iomgaa/PolyGateway
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Compare commits

base: iomgaa/PolyGateway:v1.1.0
Branches Tags
iomgaa/PolyGateway:main
iomgaa/PolyGateway:v1.3.3 iomgaa/PolyGateway:v1.3.2 iomgaa/PolyGateway:v1.3.1 iomgaa/PolyGateway:v1.3.0 iomgaa/PolyGateway:v1.2.4 iomgaa/PolyGateway:v1.2.3 iomgaa/PolyGateway:v1.2.1 iomgaa/PolyGateway:v1.2.0 iomgaa/PolyGateway:v1.1.2 iomgaa/PolyGateway:v1.1.1 iomgaa/PolyGateway:v1.1.0 iomgaa/PolyGateway:v1.0.6 iomgaa/PolyGateway:v1.0.5 iomgaa/PolyGateway:v1.0.4 iomgaa/PolyGateway:v1.0.3 iomgaa/PolyGateway:v1.0.0
...
compare: iomgaa/PolyGateway:v1.1.1
Branches Tags
iomgaa/PolyGateway:main
iomgaa/PolyGateway:v1.3.3 iomgaa/PolyGateway:v1.3.2 iomgaa/PolyGateway:v1.3.1 iomgaa/PolyGateway:v1.3.0 iomgaa/PolyGateway:v1.2.4 iomgaa/PolyGateway:v1.2.3 iomgaa/PolyGateway:v1.2.1 iomgaa/PolyGateway:v1.2.0 iomgaa/PolyGateway:v1.1.2 iomgaa/PolyGateway:v1.1.1 iomgaa/PolyGateway:v1.1.0 iomgaa/PolyGateway:v1.0.6 iomgaa/PolyGateway:v1.0.5 iomgaa/PolyGateway:v1.0.4 iomgaa/PolyGateway:v1.0.3 iomgaa/PolyGateway:v1.0.0

15 Commits
v1.1.0 ... v1.1.1

Author SHA1 Message Date
iomgaa 014fc2bfa7 chore: release 1.1.1 
Patch rather than minor: the error surface is unchanged and no public
signature moved. What downstream must notice is timing, not types — the
worst-case call duration rises to roughly max_attempts * timeout_s now
that the retry budget actually applies.

Pre-release review caught an overreaching promise in the changelog entry:
the 429 bound holds only when the stall verdict can fire at all, i.e. when
the whole scope has no progress. The verdict is a conjunction, so a call
does not die while other calls in the scope are still producing — by
design — which leaves no hard per-call ceiling in that case. That property
predates this fix and is now stated with its precondition instead of as an
unconditional guarantee.

The wiki sync in the release checklist is a no-op again: the doc site has
been down since 2026-08-02 and its landing page names CHANGELOG.md as the
version source of truth, which this commit updates.
 2026-08-06 11:57:06 -04:00
iomgaa f3e06eac89 chore: register the issue #8 design and plan in the research wiki 
Registration pages carry the chosen approach, why the split is by "which
budget the time consumes", the five rejected alternatives with reasons,
and the 3.6 correction found during independent verification.
 2026-08-06 11:09:37 -04:00
iomgaa a0a5cf7ecc fix: return 429 attempt time to the stall budget 
Independent verification found the first cut had swapped one bug for a
worse one. The budgets were split by "did we send a request", so a 429
attempt counted as productive — but 429 is exempt from the retry budget,
so its time burned neither budget. Against a queueing gateway that holds
the request for the full timeout before answering 429, a call could hang
for 301 attempts / 25.2 hours, measured, versus 301 seconds before the
change.

The split is now by which budget the time consumes: time that burns
max_attempts is excluded from stall, time that does not (429 attempts
included) belongs to stall. Measured again: back to one attempt / 301s.

Only the chat loop needs this — embedding and ocr count 429 against
max_attempts unconditionally, so the gap never existed there. The stall
verdict moved into _stalled(), which both call sites had duplicated, to
keep __call__ under the complexity gate.
 2026-08-06 10:55:51 -04:00
iomgaa bc4683d1f5 test: make the per-call clock invariant actually testable 
The concurrency case used two RetryMW instances, so instance-level sharing
was hidden by object isolation and a clock promoted to an instance
attribute passed all seven cases. Both cases now reuse one mw, and a new
one idles past the window between two calls on that instance — the shape
that would expose _entered_at pinned to process start. Mutation-checked:
promoting the clock fails the new case.
 2026-08-06 10:36:40 -04:00
iomgaa 3645e574d3 docs: record the stall metering change in architecture and changelog 
ARCHITECTURE.md 7.3 now carries the new metering and notes that the G6
ttft guard became conservative redundancy. The changelog entry leads with
what downstream must act on: the worst-case call duration rises to
max_attempts * timeout_s, and any STALL_WINDOW_S that was inflated to work
around this can go back to the default.
 2026-08-06 10:18:11 -04:00
iomgaa d05114e895 docs: align the stall window comments with the new metering 
_validate_stall still guards stall_window_s >= max ttft_timeout_s, but its
stated reason no longer holds: TTFT waiting is productive time and never
reaches the stall account. The check is harmless and stays, so the
docstring now says why it is kept rather than implying a live hazard.
.env.example dropped the "must be >= max TTFT" advice for what the window
actually measures.
 2026-08-06 10:09:06 -04:00
iomgaa 0477d9534b fix: apply the non-productive stall budget to the ocr loop 
Same failure path as the embedding loop: one timed-out attempt drains the
wall-clock window, and the next round without a runnable source declares
the scope dead in _on_no_runnable. All three governance loops now meter
stall the same way.
 2026-08-06 09:51:30 -04:00
iomgaa 6d0f3c9044 fix: apply the non-productive stall budget to the embedding loop 
The embedding loop shares the wall-clock entered_at and the same stall
verdict, so it failed the same way through a different path: one timed-out
attempt, then any round with no runnable source, and _on_no_runnable
declared the scope dead. Issue #8 only recorded the chat path; the
regression test pins this one.
 2026-08-06 09:42:07 -04:00
iomgaa 02c3d06ec6 fix: bill only non-productive waiting against the chat stall budget 
Issue #8: with timeout_s >= stall_window_s a single timed-out request
exhausted the stall window before the second attempt was even dispatched,
so LLM_MAX_RETRIES never applied and the whole scope was declared dead.

Root cause is that real attempts and non-productive waiting charged the
same wall clock, while the stall budget is the smaller of the two. The new
StallClock subtracts attempt time from the stall account, leaving the two
budgets orthogonal: attempts bill max_attempts, waiting bills
stall_window_s. The dual-condition verdict, the inf semantics of
progress_age_s, the 429 exemption and the error surface are untouched.

The productive boundary is _attempt itself, telemetry included, so a slow
recorder cannot push a call into a stalled verdict.
 2026-08-06 09:20:21 -04:00
iomgaa 573e505a4b docs: add the implementation plan for issue #8 
Six tasks: StallClock plus the chat loop, then embedding, ocr, the config
comments, the full-suite regression with doc sync, and independent
verification. Codex review raised four points, all confirmed and folded in:
a stale line reference in the fidelity section, explicit cancellation
acceptance for T2/T3 (the new attempting() wrapper now wraps their existing
cancel paths), a telemetry-boundary test pinning the design's claim that
telemetry jitter must not feed the stall verdict, and concrete test
construction for the embedding/ocr regressions.
 2026-08-06 09:09:31 -04:00
iomgaa ce2dda7d45 docs: sharpen the productive-time boundary after Codex review 
Two internal-consistency fixes from the independent design review:
the 429 saturation argument wrongly claimed exponential backoff growth
(429 skips the retry budget, so max(fails, 1) pins the delay to the base
tier), and "productive" was defined as waiting on the response while the
StallClock actually wraps all of _attempt. The boundary is now stated as
_attempt itself, including per-attempt accounting and telemetry, with the
rationale that telemetry jitter must not participate in the stall verdict.
 2026-08-06 08:16:07 -04:00
iomgaa bfe423ddf8 docs: bill only non-productive waiting against the stall budget 
Issue #8: a single request that burns its full timeout_s also exhausts
stall_window_s, so the retry budget silently never applies. Root cause is
that both budgets charge the same wall-clock time. The design makes the two
budgets orthogonal — real attempts bill the retry budget, everything else
bills the stall budget — which drops the timeout_s / stall_window_s coupling
instead of guarding it with an assembly-time check.
 2026-08-06 08:01:07 -04:00
iomgaa 9c2824ce8a fix: admit governance backend failures into scope-level unavailability (issue #7) 
A fail-closed limiter or breaker backend means the scope cannot emit a
single request, yet GovernanceBackendError sat directly under
PolyGatewayError. A caller writing only `except GatewayUnavailableError`
dropped it into the catch-all branch, so a Redis blip burned a backlog's
business failure budget into the dead letter queue over a fault a restart
would clear. It now inherits GatewayUnavailableError with a
governance_backend_down reason and a 5 second retry_after_s.

The two unknown-source sites split out into SourceNotConfiguredError,
deliberately outside the retryable family: a misconfigured source name
must burn its budget and surface rather than retry forever in silence.

README now states which errors reach callers and which the retry loop
absorbs. TransientError and SourceDeadError read like caller contracts but
never arrive, and a downstream project wrote a whole design section on
that false premise before checking the source.

Independent verification caught the split not actually holding on the only
path production uses, and caught the fix for that opening a second hole on
the accounting path. Both are fixed and pinned by tests that go through
the wrappers rather than the private methods underneath them.
 2026-08-06 06:55:38 -04:00
iomgaa 5853c3f8ff fix: keep the accounting path degrading after the wrapper change 
Letting SourceNotConfiguredError through the gate wrappers opened a hole
the recheck caught: _record_quietly only degrades GovernanceBackendError,
so an assembly defect raised from the accounting side would now escape and
destroy a response from a call that had already genuinely succeeded. That
inverts the exact invariant _record_quietly exists to hold.

Widening _record_quietly is the right fix rather than narrowing the
wrappers, because that layer degrades by what the path is (accounting, the
call is already done) rather than by which error type shows up. Narrowing
would have left 4 of 9 wrapper methods as exceptions to a rule nobody can
remember.

No backend raises it from an accounting method today, so this is a
guardrail for whoever adds source-name validation to a breaker backend.

The stub that first reported this green was wrong: its record_success
lacked count_attempt, so it raised TypeError and the wrapper relabeled it.
Fixed signature, then the test failed as it should have.

Also finishes the three-to-five leak path correction across the four
remaining spots, including the wiki summary card that indexes this design.
 2026-08-06 06:39:52 -04:00
iomgaa a57a5cea72 fix: let assembly defects pierce the gate wrappers 
Independent verification caught that the split shipped in the previous
commit did not actually hold on the only path production uses. The gate
wrappers re-raise GovernanceBackendError but nothing else, so
SourceNotConfiguredError fell into the following `except Exception` and
came back out as a governance_backend_down failure with retry_after_s=5.0.
A misconfigured source name would still retry forever and never surface.

The existing tests missed it because both of them call the private _cfg()
directly, one layer below the wrapper the governance loops actually go
through. The regression test goes through QuotaGate.

telemetry.py has to widen its terminal catch in the same commit: once the
wrapper stops relabeling the error, it is no longer a GovernanceBackendError,
and it is raised before any attempt exists, so the path would have recorded
no telemetry at all.

Also corrects the leak path count from three to five. QuotaGate.stats and
BreakerGate.retry_after_s are not wrapped by _record_quietly either.
 2026-08-06 05:57:50 -04:00
26 changed files with 1269 additions and 79 deletions
Expand all files Collapse all files
.env.example
+1 -1
View File
@@ -38,7 +38,7 @@ LLM_CIRCUIT_BREAKER_COOLDOWN=60 # 或 LLM__BREAKER__COOLDOWN_S
# LLM_TTFT_TIMEOUT=30 # 平铺看门狗缺省(成对生效) # LLM_TTFT_TIMEOUT=30 # 平铺看门狗缺省(成对生效)
# LLM_INTER_TOKEN_TIMEOUT=15 # LLM_INTER_TOKEN_TIMEOUT=15
# LLM__BREAKER__PROBE_TTL_S=240 # 缺省派生: max(2×最大源超时, cooldown, 最大源超时+5);显式值须 ≥ 最大源超时+5 # LLM__BREAKER__PROBE_TTL_S=240 # 缺省派生: max(2×最大源超时, cooldown, 最大源超时+5);显式值须 ≥ 最大源超时+5
# LLM__BACKPRESSURE__STALL_WINDOW_S=300 # stall 双条件判死窗口;须 ≥ 最大源 TTFT # LLM__BACKPRESSURE__STALL_WINDOW_S=300 # stall 双条件判死窗口;只计非生产性等待(429 退避/配额轮询/熔断冷却),与 TIMEOUT_S 无耦合,无需按 timeout×retries 放大
# LLM__BACKPRESSURE__POLL_INTERVAL_S=0.05 # LLM__BACKPRESSURE__POLL_INTERVAL_S=0.05
# LLM__SELECTOR=health_aware # health_aware(默认,M2.5) | round_robin | least_inflight # LLM__SELECTOR=health_aware # health_aware(默认,M2.5) | round_robin | least_inflight
# ── M2.5 失败率熔断通道(可选,缺省即生产推荐值)── # ── M2.5 失败率熔断通道(可选,缺省即生产推荐值)──
CHANGELOG.md
+22 -2
View File
@@ -1,5 +1,25 @@
# Changelog # Changelog
## 1.1.1(2026-08-06)
stall 判定改为非生产性等待口径(issue #8)。`timeout_s ≥ stall_window_s` 时,**一次耗满超时的请求就会让整个 scope 被判死,配置的重试次数一次都用不上**——而且没有任何报错或 warning,配置方以为自己配了 3 次重试。`stall_window_s` 默认 300 恰是个很容易被 `TIMEOUT_S` 追平的值,"只配 timeout、不配 stall"这种最常见的写法正好踩中。
根因是**两个预算重叠计费**: 真实尝试的耗时同时向重试预算(`max_attempts`)与 stall 预算(`stall_window_s`)计费,而后者更小,必然先耗尽。
### 行为变更(**请先读这一条**)
- **stall 判定的"本地超窗"条件现在只累计非生产性等待**——429 退避、配额 wait 轮询、熔断冷却;消耗重试预算的真实尝试不再计入。两个预算自此正交,划分依据是**谁消耗重试预算**: 烧 `max_attempts` 的时间不烧 `stall_window_s`,不烧 `max_attempts` 的时间(含 429 尝试本身)归 `stall_window_s` 治理。
- **`stall_window_s` 与 `timeout_s` 不再有任何耦合**,无需按 `timeout × retries` 放大。若你此前为绕开本 bug 把 `STALL_WINDOW_S` 调大过,现在可以回到默认值。
- **单次调用的最坏耗时由 `stall_window_s` 抬升到约 `max_attempts × timeout_s`**(默认配置下 3 × `TIMEOUT_S`,再加各次退避)。这是重试预算恢复生效的正确表现,但如果你的上游有调用超时,请据此复核。429 路径同样不突破这个量级——429 虽免重试预算,但其尝试耗时计入 stall 账。
**上述量级的前提是 stall 判死能够触发**,即整个 scope 无进展(`progress_age_s() > stall_window_s`)。判死是**双条件合取**,这一条未变: 若同 scope 里其他调用仍在正常出餐,本调用会继续等待换源而不判死——这正是双条件的设计意图("别人还活着,不该因我一路不顺就宣告整个 scope 死亡")。**代价是这种情形下调用级没有硬上限**,持续遭遇慢 429 的调用可以等很久。该性质由条件 B 单独门控,**早于本次修复即如此**(旧口径实测同样无界),不是本次引入;但若你需要调用级硬上限,请在调用方用 `asyncio.wait_for` 自行设置。
- 三条治理循环(chat / embedding / ocr)口径一致。**embedding 与 ocr 此前有同一缺陷**(经"先超时一次、再遇到无可用源"触发),issue 只记录了 chat 路径。
- 遥测收尾属"真实尝试"边界之内,**遥测抖动不会把一次调用推进 stalled 判决**。
### 不变
- 双条件判死的结构、`progress_age_s()` 的 `inf` 语义(从未出餐 = 全局超窗)、429 免预算、退避与 jitter 公式、`fail_fast` 分支、`AllSourcesExhausted` 的字段与 `reason` 取值(仍是 `stalled`)全部未动。**错误面零变更**,下游 `except` 写法不受影响。
- 装配期校验 `stall_window_s ≥ 最大源 ttft_timeout_s` 保留。新口径下它已是保守冗余(TTFT 等待属生产性时间),但无害且不误拒合理配置。
## 1.1.0(2026-08-06) ## 1.1.0(2026-08-06)
治理后端故障归位为 scope 级不可用(issue #7)。限流/熔断的状态后端(Redis 等)自身故障时,库按降级方向铁律 fail-closed——**整个 scope 一个请求都发不出去**,语义上就是"scope 级暂时不可用"。但 `GovernanceBackendError` 此前是 `PolyGatewayError` 的直接子类,只写 `except GatewayUnavailableError` 的调用方接不住,后果很具体: Redis 抖一下,积压任务一批批消耗业务失败预算,够到上限就进死信——**而那是运维重启一下就好的故障**。 治理后端故障归位为 scope 级不可用(issue #7)。限流/熔断的状态后端(Redis 等)自身故障时,库按降级方向铁律 fail-closed——**整个 scope 一个请求都发不出去**,语义上就是"scope 级暂时不可用"。但 `GovernanceBackendError` 此前是 `PolyGatewayError` 的直接子类,只写 `except GatewayUnavailableError` 的调用方接不住,后果很具体: Redis 抖一下,积压任务一批批消耗业务失败预算,够到上限就进死信——**而那是运维重启一下就好的故障**。
@@ -19,8 +39,8 @@
### 下游请读 ### 下游请读
- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`**,与 `AllSourcesExhausted` 同款;`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。 - **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`**,与 `AllSourcesExhausted` 同款(`per_source_reasons` 属性存在但恒为 `{}`——后端故障不针对具体某个源);`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。
- **三条闸门路径**(`try_acquire` / `try_enter` / `progress_age_s`)的后端故障会到达调用方;记账路径(`record_success` 等)仍被 `_record_quietly` 降级为 warning,这个分工不变。 - **五条闸门路径**的后端故障会到达调用方: `QuotaGate` 的 `try_acquire` / `stats` / `progress_age_s`,`BreakerGate` 的 `try_enter` / `retry_after_s`。记账路径(`record_success` / `record_failure` / `release_probe` / `mark_progress`)仍被 `_record_quietly` 降级为 warning,这个分工不变。
- **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。 - **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。
## 1.0.6(2026-08-02) ## 1.0.6(2026-08-02)
pyproject.toml
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "polygateway" name = "polygateway"
version = "1.1.0" version = "1.1.1"
description = "PolyGateway：实验室统一的大语言模型（LLM/VLM/OCR）调度与中转库——多源、限流、重试、熔断、缓存、遥测" description = "PolyGateway：实验室统一的大语言模型（LLM/VLM/OCR）调度与中转库——多源、限流、重试、熔断、缓存、遥测"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [
research-wiki/ARCHITECTURE.md
+2 -1
View File
@@ -427,8 +427,9 @@ flowchart TB
- `RedisLimiter`: 移植 CHSAnalyzer 六道闸——单条 Lua 原子检查全局并发/单源并发(ZSET 租约)/全局 RPM/单源 RPM/全局 TPM/单源 TPM;窗口 id 用 **Redis 服务器时钟**(TIME 命令)统一多进程口径。随实现移植契约测试。 - `RedisLimiter`: 移植 CHSAnalyzer 六道闸——单条 Lua 原子检查全局并发/单源并发(ZSET 租约)/全局 RPM/单源 RPM/全局 TPM/单源 TPM;窗口 id 用 **Redis 服务器时钟**(TIME 命令)统一多进程口径。随实现移植契约测试。
- `InMemoryLimiter`: 同一契约的进程内实现(semaphore + 滑动窗口计数);单进程场景下语义等价。 - `InMemoryLimiter`: 同一契约的进程内实现(semaphore + 滑动窗口计数);单进程场景下语义等价。
- **配额满行为可配**: `wait`(等待,配 stall 判定——本地等待超窗 + 全局无进展超窗双条件才判卡死)或 `fail-fast`(立即抛)。 - **配额满行为可配**: `wait`(等待,配 stall 判定——本地等待超窗 + 全局无进展超窗双条件才判卡死)或 `fail-fast`(立即抛)。
- **stall 计时口径(2026-08-06 修正,issue #8,设计 `designs/2026-08-06-issue8-stall-budget-design.md`)**: 双条件的**条件 A 只累计非生产性等待**(429 退避、配额 wait 轮询、熔断冷却),真实尝试的耗时由 `StallClock.attempting()` 从 stall 账中扣除。原实现用墙钟总耗时,使真实尝试同时向重试预算与 stall 预算计费;而 stall 预算(默认 300s)小于重试预算(`max_attempts × timeout_s`),必然先耗尽——`timeout_s ≥ stall_window_s` 时一次超时即判 scope 死,`max_attempts` **静默失效**。修正后两个预算正交,**划分依据是"谁消耗重试预算"而非"是否发出请求"**: 烧 `max_attempts` 的时间不烧 `stall_window_s`,不烧 `max_attempts` 的时间归 `stall_window_s`。**429 尝试因此也计入 stall 账**——它免重试预算,若其耗时又算生产性就两个预算都不烧,排队型网关(持满 timeout 才回 429)下调用可挂 25 小时(实施期独立验证实测,见设计 §3.6)。生产性边界即 `_attempt` 边界(含该次记账与遥测收尾),故遥测抖动不参与判死。`stall_window_s` 与 `timeout_s` 自此**无耦合**,无需按 `timeout × retries` 放大。三条治理循环(chat/embedding/ocr)共用 `middleware/retry.py` 的 `StallClock`。条件 B 的 `inf` 语义未动——新口径下"非生产性排队耗满窗口且 scope 从未出餐"判死本就正当。**残余性质(非本次引入,由条件 B 单独门控)**: 判死是双条件合取,故当同 scope 其他调用仍在正常出餐时本调用不判死(设计意图: 别人还活着就不该宣告 scope 死亡),代价是**该情形下调用级无硬上限**——持续遭遇慢 429 的调用可以等很久;需要硬上限的调用方应自行 `asyncio.wait_for`。
- 全局活性信号: `mark_progress()`/`progress_age_s()`("最近一次出餐"时刻)供背压 stall 判定,移植 `CHSAnalyzer limiter.py:193`。 - 全局活性信号: `mark_progress()`/`progress_age_s()`("最近一次出餐"时刻)供背压 stall 判定,移植 `CHSAnalyzer limiter.py:193`。
- **契约补强(2026-07-20,CHS 迁移缺口 G6)**: `settle()`/`release()` 幂等(重复调用无副作用);装配期守卫——`timeout_s ≤ permit 租约 TTL`(防租约先于请求过期)、`stall_window ≥ 最慢源 TTFT 上限`(防误判卡死),违反直接报错拒绝装配。降级方向细化(2026-07-20 M1): "报错不放行"适用于**准入侧**(try_acquire/try_enter 及选源路径消费的 source_stats/retry_after_s);已成功调用后的 settle/release 释放侧失败降级 warning——释放失败不构成放行,且不得掩盖主异常与取消。**勘误(2026-07-20 M2 设计,人类批准)**: 记账侧的 `record_success`/`record_failure`/`mark_progress` 同归此类——调用已真实完成,后端失败若冒泡会丢弃真实成功响应或掩盖原始尝试异常,故降级 warning(CHS 原版一律报错,此为有意反转;丢一次熔断记账最多延迟状态迁移且方向偏保守,epoch fencing 防污染)。 - **契约补强(2026-07-20,CHS 迁移缺口 G6)**: `settle()`/`release()` 幂等(重复调用无副作用);装配期守卫——`timeout_s ≤ permit 租约 TTL`(防租约先于请求过期)、`stall_window ≥ 最慢源 TTFT 上限`(防误判卡死;**issue #8 后为保守冗余**——TTFT 等待属生产性时间已不计入 stall,该误判在机制上不再可能,校验保留因其无害且不误拒合理配置),违反直接报错拒绝装配。降级方向细化(2026-07-20 M1): "报错不放行"适用于**准入侧**(try_acquire/try_enter 及选源路径消费的 source_stats/retry_after_s);已成功调用后的 settle/release 释放侧失败降级 warning——释放失败不构成放行,且不得掩盖主异常与取消。**勘误(2026-07-20 M2 设计,人类批准)**: 记账侧的 `record_success`/`record_failure`/`mark_progress` 同归此类——调用已真实完成,后端失败若冒泡会丢弃真实成功响应或掩盖原始尝试异常,故降级 warning(CHS 原版一律报错,此为有意反转;丢一次熔断记账最多延迟状态迁移且方向偏保守,epoch fencing 防污染)。
### 7.4 熔断 ### 7.4 熔断
research-wiki/designs/2026-08-06-governance-backend-error-design.md
+2 -2
View File
@@ -20,7 +20,7 @@
| Issue 原文 | 实际情况 | | Issue 原文 | 实际情况 |
|---|---| |---|---|
| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **三条**。`middleware/retry.py:216` 每轮循环开头的 `progress_age_s()` 同样在 catch 之外,直达调用方 | | 泄漏路径为 `try_enter` / `try_acquire` 两条 | **五条**(设计初稿写"三条",2026-08-06 独立验证时核出遗漏两条并订正): `QuotaGate` 的 `try_acquire` / `stats`(`retry.py:249`)/ `progress_age_s`(`retry.py:216`、`:305`),`BreakerGate` 的 `try_enter` / `retry_after_s`(`retry.py:292`、`:310`)。判据是该调用点是否被 `_record_quietly` 包裹——未包裹即直达调用方;OCR 与 Embedding 两个治理循环有同构的对应点 |
| (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 | | (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 |
| 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug | | 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug |
| `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 | | `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 |
@@ -131,7 +131,7 @@ Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重
| 测试 | 位置 | 先失败后通过的证据 | | 测试 | 位置 | 先失败后通过的证据 |
|---|---|---| |---|---|---|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 | | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
| 三条泄漏路径(`try_acquire`/`try_enter`/`progress_age_s`)抛出的异常携带正确 `scope` 与非零 `retry_after_s` | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` | | 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 | | `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 | | 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" | | Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" |
research-wiki/designs/2026-08-06-issue8-stall-budget-design.md
+254
View File
@@ -0,0 +1,254 @@
# stall 判定改为非生产性等待口径设计(Issue #8)
- **日期**: 2026-08-06
- **来源**: Gitea Issue #8(本机全套件跑 391.67s,1 failed;失败源于单次 300s 超时耗尽 stall 窗口,基于 1.1.0 源码核查)
- **状态**: **已批准(2026-08-06)**,待 `writing-plans`
- **触发档位**: 强制(变更治理行为——判死条件的度量口径,是库对下游的承诺)
- **方案范围**: 人类明确要求单一方案,故本文不列平行备选,仅在 §5 记录被否决路线及否决理由(体例沿用 Issue #7 设计)
## 1. 目标与非目标
| | 内容 |
|---|---|
| **G1** | 消除"单次超时即判 scope 级死亡"——`timeout_s` 与 `stall_window_s` 的隐式耦合彻底解除,重试预算在超时场景下真实可用 |
| **G2** | 使 stall 判定的度量对象与它的职责一致:**它治理的是无人治理的非生产性循环,不是已被重试预算治理的真实尝试** |
| **G3** | 三条治理循环(chat / embedding / ocr)口径一致,计时逻辑收敛为单一共享单元,杜绝第四次复制 |
| **G4** | 配置方不再需要心算 `stall_window > timeout × max_attempts`;`.env.example` 注释与实际语义对齐 |
| **非目标** | 不改 `progress_age_s()` 的 `inf` 语义(见 §3.4);不新增装配期校验(见 §5.2);不新增配置项;不改 `AllSourcesExhausted` 的字段与 `reason` 取值;不给 embedding/ocr 新增主循环判死路径(见 §5.4);不改 429 免预算、AIMD、选源、熔断任何既有行为 |
### 1.1 Issue 前提的三处修正(按 1.1.0 源码核实)
| Issue 原文 | 实际情况 |
|---|---|
| 失效点为 `retry.py:216` 一处 | **三处同构**:`retry.py:216`(主循环)、`retry.py:305` / `embedding.py:247` / `ocr.py:272`(`_on_no_runnable`)。四个判定点共用同一个墙钟 `entered_at`,故 embedding/ocr 在"先超时一次、再遇到无可用源"时同样误判——issue 只覆盖了 chat |
| 建议方向 1:装配期校验 `stall_window_s > max(timeout_s)` | **不采纳**。它把耦合固化成契约而非消除耦合,且约束值须为 `timeout × max_attempts`(本机即 900s),会让 stall 兜底迟钝到近乎失效。详见 §5.1 |
| 建议方向 2:`inf` 不参与判死 | **不采纳**。在新口径下 `inf` 从"有害恒真"变回"正确的保守默认";且它会反转已被测试钉住的既有行为。详见 §3.4 与 §5.3 |
## 2. 根因:两个预算重叠计费
`retry.py:214-215` 的注释自述这处判定是「429 免预算后的兜底,防饱和期无限循环」——它治理的对象是**非生产性循环**。但条件 A `now - entered_at > stall` 度量的是**墙钟总耗时**,无法区分两类性质相反的时间:
| 时间性质 | 构成 | 应由谁治理 | 耗尽后 |
|---|---|---|---|
| **生产性** | 一次尝试的完整生命周期(发请求、等响应含耗满 `timeout_s` 的超时/TTFT/流式读取,以及该次尝试的记账与遥测收尾) | `max_attempts`(重试预算) | `retry_exhausted` |
| **非生产性** | 429 退避、配额 wait 轮询、熔断冷却轮询、AIMD 排队 | **无人治理**(429 不计 `fails`)→ 正是 stall 的职责 | `stalled` |
**缺陷即:生产性时间同时向两个预算计费。** 而 stall 预算(默认 300s)远小于重试预算(`3 × 300s`),必然先耗尽,于是重试预算在超时场景下**永远用不上**——issue 观察到的"静默失效"就是这个重叠计费的直接后果。
`.env` 里 `TIMEOUT_S=300` 与 `_DEFAULT_STALL_WINDOW_S=300.0`(`config.py:60`)相等只是把它暴露得最快;只要 `timeout_s ≥ stall_window_s / 1`,一次超时就够。
### 2.1 两条佐证:`inf` 恒真是遗漏而非设计
| 证据 | 出处 | 含义 |
|---|---|---|
| `_PROGRESS_TTL_S = 3600 # 远大于任何 stall_window,防进度键过期造成假停滞` | `backends/redis/limiter.py:32` | 「无 progress 记录 ≠ 停滞」早已是设计共识,作者用超长 TTL 规避了"键过期"这一路径,但 TTL 再长也救不了"**从来没写过**"——冷启动是同类情形的漏网之鱼 |
| `test_global_stale_but_local_fresh_keeps_waiting` docstring 写「仅全局超窗(从未出餐 age=inf)」 | `tests/unit/test_backpressure.py:120-121` | 现有测试把 `inf` 当作"全局超窗成立"钉住了;`test_both_windows_exceeded_raises_stalled`(:89)更是**全靠 `inf` 恒真**才能触发判死 |
## 3. 选定方案:双预算正交模型
### 3.1 一句话
**stall 计时器只累计非生产性等待时间**:`stalled_s = (now − entered_at) − 真实尝试累计耗时`。
两个预算自此正交,各管一段,无缝覆盖调用的全部时间:
| 花在哪 | 烧哪个预算 |
|---|---|
| 真实尝试(`_attempt` 内),**429 除外** | 重试预算 `max_attempts` |
| 其余一切等待,**含 429 尝试本身** | stall 预算 `stall_window_s` |
> **划分依据是"谁消耗重试预算",不是"是否发出了请求"**(2026-08-06 实施期订正,见 §3.6)。初稿按后者划分,使 429 尝试两个预算都不烧。
**"生产性"的边界即 `_attempt` 的边界**——包含该次尝试的记账(`record_success`/`mark_progress`)与遥测收尾,而不止于"等响应"。这是有意的:这些收尾是"尝试已有结论"之后的动作,不是"在等待重试机会"的停滞;把它们计入 stall 会让遥测抖动参与判死,与「遥测写失败降级不冒泡」所守的"遥测不得影响主路径判决"同精神。其耗时本也在毫秒量级。
这与库内既有原则**同构**:429 不烧重试预算,所以 429 等待烧 stall 预算;真实尝试烧重试预算,所以它不烧 stall 预算。
### 3.2 为什么取补集,而不是逐处标记 sleep
两种实现都能达到 §3.1 的语义,选**取补集**(总时间减去 `_attempt` 耗时):
| 维度 | 取补集(选定) | 逐处标记 sleep(否决) |
|---|---|---|
| 埋点数量 | 每条循环 **1 处**(`_attempt` 调用点) | chat 3 处、embedding/ocr 各 2 处,共 7 处 |
| 演进安全性 | **默认安全**:将来新增任何等待路径自动计入 stall,兜底不会漏 | 默认危险:新增等待路径若忘记标记,即成新的 stall 盲区 |
| 语义可读性 | 「stall 时间 = 总时间 − 花在真实尝试上的时间」,一句话说清 | 需读者遍历全部标记点才能确认覆盖完整 |
`_attempt` 是纯生产性的:permit 获取、熔断准入、AIMD 判定全部在 `_pick_runnable` 内完成,`_attempt` 进入时已持 permit,内部只做"发请求 + 记账"。故补集口径不会把非生产性时间误算为生产性。
### 3.3 共享单元:`StallClock`
计时逻辑提取为 `middleware/retry.py` 的模块级小类,embedding/ocr 复用——沿用 `backoff_delay` 已被两者复用的既有手法(`tests/unit/test_backpressure.py:258` 记录该先例),不新建模块、不动依赖层次。
```python
class StallClock:
"""调用级 stall 计时器: 只累计非生产性等待(设计 §3.1)。
实例per调用创建, 严禁提升为实例属性——并发调用共享会互相污染。
"""
def __init__(self, now: Callable[[], float]) -> None:
self._now = now
self._entered_at = now()
self._productive_s = 0.0
def stalled_s(self) -> float:
return self._now() - self._entered_at - self._productive_s
@contextlib.asynccontextmanager
async def attempting(self):
started = self._now()
try:
yield
finally:
# 只做算术, 不吞任何异常——CancelledError 照常穿透(库铁律)
self._productive_s += self._now() - started
```
调用点改动(三处循环同款):
```python
clock = StallClock(self._now) # 替换 entered_at = self._now()
...
if clock.stalled_s() > stall and await self._quota.progress_age_s() > stall:
raise AllSourcesExhausted(..., reason="stalled", ...)
...
async with clock.attempting(): # 包裹真实尝试
outcome = await self._attempt(request, *picked, reasons, attempt_fails)
```
`_on_no_runnable` 的形参由 `entered_at: float` 改为 `clock: StallClock`(三处同改)。
### 3.4 `inf` 语义为何不动(本设计的核心权衡)
新口径下第一象限的含义变为:「**非生产性排队已耗满 `stall_window_s`,且整个 scope 从未出餐**」。此时判死是正当的——真的没有任何证据表明这个 scope 还活着,而调用方已经白等了一整个窗口。`inf` 由此从"有害的恒真"回归为"正确的保守默认"。
反过来,若同时改 `inf` 语义:
- 冷启动窗口内 stall 判定**完全失效**,429 饱和场景下 chat 主循环重新暴露无限循环风险(429 不计 `fails`,无其他兜底);
- 会反转 `test_both_windows_exceeded_raises_stalled` 钉住的行为,并与 CHS 保真蓝本分叉。
**一次改动解决问题,优于两次改动互相牵制。** 这是本设计只动条件 A 的理由。
### 3.5 429 饱和场景下兜底仍然有效(正确性验证)
修改后必须确认 stall 兜底没有被削弱。429 免预算使 `fails` 恒为 0,`retry.py` 的 `max(fails, 1)` 令退避恒定在 `backoff_base_s` 档(或取 `Retry-After` 提示的较大值),不随轮次增长。每轮构成为「一次 429 往返」+「一段恒定退避 sleep」,后者非生产性且每轮累加,`stalled_s` 单调逼近 `stall_window_s`,兜底有效。
**但这个论证在初稿里依赖一个未加保护的假设**:「429 往返是快速失败,毫秒至秒级」。§3.6 处理它不成立的情形。
### 3.6 订正:429 尝试必须退还给 stall 账(2026-08-06 实施期,独立验证发现)
**缺陷**:初稿按"是否发出请求"划分两个预算,于是 429 尝试的耗时算生产性。但 429 **不消耗重试预算**——它于是**两个预算都不烧**,掉进缝隙。§3.1 初稿声称的"无缝覆盖调用的全部时间"因此不成立。
**后果实测**(排队型网关:持满 `timeout_s` 才回 429,`timeout=300 / stall=300 / backoff_base=2 / rng=0`):
| | 尝试次数 | 墙钟 |
|---|---|---|
| 修复前(main) | 1 | 301s |
| 初稿口径 | **301** | **90,601s ≈ 25.2 小时** |
| 订正后 | 1 | 301s |
即初稿把一个 bug 换成了一个更严重的 bug——25 小时的挂起。
**订正**:划分依据改为**"谁消耗重试预算"**。429 免重试预算 → 429 尝试的耗时归 stall 治理,由 `StallClock.attempting()` yield 的句柄 `refund()` 退还。缝隙就此闭合,且这条规则比初稿更本质:两个预算按"由谁治理"划分,而非按"是否发出请求"这个表象。
**影响范围仅 chat**:embedding/ocr 无 429 免预算(无条件 `fails += 1`),429 照常烧重试预算,不存在缝隙,无需改动(与 §5.4 的分析一致)。
## 4. 旧版行为审计(stall 子系统逐条)
| 既有行为 | 处置 | 说明 |
|---|---|---|
| 双条件判死(本地超窗 ∧ 全局无进展超窗) | **保留** | 结构不变,只改条件 A 的度量口径 |
| 条件 A = 调用级累计、循环内不重置(CHS `governance.py:207`) | **保留** | `StallClock` 同样每调用一个实例、循环内不重置 |
| 条件 A 计入真实尝试耗时 | **替换** | 本设计的唯一行为变更 |
| 条件 B `progress_age_s()`,`inf` = 从未进展 | **保留** | 见 §3.4 |
| 本地 monotonic 与后端时钟刻意不混用 | **保留** | `StallClock` 只用注入的 `self._now`,不读后端时钟 |
| poll jitter ∈ [0.5p, 1.0p] 防惊群 | **保留** | 不触碰 |
| `fail_fast` 不进入 stall 判定 | **保留** | 不触碰 |
| 429 免预算(chat 独有) | **保留** | 不触碰;embedding/ocr 无此逻辑,故无对应缺口(§5.4) |
| `AllSourcesExhausted(reason="stalled")` 及其 `retry_after_s` 取值 | **保留** | 错误面零变更,下游 `except` 写法不受影响 |
**有意放弃**: 无。本设计不删除任何既有行为。
## 5. 被否决的路线
### 5.1 装配期校验 `stall_window_s > max(timeout_s)`(Issue 建议方向 1)
否决理由三条:
1. **治标**。它把"两个预算重叠计费"这个缺陷固化成一条配置契约,要求配置方绕开它,而不是消除它。
2. **约束值不可接受**。要让重试预算真正可用,须 `stall_window > timeout × max_attempts`(本机 900s)。stall 兜底随之迟钝到 900s 才触发,饱和期无限循环的防护近乎失效——**修好一个洞,挖开另一个**。
3. **挡不住残余情形**。即便配到 1200s,一次调用若在 429 轮询与超时上累计超过 1200s,条件 B 的 `inf` 仍恒真,双条件仍退化为单条件。坑只是被推远。
新口径下 `stall_window_s` 与 `timeout_s` 不再有任何耦合,**这条校验没有存在的理由**——不加校验、而是消除掉需要校验的耦合。
### 5.2 既有校验 `stall_window_s ≥ max(ttft_timeout_s)` 的处置
`config.py:240-247` 的 `_validate_stall` 被 `ARCHITECTURE.md` §7.3 记为契约补强 G6。新口径下 TTFT 等待属生产性时间,其 docstring 的理由「防把正常慢首包误判为卡死」**已不成立**。
**人类已定夺:保留校验,改写 docstring 说明新口径**。校验本身无害(不会误拒任何合理配置),保留可避免改动 ARCHITECTURE.md 既有契约、把本次改动的影响面控制在最小。docstring 改为说明"该校验在新口径下为保守冗余,TTFT 已不计入 stall"。
### 5.3 `inf` 不参与判死(Issue 建议方向 2)
见 §3.4:新口径下 `inf` 已无害,单独改它会制造冷启动兜底真空并反转既有测试。
### 5.4 给 embedding/ocr 补主循环 stall 判定
设计过程中一度提出(前提是"429 饱和时它们没有防无限循环兜底"),**核实后前提不成立,故否决**:
| 循环路径 | embedding/ocr 的兜底 |
|---|---|
| `picked is None` → `_on_no_runnable` 轮询(不烧 `fails`) | `_on_no_runnable` 内已有 stall 判定(`embedding.py:247` / `ocr.py:272`)✓ |
| 尝试失败 → `fails += 1` | `max_attempts` ✓ |
`retry.py:233-234` 的 429 免预算分支是 chat **独有**的(`embedding.py:191`、`ocr.py:216` 均为无条件 `fails += 1`,两文件亦无 `pacer`),主循环判定正是为它打的补丁。embedding/ocr 两条路径均已封闭,补齐等于凭空新增一条判死路径,使其比 chat 更易判死——纯 gold-plating。
## 6. 非功能维度
| 维度 | 回答 |
|---|---|
| **并发** | `StallClock` **每次调用创建一个实例**,是调用级局部状态,与被替换的 `entered_at` 局部变量同性质。严禁提升为实例属性(并发调用会互相污染计时)——docstring 已写明,单测钉住并发两路调用互不干扰 |
| **取消** | `attempting()` 的 `finally` 只做浮点加法,不含 `await`、不捕获任何异常,`CancelledError` 逐字穿透。既有 `test_cancellation_pierces_wait_loop` 继续有效,并新增一条"取消发生在 `_attempt` 内"的用例 |
| **降级方向** | 不变。stall 判定读取的 `progress_age_s()` 属准入侧,后端故障仍 fail-closed 抛 `GovernanceBackendError`(scope 级),不放行 |
| **幂等与重复** | `stalled_s()` 是纯读,可任意次调用;`attempting()` 可重入多次(每次尝试一次),累加语义天然幂等于"总生产性时间" |
| **持久化与原子性** | 不适用。纯进程内计时,无落盘、无后端写入,不新增任何 Redis 往返 |
| **性能** | 每次尝试新增两次 `self._now()` 调用与一次浮点加法,可忽略 |
## 7. 错误处理与测试策略
**错误分类**: 无变更。判死仍抛 `AllSourcesExhausted(reason="stalled")`,属 scope 级不可用(`GatewayUnavailableError` 家族),下游延期重投语义不变。
### 7.1 回归证据(先失败后通过)
核心用例 `test_single_timeout_does_not_exhaust_stall_budget`:`stall_window_s == timeout_s == 300`,第一次尝试推进 `FakeClock` 超过 300s 后抛 `TransientError`,第二次返回成功。
- **改前**:第二次尝试发出前即被判死,抛 `AllSourcesExhausted(reason="stalled")` → **失败**
- **改后**:重试预算正常生效,返回成功响应 → **通过**
embedding / ocr 各一条同构用例(经"先超时一次、再遇到无可用源"触发 `_on_no_runnable`)。
### 7.2 其余用例
| 用例 | 钉住什么 |
|---|---|
| 四象限现有四条(`TestStallQuadrants`) | 非生产性路径行为逐字不变;`test_both_windows_exceeded` 全程无真实尝试,`stalled_s` 等价于旧墙钟,**应原样通过** |
| `test_productive_time_excluded_from_stall` | 直接断言:仅靠真实尝试耗时无论多久都不触发判死 |
| `test_nonproductive_wait_still_triggers_stall` | 反向:纯轮询等待累满窗口仍正常判死(兜底未被削弱) |
| `test_saturation_429_still_stalls` | §3.5 的正确性验证:429 连续拒绝 + 退避,最终仍判死而非无限循环 |
| `test_cancel_inside_attempt_pierces` | 取消穿透 `attempting()` 的 `finally` |
| `test_concurrent_calls_do_not_share_clock` | 两路并发调用,一路长尝试不影响另一路的 stall 账 |
Redis 后端无需新增用例:本设计不改后端接口与 `progress_age_s()` 语义。
## 8. 交付清单(供 `writing-plans` 展开)
| # | 内容 |
|---|---|
| T1 | `middleware/retry.py` 新增 `StallClock`;主循环与 `_on_no_runnable` 改用之 |
| T2 | `embedding.py` / `ocr.py` 复用 `StallClock`,`_on_no_runnable` 形参改签名 |
| T3 | `config.py:240-247` `_validate_stall` docstring 改写(§5.2) |
| T4 | 测试:§7.1 回归三条 + §7.2 五条;`test_backpressure.py:121` docstring 订正 |
| T5 | `.env.example:41` 注释改写(删除误导性的"须 ≥ 最大源 TTFT",说明新口径);本机 `.env:37` 的临时缓解 `STALL_WINDOW_S=1200` 可回退默认(不入库,仅记录) |
| T6 | `ARCHITECTURE.md` §7.3 背压条目补记新口径与本设计指针;`CHANGELOG.md` 记治理行为变更 |
| T7 | Wiki 同步(`docs-convention.md` §2「治理行为变更」行):`解释-治理行为` + `指南-限流与熔断` |
**副作用提醒**: 修复后单次调用最坏耗时由 `stall_window_s` 抬升至 `max_attempts × timeout_s`(本机 900s)——这是重试预算恢复生效的**正确表现**,但 e2e 冒烟测试的最坏耗时随之变长,`tests/e2e` 的源 `timeout_s` 配置可能需要相应调小。
research-wiki/designs/governance-backend-error.md
+1 -1
View File
@@ -33,7 +33,7 @@ date: 2026-08-06
## 对 issue 前提的四处修正 ## 对 issue 前提的四处修正
泄漏路径是**三条**不是两条(`retry.py:216` 的 `progress_age_s()` 同样在 catch 之外);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。 泄漏路径是**五条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。 根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
research-wiki/designs/issue8-stall-budget.md
+46
View File
@@ -0,0 +1,46 @@
---
type: design
node_id: design:issue8-stall-budget
title: "stall 判定改为非生产性等待口径"
date: 2026-08-06
---
# stall 判定改为非生产性等待口径
**全文**: `designs/2026-08-06-issue8-stall-budget-design.md`(已批准 2026-08-06)|**来源**: Gitea issue #8 |**实施**: [[plan:issue8-stall-budget-plan]]
## 问题
`timeout_s ≥ stall_window_s` 时,一次耗满超时的请求即判 scope 死,`max_attempts` **静默失效**(无报错无 warning)。`stall_window_s` 默认 300 极易被 `TIMEOUT_S` 追平,"只配 timeout 不配 stall"这种最常见写法正好踩中。
## 根因
**两个预算重叠计费**:真实尝试的耗时同时向重试预算(`max_attempts`)与 stall 预算(`stall_window_s`)计费,而后者更小,必然先耗尽。
## 选定方案
`StallClock` 让 stall 只累计非生产性等待。**划分依据是"谁消耗重试预算"**,不是"是否发出请求"——烧 `max_attempts` 的时间不烧 `stall_window_s`,不烧 `max_attempts` 的时间(含 429 尝试本身)归 stall 治理。
关键理由:
- **消除耦合而非守护耦合**。`stall_window_s` 与 `timeout_s` 自此无关系,配置方不必心算 `stall > timeout × retries`。
- **`inf` 语义因此不必改**。新口径下"非生产性排队耗满窗口且 scope 从未出餐"判死本就正当,`inf` 从"有害恒真"回归为"正确的保守默认"。一次改动解决问题,优于两次改动互相牵制。
- **取补集实现**(总时间减 `_attempt` 耗时)而非逐处标记 sleep:埋点 7 处降到 3 处,且将来新增等待路径自动计入 stall,默认安全。
## 被否决的备选
| 备选 | 否决理由 |
|---|---|
| **装配期校验 `stall_window_s > max(timeout_s)`**(issue 建议方向 1) | 治标:把缺陷固化成配置契约。且约束值须为 `timeout × max_attempts`(本机 900s),使 stall 兜底迟钝到近乎失效——修好一个洞挖开另一个。仍挡不住残余情形 |
| **`inf` 不参与判死**(issue 建议方向 2) | 新口径下 `inf` 已无害。单独改它会制造冷启动兜底真空(429 免预算无其他兜底),并反转 `test_both_windows_exceeded_raises_stalled` 钉住的行为、与 CHS 蓝本分叉 |
| **逐处标记 sleep** | 埋点 7 处且默认危险:新增等待路径忘记标记即成 stall 盲区 |
| **给 embedding/ocr 补主循环 stall 判定** | 前提不成立。429 免预算是 chat 独有,embedding/ocr 无条件 `fails += 1`,两条循环路径均已封闭,补齐等于凭空新增判死路径 |
| **删除既有 ttft 装配校验** | 其理由虽已消失(TTFT 属生产性时间),但校验无害且不误拒合理配置;删除需动 ARCHITECTURE §7.3 契约 G6,超出本 issue 范围(人类定夺:保留并改注释) |
## 实施期订正(§3.6)
初稿按"是否发出请求"划分,使 429 尝试**两个预算都不烧**(429 免重试预算,其耗时又算生产性)。排队型网关持满 timeout 才回 429 时实测挂 **25.2 小时**(301 次尝试),而改前只有 301s——**把一个 bug 换成了更严重的 bug**。由独立验证发现。订正为按"谁消耗重试预算"划分,429 尝试耗时退还 stall 账,实测回到 301s。
## 不变量
双条件结构、`progress_age_s()` 的 `inf` 语义、429 免预算、退避与 jitter 公式、`fail_fast` 分支、`AllSourcesExhausted` 字段与 `reason` 取值全部未动——**错误面零变更**。
research-wiki/graph/edges.json
+17
View File
@@ -140,6 +140,16 @@
"id": "plan:governance-backend-error", "id": "plan:governance-backend-error",
"label": "实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)", "label": "实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)",
"type": "plan" "type": "plan"
},
{
"id": "design:issue8-stall-budget",
"label": "stall 判定改为非生产性等待口径",
"type": "design"
},
{
"id": "plan:issue8-stall-budget-plan",
"label": "issue #8 实施计划: stall 非生产性等待口径",
"type": "plan"
} }
], ],
"links": [ "links": [
@@ -254,6 +264,13 @@
"relation": "implements", "relation": "implements",
"evidence": "T1-T5 逐任务实现设计 §3 的五项决策与 §8 影响面清单", "evidence": "T1-T5 逐任务实现设计 §3 的五项决策与 §8 影响面清单",
"added": "2026-08-06T08:08:51.865565+00:00" "added": "2026-08-06T08:08:51.865565+00:00"
},
{
"source": "plan:issue8-stall-budget-plan",
"target": "design:issue8-stall-budget",
"relation": "implements",
"evidence": "T1-T6 实施该设计,含 §3.6 订正",
"added": "2026-08-06T14:58:01.673693+00:00"
} }
] ]
} }
research-wiki/index.md
+7 -3
View File
@@ -1,8 +1,8 @@
# Research Wiki 索引 # Research Wiki 索引
> 自动生成，更新时间：2026-08-06 08:11 UTC > 自动生成，更新时间：2026-08-06 14:58 UTC
## design (23) ## design (25)
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design` - [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design` - [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
- [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design` - [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design`
@@ -14,6 +14,7 @@
- [2026-07-31-response-observability-fields-design](designs/2026-07-31-response-observability-fields-design.md) `design:2026-07-31-response-observability-fields-design` - [2026-07-31-response-observability-fields-design](designs/2026-07-31-response-observability-fields-design.md) `design:2026-07-31-response-observability-fields-design`
- [2026-07-31-sampling-params-design](designs/2026-07-31-sampling-params-design.md) `design:2026-07-31-sampling-params-design` - [2026-07-31-sampling-params-design](designs/2026-07-31-sampling-params-design.md) `design:2026-07-31-sampling-params-design`
- [2026-08-06-governance-backend-error-design](designs/2026-08-06-governance-backend-error-design.md) `design:2026-08-06-governance-backend-error-design` - [2026-08-06-governance-backend-error-design](designs/2026-08-06-governance-backend-error-design.md) `design:2026-08-06-governance-backend-error-design`
- [2026-08-06-issue8-stall-budget-design](designs/2026-08-06-issue8-stall-budget-design.md) `design:2026-08-06-issue8-stall-budget-design`
- [est_tokens 解耦: 拆分限流预扣与遥测用量兜底(issue #2)](designs/est-tokens-decoupling.md) `design:est-tokens-decoupling` - [est_tokens 解耦: 拆分限流预扣与遥测用量兜底(issue #2)](designs/est-tokens-decoupling.md) `design:est-tokens-decoupling`
- [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2` - [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2`
- [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards` - [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards`
@@ -22,6 +23,7 @@
- [M2.5 治理韧性: 半死源隔离与健康感知调度](designs/m25-resilience.md) `design:m25-resilience` - [M2.5 治理韧性: 半死源隔离与健康感知调度](designs/m25-resilience.md) `design:m25-resilience`
- [M3 OCR 端口族设计](designs/m3-ocr.md) `design:m3-ocr` - [M3 OCR 端口族设计](designs/m3-ocr.md) `design:m3-ocr`
- [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration` - [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration`
- [stall 判定改为非生产性等待口径](designs/issue8-stall-budget.md) `design:issue8-stall-budget`
- [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields` - [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields`
- [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design` - [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design`
- [治理后端故障归位为 scope 级不可用(Issue #7)](designs/governance-backend-error.md) `design:governance-backend-error` - [治理后端故障归位为 scope 级不可用(Issue #7)](designs/governance-backend-error.md) `design:governance-backend-error`
@@ -41,7 +43,7 @@
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak` - [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens` - [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
## plan (19) ## plan (21)
- [2026-07-20-m1-core-plan](plans/2026-07-20-m1-core-plan.md) `plan:2026-07-20-m1-core-plan` - [2026-07-20-m1-core-plan](plans/2026-07-20-m1-core-plan.md) `plan:2026-07-20-m1-core-plan`
- [2026-07-20-m2-distributed-plan](plans/2026-07-20-m2-distributed-plan.md) `plan:2026-07-20-m2-distributed-plan` - [2026-07-20-m2-distributed-plan](plans/2026-07-20-m2-distributed-plan.md) `plan:2026-07-20-m2-distributed-plan`
- [2026-07-21-m25-resilience-plan](plans/2026-07-21-m25-resilience-plan.md) `plan:2026-07-21-m25-resilience-plan` - [2026-07-21-m25-resilience-plan](plans/2026-07-21-m25-resilience-plan.md) `plan:2026-07-21-m25-resilience-plan`
@@ -51,7 +53,9 @@
- [2026-07-31-response-observability-fields](plans/2026-07-31-response-observability-fields.md) `plan:2026-07-31-response-observability-fields` - [2026-07-31-response-observability-fields](plans/2026-07-31-response-observability-fields.md) `plan:2026-07-31-response-observability-fields`
- [2026-07-31-sampling-params](plans/2026-07-31-sampling-params.md) `plan:2026-07-31-sampling-params` - [2026-07-31-sampling-params](plans/2026-07-31-sampling-params.md) `plan:2026-07-31-sampling-params`
- [2026-08-06-governance-backend-error-plan](plans/2026-08-06-governance-backend-error-plan.md) `plan:2026-08-06-governance-backend-error-plan` - [2026-08-06-governance-backend-error-plan](plans/2026-08-06-governance-backend-error-plan.md) `plan:2026-08-06-governance-backend-error-plan`
- [2026-08-06-issue8-stall-budget](plans/2026-08-06-issue8-stall-budget.md) `plan:2026-08-06-issue8-stall-budget`
- [est_tokens 解耦实施计划](plans/est-tokens-decoupling.md) `plan:est-tokens-decoupling` - [est_tokens 解耦实施计划](plans/est-tokens-decoupling.md) `plan:est-tokens-decoupling`
- [issue #8 实施计划: stall 非生产性等待口径](plans/issue8-stall-budget-plan.md) `plan:issue8-stall-budget-plan`
- [M1 核心里程碑实现计划](plans/m1-core-plan.md) `plan:m1-core-plan` - [M1 核心里程碑实现计划](plans/m1-core-plan.md) `plan:m1-core-plan`
- [M2 分布式实现计划](plans/m2-distributed.md) `plan:m2-distributed` - [M2 分布式实现计划](plans/m2-distributed.md) `plan:m2-distributed`
- [M2.5 治理韧性实现计划](plans/m25-resilience.md) `plan:m25-resilience` - [M2.5 治理韧性实现计划](plans/m25-resilience.md) `plan:m25-resilience`
research-wiki/log.md
+4
View File
@@ -90,3 +90,7 @@
- [2026-08-06 08:08 UTC] 新增边: plan:governance-backend-error --implements--> design:governance-backend-error - [2026-08-06 08:08 UTC] 新增边: plan:governance-backend-error --implements--> design:governance-backend-error
- [2026-08-06 08:08 UTC] 重建索引: 57 篇页面 - [2026-08-06 08:08 UTC] 重建索引: 57 篇页面
- [2026-08-06 08:11 UTC] 重建索引: 57 篇页面 - [2026-08-06 08:11 UTC] 重建索引: 57 篇页面
- [2026-08-06 14:57 UTC] 新增 design: stall 判定改为非生产性等待口径 (design:issue8-stall-budget)
- [2026-08-06 14:58 UTC] 新增 plan: issue #8 实施计划: stall 非生产性等待口径 (plan:issue8-stall-budget-plan)
- [2026-08-06 14:58 UTC] 新增边: plan:issue8-stall-budget-plan --implements--> design:issue8-stall-budget
- [2026-08-06 14:58 UTC] 重建索引: 61 篇页面
research-wiki/plans/2026-08-06-governance-backend-error-plan.md
+34 -7
View File
@@ -107,7 +107,7 @@ class BreakerGate:
## 任务清单 ## 任务清单
### - [ ] T1: ARCHITECTURE §6.1 回补(必须先行) ### - [x] T1: ARCHITECTURE §6.1 回补(必须先行)
**文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行) **文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行)
@@ -125,7 +125,7 @@ class BreakerGate:
--- ---
### - [ ] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出 ### - [x] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出
**文件**: 改 `src/polygateway/errors.py`、`src/polygateway/__init__.py`;改 `tests/unit/test_errors.py` **文件**: 改 `src/polygateway/errors.py`、`src/polygateway/__init__.py`;改 `tests/unit/test_errors.py`
@@ -148,7 +148,7 @@ class BreakerGate:
--- ---
### - [ ] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子) ### - [x] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子)
**文件**: 改 `src/polygateway/errors.py`、`backends/redis/limiter.py`、`backends/redis/breaker.py`、`backends/memory/limiter.py`、`middleware/ratelimit.py`、`middleware/breaker.py`、`middleware/retry.py`、`ocr.py`、`embedding.py`;改 `tests/unit/test_errors.py`、`tests/unit/test_backpressure.py`、`tests/unit/test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py` **文件**: 改 `src/polygateway/errors.py`、`backends/redis/limiter.py`、`backends/redis/breaker.py`、`backends/memory/limiter.py`、`middleware/ratelimit.py`、`middleware/breaker.py`、`middleware/retry.py`、`ocr.py`、`embedding.py`;改 `tests/unit/test_errors.py`、`tests/unit/test_backpressure.py`、`tests/unit/test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py`
@@ -170,7 +170,7 @@ class BreakerGate:
- `backends/redis/breaker.py` 的 `:370 / :388 / :410 / :422 / :432`(5 处) - `backends/redis/breaker.py` 的 `:370 / :388 / :410 / :422 / :432`(5 处)
4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise` 补 `scope=self._scope`。 4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise` 补 `scope=self._scope`。
- 各方法开头的 `except GovernanceBackendError: raise` **保持不变**(后端层已填好 scope,重建实例只会重复构造,设计 §3.3)。 - ~~各方法开头的 `except GovernanceBackendError: raise` **保持不变**~~ **← 这条是错的,2026-08-06 独立验证时炸出(见 §T6)**。正确做法: 该放行必须扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise`,否则新增的兄弟类型会落进下一行的 `except Exception` 被**重新包成** `GovernanceBackendError`,使 Q1 的拆分在唯一的生产路径上完全失效。
5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序): 5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序):
@@ -186,7 +186,7 @@ class BreakerGate:
|---|---|---| |---|---|---|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 | | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 | | `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
| 三条泄漏路径(`try_acquire` / `try_enter` / `progress_age_s`)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 | | 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) | | 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 | | Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
@@ -211,7 +211,7 @@ class BreakerGate:
--- ---
### - [ ] T4: 公开错误面文档(issue #7 第二诉求) ### - [x] T4: 公开错误面文档(issue #7 第二诉求)
**文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md` **文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md`
@@ -238,7 +238,7 @@ class BreakerGate:
--- ---
### - [ ] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步 ### - [x] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步
**文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki **文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki
@@ -258,6 +258,33 @@ class BreakerGate:
--- ---
### - [x] T6: 修复独立验证炸出的阻塞缺陷(计划外,2026-08-06)
T1–T5 全绿、全部门禁通过之后,全新上下文的 verifier 用一个**走 `QuotaGate` 的**端到端用例炸出:装配缺陷在唯一的生产路径上根本没有拆出去。
**缺陷**: `QuotaGate`/`BreakerGate` 的 `except GovernanceBackendError: raise` 只放行了旧类型,新增的 `SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包成 `GovernanceBackendError`(`reason=governance_backend_down`、`retry_after_s=5.0`)。实证:
```
RAISED: GovernanceBackendError | isGatewayUnavailable=True | isSourceNotConfigured=False
| 限流后端故障(source_stats): 未知源 's1'(scope=llm)
```
即配置写错的任务照样落进"可延期重投"家族,**永远重投、永不进死信、无人告警**——正是 Q1 要防的镜像 bug,G2 等于没做。
**为什么原有测试测不出来**: T3 写的两条用例(`test_backpressure.py`、`test_redis_key_layout.py`)都直接打私有 `_cfg()`,绕过了包装器;而治理循环只经包装器访问后端。**盲区在于测试打的层次比生产路径低一层。**
**修复**(三处):
| 文件 | 改动 |
|---|---|
| `middleware/ratelimit.py` | 4 个方法的放行扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise` |
| `middleware/breaker.py` | 同上,5 个方法 |
| `middleware/telemetry.py:254` | 终态捕获元组加 `SourceNotConfiguredError`。**连带坑**: 放行生效后该异常不再是 `GovernanceBackendError`,而它在任何 attempt 之前抛出,若不显式捕获则 `emit_terminal_failure` 不触发、该路径**遥测归零**,违反"遥测必录"铁律 |
**回归测试**: `test_backpressure.py::TestUnknownSourceIsAssemblyDefect::test_survives_the_quota_gate_wrapper`(参数化覆盖 `try_acquire` / `stats`),**走包装器而非私有方法**。修前 2 failed,修后 PASS。
**同批文档订正**: 泄漏路径由"三条"改为**五条**(遗漏了 `QuotaGate.stats` 与 `BreakerGate.retry_after_s`,判据是该调用点是否被 `_record_quietly` 包裹);CHANGELOG 的 `per_source_reasons` 表述改为"属性存在但恒为 `{}`"。
## 完成后 ## 完成后
按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7。 按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7。
research-wiki/plans/2026-08-06-issue8-stall-budget.md
+276
View File
@@ -0,0 +1,276 @@
# 实施计划: stall 判定改为非生产性等待口径(Issue #8)
- **依据设计**: `research-wiki/designs/2026-08-06-issue8-stall-budget-design.md`(**已批准 2026-08-06**)
- **分支**: `feat/issue-8-stall-budget`(已建,已含设计提交 `bfe423d` + `ce2dda7`)
- **目标**: 让 stall 计时器只累计非生产性等待,解除 `timeout_s` 与 `stall_window_s` 的隐式耦合,使重试预算在超时场景下真实可用。
- **方案概述**: 新增调用级 `StallClock`(总时间减去 `_attempt` 耗时),替换三条治理循环里的墙钟 `entered_at`。判死双条件的结构、`inf` 语义、错误面、429 免预算全部不动。
- **涉及技术**: Python 3.11 asyncio、`contextlib.asynccontextmanager`、pytest + `FakeClock`。
## 保真校验适用性
**适用**。三条治理循环均为 `reference/CHSAnalyzer app/providers/governance.py:200-285` 的移植物(ARCHITECTURE.md §1.4 关键资产)。但 **`reference/` 当前不在工作区**,无法逐段比对源码,故保真基准改为两处已入库的等价证据:
1. 设计文档 §4「旧版行为审计」表——9 条既有行为逐条标注保留/替换,实施时逐条核对;
2. 代码内既有的 CHS 行号注释(`retry.py:212`「调用级累计计时,循环内不重置(CHS governance.py:207)」、`:303-304`「双条件 stall 判死(CHS governance.py:270-281)」、`:315`「jitter 防惊群(CHS governance.py:283-285)」)与 `tests/unit/test_backpressure.py:1-6` 的蓝本 docstring。
**唯一允许的语义变更是条件 A 的度量口径**(设计 §4 中标"替换"的那一行)。其余任何条件分支、退避公式、jitter 区间、状态迁移若发生行为改变,即为违规,必须回退。
## 文件结构
| 文件 | 动作 | 职责 |
|---|---|---|
| `src/polygateway/middleware/retry.py` | 修改 | 新增模块级 `StallClock`(共享单元);主循环与 `_on_no_runnable` 改用之 |
| `src/polygateway/embedding.py` | 修改 | 复用 `StallClock`;`_on_no_runnable` 改签名 |
| `src/polygateway/ocr.py` | 修改 | 同上 |
| `src/polygateway/config.py` | 修改 | `_validate_stall` docstring 改写(仅注释,不改逻辑) |
| `tests/unit/test_backpressure.py` | 修改 | 新增 `TestStallBudget` 类;订正 `:121` docstring |
| `tests/unit/test_embedding.py` | 修改 | 新增 embedding 回归用例 |
| `tests/unit/test_ocr_client.py` | 修改 | 新增 ocr 回归用例 |
| `.env.example` | 修改 | 第 41 行注释改写 |
| `research-wiki/ARCHITECTURE.md` | 修改 | §7.3 背压条目补记新口径 |
| `CHANGELOG.md` | 修改 | 记治理行为变更 |
**不创建任何新模块**。`StallClock` 放在 `retry.py`,沿用 `backoff_delay` 已被 embedding/ocr 复用的既有手法(依赖方向不变:`embedding.py:42`、`ocr.py:38` 已在 import 该模块)。
## 关键接口(跨任务消费,此处给出实际代码)
`StallClock` 由 T1 落地,T2/T3 直接消费,签名以此为准:
```python
class StallClock:
"""调用级 stall 计时器: 只累计非生产性等待(设计 §3.1)。
stall 预算治理的是"无人治理的等待"(429 退避、配额轮询、熔断冷却),
真实尝试已由重试预算 max_attempts 治理,故须从 stall 账里扣除——
两者重叠计费正是 issue #8 的根因。
每次调用创建一个实例。严禁提升为实例属性: 并发调用共享会互相污染计时。
"""
__slots__ = ("_now", "_entered_at", "_productive_s")
def __init__(self, now: Callable[[], float]) -> None:
self._now = now
self._entered_at = now()
self._productive_s = 0.0
def stalled_s(self) -> float:
"""非生产性等待累计秒数 = 总耗时 - 真实尝试耗时。"""
return self._now() - self._entered_at - self._productive_s
@contextlib.asynccontextmanager
async def attempting(self) -> AsyncIterator[None]:
"""包裹一次真实尝试, 其耗时记为生产性(边界即 _attempt 的边界)。"""
started = self._now()
try:
yield
finally:
# 只做算术, 不吞任何异常——CancelledError 逐字穿透(库铁律)
self._productive_s += self._now() - started
```
需在 `retry.py` 新增 `import contextlib`;`AsyncIterator` 从 `collections.abc` 引入(该文件已有 `from __future__ import annotations`,类型注解延迟求值,若 `TYPE_CHECKING` 块中已有 `Callable` 则复用)。
三条循环的改造模式一致:
```python
clock = StallClock(self._now) # 替换 entered_at = self._now()
...
await self._on_no_runnable(gate_rejections, reasons, clock) # 形参改类型
...
async with clock.attempting():
outcome = await self._attempt(...) # 原调用不变, 仅被包裹
```
判定式由 `self._now() - entered_at > stall` 改为 `clock.stalled_s() > stall`,**条件 B 与 `and` 结构逐字不动**。
---
## T1 — `StallClock` 落地与 chat 路径改造
- [ ] **文件**: `src/polygateway/middleware/retry.py`(修改)、`tests/unit/test_backpressure.py`(修改)
### 行为与验收标准
1. 按上文「关键接口」实现 `StallClock`,置于模块级(建议紧邻既有 `backoff_delay` 纯函数,便于 embedding/ocr 一并 import)。
2. `RetryMW.__call__`:`entered_at = self._now()`(`retry.py:212`)改为 `clock = StallClock(self._now)`;主循环判定(`:217`)改为 `clock.stalled_s() > stall`;`_attempt` 调用(`:228`)用 `async with clock.attempting():` 包裹。
3. `_on_no_runnable`(`:286-288`)形参 `entered_at: float` 改为 `clock: StallClock`,其内判定(`:306`)同步改为 `clock.stalled_s() > stall`。
4. **不得改动**:429 免预算分支(`:233-234`)、`max(fails, 1)` 退避(`:243`)、jitter 公式(`:315`)、`fail_fast` 分支、条件 B `await self._quota.progress_age_s() > stall`、`AllSourcesExhausted` 的任何字段。
5. 保留 `retry.py:212` 的 CHS 行号注释并补记新口径(说明"调用级累计、循环内不重置"仍然成立,变的只是不再计入真实尝试)。
### 测试要求(先失败后通过)
在 `tests/unit/test_backpressure.py` 新增 `class TestStallBudget`:
| 用例 | 构造 | 断言 |
|---|---|---|
| `test_single_timeout_does_not_exhaust_stall_budget` | `_STALL=300`,源 `timeout_s` 等价;脚本 `[TransientError(耗时 350s), _ok()]`——用 `FakeTransport` 配合在尝试中推进 `FakeClock` 350s | 返回成功响应。**改前**:抛 `AllSourcesExhausted(reason="stalled")` |
| `test_productive_time_excluded_from_stall` | 连续两次尝试各推进时钟 `_STALL+100`,第三次成功 | 返回成功;全程不触发 `stalled` |
| `test_nonproductive_wait_still_triggers_stall` | 沿用 `_blocked_limiter`,轮询中推进时钟超窗且不 `mark_progress` | 抛 `stalled`(兜底未被削弱) |
| `test_saturation_429_still_stalls` | 源持续抛 429(`TransientError(status_code=429)`),退避 sleep 中推进时钟 | 抛 `stalled` 而非无限循环(`BoundedSleep` 上限内)。钉住设计 §3.5 |
| `test_cancel_inside_attempt_pierces` | 在 `_attempt` 内挂起后 `task.cancel()` | 抛 `CancelledError`(`attempting()` 的 finally 不吞) |
| `test_concurrent_calls_do_not_share_clock` | 两路并发调用,一路长尝试、一路正常 | 两路互不影响;钉住 `StallClock` 不得为实例属性 |
| `test_telemetry_time_counts_as_productive` | 注入一个在 `emit_attempt` 中推进 `FakeClock` 超过 `_STALL` 的慢 emitter,transport 正常成功 | 返回成功响应,不触发 `stalled`。**钉住设计 §3.1 的边界声明**:遥测收尾属生产性,遥测抖动不得参与判死。若将来有人把 `attempting()` 的包裹范围收窄到只包 transport 调用,该不变式会被悄悄破坏而其余用例抓不到 |
**既有四象限用例(`TestStallQuadrants` 四条)必须原样通过,不得修改断言**——它们全程无真实尝试或真实尝试耗时为 0,`stalled_s()` 与旧墙钟等价。若其中任何一条需要改断言才能通过,说明实现越界,停下来复核。
同时订正 `tests/unit/test_backpressure.py:121` 的 docstring:「仅全局超窗(从未出餐 age=inf)」保持不变(该语义确实不变),但补一句说明本地口径已是非生产性等待。
### 验证命令
```bash
conda run -n PolyGateway pytest tests/unit/test_backpressure.py -v
conda run -n PolyGateway pytest tests/unit/test_retry.py -v
```
预期:全部 PASS。先在实现前跑新增用例,记录 `test_single_timeout_does_not_exhaust_stall_budget` 的 FAILED 输出作为红证据。
- [ ] **提交点**: `fix: bill only non-productive waiting against the chat stall budget`
---
## T2 — embedding 路径改造
- [ ] **文件**: `src/polygateway/embedding.py`(修改)、`tests/unit/test_embedding.py`(修改)
### 行为与验收标准
1. `embedding.py:42` 的 import 增加 `StallClock`(该行已 import `_failure_reason, backoff_delay`)。
2. `_embed_batch`(`:182`):`entered_at = self._now()` 改为 `clock = StallClock(self._now)`;`_attempt` 调用(`:188`)用 `async with clock.attempting():` 包裹;`_on_no_runnable` 传参(`:186`)改为 `clock`。
3. `_on_no_runnable`(`:230-232`)形参改 `clock: StallClock`,判定(`:248`)改 `clock.stalled_s() > stall`。
4. **不得新增主循环 stall 判定**(设计 §5.4:embedding 无 429 免预算,`fails += 1` 无条件,缺口不存在;新增等于凭空多一条判死路径)。
5. **不得改动**:`fails += 1` 的无条件性(`:191`)、`max_attempts` 判定、退避调用(`:200`)。
### 测试要求(先失败后通过)
在 `tests/unit/test_embedding.py` 新增 `test_single_timeout_does_not_exhaust_stall_budget`。
**构造方式(已核实可行,不必绕过既有 helper)**:`_embed_client`(`:219`)的 `**overrides` 直通 `EmbeddingClient.__init__`,而后者接受 `now`/`sleep`/`rng`(`embedding.py:109-111`),故可写 `_embed_client([src], script, now=clock, sleep=<推进时钟的 fake>)`。制造一轮 `_on_no_runnable` 沿用 `test_backpressure.py:75-85` `_blocked_limiter` 的手法:源 `max_concurrency=1`,测试先 `try_acquire` 占满 permit,在 fake sleep 回调里释放。helper 内的 `InMemoryLimiter` 未注入 `now` 不影响本用例——判定要的是 `progress_age_s()` 返回 `inf`(从未 `mark_progress`),与 limiter 时钟无关。
**断言**:第一次尝试推进 `FakeClock` 超过 `stall_window_s` 后抛 `TransientError`,随后经一轮 `_on_no_runnable` 再恢复,最终返回成功的 `EmbeddingResponse`。改前应抛 `AllSourcesExhausted(reason="stalled")`。
**取消穿透验收点**:既有 `test_cancel_releases_permit`(`test_embedding.py:331-339`)的取消路径**将被新的 `async with clock.attempting()` 包住**,故它是本任务的必过回归项,不得因改动而修改其断言。若它转红,说明 `attempting()` 的 `finally` 吞了 `CancelledError` 或泄漏了 permit,停下来复核而非改测试。
### 验证命令
```bash
conda run -n PolyGateway pytest tests/unit/test_embedding.py -v
```
预期:全部 PASS(含既有取消与遥测用例)。
- [ ] **提交点**: `fix: apply the non-productive stall budget to the embedding loop`
---
## T3 — ocr 路径改造
- [ ] **文件**: `src/polygateway/ocr.py`(修改)、`tests/unit/test_ocr_client.py`(修改)
### 行为与验收标准
与 T2 同构,对应行号:import(`:38`)、`_call` 的 `entered_at`(`:207`)、`_on_no_runnable` 传参(`:211`)、`_attempt` 调用(`:213`)、`_on_no_runnable` 签名(`:255-257`)与判定(`:273`)。同样**不得新增主循环 stall 判定**,不得改动 `fails += 1`(`:216`)与退避(`:225`)。
### 测试要求(先失败后通过)
`tests/unit/test_ocr_client.py` 新增与 T2 同构的 `test_single_timeout_does_not_exhaust_stall_budget`,覆盖 `recognize_text` 或 `parse_layout` 任一端点即可(两者共用 `_call`)。
**构造方式**:同 T2——经该文件既有的 `_client(...)` helper 传 `now=clock` 与推进时钟的 fake `sleep`;`_on_no_runnable` 一轮用"源 `max_concurrency=1` + 测试预先占满 permit + 在 fake sleep 回调里释放"制造。
**取消穿透验收点**:既有 `test_cancel_during_transport_releases_permit`(`test_ocr_client.py:339-348`)与其上方的退避期取消用例同样会被新包裹覆盖,均为必过回归项,不得修改断言。
### 验证命令
```bash
conda run -n PolyGateway pytest tests/unit/test_ocr_client.py tests/unit/test_monkey_ocr.py -v
```
预期:全部 PASS。
- [ ] **提交点**: `fix: apply the non-productive stall budget to the ocr loop`
---
## T4 — 配置侧注释对齐(无逻辑变更)
- [ ] **文件**: `src/polygateway/config.py`(修改)、`.env.example`(修改)
### 行为与验收标准
1. `config.py:240-241` `_validate_stall` 的 docstring 由「stall 窗口须 ≥ 最慢源 TTFT 上限,防把正常慢首包误判为卡死」改写为:说明该校验在新口径下**属保守冗余**——TTFT 等待是生产性时间,已不计入 stall;保留校验是为不改动 ARCHITECTURE.md §7.3 契约 G6(人类 2026-08-06 定夺)。**校验逻辑本身一字不改**。
2. `.env.example:41` 注释由「stall 双条件判死窗口;须 ≥ 最大源 TTFT」改写为说明它度量的是**非生产性等待**(429 退避/配额轮询/熔断冷却)累计,与 `TIMEOUT_S` 无耦合、无需按 `timeout × retries` 放大。
3. 不新增、不改名任何配置键(`_DEFAULT_STALL_WINDOW_S = 300.0` 保持不变)。
### 验证命令
```bash
conda run -n PolyGateway pytest tests/unit/test_config.py -v
conda run -n PolyGateway make lint
```
预期:全部 PASS(本任务不改逻辑,`test_config.py` 应零变化通过)。
- [ ] **提交点**: `docs: align the stall window comments with the new metering`
---
## T5 — 全套件回归与文档同步
- [ ] **文件**: `research-wiki/ARCHITECTURE.md`(修改)、`CHANGELOG.md`(修改)
### 行为与验收标准
1. 跑全套件确认零回归。**命令末尾不得接管道**(CLAUDE.md 执行模式:管道会掩盖真实退出码),需要后台跑时用 `wait`/轮询 PID 判完成。
2. `ARCHITECTURE.md` §7.3 背压条目(第 429-431 行区域)补记:stall 双条件的条件 A 现为**非生产性等待累计**,并给出本设计文档指针。既有 G6 契约行保留,补注其在新口径下为保守冗余。
3. `CHANGELOG.md` 记治理行为变更(属公共行为变更,须显式列出:单次调用最坏耗时由 `stall_window_s` 抬升至 `max_attempts × timeout_s`)。
4. 核对设计 §4 行为审计表 9 条,逐条确认实现与标注一致(保真校验检查点)。
### 副作用处置
修复后单次调用最坏耗时变为 `max_attempts × timeout_s`(本机 900s)。跑 e2e 前先评估 `tests/e2e` 的源 `timeout_s` 是否需调小,以免冒烟耗时失控。本机 `.env:37` 的临时缓解 `STALL_WINDOW_S=1200` 可回退默认值(`.env` 不入库,仅在本任务记录该动作)。
### 验证命令
```bash
conda run -n PolyGateway make lint
conda run -n PolyGateway pytest tests/unit tests/integration -v
conda run -n PolyGateway make test
```
预期:lint 通过(含 import-linter 依赖契约);单元与集成全绿;覆盖率不低于既有水平。
- [ ] **提交点**: `docs: record the stall metering change in architecture and changelog`
---
## T6 — 独立验证与 Wiki 同步
- [ ] **文件**: Gitea Wiki(独立仓库)
### 行为与验收标准
1. **派全新上下文 verifier subagent**(`verification-before-completion`,MANDATORY:跨 3 模块属里程碑级),**前台运行**(`run_in_background: false`,CLAUDE.md 执行模式)。核验对象:设计 §1 的 G1-G4 是否逐条兑现、§4 行为审计表 9 条是否与实现一致、是否出现设计未声明的语义变更、测试是否真的覆盖"先失败后通过"。
2. 按 `docs-convention.md` §2「治理行为变更」行同步 Wiki:`解释-治理行为`(stall 判定口径)、`指南-限流与熔断`(配置说明中删除"须按 timeout×retries 放大 stall"一类误导)。
3. 在 Gitea issue #8 下回帖:根因、方案、被否决的两个原建议方向及理由、影响面。
4. Wiki 注册:
```bash
.claude/tools/research_wiki.py add_entity research-wiki/ --type plan --id issue8-stall-budget --title "stall 判定改为非生产性等待口径"
.claude/tools/research_wiki.py add_edge research-wiki/ --from "plan:issue8-stall-budget" --to "design:issue8-stall-budget" --type implements --evidence "本计划实施该设计的 T1-T6"
.claude/tools/research_wiki.py rebuild_index research-wiki/
```
### 验证命令
```bash
conda run -n PolyGateway make ci
```
预期:只读验证全绿。verifier 报告须逐条对应本会话内的工具输出(证据化声明,禁止虚报)。
- [ ] **提交点**: `chore: register the issue #8 plan and sync the wiki`
---
## 任务依赖
T1 → (T2 ‖ T3) → T4 → T5 → T6。T2 与 T3 相互独立,但都依赖 T1 落地的 `StallClock`。
research-wiki/plans/governance-backend-error.md
+8
View File
@@ -34,4 +34,12 @@ date: 2026-08-06
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。 Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier)
T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。**
修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。
两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats` 与 `BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。
相关: [[governance-backend-error]](design)、[[m2-distributed]] 相关: [[governance-backend-error]](design)、[[m2-distributed]]
research-wiki/plans/issue8-stall-budget-plan.md
+35
View File
@@ -0,0 +1,35 @@
---
type: plan
node_id: plan:issue8-stall-budget-plan
title: "issue #8 实施计划: stall 非生产性等待口径"
date: 2026-08-06
---
# issue #8 实施计划: stall 非生产性等待口径
**全文**: `plans/2026-08-06-issue8-stall-budget.md` |**实现**: [[design:issue8-stall-budget]] |**分支**: `feat/issue-8-stall-budget`
## 交付
| 任务 | 内容 | 提交 |
|---|---|---|
| T1 | `StallClock` 落地 + chat 路径改造 + 8 条测试 | `02c3d06` |
| T2 | embedding 路径复用 | `6d0f3c9` |
| T3 | ocr 路径复用 | `0477d95` |
| T4 | `config.py` docstring 与 `.env.example` 注释对齐(无逻辑变更) | `d05114e` |
| T5 | 全套件回归 + `ARCHITECTURE.md` §7.3 与 `CHANGELOG` 同步 | `3645e57` |
| T6 | 独立验证(全新上下文 verifier)+ 三个问题的修复 | `bc4683d`、`a0a5cf7` |
## 测试证据(先失败后通过)
三条路径的失效链条各有一条回归用例,改前均转红于 `reason="stalled"`:`retry.py:218`、`embedding.py:250`、`ocr.py:275`。issue 只记录了 chat 路径,embedding/ocr 两条为本次核出。
## 独立验证发现的三个问题(均已修)
1. **429 缝隙(中)**: 初稿使 429 尝试两个预算都不烧,慢 429 场景实测挂 25.2 小时——**修复引入的回归**。见设计 §3.6。
2. **测试假证据(中)**: 并发用例用了两个 `RetryMW` 实例,实例级共享被对象隔离掩盖,clock 提升为实例属性时 7 条用例全部逃逸。改为复用同一 `mw` 并补"两次调用间空转超窗"用例,变异测试确认可抓。
3. **文档遗漏(轻)**: 计划要求的 `test_backpressure.py` docstring 订正漏做。
## 保真校验
治理主循环为 CHS `governance.py:200-285` 移植物,但 `reference/` 不在工作区,故以设计 §4 行为审计表 9 条 + 代码内 CHS 行号注释为基准。核对结果:标"保留"的 8 条在 `git diff` 中零出现,唯一"替换"项为条件 A 度量口径。
src/polygateway/__init__.py
+1 -1
View File
@@ -32,7 +32,7 @@ from polygateway.types import (
SourceConfig, SourceConfig,
) )
__version__ = "1.1.0" __version__ = "1.1.1"
__all__ = [ __all__ = [
"DEFAULT_PROFILES", "DEFAULT_PROFILES",
src/polygateway/config.py
+8 -1
View File
@@ -238,7 +238,14 @@ class GatewaySettings:
) )
def _validate_stall(self) -> None: def _validate_stall(self) -> None:
"""stall 窗口须 ≥ 最慢源 TTFT 上限,防把正常慢首包误判为卡死。""" """stall 窗口须 ≥ 最慢源 TTFT 上限(保守冗余,见下)。
原理由是"防把正常慢首包误判为卡死"。issue #8 起 stall 只累计**非
生产性等待**(429 退避、配额轮询、熔断冷却),TTFT 等待属生产性时间、
已不计入 stall 账,该误判在机制上不再可能。校验本身无害且不会误拒
任何合理配置,故保留——删除它需同步改动 ARCHITECTURE.md §7.3 的契约
补强 G6,超出 issue #8 的范围(2026-08-06 人类定夺)。
"""
ttfts = [s.ttft_timeout_s for s in self.sources if s.ttft_timeout_s is not None] ttfts = [s.ttft_timeout_s for s in self.sources if s.ttft_timeout_s is not None]
if ttfts and self.backpressure.stall_window_s < max(ttfts): if ttfts and self.backpressure.stall_window_s < max(ttfts):
raise ValueError( raise ValueError(
src/polygateway/embedding.py
+10 -7
View File
@@ -34,11 +34,12 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
from polygateway.middleware.ratelimit import QuotaGate from polygateway.middleware.ratelimit import QuotaGate
from polygateway.middleware.retry import _failure_reason, backoff_delay from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
from polygateway.middleware.telemetry import TelemetryEmitter from polygateway.middleware.telemetry import TelemetryEmitter
from polygateway.sources import SourceCooldownMemo from polygateway.sources import SourceCooldownMemo
from polygateway.types import ( from polygateway.types import (
@@ -178,13 +179,15 @@ class EmbeddingClient:
) -> _BatchOutcome: ) -> _BatchOutcome:
fails = 0 fails = 0
reasons: dict[str, str] = {} reasons: dict[str, str] = {}
entered_at = self._now() # 只计非生产性等待(issue #8): 真实尝试由重试预算治理,不重复烧 stall 预算
clock = StallClock(self._now)
while True: while True:
picked, gate_rejections = await self._pick_runnable(reasons) picked, gate_rejections = await self._pick_runnable(reasons)
if picked is None: if picked is None:
await self._on_no_runnable(gate_rejections, reasons, entered_at) await self._on_no_runnable(gate_rejections, reasons, clock)
continue continue
outcome = await self._attempt(batch, *picked, reasons, session_id, parent_call_id) async with clock.attempting():
outcome = await self._attempt(batch, *picked, reasons, session_id, parent_call_id)
if isinstance(outcome, _BatchOutcome): if isinstance(outcome, _BatchOutcome):
return outcome return outcome
fails += 1 fails += 1
@@ -227,7 +230,7 @@ class EmbeddingClient:
return None, gate_rejections return None, gate_rejections
async def _on_no_runnable( async def _on_no_runnable(
self, gate_rejections: int, reasons: dict[str, str], entered_at: float self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
) -> None: ) -> None:
if gate_rejections == len(self._sources): if gate_rejections == len(self._sources):
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
@@ -244,7 +247,7 @@ class EmbeddingClient:
per_source_reasons=reasons, per_source_reasons=reasons,
) )
stall = self._bp.stall_window_s stall = self._bp.stall_window_s
if self._now() - entered_at > stall and await self._quota.progress_age_s() > stall: if clock.stalled_s() > stall and await self._quota.progress_age_s() > stall:
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
raise AllSourcesExhausted( raise AllSourcesExhausted(
scope=self._scope, scope=self._scope,
@@ -326,7 +329,7 @@ class EmbeddingClient:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc) logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
async def _settle_and_release(self, permit: Permit, actual: int) -> None: async def _settle_and_release(self, permit: Permit, actual: int) -> None:
src/polygateway/middleware/breaker.py
+21 -11
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from polygateway.errors import GovernanceBackendError from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
if TYPE_CHECKING: if TYPE_CHECKING:
from polygateway.ports import GateDecision, GateUpdate, ProviderGate from polygateway.ports import GateDecision, GateUpdate, ProviderGate
@@ -22,43 +22,53 @@ class BreakerGate:
async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision: async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision:
try: try:
return await self._gate.try_enter(source.name, owner) return await self._gate.try_enter(source.name, owner)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(try_enter): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(try_enter): {exc}", scope=self._scope
) from exc
async def record_success( async def record_success(
self, entry: GateDecision, *, count_attempt: bool = True self, entry: GateDecision, *, count_attempt: bool = True
) -> GateUpdate: ) -> GateUpdate:
try: try:
return await self._gate.record_success(entry, count_attempt=count_attempt) return await self._gate.record_success(entry, count_attempt=count_attempt)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(record_success): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(record_success): {exc}", scope=self._scope
) from exc
async def record_failure( async def record_failure(
self, entry: GateDecision, reason: str, force_open: bool self, entry: GateDecision, reason: str, force_open: bool
) -> GateUpdate: ) -> GateUpdate:
try: try:
return await self._gate.record_failure(entry, reason, force_open) return await self._gate.record_failure(entry, reason, force_open)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(record_failure): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(record_failure): {exc}", scope=self._scope
) from exc
async def release_probe(self, entry: GateDecision) -> GateUpdate: async def release_probe(self, entry: GateDecision) -> GateUpdate:
try: try:
return await self._gate.release_probe(entry) return await self._gate.release_probe(entry)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(release_probe): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(release_probe): {exc}", scope=self._scope
) from exc
async def retry_after_s(self, sources: tuple[str, ...]) -> float: async def retry_after_s(self, sources: tuple[str, ...]) -> float:
try: try:
return await self._gate.retry_after_s(sources) return await self._gate.retry_after_s(sources)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(retry_after_s): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(retry_after_s): {exc}", scope=self._scope
) from exc
src/polygateway/middleware/ratelimit.py
+17 -9
View File
@@ -8,7 +8,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from polygateway.errors import GovernanceBackendError from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
if TYPE_CHECKING: if TYPE_CHECKING:
from polygateway.ports import Permit, RateLimiter from polygateway.ports import Permit, RateLimiter
@@ -26,31 +26,39 @@ class QuotaGate:
async def try_acquire(self, source: SourceConfig) -> Permit | None: async def try_acquire(self, source: SourceConfig) -> Permit | None:
try: try:
return await self._limiter.try_acquire(source.name, source.effective_est_tokens()) return await self._limiter.try_acquire(source.name, source.effective_est_tokens())
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(try_acquire): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(try_acquire): {exc}", scope=self._scope
) from exc
async def stats(self, source: SourceConfig) -> SourceStats: async def stats(self, source: SourceConfig) -> SourceStats:
try: try:
return await self._limiter.source_stats(source.name) return await self._limiter.source_stats(source.name)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(source_stats): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(source_stats): {exc}", scope=self._scope
) from exc
async def mark_progress(self) -> None: async def mark_progress(self) -> None:
try: try:
await self._limiter.mark_progress() await self._limiter.mark_progress()
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(mark_progress): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(mark_progress): {exc}", scope=self._scope
) from exc
async def progress_age_s(self) -> float: async def progress_age_s(self) -> float:
try: try:
return await self._limiter.progress_age_s() return await self._limiter.progress_age_s()
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(progress_age_s): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(progress_age_s): {exc}", scope=self._scope
) from exc
src/polygateway/middleware/retry.py
+99 -17
View File
@@ -11,6 +11,7 @@ httpx 是库的核心依赖而非实现层内部件,不违反"middleware 只依
from __future__ import annotations from __future__ import annotations
import asyncio import asyncio
import contextlib
import random import random
import time import time
import uuid import uuid
@@ -28,6 +29,7 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
@@ -38,7 +40,7 @@ from polygateway.streaming import StreamLivenessTimeout
from polygateway.types import LLMResponse from polygateway.types import LLMResponse
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Awaitable, Callable from collections.abc import AsyncIterator, Awaitable, Callable
from polygateway.ports import ( from polygateway.ports import (
GateDecision, GateDecision,
@@ -73,6 +75,65 @@ def backoff_delay(
return max(delay, retry_after) return max(delay, retry_after)
class _Attempt:
"""一次尝试的计时句柄;`refund()` 把它退还给 stall 账(见 `StallClock`)。"""
__slots__ = ("productive",)
def __init__(self) -> None:
self.productive = True
def refund(self) -> None:
"""该次尝试不消耗重试预算(429),故其耗时归 stall 治理而非重试治理。"""
self.productive = False
class StallClock:
"""调用级 stall 计时器: 只累计非生产性等待(issue #8 设计 §3.1)。
**划分依据是"谁消耗重试预算"**,不是"是否发出了请求"。消耗 `max_attempts`
的时间已被重试预算治理,从 stall 账扣除;不消耗它的时间无人治理,归 stall。
两者重叠计费正是 issue #8 的根因: stall 预算(默认 300s)小于重试预算
(3 × timeout_s),必然先耗尽,于是重试预算在超时场景下永远用不上。
"生产性"的边界即 `_attempt` 的边界,含该次尝试的记账与遥测收尾——它们是
"尝试已有结论"之后的动作,不是在等待重试机会;把它们计入 stall 会让遥测
抖动参与判死。
**例外: 429 尝试须 `refund()`**。429 免重试预算(饱和期等待而非死亡),若其
耗时又算生产性,就掉进两个预算的缝隙——排队型网关持满 timeout 才回 429 时,
每轮只有退避那一两秒进 stall 账,调用可挂满 `stall_window/backoff_base` 轮
(实测 timeout=300/base=2 时达 25 小时)。退还后缝隙闭合。
每次调用创建一个实例。严禁提升为实例属性: `_entered_at` 会固定在进程启动
时刻,使 `stalled_s()` 随进程运行时长单调增长,最终所有调用被误判 stalled。
模块级共享单元, EmbeddingClient 与 OcrClient 复用(同 `backoff_delay`)。
"""
__slots__ = ("_now", "_entered_at", "_productive_s")
def __init__(self, now: Callable[[], float]) -> None:
self._now = now
self._entered_at = now()
self._productive_s = 0.0
def stalled_s(self) -> float:
"""非生产性等待累计秒数 = 调用总耗时 - 消耗重试预算的时间。"""
return self._now() - self._entered_at - self._productive_s
@contextlib.asynccontextmanager
async def attempting(self) -> AsyncIterator[_Attempt]:
"""包裹一次真实尝试,其耗时默认记为生产性(除非被 `refund()`)。"""
handle = _Attempt()
started = self._now()
try:
yield handle
finally:
# 只做算术与取值, 不吞任何异常——CancelledError 逐字穿透(库铁律)
if handle.productive:
self._productive_s += self._now() - started
def _demote_call_failures( def _demote_call_failures(
ordered: list[SourceConfig], ordered: list[SourceConfig],
attempt_fails: dict[str, int], attempt_fails: dict[str, int],
@@ -156,6 +217,15 @@ class _Failed:
immediate: bool immediate: bool
def _is_rate_limited(outcome: LLMResponse | _Failed) -> bool:
"""429 = 服务端调度指令(gRPC pushback 语义,迭代 5): 按 Retry-After 退避但
**不消耗重试预算**——饱和窗口里等待而非死亡;其余失败照常计数。
因其免重试预算,该次尝试的耗时必须归 stall 治理(`StallClock` 的 refund)。
"""
return isinstance(outcome, _Failed) and _failure_reason(outcome.exc) == "rate_limited"
class RetryMW: class RetryMW:
"""尝试编排器;时钟/睡眠/随机全部注入,纯确定性可测(P6)。""" """尝试编排器;时钟/睡眠/随机全部注入,纯确定性可测(P6)。"""
@@ -208,12 +278,12 @@ class RetryMW:
reasons: dict[str, str] = {} reasons: dict[str, str] = {}
# 调用内失败计数(设计 §3.3): 局部状态,调用结束即弃;严禁实例属性(并发共享) # 调用内失败计数(设计 §3.3): 局部状态,调用结束即弃;严禁实例属性(并发共享)
attempt_fails: dict[str, int] = {} attempt_fails: dict[str, int] = {}
entered_at = self._now() # 调用级累计计时,循环内不重置(CHS governance.py:207) # 调用级累计计时,循环内不重置(CHS governance.py:207);issue #8 起只计
# 非生产性等待——真实尝试由重试预算治理,不再重复烧 stall 预算
clock = StallClock(self._now)
while True: while True:
# 调用级时间上限(迭代 5): 429 免预算后的兜底,防饱和期无限循环。 # 调用级时间上限(迭代 5): 429 免预算后的兜底,防饱和期无限循环
# 与 _on_no_runnable 同款双条件(CHS 口径): 本地超窗且全局无进展才判死 if await self._stalled(clock):
stall = self._bp.stall_window_s
if self._now() - entered_at > stall and await self._quota.progress_age_s() > stall:
raise AllSourcesExhausted( raise AllSourcesExhausted(
scope=self._scope, scope=self._scope,
reason="stalled", reason="stalled",
@@ -222,14 +292,17 @@ class RetryMW:
) )
picked, gate_rejections = await self._pick_runnable(reasons, attempt_fails) picked, gate_rejections = await self._pick_runnable(reasons, attempt_fails)
if picked is None: if picked is None:
await self._on_no_runnable(gate_rejections, reasons, entered_at) await self._on_no_runnable(gate_rejections, reasons, clock)
continue continue
outcome = await self._attempt(request, *picked, reasons, attempt_fails) async with clock.attempting() as attempt:
outcome = await self._attempt(request, *picked, reasons, attempt_fails)
rate_limited = _is_rate_limited(outcome)
if rate_limited:
# 免了重试预算就得进 stall 账,否则这段耗时无人治理(见 StallClock)
attempt.refund()
if isinstance(outcome, LLMResponse): if isinstance(outcome, LLMResponse):
return outcome return outcome
# 429 = 服务端调度指令(gRPC pushback 语义,迭代 5): 按 Retry-After if not rate_limited:
# 退避但不消耗重试预算——饱和窗口里等待而非死亡;其余失败照常计数
if _failure_reason(outcome.exc) != "rate_limited":
fails += 1 fails += 1
if fails >= self._retry.max_attempts: if fails >= self._retry.max_attempts:
raise AllSourcesExhausted( raise AllSourcesExhausted(
@@ -282,8 +355,20 @@ class RetryMW:
await self._settle_and_release(permit, 0) await self._settle_and_release(permit, 0)
return None, gate_rejections return None, gate_rejections
# —— 背压与 stall 判死(CHS governance.py:270-285)——
async def _stalled(self, clock: StallClock) -> bool:
"""双条件 stall 判死(CHS governance.py:270-281): 本地累计等待与全局
无进展**同时**超窗才判死——本地 monotonic 与后端时钟刻意不混用。
本地一侧只计非生产性等待(issue #8,见 `StallClock`)。短路顺序有意为之:
本地未超窗就不问后端,省一次 Redis 往返。
"""
stall = self._bp.stall_window_s
return clock.stalled_s() > stall and await self._quota.progress_age_s() > stall
async def _on_no_runnable( async def _on_no_runnable(
self, gate_rejections: int, reasons: dict[str, str], entered_at: float self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
) -> None: ) -> None:
if gate_rejections == len(self._sources): if gate_rejections == len(self._sources):
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
@@ -299,10 +384,7 @@ class RetryMW:
retry_after_s=self._bp.poll_interval_s, retry_after_s=self._bp.poll_interval_s,
per_source_reasons=reasons, per_source_reasons=reasons,
) )
# 双条件 stall 判死(CHS governance.py:270-281): 本地累计等待与全局 if await self._stalled(clock):
# 无进展**同时**超窗才判死——本地 monotonic 与后端时钟刻意不混用。
stall = self._bp.stall_window_s
if self._now() - entered_at > stall and await self._quota.progress_age_s() > stall:
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
raise AllSourcesExhausted( raise AllSourcesExhausted(
scope=self._scope, scope=self._scope,
@@ -401,7 +483,7 @@ class RetryMW:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("治理记账写回降级(不冒泡): {}", exc) logger.warning("治理记账写回降级(不冒泡): {}", exc)
def _feed_outcome(self, source_name: str, ok: bool) -> None: def _feed_outcome(self, source_name: str, ok: bool) -> None:
src/polygateway/middleware/telemetry.py
+6 -2
View File
@@ -17,7 +17,11 @@ from typing import TYPE_CHECKING
from loguru import logger from loguru import logger
from polygateway.errors import GatewayUnavailableError, GovernanceBackendError from polygateway.errors import (
GatewayUnavailableError,
GovernanceBackendError,
SourceNotConfiguredError,
)
from polygateway.middleware.cache import digest_messages from polygateway.middleware.cache import digest_messages
from polygateway.types import canonical_sampling_json, merge_sampling from polygateway.types import canonical_sampling_json, merge_sampling
@@ -247,7 +251,7 @@ class TelemetryMW:
started = self._now() started = self._now()
try: try:
response = await call_next(request) response = await call_next(request)
except (GatewayUnavailableError, GovernanceBackendError) as exc: except (GatewayUnavailableError, GovernanceBackendError, SourceNotConfiguredError) as exc:
await self._emitter.emit_terminal_failure( await self._emitter.emit_terminal_failure(
request=request, request=request,
call_id=str(uuid.uuid4()), call_id=str(uuid.uuid4()),
src/polygateway/ocr.py
+12 -7
View File
@@ -30,11 +30,12 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
from polygateway.middleware.ratelimit import QuotaGate from polygateway.middleware.ratelimit import QuotaGate
from polygateway.middleware.retry import _failure_reason, backoff_delay from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
from polygateway.middleware.telemetry import TelemetryEmitter from polygateway.middleware.telemetry import TelemetryEmitter
from polygateway.ports import OutcomeAwareSelector from polygateway.ports import OutcomeAwareSelector
from polygateway.sources import SourceCooldownMemo from polygateway.sources import SourceCooldownMemo
@@ -203,13 +204,17 @@ class OcrClient:
raise AllSourcesExhausted(scope=self._scope, reason="no_sources", retry_after_s=0.0) raise AllSourcesExhausted(scope=self._scope, reason="no_sources", retry_after_s=0.0)
fails = 0 fails = 0
reasons: dict[str, str] = {} reasons: dict[str, str] = {}
entered_at = self._now() # 只计非生产性等待(issue #8): 真实尝试由重试预算治理,不重复烧 stall 预算
clock = StallClock(self._now)
while True: while True:
picked, gate_rejections = await self._pick_runnable(reasons) picked, gate_rejections = await self._pick_runnable(reasons)
if picked is None: if picked is None:
await self._on_no_runnable(gate_rejections, reasons, entered_at) await self._on_no_runnable(gate_rejections, reasons, clock)
continue continue
outcome = await self._attempt(kind, image, *picked, reasons, session_id, parent_call_id) async with clock.attempting():
outcome = await self._attempt(
kind, image, *picked, reasons, session_id, parent_call_id
)
if isinstance(outcome, _AttemptOutcome): if isinstance(outcome, _AttemptOutcome):
return outcome return outcome
fails += 1 fails += 1
@@ -252,7 +257,7 @@ class OcrClient:
return None, gate_rejections return None, gate_rejections
async def _on_no_runnable( async def _on_no_runnable(
self, gate_rejections: int, reasons: dict[str, str], entered_at: float self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
) -> None: ) -> None:
if gate_rejections == len(self._sources): if gate_rejections == len(self._sources):
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
@@ -269,7 +274,7 @@ class OcrClient:
per_source_reasons=reasons, per_source_reasons=reasons,
) )
stall = self._bp.stall_window_s stall = self._bp.stall_window_s
if self._now() - entered_at > stall and await self._quota.progress_age_s() > stall: if clock.stalled_s() > stall and await self._quota.progress_age_s() > stall:
names = tuple(s.name for s in self._sources) names = tuple(s.name for s in self._sources)
raise AllSourcesExhausted( raise AllSourcesExhausted(
scope=self._scope, scope=self._scope,
@@ -360,7 +365,7 @@ class OcrClient:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc) logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
async def _settle_and_release(self, permit: Permit) -> None: async def _settle_and_release(self, permit: Permit) -> None:
tests/unit/test_backpressure.py
+277 -6
View File
@@ -18,6 +18,7 @@ from polygateway.errors import (
SourceNotConfiguredError, SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.ratelimit import QuotaGate
from polygateway.middleware.retry import RetryMW, backoff_delay from polygateway.middleware.retry import RetryMW, backoff_delay
from polygateway.sources import RoundRobinSelector, SourceCooldownMemo from polygateway.sources import RoundRobinSelector, SourceCooldownMemo
from polygateway.types import ( from polygateway.types import (
@@ -52,19 +53,31 @@ class BoundedSleep:
await self._side_effect(len(self.delays)) await self._side_effect(len(self.delays))
def _mw(sources, limiter, script, *, clock, sleep, rng=lambda: 0.0, quota_full="wait", gate=None): def _mw(
sources,
limiter,
script,
*,
clock,
sleep,
rng=lambda: 0.0,
quota_full="wait",
gate=None,
transport=None,
emitter=None,
):
return RetryMW( return RetryMW(
scope="llm", scope="llm",
sources=sources, sources=sources,
selector=RoundRobinSelector(), selector=RoundRobinSelector(),
limiter=limiter, limiter=limiter,
gate=gate or InMemoryGate(config=_BREAKER, now=clock), gate=gate or InMemoryGate(config=_BREAKER, now=clock),
transport=FakeTransport(script), transport=transport or FakeTransport(script),
retry=RetryPolicy(max_attempts=3, backoff_base_s=2.0, backoff_max_s=30.0), retry=RetryPolicy(max_attempts=3, backoff_base_s=2.0, backoff_max_s=30.0),
backpressure=BackpressurePolicy(stall_window_s=_STALL, poll_interval_s=0.01), backpressure=BackpressurePolicy(stall_window_s=_STALL, poll_interval_s=0.01),
quota_full=quota_full, quota_full=quota_full,
cooldown_memo=SourceCooldownMemo(now=clock), cooldown_memo=SourceCooldownMemo(now=clock),
emitter=None, emitter=emitter,
now=clock, now=clock,
sleep=sleep, sleep=sleep,
rng=rng, rng=rng,
@@ -117,7 +130,11 @@ class TestStallQuadrants:
assert resp.content == "ok" assert resp.content == "ok"
async def test_global_stale_but_local_fresh_keeps_waiting(self): async def test_global_stale_but_local_fresh_keeps_waiting(self):
"""仅全局超窗(从未出餐 age=inf): 本地才刚开始等 → 不判死。""" """仅全局超窗(从未出餐 age=inf): 本地才刚开始等 → 不判死。
`inf` 语义在 issue #8 后未变;变的是"本地"的口径——它现在度量的是
非生产性等待累计,不再是墙钟总耗时(见 TestStallBudget)。
"""
clock = FakeClock() clock = FakeClock()
src, limiter = _blocked_limiter(clock) src, limiter = _blocked_limiter(clock)
held = await limiter.try_acquire("s1", 0) held = await limiter.try_acquire("s1", 0)
@@ -177,6 +194,216 @@ class TestStallQuadrants:
await task await task
class ClockAdvancingTransport:
"""按脚本 [(推进秒数, 动作), ...] 执行: 在一次尝试内部推进时钟, 模拟真实耗时。
动作语义同 `FakeTransport`(异常即抛、"hang" 即挂起、其余为返回值)。
stall 口径的关键区分在于"时间花在哪", 故必须能让时钟只在 transport 内前进。
"""
def __init__(self, script, clock):
self.script = list(script)
self.clock = clock
self.calls = []
async def complete(self, *, messages, source, stream, overlay, call_id):
self.calls.append((source.name, call_id))
advance, action = self.script.pop(0)
self.clock.advance(advance)
if isinstance(action, Exception):
raise action
if action == "hang":
await asyncio.Event().wait()
return action
class _SlowEmitter:
"""遥测收尾中推进时钟: 钉住"遥测耗时属生产性"(设计 §3.1 边界声明)。"""
def __init__(self, clock, advance):
self._clock = clock
self._advance = advance
async def emit_attempt(self, *args, **kwargs):
self._clock.advance(self._advance)
class TestStallBudget:
"""stall 预算只计非生产性等待(issue #8 设计 §3.1)。
根因是两个预算重叠计费: 真实尝试的耗时同时烧重试预算与 stall 预算,
而 stall 预算更小必然先耗尽, 于是 max_attempts 在超时场景下永不生效。
"""
def _free_limiter(self, clock):
src = make_source()
limiter = InMemoryLimiter(
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
)
return src, limiter
async def test_single_timeout_does_not_exhaust_stall_budget(self):
"""timeout_s == stall_window_s 时, 一次超时不得判死——重试预算须真实可用。"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
# 第一次尝试耗满 300s 超时后失败, 第二次立即成功
transport = ClockAdvancingTransport(
[(_STALL + 1, TransientError("timeout", status_code=504)), (0.0, _ok())], clock
)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(), transport=transport)
resp = await mw(_REQ)
assert resp.content == "ok"
assert len(transport.calls) == 2 # 第二次尝试确实发出了
async def test_productive_time_excluded_from_stall(self):
"""连续多次长尝试也不烧 stall 预算: 它们烧的是重试预算。"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
transport = ClockAdvancingTransport(
[
(_STALL + 100, TransientError("slow", status_code=500)),
(_STALL + 100, TransientError("slow", status_code=500)),
(0.0, _ok()),
],
clock,
)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(), transport=transport)
resp = await mw(_REQ)
assert resp.content == "ok"
async def test_telemetry_time_counts_as_productive(self):
"""遥测收尾属 `_attempt` 边界内: 遥测抖动不得参与判死(设计 §3.1)。
必须走**失败**路径才有判别力: 成功后直接 return, 循环开头的 stall
判定根本不会再执行。此处让首次尝试快速失败、而遥测收尾慢得超窗,
下一轮循环开头即检验遥测耗时有没有被算进 stall 账。
"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
transport = ClockAdvancingTransport(
[(0.1, TransientError("boom", status_code=500)), (0.0, _ok())], clock
)
mw = _mw(
[src],
limiter,
[],
clock=clock,
sleep=BoundedSleep(),
transport=transport,
emitter=_SlowEmitter(clock, _STALL + 100),
)
resp = await mw(_REQ)
assert resp.content == "ok"
async def test_nonproductive_wait_still_triggers_stall(self):
"""兜底未被削弱: 纯轮询等待累满窗口仍判死。"""
clock = FakeClock()
src, limiter = _blocked_limiter(clock)
_held = await limiter.try_acquire("s1", 0)
async def advance(_n):
clock.advance(_STALL + 100)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(advance))
with pytest.raises(AllSourcesExhausted) as ei:
await mw(_REQ)
assert ei.value.reason == "stalled"
async def test_saturation_429_still_stalls(self):
"""429 免预算不烧 fails, 主循环兜底须仍能判死而非无限循环(设计 §3.5)。"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
# 429 往返本身极快(生产性可忽略), 退避 sleep 才是非生产性的大头
transport = ClockAdvancingTransport(
[(0.1, TransientError("429", status_code=429)) for _ in range(10)], clock
)
async def advance(_n):
clock.advance(_STALL)
mw = _mw(
[src], limiter, [], clock=clock, sleep=BoundedSleep(advance), transport=transport
)
with pytest.raises(AllSourcesExhausted) as ei:
await mw(_REQ)
assert ei.value.reason == "stalled" # 不是 retry_exhausted: 429 确实没烧重试预算
async def test_slow_429_does_not_escape_both_budgets(self):
"""排队型网关: 持满 timeout 才回 429。该耗时必须落进 stall 账。
429 免重试预算, 所以它的耗时若又算生产性就**两个预算都不烧**——调用
会挂满 stall_window/backoff_base 轮。修复前实测 301 次尝试、25.2 小时;
此处钉住"一轮 429 就把 stall 账推满"这个上界。
"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
transport = ClockAdvancingTransport(
[(_STALL + 1, TransientError("429", status_code=429))] * 20, clock
)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(), transport=transport)
with pytest.raises(AllSourcesExhausted) as ei:
await mw(_REQ)
assert ei.value.reason == "stalled"
# 一次持满超时的 429 即耗尽 stall 窗口, 不再无限排队
assert len(transport.calls) <= 2
async def test_cancel_inside_attempt_pierces(self):
"""取消发生在 `attempting()` 包裹内仍逐字穿透(库铁律)。"""
clock = FakeClock()
src, limiter = self._free_limiter(clock)
transport = ClockAdvancingTransport([(0.0, "hang")], clock)
mw = _mw([src], limiter, [], clock=clock, sleep=asyncio.sleep, transport=transport)
task = asyncio.create_task(mw(_REQ))
while not transport.calls:
await asyncio.sleep(0.01)
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert (await limiter.source_stats("s1")).inflight == 0 # permit 在 finally 释放
async def test_clock_is_per_call_not_per_instance(self):
"""StallClock 必须是**调用级**局部状态,不得提升为 RetryMW 实例属性。
生产形态是一个长寿命 RetryMW 跑成千上万次调用。若 clock 成了实例属性,
`_entered_at` 会固定在进程启动时刻, 每次调用的 stalled_s() 随进程运行
时长单调增长, 最终所有调用被误判 stalled——这是本用例要拦的灾难。
判别力的关键是**复用同一个 mw**: 两个 mw 实例天然隔离, 抓不到实例共享。
"""
clock = FakeClock()
src = make_source()
limiter = InMemoryLimiter(
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
)
transport = ClockAdvancingTransport([(0.0, _ok("first")), (0.0, _ok("second"))], clock)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(), transport=transport)
first = await mw(_REQ)
clock.advance(_STALL + 100) # 两次调用之间进程空转远超窗
second = await mw(_REQ)
assert (first.content, second.content) == ("first", "second")
async def test_concurrent_calls_do_not_share_clock(self):
"""并发两路共用同一个 mw: 一快一慢都能正常完成(形态冒烟)。
**这条不是回归防线**: 实测它在"clock 提为实例属性""去掉 refund""去掉
生产性扣减"三种变异下均保持绿色——共享 clock 时慢调用的耗时是作为
credit 记进共享账的,污染方向是让 stall 账**变小**(更宽松),而本用例
断言两路都成功。真正钉住调用级隔离的是上面那条
`test_clock_is_per_call_not_per_instance`。保留此条只为覆盖并发形态。
"""
clock = FakeClock()
src = make_source(max_concurrency=2)
limiter = InMemoryLimiter(
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
)
transport = ClockAdvancingTransport(
[(_STALL + 100, _ok("slow")), (0.0, _ok("fast"))], clock
)
mw = _mw([src], limiter, [], clock=clock, sleep=BoundedSleep(), transport=transport)
results = await asyncio.gather(mw(_REQ), mw(_REQ))
assert {r.content for r in results} == {"slow", "fast"}
class _GateSuccessBroken(InMemoryGate): class _GateSuccessBroken(InMemoryGate):
async def record_success(self, entry): async def record_success(self, entry):
raise GovernanceBackendError("redis 抖动", scope="llm") raise GovernanceBackendError("redis 抖动", scope="llm")
@@ -192,6 +419,12 @@ class _LimiterProgressBroken(InMemoryLimiter):
raise GovernanceBackendError("redis 抖动", scope="llm") raise GovernanceBackendError("redis 抖动", scope="llm")
class _GateSuccessMisconfigured(InMemoryGate):
# 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常
async def record_success(self, entry, *, count_attempt: bool = True):
raise SourceNotConfiguredError("未知源 's1'(scope=llm)")
class TestAccountingDegradation: class TestAccountingDegradation:
"""记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。""" """记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。"""
@@ -206,6 +439,24 @@ class TestAccountingDegradation:
resp = await mw(_REQ) resp = await mw(_REQ)
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃 assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
async def test_assembly_defect_on_accounting_path_also_degrades(self):
"""记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。
`SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若
`_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、
销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账
方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。
"""
clock = FakeClock()
src = make_source()
limiter = InMemoryLimiter(
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
)
gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock)
mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate)
resp = await mw(_REQ)
assert resp.content == "ok"
async def test_mark_progress_failure_does_not_lose_response(self): async def test_mark_progress_failure_does_not_lose_response(self):
clock = FakeClock() clock = FakeClock()
src = make_source() src = make_source()
@@ -280,14 +531,34 @@ class TestUnknownSourceIsAssemblyDefect:
# 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信 # 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信
assert not isinstance(ei.value, GatewayUnavailableError) assert not isinstance(ei.value, GatewayUnavailableError)
@pytest.mark.parametrize("method", ["try_acquire", "stats"])
async def test_survives_the_quota_gate_wrapper(self, method):
"""必须穿透 QuotaGate,否则整个拆分在生产路径上等于没做。
上面两条(以及 redis 版)打的都是私有 `_cfg`,绕过了包装器。而治理循环
只经 QuotaGate 访问后端,包装器的 `except Exception` 会把装配缺陷重新
包成 `GovernanceBackendError`——下游又拿到可重投异常,永远重投不告警。
"""
src = make_source("s1")
# 限流后端的源名单与治理循环拿到的源对不上 = 装配缺陷
limiter = InMemoryLimiter(
scope="llm", sources={"other": src}, global_limits=_NO_GLOBAL
)
gate = QuotaGate(limiter, scope="llm")
with pytest.raises(SourceNotConfiguredError) as ei:
await getattr(gate, method)(src)
assert not isinstance(ei.value, GatewayUnavailableError)
class TestGateFailuresReachCallersAsScopeLevel: class TestGateFailuresReachCallersAsScopeLevel:
"""三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。 """闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路 记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果 抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复 是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
的故障。三条路径逐一钉住,防止将来任何一条被漏掉。 的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s,
`BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly`
包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。
""" """
async def test_try_acquire_failure_is_scope_level(self): async def test_try_acquire_failure_is_scope_level(self):
tests/unit/test_embedding.py
+60
View File
@@ -173,6 +173,8 @@ from polygateway.types import ( # noqa: E402
GlobalLimits, GlobalLimits,
RetryPolicy, RetryPolicy,
) )
from tests.contracts.conftest import FakeClock # noqa: E402
from tests.unit.test_backpressure import BoundedSleep # noqa: E402
_BREAKER = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0) _BREAKER = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0)
_NO_GLOBAL = GlobalLimits(max_concurrency=0, rpm=0, tpm=0) _NO_GLOBAL = GlobalLimits(max_concurrency=0, rpm=0, tpm=0)
@@ -208,6 +210,31 @@ class ScriptedEmbedTransport:
return action return action
class _ClockAdvancingEmbedTransport:
"""按脚本 [(推进秒数, 动作), ...] 执行: 在一次尝试内部推进时钟(issue #8)。
动作语义同 `ScriptedEmbedTransport`。stall 口径要区分"时间花在哪",
故必须能让时钟只在 transport 内前进。
"""
def __init__(self, script, clock):
self.script = list(script)
self.clock = clock
self.calls = []
async def embed(self, *, texts, source, call_id):
self.calls.append((source.name, list(texts), call_id))
advance, action = self.script.pop(0)
self.clock.advance(advance)
if isinstance(action, Exception):
raise action
if action == "hang":
await asyncio.Event().wait()
if action == "ok":
return _vec_for(texts)
return action
class _MemoryRecorder: class _MemoryRecorder:
def __init__(self): def __init__(self):
self.rows = [] self.rows = []
@@ -338,6 +365,39 @@ class TestEmbedGovernance:
await task await task
assert (await limiter.source_stats("e1")).inflight == 0 assert (await limiter.source_stats("e1")).inflight == 0
async def test_single_timeout_does_not_exhaust_stall_budget(self):
"""issue #8: 一次耗满 timeout 的尝试不得吃掉 stall 预算而使重试失效。
embedding 只有 `_on_no_runnable` 一处 stall 判定, 故失效链条是
"先超时一次(墙钟耗尽) → 再遇到无可用源 → 判死"。此处正是这条路径。
"""
clock = FakeClock()
limiter = held = None # 闭包延迟求值: client 建好后才有 limiter
async def toggle_permit(n):
"""首次退避占满 permit, 迫使下一轮走 _on_no_runnable; 之后放行。"""
nonlocal held
if n == 1:
held = await limiter.try_acquire("e1", 0)
else:
await held.release()
# 第一次尝试耗满 300s 超时失败, 随后被迫走一轮 _on_no_runnable——
# stall 判定就在那里, 检验它有没有把这 300s 生产性时间算进 stall 账
transport = _ClockAdvancingEmbedTransport(
[(300.1, TransientError("timeout", status_code=504)), (0.0, "ok")], clock
)
client, limiter = _embed_client(
[_src(max_concurrency=1)],
[],
now=clock,
transport=transport,
sleep=BoundedSleep(toggle_permit),
)
resp = await client.embed(["a"])
assert resp.vectors == [[1.0]]
assert len(transport.calls) == 2 # 第二次尝试确实发出了
class TestEmbedTelemetry: class TestEmbedTelemetry:
async def test_per_batch_rows_with_digest(self): async def test_per_batch_rows_with_digest(self):
tests/unit/test_ocr_client.py
+48
View File
@@ -80,6 +80,22 @@ class ScriptedOcrTransport:
raise NotImplementedError raise NotImplementedError
class ClockAdvancingOcrTransport(ScriptedOcrTransport):
"""按脚本 [(推进秒数, 动作), ...] 在一次尝试内部推进时钟(issue #8)。
stall 口径要区分"时间花在哪",故必须能让时钟只在 transport 内前进。
"""
def __init__(self, script, clock):
super().__init__([a for _, a in script])
self._advances = [d for d, _ in script]
self.clock = clock
async def _next(self, method, source, call_id):
self.clock.advance(self._advances.pop(0))
return await super()._next(method, source, call_id)
class StaticSelector: class StaticSelector:
def order(self, sources, stats): def order(self, sources, stats):
return list(sources) return list(sources)
@@ -292,6 +308,38 @@ class TestBackpressure:
await permit.settle(0) await permit.settle(0)
await permit.release() await permit.release()
async def test_single_timeout_does_not_exhaust_stall_budget(self):
"""issue #8: 一次耗满 timeout 的尝试不得吃掉 stall 预算而使重试失效。
OCR 只有 `_on_no_runnable` 一处 stall 判定,故失效链条是"先超时一次
(墙钟耗尽)→ 再遇到无可用源 → 判死"。此处正是这条路径。
"""
clock = FakeClock()
limiter = held = None
rounds = []
async def toggle_permit(_seconds):
"""首次退避占满 permit,迫使下一轮走 _on_no_runnable;之后放行。"""
nonlocal held
rounds.append(_seconds)
if len(rounds) > 10:
raise RuntimeError("超过 10 次轮询仍未判死/未获 permit")
if len(rounds) == 1:
held = await limiter.acquire("m1", 0)
else:
await held.settle(0)
await held.release()
transport = ClockAdvancingOcrTransport(
[(300.1, TransientError("timeout", status_code=504)), (0.0, "text")], clock
)
client, limiter, _ = _client(
[_src(max_concurrency=1)], [], now=clock, sleep=toggle_permit, transport=transport
)
r = await client.recognize_text(b"jpg")
assert r.text == "LINE-1"
assert len(transport.calls) == 2 # 第二次尝试确实发出了
class FakeClock: class FakeClock:
def __init__(self, start=1000.0): def __init__(self, start=1000.0):