fix: return 429 attempt time to the stall budget

Independent verification found the first cut had swapped one bug for a
worse one. The budgets were split by "did we send a request", so a 429
attempt counted as productive — but 429 is exempt from the retry budget,
so its time burned neither budget. Against a queueing gateway that holds
the request for the full timeout before answering 429, a call could hang
for 301 attempts / 25.2 hours, measured, versus 301 seconds before the
change.

The split is now by which budget the time consumes: time that burns
max_attempts is excluded from stall, time that does not (429 attempts
included) belongs to stall. Measured again: back to one attempt / 301s.

Only the chat loop needs this — embedding and ocr count 429 against
max_attempts unconditionally, so the gap never existed there. The stall
verdict moved into _stalled(), which both call sites had duplicated, to
keep __call__ under the complexity gate.
This commit is contained in:
2026-08-06 10:55:51 -04:00
parent bc4683d1f5
commit a0a5cf7ecc
5 changed files with 104 additions and 28 deletions
+1 -1
View File
@@ -427,7 +427,7 @@ flowchart TB
- `RedisLimiter`: 移植 CHSAnalyzer 六道闸——单条 Lua 原子检查全局并发/单源并发(ZSET 租约)/全局 RPM/单源 RPM/全局 TPM/单源 TPM;窗口 id 用 **Redis 服务器时钟**(TIME 命令)统一多进程口径。随实现移植契约测试。
- `InMemoryLimiter`: 同一契约的进程内实现(semaphore + 滑动窗口计数);单进程场景下语义等价。
- **配额满行为可配**: `wait`(等待,配 stall 判定——本地等待超窗 + 全局无进展超窗双条件才判卡死)或 `fail-fast`(立即抛)。
- **stall 计时口径(2026-08-06 修正,issue #8,设计 `designs/2026-08-06-issue8-stall-budget-design.md`)**: 双条件的**条件 A 只累计非生产性等待**(429 退避、配额 wait 轮询、熔断冷却),真实尝试的耗时由 `StallClock.attempting()` 从 stall 账中扣除。原实现用墙钟总耗时,使真实尝试同时向重试预算与 stall 预算计费;而 stall 预算(默认 300s)小于重试预算(`max_attempts × timeout_s`),必然先耗尽——`timeout_s ≥ stall_window_s` 时一次超时即判 scope 死,`max_attempts` **静默失效**。修正后两个预算正交:**真实尝试烧 `max_attempts`,其余一切等待烧 `stall_window_s`**,与"429 不烧重试预算故 429 等待烧 stall 预算"同构。生产性边界即 `_attempt` 边界(含该次记账与遥测收尾),故遥测抖动不参与判死。`stall_window_s``timeout_s` 自此**无耦合**,无需按 `timeout × retries` 放大。三条治理循环(chat/embedding/ocr)共用 `middleware/retry.py``StallClock`。条件 B 的 `inf` 语义未动——新口径下"非生产性排队耗满窗口且 scope 从未出餐"判死本就正当。
- **stall 计时口径(2026-08-06 修正,issue #8,设计 `designs/2026-08-06-issue8-stall-budget-design.md`)**: 双条件的**条件 A 只累计非生产性等待**(429 退避、配额 wait 轮询、熔断冷却),真实尝试的耗时由 `StallClock.attempting()` 从 stall 账中扣除。原实现用墙钟总耗时,使真实尝试同时向重试预算与 stall 预算计费;而 stall 预算(默认 300s)小于重试预算(`max_attempts × timeout_s`),必然先耗尽——`timeout_s ≥ stall_window_s` 时一次超时即判 scope 死,`max_attempts` **静默失效**。修正后两个预算正交,**划分依据是"谁消耗重试预算"而非"是否发出请求"**: 烧 `max_attempts` 的时间不烧 `stall_window_s`,不烧 `max_attempts` 的时间归 `stall_window_s`**429 尝试因此也计入 stall 账**——它免重试预算,若其耗时又算生产性就两个预算都不烧,排队型网关(持满 timeout 才回 429)下调用可挂 25 小时(实施期独立验证实测,见设计 §3.6)。生产性边界即 `_attempt` 边界(含该次记账与遥测收尾),故遥测抖动不参与判死。`stall_window_s``timeout_s` 自此**无耦合**,无需按 `timeout × retries` 放大。三条治理循环(chat/embedding/ocr)共用 `middleware/retry.py``StallClock`。条件 B 的 `inf` 语义未动——新口径下"非生产性排队耗满窗口且 scope 从未出餐"判死本就正当。
- 全局活性信号: `mark_progress()`/`progress_age_s()`("最近一次出餐"时刻)供背压 stall 判定,移植 `CHSAnalyzer limiter.py:193`
- **契约补强(2026-07-20,CHS 迁移缺口 G6)**: `settle()`/`release()` 幂等(重复调用无副作用);装配期守卫——`timeout_s ≤ permit 租约 TTL`(防租约先于请求过期)、`stall_window ≥ 最慢源 TTFT 上限`(防误判卡死;**issue #8 后为保守冗余**——TTFT 等待属生产性时间已不计入 stall,该误判在机制上不再可能,校验保留因其无害且不误拒合理配置),违反直接报错拒绝装配。降级方向细化(2026-07-20 M1): "报错不放行"适用于**准入侧**(try_acquire/try_enter 及选源路径消费的 source_stats/retry_after_s);已成功调用后的 settle/release 释放侧失败降级 warning——释放失败不构成放行,且不得掩盖主异常与取消。**勘误(2026-07-20 M2 设计,人类批准)**: 记账侧的 `record_success`/`record_failure`/`mark_progress` 同归此类——调用已真实完成,后端失败若冒泡会丢弃真实成功响应或掩盖原始尝试异常,故降级 warning(CHS 原版一律报错,此为有意反转;丢一次熔断记账最多延迟状态迁移且方向偏保守,epoch fencing 防污染)。