feat: suppress consecutive-channel opening on evidently healthy sources

Round 8 forensics caught the healthy source circuit-opened by five
random empty completions (~20% ambient failure rate makes a 5-streak
land every ~3000 attempts), blacking out the only good source for 60s.
When the window holds min_calls samples below the failure-rate
threshold, a streak is noise and no longer opens the gate; cold-start
and low-traffic semantics are unchanged and sudden death of a warm
source is still caught by the rate channel.
This commit is contained in:
2026-07-21 15:28:34 -04:00
parent 0e0d599441
commit 121888a0eb
4 changed files with 60 additions and 5 deletions
@@ -110,6 +110,11 @@
**动因**: 第六轮实测网关限速是**按账号速率**而非并发——外部负载高峰时我方仅 26 req/min 仍吃 16% 429,AIMD(并发型)压不住速率型限流;429 每次消耗 1/3 重试预算,饱和窗口里调用被"429 链"干净杀死(16 retry_exhausted/336)。
**修正**: 携 Retry-After 的 429 是**服务端调度指令而非失败**(gRPC A6 pushback / SRE 语义): 照常退避(与 Retry-After 取大)、照常喂 AIMD/健康分,但**不消耗重试预算**;新增**调用级时间上限**(复用 `stall_window_s`,缺省 300s)在重试循环顶部兜底——持续饱和超窗即按既有 `stalled` 语义抛出,防无限循环。对下游的承诺变化: 饱和期调用最长等待 stall_window 而非快速失败(生产语义,迁移文档标注)。
### 3.39 迭代 6 补遗: 健康证据抑制连续通道(2026-07-21,第八轮取证驱动)
**动因**: 第八轮失败链取证发现健康源1 自己进 cooldown——空补全率 ~20% 下 5 连败每 ~3000 次尝试随机发生一次,连续通道(阈值 5)把唯一好源关 60s,期间调用全灭(链样本: `{'minimax_1': 'cooldown', 其余: 'cooldown/rate_limited'}`)。
**修正**(Resilience4j 精神: 率证据优于连败直觉): 窗口样本 ≥ min_calls 且失败率 < fail_rate 时**抑制连续通道开路**——有充分健康证据的源上连败是统计噪声。连续通道本职(冷启动/低流量快杀死源)保留(窗口样本不足时照常开路);温热源猝死由率通道在失败累积后接管(检出延迟从 threshold 次升至率窗口收敛,契约测试⑩钉住);SourceDead force_open 不受影响。
### 3.4 不做与预留(方案 C 组件的接入点)
- 账号级 429 共享退避: 不做;预留 = 冷却备忘 key 从 source_name 换 account_key 即可接入(LiteLLM 先例,治理粒度=配额粒度原则记入 ARCHITECTURE)。