Compare commits
12 Commits
296c765337
...
f31f7caf99
| Author | SHA1 | Date | |
|---|---|---|---|
 iomgaa
|
f31f7caf99 | ||
|
iomgaa
|
41bca375d2 | ||
|
iomgaa
|
84ee6dee84 | ||
|
iomgaa
|
c5b2b3fade | ||
|
iomgaa
|
5a025b6e5d | ||
|
iomgaa
|
d9ceaecf20 | ||
|
iomgaa
|
2a9bc44abf | ||
|
iomgaa
|
6edf4ac9de | ||
|
iomgaa
|
eb956b2cdf | ||
|
iomgaa
|
8edd3fb2cd | ||
|
iomgaa
|
942af99856 | ||
|
iomgaa
|
0b3e84b3be |
+6
-1
@@ -48,7 +48,12 @@ LLM_CIRCUIT_BREAKER_COOLDOWN=60 # 或 LLM__BREAKER__COOLDOWN_S
|
||||
# LLM__BREAKER__MAX_COOLDOWN_S=300 # 开路指数退避封顶(缺省 max(300, cooldown))
|
||||
# ── AIMD 自适应并发(M2.5,库常量非 env 键): 每源初始 8,429 ×0.5,成功 +1/limit,
|
||||
# ── ceiling = max(64, 源级 MAX_CONCURRENCY);禁用需构造函数注入自定义 pacer ──
|
||||
# LLM__QUOTA_FULL=wait # wait(默认) | fail_fast
|
||||
# LLM__QUOTA_FULL=wait # 配额满: wait(默认) | fail_fast
|
||||
# ── 熔断全拒时的处置(issue #14)。单源 scope 建议 wait: 只有一个源时
|
||||
# ── "停用这个源"等于"整个 scope 停服",fail_fast 会让开路期间的每次调用
|
||||
# ── 在几毫秒内死掉且 MAX_ATTEMPTS 一格用不上。wait 不削弱保护(等待期照样
|
||||
# ── 不发请求),只是把最坏墙钟拉长到 BACKPRESSURE__STALL_WINDOW_S ──
|
||||
# LLM__CIRCUIT_OPEN=fail_fast # 熔断开路: fail_fast(默认) | wait
|
||||
|
||||
# ══ 装配选择(PGW_*)══
|
||||
PGW_LIMITER_BACKEND=memory # memory | redis(redis 需 REDIS_URL;多进程 worker 必须 redis)
|
||||
|
||||
@@ -1,5 +1,27 @@
|
||||
# Changelog
|
||||
|
||||
## 1.2.4(2026-08-20)
|
||||
|
||||
熔断开路时,调用方第一次可以选择**等**而不是当场失败(issue #14)。此前准入侧有一格是空的:限流闸满时库允许排队(`{SCOPE}__QUOTA_FULL=wait|fail_fast`,缺省 `wait`),熔断门拒绝时**只有 fail-fast 一档且不可配**——而两者在准入语义上是同构的,都没发出请求、都带着"稍后再来"的提示。新键 `{SCOPE}__CIRCUIT_OPEN=fail_fast|wait` 补上这一格,形状与 `QUOTA_FULL` 逐项对齐。
|
||||
|
||||
**缺省是 `fail_fast`,即今天的行为**,存量部署无需改动任何配置。要改的是单源 scope:熔断的设计前提是"这个源坏了,把流量导到别的源",只配了一个源时这个前提不成立,同一段代码做的事就变成"这个源坏了,所以整个 scope 停止服务"。提交方实测:中转抖动 36 秒(22 次尝试 / 19 次 503)触发失败率通道开路,随后 30 次调用全部在 7-74 毫秒内失败,`MAX_ATTEMPTS=8` 一格没用上,一条跑了 3 小时 18 分钟的实验臂当场报废。配 `wait` 之后,熔断对配额和钱包的保护完整保留(等待期照样一个请求都不发),改变的只是调用方当场死还是排队等;代价是单次调用最坏墙钟被拉长——上限是 `STALL_WINDOW_S`(缺省 300 秒)。**但 `wait` 并不豁免重试预算**: 冷却结束后放行的探针是一次真实尝试,失败照样烧一格 `MAX_ATTEMPTS`,所以密钥失效(401/403)这类一击即熔的源通常更早以 `reason=retry_exhausted` 失败,而不是等满窗口后的 `stalled`;两者哪个先到取决于 `MAX_ATTEMPTS` 与冷却时长、`STALL_WINDOW_S` 的相对大小。库无法区分"密钥坏了"和"中转抖了",选 `wait` 就是声明"宁可等也不当场死"。
|
||||
|
||||
### 请先读这一条: `retry_after_s` 在半开状态下的取值变了(缺省档同样生效)
|
||||
|
||||
`retry_after_s` 从来没有写下来的定义,于是两个后端各自发挥、互相漂移。现在它只回答一个问题:**距离确定可再试的时刻还有多久**。健康与准入允许 → `0.0`;开路 → 剩余冷却;**半开(探针在途)→ `0.0`**,因为探针随时可能出结果,不存在确定的时刻——而 `0 = 可立即重试` 本就是这个字段的既有约定。
|
||||
|
||||
变更点在半开:此前返回的是**探针租约剩余**。那是个死锁保护参数,派生自 `max(2 × 最慢源 TIMEOUT_S, COOLDOWN_S, TIMEOUT_S + 5)`,与"这个源多久能恢复"没有任何因果关系。`TIMEOUT_S=300` 的部署里它是 600 秒,而冷却期只有 60 秒。**照它延期重投的下游,等的是一个物理上无意义的数。**
|
||||
|
||||
更重的后果在库内,提交方也没发现:这个值被写进了源冷却备忘,而备忘的 `set_until` 取更晚者、不可回退。于是——源开路、冷却到期、调用①拿到探针、并发的调用②被拒并给该源记下 600 秒本地冷却、调用①的探针成功、门恢复 CLOSED——**本进程此后仍然跳过这个健康的源将近 10 分钟**。单源下每次调用照旧抛 `CircuitOpenError`;多源部署同样中招,只是别的源接住了流量,池子越大越隐蔽。修正后备忘写进的是一个已经过期的时刻,自动回到"只记开路的确定冷却期"。
|
||||
|
||||
同批统一了两个后端在**六个出口**上的口径。其中四处是既有的分叉:Redis 在授予探针时返回探针 TTL、在写回被 fencing 拒时返回租约剩余,而内存后端一直返回 0。契约测试此前只钉了"第二个进入者会被拒绝",从没钉过它拿到的是什么数,这个盲区把分叉掩护到了今天。
|
||||
|
||||
### 其他
|
||||
|
||||
- `_pick_runnable`/`_on_no_runnable` 此前在 chat/embedding/OCR 三条治理循环里各存一份逐字复制,现收敛为 `middleware/admission.py::SourceAdmission` 一份。行为不变——差异用注入表达(调用内降权传空计数时恒等、AIMD pacer 为 `None` 时跳过),`permit` 结算的 warning 文案由三种归一为一种。
|
||||
- `GatewayUnavailableError` 的文档收回了重试职责:调用级的重试、退避、换源、等待冷却全部在库内,本异常表示那份预算已经用尽;下游据此再投属于**任务级**重试,语义不同。此前那句"业务侧 catch 本类做延期重投"读起来像在鼓励每个下游各写一份重试逻辑,而两边各写一份必然漂移。
|
||||
|
||||
|
||||
## 1.2.3(2026-08-19)
|
||||
|
||||
遥测表 `llm_calls` 的结构变更从此**由下游掌控**(issue #13)。此前两个后端都会在初始化期对下游数据库发 DDL:表不存在则建表,表存在但缺列则逐列 `ALTER TABLE ADD COLUMN`,而补列**没有任何开关**——库一升级、下次调用即自动执行。在共享的生产 Postgres 上这有三重问题:`ALTER` 取 ACCESS EXCLUSIVE 锁会排在长事务后阻塞该表其后的所有查询(而遥测是业务路径上的内联 `await`),多进程多版本共存时谁先补列是竞态,且这些 DDL 不进任何迁移记录、事后无从审计。调研过的 11 个同类系统(Celery / APScheduler / Alembic / Django contrib / Hangfire / Quartz.NET / dbt / Airbyte / Fivetran / Prefect / Airflow)里没有一个把它作为默认行为。
|
||||
|
||||
@@ -13,9 +13,9 @@
|
||||
| 多源多账号 | `{SCOPE}__{PROVIDER}__{N}__*` 配置任意多源;健康感知选源(EWMA×在途 P2C)自动避开坏源 |
|
||||
| 限流 | 并发/RPM/TPM × 全局/单源六道闸;TPM 预扣入场、按实际用量结算退款;Redis 后端跨进程原子(Lua) |
|
||||
| 错误分类重试 | 一切失败落入四分类(见下),由分类决定重试/换源/熔断;429 属 pushback 不消耗重试预算;退避含 jitter 且尊重 Retry-After |
|
||||
| 熔断 | 双通道(连续失败 + 失败率窗口,健康证据抑制误熔);半开单探针带租约(持有者死亡自动回收);epoch fencing 拒绝迟到写回;开路时长指数递增 |
|
||||
| 熔断 | 双通道(连续失败 + 失败率窗口,健康证据抑制误熔);半开单探针带租约(持有者死亡自动回收);epoch fencing 拒绝迟到写回;开路时长指数递增;**开路时当场失败还是等冷却可配**(`CIRCUIT_OPEN`,单源 scope 应配 `wait`) |
|
||||
| 自适应并发 | AIMD:429 削减、成功缓升,防止打爆上游 |
|
||||
| 背压与判死 | 配额满可选等待或快速失败;等待期按双条件判死(本地非生产性等待与全局无进展**同时**超窗)。stall 窗口只计**非生产性**等待(429 退避/配额轮询/熔断冷却),与 `TIMEOUT_S` 无耦合 |
|
||||
| 背压与判死 | 配额满与熔断开路**各自**可选等待或快速失败(`QUOTA_FULL` / `CIRCUIT_OPEN`,两键不可互相替代);等待期按双条件判死(本地非生产性等待与全局无进展**同时**超窗)。stall 窗口只计**非生产性**等待(429 退避/配额轮询/熔断冷却),与 `TIMEOUT_S` 无耦合 |
|
||||
| 响应缓存 | Redis/内存;key 含 model + messages 摘要 + namespace(缓存隔离单位)+ salt + 采样参数,多模态 content 先摘要再 hash(防毒化);可 per-call 绕过(科研重采样) |
|
||||
| 流式看门狗 | TTFT / inter-token / 总超时三层活性;thinking token 刷活性不计结果;截断流(缺 `[DONE]`)判瞬时不入缓存 |
|
||||
| 遥测与成本 | 每次调用(含缓存命中与失败)必录 24 字段;SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 |
|
||||
@@ -33,7 +33,7 @@
|
||||
|
||||
```bash
|
||||
pip install --extra-index-url https://gitea.iomgaa.online/api/packages/iomgaa/pypi/simple/ \
|
||||
"polygateway[redis,postgres,structured]>=1.2.3,<2"
|
||||
"polygateway[redis,postgres,structured]>=1.2.4,<2"
|
||||
```
|
||||
|
||||
核心仅依赖 `httpx` + `pydantic`;按需选 extras:
|
||||
@@ -400,7 +400,7 @@ SQLite 侧**不建议**对着一个大库文件跑 `DELETE` + `VACUUM`,而应**
|
||||
|---|---|
|
||||
| `{SCOPE}__{PROVIDER}__{N}__{FIELD}` | 第 N 个源;FIELD **全集** = BASE_URL/API_KEY/MODEL/TIMEOUT_S/MAX_CONCURRENCY/RPM/TPM/EST_TOKENS/TTFT_TIMEOUT_S/INTER_TOKEN_TIMEOUT_S/ENABLE_THINKING/MISSING_DONE/TRUST_ENV/EXTRA_BODY(表外的 FIELD 直接报错) |
|
||||
| `{SCOPE}__GLOBAL__*` | scope 级全局限额(跨源并发/RPM/TPM) |
|
||||
| `{SCOPE}__RETRY__*` / `BREAKER__*` / `BACKPRESSURE__*` / `SELECTOR` / `QUOTA_FULL` | per-scope 韧性参数;缺省回落平铺键(`LLM_MAX_RETRIES` 等,兼容旧项目习惯) |
|
||||
| `{SCOPE}__RETRY__*` / `BREAKER__*` / `BACKPRESSURE__*` / `SELECTOR` / `QUOTA_FULL` / `CIRCUIT_OPEN` | per-scope 韧性参数;缺省回落平铺键(`LLM_MAX_RETRIES` 等,兼容旧项目习惯) |
|
||||
| `{SCOPE}__BATCH_SIZE` / `NORMALIZE` / `EXPECTED_DIM` | 仅 `EmbeddingClient` 消费;`BATCH_SIZE` 必填(分批是行为关键,不设默认) |
|
||||
| `PGW_LIMITER_BACKEND` / `PGW_BREAKER_BACKEND` | `memory`(单进程)或 `redis`(跨进程共享,需 `REDIS_URL`) |
|
||||
| `PGW_CACHE_BACKEND` | `none` / `memory` / `redis`;非 `none` 时需 `PGW_CACHE_NAMESPACE` + `PGW_CACHE_TTL_S`(须 > 0) |
|
||||
@@ -409,6 +409,14 @@ SQLite 侧**不建议**对着一个大库文件跑 `DELETE` + `VACUUM`,而应**
|
||||
| `PGW_TELEMETRY_TEXT_CAP` | 可选正整数:遥测落库正文的字符上限(作用于每条消息的文本 `content`、多模态 part 的 `text`、`response`、`thinking`);**不设 = 不截断**,详见[合规下游的推荐配置](#6-合规下游的推荐配置) |
|
||||
| `PGW_PRICING_PATH` / `PGW_STRUCTURED_MAX_RETRIES` / `PGW_LEASE_TTL_S` | 可选:价格表(缺省则成本恒 `None`)/ 结构化重问上限(缺省 2)/ permit 租约秒数(缺省 1500,须 ≥ 最大源 `TIMEOUT_S`) |
|
||||
|
||||
**`{SCOPE}__CIRCUIT_OPEN=fail_fast|wait`(缺省 `fail_fast`)——单源 scope 请配 `wait`**
|
||||
|
||||
熔断的设计前提是"这个源坏了,把流量导到别的源"。**只配了一个源时这个前提不成立**,同一段代码做的事变成"这个源坏了,所以整个 scope 停止服务":开路期间每一次调用都在几毫秒内失败,`MAX_ATTEMPTS` 一格用不上,一个网络包都没发出去。中转抖动几十秒就足以打断一条跑了几小时的长任务。
|
||||
|
||||
`wait` 档改变的**只是**"调用方当场失败还是排队等":等待期间照样一个请求都不发,熔断对配额和钱包的保护完整保留。代价是单次调用的最坏墙钟被拉长,上限为 `{SCOPE}__BACKPRESSURE__STALL_WINDOW_S`(缺省 300 秒)。**`wait` 不豁免重试预算**——冷却结束后放行的探针是一次真实尝试,失败照样烧一格 `MAX_ATTEMPTS`;因此密钥失效(401/403)这类一击即熔的源通常更早以 `reason=retry_exhausted` 失败,而非等满窗口的 `stalled`。库无法区分"密钥坏了"和"中转抖了",选 `wait` 就是声明"宁可等也不要当场死"。多源部署保持 `fail_fast`:有源可换时,换源比等待快。
|
||||
|
||||
该键与 `{SCOPE}__QUOTA_FULL` 同形但**不可互相替代**:配额满是"排队等自己的份额"(必然轮到),熔断开路是"等这个源恢复"(未必恢复),所以两者分开配置。
|
||||
|
||||
两个易被忽略的源级键:`MISSING_DONE` 决定 SSE 缺 `[DONE]` 时的处置(`retry` 默认判瞬时重试 / `salvage` 收下已收内容并把用量可信度降为 `estimated`;零内容恒 `retry`,不受该键影响);`EXTRA_BODY` 是该源**恒定**的采样参数(JSON 对象串,并入请求体,优先级低于 `chat(overlay=...)`),禁用键 `model` / `messages` / `stream` / `stream_options` 配了直接报错,OCR 与 EMBED scope 不消费该键(配了忽略并 warning)。
|
||||
|
||||
`SCOPE` 是逻辑角色(LLM/VLM/OCR/EMBED/JUDGE/SEARCH…任意大写名),同一进程可按角色装配多个 client,各自独立配置与治理状态。
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "polygateway"
|
||||
version = "1.2.3"
|
||||
version = "1.2.4"
|
||||
description = "PolyGateway：实验室统一的大语言模型（LLM/VLM/OCR）调度与中转库——多源、限流、重试、熔断、缓存、遥测"
|
||||
# registry 包页面的正文只认这一项:缺了页面就是一片空白(1.1.2 的教训,twine 会警告
|
||||
# long_description missing 但不阻塞上传)。README 在打包时被固化进产物,发布后再改无效。
|
||||
|
||||
@@ -472,6 +472,12 @@ flowchart TB
|
||||
|
||||
**M2.5 双通道开路(2026-07-21,设计 designs/2026-07-21-m25-resilience-design.md;对 CHS 连续失败语义的有意扩展)**: P6 压测实证纯连续失败语义对"高失败率但偶尔成功"的半死源失明(10% 成功率源永不开路,吃掉 76% 尝试)。判据改为满足任一即开路——① 连续失败 ≥ 阈值(CHS 兼容,保留);② 窗口(双 30s 桶,服务器钟)样本 ≥ `min_calls`(缺省 10)且失败率 ≥ `fail_rate`(缺省 0.6)。**429 不入两通道**(限速是背压不是源故障,Envoy outlier detection 同款;交健康选源软处理);ResultInvalid/网关健康拒绝不计窗口样本(坏结果 ≠ 坏服务)。开路时长指数递增 `cooldown × 2^(streak-1)` 封顶 `max_cooldown_s`(缺省 max(300, cooldown)),仅率通道开路与探针失败重开递增 streak(连续通道误熔健康源的代价封顶单次 cooldown);CLOSED 稳定满 2×cooldown_eff 后首次成功衰减归零。探针撞 429 按无果归还语义放下家接管。原则沉淀: **治理状态的粒度必须等于配额的粒度**(限流/账号退避按配额主体建 key;缓存 key 含租户同理)。
|
||||
|
||||
**熔断拒绝补齐等待档(2026-08-19,issue #14,设计 `designs/2026-08-19-issue14-admission-wait-policy-design.md`;人类确认缺省与实施边界)**: 准入侧此前有一格是空的——限流闸满时库允许排队(`{SCOPE}__QUOTA_FULL=wait|fail_fast`,缺省 wait),熔断门拒时**只有 fail-fast 一档且不可配**。两者在准入语义上同构(都不发请求、都带 `retry_after` 提示),处置却分叉。补上 `{SCOPE}__CIRCUIT_OPEN=fail_fast|wait`(缺省 **fail_fast**,不跟随 quota_full——把最坏墙钟从毫秒抬到 stall 窗口是"快速失败 → 长时间挂起"这个最危险的方向,不能强加给存量下游)。`wait` 档下熔断的保护作用完整保留(等待期一个请求都不发),改变的只是调用方当场死还是排队等。**这一格的缺失与源数量无关**: 多源全部同时开路(共同上游挂掉、全网抖动)行为一模一样,单源只是把"全部开路"的概率从罕见变成必然;故实现上**严禁按池大小分叉**(`if len(sources) == 1` 会让行为随配置突变且无法组合测试)。等待时长按 `retry_after_s` 睡到冷却截止(而非 `poll_interval` 空转——60 秒冷却用 10ms 轮询是 6000 次往返 × 每个在途调用),抖动**上**加不缩放(对确定的截止时刻提前醒必然白醒),并夹到剩余 stall 预算,故单次调用最坏墙钟 = `stall_window_s` + 一个 poll 间隔,不随 `max_cooldown_s` 漂移。控制流必须**按拒绝原因分派**而非串行: 串行写法下 `circuit_open=wait` 不抛之后会掉进配额分支,`quota_full=fail_fast` 的调用方会收到 `reason=quota_exhausted` 而配额其实是满的。
|
||||
|
||||
**`retry_after_s` 的契约定死(同批,issue #14)**: 语义 = "距离**确定**可再试的时刻还有多久"。CLOSED/准入允许 → `0.0`(现在就能试);OPEN → 剩余冷却(确定时刻);**HALF_OPEN → `0.0`**——探针随时可能出结果,不存在确定时刻,而 `0 = 可立即重试` 本就是库既有约定。此前 HALF_OPEN 返回**探针租约剩余**,那是死锁保护参数(派生自 `max(2 × 最慢源 timeout_s, cooldown_s, timeout_s + 5)`),与"源多久能恢复"无因果关系: 现场 `TIMEOUT_S=300` 时它是 600s 而冷却只有 60s。**更重的后果不在对外报数而在库内**: 该值被喂进源冷却备忘(`SourceCooldownMemo.set_until` 取更晚者、不可回退),于是探针成功、门已恢复 CLOSED 之后,本进程仍跳过该源整整一个租约——单源下每次调用照旧判死,多源下则是"池子里少一个源"且被其他源接住流量所掩盖(issue 提交方未发现这一条)。修正后备忘写入的是已过期时刻,自动回归"只记 OPEN 的确定冷却期"。契约在**六个出口**上统一(memory 三处 + redis 六个 Lua 返回格),其中后四处是**既有的双后端分叉**(redis 在授予探针时返回 probe TTL、在 fencing 未命中时返回租约剩余,而 memory 一直是 0),由契约测试盲区掩护至今——旧用例只钉"第二个进入者被拒",从没钉它拿到什么数。
|
||||
|
||||
**准入逻辑三处收敛(同批)**: `_pick_runnable`/`_on_no_runnable` 此前在 `middleware/retry.py`、`embedding.py`、`ocr.py` 各存一份逐字复制(后两份是第一份的子集)。准入语义一直在演进(issue #8 的 stall 口径、M2.5 的 pacer、本次的等待档),每次都要三处同步。收敛为 `middleware/admission.py::SourceAdmission`,差异用注入表达而非分支: 调用内降权传空 `attempt_fails` 时恒等、AIMD pacer 为 `None` 时跳过。`QuotaGate`/`BreakerGate`/`AdaptivePacer` 由三条循环持有并与 admission **共享同一实例**(三处 `_attempt` 仍要用它们做记账写回与 `pacer.leave()`;pacer 有在途计数,分裂成两个计数器会让 admit/enter 与 leave 记到不同账上),`SourceCooldownMemo` 归 admission 独占。
|
||||
|
||||
### 7.5 响应缓存
|
||||
|
||||
**key 公式**: `sha256(canonical_json({model, messages_digest, namespace, salt, sampling}))`,前缀 `pgw:cache:`。
|
||||
@@ -578,6 +584,7 @@ src/polygateway/
|
||||
- **per-scope 韧性配置(2026-07-20,CHS 迁移缺口 G4)**: 韧性参数支持按 scope 覆盖——`{SCOPE}__RETRY__MAX_ATTEMPTS` / `{SCOPE}__BREAKER__FAIL_THRESHOLD` / `{SCOPE}__BREAKER__COOLDOWN_S` / `{SCOPE}__BACKPRESSURE__STALL_WINDOW_S` / `{SCOPE}__SELECTOR` / `{SCOPE}__GLOBAL__MAX_CONCURRENCY|RPM|TPM`(CHS 现状: VLM 与 OCR 两 scope 参数各异)。平铺键(`LLM_*`)是单 scope 场景的简写;两者并存时 scope 键优先。
|
||||
- **装配只有两条路**: `GatewayClient.from_env()`/`from_settings(settings)`(工厂,覆盖 90% 用户;补上三项目每次手写、GovDoc 缺失的"配置→client"一段)或构造函数全量依赖注入(测试/高级用户)。库内部任何组件**不得自读环境变量**(显式优于隐式)。
|
||||
- 后端选择即配置: 如 `PGW_LIMITER_BACKEND=memory|redis`、`PGW_TELEMETRY_BACKEND=sqlite|postgres`、`PGW_QUOTA_FULL=wait|fail_fast`(命名待 M1 设计文档定稿)。
|
||||
- **`{SCOPE}__CIRCUIT_OPEN=fail_fast|wait`(2026-08-19,issue #14)**: 熔断全拒时的处置,与 `{SCOPE}__QUOTA_FULL` 同形同族(上一条"后端选择即配置"里记的 `PGW_QUOTA_FULL` 是 M1 定稿前的暂拟名,实际落地为 scope 键 `{SCOPE}__QUOTA_FULL`)。缺省 **fail_fast** = 存量下游的控制流逐字不变;**单源 scope 应显式配 `wait`**。两键值域相同但语义不同故分列: 配额满是"排队等自己的份额"(必然轮到),熔断开路是"等这个源恢复"(未必恢复),调用方可能想要"配额满就等、源坏了就立刻失败"。落到 `GatewaySettings.circuit_open`(无默认值,与既有全部字段一致),校验收敛在唯一消费者 `SourceAdmission` 一处——三个客户端构造函数此前各带一份 `quota_full` 校验,再加一键就是八处复制。
|
||||
- **`PGW_TELEMETRY_SCHEMA_MODE=auto|manual`(2026-08-19,issue #13,D15)**: 可选键、**三态**——不设 = 按后端派生(sqlite→auto、postgres→manual),显式设置则两侧都可覆盖。派生只发生在 config 层一处,落到 `GatewaySettings.telemetry_auto_migrate`(无默认值,与既有全部字段一致;`telemetry_backend=none` 时无人消费,归一为 `False`),recorder 的 `auto_migrate` 是 keyword-only **必填**参数——关键行为参数不给默认值(P4),缺省规则也就不会与类签名漂移。
|
||||
- **`PGW_TELEMETRY_TEXT_CAP`(2026-08-19,issue #12)**: 可选正整数键、**二态**——不设 = 不截断(缺省)。与相邻的 `SCHEMA_MODE` 不同,这里"未设"本身就是最终答案,没有需要按后端派生的第二种缺省。落到 `GatewaySettings.telemetry_text_cap: int | None`(同样无默认值),`TelemetryEmitter.text_cap` 是 keyword-only 必填参数。值域(`> 0`)在 settings 与 emitter **两处**校验: 前者只管 env 一条路,而"构造函数全量注入"是库承诺的另一条公共装配路,`text_cap=0` 会让每条正文只剩一个省略标记(P5 不得静默)。
|
||||
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
# 熔断拒绝补齐等待档: 把"源不健康"与"调用判死"解耦
|
||||
|
||||
- **issue**: #14(dissect,单源第三方中转部署)
|
||||
- **核查基准**: HEAD 1.2.3;issue 按 1.2.1 提交,逐条复核后**全部仍然成立**(`backends/memory/breaker.py` md5 `630ed36ddeb87e08a9bac58260056046`,1.0.6→1.2.3 逐字节未变)
|
||||
- **状态**: 人类已确认(2026-08-19);经 Codex 审查修正(2026-08-19,修正点见 §3.1/§3.4/§3.5/§6 标注),待实施
|
||||
|
||||
## 1. 问题的真实形状
|
||||
|
||||
issue 把问题命名为"单源 scope 下熔断等于整体停服"。这个命名会把方案引向错误的方向——**单源不是病因,是让病灶 100% 复现的放大器**。三条独立缺陷叠加成了现场那 30 次瞬死,必须分开命名才修得干净。
|
||||
|
||||
### 1.1 缺陷一: 准入策略矩阵缺了一格
|
||||
|
||||
`_pick_runnable` 有四种"拒绝",库对它们的处置并不对称:
|
||||
|
||||
| 拒绝原因 | 计入 `gate_rejections` | 全被拒时的处置 | 可配? |
|
||||
|---|---|---|---|
|
||||
| `rate_limited`(permit 拿不到) | 否 | 走 `quota_full` 分支 | **是**(`wait`/`fail_fast`) |
|
||||
| `adaptive_paced`(AIMD 超限) | 否 | 走 `quota_full` 分支 | **是**(同上) |
|
||||
| `circuit_open`(熔断门拒) | 是 | 当场抛 `CircuitOpenError` | **否** |
|
||||
| `cooldown`(源冷却备忘) | 是 | 同上 | **否** |
|
||||
|
||||
限流闸满时库不判死、允许排队(`quota_full=wait`,缺省);熔断门拒时库**只有 fail-fast 一档且不可配**。两者在准入语义上完全同构(都不发请求、都带 `retry_after` 提示),处置却分叉。
|
||||
|
||||
**这一格的缺失与源数量无关**:多源全部同时开路(共同上游的中转挂了、一次全网抖动)时行为一模一样。单源只是把"全部开路"的概率从"罕见"变成"必然"。因此**任何形态的单源特判(`if len(sources) == 1`)都是错的**——它会让行为随池大小突变、无法组合测试,是比现状更重的债。
|
||||
|
||||
### 1.2 缺陷二: `retry_after_s` 在 HALF_OPEN 下返回了一个物理上无意义的数
|
||||
|
||||
`try_enter` 在 HALF_OPEN 拒绝时返回 `probe_expires - now`,即**探针租约的剩余时长**。而 `probe_ttl_s` 派生自 `max(2 × 最慢源 timeout_s, cooldown_s, timeout_s + 5)`(`config.py:400-407`),现场 `TIMEOUT_S=300` ⇒ **600 秒**,而冷却期只有 60 秒。
|
||||
|
||||
探针租约的长度回答的是"探针最长可以占用这个名额多久"(死锁保护参数),与"这个源多久能恢复"没有任何因果关系。两个后端同款(`backends/redis/breaker.py` 的 `TRY_ENTER`/`RETRY_AFTER` 两个 Lua 均返回 `probe_until - now`)。
|
||||
|
||||
### 1.3 缺陷三(issue 未发现,伤害最重): 恢复了的源被本进程屏蔽整个探针租约
|
||||
|
||||
缺陷二的值被喂进了源冷却备忘:
|
||||
|
||||
```text
|
||||
retry.py:354 self._memo.set_until(cand.name, self._now() + entry.retry_after_s)
|
||||
sources.py:139 self._until[name] = max(已有, until) # 取更晚者,不可回退
|
||||
```
|
||||
|
||||
于是:源 A 冷却到期 → 调用 1 拿到探针 → 并发的调用 2 被拒、拿到 600 → **给 A 记 600 秒本地冷却** → 调用 1 的探针成功、门恢复 CLOSED → **本进程此后 600 秒仍然跳过 A**,且 `reasons[A]="cooldown"` 计入 `gate_rejections`,单源下每次调用照旧抛 `CircuitOpenError`。
|
||||
|
||||
实测复现(`InMemoryGate` + 注入时钟,`cooldown_s=60`、`probe_ttl_s=600`):
|
||||
|
||||
```text
|
||||
B 决定: allowed=False state=half_open retry_after_s=600.0 <- 冷却只有 60s
|
||||
B 给 s1 记的本地冷却剩余: 600.0 秒
|
||||
探针成功后门 state: closed
|
||||
门已 CLOSED,memo.active('s1') = True
|
||||
再过 120 秒(远超 60s 冷却)memo.active = True 剩余 480.0 秒
|
||||
```
|
||||
|
||||
**这条与源数量、与是否单源都无关**:多源部署里,一个源每开路一次就会被本进程从池中除名 `probe_ttl_s`(可达 2 × timeout),池子越大越难被观测到,因为别的源接住了流量。现场那"30 次瞬死横跨 20 秒"里有多少来自这一条无法反推,但机制确凿。
|
||||
|
||||
## 2. 备选方案与否决理由
|
||||
|
||||
issue 给了 A/B/C/D 四条。逐条判:
|
||||
|
||||
| 方案 | 判定 | 理由 |
|
||||
|---|---|---|
|
||||
| A `PGW_BREAKER_BACKEND=noop` | **否决** | 关掉的是"保护"(401/403/配额耗尽的一击即熔一并失效,坏密钥持续撞墙),而诉求是"别当场判死"。且开了"治理组件可整个关掉"的先例,限流迟早跟进。三条缺陷一条都不解决 |
|
||||
| B `{SCOPE}__CIRCUIT_OPEN=wait\|fail_fast` | **采纳为主干** | 与 `quota_full` 严格同构,补的正是 §1.1 那一格。但 issue 版的 B 未答"wait 档等多久",而这个答案依赖 C |
|
||||
| C 修 HALF_OPEN 的 `retry_after_s` | **采纳,且不是"治标"** | issue 把它列为"可并行的小修"。实际上它是 B 的**前提**:wait 档要按 `retry_after` 睡,睡一个 600 秒的假数就是新事故。它还是 §1.3 的病根 |
|
||||
| D 只写文档 | **否决** | 把配置项的副作用固化成公开契约,将来动阈值逻辑即破坏;且解决不了 `force_open` |
|
||||
|
||||
**方案 = B + C,合并为一件事**:B 依赖 C 的正确性,C 修完 §1.3 自动消失。
|
||||
|
||||
## 3. 设计
|
||||
|
||||
### 3.1 `retry_after_s` 的契约定死为"确定的最早可尝试时刻"
|
||||
|
||||
| 门状态 | 返回值 | 依据 |
|
||||
|---|---|---|
|
||||
| CLOSED | `0.0` | 现状,不变 |
|
||||
| OPEN | `open_until - now` | 现状,不变。冷却截止是确定时刻 |
|
||||
| HALF_OPEN(被拒) | **`0.0`** | 探针随时可能出结果,**不存在**确定的等待时刻 |
|
||||
|
||||
`0.0` 不是新约定:`errors.py` 早已定义 `retry_after_s` 的 `0 = 可立即重试`,契约测试 `test_retry_after_semantics` 也以"健康 → 0、冷却到期 → 0"钉着这个语义。HALF_OPEN 归入"无确定等待"是同一语义的自然延伸,而非发明。
|
||||
|
||||
信息不丢失:`GateDecision.state` 已经携带 `HALF_OPEN`,调用方要区分"门闭着"与"探针在途"照样能区分。
|
||||
|
||||
**惊群由既有机制承担,不由这个数承担**:门自身的单探针租约保证第二个 caller 拿不到名额;wait 档的复查间隔由 middleware 的 `poll_interval_s` 抖动睡眠承担(§3.3)。
|
||||
|
||||
**§1.3 随之闭合**:`set_until(now + 0.0)` 写入一个已过期的截止时刻,`active()` 恒 False——HALF_OPEN 拒绝自此不再污染备忘,无需在 `retry.py` 加任何状态分支。备忘回归它唯一正当的用途:**记 OPEN 的确定冷却期**。
|
||||
|
||||
**准入被允许时恒 `0.0`**:`allowed=True` 意味着现在就能试,这个字段没有别的合理取值。
|
||||
|
||||
**改动面是五个出口,不是两个(Codex 审查修正)**。原稿只点了 `try_enter` 与 `retry_after_s()`,漏了 `GateUpdate` 那一侧;逐一核实后发现**两个后端在这两处本就已经分叉**——本 issue 的病根正是"`retry_after_s` 语义从未被定死,于是各后端各自发挥",不一并收口就是定了新契约却留两个后端不遵守:
|
||||
|
||||
| 出口 | memory 现状 | redis 现状 | 统一为 |
|
||||
|---|---|---|---|
|
||||
| `try_enter` 拒绝(OPEN) | `open_until - now` | 同 | 不变 |
|
||||
| `try_enter` 拒绝(HALF_OPEN) | `probe_expires - now` | `probe_until - now` | **`0.0`** |
|
||||
| `try_enter` **授予探针** | `0.0`(`memory:114`) | **`probe_ttl_ms`**(`redis:53`) | **`0.0`**(redis 侧改) |
|
||||
| `GateUpdate`(fencing 未命中,HALF_OPEN) | `0.0`(`memory:175-177` 非 OPEN 一律 0) | **`probe_until - now`**(`redis:127/158/258`) | **`0.0`**(redis 侧三处改) |
|
||||
| `retry_after_s()` 跨源取 min | HALF_OPEN 记 `probe_expires - now` | 同 | **HALF_OPEN 记 `0.0`** |
|
||||
|
||||
后两行是**既有缺陷**,与本 issue 同源、由契约测试盲区掩护至今(现有用例只钉"第二个进入者被拒",没钉它拿到什么数)。同源缺陷一并修,不作为独立议题。
|
||||
|
||||
memory 侧抽 `_remaining(g)` 私有纯方法供三处共用;redis 侧四个 Lua(`TRY_ENTER`/`RECORD_SUCCESS`/`RECORD_FAILURE`/`RELEASE_PROBE`)与 `RETRY_AFTER` 各改一处(Lua 无法共享函数,这是既有约束,`_WINDOW_HELPERS` 已是同款处理),由同一批双后端参数化契约用例锁死。
|
||||
|
||||
### 3.2 新配置键 `{SCOPE}__CIRCUIT_OPEN`
|
||||
|
||||
与 `quota_full` 逐项对齐,不发明新形状:
|
||||
|
||||
| 维度 | `quota_full`(既有) | `circuit_open`(新增) |
|
||||
|---|---|---|
|
||||
| 合法域 | `_QUOTA_FULL = {"wait","fail_fast"}` | `_CIRCUIT_OPEN = {"wait","fail_fast"}` |
|
||||
| 缺省 | `wait` | **`fail_fast`**(见 §3.5) |
|
||||
| env 键 | `{SCOPE}__QUOTA_FULL` | `{SCOPE}__CIRCUIT_OPEN` |
|
||||
| 装配 | settings → `GatewayClient` → 三条循环 | 同 |
|
||||
| 校验 | `_validate_backends` 表驱动 + 构造期 | 同(各加一行) |
|
||||
|
||||
改动面: `config.py`(常量 / 字段 / 校验元组 / `from_env` 各一行)、`client.py`(签名 + 透传各一处)、`SourceAdmission`(§3.4)一处。
|
||||
|
||||
### 3.3 `_on_no_runnable` 的控制流
|
||||
|
||||
现状两个分支是**串行**的。今天走不到那个坑(没有 wait 档,第一分支必抛),但**只要把第一分支改成"wait 时不抛"就会立刻踩中**:控制流会往下掉进 `quota_full` 分支,`quota_full=fail_fast` 的调用方会看到熔断等待被误报成 `reason="quota_exhausted"`。必须改成按拒绝原因分派:
|
||||
|
||||
```text
|
||||
if gate_rejections == len(sources): # 全部因熔断类原因被拒
|
||||
if circuit_open == "fail_fast": raise CircuitOpenError(retry_after=gate.retry_after_s(names))
|
||||
hint = await gate.retry_after_s(names) # OPEN 有确定值;全 HALF_OPEN 得 0
|
||||
else: # 至少一源是被配额/AIMD 挡的
|
||||
if quota_full == "fail_fast": raise AllSourcesExhausted("quota_exhausted")
|
||||
hint = 0.0
|
||||
if await self._stalled(clock): raise AllSourcesExhausted("stalled", ...)
|
||||
await self._sleep(self._nap(hint, clock))
|
||||
```
|
||||
|
||||
睡眠时长 `_nap(hint, clock)`,三条约束同时满足:
|
||||
|
||||
| 约束 | 实现 | 理由 |
|
||||
|---|---|---|
|
||||
| 不空转 | `hint > 0` 时睡到冷却结束再加抖动,而非 50ms 轮询 | 60 秒冷却下,`poll_interval=0.05` 会产生 1200 次无谓复查;memory 后端只是字典查询,**redis 后端是 1200 次往返 × 每个在途调用** |
|
||||
| 不白醒 | 抖动**上**加(`hint + poll_interval × (0.5+0.5×rng)`),不缩放 | 对一个确定的截止时刻提前醒必然被再拒一次 |
|
||||
| 等待有可解释上界 | 夹到剩余 stall 预算:`min(睡眠, stall_window - clock.stalled_s())`,下界 `poll_interval` | 最迟在 stall 窗口耗尽那一刻醒来判死,单次调用最坏墙钟 = `stall_window_s`(缺省 300s),不随 `max_cooldown_s` 漂移 |
|
||||
|
||||
`hint = 0` 时该式退化为现有的 `poll_interval × (0.5+0.5×rng)`,配额等待路径逐字不变。
|
||||
|
||||
**计时归属无需改动**:这段睡眠发生在 `clock.attempting()` 之外,自动计入 stall 账,与 ARCH §7.3 "熔断冷却属非生产性等待"的既定口径一致。
|
||||
|
||||
### 3.4 前置收敛: 准入逻辑三处复制归一
|
||||
|
||||
`_pick_runnable` / `_on_no_runnable` 目前在 `middleware/retry.py`、`embedding.py`、`ocr.py` **各有一份**,后两份是第一份的逐字子集(少 AIMD pacer 与调用内降权)。若只改 chat 一处,embedding/ocr 就成了行为分叉的角落——**那才是本次真正会留下的技术债**(CLAUDE.md 铁律痛斥的"三项目 4 处复制"的库内同款)。
|
||||
|
||||
抽 `middleware/admission.py::SourceAdmission`,持有 sources/selector/QuotaGate/BreakerGate/memo/backpressure/两个策略键/时钟三件套,暴露 `pick()` 与 `on_no_runnable()`。三条循环的差异用注入表达,不留分支:
|
||||
|
||||
| 差异 | 处理 | 行为等价性 |
|
||||
|---|---|---|
|
||||
| 调用内降权(仅 chat) | `attempt_fails` 作 `pick()` 入参 | embedding/ocr 传空 dict 时 `_demote_call_failures` 恒等返回原序(`demoted` 为空即 `return ordered`) |
|
||||
| AIMD pacer(仅 chat) | `pacer: AdaptivePacer \| None = None` | None 时跳过 `admit`/`enter`,无副作用 |
|
||||
| `_settle_and_release` 三份复制 | 提为 `middleware/` 模块级 async 函数 | chat/embedding 签名为 `(permit, actual)`,**OCR 为 `(permit)` 且体内恒 `settle(0)`**(`ocr.py:438`,Codex 审查补)。OCR 侧改为传 `0`,逐字等价;唯一可见变化是 warning 文案由"OCR permit 结算/释放失败"归一 |
|
||||
|
||||
已逐字 diff 核实(`embedding` 与 `ocr` 两份**完全相同**;chat 多出的只有上表三类)。另有两处**不在抽取边界内**、须原样保留:chat 主循环顶部额外的一次 `_stalled` 预判(`retry.py:286`),以及 OCR 的健康喂数——它们属于各自的主循环与 `_attempt`,本次一行不动。
|
||||
|
||||
**这不是任务外重构**:修复本来就必须落在这三处,"改三遍"与"抽一份改一遍"工作量相当而后者才符合 P7;且这是既有方向的延续——`StallClock` 与 `backoff_delay` 已按同一原则收敛为共享单元(ARCH §7.3)。边界严格限定在准入与无源可跑的处置,**`_attempt` 一行不动**(三者差异大: 流式 / 批 / 图)。
|
||||
|
||||
执行分两个提交:①纯重构,验收标准是全套件逐字绿、无行为变更;②在单一位置加语义。①先行以保回滚点。
|
||||
|
||||
### 3.5 缺省值取 `fail_fast`
|
||||
|
||||
`quota_full` 缺省 `wait`,但 `circuit_open` **不跟随**,理由是变更方向的危险性不对称:
|
||||
|
||||
| 取值 | 对存量下游的影响 |
|
||||
|---|---|
|
||||
| `fail_fast`(采纳) | **控制流**逐字不变(全源被熔断拒仍当场抛 `CircuitOpenError`) |
|
||||
| `wait` | 把所有人的最坏墙钟从毫秒抬到 `stall_window_s`,且是"快速失败 → 长时间挂起"这个最危险的方向 |
|
||||
|
||||
issue 的诉求本身也不是改默认值,而是**表达能力**——其 §2.3 的原话是"库对这两种情形用的是同一套默认值、且**不允许调用方表达自己属于哪一种**"。多源下 fail-fast 确实是对的(换源比等待快),单源下调用方显式配 `wait` 即可。README 与 wiki 需明写"单源 scope 建议配 `wait`"。
|
||||
|
||||
### 3.6 `errors.py` 的职责边界补写
|
||||
|
||||
issue 要求修订 `GatewayUnavailableError` 那句"业务侧 catch 本类做延期重投"——它读起来像在鼓励每个下游各写一份重试逻辑。改为明确边界:调用级的重试/退避/换源/等待**全部在库内**,本异常表示库的调用级预算(重试预算或 stall 预算)已耗尽;下游若要再投,那是**任务级重试**,语义与调用级重试不同。
|
||||
|
||||
这不是新决策,是把 ARCH §7.2 已经写明的"单层重试原则"补进 docstring。零代码风险。
|
||||
|
||||
**"缺省档零感知"须诚实收窄(Codex 审查修正)**: 缺省档保证的是**控制流**不变,不是零可见变更。`retry_after_s` 的语义修正在缺省档下同样生效——全源 HALF_OPEN 时 `CircuitOpenError.retry_after_s` 由"探针租约剩余"变为 `0.0`,而它是公开字段(`errors.py:118`)。这正是本次记 **1.3.0** 而非补丁号、且 CHANGELOG 需"请先读这一条"待遇的原因。另需注意 `GatewaySettings` 全部字段均无默认值(既有风格),新增 `circuit_open` 沿用之,直接构造该类的调用方须补一个参数。
|
||||
|
||||
## 4. 行为矩阵
|
||||
|
||||
| 场景 | `fail_fast`(缺省,= 现状) | `wait` |
|
||||
|---|---|---|
|
||||
| 单源 OPEN,冷却 60s | 立即 `CircuitOpenError(retry_after=剩余冷却)` | 睡到冷却结束(夹在 stall 预算内)→ 探针 → 成功即返回 |
|
||||
| 单源 `force_open`(401/403) | 立即失败 | 等 60 → 探针又 401(**烧掉一格 `max_attempts`**)→ 等 120 → …… 以**先耗尽的那个预算**的 reason 失败: `max_attempts` 先尽则 `retry_exhausted`,冷却累计超过 stall 预算则 `stalled`。**代价须进文档** |
|
||||
| 多源部分开路 | 不变(有源可跑就不进这个分支) | 不变 |
|
||||
| 多源全部开路 | 立即失败 | 等最早恢复的那个源(`retry_after_s` 取 min) |
|
||||
| 全部 HALF_OPEN(探针在途) | `CircuitOpenError(retry_after=0)`,语义准确(随时可能好) | `poll_interval` 抖动复查,秒级拿到探针结果 |
|
||||
| 配额满 / AIMD 超限 | 归 `quota_full` 管,逐字不变 | 逐字不变 |
|
||||
|
||||
## 5. 测试策略
|
||||
|
||||
行为变更须"先失败后通过"(CLAUDE.md 测试结果门)。分三层:
|
||||
|
||||
**契约层**(`tests/contracts/test_breaker_contract.py`,双后端参数化自动覆盖 memory + redis):
|
||||
按 §3.1 那张表**逐个出口**钉——HALF_OPEN 被拒、授予探针、`GateUpdate` fencing 未命中、`retry_after_s()` 探针在途,四处均须 `== 0.0`;OPEN 语义不变(现有 `test_retry_after_semantics` 保持绿)。现有用例只钉了"第二个进入者被拒",没钉它拿到什么数,正是这个盲区放过了两处双后端分叉。Redis 侧依赖时间快进的变体在契约层会 skip,须同步补 `tests/integration/test_redis_governance_time.py` 的真实等待变体(既有约定,不缩放时长)。
|
||||
|
||||
**单元层**(`tests/unit/test_backpressure.py` 邻域,注入时钟/睡眠/rng):
|
||||
§1.3 的回归钉子——探针成功后备忘不再屏蔽该源(直接由 §3.1 的复现脚本转化);`circuit_open=wait` 下全源开路不抛 `CircuitOpenError` 而按 `retry_after` 睡;`wait` + `quota_full=fail_fast` 组合下熔断等待**不**被误报成 `quota_exhausted`(§3.3 那个坑的钉子);`wait` 档最坏墙钟 ≤ `stall_window_s` 且判死 reason 为 `stalled`、`per_source_reasons` 含 `circuit_open`;`fail_fast` 缺省下全部现有用例逐字绿。
|
||||
|
||||
**收敛层**: §3.4 的重构提交以"三条循环现有测试全绿、零新增用例"为验收——有新增用例即说明行为被动了。
|
||||
|
||||
## 6. 非功能与已知取舍
|
||||
|
||||
| 维度 | 结论 |
|
||||
|---|---|
|
||||
| 取消穿透 | `_nap` 的长睡眠是 `await self._sleep(...)`,`CancelledError` 逐字穿透;无新增 finally 资源 |
|
||||
| 后端往返 | wait 档每个冷却周期约 1 次 gate 查询(vs. `poll_interval` 轮询的 1200 次),Redis 压力低于按现状实现的朴素 wait |
|
||||
| 遥测 | **不加列**。wait 等待期不发请求,无 attempt 行可记;调用级总等待下游可自测。进入/退出等待各打一条 `logger.info`(scope、per-source reasons、预计等待),使"等了多久"可从日志还原 |
|
||||
| 等待上界的精确值 | `_stalled` 判据是 `>` 而非 `>=`(`retry.py:368`,Codex 审查补)。睡眠恰好夹到剩余预算时,醒来 `stalled_s()` 等于窗口而不大于,不判死。故 `_nap` 夹到 `剩余预算 + poll_interval_s`,一次到位;最坏墙钟精确表述为 `stall_window_s + 一个 poll 间隔`,不是"恰好 stall_window_s" |
|
||||
| 备忘的跨进程滞后 | 本进程记了 OPEN 冷却后,即便别的进程的探针已把共享门关回 CLOSED,本进程仍会跳到本地备忘自然过期(`_pick_runnable` 先查备忘再问门)。这是备忘"以本地记录换 Redis 往返"的固有代价,误差有界(≤ 一个 cooldown),**既有性质、本次不改**;备忘是进程内存,无持久化,故不存在滚动升级残留 |
|
||||
| 无限等待 | `_stalled` 是双条件合取,同 scope 其他调用仍在出餐时本调用不判死(ARCH §7.3 已承认的残余性质)。单源全开路时无人出餐,条件 B 必然成立,会判死;多源部分开路则走不到这个分支。文档沿用既有措辞:需要硬上限的调用方自行 `asyncio.wait_for` |
|
||||
| 未解决 | `force_open` 在 wait 档下把坏密钥的失败从毫秒拖长(上限 stall 窗口)。**有意不特判**——库无法区分"密钥坏了"与"中转抖了",选 `wait` 即声明"宁可等也不当场死" |
|
||||
| 两个预算并行(整分支审查发现,2026-08-20) | `wait` **不豁免重试预算**: 冷却结束后放行的探针是一次真实尝试,失败照样烧一格 `max_attempts`(issue #8 的划分依据是"谁消耗重试预算",探针发出了真实请求,理应记在重试预算上)。故 force_open 的源常以 `retry_exhausted` 而非 `stalled` 结束。原稿 §4 只写了 stall 一种结局,已更正;由 `test_wait_does_not_exempt_probes_from_the_retry_budget` 钉住 |
|
||||
|
||||
## 7. 文档与发布
|
||||
|
||||
ARCH §7.4 增补本次决策与三条缺陷的成因;§9 配置面登记新键;README 能力表与配置表;Gitea wiki 按 `docs-convention.md` §2 同步;CHANGELOG 记为 **1.3.0**(新增配置键 + `retry_after_s` 语义变更,后者对下游可见,需"请先读这一条"待遇)。
|
||||
|
||||
`GateDecision` 的字段与 `ProviderGate` 端口签名**均不变**,故不触碰迁移兼容约束(ARCH §5.1)。
|
||||
|
||||
## 8. 已定决策(人类,2026-08-19)
|
||||
|
||||
| # | 决策 | 随之固定的实施边界 |
|
||||
|---|---|---|
|
||||
| 1 | 缺省取 **`fail_fast`**(§3.5) | 存量下游零感知;issue 提交方需自行加 `{SCOPE}__CIRCUIT_OPEN=wait`。README/wiki 必须明写"单源 scope 建议配 wait",否则这个开关等于不存在 |
|
||||
| 2 | §3.4 的三处收敛**本次一并做** | 拆为独立前置提交,验收标准是"全套件绿 + 零新增用例";该提交即回滚点 |
|
||||
@@ -185,6 +185,11 @@
|
||||
"id": "plan:plan-issue12-telemetry-retention",
|
||||
"label": "实现计划: issue12-telemetry-retention",
|
||||
"type": "plan"
|
||||
},
|
||||
{
|
||||
"id": "review:issue14-branch-review",
|
||||
"label": "整分支审查: issue #14 熔断等待档",
|
||||
"type": "review"
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
@@ -334,6 +339,20 @@
|
||||
"relation": "implements",
|
||||
"evidence": "research-wiki/plans/2026-08-19-issue12-telemetry-retention.md",
|
||||
"added": "2026-08-19T13:10:57.986963+00:00"
|
||||
},
|
||||
{
|
||||
"source": "plan:plan-issue14-admission-wait-policy",
|
||||
"target": "design:2026-08-19-issue14-admission-wait-policy-design",
|
||||
"relation": "implements",
|
||||
"evidence": "research-wiki/plans/plan-issue14-admission-wait-policy.md;T0-T8 逐节映射设计 §3.1-§3.6",
|
||||
"added": "2026-08-20T03:30:06.280582+00:00"
|
||||
},
|
||||
{
|
||||
"source": "review:issue14-branch-review",
|
||||
"target": "plan:plan-issue14-admission-wait-policy",
|
||||
"relation": "informs",
|
||||
"evidence": "Important 项促使修正 CHANGELOG/README/设计 §4/计划 T5 对 wait 档失败 reason 的描述",
|
||||
"added": "2026-08-20T05:01:16.206639+00:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,8 +1,8 @@
|
||||
# Research Wiki 索引
|
||||
|
||||
> 自动生成，更新时间：2026-08-19 13:10 UTC
|
||||
> 自动生成，更新时间：2026-08-20 05:01 UTC
|
||||
|
||||
## design (34)
|
||||
## design (35)
|
||||
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
|
||||
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
|
||||
- [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design`
|
||||
@@ -19,6 +19,7 @@
|
||||
- [2026-08-17-issue11-caller-dimensions-design](designs/2026-08-17-issue11-caller-dimensions-design.md) `design:2026-08-17-issue11-caller-dimensions-design`
|
||||
- [2026-08-19-issue12-telemetry-retention-design](designs/2026-08-19-issue12-telemetry-retention-design.md) `design:2026-08-19-issue12-telemetry-retention-design`
|
||||
- [2026-08-19-issue13-schema-mode-design](designs/2026-08-19-issue13-schema-mode-design.md) `design:2026-08-19-issue13-schema-mode-design`
|
||||
- [2026-08-19-issue14-admission-wait-policy-design](designs/2026-08-19-issue14-admission-wait-policy-design.md) `design:2026-08-19-issue14-admission-wait-policy-design`
|
||||
- [est_tokens 解耦: 拆分限流预扣与遥测用量兜底(issue #2)](designs/est-tokens-decoupling.md) `design:est-tokens-decoupling`
|
||||
- [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2`
|
||||
- [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards`
|
||||
@@ -52,7 +53,7 @@
|
||||
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
|
||||
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
|
||||
|
||||
## plan (29)
|
||||
## plan (30)
|
||||
- [2026-07-20-m1-core-plan](plans/2026-07-20-m1-core-plan.md) `plan:2026-07-20-m1-core-plan`
|
||||
- [2026-07-20-m2-distributed-plan](plans/2026-07-20-m2-distributed-plan.md) `plan:2026-07-20-m2-distributed-plan`
|
||||
- [2026-07-21-m25-resilience-plan](plans/2026-07-21-m25-resilience-plan.md) `plan:2026-07-21-m25-resilience-plan`
|
||||
@@ -74,6 +75,7 @@
|
||||
- [M2.5 治理韧性实现计划](plans/m25-resilience.md) `plan:m25-resilience`
|
||||
- [M3 OCR 实现计划](plans/m3-ocr.md) `plan:m3-ocr`
|
||||
- [M4 迁移实现计划(T0-T14)](plans/m4-migration.md) `plan:m4-migration`
|
||||
- [plan-issue14-admission-wait-policy](plans/plan-issue14-admission-wait-policy.md) `plan:plan-issue14-admission-wait-policy`
|
||||
- [响应可观测字段扩展实现计划](plans/response-observability-fields.md) `plan:response-observability-fields`
|
||||
- [实现计划: HTTP 错误响应体留存(Issue #10)](plans/issue10-error-body-retention-plan.md) `plan:issue10-error-body-retention-plan`
|
||||
- [实现计划: issue12-telemetry-retention](plans/plan-issue12-telemetry-retention.md) `plan:plan-issue12-telemetry-retention`
|
||||
@@ -83,6 +85,9 @@
|
||||
- [调用方自定义维度实现计划(issue #11)](plans/issue11-caller-dimensions.md) `plan:issue11-caller-dimensions`
|
||||
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
|
||||
|
||||
## review (1)
|
||||
- [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review`
|
||||
|
||||
## schema (1)
|
||||
- [表结构: llm_calls(遥测 22 字段)](schemas/llm-calls.md) `schema:llm-calls`
|
||||
|
||||
|
||||
@@ -114,3 +114,12 @@
|
||||
- [2026-08-19 13:10 UTC] 新增 plan: 实现计划: issue12-telemetry-retention (plan:plan-issue12-telemetry-retention)
|
||||
- [2026-08-19 13:10 UTC] 新增边: plan:plan-issue12-telemetry-retention --implements--> design:issue12-telemetry-retention
|
||||
- [2026-08-19 13:10 UTC] 重建索引: 78 篇页面
|
||||
- [2026-08-20 03:29 UTC] 新增 design: 熔断拒绝补齐等待档(issue #14) (design:issue14-admission-wait-policy)
|
||||
- [2026-08-20 03:30 UTC] 新增 plan: 实现计划: 熔断拒绝补齐等待档(issue #14) (plan:issue14-admission-wait-policy)
|
||||
- [2026-08-20 03:30 UTC] 新增边: plan:issue14-admission-wait-policy --implements--> design:issue14-admission-wait-policy
|
||||
- [2026-08-20 03:30 UTC] 重建索引: 82 篇页面
|
||||
- [2026-08-20 03:30 UTC] 重建索引: 80 篇页面
|
||||
- [2026-08-20 05:01 UTC] 新增边: review:issue14-branch-review --informs--> plan:plan-issue14-admission-wait-policy
|
||||
- [2026-08-20 05:01 UTC] 重建索引: 80 篇页面
|
||||
- [2026-08-20 05:01 UTC] 新增 review: 整分支审查: issue #14 熔断等待档 (review:issue14-branch-review)
|
||||
- [2026-08-20 05:01 UTC] 重建索引: 81 篇页面
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
# 实现计划: 熔断拒绝补齐等待档(issue #14)
|
||||
|
||||
- **设计**: `research-wiki/designs/2026-08-19-issue14-admission-wait-policy-design.md`(人类已确认 + Codex 已审)
|
||||
- **分支**: `feat/issue-14-circuit-open-policy`
|
||||
- **版本**: 1.3.0(新增配置键 + `retry_after_s` 语义变更)
|
||||
|
||||
## 目标
|
||||
|
||||
让"源不健康"不再等同于"这次调用当场判死"——补上 `{SCOPE}__CIRCUIT_OPEN=fail_fast|wait` 这一格准入策略,并把 `retry_after_s` 的语义在两个后端的五个出口上定死。
|
||||
|
||||
## 方案概述
|
||||
|
||||
三件事环环相扣: ①把 `retry_after_s` 定义为"距离**确定**可再试的时刻还有多久",HALF_OPEN 与准入允许一律 `0.0`(顺带修掉源冷却备忘被探针租约污染的 bug);②新增 `circuit_open` 策略键,`wait` 档下不抛 `CircuitOpenError` 而按 `retry_after` 睡、由 stall 预算兜底;③前置把三条治理循环里逐字复制的准入逻辑收敛成一份,否则本次修复会在 embedding/ocr 留下两个行为分叉的角落。
|
||||
|
||||
涉及技术: Python 3.11 asyncio、Redis Lua(EVALSHA)、pytest 双后端参数化契约测试。
|
||||
|
||||
## 保真校验适用性
|
||||
|
||||
**适用**。熔断状态机是 ARCHITECTURE.md §1.4 关键资产(蓝本 `reference/Video-Tree-TRM5/adapters/breaker.py` 与 `reference/CHSAnalyzer/app/coordination/provider_gate.py`),准入循环蓝本为 `reference/CHSAnalyzer/app/providers/governance.py:107-285`。T1 与 T2/T3 各带保真校验检查点。
|
||||
|
||||
## 文件结构
|
||||
|
||||
| 文件 | 动作 | 职责 |
|
||||
|---|---|---|
|
||||
| `src/polygateway/middleware/admission.py` | **新建** | `SourceAdmission`(准入与无源可跑的处置,三条循环共用)+ 模块级 `settle_and_release` |
|
||||
| `src/polygateway/middleware/retry.py` | 修改 | 删除本地 `_pick_runnable`/`_on_no_runnable`/`_settle_and_release`,改用 `SourceAdmission`;主循环与 `_attempt` 不动 |
|
||||
| `src/polygateway/embedding.py` | 修改 | 同上 |
|
||||
| `src/polygateway/ocr.py` | 修改 | 同上(注意 `_settle_and_release` 原签名只有 `permit`) |
|
||||
| `src/polygateway/backends/memory/breaker.py` | 修改 | 抽 `_remaining(g)`,三处出口共用;HALF_OPEN 与授予探针恒 `0.0` |
|
||||
| `src/polygateway/backends/redis/breaker.py` | 修改 | 五个 Lua 出口同步(`TRY_ENTER` 两处、`RECORD_SUCCESS`/`RECORD_FAILURE`/`RELEASE_PROBE` 各一处、`RETRY_AFTER` 一处) |
|
||||
| `src/polygateway/config.py` | 修改 | `_CIRCUIT_OPEN` 常量、`GatewaySettings.circuit_open` 字段、`_validate_backends` 元组、`from_env` 装载 |
|
||||
| `src/polygateway/client.py` | 修改 | 构造签名 + 透传 |
|
||||
| `src/polygateway/errors.py` | 修改 | `GatewayUnavailableError` docstring 职责边界 |
|
||||
| `tests/contracts/test_breaker_contract.py` | 修改 | 按五个出口逐个钉 `retry_after_s` |
|
||||
| `tests/integration/test_redis_governance_time.py` | 修改 | Redis 真实等待变体补 HALF_OPEN 出口 |
|
||||
| `tests/unit/test_backpressure.py` | 修改 | `circuit_open` 行为矩阵、备忘污染回归、`_nap` 上界 |
|
||||
| `tests/unit/test_config.py` | 修改 | 新键的合法域、缺省、两条装配路一致 |
|
||||
|
||||
## 关键接口(跨任务消费,此处定死)
|
||||
|
||||
`SourceAdmission` 构造与两个方法:
|
||||
|
||||
```python
|
||||
class SourceAdmission:
|
||||
def __init__(self, *, scope: str, sources: list[SourceConfig],
|
||||
selector: SourceSelector, quota: QuotaGate, breaker: BreakerGate,
|
||||
memo: SourceCooldownMemo, backpressure: BackpressurePolicy,
|
||||
quota_full: str, circuit_open: str,
|
||||
pacer: AdaptivePacer | None = None,
|
||||
health_view: Callable[[str], float] | None = None,
|
||||
now=time.monotonic, sleep=asyncio.sleep, rng=random.random) -> None: ...
|
||||
|
||||
async def pick(self, reasons: dict[str, str], attempt_fails: dict[str, int]
|
||||
) -> tuple[tuple[SourceConfig, Permit, GateDecision] | None, int]: ...
|
||||
|
||||
async def on_no_runnable(self, gate_rejections: int, reasons: dict[str, str],
|
||||
clock: StallClock) -> None: ...
|
||||
|
||||
async def stalled(self, clock: StallClock) -> bool: ...
|
||||
```
|
||||
|
||||
`quota`/`breaker`/`pacer`/`selector`/`sources` 均为**调用方传入的同一实例**(不在 admission 内新建),因为三处 `_attempt` 仍需引用它们;`memo` 则由 admission 独占。`health_view` 对应 chat 的 `self._health_view`(由 `isinstance(selector, OutcomeAwareSelector)` 在 RetryMW 构造期判定一次),embedding/ocr 传 `None`。
|
||||
|
||||
模块级结算函数(三处 `_attempt` 的 finally 与 admission 共用):
|
||||
|
||||
```python
|
||||
async def settle_and_release(permit: Permit, actual: int) -> None:
|
||||
"""finally 专用: settle 后必 release;失败降级 warning,绝不掩盖主异常/取消。"""
|
||||
```
|
||||
|
||||
睡眠时长(T5 实现,写死在 `SourceAdmission._nap`):
|
||||
|
||||
```python
|
||||
def _nap(self, hint: float, clock: StallClock) -> float:
|
||||
jitter = self._bp.poll_interval_s * (0.5 + 0.5 * self._rng())
|
||||
budget = self._bp.stall_window_s - clock.stalled_s() + self._bp.poll_interval_s
|
||||
wait = hint + jitter if hint > 0 else jitter
|
||||
return max(jitter, min(wait, budget))
|
||||
```
|
||||
|
||||
`hint == 0` 时该式退化为 `jitter`,即现有 quota-wait 行为逐字不变(`tests/unit/test_backpressure.py` 已钉 `[0.5p, 1.0p]`)。**下界取 `jitter` 而非 `poll_interval_s`(实施期修正)**: 后者会把 `rng → 0` 那半边从 `0.5p` 抬到 `1.0p`,既有的 `test_poll_jitter_bounds` 当场变红;`jitter` 同样能在预算为负时兜住不返回负数、不忙循环。`budget` 加一个 `poll_interval_s` 是因为 `_stalled` 判据是 `>` 而非 `>=`(`retry.py:368`),恰好夹到窗口不会判死。
|
||||
|
||||
**调用约束**: `_nap` 必须在 `stalled()` 判定**之后**调用。若已 stall 超窗才进来,`budget` 为负,外层 `max(poll_interval_s, ...)` 会兜成一个 poll 间隔(不会返回负数),但那意味着本该判死却又睡了一轮——顺序由 `on_no_runnable` 保证(两条路汇合后统一判 `stalled()` 再 sleep)。验算示例: `hint=60, stall_window=300, 已 stall 290, poll=0.05` → `jitter∈[0.025,0.05]`、`budget=10.05` → 返回 `10.05`,醒来累计约 `300.05` > 300,下一轮判死。
|
||||
|
||||
## 任务清单
|
||||
|
||||
### T0 — 分支与基线
|
||||
|
||||
- [ ] 建分支 `feat/issue-14-circuit-open-policy`(从 main)
|
||||
- [ ] 记录基线: `conda run -n PolyGateway python -m pytest tests/ -q` 与 `make check` + `lint-imports` 全绿,记下**本机本环境**的用例计数(执行时实测,2026-08-19 为 988 passed / 32 deselected)。该数只作同环境参照——`addopts = "-m 'not slow'"` 与 Redis 可达性都会改变它,不作硬验收
|
||||
|
||||
**验证**: `conda run -n PolyGateway python -m pytest tests/ -q` → 全 PASS;`git rev-parse --abbrev-ref HEAD` → 分支名正确
|
||||
|
||||
---
|
||||
|
||||
### T1 — 纯重构: 准入逻辑三处收敛(回滚点)
|
||||
|
||||
**动**: 新建 `src/polygateway/middleware/admission.py`;改 `middleware/retry.py`、`embedding.py`、`ocr.py`。
|
||||
|
||||
**要实现的行为**: 把 `_pick_runnable`/`_on_no_runnable`/`_stalled`/`_settle_and_release` 从三处搬进 `SourceAdmission` 与模块级 `settle_and_release`,三条循环改为持有 `SourceAdmission` 实例并调用其方法。**本任务不引入 `circuit_open` 参数**(构造签名先只收 `quota_full`,T4 再加),控制流一字不改。
|
||||
|
||||
三条循环的差异只用注入表达,不留 `if` 分支:
|
||||
|
||||
| 差异 | 处理 | 等价性依据 |
|
||||
|---|---|---|
|
||||
| 调用内降权(仅 chat) | `attempt_fails` 作 `pick()` 入参,内部无条件调 `_demote_call_failures` | 传空 dict 时 `demoted` 为空 → `return ordered` 原对象返回,恒等(`retry.py:148-150`) |
|
||||
| AIMD pacer(仅 chat) | `pacer: AdaptivePacer \| None = None` | None 时跳过 `admit()` 与 `enter()` 两个调用点,无副作用 |
|
||||
| `_settle_and_release` 签名 | OCR 原为 `(permit)`、体内恒 `settle(0)`;改为调 `settle_and_release(permit, 0)` | 逐字等价 |
|
||||
| warning 文案**三处都不同** | 归一为 "permit 结算/释放失败(不掩盖主异常)" | chat `retry.py:536` 已是该文案;embedding `embedding.py:411` 为 "embedding permit …"、OCR `ocr.py:448` 为 "OCR permit …" 将被归一(Codex 审查补,原稿只承认了 OCR)。这是本任务**唯一**的可见行为变化,须在提交信息里点名 |
|
||||
| `_stalled` 形态 | chat 已抽成方法,embedding/ocr 为内联表达式 | 两者语义逐字相同(已 diff 核实),统一用 `SourceAdmission.stalled()` |
|
||||
|
||||
**搬走 vs 共享(自审修正,这一条决定 T1 能否成立)**: 三处 `_attempt` 仍在引用 `self._breaker`(记账写回)、`self._quota`(mark_progress)、`self._pacer`(leave)、OCR 还有 `self._selector`(健康喂数,`ocr.py:426`)。因此这些字段**不搬走,而是共享同一实例**——循环保留自己的引用,构造 `SourceAdmission` 时把同一对象传进去(`AdaptivePacer` 有在途计数状态,必须是同一实例而非新建,否则 `admit`/`enter` 与 `leave` 分裂到两个计数器上)。真正搬走的只有 `_pick_runnable`/`_on_no_runnable`/`_stalled` 三个方法与 `self._memo`(仅被 `pick` 消费)。
|
||||
|
||||
**`_attempt` 的唯一改动**: `self._settle_and_release(permit, actual)` → 模块级 `settle_and_release(permit, actual)`,OCR 侧由 `(permit)` 变为 `(permit, 0)`。除此之外 `_attempt` 一行不动。原稿"三处 `_attempt` 本体不在边界内"的说法与"搬走 `_settle_and_release`"自相矛盾,此处更正。
|
||||
|
||||
**不在边界内、须原样保留**: chat 主循环顶部那次额外的 `_stalled` 预判(`retry.py:286`)、OCR 的 `_gate_on_terminal`(`ocr.py:412`)与健康喂数。
|
||||
|
||||
**保真校验检查点**: 对照 `reference/CHSAnalyzer/app/providers/governance.py:107-285`,确认搬运后 `_pick_runnable` 的候选跳过顺序(备忘 → pacer → 配额 → 熔断门)、`gate_rejections` 的计入规则(备忘与熔断门计入,pacer 与配额不计入)、`_on_no_runnable` 的三段判定顺序逐段未变。
|
||||
|
||||
**测试要求(本任务特殊)**: **不新增行为用例**。全套件绿是必要条件而非充分条件——它证明不了"逐字不变",故本任务额外要求一次**机械差异审查**: 把搬迁前后的 `pick`/`on_no_runnable` 逐语句对照,确认候选跳过顺序、`gate_rejections` 计入规则、`reasons` 的 `[]=` 与 `setdefault` 用法(两者语义不同,不可互换)一字未变。
|
||||
|
||||
**已知会碰到的既有测试**: `tests/unit/test_health_selector.py:146` 断言 `client._terminal._pacer._ceiling`,`tests/unit/test_client.py:380` 断言 `._terminal._emitter._text_cap`——这两个字段必须留在 `RetryMW` 上(与上面"共享而非搬走"一致),否则这些用例会红。
|
||||
|
||||
**验证**:
|
||||
```bash
|
||||
conda run -n PolyGateway python -m pytest tests/ -q # 期望: 全 PASS,计数与 T0 同环境基线一致
|
||||
conda run -n PolyGateway make check # 只读: ruff format --check + ruff check
|
||||
conda run -n PolyGateway lint-imports # 依赖铁律
|
||||
```
|
||||
**不要用 `make lint` 做验证**——它带 `--fix` 会自动改文件(`Makefile:11`),只读验证用 `make check` + `lint-imports`。用例计数只作**同环境**参照,不作硬验收: `pytest` 默认 `-m 'not slow'`(`pyproject.toml:51`),且无 `REDIS_URL` 时 Redis 用例 skip,计数随环境浮动。
|
||||
|
||||
import-linter 层级(`pyproject.toml:76`)允许 `middleware/admission.py` 依赖 `ports`/`types`/`errors`/`sources`(更内层),但不得 import 任何 `backends/`、`transports/`、`telemetry/`。搬迁后须清理三个原文件中失去引用的 import(`CircuitOpenError`、`QuotaGate`、`BreakerGate`、`SourceCooldownMemo` 等),否则 ruff 报未使用导入。
|
||||
|
||||
- [ ] 提交: `refactor: 把三条治理循环的准入逻辑收敛为 SourceAdmission`
|
||||
|
||||
---
|
||||
|
||||
### T2 — `retry_after_s` 语义统一(两个后端一次到位)
|
||||
|
||||
**动**: `src/polygateway/backends/memory/breaker.py`、`src/polygateway/backends/redis/breaker.py`、`tests/contracts/test_breaker_contract.py`、`tests/integration/test_redis_governance_time.py`。
|
||||
|
||||
**为什么两个后端必须同一个提交(Codex 审查修正)**: 原稿把 memory 与 redis 拆成 T2/T3 两次提交,中间 redis 侧契约用例会处于 red。但 `.claude/settings.json` 注册的 `pre-commit-guard.sh` 在检测到 `git commit` 时会跑 `pytest tests/ --tb=line -q`(`pre-commit-guard.sh:61`),红态直接卡住提交。且两者本就是**同一个契约的两个实现**,分开提交没有独立意义。
|
||||
|
||||
**要实现的行为**: `retry_after_s` = "距离**确定**可再试的时刻还有多久"。HALF_OPEN 下探针随时可能出结果,不存在确定时刻,故 `0.0`;准入被允许时同样恒 `0.0`。`0 = 可立即重试` 是库既有约定(`errors.py` 与现有契约用例"健康 → 0、冷却到期 → 0")。
|
||||
|
||||
memory 侧: 抽私有纯方法 `_remaining(g: _SourceGate) -> float`(OPEN 返回 `max(0.0, g.open_until - now)`,其余状态含 HALF_OPEN 返回 `0.0`),`try_enter` 的 HALF_OPEN 拒绝分支(`memory:148`)与 `retry_after_s()`(`memory:267`)改用它。`_snapshot`(`memory:169`)与授予探针(`memory:114`)已符合新契约,保持不变。
|
||||
|
||||
redis 侧共**六个返回格**,逐处点名(改前先确认行号仍对得上):
|
||||
|
||||
| 脚本 | 位置 | 现状 | 改为 |
|
||||
|---|---|---|---|
|
||||
| `TRY_ENTER` HALF_OPEN 拒绝 | `redis:44` | `probe_until - now` | `0` |
|
||||
| `TRY_ENTER` 授予探针 | `redis:53` | `tonumber(ARGV[2])`(= probe TTL) | `0` |
|
||||
| `RECORD_SUCCESS` fencing 未命中 | `redis:124` | half_open 取 `probe_until` | half_open 记 `0`(只 OPEN 取 `open_until - now`) |
|
||||
| `RECORD_FAILURE` fencing 未命中 | `redis:155` | 同上 | 同上 |
|
||||
| `RELEASE_PROBE` fencing 未命中 | `redis:255` | 同上 | 同上 |
|
||||
| `RETRY_AFTER` | `redis:275` | half_open 取 `probe_until` | half_open 记 `0` |
|
||||
|
||||
后四行修的是**既有的双后端语义分叉**(memory `_snapshot` 对非 OPEN 一律 `0.0`),与本 issue 同源,由契约测试盲区掩护至今——现有用例只钉"第二个进入者被拒",没钉它拿到什么数。
|
||||
|
||||
**保真校验检查点**: 状态机转换、双通道开路判据、`_cooldown_eff` 指数退避、epoch fencing 匹配条件、Lua 的原子性结构与 `redis.call('TIME')` 服务器时钟口径**一律不动**——本任务只改"对外报几"这一件事,即 return 元组里 `retry_after_ms` 那一格。改完逐脚本与 memory 实现对照走一遍状态机。
|
||||
|
||||
**测试要求**(先失败后通过,`tests/contracts/` 双后端参数化,一次覆盖 memory + redis):
|
||||
- HALF_OPEN 被拒: `decision.retry_after_s == 0.0` 且 `decision.state is GateState.HALF_OPEN`
|
||||
- 授予探针的决定: `retry_after_s == 0.0`
|
||||
- `record_*` 在 fencing 未命中且门处于 HALF_OPEN: `GateUpdate.retry_after_s == 0.0`(须同时断言 `applied is False`、`state is HALF_OPEN`,否则用例可能在别的分支上误绿)
|
||||
- `gate.retry_after_s(("s1",))` 探针在途时返回 `0.0`
|
||||
- 现有 `test_retry_after_semantics` / `test_retry_after_takes_min_across_sources` 保持绿(OPEN 语义未变)
|
||||
|
||||
**Redis 时间语义变体**: 契约层用 `clock.advance()` 的用例在 redis 参数下会 skip(`conftest.py:39` 的 `SkipClock` 哨兵),故须在 `tests/integration/test_redis_governance_time.py` 补 1:1 真实等待变体(既有约定: 不缩放时长)。该文件的 `test_meta_variants_cover_all_time_cases`(`:56`)会**机械拦截**漏配,漏了就红。
|
||||
|
||||
**验证**:
|
||||
```bash
|
||||
conda run -n PolyGateway python -m pytest tests/contracts/test_breaker_contract.py -q # 双后端全 PASS
|
||||
conda run -n PolyGateway python -m pytest tests/integration/test_redis_governance_time.py -m slow -q
|
||||
```
|
||||
第二条**必须带 `-m slow`**: `pyproject.toml:51` 的 `addopts = "-m 'not slow'"` 默认排除真实等待变体,不加就是空跑(该文件单跑 12-15 分钟)。需真实 Redis(db3),不 mock Lua 行为。
|
||||
|
||||
- [ ] 提交: `fix: 把 retry_after_s 定义为确定可再试时刻,HALF_OPEN 归零(双后端)`
|
||||
|
||||
---
|
||||
|
||||
### T3 — (已并入 T2)
|
||||
|
||||
原计划把 redis 侧拆为独立任务,因 pre-commit hook 会拦截中间红态而合并进 T2。此编号保留以免后续引用错位。
|
||||
|
||||
---
|
||||
|
||||
### T4 — 新配置键 `{SCOPE}__CIRCUIT_OPEN`
|
||||
|
||||
**动**: `src/polygateway/config.py`、`src/polygateway/client.py`、`src/polygateway/middleware/admission.py`、`embedding.py`、`ocr.py`、`tests/unit/test_config.py`。
|
||||
|
||||
**要实现的行为**: 与 `quota_full` 逐项同构,不发明新形状。
|
||||
|
||||
| 位置 | 改动 |
|
||||
|---|---|
|
||||
| `config.py` 常量区 | `_CIRCUIT_OPEN = frozenset({"wait", "fail_fast"})`,紧邻 `_QUOTA_FULL` |
|
||||
| `GatewaySettings` | 新增字段 `circuit_open: str`,**无默认值**(与该类全部既有字段一致),位置紧随 `quota_full` |
|
||||
| `_validate_backends` | 校验元组加一行 `("circuit_open", _CIRCUIT_OPEN)` |
|
||||
| `from_env` | `circuit_open=_load_choice(env, f"{scope_u}__CIRCUIT_OPEN", _CIRCUIT_OPEN, "fail_fast")` |
|
||||
| `client.py` | `GatewayClient.__init__` 加 `circuit_open: str = "fail_fast"`;`from_settings` 透传 `settings.circuit_open` |
|
||||
| `admission.py` | 构造收 `circuit_open`,同 `quota_full` 做构造期域校验并抛 `ValueError` |
|
||||
| `embedding.py` / `ocr.py` | 两个客户端的构造签名与"从 GatewayClient 派生"路径(`embedding.py:561`、`ocr.py:574` 邻域)各透传一处 |
|
||||
|
||||
**缺省取 `fail_fast`**(人类 2026-08-19 决策): 保证控制流对存量下游不变。
|
||||
|
||||
**测试要求**(先失败后通过):
|
||||
- 缺省档: 不设该键时 `settings.circuit_open == "fail_fast"`
|
||||
- 合法域: 设为 `"nope"` 时 `from_env` 与直接构造**两条路**都抛 `ValueError` 且消息点出键名/字段名
|
||||
- 两条装配路一致: `from_env` 与直接构造同一取值产出同一行为
|
||||
- `dataclasses.replace(settings, circuit_open="wait")` 仍通过全部装配守卫
|
||||
- 透传链: 从 `GatewaySettings` 一路到三条循环的 `SourceAdmission` 实例上取值正确
|
||||
|
||||
**验证**:
|
||||
```bash
|
||||
conda run -n PolyGateway python -m pytest tests/unit/test_config.py tests/unit/test_client.py -q
|
||||
```
|
||||
|
||||
- [ ] 提交: `feat: 新增 {SCOPE}__CIRCUIT_OPEN 策略键(缺省 fail_fast)`
|
||||
|
||||
---
|
||||
|
||||
### T5 — `on_no_runnable` 按原因分派 + `_nap`
|
||||
|
||||
**动**: `src/polygateway/middleware/admission.py`、`tests/unit/test_backpressure.py`。
|
||||
|
||||
**要实现的行为**: 把现状串行的两个分支改为按拒绝原因分派(伪码见设计 §3.3)。要点:
|
||||
|
||||
1. `gate_rejections == len(sources)`(全部因熔断类原因被拒)时,`fail_fast` 抛 `CircuitOpenError`(现行为),`wait` 取 `hint = await breaker.retry_after_s(names)` 后**不抛**;
|
||||
2. 否则(至少一源是被配额/AIMD 挡的)走 `quota_full` 分支,`hint = 0.0`;
|
||||
3. 两条路汇合后统一判 `stalled()`,再 `await sleep(self._nap(hint, clock))`。
|
||||
|
||||
**必须避免的坑**: 若只把第一分支改成"wait 时不抛"而不做分派,控制流会掉进 `quota_full` 分支——`quota_full=fail_fast` 的调用方会看到熔断等待被误报成 `reason="quota_exhausted"`。
|
||||
|
||||
**可观测性**: `wait` 档每轮进入等待时 `logger.info` 一条(scope、`per_source_reasons`、本次睡眠秒数)。**只此一条,不打"醒来"那条**(实施期决定): 每一轮等待各自留痕,时间线已可完整还原,而醒来后若仍被拒会立刻打下一条——补一条"醒来"只会让日志量翻倍且信息重复。**不新增遥测列**(等待期不发请求,无 attempt 行可记;调用级总耗时下游可自测)。
|
||||
|
||||
**计时归属**: 睡眠发生在 `clock.attempting()` 之外,自动计入 stall 账,与 ARCH §7.3"熔断冷却属非生产性等待"一致——**无需改 `StallClock`**。
|
||||
|
||||
**取消穿透**: `_nap` 只做算术,睡眠是裸 `await self._sleep(...)`,不得包 `try/except`。
|
||||
|
||||
**测试要求**(先失败后通过,注入时钟/睡眠/rng 保持确定性):
|
||||
- `circuit_open=wait` + 全源开路 → **不**抛 `CircuitOpenError`,而是按 `retry_after` 睡;冷却结束后拿到探针并成功返回
|
||||
- `circuit_open=wait` + `quota_full=fail_fast` + 全源开路 → **不**抛 `quota_exhausted`(这是上面那个坑的钉子)
|
||||
- `circuit_open=wait` + 冷却比 stall 预算还长 → 抛 `AllSourcesExhausted(reason="stalled")`,`per_source_reasons` 含 `circuit_open`,累计墙钟 ≤ `stall_window_s + poll_interval_s`
|
||||
- `circuit_open=wait` + 源持续 `force_open` → **`retry_exhausted` 而非 `stalled`**(整分支审查发现,原稿写错): 冷却结束后放行的探针是真实尝试,失败照样烧一格 `max_attempts`,故两个预算里先耗尽的那个决定 reason
|
||||
- 混合原因(部分 `circuit_open` + 部分 `rate_limited`)→ 走 quota 分支,`per_source_reasons` 如实混合
|
||||
- `hint == 0` 时睡眠落在 `[0.5p, 1.0p]`(现有 quota-wait 行为逐字不变)
|
||||
- `wait` 档等待中收到 `CancelledError` → 逐字穿透,in-flight permit 已释放
|
||||
- `circuit_open=fail_fast`(缺省)下,全部现有用例逐字绿
|
||||
- **备忘污染回归**(issue #14 §1.3): 探针成功后 `memo.active(源名)` 为 False,该源立即重新可选——此用例由 `/tmp/.../probe_repro.py` 的复现脚本转化而来,在 T2 之前必然 red
|
||||
|
||||
**验证**:
|
||||
```bash
|
||||
conda run -n PolyGateway python -m pytest tests/unit/test_backpressure.py tests/unit/test_retry.py -q
|
||||
conda run -n PolyGateway python -m pytest tests/ -q # 全套件
|
||||
```
|
||||
|
||||
- [ ] 提交: `feat: circuit_open=wait 下熔断拒绝改为等待而非当场判死`
|
||||
|
||||
---
|
||||
|
||||
### T6 — `errors.py` 职责边界补写
|
||||
|
||||
**动**: `src/polygateway/errors.py`。
|
||||
|
||||
**要实现的行为**: 改写 `GatewayUnavailableError` 的 docstring。现文"业务侧 catch 本类做延期重投(CHS arq 模式)"读起来像鼓励每个下游各写一份重试逻辑;改为明确边界——调用级的重试/退避/换源/等待全部在库内,本异常表示库的调用级预算(重试预算或 stall 预算)已耗尽;下游若要再投,那是**任务级重试**,语义与调用级重试不同(ARCH §7.2 单层重试原则)。
|
||||
|
||||
`retry_after_s` 那句保留并补一句: 它是"距离确定可再试的时刻",`0` 表示无确定等待(可立即重试)。
|
||||
|
||||
**测试要求**: 纯 docstring,无行为变更。验收为 `tests/unit/test_errors.py` 保持绿。
|
||||
|
||||
**验证**: `conda run -n PolyGateway python -m pytest tests/unit/test_errors.py -q`
|
||||
|
||||
- [ ] 提交: `docs: 收回 GatewayUnavailableError 的重试职责边界`
|
||||
|
||||
---
|
||||
|
||||
### T7 — 文档同步
|
||||
|
||||
**动**: `research-wiki/ARCHITECTURE.md`、`README.md`、`CHANGELOG.md`、Gitea wiki。
|
||||
|
||||
| 目标 | 内容 |
|
||||
|---|---|
|
||||
| ARCH §7.4 | 增补本次决策: 三条缺陷的成因、`retry_after_s` 的契约定义(五个出口)、`circuit_open` 策略键与缺省理由 |
|
||||
| ARCH §9 配置面 | 登记 `{SCOPE}__CIRCUIT_OPEN` |
|
||||
| README | 配置表新增该键;**明写"单源 scope 建议配 `wait`"**——缺了这句,这个开关等于不存在;核对安装命令的版本约束是否需要跟着改 |
|
||||
| CHANGELOG | 记 1.3.0,`retry_after_s` 语义变更给"请先读这一条"待遇(缺省档下 `CircuitOpenError.retry_after_s` 在全源 HALF_OPEN 时由探针租约剩余变为 0) |
|
||||
| Gitea wiki | 按 `research-wiki/docs-convention.md` §2 清单同步 |
|
||||
|
||||
**验证**: 人工逐项核对上表;`grep -n "CIRCUIT_OPEN" README.md research-wiki/ARCHITECTURE.md` 各有命中。
|
||||
|
||||
- [ ] 提交: `docs: 记录熔断等待档与 retry_after_s 契约`
|
||||
|
||||
---
|
||||
|
||||
### T8 — 合并前独立验证
|
||||
|
||||
- [ ] 派**全新上下文** verifier subagent(`verification-before-completion`),逐条核对: 设计每一节是否有对应实现、五个 `retry_after_s` 出口是否都改到、三条循环行为是否一致、测试证据是否都是"先失败后通过"
|
||||
- [ ] `conda run -n PolyGateway make check` + `conda run -n PolyGateway lint-imports` 全绿(**不用 `make lint`**,它带 `--fix` 会改文件)
|
||||
- [ ] `conda run -n PolyGateway make test` 全套件绿 + 覆盖率 ≥ 80%
|
||||
- [ ] Redis integration 套件在真实 Redis 上绿,含 `-m slow` 的时间语义变体(默认 addopts 会排除它)
|
||||
- [ ] `requesting-code-review` 走一次整分支审查
|
||||
- [ ] `finishing-a-development-branch`: `--no-ff` 合并 main,合并后在 main 上重跑 lint 与全套件
|
||||
|
||||
**注**: 发布(tag/构建/上传 registry/建 Release)按 CLAUDE.md §4.4.1 九步走,**不在本计划范围**,需人类确认后单独执行。
|
||||
|
||||
## 自审记录
|
||||
|
||||
- 设计每一节到任务的映射: §3.1→T2+T3、§3.2→T4、§3.3→T5、§3.4→T1、§3.5→T4(缺省值)+T7(文档)、§3.6→T6、§4 行为矩阵→T5 测试、§5 测试策略→T2/T3/T5、§6 非功能→T5(取消/计时/上界)
|
||||
- 无 TBD/TODO/"适当的错误处理"类占位
|
||||
- 跨任务消费的 `SourceAdmission` 签名、`settle_and_release`、`_nap` 公式已在"关键接口"写出实际代码
|
||||
- 任务顺序有硬依赖: T1(收敛)必须先于 T5(在单一位置加语义)。原 T2/T3 拆分已合并——pre-commit hook 跑全套件,任何跨提交的红态都会被拦
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
type: review
|
||||
node_id: review:issue14-branch-review
|
||||
title: "整分支审查: issue #14 熔断等待档"
|
||||
date: 2026-08-20
|
||||
---
|
||||
|
||||
# 整分支审查: issue #14 熔断等待档
|
||||
|
||||
|
||||
- **范围**: `feat/issue-14-circuit-open-policy`,296c765..5a025b6(8 提交,src 6 文件 + tests 5 文件)
|
||||
- **审查方**: Codex 全新上下文只读审查(两轮: 独立验收 + 整分支审查)
|
||||
- **结论**: **needs_changes → 修正后 approved**;Critical 0 项
|
||||
|
||||
## 发现与处置
|
||||
|
||||
| 级别 | 发现 | 核实 | 处置 |
|
||||
|---|---|---|---|
|
||||
| Important | `circuit_open=wait` + 持续 `force_open` 实际抛 `retry_exhausted` 而非文档声称的 `stalled` | **成立**。冷却结束后放行的探针是真实尝试,失败照样烧一格 `max_attempts`;审查方以单源 + 连续 `SourceDeadError("401")` 复现,本地补测试复现一致 | **改文档不改代码**——该行为符合 issue #8 确立的"划分依据是谁消耗重试预算"。修正 CHANGELOG / README / 设计 §4 行为矩阵 / 计划 T5,并补 `test_wait_does_not_exempt_probes_from_the_retry_budget` 钉死 |
|
||||
| Minor | 计划要求进入/退出等待各一条日志,实现只有进入那条 | 成立 | **保持一条**,修计划措辞: 每轮等待各自留痕已可还原时间线,醒来后若仍被拒会立刻打下一条,补"醒来"只会让日志量翻倍 |
|
||||
| — | 上一轮独立验收挑出计划 `_nap` 伪码下界与实现不一致(`poll_interval_s` vs `jitter`) | 成立 | 实现是对的(用 `poll_interval_s` 会把既有 quota 轮询的 `rng→0` 半边从 `0.5p` 抬到 `1.0p`),已回填计划 |
|
||||
|
||||
审查方两轮均确认: T1 收敛行为等价、六个 `retry_after_s` 出口齐备、备忘污染闭合、取消穿透与 permit/pacer 配对无泄漏、缺省档控制流不变。
|
||||
|
||||
## 验证证据(本会话工具输出)
|
||||
|
||||
- 全套件 `pytest tests/ -q`: **980 passed, 25 skipped, 36 deselected**(基线 967 passed;+13 为新增用例)
|
||||
- 覆盖率 `make test`: 总 **94%**(`admission.py` 93%、`config.py` 99%、`memory/breaker.py` 96%)
|
||||
- Redis 时间语义全变体 `-m slow`: **18 passed in 1151s**(19 分 11 秒,真实等待不缩放),含本次新增 4 个
|
||||
- `make check` 与 `lint-imports`: 全绿,**Contracts: 1 kept, 0 broken**
|
||||
@@ -33,7 +33,7 @@ from polygateway.types import (
|
||||
SourceConfig,
|
||||
)
|
||||
|
||||
__version__ = "1.2.3"
|
||||
__version__ = "1.2.4"
|
||||
|
||||
__all__ = [
|
||||
"DEFAULT_PROFILES",
|
||||
|
||||
@@ -100,6 +100,22 @@ class InMemoryGate:
|
||||
streak = max(1, g.reopen_streak)
|
||||
return min(self._cfg.cooldown_s * (2 ** (streak - 1)), self._cfg.max_cooldown_s)
|
||||
|
||||
def _remaining(self, g: _SourceGate) -> float:
|
||||
"""距离**确定**可再试的时刻还有多久(issue #14 的契约定义)。
|
||||
|
||||
OPEN 的冷却截止是确定时刻;HALF_OPEN 下探针随时可能出结果,**不存在**
|
||||
确定时刻,故 `0.0`——`0 = 可立即重试` 是库既有约定。此前这里返回探针
|
||||
租约剩余,而租约长度是死锁保护参数(派生自 `2 × 最慢源 timeout`),与
|
||||
"源多久能恢复"无因果关系;它还被喂进源冷却备忘,而备忘 `set_until`
|
||||
取更晚者不可回退,于是门恢复 CLOSED 后本进程仍跳过该源整整一个租约。
|
||||
|
||||
三个出口(`try_enter` 拒绝、`_snapshot`、`retry_after_s`)共用本方法,
|
||||
避免同一语义在三处各算一遍而漂移。
|
||||
"""
|
||||
if g.state is GateState.OPEN:
|
||||
return max(0.0, g.open_until - self._now())
|
||||
return 0.0
|
||||
|
||||
def _grant_probe(self, g: _SourceGate, source_name: str, owner: str) -> GateDecision:
|
||||
g.state = GateState.HALF_OPEN
|
||||
g.probe_owner = owner
|
||||
@@ -140,7 +156,7 @@ class InMemoryGate:
|
||||
epoch=g.epoch,
|
||||
is_probe=False,
|
||||
probe_owner=None,
|
||||
retry_after_s=g.open_until - now,
|
||||
retry_after_s=self._remaining(g),
|
||||
)
|
||||
# HALF_OPEN: 探针在途;租约过期则接管,否则拒绝(防惊群)
|
||||
if now >= g.probe_expires:
|
||||
@@ -152,7 +168,7 @@ class InMemoryGate:
|
||||
epoch=g.epoch,
|
||||
is_probe=False,
|
||||
probe_owner=None,
|
||||
retry_after_s=g.probe_expires - now,
|
||||
retry_after_s=self._remaining(g),
|
||||
)
|
||||
|
||||
def _fenced(self, g: _SourceGate, entry: GateDecision) -> bool:
|
||||
@@ -172,9 +188,7 @@ class InMemoryGate:
|
||||
state=g.state,
|
||||
epoch=g.epoch,
|
||||
failure_count=g.fails,
|
||||
retry_after_s=max(0.0, g.open_until - self._now())
|
||||
if g.state is GateState.OPEN
|
||||
else 0.0,
|
||||
retry_after_s=self._remaining(g),
|
||||
)
|
||||
|
||||
def _open(self, g: _SourceGate, reason: str, *, bump_streak: bool) -> None:
|
||||
@@ -258,14 +272,4 @@ class InMemoryGate:
|
||||
"""集合中最早可尝试时间;健康/到期返回 0。"""
|
||||
if not sources:
|
||||
raise ValueError("sources 不能为空")
|
||||
now = self._now()
|
||||
waits = []
|
||||
for name in sources:
|
||||
g = self._gate(name)
|
||||
if g.state is GateState.OPEN:
|
||||
waits.append(max(0.0, g.open_until - now))
|
||||
elif g.state is GateState.HALF_OPEN:
|
||||
waits.append(max(0.0, g.probe_expires - now))
|
||||
else:
|
||||
waits.append(0.0)
|
||||
return min(waits)
|
||||
return min(self._remaining(self._gate(name)) for name in sources)
|
||||
|
||||
@@ -42,7 +42,8 @@ if state == 'open' and now < open_until then
|
||||
return {0, state, epoch, 0, '', open_until - now}
|
||||
end
|
||||
if state == 'half_open' and now < probe_until then
|
||||
return {0, state, epoch, 0, '', probe_until - now}
|
||||
-- 探针在途: 无确定的可再试时刻 → 0(issue #14,与 memory `_remaining` 同口径)
|
||||
return {0, state, epoch, 0, '', 0}
|
||||
end
|
||||
|
||||
local next_probe_until = now + tonumber(ARGV[2])
|
||||
@@ -50,7 +51,7 @@ redis.call('HSET', KEYS[1],
|
||||
'state', 'half_open',
|
||||
'probe_owner', ARGV[1],
|
||||
'probe_until', next_probe_until)
|
||||
return {1, 'half_open', epoch, 1, ARGV[1], tonumber(ARGV[2])}
|
||||
return {1, 'half_open', epoch, 1, ARGV[1], 0}
|
||||
"""
|
||||
|
||||
# M2.5 窗口/退避公共片段(拼接进 success/failure 脚本;Lua 脚本间无法共享函数)
|
||||
@@ -124,8 +125,6 @@ end
|
||||
local deadline = 0
|
||||
if state == 'open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
|
||||
elseif state == 'half_open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
|
||||
end
|
||||
return {0, state, epoch, failures, math.max(deadline - now, 0)}
|
||||
"""
|
||||
@@ -156,8 +155,6 @@ if not matches then
|
||||
local deadline = 0
|
||||
if state == 'open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
|
||||
elseif state == 'half_open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
|
||||
end
|
||||
return {0, state, epoch, failures, math.max(deadline - now, 0)}
|
||||
end
|
||||
@@ -255,8 +252,6 @@ end
|
||||
local deadline = 0
|
||||
if state == 'open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
|
||||
elseif state == 'half_open' then
|
||||
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
|
||||
end
|
||||
return {0, state, epoch, failures, math.max(deadline - now, 0)}
|
||||
"""
|
||||
@@ -272,9 +267,6 @@ for _, key in ipairs(KEYS) do
|
||||
if state == 'open' then
|
||||
local deadline = tonumber(redis.call('HGET', key, 'open_until') or '0')
|
||||
remaining = math.max(deadline - now, 0)
|
||||
elseif state == 'half_open' then
|
||||
local deadline = tonumber(redis.call('HGET', key, 'probe_until') or '0')
|
||||
remaining = math.max(deadline - now, 0)
|
||||
end
|
||||
if minimum == nil or remaining < minimum then minimum = remaining end
|
||||
end
|
||||
|
||||
@@ -134,6 +134,7 @@ class GatewayClient:
|
||||
retry: RetryPolicy,
|
||||
backpressure: BackpressurePolicy,
|
||||
quota_full: str = "wait",
|
||||
circuit_open: str = "fail_fast",
|
||||
telemetry: TelemetryRecorder | None = None,
|
||||
pricing: PricingTable | None = None,
|
||||
text_cap: int | None = None,
|
||||
@@ -162,6 +163,7 @@ class GatewayClient:
|
||||
retry=retry,
|
||||
backpressure=backpressure,
|
||||
quota_full=quota_full,
|
||||
circuit_open=circuit_open,
|
||||
cooldown_memo=SourceCooldownMemo(now=now),
|
||||
# AIMD ceiling 尊重源级静态并发上限(独立核验 I1: 不得静默钳制大于 64 的配置)
|
||||
pacer=AdaptivePacer(
|
||||
@@ -313,6 +315,7 @@ class GatewayClient:
|
||||
retry=settings.retry,
|
||||
backpressure=settings.backpressure,
|
||||
quota_full=settings.quota_full,
|
||||
circuit_open=settings.circuit_open,
|
||||
telemetry=telemetry if telemetry is not None else _build_telemetry(settings),
|
||||
pricing=PricingTable.from_file(settings.pricing_path)
|
||||
if settings.pricing_path is not None
|
||||
|
||||
@@ -50,6 +50,9 @@ _SOURCE_FIELDS: dict[str, tuple[str, str]] = {
|
||||
_RESERVED_SEGMENTS = frozenset({"GLOBAL", "RETRY", "BREAKER", "BACKPRESSURE"})
|
||||
_SELECTORS = frozenset({"round_robin", "least_inflight", "health_aware"})
|
||||
_QUOTA_FULL = frozenset({"wait", "fail_fast"})
|
||||
# 熔断全拒时的处置(issue #14);值域与 _QUOTA_FULL 相同但语义不同——配额满是
|
||||
# "排队等自己的份额"(必然轮到),熔断开路是"等源恢复"(未必恢复),故分列两键
|
||||
_CIRCUIT_OPEN = frozenset({"wait", "fail_fast"})
|
||||
# 后端合法域: env 解析与构造期校验共用一份定义,避免两处分叉
|
||||
_LIMITER_BACKENDS = frozenset({"memory", "redis"})
|
||||
_BREAKER_BACKENDS = frozenset({"memory", "redis"})
|
||||
@@ -128,6 +131,9 @@ class GatewaySettings:
|
||||
backpressure: BackpressurePolicy
|
||||
selector: str
|
||||
quota_full: str
|
||||
# 熔断全拒时是当场判死还是等冷却过去(issue #14);缺省 fail_fast 保持
|
||||
# 存量下游的控制流不变,单源 scope 应显式配 wait
|
||||
circuit_open: str
|
||||
limiter_backend: str
|
||||
breaker_backend: str
|
||||
cache_backend: str
|
||||
@@ -203,6 +209,7 @@ class GatewaySettings:
|
||||
("telemetry_backend", _TELEMETRY_BACKENDS),
|
||||
("selector", _SELECTORS),
|
||||
("quota_full", _QUOTA_FULL),
|
||||
("circuit_open", _CIRCUIT_OPEN),
|
||||
):
|
||||
value = getattr(self, field)
|
||||
if value not in allowed:
|
||||
@@ -320,6 +327,7 @@ class GatewaySettings:
|
||||
backpressure=_load_backpressure(scope_u, env),
|
||||
selector=_load_choice(env, f"{scope_u}__SELECTOR", _SELECTORS, "health_aware"),
|
||||
quota_full=_load_choice(env, f"{scope_u}__QUOTA_FULL", _QUOTA_FULL, "wait"),
|
||||
circuit_open=_load_choice(env, f"{scope_u}__CIRCUIT_OPEN", _CIRCUIT_OPEN, "fail_fast"),
|
||||
**_load_pgw(env),
|
||||
)
|
||||
|
||||
|
||||
@@ -28,7 +28,6 @@ from loguru import logger
|
||||
from polygateway.config import EmbeddingSettings
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
CircuitOpenError,
|
||||
GovernanceBackendError,
|
||||
PolyGatewayError,
|
||||
RequestRejectedError,
|
||||
@@ -37,11 +36,11 @@ from polygateway.errors import (
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.admission import SourceAdmission, settle_and_release
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
from polygateway.middleware.ratelimit import QuotaGate
|
||||
from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
|
||||
from polygateway.middleware.telemetry import TelemetryEmitter
|
||||
from polygateway.sources import SourceCooldownMemo
|
||||
from polygateway.types import (
|
||||
ChatRequest,
|
||||
EmbeddingResponse,
|
||||
@@ -102,6 +101,7 @@ class EmbeddingClient:
|
||||
retry: RetryPolicy,
|
||||
backpressure: BackpressurePolicy,
|
||||
quota_full: str = "wait",
|
||||
circuit_open: str = "fail_fast",
|
||||
telemetry: TelemetryRecorder | None = None,
|
||||
pricing: PricingTable | None = None,
|
||||
text_cap: int | None = None,
|
||||
@@ -114,21 +114,16 @@ class EmbeddingClient:
|
||||
) -> None:
|
||||
if batch_size < 1:
|
||||
raise ValueError("batch_size 必须 ≥ 1")
|
||||
if quota_full not in ("wait", "fail_fast"):
|
||||
raise ValueError(f"quota_full 必须是 wait|fail_fast: {quota_full!r}")
|
||||
if expected_dim is not None and expected_dim < 1:
|
||||
raise ValueError("expected_dim 必须 ≥ 1")
|
||||
self._scope = scope
|
||||
# embed payload 硬编码 {model, input},带 extra_body 的源必须先剥离,
|
||||
# 否则遥测会记录一个从未发出的采样参数(issue #4 决策 G)
|
||||
self._sources = strip_unsupported_extra_body(list(sources), path="embedding")
|
||||
self._selector = selector
|
||||
self._quota = QuotaGate(limiter, scope=self._scope)
|
||||
self._breaker = BreakerGate(breaker, scope=self._scope)
|
||||
self._transport = transport
|
||||
self._retry = retry
|
||||
self._bp = backpressure
|
||||
self._quota_full = quota_full
|
||||
self._emitter = (
|
||||
TelemetryEmitter(telemetry, pricing=pricing, text_cap=text_cap) if telemetry else None
|
||||
)
|
||||
@@ -137,10 +132,23 @@ class EmbeddingClient:
|
||||
self._batch_size = batch_size
|
||||
self._normalize = normalize
|
||||
self._expected_dim = expected_dim
|
||||
self._memo = SourceCooldownMemo(now=now)
|
||||
self._now = now
|
||||
self._sleep = sleep
|
||||
self._rng = rng
|
||||
# 准入编排三条循环共用一份(issue #14);冷却备忘由它独占
|
||||
self._admission = SourceAdmission(
|
||||
scope=self._scope,
|
||||
sources=self._sources,
|
||||
selector=selector,
|
||||
quota=self._quota,
|
||||
breaker=self._breaker,
|
||||
backpressure=backpressure,
|
||||
quota_full=quota_full,
|
||||
circuit_open=circuit_open,
|
||||
now=now,
|
||||
sleep=sleep,
|
||||
rng=rng,
|
||||
)
|
||||
self._closed = False
|
||||
|
||||
async def embed(
|
||||
@@ -207,9 +215,9 @@ class EmbeddingClient:
|
||||
# 只计非生产性等待(issue #8): 真实尝试由重试预算治理,不重复烧 stall 预算
|
||||
clock = StallClock(self._now)
|
||||
while True:
|
||||
picked, gate_rejections = await self._pick_runnable(reasons)
|
||||
picked, gate_rejections = await self._admission.pick(reasons, {})
|
||||
if picked is None:
|
||||
await self._on_no_runnable(gate_rejections, reasons, clock)
|
||||
await self._admission.on_no_runnable(gate_rejections, reasons, clock)
|
||||
continue
|
||||
async with clock.attempting():
|
||||
outcome = await self._attempt(
|
||||
@@ -228,62 +236,6 @@ class EmbeddingClient:
|
||||
if not outcome.immediate:
|
||||
await self._sleep(backoff_delay(self._retry, fails, outcome.exc, self._rng))
|
||||
|
||||
async def _pick_runnable(
|
||||
self, reasons: dict[str, str]
|
||||
) -> tuple[tuple[SourceConfig, Permit, GateDecision] | None, int]:
|
||||
stats = {s.name: await self._quota.stats(s) for s in self._sources}
|
||||
gate_rejections = 0
|
||||
for cand in self._selector.order(self._sources, stats):
|
||||
if self._memo.active(cand.name):
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "cooldown"
|
||||
continue
|
||||
permit = await self._quota.try_acquire(cand)
|
||||
if permit is None:
|
||||
reasons.setdefault(cand.name, "rate_limited")
|
||||
continue
|
||||
entry = None
|
||||
try:
|
||||
entry = await self._breaker.try_enter(cand, uuid.uuid4().hex)
|
||||
finally:
|
||||
if entry is None:
|
||||
await self._settle_and_release(permit, 0)
|
||||
if entry.allowed:
|
||||
return (cand, permit, entry), gate_rejections
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "circuit_open"
|
||||
self._memo.set_until(cand.name, self._now() + entry.retry_after_s)
|
||||
await self._settle_and_release(permit, 0)
|
||||
return None, gate_rejections
|
||||
|
||||
async def _on_no_runnable(
|
||||
self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
|
||||
) -> None:
|
||||
if gate_rejections == len(self._sources):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
stall = self._bp.stall_window_s
|
||||
if clock.stalled_s() > stall and await self._quota.progress_age_s() > stall:
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
await self._sleep(self._bp.poll_interval_s * (0.5 + 0.5 * self._rng()))
|
||||
|
||||
async def _attempt(
|
||||
self,
|
||||
batch: list[str],
|
||||
@@ -377,7 +329,7 @@ class EmbeddingClient:
|
||||
)
|
||||
return _FailedBatch(exc, immediate=dead)
|
||||
finally:
|
||||
await self._settle_and_release(permit, actual)
|
||||
await settle_and_release(permit, actual)
|
||||
|
||||
# —— 辅助 ——
|
||||
|
||||
@@ -399,17 +351,6 @@ class EmbeddingClient:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
||||
|
||||
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
||||
try:
|
||||
try:
|
||||
await permit.settle(actual)
|
||||
finally:
|
||||
await permit.release()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("embedding permit 结算/释放失败(不掩盖主异常): {}", exc)
|
||||
|
||||
async def _emit(
|
||||
self,
|
||||
batch: list[str],
|
||||
@@ -559,6 +500,7 @@ class EmbeddingClient:
|
||||
retry=gw.retry,
|
||||
backpressure=gw.backpressure,
|
||||
quota_full=gw.quota_full,
|
||||
circuit_open=gw.circuit_open,
|
||||
telemetry=telemetry if telemetry is not None else _build_telemetry(gw),
|
||||
pricing=PricingTable.from_file(gw.pricing_path)
|
||||
if gw.pricing_path is not None
|
||||
|
||||
@@ -116,9 +116,20 @@ class ResultInvalidError(PolyGatewayError):
|
||||
|
||||
|
||||
class GatewayUnavailableError(PolyGatewayError):
|
||||
"""scope 级暂时不可用;业务侧 catch 本类做延期重投(CHS arq 模式)。
|
||||
"""scope 级暂时不可用: 库的**调用级**预算已经耗尽。
|
||||
|
||||
`retry_after_s` 非可选(0 = 可立即重试),承 CHS ProviderUnavailableError。
|
||||
**职责边界(issue #14)**: 调用级的重试、退避、换源、等待冷却全部在库内,
|
||||
不需要下游再写一层——两边各写一份必然漂移(库调了退避曲线而下游不知道,
|
||||
下游改了等待上限而库的遥测算不进去),漂移之后"这次调用到底等了多久、
|
||||
试了几次"就没有单一事实源答得出来。本异常表示那份预算(重试预算或 stall
|
||||
预算)已经用完。下游据此再投是**任务级重试**,与调用级重试语义不同,由
|
||||
业务自行在库外包(ARCH §7.2 单层重试原则)。
|
||||
|
||||
熔断开路时是当场抛本类还是先等冷却过去,由 `{SCOPE}__CIRCUIT_OPEN`
|
||||
决定(缺省 fail_fast;单源 scope 建议配 wait)。
|
||||
|
||||
`retry_after_s` 非可选,语义是"距离**确定**可再试的时刻还有多久";
|
||||
`0` 表示不存在确定的等待时刻(可立即重试),承 CHS ProviderUnavailableError。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
"""SourceAdmission: 一次尝试的准入编排,三条治理循环(chat/embedding/ocr)共用一份。
|
||||
|
||||
**收敛缘由(issue #14)**: 本模块的两个方法此前在 `middleware/retry.py`、
|
||||
`embedding.py`、`ocr.py` 各存一份逐字复制(后两份是第一份的子集)。准入语义
|
||||
一直在演进——issue #8 改过 stall 口径、M2.5 加过 AIMD pacer、issue #14 要加
|
||||
熔断等待档——每演进一次就要三处同步,漏一处即行为分叉。三份复制正是库铁律
|
||||
痛斥的那种模式(遥测"三项目 4 处复制"的教训),只不过这次发生在库内部。
|
||||
|
||||
**职责边界**: 只管"挑出一个可跑的源"与"一个都挑不出来时怎么办";一次尝试
|
||||
本身(transport 调用、记账写回、逐次遥测)仍归各循环的 `_attempt`。
|
||||
|
||||
**共享而非持有**: `QuotaGate`/`BreakerGate`/`AdaptivePacer`/`SourceSelector` 由
|
||||
调用方构造后传入**同一实例**——三处 `_attempt` 仍要用它们做记账写回与
|
||||
`pacer.leave()`。pacer 尤其不能各建一个: 它有在途计数,分裂成两个计数器会让
|
||||
`admit`/`enter` 与 `leave` 记到不同账上。`SourceCooldownMemo` 只被准入消费,
|
||||
由本类独占。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import random
|
||||
import time
|
||||
import uuid
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from polygateway.errors import AllSourcesExhausted, CircuitOpenError
|
||||
from polygateway.sources import SourceCooldownMemo
|
||||
|
||||
# 两个准入策略键共用的值域;校验只此一处,不在各客户端重复
|
||||
_POLICIES = frozenset({"wait", "fail_fast"})
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
from polygateway.middleware.ratelimit import QuotaGate
|
||||
from polygateway.middleware.retry import StallClock
|
||||
from polygateway.ports import GateDecision, Permit, SourceSelector
|
||||
from polygateway.sources import AdaptivePacer
|
||||
from polygateway.types import BackpressurePolicy, SourceConfig
|
||||
|
||||
|
||||
async def settle_and_release(permit: Permit, actual: int) -> None:
|
||||
"""finally 专用: settle 后必 release;失败降级 warning,绝不掩盖主异常/取消。
|
||||
|
||||
三条循环的 `_attempt` 与本模块的准入拒绝路径共用这一份(此前三处逐字复制,
|
||||
仅 warning 文案不同)。
|
||||
"""
|
||||
try:
|
||||
try:
|
||||
await permit.settle(actual)
|
||||
finally:
|
||||
await permit.release()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("permit 结算/释放失败(不掩盖主异常): {}", exc)
|
||||
|
||||
|
||||
def _demote_call_failures(
|
||||
ordered: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float] | None,
|
||||
) -> list[SourceConfig]:
|
||||
"""调用内降权(设计 §3.3/§3.36): 失败 ≥2 次且存在可信替代才让位。
|
||||
|
||||
可信替代 = 某未失败候选 health ≥ 0.5 × 失败源 health——异构池里健康源
|
||||
偶发失败不该被推向已知坏源(第三轮教训: 期望成功率 83% vs 10%)。
|
||||
无健康视图(round_robin 等)保持无条件降权(冷启动保护)。
|
||||
|
||||
`attempt_fails` 为空时恒等返回原列表对象——embedding/ocr 不维护调用内
|
||||
失败计数,故对它们这一步是零成本的空操作,无需在调用侧加分支。
|
||||
"""
|
||||
demoted = [s for s in ordered if attempt_fails.get(s.name, 0) >= 2]
|
||||
if not demoted or len(demoted) == len(ordered):
|
||||
return ordered
|
||||
if health is None:
|
||||
return _move_to_tail(ordered, demoted)
|
||||
return _health_gated_reorder(ordered, demoted, attempt_fails, health)
|
||||
|
||||
|
||||
def _move_to_tail(ordered: list[SourceConfig], demoted: list[SourceConfig]) -> list[SourceConfig]:
|
||||
"""无健康视图: 无条件移尾(冷启动保护原语义)。"""
|
||||
names = {d.name for d in demoted}
|
||||
return [s for s in ordered if s.name not in names] + demoted
|
||||
|
||||
|
||||
def _health_gated_reorder(
|
||||
ordered: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""健康门槛降权: 无可信替代则原地重试;有则插到可信替代之后。"""
|
||||
demoted = _credible_demotions(ordered, demoted, attempt_fails, health)
|
||||
if not demoted:
|
||||
return ordered
|
||||
names = {d.name for d in demoted}
|
||||
rest = [s for s in ordered if s.name not in names]
|
||||
return _insert_after_credible(rest, demoted, health)
|
||||
|
||||
|
||||
def _insert_after_credible(
|
||||
rest: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""插入位置(第四轮教训): 被降权源排在可信替代之后、不可信源之前——
|
||||
可信替代被限流闸/熔断跳过时,下一候选是失败源本身而非垃圾源。"""
|
||||
bar = 0.5 * max(health(d.name) for d in demoted)
|
||||
credible = [s for s in rest if health(s.name) >= bar]
|
||||
junk = [s for s in rest if health(s.name) < bar]
|
||||
return credible + demoted + junk
|
||||
|
||||
|
||||
def _credible_demotions(
|
||||
ordered: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""健康门槛过滤: 仅当存在"健康分 ≥ 失败源一半"的未失败候选,让位才有意义。"""
|
||||
alts = [o for o in ordered if attempt_fails.get(o.name, 0) < 2]
|
||||
return [s for s in demoted if any(health(o.name) >= 0.5 * health(s.name) for o in alts)]
|
||||
|
||||
|
||||
class SourceAdmission:
|
||||
"""准入编排器(CHS `governance.py:107-285` 同款);时钟/睡眠/随机全部注入。"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
scope: str,
|
||||
sources: list[SourceConfig],
|
||||
selector: SourceSelector,
|
||||
quota: QuotaGate,
|
||||
breaker: BreakerGate,
|
||||
backpressure: BackpressurePolicy,
|
||||
quota_full: str,
|
||||
circuit_open: str,
|
||||
memo: SourceCooldownMemo | None = None,
|
||||
pacer: AdaptivePacer | None = None,
|
||||
health_view: Callable[[str], float] | None = None,
|
||||
now: Callable[[], float] = time.monotonic,
|
||||
sleep: Callable[[float], object] = asyncio.sleep,
|
||||
rng: Callable[[], float] = random.random,
|
||||
) -> None:
|
||||
for name, value in (("quota_full", quota_full), ("circuit_open", circuit_open)):
|
||||
if value not in _POLICIES:
|
||||
raise ValueError(f"{name} 必须是 wait|fail_fast: {value!r}")
|
||||
self._scope = scope
|
||||
self._sources = sources
|
||||
self._selector = selector
|
||||
self._quota = quota
|
||||
self._breaker = breaker
|
||||
self._bp = backpressure
|
||||
self._quota_full = quota_full
|
||||
self._circuit_open = circuit_open
|
||||
self._memo = memo or SourceCooldownMemo(now=now)
|
||||
self._pacer = pacer
|
||||
self._health_view = health_view
|
||||
self._now = now
|
||||
self._sleep = sleep
|
||||
self._rng = rng
|
||||
|
||||
# —— 选源与准入(CHS _pick_runnable 120-167)——
|
||||
|
||||
async def pick(
|
||||
self, reasons: dict[str, str], attempt_fails: dict[str, int]
|
||||
) -> tuple[tuple[SourceConfig, Permit, GateDecision] | None, int]:
|
||||
"""挑出第一个过闸的候选;返回 (选中三元组 | None, 熔断类拒绝计数)。"""
|
||||
stats = {s.name: await self._quota.stats(s) for s in self._sources}
|
||||
gate_rejections = 0
|
||||
ordered = _demote_call_failures(
|
||||
self._selector.order(self._sources, stats), attempt_fails, self._health_view
|
||||
)
|
||||
for cand in ordered:
|
||||
if self._memo.active(cand.name):
|
||||
# 冷却备忘跳过也计入拒绝数,保住 circuit_open 判据(CHS 同款)
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "cooldown"
|
||||
continue
|
||||
if self._pacer is not None and not self._pacer.admit(cand.name):
|
||||
# AIMD 超限: 不计 gate_rejections → 走 quota-wait 排队,不误判熔断
|
||||
reasons.setdefault(cand.name, "adaptive_paced")
|
||||
continue
|
||||
permit = await self._quota.try_acquire(cand)
|
||||
if permit is None:
|
||||
reasons.setdefault(cand.name, "rate_limited")
|
||||
continue
|
||||
entry = None
|
||||
try:
|
||||
entry = await self._breaker.try_enter(cand, uuid.uuid4().hex)
|
||||
finally:
|
||||
# try_enter 未归还 entry(异常/取消)→ 释放已占 permit,不吞任何异常
|
||||
if entry is None:
|
||||
await settle_and_release(permit, 0)
|
||||
if entry.allowed:
|
||||
if self._pacer is not None:
|
||||
self._pacer.enter(cand.name)
|
||||
return (cand, permit, entry), gate_rejections
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "circuit_open"
|
||||
# 开路源本地记冷却,避免每轮白烧 RPM 探测(CHS governance.py:107)
|
||||
self._memo.set_until(cand.name, self._now() + entry.retry_after_s)
|
||||
await settle_and_release(permit, 0)
|
||||
return None, gate_rejections
|
||||
|
||||
# —— 背压与 stall 判死(CHS governance.py:270-285)——
|
||||
|
||||
async def stalled(self, clock: StallClock) -> bool:
|
||||
"""双条件 stall 判死(CHS governance.py:270-281): 本地累计等待与全局
|
||||
无进展**同时**超窗才判死——本地 monotonic 与后端时钟刻意不混用。
|
||||
|
||||
本地一侧只计非生产性等待(issue #8,见 `StallClock`)。短路顺序有意为之:
|
||||
本地未超窗就不问后端,省一次 Redis 往返。
|
||||
"""
|
||||
stall = self._bp.stall_window_s
|
||||
return clock.stalled_s() > stall and await self._quota.progress_age_s() > stall
|
||||
|
||||
async def on_no_runnable(
|
||||
self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
|
||||
) -> None:
|
||||
"""一个源都挑不出来时的处置: **按拒绝原因分派**到各自的策略。
|
||||
|
||||
分派而非串行是硬要求(issue #14): 串行写法下 `circuit_open=wait` 不抛
|
||||
之后会径直掉进配额分支,`quota_full=fail_fast` 的调用方于是收到一个
|
||||
`reason=quota_exhausted` 的异常——而配额其实是满的,坏的是熔断门。
|
||||
"""
|
||||
names = tuple(s.name for s in self._sources)
|
||||
if gate_rejections == len(self._sources):
|
||||
# 全部因熔断类原因(门开路 / 本地冷却备忘)被拒
|
||||
if self._circuit_open == "fail_fast":
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
# wait: 保护作用完整保留(这一轮照样一个请求都不发),改变的只是
|
||||
# 调用方当场死还是排队等——多源可换源故 fail-fast 对,单源无源可换
|
||||
hint = await self._breaker.retry_after_s(names)
|
||||
else:
|
||||
# 至少一个源是被配额/AIMD 挡的,归 quota_full 管
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
hint = 0.0
|
||||
if await self.stalled(clock):
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
nap = self._nap(hint, clock)
|
||||
if hint > 0:
|
||||
logger.info("熔断开路等待 {:.1f}s 后重试(scope={}, 原因={})", nap, self._scope, reasons)
|
||||
await self._sleep(nap)
|
||||
|
||||
def _nap(self, hint: float, clock: StallClock) -> float:
|
||||
"""本轮等待多久。**必须在 `stalled()` 判定之后调用**(预算可能已耗尽)。
|
||||
|
||||
`hint > 0`(熔断开路有确定的冷却截止)时睡到那个时刻,而不是按
|
||||
`poll_interval` 空转——60 秒冷却用 10ms 轮询是 6000 次空转,内存后端
|
||||
只是查字典,Redis 后端则是 6000 次往返 × 每个在途调用。抖动**上**加
|
||||
而非缩放(既有 quota 路径是 `[0.5p, 1.0p]`): 对一个确定的截止时刻提前
|
||||
醒来必然被再拒一次,白跑一趟。
|
||||
|
||||
两档都夹到剩余 stall 预算,故单次调用的最坏墙钟是 `stall_window_s`
|
||||
加一个 poll 间隔,不随 `max_cooldown_s` 漂移。多加的那一格是因为
|
||||
`stalled()` 判据是 `>` 而非 `>=`——恰好睡到窗口边界不判死,留这一格
|
||||
让下一轮必定判死。`hint == 0` 时整个式子退化为既有的 jitter 轮询。
|
||||
"""
|
||||
jitter = self._bp.poll_interval_s * (0.5 + 0.5 * self._rng())
|
||||
budget = self._bp.stall_window_s - clock.stalled_s() + self._bp.poll_interval_s
|
||||
wait = hint + jitter if hint > 0 else jitter
|
||||
# 下界取 jitter 而非 poll_interval: 既有 quota 轮询是 [0.5p, 1.0p],用
|
||||
# poll_interval 兜底会把 rng→0 那半边抬上去。预算为负时(本地已超窗但
|
||||
# 全局仍在出餐,故 stalled() 不判死)靠它退回正常轮询节奏,不忙循环。
|
||||
return max(jitter, min(wait, budget))
|
||||
@@ -23,7 +23,6 @@ from loguru import logger
|
||||
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
CircuitOpenError,
|
||||
GovernanceBackendError,
|
||||
PolyGatewayError,
|
||||
RequestRejectedError,
|
||||
@@ -32,10 +31,11 @@ from polygateway.errors import (
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.admission import SourceAdmission, settle_and_release
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
from polygateway.middleware.ratelimit import QuotaGate
|
||||
from polygateway.ports import OutcomeAwareSelector
|
||||
from polygateway.sources import AdaptivePacer, SourceCooldownMemo
|
||||
from polygateway.sources import AdaptivePacer
|
||||
from polygateway.streaming import StreamLivenessTimeout
|
||||
from polygateway.types import LLMResponse
|
||||
|
||||
@@ -50,6 +50,7 @@ if TYPE_CHECKING:
|
||||
SourceSelector,
|
||||
Transport,
|
||||
)
|
||||
from polygateway.sources import SourceCooldownMemo
|
||||
from polygateway.types import (
|
||||
BackpressurePolicy,
|
||||
ChatRequest,
|
||||
@@ -134,70 +135,6 @@ class StallClock:
|
||||
self._productive_s += self._now() - started
|
||||
|
||||
|
||||
def _demote_call_failures(
|
||||
ordered: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float] | None,
|
||||
) -> list[SourceConfig]:
|
||||
"""调用内降权(设计 §3.3/§3.36): 失败 ≥2 次且存在可信替代才让位。
|
||||
|
||||
可信替代 = 某未失败候选 health ≥ 0.5 × 失败源 health——异构池里健康源
|
||||
偶发失败不该被推向已知坏源(第三轮教训: 期望成功率 83% vs 10%)。
|
||||
无健康视图(round_robin 等)保持无条件降权(冷启动保护)。
|
||||
"""
|
||||
demoted = [s for s in ordered if attempt_fails.get(s.name, 0) >= 2]
|
||||
if not demoted or len(demoted) == len(ordered):
|
||||
return ordered
|
||||
if health is None:
|
||||
return _move_to_tail(ordered, demoted)
|
||||
return _health_gated_reorder(ordered, demoted, attempt_fails, health)
|
||||
|
||||
|
||||
def _move_to_tail(ordered: list[SourceConfig], demoted: list[SourceConfig]) -> list[SourceConfig]:
|
||||
"""无健康视图: 无条件移尾(冷启动保护原语义)。"""
|
||||
names = {d.name for d in demoted}
|
||||
return [s for s in ordered if s.name not in names] + demoted
|
||||
|
||||
|
||||
def _health_gated_reorder(
|
||||
ordered: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""健康门槛降权: 无可信替代则原地重试;有则插到可信替代之后。"""
|
||||
demoted = _credible_demotions(ordered, demoted, attempt_fails, health)
|
||||
if not demoted:
|
||||
return ordered
|
||||
names = {d.name for d in demoted}
|
||||
rest = [s for s in ordered if s.name not in names]
|
||||
return _insert_after_credible(rest, demoted, health)
|
||||
|
||||
|
||||
def _insert_after_credible(
|
||||
rest: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""插入位置(第四轮教训): 被降权源排在可信替代之后、不可信源之前——
|
||||
可信替代被限流闸/熔断跳过时,下一候选是失败源本身而非垃圾源。"""
|
||||
bar = 0.5 * max(health(d.name) for d in demoted)
|
||||
credible = [s for s in rest if health(s.name) >= bar]
|
||||
junk = [s for s in rest if health(s.name) < bar]
|
||||
return credible + demoted + junk
|
||||
|
||||
|
||||
def _credible_demotions(
|
||||
ordered: list[SourceConfig],
|
||||
demoted: list[SourceConfig],
|
||||
attempt_fails: dict[str, int],
|
||||
health: Callable[[str], float],
|
||||
) -> list[SourceConfig]:
|
||||
"""健康门槛过滤: 仅当存在"健康分 ≥ 失败源一半"的未失败候选,让位才有意义。"""
|
||||
alts = [o for o in ordered if attempt_fails.get(o.name, 0) < 2]
|
||||
return [s for s in demoted if any(health(o.name) >= 0.5 * health(s.name) for o in alts)]
|
||||
|
||||
|
||||
def _failure_reason(exc: PolyGatewayError) -> str:
|
||||
"""失败原因归类(CHS governance.py:169 同款)。"""
|
||||
if isinstance(exc, SourceDeadError):
|
||||
@@ -241,6 +178,7 @@ class RetryMW:
|
||||
retry: RetryPolicy,
|
||||
backpressure: BackpressurePolicy,
|
||||
quota_full: str = "wait",
|
||||
circuit_open: str = "fail_fast",
|
||||
cooldown_memo: SourceCooldownMemo | None = None,
|
||||
pacer: AdaptivePacer | None = None,
|
||||
emitter: object | None = None,
|
||||
@@ -248,27 +186,39 @@ class RetryMW:
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
rng: Callable[[], float] = random.random,
|
||||
) -> None:
|
||||
if quota_full not in ("wait", "fail_fast"):
|
||||
raise ValueError(f"quota_full 必须是 wait|fail_fast: {quota_full!r}")
|
||||
self._scope = scope
|
||||
self._sources = list(sources)
|
||||
self._selector = selector
|
||||
# 记账写回与 pacer 结算仍在 `_attempt` 内,故这三者由本类持有并与
|
||||
# `SourceAdmission` **共享同一实例**(pacer 有在途计数,不可分裂)
|
||||
self._quota = QuotaGate(limiter, scope=self._scope)
|
||||
self._breaker = BreakerGate(gate, scope=self._scope)
|
||||
self._transport = transport
|
||||
self._retry = retry
|
||||
self._bp = backpressure
|
||||
self._quota_full = quota_full
|
||||
self._memo = cooldown_memo or SourceCooldownMemo(now=now)
|
||||
# M2.5: 选源器可选健康喂数端口,构造期 isinstance 判定一次(设计 §3.2)
|
||||
self._outcome_sink = selector if isinstance(selector, OutcomeAwareSelector) else None
|
||||
self._health_view = self._outcome_sink.health if self._outcome_sink else None
|
||||
# M2.5 §3.35: AIMD 自适应并发——429 收紧、成功回涨,超限调用排队不烧预算
|
||||
self._pacer = pacer or AdaptivePacer(ceiling=64.0)
|
||||
self._emitter = emitter
|
||||
self._now = now
|
||||
self._sleep = sleep
|
||||
self._rng = rng
|
||||
# 准入编排三条循环共用一份(issue #14);冷却备忘由它独占
|
||||
self._admission = SourceAdmission(
|
||||
scope=self._scope,
|
||||
sources=self._sources,
|
||||
selector=selector,
|
||||
quota=self._quota,
|
||||
breaker=self._breaker,
|
||||
backpressure=backpressure,
|
||||
quota_full=quota_full,
|
||||
circuit_open=circuit_open,
|
||||
memo=cooldown_memo,
|
||||
pacer=self._pacer,
|
||||
health_view=self._outcome_sink.health if self._outcome_sink else None,
|
||||
now=now,
|
||||
sleep=sleep,
|
||||
rng=rng,
|
||||
)
|
||||
|
||||
async def __call__(self, request: ChatRequest) -> LLMResponse:
|
||||
"""执行治理调用;scope 级失败按 §6.1 携结构化字段上抛。"""
|
||||
@@ -283,16 +233,16 @@ class RetryMW:
|
||||
clock = StallClock(self._now)
|
||||
while True:
|
||||
# 调用级时间上限(迭代 5): 429 免预算后的兜底,防饱和期无限循环
|
||||
if await self._stalled(clock):
|
||||
if await self._admission.stalled(clock):
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=self._retry.backoff_base_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
picked, gate_rejections = await self._pick_runnable(reasons, attempt_fails)
|
||||
picked, gate_rejections = await self._admission.pick(reasons, attempt_fails)
|
||||
if picked is None:
|
||||
await self._on_no_runnable(gate_rejections, reasons, clock)
|
||||
await self._admission.on_no_runnable(gate_rejections, reasons, clock)
|
||||
continue
|
||||
async with clock.attempting() as attempt:
|
||||
outcome = await self._attempt(request, *picked, reasons, attempt_fails)
|
||||
@@ -314,87 +264,6 @@ class RetryMW:
|
||||
if not outcome.immediate:
|
||||
await self._sleep(self._backoff_delay(max(fails, 1), outcome.exc))
|
||||
|
||||
# —— 选源与准入(CHS _pick_runnable 120-167)——
|
||||
|
||||
async def _pick_runnable(
|
||||
self, reasons: dict[str, str], attempt_fails: dict[str, int]
|
||||
) -> tuple[tuple[SourceConfig, Permit, GateDecision] | None, int]:
|
||||
stats = {s.name: await self._quota.stats(s) for s in self._sources}
|
||||
gate_rejections = 0
|
||||
ordered = _demote_call_failures(
|
||||
self._selector.order(self._sources, stats), attempt_fails, self._health_view
|
||||
)
|
||||
for cand in ordered:
|
||||
if self._memo.active(cand.name):
|
||||
# 冷却备忘跳过也计入拒绝数,保住 circuit_open 判据(CHS 同款)
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "cooldown"
|
||||
continue
|
||||
if not self._pacer.admit(cand.name):
|
||||
# AIMD 超限: 不计 gate_rejections → 走 quota-wait 排队,不误判熔断
|
||||
reasons.setdefault(cand.name, "adaptive_paced")
|
||||
continue
|
||||
permit = await self._quota.try_acquire(cand)
|
||||
if permit is None:
|
||||
reasons.setdefault(cand.name, "rate_limited")
|
||||
continue
|
||||
entry = None
|
||||
try:
|
||||
entry = await self._breaker.try_enter(cand, uuid.uuid4().hex)
|
||||
finally:
|
||||
# try_enter 未归还 entry(异常/取消)→ 释放已占 permit,不吞任何异常
|
||||
if entry is None:
|
||||
await self._settle_and_release(permit, 0)
|
||||
if entry.allowed:
|
||||
self._pacer.enter(cand.name)
|
||||
return (cand, permit, entry), gate_rejections
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "circuit_open"
|
||||
# 开路源本地记冷却,避免每轮白烧 RPM 探测(CHS governance.py:107)
|
||||
self._memo.set_until(cand.name, self._now() + entry.retry_after_s)
|
||||
await self._settle_and_release(permit, 0)
|
||||
return None, gate_rejections
|
||||
|
||||
# —— 背压与 stall 判死(CHS governance.py:270-285)——
|
||||
|
||||
async def _stalled(self, clock: StallClock) -> bool:
|
||||
"""双条件 stall 判死(CHS governance.py:270-281): 本地累计等待与全局
|
||||
无进展**同时**超窗才判死——本地 monotonic 与后端时钟刻意不混用。
|
||||
|
||||
本地一侧只计非生产性等待(issue #8,见 `StallClock`)。短路顺序有意为之:
|
||||
本地未超窗就不问后端,省一次 Redis 往返。
|
||||
"""
|
||||
stall = self._bp.stall_window_s
|
||||
return clock.stalled_s() > stall and await self._quota.progress_age_s() > stall
|
||||
|
||||
async def _on_no_runnable(
|
||||
self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
|
||||
) -> None:
|
||||
if gate_rejections == len(self._sources):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
if await self._stalled(clock):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
# jitter ∈ [0.5p, 1.0p] 防惊群(CHS governance.py:283-285)
|
||||
await self._sleep(self._bp.poll_interval_s * (0.5 + 0.5 * self._rng()))
|
||||
|
||||
# —— 单次尝试(CHS run 200-268)——
|
||||
|
||||
async def _attempt(
|
||||
@@ -460,7 +329,7 @@ class RetryMW:
|
||||
return _Failed(exc, immediate=dead)
|
||||
finally:
|
||||
self._pacer.leave(source.name)
|
||||
await self._settle_and_release(permit, actual)
|
||||
await settle_and_release(permit, actual)
|
||||
|
||||
async def _on_rejected(
|
||||
self, exc: RequestRejectedError, source: SourceConfig, entry: GateDecision
|
||||
@@ -523,18 +392,6 @@ class RetryMW:
|
||||
reasoning_tokens=result.reasoning_tokens,
|
||||
)
|
||||
|
||||
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
||||
"""finally 专用: settle 后必 release;失败降级 warning,绝不掩盖主异常/取消。"""
|
||||
try:
|
||||
try:
|
||||
await permit.settle(actual)
|
||||
finally:
|
||||
await permit.release()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("permit 结算/释放失败(不掩盖主异常): {}", exc)
|
||||
|
||||
async def _emit(
|
||||
self,
|
||||
request: ChatRequest,
|
||||
|
||||
+20
-78
@@ -24,7 +24,6 @@ from loguru import logger
|
||||
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
CircuitOpenError,
|
||||
GovernanceBackendError,
|
||||
PolyGatewayError,
|
||||
RequestRejectedError,
|
||||
@@ -33,12 +32,12 @@ from polygateway.errors import (
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.admission import SourceAdmission, settle_and_release
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
from polygateway.middleware.ratelimit import QuotaGate
|
||||
from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
|
||||
from polygateway.middleware.telemetry import TelemetryEmitter
|
||||
from polygateway.ports import OutcomeAwareSelector
|
||||
from polygateway.sources import SourceCooldownMemo
|
||||
from polygateway.types import (
|
||||
ChatRequest,
|
||||
LLMResponse,
|
||||
@@ -108,14 +107,13 @@ class OcrClient:
|
||||
retry: RetryPolicy,
|
||||
backpressure: BackpressurePolicy,
|
||||
quota_full: str = "wait",
|
||||
circuit_open: str = "fail_fast",
|
||||
telemetry: TelemetryRecorder | None = None,
|
||||
text_cap: int | None = None,
|
||||
now: Callable[[], float] = time.monotonic,
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
rng: Callable[[], float] = random.random,
|
||||
) -> None:
|
||||
if quota_full not in ("wait", "fail_fast"):
|
||||
raise ValueError(f"quota_full 必须是 wait|fail_fast: {quota_full!r}")
|
||||
self._scope = scope
|
||||
# MonkeyOCR 只发 multipart 表单,带 extra_body 的源必须先剥离,否则
|
||||
# 遥测会记录一个从未发出的采样参数(issue #4 决策 G)
|
||||
@@ -126,14 +124,25 @@ class OcrClient:
|
||||
self._breaker = BreakerGate(breaker, scope=self._scope)
|
||||
self._transport = transport
|
||||
self._retry = retry
|
||||
self._bp = backpressure
|
||||
self._quota_full = quota_full
|
||||
self._emitter = TelemetryEmitter(telemetry, text_cap=text_cap) if telemetry else None
|
||||
self._telemetry = telemetry
|
||||
self._memo = SourceCooldownMemo(now=now)
|
||||
self._now = now
|
||||
self._sleep = sleep
|
||||
self._rng = rng
|
||||
# 准入编排三条循环共用一份(issue #14);冷却备忘由它独占
|
||||
self._admission = SourceAdmission(
|
||||
scope=self._scope,
|
||||
sources=self._sources,
|
||||
selector=selector,
|
||||
quota=self._quota,
|
||||
breaker=self._breaker,
|
||||
backpressure=backpressure,
|
||||
quota_full=quota_full,
|
||||
circuit_open=circuit_open,
|
||||
now=now,
|
||||
sleep=sleep,
|
||||
rng=rng,
|
||||
)
|
||||
self._closed = False
|
||||
|
||||
# —— 公共端口(OcrTextPort / OcrLayoutPort)——
|
||||
@@ -234,9 +243,9 @@ class OcrClient:
|
||||
# 只计非生产性等待(issue #8): 真实尝试由重试预算治理,不重复烧 stall 预算
|
||||
clock = StallClock(self._now)
|
||||
while True:
|
||||
picked, gate_rejections = await self._pick_runnable(reasons)
|
||||
picked, gate_rejections = await self._admission.pick(reasons, {})
|
||||
if picked is None:
|
||||
await self._on_no_runnable(gate_rejections, reasons, clock)
|
||||
await self._admission.on_no_runnable(gate_rejections, reasons, clock)
|
||||
continue
|
||||
async with clock.attempting():
|
||||
outcome = await self._attempt(
|
||||
@@ -255,62 +264,6 @@ class OcrClient:
|
||||
if not outcome.immediate:
|
||||
await self._sleep(backoff_delay(self._retry, fails, outcome.exc, self._rng))
|
||||
|
||||
async def _pick_runnable(
|
||||
self, reasons: dict[str, str]
|
||||
) -> tuple[tuple[SourceConfig, Permit, GateDecision] | None, int]:
|
||||
stats = {s.name: await self._quota.stats(s) for s in self._sources}
|
||||
gate_rejections = 0
|
||||
for cand in self._selector.order(self._sources, stats):
|
||||
if self._memo.active(cand.name):
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "cooldown"
|
||||
continue
|
||||
permit = await self._quota.try_acquire(cand)
|
||||
if permit is None:
|
||||
reasons.setdefault(cand.name, "rate_limited")
|
||||
continue
|
||||
entry = None
|
||||
try:
|
||||
entry = await self._breaker.try_enter(cand, uuid.uuid4().hex)
|
||||
finally:
|
||||
if entry is None:
|
||||
await self._settle_and_release(permit)
|
||||
if entry.allowed:
|
||||
return (cand, permit, entry), gate_rejections
|
||||
gate_rejections += 1
|
||||
reasons[cand.name] = "circuit_open"
|
||||
self._memo.set_until(cand.name, self._now() + entry.retry_after_s)
|
||||
await self._settle_and_release(permit)
|
||||
return None, gate_rejections
|
||||
|
||||
async def _on_no_runnable(
|
||||
self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
|
||||
) -> None:
|
||||
if gate_rejections == len(self._sources):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
stall = self._bp.stall_window_s
|
||||
if clock.stalled_s() > stall and await self._quota.progress_age_s() > stall:
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
await self._sleep(self._bp.poll_interval_s * (0.5 + 0.5 * self._rng()))
|
||||
|
||||
async def _attempt(
|
||||
self,
|
||||
kind: _OcrKind,
|
||||
@@ -398,7 +351,7 @@ class OcrClient:
|
||||
)
|
||||
return _FailedAttempt(exc, immediate=dead)
|
||||
finally:
|
||||
await self._settle_and_release(permit)
|
||||
await settle_and_release(permit, 0)
|
||||
|
||||
async def _invoke(
|
||||
self, kind: _OcrKind, image: bytes, source: SourceConfig, call_id: str
|
||||
@@ -435,18 +388,6 @@ class OcrClient:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
||||
|
||||
async def _settle_and_release(self, permit: Permit) -> None:
|
||||
"""settle 恒 0: OCR 无 token 计费(设计 §5 差异①)。"""
|
||||
try:
|
||||
try:
|
||||
await permit.settle(0)
|
||||
finally:
|
||||
await permit.release()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("OCR permit 结算/释放失败(不掩盖主异常): {}", exc)
|
||||
|
||||
async def _emit(
|
||||
self,
|
||||
kind: _OcrKind,
|
||||
@@ -572,6 +513,7 @@ class OcrClient:
|
||||
retry=gw.retry,
|
||||
backpressure=gw.backpressure,
|
||||
quota_full=gw.quota_full,
|
||||
circuit_open=gw.circuit_open,
|
||||
telemetry=telemetry if telemetry is not None else _build_telemetry(gw),
|
||||
# OCR 行与 chat 行写同一张 llm_calls;漏传这一条,同表内就一半受控
|
||||
# 一半不受控(issue #12)
|
||||
|
||||
@@ -291,6 +291,50 @@ class TestRetryAfter:
|
||||
await _open_gate(gate, "s1") # s1 开路;s2 健康
|
||||
assert await gate.retry_after_s(("s1", "s2")) == 0.0
|
||||
|
||||
async def test_half_open_rejection_reports_no_certain_wait(self, gate_factory, clock):
|
||||
"""探针在途时被拒 → 0.0(issue #14): 探针随时可能出结果,不存在确定时刻。
|
||||
|
||||
旧行为返回探针租约剩余,而租约长度是**死锁保护参数**(派生自
|
||||
`2 × 最慢源 timeout`),与"这个源多久能恢复"没有因果关系。现场
|
||||
`TIMEOUT_S=300` 时它是 600s,而冷却期只有 60s。
|
||||
"""
|
||||
gate = gate_factory(_CFG)
|
||||
await _open_gate(gate)
|
||||
clock.advance(_CFG.cooldown_s + 1)
|
||||
probe = await gate.try_enter("s1", "w1")
|
||||
assert probe.is_probe
|
||||
blocked = await gate.try_enter("s1", "w2")
|
||||
assert not blocked.allowed and blocked.state is GateState.HALF_OPEN
|
||||
assert blocked.retry_after_s == 0.0
|
||||
|
||||
async def test_probe_grant_reports_no_certain_wait(self, gate_factory, clock):
|
||||
"""准入被允许 → 恒 0.0(现在就能试);此前 redis 侧返回探针 TTL。"""
|
||||
gate = gate_factory(_CFG)
|
||||
await _open_gate(gate)
|
||||
clock.advance(_CFG.cooldown_s + 1)
|
||||
probe = await gate.try_enter("s1", "w1")
|
||||
assert probe.allowed and probe.is_probe
|
||||
assert probe.retry_after_s == 0.0
|
||||
|
||||
async def test_retry_after_zero_while_probe_in_flight(self, gate_factory, clock):
|
||||
"""集合查询同口径: 探针在途的源不贡献等待时间。"""
|
||||
gate = gate_factory(_CFG)
|
||||
await _open_gate(gate)
|
||||
clock.advance(_CFG.cooldown_s + 1)
|
||||
assert (await gate.try_enter("s1", "w1")).is_probe
|
||||
assert await gate.retry_after_s(("s1",)) == 0.0
|
||||
|
||||
async def test_fenced_write_in_half_open_reports_no_certain_wait(self, gate_factory, clock):
|
||||
"""写回被 fencing 拒时的快照同口径;此前 redis 侧返回探针租约剩余。"""
|
||||
gate = gate_factory(_CFG)
|
||||
stale = await gate.try_enter("s1", "slow-worker") # epoch 0 的旧 entry
|
||||
await _open_gate(gate) # 他人开路,epoch 推进
|
||||
clock.advance(_CFG.cooldown_s + 1)
|
||||
assert (await gate.try_enter("s1", "w1")).is_probe # 门此刻 HALF_OPEN
|
||||
update = await gate.record_success(stale)
|
||||
assert not update.applied and update.state is GateState.HALF_OPEN
|
||||
assert update.retry_after_s == 0.0
|
||||
|
||||
|
||||
class TestConsecutiveSuppression:
|
||||
"""迭代 6: 窗口证据充足且健康时,连败是噪声,不开路(设计 §3.39)。"""
|
||||
|
||||
@@ -327,3 +327,49 @@ async def test_variant_probe_rate_limited_releases_not_hangs(redis_client):
|
||||
assert update.applied
|
||||
nxt = await gate.try_enter("s1", "w2")
|
||||
assert nxt.allowed and nxt.is_probe # 立即可再探,不等 probe_ttl
|
||||
|
||||
|
||||
# —— issue #14: retry_after_s = 距离**确定**可再试的时刻,HALF_OPEN 无确定时刻 ——
|
||||
|
||||
|
||||
@pytestmark_slow
|
||||
async def test_variant_half_open_rejection_reports_no_certain_wait(redis_client):
|
||||
gate = _gate(redis_client)
|
||||
await _open_gate(gate)
|
||||
await asyncio.sleep(_CFG.cooldown_s + 1)
|
||||
probe = await gate.try_enter("s1", "w1")
|
||||
assert probe.is_probe
|
||||
blocked = await gate.try_enter("s1", "w2")
|
||||
assert not blocked.allowed and blocked.state is GateState.HALF_OPEN
|
||||
assert blocked.retry_after_s == 0.0
|
||||
|
||||
|
||||
@pytestmark_slow
|
||||
async def test_variant_probe_grant_reports_no_certain_wait(redis_client):
|
||||
gate = _gate(redis_client)
|
||||
await _open_gate(gate)
|
||||
await asyncio.sleep(_CFG.cooldown_s + 1)
|
||||
probe = await gate.try_enter("s1", "w1")
|
||||
assert probe.allowed and probe.is_probe
|
||||
assert probe.retry_after_s == 0.0
|
||||
|
||||
|
||||
@pytestmark_slow
|
||||
async def test_variant_retry_after_zero_while_probe_in_flight(redis_client):
|
||||
gate = _gate(redis_client)
|
||||
await _open_gate(gate)
|
||||
await asyncio.sleep(_CFG.cooldown_s + 1)
|
||||
assert (await gate.try_enter("s1", "w1")).is_probe
|
||||
assert await gate.retry_after_s(("s1",)) == 0.0
|
||||
|
||||
|
||||
@pytestmark_slow
|
||||
async def test_variant_fenced_write_in_half_open_reports_no_certain_wait(redis_client):
|
||||
gate = _gate(redis_client)
|
||||
stale = await gate.try_enter("s1", "slow-worker") # epoch 0 的旧 entry
|
||||
await _open_gate(gate) # 他人开路,epoch 推进
|
||||
await asyncio.sleep(_CFG.cooldown_s + 1)
|
||||
assert (await gate.try_enter("s1", "w1")).is_probe # 门此刻 HALF_OPEN
|
||||
update = await gate.record_success(stale)
|
||||
assert not update.applied and update.state is GateState.HALF_OPEN
|
||||
assert update.retry_after_s == 0.0
|
||||
|
||||
@@ -13,8 +13,10 @@ from polygateway.backends.memory.breaker import InMemoryGate
|
||||
from polygateway.backends.memory.limiter import InMemoryLimiter
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
CircuitOpenError,
|
||||
GatewayUnavailableError,
|
||||
GovernanceBackendError,
|
||||
SourceDeadError,
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
@@ -62,6 +64,7 @@ def _mw(
|
||||
sleep,
|
||||
rng=lambda: 0.0,
|
||||
quota_full="wait",
|
||||
circuit_open="fail_fast",
|
||||
gate=None,
|
||||
transport=None,
|
||||
emitter=None,
|
||||
@@ -76,6 +79,7 @@ def _mw(
|
||||
retry=RetryPolicy(max_attempts=3, backoff_base_s=2.0, backoff_max_s=30.0),
|
||||
backpressure=BackpressurePolicy(stall_window_s=_STALL, poll_interval_s=0.01),
|
||||
quota_full=quota_full,
|
||||
circuit_open=circuit_open,
|
||||
cooldown_memo=SourceCooldownMemo(now=clock),
|
||||
emitter=emitter,
|
||||
now=clock,
|
||||
@@ -593,3 +597,201 @@ class TestGateFailuresReachCallersAsScopeLevel:
|
||||
await QuotaGate(_Broken(), scope="LLM").progress_age_s()
|
||||
assert ei.value.scope == "llm"
|
||||
assert ei.value.reason == "governance_backend_down"
|
||||
|
||||
|
||||
class TestCircuitOpenPolicy:
|
||||
"""issue #14: 熔断全拒时是当场判死还是等冷却过去。
|
||||
|
||||
缺省 fail_fast 即历史行为(TestStallQuadrants 等既有用例照旧覆盖);
|
||||
本类钉的是 wait 档,以及两条策略互不串线。
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
async def _opened_gate(clock, cfg=_BREAKER):
|
||||
gate = InMemoryGate(config=cfg, now=clock)
|
||||
for _ in range(cfg.fail_threshold):
|
||||
entry = await gate.try_enter("s1", "w")
|
||||
await gate.record_failure(entry, "network_error", False)
|
||||
return gate
|
||||
|
||||
@staticmethod
|
||||
def _free_limiter(clock, src):
|
||||
return InMemoryLimiter(
|
||||
scope="llm",
|
||||
sources={"s1": src},
|
||||
global_limits=_NO_GLOBAL,
|
||||
lease_ttl_s=10_000.0,
|
||||
now=clock,
|
||||
)
|
||||
|
||||
async def test_fail_fast_is_the_default(self):
|
||||
"""缺省档逐字保持历史行为: 全源开路当场抛 CircuitOpenError。"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[],
|
||||
clock=clock,
|
||||
sleep=BoundedSleep(),
|
||||
gate=await self._opened_gate(clock),
|
||||
)
|
||||
with pytest.raises(CircuitOpenError) as ei:
|
||||
await mw(_REQ)
|
||||
assert ei.value.reason == "circuit_open"
|
||||
|
||||
async def test_wait_sleeps_out_the_cooldown_instead_of_dying(self):
|
||||
"""wait 档: 睡到冷却结束再来一轮,拿到探针后正常返回。
|
||||
|
||||
睡的是**冷却剩余**而不是 poll_interval——60 秒冷却用 10ms 轮询要空转
|
||||
6000 次,memory 后端只是查字典,Redis 后端则是 6000 次往返 × 每个在途调用。
|
||||
"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
sleep = BoundedSleep()
|
||||
|
||||
async def advance(_n):
|
||||
clock.advance(sleep.delays[-1])
|
||||
|
||||
sleep._side_effect = advance
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[_ok()],
|
||||
clock=clock,
|
||||
sleep=sleep,
|
||||
gate=await self._opened_gate(clock),
|
||||
circuit_open="wait",
|
||||
)
|
||||
resp = await mw(_REQ)
|
||||
assert resp.content == "ok"
|
||||
# 一觉睡到冷却结束(jitter 上加,rng=0 → +0.5×poll),不是 poll 空转
|
||||
assert sleep.delays[0] == pytest.approx(_BREAKER.cooldown_s + 0.005)
|
||||
|
||||
async def test_wait_does_not_leak_into_the_quota_branch(self):
|
||||
"""两条策略互不串线: circuit_open=wait 配 quota_full=fail_fast 时,
|
||||
熔断等待**不得**被当成配额耗尽上报——串线会让调用方拿到一个
|
||||
reason=quota_exhausted 的异常,而配额其实是满的。"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
sleep = BoundedSleep()
|
||||
|
||||
async def advance(_n):
|
||||
clock.advance(sleep.delays[-1])
|
||||
|
||||
sleep._side_effect = advance
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[_ok()],
|
||||
clock=clock,
|
||||
sleep=sleep,
|
||||
gate=await self._opened_gate(clock),
|
||||
quota_full="fail_fast",
|
||||
circuit_open="wait",
|
||||
)
|
||||
assert (await mw(_REQ)).content == "ok"
|
||||
|
||||
async def test_wait_still_dies_when_cooldown_outlasts_the_stall_budget(self):
|
||||
"""等待有可解释的上界: 冷却比 stall 预算还长时,在窗口耗尽处判死。
|
||||
|
||||
单次睡眠夹到剩余 stall 预算,故最坏墙钟 = stall_window + 一个 poll,
|
||||
不随 max_cooldown_s 漂移。
|
||||
"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
long_cooldown = BreakerConfig(
|
||||
fail_threshold=3, cooldown_s=1000.0, probe_ttl_s=2000.0, max_cooldown_s=1000.0
|
||||
)
|
||||
sleep = BoundedSleep()
|
||||
|
||||
async def advance(_n):
|
||||
clock.advance(sleep.delays[-1])
|
||||
|
||||
sleep._side_effect = advance
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[],
|
||||
clock=clock,
|
||||
sleep=sleep,
|
||||
gate=await self._opened_gate(clock, long_cooldown),
|
||||
circuit_open="wait",
|
||||
)
|
||||
with pytest.raises(AllSourcesExhausted) as ei:
|
||||
await mw(_REQ)
|
||||
assert ei.value.reason == "stalled"
|
||||
assert ei.value.per_source_reasons == {"s1": "circuit_open"}
|
||||
assert sleep.delays[0] == pytest.approx(_STALL + 0.01) # 夹到预算 + 一个 poll
|
||||
|
||||
async def test_wait_loop_stays_cancellable(self):
|
||||
"""取消穿透(铁律): 熔断等待中的取消不得被吞。"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[],
|
||||
clock=clock,
|
||||
sleep=asyncio.sleep,
|
||||
gate=await self._opened_gate(clock),
|
||||
circuit_open="wait",
|
||||
)
|
||||
task = asyncio.create_task(mw(_REQ))
|
||||
await asyncio.sleep(0.03)
|
||||
task.cancel()
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
async def test_wait_does_not_exempt_probes_from_the_retry_budget(self):
|
||||
"""wait 档不豁免重试预算: 探针是**真实尝试**,失败照样烧 max_attempts。
|
||||
|
||||
故 force_open 的源(401/403/欠费一击即熔,不看任何阈值)在 wait 档下并
|
||||
**不是**"等满 stall 窗口才死"——两个预算哪个先耗尽就以哪个的 reason
|
||||
失败。这里 max_attempts=3 而冷却只累计 120s < stall_window=300s,故
|
||||
先到的是重试预算。参数换成"冷却累计超过 stall 预算"则先到 stalled
|
||||
(见 test_wait_still_dies_when_cooldown_outlasts_the_stall_budget)。
|
||||
|
||||
这与 issue #8 确立的划分一致: 划分依据是"谁消耗重试预算",探针发出了
|
||||
真实请求,理应记在重试预算上而不是 stall 账上。
|
||||
"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
sleep = BoundedSleep()
|
||||
|
||||
async def advance(_n):
|
||||
clock.advance(sleep.delays[-1])
|
||||
|
||||
sleep._side_effect = advance
|
||||
mw = _mw(
|
||||
[src],
|
||||
self._free_limiter(clock, src),
|
||||
[SourceDeadError("401"), SourceDeadError("401"), SourceDeadError("401")],
|
||||
clock=clock,
|
||||
sleep=sleep,
|
||||
circuit_open="wait",
|
||||
)
|
||||
with pytest.raises(AllSourcesExhausted) as ei:
|
||||
await mw(_REQ)
|
||||
assert ei.value.reason == "retry_exhausted"
|
||||
assert clock.t - 1000.0 < _STALL # 远未等满 stall 窗口
|
||||
|
||||
async def test_half_open_rejection_does_not_blacklist_a_recovered_source(self):
|
||||
"""issue #14 §1.3 回归: 探针成功后本进程立即可再选该源。
|
||||
|
||||
此前 HALF_OPEN 拒绝把探针租约(派生自 2 × timeout,现场 600s)写进冷却
|
||||
备忘,而 `set_until` 取更晚者、不可回退——门恢复 CLOSED 之后本进程仍
|
||||
跳过该源整整一个租约,单源下每次调用照旧判死。多源部署同样中招,只是
|
||||
被别的源接住流量掩盖了。
|
||||
"""
|
||||
clock = FakeClock()
|
||||
cfg = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=600.0)
|
||||
gate = await self._opened_gate(clock, cfg)
|
||||
memo = SourceCooldownMemo(now=clock)
|
||||
clock.advance(cfg.cooldown_s + 1)
|
||||
probe = await gate.try_enter("s1", "w1")
|
||||
blocked = await gate.try_enter("s1", "w2") # 并发调用撞上在途探针
|
||||
assert not blocked.allowed
|
||||
memo.set_until("s1", clock() + blocked.retry_after_s) # 准入路径的写法
|
||||
await gate.record_success(probe) # 探针成功 → 门恢复 CLOSED
|
||||
assert not memo.active("s1")
|
||||
|
||||
@@ -170,6 +170,18 @@ class TestResilienceKeys:
|
||||
with pytest.raises(ValueError, match="probe"):
|
||||
GatewaySettings.from_env("LLM", env=_env(**{"LLM__BREAKER__PROBE_TTL_S": "45"}))
|
||||
|
||||
def test_circuit_open_defaults_to_fail_fast(self):
|
||||
"""issue #14: 熔断拒绝的处置策略。
|
||||
|
||||
缺省**不跟随** quota_full 的 wait——把最坏墙钟从毫秒抬到 stall 窗口
|
||||
是"快速失败 → 长时间挂起"这个最危险的方向,不能强加给存量下游。
|
||||
"""
|
||||
assert GatewaySettings.from_env("LLM", env=_env()).circuit_open == "fail_fast"
|
||||
waiting = GatewaySettings.from_env("LLM", env=_env(**{"LLM__CIRCUIT_OPEN": "wait"}))
|
||||
assert waiting.circuit_open == "wait"
|
||||
with pytest.raises(ValueError, match="CIRCUIT_OPEN"):
|
||||
GatewaySettings.from_env("LLM", env=_env(**{"LLM__CIRCUIT_OPEN": "block"}))
|
||||
|
||||
def test_selector_and_quota_full(self):
|
||||
# M2.5: 缺省选源改 health_aware(生产级默认);显式配置者不变
|
||||
s = GatewaySettings.from_env("LLM", env=_env())
|
||||
@@ -589,6 +601,7 @@ class TestCrossFieldInvariants:
|
||||
("telemetry_backend", "redis"),
|
||||
("selector", "random"),
|
||||
("quota_full", "block"),
|
||||
("circuit_open", "block"),
|
||||
],
|
||||
)
|
||||
def test_enum_field_rejects_value_outside_domain(self, field, bad_value):
|
||||
|
||||
@@ -629,7 +629,7 @@ class TestDemotionInsertPosition:
|
||||
|
||||
async def test_demoted_lands_before_junk_sources(self):
|
||||
# a 失败 2 次;b 可信(0.9)但会被跳过时,第三候选应是 a 而非垃圾源 c
|
||||
from polygateway.middleware.retry import _demote_call_failures
|
||||
from polygateway.middleware.admission import _demote_call_failures
|
||||
|
||||
srcs = [_src("a"), _src("b"), _src("c")]
|
||||
health = {"a": 0.9, "b": 0.9, "c": 0.05}.__getitem__
|
||||
@@ -637,7 +637,7 @@ class TestDemotionInsertPosition:
|
||||
assert [s.name for s in out] == ["b", "a", "c"]
|
||||
|
||||
async def test_health_blind_demotion_still_tail(self):
|
||||
from polygateway.middleware.retry import _demote_call_failures
|
||||
from polygateway.middleware.admission import _demote_call_failures
|
||||
|
||||
srcs = [_src("a"), _src("b"), _src("c")]
|
||||
out = _demote_call_failures(srcs, {"a": 2}, None)
|
||||
|
||||
Reference in New Issue
Block a user