fix: admit governance backend failures into scope-level unavailability (issue #7)
A fail-closed limiter or breaker backend means the scope cannot emit a single request, yet GovernanceBackendError sat directly under PolyGatewayError. A caller writing only `except GatewayUnavailableError` dropped it into the catch-all branch, so a Redis blip burned a backlog's business failure budget into the dead letter queue over a fault a restart would clear. It now inherits GatewayUnavailableError with a governance_backend_down reason and a 5 second retry_after_s. The two unknown-source sites split out into SourceNotConfiguredError, deliberately outside the retryable family: a misconfigured source name must burn its budget and surface rather than retry forever in silence. README now states which errors reach callers and which the retry loop absorbs. TransientError and SourceDeadError read like caller contracts but never arrive, and a downstream project wrote a whole design section on that false premise before checking the source. Independent verification caught the split not actually holding on the only path production uses, and caught the fix for that opening a second hole on the accounting path. Both are fixed and pinned by tests that go through the wrappers rather than the private methods underneath them.
This commit is contained in:
@@ -1,5 +1,28 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 1.1.0(2026-08-06)
|
||||||
|
|
||||||
|
治理后端故障归位为 scope 级不可用(issue #7)。限流/熔断的状态后端(Redis 等)自身故障时,库按降级方向铁律 fail-closed——**整个 scope 一个请求都发不出去**,语义上就是"scope 级暂时不可用"。但 `GovernanceBackendError` 此前是 `PolyGatewayError` 的直接子类,只写 `except GatewayUnavailableError` 的调用方接不住,后果很具体: Redis 抖一下,积压任务一批批消耗业务失败预算,够到上限就进死信——**而那是运维重启一下就好的故障**。
|
||||||
|
|
||||||
|
### 行为变更(**请先读这一条**)
|
||||||
|
|
||||||
|
- **`GovernanceBackendError` 现在能被 `except GatewayUnavailableError` 捕获。** 它改为继承该类,`reason` 恒为新增的 `governance_backend_down`。**下游对后端故障的处置路线因此改变**: 从"落进兜底分支、按业务失败处置"变为"按 scope 级不可用延期重投、不消耗失败预算"。这正是本次修复的目标,但升级前请确认下游的兜底分支没有依赖旧行为(例如靠它触发告警)。既有的 `except GovernanceBackendError` **继续有效**——加父类是扩大捕获面,不是破坏。
|
||||||
|
- **配置写错(源名与限流后端配置不匹配)现在抛 `SourceNotConfiguredError` 而非 `GovernanceBackendError`。** 该类**有意不在** `GatewayUnavailableError` 之下: 那是装配缺陷不是暂时故障,必须消耗失败预算、进死信、让人看见。若随整类归入可重投家族,配置写错的任务会永远重投且无人告警——恰是本次要修的 bug 的镜像。
|
||||||
|
- **`GovernanceBackendError` 的构造签名增加必填 keyword `scope`。** 库内 20 处构造点已全部更新;若下游有自行构造该异常的代码(罕见)需同步补 `scope`。
|
||||||
|
|
||||||
|
### 新增
|
||||||
|
|
||||||
|
- **`SourceNotConfiguredError`**(公共导出)。源名不在限流后端配置字典中时抛出,正常不可达,属装配缺陷。
|
||||||
|
- **`GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0`**,`GovernanceBackendError.retry_after_s` 的默认值。**不是环境配置项**——后端恢复时间物理上不可知(不同于熔断冷却有确定到期时刻),故取保守固定值。**不取 0**: 那会让积压任务零延迟同时冲击已挂掉的后端,把一次故障放大成一场风暴。
|
||||||
|
- **scope 级 `reason` 值域增 `governance_backend_down`**(由 5 值扩为 6 值)。
|
||||||
|
- **README 新增"哪些异常会到达调用方"两列表**。四分类里 `TransientError` / `SourceDeadError` 被重试循环接住、耗尽时包成 `AllSourcesExhausted`,**根本到不了调用方**,而这只看类型树与 docstring 读不出来——曾让下游据此写错整段设计文档。
|
||||||
|
|
||||||
|
### 下游请读
|
||||||
|
|
||||||
|
- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`**,与 `AllSourcesExhausted` 同款(`per_source_reasons` 属性存在但恒为 `{}`——后端故障不针对具体某个源);`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。
|
||||||
|
- **五条闸门路径**的后端故障会到达调用方: `QuotaGate` 的 `try_acquire` / `stats` / `progress_age_s`,`BreakerGate` 的 `try_enter` / `retry_after_s`。记账路径(`record_success` / `record_failure` / `release_probe` / `mark_progress`)仍被 `_record_quietly` 降级为 warning,这个分工不变。
|
||||||
|
- **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。
|
||||||
|
|
||||||
## 1.0.6(2026-08-02)
|
## 1.0.6(2026-08-02)
|
||||||
|
|
||||||
推理开关能力建模与 `reasoning_tokens` 采集。`enable_thinking=False` 此前对 `minimax` / `openai` 两类源**完全不产生效果**——两个 profile 的 thinking 两档皆为空字典,`payload.update({})` 是空操作,而配置方以为关掉了推理。这比"不提供这个开关"更危险:不提供的话调用方会去找别的办法,提供了但静默失效,调用方就带着一个错误的前提往下走。一个下游项目正卡在这上面。
|
推理开关能力建模与 `reasoning_tokens` 采集。`enable_thinking=False` 此前对 `minimax` / `openai` 两类源**完全不产生效果**——两个 profile 的 thinking 两档皆为空字典,`payload.update({})` 是空操作,而配置方以为关掉了推理。这比"不提供这个开关"更危险:不提供的话调用方会去找别的办法,提供了但静默失效,调用方就带着一个错误的前提往下走。一个下游项目正卡在这上面。
|
||||||
|
|||||||
@@ -28,6 +28,12 @@ make ci # 只读验证(check + test)
|
|||||||
|
|
||||||
> **档位原则(Fable 5 适配,2026-07 调研决策)**: 约束"边界与验收",不规定思考步骤。强制档(MANDATORY)是硬门;其余由模型按 skill description 自判,自判标准是任务实质(规模/风险/是否触及公共承诺),不是省事。硬边界(reference/ 只读、危险命令、提交质量门)由 `.claude/settings.json` 注册的 hooks **确定性执行**,不依赖提示词自觉。
|
> **档位原则(Fable 5 适配,2026-07 调研决策)**: 约束"边界与验收",不规定思考步骤。强制档(MANDATORY)是硬门;其余由模型按 skill description 自判,自判标准是任务实质(规模/风险/是否触及公共承诺),不是省事。硬边界(reference/ 只读、危险命令、提交质量门)由 `.claude/settings.json` 注册的 hooks **确定性执行**,不依赖提示词自觉。
|
||||||
|
|
||||||
|
> [!CRITICAL]
|
||||||
|
> **执行模式: subagent 与 Codex 一律前台(2026-08-06 人类指令)**
|
||||||
|
> 一切 subagent(verifier、`subagent-driven-development` 执行器、Explore 等)与 Codex 调用**必须前台运行**——`Agent` 工具传 `run_in_background: false`,`/codex:rescue` 带 `--wait`,**禁止**后台派发后继续做别的事。
|
||||||
|
> **理由(实测教训)**: 后台完成通知不可靠——管道会掩盖真实退出码(`pytest ... | tail` 让失败跑报成 exit 0),等待脚本的 `pgrep -f` 会自匹配成死循环,于是出现"任务早完成却没人知道"和"任务挂了也没人知道"两种失败,且两种都以"看起来还在跑"的形态呈现,无法从外部区分。前台运行牺牲并行度换取状态确定性,这个交换在本项目是划算的。
|
||||||
|
> **同一理由适用于长跑命令**: 需要后台跑时(如全套件测试),命令末尾**不得接管道**,否则退出码失真;要判完成用 `wait`/轮询 PID,不要用会匹配到自身的 `pgrep -f "<完整命令串>"`。
|
||||||
|
|
||||||
### Phase 1: 规划与设计
|
### Phase 1: 规划与设计
|
||||||
1. 涉及**公共 API、端口签名、架构边界、新子系统**的变更**必须**调用 `brainstorming`(产出 2-3 备选方案+权衡)并经**人类确认**后实施;其余任务自判(判据: 是否改变库对下游的承诺)。动手前查阅 `research-wiki/`(单一事实源)。
|
1. 涉及**公共 API、端口签名、架构边界、新子系统**的变更**必须**调用 `brainstorming`(产出 2-3 备选方案+权衡)并经**人类确认**后实施;其余任务自判(判据: 是否改变库对下游的承诺)。动手前查阅 `research-wiki/`(单一事实源)。
|
||||||
2. 功能产生运行时数据时**必须**调用 `structured-logging`。
|
2. 功能产生运行时数据时**必须**调用 `structured-logging`。
|
||||||
|
|||||||
@@ -124,6 +124,21 @@ except RequestRejectedError:
|
|||||||
|
|
||||||
预算耗尽/全源熔断时抛 `GatewayUnavailableError` 族(`CircuitOpenError` / `AllSourcesExhausted`),携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`,供任务队列做延期重投。
|
预算耗尽/全源熔断时抛 `GatewayUnavailableError` 族(`CircuitOpenError` / `AllSourcesExhausted`),携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`,供任务队列做延期重投。
|
||||||
|
|
||||||
|
### 哪些异常会到达调用方
|
||||||
|
|
||||||
|
上表的"库内行为"一列描述的是**治理动作**,不是调用方要处理的东西。四类里有两类**根本到不了调用方**——它们被重试循环接住,预算耗尽时统一包成 `AllSourcesExhausted`。这个区分只看类型树和 docstring 是读不出来的,曾让下游据此写错整段设计文档,故在此列明:
|
||||||
|
|
||||||
|
| 会到达调用方 | 库内吸收(不必 catch) |
|
||||||
|
|---|---|
|
||||||
|
| `GatewayUnavailableError` 族——`CircuitOpenError` / `AllSourcesExhausted` / `GovernanceBackendError` | `TransientError`(退避后换源重试,耗尽即转为 `AllSourcesExhausted`) |
|
||||||
|
| `RequestRejectedError` | `SourceDeadError`(立即熔断该源并换源,同上) |
|
||||||
|
| `ResultInvalidError` | |
|
||||||
|
| `SourceNotConfiguredError` | |
|
||||||
|
|
||||||
|
**`GovernanceBackendError` 属于第一列**: 限流/熔断的状态后端(如 Redis)自身故障时库 fail-closed——一个请求都发不出去,这就是"整个 scope 暂时不可用"。它继承 `GatewayUnavailableError`,所以 §4 那段 `except GatewayUnavailableError` 一条即覆盖完整,无需为它单列分支。`retry_after_s` 默认 5 秒(后端恢复时间不可知,取 0 会让积压任务零延迟冲击已挂掉的后端)。
|
||||||
|
|
||||||
|
**`SourceNotConfiguredError` 有意不在第一列的族内**: 源名不在限流后端的配置字典中是**装配缺陷**而非暂时故障,它应当消耗失败预算、进死信、让人看见——归入可重投家族只会让配置写错的任务永远重投且无人告警。
|
||||||
|
|
||||||
## 配置参考
|
## 配置参考
|
||||||
|
|
||||||
配置只有两条装配路径:`from_env()`(读 `.env`/环境变量)或构造函数全量注入(测试/高级);库内部任何组件不自读环境变量。键名全集见 [.env.example](.env.example),约定速览:
|
配置只有两条装配路径:`from_env()`(读 `.env`/环境变量)或构造函数全量注入(测试/高级);库内部任何组件不自读环境变量。键名全集见 [.env.example](.env.example),约定速览:
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "polygateway"
|
name = "polygateway"
|
||||||
version = "1.0.6"
|
version = "1.1.0"
|
||||||
description = "PolyGateway：实验室统一的大语言模型（LLM/VLM/OCR）调度与中转库——多源、限流、重试、熔断、缓存、遥测"
|
description = "PolyGateway：实验室统一的大语言模型（LLM/VLM/OCR）调度与中转库——多源、限流、重试、熔断、缓存、遥测"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
@@ -376,8 +376,14 @@ flowchart TB
|
|||||||
| `RequestRejectedError` | 400/请求格式错/坏输入(如不支持的图像格式) | ❌ | ❌ | ❌ |
|
| `RequestRejectedError` | 400/请求格式错/坏输入(如不支持的图像格式) | ❌ | ❌ | ❌ |
|
||||||
| `ResultInvalidError` | 调用成功但内容不可解析(JSON 修不好、ZIP 缺关键文件) | ❌(仅 D14 结构化阶梯的有界带反馈重问,不入 transport 重试计数) | ❌ | ❌(熔断记**成功**) |
|
| `ResultInvalidError` | 调用成功但内容不可解析(JSON 修不好、ZIP 缺关键文件) | ❌(仅 D14 结构化阶梯的有界带反馈重问,不入 transport 重试计数) | ❌ | ❌(熔断记**成功**) |
|
||||||
| `CircuitOpenError` / `AllSourcesExhausted` | 开路 / 全源耗尽 | 调用方决定: wait / fail-fast 可配 | — | — |
|
| `CircuitOpenError` / `AllSourcesExhausted` | 开路 / 全源耗尽 | 调用方决定: wait / fail-fast 可配 | — | — |
|
||||||
|
| `GovernanceBackendError` | 限流/熔断**状态后端自身**故障(Redis 挂等);降级方向 fail-closed,故一个请求都发不出去 | 调用方决定(同 scope 级: 延期重投) | — | — |
|
||||||
|
| `SourceNotConfiguredError` | 源名不在限流后端配置字典中——**装配缺陷**,非调用失败,正常不可达 | ❌ | ❌ | ❌ |
|
||||||
|
|
||||||
**scope 级不可用的结构化语义(2026-07-20,CHS 迁移缺口 G1;2026-07-20 M1 设计勘误修订)**: `AllSourcesExhausted`/`CircuitOpenError` 必须携带结构化字段——`retry_after_s: float`(**非可选**,承 CHS `ProviderUnavailableError` 同款,0 表示可立即重试;取各源冷却与 Retry-After 的最小值)、`reason` 枚举、`per_source_reasons: dict[str, str]`。reason 两层值域(M1 设计 §3 勘误: 本节初版所列 7 值与 CHS `errors.py:143-153` 实际值域不符,重组如下)——scope 级 `reason`: circuit_open / retry_exhausted / stalled / quota_exhausted / no_sources;`per_source_reasons` 值: network_error / timeout / rate_limited / source_dead / circuit_open / cooldown。CHS 的"scope 级不可用 → arq 延期重投、不消耗业务失败预算"(`workers/tracking.py:406-428`)依赖 `retry_after_s` 复现。
|
**scope 级不可用的结构化语义(2026-07-20,CHS 迁移缺口 G1;2026-07-20 M1 设计勘误修订)**: `AllSourcesExhausted`/`CircuitOpenError` 必须携带结构化字段——`retry_after_s: float`(**非可选**,承 CHS `ProviderUnavailableError` 同款,0 表示可立即重试;取各源冷却与 Retry-After 的最小值)、`reason` 枚举、`per_source_reasons: dict[str, str]`。reason 两层值域(M1 设计 §3 勘误: 本节初版所列 7 值与 CHS `errors.py:143-153` 实际值域不符,重组如下)——scope 级 `reason`: circuit_open / retry_exhausted / stalled / quota_exhausted / no_sources / **governance_backend_down**(2026-08-06 增,见下);`per_source_reasons` 值: network_error / timeout / rate_limited / source_dead / circuit_open / cooldown。CHS 的"scope 级不可用 → arq 延期重投、不消耗业务失败预算"(`workers/tracking.py:406-428`)依赖 `retry_after_s` 复现。
|
||||||
|
|
||||||
|
**治理后端故障归位(2026-08-06,Gitea issue #7;设计 `designs/2026-08-06-governance-backend-error-design.md`)**: `GovernanceBackendError` 自 M2 引入分布式后端时新增,但**当时未回补本表**,于是它在"调用方视角的分类学"里一直没有位置——本次归位同时补上这个遗漏。它此前是 `PolyGatewayError` 的直接子类,而语义上 fail-closed 意味着整个 scope 发不出任何请求,正是 scope 级不可用;下游只写 `except GatewayUnavailableError` 会把它落进兜底分支,导致"Redis 抖一下 → 积压任务消耗业务失败预算 → 进死信",而那是运维重启即可恢复的故障。现改为继承 `GatewayUnavailableError`,`reason` 恒为 `governance_backend_down`,`retry_after_s` 默认取常量 `GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0`——**不取 0**,因为后端恢复时间物理上不可知(不同于熔断冷却有确定到期时刻),而 0 会让积压任务零延迟同时冲击已挂掉的后端。
|
||||||
|
|
||||||
|
同批拆出 `SourceNotConfiguredError`: 限流后端 `_cfg()` 遇到源名不在配置字典中时原先也抛 `GovernanceBackendError`,但那是装配缺陷而非后端故障。若随整类归入"可延期重投",配置写错的任务会**永远重投、永不进死信、无人告警**——恰是本次要修的 bug 的镜像。故它有意留在 `GatewayUnavailableError` 之外,让缺陷消耗失败预算并浮出水面。它与四分类的关系见 §6.3 之外的第三论域说明: 四分类的论域是 transport 层翻译的**调用失败**(§6.2),scope 级不可用回答"整个 scope 还能不能用",而装配缺陷根本不该进入治理循环被"决定"。
|
||||||
|
|
||||||
### 6.2 翻译规则(transport 层职责)
|
### 6.2 翻译规则(transport 层职责)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,181 @@
|
|||||||
|
# 治理后端故障归位为 scope 级不可用设计(Issue #7)
|
||||||
|
|
||||||
|
- **日期**: 2026-08-06
|
||||||
|
- **来源**: Gitea Issue #7(下游 CHSAnalyzer3 按异常类型分流失败,基于 1.0.1 源码核查)
|
||||||
|
- **状态**: **已批准(2026-08-06)**,待 `writing-plans`
|
||||||
|
- **触发档位**: 强制(变更 `errors.py` 公共错误类型树 = 库对下游的承诺)
|
||||||
|
- **方案范围**: 人类已选定方向 A′ 并明确要求单一方案,故本文不列平行备选,仅在 §4 记录被否决路线及否决理由
|
||||||
|
|
||||||
|
## 1. 目标与非目标
|
||||||
|
|
||||||
|
| | 内容 |
|
||||||
|
|---|---|
|
||||||
|
| **G1** | `GovernanceBackendError` 归入 `GatewayUnavailableError` 之下,使"该延期重投的失败"在类型上闭合——调用方一条 `except GatewayUnavailableError` 覆盖完整,漏接在物理上不可能 |
|
||||||
|
| **G2** | 把混在同一类里的**装配期缺陷**("未知源")拆出去,使其**不**被误判为可重投 |
|
||||||
|
| **G3** | `retry_after_s` 取非零值,避免后端故障期间下游零延迟批量重投形成忙循环 |
|
||||||
|
| **G4** | 公开错误面文档化:README 增"会到达调用方 / 库内吸收"两列表,`ARCHITECTURE.md` §6.1 回补缺失的 `GovernanceBackendError` 行 |
|
||||||
|
| **非目标** | 不改 fail-closed 降级方向(限流/熔断后端不可用 → 报错而非放行,库铁律不动);不改后端重连/健康探测;不新增配置项;不改 `TransientError`/`SourceDeadError` 的库内吸收行为 |
|
||||||
|
|
||||||
|
### 1.1 Issue 前提的四处修正(按 1.0.6 源码核实)
|
||||||
|
|
||||||
|
| Issue 原文 | 实际情况 |
|
||||||
|
|---|---|
|
||||||
|
| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **五条**(设计初稿写"三条",2026-08-06 独立验证时核出遗漏两条并订正): `QuotaGate` 的 `try_acquire` / `stats`(`retry.py:249`)/ `progress_age_s`(`retry.py:216`、`:305`),`BreakerGate` 的 `try_enter` / `retry_after_s`(`retry.py:292`、`:310`)。判据是该调用点是否被 `_record_quietly` 包裹——未包裹即直达调用方;OCR 与 Embedding 两个治理循环有同构的对应点 |
|
||||||
|
| (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 |
|
||||||
|
| 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug |
|
||||||
|
| `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 |
|
||||||
|
|
||||||
|
另需记录一处根因:`ARCHITECTURE.md:372-378` §6.1 的错误分类表里 `GovernanceBackendError` **一次都没出现**。它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来就没有位置——README 的遗漏是这个遗漏的下游后果。
|
||||||
|
|
||||||
|
## 2. 影响面的决定性前提(改动安全性的依据)
|
||||||
|
|
||||||
|
| 事实 | 证据 | 含义 |
|
||||||
|
|---|---|---|
|
||||||
|
| 库内仅一处 `except GatewayUnavailableError` | `middleware/telemetry.py:250`,写法为 `except (GatewayUnavailableError, GovernanceBackendError)` | 变成父子关系后该处由"并列捕获"退化为"父类捕获",**行为逐字不变**,库内零回归 |
|
||||||
|
| 加父类是纯扩大 | 下游既有 `except GovernanceBackendError` 全部照旧命中 | 不违反 CLAUDE.md §4.3「已被下游消费的公共类型只增不删不改名」 |
|
||||||
|
| `QuotaGate`/`BreakerGate` 是后端异常的唯一入口 | 两类 docstring 自述,三处装配 `retry.py:186` / `ocr.py:122` / `embedding.py:123` | scope 注入点收敛为 2 个类、3 处装配 |
|
||||||
|
| 三个装配点都持有 `self._scope` | `retry.py:183`、`ocr.py:116`、`embedding.py:118` | 注入无需新增上游参数传递链 |
|
||||||
|
|
||||||
|
## 3. 选定方案
|
||||||
|
|
||||||
|
### 3.1 类型树变更
|
||||||
|
|
||||||
|
`SCOPE_REASONS` 增枚举值 `governance_backend_down`;`GovernanceBackendError` 改继承 `GatewayUnavailableError`,`reason` 恒为该值(与 `CircuitOpenError` 恒为 `circuit_open` 同构,是本库已有的表达手法)。
|
||||||
|
|
||||||
|
构造签名保持"首参为 message"的位置参数形态,以免 22 处构造点与既有测试全部改写:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class GovernanceBackendError(GatewayUnavailableError):
|
||||||
|
def __init__(self, message, *, scope, retry_after_s=GOVERNANCE_BACKEND_RETRY_AFTER_S,
|
||||||
|
source_name=None):
|
||||||
|
super().__init__(scope=scope, reason="governance_backend_down",
|
||||||
|
retry_after_s=retry_after_s, source_name=source_name)
|
||||||
|
self.args = (message,) # 见 §3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
`scope` 为必填 keyword(P4 显式优于隐式:它在三层调用点全部可得,给默认值只会掩盖装配疏漏)。
|
||||||
|
|
||||||
|
### 3.2 `retry_after_s` 的取值(本设计的核心权衡)
|
||||||
|
|
||||||
|
Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重投,Redis 挂掉期间队列里积压的任务将以零延迟批量重投,对着一个已经挂掉的后端打忙循环——把一次故障放大成一场风暴。这与本 issue 想修的问题同源:都是"分类正确但处置参数错误"。
|
||||||
|
|
||||||
|
已考虑并否决的两个替代取值:
|
||||||
|
|
||||||
|
| 取值 | 否决理由 |
|
||||||
|
|---|---|
|
||||||
|
| 复用 `BackpressureConfig.poll_interval_s`(与 `quota_exhausted` 同源,`retry.py:299` 有先例) | 该值只有三个装配点持有,后端层 11 处构造点拿不到;为此给 `RedisLimiter`/`RedisBreaker` 增构造参数,是让后端层去持有"重投策略"——违反 P7(决策逻辑与状态存储分离),后端只该知道"我坏了",不该知道这在治理上意味着什么 |
|
||||||
|
| 新增配置项 `PGW_GOVERNANCE_BACKEND_RETRY_AFTER_S` | YAGNI。目前无任何下游表达过需要调它;真需要时下游可完全忽略 `exc.retry_after_s` 用自有退避 |
|
||||||
|
|
||||||
|
**选定**:`errors.py` 模块级常量 `GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0`,作为构造默认值,docstring 写明理由——后端恢复时间物理上不可知(不同于熔断冷却有确定到期时刻),取一个保守固定值;下游若有自己的退避策略可忽略此值。本库对 scope 级异常硬编码语义值已有先例(`retry.py:206` 的 `no_sources` 取 `0.0`)。
|
||||||
|
|
||||||
|
它**不是环境配置项**,故不落 CLAUDE.md §4.5「严禁硬编码默认值」的论域——§4.5 约束的是 `pydantic-settings` + `.env` 管辖的工程配置(超时、并发、限额),而本常量是异常自身携带的语义默认值,与 `no_sources` 取 `0.0` 同性质。docstring 需显式写明这一点,避免后来者误加环境键。
|
||||||
|
|
||||||
|
### 3.3 `scope` 的三层来源
|
||||||
|
|
||||||
|
| 层 | 构造点数 | scope 来源 | 改动 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `backends/redis/limiter.py` | 6 | `self._scope`(`:170`) | 补 `scope=self._scope` |
|
||||||
|
| `backends/redis/breaker.py` | 5 | `self._scope`(`:291`) | 补 `scope=self._scope` |
|
||||||
|
| `middleware/breaker.py` `BreakerGate` | 5 | **需注入** | 构造函数增 `scope: str`,三处装配传 `self._scope` |
|
||||||
|
| `middleware/ratelimit.py` `QuotaGate` | 4 | **需注入** | 同上 |
|
||||||
|
|
||||||
|
包装器对后端自抛异常的 `except GovernanceBackendError: raise` 原样放行**保持不变**——后端层已填好 scope,重建实例只会制造"同一异常构造两次"的怪味且覆盖值相同。
|
||||||
|
|
||||||
|
### 3.4 "未知源"拆分为独立错误类
|
||||||
|
|
||||||
|
`backends/memory/limiter.py:92` 与 `backends/redis/limiter.py:198` 的 `_cfg()` 在源名不在配置字典中时抛 `GovernanceBackendError`。**这不是后端故障**,是限流后端拿到的源列表与治理循环的对不上——装配期缺陷,正常不可达。
|
||||||
|
|
||||||
|
若随整类归入"延期重投、不扣失败预算",配置写错的任务将**永远重投、永远不进死信**,运维永远收不到告警——正是本 issue 要修的 bug 的镜像。
|
||||||
|
|
||||||
|
新增 `SourceNotConfiguredError(PolyGatewayError)`,**有意不放在** `GatewayUnavailableError` 之下:下游默认按"任务的错"处置 → 扣失败预算 → 进死信 → 人能看见。这是缺陷该有的可见性。该类进 `__init__.py` 公共导出(下游可选择性识别,但不识别也能得到正确处置)。
|
||||||
|
|
||||||
|
### 3.5 message 保全
|
||||||
|
|
||||||
|
`GatewayUnavailableError.__init__` 会把 message 覆盖为 `f"{scope} 网关暂时不可用: {reason}"`,而 22 处构造点携带的诊断串(如 `限流后端 try_acquire 失败: {exc}`)是排障的主要线索,不可丢。方案是 `super().__init__()` 后覆写 `self.args = (message,)`,使 `str(exc)` 仍为原诊断串,而 `scope`/`reason`/`retry_after_s` 作为结构化字段并存。父类不动——它的 message 生成逻辑对 `CircuitOpenError`/`AllSourcesExhausted` 仍然正确。
|
||||||
|
|
||||||
|
## 4. 被否决的路线
|
||||||
|
|
||||||
|
| 路线 | 否决理由 |
|
||||||
|
|---|---|
|
||||||
|
| **B: 只补文档,类型树不动** | 正确性依赖每个下游都读到那句话。本库下游不止一个,且本 issue 本身就是"文档读不出来"引发的——同一个失效模式不能用同一种药治 |
|
||||||
|
| **C: 类型树不动,在 RetryMW 边界包成 `AllSourcesExhausted`** | 比 A′ 更具破坏性:下游现有 `except GovernanceBackendError` 会直接失效。加父类是扩大,换类型是破坏 |
|
||||||
|
| **D: 后端层不再构造该异常,原始异常穿透由包装器统一翻译**(初评时倾向,已否决) | `backends/redis/limiter.py:133,151` 的 `RedisPermit.release/settle` 依赖 `except GovernanceBackendError` 实现**释放侧降级**(失败只 warning 不冒泡)。原始 redis 异常穿透后该处接不住,会破坏这条既有降级行为;改为 `except Exception` 则违反 P5 |
|
||||||
|
|
||||||
|
## 5. 行为审计(既有行为逐条标注)
|
||||||
|
|
||||||
|
| 既有行为 | 出处 | 处置 |
|
||||||
|
|---|---|---|
|
||||||
|
| 限流/熔断后端不可用 → 报错而非放行(fail-closed) | 库铁律 | **保留**,一字不改 |
|
||||||
|
| 记账路径后端故障降级为 warning | `middleware/retry.py:404` `_record_quietly` | **保留**。仅闸门路径需要到达调用方 |
|
||||||
|
| permit `release`/`settle` 失败降级 warning | `redis/limiter.py:133,151` | **保留**(§4 路线 D 因此被否决) |
|
||||||
|
| 遥测对后端故障发 `emit_terminal_failure` | `middleware/telemetry.py:250` | **保留**,父子关系后由父类分支承接,行为不变 |
|
||||||
|
| `except GovernanceBackendError: raise` 原样放行 | 包装器 9 处 | **保留** |
|
||||||
|
| "未知源"抛 `GovernanceBackendError` | `memory/limiter.py:92`、`redis/limiter.py:198` | **替换**为 `SourceNotConfiguredError`(§3.4) |
|
||||||
|
| `str(exc)` 为诊断串 | 22 处 | **保留**(§3.5 显式保全) |
|
||||||
|
|
||||||
|
## 6. 非功能维度
|
||||||
|
|
||||||
|
| 维度 | 回答 |
|
||||||
|
|---|---|
|
||||||
|
| **并发与取消** | 不适用于新增并发路径。异常构造是纯同步无状态操作,不引入共享状态。`CancelledError` 穿透路径完全不受影响——本设计不新增任何 `except` 子句,`_record_quietly` 中 `except asyncio.CancelledError`(`:402`)先于 `except GovernanceBackendError`(`:404`)的顺序不动 |
|
||||||
|
| **降级方向** | 不变。fail-closed 是本类存在的理由,本设计只改"它被归入哪一类",不改"它是否被抛出" |
|
||||||
|
| **幂等与重复** | 异常类型变更不涉及幂等性。需注意的是下游行为改变:同一次后端故障从"扣失败预算"变为"延期重投",重投次数由下游队列策略决定——这正是期望的变更,已在 CHANGELOG 行为变更段声明 |
|
||||||
|
| **持久化与原子性** | 无持久化改动。遥测落库路径(`emit_terminal_failure`)的字段与调用时机均不变 |
|
||||||
|
|
||||||
|
## 7. 错误处理与测试策略
|
||||||
|
|
||||||
|
新失败面只有一个:`SourceNotConfiguredError`,它落在四分类之外。这**不违反** CLAUDE.md §4.2「一切失败必须落入四分类」——该铁律的论域是 **transport 层翻译的调用失败**(`ARCHITECTURE.md` §6.2 的翻译规则表逐条对应 HTTP 状态码与解析失败),而本库已有一整族异常合法地处在四分类之外:`GatewayUnavailableError` / `CircuitOpenError` / `AllSourcesExhausted` 都不是四分类之一,`ARCHITECTURE.md` §6.1 把它们单列一行,因为它们回答的是另一个问题——"整个 scope 还能不能用",而非"这一次调用怎么失败的"。
|
||||||
|
|
||||||
|
`SourceNotConfiguredError` 属于第三个论域:**装配缺陷**(配置与治理循环不一致,正常不可达)。四分类决定重试/换源/熔断,而装配缺陷根本不该进入治理循环去被"决定",它应当立刻失败并让人看见。将其塞进四分类中的任何一类都会赋予它一份不该有的治理语义(如 `RequestRejectedError` 会让下游以为请求本身有问题、去修请求)。§9 Q1 保留了"复用 `RequestRejectedError`"作为备选供人类权衡。
|
||||||
|
|
||||||
|
| 测试 | 位置 | 先失败后通过的证据 |
|
||||||
|
|---|---|---|
|
||||||
|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
|
||||||
|
| 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
|
||||||
|
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
|
||||||
|
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
|
||||||
|
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" |
|
||||||
|
|
||||||
|
## 8. 影响面清单
|
||||||
|
|
||||||
|
| 类别 | 内容 |
|
||||||
|
|---|---|
|
||||||
|
| **源码** | `errors.py`(新常量+新类+继承变更)、`backends/redis/limiter.py`(7)、`backends/redis/breaker.py`(5)、`backends/memory/limiter.py`(1)、`middleware/breaker.py`(6:构造函数+5 处)、`middleware/ratelimit.py`(5)、`middleware/retry.py`/`ocr.py`/`embedding.py`(各 1 行装配)、`__init__.py`(导出新类) |
|
||||||
|
| **测试** | `tests/unit/test_errors.py`、`test_backpressure.py`、`test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py` |
|
||||||
|
| **文档** | `README.md` §"错误模型"增两列表 + `GovernanceBackendError` 行;`ARCHITECTURE.md` §6.1 回补该类并记录本次归位;`migrations/chsanalyzer.md` G1 条目补注;`CHANGELOG.md` 1.1.0;按 `docs-convention.md` §2 同步 Gitea Wiki |
|
||||||
|
| **版本** | **1.1.0**。有行为变更(下游对后端故障的处置路线改变)但无 API 破坏(加父类是扩大),按语义化版本走 minor |
|
||||||
|
| **下游** | CHSAnalyzer3 当前在 1.0.1。升级后 `except GatewayUnavailableError` 即覆盖后端故障,其现有 `except GovernanceBackendError`(若有)继续有效,无需改代码即可获得修复 |
|
||||||
|
|
||||||
|
### 8.1 执行顺序(单一事实源纪律)
|
||||||
|
|
||||||
|
`ARCHITECTURE.md` 是架构单一事实源,`SCOPE_REASONS` 新增值域与 `GovernanceBackendError` 的归位都与其 §6.1 现状冲突。因此 **§6.1 的修订必须先于或同批于代码实现落地**,不得"先改代码、事后补文档"。具体为:人类批准本设计后,`writing-plans` 的第一项任务即为修订 `ARCHITECTURE.md` §6.1(补 `GovernanceBackendError` 与 `SourceNotConfiguredError` 行、scope 级 reason 值域增 `governance_backend_down`、记录本次归位的理由与日期),与实现同一分支、同批提交。
|
||||||
|
|
||||||
|
## 9. 待人类确认的决策点
|
||||||
|
|
||||||
|
(编号用 Q 前缀,避免与 `ARCHITECTURE.md` 的架构决策 D1–D14 混淆)
|
||||||
|
|
||||||
|
**三点均已由人类拍板(2026-08-06),全部采纳本文的选择:**
|
||||||
|
|
||||||
|
| # | 决策 | 裁定 | 被否决的备选及理由 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Q1 | "未知源"归到哪 | ✅ **拆为 `SourceNotConfiguredError`**,不在 `GatewayUnavailableError` 之下(§3.4) | ① 沿用 `GovernanceBackendError`——配置写错的任务将无限重投、永不进死信、无人发现;② 复用 `RequestRejectedError`——治理行为与选定方案**完全等价**,但名称误导:下游会去查 prompt 而非配置文件 |
|
||||||
|
| Q2 | `retry_after_s` 取值 | ✅ **常量 `5.0`**(§3.2) | 取 0 会让积压任务零延迟同时冲击已挂掉的后端,把一次故障放大成风暴 |
|
||||||
|
| Q3 | 新类是否公共导出 | ✅ **导出**(进 `__init__.py`) | 不导出则下游无法给"配置写错"单独接告警,而导出无成本 |
|
||||||
|
|
||||||
|
## 10. 审批记录
|
||||||
|
|
||||||
|
| 阶段 | 状态 |
|
||||||
|
|---|---|
|
||||||
|
| Claude 自审 | 已完成(全部结论对应本会话内 grep/read 输出;§3.5 的 `self.args` 保全机制经 conda 环境实跑验证) |
|
||||||
|
| Codex 独立审 | 已完成(2026-08-06),4 条意见逐条核验见下 |
|
||||||
|
| 人类审批 | ✅ **已批准(2026-08-06)**。方向 A′ 于设计前即由人类选定;Q1–Q3 三个决策点逐条拍板,全部采纳本文选择(见 §9)。可进入 `writing-plans` |
|
||||||
|
|
||||||
|
### 10.1 Codex 意见的核验结果
|
||||||
|
|
||||||
|
| 意见 | 判定 | 处置 |
|
||||||
|
|---|---|---|
|
||||||
|
| ARCHITECTURE §6.1 未同步前实施违反单一事实源(判为阻塞) | **实质成立**,但性质是执行顺序而非设计缺陷——§8 本已把 §6.1 回补列入影响面 | 新增 §8.1 明确"架构文档修订先于/同批于实现" |
|
||||||
|
| §6.1 错误分类表未承认 `GovernanceBackendError`(判为阻塞) | **与上条同源**,且 §1.1 已自陈此为根因 | 同上,由 §8.1 覆盖 |
|
||||||
|
| `SourceNotConfiguredError` 落在四分类外违反 §4.2 铁律(判为阻塞) | **部分成立**:铁律论域被误读——`GatewayUnavailableError` 族本就合法处在四分类之外(§6.1 单列一行)。但原文表述确会引起该疑虑 | §7 补写三个论域的划分论证;§9 Q1 增列"复用 `RequestRejectedError`"备选交人类权衡 |
|
||||||
|
| 硬编码常量与 §4.5 存在张力(建议性) | **成立** | §3.2 补写"非环境配置项"及 docstring 要求 |
|
||||||
|
| Q 编号与架构 D1–D14 混淆(建议性) | **成立** | §9 决策点编号由 `D` 改为 `Q` |
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:governance-backend-error
|
||||||
|
title: "治理后端故障归位为 scope 级不可用(Issue #7)"
|
||||||
|
date: 2026-08-06
|
||||||
|
---
|
||||||
|
|
||||||
|
# 治理后端故障归位为 scope 级不可用(Issue #7)
|
||||||
|
|
||||||
|
全文见 `2026-08-06-governance-backend-error-design.md`。来源: Gitea Issue #7(下游 CHSAnalyzer3 按异常类型分流失败)。**状态: 已批准(2026-08-06,人类逐条拍板 Q1/Q2/Q3),待 `writing-plans`。**
|
||||||
|
|
||||||
|
问题: 限流/熔断状态后端故障时库 fail-closed,一个请求都发不出去——语义上就是 scope 级不可用,但 `GovernanceBackendError` 是 `PolyGatewayError` 的**直接子类**,只写 `except GatewayUnavailableError` 的调用方接不住,于是 Redis 抖一下,积压任务一批批消耗业务失败预算进死信,而那是运维重启就好的故障。
|
||||||
|
|
||||||
|
## 选定方案
|
||||||
|
|
||||||
|
| 决策 | 选定 | 关键理由 |
|
||||||
|
|---|---|---|
|
||||||
|
| A 类型树 | `GovernanceBackendError` 改继承 `GatewayUnavailableError`,`SCOPE_REASONS` 增 `governance_backend_down`,`reason` 恒为该值 | 加父类是**扩大**不是破坏(既有 `except GovernanceBackendError` 照旧命中);库内仅 `telemetry.py:250` 一处捕父类且已并列写两者,**零回归** |
|
||||||
|
| B `retry_after_s` | 模块常量 `GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0`,非环境配置项 | 后端恢复时间物理上不可知(不同于熔断冷却有确定到期时刻);取 0 会让积压任务零延迟批量重投,把一次故障放大成风暴 |
|
||||||
|
| C scope 来源 | 后端层用 `self._scope`;`QuotaGate`/`BreakerGate` 构造函数注入,三处装配(`retry.py`/`ocr.py`/`embedding.py`)各传一行 | 两个包装器是后端异常的唯一入口,注入点收敛;三处装配本就持有 `self._scope` |
|
||||||
|
| D 未知源拆分 | `_cfg()` 的 2 处改抛新增的 `SourceNotConfiguredError`,**有意不放在** `GatewayUnavailableError` 之下 | 那是装配缺陷不是后端故障;随整类归入"可重投"会让配置写错的任务永远重投、永不进死信——本 issue 要修的 bug 的镜像 |
|
||||||
|
| E message 保全 | `super().__init__()` 后覆写 `self.args = (message,)` | 父类会把 message 覆盖为 `f"{scope} 网关暂时不可用: {reason}"`,而 22 处构造点的诊断串是排障主线索。机制已实跑验证 |
|
||||||
|
|
||||||
|
## 被否决的备选
|
||||||
|
|
||||||
|
| 备选 | 否决原因 |
|
||||||
|
|---|---|
|
||||||
|
| B(issue 原议): 只补文档,类型树不动 | 正确性依赖每个下游都读到那句话;本 issue 本身就是"文档读不出来"引发的,同一失效模式不能用同一种药治 |
|
||||||
|
| C: 在 RetryMW 边界包成 `AllSourcesExhausted` | 比选定方案更具破坏性——下游现有 `except GovernanceBackendError` 直接失效 |
|
||||||
|
| D: 后端层不再构造该异常,原始异常穿透由包装器统一翻译 | 初评时倾向。`redis/limiter.py:133,151` 的 `RedisPermit.release/settle` 依赖 `except GovernanceBackendError` 实现**释放侧降级**,穿透后接不住会破坏该既有行为;改 `except Exception` 则违反 P5 |
|
||||||
|
| `retry_after_s` 复用 `BackpressureConfig.poll_interval_s` | 该值只有三个装配点持有,为此给后端加构造参数等于让状态存储层持有重投策略,违反 P7 |
|
||||||
|
| 新增配置项 `PGW_GOVERNANCE_BACKEND_RETRY_AFTER_S` | YAGNI;无下游表达过需要,真需要时下游可忽略该字段用自有退避 |
|
||||||
|
|
||||||
|
## 对 issue 前提的四处修正
|
||||||
|
|
||||||
|
泄漏路径是**五条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
|
||||||
|
|
||||||
|
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
|
||||||
|
|
||||||
|
## 独立审查修正(2026-08-06, Codex)
|
||||||
|
|
||||||
|
4 条意见逐条核验: 两条"架构文档未同步"实质成立但性质是执行顺序 → 新增 §8.1 钉死"`ARCHITECTURE.md` §6.1 修订先于/同批于实现";"新错误类违反四分类铁律"**部分成立**——铁律论域被误读(`GatewayUnavailableError` 族本就合法处在四分类之外),但原表述确会引起疑虑 → §7 补写三论域划分论证,并把"复用 `RequestRejectedError`"增列为待人类权衡的备选;两条建议性意见(常量非配置项的说明、决策编号 `D`→`Q` 防与架构 D1–D14 混淆)已采纳。
|
||||||
|
|
||||||
|
相关: [[m2-distributed]]、[[m1-core-design]]、[[m25-resilience]]
|
||||||
@@ -130,6 +130,16 @@
|
|||||||
"id": "plan:sampling-params-plan",
|
"id": "plan:sampling-params-plan",
|
||||||
"label": "采样参数透传实现计划(issue #4)",
|
"label": "采样参数透传实现计划(issue #4)",
|
||||||
"type": "plan"
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:governance-backend-error",
|
||||||
|
"label": "治理后端故障归位为 scope 级不可用(Issue #7)",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:governance-backend-error",
|
||||||
|
"label": "实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)",
|
||||||
|
"type": "plan"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"links": [
|
"links": [
|
||||||
@@ -237,6 +247,13 @@
|
|||||||
"relation": "implements",
|
"relation": "implements",
|
||||||
"evidence": "T1-T10 逐条实现设计的 D1-D6 六个决策与 §11 九条验收标准",
|
"evidence": "T1-T10 逐条实现设计的 D1-D6 六个决策与 §11 九条验收标准",
|
||||||
"added": "2026-08-02T09:49:48.126539+00:00"
|
"added": "2026-08-02T09:49:48.126539+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:governance-backend-error",
|
||||||
|
"target": "design:governance-backend-error",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "T1-T5 逐任务实现设计 §3 的五项决策与 §8 影响面清单",
|
||||||
|
"added": "2026-08-06T08:08:51.865565+00:00"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
# Research Wiki 索引
|
# Research Wiki 索引
|
||||||
|
|
||||||
> 自动生成，更新时间：2026-08-02 10:55 UTC
|
> 自动生成，更新时间：2026-08-06 08:11 UTC
|
||||||
|
|
||||||
## design (21)
|
## design (23)
|
||||||
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
|
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
|
||||||
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
|
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
|
||||||
- [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design`
|
- [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design`
|
||||||
@@ -13,6 +13,7 @@
|
|||||||
- [2026-07-30-settings-invariants-round-2-design](designs/2026-07-30-settings-invariants-round-2-design.md) `design:2026-07-30-settings-invariants-round-2-design`
|
- [2026-07-30-settings-invariants-round-2-design](designs/2026-07-30-settings-invariants-round-2-design.md) `design:2026-07-30-settings-invariants-round-2-design`
|
||||||
- [2026-07-31-response-observability-fields-design](designs/2026-07-31-response-observability-fields-design.md) `design:2026-07-31-response-observability-fields-design`
|
- [2026-07-31-response-observability-fields-design](designs/2026-07-31-response-observability-fields-design.md) `design:2026-07-31-response-observability-fields-design`
|
||||||
- [2026-07-31-sampling-params-design](designs/2026-07-31-sampling-params-design.md) `design:2026-07-31-sampling-params-design`
|
- [2026-07-31-sampling-params-design](designs/2026-07-31-sampling-params-design.md) `design:2026-07-31-sampling-params-design`
|
||||||
|
- [2026-08-06-governance-backend-error-design](designs/2026-08-06-governance-backend-error-design.md) `design:2026-08-06-governance-backend-error-design`
|
||||||
- [est_tokens 解耦: 拆分限流预扣与遥测用量兜底(issue #2)](designs/est-tokens-decoupling.md) `design:est-tokens-decoupling`
|
- [est_tokens 解耦: 拆分限流预扣与遥测用量兜底(issue #2)](designs/est-tokens-decoupling.md) `design:est-tokens-decoupling`
|
||||||
- [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2`
|
- [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2`
|
||||||
- [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards`
|
- [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards`
|
||||||
@@ -23,6 +24,7 @@
|
|||||||
- [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration`
|
- [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration`
|
||||||
- [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields`
|
- [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields`
|
||||||
- [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design`
|
- [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design`
|
||||||
|
- [治理后端故障归位为 scope 级不可用(Issue #7)](designs/governance-backend-error.md) `design:governance-backend-error`
|
||||||
- [采样参数透传设计(issue #4)](designs/sampling-params.md) `design:sampling-params`
|
- [采样参数透传设计(issue #4)](designs/sampling-params.md) `design:sampling-params`
|
||||||
|
|
||||||
## finding (12)
|
## finding (12)
|
||||||
@@ -39,7 +41,7 @@
|
|||||||
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
|
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
|
||||||
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
|
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
|
||||||
|
|
||||||
## plan (17)
|
## plan (19)
|
||||||
- [2026-07-20-m1-core-plan](plans/2026-07-20-m1-core-plan.md) `plan:2026-07-20-m1-core-plan`
|
- [2026-07-20-m1-core-plan](plans/2026-07-20-m1-core-plan.md) `plan:2026-07-20-m1-core-plan`
|
||||||
- [2026-07-20-m2-distributed-plan](plans/2026-07-20-m2-distributed-plan.md) `plan:2026-07-20-m2-distributed-plan`
|
- [2026-07-20-m2-distributed-plan](plans/2026-07-20-m2-distributed-plan.md) `plan:2026-07-20-m2-distributed-plan`
|
||||||
- [2026-07-21-m25-resilience-plan](plans/2026-07-21-m25-resilience-plan.md) `plan:2026-07-21-m25-resilience-plan`
|
- [2026-07-21-m25-resilience-plan](plans/2026-07-21-m25-resilience-plan.md) `plan:2026-07-21-m25-resilience-plan`
|
||||||
@@ -48,6 +50,7 @@
|
|||||||
- [2026-07-30-est-tokens-decoupling-plan](plans/2026-07-30-est-tokens-decoupling-plan.md) `plan:2026-07-30-est-tokens-decoupling-plan`
|
- [2026-07-30-est-tokens-decoupling-plan](plans/2026-07-30-est-tokens-decoupling-plan.md) `plan:2026-07-30-est-tokens-decoupling-plan`
|
||||||
- [2026-07-31-response-observability-fields](plans/2026-07-31-response-observability-fields.md) `plan:2026-07-31-response-observability-fields`
|
- [2026-07-31-response-observability-fields](plans/2026-07-31-response-observability-fields.md) `plan:2026-07-31-response-observability-fields`
|
||||||
- [2026-07-31-sampling-params](plans/2026-07-31-sampling-params.md) `plan:2026-07-31-sampling-params`
|
- [2026-07-31-sampling-params](plans/2026-07-31-sampling-params.md) `plan:2026-07-31-sampling-params`
|
||||||
|
- [2026-08-06-governance-backend-error-plan](plans/2026-08-06-governance-backend-error-plan.md) `plan:2026-08-06-governance-backend-error-plan`
|
||||||
- [est_tokens 解耦实施计划](plans/est-tokens-decoupling.md) `plan:est-tokens-decoupling`
|
- [est_tokens 解耦实施计划](plans/est-tokens-decoupling.md) `plan:est-tokens-decoupling`
|
||||||
- [M1 核心里程碑实现计划](plans/m1-core-plan.md) `plan:m1-core-plan`
|
- [M1 核心里程碑实现计划](plans/m1-core-plan.md) `plan:m1-core-plan`
|
||||||
- [M2 分布式实现计划](plans/m2-distributed.md) `plan:m2-distributed`
|
- [M2 分布式实现计划](plans/m2-distributed.md) `plan:m2-distributed`
|
||||||
@@ -55,6 +58,7 @@
|
|||||||
- [M3 OCR 实现计划](plans/m3-ocr.md) `plan:m3-ocr`
|
- [M3 OCR 实现计划](plans/m3-ocr.md) `plan:m3-ocr`
|
||||||
- [M4 迁移实现计划(T0-T14)](plans/m4-migration.md) `plan:m4-migration`
|
- [M4 迁移实现计划(T0-T14)](plans/m4-migration.md) `plan:m4-migration`
|
||||||
- [响应可观测字段扩展实现计划](plans/response-observability-fields.md) `plan:response-observability-fields`
|
- [响应可观测字段扩展实现计划](plans/response-observability-fields.md) `plan:response-observability-fields`
|
||||||
|
- [实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)](plans/governance-backend-error.md) `plan:governance-backend-error`
|
||||||
- [推理开关能力建模与 reasoning_tokens 采集实施计划(issue #5 + #6)](plans/2026-08-02-thinking-capability.md) `plan:2026-08-02-thinking-capability`
|
- [推理开关能力建模与 reasoning_tokens 采集实施计划(issue #5 + #6)](plans/2026-08-02-thinking-capability.md) `plan:2026-08-02-thinking-capability`
|
||||||
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
|
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
|
||||||
|
|
||||||
|
|||||||
@@ -84,3 +84,9 @@
|
|||||||
- [2026-08-02 09:49 UTC] 重建索引: 53 篇页面
|
- [2026-08-02 09:49 UTC] 重建索引: 53 篇页面
|
||||||
- [2026-08-02 10:55 UTC] 重建索引: 53 篇页面
|
- [2026-08-02 10:55 UTC] 重建索引: 53 篇页面
|
||||||
- [2026-08-02 10:55 UTC] 更新 finding: 补 §2.5 输出长度不是有效判别量(e2e 各 15 轮实测)
|
- [2026-08-02 10:55 UTC] 更新 finding: 补 §2.5 输出长度不是有效判别量(e2e 各 15 轮实测)
|
||||||
|
- [2026-08-06 06:37 UTC] 新增 design: 治理后端故障归位为 scope 级不可用(Issue #7) (design:governance-backend-error)
|
||||||
|
- [2026-08-06 06:38 UTC] 重建索引: 55 篇页面
|
||||||
|
- [2026-08-06 08:08 UTC] 新增 plan: 实现计划: 治理后端故障归位为 scope 级不可用(Issue #7) (plan:governance-backend-error)
|
||||||
|
- [2026-08-06 08:08 UTC] 新增边: plan:governance-backend-error --implements--> design:governance-backend-error
|
||||||
|
- [2026-08-06 08:08 UTC] 重建索引: 57 篇页面
|
||||||
|
- [2026-08-06 08:11 UTC] 重建索引: 57 篇页面
|
||||||
|
|||||||
@@ -181,7 +181,7 @@ stack = ExtractionProviderStack(
|
|||||||
| R4 | 六道闸+契约 5 条、服务器时钟窗口、settle 落 acquire 窗口、transient 按 est 保守结算 | M2 | §7.3 大体覆盖 |
|
| R4 | 六道闸+契约 5 条、服务器时钟窗口、settle 落 acquire 窗口、transient 按 est 保守结算 | M2 | §7.3 大体覆盖 |
|
||||||
| R5 | RequestRejected 二分(真实响应记成功/本地拒绝释放探针);换源重试跨源计数口径 | M2 | 须进 M2 设计 |
|
| R5 | RequestRejected 二分(真实响应记成功/本地拒绝释放探针);换源重试跨源计数口径 | M2 | 须进 M2 设计 |
|
||||||
| R6 | OCR ZIP 协议 + bbox 数值防御下沉;OCR Usage=0;glm 白名单预留 | M3 | §7.10 已覆盖 |
|
| R6 | OCR ZIP 协议 + bbox 数值防御下沉;OCR Usage=0;glm 白名单预留 | M3 | §7.10 已覆盖 |
|
||||||
| **G1** | ✅ 已闭(M3 核实): 库 `GatewayUnavailableError` 一族自 M1 起携 `scope/reason/retry_after_s/per_source_reasons`(errors.py:74-105),chat/embedding/OCR 三循环抛出点均已填充且有契约测试钉住;项目侧仅剩约 10 行翻译 shim(库异常 → ProviderUnavailableError)或 tracking.py 直接 except 库异常 | M2 | 已闭 |
|
| **G1** | ✅ 已闭(M3 核实): 库 `GatewayUnavailableError` 一族自 M1 起携 `scope/reason/retry_after_s/per_source_reasons`(errors.py:74-105),chat/embedding/OCR 三循环抛出点均已填充且有契约测试钉住;项目侧仅剩约 10 行翻译 shim(库异常 → ProviderUnavailableError)或 tracking.py 直接 except 库异常。**2026-08-06 补(issue #7,库 1.1.0)**: 治理后端故障(`GovernanceBackendError`,Redis 挂等 fail-closed 情形)此前**不在**该族内,`except GatewayUnavailableError` 接不住,会落进 `_TERMINAL` 兜底而消耗业务失败预算;现已归入该族(`reason=governance_backend_down`,`retry_after_s` 默认 5.0),tracking.py 一条 except 即覆盖完整,**无需为它单列分支**。同批新增的 `SourceNotConfiguredError`(源名与配置不匹配的装配缺陷)**有意在族外**,应当落进 `_TERMINAL` 让配置错误浮出水面 | M2 | 已闭 |
|
||||||
| **G2** | ✅ 已闭(2026-07-30 核实): `est_tokens` 已进 ARCH §7.7 SourceConfig 字段清单,且 §7.3 `try_acquire` 的 est 来源已定义为 `SourceConfig.effective_est_tokens()`(显式值优先,否则按 `tpm // 60` 派生)。双职责一并拆开:该字段只剩 TPM 预扣的可选调优覆盖,usage 缺失不再由它兜底(见 §7 行 151 的推翻判定) | M2 | 已闭 |
|
| **G2** | ✅ 已闭(2026-07-30 核实): `est_tokens` 已进 ARCH §7.7 SourceConfig 字段清单,且 §7.3 `try_acquire` 的 est 来源已定义为 `SourceConfig.effective_est_tokens()`(显式值优先,否则按 `tpm // 60` 派生)。双职责一并拆开:该字段只剩 TPM 预扣的可选调优覆盖,usage 缺失不再由它兜底(见 §7 行 151 的推翻判定) | M2 | 已闭 |
|
||||||
| **G3** | ⚠️ §4.3 层序图文矛盾:图示 熔断→限流→重试(重试最内),但理由要求"每次重试重新过限流闸"且熔断/限流是 per-source 的、选源在重试循环内(governance.py:120-167 实践为每次尝试执行 选源→冷却备忘→permit→熔断门)。洋葱不澄清"逐次准入"机制则多源语义无法成立 | M2 | **架构缺口**,澄清 §4.3/§4.4 |
|
| **G3** | ⚠️ §4.3 层序图文矛盾:图示 熔断→限流→重试(重试最内),但理由要求"每次重试重新过限流闸"且熔断/限流是 per-source 的、选源在重试循环内(governance.py:120-167 实践为每次尝试执行 选源→冷却备忘→permit→熔断门)。洋葱不澄清"逐次准入"机制则多源语义无法成立 | M2 | **架构缺口**,澄清 §4.3/§4.4 |
|
||||||
| **G4** | ⚠️ per-scope 韧性配置命名(`{SCOPE}__RETRY__*`/`BREAKER__*`/`BACKPRESSURE__*`/`SELECTOR`/`GLOBAL__*`)未进 ARCH §9,现文只有平铺 `LLM_*` 键;CHSAnalyzer 的 VLM/OCR 两 scope 参数各异,平铺键无法表达 | M2 | **架构缺口**,修订 §9 |
|
| **G4** | ⚠️ per-scope 韧性配置命名(`{SCOPE}__RETRY__*`/`BREAKER__*`/`BACKPRESSURE__*`/`SELECTOR`/`GLOBAL__*`)未进 ARCH §9,现文只有平铺 `LLM_*` 键;CHSAnalyzer 的 VLM/OCR 两 scope 参数各异,平铺键无法表达 | M2 | **架构缺口**,修订 §9 |
|
||||||
|
|||||||
@@ -0,0 +1,290 @@
|
|||||||
|
# 实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)
|
||||||
|
|
||||||
|
- **设计**: `research-wiki/designs/2026-08-06-governance-backend-error-design.md`(已批准 2026-08-06,Q1/Q2/Q3 逐条拍板)
|
||||||
|
- **分支**: `feat/issue-7-governance-backend-error`
|
||||||
|
- **目标**: 让"限流/熔断后端故障"在类型上落入 `GatewayUnavailableError`,使调用方一条 `except` 覆盖完整;同时把混在同一类里的装配缺陷拆出去,避免配置写错的任务永远重投。
|
||||||
|
- **方案概述**: `GovernanceBackendError` 改继承 `GatewayUnavailableError`(新 reason `governance_backend_down`,`retry_after_s` 默认 5.0);两处"未知源"改抛新增的 `SourceNotConfiguredError`(**不**在 scope 级家族内);`scope` 由后端层 `self._scope` 与两个 gate 包装器注入。
|
||||||
|
- **涉及技术**: Python 3.11+,pytest(含真实 Redis 的 integration),radon/ruff 门禁。
|
||||||
|
|
||||||
|
## 保真校验(适用)
|
||||||
|
|
||||||
|
本计划触及 ARCHITECTURE.md §1.4 索引的移植蓝本:错误分类(`reference/CHSAnalyzer/app/domain/errors.py`)与限流/熔断(`reference/CHSAnalyzer/app/coordination/`)。
|
||||||
|
|
||||||
|
本次**有意变更**的语义只有一条,已在设计 §3.1 声明:`GovernanceBackendError` 的类型归属(CHS 的 `LimiterError` 是独立异常,本库将其提升为 scope 级不可用的一员)。除此之外,下列承自 CHS 的语义**不得被顺带改动**,每个任务完成前逐条自查:
|
||||||
|
|
||||||
|
| 不得改动 | 出处 |
|
||||||
|
|---|---|
|
||||||
|
| `retry_after_s` 非可选、`0 = 可立即重试` | `errors.py:74-78` |
|
||||||
|
| `SCOPE_REASONS` 既有 5 值与 `SOURCE_REASONS` 既有 7 值 | `errors.py:7-20` |
|
||||||
|
| fail-closed 降级方向(限流/熔断后端挂 → 报错而非放行) | 库铁律 |
|
||||||
|
| 记账路径降级为 warning、闸门路径上抛的分工 | `middleware/retry.py:404` |
|
||||||
|
| `RedisPermit.release/settle` 的释放侧降级 | `backends/redis/limiter.py:133,151` |
|
||||||
|
|
||||||
|
## 文件结构
|
||||||
|
|
||||||
|
| 文件 | 职责 | 动作 |
|
||||||
|
|---|---|---|
|
||||||
|
| `research-wiki/ARCHITECTURE.md` | 架构单一事实源 §6.1 错误分类表 | 修改(**必须先行**,见设计 §8.1) |
|
||||||
|
| `src/polygateway/errors.py` | 错误类型树内核 | 修改: 新常量、新 reason、新类、继承变更 |
|
||||||
|
| `src/polygateway/__init__.py` | 公共 API 面 | 修改: 导出新类 |
|
||||||
|
| `src/polygateway/backends/redis/limiter.py` | Redis 限流后端 | 修改: 6 处补 scope、1 处换新类 |
|
||||||
|
| `src/polygateway/backends/redis/breaker.py` | Redis 熔断后端 | 修改: 5 处补 scope |
|
||||||
|
| `src/polygateway/backends/memory/limiter.py` | 内存限流后端 | 修改: 1 处换新类 |
|
||||||
|
| `src/polygateway/middleware/ratelimit.py` | `QuotaGate` 包装器 | 修改: 构造增 scope、4 处补 scope |
|
||||||
|
| `src/polygateway/middleware/breaker.py` | `BreakerGate` 包装器 | 修改: 构造增 scope、5 处补 scope |
|
||||||
|
| `src/polygateway/middleware/retry.py` / `ocr.py` / `embedding.py` | 三处 gate 装配 | 修改: 各 2 行传 scope |
|
||||||
|
| `tests/unit/test_errors.py` | 错误类型契约 | 修改 |
|
||||||
|
| `tests/unit/test_backpressure.py` | 后端故障传播 | 修改 |
|
||||||
|
| `tests/unit/test_redis_key_layout.py` | 未知源行为 | 修改 |
|
||||||
|
| `tests/integration/test_redis_cross_connection.py` | 真实 Redis 掉线 | 修改 |
|
||||||
|
| `README.md` / `research-wiki/migrations/chsanalyzer.md` / `CHANGELOG.md` / `pyproject.toml` | 文档与版本 | 修改 |
|
||||||
|
|
||||||
|
## 关键接口(跨任务消费,此处写死)
|
||||||
|
|
||||||
|
`errors.py` 新增与变更部分:
|
||||||
|
|
||||||
|
```python
|
||||||
|
GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0
|
||||||
|
"""治理后端故障的建议重投间隔(秒)。
|
||||||
|
|
||||||
|
**不是环境配置项**——后端恢复时间物理上不可知(不同于熔断冷却有确定到期
|
||||||
|
时刻),故取一个保守固定值;下游有自己的退避策略时可忽略本字段。取 0 会让
|
||||||
|
积压任务零延迟同时冲击已挂掉的后端(issue #7 §3.2)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class SourceNotConfiguredError(PolyGatewayError):
|
||||||
|
"""源名不在限流后端的配置字典中: 装配缺陷,正常不可达。
|
||||||
|
|
||||||
|
**有意不在** `GatewayUnavailableError` 之下: 它不是"暂时不可用"而是
|
||||||
|
"配置写错了",必须消耗失败预算进死信让人看见;归入可重投家族会让配置
|
||||||
|
错误的任务永远重投、永不告警(issue #7 §3.4)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class GovernanceBackendError(GatewayUnavailableError):
|
||||||
|
"""限流/熔断状态后端自身故障: 必须报错而非放行(防击穿网关,降级方向铁律)。
|
||||||
|
|
||||||
|
继承 `GatewayUnavailableError`: fail-closed 时一个请求都发不出去,语义
|
||||||
|
上即 scope 级不可用,调用方一条 except 即可覆盖(issue #7)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
message: str,
|
||||||
|
*,
|
||||||
|
scope: str,
|
||||||
|
retry_after_s: float = GOVERNANCE_BACKEND_RETRY_AFTER_S,
|
||||||
|
source_name: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
scope=scope,
|
||||||
|
reason="governance_backend_down",
|
||||||
|
retry_after_s=retry_after_s,
|
||||||
|
source_name=source_name,
|
||||||
|
)
|
||||||
|
# 父类会把 message 覆写为 "{scope} 网关暂时不可用: {reason}",而各构造点
|
||||||
|
# 携带的诊断串是排障主线索,必须保住(设计 §3.5,机制已实跑验证)
|
||||||
|
self.args = (message,)
|
||||||
|
```
|
||||||
|
|
||||||
|
两个 gate 包装器的构造签名(`scope` 为 keyword-only 必填):
|
||||||
|
|
||||||
|
```python
|
||||||
|
class QuotaGate:
|
||||||
|
def __init__(self, limiter: RateLimiter, *, scope: str) -> None:
|
||||||
|
self._limiter = limiter
|
||||||
|
self._scope = scope
|
||||||
|
|
||||||
|
|
||||||
|
class BreakerGate:
|
||||||
|
def __init__(self, gate: ProviderGate, *, scope: str) -> None:
|
||||||
|
self._gate = gate
|
||||||
|
self._scope = scope
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 任务清单
|
||||||
|
|
||||||
|
### - [x] T1: ARCHITECTURE §6.1 回补(必须先行)
|
||||||
|
|
||||||
|
**文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行)
|
||||||
|
|
||||||
|
**行为**: 在错误分类表补两行——`GovernanceBackendError`(scope 级不可用,reason 恒为 `governance_backend_down`)与 `SourceNotConfiguredError`(装配缺陷,不重试不换源,消耗失败预算);scope 级 `reason` 值域由 5 值扩为 6 值,增 `governance_backend_down`。同时记录本次归位的理由与日期,并说明根因(该类是 M2 引入分布式后端时新增,当时未回补本表)。
|
||||||
|
|
||||||
|
**为什么先行**: `ARCHITECTURE.md` 是单一事实源,新 reason 值域与其现状冲突;先改代码后补文档等于让实现与事实源脱节(设计 §8.1)。
|
||||||
|
|
||||||
|
**验收**: §6.1 表格含上述两行;reason 值域文字与 `errors.py` 将要写入的 `SCOPE_REASONS` 逐字一致。
|
||||||
|
|
||||||
|
**测试要求**: 纯文档,无测试证据要求。
|
||||||
|
|
||||||
|
**验证**: `grep -n "governance_backend_down\|SourceNotConfiguredError" research-wiki/ARCHITECTURE.md` → 至少各 1 处命中。
|
||||||
|
|
||||||
|
**提交**: `docs: admit governance backend failures into the scope-level error model`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### - [x] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出
|
||||||
|
|
||||||
|
**文件**: 改 `src/polygateway/errors.py`、`src/polygateway/__init__.py`;改 `tests/unit/test_errors.py`
|
||||||
|
|
||||||
|
**行为**:
|
||||||
|
1. 加模块级常量 `GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0`(docstring 逐字见上文"关键接口");
|
||||||
|
2. `SCOPE_REASONS` 增 `"governance_backend_down"`;
|
||||||
|
3. 新增 `SourceNotConfiguredError(PolyGatewayError)`(定义逐字见上文);
|
||||||
|
4. `__init__.py` 的 import 块与 `__all__` 各增 `SourceNotConfiguredError`(`__all__` 保持字母序: `SourceDeadError` → **`SourceNotConfiguredError`** → `TransientError`,即插在 `SourceDeadError` **之后**)。
|
||||||
|
|
||||||
|
**本任务不动 `GovernanceBackendError`**——它是纯增量,不破坏任何既有调用点,可独立提交且全套件保持通过。
|
||||||
|
|
||||||
|
**测试要求(先失败后通过)**:
|
||||||
|
- 新增用例断言 `SourceNotConfiguredError` **不是** `GatewayUnavailableError` 的子类,且是 `PolyGatewayError` 的子类。改前该类不存在 → `ImportError`;改后 PASS。
|
||||||
|
- 新增用例断言 `"governance_backend_down" in SCOPE_REASONS`,且 `GatewayUnavailableError(scope="llm", reason="governance_backend_down", retry_after_s=0.0)` 可构造。改前 `reason` 校验抛 `ValueError` → 用例失败;改后 PASS。
|
||||||
|
- 新增用例断言 `from polygateway import SourceNotConfiguredError` 可用。
|
||||||
|
|
||||||
|
**验证**: `conda run -n PolyGateway pytest tests/unit/test_errors.py -v` → 全 PASS;`conda run -n PolyGateway pytest tests/ -q` → 与改动前同样全绿(纯增量不应影响任何既有用例)。
|
||||||
|
|
||||||
|
**提交**: `feat: add SourceNotConfiguredError and the governance backend reason`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### - [x] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子)
|
||||||
|
|
||||||
|
**文件**: 改 `src/polygateway/errors.py`、`backends/redis/limiter.py`、`backends/redis/breaker.py`、`backends/memory/limiter.py`、`middleware/ratelimit.py`、`middleware/breaker.py`、`middleware/retry.py`、`ocr.py`、`embedding.py`;改 `tests/unit/test_errors.py`、`tests/unit/test_backpressure.py`、`tests/unit/test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py`
|
||||||
|
|
||||||
|
**为什么必须原子**: `scope` 是必填 keyword,继承变更与全部构造点若分批提交,中间状态会 `TypeError`,门禁跑不过。
|
||||||
|
|
||||||
|
**行为**:
|
||||||
|
|
||||||
|
1. `errors.py`: `GovernanceBackendError` 改继承 `GatewayUnavailableError` 并覆写 `__init__`(逐字见上文"关键接口")。
|
||||||
|
|
||||||
|
2. **两处未知源改抛新类**(设计 §3.4,Q1 已拍板):
|
||||||
|
|
||||||
|
| 位置 | 改为 |
|
||||||
|
|---|---|
|
||||||
|
| `backends/redis/limiter.py:198` | `raise SourceNotConfiguredError(f"未知源 {source_key!r}(scope={self._scope})")` |
|
||||||
|
| `backends/memory/limiter.py:92` | 同上 |
|
||||||
|
|
||||||
|
3. **后端层 11 处补 `scope=self._scope`**(该属性已存在: redis limiter `:170`、redis breaker `:291`、memory limiter 同名字段):
|
||||||
|
- `backends/redis/limiter.py` 的 `:250 / :268 / :275 / :286 / :298 / :305`(6 处)
|
||||||
|
- `backends/redis/breaker.py` 的 `:370 / :388 / :410 / :422 / :432`(5 处)
|
||||||
|
|
||||||
|
4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise` 补 `scope=self._scope`。
|
||||||
|
- ~~各方法开头的 `except GovernanceBackendError: raise` **保持不变**~~ **← 这条是错的,2026-08-06 独立验证时炸出(见 §T6)**。正确做法: 该放行必须扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise`,否则新增的兄弟类型会落进下一行的 `except Exception` 被**重新包成** `GovernanceBackendError`,使 Q1 的拆分在唯一的生产路径上完全失效。
|
||||||
|
|
||||||
|
5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序):
|
||||||
|
|
||||||
|
| 文件 | 行 | 改为 |
|
||||||
|
|---|---|---|
|
||||||
|
| `middleware/retry.py` | 186-187 | `QuotaGate(limiter, scope=self._scope)` / `BreakerGate(gate, scope=self._scope)` |
|
||||||
|
| `ocr.py` | 122-123 | 同款 |
|
||||||
|
| `embedding.py` | 123-124 | 同款 |
|
||||||
|
|
||||||
|
**测试要求(先失败后通过,逐条对应)**:
|
||||||
|
|
||||||
|
| 用例 | 文件 | 改前为何失败 |
|
||||||
|
|---|---|---|
|
||||||
|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
|
||||||
|
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
|
||||||
|
| 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
|
||||||
|
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
|
||||||
|
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
|
||||||
|
|
||||||
|
**必须同批更新的既有测试构造点**(新签名为 keyword-only 必填,漏改即 `TypeError: missing required keyword-only argument`,门禁直接红):
|
||||||
|
|
||||||
|
| 位置 | 现状 | 改为 |
|
||||||
|
|---|---|---|
|
||||||
|
| `tests/unit/test_backpressure.py:176 / :181 / :186` | `raise GovernanceBackendError("redis 抖动")` | 补 `scope=`(任意测试 scope,如 `"llm"`) |
|
||||||
|
| `tests/unit/test_errors.py:89` | `exc = GovernanceBackendError("redis down")` | 同上;该用例现断言它**不属于**可重试分类,须一并改为断言它**是** `GatewayUnavailableError` |
|
||||||
|
| `tests/unit/test_backpressure.py:255 / :257` | `QuotaGate(_L())` / `QuotaGate(_Broken())` | `QuotaGate(_L(), scope="llm")` 等 |
|
||||||
|
|
||||||
|
**保真校验检查点**: 提交前对照上文"保真校验"五条逐条自查,确认无一被顺带改动。特别核对 `RedisPermit.release/settle`(`redis/limiter.py:133,151`)的 `except GovernanceBackendError` 仍能接住释放侧失败——该处是设计 §4 否决"让原始异常穿透"路线的直接原因。
|
||||||
|
|
||||||
|
**验证**:
|
||||||
|
- `conda run -n PolyGateway pytest tests/unit tests/contracts -v` → 全 PASS
|
||||||
|
- `conda run -n PolyGateway pytest tests/integration -v` → 全 PASS(需真实 Redis)
|
||||||
|
- `conda run -n PolyGateway pytest tests/ -q` → `0 failed`
|
||||||
|
- `conda run -n PolyGateway radon cc src -n C -s` → 无输出
|
||||||
|
- `make lint` → import-linter 契约全绿(本次不新增跨层依赖,应无变化)
|
||||||
|
|
||||||
|
**提交**: `fix: reparent governance backend failures under GatewayUnavailableError (issue #7)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### - [x] T4: 公开错误面文档(issue #7 第二诉求)
|
||||||
|
|
||||||
|
**文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md`
|
||||||
|
|
||||||
|
**行为**:
|
||||||
|
1. README 增一张两列表,明确区分**会到达调用方**与**库内吸收**:
|
||||||
|
|
||||||
|
| 会到达调用方 | 库内吸收 |
|
||||||
|
|---|---|
|
||||||
|
| `GatewayUnavailableError` 族(`CircuitOpenError` / `AllSourcesExhausted` / `GovernanceBackendError`) | `TransientError` |
|
||||||
|
| `RequestRejectedError` | `SourceDeadError` |
|
||||||
|
| `ResultInvalidError` | |
|
||||||
|
| `SourceNotConfiguredError` | |
|
||||||
|
|
||||||
|
2. 在该表下补一句说明: `TransientError` / `SourceDeadError` 的 docstring 描述的是**库内治理行为**,它们被 `middleware/retry.py:365` 接住并在预算耗尽时包成 `AllSourcesExhausted`,**不会**到达调用方——issue #7 记载下游曾据此写错整段设计文档。
|
||||||
|
3. `migrations/chsanalyzer.md` 的 G1 条目补注:后端故障现已并入 `GatewayUnavailableError`,项目侧 `except GatewayUnavailableError` 一条即覆盖完整,无需为 `GovernanceBackendError` 单列分支。
|
||||||
|
|
||||||
|
**验收**: 调用方仅读 README 即可判断该 catch 什么,无需读 `middleware/retry.py`。
|
||||||
|
|
||||||
|
**测试要求**: 纯文档,无测试证据要求。
|
||||||
|
|
||||||
|
**验证**: `grep -n "库内吸收" README.md` → 命中。
|
||||||
|
|
||||||
|
**提交**: `docs: publish which errors reach callers and which the library absorbs`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### - [x] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步
|
||||||
|
|
||||||
|
**文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki
|
||||||
|
|
||||||
|
**行为**: 版本 `1.0.6` → `1.1.0`(有行为变更但无 API 破坏:加父类是扩大)。CHANGELOG 需写明:
|
||||||
|
|
||||||
|
- **行为变更**: 后端故障从"落入调用方兜底分支"变为"被 `except GatewayUnavailableError` 捕获";下游据此把它按"延期重投、不消耗失败预算"处置,这正是修复目标,但**处置路线确实变了**,升级前须确认下游的兜底分支没有依赖它。
|
||||||
|
- **新增**: `SourceNotConfiguredError`(公共导出)、`GOVERNANCE_BACKEND_RETRY_AFTER_S`、scope 级 reason `governance_backend_down`。
|
||||||
|
- **下游请读**: `GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`(默认 5.0)/`per_source_reasons`;`str(exc)` 仍是原诊断串,结构化字段并存。配置写错(源名不匹配)现在抛 `SourceNotConfiguredError` 而非 `GovernanceBackendError`,它**不**属于可重投家族——这是有意的,目的是让装配缺陷进死信而不是永远重投。
|
||||||
|
|
||||||
|
**验收**: 版本三处一致(`pyproject.toml` / `__init__.py` / CHANGELOG 标题);CLAUDE.md §6 要求"版本 bump 提交不得裸发",故本任务必须与 wiki 同步同批。
|
||||||
|
|
||||||
|
**测试要求**: 无行为变更,`pytest tests/ -q` 保持全绿即可。
|
||||||
|
|
||||||
|
**验证**: `grep -n "1.1.0" pyproject.toml src/polygateway/__init__.py CHANGELOG.md` → 三处命中。
|
||||||
|
|
||||||
|
**提交**: `chore: release 1.1.0`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### - [x] T6: 修复独立验证炸出的阻塞缺陷(计划外,2026-08-06)
|
||||||
|
|
||||||
|
T1–T5 全绿、全部门禁通过之后,全新上下文的 verifier 用一个**走 `QuotaGate` 的**端到端用例炸出:装配缺陷在唯一的生产路径上根本没有拆出去。
|
||||||
|
|
||||||
|
**缺陷**: `QuotaGate`/`BreakerGate` 的 `except GovernanceBackendError: raise` 只放行了旧类型,新增的 `SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包成 `GovernanceBackendError`(`reason=governance_backend_down`、`retry_after_s=5.0`)。实证:
|
||||||
|
|
||||||
|
```
|
||||||
|
RAISED: GovernanceBackendError | isGatewayUnavailable=True | isSourceNotConfigured=False
|
||||||
|
| 限流后端故障(source_stats): 未知源 's1'(scope=llm)
|
||||||
|
```
|
||||||
|
|
||||||
|
即配置写错的任务照样落进"可延期重投"家族,**永远重投、永不进死信、无人告警**——正是 Q1 要防的镜像 bug,G2 等于没做。
|
||||||
|
|
||||||
|
**为什么原有测试测不出来**: T3 写的两条用例(`test_backpressure.py`、`test_redis_key_layout.py`)都直接打私有 `_cfg()`,绕过了包装器;而治理循环只经包装器访问后端。**盲区在于测试打的层次比生产路径低一层。**
|
||||||
|
|
||||||
|
**修复**(三处):
|
||||||
|
|
||||||
|
| 文件 | 改动 |
|
||||||
|
|---|---|
|
||||||
|
| `middleware/ratelimit.py` | 4 个方法的放行扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise` |
|
||||||
|
| `middleware/breaker.py` | 同上,5 个方法 |
|
||||||
|
| `middleware/telemetry.py:254` | 终态捕获元组加 `SourceNotConfiguredError`。**连带坑**: 放行生效后该异常不再是 `GovernanceBackendError`,而它在任何 attempt 之前抛出,若不显式捕获则 `emit_terminal_failure` 不触发、该路径**遥测归零**,违反"遥测必录"铁律 |
|
||||||
|
|
||||||
|
**回归测试**: `test_backpressure.py::TestUnknownSourceIsAssemblyDefect::test_survives_the_quota_gate_wrapper`(参数化覆盖 `try_acquire` / `stats`),**走包装器而非私有方法**。修前 2 failed,修后 PASS。
|
||||||
|
|
||||||
|
**同批文档订正**: 泄漏路径由"三条"改为**五条**(遗漏了 `QuotaGate.stats` 与 `BreakerGate.retry_after_s`,判据是该调用点是否被 `_record_quietly` 包裹);CHANGELOG 的 `per_source_reasons` 表述改为"属性存在但恒为 `{}`"。
|
||||||
|
|
||||||
|
## 完成后
|
||||||
|
|
||||||
|
按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7。
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:governance-backend-error
|
||||||
|
title: "实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)"
|
||||||
|
date: 2026-08-06
|
||||||
|
---
|
||||||
|
|
||||||
|
# 实现计划: 治理后端故障归位为 scope 级不可用(Issue #7)
|
||||||
|
|
||||||
|
全文见 `2026-08-06-governance-backend-error-plan.md`。实现设计 [[governance-backend-error]](已批准 2026-08-06)。
|
||||||
|
|
||||||
|
## 五个任务
|
||||||
|
|
||||||
|
| # | 任务 | 关键约束 |
|
||||||
|
|---|---|---|
|
||||||
|
| T1 | `ARCHITECTURE.md` §6.1 回补两行 + reason 值域扩为 6 值 | **必须先行**——单一事实源纪律,先改代码后补文档等于让实现与事实源脱节(设计 §8.1) |
|
||||||
|
| T2 | `errors.py` 纯增量: 新常量、新 reason、`SourceNotConfiguredError` + 顶层导出 | 刻意不动 `GovernanceBackendError`,故全套件保持通过,可独立提交 |
|
||||||
|
| T3 | `GovernanceBackendError` 归位 + 22 处构造点 + gate scope 注入 + 全部受影响测试 | **必须原子**: `scope` 是必填 keyword,分批提交的中间状态会 `TypeError` |
|
||||||
|
| T4 | README 公开错误面两列表 + 迁移文档补注 | issue #7 的第二诉求,作者认为比第一条更值得改 |
|
||||||
|
| T5 | 版本 1.1.0 + CHANGELOG + Wiki 同步 | 加父类是扩大不是破坏,故 minor 而非 major |
|
||||||
|
|
||||||
|
## 保真校验(适用)
|
||||||
|
|
||||||
|
触及 ARCHITECTURE §1.4 的移植蓝本(CHS `app/domain/errors.py` 与 `app/coordination/`)。本次**有意变更**的语义仅一条(`GovernanceBackendError` 的类型归属);`retry_after_s` 非可选语义、两个 reason 值域、fail-closed 方向、记账/闸门分工、`RedisPermit` 释放侧降级五条**不得被顺带改动**,每任务完成前逐条自查。
|
||||||
|
|
||||||
|
## 独立审查修正(2026-08-06, Codex)
|
||||||
|
|
||||||
|
4 条意见全部核实属实并已折回:
|
||||||
|
|
||||||
|
1. **T3 测试证据定位错误**(重要)——原写"复用 `test_backpressure.py:176-186` 的注入桩"覆盖三条泄漏路径,实测那三个桩是 `record_success`/`record_failure`/`mark_progress` 的**记账侧降级**,与闸门路径无关;`try_acquire`/`try_enter` 全无覆盖。已改为"三条桩都要新增"并写明现状。
|
||||||
|
2. **T3 漏了既有测试构造点**(重要)——新签名 keyword-only 必填,`test_backpressure.py:176/181/186`、`test_errors.py:89` 的裸 `GovernanceBackendError("...")` 与 `:255/:257` 的 `QuotaGate(_L())` 漏改即 `TypeError`。已补一张同批更新清单。
|
||||||
|
3. **`__all__` 插入位置写反**(次要)——按字母序应在 `SourceDeadError` **之后**而非之前。已改。
|
||||||
|
4. **设计中 telemetry 行号过时**(次要)——`:210` → `:250`,系本分支加 `_AttemptUsage` 造成的漂移。设计与摘要页已同步更新。
|
||||||
|
|
||||||
|
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
|
||||||
|
|
||||||
|
## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier)
|
||||||
|
|
||||||
|
T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。**
|
||||||
|
|
||||||
|
修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。
|
||||||
|
|
||||||
|
两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats` 与 `BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。
|
||||||
|
|
||||||
|
相关: [[governance-backend-error]](design)、[[m2-distributed]]
|
||||||
@@ -17,6 +17,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.ocr import OcrClient
|
from polygateway.ocr import OcrClient
|
||||||
@@ -31,7 +32,7 @@ from polygateway.types import (
|
|||||||
SourceConfig,
|
SourceConfig,
|
||||||
)
|
)
|
||||||
|
|
||||||
__version__ = "1.0.6"
|
__version__ = "1.1.0"
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"DEFAULT_PROFILES",
|
"DEFAULT_PROFILES",
|
||||||
@@ -58,6 +59,7 @@ __all__ = [
|
|||||||
"ResultInvalidError",
|
"ResultInvalidError",
|
||||||
"SourceConfig",
|
"SourceConfig",
|
||||||
"SourceDeadError",
|
"SourceDeadError",
|
||||||
|
"SourceNotConfiguredError",
|
||||||
"TransientError",
|
"TransientError",
|
||||||
"__version__",
|
"__version__",
|
||||||
"gather_bounded",
|
"gather_bounded",
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import time
|
|||||||
import uuid
|
import uuid
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from polygateway.errors import GovernanceBackendError
|
from polygateway.errors import SourceNotConfiguredError
|
||||||
from polygateway.types import GlobalLimits, SourceConfig, SourceStats
|
from polygateway.types import GlobalLimits, SourceConfig, SourceStats
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
@@ -89,7 +89,7 @@ class InMemoryLimiter:
|
|||||||
def _cfg(self, source_key: str) -> SourceConfig:
|
def _cfg(self, source_key: str) -> SourceConfig:
|
||||||
cfg = self._sources.get(source_key)
|
cfg = self._sources.get(source_key)
|
||||||
if cfg is None:
|
if cfg is None:
|
||||||
raise GovernanceBackendError(f"未知源 {source_key!r}(scope={self._scope})")
|
raise SourceNotConfiguredError(f"未知源 {source_key!r}(scope={self._scope})")
|
||||||
return cfg
|
return cfg
|
||||||
|
|
||||||
def _window(self) -> int:
|
def _window(self) -> int:
|
||||||
|
|||||||
@@ -367,7 +367,7 @@ class RedisGate:
|
|||||||
keys=[self._key(source_name)], args=[owner, self._probe_ttl_ms]
|
keys=[self._key(source_name)], args=[owner, self._probe_ttl_ms]
|
||||||
)
|
)
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端 try_enter 失败: {exc}") from exc
|
raise GovernanceBackendError(f"熔断后端 try_enter 失败: {exc}", scope=self._scope) from exc
|
||||||
return self._decision(source_name, result)
|
return self._decision(source_name, result)
|
||||||
|
|
||||||
async def record_success(
|
async def record_success(
|
||||||
@@ -385,7 +385,7 @@ class RedisGate:
|
|||||||
try:
|
try:
|
||||||
result = await self._success_lua(keys=[self._key(entry.source_name)], args=args)
|
result = await self._success_lua(keys=[self._key(entry.source_name)], args=args)
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端 record_success 失败: {exc}") from exc
|
raise GovernanceBackendError(f"熔断后端 record_success 失败: {exc}", scope=self._scope) from exc
|
||||||
return self._update(result)
|
return self._update(result)
|
||||||
|
|
||||||
async def record_failure(
|
async def record_failure(
|
||||||
@@ -407,7 +407,7 @@ class RedisGate:
|
|||||||
try:
|
try:
|
||||||
result = await self._failure_lua(keys=[self._key(entry.source_name)], args=args)
|
result = await self._failure_lua(keys=[self._key(entry.source_name)], args=args)
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端 record_failure 失败: {exc}") from exc
|
raise GovernanceBackendError(f"熔断后端 record_failure 失败: {exc}", scope=self._scope) from exc
|
||||||
return self._update(result)
|
return self._update(result)
|
||||||
|
|
||||||
async def release_probe(self, entry: GateDecision) -> GateUpdate:
|
async def release_probe(self, entry: GateDecision) -> GateUpdate:
|
||||||
@@ -419,7 +419,7 @@ class RedisGate:
|
|||||||
keys=[self._key(entry.source_name)], args=[entry.epoch, entry.probe_owner]
|
keys=[self._key(entry.source_name)], args=[entry.epoch, entry.probe_owner]
|
||||||
)
|
)
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端 release_probe 失败: {exc}") from exc
|
raise GovernanceBackendError(f"熔断后端 release_probe 失败: {exc}", scope=self._scope) from exc
|
||||||
return self._update(result)
|
return self._update(result)
|
||||||
|
|
||||||
async def retry_after_s(self, sources: tuple[str, ...]) -> float:
|
async def retry_after_s(self, sources: tuple[str, ...]) -> float:
|
||||||
@@ -429,7 +429,7 @@ class RedisGate:
|
|||||||
try:
|
try:
|
||||||
result = await self._retry_after_lua(keys=[self._key(s) for s in sources])
|
result = await self._retry_after_lua(keys=[self._key(s) for s in sources])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端 retry_after_s 失败: {exc}") from exc
|
raise GovernanceBackendError(f"熔断后端 retry_after_s 失败: {exc}", scope=self._scope) from exc
|
||||||
return int(result) / 1000.0
|
return int(result) / 1000.0
|
||||||
|
|
||||||
async def aclose(self) -> None:
|
async def aclose(self) -> None:
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ from typing import TYPE_CHECKING
|
|||||||
from loguru import logger
|
from loguru import logger
|
||||||
from redis.exceptions import RedisError
|
from redis.exceptions import RedisError
|
||||||
|
|
||||||
from polygateway.errors import GovernanceBackendError
|
from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
|
||||||
from polygateway.types import GlobalLimits, SourceConfig, SourceStats
|
from polygateway.types import GlobalLimits, SourceConfig, SourceStats
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
@@ -195,7 +195,7 @@ class RedisLimiter:
|
|||||||
def _cfg(self, source_key: str) -> SourceConfig:
|
def _cfg(self, source_key: str) -> SourceConfig:
|
||||||
cfg = self._sources.get(source_key)
|
cfg = self._sources.get(source_key)
|
||||||
if cfg is None:
|
if cfg is None:
|
||||||
raise GovernanceBackendError(f"未知源 {source_key!r}(scope={self._scope})")
|
raise SourceNotConfiguredError(f"未知源 {source_key!r}(scope={self._scope})")
|
||||||
return cfg
|
return cfg
|
||||||
|
|
||||||
def _lease_keys(self, source_key: str) -> tuple[str, str]:
|
def _lease_keys(self, source_key: str) -> tuple[str, str]:
|
||||||
@@ -247,7 +247,7 @@ class RedisLimiter:
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 try_acquire 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 try_acquire 失败: {exc}", scope=self._scope) from exc
|
||||||
if ok != 1:
|
if ok != 1:
|
||||||
return None
|
return None
|
||||||
return _RedisPermit(self, source_key, lease_id, est_tokens, window)
|
return _RedisPermit(self, source_key, lease_id, est_tokens, window)
|
||||||
@@ -265,14 +265,14 @@ class RedisLimiter:
|
|||||||
try:
|
try:
|
||||||
await self._release_lua(keys=[gl, sl], args=[lease_id])
|
await self._release_lua(keys=[gl, sl], args=[lease_id])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 release 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 release 失败: {exc}", scope=self._scope) from exc
|
||||||
|
|
||||||
async def _settle_tpm(self, source_key: str, delta: int, window: int) -> None:
|
async def _settle_tpm(self, source_key: str, delta: int, window: int) -> None:
|
||||||
wk = self._window_keys(source_key, window)
|
wk = self._window_keys(source_key, window)
|
||||||
try:
|
try:
|
||||||
await self._settle_lua(keys=[wk["g_tpm"], wk["s_tpm"]], args=[delta, _WINDOW_TTL_S])
|
await self._settle_lua(keys=[wk["g_tpm"], wk["s_tpm"]], args=[delta, _WINDOW_TTL_S])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 settle 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 settle 失败: {exc}", scope=self._scope) from exc
|
||||||
|
|
||||||
async def source_stats(self, source_key: str) -> SourceStats:
|
async def source_stats(self, source_key: str) -> SourceStats:
|
||||||
"""当前窗口快照;读侧 clamp ≥0(展示口径,存储保留负值)。"""
|
"""当前窗口快照;读侧 clamp ≥0(展示口径,存储保留负值)。"""
|
||||||
@@ -283,7 +283,7 @@ class RedisLimiter:
|
|||||||
wk = self._window_keys(source_key, window)
|
wk = self._window_keys(source_key, window)
|
||||||
res = await self._stats_lua(keys=[sl, wk["s_rpm"], wk["s_tpm"]])
|
res = await self._stats_lua(keys=[sl, wk["s_rpm"], wk["s_tpm"]])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 source_stats 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 source_stats 失败: {exc}", scope=self._scope) from exc
|
||||||
return SourceStats(
|
return SourceStats(
|
||||||
inflight=int(res[0]),
|
inflight=int(res[0]),
|
||||||
rpm_used=max(0, int(res[1])),
|
rpm_used=max(0, int(res[1])),
|
||||||
@@ -295,14 +295,14 @@ class RedisLimiter:
|
|||||||
try:
|
try:
|
||||||
await self._progress_mark_lua(keys=[self._progress_key()], args=[_PROGRESS_TTL_S])
|
await self._progress_mark_lua(keys=[self._progress_key()], args=[_PROGRESS_TTL_S])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 mark_progress 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 mark_progress 失败: {exc}", scope=self._scope) from exc
|
||||||
|
|
||||||
async def progress_age_s(self) -> float:
|
async def progress_age_s(self) -> float:
|
||||||
"""距上次全局成功的秒数;仅键缺失(-1)= 从未进展 → inf(CHS limiter.py:208)。"""
|
"""距上次全局成功的秒数;仅键缺失(-1)= 从未进展 → inf(CHS limiter.py:208)。"""
|
||||||
try:
|
try:
|
||||||
res = await self._progress_age_lua(keys=[self._progress_key()])
|
res = await self._progress_age_lua(keys=[self._progress_key()])
|
||||||
except RedisError as exc:
|
except RedisError as exc:
|
||||||
raise GovernanceBackendError(f"限流后端 progress_age_s 失败: {exc}") from exc
|
raise GovernanceBackendError(f"限流后端 progress_age_s 失败: {exc}", scope=self._scope) from exc
|
||||||
return float("inf") if int(res) == -1 else int(res) / 1000.0
|
return float("inf") if int(res) == -1 else int(res) / 1000.0
|
||||||
|
|
||||||
async def aclose(self) -> None:
|
async def aclose(self) -> None:
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -120,8 +121,8 @@ class EmbeddingClient:
|
|||||||
# 否则遥测会记录一个从未发出的采样参数(issue #4 决策 G)
|
# 否则遥测会记录一个从未发出的采样参数(issue #4 决策 G)
|
||||||
self._sources = strip_unsupported_extra_body(list(sources), path="embedding")
|
self._sources = strip_unsupported_extra_body(list(sources), path="embedding")
|
||||||
self._selector = selector
|
self._selector = selector
|
||||||
self._quota = QuotaGate(limiter)
|
self._quota = QuotaGate(limiter, scope=self._scope)
|
||||||
self._breaker = BreakerGate(breaker)
|
self._breaker = BreakerGate(breaker, scope=self._scope)
|
||||||
self._transport = transport
|
self._transport = transport
|
||||||
self._retry = retry
|
self._retry = retry
|
||||||
self._bp = backpressure
|
self._bp = backpressure
|
||||||
@@ -326,7 +327,7 @@ class EmbeddingClient:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
||||||
|
|||||||
@@ -5,8 +5,22 @@
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
SCOPE_REASONS = frozenset(
|
SCOPE_REASONS = frozenset(
|
||||||
{"circuit_open", "retry_exhausted", "stalled", "quota_exhausted", "no_sources"}
|
{
|
||||||
|
"circuit_open",
|
||||||
|
"retry_exhausted",
|
||||||
|
"stalled",
|
||||||
|
"quota_exhausted",
|
||||||
|
"no_sources",
|
||||||
|
"governance_backend_down", # issue #7: 限流/熔断后端故障(fail-closed → 整个 scope 发不出请求)
|
||||||
|
}
|
||||||
)
|
)
|
||||||
|
GOVERNANCE_BACKEND_RETRY_AFTER_S = 5.0
|
||||||
|
"""治理后端故障的建议重投间隔(秒)。
|
||||||
|
|
||||||
|
**不是环境配置项**——后端恢复时间物理上不可知(不同于熔断冷却有确定到期时刻),
|
||||||
|
故取一个保守固定值;下游有自己的退避策略时可忽略本字段。取 0 会让积压任务零延迟
|
||||||
|
同时冲击已挂掉的后端,把一次故障放大成一场风暴(issue #7 §3.2)。
|
||||||
|
"""
|
||||||
SOURCE_REASONS = frozenset(
|
SOURCE_REASONS = frozenset(
|
||||||
{
|
{
|
||||||
"network_error",
|
"network_error",
|
||||||
@@ -127,5 +141,38 @@ class AllSourcesExhausted(GatewayUnavailableError): # noqa: N818 — ARCH §6.1
|
|||||||
"""重试预算耗尽 / 无可用源 / 配额 fail-fast 等 scope 级失败。"""
|
"""重试预算耗尽 / 无可用源 / 配额 fail-fast 等 scope 级失败。"""
|
||||||
|
|
||||||
|
|
||||||
class GovernanceBackendError(PolyGatewayError):
|
class SourceNotConfiguredError(PolyGatewayError):
|
||||||
"""限流/熔断状态后端自身故障: 必须报错而非放行(防击穿网关,降级方向铁律)。"""
|
"""源名不在限流后端的配置字典中: 装配缺陷,正常不可达。
|
||||||
|
|
||||||
|
**有意不在** `GatewayUnavailableError` 之下: 它不是"暂时不可用"而是"配置写
|
||||||
|
错了",必须消耗失败预算进死信让人看见;归入可重投家族会让配置错误的任务永远
|
||||||
|
重投、永不告警——正是 issue #7 要修的那个 bug 的镜像(§3.4)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class GovernanceBackendError(GatewayUnavailableError):
|
||||||
|
"""限流/熔断状态后端自身故障: 必须报错而非放行(防击穿网关,降级方向铁律)。
|
||||||
|
|
||||||
|
继承 `GatewayUnavailableError`(issue #7): fail-closed 意味着整个 scope 一个
|
||||||
|
请求都发不出去,语义上即 scope 级不可用。此前它是 `PolyGatewayError` 的直接
|
||||||
|
子类,只写 `except GatewayUnavailableError` 的调用方接不住,后果是"Redis 抖
|
||||||
|
一下 → 积压任务消耗业务失败预算 → 进死信",而那是运维重启即可恢复的故障。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
message: str,
|
||||||
|
*,
|
||||||
|
scope: str,
|
||||||
|
retry_after_s: float = GOVERNANCE_BACKEND_RETRY_AFTER_S,
|
||||||
|
source_name: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
scope=scope,
|
||||||
|
reason="governance_backend_down",
|
||||||
|
retry_after_s=retry_after_s,
|
||||||
|
source_name=source_name,
|
||||||
|
)
|
||||||
|
# 父类会把 message 覆写为 "{scope} 网关暂时不可用: {reason}",而各构造点
|
||||||
|
# 携带的诊断串(如"限流后端 try_acquire 失败: ...")是排障主线索,必须保住
|
||||||
|
self.args = (message,)
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from polygateway.errors import GovernanceBackendError
|
from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from polygateway.ports import GateDecision, GateUpdate, ProviderGate
|
from polygateway.ports import GateDecision, GateUpdate, ProviderGate
|
||||||
@@ -14,49 +14,61 @@ if TYPE_CHECKING:
|
|||||||
class BreakerGate:
|
class BreakerGate:
|
||||||
"""RetryMW 面向熔断后端的唯一入口;包装一切后端异常。"""
|
"""RetryMW 面向熔断后端的唯一入口;包装一切后端异常。"""
|
||||||
|
|
||||||
def __init__(self, gate: ProviderGate) -> None:
|
def __init__(self, gate: ProviderGate, *, scope: str) -> None:
|
||||||
self._gate = gate
|
self._gate = gate
|
||||||
|
# 后端故障即 scope 级不可用,异常须携 scope 供调用方定位(issue #7 §3.3)
|
||||||
|
self._scope = scope
|
||||||
|
|
||||||
async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision:
|
async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision:
|
||||||
try:
|
try:
|
||||||
return await self._gate.try_enter(source.name, owner)
|
return await self._gate.try_enter(source.name, owner)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端故障(try_enter): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"熔断后端故障(try_enter): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def record_success(
|
async def record_success(
|
||||||
self, entry: GateDecision, *, count_attempt: bool = True
|
self, entry: GateDecision, *, count_attempt: bool = True
|
||||||
) -> GateUpdate:
|
) -> GateUpdate:
|
||||||
try:
|
try:
|
||||||
return await self._gate.record_success(entry, count_attempt=count_attempt)
|
return await self._gate.record_success(entry, count_attempt=count_attempt)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端故障(record_success): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"熔断后端故障(record_success): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def record_failure(
|
async def record_failure(
|
||||||
self, entry: GateDecision, reason: str, force_open: bool
|
self, entry: GateDecision, reason: str, force_open: bool
|
||||||
) -> GateUpdate:
|
) -> GateUpdate:
|
||||||
try:
|
try:
|
||||||
return await self._gate.record_failure(entry, reason, force_open)
|
return await self._gate.record_failure(entry, reason, force_open)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端故障(record_failure): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"熔断后端故障(record_failure): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def release_probe(self, entry: GateDecision) -> GateUpdate:
|
async def release_probe(self, entry: GateDecision) -> GateUpdate:
|
||||||
try:
|
try:
|
||||||
return await self._gate.release_probe(entry)
|
return await self._gate.release_probe(entry)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端故障(release_probe): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"熔断后端故障(release_probe): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def retry_after_s(self, sources: tuple[str, ...]) -> float:
|
async def retry_after_s(self, sources: tuple[str, ...]) -> float:
|
||||||
try:
|
try:
|
||||||
return await self._gate.retry_after_s(sources)
|
return await self._gate.retry_after_s(sources)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"熔断后端故障(retry_after_s): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"熔断后端故障(retry_after_s): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from polygateway.errors import GovernanceBackendError
|
from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from polygateway.ports import Permit, RateLimiter
|
from polygateway.ports import Permit, RateLimiter
|
||||||
@@ -18,37 +18,47 @@ if TYPE_CHECKING:
|
|||||||
class QuotaGate:
|
class QuotaGate:
|
||||||
"""RetryMW 面向限流后端的唯一入口;包装一切后端异常。"""
|
"""RetryMW 面向限流后端的唯一入口;包装一切后端异常。"""
|
||||||
|
|
||||||
def __init__(self, limiter: RateLimiter) -> None:
|
def __init__(self, limiter: RateLimiter, *, scope: str) -> None:
|
||||||
self._limiter = limiter
|
self._limiter = limiter
|
||||||
|
# 后端故障即 scope 级不可用,异常须携 scope 供调用方定位(issue #7 §3.3)
|
||||||
|
self._scope = scope
|
||||||
|
|
||||||
async def try_acquire(self, source: SourceConfig) -> Permit | None:
|
async def try_acquire(self, source: SourceConfig) -> Permit | None:
|
||||||
try:
|
try:
|
||||||
return await self._limiter.try_acquire(source.name, source.effective_est_tokens())
|
return await self._limiter.try_acquire(source.name, source.effective_est_tokens())
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"限流后端故障(try_acquire): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"限流后端故障(try_acquire): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def stats(self, source: SourceConfig) -> SourceStats:
|
async def stats(self, source: SourceConfig) -> SourceStats:
|
||||||
try:
|
try:
|
||||||
return await self._limiter.source_stats(source.name)
|
return await self._limiter.source_stats(source.name)
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"限流后端故障(source_stats): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"限流后端故障(source_stats): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def mark_progress(self) -> None:
|
async def mark_progress(self) -> None:
|
||||||
try:
|
try:
|
||||||
await self._limiter.mark_progress()
|
await self._limiter.mark_progress()
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"限流后端故障(mark_progress): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"限流后端故障(mark_progress): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|
||||||
async def progress_age_s(self) -> float:
|
async def progress_age_s(self) -> float:
|
||||||
try:
|
try:
|
||||||
return await self._limiter.progress_age_s()
|
return await self._limiter.progress_age_s()
|
||||||
except GovernanceBackendError:
|
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||||
raise
|
raise
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise GovernanceBackendError(f"限流后端故障(progress_age_s): {exc}") from exc
|
raise GovernanceBackendError(
|
||||||
|
f"限流后端故障(progress_age_s): {exc}", scope=self._scope
|
||||||
|
) from exc
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -183,8 +184,8 @@ class RetryMW:
|
|||||||
self._scope = scope
|
self._scope = scope
|
||||||
self._sources = list(sources)
|
self._sources = list(sources)
|
||||||
self._selector = selector
|
self._selector = selector
|
||||||
self._quota = QuotaGate(limiter)
|
self._quota = QuotaGate(limiter, scope=self._scope)
|
||||||
self._breaker = BreakerGate(gate)
|
self._breaker = BreakerGate(gate, scope=self._scope)
|
||||||
self._transport = transport
|
self._transport = transport
|
||||||
self._retry = retry
|
self._retry = retry
|
||||||
self._bp = backpressure
|
self._bp = backpressure
|
||||||
@@ -401,7 +402,7 @@ class RetryMW:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
def _feed_outcome(self, source_name: str, ok: bool) -> None:
|
def _feed_outcome(self, source_name: str, ok: bool) -> None:
|
||||||
|
|||||||
@@ -12,11 +12,16 @@ import asyncio
|
|||||||
import json
|
import json
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
|
from dataclasses import dataclass
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from polygateway.errors import GatewayUnavailableError, GovernanceBackendError
|
from polygateway.errors import (
|
||||||
|
GatewayUnavailableError,
|
||||||
|
GovernanceBackendError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
|
)
|
||||||
from polygateway.middleware.cache import digest_messages
|
from polygateway.middleware.cache import digest_messages
|
||||||
from polygateway.types import canonical_sampling_json, merge_sampling
|
from polygateway.types import canonical_sampling_json, merge_sampling
|
||||||
|
|
||||||
@@ -28,6 +33,44 @@ if TYPE_CHECKING:
|
|||||||
from polygateway.types import ChatRequest, LLMResponse, SourceConfig
|
from polygateway.types import ChatRequest, LLMResponse, SourceConfig
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _AttemptUsage:
|
||||||
|
"""一次尝试的用量视图;默认值即"失败尝试"档(无用量可言,记 0 并标 unavailable)。
|
||||||
|
|
||||||
|
存在的理由是把 `emit_attempt` 里逐字段重复的 `X if response else Y` 收敛为
|
||||||
|
一处判定——十处三元把该方法推到圈复杂度 C,而它们表达的是同一件事。
|
||||||
|
"""
|
||||||
|
|
||||||
|
response_text: str = ""
|
||||||
|
thinking: str = ""
|
||||||
|
prompt_tokens: int = 0
|
||||||
|
completion_tokens: int = 0
|
||||||
|
usage_source: str = "unavailable"
|
||||||
|
ttft_ms: float | None = None
|
||||||
|
max_inter_token_ms: float | None = None
|
||||||
|
cached_prompt_tokens: int | None = None
|
||||||
|
model_reported: str | None = None
|
||||||
|
reasoning_tokens: int | None = None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def of(cls, response: LLMResponse | None) -> _AttemptUsage:
|
||||||
|
"""从响应取用量;`None`(失败尝试)返回全默认视图。"""
|
||||||
|
if response is None:
|
||||||
|
return cls()
|
||||||
|
return cls(
|
||||||
|
response_text=response.content,
|
||||||
|
thinking=response.thinking,
|
||||||
|
prompt_tokens=response.prompt_tokens,
|
||||||
|
completion_tokens=response.completion_tokens,
|
||||||
|
usage_source=response.usage_source,
|
||||||
|
ttft_ms=response.ttft_ms,
|
||||||
|
max_inter_token_ms=response.max_inter_token_ms,
|
||||||
|
cached_prompt_tokens=response.cached_prompt_tokens,
|
||||||
|
model_reported=response.model_reported,
|
||||||
|
reasoning_tokens=response.reasoning_tokens,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TelemetryEmitter:
|
class TelemetryEmitter:
|
||||||
"""从请求与结果组装 21 字段并写入 recorder;一切写失败降级 warning。"""
|
"""从请求与结果组装 21 字段并写入 recorder;一切写失败降级 warning。"""
|
||||||
|
|
||||||
@@ -46,25 +89,26 @@ class TelemetryEmitter:
|
|||||||
error: str | None,
|
error: str | None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""逐次尝试记录(RetryMW 调用);失败尝试无用量可言,记 0 并标 unavailable。"""
|
"""逐次尝试记录(RetryMW 调用);失败尝试无用量可言,记 0 并标 unavailable。"""
|
||||||
|
usage = _AttemptUsage.of(response)
|
||||||
await self._record(
|
await self._record(
|
||||||
request=request,
|
request=request,
|
||||||
call_id=call_id,
|
call_id=call_id,
|
||||||
model=source.model,
|
model=source.model,
|
||||||
provider=source.provider,
|
provider=source.provider,
|
||||||
source_name=source.name,
|
source_name=source.name,
|
||||||
response_text=response.content if response else "",
|
response_text=usage.response_text,
|
||||||
thinking=response.thinking if response else "",
|
thinking=usage.thinking,
|
||||||
prompt_tokens=response.prompt_tokens if response else 0,
|
prompt_tokens=usage.prompt_tokens,
|
||||||
completion_tokens=response.completion_tokens if response else 0,
|
completion_tokens=usage.completion_tokens,
|
||||||
usage_source=response.usage_source if response else "unavailable",
|
usage_source=usage.usage_source,
|
||||||
latency_ms=latency_ms,
|
latency_ms=latency_ms,
|
||||||
ttft_ms=response.ttft_ms if response else None,
|
ttft_ms=usage.ttft_ms,
|
||||||
max_inter_token_ms=response.max_inter_token_ms if response else None,
|
max_inter_token_ms=usage.max_inter_token_ms,
|
||||||
cache_hit=False,
|
cache_hit=False,
|
||||||
error=error,
|
error=error,
|
||||||
cached_prompt_tokens=response.cached_prompt_tokens if response else None,
|
cached_prompt_tokens=usage.cached_prompt_tokens,
|
||||||
model_reported=response.model_reported if response else None,
|
model_reported=usage.model_reported,
|
||||||
reasoning_tokens=response.reasoning_tokens if response else None,
|
reasoning_tokens=usage.reasoning_tokens,
|
||||||
# 唯一有"生效源"的入口,故是唯一能并上 extra_body 的(设计决策 D)
|
# 唯一有"生效源"的入口,故是唯一能并上 extra_body 的(设计决策 D)
|
||||||
sampling=canonical_sampling_json(merge_sampling(source.extra_body, request.sampling)),
|
sampling=canonical_sampling_json(merge_sampling(source.extra_body, request.sampling)),
|
||||||
)
|
)
|
||||||
@@ -207,7 +251,7 @@ class TelemetryMW:
|
|||||||
started = self._now()
|
started = self._now()
|
||||||
try:
|
try:
|
||||||
response = await call_next(request)
|
response = await call_next(request)
|
||||||
except (GatewayUnavailableError, GovernanceBackendError) as exc:
|
except (GatewayUnavailableError, GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
await self._emitter.emit_terminal_failure(
|
await self._emitter.emit_terminal_failure(
|
||||||
request=request,
|
request=request,
|
||||||
call_id=str(uuid.uuid4()),
|
call_id=str(uuid.uuid4()),
|
||||||
|
|||||||
@@ -30,6 +30,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -119,8 +120,8 @@ class OcrClient:
|
|||||||
self._sources = strip_unsupported_extra_body(list(sources), path="OCR")
|
self._sources = strip_unsupported_extra_body(list(sources), path="OCR")
|
||||||
self._selector = selector
|
self._selector = selector
|
||||||
self._feed_health = isinstance(selector, OutcomeAwareSelector)
|
self._feed_health = isinstance(selector, OutcomeAwareSelector)
|
||||||
self._quota = QuotaGate(limiter)
|
self._quota = QuotaGate(limiter, scope=self._scope)
|
||||||
self._breaker = BreakerGate(breaker)
|
self._breaker = BreakerGate(breaker, scope=self._scope)
|
||||||
self._transport = transport
|
self._transport = transport
|
||||||
self._retry = retry
|
self._retry = retry
|
||||||
self._bp = backpressure
|
self._bp = backpressure
|
||||||
@@ -360,7 +361,7 @@ class OcrClient:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
async def _settle_and_release(self, permit: Permit) -> None:
|
async def _settle_and_release(self, permit: Permit) -> None:
|
||||||
|
|||||||
@@ -16,7 +16,11 @@ import pytest
|
|||||||
from polygateway.backends.redis.breaker import RedisGate
|
from polygateway.backends.redis.breaker import RedisGate
|
||||||
from polygateway.backends.redis.limiter import RedisLimiter
|
from polygateway.backends.redis.limiter import RedisLimiter
|
||||||
from polygateway.client import GatewayClient
|
from polygateway.client import GatewayClient
|
||||||
from polygateway.errors import AllSourcesExhausted, GovernanceBackendError
|
from polygateway.errors import (
|
||||||
|
AllSourcesExhausted,
|
||||||
|
GatewayUnavailableError,
|
||||||
|
GovernanceBackendError,
|
||||||
|
)
|
||||||
from polygateway.sources import RoundRobinSelector
|
from polygateway.sources import RoundRobinSelector
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
BackpressurePolicy,
|
BackpressurePolicy,
|
||||||
@@ -225,7 +229,11 @@ async def test_cancel_in_flight_releases_lease(clients):
|
|||||||
|
|
||||||
|
|
||||||
async def test_redis_down_admission_fails_closed():
|
async def test_redis_down_admission_fails_closed():
|
||||||
"""Redis 不可达 → 准入侧抛 GovernanceBackendError,绝不放行(库铁律)。"""
|
"""Redis 不可达 → 准入侧报错绝不放行(库铁律),且以 scope 级形态到达调用方。
|
||||||
|
|
||||||
|
issue #7: 调用方只写 `except GatewayUnavailableError` 就该覆盖后端故障——
|
||||||
|
真实 Redis 掉线是这条链路唯一的端到端证据,故断言收紧到 scope 级语义。
|
||||||
|
"""
|
||||||
import redis.asyncio as aioredis
|
import redis.asyncio as aioredis
|
||||||
|
|
||||||
dead = aioredis.from_url(
|
dead = aioredis.from_url(
|
||||||
@@ -240,9 +248,12 @@ async def test_redis_down_admission_fails_closed():
|
|||||||
lease_ttl_s=30.0,
|
lease_ttl_s=30.0,
|
||||||
)
|
)
|
||||||
gate = RedisGate(config=_CFG, redis=dead, scope="t-dead")
|
gate = RedisGate(config=_CFG, redis=dead, scope="t-dead")
|
||||||
with pytest.raises(GovernanceBackendError):
|
for call in (limiter.try_acquire("s1", 0), gate.try_enter("s1", "w")):
|
||||||
await limiter.try_acquire("s1", 0)
|
with pytest.raises(GatewayUnavailableError) as ei:
|
||||||
with pytest.raises(GovernanceBackendError):
|
await call
|
||||||
await gate.try_enter("s1", "w")
|
assert isinstance(ei.value, GovernanceBackendError)
|
||||||
|
assert ei.value.reason == "governance_backend_down"
|
||||||
|
assert ei.value.scope == "t-dead"
|
||||||
|
assert ei.value.retry_after_s > 0
|
||||||
finally:
|
finally:
|
||||||
await dead.aclose()
|
await dead.aclose()
|
||||||
|
|||||||
@@ -11,7 +11,14 @@ import pytest
|
|||||||
|
|
||||||
from polygateway.backends.memory.breaker import InMemoryGate
|
from polygateway.backends.memory.breaker import InMemoryGate
|
||||||
from polygateway.backends.memory.limiter import InMemoryLimiter
|
from polygateway.backends.memory.limiter import InMemoryLimiter
|
||||||
from polygateway.errors import AllSourcesExhausted, GovernanceBackendError, TransientError
|
from polygateway.errors import (
|
||||||
|
AllSourcesExhausted,
|
||||||
|
GatewayUnavailableError,
|
||||||
|
GovernanceBackendError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
|
TransientError,
|
||||||
|
)
|
||||||
|
from polygateway.middleware.ratelimit import QuotaGate
|
||||||
from polygateway.middleware.retry import RetryMW, backoff_delay
|
from polygateway.middleware.retry import RetryMW, backoff_delay
|
||||||
from polygateway.sources import RoundRobinSelector, SourceCooldownMemo
|
from polygateway.sources import RoundRobinSelector, SourceCooldownMemo
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
@@ -173,17 +180,23 @@ class TestStallQuadrants:
|
|||||||
|
|
||||||
class _GateSuccessBroken(InMemoryGate):
|
class _GateSuccessBroken(InMemoryGate):
|
||||||
async def record_success(self, entry):
|
async def record_success(self, entry):
|
||||||
raise GovernanceBackendError("redis 抖动")
|
raise GovernanceBackendError("redis 抖动", scope="llm")
|
||||||
|
|
||||||
|
|
||||||
class _GateFailureBroken(InMemoryGate):
|
class _GateFailureBroken(InMemoryGate):
|
||||||
async def record_failure(self, entry, reason, force_open):
|
async def record_failure(self, entry, reason, force_open):
|
||||||
raise GovernanceBackendError("redis 抖动")
|
raise GovernanceBackendError("redis 抖动", scope="llm")
|
||||||
|
|
||||||
|
|
||||||
class _LimiterProgressBroken(InMemoryLimiter):
|
class _LimiterProgressBroken(InMemoryLimiter):
|
||||||
async def mark_progress(self):
|
async def mark_progress(self):
|
||||||
raise GovernanceBackendError("redis 抖动")
|
raise GovernanceBackendError("redis 抖动", scope="llm")
|
||||||
|
|
||||||
|
|
||||||
|
class _GateSuccessMisconfigured(InMemoryGate):
|
||||||
|
# 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常
|
||||||
|
async def record_success(self, entry, *, count_attempt: bool = True):
|
||||||
|
raise SourceNotConfiguredError("未知源 's1'(scope=llm)")
|
||||||
|
|
||||||
|
|
||||||
class TestAccountingDegradation:
|
class TestAccountingDegradation:
|
||||||
@@ -200,6 +213,24 @@ class TestAccountingDegradation:
|
|||||||
resp = await mw(_REQ)
|
resp = await mw(_REQ)
|
||||||
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
|
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
|
||||||
|
|
||||||
|
async def test_assembly_defect_on_accounting_path_also_degrades(self):
|
||||||
|
"""记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。
|
||||||
|
|
||||||
|
`SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若
|
||||||
|
`_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、
|
||||||
|
销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账
|
||||||
|
方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。
|
||||||
|
"""
|
||||||
|
clock = FakeClock()
|
||||||
|
src = make_source()
|
||||||
|
limiter = InMemoryLimiter(
|
||||||
|
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
|
||||||
|
)
|
||||||
|
gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock)
|
||||||
|
mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate)
|
||||||
|
resp = await mw(_REQ)
|
||||||
|
assert resp.content == "ok"
|
||||||
|
|
||||||
async def test_mark_progress_failure_does_not_lose_response(self):
|
async def test_mark_progress_failure_does_not_lose_response(self):
|
||||||
clock = FakeClock()
|
clock = FakeClock()
|
||||||
src = make_source()
|
src = make_source()
|
||||||
@@ -252,6 +283,91 @@ class TestQuotaGateProgressAge:
|
|||||||
async def progress_age_s(self):
|
async def progress_age_s(self):
|
||||||
raise OSError("down")
|
raise OSError("down")
|
||||||
|
|
||||||
assert await QuotaGate(_L()).progress_age_s() == 12.5
|
assert await QuotaGate(_L(), scope="llm").progress_age_s() == 12.5
|
||||||
with pytest.raises(GovernanceBackendError):
|
with pytest.raises(GovernanceBackendError):
|
||||||
await QuotaGate(_Broken()).progress_age_s()
|
await QuotaGate(_Broken(), scope="llm").progress_age_s()
|
||||||
|
|
||||||
|
|
||||||
|
class TestUnknownSourceIsAssemblyDefect:
|
||||||
|
"""未知源 = 限流后端的源名单与治理循环对不上,是装配缺陷不是后端故障。
|
||||||
|
|
||||||
|
两个后端行为必须一致(Redis 版对应用例在 `test_redis_key_layout.py::
|
||||||
|
TestConversions::test_unknown_source_rejected`);内存版此前无覆盖,
|
||||||
|
该分支从未被测过(issue #7 §3.4)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_memory_limiter_rejects_unknown_source(self):
|
||||||
|
limiter = InMemoryLimiter(
|
||||||
|
scope="llm", sources={"s1": make_source("s1")}, global_limits=_NO_GLOBAL
|
||||||
|
)
|
||||||
|
with pytest.raises(SourceNotConfiguredError) as ei:
|
||||||
|
limiter._cfg("nope")
|
||||||
|
# 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信
|
||||||
|
assert not isinstance(ei.value, GatewayUnavailableError)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", ["try_acquire", "stats"])
|
||||||
|
async def test_survives_the_quota_gate_wrapper(self, method):
|
||||||
|
"""必须穿透 QuotaGate,否则整个拆分在生产路径上等于没做。
|
||||||
|
|
||||||
|
上面两条(以及 redis 版)打的都是私有 `_cfg`,绕过了包装器。而治理循环
|
||||||
|
只经 QuotaGate 访问后端,包装器的 `except Exception` 会把装配缺陷重新
|
||||||
|
包成 `GovernanceBackendError`——下游又拿到可重投异常,永远重投不告警。
|
||||||
|
"""
|
||||||
|
src = make_source("s1")
|
||||||
|
# 限流后端的源名单与治理循环拿到的源对不上 = 装配缺陷
|
||||||
|
limiter = InMemoryLimiter(
|
||||||
|
scope="llm", sources={"other": src}, global_limits=_NO_GLOBAL
|
||||||
|
)
|
||||||
|
gate = QuotaGate(limiter, scope="llm")
|
||||||
|
with pytest.raises(SourceNotConfiguredError) as ei:
|
||||||
|
await getattr(gate, method)(src)
|
||||||
|
assert not isinstance(ei.value, GatewayUnavailableError)
|
||||||
|
|
||||||
|
|
||||||
|
class TestGateFailuresReachCallersAsScopeLevel:
|
||||||
|
"""闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
|
||||||
|
|
||||||
|
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
|
||||||
|
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
|
||||||
|
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
|
||||||
|
的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s,
|
||||||
|
`BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly`
|
||||||
|
包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def test_try_acquire_failure_is_scope_level(self):
|
||||||
|
from polygateway.middleware.ratelimit import QuotaGate
|
||||||
|
|
||||||
|
class _Broken:
|
||||||
|
async def try_acquire(self, name, est):
|
||||||
|
raise OSError("down")
|
||||||
|
|
||||||
|
with pytest.raises(GatewayUnavailableError) as ei:
|
||||||
|
await QuotaGate(_Broken(), scope="LLM").try_acquire(make_source("s1"))
|
||||||
|
assert ei.value.scope == "llm"
|
||||||
|
assert ei.value.reason == "governance_backend_down"
|
||||||
|
assert ei.value.retry_after_s > 0 # 0 会让积压任务零延迟冲击已挂的后端
|
||||||
|
|
||||||
|
async def test_try_enter_failure_is_scope_level(self):
|
||||||
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
|
|
||||||
|
class _Broken:
|
||||||
|
async def try_enter(self, name, owner):
|
||||||
|
raise OSError("down")
|
||||||
|
|
||||||
|
with pytest.raises(GatewayUnavailableError) as ei:
|
||||||
|
await BreakerGate(_Broken(), scope="LLM").try_enter(make_source("s1"), "owner")
|
||||||
|
assert ei.value.scope == "llm"
|
||||||
|
assert ei.value.reason == "governance_backend_down"
|
||||||
|
|
||||||
|
async def test_progress_age_failure_is_scope_level(self):
|
||||||
|
from polygateway.middleware.ratelimit import QuotaGate
|
||||||
|
|
||||||
|
class _Broken:
|
||||||
|
async def progress_age_s(self):
|
||||||
|
raise OSError("down")
|
||||||
|
|
||||||
|
with pytest.raises(GatewayUnavailableError) as ei:
|
||||||
|
await QuotaGate(_Broken(), scope="LLM").progress_age_s()
|
||||||
|
assert ei.value.scope == "llm"
|
||||||
|
assert ei.value.reason == "governance_backend_down"
|
||||||
|
|||||||
@@ -3,6 +3,8 @@
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from polygateway.errors import (
|
from polygateway.errors import (
|
||||||
|
GOVERNANCE_BACKEND_RETRY_AFTER_S,
|
||||||
|
SCOPE_REASONS,
|
||||||
AllSourcesExhausted,
|
AllSourcesExhausted,
|
||||||
CircuitOpenError,
|
CircuitOpenError,
|
||||||
GatewayUnavailableError,
|
GatewayUnavailableError,
|
||||||
@@ -11,6 +13,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -86,6 +89,56 @@ class TestGatewayUnavailable:
|
|||||||
class TestBackendFailure:
|
class TestBackendFailure:
|
||||||
def test_governance_backend_error_is_not_transient(self):
|
def test_governance_backend_error_is_not_transient(self):
|
||||||
"""限流/熔断后端故障必须报错不放行,且不落入可重试分类。"""
|
"""限流/熔断后端故障必须报错不放行,且不落入可重试分类。"""
|
||||||
exc = GovernanceBackendError("redis down")
|
exc = GovernanceBackendError("redis down", scope="llm")
|
||||||
assert isinstance(exc, PolyGatewayError)
|
assert isinstance(exc, PolyGatewayError)
|
||||||
assert not isinstance(exc, TransientError)
|
assert not isinstance(exc, TransientError)
|
||||||
|
|
||||||
|
def test_is_scope_level_unavailability(self):
|
||||||
|
"""fail-closed 时整个 scope 一个请求都发不出去,调用方一条 except 应覆盖(issue #7)。"""
|
||||||
|
exc = GovernanceBackendError("限流后端 try_acquire 失败: boom", scope="LLM")
|
||||||
|
assert isinstance(exc, GatewayUnavailableError)
|
||||||
|
assert exc.reason == "governance_backend_down"
|
||||||
|
assert exc.scope == "llm" # 与既有 scope 级异常同款: 归一化小写
|
||||||
|
assert exc.retry_after_s == GOVERNANCE_BACKEND_RETRY_AFTER_S
|
||||||
|
|
||||||
|
def test_diagnostic_message_survives_reparenting(self):
|
||||||
|
"""父类把 message 覆写为模板串,而各构造点的诊断串是排障主线索(§3.5)。"""
|
||||||
|
exc = GovernanceBackendError("熔断后端 try_enter 失败: boom", scope="llm")
|
||||||
|
assert str(exc) == "熔断后端 try_enter 失败: boom"
|
||||||
|
|
||||||
|
def test_retry_after_overridable(self):
|
||||||
|
exc = GovernanceBackendError("redis down", scope="llm", retry_after_s=30.0)
|
||||||
|
assert exc.retry_after_s == 30.0
|
||||||
|
|
||||||
|
|
||||||
|
class TestSourceNotConfigured:
|
||||||
|
"""装配缺陷有意留在 scope 级家族之外(issue #7 §3.4,Q1 人类拍板)。"""
|
||||||
|
|
||||||
|
def test_is_domain_error_but_not_scope_level(self):
|
||||||
|
exc = SourceNotConfiguredError("未知源 'nope'(scope=llm)")
|
||||||
|
assert isinstance(exc, PolyGatewayError)
|
||||||
|
# 关键断言: 归入可重投家族会让配置写错的任务永远重投、永不进死信
|
||||||
|
assert not isinstance(exc, GatewayUnavailableError)
|
||||||
|
|
||||||
|
def test_exported_at_package_top_level(self):
|
||||||
|
import polygateway
|
||||||
|
|
||||||
|
assert polygateway.SourceNotConfiguredError is SourceNotConfiguredError
|
||||||
|
assert "SourceNotConfiguredError" in polygateway.__all__
|
||||||
|
|
||||||
|
|
||||||
|
class TestGovernanceBackendReason:
|
||||||
|
"""新 scope 级 reason 值域(issue #7 §3.1)。"""
|
||||||
|
|
||||||
|
def test_reason_admitted_to_scope_domain(self):
|
||||||
|
assert "governance_backend_down" in SCOPE_REASONS
|
||||||
|
|
||||||
|
def test_gateway_unavailable_accepts_the_new_reason(self):
|
||||||
|
exc = AllSourcesExhausted(
|
||||||
|
scope="LLM", reason="governance_backend_down", retry_after_s=0.0
|
||||||
|
)
|
||||||
|
assert exc.reason == "governance_backend_down"
|
||||||
|
|
||||||
|
def test_retry_after_default_is_non_zero(self):
|
||||||
|
"""取 0 会让积压任务零延迟冲击已挂掉的后端(§3.2)。"""
|
||||||
|
assert GOVERNANCE_BACKEND_RETRY_AFTER_S > 0
|
||||||
|
|||||||
@@ -68,10 +68,13 @@ class TestConversions:
|
|||||||
_limiter(lease_ttl_s=0)
|
_limiter(lease_ttl_s=0)
|
||||||
|
|
||||||
def test_unknown_source_rejected(self):
|
def test_unknown_source_rejected(self):
|
||||||
from polygateway.errors import GovernanceBackendError
|
"""未知源是装配缺陷,不是后端故障(issue #7 §3.4)。"""
|
||||||
|
from polygateway.errors import GatewayUnavailableError, SourceNotConfiguredError
|
||||||
|
|
||||||
with pytest.raises(GovernanceBackendError):
|
with pytest.raises(SourceNotConfiguredError) as ei:
|
||||||
_limiter()._cfg("nope")
|
_limiter()._cfg("nope")
|
||||||
|
# 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信
|
||||||
|
assert not isinstance(ei.value, GatewayUnavailableError)
|
||||||
|
|
||||||
|
|
||||||
class TestLuaFidelity:
|
class TestLuaFidelity:
|
||||||
|
|||||||
Reference in New Issue
Block a user