3 Commits

Author SHA1 Message Date
iomgaa 9c2824ce8a fix: admit governance backend failures into scope-level unavailability (issue #7)
A fail-closed limiter or breaker backend means the scope cannot emit a
single request, yet GovernanceBackendError sat directly under
PolyGatewayError. A caller writing only `except GatewayUnavailableError`
dropped it into the catch-all branch, so a Redis blip burned a backlog's
business failure budget into the dead letter queue over a fault a restart
would clear. It now inherits GatewayUnavailableError with a
governance_backend_down reason and a 5 second retry_after_s.

The two unknown-source sites split out into SourceNotConfiguredError,
deliberately outside the retryable family: a misconfigured source name
must burn its budget and surface rather than retry forever in silence.

README now states which errors reach callers and which the retry loop
absorbs. TransientError and SourceDeadError read like caller contracts but
never arrive, and a downstream project wrote a whole design section on
that false premise before checking the source.

Independent verification caught the split not actually holding on the only
path production uses, and caught the fix for that opening a second hole on
the accounting path. Both are fixed and pinned by tests that go through
the wrappers rather than the private methods underneath them.
2026-08-06 06:55:38 -04:00
iomgaa 5853c3f8ff fix: keep the accounting path degrading after the wrapper change
Letting SourceNotConfiguredError through the gate wrappers opened a hole
the recheck caught: _record_quietly only degrades GovernanceBackendError,
so an assembly defect raised from the accounting side would now escape and
destroy a response from a call that had already genuinely succeeded. That
inverts the exact invariant _record_quietly exists to hold.

Widening _record_quietly is the right fix rather than narrowing the
wrappers, because that layer degrades by what the path is (accounting, the
call is already done) rather than by which error type shows up. Narrowing
would have left 4 of 9 wrapper methods as exceptions to a rule nobody can
remember.

No backend raises it from an accounting method today, so this is a
guardrail for whoever adds source-name validation to a breaker backend.

The stub that first reported this green was wrong: its record_success
lacked count_attempt, so it raised TypeError and the wrapper relabeled it.
Fixed signature, then the test failed as it should have.

Also finishes the three-to-five leak path correction across the four
remaining spots, including the wiki summary card that indexes this design.
2026-08-06 06:39:52 -04:00
iomgaa a57a5cea72 fix: let assembly defects pierce the gate wrappers
Independent verification caught that the split shipped in the previous
commit did not actually hold on the only path production uses. The gate
wrappers re-raise GovernanceBackendError but nothing else, so
SourceNotConfiguredError fell into the following `except Exception` and
came back out as a governance_backend_down failure with retry_after_s=5.0.
A misconfigured source name would still retry forever and never surface.

The existing tests missed it because both of them call the private _cfg()
directly, one layer below the wrapper the governance loops actually go
through. The regression test goes through QuotaGate.

telemetry.py has to widen its terminal catch in the same commit: once the
wrapper stops relabeling the error, it is no longer a GovernanceBackendError,
and it is raised before any attempt exists, so the path would have recorded
no telemetry at all.

Also corrects the leak path count from three to five. QuotaGate.stats and
BreakerGate.retry_after_s are not wrapped by _record_quietly either.
2026-08-06 05:57:50 -04:00
12 changed files with 144 additions and 39 deletions
+2 -2
View File
@@ -19,8 +19,8 @@
### 下游请读 ### 下游请读
- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`**,与 `AllSourcesExhausted` 同款;`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。 - **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`**,与 `AllSourcesExhausted` 同款(`per_source_reasons` 属性存在但恒为 `{}`——后端故障不针对具体某个源);`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。
- **条闸门路径**(`try_acquire` / `try_enter` / `progress_age_s`)的后端故障会到达调用方;记账路径(`record_success`)仍被 `_record_quietly` 降级为 warning,这个分工不变。 - **条闸门路径**的后端故障会到达调用方: `QuotaGate``try_acquire` / `stats` / `progress_age_s`,`BreakerGate``try_enter` / `retry_after_s`。记账路径(`record_success` / `record_failure` / `release_probe` / `mark_progress`)仍被 `_record_quietly` 降级为 warning,这个分工不变。
- **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。 - **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。
## 1.0.6(2026-08-02) ## 1.0.6(2026-08-02)
@@ -20,7 +20,7 @@
| Issue 原文 | 实际情况 | | Issue 原文 | 实际情况 |
|---|---| |---|---|
| 泄漏路径为 `try_enter` / `try_acquire` 两条 | ****`middleware/retry.py:216` 每轮循环开头的 `progress_age_s()` 同样在 catch 之外,直达调用方 | | 泄漏路径为 `try_enter` / `try_acquire` 两条 | ****(设计初稿写"三条",2026-08-06 独立验证时核出遗漏两条并订正): `QuotaGate``try_acquire` / `stats`(`retry.py:249`)/ `progress_age_s`(`retry.py:216``:305`),`BreakerGate``try_enter` / `retry_after_s`(`retry.py:292``:310`)。判据是该调用点是否被 `_record_quietly` 包裹——未包裹即直达调用方;OCR 与 Embedding 两个治理循环有同构的对应点 |
| (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 | | (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 |
| 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug | | 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug |
| `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 | | `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 |
@@ -131,7 +131,7 @@ Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重
| 测试 | 位置 | 先失败后通过的证据 | | 测试 | 位置 | 先失败后通过的证据 |
|---|---|---| |---|---|---|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 | | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
| 三条泄漏路径(`try_acquire`/`try_enter`/`progress_age_s`)抛出的异常携带正确 `scope` 与非零 `retry_after_s` | `tests/unit/test_backpressure.py`**三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s``:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` | | 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py`**三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s``:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 | | `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 | | 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError``reason == governance_backend_down`" | | Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError``reason == governance_backend_down`" |
@@ -33,7 +33,7 @@ date: 2026-08-06
## 对 issue 前提的四处修正 ## 对 issue 前提的四处修正
泄漏路径是**条**不是两条(`retry.py:216``progress_age_s()` 同样在 catch 之外);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。 泄漏路径是**条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。 根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
@@ -107,7 +107,7 @@ class BreakerGate:
## 任务清单 ## 任务清单
### - [ ] T1: ARCHITECTURE §6.1 回补(必须先行) ### - [x] T1: ARCHITECTURE §6.1 回补(必须先行)
**文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行) **文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行)
@@ -125,7 +125,7 @@ class BreakerGate:
--- ---
### - [ ] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出 ### - [x] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出
**文件**: 改 `src/polygateway/errors.py``src/polygateway/__init__.py`;改 `tests/unit/test_errors.py` **文件**: 改 `src/polygateway/errors.py``src/polygateway/__init__.py`;改 `tests/unit/test_errors.py`
@@ -148,7 +148,7 @@ class BreakerGate:
--- ---
### - [ ] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子) ### - [x] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子)
**文件**: 改 `src/polygateway/errors.py``backends/redis/limiter.py``backends/redis/breaker.py``backends/memory/limiter.py``middleware/ratelimit.py``middleware/breaker.py``middleware/retry.py``ocr.py``embedding.py`;改 `tests/unit/test_errors.py``tests/unit/test_backpressure.py``tests/unit/test_redis_key_layout.py``tests/integration/test_redis_cross_connection.py` **文件**: 改 `src/polygateway/errors.py``backends/redis/limiter.py``backends/redis/breaker.py``backends/memory/limiter.py``middleware/ratelimit.py``middleware/breaker.py``middleware/retry.py``ocr.py``embedding.py`;改 `tests/unit/test_errors.py``tests/unit/test_backpressure.py``tests/unit/test_redis_key_layout.py``tests/integration/test_redis_cross_connection.py`
@@ -170,7 +170,7 @@ class BreakerGate:
- `backends/redis/breaker.py``:370 / :388 / :410 / :422 / :432`(5 处) - `backends/redis/breaker.py``:370 / :388 / :410 / :422 / :432`(5 处)
4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise``scope=self._scope` 4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise``scope=self._scope`
- 各方法开头的 `except GovernanceBackendError: raise` **保持不变**(后端层已填好 scope,重建实例只会重复构造,设计 §3.3) - ~~各方法开头的 `except GovernanceBackendError: raise` **保持不变**~~ **← 这条是错的,2026-08-06 独立验证时炸出(见 §T6)**。正确做法: 该放行必须扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise`,否则新增的兄弟类型会落进下一行的 `except Exception` 被**重新包成** `GovernanceBackendError`,使 Q1 的拆分在唯一的生产路径上完全失效
5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序): 5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序):
@@ -186,7 +186,7 @@ class BreakerGate:
|---|---|---| |---|---|---|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"``retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 | | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"``retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 | | `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
| 三条泄漏路径(`try_acquire` / `try_enter` / `progress_age_s`)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_backpressure.py`**三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 | | 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py`**三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92``_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) | | 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92``_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 | | Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
@@ -211,7 +211,7 @@ class BreakerGate:
--- ---
### - [ ] T4: 公开错误面文档(issue #7 第二诉求) ### - [x] T4: 公开错误面文档(issue #7 第二诉求)
**文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md` **文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md`
@@ -238,7 +238,7 @@ class BreakerGate:
--- ---
### - [ ] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步 ### - [x] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步
**文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki **文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki
@@ -258,6 +258,33 @@ class BreakerGate:
--- ---
### - [x] T6: 修复独立验证炸出的阻塞缺陷(计划外,2026-08-06)
T1–T5 全绿、全部门禁通过之后,全新上下文的 verifier 用一个**走 `QuotaGate` 的**端到端用例炸出:装配缺陷在唯一的生产路径上根本没有拆出去。
**缺陷**: `QuotaGate`/`BreakerGate``except GovernanceBackendError: raise` 只放行了旧类型,新增的 `SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包成 `GovernanceBackendError`(`reason=governance_backend_down``retry_after_s=5.0`)。实证:
```
RAISED: GovernanceBackendError | isGatewayUnavailable=True | isSourceNotConfigured=False
| 限流后端故障(source_stats): 未知源 's1'(scope=llm)
```
即配置写错的任务照样落进"可延期重投"家族,**永远重投、永不进死信、无人告警**——正是 Q1 要防的镜像 bug,G2 等于没做。
**为什么原有测试测不出来**: T3 写的两条用例(`test_backpressure.py``test_redis_key_layout.py`)都直接打私有 `_cfg()`,绕过了包装器;而治理循环只经包装器访问后端。**盲区在于测试打的层次比生产路径低一层。**
**修复**(三处):
| 文件 | 改动 |
|---|---|
| `middleware/ratelimit.py` | 4 个方法的放行扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise` |
| `middleware/breaker.py` | 同上,5 个方法 |
| `middleware/telemetry.py:254` | 终态捕获元组加 `SourceNotConfiguredError`。**连带坑**: 放行生效后该异常不再是 `GovernanceBackendError`,而它在任何 attempt 之前抛出,若不显式捕获则 `emit_terminal_failure` 不触发、该路径**遥测归零**,违反"遥测必录"铁律 |
**回归测试**: `test_backpressure.py::TestUnknownSourceIsAssemblyDefect::test_survives_the_quota_gate_wrapper`(参数化覆盖 `try_acquire` / `stats`),**走包装器而非私有方法**。修前 2 failed,修后 PASS。
**同批文档订正**: 泄漏路径由"三条"改为**五条**(遗漏了 `QuotaGate.stats``BreakerGate.retry_after_s`,判据是该调用点是否被 `_record_quietly` 包裹);CHANGELOG 的 `per_source_reasons` 表述改为"属性存在但恒为 `{}`"。
## 完成后 ## 完成后
按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7 按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7
@@ -34,4 +34,12 @@ date: 2026-08-06
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。 Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier)
T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。**
修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。
两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats``BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。
相关: [[governance-backend-error]](design)、[[m2-distributed]] 相关: [[governance-backend-error]](design)、[[m2-distributed]]
+2 -1
View File
@@ -34,6 +34,7 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
@@ -326,7 +327,7 @@ class EmbeddingClient:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc) logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
async def _settle_and_release(self, permit: Permit, actual: int) -> None: async def _settle_and_release(self, permit: Permit, actual: int) -> None:
+21 -11
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from polygateway.errors import GovernanceBackendError from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
if TYPE_CHECKING: if TYPE_CHECKING:
from polygateway.ports import GateDecision, GateUpdate, ProviderGate from polygateway.ports import GateDecision, GateUpdate, ProviderGate
@@ -22,43 +22,53 @@ class BreakerGate:
async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision: async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision:
try: try:
return await self._gate.try_enter(source.name, owner) return await self._gate.try_enter(source.name, owner)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(try_enter): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(try_enter): {exc}", scope=self._scope
) from exc
async def record_success( async def record_success(
self, entry: GateDecision, *, count_attempt: bool = True self, entry: GateDecision, *, count_attempt: bool = True
) -> GateUpdate: ) -> GateUpdate:
try: try:
return await self._gate.record_success(entry, count_attempt=count_attempt) return await self._gate.record_success(entry, count_attempt=count_attempt)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(record_success): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(record_success): {exc}", scope=self._scope
) from exc
async def record_failure( async def record_failure(
self, entry: GateDecision, reason: str, force_open: bool self, entry: GateDecision, reason: str, force_open: bool
) -> GateUpdate: ) -> GateUpdate:
try: try:
return await self._gate.record_failure(entry, reason, force_open) return await self._gate.record_failure(entry, reason, force_open)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(record_failure): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(record_failure): {exc}", scope=self._scope
) from exc
async def release_probe(self, entry: GateDecision) -> GateUpdate: async def release_probe(self, entry: GateDecision) -> GateUpdate:
try: try:
return await self._gate.release_probe(entry) return await self._gate.release_probe(entry)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(release_probe): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(release_probe): {exc}", scope=self._scope
) from exc
async def retry_after_s(self, sources: tuple[str, ...]) -> float: async def retry_after_s(self, sources: tuple[str, ...]) -> float:
try: try:
return await self._gate.retry_after_s(sources) return await self._gate.retry_after_s(sources)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"熔断后端故障(retry_after_s): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"熔断后端故障(retry_after_s): {exc}", scope=self._scope
) from exc
+17 -9
View File
@@ -8,7 +8,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from polygateway.errors import GovernanceBackendError from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
if TYPE_CHECKING: if TYPE_CHECKING:
from polygateway.ports import Permit, RateLimiter from polygateway.ports import Permit, RateLimiter
@@ -26,31 +26,39 @@ class QuotaGate:
async def try_acquire(self, source: SourceConfig) -> Permit | None: async def try_acquire(self, source: SourceConfig) -> Permit | None:
try: try:
return await self._limiter.try_acquire(source.name, source.effective_est_tokens()) return await self._limiter.try_acquire(source.name, source.effective_est_tokens())
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(try_acquire): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(try_acquire): {exc}", scope=self._scope
) from exc
async def stats(self, source: SourceConfig) -> SourceStats: async def stats(self, source: SourceConfig) -> SourceStats:
try: try:
return await self._limiter.source_stats(source.name) return await self._limiter.source_stats(source.name)
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(source_stats): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(source_stats): {exc}", scope=self._scope
) from exc
async def mark_progress(self) -> None: async def mark_progress(self) -> None:
try: try:
await self._limiter.mark_progress() await self._limiter.mark_progress()
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(mark_progress): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(mark_progress): {exc}", scope=self._scope
) from exc
async def progress_age_s(self) -> float: async def progress_age_s(self) -> float:
try: try:
return await self._limiter.progress_age_s() return await self._limiter.progress_age_s()
except GovernanceBackendError: except (GovernanceBackendError, SourceNotConfiguredError):
raise raise
except Exception as exc: except Exception as exc:
raise GovernanceBackendError(f"限流后端故障(progress_age_s): {exc}", scope=self._scope) from exc raise GovernanceBackendError(
f"限流后端故障(progress_age_s): {exc}", scope=self._scope
) from exc
+2 -1
View File
@@ -28,6 +28,7 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
@@ -401,7 +402,7 @@ class RetryMW:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("治理记账写回降级(不冒泡): {}", exc) logger.warning("治理记账写回降级(不冒泡): {}", exc)
def _feed_outcome(self, source_name: str, ok: bool) -> None: def _feed_outcome(self, source_name: str, ok: bool) -> None:
+6 -2
View File
@@ -17,7 +17,11 @@ from typing import TYPE_CHECKING
from loguru import logger from loguru import logger
from polygateway.errors import GatewayUnavailableError, GovernanceBackendError from polygateway.errors import (
GatewayUnavailableError,
GovernanceBackendError,
SourceNotConfiguredError,
)
from polygateway.middleware.cache import digest_messages from polygateway.middleware.cache import digest_messages
from polygateway.types import canonical_sampling_json, merge_sampling from polygateway.types import canonical_sampling_json, merge_sampling
@@ -247,7 +251,7 @@ class TelemetryMW:
started = self._now() started = self._now()
try: try:
response = await call_next(request) response = await call_next(request)
except (GatewayUnavailableError, GovernanceBackendError) as exc: except (GatewayUnavailableError, GovernanceBackendError, SourceNotConfiguredError) as exc:
await self._emitter.emit_terminal_failure( await self._emitter.emit_terminal_failure(
request=request, request=request,
call_id=str(uuid.uuid4()), call_id=str(uuid.uuid4()),
+2 -1
View File
@@ -30,6 +30,7 @@ from polygateway.errors import (
RequestRejectedError, RequestRejectedError,
ResultInvalidError, ResultInvalidError,
SourceDeadError, SourceDeadError,
SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.breaker import BreakerGate from polygateway.middleware.breaker import BreakerGate
@@ -360,7 +361,7 @@ class OcrClient:
await write_back await write_back
except asyncio.CancelledError: except asyncio.CancelledError:
raise raise
except GovernanceBackendError as exc: except (GovernanceBackendError, SourceNotConfiguredError) as exc:
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc) logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
async def _settle_and_release(self, permit: Permit) -> None: async def _settle_and_release(self, permit: Permit) -> None:
+47 -2
View File
@@ -18,6 +18,7 @@ from polygateway.errors import (
SourceNotConfiguredError, SourceNotConfiguredError,
TransientError, TransientError,
) )
from polygateway.middleware.ratelimit import QuotaGate
from polygateway.middleware.retry import RetryMW, backoff_delay from polygateway.middleware.retry import RetryMW, backoff_delay
from polygateway.sources import RoundRobinSelector, SourceCooldownMemo from polygateway.sources import RoundRobinSelector, SourceCooldownMemo
from polygateway.types import ( from polygateway.types import (
@@ -192,6 +193,12 @@ class _LimiterProgressBroken(InMemoryLimiter):
raise GovernanceBackendError("redis 抖动", scope="llm") raise GovernanceBackendError("redis 抖动", scope="llm")
class _GateSuccessMisconfigured(InMemoryGate):
# 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常
async def record_success(self, entry, *, count_attempt: bool = True):
raise SourceNotConfiguredError("未知源 's1'(scope=llm)")
class TestAccountingDegradation: class TestAccountingDegradation:
"""记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。""" """记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。"""
@@ -206,6 +213,24 @@ class TestAccountingDegradation:
resp = await mw(_REQ) resp = await mw(_REQ)
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃 assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
async def test_assembly_defect_on_accounting_path_also_degrades(self):
"""记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。
`SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若
`_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、
销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账
方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。
"""
clock = FakeClock()
src = make_source()
limiter = InMemoryLimiter(
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
)
gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock)
mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate)
resp = await mw(_REQ)
assert resp.content == "ok"
async def test_mark_progress_failure_does_not_lose_response(self): async def test_mark_progress_failure_does_not_lose_response(self):
clock = FakeClock() clock = FakeClock()
src = make_source() src = make_source()
@@ -280,14 +305,34 @@ class TestUnknownSourceIsAssemblyDefect:
# 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信 # 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信
assert not isinstance(ei.value, GatewayUnavailableError) assert not isinstance(ei.value, GatewayUnavailableError)
@pytest.mark.parametrize("method", ["try_acquire", "stats"])
async def test_survives_the_quota_gate_wrapper(self, method):
"""必须穿透 QuotaGate,否则整个拆分在生产路径上等于没做。
上面两条(以及 redis 版)打的都是私有 `_cfg`,绕过了包装器。而治理循环
只经 QuotaGate 访问后端,包装器的 `except Exception` 会把装配缺陷重新
包成 `GovernanceBackendError`——下游又拿到可重投异常,永远重投不告警。
"""
src = make_source("s1")
# 限流后端的源名单与治理循环拿到的源对不上 = 装配缺陷
limiter = InMemoryLimiter(
scope="llm", sources={"other": src}, global_limits=_NO_GLOBAL
)
gate = QuotaGate(limiter, scope="llm")
with pytest.raises(SourceNotConfiguredError) as ei:
await getattr(gate, method)(src)
assert not isinstance(ei.value, GatewayUnavailableError)
class TestGateFailuresReachCallersAsScopeLevel: class TestGateFailuresReachCallersAsScopeLevel:
"""三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。 """闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路 记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果 抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复 是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
的故障。三条路径逐一钉住,防止将来任何一条被漏掉。 的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s,
`BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly`
包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。
""" """
async def test_try_acquire_failure_is_scope_level(self): async def test_try_acquire_failure_is_scope_level(self):