diff --git a/research-wiki/designs/2026-08-06-governance-backend-error-design.md b/research-wiki/designs/2026-08-06-governance-backend-error-design.md index 0203ad9..1f39683 100644 --- a/research-wiki/designs/2026-08-06-governance-backend-error-design.md +++ b/research-wiki/designs/2026-08-06-governance-backend-error-design.md @@ -131,7 +131,7 @@ Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重 | 测试 | 位置 | 先失败后通过的证据 | |---|---|---| | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 | -| 三条泄漏路径(`try_acquire`/`try_enter`/`progress_age_s`)抛出的异常携带正确 `scope` 与非零 `retry_after_s` | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` | +| 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` | | `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 | | 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 | | Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" | diff --git a/research-wiki/designs/governance-backend-error.md b/research-wiki/designs/governance-backend-error.md index a126bde..969150d 100644 --- a/research-wiki/designs/governance-backend-error.md +++ b/research-wiki/designs/governance-backend-error.md @@ -33,7 +33,7 @@ date: 2026-08-06 ## 对 issue 前提的四处修正 -泄漏路径是**三条**不是两条(`retry.py:216` 的 `progress_age_s()` 同样在 catch 之外);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。 +泄漏路径是**五条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。 根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。 diff --git a/research-wiki/plans/2026-08-06-governance-backend-error-plan.md b/research-wiki/plans/2026-08-06-governance-backend-error-plan.md index dc661e3..84aa53a 100644 --- a/research-wiki/plans/2026-08-06-governance-backend-error-plan.md +++ b/research-wiki/plans/2026-08-06-governance-backend-error-plan.md @@ -186,7 +186,7 @@ class BreakerGate: |---|---|---| | `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 | | `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 | -| 三条泄漏路径(`try_acquire` / `try_enter` / `progress_age_s`)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 | +| 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 | | 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) | | Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 | diff --git a/research-wiki/plans/governance-backend-error.md b/research-wiki/plans/governance-backend-error.md index f72eedd..8aae286 100644 --- a/research-wiki/plans/governance-backend-error.md +++ b/research-wiki/plans/governance-backend-error.md @@ -34,4 +34,12 @@ date: 2026-08-06 Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。 +## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier) + +T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。** + +修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。 + +两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats` 与 `BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。 + 相关: [[governance-backend-error]](design)、[[m2-distributed]] diff --git a/src/polygateway/embedding.py b/src/polygateway/embedding.py index 2497944..55e501a 100644 --- a/src/polygateway/embedding.py +++ b/src/polygateway/embedding.py @@ -34,6 +34,7 @@ from polygateway.errors import ( RequestRejectedError, ResultInvalidError, SourceDeadError, + SourceNotConfiguredError, TransientError, ) from polygateway.middleware.breaker import BreakerGate @@ -326,7 +327,7 @@ class EmbeddingClient: await write_back except asyncio.CancelledError: raise - except GovernanceBackendError as exc: + except (GovernanceBackendError, SourceNotConfiguredError) as exc: logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc) async def _settle_and_release(self, permit: Permit, actual: int) -> None: diff --git a/src/polygateway/middleware/retry.py b/src/polygateway/middleware/retry.py index 40d43a0..7fbe421 100644 --- a/src/polygateway/middleware/retry.py +++ b/src/polygateway/middleware/retry.py @@ -28,6 +28,7 @@ from polygateway.errors import ( RequestRejectedError, ResultInvalidError, SourceDeadError, + SourceNotConfiguredError, TransientError, ) from polygateway.middleware.breaker import BreakerGate @@ -401,7 +402,7 @@ class RetryMW: await write_back except asyncio.CancelledError: raise - except GovernanceBackendError as exc: + except (GovernanceBackendError, SourceNotConfiguredError) as exc: logger.warning("治理记账写回降级(不冒泡): {}", exc) def _feed_outcome(self, source_name: str, ok: bool) -> None: diff --git a/src/polygateway/ocr.py b/src/polygateway/ocr.py index 2015d06..6d5bd53 100644 --- a/src/polygateway/ocr.py +++ b/src/polygateway/ocr.py @@ -30,6 +30,7 @@ from polygateway.errors import ( RequestRejectedError, ResultInvalidError, SourceDeadError, + SourceNotConfiguredError, TransientError, ) from polygateway.middleware.breaker import BreakerGate @@ -360,7 +361,7 @@ class OcrClient: await write_back except asyncio.CancelledError: raise - except GovernanceBackendError as exc: + except (GovernanceBackendError, SourceNotConfiguredError) as exc: logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc) async def _settle_and_release(self, permit: Permit) -> None: diff --git a/tests/unit/test_backpressure.py b/tests/unit/test_backpressure.py index b64365c..302de45 100644 --- a/tests/unit/test_backpressure.py +++ b/tests/unit/test_backpressure.py @@ -193,6 +193,12 @@ class _LimiterProgressBroken(InMemoryLimiter): raise GovernanceBackendError("redis 抖动", scope="llm") +class _GateSuccessMisconfigured(InMemoryGate): + # 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常 + async def record_success(self, entry, *, count_attempt: bool = True): + raise SourceNotConfiguredError("未知源 's1'(scope=llm)") + + class TestAccountingDegradation: """记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。""" @@ -207,6 +213,24 @@ class TestAccountingDegradation: resp = await mw(_REQ) assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃 + async def test_assembly_defect_on_accounting_path_also_degrades(self): + """记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。 + + `SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若 + `_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、 + 销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账 + 方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。 + """ + clock = FakeClock() + src = make_source() + limiter = InMemoryLimiter( + scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock + ) + gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock) + mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate) + resp = await mw(_REQ) + assert resp.content == "ok" + async def test_mark_progress_failure_does_not_lose_response(self): clock = FakeClock() src = make_source() @@ -301,12 +325,14 @@ class TestUnknownSourceIsAssemblyDefect: class TestGateFailuresReachCallersAsScopeLevel: - """三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。 + """闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。 记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路 抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果 是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复 - 的故障。三条路径逐一钉住,防止将来任何一条被漏掉。 + 的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s, + `BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly` + 包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。 """ async def test_try_acquire_failure_is_scope_level(self):