fix: keep the accounting path degrading after the wrapper change
Letting SourceNotConfiguredError through the gate wrappers opened a hole the recheck caught: _record_quietly only degrades GovernanceBackendError, so an assembly defect raised from the accounting side would now escape and destroy a response from a call that had already genuinely succeeded. That inverts the exact invariant _record_quietly exists to hold. Widening _record_quietly is the right fix rather than narrowing the wrappers, because that layer degrades by what the path is (accounting, the call is already done) rather than by which error type shows up. Narrowing would have left 4 of 9 wrapper methods as exceptions to a rule nobody can remember. No backend raises it from an accounting method today, so this is a guardrail for whoever adds source-name validation to a breaker backend. The stub that first reported this green was wrong: its record_success lacked count_attempt, so it raised TypeError and the wrapper relabeled it. Fixed signature, then the test failed as it should have. Also finishes the three-to-five leak path correction across the four remaining spots, including the wiki summary card that indexes this design.
This commit is contained in:
@@ -131,7 +131,7 @@ Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重
|
|||||||
| 测试 | 位置 | 先失败后通过的证据 |
|
| 测试 | 位置 | 先失败后通过的证据 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
|
||||||
| 三条泄漏路径(`try_acquire`/`try_enter`/`progress_age_s`)抛出的异常携带正确 `scope` 与非零 `retry_after_s` | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
|
| 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
|
||||||
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
|
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
|
||||||
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
|
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
|
||||||
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" |
|
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" |
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ date: 2026-08-06
|
|||||||
|
|
||||||
## 对 issue 前提的四处修正
|
## 对 issue 前提的四处修正
|
||||||
|
|
||||||
泄漏路径是**三条**不是两条(`retry.py:216` 的 `progress_age_s()` 同样在 catch 之外);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
|
泄漏路径是**五条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
|
||||||
|
|
||||||
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
|
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
|
||||||
|
|
||||||
|
|||||||
@@ -186,7 +186,7 @@ class BreakerGate:
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
|
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
|
||||||
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
|
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
|
||||||
| 三条泄漏路径(`try_acquire` / `try_enter` / `progress_age_s`)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
|
| 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
|
||||||
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
|
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
|
||||||
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
|
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
|
||||||
|
|
||||||
|
|||||||
@@ -34,4 +34,12 @@ date: 2026-08-06
|
|||||||
|
|
||||||
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
|
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
|
||||||
|
|
||||||
|
## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier)
|
||||||
|
|
||||||
|
T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。**
|
||||||
|
|
||||||
|
修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。
|
||||||
|
|
||||||
|
两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats` 与 `BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。
|
||||||
|
|
||||||
相关: [[governance-backend-error]](design)、[[m2-distributed]]
|
相关: [[governance-backend-error]](design)、[[m2-distributed]]
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -326,7 +327,7 @@ class EmbeddingClient:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -401,7 +402,7 @@ class RetryMW:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
def _feed_outcome(self, source_name: str, ok: bool) -> None:
|
def _feed_outcome(self, source_name: str, ok: bool) -> None:
|
||||||
|
|||||||
@@ -30,6 +30,7 @@ from polygateway.errors import (
|
|||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
ResultInvalidError,
|
ResultInvalidError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
|
SourceNotConfiguredError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.breaker import BreakerGate
|
from polygateway.middleware.breaker import BreakerGate
|
||||||
@@ -360,7 +361,7 @@ class OcrClient:
|
|||||||
await write_back
|
await write_back
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
except GovernanceBackendError as exc:
|
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||||
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
||||||
|
|
||||||
async def _settle_and_release(self, permit: Permit) -> None:
|
async def _settle_and_release(self, permit: Permit) -> None:
|
||||||
|
|||||||
@@ -193,6 +193,12 @@ class _LimiterProgressBroken(InMemoryLimiter):
|
|||||||
raise GovernanceBackendError("redis 抖动", scope="llm")
|
raise GovernanceBackendError("redis 抖动", scope="llm")
|
||||||
|
|
||||||
|
|
||||||
|
class _GateSuccessMisconfigured(InMemoryGate):
|
||||||
|
# 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常
|
||||||
|
async def record_success(self, entry, *, count_attempt: bool = True):
|
||||||
|
raise SourceNotConfiguredError("未知源 's1'(scope=llm)")
|
||||||
|
|
||||||
|
|
||||||
class TestAccountingDegradation:
|
class TestAccountingDegradation:
|
||||||
"""记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。"""
|
"""记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。"""
|
||||||
|
|
||||||
@@ -207,6 +213,24 @@ class TestAccountingDegradation:
|
|||||||
resp = await mw(_REQ)
|
resp = await mw(_REQ)
|
||||||
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
|
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
|
||||||
|
|
||||||
|
async def test_assembly_defect_on_accounting_path_also_degrades(self):
|
||||||
|
"""记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。
|
||||||
|
|
||||||
|
`SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若
|
||||||
|
`_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、
|
||||||
|
销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账
|
||||||
|
方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。
|
||||||
|
"""
|
||||||
|
clock = FakeClock()
|
||||||
|
src = make_source()
|
||||||
|
limiter = InMemoryLimiter(
|
||||||
|
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
|
||||||
|
)
|
||||||
|
gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock)
|
||||||
|
mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate)
|
||||||
|
resp = await mw(_REQ)
|
||||||
|
assert resp.content == "ok"
|
||||||
|
|
||||||
async def test_mark_progress_failure_does_not_lose_response(self):
|
async def test_mark_progress_failure_does_not_lose_response(self):
|
||||||
clock = FakeClock()
|
clock = FakeClock()
|
||||||
src = make_source()
|
src = make_source()
|
||||||
@@ -301,12 +325,14 @@ class TestUnknownSourceIsAssemblyDefect:
|
|||||||
|
|
||||||
|
|
||||||
class TestGateFailuresReachCallersAsScopeLevel:
|
class TestGateFailuresReachCallersAsScopeLevel:
|
||||||
"""三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
|
"""闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
|
||||||
|
|
||||||
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
|
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
|
||||||
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
|
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
|
||||||
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
|
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
|
||||||
的故障。三条路径逐一钉住,防止将来任何一条被漏掉。
|
的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s,
|
||||||
|
`BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly`
|
||||||
|
包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
async def test_try_acquire_failure_is_scope_level(self):
|
async def test_try_acquire_failure_is_scope_level(self):
|
||||||
|
|||||||
Reference in New Issue
Block a user