diff --git a/CHANGELOG.md b/CHANGELOG.md index 7268140..039326a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,8 +19,8 @@ ### 下游请读 -- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`**,与 `AllSourcesExhausted` 同款;`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。 -- **三条闸门路径**(`try_acquire` / `try_enter` / `progress_age_s`)的后端故障会到达调用方;记账路径(`record_success` 等)仍被 `_record_quietly` 降级为 warning,这个分工不变。 +- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`**,与 `AllSourcesExhausted` 同款(`per_source_reasons` 属性存在但恒为 `{}`——后端故障不针对具体某个源);`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。 +- **五条闸门路径**的后端故障会到达调用方: `QuotaGate` 的 `try_acquire` / `stats` / `progress_age_s`,`BreakerGate` 的 `try_enter` / `retry_after_s`。记账路径(`record_success` / `record_failure` / `release_probe` / `mark_progress`)仍被 `_record_quietly` 降级为 warning,这个分工不变。 - **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。 ## 1.0.6(2026-08-02) diff --git a/research-wiki/designs/2026-08-06-governance-backend-error-design.md b/research-wiki/designs/2026-08-06-governance-backend-error-design.md index cf01841..0203ad9 100644 --- a/research-wiki/designs/2026-08-06-governance-backend-error-design.md +++ b/research-wiki/designs/2026-08-06-governance-backend-error-design.md @@ -20,7 +20,7 @@ | Issue 原文 | 实际情况 | |---|---| -| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **三条**。`middleware/retry.py:216` 每轮循环开头的 `progress_age_s()` 同样在 catch 之外,直达调用方 | +| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **五条**(设计初稿写"三条",2026-08-06 独立验证时核出遗漏两条并订正): `QuotaGate` 的 `try_acquire` / `stats`(`retry.py:249`)/ `progress_age_s`(`retry.py:216`、`:305`),`BreakerGate` 的 `try_enter` / `retry_after_s`(`retry.py:292`、`:310`)。判据是该调用点是否被 `_record_quietly` 包裹——未包裹即直达调用方;OCR 与 Embedding 两个治理循环有同构的对应点 | | (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 | | 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug | | `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 | diff --git a/research-wiki/plans/2026-08-06-governance-backend-error-plan.md b/research-wiki/plans/2026-08-06-governance-backend-error-plan.md index 6758adb..dc661e3 100644 --- a/research-wiki/plans/2026-08-06-governance-backend-error-plan.md +++ b/research-wiki/plans/2026-08-06-governance-backend-error-plan.md @@ -107,7 +107,7 @@ class BreakerGate: ## 任务清单 -### - [ ] T1: ARCHITECTURE §6.1 回补(必须先行) +### - [x] T1: ARCHITECTURE §6.1 回补(必须先行) **文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行) @@ -125,7 +125,7 @@ class BreakerGate: --- -### - [ ] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出 +### - [x] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出 **文件**: 改 `src/polygateway/errors.py`、`src/polygateway/__init__.py`;改 `tests/unit/test_errors.py` @@ -148,7 +148,7 @@ class BreakerGate: --- -### - [ ] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子) +### - [x] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子) **文件**: 改 `src/polygateway/errors.py`、`backends/redis/limiter.py`、`backends/redis/breaker.py`、`backends/memory/limiter.py`、`middleware/ratelimit.py`、`middleware/breaker.py`、`middleware/retry.py`、`ocr.py`、`embedding.py`;改 `tests/unit/test_errors.py`、`tests/unit/test_backpressure.py`、`tests/unit/test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py` @@ -170,7 +170,7 @@ class BreakerGate: - `backends/redis/breaker.py` 的 `:370 / :388 / :410 / :422 / :432`(5 处) 4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise` 补 `scope=self._scope`。 - - 各方法开头的 `except GovernanceBackendError: raise` **保持不变**(后端层已填好 scope,重建实例只会重复构造,设计 §3.3)。 + - ~~各方法开头的 `except GovernanceBackendError: raise` **保持不变**~~ **← 这条是错的,2026-08-06 独立验证时炸出(见 §T6)**。正确做法: 该放行必须扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise`,否则新增的兄弟类型会落进下一行的 `except Exception` 被**重新包成** `GovernanceBackendError`,使 Q1 的拆分在唯一的生产路径上完全失效。 5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序): @@ -211,7 +211,7 @@ class BreakerGate: --- -### - [ ] T4: 公开错误面文档(issue #7 第二诉求) +### - [x] T4: 公开错误面文档(issue #7 第二诉求) **文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md` @@ -238,7 +238,7 @@ class BreakerGate: --- -### - [ ] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步 +### - [x] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步 **文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki @@ -258,6 +258,33 @@ class BreakerGate: --- +### - [x] T6: 修复独立验证炸出的阻塞缺陷(计划外,2026-08-06) + +T1–T5 全绿、全部门禁通过之后,全新上下文的 verifier 用一个**走 `QuotaGate` 的**端到端用例炸出:装配缺陷在唯一的生产路径上根本没有拆出去。 + +**缺陷**: `QuotaGate`/`BreakerGate` 的 `except GovernanceBackendError: raise` 只放行了旧类型,新增的 `SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包成 `GovernanceBackendError`(`reason=governance_backend_down`、`retry_after_s=5.0`)。实证: + +``` +RAISED: GovernanceBackendError | isGatewayUnavailable=True | isSourceNotConfigured=False + | 限流后端故障(source_stats): 未知源 's1'(scope=llm) +``` + +即配置写错的任务照样落进"可延期重投"家族,**永远重投、永不进死信、无人告警**——正是 Q1 要防的镜像 bug,G2 等于没做。 + +**为什么原有测试测不出来**: T3 写的两条用例(`test_backpressure.py`、`test_redis_key_layout.py`)都直接打私有 `_cfg()`,绕过了包装器;而治理循环只经包装器访问后端。**盲区在于测试打的层次比生产路径低一层。** + +**修复**(三处): + +| 文件 | 改动 | +|---|---| +| `middleware/ratelimit.py` | 4 个方法的放行扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise` | +| `middleware/breaker.py` | 同上,5 个方法 | +| `middleware/telemetry.py:254` | 终态捕获元组加 `SourceNotConfiguredError`。**连带坑**: 放行生效后该异常不再是 `GovernanceBackendError`,而它在任何 attempt 之前抛出,若不显式捕获则 `emit_terminal_failure` 不触发、该路径**遥测归零**,违反"遥测必录"铁律 | + +**回归测试**: `test_backpressure.py::TestUnknownSourceIsAssemblyDefect::test_survives_the_quota_gate_wrapper`(参数化覆盖 `try_acquire` / `stats`),**走包装器而非私有方法**。修前 2 failed,修后 PASS。 + +**同批文档订正**: 泄漏路径由"三条"改为**五条**(遗漏了 `QuotaGate.stats` 与 `BreakerGate.retry_after_s`,判据是该调用点是否被 `_record_quietly` 包裹);CHANGELOG 的 `per_source_reasons` 表述改为"属性存在但恒为 `{}`"。 + ## 完成后 按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7。 diff --git a/src/polygateway/middleware/breaker.py b/src/polygateway/middleware/breaker.py index 4da1716..36a93f7 100644 --- a/src/polygateway/middleware/breaker.py +++ b/src/polygateway/middleware/breaker.py @@ -4,7 +4,7 @@ from __future__ import annotations from typing import TYPE_CHECKING -from polygateway.errors import GovernanceBackendError +from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError if TYPE_CHECKING: from polygateway.ports import GateDecision, GateUpdate, ProviderGate @@ -22,43 +22,53 @@ class BreakerGate: async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision: try: return await self._gate.try_enter(source.name, owner) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"熔断后端故障(try_enter): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"熔断后端故障(try_enter): {exc}", scope=self._scope + ) from exc async def record_success( self, entry: GateDecision, *, count_attempt: bool = True ) -> GateUpdate: try: return await self._gate.record_success(entry, count_attempt=count_attempt) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"熔断后端故障(record_success): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"熔断后端故障(record_success): {exc}", scope=self._scope + ) from exc async def record_failure( self, entry: GateDecision, reason: str, force_open: bool ) -> GateUpdate: try: return await self._gate.record_failure(entry, reason, force_open) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"熔断后端故障(record_failure): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"熔断后端故障(record_failure): {exc}", scope=self._scope + ) from exc async def release_probe(self, entry: GateDecision) -> GateUpdate: try: return await self._gate.release_probe(entry) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"熔断后端故障(release_probe): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"熔断后端故障(release_probe): {exc}", scope=self._scope + ) from exc async def retry_after_s(self, sources: tuple[str, ...]) -> float: try: return await self._gate.retry_after_s(sources) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"熔断后端故障(retry_after_s): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"熔断后端故障(retry_after_s): {exc}", scope=self._scope + ) from exc diff --git a/src/polygateway/middleware/ratelimit.py b/src/polygateway/middleware/ratelimit.py index 7d9e599..a71cee1 100644 --- a/src/polygateway/middleware/ratelimit.py +++ b/src/polygateway/middleware/ratelimit.py @@ -8,7 +8,7 @@ from __future__ import annotations from typing import TYPE_CHECKING -from polygateway.errors import GovernanceBackendError +from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError if TYPE_CHECKING: from polygateway.ports import Permit, RateLimiter @@ -26,31 +26,39 @@ class QuotaGate: async def try_acquire(self, source: SourceConfig) -> Permit | None: try: return await self._limiter.try_acquire(source.name, source.effective_est_tokens()) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"限流后端故障(try_acquire): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"限流后端故障(try_acquire): {exc}", scope=self._scope + ) from exc async def stats(self, source: SourceConfig) -> SourceStats: try: return await self._limiter.source_stats(source.name) - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"限流后端故障(source_stats): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"限流后端故障(source_stats): {exc}", scope=self._scope + ) from exc async def mark_progress(self) -> None: try: await self._limiter.mark_progress() - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"限流后端故障(mark_progress): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"限流后端故障(mark_progress): {exc}", scope=self._scope + ) from exc async def progress_age_s(self) -> float: try: return await self._limiter.progress_age_s() - except GovernanceBackendError: + except (GovernanceBackendError, SourceNotConfiguredError): raise except Exception as exc: - raise GovernanceBackendError(f"限流后端故障(progress_age_s): {exc}", scope=self._scope) from exc + raise GovernanceBackendError( + f"限流后端故障(progress_age_s): {exc}", scope=self._scope + ) from exc diff --git a/src/polygateway/middleware/telemetry.py b/src/polygateway/middleware/telemetry.py index 491ee9e..1d81e1c 100644 --- a/src/polygateway/middleware/telemetry.py +++ b/src/polygateway/middleware/telemetry.py @@ -17,7 +17,11 @@ from typing import TYPE_CHECKING from loguru import logger -from polygateway.errors import GatewayUnavailableError, GovernanceBackendError +from polygateway.errors import ( + GatewayUnavailableError, + GovernanceBackendError, + SourceNotConfiguredError, +) from polygateway.middleware.cache import digest_messages from polygateway.types import canonical_sampling_json, merge_sampling @@ -247,7 +251,7 @@ class TelemetryMW: started = self._now() try: response = await call_next(request) - except (GatewayUnavailableError, GovernanceBackendError) as exc: + except (GatewayUnavailableError, GovernanceBackendError, SourceNotConfiguredError) as exc: await self._emitter.emit_terminal_failure( request=request, call_id=str(uuid.uuid4()), diff --git a/tests/unit/test_backpressure.py b/tests/unit/test_backpressure.py index cd2ee73..b64365c 100644 --- a/tests/unit/test_backpressure.py +++ b/tests/unit/test_backpressure.py @@ -18,6 +18,7 @@ from polygateway.errors import ( SourceNotConfiguredError, TransientError, ) +from polygateway.middleware.ratelimit import QuotaGate from polygateway.middleware.retry import RetryMW, backoff_delay from polygateway.sources import RoundRobinSelector, SourceCooldownMemo from polygateway.types import ( @@ -280,6 +281,24 @@ class TestUnknownSourceIsAssemblyDefect: # 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信 assert not isinstance(ei.value, GatewayUnavailableError) + @pytest.mark.parametrize("method", ["try_acquire", "stats"]) + async def test_survives_the_quota_gate_wrapper(self, method): + """必须穿透 QuotaGate,否则整个拆分在生产路径上等于没做。 + + 上面两条(以及 redis 版)打的都是私有 `_cfg`,绕过了包装器。而治理循环 + 只经 QuotaGate 访问后端,包装器的 `except Exception` 会把装配缺陷重新 + 包成 `GovernanceBackendError`——下游又拿到可重投异常,永远重投不告警。 + """ + src = make_source("s1") + # 限流后端的源名单与治理循环拿到的源对不上 = 装配缺陷 + limiter = InMemoryLimiter( + scope="llm", sources={"other": src}, global_limits=_NO_GLOBAL + ) + gate = QuotaGate(limiter, scope="llm") + with pytest.raises(SourceNotConfiguredError) as ei: + await getattr(gate, method)(src) + assert not isinstance(ei.value, GatewayUnavailableError) + class TestGateFailuresReachCallersAsScopeLevel: """三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。