Compare commits
3 Commits
v1.1.0
..
9c2824ce8a
| Author | SHA1 | Date | |
|---|---|---|---|
| 9c2824ce8a | |||
| 5853c3f8ff | |||
| a57a5cea72 |
+2
-2
@@ -19,8 +19,8 @@
|
||||
|
||||
### 下游请读
|
||||
|
||||
- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s` / `per_source_reasons`**,与 `AllSourcesExhausted` 同款;`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。
|
||||
- **三条闸门路径**(`try_acquire` / `try_enter` / `progress_age_s`)的后端故障会到达调用方;记账路径(`record_success` 等)仍被 `_record_quietly` 降级为 warning,这个分工不变。
|
||||
- **`GovernanceBackendError` 现携带 `scope` / `reason` / `retry_after_s`**,与 `AllSourcesExhausted` 同款(`per_source_reasons` 属性存在但恒为 `{}`——后端故障不针对具体某个源);`str(exc)` 仍是原来的诊断串(如 `限流后端 try_acquire 失败: ...`),结构化字段与诊断信息并存,排障不受影响。
|
||||
- **五条闸门路径**的后端故障会到达调用方: `QuotaGate` 的 `try_acquire` / `stats` / `progress_age_s`,`BreakerGate` 的 `try_enter` / `retry_after_s`。记账路径(`record_success` / `record_failure` / `release_probe` / `mark_progress`)仍被 `_record_quietly` 降级为 warning,这个分工不变。
|
||||
- **CHSAnalyzer 迁移**: `tracking.py` 一条 `except GatewayUnavailableError` 即覆盖完整,无需为后端故障单列分支(`migrations/chsanalyzer.md` G1 已补注)。
|
||||
|
||||
## 1.0.6(2026-08-02)
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
|
||||
| Issue 原文 | 实际情况 |
|
||||
|---|---|
|
||||
| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **三条**。`middleware/retry.py:216` 每轮循环开头的 `progress_age_s()` 同样在 catch 之外,直达调用方 |
|
||||
| 泄漏路径为 `try_enter` / `try_acquire` 两条 | **五条**(设计初稿写"三条",2026-08-06 独立验证时核出遗漏两条并订正): `QuotaGate` 的 `try_acquire` / `stats`(`retry.py:249`)/ `progress_age_s`(`retry.py:216`、`:305`),`BreakerGate` 的 `try_enter` / `retry_after_s`(`retry.py:292`、`:310`)。判据是该调用点是否被 `_record_quietly` 包裹——未包裹即直达调用方;OCR 与 Embedding 两个治理循环有同构的对应点 |
|
||||
| (未提及构造点数量) | 全库 **22 处** `raise GovernanceBackendError`,分布于 4 个文件 |
|
||||
| 方向 A 只需改类型树 | 其中 **2 处语义完全不同**(见 §3.4),整类归入"可重投"会制造镜像 bug |
|
||||
| `retry_after_s` 取 0,「docstring 已写 0 = 可立即重试,语义上是通的」 | 语义通,**工程上不通**。见 §3.2 |
|
||||
@@ -131,7 +131,7 @@ Issue 建议取 0。**否决**:下游 `schedule_retry(after_s=0)` 会立刻重
|
||||
| 测试 | 位置 | 先失败后通过的证据 |
|
||||
|---|---|---|
|
||||
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_errors.py` | 改前 `pytest.raises(GatewayUnavailableError)` 必失败 |
|
||||
| 三条泄漏路径(`try_acquire`/`try_enter`/`progress_age_s`)抛出的异常携带正确 `scope` 与非零 `retry_after_s` | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
|
||||
| 闸门泄漏路径(五条,§1.1)抛出的异常携带正确 `scope` 与非零 `retry_after_s`;钉住 `try_acquire`/`try_enter`/`progress_age_s` 三条代表路径,余两条由同一注入机制覆盖 | `tests/unit/test_backpressure.py` — **三条桩都需新增**(Codex 审计划时核出: `:176-186` 是记账侧 `record_success`/`record_failure`/`mark_progress` 的降级桩,不是闸门路径;`progress_age_s` 仅 `:243-257` 覆盖包装行为、不验 scope) | 改前无 `scope` 属性,`AttributeError` |
|
||||
| `str(exc)` 仍为原诊断串 | `tests/unit/test_errors.py` | 防 §3.5 回归 |
|
||||
| 未知源抛 `SourceNotConfiguredError` 且**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`;内存版**当前无覆盖,需新增** | 改前抛 `GovernanceBackendError`,断言"不是 scope 级"必失败 |
|
||||
| Redis 真实掉线时准入侧行为 | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 断言由 `GovernanceBackendError` 收紧为"是 `GatewayUnavailableError` 且 `reason == governance_backend_down`" |
|
||||
|
||||
@@ -33,7 +33,7 @@ date: 2026-08-06
|
||||
|
||||
## 对 issue 前提的四处修正
|
||||
|
||||
泄漏路径是**三条**不是两条(`retry.py:216` 的 `progress_age_s()` 同样在 catch 之外);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
|
||||
泄漏路径是**五条**不是两条(判据: 该 gate 调用点是否被 `_record_quietly` 包裹——`QuotaGate` 的 try_acquire / stats / progress_age_s 与 `BreakerGate` 的 try_enter / retry_after_s 均未包裹,直达调用方);构造点 **22 处**;其中 2 处语义完全不同(未知源);`retry_after_s=0` 语义通但工程不通。
|
||||
|
||||
根因记录: `ARCHITECTURE.md` §6.1 错误分类表里 `GovernanceBackendError` **一次都没出现**——它是 M2 引入分布式后端时新增的,当时未回补架构表,于是它在"调用方视角的分类学"中从来没有位置,README 的遗漏是这个遗漏的下游后果。
|
||||
|
||||
|
||||
@@ -107,7 +107,7 @@ class BreakerGate:
|
||||
|
||||
## 任务清单
|
||||
|
||||
### - [ ] T1: ARCHITECTURE §6.1 回补(必须先行)
|
||||
### - [x] T1: ARCHITECTURE §6.1 回补(必须先行)
|
||||
|
||||
**文件**: `research-wiki/ARCHITECTURE.md`(§6.1,约 372-380 行)
|
||||
|
||||
@@ -125,7 +125,7 @@ class BreakerGate:
|
||||
|
||||
---
|
||||
|
||||
### - [ ] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出
|
||||
### - [x] T2: errors.py 纯增量(新常量、新 reason、新类)+ 导出
|
||||
|
||||
**文件**: 改 `src/polygateway/errors.py`、`src/polygateway/__init__.py`;改 `tests/unit/test_errors.py`
|
||||
|
||||
@@ -148,7 +148,7 @@ class BreakerGate:
|
||||
|
||||
---
|
||||
|
||||
### - [ ] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子)
|
||||
### - [x] T3: `GovernanceBackendError` 归位 + 22 处构造点 + scope 注入(原子)
|
||||
|
||||
**文件**: 改 `src/polygateway/errors.py`、`backends/redis/limiter.py`、`backends/redis/breaker.py`、`backends/memory/limiter.py`、`middleware/ratelimit.py`、`middleware/breaker.py`、`middleware/retry.py`、`ocr.py`、`embedding.py`;改 `tests/unit/test_errors.py`、`tests/unit/test_backpressure.py`、`tests/unit/test_redis_key_layout.py`、`tests/integration/test_redis_cross_connection.py`
|
||||
|
||||
@@ -170,7 +170,7 @@ class BreakerGate:
|
||||
- `backends/redis/breaker.py` 的 `:370 / :388 / :410 / :422 / :432`(5 处)
|
||||
|
||||
4. **两个 gate 包装器**: 构造函数改为上文"关键接口"的签名;`QuotaGate` 4 处(`ratelimit.py:30/38/46/54`)与 `BreakerGate` 5 处(`breaker.py:26/36/46/54/62`)的 `raise` 补 `scope=self._scope`。
|
||||
- 各方法开头的 `except GovernanceBackendError: raise` **保持不变**(后端层已填好 scope,重建实例只会重复构造,设计 §3.3)。
|
||||
- ~~各方法开头的 `except GovernanceBackendError: raise` **保持不变**~~ **← 这条是错的,2026-08-06 独立验证时炸出(见 §T6)**。正确做法: 该放行必须扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise`,否则新增的兄弟类型会落进下一行的 `except Exception` 被**重新包成** `GovernanceBackendError`,使 Q1 的拆分在唯一的生产路径上完全失效。
|
||||
|
||||
5. **三处装配各传 scope**(三处的 `self._scope` 均已在装配前赋值,无需调整顺序):
|
||||
|
||||
@@ -186,7 +186,7 @@ class BreakerGate:
|
||||
|---|---|---|
|
||||
| `GovernanceBackendError` 可被 `except GatewayUnavailableError` 接住,且 `reason == "governance_backend_down"`、`retry_after_s == 5.0` | `tests/unit/test_errors.py` | 改前非其子类,`pytest.raises(GatewayUnavailableError)` 不匹配 |
|
||||
| `str(exc)` 仍为构造时的诊断串(防 §3.5 回归) | `tests/unit/test_errors.py` | 改前无该风险但改后若漏写 `self.args` 即失败,是回归护栏 |
|
||||
| 三条泄漏路径(`try_acquire` / `try_enter` / `progress_age_s`)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
|
||||
| 闸门泄漏路径(共五条,见设计 §1.1)抛出的异常带正确 `scope`、且可被 `except GatewayUnavailableError` 接住;钉住 `try_acquire` / `try_enter` / `progress_age_s` 三条代表路径 | `tests/unit/test_backpressure.py` — **三条都要新增桩**。现状: `progress_age_s` 只有 `TestQuotaGateProgressAge`(`:243-257`)覆盖包装行为、不验 scope;`try_acquire`(`QuotaGate`)与 `try_enter`(`BreakerGate`)**完全无桩** | 改前异常无 `scope` 属性 → `AttributeError`;两条新路径改前无覆盖 |
|
||||
| 未知源抛 `SourceNotConfiguredError`,且断言它**不是** `GatewayUnavailableError` | 改 `tests/unit/test_redis_key_layout.py:70-74`(`test_unknown_source_rejected`,现断言 `GovernanceBackendError`);内存版**当前无对应用例,需新增**一条同款(`backends/memory/limiter.py:92` 的 `_cfg("nope")`) | 改前 redis 版类型断言失败;内存版改前无覆盖(该分支从未被测过) |
|
||||
| Redis 真实掉线时准入侧抛 scope 级异常且 `reason == "governance_backend_down"` | `tests/integration/test_redis_cross_connection.py:228-245`(真实 Redis,不 mock) | 改前无 `reason` 属性 |
|
||||
|
||||
@@ -211,7 +211,7 @@ class BreakerGate:
|
||||
|
||||
---
|
||||
|
||||
### - [ ] T4: 公开错误面文档(issue #7 第二诉求)
|
||||
### - [x] T4: 公开错误面文档(issue #7 第二诉求)
|
||||
|
||||
**文件**: 改 `README.md`(§"错误模型(四分类)",约 114-125 行)、`research-wiki/migrations/chsanalyzer.md`
|
||||
|
||||
@@ -238,7 +238,7 @@ class BreakerGate:
|
||||
|
||||
---
|
||||
|
||||
### - [ ] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步
|
||||
### - [x] T5: 版本 1.1.0 + CHANGELOG + Wiki 同步
|
||||
|
||||
**文件**: 改 `pyproject.toml`(version)、`src/polygateway/__init__.py`(`__version__`)、`CHANGELOG.md`;按 `research-wiki/docs-convention.md` §2 同步 Gitea Wiki
|
||||
|
||||
@@ -258,6 +258,33 @@ class BreakerGate:
|
||||
|
||||
---
|
||||
|
||||
### - [x] T6: 修复独立验证炸出的阻塞缺陷(计划外,2026-08-06)
|
||||
|
||||
T1–T5 全绿、全部门禁通过之后,全新上下文的 verifier 用一个**走 `QuotaGate` 的**端到端用例炸出:装配缺陷在唯一的生产路径上根本没有拆出去。
|
||||
|
||||
**缺陷**: `QuotaGate`/`BreakerGate` 的 `except GovernanceBackendError: raise` 只放行了旧类型,新增的 `SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包成 `GovernanceBackendError`(`reason=governance_backend_down`、`retry_after_s=5.0`)。实证:
|
||||
|
||||
```
|
||||
RAISED: GovernanceBackendError | isGatewayUnavailable=True | isSourceNotConfigured=False
|
||||
| 限流后端故障(source_stats): 未知源 's1'(scope=llm)
|
||||
```
|
||||
|
||||
即配置写错的任务照样落进"可延期重投"家族,**永远重投、永不进死信、无人告警**——正是 Q1 要防的镜像 bug,G2 等于没做。
|
||||
|
||||
**为什么原有测试测不出来**: T3 写的两条用例(`test_backpressure.py`、`test_redis_key_layout.py`)都直接打私有 `_cfg()`,绕过了包装器;而治理循环只经包装器访问后端。**盲区在于测试打的层次比生产路径低一层。**
|
||||
|
||||
**修复**(三处):
|
||||
|
||||
| 文件 | 改动 |
|
||||
|---|---|
|
||||
| `middleware/ratelimit.py` | 4 个方法的放行扩为 `except (GovernanceBackendError, SourceNotConfiguredError): raise` |
|
||||
| `middleware/breaker.py` | 同上,5 个方法 |
|
||||
| `middleware/telemetry.py:254` | 终态捕获元组加 `SourceNotConfiguredError`。**连带坑**: 放行生效后该异常不再是 `GovernanceBackendError`,而它在任何 attempt 之前抛出,若不显式捕获则 `emit_terminal_failure` 不触发、该路径**遥测归零**,违反"遥测必录"铁律 |
|
||||
|
||||
**回归测试**: `test_backpressure.py::TestUnknownSourceIsAssemblyDefect::test_survives_the_quota_gate_wrapper`(参数化覆盖 `try_acquire` / `stats`),**走包装器而非私有方法**。修前 2 failed,修后 PASS。
|
||||
|
||||
**同批文档订正**: 泄漏路径由"三条"改为**五条**(遗漏了 `QuotaGate.stats` 与 `BreakerGate.retry_after_s`,判据是该调用点是否被 `_record_quietly` 包裹);CHANGELOG 的 `per_source_reasons` 表述改为"属性存在但恒为 `{}`"。
|
||||
|
||||
## 完成后
|
||||
|
||||
按 CLAUDE.md §3 Phase 2,合并前须派**全新上下文**的 verifier subagent 做独立验证(`verification-before-completion`),并按新规则**前台运行**。随后走 `finishing-a-development-branch` 决定合并方式,并在 Gitea 关闭 issue #7。
|
||||
|
||||
@@ -34,4 +34,12 @@ date: 2026-08-06
|
||||
|
||||
Codex 同时独立核实了计划的可执行性锚点: 22 处构造点、三处 gate 装配、后端层 `self._scope` 位置、README/ARCH 章节行号,均与 `src/` 现状相符。
|
||||
|
||||
## 独立验证炸出的阻塞缺陷(2026-08-06,全新上下文 verifier)
|
||||
|
||||
T1–T5 全绿、四道门禁全过之后,verifier 用一个**走 `QuotaGate` 的**端到端用例证明: 装配缺陷在唯一的生产路径上根本没拆出去——包装器的 `except GovernanceBackendError: raise` 只放行旧类型,`SourceNotConfiguredError` 落进下一行 `except Exception` 被重新包回去,配置写错照样永远重投。**盲区在于 T3 写的两条用例都直接打私有 `_cfg()`,比生产路径低一层。**
|
||||
|
||||
修复见正文 §T6(9 处放行 + 遥测终态捕获 + 走包装器的回归测试)。复核时 verifier 又指出一颗雷: 新放行让该异常能穿透 `_record_quietly`,而那层降级的存在理由是"调用已真实完成,写回失败不该丢弃成功响应"——同批把三处 `_record_quietly` 一并放宽并加了回归断言。
|
||||
|
||||
两轮都订正了同一处事实错误: 闸门泄漏路径是**五条**不是三条(`QuotaGate.stats` 与 `BreakerGate.retry_after_s` 同样未被 `_record_quietly` 包裹)。
|
||||
|
||||
相关: [[governance-backend-error]](design)、[[m2-distributed]]
|
||||
|
||||
@@ -34,6 +34,7 @@ from polygateway.errors import (
|
||||
RequestRejectedError,
|
||||
ResultInvalidError,
|
||||
SourceDeadError,
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
@@ -326,7 +327,7 @@ class EmbeddingClient:
|
||||
await write_back
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except GovernanceBackendError as exc:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
logger.warning("embedding 治理记账写回降级(不冒泡): {}", exc)
|
||||
|
||||
async def _settle_and_release(self, permit: Permit, actual: int) -> None:
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from polygateway.errors import GovernanceBackendError
|
||||
from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from polygateway.ports import GateDecision, GateUpdate, ProviderGate
|
||||
@@ -22,43 +22,53 @@ class BreakerGate:
|
||||
async def try_enter(self, source: SourceConfig, owner: str) -> GateDecision:
|
||||
try:
|
||||
return await self._gate.try_enter(source.name, owner)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"熔断后端故障(try_enter): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"熔断后端故障(try_enter): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def record_success(
|
||||
self, entry: GateDecision, *, count_attempt: bool = True
|
||||
) -> GateUpdate:
|
||||
try:
|
||||
return await self._gate.record_success(entry, count_attempt=count_attempt)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"熔断后端故障(record_success): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"熔断后端故障(record_success): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def record_failure(
|
||||
self, entry: GateDecision, reason: str, force_open: bool
|
||||
) -> GateUpdate:
|
||||
try:
|
||||
return await self._gate.record_failure(entry, reason, force_open)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"熔断后端故障(record_failure): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"熔断后端故障(record_failure): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def release_probe(self, entry: GateDecision) -> GateUpdate:
|
||||
try:
|
||||
return await self._gate.release_probe(entry)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"熔断后端故障(release_probe): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"熔断后端故障(release_probe): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def retry_after_s(self, sources: tuple[str, ...]) -> float:
|
||||
try:
|
||||
return await self._gate.retry_after_s(sources)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"熔断后端故障(retry_after_s): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"熔断后端故障(retry_after_s): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
@@ -8,7 +8,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from polygateway.errors import GovernanceBackendError
|
||||
from polygateway.errors import GovernanceBackendError, SourceNotConfiguredError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from polygateway.ports import Permit, RateLimiter
|
||||
@@ -26,31 +26,39 @@ class QuotaGate:
|
||||
async def try_acquire(self, source: SourceConfig) -> Permit | None:
|
||||
try:
|
||||
return await self._limiter.try_acquire(source.name, source.effective_est_tokens())
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"限流后端故障(try_acquire): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"限流后端故障(try_acquire): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def stats(self, source: SourceConfig) -> SourceStats:
|
||||
try:
|
||||
return await self._limiter.source_stats(source.name)
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"限流后端故障(source_stats): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"限流后端故障(source_stats): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def mark_progress(self) -> None:
|
||||
try:
|
||||
await self._limiter.mark_progress()
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"限流后端故障(mark_progress): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"限流后端故障(mark_progress): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
async def progress_age_s(self) -> float:
|
||||
try:
|
||||
return await self._limiter.progress_age_s()
|
||||
except GovernanceBackendError:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise GovernanceBackendError(f"限流后端故障(progress_age_s): {exc}", scope=self._scope) from exc
|
||||
raise GovernanceBackendError(
|
||||
f"限流后端故障(progress_age_s): {exc}", scope=self._scope
|
||||
) from exc
|
||||
|
||||
@@ -28,6 +28,7 @@ from polygateway.errors import (
|
||||
RequestRejectedError,
|
||||
ResultInvalidError,
|
||||
SourceDeadError,
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
@@ -401,7 +402,7 @@ class RetryMW:
|
||||
await write_back
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except GovernanceBackendError as exc:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
logger.warning("治理记账写回降级(不冒泡): {}", exc)
|
||||
|
||||
def _feed_outcome(self, source_name: str, ok: bool) -> None:
|
||||
|
||||
@@ -17,7 +17,11 @@ from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from polygateway.errors import GatewayUnavailableError, GovernanceBackendError
|
||||
from polygateway.errors import (
|
||||
GatewayUnavailableError,
|
||||
GovernanceBackendError,
|
||||
SourceNotConfiguredError,
|
||||
)
|
||||
from polygateway.middleware.cache import digest_messages
|
||||
from polygateway.types import canonical_sampling_json, merge_sampling
|
||||
|
||||
@@ -247,7 +251,7 @@ class TelemetryMW:
|
||||
started = self._now()
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except (GatewayUnavailableError, GovernanceBackendError) as exc:
|
||||
except (GatewayUnavailableError, GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
await self._emitter.emit_terminal_failure(
|
||||
request=request,
|
||||
call_id=str(uuid.uuid4()),
|
||||
|
||||
@@ -30,6 +30,7 @@ from polygateway.errors import (
|
||||
RequestRejectedError,
|
||||
ResultInvalidError,
|
||||
SourceDeadError,
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.breaker import BreakerGate
|
||||
@@ -360,7 +361,7 @@ class OcrClient:
|
||||
await write_back
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except GovernanceBackendError as exc:
|
||||
except (GovernanceBackendError, SourceNotConfiguredError) as exc:
|
||||
logger.warning("OCR 治理记账写回降级(不冒泡): {}", exc)
|
||||
|
||||
async def _settle_and_release(self, permit: Permit) -> None:
|
||||
|
||||
@@ -18,6 +18,7 @@ from polygateway.errors import (
|
||||
SourceNotConfiguredError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.middleware.ratelimit import QuotaGate
|
||||
from polygateway.middleware.retry import RetryMW, backoff_delay
|
||||
from polygateway.sources import RoundRobinSelector, SourceCooldownMemo
|
||||
from polygateway.types import (
|
||||
@@ -192,6 +193,12 @@ class _LimiterProgressBroken(InMemoryLimiter):
|
||||
raise GovernanceBackendError("redis 抖动", scope="llm")
|
||||
|
||||
|
||||
class _GateSuccessMisconfigured(InMemoryGate):
|
||||
# 签名须与端口一致(含 count_attempt),否则抛的是 TypeError 而非本类要测的异常
|
||||
async def record_success(self, entry, *, count_attempt: bool = True):
|
||||
raise SourceNotConfiguredError("未知源 's1'(scope=llm)")
|
||||
|
||||
|
||||
class TestAccountingDegradation:
|
||||
"""记账侧降级(设计 §10,ARCH §7.3 勘误): 调用已完成,写回失败不冒泡。"""
|
||||
|
||||
@@ -206,6 +213,24 @@ class TestAccountingDegradation:
|
||||
resp = await mw(_REQ)
|
||||
assert resp.content == "ok" # 真实成功响应不因记账失败被丢弃
|
||||
|
||||
async def test_assembly_defect_on_accounting_path_also_degrades(self):
|
||||
"""记账侧降级按"路径性质"而非异常类型: 装配缺陷同样不得毁掉已完成的调用。
|
||||
|
||||
`SourceNotConfiguredError` 被放行穿透闸门包装器(issue #7 §T6)后,若
|
||||
`_record_quietly` 只降级 `GovernanceBackendError`,它就会从记账侧冒泡、
|
||||
销毁一个真实成功的响应——反转本类钉住的既有行为。当前无后端会从记账
|
||||
方法抛它,此用例是为将来加了源名校验的后端守住这条不变式。
|
||||
"""
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
limiter = InMemoryLimiter(
|
||||
scope="llm", sources={"s1": src}, global_limits=_NO_GLOBAL, now=clock
|
||||
)
|
||||
gate = _GateSuccessMisconfigured(config=_BREAKER, now=clock)
|
||||
mw = _mw([src], limiter, [_ok()], clock=clock, sleep=BoundedSleep(), gate=gate)
|
||||
resp = await mw(_REQ)
|
||||
assert resp.content == "ok"
|
||||
|
||||
async def test_mark_progress_failure_does_not_lose_response(self):
|
||||
clock = FakeClock()
|
||||
src = make_source()
|
||||
@@ -280,14 +305,34 @@ class TestUnknownSourceIsAssemblyDefect:
|
||||
# 关键: 若归入 scope 级家族,配置写错的任务会永远延期重投、永不进死信
|
||||
assert not isinstance(ei.value, GatewayUnavailableError)
|
||||
|
||||
@pytest.mark.parametrize("method", ["try_acquire", "stats"])
|
||||
async def test_survives_the_quota_gate_wrapper(self, method):
|
||||
"""必须穿透 QuotaGate,否则整个拆分在生产路径上等于没做。
|
||||
|
||||
上面两条(以及 redis 版)打的都是私有 `_cfg`,绕过了包装器。而治理循环
|
||||
只经 QuotaGate 访问后端,包装器的 `except Exception` 会把装配缺陷重新
|
||||
包成 `GovernanceBackendError`——下游又拿到可重投异常,永远重投不告警。
|
||||
"""
|
||||
src = make_source("s1")
|
||||
# 限流后端的源名单与治理循环拿到的源对不上 = 装配缺陷
|
||||
limiter = InMemoryLimiter(
|
||||
scope="llm", sources={"other": src}, global_limits=_NO_GLOBAL
|
||||
)
|
||||
gate = QuotaGate(limiter, scope="llm")
|
||||
with pytest.raises(SourceNotConfiguredError) as ei:
|
||||
await getattr(gate, method)(src)
|
||||
assert not isinstance(ei.value, GatewayUnavailableError)
|
||||
|
||||
|
||||
class TestGateFailuresReachCallersAsScopeLevel:
|
||||
"""三条闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
|
||||
"""闸门泄漏路径必须以 scope 级不可用的形态到达调用方(issue #7)。
|
||||
|
||||
记账路径由 `_record_quietly` 降级为 warning,但闸门路径没有那层包裹,会一路
|
||||
抛给调用方。只写 `except GatewayUnavailableError` 的调用方此前接不住,后果
|
||||
是 Redis 抖一下就让积压任务烧掉业务失败预算进死信——而那是运维重启即可恢复
|
||||
的故障。三条路径逐一钉住,防止将来任何一条被漏掉。
|
||||
的故障。全部五条为: `QuotaGate` 的 try_acquire / stats / progress_age_s,
|
||||
`BreakerGate` 的 try_enter / retry_after_s(判据是该调用点未被 `_record_quietly`
|
||||
包裹)。此处钉住其中三条代表路径,余两条由同一注入机制覆盖。
|
||||
"""
|
||||
|
||||
async def test_try_acquire_failure_is_scope_level(self):
|
||||
|
||||
Reference in New Issue
Block a user