5 Commits

Author SHA1 Message Date
iomgaa afd6101c08 test: cover the env-key messages left unguarded by mutation testing
Mutation testing showed the negative structured-retries and expected-dim checks
in the env parsing path could be deleted with every test still passing. Their
value is the env key name in the message, so they need tests that assert it.

Changelog now states the real scope of this release and warns that normalising
scope moves the Redis keys, the one change here that silently relocates runtime
state. Records the breaker threshold derivation as deliberately env-only so it
does not resurface as another round.
2026-07-30 02:31:51 -04:00
iomgaa 726f26d8bd fix: normalise scope and blank strings on the construction path too
The verifier found four more env-only behaviours of the same class the branch
was already fixing. The worst is scope: it goes straight into the Redis keys
(pgw:limit:{scope}, pgw:gate:{scope}), so one process using from_env("LLM")
and another constructing scope="LLM" by hand split the rate limit and breaker
state across two namespaces, each tracking its own quota, with no error.

Blank redis_url and pricing_path now collapse to None as from_env has always
done, so they fall into the required-field checks instead of reaching the redis
client as an unparseable URL. EmbeddingSettings gains the __post_init__ it never
had, moving its batch_size and expected_dim checks off the from_env-only path.

Also adds the cache backend whitelist test that mutation testing showed missing.
2026-07-30 02:15:32 -04:00
iomgaa c9fdff9d55 fix: keep credentials out of the DSN rewrite warning
The warning added earlier in this branch logged the whole Postgres DSN, password
included, and nothing else in the library has ever printed a connection string.
It now reports only the scheme segment, which is the part that actually changed.

Regression test asserts the password and host/path never reach the log.
2026-07-30 01:13:45 -04:00
iomgaa a65b504a3d fix: consolidate remaining assembly validation into GatewaySettings
Round two of the from_env-only validation problem. Fifteen checks still lived
in the env parsing functions: six enum domains, the redis_url requirement for
redis-backed limiter/breaker/cache, cache namespace and TTL, telemetry path and
DSN, non-negative structured retries and non-blank scope. from_settings and
direct construction bypassed all of them.

The five asserts in client.py that claimed config had already validated
redis_url and the telemetry targets now hold on every path, so they revert to
what CLAUDE.md permits: internal invariant declarations that also narrow the
Optional for type checkers. Their comments now name the method that guarantees
them, since the previous wording is exactly what went stale.

Postgres DSNs built by hand now get the SQLAlchemy +driver suffix stripped the
way from_env has always stripped it, with a warning so the rewrite is not
silent. The env path strips earlier, so it stays quiet.
2026-07-30 00:58:33 -04:00
iomgaa 8c9e1179bc docs: design second round of settings validation consolidation
The independent verifier found 15 more checks still living only in from_env:
six enum domains, seven conditional-required pairs and two scalar ranges.
More severe than round one because client.py has five asserts that claim
config already validated the redis_url and telemetry paths, which is false on
the from_settings path.

Also records a normalisation gap the verifier missed: _load_pg_dsn strips the
SQLAlchemy +asyncpg suffix, so a hand-built DSN reaches asyncpg unstripped.
2026-07-30 00:46:44 -04:00
12 changed files with 546 additions and 24 deletions
+20
View File
@@ -1,5 +1,25 @@
# Changelog
## 1.0.2(2026-07-30)
1.0.1 的续作:那一版把三条跨字段守卫收进构造期后,独立验证发现 `from_env` 上还留着同一类的 15 条校验与 4 条规范化,一并收拢。
### 修复
- **后端选择与条件必填项在任何构造路径上都校验。** 以下此前只有 `from_env` 拦得住,`from_settings()` 与直接构造一律放行:`limiter_backend`/`breaker_backend`/`cache_backend`/`telemetry_backend`/`selector`/`quota_full` 六个字段的合法域;取 `redis` 的后端必须有 `redis_url`;启用缓存必须有 `cache_namespace` 与正 `cache_ttl_s`;`telemetry_backend``sqlite`/`postgres` 时对应的路径/DSN 必填;`structured_max_retries` 非负;`scope` 非空。
- **`client.py` 五处断言的前提现在真的成立。** `assert settings.redis_url is not None # 内部不变量: config 已校验` 之类的注释此前在 `from_settings` 路上是假的:断言开启时抛不含任何字段信息的 `AssertionError`,`python -O` 下断言被移除、错误退化为 redis 库抛出的连接串解析异常。注释已改为点明由哪个校验方法保证。
- **构造路补齐了 `from_env` 一直在做的规范化**,两条装配路对同一输入产出同一个值:
- `scope` 小写并去空白。它直接进 Redis key(`pgw:limit:{scope}:…``pgw:gate:{scope}:…`),此前一个进程走 `from_env("LLM")` 拿到 `llm`、另一个直接构造传 `"LLM"`,**同一逻辑 scope 的限流与熔断状态会分裂到两套命名空间**,各记各的配额与熔断状态,分布式治理静默失效且不报错。
- `redis_url``pricing_path` 的空串归 `None`。留着空串会骗过 `is None` 判断,把错误推迟成 redis 客户端的连接串解析异常或 `Is a directory: '.'`
- Postgres DSN 剥掉 SQLAlchemy 驱动后缀(`postgresql+asyncpg://…``+asyncpg` asyncpg 不认)。这一条剥的时候会发一条 warning——库动了调用方给的值,不该静默;日志只出现 scheme 段,DSN 带密码,整串不进日志。经 `from_env` 装配的不受影响也不会有这条 warning(`_load_pg_dsn` 早就剥干净了)。
- **`EmbeddingSettings``batch_size` / `expected_dim` 域校验也移入构造期**,此前只有 `EmbeddingSettings.from_env` 校验,直接构造出 `batch_size=-3` 要到 `EmbeddingClient` 构造时才 fail-loud。
### 行为收紧(下游请读)
同 1.0.1:经 `from_env()` 装配的调用方**不受影响**。手工构造 `GatewaySettings` 或对它 `dataclasses.replace` 的调用方,若配置组合非法,现在会在构造期抛 `ValueError` 并点出字段名,而不是留到运行时表现为静默不建后端、裸 `AssertionError` 或第三方库的天书报错。
**一处静默改值需要留意**:此前手工构造传 `scope="LLM"`(非全小写)的调用方,升级后 scope 会被规范化为 `llm`,**Redis key 随之从 `pgw:limit:LLM:…` 切到 `pgw:limit:llm:…`**。这正是本次要修的问题——旧行为下这批 key 与 `from_env` 装配的进程根本不在同一命名空间;但切换发生的那一刻,旧键上的在途租约会被遗弃,靠 TTL 自愈。滚动升级期间建议留意限流配额短暂偏松。
## 1.0.1(2026-07-30)
### 修复
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "polygateway"
version = "1.0.1"
version = "1.0.2"
description = "PolyGateway:实验室统一的大语言模型(LLM/VLM/OCR)调度与中转库——多源、限流、重试、熔断、缓存、遥测"
requires-python = ">=3.11"
dependencies = [
@@ -0,0 +1,164 @@
# GatewaySettings 装配校验补齐(第二轮)
- **日期**: 2026-07-30;**状态**: **已批准并实施**(2026-07-30 人类门通过;§9 结论、§10 实施留痕)
- **缘起**: [2026-07-29-settings-invariant-guards-design.md](2026-07-29-settings-invariant-guards-design.md) §9.1 —— 独立 verifier 在第一轮交付后发现,`from_env` 上还留着一批同族校验;本设计是那一轮的续作,**同一个 bug 类的剩余部分**
- **上游依据**: 第一轮设计 §2 已批准的方案 A(不变量归属于类,不归属于某个工厂);CLAUDE.md §4.3(assert 仅用于内部不变量)、§4.5(装配只有两条路)
## 1. 待收拢的校验清单(逐条实测确认只在 `from_env` 生效)
### A. 枚举合法域(6 条)
| 字段 | 合法域 | 现居 |
|---|---|---|
| `limiter_backend` / `breaker_backend` | `{memory, redis}` | `_load_pgw`(经 `_load_choice`) |
| `cache_backend` | `{redis, memory, none}` | `_load_pgw` 内联 |
| `telemetry_backend` | `{sqlite, postgres, none}` | `_load_pgw` 内联 |
| `selector` | `_SELECTORS` | `from_env``_load_choice` |
| `quota_full` | `_QUOTA_FULL` | `from_env``_load_choice` |
直接构造传 `selector="random"``cache_backend="rediss"` 一律放行,后果是装配时落进 `_build_*` 的 else 分支或静默不建后端。
### B. 条件必填(7 条,跨字段)
| 条件 | 要求 | 违反后果 |
|---|---|---|
| `limiter_backend`/`breaker_backend`/`cache_backend``redis` | `redis_url` 非空 | **见 §2**,最严重 |
| `cache_backend != "none"` | `cache_namespace` 非空 | 缓存 key 失去租户隔离——踩"无缓存毒化"铁律 |
| `cache_backend != "none"` | `cache_ttl_s > 0` | `from_env` 明令禁止的"永不过期"从另一条路进来 |
| `telemetry_backend == "sqlite"` | `telemetry_sqlite_path` 非空 | 断言炸或写空路径 |
| `telemetry_backend == "postgres"` | `telemetry_pg_dsn` 非空 | 同上 |
### C. 标量域(2 条)
`structured_max_retries ≥ 0`;`scope` 非空(空 scope 会污染遥测与缓存命名空间)。
## 2. 为什么这批比第一轮更严重:`client.py` 的断言前提为假
`client.py` 有 5 处断言**明文声称这个前提已经成立**:
```python
assert settings.redis_url is not None # 内部不变量: config 已校验
```
位置:`client.py:262/282/302`(redis_url)、`:312`(pg_dsn)、`:316`(sqlite_path)。走 `from_settings` 时该注释是假的,verifier 实测:
| 运行方式 | 结果 |
|---|---|
| 断言开启 | `AssertionError()` —— 裸断言,不点字段、不说原因 |
| `python -O` | 断言消失,退化为 redis 库的 `ValueError: Redis URL must specify one of the following schemes...` |
后者正是 CLAUDE.md §4.3 禁止的"assert 承担生产校验"。
**但注意结论的方向**:这 5 处 assert 本身不是要修的东西——它们要的前提是对的,错的是没人保证这个前提。§4 给出处置。
## 3. 方案
沿用第一轮已批准的方案 A,不重新论证:全部收进 `GatewaySettings.__post_init__`,新增三个私有方法与既有四个并列。
| 方法 | 覆盖 |
|---|---|
| `_validate_backends` | A 类 6 条枚举 + B 类 redis_url 三条件 |
| `_validate_cache` | `cache_namespace` 非空、`cache_ttl_s > 0`(仅 `cache_backend != "none"` 时) |
| `_validate_telemetry` | sqlite path / postgres dsn 条件必填 + §5 的 DSN 形态 |
标量两条(`structured_max_retries``scope`)并入 `_validate_sources` 改名后的 `_validate_identity`,与 `SourceConfig._validate_identity` 同名同职。
枚举合法域上提为模块级 frozenset 常量(`_LIMITER_BACKENDS` 等),`_load_pgw``__post_init__` 共用一份,消除现有的内联字面量重复。
**否决的替代**:在 `_build_limiter`/`_build_cache` 等工厂函数里逐个补显式检查。理由同第一轮 §2 方案 B——校验散落在消费点,每加一个后端就多一处要同步,且 `dataclasses.replace` 仍绕过。
## 4. 5 处 assert 的处置:**保留,不改**
修好构造期校验后,`settings.redis_url is not None` 就真的成了内部不变量——CLAUDE.md §4.3 原文"assert 仅用于内部不变量"说的正是这种用法,同时它给类型检查器收窄了 `str | None`。此时删掉 assert 反而丢失类型信息,改成 `raise` 则是在防御一个已被构造期排除的情况(死代码)。
**要改的是注释**:`# 内部不变量: config 已校验` 应点明由谁保证,例如 `# 内部不变量: GatewaySettings._validate_backends 已保证`。前一轮的教训就是这类注释会随时间变成谎言。
## 5. Postgres DSN:校验而非规范化(本轮唯一的新决策)
`_load_pg_dsn``from_env` 读到的 DSN 做了**规范化**:剥掉 SQLAlchemy 风格的 `+asyncpg` 驱动后缀(asyncpg 不认)。直接构造那条路不会剥,`postgresql+asyncpg://...` 会原样送进 asyncpg 然后在首次写遥测时才炸。
| 选项 | 权衡 |
|---|---|
| A. 构造期校验,含 `+driver` 即报错 | 显式,库不碰用户给的值;但两条装配路对同一输入接受度不同 |
| B. 构造期静默剥后缀 | 两条路完全对齐;但 frozen 类在构造期悄悄改字段,调用方不知情 |
| **C. 构造期剥后缀 + `logger.warning`(用户 2026-07-30 拍板)** | 两条路行为对齐,同时不静默——调用方在日志里看得见库动了他的值,想根治就自己改 DSN |
选 C。实现要点:`object.__setattr__` 改 frozen 字段(`SourceConfig` 无此先例,但 frozen 的约束是对**外部**不可变,构造期规范化是既有 dataclass 惯用法);warning 走 loguru(核心依赖,库内 `ocr.py:183`/`embedding.py:318` 同款用法)。
**warning 不会打扰 env 用户**:`_load_pg_dsn` 保留现有的剥离逻辑,`from_env` 传给构造函数时 DSN 已经干净,`__post_init__` 无事可做。只有手工构造传了带后缀的 DSN 才会触发。三项目 `.env` 里那些 SQLAlchemy 写法不会每次装配刷一条 warning。
代价是同一件事有两处剥离逻辑。用同一个模块级 helper `_strip_dsn_driver(dsn)` 供两处调用,避免实现分叉。
## 6. 行为审计
| 现有行为 | 处置 |
|---|---|
| `from_env` 对上述 15 条的校验与报错 | **全部保留**,时机提前到 `cls(...)`;`_load_*` 内联检查删除,避免同一约束两处维护 |
| `_load_pg_dsn``+driver` | **保留**,继续只在 env 路径生效(§5) |
| `_load_choice``default` 语义(键缺失时取默认) | **保留**,那是 env 解析职责,不是不变量 |
| `_load_breaker` 的有效阈值派生 `max(配置值, 源级并发×2)` | **有意保留在 env 层**(verifier 二次核验点名,记此备案免成"第五批")。它是**派生**不是校验/规范化:两路产出确实不同(env 装配 threshold=5/并发=100 得 200,直接构造得 5),但派生依赖的是"用户没显式表态时库替他选一个合理值"的 env 语义;代码构造那条路,调用方给什么就是什么表态。其跨字段下限风险由 `_validate_probe` 在构造期兜底 |
| 直接构造出上述任一非法组合 → 静默成功 | **有意替换**为构造期 `ValueError` |
| `client.py` 5 处 assert | **保留**,仅改注释(§4) |
| 异常类型 | 一律 `ValueError`,与第一轮及既有装配错误一致 |
**有意放弃**:不校验 `pricing_path` 指向的文件是否存在(I/O 不属于配置校验,`PricingTable.from_file` 自会报错);不强制 `cache_backend == "none"` 时 namespace/ttl 必须为 None(多余字段无害)。
## 7. 非功能维度
与第一轮同构,不重复论证:`__post_init__` 纯同步计算无 I/O(不适用并发/取消/持久化);装配期属准入侧,报错不放行;`__post_init__` 不改字段故幂等。**性能**:新增约 10 次字符串比较,第一轮实测单次构造 1.45 µs 且库内无热路径构造 `GatewaySettings`,可忽略。
## 8. 测试策略
`tests/unit/test_config.py::TestCrossFieldInvariants` 扩充(不新建类,同族不变量归一处):
| 用例组 | 断言 |
|---|---|
| 6 条枚举各一条非法值 | 抛 `ValueError`,消息含字段名与合法域 |
| redis_url 三条件(limiter/breaker/cache 各一) | 抛 `ValueError`,消息点明需要 `redis_url` |
| cache namespace 缺失 / ttl ≤ 0 | 抛 `ValueError` |
| telemetry sqlite path / pg dsn 缺失 | 抛 `ValueError` |
| `structured_max_retries=-1``scope=""` | 抛 `ValueError` |
| pg dsn 含 `+asyncpg`(直接构造) | 后缀被剥,字段值为干净 DSN,且发出一条 warning(用 `caplog`/loguru sink 断言) |
| pg dsn 干净(直接构造)、或经 `from_env` 传入 | **不发** warning——env 路已在 `_load_pg_dsn` 剥过,不该刷噪音 |
| 合法组合(每种 backend 组合各一) | 构造成功——收紧的是错的那些 |
| **回归护栏**:`GatewayClient.from_settings` 走 redis 三后端的合法配置 | 装配成功,证明 assert 前提真的被保证了 |
TDD:先跑出红,预计 ≥14 条失败。要求同第一轮——每条实现改动都要有对应测试能杀死它。
版本:**1.0.2**(patch),CHANGELOG 同样单列"行为收紧"小节。
## 9. 人类拍板结论(2026-07-30)
| 问题 | 结论 |
|---|---|
| §5 DSN 处置 | **选 C**:构造期剥后缀 + `logger.warning`。不静默改用户的值,也不让两条装配路产出不一致 |
| §4 assert 处置 | **保留,只改注释**,点明由哪个方法保证前提 |
| 方案主体 | 沿用第一轮已批准的方案 A,无需重新论证 |
| 版本 | 1.0.2(patch) |
| **范围追加**(实施中经 verifier 发现后拍板) | G1-G4 四条同族遗漏一并纳入本轮;G1 的 scope 规范化取**静默**小写+strip(不告警——`from_env` 一直静默小写,scope 大小写不承载语义) |
## 10. 实施留痕
分支 `fix/settings-invariants-round-2`。TDD 两段:主体 15 条先 **16 failed**、G1-G4 追加 **9 failed**,实现后全绿(547 passed / 14 skipped,1.0.1 基线 516)。
### 10.1 独立 verifier 的关键发现
第一次核验判**有阻塞**,已修:
- **阻塞(本轮新引入)**:DSN 剥离的 warning 打印了完整连接串,**含明文密码**,而库内此前从无任何地方打印连接串——违反 P5。已改为只报 scheme 段变化,并补回归测试断言密码与 host/path 不进日志。
- **变异测试 27/28 被杀**,唯一存活的是 `_load_pgw``PGW_CACHE_BACKEND` 域检查删掉后仍全绿(该 env 层 raise 零覆盖)。已补 `test_cache_backend_whitelist`,与既有 `test_telemetry_backend_whitelist` 对称。
- **assert 处置经独立核验成立**:遍历所有可达构造路径均无法制造 assert 失败,`python -O` 下同样在构造期被拦(旧病症消失);唯一能触发的是 `object.__new__` 绕过 `__post_init__` 的人造路径,非公共 API。
- **frozen 语义无副作用**:`object.__setattr__``hash`/相等性/集合去重正常,`replace` 幂等不重复告警,`pickle`/`deepcopy` 不触发 `__post_init__` 故不重复告警,对外仍抛 `FrozenInstanceError`
### 10.2 G1-G4:第三批遗漏(已纳入本轮)
verifier 通读 `_load_*` 后发现,除设计 §1 的 15 条外还有四条**规范化**只在 env 路生效——与本轮所修的 DSN 是同一类:
| | 内容 | 危害 |
|---|---|---|
| G1 | `scope` 小写化 | **最严重**:scope 进 Redis key,大小写不一致使限流/熔断状态分裂到两套命名空间,分布式治理静默失效 |
| G2 | `redis_url` 空串归 None | 空串骗过 `is None`,退化为 redis 客户端的连接串天书报错——正是本轮 CHANGELOG 声称已消除的那种 |
| G3 | `pricing_path` 空串归 None | 退化为 `Is a directory: '.'` |
| G4 | `EmbeddingSettings.batch_size`/`expected_dim` 域 | 该类无 `__post_init__`;晚一步到 client 构造才 fail-loud |
统一收进新增的 `GatewaySettings._normalize()`(在全部 `_validate_*` 之前跑)与 `EmbeddingSettings.__post_init__`。DSN 后缀因需看 backend 且需告警,规范化留在 `_validate_telemetry`
@@ -0,0 +1,17 @@
---
type: design
node_id: design:settings-invariants-round-2
title: "GatewaySettings 装配校验补齐(第二轮)"
date: 2026-07-30
---
# GatewaySettings 装配校验补齐(第二轮)
全文见 [2026-07-30-settings-invariants-round-2-design.md](2026-07-30-settings-invariants-round-2-design.md)。第一轮见 [settings-invariant-guards](settings-invariant-guards.md)。
- **缘起**: 第一轮交付后独立 verifier 发现 `from_env` 上还留着 15 条同族校验(枚举合法域 6、条件必填 7、标量域 2),`from_settings` 与直接构造全部放行。
- **严重性高于第一轮**: `client.py:262/282/302/312/316` 有 5 处 `assert ... # 内部不变量: config 已校验` 明文依赖这个前提;实测断言开启抛裸 `AssertionError`,`python -O` 下退化为 redis 库天书。
- **方案**: 沿用第一轮已批准的方案 A,不重新论证;新增 `_validate_backends/_validate_cache/_validate_telemetry`,枚举合法域上提为模块级常量供 `_load_pgw` 与构造期共用。
- **assert 处置**: **保留不改**——前提一旦由构造期保证,它就是 CLAUDE.md §4.3 认可的内部不变量用法且给类型检查器收窄 `str | None`;只改那句会变成谎言的注释,点明由哪个方法保证。
- **本轮唯一新决策**: `_load_pg_dsn``+asyncpg` 驱动后缀是**规范化**不是校验,直接构造那条路不会剥。选校验拒绝(显式)而非构造期 `object.__setattr__` 剥后缀(在用户背后改 frozen 字段)。两条路接受度不同是有意的:env 路要吃三项目历史遗留的 SQLAlchemy DSN 写法,代码构造路没有历史包袱。
- **版本**: 1.0.2(patch),CHANGELOG 单列"行为收紧"小节。
+5
View File
@@ -95,6 +95,11 @@
"id": "design:settings-invariant-guards",
"label": "GatewaySettings 跨字段不变量守卫的生效范围",
"type": "design"
},
{
"id": "design:settings-invariants-round-2",
"label": "GatewaySettings 装配校验补齐(第二轮)",
"type": "design"
}
],
"links": [
+4 -2
View File
@@ -1,14 +1,16 @@
# Research Wiki 索引
> 自动生成,更新时间:2026-07-30 03:44 UTC
> 自动生成,更新时间:2026-07-30 04:44 UTC
## design (12)
## design (14)
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
- [2026-07-21-m25-resilience-design](designs/2026-07-21-m25-resilience-design.md) `design:2026-07-21-m25-resilience-design`
- [2026-07-21-m3-ocr-design](designs/2026-07-21-m3-ocr-design.md) `design:2026-07-21-m3-ocr-design`
- [2026-07-22-m4-migration-design](designs/2026-07-22-m4-migration-design.md) `design:2026-07-22-m4-migration-design`
- [2026-07-29-settings-invariant-guards-design](designs/2026-07-29-settings-invariant-guards-design.md) `design:2026-07-29-settings-invariant-guards-design`
- [2026-07-30-settings-invariants-round-2-design](designs/2026-07-30-settings-invariants-round-2-design.md) `design:2026-07-30-settings-invariants-round-2-design`
- [GatewaySettings 装配校验补齐(第二轮)](designs/settings-invariants-round-2.md) `design:settings-invariants-round-2`
- [GatewaySettings 跨字段不变量守卫的生效范围](designs/settings-invariant-guards.md) `design:settings-invariant-guards`
- [M1 核心里程碑设计:公共签名冻结与治理栈落地](designs/m1-core-design.md) `design:m1-core-design`
- [M2 分布式:Redis 治理后端+背压+Postgres 遥测+pricing+Embedding+压测 harness](designs/m2-distributed.md) `design:m2-distributed`
+2
View File
@@ -48,3 +48,5 @@
- [2026-07-22 14:36 UTC] 重建索引: 34 篇页面
- [2026-07-30 03:44 UTC] 新增 design: GatewaySettings 跨字段不变量守卫的生效范围 (design:settings-invariant-guards)
- [2026-07-30 03:44 UTC] 重建索引: 36 篇页面
- [2026-07-30 04:44 UTC] 新增 design: GatewaySettings 装配校验补齐(第二轮) (design:settings-invariants-round-2)
- [2026-07-30 04:44 UTC] 重建索引: 38 篇页面
+1 -1
View File
@@ -31,7 +31,7 @@ from polygateway.types import (
SourceConfig,
)
__version__ = "1.0.1"
__version__ = "1.0.2"
__all__ = [
"DEFAULT_PROFILES",
+5 -5
View File
@@ -259,7 +259,7 @@ def _build_limiter(settings: GatewaySettings, sources: list[SourceConfig]) -> Ra
if settings.limiter_backend == "redis":
from polygateway.backends.redis.limiter import RedisLimiter
assert settings.redis_url is not None # 内部不变量: config 已校验
assert settings.redis_url is not None # 内部不变量: _validate_backends 已保证
return RedisLimiter.from_url(
settings.redis_url,
scope=settings.scope,
@@ -279,7 +279,7 @@ def _build_breaker(settings: GatewaySettings) -> ProviderGate:
if settings.breaker_backend == "redis":
from polygateway.backends.redis.breaker import RedisGate
assert settings.redis_url is not None # 内部不变量: config 已校验
assert settings.redis_url is not None # 内部不变量: _validate_backends 已保证
return RedisGate.from_url(settings.redis_url, config=settings.breaker, scope=settings.scope)
return InMemoryGate(config=settings.breaker)
@@ -299,7 +299,7 @@ def _build_cache(settings: GatewaySettings) -> CacheBackend | None:
return InMemoryCache()
from polygateway.backends.redis_cache import RedisCache
assert settings.redis_url is not None # 内部不变量: config 已校验
assert settings.redis_url is not None # 内部不变量: _validate_backends 已保证
return RedisCache.from_url(settings.redis_url)
@@ -309,11 +309,11 @@ def _build_telemetry(settings: GatewaySettings) -> TelemetryRecorder | None:
if settings.telemetry_backend == "postgres":
from polygateway.telemetry.postgres import PostgresRecorder
assert settings.telemetry_pg_dsn is not None # 内部不变量: config 已校验
assert settings.telemetry_pg_dsn is not None # 内部不变量: _validate_telemetry 已保证
return PostgresRecorder(settings.telemetry_pg_dsn)
from polygateway.telemetry.sqlite import SQLiteRecorder
assert settings.telemetry_sqlite_path is not None # 内部不变量: config 已校验
assert settings.telemetry_sqlite_path is not None # 内部不变量: _validate_telemetry 已保证
return SQLiteRecorder(settings.telemetry_sqlite_path)
+114 -14
View File
@@ -16,6 +16,7 @@ from dataclasses import dataclass
from typing import TYPE_CHECKING
from dotenv import dotenv_values
from loguru import logger
from polygateway.types import (
BackpressurePolicy,
@@ -47,6 +48,12 @@ _SOURCE_FIELDS: dict[str, tuple[str, str]] = {
_RESERVED_SEGMENTS = frozenset({"GLOBAL", "RETRY", "BREAKER", "BACKPRESSURE"})
_SELECTORS = frozenset({"round_robin", "least_inflight", "health_aware"})
_QUOTA_FULL = frozenset({"wait", "fail_fast"})
# 后端合法域: env 解析与构造期校验共用一份定义,避免两处分叉
_LIMITER_BACKENDS = frozenset({"memory", "redis"})
_BREAKER_BACKENDS = frozenset({"memory", "redis"})
_CACHE_BACKENDS = frozenset({"redis", "memory", "none"})
_TELEMETRY_BACKENDS = frozenset({"sqlite", "postgres", "none"})
_REDIS_DEPENDENT_BACKENDS = ("limiter_backend", "breaker_backend", "cache_backend")
# 背压默认(M1 仅 poll 生效;CHS _BACKOFF_S=0.05 同源)
_DEFAULT_STALL_WINDOW_S = 300.0
_DEFAULT_POLL_INTERVAL_S = 0.05
@@ -122,15 +129,94 @@ class GatewaySettings:
lease_ttl_s: float
def __post_init__(self) -> None:
self._validate_sources()
self._normalize()
self._validate_identity()
self._validate_backends()
self._validate_cache()
self._validate_telemetry()
self._validate_lease()
self._validate_stall()
self._validate_probe()
def _validate_sources(self) -> None:
"""零源的配置装出来选源必然失败,构造期即拒。"""
def _normalize(self) -> None:
"""把 `from_env` 一直在做的规范化补到构造路上,两条路必须产出同一个值。
`scope` 最要紧: 它直接进 Redis key(`pgw:limit:{scope}:…`/`pgw:gate:{scope}:…`)。
一个进程走 `from_env("LLM")` 拿到 "llm"、另一个直接构造传 "LLM",同一逻辑
scope 的限流与熔断状态会分裂到两套命名空间,各记各的,治理静默失效且不报错。
空串归 None 同理: 留着空串会骗过 `is None` 判断,把错误推迟到 redis 客户端
抛连接串解析异常。`telemetry_pg_dsn` 的驱动后缀因为要看 backend 且需告警,
规范化留在 `_validate_telemetry`。
"""
normalized_scope = self.scope.strip().lower()
if normalized_scope != self.scope:
object.__setattr__(self, "scope", normalized_scope)
for field in ("redis_url", "pricing_path"):
if getattr(self, field) == "":
object.__setattr__(self, field, None)
def _validate_identity(self) -> None:
"""本类自身字段的基本域: 空 scope 会污染遥测与缓存命名空间;零源必然选源失败。"""
if not self.scope.strip():
raise ValueError("GatewaySettings.scope 不能为空")
if not self.sources:
raise ValueError("GatewaySettings.sources 不能为空: 至少一个源")
if self.structured_max_retries < 0:
raise ValueError(f"structured_max_retries 不能为负: {self.structured_max_retries}")
def _validate_backends(self) -> None:
"""后端选择必须落在合法域内,取 redis 的还必须有连接串。
域外取值此前只有 `from_env` 拦得住,直接构造会一路走到 `client.py` 的
`_build_*`,落进 else 分支静默不建后端,或撞上那里的断言。
"""
for field, allowed in (
("limiter_backend", _LIMITER_BACKENDS),
("breaker_backend", _BREAKER_BACKENDS),
("cache_backend", _CACHE_BACKENDS),
("telemetry_backend", _TELEMETRY_BACKENDS),
("selector", _SELECTORS),
("quota_full", _QUOTA_FULL),
):
value = getattr(self, field)
if value not in allowed:
raise ValueError(f"{field} 非法值 {value!r};允许: {sorted(allowed)}")
on_redis = [f for f in _REDIS_DEPENDENT_BACKENDS if getattr(self, f) == "redis"]
if on_redis and self.redis_url is None:
raise ValueError(f"{''.join(on_redis)} 取 redis 时必须提供 redis_url")
def _validate_cache(self) -> None:
"""启用缓存必须有命名空间与正 TTL(缺命名空间即失去租户隔离,会毒化缓存)。"""
if self.cache_backend == "none":
return
if not self.cache_namespace:
raise ValueError("启用缓存时 cache_namespace 不能为空: 缓存 key 靠它做租户隔离")
if self.cache_ttl_s is None or self.cache_ttl_s <= 0:
raise ValueError(f"cache_ttl_s 必须 > 0(禁止永不过期): {self.cache_ttl_s}")
def _validate_telemetry(self) -> None:
"""遥测后端各自的落点必填;顺带剥掉 asyncpg 不认的 SQLAlchemy 驱动后缀。
剥而不是拒: 两条装配路对同一 DSN 应产出同一结果。但不静默——`from_env`
那条路在 `_load_pg_dsn` 就剥干净了,能走到这里的只有手工构造的调用方,
他有权知道库动了他给的值。
"""
if self.telemetry_backend == "sqlite" and not self.telemetry_sqlite_path:
raise ValueError("telemetry_backend=sqlite 时必须提供 telemetry_sqlite_path")
if self.telemetry_backend != "postgres":
return
if not self.telemetry_pg_dsn:
raise ValueError("telemetry_backend=postgres 时必须提供 telemetry_pg_dsn")
stripped = _strip_dsn_driver(self.telemetry_pg_dsn)
if stripped != self.telemetry_pg_dsn:
# 只报 scheme 段: DSN 带密码,整串不得进日志(P5 敏感信息只走 .env)
logger.warning(
"telemetry_pg_dsn 的 scheme 含 asyncpg 不认的驱动后缀,已由 {} 剥为 {}",
self.telemetry_pg_dsn.partition("://")[0],
stripped.partition("://")[0],
)
object.__setattr__(self, "telemetry_pg_dsn", stripped)
def _validate_lease(self) -> None:
"""调用超时须 ≤ permit 租约 TTL,防租约先于请求过期使并发超出配额。"""
@@ -318,17 +404,15 @@ def _load_choice(env: Mapping[str, str], key: str, allowed: frozenset[str], defa
def _load_pgw(env: Mapping[str, str]) -> dict[str, object]:
limiter_backend = _load_choice(
env, "PGW_LIMITER_BACKEND", frozenset({"memory", "redis"}), "memory"
)
breaker_backend = _load_choice(
env, "PGW_BREAKER_BACKEND", frozenset({"memory", "redis"}), "memory"
)
# 合法域与构造期守卫共用常量;此处的检查保留是为了报错能点出 env 键名,
# 构造期那道点的是字段名(两类调用方各看得懂自己那套)
limiter_backend = _load_choice(env, "PGW_LIMITER_BACKEND", _LIMITER_BACKENDS, "memory")
breaker_backend = _load_choice(env, "PGW_BREAKER_BACKEND", _BREAKER_BACKENDS, "memory")
_, cache_backend = _require(env, "PGW_CACHE_BACKEND")
_, telemetry_backend = _require(env, "PGW_TELEMETRY_BACKEND")
if cache_backend not in ("redis", "memory", "none"):
if cache_backend not in _CACHE_BACKENDS:
raise ValueError(f"PGW_CACHE_BACKEND 非法值 {cache_backend!r}")
if telemetry_backend not in ("sqlite", "postgres", "none"):
if telemetry_backend not in _TELEMETRY_BACKENDS:
raise ValueError(f"PGW_TELEMETRY_BACKEND 非法值 {telemetry_backend!r}")
redis_url = env.get("REDIS_URL") or None
if "redis" in (limiter_backend, breaker_backend) and redis_url is None:
@@ -350,13 +434,22 @@ def _load_pgw(env: Mapping[str, str]) -> dict[str, object]:
}
def _load_pg_dsn(env: Mapping[str, str]) -> str:
"""读取 Postgres DSN 并剥 SQLAlchemy 风格驱动后缀(asyncpg 不认 `+driver`)"""
_, dsn = _require(env, "PGW_TELEMETRY_PG_DSN")
def _strip_dsn_driver(dsn: str) -> str:
"""剥 SQLAlchemy 风格的 `+driver` 后缀(asyncpg 不认);已干净的原样返回"""
scheme, sep, rest = dsn.partition("://")
return f"{scheme.partition('+')[0]}{sep}{rest}"
def _load_pg_dsn(env: Mapping[str, str]) -> str:
"""读取 Postgres DSN 并剥驱动后缀。
env 路在此剥干净,构造期那道就无事可做——三项目 `.env` 里的 SQLAlchemy
写法不会每次装配都刷一条 warning。
"""
_, dsn = _require(env, "PGW_TELEMETRY_PG_DSN")
return _strip_dsn_driver(dsn)
def _load_cache_keys(
env: Mapping[str, str], cache_backend: str, redis_url: str | None
) -> dict[str, object]:
@@ -399,6 +492,13 @@ class EmbeddingSettings:
normalize: bool = False
expected_dim: int | None = None
def __post_init__(self) -> None:
"""自身字段的域校验;内嵌的 gateway 由 `GatewaySettings.__post_init__` 自己把关。"""
if self.batch_size < 1:
raise ValueError(f"EmbeddingSettings.batch_size 必须 ≥ 1: {self.batch_size}")
if self.expected_dim is not None and self.expected_dim < 1:
raise ValueError(f"EmbeddingSettings.expected_dim 必须 ≥ 1: {self.expected_dim}")
@classmethod
def from_env(
cls,
+208 -1
View File
@@ -1,11 +1,13 @@
"""config.py 配置聚合测试(设计 §8): 多源命名、键优先级、缺失报错。"""
import contextlib
import dataclasses
import pytest
from loguru import logger
from polygateway.client import GatewayClient
from polygateway.config import GatewaySettings, OcrSettings
from polygateway.config import EmbeddingSettings, GatewaySettings, OcrSettings
_BASE_ENV = {
"LLM__QWEN__1__BASE_URL": "https://gw-a.example/v1",
@@ -22,6 +24,17 @@ _BASE_ENV = {
}
@contextlib.contextmanager
def _captured_warnings():
"""捕获库发出的 WARNING;loguru 不经标准 logging,pytest 的 caplog 抓不到。"""
messages: list[str] = []
sink_id = logger.add(messages.append, level="WARNING")
try:
yield messages
finally:
logger.remove(sink_id)
def _env(**overrides):
env = dict(_BASE_ENV)
env.update({k: v for k, v in overrides.items() if v is not None})
@@ -158,6 +171,11 @@ class TestAssemblyGuards:
s2 = GatewaySettings.from_env("LLM", env=_env(PGW_STRUCTURED_MAX_RETRIES="0"))
assert s2.structured_max_retries == 0
def test_negative_structured_retries_rejected_with_env_key(self):
"""env 层的检查保留是为了报错能点出键名(构造期那道点的是字段名)。"""
with pytest.raises(ValueError, match="PGW_STRUCTURED_MAX_RETRIES"):
GatewaySettings.from_env("LLM", env=_env(PGW_STRUCTURED_MAX_RETRIES="-1"))
def test_cache_requires_namespace_and_ttl(self):
env = _env(PGW_CACHE_BACKEND="memory")
with pytest.raises(ValueError, match="NAMESPACE"):
@@ -258,6 +276,11 @@ class TestAssemblyGuards:
with pytest.raises(ValueError, match="TELEMETRY_BACKEND"):
GatewaySettings.from_env("LLM", env=_env(PGW_TELEMETRY_BACKEND="mysql"))
def test_cache_backend_whitelist(self):
"""对称于上一条: env 层的域检查保留是为了报错能点出键名,得有测试守着。"""
with pytest.raises(ValueError, match="CACHE_BACKEND"):
GatewaySettings.from_env("LLM", env=_env(PGW_CACHE_BACKEND="rediss"))
def test_pricing_path_optional(self):
assert GatewaySettings.from_env("LLM", env=_env()).pricing_path is None
s = GatewaySettings.from_env("LLM", env=_env(PGW_PRICING_PATH="conf/prices.json"))
@@ -413,3 +436,187 @@ class TestCrossFieldInvariants:
base = self._base()
with pytest.raises(ValueError, match="lease_ttl_s"):
OcrSettings(gateway=dataclasses.replace(base, lease_ttl_s=1.0))
# —— 第二轮(设计 2026-07-30): 后端枚举合法域 ——
@pytest.mark.parametrize(
("field", "bad_value"),
[
("limiter_backend", "rediss"),
("breaker_backend", "sqlite"),
("cache_backend", "postgres"),
("telemetry_backend", "redis"),
("selector", "random"),
("quota_full", "block"),
],
)
def test_enum_field_rejects_value_outside_domain(self, field, bad_value):
"""域外取值此前只有 from_env 拦得住,直接构造会落进 _build_* 的 else 分支。"""
base = self._base()
with pytest.raises(ValueError, match=field):
dataclasses.replace(base, **{field: bad_value})
# —— 条件必填: 取 redis 的后端必须有 redis_url ——
@pytest.mark.parametrize("field", ["limiter_backend", "breaker_backend"])
def test_redis_backend_requires_redis_url(self, field):
"""client.py 的 assert settings.redis_url is not None 依赖的正是这条。"""
base = self._base() # redis_url=None
with pytest.raises(ValueError, match="redis_url"):
dataclasses.replace(base, **{field: "redis"})
def test_redis_cache_requires_redis_url(self):
base = self._base()
with pytest.raises(ValueError, match="redis_url"):
dataclasses.replace(base, cache_backend="redis", cache_namespace="ns", cache_ttl_s=60)
# —— 条件必填: 启用缓存必须有命名空间与正 TTL ——
def test_cache_requires_namespace(self):
"""缺命名空间即失去租户隔离,踩"无缓存毒化"铁律。"""
base = self._base()
with pytest.raises(ValueError, match="cache_namespace"):
dataclasses.replace(base, cache_backend="memory", cache_ttl_s=60)
def test_cache_ttl_must_be_positive(self):
"""from_env 明令禁止的"永不过期"不能从另一条路进来。"""
base = self._base()
with pytest.raises(ValueError, match="cache_ttl_s"):
dataclasses.replace(base, cache_backend="memory", cache_namespace="ns", cache_ttl_s=0)
# —— 条件必填: 遥测后端各自的落点 ——
def test_sqlite_telemetry_requires_path(self):
base = self._base()
with pytest.raises(ValueError, match="telemetry_sqlite_path"):
dataclasses.replace(base, telemetry_backend="sqlite")
def test_postgres_telemetry_requires_dsn(self):
base = self._base()
with pytest.raises(ValueError, match="telemetry_pg_dsn"):
dataclasses.replace(base, telemetry_backend="postgres")
# —— 标量域 ——
def test_negative_structured_retries_rejected(self):
base = self._base()
with pytest.raises(ValueError, match="structured_max_retries"):
dataclasses.replace(base, structured_max_retries=-1)
def test_blank_scope_rejected(self):
"""空 scope 会污染遥测与缓存命名空间。"""
base = self._base()
with pytest.raises(ValueError, match="scope"):
dataclasses.replace(base, scope=" ")
# —— 合法组合仍可构造(收紧的是错的那些)——
def test_full_redis_stack_constructible(self):
base = self._base()
settings = dataclasses.replace(
base,
limiter_backend="redis",
breaker_backend="redis",
cache_backend="redis",
cache_namespace="ns",
cache_ttl_s=60,
redis_url="redis://127.0.0.1:6379/3",
)
assert settings.cache_ttl_s == 60 and settings.redis_url is not None
# —— Postgres DSN: 剥 SQLAlchemy 驱动后缀并出声(设计 §5 方案 C)——
def test_sqlalchemy_dsn_suffix_stripped_with_warning(self):
"""asyncpg 不认 `+driver`;库替调用方剥掉,但不静默——日志里看得见。"""
base = self._base()
with _captured_warnings() as warnings:
settings = dataclasses.replace(
base,
telemetry_backend="postgres",
telemetry_pg_dsn="postgresql+asyncpg://u:s3cret@h/db",
)
assert settings.telemetry_pg_dsn == "postgresql://u:s3cret@h/db"
assert any("asyncpg" in m for m in warnings)
def test_dsn_warning_does_not_leak_credentials(self):
"""DSN 带密码,日志只能出现 scheme 段(P5: 敏感信息只走 .env)。"""
base = self._base()
with _captured_warnings() as warnings:
dataclasses.replace(
base,
telemetry_backend="postgres",
telemetry_pg_dsn="postgresql+asyncpg://u:s3cret@h/db",
)
assert warnings and not any("s3cret" in m or "@h/db" in m for m in warnings)
def test_env_path_strips_dsn_without_warning(self):
"""env 路已在 _load_pg_dsn 剥过,不该给三项目的历史 DSN 写法刷噪音。"""
with _captured_warnings() as warnings:
settings = GatewaySettings.from_env(
"LLM",
env=_env(
PGW_TELEMETRY_BACKEND="postgres",
PGW_TELEMETRY_PG_DSN="postgresql+asyncpg://u@h/db",
),
)
assert settings.telemetry_pg_dsn == "postgresql://u@h/db"
assert not warnings
# —— 构造期规范化: env 路一直在做的,构造路也要做(否则两条路产出不同的值)——
@pytest.mark.parametrize("raw", ["LLM", " llm ", " LLM "])
def test_scope_normalized_on_direct_construction(self, raw):
"""scope 直接进 Redis key(pgw:limit:{scope}:…)。
大小写不一致会让同一逻辑 scope 的限流/熔断状态分裂到两套命名空间——
两边各记各的配额与熔断状态,分布式治理静默失效且不报错。
"""
base = self._base()
assert dataclasses.replace(base, scope=raw).scope == "llm"
def test_blank_redis_url_normalized_to_none(self):
"""空串此前只有 env 路归 None,构造路留着它骗过 `is None` 判断。"""
base = self._base()
assert dataclasses.replace(base, redis_url="").redis_url is None
def test_blank_redis_url_still_blocks_redis_backend(self):
"""归 None 后必须落进条件必填,而不是放行到 redis 库去抛连接串天书。"""
base = self._base()
with pytest.raises(ValueError, match="redis_url"):
dataclasses.replace(base, limiter_backend="redis", redis_url="")
def test_blank_pricing_path_normalized_to_none(self):
base = self._base()
assert dataclasses.replace(base, pricing_path="").pricing_path is None
# —— EmbeddingSettings 自身的字段域(此前只有 from_env 校验)——
@pytest.mark.parametrize("bad", [0, -3])
def test_embedding_settings_rejects_non_positive_batch_size(self, bad):
base = self._base()
with pytest.raises(ValueError, match="batch_size"):
EmbeddingSettings(gateway=base, batch_size=bad)
def test_embedding_settings_rejects_non_positive_expected_dim(self):
base = self._base()
with pytest.raises(ValueError, match="expected_dim"):
EmbeddingSettings(gateway=base, batch_size=8, expected_dim=0)
def test_embedding_settings_accepts_valid_values(self):
base = self._base()
settings = EmbeddingSettings(gateway=base, batch_size=8, expected_dim=1024)
assert settings.batch_size == 8 and settings.expected_dim == 1024
# —— 回归护栏: client.py 的 assert 前提确实被保证了 ——
def test_factory_accepts_valid_redis_stack(self):
"""补齐校验后,client.py:262/282/302 的 assert 退回成纯内部不变量声明。"""
base = self._base()
settings = dataclasses.replace(
base,
limiter_backend="redis",
breaker_backend="redis",
redis_url="redis://127.0.0.1:6379/3",
)
client = GatewayClient.from_settings(settings)
assert client is not None
+5
View File
@@ -358,6 +358,11 @@ class TestEmbeddingSettings:
s = EmbeddingSettings.from_env("EMBED", env=env)
assert s.normalize is True and s.expected_dim == 768
def test_expected_dim_must_be_positive(self):
"""env 层的检查保留是为了报错能点出键名(构造期那道点的是字段名)。"""
with pytest.raises(ValueError, match="EXPECTED_DIM"):
EmbeddingSettings.from_env("EMBED", env={**self._ENV, "EMBED__EXPECTED_DIM": "0"})
def test_from_settings_assembles_client(self):
s = EmbeddingSettings.from_env("EMBED", env=self._ENV)
client = EmbeddingClient.from_settings(s)