fix: judge telemetry failures by nature, not by step

The pool exhaustion in issue #15 was fatal only because min_size=10 forced
a transient error to surface at pool creation, and that step was hardcoded
to permanent death. Step is the wrong axis: it conflates "the DSN cannot
be parsed" with "someone else holds all the connections right now".

Failures are now classified by two rules. Fatal means the cause lies
entirely inside this process and cannot change, which only the
construction-time DSN satisfies. Everything else splits on whether the
failure has anything to do with this row's data: row-level failures drop
one row and keep trying, environment-level failures cool down for 60s and
then get exactly one retry, so a restarted database or a DBA creating the
table heals on its own.

42703 (missing column) is the single named exception and stays row-level
even though every row fails alike: issue #13 promised that the manual mode
trims the INSERT and exposes drift per row, and that promise outranks the
rule. Any future exception owes the same argument.

The _failed boolean is gone; the tracker is the only degradation state,
because two copies of the same fact drift apart. Closing stays outside
that state: it is the caller's own decision, not an anomaly to recover
from, so the snapshot reports it through dropped_rows and the drop reason
instead of raising the degraded flag on every clean shutdown.
This commit is contained in:
2026-08-24 10:18:53 -04:00
parent bc071c6f41
commit eef2fdc5df
3 changed files with 374 additions and 58 deletions
+7 -7
View File
@@ -413,13 +413,13 @@ class TestLeastPrivilegeDeployment:
await conn.close()
async def test_records_land_without_schema_create_privilege(self, least_privilege_dsn):
"""修复前: 建表被拒 → _failed → 整个进程一条不落(下游 150 次调用全丢)。"""
"""修复前: 建表被拒 → 整体判死 → 整个进程一条不落(下游 150 次调用全丢)。"""
low_dsn, schema = least_privilege_dsn
recorder = _recorder(low_dsn, auto_migrate=True)
try:
await _record_minimal(recorder, call_id=_cid("lp1"))
await _record_minimal(recorder, call_id=_cid("lp2"), cost=1.5)
assert recorder._failed is False # 判死开关不得被建表权限触发
assert recorder.telemetry_status.degraded is False # 建表权限不得触发降级
rows = await _fetch(
low_dsn,
"SELECT call_id, cost FROM llm_calls WHERE call_id LIKE $1 ORDER BY call_id",
@@ -665,15 +665,15 @@ class TestCallerDimensionsAcceptance:
async def test_backfill_failure_degrades_per_row_not_wholesale(
self, least_privilege_pre_tenant_dsn, captured_warnings
):
"""补列失败的降级方向: 记 warning、不置 `_failed`、后续 INSERT 仍照发。
"""补列失败的降级方向: 记 warning、不整体降级、后续 INSERT 仍照发。
置 `_failed` 会让整个进程从此一条遥测都不写(比逐行丢弃严重得多),
且一旦 DBA 补上列也不会自愈——必须等重启
整体降级会让整个进程停写(比逐行丢弃严重得多),而缺列(SQLSTATE 42703)
是判据的唯一具名例外: 必须逐行暴露,好让下游看见 schema 漂移(issue #13)
"""
recorder = _recorder(least_privilege_pre_tenant_dsn, auto_migrate=True)
try:
await _record_minimal(recorder, call_id=_cid("lpp1")) # 不得抛
assert recorder._failed is False
assert recorder.telemetry_status.degraded is False
assert any("补列失败" in m for m in captured_warnings)
# 缺列的表上 INSERT 必然失败;逐行 warning 正是"INSERT 照发了"的证据
assert any("写入失败" in m for m in captured_warnings)
@@ -867,7 +867,7 @@ class TestManualSchemaModeAcceptance:
assert [m for m in captured_warnings if "补列失败" in m] == []
assert [m for m in captured_warnings if "写入失败" in m] == []
assert recorder._failed is False
assert recorder.telemetry_status.degraded is False
notices = [m for m in captured_warnings if "auto_migrate=False" in m]
assert len(notices) == 1 # 准备期一次,第二行不再重复
assert "以下维度不会被记录: tenant_id, meta" in notices[0]