feat: make telemetry degradation a first-class state

Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.

Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.

For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.

Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
This commit is contained in:
2026-08-24 08:57:23 -04:00
parent e69ca4c82c
commit f958138e83
11 changed files with 542 additions and 5 deletions
+17 -1
View File
@@ -28,10 +28,13 @@ from polygateway.telemetry.schema import (
insert_sql,
missing_columns_warning,
)
from polygateway.telemetry.status import TelemetryStatusTracker
if TYPE_CHECKING:
import asyncpg
from polygateway.types import TelemetryStatus
# 探测表是否存在;不需要任何权限,且与 INSERT 走同一套 search_path 解析
_TABLE_EXISTS = "SELECT to_regclass('llm_calls')"
@@ -72,8 +75,16 @@ class PostgresRecorder:
self._insert = insert_sql("postgres", COLUMNS)
self._schema_ready = False
self._failed = False # 结构性降级标志: 置位后所有写入短路
# 降级的可编程出口与节流日志;`_failed` 与它并存是 issue #15 的过渡态,
# 判据改造(冷却自愈)落地时状态收归 tracker 一处
self._status = TelemetryStatusTracker(backend="postgres")
self._init_lock = asyncio.Lock()
@property
def telemetry_status(self) -> TelemetryStatus:
"""当前可写状态快照(ports.TelemetryStatusProvider)。"""
return self._status.snapshot()
async def _ensure_ready(self) -> asyncpg.Pool | None:
"""lazy 建池+备表;判死只认「确定写不进去」(issue #9),其余失败都留活路。"""
if self._failed:
@@ -104,7 +115,7 @@ class PostgresRecorder:
# 池建不出来 = 确定写不进去;且每次调用重试都要内联吞掉 connect
# 超时,而遥测是业务路径上的 await —— 此处必须永久降级
self._failed = True
logger.warning("Postgres 遥测建池失败,后续记录降级为 no-op: {}", exc)
self._status.enter_degraded(f"建池失败: {exc}", fatal=True, cooldown_s=None)
return None
return self._pool
@@ -122,6 +133,9 @@ class PostgresRecorder:
return None
if columns is None:
self._failed = True
self._status.enter_degraded(
"表 llm_calls 不存在且建不出来(记录无处可落)", fatal=True, cooldown_s=None
)
return None
# 写入列、语句与就绪标志必须**一起**生效: `_ensure_ready` 只看 `_schema_ready`
# 就绕开 `_init_lock` 直接返回池,先置就绪会开出"已就绪但语句还是旧的"的窗口
@@ -232,6 +246,8 @@ class PostgresRecorder:
"""
pool = await self._ensure_ready()
if pool is None:
# 降级期间静默 return 就是 issue #15 的破口: 丢行必须计数且节流出声
self._status.record_drop("遥测已降级")
return
row = tuple(fields[col] for col in self._columns)
try: