feat: make telemetry degradation a first-class state
Telemetry degradation used to be a single warning and a private boolean. In a long-running process that is indistinguishable from telemetry working: issue #15 was only found by hand-reconciling milestone log lines against llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite side was worse — once init failed, every write returned without even a log line. Degradation now has one shared owner. TelemetryStatusTracker holds the state machine (enter/recover/drop/should-retry), announces entry and recovery once each, and repeats the drop count under a row-and-time double threshold so a degraded backend neither floods the log nor goes quiet. Both recorders hold one; both count the rows they drop. For programmatic consumers, TelemetryStatus is a frozen snapshot exposed as telemetry_status on all three clients, resolved through a single isinstance check. It is a separate optional port rather than a member of TelemetryRecorder: that protocol is @runtime_checkable, so adding an attribute would make every implementation that only defines record_llm_call stop satisfying it — downstream isinstance assertions would break on upgrade. The existing assertion in test_ports.py is what keeps that decision honest. Failure criteria are deliberately untouched here: Postgres still treats a pool failure as permanent, only now visibly. `_failed` and the tracker therefore both carry the verdict for the span of this one change; the cooldown rework collapses them into the tracker alone.
This commit is contained in:
@@ -28,10 +28,13 @@ from polygateway.telemetry.schema import (
|
||||
insert_sql,
|
||||
missing_columns_warning,
|
||||
)
|
||||
from polygateway.telemetry.status import TelemetryStatusTracker
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import asyncpg
|
||||
|
||||
from polygateway.types import TelemetryStatus
|
||||
|
||||
# 探测表是否存在;不需要任何权限,且与 INSERT 走同一套 search_path 解析
|
||||
_TABLE_EXISTS = "SELECT to_regclass('llm_calls')"
|
||||
|
||||
@@ -72,8 +75,16 @@ class PostgresRecorder:
|
||||
self._insert = insert_sql("postgres", COLUMNS)
|
||||
self._schema_ready = False
|
||||
self._failed = False # 结构性降级标志: 置位后所有写入短路
|
||||
# 降级的可编程出口与节流日志;`_failed` 与它并存是 issue #15 的过渡态,
|
||||
# 判据改造(冷却自愈)落地时状态收归 tracker 一处
|
||||
self._status = TelemetryStatusTracker(backend="postgres")
|
||||
self._init_lock = asyncio.Lock()
|
||||
|
||||
@property
|
||||
def telemetry_status(self) -> TelemetryStatus:
|
||||
"""当前可写状态快照(ports.TelemetryStatusProvider)。"""
|
||||
return self._status.snapshot()
|
||||
|
||||
async def _ensure_ready(self) -> asyncpg.Pool | None:
|
||||
"""lazy 建池+备表;判死只认「确定写不进去」(issue #9),其余失败都留活路。"""
|
||||
if self._failed:
|
||||
@@ -104,7 +115,7 @@ class PostgresRecorder:
|
||||
# 池建不出来 = 确定写不进去;且每次调用重试都要内联吞掉 connect
|
||||
# 超时,而遥测是业务路径上的 await —— 此处必须永久降级
|
||||
self._failed = True
|
||||
logger.warning("Postgres 遥测建池失败,后续记录降级为 no-op: {}", exc)
|
||||
self._status.enter_degraded(f"建池失败: {exc}", fatal=True, cooldown_s=None)
|
||||
return None
|
||||
return self._pool
|
||||
|
||||
@@ -122,6 +133,9 @@ class PostgresRecorder:
|
||||
return None
|
||||
if columns is None:
|
||||
self._failed = True
|
||||
self._status.enter_degraded(
|
||||
"表 llm_calls 不存在且建不出来(记录无处可落)", fatal=True, cooldown_s=None
|
||||
)
|
||||
return None
|
||||
# 写入列、语句与就绪标志必须**一起**生效: `_ensure_ready` 只看 `_schema_ready`
|
||||
# 就绕开 `_init_lock` 直接返回池,先置就绪会开出"已就绪但语句还是旧的"的窗口
|
||||
@@ -232,6 +246,8 @@ class PostgresRecorder:
|
||||
"""
|
||||
pool = await self._ensure_ready()
|
||||
if pool is None:
|
||||
# 降级期间静默 return 就是 issue #15 的破口: 丢行必须计数且节流出声
|
||||
self._status.record_drop("遥测已降级")
|
||||
return
|
||||
row = tuple(fields[col] for col in self._columns)
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user