feat: make telemetry degradation a first-class state

Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.

Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.

For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.

Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
This commit is contained in:
2026-08-24 08:57:23 -04:00
parent e69ca4c82c
commit f958138e83
11 changed files with 542 additions and 5 deletions
+17 -1
View File
@@ -19,6 +19,7 @@ import asyncio
import sqlite3
import threading
from pathlib import Path
from typing import TYPE_CHECKING
from loguru import logger
@@ -29,6 +30,10 @@ from polygateway.telemetry.schema import (
insert_sql,
missing_columns_warning,
)
from polygateway.telemetry.status import TelemetryStatusTracker
if TYPE_CHECKING:
from polygateway.types import TelemetryStatus
class SQLiteRecorder:
@@ -45,6 +50,9 @@ class SQLiteRecorder:
config 一处,不与本类签名漂移(设计 D-c)。
"""
self._auto_migrate = auto_migrate
# SQLite 侧本次只做可见性: 它的失败模式(目录不可写、文件损坏)在装配期
# 就暴露给下游,不是"跑到一半悄悄断",故降级恒为 fatal,不做冷却重连
self._status = TelemetryStatusTracker(backend="sqlite")
self._lock = threading.Lock()
self._conn: sqlite3.Connection | None = None
# 先按全量列定型: 连接失败/探测失败时保守沿用全量(今天的行为)
@@ -60,9 +68,14 @@ class SQLiteRecorder:
conn.commit()
self._conn = conn
except (OSError, sqlite3.Error) as exc:
logger.warning("SQLite 遥测初始化失败,后续记录降级为 no-op: {}", exc)
self._status.enter_degraded(f"初始化失败: {exc}", fatal=True, cooldown_s=None)
self._prepare_columns()
@property
def telemetry_status(self) -> TelemetryStatus:
"""当前可写状态快照(ports.TelemetryStatusProvider)。"""
return self._status.snapshot()
def _prepare_columns(self) -> None:
"""探测现有列后定型写入: auto 档补齐缺列,manual 档改为裁剪写入(issue #13)。
@@ -136,6 +149,9 @@ class SQLiteRecorder:
占位符同序——两者必须一起改,分开改就是把值写进错位的列。
"""
if self._conn is None:
# 改前这里是**裸 return**: 初始化失败后每一行都无声消失,长跑进程里
# 与"遥测正常"外观上完全一致(设计 §1.4 的直接钉子)
self._status.record_drop("遥测已降级")
return
row = tuple(fields[col] for col in self._columns)
try: