feat: make telemetry degradation a first-class state
Telemetry degradation used to be a single warning and a private boolean. In a long-running process that is indistinguishable from telemetry working: issue #15 was only found by hand-reconciling milestone log lines against llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite side was worse — once init failed, every write returned without even a log line. Degradation now has one shared owner. TelemetryStatusTracker holds the state machine (enter/recover/drop/should-retry), announces entry and recovery once each, and repeats the drop count under a row-and-time double threshold so a degraded backend neither floods the log nor goes quiet. Both recorders hold one; both count the rows they drop. For programmatic consumers, TelemetryStatus is a frozen snapshot exposed as telemetry_status on all three clients, resolved through a single isinstance check. It is a separate optional port rather than a member of TelemetryRecorder: that protocol is @runtime_checkable, so adding an attribute would make every implementation that only defines record_llm_call stop satisfying it — downstream isinstance assertions would break on upgrade. The existing assertion in test_ports.py is what keeps that decision honest. Failure criteria are deliberately untouched here: Postgres still treats a pool failure as permanent, only now visibly. `_failed` and the tracker therefore both carry the verdict for the span of this one change; the cooldown rework collapses them into the tracker alone.
This commit is contained in:
@@ -19,6 +19,7 @@ import asyncio
|
||||
import sqlite3
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
@@ -29,6 +30,10 @@ from polygateway.telemetry.schema import (
|
||||
insert_sql,
|
||||
missing_columns_warning,
|
||||
)
|
||||
from polygateway.telemetry.status import TelemetryStatusTracker
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from polygateway.types import TelemetryStatus
|
||||
|
||||
|
||||
class SQLiteRecorder:
|
||||
@@ -45,6 +50,9 @@ class SQLiteRecorder:
|
||||
config 一处,不与本类签名漂移(设计 D-c)。
|
||||
"""
|
||||
self._auto_migrate = auto_migrate
|
||||
# SQLite 侧本次只做可见性: 它的失败模式(目录不可写、文件损坏)在装配期
|
||||
# 就暴露给下游,不是"跑到一半悄悄断",故降级恒为 fatal,不做冷却重连
|
||||
self._status = TelemetryStatusTracker(backend="sqlite")
|
||||
self._lock = threading.Lock()
|
||||
self._conn: sqlite3.Connection | None = None
|
||||
# 先按全量列定型: 连接失败/探测失败时保守沿用全量(今天的行为)
|
||||
@@ -60,9 +68,14 @@ class SQLiteRecorder:
|
||||
conn.commit()
|
||||
self._conn = conn
|
||||
except (OSError, sqlite3.Error) as exc:
|
||||
logger.warning("SQLite 遥测初始化失败,后续记录降级为 no-op: {}", exc)
|
||||
self._status.enter_degraded(f"初始化失败: {exc}", fatal=True, cooldown_s=None)
|
||||
self._prepare_columns()
|
||||
|
||||
@property
|
||||
def telemetry_status(self) -> TelemetryStatus:
|
||||
"""当前可写状态快照(ports.TelemetryStatusProvider)。"""
|
||||
return self._status.snapshot()
|
||||
|
||||
def _prepare_columns(self) -> None:
|
||||
"""探测现有列后定型写入: auto 档补齐缺列,manual 档改为裁剪写入(issue #13)。
|
||||
|
||||
@@ -136,6 +149,9 @@ class SQLiteRecorder:
|
||||
占位符同序——两者必须一起改,分开改就是把值写进错位的列。
|
||||
"""
|
||||
if self._conn is None:
|
||||
# 改前这里是**裸 return**: 初始化失败后每一行都无声消失,长跑进程里
|
||||
# 与"遥测正常"外观上完全一致(设计 §1.4 的直接钉子)
|
||||
self._status.record_drop("遥测已降级")
|
||||
return
|
||||
row = tuple(fields[col] for col in self._columns)
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user