feat: make telemetry degradation a first-class state
Telemetry degradation used to be a single warning and a private boolean. In a long-running process that is indistinguishable from telemetry working: issue #15 was only found by hand-reconciling milestone log lines against llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite side was worse — once init failed, every write returned without even a log line. Degradation now has one shared owner. TelemetryStatusTracker holds the state machine (enter/recover/drop/should-retry), announces entry and recovery once each, and repeats the drop count under a row-and-time double threshold so a degraded backend neither floods the log nor goes quiet. Both recorders hold one; both count the rows they drop. For programmatic consumers, TelemetryStatus is a frozen snapshot exposed as telemetry_status on all three clients, resolved through a single isinstance check. It is a separate optional port rather than a member of TelemetryRecorder: that protocol is @runtime_checkable, so adding an attribute would make every implementation that only defines record_llm_call stop satisfying it — downstream isinstance assertions would break on upgrade. The existing assertion in test_ports.py is what keeps that decision honest. Failure criteria are deliberately untouched here: Postgres still treats a pool failure as permanent, only now visibly. `_failed` and the tracker therefore both carry the verdict for the span of this one change; the cooldown rework collapses them into the tracker alone.
This commit is contained in:
@@ -258,6 +258,31 @@ class SourceStats:
|
||||
tpm_used: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TelemetryStatus:
|
||||
"""遥测后端的可写状态快照;degraded 期间下游可据此对账(issue #15)。
|
||||
|
||||
不叫 `health`: 库内 `health` 一律指**源的健康度**(`OcrTransport.check_health`
|
||||
探活、`SourceSelector.health` 成功率 EWMA),而这里描述的是"这个 recorder
|
||||
现在能不能写、为什么不能、丢了多少",是状态不是评分(设计 §3.3)。
|
||||
|
||||
时长一律给**相对秒数**而非绝对时间戳: 库内的时钟是 monotonic,把它的读数
|
||||
交给下游会与 wall clock 混淆成两个不可比的时间轴。
|
||||
"""
|
||||
|
||||
degraded: bool
|
||||
fatal: bool
|
||||
"""True = 本进程内不可恢复(仅 DSN 不可解析一类配置级失败),需改配置并重启。"""
|
||||
reason: str | None
|
||||
"""降级原因;未降级为 None。"""
|
||||
degraded_for_s: float | None
|
||||
"""已降级时长;未降级为 None。"""
|
||||
dropped_rows: int
|
||||
"""累计丢弃行数;**进程生命周期内单调不减**——恢复不等于没丢过。"""
|
||||
retry_after_s: float | None
|
||||
"""距下次重新准备的秒数;fatal 或未降级为 None,冷却已到期为 0.0。"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TransportResult:
|
||||
"""transport 单次原始调用的产物;治理字段由 RetryMW 补齐为 LLMResponse。"""
|
||||
|
||||
Reference in New Issue
Block a user