feat: make telemetry degradation a first-class state
Telemetry degradation used to be a single warning and a private boolean. In a long-running process that is indistinguishable from telemetry working: issue #15 was only found by hand-reconciling milestone log lines against llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite side was worse — once init failed, every write returned without even a log line. Degradation now has one shared owner. TelemetryStatusTracker holds the state machine (enter/recover/drop/should-retry), announces entry and recovery once each, and repeats the drop count under a row-and-time double threshold so a degraded backend neither floods the log nor goes quiet. Both recorders hold one; both count the rows they drop. For programmatic consumers, TelemetryStatus is a frozen snapshot exposed as telemetry_status on all three clients, resolved through a single isinstance check. It is a separate optional port rather than a member of TelemetryRecorder: that protocol is @runtime_checkable, so adding an attribute would make every implementation that only defines record_llm_call stop satisfying it — downstream isinstance assertions would break on upgrade. The existing assertion in test_ports.py is what keeps that decision honest. Failure criteria are deliberately untouched here: Postgres still treats a pool failure as permanent, only now visibly. `_failed` and the tracker therefore both carry the verdict for the span of this one change; the cooldown rework collapses them into the tracker alone.
This commit is contained in:
@@ -835,3 +835,40 @@ class TestRedisCacheOwnership:
|
||||
await cache.aclose()
|
||||
await cache.aclose() # 幂等: 不重复关
|
||||
assert built.closed == 1
|
||||
|
||||
|
||||
class TestTelemetryStatusExposure:
|
||||
"""降级状态的只读出口: 一处 isinstance 判定,三个 client 各钉一次(设计 §3.3)。"""
|
||||
|
||||
def _recorder(self, tmp_path):
|
||||
from polygateway.telemetry.sqlite import SQLiteRecorder
|
||||
|
||||
return SQLiteRecorder(tmp_path / "telemetry.db", auto_migrate=True)
|
||||
|
||||
def _assert_snapshot(self, status):
|
||||
from polygateway.types import TelemetryStatus
|
||||
|
||||
assert isinstance(status, TelemetryStatus)
|
||||
assert status.degraded is False
|
||||
|
||||
def test_gateway_client_without_telemetry_reports_none(self):
|
||||
assert _client().telemetry_status is None
|
||||
|
||||
def test_gateway_client_with_foreign_recorder_reports_none(self):
|
||||
"""注入的第三方 recorder 不提供状态 → None,绝不得抛 AttributeError。"""
|
||||
assert _client(telemetry=_Closable()).telemetry_status is None
|
||||
|
||||
def test_gateway_client_with_builtin_recorder_reports_snapshot(self, tmp_path):
|
||||
self._assert_snapshot(_client(telemetry=self._recorder(tmp_path)).telemetry_status)
|
||||
|
||||
def test_embedding_client_exposes_the_same_outlet(self, tmp_path):
|
||||
assert _embedding_client().telemetry_status is None
|
||||
assert _embedding_client(telemetry=_Closable()).telemetry_status is None
|
||||
self._assert_snapshot(
|
||||
_embedding_client(telemetry=self._recorder(tmp_path)).telemetry_status
|
||||
)
|
||||
|
||||
def test_ocr_client_exposes_the_same_outlet(self, tmp_path):
|
||||
assert _ocr_client().telemetry_status is None
|
||||
assert _ocr_client(telemetry=_Closable()).telemetry_status is None
|
||||
self._assert_snapshot(_ocr_client(telemetry=self._recorder(tmp_path)).telemetry_status)
|
||||
|
||||
Reference in New Issue
Block a user