feat: make telemetry degradation a first-class state

Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.

Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.

For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.

Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
This commit is contained in:
2026-08-24 08:57:23 -04:00
parent e69ca4c82c
commit f958138e83
11 changed files with 542 additions and 5 deletions
+24 -1
View File
@@ -16,9 +16,10 @@ from polygateway.ports import (
SourceSelector,
StructuredOutputStrategy,
TelemetryRecorder,
TelemetryStatusProvider,
Transport,
)
from polygateway.types import LLMResponse, SourceStats
from polygateway.types import LLMResponse, SourceStats, TelemetryStatus
def _resp() -> LLMResponse:
@@ -141,6 +142,28 @@ def test_protocols_are_runtime_checkable(impl, protocol):
assert isinstance(impl, protocol)
class _DummyStatusProvider(_DummyRecorder):
@property
def telemetry_status(self) -> TelemetryStatus:
return TelemetryStatus(
degraded=False,
fatal=False,
reason=None,
degraded_for_s=None,
dropped_rows=0,
retry_after_s=None,
)
def test_status_provider_is_a_separate_optional_port():
"""状态**不得**并进 TelemetryRecorder: 那会让只实现 record_llm_call 的对象
当场不再满足 @runtime_checkable 的结构检查(设计 §3.3,Codex 审查)。"""
assert isinstance(_DummyStatusProvider(), TelemetryStatusProvider)
assert isinstance(_DummyStatusProvider(), TelemetryRecorder)
assert not isinstance(_DummyRecorder(), TelemetryStatusProvider)
assert isinstance(_DummyRecorder(), TelemetryRecorder) # 这条断言是那条决策的执法点
def _decision(**overrides) -> GateDecision:
base = {
"source_name": "qwen_1",