feat: make telemetry degradation a first-class state

Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.

Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.

For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.

Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
This commit is contained in:
2026-08-24 08:57:23 -04:00
parent e69ca4c82c
commit f958138e83
11 changed files with 542 additions and 5 deletions
+11 -1
View File
@@ -25,7 +25,7 @@ from typing import TYPE_CHECKING, Any
from loguru import logger
from polygateway.client import _aclose_component
from polygateway.client import _aclose_component, _telemetry_status_of
from polygateway.config import EmbeddingSettings
from polygateway.errors import (
AllSourcesExhausted,
@@ -46,6 +46,7 @@ from polygateway.types import (
ChatRequest,
EmbeddingResponse,
LLMResponse,
TelemetryStatus,
strip_unsupported_extra_body,
validate_caller_dimensions,
)
@@ -454,6 +455,15 @@ class EmbeddingClient:
known = [c for c in costs if c is not None]
return sum(known) if known else None
@property
def telemetry_status(self) -> TelemetryStatus | None:
"""遥测后端的可写状态;无遥测或注入的 recorder 不提供状态时为 None。
判定收敛在 `_telemetry_status_of` 一处(不是三处各自探测): 三个 client
的 `aclose` 曾各持一份逐字复制,漂移的结果就是越权关闭(设计 §3.3/§3.4)。
"""
return _telemetry_status_of(self._telemetry)
async def aclose(self) -> None:
"""幂等释放**自建**资源(与 GatewayClient 对称);注入的组件一律不碰。"""
if self._closed: