feat: make telemetry degradation a first-class state

Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.

Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.

For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.

Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
This commit is contained in:
2026-08-24 08:57:23 -04:00
parent e69ca4c82c
commit f958138e83
11 changed files with 542 additions and 5 deletions
+24
View File
@@ -24,6 +24,7 @@ from polygateway.middleware.cache import CacheMW
from polygateway.middleware.retry import RetryMW
from polygateway.middleware.structured import StructuredMW
from polygateway.middleware.telemetry import TelemetryEmitter, TelemetryMW
from polygateway.ports import TelemetryStatusProvider
from polygateway.pricing import PricingTable
from polygateway.providers import get_capability, get_provider, resolve_thinking
from polygateway.sources import (
@@ -37,6 +38,7 @@ from polygateway.transports.openai_compat import OpenAICompatTransport
from polygateway.types import (
ChatRequest,
LLMResponse,
TelemetryStatus,
validate_caller_dimensions,
validate_request_overlay,
)
@@ -135,6 +137,19 @@ async def _aclose_component(component: object | None) -> None:
close()
def _telemetry_status_of(telemetry: TelemetryRecorder | None) -> TelemetryStatus | None:
"""三个 client 共用的状态取值点: 不提供状态的 recorder 一律返回 None。
判定写成 `isinstance(可选端口)` 而不是裸 `getattr`: 两者运行时都是结构检查
(`@runtime_checkable` 按属性存在性判定),差别在**契约有没有名字**——端口是
写进 `ports.py` 的公开承诺,下游可以照着实现;散落的 `getattr` 不是,而
`aclose` 当年正是被复制成三份鸭子类型探测才漂移出越权关闭(设计 §3.3/§3.4)。
"""
if isinstance(telemetry, TelemetryStatusProvider):
return telemetry.telemetry_status
return None
def _mark_owned_components(
client: Any,
*,
@@ -252,6 +267,15 @@ class GatewayClient:
self._owns_breaker = False
self._closed = False
@property
def telemetry_status(self) -> TelemetryStatus | None:
"""遥测后端的可写状态;无遥测或注入的 recorder 不提供状态时为 None。
判定收敛在 `_telemetry_status_of` 一处(不是三处各自探测): 三个 client
的 `aclose` 曾各持一份逐字复制,漂移的结果就是越权关闭(设计 §3.3/§3.4)。
"""
return _telemetry_status_of(self._telemetry)
async def chat(
self,
messages: list[dict[str, Any]],