feat: make telemetry degradation a first-class state
Telemetry degradation used to be a single warning and a private boolean. In a long-running process that is indistinguishable from telemetry working: issue #15 was only found by hand-reconciling milestone log lines against llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite side was worse — once init failed, every write returned without even a log line. Degradation now has one shared owner. TelemetryStatusTracker holds the state machine (enter/recover/drop/should-retry), announces entry and recovery once each, and repeats the drop count under a row-and-time double threshold so a degraded backend neither floods the log nor goes quiet. Both recorders hold one; both count the rows they drop. For programmatic consumers, TelemetryStatus is a frozen snapshot exposed as telemetry_status on all three clients, resolved through a single isinstance check. It is a separate optional port rather than a member of TelemetryRecorder: that protocol is @runtime_checkable, so adding an attribute would make every implementation that only defines record_llm_call stop satisfying it — downstream isinstance assertions would break on upgrade. The existing assertion in test_ports.py is what keeps that decision honest. Failure criteria are deliberately untouched here: Postgres still treats a pool failure as permanent, only now visibly. `_failed` and the tracker therefore both carry the verdict for the span of this one change; the cooldown rework collapses them into the tracker alone.
This commit is contained in:
@@ -24,6 +24,7 @@ from polygateway.middleware.cache import CacheMW
|
||||
from polygateway.middleware.retry import RetryMW
|
||||
from polygateway.middleware.structured import StructuredMW
|
||||
from polygateway.middleware.telemetry import TelemetryEmitter, TelemetryMW
|
||||
from polygateway.ports import TelemetryStatusProvider
|
||||
from polygateway.pricing import PricingTable
|
||||
from polygateway.providers import get_capability, get_provider, resolve_thinking
|
||||
from polygateway.sources import (
|
||||
@@ -37,6 +38,7 @@ from polygateway.transports.openai_compat import OpenAICompatTransport
|
||||
from polygateway.types import (
|
||||
ChatRequest,
|
||||
LLMResponse,
|
||||
TelemetryStatus,
|
||||
validate_caller_dimensions,
|
||||
validate_request_overlay,
|
||||
)
|
||||
@@ -135,6 +137,19 @@ async def _aclose_component(component: object | None) -> None:
|
||||
close()
|
||||
|
||||
|
||||
def _telemetry_status_of(telemetry: TelemetryRecorder | None) -> TelemetryStatus | None:
|
||||
"""三个 client 共用的状态取值点: 不提供状态的 recorder 一律返回 None。
|
||||
|
||||
判定写成 `isinstance(可选端口)` 而不是裸 `getattr`: 两者运行时都是结构检查
|
||||
(`@runtime_checkable` 按属性存在性判定),差别在**契约有没有名字**——端口是
|
||||
写进 `ports.py` 的公开承诺,下游可以照着实现;散落的 `getattr` 不是,而
|
||||
`aclose` 当年正是被复制成三份鸭子类型探测才漂移出越权关闭(设计 §3.3/§3.4)。
|
||||
"""
|
||||
if isinstance(telemetry, TelemetryStatusProvider):
|
||||
return telemetry.telemetry_status
|
||||
return None
|
||||
|
||||
|
||||
def _mark_owned_components(
|
||||
client: Any,
|
||||
*,
|
||||
@@ -252,6 +267,15 @@ class GatewayClient:
|
||||
self._owns_breaker = False
|
||||
self._closed = False
|
||||
|
||||
@property
|
||||
def telemetry_status(self) -> TelemetryStatus | None:
|
||||
"""遥测后端的可写状态;无遥测或注入的 recorder 不提供状态时为 None。
|
||||
|
||||
判定收敛在 `_telemetry_status_of` 一处(不是三处各自探测): 三个 client
|
||||
的 `aclose` 曾各持一份逐字复制,漂移的结果就是越权关闭(设计 §3.3/§3.4)。
|
||||
"""
|
||||
return _telemetry_status_of(self._telemetry)
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
|
||||
Reference in New Issue
Block a user