fix: close the failure modes review found in the new code

Three of them were the same shape as the bug this branch exists to fix:
something goes wrong, the library swallows it, and the caller is left
with a number that means the opposite of what happened.

The throttle key had no source in it. Five sources on one model is the
normal case here, so the first one to break would warn once and silence
the other four for the life of the process, and the message never said
which gateway to look at.

An unknown verdict in a cached entry threw away the whole response. The
rehydrator tolerates unknown fields but not unknown values of a known
field, so two library versions sharing a Redis would each invalidate
the other's entries: halved hit rate, and the only log line says the
cache rebuild failed. A purely observational field should not be able
to void a response whose content is intact.

Normalising for telemetry now degrades instead of raising, both for a
bare string and for a value outside the domain. Either one used to
reach the same except and cost the whole row, which is exactly how
1.3.0 lost nineteen calls without anyone noticing.
This commit is contained in:
2026-08-26 02:37:24 -04:00
parent c0b544d233
commit 1307a02b92
9 changed files with 270 additions and 30 deletions
+52
View File
@@ -9,6 +9,7 @@ import subprocess
from pathlib import Path
import pytest
from loguru import logger
from polygateway.backends.memory.breaker import InMemoryGate
from polygateway.backends.memory.limiter import InMemoryLimiter
@@ -1169,6 +1170,57 @@ class TestEmitterThinkingObservation:
assert value == "observed"
assert type(value) is str # 不是 ThinkingObservation: 子类实例不得下沉到 recorder
async def test_a_bare_string_verdict_still_lands(self):
"""下游填裸 str 时**整行**不得丢失(遥测必录)。
`LLMResponse` 是无运行时校验的 frozen dataclass,写
`LLMResponse(..., thinking_observation="observed")` 完全自然且 `==` 比较
照常成立;若 emitter 直接取 `.value`,这里会抛 `AttributeError` 并被
`_record` 的 `except Exception` 吞成一条泛化 warning——丢的不是这一列,
是整行,正是 1.3.0 那次"19 次调用一行未落"的同款形态。
"""
rec = _MemoryRecorder()
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
request=_REQ,
source=_source(),
call_id="c",
latency_ms=1,
response=_resp(thinking_observation="observed"),
error=None,
)
assert len(rec.rows) == 1, "整行被吞了"
value = rec.rows[0]["thinking_observation"]
assert value == "observed"
assert type(value) is str
async def test_an_out_of_domain_verdict_degrades_but_keeps_the_row(self):
"""域外取值挡在落库前,但**降级不丢行**: 列的取值域由库守,代价不是整行。
直接 `ThinkingObservation(x).value` 会在这里抛 `ValueError`,同样被
`_record` 的 `except Exception` 吞成丢整行——那只修好了裸 str 一半,
口误值(大小写不符、拼错)对测试替身同样自然。故降级为 `unknown`
(对库而言本次确实判不出来)并单独告警,与缓存回放的方向选择一致。
"""
rec = _MemoryRecorder()
messages: list[str] = []
sink_id = logger.add(messages.append, level="WARNING")
try:
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
request=_REQ,
source=_source(),
call_id="c",
latency_ms=1,
response=_resp(thinking_observation="OBSERVED"), # 大小写不符即域外
error=None,
)
finally:
logger.remove(sink_id)
assert len(rec.rows) == 1, "整行被吞了"
assert rec.rows[0]["thinking_observation"] == "unknown"
hits = [m for m in messages if "OBSERVED" in m]
assert len(hits) == 1, f"域外取值必须单独告警: {messages}"
assert [m for m in messages if "遥测记录失败" in m] == []
async def test_cache_hit_replays_the_recorded_verdict(self):
"""缓存命中回放历史那次的裁定: 与 model/prompt_tokens 同一口径。"""
rec = _MemoryRecorder()