diff --git a/README.md b/README.md index 217ea89..8714bbe 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ | 响应缓存 | Redis/内存;key 含 model + messages 摘要 + namespace(缓存隔离单位)+ salt + 采样参数,多模态 content 先摘要再 hash(防毒化);可 per-call 绕过(科研重采样) | | 流式看门狗 | TTFT / inter-token / 总超时三层活性;thinking token 刷活性不计结果;截断流(缺 `[DONE]`)判瞬时不入缓存 | | 推理可观测性 | "这次到底推理没推理"由多信号裁定(推理正文压倒 usage 明细),三态落在 `LLMResponse.thinking_observation`:`observed` / `absent` / `unknown`——**`unknown` 是"本次判不出",不是"没推理"**;请求方向与实测观测矛盾时按 `(模型, 方向)` 各告警一次(能力表过期、开启未生效、注入了却观测不到);裁定结果随遥测落库 | -| 遥测与成本 | 每次调用(含缓存命中与失败)必录 25 字段;SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 | +| 遥测与成本 | 每次调用(含缓存命中与失败)必录 26 字段;SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 | | 遥测的资源与降级 | Postgres 池**闲时占 0 条连接**、忙时上限可配(`PGW_TELEMETRY_PG_POOL_MAX`,缺省 4),每次写入有硬预算(`PGW_TELEMETRY_PG_WRITE_TIMEOUT_S`,缺省 5s);后端不可用是**可恢复的降级**(冷却 60s 后自动重试,DBA 建完表/放开权限即自愈),永久失能只留给 DSN 本身写错;降级状态可编程查询——`client.telemetry_status` 给出 `degraded`/`fatal`/`reason`/`dropped_rows` 等只读快照,不必再靠人工对账。**对账要同时看 `degraded` 与 `dropped_rows`**: 池饱和超预算丢的行走行级丢弃,`degraded` 保持 `False`(后端没挂,是本进程并发超了),只按 `degraded` 告警会看不见这一类丢行——而它恰是 `pool_max` 配小了的唯一信号 | | 调用方维度 | 每次调用可带 `tenant_id`(遥测表的真实列,可挂 RLS、可建复合索引)与 `meta`(≤16 个自定义 KV);四个公共方法全覆盖,校验超限即报错;**库只交付列,不启用 RLS、不建索引** | | 遥测表治理 | `llm_calls` 是**下游的表**:PG 侧缺省**不再自动 `ALTER` 补列**(`PGW_TELEMETRY_SCHEMA_MODE` 三态,不设则 sqlite→auto、postgres→manual),manual 档点名缺列并按现有列裁剪写入;`telemetry_schema_sql(backend)` 自取可粘进迁移文件的建表/补列 SQL;`PGW_TELEMETRY_TEXT_CAP` 限正文长度(**不设 = 存全文**);保留期与访问控制走[生产部署 DDL 模板](#生产部署-ddl-模板postgresql)加 `tools/telemetry_retention.py` | diff --git a/research-wiki/index.md b/research-wiki/index.md index 1dcd691..ebf56d3 100644 --- a/research-wiki/index.md +++ b/research-wiki/index.md @@ -103,7 +103,7 @@ - [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review` ## schema (1) -- [表结构: llm_calls(遥测 25 字段)](schemas/llm-calls.md) `schema:llm-calls` +- [表结构: llm_calls(遥测 26 字段)](schemas/llm-calls.md) `schema:llm-calls` ## metric (2) - [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success` diff --git a/research-wiki/schemas/llm-calls.md b/research-wiki/schemas/llm-calls.md index ad82a3e..a40dd97 100644 --- a/research-wiki/schemas/llm-calls.md +++ b/research-wiki/schemas/llm-calls.md @@ -1,11 +1,11 @@ --- type: schema node_id: schema:llm-calls -title: "表结构: llm_calls(遥测 25 字段)" +title: "表结构: llm_calls(遥测 26 字段)" date: 2026-07-20 --- -# 表结构: llm_calls(遥测 25 字段) +# 表结构: llm_calls(遥测 26 字段) ## 列定义(冻结,M1 设计 §4.4 / ARCH §7.8) @@ -31,6 +31,7 @@ date: 2026-07-20 | tenant_id | TEXT NOT NULL DEFAULT '' | 调用方租户(2026-08-17,issue #11);**缺省落哨兵空串而非 NULL**——PG 的 RLS `USING` 对返回 NULL 的行一律隐藏且不报错,NULL 的租户不是「未归属」而是对所有人永久不可见 | | meta | TEXT / JSONB NOT NULL DEFAULT '' / '{}' | 调用方自定义维度(同批,≤16 个 KV);SQLite 存 canonical JSON 串,PG 存 JSONB | | thinking_observation | TEXT | 本次推理是否真的发生的三态裁定(2026-08-25,issue #16/#17);`observed` / `absent` / `unknown`。见下方口径 | +| reasoning_effort | TEXT | 本次调用**实际发出**的推理档位(2026-09-04,issue #20);八档 `Effort` 字面量之一,NULL = 调用方未表态(与 `none`「明确要求不推理」不可混同)。见下方口径 | ## usage/成本口径(2026-07-30,est_tokens 解耦) @@ -104,6 +105,34 @@ ORDER BY model, calls DESC; 三条限定各有理由: `cache_hit = false` 与 `cost`/`cached_prompt_tokens` 同源——缓存命中行原样回放历史观测值,计入即重复计数;`error IS NULL` 排除失败尝试与终态失败行,那些行的本列恒为 `unknown`(无响应可裁定,默认值本身不撒谎),混进来会把「观测不到」的占比整体抬高;时间窗是为了让**变化**可见——某模型的 `unknown` 占比从 0 跳到 100%,正是它停报推理信号的那一天。补列之前写入的历史行本列为 NULL,与 `unknown` 是两回事(前者是那时还没有这一列),跨版本对比须显式区分。 +## 推理档位口径(2026-09-04,issue #20) + +`reasoning_effort` 回答的是「这一行跑在哪一档」——补列之前,25 列里没有任何一列答得出,于是「不同档位是不是真有用」在数据侧无从分组。NULL 有两个来源(调用方未表态 / 档位名读不懂),两者都**不可**折叠进 `none`:`none` 是一次「要求不推理」的表态。 + +三个 emit 入口的取值同样各自定死,与 `sampling` 同构: + +| 入口 | 有生效源? | 记什么 | +|---|---|---| +| `emit_attempt`(成功) | 有 | `response.applied_effort`——transport 裁定的**实发档** | +| `emit_attempt`(失败) | 有 | `effective_effort(请求级 > 源级 > enable_thinking)` 的**请求档** | +| `emit_cache_hit` / `emit_terminal_failure` | 无 | 仅 `request.reasoning_effort` | + +成功行必须读实发档而非重算: 源上开了 `EFFORT_FALLBACK=nearest` 时请求 `medium` 而模型只有 low/high/max,实发的是 `low`,重算会把整行挂在一个从未发出过的档下。失败尝试没有响应,实发档无从得知,故退回请求档——于是开了映射的源上**成功行与失败行不是同一把尺子**,跨 `error IS NULL` 混合统计前必须显式分开。仍然记而不留空,是因为档位错误(`resolve_thinking` 的 Phase 2/4/5)根本没发 HTTP 就被拒,这类行记的正是**被拒绝的那一档**,而「哪一档配错了」正是排障要的信号。 + +OCR / embedding 路径的该列**恒为 NULL**(`emit_attempt(reasoning_applies=False)`),理由与 `sampling` 逐字相同: 两条路径的 payload 不带推理参数,源上即便误配了 `ENABLE_THINKING`,记一个档也是记录一个从未发出的参数。 + +按档位看推理产出,即压测「高档是不是真的多想」的基本查询: + +```sql +SELECT model, reasoning_effort, + count(*) AS calls, + round(avg(reasoning_tokens)) AS avg_reasoning_tokens +FROM llm_calls +WHERE cache_hit = false AND error IS NULL AND reasoning_effort IS NOT NULL +GROUP BY model, reasoning_effort +ORDER BY model, calls DESC; +``` + ## 埋点位置(单一 helper 铁律) - `middleware/telemetry.py::TelemetryEmitter` 是全库**唯一** `record_llm_call` 调用点; diff --git a/src/polygateway/embedding.py b/src/polygateway/embedding.py index 068d547..ac042ca 100644 --- a/src/polygateway/embedding.py +++ b/src/polygateway/embedding.py @@ -411,6 +411,9 @@ class EmbeddingClient: latency_ms=int((self._now() - started) * 1000), response=response, error=None if error is None else str(error), + # embedding payload 硬编码 {model, input},从不带推理参数;源上即便 + # 误配了 ENABLE_THINKING,记一个档也是替这次调用声称它没做过的事 + reasoning_applies=False, ) def _merge(self, outcomes: list[_BatchOutcome]) -> EmbeddingResponse: diff --git a/src/polygateway/middleware/retry.py b/src/polygateway/middleware/retry.py index 5c4c7d4..09a0236 100644 --- a/src/polygateway/middleware/retry.py +++ b/src/polygateway/middleware/retry.py @@ -421,6 +421,8 @@ class RetryMW: latency_ms=int((self._now() - started) * 1000), response=response, error=None if error is None else str(error), + # chat 路径是唯一带推理参数的路径,故实发档由这里的响应说了算 + reasoning_applies=True, ) except asyncio.CancelledError: raise diff --git a/src/polygateway/middleware/telemetry.py b/src/polygateway/middleware/telemetry.py index b330199..3b4c0f6 100644 --- a/src/polygateway/middleware/telemetry.py +++ b/src/polygateway/middleware/telemetry.py @@ -23,7 +23,8 @@ from polygateway.errors import ( SourceNotConfiguredError, ) from polygateway.middleware.cache import digest_messages -from polygateway.types import ThinkingObservation, canonical_sampling_json, merge_sampling +from polygateway.thinking import effective_effort +from polygateway.types import Effort, ThinkingObservation, canonical_sampling_json, merge_sampling if TYPE_CHECKING: from collections.abc import Callable, Mapping @@ -80,6 +81,70 @@ def _normalize_observation(raw: object) -> str: return ThinkingObservation.UNKNOWN.value +def _normalize_effort(raw: object) -> str | None: + """实际档位 → 落库用的裸 str;不表态与域外取值都落 `NULL`。 + + **不写 `raw.value`**,理由与 `_normalize_observation` 逐字相同: `LLMResponse` + 是无运行时校验的 frozen dataclass,测试替身写 `applied_effort="low"` 完全自然, + 而 `.value` 会当场抛 `AttributeError`,被 `_record` 的 `except Exception` 吞成 + 一条泛化 warning —— 丢的不是这一列,是**整行**。 + + 域外取值降级为 `None` 而不抛,方向与 `CacheMW._coerce_applied_effort` 一致 + (设计 §4.4): 多项目共用一套后端时,更新版本的进程可能带来本版没有的档位名, + 归因字段不该有能力废掉一整行遥测。降级到 `None` 也是唯一诚实的说法——库确实 + 不知道这次跑在哪档,随便挑一档等于替上游声称了一件它没说过的事。 + + 注意 `None` 在本列有**两个**来源(不表态 / 读不懂),二者都不可折叠进 `'none'`: + `'none'` 是"明确要求不推理",是一次表态。 + """ + if raw is None: + return None + try: + return Effort(raw).value + except ValueError: + logger.warning( + "推理档位取值 {!r} 不在本版档位词汇内,本行 reasoning_effort 降级记为 NULL" + "(其余列照常落库)", + raw, + ) + return None + + +def _attempt_effort( + *, + request: ChatRequest, + source: SourceConfig, + response: LLMResponse | None, + applies: bool, +) -> str | None: + """一次尝试该记哪一档: 成功读**实发档**,失败退回**请求档**(设计 §6)。 + + 成功行一律读 `response.applied_effort` 而**绝不重算**: 源上开了 + `EFFORT_FALLBACK=nearest` 时,请求 `medium` 而模型只有 low/high/max,实发的是 + `low`;此处重算 `effective_effort` 必然算成请求档,于是整行被挂在一个从未发出 + 过的分组下——而两个值在没开映射的源上恒等,这个错在本地跑不出来。 + + 失败尝试没有响应,实发档无从得知,故退回请求档并**接受这层含义差别**: 开了映射 + 的源上,成功行是映射后的档、失败行是请求档,两种行不是同一把尺子。仍然记而不是 + 留空,是因为档位错误(`resolve_thinking` 的 Phase 2/4/5)根本没发 HTTP 就被拒, + 这类行记的正是**被拒绝的那一档**——"哪一档配错了"是压测与排障要的信号。 + + 回落走 `effective_effort` 而非裸读两个字段: `enable_thinking` 也是一次表态 + (语法糖),漏掉它就会把一次明确要求推理的调用记成"没表态"。 + """ + if not applies: + return None + if response is not None: + return _normalize_effort(response.applied_effort) + return _normalize_effort( + effective_effort( + request_effort=request.reasoning_effort, + source_effort=source.reasoning_effort, + enable_thinking=source.enable_thinking, + ) + ) + + def _cap_text(text: str, cap: int | None) -> str: """超出 cap 时头部硬切并附省略标记 `…(略 N 字)`;cap 为 None 原样返回。""" if cap is None or len(text) <= cap: @@ -161,7 +226,7 @@ class _AttemptUsage: class TelemetryEmitter: - """从请求与结果组装 25 字段并写入 recorder;一切写失败降级 warning。""" + """从请求与结果组装 26 字段并写入 recorder;一切写失败降级 warning。""" def __init__( self, @@ -192,8 +257,17 @@ class TelemetryEmitter: latency_ms: int, response: LLMResponse | None, error: str | None, + reasoning_applies: bool, ) -> None: - """逐次尝试记录(RetryMW 调用);失败尝试无用量可言,记 0 并标 unavailable。""" + """逐次尝试记录(三个 Client 的重试层调用);失败尝试无用量可言,记 0 并标 unavailable。 + + `reasoning_applies` 声明**这条调用路径有没有推理语义**: chat 路径为 + `True`,embedding / OCR 路径为 `False`。它不能由 emitter 自己推断——三条路径 + 共用同一个 `SourceConfig` 类型,一个误配了 `ENABLE_THINKING` 的 embedding 源 + 会让下面的回落算出 `auto`,给一次从来不带推理参数的调用挂上一个从未发出过的 + 档。**不设默认值**: 与 `TelemetryRecorder` 同一约定,库外无第三方调用者,漏传 + 当场 TypeError,好过被静默当成"没表态"。 + """ usage = _AttemptUsage.of(response) await self._record( request=request, @@ -219,6 +293,9 @@ class TelemetryEmitter: sampling=canonical_sampling_json(merge_sampling(source.extra_body, request.sampling)), tenant_id=request.tenant_id, meta=request.meta, + reasoning_effort=_attempt_effort( + request=request, source=source, response=response, applies=reasoning_applies + ), ) async def emit_cache_hit(self, *, request: ChatRequest, response: LLMResponse) -> None: @@ -254,6 +331,9 @@ class TelemetryEmitter: # 记到上一个租户头上,两边的账同时错且无任何报错(issue #11 设计 §4.3) tenant_id=request.tenant_id, meta=request.meta, + # 与 sampling 同一口径: 命中行没有选中源,源级档位与 `nearest` 映射 + # 都无从谈起,只记调用方这次要的档(response 里那个是历史那次实发的) + reasoning_effort=_normalize_effort(request.reasoning_effort), ) async def emit_terminal_failure( @@ -286,6 +366,8 @@ class TelemetryEmitter: # 源不可知,但租户归属是已知的——终态失败行恰是审计最需要的 tenant_id=request.tenant_id, meta=request.meta, + # 可能根本没选出源,故与 sampling 同样只取请求档 + reasoning_effort=_normalize_effort(request.reasoning_effort), ) async def _record( @@ -317,6 +399,10 @@ class TelemetryEmitter: # issue #11: 未归一化的调用方维度,归一化在本方法内收口(recorder 只落库) tenant_id: str | None, meta: Mapping[str, Any], + # issue #20: 已由各入口按自己的口径定型成裸 str/None(口径差别见三个入口的 + # 注释),本方法只搬运——把定型放这里就得再传一遍 response/source,等于把 + # "唯一 record_llm_call 调用点"换成"两处口径判断",那正是要避免的复制 + reasoning_effort: str | None, ) -> None: try: # 成本换算(M2 §6): 成功行按单价换算;缓存命中 0.0(未产生新调用); @@ -370,6 +456,7 @@ class TelemetryEmitter: # 保证接受,而遥测写失败只降级成一条 warning——不会当场炸,只会让 # Postgres 那一路悄悄少一列数据 thinking_observation=_normalize_observation(thinking_observation), + reasoning_effort=reasoning_effort, ) except asyncio.CancelledError: raise diff --git a/src/polygateway/ocr.py b/src/polygateway/ocr.py index d2b486b..7dc0d35 100644 --- a/src/polygateway/ocr.py +++ b/src/polygateway/ocr.py @@ -455,6 +455,8 @@ class OcrClient: latency_ms=latency_ms, response=response, error=error_text, + # OCR 走 MonkeyOCR 自有端点,没有推理参数可言(理由同 embedding) + reasoning_applies=False, ) @staticmethod diff --git a/src/polygateway/ports.py b/src/polygateway/ports.py index 020e943..ed5b20b 100644 --- a/src/polygateway/ports.py +++ b/src/polygateway/ports.py @@ -271,7 +271,7 @@ class TelemetryStatusProvider(Protocol): @runtime_checkable class TelemetryRecorder(Protocol): - """遥测后端;25 字段冻结(M1 设计 §4.4 + issue #3/#4/#11/#16),唯一调用点是 TelemetryEmitter。 + """遥测后端;26 字段冻结(M1 设计 §4.4 + issue #3/#4/#11/#16/#20),唯一调用点是 TelemetryEmitter。 新增参数不设默认值: 库外无第三方实现者(三项目迁移时删除了各自的同名 Protocol),完整签名的成本为零,而少写一列会被 emitter 的降级吞成 warning。 @@ -281,6 +281,9 @@ class TelemetryRecorder(Protocol): `thinking_observation` 同理: emitter 已把 `ThinkingObservation` 取成 `.value` 的裸 `str`(`StrEnum` 是 `str` 子类,而 asyncpg 的参数编码对子类不保证接受, 遥测写失败又只降级成 warning——PG 那一路会静默少一列数据)。 + `reasoning_effort` 同一先例(issue #20): emitter 已把 `Effort` 取成 `.value` + 的裸 `str`,`None` 表示调用方没表态——它与 `'none'`(明确要求不推理)不可折叠。 + recorder 只负责落库,不做任何语义判断,与 `sampling` 列由 `canonical_sampling_json()` 在 emitter 侧定型是同一先例。 """ @@ -313,4 +316,5 @@ class TelemetryRecorder(Protocol): tenant_id: str, meta: str, thinking_observation: str, + reasoning_effort: str | None, ) -> None: ... diff --git a/src/polygateway/telemetry/schema.py b/src/polygateway/telemetry/schema.py index 93e3363..46ada8d 100644 --- a/src/polygateway/telemetry/schema.py +++ b/src/polygateway/telemetry/schema.py @@ -5,8 +5,8 @@ 多处各存一份必然漂移,而漂移的表现是"下游照打印的 SQL 建完表,库仍报缺列"。 **`COLUMNS` 是 INSERT 字段序,不是物理列序**: 数据库自填的 `created_at` 不在其中(它带 -`DEFAULT now()` / `datetime('now')`,库从不显式写它)。物理表列 = 25 个 INSERT 字段 + -`created_at` = 26;列数断言一律按物理列数写,两套口径混用是最易错处。 +`DEFAULT now()` / `datetime('now')`,库从不显式写它)。物理表列 = 26 个 INSERT 字段 + +`created_at` = 27;列数断言一律按物理列数写,两套口径混用是最易错处。 本模块只依赖标准库: `telemetry/` 与 `backends/`、`transports/`、`structured/` 同层且 互不依赖(import-linter 契约执法)。 @@ -51,7 +51,8 @@ CREATE TABLE IF NOT EXISTS llm_calls ( reasoning_tokens INTEGER, tenant_id TEXT NOT NULL DEFAULT '', meta TEXT NOT NULL DEFAULT '{}', - thinking_observation TEXT + thinking_observation TEXT, + reasoning_effort TEXT ); """ @@ -82,7 +83,8 @@ CREATE TABLE IF NOT EXISTS llm_calls ( reasoning_tokens INTEGER, tenant_id TEXT NOT NULL DEFAULT '', meta JSONB NOT NULL DEFAULT '{}'::jsonb, - thinking_observation TEXT + thinking_observation TEXT, + reasoning_effort TEXT ); """ @@ -100,6 +102,9 @@ SQLITE_BACKFILL = ( # 可空: 补列之前的行没有裁定结果,NULL 如实表达"这行根本没记过这件事", # 与哨兵串 'unknown'(库确实裁过但判不出来)是两回事,不得混同 ("thinking_observation", "TEXT"), + # 同样可空,但这里 NULL 表达的是"调用方没表态"(issue #20): 它与 'none' + # (明确要求不推理)是两回事,折叠成任一档都等于替上游声称了它没说过的事 + ("reasoning_effort", "TEXT"), ) # PG 补列的列定义。语句由此派生成两份文本(见下),使"库内执行的那份"与"打印给 @@ -114,6 +119,7 @@ _PG_BACKFILL_DECLS = ( ("meta", "JSONB NOT NULL DEFAULT '{}'::jsonb"), # 可空,理由同 SQLITE_BACKFILL 同名项 ("thinking_observation", "TEXT"), + ("reasoning_effort", "TEXT"), ) # 新列排在 created_at 之后: 与旧表 ALTER 追加的位置一致(见 SQLITE_BACKFILL 同款注释)。 @@ -151,6 +157,7 @@ COLUMNS = ( "tenant_id", "meta", "thinking_observation", + "reasoning_effort", ) _COLUMN_SET = frozenset(COLUMNS) diff --git a/src/polygateway/telemetry/sqlite.py b/src/polygateway/telemetry/sqlite.py index cc46f2c..7781530 100644 --- a/src/polygateway/telemetry/sqlite.py +++ b/src/polygateway/telemetry/sqlite.py @@ -143,7 +143,7 @@ class SQLiteRecorder: logger.warning("SQLite 遥测补列失败(写入将逐行降级): {}", exc) async def record_llm_call(self, **fields: object) -> None: - """写一行遥测;字段集合即 25 字段冻结签名(ports.TelemetryRecorder)。 + """写一行遥测;字段集合即 26 字段冻结签名(ports.TelemetryRecorder)。 取值按 `self._columns`(manual 档可能已被裁剪),与 `self._insert` 的 占位符同序——两者必须一起改,分开改就是把值写进错位的列。 diff --git a/tests/integration/test_postgres_telemetry.py b/tests/integration/test_postgres_telemetry.py index 0159873..380e223 100644 --- a/tests/integration/test_postgres_telemetry.py +++ b/tests/integration/test_postgres_telemetry.py @@ -57,6 +57,7 @@ _EXPECTED_COLUMNS = [ "tenant_id", "meta", "thinking_observation", + "reasoning_effort", ] @@ -127,6 +128,8 @@ async def _record_minimal( "meta": "{}", # 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str "thinking_observation": "unknown", + # 同理: `Effort` 归一成裸 str,不表态则是 None(与 'low' 必须分得开) + "reasoning_effort": None, } fields.update(overrides) await recorder.record_llm_call(**fields) @@ -225,11 +228,12 @@ class TestObservabilityColumns: await _record_minimal( recorder, call_id="samp", sampling='{"seed": 42, "temperature": 0}' ) + await _record_minimal(recorder, call_id="tier", reasoning_effort="low") rows = await _fetch( sandbox.dsn, - "SELECT call_id, cached_prompt_tokens, model_reported, sampling FROM llm_calls " - "WHERE call_id = ANY($1::text[])", - ["hit", "zero", "model", "samp"], + "SELECT call_id, cached_prompt_tokens, model_reported, sampling, " + "reasoning_effort FROM llm_calls WHERE call_id = ANY($1::text[])", + ["hit", "zero", "model", "samp", "tier"], ) by_id = {r["call_id"]: r for r in rows} assert by_id["hit"]["cached_prompt_tokens"] == 64 @@ -239,6 +243,10 @@ class TestObservabilityColumns: # issue #4: PG 侧也须验非空 sampling 能读回原值(不只是列存在) assert json.loads(by_id["samp"]["sampling"]) == {"seed": 42, "temperature": 0} assert by_id["hit"]["sampling"] is None + # issue #20: PG 侧同样要验档位读得回来——emitter 落的是裸 str, + # 若哪天回退成 `Effort` 实例,asyncpg 编码不保证接受,写入会整行降级 + assert by_id["tier"]["reasoning_effort"] == "low" + assert by_id["hit"]["reasoning_effort"] is None # 不表态是 NULL finally: await recorder.aclose() @@ -580,11 +588,13 @@ _PRE_TENANT_INSERT = ( ) -# `_PRE_TENANT_DDL` 的物理列(23 个): 由 `_EXPECTED_COLUMNS` 去掉此后新增的三列 +# `_PRE_TENANT_DDL` 的物理列(23 个): 由 `_EXPECTED_COLUMNS` 去掉此后新增的四列 # 派生而非另抄一份——两份常量必然漂移,而漂移的表现是"manual 档没补列"这条断言假绿。 -# 去掉后的顺序与 DDL 逐字一致(这三列在 DDL 里本就排在末尾)。 +# 去掉后的顺序与 DDL 逐字一致(这四列在 DDL 里本就排在末尾)。 _PRE_TENANT_COLUMNS = [ - c for c in _EXPECTED_COLUMNS if c not in ("tenant_id", "meta", "thinking_observation") + c + for c in _EXPECTED_COLUMNS + if c not in ("tenant_id", "meta", "thinking_observation", "reasoning_effort") ] # 回读要逐列比对的字段: 物理列去掉库从不显式写的 created_at,恰好 22 个 @@ -693,7 +703,7 @@ class TestCallerDimensionsAcceptance: "WHERE table_schema = $1 AND table_name = 'llm_calls' ORDER BY ordinal_position", schema, ) - # 22 → 25 个 recorder 字段(加 created_at 共 26 个物理列),且新列追加在末尾 + # 22 → 26 个 recorder 字段(加 created_at 共 27 个物理列),且新列追加在末尾 assert [r["column_name"] for r in cols] == _EXPECTED_COLUMNS rows = await _fetch( schema_dsn, @@ -869,7 +879,7 @@ class TestManualSchemaModeAcceptance: """22 字段旧表 + manual: 列一个不加,行照常落库,缺的三维度静默不写。 与 `test_pre_tenant_table_gains_columns_and_old_rows_stay_auditable` 恰成对照: - 同一张表、同一份负载,只有 `auto_migrate` 不同,列数就必须是 23 与 26 之别。 + 同一张表、同一份负载,只有 `auto_migrate` 不同,列数就必须是 23 与 27 之别。 """ schema_dsn, schema = pre_tenant_schema recorder = _recorder(schema_dsn, auto_migrate=False) @@ -898,8 +908,11 @@ class TestManualSchemaModeAcceptance: assert [m for m in captured_warnings if "补列失败" in m] == [] notices = [m for m in captured_warnings if "auto_migrate=False" in m] assert len(notices) == 1 # 准备期一次讲清,不逐行刷屏 - # 逐字钉住三个维度: 前缀断言会让将来漏进告警的新列照样绿 - assert "以下维度不会被记录: tenant_id, meta, thinking_observation。" in notices[0] + # 逐字钉住四个维度: 前缀断言会让将来漏进告警的新列照样绿 + assert ( + "以下维度不会被记录: tenant_id, meta, thinking_observation, reasoning_effort。" + in notices[0] + ) finally: await recorder.aclose() @@ -925,8 +938,11 @@ class TestManualSchemaModeAcceptance: assert recorder.telemetry_status.degraded is False notices = [m for m in captured_warnings if "auto_migrate=False" in m] assert len(notices) == 1 # 准备期一次,第二行不再重复 - # 逐字钉住三个维度: 前缀断言会让将来漏进告警的新列照样绿 - assert "以下维度不会被记录: tenant_id, meta, thinking_observation。" in notices[0] + # 逐字钉住四个维度: 前缀断言会让将来漏进告警的新列照样绿 + assert ( + "以下维度不会被记录: tenant_id, meta, thinking_observation, reasoning_effort。" + in notices[0] + ) # 提示里的 SQL 必须可直接粘贴执行,而不是只报个列名 assert ( "ALTER TABLE llm_calls ADD COLUMN tenant_id TEXT NOT NULL DEFAULT '';" in notices[0] @@ -984,7 +1000,7 @@ class TestPublishedSchemaScript: await _execute_script(fresh_dsn, script) actual = [r["column_name"] for r in await _fetch(fresh_dsn, _PHYSICAL_COLUMNS_SQL, schema)] - # 物理列 = 25 个 INSERT 字段 + 库从不显式写的 created_at;对着库常量比,不另抄一份 + # 物理列 = 26 个 INSERT 字段 + 库从不显式写的 created_at;对着库常量比,不另抄一份 assert set(actual) == set(COLUMNS) | {"created_at"} # 列序也不许漂: 新列必须排在 created_at 之后,否则新建库与 ALTER 升级的列序分叉 assert actual == _EXPECTED_COLUMNS diff --git a/tests/unit/test_cache.py b/tests/unit/test_cache.py index a46069b..c83e423 100644 --- a/tests/unit/test_cache.py +++ b/tests/unit/test_cache.py @@ -584,6 +584,7 @@ class TestTelemetryCapDoesNotPoisonTheCacheKey: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) # 截断确实发生了(否则本用例恒真) logged = json.loads(rec.rows[0]["messages"]) diff --git a/tests/unit/test_openai_compat.py b/tests/unit/test_openai_compat.py index 01ee5eb..3d2c525 100644 --- a/tests/unit/test_openai_compat.py +++ b/tests/unit/test_openai_compat.py @@ -123,6 +123,7 @@ async def _recorded_cost(result, source): latency_ms=1, response=response, error=None, + reasoning_applies=True, ) return recorder.rows[0]["cost"] diff --git a/tests/unit/test_ports.py b/tests/unit/test_ports.py index b0d8c5d..f772271 100644 --- a/tests/unit/test_ports.py +++ b/tests/unit/test_ports.py @@ -275,7 +275,9 @@ class TestTelemetryRecorderSignature: params = inspect.signature(TelemetryRecorder.record_llm_call).parameters assert {"tenant_id", "meta"} <= set(params) - @pytest.mark.parametrize("name", ["tenant_id", "meta", "thinking_observation"]) + @pytest.mark.parametrize( + "name", ["tenant_id", "meta", "thinking_observation", "reasoning_effort"] + ) def test_caller_dimensions_have_no_default(self, name): import inspect diff --git a/tests/unit/test_pricing.py b/tests/unit/test_pricing.py index cdd76b4..de031a1 100644 --- a/tests/unit/test_pricing.py +++ b/tests/unit/test_pricing.py @@ -177,6 +177,7 @@ class TestEmitterCost: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] == pytest.approx(7.2) @@ -196,6 +197,7 @@ class TestEmitterCost: latency_ms=1, response=None, error="TransientError: boom", + reasoning_applies=True, ) assert rec.rows[0]["cost"] is None @@ -209,6 +211,7 @@ class TestEmitterCost: latency_ms=1, response=_resp(model="mystery"), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] is None @@ -223,5 +226,6 @@ class TestEmitterCost: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] is None diff --git a/tests/unit/test_telemetry.py b/tests/unit/test_telemetry.py index c91188e..f9b4867 100644 --- a/tests/unit/test_telemetry.py +++ b/tests/unit/test_telemetry.py @@ -25,6 +25,7 @@ from polygateway.types import ( BackpressurePolicy, BreakerConfig, ChatRequest, + Effort, EmbeddingTransportResult, GlobalLimits, LLMResponse, @@ -63,6 +64,7 @@ _EXPECTED_COLUMNS = [ "tenant_id", "meta", "thinking_observation", + "reasoning_effort", ] @@ -132,6 +134,8 @@ async def _record_minimal(recorder, call_id="c1", **overrides): "meta": "{}", # 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str "thinking_observation": "unknown", + # 同理: `Effort` 归一成裸 str,不表态则是 None(与 'low' 必须分得开) + "reasoning_effort": None, } fields.update(overrides) await recorder.record_llm_call(**fields) @@ -177,16 +181,17 @@ _FROZEN_SQLITE_INSERT = ( "INSERT OR IGNORE INTO llm_calls (call_id, parent_call_id, session_id, model, provider, " "source_name, messages, response, thinking, prompt_tokens, completion_tokens, usage_source, " "latency_ms, ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, " - "model_reported, sampling, reasoning_tokens, tenant_id, meta, thinking_observation) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" + "model_reported, sampling, reasoning_tokens, tenant_id, meta, thinking_observation, " + "reasoning_effort) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" ) _FROZEN_PG_INSERT = ( "INSERT INTO llm_calls (call_id, parent_call_id, session_id, model, provider, source_name, " "messages, response, thinking, prompt_tokens, completion_tokens, usage_source, latency_ms, " "ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, model_reported, " - "sampling, reasoning_tokens, tenant_id, meta, thinking_observation) " + "sampling, reasoning_tokens, tenant_id, meta, thinking_observation, reasoning_effort) " "VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, " - "$19, $20, $21, $22, $23, $24, $25) " + "$19, $20, $21, $22, $23, $24, $25, $26) " # 无冲突目标(issue #13 Task 2): 带 `(call_id)` 的版本在按 created_at 分区、 # 主键为 (call_id, created_at) 的表上匹配不到约束,PG 直接拒收整条写入 "ON CONFLICT DO NOTHING" @@ -212,7 +217,7 @@ class TestSchemaModule: # COLUMNS 是 INSERT 字段序,不含数据库自填的 created_at assert list(COLUMNS) == [c for c in _EXPECTED_COLUMNS if c != "created_at"] - assert len(COLUMNS) == 25 + assert len(COLUMNS) == 26 # 两端 DDL 的列出现顺序 == 物理列序(created_at 在第 19 位) for ddl in (SQLITE_DDL, PG_DDL): assert _first_occurrence_order(ddl, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS @@ -226,8 +231,8 @@ class TestSchemaModule: "ALTER TABLE llm_calls ADD COLUMN cached_prompt_tokens INTEGER", ) assert PG_BACKFILL[-1] == ( - "thinking_observation", - "ALTER TABLE llm_calls ADD COLUMN thinking_observation TEXT", + "reasoning_effort", + "ALTER TABLE llm_calls ADD COLUMN reasoning_effort TEXT", ) assert all("IF NOT EXISTS" not in stmt for _, stmt in PG_BACKFILL) @@ -275,7 +280,7 @@ class TestSchemaModule: pg = telemetry_schema_sql("postgres") lite = telemetry_schema_sql("sqlite") for script in (pg, lite): - # 25 个 INSERT 字段 + created_at 全在,且首次出现顺序与建表 DDL 一致 + # 26 个 INSERT 字段 + created_at 全在,且首次出现顺序与建表 DDL 一致 assert _first_occurrence_order(script, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS assert "CREATE TABLE IF NOT EXISTS llm_calls" in script # 人执行的那份必须幂等: PG 用 ADD COLUMN IF NOT EXISTS(与库内那份有意不同) @@ -313,7 +318,7 @@ class TestBackendColumnParity: """新列只能追加在末尾: 旧表经 ALTER 补列必落末尾,插在中间会让两条路径分叉。""" from polygateway.telemetry.schema import COLUMNS - assert COLUMNS[-3:] == ("tenant_id", "meta", "thinking_observation") + assert COLUMNS[-4:] == ("tenant_id", "meta", "thinking_observation", "reasoning_effort") class TestSQLiteRecorder: @@ -405,6 +410,27 @@ class TestSQLiteRecorder: assert rows["t-absent"] == "absent" # 观测到"确实没推理",与"看不出来"不是一回事 assert rows["t-unknown"] == "unknown" + async def test_reasoning_effort_column_round_trips(self, tmp_path): + """issue #20: 实际档位落库,事后才分得清"这一行跑在哪档"。 + + 断言裸串而非枚举,理由与 `thinking_observation` 逐字相同: `StrEnum` 是 + `str` 子类,而 asyncpg 对子类编码不保证接受,遥测写失败只降级一条 warning + ——PG 那一路会静默少一列,SQLite 本地全绿也发现不了。 + """ + recorder = SQLiteRecorder(tmp_path / "t.db", auto_migrate=True) + await _record_minimal(recorder, call_id="e-low", reasoning_effort="low") + await _record_minimal(recorder, call_id="e-max", reasoning_effort="max") + await _record_minimal(recorder, call_id="e-silent") + recorder.close() + rows = dict( + sqlite3.connect(tmp_path / "t.db") + .execute("SELECT call_id, reasoning_effort FROM llm_calls") + .fetchall() + ) + assert rows["e-low"] == "low" + assert rows["e-max"] == "max" + assert rows["e-silent"] is None # 不表态是 NULL,与任何一档都分得开 + async def test_sampling_column_round_trips(self, tmp_path): """issue #4: 采样参数落库,否则事后无法证明某批数据跑在什么温度下。""" recorder = SQLiteRecorder(tmp_path / "t.db", auto_migrate=True) @@ -568,7 +594,7 @@ class TestSQLiteCallerDimensionsAcceptance: conn = sqlite3.connect(db) cols = [r[1] for r in conn.execute("PRAGMA table_info(llm_calls)")] - assert cols == _EXPECTED_COLUMNS # 22 → 25 个 recorder 字段(+ created_at 共 26 物理列) + assert cols == _EXPECTED_COLUMNS # 22 → 26 个 recorder 字段(+ created_at 共 27 物理列) rows = dict(conn.execute("SELECT call_id, tenant_id FROM llm_calls").fetchall()) assert rows["new-row"] == "tenant-a" assert rows["old-row"] == "" # 不是 None: NULL 会被 RLS 静默吞掉 @@ -610,7 +636,7 @@ class TestSQLiteCallerDimensionsAcceptance: stale = sqlite3.connect(db) assert [r[1] for r in stale.execute("PRAGMA table_info(llm_calls)")] == ( - _EXPECTED_COLUMNS[:-3] + _EXPECTED_COLUMNS[:-4] ) # 补列确实没成功,用例不是在只读库上空转 @@ -618,7 +644,7 @@ class TestSQLiteSchemaMode: """issue #13: `auto_migrate` 两档——auto 保持自动补列,manual 只裁剪写入不发 DDL。 列数断言一律按**物理列数**写: 旧表 22 个 INSERT 字段 + `created_at` = 23, - 补齐后 25 + `created_at` = 26。混用 INSERT 字段数与物理列数是本处最易错的地方。 + 补齐后 26 + `created_at` = 27。混用 INSERT 字段数与物理列数是本处最易错的地方。 """ def _physical_columns(self, db: Path) -> list[str]: @@ -657,7 +683,7 @@ class TestSQLiteSchemaMode: assert "ALTER TABLE" in message # 给出可直接执行的补列 SQL async def test_auto_mode_still_upgrades_the_legacy_table(self, tmp_path): - """auto + 同款旧表: 现状回归,补列后物理列数 23 → 26。""" + """auto + 同款旧表: 现状回归,补列后物理列数 23 → 27。""" db = tmp_path / "auto_legacy.db" _make_pre_tenant_db(db) @@ -666,10 +692,10 @@ class TestSQLiteSchemaMode: recorder.close() assert self._physical_columns(db) == _EXPECTED_COLUMNS - assert len(self._physical_columns(db)) == 26 + assert len(self._physical_columns(db)) == 27 async def test_manual_mode_still_creates_a_fresh_table(self, tmp_path): - """manual 只管 ALTER,不管 CREATE: 全新库照建,26 个物理列齐全(设计 §4.2)。""" + """manual 只管 ALTER,不管 CREATE: 全新库照建,27 个物理列齐全(设计 §4.2)。""" db = tmp_path / "manual_fresh.db" recorder = SQLiteRecorder(db, auto_migrate=False) await _record_minimal(recorder, call_id="c-fresh", tenant_id="tenant-a") @@ -894,6 +920,7 @@ class TestPostgresBackfillDiscipline: "tenant_id", "meta", "thinking_observation", + "reasoning_effort", ] def _recorder(self, conn): @@ -1120,6 +1147,7 @@ class TestEmitterRecorderContract: latency_ms=42, response=_resp(), error=None, + reasoning_applies=True, ) assert set(rec.rows[0]) == set(COLUMNS) @@ -1137,6 +1165,7 @@ class TestEmitterRecorderContract: latency_ms=1, response=None, error="boom", + reasoning_applies=True, ) elif emit == "cache_hit": await emitter.emit_cache_hit(request=_REQ, response=_resp()) @@ -1165,6 +1194,7 @@ class TestEmitterThinkingObservation: latency_ms=1, response=_resp(thinking_observation=ThinkingObservation.OBSERVED), error=None, + reasoning_applies=True, ) value = rec.rows[0]["thinking_observation"] assert value == "observed" @@ -1187,6 +1217,7 @@ class TestEmitterThinkingObservation: latency_ms=1, response=_resp(thinking_observation="observed"), error=None, + reasoning_applies=True, ) assert len(rec.rows) == 1, "整行被吞了" value = rec.rows[0]["thinking_observation"] @@ -1212,6 +1243,7 @@ class TestEmitterThinkingObservation: latency_ms=1, response=_resp(thinking_observation="OBSERVED"), # 大小写不符即域外 error=None, + reasoning_applies=True, ) finally: logger.remove(sink_id) @@ -1250,10 +1282,190 @@ class TestEmitterThinkingObservation: latency_ms=1, response=None, error="boom", + reasoning_applies=True, ) assert rec.rows[0]["thinking_observation"] == "unknown" +class TestEmitterReasoningEffort: + """issue #20: 每行记下这次调用**实际跑在哪档**,否则压测无从分组。 + + 三个入口的取值口径**有意不同**,故逐个钉死: 只有 `emit_attempt` 手上有生效源, + 它才谈得上"实际档";另两个入口没有选中源,源级档位无从谈起,只能记请求档。 + 与 `sampling` 列的现有做法同构。 + """ + + async def test_attempt_records_the_tier_the_transport_applied(self): + """`nearest` 映射后成功行记的是**映射后**的档,不是请求档。 + + 请求 `medium`、模型只有 low/high/max 时二者分叉(实发 `low`)。emitter 若 + "顺手"重算 `effective_effort`,记的就是一个从未发出过的档,而两个值在没开 + 映射的源上恒等——本地跑不开映射的源永远看不出这个错。 + """ + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=ChatRequest( + messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.MEDIUM + ), + source=_source(effort_fallback="nearest"), + call_id="c", + latency_ms=1, + response=_resp(applied_effort=Effort.LOW), + error=None, + reasoning_applies=True, + ) + value = rec.rows[0]["reasoning_effort"] + assert value == "low" # 不是 medium: 那一档从未发出去过 + assert type(value) is str # 不是 Effort: 子类实例不得下沉到 recorder + + async def test_failed_attempt_falls_back_to_the_requested_tier(self): + """失败尝试没有响应,实际档不可知,记请求档并接受这层含义差别。 + + 档位错误(resolve 的 Phase 2/4/5)根本没发 HTTP,却照样经 + `RequestRejectedError` 走到这里——记的正是**被拒绝的那一档**,这对 + "哪一档配错了" 是有用信号,不该被过滤掉。 + """ + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=_REQ, + source=_source(reasoning_effort=Effort.HIGH), + call_id="c", + latency_ms=1, + response=None, + error="boom", + reasoning_applies=True, + ) + assert rec.rows[0]["reasoning_effort"] == "high" + + async def test_failed_attempt_resolves_the_syntactic_sugar_too(self): + """回落走 `effective_effort` 而非裸读字段: `enable_thinking` 也是表态。""" + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=_REQ, + source=_source(enable_thinking=True), + call_id="c", + latency_ms=1, + response=None, + error="boom", + reasoning_applies=True, + ) + assert rec.rows[0]["reasoning_effort"] == "auto" + + async def test_cache_hit_records_the_request_tier_not_the_replayed_one(self): + """命中行没有选中源,故记请求档;与 model/prompt_tokens 的回放口径相反。""" + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_cache_hit( + request=ChatRequest( + messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.MEDIUM + ), + response=_resp(cache_hit=True, applied_effort=Effort.LOW), + ) + assert rec.rows[0]["reasoning_effort"] == "medium" + + async def test_terminal_failure_records_the_request_tier(self): + """终态失败可能根本没选出源,源级档位无从谈起。""" + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_terminal_failure( + request=ChatRequest( + messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.XHIGH + ), + call_id="c", + latency_ms=1, + error="dead", + ) + value = rec.rows[0]["reasoning_effort"] + assert value == "xhigh" + assert type(value) is str + + @pytest.mark.parametrize("emit", ["attempt", "cache_hit", "terminal_failure"]) + async def test_silence_lands_as_null(self, emit): + """谁都没表态时落 `NULL`: `None` 与 `'low'` 必须分得开(设计 §6)。 + + 库并不观测模型内部的默认档,记一个推定值等于把"没看见"说成"发生了"。 + """ + rec = _MemoryRecorder() + emitter = TelemetryEmitter(rec, text_cap=None) + if emit == "attempt": + await emitter.emit_attempt( + request=_REQ, + source=_source(), + call_id="c", + latency_ms=1, + response=_resp(), + error=None, + reasoning_applies=True, + ) + elif emit == "cache_hit": + await emitter.emit_cache_hit(request=_REQ, response=_resp(cache_hit=True)) + else: + await emitter.emit_terminal_failure( + request=_REQ, call_id="c", latency_ms=1, error="dead" + ) + assert rec.rows[0]["reasoning_effort"] is None + + async def test_a_reasonless_path_never_records_a_tier(self): + """embedding/OCR 走同一个 emitter,但它们的 payload 里没有推理参数。 + + 源上误配了 `ENABLE_THINKING` 时,回落若照算就会给一次 embedding 失败 + 挂上 `auto` ——那一档从来没有、也不可能被发出去。 + """ + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=_REQ, + source=_source(enable_thinking=True), + call_id="c", + latency_ms=1, + response=None, + error="boom", + reasoning_applies=False, + ) + assert rec.rows[0]["reasoning_effort"] is None + + async def test_an_out_of_domain_tier_degrades_but_keeps_the_row(self): + """域外取值降级为 `NULL` 且**不丢整行**(遥测必录);与缓存回放同一方向。 + + `LLMResponse` 无运行时校验,测试替身写裸串完全自然;直接 `Effort(raw)` 会 + 抛 `ValueError`,被 `_record` 的 `except Exception` 吞成丢整行。 + """ + rec = _MemoryRecorder() + messages: list[str] = [] + sink_id = logger.add(messages.append, level="WARNING") + try: + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=_REQ, + source=_source(), + call_id="c", + latency_ms=1, + response=_resp(applied_effort="lowest"), + error=None, + reasoning_applies=True, + ) + finally: + logger.remove(sink_id) + assert len(rec.rows) == 1, "整行被吞了" + assert rec.rows[0]["reasoning_effort"] is None + hits = [m for m in messages if "lowest" in m] + assert len(hits) == 1, f"域外取值必须单独告警: {messages}" + assert [m for m in messages if "遥测记录失败" in m] == [] + + async def test_a_bare_string_tier_still_lands(self): + """裸串在域内时照常归一并落库,整行不得丢失。""" + rec = _MemoryRecorder() + await TelemetryEmitter(rec, text_cap=None).emit_attempt( + request=_REQ, + source=_source(), + call_id="c", + latency_ms=1, + response=_resp(applied_effort="max"), + error=None, + reasoning_applies=True, + ) + assert len(rec.rows) == 1, "整行被吞了" + value = rec.rows[0]["reasoning_effort"] + assert value == "max" + assert type(value) is str + + class TestEmitterObservabilityFields: """issue #3: 三个入口各自的取值口径(设计 §5 表)。""" @@ -1266,6 +1478,7 @@ class TestEmitterObservabilityFields: latency_ms=42, response=_resp(cached_prompt_tokens=64, model_reported="m-real", reasoning_tokens=7), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cached_prompt_tokens"] == 64 assert rec.rows[0]["model_reported"] == "m-real" @@ -1280,6 +1493,7 @@ class TestEmitterObservabilityFields: latency_ms=7, response=None, error="boom", + reasoning_applies=True, ) assert rec.rows[0]["cached_prompt_tokens"] is None assert rec.rows[0]["model_reported"] is None @@ -1330,6 +1544,7 @@ class TestEmitterSamplingColumn: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) assert json.loads(rec.rows[0]["sampling"]) == {"seed": 42, "temperature": 0} @@ -1344,6 +1559,7 @@ class TestEmitterSamplingColumn: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) await emitter.emit_cache_hit(request=self._SAMPLED, response=_resp()) await emitter.emit_terminal_failure( @@ -1376,6 +1592,7 @@ class TestEmitterSamplingColumn: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) assert rec.rows[0]["sampling"] is None @@ -1408,6 +1625,7 @@ class TestEmitterCallerDimensions: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) elif emit == "cache_hit": await emitter.emit_cache_hit(request=self._REQ_A, response=_resp(cache_hit=True)) @@ -1461,6 +1679,7 @@ class TestEmitterCallerDimensions: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) row = rec.rows[0] assert row["tenant_id"] == "" @@ -1521,6 +1740,7 @@ class TestCostWithCachedTier: latency_ms=1, response=full, error=None, + reasoning_applies=True, ) await emitter.emit_attempt( request=_REQ, @@ -1531,6 +1751,7 @@ class TestCostWithCachedTier: prompt_tokens=1_000_000, completion_tokens=0, cached_prompt_tokens=600_000 ), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] == pytest.approx(10.0) assert rec.rows[1]["cost"] == pytest.approx(5.2) # 400k×10 + 600k×2 @@ -1553,6 +1774,7 @@ class TestCostWithCachedTier: latency_ms=1, response=_resp(usage_source="unavailable", cached_prompt_tokens=5), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] is None @@ -1568,6 +1790,7 @@ class TestEmitter: latency_ms=42, response=_resp(), error=None, + reasoning_applies=True, ) row = rec.rows[0] assert row["call_id"] == "cid-1" and row["error"] is None @@ -1584,6 +1807,7 @@ class TestEmitter: latency_ms=7, response=None, error="TransientError: boom", + reasoning_applies=True, ) row = rec.rows[0] assert row["error"].startswith("TransientError") @@ -1615,6 +1839,7 @@ class TestEmitter: usage_source="unavailable", prompt_tokens=prompt, completion_tokens=completion ), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] is None @@ -1628,6 +1853,7 @@ class TestEmitter: latency_ms=42, response=_resp(prompt_tokens=0, completion_tokens=4000), error=None, + reasoning_applies=True, ) assert rec.rows[0]["cost"] == pytest.approx(0.032) @@ -1661,6 +1887,7 @@ class TestEmitter: latency_ms=1, response=None, error="x", + reasoning_applies=True, ) assert len(rec.rows[0]["messages"]) < 500 # base64 不整段进库(VT R12) @@ -1677,6 +1904,7 @@ class TestEmitter: latency_ms=1, response=_resp(), error=None, + reasoning_applies=True, ) # 不抛(降级不冒泡) @@ -1773,6 +2001,7 @@ async def _emit_with_cap(messages, *, cap, response=_LONG, thinking=_LONG): latency_ms=1, response=_resp(content=response, thinking=thinking), error=None, + reasoning_applies=True, ) return rec.rows[0] diff --git a/tests/unit/test_usage_source_domain.py b/tests/unit/test_usage_source_domain.py index 7dd3459..8c583a5 100644 --- a/tests/unit/test_usage_source_domain.py +++ b/tests/unit/test_usage_source_domain.py @@ -262,6 +262,7 @@ async def test_emit_attempt_success_stays_in_domain(emitted): latency_ms=10, response=_resp(emitted), error=None, + reasoning_applies=True, ) assert recorder.rows[0]["usage_source"] in USAGE_SOURCES @@ -276,6 +277,7 @@ async def test_emit_attempt_failed_attempt_stays_in_domain(): latency_ms=10, response=None, error="boom", + reasoning_applies=True, ) assert recorder.rows[0]["usage_source"] in USAGE_SOURCES