feat: record which tier a call actually ran at
Twenty-five columns and not one of them answered "which tier was this?", so the question the whole issue exists to settle - does a higher tier buy anything - had no way to group its data. The three emit entry points deliberately disagree, the way sampling already does. A successful attempt records what the transport actually sent: with EFFORT_FALLBACK=nearest a request for medium goes out as low, and recomputing here would file the row under a tier that never left the process. A failed attempt has no response to read, so it falls back to the requested tier - which is exactly right for the tier errors that are rejected before any HTTP happens, because the rejected tier is the signal. Cache hits and terminal failures have no chosen source at all, so a source-level tier is not a thing they could report. emit_attempt now demands to be told whether the path reasons at all. Embedding and OCR share the emitter but never send reasoning parameters; without the flag a source that mistakenly carries ENABLE_THINKING would hang a tier on a call that could not possibly have run at one. The value lands as a plain str. StrEnum is a str subclass and asyncpg promises nothing about encoding subclasses, and a telemetry write that fails is only a warning - Postgres would just quietly lose the column. NULL means nobody declared a tier, which is not the same statement as 'none', and the two must never be folded together.
This commit is contained in:
@@ -19,7 +19,7 @@
|
|||||||
| 响应缓存 | Redis/内存;key 含 model + messages 摘要 + namespace(缓存隔离单位)+ salt + 采样参数,多模态 content 先摘要再 hash(防毒化);可 per-call 绕过(科研重采样) |
|
| 响应缓存 | Redis/内存;key 含 model + messages 摘要 + namespace(缓存隔离单位)+ salt + 采样参数,多模态 content 先摘要再 hash(防毒化);可 per-call 绕过(科研重采样) |
|
||||||
| 流式看门狗 | TTFT / inter-token / 总超时三层活性;thinking token 刷活性不计结果;截断流(缺 `[DONE]`)判瞬时不入缓存 |
|
| 流式看门狗 | TTFT / inter-token / 总超时三层活性;thinking token 刷活性不计结果;截断流(缺 `[DONE]`)判瞬时不入缓存 |
|
||||||
| 推理可观测性 | "这次到底推理没推理"由多信号裁定(推理正文压倒 usage 明细),三态落在 `LLMResponse.thinking_observation`:`observed` / `absent` / `unknown`——**`unknown` 是"本次判不出",不是"没推理"**;请求方向与实测观测矛盾时按 `(模型, 方向)` 各告警一次(能力表过期、开启未生效、注入了却观测不到);裁定结果随遥测落库 |
|
| 推理可观测性 | "这次到底推理没推理"由多信号裁定(推理正文压倒 usage 明细),三态落在 `LLMResponse.thinking_observation`:`observed` / `absent` / `unknown`——**`unknown` 是"本次判不出",不是"没推理"**;请求方向与实测观测矛盾时按 `(模型, 方向)` 各告警一次(能力表过期、开启未生效、注入了却观测不到);裁定结果随遥测落库 |
|
||||||
| 遥测与成本 | 每次调用(含缓存命中与失败)必录 25 字段;SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 |
|
| 遥测与成本 | 每次调用(含缓存命中与失败)必录 26 字段;SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 |
|
||||||
| 遥测的资源与降级 | Postgres 池**闲时占 0 条连接**、忙时上限可配(`PGW_TELEMETRY_PG_POOL_MAX`,缺省 4),每次写入有硬预算(`PGW_TELEMETRY_PG_WRITE_TIMEOUT_S`,缺省 5s);后端不可用是**可恢复的降级**(冷却 60s 后自动重试,DBA 建完表/放开权限即自愈),永久失能只留给 DSN 本身写错;降级状态可编程查询——`client.telemetry_status` 给出 `degraded`/`fatal`/`reason`/`dropped_rows` 等只读快照,不必再靠人工对账。**对账要同时看 `degraded` 与 `dropped_rows`**: 池饱和超预算丢的行走行级丢弃,`degraded` 保持 `False`(后端没挂,是本进程并发超了),只按 `degraded` 告警会看不见这一类丢行——而它恰是 `pool_max` 配小了的唯一信号 |
|
| 遥测的资源与降级 | Postgres 池**闲时占 0 条连接**、忙时上限可配(`PGW_TELEMETRY_PG_POOL_MAX`,缺省 4),每次写入有硬预算(`PGW_TELEMETRY_PG_WRITE_TIMEOUT_S`,缺省 5s);后端不可用是**可恢复的降级**(冷却 60s 后自动重试,DBA 建完表/放开权限即自愈),永久失能只留给 DSN 本身写错;降级状态可编程查询——`client.telemetry_status` 给出 `degraded`/`fatal`/`reason`/`dropped_rows` 等只读快照,不必再靠人工对账。**对账要同时看 `degraded` 与 `dropped_rows`**: 池饱和超预算丢的行走行级丢弃,`degraded` 保持 `False`(后端没挂,是本进程并发超了),只按 `degraded` 告警会看不见这一类丢行——而它恰是 `pool_max` 配小了的唯一信号 |
|
||||||
| 调用方维度 | 每次调用可带 `tenant_id`(遥测表的真实列,可挂 RLS、可建复合索引)与 `meta`(≤16 个自定义 KV);四个公共方法全覆盖,校验超限即报错;**库只交付列,不启用 RLS、不建索引** |
|
| 调用方维度 | 每次调用可带 `tenant_id`(遥测表的真实列,可挂 RLS、可建复合索引)与 `meta`(≤16 个自定义 KV);四个公共方法全覆盖,校验超限即报错;**库只交付列,不启用 RLS、不建索引** |
|
||||||
| 遥测表治理 | `llm_calls` 是**下游的表**:PG 侧缺省**不再自动 `ALTER` 补列**(`PGW_TELEMETRY_SCHEMA_MODE` 三态,不设则 sqlite→auto、postgres→manual),manual 档点名缺列并按现有列裁剪写入;`telemetry_schema_sql(backend)` 自取可粘进迁移文件的建表/补列 SQL;`PGW_TELEMETRY_TEXT_CAP` 限正文长度(**不设 = 存全文**);保留期与访问控制走[生产部署 DDL 模板](#生产部署-ddl-模板postgresql)加 `tools/telemetry_retention.py` |
|
| 遥测表治理 | `llm_calls` 是**下游的表**:PG 侧缺省**不再自动 `ALTER` 补列**(`PGW_TELEMETRY_SCHEMA_MODE` 三态,不设则 sqlite→auto、postgres→manual),manual 档点名缺列并按现有列裁剪写入;`telemetry_schema_sql(backend)` 自取可粘进迁移文件的建表/补列 SQL;`PGW_TELEMETRY_TEXT_CAP` 限正文长度(**不设 = 存全文**);保留期与访问控制走[生产部署 DDL 模板](#生产部署-ddl-模板postgresql)加 `tools/telemetry_retention.py` |
|
||||||
|
|||||||
@@ -103,7 +103,7 @@
|
|||||||
- [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review`
|
- [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review`
|
||||||
|
|
||||||
## schema (1)
|
## schema (1)
|
||||||
- [表结构: llm_calls(遥测 25 字段)](schemas/llm-calls.md) `schema:llm-calls`
|
- [表结构: llm_calls(遥测 26 字段)](schemas/llm-calls.md) `schema:llm-calls`
|
||||||
|
|
||||||
## metric (2)
|
## metric (2)
|
||||||
- [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success`
|
- [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success`
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
---
|
---
|
||||||
type: schema
|
type: schema
|
||||||
node_id: schema:llm-calls
|
node_id: schema:llm-calls
|
||||||
title: "表结构: llm_calls(遥测 25 字段)"
|
title: "表结构: llm_calls(遥测 26 字段)"
|
||||||
date: 2026-07-20
|
date: 2026-07-20
|
||||||
---
|
---
|
||||||
|
|
||||||
# 表结构: llm_calls(遥测 25 字段)
|
# 表结构: llm_calls(遥测 26 字段)
|
||||||
|
|
||||||
|
|
||||||
## 列定义(冻结,M1 设计 §4.4 / ARCH §7.8)
|
## 列定义(冻结,M1 设计 §4.4 / ARCH §7.8)
|
||||||
@@ -31,6 +31,7 @@ date: 2026-07-20
|
|||||||
| tenant_id | TEXT NOT NULL DEFAULT '' | 调用方租户(2026-08-17,issue #11);**缺省落哨兵空串而非 NULL**——PG 的 RLS `USING` 对返回 NULL 的行一律隐藏且不报错,NULL 的租户不是「未归属」而是对所有人永久不可见 |
|
| tenant_id | TEXT NOT NULL DEFAULT '' | 调用方租户(2026-08-17,issue #11);**缺省落哨兵空串而非 NULL**——PG 的 RLS `USING` 对返回 NULL 的行一律隐藏且不报错,NULL 的租户不是「未归属」而是对所有人永久不可见 |
|
||||||
| meta | TEXT / JSONB NOT NULL DEFAULT '' / '{}' | 调用方自定义维度(同批,≤16 个 KV);SQLite 存 canonical JSON 串,PG 存 JSONB |
|
| meta | TEXT / JSONB NOT NULL DEFAULT '' / '{}' | 调用方自定义维度(同批,≤16 个 KV);SQLite 存 canonical JSON 串,PG 存 JSONB |
|
||||||
| thinking_observation | TEXT | 本次推理是否真的发生的三态裁定(2026-08-25,issue #16/#17);`observed` / `absent` / `unknown`。见下方口径 |
|
| thinking_observation | TEXT | 本次推理是否真的发生的三态裁定(2026-08-25,issue #16/#17);`observed` / `absent` / `unknown`。见下方口径 |
|
||||||
|
| reasoning_effort | TEXT | 本次调用**实际发出**的推理档位(2026-09-04,issue #20);八档 `Effort` 字面量之一,NULL = 调用方未表态(与 `none`「明确要求不推理」不可混同)。见下方口径 |
|
||||||
|
|
||||||
## usage/成本口径(2026-07-30,est_tokens 解耦)
|
## usage/成本口径(2026-07-30,est_tokens 解耦)
|
||||||
|
|
||||||
@@ -104,6 +105,34 @@ ORDER BY model, calls DESC;
|
|||||||
|
|
||||||
三条限定各有理由: `cache_hit = false` 与 `cost`/`cached_prompt_tokens` 同源——缓存命中行原样回放历史观测值,计入即重复计数;`error IS NULL` 排除失败尝试与终态失败行,那些行的本列恒为 `unknown`(无响应可裁定,默认值本身不撒谎),混进来会把「观测不到」的占比整体抬高;时间窗是为了让**变化**可见——某模型的 `unknown` 占比从 0 跳到 100%,正是它停报推理信号的那一天。补列之前写入的历史行本列为 NULL,与 `unknown` 是两回事(前者是那时还没有这一列),跨版本对比须显式区分。
|
三条限定各有理由: `cache_hit = false` 与 `cost`/`cached_prompt_tokens` 同源——缓存命中行原样回放历史观测值,计入即重复计数;`error IS NULL` 排除失败尝试与终态失败行,那些行的本列恒为 `unknown`(无响应可裁定,默认值本身不撒谎),混进来会把「观测不到」的占比整体抬高;时间窗是为了让**变化**可见——某模型的 `unknown` 占比从 0 跳到 100%,正是它停报推理信号的那一天。补列之前写入的历史行本列为 NULL,与 `unknown` 是两回事(前者是那时还没有这一列),跨版本对比须显式区分。
|
||||||
|
|
||||||
|
## 推理档位口径(2026-09-04,issue #20)
|
||||||
|
|
||||||
|
`reasoning_effort` 回答的是「这一行跑在哪一档」——补列之前,25 列里没有任何一列答得出,于是「不同档位是不是真有用」在数据侧无从分组。NULL 有两个来源(调用方未表态 / 档位名读不懂),两者都**不可**折叠进 `none`:`none` 是一次「要求不推理」的表态。
|
||||||
|
|
||||||
|
三个 emit 入口的取值同样各自定死,与 `sampling` 同构:
|
||||||
|
|
||||||
|
| 入口 | 有生效源? | 记什么 |
|
||||||
|
|---|---|---|
|
||||||
|
| `emit_attempt`(成功) | 有 | `response.applied_effort`——transport 裁定的**实发档** |
|
||||||
|
| `emit_attempt`(失败) | 有 | `effective_effort(请求级 > 源级 > enable_thinking)` 的**请求档** |
|
||||||
|
| `emit_cache_hit` / `emit_terminal_failure` | 无 | 仅 `request.reasoning_effort` |
|
||||||
|
|
||||||
|
成功行必须读实发档而非重算: 源上开了 `EFFORT_FALLBACK=nearest` 时请求 `medium` 而模型只有 low/high/max,实发的是 `low`,重算会把整行挂在一个从未发出过的档下。失败尝试没有响应,实发档无从得知,故退回请求档——于是开了映射的源上**成功行与失败行不是同一把尺子**,跨 `error IS NULL` 混合统计前必须显式分开。仍然记而不留空,是因为档位错误(`resolve_thinking` 的 Phase 2/4/5)根本没发 HTTP 就被拒,这类行记的正是**被拒绝的那一档**,而「哪一档配错了」正是排障要的信号。
|
||||||
|
|
||||||
|
OCR / embedding 路径的该列**恒为 NULL**(`emit_attempt(reasoning_applies=False)`),理由与 `sampling` 逐字相同: 两条路径的 payload 不带推理参数,源上即便误配了 `ENABLE_THINKING`,记一个档也是记录一个从未发出的参数。
|
||||||
|
|
||||||
|
按档位看推理产出,即压测「高档是不是真的多想」的基本查询:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT model, reasoning_effort,
|
||||||
|
count(*) AS calls,
|
||||||
|
round(avg(reasoning_tokens)) AS avg_reasoning_tokens
|
||||||
|
FROM llm_calls
|
||||||
|
WHERE cache_hit = false AND error IS NULL AND reasoning_effort IS NOT NULL
|
||||||
|
GROUP BY model, reasoning_effort
|
||||||
|
ORDER BY model, calls DESC;
|
||||||
|
```
|
||||||
|
|
||||||
## 埋点位置(单一 helper 铁律)
|
## 埋点位置(单一 helper 铁律)
|
||||||
|
|
||||||
- `middleware/telemetry.py::TelemetryEmitter` 是全库**唯一** `record_llm_call` 调用点;
|
- `middleware/telemetry.py::TelemetryEmitter` 是全库**唯一** `record_llm_call` 调用点;
|
||||||
|
|||||||
@@ -411,6 +411,9 @@ class EmbeddingClient:
|
|||||||
latency_ms=int((self._now() - started) * 1000),
|
latency_ms=int((self._now() - started) * 1000),
|
||||||
response=response,
|
response=response,
|
||||||
error=None if error is None else str(error),
|
error=None if error is None else str(error),
|
||||||
|
# embedding payload 硬编码 {model, input},从不带推理参数;源上即便
|
||||||
|
# 误配了 ENABLE_THINKING,记一个档也是替这次调用声称它没做过的事
|
||||||
|
reasoning_applies=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _merge(self, outcomes: list[_BatchOutcome]) -> EmbeddingResponse:
|
def _merge(self, outcomes: list[_BatchOutcome]) -> EmbeddingResponse:
|
||||||
|
|||||||
@@ -421,6 +421,8 @@ class RetryMW:
|
|||||||
latency_ms=int((self._now() - started) * 1000),
|
latency_ms=int((self._now() - started) * 1000),
|
||||||
response=response,
|
response=response,
|
||||||
error=None if error is None else str(error),
|
error=None if error is None else str(error),
|
||||||
|
# chat 路径是唯一带推理参数的路径,故实发档由这里的响应说了算
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
|
|||||||
@@ -23,7 +23,8 @@ from polygateway.errors import (
|
|||||||
SourceNotConfiguredError,
|
SourceNotConfiguredError,
|
||||||
)
|
)
|
||||||
from polygateway.middleware.cache import digest_messages
|
from polygateway.middleware.cache import digest_messages
|
||||||
from polygateway.types import ThinkingObservation, canonical_sampling_json, merge_sampling
|
from polygateway.thinking import effective_effort
|
||||||
|
from polygateway.types import Effort, ThinkingObservation, canonical_sampling_json, merge_sampling
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from collections.abc import Callable, Mapping
|
from collections.abc import Callable, Mapping
|
||||||
@@ -80,6 +81,70 @@ def _normalize_observation(raw: object) -> str:
|
|||||||
return ThinkingObservation.UNKNOWN.value
|
return ThinkingObservation.UNKNOWN.value
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_effort(raw: object) -> str | None:
|
||||||
|
"""实际档位 → 落库用的裸 str;不表态与域外取值都落 `NULL`。
|
||||||
|
|
||||||
|
**不写 `raw.value`**,理由与 `_normalize_observation` 逐字相同: `LLMResponse`
|
||||||
|
是无运行时校验的 frozen dataclass,测试替身写 `applied_effort="low"` 完全自然,
|
||||||
|
而 `.value` 会当场抛 `AttributeError`,被 `_record` 的 `except Exception` 吞成
|
||||||
|
一条泛化 warning —— 丢的不是这一列,是**整行**。
|
||||||
|
|
||||||
|
域外取值降级为 `None` 而不抛,方向与 `CacheMW._coerce_applied_effort` 一致
|
||||||
|
(设计 §4.4): 多项目共用一套后端时,更新版本的进程可能带来本版没有的档位名,
|
||||||
|
归因字段不该有能力废掉一整行遥测。降级到 `None` 也是唯一诚实的说法——库确实
|
||||||
|
不知道这次跑在哪档,随便挑一档等于替上游声称了一件它没说过的事。
|
||||||
|
|
||||||
|
注意 `None` 在本列有**两个**来源(不表态 / 读不懂),二者都不可折叠进 `'none'`:
|
||||||
|
`'none'` 是"明确要求不推理",是一次表态。
|
||||||
|
"""
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return Effort(raw).value
|
||||||
|
except ValueError:
|
||||||
|
logger.warning(
|
||||||
|
"推理档位取值 {!r} 不在本版档位词汇内,本行 reasoning_effort 降级记为 NULL"
|
||||||
|
"(其余列照常落库)",
|
||||||
|
raw,
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _attempt_effort(
|
||||||
|
*,
|
||||||
|
request: ChatRequest,
|
||||||
|
source: SourceConfig,
|
||||||
|
response: LLMResponse | None,
|
||||||
|
applies: bool,
|
||||||
|
) -> str | None:
|
||||||
|
"""一次尝试该记哪一档: 成功读**实发档**,失败退回**请求档**(设计 §6)。
|
||||||
|
|
||||||
|
成功行一律读 `response.applied_effort` 而**绝不重算**: 源上开了
|
||||||
|
`EFFORT_FALLBACK=nearest` 时,请求 `medium` 而模型只有 low/high/max,实发的是
|
||||||
|
`low`;此处重算 `effective_effort` 必然算成请求档,于是整行被挂在一个从未发出
|
||||||
|
过的分组下——而两个值在没开映射的源上恒等,这个错在本地跑不出来。
|
||||||
|
|
||||||
|
失败尝试没有响应,实发档无从得知,故退回请求档并**接受这层含义差别**: 开了映射
|
||||||
|
的源上,成功行是映射后的档、失败行是请求档,两种行不是同一把尺子。仍然记而不是
|
||||||
|
留空,是因为档位错误(`resolve_thinking` 的 Phase 2/4/5)根本没发 HTTP 就被拒,
|
||||||
|
这类行记的正是**被拒绝的那一档**——"哪一档配错了"是压测与排障要的信号。
|
||||||
|
|
||||||
|
回落走 `effective_effort` 而非裸读两个字段: `enable_thinking` 也是一次表态
|
||||||
|
(语法糖),漏掉它就会把一次明确要求推理的调用记成"没表态"。
|
||||||
|
"""
|
||||||
|
if not applies:
|
||||||
|
return None
|
||||||
|
if response is not None:
|
||||||
|
return _normalize_effort(response.applied_effort)
|
||||||
|
return _normalize_effort(
|
||||||
|
effective_effort(
|
||||||
|
request_effort=request.reasoning_effort,
|
||||||
|
source_effort=source.reasoning_effort,
|
||||||
|
enable_thinking=source.enable_thinking,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _cap_text(text: str, cap: int | None) -> str:
|
def _cap_text(text: str, cap: int | None) -> str:
|
||||||
"""超出 cap 时头部硬切并附省略标记 `…(略 N 字)`;cap 为 None 原样返回。"""
|
"""超出 cap 时头部硬切并附省略标记 `…(略 N 字)`;cap 为 None 原样返回。"""
|
||||||
if cap is None or len(text) <= cap:
|
if cap is None or len(text) <= cap:
|
||||||
@@ -161,7 +226,7 @@ class _AttemptUsage:
|
|||||||
|
|
||||||
|
|
||||||
class TelemetryEmitter:
|
class TelemetryEmitter:
|
||||||
"""从请求与结果组装 25 字段并写入 recorder;一切写失败降级 warning。"""
|
"""从请求与结果组装 26 字段并写入 recorder;一切写失败降级 warning。"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
@@ -192,8 +257,17 @@ class TelemetryEmitter:
|
|||||||
latency_ms: int,
|
latency_ms: int,
|
||||||
response: LLMResponse | None,
|
response: LLMResponse | None,
|
||||||
error: str | None,
|
error: str | None,
|
||||||
|
reasoning_applies: bool,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""逐次尝试记录(RetryMW 调用);失败尝试无用量可言,记 0 并标 unavailable。"""
|
"""逐次尝试记录(三个 Client 的重试层调用);失败尝试无用量可言,记 0 并标 unavailable。
|
||||||
|
|
||||||
|
`reasoning_applies` 声明**这条调用路径有没有推理语义**: chat 路径为
|
||||||
|
`True`,embedding / OCR 路径为 `False`。它不能由 emitter 自己推断——三条路径
|
||||||
|
共用同一个 `SourceConfig` 类型,一个误配了 `ENABLE_THINKING` 的 embedding 源
|
||||||
|
会让下面的回落算出 `auto`,给一次从来不带推理参数的调用挂上一个从未发出过的
|
||||||
|
档。**不设默认值**: 与 `TelemetryRecorder` 同一约定,库外无第三方调用者,漏传
|
||||||
|
当场 TypeError,好过被静默当成"没表态"。
|
||||||
|
"""
|
||||||
usage = _AttemptUsage.of(response)
|
usage = _AttemptUsage.of(response)
|
||||||
await self._record(
|
await self._record(
|
||||||
request=request,
|
request=request,
|
||||||
@@ -219,6 +293,9 @@ class TelemetryEmitter:
|
|||||||
sampling=canonical_sampling_json(merge_sampling(source.extra_body, request.sampling)),
|
sampling=canonical_sampling_json(merge_sampling(source.extra_body, request.sampling)),
|
||||||
tenant_id=request.tenant_id,
|
tenant_id=request.tenant_id,
|
||||||
meta=request.meta,
|
meta=request.meta,
|
||||||
|
reasoning_effort=_attempt_effort(
|
||||||
|
request=request, source=source, response=response, applies=reasoning_applies
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
async def emit_cache_hit(self, *, request: ChatRequest, response: LLMResponse) -> None:
|
async def emit_cache_hit(self, *, request: ChatRequest, response: LLMResponse) -> None:
|
||||||
@@ -254,6 +331,9 @@ class TelemetryEmitter:
|
|||||||
# 记到上一个租户头上,两边的账同时错且无任何报错(issue #11 设计 §4.3)
|
# 记到上一个租户头上,两边的账同时错且无任何报错(issue #11 设计 §4.3)
|
||||||
tenant_id=request.tenant_id,
|
tenant_id=request.tenant_id,
|
||||||
meta=request.meta,
|
meta=request.meta,
|
||||||
|
# 与 sampling 同一口径: 命中行没有选中源,源级档位与 `nearest` 映射
|
||||||
|
# 都无从谈起,只记调用方这次要的档(response 里那个是历史那次实发的)
|
||||||
|
reasoning_effort=_normalize_effort(request.reasoning_effort),
|
||||||
)
|
)
|
||||||
|
|
||||||
async def emit_terminal_failure(
|
async def emit_terminal_failure(
|
||||||
@@ -286,6 +366,8 @@ class TelemetryEmitter:
|
|||||||
# 源不可知,但租户归属是已知的——终态失败行恰是审计最需要的
|
# 源不可知,但租户归属是已知的——终态失败行恰是审计最需要的
|
||||||
tenant_id=request.tenant_id,
|
tenant_id=request.tenant_id,
|
||||||
meta=request.meta,
|
meta=request.meta,
|
||||||
|
# 可能根本没选出源,故与 sampling 同样只取请求档
|
||||||
|
reasoning_effort=_normalize_effort(request.reasoning_effort),
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _record(
|
async def _record(
|
||||||
@@ -317,6 +399,10 @@ class TelemetryEmitter:
|
|||||||
# issue #11: 未归一化的调用方维度,归一化在本方法内收口(recorder 只落库)
|
# issue #11: 未归一化的调用方维度,归一化在本方法内收口(recorder 只落库)
|
||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
meta: Mapping[str, Any],
|
meta: Mapping[str, Any],
|
||||||
|
# issue #20: 已由各入口按自己的口径定型成裸 str/None(口径差别见三个入口的
|
||||||
|
# 注释),本方法只搬运——把定型放这里就得再传一遍 response/source,等于把
|
||||||
|
# "唯一 record_llm_call 调用点"换成"两处口径判断",那正是要避免的复制
|
||||||
|
reasoning_effort: str | None,
|
||||||
) -> None:
|
) -> None:
|
||||||
try:
|
try:
|
||||||
# 成本换算(M2 §6): 成功行按单价换算;缓存命中 0.0(未产生新调用);
|
# 成本换算(M2 §6): 成功行按单价换算;缓存命中 0.0(未产生新调用);
|
||||||
@@ -370,6 +456,7 @@ class TelemetryEmitter:
|
|||||||
# 保证接受,而遥测写失败只降级成一条 warning——不会当场炸,只会让
|
# 保证接受,而遥测写失败只降级成一条 warning——不会当场炸,只会让
|
||||||
# Postgres 那一路悄悄少一列数据
|
# Postgres 那一路悄悄少一列数据
|
||||||
thinking_observation=_normalize_observation(thinking_observation),
|
thinking_observation=_normalize_observation(thinking_observation),
|
||||||
|
reasoning_effort=reasoning_effort,
|
||||||
)
|
)
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
raise
|
raise
|
||||||
|
|||||||
@@ -455,6 +455,8 @@ class OcrClient:
|
|||||||
latency_ms=latency_ms,
|
latency_ms=latency_ms,
|
||||||
response=response,
|
response=response,
|
||||||
error=error_text,
|
error=error_text,
|
||||||
|
# OCR 走 MonkeyOCR 自有端点,没有推理参数可言(理由同 embedding)
|
||||||
|
reasoning_applies=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
|
|||||||
@@ -271,7 +271,7 @@ class TelemetryStatusProvider(Protocol):
|
|||||||
|
|
||||||
@runtime_checkable
|
@runtime_checkable
|
||||||
class TelemetryRecorder(Protocol):
|
class TelemetryRecorder(Protocol):
|
||||||
"""遥测后端;25 字段冻结(M1 设计 §4.4 + issue #3/#4/#11/#16),唯一调用点是 TelemetryEmitter。
|
"""遥测后端;26 字段冻结(M1 设计 §4.4 + issue #3/#4/#11/#16/#20),唯一调用点是 TelemetryEmitter。
|
||||||
|
|
||||||
新增参数不设默认值: 库外无第三方实现者(三项目迁移时删除了各自的同名
|
新增参数不设默认值: 库外无第三方实现者(三项目迁移时删除了各自的同名
|
||||||
Protocol),完整签名的成本为零,而少写一列会被 emitter 的降级吞成 warning。
|
Protocol),完整签名的成本为零,而少写一列会被 emitter 的降级吞成 warning。
|
||||||
@@ -281,6 +281,9 @@ class TelemetryRecorder(Protocol):
|
|||||||
`thinking_observation` 同理: emitter 已把 `ThinkingObservation` 取成 `.value`
|
`thinking_observation` 同理: emitter 已把 `ThinkingObservation` 取成 `.value`
|
||||||
的裸 `str`(`StrEnum` 是 `str` 子类,而 asyncpg 的参数编码对子类不保证接受,
|
的裸 `str`(`StrEnum` 是 `str` 子类,而 asyncpg 的参数编码对子类不保证接受,
|
||||||
遥测写失败又只降级成 warning——PG 那一路会静默少一列数据)。
|
遥测写失败又只降级成 warning——PG 那一路会静默少一列数据)。
|
||||||
|
`reasoning_effort` 同一先例(issue #20): emitter 已把 `Effort` 取成 `.value`
|
||||||
|
的裸 `str`,`None` 表示调用方没表态——它与 `'none'`(明确要求不推理)不可折叠。
|
||||||
|
|
||||||
recorder 只负责落库,不做任何语义判断,与 `sampling` 列由
|
recorder 只负责落库,不做任何语义判断,与 `sampling` 列由
|
||||||
`canonical_sampling_json()` 在 emitter 侧定型是同一先例。
|
`canonical_sampling_json()` 在 emitter 侧定型是同一先例。
|
||||||
"""
|
"""
|
||||||
@@ -313,4 +316,5 @@ class TelemetryRecorder(Protocol):
|
|||||||
tenant_id: str,
|
tenant_id: str,
|
||||||
meta: str,
|
meta: str,
|
||||||
thinking_observation: str,
|
thinking_observation: str,
|
||||||
|
reasoning_effort: str | None,
|
||||||
) -> None: ...
|
) -> None: ...
|
||||||
|
|||||||
@@ -5,8 +5,8 @@
|
|||||||
多处各存一份必然漂移,而漂移的表现是"下游照打印的 SQL 建完表,库仍报缺列"。
|
多处各存一份必然漂移,而漂移的表现是"下游照打印的 SQL 建完表,库仍报缺列"。
|
||||||
|
|
||||||
**`COLUMNS` 是 INSERT 字段序,不是物理列序**: 数据库自填的 `created_at` 不在其中(它带
|
**`COLUMNS` 是 INSERT 字段序,不是物理列序**: 数据库自填的 `created_at` 不在其中(它带
|
||||||
`DEFAULT now()` / `datetime('now')`,库从不显式写它)。物理表列 = 25 个 INSERT 字段 +
|
`DEFAULT now()` / `datetime('now')`,库从不显式写它)。物理表列 = 26 个 INSERT 字段 +
|
||||||
`created_at` = 26;列数断言一律按物理列数写,两套口径混用是最易错处。
|
`created_at` = 27;列数断言一律按物理列数写,两套口径混用是最易错处。
|
||||||
|
|
||||||
本模块只依赖标准库: `telemetry/` 与 `backends/`、`transports/`、`structured/` 同层且
|
本模块只依赖标准库: `telemetry/` 与 `backends/`、`transports/`、`structured/` 同层且
|
||||||
互不依赖(import-linter 契约执法)。
|
互不依赖(import-linter 契约执法)。
|
||||||
@@ -51,7 +51,8 @@ CREATE TABLE IF NOT EXISTS llm_calls (
|
|||||||
reasoning_tokens INTEGER,
|
reasoning_tokens INTEGER,
|
||||||
tenant_id TEXT NOT NULL DEFAULT '',
|
tenant_id TEXT NOT NULL DEFAULT '',
|
||||||
meta TEXT NOT NULL DEFAULT '{}',
|
meta TEXT NOT NULL DEFAULT '{}',
|
||||||
thinking_observation TEXT
|
thinking_observation TEXT,
|
||||||
|
reasoning_effort TEXT
|
||||||
);
|
);
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -82,7 +83,8 @@ CREATE TABLE IF NOT EXISTS llm_calls (
|
|||||||
reasoning_tokens INTEGER,
|
reasoning_tokens INTEGER,
|
||||||
tenant_id TEXT NOT NULL DEFAULT '',
|
tenant_id TEXT NOT NULL DEFAULT '',
|
||||||
meta JSONB NOT NULL DEFAULT '{}'::jsonb,
|
meta JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||||
thinking_observation TEXT
|
thinking_observation TEXT,
|
||||||
|
reasoning_effort TEXT
|
||||||
);
|
);
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -100,6 +102,9 @@ SQLITE_BACKFILL = (
|
|||||||
# 可空: 补列之前的行没有裁定结果,NULL 如实表达"这行根本没记过这件事",
|
# 可空: 补列之前的行没有裁定结果,NULL 如实表达"这行根本没记过这件事",
|
||||||
# 与哨兵串 'unknown'(库确实裁过但判不出来)是两回事,不得混同
|
# 与哨兵串 'unknown'(库确实裁过但判不出来)是两回事,不得混同
|
||||||
("thinking_observation", "TEXT"),
|
("thinking_observation", "TEXT"),
|
||||||
|
# 同样可空,但这里 NULL 表达的是"调用方没表态"(issue #20): 它与 'none'
|
||||||
|
# (明确要求不推理)是两回事,折叠成任一档都等于替上游声称了它没说过的事
|
||||||
|
("reasoning_effort", "TEXT"),
|
||||||
)
|
)
|
||||||
|
|
||||||
# PG 补列的列定义。语句由此派生成两份文本(见下),使"库内执行的那份"与"打印给
|
# PG 补列的列定义。语句由此派生成两份文本(见下),使"库内执行的那份"与"打印给
|
||||||
@@ -114,6 +119,7 @@ _PG_BACKFILL_DECLS = (
|
|||||||
("meta", "JSONB NOT NULL DEFAULT '{}'::jsonb"),
|
("meta", "JSONB NOT NULL DEFAULT '{}'::jsonb"),
|
||||||
# 可空,理由同 SQLITE_BACKFILL 同名项
|
# 可空,理由同 SQLITE_BACKFILL 同名项
|
||||||
("thinking_observation", "TEXT"),
|
("thinking_observation", "TEXT"),
|
||||||
|
("reasoning_effort", "TEXT"),
|
||||||
)
|
)
|
||||||
|
|
||||||
# 新列排在 created_at 之后: 与旧表 ALTER 追加的位置一致(见 SQLITE_BACKFILL 同款注释)。
|
# 新列排在 created_at 之后: 与旧表 ALTER 追加的位置一致(见 SQLITE_BACKFILL 同款注释)。
|
||||||
@@ -151,6 +157,7 @@ COLUMNS = (
|
|||||||
"tenant_id",
|
"tenant_id",
|
||||||
"meta",
|
"meta",
|
||||||
"thinking_observation",
|
"thinking_observation",
|
||||||
|
"reasoning_effort",
|
||||||
)
|
)
|
||||||
|
|
||||||
_COLUMN_SET = frozenset(COLUMNS)
|
_COLUMN_SET = frozenset(COLUMNS)
|
||||||
|
|||||||
@@ -143,7 +143,7 @@ class SQLiteRecorder:
|
|||||||
logger.warning("SQLite 遥测补列失败(写入将逐行降级): {}", exc)
|
logger.warning("SQLite 遥测补列失败(写入将逐行降级): {}", exc)
|
||||||
|
|
||||||
async def record_llm_call(self, **fields: object) -> None:
|
async def record_llm_call(self, **fields: object) -> None:
|
||||||
"""写一行遥测;字段集合即 25 字段冻结签名(ports.TelemetryRecorder)。
|
"""写一行遥测;字段集合即 26 字段冻结签名(ports.TelemetryRecorder)。
|
||||||
|
|
||||||
取值按 `self._columns`(manual 档可能已被裁剪),与 `self._insert` 的
|
取值按 `self._columns`(manual 档可能已被裁剪),与 `self._insert` 的
|
||||||
占位符同序——两者必须一起改,分开改就是把值写进错位的列。
|
占位符同序——两者必须一起改,分开改就是把值写进错位的列。
|
||||||
|
|||||||
@@ -57,6 +57,7 @@ _EXPECTED_COLUMNS = [
|
|||||||
"tenant_id",
|
"tenant_id",
|
||||||
"meta",
|
"meta",
|
||||||
"thinking_observation",
|
"thinking_observation",
|
||||||
|
"reasoning_effort",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -127,6 +128,8 @@ async def _record_minimal(
|
|||||||
"meta": "{}",
|
"meta": "{}",
|
||||||
# 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str
|
# 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str
|
||||||
"thinking_observation": "unknown",
|
"thinking_observation": "unknown",
|
||||||
|
# 同理: `Effort` 归一成裸 str,不表态则是 None(与 'low' 必须分得开)
|
||||||
|
"reasoning_effort": None,
|
||||||
}
|
}
|
||||||
fields.update(overrides)
|
fields.update(overrides)
|
||||||
await recorder.record_llm_call(**fields)
|
await recorder.record_llm_call(**fields)
|
||||||
@@ -225,11 +228,12 @@ class TestObservabilityColumns:
|
|||||||
await _record_minimal(
|
await _record_minimal(
|
||||||
recorder, call_id="samp", sampling='{"seed": 42, "temperature": 0}'
|
recorder, call_id="samp", sampling='{"seed": 42, "temperature": 0}'
|
||||||
)
|
)
|
||||||
|
await _record_minimal(recorder, call_id="tier", reasoning_effort="low")
|
||||||
rows = await _fetch(
|
rows = await _fetch(
|
||||||
sandbox.dsn,
|
sandbox.dsn,
|
||||||
"SELECT call_id, cached_prompt_tokens, model_reported, sampling FROM llm_calls "
|
"SELECT call_id, cached_prompt_tokens, model_reported, sampling, "
|
||||||
"WHERE call_id = ANY($1::text[])",
|
"reasoning_effort FROM llm_calls WHERE call_id = ANY($1::text[])",
|
||||||
["hit", "zero", "model", "samp"],
|
["hit", "zero", "model", "samp", "tier"],
|
||||||
)
|
)
|
||||||
by_id = {r["call_id"]: r for r in rows}
|
by_id = {r["call_id"]: r for r in rows}
|
||||||
assert by_id["hit"]["cached_prompt_tokens"] == 64
|
assert by_id["hit"]["cached_prompt_tokens"] == 64
|
||||||
@@ -239,6 +243,10 @@ class TestObservabilityColumns:
|
|||||||
# issue #4: PG 侧也须验非空 sampling 能读回原值(不只是列存在)
|
# issue #4: PG 侧也须验非空 sampling 能读回原值(不只是列存在)
|
||||||
assert json.loads(by_id["samp"]["sampling"]) == {"seed": 42, "temperature": 0}
|
assert json.loads(by_id["samp"]["sampling"]) == {"seed": 42, "temperature": 0}
|
||||||
assert by_id["hit"]["sampling"] is None
|
assert by_id["hit"]["sampling"] is None
|
||||||
|
# issue #20: PG 侧同样要验档位读得回来——emitter 落的是裸 str,
|
||||||
|
# 若哪天回退成 `Effort` 实例,asyncpg 编码不保证接受,写入会整行降级
|
||||||
|
assert by_id["tier"]["reasoning_effort"] == "low"
|
||||||
|
assert by_id["hit"]["reasoning_effort"] is None # 不表态是 NULL
|
||||||
finally:
|
finally:
|
||||||
await recorder.aclose()
|
await recorder.aclose()
|
||||||
|
|
||||||
@@ -580,11 +588,13 @@ _PRE_TENANT_INSERT = (
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# `_PRE_TENANT_DDL` 的物理列(23 个): 由 `_EXPECTED_COLUMNS` 去掉此后新增的三列
|
# `_PRE_TENANT_DDL` 的物理列(23 个): 由 `_EXPECTED_COLUMNS` 去掉此后新增的四列
|
||||||
# 派生而非另抄一份——两份常量必然漂移,而漂移的表现是"manual 档没补列"这条断言假绿。
|
# 派生而非另抄一份——两份常量必然漂移,而漂移的表现是"manual 档没补列"这条断言假绿。
|
||||||
# 去掉后的顺序与 DDL 逐字一致(这三列在 DDL 里本就排在末尾)。
|
# 去掉后的顺序与 DDL 逐字一致(这四列在 DDL 里本就排在末尾)。
|
||||||
_PRE_TENANT_COLUMNS = [
|
_PRE_TENANT_COLUMNS = [
|
||||||
c for c in _EXPECTED_COLUMNS if c not in ("tenant_id", "meta", "thinking_observation")
|
c
|
||||||
|
for c in _EXPECTED_COLUMNS
|
||||||
|
if c not in ("tenant_id", "meta", "thinking_observation", "reasoning_effort")
|
||||||
]
|
]
|
||||||
|
|
||||||
# 回读要逐列比对的字段: 物理列去掉库从不显式写的 created_at,恰好 22 个
|
# 回读要逐列比对的字段: 物理列去掉库从不显式写的 created_at,恰好 22 个
|
||||||
@@ -693,7 +703,7 @@ class TestCallerDimensionsAcceptance:
|
|||||||
"WHERE table_schema = $1 AND table_name = 'llm_calls' ORDER BY ordinal_position",
|
"WHERE table_schema = $1 AND table_name = 'llm_calls' ORDER BY ordinal_position",
|
||||||
schema,
|
schema,
|
||||||
)
|
)
|
||||||
# 22 → 25 个 recorder 字段(加 created_at 共 26 个物理列),且新列追加在末尾
|
# 22 → 26 个 recorder 字段(加 created_at 共 27 个物理列),且新列追加在末尾
|
||||||
assert [r["column_name"] for r in cols] == _EXPECTED_COLUMNS
|
assert [r["column_name"] for r in cols] == _EXPECTED_COLUMNS
|
||||||
rows = await _fetch(
|
rows = await _fetch(
|
||||||
schema_dsn,
|
schema_dsn,
|
||||||
@@ -869,7 +879,7 @@ class TestManualSchemaModeAcceptance:
|
|||||||
"""22 字段旧表 + manual: 列一个不加,行照常落库,缺的三维度静默不写。
|
"""22 字段旧表 + manual: 列一个不加,行照常落库,缺的三维度静默不写。
|
||||||
|
|
||||||
与 `test_pre_tenant_table_gains_columns_and_old_rows_stay_auditable` 恰成对照:
|
与 `test_pre_tenant_table_gains_columns_and_old_rows_stay_auditable` 恰成对照:
|
||||||
同一张表、同一份负载,只有 `auto_migrate` 不同,列数就必须是 23 与 26 之别。
|
同一张表、同一份负载,只有 `auto_migrate` 不同,列数就必须是 23 与 27 之别。
|
||||||
"""
|
"""
|
||||||
schema_dsn, schema = pre_tenant_schema
|
schema_dsn, schema = pre_tenant_schema
|
||||||
recorder = _recorder(schema_dsn, auto_migrate=False)
|
recorder = _recorder(schema_dsn, auto_migrate=False)
|
||||||
@@ -898,8 +908,11 @@ class TestManualSchemaModeAcceptance:
|
|||||||
assert [m for m in captured_warnings if "补列失败" in m] == []
|
assert [m for m in captured_warnings if "补列失败" in m] == []
|
||||||
notices = [m for m in captured_warnings if "auto_migrate=False" in m]
|
notices = [m for m in captured_warnings if "auto_migrate=False" in m]
|
||||||
assert len(notices) == 1 # 准备期一次讲清,不逐行刷屏
|
assert len(notices) == 1 # 准备期一次讲清,不逐行刷屏
|
||||||
# 逐字钉住三个维度: 前缀断言会让将来漏进告警的新列照样绿
|
# 逐字钉住四个维度: 前缀断言会让将来漏进告警的新列照样绿
|
||||||
assert "以下维度不会被记录: tenant_id, meta, thinking_observation。" in notices[0]
|
assert (
|
||||||
|
"以下维度不会被记录: tenant_id, meta, thinking_observation, reasoning_effort。"
|
||||||
|
in notices[0]
|
||||||
|
)
|
||||||
finally:
|
finally:
|
||||||
await recorder.aclose()
|
await recorder.aclose()
|
||||||
|
|
||||||
@@ -925,8 +938,11 @@ class TestManualSchemaModeAcceptance:
|
|||||||
assert recorder.telemetry_status.degraded is False
|
assert recorder.telemetry_status.degraded is False
|
||||||
notices = [m for m in captured_warnings if "auto_migrate=False" in m]
|
notices = [m for m in captured_warnings if "auto_migrate=False" in m]
|
||||||
assert len(notices) == 1 # 准备期一次,第二行不再重复
|
assert len(notices) == 1 # 准备期一次,第二行不再重复
|
||||||
# 逐字钉住三个维度: 前缀断言会让将来漏进告警的新列照样绿
|
# 逐字钉住四个维度: 前缀断言会让将来漏进告警的新列照样绿
|
||||||
assert "以下维度不会被记录: tenant_id, meta, thinking_observation。" in notices[0]
|
assert (
|
||||||
|
"以下维度不会被记录: tenant_id, meta, thinking_observation, reasoning_effort。"
|
||||||
|
in notices[0]
|
||||||
|
)
|
||||||
# 提示里的 SQL 必须可直接粘贴执行,而不是只报个列名
|
# 提示里的 SQL 必须可直接粘贴执行,而不是只报个列名
|
||||||
assert (
|
assert (
|
||||||
"ALTER TABLE llm_calls ADD COLUMN tenant_id TEXT NOT NULL DEFAULT '';" in notices[0]
|
"ALTER TABLE llm_calls ADD COLUMN tenant_id TEXT NOT NULL DEFAULT '';" in notices[0]
|
||||||
@@ -984,7 +1000,7 @@ class TestPublishedSchemaScript:
|
|||||||
|
|
||||||
await _execute_script(fresh_dsn, script)
|
await _execute_script(fresh_dsn, script)
|
||||||
actual = [r["column_name"] for r in await _fetch(fresh_dsn, _PHYSICAL_COLUMNS_SQL, schema)]
|
actual = [r["column_name"] for r in await _fetch(fresh_dsn, _PHYSICAL_COLUMNS_SQL, schema)]
|
||||||
# 物理列 = 25 个 INSERT 字段 + 库从不显式写的 created_at;对着库常量比,不另抄一份
|
# 物理列 = 26 个 INSERT 字段 + 库从不显式写的 created_at;对着库常量比,不另抄一份
|
||||||
assert set(actual) == set(COLUMNS) | {"created_at"}
|
assert set(actual) == set(COLUMNS) | {"created_at"}
|
||||||
# 列序也不许漂: 新列必须排在 created_at 之后,否则新建库与 ALTER 升级的列序分叉
|
# 列序也不许漂: 新列必须排在 created_at 之后,否则新建库与 ALTER 升级的列序分叉
|
||||||
assert actual == _EXPECTED_COLUMNS
|
assert actual == _EXPECTED_COLUMNS
|
||||||
|
|||||||
@@ -584,6 +584,7 @@ class TestTelemetryCapDoesNotPoisonTheCacheKey:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
# 截断确实发生了(否则本用例恒真)
|
# 截断确实发生了(否则本用例恒真)
|
||||||
logged = json.loads(rec.rows[0]["messages"])
|
logged = json.loads(rec.rows[0]["messages"])
|
||||||
|
|||||||
@@ -123,6 +123,7 @@ async def _recorded_cost(result, source):
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=response,
|
response=response,
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
return recorder.rows[0]["cost"]
|
return recorder.rows[0]["cost"]
|
||||||
|
|
||||||
|
|||||||
@@ -275,7 +275,9 @@ class TestTelemetryRecorderSignature:
|
|||||||
params = inspect.signature(TelemetryRecorder.record_llm_call).parameters
|
params = inspect.signature(TelemetryRecorder.record_llm_call).parameters
|
||||||
assert {"tenant_id", "meta"} <= set(params)
|
assert {"tenant_id", "meta"} <= set(params)
|
||||||
|
|
||||||
@pytest.mark.parametrize("name", ["tenant_id", "meta", "thinking_observation"])
|
@pytest.mark.parametrize(
|
||||||
|
"name", ["tenant_id", "meta", "thinking_observation", "reasoning_effort"]
|
||||||
|
)
|
||||||
def test_caller_dimensions_have_no_default(self, name):
|
def test_caller_dimensions_have_no_default(self, name):
|
||||||
import inspect
|
import inspect
|
||||||
|
|
||||||
|
|||||||
@@ -177,6 +177,7 @@ class TestEmitterCost:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] == pytest.approx(7.2)
|
assert rec.rows[0]["cost"] == pytest.approx(7.2)
|
||||||
|
|
||||||
@@ -196,6 +197,7 @@ class TestEmitterCost:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=None,
|
response=None,
|
||||||
error="TransientError: boom",
|
error="TransientError: boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] is None
|
assert rec.rows[0]["cost"] is None
|
||||||
|
|
||||||
@@ -209,6 +211,7 @@ class TestEmitterCost:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(model="mystery"),
|
response=_resp(model="mystery"),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] is None
|
assert rec.rows[0]["cost"] is None
|
||||||
|
|
||||||
@@ -223,5 +226,6 @@ class TestEmitterCost:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] is None
|
assert rec.rows[0]["cost"] is None
|
||||||
|
|||||||
+244
-15
@@ -25,6 +25,7 @@ from polygateway.types import (
|
|||||||
BackpressurePolicy,
|
BackpressurePolicy,
|
||||||
BreakerConfig,
|
BreakerConfig,
|
||||||
ChatRequest,
|
ChatRequest,
|
||||||
|
Effort,
|
||||||
EmbeddingTransportResult,
|
EmbeddingTransportResult,
|
||||||
GlobalLimits,
|
GlobalLimits,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
@@ -63,6 +64,7 @@ _EXPECTED_COLUMNS = [
|
|||||||
"tenant_id",
|
"tenant_id",
|
||||||
"meta",
|
"meta",
|
||||||
"thinking_observation",
|
"thinking_observation",
|
||||||
|
"reasoning_effort",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -132,6 +134,8 @@ async def _record_minimal(recorder, call_id="c1", **overrides):
|
|||||||
"meta": "{}",
|
"meta": "{}",
|
||||||
# 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str
|
# 同样已由 emitter 归一化: 枚举取 .value 后才下沉,recorder 只见裸 str
|
||||||
"thinking_observation": "unknown",
|
"thinking_observation": "unknown",
|
||||||
|
# 同理: `Effort` 归一成裸 str,不表态则是 None(与 'low' 必须分得开)
|
||||||
|
"reasoning_effort": None,
|
||||||
}
|
}
|
||||||
fields.update(overrides)
|
fields.update(overrides)
|
||||||
await recorder.record_llm_call(**fields)
|
await recorder.record_llm_call(**fields)
|
||||||
@@ -177,16 +181,17 @@ _FROZEN_SQLITE_INSERT = (
|
|||||||
"INSERT OR IGNORE INTO llm_calls (call_id, parent_call_id, session_id, model, provider, "
|
"INSERT OR IGNORE INTO llm_calls (call_id, parent_call_id, session_id, model, provider, "
|
||||||
"source_name, messages, response, thinking, prompt_tokens, completion_tokens, usage_source, "
|
"source_name, messages, response, thinking, prompt_tokens, completion_tokens, usage_source, "
|
||||||
"latency_ms, ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, "
|
"latency_ms, ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, "
|
||||||
"model_reported, sampling, reasoning_tokens, tenant_id, meta, thinking_observation) "
|
"model_reported, sampling, reasoning_tokens, tenant_id, meta, thinking_observation, "
|
||||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
"reasoning_effort) "
|
||||||
|
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
||||||
)
|
)
|
||||||
_FROZEN_PG_INSERT = (
|
_FROZEN_PG_INSERT = (
|
||||||
"INSERT INTO llm_calls (call_id, parent_call_id, session_id, model, provider, source_name, "
|
"INSERT INTO llm_calls (call_id, parent_call_id, session_id, model, provider, source_name, "
|
||||||
"messages, response, thinking, prompt_tokens, completion_tokens, usage_source, latency_ms, "
|
"messages, response, thinking, prompt_tokens, completion_tokens, usage_source, latency_ms, "
|
||||||
"ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, model_reported, "
|
"ttft_ms, max_inter_token_ms, cache_hit, error, cost, cached_prompt_tokens, model_reported, "
|
||||||
"sampling, reasoning_tokens, tenant_id, meta, thinking_observation) "
|
"sampling, reasoning_tokens, tenant_id, meta, thinking_observation, reasoning_effort) "
|
||||||
"VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, "
|
"VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, "
|
||||||
"$19, $20, $21, $22, $23, $24, $25) "
|
"$19, $20, $21, $22, $23, $24, $25, $26) "
|
||||||
# 无冲突目标(issue #13 Task 2): 带 `(call_id)` 的版本在按 created_at 分区、
|
# 无冲突目标(issue #13 Task 2): 带 `(call_id)` 的版本在按 created_at 分区、
|
||||||
# 主键为 (call_id, created_at) 的表上匹配不到约束,PG 直接拒收整条写入
|
# 主键为 (call_id, created_at) 的表上匹配不到约束,PG 直接拒收整条写入
|
||||||
"ON CONFLICT DO NOTHING"
|
"ON CONFLICT DO NOTHING"
|
||||||
@@ -212,7 +217,7 @@ class TestSchemaModule:
|
|||||||
|
|
||||||
# COLUMNS 是 INSERT 字段序,不含数据库自填的 created_at
|
# COLUMNS 是 INSERT 字段序,不含数据库自填的 created_at
|
||||||
assert list(COLUMNS) == [c for c in _EXPECTED_COLUMNS if c != "created_at"]
|
assert list(COLUMNS) == [c for c in _EXPECTED_COLUMNS if c != "created_at"]
|
||||||
assert len(COLUMNS) == 25
|
assert len(COLUMNS) == 26
|
||||||
# 两端 DDL 的列出现顺序 == 物理列序(created_at 在第 19 位)
|
# 两端 DDL 的列出现顺序 == 物理列序(created_at 在第 19 位)
|
||||||
for ddl in (SQLITE_DDL, PG_DDL):
|
for ddl in (SQLITE_DDL, PG_DDL):
|
||||||
assert _first_occurrence_order(ddl, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS
|
assert _first_occurrence_order(ddl, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS
|
||||||
@@ -226,8 +231,8 @@ class TestSchemaModule:
|
|||||||
"ALTER TABLE llm_calls ADD COLUMN cached_prompt_tokens INTEGER",
|
"ALTER TABLE llm_calls ADD COLUMN cached_prompt_tokens INTEGER",
|
||||||
)
|
)
|
||||||
assert PG_BACKFILL[-1] == (
|
assert PG_BACKFILL[-1] == (
|
||||||
"thinking_observation",
|
"reasoning_effort",
|
||||||
"ALTER TABLE llm_calls ADD COLUMN thinking_observation TEXT",
|
"ALTER TABLE llm_calls ADD COLUMN reasoning_effort TEXT",
|
||||||
)
|
)
|
||||||
assert all("IF NOT EXISTS" not in stmt for _, stmt in PG_BACKFILL)
|
assert all("IF NOT EXISTS" not in stmt for _, stmt in PG_BACKFILL)
|
||||||
|
|
||||||
@@ -275,7 +280,7 @@ class TestSchemaModule:
|
|||||||
pg = telemetry_schema_sql("postgres")
|
pg = telemetry_schema_sql("postgres")
|
||||||
lite = telemetry_schema_sql("sqlite")
|
lite = telemetry_schema_sql("sqlite")
|
||||||
for script in (pg, lite):
|
for script in (pg, lite):
|
||||||
# 25 个 INSERT 字段 + created_at 全在,且首次出现顺序与建表 DDL 一致
|
# 26 个 INSERT 字段 + created_at 全在,且首次出现顺序与建表 DDL 一致
|
||||||
assert _first_occurrence_order(script, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS
|
assert _first_occurrence_order(script, _EXPECTED_COLUMNS) == _EXPECTED_COLUMNS
|
||||||
assert "CREATE TABLE IF NOT EXISTS llm_calls" in script
|
assert "CREATE TABLE IF NOT EXISTS llm_calls" in script
|
||||||
# 人执行的那份必须幂等: PG 用 ADD COLUMN IF NOT EXISTS(与库内那份有意不同)
|
# 人执行的那份必须幂等: PG 用 ADD COLUMN IF NOT EXISTS(与库内那份有意不同)
|
||||||
@@ -313,7 +318,7 @@ class TestBackendColumnParity:
|
|||||||
"""新列只能追加在末尾: 旧表经 ALTER 补列必落末尾,插在中间会让两条路径分叉。"""
|
"""新列只能追加在末尾: 旧表经 ALTER 补列必落末尾,插在中间会让两条路径分叉。"""
|
||||||
from polygateway.telemetry.schema import COLUMNS
|
from polygateway.telemetry.schema import COLUMNS
|
||||||
|
|
||||||
assert COLUMNS[-3:] == ("tenant_id", "meta", "thinking_observation")
|
assert COLUMNS[-4:] == ("tenant_id", "meta", "thinking_observation", "reasoning_effort")
|
||||||
|
|
||||||
|
|
||||||
class TestSQLiteRecorder:
|
class TestSQLiteRecorder:
|
||||||
@@ -405,6 +410,27 @@ class TestSQLiteRecorder:
|
|||||||
assert rows["t-absent"] == "absent" # 观测到"确实没推理",与"看不出来"不是一回事
|
assert rows["t-absent"] == "absent" # 观测到"确实没推理",与"看不出来"不是一回事
|
||||||
assert rows["t-unknown"] == "unknown"
|
assert rows["t-unknown"] == "unknown"
|
||||||
|
|
||||||
|
async def test_reasoning_effort_column_round_trips(self, tmp_path):
|
||||||
|
"""issue #20: 实际档位落库,事后才分得清"这一行跑在哪档"。
|
||||||
|
|
||||||
|
断言裸串而非枚举,理由与 `thinking_observation` 逐字相同: `StrEnum` 是
|
||||||
|
`str` 子类,而 asyncpg 对子类编码不保证接受,遥测写失败只降级一条 warning
|
||||||
|
——PG 那一路会静默少一列,SQLite 本地全绿也发现不了。
|
||||||
|
"""
|
||||||
|
recorder = SQLiteRecorder(tmp_path / "t.db", auto_migrate=True)
|
||||||
|
await _record_minimal(recorder, call_id="e-low", reasoning_effort="low")
|
||||||
|
await _record_minimal(recorder, call_id="e-max", reasoning_effort="max")
|
||||||
|
await _record_minimal(recorder, call_id="e-silent")
|
||||||
|
recorder.close()
|
||||||
|
rows = dict(
|
||||||
|
sqlite3.connect(tmp_path / "t.db")
|
||||||
|
.execute("SELECT call_id, reasoning_effort FROM llm_calls")
|
||||||
|
.fetchall()
|
||||||
|
)
|
||||||
|
assert rows["e-low"] == "low"
|
||||||
|
assert rows["e-max"] == "max"
|
||||||
|
assert rows["e-silent"] is None # 不表态是 NULL,与任何一档都分得开
|
||||||
|
|
||||||
async def test_sampling_column_round_trips(self, tmp_path):
|
async def test_sampling_column_round_trips(self, tmp_path):
|
||||||
"""issue #4: 采样参数落库,否则事后无法证明某批数据跑在什么温度下。"""
|
"""issue #4: 采样参数落库,否则事后无法证明某批数据跑在什么温度下。"""
|
||||||
recorder = SQLiteRecorder(tmp_path / "t.db", auto_migrate=True)
|
recorder = SQLiteRecorder(tmp_path / "t.db", auto_migrate=True)
|
||||||
@@ -568,7 +594,7 @@ class TestSQLiteCallerDimensionsAcceptance:
|
|||||||
|
|
||||||
conn = sqlite3.connect(db)
|
conn = sqlite3.connect(db)
|
||||||
cols = [r[1] for r in conn.execute("PRAGMA table_info(llm_calls)")]
|
cols = [r[1] for r in conn.execute("PRAGMA table_info(llm_calls)")]
|
||||||
assert cols == _EXPECTED_COLUMNS # 22 → 25 个 recorder 字段(+ created_at 共 26 物理列)
|
assert cols == _EXPECTED_COLUMNS # 22 → 26 个 recorder 字段(+ created_at 共 27 物理列)
|
||||||
rows = dict(conn.execute("SELECT call_id, tenant_id FROM llm_calls").fetchall())
|
rows = dict(conn.execute("SELECT call_id, tenant_id FROM llm_calls").fetchall())
|
||||||
assert rows["new-row"] == "tenant-a"
|
assert rows["new-row"] == "tenant-a"
|
||||||
assert rows["old-row"] == "" # 不是 None: NULL 会被 RLS 静默吞掉
|
assert rows["old-row"] == "" # 不是 None: NULL 会被 RLS 静默吞掉
|
||||||
@@ -610,7 +636,7 @@ class TestSQLiteCallerDimensionsAcceptance:
|
|||||||
|
|
||||||
stale = sqlite3.connect(db)
|
stale = sqlite3.connect(db)
|
||||||
assert [r[1] for r in stale.execute("PRAGMA table_info(llm_calls)")] == (
|
assert [r[1] for r in stale.execute("PRAGMA table_info(llm_calls)")] == (
|
||||||
_EXPECTED_COLUMNS[:-3]
|
_EXPECTED_COLUMNS[:-4]
|
||||||
) # 补列确实没成功,用例不是在只读库上空转
|
) # 补列确实没成功,用例不是在只读库上空转
|
||||||
|
|
||||||
|
|
||||||
@@ -618,7 +644,7 @@ class TestSQLiteSchemaMode:
|
|||||||
"""issue #13: `auto_migrate` 两档——auto 保持自动补列,manual 只裁剪写入不发 DDL。
|
"""issue #13: `auto_migrate` 两档——auto 保持自动补列,manual 只裁剪写入不发 DDL。
|
||||||
|
|
||||||
列数断言一律按**物理列数**写: 旧表 22 个 INSERT 字段 + `created_at` = 23,
|
列数断言一律按**物理列数**写: 旧表 22 个 INSERT 字段 + `created_at` = 23,
|
||||||
补齐后 25 + `created_at` = 26。混用 INSERT 字段数与物理列数是本处最易错的地方。
|
补齐后 26 + `created_at` = 27。混用 INSERT 字段数与物理列数是本处最易错的地方。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def _physical_columns(self, db: Path) -> list[str]:
|
def _physical_columns(self, db: Path) -> list[str]:
|
||||||
@@ -657,7 +683,7 @@ class TestSQLiteSchemaMode:
|
|||||||
assert "ALTER TABLE" in message # 给出可直接执行的补列 SQL
|
assert "ALTER TABLE" in message # 给出可直接执行的补列 SQL
|
||||||
|
|
||||||
async def test_auto_mode_still_upgrades_the_legacy_table(self, tmp_path):
|
async def test_auto_mode_still_upgrades_the_legacy_table(self, tmp_path):
|
||||||
"""auto + 同款旧表: 现状回归,补列后物理列数 23 → 26。"""
|
"""auto + 同款旧表: 现状回归,补列后物理列数 23 → 27。"""
|
||||||
db = tmp_path / "auto_legacy.db"
|
db = tmp_path / "auto_legacy.db"
|
||||||
_make_pre_tenant_db(db)
|
_make_pre_tenant_db(db)
|
||||||
|
|
||||||
@@ -666,10 +692,10 @@ class TestSQLiteSchemaMode:
|
|||||||
recorder.close()
|
recorder.close()
|
||||||
|
|
||||||
assert self._physical_columns(db) == _EXPECTED_COLUMNS
|
assert self._physical_columns(db) == _EXPECTED_COLUMNS
|
||||||
assert len(self._physical_columns(db)) == 26
|
assert len(self._physical_columns(db)) == 27
|
||||||
|
|
||||||
async def test_manual_mode_still_creates_a_fresh_table(self, tmp_path):
|
async def test_manual_mode_still_creates_a_fresh_table(self, tmp_path):
|
||||||
"""manual 只管 ALTER,不管 CREATE: 全新库照建,26 个物理列齐全(设计 §4.2)。"""
|
"""manual 只管 ALTER,不管 CREATE: 全新库照建,27 个物理列齐全(设计 §4.2)。"""
|
||||||
db = tmp_path / "manual_fresh.db"
|
db = tmp_path / "manual_fresh.db"
|
||||||
recorder = SQLiteRecorder(db, auto_migrate=False)
|
recorder = SQLiteRecorder(db, auto_migrate=False)
|
||||||
await _record_minimal(recorder, call_id="c-fresh", tenant_id="tenant-a")
|
await _record_minimal(recorder, call_id="c-fresh", tenant_id="tenant-a")
|
||||||
@@ -894,6 +920,7 @@ class TestPostgresBackfillDiscipline:
|
|||||||
"tenant_id",
|
"tenant_id",
|
||||||
"meta",
|
"meta",
|
||||||
"thinking_observation",
|
"thinking_observation",
|
||||||
|
"reasoning_effort",
|
||||||
]
|
]
|
||||||
|
|
||||||
def _recorder(self, conn):
|
def _recorder(self, conn):
|
||||||
@@ -1120,6 +1147,7 @@ class TestEmitterRecorderContract:
|
|||||||
latency_ms=42,
|
latency_ms=42,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert set(rec.rows[0]) == set(COLUMNS)
|
assert set(rec.rows[0]) == set(COLUMNS)
|
||||||
|
|
||||||
@@ -1137,6 +1165,7 @@ class TestEmitterRecorderContract:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=None,
|
response=None,
|
||||||
error="boom",
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
elif emit == "cache_hit":
|
elif emit == "cache_hit":
|
||||||
await emitter.emit_cache_hit(request=_REQ, response=_resp())
|
await emitter.emit_cache_hit(request=_REQ, response=_resp())
|
||||||
@@ -1165,6 +1194,7 @@ class TestEmitterThinkingObservation:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(thinking_observation=ThinkingObservation.OBSERVED),
|
response=_resp(thinking_observation=ThinkingObservation.OBSERVED),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
value = rec.rows[0]["thinking_observation"]
|
value = rec.rows[0]["thinking_observation"]
|
||||||
assert value == "observed"
|
assert value == "observed"
|
||||||
@@ -1187,6 +1217,7 @@ class TestEmitterThinkingObservation:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(thinking_observation="observed"),
|
response=_resp(thinking_observation="observed"),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert len(rec.rows) == 1, "整行被吞了"
|
assert len(rec.rows) == 1, "整行被吞了"
|
||||||
value = rec.rows[0]["thinking_observation"]
|
value = rec.rows[0]["thinking_observation"]
|
||||||
@@ -1212,6 +1243,7 @@ class TestEmitterThinkingObservation:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(thinking_observation="OBSERVED"), # 大小写不符即域外
|
response=_resp(thinking_observation="OBSERVED"), # 大小写不符即域外
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
finally:
|
finally:
|
||||||
logger.remove(sink_id)
|
logger.remove(sink_id)
|
||||||
@@ -1250,10 +1282,190 @@ class TestEmitterThinkingObservation:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=None,
|
response=None,
|
||||||
error="boom",
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["thinking_observation"] == "unknown"
|
assert rec.rows[0]["thinking_observation"] == "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmitterReasoningEffort:
|
||||||
|
"""issue #20: 每行记下这次调用**实际跑在哪档**,否则压测无从分组。
|
||||||
|
|
||||||
|
三个入口的取值口径**有意不同**,故逐个钉死: 只有 `emit_attempt` 手上有生效源,
|
||||||
|
它才谈得上"实际档";另两个入口没有选中源,源级档位无从谈起,只能记请求档。
|
||||||
|
与 `sampling` 列的现有做法同构。
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def test_attempt_records_the_tier_the_transport_applied(self):
|
||||||
|
"""`nearest` 映射后成功行记的是**映射后**的档,不是请求档。
|
||||||
|
|
||||||
|
请求 `medium`、模型只有 low/high/max 时二者分叉(实发 `low`)。emitter 若
|
||||||
|
"顺手"重算 `effective_effort`,记的就是一个从未发出过的档,而两个值在没开
|
||||||
|
映射的源上恒等——本地跑不开映射的源永远看不出这个错。
|
||||||
|
"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=ChatRequest(
|
||||||
|
messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.MEDIUM
|
||||||
|
),
|
||||||
|
source=_source(effort_fallback="nearest"),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=_resp(applied_effort=Effort.LOW),
|
||||||
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
value = rec.rows[0]["reasoning_effort"]
|
||||||
|
assert value == "low" # 不是 medium: 那一档从未发出去过
|
||||||
|
assert type(value) is str # 不是 Effort: 子类实例不得下沉到 recorder
|
||||||
|
|
||||||
|
async def test_failed_attempt_falls_back_to_the_requested_tier(self):
|
||||||
|
"""失败尝试没有响应,实际档不可知,记请求档并接受这层含义差别。
|
||||||
|
|
||||||
|
档位错误(resolve 的 Phase 2/4/5)根本没发 HTTP,却照样经
|
||||||
|
`RequestRejectedError` 走到这里——记的正是**被拒绝的那一档**,这对
|
||||||
|
"哪一档配错了" 是有用信号,不该被过滤掉。
|
||||||
|
"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(reasoning_effort=Effort.HIGH),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=None,
|
||||||
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
assert rec.rows[0]["reasoning_effort"] == "high"
|
||||||
|
|
||||||
|
async def test_failed_attempt_resolves_the_syntactic_sugar_too(self):
|
||||||
|
"""回落走 `effective_effort` 而非裸读字段: `enable_thinking` 也是表态。"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(enable_thinking=True),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=None,
|
||||||
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
assert rec.rows[0]["reasoning_effort"] == "auto"
|
||||||
|
|
||||||
|
async def test_cache_hit_records_the_request_tier_not_the_replayed_one(self):
|
||||||
|
"""命中行没有选中源,故记请求档;与 model/prompt_tokens 的回放口径相反。"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_cache_hit(
|
||||||
|
request=ChatRequest(
|
||||||
|
messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.MEDIUM
|
||||||
|
),
|
||||||
|
response=_resp(cache_hit=True, applied_effort=Effort.LOW),
|
||||||
|
)
|
||||||
|
assert rec.rows[0]["reasoning_effort"] == "medium"
|
||||||
|
|
||||||
|
async def test_terminal_failure_records_the_request_tier(self):
|
||||||
|
"""终态失败可能根本没选出源,源级档位无从谈起。"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_terminal_failure(
|
||||||
|
request=ChatRequest(
|
||||||
|
messages=[{"role": "user", "content": "hi"}], reasoning_effort=Effort.XHIGH
|
||||||
|
),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
error="dead",
|
||||||
|
)
|
||||||
|
value = rec.rows[0]["reasoning_effort"]
|
||||||
|
assert value == "xhigh"
|
||||||
|
assert type(value) is str
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("emit", ["attempt", "cache_hit", "terminal_failure"])
|
||||||
|
async def test_silence_lands_as_null(self, emit):
|
||||||
|
"""谁都没表态时落 `NULL`: `None` 与 `'low'` 必须分得开(设计 §6)。
|
||||||
|
|
||||||
|
库并不观测模型内部的默认档,记一个推定值等于把"没看见"说成"发生了"。
|
||||||
|
"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
emitter = TelemetryEmitter(rec, text_cap=None)
|
||||||
|
if emit == "attempt":
|
||||||
|
await emitter.emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=_resp(),
|
||||||
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
elif emit == "cache_hit":
|
||||||
|
await emitter.emit_cache_hit(request=_REQ, response=_resp(cache_hit=True))
|
||||||
|
else:
|
||||||
|
await emitter.emit_terminal_failure(
|
||||||
|
request=_REQ, call_id="c", latency_ms=1, error="dead"
|
||||||
|
)
|
||||||
|
assert rec.rows[0]["reasoning_effort"] is None
|
||||||
|
|
||||||
|
async def test_a_reasonless_path_never_records_a_tier(self):
|
||||||
|
"""embedding/OCR 走同一个 emitter,但它们的 payload 里没有推理参数。
|
||||||
|
|
||||||
|
源上误配了 `ENABLE_THINKING` 时,回落若照算就会给一次 embedding 失败
|
||||||
|
挂上 `auto` ——那一档从来没有、也不可能被发出去。
|
||||||
|
"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(enable_thinking=True),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=None,
|
||||||
|
error="boom",
|
||||||
|
reasoning_applies=False,
|
||||||
|
)
|
||||||
|
assert rec.rows[0]["reasoning_effort"] is None
|
||||||
|
|
||||||
|
async def test_an_out_of_domain_tier_degrades_but_keeps_the_row(self):
|
||||||
|
"""域外取值降级为 `NULL` 且**不丢整行**(遥测必录);与缓存回放同一方向。
|
||||||
|
|
||||||
|
`LLMResponse` 无运行时校验,测试替身写裸串完全自然;直接 `Effort(raw)` 会
|
||||||
|
抛 `ValueError`,被 `_record` 的 `except Exception` 吞成丢整行。
|
||||||
|
"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
messages: list[str] = []
|
||||||
|
sink_id = logger.add(messages.append, level="WARNING")
|
||||||
|
try:
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=_resp(applied_effort="lowest"),
|
||||||
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
logger.remove(sink_id)
|
||||||
|
assert len(rec.rows) == 1, "整行被吞了"
|
||||||
|
assert rec.rows[0]["reasoning_effort"] is None
|
||||||
|
hits = [m for m in messages if "lowest" in m]
|
||||||
|
assert len(hits) == 1, f"域外取值必须单独告警: {messages}"
|
||||||
|
assert [m for m in messages if "遥测记录失败" in m] == []
|
||||||
|
|
||||||
|
async def test_a_bare_string_tier_still_lands(self):
|
||||||
|
"""裸串在域内时照常归一并落库,整行不得丢失。"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
await TelemetryEmitter(rec, text_cap=None).emit_attempt(
|
||||||
|
request=_REQ,
|
||||||
|
source=_source(),
|
||||||
|
call_id="c",
|
||||||
|
latency_ms=1,
|
||||||
|
response=_resp(applied_effort="max"),
|
||||||
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
|
)
|
||||||
|
assert len(rec.rows) == 1, "整行被吞了"
|
||||||
|
value = rec.rows[0]["reasoning_effort"]
|
||||||
|
assert value == "max"
|
||||||
|
assert type(value) is str
|
||||||
|
|
||||||
|
|
||||||
class TestEmitterObservabilityFields:
|
class TestEmitterObservabilityFields:
|
||||||
"""issue #3: 三个入口各自的取值口径(设计 §5 表)。"""
|
"""issue #3: 三个入口各自的取值口径(设计 §5 表)。"""
|
||||||
|
|
||||||
@@ -1266,6 +1478,7 @@ class TestEmitterObservabilityFields:
|
|||||||
latency_ms=42,
|
latency_ms=42,
|
||||||
response=_resp(cached_prompt_tokens=64, model_reported="m-real", reasoning_tokens=7),
|
response=_resp(cached_prompt_tokens=64, model_reported="m-real", reasoning_tokens=7),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cached_prompt_tokens"] == 64
|
assert rec.rows[0]["cached_prompt_tokens"] == 64
|
||||||
assert rec.rows[0]["model_reported"] == "m-real"
|
assert rec.rows[0]["model_reported"] == "m-real"
|
||||||
@@ -1280,6 +1493,7 @@ class TestEmitterObservabilityFields:
|
|||||||
latency_ms=7,
|
latency_ms=7,
|
||||||
response=None,
|
response=None,
|
||||||
error="boom",
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cached_prompt_tokens"] is None
|
assert rec.rows[0]["cached_prompt_tokens"] is None
|
||||||
assert rec.rows[0]["model_reported"] is None
|
assert rec.rows[0]["model_reported"] is None
|
||||||
@@ -1330,6 +1544,7 @@ class TestEmitterSamplingColumn:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert json.loads(rec.rows[0]["sampling"]) == {"seed": 42, "temperature": 0}
|
assert json.loads(rec.rows[0]["sampling"]) == {"seed": 42, "temperature": 0}
|
||||||
|
|
||||||
@@ -1344,6 +1559,7 @@ class TestEmitterSamplingColumn:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
await emitter.emit_cache_hit(request=self._SAMPLED, response=_resp())
|
await emitter.emit_cache_hit(request=self._SAMPLED, response=_resp())
|
||||||
await emitter.emit_terminal_failure(
|
await emitter.emit_terminal_failure(
|
||||||
@@ -1376,6 +1592,7 @@ class TestEmitterSamplingColumn:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["sampling"] is None
|
assert rec.rows[0]["sampling"] is None
|
||||||
|
|
||||||
@@ -1408,6 +1625,7 @@ class TestEmitterCallerDimensions:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
elif emit == "cache_hit":
|
elif emit == "cache_hit":
|
||||||
await emitter.emit_cache_hit(request=self._REQ_A, response=_resp(cache_hit=True))
|
await emitter.emit_cache_hit(request=self._REQ_A, response=_resp(cache_hit=True))
|
||||||
@@ -1461,6 +1679,7 @@ class TestEmitterCallerDimensions:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
row = rec.rows[0]
|
row = rec.rows[0]
|
||||||
assert row["tenant_id"] == ""
|
assert row["tenant_id"] == ""
|
||||||
@@ -1521,6 +1740,7 @@ class TestCostWithCachedTier:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=full,
|
response=full,
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
await emitter.emit_attempt(
|
await emitter.emit_attempt(
|
||||||
request=_REQ,
|
request=_REQ,
|
||||||
@@ -1531,6 +1751,7 @@ class TestCostWithCachedTier:
|
|||||||
prompt_tokens=1_000_000, completion_tokens=0, cached_prompt_tokens=600_000
|
prompt_tokens=1_000_000, completion_tokens=0, cached_prompt_tokens=600_000
|
||||||
),
|
),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] == pytest.approx(10.0)
|
assert rec.rows[0]["cost"] == pytest.approx(10.0)
|
||||||
assert rec.rows[1]["cost"] == pytest.approx(5.2) # 400k×10 + 600k×2
|
assert rec.rows[1]["cost"] == pytest.approx(5.2) # 400k×10 + 600k×2
|
||||||
@@ -1553,6 +1774,7 @@ class TestCostWithCachedTier:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(usage_source="unavailable", cached_prompt_tokens=5),
|
response=_resp(usage_source="unavailable", cached_prompt_tokens=5),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] is None
|
assert rec.rows[0]["cost"] is None
|
||||||
|
|
||||||
@@ -1568,6 +1790,7 @@ class TestEmitter:
|
|||||||
latency_ms=42,
|
latency_ms=42,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
row = rec.rows[0]
|
row = rec.rows[0]
|
||||||
assert row["call_id"] == "cid-1" and row["error"] is None
|
assert row["call_id"] == "cid-1" and row["error"] is None
|
||||||
@@ -1584,6 +1807,7 @@ class TestEmitter:
|
|||||||
latency_ms=7,
|
latency_ms=7,
|
||||||
response=None,
|
response=None,
|
||||||
error="TransientError: boom",
|
error="TransientError: boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
row = rec.rows[0]
|
row = rec.rows[0]
|
||||||
assert row["error"].startswith("TransientError")
|
assert row["error"].startswith("TransientError")
|
||||||
@@ -1615,6 +1839,7 @@ class TestEmitter:
|
|||||||
usage_source="unavailable", prompt_tokens=prompt, completion_tokens=completion
|
usage_source="unavailable", prompt_tokens=prompt, completion_tokens=completion
|
||||||
),
|
),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] is None
|
assert rec.rows[0]["cost"] is None
|
||||||
|
|
||||||
@@ -1628,6 +1853,7 @@ class TestEmitter:
|
|||||||
latency_ms=42,
|
latency_ms=42,
|
||||||
response=_resp(prompt_tokens=0, completion_tokens=4000),
|
response=_resp(prompt_tokens=0, completion_tokens=4000),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert rec.rows[0]["cost"] == pytest.approx(0.032)
|
assert rec.rows[0]["cost"] == pytest.approx(0.032)
|
||||||
|
|
||||||
@@ -1661,6 +1887,7 @@ class TestEmitter:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=None,
|
response=None,
|
||||||
error="x",
|
error="x",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert len(rec.rows[0]["messages"]) < 500 # base64 不整段进库(VT R12)
|
assert len(rec.rows[0]["messages"]) < 500 # base64 不整段进库(VT R12)
|
||||||
|
|
||||||
@@ -1677,6 +1904,7 @@ class TestEmitter:
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(),
|
response=_resp(),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
) # 不抛(降级不冒泡)
|
) # 不抛(降级不冒泡)
|
||||||
|
|
||||||
|
|
||||||
@@ -1773,6 +2001,7 @@ async def _emit_with_cap(messages, *, cap, response=_LONG, thinking=_LONG):
|
|||||||
latency_ms=1,
|
latency_ms=1,
|
||||||
response=_resp(content=response, thinking=thinking),
|
response=_resp(content=response, thinking=thinking),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
return rec.rows[0]
|
return rec.rows[0]
|
||||||
|
|
||||||
|
|||||||
@@ -262,6 +262,7 @@ async def test_emit_attempt_success_stays_in_domain(emitted):
|
|||||||
latency_ms=10,
|
latency_ms=10,
|
||||||
response=_resp(emitted),
|
response=_resp(emitted),
|
||||||
error=None,
|
error=None,
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert recorder.rows[0]["usage_source"] in USAGE_SOURCES
|
assert recorder.rows[0]["usage_source"] in USAGE_SOURCES
|
||||||
|
|
||||||
@@ -276,6 +277,7 @@ async def test_emit_attempt_failed_attempt_stays_in_domain():
|
|||||||
latency_ms=10,
|
latency_ms=10,
|
||||||
response=None,
|
response=None,
|
||||||
error="boom",
|
error="boom",
|
||||||
|
reasoning_applies=True,
|
||||||
)
|
)
|
||||||
assert recorder.rows[0]["usage_source"] in USAGE_SOURCES
|
assert recorder.rows[0]["usage_source"] in USAGE_SOURCES
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user