test: judge reasoning by what the library actually observed
The four cases were red because the criterion could not see the evidence. reasoning_tokens has been None on this route ever since MiniMax stopped reporting completion_tokens_details, while the same call carried 185 characters of reasoning prose the assertions never looked at. L5 asserted something that cannot happen. M3 returns neither prose nor usage detail over the plain endpoint, so demanding that the non-streaming path observe reasoning could never pass. It now asserts what is true and worth holding: the prompt_tokens anchor still separates the two directions, so the parameter did reach the model, and the verdict is not ABSENT, so the library marked the gap honestly instead of dressing it up as no reasoning. _ON_MIN_COMPLETION is gone. The two directions overlap in output length — 46 at most disabled, 13 at least enabled — so that fallback drew a line through noise and only made the criterion look defended.
This commit is contained in:
@@ -1,21 +1,27 @@
|
||||
"""真实 API 验证推理开关与 reasoning_tokens(issue #5 + #6)。
|
||||
"""真实 API 验证推理开关与推理可观测性(issue #5 + #6;判据于 #16/#17 重建)。
|
||||
|
||||
本组用例**必须真跑**: 改动的正确性与具体模型强相关,mock 只能验证代码路径,
|
||||
验证不了"这个参数在这个模型上到底关没关掉推理"。
|
||||
|
||||
两条判据纪律(来自 findings §4c 的实测教训):
|
||||
三条判据纪律(第 1、2 条来自 findings §4c,第 1 条的推翻与第 3 条来自
|
||||
`findings/2026-08-25-thinking-observability-regression.md`):
|
||||
|
||||
1. **判别量只能是 `reasoning_tokens`,不能是 `completion_tokens`。** 两档的输出
|
||||
长度分布**是重叠的**: 实测关闭档最高 46 token(模型偶尔把解题过程写进正文),
|
||||
开启档最低 13 token(medium 档想得少的那几轮),按长度阈值判两边都会误判。
|
||||
而 `reasoning_tokens` 在同一批 30 轮里干净分开——关闭 15/15 为 None,
|
||||
开启 15/15 大于 0。
|
||||
2. **另配一个不含魔数的确定性锚点**(见 L2b): 同一模型上,关闭档的
|
||||
1. **判别量是库裁定的三态 `thinking_observation`,既不是 `reasoning_tokens`
|
||||
也不是 `completion_tokens`。** 长度判据早已排除: 两档的输出长度分布**是
|
||||
重叠的**(实测关闭档最高 46 token、开启档最低 13 token),按阈值判两边都会
|
||||
误判。而 `reasoning_tokens` 这个曾经"干净分开"的判据也已失效——MiniMax
|
||||
这一路上游不再返回 `usage.completion_tokens_details`,该字段恒 `None`;同一
|
||||
次调用里库明明拿得到 185 字符推理正文,单看 token 计数却把"推理正常"读成
|
||||
"没推理"(2026-08-25 findings §3.4/结论③,四条用例因此假红)。三态裁定同时
|
||||
看正文与计数: **正文是事实本身,token 计数只是对事实的转述**。
|
||||
2. **另配一个不含魔数的确定性锚点**(见 L2b、L5): 同一模型上,关闭档的
|
||||
`prompt_tokens` 严格小于开启档——供应商在开启时注入了推理指令,输入侧
|
||||
token 数随之变大。这是相对比较,不硬编码任何具体数值。
|
||||
3. **关闭方向要求每轮满足,开启方向只要求多数轮满足。** 中转在上游不返回
|
||||
usage 时会本地补算并吃掉 `completion_tokens_details`(findings §4c),
|
||||
开启方向因此可能偶尔观测不到;关闭方向不受影响。
|
||||
token 数随之变大。这是相对比较,不硬编码任何具体数值;且它不依赖上游是否
|
||||
回传推理正文,所以在"观测不到推理"的非流式路径上依然作数。
|
||||
3. **`UNKNOWN` 不等于"没推理",不能拿它判红。** 关闭方向要求每轮"未观测到
|
||||
推理"(`UNKNOWN` 计入满足——它没有证伪力),其证伪力来自: 模型若偷偷推理了,
|
||||
可观测路径会翻成 `OBSERVED`。开启方向只要求多数轮 `OBSERVED`;M3 非流式
|
||||
路径整片观测不到,该档由 L5 用另一套断言覆盖。
|
||||
|
||||
源不可用一律 `skip` 并在报告中记为「未覆盖」,**绝不静默计入通过**。
|
||||
"""
|
||||
@@ -30,7 +36,7 @@ from pathlib import Path
|
||||
import pytest
|
||||
from dotenv import dotenv_values
|
||||
|
||||
from polygateway import GatewayClient, GatewaySettings
|
||||
from polygateway import GatewayClient, GatewaySettings, ThinkingObservation
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
RequestRejectedError,
|
||||
@@ -42,10 +48,11 @@ from polygateway.thinking import DEFAULT_CAPABILITIES, get_capability
|
||||
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
|
||||
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
|
||||
|
||||
# slow: 本组 137 次真实调用、约 7 分钟,且判据是统计性的——网络抖动会让它偶发
|
||||
# 失败(实测有一次 network_error 连续三次耗尽源)。让它阻断 `make ci` 会把测试
|
||||
# 变成噪声源,故沿用项目既有的 slow 标记默认排除,合并前用 `-m slow` 显式真跑并
|
||||
# 存档报告。"不自动门控"不等于"可跳过"。
|
||||
# slow: 本组 92 次真实调用、约 4 分半(2026-08-26 判据换三态后实测;此前记的
|
||||
# "137 次、约 7 分钟"已被证伪,别照旧值估 CI 预算),且判据是统计性的——网络抖动
|
||||
# 会让它偶发失败(实测有一次 network_error 连续三次耗尽源)。让它阻断 `make ci`
|
||||
# 会把测试变成噪声源,故沿用项目既有的 slow 标记默认排除,合并前用 `-m slow`
|
||||
# 显式真跑并存档报告。"不自动门控"不等于"可跳过"。
|
||||
pytestmark = [
|
||||
pytest.mark.slow,
|
||||
pytest.mark.skipif(
|
||||
@@ -60,9 +67,6 @@ _ROUNDS = int(os.environ.get("PGW_E2E_THINKING_ROUNDS", "10"))
|
||||
# 开着时则是几百——两档之间隔着一个数量级,判据不必卡在噪声里
|
||||
_PROMPT = "一个笼子里有若干鸡和兔,共 35 个头、94 只脚。鸡和兔各有多少只?只输出两个数字。"
|
||||
|
||||
_ON_MIN_COMPLETION = 100
|
||||
"""仅用于 `reasoning_tokens` 被中转吃掉时的退路;关闭方向不设长度门(见 `_reasoning_off`)。"""
|
||||
|
||||
_ROWS: list[dict] = []
|
||||
|
||||
# 显式映射,不按模型名猜 provider —— 那正是 D11 要消灭的东西(providers.py 开篇)。
|
||||
@@ -107,6 +111,10 @@ async def _run_rounds(rounds: int, *, stream: bool = True, **source_overrides) -
|
||||
"prompt_tokens": resp.prompt_tokens,
|
||||
"completion_tokens": resp.completion_tokens,
|
||||
"reasoning_tokens": resp.reasoning_tokens,
|
||||
# 结论与证据一起入报告: 只记 observation 会让"为什么这么判"
|
||||
# 不可复核,而 thinking_chars 正是本次改判的直接证据
|
||||
"thinking_observation": resp.thinking_observation,
|
||||
"thinking_chars": len(resp.thinking),
|
||||
"content": resp.content[:60],
|
||||
}
|
||||
)
|
||||
@@ -128,26 +136,32 @@ def _record(matrix_id: str, desc: str, status: str, detail, observations=None) -
|
||||
|
||||
|
||||
def _reasoning_off(obs: dict) -> bool:
|
||||
"""关闭方向: 只看 reasoning_tokens。
|
||||
"""关闭方向: 只要没观测到推理即算满足。
|
||||
|
||||
`UNKNOWN` 计入满足是有意的: 它没有证伪力(本次无任何信号,判不出来),拿它
|
||||
判红等于每次关闭调用都喊一遍。本判据真正的证伪力在于——模型若偷偷推理了,
|
||||
可观测路径会把裁定翻成 `OBSERVED`。
|
||||
|
||||
**刻意不设 completion_tokens 上限**: 实测关闭档偶尔会到 46 token(模型没照做
|
||||
"只输出两个数字",把解题过程写进了正文),而那是正文不是推理。加长度门只会
|
||||
把这种正常波动误判成"没关掉"。
|
||||
"""
|
||||
return obs["reasoning_tokens"] in (None, 0)
|
||||
return obs["thinking_observation"] != ThinkingObservation.OBSERVED
|
||||
|
||||
|
||||
def _reasoning_on(obs: dict) -> bool:
|
||||
"""开启方向: 有 reasoning_tokens 就以它为准,它是本次改动引入的直接判据。
|
||||
"""开启方向: 观测到推理即为真。
|
||||
|
||||
不能拿 completion_tokens 当开启方向的主判据: medium 档的推理量方差极大
|
||||
(实测 15 轮跨 7-170 token),按长度阈值判会把"推理了但想得少"误判成没推理。
|
||||
仅当中转吃掉了 ctd(reasoning_tokens is None)才退回长度判据。
|
||||
判据从 `reasoning_tokens` 换成库的三态裁定,因为 MiniMax 这一路已不再上报
|
||||
`completion_tokens_details`(2026-08-25 findings 结论②),该字段恒 `None`;
|
||||
而库在同一次调用里拿得到 185 字符推理正文(findings §3.4)——旧判据看不见
|
||||
它,L2/L3b/L4/L5 四条因此假红。
|
||||
|
||||
也不能退回 completion_tokens 当判据: medium 档的推理量方差极大(实测 15 轮
|
||||
跨 7-170 token),两档分布还与关闭档重叠,按长度阈值判会把"推理了但想得少"
|
||||
误判成没推理。
|
||||
"""
|
||||
reasoning = obs["reasoning_tokens"]
|
||||
if reasoning is not None:
|
||||
return reasoning > 0
|
||||
return obs["completion_tokens"] > _ON_MIN_COMPLETION
|
||||
return obs["thinking_observation"] == ThinkingObservation.OBSERVED
|
||||
|
||||
|
||||
def _skip_if_unreachable(exc: Exception, matrix_id: str, desc: str):
|
||||
@@ -163,14 +177,18 @@ def _write_report():
|
||||
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
path = _OUT_DIR / f"test_thinking_live_{ts}.md"
|
||||
lines = [
|
||||
"# 推理开关与 reasoning_tokens 真实 API 验证",
|
||||
"# 推理开关与推理可观测性真实 API 验证",
|
||||
"",
|
||||
f"- 时间: {ts}",
|
||||
f"- 每档轮数: {_ROUNDS}",
|
||||
"- 关闭判据: **每轮** reasoning_tokens in (None, 0);刻意不设输出长度上限"
|
||||
"(两档的 completion 分布重叠: 实测关闭档最高 46、开启档最低 13)",
|
||||
f"- 开启判据: **多数轮** reasoning_tokens > 0(被中转吃掉时退回 completion > {_ON_MIN_COMPLETION})",
|
||||
"- 确定性锚点(L2b): 关闭档 prompt_tokens 最大值 < 开启档最小值,相对比较无魔数",
|
||||
"- 判别量: 库裁定的三态 `thinking_observation`(OBSERVED/ABSENT/UNKNOWN),"
|
||||
"由推理正文与 reasoning_tokens 共同裁定 —— 正文是事实,token 计数只是转述",
|
||||
"- 关闭判据: **每轮** observation != OBSERVED(UNKNOWN 计入满足,它没有证伪力);"
|
||||
"刻意不设输出长度上限(两档的 completion 分布重叠: 实测关闭档最高 46、开启档最低 13)",
|
||||
"- 开启判据: **多数轮** observation == OBSERVED",
|
||||
"- 确定性锚点(L2b、L5): 关闭档 prompt_tokens 最大值 < 开启档最小值,相对比较无魔数",
|
||||
"- L5(非流式): M3 该路径推理已计费却不回传正文,故不断言「观测到推理」,"
|
||||
"改断锚点可分 + 开启档不被误判为 ABSENT",
|
||||
"",
|
||||
"## 矩阵结论",
|
||||
"",
|
||||
@@ -206,7 +224,7 @@ class TestMiniMaxM3:
|
||||
"L1",
|
||||
"enable_thinking=False(流式)",
|
||||
"PASS" if len(offs) == len(obs) else "FAIL",
|
||||
f"{len(offs)}/{len(obs)} 轮确认未推理",
|
||||
f"{len(offs)}/{len(obs)} 轮未观测到推理",
|
||||
obs,
|
||||
)
|
||||
assert len(offs) == len(obs), f"关闭方向要求每轮满足: {obs}"
|
||||
@@ -256,7 +274,7 @@ class TestMiniMaxM3:
|
||||
"L3",
|
||||
"enable_thinking=None(不干预,基线)",
|
||||
"PASS" if len(quiet) == len(obs) else "FAIL",
|
||||
f"{len(quiet)}/{len(obs)} 轮未推理(M3 默认档本就不推理)",
|
||||
f"{len(quiet)}/{len(obs)} 轮未观测到推理(M3 默认档本就不推理)",
|
||||
obs,
|
||||
)
|
||||
assert len(quiet) == len(obs), f"M3 默认档不应推理: {obs}"
|
||||
@@ -271,6 +289,12 @@ class TestMiniMaxM3:
|
||||
判别方法: 发一个**非法值**。若未知值会被静默丢弃,它的表现应与"不注入"
|
||||
一致(不推理);实测它反而开启了推理,说明网关认这个键、只是不认这个值。
|
||||
既然非法值与 `none` 的表现不同,`none` 就必然是被识别的枚举值。
|
||||
|
||||
**该手法不可移植,只对"认这个键但不校验值"的 provider 成立**: minimax 对
|
||||
非法 `reasoning_effort` 返回 200 且照常推理(2026-08-25 findings §5:
|
||||
prompt 207,介于基线 194 与 medium 216 之间,走了第三条模板路径);而 qwen
|
||||
对同样的值直接返回 **HTTP 400**。把本用例套到 qwen 那类会校验值的 provider
|
||||
上,拿到的会是异常而非"不推理",是假红。
|
||||
"""
|
||||
rounds = max(3, _ROUNDS // 3)
|
||||
bogus = await _run_rounds(
|
||||
@@ -318,23 +342,50 @@ class TestMiniMaxM3:
|
||||
)
|
||||
assert len(ons) * 2 > len(obs), f"extra_body 未能覆盖 profile: {obs}"
|
||||
|
||||
async def test_l5_non_stream_path_matches_stream(self):
|
||||
"""非流式快路径独立于流式实现,采集与注入都要各自验一遍。"""
|
||||
async def test_l5_non_stream_path_is_distinguishable_and_honestly_unknown(self):
|
||||
"""非流式快路径: 参数确实到达了模型,而推理信号被如实标成"观测不到"。
|
||||
|
||||
**本用例不能断言"非流式开启档观测到推理"——那永远不成立**: M3 在非流式
|
||||
路径下推理段确实产生并计费(2026-08-25 findings §3.4: 开启档 completion 53
|
||||
vs 关闭档 3),但 `message` 里没有 `reasoning_content`、`usage` 里也没有
|
||||
`completion_tokens_details`,推理内容整体不回传。**这是上游行为,库修不了;
|
||||
库能做也必须做的是让它可见**——下游在为看不见的东西付费,不该由库替它
|
||||
沉默。
|
||||
|
||||
故改断两件在非流式下真实成立的事:
|
||||
其一 `prompt_tokens` 锚点仍把两档分开(判据形态照抄 L2b,证明注入到达了模型,
|
||||
排除"非流式路径把参数弄丢了"这一伪解释);
|
||||
其二开启档的裁定**不是 `ABSENT`**——`ABSENT` 的语义是"上游明确上报未推理",
|
||||
而实情是"判不出来"(`UNKNOWN`),库若把后者伪装成前者,正是 issue #16/#17 里
|
||||
那个静默错觉。这里断 `!= ABSENT` 而非 `== UNKNOWN`,是为了留出上游哪天开始
|
||||
回传正文的余地: 那时裁定会翻成 `OBSERVED`,是好事,不该让它把测试判红。
|
||||
"""
|
||||
rounds = max(3, _ROUNDS // 2)
|
||||
off = await _run_rounds(rounds, stream=False, model="MiniMax-M3", enable_thinking=False)
|
||||
on = await _run_rounds(rounds, stream=False, model="MiniMax-M3", enable_thinking=True)
|
||||
offs = [o for o in off if _reasoning_off(o)]
|
||||
ons = [o for o in on if _reasoning_on(o)]
|
||||
ok = len(offs) == len(off) and len(ons) * 2 > len(on)
|
||||
off_max = max(o["prompt_tokens"] for o in off)
|
||||
on_min = min(o["prompt_tokens"] for o in on)
|
||||
not_absent = [o for o in on if o["thinking_observation"] != ThinkingObservation.ABSENT]
|
||||
on_states = Counter(str(o["thinking_observation"]) for o in on)
|
||||
ok = len(offs) == len(off) and off_max < on_min and len(not_absent) == len(on)
|
||||
_record(
|
||||
"L5",
|
||||
"非流式路径重跑 L1/L2",
|
||||
"非流式: prompt 锚点可分 + 开启档如实标 UNKNOWN 而非 ABSENT",
|
||||
"PASS" if ok else "FAIL",
|
||||
f"关闭 {len(offs)}/{len(off)} 轮,开启 {len(ons)}/{len(on)} 轮",
|
||||
f"关闭 {len(offs)}/{len(off)} 轮未观测到推理;"
|
||||
f"关闭档 prompt 最大 {off_max} < 开启档最小 {on_min};"
|
||||
f"开启档裁定分布 {dict(on_states)}",
|
||||
off + on,
|
||||
)
|
||||
assert len(offs) == len(off), f"非流式关闭方向未满足: {off}"
|
||||
assert len(ons) * 2 > len(on), f"非流式开启方向未满足: {on}"
|
||||
assert off_max < on_min, (
|
||||
f"非流式两档 prompt_tokens 未分开(关闭最大 {off_max},开启最小 {on_min}): "
|
||||
f"开启参数可能没到达模型"
|
||||
)
|
||||
assert len(not_absent) == len(on), (
|
||||
f"非流式开启档被裁成 ABSENT(声称上游明确上报未推理),而实情是观测不到: {on}"
|
||||
)
|
||||
|
||||
|
||||
class TestOtherProviders:
|
||||
@@ -357,7 +408,7 @@ class TestOtherProviders:
|
||||
matrix,
|
||||
desc,
|
||||
"PASS" if len(offs) == len(obs) else "FAIL",
|
||||
f"{len(offs)}/{len(obs)} 轮确认未推理",
|
||||
f"{len(offs)}/{len(obs)} 轮未观测到推理",
|
||||
obs,
|
||||
)
|
||||
assert len(offs) == len(obs), f"{provider} 关闭方向未满足: {obs}"
|
||||
@@ -395,7 +446,7 @@ class TestCapabilityDrift:
|
||||
"L8",
|
||||
desc,
|
||||
"PASS" if len(offs) == len(obs) else "FAIL(能力表已漂移)",
|
||||
f"实测 {dict(verdict)};声明 can_disable=True 要求每轮关闭",
|
||||
f"实测未观测到推理 {dict(verdict)}(True=满足);声明 can_disable=True 要求每轮满足",
|
||||
obs,
|
||||
)
|
||||
assert len(offs) == len(obs), (
|
||||
|
||||
Reference in New Issue
Block a user