test: replace the guessed tier table with what the gateway actually does

T10 经 new-api 中转对 26 个可调用模型逐个实测(约 500 次真实调用),把
DEFAULT_CAPABILITIES 里的文档推定换成实测结论,并把实测本身固化成可复跑、
可只跑单模型的 slow 用例。

三处与文档推定不符:
- kimi-k3 实测可关(关闭档 completion 恒 9 且与 max 档锚点可分),T1 的保守
  登记被推翻,OpenRouter 的 mandatory:false 胜出;
- MiniMax-M3 的 auto 实测不推理(minimax 的"开"是不注入任何参数,而 M3 默认档
  本就不推理),六个强度值反而全部生效且彼此等价,故清单由 (none, auto) 换成
  none + 六个强度值;
- glm-5.3 关不掉推理坐实(请求 none 后 4/5 轮仍观测到推理),issue #20 的核心
  争议就此了结——当时短提示词下的 rt≈1.2 是采样噪声。

新增 kimi-for-coding(现有该型号自己的实测证据,不再走 Phase 3)。

用例带三条判据: 关闭方向要求每轮未观测到推理;短提示词的"关掉了"必须过长
上下文复核(glm-5.3-flash 正是短 5/5 未观测、长上下文 2/3 露馅);上游整片
不回传推理信号时另取无魔数锚点,不许把"没看见"写成"没发生"。另加一道模型
身份核对——该渠道把 glm-5/5.1/5.2 全部路由到 glm-5.3,那三组数据一律作废。

未覆盖 9 个型号(claude 三个 7 天限额用尽、gemini 两个上游报错、gpt-5.4 限流、
glm 三个被路由),evidence 逐条写明原因,不得被后续文档写成"已实测"。

单元测试里三条以 MiniMax-M3 当"纯开关型"样本的用例改用 glm-4.6v——实测后
M3 不再是那个形状,而 glm-4.6v 是全表证据最硬的 (none, auto)。
This commit is contained in:
2026-09-05 10:35:30 -04:00
parent f9b357b9d7
commit f5e6fafe8d
3 changed files with 724 additions and 70 deletions
+158 -60
View File
@@ -108,111 +108,218 @@ class ThinkingCapability:
return any(e not in (Effort.NONE, Effort.AUTO) for e in self.supported_efforts) return any(e not in (Effort.NONE, Effort.AUTO) for e in self.supported_efforts)
# 证据分类,evidence 里必须自报家门: # 证据分类,evidence 里必须自报家门:
# 实测 = 经 new-api 中转打过真实请求(最硬,不得被文档推定覆盖); # 实测 = 经 new-api 中转打过真实请求(最硬,不得被文档推定覆盖);
# 文档推定 = 官方文档 / OpenRouter / cherry-studio / LiteLLM 四方交叉(待实测校正)。 # 文档推定 = 官方文档 / OpenRouter / cherry-studio / LiteLLM 四方交叉;
# 实测未覆盖 = T10 试过但拿不到数据(渠道限额/上游报错/被路由到别的模型),
# 此时**必须写明原因**——"没测到"与"测了没问题"是两回事,T9 之类的下游
# 文档任务不得把前者写成后者。
_MEASURED = "2026-08-02 经 new-api 中转实测" _MEASURED = "2026-08-02 经 new-api 中转实测"
_DOC = "2026-09-04 文档推定(官方文档 + OpenRouter + cherry-studio + LiteLLM 四方交叉),待经 new-api 实测" _T10 = "2026-09-05 经 new-api 中转实测(T10: 短提示词 N=5,声称可关的再加长上下文 N=3 复核)"
_DOC = "2026-09-04 文档推定(官方文档 + OpenRouter + cherry-studio + LiteLLM 四方交叉)"
# T10 的三条判据(报告见 tests/outputs/thinking/,用例见 tests/e2e/test_thinking_live.py):
# ① 关闭方向要求**每轮**未观测到推理,任一轮观测到即证伪;
# ② 短提示词下的"关掉了"必须过长上下文复核——glm-5.3-flash 正是短提示词 5/5
# 未观测到推理、5000 token 长上下文下 2/3 轮露馅(issue #20 的原始现象);
# ③ 上游整片不回传推理信号(kimi/MiniMax/qwen/gpt 这几路的关闭档都是)时,
# "没看见"不算"没发生",另取一个无魔数锚点: 关闭档的 completion_tokens
# 必须严格小于 max 档。
DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType( DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
{ {
# —— 实测条目(2026-08-02/08-25),证据原文保留 —— # —— MiniMax ——
"MiniMax-M3": ThinkingCapability( "MiniMax-M3": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO), supported_efforts=(
Effort.NONE,
Effort.MINIMAL,
Effort.LOW,
Effort.MEDIUM,
Effort.HIGH,
Effort.XHIGH,
Effort.MAX,
),
evidence=( evidence=(
"2026-08-02 经 new-api 中转实测 N=10: reasoning_effort=none 稳定关闭,零跳变;" f"{_T10}: reasoning_effort=none 关闭成立(短 5/5 + 长上下文 3/3 未观测到推理,"
"2026-08-25 复测依然成立(prompt 194 = 基线、completion 3、无推理正文)。" "completion 恒 3 token,且与 max 档 completion 57-173 锚点可分);六个强度值各 N=5 "
"两条限制(findings 2026-08-25-thinking-observability-regression §3.1/§5): " "全部观测到推理,rt 分布完全重叠(minimal 64-124 / low 55-112 / medium 51-104 / "
"① 非流式路径观测不到推理信号——推理已计费,但正文与 usage 明细都不回传;" "high 62-128 / xhigh 58-118 / max 57-170)——**它们是''的六种写法,不是六个深度档**,"
"② enable_thinking / thinking:{type:enabled} 对本模型无效,仅 reasoning_effort 是真开关" "MiniMax 官方只有开/关两态,配哪一个都一样贵"
"无强度档: 官方只有开/关两态(thinking.type disabled/adaptive)" "**`auto` 已从清单移除**: minimax 的''在 wire 上是 on_base={}(什么参数都不注入),"
"而 M3 的默认档实测不推理,故 auto 在这条路上表达不了''(N=5 全部未观测到推理)。"
"注意 `resolve_thinking` 的 Phase 5 无条件放行 auto,故存量 ENABLE_THINKING=true 仍会"
"静默落到不推理——那是 wire 的缺口,能力表堵不住,已记入 T10 报告待单独立项。"
f"历史: {_MEASURED} N=10 同样成立;enable_thinking / thinking:{{type}} 两种写法对本模型"
"无效,reasoning_effort 才是真开关(findings 2026-08-25 §3.1/§5)。"
"另: 2026-08-25 记录的'MiniMax 这一路已停报 completion_tokens_details'本次**不再成立**"
"——开启档 rt 有值,只有关闭档整片缺 details"
), ),
), ),
"MiniMax-M2.7": ThinkingCapability( "MiniMax-M2.7": ThinkingCapability(
supported_efforts=(Effort.AUTO,), supported_efforts=(Effort.AUTO,),
evidence=( evidence=(
"2026-08-02 实测 reasoning_effort=none / thinking:{disabled} / thinking:{adaptive} " f"{_T10}: 请求 none 时 5/5 轮仍观测到推理(rt 100-161、推理正文 274-482 字符),"
"各 N=3 全部无效;OpenRouter 注册表登记 mandatory:true,models.dev 登记无控制手段" "**关不掉**成立;auto 档 5/5 观测到推理"
"MiniMax 官方亦承认 M2.x 接受 disabled 但推理仍开着" f"历史({_MEASURED}): reasoning_effort=none / thinking:{{disabled}} / thinking:{{adaptive}} "
"各 N=3 全部无效;OpenRouter 登记 mandatory:true,models.dev 登记无控制手段"
), ),
), ),
"MiniMax-M2.5": ThinkingCapability( "MiniMax-M2.5": ThinkingCapability(
supported_efforts=(Effort.AUTO,), supported_efforts=(Effort.AUTO,),
evidence="2026-08-02 实测同 M2.7: 三种形态各 N=3 全部无效;外部注册表同样登记为强制推理", evidence=(
f"{_T10}: 请求 none 时 5/5 轮仍观测到推理(rt 104-158),**关不掉**成立;"
"auto 档 5/5 观测到推理(rt 121-245)。"
f"历史({_MEASURED}): 三种形态各 N=3 全部无效;外部注册表同样登记为强制推理"
), ),
),
# —— qwen(百炼系,开关型) ——
"qwen3.7-plus": ThinkingCapability( "qwen3.7-plus": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO), supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=( evidence=(
"2026-08-02 实测 enable_thinking=false 关闭(completion 5 token,无推理)。" f"{_T10}: none 关闭成立(短 5/5 + 长 3/3 未观测到推理,且与 auto 档 completion 锚点可分);"
"无强度档: OpenRouter 登记本型号只支持 reasoning 开关,不支持 reasoning_effort" "auto 档 5/5 观测到推理。无强度档: 该 provider 的 wire 没有 effort_key,请求 max 当场被库"
"拒(百炼靠 thinking_budget 调深度,预算型控制本库不支持)。"
f"历史({_MEASURED}): enable_thinking=false 关闭(completion 5 token)"
), ),
), ),
"qwen3.7-max": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_T10}: 同 qwen3.7-plus——none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
),
"qwen3.6-plus": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
),
"qwen3.5-flash": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
),
"qwen-plus-latest": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
),
# —— deepseek ——
"deepseek-v4-pro": ThinkingCapability( "deepseek-v4-pro": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
evidence=( evidence=(
"关闭档为 2026-08-02 实测(thinking:{type:disabled},completion 3 token,无推理);" f"{_T10}: none 关闭成立(短 5/5 + 长 3/3 未观测到推理,与 max 档锚点可分);"
f"强度档为{_DOC}: 官方 thinking_mode 文档列 Non-think/Think High/Think Max 三态,默认 high" "high / max 各 N=5 全部观测到推理(rt 59-73 / 56-69,推理正文 max 档明显更长: "
"135-186 vs 101-124 字符)。默认档按官方 thinking_mode 文档为 high"
), ),
), ),
# —— 文档推定条目(2026-09-04),待 T10 经 new-api 实测校正 ——
"deepseek-v4-flash": ThinkingCapability( "deepseek-v4-flash": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
evidence=f"{_DOC}: 官方文档「deepseek-v4-flash 与 deepseek-v4-pro 一致」,默认 high", evidence=(
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3,锚点可分);high / max 各 N=5 全部观测到推理"
"(rt 16-36 / 12-42)。与 v4-pro 同档,印证官方'与 deepseek-v4-pro 一致'的说法"
),
), ),
"deepseek-v4-flash-vision-exp": ThinkingCapability( "deepseek-v4-flash-vision-exp": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
evidence=f"{_DOC}: 同 v4-flash 一档(OpenRouter 登记支持 reasoning_effort)", evidence=(
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3,锚点可分);high / max 各 N=5 全部观测到推理"
"(rt 12-17 / 17-34)"
), ),
),
# —— 智谱 ——
"glm-5.3": ThinkingCapability( "glm-5.3": ThinkingCapability(
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
evidence=( evidence=(
f"{_DOC}: **推理不可关闭**——智谱官方文档明确 thinking.type 只接受 enabled," f"{_T10}: **推理不可关闭已实测坐实**——请求 none(注入 thinking:{{type:disabled}})后 "
"官方迁移建议是改用 enabled + reasoning_effort=low;cherry-studio 标 toggle:false、" "5 轮里 4 轮仍观测到推理(rt 7、推理正文 12 字符),只有 1 轮 rt=0;"
"OpenRouter 标 mandatory:true,三源一致。默认 max" "low / high / max 各 N=5 全部观测到推理(rt 55-77 / 47-63 / 48-60,三档分不出深浅)"
"注: issue #20 实测的 reasoning_effort=none 是**未定义值**,短提示词下 rt≈1.2 像是关了," "这一条了结了 issue #20 的核心争议: 当时短提示词下 rt≈1.2 看着像关掉了,实为采样噪声。"
"5552 token 长上下文下跳到 0/54/167 即露馅" f"文档侧三源一致({_DOC}): 智谱官方 thinking.type 只接受 enabled、迁移建议改用 "
"enabled + reasoning_effort=low,cherry-studio 标 toggle:false,OpenRouter 标 mandatory:true。"
"默认 max。**注意本渠道不校验档位值**: 未登记的 medium 也会被照单接受(实测 rt 62),"
"'网关没报错'在这一路上不构成'该档受支持'的证据"
), ),
), ),
"glm-5.3-flash": ThinkingCapability( "glm-5.3-flash": ThinkingCapability(
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
evidence=f"{_DOC}: 同 glm-5.3(cherry-studio 的 pattern 'glm-5[.-]3' 覆盖两者),默认 max", evidence=(
f"{_T10}: **推理不可关闭,且是判据②唯一的现役样本**——请求 none 时短提示词 5/5 轮"
"未观测到推理(看着完全像关掉了),换成 5000 token 长上下文后 3 轮里 2 轮露馅"
"(rt=2、有推理正文)。只跑短提示词的实测会在这个模型上得出相反结论。"
"low / high / max 各 N=5 全部观测到推理(rt 8-60 / 27-91 / 27-70)。默认 max"
),
), ),
"glm-5.2": ThinkingCapability( "glm-5.2": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX), supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
evidence=( evidence=(
f"{_DOC}: cherry-studio 登记 none/high/max(官方端点默认 max,百炼上默认 high)。" f"{_DOC}: cherry-studio 登记 none/high/max(官方端点默认 max,百炼上默认 high)。"
"注意: issue #20 记录本渠道对 glm-5.2 的请求 6/6 回报 model=glm-5.3,疑被路由,实测时须核对 model_reported" "**T10 实测未覆盖——该渠道把本型号路由到了别的模型**: 请求 glm-5.2 时 5/5 轮回报 "
"model=glm-5.3(issue #20 记录的 6/6 复现),拿到的行为不属于本型号,故整组数据作废、"
"本行仍是文档推定。**下游风险**: 在本渠道上给 glm-5.2 配 none,库会照本行放行,"
"而真正服务请求的 glm-5.3 关不掉推理——运行期 reconcile 会喊,但那是事后"
), ),
), ),
"glm-5": ThinkingCapability( "glm-5": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO), supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: OpenRouter 登记只支持 reasoning 开关、无 reasoning_effort;cherry-studio 标 toggle:true", evidence=(
f"{_DOC}: OpenRouter 登记只支持 reasoning 开关、无 reasoning_effort;cherry-studio 标 toggle:true。"
"**T10 实测未覆盖**: 与 glm-5.2 同因——5/5 轮回报 model=glm-5.3,数据不属于本型号"
),
), ),
"glm-5.1": ThinkingCapability( "glm-5.1": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO), supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: 同 glm-5(OpenRouter reasoning.mandatory=false 且无 supported_efforts)", evidence=(
f"{_DOC}: 同 glm-5(OpenRouter reasoning.mandatory=false 且无 supported_efforts)。"
"**T10 实测未覆盖**: 5/5 轮回报 model=glm-5.3,数据不属于本型号"
),
), ),
"glm-4.6v": ThinkingCapability( "glm-4.6v": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO), supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: OpenRouter 登记无 reasoning_effort;VLM,推理控制同 glm-4.x 系开关型",
),
"kimi-k3": ThinkingCapability(
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
evidence=( evidence=(
f"{_DOC}: 官方 reasoning_effort 三档 low/high/max,默认 max。" f"{_T10}: none 关闭成立,且是全表**证据最硬**的一条——短 5/5 + 长 3/3 全部裁定 ABSENT"
"**保守登记为不可关**——官方档位表无 none,而 OpenRouter 标 mandatory:false,两源分歧待实测;" "(上游明确上报 reasoning_tokens=0,不是'看不见'),无需锚点旁证;"
"保守方向的代价是下游配 none 会报错并被指向 low,反方向的代价是静默失效(issue #20 的病)。" "auto 档 5/5 观测到推理(rt 57-153)。model_reported 与请求一致,未被路由"
"另: 官方提示切换档位会使 prefix cache 失效,不宜在会话中途改档"
), ),
), ),
# —— 月之暗面 ——
"kimi-k3": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.LOW, Effort.HIGH, Effort.MAX),
evidence=(
f"{_T10}: **可关闭——推翻 T1 的保守登记**。请求 none(注入 thinking:{{type:disabled}})后"
"短 5/5 + 长上下文 3/3 轮无任何推理信号,completion 恒 9 token;同一模型 max 档 "
"completion 明显更大且带推理正文(rt 33-146),锚点可分——故'没看见'这次有正面证据支撑。"
"两源分歧由此了结: OpenRouter 的 mandatory:false 是对的,官方档位表没列 none 只是没列。"
"low / high / max 各 N=5 全部观测到推理(rt 21-53 / 38-60 / 33-146)。"
"**model_reported 是 `k3`**(别名,非串台)。官方提示切换档位会使 prefix cache 失效,"
"不宜在会话中途改档"
),
),
"kimi-for-coding": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.LOW, Effort.HIGH, Effort.MAX),
evidence=(
f"{_T10}: 本型号在 T1 时因'档位清单无直接证据'走 Phase 3 不登记(设计 §8 第三档),"
"现有它自己的实测证据故补登。none: 短 5/5 + 长 3/3 无推理信号、completion 恒 2 token,"
"与开启档锚点可分;low / high / max 各 N=3 全部观测到推理(rt 8-40 / 25-75 / 62-85)。"
"档位词汇沿用月之暗面官方的 low/high/max: 本渠道对 moonshot **不校验档位值**"
"(minimal/medium/xhigh 照样返回 200 并推理),故'没被拒'不构成'受支持',"
"登记一个厂商没声明的档等于替它做承诺"
),
),
# —— OpenAI ——
"gpt-5.4": ThinkingCapability( "gpt-5.4": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH), supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
evidence=f"{_DOC}: OpenRouter 登记 none/low/medium/high/xhigh,默认 medium;LiteLLM 登记 minimal 不支持", evidence=(
f"{_DOC}: OpenRouter 登记 none/low/medium/high/xhigh,默认 medium;LiteLLM 登记 minimal 不支持。"
"**T10 实测未覆盖**: 该渠道本型号所有账号限流(429 All available accounts are "
"currently rate-limited),5/5 轮失败。同代的 gpt-5.5 已实测且与本清单逐字相符"
),
), ),
"gpt-5.5": ThinkingCapability( "gpt-5.5": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH), supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
evidence=f"{_DOC}: 同 gpt-5.4(OpenRouter supported_efforts 一致,默认 medium)", evidence=(
f"{_T10}: 清单**逐条对上**,是全表验证最完整的一行。none 关闭成立(短 5/5 + 长 3/3,"
"锚点可分);low/medium/high/xhigh 各 N=5 全部观测到推理,且 rt 随档位单调上升"
"(18-21 / 18-22 / 22-34 / 35-65)——本渠道上少见的、档位真的分得开的模型;"
"清单外的 max 与 minimal 各 N=3 全部被上游 400 拒("
"Unsupported value),说明这一路**会校验档位值**,与 zhipu/moonshot 的照单全收相反"
), ),
),
# —— Anthropic ——
"claude-opus-5": ThinkingCapability( "claude-opus-5": ThinkingCapability(
supported_efforts=( supported_efforts=(
Effort.NONE, Effort.NONE,
@@ -225,7 +332,8 @@ DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
evidence=( evidence=(
f"{_DOC}: Anthropic 官方 adaptive thinking + output_config.effort 五档(low/medium/high/" f"{_DOC}: Anthropic 官方 adaptive thinking + output_config.effort 五档(low/medium/high/"
"xhigh/max),默认 high;OpenRouter 标 mandatory:false 故可关。" "xhigh/max),默认 high;OpenRouter 标 mandatory:false 故可关。"
"关闭档依赖 new-api 把 reasoning_effort=none 转成 thinking 关闭形态,待实测确认" "**T10 实测未覆盖**: 该渠道 claude 全系返回 429「api key 7天限额已用完」,5/5 轮失败。"
"关闭档仍依赖 new-api 把 reasoning_effort=none 转成 thinking 关闭形态,未经验证"
), ),
), ),
"claude-sonnet-5": ThinkingCapability( "claude-sonnet-5": ThinkingCapability(
@@ -237,32 +345,22 @@ DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
Effort.XHIGH, Effort.XHIGH,
Effort.MAX, Effort.MAX,
), ),
evidence=f"{_DOC}: 同 claude-opus-5(OpenRouter supported_efforts 与默认档一致)", evidence=(
f"{_DOC}: 同 claude-opus-5(OpenRouter supported_efforts 与默认档一致)。"
"**T10 实测未覆盖**: 同因 429「api key 7天限额已用完」"
), ),
),
# —— Google ——
"gemini-3.1-pro": ThinkingCapability( "gemini-3.1-pro": ThinkingCapability(
supported_efforts=(Effort.LOW, Effort.MEDIUM, Effort.HIGH), supported_efforts=(Effort.LOW, Effort.MEDIUM, Effort.HIGH),
evidence=( evidence=(
f"{_DOC}: **推理不可关闭**——Google 官方文档明确 Gemini 3 Pro / 3.1 Pro 无法关闭思考," f"{_DOC}: **推理不可关闭**——Google 官方文档明确 Gemini 3 Pro / 3.1 Pro 无法关闭思考,"
"OpenRouter 亦标 mandatory:true。thinking_level 三档;默认档两源打架" "OpenRouter 亦标 mandatory:true。thinking_level 三档;默认档两源打架"
"(官方文档说 HIGH,OpenRouter 说 medium),待实测" "(官方文档说 HIGH,OpenRouter 说 medium)"
"**T10 实测未覆盖**: 该渠道本型号上游报错(bad_response_status_code / openai_error),"
"5/5 轮失败,连默认档基线都没取到,两源分歧仍悬着"
), ),
), ),
"qwen-plus-latest": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: 百炼 enable_thinking 开关型(thinking_budget 是 token 预算,本库不支持预算型)",
),
"qwen3.5-flash": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: 同 qwen-plus-latest(OpenRouter 登记无 reasoning_effort)",
),
"qwen3.6-plus": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: 同 qwen-plus-latest(OpenRouter 登记无 reasoning_effort)",
),
"qwen3.7-max": ThinkingCapability(
supported_efforts=(Effort.NONE, Effort.AUTO),
evidence=f"{_DOC}: 同 qwen3.7-plus 一代(OpenRouter 登记无 reasoning_effort)",
),
} }
) )
"""在用模型的推理能力登记(YAGNI: 不覆盖全世界,未登记走 `resolve_thinking` 退化)。""" """在用模型的推理能力登记(YAGNI: 不覆盖全世界,未登记走 `resolve_thinking` 退化)。"""
+552 -1
View File
@@ -26,12 +26,15 @@
源不可用一律 `skip` 并在报告中记为「未覆盖」,**绝不静默计入通过**。 源不可用一律 `skip` 并在报告中记为「未覆盖」,**绝不静默计入通过**。
""" """
import asyncio
import dataclasses import dataclasses
import json import json
import os import os
from collections import Counter from collections import Counter
from collections.abc import Mapping
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from types import MappingProxyType
import pytest import pytest
from dotenv import dotenv_values from dotenv import dotenv_values
@@ -39,11 +42,13 @@ from dotenv import dotenv_values
from polygateway import GatewayClient, GatewaySettings, ThinkingObservation from polygateway import GatewayClient, GatewaySettings, ThinkingObservation
from polygateway.errors import ( from polygateway.errors import (
AllSourcesExhausted, AllSourcesExhausted,
GatewayUnavailableError,
RequestRejectedError, RequestRejectedError,
SourceDeadError, SourceDeadError,
TransientError, TransientError,
) )
from polygateway.thinking import DEFAULT_CAPABILITIES, get_capability from polygateway.thinking import DEFAULT_CAPABILITIES, ThinkingCapability, get_capability
from polygateway.types import EFFORT_ORDER, Effort
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None} _ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV) _HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
@@ -77,7 +82,28 @@ _MODEL_PROVIDER = {
"MiniMax-M2.7": "minimax", "MiniMax-M2.7": "minimax",
"MiniMax-M2.5": "minimax", "MiniMax-M2.5": "minimax",
"qwen3.7-plus": "qwen", "qwen3.7-plus": "qwen",
"qwen3.7-max": "qwen",
"qwen3.6-plus": "qwen",
"qwen3.5-flash": "qwen",
"qwen-plus-latest": "qwen",
"deepseek-v4-pro": "deepseek", "deepseek-v4-pro": "deepseek",
"deepseek-v4-flash": "deepseek",
"deepseek-v4-flash-vision-exp": "deepseek",
"glm-5.3": "zhipu",
"glm-5.3-flash": "zhipu",
"glm-5.2": "zhipu",
"glm-5.1": "zhipu",
"glm-5": "zhipu",
"glm-4.6v": "zhipu",
"kimi-k3": "moonshot",
"kimi-for-coding": "moonshot",
"gpt-5.4": "openai",
"gpt-5.5": "openai",
"claude-opus-5": "anthropic",
"claude-sonnet-5": "anthropic",
"claude-haiku-5": "anthropic",
"gemini-3.1-pro": "google",
"gemini-3-flash": "google",
} }
@@ -519,3 +545,528 @@ class TestAssemblyGuardAgainstRealConfig:
finally: finally:
await client.aclose() await client.aclose()
_record("L9", "绕过装配守卫时 transport 兜底", "PASS", "RequestRejectedError,属四分类") _record("L9", "绕过装配守卫时 transport 兜底", "PASS", "RequestRejectedError,属四分类")
# ══════════════════════════════════════════════════════════════════════════════
# T10: 逐模型档位实测(方法论沿用 issue #20)
#
# 本节与上面的 L1-L9 分工不同: 上面验的是**库的行为**(注入到没到、观测准不准),
# 这里验的是**能力表的内容**(`DEFAULT_CAPABILITIES` 里那 20 多条声明是不是真的)。
# 二者判据可以共用,数据源却必须分开——能力表实测要**绕过能力表**才有意义,
# 否则拿待验证的声明去挡请求,等于用结论证明前提。
#
# 判据(三条,均沿用已有纪律):
# ① 关闭方向: 每轮 `thinking_observation != OBSERVED` 才算真关掉;任一轮
# OBSERVED 即证伪(推理正文是事实本身,不需要多数票)。
# ② **短提示词的"关掉了"必须经长上下文复核**: issue #20 实测 GLM 系在短提示词
# 下 reasoning_tokens≈1.2 像是关了,5552 token 长上下文下跳到 0/54/167 即露馅。
# 短提示词下推理量本就趋近于 0,分不出"关了"与"没什么可想的"。
# ③ 开启方向: 多数轮 OBSERVED(单轮抖动不判红,与 L2 同口径)。
# ④ 关闭结论**不许只靠 `UNKNOWN`**: 上游整片不回传推理信号时(kimi、MiniMax 两路
# 都是),"没看见"不是"没发生"。此时补一个不含魔数的锚点——关闭档的
# `completion_tokens` 必须严格小于 `max` 档,否则结论记为「判不出来」。
# ══════════════════════════════════════════════════════════════════════════════
_TIER_OUT_DIR = Path("tests/outputs/thinking")
_TIER_ROUNDS = int(os.environ.get("PGW_E2E_TIER_ROUNDS", "5"))
_TIER_LONG_ROUNDS = int(os.environ.get("PGW_E2E_TIER_LONG_ROUNDS", "3"))
# 共用生产网关,宁慢勿冲(人类 2026-09-05 指令): 默认 3,可下调不建议上调
_TIER_CONCURRENCY = int(os.environ.get("PGW_E2E_TIER_CONCURRENCY", "3"))
# 固定短提示词: 答案本身约 4 token,推理 token 的信噪比高(issue #20 同款)
_TIER_PROMPT = "23 乘以 47 等于多少?只回答一个数字,不要解释。"
# 长上下文对照组(判据②)。填充文本与题目无关且不含任何业务领域词汇(零业务假设
# 铁律),只为把输入撑到数千 token;题目放在最后,避免被当成"读完就忘"的前缀
_TIER_LONG_PROMPT = (
"\n".join(
f"{i:04d}. 这是一段与题目无关的填充文字,仅用于把上下文撑到数千 token,"
"以复核短提示词下得到的关闭结论在长上下文下是否依然成立。"
for i in range(120)
)
+ "\n\n"
+ _TIER_PROMPT
)
_ALL_EFFORTS: tuple[Effort, ...] = (*EFFORT_ORDER, Effort.AUTO)
_PROBE_ROWS: list[dict] = []
def _tier_settings(model: str) -> GatewaySettings:
"""探测用配置: 生产口径的超时,但**重试预算压到 1 次**。
压重试是因为探测里"这一轮失败"本身就是数据(逐轮进报告),库替它重试只会
"渠道当下不可用"变成三倍等待——2026-09-05 实测 claude 系 7 天限额用尽时
每轮 429,三次重试让单个模型阻塞三分钟以上,26 个模型跑不完。
**单次请求的超时不动**(仍是 .env 的生产值 300s): §4.6 那条"测试超时不得紧于
生产配置"防的是把慢而正常的模型误判成不可用,那个风险在这里照旧存在。重试次数
与背压窗口不属于同一类——它们决定"失败之后还等多久",而不是"多慢算失败";
一个真在出字的模型永远碰不到这两者。
"""
base = GatewaySettings.from_env(
"LLM",
env={
**_ENV,
"PGW_CACHE_BACKEND": "none",
"LLM_MAX_RETRIES": "1",
# 探测是**单源**的,没有别的源可换。生产值 1200s 的 stall window 在这里
# 只会把"这个模型当下不可用"拖成 20 分钟一轮: 2026-09-05 实测 claude 系
# 7 天限额用尽返回 429 且不带 Retry-After,库据此判"无可运行源"并按背压
# 语义等到窗口耗尽(实测把窗口调到 45s 即在 46.7s 报 stalled)。多源生产
# 场景下这段等待是有意义的(等别的源恢复),探测场景下等不到任何东西
"LLM__BACKPRESSURE__STALL_WINDOW_S": "60",
},
)
source = dataclasses.replace(
base.sources[0],
provider=_MODEL_PROVIDER[model],
model=model,
enable_thinking=None,
reasoning_effort=None,
)
return dataclasses.replace(base, sources=(source,))
def _probe_capabilities(model: str) -> dict[str, ThinkingCapability]:
"""临时全档能力表: **实测的对象正是能力表本身**,不能拿它当前提去挡请求。
不传 `capabilities={}`(即"未登记")的理由是噪声: 那条路会走 Phase 3,每轮都
warning 一句"能力未登记",几百轮下来把真正的告警淹没。全档表让五关全部放行,
请求原样发出去,由上游而不是由库来回答"这一档到底行不行"
"""
return {model: ThinkingCapability(_ALL_EFFORTS, evidence="T10 实测临时表(不进 DEFAULT)")}
async def _probe_effort(
model: str, effort: Effort, *, rounds: int, prompt: str, prompt_kind: str
) -> list[dict]:
"""对一个 (模型, 档位) 打 N 轮真实请求,逐轮记录;失败轮记 `error` 而不冒泡。
失败不冒泡是本函数与 `_run_rounds` 的唯一区别: 这里"上游拒绝这一档"本身就是
**实测结论**(HTTP 400 = 该档不被接受),把它抛出去会让数据采集半途而废。
只吞四分类与 `AllSourcesExhausted`——库自身的 `ValueError` 等仍然冒泡,那是
bug 不是数据。
"""
client = GatewayClient.from_settings(
_tier_settings(model), capabilities=_probe_capabilities(model)
)
semaphore = asyncio.Semaphore(_TIER_CONCURRENCY)
async def _one(index: int) -> dict:
base = {"round": index + 1, "effort": effort.value, "prompt_kind": prompt_kind}
async with semaphore:
try:
resp = await client.chat(
[{"role": "user", "content": prompt}],
stream=True,
reasoning_effort=effort,
cache_salt=f"tier-probe-{model}-{effort.value}-{prompt_kind}-{index}",
)
except (
RequestRejectedError,
GatewayUnavailableError,
SourceDeadError,
TransientError,
) as exc:
# 捕 `GatewayUnavailableError` 而不是只捕 `AllSourcesExhausted`:
# 某个模型在网关上不通时,连续失败会把熔断门打开,后续轮次抛的是
# `CircuitOpenError`(同一父类的兄弟)。只捕子类会让"源不可用"这
# 件事在第 N 轮换个类型冒出去,把数据采集打断成一次红测
return {**base, "error": f"{type(exc).__name__}: {str(exc)[:160]}"}
return {
**base,
"error": None,
"prompt_tokens": resp.prompt_tokens,
"completion_tokens": resp.completion_tokens,
"reasoning_tokens": resp.reasoning_tokens,
"thinking_chars": len(resp.thinking),
"thinking_observation": resp.thinking_observation,
"applied_effort": resp.applied_effort,
# 核对模型身份: issue #20 记录本渠道对 glm-5.2 的请求 6/6 回报
# model=glm-5.3。凡结论依赖模型身份的,对不上即数据不可信
"model_reported": resp.model_reported,
"content": resp.content[:40],
}
try:
return list(await asyncio.gather(*(_one(i) for i in range(rounds))))
finally:
await client.aclose()
def _probe_ok(obs: dict) -> bool:
return obs["error"] is None
def _probe_quiet(obs: dict) -> bool:
"""成功且未观测到推理(判据①的满足条件);失败轮不算"安静",它没有观测。"""
return _probe_ok(obs) and obs["thinking_observation"] != ThinkingObservation.OBSERVED
def _probe_observed(obs: dict) -> bool:
return _probe_ok(obs) and obs["thinking_observation"] == ThinkingObservation.OBSERVED
def _rt_summary(observations: list[dict]) -> str:
"""报告里的一行摘要: rt 观测值序列 + 裁定分布 + 身份核对,三样缺一不可复核。"""
ok = [o for o in observations if _probe_ok(o)]
if not ok:
return f"全部 {len(observations)} 轮失败: {observations[0]['error']}"
rts = [o["reasoning_tokens"] for o in ok]
verdicts = Counter(str(o["thinking_observation"]) for o in ok)
reported = sorted({str(o["model_reported"]) for o in ok})
failed = len(observations) - len(ok)
tail = f";{failed} 轮失败" if failed else ""
return (
f"rt={rts};裁定 {dict(verdicts)};thinking_chars="
f"{[o['thinking_chars'] for o in ok]};model_reported={reported}{tail}"
)
# 已知的合法别名: 供应商回报的名字与配置里的别名本就可以不同(月之暗面回
# `k3`、Google 回 `-preview` 后缀)。**显式登记而不是按前缀猜**——猜的话
# `glm-5.2 → glm-5.3` 这种真·串台也会被当成"同族别名"放过,而那正是本表要抓的
_MODEL_REPORTED_ALIASES: Mapping[str, frozenset[str]] = MappingProxyType(
{
"kimi-k3": frozenset({"k3"}),
"kimi-for-coding": frozenset({"k3"}),
"gemini-3-flash": frozenset({"gemini-3-flash-preview"}),
"gemini-3.1-pro": frozenset({"gemini-3.1-pro-preview"}),
}
)
def _identity_mismatch(model: str, observations: list[dict]) -> list[str]:
"""响应体里的 `model` 与请求的模型对不上 → 本次数据说的不是这个模型。
issue #20 就栽在这里: 该渠道对 `glm-5.2` 的请求 6/6 回报 `model=glm-5.3`,
照单全收的话,能力表里 glm-5.2 那一行记的其实是 glm-5.3 的行为。凡结论依赖
模型身份的,对不上就必须当场作废,而不是打个折扣继续用。
`None`(上游未上报)不算不符: 那是"没说",不是"说了别的"
"""
allowed = {model, *_MODEL_REPORTED_ALIASES.get(model, frozenset())}
return sorted(
{
o["model_reported"]
for o in observations
if _probe_ok(o)
and o["model_reported"] is not None
and o["model_reported"] not in allowed
}
)
async def _anchor_off_against_on(
model: str, off_observations: list[dict]
) -> tuple[list[dict], Effort | None, bool]:
"""判据④: 拿"开启档的 completion 明显更大"给关闭结论补一个正面证据。
需要它是因为 `UNKNOWN` 的语义: 它是"本次没有任何信号,判不出来",不是"没推理"
(`observe_thinking` 的 docstring 把这条写死了)。kimi 与 MiniMax 这两路上游都
不回传 `completion_tokens_details`,关闭档整片 `UNKNOWN`——此时若直接把"没看见"
读成"关掉了",库就会登记一个自己从未验证过的 `none`,而下游据此以为省了钱。
锚点取 `completion_tokens` 的相对比较(关闭档最大值 < 开启档最小值),**不含
任何魔数**: 推理段计在 completion 里,真开着时两档差一个数量级(实测 kimi-k3
关闭档恒 9 token)。取 `max` 档而非 `auto`: 后者对 minimax 一路等于"什么都不注入"
(`on_base={}`),那是模型默认档而不是"",拿它当对照组会把 M3 这种默认不推理的
模型判成"分不开"。`max` 打不通时才退到 `auto`。
"""
off_usable = [o for o in off_observations if _probe_ok(o)]
for tier in (Effort.MAX, Effort.AUTO):
anchor = await _probe_effort(
model,
tier,
rounds=_TIER_LONG_ROUNDS,
prompt=_TIER_PROMPT,
prompt_kind=f"anchor({tier.value})",
)
on_usable = [o for o in anchor if _probe_ok(o)]
if not on_usable:
continue
off_max = max(o["completion_tokens"] for o in off_usable)
on_min = min(o["completion_tokens"] for o in on_usable)
return anchor, tier, off_max < on_min
return [], None, False
def _probe_record(model: str, phase: str, verdict: str, detail: str, observations: list[dict]):
_PROBE_ROWS.append(
{
"model": model,
"provider": _MODEL_PROVIDER[model],
"phase": phase,
"verdict": verdict,
"detail": detail,
"observations": observations,
}
)
@pytest.fixture(scope="module", autouse=True)
def _write_tier_report():
"""T10 报告独立成文件: 它的读者是"能力表该怎么改",与 L1-L9 的"库对不对"不同。"""
yield
if not _PROBE_ROWS:
return
_TIER_OUT_DIR.mkdir(parents=True, exist_ok=True)
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
path = _TIER_OUT_DIR / f"tier_probe_{ts}.md"
lines = [
"# 推理档位能力表实测(T10,经 new-api 中转)",
"",
f"- 时间: {ts}",
f"- 短提示词轮数: {_TIER_ROUNDS};长上下文复核轮数: {_TIER_LONG_ROUNDS};"
f"并发: {_TIER_CONCURRENCY}(共用生产网关,宁慢勿冲)",
f"- 短提示词: `{_TIER_PROMPT}`",
f"- 长上下文: 同题 + {len(_TIER_LONG_PROMPT)} 字符无关填充(判据②)",
"- 判据: 关闭方向要求**每轮**未观测到推理,且短提示词的「关掉了」必须经长上下文复核;"
"开启方向要求多数轮 OBSERVED",
"- 能力表在探测时被临时替换为全档表: 实测的对象正是它,不能拿它挡请求",
"",
"## 逐模型结论",
"",
"| 模型 | provider | 阶段 | 结论 | 观测 |",
"|---|---|---|---|---|",
]
total = 0
for row in _PROBE_ROWS:
detail = str(row["detail"]).replace("|", "\\|").replace("\n", " ")[:220]
lines.append(
f"| {row['model']} | {row['provider']} | {row['phase']} | {row['verdict']} | {detail} |"
)
total += len(row["observations"])
lines += ["", f"**总真实调用次数: {total}**", "", "## 逐轮原始观测", ""]
for row in _PROBE_ROWS:
if not row["observations"]:
continue
lines += [f"### {row['model']}{row['phase']}", "", "```json"]
lines.append(json.dumps(row["observations"], ensure_ascii=False, indent=2, default=str))
lines += ["```", ""]
path.write_text("\n".join(lines), encoding="utf-8")
print(f"\n[T10 报告] {path}")
class TestTierProbe:
"""能力表实测。可只跑单个模型: `-k "test_t10 and glm-5.3"`。"""
@pytest.mark.parametrize("model", sorted(_MODEL_PROVIDER))
async def test_t10_none_direction_matches_declaration(self, model):
"""「这个模型到底关不关得掉」——能力表里唯一会**报错**的那条声明。
它是本节最要紧的一条: `Effort.NONE` 在不在清单里,决定 Phase 4 是放行还是
当场报错。声明错了,两个方向的代价都很实在——多写了 `none` 会让下游以为
关掉了(issue #20 的静默失效),漏写了会把一条本来可用的路堵死。
"""
short = await _probe_effort(
model, Effort.NONE, rounds=_TIER_ROUNDS, prompt=_TIER_PROMPT, prompt_kind="short"
)
# 上游拒绝这一档(400)是**结论**而非故障: 它等价于"关不掉";
# 其余失败(渠道下线/超时)才是源不可用,按既有纪律记为未覆盖
rejected = [o for o in short if o["error"] and o["error"].startswith("RequestRejected")]
usable = [o for o in short if _probe_ok(o)]
# **按可用轮判,而不是一有失败就整条跳过**: 共用网关上偶发 429/503 是常态,
# 一票否决会让整张表因为一次抖动而没有数据。样本低于 3 轮才是真的没结论
if not rejected and len(usable) < min(3, _TIER_ROUNDS):
broken = [o for o in short if o["error"]]
_probe_record(model, "none 方向", "SKIP(源不可用)", _rt_summary(short), short)
pytest.skip(f"{model} 源不可用,已记为未覆盖: {broken[0]['error'][:120]}")
strangers = _identity_mismatch(model, short)
if strangers:
_probe_record(
model,
"none 方向",
"SKIP(身份不符,数据不可信)",
f"该渠道把请求回报成 {strangers};{_rt_summary(short)}",
short,
)
pytest.skip(f"{model} 被该渠道路由到 {strangers},本次观测说的不是这个模型")
observations = list(short)
measured_can_disable = not rejected and all(_probe_quiet(o) for o in usable)
note = ""
if measured_can_disable:
# 判据②: 短提示词下"看起来关了"必须过长上下文这一关
long_ctx = await _probe_effort(
model,
Effort.NONE,
rounds=_TIER_LONG_ROUNDS,
prompt=_TIER_LONG_PROMPT,
prompt_kind="long",
)
observations += long_ctx
usable = [o for o in long_ctx if _probe_ok(o)]
if not usable:
note = ";长上下文复核未跑通,结论只在短提示词下成立"
else:
measured_can_disable = all(_probe_quiet(o) for o in usable)
note = ";长上下文复核" + ("同样未观测到推理" if measured_can_disable else "露馅")
if measured_can_disable and not any(
o["thinking_observation"] is ThinkingObservation.ABSENT for o in observations
):
# 判据④: 全程 `UNKNOWN` 时,"关掉了"是一句没有正面证据的话
anchor, anchor_tier, separable = await _anchor_off_against_on(model, observations)
observations += anchor
if anchor_tier is None:
note += ";锚点未跑通,关闭结论缺正面证据"
elif separable:
note += f";锚点可分(关闭档 completion 严格小于 {anchor_tier.value} 档)"
else:
measured_can_disable = None
note += f";**锚点不可分**(与 {anchor_tier.value} 档的 completion 分不开),判不出来"
detail = f"实测 can_disable={measured_can_disable}{note}。短: {_rt_summary(short)}" + (
f" ‖ 后续: {_rt_summary(observations[len(short) :])}"
if len(observations) > len(short)
else ""
)
if measured_can_disable is None:
_probe_record(model, "none 方向", "INCONCLUSIVE(无正面证据)", detail, observations)
pytest.skip(f"{model} 判不出来,已记为未覆盖: {detail[:160]}")
capability = get_capability(model)
if capability is None:
_probe_record(model, "none 方向", "DATA(未登记)", detail, observations)
pytest.skip(f"{model} 未登记(设计 §8 第三档),本条只采数据: {detail[:120]}")
agrees = measured_can_disable == capability.can_disable
_probe_record(
model,
"none 方向",
"PASS" if agrees else "FAIL(能力表已漂移)",
f"声明 can_disable={capability.can_disable};{detail}",
observations,
)
assert agrees, (
f"{model} 的能力表与实测不符: 声明 can_disable={capability.can_disable},"
f"实测 {measured_can_disable}{detail}"
)
@pytest.mark.parametrize("model", sorted(DEFAULT_CAPABILITIES))
async def test_t10_declared_tiers_actually_reason(self, model):
"""已登记的每个**开启档**都必须被上游接受,且真的推理。
证伪力只在"被拒""没推理"两件事上——**不断言档位之间的 rt 高低**:
设计 §4.3 已定,同一档 rt 实测在 8~56 之间跳,拿它比大小必然是噪声。
故本条能证伪的是"登记了一个上游根本不认的档",不是"档位排序对不对"
"""
capability = get_capability(model)
tiers = [e for e in capability.supported_efforts if e is not Effort.NONE]
if not tiers:
pytest.skip(f"{model} 只登记了 none,没有开启档可验")
failures = []
for tier in tiers:
observations = await _probe_effort(
model, tier, rounds=_TIER_ROUNDS, prompt=_TIER_PROMPT, prompt_kind="short"
)
rejected = [
o for o in observations if o["error"] and o["error"].startswith("RequestRejected")
]
usable = [o for o in observations if _probe_ok(o)]
observed = [o for o in observations if _probe_observed(o)]
strangers = _identity_mismatch(model, observations)
if strangers:
# 与 none 方向同一条纪律: 回报的不是这个模型,这组数就不是它的
_probe_record(
model,
f"档位 {tier.value}",
"SKIP(身份不符,数据不可信)",
f"该渠道把请求回报成 {strangers};{_rt_summary(observations)}",
observations,
)
pytest.skip(f"{model} 被该渠道路由到 {strangers},本次观测说的不是这个模型")
if rejected:
verdict, problem = "FAIL(上游拒绝该档)", f"{tier.value}: 上游拒绝"
elif not usable:
verdict, problem = "SKIP(源不可用)", None
elif len(observed) * 2 > len(usable):
verdict, problem = "PASS", None
else:
verdict, problem = "FAIL(该档未推理)", f"{tier.value}: 多数轮未观测到推理"
if problem:
failures.append(problem)
_probe_record(
model, f"档位 {tier.value}", verdict, _rt_summary(observations), observations
)
assert not failures, f"{model} 登记的档位与实测不符: {failures}"
@pytest.mark.parametrize("model", ["gemini-3.1-pro", "gpt-5.5", "glm-5.3"])
async def test_t10_no_opinion_stays_no_opinion(self, model):
"""不表态时库**不推定**模型自己的默认档(Phase 1),顺带采下默认档的 rt 基线。
为什么给这三个模型单列一条: 它们的「厂商默认档」是 evidence 里写着、却最容易
写错的一格(Gemini 3.1 Pro 官方文档说 HIGH、OpenRouter 说 medium,两源打架),
而默认档写错会误导下游估成本。库本身不依赖这个值——**它不表态就什么都不注入**,
这正是本条断言的东西;默认档的 rt 观测只作报告里的旁证,**不作断言**: 单一模型上
rt 与档位没有可判定的函数关系(设计 §4.3),拿它反推默认档只能存疑,不能定论。
2026-09-05: gemini 一路当下在本渠道上游报错,claude 一路 7 天限额用尽,故把
另两格换成当下可测的 gpt-5.5 与 glm-5.3;gemini 留着,渠道恢复即有数。
"""
client = GatewayClient.from_settings(
_tier_settings(model), capabilities=_probe_capabilities(model)
)
observations = []
try:
for i in range(_TIER_ROUNDS):
try:
resp = await client.chat(
[{"role": "user", "content": _TIER_PROMPT}],
stream=True,
cache_salt=f"tier-default-{model}-{i}",
)
except (
RequestRejectedError,
GatewayUnavailableError,
SourceDeadError,
TransientError,
) as exc:
observations.append(
{
"round": i + 1,
"effort": "(不表态)",
"prompt_kind": "short",
"error": f"{type(exc).__name__}: {str(exc)[:160]}",
}
)
continue
observations.append(
{
"round": i + 1,
"effort": "(不表态)",
"prompt_kind": "short",
"error": None,
"prompt_tokens": resp.prompt_tokens,
"completion_tokens": resp.completion_tokens,
"reasoning_tokens": resp.reasoning_tokens,
"thinking_chars": len(resp.thinking),
"thinking_observation": resp.thinking_observation,
"applied_effort": resp.applied_effort,
"model_reported": resp.model_reported,
"content": resp.content[:40],
}
)
finally:
await client.aclose()
usable = [o for o in observations if _probe_ok(o)]
if not usable:
_probe_record(
model,
"默认档基线(不表态)",
"SKIP(源不可用)",
_rt_summary(observations),
observations,
)
pytest.skip(f"{model} 源不可用,已记为未覆盖: {observations[0]['error'][:120]}")
leaked = [o for o in usable if o["applied_effort"] is not None]
_probe_record(
model,
"默认档基线(不表态)",
"PASS" if not leaked else "FAIL(库替模型推定了默认档)",
_rt_summary(observations),
observations,
)
assert not leaked, f"{model}: 不表态时 applied_effort 应为 None,实测 {leaked}"
+14 -9
View File
@@ -311,10 +311,15 @@ class TestResolveThinking:
assert "low" in msg and "high" in msg and "max" in msg assert "low" in msg and "high" in msg and "max" in msg
def test_phase5_says_toggle_only_for_switch_model(self): def test_phase5_says_toggle_only_for_switch_model(self):
"""纯开关型模型没有档位,对它说"可选档位"是错的(设计 §3.2 第三个派生量)。""" """纯开关型模型没有档位,对它说"可选档位"是错的(设计 §3.2 第三个派生量)。
cap = get_capability("MiniMax-M3") # (none, auto): 能开能关,但没有强度档
样本 2026-09-05 由 MiniMax-M3 换成 glm-4.6v: T10 实测 M3 的六个强度值全部生效,
它不再是纯开关型;glm-4.6v 是实测证据最硬的 (none, auto) 模型,且 zhipu 的 wire
有 effort_key——这两点缺一不可,否则命中的是"该 provider 没有档位键"那条分支。
"""
cap = get_capability("glm-4.6v") # (none, auto): 能开能关,但没有强度档
with pytest.raises(ThinkingUnsupportedError) as exc: with pytest.raises(ThinkingUnsupportedError) as exc:
resolve_thinking(get_provider("minimax"), cap, Effort.HIGH, model="MiniMax-M3") resolve_thinking(get_provider("zhipu"), cap, Effort.HIGH, model="glm-4.6v")
msg = str(exc.value) msg = str(exc.value)
assert "可选档位" not in msg assert "可选档位" not in msg
assert "该模型只有开关" in msg assert "该模型只有开关" in msg
@@ -328,10 +333,10 @@ class TestResolveThinking:
) )
with pytest.raises(ThinkingUnsupportedError) as switch: with pytest.raises(ThinkingUnsupportedError) as switch:
resolve_thinking( resolve_thinking(
get_provider("minimax"), get_provider("zhipu"),
get_capability("MiniMax-M3"), get_capability("glm-4.6v"),
Effort.MEDIUM, Effort.MEDIUM,
model="MiniMax-M3", model="glm-4.6v",
) )
assert str(tiered.value) != str(switch.value) assert str(tiered.value) != str(switch.value)
@@ -386,12 +391,12 @@ class TestResolveThinking:
def test_nearest_never_turns_reasoning_off(self): def test_nearest_never_turns_reasoning_off(self):
"""请求"想得浅一点"绝不能被映射成"别想了": 那是方向反转,不是省钱。""" """请求"想得浅一点"绝不能被映射成"别想了": 那是方向反转,不是省钱。"""
cap = get_capability("MiniMax-M3") # (none, auto) cap = get_capability("glm-4.6v") # (none, auto)
got = resolve_thinking( got = resolve_thinking(
get_provider("minimax"), cap, Effort.HIGH, model="MiniMax-M3", fallback="nearest" get_provider("zhipu"), cap, Effort.HIGH, model="glm-4.6v", fallback="nearest"
) )
assert got.applied_effort is Effort.AUTO assert got.applied_effort is Effort.AUTO
assert got.payload == {} assert got.payload == {"thinking": {"type": "enabled"}}
def test_nearest_still_errors_when_no_on_tier_exists(self): def test_nearest_still_errors_when_no_on_tier_exists(self):
"""只能关不能开的模型,映射无解——报错而非挑一个反向的档。""" """只能关不能开的模型,映射无解——报错而非挑一个反向的档。"""