test: replace the guessed tier table with what the gateway actually does
T10 经 new-api 中转对 26 个可调用模型逐个实测(约 500 次真实调用),把 DEFAULT_CAPABILITIES 里的文档推定换成实测结论,并把实测本身固化成可复跑、 可只跑单模型的 slow 用例。 三处与文档推定不符: - kimi-k3 实测可关(关闭档 completion 恒 9 且与 max 档锚点可分),T1 的保守 登记被推翻,OpenRouter 的 mandatory:false 胜出; - MiniMax-M3 的 auto 实测不推理(minimax 的"开"是不注入任何参数,而 M3 默认档 本就不推理),六个强度值反而全部生效且彼此等价,故清单由 (none, auto) 换成 none + 六个强度值; - glm-5.3 关不掉推理坐实(请求 none 后 4/5 轮仍观测到推理),issue #20 的核心 争议就此了结——当时短提示词下的 rt≈1.2 是采样噪声。 新增 kimi-for-coding(现有该型号自己的实测证据,不再走 Phase 3)。 用例带三条判据: 关闭方向要求每轮未观测到推理;短提示词的"关掉了"必须过长 上下文复核(glm-5.3-flash 正是短 5/5 未观测、长上下文 2/3 露馅);上游整片 不回传推理信号时另取无魔数锚点,不许把"没看见"写成"没发生"。另加一道模型 身份核对——该渠道把 glm-5/5.1/5.2 全部路由到 glm-5.3,那三组数据一律作废。 未覆盖 9 个型号(claude 三个 7 天限额用尽、gemini 两个上游报错、gpt-5.4 限流、 glm 三个被路由),evidence 逐条写明原因,不得被后续文档写成"已实测"。 单元测试里三条以 MiniMax-M3 当"纯开关型"样本的用例改用 glm-4.6v——实测后 M3 不再是那个形状,而 glm-4.6v 是全表证据最硬的 (none, auto)。
This commit is contained in:
+158
-60
@@ -108,111 +108,218 @@ class ThinkingCapability:
|
|||||||
return any(e not in (Effort.NONE, Effort.AUTO) for e in self.supported_efforts)
|
return any(e not in (Effort.NONE, Effort.AUTO) for e in self.supported_efforts)
|
||||||
|
|
||||||
|
|
||||||
# 证据分两类,evidence 里必须自报家门:
|
# 证据分三类,evidence 里必须自报家门:
|
||||||
# 实测 = 经 new-api 中转打过真实请求(最硬,不得被文档推定覆盖);
|
# 实测 = 经 new-api 中转打过真实请求(最硬,不得被文档推定覆盖);
|
||||||
# 文档推定 = 官方文档 / OpenRouter / cherry-studio / LiteLLM 四方交叉(待实测校正)。
|
# 文档推定 = 官方文档 / OpenRouter / cherry-studio / LiteLLM 四方交叉;
|
||||||
|
# 实测未覆盖 = T10 试过但拿不到数据(渠道限额/上游报错/被路由到别的模型),
|
||||||
|
# 此时**必须写明原因**——"没测到"与"测了没问题"是两回事,T9 之类的下游
|
||||||
|
# 文档任务不得把前者写成后者。
|
||||||
_MEASURED = "2026-08-02 经 new-api 中转实测"
|
_MEASURED = "2026-08-02 经 new-api 中转实测"
|
||||||
_DOC = "2026-09-04 文档推定(官方文档 + OpenRouter + cherry-studio + LiteLLM 四方交叉),待经 new-api 实测"
|
_T10 = "2026-09-05 经 new-api 中转实测(T10: 短提示词 N=5,声称可关的再加长上下文 N=3 复核)"
|
||||||
|
_DOC = "2026-09-04 文档推定(官方文档 + OpenRouter + cherry-studio + LiteLLM 四方交叉)"
|
||||||
|
|
||||||
|
# T10 的三条判据(报告见 tests/outputs/thinking/,用例见 tests/e2e/test_thinking_live.py):
|
||||||
|
# ① 关闭方向要求**每轮**未观测到推理,任一轮观测到即证伪;
|
||||||
|
# ② 短提示词下的"关掉了"必须过长上下文复核——glm-5.3-flash 正是短提示词 5/5
|
||||||
|
# 未观测到推理、5000 token 长上下文下 2/3 轮露馅(issue #20 的原始现象);
|
||||||
|
# ③ 上游整片不回传推理信号(kimi/MiniMax/qwen/gpt 这几路的关闭档都是)时,
|
||||||
|
# "没看见"不算"没发生",另取一个无魔数锚点: 关闭档的 completion_tokens
|
||||||
|
# 必须严格小于 max 档。
|
||||||
|
|
||||||
DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
|
DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
|
||||||
{
|
{
|
||||||
# —— 实测条目(2026-08-02/08-25),证据原文保留 ——
|
# —— MiniMax ——
|
||||||
"MiniMax-M3": ThinkingCapability(
|
"MiniMax-M3": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
supported_efforts=(
|
||||||
|
Effort.NONE,
|
||||||
|
Effort.MINIMAL,
|
||||||
|
Effort.LOW,
|
||||||
|
Effort.MEDIUM,
|
||||||
|
Effort.HIGH,
|
||||||
|
Effort.XHIGH,
|
||||||
|
Effort.MAX,
|
||||||
|
),
|
||||||
evidence=(
|
evidence=(
|
||||||
"2026-08-02 经 new-api 中转实测 N=10: reasoning_effort=none 稳定关闭,零跳变;"
|
f"{_T10}: reasoning_effort=none 关闭成立(短 5/5 + 长上下文 3/3 未观测到推理,"
|
||||||
"2026-08-25 复测依然成立(prompt 194 = 基线、completion 3、无推理正文)。"
|
"completion 恒 3 token,且与 max 档 completion 57-173 锚点可分);六个强度值各 N=5 "
|
||||||
"两条限制(findings 2026-08-25-thinking-observability-regression §3.1/§5): "
|
"全部观测到推理,rt 分布完全重叠(minimal 64-124 / low 55-112 / medium 51-104 / "
|
||||||
"① 非流式路径观测不到推理信号——推理已计费,但正文与 usage 明细都不回传;"
|
"high 62-128 / xhigh 58-118 / max 57-170)——**它们是'开'的六种写法,不是六个深度档**,"
|
||||||
"② enable_thinking / thinking:{type:enabled} 对本模型无效,仅 reasoning_effort 是真开关。"
|
"MiniMax 官方只有开/关两态,配哪一个都一样贵。"
|
||||||
"无强度档: 官方只有开/关两态(thinking.type disabled/adaptive)"
|
"**`auto` 已从清单移除**: minimax 的'开'在 wire 上是 on_base={}(什么参数都不注入),"
|
||||||
|
"而 M3 的默认档实测不推理,故 auto 在这条路上表达不了'开'(N=5 全部未观测到推理)。"
|
||||||
|
"注意 `resolve_thinking` 的 Phase 5 无条件放行 auto,故存量 ENABLE_THINKING=true 仍会"
|
||||||
|
"静默落到不推理——那是 wire 的缺口,能力表堵不住,已记入 T10 报告待单独立项。"
|
||||||
|
f"历史: {_MEASURED} N=10 同样成立;enable_thinking / thinking:{{type}} 两种写法对本模型"
|
||||||
|
"无效,reasoning_effort 才是真开关(findings 2026-08-25 §3.1/§5)。"
|
||||||
|
"另: 2026-08-25 记录的'MiniMax 这一路已停报 completion_tokens_details'本次**不再成立**"
|
||||||
|
"——开启档 rt 有值,只有关闭档整片缺 details"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"MiniMax-M2.7": ThinkingCapability(
|
"MiniMax-M2.7": ThinkingCapability(
|
||||||
supported_efforts=(Effort.AUTO,),
|
supported_efforts=(Effort.AUTO,),
|
||||||
evidence=(
|
evidence=(
|
||||||
"2026-08-02 实测 reasoning_effort=none / thinking:{disabled} / thinking:{adaptive} "
|
f"{_T10}: 请求 none 时 5/5 轮仍观测到推理(rt 100-161、推理正文 274-482 字符),"
|
||||||
"各 N=3 全部无效;OpenRouter 注册表登记 mandatory:true,models.dev 登记无控制手段。"
|
"**关不掉**成立;auto 档 5/5 观测到推理。"
|
||||||
"MiniMax 官方亦承认 M2.x 接受 disabled 但推理仍开着"
|
f"历史({_MEASURED}): reasoning_effort=none / thinking:{{disabled}} / thinking:{{adaptive}} "
|
||||||
|
"各 N=3 全部无效;OpenRouter 登记 mandatory:true,models.dev 登记无控制手段"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"MiniMax-M2.5": ThinkingCapability(
|
"MiniMax-M2.5": ThinkingCapability(
|
||||||
supported_efforts=(Effort.AUTO,),
|
supported_efforts=(Effort.AUTO,),
|
||||||
evidence="2026-08-02 实测同 M2.7: 三种形态各 N=3 全部无效;外部注册表同样登记为强制推理",
|
evidence=(
|
||||||
|
f"{_T10}: 请求 none 时 5/5 轮仍观测到推理(rt 104-158),**关不掉**成立;"
|
||||||
|
"auto 档 5/5 观测到推理(rt 121-245)。"
|
||||||
|
f"历史({_MEASURED}): 三种形态各 N=3 全部无效;外部注册表同样登记为强制推理"
|
||||||
),
|
),
|
||||||
|
),
|
||||||
|
# —— qwen(百炼系,开关型) ——
|
||||||
"qwen3.7-plus": ThinkingCapability(
|
"qwen3.7-plus": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
evidence=(
|
evidence=(
|
||||||
"2026-08-02 实测 enable_thinking=false 关闭(completion 5 token,无推理)。"
|
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3 未观测到推理,且与 auto 档 completion 锚点可分);"
|
||||||
"无强度档: OpenRouter 登记本型号只支持 reasoning 开关,不支持 reasoning_effort"
|
"auto 档 5/5 观测到推理。无强度档: 该 provider 的 wire 没有 effort_key,请求 max 当场被库"
|
||||||
|
"拒(百炼靠 thinking_budget 调深度,预算型控制本库不支持)。"
|
||||||
|
f"历史({_MEASURED}): enable_thinking=false 关闭(completion 5 token)"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
"qwen3.7-max": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
|
evidence=f"{_T10}: 同 qwen3.7-plus——none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
|
||||||
|
),
|
||||||
|
"qwen3.6-plus": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
|
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
|
||||||
|
),
|
||||||
|
"qwen3.5-flash": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
|
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
|
||||||
|
),
|
||||||
|
"qwen-plus-latest": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
|
evidence=f"{_T10}: none 短 5/5 + 长 3/3 关闭且锚点可分,auto 档 5/5 观测到推理",
|
||||||
|
),
|
||||||
|
# —— deepseek ——
|
||||||
"deepseek-v4-pro": ThinkingCapability(
|
"deepseek-v4-pro": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
||||||
evidence=(
|
evidence=(
|
||||||
"关闭档为 2026-08-02 实测(thinking:{type:disabled},completion 3 token,无推理);"
|
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3 未观测到推理,与 max 档锚点可分);"
|
||||||
f"强度档为{_DOC}: 官方 thinking_mode 文档列 Non-think/Think High/Think Max 三态,默认 high"
|
"high / max 各 N=5 全部观测到推理(rt 59-73 / 56-69,推理正文 max 档明显更长: "
|
||||||
|
"135-186 vs 101-124 字符)。默认档按官方 thinking_mode 文档为 high"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
# —— 文档推定条目(2026-09-04),待 T10 经 new-api 实测校正 ——
|
|
||||||
"deepseek-v4-flash": ThinkingCapability(
|
"deepseek-v4-flash": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
||||||
evidence=f"{_DOC}: 官方文档「deepseek-v4-flash 与 deepseek-v4-pro 一致」,默认 high",
|
evidence=(
|
||||||
|
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3,锚点可分);high / max 各 N=5 全部观测到推理"
|
||||||
|
"(rt 16-36 / 12-42)。与 v4-pro 同档,印证官方'与 deepseek-v4-pro 一致'的说法"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
"deepseek-v4-flash-vision-exp": ThinkingCapability(
|
"deepseek-v4-flash-vision-exp": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
||||||
evidence=f"{_DOC}: 同 v4-flash 一档(OpenRouter 登记支持 reasoning_effort)",
|
evidence=(
|
||||||
|
f"{_T10}: none 关闭成立(短 5/5 + 长 3/3,锚点可分);high / max 各 N=5 全部观测到推理"
|
||||||
|
"(rt 12-17 / 17-34)"
|
||||||
),
|
),
|
||||||
|
),
|
||||||
|
# —— 智谱 ——
|
||||||
"glm-5.3": ThinkingCapability(
|
"glm-5.3": ThinkingCapability(
|
||||||
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
|
||||||
evidence=(
|
evidence=(
|
||||||
f"{_DOC}: **推理不可关闭**——智谱官方文档明确 thinking.type 只接受 enabled,"
|
f"{_T10}: **推理不可关闭已实测坐实**——请求 none(注入 thinking:{{type:disabled}})后 "
|
||||||
"官方迁移建议是改用 enabled + reasoning_effort=low;cherry-studio 标 toggle:false、"
|
"5 轮里 4 轮仍观测到推理(rt 7、推理正文 12 字符),只有 1 轮 rt=0;"
|
||||||
"OpenRouter 标 mandatory:true,三源一致。默认 max。"
|
"low / high / max 各 N=5 全部观测到推理(rt 55-77 / 47-63 / 48-60,三档分不出深浅)。"
|
||||||
"注: issue #20 实测的 reasoning_effort=none 是**未定义值**,短提示词下 rt≈1.2 像是关了,"
|
"这一条了结了 issue #20 的核心争议: 当时短提示词下 rt≈1.2 看着像关掉了,实为采样噪声。"
|
||||||
"5552 token 长上下文下跳到 0/54/167 即露馅"
|
f"文档侧三源一致({_DOC}): 智谱官方 thinking.type 只接受 enabled、迁移建议改用 "
|
||||||
|
"enabled + reasoning_effort=low,cherry-studio 标 toggle:false,OpenRouter 标 mandatory:true。"
|
||||||
|
"默认 max。**注意本渠道不校验档位值**: 未登记的 medium 也会被照单接受(实测 rt 62),"
|
||||||
|
"故'网关没报错'在这一路上不构成'该档受支持'的证据"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"glm-5.3-flash": ThinkingCapability(
|
"glm-5.3-flash": ThinkingCapability(
|
||||||
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
|
||||||
evidence=f"{_DOC}: 同 glm-5.3(cherry-studio 的 pattern 'glm-5[.-]3' 覆盖两者),默认 max",
|
evidence=(
|
||||||
|
f"{_T10}: **推理不可关闭,且是判据②唯一的现役样本**——请求 none 时短提示词 5/5 轮"
|
||||||
|
"未观测到推理(看着完全像关掉了),换成 5000 token 长上下文后 3 轮里 2 轮露馅"
|
||||||
|
"(rt=2、有推理正文)。只跑短提示词的实测会在这个模型上得出相反结论。"
|
||||||
|
"low / high / max 各 N=5 全部观测到推理(rt 8-60 / 27-91 / 27-70)。默认 max"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
"glm-5.2": ThinkingCapability(
|
"glm-5.2": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
supported_efforts=(Effort.NONE, Effort.HIGH, Effort.MAX),
|
||||||
evidence=(
|
evidence=(
|
||||||
f"{_DOC}: cherry-studio 登记 none/high/max(官方端点默认 max,百炼上默认 high)。"
|
f"{_DOC}: cherry-studio 登记 none/high/max(官方端点默认 max,百炼上默认 high)。"
|
||||||
"注意: issue #20 记录本渠道对 glm-5.2 的请求 6/6 回报 model=glm-5.3,疑被路由,实测时须核对 model_reported"
|
"**T10 实测未覆盖——该渠道把本型号路由到了别的模型**: 请求 glm-5.2 时 5/5 轮回报 "
|
||||||
|
"model=glm-5.3(issue #20 记录的 6/6 复现),拿到的行为不属于本型号,故整组数据作废、"
|
||||||
|
"本行仍是文档推定。**下游风险**: 在本渠道上给 glm-5.2 配 none,库会照本行放行,"
|
||||||
|
"而真正服务请求的 glm-5.3 关不掉推理——运行期 reconcile 会喊,但那是事后"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"glm-5": ThinkingCapability(
|
"glm-5": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
evidence=f"{_DOC}: OpenRouter 登记只支持 reasoning 开关、无 reasoning_effort;cherry-studio 标 toggle:true",
|
evidence=(
|
||||||
|
f"{_DOC}: OpenRouter 登记只支持 reasoning 开关、无 reasoning_effort;cherry-studio 标 toggle:true。"
|
||||||
|
"**T10 实测未覆盖**: 与 glm-5.2 同因——5/5 轮回报 model=glm-5.3,数据不属于本型号"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
"glm-5.1": ThinkingCapability(
|
"glm-5.1": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
evidence=f"{_DOC}: 同 glm-5(OpenRouter reasoning.mandatory=false 且无 supported_efforts)",
|
evidence=(
|
||||||
|
f"{_DOC}: 同 glm-5(OpenRouter reasoning.mandatory=false 且无 supported_efforts)。"
|
||||||
|
"**T10 实测未覆盖**: 5/5 轮回报 model=glm-5.3,数据不属于本型号"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
"glm-4.6v": ThinkingCapability(
|
"glm-4.6v": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
supported_efforts=(Effort.NONE, Effort.AUTO),
|
||||||
evidence=f"{_DOC}: OpenRouter 登记无 reasoning_effort;VLM,推理控制同 glm-4.x 系开关型",
|
|
||||||
),
|
|
||||||
"kimi-k3": ThinkingCapability(
|
|
||||||
supported_efforts=(Effort.LOW, Effort.HIGH, Effort.MAX),
|
|
||||||
evidence=(
|
evidence=(
|
||||||
f"{_DOC}: 官方 reasoning_effort 三档 low/high/max,默认 max。"
|
f"{_T10}: none 关闭成立,且是全表**证据最硬**的一条——短 5/5 + 长 3/3 全部裁定 ABSENT"
|
||||||
"**保守登记为不可关**——官方档位表无 none,而 OpenRouter 标 mandatory:false,两源分歧待实测;"
|
"(上游明确上报 reasoning_tokens=0,不是'看不见'),无需锚点旁证;"
|
||||||
"保守方向的代价是下游配 none 会报错并被指向 low,反方向的代价是静默失效(issue #20 的病)。"
|
"auto 档 5/5 观测到推理(rt 57-153)。model_reported 与请求一致,未被路由"
|
||||||
"另: 官方提示切换档位会使 prefix cache 失效,不宜在会话中途改档"
|
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
|
# —— 月之暗面 ——
|
||||||
|
"kimi-k3": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.LOW, Effort.HIGH, Effort.MAX),
|
||||||
|
evidence=(
|
||||||
|
f"{_T10}: **可关闭——推翻 T1 的保守登记**。请求 none(注入 thinking:{{type:disabled}})后"
|
||||||
|
"短 5/5 + 长上下文 3/3 轮无任何推理信号,completion 恒 9 token;同一模型 max 档 "
|
||||||
|
"completion 明显更大且带推理正文(rt 33-146),锚点可分——故'没看见'这次有正面证据支撑。"
|
||||||
|
"两源分歧由此了结: OpenRouter 的 mandatory:false 是对的,官方档位表没列 none 只是没列。"
|
||||||
|
"low / high / max 各 N=5 全部观测到推理(rt 21-53 / 38-60 / 33-146)。"
|
||||||
|
"**model_reported 是 `k3`**(别名,非串台)。官方提示切换档位会使 prefix cache 失效,"
|
||||||
|
"不宜在会话中途改档"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
"kimi-for-coding": ThinkingCapability(
|
||||||
|
supported_efforts=(Effort.NONE, Effort.LOW, Effort.HIGH, Effort.MAX),
|
||||||
|
evidence=(
|
||||||
|
f"{_T10}: 本型号在 T1 时因'档位清单无直接证据'走 Phase 3 不登记(设计 §8 第三档),"
|
||||||
|
"现有它自己的实测证据故补登。none: 短 5/5 + 长 3/3 无推理信号、completion 恒 2 token,"
|
||||||
|
"与开启档锚点可分;low / high / max 各 N=3 全部观测到推理(rt 8-40 / 25-75 / 62-85)。"
|
||||||
|
"档位词汇沿用月之暗面官方的 low/high/max: 本渠道对 moonshot **不校验档位值**"
|
||||||
|
"(minimal/medium/xhigh 照样返回 200 并推理),故'没被拒'不构成'受支持',"
|
||||||
|
"登记一个厂商没声明的档等于替它做承诺"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
# —— OpenAI ——
|
||||||
"gpt-5.4": ThinkingCapability(
|
"gpt-5.4": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
|
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
|
||||||
evidence=f"{_DOC}: OpenRouter 登记 none/low/medium/high/xhigh,默认 medium;LiteLLM 登记 minimal 不支持",
|
evidence=(
|
||||||
|
f"{_DOC}: OpenRouter 登记 none/low/medium/high/xhigh,默认 medium;LiteLLM 登记 minimal 不支持。"
|
||||||
|
"**T10 实测未覆盖**: 该渠道本型号所有账号限流(429 All available accounts are "
|
||||||
|
"currently rate-limited),5/5 轮失败。同代的 gpt-5.5 已实测且与本清单逐字相符"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
"gpt-5.5": ThinkingCapability(
|
"gpt-5.5": ThinkingCapability(
|
||||||
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
|
supported_efforts=(Effort.NONE, Effort.LOW, Effort.MEDIUM, Effort.HIGH, Effort.XHIGH),
|
||||||
evidence=f"{_DOC}: 同 gpt-5.4(OpenRouter supported_efforts 一致,默认 medium)",
|
evidence=(
|
||||||
|
f"{_T10}: 清单**逐条对上**,是全表验证最完整的一行。none 关闭成立(短 5/5 + 长 3/3,"
|
||||||
|
"锚点可分);low/medium/high/xhigh 各 N=5 全部观测到推理,且 rt 随档位单调上升"
|
||||||
|
"(18-21 / 18-22 / 22-34 / 35-65)——本渠道上少见的、档位真的分得开的模型;"
|
||||||
|
"清单外的 max 与 minimal 各 N=3 全部被上游 400 拒("
|
||||||
|
"Unsupported value),说明这一路**会校验档位值**,与 zhipu/moonshot 的照单全收相反"
|
||||||
),
|
),
|
||||||
|
),
|
||||||
|
# —— Anthropic ——
|
||||||
"claude-opus-5": ThinkingCapability(
|
"claude-opus-5": ThinkingCapability(
|
||||||
supported_efforts=(
|
supported_efforts=(
|
||||||
Effort.NONE,
|
Effort.NONE,
|
||||||
@@ -225,7 +332,8 @@ DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
|
|||||||
evidence=(
|
evidence=(
|
||||||
f"{_DOC}: Anthropic 官方 adaptive thinking + output_config.effort 五档(low/medium/high/"
|
f"{_DOC}: Anthropic 官方 adaptive thinking + output_config.effort 五档(low/medium/high/"
|
||||||
"xhigh/max),默认 high;OpenRouter 标 mandatory:false 故可关。"
|
"xhigh/max),默认 high;OpenRouter 标 mandatory:false 故可关。"
|
||||||
"关闭档依赖 new-api 把 reasoning_effort=none 转成 thinking 关闭形态,待实测确认"
|
"**T10 实测未覆盖**: 该渠道 claude 全系返回 429「api key 7天限额已用完」,5/5 轮失败。"
|
||||||
|
"关闭档仍依赖 new-api 把 reasoning_effort=none 转成 thinking 关闭形态,未经验证"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"claude-sonnet-5": ThinkingCapability(
|
"claude-sonnet-5": ThinkingCapability(
|
||||||
@@ -237,32 +345,22 @@ DEFAULT_CAPABILITIES: Mapping[str, ThinkingCapability] = MappingProxyType(
|
|||||||
Effort.XHIGH,
|
Effort.XHIGH,
|
||||||
Effort.MAX,
|
Effort.MAX,
|
||||||
),
|
),
|
||||||
evidence=f"{_DOC}: 同 claude-opus-5(OpenRouter supported_efforts 与默认档一致)",
|
evidence=(
|
||||||
|
f"{_DOC}: 同 claude-opus-5(OpenRouter supported_efforts 与默认档一致)。"
|
||||||
|
"**T10 实测未覆盖**: 同因 429「api key 7天限额已用完」"
|
||||||
),
|
),
|
||||||
|
),
|
||||||
|
# —— Google ——
|
||||||
"gemini-3.1-pro": ThinkingCapability(
|
"gemini-3.1-pro": ThinkingCapability(
|
||||||
supported_efforts=(Effort.LOW, Effort.MEDIUM, Effort.HIGH),
|
supported_efforts=(Effort.LOW, Effort.MEDIUM, Effort.HIGH),
|
||||||
evidence=(
|
evidence=(
|
||||||
f"{_DOC}: **推理不可关闭**——Google 官方文档明确 Gemini 3 Pro / 3.1 Pro 无法关闭思考,"
|
f"{_DOC}: **推理不可关闭**——Google 官方文档明确 Gemini 3 Pro / 3.1 Pro 无法关闭思考,"
|
||||||
"OpenRouter 亦标 mandatory:true。thinking_level 三档;默认档两源打架"
|
"OpenRouter 亦标 mandatory:true。thinking_level 三档;默认档两源打架"
|
||||||
"(官方文档说 HIGH,OpenRouter 说 medium),待实测"
|
"(官方文档说 HIGH,OpenRouter 说 medium)。"
|
||||||
|
"**T10 实测未覆盖**: 该渠道本型号上游报错(bad_response_status_code / openai_error),"
|
||||||
|
"5/5 轮失败,连默认档基线都没取到,两源分歧仍悬着"
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
"qwen-plus-latest": ThinkingCapability(
|
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
|
||||||
evidence=f"{_DOC}: 百炼 enable_thinking 开关型(thinking_budget 是 token 预算,本库不支持预算型)",
|
|
||||||
),
|
|
||||||
"qwen3.5-flash": ThinkingCapability(
|
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
|
||||||
evidence=f"{_DOC}: 同 qwen-plus-latest(OpenRouter 登记无 reasoning_effort)",
|
|
||||||
),
|
|
||||||
"qwen3.6-plus": ThinkingCapability(
|
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
|
||||||
evidence=f"{_DOC}: 同 qwen-plus-latest(OpenRouter 登记无 reasoning_effort)",
|
|
||||||
),
|
|
||||||
"qwen3.7-max": ThinkingCapability(
|
|
||||||
supported_efforts=(Effort.NONE, Effort.AUTO),
|
|
||||||
evidence=f"{_DOC}: 同 qwen3.7-plus 一代(OpenRouter 登记无 reasoning_effort)",
|
|
||||||
),
|
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
"""在用模型的推理能力登记(YAGNI: 不覆盖全世界,未登记走 `resolve_thinking` 退化)。"""
|
"""在用模型的推理能力登记(YAGNI: 不覆盖全世界,未登记走 `resolve_thinking` 退化)。"""
|
||||||
|
|||||||
@@ -26,12 +26,15 @@
|
|||||||
源不可用一律 `skip` 并在报告中记为「未覆盖」,**绝不静默计入通过**。
|
源不可用一律 `skip` 并在报告中记为「未覆盖」,**绝不静默计入通过**。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import dataclasses
|
import dataclasses
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from collections import Counter
|
from collections import Counter
|
||||||
|
from collections.abc import Mapping
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from dotenv import dotenv_values
|
from dotenv import dotenv_values
|
||||||
@@ -39,11 +42,13 @@ from dotenv import dotenv_values
|
|||||||
from polygateway import GatewayClient, GatewaySettings, ThinkingObservation
|
from polygateway import GatewayClient, GatewaySettings, ThinkingObservation
|
||||||
from polygateway.errors import (
|
from polygateway.errors import (
|
||||||
AllSourcesExhausted,
|
AllSourcesExhausted,
|
||||||
|
GatewayUnavailableError,
|
||||||
RequestRejectedError,
|
RequestRejectedError,
|
||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.thinking import DEFAULT_CAPABILITIES, get_capability
|
from polygateway.thinking import DEFAULT_CAPABILITIES, ThinkingCapability, get_capability
|
||||||
|
from polygateway.types import EFFORT_ORDER, Effort
|
||||||
|
|
||||||
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
|
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
|
||||||
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
|
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
|
||||||
@@ -77,7 +82,28 @@ _MODEL_PROVIDER = {
|
|||||||
"MiniMax-M2.7": "minimax",
|
"MiniMax-M2.7": "minimax",
|
||||||
"MiniMax-M2.5": "minimax",
|
"MiniMax-M2.5": "minimax",
|
||||||
"qwen3.7-plus": "qwen",
|
"qwen3.7-plus": "qwen",
|
||||||
|
"qwen3.7-max": "qwen",
|
||||||
|
"qwen3.6-plus": "qwen",
|
||||||
|
"qwen3.5-flash": "qwen",
|
||||||
|
"qwen-plus-latest": "qwen",
|
||||||
"deepseek-v4-pro": "deepseek",
|
"deepseek-v4-pro": "deepseek",
|
||||||
|
"deepseek-v4-flash": "deepseek",
|
||||||
|
"deepseek-v4-flash-vision-exp": "deepseek",
|
||||||
|
"glm-5.3": "zhipu",
|
||||||
|
"glm-5.3-flash": "zhipu",
|
||||||
|
"glm-5.2": "zhipu",
|
||||||
|
"glm-5.1": "zhipu",
|
||||||
|
"glm-5": "zhipu",
|
||||||
|
"glm-4.6v": "zhipu",
|
||||||
|
"kimi-k3": "moonshot",
|
||||||
|
"kimi-for-coding": "moonshot",
|
||||||
|
"gpt-5.4": "openai",
|
||||||
|
"gpt-5.5": "openai",
|
||||||
|
"claude-opus-5": "anthropic",
|
||||||
|
"claude-sonnet-5": "anthropic",
|
||||||
|
"claude-haiku-5": "anthropic",
|
||||||
|
"gemini-3.1-pro": "google",
|
||||||
|
"gemini-3-flash": "google",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -519,3 +545,528 @@ class TestAssemblyGuardAgainstRealConfig:
|
|||||||
finally:
|
finally:
|
||||||
await client.aclose()
|
await client.aclose()
|
||||||
_record("L9", "绕过装配守卫时 transport 兜底", "PASS", "RequestRejectedError,属四分类")
|
_record("L9", "绕过装配守卫时 transport 兜底", "PASS", "RequestRejectedError,属四分类")
|
||||||
|
|
||||||
|
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
# T10: 逐模型档位实测(方法论沿用 issue #20)
|
||||||
|
#
|
||||||
|
# 本节与上面的 L1-L9 分工不同: 上面验的是**库的行为**(注入到没到、观测准不准),
|
||||||
|
# 这里验的是**能力表的内容**(`DEFAULT_CAPABILITIES` 里那 20 多条声明是不是真的)。
|
||||||
|
# 二者判据可以共用,数据源却必须分开——能力表实测要**绕过能力表**才有意义,
|
||||||
|
# 否则拿待验证的声明去挡请求,等于用结论证明前提。
|
||||||
|
#
|
||||||
|
# 判据(三条,均沿用已有纪律):
|
||||||
|
# ① 关闭方向: 每轮 `thinking_observation != OBSERVED` 才算真关掉;任一轮
|
||||||
|
# OBSERVED 即证伪(推理正文是事实本身,不需要多数票)。
|
||||||
|
# ② **短提示词的"关掉了"必须经长上下文复核**: issue #20 实测 GLM 系在短提示词
|
||||||
|
# 下 reasoning_tokens≈1.2 像是关了,5552 token 长上下文下跳到 0/54/167 即露馅。
|
||||||
|
# 短提示词下推理量本就趋近于 0,分不出"关了"与"没什么可想的"。
|
||||||
|
# ③ 开启方向: 多数轮 OBSERVED(单轮抖动不判红,与 L2 同口径)。
|
||||||
|
# ④ 关闭结论**不许只靠 `UNKNOWN`**: 上游整片不回传推理信号时(kimi、MiniMax 两路
|
||||||
|
# 都是),"没看见"不是"没发生"。此时补一个不含魔数的锚点——关闭档的
|
||||||
|
# `completion_tokens` 必须严格小于 `max` 档,否则结论记为「判不出来」。
|
||||||
|
# ══════════════════════════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
_TIER_OUT_DIR = Path("tests/outputs/thinking")
|
||||||
|
_TIER_ROUNDS = int(os.environ.get("PGW_E2E_TIER_ROUNDS", "5"))
|
||||||
|
_TIER_LONG_ROUNDS = int(os.environ.get("PGW_E2E_TIER_LONG_ROUNDS", "3"))
|
||||||
|
# 共用生产网关,宁慢勿冲(人类 2026-09-05 指令): 默认 3,可下调不建议上调
|
||||||
|
_TIER_CONCURRENCY = int(os.environ.get("PGW_E2E_TIER_CONCURRENCY", "3"))
|
||||||
|
|
||||||
|
# 固定短提示词: 答案本身约 4 token,推理 token 的信噪比高(issue #20 同款)
|
||||||
|
_TIER_PROMPT = "23 乘以 47 等于多少?只回答一个数字,不要解释。"
|
||||||
|
|
||||||
|
# 长上下文对照组(判据②)。填充文本与题目无关且不含任何业务领域词汇(零业务假设
|
||||||
|
# 铁律),只为把输入撑到数千 token;题目放在最后,避免被当成"读完就忘"的前缀
|
||||||
|
_TIER_LONG_PROMPT = (
|
||||||
|
"\n".join(
|
||||||
|
f"{i:04d}. 这是一段与题目无关的填充文字,仅用于把上下文撑到数千 token,"
|
||||||
|
"以复核短提示词下得到的关闭结论在长上下文下是否依然成立。"
|
||||||
|
for i in range(120)
|
||||||
|
)
|
||||||
|
+ "\n\n"
|
||||||
|
+ _TIER_PROMPT
|
||||||
|
)
|
||||||
|
|
||||||
|
_ALL_EFFORTS: tuple[Effort, ...] = (*EFFORT_ORDER, Effort.AUTO)
|
||||||
|
|
||||||
|
_PROBE_ROWS: list[dict] = []
|
||||||
|
|
||||||
|
|
||||||
|
def _tier_settings(model: str) -> GatewaySettings:
|
||||||
|
"""探测用配置: 生产口径的超时,但**重试预算压到 1 次**。
|
||||||
|
|
||||||
|
压重试是因为探测里"这一轮失败"本身就是数据(逐轮进报告),库替它重试只会
|
||||||
|
把"渠道当下不可用"变成三倍等待——2026-09-05 实测 claude 系 7 天限额用尽时
|
||||||
|
每轮 429,三次重试让单个模型阻塞三分钟以上,26 个模型跑不完。
|
||||||
|
|
||||||
|
**单次请求的超时不动**(仍是 .env 的生产值 300s): §4.6 那条"测试超时不得紧于
|
||||||
|
生产配置"防的是把慢而正常的模型误判成不可用,那个风险在这里照旧存在。重试次数
|
||||||
|
与背压窗口不属于同一类——它们决定"失败之后还等多久",而不是"多慢算失败";
|
||||||
|
一个真在出字的模型永远碰不到这两者。
|
||||||
|
"""
|
||||||
|
base = GatewaySettings.from_env(
|
||||||
|
"LLM",
|
||||||
|
env={
|
||||||
|
**_ENV,
|
||||||
|
"PGW_CACHE_BACKEND": "none",
|
||||||
|
"LLM_MAX_RETRIES": "1",
|
||||||
|
# 探测是**单源**的,没有别的源可换。生产值 1200s 的 stall window 在这里
|
||||||
|
# 只会把"这个模型当下不可用"拖成 20 分钟一轮: 2026-09-05 实测 claude 系
|
||||||
|
# 7 天限额用尽返回 429 且不带 Retry-After,库据此判"无可运行源"并按背压
|
||||||
|
# 语义等到窗口耗尽(实测把窗口调到 45s 即在 46.7s 报 stalled)。多源生产
|
||||||
|
# 场景下这段等待是有意义的(等别的源恢复),探测场景下等不到任何东西
|
||||||
|
"LLM__BACKPRESSURE__STALL_WINDOW_S": "60",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
source = dataclasses.replace(
|
||||||
|
base.sources[0],
|
||||||
|
provider=_MODEL_PROVIDER[model],
|
||||||
|
model=model,
|
||||||
|
enable_thinking=None,
|
||||||
|
reasoning_effort=None,
|
||||||
|
)
|
||||||
|
return dataclasses.replace(base, sources=(source,))
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_capabilities(model: str) -> dict[str, ThinkingCapability]:
|
||||||
|
"""临时全档能力表: **实测的对象正是能力表本身**,不能拿它当前提去挡请求。
|
||||||
|
|
||||||
|
不传 `capabilities={}`(即"未登记")的理由是噪声: 那条路会走 Phase 3,每轮都
|
||||||
|
warning 一句"能力未登记",几百轮下来把真正的告警淹没。全档表让五关全部放行,
|
||||||
|
请求原样发出去,由上游而不是由库来回答"这一档到底行不行"。
|
||||||
|
"""
|
||||||
|
return {model: ThinkingCapability(_ALL_EFFORTS, evidence="T10 实测临时表(不进 DEFAULT)")}
|
||||||
|
|
||||||
|
|
||||||
|
async def _probe_effort(
|
||||||
|
model: str, effort: Effort, *, rounds: int, prompt: str, prompt_kind: str
|
||||||
|
) -> list[dict]:
|
||||||
|
"""对一个 (模型, 档位) 打 N 轮真实请求,逐轮记录;失败轮记 `error` 而不冒泡。
|
||||||
|
|
||||||
|
失败不冒泡是本函数与 `_run_rounds` 的唯一区别: 这里"上游拒绝这一档"本身就是
|
||||||
|
**实测结论**(HTTP 400 = 该档不被接受),把它抛出去会让数据采集半途而废。
|
||||||
|
只吞四分类与 `AllSourcesExhausted`——库自身的 `ValueError` 等仍然冒泡,那是
|
||||||
|
bug 不是数据。
|
||||||
|
"""
|
||||||
|
client = GatewayClient.from_settings(
|
||||||
|
_tier_settings(model), capabilities=_probe_capabilities(model)
|
||||||
|
)
|
||||||
|
semaphore = asyncio.Semaphore(_TIER_CONCURRENCY)
|
||||||
|
|
||||||
|
async def _one(index: int) -> dict:
|
||||||
|
base = {"round": index + 1, "effort": effort.value, "prompt_kind": prompt_kind}
|
||||||
|
async with semaphore:
|
||||||
|
try:
|
||||||
|
resp = await client.chat(
|
||||||
|
[{"role": "user", "content": prompt}],
|
||||||
|
stream=True,
|
||||||
|
reasoning_effort=effort,
|
||||||
|
cache_salt=f"tier-probe-{model}-{effort.value}-{prompt_kind}-{index}",
|
||||||
|
)
|
||||||
|
except (
|
||||||
|
RequestRejectedError,
|
||||||
|
GatewayUnavailableError,
|
||||||
|
SourceDeadError,
|
||||||
|
TransientError,
|
||||||
|
) as exc:
|
||||||
|
# 捕 `GatewayUnavailableError` 而不是只捕 `AllSourcesExhausted`:
|
||||||
|
# 某个模型在网关上不通时,连续失败会把熔断门打开,后续轮次抛的是
|
||||||
|
# `CircuitOpenError`(同一父类的兄弟)。只捕子类会让"源不可用"这
|
||||||
|
# 件事在第 N 轮换个类型冒出去,把数据采集打断成一次红测
|
||||||
|
return {**base, "error": f"{type(exc).__name__}: {str(exc)[:160]}"}
|
||||||
|
return {
|
||||||
|
**base,
|
||||||
|
"error": None,
|
||||||
|
"prompt_tokens": resp.prompt_tokens,
|
||||||
|
"completion_tokens": resp.completion_tokens,
|
||||||
|
"reasoning_tokens": resp.reasoning_tokens,
|
||||||
|
"thinking_chars": len(resp.thinking),
|
||||||
|
"thinking_observation": resp.thinking_observation,
|
||||||
|
"applied_effort": resp.applied_effort,
|
||||||
|
# 核对模型身份: issue #20 记录本渠道对 glm-5.2 的请求 6/6 回报
|
||||||
|
# model=glm-5.3。凡结论依赖模型身份的,对不上即数据不可信
|
||||||
|
"model_reported": resp.model_reported,
|
||||||
|
"content": resp.content[:40],
|
||||||
|
}
|
||||||
|
|
||||||
|
try:
|
||||||
|
return list(await asyncio.gather(*(_one(i) for i in range(rounds))))
|
||||||
|
finally:
|
||||||
|
await client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_ok(obs: dict) -> bool:
|
||||||
|
return obs["error"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_quiet(obs: dict) -> bool:
|
||||||
|
"""成功且未观测到推理(判据①的满足条件);失败轮不算"安静",它没有观测。"""
|
||||||
|
return _probe_ok(obs) and obs["thinking_observation"] != ThinkingObservation.OBSERVED
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_observed(obs: dict) -> bool:
|
||||||
|
return _probe_ok(obs) and obs["thinking_observation"] == ThinkingObservation.OBSERVED
|
||||||
|
|
||||||
|
|
||||||
|
def _rt_summary(observations: list[dict]) -> str:
|
||||||
|
"""报告里的一行摘要: rt 观测值序列 + 裁定分布 + 身份核对,三样缺一不可复核。"""
|
||||||
|
ok = [o for o in observations if _probe_ok(o)]
|
||||||
|
if not ok:
|
||||||
|
return f"全部 {len(observations)} 轮失败: {observations[0]['error']}"
|
||||||
|
rts = [o["reasoning_tokens"] for o in ok]
|
||||||
|
verdicts = Counter(str(o["thinking_observation"]) for o in ok)
|
||||||
|
reported = sorted({str(o["model_reported"]) for o in ok})
|
||||||
|
failed = len(observations) - len(ok)
|
||||||
|
tail = f";{failed} 轮失败" if failed else ""
|
||||||
|
return (
|
||||||
|
f"rt={rts};裁定 {dict(verdicts)};thinking_chars="
|
||||||
|
f"{[o['thinking_chars'] for o in ok]};model_reported={reported}{tail}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# 已知的合法别名: 供应商回报的名字与配置里的别名本就可以不同(月之暗面回
|
||||||
|
# `k3`、Google 回 `-preview` 后缀)。**显式登记而不是按前缀猜**——猜的话
|
||||||
|
# `glm-5.2 → glm-5.3` 这种真·串台也会被当成"同族别名"放过,而那正是本表要抓的
|
||||||
|
_MODEL_REPORTED_ALIASES: Mapping[str, frozenset[str]] = MappingProxyType(
|
||||||
|
{
|
||||||
|
"kimi-k3": frozenset({"k3"}),
|
||||||
|
"kimi-for-coding": frozenset({"k3"}),
|
||||||
|
"gemini-3-flash": frozenset({"gemini-3-flash-preview"}),
|
||||||
|
"gemini-3.1-pro": frozenset({"gemini-3.1-pro-preview"}),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _identity_mismatch(model: str, observations: list[dict]) -> list[str]:
|
||||||
|
"""响应体里的 `model` 与请求的模型对不上 → 本次数据说的不是这个模型。
|
||||||
|
|
||||||
|
issue #20 就栽在这里: 该渠道对 `glm-5.2` 的请求 6/6 回报 `model=glm-5.3`,
|
||||||
|
照单全收的话,能力表里 glm-5.2 那一行记的其实是 glm-5.3 的行为。凡结论依赖
|
||||||
|
模型身份的,对不上就必须当场作废,而不是打个折扣继续用。
|
||||||
|
|
||||||
|
`None`(上游未上报)不算不符: 那是"没说",不是"说了别的"。
|
||||||
|
"""
|
||||||
|
allowed = {model, *_MODEL_REPORTED_ALIASES.get(model, frozenset())}
|
||||||
|
return sorted(
|
||||||
|
{
|
||||||
|
o["model_reported"]
|
||||||
|
for o in observations
|
||||||
|
if _probe_ok(o)
|
||||||
|
and o["model_reported"] is not None
|
||||||
|
and o["model_reported"] not in allowed
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _anchor_off_against_on(
|
||||||
|
model: str, off_observations: list[dict]
|
||||||
|
) -> tuple[list[dict], Effort | None, bool]:
|
||||||
|
"""判据④: 拿"开启档的 completion 明显更大"给关闭结论补一个正面证据。
|
||||||
|
|
||||||
|
需要它是因为 `UNKNOWN` 的语义: 它是"本次没有任何信号,判不出来",不是"没推理"
|
||||||
|
(`observe_thinking` 的 docstring 把这条写死了)。kimi 与 MiniMax 这两路上游都
|
||||||
|
不回传 `completion_tokens_details`,关闭档整片 `UNKNOWN`——此时若直接把"没看见"
|
||||||
|
读成"关掉了",库就会登记一个自己从未验证过的 `none`,而下游据此以为省了钱。
|
||||||
|
|
||||||
|
锚点取 `completion_tokens` 的相对比较(关闭档最大值 < 开启档最小值),**不含
|
||||||
|
任何魔数**: 推理段计在 completion 里,真开着时两档差一个数量级(实测 kimi-k3
|
||||||
|
关闭档恒 9 token)。取 `max` 档而非 `auto`: 后者对 minimax 一路等于"什么都不注入"
|
||||||
|
(`on_base={}`),那是模型默认档而不是"开",拿它当对照组会把 M3 这种默认不推理的
|
||||||
|
模型判成"分不开"。`max` 打不通时才退到 `auto`。
|
||||||
|
"""
|
||||||
|
off_usable = [o for o in off_observations if _probe_ok(o)]
|
||||||
|
for tier in (Effort.MAX, Effort.AUTO):
|
||||||
|
anchor = await _probe_effort(
|
||||||
|
model,
|
||||||
|
tier,
|
||||||
|
rounds=_TIER_LONG_ROUNDS,
|
||||||
|
prompt=_TIER_PROMPT,
|
||||||
|
prompt_kind=f"anchor({tier.value})",
|
||||||
|
)
|
||||||
|
on_usable = [o for o in anchor if _probe_ok(o)]
|
||||||
|
if not on_usable:
|
||||||
|
continue
|
||||||
|
off_max = max(o["completion_tokens"] for o in off_usable)
|
||||||
|
on_min = min(o["completion_tokens"] for o in on_usable)
|
||||||
|
return anchor, tier, off_max < on_min
|
||||||
|
return [], None, False
|
||||||
|
|
||||||
|
|
||||||
|
def _probe_record(model: str, phase: str, verdict: str, detail: str, observations: list[dict]):
|
||||||
|
_PROBE_ROWS.append(
|
||||||
|
{
|
||||||
|
"model": model,
|
||||||
|
"provider": _MODEL_PROVIDER[model],
|
||||||
|
"phase": phase,
|
||||||
|
"verdict": verdict,
|
||||||
|
"detail": detail,
|
||||||
|
"observations": observations,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module", autouse=True)
|
||||||
|
def _write_tier_report():
|
||||||
|
"""T10 报告独立成文件: 它的读者是"能力表该怎么改",与 L1-L9 的"库对不对"不同。"""
|
||||||
|
yield
|
||||||
|
if not _PROBE_ROWS:
|
||||||
|
return
|
||||||
|
_TIER_OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
path = _TIER_OUT_DIR / f"tier_probe_{ts}.md"
|
||||||
|
lines = [
|
||||||
|
"# 推理档位能力表实测(T10,经 new-api 中转)",
|
||||||
|
"",
|
||||||
|
f"- 时间: {ts}",
|
||||||
|
f"- 短提示词轮数: {_TIER_ROUNDS};长上下文复核轮数: {_TIER_LONG_ROUNDS};"
|
||||||
|
f"并发: {_TIER_CONCURRENCY}(共用生产网关,宁慢勿冲)",
|
||||||
|
f"- 短提示词: `{_TIER_PROMPT}`",
|
||||||
|
f"- 长上下文: 同题 + {len(_TIER_LONG_PROMPT)} 字符无关填充(判据②)",
|
||||||
|
"- 判据: 关闭方向要求**每轮**未观测到推理,且短提示词的「关掉了」必须经长上下文复核;"
|
||||||
|
"开启方向要求多数轮 OBSERVED",
|
||||||
|
"- 能力表在探测时被临时替换为全档表: 实测的对象正是它,不能拿它挡请求",
|
||||||
|
"",
|
||||||
|
"## 逐模型结论",
|
||||||
|
"",
|
||||||
|
"| 模型 | provider | 阶段 | 结论 | 观测 |",
|
||||||
|
"|---|---|---|---|---|",
|
||||||
|
]
|
||||||
|
total = 0
|
||||||
|
for row in _PROBE_ROWS:
|
||||||
|
detail = str(row["detail"]).replace("|", "\\|").replace("\n", " ")[:220]
|
||||||
|
lines.append(
|
||||||
|
f"| {row['model']} | {row['provider']} | {row['phase']} | {row['verdict']} | {detail} |"
|
||||||
|
)
|
||||||
|
total += len(row["observations"])
|
||||||
|
lines += ["", f"**总真实调用次数: {total}**", "", "## 逐轮原始观测", ""]
|
||||||
|
for row in _PROBE_ROWS:
|
||||||
|
if not row["observations"]:
|
||||||
|
continue
|
||||||
|
lines += [f"### {row['model']} — {row['phase']}", "", "```json"]
|
||||||
|
lines.append(json.dumps(row["observations"], ensure_ascii=False, indent=2, default=str))
|
||||||
|
lines += ["```", ""]
|
||||||
|
path.write_text("\n".join(lines), encoding="utf-8")
|
||||||
|
print(f"\n[T10 报告] {path}")
|
||||||
|
|
||||||
|
|
||||||
|
class TestTierProbe:
|
||||||
|
"""能力表实测。可只跑单个模型: `-k "test_t10 and glm-5.3"`。"""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("model", sorted(_MODEL_PROVIDER))
|
||||||
|
async def test_t10_none_direction_matches_declaration(self, model):
|
||||||
|
"""「这个模型到底关不关得掉」——能力表里唯一会**报错**的那条声明。
|
||||||
|
|
||||||
|
它是本节最要紧的一条: `Effort.NONE` 在不在清单里,决定 Phase 4 是放行还是
|
||||||
|
当场报错。声明错了,两个方向的代价都很实在——多写了 `none` 会让下游以为
|
||||||
|
关掉了(issue #20 的静默失效),漏写了会把一条本来可用的路堵死。
|
||||||
|
"""
|
||||||
|
short = await _probe_effort(
|
||||||
|
model, Effort.NONE, rounds=_TIER_ROUNDS, prompt=_TIER_PROMPT, prompt_kind="short"
|
||||||
|
)
|
||||||
|
# 上游拒绝这一档(400)是**结论**而非故障: 它等价于"关不掉";
|
||||||
|
# 其余失败(渠道下线/超时)才是源不可用,按既有纪律记为未覆盖
|
||||||
|
rejected = [o for o in short if o["error"] and o["error"].startswith("RequestRejected")]
|
||||||
|
usable = [o for o in short if _probe_ok(o)]
|
||||||
|
# **按可用轮判,而不是一有失败就整条跳过**: 共用网关上偶发 429/503 是常态,
|
||||||
|
# 一票否决会让整张表因为一次抖动而没有数据。样本低于 3 轮才是真的没结论
|
||||||
|
if not rejected and len(usable) < min(3, _TIER_ROUNDS):
|
||||||
|
broken = [o for o in short if o["error"]]
|
||||||
|
_probe_record(model, "none 方向", "SKIP(源不可用)", _rt_summary(short), short)
|
||||||
|
pytest.skip(f"{model} 源不可用,已记为未覆盖: {broken[0]['error'][:120]}")
|
||||||
|
|
||||||
|
strangers = _identity_mismatch(model, short)
|
||||||
|
if strangers:
|
||||||
|
_probe_record(
|
||||||
|
model,
|
||||||
|
"none 方向",
|
||||||
|
"SKIP(身份不符,数据不可信)",
|
||||||
|
f"该渠道把请求回报成 {strangers};{_rt_summary(short)}",
|
||||||
|
short,
|
||||||
|
)
|
||||||
|
pytest.skip(f"{model} 被该渠道路由到 {strangers},本次观测说的不是这个模型")
|
||||||
|
|
||||||
|
observations = list(short)
|
||||||
|
measured_can_disable = not rejected and all(_probe_quiet(o) for o in usable)
|
||||||
|
note = ""
|
||||||
|
if measured_can_disable:
|
||||||
|
# 判据②: 短提示词下"看起来关了"必须过长上下文这一关
|
||||||
|
long_ctx = await _probe_effort(
|
||||||
|
model,
|
||||||
|
Effort.NONE,
|
||||||
|
rounds=_TIER_LONG_ROUNDS,
|
||||||
|
prompt=_TIER_LONG_PROMPT,
|
||||||
|
prompt_kind="long",
|
||||||
|
)
|
||||||
|
observations += long_ctx
|
||||||
|
usable = [o for o in long_ctx if _probe_ok(o)]
|
||||||
|
if not usable:
|
||||||
|
note = ";长上下文复核未跑通,结论只在短提示词下成立"
|
||||||
|
else:
|
||||||
|
measured_can_disable = all(_probe_quiet(o) for o in usable)
|
||||||
|
note = ";长上下文复核" + ("同样未观测到推理" if measured_can_disable else "露馅")
|
||||||
|
|
||||||
|
if measured_can_disable and not any(
|
||||||
|
o["thinking_observation"] is ThinkingObservation.ABSENT for o in observations
|
||||||
|
):
|
||||||
|
# 判据④: 全程 `UNKNOWN` 时,"关掉了"是一句没有正面证据的话
|
||||||
|
anchor, anchor_tier, separable = await _anchor_off_against_on(model, observations)
|
||||||
|
observations += anchor
|
||||||
|
if anchor_tier is None:
|
||||||
|
note += ";锚点未跑通,关闭结论缺正面证据"
|
||||||
|
elif separable:
|
||||||
|
note += f";锚点可分(关闭档 completion 严格小于 {anchor_tier.value} 档)"
|
||||||
|
else:
|
||||||
|
measured_can_disable = None
|
||||||
|
note += f";**锚点不可分**(与 {anchor_tier.value} 档的 completion 分不开),判不出来"
|
||||||
|
|
||||||
|
detail = f"实测 can_disable={measured_can_disable}{note}。短: {_rt_summary(short)}" + (
|
||||||
|
f" ‖ 后续: {_rt_summary(observations[len(short) :])}"
|
||||||
|
if len(observations) > len(short)
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
if measured_can_disable is None:
|
||||||
|
_probe_record(model, "none 方向", "INCONCLUSIVE(无正面证据)", detail, observations)
|
||||||
|
pytest.skip(f"{model} 判不出来,已记为未覆盖: {detail[:160]}")
|
||||||
|
capability = get_capability(model)
|
||||||
|
if capability is None:
|
||||||
|
_probe_record(model, "none 方向", "DATA(未登记)", detail, observations)
|
||||||
|
pytest.skip(f"{model} 未登记(设计 §8 第三档),本条只采数据: {detail[:120]}")
|
||||||
|
agrees = measured_can_disable == capability.can_disable
|
||||||
|
_probe_record(
|
||||||
|
model,
|
||||||
|
"none 方向",
|
||||||
|
"PASS" if agrees else "FAIL(能力表已漂移)",
|
||||||
|
f"声明 can_disable={capability.can_disable};{detail}",
|
||||||
|
observations,
|
||||||
|
)
|
||||||
|
assert agrees, (
|
||||||
|
f"{model} 的能力表与实测不符: 声明 can_disable={capability.can_disable},"
|
||||||
|
f"实测 {measured_can_disable}。{detail}"
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("model", sorted(DEFAULT_CAPABILITIES))
|
||||||
|
async def test_t10_declared_tiers_actually_reason(self, model):
|
||||||
|
"""已登记的每个**开启档**都必须被上游接受,且真的推理。
|
||||||
|
|
||||||
|
证伪力只在"被拒"与"没推理"两件事上——**不断言档位之间的 rt 高低**:
|
||||||
|
设计 §4.3 已定,同一档 rt 实测在 8~56 之间跳,拿它比大小必然是噪声。
|
||||||
|
故本条能证伪的是"登记了一个上游根本不认的档",不是"档位排序对不对"。
|
||||||
|
"""
|
||||||
|
capability = get_capability(model)
|
||||||
|
tiers = [e for e in capability.supported_efforts if e is not Effort.NONE]
|
||||||
|
if not tiers:
|
||||||
|
pytest.skip(f"{model} 只登记了 none,没有开启档可验")
|
||||||
|
failures = []
|
||||||
|
for tier in tiers:
|
||||||
|
observations = await _probe_effort(
|
||||||
|
model, tier, rounds=_TIER_ROUNDS, prompt=_TIER_PROMPT, prompt_kind="short"
|
||||||
|
)
|
||||||
|
rejected = [
|
||||||
|
o for o in observations if o["error"] and o["error"].startswith("RequestRejected")
|
||||||
|
]
|
||||||
|
usable = [o for o in observations if _probe_ok(o)]
|
||||||
|
observed = [o for o in observations if _probe_observed(o)]
|
||||||
|
strangers = _identity_mismatch(model, observations)
|
||||||
|
if strangers:
|
||||||
|
# 与 none 方向同一条纪律: 回报的不是这个模型,这组数就不是它的
|
||||||
|
_probe_record(
|
||||||
|
model,
|
||||||
|
f"档位 {tier.value}",
|
||||||
|
"SKIP(身份不符,数据不可信)",
|
||||||
|
f"该渠道把请求回报成 {strangers};{_rt_summary(observations)}",
|
||||||
|
observations,
|
||||||
|
)
|
||||||
|
pytest.skip(f"{model} 被该渠道路由到 {strangers},本次观测说的不是这个模型")
|
||||||
|
if rejected:
|
||||||
|
verdict, problem = "FAIL(上游拒绝该档)", f"{tier.value}: 上游拒绝"
|
||||||
|
elif not usable:
|
||||||
|
verdict, problem = "SKIP(源不可用)", None
|
||||||
|
elif len(observed) * 2 > len(usable):
|
||||||
|
verdict, problem = "PASS", None
|
||||||
|
else:
|
||||||
|
verdict, problem = "FAIL(该档未推理)", f"{tier.value}: 多数轮未观测到推理"
|
||||||
|
if problem:
|
||||||
|
failures.append(problem)
|
||||||
|
_probe_record(
|
||||||
|
model, f"档位 {tier.value}", verdict, _rt_summary(observations), observations
|
||||||
|
)
|
||||||
|
assert not failures, f"{model} 登记的档位与实测不符: {failures}"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("model", ["gemini-3.1-pro", "gpt-5.5", "glm-5.3"])
|
||||||
|
async def test_t10_no_opinion_stays_no_opinion(self, model):
|
||||||
|
"""不表态时库**不推定**模型自己的默认档(Phase 1),顺带采下默认档的 rt 基线。
|
||||||
|
|
||||||
|
为什么给这三个模型单列一条: 它们的「厂商默认档」是 evidence 里写着、却最容易
|
||||||
|
写错的一格(Gemini 3.1 Pro 官方文档说 HIGH、OpenRouter 说 medium,两源打架),
|
||||||
|
而默认档写错会误导下游估成本。库本身不依赖这个值——**它不表态就什么都不注入**,
|
||||||
|
这正是本条断言的东西;默认档的 rt 观测只作报告里的旁证,**不作断言**: 单一模型上
|
||||||
|
rt 与档位没有可判定的函数关系(设计 §4.3),拿它反推默认档只能存疑,不能定论。
|
||||||
|
|
||||||
|
2026-09-05: gemini 一路当下在本渠道上游报错,claude 一路 7 天限额用尽,故把
|
||||||
|
另两格换成当下可测的 gpt-5.5 与 glm-5.3;gemini 留着,渠道恢复即有数。
|
||||||
|
"""
|
||||||
|
client = GatewayClient.from_settings(
|
||||||
|
_tier_settings(model), capabilities=_probe_capabilities(model)
|
||||||
|
)
|
||||||
|
observations = []
|
||||||
|
try:
|
||||||
|
for i in range(_TIER_ROUNDS):
|
||||||
|
try:
|
||||||
|
resp = await client.chat(
|
||||||
|
[{"role": "user", "content": _TIER_PROMPT}],
|
||||||
|
stream=True,
|
||||||
|
cache_salt=f"tier-default-{model}-{i}",
|
||||||
|
)
|
||||||
|
except (
|
||||||
|
RequestRejectedError,
|
||||||
|
GatewayUnavailableError,
|
||||||
|
SourceDeadError,
|
||||||
|
TransientError,
|
||||||
|
) as exc:
|
||||||
|
observations.append(
|
||||||
|
{
|
||||||
|
"round": i + 1,
|
||||||
|
"effort": "(不表态)",
|
||||||
|
"prompt_kind": "short",
|
||||||
|
"error": f"{type(exc).__name__}: {str(exc)[:160]}",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
observations.append(
|
||||||
|
{
|
||||||
|
"round": i + 1,
|
||||||
|
"effort": "(不表态)",
|
||||||
|
"prompt_kind": "short",
|
||||||
|
"error": None,
|
||||||
|
"prompt_tokens": resp.prompt_tokens,
|
||||||
|
"completion_tokens": resp.completion_tokens,
|
||||||
|
"reasoning_tokens": resp.reasoning_tokens,
|
||||||
|
"thinking_chars": len(resp.thinking),
|
||||||
|
"thinking_observation": resp.thinking_observation,
|
||||||
|
"applied_effort": resp.applied_effort,
|
||||||
|
"model_reported": resp.model_reported,
|
||||||
|
"content": resp.content[:40],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
await client.aclose()
|
||||||
|
usable = [o for o in observations if _probe_ok(o)]
|
||||||
|
if not usable:
|
||||||
|
_probe_record(
|
||||||
|
model,
|
||||||
|
"默认档基线(不表态)",
|
||||||
|
"SKIP(源不可用)",
|
||||||
|
_rt_summary(observations),
|
||||||
|
observations,
|
||||||
|
)
|
||||||
|
pytest.skip(f"{model} 源不可用,已记为未覆盖: {observations[0]['error'][:120]}")
|
||||||
|
leaked = [o for o in usable if o["applied_effort"] is not None]
|
||||||
|
_probe_record(
|
||||||
|
model,
|
||||||
|
"默认档基线(不表态)",
|
||||||
|
"PASS" if not leaked else "FAIL(库替模型推定了默认档)",
|
||||||
|
_rt_summary(observations),
|
||||||
|
observations,
|
||||||
|
)
|
||||||
|
assert not leaked, f"{model}: 不表态时 applied_effort 应为 None,实测 {leaked}"
|
||||||
|
|||||||
@@ -311,10 +311,15 @@ class TestResolveThinking:
|
|||||||
assert "low" in msg and "high" in msg and "max" in msg
|
assert "low" in msg and "high" in msg and "max" in msg
|
||||||
|
|
||||||
def test_phase5_says_toggle_only_for_switch_model(self):
|
def test_phase5_says_toggle_only_for_switch_model(self):
|
||||||
"""纯开关型模型没有档位,对它说"可选档位"是错的(设计 §3.2 第三个派生量)。"""
|
"""纯开关型模型没有档位,对它说"可选档位"是错的(设计 §3.2 第三个派生量)。
|
||||||
cap = get_capability("MiniMax-M3") # (none, auto): 能开能关,但没有强度档
|
|
||||||
|
样本 2026-09-05 由 MiniMax-M3 换成 glm-4.6v: T10 实测 M3 的六个强度值全部生效,
|
||||||
|
它不再是纯开关型;glm-4.6v 是实测证据最硬的 (none, auto) 模型,且 zhipu 的 wire
|
||||||
|
有 effort_key——这两点缺一不可,否则命中的是"该 provider 没有档位键"那条分支。
|
||||||
|
"""
|
||||||
|
cap = get_capability("glm-4.6v") # (none, auto): 能开能关,但没有强度档
|
||||||
with pytest.raises(ThinkingUnsupportedError) as exc:
|
with pytest.raises(ThinkingUnsupportedError) as exc:
|
||||||
resolve_thinking(get_provider("minimax"), cap, Effort.HIGH, model="MiniMax-M3")
|
resolve_thinking(get_provider("zhipu"), cap, Effort.HIGH, model="glm-4.6v")
|
||||||
msg = str(exc.value)
|
msg = str(exc.value)
|
||||||
assert "可选档位" not in msg
|
assert "可选档位" not in msg
|
||||||
assert "该模型只有开关" in msg
|
assert "该模型只有开关" in msg
|
||||||
@@ -328,10 +333,10 @@ class TestResolveThinking:
|
|||||||
)
|
)
|
||||||
with pytest.raises(ThinkingUnsupportedError) as switch:
|
with pytest.raises(ThinkingUnsupportedError) as switch:
|
||||||
resolve_thinking(
|
resolve_thinking(
|
||||||
get_provider("minimax"),
|
get_provider("zhipu"),
|
||||||
get_capability("MiniMax-M3"),
|
get_capability("glm-4.6v"),
|
||||||
Effort.MEDIUM,
|
Effort.MEDIUM,
|
||||||
model="MiniMax-M3",
|
model="glm-4.6v",
|
||||||
)
|
)
|
||||||
assert str(tiered.value) != str(switch.value)
|
assert str(tiered.value) != str(switch.value)
|
||||||
|
|
||||||
@@ -386,12 +391,12 @@ class TestResolveThinking:
|
|||||||
|
|
||||||
def test_nearest_never_turns_reasoning_off(self):
|
def test_nearest_never_turns_reasoning_off(self):
|
||||||
"""请求"想得浅一点"绝不能被映射成"别想了": 那是方向反转,不是省钱。"""
|
"""请求"想得浅一点"绝不能被映射成"别想了": 那是方向反转,不是省钱。"""
|
||||||
cap = get_capability("MiniMax-M3") # (none, auto)
|
cap = get_capability("glm-4.6v") # (none, auto)
|
||||||
got = resolve_thinking(
|
got = resolve_thinking(
|
||||||
get_provider("minimax"), cap, Effort.HIGH, model="MiniMax-M3", fallback="nearest"
|
get_provider("zhipu"), cap, Effort.HIGH, model="glm-4.6v", fallback="nearest"
|
||||||
)
|
)
|
||||||
assert got.applied_effort is Effort.AUTO
|
assert got.applied_effort is Effort.AUTO
|
||||||
assert got.payload == {}
|
assert got.payload == {"thinking": {"type": "enabled"}}
|
||||||
|
|
||||||
def test_nearest_still_errors_when_no_on_tier_exists(self):
|
def test_nearest_still_errors_when_no_on_tier_exists(self):
|
||||||
"""只能关不能开的模型,映射无解——报错而非挑一个反向的档。"""
|
"""只能关不能开的模型,映射无解——报错而非挑一个反向的档。"""
|
||||||
|
|||||||
Reference in New Issue
Block a user