The verifier caught that the disable-direction evidence only proved "no regression", not "actually took effect": on M3 the disabled runs and the no-opinion baseline are identically distributed, because that model does not reason by default anyway. So the disable runs alone cannot rule out the very failure mode issue #5 is about -- the parameter being silently dropped upstream. The bogus-value experiment that does rule it out was sitting in the findings document instead of the test suite; it is now case L3b, and the L3 assertion that could never fail is gone. Also from the review: the e2e helper caught bare Exception, which would have disguised a library bug as an unavailable source, exactly the silence the reporting discipline exists to prevent; the unregistered model warning fired on every request instead of once per source; and the transport caught ValueError broadly enough to mislabel unrelated errors, now narrowed to a dedicated ThinkingUnsupportedError. The design and plan still described the original judgement criteria, which the measurements had already overturned. Both now match what the tests actually do, and the design no longer claims the only new failure surface is the openai one -- dissect configures MiniMax-M2.7 with ENABLE_THINKING=false and will fail at assembly, which has to be coordinated before this merges.
This commit is contained in:
@@ -209,7 +209,7 @@ e2e 不进 CI 自动门的理由是外部不可用会误伤:实测中 kimi 渠
|
||||
|
||||
### 9.3 三条必须遵守的测试纪律
|
||||
|
||||
**(a)主判据选不会被中转污染的量。** `reasoning_tokens` 会被 new-api 的本地补算吃掉(实测 6:4 随机),单轮断言必然 flaky;而 `completion_tokens` 在补算路径下依然有值。因此**"是否关闭"的主判据用 `completion_tokens` 阈值,`reasoning_tokens` 作辅助**。这是本次实测最重要的工程教训之一。
|
||||
**(a)判别量只能是 `reasoning_tokens`。**(2026-08-02 e2e 实测修正:本节初稿写的是"主判据用 `completion_tokens`",被数据推翻。)两档的输出长度分布**重叠**——关闭档实测最高 46(模型偶尔把解题过程写进正文),开启档最低 13(medium 档想得少的轮次),按长度阈值判两个方向都会误判;而 `reasoning_tokens` 在同一批 30 轮里干净分开。`completion_tokens` 仅作 `reasoning_tokens` 被中转吃掉时的退路。另配一个不含魔数的确定性锚点:关闭档 `prompt_tokens` 严格小于开启档(实测 194 < 207)。
|
||||
|
||||
**(b)多轮 + 计数判定,不用单轮判定。** 关闭方向要求**每轮**都满足(关掉后 `completion_tokens` 极稳定,实测 4–10);开启方向只要求**多数轮**满足(推理量方差大)。
|
||||
|
||||
@@ -249,7 +249,9 @@ e2e 不进 CI 自动门的理由是外部不可用会误伤:实测中 kimi 渠
|
||||
|
||||
**能力表的正确性依赖实测,且经中转。** 三条 MiniMax 结论均在自建 new-api 中转下取得,直连官方端点未验证;表中每条 `evidence` 须写明这一点。若下游改为直连,L8 漂移哨兵是发现失真的第一道防线。
|
||||
|
||||
**三个下游零破坏**:VT / CHS / GovDoc 的 thinking 用法均为二元,本方案不改公开字段形态。新增的失败面仅有 `provider=openai` + 配了 `ENABLE_THINKING` 这一组合,经全仓与 dissect 检索当前无此用法。
|
||||
**新增两处失败面,其中一处会立刻打挂 dissect。**(2026-08-02 独立核验修正:本节初稿只列了 `openai` 那一处,遗漏了 M2.x。)其一是 `provider=openai` + 配了 `ENABLE_THINKING`,经全仓与 dissect 检索当前无此用法(dissect 的 K3 scope 用 `provider=openai` 但未配该项)。其二是**关不掉推理的模型 + `ENABLE_THINKING=false`**,而 `dissect/.env:80,85` 正是 `MiniMax-M2.7` + `false` —— 合并后该 scope 装配即抛 `ValueError`,实验链启动就挂。这是本设计的**预期行为**(给不了语义保证就必须说),但必须与 dissect 协调后再合并,不能突然打挂它。
|
||||
|
||||
**三个参考下游零破坏**:VT / CHS / GovDoc 的 thinking 用法均为二元,本方案不改公开字段形态。
|
||||
|
||||
## 13. 另立 issue(不在本次范围)
|
||||
|
||||
|
||||
@@ -71,6 +71,10 @@ date: 2026-08-02
|
||||
|
||||
**结论:M2.x 的推理是模型固有属性,不是参数没找对。** 任何库层改动都无法让它关闭;唯一诚实的做法是如实报错。
|
||||
|
||||
### 2.4 M3 的稳定性
|
||||
|
||||
同一请求打 10 次,`(prompt_tokens, 是否上报 ctd)` 全部为 `(194, False)`,零跳变——`enable_thinking=False` 的修复可以建立在 M3 上。
|
||||
|
||||
### 2.5 输出长度不是有效判别量(2026-08-02 e2e 补测,各 15 轮)
|
||||
|
||||
初版判据用 `completion_tokens` 阈值区分推理开关,被自己的数据证伪:
|
||||
@@ -86,10 +90,6 @@ date: 2026-08-02
|
||||
|
||||
另有一个不含魔数的确定性锚点:同一模型上关闭档的 `prompt_tokens` 严格小于开启档(实测 194 < 207),因为供应商在开启时向模板注入了推理指令。这是相对比较,供应商改模板也不会失效。
|
||||
|
||||
### 2.4 M3 的稳定性
|
||||
|
||||
同一请求打 10 次,`(prompt_tokens, 是否上报 ctd)` 全部为 `(194, False)`,零跳变——`enable_thinking=False` 的修复可以建立在 M3 上。
|
||||
|
||||
## 3. qwen / deepseek:现有 profile 正确
|
||||
|
||||
| 模型 | `enable_thinking=false` | `thinking:{disabled}` | `reasoning_effort=none` | 现有 profile |
|
||||
|
||||
@@ -235,9 +235,11 @@ def _fingerprint_mark(s: SourceConfig) -> str:
|
||||
|
||||
**三条必须遵守的测试纪律**:
|
||||
|
||||
其一,**主判据用 `completion_tokens`,`reasoning_tokens` 只作辅助**。中转在上游不返回 usage 时会本地补算并吃掉 `completion_tokens_details`(findings §4c 实测同一请求 10 轮呈 6:4 双峰),拿它做单轮断言必然 flaky;而 `completion_tokens` 在补算路径下依然有值。
|
||||
其一,**判别量只能是 `reasoning_tokens`**。(执行时按 e2e 实测修正:本条初稿写的是「主判据用 `completion_tokens`」,被数据推翻——两档的输出长度分布**重叠**,关闭档实测最高 46、开启档最低 13,按长度阈值判两个方向都会误判。)`completion_tokens` 仅作 `reasoning_tokens` 被中转吃掉时的退路(findings §4c、§2.5)。
|
||||
|
||||
其二,**关闭方向要求每轮满足,开启方向只要求多数轮满足**。关掉后 `completion_tokens` 极稳定(实测 4–10),推理量则方差大。
|
||||
其二,**关闭方向要求每轮满足,开启方向只要求多数轮满足**。中转吃掉 ctd 时开启方向可能偶尔观测不到,关闭方向不受影响。
|
||||
|
||||
其四,**必须有不依赖输出侧噪声的锚点**:L2b 比较两档的 `prompt_tokens`(相对比较,无魔数),L3b 用非法值反证 `none` 是被识别而非被静默丢弃——后者正是 issue #5 的原始故障形态,不排除它,关闭方向的证据就只到「未回归」,够不到「已生效」。
|
||||
|
||||
其三,**源不可用必须跳过并在报告中显式记为「未覆盖」**,不得静默计入通过(实测中 kimi 渠道 429 后被中转下线并返回 404)。报告要能一眼看出哪些矩阵行没跑到。
|
||||
|
||||
|
||||
@@ -87,6 +87,17 @@ DEFAULT_PROFILES: Mapping[str, ProviderProfile] = MappingProxyType(
|
||||
)
|
||||
|
||||
|
||||
class ThinkingUnsupportedError(ValueError):
|
||||
"""推理开关无法满足: 形态未知或该模型不支持该方向(issue #5)。
|
||||
|
||||
是 `ValueError` 的子类而非 `errors.py` 四分类之一——它描述的是**配置**
|
||||
不可满足(装配期就该炸),不是一次调用的运行时失败。transport 在请求期
|
||||
捕获它并翻译为 `RequestRejectedError` 再进四分类。单列一个类型是为了让
|
||||
捕获点能精确到它,而不是宽catch 整个 `ValueError`(那会把序列化等无关
|
||||
错误误贴成"推理开关无法满足")。
|
||||
"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThinkingCapability:
|
||||
"""某个**具体模型**能否关闭推理(issue #5);登记必须附实测证据与日期。
|
||||
@@ -162,6 +173,7 @@ def resolve_thinking(
|
||||
enable_thinking: bool | None,
|
||||
*,
|
||||
model: str,
|
||||
warn_unregistered: bool = True,
|
||||
) -> Mapping[str, Any]:
|
||||
"""三态 + 两层能力 → 请求体注入片段;不可满足时 ValueError。
|
||||
|
||||
@@ -172,6 +184,9 @@ def resolve_thinking(
|
||||
|
||||
`model` 只用于错误与告警文案: 报错能定位到具体模型才有可操作性,而
|
||||
`capability` 为 None(未登记)时无从从别处取得模型名。
|
||||
|
||||
`warn_unregistered=False` 供请求热路径去重用: 装配期已经喊过一次,逐次
|
||||
调用再喊只会刷屏。判定结果不受此参数影响。
|
||||
"""
|
||||
# Phase 1: 调用方不表态 —— 与 False 严格区分,用模型默认档
|
||||
if enable_thinking is None:
|
||||
@@ -180,31 +195,36 @@ def resolve_thinking(
|
||||
direction = "thinking_on" if enable_thinking else "thinking_off"
|
||||
# Phase 2: 形态未知 —— 提供了开关却不知道怎么发,静默放行就是欺骗调用方
|
||||
if slot is None:
|
||||
raise ValueError(
|
||||
raise ThinkingUnsupportedError(
|
||||
f"provider {profile.name!r} 的 {direction} 形态未知(模型 {model!r}): "
|
||||
f"本库不知道该 provider 如何表达这一档。请用 register_provider 注册形态,"
|
||||
f"或改用 SourceConfig.extra_body 直接下发供应商参数"
|
||||
)
|
||||
# Phase 3: 能力未登记 —— 新模型上线不该被库挡住,但也不该假装成功
|
||||
if capability is None:
|
||||
logger.warning(
|
||||
"模型 {} 的推理能力未登记,按 provider {} 的形态尽力注入 {};"
|
||||
"若该模型实际不支持这一档,本次设置将静默失效。实测后请用 register_capability 登记",
|
||||
model,
|
||||
profile.name,
|
||||
dict(slot),
|
||||
)
|
||||
if warn_unregistered:
|
||||
_warn_unregistered(model, profile, slot)
|
||||
return slot
|
||||
# Phase 4: 明确不支持关闭 —— 调用方要的是"不推理"的语义保证,给不了必须说
|
||||
if enable_thinking is False and not capability.can_disable:
|
||||
raise ValueError(
|
||||
raise ThinkingUnsupportedError(
|
||||
f"模型 {model!r} 无法关闭推理,enable_thinking=False 无法满足: "
|
||||
f"{capability.evidence}。该模型的推理是固有属性,任何参数都关不掉——"
|
||||
f"若实验需要关闭思维链,请换用支持关闭的模型"
|
||||
f"需要关闭思维链请换用支持关闭的模型"
|
||||
)
|
||||
return slot
|
||||
|
||||
|
||||
def _warn_unregistered(model: str, profile: ProviderProfile, slot: Mapping[str, Any]) -> None:
|
||||
logger.warning(
|
||||
"模型 {} 的推理能力未登记,按 provider {} 的形态尽力注入 {};"
|
||||
"若该模型实际不支持这一档,本次设置将静默失效。实测后请用 register_capability 登记",
|
||||
model,
|
||||
profile.name,
|
||||
dict(slot),
|
||||
)
|
||||
|
||||
|
||||
def get_provider(
|
||||
name: str, *, registry: Mapping[str, ProviderProfile] | None = None
|
||||
) -> ProviderProfile:
|
||||
|
||||
@@ -24,6 +24,7 @@ from polygateway.errors import (
|
||||
from polygateway.providers import (
|
||||
ProviderProfile,
|
||||
ThinkingCapability,
|
||||
ThinkingUnsupportedError,
|
||||
get_capability,
|
||||
get_provider,
|
||||
resolve_thinking,
|
||||
@@ -295,6 +296,9 @@ class OpenAICompatTransport:
|
||||
) -> None:
|
||||
self._registry = registry
|
||||
self._capabilities = capabilities
|
||||
# 未登记模型只喊一次: 装配期已喊过,逐次调用再喊是日志洪水。
|
||||
# 实例级而非模块级 —— 模块级可变状态违反纯 asyncio 中立铁律
|
||||
self._warned_models: set[str] = set()
|
||||
self._client_factory = client_factory or _default_client_factory
|
||||
self._clients: dict[str, httpx.AsyncClient] = {}
|
||||
|
||||
@@ -320,8 +324,16 @@ class OpenAICompatTransport:
|
||||
# 形态(provider 级)与能力(model 级)在此相遇;不可满足时 ValueError,
|
||||
# 由 complete() 翻译为四分类之一(issue #5)
|
||||
capability = get_capability(source.model, table=self._capabilities)
|
||||
first_time = source.model not in self._warned_models
|
||||
self._warned_models.add(source.model)
|
||||
payload.update(
|
||||
resolve_thinking(profile, capability, source.enable_thinking, model=source.model)
|
||||
resolve_thinking(
|
||||
profile,
|
||||
capability,
|
||||
source.enable_thinking,
|
||||
model=source.model,
|
||||
warn_unregistered=first_time,
|
||||
)
|
||||
)
|
||||
# 顺序即优先级(issue #4 设计决策 A): 配置级 extra_body 在前,调用级
|
||||
# overlay(含结构化注入)在后覆盖之。两行不可调换
|
||||
@@ -344,8 +356,9 @@ class OpenAICompatTransport:
|
||||
payload = self._build_payload(
|
||||
messages=messages, source=source, profile=profile, stream=stream, overlay=overlay
|
||||
)
|
||||
except ValueError as exc:
|
||||
# 推理开关不可满足是**请求本身**的问题: 换源重试都救不了它
|
||||
except ThinkingUnsupportedError as exc:
|
||||
# 推理开关不可满足是**请求本身**的问题: 换源重试都救不了它。只捕这个
|
||||
# 专用类型而非宽 catch ValueError —— 后者会把序列化等无关错误误贴标签
|
||||
raise RequestRejectedError(
|
||||
f"{source.name} 推理开关无法满足: {exc}",
|
||||
source_name=source.name,
|
||||
|
||||
@@ -31,7 +31,12 @@ import pytest
|
||||
from dotenv import dotenv_values
|
||||
|
||||
from polygateway import GatewayClient, GatewaySettings
|
||||
from polygateway.errors import RequestRejectedError
|
||||
from polygateway.errors import (
|
||||
AllSourcesExhausted,
|
||||
RequestRejectedError,
|
||||
SourceDeadError,
|
||||
TransientError,
|
||||
)
|
||||
from polygateway.providers import DEFAULT_CAPABILITIES, get_capability
|
||||
|
||||
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
|
||||
@@ -236,8 +241,53 @@ class TestMiniMaxM3:
|
||||
|
||||
async def test_l3_no_opinion_is_the_model_default(self):
|
||||
obs = await _run_rounds(_ROUNDS, model="MiniMax-M3", enable_thinking=None)
|
||||
_record("L3", "enable_thinking=None(不干预,基线)", "PASS", "仅记录基线,不断言方向", obs)
|
||||
assert len(obs) == _ROUNDS
|
||||
# M3 的默认档实测就是不推理(findings §2.1),所以不干预时也应观测不到推理。
|
||||
# 注意这**不能**反过来证明关闭方向生效 —— L1 与本行同分布,区分二者的是
|
||||
# L2b 的 prompt_tokens 与 L3b 的乱码值反证
|
||||
quiet = [o for o in obs if _reasoning_off(o)]
|
||||
_record(
|
||||
"L3",
|
||||
"enable_thinking=None(不干预,基线)",
|
||||
"PASS" if len(quiet) == len(obs) else "FAIL",
|
||||
f"{len(quiet)}/{len(obs)} 轮未推理(M3 默认档本就不推理)",
|
||||
obs,
|
||||
)
|
||||
assert len(quiet) == len(obs), f"M3 默认档不应推理: {obs}"
|
||||
|
||||
async def test_l3b_none_is_recognised_not_silently_dropped(self):
|
||||
"""反证: 关闭方向的观测必须排除"参数被静默丢弃"这一伪解释。
|
||||
|
||||
L1(关闭)与 L3(不干预)在 M3 上**同分布**——因为 M3 默认档本就不推理。
|
||||
所以 L1 单独看不能区分"`none` 真的被消费"与"`none` 被中转吞了",而后者
|
||||
正是 issue #5 的原始故障形态(`enable_thinking` 就是这么被吞的)。
|
||||
|
||||
判别方法: 发一个**非法值**。若未知值会被静默丢弃,它的表现应与"不注入"
|
||||
一致(不推理);实测它反而开启了推理,说明网关认这个键、只是不认这个值。
|
||||
既然非法值与 `none` 的表现不同,`none` 就必然是被识别的枚举值。
|
||||
"""
|
||||
rounds = max(3, _ROUNDS // 3)
|
||||
bogus = await _run_rounds(
|
||||
rounds,
|
||||
model="MiniMax-M3",
|
||||
enable_thinking=None,
|
||||
extra_body={"reasoning_effort": "definitely-not-a-real-level"},
|
||||
)
|
||||
off = await _run_rounds(rounds, model="MiniMax-M3", enable_thinking=False)
|
||||
bogus_on = [o for o in bogus if _reasoning_on(o)]
|
||||
off_quiet = [o for o in off if _reasoning_off(o)]
|
||||
ok = len(bogus_on) * 2 > len(bogus) and len(off_quiet) == len(off)
|
||||
_record(
|
||||
"L3b",
|
||||
"非法值反证 none 被识别",
|
||||
"PASS" if ok else "FAIL",
|
||||
f"非法值 {len(bogus_on)}/{len(bogus)} 轮推理,none {len(off_quiet)}/{len(off)} 轮不推理"
|
||||
"(两者表现不同 ⇒ none 非被丢弃)",
|
||||
bogus + off,
|
||||
)
|
||||
assert len(bogus_on) * 2 > len(bogus), (
|
||||
f"非法值未开启推理,无法排除'未知值被静默丢弃'这一伪解释: {bogus}"
|
||||
)
|
||||
assert len(off_quiet) == len(off), f"none 未关闭推理: {off}"
|
||||
|
||||
async def test_l4_extra_body_overrides_the_profile(self):
|
||||
"""profile 注入 none,extra_body 要求 high —— 后者必须赢(优先级不可调换)。
|
||||
@@ -291,7 +341,9 @@ class TestOtherProviders:
|
||||
desc = f"{provider} enable_thinking=False"
|
||||
try:
|
||||
obs = await _run_rounds(_ROUNDS, provider=provider, model=model, enable_thinking=False)
|
||||
except Exception as exc: # 渠道未开通/下线: 记为未覆盖
|
||||
except (AllSourcesExhausted, SourceDeadError, TransientError) as exc:
|
||||
# 只吞网关/网络类失败。**不吞 ValueError / RequestRejected** ——
|
||||
# 那两类正是本次改动最可能的误伤方向,吞掉就成了纪律(c)要防的静默
|
||||
_skip_if_unreachable(exc, matrix, desc)
|
||||
offs = [o for o in obs if _reasoning_off(o)]
|
||||
_record(
|
||||
@@ -328,7 +380,7 @@ class TestCapabilityDrift:
|
||||
return
|
||||
try:
|
||||
obs = await _run_rounds(rounds, provider=provider, model=model, enable_thinking=False)
|
||||
except Exception as exc:
|
||||
except (AllSourcesExhausted, SourceDeadError, TransientError) as exc:
|
||||
_skip_if_unreachable(exc, "L8", desc)
|
||||
offs = [o for o in obs if _reasoning_off(o)]
|
||||
verdict = Counter(_reasoning_off(o) for o in obs)
|
||||
|
||||
@@ -7,6 +7,7 @@ import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from loguru import logger
|
||||
|
||||
from polygateway.errors import (
|
||||
RequestRejectedError,
|
||||
@@ -557,6 +558,45 @@ class TestRequestShaping:
|
||||
with pytest.raises(RequestRejectedError, match="MiniMax-M2.7"):
|
||||
await _complete(_transport_for(handler), source)
|
||||
|
||||
async def test_unregistered_model_warns_only_once_per_source(self):
|
||||
"""未登记模型的告警不能打在请求热路径上: 装配期已喊过,逐次再喊是刷屏。"""
|
||||
|
||||
def handler(request):
|
||||
return _sse_stream(_chunk(content="x"), _chunk(usage=_USAGE))
|
||||
|
||||
source = _source(name="mm", provider="minimax", model="MiniMax-M99", enable_thinking=False)
|
||||
transport = _transport_for(handler)
|
||||
messages: list[str] = []
|
||||
sink_id = logger.add(messages.append, level="WARNING")
|
||||
try:
|
||||
await _complete(transport, source)
|
||||
await _complete(transport, source)
|
||||
await _complete(transport, source)
|
||||
finally:
|
||||
logger.remove(sink_id)
|
||||
hits = [m for m in messages if "MiniMax-M99" in m]
|
||||
assert len(hits) == 1, f"三次调用应只告警一次,实得 {len(hits)} 次"
|
||||
|
||||
async def test_unrelated_value_error_is_not_mislabelled(self, monkeypatch):
|
||||
"""只捕 ThinkingUnsupportedError: 无关的 ValueError 不该被贴成推理开关的错。
|
||||
|
||||
今天 `_build_payload` 里只有 resolve_thinking 会抛 ValueError,所以这条
|
||||
是防御未来 —— 但正因如此才要钉住: 将来谁在那里加一处校验,宽 catch 会
|
||||
把它的错误信息盖掉,而这个用例会先红。
|
||||
"""
|
||||
|
||||
def handler(request): # pragma: no cover - 不该走到发请求
|
||||
raise AssertionError("请求不该发出")
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise ValueError("故意的无关错误")
|
||||
|
||||
monkeypatch.setattr("polygateway.transports.openai_compat.resolve_thinking", _boom)
|
||||
with pytest.raises(ValueError, match="故意的无关错误") as exc:
|
||||
await _complete(_transport_for(handler), _source(enable_thinking=False))
|
||||
assert "推理开关" not in str(exc.value)
|
||||
assert not isinstance(exc.value, RequestRejectedError)
|
||||
|
||||
async def test_unknown_shape_is_rejected(self):
|
||||
def handler(request): # pragma: no cover - 不该走到发请求
|
||||
raise AssertionError("请求不该发出")
|
||||
|
||||
Reference in New Issue
Block a user