The verifier caught that the disable-direction evidence only proved "no regression", not "actually took effect": on M3 the disabled runs and the no-opinion baseline are identically distributed, because that model does not reason by default anyway. So the disable runs alone cannot rule out the very failure mode issue #5 is about -- the parameter being silently dropped upstream. The bogus-value experiment that does rule it out was sitting in the findings document instead of the test suite; it is now case L3b, and the L3 assertion that could never fail is gone. Also from the review: the e2e helper caught bare Exception, which would have disguised a library bug as an unavailable source, exactly the silence the reporting discipline exists to prevent; the unregistered model warning fired on every request instead of once per source; and the transport caught ValueError broadly enough to mislabel unrelated errors, now narrowed to a dedicated ThinkingUnsupportedError. The design and plan still described the original judgement criteria, which the measurements had already overturned. Both now match what the tests actually do, and the design no longer claims the only new failure surface is the openai one -- dissect configures MiniMax-M2.7 with ENABLE_THINKING=false and will fail at assembly, which has to be coordinated before this merges.
This commit is contained in:
@@ -209,7 +209,7 @@ e2e 不进 CI 自动门的理由是外部不可用会误伤:实测中 kimi 渠
|
||||
|
||||
### 9.3 三条必须遵守的测试纪律
|
||||
|
||||
**(a)主判据选不会被中转污染的量。** `reasoning_tokens` 会被 new-api 的本地补算吃掉(实测 6:4 随机),单轮断言必然 flaky;而 `completion_tokens` 在补算路径下依然有值。因此**"是否关闭"的主判据用 `completion_tokens` 阈值,`reasoning_tokens` 作辅助**。这是本次实测最重要的工程教训之一。
|
||||
**(a)判别量只能是 `reasoning_tokens`。**(2026-08-02 e2e 实测修正:本节初稿写的是"主判据用 `completion_tokens`",被数据推翻。)两档的输出长度分布**重叠**——关闭档实测最高 46(模型偶尔把解题过程写进正文),开启档最低 13(medium 档想得少的轮次),按长度阈值判两个方向都会误判;而 `reasoning_tokens` 在同一批 30 轮里干净分开。`completion_tokens` 仅作 `reasoning_tokens` 被中转吃掉时的退路。另配一个不含魔数的确定性锚点:关闭档 `prompt_tokens` 严格小于开启档(实测 194 < 207)。
|
||||
|
||||
**(b)多轮 + 计数判定,不用单轮判定。** 关闭方向要求**每轮**都满足(关掉后 `completion_tokens` 极稳定,实测 4–10);开启方向只要求**多数轮**满足(推理量方差大)。
|
||||
|
||||
@@ -249,7 +249,9 @@ e2e 不进 CI 自动门的理由是外部不可用会误伤:实测中 kimi 渠
|
||||
|
||||
**能力表的正确性依赖实测,且经中转。** 三条 MiniMax 结论均在自建 new-api 中转下取得,直连官方端点未验证;表中每条 `evidence` 须写明这一点。若下游改为直连,L8 漂移哨兵是发现失真的第一道防线。
|
||||
|
||||
**三个下游零破坏**:VT / CHS / GovDoc 的 thinking 用法均为二元,本方案不改公开字段形态。新增的失败面仅有 `provider=openai` + 配了 `ENABLE_THINKING` 这一组合,经全仓与 dissect 检索当前无此用法。
|
||||
**新增两处失败面,其中一处会立刻打挂 dissect。**(2026-08-02 独立核验修正:本节初稿只列了 `openai` 那一处,遗漏了 M2.x。)其一是 `provider=openai` + 配了 `ENABLE_THINKING`,经全仓与 dissect 检索当前无此用法(dissect 的 K3 scope 用 `provider=openai` 但未配该项)。其二是**关不掉推理的模型 + `ENABLE_THINKING=false`**,而 `dissect/.env:80,85` 正是 `MiniMax-M2.7` + `false` —— 合并后该 scope 装配即抛 `ValueError`,实验链启动就挂。这是本设计的**预期行为**(给不了语义保证就必须说),但必须与 dissect 协调后再合并,不能突然打挂它。
|
||||
|
||||
**三个参考下游零破坏**:VT / CHS / GovDoc 的 thinking 用法均为二元,本方案不改公开字段形态。
|
||||
|
||||
## 13. 另立 issue(不在本次范围)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user