test: verify the thinking switch against the live API (issue #5, #6)

A sixteen-row matrix over 127 real calls: disable and enable on
MiniMax-M3 in both streaming and non-streaming mode, extra_body winning
over the profile slot, qwen and deepseek still disabling correctly, a
drift sentinel that re-derives every registered capability from live
behaviour, and the assembly guard refusing the models that cannot
comply.

Two judgement criteria had to be corrected by the data they were meant
to judge. Output length cannot separate the two regimes at all -- the
disabled runs reach 46 tokens when the model narrates its working in
the visible answer, and the enabled runs drop to 13 when medium effort
barely thinks. reasoning_tokens separates them cleanly in both
directions, which is precisely what issue #6 was collected for. A
second anchor compares prompt_tokens between the two regimes: the
vendor injects a reasoning instruction when thinking is on, so the
input side grows, and comparing the two runs relatively avoids
hardcoding any vendor number.

Provider names are mapped explicitly rather than guessed from the model
string; guessing had silently skipped the qwen row behind a "source
unavailable" reason that was not true.
This commit is contained in:
2026-08-02 06:55:38 -04:00
parent 82f4ec4910
commit 4c135075b3
6 changed files with 437 additions and 5 deletions
+2 -2
View File
@@ -1,6 +1,6 @@
# Research Wiki 索引
> 自动生成,更新时间:2026-08-02 09:49 UTC
> 自动生成,更新时间:2026-08-02 10:55 UTC
## design (21)
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
@@ -59,7 +59,7 @@
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
## schema (1)
- [表结构: llm_calls(遥测 21 字段)](schemas/llm-calls.md) `schema:llm-calls`
- [表结构: llm_calls(遥测 22 字段)](schemas/llm-calls.md) `schema:llm-calls`
## metric (2)
- [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success`