Files
PolyGateway/research-wiki/schemas
iomgaa 4c135075b3 test: verify the thinking switch against the live API (issue #5, #6)
A sixteen-row matrix over 127 real calls: disable and enable on
MiniMax-M3 in both streaming and non-streaming mode, extra_body winning
over the profile slot, qwen and deepseek still disabling correctly, a
drift sentinel that re-derives every registered capability from live
behaviour, and the assembly guard refusing the models that cannot
comply.

Two judgement criteria had to be corrected by the data they were meant
to judge. Output length cannot separate the two regimes at all -- the
disabled runs reach 46 tokens when the model narrates its working in
the visible answer, and the enabled runs drop to 13 when medium effort
barely thinks. reasoning_tokens separates them cleanly in both
directions, which is precisely what issue #6 was collected for. A
second anchor compares prompt_tokens between the two regimes: the
vendor injects a reasoning instruction when thinking is on, so the
input side grows, and comparing the two runs relatively avoids
hardcoding any vendor number.

Provider names are mapped explicitly rather than guessed from the model
string; guessing had silently skipped the qwen row behind a "source
unavailable" reason that was not true.
2026-08-02 06:55:38 -04:00
..