Commit Graph

15 Commits

Author SHA1 Message Date
iomgaa 48805cb9fb fix: address the independent verification findings (issue #5, #6)
The verifier caught that the disable-direction evidence only proved "no
regression", not "actually took effect": on M3 the disabled runs and the
no-opinion baseline are identically distributed, because that model does
not reason by default anyway. So the disable runs alone cannot rule out
the very failure mode issue #5 is about -- the parameter being silently
dropped upstream. The bogus-value experiment that does rule it out was
sitting in the findings document instead of the test suite; it is now
case L3b, and the L3 assertion that could never fail is gone.

Also from the review: the e2e helper caught bare Exception, which would
have disguised a library bug as an unavailable source, exactly the
silence the reporting discipline exists to prevent; the unregistered
model warning fired on every request instead of once per source; and the
transport caught ValueError broadly enough to mislabel unrelated errors,
now narrowed to a dedicated ThinkingUnsupportedError.

The design and plan still described the original judgement criteria,
which the measurements had already overturned. Both now match what the
tests actually do, and the design no longer claims the only new failure
surface is the openai one -- dissect configures MiniMax-M2.7 with
ENABLE_THINKING=false and will fail at assembly, which has to be
coordinated before this merges.
2026-08-02 07:40:06 -04:00
iomgaa 4c135075b3 test: verify the thinking switch against the live API (issue #5, #6)
A sixteen-row matrix over 127 real calls: disable and enable on
MiniMax-M3 in both streaming and non-streaming mode, extra_body winning
over the profile slot, qwen and deepseek still disabling correctly, a
drift sentinel that re-derives every registered capability from live
behaviour, and the assembly guard refusing the models that cannot
comply.

Two judgement criteria had to be corrected by the data they were meant
to judge. Output length cannot separate the two regimes at all -- the
disabled runs reach 46 tokens when the model narrates its working in
the visible answer, and the enabled runs drop to 13 when medium effort
barely thinks. reasoning_tokens separates them cleanly in both
directions, which is precisely what issue #6 was collected for. A
second anchor compares prompt_tokens between the two regimes: the
vendor injects a reasoning instruction when thinking is on, so the
input side grows, and comparing the two runs relatively avoids
hardcoding any vendor number.

Provider names are mapped explicitly rather than guessed from the model
string; guessing had silently skipped the qwen row behind a "source
unavailable" reason that was not true.
2026-08-02 06:55:38 -04:00
iomgaa 781579bf36 docs: record thinking-switch findings and capability design (issue #5, #6)
Findings: live-API measurements across MiniMax M3/M2.7/M2.5, qwen and
deepseek, plus a survey of how nine unified gateways model per-model
parameter divergence. Key facts: reasoning_effort is MiniMax's real
switch, M2.x reasoning is mandatory and cannot be disabled, and the
relay's local token-count fallback silently drops reasoning_tokens.

Design: keep the parameter shape at provider level, push capability
down to model level, split "unknown" / "unsupported" / "no opinion"
into three distinct values, and fail at assembly time when a model
cannot honour enable_thinking=False.
2026-08-02 05:42:05 -04:00
iomgaa b2ee4fa383 docs: record v1.0.0 release and close out M4 roadmap status 2026-07-22 11:47:16 -04:00
iomgaa 7ac51f1f55 docs: register m4-acceptance finding in wiki 2026-07-22 10:36:33 -04:00
iomgaa f50df42084 docs: record M4 two-project migration acceptance 2026-07-22 10:36:33 -04:00
iomgaa d883d6e9c7 docs: record verifier fixes and P7 confirmation round 2026-07-22 02:42:33 -04:00
iomgaa 18d4f60665 docs: record P7 OCR soak acceptance at 99.73 percent 2026-07-22 01:37:17 -04:00
iomgaa a06761917e fix: address M2.5 verifier findings before merge
AIMD ceiling now respects per-source max_concurrency and the pacer is
assembled explicitly in the client; MIN_CALLS parses as strict int;
acceptance doc corrects source-5 attempt count to 549; design and
migration notes aligned with implemented 429/stall/suppression
semantics and AIMD constants documented.
2026-07-21 21:10:02 -04:00
iomgaa 9d28e03d49 docs: record M2.5 acceptance at 98.96 percent on unchanged P6 2026-07-21 17:34:13 -04:00
iomgaa 1a8931613a docs: correct P6 finding per verifier review 2026-07-21 07:42:54 -04:00
iomgaa 8c178a249c docs: record P6 soak baseline and scoreboard artifact fixes 2026-07-21 07:22:34 -04:00
iomgaa 326dd24dd1 fix: address M2 verifier findings in soak harness 2026-07-21 02:05:49 -04:00
iomgaa e101d82400 docs: record completed soak corpus pull and redis isolation rules 2026-07-20 22:37:00 -04:00
iomgaa 2e9e2b7c36 docs: add M2 soak-test scenario matrix and data manifest
Records the human decision to test against the real gateway with real
data instead of building a mock gateway, six execution scenarios with
fault-source mixing, hard invariants scored from our own telemetry,
and the corpus inventory pulled from the lab server.
2026-07-20 22:27:47 -04:00