Whether a call actually reasoned is now a first-class return value
(issue #16 + #17). The issues blamed MiniMax-M3 for no longer
reasoning; probing the live gateway showed the opposite. M3 reasons
fine — 124 characters of it over SSE — and what changed is that the
MiniMax route stopped reporting completion_tokens_details while qwen
and deepseek still do. The library had staked the whole question on
that one field, so it held 185 characters of reasoning prose and
reported no reasoning.
ThinkingObservation says observed, absent, or unknown, and unknown
means the call left no signal rather than that nothing happened. The
verdict is reconciled against the capability table on every call, so a
declaration going stale becomes a warning instead of a silent illusion
— the M3 evidence had sat unchecked for twenty-three days. It lands in
telemetry too, because this surfaced only when someone ran a suite that
is excluded by default and had not run in eighteen days.
The reasoning_tokens docstring was still teaching downstream to treat
None or 0 as no reasoning. The changelog and the schema page had both
been corrected; the docstring had not, and it is the copy that ships in
the wheel and shows up on hover. Someone writing a report from it would
have counted every real MiniMax reasoning call as not reasoning, which
is issue #16 all over again with the tests green.
The original wording stays, since reading pre-1.3.1 rows still needs
it. What follows it now says when it expired and what to read instead.
Two more places had drifted the same way: the changelog and the
architecture doc described the throttle and the cache fallback as they
were before this review, which is to say as the opposite of what the
code now does.
The claim that the two throttle sets would suppress each other does not
survive checking, as the mutation testing showed: their key spaces do
not overlap. Keeping them apart is still right, but for the honest
reason, which is that the two warnings have unrelated lifetimes.
Three of them were the same shape as the bug this branch exists to fix:
something goes wrong, the library swallows it, and the caller is left
with a number that means the opposite of what happened.
The throttle key had no source in it. Five sources on one model is the
normal case here, so the first one to break would warn once and silence
the other four for the life of the process, and the message never said
which gateway to look at.
An unknown verdict in a cached entry threw away the whole response. The
rehydrator tolerates unknown fields but not unknown values of a known
field, so two library versions sharing a Redis would each invalidate
the other's entries: halved hit rate, and the only log line says the
cache rebuild failed. A purely observational field should not be able
to void a response whose content is intact.
Normalising for telemetry now degrades instead of raising, both for a
bare string and for a value outside the domain. Either one used to
reach the same except and cost the whole row, which is exactly how
1.3.0 lost nineteen calls without anyone noticing.
The telemetry field count is taken from inspect.signature, not from
memory, because that is the one the release checklist keeps catching.
llm-calls.md said 22 and was two rounds stale; fixing the title alone
would have left the table contradicting it, so tenant_id and meta are
documented too.
The production template needed no new column — it derives them with
LIKE. What it gained is an assertion that it must keep deriving them
and must not inline a column name, which is the drift that could
actually happen.
The changelog leads with the three breaking items. A patch number
carries no warning by design, so the entry has to.
The four cases were red because the criterion could not see the
evidence. reasoning_tokens has been None on this route ever since
MiniMax stopped reporting completion_tokens_details, while the same
call carried 185 characters of reasoning prose the assertions never
looked at.
L5 asserted something that cannot happen. M3 returns neither prose nor
usage detail over the plain endpoint, so demanding that the
non-streaming path observe reasoning could never pass. It now asserts
what is true and worth holding: the prompt_tokens anchor still
separates the two directions, so the parameter did reach the model, and
the verdict is not ABSENT, so the library marked the gap honestly
instead of dressing it up as no reasoning.
_ON_MIN_COMPLETION is gone. The two directions overlap in output length
— 46 at most disabled, 13 at least enabled — so that fallback drew a
line through noise and only made the criterion look defended.
The README's production template does not hand-write its columns; it
derives them with LIKE from the seed table, and the prose right above it
says so. Telling an executor to add a column there would have made
Postgres reject a duplicate, turned TestProductionTemplate red, and
broken deployment for anyone following it.
The claim came from another task's report and went into the plan without
opening the README. A finding relayed across tasks is a lead to verify,
not a fact. What replaces it is a shape assertion — the template must
derive via LIKE and must not inline any column name — which pins the
real risk of someone copying columns in later.
Also adds the Gitea wiki sync the plan had missed: docs-convention makes
a version bump commit illegal on its own.
The README carries a hand-written production DDL template that no test
ever compares against COLUMNS, so it can fall a column behind and stay
green. Downstream deploying from it would get a table without the new
column and the library would silently trim it — the same silence this
issue exists to remove. Task 9 now fixes the template and adds the
same-source assertion.
Also records two things the implementation disproved: caplog cannot see
loguru output, and reconcile_thinking has to be defined after the
dataclass it annotates, since this module evaluates annotations eagerly.
And the column-count table was incomplete — six more spots go red.
This issue surfaced only because someone ran a slow suite that is
excluded by default and had not been run for eighteen days. As a column
it becomes a query: which model stopped being observable, and when.
The emitter unwraps the enum to a plain str at the single _record exit.
asyncpg makes no promise about encoding a str subclass, and a telemetry
write that fails is downgraded to one warning — it would not crash, it
would just quietly cost the Postgres path a column. Normalising at the
emitter follows what tenant_id, meta and sampling already do.
The column is appended last in COLUMNS and in both DDLs. An existing
table can only take ALTER at the end, so putting it anywhere else
forks the physical column order between a freshly built database and a
backfilled one.
asdict keeps the enum and json.dumps writes it as a string because
StrEnum is a str subclass, but nothing turns it back on the way in, so
a cache hit returned a plain str where the annotation promised an enum.
Verified end to end rather than assumed from the subclass relation.
A value outside the domain now raises inside the existing guard and the
call falls back to source, which is the right direction for a poisoned
or stale cache entry. Entries written before this column existed still
replay: the guard checks for the key first, and a test pins that, since
turning it into an unconditional conversion would quietly turn every
pre-upgrade entry into a permanent miss.
The M3 evidence sat at 08-02 for twenty-three days while nobody could
tell whether it still held. A declaration that goes stale in silence is
the failure this issue is really about, so the library now compares
what it declared against what it just observed and says so when the two
part ways.
Judgement is separated from logging: reconcile_thinking returns the
warning text, so tests assert on the text instead of parsing logs.
Two cases that look alike are kept apart — a model whose capability is
registered gets a drift warning quoting its evidence, an unregistered
one is never told the table said anything, because it never did.
False x UNKNOWN stays silent on purpose. UNKNOWN cannot falsify
anything, and warning on it would fire on every disabled call M3 makes
over the plain endpoint. A warning that always fires is not a warning.
can_disable stays true — reasoning_effort=none still lands prompt 194,
completion 3, no prose. What the retest added are two limits worth
recording: the verdict is unobservable on the non-streaming path, where
reasoning is billed but neither prose nor usage detail comes back, and
enable_thinking / thinking:{enabled} remain inert on this model.
No behaviour changed, so there is no failing test to show first. The
evidence for a declaration that still holds is the retest itself, not
a unit test the library could write about its own claim.
Both assembly paths fill it, streaming and non-streaming alike. Filling
only one is exactly the divergence this issue exposed: M3 returns
reasoning prose over SSE and nothing at all over the plain endpoint, so
a verdict computed on one path says nothing about the other.
The field defaults to UNKNOWN on both TransportResult and LLMResponse.
A transport that does not judge should not get to declare absence on
the provider's behalf, and a default that stays silent is the only one
that cannot lie.
Both recorders are (self, **fields), not explicit parameter lists, so a
new column needs no signature change on them — COLUMNS plus an emitter
that passes it is enough. The port Protocol stays explicit because that
is where the emitter's contract and the freeze test anchor.
providers.py had been holding two jobs: the registry of what each
provider looks like, and the decisions made from those declarations.
Adding response-side judgement would have made it the module for
everything about reasoning, so the decisions move to thinking.py and
the registry keeps only profiles and their lookup.
Moving a module breaks any deep-path import of what moved, so the six
public symbols are promoted to the package root at the same time. The
top level is this library's stated API surface; giving downstream a
stable name to import is what makes the next reorganisation harmless.
observe_thinking stays unexported — downstream reads the verdict off
LLMResponse, and exporting it would be a permanent promise for nothing.
reasoning_tokens=None has been carrying two meanings at once, no
reasoning and no report, and the library resolved the ambiguity by
quietly claiming the first. ThinkingObservation splits them: UNKNOWN
says the call left no signal, ABSENT says the provider reported zero.
The verdict ranks evidence by hardness. Reasoning prose is the fact
itself; reasoning_tokens is a report about the fact, so a missing
report cannot overrule prose that is right there. The prose check
strips first, since a gateway that returns whitespace is not evidence.
The enum lives in types.py, not in the new thinking.py, because
LLMResponse is typed on it and the innermost layer must not import a
decision module.
Everything up to and including the full slow suite runs on the branch
without asking. Merging to main, pushing a tag, and uploading to the
registry cannot be taken back, and a registry version number cannot be
reused, so those wait for an explicit yes.
Also settles three things an executor would have tripped on: the
_AttemptUsage field is typed as the enum with .value applied only at
the recorder boundary, __all__ is not in strict alphabetical order, and
the port signature test freezes two params rather than the full list.
Each task carries its own failing-then-passing evidence and a command
whose output decides whether it is done. Two traps are called out where
an executor would otherwise walk into them: the enum has to live in
types.py or import-linter rejects the layering, and the 24 in
test_telemetry.py line 1787 counts OCR placeholder characters, not
telemetry columns.
The enum belonged in types.py all along: making LLMResponse field-typed
on a symbol defined in thinking.py would have had the innermost layer
import a decision module, and import-linter would have caught it only
after the code was written.
The 4.1 table claimed reconciliation could still catch a failed disable
while section 5 said UNKNOWN never speaks. UNKNOWN has no falsifying
power; the guarantee only covers observable paths, and the doc now says
so instead of pretending otherwise.
Section 12 was written against a misreading: _record already is the
single helper the ironclad rule asks for, so there was no debt to
decline. Landing sites had missed ports.py, whose record_llm_call
freezes 24 explicit params with no defaults, and cache.py, where
_rehydrate revives the enum as a bare string.
The design argued for 1.4.0 because a broken deep-path import hidden
behind a patch bump is a debt handed to downstream. The call is 1.3.1.
Since the version number no longer carries the warning, the CHANGELOG
has to: breaking items and their fixes go first in the entry, following
the 1.3.0 read-this-first form.
The four red e2e cases were blamed on MiniMax-M3 no longer reasoning.
Raw gateway probes show the opposite: M3 reasons fine (124 chars of
reasoning_content, prompt 194 to 216, completion 3 to 60). What changed
is that the MiniMax route stopped returning completion_tokens_details,
while qwen and deepseek still do on the same gateway and key. The
library already holds 185 chars of proof in LLMResponse.thinking and
never feeds it into any verdict.
The design turns that verdict into a first-class return value judged
from multiple signals, says UNKNOWN when a single response cannot tell,
and reconciles it against the capability table so a stale declaration
becomes a warning instead of a silent illusion.
ARCHITECTURE 7.8 gains the pool resource semantics, the two-sentence
failure verdict and the two config keys; the ownership rule lands in a
new 4.5 because it is a cross-subsystem discipline, not a telemetry
convention. CHANGELOG leads with the three items downstream must read
first: the 3.12 floor, the connection count going from 10 per client to
on demand, and aclose no longer closing injected components.
client.telemetry_status exists so downstream can reconcile telemetry
programmatically, but annotating its return type meant reaching into
polygateway.types while the convention here is that the top-level
exports are the public API surface. The port itself stays unexported:
nobody outside the library implements it.
The status snapshot reports elapsed time, so asserting retry_after_s
against the real monotonic clock was really asserting that a few lines
of code take zero time; it failed at 59.99993 vs 60.0. The recorder
already accepts an injected clock for exactly this reason.
The min_size=10 default survived to 1.2.4 because every PG test injected a
pool and thus skipped the pool-building path entirely. Unit tests now assert
the create_pool arguments, but "we passed min_size=0" and "the server really
opened that many backends" are two different claims, and only a real instance
can settle the second one. Count via a run-unique application_name carried on
the DSN: the instance is shared with other projects, so counting by database
or role would fold their connections into ours and make the case flaky by
construction.
Degradation is exercised through an unreachable DSN rather than by exhausting
the shared instance's connections. A refused connection lands in the same
class as exhaustion, and the fake clock lets the 60s cooldown be observed
without sleeping. retry_after_s is the signal that separates a real retry
(which renews the window) from the cheap short circuit (which does not).
Evidence: with create_pool reverted to its pre-fix form both cases go red
(observed 10 backends after a single write, and refusal surfacing at pool
creation instead of at prepare time).
The pool exhaustion in issue #15 was fatal only because min_size=10 forced
a transient error to surface at pool creation, and that step was hardcoded
to permanent death. Step is the wrong axis: it conflates "the DSN cannot
be parsed" with "someone else holds all the connections right now".
Failures are now classified by two rules. Fatal means the cause lies
entirely inside this process and cannot change, which only the
construction-time DSN satisfies. Everything else splits on whether the
failure has anything to do with this row's data: row-level failures drop
one row and keep trying, environment-level failures cool down for 60s and
then get exactly one retry, so a restarted database or a DBA creating the
table heals on its own.
42703 (missing column) is the single named exception and stays row-level
even though every row fails alike: issue #13 promised that the manual mode
trims the INSERT and exposes drift per row, and that promise outranks the
rule. Any future exception owes the same argument.
The _failed boolean is gone; the tracker is the only degradation state,
because two copies of the same fact drift apart. Closing stays outside
that state: it is the caller's own decision, not an anomaly to recover
from, so the snapshot reports it through dropped_rows and the drop reason
instead of raising the degraded flag on every clean shutdown.
Closing was the last unbounded wait on the shutdown path: asyncpg's
Pool.close() awaits wait_until_released() on every holder, so a single
in-flight connection parks the caller forever (60s only buys a warning).
It now runs under asyncio.wait_for and terminates the pool on timeout;
external cancellation still propagates untouched.
Closing is also final now. Clearing _pool used to leave the recorder free
to build a fresh pool on the next write - worse in the injected case,
where the owner believes it still holds every connection while the
recorder quietly opened its own. Recovery is a runtime concern (cooldown
retry), not a side effect of shutdown, so writes after aclose short out
and count the dropped row with a reason of their own.
Also covers the release/terminate fallback left untested by the pool
work: the fake pool needed for the close cases makes it nearly free.
The pool was the only external resource in the library that pre-allocated:
asyncpg's default min_size=10 turned pool creation into an all-or-nothing
action, so on a shared instance running low on connection budget the first
thing to fall over was the one component that must not fail silently
(4 clients x 10 = 40 idle connections just to write telemetry).
min_size=0 means "do not pre-connect" - asyncpg only builds holders - so
pool creation becomes free and never touches the database; connection
failures then land on acquire, the path that already drops one row and lets
the pool recover. max_size and the write budget become the library's
explicit statement about its own footprint, configurable through two new
keys whose defaults live in config alone (the recorder parameters are
required keyword-only, same discipline as auto_migrate).
The whole write - prepare, acquire, execute - now runs inside one
asyncio.timeout: acquire used to have no timeout at all, so a full pool
would hang forever on the caller's path. Release is explicit rather than
`async with`, because asyncpg shields release and reuses the acquire
timeout, which would let a single telemetry write consume twice the budget.
Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.
Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.
For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.
Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
A client used to close whatever transport, recorder or cache it happened
to hold, injected or not, so the first client to shut down killed the
backend its siblings were still using. That is why the explicit-sharing
path the architecture prescribes was unusable in practice and downstream
projects fell back to one private instance per client. The mirror image
of the same gap: the redis clients the factories build for the limiter
and the breaker were never closed at all, because nobody kept a
reference to them once they were handed to the retry middleware.
Ownership is now stated once, the way RedisLimiter already stated it:
whoever builds a resource closes it, injected ones are left alone. The
constructor is the full-injection path, so it owns nothing by default
and only the factories mark what they built. RedisCache gains the same
rule for its own client, and the three copies of the "probe for aclose,
fall back to close" dance collapse into a single helper so the next
correction cannot land in only one of them.
The design traces the incident to four stacked defects rather than one bad
default: the pool is the only resource in the library that pre-allocates,
the kill switch keys off which step failed instead of what failed, the
degraded state can neither recover nor be observed, and the ownership rules
make the sanctioned sharing path unusable.
The plan sequences the tracker ahead of the pool and failure work so every
commit stays green, and records two facts the implementer needs up front:
the pool-construction path has zero test coverage today, and the commit
gate runs the full suite plus a complexity ceiling.
The telemetry write budget needs asyncio.timeout, whose uncancel accounting
was only fixed after 3.11.1 — pinning the floor at 3.12 removes that hazard
instead of working around it.
Raising ruff's target-version turns on UP047, so gather_bounded,
_anext_within and stream_with_liveness_timeouts move to def f[T](...) and
the two module-level TypeVars go away. That syntax is a SyntaxError on
3.11, so it can only land together with the version bump.
The pre-commit hook runs the whole suite, and tests/e2e/ talks to a real
LLM gateway, so whether a commit is allowed depended on how fast that
gateway happened to be. During the issue 14 work it blocked two commits
on two different cases; both passed when rerun alone, and the suite went
from 165s to 336s that hour.
The wasted minutes are not the real cost. Retrying on red teaches you to
read "test failed" as "gateway was slow", and a genuinely flaky bug then
gets retried away too. An alarm that cries wolf stops being an alarm.
test_thinking_live.py already carried the slow marker; the other three
files now match it, and the release checklist gains an explicit
`pytest -m slow` step so they still run where a human is watching --
without that step this change would just delete the coverage.
Also raises test_flat_legacy_keys_assemble's LLM_TIMEOUT from 120 to
300, matching .env. At 120 the case allowed half of what production
allows, on a gateway that needs the full 300 -- it measured 116s in a
solo run. The assertion is that the flat key name parses into
SourceConfig.timeout_s; the value itself was never under test.
Close issue #14: an open circuit could only kill the call on the spot.
Three things. retry_after_s now means "how long until a retry is
certainly worth attempting", so a half-open gate and an admitted probe
both report 0.0 -- which also closes a bug the issue never spotted: that
value was fed into the source cooldown memo, whose set_until only moves
forward, so a source stayed skipped in-process for a whole probe lease
(up to 2x timeout) after its probe succeeded and the gate closed. Multi
source deployments were hit too; other sources just absorbed the load.
{SCOPE}__CIRCUIT_OPEN=fail_fast|wait fills the missing cell of the
admission matrix, shaped like QUOTA_FULL. Default fail_fast keeps every
existing control flow byte-identical; single-source scopes want wait.
And the admission logic that all three governance loops had copied
verbatim now lives once, in SourceAdmission -- otherwise this fix would
have left embedding and OCR behind as divergent corners.
README first, since packaging freezes whatever it says at build time:
version pin bumped, and the capability table now mentions that an open
circuit can wait as well as fail fast. Verified the numeric claims by
measurement rather than memory -- record_llm_call still takes 24 fields,
schema.COLUMNS still has 24, meta still caps at 16 keys.
Records what the two Codex review rounds found, which findings held up
under verification, and how each was resolved -- including the one that
changed docs rather than code. Also lists the evidence behind the
completion claim: suite counts, coverage, the 19-minute real-wait Redis
run, and the import contract.
Branch review caught the docs claiming something the code does not do.
CHANGELOG, README and the design's behaviour matrix all said a
force-opened source under circuit_open=wait waits out the full stall
window. It does not: the probe let through after each cooldown is a
real attempt, so it burns a max_attempts slot like any other, and a
401 source usually runs out of retry budget first -- reason is
retry_exhausted, not stalled. Which budget wins depends on
max_attempts against the cooldowns and the stall window.
The behaviour is right; only the prose was wrong. Charging the probe
to the retry budget is exactly the split issue #8 settled: the
question is who spends max_attempts, and a probe does send a real
request. A test now pins it so the claim cannot drift again.
Also drops the planned "woke up" log line. Each wait round already
logs on entry with its duration, and a still-blocked wake-up logs the
next round immediately, so a second line would only double the volume.
README gains the key with the reason a single-source scope wants wait,
and the price of choosing it. .env.example carries the same warning
since README points at it as the full key list. ARCHITECTURE 7.4 records
why the missing cell is unrelated to source count -- and why keying on
len(sources) would be the worse debt -- plus the six-exit retry_after_s
contract and the admission convergence; 9 registers the key.
CHANGELOG stays unreleased per the release checklist: the version bump
belongs to the release run, not here. Its "read this first" section
covers the half-open retry_after_s change, which is visible even on the
default fail_fast setting.
iomgaa