ARCHITECTURE 7.8 gains the pool resource semantics, the two-sentence
failure verdict and the two config keys; the ownership rule lands in a
new 4.5 because it is a cross-subsystem discipline, not a telemetry
convention. CHANGELOG leads with the three items downstream must read
first: the 3.12 floor, the connection count going from 10 per client to
on demand, and aclose no longer closing injected components.
client.telemetry_status exists so downstream can reconcile telemetry
programmatically, but annotating its return type meant reaching into
polygateway.types while the convention here is that the top-level
exports are the public API surface. The port itself stays unexported:
nobody outside the library implements it.
The status snapshot reports elapsed time, so asserting retry_after_s
against the real monotonic clock was really asserting that a few lines
of code take zero time; it failed at 59.99993 vs 60.0. The recorder
already accepts an injected clock for exactly this reason.
The min_size=10 default survived to 1.2.4 because every PG test injected a
pool and thus skipped the pool-building path entirely. Unit tests now assert
the create_pool arguments, but "we passed min_size=0" and "the server really
opened that many backends" are two different claims, and only a real instance
can settle the second one. Count via a run-unique application_name carried on
the DSN: the instance is shared with other projects, so counting by database
or role would fold their connections into ours and make the case flaky by
construction.
Degradation is exercised through an unreachable DSN rather than by exhausting
the shared instance's connections. A refused connection lands in the same
class as exhaustion, and the fake clock lets the 60s cooldown be observed
without sleeping. retry_after_s is the signal that separates a real retry
(which renews the window) from the cheap short circuit (which does not).
Evidence: with create_pool reverted to its pre-fix form both cases go red
(observed 10 backends after a single write, and refusal surfacing at pool
creation instead of at prepare time).
The pool exhaustion in issue #15 was fatal only because min_size=10 forced
a transient error to surface at pool creation, and that step was hardcoded
to permanent death. Step is the wrong axis: it conflates "the DSN cannot
be parsed" with "someone else holds all the connections right now".
Failures are now classified by two rules. Fatal means the cause lies
entirely inside this process and cannot change, which only the
construction-time DSN satisfies. Everything else splits on whether the
failure has anything to do with this row's data: row-level failures drop
one row and keep trying, environment-level failures cool down for 60s and
then get exactly one retry, so a restarted database or a DBA creating the
table heals on its own.
42703 (missing column) is the single named exception and stays row-level
even though every row fails alike: issue #13 promised that the manual mode
trims the INSERT and exposes drift per row, and that promise outranks the
rule. Any future exception owes the same argument.
The _failed boolean is gone; the tracker is the only degradation state,
because two copies of the same fact drift apart. Closing stays outside
that state: it is the caller's own decision, not an anomaly to recover
from, so the snapshot reports it through dropped_rows and the drop reason
instead of raising the degraded flag on every clean shutdown.
Closing was the last unbounded wait on the shutdown path: asyncpg's
Pool.close() awaits wait_until_released() on every holder, so a single
in-flight connection parks the caller forever (60s only buys a warning).
It now runs under asyncio.wait_for and terminates the pool on timeout;
external cancellation still propagates untouched.
Closing is also final now. Clearing _pool used to leave the recorder free
to build a fresh pool on the next write - worse in the injected case,
where the owner believes it still holds every connection while the
recorder quietly opened its own. Recovery is a runtime concern (cooldown
retry), not a side effect of shutdown, so writes after aclose short out
and count the dropped row with a reason of their own.
Also covers the release/terminate fallback left untested by the pool
work: the fake pool needed for the close cases makes it nearly free.
The pool was the only external resource in the library that pre-allocated:
asyncpg's default min_size=10 turned pool creation into an all-or-nothing
action, so on a shared instance running low on connection budget the first
thing to fall over was the one component that must not fail silently
(4 clients x 10 = 40 idle connections just to write telemetry).
min_size=0 means "do not pre-connect" - asyncpg only builds holders - so
pool creation becomes free and never touches the database; connection
failures then land on acquire, the path that already drops one row and lets
the pool recover. max_size and the write budget become the library's
explicit statement about its own footprint, configurable through two new
keys whose defaults live in config alone (the recorder parameters are
required keyword-only, same discipline as auto_migrate).
The whole write - prepare, acquire, execute - now runs inside one
asyncio.timeout: acquire used to have no timeout at all, so a full pool
would hang forever on the caller's path. Release is explicit rather than
`async with`, because asyncpg shields release and reuses the acquire
timeout, which would let a single telemetry write consume twice the budget.
Telemetry degradation used to be a single warning and a private boolean.
In a long-running process that is indistinguishable from telemetry working:
issue #15 was only found by hand-reconciling milestone log lines against
llm_calls rows, after 19 calls had silently gone unrecorded. The SQLite
side was worse — once init failed, every write returned without even a
log line.
Degradation now has one shared owner. TelemetryStatusTracker holds the
state machine (enter/recover/drop/should-retry), announces entry and
recovery once each, and repeats the drop count under a row-and-time
double threshold so a degraded backend neither floods the log nor goes
quiet. Both recorders hold one; both count the rows they drop.
For programmatic consumers, TelemetryStatus is a frozen snapshot exposed
as telemetry_status on all three clients, resolved through a single
isinstance check. It is a separate optional port rather than a member of
TelemetryRecorder: that protocol is @runtime_checkable, so adding an
attribute would make every implementation that only defines
record_llm_call stop satisfying it — downstream isinstance assertions
would break on upgrade. The existing assertion in test_ports.py is what
keeps that decision honest.
Failure criteria are deliberately untouched here: Postgres still treats a
pool failure as permanent, only now visibly. `_failed` and the tracker
therefore both carry the verdict for the span of this one change; the
cooldown rework collapses them into the tracker alone.
A client used to close whatever transport, recorder or cache it happened
to hold, injected or not, so the first client to shut down killed the
backend its siblings were still using. That is why the explicit-sharing
path the architecture prescribes was unusable in practice and downstream
projects fell back to one private instance per client. The mirror image
of the same gap: the redis clients the factories build for the limiter
and the breaker were never closed at all, because nobody kept a
reference to them once they were handed to the retry middleware.
Ownership is now stated once, the way RedisLimiter already stated it:
whoever builds a resource closes it, injected ones are left alone. The
constructor is the full-injection path, so it owns nothing by default
and only the factories mark what they built. RedisCache gains the same
rule for its own client, and the three copies of the "probe for aclose,
fall back to close" dance collapse into a single helper so the next
correction cannot land in only one of them.
The design traces the incident to four stacked defects rather than one bad
default: the pool is the only resource in the library that pre-allocates,
the kill switch keys off which step failed instead of what failed, the
degraded state can neither recover nor be observed, and the ownership rules
make the sanctioned sharing path unusable.
The plan sequences the tracker ahead of the pool and failure work so every
commit stays green, and records two facts the implementer needs up front:
the pool-construction path has zero test coverage today, and the commit
gate runs the full suite plus a complexity ceiling.
The telemetry write budget needs asyncio.timeout, whose uncancel accounting
was only fixed after 3.11.1 — pinning the floor at 3.12 removes that hazard
instead of working around it.
Raising ruff's target-version turns on UP047, so gather_bounded,
_anext_within and stream_with_liveness_timeouts move to def f[T](...) and
the two module-level TypeVars go away. That syntax is a SyntaxError on
3.11, so it can only land together with the version bump.
The pre-commit hook runs the whole suite, and tests/e2e/ talks to a real
LLM gateway, so whether a commit is allowed depended on how fast that
gateway happened to be. During the issue 14 work it blocked two commits
on two different cases; both passed when rerun alone, and the suite went
from 165s to 336s that hour.
The wasted minutes are not the real cost. Retrying on red teaches you to
read "test failed" as "gateway was slow", and a genuinely flaky bug then
gets retried away too. An alarm that cries wolf stops being an alarm.
test_thinking_live.py already carried the slow marker; the other three
files now match it, and the release checklist gains an explicit
`pytest -m slow` step so they still run where a human is watching --
without that step this change would just delete the coverage.
Also raises test_flat_legacy_keys_assemble's LLM_TIMEOUT from 120 to
300, matching .env. At 120 the case allowed half of what production
allows, on a gateway that needs the full 300 -- it measured 116s in a
solo run. The assertion is that the flat key name parses into
SourceConfig.timeout_s; the value itself was never under test.
Close issue #14: an open circuit could only kill the call on the spot.
Three things. retry_after_s now means "how long until a retry is
certainly worth attempting", so a half-open gate and an admitted probe
both report 0.0 -- which also closes a bug the issue never spotted: that
value was fed into the source cooldown memo, whose set_until only moves
forward, so a source stayed skipped in-process for a whole probe lease
(up to 2x timeout) after its probe succeeded and the gate closed. Multi
source deployments were hit too; other sources just absorbed the load.
{SCOPE}__CIRCUIT_OPEN=fail_fast|wait fills the missing cell of the
admission matrix, shaped like QUOTA_FULL. Default fail_fast keeps every
existing control flow byte-identical; single-source scopes want wait.
And the admission logic that all three governance loops had copied
verbatim now lives once, in SourceAdmission -- otherwise this fix would
have left embedding and OCR behind as divergent corners.
README first, since packaging freezes whatever it says at build time:
version pin bumped, and the capability table now mentions that an open
circuit can wait as well as fail fast. Verified the numeric claims by
measurement rather than memory -- record_llm_call still takes 24 fields,
schema.COLUMNS still has 24, meta still caps at 16 keys.
Records what the two Codex review rounds found, which findings held up
under verification, and how each was resolved -- including the one that
changed docs rather than code. Also lists the evidence behind the
completion claim: suite counts, coverage, the 19-minute real-wait Redis
run, and the import contract.
Branch review caught the docs claiming something the code does not do.
CHANGELOG, README and the design's behaviour matrix all said a
force-opened source under circuit_open=wait waits out the full stall
window. It does not: the probe let through after each cooldown is a
real attempt, so it burns a max_attempts slot like any other, and a
401 source usually runs out of retry budget first -- reason is
retry_exhausted, not stalled. Which budget wins depends on
max_attempts against the cooldowns and the stall window.
The behaviour is right; only the prose was wrong. Charging the probe
to the retry budget is exactly the split issue #8 settled: the
question is who spends max_attempts, and a probe does send a real
request. A test now pins it so the claim cannot drift again.
Also drops the planned "woke up" log line. Each wait round already
logs on entry with its duration, and a still-blocked wake-up logs the
next round immediately, so a second line would only double the volume.
README gains the key with the reason a single-source scope wants wait,
and the price of choosing it. .env.example carries the same warning
since README points at it as the full key list. ARCHITECTURE 7.4 records
why the missing cell is unrelated to source count -- and why keying on
len(sources) would be the worse debt -- plus the six-exit retry_after_s
contract and the admission convergence; 9 registers the key.
CHANGELOG stays unreleased per the release checklist: the version bump
belongs to the release run, not here. Its "read this first" section
covers the half-open retry_after_s change, which is visible even on the
default fail_fast setting.
The GatewayUnavailableError docstring told callers to catch it and
retry later, which reads as an invitation for every downstream to write
its own retry layer. Two layers drift -- the library retunes its
backoff and the caller never hears, the caller changes its patience and
the telemetry cannot see it -- and after that nothing can answer how
long a call actually waited or how many attempts it made.
Call-level retry, backoff, source switching and cooldown waiting all
live in the library. The exception means that budget is spent. Retrying
past it is task-level retry, a different thing, and stays outside
(ARCH 7.2, single-layer retry). Also states what retry_after_s means
now and points at CIRCUIT_OPEN.
on_no_runnable now dispatches on why every source was rejected instead
of falling through two serial branches. Under wait, a fully open circuit
sleeps out the cooldown and comes back for another round; the breaker's
protection is untouched (still not a single request leaves during the
wait, so no quota or money burns) -- what changes is whether the caller
dies on the spot or queues.
Dispatching is not cosmetic. Left serial, wait would fall into the quota
branch and a caller with quota_full=fail_fast would get a
quota_exhausted error while its quota was in fact fine.
_nap sleeps to the cooldown deadline rather than polling every 10ms,
which for a 60s cooldown is 6000 round trips per in-flight call on the
Redis backend. Jitter is added on top instead of scaling the wait, since
waking early before a known deadline just earns another rejection. Both
arms clamp to the remaining stall budget, so the worst case per call is
stall_window plus one poll and does not drift with max_cooldown_s. The
clamp's lower bound is the jitter itself, not poll_interval -- the
latter would have lifted the existing [0.5p, 1.0p] quota polling.
Limiter rejections have always chosen between waiting and failing fast;
breaker rejections had no such choice. The new key is the missing cell
of that matrix, shaped exactly like QUOTA_FULL so there is nothing new
to learn. It defaults to fail_fast: flipping the default would move
every existing deployment's worst-case wall clock from milliseconds to
the stall window, which is the wrong direction to impose on anyone.
Single-source scopes are the ones that want wait, and they now have a
way to say so.
The two keys stay separate despite sharing a domain, because a full
quota is "queue for your share" (your turn always comes) while an open
circuit is "wait for the source to recover" (it might not).
Policy validation collapses into SourceAdmission, the only consumer.
The three client constructors used to each carry their own copy of the
quota_full check; adding a second key there would have made eight
copies of the same two lines. Rejection timing and message are
unchanged -- admission is built inside those constructors.
This commit only wires the key through; the control flow that reads it
lands next.
retry_after_s never had a written definition, so each backend improvised
and they drifted apart. It now answers exactly one question: how long
until a retry is *certainly* worth attempting. OPEN has such a moment
(the cooldown deadline); HALF_OPEN does not, because the probe can come
back at any time -- so it reports 0.0, which already means "retry now"
elsewhere in the library.
Six exits are brought in line. The half-open rejection is the one issue
14 reported: it returned the probe lease remainder, a deadlock-guard
value derived from 2x the slowest timeout, so a 60s cooldown told
callers to wait 600s. Worse, retry.py fed that number into the source
cooldown memo, whose set_until only moves forward -- a source stayed
skipped in-process for the whole lease even after its probe succeeded
and the gate closed. That now writes an already-expired deadline, so
the memo goes back to recording only real OPEN cooldowns.
The other five were pre-existing memory/redis divergences hidden by a
contract-test blind spot (the suite pinned that a second caller gets
rejected, never what number it got): redis reported the probe TTL on
grant and the lease remainder on fenced-out writes, where memory has
always reported 0. Contract cases now pin all four half-open exits on
both backends, with 1:1 real-wait variants for redis since the
fake-clock ones skip there.
_pick_runnable and _on_no_runnable lived in three copies (retry.py,
embedding.py, ocr.py), the latter two being verbatim subsets of the
first. Admission semantics keep evolving -- issue #8 changed the stall
accounting, M2.5 added the AIMD pacer, issue #14 is about to add a wait
policy -- and every round had to be applied three times.
SourceAdmission now owns picking a runnable source and deciding what
happens when none is available. The three loops keep their QuotaGate,
BreakerGate and pacer references because _attempt still needs them for
write-back and pacer.leave(); those instances are shared, not rebuilt
(a second pacer would split the in-flight counter). The cooldown memo
moves in wholesale since only admission consumes it.
Behaviour is unchanged: pick differs from the old chat copy only by the
pacer None-guards, on_no_runnable is verbatim identical, and the suite
reports the same 967 passed / 21 skipped / 32 deselected as before. The
one visible change is the settle-and-release warning text, which had
three variants ("permit", "embedding permit", "OCR permit") and is now
one. Tests importing _demote_call_failures follow it to its new home.
The breaker conflates "this source is unhealthy" with "kill this call
now". Limiter rejections already choose between wait and fail_fast;
breaker rejections had no such choice, so a single-source scope loses
its whole retry budget the moment the gate opens.
Design adds {SCOPE}__CIRCUIT_OPEN (default fail_fast, so existing
deployments keep their control flow) and pins retry_after_s to "time
until a *certain* retry moment" across all six gate exits. The latter
also fixes a separate bug the issue missed: a half-open rejection fed
the probe lease (up to 2x timeout) into the source cooldown memo, whose
set_until only moves forward -- so a recovered source stayed blacklisted
in-process long after the gate closed. That one bites multi-source
deployments too, it is just hidden when other sources absorb the load.
Human-approved 2026-08-19; both documents revised after Codex review.
Dates the unreleased section as 1.2.3 (2026-08-19) and moves both version
strings from 1.2.1 in lockstep. The human picked a patch number knowing
this release carries five breaking changes; that is deliberate.
Two lines added to the upgrade hints: the install pin move, matching what
1.2.1 recorded for its own, and a pointer saying the zero-row RLS
self-check now also lives in the README, since CHANGELOG.md never reaches
anyone who only reads the packaged README.
The README is the only prose the sdist freezes, so anything a downstream
needs after `pip install` has to be in it before the build.
Four gaps: the install pin still floored at 1.2.1, which lets an explicit
install land on a version without the schema mode or the text cap the
same README documents; the capability table never mentioned either new
key, telemetry_schema_sql, the retention script, or the DDL template; the
"your llm_calls may be silently empty" warning about the 1.2.1 RLS
template lived only in CHANGELOG.md, which is not in the sdist; and both
references to tools/telemetry_retention.py read as if pip shipped it.
The RLS note goes above the pg-template:rls anchor, not between it and
the fence, so the block parser in test_postgres_telemetry.py still finds
all seven blocks. Telemetry field count re-measured against
inspect.signature(TelemetryRecorder.record_llm_call) and schema.COLUMNS:
still 24, so the table's number stands.
issue #12: downstreams now have a way to control what the telemetry table
keeps, for how long, and who can read it. PGW_TELEMETRY_TEXT_CAP caps
message bodies, responses and thinking at the single telemetry call site
-- default None, so nothing changes unless asked. Retention ships as
tools/telemetry_retention.py, dry-run by default and stepping aside for
DROP PARTITION on partitioned tables, so the library itself never holds
DELETE rights.
The README gains a production deployment template -- three roles,
REVOKE UPDATE/DELETE, RANGE partitioning, RLS -- whose SQL the
integration test parses out of the README itself and runs against a real
Postgres, so the document cannot drift from what works. Writing it
surfaced a defect in the 1.2.1 RLS template: it bound the write-side
policy to a GUC the recorder never sets, which rejected every INSERT and
left the table silently empty.
_validate carried the whole matrix in one function (cc C/13, over the
branch quality gate). Splitting it by what is actually being checked —
shared, sqlite-only, postgres-only — puts every piece at A/B.
Ordering is the part that had to survive: the chain's order is the error
messages' priority, so a run with several bad flags still reports the
same one it did before. The --vacuum/--apply pairing therefore stays in
the shared step ahead of the backend branch, where it was; it is a "do
not rewrite the whole file when you only meant to look" rule, which
holds before the question of which backend a flag belongs to.
No behavior change: all nine parser.error strings are byte-identical and
in the same order, and the eight usage-error cases pass unmodified.
The unreleased entry now covers both issues as one release note: #13 hands
schema control to downstreams, #12 hands over the other half — deleting
data — and ships three knobs that change nothing by default.
Top of the section is the 1.2.1 RLS template defect Task 4 found. That
template bound the write-side policy to app.tenant_id, but PostgresRecorder
writes every tenant through one pool and never calls set_config, so every
INSERT is rejected — and telemetry degrades silently, so the symptom is an
empty table, not an error. The entry says how to check for it (count rows
with a BYPASSRLS role; grep the per-row write warning) and what the new
WITH CHECK (true) template trades away.
ARCHITECTURE gets #12's half of D15: the library must not even hold the
means to delete, because REVOKE UPDATE, DELETE and a retention policy can
only be reconciled by DROP PARTITION (owner) rather than DELETE (app).
7.8 and 9 record the text cap, its default of no truncation, and why the
cut is per text rather than over the serialized JSON.
The library only ever SELECTs/INSERTs into llm_calls (D15), so expiring
rows has to live outside it — holding DELETE would contradict the
REVOKE UPDATE, DELETE the deployment template recommends.
tools/telemetry_retention.py is dry-run by default and prints the row
count, the created_at window and the tenant_id spread so an operator can
tell whether the rows about to go are the intended ones. The Postgres
branch refuses partitioned targets with exit code 3 (DETACH/DROP
PARTITION is O(1); DELETE is not) and otherwise deletes in per-batch
transactions. Missing asyncpg exits 2 rather than degrading quietly:
this is an ops tool, and a silent "0 rows" reads as "already clean".
Exit codes are the contract with the scheduler, so argparse errors were
moved off 2 (now 1) to keep "bad flags" distinguishable from "cannot
reach the database".
The Postgres cases run against the real instance in throwaway schemas —
never public.llm_calls — and the batch case asserts the shared table's
row count is unchanged, so a search_path that failed to apply lands as a
red test instead of a deletion.
`PGW_TELEMETRY_TEXT_CAP` now reaches the emitter on every assembly path.
Unset means no truncation, which stays the default: a truncated row is
no longer audit evidence and cannot be replayed, and downstreams rely on
that today. The flip side — contracts and bids sitting in `llm_calls`
indefinitely, multi-tenant — is spelled out in `.env.example` so readers
can weigh both.
All three `from_settings` paths are wired (chat, embedding, OCR): they
write the same table, so capping only chat would leave half of it
uncontrolled. `TelemetryEmitter.__init__` now rejects `text_cap <= 0`;
it is the single point where the three clients converge, so the direct
construction path — a public assembly route the settings guard never
sees — is covered too. `0` would otherwise reduce every body to a bare
elision marker.
Chat rows stored full message and response text with no upper bound, so
downstream contracts and tenders lived in llm_calls indefinitely. Add
_cap_text/_cap_messages in the single telemetry exit (_record), applied
after digest_messages and before json.dumps, plus to response/thinking.
Capping is per text, not over the serialized JSON: cutting the whole
string would emit invalid JSON into an unvalidated TEXT column. The cap
builds new dicts and never mutates in place — digest_messages passes
non-list content straight through as the same object, so an in-place cut
would silently poison the caller's messages and the cache key.
text_cap is required on TelemetryEmitter (internal class, three known
construction sites) and defaults to None on the three public clients, so
the default behaviour stays byte-for-byte identical. Settings wiring
lands separately.
issue #13: the library no longer alters a downstream Postgres table on
its own. PGW_TELEMETRY_SCHEMA_MODE is tri-state and defaults by backend
-- SQLite keeps auto-migrating a local file, Postgres switches to manual,
where a stale table gets a named warning with runnable SQL and the INSERT
is trimmed to the columns that exist rather than dropping every row.
Schema constants now live in telemetry/schema.py so the SQL the library
prints cannot drift from the DDL it runs, and telemetry_schema_sql is
exported for downstreams writing their own migrations. The PG write drops
its conflict target, which partitioned tables require and which issue #12
depends on.
PG requires a partitioned table's unique constraints to include the
partition key, so issue #12's RANGE partitioning on created_at forces
the primary key to (call_id, created_at). The old
`ON CONFLICT (call_id) DO NOTHING` then matches no constraint and PG
rejects every row with
there is no unique or exclusion constraint matching the
ON CONFLICT specification
which the recorder swallows as a per-row warning: telemetry would go
silently dark under a partitioned deployment. The target-free form is
valid on both table shapes and is literally equivalent on a plain table
(the primary key is its only unique constraint). SQLite's
`INSERT OR IGNORE` already carries no target and is untouched.
Integration coverage on the real PG instance, both inside self-created
temp schemas: a plain table still keeps one row per call_id, and a
table partitioned by created_at now accepts writes and reads them back.
The second case was red before this change with the error above.
Not related to the schema work. These three fail ruff format --check on
main as well -- the pinned ruff is newer than whatever last formatted
them -- and a red make check makes the per-task quality gate useless for
everything that follows.
iomgaa