AIMD ceiling now respects per-source max_concurrency and the pacer is
assembled explicitly in the client; MIN_CALLS parses as strict int;
acceptance doc corrects source-5 attempt count to 549; design and
migration notes aligned with implemented 429/stall/suppression
semantics and AIMD constants documented.
Round 6 hit account-level rate throttling the concurrency AIMD cannot
absorb: at 26 req/min the gateway still returned 16% 429s and each one
burned a third of the retry budget. Retry-After-guided 429s now back
off without consuming attempts (gRPC pushback semantics); the retry
loop gains a per-call ceiling using the same dual-condition stall
verdict as quota-wait (local window exceeded AND no global progress).
Round 5 left three residual failure classes; ladder exhaustion (3.2%
of structured calls with a single re-ask) and 429 leakage (5.8%, AIMD
oscillating above the sustainable point) are addressable: re-ask
default goes 1 to 2 (conservative vs instructor's 3) and the AIMD cut
factor drops to 0.5.
Round 4 showed tail placement routes the third attempt to junk sources
whenever the credible alternative is gate-skipped (tight-RPM source
admitted as credible, then skipped by the limiter, falling through to
the flapping watchdog source). Demoted sources now sit between credible
and non-credible candidates.
Round 3 showed unconditional yield-after-two-failures pushes the third
attempt onto known-bad sources in heterogeneous pools (83% vs 10%
expected success). OutcomeAwareSelector now exposes health(); a failed
source only yields when some untried candidate scores at least half its
health. Health-blind selectors keep the unconditional rule.
P6 round 2 showed routing convergence turns account-level 429s into
the binding constraint (62% throttle rate at full concurrency). Each
source now carries a local AIMD limit: multiplicative cut on 429,
additive growth on success. Over-limit picks queue via the existing
quota-wait poll instead of burning retry budget or tripping the
circuit-open verdict.
RetryMW keeps a per-call failure map (local, never instance state):
a source failing twice in one call yields to the next candidate.
Attempt outcomes feed OutcomeAwareSelector behind a swallow-and-warn
guard; ResultInvalid and provider-rejected paths record success with
count_attempt=False so the breaker window stays clean. Same accounting
applied in EmbeddingClient.
Score is success-rate EWMA over (1 + inflight) with a 0.05 exploration
floor so quarantined sources can prove recovery; EWMA climb doubles as
slow-start. New optional OutcomeAwareSelector port feeds attempt
outcomes.
Threshold auto-raise now uses per-source concurrency only (the M2
global-concurrency formula neutered the breaker at scale). Four new
optional keys: MIN_CALLS, FAIL_RATE, WINDOW_S, MAX_COOLDOWN_S.
Verifier adversarial cases showed pooled-room exemption could hide real
breaches: edge rows may only borrow from the neighbor on their own side,
and neighbor room is consumed globally so two windows cannot claim the
same slot. Also parse SQLite created_at as UTC, guard the dispatch
semaphore on progress-callback failure, and print a caveat that rescore
live checks reflect current db3 state.
RPM invariant now buckets by admit time on the Redis server clock,
exempts +/-2s boundary jitter, and excludes cache-hit rows that never
consumed a limiter slot. RSS sampling reports current ps RSS instead of
the monotonic ru_maxrss peak. Dispatch is paced by the concurrency
semaphore so --max-hours stays live and memory stays bounded. Add
--rescore RUN_ID to re-judge a finished run offline.
Classify empty completions as transient per human ruling (fixes flaky
real-gateway smoke and prevents caching empty responses), rename the
factory injection parameter gate to breaker per the frozen design,
rewrite the probe-entry cleanup without except BaseException, declare
python-dotenv explicitly, add a mid-backoff cancellation test, and
record all implementation errata in the design and architecture docs.
Includes config aggregation for multi-source env keys, from_env and
from_settings factories with explicit shared-backend injection,
gather_bounded, top-level exports, tightened import-linter layers with
the gate removed from the Makefile, and the finalized .env.example.
iomgaa