feat: let circuit_open=wait queue instead of killing the call
on_no_runnable now dispatches on why every source was rejected instead of falling through two serial branches. Under wait, a fully open circuit sleeps out the cooldown and comes back for another round; the breaker's protection is untouched (still not a single request leaves during the wait, so no quota or money burns) -- what changes is whether the caller dies on the spot or queues. Dispatching is not cosmetic. Left serial, wait would fall into the quota branch and a caller with quota_full=fail_fast would get a quota_exhausted error while its quota was in fact fine. _nap sleeps to the cooldown deadline rather than polling every 10ms, which for a 60s cooldown is 6000 round trips per in-flight call on the Redis backend. Jitter is added on top instead of scaling the wait, since waking early before a known deadline just earns another rejection. Both arms clamp to the remaining stall budget, so the worst case per call is stall_window plus one poll and does not drift with max_cooldown_s. The clamp's lower bound is the jitter itself, not poll_interval -- the latter would have lifted the existing [0.5p, 1.0p] quota polling.
This commit is contained in:
@@ -224,28 +224,66 @@ class SourceAdmission:
|
||||
async def on_no_runnable(
|
||||
self, gate_rejections: int, reasons: dict[str, str], clock: StallClock
|
||||
) -> None:
|
||||
"""一个源都挑不出来时的处置: 判死或等待一轮再来。"""
|
||||
"""一个源都挑不出来时的处置: **按拒绝原因分派**到各自的策略。
|
||||
|
||||
分派而非串行是硬要求(issue #14): 串行写法下 `circuit_open=wait` 不抛
|
||||
之后会径直掉进配额分支,`quota_full=fail_fast` 的调用方于是收到一个
|
||||
`reason=quota_exhausted` 的异常——而配额其实是满的,坏的是熔断门。
|
||||
"""
|
||||
names = tuple(s.name for s in self._sources)
|
||||
if gate_rejections == len(self._sources):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
# 全部因熔断类原因(门开路 / 本地冷却备忘)被拒
|
||||
if self._circuit_open == "fail_fast":
|
||||
raise CircuitOpenError(
|
||||
scope=self._scope,
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
# wait: 保护作用完整保留(这一轮照样一个请求都不发),改变的只是
|
||||
# 调用方当场死还是排队等——多源可换源故 fail-fast 对,单源无源可换
|
||||
hint = await self._breaker.retry_after_s(names)
|
||||
else:
|
||||
# 至少一个源是被配额/AIMD 挡的,归 quota_full 管
|
||||
if self._quota_full == "fail_fast":
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="quota_exhausted",
|
||||
retry_after_s=self._bp.poll_interval_s,
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
hint = 0.0
|
||||
if await self.stalled(clock):
|
||||
names = tuple(s.name for s in self._sources)
|
||||
raise AllSourcesExhausted(
|
||||
scope=self._scope,
|
||||
reason="stalled",
|
||||
retry_after_s=await self._breaker.retry_after_s(names),
|
||||
per_source_reasons=reasons,
|
||||
)
|
||||
# jitter ∈ [0.5p, 1.0p] 防惊群(CHS governance.py:283-285)
|
||||
await self._sleep(self._bp.poll_interval_s * (0.5 + 0.5 * self._rng()))
|
||||
nap = self._nap(hint, clock)
|
||||
if hint > 0:
|
||||
logger.info(
|
||||
"熔断开路等待 {:.1f}s 后重试(scope={}, 原因={})", nap, self._scope, reasons
|
||||
)
|
||||
await self._sleep(nap)
|
||||
|
||||
def _nap(self, hint: float, clock: StallClock) -> float:
|
||||
"""本轮等待多久。**必须在 `stalled()` 判定之后调用**(预算可能已耗尽)。
|
||||
|
||||
`hint > 0`(熔断开路有确定的冷却截止)时睡到那个时刻,而不是按
|
||||
`poll_interval` 空转——60 秒冷却用 10ms 轮询是 6000 次空转,内存后端
|
||||
只是查字典,Redis 后端则是 6000 次往返 × 每个在途调用。抖动**上**加
|
||||
而非缩放(既有 quota 路径是 `[0.5p, 1.0p]`): 对一个确定的截止时刻提前
|
||||
醒来必然被再拒一次,白跑一趟。
|
||||
|
||||
两档都夹到剩余 stall 预算,故单次调用的最坏墙钟是 `stall_window_s`
|
||||
加一个 poll 间隔,不随 `max_cooldown_s` 漂移。多加的那一格是因为
|
||||
`stalled()` 判据是 `>` 而非 `>=`——恰好睡到窗口边界不判死,留这一格
|
||||
让下一轮必定判死。`hint == 0` 时整个式子退化为既有的 jitter 轮询。
|
||||
"""
|
||||
jitter = self._bp.poll_interval_s * (0.5 + 0.5 * self._rng())
|
||||
budget = self._bp.stall_window_s - clock.stalled_s() + self._bp.poll_interval_s
|
||||
wait = hint + jitter if hint > 0 else jitter
|
||||
# 下界取 jitter 而非 poll_interval: 既有 quota 轮询是 [0.5p, 1.0p],用
|
||||
# poll_interval 兜底会把 rng→0 那半边抬上去。预算为负时(本地已超窗但
|
||||
# 全局仍在出餐,故 stalled() 不判死)靠它退回正常轮询节奏,不忙循环。
|
||||
return max(jitter, min(wait, budget))
|
||||
|
||||
Reference in New Issue
Block a user