iomgaa/PolyGateway
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases 9 Wiki Activity

fix: pin retry_after_s to the next certain retry moment

Browse Source 
retry_after_s never had a written definition, so each backend improvised
and they drifted apart. It now answers exactly one question: how long
until a retry is *certainly* worth attempting. OPEN has such a moment
(the cooldown deadline); HALF_OPEN does not, because the probe can come
back at any time -- so it reports 0.0, which already means "retry now"
elsewhere in the library.

Six exits are brought in line. The half-open rejection is the one issue
14 reported: it returned the probe lease remainder, a deadlock-guard
value derived from 2x the slowest timeout, so a 60s cooldown told
callers to wait 600s. Worse, retry.py fed that number into the source
cooldown memo, whose set_until only moves forward -- a source stayed
skipped in-process for the whole lease even after its probe succeeded
and the gate closed. That now writes an already-expired deadline, so
the memo goes back to recording only real OPEN cooldowns.

The other five were pre-existing memory/redis divergences hidden by a
contract-test blind spot (the suite pinned that a second caller gets
rejected, never what number it got): redis reported the probe TTL on
grant and the lease remainder on fenced-out writes, where memory has
always reported 0. Contract cases now pin all four half-open exits on
both backends, with 1:1 real-wait variants for redis since the
fake-clock ones skip there.
This commit is contained in:
iomgaa
2026-08-20 00:09:20 -04:00
parent 942af99856
commit 8edd3fb2cd
4 changed files with 114 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/polygateway/backends/memory/breaker.py
+20 -16
View File
@@ -100,6 +100,22 @@ class InMemoryGate:
streak = max(1, g.reopen_streak)
return min(self._cfg.cooldown_s * (2 ** (streak - 1)), self._cfg.max_cooldown_s)
def _remaining(self, g: _SourceGate) -> float:
"""距离**确定**可再试的时刻还有多久(issue #14 的契约定义)。
OPEN 的冷却截止是确定时刻;HALF_OPEN 下探针随时可能出结果,**不存在**
确定时刻,故 `0.0`——`0 = 可立即重试` 是库既有约定。此前这里返回探针
租约剩余,而租约长度是死锁保护参数(派生自 `2 × 最慢源 timeout`),与
"源多久能恢复"无因果关系;它还被喂进源冷却备忘,而备忘 `set_until`
取更晚者不可回退,于是门恢复 CLOSED 后本进程仍跳过该源整整一个租约。
三个出口(`try_enter` 拒绝、`_snapshot`、`retry_after_s`)共用本方法,
避免同一语义在三处各算一遍而漂移。
"""
if g.state is GateState.OPEN:
return max(0.0, g.open_until - self._now())
return 0.0
def _grant_probe(self, g: _SourceGate, source_name: str, owner: str) -> GateDecision:
g.state = GateState.HALF_OPEN
g.probe_owner = owner
@@ -140,7 +156,7 @@ class InMemoryGate:
epoch=g.epoch,
is_probe=False,
probe_owner=None,
retry_after_s=g.open_until - now,
retry_after_s=self._remaining(g),
)
# HALF_OPEN: 探针在途;租约过期则接管,否则拒绝(防惊群)
if now >= g.probe_expires:
@@ -152,7 +168,7 @@ class InMemoryGate:
epoch=g.epoch,
is_probe=False,
probe_owner=None,
retry_after_s=g.probe_expires - now,
retry_after_s=self._remaining(g),
)
def _fenced(self, g: _SourceGate, entry: GateDecision) -> bool:
@@ -172,9 +188,7 @@ class InMemoryGate:
state=g.state,
epoch=g.epoch,
failure_count=g.fails,
retry_after_s=max(0.0, g.open_until - self._now())
if g.state is GateState.OPEN
else 0.0,
retry_after_s=self._remaining(g),
)
def _open(self, g: _SourceGate, reason: str, *, bump_streak: bool) -> None:
@@ -258,14 +272,4 @@ class InMemoryGate:
"""集合中最早可尝试时间;健康/到期返回 0。"""
if not sources:
raise ValueError("sources 不能为空")
now = self._now()
waits = []
for name in sources:
g = self._gate(name)
if g.state is GateState.OPEN:
waits.append(max(0.0, g.open_until - now))
elif g.state is GateState.HALF_OPEN:
waits.append(max(0.0, g.probe_expires - now))
else:
waits.append(0.0)
return min(waits)
return min(self._remaining(self._gate(name)) for name in sources)
src/polygateway/backends/redis/breaker.py
+3 -11
View File
@@ -42,7 +42,8 @@ if state == 'open' and now < open_until then
return {0, state, epoch, 0, '', open_until - now}
end
if state == 'half_open' and now < probe_until then
return {0, state, epoch, 0, '', probe_until - now}
-- 探针在途: 无确定的可再试时刻 → 0(issue #14,与 memory `_remaining` 同口径)
return {0, state, epoch, 0, '', 0}
end
local next_probe_until = now + tonumber(ARGV[2])
@@ -50,7 +51,7 @@ redis.call('HSET', KEYS[1],
'state', 'half_open',
'probe_owner', ARGV[1],
'probe_until', next_probe_until)
return {1, 'half_open', epoch, 1, ARGV[1], tonumber(ARGV[2])}
return {1, 'half_open', epoch, 1, ARGV[1], 0}
"""
# M2.5 窗口/退避公共片段(拼接进 success/failure 脚本;Lua 脚本间无法共享函数)
@@ -124,8 +125,6 @@ end
local deadline = 0
if state == 'open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
elseif state == 'half_open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
end
return {0, state, epoch, failures, math.max(deadline - now, 0)}
"""
@@ -156,8 +155,6 @@ if not matches then
local deadline = 0
if state == 'open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
elseif state == 'half_open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
end
return {0, state, epoch, failures, math.max(deadline - now, 0)}
end
@@ -255,8 +252,6 @@ end
local deadline = 0
if state == 'open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'open_until') or '0')
elseif state == 'half_open' then
deadline = tonumber(redis.call('HGET', KEYS[1], 'probe_until') or '0')
end
return {0, state, epoch, failures, math.max(deadline - now, 0)}
"""
@@ -272,9 +267,6 @@ for _, key in ipairs(KEYS) do
if state == 'open' then
local deadline = tonumber(redis.call('HGET', key, 'open_until') or '0')
remaining = math.max(deadline - now, 0)
elseif state == 'half_open' then
local deadline = tonumber(redis.call('HGET', key, 'probe_until') or '0')
remaining = math.max(deadline - now, 0)
end
if minimum == nil or remaining < minimum then minimum = remaining end
end