fix: keep a low-tier answer out of the cache slot a max-tier one filled

The per-call reasoning tier never reached the cache key, and the model
fingerprint could not stand in for it: the fingerprint is computed once at
assembly time, so two calls on the same client asking for low and max looked
identical to it. Same messages, different tiers, one shared entry -- the
verbatim replay of issue #4's five seeds all hitting the same response.

Source-level tiers join the fingerprint under the same rule enable_thinking
already follows (appended only when the source takes a position), and the
filter that decides which sources enter the mark set is widened to match --
without that, a source configured with nothing but REASONING_EFFORT would
never reach _fingerprint_mark at all.

None (no opinion) and Effort.NONE (asked not to reason) stay distinct keys.
Sources that opine on neither keep byte-identical keys and fingerprints, so
nothing existing cold-starts.
This commit is contained in:
2026-09-05 02:39:38 -04:00
parent 80a8013642
commit 33c8e8274b
4 changed files with 137 additions and 10 deletions
+24 -8
View File
@@ -99,14 +99,21 @@ def _guard_thinking(
def _fingerprint_mark(source: SourceConfig) -> str:
"""单源的指纹标记;`enable_thinking` 仅在**表态时**追加。
"""单源的指纹标记;`enable_thinking` 与 `reasoning_effort` 仅在**表态时**追加。
只在表态时追加不是省事: 这样只配了 `extra_body` 的存量源字面量与 issue #4
时期逐字相同,升级本版本不会给它们平白来一次全量缓存冷启动。
`reasoning_effort`(issue #20)与 `enable_thinking` 同规则、同理由: 它一旦真正
改变请求体,"把源级档位从 low 改成 max 后重启"就会读到 low 档时缓存的旧响应。
两者取值域不相交(`"none"`/`"low"`… vs `true`/`false`),故追加进同一个列表也
不会把两种写法摘要成同一身份。
"""
parts: list[Any] = [source.model, dict(source.extra_body)]
if source.enable_thinking is not None:
parts.append(source.enable_thinking)
if source.reasoning_effort is not None:
parts.append(source.reasoning_effort)
return json.dumps(parts, sort_keys=True, ensure_ascii=False)
@@ -114,16 +121,25 @@ def build_model_fingerprint(sources: Iterable[SourceConfig]) -> str:
"""缓存 key 的模型身份: 多源 scope = 排序去重的 model 合集。
配置级采样参数(`extra_body`)必须参与,否则把 temperature 从 0 改成 1
后重启仍会读到旧缓存(issue #4 设计决策 C)。`enable_thinking` 同理
(issue #5): 它一旦真正改变请求体,"关掉推理后重启"就会读到开着推理时
缓存的旧响应。全源者皆未表态时字面量与历史实现逐字相同,不触发存量
缓存冷启动。
后重启仍会读到旧缓存(issue #4 设计决策 C)。`enable_thinking`(issue #5)与
源级 `reasoning_effort`(issue #20)同理: 它一旦真正改变请求体,"关掉推理后
重启"就会读到开着推理时缓存的旧响应。全源者皆未表态时字面量与历史实现逐字
相同,不触发存量缓存冷启动。
注意本指纹是**装配期**算出的**集合级**身份,覆盖不到逐次调用变化的请求级档位
——后者由 `build_cache_key` 的 `reasoning_effort` 参数单独承担(ARCH §7.5)。
"""
fingerprint = ",".join(sorted({s.model for s in sources}))
# 按 (model, extra_body[, enable_thinking]) 而非源名摘要: 语义是"本 scope
# 会用哪些(模型, 请求形态)组合",改源名不该误触全量冷启动
# 按 (model, extra_body[, enable_thinking][, reasoning_effort]) 而非源名摘要:
# 语义是"本 scope 会用哪些(模型, 请求形态)组合",改源名不该误触全量冷启动
# 过滤条件必须与 `_fingerprint_mark` 追加的字段逐项对齐: 漏掉一项,只配了该项
# 的源根本进不了 marks,`_fingerprint_mark` 改了也白改
marks = sorted(
{_fingerprint_mark(s) for s in sources if s.extra_body or s.enable_thinking is not None}
{
_fingerprint_mark(s)
for s in sources
if s.extra_body or s.enable_thinking is not None or s.reasoning_effort is not None
}
)
if marks:
digest = hashlib.sha256("".join(marks).encode("utf-8")).hexdigest()
+15 -1
View File
@@ -17,7 +17,7 @@ from typing import TYPE_CHECKING, Any
from loguru import logger
from polygateway.types import ChatRequest, LLMResponse, ThinkingObservation
from polygateway.types import ChatRequest, Effort, LLMResponse, ThinkingObservation
if TYPE_CHECKING:
from collections.abc import Mapping
@@ -83,12 +83,21 @@ def build_cache_key(
salt: str | None,
*,
sampling: Mapping[str, Any] | None = None,
reasoning_effort: Effort | None = None,
) -> str:
"""缓存 key 公式;salt 仅非 None 时参与(VT 旧键语义: 不传 salt 键形不变)。
`sampling` 仅**非空**时参与(与 salt 的"仅非 None"不同——空串是有意义的
salt,而空采样参数与不传无语义差别)。它必须进 key: 否则同 messages 跑 5 个
seed 会全部命中第一次的响应,标准差恒为 0 且不报错(issue #4 决策 C)。
`reasoning_effort` 是**请求级**档位(issue #20),仅非 `None` 时参与。它不能靠
`model_fingerprint` 代劳: 后者是**装配期**算出的集合级指纹,一次调用改档位不会
让它变一个字节;不进 key 则同 messages 跑 low 与 max 互相命中,是 issue #4
「5 个 seed 全命中同一响应」的逐字翻版。
判据用 `is not None` 而非真值: `Effort.NONE`(明确要求不推理)与 `None`
(不表态)语义不同——前者拿到的是没有推理过程的响应,合并即毒化。
"""
key_obj: dict[str, Any] = {
"model": model_fingerprint,
@@ -99,6 +108,8 @@ def build_cache_key(
key_obj["salt"] = salt
if sampling:
key_obj["sampling"] = dict(sampling)
if reasoning_effort is not None:
key_obj["reasoning_effort"] = str(reasoning_effort)
payload = json.dumps(key_obj, sort_keys=True, ensure_ascii=False)
return _KEY_PREFIX + hashlib.sha256(payload.encode("utf-8")).hexdigest()
@@ -139,6 +150,9 @@ class CacheMW:
namespace,
request.cache_salt,
sampling=request.sampling,
# 请求级档位必须逐次进 key: `self._fingerprint` 是装配期的集合级指纹,
# 同一个 client 上 low 与 max 两次调用在它眼里毫无分别(issue #20)
reasoning_effort=request.reasoning_effort,
)
cached = await self._safe_get(key)
if cached is not None: