test: skip L8 when the channel drops the model instead of failing
The 2:25 slow run left exactly one red: kimi-for-coding answers 404 model_not_found because the channel removed it from the account group between 09:44 (four green probes, correct model_reported) and 15:00. L8 was reading that as "the capability table drifted", which is a statement about the model the channel no longer serves. The 404/model_not_found rule already used by T10 now lives in one helper and is applied on the L1-L9 side too, via the same unreachable fallback: that one rejection skips and records an uncovered row, every other RequestRejectedError still bubbles, since those are the real failures this suite exists to catch.
This commit is contained in:
@@ -200,6 +200,21 @@ def _skip_if_unreachable(exc: Exception, matrix_id: str, desc: str):
|
||||
pytest.skip(f"{matrix_id} 源不可用,已记为未覆盖: {str(exc)[:120]}")
|
||||
|
||||
|
||||
def _is_model_not_found(status_code: int | None, body_text: str) -> bool:
|
||||
"""上游说的是**该渠道根本没有这个型号**(404 `model_not_found`),不是"这次请求有问题"。
|
||||
|
||||
2026-09-05 实测: kimi-for-coding 上午 09:44 四项全 PASS 且 `model_reported` 正确,
|
||||
15:00 起变成 `404 | {"error":{...,"type":"model_not_found"}}` —— 渠道把它从账号组里
|
||||
摘掉了。它与"上游拒绝这一档"(400 invalid tier)完全不是一件事: 后者是**关于档位/
|
||||
能力的结论**,前者对它们一无所知,只说明源当下不可用。混为一谈会让一次渠道调整变成
|
||||
"能力表漂移"的假红,严重时反过来把能力表改错。
|
||||
|
||||
判据取 `status_code` 与响应体里的 `type` 字段(机器可判的那格),不做整条 message
|
||||
的模糊匹配 —— message 里还拼着源名与库自己的话,匹配它等于赌文案不变。
|
||||
"""
|
||||
return status_code == 404 and "model_not_found" in (body_text or "")
|
||||
|
||||
|
||||
async def _rounds_or_skip(matrix_id: str, desc: str, rounds: int, **source_overrides) -> list[dict]:
|
||||
"""`_run_rounds` 加上"源不可用即记为未覆盖"的兜底(本模块 docstring 的纪律)。
|
||||
|
||||
@@ -208,8 +223,9 @@ async def _rounds_or_skip(matrix_id: str, desc: str, rounds: int, **source_overr
|
||||
`_record()` **之前**,报告里连一行「未覆盖」都不会留下——事后翻报告只看到该
|
||||
矩阵行凭空消失,判断不出当时到底发生了什么。
|
||||
|
||||
只吞网关/网络三类。**不吞 `ValueError` / `RequestRejectedError`**: 前者是装配
|
||||
守卫,后者是"请求本身被拒",两者都是本组要抓的真失败,吞掉即成静默。
|
||||
只吞网关/网络三类,外加"该渠道没有这个型号"这**一种**被拒(见
|
||||
`_is_model_not_found`)。**其余 `ValueError` / `RequestRejectedError` 照旧冒泡**:
|
||||
前者是装配守卫,后者是"请求本身被拒",两者都是本组要抓的真失败,吞掉即成静默。
|
||||
"""
|
||||
try:
|
||||
return await _run_rounds(rounds, **source_overrides)
|
||||
@@ -217,6 +233,11 @@ async def _rounds_or_skip(matrix_id: str, desc: str, rounds: int, **source_overr
|
||||
# `_skip_if_unreachable` 内部 `pytest.skip` 必抛,此处不会落到函数末尾
|
||||
_skip_if_unreachable(exc, matrix_id, desc)
|
||||
raise # pragma: no cover —— 只为让静态读者看清控制流不会往下走
|
||||
except RequestRejectedError as exc:
|
||||
if not _is_model_not_found(exc.status_code, exc.body_text):
|
||||
raise
|
||||
_skip_if_unreachable(exc, matrix_id, desc)
|
||||
raise # pragma: no cover
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
@@ -522,6 +543,10 @@ class TestCapabilityDrift:
|
||||
身份不符一律 SKIP 记为未覆盖: 那是外部渠道问题,不是能力表的证据。
|
||||
**三个型号一视同仁**,不能只挡报错的那两个: glm-5.2 这次侥幸 PASS(被路由到的
|
||||
glm-5.3 那几轮恰好没推理),而侥幸绿的数据与红的数据一样不可信。
|
||||
|
||||
另一种同类外部状况走 `_rounds_or_skip`: 渠道把型号从账号组里摘了(404
|
||||
`model_not_found`),同样是源不可用而非能力表漂移(2026-09-05 kimi-for-coding
|
||||
当天从 PASS 变 404,旧版把它记成了一条 FAIL)。
|
||||
"""
|
||||
cap = get_capability(model)
|
||||
provider = _MODEL_PROVIDER[model]
|
||||
@@ -535,10 +560,9 @@ class TestCapabilityDrift:
|
||||
)
|
||||
_record("L8", desc, "PASS", "装配期按声明拒绝,与实测一致")
|
||||
return
|
||||
try:
|
||||
obs = await _run_rounds(rounds, provider=provider, model=model, enable_thinking=False)
|
||||
except (AllSourcesExhausted, SourceDeadError, TransientError) as exc:
|
||||
_skip_if_unreachable(exc, "L8", desc)
|
||||
obs = await _rounds_or_skip(
|
||||
"L8", desc, rounds, provider=provider, model=model, enable_thinking=False
|
||||
)
|
||||
strangers = _identity_mismatch(model, obs)
|
||||
if strangers:
|
||||
_record(
|
||||
@@ -793,18 +817,12 @@ def _probe_ok(obs: dict) -> bool:
|
||||
|
||||
|
||||
def _model_missing(obs: dict) -> bool:
|
||||
"""这一轮失败的原因是**该渠道根本没有这个型号**(404 `model_not_found`)。
|
||||
"""`_is_model_not_found` 的逐轮字典适配:这一轮是"该渠道没有这个型号"而失败的。
|
||||
|
||||
2026-09-05 实测: kimi-for-coding 上午 09:44 四项全 PASS 且 `model_reported`
|
||||
正确,15:00 就变成 `404 | {"error":{...,"type":"model_not_found"}}` —— 渠道
|
||||
把它从账号组里摘掉了。这与"上游拒绝这一档"(400 invalid tier)完全不是一件事:
|
||||
后者是**关于档位的结论**,前者对档位一无所知,只说明源当下不可用。混为一谈会
|
||||
让一次渠道调整变成"能力表漂移"的假红,严重时反过来把能力表改错。
|
||||
|
||||
判据取 `status_code` 与响应体里的 `type` 字段(机器可判的那格),不做整条
|
||||
message 的模糊匹配 —— message 里还拼着源名与库自己的话,匹配它等于赌文案不变。
|
||||
判据本身与 L1-L9 共用一份(理由见 `_is_model_not_found`),此处只负责从失败轮
|
||||
里取出那两格 —— 两处各写一份迟早只改一处。
|
||||
"""
|
||||
return obs.get("error_status") == 404 and "model_not_found" in (obs.get("error_body") or "")
|
||||
return _is_model_not_found(obs.get("error_status"), obs.get("error_body") or "")
|
||||
|
||||
|
||||
def _probe_rejected(obs: dict) -> bool:
|
||||
|
||||
Reference in New Issue
Block a user