fix: guard floor phase against n_trainval budget + assert epsilon on output

Codex Task 8 审查修复:
- Important 1: _satisfy_floors 每步移入前检查预算,floor 需求超 n_trainval 时
  抛 InfeasibleSplitError(fail loud),保证 trainval 永不超额挤占 test;补预算超限测试。
- Important 2: 确定性测试末尾用 _epsilon_ok 断言产出 test 真满足 ε(两维偏差回归护栏),
  并断言 trainval <= n_trainval。
- Minor: fixture docstring 注明结构真实 / signal 二次构造,正式运行由真实诊断替换。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-15 12:38:33 -04:00
parent c3187167c8
commit b497db97ba
2 changed files with 42 additions and 4 deletions
+31 -2
View File
@@ -54,8 +54,9 @@ def _load_adhoc_predictions() -> list[dict]:
def _real_shaped_video_records() -> list:
"""用真实预测结构 + 二次构造 T2 信号构建 300 个 VideoRecord。
题型 / 难度画像取自真实 infer_adhoc 预测;因真实诊断尚未跑,T2 信号是二次构造
对高信号题型(Counting / Action Reasoning)的错题标 T2 defecterror_type 循环取 4 类,
注意:视频 type / difficulty 结构真实(取自真实 infer_adhoc 预测),signal 叠加是二次构造
pre-diagnosis 阶段固有限制,真实诊断尚未跑);正式运行时 T2 信号由真实诊断产物替换。
这里对高信号题型(Counting / Action Reasoning)的错题标 T2 defecterror_type 循环取 4 类,
使 floor 约束有料、多样性格子有区分度。
返回:
@@ -158,6 +159,7 @@ def test_build_video_records_covers_all_videos_with_difficulty_and_types():
def test_select_split_video_disjoint_and_floor_and_deterministic():
from app.harness.split_selection import (
SelectConfig,
_epsilon_ok,
derive_reportable_types,
select_split,
)
@@ -181,6 +183,10 @@ def test_select_split_video_disjoint_and_floor_and_deterministic():
v.wrong_by_type.get("Counting Problem", 0) for v in videos if v.video_id in trainval_ids
)
assert counting_defects >= 3 # floor 硬约束满足
assert len(a.trainval) <= cfg.n_trainval # trainval 永不超预算
test_ids = set(a.test)
test_records = [v for v in videos if v.video_id in test_ids]
assert _epsilon_ok(test_records, videos, cfg) # 产出 test 真满足 ε(回归护栏)
def test_infeasible_floor_vs_epsilon_raises():
@@ -204,6 +210,29 @@ def test_infeasible_floor_vs_epsilon_raises():
)
def test_infeasible_floor_exceeds_n_trainval_budget_raises():
from app.harness.split_selection import (
InfeasibleSplitError,
SelectConfig,
select_split,
)
# floor 需求(Action Reasoning 48 缺陷)远超 n_trainval=1 预算:单视频至多带 1 题 AR 缺陷,
# 无法在 1 个 trainval 名额内满足 floor=40 → 抛 InfeasibleSplitError(预算不足)。
videos = _real_shaped_video_records()
with pytest.raises(InfeasibleSplitError):
select_split(
videos,
config=SelectConfig(
n_trainval=1,
floor_k={"Action Reasoning": 40},
epsilon=1.0,
reportable_types=set(),
seed=3,
),
)
def test_derive_reportable_types():
from app.harness.split_selection import derive_reportable_types