fix: address whole-impl review (INFRA T0 rows, reproducible manifest, evolution_target report, dead config, canonical DRY)

C-1: persist_infra_t0_rows 补 INFRA/空预测错题的 T0 信号行(不进诊断故须单独落库),run_pipeline 加 Phase 0,dry-run 用假数据走通。
C-2: CLI 加 --generated-at,真实运行默认盖真实 UTC now,可显式固定以字节级复现 manifest。
I-1: coverage_report 增 evolution_target_distribution(T2 信号按 tool/skill/system 计数)。
I-2: 删除 PoolConfig 死字段 n_trainval/floor_k/epsilon/report_floor/val_wrong_min(grep 确认无消费者,视频级切分用独立 VideoSplitConfig/SplitBuildConfig/SelectConfig)。
I-3: 抽共享 load_canonical_predictions(db_path, run_id),CLI 与 build_split 共用;消除 canonical 取行 + correct 判定重复。
M-1: build_split docstring 注明 val_wrong_min-agnostic 契约(McNemar 护栏由 CLI 冻结后执行,Task 11 契约)。
This commit is contained in:
2026-07-15 13:39:14 -04:00
parent 02b8145b7f
commit 8fef7ced42
6 changed files with 302 additions and 134 deletions
+6 -53
View File
@@ -1,10 +1,12 @@
"""视频级切分科研旋钮单元测试:诊断指纹 + val_wrong_min 功效护栏 + PoolConfig 新字段
"""视频级切分科研旋钮单元测试:诊断指纹 + val_wrong_min 功效护栏。
覆盖:
- diag_fingerprint 对 (prompt 版本 / 模型 / 代码版本) 三元组确定且敏感;
- split_by_video_assignment 的 val_wrong_min 门控 fail loud(验证信号不足即报错);
- val_wrong_min 默认 0 时行为与 Task 11 现有调用完全一致(不回归)
- PoolConfig 能接收视频级切分的五个新旋钮字段(纯 dataclass 装配)。
- val_wrong_min 默认 0 时行为与 Task 11 现有调用完全一致(不回归)
注:结果驱动视频级切分不复用 PoolConfig——它有独立的 VideoSplitConfig /
SplitBuildConfig / SelectConfig,故 PoolConfig 不承载视频级切分旋钮(无死配置面)。
"""
from __future__ import annotations
@@ -13,7 +15,7 @@ import pytest
from app.harness.pools import InsufficientValSignal, split_by_video_assignment
from app.harness.split_selection import diag_fingerprint
from core.types import GeneratedQuestion, PoolConfig
from core.types import GeneratedQuestion
def _q(qid: str, vid: str, tt: str = "Counting Problem") -> GeneratedQuestion:
@@ -64,52 +66,3 @@ def test_val_wrong_min_default_zero_no_regression():
qs, assignment, correctness=correctness, val_ratio=1.0, seed=0
)
assert len(pools.validation) == 2 # 未抛异常,正常返回
def test_pool_config_accepts_video_split_knobs():
"""PoolConfig 能接收视频级切分五个新旋钮字段(默认惰性,不破坏现有构造点)。"""
cfg = PoolConfig(
task_types=None,
seed=0,
baseline_run_id="infer_adhoc",
diag_size=200,
diag_correct_ratio=0.5,
val_size=30,
val_correct_ratio=0.5,
test_size=60,
eval_min_per_class=2,
train_ratio=0.667,
test_questions_dir=None,
n_trainval=100,
floor_k={"Counting Problem": 3},
epsilon=0.1,
report_floor=27,
val_wrong_min=20,
)
assert cfg.n_trainval == 100
assert cfg.floor_k == {"Counting Problem": 3}
assert cfg.epsilon == 0.1
assert cfg.report_floor == 27
assert cfg.val_wrong_min == 20
def test_pool_config_video_split_knobs_default_inert():
"""未传视频级切分字段时默认惰性(0 / 空 dict),不破坏 GlobalPoolStrategy 现有构造。"""
cfg = PoolConfig(
task_types=None,
seed=0,
baseline_run_id="run_1",
diag_size=200,
diag_correct_ratio=0.5,
val_size=30,
val_correct_ratio=0.5,
test_size=60,
eval_min_per_class=2,
train_ratio=0.667,
test_questions_dir=None,
)
assert cfg.n_trainval == 0
assert cfg.floor_k == {}
assert cfg.epsilon == 0.0
assert cfg.report_floor == 0
assert cfg.val_wrong_min == 0