fix: address whole-impl review (INFRA T0 rows, reproducible manifest, evolution_target report, dead config, canonical DRY)

C-1: persist_infra_t0_rows 补 INFRA/空预测错题的 T0 信号行(不进诊断故须单独落库),run_pipeline 加 Phase 0,dry-run 用假数据走通。
C-2: CLI 加 --generated-at,真实运行默认盖真实 UTC now,可显式固定以字节级复现 manifest。
I-1: coverage_report 增 evolution_target_distribution(T2 信号按 tool/skill/system 计数)。
I-2: 删除 PoolConfig 死字段 n_trainval/floor_k/epsilon/report_floor/val_wrong_min(grep 确认无消费者,视频级切分用独立 VideoSplitConfig/SplitBuildConfig/SelectConfig)。
I-3: 抽共享 load_canonical_predictions(db_path, run_id),CLI 与 build_split 共用;消除 canonical 取行 + correct 判定重复。
M-1: build_split docstring 注明 val_wrong_min-agnostic 契约(McNemar 护栏由 CLI 冻结后执行,Task 11 契约)。
This commit is contained in:
2026-07-15 13:39:14 -04:00
parent 02b8145b7f
commit 8fef7ced42
6 changed files with 302 additions and 134 deletions
+64 -16
View File
@@ -119,6 +119,11 @@ def build_split(
test → 加载题库并以视频归属切三池 → 原子冻结 pools.json → 写溯源 manifest →
六条防御断言 fail-fast 校验。
契约(Task 11,非疏漏):build_split 有意保持 val_wrong_min-agnostic——内部调
split_by_video_assignment 时不传 val_wrong_min(默认 0,不校验 validation 错题
数)。McNemar 功效护栏是切分**冻结后**的独立校验,由 CLI 的 check_mcnemar_power
在 build_split 返回后执行;切分构造本身不因功效阈失败,二者关注点分离。
参数:
db_path: harness.db 路径(只读读取 predictions,不改动)。
baseline_run_id: 基线 run 标识(如 "infer_adhoc")。
@@ -138,7 +143,7 @@ def build_split(
ValueError: 上游依赖校验失败(如 correctness 缺题、assignment 非法)。
"""
# Phase 1: canonical 基线预测 + 诊断信号。
preds = _read_canonical_predictions(db_path, baseline_run_id)
preds = load_canonical_predictions(db_path, baseline_run_id)
signal_rows_raw = signal_store.load(baseline_run_id, diag_fingerprint)
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint)
signal_rows = [
@@ -226,31 +231,38 @@ def _normalize_choice(choice: str | None) -> str:
return (choice or "").strip().upper()[:1]
def _read_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
def load_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
"""从 harness.db 只读取指定 run 每题首行(ORDER BY rowid)为 canonical 预测。
共享口径 helperCLI(可诊断错题筛选 + INFRA T0 补录)与 build_split(切分)
共用同一"每 qid 取 rowid 最小首行 + 归一化 correct 判定"口径,消除两处重复实现。
同一 question_id 可能有多行(重跑 / 补测),canonical 口径取 rowid 最小的首行,
保证 distinct question 计数与对错判定确定。correct = 预测与答案归一后逐字符相等。
保证 distinct question 计数与对错判定确定。correct = 预测与答案各自归一
(strip → 大写 → 取首字母)后逐字符相等。
口径边界:旧 build_or_load_pools 的 legacy 池构建路径(app/harness/pools.py)是
另一条独立既有链路,不共用本 helper,两者刻意不统一(本次不动 legacy 路径)。
参数:
db_path: harness.db 路径。
db_path: harness.db 路径URI mode=ro 只读打开,绝不改动基线 db)
baseline_run_id: 基线 run 标识。
返回:
canonical 预测行列表,每行 {video_id, question_id, task_type, correct}。
canonical 预测行列表,每行含 question_id / video_id / task_type /
prediction / answer / stop_reason / correctbool)。按 rowid 升序去重,
每 qid 保留首行。
异常:
ValueError: 该 run 无任何预测行(fail-fast,不返回空切分)。
实现细节:
以 URI mode=ro 打开只读连接,绝不改动基线 db;按 rowid 升序遍历,
首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
按 rowid 升序遍历,首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
"""
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn.row_factory = sqlite3.Row
try:
rows = conn.execute(
"SELECT question_id, video_id, task_type, prediction, answer "
"SELECT question_id, video_id, task_type, prediction, answer, stop_reason "
"FROM predictions WHERE run_id = ? ORDER BY rowid",
(baseline_run_id,),
).fetchall()
@@ -266,6 +278,9 @@ def _read_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dic
"question_id": qid,
"video_id": row["video_id"],
"task_type": row["task_type"],
"prediction": row["prediction"],
"answer": row["answer"],
"stop_reason": row["stop_reason"],
"correct": _normalize_choice(row["prediction"]) == _normalize_choice(row["answer"]),
}
if not canonical:
@@ -367,7 +382,13 @@ def _build_coverage_report(
返回:
覆盖报告字典,含 cells_covered / grid_total / floor_satisfied /
test_representativeness_deviation / tier_distribution
test_representativeness_deviation / tier_distribution /
evolution_target_distribution。
实现细节:
evolution_target_distribution 只统计 T2 信号(可训练缺陷),按
tool / skill / system 计数,报告"哪层参数组拿到梯度"T0/T1/uncertain 行
evolution_target 恒为 None,不入该分布。
"""
by_id = {v.video_id: v for v in videos}
trainval = [by_id[vid] for vid in assignment_obj.trainval]
@@ -395,13 +416,7 @@ def _build_coverage_report(
diff_buckets,
)
tier_counts = Counter(row.tier for row in signal_rows_raw)
total_signals = sum(tier_counts.values())
tier_distribution = (
{tier: count / total_signals for tier, count in tier_counts.items()}
if total_signals
else {}
)
tier_distribution, evolution_target_distribution = _signal_distributions(signal_rows_raw)
return {
"cells_covered": len(covered_cells),
@@ -413,9 +428,42 @@ def _build_coverage_report(
"epsilon": config.epsilon,
},
"tier_distribution": tier_distribution,
"evolution_target_distribution": evolution_target_distribution,
}
def _signal_distributions(
signal_rows_raw: list[DiagnosisSignalRow],
) -> tuple[dict[str, float], dict[str, int]]:
"""由诊断信号行算 tier 占比分布与 T2 进化目标计数分布。
参数:
signal_rows_raw: 诊断信号行。
返回:
(tier_distribution, evolution_target_distribution) 二元组:
- tier_distribution: {tier: 占比},无信号时为空 dict;
- evolution_target_distribution: 仅统计 T2(可训练缺陷)信号,按
tool / skill / system 计数,报告哪层参数组拿到梯度;T0/T1/uncertain 行
evolution_target 恒为 None,不入该分布。
"""
tier_counts = Counter(row.tier for row in signal_rows_raw)
total_signals = sum(tier_counts.values())
tier_distribution = (
{tier: count / total_signals for tier, count in tier_counts.items()}
if total_signals
else {}
)
evolution_target_distribution = dict(
Counter(
row.evolution_target
for row in signal_rows_raw
if row.tier == "T2" and row.evolution_target is not None
)
)
return tier_distribution, evolution_target_distribution
def _assert_split_invariants(
*,
pools: Pools,