fix: address whole-impl review (INFRA T0 rows, reproducible manifest, evolution_target report, dead config, canonical DRY)
C-1: persist_infra_t0_rows 补 INFRA/空预测错题的 T0 信号行(不进诊断故须单独落库),run_pipeline 加 Phase 0,dry-run 用假数据走通。 C-2: CLI 加 --generated-at,真实运行默认盖真实 UTC now,可显式固定以字节级复现 manifest。 I-1: coverage_report 增 evolution_target_distribution(T2 信号按 tool/skill/system 计数)。 I-2: 删除 PoolConfig 死字段 n_trainval/floor_k/epsilon/report_floor/val_wrong_min(grep 确认无消费者,视频级切分用独立 VideoSplitConfig/SplitBuildConfig/SelectConfig)。 I-3: 抽共享 load_canonical_predictions(db_path, run_id),CLI 与 build_split 共用;消除 canonical 取行 + correct 判定重复。 M-1: build_split docstring 注明 val_wrong_min-agnostic 契约(McNemar 护栏由 CLI 冻结后执行,Task 11 契约)。
This commit is contained in:
+64
-16
@@ -119,6 +119,11 @@ def build_split(
|
||||
test → 加载题库并以视频归属切三池 → 原子冻结 pools.json → 写溯源 manifest →
|
||||
六条防御断言 fail-fast 校验。
|
||||
|
||||
契约(Task 11,非疏漏):build_split 有意保持 val_wrong_min-agnostic——内部调
|
||||
split_by_video_assignment 时不传 val_wrong_min(默认 0,不校验 validation 错题
|
||||
数)。McNemar 功效护栏是切分**冻结后**的独立校验,由 CLI 的 check_mcnemar_power
|
||||
在 build_split 返回后执行;切分构造本身不因功效阈失败,二者关注点分离。
|
||||
|
||||
参数:
|
||||
db_path: harness.db 路径(只读读取 predictions,不改动)。
|
||||
baseline_run_id: 基线 run 标识(如 "infer_adhoc")。
|
||||
@@ -138,7 +143,7 @@ def build_split(
|
||||
ValueError: 上游依赖校验失败(如 correctness 缺题、assignment 非法)。
|
||||
"""
|
||||
# Phase 1: canonical 基线预测 + 诊断信号。
|
||||
preds = _read_canonical_predictions(db_path, baseline_run_id)
|
||||
preds = load_canonical_predictions(db_path, baseline_run_id)
|
||||
signal_rows_raw = signal_store.load(baseline_run_id, diag_fingerprint)
|
||||
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint)
|
||||
signal_rows = [
|
||||
@@ -226,31 +231,38 @@ def _normalize_choice(choice: str | None) -> str:
|
||||
return (choice or "").strip().upper()[:1]
|
||||
|
||||
|
||||
def _read_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
|
||||
def load_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
|
||||
"""从 harness.db 只读取指定 run 每题首行(ORDER BY rowid)为 canonical 预测。
|
||||
|
||||
共享口径 helper:CLI(可诊断错题筛选 + INFRA T0 补录)与 build_split(切分)
|
||||
共用同一"每 qid 取 rowid 最小首行 + 归一化 correct 判定"口径,消除两处重复实现。
|
||||
同一 question_id 可能有多行(重跑 / 补测),canonical 口径取 rowid 最小的首行,
|
||||
保证 distinct question 计数与对错判定确定。correct = 预测与答案归一后逐字符相等。
|
||||
保证 distinct question 计数与对错判定确定。correct = 预测与答案各自归一
|
||||
(strip → 大写 → 取首字母)后逐字符相等。
|
||||
|
||||
口径边界:旧 build_or_load_pools 的 legacy 池构建路径(app/harness/pools.py)是
|
||||
另一条独立既有链路,不共用本 helper,两者刻意不统一(本次不动 legacy 路径)。
|
||||
|
||||
参数:
|
||||
db_path: harness.db 路径。
|
||||
db_path: harness.db 路径(URI mode=ro 只读打开,绝不改动基线 db)。
|
||||
baseline_run_id: 基线 run 标识。
|
||||
|
||||
返回:
|
||||
canonical 预测行列表,每行 {video_id, question_id, task_type, correct}。
|
||||
canonical 预测行列表,每行含 question_id / video_id / task_type /
|
||||
prediction / answer / stop_reason / correct(bool)。按 rowid 升序去重,
|
||||
每 qid 保留首行。
|
||||
|
||||
异常:
|
||||
ValueError: 该 run 无任何预测行(fail-fast,不返回空切分)。
|
||||
|
||||
实现细节:
|
||||
以 URI mode=ro 打开只读连接,绝不改动基线 db;按 rowid 升序遍历,
|
||||
首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
|
||||
按 rowid 升序遍历,首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
|
||||
"""
|
||||
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT question_id, video_id, task_type, prediction, answer "
|
||||
"SELECT question_id, video_id, task_type, prediction, answer, stop_reason "
|
||||
"FROM predictions WHERE run_id = ? ORDER BY rowid",
|
||||
(baseline_run_id,),
|
||||
).fetchall()
|
||||
@@ -266,6 +278,9 @@ def _read_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dic
|
||||
"question_id": qid,
|
||||
"video_id": row["video_id"],
|
||||
"task_type": row["task_type"],
|
||||
"prediction": row["prediction"],
|
||||
"answer": row["answer"],
|
||||
"stop_reason": row["stop_reason"],
|
||||
"correct": _normalize_choice(row["prediction"]) == _normalize_choice(row["answer"]),
|
||||
}
|
||||
if not canonical:
|
||||
@@ -367,7 +382,13 @@ def _build_coverage_report(
|
||||
|
||||
返回:
|
||||
覆盖报告字典,含 cells_covered / grid_total / floor_satisfied /
|
||||
test_representativeness_deviation / tier_distribution。
|
||||
test_representativeness_deviation / tier_distribution /
|
||||
evolution_target_distribution。
|
||||
|
||||
实现细节:
|
||||
evolution_target_distribution 只统计 T2 信号(可训练缺陷),按
|
||||
tool / skill / system 计数,报告"哪层参数组拿到梯度";T0/T1/uncertain 行
|
||||
evolution_target 恒为 None,不入该分布。
|
||||
"""
|
||||
by_id = {v.video_id: v for v in videos}
|
||||
trainval = [by_id[vid] for vid in assignment_obj.trainval]
|
||||
@@ -395,13 +416,7 @@ def _build_coverage_report(
|
||||
diff_buckets,
|
||||
)
|
||||
|
||||
tier_counts = Counter(row.tier for row in signal_rows_raw)
|
||||
total_signals = sum(tier_counts.values())
|
||||
tier_distribution = (
|
||||
{tier: count / total_signals for tier, count in tier_counts.items()}
|
||||
if total_signals
|
||||
else {}
|
||||
)
|
||||
tier_distribution, evolution_target_distribution = _signal_distributions(signal_rows_raw)
|
||||
|
||||
return {
|
||||
"cells_covered": len(covered_cells),
|
||||
@@ -413,9 +428,42 @@ def _build_coverage_report(
|
||||
"epsilon": config.epsilon,
|
||||
},
|
||||
"tier_distribution": tier_distribution,
|
||||
"evolution_target_distribution": evolution_target_distribution,
|
||||
}
|
||||
|
||||
|
||||
def _signal_distributions(
|
||||
signal_rows_raw: list[DiagnosisSignalRow],
|
||||
) -> tuple[dict[str, float], dict[str, int]]:
|
||||
"""由诊断信号行算 tier 占比分布与 T2 进化目标计数分布。
|
||||
|
||||
参数:
|
||||
signal_rows_raw: 诊断信号行。
|
||||
|
||||
返回:
|
||||
(tier_distribution, evolution_target_distribution) 二元组:
|
||||
- tier_distribution: {tier: 占比},无信号时为空 dict;
|
||||
- evolution_target_distribution: 仅统计 T2(可训练缺陷)信号,按
|
||||
tool / skill / system 计数,报告哪层参数组拿到梯度;T0/T1/uncertain 行
|
||||
evolution_target 恒为 None,不入该分布。
|
||||
"""
|
||||
tier_counts = Counter(row.tier for row in signal_rows_raw)
|
||||
total_signals = sum(tier_counts.values())
|
||||
tier_distribution = (
|
||||
{tier: count / total_signals for tier, count in tier_counts.items()}
|
||||
if total_signals
|
||||
else {}
|
||||
)
|
||||
evolution_target_distribution = dict(
|
||||
Counter(
|
||||
row.evolution_target
|
||||
for row in signal_rows_raw
|
||||
if row.tier == "T2" and row.evolution_target is not None
|
||||
)
|
||||
)
|
||||
return tier_distribution, evolution_target_distribution
|
||||
|
||||
|
||||
def _assert_split_invariants(
|
||||
*,
|
||||
pools: Pools,
|
||||
|
||||
Reference in New Issue
Block a user