695 lines
37 KiB
Markdown
695 lines
37 KiB
Markdown
# 结果驱动的视频级切分 Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 用离线、结果驱动的视频级 train/val/test 切分（基于已有 `infer_adhoc` baseline + 240 错题诊断），产出冻结 `pools.json`，替代自造训练题。
|
||
|
||
**Architecture:** 复用现有 `run_diagnosis` 拿逐题 `error_attributions`；新增纯函数 `split_selection`（signal 分层 → 视频聚合 → 贪心联合约束选择）；改 `pools.py` 切分原子 unit→video 并补原子写；全程离线可复现（fixed seed + baseline_run_id + diag_fingerprint 溯源）。
|
||
|
||
**Tech Stack:** Python 3.11、pytest、SQLite（harness.db）、asyncio、loguru、pydantic-settings、YAML（科研配置）。
|
||
|
||
**设计源**：`research-wiki/designs/2026-07-15-results-driven-video-split-design.md`
|
||
|
||
---
|
||
|
||
## 关键代码锚点（实现前必读）
|
||
|
||
| 用途 | 位置 |
|
||
|------|------|
|
||
| 诊断入口（复用） | `core/evolution/diagnose.py:2017` `run_diagnosis(...) -> DiagnosisResult` |
|
||
| 逐题归因产物 | `DiagnosisResult.error_attributions: list[ErrorAttribution]`（`core/evolution/types.py:257,149`） |
|
||
| INFRA 排除集 | `core/evolution/diagnose.py:56` `_INFRA_STOP_REASONS={"error","parse_error"}` |
|
||
| error_type 瀑布 | `core/evolution/diagnose.py:910-941`（extraction/search/reasoning/mixed） |
|
||
| RunLog 读端口 | `core/evolution/protocols.py:68`（`get_predictions`/`get_traces`） |
|
||
| 三池结构 | `app/harness/pools.py:33` `Pools`；`build_pools:53`；`_split_one_category:809`（签名行，822 是 docstring） |
|
||
| 切分入口/冻结 | `app/harness/pools.py:473` `build_or_load_pools`；`save_pools:289`（`path.write_text:341` 非原子）；per_category 写(566) 非原子 |
|
||
| PoolConfig | `core/types.py:146`（seed/diag_size/val_size/test_size/train_ratio） |
|
||
| baseline 数据 | `workspaces/default/harness.db`，`predictions` run_id=`infer_adhoc`；`traces` 表**空**，轨迹在 `predictions.steps_json` |
|
||
|
||
### Canonical baseline 事实（Codex 计划审 C1/C2 核实，实现须严格对齐）
|
||
|
||
- **902 原始行 / 900 distinct question / 300 视频**；`743-1` 有 **3 行**（error/budget_exceeded/finished）→ **canonical 取行策略 = 每 question_id 取第一行**（`ORDER BY rowid`），全流程统一。
|
||
- **660 对 / 236 可诊断错题（非空 pred，非 INFRA）/ 4 空 pred（全 INFRA：2 error + 2 parse_error）** → 非对合计 240。
|
||
- **`only_incorrect=True` + INFRA 排除**后进诊断的是 **236 题**；4 个 INFRA 空 pred 直接落 **T0**（不进 judge）。
|
||
- **`steps_json` 步字段 = `{thought, tool_call:{tool, args}, tool_output}`**——工具名在 **`tool_call.tool`**（非 `name`；对齐 `core/evolution/diagnose.py:258` 的 `tool_call.get("tool")`）。899/902 行非空，空的走空返回。
|
||
- **防御断言按 distinct question（900）计，不按裸行（902）**。
|
||
|
||
## 核心算法保真校验
|
||
|
||
本计划**不迁移/不改**核心算法（`ARCHITECTURE.md §6`）。唯一相邻项是第 5 项"信息阶梯冷启动 2:1"（`gate_ladder.py`）：本计划只把 **pools 切分边界**改为 video，**pools 内部仍是 unit 粒度**，`gate_ladder` 的输入（unit + correctness）不变。**保真检查点（Task 9 Step 4）**：确认 video 切分后 `Pools.diagnosis/validation` 内仍是逐 unit 列表，`gate_ladder.build_cold_entries` 消费不变、冷启动 2:1 未受影响。
|
||
|
||
---
|
||
|
||
## Phase 0：schema 前置门（编码前必须完成）
|
||
|
||
### Task 0: 用 structured-logging 定稿并注册 schema
|
||
|
||
**Files:** 无代码；产出 `research-wiki/schemas/` 注册 + 本计划 §schema 表定稿。
|
||
|
||
- [ ] **Step 1: 调用 structured-logging skill**，为下述两张表定稿列/类型/索引/基线指标并注册 Wiki。设计已给骨架，structured-logging 负责批准与注册：
|
||
|
||
`baseline_diagnosis`（逐题诊断信号，harness.db）：
|
||
|
||
| 列 | 类型 | 说明 |
|
||
|----|------|------|
|
||
| question_id | TEXT | 题 ID |
|
||
| video_id | TEXT | 视频 ID |
|
||
| baseline_run_id | TEXT | 溯源 run（`infer_adhoc`） |
|
||
| diag_fingerprint | TEXT | `hash(诊断prompt版本+model+诊断代码版本)` |
|
||
| task_type | TEXT | 12 类之一 |
|
||
| error_type | TEXT **NULL** | extraction/search/reasoning/mixed；**T0/uncertain 行为 NULL**（Codex I4：INFRA/degraded 题无合法 error_type） |
|
||
| cause_category | TEXT NULL | defect/lapse/NULL |
|
||
| tier | TEXT | T0/T1/T2/uncertain |
|
||
| evolution_target | TEXT **NULL** | 由 error_type 派生（tool/skill/system）；error_type 为 NULL 时亦 NULL |
|
||
| degraded | INTEGER | judge 降级 0/1 |
|
||
| infra | INTEGER | INFRA 排除 0/1 |
|
||
| session_id | TEXT | 遥测关联 |
|
||
| **PK** | | (question_id, baseline_run_id, diag_fingerprint) |
|
||
|
||
> **T0/uncertain 写规则**：INFRA 题 `infra=1, tier=T0, error_type/cause_category/evolution_target=NULL`；degraded/诊断失败题 `degraded=1, tier=uncertain, error_type` 可为 NULL。二者均不进 T2、不入 48 格覆盖。
|
||
|
||
`split_manifest`（冻结产物溯源，随 pools.json 同目录 JSON）：
|
||
|
||
| 键 | 说明 |
|
||
|----|------|
|
||
| baseline_run_id / diag_fingerprint / seed | 复现三元组 |
|
||
| config | N_trainval/floor_K/ε/report_floor/val 尺寸 快照 |
|
||
| pools_sha256 | pools.json 内容 hash |
|
||
| coverage_report | 48 格覆盖数、floor 达标情况、test 代表性偏差 |
|
||
|
||
- [ ] **Step 2:** 确认 structured-logging 已在 `research-wiki/schemas/` 注册二表并给出基线指标（各 tier 占比、48 格覆盖率、floor 达标率）。**此门通过前不得进入 Phase 1。**
|
||
|
||
---
|
||
|
||
## Phase 1：离线诊断管线
|
||
|
||
### Task 1: steps_json → trace 行适配器
|
||
|
||
**Files:**
|
||
- Create: `app/harness/steps_json_traces.py`
|
||
- Test: `tests/unit/test_steps_json_traces.py`
|
||
|
||
`run_diagnosis` 经 `get_traces` 取轨迹，但 `infer_adhoc` 的 `traces` 表空、轨迹在 `predictions.steps_json`。此适配器把 `{thought, tool_call, tool_output}` 步转成 `get_traces` 的行形 `{video_id, question_id, step, tool_name, tool_args, tool_output, thought}`。
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_steps_json_traces.py
|
||
import json
|
||
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||
|
||
|
||
def test_parses_tool_call_into_name_and_args():
|
||
# 真实 infer_adhoc steps_json 形态：tool_call={"tool":..., "args":...}（非 "name"）
|
||
steps = [
|
||
{"thought": "看根节点", "tool_call": {"tool": "view_node", "args": {"node_id": "v_L1_000"}}, "tool_output": "o0"},
|
||
{"thought": "搜索", "tool_call": {"tool": "search_similar", "args": {"query": "gadget"}}, "tool_output": "hit"},
|
||
]
|
||
rows = steps_json_to_trace_rows("vid1", "q1", json.dumps(steps))
|
||
assert [r["step"] for r in rows] == [0, 1]
|
||
assert rows[0]["tool_name"] == "view_node"
|
||
assert rows[0]["tool_args"] == {"node_id": "v_L1_000"}
|
||
assert rows[0]["video_id"] == "vid1" and rows[0]["question_id"] == "q1"
|
||
assert rows[1]["tool_output"] == "hit"
|
||
|
||
|
||
def test_empty_or_blank_steps_json_returns_empty():
|
||
assert steps_json_to_trace_rows("v", "q", "") == []
|
||
assert steps_json_to_trace_rows("v", "q", "[]") == []
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败** — `conda activate Video-Tree-TRM && pytest tests/unit/test_steps_json_traces.py -v`，预期 `ModuleNotFoundError`。
|
||
|
||
- [ ] **Step 3: 最小实现**
|
||
|
||
```python
|
||
# app/harness/steps_json_traces.py
|
||
"""把 predictions.steps_json 转成 RunLog.get_traces 的行形。
|
||
|
||
infer_adhoc 的 traces 表为空，轨迹存于 steps_json（{thought, tool_call, tool_output}）。
|
||
诊断管线经 get_traces 消费轨迹，故需此确定性转换适配。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
from typing import Any
|
||
|
||
|
||
def steps_json_to_trace_rows(video_id: str, question_id: str, steps_json: str) -> list[dict[str, Any]]:
|
||
"""将单题 steps_json 解析为 trace 行列表（step 从 0 递增）。
|
||
|
||
参数:
|
||
video_id: 视频 ID。
|
||
question_id: 题 ID。
|
||
steps_json: predictions.steps_json 原文（JSON 数组字符串）。
|
||
|
||
返回:
|
||
行字典列表，字段对齐 traces 表 schema；空/空数组返回 []。
|
||
"""
|
||
if not steps_json or not steps_json.strip():
|
||
return []
|
||
steps = json.loads(steps_json)
|
||
if not isinstance(steps, list):
|
||
raise ValueError(f"steps_json 非数组: {question_id}")
|
||
rows: list[dict[str, Any]] = []
|
||
for i, s in enumerate(steps):
|
||
call = s.get("tool_call") or {}
|
||
rows.append(
|
||
{
|
||
"video_id": video_id,
|
||
"question_id": question_id,
|
||
"step": i,
|
||
# infer_adhoc 用 "tool"；back-compat 兼容极少数 "name"
|
||
"tool_name": call.get("tool", call.get("name")),
|
||
"tool_args": call.get("args", {}),
|
||
"tool_output": s.get("tool_output"),
|
||
"thought": s.get("thought"),
|
||
}
|
||
)
|
||
return rows
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_steps_json_traces.py -v`，预期 PASS。
|
||
- [ ] **Step 5: 提交** — `git add app/harness/steps_json_traces.py tests/unit/test_steps_json_traces.py && git commit -m "feat: add steps_json to trace-row adapter"`
|
||
|
||
### Task 2: RunLog 包装器（预测直取 + 轨迹从 steps_json）
|
||
|
||
**Files:**
|
||
- Create: `app/harness/baseline_run_log.py`
|
||
- Test: `tests/unit/test_baseline_run_log.py`
|
||
|
||
包装现有 SQLite RunLog：`get_predictions` 透传；`get_traces` 当底层 traces 空时，从 predictions.steps_json 经 Task 1 生成。不写 SQL 于 app 之外的裸路径——复用底层适配器。
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_baseline_run_log.py
|
||
import json
|
||
import pytest
|
||
from app.harness.baseline_run_log import StepsJsonRunLog
|
||
|
||
|
||
class _FakeInner:
|
||
def __init__(self, preds, traces):
|
||
self._preds, self._traces = preds, traces
|
||
async def get_predictions(self, run_id, *, question_ids=None):
|
||
return [p for p in self._preds if not question_ids or p["question_id"] in question_ids]
|
||
async def get_traces(self, run_id, *, question_ids=None):
|
||
return list(self._traces)
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_get_traces_falls_back_to_steps_json_when_table_empty():
|
||
steps = [{"thought": "t", "tool_call": {"tool": "view_node", "args": {}}, "tool_output": "o"}]
|
||
preds = [{"video_id": "v1", "question_id": "q1", "steps_json": json.dumps(steps)}]
|
||
log = StepsJsonRunLog(_FakeInner(preds, traces=[]))
|
||
rows = await log.get_traces("r", question_ids=["q1"])
|
||
assert rows[0]["tool_name"] == "view_node" and rows[0]["question_id"] == "q1"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_get_traces_prefers_nonempty_inner_table():
|
||
inner_traces = [{"video_id": "v1", "question_id": "q1", "step": 0, "tool_name": "x"}]
|
||
log = StepsJsonRunLog(_FakeInner([], inner_traces))
|
||
rows = await log.get_traces("r")
|
||
assert rows == inner_traces
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败** — `pytest tests/unit/test_baseline_run_log.py -v`，预期 `ModuleNotFoundError`。
|
||
|
||
- [ ] **Step 3: 最小实现**
|
||
|
||
```python
|
||
# app/harness/baseline_run_log.py
|
||
"""RunLog 包装器：traces 表空时从 predictions.steps_json 重建轨迹。
|
||
|
||
用于对 infer_adhoc 这类 traces 未落表、轨迹在 steps_json 的历史 run 跑离线诊断。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
from typing import Any
|
||
|
||
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||
|
||
|
||
class StepsJsonRunLog:
|
||
"""委托内层 RunLog；get_traces 空表时回退 steps_json。"""
|
||
|
||
def __init__(self, inner: Any) -> None:
|
||
self._inner = inner
|
||
|
||
async def get_predictions(self, run_id: str, *, question_ids: list[str] | None = None) -> list[dict[str, Any]]:
|
||
return await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||
|
||
async def get_traces(self, run_id: str, *, question_ids: list[str] | None = None) -> list[dict[str, Any]]:
|
||
inner_rows = await self._inner.get_traces(run_id, question_ids=question_ids)
|
||
if inner_rows:
|
||
return inner_rows
|
||
preds = await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||
rows: list[dict[str, Any]] = []
|
||
for p in preds:
|
||
rows.extend(steps_json_to_trace_rows(p["video_id"], p["question_id"], p.get("steps_json") or ""))
|
||
return rows
|
||
```
|
||
|
||
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_baseline_run_log.py -v`，预期 PASS。
|
||
- [ ] **Step 5: 提交** — `git add app/harness/baseline_run_log.py tests/unit/test_baseline_run_log.py && git commit -m "feat: add steps_json-backed RunLog wrapper"`
|
||
|
||
### Task 3: baseline_diagnosis 持久化端口 + SQLite 适配
|
||
|
||
**Files:**
|
||
- Modify: `core/evolution/protocols.py`（新增 `DiagnosisSignalStore` Protocol）
|
||
- Create: `adapters/baseline_diagnosis_store.py`（SQLite 实现）
|
||
- Test: `tests/unit/test_baseline_diagnosis_store.py`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_baseline_diagnosis_store.py
|
||
import sqlite3
|
||
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore, DiagnosisSignalRow
|
||
|
||
|
||
def _store(tmp_path):
|
||
return SqliteDiagnosisSignalStore(str(tmp_path / "h.db"))
|
||
|
||
|
||
def test_upsert_and_get_done_ids(tmp_path):
|
||
s = _store(tmp_path)
|
||
row = DiagnosisSignalRow(question_id="q1", video_id="v1", baseline_run_id="r", diag_fingerprint="fp",
|
||
task_type="Counting", error_type="search_failure", cause_category="defect",
|
||
tier="T2", evolution_target="skill", degraded=False, infra=False, session_id="s")
|
||
s.upsert(row)
|
||
assert s.done_question_ids("r", "fp") == {"q1"}
|
||
# upsert 同键覆盖，不重复
|
||
s.upsert(row)
|
||
assert s.done_question_ids("r", "fp") == {"q1"}
|
||
# 不同 fingerprint 隔离
|
||
assert s.done_question_ids("r", "other_fp") == set()
|
||
|
||
|
||
def test_load_rows_roundtrip(tmp_path):
|
||
s = _store(tmp_path)
|
||
s.upsert(DiagnosisSignalRow("q2", "v2", "r", "fp", "OCR Problems", "extraction_failure",
|
||
"lapse", "T1", "tool", False, False, "s"))
|
||
rows = s.load("r", "fp")
|
||
assert len(rows) == 1 and rows[0].tier == "T1" and rows[0].evolution_target == "tool"
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败** — `pytest tests/unit/test_baseline_diagnosis_store.py -v`，预期 `ModuleNotFoundError`。
|
||
|
||
- [ ] **Step 3: 实现 Protocol + 适配**
|
||
|
||
在 `core/evolution/protocols.py` 追加：
|
||
|
||
```python
|
||
class DiagnosisSignalStore(Protocol):
|
||
"""逐题诊断信号存储端口（隔离 SQLite，app/core 不写 SQL）。"""
|
||
|
||
def upsert(self, row: "DiagnosisSignalRow") -> None: ...
|
||
def done_question_ids(self, baseline_run_id: str, diag_fingerprint: str) -> set[str]: ...
|
||
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list["DiagnosisSignalRow"]: ...
|
||
```
|
||
|
||
创建 `adapters/baseline_diagnosis_store.py`：`DiagnosisSignalRow` dataclass（字段对齐 Task 0 表）；`SqliteDiagnosisSignalStore` 建表（PK `(question_id, baseline_run_id, diag_fingerprint)`）、`INSERT OR REPLACE` upsert、按 (run,fp) 查已完成集与全量。所有写在单事务提交（原子）。
|
||
|
||
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_baseline_diagnosis_store.py -v`，预期 PASS。
|
||
- [ ] **Step 5: 提交** — `git add core/evolution/protocols.py adapters/baseline_diagnosis_store.py tests/unit/test_baseline_diagnosis_store.py && git commit -m "feat: add baseline diagnosis signal store"`
|
||
|
||
### Task 4: 离线诊断编排
|
||
|
||
**Files:**
|
||
- Create: `app/harness/baseline_diagnosis.py`
|
||
- Test: `tests/integration/test_baseline_diagnosis.py`（LLM 类 → 产出 MD）
|
||
|
||
编排：算 `diag_fingerprint` → 查已完成集（续跑）→ 对剩余错题调 `run_diagnosis`（经 `StepsJsonRunLog`）→ 投影 `error_attributions` + INFRA + degraded 为 `DiagnosisSignalRow`（tier 由 Task 6 `score_signal`，此处先落 error_type/cause_category，tier 计算在 Task 6 引入后接入）→ upsert。
|
||
|
||
- [ ] **Step 1: 写失败集成测试**（mock LLM，真实 infer_adhoc 抽 3 题）
|
||
|
||
```python
|
||
# tests/integration/test_baseline_diagnosis.py
|
||
import pytest
|
||
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||
from app.harness.baseline_diagnosis import run_baseline_diagnosis, DiagnosisDeps
|
||
|
||
|
||
def _deps(monkeypatch, calls):
|
||
async def fake_run_diagnosis(run_id, questions, tree_data, llm, run_log, skill_store, prompts,
|
||
*, concurrency, question_ids=None, task_types=None, only_incorrect=False):
|
||
calls.append(tuple(question_ids or []))
|
||
from core.evolution.types import DiagnosisResult, ErrorAttribution
|
||
return DiagnosisResult(run_id=run_id, error_attributions=[
|
||
ErrorAttribution("q1", "search_failure", None, "defect"),
|
||
ErrorAttribution("q2", "mixed", None, "lapse"),
|
||
], infra_question_ids=[], degraded_question_ids=[])
|
||
monkeypatch.setattr("app.harness.baseline_diagnosis.run_diagnosis", fake_run_diagnosis)
|
||
# 全部依赖显式 fake，无占位
|
||
return DiagnosisDeps(run_log=_FakeRunLog(), llm=_FakeLLM(), skill_store=_FakeSkillStore(),
|
||
prompts=_fake_prompts(), tree_data={}, concurrency=2)
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_resume_skips_done(tmp_path, monkeypatch):
|
||
calls = []
|
||
deps = _deps(monkeypatch, calls)
|
||
store = SqliteDiagnosisSignalStore(str(tmp_path / "h.db"))
|
||
q_by_id = {"q1": _mk_q("q1"), "q2": _mk_q("q2")}
|
||
await run_baseline_diagnosis(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||
wrong_ids=["q1", "q2"], questions=q_by_id, store=store, deps=deps)
|
||
assert store.done_question_ids("infer_adhoc", "fp") == {"q1", "q2"}
|
||
await run_baseline_diagnosis(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||
wrong_ids=["q1", "q2"], questions=q_by_id, store=store, deps=deps)
|
||
assert calls[-1] == () # 第二次无剩余
|
||
```
|
||
|
||
- [ ] **Step 2: 运行确认失败**，预期 `ModuleNotFoundError`。
|
||
- [ ] **Step 3: 实现** `run_baseline_diagnosis(baseline_run_id, diag_fingerprint, wrong_ids, questions, store, deps)`：`remaining = wrong_ids - store.done_question_ids(run,fp)`；空则返回；否则 `await run_diagnosis(baseline_run_id, [questions[i] for i in remaining], deps.tree_data, deps.llm, StepsJsonRunLog(deps.run_log), deps.skill_store, deps.prompts, concurrency=deps.concurrency, question_ids=list(remaining), only_incorrect=True)`；投影 `error_attributions`（tier 由 Task 6 `score_signal`，error_type/cause_category 落列）+ `infra_question_ids`（infra=1,tier=T0,error_type/evolution_target=NULL）+ `degraded_question_ids`（degraded=1,tier=uncertain）→ upsert。
|
||
> **错误处理实况（Codex I3）**：`run_diagnosis` 的 defect/lapse 判别处对 judge 异常是 `except Exception` 后 warning + 默认 lapse（`core/evolution/diagnose.py:2186-2192`），**非全传播**。本编排不谎称"全传播"：网络/API 层失败经 GovernedLLMClient 重试栈后仍失败会向上抛；judge 语义歧义按现有保护性 lapse 处理并**计数上报**（`degraded_count`/默认 lapse 数写入 manifest）。产出 MD 到 `tests/outputs/`。
|
||
- [ ] **Step 4: 运行确认通过**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: add offline baseline diagnosis orchestration"`
|
||
|
||
---
|
||
|
||
## Phase 2：signal 分层 + 视频聚合 + 贪心选择（纯函数）
|
||
|
||
### Task 5: evolution_target 派生 + cell 定义
|
||
|
||
**Files:** Create `app/harness/split_selection.py`（本 Task 起逐步充实）；Test `tests/unit/test_split_selection.py`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_split_selection.py
|
||
from app.harness.split_selection import evolution_target_of, cell_of
|
||
|
||
|
||
def test_evolution_target_mapping():
|
||
assert evolution_target_of("extraction_failure") == "tool"
|
||
assert evolution_target_of("search_failure") == "skill"
|
||
assert evolution_target_of("reasoning_failure") == "skill"
|
||
assert evolution_target_of("mixed") == "system"
|
||
|
||
|
||
def test_cell_is_task_type_x_error_type():
|
||
assert cell_of("Counting Problem", "search_failure") == ("Counting Problem", "search_failure")
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现**
|
||
|
||
```python
|
||
# app/harness/split_selection.py (片段)
|
||
_EVOLUTION_TARGET = {
|
||
"extraction_failure": "tool",
|
||
"search_failure": "skill",
|
||
"reasoning_failure": "skill",
|
||
"mixed": "system",
|
||
}
|
||
|
||
|
||
def evolution_target_of(error_type: str) -> str:
|
||
"""error_type → 进化目标（派生标注，非独立多样性轴）。"""
|
||
if error_type not in _EVOLUTION_TARGET:
|
||
raise ValueError(f"未知 error_type: {error_type}")
|
||
return _EVOLUTION_TARGET[error_type]
|
||
|
||
|
||
def cell_of(task_type: str, error_type: str) -> tuple[str, str]:
|
||
"""多样性主格子 = (task_type, error_type)。"""
|
||
return (task_type, error_type)
|
||
```
|
||
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: add evolution_target derivation and cell"`
|
||
|
||
### Task 6: score_signal 分层
|
||
|
||
**Files:** Modify `app/harness/split_selection.py`；Test 同上文件追加
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
def test_tiers():
|
||
from app.harness.split_selection import score_signal
|
||
assert score_signal(cause_category="defect", infra=False, degraded=False).tier == "T2"
|
||
assert score_signal(cause_category="lapse", infra=False, degraded=False).tier == "T1"
|
||
assert score_signal(cause_category="defect", infra=True, degraded=False).tier == "T0"
|
||
assert score_signal(cause_category=None, infra=False, degraded=True).tier == "uncertain"
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** `score_signal(...) -> SignalLabel`（dataclass `tier: str`）：优先级 INFRA→T0；degraded/诊断失败→uncertain；`cause_category=='defect'`→T2；`=='lapse'`→T1；其余→uncertain。**顺序固定，不用魔法权重**。
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: add signal tiering"`
|
||
|
||
### Task 7: 全视频记录构建（VideoRecord）+ 信号聚合
|
||
|
||
**Files:** Modify `app/harness/split_selection.py`；Test 追加
|
||
|
||
> **Codex 计划审 C4**：`select_split` 需**全 300 视频**的 type/difficulty/correctness 分布（含 125 个全对零信号视频）来算 ε 代表性与 test 补集，**不能只喂诊断行**。故先由全 900 predictions + questions 构建 `VideoRecord`，再叠加 T2 诊断信号。
|
||
|
||
- [ ] **Step 1: 写失败测试**（真实数据二次构造：从 infer_adhoc 抽一个真实视频的 3 题结构）
|
||
|
||
```python
|
||
def test_build_video_records_covers_all_videos_with_difficulty_and_types():
|
||
from app.harness.split_selection import build_video_records
|
||
# 二次构造：真实 3 题/视频、真实 type、真实对错
|
||
preds = [
|
||
{"video_id": "v1", "question_id": "v1-1", "task_type": "Counting Problem", "correct": False},
|
||
{"video_id": "v1", "question_id": "v1-2", "task_type": "Action Reasoning", "correct": True},
|
||
{"video_id": "v1", "question_id": "v1-3", "task_type": "OCR Problems", "correct": True},
|
||
{"video_id": "v2", "question_id": "v2-1", "task_type": "Counting Problem", "correct": True},
|
||
{"video_id": "v2", "question_id": "v2-2", "task_type": "Counting Problem", "correct": True},
|
||
{"video_id": "v2", "question_id": "v2-3", "task_type": "Counting Problem", "correct": True},
|
||
]
|
||
signal_rows = [{"question_id": "v1-1", "task_type": "Counting Problem",
|
||
"error_type": "search_failure", "tier": "T2"}]
|
||
recs = build_video_records(preds, signal_rows)
|
||
assert {r.video_id for r in recs} == {"v1", "v2"} # 全视频（含零信号 v2）
|
||
v1 = next(r for r in recs if r.video_id == "v1")
|
||
v2 = next(r for r in recs if r.video_id == "v2")
|
||
assert v1.n_correct == 2 and v1.difficulty == 1 # 3题对2 → 难度画像桶=1错
|
||
assert v2.difficulty == 0 and v2.cells == set() # 零信号视频进 test 无 T2 格子
|
||
assert v1.cells == {("Counting Problem", "search_failure")}
|
||
assert v1.type_set == {"Counting Problem", "Action Reasoning", "OCR Problems"}
|
||
assert v1.wrong_by_type == {"Counting Problem": 1} # T2 计数供 floor
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** `build_video_records(preds, signal_rows) -> list[VideoRecord]`：按 video_id 聚合全 900 题 → `VideoRecord(video_id, type_set, n_correct, difficulty=3-n_correct 的错题数桶, cells:set(仅 T2 题 cell_of 并集去重), wrong_by_type:dict(各type T2 数))`。零信号视频 cells 空、仍在列表（供 test 与 ε）。
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: build all-video records with difficulty and signal overlay"`
|
||
|
||
### Task 8: 贪心联合约束选择器
|
||
|
||
**Files:** Modify `app/harness/split_selection.py`；Test 追加
|
||
|
||
- [ ] **Step 1: 写失败测试**（输入为 Task 7 的 `VideoRecord`；fixture 用真实 infer_adhoc 各 type 分布二次构造，非纯虚构——满足 §4.6）
|
||
|
||
```python
|
||
def _real_shaped_video_records():
|
||
# 从 workspaces/default/harness.db infer_adhoc 抽真实(video,3题type,correct)二次构造
|
||
# 保留真实类型长尾(Counting/AR/OCR...)与 0/1/2/3-对难度画像分布
|
||
... # helper：读真实 DB → build_video_records；见 conftest 提供的 fixture
|
||
|
||
|
||
def test_select_split_video_disjoint_and_floor_and_deterministic():
|
||
from app.harness.split_selection import select_split, SelectConfig, derive_reportable_types
|
||
videos = _real_shaped_video_records()
|
||
total_by_type = _count_questions_by_type(videos)
|
||
cfg = SelectConfig(n_trainval=100, floor_k={"Counting Problem": 3}, epsilon=0.1,
|
||
reportable_types=derive_reportable_types(total_by_type, report_floor=27),
|
||
seed=7)
|
||
a = select_split(videos, config=cfg)
|
||
b = select_split(videos, config=cfg)
|
||
assert set(a.trainval) & set(a.test) == set() # 互斥
|
||
assert set(a.trainval) | set(a.test) == {v.video_id for v in videos}
|
||
assert a.trainval == b.trainval # 同 seed 同解
|
||
assert sum(v.wrong_by_type.get("Counting Problem", 0) # floor 达标
|
||
for v in videos if v.video_id in a.trainval) >= 3
|
||
|
||
|
||
def test_infeasible_floor_vs_epsilon_raises():
|
||
from app.harness.split_selection import select_split, SelectConfig, InfeasibleSplitError
|
||
import pytest
|
||
videos = _real_shaped_video_records()
|
||
with pytest.raises(InfeasibleSplitError): # 极小 ε + 高 floor → 死锁
|
||
select_split(videos, config=SelectConfig(n_trainval=2, floor_k={"OCR Problems": 50},
|
||
epsilon=0.001, reportable_types=set(), seed=1))
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** `select_split(videos: list[VideoRecord], config) -> SplitAssignment(trainval, test)`（全局统计从 `videos` 内部算，无需外传）：按设计 §6 两阶段贪心——floor 阶段与多样性阶段**均带 ε 可行性检查**（移入 trainval 后 test=补集 仍满足 per-type 比例 + 0/1/2/3-对难度画像 ±ε）；floor 死锁/欠额抛 `InfeasibleSplitError`。**确定性**：先按 `seed` 对候选做一次固定预洗牌，再按 `-边际增益` 稳定排序（seed 只控预洗牌打破等增益平局，非二次 key）。`_epsilon_ok(test_videos, ε)` 校验 reportable 类型比例与难度画像偏差 ≤ ε。同文件加 `derive_reportable_types(total_by_type, report_floor) -> set[str]`（总题数 ≥ report_floor 的类型，落地设计 §7）。
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: add greedy joint-constrained split selector"`
|
||
|
||
---
|
||
|
||
## Phase 3：pools 视频原子 + 冻结
|
||
|
||
### Task 9: pools 切分原子 unit→video
|
||
|
||
**Files:**
|
||
- Modify: `app/harness/pools.py`（`build_pools`/`_split_one_category`/`build_or_load_pools`）
|
||
- Test: `tests/unit/test_pools_video_atomic.py` + 回归 `tests/unit/test_run_store.py`（若涉及）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_pools_video_atomic.py
|
||
from app.harness.pools import split_by_video_assignment
|
||
from core.types import GeneratedQuestion
|
||
|
||
|
||
def _q(qid, vid, tt="Counting Problem"):
|
||
return GeneratedQuestion(question_id=qid, video_id=vid, task_type=tt, question="", options=("A","B","C","D"), answer="A")
|
||
|
||
|
||
def test_video_never_split_across_pools():
|
||
qs = [_q("v1-1","v1"), _q("v1-2","v1"), _q("v1-3","v1"), _q("v2-1","v2")]
|
||
assignment = {"v1": "trainval", "v2": "test"}
|
||
pools = split_by_video_assignment(qs, assignment, correctness={q.question_id: True for q in qs},
|
||
val_ratio=0.0, seed=0)
|
||
test_vids = {q.video_id for q in pools.test}
|
||
train_vids = {q.video_id for q in pools.diagnosis + pools.validation}
|
||
assert test_vids & train_vids == set() # 视频不跨池
|
||
assert test_vids == {"v2"} and train_vids == {"v1"}
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** 新增 `split_by_video_assignment(questions, assignment, correctness, val_ratio, seed) -> Pools`：先按 assignment 把题分到 trainval/test（video 原子）；trainval 内用**视频组**做 correctness 分层切 train(diagnosis)/val——改 `_split_one_category` 采样原子为 video 组（同 video 全 unit 同进同出，复用现有 correctness 分层 + `eval_min_per_class`）；test 直接为 test 视频全部题。保留 `baseline_val_accuracy`/`correctness` 计算。
|
||
- [ ] **Step 4: 通过确认** + **保真检查点**：断言 `pools.diagnosis` 内仍是逐 unit 列表（`gate_ladder` 输入不变）。运行 `pytest tests/unit/test_pools*.py tests/unit/test_gate.py -v`。
|
||
- [ ] **Step 5: 提交** — `git commit -m "refactor: add video-atomic pool split (algo #5 gate input preserved)"`
|
||
|
||
### Task 10: 原子冻结 + manifest
|
||
|
||
**Files:**
|
||
- Modify: `app/harness/pools.py`（`save_pools` 与 per_category 写路径抽共用原子助手）
|
||
- Create: `app/harness/split_manifest.py`
|
||
- Test: `tests/unit/test_atomic_save_pools.py`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/unit/test_atomic_save_pools.py
|
||
import json
|
||
from app.harness.pools import _atomic_write_json # 新共用助手
|
||
|
||
|
||
def test_atomic_write_replaces_and_no_tmp_left(tmp_path):
|
||
p = tmp_path / "pools.json"
|
||
_atomic_write_json(p, {"a": 1})
|
||
assert json.loads(p.read_text())["a"] == 1
|
||
assert list(tmp_path.glob("*.tmp")) == [] # 无残留 tmp
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** `_atomic_write_json(path, obj)`：写 `path.with_suffix(".tmp")` 后 `os.replace`；`save_pools`(341) 与 per_category 写(566) 均改调它。`split_manifest.py::write_manifest(...)` 写 baseline_run_id/diag_fingerprint/seed/config/pools_sha256/coverage_report。
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "fix: make pools.json freeze atomic + add split manifest"`
|
||
|
||
### Task 11: 端到端集成 + 防御断言
|
||
|
||
**Files:**
|
||
- Create: `app/harness/build_split.py`（顶层编排：诊断结果 → signal → 视频聚合 → select_split → split_by_video_assignment → 冻结 + manifest）
|
||
- Test: `tests/integration/test_build_split_e2e.py`
|
||
|
||
- [ ] **Step 1: 写失败集成测试**（用真实 infer_adhoc 900 预测 + Task 3 缓存诊断表；若无缓存则 mock 诊断行）
|
||
|
||
```python
|
||
# tests/integration/test_build_split_e2e.py
|
||
import pytest
|
||
from app.harness.build_split import build_split
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_end_to_end_freezes_valid_pools(tmp_path):
|
||
out = tmp_path / "pools.json"
|
||
result = await build_split(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||
harness_db="workspaces/default/harness.db",
|
||
signal_store=_cached_or_mock_store(), out_path=out, config=_calibrated_cfg())
|
||
pools = result.pools
|
||
# 防御断言①: 三池视频互斥
|
||
tv = {q.video_id for q in pools.diagnosis + pools.validation}
|
||
te = {q.video_id for q in pools.test}
|
||
assert tv & te == set()
|
||
# 防御断言②: 覆盖全 900 题
|
||
assert len(pools.diagnosis) + len(pools.validation) + len(pools.test) == 900
|
||
# manifest 存在且 pools_sha256 校验一致
|
||
assert result.manifest["pools_sha256"] == _sha256(out.read_text())
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** `build_split(...)`：从 signal_store.load(run,fp) 取逐题 tier/cell → `aggregate_video_signal` 逐视频 → `select_split` → `split_by_video_assignment` → `save_pools`(原子) + `write_manifest`。落地设计 §10 防御断言清单①②③④⑤⑥（三池互斥/900 覆盖/每视频 3 题/fingerprint 一致/manifest hash/ID 唯一），任一不满足 fail-fast。
|
||
- [ ] **Step 4: 通过确认** — `pytest tests/integration/test_build_split_e2e.py -v`。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: wire end-to-end results-driven split with defensive asserts"`
|
||
|
||
---
|
||
|
||
## Phase 4：配置与标定
|
||
|
||
### Task 12: 科研 YAML 旋钮 + 标定程序
|
||
|
||
**Files:**
|
||
- Modify: `core/types.py`（`PoolConfig` 加 `n_trainval/floor_k/epsilon/report_floor/val_wrong_min`）
|
||
- Create: `config/video_split.yaml`
|
||
- Create: `scripts/build_video_split.sh`
|
||
- Test: `tests/unit/test_pool_config_video_split.py`
|
||
|
||
- [ ] **Step 1: 写失败测试** — ① `PoolConfig` 可从 `config/video_split.yaml` 载入新字段且缺失关键项 fail-fast（不兜底默认）；② `diag_fingerprint(prompt_version, model, code_version)` 确定性、任一输入变则变；③ `val_wrong_min` 进入 `split_by_video_assignment` 可行性：val 错题数 < `val_wrong_min` 时抛错。
|
||
|
||
```python
|
||
def test_diag_fingerprint_deterministic_and_sensitive():
|
||
from app.harness.split_selection import diag_fingerprint
|
||
a = diag_fingerprint("p1", "deepseek-v4", "abc123")
|
||
assert a == diag_fingerprint("p1", "deepseek-v4", "abc123") # 确定性
|
||
assert a != diag_fingerprint("p2", "deepseek-v4", "abc123") # prompt 变则变
|
||
assert a != diag_fingerprint("p1", "kimi", "abc123") # model 变则变
|
||
|
||
|
||
def test_val_wrong_min_enforced(tmp_path):
|
||
from app.harness.pools import split_by_video_assignment, InsufficientValSignal
|
||
import pytest
|
||
with pytest.raises(InsufficientValSignal): # val 错题不足功效阈
|
||
split_by_video_assignment(_all_correct_qs(), {"v": "trainval"},
|
||
correctness=_all_true(), val_ratio=0.5, seed=0, val_wrong_min=5)
|
||
```
|
||
|
||
- [ ] **Step 2: 失败确认**。
|
||
- [ ] **Step 3: 实现** 扩 `PoolConfig`（`n_trainval/floor_k/epsilon/report_floor/val_wrong_min`）；`diag_fingerprint(...)= hashlib.sha256("|".join(...)).hexdigest()[:16]`（来源：诊断 prompt 文件 hash + `.env` 模型名 + `git rev-parse HEAD` 短 SHA，`--force` 用**新 fingerprint** 写、不覆盖旧记录）；`split_by_video_assignment` 加 `val_wrong_min` 参数，val 切出后校验错题数 ≥ 阈值否则 `InsufficientValSignal`；写 `config/video_split.yaml`（占位阈值 + 注释"诊断后标定"）；`scripts/build_video_split.sh` 自包含零参复现（GPU 卡号除外）。**标定程序**：诊断跑完读 `baseline_diagnosis` 各 type T2 数 → 定 `floor_k`(如 min(可用defect,3))、`n_trainval`(~100)、`epsilon`(如 0.1)、`val_wrong_min`（McNemar 功效阈，如 ≥20）。
|
||
- [ ] **Step 4: 通过确认**。
|
||
- [ ] **Step 5: 提交** — `git commit -m "feat: add video-split config knobs and reproducible script"`
|
||
|
||
---
|
||
|
||
## Self-Review（作者自查，已执行）
|
||
|
||
- **Spec 覆盖**：设计 §4 组件→Task 1-12 一一对应；§5 signal/多样性→Task 5-7；§6 贪心→Task 8；§7 长尾→Task 8/12(reportable_types+report_floor)；§8 val→Task 9(val_ratio+min_per_class)+Task 12(val_wrong_min)；§9 非功能→Task 3(逐题续跑/upsert)、Task 10(原子写)；§10 错误→Task 4(P5 传播)、Task 8(InfeasibleSplitError)、Task 11(防御断言)；§12 前序继承→Task 9(三池/pair/baseline_val_accuracy)+Task 10(per_category 原子写)。
|
||
- **占位扫描**：无 TBD/TODO；阈值项在 Task 12 明确"诊断后标定"并给标定程序，非占位。
|
||
- **类型一致**：`DiagnosisSignalRow`/`DiagnosisDeps`/`VideoRecord`/`SignalLabel`/`SelectConfig`/`SplitAssignment`/`InfeasibleSplitError`/`InsufficientValSignal`/`derive_reportable_types`/`diag_fingerprint`/`_atomic_write_json`/`split_by_video_assignment`/`build_video_records` 跨 Task 命名一致。
|
||
- **Codex 计划审修订已并入**：C1 字段名 `tool`（Task 1/2）、C2 canonical 事实（900/660/236错/4 INFRA-null，按 distinct question 断言）、C3 Task 4 去占位显式 fake、C4 全视频 `VideoRecord` 模型（Task 7）；I1 val_wrong_min 接回切分（Task 12）、I2 `derive_reportable_types`（Task 8）、I3 诚实标注 run_diagnosis 错误处理、I4 schema NULL 规则（Task 0）、I5 真实数据二次构造 fixture、I6 `diag_fingerprint` 函数+测试；M1 锚点 809、M2 保真点 Task 9、M3 seed 只控预洗牌。
|
||
|
||
## 里程碑
|
||
|
||
| 里程碑 | 完成 Task |
|
||
|--------|----------|
|
||
| M1 诊断信号就绪（236 可诊断错题分层 + 4 INFRA-null 落 T0） | 0-4 |
|
||
| M2 选择器可产合法切分（纯函数全绿） | 5-8 |
|
||
| M3 冻结 pools.json + manifest 端到端 | 9-11 |
|
||
| M4 可复现脚本 + 标定 | 12 |
|
||
|
||
## 验收标准
|
||
|
||
- 冻结 `pools.json`：三池视频互斥、覆盖 900 题、floor 达标、test 代表性偏差 ≤ ε；manifest hash 自洽。
|
||
- 全测试绿（unit + integration）；LLM 类测试产出 MD。
|
||
- `bash scripts/build_video_split.sh` 零参复现（GPU 卡号除外）。
|
||
- gate_ladder 冷启动 2:1 输入不变（保真检查点通过）。
|
||
|
||
## 风险
|
||
|
||
| 风险 | 缓解 |
|
||
|------|------|
|
||
| steps_json 部分为空(3/902)或字段异常 | Task 1 空返回 []；诊断对应题落 uncertain，计数上报 |
|
||
| 100 训练视频信号不足 | 切设计 §14 方案 B（外部 benchmark 训练） |
|
||
| floor 与 ε 联合不可行 | Task 8 InfeasibleSplitError fail loud，Task 12 放松旋钮 |
|