Video-Tree-TRM5/research-wiki/plans/2026-07-14-action-recognition-training.md

# Action Recognition 单题型首次训练实验 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** 用 Action Recognition 单题型端到端验证训练管线，包含 3 处代码修改、1 套实验配置、1 个前置脚本。

**Architecture:** 修改 `PerCategoryPoolStrategy.build()` 使 test 池受 `task_types` 过滤；RunConfig 新增 `run_holdout_eval` 开关控制 epoch 内四向 held-out；`load_config` 修复 YAML list→tuple 转换；新建实验 YAML 和 sh 脚本驱动训练。

**Tech Stack:** Python 3.11, pytest, YAML, bash

---

### Task 0: 清理 v2-360 目录中的 backup 文件

**Files:**
- Modify: `store/questions/generated-v2-360/`（重命名文件）

`load_benchmark()` 会加载目录下所有 `*.json`，`accepted_questions_backup_220.json` 含 18 道重复 AR 题会干扰训练。

- [ ] **Step 1: 重命名 backup 文件使其不被 load_benchmark 加载**

```bash
mv store/questions/generated-v2-360/accepted_questions_backup_220.json \
   store/questions/generated-v2-360/accepted_questions_backup_220.json.bak
```

- [ ] **Step 2: 验证 load_benchmark 只加载 180 题**

```bash
conda activate Video-Tree-TRM & python -c "
from pathlib import Path
from app.question_gen import load_benchmark
qs = load_benchmark(Path('store/questions/generated-v2-360'))
print(f'Total: {len(qs)}')
ar = [q for q in qs if q.task_type == 'Action Recognition']
print(f'Action Recognition: {len(ar)}')
assert len(qs) == 180, f'Expected 180, got {len(qs)}'
assert len(ar) == 30, f'Expected 30 AR, got {len(ar)}'
print('OK')
"
```

预期：Total: 180, Action Recognition: 30, OK

- [ ] **Step 3: 提交**

```bash
git add -A store/questions/generated-v2-360/
git commit -m "chore: rename v2-360 backup JSON to .bak to exclude from load_benchmark"
```

---

### Task 1: PerCategoryPoolStrategy test 池 task_types 过滤

**Files:**
- Modify: `app/harness/pools.py:600-606`
- Test: `tests/unit/test_harness_pools.py`

- [ ] **Step 1: 写失败测试 — test 池按 task_types 过滤**

在 `tests/unit/test_harness_pools.py` 的 `TestPerCategoryPoolStrategy` 类末尾新增。
注意：`load_benchmark` 要求每个 JSON 文件内容为**题目数组**（`[{...}]`），不是单个 dict。

```python
def test_per_category_test_pool_filtered_by_task_types(self, tmp_path: Path):
    """test_questions_dir 含多题型时，test 池只保留 task_types 指定的题型。"""
    test_dir = tmp_path / "test_questions"
    test_dir.mkdir()
    for tt in ("Action Recognition", "Object Reasoning", "Counting Problem"):
        items = []
        for i in range(10):
            qid = f"{tt.replace(' ', '_')}_{i:03d}"
            items.append({
                "question_id": qid,
                "video_id": "v1",
                "task_type": tt,
                "question": f"Q {qid}?",
                "options": ["A. a", "B. b", "C. c", "D. d"],
                "answer": "A",
            })
        slug = tt.lower().replace(" ", "_")
        (test_dir / f"{slug}.json").write_text(
            json.dumps(items, ensure_ascii=False), encoding="utf-8"
        )

    questions = [_make_question(f"ar_{i:03d}", "Action Recognition") for i in range(30)]
    correctness = {q.question_id: (i < 20) for i, q in enumerate(questions)}

    config = PoolConfig(
        task_types=("Action Recognition",),
        seed=42,
        baseline_run_id="bl",
        diag_size=0,
        diag_correct_ratio=0.0,
        val_size=0,
        val_correct_ratio=0.0,
        test_size=0,
        eval_min_per_class=0,
        train_ratio=20 / 30,
        test_questions_dir=test_dir,
    )
    strategy = PerCategoryPoolStrategy()
    pools = strategy.build(questions, correctness, config)

    assert len(pools.test) == 10
    assert all(q.task_type == "Action Recognition" for q in pools.test)

def test_per_category_test_pool_no_filter_when_task_types_none(self, tmp_path: Path):
    """task_types=None 时 test 池不过滤，保留全部题型。"""
    test_dir = tmp_path / "test_questions"
    test_dir.mkdir()
    for tt in ("Action Recognition", "Object Reasoning"):
        items = []
        for i in range(5):
            qid = f"{tt.replace(' ', '_')}_{i:03d}"
            items.append({
                "question_id": qid,
                "video_id": "v1",
                "task_type": tt,
                "question": f"Q {qid}?",
                "options": ["A. a", "B. b", "C. c", "D. d"],
                "answer": "A",
            })
        slug = tt.lower().replace(" ", "_")
        (test_dir / f"{slug}.json").write_text(
            json.dumps(items, ensure_ascii=False), encoding="utf-8"
        )

    questions = [_make_question(f"q_{i:03d}", "Action Recognition") for i in range(10)]
    correctness = {q.question_id: True for q in questions}

    config = PoolConfig(
        task_types=None,
        seed=42,
        baseline_run_id="bl",
        diag_size=0,
        diag_correct_ratio=0.0,
        val_size=0,
        val_correct_ratio=0.0,
        test_size=0,
        eval_min_per_class=0,
        train_ratio=0.667,
        test_questions_dir=test_dir,
    )
    strategy = PerCategoryPoolStrategy()
    pools = strategy.build(questions, correctness, config)

    assert len(pools.test) == 10
```

- [ ] **Step 2: 运行测试验证失败**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types -v
```

预期：FAIL — `assert len(pools.test) == 10` 失败（实际 30 题，未过滤）。

- [ ] **Step 3: 实现 test 池过滤**

修改 `app/harness/pools.py` `PerCategoryPoolStrategy.build()` 的 Phase 4：

```python
        # Phase 4: test 池（从外部目录加载，无则空）
        test: list[GeneratedQuestion] = []
        if config.test_questions_dir is not None:
            from app.question_gen import load_benchmark

            test = load_benchmark(config.test_questions_dir)
            if config.task_types is not None:
                allowed = set(config.task_types)
                test = [q for q in test if q.task_type in allowed]
```

- [ ] **Step 4: 运行测试验证通过**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_no_filter_when_task_types_none -v
```

预期：PASS

- [ ] **Step 5: 运行全部 pool 测试确保无回归**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/integration/test_pool_strategy.py -v
```

预期：全部 PASS

- [ ] **Step 6: 提交**

```bash
git add app/harness/pools.py tests/unit/test_harness_pools.py
git commit -m "feat(pools): filter test pool by task_types in PerCategoryPoolStrategy"
```

---

### Task 2: RunConfig 新增 run_holdout_eval + load_config list→tuple 修复 + Runner 条件跳过

**Files:**
- Modify: `app/harness/config.py:147` (字段) + `app/harness/config.py:437` (list→tuple)
- Modify: `app/harness/runner.py:1371`
- Modify: `main.py:205`
- Test: `tests/unit/test_harness_pools.py`

- [ ] **Step 1: 写失败测试 — RunConfig 新字段**

在 `tests/unit/test_harness_pools.py` 文件末尾新增：

```python
class TestRunHoldoutEvalConfig:
    """run_holdout_eval 字段校验。"""

    def test_default_true(self):
        """run_holdout_eval 默认值为 True。"""
        from app.harness.config import RunConfig

        config = RunConfig(
            workspace_dir=Path("/tmp/ws"),
            store_dir=Path("/tmp/store"),
            mode="train",
            concurrency=4,
            max_steps=10,
            skill_mode="auto",
            n_samples=0,
            questions="benchmarks/Video-MME",
            skills_version="v1",
            prompts_version="v1",
            epochs=1,
            diag_size=100,
            diag_correct_ratio=0.5,
            val_size=30,
            val_correct_ratio=0.5,
            edit_budget_start=5,
            edit_budget_end=2,
            batch_size=15,
            min_class_per_batch=2,
            eval_min_per_class=2,
            early_stop_patience=4,
            test_size=30,
            use_slow_momentum=True,
            gate_e_confirm=20.0,
            gate_e_provisional=3.0,
            gate_w_net_min=2,
            gate_delta_min=0.02,
            gate_lambda_dir=-0.642,
            gate_e_rollback=10.0,
            gate_block=8,
            gate_n_max=40,
            gate_p_low=0.05,
            gate_p_high=0.95,
            gate_probe_quota=0.2,
            gate_gamma_decay=0.9,
            gate_cooldown_steps=2,
            gate_guard_err=0.10,
            skill_update_mode="patch",
            appendix_consolidate_threshold=6,
            run_id="test_run",
        )
        assert config.run_holdout_eval is True

    def test_explicit_false(self):
        """run_holdout_eval 可设为 False。"""
        from app.harness.config import RunConfig

        config = RunConfig(
            workspace_dir=Path("/tmp/ws"),
            store_dir=Path("/tmp/store"),
            mode="train",
            concurrency=4,
            max_steps=10,
            skill_mode="auto",
            n_samples=0,
            questions="benchmarks/Video-MME",
            skills_version="v1",
            prompts_version="v1",
            epochs=1,
            diag_size=100,
            diag_correct_ratio=0.5,
            val_size=30,
            val_correct_ratio=0.5,
            edit_budget_start=5,
            edit_budget_end=2,
            batch_size=15,
            min_class_per_batch=2,
            eval_min_per_class=2,
            early_stop_patience=4,
            test_size=30,
            use_slow_momentum=True,
            gate_e_confirm=20.0,
            gate_e_provisional=3.0,
            gate_w_net_min=2,
            gate_delta_min=0.02,
            gate_lambda_dir=-0.642,
            gate_e_rollback=10.0,
            gate_block=8,
            gate_n_max=40,
            gate_p_low=0.05,
            gate_p_high=0.95,
            gate_probe_quota=0.2,
            gate_gamma_decay=0.9,
            gate_cooldown_steps=2,
            gate_guard_err=0.10,
            skill_update_mode="patch",
            appendix_consolidate_threshold=6,
            run_id="test_run",
            run_holdout_eval=False,
        )
        assert config.run_holdout_eval is False
```

- [ ] **Step 2: 运行测试验证失败**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v
```

预期：FAIL — `TypeError: __init__() got an unexpected keyword argument 'run_holdout_eval'`

- [ ] **Step 3: RunConfig 新增 run_holdout_eval 字段**

在 `app/harness/config.py` 的有默认值字段区（`test_questions` 后面）新增：

```python
    test_questions: str = "benchmarks/Video-MME"
    run_holdout_eval: bool = True
```

- [ ] **Step 4: 运行测试验证通过**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v
```

预期：PASS

- [ ] **Step 5: load_config 修复 YAML list→tuple 转换**

`RunConfig.task_types` 类型为 `tuple[str, ...] | None`，但 YAML list 加载后不转换。
在 `app/harness/config.py` `load_config()` Phase 4（类型转换区）后面新增：

```python
    # Phase 4: 类型转换 — 路径字段转 Path
    for field_name in _PATH_FIELDS:
        if field_name in yaml_data:
            yaml_data[field_name] = Path(yaml_data[field_name])

    # Phase 4b: 类型转换 — task_types list → tuple
    if "task_types" in yaml_data and yaml_data["task_types"] is not None:
        yaml_data["task_types"] = tuple(yaml_data["task_types"])
```

- [ ] **Step 6: main.py 新增 CLI 开关**

在 `main.py` `_build_parser()` 的 `--test-questions` 后面新增：

```python
    parser.add_argument("--test-questions", type=str, dest="test_questions")
    parser.add_argument(
        "--no-run-holdout-eval",
        action="store_true",
        dest="no_run_holdout_eval",
    )
    return parser
```

在 `main()` 中 `cli_overrides` 构建处（约第 269 行 `cli_overrides = ...` 之后）处理取反映射：

```python
    cli_overrides = {k: v for k, v in cli_args.items() if k != "config"}
    if cli_overrides.get("no_run_holdout_eval"):
        cli_overrides["run_holdout_eval"] = False
    cli_overrides.pop("no_run_holdout_eval", None)
```

- [ ] **Step 7: Runner `_slow_update_cycle` 条件跳过 holdout**

修改 `app/harness/runner.py` `_slow_update_cycle` 的 Phase 9（约第 1371 行）：

将：
```python
        await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)
```

改为：
```python
        if self._config.run_holdout_eval:
            await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)
```

- [ ] **Step 8: 运行全部测试确认无回归**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/unit/test_harness_store.py tests/integration/test_pool_strategy.py -v
```

预期：全部 PASS

- [ ] **Step 9: 提交**

```bash
git add app/harness/config.py app/harness/runner.py main.py tests/unit/test_harness_pools.py
git commit -m "feat(config): add run_holdout_eval toggle and fix YAML task_types list-to-tuple"
```

---

### Task 3: 实验配置文件

**Files:**
- Create: `config/train_action_recognition.yaml`

- [ ] **Step 1: 创建实验 YAML**

```yaml
# config/train_action_recognition.yaml
# Action Recognition 单题型首次训练实验
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md

harness:
  workspace_dir: "workspaces/train-action-recognition"
  store_dir: store
  mode: train
  run_id: train_ar_v1
  concurrency: 24
  max_steps: 40
  skill_mode: auto
  n_samples: 0
  questions: "generated-v2-360"
  skills_version: v1
  prompts_version: v1
  epochs: 3
  # CE-Gate 参数（沿用 default.yaml）
  gate_e_confirm: 20.0
  gate_e_provisional: 3.0
  gate_w_net_min: 2
  gate_delta_min: 0.02
  gate_lambda_dir: -0.642
  gate_e_rollback: 10.0
  gate_block: 8
  gate_n_max: 40
  gate_p_low: 0.05
  gate_p_high: 0.95
  gate_probe_quota: 0.2
  gate_gamma_decay: 0.9
  gate_cooldown_steps: 2
  gate_guard_err: 0.10
  # 进化参数
  edit_budget_start: 5
  edit_budget_end: 2
  skill_update_mode: patch
  appendix_consolidate_threshold: 6
  # 池配置 — per_category 单题型
  pool_split_mode: per_category
  task_types:
    - "Action Recognition"
  train_ratio: 0.667
  test_questions: "benchmarks/Video-MME"
  run_holdout_eval: false
  # mini-batch
  batch_size: 10
  min_class_per_batch: 2
  batch_correct_ratio: 0.5
  momentum_samples: 20
  eval_min_per_class: 2
  early_stop_patience: 4
  test_size: 63
  diag_size: 20
  diag_correct_ratio: 0.5
  val_size: 10
  val_correct_ratio: 0.5
  use_slow_momentum: true

embed:
  backend: "local"
  model_name: "BAAI/bge-base-zh-v1.5"
  embed_dim: 768
  device: "cuda"
```

- [ ] **Step 2: 验证 YAML 可正确加载为 RunConfig**

```bash
conda activate Video-Tree-TRM & python -c "
from app.harness.config import load_config
from pathlib import Path
config = load_config(Path('config/train_action_recognition.yaml'))
assert config.task_types == ('Action Recognition',), f'task_types={config.task_types}'
assert config.run_holdout_eval is False
assert config.pool_split_mode == 'per_category'
print('Config loaded OK')
"
```

预期：Config loaded OK

- [ ] **Step 3: 提交**

```bash
git add config/train_action_recognition.yaml
git commit -m "config: add train_action_recognition experiment YAML"
```

---

### Task 4: 前置脚本 — baseline infer + seed 创建 + 训练

**Files:**
- Create: `scripts/train_action_recognition.sh`

- [ ] **Step 1: 创建脚本**

```bash
#!/usr/bin/env bash
# Action Recognition 单题型训练实验
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
#
# 三阶段:
#   Phase 0: baseline infer (v2-360 Action Recognition 30 题)
#   Phase 1: create seed (v2ar-baseline)
#   Phase 2: train (3 epochs, per_category)
#
# 用法:
#   CUDA_VISIBLE_DEVICES=0 bash scripts/train_action_recognition.sh
#   MODE=mock bash scripts/train_action_recognition.sh   # 跳过 Phase 0/1
set -euo pipefail

cd "$(dirname "$0")/.."

CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
export CUDA_VISIBLE_DEVICES

export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTHONUNBUFFERED=1

set -a
source .env
set +a

PYTHON="$(conda run -n Video-Tree-TRM which python)"

# ── Phase 0: Baseline infer ──
if [[ "${MODE:-}" != "mock" ]]; then
    echo "=== Phase 0: Baseline infer (v2-360 Action Recognition 30 题) ==="
    "${PYTHON}" main.py \
        --config config/default.yaml \
        --workspace-dir workspaces/default \
        --store-dir store \
        --mode infer \
        --concurrency 24 \
        --max-steps 40 \
        --skill-mode auto \
        --n-samples 0 \
        --questions "generated-v2-360" \
        --skills-version v1 \
        --prompts-version v1 \
        --run-id v2ar_baseline \
        --task-types "Action Recognition"
fi

# ── Phase 1: Create seed ──
if [[ "${MODE:-}" != "mock" && ! -d "store/seeds/v2ar-baseline" ]]; then
    echo "=== Phase 1: Create seed v2ar-baseline ==="
    "${PYTHON}" -c "
from pathlib import Path
from app.harness.store import extract_run_db, init_seed
import tempfile

tmp = Path(tempfile.mkdtemp()) / 'baseline.db'
extract_run_db(
    Path('workspaces/default/harness.db'),
    tmp,
    'infer_v2ar_baseline',
)
init_seed(
    store_dir=Path('store'),
    name='v2ar-baseline',
    skills_dir=Path('store/skills/v1'),
    prompts_dir=Path('store/prompts/v1'),
    baseline_db=tmp,
    baseline_run_id='infer_v2ar_baseline',
    parent=None,
    description='v2-360 Action Recognition 30 题 baseline (skills/v1)',
)
tmp.unlink()
print('Seed created: store/seeds/v2ar-baseline/')
"
elif [[ -d "store/seeds/v2ar-baseline" ]]; then
    echo "=== Phase 1: Seed v2ar-baseline 已存在，跳过 ==="
fi

# ── Phase 2: Train ──
echo "=== Phase 2: Train (3 epochs, Action Recognition) ==="
"${PYTHON}" main.py \
    --config config/train_action_recognition.yaml \
    --fresh \
    --seed v2ar-baseline

echo "=== 训练完成 ==="
echo "结果查看:"
echo "  cat workspaces/train-action-recognition/analyses/final_test_eval.json"
echo "  sqlite3 workspaces/train-action-recognition/harness.db 'SELECT * FROM dual_metric'"
```

- [ ] **Step 2: 设置可执行权限并验证语法**

```bash
chmod +x scripts/train_action_recognition.sh
bash -n scripts/train_action_recognition.sh
```

预期：无语法错误

- [ ] **Step 3: 提交**

```bash
git add scripts/train_action_recognition.sh
git commit -m "scripts: add train_action_recognition experiment script"
```

---

### Task 5: lint 检查 + 全量回归测试

- [ ] **Step 1: Ruff 格式化与检查**

```bash
conda activate Video-Tree-TRM & ruff format app/ core/ && ruff check app/ core/ --fix
```

预期：无错误

- [ ] **Step 2: 全量测试**

```bash
conda activate Video-Tree-TRM & pytest tests/unit/ tests/integration/ -v --tb=short
```

预期：全部 PASS

- [ ] **Step 3: 最终提交（如有 lint 修复）**

```bash
git add -A && git commit -m "chore: lint and format training experiment changes"
```

---

## 核心算法保真校验

本计划不涉及核心算法迁移。修改仅限于：
- `PerCategoryPoolStrategy.build()` 新增 3 行 test 池过滤（不改 train/val 切分逻辑）
- `RunConfig` 新增 1 个 bool 字段
- `load_config` 新增 2 行 list→tuple 转换
- `_slow_update_cycle` 新增 1 行 `if` 条件（不改 holdout 内部逻辑）

保真校验不适用。