Video-Tree-TRM5/research-wiki/plans/2026-07-14-action-recognition-training.md

Action Recognition 单题型首次训练实验 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: 用 Action Recognition 单题型端到端验证训练管线，包含 3 处代码修改、1 套实验配置、1 个前置脚本。

Architecture: 修改 PerCategoryPoolStrategy.build() 使 test 池受 task_types 过滤；RunConfig 新增 run_holdout_eval 开关控制 epoch 内四向 held-out；load_config 修复 YAML list→tuple 转换；新建实验 YAML 和 sh 脚本驱动训练。

Tech Stack: Python 3.11, pytest, YAML, bash

---

Task 0: 清理 v2-360 目录中的 backup 文件

Files:

• Modify: store/questions/generated-v2-360/（重命名文件）

load_benchmark() 会加载目录下所有 *.json，accepted_questions_backup_220.json 含 18 道重复 AR 题会干扰训练。

• Step 1: 重命名 backup 文件使其不被 load_benchmark 加载

mv store/questions/generated-v2-360/accepted_questions_backup_220.json \
   store/questions/generated-v2-360/accepted_questions_backup_220.json.bak

• Step 2: 验证 load_benchmark 只加载 180 题

conda activate Video-Tree-TRM & python -c "
from pathlib import Path
from app.question_gen import load_benchmark
qs = load_benchmark(Path('store/questions/generated-v2-360'))
print(f'Total: {len(qs)}')
ar = [q for q in qs if q.task_type == 'Action Recognition']
print(f'Action Recognition: {len(ar)}')
assert len(qs) == 180, f'Expected 180, got {len(qs)}'
assert len(ar) == 30, f'Expected 30 AR, got {len(ar)}'
print('OK')
"

预期：Total: 180, Action Recognition: 30, OK

• Step 3: 提交

git add -A store/questions/generated-v2-360/
git commit -m "chore: rename v2-360 backup JSON to .bak to exclude from load_benchmark"

---

Task 1: PerCategoryPoolStrategy test 池 task_types 过滤

Files:

• Modify: app/harness/pools.py:600-606

• Test: tests/unit/test_harness_pools.py

• Step 1: 写失败测试 — test 池按 task_types 过滤

在 tests/unit/test_harness_pools.py 的 TestPerCategoryPoolStrategy 类末尾新增。 注意：load_benchmark 要求每个 JSON 文件内容为题目数组（[{...}]），不是单个 dict。

def test_per_category_test_pool_filtered_by_task_types(self, tmp_path: Path):
    """test_questions_dir 含多题型时，test 池只保留 task_types 指定的题型。"""
    test_dir = tmp_path / "test_questions"
    test_dir.mkdir()
    for tt in ("Action Recognition", "Object Reasoning", "Counting Problem"):
        items = []
        for i in range(10):
            qid = f"{tt.replace(' ', '_')}_{i:03d}"
            items.append({
                "question_id": qid,
                "video_id": "v1",
                "task_type": tt,
                "question": f"Q {qid}?",
                "options": ["A. a", "B. b", "C. c", "D. d"],
                "answer": "A",
            })
        slug = tt.lower().replace(" ", "_")
        (test_dir / f"{slug}.json").write_text(
            json.dumps(items, ensure_ascii=False), encoding="utf-8"
        )

    questions = [_make_question(f"ar_{i:03d}", "Action Recognition") for i in range(30)]
    correctness = {q.question_id: (i < 20) for i, q in enumerate(questions)}

    config = PoolConfig(
        task_types=("Action Recognition",),
        seed=42,
        baseline_run_id="bl",
        diag_size=0,
        diag_correct_ratio=0.0,
        val_size=0,
        val_correct_ratio=0.0,
        test_size=0,
        eval_min_per_class=0,
        train_ratio=20 / 30,
        test_questions_dir=test_dir,
    )
    strategy = PerCategoryPoolStrategy()
    pools = strategy.build(questions, correctness, config)

    assert len(pools.test) == 10
    assert all(q.task_type == "Action Recognition" for q in pools.test)

def test_per_category_test_pool_no_filter_when_task_types_none(self, tmp_path: Path):
    """task_types=None 时 test 池不过滤，保留全部题型。"""
    test_dir = tmp_path / "test_questions"
    test_dir.mkdir()
    for tt in ("Action Recognition", "Object Reasoning"):
        items = []
        for i in range(5):
            qid = f"{tt.replace(' ', '_')}_{i:03d}"
            items.append({
                "question_id": qid,
                "video_id": "v1",
                "task_type": tt,
                "question": f"Q {qid}?",
                "options": ["A. a", "B. b", "C. c", "D. d"],
                "answer": "A",
            })
        slug = tt.lower().replace(" ", "_")
        (test_dir / f"{slug}.json").write_text(
            json.dumps(items, ensure_ascii=False), encoding="utf-8"
        )

    questions = [_make_question(f"q_{i:03d}", "Action Recognition") for i in range(10)]
    correctness = {q.question_id: True for q in questions}

    config = PoolConfig(
        task_types=None,
        seed=42,
        baseline_run_id="bl",
        diag_size=0,
        diag_correct_ratio=0.0,
        val_size=0,
        val_correct_ratio=0.0,
        test_size=0,
        eval_min_per_class=0,
        train_ratio=0.667,
        test_questions_dir=test_dir,
    )
    strategy = PerCategoryPoolStrategy()
    pools = strategy.build(questions, correctness, config)

    assert len(pools.test) == 10

• Step 2: 运行测试验证失败

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types -v

预期：FAIL — assert len(pools.test) == 10 失败（实际 30 题，未过滤）。

• Step 3: 实现 test 池过滤

修改 app/harness/pools.py PerCategoryPoolStrategy.build() 的 Phase 4：

        # Phase 4: test 池（从外部目录加载，无则空）
        test: list[GeneratedQuestion] = []
        if config.test_questions_dir is not None:
            from app.question_gen import load_benchmark

            test = load_benchmark(config.test_questions_dir)
            if config.task_types is not None:
                allowed = set(config.task_types)
                test = [q for q in test if q.task_type in allowed]

• Step 4: 运行测试验证通过

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_no_filter_when_task_types_none -v

预期：PASS

• Step 5: 运行全部 pool 测试确保无回归

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/integration/test_pool_strategy.py -v

预期：全部 PASS

• Step 6: 提交

git add app/harness/pools.py tests/unit/test_harness_pools.py
git commit -m "feat(pools): filter test pool by task_types in PerCategoryPoolStrategy"

---

Task 2: RunConfig 新增 run_holdout_eval + load_config list→tuple 修复 + Runner 条件跳过

Files:

• Modify: app/harness/config.py:147 (字段) + app/harness/config.py:437 (list→tuple)

• Modify: app/harness/runner.py:1371

• Modify: main.py:205

• Test: tests/unit/test_harness_pools.py

• Step 1: 写失败测试 — RunConfig 新字段

在 tests/unit/test_harness_pools.py 文件末尾新增：

class TestRunHoldoutEvalConfig:
    """run_holdout_eval 字段校验。"""

    def test_default_true(self):
        """run_holdout_eval 默认值为 True。"""
        from app.harness.config import RunConfig

        config = RunConfig(
            workspace_dir=Path("/tmp/ws"),
            store_dir=Path("/tmp/store"),
            mode="train",
            concurrency=4,
            max_steps=10,
            skill_mode="auto",
            n_samples=0,
            questions="benchmarks/Video-MME",
            skills_version="v1",
            prompts_version="v1",
            epochs=1,
            diag_size=100,
            diag_correct_ratio=0.5,
            val_size=30,
            val_correct_ratio=0.5,
            edit_budget_start=5,
            edit_budget_end=2,
            batch_size=15,
            min_class_per_batch=2,
            eval_min_per_class=2,
            early_stop_patience=4,
            test_size=30,
            use_slow_momentum=True,
            gate_e_confirm=20.0,
            gate_e_provisional=3.0,
            gate_w_net_min=2,
            gate_delta_min=0.02,
            gate_lambda_dir=-0.642,
            gate_e_rollback=10.0,
            gate_block=8,
            gate_n_max=40,
            gate_p_low=0.05,
            gate_p_high=0.95,
            gate_probe_quota=0.2,
            gate_gamma_decay=0.9,
            gate_cooldown_steps=2,
            gate_guard_err=0.10,
            skill_update_mode="patch",
            appendix_consolidate_threshold=6,
            run_id="test_run",
        )
        assert config.run_holdout_eval is True

    def test_explicit_false(self):
        """run_holdout_eval 可设为 False。"""
        from app.harness.config import RunConfig

        config = RunConfig(
            workspace_dir=Path("/tmp/ws"),
            store_dir=Path("/tmp/store"),
            mode="train",
            concurrency=4,
            max_steps=10,
            skill_mode="auto",
            n_samples=0,
            questions="benchmarks/Video-MME",
            skills_version="v1",
            prompts_version="v1",
            epochs=1,
            diag_size=100,
            diag_correct_ratio=0.5,
            val_size=30,
            val_correct_ratio=0.5,
            edit_budget_start=5,
            edit_budget_end=2,
            batch_size=15,
            min_class_per_batch=2,
            eval_min_per_class=2,
            early_stop_patience=4,
            test_size=30,
            use_slow_momentum=True,
            gate_e_confirm=20.0,
            gate_e_provisional=3.0,
            gate_w_net_min=2,
            gate_delta_min=0.02,
            gate_lambda_dir=-0.642,
            gate_e_rollback=10.0,
            gate_block=8,
            gate_n_max=40,
            gate_p_low=0.05,
            gate_p_high=0.95,
            gate_probe_quota=0.2,
            gate_gamma_decay=0.9,
            gate_cooldown_steps=2,
            gate_guard_err=0.10,
            skill_update_mode="patch",
            appendix_consolidate_threshold=6,
            run_id="test_run",
            run_holdout_eval=False,
        )
        assert config.run_holdout_eval is False

• Step 2: 运行测试验证失败

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v

预期：FAIL — TypeError: __init__() got an unexpected keyword argument 'run_holdout_eval'

• Step 3: RunConfig 新增 run_holdout_eval 字段

在 app/harness/config.py 的有默认值字段区（test_questions 后面）新增：

    test_questions: str = "benchmarks/Video-MME"
    run_holdout_eval: bool = True

• Step 4: 运行测试验证通过

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v

预期：PASS

• Step 5: load_config 修复 YAML list→tuple 转换

RunConfig.task_types 类型为 tuple[str, ...] | None，但 YAML list 加载后不转换。 在 app/harness/config.py load_config() Phase 4（类型转换区）后面新增：

    # Phase 4: 类型转换 — 路径字段转 Path
    for field_name in _PATH_FIELDS:
        if field_name in yaml_data:
            yaml_data[field_name] = Path(yaml_data[field_name])

    # Phase 4b: 类型转换 — task_types list → tuple
    if "task_types" in yaml_data and yaml_data["task_types"] is not None:
        yaml_data["task_types"] = tuple(yaml_data["task_types"])

• Step 6: main.py 新增 CLI 开关

在 main.py _build_parser() 的 --test-questions 后面新增：

    parser.add_argument("--test-questions", type=str, dest="test_questions")
    parser.add_argument(
        "--no-run-holdout-eval",
        action="store_true",
        dest="no_run_holdout_eval",
    )
    return parser

在 main() 中 cli_overrides 构建处（约第 269 行 cli_overrides = ... 之后）处理取反映射：

    cli_overrides = {k: v for k, v in cli_args.items() if k != "config"}
    if cli_overrides.get("no_run_holdout_eval"):
        cli_overrides["run_holdout_eval"] = False
    cli_overrides.pop("no_run_holdout_eval", None)

• Step 7: Runner _slow_update_cycle 条件跳过 holdout

修改 app/harness/runner.py _slow_update_cycle 的 Phase 9（约第 1371 行）：

将：

        await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)

改为：

        if self._config.run_holdout_eval:
            await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)

• Step 8: 运行全部测试确认无回归

conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/unit/test_harness_store.py tests/integration/test_pool_strategy.py -v

预期：全部 PASS

• Step 9: 提交

git add app/harness/config.py app/harness/runner.py main.py tests/unit/test_harness_pools.py
git commit -m "feat(config): add run_holdout_eval toggle and fix YAML task_types list-to-tuple"

---

Task 3: 实验配置文件

Files:

• Create: config/train_action_recognition.yaml

• Step 1: 创建实验 YAML

# config/train_action_recognition.yaml
# Action Recognition 单题型首次训练实验
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md

harness:
  workspace_dir: "workspaces/train-action-recognition"
  store_dir: store
  mode: train
  run_id: train_ar_v1
  concurrency: 24
  max_steps: 40
  skill_mode: auto
  n_samples: 0
  questions: "generated-v2-360"
  skills_version: v1
  prompts_version: v1
  epochs: 3
  # CE-Gate 参数（沿用 default.yaml）
  gate_e_confirm: 20.0
  gate_e_provisional: 3.0
  gate_w_net_min: 2
  gate_delta_min: 0.02
  gate_lambda_dir: -0.642
  gate_e_rollback: 10.0
  gate_block: 8
  gate_n_max: 40
  gate_p_low: 0.05
  gate_p_high: 0.95
  gate_probe_quota: 0.2
  gate_gamma_decay: 0.9
  gate_cooldown_steps: 2
  gate_guard_err: 0.10
  # 进化参数
  edit_budget_start: 5
  edit_budget_end: 2
  skill_update_mode: patch
  appendix_consolidate_threshold: 6
  # 池配置 — per_category 单题型
  pool_split_mode: per_category
  task_types:
    - "Action Recognition"
  train_ratio: 0.667
  test_questions: "benchmarks/Video-MME"
  run_holdout_eval: false
  # mini-batch
  batch_size: 10
  min_class_per_batch: 2
  batch_correct_ratio: 0.5
  momentum_samples: 20
  eval_min_per_class: 2
  early_stop_patience: 4
  test_size: 63
  diag_size: 20
  diag_correct_ratio: 0.5
  val_size: 10
  val_correct_ratio: 0.5
  use_slow_momentum: true

embed:
  backend: "local"
  model_name: "BAAI/bge-base-zh-v1.5"
  embed_dim: 768
  device: "cuda"

• Step 2: 验证 YAML 可正确加载为 RunConfig

conda activate Video-Tree-TRM & python -c "
from app.harness.config import load_config
from pathlib import Path
config = load_config(Path('config/train_action_recognition.yaml'))
assert config.task_types == ('Action Recognition',), f'task_types={config.task_types}'
assert config.run_holdout_eval is False
assert config.pool_split_mode == 'per_category'
print('Config loaded OK')
"

预期：Config loaded OK

• Step 3: 提交

git add config/train_action_recognition.yaml
git commit -m "config: add train_action_recognition experiment YAML"

---

Task 4: 前置脚本 — baseline infer + seed 创建 + 训练

Files:

• Create: scripts/train_action_recognition.sh

• Step 1: 创建脚本

#!/usr/bin/env bash
# Action Recognition 单题型训练实验
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
#
# 三阶段:
#   Phase 0: baseline infer (v2-360 Action Recognition 30 题)
#   Phase 1: create seed (v2ar-baseline)
#   Phase 2: train (3 epochs, per_category)
#
# 用法:
#   CUDA_VISIBLE_DEVICES=0 bash scripts/train_action_recognition.sh
#   MODE=mock bash scripts/train_action_recognition.sh   # 跳过 Phase 0/1
set -euo pipefail

cd "$(dirname "$0")/.."

CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
export CUDA_VISIBLE_DEVICES

export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTHONUNBUFFERED=1

set -a
source .env
set +a

PYTHON="$(conda run -n Video-Tree-TRM which python)"

# ── Phase 0: Baseline infer ──
if [[ "${MODE:-}" != "mock" ]]; then
    echo "=== Phase 0: Baseline infer (v2-360 Action Recognition 30 题) ==="
    "${PYTHON}" main.py \
        --config config/default.yaml \
        --workspace-dir workspaces/default \
        --store-dir store \
        --mode infer \
        --concurrency 24 \
        --max-steps 40 \
        --skill-mode auto \
        --n-samples 0 \
        --questions "generated-v2-360" \
        --skills-version v1 \
        --prompts-version v1 \
        --run-id v2ar_baseline \
        --task-types "Action Recognition"
fi

# ── Phase 1: Create seed ──
if [[ "${MODE:-}" != "mock" && ! -d "store/seeds/v2ar-baseline" ]]; then
    echo "=== Phase 1: Create seed v2ar-baseline ==="
    "${PYTHON}" -c "
from pathlib import Path
from app.harness.store import extract_run_db, init_seed
import tempfile

tmp = Path(tempfile.mkdtemp()) / 'baseline.db'
extract_run_db(
    Path('workspaces/default/harness.db'),
    tmp,
    'infer_v2ar_baseline',
)
init_seed(
    store_dir=Path('store'),
    name='v2ar-baseline',
    skills_dir=Path('store/skills/v1'),
    prompts_dir=Path('store/prompts/v1'),
    baseline_db=tmp,
    baseline_run_id='infer_v2ar_baseline',
    parent=None,
    description='v2-360 Action Recognition 30 题 baseline (skills/v1)',
)
tmp.unlink()
print('Seed created: store/seeds/v2ar-baseline/')
"
elif [[ -d "store/seeds/v2ar-baseline" ]]; then
    echo "=== Phase 1: Seed v2ar-baseline 已存在，跳过 ==="
fi

# ── Phase 2: Train ──
echo "=== Phase 2: Train (3 epochs, Action Recognition) ==="
"${PYTHON}" main.py \
    --config config/train_action_recognition.yaml \
    --fresh \
    --seed v2ar-baseline

echo "=== 训练完成 ==="
echo "结果查看:"
echo "  cat workspaces/train-action-recognition/analyses/final_test_eval.json"
echo "  sqlite3 workspaces/train-action-recognition/harness.db 'SELECT * FROM dual_metric'"

• Step 2: 设置可执行权限并验证语法

chmod +x scripts/train_action_recognition.sh
bash -n scripts/train_action_recognition.sh

预期：无语法错误

• Step 3: 提交

git add scripts/train_action_recognition.sh
git commit -m "scripts: add train_action_recognition experiment script"

---

Task 5: lint 检查 + 全量回归测试

• Step 1: Ruff 格式化与检查

conda activate Video-Tree-TRM & ruff format app/ core/ && ruff check app/ core/ --fix

预期：无错误

• Step 2: 全量测试

conda activate Video-Tree-TRM & pytest tests/unit/ tests/integration/ -v --tb=short

预期：全部 PASS

• Step 3: 最终提交（如有 lint 修复）

git add -A && git commit -m "chore: lint and format training experiment changes"

---

核心算法保真校验

本计划不涉及核心算法迁移。修改仅限于：

• PerCategoryPoolStrategy.build() 新增 3 行 test 池过滤（不改 train/val 切分逻辑）
• RunConfig 新增 1 个 bool 字段
• load_config 新增 2 行 list→tuple 转换
• _slow_update_cycle 新增 1 行 if 条件（不改 holdout 内部逻辑）

保真校验不适用。