feat(harness): add Action Recognition training experiment
- PerCategoryPoolStrategy: filter test pool by task_types - RunConfig: add run_holdout_eval toggle (default true) - load_config: fix YAML task_types list-to-tuple conversion - Runner: conditionally skip _holdout_four_way when disabled - CLI: add --no-run-holdout-eval flag - New config/train_action_recognition.yaml (3 epochs, per_category) - New scripts/train_action_recognition.sh (baseline + seed + train)
This commit is contained in:
@@ -0,0 +1,661 @@
|
||||
# Action Recognition 单题型首次训练实验 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 用 Action Recognition 单题型端到端验证训练管线，包含 3 处代码修改、1 套实验配置、1 个前置脚本。
|
||||
|
||||
**Architecture:** 修改 `PerCategoryPoolStrategy.build()` 使 test 池受 `task_types` 过滤；RunConfig 新增 `run_holdout_eval` 开关控制 epoch 内四向 held-out；`load_config` 修复 YAML list→tuple 转换；新建实验 YAML 和 sh 脚本驱动训练。
|
||||
|
||||
**Tech Stack:** Python 3.11, pytest, YAML, bash
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 清理 v2-360 目录中的 backup 文件
|
||||
|
||||
**Files:**
|
||||
- Modify: `store/questions/generated-v2-360/`（重命名文件）
|
||||
|
||||
`load_benchmark()` 会加载目录下所有 `*.json`，`accepted_questions_backup_220.json` 含 18 道重复 AR 题会干扰训练。
|
||||
|
||||
- [ ] **Step 1: 重命名 backup 文件使其不被 load_benchmark 加载**
|
||||
|
||||
```bash
|
||||
mv store/questions/generated-v2-360/accepted_questions_backup_220.json \
|
||||
store/questions/generated-v2-360/accepted_questions_backup_220.json.bak
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证 load_benchmark 只加载 180 题**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & python -c "
|
||||
from pathlib import Path
|
||||
from app.question_gen import load_benchmark
|
||||
qs = load_benchmark(Path('store/questions/generated-v2-360'))
|
||||
print(f'Total: {len(qs)}')
|
||||
ar = [q for q in qs if q.task_type == 'Action Recognition']
|
||||
print(f'Action Recognition: {len(ar)}')
|
||||
assert len(qs) == 180, f'Expected 180, got {len(qs)}'
|
||||
assert len(ar) == 30, f'Expected 30 AR, got {len(ar)}'
|
||||
print('OK')
|
||||
"
|
||||
```
|
||||
|
||||
预期：Total: 180, Action Recognition: 30, OK
|
||||
|
||||
- [ ] **Step 3: 提交**
|
||||
|
||||
```bash
|
||||
git add -A store/questions/generated-v2-360/
|
||||
git commit -m "chore: rename v2-360 backup JSON to .bak to exclude from load_benchmark"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 1: PerCategoryPoolStrategy test 池 task_types 过滤
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/harness/pools.py:600-606`
|
||||
- Test: `tests/unit/test_harness_pools.py`
|
||||
|
||||
- [ ] **Step 1: 写失败测试 — test 池按 task_types 过滤**
|
||||
|
||||
在 `tests/unit/test_harness_pools.py` 的 `TestPerCategoryPoolStrategy` 类末尾新增。
|
||||
注意：`load_benchmark` 要求每个 JSON 文件内容为**题目数组**（`[{...}]`），不是单个 dict。
|
||||
|
||||
```python
|
||||
def test_per_category_test_pool_filtered_by_task_types(self, tmp_path: Path):
|
||||
"""test_questions_dir 含多题型时，test 池只保留 task_types 指定的题型。"""
|
||||
test_dir = tmp_path / "test_questions"
|
||||
test_dir.mkdir()
|
||||
for tt in ("Action Recognition", "Object Reasoning", "Counting Problem"):
|
||||
items = []
|
||||
for i in range(10):
|
||||
qid = f"{tt.replace(' ', '_')}_{i:03d}"
|
||||
items.append({
|
||||
"question_id": qid,
|
||||
"video_id": "v1",
|
||||
"task_type": tt,
|
||||
"question": f"Q {qid}?",
|
||||
"options": ["A. a", "B. b", "C. c", "D. d"],
|
||||
"answer": "A",
|
||||
})
|
||||
slug = tt.lower().replace(" ", "_")
|
||||
(test_dir / f"{slug}.json").write_text(
|
||||
json.dumps(items, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
|
||||
questions = [_make_question(f"ar_{i:03d}", "Action Recognition") for i in range(30)]
|
||||
correctness = {q.question_id: (i < 20) for i, q in enumerate(questions)}
|
||||
|
||||
config = PoolConfig(
|
||||
task_types=("Action Recognition",),
|
||||
seed=42,
|
||||
baseline_run_id="bl",
|
||||
diag_size=0,
|
||||
diag_correct_ratio=0.0,
|
||||
val_size=0,
|
||||
val_correct_ratio=0.0,
|
||||
test_size=0,
|
||||
eval_min_per_class=0,
|
||||
train_ratio=20 / 30,
|
||||
test_questions_dir=test_dir,
|
||||
)
|
||||
strategy = PerCategoryPoolStrategy()
|
||||
pools = strategy.build(questions, correctness, config)
|
||||
|
||||
assert len(pools.test) == 10
|
||||
assert all(q.task_type == "Action Recognition" for q in pools.test)
|
||||
|
||||
def test_per_category_test_pool_no_filter_when_task_types_none(self, tmp_path: Path):
|
||||
"""task_types=None 时 test 池不过滤，保留全部题型。"""
|
||||
test_dir = tmp_path / "test_questions"
|
||||
test_dir.mkdir()
|
||||
for tt in ("Action Recognition", "Object Reasoning"):
|
||||
items = []
|
||||
for i in range(5):
|
||||
qid = f"{tt.replace(' ', '_')}_{i:03d}"
|
||||
items.append({
|
||||
"question_id": qid,
|
||||
"video_id": "v1",
|
||||
"task_type": tt,
|
||||
"question": f"Q {qid}?",
|
||||
"options": ["A. a", "B. b", "C. c", "D. d"],
|
||||
"answer": "A",
|
||||
})
|
||||
slug = tt.lower().replace(" ", "_")
|
||||
(test_dir / f"{slug}.json").write_text(
|
||||
json.dumps(items, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
|
||||
questions = [_make_question(f"q_{i:03d}", "Action Recognition") for i in range(10)]
|
||||
correctness = {q.question_id: True for q in questions}
|
||||
|
||||
config = PoolConfig(
|
||||
task_types=None,
|
||||
seed=42,
|
||||
baseline_run_id="bl",
|
||||
diag_size=0,
|
||||
diag_correct_ratio=0.0,
|
||||
val_size=0,
|
||||
val_correct_ratio=0.0,
|
||||
test_size=0,
|
||||
eval_min_per_class=0,
|
||||
train_ratio=0.667,
|
||||
test_questions_dir=test_dir,
|
||||
)
|
||||
strategy = PerCategoryPoolStrategy()
|
||||
pools = strategy.build(questions, correctness, config)
|
||||
|
||||
assert len(pools.test) == 10
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 运行测试验证失败**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types -v
|
||||
```
|
||||
|
||||
预期：FAIL — `assert len(pools.test) == 10` 失败（实际 30 题，未过滤）。
|
||||
|
||||
- [ ] **Step 3: 实现 test 池过滤**
|
||||
|
||||
修改 `app/harness/pools.py` `PerCategoryPoolStrategy.build()` 的 Phase 4：
|
||||
|
||||
```python
|
||||
# Phase 4: test 池（从外部目录加载，无则空）
|
||||
test: list[GeneratedQuestion] = []
|
||||
if config.test_questions_dir is not None:
|
||||
from app.question_gen import load_benchmark
|
||||
|
||||
test = load_benchmark(config.test_questions_dir)
|
||||
if config.task_types is not None:
|
||||
allowed = set(config.task_types)
|
||||
test = [q for q in test if q.task_type in allowed]
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 运行测试验证通过**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_filtered_by_task_types tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy::test_per_category_test_pool_no_filter_when_task_types_none -v
|
||||
```
|
||||
|
||||
预期：PASS
|
||||
|
||||
- [ ] **Step 5: 运行全部 pool 测试确保无回归**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/integration/test_pool_strategy.py -v
|
||||
```
|
||||
|
||||
预期：全部 PASS
|
||||
|
||||
- [ ] **Step 6: 提交**
|
||||
|
||||
```bash
|
||||
git add app/harness/pools.py tests/unit/test_harness_pools.py
|
||||
git commit -m "feat(pools): filter test pool by task_types in PerCategoryPoolStrategy"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: RunConfig 新增 run_holdout_eval + load_config list→tuple 修复 + Runner 条件跳过
|
||||
|
||||
**Files:**
|
||||
- Modify: `app/harness/config.py:147` (字段) + `app/harness/config.py:437` (list→tuple)
|
||||
- Modify: `app/harness/runner.py:1371`
|
||||
- Modify: `main.py:205`
|
||||
- Test: `tests/unit/test_harness_pools.py`
|
||||
|
||||
- [ ] **Step 1: 写失败测试 — RunConfig 新字段**
|
||||
|
||||
在 `tests/unit/test_harness_pools.py` 文件末尾新增：
|
||||
|
||||
```python
|
||||
class TestRunHoldoutEvalConfig:
|
||||
"""run_holdout_eval 字段校验。"""
|
||||
|
||||
def test_default_true(self):
|
||||
"""run_holdout_eval 默认值为 True。"""
|
||||
from app.harness.config import RunConfig
|
||||
|
||||
config = RunConfig(
|
||||
workspace_dir=Path("/tmp/ws"),
|
||||
store_dir=Path("/tmp/store"),
|
||||
mode="train",
|
||||
concurrency=4,
|
||||
max_steps=10,
|
||||
skill_mode="auto",
|
||||
n_samples=0,
|
||||
questions="benchmarks/Video-MME",
|
||||
skills_version="v1",
|
||||
prompts_version="v1",
|
||||
epochs=1,
|
||||
diag_size=100,
|
||||
diag_correct_ratio=0.5,
|
||||
val_size=30,
|
||||
val_correct_ratio=0.5,
|
||||
edit_budget_start=5,
|
||||
edit_budget_end=2,
|
||||
batch_size=15,
|
||||
min_class_per_batch=2,
|
||||
eval_min_per_class=2,
|
||||
early_stop_patience=4,
|
||||
test_size=30,
|
||||
use_slow_momentum=True,
|
||||
gate_e_confirm=20.0,
|
||||
gate_e_provisional=3.0,
|
||||
gate_w_net_min=2,
|
||||
gate_delta_min=0.02,
|
||||
gate_lambda_dir=-0.642,
|
||||
gate_e_rollback=10.0,
|
||||
gate_block=8,
|
||||
gate_n_max=40,
|
||||
gate_p_low=0.05,
|
||||
gate_p_high=0.95,
|
||||
gate_probe_quota=0.2,
|
||||
gate_gamma_decay=0.9,
|
||||
gate_cooldown_steps=2,
|
||||
gate_guard_err=0.10,
|
||||
skill_update_mode="patch",
|
||||
appendix_consolidate_threshold=6,
|
||||
run_id="test_run",
|
||||
)
|
||||
assert config.run_holdout_eval is True
|
||||
|
||||
def test_explicit_false(self):
|
||||
"""run_holdout_eval 可设为 False。"""
|
||||
from app.harness.config import RunConfig
|
||||
|
||||
config = RunConfig(
|
||||
workspace_dir=Path("/tmp/ws"),
|
||||
store_dir=Path("/tmp/store"),
|
||||
mode="train",
|
||||
concurrency=4,
|
||||
max_steps=10,
|
||||
skill_mode="auto",
|
||||
n_samples=0,
|
||||
questions="benchmarks/Video-MME",
|
||||
skills_version="v1",
|
||||
prompts_version="v1",
|
||||
epochs=1,
|
||||
diag_size=100,
|
||||
diag_correct_ratio=0.5,
|
||||
val_size=30,
|
||||
val_correct_ratio=0.5,
|
||||
edit_budget_start=5,
|
||||
edit_budget_end=2,
|
||||
batch_size=15,
|
||||
min_class_per_batch=2,
|
||||
eval_min_per_class=2,
|
||||
early_stop_patience=4,
|
||||
test_size=30,
|
||||
use_slow_momentum=True,
|
||||
gate_e_confirm=20.0,
|
||||
gate_e_provisional=3.0,
|
||||
gate_w_net_min=2,
|
||||
gate_delta_min=0.02,
|
||||
gate_lambda_dir=-0.642,
|
||||
gate_e_rollback=10.0,
|
||||
gate_block=8,
|
||||
gate_n_max=40,
|
||||
gate_p_low=0.05,
|
||||
gate_p_high=0.95,
|
||||
gate_probe_quota=0.2,
|
||||
gate_gamma_decay=0.9,
|
||||
gate_cooldown_steps=2,
|
||||
gate_guard_err=0.10,
|
||||
skill_update_mode="patch",
|
||||
appendix_consolidate_threshold=6,
|
||||
run_id="test_run",
|
||||
run_holdout_eval=False,
|
||||
)
|
||||
assert config.run_holdout_eval is False
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 运行测试验证失败**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v
|
||||
```
|
||||
|
||||
预期：FAIL — `TypeError: __init__() got an unexpected keyword argument 'run_holdout_eval'`
|
||||
|
||||
- [ ] **Step 3: RunConfig 新增 run_holdout_eval 字段**
|
||||
|
||||
在 `app/harness/config.py` 的有默认值字段区（`test_questions` 后面）新增：
|
||||
|
||||
```python
|
||||
test_questions: str = "benchmarks/Video-MME"
|
||||
run_holdout_eval: bool = True
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 运行测试验证通过**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py::TestRunHoldoutEvalConfig -v
|
||||
```
|
||||
|
||||
预期：PASS
|
||||
|
||||
- [ ] **Step 5: load_config 修复 YAML list→tuple 转换**
|
||||
|
||||
`RunConfig.task_types` 类型为 `tuple[str, ...] | None`，但 YAML list 加载后不转换。
|
||||
在 `app/harness/config.py` `load_config()` Phase 4（类型转换区）后面新增：
|
||||
|
||||
```python
|
||||
# Phase 4: 类型转换 — 路径字段转 Path
|
||||
for field_name in _PATH_FIELDS:
|
||||
if field_name in yaml_data:
|
||||
yaml_data[field_name] = Path(yaml_data[field_name])
|
||||
|
||||
# Phase 4b: 类型转换 — task_types list → tuple
|
||||
if "task_types" in yaml_data and yaml_data["task_types"] is not None:
|
||||
yaml_data["task_types"] = tuple(yaml_data["task_types"])
|
||||
```
|
||||
|
||||
- [ ] **Step 6: main.py 新增 CLI 开关**
|
||||
|
||||
在 `main.py` `_build_parser()` 的 `--test-questions` 后面新增：
|
||||
|
||||
```python
|
||||
parser.add_argument("--test-questions", type=str, dest="test_questions")
|
||||
parser.add_argument(
|
||||
"--no-run-holdout-eval",
|
||||
action="store_true",
|
||||
dest="no_run_holdout_eval",
|
||||
)
|
||||
return parser
|
||||
```
|
||||
|
||||
在 `main()` 中 `cli_overrides` 构建处（约第 269 行 `cli_overrides = ...` 之后）处理取反映射：
|
||||
|
||||
```python
|
||||
cli_overrides = {k: v for k, v in cli_args.items() if k != "config"}
|
||||
if cli_overrides.get("no_run_holdout_eval"):
|
||||
cli_overrides["run_holdout_eval"] = False
|
||||
cli_overrides.pop("no_run_holdout_eval", None)
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Runner `_slow_update_cycle` 条件跳过 holdout**
|
||||
|
||||
修改 `app/harness/runner.py` `_slow_update_cycle` 的 Phase 9（约第 1371 行）：
|
||||
|
||||
将：
|
||||
```python
|
||||
await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)
|
||||
```
|
||||
|
||||
改为：
|
||||
```python
|
||||
if self._config.run_holdout_eval:
|
||||
await self._holdout_four_way(epoch, pools, state, eval_skills_version, eval_prompts_version)
|
||||
```
|
||||
|
||||
- [ ] **Step 8: 运行全部测试确认无回归**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/test_harness_pools.py tests/unit/test_harness_store.py tests/integration/test_pool_strategy.py -v
|
||||
```
|
||||
|
||||
预期：全部 PASS
|
||||
|
||||
- [ ] **Step 9: 提交**
|
||||
|
||||
```bash
|
||||
git add app/harness/config.py app/harness/runner.py main.py tests/unit/test_harness_pools.py
|
||||
git commit -m "feat(config): add run_holdout_eval toggle and fix YAML task_types list-to-tuple"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: 实验配置文件
|
||||
|
||||
**Files:**
|
||||
- Create: `config/train_action_recognition.yaml`
|
||||
|
||||
- [ ] **Step 1: 创建实验 YAML**
|
||||
|
||||
```yaml
|
||||
# config/train_action_recognition.yaml
|
||||
# Action Recognition 单题型首次训练实验
|
||||
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
|
||||
|
||||
harness:
|
||||
workspace_dir: "workspaces/train-action-recognition"
|
||||
store_dir: store
|
||||
mode: train
|
||||
run_id: train_ar_v1
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "generated-v2-360"
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 3
|
||||
# CE-Gate 参数（沿用 default.yaml）
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_block: 8
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
# 进化参数
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
# 池配置 — per_category 单题型
|
||||
pool_split_mode: per_category
|
||||
task_types:
|
||||
- "Action Recognition"
|
||||
train_ratio: 0.667
|
||||
test_questions: "benchmarks/Video-MME"
|
||||
run_holdout_eval: false
|
||||
# mini-batch
|
||||
batch_size: 10
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
early_stop_patience: 4
|
||||
test_size: 63
|
||||
diag_size: 20
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 10
|
||||
val_correct_ratio: 0.5
|
||||
use_slow_momentum: true
|
||||
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证 YAML 可正确加载为 RunConfig**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & python -c "
|
||||
from app.harness.config import load_config
|
||||
from pathlib import Path
|
||||
config = load_config(Path('config/train_action_recognition.yaml'))
|
||||
assert config.task_types == ('Action Recognition',), f'task_types={config.task_types}'
|
||||
assert config.run_holdout_eval is False
|
||||
assert config.pool_split_mode == 'per_category'
|
||||
print('Config loaded OK')
|
||||
"
|
||||
```
|
||||
|
||||
预期：Config loaded OK
|
||||
|
||||
- [ ] **Step 3: 提交**
|
||||
|
||||
```bash
|
||||
git add config/train_action_recognition.yaml
|
||||
git commit -m "config: add train_action_recognition experiment YAML"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: 前置脚本 — baseline infer + seed 创建 + 训练
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/train_action_recognition.sh`
|
||||
|
||||
- [ ] **Step 1: 创建脚本**
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Action Recognition 单题型训练实验
|
||||
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
|
||||
#
|
||||
# 三阶段:
|
||||
# Phase 0: baseline infer (v2-360 Action Recognition 30 题)
|
||||
# Phase 1: create seed (v2ar-baseline)
|
||||
# Phase 2: train (3 epochs, per_category)
|
||||
#
|
||||
# 用法:
|
||||
# CUDA_VISIBLE_DEVICES=0 bash scripts/train_action_recognition.sh
|
||||
# MODE=mock bash scripts/train_action_recognition.sh # 跳过 Phase 0/1
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
|
||||
export CUDA_VISIBLE_DEVICES
|
||||
|
||||
export HF_HUB_OFFLINE=1
|
||||
export TRANSFORMERS_OFFLINE=1
|
||||
export PYTHONUNBUFFERED=1
|
||||
|
||||
set -a
|
||||
source .env
|
||||
set +a
|
||||
|
||||
PYTHON="$(conda run -n Video-Tree-TRM which python)"
|
||||
|
||||
# ── Phase 0: Baseline infer ──
|
||||
if [[ "${MODE:-}" != "mock" ]]; then
|
||||
echo "=== Phase 0: Baseline infer (v2-360 Action Recognition 30 题) ==="
|
||||
"${PYTHON}" main.py \
|
||||
--config config/default.yaml \
|
||||
--workspace-dir workspaces/default \
|
||||
--store-dir store \
|
||||
--mode infer \
|
||||
--concurrency 24 \
|
||||
--max-steps 40 \
|
||||
--skill-mode auto \
|
||||
--n-samples 0 \
|
||||
--questions "generated-v2-360" \
|
||||
--skills-version v1 \
|
||||
--prompts-version v1 \
|
||||
--run-id v2ar_baseline \
|
||||
--task-types "Action Recognition"
|
||||
fi
|
||||
|
||||
# ── Phase 1: Create seed ──
|
||||
if [[ "${MODE:-}" != "mock" && ! -d "store/seeds/v2ar-baseline" ]]; then
|
||||
echo "=== Phase 1: Create seed v2ar-baseline ==="
|
||||
"${PYTHON}" -c "
|
||||
from pathlib import Path
|
||||
from app.harness.store import extract_run_db, init_seed
|
||||
import tempfile
|
||||
|
||||
tmp = Path(tempfile.mkdtemp()) / 'baseline.db'
|
||||
extract_run_db(
|
||||
Path('workspaces/default/harness.db'),
|
||||
tmp,
|
||||
'infer_v2ar_baseline',
|
||||
)
|
||||
init_seed(
|
||||
store_dir=Path('store'),
|
||||
name='v2ar-baseline',
|
||||
skills_dir=Path('store/skills/v1'),
|
||||
prompts_dir=Path('store/prompts/v1'),
|
||||
baseline_db=tmp,
|
||||
baseline_run_id='infer_v2ar_baseline',
|
||||
parent=None,
|
||||
description='v2-360 Action Recognition 30 题 baseline (skills/v1)',
|
||||
)
|
||||
tmp.unlink()
|
||||
print('Seed created: store/seeds/v2ar-baseline/')
|
||||
"
|
||||
elif [[ -d "store/seeds/v2ar-baseline" ]]; then
|
||||
echo "=== Phase 1: Seed v2ar-baseline 已存在，跳过 ==="
|
||||
fi
|
||||
|
||||
# ── Phase 2: Train ──
|
||||
echo "=== Phase 2: Train (3 epochs, Action Recognition) ==="
|
||||
"${PYTHON}" main.py \
|
||||
--config config/train_action_recognition.yaml \
|
||||
--fresh \
|
||||
--seed v2ar-baseline
|
||||
|
||||
echo "=== 训练完成 ==="
|
||||
echo "结果查看:"
|
||||
echo " cat workspaces/train-action-recognition/analyses/final_test_eval.json"
|
||||
echo " sqlite3 workspaces/train-action-recognition/harness.db 'SELECT * FROM dual_metric'"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 设置可执行权限并验证语法**
|
||||
|
||||
```bash
|
||||
chmod +x scripts/train_action_recognition.sh
|
||||
bash -n scripts/train_action_recognition.sh
|
||||
```
|
||||
|
||||
预期：无语法错误
|
||||
|
||||
- [ ] **Step 3: 提交**
|
||||
|
||||
```bash
|
||||
git add scripts/train_action_recognition.sh
|
||||
git commit -m "scripts: add train_action_recognition experiment script"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: lint 检查 + 全量回归测试
|
||||
|
||||
- [ ] **Step 1: Ruff 格式化与检查**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & ruff format app/ core/ && ruff check app/ core/ --fix
|
||||
```
|
||||
|
||||
预期：无错误
|
||||
|
||||
- [ ] **Step 2: 全量测试**
|
||||
|
||||
```bash
|
||||
conda activate Video-Tree-TRM & pytest tests/unit/ tests/integration/ -v --tb=short
|
||||
```
|
||||
|
||||
预期：全部 PASS
|
||||
|
||||
- [ ] **Step 3: 最终提交（如有 lint 修复）**
|
||||
|
||||
```bash
|
||||
git add -A && git commit -m "chore: lint and format training experiment changes"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 核心算法保真校验
|
||||
|
||||
本计划不涉及核心算法迁移。修改仅限于：
|
||||
- `PerCategoryPoolStrategy.build()` 新增 3 行 test 池过滤（不改 train/val 切分逻辑）
|
||||
- `RunConfig` 新增 1 个 bool 字段
|
||||
- `load_config` 新增 2 行 list→tuple 转换
|
||||
- `_slow_update_cycle` 新增 1 行 `if` 条件（不改 holdout 内部逻辑）
|
||||
|
||||
保真校验不适用。
|
||||
Reference in New Issue
Block a user