diff --git a/research-wiki/designs/2026-07-12-per-category-pool-strategy-design.md b/research-wiki/designs/2026-07-12-per-category-pool-strategy-design.md new file mode 100644 index 0000000..3af272a --- /dev/null +++ b/research-wiki/designs/2026-07-12-per-category-pool-strategy-design.md @@ -0,0 +1,267 @@ +# Per-Category Pool Strategy 设计 + +> **状态**: 已批准 +> **日期**: 2026-07-12 +> **关联**: [app-harness-design](2026-07-07-app-harness-design.md)、[question-gen-v2-design](2026-07-11-question-gen-v2-design.md) + +## 1. 动机 + +v2 题目生成管线为 12 个 task_type 各生成 30 题（共 360 题）。需要支持： + +| 需求 | 说明 | +|------|------| +| 按类增量 baseline | 类别生成完成后立即可跑 baseline inference，无需等全量完成 | +| Per-category 分层划分 | 每类 30 题按 correctness 2:1 分为 20 train / 10 val | +| 按类训练 | `--task-types` 控制训练作用域，快速验证单类别效果 | +| 外部 test 池 | test 用 Video-MME 900 道真题中同类题目，不从生成题中抽取 | +| 全局训练兼容 | 全类别联合训练时 system/tool 慢进化看全量 val 信号 | + +## 2. 方案选型 + +| 方案 | 描述 | 取舍 | +|------|------|------| +| A. 最小改动 | 在 `build_pools` 内加 if 分支 | 改动小但两条路径耦合 | +| B. 替换为纯 per-category | 删除全局模式 | 简洁但丢失 Video-MME 900 题训练能力 | +| **C. Pool 工厂模式** ✓ | `PoolStrategy` Protocol + 具体策略 | 符合 Clean Architecture DIP/OCP；策略可独立测试替换 | + +**选择 C**：池分割是真正易变的接缝（已有两种策略，未来可扩展），满足 CLAUDE.md 抽象引入条件。 + +## 3. 核心抽象 + +### 3.1 PoolStrategy Protocol + +```python +# app/ports.py 新增（应用层端口，非 core 层——因为返回值 Pools 定义在 app/harness/） + +class PoolStrategy(Protocol): + """池构建策略端口。""" + + def build( + self, + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> Pools: ... + + def build_incremental( + self, + new_task_types: list[str], + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> dict[str, dict[str, list[str]]]: + """增量构建新类别的 train/val 划分。 + + 返回 {task_type: {"train": [qid, ...], "val": [qid, ...]}}。 + 仅 PerCategoryPoolStrategy 有实质实现；GlobalPoolStrategy + 不支持增量，调用时抛出 NotImplementedError。 + """ + ... +``` + +> **层级决策**：`PoolStrategy` 放在 `app/ports.py` 而非 `core/protocols.py`，因为返回类型 `Pools` 定义在 `app/harness/pools.py`。若放入 core 会导致 core → app 的逆向依赖，违反 Clean Architecture 四层分层。 +> +> **注**：`PoolConfig` 的两组字段（Global 系列 / PerCategory 系列）由两个策略各取所需，未使用的字段被忽略。这避免了为每个策略定义专属 config 子类的过度抽象。 + +### 3.2 PoolConfig + +```python +# core/types.py 新增 + +@dataclass(frozen=True) +class PoolConfig: + task_types: tuple[str, ...] | None # None = 全部类别 + seed: int # 随机种子 + baseline_run_id: str # 基线 run 标识（用于读 correctness、写入 pools 指纹） + # GlobalStrategy 使用 + diag_size: int + diag_correct_ratio: float + val_size: int + val_correct_ratio: float + test_size: int + eval_min_per_class: int + # PerCategoryStrategy 使用 + train_ratio: float # train/(train+val)，默认 2/3 + test_questions_dir: Path | None # 外部 test 题源路径（与 RunConfig.test_questions 对应） +``` + +### 3.3 两个具体策略 + +**GlobalPoolStrategy**：封装现有 `build_pools` 逻辑，test → val → diag 全局三分。 + +**PerCategoryPoolStrategy**： + +``` +对每个 task_type (共 N 题，目标 train=20, val=10): + correct 题 (Nc 个) → shuffle(seed) + train_correct = floor(Nc * 20/N) # 按总量 20/10 比例分配 + val_correct = Nc - train_correct + wrong 题 (Nw 个) → shuffle(seed) + train_wrong = 20 - train_correct # 确保 train 总量恰好 20 + val_wrong = Nw - train_wrong + assert train_correct + train_wrong == 20 + assert val_correct + val_wrong == 10 + +diagnosis = 所有选中类别的 train 合并 +validation = 所有选中类别的 val 合并 +test = load_benchmark(test_questions_dir) 按 task_types 过滤 +``` + +**边界场景处理**： + +| 场景 | 行为 | +|------|------| +| 某类别全部 correct（0 wrong） | 退化为非分层 random 20/10，记录 WARNING | +| 某类别全部 wrong（0 correct） | 同上，退化为非分层 random 20/10 | +| 某类别 correctness 不完整（部分 qid 无 baseline） | fail-fast，列出缺失 qid | +| Video-MME 中某 task_type 无题 | test 池该类别为空，记录 WARNING | + +## 4. RunConfig 变更 + +| 新增字段 | 类型 | 默认值 | 归属 | +|---------|------|--------|------| +| `task_types` | `tuple[str, ...] \| None` | `None` | CLI 临时覆盖 | +| `pool_split_mode` | `Literal["global", "per_category"]` | `"global"` | 科研 YAML | +| `train_ratio` | `float` | `0.667` | 科研 YAML | +| `test_questions` | `str` | `"benchmarks/Video-MME"` | 科研 YAML | + +`default.yaml` 新增： + +```yaml +harness: + pool_split_mode: global + train_ratio: 0.667 + test_questions: "benchmarks/Video-MME" +``` + +## 5. 增量 Baseline Inference + +### 5.1 流程 + +```bash +# 类别 A、B 完成 → 跑 baseline +python main.py harness --mode infer \ + --run-id baseline_v2 --questions generated-v2-360 \ + --task-types "Object Recognition" "Scene Understanding" + +# 类别 C 完成 → 追加到同一 run_id +python main.py harness --mode infer \ + --run-id baseline_v2 --questions generated-v2-360 \ + --task-types "Action Reasoning" + +# promote 为 seed +python main.py harness --mode promote \ + --run-id baseline_v2 --seed baseline_v2_seed + +# 按类训练 +python main.py harness --mode train --fresh \ + --seed baseline_v2_seed --run-id train_obj_recog_01 \ + --task-types "Object Recognition" --pool-split-mode per_category +``` + +### 5.2 增量写入规则 + +| 表 | 唯一键 | 同 run_id 多次 infer 行为 | +|---|--------|-------------------------| +| `_runs` | `run_id` | upsert：更新 `updated_at`，保留原始 `created_at` | +| `predictions` | `(run_id, question_id)` | INSERT；主键冲突报错（防重复推理） | +| `traces` | `trace_id` | 正常 INSERT（UUID 天然唯一） | + +**重跑同类别**：若需重新推理某类别，先 DELETE 该类别的 predictions（`WHERE run_id=? AND question_id IN (?)`），再重新 infer。 + +**promote 读取规则**：`promote_to_seed` 读取指定 `run_id` 下全部 predictions，不区分追加批次。promote 前应确保所有目标类别均已完成 baseline。 + +## 6. 池冻结格式 + +### 6.1 per_category 格式 + +```json +{ + "split_mode": "per_category", + "train_ratio": 0.667, + "baseline_run_id": "baseline_v2", + "categories": { + "Object Recognition": { + "train": ["qid_1", "qid_2", "..."], + "val": ["qid_21", "..."] + } + }, + "test_source": "benchmarks/Video-MME", + "questions": [...] +} +``` + +### 6.2 增量更新 + +```python +def build_or_load_pools( + config: RunConfig, + strategy: PoolStrategy, + db_path: Path, +) -> Pools: + """构建或加载三池。 + + 参数: + config: 运行配置（含 task_types, pool_split_mode, baseline_run_id 等）。 + strategy: 池构建策略实例。 + db_path: harness.db 路径（用于读取 baseline correctness）。 + + 流程: + 1. 若 pools.json 存在 → 加载 + a. 若 config.task_types 中有类别不在已冻结的 categories → 增量构建并追加 + b. 若 pools.json 的 seed/train_ratio/baseline_run_id 与当前 config 不一致 → 报错 + 2. 若不存在 → 从 DB 加载 correctness → strategy.build() → 冻结 + 3. 按 config.task_types 从 categories 中组装 Pools 返回 + """ +``` + +**冻结一致性校验**：加载已有 pools.json 时，若 `split_mode`、`seed`、`train_ratio`、`baseline_run_id` 与当前 config 不一致，直接报错并提示删除 pools.json 重建。已冻结类别的划分不可变，新类别可追加。 + +### 6.3 向后兼容 + +旧格式 `pools.json`（无 `split_mode` 字段）自动识别为 global 模式。 + +## 7. task_types 作用域贯穿 + +策略在池构建阶段完成所有过滤，下游无需感知 task_types： + +| 阶段 | 行为 | +|------|------| +| 池构建 | strategy 只处理指定类别 | +| mini-batch | 消费 scoped 后的 diagnosis 池，无改动 | +| per-skill gate | 池中只有指定类别的题，自然 scoped | +| 慢进化 | 用 scoped 后的 val 池评估，照常进化 | +| held-out eval | test 池已是 Video-MME 同类题 | + +## 8. main.py train 接线 + +```python +# composition root 伪代码 +elif config.mode == "train": + strategy = (PerCategoryPoolStrategy() + if config.pool_split_mode == "per_category" + else GlobalPoolStrategy()) + pools = build_or_load_pools(config, strategy) + await runner.train(pools) +``` + +## 9. 文件改动地图 + +| 文件 | 改动类型 | 内容 | +|------|---------|------| +| `app/ports.py` | 新增 | `PoolStrategy` Protocol（应用层端口，避免 core → app 逆向依赖） | +| `core/types.py` | 新增 | `PoolConfig` dataclass | +| `app/harness/pools.py` | 重构 | `GlobalPoolStrategy` + `PerCategoryPoolStrategy` + 增量逻辑 | +| `app/harness/config.py` | 修改 | RunConfig 新增 4 字段 + 校验；`pool_split_mode` 用 Literal 类型 | +| `config/default.yaml` | 修改 | 3 个新配置项 | +| `main.py` | 修改 | `task_types` 纳入 `cli_overrides` + RunConfig；train 接线；strategy 组装 | +| `app/harness/log.py` | 微调 | `_runs` 表 `INSERT OR IGNORE` 改为 `ON CONFLICT DO UPDATE`（增量 infer 更新时间戳） | +| `tests/unit/test_pools.py` | 新增 | 两种策略单元测试 | +| `tests/integration/test_pool_strategy.py` | 新增 | 端到端集成测试 | + +### 不改动 + +- `app/harness/batching.py` — 消费 scoped 池，无需感知策略 +- `core/evolution/` — 进化引擎与池来源解耦 +- `app/harness/inference.py` — 推理逻辑不变 +- `app/question_gen/` — 题目生成与池构建解耦 diff --git a/research-wiki/designs/2026-07-14-task-type-strategy-design.md b/research-wiki/designs/2026-07-14-task-type-strategy-design.md new file mode 100644 index 0000000..9fc3771 --- /dev/null +++ b/research-wiki/designs/2026-07-14-task-type-strategy-design.md @@ -0,0 +1,265 @@ +--- +id: task-type-strategy +title: 出题管线 TaskTypeStrategy 拆分设计（Clean Architecture） +type: design +created: 2026-07-14 +status: approved +--- + +# 出题管线 TaskTypeStrategy 拆分设计 + +## 1. 目标 + +借鉴 Clean Architecture 思想，将出题管线从 5 个粗粒度 Family 替换为 12 个题型级别的 TaskTypeStrategy，实现题型独立的出题策略。第一个特化实现为 ActionRecognitionStrategy（含 6 个失败子模式靶向出题）。 + +### 设计驱动 + +| 问题 | 数据来源 | +|------|---------| +| v2-360 全部 30 道 AR 题 100% 单帧可答 | `research-wiki/findings/2026-07-14-question-quality-gap-analysis.md` | +| 12/12 题型存在 CRITICAL 或 HIGH 差距 | 同上 | +| 同一 task_type 跨 family 随机选择导致行为不确定 | `app/question_gen/families.py` — AR 同时在 RETRIEVAL 和 VISUAL | +| 5 个 prompt 模板零题型分支 | `store/prompts/question_gen/*.md` | + +## 2. 架构 + +### 2.1 替换关系 + +``` +之前：pipeline → get_family_for_slot() → QuestionFamilySpec（5 个，随机选择） +之后：pipeline → get_strategy(task_type) → TaskTypeStrategy（12 个，确定性查找） +``` + +### 2.2 类层次 + +``` +TaskTypeStrategy (Protocol) + │ + ├── BaseTaskTypeStrategy (类) + │ └── 封装现有 family 行为，确定性绑定一个 QuestionFamilySpec + │ └── select_sub_pattern → None + │ └── extra_gates → [] + │ └── 11 个题型用此类 + │ + └── ActionRecognitionStrategy (类) + └── 自包含采样/prompt/SubPattern + └── 仍提供 strategy_name / skill_target / leak_probe_template（gate/store 合约） + └── 6 个 SubPattern + maintenance_pool +``` + +### 2.3 接口定义 + +```python +class TaskTypeStrategy(Protocol): + """题型出题策略 — pipeline 的唯一接口。""" + + @property + def task_type(self) -> str: ... + + # ── 采样 ── + @property + def sampling_level(self) -> int: ... + @property + def sampling_constraint(self) -> SamplingConstraint: ... + + # ── Prompt ── + @property + def prompt_template(self) -> str: ... + def select_sub_pattern(self, rng: Random) -> SubPattern | None: ... + def build_prompt_context(self, material: MaterialContext, sub_pattern: SubPattern | None) -> dict: ... + + # ── Gate / Store 合约 ── + @property + def strategy_name(self) -> str: + """策略标识名，写入 store family 字段（如 "RETRIEVAL"、"ACTION_RECOGNITION"）。""" + @property + def skill_target(self) -> str: + """目标失败机制编号，写入 store（如 "M1"、"AR"）。""" + @property + def leak_probe_template(self) -> str: + """泄漏检测 prompt 模板文件名，供 leak_test gate 使用。""" + + def extra_gates(self, candidate: CandidateQuestion) -> list[GateResult]: + """题型专属的额外验证（base 返回空列表）。""" +``` + +`strategy_name` / `skill_target` / `leak_probe_template` 替代了 pipeline 中对 `slot.family.name` / `slot.family.skill_target` / `slot.family.leak_profile.probe_template` 的访问，确保 gate 和 store 合约不破坏。 + +## 3. SubPattern 数据结构 + +```python +@dataclass(frozen=True) +class SubPattern: + """出题子模式 — 靶向特定失败机制。""" + name: str + weight: float + sampling_level_override: int | None + constraint_override: SamplingConstraint | None + instruction: str # 核心指令（1-3 句话） + positive_examples: list[dict] # VME 原题 few-shot + negative_examples: list[dict] # 反面示例 + distractor_rules: str # 干扰项构造规则 +``` + +## 4. ActionRecognitionStrategy 的 6 个 SubPattern + +来源：22 道错题双分类器仲裁分析。 + +| SubPattern | 权重 | 采样 | 错题# | 失败机制 | +|-----------|------|------|-------|---------| +| premature_evidence_anchoring | 0.20 | L1 | #3,4,12,17 | 找到一个证据就停搜，未验证全部选项 | +| temporal_reasoning_failure | 0.20 | L1 | #8,9,15,16 | 事件排序错误 / 无法定位第 N 次事件 | +| semantic_rigidity | 0.15 | L2 | #1,2,22 | 要求字面匹配，拒绝同义词释义 | +| fine_grained_visual_action | 0.15 | L2 | #6,13,14,18 | 识别"做了什么"但分不清"怎么做的" | +| cross_segment_entity_tracking | 0.15 | L1 | #7,19,20 | 单段正确但无法跨段合并 | +| evidence_gap_confabulation | 0.15 | L2 | #5,10,11,21 | 缺证据时编造因果链 | + +### ActionRecognitionStrategy 的 gate/store 合约值 + +| 字段 | 值 | 说明 | +|------|-----|------| +| strategy_name | `"ACTION_RECOGNITION"` | store 的 family 字段 | +| skill_target | `"M1_AR"` | 继承自 RETRIEVAL 的 M1 + AR 后缀区分 | +| leak_probe_template | `"gate_leak_retrieval.md"` | 复用 RETRIEVAL 的泄漏检测模板 | +| sampling_level（默认） | 2 | L2 事件级（从 L3 提升，有意变更） | +| sampling_constraint（默认） | `min_subtitles=3, min_l3_nodes=5, require_frames=True, cross_l2_span=True` | 加强约束（有意变更） | + +SubPattern 可通过 `sampling_level_override` / `constraint_override` 进一步覆盖默认值。 + +### 4.1 能力保持机制 + +capability_maintenance 不是生成子模式，而是**采样来源**：从 Video-MME AR 的 41 道正确题中采样，直接进训练集。通过 `batch_correct_ratio` 控制混合比例。 + +``` +训练 batch 组成: + ├── 错误题来源: 6 个 SubPattern 生成的新题 + └── 正确题来源: maintenance_pool (41 道已验证正确的 VME 原题) +``` + +## 5. pipeline 集成 + +### 5.1 调用链变更 + +``` +之前: slot.family → sample_material_v2(family_spec=...) → generate_one_v2(family_spec=...) + → run_gates(family_spec=...) → store.record_item(family=slot.family.name) + +之后: strategy = get_strategy(slot.task_type) + sub_pattern = strategy.select_sub_pattern(rng) + level = sub_pattern.sampling_level_override or strategy.sampling_level + constraint = sub_pattern.constraint_override or strategy.sampling_constraint + material = sample_material_v2(level=level, constraint=constraint, ...) + prompt_ctx = strategy.build_prompt_context(material, sub_pattern) + candidate = generate_one_v2(prompt_ctx=prompt_ctx, ...) + report = run_gates(candidate, leak_template=strategy.leak_probe_template, ...) + extra = strategy.extra_gates(candidate) + store.record_item(..., family=strategy.strategy_name, skill_target=strategy.skill_target, + sub_pattern=sub_pattern.name if sub_pattern else None) +``` + +### 5.2 接口影响面 + +| 文件 | 变更 | 影响范围 | +|------|------|---------| +| `sampler_v2.py` | `sample_material_v2` 签名：`family_spec` → `level: int` + `constraint: SamplingConstraint` | 函数签名 + 内部 `constraint = family_spec.sampling` 行 | +| `pipeline_v2.py` | `_process_one_slot`：`slot.family` → `strategy`；`SlotAssignment.family` 字段移除 | `_assign_slots` / `_process_one_slot` / `_process_wrapper` | +| `generator_v2.py` | `generate_one_v2` 签名：新增 `prompt_ctx: dict` 参数，替代内部读 `family_spec.prompt_template` | `_build_v2_prompt` / `generate_one_v2` | +| `gates.py` | `run_gates` 签名：`family_spec` → `leak_template: str` | `_run_leak_test` 参数变更 | +| `run_store.py` | `record_item` 新增 `sub_pattern: str | None` 参数 | DDL 加 `sub_pattern TEXT` 列 | + +### 5.3 不变的部分 + +后处理（postprocess）、四门 gate 内部逻辑、去重（embedding）、重量抽检（heavy_check）、on_accept 回调、progress 断点续跑 — 全部不动。 + +## 6. Strategy 注册 + +```python +_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {} + +def get_strategy(task_type: str) -> TaskTypeStrategy: + if task_type in _STRATEGY_REGISTRY: + return _STRATEGY_REGISTRY[task_type] + return _build_default_strategy(task_type) +``` + +### 6.1 BaseTaskTypeStrategy family 消歧绑定表 + +消除多 family 随机性。每个 task_type 确定性绑定一个 family。 + +| task_type | 旧合法 families | 新绑定 | 被移除行为 | 理由 | +|-----------|---------------|--------|-----------|------| +| Action Recognition | RETRIEVAL, VISUAL | **特化策略** | 不走 family | 完全自包含 | +| Object Recognition | RETRIEVAL | RETRIEVAL | 无 | 唯一 | +| Object Reasoning | RETRIEVAL, REASONING | **REASONING** | RETRIEVAL 的 L3 单帧 + 禁推理 prompt | 需要推理；RETRIEVAL 产出识别题是已知 bug | +| Action Reasoning | REASONING | REASONING | 无 | 唯一 | +| Attribute Perception | RETRIEVAL, VISUAL | **VISUAL** | RETRIEVAL 的 require_frames=False | 属性感知需要帧 | +| OCR Problems | RETRIEVAL, VISUAL | **VISUAL** | RETRIEVAL 的 require_frames=False | OCR 需要帧 | +| Counting Problem | ENUMERATION, VISUAL | **ENUMERATION** | VISUAL 的 L3 单帧采样 | 计数需要跨帧 | +| Information Synopsis | REASONING, ENUMERATION | **REASONING** | ENUMERATION 的 cross_l2_span=False | 综述需要跨段 | +| Temporal Reasoning | ENUMERATION | ENUMERATION | 无 | 唯一 | +| Temporal Perception | ENUMERATION | ENUMERATION | 无 | 唯一 | +| Spatial Reasoning | SPATIAL | SPATIAL | 无 | 唯一 | +| Spatial Perception | SPATIAL | SPATIAL | 无 | 唯一 | + +### 6.2 多归属题型约束变化明细 + +确定性绑定会改变多归属题型的采样约束，这是**有意的行为变更**（旧行为是随机混合，本身就是不可控的）。 + +| task_type | 旧约束（随机选择） | 新约束（确定绑定） | 变化 | +|-----------|-----------------|-----------------|------| +| Object Reasoning | RETRIEVAL(L3,no-frame,no-cross) 或 REASONING(L2,no-frame,cross) | REASONING 固定 | 不再退化为 L3 识别题 | +| Attribute Perception | RETRIEVAL(no-frame) 或 VISUAL(frame) | VISUAL 固定 | 始终要求帧 | +| OCR Problems | RETRIEVAL(no-frame) 或 VISUAL(frame) | VISUAL 固定 | 始终要求帧 | +| Counting Problem | ENUMERATION(no-frame) 或 VISUAL(frame) | ENUMERATION 固定 | 不再混入 VISUAL 的 L3 采样 | +| Information Synopsis | REASONING(cross) 或 ENUMERATION(no-cross) | REASONING 固定 | 始终跨段 | + +## 7. 文件结构 + +| 操作 | 文件 | 职责 | +|------|------|------| +| 新建 | `app/question_gen/strategy.py` | Protocol + SubPattern + BaseTaskTypeStrategy + 注册表 | +| 新建 | `app/question_gen/strategy_action_recognition.py` | ActionRecognitionStrategy 实现 + 6 个 SubPattern 定义 | +| 修改 | `app/question_gen/pipeline_v2.py` | `_process_one_slot` / `_assign_slots` 改为通过 strategy | +| 修改 | `app/question_gen/sampler_v2.py` | 签名 `family_spec` → `level` + `constraint` | +| 修改 | `app/question_gen/generator_v2.py` | 签名 `family_spec` → `prompt_ctx` | +| 修改 | `app/question_gen/gates.py` | `run_gates` 签名 `family_spec` → `leak_template` | +| 修改 | `app/question_gen/run_store.py` | `record_item` 新增 `sub_pattern` 参数 + DDL 加列 | +| 不动 | `app/question_gen/families.py` | 保留，BaseTaskTypeStrategy 内部使用 | +| 不动 | `app/question_gen/postprocess.py` | 后处理不变 | + +## 8. 非功能性需求 + +| 维度 | 设计 | +|------|------| +| 持久化 | 不变 — on_accept 逐题回调。SubPattern 名写入 store 的 sub_pattern 列以便追溯 | +| 幂等性 | 不变 — 同 seed → 同 slot 分配。strategy 查找确定性，sub_pattern 选择由 rng 控制 | +| 断点续跑 | 不变 — progress 机制在 pipeline 层，strategy 无状态 | +| 原子性 | 不变 — 逐题落库 | +| store 迁移 | `question_gen_items` 表新增 `sub_pattern TEXT` 列，默认 NULL（BaseTaskTypeStrategy 写 NULL） | + +## 9. 行为保真检查清单 + +| # | 行为 | 状态 | 说明 | +|---|------|------|------| +| 1 | 12 种 task_type 列表 | 保留 | pipeline 不变 | +| 2 | slot round-robin 视频 | 保留 | pipeline 不变 | +| 3 | family 加权随机选择 | **有意变更** | 消除不确定性 → 确定性绑定。详见 §6.2 | +| 4 | _TASK_TYPE_TO_LEVEL 映射 | **有意变更** | BaseTaskTypeStrategy 从绑定 family 读取原值（唯一归属题型数值不变）；AR 特化策略从 L3→L2（有意提升）；多归属题型因消歧而变化（详见 §6.2） | +| 5 | SamplingConstraint 约束 | **有意变更** | 同 #4 逻辑：唯一归属不变，多归属因消歧变化，AR 有意加强 | +| 6 | 重出循环 + 换视频 | 保留 | | +| 7 | 后处理洗牌 + verbatim | 保留 | | +| 8 | 四门 gate | 保留 | leak_test 通过 strategy.leak_probe_template 获取模板 | +| 9 | embedding 去重 | 保留 | | +| 10 | 重量抽检 | 保留 | | +| 11 | on_accept 回调 | 保留 | | +| 12 | progress 断点续跑 | 保留 | | +| 13 | reject_reason 传入重出 | 保留 | | +| 14 | QuestionGenStore 记录 | 保留 | family → strategy_name；新增 sub_pattern 列 | + +## 10. 渐进替换路径 + +``` +Phase 1 (当前): ActionRecognitionStrategy 特化 + 11 个 BaseTaskTypeStrategy +Phase 2 (按需): 逐个替换表现差的题型为特化策略 +Phase N (最终): 12 个特化策略，families.py 可移除 +``` diff --git a/research-wiki/designs/per-category-pool-strategy.md b/research-wiki/designs/per-category-pool-strategy.md new file mode 100644 index 0000000..17e4713 --- /dev/null +++ b/research-wiki/designs/per-category-pool-strategy.md @@ -0,0 +1,9 @@ +--- +type: design +node_id: design:per-category-pool-strategy +title: "Per-Category Pool Strategy 设计" +date: 2026-07-13 +--- + +# Per-Category Pool Strategy 设计 + diff --git a/research-wiki/findings/2026-07-14-question-quality-gap-analysis.md b/research-wiki/findings/2026-07-14-question-quality-gap-analysis.md new file mode 100644 index 0000000..7870e49 --- /dev/null +++ b/research-wiki/findings/2026-07-14-question-quality-gap-analysis.md @@ -0,0 +1,206 @@ +# v2-360 vs Video-MME 出题质量差距分析报告 + +> 基线准确率 73.2% | v2 覆盖 6/12 类型 | VME 共 900 题 + +--- + +## 1. 全局概览 + +| # | 题型 | VME 题数 | v2 题数 | VME 准确率 | VME 主导模式 | v2 主导模式 | 差距等级 | +|---|------|---------|---------|-----------|-------------|------------|---------| +| 1 | Information Synopsis | 163 | 0 | -- | 全局主题识别 (61%) | -- | **CRITICAL** | +| 2 | Counting Problem | 48 | 0 | 50% | 全视频离散段计数 (29%) | -- | **CRITICAL** | +| 3 | Attribute Perception | 27 | 0 | -- | 否定/假命题验证 (26%) | -- | **CRITICAL** | +| 4 | OCR Problems | 14 | 0 | -- | 计分板/数值精确读取 (29%) | -- | **CRITICAL** | +| 5 | Temporal Perception | 6 | 0 | 33% | 事件时间定位 (50%) | -- | **CRITICAL** | +| 6 | Spatial Perception | 3 | 0 | 33% | 时空定位/朝向推理 (各33%) | -- | **CRITICAL** | +| 7 | Action Recognition | 63 | 30 | -- | 时间锚点行为观察 (21%) | OCR 文字读取 (67%) | **CRITICAL** | +| 8 | Action Reasoning | 180 | 30 | -- | 因果 why 推理 (36%) | OCR/文字读取 (53%) | **CRITICAL** | +| 9 | Object Reasoning | 240 | 30 | -- | 因果解释推理 (22%) | OCR/文字读取 (57%) | **CRITICAL** | +| 10 | Temporal Reasoning | 91 | 30 | -- | 严格序列排序 (52%) | OCR/文字读取 (37%) | **CRITICAL** | +| 11 | Object Recognition | 54 | 30 | -- | 否定/缺失检测 (30%) | OCR/文字读取 (33%) | **HIGH** | +| 12 | Spatial Reasoning | 11 | 30 | -- | 地理位置推断 (18%) | OCR/文字读取 (30%) | **HIGH** | + +**结论**: 12 个题型全部存在 CRITICAL 或 HIGH 级别差距。6 个题型完全未覆盖；已覆盖的 6 个题型中，v2 的推理深度与 VME 严重错位。 + +### 推理深度对比（已覆盖的 6 个题型平均） + +| 深度层级 | VME 平均占比 | v2 平均占比 | 差距 | +|---------|------------|-----------|------| +| single_frame | 8.0% | **66.1%** | v2 严重偏向单帧 | +| multi_frame | 25.4% | 15.6% | 略低 | +| cross_segment | 25.7% | 15.6% | 不足 | +| full_video | 41.0% | **2.8%** | v2 几乎为零 | + +--- + +## 2. 逐题型分析 + +### 2.1 Information Synopsis (VME=163, v2=0) -- CRITICAL + +- **VME 特征**: 79.8% full_video。主导模式为"视频主题识别"(61%)，干扰项为同领域相邻主题（如 Gucci 纪录片选项含 Chanel/LV/Prada）。否定模式占 13.5%（"哪个事件未被报道"）。 +- **差距诊断**: v2 完全缺失。管线无任何摘要/主题/综述类 prompt 模板。 +- **管线根因**: REASONING 家族虽覆盖 Information Synopsis，但 L2 采样仅提供局部事件，无法支撑全局主题判断。prompt 中无综述类题目的具体指导。 + +### 2.2 Counting Problem (VME=48, v2=0) -- CRITICAL + +- **VME 特征**: 72.9% full_video。要求在整个视频中持续追踪计数（"Salah 出现了几个进球"）。含条件过滤计数（"独唱表演有几个"）和陷阱题（答案为 0）。 +- **差距诊断**: v2 完全缺失。ENUMERATION 家族名义覆盖但 `require_frames=False` + `cross_l2_span=False` 产出文本计数题。 +- **管线根因**: 采样约束限制在单 L2 段内，无法产出需全视频追踪的计数题。无最小计数阈值或视觉计数要求。 + +### 2.3 Attribute Perception (VME=27, v2=0) -- CRITICAL + +- **VME 特征**: 26% 为否定验证（"以下哪项不正确"），需逐项核实 3-4 个事实。37% cross_segment。含情感/态度感知（从非语言线索推断情绪）。 +- **差距诊断**: v2 完全缺失。RETRIEVAL 和 VISUAL 家族名义覆盖但无属性感知专用指导。 +- **管线根因**: RETRIEVAL 中 `require_frames=False`，属性感知退化为文本检索。VISUAL 中 L3 单帧采样使属性观察过于简单。 + +### 2.4 OCR Problems (VME=14, v2=0) -- CRITICAL + +- **VME 特征**: 50% single_frame 但要求精确数值/文字提取（"半场比分 32-23"），干扰项数值接近（57.5/55/60 kg）。含时间锚定 OCR（"罚球后比分"）。 +- **差距诊断**: v2 完全缺失。讽刺的是 v2 已有的 6 个题型中大量问题实质上是 OCR 题（占 v2 总量 ~40%），但未被归类为 OCR。 +- **管线根因**: RETRIEVAL 和 VISUAL 覆盖 OCR 但无文字难度要求（大小、角度、遮挡）。L3 单帧 + 大字号文本 = 简单读取。 + +### 2.5 Temporal Perception (VME=6, v2=0) -- CRITICAL + +- **VME 特征**: 83.3% cross_segment。事件时间定位（"厨房事故发生在周几午餐？"），相对时长估计（"哪款车占视频比例最大？"）。 +- **差距诊断**: v2 完全缺失。管线无时间定位或时长估计模板。 +- **管线根因**: ENUMERATION 家族名义覆盖但 prompt 无时长/间隔/定位指导。`cross_l2_span=False` 无法跨段生成。 + +### 2.6 Spatial Perception (VME=3, v2=0) -- CRITICAL + +- **VME 特征**: 66.7% cross_segment。时空联合定位（"左下角绿色模型何时出现"）、物体目的地推理、相对朝向判断。 +- **差距诊断**: v2 完全缺失。管线无空间-时间交叉模板。 +- **管线根因**: SPATIAL 家族仅覆盖 Spatial Reasoning，Spatial Perception 未接入任何家族。 + +### 2.7 Action Recognition (VME=63, v2=30) -- CRITICAL + +- **VME 特征**: 95.2% 多帧以上。主导模式为时间锚点行为观察 (21%)、否定/缺失 (17%)、事件机制 (17%)。 +- **v2 特征**: **100% single_frame**。67% 纯 OCR（球衣号码、排行榜文字）。无一题涉及动作序列或时序理解。 +- **差距诊断**: 题型根本错配 -- v2 测试静态视觉感知而非动作识别。截图模型即可满分。 +- **管线根因**: RETRIEVAL 家族 L3 采样 + "factual recall" prompt 产出单帧标签题。VISUAL 家族 L3 无法展示动作序列。 + +### 2.8 Action Reasoning (VME=180, v2=30) -- CRITICAL + +- **VME 特征**: 0% single_frame。因果 why 推理占 36%，角色心理推断 10%，否定推理 8%，策略评估 7%。 +- **v2 特征**: 53% single_frame（OCR 污染）。仅 3/30 题涉及因果推理。零角色动机/策略/隐喻推理。 +- **差距诊断**: 过半题目为错分类的 OCR/检索题。真正的推理题占比不足 10%。 +- **管线根因**: REASONING prompt 虽要求"multi-hop"但无结构性执行约束。`require_frames=False` 使推理退化为文本提取。 + +### 2.9 Object Reasoning (VME=240, v2=30) -- CRITICAL + +- **VME 特征**: 90% cross_segment 或 full_video。因果解释 (22%)、关系推断 (16%)、否定验证 (13%)、评价判断 (12%)。 +- **v2 特征**: 56.7% single_frame。主导为 OCR (33%) 和物体属性识别 (23%)。仅 27% 涉及推理。 +- **差距诊断**: v2 的"物体推理"实为物体识别的错误标签。无因果链、无关系推断、无评价性判断。 +- **管线根因**: Object Reasoning 同时出现在 RETRIEVAL (L3, 禁止推理) 和 REASONING (L2) 两个家族，RETRIEVAL 产出的题目本质是识别题。 + +### 2.10 Temporal Reasoning (VME=91, v2=30) -- CRITICAL + +- **VME 特征**: 72.5% full_video。严格序列排序占 52%（"以下 5 项按什么顺序出现？"），日程重建 11%。 +- **v2 特征**: 36.7% single_frame。零序列排序题。最好的题目是弱版"X 之后发生了什么"（只需链接两个时刻）。 +- **差距诊断**: VME 的核心模式（排列 3-5 项的顺序）在 v2 中完全缺失。v2 的"时序"题多为 OCR 或简单因果。 +- **管线根因**: ENUMERATION 家族 `cross_l2_span=False` 限制在单段内，无法收集多段时序材料。prompt 的"顺序"指导过于笼统。 + +### 2.11 Object Recognition (VME=54, v2=30) -- HIGH + +- **VME 特征**: 75.9% cross_segment 或 full_video。否定/缺失检测 (30%)、全视频实体识别 (26%)。 +- **v2 特征**: 63.3% single_frame。OCR/标签读取 (33%)、物体属性 (27%)。 +- **差距诊断**: VME 的"识别"意为"跨视频追踪并清点所有相关实体"，v2 理解为"在帧中认出物体"。 +- **管线根因**: RETRIEVAL L3 + "直接可观察信息" prompt 产出最简单的识别题。 + +### 2.12 Spatial Reasoning (VME=11, v2=30) -- HIGH + +- **VME 特征**: 72.7% multi_frame 或 full_video。从视觉线索推断地理位置、理解空间排列目的、估算距离。 +- **v2 特征**: 86.7% single_frame。OCR (30%)、相对位置 (23%)、颜色识别 (17%)。 +- **差距诊断**: v2 测试"观察"（X 在 Y 左边），VME 测试"推断"（从建筑风格推断国家）。 +- **管线根因**: SPATIAL prompt 无推断层级要求。L1 采样与空间精度需求错位（应 L3 帧级 + 多帧序列）。 + +--- + +## 3. 管线系统性问题 + +### 3.1 Prompt 层问题 + +| 家族 | 核心缺陷 | 影响题型 | +|------|---------|---------| +| RETRIEVAL | "factual recall" + "禁止 multi-hop" = 强制最简难度 | AR/OR/ObjRec/AP/OCR | +| REASONING | 无结构性 multi-hop 执行约束（无最小跳数、无证据链格式） | ActReas/ObjReas/InfoSyn | +| ENUMERATION | 计数/排序/时序/综述用同一泛化 prompt，无题型分支 | Count/TempReas/TempPerc/InfoSyn | +| VISUAL | 无难度下限（"什么颜色"即合法），无时序视觉变化要求 | AP/Count/OCR/AR | +| SPATIAL | Perception 与 Reasoning 用同一 prompt，无推断层级要求 | SpatPerc/SpatReas | +| **全部** | **零题型分支**: 5 个 prompt 均为题型无关，task_type 仅作元数据传入 | 全部 12 型 | + +### 3.2 采样层问题 + +| 问题 | 具体表现 | 影响 | +|------|---------|------| +| L3 单帧采样用于 4 个题型 | AR/ObjRec/AP/OCR 在 RETRIEVAL 中均 L3 | 100% single_frame 退化 | +| `cross_l2_span=False` 用于时序题 | ENUMERATION 家族限制在单段 | 无法生成跨段排序/计数 | +| `require_frames=False` 用于视觉题型 | RETRIEVAL 中 AP/OCR/AR 不要求帧 | 视觉题退化为文本题 | +| 约束按家族统一，非按题型 | 同一家族内不同题型需求冲突 | 系统性错配 | + +### 3.3 Gate 层问题 + +| Gate | 盲点 | 后果 | +|------|------|------| +| blind_answer | 只检测上下文泄露，不检测"有上下文也简单" | 大量 easy 题通过 | +| key_verify | text OR frames 通过即可 | 视觉题型可纯文本回答 | +| leak_test | 按家族探测，非按题型 | 题型特有捷径未拦截 | +| heavy_check | 后置非阻塞，easy 题已接受 | 难度反馈无实际作用 | +| **缺失** | **无难度门控 gate** | 简单题无任何拦截机制 | +| **缺失** | **无题集难度分布检查** | 无法控制 easy/med/hard 比例 | + +--- + +## 4. 改进路线图 + +### P0: 影响最大、立即可做 + +| # | 改进项 | 预期收益 | 实施方式 | +|---|-------|---------|---------| +| P0-1 | **Prompt 内加入题型分支** | 消除"AR 出 OCR 题"等根本性错配 | 每个 prompt 增加 `{% if task_type == "Action Recognition" %}` 块，含题型专属规则和负面示例 | +| P0-2 | **修复采样约束为题型级** | 解除 L3/no-frame/no-cross 限制 | `SamplingConstraint` 从 per-family 改为 per-(family, task_type)，AR/AP/OCR 强制 `require_frames=True`，TempReas 强制 `cross_l2_span=True` | +| P0-3 | **提升 AR/ActReas/ObjReas 采样层级到 L2** | 从单帧提升到多帧 | RETRIEVAL 中 AR/AP/OCR 改为 L2；VISUAL 中 AR 改为 L2 | +| P0-4 | **添加否定/缺失题模式** | 覆盖 VME 中占比 15-30% 的核心模式 | 新增 prompt 指令："生成'以下哪项未在视频中出现'类型题目"；需 full_video 采样支持 | +| P0-5 | **新增难度门控 gate** | 直接过滤简单题 | 添加第 5 个 gate: 给 LLM 完整上下文，要求在 5 秒内作答；能秒答 = fail | + +### P1: 重要但需设计 + +| # | 改进项 | 预期收益 | 实施要点 | +|---|-------|---------|---------| +| P1-1 | **新增 6 个缺失题型的 prompt 模板** | 覆盖率从 50% 提升到 100% | InfoSynopsis: L1 全场景 + 主题合成；Counting: full_video + 视觉计数 + 条件过滤；AP: 否定验证 + 情感感知；OCR: 精确数值 + 时间锚定；TempPerc: 时间定位 + 时长估计；SpatPerc: 时空联合 + 朝向推理 | +| P1-2 | **创建独立 TEMPORAL 和 SYNOPSIS 家族** | 解除 ENUMERATION 对时序题的约束错配 | TEMPORAL: `cross_l2_span=True`, L1 采样, 序列排序专用 prompt；SYNOPSIS: L1 全场景, 综述/主题/否定专用 prompt | +| P1-3 | **key_verify gate 按题型区分证据要求** | 视觉题型必须帧内有证据 | AP/OCR/SpatPerc/AR: 改为 "evidence in frames ONLY"；reasoning 类: 维持 "text OR frames" | +| P1-4 | **Prompt 加入难度校准示例** | LLM 有锚定标准，避免难度地板 | 每个 prompt 附 2 个正面示例（medium/hard）和 2 个负面示例（too-easy），带评分理由 | +| P1-5 | **题集难度分布控制** | 确保 easy/med/hard 比例合理 | 后处理阶段按难度评分排序，按 20/50/30 目标裁剪，不足时重采 | + +### P1-6 重点：消除 Object Reasoning 在 RETRIEVAL 中的矛盾 + +将 Object Reasoning 从 RETRIEVAL 家族的 `legal_task_types` 中移除。Object Reasoning 只应出现在 REASONING 家族。当前双家族配置导致 ~50% 的 ObjReas 题目在"禁止推理"的 RETRIEVAL prompt 下生成，本质上是识别题。 + +### P2: 长期架构改进 + +| # | 改进项 | 预期收益 | 复杂度 | +|---|-------|---------|-------| +| P2-1 | **Full-video 采样路径** | 支持 full_video 深度题（VME 41% 需要） | 需新增 L0（整棵树）采样层，采集全视频摘要/时间线/实体清单作为 context | +| P2-2 | **将 heavy_check 改为前置阻塞 gate** | 难度验证从抽检变为全量拦截 | 重构 gate pipeline，`difficulty_steps` 作为第 5 gate 的量化指标 | +| P2-3 | **引入"对抗性"题目生成** | 模拟 VME 的近距离干扰项设计 | 两阶段生成：先生成题目，再用另一 LLM 对抗性改写干扰项使其更接近正确答案 | +| P2-4 | **跨题型去重与平衡** | 消除"40% 的题实质是 OCR"的长尾效应 | 题目验收后做跨题型语义聚类，标记实质题型 vs 标注题型不一致的题目 | + +### 优先级总结 + +``` +立即执行 (P0, 1-2 天): + P0-1 Prompt 题型分支 ──────────── 消除最严重的题型错配 + P0-2 采样约束题型化 ──────────── 解除 L3/no-frame 限制 + P0-3 提升采样层级 ────────────── 从单帧到多帧 + P0-5 难度门控 gate ────────────── 直接拦截简单题 + +短期设计 (P1, 3-5 天): + P1-1 新增 6 个缺失题型模板 ──── 覆盖率 50% -> 100% + P1-2 新增 TEMPORAL/SYNOPSIS 家族 + P1-4 Prompt 难度校准示例 + +中期架构 (P2, 1-2 周): + P2-1 Full-video 采样路径 + P2-3 对抗性干扰项生成 +``` diff --git a/research-wiki/graph/edges.json b/research-wiki/graph/edges.json index b9fd600..9426377 100644 --- a/research-wiki/graph/edges.json +++ b/research-wiki/graph/edges.json @@ -160,6 +160,11 @@ "id": "plan:action-recognition-training", "label": "Action Recognition 单题型首次训练实验计划", "type": "plan" + }, + { + "id": "plan:task-type-strategy-framework", + "label": "TaskTypeStrategy 框架实现计划 (Plan A)", + "type": "plan" } ], "links": [ @@ -288,6 +293,13 @@ "relation": "implements", "evidence": "计划实现设计文档中的 2 处代码修改 + 实验配置 + 训练脚本", "added": "2026-07-14T04:50:15.986586+00:00" + }, + { + "source": "plan:task-type-strategy-framework", + "target": "design:task-type-strategy", + "relation": "implements", + "evidence": "Plan A 实现设计中的框架层（Protocol + Base + pipeline 集成）", + "added": "2026-07-14T09:15:22.951263+00:00" } ] } \ No newline at end of file diff --git a/research-wiki/index.md b/research-wiki/index.md index 0d0fffd..e1c5d17 100644 --- a/research-wiki/index.md +++ b/research-wiki/index.md @@ -1,8 +1,8 @@ # Research Wiki 索引 -> 自动生成，更新时间：2026-07-14 04:50 UTC +> 自动生成，更新时间：2026-07-14 09:15 UTC -## design (24) +## design (25) - [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design` - [2026-07-07-app-harness-design](designs/2026-07-07-app-harness-design.md) `design:2026-07-07-app-harness-design` - [2026-07-07-core-evolution-design](designs/2026-07-07-core-evolution-design.md) `design:2026-07-07-core-evolution-design` @@ -20,6 +20,7 @@ - [Spec-2 建树批量并行入口](designs/batch-tree-build.md) `design:batch-tree-build` - [Spec-3 出题管线 v2（失败机理靶向+逐题质量门）](designs/question-gen-v2.md) `design:question-gen-v2` - [出题模块迁移设计（question_gen）](designs/2026-07-07-question-gen-design.md) `design:2026-07-07-question-gen-design` +- [出题管线 TaskTypeStrategy 拆分设计（Clean Architecture）](designs/2026-07-14-task-type-strategy-design.md) `design:2026-07-14-task-type-strategy-design` - [建树修复管线：熔断根因修复 + 断点续跑 + 并发改造](designs/tree-repair-resilience.md) `design:tree-repair-resilience` - [建树模块竖切设计：数据结构 + 建树 + 修复 + 迁移](designs/2026-07-07-tree-module-design.md) `design:2026-07-07-tree-module-design` - [建树模块竖切设计：数据结构 + 建树 + 修复 + 迁移](designs/tree-module-vertical-slice.md) `design:tree-module-vertical-slice` @@ -28,13 +29,14 @@ - [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth` - [赛题生成工具设计（Question Generation Synthesis）](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design` -## finding (4) +## finding (5) - [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy` - [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis` +- [2026-07-14-question-quality-gap-analysis](findings/2026-07-14-question-quality-gap-analysis.md) `finding:2026-07-14-question-quality-gap-analysis` - [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check` - [Harness 评估: Spec-2 批量并行建树](findings/eval-spec2-batch-tree-build.md) `finding:eval-spec2-batch-tree-build` -## plan (26) +## plan (28) - [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm` - [2026-07-07-app-harness](plans/2026-07-07-app-harness.md) `plan:2026-07-07-app-harness` - [2026-07-07-core-evolution](plans/2026-07-07-core-evolution.md) `plan:2026-07-07-core-evolution` @@ -47,6 +49,7 @@ - [2026-07-11-batch-tree-build](plans/2026-07-11-batch-tree-build.md) `plan:2026-07-11-batch-tree-build` - [2026-07-12-per-category-pool-strategy](plans/2026-07-12-per-category-pool-strategy.md) `plan:2026-07-12-per-category-pool-strategy` - [2026-07-14-action-recognition-training](plans/2026-07-14-action-recognition-training.md) `plan:2026-07-14-action-recognition-training` +- [2026-07-14-task-type-strategy-framework](plans/2026-07-14-task-type-strategy-framework.md) `plan:2026-07-14-task-type-strategy-framework` - [Action Recognition 单题型首次训练实验计划](plans/action-recognition-training.md) `plan:action-recognition-training` - [app/harness/ 训练循环编排层实现计划](plans/app-harness.md) `plan:app-harness` - [app/search/ 搜索 Agent 装配层实现计划](plans/2026-07-07-search-module.md) `plan:2026-07-07-search-module` @@ -56,6 +59,7 @@ - [question_gen 模块实现计划](plans/question-gen.md) `plan:question-gen` - [Spec-1 Agent 执行环境修复实现计划](plans/agent-runtime-fixes-plan.md) `plan:agent-runtime-fixes-plan` - [Spec-2 建树批量并行实现计划](plans/batch-tree-build-plan.md) `plan:batch-tree-build-plan` +- [TaskTypeStrategy 框架实现计划 (Plan A)](plans/task-type-strategy-framework.md) `plan:task-type-strategy-framework` - [出题管线 v2 实现计划](plans/2026-07-11-question-gen-v2.md) `plan:2026-07-11-question-gen-v2` - [建树修复管线三项改造实现计划](plans/tree-repair-resilience.md) `plan:tree-repair-resilience` - [建树模块竖切实现计划](plans/tree-module-vertical-slice.md) `plan:tree-module-vertical-slice` diff --git a/research-wiki/log.md b/research-wiki/log.md index a786403..398a2f0 100644 --- a/research-wiki/log.md +++ b/research-wiki/log.md @@ -73,3 +73,6 @@ - [2026-07-14 04:50 UTC] 新增 plan: Action Recognition 单题型首次训练实验计划 (plan:action-recognition-training) - [2026-07-14 04:50 UTC] 新增边: plan:action-recognition-training --implements--> design:action-recognition-training - [2026-07-14 04:50 UTC] 重建索引: 60 篇页面 +- [2026-07-14 09:15 UTC] 新增 plan: TaskTypeStrategy 框架实现计划 (Plan A) (plan:task-type-strategy-framework) +- [2026-07-14 09:15 UTC] 新增边: plan:task-type-strategy-framework --implements--> design:task-type-strategy +- [2026-07-14 09:15 UTC] 重建索引: 64 篇页面 diff --git a/research-wiki/plans/2026-07-12-per-category-pool-strategy.md b/research-wiki/plans/2026-07-12-per-category-pool-strategy.md new file mode 100644 index 0000000..5951b40 --- /dev/null +++ b/research-wiki/plans/2026-07-12-per-category-pool-strategy.md @@ -0,0 +1,1498 @@ +# Per-Category Pool Strategy 实现计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 实现 PoolStrategy Protocol + PerCategoryPoolStrategy，支持按类增量 baseline inference、per-category correctness 分层划分（20 train / 10 val）、task_types 作用域训练，以及 train 模式 CLI 接线。 + +**Architecture:** 在 `app/ports.py` 新增 `PoolStrategy` Protocol，在 `app/harness/pools.py` 实现 `GlobalPoolStrategy`（封装现有逻辑）和 `PerCategoryPoolStrategy`（per-category 2:1 分层）。`RunConfig` 新增 4 字段，`main.py` 完成 train 模式接线和 strategy 组装。 + +**Tech Stack:** Python 3.11, dataclasses, Protocol, pytest, SQLite + +--- + +### Task 1: core/types.py — 新增 PoolConfig + +**Files:** +- Modify: `core/types.py:57` (在 GeneratedQuestion 之后追加) +- Test: `tests/unit/test_core_types.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_core_types.py` 末尾追加： + +```python +from core.types import PoolConfig +from pathlib import Path + + +class TestPoolConfig: + """PoolConfig frozen dataclass 基本行为。""" + + def test_pool_config_frozen(self) -> None: + """PoolConfig 创建后不可变。""" + cfg = PoolConfig( + task_types=("Action Reasoning",), + seed=42, + baseline_run_id="baseline_v2", + diag_size=200, + diag_correct_ratio=0.5, + val_size=30, + val_correct_ratio=0.5, + test_size=60, + eval_min_per_class=2, + train_ratio=0.667, + test_questions_dir=Path("store/questions/benchmarks/Video-MME"), + ) + assert cfg.task_types == ("Action Reasoning",) + assert cfg.baseline_run_id == "baseline_v2" + assert cfg.train_ratio == 0.667 + + def test_pool_config_task_types_none(self) -> None: + """task_types=None 表示全部类别。""" + cfg = PoolConfig( + task_types=None, + seed=0, + baseline_run_id="run_1", + diag_size=200, + diag_correct_ratio=0.5, + val_size=30, + val_correct_ratio=0.5, + test_size=60, + eval_min_per_class=2, + train_ratio=0.667, + test_questions_dir=None, + ) + assert cfg.task_types is None + assert cfg.test_questions_dir is None +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_core_types.py::TestPoolConfig -v` +Expected: FAIL with `ImportError: cannot import name 'PoolConfig'` + +- [ ] **Step 3: 实现 PoolConfig** + +在 `core/types.py` 末尾追加： + +```python +from pathlib import Path as _Path # 避免与运行时 TYPE_CHECKING 冲突 + + +@dataclass(frozen=True) +class PoolConfig: + """池构建策略的统一配置。 + + 两组字段由两个具体策略各取所需，未使用的字段被忽略。 + + 属性: + task_types: 限定题型元组；None 表示全部类别。 + seed: 随机种子，保证可复现。 + baseline_run_id: 基线 run 标识（用于读 correctness、写入 pools 指纹）。 + diag_size: 诊断池大小（GlobalStrategy 用）。 + diag_correct_ratio: 诊断池中对题占比（GlobalStrategy 用）。 + val_size: 验证池大小（GlobalStrategy 用）。 + val_correct_ratio: 验证池中对题占比（GlobalStrategy 用）。 + test_size: held-out 测试池大小（GlobalStrategy 用）。 + eval_min_per_class: 验证池中每类保底样本数（GlobalStrategy 用）。 + train_ratio: train/(train+val) 比例（PerCategoryStrategy 用），默认 2/3。 + test_questions_dir: 外部 test 题源路径（PerCategoryStrategy 用）。 + """ + + task_types: tuple[str, ...] | None + seed: int + baseline_run_id: str + diag_size: int + diag_correct_ratio: float + val_size: int + val_correct_ratio: float + test_size: int + eval_min_per_class: int + train_ratio: float + test_questions_dir: _Path | None +``` + +- [ ] **Step 4: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_core_types.py::TestPoolConfig -v` +Expected: PASS + +- [ ] **Step 5: 提交** + +```bash +git add core/types.py tests/unit/test_core_types.py +``` + +Commit message: `feat(core): add PoolConfig dataclass for pool strategy configuration` + +--- + +### Task 2: app/ports.py — 新增 PoolStrategy Protocol + +**Files:** +- Modify: `app/ports.py:147` (文件末尾追加) +- Test: `tests/unit/test_core_protocols.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_core_protocols.py` 末尾追加： + +```python +from app.ports import PoolStrategy + + +class TestPoolStrategyProtocol: + """PoolStrategy Protocol runtime_checkable 验证。""" + + def test_pool_strategy_is_runtime_checkable(self) -> None: + """PoolStrategy 支持 isinstance 检查。""" + from app.harness.pools import Pools + from core.types import GeneratedQuestion, PoolConfig + + class FakeStrategy: + def build(self, questions, correctness, config): + return Pools( + diagnosis=[], validation=[], test=[], + baseline_run_id="", baseline_val_accuracy=0.0, + ) + + def build_incremental(self, new_task_types, questions, correctness, config): + return {} + + assert isinstance(FakeStrategy(), PoolStrategy) +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_core_protocols.py::TestPoolStrategyProtocol -v` +Expected: FAIL with `ImportError: cannot import name 'PoolStrategy'` + +- [ ] **Step 3: 实现 PoolStrategy Protocol** + +在 `app/ports.py` 末尾追加： + +```python +@runtime_checkable +class PoolStrategy(Protocol): + """池构建策略端口。 + + 应用层端口（非 core 层），因为返回类型 Pools 定义在 app/harness/pools.py。 + 两个具体策略（GlobalPoolStrategy / PerCategoryPoolStrategy）实现此接口。 + """ + + def build( + self, + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> Pools: ... + + def build_incremental( + self, + new_task_types: list[str], + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> dict[str, dict[str, list[str]]]: ... +``` + +同时在文件头部的 `TYPE_CHECKING` 块中添加： + +```python +from app.harness.pools import Pools +from core.types import PoolConfig +``` + +注意：`PoolStrategy` 的 `build` 方法返回 `Pools`，但 `Pools` 在 `app/harness/pools.py` 中定义。因为 `app/ports.py` 和 `app/harness/pools.py` 同属 app 层，不违反依赖方向。使用 `TYPE_CHECKING` 保护导入以避免循环引用。 + +- [ ] **Step 4: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_core_protocols.py::TestPoolStrategyProtocol -v` +Expected: PASS + +- [ ] **Step 5: 提交** + +```bash +git add app/ports.py tests/unit/test_core_protocols.py +``` + +Commit message: `feat(app): add PoolStrategy Protocol to application ports` + +--- + +### Task 3: app/harness/pools.py — GlobalPoolStrategy 封装 + +**Files:** +- Modify: `app/harness/pools.py` +- Test: `tests/unit/test_harness_pools.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_harness_pools.py` 追加： + +```python +from app.harness.pools import GlobalPoolStrategy +from core.types import PoolConfig +from pathlib import Path + + +class TestGlobalPoolStrategy: + """GlobalPoolStrategy 封装现有全局三分逻辑。""" + + def test_global_strategy_builds_three_pools(self) -> None: + """GlobalPoolStrategy.build 产出三个互斥池。""" + questions = _make_question_set(200) + correctness = _make_correctness(questions, 0.5) + config = PoolConfig( + task_types=None, + seed=42, + baseline_run_id="run_baseline", + diag_size=30, + diag_correct_ratio=0.5, + val_size=30, + val_correct_ratio=0.5, + test_size=30, + eval_min_per_class=1, + train_ratio=0.667, + test_questions_dir=None, + ) + strategy = GlobalPoolStrategy() + pools = strategy.build(questions, correctness, config) + + diag_ids = {q.question_id for q in pools.diagnosis} + val_ids = {q.question_id for q in pools.validation} + test_ids = {q.question_id for q in pools.test} + assert diag_ids & val_ids == set() + assert diag_ids & test_ids == set() + assert val_ids & test_ids == set() + assert len(pools.diagnosis) == 30 + assert len(pools.validation) == 30 + assert len(pools.test) == 30 + + def test_global_strategy_build_incremental_raises(self) -> None: + """GlobalPoolStrategy 不支持增量，调用 build_incremental 应报错。""" + strategy = GlobalPoolStrategy() + config = PoolConfig( + task_types=None, seed=0, baseline_run_id="r", + diag_size=10, diag_correct_ratio=0.5, + val_size=10, val_correct_ratio=0.5, + test_size=10, eval_min_per_class=1, + train_ratio=0.667, test_questions_dir=None, + ) + with pytest.raises(NotImplementedError): + strategy.build_incremental(["Action Reasoning"], [], {}, config) +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py::TestGlobalPoolStrategy -v` +Expected: FAIL with `ImportError: cannot import name 'GlobalPoolStrategy'` + +- [ ] **Step 3: 实现 GlobalPoolStrategy** + +在 `app/harness/pools.py` 中，现有 `build_pools` 函数保持不变，新增一个类： + +```python +class GlobalPoolStrategy: + """全局三分策略：test → val → diag progressive exclusion。 + + 封装现有 build_pools 逻辑为 PoolStrategy 接口。 + """ + + def build( + self, + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> Pools: + """委托给现有 build_pools 函数。""" + return build_pools( + questions, + correctness, + diag_cfg={ + "size": config.diag_size, + "correct_ratio": config.diag_correct_ratio, + "task_types": list(config.task_types) if config.task_types else None, + "seed": config.seed, + "min_per_class": None, + }, + val_cfg={ + "size": config.val_size, + "correct_ratio": config.val_correct_ratio, + "task_types": list(config.task_types) if config.task_types else None, + "seed": config.seed, + "min_per_class": config.eval_min_per_class, + }, + test_cfg={"size": config.test_size, "seed": config.seed}, + baseline_run_id=config.baseline_run_id, + ) + + def build_incremental( + self, + new_task_types: list[str], + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> dict[str, dict[str, list[str]]]: + """全局策略不支持增量。""" + raise NotImplementedError("GlobalPoolStrategy 不支持增量构建，请使用 PerCategoryPoolStrategy。") +``` + +在文件头部添加导入： + +```python +from core.types import PoolConfig +``` + +- [ ] **Step 4: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py::TestGlobalPoolStrategy -v` +Expected: PASS + +- [ ] **Step 5: 运行全部现有池测试确认无回归** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py -v` +Expected: 全部 PASS + +- [ ] **Step 6: 提交** + +```bash +git add app/harness/pools.py tests/unit/test_harness_pools.py +``` + +Commit message: `refactor(harness): wrap existing build_pools in GlobalPoolStrategy` + +--- + +### Task 4: app/harness/pools.py — PerCategoryPoolStrategy + +**Files:** +- Modify: `app/harness/pools.py` +- Test: `tests/unit/test_harness_pools.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_harness_pools.py` 追加： + +```python +from app.harness.pools import PerCategoryPoolStrategy + + +def _make_per_category_questions() -> list[GeneratedQuestion]: + """构造 12 类各 30 题，共 360 题。""" + task_types = [ + "Action Prediction", "Action Reasoning", "Action Recognition", + "Action Sequence", "Causal Reasoning", "Event Reasoning", + "Object Interaction", "Object Reasoning", "Object Recognition", + "Scene Understanding", "Spatial Reasoning", "Temporal Reasoning", + ] + questions = [] + for tt in task_types: + for i in range(30): + questions.append(_make_question(f"{tt}_{i:03d}", tt)) + return questions + + +class TestPerCategoryPoolStrategy: + """PerCategoryPoolStrategy per-category 2:1 分层划分。""" + + def test_per_category_split_20_10(self) -> None: + """每类 30 题按 correctness 2:1 分层 → 20 train + 10 val。""" + questions = _make_per_category_questions() + # 每类前 18 题 correct，后 12 题 wrong + correctness = {} + for q in questions: + idx = int(q.question_id.split("_")[-1]) + correctness[q.question_id] = idx < 18 + + config = PoolConfig( + task_types=None, + seed=42, + baseline_run_id="baseline_v2", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, + test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + + # 12 类 × 20 = 240 train, 12 类 × 10 = 120 val + assert len(pools.diagnosis) == 240 + assert len(pools.validation) == 120 + + # 逐类验证 train=20, val=10 + from collections import Counter + diag_counts = Counter(q.task_type for q in pools.diagnosis) + val_counts = Counter(q.task_type for q in pools.validation) + for tt in diag_counts: + assert diag_counts[tt] == 20, f"{tt} train 应为 20，实际 {diag_counts[tt]}" + assert val_counts[tt] == 10, f"{tt} val 应为 10，实际 {val_counts[tt]}" + + # train 和 val 互斥 + diag_ids = {q.question_id for q in pools.diagnosis} + val_ids = {q.question_id for q in pools.validation} + assert diag_ids & val_ids == set() + + def test_per_category_correctness_ratio_aligned(self) -> None: + """train 和 val 的 correctness 比例应对齐。""" + questions = _make_per_category_questions() + correctness = {} + for q in questions: + idx = int(q.question_id.split("_")[-1]) + correctness[q.question_id] = idx < 18 # 18/30 = 60% correct + + config = PoolConfig( + task_types=("Action Reasoning",), + seed=42, + baseline_run_id="baseline_v2", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, + test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + + assert len(pools.diagnosis) == 20 + assert len(pools.validation) == 10 + + diag_correct = sum(1 for q in pools.diagnosis if correctness[q.question_id]) + val_correct = sum(1 for q in pools.validation if correctness[q.question_id]) + # 18 correct: floor(18 * 20/30) = 12 train, 6 val + assert diag_correct == 12 + assert val_correct == 6 + + def test_per_category_all_correct_degrades(self) -> None: + """某类全部 correct（0 wrong）时退化为非分层 random 20/10。""" + questions = [_make_question(f"q_{i:03d}", "Object Recognition") for i in range(30)] + correctness = {q.question_id: True for q in questions} + + config = PoolConfig( + task_types=None, seed=42, baseline_run_id="r", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + assert len(pools.diagnosis) == 20 + assert len(pools.validation) == 10 + + def test_per_category_missing_correctness_fails(self) -> None: + """correctness 不完整时 fail-fast。""" + questions = [_make_question(f"q_{i:03d}", "Object Recognition") for i in range(30)] + correctness = {q.question_id: True for q in questions[:25]} # 缺 5 题 + + config = PoolConfig( + task_types=None, seed=42, baseline_run_id="r", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + with pytest.raises(ValueError, match="correctness 缺失"): + strategy.build(questions, correctness, config) + + def test_per_category_task_types_filter(self) -> None: + """task_types 过滤只处理指定类别。""" + questions = _make_per_category_questions() + correctness = {q.question_id: True for q in questions} + + config = PoolConfig( + task_types=("Action Reasoning", "Scene Understanding"), + seed=42, baseline_run_id="r", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + assert len(pools.diagnosis) == 40 # 2 类 × 20 + assert len(pools.validation) == 20 # 2 类 × 10 + types_in_diag = {q.task_type for q in pools.diagnosis} + assert types_in_diag == {"Action Reasoning", "Scene Understanding"} +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy -v` +Expected: FAIL with `ImportError: cannot import name 'PerCategoryPoolStrategy'` + +- [ ] **Step 3: 实现 PerCategoryPoolStrategy** + +在 `app/harness/pools.py` 中追加： + +```python +import math +from loguru import logger + + +class PerCategoryPoolStrategy: + """Per-category correctness 分层策略。 + + 每个 task_type 内部按 correct/wrong 分层， + 各自按 train_ratio 比例分配到 train(diagnosis) 和 val 池。 + test 从外部 benchmark 加载。 + """ + + def build( + self, + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> Pools: + """按类别分层构建 train/val 池。 + + 参数: + questions: 题目全集（可含多类别）。 + correctness: question_id -> 基线是否答对。 + config: 池配置（使用 task_types, seed, train_ratio, test_questions_dir）。 + + 返回: + Pools（diagnosis=train 合并, validation=val 合并, test=外部 benchmark）。 + """ + filtered = questions + if config.task_types is not None: + allowed = set(config.task_types) + filtered = [q for q in questions if q.task_type in allowed] + + # 按 task_type 分组 + by_type: dict[str, list[GeneratedQuestion]] = {} + for q in filtered: + by_type.setdefault(q.task_type, []).append(q) + + rng = random.Random(config.seed) + all_train: list[GeneratedQuestion] = [] + all_val: list[GeneratedQuestion] = [] + + for task_type in sorted(by_type): + type_qs = by_type[task_type] + train_qs, val_qs = self._split_one_category( + type_qs, correctness, config.train_ratio, rng, task_type, + ) + all_train.extend(train_qs) + all_val.extend(val_qs) + + # test 从外部 benchmark 加载 + test_qs: list[GeneratedQuestion] = [] + if config.test_questions_dir is not None: + from app.question_gen import load_benchmark + all_test = load_benchmark(config.test_questions_dir) + if config.task_types is not None: + allowed = set(config.task_types) + test_qs = [q for q in all_test if q.task_type in allowed] + else: + test_qs = all_test + if not test_qs: + logger.warning("test 池为空：test_questions_dir 中无匹配的 task_type") + + # baseline_val_accuracy + val_correct = sum(1 for q in all_val if correctness.get(q.question_id)) + baseline_val_acc = val_correct / len(all_val) if all_val else 0.0 + + # 汇总 correctness + all_correctness = { + q.question_id: correctness.get(q.question_id, False) + for q in all_train + all_val + test_qs + } + + return Pools( + diagnosis=all_train, + validation=all_val, + test=test_qs, + baseline_run_id=config.baseline_run_id, + baseline_val_accuracy=baseline_val_acc, + correctness=all_correctness, + ) + + def _split_one_category( + self, + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + train_ratio: float, + rng: random.Random, + task_type: str, + ) -> tuple[list[GeneratedQuestion], list[GeneratedQuestion]]: + """单类别按 correctness 分层划分。 + + 参数: + questions: 该类别全部题目。 + correctness: question_id -> 基线是否答对。 + train_ratio: train/(train+val) 比例。 + rng: 随机数发生器。 + task_type: 类别名（用于日志）。 + + 返回: + (train 题目列表, val 题目列表)。 + """ + n_total = len(questions) + n_train = round(n_total * train_ratio) + n_val = n_total - n_train + + # 校验 correctness 完整性 + missing = [q.question_id for q in questions if q.question_id not in correctness] + if missing: + raise ValueError( + f"类别 {task_type!r} 的 correctness 缺失 {len(missing)} 题: " + + ", ".join(missing[:5]) + + ("..." if len(missing) > 5 else "") + ) + + correct = [q for q in questions if correctness[q.question_id]] + wrong = [q for q in questions if not correctness[q.question_id]] + + # 边界：全 correct 或全 wrong → 退化为非分层 + if not wrong or not correct: + logger.warning( + "类别 {!r} 全部 {} ({} 题)，退化为非分层 random {}/{}", + task_type, + "correct" if not wrong else "wrong", + n_total, n_train, n_val, + ) + shuffled = list(questions) + rng.shuffle(shuffled) + return shuffled[:n_train], shuffled[n_train:] + + # 按比例分配 correct/wrong 到 train + n_correct = len(correct) + train_correct = math.floor(n_correct * n_train / n_total) + train_wrong = n_train - train_correct + val_correct = n_correct - train_correct + val_wrong = len(wrong) - train_wrong + + assert train_correct + train_wrong == n_train + assert val_correct + val_wrong == n_val + + rng.shuffle(correct) + rng.shuffle(wrong) + + train_qs = correct[:train_correct] + wrong[:train_wrong] + val_qs = correct[train_correct:] + wrong[train_wrong:] + return train_qs, val_qs + + def build_incremental( + self, + new_task_types: list[str], + questions: list[GeneratedQuestion], + correctness: dict[str, bool], + config: PoolConfig, + ) -> dict[str, dict[str, list[str]]]: + """增量构建新类别的 train/val 划分。 + + 参数: + new_task_types: 待新增的类别列表。 + questions: 题目全集。 + correctness: question_id -> 基线是否答对。 + config: 池配置。 + + 返回: + {task_type: {"train": [qid, ...], "val": [qid, ...]}}。 + """ + rng = random.Random(config.seed) + result: dict[str, dict[str, list[str]]] = {} + + by_type: dict[str, list[GeneratedQuestion]] = {} + for q in questions: + if q.task_type in new_task_types: + by_type.setdefault(q.task_type, []).append(q) + + for task_type in sorted(by_type): + train_qs, val_qs = self._split_one_category( + by_type[task_type], correctness, config.train_ratio, rng, task_type, + ) + result[task_type] = { + "train": [q.question_id for q in train_qs], + "val": [q.question_id for q in val_qs], + } + return result +``` + +在文件头部添加 `import math` 和 `import random`（`random` 已有，`math` 需新增）。 + +- [ ] **Step 4: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py::TestPerCategoryPoolStrategy -v` +Expected: 全部 PASS + +- [ ] **Step 5: 运行全部池测试确认无回归** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py -v` +Expected: 全部 PASS + +- [ ] **Step 6: 提交** + +```bash +git add app/harness/pools.py tests/unit/test_harness_pools.py +``` + +Commit message: `feat(harness): add PerCategoryPoolStrategy with correctness-stratified 2:1 split` + +--- + +### Task 5: app/harness/config.py — RunConfig 新增字段（原 Task 6，提前以消除前向引用） + +**Files:** +- Modify: `app/harness/pools.py:170-288` +- Test: `tests/unit/test_harness_pools.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_harness_pools.py` 追加： + +```python +class TestPerCategorySaveLoad: + """per_category 格式的 pools.json 冻结/加载。""" + + def test_save_load_per_category_roundtrip(self, tmp_path: Path) -> None: + """per_category 模式 save → load 往返一致。""" + questions = _make_per_category_questions() + correctness = {q.question_id: True for q in questions} + config = PoolConfig( + task_types=("Action Reasoning",), + seed=42, baseline_run_id="baseline_v2", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, test_questions_dir=None, + ) + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + + pools_path = tmp_path / "pools.json" + save_pools(pools, pools_path, split_mode="per_category", config=config) + loaded = load_pools(pools_path) + + assert loaded.baseline_run_id == pools.baseline_run_id + assert len(loaded.diagnosis) == len(pools.diagnosis) + assert len(loaded.validation) == len(pools.validation) + + def test_load_per_category_rejects_mismatched_config(self, tmp_path: Path) -> None: + """加载时 seed/train_ratio 不匹配 → 报错。""" + data = { + "split_mode": "per_category", + "train_ratio": 0.667, + "seed": 42, + "baseline_run_id": "r1", + "baseline_val_accuracy": 0.5, + "correctness": {}, + "categories": {}, + "diagnosis": [], + "validation": [], + "test": [], + } + pools_path = tmp_path / "pools.json" + pools_path.write_text(json.dumps(data), encoding="utf-8") + + # 正常加载应通过 + loaded = load_pools(pools_path) + assert loaded.baseline_run_id == "r1" +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py::TestPerCategorySaveLoad -v` +Expected: FAIL（`save_pools` 不接受 `split_mode` 参数） + +- [ ] **Step 3: 扩展 save_pools 和 load_pools** + +修改 `save_pools` 签名和实现，支持 per_category 格式： + +```python +def save_pools( + pools: Pools, + path: Path, + *, + split_mode: str = "global", + config: PoolConfig | None = None, +) -> None: + """将三池及基线指标冻结为 JSON。 + + 参数: + pools: 待冻结的三池。 + path: 目标 JSON 文件路径。 + split_mode: 分割模式标签（"global" / "per_category"）。 + config: 池配置（per_category 模式下需要记录 seed/train_ratio）。 + """ + data: dict = { + "split_mode": split_mode, + "baseline_run_id": pools.baseline_run_id, + "baseline_val_accuracy": pools.baseline_val_accuracy, + "correctness": pools.correctness, + "diagnosis": [_q_to_dict(q) for q in pools.diagnosis], + "validation": [_q_to_dict(q) for q in pools.validation], + "test": [_q_to_dict(q) for q in pools.test], + } + if split_mode == "per_category" and config is not None: + # 按类别记录 train/val qid 分配 + categories: dict[str, dict[str, list[str]]] = {} + for q in pools.diagnosis: + categories.setdefault(q.task_type, {"train": [], "val": []})["train"].append( + q.question_id + ) + for q in pools.validation: + categories.setdefault(q.task_type, {"train": [], "val": []})["val"].append( + q.question_id + ) + data["categories"] = categories + data["seed"] = config.seed + data["train_ratio"] = config.train_ratio + if config.test_questions_dir is not None: + data["test_source"] = str(config.test_questions_dir) + path.write_text( + json.dumps(data, ensure_ascii=False, indent=2), + encoding="utf-8", + ) +``` + +修改 `load_pools` 以兼容新旧格式： + +```python +def load_pools(path: Path) -> Pools: + """从 JSON 恢复冻结的三池。兼容 global 和 per_category 格式。""" + d = json.loads(path.read_text(encoding="utf-8")) + if "test" not in d: + raise ValueError( + f"{path} 为旧格式 pools.json（缺 test 池），" + "请删除后重新切分。" + ) + return Pools( + diagnosis=[_dict_to_q(x) for x in d["diagnosis"]], + validation=[_dict_to_q(x) for x in d["validation"]], + test=[_dict_to_q(x) for x in d["test"]], + baseline_run_id=d["baseline_run_id"], + baseline_val_accuracy=d["baseline_val_accuracy"], + correctness=d.get("correctness", {}), + ) +``` + +- [ ] **Step 4: 重构 build_or_load_pools 接受 strategy** + +```python +def build_or_load_pools( + config: RunConfig, + strategy: PoolStrategy, + db_path: Path, +) -> Pools: + """构建或加载三池。 + + 参数: + config: 运行配置。 + strategy: 池构建策略实例。 + db_path: harness.db 路径（用于读取 baseline correctness）。 + + 返回: + 冻结的三池 Pools。 + """ + from app.harness.log import HarnessLog + from app.harness.workspace import resolve_paths + from app.question_gen import load_benchmark + + pools_path = config.workspace_dir / "pools.json" + + # baseline_run_id 从 seed 解析（train 的 config.run_id 是训练 ID，非 baseline） + from app.harness.workspace import resolve_paths as _resolve + _paths = _resolve(config.workspace_dir) + manifest = json.loads((_paths.workspace_dir / "manifest.json").read_text(encoding="utf-8")) + baseline_run_id = manifest.get("baseline_run_id", config.run_id or "infer_adhoc") + + if pools_path.exists(): + loaded_data = json.loads(pools_path.read_text(encoding="utf-8")) + stored_mode = loaded_data.get("split_mode", "global") + if stored_mode == "per_category": + # 一致性校验 + if loaded_data.get("seed") != getattr(_to_pool_config(config), "seed", None): + raise ValueError( + f"pools.json 的 seed({loaded_data.get('seed')}) 与当前配置不一致，" + "请删除 pools.json 重建。" + ) + # 检查是否有新类别需要增量 + existing_categories = set(loaded_data.get("categories", {}).keys()) + requested = set(config.task_types) if config.task_types else set() + new_types = requested - existing_categories + if new_types: + paths = resolve_paths(config.workspace_dir) + questions = load_benchmark(paths.questions_dir) + with HarnessLog(str(db_path), baseline_run_id) as log: + rows = log.query( + "SELECT question_id, prediction, answer FROM predictions WHERE run_id=?", + (baseline_run_id,), + ) + correctness = {r["question_id"]: r["prediction"] == r["answer"] for r in rows} + pool_config = _to_pool_config(config) + incremental = strategy.build_incremental( + list(new_types), questions, correctness, pool_config, + ) + loaded_data["categories"].update(incremental) + # 重建 pools 并保存 + pools_path.write_text( + json.dumps(loaded_data, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + return load_pools(pools_path) + + # 首次构建 + paths = resolve_paths(config.workspace_dir) + questions = load_benchmark(paths.questions_dir) + with HarnessLog(str(db_path), baseline_run_id) as log: + rows = log.query( + "SELECT question_id, prediction, answer FROM predictions WHERE run_id=?", + (baseline_run_id,), + ) + correctness = {r["question_id"]: r["prediction"] == r["answer"] for r in rows} + pool_config = _to_pool_config(config) + pools = strategy.build(questions, correctness, pool_config) + save_pools(pools, pools_path, split_mode=config.pool_split_mode, config=pool_config) + return pools +``` + +新增辅助函数 `_to_pool_config`： + +```python +def _to_pool_config(config: RunConfig) -> PoolConfig: + """从 RunConfig 提取 PoolConfig。""" + return PoolConfig( + task_types=config.task_types, + seed=0, + baseline_run_id=config.run_id if config.run_id else "infer_adhoc", + diag_size=config.diag_size, + diag_correct_ratio=config.diag_correct_ratio, + val_size=config.val_size, + val_correct_ratio=config.val_correct_ratio, + test_size=config.test_size, + eval_min_per_class=config.eval_min_per_class, + train_ratio=config.train_ratio, + test_questions_dir=( + Path(config.store_dir) / "questions" / config.test_questions + if hasattr(config, "test_questions") and config.test_questions + else None + ), + ) +``` + +- [ ] **Step 5: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py -v` +Expected: 全部 PASS + +- [ ] **Step 6: 提交** + +```bash +git add app/harness/pools.py tests/unit/test_harness_pools.py +``` + +Commit message: `feat(harness): refactor build_or_load_pools to accept PoolStrategy + per_category freeze format` + +--- + +### Task 6: pools.py — 重构 build_or_load_pools + per_category 冻结格式（原 Task 5，后移以依赖 Task 5 的 RunConfig 字段） + +**Files:** +- Modify: `app/harness/config.py:131-138` (在有默认值字段区追加) +- Modify: `app/harness/config.py:259-298` (校验逻辑调整) +- Test: `tests/unit/test_harness_config.py` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/test_harness_config.py` 末尾追加： + +```python +class TestRunConfigNewFields: + """RunConfig 新增字段校验。""" + + def test_pool_split_mode_valid(self, base_config: dict) -> None: + """pool_split_mode 合法值。""" + base_config["pool_split_mode"] = "per_category" + cfg = RunConfig(**base_config) + assert cfg.pool_split_mode == "per_category" + + def test_pool_split_mode_invalid(self, base_config: dict) -> None: + """pool_split_mode 非法值应报错。""" + base_config["pool_split_mode"] = "invalid" + from app.harness.config import _validate + cfg = RunConfig(**base_config) + with pytest.raises(ValueError, match="pool_split_mode"): + _validate(cfg) + + def test_task_types_tuple(self, base_config: dict) -> None: + """task_types 接受 tuple。""" + base_config["task_types"] = ("Action Reasoning", "Scene Understanding") + cfg = RunConfig(**base_config) + assert cfg.task_types == ("Action Reasoning", "Scene Understanding") + + def test_task_types_none(self, base_config: dict) -> None: + """task_types 默认 None。""" + cfg = RunConfig(**base_config) + assert cfg.task_types is None + + def test_train_ratio_range(self, base_config: dict) -> None: + """train_ratio 必须在 (0, 1) 内。""" + base_config["train_ratio"] = 1.5 + from app.harness.config import _validate + cfg = RunConfig(**base_config) + with pytest.raises(ValueError, match="train_ratio"): + _validate(cfg) + + def test_per_category_skips_val_size_check(self, base_config: dict) -> None: + """per_category 模式跳过 val_size >= eval_min_per_class * 12 的校验。""" + base_config["pool_split_mode"] = "per_category" + base_config["val_size"] = 1 # 远小于 12*2=24，全局模式会报错 + from app.harness.config import _validate + cfg = RunConfig(**base_config) + _validate(cfg) # 不应报错 +``` + +注意：需要检查已有测试的 `base_config` fixture，确保它包含新字段的默认值。 + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_config.py::TestRunConfigNewFields -v` +Expected: FAIL + +- [ ] **Step 3: 修改 RunConfig** + +在 `app/harness/config.py` 的 `RunConfig` 有默认值区（line 131-138 之后）追加： + +```python + task_types: tuple[str, ...] | None = None + pool_split_mode: str = "global" + train_ratio: float = 0.667 + test_questions: str = "benchmarks/Video-MME" +``` + +在 `_VALID_POOL_SPLIT_MODES` 常量区追加： + +```python +_VALID_POOL_SPLIT_MODES = {"global", "per_category"} +``` + +- [ ] **Step 4: 修改校验逻辑** + +在 `_validate_basic` 中追加 `pool_split_mode` 和 `train_ratio` 校验： + +```python + if config.pool_split_mode not in _VALID_POOL_SPLIT_MODES: + raise ValueError( + f"pool_split_mode 必须为 {_VALID_POOL_SPLIT_MODES} 之一，" + f"实际: {config.pool_split_mode!r}" + ) + if not (0 < config.train_ratio < 1): + raise ValueError( + f"train_ratio 必须在 (0, 1) 内，实际: {config.train_ratio}" + ) +``` + +修改 `_validate_minibatch` 中的 `val_size` 校验，per_category 模式跳过： + +将现有的： +```python + floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT + if config.val_size < floor: +``` + +改为： +```python + if config.pool_split_mode != "per_category": + floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT + if config.val_size < floor: +``` + +- [ ] **Step 5: 更新 base_config fixture** + +在 `tests/unit/test_harness_config.py` 的 `base_config` fixture 中追加新字段默认值： + +```python + "task_types": None, + "pool_split_mode": "global", + "train_ratio": 0.667, + "test_questions": "benchmarks/Video-MME", +``` + +- [ ] **Step 6: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_config.py -v` +Expected: 全部 PASS + +- [ ] **Step 7: 提交** + +```bash +git add app/harness/config.py tests/unit/test_harness_config.py +``` + +Commit message: `feat(harness): add task_types, pool_split_mode, train_ratio, test_questions to RunConfig` + +--- + +### Task 7: config/default.yaml — 新增配置项 + +**Files:** +- Modify: `config/default.yaml` + +- [ ] **Step 1: 在 harness 段末尾追加** + +在 `config/default.yaml` 的 `harness:` 段 `early_stop_patience: 8` 和 `use_slow_momentum: true` 之后追加： + +```yaml + # 池构建策略 + pool_split_mode: global # global | per_category + train_ratio: 0.667 # per_category 模式下 train/(train+val) 比例 + test_questions: "benchmarks/Video-MME" # test 池的题目来源 +``` + +- [ ] **Step 2: 验证 YAML 可解析** + +Run: `conda activate Video-Tree-TRM && python -c "import yaml; yaml.safe_load(open('config/default.yaml'))"` +Expected: 无输出（成功） + +- [ ] **Step 3: 提交** + +```bash +git add config/default.yaml +``` + +Commit message: `config: add pool_split_mode, train_ratio, test_questions to default.yaml` + +--- + +### Task 8: app/harness/log.py — _runs 表 upsert + +**Files:** +- Modify: `app/harness/log.py:72-78` + +- [ ] **Step 1: 写失败测试** + +在 `tests/unit/` 新建或追加到已有 test 文件： + +```python +# tests/unit/test_harness_log.py + +from app.harness.log import HarnessLog + + +class TestHarnessLogUpsert: + """_runs 表 upsert 行为。""" + + def test_same_run_id_updates_started_at(self, tmp_path) -> None: + """同 run_id 第二次创建 HarnessLog 应更新 started_at。""" + db = str(tmp_path / "test.db") + with HarnessLog(db, "run_1", git_sha="abc") as log1: + rows = log1.query("SELECT started_at FROM _runs WHERE run_id='run_1'") + first_time = rows[0]["started_at"] + + import time + time.sleep(0.01) + + with HarnessLog(db, "run_1", git_sha="abc") as log2: + rows = log2.query("SELECT started_at FROM _runs WHERE run_id='run_1'") + second_time = rows[0]["started_at"] + + assert second_time > first_time + # 应只有一行 + with HarnessLog(db, "run_1") as log3: + rows = log3.query("SELECT COUNT(*) as cnt FROM _runs WHERE run_id='run_1'") + assert rows[0]["cnt"] == 1 +``` + +- [ ] **Step 2: 运行测试确认失败** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_log.py::TestHarnessLogUpsert -v` +Expected: FAIL（当前 INSERT OR IGNORE 不更新 started_at） + +- [ ] **Step 3: 修改 HarnessLog.__init__** + +将 `app/harness/log.py` line 72-78 的： + +```python + self._conn.execute( + "INSERT OR IGNORE INTO _runs" + " (run_id, git_sha, started_at, config, status)" + " VALUES (?, ?, ?, ?, ?)", + (run_id, resolved_sha, _now_iso(), config_json, "running"), + ) +``` + +改为： + +```python + self._conn.execute( + "INSERT INTO _runs" + " (run_id, git_sha, started_at, config, status)" + " VALUES (?, ?, ?, ?, ?)" + " ON CONFLICT(run_id) DO UPDATE SET" + " started_at=excluded.started_at," + " config=excluded.config," + " status=excluded.status", + (run_id, resolved_sha, _now_iso(), config_json, "running"), + ) +``` + +- [ ] **Step 4: 运行测试确认通过** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_log.py -v` +Expected: PASS + +- [ ] **Step 5: 提交** + +```bash +git add app/harness/log.py tests/unit/test_harness_log.py +``` + +Commit message: `fix(harness): change _runs INSERT OR IGNORE to ON CONFLICT DO UPDATE for incremental infer` + +--- + +### Task 9: main.py — task_types 纳入 config + train 接线 + +**Files:** +- Modify: `main.py:258,293-298` + +- [ ] **Step 1: 将 task_types 纳入 cli_overrides** + +修改 `main.py` line 258 的： + +```python + cli_overrides = {k: v for k, v in vars(args).items() if k not in ("config", "task_types")} +``` + +改为： + +```python + cli_args = vars(args) + # task_types: list -> tuple（RunConfig 要求 tuple） + if cli_args.get("task_types") is not None: + cli_args["task_types"] = tuple(cli_args["task_types"]) + cli_overrides = {k: v for k, v in cli_args.items() if k != "config"} +``` + +- [ ] **Step 2: 添加 --pool-split-mode CLI 参数** + +在 `_build_parser()` 中追加： + +```python + parser.add_argument( + "--pool-split-mode", + choices=["global", "per_category"], + dest="pool_split_mode", + ) + parser.add_argument("--train-ratio", type=float, dest="train_ratio") + parser.add_argument("--test-questions", type=str, dest="test_questions") +``` + +- [ ] **Step 3: 接线 train 模式** + +修改 `main.py` line 293-298 的： + +```python + if config.mode == "infer": + task_types = getattr(args, "task_types", None) + result = asyncio.run(runner.infer(task_types=task_types)) + _log_result(result) + else: + raise SystemExit(f"模式 {config.mode!r} 尚未实现") +``` + +改为： + +```python + if config.mode == "infer": + result = asyncio.run(runner.infer(task_types=config.task_types)) + _log_result(result) + elif config.mode == "train": + from app.harness.pools import ( + GlobalPoolStrategy, + PerCategoryPoolStrategy, + build_or_load_pools, + ) + from app.harness.workspace import resolve_paths + + strategy = ( + PerCategoryPoolStrategy() + if config.pool_split_mode == "per_category" + else GlobalPoolStrategy() + ) + paths = resolve_paths(config.workspace_dir) + pools = build_or_load_pools(config, strategy, paths.db_path) + asyncio.run(runner.train(pools)) + else: + raise SystemExit(f"模式 {config.mode!r} 尚未实现") +``` + +- [ ] **Step 4: 验证 CLI 解析** + +Run: `conda activate Video-Tree-TRM && python main.py harness --help` +Expected: 输出中包含 `--pool-split-mode`, `--train-ratio`, `--test-questions`, `--task-types` + +- [ ] **Step 5: 提交** + +```bash +git add main.py +``` + +Commit message: `feat(cli): wire train mode with PoolStrategy selection and task_types in RunConfig` + +--- + +### Task 10: 集成测试 + +**Files:** +- Create: `tests/integration/test_pool_strategy.py` + +- [ ] **Step 1: 编写集成测试** + +```python +"""PerCategoryPoolStrategy 端到端集成测试。 + +验证从构造题目 → 伪造 baseline → 池构建 → 冻结 → 加载的完整流程。 +""" + +from __future__ import annotations + +import json +from collections import Counter +from pathlib import Path + +import pytest + +from app.harness.pools import ( + PerCategoryPoolStrategy, + load_pools, + save_pools, +) +from core.types import GeneratedQuestion, PoolConfig + + +def _make_question(qid: str, task_type: str) -> GeneratedQuestion: + """构造测试用 GeneratedQuestion。""" + return GeneratedQuestion( + question_id=qid, video_id="v1", task_type=task_type, + question=f"Q {qid}?", + options=("A. a", "B. b", "C. c", "D. d"), + answer="A", source_nodes=("n1",), difficulty="medium", + ) + + +class TestPerCategoryE2E: + """端到端：构建 → 冻结 → 加载 → 校验。""" + + def test_full_flow(self, tmp_path: Path) -> None: + """完整流程：12 类各 30 题 → 策略构建 → 冻结 → 加载 → 三池校验。""" + task_types = [ + "Action Prediction", "Action Reasoning", "Action Recognition", + "Action Sequence", "Causal Reasoning", "Event Reasoning", + "Object Interaction", "Object Reasoning", "Object Recognition", + "Scene Understanding", "Spatial Reasoning", "Temporal Reasoning", + ] + questions = [] + for tt in task_types: + for i in range(30): + questions.append(_make_question(f"{tt}_{i:03d}", tt)) + + # 每类前 18 correct，后 12 wrong + correctness = {} + for q in questions: + idx = int(q.question_id.split("_")[-1]) + correctness[q.question_id] = idx < 18 + + config = PoolConfig( + task_types=None, seed=42, baseline_run_id="baseline_v2", + diag_size=0, diag_correct_ratio=0.0, + val_size=0, val_correct_ratio=0.0, + test_size=0, eval_min_per_class=0, + train_ratio=20 / 30, test_questions_dir=None, + ) + + strategy = PerCategoryPoolStrategy() + pools = strategy.build(questions, correctness, config) + + # 验证总量 + assert len(pools.diagnosis) == 240 + assert len(pools.validation) == 120 + + # 验证逐类均匀 + diag_by_type = Counter(q.task_type for q in pools.diagnosis) + val_by_type = Counter(q.task_type for q in pools.validation) + for tt in task_types: + assert diag_by_type[tt] == 20 + assert val_by_type[tt] == 10 + + # 验证互斥 + diag_ids = {q.question_id for q in pools.diagnosis} + val_ids = {q.question_id for q in pools.validation} + assert diag_ids & val_ids == set() + + # 验证 correctness 对齐 + for tt in task_types: + tt_diag = [q for q in pools.diagnosis if q.task_type == tt] + tt_val = [q for q in pools.validation if q.task_type == tt] + diag_ratio = sum(1 for q in tt_diag if correctness[q.question_id]) / len(tt_diag) + val_ratio = sum(1 for q in tt_val if correctness[q.question_id]) / len(tt_val) + assert abs(diag_ratio - val_ratio) < 0.05, ( + f"{tt}: train ratio {diag_ratio:.2f} vs val ratio {val_ratio:.2f}" + ) + + # 冻结 → 加载 + pools_path = tmp_path / "pools.json" + save_pools(pools, pools_path, split_mode="per_category", config=config) + loaded = load_pools(pools_path) + assert len(loaded.diagnosis) == 240 + assert len(loaded.validation) == 120 + + # 验证冻结格式 + data = json.loads(pools_path.read_text()) + assert data["split_mode"] == "per_category" + assert "categories" in data + assert len(data["categories"]) == 12 +``` + +- [ ] **Step 2: 运行测试** + +Run: `conda activate Video-Tree-TRM && pytest tests/integration/test_pool_strategy.py -v` +Expected: PASS + +- [ ] **Step 3: 运行全量测试确认无回归** + +Run: `conda activate Video-Tree-TRM && pytest tests/unit/test_harness_pools.py tests/unit/test_harness_config.py tests/unit/test_core_types.py tests/unit/test_core_protocols.py -v` +Expected: 全部 PASS + +- [ ] **Step 4: 提交** + +```bash +git add tests/integration/test_pool_strategy.py +``` + +Commit message: `test(integration): add PerCategoryPoolStrategy end-to-end test` + +--- + +### Task 11: lint + 最终验证 + +**Files:** 无新增 + +- [ ] **Step 1: 代码格式化** + +Run: `conda activate Video-Tree-TRM && ruff format app/ core/ tests/` +Run: `conda activate Video-Tree-TRM && ruff check app/ core/ tests/ --fix` + +- [ ] **Step 2: 全量测试** + +Run: `conda activate Video-Tree-TRM && pytest tests/ -v --tb=short` +Expected: 全部 PASS + +- [ ] **Step 3: 修复任何问题后提交** + +```bash +git add -A +``` + +Commit message: `chore: lint and format per-category pool strategy implementation` + +--- + +## 保真校验 + +本计划不涉及核心算法迁移（13 项均不涉及）。PoolStrategy 是新增抽象，`GlobalPoolStrategy` 封装的 `build_pools` 保持原有逻辑不变。保真校验不适用。 diff --git a/research-wiki/plans/2026-07-14-task-type-strategy-framework.md b/research-wiki/plans/2026-07-14-task-type-strategy-framework.md new file mode 100644 index 0000000..b1fdff9 --- /dev/null +++ b/research-wiki/plans/2026-07-14-task-type-strategy-framework.md @@ -0,0 +1,751 @@ +# TaskTypeStrategy 框架实现计划 (Plan A) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 将出题管线从 QuestionFamilySpec 替换为 TaskTypeStrategy 接口，12 个题型（含 AR 临时 Base）用 BaseTaskTypeStrategy 封装现有 family 行为，pipeline/sampler/generator/gates/store 签名适配。 + +**Architecture:** 新建 `strategy.py`（Protocol + SubPattern + Base + 注册表），修改 pipeline/sampler/generator/gates/store 5 个文件的函数签名，从 `family_spec` 参数改为 strategy 提供的离散参数。BaseTaskTypeStrategy 内部委托给绑定的 QuestionFamilySpec，确保 12 个题型行为不变。AR 临时绑定 VISUAL family（Plan B 替换为特化策略）。 + +**Codex 审查修复:** +- C1: sub_pattern 空值判护 — `sub_pattern.x if sub_pattern else strategy.x` +- C2: AR 在 Plan A 临时绑定 VISUAL family，Plan B 替换为 ActionRecognitionStrategy +- C3: sampler 异常消息同步改为使用 task_type +- C4: run_store 加幂等 ALTER TABLE 迁移 +- I1: pipeline 显式调用 extra_gates 并合并结果 +- I2: 移除 build_prompt_context，改为离散参数（prompt_template/strategy_name/skill_target/sub_pattern_instruction） + +**Tech Stack:** Python 3.11, pytest, Protocol (typing) + +**关联设计:** `research-wiki/designs/2026-07-14-task-type-strategy-design.md` + +**范围:** 仅 Plan A（框架 + Base）。ActionRecognitionStrategy 特化实现在 Plan B。 + +--- + +### Task 1: strategy.py — Protocol + SubPattern + BaseTaskTypeStrategy + 注册表 + +**Files:** +- Create: `app/question_gen/strategy.py` +- Test: `tests/unit/test_strategy.py` + +- [ ] **Step 1: 写失败测试 — Protocol 和 Base 行为** + +```python +# tests/unit/test_strategy.py +"""TaskTypeStrategy Protocol 与 BaseTaskTypeStrategy 单元测试。""" + +from __future__ import annotations + +import random +from pathlib import Path + +import pytest + +from app.question_gen.families import ( + ENUMERATION_FAMILY, + REASONING_FAMILY, + RETRIEVAL_FAMILY, + SPATIAL_FAMILY, + VISUAL_FAMILY, +) +from app.question_gen.strategy import ( + BaseTaskTypeStrategy, + SubPattern, + get_strategy, + register_strategy, +) + + +class TestBaseTaskTypeStrategy: + """BaseTaskTypeStrategy 封装 family 行为。""" + + def test_task_type_and_strategy_name(self): + """task_type 和 strategy_name 正确返回。""" + s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY) + assert s.task_type == "Object Recognition" + assert s.strategy_name == "RETRIEVAL" + + def test_sampling_from_family(self): + """采样参数从绑定的 family 读取。""" + s = BaseTaskTypeStrategy(task_type="Temporal Reasoning", family=ENUMERATION_FAMILY) + assert s.sampling_level == 1 + assert s.sampling_constraint.min_l3_nodes == 5 + + def test_skill_target_from_family(self): + """skill_target 从 family 读取。""" + s = BaseTaskTypeStrategy(task_type="Action Reasoning", family=REASONING_FAMILY) + assert s.skill_target == "M2" + + def test_leak_probe_template(self): + """leak_probe_template 从 family.leak_profile 读取。""" + s = BaseTaskTypeStrategy(task_type="Spatial Reasoning", family=SPATIAL_FAMILY) + assert s.leak_probe_template == "gate_leak_spatial.md" + + def test_prompt_template_from_family(self): + """prompt_template 从 family 读取。""" + s = BaseTaskTypeStrategy(task_type="OCR Problems", family=VISUAL_FAMILY) + assert s.prompt_template == "visual.md" + + def test_select_sub_pattern_returns_none(self): + """BaseTaskTypeStrategy 无子模式。""" + s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY) + rng = random.Random(42) + assert s.select_sub_pattern(rng) is None + + def test_extra_gates_returns_empty(self): + """BaseTaskTypeStrategy 无额外 gate。""" + s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY) + assert s.extra_gates(None) == [] + + +class TestSamplingLevelMapping: + """BaseTaskTypeStrategy 的 sampling_level 从 _TASK_TYPE_TO_LEVEL 读取。""" + + def test_l3_types(self): + """L3 题型。""" + for tt in ("Object Recognition",): + s = BaseTaskTypeStrategy(task_type=tt, family=RETRIEVAL_FAMILY) + assert s.sampling_level == 3, f"{tt} should be L3" + + def test_l2_types(self): + """L2 题型。""" + s = BaseTaskTypeStrategy(task_type="Action Reasoning", family=REASONING_FAMILY) + assert s.sampling_level == 2 + + def test_l1_types(self): + """L1 题型。""" + s = BaseTaskTypeStrategy(task_type="Temporal Reasoning", family=ENUMERATION_FAMILY) + assert s.sampling_level == 1 + + +class TestStrategyRegistry: + """注册表查找。""" + + def test_get_unregistered_returns_base(self): + """未注册题型返回 BaseTaskTypeStrategy。""" + s = get_strategy("Spatial Reasoning") + assert isinstance(s, BaseTaskTypeStrategy) + assert s.task_type == "Spatial Reasoning" + + def test_register_and_get(self): + """注册后 get 返回注册的策略。""" + custom = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY) + register_strategy(custom) + assert get_strategy("Object Recognition") is custom + + +class TestSubPattern: + """SubPattern 数据类。""" + + def test_frozen(self): + """SubPattern 不可变。""" + sp = SubPattern( + name="test", + weight=0.5, + sampling_level_override=None, + constraint_override=None, + instruction="test instruction", + positive_examples=[], + negative_examples=[], + distractor_rules="", + ) + with pytest.raises(AttributeError): + sp.name = "changed" +``` + +- [ ] **Step 2: 运行测试验证失败** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/test_strategy.py -v +``` + +预期：ImportError — `app.question_gen.strategy` 不存在。 + +- [ ] **Step 3: 实现 strategy.py** + +```python +# app/question_gen/strategy.py +"""题型出题策略 — Clean Architecture 的 Strategy 层。 + +将出题管线从 5 个粗粒度 QuestionFamilySpec 替换为 12 个题型级别的 +TaskTypeStrategy。BaseTaskTypeStrategy 封装现有 family 行为， +特化策略（如 ActionRecognitionStrategy）自包含。 + +典型用法:: + + strategy = get_strategy("Action Reasoning") + level = strategy.sampling_level + constraint = strategy.sampling_constraint +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable + +from app.question_gen.families import ( + ENUMERATION_FAMILY, + REASONING_FAMILY, + RETRIEVAL_FAMILY, + SPATIAL_FAMILY, + VISUAL_FAMILY, + QuestionFamilySpec, + SamplingConstraint, +) +from app.question_gen.sampler_v2 import _TASK_TYPE_TO_LEVEL + +if TYPE_CHECKING: + import random + + +# --------------------------------------------------------------------------- +# SubPattern +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class SubPattern: + """出题子模式 — 靶向特定失败机制。 + + 属性: + name: 子模式标识名。 + weight: 选择权重（按比例分配）。 + sampling_level_override: 覆盖策略默认采样层级（None = 不覆盖）。 + constraint_override: 覆盖策略默认采样约束（None = 不覆盖）。 + instruction: 核心出题指令（1-3 句话）。 + positive_examples: VME 原题 few-shot 示范。 + negative_examples: 反面示例。 + distractor_rules: 干扰项构造规则。 + """ + + name: str + weight: float + sampling_level_override: int | None + constraint_override: SamplingConstraint | None + instruction: str + positive_examples: list[dict] = field(default_factory=list) + negative_examples: list[dict] = field(default_factory=list) + distractor_rules: str = "" + + +# --------------------------------------------------------------------------- +# Protocol +# --------------------------------------------------------------------------- + + +@runtime_checkable +class TaskTypeStrategy(Protocol): + """题型出题策略 — pipeline 的唯一接口。""" + + @property + def task_type(self) -> str: ... + + @property + def sampling_level(self) -> int: ... + + @property + def sampling_constraint(self) -> SamplingConstraint: ... + + @property + def prompt_template(self) -> str: ... + + @property + def strategy_name(self) -> str: ... + + @property + def skill_target(self) -> str: ... + + @property + def leak_probe_template(self) -> str: ... + + def select_sub_pattern(self, rng: random.Random) -> SubPattern | None: ... + + def build_prompt_context( + self, material: Any, sub_pattern: SubPattern | None + ) -> dict: ... + + def extra_gates(self, candidate: Any) -> list: ... + + +# --------------------------------------------------------------------------- +# BaseTaskTypeStrategy +# --------------------------------------------------------------------------- + +# 消歧绑定表：多归属题型确定性绑定到一个 family +_TASK_TYPE_TO_FAMILY: dict[str, QuestionFamilySpec] = { + "Action Recognition": VISUAL_FAMILY, # Plan A 临时绑定；Plan B 替换为特化策略 + "Object Recognition": RETRIEVAL_FAMILY, + "Object Reasoning": REASONING_FAMILY, + "Action Reasoning": REASONING_FAMILY, + "Attribute Perception": VISUAL_FAMILY, + "OCR Problems": VISUAL_FAMILY, + "Counting Problem": ENUMERATION_FAMILY, + "Information Synopsis": REASONING_FAMILY, + "Temporal Reasoning": ENUMERATION_FAMILY, + "Temporal Perception": ENUMERATION_FAMILY, + "Spatial Reasoning": SPATIAL_FAMILY, + "Spatial Perception": SPATIAL_FAMILY, +} + + +class BaseTaskTypeStrategy: + """封装现有 QuestionFamilySpec 行为的默认策略。 + + 所有属性委托给绑定的 family，确保未特化题型的行为不变。 + + 参数: + task_type: 题型名。 + family: 绑定的 QuestionFamilySpec。 + """ + + def __init__(self, task_type: str, family: QuestionFamilySpec) -> None: + self._task_type = task_type + self._family = family + + @property + def task_type(self) -> str: + return self._task_type + + @property + def sampling_level(self) -> int: + return _TASK_TYPE_TO_LEVEL[self._task_type] + + @property + def sampling_constraint(self) -> SamplingConstraint: + return self._family.sampling + + @property + def prompt_template(self) -> str: + return self._family.prompt_template + + @property + def strategy_name(self) -> str: + return self._family.name + + @property + def skill_target(self) -> str: + return self._family.skill_target + + @property + def leak_probe_template(self) -> str: + return self._family.leak_profile.probe_template + + def select_sub_pattern(self, rng: random.Random) -> SubPattern | None: + """BaseTaskTypeStrategy 无子模式。""" + return None + + def build_prompt_context( + self, material: Any, sub_pattern: SubPattern | None + ) -> dict: + """返回基础 prompt 上下文（family_name + prompt_template）。""" + return { + "family_name": self._family.name, + "prompt_template": self._family.prompt_template, + } + + def extra_gates(self, candidate: Any) -> list: + """BaseTaskTypeStrategy 无额外 gate。""" + return [] + + +# --------------------------------------------------------------------------- +# 注册表 +# --------------------------------------------------------------------------- + +_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {} + + +def register_strategy(strategy: TaskTypeStrategy) -> None: + """注册一个题型策略。同一 task_type 重复注册会覆盖。 + + 参数: + strategy: 实现 TaskTypeStrategy 接口的策略实例。 + """ + _STRATEGY_REGISTRY[strategy.task_type] = strategy + + +def get_strategy(task_type: str) -> TaskTypeStrategy: + """获取题型策略。未注册的自动创建 BaseTaskTypeStrategy。 + + 参数: + task_type: 题型名。 + + 返回: + TaskTypeStrategy 实例。 + + 异常: + KeyError: task_type 不在消歧绑定表和注册表中。 + """ + if task_type in _STRATEGY_REGISTRY: + return _STRATEGY_REGISTRY[task_type] + return _build_default_strategy(task_type) + + +def _build_default_strategy(task_type: str) -> BaseTaskTypeStrategy: + """根据消歧绑定表自动构造 BaseTaskTypeStrategy。 + + 参数: + task_type: 题型名。 + + 返回: + BaseTaskTypeStrategy 实例。 + + 异常: + KeyError: task_type 不在消歧绑定表中。 + """ + family = _TASK_TYPE_TO_FAMILY[task_type] + return BaseTaskTypeStrategy(task_type=task_type, family=family) +``` + +- [ ] **Step 4: 运行测试验证通过** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/test_strategy.py -v +``` + +预期：全部 PASS + +- [ ] **Step 5: 提交** + +```bash +git add app/question_gen/strategy.py tests/unit/test_strategy.py +git commit -m "feat(question_gen): add TaskTypeStrategy Protocol and BaseTaskTypeStrategy" +``` + +--- + +### Task 2: sampler_v2.py 签名适配 + +**Files:** +- Modify: `app/question_gen/sampler_v2.py:495-530` +- Test: `tests/unit/test_sampler_v2.py`（现有测试应继续通过） + +- [ ] **Step 1: 修改 sample_material_v2 签名** + +将 `family_spec: QuestionFamilySpec` 参数替换为 `level: int` + `constraint: SamplingConstraint`： + +在 `app/question_gen/sampler_v2.py` 中，修改 `sample_material_v2` 函数签名和内部使用： + +```python +def sample_material_v2( + tree: TreeIndex, + task_type: str, + used_node_ids: set[str], + rng: random.Random, + *, + level: int, + constraint: SamplingConstraint, + max_attempts: int = 10, +) -> MaterialContext: +``` + +函数体内将 `level = _TASK_TYPE_TO_LEVEL[task_type]` 和 `constraint = family_spec.sampling` 两行删除（改为直接使用参数）。 + +- [ ] **Step 2: 修改 pipeline_v2.py 中的调用点** + +在 `_process_one_slot` 中（约 371 行），将： + +```python +material = sample_material_v2( + tree=current_tree, + family_spec=slot.family, + task_type=slot.task_type, + used_node_ids=used_node_ids, + rng=rng, +) +``` + +改为： + +```python +material = sample_material_v2( + tree=current_tree, + task_type=slot.task_type, + used_node_ids=used_node_ids, + rng=rng, + level=strategy.sampling_level, + constraint=strategy.sampling_constraint, +) +``` + +（`strategy` 变量在 Task 4 中引入，此处先改签名。） + +- [ ] **Step 3: 运行 sampler 测试** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/test_sampler_v2.py -v +``` + +预期：需要更新测试中的调用签名。修复后全部 PASS。 + +- [ ] **Step 4: 提交** + +```bash +git add app/question_gen/sampler_v2.py tests/unit/test_sampler_v2.py +git commit -m "refactor(sampler): replace family_spec param with level+constraint" +``` + +--- + +### Task 3: generator_v2.py + gates.py + run_store.py 签名适配 + +**Files:** +- Modify: `app/question_gen/generator_v2.py:88-104, 112-194, 351-420` +- Modify: `app/question_gen/gates.py:399-431, 439-494` +- Modify: `app/question_gen/run_store.py:110-128, 242-290` + +- [ ] **Step 1: generator_v2.py — _load_prompt_template 改为接收文件名** + +```python +def _load_prompt_template(template_name: str) -> str: + path = _PROMPTS_DIR / template_name + if not path.exists(): + msg = f"Prompt 模板文件不存在: {path}" + raise FileNotFoundError(msg) + return path.read_text(encoding="utf-8") +``` + +- [ ] **Step 2: generator_v2.py — _build_v2_prompt 改为接收离散参数** + +```python +def _build_v2_prompt( + prompt_template: str, + strategy_name: str, + material: MaterialContext, + task_type: str, + seq: int, + *, + reject_reason: str | None = None, + sub_pattern_instruction: str | None = None, +) -> tuple[list[dict[str, str]], list[str]]: +``` + +函数体内： +- `template_content = _load_prompt_template(family_spec)` → `template_content = _load_prompt_template(prompt_template)` +- `family_spec.name` → `strategy_name` +- 在 `reject_reason` 注入之后、输出格式之前，如果 `sub_pattern_instruction` 非 None，追加： + +```python +if sub_pattern_instruction is not None: + user_parts.append(f"\n## Special Focus:\n{sub_pattern_instruction}") +``` + +- [ ] **Step 3: generator_v2.py — generate_one_v2 签名适配** + +```python +async def generate_one_v2( + vlm: VLMProvider, + tree: TreeIndex, + material: MaterialContext, + task_type: str, + seq: int, + *, + video_id: str, + prompt_template: str, + strategy_name: str, + skill_target: str, + reject_reason: str | None = None, + sub_pattern_instruction: str | None = None, + session_id: str, +) -> CandidateQuestion: +``` + +函数体内： +- `_build_v2_prompt(family_spec=family_spec, ...)` → `_build_v2_prompt(prompt_template=prompt_template, strategy_name=strategy_name, ..., sub_pattern_instruction=sub_pattern_instruction)` +- `family_spec.name` → `strategy_name` +- `family_spec.skill_target` → `skill_target` + +- [ ] **Step 4: gates.py — _gate_leak_test 和 run_gates 签名适配** + +`_gate_leak_test`： + +```python +async def _gate_leak_test( + candidate: CandidateQuestion, + leak_probe_template: str, + llm: LLMProvider, + *, + session_id: str, +) -> GateResult: +``` + +函数体内：`probe_template_name = family_spec.leak_profile.probe_template` → `probe_template_name = leak_probe_template` + +`run_gates`： + +```python +async def run_gates( + candidate: CandidateQuestion, + tree: TreeIndex, + llm: LLMProvider, + leak_probe_template: str, + postprocess: PostprocessResult, + *, + vlm: VLMProvider | None = None, + session_id: str, +) -> GateReport: +``` + +函数体内：`_gate_leak_test(candidate, family_spec, llm, ...)` → `_gate_leak_test(candidate, leak_probe_template, llm, ...)` + +- [ ] **Step 5: run_store.py — DDL 加 sub_pattern 列 + record_item 加参数** + +DDL `_DDL_ITEMS` 在 `difficulty_steps` 行后加： + +```sql + sub_pattern TEXT, +``` + +`record_item` 新增参数 `sub_pattern: str | None = None`，INSERT 语句加入该列。 + +同时在 `_init_schema` 方法中加幂等迁移（兼容已有数据库）： + +```python +# 幂等迁移：为已有表加 sub_pattern 列 +cols = {r[1] for r in self._conn.execute("PRAGMA table_info(question_gen_items)")} +if "sub_pattern" not in cols: + self._conn.execute("ALTER TABLE question_gen_items ADD COLUMN sub_pattern TEXT") +``` + +- [ ] **Step 6: 运行全部出题管线测试** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/test_generate_questions.py tests/unit/test_gates.py tests/unit/test_run_store.py tests/unit/test_sampler_v2.py tests/unit/test_families.py tests/integration/test_pipeline_v2.py tests/integration/test_cli_generate_v2.py -v --tb=short +``` + +预期：需更新测试中的调用签名。修复后全部 PASS。 + +- [ ] **Step 7: 提交** + +```bash +git add app/question_gen/generator_v2.py app/question_gen/gates.py app/question_gen/run_store.py +git commit -m "refactor(question_gen): adapt generator/gates/store signatures for strategy" +``` + +--- + +### Task 4: pipeline_v2.py 集成 — 用 strategy 替换 family + +**Files:** +- Modify: `app/question_gen/pipeline_v2.py:37, 57-73, 170-211, 300-530` + +- [ ] **Step 1: SlotAssignment 移除 family 字段，改为 strategy_name** + +```python +@dataclass(frozen=True) +class SlotAssignment: + slot_id: str + video_id: str + task_type: str + seq: int +``` + +- [ ] **Step 2: _assign_slots 移除 family 选择逻辑** + +```python +def _assign_slots( + video_ids: list[str], + task_types: list[str], + per_type: int, +) -> list[SlotAssignment]: + slots: list[SlotAssignment] = [] + global_seq = 0 + for task_type in task_types: + for i in range(per_type): + video_id = video_ids[i % len(video_ids)] + global_seq += 1 + slot_id = f"{task_type}_{global_seq:04d}" + slots.append(SlotAssignment( + slot_id=slot_id, + video_id=video_id, + task_type=task_type, + seq=global_seq, + )) + return slots +``` + +不再调用 `get_family_for_slot`，不再需要 `family_ratios` 和 `rng` 参数。 + +- [ ] **Step 3: _process_one_slot 通过 strategy 获取参数** + +在函数开头加： + +```python +from app.question_gen.strategy import get_strategy +strategy = get_strategy(slot.task_type) +sub_pattern = strategy.select_sub_pattern(rng) +``` + +替换所有 `slot.family` 引用： +- `slot.family.name` → `strategy.strategy_name` +- `slot.family.skill_target` → `strategy.skill_target` +- `sample_material_v2(...)` → 采样参数带空值判护： + ```python + level = sub_pattern.sampling_level_override if sub_pattern and sub_pattern.sampling_level_override is not None else strategy.sampling_level + constraint = sub_pattern.constraint_override if sub_pattern and sub_pattern.constraint_override is not None else strategy.sampling_constraint + sample_material_v2(..., level=level, constraint=constraint, ...) + ``` +- `generate_one_v2(...)` → `generate_one_v2(..., prompt_template=strategy.prompt_template, strategy_name=strategy.strategy_name, skill_target=strategy.skill_target, sub_pattern_instruction=sub_pattern.instruction if sub_pattern else None, ...)` +- `run_gates(...)` → `run_gates(..., leak_probe_template=strategy.leak_probe_template, ...)` +- `store.record_item(...)` → `store.record_item(..., family=strategy.strategy_name, skill_target=strategy.skill_target, sub_pattern=sub_pattern.name if sub_pattern else None, ...)` +- gate 后追加 extra_gates 调用： + ```python + extra_results = strategy.extra_gates(candidate) + if any(r.verdict == GateVerdict.FAIL for r in extra_results): + prev_reason = "; ".join(r.reason for r in extra_results if r.verdict == GateVerdict.FAIL) + continue + ``` + +- [ ] **Step 4: run_pipeline_v2 移除 family_ratios 依赖** + +`_assign_slots` 调用处移除 `config.family_ratios` 和 `rng` 参数。 + +- [ ] **Step 5: 运行集成测试** + +```bash +conda activate Video-Tree-TRM & pytest tests/integration/test_pipeline_v2.py tests/integration/test_cli_generate_v2.py -v --tb=short +``` + +预期：需更新测试中的 SlotAssignment/family 引用。修复后全部 PASS。 + +- [ ] **Step 6: 提交** + +```bash +git add app/question_gen/pipeline_v2.py +git commit -m "feat(pipeline): replace QuestionFamilySpec with TaskTypeStrategy" +``` + +--- + +### Task 5: 全量回归测试 + lint + +- [ ] **Step 1: Ruff** + +```bash +conda activate Video-Tree-TRM & ruff format app/question_gen/ tests/ && ruff check app/question_gen/ --fix +``` + +- [ ] **Step 2: 全量测试** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/ tests/integration/ -v --tb=short +``` + +预期：1160+ 全部 PASS + +- [ ] **Step 3: 提交** + +```bash +git add -A && git commit -m "chore: lint and format strategy framework changes" +``` + +--- + +## 核心算法保真校验 + +本计划不涉及核心算法迁移。修改仅限于出题管线的接口签名替换： +- sampler/generator/gates/store 的函数签名从 `family_spec` 改为离散参数 +- pipeline 的 slot 分配从 family 随机选择改为 strategy 确定性查找 +- 所有核心出题逻辑（重出循环、后处理、四门 gate 内部、去重、heavy_check）不变 + +保真校验不适用。 diff --git a/research-wiki/plans/action-recognition-training.md b/research-wiki/plans/action-recognition-training.md new file mode 100644 index 0000000..82f7cb9 --- /dev/null +++ b/research-wiki/plans/action-recognition-training.md @@ -0,0 +1,9 @@ +--- +type: plan +node_id: plan:action-recognition-training +title: "Action Recognition 单题型首次训练实验计划" +date: 2026-07-14 +--- + +# Action Recognition 单题型首次训练实验计划 + diff --git a/research-wiki/plans/per-category-pool-strategy.md b/research-wiki/plans/per-category-pool-strategy.md new file mode 100644 index 0000000..4e5e347 --- /dev/null +++ b/research-wiki/plans/per-category-pool-strategy.md @@ -0,0 +1,9 @@ +--- +type: plan +node_id: plan:per-category-pool-strategy +title: "Per-Category Pool Strategy 实现计划" +date: 2026-07-13 +--- + +# Per-Category Pool Strategy 实现计划 + diff --git a/research-wiki/plans/task-type-strategy-framework.md b/research-wiki/plans/task-type-strategy-framework.md new file mode 100644 index 0000000..ad14302 --- /dev/null +++ b/research-wiki/plans/task-type-strategy-framework.md @@ -0,0 +1,9 @@ +--- +type: plan +node_id: plan:task-type-strategy-framework +title: "TaskTypeStrategy 框架实现计划 (Plan A)" +date: 2026-07-14 +--- + +# TaskTypeStrategy 框架实现计划 (Plan A) +