docs: add TaskTypeStrategy design + Plan A framework plan
This commit is contained in:
@@ -0,0 +1,267 @@
|
|||||||
|
# Per-Category Pool Strategy 设计
|
||||||
|
|
||||||
|
> **状态**: 已批准
|
||||||
|
> **日期**: 2026-07-12
|
||||||
|
> **关联**: [app-harness-design](2026-07-07-app-harness-design.md)、[question-gen-v2-design](2026-07-11-question-gen-v2-design.md)
|
||||||
|
|
||||||
|
## 1. 动机
|
||||||
|
|
||||||
|
v2 题目生成管线为 12 个 task_type 各生成 30 题(共 360 题)。需要支持:
|
||||||
|
|
||||||
|
| 需求 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 按类增量 baseline | 类别生成完成后立即可跑 baseline inference,无需等全量完成 |
|
||||||
|
| Per-category 分层划分 | 每类 30 题按 correctness 2:1 分为 20 train / 10 val |
|
||||||
|
| 按类训练 | `--task-types` 控制训练作用域,快速验证单类别效果 |
|
||||||
|
| 外部 test 池 | test 用 Video-MME 900 道真题中同类题目,不从生成题中抽取 |
|
||||||
|
| 全局训练兼容 | 全类别联合训练时 system/tool 慢进化看全量 val 信号 |
|
||||||
|
|
||||||
|
## 2. 方案选型
|
||||||
|
|
||||||
|
| 方案 | 描述 | 取舍 |
|
||||||
|
|------|------|------|
|
||||||
|
| A. 最小改动 | 在 `build_pools` 内加 if 分支 | 改动小但两条路径耦合 |
|
||||||
|
| B. 替换为纯 per-category | 删除全局模式 | 简洁但丢失 Video-MME 900 题训练能力 |
|
||||||
|
| **C. Pool 工厂模式** ✓ | `PoolStrategy` Protocol + 具体策略 | 符合 Clean Architecture DIP/OCP;策略可独立测试替换 |
|
||||||
|
|
||||||
|
**选择 C**:池分割是真正易变的接缝(已有两种策略,未来可扩展),满足 CLAUDE.md 抽象引入条件。
|
||||||
|
|
||||||
|
## 3. 核心抽象
|
||||||
|
|
||||||
|
### 3.1 PoolStrategy Protocol
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/ports.py 新增(应用层端口,非 core 层——因为返回值 Pools 定义在 app/harness/)
|
||||||
|
|
||||||
|
class PoolStrategy(Protocol):
|
||||||
|
"""池构建策略端口。"""
|
||||||
|
|
||||||
|
def build(
|
||||||
|
self,
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
config: PoolConfig,
|
||||||
|
) -> Pools: ...
|
||||||
|
|
||||||
|
def build_incremental(
|
||||||
|
self,
|
||||||
|
new_task_types: list[str],
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
config: PoolConfig,
|
||||||
|
) -> dict[str, dict[str, list[str]]]:
|
||||||
|
"""增量构建新类别的 train/val 划分。
|
||||||
|
|
||||||
|
返回 {task_type: {"train": [qid, ...], "val": [qid, ...]}}。
|
||||||
|
仅 PerCategoryPoolStrategy 有实质实现;GlobalPoolStrategy
|
||||||
|
不支持增量,调用时抛出 NotImplementedError。
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
> **层级决策**:`PoolStrategy` 放在 `app/ports.py` 而非 `core/protocols.py`,因为返回类型 `Pools` 定义在 `app/harness/pools.py`。若放入 core 会导致 core → app 的逆向依赖,违反 Clean Architecture 四层分层。
|
||||||
|
>
|
||||||
|
> **注**:`PoolConfig` 的两组字段(Global 系列 / PerCategory 系列)由两个策略各取所需,未使用的字段被忽略。这避免了为每个策略定义专属 config 子类的过度抽象。
|
||||||
|
|
||||||
|
### 3.2 PoolConfig
|
||||||
|
|
||||||
|
```python
|
||||||
|
# core/types.py 新增
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PoolConfig:
|
||||||
|
task_types: tuple[str, ...] | None # None = 全部类别
|
||||||
|
seed: int # 随机种子
|
||||||
|
baseline_run_id: str # 基线 run 标识(用于读 correctness、写入 pools 指纹)
|
||||||
|
# GlobalStrategy 使用
|
||||||
|
diag_size: int
|
||||||
|
diag_correct_ratio: float
|
||||||
|
val_size: int
|
||||||
|
val_correct_ratio: float
|
||||||
|
test_size: int
|
||||||
|
eval_min_per_class: int
|
||||||
|
# PerCategoryStrategy 使用
|
||||||
|
train_ratio: float # train/(train+val),默认 2/3
|
||||||
|
test_questions_dir: Path | None # 外部 test 题源路径(与 RunConfig.test_questions 对应)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.3 两个具体策略
|
||||||
|
|
||||||
|
**GlobalPoolStrategy**:封装现有 `build_pools` 逻辑,test → val → diag 全局三分。
|
||||||
|
|
||||||
|
**PerCategoryPoolStrategy**:
|
||||||
|
|
||||||
|
```
|
||||||
|
对每个 task_type (共 N 题,目标 train=20, val=10):
|
||||||
|
correct 题 (Nc 个) → shuffle(seed)
|
||||||
|
train_correct = floor(Nc * 20/N) # 按总量 20/10 比例分配
|
||||||
|
val_correct = Nc - train_correct
|
||||||
|
wrong 题 (Nw 个) → shuffle(seed)
|
||||||
|
train_wrong = 20 - train_correct # 确保 train 总量恰好 20
|
||||||
|
val_wrong = Nw - train_wrong
|
||||||
|
assert train_correct + train_wrong == 20
|
||||||
|
assert val_correct + val_wrong == 10
|
||||||
|
|
||||||
|
diagnosis = 所有选中类别的 train 合并
|
||||||
|
validation = 所有选中类别的 val 合并
|
||||||
|
test = load_benchmark(test_questions_dir) 按 task_types 过滤
|
||||||
|
```
|
||||||
|
|
||||||
|
**边界场景处理**:
|
||||||
|
|
||||||
|
| 场景 | 行为 |
|
||||||
|
|------|------|
|
||||||
|
| 某类别全部 correct(0 wrong) | 退化为非分层 random 20/10,记录 WARNING |
|
||||||
|
| 某类别全部 wrong(0 correct) | 同上,退化为非分层 random 20/10 |
|
||||||
|
| 某类别 correctness 不完整(部分 qid 无 baseline) | fail-fast,列出缺失 qid |
|
||||||
|
| Video-MME 中某 task_type 无题 | test 池该类别为空,记录 WARNING |
|
||||||
|
|
||||||
|
## 4. RunConfig 变更
|
||||||
|
|
||||||
|
| 新增字段 | 类型 | 默认值 | 归属 |
|
||||||
|
|---------|------|--------|------|
|
||||||
|
| `task_types` | `tuple[str, ...] \| None` | `None` | CLI 临时覆盖 |
|
||||||
|
| `pool_split_mode` | `Literal["global", "per_category"]` | `"global"` | 科研 YAML |
|
||||||
|
| `train_ratio` | `float` | `0.667` | 科研 YAML |
|
||||||
|
| `test_questions` | `str` | `"benchmarks/Video-MME"` | 科研 YAML |
|
||||||
|
|
||||||
|
`default.yaml` 新增:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
harness:
|
||||||
|
pool_split_mode: global
|
||||||
|
train_ratio: 0.667
|
||||||
|
test_questions: "benchmarks/Video-MME"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. 增量 Baseline Inference
|
||||||
|
|
||||||
|
### 5.1 流程
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 类别 A、B 完成 → 跑 baseline
|
||||||
|
python main.py harness --mode infer \
|
||||||
|
--run-id baseline_v2 --questions generated-v2-360 \
|
||||||
|
--task-types "Object Recognition" "Scene Understanding"
|
||||||
|
|
||||||
|
# 类别 C 完成 → 追加到同一 run_id
|
||||||
|
python main.py harness --mode infer \
|
||||||
|
--run-id baseline_v2 --questions generated-v2-360 \
|
||||||
|
--task-types "Action Reasoning"
|
||||||
|
|
||||||
|
# promote 为 seed
|
||||||
|
python main.py harness --mode promote \
|
||||||
|
--run-id baseline_v2 --seed baseline_v2_seed
|
||||||
|
|
||||||
|
# 按类训练
|
||||||
|
python main.py harness --mode train --fresh \
|
||||||
|
--seed baseline_v2_seed --run-id train_obj_recog_01 \
|
||||||
|
--task-types "Object Recognition" --pool-split-mode per_category
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 增量写入规则
|
||||||
|
|
||||||
|
| 表 | 唯一键 | 同 run_id 多次 infer 行为 |
|
||||||
|
|---|--------|-------------------------|
|
||||||
|
| `_runs` | `run_id` | upsert:更新 `updated_at`,保留原始 `created_at` |
|
||||||
|
| `predictions` | `(run_id, question_id)` | INSERT;主键冲突报错(防重复推理) |
|
||||||
|
| `traces` | `trace_id` | 正常 INSERT(UUID 天然唯一) |
|
||||||
|
|
||||||
|
**重跑同类别**:若需重新推理某类别,先 DELETE 该类别的 predictions(`WHERE run_id=? AND question_id IN (?)`),再重新 infer。
|
||||||
|
|
||||||
|
**promote 读取规则**:`promote_to_seed` 读取指定 `run_id` 下全部 predictions,不区分追加批次。promote 前应确保所有目标类别均已完成 baseline。
|
||||||
|
|
||||||
|
## 6. 池冻结格式
|
||||||
|
|
||||||
|
### 6.1 per_category 格式
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"split_mode": "per_category",
|
||||||
|
"train_ratio": 0.667,
|
||||||
|
"baseline_run_id": "baseline_v2",
|
||||||
|
"categories": {
|
||||||
|
"Object Recognition": {
|
||||||
|
"train": ["qid_1", "qid_2", "..."],
|
||||||
|
"val": ["qid_21", "..."]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"test_source": "benchmarks/Video-MME",
|
||||||
|
"questions": [...]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 增量更新
|
||||||
|
|
||||||
|
```python
|
||||||
|
def build_or_load_pools(
|
||||||
|
config: RunConfig,
|
||||||
|
strategy: PoolStrategy,
|
||||||
|
db_path: Path,
|
||||||
|
) -> Pools:
|
||||||
|
"""构建或加载三池。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
config: 运行配置(含 task_types, pool_split_mode, baseline_run_id 等)。
|
||||||
|
strategy: 池构建策略实例。
|
||||||
|
db_path: harness.db 路径(用于读取 baseline correctness)。
|
||||||
|
|
||||||
|
流程:
|
||||||
|
1. 若 pools.json 存在 → 加载
|
||||||
|
a. 若 config.task_types 中有类别不在已冻结的 categories → 增量构建并追加
|
||||||
|
b. 若 pools.json 的 seed/train_ratio/baseline_run_id 与当前 config 不一致 → 报错
|
||||||
|
2. 若不存在 → 从 DB 加载 correctness → strategy.build() → 冻结
|
||||||
|
3. 按 config.task_types 从 categories 中组装 Pools 返回
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
|
||||||
|
**冻结一致性校验**:加载已有 pools.json 时,若 `split_mode`、`seed`、`train_ratio`、`baseline_run_id` 与当前 config 不一致,直接报错并提示删除 pools.json 重建。已冻结类别的划分不可变,新类别可追加。
|
||||||
|
|
||||||
|
### 6.3 向后兼容
|
||||||
|
|
||||||
|
旧格式 `pools.json`(无 `split_mode` 字段)自动识别为 global 模式。
|
||||||
|
|
||||||
|
## 7. task_types 作用域贯穿
|
||||||
|
|
||||||
|
策略在池构建阶段完成所有过滤,下游无需感知 task_types:
|
||||||
|
|
||||||
|
| 阶段 | 行为 |
|
||||||
|
|------|------|
|
||||||
|
| 池构建 | strategy 只处理指定类别 |
|
||||||
|
| mini-batch | 消费 scoped 后的 diagnosis 池,无改动 |
|
||||||
|
| per-skill gate | 池中只有指定类别的题,自然 scoped |
|
||||||
|
| 慢进化 | 用 scoped 后的 val 池评估,照常进化 |
|
||||||
|
| held-out eval | test 池已是 Video-MME 同类题 |
|
||||||
|
|
||||||
|
## 8. main.py train 接线
|
||||||
|
|
||||||
|
```python
|
||||||
|
# composition root 伪代码
|
||||||
|
elif config.mode == "train":
|
||||||
|
strategy = (PerCategoryPoolStrategy()
|
||||||
|
if config.pool_split_mode == "per_category"
|
||||||
|
else GlobalPoolStrategy())
|
||||||
|
pools = build_or_load_pools(config, strategy)
|
||||||
|
await runner.train(pools)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. 文件改动地图
|
||||||
|
|
||||||
|
| 文件 | 改动类型 | 内容 |
|
||||||
|
|------|---------|------|
|
||||||
|
| `app/ports.py` | 新增 | `PoolStrategy` Protocol(应用层端口,避免 core → app 逆向依赖) |
|
||||||
|
| `core/types.py` | 新增 | `PoolConfig` dataclass |
|
||||||
|
| `app/harness/pools.py` | 重构 | `GlobalPoolStrategy` + `PerCategoryPoolStrategy` + 增量逻辑 |
|
||||||
|
| `app/harness/config.py` | 修改 | RunConfig 新增 4 字段 + 校验;`pool_split_mode` 用 Literal 类型 |
|
||||||
|
| `config/default.yaml` | 修改 | 3 个新配置项 |
|
||||||
|
| `main.py` | 修改 | `task_types` 纳入 `cli_overrides` + RunConfig;train 接线;strategy 组装 |
|
||||||
|
| `app/harness/log.py` | 微调 | `_runs` 表 `INSERT OR IGNORE` 改为 `ON CONFLICT DO UPDATE`(增量 infer 更新时间戳) |
|
||||||
|
| `tests/unit/test_pools.py` | 新增 | 两种策略单元测试 |
|
||||||
|
| `tests/integration/test_pool_strategy.py` | 新增 | 端到端集成测试 |
|
||||||
|
|
||||||
|
### 不改动
|
||||||
|
|
||||||
|
- `app/harness/batching.py` — 消费 scoped 池,无需感知策略
|
||||||
|
- `core/evolution/` — 进化引擎与池来源解耦
|
||||||
|
- `app/harness/inference.py` — 推理逻辑不变
|
||||||
|
- `app/question_gen/` — 题目生成与池构建解耦
|
||||||
@@ -0,0 +1,265 @@
|
|||||||
|
---
|
||||||
|
id: task-type-strategy
|
||||||
|
title: 出题管线 TaskTypeStrategy 拆分设计(Clean Architecture)
|
||||||
|
type: design
|
||||||
|
created: 2026-07-14
|
||||||
|
status: approved
|
||||||
|
---
|
||||||
|
|
||||||
|
# 出题管线 TaskTypeStrategy 拆分设计
|
||||||
|
|
||||||
|
## 1. 目标
|
||||||
|
|
||||||
|
借鉴 Clean Architecture 思想,将出题管线从 5 个粗粒度 Family 替换为 12 个题型级别的 TaskTypeStrategy,实现题型独立的出题策略。第一个特化实现为 ActionRecognitionStrategy(含 6 个失败子模式靶向出题)。
|
||||||
|
|
||||||
|
### 设计驱动
|
||||||
|
|
||||||
|
| 问题 | 数据来源 |
|
||||||
|
|------|---------|
|
||||||
|
| v2-360 全部 30 道 AR 题 100% 单帧可答 | `research-wiki/findings/2026-07-14-question-quality-gap-analysis.md` |
|
||||||
|
| 12/12 题型存在 CRITICAL 或 HIGH 差距 | 同上 |
|
||||||
|
| 同一 task_type 跨 family 随机选择导致行为不确定 | `app/question_gen/families.py` — AR 同时在 RETRIEVAL 和 VISUAL |
|
||||||
|
| 5 个 prompt 模板零题型分支 | `store/prompts/question_gen/*.md` |
|
||||||
|
|
||||||
|
## 2. 架构
|
||||||
|
|
||||||
|
### 2.1 替换关系
|
||||||
|
|
||||||
|
```
|
||||||
|
之前:pipeline → get_family_for_slot() → QuestionFamilySpec(5 个,随机选择)
|
||||||
|
之后:pipeline → get_strategy(task_type) → TaskTypeStrategy(12 个,确定性查找)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 类层次
|
||||||
|
|
||||||
|
```
|
||||||
|
TaskTypeStrategy (Protocol)
|
||||||
|
│
|
||||||
|
├── BaseTaskTypeStrategy (类)
|
||||||
|
│ └── 封装现有 family 行为,确定性绑定一个 QuestionFamilySpec
|
||||||
|
│ └── select_sub_pattern → None
|
||||||
|
│ └── extra_gates → []
|
||||||
|
│ └── 11 个题型用此类
|
||||||
|
│
|
||||||
|
└── ActionRecognitionStrategy (类)
|
||||||
|
└── 自包含采样/prompt/SubPattern
|
||||||
|
└── 仍提供 strategy_name / skill_target / leak_probe_template(gate/store 合约)
|
||||||
|
└── 6 个 SubPattern + maintenance_pool
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.3 接口定义
|
||||||
|
|
||||||
|
```python
|
||||||
|
class TaskTypeStrategy(Protocol):
|
||||||
|
"""题型出题策略 — pipeline 的唯一接口。"""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def task_type(self) -> str: ...
|
||||||
|
|
||||||
|
# ── 采样 ──
|
||||||
|
@property
|
||||||
|
def sampling_level(self) -> int: ...
|
||||||
|
@property
|
||||||
|
def sampling_constraint(self) -> SamplingConstraint: ...
|
||||||
|
|
||||||
|
# ── Prompt ──
|
||||||
|
@property
|
||||||
|
def prompt_template(self) -> str: ...
|
||||||
|
def select_sub_pattern(self, rng: Random) -> SubPattern | None: ...
|
||||||
|
def build_prompt_context(self, material: MaterialContext, sub_pattern: SubPattern | None) -> dict: ...
|
||||||
|
|
||||||
|
# ── Gate / Store 合约 ──
|
||||||
|
@property
|
||||||
|
def strategy_name(self) -> str:
|
||||||
|
"""策略标识名,写入 store family 字段(如 "RETRIEVAL"、"ACTION_RECOGNITION")。"""
|
||||||
|
@property
|
||||||
|
def skill_target(self) -> str:
|
||||||
|
"""目标失败机制编号,写入 store(如 "M1"、"AR")。"""
|
||||||
|
@property
|
||||||
|
def leak_probe_template(self) -> str:
|
||||||
|
"""泄漏检测 prompt 模板文件名,供 leak_test gate 使用。"""
|
||||||
|
|
||||||
|
def extra_gates(self, candidate: CandidateQuestion) -> list[GateResult]:
|
||||||
|
"""题型专属的额外验证(base 返回空列表)。"""
|
||||||
|
```
|
||||||
|
|
||||||
|
`strategy_name` / `skill_target` / `leak_probe_template` 替代了 pipeline 中对 `slot.family.name` / `slot.family.skill_target` / `slot.family.leak_profile.probe_template` 的访问,确保 gate 和 store 合约不破坏。
|
||||||
|
|
||||||
|
## 3. SubPattern 数据结构
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SubPattern:
|
||||||
|
"""出题子模式 — 靶向特定失败机制。"""
|
||||||
|
name: str
|
||||||
|
weight: float
|
||||||
|
sampling_level_override: int | None
|
||||||
|
constraint_override: SamplingConstraint | None
|
||||||
|
instruction: str # 核心指令(1-3 句话)
|
||||||
|
positive_examples: list[dict] # VME 原题 few-shot
|
||||||
|
negative_examples: list[dict] # 反面示例
|
||||||
|
distractor_rules: str # 干扰项构造规则
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. ActionRecognitionStrategy 的 6 个 SubPattern
|
||||||
|
|
||||||
|
来源:22 道错题双分类器仲裁分析。
|
||||||
|
|
||||||
|
| SubPattern | 权重 | 采样 | 错题# | 失败机制 |
|
||||||
|
|-----------|------|------|-------|---------|
|
||||||
|
| premature_evidence_anchoring | 0.20 | L1 | #3,4,12,17 | 找到一个证据就停搜,未验证全部选项 |
|
||||||
|
| temporal_reasoning_failure | 0.20 | L1 | #8,9,15,16 | 事件排序错误 / 无法定位第 N 次事件 |
|
||||||
|
| semantic_rigidity | 0.15 | L2 | #1,2,22 | 要求字面匹配,拒绝同义词释义 |
|
||||||
|
| fine_grained_visual_action | 0.15 | L2 | #6,13,14,18 | 识别"做了什么"但分不清"怎么做的" |
|
||||||
|
| cross_segment_entity_tracking | 0.15 | L1 | #7,19,20 | 单段正确但无法跨段合并 |
|
||||||
|
| evidence_gap_confabulation | 0.15 | L2 | #5,10,11,21 | 缺证据时编造因果链 |
|
||||||
|
|
||||||
|
### ActionRecognitionStrategy 的 gate/store 合约值
|
||||||
|
|
||||||
|
| 字段 | 值 | 说明 |
|
||||||
|
|------|-----|------|
|
||||||
|
| strategy_name | `"ACTION_RECOGNITION"` | store 的 family 字段 |
|
||||||
|
| skill_target | `"M1_AR"` | 继承自 RETRIEVAL 的 M1 + AR 后缀区分 |
|
||||||
|
| leak_probe_template | `"gate_leak_retrieval.md"` | 复用 RETRIEVAL 的泄漏检测模板 |
|
||||||
|
| sampling_level(默认) | 2 | L2 事件级(从 L3 提升,有意变更) |
|
||||||
|
| sampling_constraint(默认) | `min_subtitles=3, min_l3_nodes=5, require_frames=True, cross_l2_span=True` | 加强约束(有意变更) |
|
||||||
|
|
||||||
|
SubPattern 可通过 `sampling_level_override` / `constraint_override` 进一步覆盖默认值。
|
||||||
|
|
||||||
|
### 4.1 能力保持机制
|
||||||
|
|
||||||
|
capability_maintenance 不是生成子模式,而是**采样来源**:从 Video-MME AR 的 41 道正确题中采样,直接进训练集。通过 `batch_correct_ratio` 控制混合比例。
|
||||||
|
|
||||||
|
```
|
||||||
|
训练 batch 组成:
|
||||||
|
├── 错误题来源: 6 个 SubPattern 生成的新题
|
||||||
|
└── 正确题来源: maintenance_pool (41 道已验证正确的 VME 原题)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. pipeline 集成
|
||||||
|
|
||||||
|
### 5.1 调用链变更
|
||||||
|
|
||||||
|
```
|
||||||
|
之前: slot.family → sample_material_v2(family_spec=...) → generate_one_v2(family_spec=...)
|
||||||
|
→ run_gates(family_spec=...) → store.record_item(family=slot.family.name)
|
||||||
|
|
||||||
|
之后: strategy = get_strategy(slot.task_type)
|
||||||
|
sub_pattern = strategy.select_sub_pattern(rng)
|
||||||
|
level = sub_pattern.sampling_level_override or strategy.sampling_level
|
||||||
|
constraint = sub_pattern.constraint_override or strategy.sampling_constraint
|
||||||
|
material = sample_material_v2(level=level, constraint=constraint, ...)
|
||||||
|
prompt_ctx = strategy.build_prompt_context(material, sub_pattern)
|
||||||
|
candidate = generate_one_v2(prompt_ctx=prompt_ctx, ...)
|
||||||
|
report = run_gates(candidate, leak_template=strategy.leak_probe_template, ...)
|
||||||
|
extra = strategy.extra_gates(candidate)
|
||||||
|
store.record_item(..., family=strategy.strategy_name, skill_target=strategy.skill_target,
|
||||||
|
sub_pattern=sub_pattern.name if sub_pattern else None)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 接口影响面
|
||||||
|
|
||||||
|
| 文件 | 变更 | 影响范围 |
|
||||||
|
|------|------|---------|
|
||||||
|
| `sampler_v2.py` | `sample_material_v2` 签名:`family_spec` → `level: int` + `constraint: SamplingConstraint` | 函数签名 + 内部 `constraint = family_spec.sampling` 行 |
|
||||||
|
| `pipeline_v2.py` | `_process_one_slot`:`slot.family` → `strategy`;`SlotAssignment.family` 字段移除 | `_assign_slots` / `_process_one_slot` / `_process_wrapper` |
|
||||||
|
| `generator_v2.py` | `generate_one_v2` 签名:新增 `prompt_ctx: dict` 参数,替代内部读 `family_spec.prompt_template` | `_build_v2_prompt` / `generate_one_v2` |
|
||||||
|
| `gates.py` | `run_gates` 签名:`family_spec` → `leak_template: str` | `_run_leak_test` 参数变更 |
|
||||||
|
| `run_store.py` | `record_item` 新增 `sub_pattern: str | None` 参数 | DDL 加 `sub_pattern TEXT` 列 |
|
||||||
|
|
||||||
|
### 5.3 不变的部分
|
||||||
|
|
||||||
|
后处理(postprocess)、四门 gate 内部逻辑、去重(embedding)、重量抽检(heavy_check)、on_accept 回调、progress 断点续跑 — 全部不动。
|
||||||
|
|
||||||
|
## 6. Strategy 注册
|
||||||
|
|
||||||
|
```python
|
||||||
|
_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {}
|
||||||
|
|
||||||
|
def get_strategy(task_type: str) -> TaskTypeStrategy:
|
||||||
|
if task_type in _STRATEGY_REGISTRY:
|
||||||
|
return _STRATEGY_REGISTRY[task_type]
|
||||||
|
return _build_default_strategy(task_type)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.1 BaseTaskTypeStrategy family 消歧绑定表
|
||||||
|
|
||||||
|
消除多 family 随机性。每个 task_type 确定性绑定一个 family。
|
||||||
|
|
||||||
|
| task_type | 旧合法 families | 新绑定 | 被移除行为 | 理由 |
|
||||||
|
|-----------|---------------|--------|-----------|------|
|
||||||
|
| Action Recognition | RETRIEVAL, VISUAL | **特化策略** | 不走 family | 完全自包含 |
|
||||||
|
| Object Recognition | RETRIEVAL | RETRIEVAL | 无 | 唯一 |
|
||||||
|
| Object Reasoning | RETRIEVAL, REASONING | **REASONING** | RETRIEVAL 的 L3 单帧 + 禁推理 prompt | 需要推理;RETRIEVAL 产出识别题是已知 bug |
|
||||||
|
| Action Reasoning | REASONING | REASONING | 无 | 唯一 |
|
||||||
|
| Attribute Perception | RETRIEVAL, VISUAL | **VISUAL** | RETRIEVAL 的 require_frames=False | 属性感知需要帧 |
|
||||||
|
| OCR Problems | RETRIEVAL, VISUAL | **VISUAL** | RETRIEVAL 的 require_frames=False | OCR 需要帧 |
|
||||||
|
| Counting Problem | ENUMERATION, VISUAL | **ENUMERATION** | VISUAL 的 L3 单帧采样 | 计数需要跨帧 |
|
||||||
|
| Information Synopsis | REASONING, ENUMERATION | **REASONING** | ENUMERATION 的 cross_l2_span=False | 综述需要跨段 |
|
||||||
|
| Temporal Reasoning | ENUMERATION | ENUMERATION | 无 | 唯一 |
|
||||||
|
| Temporal Perception | ENUMERATION | ENUMERATION | 无 | 唯一 |
|
||||||
|
| Spatial Reasoning | SPATIAL | SPATIAL | 无 | 唯一 |
|
||||||
|
| Spatial Perception | SPATIAL | SPATIAL | 无 | 唯一 |
|
||||||
|
|
||||||
|
### 6.2 多归属题型约束变化明细
|
||||||
|
|
||||||
|
确定性绑定会改变多归属题型的采样约束,这是**有意的行为变更**(旧行为是随机混合,本身就是不可控的)。
|
||||||
|
|
||||||
|
| task_type | 旧约束(随机选择) | 新约束(确定绑定) | 变化 |
|
||||||
|
|-----------|-----------------|-----------------|------|
|
||||||
|
| Object Reasoning | RETRIEVAL(L3,no-frame,no-cross) 或 REASONING(L2,no-frame,cross) | REASONING 固定 | 不再退化为 L3 识别题 |
|
||||||
|
| Attribute Perception | RETRIEVAL(no-frame) 或 VISUAL(frame) | VISUAL 固定 | 始终要求帧 |
|
||||||
|
| OCR Problems | RETRIEVAL(no-frame) 或 VISUAL(frame) | VISUAL 固定 | 始终要求帧 |
|
||||||
|
| Counting Problem | ENUMERATION(no-frame) 或 VISUAL(frame) | ENUMERATION 固定 | 不再混入 VISUAL 的 L3 采样 |
|
||||||
|
| Information Synopsis | REASONING(cross) 或 ENUMERATION(no-cross) | REASONING 固定 | 始终跨段 |
|
||||||
|
|
||||||
|
## 7. 文件结构
|
||||||
|
|
||||||
|
| 操作 | 文件 | 职责 |
|
||||||
|
|------|------|------|
|
||||||
|
| 新建 | `app/question_gen/strategy.py` | Protocol + SubPattern + BaseTaskTypeStrategy + 注册表 |
|
||||||
|
| 新建 | `app/question_gen/strategy_action_recognition.py` | ActionRecognitionStrategy 实现 + 6 个 SubPattern 定义 |
|
||||||
|
| 修改 | `app/question_gen/pipeline_v2.py` | `_process_one_slot` / `_assign_slots` 改为通过 strategy |
|
||||||
|
| 修改 | `app/question_gen/sampler_v2.py` | 签名 `family_spec` → `level` + `constraint` |
|
||||||
|
| 修改 | `app/question_gen/generator_v2.py` | 签名 `family_spec` → `prompt_ctx` |
|
||||||
|
| 修改 | `app/question_gen/gates.py` | `run_gates` 签名 `family_spec` → `leak_template` |
|
||||||
|
| 修改 | `app/question_gen/run_store.py` | `record_item` 新增 `sub_pattern` 参数 + DDL 加列 |
|
||||||
|
| 不动 | `app/question_gen/families.py` | 保留,BaseTaskTypeStrategy 内部使用 |
|
||||||
|
| 不动 | `app/question_gen/postprocess.py` | 后处理不变 |
|
||||||
|
|
||||||
|
## 8. 非功能性需求
|
||||||
|
|
||||||
|
| 维度 | 设计 |
|
||||||
|
|------|------|
|
||||||
|
| 持久化 | 不变 — on_accept 逐题回调。SubPattern 名写入 store 的 sub_pattern 列以便追溯 |
|
||||||
|
| 幂等性 | 不变 — 同 seed → 同 slot 分配。strategy 查找确定性,sub_pattern 选择由 rng 控制 |
|
||||||
|
| 断点续跑 | 不变 — progress 机制在 pipeline 层,strategy 无状态 |
|
||||||
|
| 原子性 | 不变 — 逐题落库 |
|
||||||
|
| store 迁移 | `question_gen_items` 表新增 `sub_pattern TEXT` 列,默认 NULL(BaseTaskTypeStrategy 写 NULL) |
|
||||||
|
|
||||||
|
## 9. 行为保真检查清单
|
||||||
|
|
||||||
|
| # | 行为 | 状态 | 说明 |
|
||||||
|
|---|------|------|------|
|
||||||
|
| 1 | 12 种 task_type 列表 | 保留 | pipeline 不变 |
|
||||||
|
| 2 | slot round-robin 视频 | 保留 | pipeline 不变 |
|
||||||
|
| 3 | family 加权随机选择 | **有意变更** | 消除不确定性 → 确定性绑定。详见 §6.2 |
|
||||||
|
| 4 | _TASK_TYPE_TO_LEVEL 映射 | **有意变更** | BaseTaskTypeStrategy 从绑定 family 读取原值(唯一归属题型数值不变);AR 特化策略从 L3→L2(有意提升);多归属题型因消歧而变化(详见 §6.2) |
|
||||||
|
| 5 | SamplingConstraint 约束 | **有意变更** | 同 #4 逻辑:唯一归属不变,多归属因消歧变化,AR 有意加强 |
|
||||||
|
| 6 | 重出循环 + 换视频 | 保留 | |
|
||||||
|
| 7 | 后处理洗牌 + verbatim | 保留 | |
|
||||||
|
| 8 | 四门 gate | 保留 | leak_test 通过 strategy.leak_probe_template 获取模板 |
|
||||||
|
| 9 | embedding 去重 | 保留 | |
|
||||||
|
| 10 | 重量抽检 | 保留 | |
|
||||||
|
| 11 | on_accept 回调 | 保留 | |
|
||||||
|
| 12 | progress 断点续跑 | 保留 | |
|
||||||
|
| 13 | reject_reason 传入重出 | 保留 | |
|
||||||
|
| 14 | QuestionGenStore 记录 | 保留 | family → strategy_name;新增 sub_pattern 列 |
|
||||||
|
|
||||||
|
## 10. 渐进替换路径
|
||||||
|
|
||||||
|
```
|
||||||
|
Phase 1 (当前): ActionRecognitionStrategy 特化 + 11 个 BaseTaskTypeStrategy
|
||||||
|
Phase 2 (按需): 逐个替换表现差的题型为特化策略
|
||||||
|
Phase N (最终): 12 个特化策略,families.py 可移除
|
||||||
|
```
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:per-category-pool-strategy
|
||||||
|
title: "Per-Category Pool Strategy 设计"
|
||||||
|
date: 2026-07-13
|
||||||
|
---
|
||||||
|
|
||||||
|
# Per-Category Pool Strategy 设计
|
||||||
|
|
||||||
@@ -0,0 +1,206 @@
|
|||||||
|
# v2-360 vs Video-MME 出题质量差距分析报告
|
||||||
|
|
||||||
|
> 基线准确率 73.2% | v2 覆盖 6/12 类型 | VME 共 900 题
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 全局概览
|
||||||
|
|
||||||
|
| # | 题型 | VME 题数 | v2 题数 | VME 准确率 | VME 主导模式 | v2 主导模式 | 差距等级 |
|
||||||
|
|---|------|---------|---------|-----------|-------------|------------|---------|
|
||||||
|
| 1 | Information Synopsis | 163 | 0 | -- | 全局主题识别 (61%) | -- | **CRITICAL** |
|
||||||
|
| 2 | Counting Problem | 48 | 0 | 50% | 全视频离散段计数 (29%) | -- | **CRITICAL** |
|
||||||
|
| 3 | Attribute Perception | 27 | 0 | -- | 否定/假命题验证 (26%) | -- | **CRITICAL** |
|
||||||
|
| 4 | OCR Problems | 14 | 0 | -- | 计分板/数值精确读取 (29%) | -- | **CRITICAL** |
|
||||||
|
| 5 | Temporal Perception | 6 | 0 | 33% | 事件时间定位 (50%) | -- | **CRITICAL** |
|
||||||
|
| 6 | Spatial Perception | 3 | 0 | 33% | 时空定位/朝向推理 (各33%) | -- | **CRITICAL** |
|
||||||
|
| 7 | Action Recognition | 63 | 30 | -- | 时间锚点行为观察 (21%) | OCR 文字读取 (67%) | **CRITICAL** |
|
||||||
|
| 8 | Action Reasoning | 180 | 30 | -- | 因果 why 推理 (36%) | OCR/文字读取 (53%) | **CRITICAL** |
|
||||||
|
| 9 | Object Reasoning | 240 | 30 | -- | 因果解释推理 (22%) | OCR/文字读取 (57%) | **CRITICAL** |
|
||||||
|
| 10 | Temporal Reasoning | 91 | 30 | -- | 严格序列排序 (52%) | OCR/文字读取 (37%) | **CRITICAL** |
|
||||||
|
| 11 | Object Recognition | 54 | 30 | -- | 否定/缺失检测 (30%) | OCR/文字读取 (33%) | **HIGH** |
|
||||||
|
| 12 | Spatial Reasoning | 11 | 30 | -- | 地理位置推断 (18%) | OCR/文字读取 (30%) | **HIGH** |
|
||||||
|
|
||||||
|
**结论**: 12 个题型全部存在 CRITICAL 或 HIGH 级别差距。6 个题型完全未覆盖;已覆盖的 6 个题型中,v2 的推理深度与 VME 严重错位。
|
||||||
|
|
||||||
|
### 推理深度对比(已覆盖的 6 个题型平均)
|
||||||
|
|
||||||
|
| 深度层级 | VME 平均占比 | v2 平均占比 | 差距 |
|
||||||
|
|---------|------------|-----------|------|
|
||||||
|
| single_frame | 8.0% | **66.1%** | v2 严重偏向单帧 |
|
||||||
|
| multi_frame | 25.4% | 15.6% | 略低 |
|
||||||
|
| cross_segment | 25.7% | 15.6% | 不足 |
|
||||||
|
| full_video | 41.0% | **2.8%** | v2 几乎为零 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 逐题型分析
|
||||||
|
|
||||||
|
### 2.1 Information Synopsis (VME=163, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 79.8% full_video。主导模式为"视频主题识别"(61%),干扰项为同领域相邻主题(如 Gucci 纪录片选项含 Chanel/LV/Prada)。否定模式占 13.5%("哪个事件未被报道")。
|
||||||
|
- **差距诊断**: v2 完全缺失。管线无任何摘要/主题/综述类 prompt 模板。
|
||||||
|
- **管线根因**: REASONING 家族虽覆盖 Information Synopsis,但 L2 采样仅提供局部事件,无法支撑全局主题判断。prompt 中无综述类题目的具体指导。
|
||||||
|
|
||||||
|
### 2.2 Counting Problem (VME=48, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 72.9% full_video。要求在整个视频中持续追踪计数("Salah 出现了几个进球")。含条件过滤计数("独唱表演有几个")和陷阱题(答案为 0)。
|
||||||
|
- **差距诊断**: v2 完全缺失。ENUMERATION 家族名义覆盖但 `require_frames=False` + `cross_l2_span=False` 产出文本计数题。
|
||||||
|
- **管线根因**: 采样约束限制在单 L2 段内,无法产出需全视频追踪的计数题。无最小计数阈值或视觉计数要求。
|
||||||
|
|
||||||
|
### 2.3 Attribute Perception (VME=27, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 26% 为否定验证("以下哪项不正确"),需逐项核实 3-4 个事实。37% cross_segment。含情感/态度感知(从非语言线索推断情绪)。
|
||||||
|
- **差距诊断**: v2 完全缺失。RETRIEVAL 和 VISUAL 家族名义覆盖但无属性感知专用指导。
|
||||||
|
- **管线根因**: RETRIEVAL 中 `require_frames=False`,属性感知退化为文本检索。VISUAL 中 L3 单帧采样使属性观察过于简单。
|
||||||
|
|
||||||
|
### 2.4 OCR Problems (VME=14, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 50% single_frame 但要求精确数值/文字提取("半场比分 32-23"),干扰项数值接近(57.5/55/60 kg)。含时间锚定 OCR("罚球后比分")。
|
||||||
|
- **差距诊断**: v2 完全缺失。讽刺的是 v2 已有的 6 个题型中大量问题实质上是 OCR 题(占 v2 总量 ~40%),但未被归类为 OCR。
|
||||||
|
- **管线根因**: RETRIEVAL 和 VISUAL 覆盖 OCR 但无文字难度要求(大小、角度、遮挡)。L3 单帧 + 大字号文本 = 简单读取。
|
||||||
|
|
||||||
|
### 2.5 Temporal Perception (VME=6, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 83.3% cross_segment。事件时间定位("厨房事故发生在周几午餐?"),相对时长估计("哪款车占视频比例最大?")。
|
||||||
|
- **差距诊断**: v2 完全缺失。管线无时间定位或时长估计模板。
|
||||||
|
- **管线根因**: ENUMERATION 家族名义覆盖但 prompt 无时长/间隔/定位指导。`cross_l2_span=False` 无法跨段生成。
|
||||||
|
|
||||||
|
### 2.6 Spatial Perception (VME=3, v2=0) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 66.7% cross_segment。时空联合定位("左下角绿色模型何时出现")、物体目的地推理、相对朝向判断。
|
||||||
|
- **差距诊断**: v2 完全缺失。管线无空间-时间交叉模板。
|
||||||
|
- **管线根因**: SPATIAL 家族仅覆盖 Spatial Reasoning,Spatial Perception 未接入任何家族。
|
||||||
|
|
||||||
|
### 2.7 Action Recognition (VME=63, v2=30) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 95.2% 多帧以上。主导模式为时间锚点行为观察 (21%)、否定/缺失 (17%)、事件机制 (17%)。
|
||||||
|
- **v2 特征**: **100% single_frame**。67% 纯 OCR(球衣号码、排行榜文字)。无一题涉及动作序列或时序理解。
|
||||||
|
- **差距诊断**: 题型根本错配 -- v2 测试静态视觉感知而非动作识别。截图模型即可满分。
|
||||||
|
- **管线根因**: RETRIEVAL 家族 L3 采样 + "factual recall" prompt 产出单帧标签题。VISUAL 家族 L3 无法展示动作序列。
|
||||||
|
|
||||||
|
### 2.8 Action Reasoning (VME=180, v2=30) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 0% single_frame。因果 why 推理占 36%,角色心理推断 10%,否定推理 8%,策略评估 7%。
|
||||||
|
- **v2 特征**: 53% single_frame(OCR 污染)。仅 3/30 题涉及因果推理。零角色动机/策略/隐喻推理。
|
||||||
|
- **差距诊断**: 过半题目为错分类的 OCR/检索题。真正的推理题占比不足 10%。
|
||||||
|
- **管线根因**: REASONING prompt 虽要求"multi-hop"但无结构性执行约束。`require_frames=False` 使推理退化为文本提取。
|
||||||
|
|
||||||
|
### 2.9 Object Reasoning (VME=240, v2=30) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 90% cross_segment 或 full_video。因果解释 (22%)、关系推断 (16%)、否定验证 (13%)、评价判断 (12%)。
|
||||||
|
- **v2 特征**: 56.7% single_frame。主导为 OCR (33%) 和物体属性识别 (23%)。仅 27% 涉及推理。
|
||||||
|
- **差距诊断**: v2 的"物体推理"实为物体识别的错误标签。无因果链、无关系推断、无评价性判断。
|
||||||
|
- **管线根因**: Object Reasoning 同时出现在 RETRIEVAL (L3, 禁止推理) 和 REASONING (L2) 两个家族,RETRIEVAL 产出的题目本质是识别题。
|
||||||
|
|
||||||
|
### 2.10 Temporal Reasoning (VME=91, v2=30) -- CRITICAL
|
||||||
|
|
||||||
|
- **VME 特征**: 72.5% full_video。严格序列排序占 52%("以下 5 项按什么顺序出现?"),日程重建 11%。
|
||||||
|
- **v2 特征**: 36.7% single_frame。零序列排序题。最好的题目是弱版"X 之后发生了什么"(只需链接两个时刻)。
|
||||||
|
- **差距诊断**: VME 的核心模式(排列 3-5 项的顺序)在 v2 中完全缺失。v2 的"时序"题多为 OCR 或简单因果。
|
||||||
|
- **管线根因**: ENUMERATION 家族 `cross_l2_span=False` 限制在单段内,无法收集多段时序材料。prompt 的"顺序"指导过于笼统。
|
||||||
|
|
||||||
|
### 2.11 Object Recognition (VME=54, v2=30) -- HIGH
|
||||||
|
|
||||||
|
- **VME 特征**: 75.9% cross_segment 或 full_video。否定/缺失检测 (30%)、全视频实体识别 (26%)。
|
||||||
|
- **v2 特征**: 63.3% single_frame。OCR/标签读取 (33%)、物体属性 (27%)。
|
||||||
|
- **差距诊断**: VME 的"识别"意为"跨视频追踪并清点所有相关实体",v2 理解为"在帧中认出物体"。
|
||||||
|
- **管线根因**: RETRIEVAL L3 + "直接可观察信息" prompt 产出最简单的识别题。
|
||||||
|
|
||||||
|
### 2.12 Spatial Reasoning (VME=11, v2=30) -- HIGH
|
||||||
|
|
||||||
|
- **VME 特征**: 72.7% multi_frame 或 full_video。从视觉线索推断地理位置、理解空间排列目的、估算距离。
|
||||||
|
- **v2 特征**: 86.7% single_frame。OCR (30%)、相对位置 (23%)、颜色识别 (17%)。
|
||||||
|
- **差距诊断**: v2 测试"观察"(X 在 Y 左边),VME 测试"推断"(从建筑风格推断国家)。
|
||||||
|
- **管线根因**: SPATIAL prompt 无推断层级要求。L1 采样与空间精度需求错位(应 L3 帧级 + 多帧序列)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 管线系统性问题
|
||||||
|
|
||||||
|
### 3.1 Prompt 层问题
|
||||||
|
|
||||||
|
| 家族 | 核心缺陷 | 影响题型 |
|
||||||
|
|------|---------|---------|
|
||||||
|
| RETRIEVAL | "factual recall" + "禁止 multi-hop" = 强制最简难度 | AR/OR/ObjRec/AP/OCR |
|
||||||
|
| REASONING | 无结构性 multi-hop 执行约束(无最小跳数、无证据链格式) | ActReas/ObjReas/InfoSyn |
|
||||||
|
| ENUMERATION | 计数/排序/时序/综述用同一泛化 prompt,无题型分支 | Count/TempReas/TempPerc/InfoSyn |
|
||||||
|
| VISUAL | 无难度下限("什么颜色"即合法),无时序视觉变化要求 | AP/Count/OCR/AR |
|
||||||
|
| SPATIAL | Perception 与 Reasoning 用同一 prompt,无推断层级要求 | SpatPerc/SpatReas |
|
||||||
|
| **全部** | **零题型分支**: 5 个 prompt 均为题型无关,task_type 仅作元数据传入 | 全部 12 型 |
|
||||||
|
|
||||||
|
### 3.2 采样层问题
|
||||||
|
|
||||||
|
| 问题 | 具体表现 | 影响 |
|
||||||
|
|------|---------|------|
|
||||||
|
| L3 单帧采样用于 4 个题型 | AR/ObjRec/AP/OCR 在 RETRIEVAL 中均 L3 | 100% single_frame 退化 |
|
||||||
|
| `cross_l2_span=False` 用于时序题 | ENUMERATION 家族限制在单段 | 无法生成跨段排序/计数 |
|
||||||
|
| `require_frames=False` 用于视觉题型 | RETRIEVAL 中 AP/OCR/AR 不要求帧 | 视觉题退化为文本题 |
|
||||||
|
| 约束按家族统一,非按题型 | 同一家族内不同题型需求冲突 | 系统性错配 |
|
||||||
|
|
||||||
|
### 3.3 Gate 层问题
|
||||||
|
|
||||||
|
| Gate | 盲点 | 后果 |
|
||||||
|
|------|------|------|
|
||||||
|
| blind_answer | 只检测上下文泄露,不检测"有上下文也简单" | 大量 easy 题通过 |
|
||||||
|
| key_verify | text OR frames 通过即可 | 视觉题型可纯文本回答 |
|
||||||
|
| leak_test | 按家族探测,非按题型 | 题型特有捷径未拦截 |
|
||||||
|
| heavy_check | 后置非阻塞,easy 题已接受 | 难度反馈无实际作用 |
|
||||||
|
| **缺失** | **无难度门控 gate** | 简单题无任何拦截机制 |
|
||||||
|
| **缺失** | **无题集难度分布检查** | 无法控制 easy/med/hard 比例 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 改进路线图
|
||||||
|
|
||||||
|
### P0: 影响最大、立即可做
|
||||||
|
|
||||||
|
| # | 改进项 | 预期收益 | 实施方式 |
|
||||||
|
|---|-------|---------|---------|
|
||||||
|
| P0-1 | **Prompt 内加入题型分支** | 消除"AR 出 OCR 题"等根本性错配 | 每个 prompt 增加 `{% if task_type == "Action Recognition" %}` 块,含题型专属规则和负面示例 |
|
||||||
|
| P0-2 | **修复采样约束为题型级** | 解除 L3/no-frame/no-cross 限制 | `SamplingConstraint` 从 per-family 改为 per-(family, task_type),AR/AP/OCR 强制 `require_frames=True`,TempReas 强制 `cross_l2_span=True` |
|
||||||
|
| P0-3 | **提升 AR/ActReas/ObjReas 采样层级到 L2** | 从单帧提升到多帧 | RETRIEVAL 中 AR/AP/OCR 改为 L2;VISUAL 中 AR 改为 L2 |
|
||||||
|
| P0-4 | **添加否定/缺失题模式** | 覆盖 VME 中占比 15-30% 的核心模式 | 新增 prompt 指令:"生成'以下哪项未在视频中出现'类型题目";需 full_video 采样支持 |
|
||||||
|
| P0-5 | **新增难度门控 gate** | 直接过滤简单题 | 添加第 5 个 gate: 给 LLM 完整上下文,要求在 5 秒内作答;能秒答 = fail |
|
||||||
|
|
||||||
|
### P1: 重要但需设计
|
||||||
|
|
||||||
|
| # | 改进项 | 预期收益 | 实施要点 |
|
||||||
|
|---|-------|---------|---------|
|
||||||
|
| P1-1 | **新增 6 个缺失题型的 prompt 模板** | 覆盖率从 50% 提升到 100% | InfoSynopsis: L1 全场景 + 主题合成;Counting: full_video + 视觉计数 + 条件过滤;AP: 否定验证 + 情感感知;OCR: 精确数值 + 时间锚定;TempPerc: 时间定位 + 时长估计;SpatPerc: 时空联合 + 朝向推理 |
|
||||||
|
| P1-2 | **创建独立 TEMPORAL 和 SYNOPSIS 家族** | 解除 ENUMERATION 对时序题的约束错配 | TEMPORAL: `cross_l2_span=True`, L1 采样, 序列排序专用 prompt;SYNOPSIS: L1 全场景, 综述/主题/否定专用 prompt |
|
||||||
|
| P1-3 | **key_verify gate 按题型区分证据要求** | 视觉题型必须帧内有证据 | AP/OCR/SpatPerc/AR: 改为 "evidence in frames ONLY";reasoning 类: 维持 "text OR frames" |
|
||||||
|
| P1-4 | **Prompt 加入难度校准示例** | LLM 有锚定标准,避免难度地板 | 每个 prompt 附 2 个正面示例(medium/hard)和 2 个负面示例(too-easy),带评分理由 |
|
||||||
|
| P1-5 | **题集难度分布控制** | 确保 easy/med/hard 比例合理 | 后处理阶段按难度评分排序,按 20/50/30 目标裁剪,不足时重采 |
|
||||||
|
|
||||||
|
### P1-6 重点:消除 Object Reasoning 在 RETRIEVAL 中的矛盾
|
||||||
|
|
||||||
|
将 Object Reasoning 从 RETRIEVAL 家族的 `legal_task_types` 中移除。Object Reasoning 只应出现在 REASONING 家族。当前双家族配置导致 ~50% 的 ObjReas 题目在"禁止推理"的 RETRIEVAL prompt 下生成,本质上是识别题。
|
||||||
|
|
||||||
|
### P2: 长期架构改进
|
||||||
|
|
||||||
|
| # | 改进项 | 预期收益 | 复杂度 |
|
||||||
|
|---|-------|---------|-------|
|
||||||
|
| P2-1 | **Full-video 采样路径** | 支持 full_video 深度题(VME 41% 需要) | 需新增 L0(整棵树)采样层,采集全视频摘要/时间线/实体清单作为 context |
|
||||||
|
| P2-2 | **将 heavy_check 改为前置阻塞 gate** | 难度验证从抽检变为全量拦截 | 重构 gate pipeline,`difficulty_steps` 作为第 5 gate 的量化指标 |
|
||||||
|
| P2-3 | **引入"对抗性"题目生成** | 模拟 VME 的近距离干扰项设计 | 两阶段生成:先生成题目,再用另一 LLM 对抗性改写干扰项使其更接近正确答案 |
|
||||||
|
| P2-4 | **跨题型去重与平衡** | 消除"40% 的题实质是 OCR"的长尾效应 | 题目验收后做跨题型语义聚类,标记实质题型 vs 标注题型不一致的题目 |
|
||||||
|
|
||||||
|
### 优先级总结
|
||||||
|
|
||||||
|
```
|
||||||
|
立即执行 (P0, 1-2 天):
|
||||||
|
P0-1 Prompt 题型分支 ──────────── 消除最严重的题型错配
|
||||||
|
P0-2 采样约束题型化 ──────────── 解除 L3/no-frame 限制
|
||||||
|
P0-3 提升采样层级 ────────────── 从单帧到多帧
|
||||||
|
P0-5 难度门控 gate ────────────── 直接拦截简单题
|
||||||
|
|
||||||
|
短期设计 (P1, 3-5 天):
|
||||||
|
P1-1 新增 6 个缺失题型模板 ──── 覆盖率 50% -> 100%
|
||||||
|
P1-2 新增 TEMPORAL/SYNOPSIS 家族
|
||||||
|
P1-4 Prompt 难度校准示例
|
||||||
|
|
||||||
|
中期架构 (P2, 1-2 周):
|
||||||
|
P2-1 Full-video 采样路径
|
||||||
|
P2-3 对抗性干扰项生成
|
||||||
|
```
|
||||||
@@ -160,6 +160,11 @@
|
|||||||
"id": "plan:action-recognition-training",
|
"id": "plan:action-recognition-training",
|
||||||
"label": "Action Recognition 单题型首次训练实验计划",
|
"label": "Action Recognition 单题型首次训练实验计划",
|
||||||
"type": "plan"
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:task-type-strategy-framework",
|
||||||
|
"label": "TaskTypeStrategy 框架实现计划 (Plan A)",
|
||||||
|
"type": "plan"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"links": [
|
"links": [
|
||||||
@@ -288,6 +293,13 @@
|
|||||||
"relation": "implements",
|
"relation": "implements",
|
||||||
"evidence": "计划实现设计文档中的 2 处代码修改 + 实验配置 + 训练脚本",
|
"evidence": "计划实现设计文档中的 2 处代码修改 + 实验配置 + 训练脚本",
|
||||||
"added": "2026-07-14T04:50:15.986586+00:00"
|
"added": "2026-07-14T04:50:15.986586+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:task-type-strategy-framework",
|
||||||
|
"target": "design:task-type-strategy",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "Plan A 实现设计中的框架层(Protocol + Base + pipeline 集成)",
|
||||||
|
"added": "2026-07-14T09:15:22.951263+00:00"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
# Research Wiki 索引
|
# Research Wiki 索引
|
||||||
|
|
||||||
> 自动生成,更新时间:2026-07-14 04:50 UTC
|
> 自动生成,更新时间:2026-07-14 09:15 UTC
|
||||||
|
|
||||||
## design (24)
|
## design (25)
|
||||||
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
||||||
- [2026-07-07-app-harness-design](designs/2026-07-07-app-harness-design.md) `design:2026-07-07-app-harness-design`
|
- [2026-07-07-app-harness-design](designs/2026-07-07-app-harness-design.md) `design:2026-07-07-app-harness-design`
|
||||||
- [2026-07-07-core-evolution-design](designs/2026-07-07-core-evolution-design.md) `design:2026-07-07-core-evolution-design`
|
- [2026-07-07-core-evolution-design](designs/2026-07-07-core-evolution-design.md) `design:2026-07-07-core-evolution-design`
|
||||||
@@ -20,6 +20,7 @@
|
|||||||
- [Spec-2 建树批量并行入口](designs/batch-tree-build.md) `design:batch-tree-build`
|
- [Spec-2 建树批量并行入口](designs/batch-tree-build.md) `design:batch-tree-build`
|
||||||
- [Spec-3 出题管线 v2(失败机理靶向+逐题质量门)](designs/question-gen-v2.md) `design:question-gen-v2`
|
- [Spec-3 出题管线 v2(失败机理靶向+逐题质量门)](designs/question-gen-v2.md) `design:question-gen-v2`
|
||||||
- [出题模块迁移设计(question_gen)](designs/2026-07-07-question-gen-design.md) `design:2026-07-07-question-gen-design`
|
- [出题模块迁移设计(question_gen)](designs/2026-07-07-question-gen-design.md) `design:2026-07-07-question-gen-design`
|
||||||
|
- [出题管线 TaskTypeStrategy 拆分设计(Clean Architecture)](designs/2026-07-14-task-type-strategy-design.md) `design:2026-07-14-task-type-strategy-design`
|
||||||
- [建树修复管线:熔断根因修复 + 断点续跑 + 并发改造](designs/tree-repair-resilience.md) `design:tree-repair-resilience`
|
- [建树修复管线:熔断根因修复 + 断点续跑 + 并发改造](designs/tree-repair-resilience.md) `design:tree-repair-resilience`
|
||||||
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/2026-07-07-tree-module-design.md) `design:2026-07-07-tree-module-design`
|
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/2026-07-07-tree-module-design.md) `design:2026-07-07-tree-module-design`
|
||||||
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/tree-module-vertical-slice.md) `design:tree-module-vertical-slice`
|
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/tree-module-vertical-slice.md) `design:tree-module-vertical-slice`
|
||||||
@@ -28,13 +29,14 @@
|
|||||||
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
||||||
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
||||||
|
|
||||||
## finding (4)
|
## finding (5)
|
||||||
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
||||||
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
||||||
|
- [2026-07-14-question-quality-gap-analysis](findings/2026-07-14-question-quality-gap-analysis.md) `finding:2026-07-14-question-quality-gap-analysis`
|
||||||
- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check`
|
- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check`
|
||||||
- [Harness 评估: Spec-2 批量并行建树](findings/eval-spec2-batch-tree-build.md) `finding:eval-spec2-batch-tree-build`
|
- [Harness 评估: Spec-2 批量并行建树](findings/eval-spec2-batch-tree-build.md) `finding:eval-spec2-batch-tree-build`
|
||||||
|
|
||||||
## plan (26)
|
## plan (28)
|
||||||
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
||||||
- [2026-07-07-app-harness](plans/2026-07-07-app-harness.md) `plan:2026-07-07-app-harness`
|
- [2026-07-07-app-harness](plans/2026-07-07-app-harness.md) `plan:2026-07-07-app-harness`
|
||||||
- [2026-07-07-core-evolution](plans/2026-07-07-core-evolution.md) `plan:2026-07-07-core-evolution`
|
- [2026-07-07-core-evolution](plans/2026-07-07-core-evolution.md) `plan:2026-07-07-core-evolution`
|
||||||
@@ -47,6 +49,7 @@
|
|||||||
- [2026-07-11-batch-tree-build](plans/2026-07-11-batch-tree-build.md) `plan:2026-07-11-batch-tree-build`
|
- [2026-07-11-batch-tree-build](plans/2026-07-11-batch-tree-build.md) `plan:2026-07-11-batch-tree-build`
|
||||||
- [2026-07-12-per-category-pool-strategy](plans/2026-07-12-per-category-pool-strategy.md) `plan:2026-07-12-per-category-pool-strategy`
|
- [2026-07-12-per-category-pool-strategy](plans/2026-07-12-per-category-pool-strategy.md) `plan:2026-07-12-per-category-pool-strategy`
|
||||||
- [2026-07-14-action-recognition-training](plans/2026-07-14-action-recognition-training.md) `plan:2026-07-14-action-recognition-training`
|
- [2026-07-14-action-recognition-training](plans/2026-07-14-action-recognition-training.md) `plan:2026-07-14-action-recognition-training`
|
||||||
|
- [2026-07-14-task-type-strategy-framework](plans/2026-07-14-task-type-strategy-framework.md) `plan:2026-07-14-task-type-strategy-framework`
|
||||||
- [Action Recognition 单题型首次训练实验计划](plans/action-recognition-training.md) `plan:action-recognition-training`
|
- [Action Recognition 单题型首次训练实验计划](plans/action-recognition-training.md) `plan:action-recognition-training`
|
||||||
- [app/harness/ 训练循环编排层实现计划](plans/app-harness.md) `plan:app-harness`
|
- [app/harness/ 训练循环编排层实现计划](plans/app-harness.md) `plan:app-harness`
|
||||||
- [app/search/ 搜索 Agent 装配层实现计划](plans/2026-07-07-search-module.md) `plan:2026-07-07-search-module`
|
- [app/search/ 搜索 Agent 装配层实现计划](plans/2026-07-07-search-module.md) `plan:2026-07-07-search-module`
|
||||||
@@ -56,6 +59,7 @@
|
|||||||
- [question_gen 模块实现计划](plans/question-gen.md) `plan:question-gen`
|
- [question_gen 模块实现计划](plans/question-gen.md) `plan:question-gen`
|
||||||
- [Spec-1 Agent 执行环境修复实现计划](plans/agent-runtime-fixes-plan.md) `plan:agent-runtime-fixes-plan`
|
- [Spec-1 Agent 执行环境修复实现计划](plans/agent-runtime-fixes-plan.md) `plan:agent-runtime-fixes-plan`
|
||||||
- [Spec-2 建树批量并行实现计划](plans/batch-tree-build-plan.md) `plan:batch-tree-build-plan`
|
- [Spec-2 建树批量并行实现计划](plans/batch-tree-build-plan.md) `plan:batch-tree-build-plan`
|
||||||
|
- [TaskTypeStrategy 框架实现计划 (Plan A)](plans/task-type-strategy-framework.md) `plan:task-type-strategy-framework`
|
||||||
- [出题管线 v2 实现计划](plans/2026-07-11-question-gen-v2.md) `plan:2026-07-11-question-gen-v2`
|
- [出题管线 v2 实现计划](plans/2026-07-11-question-gen-v2.md) `plan:2026-07-11-question-gen-v2`
|
||||||
- [建树修复管线三项改造实现计划](plans/tree-repair-resilience.md) `plan:tree-repair-resilience`
|
- [建树修复管线三项改造实现计划](plans/tree-repair-resilience.md) `plan:tree-repair-resilience`
|
||||||
- [建树模块竖切实现计划](plans/tree-module-vertical-slice.md) `plan:tree-module-vertical-slice`
|
- [建树模块竖切实现计划](plans/tree-module-vertical-slice.md) `plan:tree-module-vertical-slice`
|
||||||
|
|||||||
@@ -73,3 +73,6 @@
|
|||||||
- [2026-07-14 04:50 UTC] 新增 plan: Action Recognition 单题型首次训练实验计划 (plan:action-recognition-training)
|
- [2026-07-14 04:50 UTC] 新增 plan: Action Recognition 单题型首次训练实验计划 (plan:action-recognition-training)
|
||||||
- [2026-07-14 04:50 UTC] 新增边: plan:action-recognition-training --implements--> design:action-recognition-training
|
- [2026-07-14 04:50 UTC] 新增边: plan:action-recognition-training --implements--> design:action-recognition-training
|
||||||
- [2026-07-14 04:50 UTC] 重建索引: 60 篇页面
|
- [2026-07-14 04:50 UTC] 重建索引: 60 篇页面
|
||||||
|
- [2026-07-14 09:15 UTC] 新增 plan: TaskTypeStrategy 框架实现计划 (Plan A) (plan:task-type-strategy-framework)
|
||||||
|
- [2026-07-14 09:15 UTC] 新增边: plan:task-type-strategy-framework --implements--> design:task-type-strategy
|
||||||
|
- [2026-07-14 09:15 UTC] 重建索引: 64 篇页面
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,751 @@
|
|||||||
|
# TaskTypeStrategy 框架实现计划 (Plan A)
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 将出题管线从 QuestionFamilySpec 替换为 TaskTypeStrategy 接口,12 个题型(含 AR 临时 Base)用 BaseTaskTypeStrategy 封装现有 family 行为,pipeline/sampler/generator/gates/store 签名适配。
|
||||||
|
|
||||||
|
**Architecture:** 新建 `strategy.py`(Protocol + SubPattern + Base + 注册表),修改 pipeline/sampler/generator/gates/store 5 个文件的函数签名,从 `family_spec` 参数改为 strategy 提供的离散参数。BaseTaskTypeStrategy 内部委托给绑定的 QuestionFamilySpec,确保 12 个题型行为不变。AR 临时绑定 VISUAL family(Plan B 替换为特化策略)。
|
||||||
|
|
||||||
|
**Codex 审查修复:**
|
||||||
|
- C1: sub_pattern 空值判护 — `sub_pattern.x if sub_pattern else strategy.x`
|
||||||
|
- C2: AR 在 Plan A 临时绑定 VISUAL family,Plan B 替换为 ActionRecognitionStrategy
|
||||||
|
- C3: sampler 异常消息同步改为使用 task_type
|
||||||
|
- C4: run_store 加幂等 ALTER TABLE 迁移
|
||||||
|
- I1: pipeline 显式调用 extra_gates 并合并结果
|
||||||
|
- I2: 移除 build_prompt_context,改为离散参数(prompt_template/strategy_name/skill_target/sub_pattern_instruction)
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11, pytest, Protocol (typing)
|
||||||
|
|
||||||
|
**关联设计:** `research-wiki/designs/2026-07-14-task-type-strategy-design.md`
|
||||||
|
|
||||||
|
**范围:** 仅 Plan A(框架 + Base)。ActionRecognitionStrategy 特化实现在 Plan B。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: strategy.py — Protocol + SubPattern + BaseTaskTypeStrategy + 注册表
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/question_gen/strategy.py`
|
||||||
|
- Test: `tests/unit/test_strategy.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试 — Protocol 和 Base 行为**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_strategy.py
|
||||||
|
"""TaskTypeStrategy Protocol 与 BaseTaskTypeStrategy 单元测试。"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import random
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.question_gen.families import (
|
||||||
|
ENUMERATION_FAMILY,
|
||||||
|
REASONING_FAMILY,
|
||||||
|
RETRIEVAL_FAMILY,
|
||||||
|
SPATIAL_FAMILY,
|
||||||
|
VISUAL_FAMILY,
|
||||||
|
)
|
||||||
|
from app.question_gen.strategy import (
|
||||||
|
BaseTaskTypeStrategy,
|
||||||
|
SubPattern,
|
||||||
|
get_strategy,
|
||||||
|
register_strategy,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestBaseTaskTypeStrategy:
|
||||||
|
"""BaseTaskTypeStrategy 封装 family 行为。"""
|
||||||
|
|
||||||
|
def test_task_type_and_strategy_name(self):
|
||||||
|
"""task_type 和 strategy_name 正确返回。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY)
|
||||||
|
assert s.task_type == "Object Recognition"
|
||||||
|
assert s.strategy_name == "RETRIEVAL"
|
||||||
|
|
||||||
|
def test_sampling_from_family(self):
|
||||||
|
"""采样参数从绑定的 family 读取。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Temporal Reasoning", family=ENUMERATION_FAMILY)
|
||||||
|
assert s.sampling_level == 1
|
||||||
|
assert s.sampling_constraint.min_l3_nodes == 5
|
||||||
|
|
||||||
|
def test_skill_target_from_family(self):
|
||||||
|
"""skill_target 从 family 读取。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Action Reasoning", family=REASONING_FAMILY)
|
||||||
|
assert s.skill_target == "M2"
|
||||||
|
|
||||||
|
def test_leak_probe_template(self):
|
||||||
|
"""leak_probe_template 从 family.leak_profile 读取。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Spatial Reasoning", family=SPATIAL_FAMILY)
|
||||||
|
assert s.leak_probe_template == "gate_leak_spatial.md"
|
||||||
|
|
||||||
|
def test_prompt_template_from_family(self):
|
||||||
|
"""prompt_template 从 family 读取。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="OCR Problems", family=VISUAL_FAMILY)
|
||||||
|
assert s.prompt_template == "visual.md"
|
||||||
|
|
||||||
|
def test_select_sub_pattern_returns_none(self):
|
||||||
|
"""BaseTaskTypeStrategy 无子模式。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY)
|
||||||
|
rng = random.Random(42)
|
||||||
|
assert s.select_sub_pattern(rng) is None
|
||||||
|
|
||||||
|
def test_extra_gates_returns_empty(self):
|
||||||
|
"""BaseTaskTypeStrategy 无额外 gate。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY)
|
||||||
|
assert s.extra_gates(None) == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestSamplingLevelMapping:
|
||||||
|
"""BaseTaskTypeStrategy 的 sampling_level 从 _TASK_TYPE_TO_LEVEL 读取。"""
|
||||||
|
|
||||||
|
def test_l3_types(self):
|
||||||
|
"""L3 题型。"""
|
||||||
|
for tt in ("Object Recognition",):
|
||||||
|
s = BaseTaskTypeStrategy(task_type=tt, family=RETRIEVAL_FAMILY)
|
||||||
|
assert s.sampling_level == 3, f"{tt} should be L3"
|
||||||
|
|
||||||
|
def test_l2_types(self):
|
||||||
|
"""L2 题型。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Action Reasoning", family=REASONING_FAMILY)
|
||||||
|
assert s.sampling_level == 2
|
||||||
|
|
||||||
|
def test_l1_types(self):
|
||||||
|
"""L1 题型。"""
|
||||||
|
s = BaseTaskTypeStrategy(task_type="Temporal Reasoning", family=ENUMERATION_FAMILY)
|
||||||
|
assert s.sampling_level == 1
|
||||||
|
|
||||||
|
|
||||||
|
class TestStrategyRegistry:
|
||||||
|
"""注册表查找。"""
|
||||||
|
|
||||||
|
def test_get_unregistered_returns_base(self):
|
||||||
|
"""未注册题型返回 BaseTaskTypeStrategy。"""
|
||||||
|
s = get_strategy("Spatial Reasoning")
|
||||||
|
assert isinstance(s, BaseTaskTypeStrategy)
|
||||||
|
assert s.task_type == "Spatial Reasoning"
|
||||||
|
|
||||||
|
def test_register_and_get(self):
|
||||||
|
"""注册后 get 返回注册的策略。"""
|
||||||
|
custom = BaseTaskTypeStrategy(task_type="Object Recognition", family=RETRIEVAL_FAMILY)
|
||||||
|
register_strategy(custom)
|
||||||
|
assert get_strategy("Object Recognition") is custom
|
||||||
|
|
||||||
|
|
||||||
|
class TestSubPattern:
|
||||||
|
"""SubPattern 数据类。"""
|
||||||
|
|
||||||
|
def test_frozen(self):
|
||||||
|
"""SubPattern 不可变。"""
|
||||||
|
sp = SubPattern(
|
||||||
|
name="test",
|
||||||
|
weight=0.5,
|
||||||
|
sampling_level_override=None,
|
||||||
|
constraint_override=None,
|
||||||
|
instruction="test instruction",
|
||||||
|
positive_examples=[],
|
||||||
|
negative_examples=[],
|
||||||
|
distractor_rules="",
|
||||||
|
)
|
||||||
|
with pytest.raises(AttributeError):
|
||||||
|
sp.name = "changed"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行测试验证失败**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/unit/test_strategy.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:ImportError — `app.question_gen.strategy` 不存在。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现 strategy.py**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/question_gen/strategy.py
|
||||||
|
"""题型出题策略 — Clean Architecture 的 Strategy 层。
|
||||||
|
|
||||||
|
将出题管线从 5 个粗粒度 QuestionFamilySpec 替换为 12 个题型级别的
|
||||||
|
TaskTypeStrategy。BaseTaskTypeStrategy 封装现有 family 行为,
|
||||||
|
特化策略(如 ActionRecognitionStrategy)自包含。
|
||||||
|
|
||||||
|
典型用法::
|
||||||
|
|
||||||
|
strategy = get_strategy("Action Reasoning")
|
||||||
|
level = strategy.sampling_level
|
||||||
|
constraint = strategy.sampling_constraint
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||||
|
|
||||||
|
from app.question_gen.families import (
|
||||||
|
ENUMERATION_FAMILY,
|
||||||
|
REASONING_FAMILY,
|
||||||
|
RETRIEVAL_FAMILY,
|
||||||
|
SPATIAL_FAMILY,
|
||||||
|
VISUAL_FAMILY,
|
||||||
|
QuestionFamilySpec,
|
||||||
|
SamplingConstraint,
|
||||||
|
)
|
||||||
|
from app.question_gen.sampler_v2 import _TASK_TYPE_TO_LEVEL
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
import random
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SubPattern
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SubPattern:
|
||||||
|
"""出题子模式 — 靶向特定失败机制。
|
||||||
|
|
||||||
|
属性:
|
||||||
|
name: 子模式标识名。
|
||||||
|
weight: 选择权重(按比例分配)。
|
||||||
|
sampling_level_override: 覆盖策略默认采样层级(None = 不覆盖)。
|
||||||
|
constraint_override: 覆盖策略默认采样约束(None = 不覆盖)。
|
||||||
|
instruction: 核心出题指令(1-3 句话)。
|
||||||
|
positive_examples: VME 原题 few-shot 示范。
|
||||||
|
negative_examples: 反面示例。
|
||||||
|
distractor_rules: 干扰项构造规则。
|
||||||
|
"""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
weight: float
|
||||||
|
sampling_level_override: int | None
|
||||||
|
constraint_override: SamplingConstraint | None
|
||||||
|
instruction: str
|
||||||
|
positive_examples: list[dict] = field(default_factory=list)
|
||||||
|
negative_examples: list[dict] = field(default_factory=list)
|
||||||
|
distractor_rules: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Protocol
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class TaskTypeStrategy(Protocol):
|
||||||
|
"""题型出题策略 — pipeline 的唯一接口。"""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def task_type(self) -> str: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sampling_level(self) -> int: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sampling_constraint(self) -> SamplingConstraint: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prompt_template(self) -> str: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def strategy_name(self) -> str: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def skill_target(self) -> str: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def leak_probe_template(self) -> str: ...
|
||||||
|
|
||||||
|
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None: ...
|
||||||
|
|
||||||
|
def build_prompt_context(
|
||||||
|
self, material: Any, sub_pattern: SubPattern | None
|
||||||
|
) -> dict: ...
|
||||||
|
|
||||||
|
def extra_gates(self, candidate: Any) -> list: ...
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# BaseTaskTypeStrategy
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# 消歧绑定表:多归属题型确定性绑定到一个 family
|
||||||
|
_TASK_TYPE_TO_FAMILY: dict[str, QuestionFamilySpec] = {
|
||||||
|
"Action Recognition": VISUAL_FAMILY, # Plan A 临时绑定;Plan B 替换为特化策略
|
||||||
|
"Object Recognition": RETRIEVAL_FAMILY,
|
||||||
|
"Object Reasoning": REASONING_FAMILY,
|
||||||
|
"Action Reasoning": REASONING_FAMILY,
|
||||||
|
"Attribute Perception": VISUAL_FAMILY,
|
||||||
|
"OCR Problems": VISUAL_FAMILY,
|
||||||
|
"Counting Problem": ENUMERATION_FAMILY,
|
||||||
|
"Information Synopsis": REASONING_FAMILY,
|
||||||
|
"Temporal Reasoning": ENUMERATION_FAMILY,
|
||||||
|
"Temporal Perception": ENUMERATION_FAMILY,
|
||||||
|
"Spatial Reasoning": SPATIAL_FAMILY,
|
||||||
|
"Spatial Perception": SPATIAL_FAMILY,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BaseTaskTypeStrategy:
|
||||||
|
"""封装现有 QuestionFamilySpec 行为的默认策略。
|
||||||
|
|
||||||
|
所有属性委托给绑定的 family,确保未特化题型的行为不变。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
task_type: 题型名。
|
||||||
|
family: 绑定的 QuestionFamilySpec。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, task_type: str, family: QuestionFamilySpec) -> None:
|
||||||
|
self._task_type = task_type
|
||||||
|
self._family = family
|
||||||
|
|
||||||
|
@property
|
||||||
|
def task_type(self) -> str:
|
||||||
|
return self._task_type
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sampling_level(self) -> int:
|
||||||
|
return _TASK_TYPE_TO_LEVEL[self._task_type]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sampling_constraint(self) -> SamplingConstraint:
|
||||||
|
return self._family.sampling
|
||||||
|
|
||||||
|
@property
|
||||||
|
def prompt_template(self) -> str:
|
||||||
|
return self._family.prompt_template
|
||||||
|
|
||||||
|
@property
|
||||||
|
def strategy_name(self) -> str:
|
||||||
|
return self._family.name
|
||||||
|
|
||||||
|
@property
|
||||||
|
def skill_target(self) -> str:
|
||||||
|
return self._family.skill_target
|
||||||
|
|
||||||
|
@property
|
||||||
|
def leak_probe_template(self) -> str:
|
||||||
|
return self._family.leak_profile.probe_template
|
||||||
|
|
||||||
|
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None:
|
||||||
|
"""BaseTaskTypeStrategy 无子模式。"""
|
||||||
|
return None
|
||||||
|
|
||||||
|
def build_prompt_context(
|
||||||
|
self, material: Any, sub_pattern: SubPattern | None
|
||||||
|
) -> dict:
|
||||||
|
"""返回基础 prompt 上下文(family_name + prompt_template)。"""
|
||||||
|
return {
|
||||||
|
"family_name": self._family.name,
|
||||||
|
"prompt_template": self._family.prompt_template,
|
||||||
|
}
|
||||||
|
|
||||||
|
def extra_gates(self, candidate: Any) -> list:
|
||||||
|
"""BaseTaskTypeStrategy 无额外 gate。"""
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 注册表
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def register_strategy(strategy: TaskTypeStrategy) -> None:
|
||||||
|
"""注册一个题型策略。同一 task_type 重复注册会覆盖。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
strategy: 实现 TaskTypeStrategy 接口的策略实例。
|
||||||
|
"""
|
||||||
|
_STRATEGY_REGISTRY[strategy.task_type] = strategy
|
||||||
|
|
||||||
|
|
||||||
|
def get_strategy(task_type: str) -> TaskTypeStrategy:
|
||||||
|
"""获取题型策略。未注册的自动创建 BaseTaskTypeStrategy。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
task_type: 题型名。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
TaskTypeStrategy 实例。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
KeyError: task_type 不在消歧绑定表和注册表中。
|
||||||
|
"""
|
||||||
|
if task_type in _STRATEGY_REGISTRY:
|
||||||
|
return _STRATEGY_REGISTRY[task_type]
|
||||||
|
return _build_default_strategy(task_type)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_default_strategy(task_type: str) -> BaseTaskTypeStrategy:
|
||||||
|
"""根据消歧绑定表自动构造 BaseTaskTypeStrategy。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
task_type: 题型名。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
BaseTaskTypeStrategy 实例。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
KeyError: task_type 不在消歧绑定表中。
|
||||||
|
"""
|
||||||
|
family = _TASK_TYPE_TO_FAMILY[task_type]
|
||||||
|
return BaseTaskTypeStrategy(task_type=task_type, family=family)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试验证通过**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/unit/test_strategy.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:全部 PASS
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/question_gen/strategy.py tests/unit/test_strategy.py
|
||||||
|
git commit -m "feat(question_gen): add TaskTypeStrategy Protocol and BaseTaskTypeStrategy"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: sampler_v2.py 签名适配
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/question_gen/sampler_v2.py:495-530`
|
||||||
|
- Test: `tests/unit/test_sampler_v2.py`(现有测试应继续通过)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 修改 sample_material_v2 签名**
|
||||||
|
|
||||||
|
将 `family_spec: QuestionFamilySpec` 参数替换为 `level: int` + `constraint: SamplingConstraint`:
|
||||||
|
|
||||||
|
在 `app/question_gen/sampler_v2.py` 中,修改 `sample_material_v2` 函数签名和内部使用:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def sample_material_v2(
|
||||||
|
tree: TreeIndex,
|
||||||
|
task_type: str,
|
||||||
|
used_node_ids: set[str],
|
||||||
|
rng: random.Random,
|
||||||
|
*,
|
||||||
|
level: int,
|
||||||
|
constraint: SamplingConstraint,
|
||||||
|
max_attempts: int = 10,
|
||||||
|
) -> MaterialContext:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数体内将 `level = _TASK_TYPE_TO_LEVEL[task_type]` 和 `constraint = family_spec.sampling` 两行删除(改为直接使用参数)。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 修改 pipeline_v2.py 中的调用点**
|
||||||
|
|
||||||
|
在 `_process_one_slot` 中(约 371 行),将:
|
||||||
|
|
||||||
|
```python
|
||||||
|
material = sample_material_v2(
|
||||||
|
tree=current_tree,
|
||||||
|
family_spec=slot.family,
|
||||||
|
task_type=slot.task_type,
|
||||||
|
used_node_ids=used_node_ids,
|
||||||
|
rng=rng,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
改为:
|
||||||
|
|
||||||
|
```python
|
||||||
|
material = sample_material_v2(
|
||||||
|
tree=current_tree,
|
||||||
|
task_type=slot.task_type,
|
||||||
|
used_node_ids=used_node_ids,
|
||||||
|
rng=rng,
|
||||||
|
level=strategy.sampling_level,
|
||||||
|
constraint=strategy.sampling_constraint,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
(`strategy` 变量在 Task 4 中引入,此处先改签名。)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 运行 sampler 测试**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/unit/test_sampler_v2.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:需要更新测试中的调用签名。修复后全部 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/question_gen/sampler_v2.py tests/unit/test_sampler_v2.py
|
||||||
|
git commit -m "refactor(sampler): replace family_spec param with level+constraint"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: generator_v2.py + gates.py + run_store.py 签名适配
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/question_gen/generator_v2.py:88-104, 112-194, 351-420`
|
||||||
|
- Modify: `app/question_gen/gates.py:399-431, 439-494`
|
||||||
|
- Modify: `app/question_gen/run_store.py:110-128, 242-290`
|
||||||
|
|
||||||
|
- [ ] **Step 1: generator_v2.py — _load_prompt_template 改为接收文件名**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _load_prompt_template(template_name: str) -> str:
|
||||||
|
path = _PROMPTS_DIR / template_name
|
||||||
|
if not path.exists():
|
||||||
|
msg = f"Prompt 模板文件不存在: {path}"
|
||||||
|
raise FileNotFoundError(msg)
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: generator_v2.py — _build_v2_prompt 改为接收离散参数**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _build_v2_prompt(
|
||||||
|
prompt_template: str,
|
||||||
|
strategy_name: str,
|
||||||
|
material: MaterialContext,
|
||||||
|
task_type: str,
|
||||||
|
seq: int,
|
||||||
|
*,
|
||||||
|
reject_reason: str | None = None,
|
||||||
|
sub_pattern_instruction: str | None = None,
|
||||||
|
) -> tuple[list[dict[str, str]], list[str]]:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数体内:
|
||||||
|
- `template_content = _load_prompt_template(family_spec)` → `template_content = _load_prompt_template(prompt_template)`
|
||||||
|
- `family_spec.name` → `strategy_name`
|
||||||
|
- 在 `reject_reason` 注入之后、输出格式之前,如果 `sub_pattern_instruction` 非 None,追加:
|
||||||
|
|
||||||
|
```python
|
||||||
|
if sub_pattern_instruction is not None:
|
||||||
|
user_parts.append(f"\n## Special Focus:\n{sub_pattern_instruction}")
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: generator_v2.py — generate_one_v2 签名适配**
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def generate_one_v2(
|
||||||
|
vlm: VLMProvider,
|
||||||
|
tree: TreeIndex,
|
||||||
|
material: MaterialContext,
|
||||||
|
task_type: str,
|
||||||
|
seq: int,
|
||||||
|
*,
|
||||||
|
video_id: str,
|
||||||
|
prompt_template: str,
|
||||||
|
strategy_name: str,
|
||||||
|
skill_target: str,
|
||||||
|
reject_reason: str | None = None,
|
||||||
|
sub_pattern_instruction: str | None = None,
|
||||||
|
session_id: str,
|
||||||
|
) -> CandidateQuestion:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数体内:
|
||||||
|
- `_build_v2_prompt(family_spec=family_spec, ...)` → `_build_v2_prompt(prompt_template=prompt_template, strategy_name=strategy_name, ..., sub_pattern_instruction=sub_pattern_instruction)`
|
||||||
|
- `family_spec.name` → `strategy_name`
|
||||||
|
- `family_spec.skill_target` → `skill_target`
|
||||||
|
|
||||||
|
- [ ] **Step 4: gates.py — _gate_leak_test 和 run_gates 签名适配**
|
||||||
|
|
||||||
|
`_gate_leak_test`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def _gate_leak_test(
|
||||||
|
candidate: CandidateQuestion,
|
||||||
|
leak_probe_template: str,
|
||||||
|
llm: LLMProvider,
|
||||||
|
*,
|
||||||
|
session_id: str,
|
||||||
|
) -> GateResult:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数体内:`probe_template_name = family_spec.leak_profile.probe_template` → `probe_template_name = leak_probe_template`
|
||||||
|
|
||||||
|
`run_gates`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def run_gates(
|
||||||
|
candidate: CandidateQuestion,
|
||||||
|
tree: TreeIndex,
|
||||||
|
llm: LLMProvider,
|
||||||
|
leak_probe_template: str,
|
||||||
|
postprocess: PostprocessResult,
|
||||||
|
*,
|
||||||
|
vlm: VLMProvider | None = None,
|
||||||
|
session_id: str,
|
||||||
|
) -> GateReport:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数体内:`_gate_leak_test(candidate, family_spec, llm, ...)` → `_gate_leak_test(candidate, leak_probe_template, llm, ...)`
|
||||||
|
|
||||||
|
- [ ] **Step 5: run_store.py — DDL 加 sub_pattern 列 + record_item 加参数**
|
||||||
|
|
||||||
|
DDL `_DDL_ITEMS` 在 `difficulty_steps` 行后加:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
sub_pattern TEXT,
|
||||||
|
```
|
||||||
|
|
||||||
|
`record_item` 新增参数 `sub_pattern: str | None = None`,INSERT 语句加入该列。
|
||||||
|
|
||||||
|
同时在 `_init_schema` 方法中加幂等迁移(兼容已有数据库):
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 幂等迁移:为已有表加 sub_pattern 列
|
||||||
|
cols = {r[1] for r in self._conn.execute("PRAGMA table_info(question_gen_items)")}
|
||||||
|
if "sub_pattern" not in cols:
|
||||||
|
self._conn.execute("ALTER TABLE question_gen_items ADD COLUMN sub_pattern TEXT")
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: 运行全部出题管线测试**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/unit/test_generate_questions.py tests/unit/test_gates.py tests/unit/test_run_store.py tests/unit/test_sampler_v2.py tests/unit/test_families.py tests/integration/test_pipeline_v2.py tests/integration/test_cli_generate_v2.py -v --tb=short
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:需更新测试中的调用签名。修复后全部 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 7: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/question_gen/generator_v2.py app/question_gen/gates.py app/question_gen/run_store.py
|
||||||
|
git commit -m "refactor(question_gen): adapt generator/gates/store signatures for strategy"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: pipeline_v2.py 集成 — 用 strategy 替换 family
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/question_gen/pipeline_v2.py:37, 57-73, 170-211, 300-530`
|
||||||
|
|
||||||
|
- [ ] **Step 1: SlotAssignment 移除 family 字段,改为 strategy_name**
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SlotAssignment:
|
||||||
|
slot_id: str
|
||||||
|
video_id: str
|
||||||
|
task_type: str
|
||||||
|
seq: int
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: _assign_slots 移除 family 选择逻辑**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _assign_slots(
|
||||||
|
video_ids: list[str],
|
||||||
|
task_types: list[str],
|
||||||
|
per_type: int,
|
||||||
|
) -> list[SlotAssignment]:
|
||||||
|
slots: list[SlotAssignment] = []
|
||||||
|
global_seq = 0
|
||||||
|
for task_type in task_types:
|
||||||
|
for i in range(per_type):
|
||||||
|
video_id = video_ids[i % len(video_ids)]
|
||||||
|
global_seq += 1
|
||||||
|
slot_id = f"{task_type}_{global_seq:04d}"
|
||||||
|
slots.append(SlotAssignment(
|
||||||
|
slot_id=slot_id,
|
||||||
|
video_id=video_id,
|
||||||
|
task_type=task_type,
|
||||||
|
seq=global_seq,
|
||||||
|
))
|
||||||
|
return slots
|
||||||
|
```
|
||||||
|
|
||||||
|
不再调用 `get_family_for_slot`,不再需要 `family_ratios` 和 `rng` 参数。
|
||||||
|
|
||||||
|
- [ ] **Step 3: _process_one_slot 通过 strategy 获取参数**
|
||||||
|
|
||||||
|
在函数开头加:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from app.question_gen.strategy import get_strategy
|
||||||
|
strategy = get_strategy(slot.task_type)
|
||||||
|
sub_pattern = strategy.select_sub_pattern(rng)
|
||||||
|
```
|
||||||
|
|
||||||
|
替换所有 `slot.family` 引用:
|
||||||
|
- `slot.family.name` → `strategy.strategy_name`
|
||||||
|
- `slot.family.skill_target` → `strategy.skill_target`
|
||||||
|
- `sample_material_v2(...)` → 采样参数带空值判护:
|
||||||
|
```python
|
||||||
|
level = sub_pattern.sampling_level_override if sub_pattern and sub_pattern.sampling_level_override is not None else strategy.sampling_level
|
||||||
|
constraint = sub_pattern.constraint_override if sub_pattern and sub_pattern.constraint_override is not None else strategy.sampling_constraint
|
||||||
|
sample_material_v2(..., level=level, constraint=constraint, ...)
|
||||||
|
```
|
||||||
|
- `generate_one_v2(...)` → `generate_one_v2(..., prompt_template=strategy.prompt_template, strategy_name=strategy.strategy_name, skill_target=strategy.skill_target, sub_pattern_instruction=sub_pattern.instruction if sub_pattern else None, ...)`
|
||||||
|
- `run_gates(...)` → `run_gates(..., leak_probe_template=strategy.leak_probe_template, ...)`
|
||||||
|
- `store.record_item(...)` → `store.record_item(..., family=strategy.strategy_name, skill_target=strategy.skill_target, sub_pattern=sub_pattern.name if sub_pattern else None, ...)`
|
||||||
|
- gate 后追加 extra_gates 调用:
|
||||||
|
```python
|
||||||
|
extra_results = strategy.extra_gates(candidate)
|
||||||
|
if any(r.verdict == GateVerdict.FAIL for r in extra_results):
|
||||||
|
prev_reason = "; ".join(r.reason for r in extra_results if r.verdict == GateVerdict.FAIL)
|
||||||
|
continue
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: run_pipeline_v2 移除 family_ratios 依赖**
|
||||||
|
|
||||||
|
`_assign_slots` 调用处移除 `config.family_ratios` 和 `rng` 参数。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行集成测试**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/integration/test_pipeline_v2.py tests/integration/test_cli_generate_v2.py -v --tb=short
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:需更新测试中的 SlotAssignment/family 引用。修复后全部 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/question_gen/pipeline_v2.py
|
||||||
|
git commit -m "feat(pipeline): replace QuestionFamilySpec with TaskTypeStrategy"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: 全量回归测试 + lint
|
||||||
|
|
||||||
|
- [ ] **Step 1: Ruff**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & ruff format app/question_gen/ tests/ && ruff check app/question_gen/ --fix
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 全量测试**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda activate Video-Tree-TRM & pytest tests/unit/ tests/integration/ -v --tb=short
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:1160+ 全部 PASS
|
||||||
|
|
||||||
|
- [ ] **Step 3: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add -A && git commit -m "chore: lint and format strategy framework changes"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
本计划不涉及核心算法迁移。修改仅限于出题管线的接口签名替换:
|
||||||
|
- sampler/generator/gates/store 的函数签名从 `family_spec` 改为离散参数
|
||||||
|
- pipeline 的 slot 分配从 family 随机选择改为 strategy 确定性查找
|
||||||
|
- 所有核心出题逻辑(重出循环、后处理、四门 gate 内部、去重、heavy_check)不变
|
||||||
|
|
||||||
|
保真校验不适用。
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:action-recognition-training
|
||||||
|
title: "Action Recognition 单题型首次训练实验计划"
|
||||||
|
date: 2026-07-14
|
||||||
|
---
|
||||||
|
|
||||||
|
# Action Recognition 单题型首次训练实验计划
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:per-category-pool-strategy
|
||||||
|
title: "Per-Category Pool Strategy 实现计划"
|
||||||
|
date: 2026-07-13
|
||||||
|
---
|
||||||
|
|
||||||
|
# Per-Category Pool Strategy 实现计划
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:task-type-strategy-framework
|
||||||
|
title: "TaskTypeStrategy 框架实现计划 (Plan A)"
|
||||||
|
date: 2026-07-14
|
||||||
|
---
|
||||||
|
|
||||||
|
# TaskTypeStrategy 框架实现计划 (Plan A)
|
||||||
|
|
||||||
Reference in New Issue
Block a user