docs(plans): add Spec-3 question-gen-v2 implementation plan
9 tasks: type extension → postprocess → families → sampler_v2 → generator_v2 → gates → run_store → pipeline_v2 → CLI integration. Includes structured-logging schemas/metrics and Codex review revisions.
This commit is contained in:
@@ -0,0 +1,97 @@
|
||||
---
|
||||
type: schema
|
||||
node_id: schema:question-gen-items
|
||||
title: "表结构: question_gen_items（逐题门判定）"
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 表结构: question_gen_items（逐题门判定）
|
||||
|
||||
出题管线 v2 的逐题门判定记录。每题每轮（含重出）产生一行。
|
||||
|
||||
## 列定义
|
||||
|
||||
| 列名 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `item_id` | TEXT | PK | 题目唯一 ID（UUID，每轮独立） |
|
||||
| `run_id` | TEXT | NOT NULL FK→question_gen_runs | 关联批次 |
|
||||
| `slot_id` | TEXT | NOT NULL | 逻辑 slot 标识（同 slot 多次重出共享） |
|
||||
| `video_id` | TEXT | NOT NULL | 视频 ID |
|
||||
| `family` | TEXT | NOT NULL | 题族（retrieval/reasoning/enumeration/visual/spatial） |
|
||||
| `task_type` | TEXT | NOT NULL | Video-MME 12 类主标签 |
|
||||
| `skill_target` | TEXT | | M1-M5 + 题族子标签 |
|
||||
| `attempt` | INTEGER | NOT NULL | 当前重出轮次（1-based，≤3） |
|
||||
| `gate_key_verify` | TEXT | | pass / reject / NULL（未执行） |
|
||||
| `gate_blind_answer` | TEXT | | pass / reject / NULL |
|
||||
| `gate_multi_true` | TEXT | | pass / reject / NULL |
|
||||
| `gate_leak_test` | TEXT | | pass / reject / NULL |
|
||||
| `reject_reason` | TEXT | | 拒因文本（首个拒绝门的判定摘要） |
|
||||
| `final_status` | TEXT | NOT NULL | accepted / rejected / pending |
|
||||
| `difficulty_steps` | INTEGER | | 重量抽检产出 Agent 步数（NULL=未抽检） |
|
||||
| `created_at` | TEXT | NOT NULL | ISO8601 |
|
||||
|
||||
## DDL
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS question_gen_items (
|
||||
item_id TEXT PRIMARY KEY,
|
||||
run_id TEXT NOT NULL REFERENCES question_gen_runs(run_id),
|
||||
slot_id TEXT NOT NULL,
|
||||
video_id TEXT NOT NULL,
|
||||
family TEXT NOT NULL,
|
||||
task_type TEXT NOT NULL,
|
||||
skill_target TEXT,
|
||||
attempt INTEGER NOT NULL DEFAULT 1,
|
||||
gate_key_verify TEXT,
|
||||
gate_blind_answer TEXT,
|
||||
gate_multi_true TEXT,
|
||||
gate_leak_test TEXT,
|
||||
reject_reason TEXT,
|
||||
final_status TEXT NOT NULL DEFAULT 'pending',
|
||||
difficulty_steps INTEGER,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_items_run ON question_gen_items(run_id);
|
||||
CREATE INDEX idx_items_family ON question_gen_items(family);
|
||||
CREATE INDEX idx_items_task_type ON question_gen_items(task_type);
|
||||
CREATE INDEX idx_items_slot ON question_gen_items(slot_id);
|
||||
```
|
||||
|
||||
## 埋点位置
|
||||
|
||||
| 代码位置 | 动作 | 时机 |
|
||||
|---------|------|------|
|
||||
| `synthesizer.py::generate_one_v2` 返回后 | INSERT（gates=NULL, status=pending） | 每题生成后 |
|
||||
| `gates.py::run_gates` 返回后 | UPDATE gate_* 列 + reject_reason + final_status | 门跑完后 |
|
||||
| 重出循环入口 | INSERT 新行（same slot_id, attempt+1） | 重出时 |
|
||||
| `heavy_check.py` 完成 | UPDATE difficulty_steps | 抽检完成 |
|
||||
|
||||
## 查询模式
|
||||
|
||||
```sql
|
||||
-- 各门拦截率
|
||||
SELECT
|
||||
gate_key_verify,
|
||||
COUNT(*) as cnt
|
||||
FROM question_gen_items WHERE run_id=?
|
||||
GROUP BY gate_key_verify;
|
||||
|
||||
-- 按题族切片拦截率
|
||||
SELECT family,
|
||||
SUM(CASE WHEN final_status='rejected' THEN 1 ELSE 0 END) as rejected,
|
||||
COUNT(*) as total
|
||||
FROM question_gen_items WHERE run_id=?
|
||||
GROUP BY family;
|
||||
|
||||
-- 重出收敛率
|
||||
SELECT
|
||||
COUNT(DISTINCT slot_id) FILTER (WHERE final_status='accepted') as converged,
|
||||
COUNT(DISTINCT slot_id) as total_slots
|
||||
FROM question_gen_items WHERE run_id=?;
|
||||
|
||||
-- 按 task_type 分布
|
||||
SELECT task_type, COUNT(*) FROM question_gen_items
|
||||
WHERE run_id=? AND final_status='accepted'
|
||||
GROUP BY task_type;
|
||||
```
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
type: schema
|
||||
node_id: schema:question-gen-runs
|
||||
title: "表结构: question_gen_runs（出题批次）"
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 表结构: question_gen_runs（出题批次）
|
||||
|
||||
出题管线 v2 的批次级运行记录。每次 `tools/generate_questions.py generate` 执行产生一行。
|
||||
|
||||
## 列定义
|
||||
|
||||
| 列名 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `run_id` | TEXT | PK | 生成批次唯一 ID（UUID） |
|
||||
| `git_sha` | TEXT | NOT NULL | 代码版本（HEAD commit） |
|
||||
| `config_snapshot` | TEXT | | 科研配置 YAML 快照（JSON 序列化） |
|
||||
| `started_at` | TEXT | NOT NULL | ISO8601 开始时间 |
|
||||
| `finished_at` | TEXT | | ISO8601 结束时间（运行中为 NULL） |
|
||||
| `status` | TEXT | NOT NULL | running / completed / failed |
|
||||
| `total_slots` | INTEGER | | 目标题数（默认 240） |
|
||||
| `accepted` | INTEGER | | 过门题数 |
|
||||
| `rejected` | INTEGER | | 最终拒绝题数 |
|
||||
| `heavy_sampled` | INTEGER | | 重量抽检题数 |
|
||||
|
||||
## DDL
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS question_gen_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
git_sha TEXT NOT NULL,
|
||||
config_snapshot TEXT,
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'running',
|
||||
total_slots INTEGER,
|
||||
accepted INTEGER,
|
||||
rejected INTEGER,
|
||||
heavy_sampled INTEGER
|
||||
);
|
||||
```
|
||||
|
||||
## 埋点位置
|
||||
|
||||
| 代码位置 | 动作 | 时机 |
|
||||
|---------|------|------|
|
||||
| `tools/generate_questions.py::generate` 入口 | INSERT（status=running） | 批次开始 |
|
||||
| `tools/generate_questions.py::generate` 结尾 | UPDATE（finished_at/status/统计汇总） | 批次结束 |
|
||||
|
||||
## 查询模式
|
||||
|
||||
```sql
|
||||
-- 最近一次成功运行
|
||||
SELECT * FROM question_gen_runs WHERE status='completed' ORDER BY started_at DESC LIMIT 1;
|
||||
-- 产能统计
|
||||
SELECT accepted, rejected, heavy_sampled, total_slots FROM question_gen_runs WHERE run_id=?;
|
||||
```
|
||||
Reference in New Issue
Block a user