docs: add question-gen v3 construction-paradigm design and phase1 plan
Complete v3 planning: construction-first paradigm (frame-perception grounded fact extraction + 4-family independent judges + 6-layer verification + QuestionUnit contract), adversarial audit, paradigm-shift finding, real-data spike validation, logging schema, and phase1 contract implementation plan.
This commit is contained in:
@@ -0,0 +1,88 @@
|
||||
---
|
||||
id: v3-question-gen-logging
|
||||
title: 表结构 v3 出题日志/观测(unit_verdict / collapse_metrics / quarantine / facts / resume)
|
||||
type: schema
|
||||
created: 2026-07-15
|
||||
---
|
||||
|
||||
# v3 出题运行时数据 schema
|
||||
|
||||
**载体**:扩展 `app/question_gen/run_store.py`(同一 SQLite,不新建 run_store_v3)。所有 LLM/VLM 调用 telemetry 走 `GovernedLLMClient` 自动录(CLAUDE.md §4.8),本 schema 只管**出题领域观测**。落库统一 `store.insert(...)`,禁裸 SQL 散落。
|
||||
|
||||
## 表 1:`facts`(帧感知抽取 Fact schema,§9)
|
||||
|
||||
| 列 | 类型 | 说明 |
|
||||
|----|------|------|
|
||||
| fact_id | TEXT PK | UUID |
|
||||
| video_id / segment_id | TEXT | 溯源 |
|
||||
| subject / action / object | TEXT | 结构化绑定 |
|
||||
| frame_ids | TEXT(JSON) | 感知所用帧 |
|
||||
| polarity | TEXT | 真/假 |
|
||||
| fact_type | TEXT | binding/state/manner/order/evidence |
|
||||
| difficulty_tier | INT | 感知难度分层(C1;manner/计数=高)|
|
||||
| verifier_refs | TEXT(JSON) | qwen/MiniMax 各自裁决 |
|
||||
| cross_agree | INT | 双 VLM 是否一致 |
|
||||
| negative_at_target | TEXT | 目标点为假的核实结果(C5)|
|
||||
| session_id | TEXT | epoch/step 关联 |
|
||||
|
||||
## 表 2:`unit_verdict`(六层验证每层裁决,§6)
|
||||
|
||||
| 列 | 类型 | 说明 |
|
||||
|----|------|------|
|
||||
| unit_id | TEXT | pair/single 单元 |
|
||||
| pair_id | TEXT? | pair 关联 |
|
||||
| sub_pattern | TEXT | 6 子模式之一 |
|
||||
| stage | INT | 层 1-6 |
|
||||
| verdict | TEXT | pass/fail/abstain |
|
||||
| reason | TEXT | 拒因 |
|
||||
| metric_value | REAL? | 该层量化值 |
|
||||
| model | TEXT? | 裁判模型 |
|
||||
| session_id | TEXT | |
|
||||
|
||||
## 表 3:`collapse_metrics`(配对坍缩度量,§7)
|
||||
|
||||
| 列 | 类型 | 说明 |
|
||||
|----|------|------|
|
||||
| pair_id | TEXT PK | |
|
||||
| text_only_acc / single_frame_acc / placebo_drop | REAL | 模型探针类 |
|
||||
| majority_vote_hit / slot_chi2 / distractor_min_dist | REAL | 纯结构类 |
|
||||
| multiformat_consistency / subtitle_answerability | REAL | 探针类 |
|
||||
|
||||
## 表 4:`quarantine`(失败题指纹去重,§10)
|
||||
|
||||
| 列 | 类型 | 说明 |
|
||||
|----|------|------|
|
||||
| content_fingerprint | TEXT PK | 题面语义内容指纹 |
|
||||
| sub_pattern / quarantine_reason / round_no | TEXT/TEXT/INT | |
|
||||
| ts | TEXT | 时间戳(外部传入,禁进程内 now)|
|
||||
|
||||
## 表 5:`resume_state`(断点续跑,§12)
|
||||
|
||||
| 列 | 类型 | 说明 |
|
||||
|----|------|------|
|
||||
| unit_id | TEXT PK | |
|
||||
| status | TEXT | pending/accepted/rejected |
|
||||
| config_fingerprint | TEXT | 失效检测(求解器/裁判 config 变更作废)|
|
||||
| seq_offset | INT | 补构造续编防撞 |
|
||||
|
||||
## 基线指标(进科研 YAML + run 快照,首次运行后建基线)
|
||||
|
||||
| 指标 | 阈值/口径 | 类别 |
|
||||
|------|----------|------|
|
||||
| 逐维众数命中率 | ≈随机(0.25) | 纯结构(硬地锚)|
|
||||
| pair 关系可解率 | ≈1/16 | 纯结构 |
|
||||
| 双正解率 | 基线 33%→≤5% | 探针+人工 |
|
||||
| 各门通过率 / backfill_yield_by_round | 观测,防减产不可行区 | 系统 |
|
||||
| judge_disagreement_by_subpattern / shared_error_rate | 观测(人工小样本估)| 探针+人工 |
|
||||
| negative_at_target 剔除率 | spike 实测 ~17%(基线)| 结构 |
|
||||
|
||||
## 埋点位置(交 writing-plans,必须作显式步骤)
|
||||
|
||||
| 埋点 | 文件/函数 | 调用 |
|
||||
|------|----------|------|
|
||||
| Fact 落库 | `grounded_fact_extractor`(新建) | `store.insert_fact(...)` |
|
||||
| 逐层裁决 | `pipeline`(主编排)每层出口 | `store.insert_unit_verdict(stage, verdict, reason, ...)` |
|
||||
| 坍缩度量 | `collapse_metric`(新建) | `store.insert_collapse_metrics(...)` |
|
||||
| quarantine | `pipeline` 拒题路径 | `store.quarantine(content_fingerprint, reason)` + 补构造前查黑名单 |
|
||||
| 续跑状态 | `pipeline` on_accept/on_reject | `store.upsert_resume_state(...)` |
|
||||
| LLM/VLM telemetry | 所有 VLM/LLM 调用 | GovernedLLMClient 自动(session_id 透传)|
|
||||
Reference in New Issue
Block a user