docs(plans): add Spec-3 question-gen-v2 implementation plan
9 tasks: type extension → postprocess → families → sampler_v2 → generator_v2 → gates → run_store → pipeline_v2 → CLI integration. Includes structured-logging schemas/metrics and Codex review revisions.
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: metric
|
||||
node_id: metric:qgen-acceptance-rate
|
||||
title: 最终接受率
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 最终接受率
|
||||
|
||||
## 定义
|
||||
|
||||
最终过门题数 / 目标题数(total_slots)。衡量管线整体产能。
|
||||
|
||||
## 判定方式
|
||||
|
||||
- **类型**: 硬性(下限约束)
|
||||
- **阈值**: ≥ 85%(低于说明管线产能不足,需扩大候选池或降低门阈值)
|
||||
- **计算**: `accepted / total_slots`
|
||||
- **数据源**: `question_gen_runs`
|
||||
|
||||
## 基线
|
||||
|
||||
待首次运行后建立。
|
||||
|
||||
## SQL
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
CAST(accepted AS REAL) / total_slots AS acceptance_rate
|
||||
FROM question_gen_runs
|
||||
WHERE run_id=?;
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
type: metric
|
||||
node_id: metric:qgen-answer-uniformity
|
||||
title: 答案位置分布均匀性
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 答案位置分布均匀性
|
||||
|
||||
## 定义
|
||||
|
||||
最终接受题目中,正确答案在 A/B/C/D 四个位置的分布是否均匀。
|
||||
|
||||
## 判定方式
|
||||
|
||||
- **类型**: 硬性(卡方检验)
|
||||
- **阈值**: χ² 检验 p > 0.05(不拒绝均匀分布假设)
|
||||
- **计算**: 对 accepted 题目统计 correct_answer 的 A/B/C/D 频次,做 χ²(df=3) 检验
|
||||
- **数据源**: 题目 JSON 文件(correct_answer 字段)
|
||||
|
||||
## 基线
|
||||
|
||||
设计 §7.3 要求 shuffle 后必须均匀;不依赖历史基线。
|
||||
|
||||
## 验证方式
|
||||
|
||||
```python
|
||||
from scipy.stats import chisquare
|
||||
observed = [count_A, count_B, count_C, count_D]
|
||||
expected = [total/4] * 4
|
||||
stat, p = chisquare(observed, expected)
|
||||
assert p > 0.05, f"答案位置分布不均匀: p={p:.4f}"
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
type: metric
|
||||
node_id: metric:qgen-gate-rejection-rate
|
||||
title: 出题四门总拦截率
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 出题四门总拦截率
|
||||
|
||||
## 定义
|
||||
|
||||
轻量四门(键验证 + 盲答 + 多真 + 泄漏)在首次尝试中拒绝的题目占比。
|
||||
|
||||
## 判定方式
|
||||
|
||||
- **类型**: 硬性(上限约束)
|
||||
- **阈值**: ≤ 60%(超过说明生成器质量过差,prompt 模板需大幅迭代)
|
||||
- **计算**: `COUNT(attempt=1 AND final_status='rejected') / COUNT(attempt=1)`
|
||||
- **数据源**: `question_gen_items` WHERE `attempt=1`
|
||||
|
||||
## 基线
|
||||
|
||||
待首次运行后建立。
|
||||
|
||||
## SQL
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
CAST(SUM(CASE WHEN final_status='rejected' THEN 1 ELSE 0 END) AS REAL) /
|
||||
COUNT(*) AS rejection_rate
|
||||
FROM question_gen_items
|
||||
WHERE run_id=? AND attempt=1;
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
type: metric
|
||||
node_id: metric:qgen-regen-convergence
|
||||
title: 重出收敛率(3轮内过门)
|
||||
date: 2026-07-12
|
||||
---
|
||||
|
||||
# 重出收敛率(3轮内过门)
|
||||
|
||||
## 定义
|
||||
|
||||
所有 slot 中,在 ≤3 次尝试内最终被接受的 slot 占总 slot 的比例。
|
||||
|
||||
## 判定方式
|
||||
|
||||
- **类型**: 硬性(下限约束)
|
||||
- **阈值**: ≥ 80%(低于说明 prompt 模板或拒因反馈机制需迭代)
|
||||
- **计算**: `COUNT(DISTINCT slot_id WHERE accepted) / COUNT(DISTINCT slot_id)`
|
||||
- **数据源**: `question_gen_items`
|
||||
|
||||
## 基线
|
||||
|
||||
待首次运行后建立。
|
||||
|
||||
## SQL
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
CAST(COUNT(DISTINCT CASE WHEN final_status='accepted' THEN slot_id END) AS REAL) /
|
||||
COUNT(DISTINCT slot_id) AS convergence_rate
|
||||
FROM question_gen_items
|
||||
WHERE run_id=?;
|
||||
```
|
||||
Reference in New Issue
Block a user