iomgaa/Video-Tree-TRM5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: snapshot in-progress question-gen work before preflight fixes

Browse Source
This commit is contained in:
iomgaa
2026-07-16 04:12:21 -04:00
parent 11a5545f57
commit a4c429b247
39 changed files with 738 additions and 283 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/brainstorming/SKILL.md
+9 -1
View File
@@ -22,6 +22,7 @@ Every project goes through this process. A todo list, a single-function utility,
You MUST create a task for each of these items and complete them in order:
1. **Explore project context** — check files, docs, recent commits
1.5. **Prior-version audit (mandatory for rewrites/refactors)** — if the task replaces or rewrites an existing module, list every behavior of the old version (including persistence, crash recovery, idempotency, resume) and confirm each is kept, replaced, or deliberately dropped. Undocumented implicit drops = bugs.
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
@@ -92,7 +93,12 @@ digraph brainstorming {
- Once you believe you understand what you're building, present the design
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
- Ask after each section whether it looks right so far
- Cover: architecture, components, data flow, error handling, testing
- Cover: architecture, components, data flow, error handling, testing, **non-functional requirements** (see below)
- **Non-functional requirements (mandatory section):** Every design MUST explicitly address these four dimensions — even if the answer is "not applicable":
- **Persistence strategy:** When does data hit disk? How much is lost on crash? Overwrite or append?
- **Idempotency:** Is the same operation safe to repeat? Does it produce the same result?
- **Resume/checkpoint:** Can the process recover from interruption? How is progress persisted?
- **Atomicity:** Are writes atomic? Can a partial write corrupt data?
- Be ready to go back and clarify if something doesn't make sense
**Design for isolation and clarity:**
@@ -124,6 +130,8 @@ After writing the spec document, look at it with fresh eyes:
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
5. **Non-functional coverage:** Does the design explicitly address persistence, idempotency, resume, and atomicity? If any dimension is missing, add it now — even if the answer is "not applicable."
6. **Prior-version regression check (rewrites only):** If this replaces an existing module, confirm every behavior from the prior-version audit (step 1.5) is accounted for in the design. Any gap = a spec bug.
Fix any issues inline. No need to re-review — just fix and move on.
CLAUDE.md
+17 -3
View File
@@ -96,9 +96,10 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
### Phase 1: 规划与设计 (Planning)
1. **需求探索**: 涉及创建新功能、新组件、修改行为时，**必须**先调用 `brainstorming` skill 进行需求探索与设计。无论用户的指令多么具体、改动多么简单，都不得跳过此步骤（除非用户显式说"跳过 brainstorming"）。
2. **查阅规格 & 讨论**: 仔细阅读 `research-wiki/`（单一事实源）下对应的文档，了解项目最新情况。对于不理解的地方请与人类进行多轮讨论，确保理解人类的设计意图。
3. **日志方案设计**: 功能会产生运行时数据时，**必须**调用 `structured-logging` skill 设计日志方案。
4. **撰写计划**: 正式编码前，**必须**调用 `writing-plans` skill 撰写实现计划。
5. **审核门控（差异化）**:
3. **前序版本对照（重写/重构时强制）**: 当任务涉及重写或重构已有模块时，**必须**列出前序版本的所有行为（包括持久化策略、崩溃恢复、幂等性、断点续跑等非功能性行为），逐一确认新版本是保留、替代、还是删除。未经确认的隐式删除 = bug。
4. **日志方案设计**: 功能会产生运行时数据时，**必须**调用 `structured-logging` skill 设计日志方案。
5. **撰写计划**: 正式编码前，**必须**调用 `writing-plans` skill 撰写实现计划。
6. **审核门控（差异化）**:
- **design：Claude 自审 → Codex 审 → 人类审**（保留人类门，批准后方可进入计划阶段）。
- **plan：Claude 自审 → Codex 审 → 直接执行**（无 plan 人类门）；plan 经 Claude 自审 + Codex 审通过后直接进入 Phase 2 执行。
@@ -159,6 +160,19 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
- **功能修改**:
- **必须** 不考虑向后兼容，直接修改原文件。代码简洁性优先。
### 4.2.1 设计文档非功能性需求覆盖（强制）
> **教训来源**: v2 出题管线重写时未继承 v1 的逐题追加持久化策略，导致多次 run 的题目丢失。
设计文档**必须**显式覆盖以下非功能性维度（即使答案是"不适用"也要写明）：
| 维度 | 必答问题 |
|------|---------|
| **持久化策略** | 数据何时落盘？崩溃时最多丢多少？是覆盖写还是追加？ |
| **幂等性** | 同一操作重复执行是否安全？结果是否一致？ |
| **断点续跑** | 中断后重启能否从断点恢复？进度如何持久化？ |
| **原子性** | 写操作是否原子？部分写入是否会损坏数据？ |
### 4.3 Git 工作流规范
- **Feature Branch**: 所有开发工作在 feature 分支上进行，**严禁**直接在 main/master 上修改。
- **增量提交**: 频繁提交，每个提交有明确的语义。
app/harness/gate_ladder.py
+5 -19
View File
@@ -27,7 +27,7 @@ from typing import TYPE_CHECKING
from loguru import logger
from app.harness.question_units import build_units
from app.harness.question_units import build_units, unit_correctness
if TYPE_CHECKING:
from pathlib import Path
@@ -65,20 +65,6 @@ class LadderEntry:
p_hat: float
def _unit_correct(unit: QuestionUnit, correctness: dict[str, bool]) -> bool:
"""单元级正确性：AR pair 双向 AND，single 即单题；单元错 = 任一成员错。
参数:
unit: 目标单元。
correctness: question_id -> 是否答对（缺项按未答对处理，与迁移前
correctness.get(qid, False) 的默认语义一致，不改判定）。
返回:
单元内所有成员均答对时 True，否则 False。
"""
return all(correctness.get(q.question_id, False) for q in unit.questions)
def build_cold_entries(
units: list[QuestionUnit],
correctness: dict[str, bool],
@@ -90,7 +76,7 @@ def build_cold_entries(
参数:
units: 该题型的全部候选单元（已排除 test 池；AR pair 已折叠成单元）。
correctness: question_id -> 种子基线是否答对（900 题全量逐题对错）。
单元级对错由 _unit_correct 折叠（任一成员错 → 单元错）。
单元级对错由 unit_correctness(strict=False) 折叠（任一成员错 → 单元错）。
probe_quota: 从错 unit 中随机抽出插到梯尾的探针比例（防"解锁新能力"盲区）。
seed: 洗牌种子，保证确定性重建。
@@ -104,8 +90,8 @@ def build_cold_entries(
错错对 2:1 交错（一方耗尽后顺排另一方）-> 探针追加尾部。
"""
rng = random.Random(seed)
wrong = [u for u in units if not _unit_correct(u, correctness)]
right = [u for u in units if _unit_correct(u, correctness)]
wrong = [u for u in units if not unit_correctness(u, correctness, strict=False)]
right = [u for u in units if unit_correctness(u, correctness, strict=False)]
rng.shuffle(wrong)
rng.shuffle(right)
@@ -125,7 +111,7 @@ def build_cold_entries(
interleaved.extend(probes)
def _p0(u: QuestionUnit) -> float:
return 2 / 3 if _unit_correct(u, correctness) else 1 / 3
return 2 / 3 if unit_correctness(u, correctness, strict=False) else 1 / 3
return [LadderEntry(u.unit_id, _p0(u)) for u in interleaved]
app/harness/question_units.py
+19 -8
View File
@@ -96,24 +96,33 @@ def validate_units(units: list[QuestionUnit]) -> list[QuestionUnit]:
return units
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool]) -> bool:
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool], *, strict: bool = True) -> bool:
"""计算单元级正确性：AR pair 走双向 AND，single 即单题正确性。
参数:
unit: 目标单元。
per_q: 题目 question_id → 该题是否作答正确的映射。
strict: 缺键策略。True（默认）时以 per_q[q.question_id] 取值，缺任一题
触发 KeyError（防静默兜底，强制上游先补齐全部单题结果）；False 时以
per_q.get(q.question_id, False) 取值，缺键计 False（宽松口径，供池
构建 / gate 冷启动 / 采样等"缺基线对错即视为未答对"的调用点复用）。
返回:
单元内所有题目均正确时为 True，否则 False。
关键实现:
直接以 per_q[q.question_id] 取值，缺任一题触发 KeyError（防静默兜底），
强制上游先补齐全部单题结果再计单元正确性。
pool 构建（pools）、gate 冷启动（gate_ladder）、分层采样（loader）三处
原各自持有的 loose 版 _unit_correct 副本统一收敛到本函数 strict=False 分支，
消除重复逻辑与 missing-key 策略分叉。
"""
return all(per_q[q.question_id] for q in unit.questions)
if strict:
return all(per_q[q.question_id] for q in unit.questions)
return all(per_q.get(q.question_id, False) for q in unit.questions)
def unit_correctness_view(units: list[QuestionUnit], per_q: dict[str, bool]) -> dict[str, bool]:
def unit_correctness_view(
units: list[QuestionUnit], per_q: dict[str, bool], *, strict: bool = True
) -> dict[str, bool]:
"""把逐题对错折叠成单元级视图：unit_id → 单元是否整体正确。
进化引擎（gate e-process / quadrant / probation / pair_block / compute_accuracy）
@@ -123,13 +132,15 @@ def unit_correctness_view(units: list[QuestionUnit], per_q: dict[str, bool]) ->
参数:
units: 目标单元列表（single 或 pair）。
per_q: 题目 question_id → 该题是否作答正确（唯一逐题溯源来源）。
strict: 缺键策略，透传给 unit_correctness。True（默认）缺任一题 raise
KeyError；False 缺键计 False（宽松口径）。
返回:
unit_id → 单元级正确性。single 的 unit_id 等于其 question_id，
pair 的 unit_id 等于共享 pair_id。
关键实现:
逐单元复用 unit_correctness（内部以 per_q[q.question_id] 取值，缺任一题
触发 KeyError），禁静默兜底、强制上游先补齐全部单题结果。
逐单元复用 unit_correctness（strict 透传），默认 strict 禁静默兜底、
强制上游先补齐全部单题结果。
"""
return {u.unit_id: unit_correctness(u, per_q) for u in units}
return {u.unit_id: unit_correctness(u, per_q, strict=strict) for u in units}
app/question_gen/families.py
+12 -23
View File
@@ -81,18 +81,11 @@ RETRIEVAL_FAMILY = QuestionFamilySpec(
),
legal_task_types=frozenset(
[
"Action Recognition",
"Action Reasoning",
"Action Prediction",
"Action Sequence",
"Object Recognition",
"Object Reasoning",
"Object Interaction",
"Scene Understanding",
"Event Reasoning",
"Causal Reasoning",
"Temporal Reasoning",
"Spatial Reasoning",
"Action Recognition",
"Attribute Perception",
"OCR Problems",
]
),
leak_profile=LeakTestProfile(
@@ -116,10 +109,7 @@ REASONING_FAMILY = QuestionFamilySpec(
[
"Action Reasoning",
"Object Reasoning",
"Event Reasoning",
"Causal Reasoning",
"Temporal Reasoning",
"Spatial Reasoning",
"Information Synopsis",
]
),
leak_profile=LeakTestProfile(
@@ -141,10 +131,10 @@ ENUMERATION_FAMILY = QuestionFamilySpec(
),
legal_task_types=frozenset(
[
"Action Sequence",
"Object Recognition",
"Object Interaction",
"Scene Understanding",
"Counting Problem",
"Temporal Reasoning",
"Temporal Perception",
"Information Synopsis",
]
),
leak_profile=LeakTestProfile(
@@ -166,10 +156,10 @@ VISUAL_FAMILY = QuestionFamilySpec(
),
legal_task_types=frozenset(
[
"Object Recognition",
"Scene Understanding",
"Attribute Perception",
"Counting Problem",
"OCR Problems",
"Action Recognition",
"Spatial Reasoning",
]
),
leak_profile=LeakTestProfile(
@@ -191,9 +181,8 @@ SPATIAL_FAMILY = QuestionFamilySpec(
),
legal_task_types=frozenset(
[
"Spatial Perception",
"Spatial Reasoning",
"Object Interaction",
"Scene Understanding",
]
),
leak_profile=LeakTestProfile(
app/question_gen/loader.py
+10 -15
View File
@@ -124,19 +124,6 @@ def stratified_sample(
return flatten_units(sampled)
def _unit_correct(unit: QuestionUnit, correctness: dict[str, bool]) -> bool:
"""单元级正确性：成员全部答对才算对（缺失按 False，宽松口径）。
参数:
unit: 目标单元（single 1 题，pair 2 题）。
correctness: question_id -> 基线是否答对。
返回:
pair 走双向 AND、single 即单题正确性；任一成员缺失或答错即 False。
"""
return all(correctness.get(q.question_id, False) for q in unit.questions)
def _ratio_stratified_sample(
pool: list[QuestionUnit],
correctness: dict[str, bool],
@@ -158,9 +145,17 @@ def _ratio_stratified_sample(
异常:
ValueError: 对单元或错单元层不足。
关键实现:
unit_correctness 采用函数内延迟导入：loader 属 question_gen，
question_units 属 harness，模块级导入将触发循环依赖（沿用 build_units /
flatten_units 的既有做法）。以 strict=False 保持"缺基线对错即视为未答对"的
原 loose 语义不变。
"""
correct = [u for u in pool if _unit_correct(u, correctness)]
wrong = [u for u in pool if not _unit_correct(u, correctness)]
from app.harness.question_units import unit_correctness
correct = [u for u in pool if unit_correctness(u, correctness, strict=False)]
wrong = [u for u in pool if not unit_correctness(u, correctness, strict=False)]
n_correct = round(size * correct_ratio)
n_wrong = size - n_correct
if len(correct) < n_correct or len(wrong) < n_wrong:
config/question_gen_180_补.yaml
+81
View File
@@ -0,0 +1,81 @@
# 补生 Video-MME 6 类缺失题型（180 题）
# 原 360 题中有 6 类错误类型已归档，此配置只生成缺失的 6 类。
# ── 建树模块 ──（与 question_gen_360.yaml 一致）
tree:
max_paragraphs_per_l2: 5
l1_segment_duration: 600.0
l2_clip_duration: 60.0
l3_fps: 0.5
l2_representative_frames: 6
cache_dir: "cache/trees"
concurrency: 16
subtitle_inject: true
srt_window_sec: 5.0
# ── Embedding ──
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
# ── Harness ──（占位，出题不使用）
harness:
workspace_dir: "workspaces/default"
store_dir: store
mode: infer
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME"
skills_version: v1
prompts_version: v1
epochs: 1
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
diag_size: 200
diag_correct_ratio: 0.5
val_size: 30
val_correct_ratio: 0.5
test_size: 60
batch_size: 15
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
early_stop_patience: 8
use_slow_momentum: true
# ── 出题管线 v2 ──
question_gen_v2:
family_ratios:
retrieval: 0.30
reasoning: 0.25
enumeration: 0.20
visual: 0.15
spatial: 0.10
dedup_threshold: 0.85
retry_limit: 10
heavy_sample_rate: 0.15
output_dir: "store/questions/generated-v2-180补"
per_type: 30 # 6 类 x 30 = 180 题
concurrency: 24
seed: 43 # 不同于原始 seed=42，避免生成相同题目
config/question_gen_360.yaml
+87
View File
@@ -0,0 +1,87 @@
# config/default.yaml
# 科研实验配置默认值来源（会在实验中反复扫动/对比的参数）。
# 工程配置（少变、敏感）由 .env / pydantic-settings 管理，不在此文件。
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
# ── 建树模块 ──
tree:
max_paragraphs_per_l2: 5
l1_segment_duration: 600.0 # L1 段时长（秒）
l2_clip_duration: 60.0 # L2 clip 时长（秒）
l3_fps: 0.5 # L3 帧提取频率（帧/秒）
l2_representative_frames: 6 # L2 VLM 描述用的代表帧数
cache_dir: "cache/trees"
concurrency: 16 # asyncio Semaphore 上限
subtitle_inject: true # 建树时是否注入 SRT 字幕
srt_window_sec: 5.0 # 字幕匹配时间窗口（前后各 N 秒）
# ── Embedding ──
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
# ── Harness 自进化循环 ──
harness:
workspace_dir: "workspaces/default"
store_dir: store
mode: infer
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME"
skills_version: v1
prompts_version: v1
epochs: 1
# CE-Gate 参数
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 数据池
diag_size: 200
diag_correct_ratio: 0.5
val_size: 30
val_correct_ratio: 0.5
test_size: 60
# mini-batch
batch_size: 15
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
early_stop_patience: 8
use_slow_momentum: true
# ── 出题管线 v2 ──
question_gen_v2:
family_ratios:
retrieval: 0.30
reasoning: 0.25
enumeration: 0.20
visual: 0.15
spatial: 0.10
dedup_threshold: 0.85
retry_limit: 10
heavy_sample_rate: 0.15
output_dir: "store/questions/generated-v2-360"
per_type: 30 # 12 类 x 30 = 360 题
concurrency: 24
seed: 42
config/train_ar30.yaml
+64
View File
@@ -0,0 +1,64 @@
# config/train_ar30.yaml
# Action Recognition 训练 — 基于 SubPattern 靶向生成的 30 题
# 对比基线: v2-360 的 AR 题（100% 单帧，训练无效）
# 本次: AR30 题（6 种失败子模式靶向，跨段时序）
harness:
workspace_dir: "workspaces/train-ar30"
store_dir: store
mode: train
run_id: train_ar30_v1
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "generated-ar30"
skills_version: v1
prompts_version: v1
epochs: 3
# CE-Gate 参数（沿用 default.yaml）
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 池配置 — per_category 单题型
pool_split_mode: per_category
task_types:
- "Action Recognition"
train_ratio: 0.667
test_questions: "benchmarks/Video-MME"
run_holdout_eval: false
# mini-batch
batch_size: 10
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
early_stop_patience: 4
test_size: 63
diag_size: 20
diag_correct_ratio: 0.5
val_size: 10
val_correct_ratio: 0.5
use_slow_momentum: true
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
config/video_split.yaml
+12 -4
View File
@@ -14,10 +14,18 @@ video_split:
val_wrong_min: 20 # validation 池最少错题数（McNemar 检验功效阈 ≈ 20，低于则信号不足）
val_ratio: 0.3 # validation 占 trainval 视频组总数的比例
seed: 7 # 贪心选择器预洗牌 + 视频组题级切分种子（打破等增益 / 等槽平局）
floor_k: # 各高信号 task_type 的 T2 defect 下限（硬约束）—— 占位，标定后替换
Counting Problem: 3 # 取克制值 min(诊断可用 defect 数, 3)，避免把信号全抽进 trainval
Object Reasoning: 3
Action Reasoning: 3
floor_k: # 各 task_type 的 T2 defect 下限（硬约束）—— 均衡覆盖全 11 类，标定于 1cb1c203 真实 T2 分布
Object Reasoning: 5 # T2=25
Information Synopsis: 5 # T2=13
Action Reasoning: 5 # T2=11
Counting Problem: 5 # T2=10
Temporal Reasoning: 2 # T2=5
Object Recognition: 2 # T2=5
Action Recognition: 2 # T2=5
Attribute Perception: 1 # T2=3
Temporal Perception: 1 # T2=2
OCR Problems: 1 # T2=2
Spatial Perception: 1 # T2=1
diag: # 诊断口径指纹三分量（隔离不同诊断配置的信号，参与主键）
prompt_version: diagnose_v1 # 诊断 prompt 版本标识（换 prompt 即换指纹，旧记录不被覆盖）
research-wiki/reviews/2026-07-16-preflight-train-review.md
+51
View File
@@ -0,0 +1,51 @@
# 训练前多维度审查（video-split 冻结切分 → Video-MME 900 训练）
> 2026-07-16。8 维度并行审查 + 每条发现 2 名独立验证员对抗核实：**19 条确认 / 4 条存疑 / 12 条误报被反驳**。
> 场景锚定：global 冻结池 210/90/600、12 题型、baseline=infer_adhoc、concurrency=24、3 epochs、skill_update_mode=patch。
> 关键条目（P0-1/P0-2/微型题型）已由主会话人工复核确认。
## P0 —— 不修则训练无效或必崩
| # | 位置 | 缺陷 | 后果 |
|---|------|------|------|
| P0-1 | `runner.py:2272` + `prompts/` | 进化模板 `evolve_skill/system/tool/rank.md`、`consolidate_system.md` 全部缺失（TRM4 有），`_load_evolve_prompts` 用 `else ""` 静默兜底空串 | 全部进化 LLM 调用以空 system prompt 运行，edits 恒空 → 3 epochs 表面正常跑完但 **零进化**。已跑过的 train-ar30 / train-action-recognition 同样受影响，结果需回查 |
| P0-2 | `inference.py:462` | traces 表只建表、全仓库无写入（TRM4 TracePlugin 未迁移）；训练轨迹只存 `predictions.steps_json` | 训练内 `run_diagnosis` 经 `get_traces` 拿空轨迹 → 诊断瀑布坍缩（算法保真 #7 失效）。离线管线已有 `StepsJsonRunLog` 适配器可复用 |
| P0-3 | `runner.py:2044` / `validate.py:682` / `gate.py:99` | 微型题型三连雷（根因同一）：新冻结切分中 Temporal Perception、Spatial Perception、Spatial Reasoning 在 val 池 **0 题** → `_class_baseline_acc` AssertionError；非 test 单元仅 2 个 → 案例包排除后阶梯为空 raise ValueError；n_plan=1 时 e-process 结构性 reject_inertia | 训练中途必崩两处 + 微型类白烧进化成本。建议训练 task_types 排除微型类，或切分加 per-class val 下限 |
| P0-4 | `inference.py:423,446` | prediction 未归一化且落库 insert 在 try 块之外；LLM 提交 `{"answer": ["B"]}` 等非标量 → sqlite 绑定异常击穿整个 gather | 训练崩溃，且 Redis 缓存重放使 resume 后确定性复现 → 死循环 |
| P0-5 | `runner.py:316` vs `config.py:63` | early_stop_patience 实际按 **step** 计数（每 epoch 一次性累加 ~20），文档语义是"轮" | patience=4 时 epoch 1 只要没严格超过 baseline 就终止全部训练，交付 v1 基线 |
| P0-6 | `momentum.py:151` | `prompts/slow_momentum.md` 缺失，`use_slow_momentum=true` 下无条件 read_text | 修复 P0-1 后必现：首个 accept 的 epoch 末 FileNotFoundError，崩在慢更新中段（叠加慢更新非幂等 → resume 二次污染） |
## P1 —— 信号污染类，强烈建议训练前修
| # | 位置 | 缺陷 |
|---|------|------|
| P1-1 | `main.py:93` + `redis_cache.py:42` | REDIS_CACHE_TTL=0 → 永不过期；缓存键仅 hash(model+messages)，不含采样参数。prompt 未被进化修改的题跨 epoch 逐字节重放首次采样，γ-EMA/e-process 把重放当独立证据 |
| P1-2 | `llm.py:116,575` | SSE 流截断（无 [DONE]）当成功处理并写入永不过期缓存 → 半截答案永久毒化 |
| P1-3 | `llm.py:182` | httpx.RemoteProtocolError/ReadError/ConnectTimeout/PoolTimeout 不在瞬时错误清单，零重试直接落错题；8 次重试预算对最常见断连完全无效 |
| P1-4 | `validate.py:304,593` | 基线臂 INFRA 错误（prediction=None）折叠为"基线答错"永久写 BaselineCache（内容寻址无失效），INFRA 护栏在缓存写入之后才检查 |
| P1-5 | `diagnose.py:2194` + `runner.py:1019` | cause_category=None（judge 基础设施异常）与 degraded 题在训练链路中进 defect 正文进化路径，反转"判不准默认 lapse"的保护方向；runner 对 degraded_count 零检查 |
| P1-6 | `patch.py:343,320` | 冻结区判定只查 edit target 起点不查跨度：起点在正文、末端延伸进 appendix/momentum 区的 delete/replace 被放行 → marker 破坏 → epoch≥2 时 `appendix_region_bounds` ValueError 崩溃 |
| P1-7 | `workspace.py:282,311,365` | manifest.json 全部写路径为裸 write_text 非原子写（checkpoint.py 已有 tmp+replace 先例）；训练高频重写，截断即 workspace 不可恢复 |
## P2 —— 操作规程可规避 / 影响半径小
| # | 位置 | 缺陷 | 规避 |
|---|------|------|------|
| P2-1 | `video_split_cli.py:557` | 冻结产物无覆盖保护（承诺的 --force 门不存在），commit 换 SHA 后重跑会静默替换 pools.json | 立即备份当前冻结产物并记录 sha256；训练前不再重跑切分脚本 |
| P2-2 | `log.py:77` | 只读查询也以 baseline_run_id 打开 HarnessLog → upsert 改写 infer_adhoc 的 _runs 溯源元数据 | 违反 log.py 自身 docstring 约定，宜改走 RunLogImpl |
| P2-3 | `diagnose.py:2081` / `inference.py:461` | predictions/traces 无主键 + step 中途崩溃 resume 同 run_id 重跑 → 重复行双计入诊断统计 | 避免中途 kill；后续补去重 |
| P2-4 | `diagnose.py:2194`（离线） | 孤立 C3 瞬时失败 → 题永久 uncertain 且断点续跑不重试（影响个位数题；C1/C2 失败会 fail-loud 全崩，批量污染不可达） | 检查 uncertain 占比（本轮 3/236） |
| P2-5 | `runner.py:1444` | R2 的 dual_metric "final" 行在 revert 判定前落库，回退版本留下污染记录（存疑级） | harness-eval 读数时注意 |
| P2-6 | `breaker.py:36` | 熔断半开无单探针语义，cooldown 到期 24 并发同时放行（存疑级） | 持续故障时人工暂停 |
| P2-7 | `runner.py:2286` | `prompts/span_eval_user.md` 缺失（同款空串兜底），但 diagnose.py 实际未消费该字段 | 迁移时顺手补 |
## 反驳的 12 条（误报，不需处理）
McNemar 护栏时序、diag_fingerprint 声明式指纹、T0 计入 val 错题、sub_pattern 序列化丢失、correctness 重复行、双题目权威、resume 结构键缺池参数、evolve_single_tool TypeError、momentum guidance 消毒、缓存反序列化无防护、CLI store_true 覆盖 YAML、load_config 静默过滤未知键 —— 均经双验证员核实为不可达或有上游防御。
## 结合前一轮预检的完整训练前 checklist
1. 接线：seed 携带 pools.json 机制（或 frozen_pools_path 配置）+ 新建 adhoc-baseline seed + 训练 yaml/sh（`questions: benchmarks/Video-MME`、run_holdout_eval 显式决策）。
2. P0-1~P0-6 全部修复；P1 按成本尽量修（P1-1 至少给训练 run 加缓存 salt 或 TTL）。
3. 备份当前冻结产物（P2-1）。
4. 训练启动前预检脚本：val per-class 覆盖、微型题型排除、evolve/diagnose 模板存在性 fail-loud。
scripts/generate_questions_v2.sh
+3 -1
View File
@@ -26,10 +26,12 @@ set -euo pipefail
cd "$(dirname "$0")/.."
STORE_DIR="${STORE_DIR:-store}"
CONFIG="${CONFIG:-config/default.yaml}"
CONFIG="${CONFIG:-config/question_gen_360.yaml}"
DB_PATH="${DB_PATH:-logs/question_gen.db}"
export PYTHONUNBUFFERED=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
# shellcheck source=../.env
source .env
scripts/generate_questions_v2_补6类.sh
Executable
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# 补生 Video-MME 6 类缺失题型（180 题）
# 背景：原 360 题中 6 类 task_type 使用了错误的自创类型，
# 已归档为 archived_wrong_task_types.tar.gz。
# 本脚本只生成缺失的 6 类，生成完成后手动合并到 accepted_questions.json。
set -euo pipefail
cd "$(dirname "$0")/.."
STORE_DIR="${STORE_DIR:-store}"
CONFIG="${CONFIG:-config/question_gen_180_补.yaml}"
DB_PATH="${DB_PATH:-logs/question_gen_补6类.db}"
export PYTHONUNBUFFERED=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
# shellcheck source=../.env
source .env
[ "${MODE:-}" = "mock" ] && export LLM_MOCK=1 VLM_MOCK=1
PYTHON="$(conda run -n Video-Tree-TRM which python)"
"${PYTHON}" tools/generate_questions.py generate-v2 \
--store-dir "$STORE_DIR" \
--config "$CONFIG" \
--db-path "$DB_PATH" \
--task-types \
"Attribute Perception" \
"Counting Problem" \
"Information Synopsis" \
"OCR Problems" \
"Spatial Perception" \
"Temporal Perception" \
${SEED:+--seed "$SEED"}
scripts/train_ar30.sh
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Action Recognition 训练 — 基于 SubPattern 靶向生成的 AR30 题
#
# 三阶段:
# Phase 0: baseline infer (AR30 题 + VME benchmark 作为 test)
# Phase 1: create seed (ar30-baseline)
# Phase 2: train (3 epochs, per_category)
#
# 用法:
# CUDA_VISIBLE_DEVICES=0 bash scripts/train_ar30.sh
# MODE=mock bash scripts/train_ar30.sh # 跳过 Phase 0/1
#
# 与上次训练的区别:
# - 题目来源: generated-ar30（SubPattern 靶向）替代 generated-v2-360（OCR 污染）
# - seed 名: ar30-baseline（独立于旧的 v2ar-baseline）
# - workspace: workspaces/train-ar30
set -euo pipefail
cd "$(dirname "$0")/.."
CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
export CUDA_VISIBLE_DEVICES
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export PYTHONUNBUFFERED=1
set -a
source .env
set +a
PYTHON="$(conda run -n Video-Tree-TRM which python)"
# ── Phase 0: Baseline infer（用 AR30 新题跑基线推理）──
if [[ "${MODE:-}" != "mock" ]]; then
echo "=== Phase 0: Baseline infer (AR30 新题 30 题) ==="
"${PYTHON}" main.py \
--config config/train_ar30.yaml \
--workspace-dir workspaces/default \
--store-dir store \
--mode infer \
--concurrency 24 \
--max-steps 40 \
--skill-mode auto \
--n-samples 0 \
--questions "generated-ar30" \
--skills-version v1 \
--prompts-version v1 \
--run-id ar30_baseline \
--task-types "Action Recognition"
fi
# ── Phase 1: Create seed ──
if [[ "${MODE:-}" != "mock" && ! -d "store/seeds/ar30-baseline" ]]; then
echo "=== Phase 1: Create seed ar30-baseline ==="
"${PYTHON}" -c "
from pathlib import Path
from app.harness.store import extract_run_db, init_seed
import tempfile
tmp = Path(tempfile.mkdtemp()) / 'baseline.db'
extract_run_db(
Path('workspaces/default/harness.db'),
tmp,
'infer_ar30_baseline',
)
init_seed(
store_dir=Path('store'),
name='ar30-baseline',
skills_dir=Path('store/skills/v1'),
prompts_dir=Path('store/prompts/v1'),
baseline_db=tmp,
baseline_run_id='infer_ar30_baseline',
parent=None,
description='AR30 SubPattern 靶向题 baseline (skills/v1)',
)
tmp.unlink()
print('Seed created: store/seeds/ar30-baseline/')
"
elif [[ -d "store/seeds/ar30-baseline" ]]; then
echo "=== Phase 1: Seed ar30-baseline 已存在，跳过 ==="
fi
# ── Phase 2: Train ──
echo "=== Phase 2: Train (3 epochs, AR30) ==="
"${PYTHON}" main.py \
--config config/train_ar30.yaml \
--fresh \
--seed ar30-baseline
echo "=== 训练完成 ==="
echo "结果查看:"
echo " cat workspaces/train-ar30/analyses/final_test_eval.json"
echo " sqlite3 workspaces/train-ar30/harness.db 'SELECT * FROM dual_metric'"
store/prompts/question_gen/enumeration.md
+7
View File
@@ -17,6 +17,13 @@ Your task: Generate an **enumeration** multiple-choice question that tests count
- Avoid trivially small counts (e.g., "How many people?" when only 1 is visible).
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Prohibited Patterns
- Do NOT fabricate counts or list items absent from the provided material — no invented quantities, names, or sequences that are not explicitly confirmable from the subtitles or frames.
- Do NOT construct numerical options in ascending or descending order where the correct answer is the extreme value — shuffle the magnitudes across options.
- Do NOT write a question where multiple counting interpretations could yield different correct answers — the scope of "what to count" must be unambiguous.
- Do NOT ask questions answerable by common sense about typical quantities (e.g., "How many wheels does the car have?") — the count must require watching this specific content.
## Output
Respond with ONLY a valid JSON object. No additional text.
store/prompts/question_gen/gate_key_verify.md
+8 -5
View File
@@ -4,9 +4,11 @@ You are a quality-control judge for video understanding questions.
## Task
Given the source material from a video and a multiple-choice question with its designated correct answer, determine whether the correct answer is **supported by evidence** in the source material.
Given the source material (text descriptions AND video frames) from a video and a multiple-choice question with its designated correct answer, determine whether the correct answer is **supported by evidence** in the source material.
## Source Material
**IMPORTANT:** You can see both the text descriptions AND the actual video frames. The correct answer may be visually evident in the frames even if not explicitly mentioned in the text descriptions. Use BOTH modalities for your judgment.
## Source Material (Text)
{source_text}
@@ -24,9 +26,10 @@ Given the source material from a video and a multiple-choice question with its d
## Instructions
1. Read the source material carefully.
2. Determine if the designated correct answer can be derived or inferred from the source material.
3. If evidence supports the answer, verdict is "pass". If not, verdict is "fail".
1. Examine the video frames carefully for visual evidence.
2. Read the text descriptions for contextual evidence.
3. If evidence from EITHER the frames OR the text supports the answer, verdict is "pass".
4. Only verdict "fail" if NEITHER the frames NOR the text provide any support for the answer.
## Response Format (strict JSON)
store/prompts/question_gen/reasoning.md
+7
View File
@@ -17,6 +17,13 @@ Your task: Generate a **multi-hop reasoning** multiple-choice question that requ
- The reasoning chain should be verifiable from the provided material.
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Prohibited Patterns
- Do NOT fabricate details absent from the provided material — no invented names, numbers, dialogue lines, or events that are not explicitly present in the subtitles or visually confirmed in the frames.
- Do NOT construct options where the correct answer is the largest number, the last item in a sequence, or the most visually salient choice — these patterns let a test-taker guess without understanding the content.
- Do NOT write a question where multiple options could reasonably be considered correct — each distractor must be clearly wrong given the source material.
- Do NOT ask questions answerable by common sense or world knowledge alone (e.g., "What happens after X?" when the causal link is obvious) — the reasoning chain must depend on video-specific evidence.
## Output
Respond with ONLY a valid JSON object. No additional text.
store/prompts/question_gen/retrieval.md
+7
View File
@@ -18,6 +18,13 @@ Your task: Generate a **factual retrieval** multiple-choice question that tests
- Avoid negation in the question stem (e.g., "Which of the following is NOT...").
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Prohibited Patterns
- Do NOT fabricate details absent from the provided material — no invented names, numbers, dialogue lines, or events that are not explicitly present in the subtitles or visually confirmed in the frames.
- Do NOT construct options where the correct answer is the largest number, the last item in a sequence, or the most visually salient choice — these patterns let a test-taker guess without understanding the content.
- Do NOT write a question where multiple options could reasonably be considered correct — each distractor must be clearly wrong given the source material.
- Do NOT ask questions answerable by common sense or world knowledge alone (e.g., "What color is the sky?") — the question must require having seen this specific video.
## Output
Respond with ONLY a valid JSON object. No additional text.
store/prompts/question_gen/spatial.md
+7
View File
@@ -18,6 +18,13 @@ Your task: Generate a **spatial relationship** multiple-choice question that tes
- Spatial references must be unambiguous given the visual content.
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Prohibited Patterns
- Do NOT fabricate spatial details absent from the provided material — no invented positions, distances, or arrangements that are not visually confirmed in the frames.
- Do NOT construct options where the correct answer follows an obvious spatial pattern (e.g., always "left", always the closest) — randomize spatial references across options.
- Do NOT write a question where multiple spatial interpretations could be correct — the spatial relationship must be unambiguous from the frames.
- Do NOT ask questions answerable by common sense about typical spatial layouts (e.g., "Where is the audience relative to the stage?") — the question must require observing this specific scene.
## Output
Respond with ONLY a valid JSON object. No additional text.
store/prompts/question_gen/visual.md
+7
View File
@@ -17,6 +17,13 @@ Your task: Generate a **visual detail** multiple-choice question that requires o
- Avoid questions about things that are typically described in subtitles (dialogue content, narration).
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Prohibited Patterns
- Do NOT fabricate visual details absent from the frames — no invented colors, text overlays, logos, or object appearances that you cannot directly see in the provided images.
- Do NOT construct options where the correct answer is the most visually striking or salient choice — distractors should be equally plausible to someone who glanced briefly.
- Do NOT write a question where multiple options could reasonably match what is shown — each distractor must be clearly inconsistent with the frames.
- Do NOT ask questions answerable without the frames (e.g., typical object colors, standard uniforms) — the visual detail must be specific to these frames.
## Output
Respond with ONLY a valid JSON object. No additional text.
tests/integration/test_agent_governed_e2e.py
+2 -2
View File
@@ -49,8 +49,8 @@ def _streaming_result(content: str, thinking: str = "") -> tuple:
return (
content,
thinking,
50.0, # ttft_ms
10.0, # max_inter_token_ms
50.0, # ttft_ms
10.0, # max_inter_token_ms
{"prompt_tokens": 10, "completion_tokens": 5},
)
tests/integration/test_tree_build_e2e.py
+12 -7
View File
@@ -7,15 +7,20 @@
from __future__ import annotations
import numpy as np
import pytest
from app.tree.index import (
IndexMeta, TreeIndex, L1Node, L1Card,
L2Node, L2Card, L3Node, L3Card,
)
from app.tree.verify import verify_tree
from app.tree.subtitle import SRTEntry, assign_subtitles_voronoi
from app.tree.environment import TreeEnvironment
from app.tree.index import (
IndexMeta,
L1Card,
L1Node,
L2Card,
L2Node,
L3Card,
L3Node,
TreeIndex,
)
from app.tree.subtitle import SRTEntry, assign_subtitles_voronoi
from app.tree.verify import verify_tree
class TestTreeModuleE2E:
tests/unit/test_agent_protocols.py
+3
View File
@@ -1,4 +1,5 @@
"""core/agent/protocols.py 单元测试。"""
from __future__ import annotations
from typing import Any
@@ -18,9 +19,11 @@ class _FakeDispatcher:
def test_fake_dispatcher_satisfies_protocol() -> None:
assert isinstance(_FakeDispatcher(), ToolDispatcher)
def test_plain_object_not_dispatcher() -> None:
assert not isinstance(object(), ToolDispatcher)
def test_hookspec_can_register() -> None:
pm = pluggy.PluginManager("agent_loop")
pm.add_hookspecs(AgentLoopSpec)
tests/unit/test_agent_types.py
+10 -3
View File
@@ -1,4 +1,5 @@
"""core/agent/types.py 单元测试。"""
from __future__ import annotations
from core.agent.types import LoopResult, Step
@@ -31,12 +32,18 @@ class TestLoopResult:
def test_with_steps(self) -> None:
step = Step(
thought="t", reflect={}, plan={},
thought="t",
reflect={},
plan={},
tool_call={"tool": "t", "args": {}},
tool_output="o", raw_content="r", call_id="c",
tool_output="o",
raw_content="r",
call_id="c",
)
lr = LoopResult(
result={"answer": "42"}, steps=[step], steps_used=1,
result={"answer": "42"},
steps=[step],
steps_used=1,
token_usage={"prompt_tokens": 100, "completion_tokens": 50},
stop_reason="finished",
)
tests/unit/test_core_protocols.py
+48 -15
View File
@@ -1,11 +1,10 @@
"""core/protocols.py 单元测试 — 验证 Protocol 可 runtime_checkable。"""
from __future__ import annotations
from pathlib import Path
from typing import Any
import pytest
from core.protocols import LLMProvider, TelemetryRecorder, VLMProvider
from core.types import LLMResponse
@@ -19,9 +18,17 @@ class _FakeLLM:
parent_call_id: str | None = None,
) -> LLMResponse:
return LLMResponse(
content="ok", thinking="", model="m", provider="p",
prompt_tokens=1, completion_tokens=1, latency_ms=1,
ttft_ms=None, max_inter_token_ms=None, cache_hit=False, call_id="c",
content="ok",
thinking="",
model="m",
provider="p",
prompt_tokens=1,
completion_tokens=1,
latency_ms=1,
ttft_ms=None,
max_inter_token_ms=None,
cache_hit=False,
call_id="c",
)
@@ -35,19 +42,39 @@ class _FakeVLM:
parent_call_id: str | None = None,
) -> LLMResponse:
return LLMResponse(
content="ok", thinking="", model="m", provider="p",
prompt_tokens=1, completion_tokens=1, latency_ms=1,
ttft_ms=None, max_inter_token_ms=None, cache_hit=False, call_id="c",
content="ok",
thinking="",
model="m",
provider="p",
prompt_tokens=1,
completion_tokens=1,
latency_ms=1,
ttft_ms=None,
max_inter_token_ms=None,
cache_hit=False,
call_id="c",
)
class _FakeTelemetry:
async def record_llm_call(
self, *, call_id: str, parent_call_id: str | None, session_id: str | None,
model_name: str, provider: str, messages: str, response: str, thinking: str,
prompt_tokens: int, completion_tokens: int, latency_ms: int,
ttft_ms: float | None, max_inter_token_ms: float | None,
cache_hit: bool, error: str | None,
self,
*,
call_id: str,
parent_call_id: str | None,
session_id: str | None,
model_name: str,
provider: str,
messages: str,
response: str,
thinking: str,
prompt_tokens: int,
completion_tokens: int,
latency_ms: int,
ttft_ms: float | None,
max_inter_token_ms: float | None,
cache_hit: bool,
error: str | None,
) -> None:
pass
@@ -55,12 +82,15 @@ class _FakeTelemetry:
def test_fake_llm_satisfies_protocol() -> None:
assert isinstance(_FakeLLM(), LLMProvider)
def test_fake_vlm_satisfies_protocol() -> None:
assert isinstance(_FakeVLM(), VLMProvider)
def test_fake_telemetry_satisfies_protocol() -> None:
assert isinstance(_FakeTelemetry(), TelemetryRecorder)
def test_plain_object_does_not_satisfy() -> None:
assert not isinstance(object(), LLMProvider)
assert not isinstance(object(), VLMProvider)
@@ -80,8 +110,11 @@ class TestPoolStrategyProtocol:
class FakeStrategy:
def build(self, questions, correctness, config):
return Pools(
diagnosis=[], validation=[], test=[],
baseline_run_id="", baseline_val_accuracy=0.0,
diagnosis=[],
validation=[],
test=[],
baseline_run_id="",
baseline_val_accuracy=0.0,
)
def build_incremental(self, new_task_types, questions, correctness, config):
tests/unit/test_families.py
+9 -9
View File
@@ -14,21 +14,21 @@ from app.question_gen.families import (
get_family_for_slot,
)
# 12 种任务类型（来自 harness config）
# Video-MME 12 种任务类型
ALL_TASK_TYPES: frozenset[str] = frozenset(
[
"Action Recognition",
"Action Reasoning",
"Action Prediction",
"Action Sequence",
"Attribute Perception",
"Counting Problem",
"Information Synopsis",
"Object Recognition",
"Object Reasoning",
"Object Interaction",
"Scene Understanding",
"Event Reasoning",
"Causal Reasoning",
"Temporal Reasoning",
"OCR Problems",
"Spatial Perception",
"Spatial Reasoning",
"Temporal Perception",
"Temporal Reasoning",
]
)
@@ -163,7 +163,7 @@ class TestGetFamilyForSlot:
def test_no_legal_family_raises(self) -> None:
"""所有 family 权重为 0 时合法族为空，应抛出 ValueError。"""
rng = random.Random(0)
# Spatial Reasoning 合法族: RETRIEVAL, REASONING, VISUAL, SPATIAL
# Spatial Reasoning 合法族: SPATIAL
# 如果 ratios 中只含不合法的族名，应抛错
with pytest.raises(ValueError, match="合法"):
get_family_for_slot(
tests/unit/test_gate.py
+1 -3
View File
@@ -4,12 +4,10 @@
的核心路径与边界条件。
"""
import math
import pytest
from core.evolution.gate import compute_e_value, gate_decision, probation_verdict
from core.evolution.types import GateParams, GateVerdict
from core.evolution.types import GateParams
_PARAMS = GateParams(
e_confirm=20.0,
tests/unit/test_generate_questions.py
-1
View File
@@ -11,7 +11,6 @@ import sys
from pathlib import Path
import numpy as np
import pytest
# 确保项目根目录在 sys.path 中
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
tests/unit/test_harness_log.py
+4 -12
View File
@@ -112,9 +112,7 @@ class TestHarnessLog:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
row = conn.execute(
"SELECT status FROM _runs WHERE run_id = ?", (run_id,)
).fetchone()
row = conn.execute("SELECT status FROM _runs WHERE run_id = ?", (run_id,)).fetchone()
conn.close()
assert row["status"] == "failed"
@@ -130,9 +128,7 @@ class TestHarnessLog:
log.insert("t", {"x": 42})
conn = sqlite3.connect(db_path)
count = conn.execute(
"SELECT COUNT(*) FROM _runs WHERE run_id = ?", (run_id,)
).fetchone()[0]
count = conn.execute("SELECT COUNT(*) FROM _runs WHERE run_id = ?", (run_id,)).fetchone()[0]
conn.close()
assert count == 1, "ON CONFLICT DO UPDATE 应保证 _runs 只有一行"
@@ -151,9 +147,7 @@ class TestHarnessLog:
with HarnessLog(db_path, run_id) as log:
log.create_table("batch", {"epoch": "INTEGER", "loss": "REAL"})
log.insert_many("batch", records)
rows = log.query(
"SELECT * FROM batch WHERE run_id = ? ORDER BY epoch", (run_id,)
)
rows = log.query("SELECT * FROM batch WHERE run_id = ? ORDER BY epoch", (run_id,))
assert len(rows) == 5
assert [r["epoch"] for r in rows] == [0, 1, 2, 3, 4]
@@ -163,9 +157,7 @@ class TestHarnessLog:
with HarnessLog(db_path, run_id) as log:
log.log_event("train_start", {"epoch": 1, "lr": 0.001})
log.log_event("train_end", {"epoch": 1, "loss": 0.42})
rows = log.query(
"SELECT * FROM _events WHERE run_id = ? ORDER BY id", (run_id,)
)
rows = log.query("SELECT * FROM _events WHERE run_id = ? ORDER BY id", (run_id,))
assert len(rows) == 2
assert rows[0]["event_type"] == "train_start"
tests/unit/test_harness_store.py
+31 -48
View File
@@ -9,7 +9,6 @@ import pytest
from app.harness.store import (
_parse_version,
_write_meta,
advance_version,
extract_run_db,
init_seed,
@@ -21,7 +20,6 @@ from app.harness.store import (
read_seed,
)
# ---------------------------------------------------------------------------
# _parse_version
# ---------------------------------------------------------------------------
@@ -52,7 +50,7 @@ class TestParseVersion:
class TestListVersions:
"""list_versions 按数字排序，v10 排在 v2 后。"""
def test_list_versions_numeric_sort(self, tmp_path: "Path") -> None:
def test_list_versions_numeric_sort(self, tmp_path: Path) -> None:
"""v10 必须排在 v2 后面（非字典序）。"""
store = tmp_path / "store"
resource = store / "skills"
@@ -62,11 +60,11 @@ class TestListVersions:
result = list_versions(store, "skills")
assert result == ["v1", "v2", "v3", "v10", "v20"]
def test_list_versions_empty(self, tmp_path: "Path") -> None:
def test_list_versions_empty(self, tmp_path: Path) -> None:
store = tmp_path / "store"
assert list_versions(store, "skills") == []
def test_list_versions_ignores_non_version_dirs(self, tmp_path: "Path") -> None:
def test_list_versions_ignores_non_version_dirs(self, tmp_path: Path) -> None:
"""非 v\\d+ 格式的目录被忽略。"""
store = tmp_path / "store"
resource = store / "skills"
@@ -85,11 +83,11 @@ class TestListVersions:
class TestNextVersion:
"""next_version 返回下一个可用版本号。"""
def test_next_version_empty(self, tmp_path: "Path") -> None:
def test_next_version_empty(self, tmp_path: Path) -> None:
store = tmp_path / "store"
assert next_version(store, "skills") == "v1"
def test_next_version_after_existing(self, tmp_path: "Path") -> None:
def test_next_version_after_existing(self, tmp_path: Path) -> None:
store = tmp_path / "store"
resource = store / "skills"
resource.mkdir(parents=True)
@@ -97,7 +95,7 @@ class TestNextVersion:
(resource / "v2").mkdir()
assert next_version(store, "skills") == "v3"
def test_next_version_with_gap(self, tmp_path: "Path") -> None:
def test_next_version_with_gap(self, tmp_path: Path) -> None:
"""v1 和 v10 之间有 gap，next 应为 v11。"""
store = tmp_path / "store"
resource = store / "skills"
@@ -115,7 +113,7 @@ class TestNextVersion:
class TestAdvanceVersion:
"""advance_version copytree + _write_meta。"""
def test_advance_version(self, tmp_path: "Path") -> None:
def test_advance_version(self, tmp_path: Path) -> None:
store = tmp_path / "store"
resource = store / "skills"
resource.mkdir(parents=True)
@@ -149,7 +147,7 @@ class TestAdvanceVersion:
class TestInitStore:
"""init_store 初始化 Store 目录结构。"""
def test_init_store(self, tmp_path: "Path") -> None:
def test_init_store(self, tmp_path: Path) -> None:
videos = tmp_path / "videos_src"
videos.mkdir()
(videos / "v001").mkdir()
@@ -172,13 +170,11 @@ class TestInitStore:
assert (store / "skills" / "v1" / "search.md").read_text() == "skill"
assert (store / "prompts" / "v1" / "system.md").read_text() == "prompt"
skills_meta = json.loads(
(store / "skills" / "v1" / "meta.json").read_text()
)
skills_meta = json.loads((store / "skills" / "v1" / "meta.json").read_text())
assert skills_meta["version"] == "v1"
assert skills_meta["source"] == "manual"
def test_init_store_exists_raises(self, tmp_path: "Path") -> None:
def test_init_store_exists_raises(self, tmp_path: Path) -> None:
store = tmp_path / "store"
store.mkdir()
with pytest.raises(FileExistsError, match="Store 已存在"):
@@ -205,13 +201,9 @@ def _make_seed_fixtures(tmp_path):
baseline_db = tmp_path / "base.db"
conn = sqlite3.connect(baseline_db)
conn.execute(
"CREATE TABLE _runs (run_id TEXT PRIMARY KEY, status TEXT)"
)
conn.execute("CREATE TABLE _runs (run_id TEXT PRIMARY KEY, status TEXT)")
conn.execute("INSERT INTO _runs VALUES ('r1', 'done')")
conn.execute(
"CREATE TABLE predictions (run_id TEXT, question_id TEXT, answer TEXT)"
)
conn.execute("CREATE TABLE predictions (run_id TEXT, question_id TEXT, answer TEXT)")
conn.execute("INSERT INTO predictions VALUES ('r1', 'q1', 'A')")
conn.commit()
conn.close()
@@ -222,7 +214,7 @@ def _make_seed_fixtures(tmp_path):
class TestInitSeed:
"""init_seed 创建种子目录。"""
def test_init_seed(self, tmp_path: "Path") -> None:
def test_init_seed(self, tmp_path: Path) -> None:
store, skills_dir, prompts_dir, baseline_db = _make_seed_fixtures(tmp_path)
seed_dir = init_seed(
store,
@@ -245,25 +237,23 @@ class TestInitSeed:
assert meta["description"] == "初始种子"
assert "created_at" in meta
def test_init_seed_exists_raises(self, tmp_path: "Path") -> None:
def test_init_seed_exists_raises(self, tmp_path: Path) -> None:
store, skills_dir, prompts_dir, baseline_db = _make_seed_fixtures(tmp_path)
init_seed(store, "dup", skills_dir, prompts_dir, baseline_db, "r1", None, "first")
with pytest.raises(FileExistsError, match="种子已存在"):
init_seed(
store, "dup", skills_dir, prompts_dir, baseline_db, "r1", None, "second"
)
init_seed(store, "dup", skills_dir, prompts_dir, baseline_db, "r1", None, "second")
class TestListSeeds:
"""list_seeds 列出所有种子。"""
def test_list_seeds(self, tmp_path: "Path") -> None:
def test_list_seeds(self, tmp_path: Path) -> None:
store, skills_dir, prompts_dir, baseline_db = _make_seed_fixtures(tmp_path)
init_seed(store, "beta", skills_dir, prompts_dir, baseline_db, "r1", None, "b")
init_seed(store, "alpha", skills_dir, prompts_dir, baseline_db, "r1", None, "a")
assert list_seeds(store) == ["alpha", "beta"]
def test_list_seeds_empty(self, tmp_path: "Path") -> None:
def test_list_seeds_empty(self, tmp_path: Path) -> None:
store = tmp_path / "store"
assert list_seeds(store) == []
@@ -271,14 +261,14 @@ class TestListSeeds:
class TestReadSeed:
"""read_seed 读取 seed.json。"""
def test_read_seed(self, tmp_path: "Path") -> None:
def test_read_seed(self, tmp_path: Path) -> None:
store, skills_dir, prompts_dir, baseline_db = _make_seed_fixtures(tmp_path)
init_seed(store, "s1", skills_dir, prompts_dir, baseline_db, "r1", None, "desc")
meta = read_seed(store, "s1")
assert meta["baseline_run_id"] == "r1"
assert meta["description"] == "desc"
def test_read_seed_not_found(self, tmp_path: "Path") -> None:
def test_read_seed_not_found(self, tmp_path: Path) -> None:
store = tmp_path / "store"
store.mkdir()
with pytest.raises(FileNotFoundError, match="种子不存在"):
@@ -296,22 +286,17 @@ class TestExtractRunDb:
def _make_src_db(self, path):
"""创建带 _runs + predictions 表的源 db。"""
conn = sqlite3.connect(path)
conn.execute(
"CREATE TABLE _runs (run_id TEXT PRIMARY KEY, status TEXT)"
)
conn.execute("CREATE TABLE _runs (run_id TEXT PRIMARY KEY, status TEXT)")
conn.execute("INSERT INTO _runs VALUES ('r1', 'done')")
conn.execute("INSERT INTO _runs VALUES ('r2', 'done')")
conn.execute(
"CREATE TABLE predictions "
"(run_id TEXT, question_id TEXT, answer TEXT)"
)
conn.execute("CREATE TABLE predictions (run_id TEXT, question_id TEXT, answer TEXT)")
conn.execute("INSERT INTO predictions VALUES ('r1', 'q1', 'A')")
conn.execute("INSERT INTO predictions VALUES ('r1', 'q2', 'B')")
conn.execute("INSERT INTO predictions VALUES ('r2', 'q1', 'C')")
conn.commit()
conn.close()
def test_extract_run_db_preserves_pk(self, tmp_path: "Path") -> None:
def test_extract_run_db_preserves_pk(self, tmp_path: Path) -> None:
"""原始 CREATE 保留主键约束。"""
src = tmp_path / "src.db"
dst = tmp_path / "dst.db"
@@ -334,7 +319,7 @@ class TestExtractRunDb:
assert len(preds) == 2
conn.close()
def test_extract_run_db_missing_table(self, tmp_path: "Path") -> None:
def test_extract_run_db_missing_table(self, tmp_path: Path) -> None:
"""源 db 无目标表时报错。"""
src = tmp_path / "src.db"
dst = tmp_path / "dst.db"
@@ -345,7 +330,7 @@ class TestExtractRunDb:
with pytest.raises(RuntimeError, match="源 db 无表"):
extract_run_db(src, dst, "r1")
def test_extract_run_db_no_rows(self, tmp_path: "Path") -> None:
def test_extract_run_db_no_rows(self, tmp_path: Path) -> None:
"""目标 run_id 不存在时报错。"""
src = tmp_path / "src.db"
dst = tmp_path / "dst.db"
@@ -382,9 +367,7 @@ def _make_promote_fixtures(tmp_path):
prompts_version TEXT
)
""")
conn.execute(
"INSERT INTO _runs VALUES ('eval_001', 'v2', 'v2')"
)
conn.execute("INSERT INTO _runs VALUES ('eval_001', 'v2', 'v2')")
conn.execute("""
CREATE TABLE predictions (
run_id TEXT, question_id TEXT, answer TEXT
@@ -400,7 +383,7 @@ def _make_promote_fixtures(tmp_path):
class TestPromoteToSeed:
"""promote_to_seed 固化 workspace 版本为种子。"""
def test_promote_to_seed_success(self, tmp_path: "Path") -> None:
def test_promote_to_seed_success(self, tmp_path: Path) -> None:
ws, store = _make_promote_fixtures(tmp_path)
seed_dir = promote_to_seed(ws, store, "v2", "eval_001", "evolved-seed", "good")
assert seed_dir == store / "seeds" / "evolved-seed"
@@ -411,13 +394,13 @@ class TestPromoteToSeed:
assert meta["baseline_run_id"] == "eval_001"
assert meta["parent"] == "ws:v2"
def test_promote_to_seed_version_mismatch(self, tmp_path: "Path") -> None:
def test_promote_to_seed_version_mismatch(self, tmp_path: Path) -> None:
"""eval run 的 skills_version 与 --version 不符时报错。"""
ws, store = _make_promote_fixtures(tmp_path)
with pytest.raises(ValueError, match="版本.*不符"):
promote_to_seed(ws, store, "v3", "eval_001", "bad", "mismatch")
def test_promote_to_seed_null_version(self, tmp_path: "Path") -> None:
def test_promote_to_seed_null_version(self, tmp_path: Path) -> None:
"""eval run 的版本为 NULL 时报错。"""
ws = tmp_path / "ws2"
ws.mkdir()
@@ -438,20 +421,20 @@ class TestPromoteToSeed:
with pytest.raises(ValueError, match="NULL"):
promote_to_seed(ws, store, "v1", "eval_null", "bad", "null ver")
def test_promote_to_seed_run_not_found(self, tmp_path: "Path") -> None:
def test_promote_to_seed_run_not_found(self, tmp_path: Path) -> None:
"""eval run 不存在时报错。"""
ws, store = _make_promote_fixtures(tmp_path)
with pytest.raises(ValueError, match="eval run 不存在"):
promote_to_seed(ws, store, "v1", "nonexistent", "bad", "no run")
def test_promote_cleanup_tmp_db(self, tmp_path: "Path") -> None:
def test_promote_cleanup_tmp_db(self, tmp_path: Path) -> None:
"""finally 清理临时 db 文件。"""
ws, store = _make_promote_fixtures(tmp_path)
promote_to_seed(ws, store, "v2", "eval_001", "clean-test", "cleanup")
# 临时 db 应已清理
assert not (ws / "_promote_tmp.db").exists()
def test_promote_cleanup_tmp_db_on_error(self, tmp_path: "Path") -> None:
def test_promote_cleanup_tmp_db_on_error(self, tmp_path: Path) -> None:
"""即使 init_seed 失败（同名种子），临时 db 也应被清理。"""
ws, store = _make_promote_fixtures(tmp_path)
promote_to_seed(ws, store, "v2", "eval_001", "first", "first time")
tests/unit/test_patch.py
+3 -9
View File
@@ -259,9 +259,7 @@ class TestApplyPatch:
def test_insert_after_protected_skip(self) -> None:
content = "# Title\n\nFROZEN BLOCK\n\nrest"
edits = [{"op": "insert_after", "target": "FROZEN BLOCK", "content": "nope"}]
out, reports = apply_patch_with_report(
content, edits, protected_spans=["FROZEN BLOCK"]
)
out, reports = apply_patch_with_report(content, edits, protected_spans=["FROZEN BLOCK"])
assert out == content
assert reports[0]["status"] == "skipped_protected"
@@ -276,9 +274,7 @@ class TestApplyPatch:
def test_replace_protected_skip(self) -> None:
content = "# Title\n\nprotected\n\nrest"
edits = [{"op": "replace", "target": "protected", "content": "nope"}]
out, reports = apply_patch_with_report(
content, edits, protected_spans=["protected"]
)
out, reports = apply_patch_with_report(content, edits, protected_spans=["protected"])
assert "protected" in out
assert reports[0]["status"] == "skipped_protected"
@@ -335,9 +331,7 @@ class TestApplyPatch:
{"op": "append", "target": "", "content": "prefix text"},
{"op": "replace", "target": protected, "content": "nope"},
]
out, reports = apply_patch_with_report(
content, edits, protected_spans=[protected]
)
out, reports = apply_patch_with_report(content, edits, protected_spans=[protected])
# append 在 FREEZE 之前插入，坐标右移后 replace 仍能检测冻结区
assert reports[1]["status"] == "skipped_protected"
tests/unit/test_redis_cache.py
+1 -3
View File
@@ -61,9 +61,7 @@ async def test_cache_miss_returns_none(fake_redis: object) -> None:
@pytest.mark.asyncio
async def test_cache_roundtrip(
fake_redis: object, sample_response: LLMResponse
) -> None:
async def test_cache_roundtrip(fake_redis: object, sample_response: LLMResponse) -> None:
"""set 后 get 应返回相同内容。"""
cache = RedisResponseCache(redis=fake_redis, ttl_s=300)
await cache.set("gpt-4o", MESSAGES, sample_response)
tests/unit/test_repair_detector.py
+1 -1
View File
@@ -14,7 +14,7 @@ from app.tree.index import (
L3Node,
TreeIndex,
)
from app.tree.repair.detector import NodeIssue, detect_issues
from app.tree.repair.detector import detect_issues
if TYPE_CHECKING:
from pathlib import Path
tests/unit/test_repair_progress.py
+8
View File
@@ -1,4 +1,5 @@
"""修复管线断点续跑 progress 管理测试。"""
from __future__ import annotations
import asyncio
@@ -10,6 +11,7 @@ import pytest
def test_load_progress_missing_file(tmp_path):
"""progress 文件不存在时返回空集合。"""
from tools.repair_trees import load_progress
result = load_progress(tmp_path / "nonexistent.json")
assert result == set()
@@ -17,6 +19,7 @@ def test_load_progress_missing_file(tmp_path):
def test_load_progress_valid_file(tmp_path):
"""正常读取已有 progress 文件。"""
from tools.repair_trees import load_progress
path = tmp_path / "progress.json"
path.write_text(json.dumps({"finished_video_ids": ["vid_a", "vid_b"]}))
result = load_progress(path)
@@ -26,6 +29,7 @@ def test_load_progress_valid_file(tmp_path):
def test_load_progress_corrupted_file(tmp_path):
"""损坏的 JSON 文件返回空集合（不抛异常）。"""
from tools.repair_trees import load_progress
path = tmp_path / "progress.json"
path.write_text("{invalid json")
result = load_progress(path)
@@ -36,6 +40,7 @@ def test_load_progress_corrupted_file(tmp_path):
async def test_save_progress_atomic(tmp_path):
"""save_progress 原子写入，并发调用不丢失更新。"""
from tools.repair_trees import save_progress
path = tmp_path / "progress.json"
lock = asyncio.Lock()
await save_progress(path, lock, "vid_a")
@@ -48,6 +53,7 @@ async def test_save_progress_atomic(tmp_path):
async def test_save_progress_concurrent(tmp_path):
"""16 路并发 save_progress 不丢失更新。"""
from tools.repair_trees import save_progress
path = tmp_path / "progress.json"
lock = asyncio.Lock()
tasks = [save_progress(path, lock, f"vid_{i}") for i in range(16)]
@@ -59,6 +65,7 @@ async def test_save_progress_concurrent(tmp_path):
def test_should_skip_finished():
"""已在 finished 集合中的视频应跳过。"""
from tools.repair_trees import should_skip_video
finished = {"vid_a", "vid_b"}
assert should_skip_video("vid_a", finished, reaggregate_all=False) is True
assert should_skip_video("vid_c", finished, reaggregate_all=False) is False
@@ -67,5 +74,6 @@ def test_should_skip_finished():
def test_should_skip_reaggregate_all_forces_rerun():
"""--reaggregate-all 标志强制不跳过。"""
from tools.repair_trees import should_skip_video
finished = {"vid_a"}
assert should_skip_video("vid_a", finished, reaggregate_all=True) is False
tests/unit/test_run_store.py
+1 -1
View File
@@ -138,7 +138,7 @@ class TestQuestionGenStore:
slot_id="slot-0",
video_id="v002",
family="reasoning",
task_type="Action Sequence",
task_type="Action Reasoning",
skill_target="M2",
attempt=1,
question_text="为什么这样做？",
tests/unit/test_search_summarizer.py
+35 -54
View File
@@ -219,11 +219,7 @@ class TestCheckAnchors:
def test_no_info_statement_not_counted(self) -> None:
"""声明句"未包含…相关…信息"不计入 n_assertions。"""
anchor_map = {"s1": "行1"}
summary = (
"[相关信息]\n"
"- 该节点未包含与问题直接相关的信息\n"
"- 关键发现(s1)"
)
summary = "[相关信息]\n- 该节点未包含与问题直接相关的信息\n- 关键发现(s1)"
_, stats = check_anchors(summary, anchor_map)
assert stats["n_assertions"] == 1 # 声明句不计
assert stats["n_anchored"] == 1
@@ -271,9 +267,7 @@ class TestAssembleAnchoredOutput:
"""ids_expand 模式：保留行号 + 附加引文段。"""
anchor_map = {"s1": "第一行内容", "s2": "第二行内容"}
summary = "关键发现(s1,s2)"
result, stats = assemble_anchored_output(
summary, anchor_map, "ids_expand"
)
result, stats = assemble_anchored_output(summary, anchor_map, "ids_expand")
assert "(s1,s2)" in result
assert "[引文]" in result
assert 's1: "第一行内容"' in result
@@ -284,9 +278,7 @@ class TestAssembleAnchoredOutput:
"""expand_only 模式：剥除行号 + 附加引文段。"""
anchor_map = {"s1": "第一行内容"}
summary = "关键发现(s1)"
result, stats = assemble_anchored_output(
summary, anchor_map, "expand_only"
)
result, stats = assemble_anchored_output(summary, anchor_map, "expand_only")
assert "(s1)" not in result
assert "[引文]" in result
assert 's1: "第一行内容"' in result
@@ -297,21 +289,15 @@ class TestAssembleAnchoredOutput:
anchor_map = {f"s{i}": f"行{i}" for i in range(1, 10)}
refs = ",".join(f"s{i}" for i in range(1, 10))
summary = f"发现({refs})"
result, stats = assemble_anchored_output(
summary, anchor_map, "ids_expand"
)
result, stats = assemble_anchored_output(summary, anchor_map, "ids_expand")
assert stats["n_expanded"] == 5
def test_max_chars_cap(self) -> None:
"""总字符超过 800 时截断。"""
anchor_map = {
f"s{i}": "A" * 300 for i in range(1, 6)
}
anchor_map = {f"s{i}": "A" * 300 for i in range(1, 6)}
refs = ",".join(f"s{i}" for i in range(1, 6))
summary = f"发现({refs})"
result, stats = assemble_anchored_output(
summary, anchor_map, "ids_expand"
)
result, stats = assemble_anchored_output(summary, anchor_map, "ids_expand")
# 300 字符原文 + 前缀 ≈ 310+ 每条，800 / 310 ≈ 2 条
assert stats["n_expanded"] < 5
@@ -319,9 +305,7 @@ class TestAssembleAnchoredOutput:
"""单行超 200 字符截断并标记 n_trunc。"""
anchor_map = {"s1": "A" * 250}
summary = "发现(s1)"
result, stats = assemble_anchored_output(
summary, anchor_map, "ids_expand"
)
result, stats = assemble_anchored_output(summary, anchor_map, "ids_expand")
assert stats["n_trunc"] == 1
assert "…" in result
@@ -388,10 +372,12 @@ class TestSummarizeNode:
async def test_anchor_mode(self, prompts_dir: Path) -> None:
"""锚模式：check_anchors + assemble。"""
anchor_map = {"s1": "第一行", "s2": "第二行"}
llm = FakeLLMProvider([
"[相关信息]\n- 关键发现(s1)\n- 补充(s2)",
"核实通过",
])
llm = FakeLLMProvider(
[
"[相关信息]\n- 关键发现(s1)\n- 补充(s2)",
"核实通过",
]
)
result = await summarize_node(
llm,
"带行号的内容",
@@ -409,10 +395,12 @@ class TestSummarizeNode:
"""锚模式 stats_sink 回调接收完整统计。"""
anchor_map = {"s1": "第一行"}
collected: list[dict] = []
llm = FakeLLMProvider([
"[相关信息]\n- 关键发现(s1)",
"核实通过",
])
llm = FakeLLMProvider(
[
"[相关信息]\n- 关键发现(s1)",
"核实通过",
]
)
await summarize_node(
llm,
"内容",
@@ -471,9 +459,7 @@ class TestSummarizeChildren:
{"id": "n2", "time_range": (30.0, 60.0), "summary": "中间"},
]
llm = FakeLLMProvider(["相关性标注结果", "核实通过"])
result = await summarize_children(
llm, children_info, "问题", prompts_dir
)
result = await summarize_children(llm, children_info, "问题", prompts_dir)
assert "相关性标注结果" in result
assert "[核实] 核实通过" in result
@@ -484,25 +470,19 @@ class TestSummarizeChildren:
{"id": "n1", "time_range": (0.0, 30.0), "summary": "开头"},
]
llm = FailingLLMProvider("网络错误")
result = await summarize_children(
llm, children_info, "问题", prompts_dir
)
result = await summarize_children(llm, children_info, "问题", prompts_dir)
assert "n1" in result
assert "0-30s" in result
assert "开头" in result
@pytest.mark.asyncio()
async def test_verify_failure_returns_extract_only(
self, prompts_dir: Path
) -> None:
async def test_verify_failure_returns_extract_only(self, prompts_dir: Path) -> None:
"""核实轮失败仍返回提取结果。"""
children_info = [
{"id": "n1", "time_range": (0.0, 30.0), "summary": "开头"},
]
llm = FailOnNthLLMProvider(["标注结果"], fail_on=2)
result = await summarize_children(
llm, children_info, "问题", prompts_dir
)
result = await summarize_children(llm, children_info, "问题", prompts_dir)
assert "标注结果" in result
@@ -513,19 +493,22 @@ class TestSummarizeNodesBatch:
async def test_batch_normal(self, prompts_dir: Path) -> None:
"""并发三个节点，结果顺序与输入一致。"""
# 每个节点需要 2 轮 LLM 调用（提取 + 核实）
llm = FakeLLMProvider([
"摘要A", "核实A",
"摘要B", "核实B",
"摘要C", "核实C",
])
llm = FakeLLMProvider(
[
"摘要A",
"核实A",
"摘要B",
"核实B",
"摘要C",
"核实C",
]
)
items = [
("n1", "内容1", "extra1"),
("n2", "内容2", "extra2"),
("n3", "内容3", "extra3"),
]
results = await summarize_nodes_batch(
llm, items, "问题", prompts_dir
)
results = await summarize_nodes_batch(llm, items, "问题", prompts_dir)
assert len(results) == 3
assert results[0][0] == "n1"
assert results[1][0] == "n2"
@@ -538,7 +521,5 @@ class TestSummarizeNodesBatch:
async def test_batch_empty(self, prompts_dir: Path) -> None:
"""空列表返回空结果。"""
llm = FakeLLMProvider([])
results = await summarize_nodes_batch(
llm, [], "问题", prompts_dir
)
results = await summarize_nodes_batch(llm, [], "问题", prompts_dir)
assert results == []
tests/unit/test_search_vision.py
+11 -33
View File
@@ -126,9 +126,7 @@ def prompts_dir(tmp_path: Path) -> Path:
class TestObserveFrameNormal:
"""两轮正常执行（verify=True）。"""
def test_two_round_normal(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_two_round_normal(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["raw evidence", "verified ok"])
@@ -158,9 +156,7 @@ class TestObserveFrameNormal:
class TestObserveFrameExtractOnly:
"""verify=False 仅执行提取轮。"""
def test_extract_only(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_extract_only(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["only extract"])
@@ -183,9 +179,7 @@ class TestObserveFrameExtractOnly:
class TestObserveFrameOCRInjection:
"""OCR 注入：文本非空时并置于问题前。"""
def test_ocr_injected(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_ocr_injected(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["evidence with ocr"])
@@ -216,9 +210,7 @@ class TestObserveFrameOCRInjection:
class TestObserveFrameOCRFailDegrades:
"""OCR 转录抛出异常时降级：不注入 OCR、ocr_failed=1。"""
def test_ocr_failure_degrades(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_ocr_failure_degrades(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["evidence no ocr"])
@@ -245,9 +237,7 @@ class TestObserveFrameOCRFailDegrades:
class TestObserveFrameOCRNone:
"""ocr=None 时不执行转录。"""
def test_ocr_none(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_ocr_none(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["no ocr"])
@@ -274,9 +264,7 @@ class TestObserveFrameOCRNone:
class TestObserveFrameVLMExtractFailure:
"""VLM 提取轮失败 → 返回 [VL错误]。"""
def test_vlm_extract_failure(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_vlm_extract_failure(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(raises=[RuntimeError("VLM timeout")])
@@ -302,9 +290,7 @@ class TestObserveFrameVLMExtractFailure:
class TestObserveFrameVLMVerifyFailureDegrades:
"""VLM 验证轮失败 → 降级：保留提取结果 + [验证] 跳过。"""
def test_vlm_verify_failure_degrades(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_vlm_verify_failure_degrades(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(
@@ -359,9 +345,7 @@ class TestObserveFrameFileMissing:
class TestObserveFrameStatsKeys:
"""stats 包含全部五个预期键。"""
def test_stats_keys_complete(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_stats_keys_complete(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["evidence"])
@@ -386,9 +370,7 @@ class TestObserveFrameStatsKeys:
class TestObserveFrameDiscrepancyAndAbstain:
"""VLM 返回含 '分歧' 或 '[证据不存在]' 时对应 stats 标记。"""
def test_discrepancy_flag(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_discrepancy_flag(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["发现分歧：OCR 与画面不一致"])
@@ -408,9 +390,7 @@ class TestObserveFrameDiscrepancyAndAbstain:
assert collected[0]["discrepancy"] == 1
def test_abstain_flag(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_abstain_flag(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["[证据不存在] 无法判断"])
@@ -434,9 +414,7 @@ class TestObserveFrameDiscrepancyAndAbstain:
class TestObserveFrameTelemetryPassthrough:
"""session_id 和 parent_call_id 透传到 VLM 调用。"""
def test_telemetry_passthrough(
self, frame_files: list[Path], prompts_dir: Path
) -> None:
def test_telemetry_passthrough(self, frame_files: list[Path], prompts_dir: Path) -> None:
from app.search.vision import observe_frame
vlm = FakeVLMProvider(responses=["evidence"])
tests/unit/test_smoke.py
+3
View File
@@ -4,16 +4,19 @@
def test_core_importable():
"""core 包可导入。"""
import core
assert core is not None
def test_app_importable():
"""app 包可导入。"""
import app
assert app is not None
def test_adapters_importable():
"""adapters 包可导入。"""
import adapters
assert adapters is not None
tests/unit/test_telemetry.py
+10 -3
View File
@@ -1,4 +1,5 @@
"""adapters/telemetry.py 单元测试 — SQLiteTelemetryRecorder。"""
from __future__ import annotations
import sqlite3
@@ -9,11 +10,11 @@ import pytest
from adapters.telemetry import SQLiteTelemetryRecorder
from core.protocols import TelemetryRecorder
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture()
def db_path(tmp_path):
"""返回临时数据库路径。"""
@@ -51,6 +52,7 @@ def _make_call_kwargs(*, cache_hit: bool = False, error: str | None = None):
# Tests
# ---------------------------------------------------------------------------
def test_satisfies_protocol(recorder):
"""SQLiteTelemetryRecorder 满足 TelemetryRecorder Protocol。"""
assert isinstance(recorder, TelemetryRecorder)
@@ -86,7 +88,9 @@ async def test_record_with_error(recorder, db_path):
conn = sqlite3.connect(str(db_path))
conn.row_factory = sqlite3.Row
row = conn.execute("SELECT error FROM llm_calls WHERE call_id = ?", (kwargs["call_id"],)).fetchone()
row = conn.execute(
"SELECT error FROM llm_calls WHERE call_id = ?", (kwargs["call_id"],)
).fetchone()
conn.close()
assert row["error"] == "RateLimitError: 429"
@@ -100,7 +104,9 @@ async def test_record_cache_hit(recorder, db_path):
conn = sqlite3.connect(str(db_path))
conn.row_factory = sqlite3.Row
row = conn.execute("SELECT cache_hit FROM llm_calls WHERE call_id = ?", (kwargs["call_id"],)).fetchone()
row = conn.execute(
"SELECT cache_hit FROM llm_calls WHERE call_id = ?", (kwargs["call_id"],)
).fetchone()
conn.close()
assert row["cache_hit"] == 1
@@ -130,6 +136,7 @@ async def test_db_error_does_not_propagate(tmp_path):
async def test_concurrent_writes_no_lock_error(recorder, db_path):
"""16 路并发 record_llm_call 应全部成功，无 database is locked 错误。"""
import asyncio
tasks = []
for _ in range(16):
kwargs = _make_call_kwargs()