iomgaa/Video-Tree-TRM5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
259 Commits 2 Branches 0 Tags
b1f15ddb3af42a66d20df0c6302743024930b1e2
Commit Graph

259 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
iomgaa 206c553143 feat(question_gen): add v2 pipeline with retry loop and heavy check 
- PipelineConfig: YAML-driven configuration with family_ratios, retry,
  concurrency, dedup threshold, and heavy sampling rate
- _assign_slots: deterministic round-robin slot assignment across videos
  with per-family weighted random selection
- _process_one_slot: full retry loop (generate → postprocess → gates →
  dedup) with reject-reason feedback to VLM on retry
- _heavy_check_one: blind LLM agent trial-answer for difficulty_steps
- run_pipeline_v2: orchestration with semaphore-bounded concurrency,
  progress/resume support, and store integration
- is_duplicate: cosine similarity dedup against embedding pool

Tests: 11 integration tests covering slot assignment, retry behavior,
max-retries exhaustion, full pipeline flow, progress resume, heavy
sampling, and store record completeness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:55:15 -04:00
iomgaa 6d6eb8e3a3 refactor(question_gen): extract _validate_parsed_fields to reduce CC 
Split field validation logic out of _parse_v2_response into a dedicated
_validate_parsed_fields helper. This brings _parse_v2_response from CC=11
(grade C) down to CC=3 (grade A). The extracted validator is CC=9 (grade B).
No grade-C functions remain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:43:35 -04:00
iomgaa cf51d2de9d feat(question_gen): add v2 generator with per-family prompt templates 
Implement generator_v2.py with:
- CandidateQuestion dataclass (canonical location)
- _load_prompt_template: loads per-family .md from store/prompts/
- _build_v2_prompt: constructs system+user messages with material context
- _parse_v2_response: JSON extraction, json_repair, field validation
- generate_one_v2: async VLM call orchestration with reject_reason support

Add 5 family-specific prompt templates:
- retrieval.md: factual recall from visible content
- reasoning.md: multi-hop inference across segments
- enumeration.md: counting/listing entities and actions
- visual.md: visual details requiring frame observation
- spatial.md: spatial relationships between objects/people

Tests: 11 unit tests covering prompt build, parse, and e2e generation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:41:36 -04:00
iomgaa 9053233f99 fix(question_gen): gates reject_reason returns raw reason; short-circuit skips LLM 
- GateReport.reject_reason now returns gate.reason directly (no [name] prefix)
- verbatim short-circuit sets other 3 gates to SKIP without calling LLM
- test_high_verbatim_shortcircuits asserts zero LLM calls and SKIP verdicts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:33:57 -04:00
iomgaa 9f739e831d refactor(question_gen): reduce cyclomatic complexity in sampler_v2 
Extract shared _resolve_subtree helper to eliminate repeated tri-level
node resolution. Break _validate_sampling_constraints into focused
single-purpose helpers:
- _count_l3_descendants
- _has_frames
- _count_subtitles
- _resolve_subtree / _find_l3_parent

Extract _subtitles_from_l2_list and _frames_from_l2_list to simplify
collection functions.

Complexity improvements:
- _validate_sampling_constraints: D(23) -> B(8)
- _collect_subtitle_sentences: C(16) -> A(3)
- _collect_frame_paths: C(13) -> A(2)

All functions now grade B or better per radon cc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:33:48 -04:00
iomgaa f74711cd11 feat(question_gen): add v2 material sampler with family constraints 
Implement sample_material_v2 module that samples tree nodes with
QuestionFamilySpec-aware constraint validation, providing richer
MaterialContext output (subtitles, cross-L2 context, frame paths).

Key components:
- AnchorContext/MaterialContext frozen dataclasses
- _validate_sampling_constraints: multi-level constraint checking
- _collect_subtitle_sentences: subtree subtitle extraction
- _collect_cross_l2_context: peer L2 event descriptions
- sample_material_v2: main entry with retry-on-constraint-violation

Tests: 11 unit tests covering normal sampling, used-node exclusion,
constraint violation retries, cross-L2 population, and subtitle
collection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:30:56 -04:00
iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check 
Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:30:13 -04:00
iomgaa 9627ac9cf9 fix(question_gen): raise ValueError on UPDATE of missing rows 
record_run_end, update_gates, and update_difficulty now check
cursor.rowcount after UPDATE+commit and raise ValueError if 0 rows
were affected. Prevents silent telemetry loss.

Adds three negative-path tests:
- test_record_run_end_missing_run_raises
- test_update_gates_missing_item_raises
- test_update_difficulty_missing_item_raises

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:23:48 -04:00
iomgaa 7abe92eb1c fix(question_gen): check_verbatim covers question_text + add missing blacklist patterns 
- check_verbatim now computes n-gram overlap for BOTH question_text and
  correct_option vs source texts, returning max(question_ratio, option_ratio).
  Extracted _ngram_overlap_ratio helper for reuse.

- Added 4 missing blacklist patterns: 'this segment', 'this frame',
  'the current frame', 'frame summary'.

- Added 5 new test cases covering the above changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:23:37 -04:00
iomgaa 9525726133 fix(question_gen): rename finished_at to ended_at per schema spec 
Aligns DDL column name with research-wiki/schemas/question-gen-runs.md
which specifies 'ended_at' (not 'finished_at').

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:18:52 -04:00
iomgaa 74686dde68 refactor(question_gen): reduce check_forbidden_material complexity to A(3) 
Extract _match_any helper and declarative _FORBIDDEN_MATERIAL_RULES table
to replace repetitive per-category for-loops. Reduces cyclomatic complexity
from C(11) to A(3).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:16:34 -04:00
iomgaa 6a5424a618 feat(question_gen): add SQLite run store for generation telemetry 
Implements QuestionGenStore with:
- Idempotent schema initialization (question_gen_runs + question_gen_items)
- Run lifecycle: record_run_start / record_run_end / get_run_stats
- Per-item recording: record_item / update_gates / update_difficulty
- GateReportLike Protocol for duck-type gate report compatibility
- WAL mode + foreign keys + check_same_thread=False

DDL aligns with research-wiki/schemas/question-gen-{runs,items}.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:15:53 -04:00
iomgaa c83d771923 feat(question_gen): add deterministic postprocess layer 
Add app/question_gen/postprocess.py with zero-LLM deterministic
post-processing for generated questions:

- shuffle_options: deterministic option permutation with answer remapping
- check_referent_blacklist: detect self-referential language (this clip, etc.)
- check_verbatim: word-level n-gram overlap ratio measurement
- has_time_anchor: timestamp and temporal phrase detection
- check_forbidden_material: T1/T7 source material validation
- run_postprocess: orchestration returning PostprocessResult

Tests: 31 unit tests covering all functions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:14:43 -04:00
iomgaa 75e6d8c550 feat(question_gen): add 5 question family specs with sampling constraints 
Define QuestionFamilySpec, LeakTestProfile, SamplingConstraint dataclasses
and instantiate 5 families (RETRIEVAL/REASONING/ENUMERATION/VISUAL/SPATIAL)
targeting failure mechanisms M1-M5. Implement get_family_for_slot with
legal-type filtering + weighted random selection.

13 unit tests cover: full task-type coverage, skill_target uniqueness,
deterministic seeding, invalid input errors, and chi-square distribution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:14:22 -04:00
iomgaa 811ffa648b feat(types): extend GeneratedQuestion with skill_target & difficulty_steps 
- Add skill_target (str | None) and difficulty_steps (int | None) fields
  to GeneratedQuestion dataclass with field(default=None)
- Update loader.py to pass new fields from JSON (backward-compatible)
- Update pools.py _q_to_dict/_dict_to_q for serialization compat
- Add question_gen_v2 config section to default.yaml
- Add comprehensive test coverage (7 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-11 23:05:38 -04:00
iomgaa 043d4aa46f docs(plans): add Spec-3 question-gen-v2 implementation plan 
9 tasks: type extension → postprocess → families → sampler_v2 →
generator_v2 → gates → run_store → pipeline_v2 → CLI integration.
Includes structured-logging schemas/metrics and Codex review revisions.
 2026-07-11 22:57:47 -04:00
iomgaa 8c9adfd3fa docs(wiki): record Spec-2 harness-eval finding (N/A — pure tooling addition) 2026-07-11 12:13:22 -04:00
iomgaa a51b3c4238 fix(tools): validate concurrency params to prevent zero-semaphore deadlock 
Codex 最终审查发现 video_concurrency=0 或 TREE_BUILD_API_CONCURRENCY=0
会创建永不放行的 Semaphore 导致整批挂死。追加 >= 1 校验，fail-fast 报错。
 2026-07-11 12:11:48 -04:00
iomgaa 0290005129 fix(scripts): align build_trees.sh with project shell conventions 
Codex 审查发现两处偏离既有脚本惯例：
1. conda activate 改为 PYTHON=$(conda run ...) + ${PYTHON} 模式
2. 追加 source .env 确保 TREE_BUILD_API_CONCURRENCY 等工程配置可用
 2026-07-11 12:07:14 -04:00
iomgaa 928ec8c37b feat(scripts): add batch tree build entry with smoke-test guide 2026-07-11 12:04:34 -04:00
iomgaa c5bf7e1a61 fix(tools): widen error isolation to cover full single-video pipeline 
Codex 功能质量审查发现 _build_one 的 try/except 仅包裹 build_async，
构造器、SRT 解析、save_json、save_progress 抛异常会逃逸到 gather 中止整批。
将 try 边界扩大到覆盖完整单视频流程（构建→建树→保存→进度记录）。
 2026-07-11 12:02:21 -04:00
iomgaa 56fe138a46 feat(tools): batch tree build orchestration with shared API semaphore 2026-07-11 11:56:36 -04:00
iomgaa 978ddef91b feat(tools): add build_trees skeleton with discovery helpers 2026-07-11 11:41:22 -04:00
iomgaa e9073bfdc2 feat(tree): expose build_async and accept injected API semaphore 
Core algorithms #1/#2/#3 unchanged: only entry wrapping and semaphore
source switch (injected vs self-built); build logic untouched.
- rename _build_async to public build_async (body unchanged)
- __init__ accepts keyword-only api_semaphore for cross-video sharing
- default path (no injection) is verbatim-equivalent to previous code
 2026-07-11 11:14:45 -04:00
iomgaa 25f5537974 docs(plans): add Spec-2 batch tree build implementation plan 2026-07-11 11:11:02 -04:00
iomgaa 1d2385291a docs(wiki): record Spec-1 harness-eval finding (all metrics pass) 2026-07-11 09:33:53 -04:00
iomgaa 40b156cf69 docs(wiki): record final Codex PASS for recipe redesign round 2026-07-11 09:08:27 -04:00
iomgaa ebbd112d50 fix(tree): move Iterable import into type-checking block 2026-07-11 09:05:04 -04:00
iomgaa a046862ca1 docs(wiki): record build-tree recipe redesign and diagnosis stage tags 2026-07-11 09:04:57 -04:00
iomgaa f0284199b8 refactor(tree): narrow _clean_join parameter type to Iterable 2026-07-11 09:04:25 -04:00
iomgaa 1c21e215e2 test(tree): cover L1 entity fields and block ordering in view_node 2026-07-11 09:02:03 -04:00
iomgaa a2c6a97621 refactor(tree): extract _clean_join to keep entity-fields complexity at B 
质量门要求新增代码复杂度不超过 B 级：将 node_entity_fields 中重复的
"清洗-去重-拼接"逻辑抽为模块级纯函数 _clean_join，主体收敛为两次
调用。radon：node_entity_fields C(13) → B(6)，_clean_join A(5)。

算法 #11（树环境语义搜索）数据访问层，行为不变（既有 4 测试未改全绿）。
 2026-07-11 08:56:17 -04:00
iomgaa 4e0e05210d feat(search): append raw entity fields after view_node summary 
view_node 按题两轮摘要（summarize_node）会吞掉 entities/visible_text
字段信号，Agent 站在证据节点上仍漏读实体（benchmark 错题 M1，案例
786-2、872-3、750-1）。dispatcher 侧在摘要后确定性追加 [实体]/[画面
文字] 原文区块，LLM 无法吞掉。

- TreeEnvironment.node_entity_fields：按层级取 card 实体字段原文，
  去空白、去重、分号拼接；空字段省键；未知节点抛 KeyError
- _handle_view_node Phase 2.5：摘要后、子节点概览前追加实体区块
- 附带 ruff format 修正 test_tree_environment.py 两处既有格式

算法 #11（树环境语义搜索）数据访问层扩展，不改搜索算法本身。
 2026-07-11 08:53:56 -04:00
iomgaa 291a8108e1 test(agent): tighten type annotations in step-retry tests 2026-07-11 08:48:04 -04:00
iomgaa badfcce4cb fix(agent): validate non-empty retry delays; pin exhaustion semantics 
核心算法 #10（Agent Loop）：Codex 质量审查跟进，仅加固防御与
可观测，不改变循环语义。
- __init__ 校验 step_retry_delays 非空，空序列直接抛 ValueError
  （P5 fail-fast，避免首次可重试异常时 IndexError 掩盖原始 LLM
  异常、违背方法契约）
- 补耗尽语义窄测试（耗尽后原样抛出最后一次原始异常）与空序列
  构造校验测试
- run() 最终失败日志补异常类型名，便于回溯归因
 2026-07-11 08:44:47 -04:00
iomgaa e3184c11f9 feat(agent): step-level retry for transient LLM errors (20s/40s backoff) 
核心算法 #10（Agent Loop）：仅加固异常路径的韧性兜底，不改变
解析协议、hook 时序与步数语义。benchmark 错题 796-3 显示一次
SSL BAD_RECORD_MAC 穿透 GovernedLLMClient 重试栈后废掉 13 步
已积累上下文；本次在 run() Phase 1 增加步级重试（默认 2 次，
20s/40s 退避），可重试异常限定 (TimeoutError, OSError)，非可
重试异常照旧 fail-fast 整题终止，行为与现状一致（P5 显式异常）。
为满足 radon C 级复杂度约束，重试循环抽取为私有方法
_call_llm_with_step_retry，行为不变。
 2026-07-11 08:37:35 -04:00
iomgaa cf529f2c8f test(agent): add docstring to empty-content boundary test 2026-07-11 08:33:36 -04:00
iomgaa 439dc29b3b fix(agent): reject argless action in normalization; add boundary test 
核心算法 #10（Agent Loop）：修复 Codex 质量审查 Critical——
_normalize_action 仅在除 tool 外至少存在一个平铺参数键时才收拢，
{"tool": "x"} 无参结构不再被静默升级为空 args 合法结构，照旧
返回 None 走 retry 追问路径。补边界测试 + 测试辅助方法类型注解
与中文 docstring。
 2026-07-11 08:28:59 -04:00
iomgaa 6034d4172d test(question_gen): update __all__ assertion to include synthesizer exports 2026-07-11 08:21:06 -04:00
iomgaa a31b1fbf37 test(question_gen): drop tests for removed calibrate pairing validator 2026-07-11 08:19:45 -04:00
iomgaa 8d84d5e236 fix(agent): normalize fenced and flat-args LLM outputs in parser 
核心算法 #10（Agent Loop）：仅加固 _parse_response 解析路径——
剥除 ```json 围栏 + 收拢 action 平铺参数（deepseek 稳定输出变体，
案例 637-3/615-3 三连拒 0 步阵亡）。Thinking+JSON 协议、json_repair
兜底链、hook 时序与步数语义均未改动。
 2026-07-11 08:16:28 -04:00
iomgaa 39c6352781 docs(plans): add Spec-1 implementation plan with Codex review fixes 2026-07-11 08:12:07 -04:00
iomgaa ef402c46a2 docs(designs): apply Codex review fixes to Spec-1/2/3 2026-07-11 07:54:48 -04:00
iomgaa a5666b4f16 docs(wiki): record tree-figure style unification and review fixes 2026-07-11 07:52:22 -04:00
iomgaa a79c2ec753 docs(wiki): add three-round analysis findings and Spec-1/2/3 designs 2026-07-11 07:45:51 -04:00
iomgaa 658e62054e feat(scripts): add generated-questions infer and calibrate entry 2026-07-11 07:45:42 -04:00
iomgaa da70eb6e23 refactor(question_gen): slim calibrate to compare two existing runs 2026-07-11 07:45:42 -04:00
iomgaa 307c64c388 fix(harness): infer mode honors CLI --questions over manifest 2026-07-11 07:45:33 -04:00
iomgaa 735471df3c fix(question_gen): Information Synopsis 去重阈值放宽至 0.90 
该题型天然高相似（benchmark 均值 0.66，中位数 0.67），
通用阈值 0.75/0.85 导致几乎所有新题被误判重复。
按题型覆盖为 0.90，只拦截真正的复制品。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-10 08:55:47 -04:00
iomgaa 25c8d5ec42 fix(question_gen): 每次重试换视频避免同视频反复去重失败 
Information Synopsis 163 道 benchmark 题，同一视频反复出题
必然相似。改为每次重试随机选不同视频。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-10 08:15:03 -04:00