iomgaa/Video-Tree-TRM5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
241 Commits 2 Branches 0 Tags
453cf62088752e2905090abd8f1dd0b1d9e81ac3
Commit Graph

21 Commits

Author SHA1 Message Date
iomgaa 36c712defa feat(pipeline): replace QuestionFamilySpec with TaskTypeStrategy 
- SlotAssignment: remove family field, strategy looked up at process time
- PipelineConfig: remove family_ratios field
- _assign_slots: remove family_ratios and rng params (pure deterministic)
- _process_one_slot: use get_strategy() for sampling, generation, gates
- Add sub_pattern support (level/constraint override, instruction injection)
- Add strategy.extra_gates() check after standard gates
- load_pipeline_config: stop reading family_ratios from YAML
- Update tools/generate_questions.py seed override and dry-run log
- Update all integration tests to match new API

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 06:05:23 -04:00
iomgaa eecb86e27a feat(question_gen): add generate-v2 CLI subcommand and experiment script 
- Add generate-v2 subparser with --config, --store-dir, --db-path,
  --seed, and --dry-run arguments to tools/generate_questions.py
- Implement _run_generate_v2 async handler: config loading, video
  discovery, DI client construction, TreeIndex loading, pipeline
  invocation, and result persistence
- Add scripts/generate_questions_v2.sh following build_trees.sh
  conventions (source .env, conda run python path, MODE=mock support)
- Update app/question_gen/__init__.py to export full v2 public API:
  run_pipeline_v2, PipelineConfig, PipelineResult, QuestionFamilySpec,
  ALL_FAMILIES, CandidateQuestion, generate_one_v2, GateReport, run_gates
- Add QuestionGenStore.load_progress() for pipeline resumption
- Add integration tests for CLI help and dry-run behavior
- Update test_question_gen_api to match expanded __all__

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-12 00:04:42 -04:00
iomgaa a51b3c4238 fix(tools): validate concurrency params to prevent zero-semaphore deadlock 
Codex 最终审查发现 video_concurrency=0 或 TREE_BUILD_API_CONCURRENCY=0
会创建永不放行的 Semaphore 导致整批挂死。追加 >= 1 校验，fail-fast 报错。
 2026-07-11 12:11:48 -04:00
iomgaa c5bf7e1a61 fix(tools): widen error isolation to cover full single-video pipeline 
Codex 功能质量审查发现 _build_one 的 try/except 仅包裹 build_async，
构造器、SRT 解析、save_json、save_progress 抛异常会逃逸到 gather 中止整批。
将 try 边界扩大到覆盖完整单视频流程（构建→建树→保存→进度记录）。
 2026-07-11 12:02:21 -04:00
iomgaa 56fe138a46 feat(tools): batch tree build orchestration with shared API semaphore 2026-07-11 11:56:36 -04:00
iomgaa 978ddef91b feat(tools): add build_trees skeleton with discovery helpers 2026-07-11 11:41:22 -04:00
iomgaa da70eb6e23 refactor(question_gen): slim calibrate to compare two existing runs 2026-07-11 07:45:42 -04:00
iomgaa 735471df3c fix(question_gen): Information Synopsis 去重阈值放宽至 0.90 
该题型天然高相似（benchmark 均值 0.66，中位数 0.67），
通用阈值 0.75/0.85 导致几乎所有新题被误判重复。
按题型覆盖为 0.90，只拦截真正的复制品。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-10 08:55:47 -04:00
iomgaa 25c8d5ec42 fix(question_gen): 每次重试换视频避免同视频反复去重失败 
Information Synopsis 163 道 benchmark 题，同一视频反复出题
必然相似。改为每次重试随机选不同视频。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-10 08:15:03 -04:00
iomgaa 8d11513e54 fix: 禁止日志缓存，确保所有日志立刻输出 
- scripts/*.sh: PYTHONUNBUFFERED=1
- tools/generate_questions.py: loguru file sink enqueue=False
- main.py: loguru file sink enqueue=False

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 23:15:02 -04:00
iomgaa c191be16e2 fix(question_gen): L3 frame_path 拼接视频目录前缀 
tree.json 中 frame_path 是相对路径（frames/xxx.jpg），
generate_questions 需要拼上 store/videos/{video_id}/ 前缀供 VLM 读取。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 23:10:18 -04:00
iomgaa d3be9b1322 refactor(tree): subtitle 迁入 L3Card/L2Card + 建树管线修正 
- L3Card/L2Card 新增 subtitle: str 字段（L1Card 不加）
- L3Node 移除 subtitle 字段（数据迁入 Card）
- assign_subtitles_voronoi 改写 Card.subtitle + L2 聚合
- _collect_card_strings 增加 skip_fields 排除 subtitle
- _node_full_text/_node_anchored_text 保持 字幕:/[sN] 语义
- get_subtitle 读 Card.subtitle（L2/L3）
- verify.py/synthesizer.py: l3.subtitle → l3.card.subtitle
- 迁移脚本 tools/migrate_subtitle_to_card.py（幂等，300 棵树已迁移）
- 9→6 个测试文件适配（3 个无需改动）

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 11:57:41 -04:00
iomgaa 45403b23b4 feat(repair): regenerator + supplement 防御修复 + 迁移脚本 
- 新增 app/tree/repair/regenerator.py（VLM 重生成 + 级联修复）
- supplement.py: deduplicate_field str() 防御 + inject_value strip
- patch.py: ruff format 格式化
- repair_trees.sh: conda source 激活修复
- 新增 migrate_from_trm4.sh 迁移工具
- enhance/__init__.py → repair/__init__.py 重命名

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 07:53:59 -04:00
iomgaa f57ee45dc0 fix: Codex 全量审查修正 
Critical:
- C1: assemble_mode 'plain' → 'ids'（合法枚举值）
- C2: question_id 加入 task_type slug 避免跨题型冲突

Important/Minor:
- generate_one 移除未用的 embed_fn/similarity_threshold 参数
- config.py 注释 11→12 同步
- 测试 question_id 断言更新

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 07:40:27 -04:00
iomgaa 93c9be8bfa feat(tools): generate_questions.py calibrate 子命令 
- Fisher exact test + effect size 组合判定（PASS/WARN/FAIL）
- 按 video_id 分组推理，避免跨视频树错用
- baseline 支持从 DB 读取或自动跑推理
- 对比表输出 + 退出码控制

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 05:47:52 -04:00
iomgaa 11f3c90200 feat(tools): generate_questions.py generate 子命令 
- VLM 出题 + embedding 去重 + 断点续跑 + 并发控制
- 单线程汇总点保证去重原子性
- 18 个单元测试覆盖 progress/exemplar/pool rebuild/JSON append

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 05:42:50 -04:00
iomgaa d6bcf41336 fix(repair): Codex 审查修正 — progress 完成判据加严 + load_progress 防御增强 
1. 保存 progress 前重新 detect_issues，只有 empty_field 清零才记 finished
   （修复 L2/L1 空字段被检测但未修复仍写 finished 的问题）
2. load_progress 增加 AttributeError 捕获（防 JSON 非 dict 形态崩溃）
 2026-07-09 00:30:23 -04:00
iomgaa 4612123ec4 style: ruff format telemetry + repair_trees, remove unused SRTEntry import 2026-07-09 00:25:36 -04:00
iomgaa 847def4a03 feat(repair): asyncio.Semaphore 并发 + 断点续跑 + CLI 参数 
--concurrency 默认 16，--reaggregate-all 强制全量重聚合。
Semaphore 限视频并发数，视频内四步串行。progress 文件
asyncio.Lock + os.replace 原子写入。熔断阈值 max(.env, concurrency*2)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 00:23:40 -04:00
iomgaa afe80a8b32 feat(repair): 断点续跑 progress 文件管理 
load_progress / save_progress(asyncio.Lock + os.replace 原子写入)
/ should_skip_video。支持并发安全的读改写和 --reaggregate-all 兜底。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-09 00:23:04 -04:00
iomgaa faf6f1bd63 fix(tools): handle None card in flat tree conversion 2026-07-07 03:13:38 -04:00