iomgaa/Video-Tree-TRM5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
506 Commits 2 Branches 0 Tags
main
Commit Graph

258 Commits

Author SHA1 Message Date
iomgaa d6a3107e4e feat(question_gen): loader 按 unit 分层采样 + load_benchmark 读回 pair 字段 
stratified_sample 先 build_units 聚合，以 QuestionUnit 为采样原子做
分层/去重/补足/rng.sample，返回前 flatten_units 展开为逐题列表；
size/correct_ratio/min_per_class 均按 unit 计数，单元正确性走成员 AND，
孪生对两题永不被劈开。纯 single 输入下 build_units 1:1 折叠、顺序不变，
rng 消耗与旧逐题实现字节级一致（新增回归测试守护）。

_backfill_per_class candidates 改按 unit 枚举去重；build_units/flatten_units
函数内延迟导入以规避 question_gen<->harness 循环依赖（沿用 adversarial_filter）。

load_benchmark 反序列化补 pair_id/question_role/flip_axis/unit_id 四字段，
用 .get 兼容旧 JSON（缺失退化为 single，unit_id 由 __post_init__ 回填）。

pools._sample_excluding 随之改为透传 flatten_units(candidates) 给已单元化的
stratified_sample（不再用 lone pair-original 代表），行为对 single-only 保持等价。
 2026-07-15 06:27:41 -04:00
iomgaa ddb9a44f75 feat(pools): 三池切分以 unit 为原子，孪生对同池不被拆散 
build_pools/_sample_excluding 与 PerCategoryPoolStrategy._split_one_category/
build_incremental 两条切分路径均改为以 QuestionUnit 为采样原子：progressive
exclusion 互斥集合与 train/val 分层划分都按 unit_id 计数（pair 计 1 个 unit），
命中单元整体展开，AR 孪生对两题永不落入不同池/split。

复用 app.harness.question_units 的 build_units/flatten_units，不重写分组逻辑。
single-only 输入下 unit 与 question 一一对应、rng 消耗量不变，采样与划分结果
与逐题口径完全一致；抽出 _unit_correct/_assert_correctness_complete 两个 helper
将 _split_one_category 复杂度压回基线以下。

新增 tests/unit/test_pools_pair_atomic.py 覆盖两条路径的 pair 原子性回归。
 2026-07-15 06:09:14 -04:00
iomgaa bef46636fe feat: add question_units helper as pair contract entry point 
build_units/flatten_units/validate_units/unit_correctness——pair 契约唯一入口。
build_units 对孤儿/超员/角色缺失重复 fail-fast raise ValueError（防 next 静默
StopIteration）；unit_correctness 走 per_q[qid] KeyError 防静默兜底。
 2026-07-15 05:52:30 -04:00
iomgaa 7ef9b99217 feat: add QuestionUnit entity and GeneratedQuestion pair fields 
新增 QuestionUnit 契约实体（kind/unit_id/task_type/questions/unit_hash + size/from_single/from_pair），
作为 AR pair 孪生对贯穿评测/训练 harness 的最小调度单元。
GeneratedQuestion 追加 unit_id/pair_id/question_role/flip_axis 四字段，
__post_init__ 回填 unit_id，默认值保证现有非 AR single 题构造点行为不变。
 2026-07-15 05:45:34 -04:00
iomgaa b1c1bf7aac fix: isolate selector VLM exceptions at slot level (no batch crash) 2026-07-14 21:08:00 -04:00
iomgaa f36eb66c18 fix: tolerate backfill under-delivery, fix predict None-fill, drop dead session_id 2026-07-14 17:07:55 -04:00
iomgaa 1d222d9f18 feat: wire real agent runner, backfill assembly and adversarial-filter CLI 2026-07-14 16:58:32 -04:00
iomgaa 73d0bb9190 feat: add adversarial round loop with backfill iteration and difficulty report 2026-07-14 16:40:05 -04:00
iomgaa c1565a01c2 test: assert stage/verdict/pair_id and Q-refusal flip_skipped path 2026-07-14 16:33:09 -04:00
iomgaa 15aee0cfc1 feat: add flip gate reusing P prediction and mirror-Q agent run 2026-07-14 16:26:51 -04:00
iomgaa 8731e448fe fix: harden mirror material rebuild (empty/invalid frames, dedup, observability) 2026-07-14 16:16:38 -04:00
iomgaa 4af99b700b feat: add mirror question generation with canonical distinctness check 2026-07-14 16:08:13 -04:00
iomgaa d4e9852864 feat: add cheater gate with resume-safe survivor recovery 2026-07-14 16:00:14 -04:00
iomgaa 24ed7ca322 fix: harden canonical_answer_text and align decision-core signatures to spec 2026-07-14 15:55:36 -04:00
iomgaa c109f2257a feat: add pure decision core for adversarial filter (hash/fingerprint/canonical/flip) 2026-07-14 15:51:28 -04:00
iomgaa 334fbbc94d feat: add AdversarialFilterConfig for Phase B post-hoc filter layer 2026-07-14 15:47:21 -04:00
iomgaa ac115d96fb test: cover backfill params end-to-end (mutation isolation + seq_offset) 2026-07-14 15:44:43 -04:00
iomgaa 8b9e8aa19f feat: add optional backfill params to run_pipeline_v2 2026-07-14 15:39:57 -04:00
iomgaa d77cbc95eb feat: add adversarial_verdicts table with resume and terminal-verdict query 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-14 15:35:11 -04:00
iomgaa e41a2b0d08 test: guard non-AR flip isolation and AR flip-capable set 2026-07-14 15:30:18 -04:00
iomgaa f12dd7e559 feat: add supports_flip/flip_axis to SubPattern for Phase B flip gate 2026-07-14 15:27:21 -04:00
iomgaa 96e314c3a0 fix: mark item rejected on selector_error for consistent bookkeeping 
selector_error 分支（捕获 ValueError/FileNotFoundError）此前只返回 reason，
未 mark_item_rejected，导致 Phase 3 已 record 的 pending attempt 行永远停在
pending；而 hard-fail 分支会标记 rejected。两条失败路径落库风格现统一为
mark_item_rejected（异常路径无 outcome/observation，故不写 selector_scores）。

补单测 test_apply_grounded_selector_marks_rejected_on_error 守卫该路径。
 2026-07-14 14:49:41 -04:00
iomgaa 46ac848176 fix: converge non-numeric selector scores to ValueError 2026-07-14 14:47:21 -04:00
iomgaa 76f719018c feat: loosen multi_true gate to qualifier-scoped correctness 2026-07-14 14:33:22 -04:00
iomgaa b1f15ddb3a fix: keep cross_segment rule single-dimension, guard no-absent-events 2026-07-14 14:31:30 -04:00
iomgaa 58278c6de4 feat: enforce single-dimension counterfactual in AR distractor rules 2026-07-14 14:28:32 -04:00
iomgaa b13eab0659 feat: wire grounded selector into AR slot processing 
将 Task 5 的 grounded selector 织入 AR 出题路径（Phase 3.5，位于
record_item 与 postprocess 之间），仅在 strategy.uses_grounded_selector
为真时进入。observation 始终落库（含 hard-fail），硬失败走重出。
PipelineConfig 新增 candidate_pool_size/selector_delta_low/
selector_delta_high 三参，YAML 与 CLI seed override 同步。
 2026-07-14 14:15:50 -04:00
iomgaa d0194f5840 fix: degrade distractor pool gracefully on malformed VLM response 2026-07-14 14:11:13 -04:00
iomgaa 8a54055d02 feat: add grounded distractor selector with visual scoring 2026-07-14 14:06:07 -04:00
iomgaa 207e834f30 feat: add selector_scores observation column to question_gen_items 2026-07-14 13:59:29 -04:00
iomgaa 3f984acc18 feat: add uses_grounded_selector strategy switch (AR only) 2026-07-14 13:53:17 -04:00
iomgaa 2608a3841f style: reformat sub_pattern round-trip tests 2026-07-14 13:51:51 -04:00
iomgaa e68e4b7d57 fix: restore sub_pattern when loading benchmark JSON 2026-07-14 13:51:05 -04:00
iomgaa 111c88488f style: format sub_pattern test file 2026-07-14 13:46:25 -04:00
iomgaa ae0a718f67 feat: thread and persist sub_pattern into accepted questions 2026-07-14 13:45:31 -04:00
iomgaa 25f2a845ff fix: run_gates must use current_tree after video resample 2026-07-14 13:40:46 -04:00
iomgaa 84b52a0311 feat(pools): auto-supplement maintenance correct questions in PerCategoryPoolStrategy 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 10:41:46 -04:00
iomgaa d7d7ce5bdc feat(question_gen): register ActionRecognitionStrategy, replace temp VISUAL binding 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 06:51:50 -04:00
iomgaa d0a8019fe1 feat(question_gen): add ActionRecognitionStrategy with 6 SubPatterns 
Self-contained strategy targeting 6 Agent failure modes in Action
Recognition: premature_evidence_anchoring, temporal_reasoning_failure,
semantic_rigidity, fine_grained_visual_action,
cross_segment_entity_tracking, evidence_gap_confabulation.

- L2 default sampling (upgrade from L3) with 3 patterns overriding to L1
- Weighted random SubPattern selection (0.20/0.20/0.15/0.15/0.15/0.15)
- Each SubPattern includes instruction, examples, distractor rules
- Satisfies TaskTypeStrategy Protocol without extending BaseTaskTypeStrategy
- 32 unit tests covering all properties, definitions, and selection behavior

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 06:49:03 -04:00
iomgaa 36c712defa feat(pipeline): replace QuestionFamilySpec with TaskTypeStrategy 
- SlotAssignment: remove family field, strategy looked up at process time
- PipelineConfig: remove family_ratios field
- _assign_slots: remove family_ratios and rng params (pure deterministic)
- _process_one_slot: use get_strategy() for sampling, generation, gates
- Add sub_pattern support (level/constraint override, instruction injection)
- Add strategy.extra_gates() check after standard gates
- load_pipeline_config: stop reading family_ratios from YAML
- Update tools/generate_questions.py seed override and dry-run log
- Update all integration tests to match new API

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 06:05:23 -04:00
iomgaa b9616de21e fix(tests): update test_generator_v2 to use new generator signatures 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 05:55:08 -04:00
iomgaa afa77173e3 refactor(question_gen): adapt generator/gates/store signatures for strategy 
- generator_v2: _load_prompt_template takes template_name str instead of
  QuestionFamilySpec; _build_v2_prompt takes prompt_template + strategy_name
  + sub_pattern_instruction; generate_one_v2 takes discrete params
  (prompt_template, strategy_name, skill_target, sub_pattern_instruction)
- gates: _gate_leak_test and run_gates take leak_probe_template str
  instead of QuestionFamilySpec
- run_store: add sub_pattern column to DDL + idempotent migration;
  record_item accepts optional sub_pattern param
- Remove QuestionFamilySpec imports from generator_v2 and gates modules
- Update test call sites accordingly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 05:50:47 -04:00
iomgaa c49d0ff12f fix(sampler): validate level param rejects invalid values 
Add ValueError guard at the top of sample_material_v2 for level not in
{1, 2, 3}, preventing silent fallthrough to L1 sampling. Add unit test
for the new validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 05:43:28 -04:00
iomgaa e2325b6535 refactor(sampler): replace family_spec param with level+constraint 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 05:38:14 -04:00
iomgaa b6b6a48503 feat(question_gen): add TaskTypeStrategy Protocol and BaseTaskTypeStrategy 
- TaskTypeStrategy Protocol: pipeline 的唯一接口，定义 task_type、
  sampling_level、sampling_constraint、prompt_template 等属性
- SubPattern frozen dataclass: 出题子模式，靶向特定失败机制
- BaseTaskTypeStrategy: 封装现有 QuestionFamilySpec 行为的默认策略，
  所有属性委托给绑定的 family
- _TASK_TYPE_TO_FAMILY: 消歧绑定表，12 个题型确定性绑定到 1 个 family
- register_strategy/get_strategy: 注册表 API，未注册题型自动创建
  BaseTaskTypeStrategy
- 13 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-14 05:24:29 -04:00
iomgaa dec7346da3 feat(harness): add Action Recognition training experiment 
- PerCategoryPoolStrategy: filter test pool by task_types
- RunConfig: add run_holdout_eval toggle (default true)
- load_config: fix YAML task_types list-to-tuple conversion
- Runner: conditionally skip _holdout_four_way when disabled
- CLI: add --no-run-holdout-eval flag
- New config/train_action_recognition.yaml (3 epochs, per_category)
- New scripts/train_action_recognition.sh (baseline + seed + train)
 2026-07-14 00:58:54 -04:00
iomgaa 37d4519905 chore: lint and format per-category pool strategy implementation 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-12 22:58:10 -04:00
iomgaa 671db2f88c test(integration): add PerCategoryPoolStrategy end-to-end test 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-12 22:56:42 -04:00
iomgaa c66a00c924 feat(harness): refactor build_or_load_pools to accept PoolStrategy + per_category freeze format 
- save_pools: extended with split_mode and config params; per_category
  mode writes categories metadata (seed, train_ratio, test_source) for
  incremental append and consistency validation
- load_pools: compatible with both old format (no split_mode) and new
  format; extra metadata fields ignored during load
- build_or_load_pools: signature changed to (config, strategy, db_path);
  baseline_run_id read from seed.json (not config.run_id); per_category
  mode does consistency check on reload and supports incremental category
  append via strategy.build_incremental
- Added _to_pool_config, _read_baseline_run_id,
  _validate_per_category_consistency helpers
- Tests: TestPerCategorySaveLoad with 5 test cases covering roundtrip,
  missing config error, global split_mode field, legacy format compat,
  multi-type categories

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-12 22:53:19 -04:00
iomgaa e5b07ac974 feat(harness): add task_types, pool_split_mode, train_ratio, test_questions to RunConfig 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-07-12 22:46:28 -04:00