load_benchmark used the JSON filename stem as video_id, which broke
v2-360 questions (all 180 questions got video_id='accepted_questions').
Now uses qa['video_id'] when present, falls back to filename for
Video-MME format compatibility.
- Add --pool-split-mode, --train-ratio, --test-questions CLI arguments
- Include task_types in cli_overrides (convert list to tuple for RunConfig)
- Wire train mode branch: select strategy based on pool_split_mode,
resolve workspace paths, build pools, and call runner.train()
- Infer mode now reads task_types from config instead of raw args
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- save_pools: extended with split_mode and config params; per_category
mode writes categories metadata (seed, train_ratio, test_source) for
incremental append and consistency validation
- load_pools: compatible with both old format (no split_mode) and new
format; extra metadata fields ignored during load
- build_or_load_pools: signature changed to (config, strategy, db_path);
baseline_run_id read from seed.json (not config.run_id); per_category
mode does consistency check on reload and supports incremental category
append via strategy.build_incremental
- Added _to_pool_config, _read_baseline_run_id,
_validate_per_category_consistency helpers
- Tests: TestPerCategorySaveLoad with 5 test cases covering roundtrip,
missing config error, global split_mode field, legacy format compat,
multi-type categories
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Apply postprocess shuffle result (pp.options, pp.answer) to final
GeneratedQuestion output instead of using original candidate values.
2. Record dedup rejection in store via new mark_item_rejected() method,
preventing items from staying as 'accepted' after dedup rejects them.
3. Add .flatten() to embed_fn outputs in _is_duplicate and embed_pool
append to handle 2D (1,D) arrays from embedding implementations.
4. Validate exactly 4 options in _validate_parsed_fields (was >= 2),
matching the A-D answer constraint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add generate-v2 subparser with --config, --store-dir, --db-path,
--seed, and --dry-run arguments to tools/generate_questions.py
- Implement _run_generate_v2 async handler: config loading, video
discovery, DI client construction, TreeIndex loading, pipeline
invocation, and result persistence
- Add scripts/generate_questions_v2.sh following build_trees.sh
conventions (source .env, conda run python path, MODE=mock support)
- Update app/question_gen/__init__.py to export full v2 public API:
run_pipeline_v2, PipelineConfig, PipelineResult, QuestionFamilySpec,
ALL_FAMILIES, CandidateQuestion, generate_one_v2, GateReport, run_gates
- Add QuestionGenStore.load_progress() for pipeline resumption
- Add integration tests for CLI help and dry-run behavior
- Update test_question_gen_api to match expanded __all__
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract _get_git_sha, _filter_pending_slots, and _run_heavy_sampling
from run_pipeline_v2. Reduces cyclomatic complexity from C(15) to B(6).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PipelineConfig: YAML-driven configuration with family_ratios, retry,
concurrency, dedup threshold, and heavy sampling rate
- _assign_slots: deterministic round-robin slot assignment across videos
with per-family weighted random selection
- _process_one_slot: full retry loop (generate → postprocess → gates →
dedup) with reject-reason feedback to VLM on retry
- _heavy_check_one: blind LLM agent trial-answer for difficulty_steps
- run_pipeline_v2: orchestration with semaphore-bounded concurrency,
progress/resume support, and store integration
- is_duplicate: cosine similarity dedup against embedding pool
Tests: 11 integration tests covering slot assignment, retry behavior,
max-retries exhaustion, full pipeline flow, progress resume, heavy
sampling, and store record completeness.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split field validation logic out of _parse_v2_response into a dedicated
_validate_parsed_fields helper. This brings _parse_v2_response from CC=11
(grade C) down to CC=3 (grade A). The extracted validator is CC=9 (grade B).
No grade-C functions remain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GateReport.reject_reason now returns gate.reason directly (no [name] prefix)
- verbatim short-circuit sets other 3 gates to SKIP without calling LLM
- test_high_verbatim_shortcircuits asserts zero LLM calls and SKIP verdicts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
record_run_end, update_gates, and update_difficulty now check
cursor.rowcount after UPDATE+commit and raise ValueError if 0 rows
were affected. Prevents silent telemetry loss.
Adds three negative-path tests:
- test_record_run_end_missing_run_raises
- test_update_gates_missing_item_raises
- test_update_difficulty_missing_item_raises
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- check_verbatim now computes n-gram overlap for BOTH question_text and
correct_option vs source texts, returning max(question_ratio, option_ratio).
Extracted _ngram_overlap_ratio helper for reuse.
- Added 4 missing blacklist patterns: 'this segment', 'this frame',
'the current frame', 'frame summary'.
- Added 5 new test cases covering the above changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Aligns DDL column name with research-wiki/schemas/question-gen-runs.md
which specifies 'ended_at' (not 'finished_at').
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract _match_any helper and declarative _FORBIDDEN_MATERIAL_RULES table
to replace repetitive per-category for-loops. Reduces cyclomatic complexity
from C(11) to A(3).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add skill_target (str | None) and difficulty_steps (int | None) fields
to GeneratedQuestion dataclass with field(default=None)
- Update loader.py to pass new fields from JSON (backward-compatible)
- Update pools.py _q_to_dict/_dict_to_q for serialization compat
- Add question_gen_v2 config section to default.yaml
- Add comprehensive test coverage (7 tests)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core algorithms #1/#2/#3 unchanged: only entry wrapping and semaphore
source switch (injected vs self-built); build logic untouched.
- rename _build_async to public build_async (body unchanged)
- __init__ accepts keyword-only api_semaphore for cross-video sharing
- default path (no injection) is verbatim-equivalent to previous code
iomgaa