1. Apply postprocess shuffle result (pp.options, pp.answer) to final
GeneratedQuestion output instead of using original candidate values.
2. Record dedup rejection in store via new mark_item_rejected() method,
preventing items from staying as 'accepted' after dedup rejects them.
3. Add .flatten() to embed_fn outputs in _is_duplicate and embed_pool
append to handle 2D (1,D) arrays from embedding implementations.
4. Validate exactly 4 options in _validate_parsed_fields (was >= 2),
matching the A-D answer constraint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract _get_git_sha, _filter_pending_slots, and _run_heavy_sampling
from run_pipeline_v2. Reduces cyclomatic complexity from C(15) to B(6).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PipelineConfig: YAML-driven configuration with family_ratios, retry,
concurrency, dedup threshold, and heavy sampling rate
- _assign_slots: deterministic round-robin slot assignment across videos
with per-family weighted random selection
- _process_one_slot: full retry loop (generate → postprocess → gates →
dedup) with reject-reason feedback to VLM on retry
- _heavy_check_one: blind LLM agent trial-answer for difficulty_steps
- run_pipeline_v2: orchestration with semaphore-bounded concurrency,
progress/resume support, and store integration
- is_duplicate: cosine similarity dedup against embedding pool
Tests: 11 integration tests covering slot assignment, retry behavior,
max-retries exhaustion, full pipeline flow, progress resume, heavy
sampling, and store record completeness.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
iomgaa