feat(question_gen): add v2 generator with per-family prompt templates

Implement generator_v2.py with:
- CandidateQuestion dataclass (canonical location)
- _load_prompt_template: loads per-family .md from store/prompts/
- _build_v2_prompt: constructs system+user messages with material context
- _parse_v2_response: JSON extraction, json_repair, field validation
- generate_one_v2: async VLM call orchestration with reject_reason support

Add 5 family-specific prompt templates:
- retrieval.md: factual recall from visible content
- reasoning.md: multi-hop inference across segments
- enumeration.md: counting/listing entities and actions
- visual.md: visual details requiring frame observation
- spatial.md: spatial relationships between objects/people

Tests: 11 unit tests covering prompt build, parse, and e2e generation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-11 23:41:36 -04:00
parent 9053233f99
commit cf51d2de9d
7 changed files with 928 additions and 0 deletions
+22
View File
@@ -0,0 +1,22 @@
You are a question generator for video understanding benchmarks.
Your task: Generate an **enumeration** multiple-choice question that tests counting, listing, or identifying the number/set of specific entities or actions in the video content.
## Guidelines
- The question MUST require counting entities, listing items, or identifying sequences of actions.
- Focus on "How many...", "Which of the following are all...", "In what order..." style questions.
- The answer should require careful attention to all relevant parts of the content — partial viewing should not suffice.
- Distractors should represent common counting errors (off-by-one, missing/extra items, wrong order).
## Quality Requirements
- Question must be grammatically correct and unambiguous.
- All four options must be parallel in structure and length.
- The correct answer must not be identifiable by option length or format alone.
- Avoid trivially small counts (e.g., "How many people?" when only 1 is visible).
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Output
Respond with ONLY a valid JSON object. No additional text.
+22
View File
@@ -0,0 +1,22 @@
You are a question generator for video understanding benchmarks.
Your task: Generate a **multi-hop reasoning** multiple-choice question that requires connecting information from multiple segments of the video to arrive at the correct answer.
## Guidelines
- The question MUST require reasoning across at least two distinct pieces of information (temporal, causal, or logical connections).
- The answer should NOT be directly stated in any single subtitle or frame — it must be inferred by combining evidence.
- Test causal chains, temporal ordering, or logical deductions that span multiple events.
- Distractors should represent common reasoning errors (e.g., reversed causality, incorrect temporal ordering).
## Quality Requirements
- Question must be grammatically correct and unambiguous.
- All four options must be parallel in structure and length.
- The correct answer must not be identifiable from linguistic cues alone.
- The reasoning chain should be verifiable from the provided material.
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Output
Respond with ONLY a valid JSON object. No additional text.
+23
View File
@@ -0,0 +1,23 @@
You are a question generator for video understanding benchmarks.
Your task: Generate a **factual retrieval** multiple-choice question that tests whether the answerer can recall specific information directly observable in the provided video content.
## Guidelines
- The question MUST target factual recall — the answer should be directly stated or clearly shown in the source material.
- The correct answer must be unambiguously supported by the subtitle text or visual content.
- Distractors (wrong options) must be plausible but clearly incorrect given the source material.
- Do NOT require multi-hop reasoning or inference beyond the directly presented facts.
- The question should be answerable ONLY by someone who has seen/read the source content — avoid common-sense questions.
## Quality Requirements
- Question must be grammatically correct and unambiguous.
- All four options must be parallel in structure and length.
- The correct answer must not be identifiable from linguistic cues alone.
- Avoid negation in the question stem (e.g., "Which of the following is NOT...").
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Output
Respond with ONLY a valid JSON object. No additional text.
+23
View File
@@ -0,0 +1,23 @@
You are a question generator for video understanding benchmarks.
Your task: Generate a **spatial relationship** multiple-choice question that tests understanding of spatial arrangements, positions, and relationships between objects or people in the video.
## Guidelines
- The question MUST focus on spatial relationships: relative positions, directions, distances, containment, or spatial changes.
- Test understanding of "where" things are, how they relate spatially, or how spatial arrangements change over time.
- Use spatial language: "left/right of", "above/below", "between", "inside/outside", "closer/farther", "facing".
- The answer should require spatial reasoning that goes beyond simple object identification.
- Distractors should represent common spatial confusion (mirror reversals, misremembered positions).
## Quality Requirements
- Question must be grammatically correct and unambiguous.
- All four options must be parallel in structure and length.
- The correct answer must not be identifiable from linguistic cues alone.
- Spatial references must be unambiguous given the visual content.
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Output
Respond with ONLY a valid JSON object. No additional text.
+22
View File
@@ -0,0 +1,22 @@
You are a question generator for video understanding benchmarks.
Your task: Generate a **visual detail** multiple-choice question that requires observing specific visual information from the video frames — details that cannot be answered from subtitles or text alone.
## Guidelines
- The question MUST target visual details: colors, shapes, positions, appearances, visual states, or visual actions.
- The answer should be verifiable ONLY by looking at the actual frames — subtitle text alone must NOT suffice.
- Focus on concrete visual observations: what objects look like, their appearance, visual relationships.
- Distractors should be visually plausible alternatives that could be confused without careful observation.
## Quality Requirements
- Question must be grammatically correct and unambiguous.
- All four options must be parallel in structure and length.
- The correct answer must not be identifiable from linguistic cues alone.
- Avoid questions about things that are typically described in subtitles (dialogue content, narration).
- Each option must begin with "A. ", "B. ", "C. ", or "D. ".
## Output
Respond with ONLY a valid JSON object. No additional text.