feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-11 23:30:13 -04:00
parent 9627ac9cf9
commit 271d1682c9
10 changed files with 1030 additions and 0 deletions
@@ -0,0 +1,24 @@
# Blind Answer Gate
You are evaluating whether a multiple-choice question can be answered correctly **without any video context**, using only common sense or option patterns.
## Question
{question}
## Options
{options}
## Instructions
1. Read the question and options WITHOUT any source material.
2. Try to determine the correct answer using only common sense, option length patterns, or grammatical cues.
3. If you can confidently pick the correct answer → verdict "fail" (the question leaks information).
4. If you cannot determine the answer without context → verdict "pass" (the question genuinely requires video understanding).
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,35 @@
# Key Verify Gate
You are a quality-control judge for video understanding questions.
## Task
Given the source material from a video and a multiple-choice question with its designated correct answer, determine whether the correct answer is **supported by evidence** in the source material.
## Source Material
{source_text}
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
1. Read the source material carefully.
2. Determine if the designated correct answer can be derived or inferred from the source material.
3. If evidence supports the answer, verdict is "pass". If not, verdict is "fail".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,32 @@
# Leak Test: Option Length Shortcut (ENUMERATION Family)
You are detecting whether a question exploits option length patterns as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by exploiting the length or specificity of options, without actually understanding the video content.
1. Is the correct answer notably longer or more specific than distractors?
2. Are distractor options clearly shorter, vaguer, or less detailed than the correct answer?
3. Could a student "game" this question by always picking the longest/most-detailed option?
If option length is an exploitable shortcut → verdict "fail".
If options are roughly balanced in length and specificity → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,32 @@
# Leak Test: Frequency Shortcut (REASONING Family)
You are detecting whether a question exploits word frequency patterns as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by picking the option that shares the most words or phrases with the question stem, without genuine causal/logical reasoning.
1. Does the correct answer have significantly more word overlap with the question than distractors?
2. Are distractor options phrased in notably different vocabulary from the question?
3. Could a student "game" this question by matching keywords between question and options?
If frequency-based word matching is an exploitable shortcut → verdict "fail".
If the question requires genuine reasoning → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,32 @@
# Leak Test: Temporal Proximity Shortcut (RETRIEVAL Family)
You are detecting whether a question exploits temporal proximity patterns as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by simply picking the event that is temporally closest to the question's time reference, without truly understanding the content.
1. Does the correct answer correspond to the most recently mentioned or chronologically nearest event?
2. Are distractor options clearly from distant time points, making elimination trivial?
3. Could a student "game" this question by always picking the temporally proximate option?
If temporal proximity is an exploitable shortcut → verdict "fail".
If the question requires genuine content understanding → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,32 @@
# Leak Test: Spatial Default Shortcut (SPATIAL Family)
You are detecting whether a question exploits spatial default assumptions as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by relying on common spatial assumptions (e.g., "center of frame", "left to right", "foreground") without actual visual understanding.
1. Is the correct answer the "default" spatial position people would assume (e.g., center, front)?
2. Are distractor options in positions that seem intuitively unlikely (e.g., extreme edges, behind)?
3. Could a student "game" this question by always picking the spatially default/expected option?
If spatial defaults are an exploitable shortcut → verdict "fail".
If the question requires genuine spatial reasoning → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,32 @@
# Leak Test: Visual Salience Shortcut (VISUAL Family)
You are detecting whether a question exploits visual salience patterns as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by simply picking the most visually salient or "obvious" object/action, without careful visual analysis.
1. Is the correct answer the most prominent or commonly expected element in such a scene?
2. Are distractor options obviously implausible or uncommon objects/actions for the described setting?
3. Could a student "game" this question by guessing the most visually dominant element?
If visual salience is an exploitable shortcut → verdict "fail".
If the question requires careful visual discrimination → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```
@@ -0,0 +1,28 @@
# Multi-True Gate
You are checking whether a multiple-choice question has **more than one plausibly correct answer** given the source material.
## Source Material
{source_text}
## Question
{question}
## Options
{options}
## Instructions
1. Read the source material and the question carefully.
2. For each option, assess whether it could be considered a correct or plausible answer given the source.
3. If exactly ONE option is clearly correct → verdict "pass".
4. If TWO or more options are plausibly correct → verdict "fail".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation listing plausible options if multiple"}}
```