feat(question_gen): add lightweight 4-gate quality check
Implement 4 concurrent LLM-based quality gates for generated questions: - key_verify: validates answer evidence in source material - blind_answer: rejects questions answerable without video context - multi_true: detects ambiguous multi-correct options - leak_test: per-family shortcut detection (5 probe templates) Includes run_gates orchestrator with verbatim_ratio short-circuit, JSON response parsing with fallback, and 9 unit tests (all passing). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# Blind Answer Gate
|
||||
|
||||
You are evaluating whether a multiple-choice question can be answered correctly **without any video context**, using only common sense or option patterns.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Read the question and options WITHOUT any source material.
|
||||
2. Try to determine the correct answer using only common sense, option length patterns, or grammatical cues.
|
||||
3. If you can confidently pick the correct answer → verdict "fail" (the question leaks information).
|
||||
4. If you cannot determine the answer without context → verdict "pass" (the question genuinely requires video understanding).
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,35 @@
|
||||
# Key Verify Gate
|
||||
|
||||
You are a quality-control judge for video understanding questions.
|
||||
|
||||
## Task
|
||||
|
||||
Given the source material from a video and a multiple-choice question with its designated correct answer, determine whether the correct answer is **supported by evidence** in the source material.
|
||||
|
||||
## Source Material
|
||||
|
||||
{source_text}
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Read the source material carefully.
|
||||
2. Determine if the designated correct answer can be derived or inferred from the source material.
|
||||
3. If evidence supports the answer, verdict is "pass". If not, verdict is "fail".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Option Length Shortcut (ENUMERATION Family)
|
||||
|
||||
You are detecting whether a question exploits option length patterns as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by exploiting the length or specificity of options, without actually understanding the video content.
|
||||
|
||||
1. Is the correct answer notably longer or more specific than distractors?
|
||||
2. Are distractor options clearly shorter, vaguer, or less detailed than the correct answer?
|
||||
3. Could a student "game" this question by always picking the longest/most-detailed option?
|
||||
|
||||
If option length is an exploitable shortcut → verdict "fail".
|
||||
If options are roughly balanced in length and specificity → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Frequency Shortcut (REASONING Family)
|
||||
|
||||
You are detecting whether a question exploits word frequency patterns as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by picking the option that shares the most words or phrases with the question stem, without genuine causal/logical reasoning.
|
||||
|
||||
1. Does the correct answer have significantly more word overlap with the question than distractors?
|
||||
2. Are distractor options phrased in notably different vocabulary from the question?
|
||||
3. Could a student "game" this question by matching keywords between question and options?
|
||||
|
||||
If frequency-based word matching is an exploitable shortcut → verdict "fail".
|
||||
If the question requires genuine reasoning → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Temporal Proximity Shortcut (RETRIEVAL Family)
|
||||
|
||||
You are detecting whether a question exploits temporal proximity patterns as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by simply picking the event that is temporally closest to the question's time reference, without truly understanding the content.
|
||||
|
||||
1. Does the correct answer correspond to the most recently mentioned or chronologically nearest event?
|
||||
2. Are distractor options clearly from distant time points, making elimination trivial?
|
||||
3. Could a student "game" this question by always picking the temporally proximate option?
|
||||
|
||||
If temporal proximity is an exploitable shortcut → verdict "fail".
|
||||
If the question requires genuine content understanding → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Spatial Default Shortcut (SPATIAL Family)
|
||||
|
||||
You are detecting whether a question exploits spatial default assumptions as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by relying on common spatial assumptions (e.g., "center of frame", "left to right", "foreground") without actual visual understanding.
|
||||
|
||||
1. Is the correct answer the "default" spatial position people would assume (e.g., center, front)?
|
||||
2. Are distractor options in positions that seem intuitively unlikely (e.g., extreme edges, behind)?
|
||||
3. Could a student "game" this question by always picking the spatially default/expected option?
|
||||
|
||||
If spatial defaults are an exploitable shortcut → verdict "fail".
|
||||
If the question requires genuine spatial reasoning → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Visual Salience Shortcut (VISUAL Family)
|
||||
|
||||
You are detecting whether a question exploits visual salience patterns as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by simply picking the most visually salient or "obvious" object/action, without careful visual analysis.
|
||||
|
||||
1. Is the correct answer the most prominent or commonly expected element in such a scene?
|
||||
2. Are distractor options obviously implausible or uncommon objects/actions for the described setting?
|
||||
3. Could a student "game" this question by guessing the most visually dominant element?
|
||||
|
||||
If visual salience is an exploitable shortcut → verdict "fail".
|
||||
If the question requires careful visual discrimination → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
@@ -0,0 +1,28 @@
|
||||
# Multi-True Gate
|
||||
|
||||
You are checking whether a multiple-choice question has **more than one plausibly correct answer** given the source material.
|
||||
|
||||
## Source Material
|
||||
|
||||
{source_text}
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Read the source material and the question carefully.
|
||||
2. For each option, assess whether it could be considered a correct or plausible answer given the source.
|
||||
3. If exactly ONE option is clearly correct → verdict "pass".
|
||||
4. If TWO or more options are plausibly correct → verdict "fail".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation listing plausible options if multiple"}}
|
||||
```
|
||||
Reference in New Issue
Block a user