feat(question_gen): add lightweight 4-gate quality check
Implement 4 concurrent LLM-based quality gates for generated questions: - key_verify: validates answer evidence in source material - blind_answer: rejects questions answerable without video context - multi_true: detects ambiguous multi-correct options - leak_test: per-family shortcut detection (5 probe templates) Includes run_gates orchestrator with verbatim_ratio short-circuit, JSON response parsing with fallback, and 9 unit tests (all passing). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
# Leak Test: Spatial Default Shortcut (SPATIAL Family)
|
||||
|
||||
You are detecting whether a question exploits spatial default assumptions as a shortcut.
|
||||
|
||||
## Question
|
||||
|
||||
{question}
|
||||
|
||||
## Options
|
||||
|
||||
{options}
|
||||
|
||||
## Designated Correct Answer
|
||||
|
||||
{answer}
|
||||
|
||||
## Instructions
|
||||
|
||||
Check if the question can be answered by relying on common spatial assumptions (e.g., "center of frame", "left to right", "foreground") without actual visual understanding.
|
||||
|
||||
1. Is the correct answer the "default" spatial position people would assume (e.g., center, front)?
|
||||
2. Are distractor options in positions that seem intuitively unlikely (e.g., extreme edges, behind)?
|
||||
3. Could a student "game" this question by always picking the spatially default/expected option?
|
||||
|
||||
If spatial defaults are an exploitable shortcut → verdict "fail".
|
||||
If the question requires genuine spatial reasoning → verdict "pass".
|
||||
|
||||
## Response Format (strict JSON)
|
||||
|
||||
```json
|
||||
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
|
||||
```
|
||||
Reference in New Issue
Block a user