feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-11 23:30:13 -04:00
parent 9627ac9cf9
commit 271d1682c9
10 changed files with 1030 additions and 0 deletions
@@ -0,0 +1,32 @@
# Leak Test: Frequency Shortcut (REASONING Family)
You are detecting whether a question exploits word frequency patterns as a shortcut.
## Question
{question}
## Options
{options}
## Designated Correct Answer
{answer}
## Instructions
Check if the question can be answered by picking the option that shares the most words or phrases with the question stem, without genuine causal/logical reasoning.
1. Does the correct answer have significantly more word overlap with the question than distractors?
2. Are distractor options phrased in notably different vocabulary from the question?
3. Could a student "game" this question by matching keywords between question and options?
If frequency-based word matching is an exploitable shortcut → verdict "fail".
If the question requires genuine reasoning → verdict "pass".
## Response Format (strict JSON)
```json
{{"verdict": "pass" or "fail", "reason": "brief explanation"}}
```