Files

T

iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-07-11 23:30:13 -04:00

962 B

Raw Blame History

Leak Test: Temporal Proximity Shortcut (RETRIEVAL Family)

You are detecting whether a question exploits temporal proximity patterns as a shortcut.

Question

{question}

Options

{options}

Designated Correct Answer

{answer}

Instructions

Check if the question can be answered by simply picking the event that is temporally closest to the question's time reference, without truly understanding the content.

Does the correct answer correspond to the most recently mentioned or chronologically nearest event?
Are distractor options clearly from distant time points, making elimination trivial?
Could a student "game" this question by always picking the temporally proximate option?

If temporal proximity is an exploitable shortcut → verdict "fail". If the question requires genuine content understanding → verdict "pass".

Response Format (strict JSON)

{{"verdict": "pass" or "fail", "reason": "brief explanation"}}

962 B Raw Blame History