Files

T

iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-07-11 23:30:13 -04:00

924 B

Raw Blame History

Leak Test: Visual Salience Shortcut (VISUAL Family)

You are detecting whether a question exploits visual salience patterns as a shortcut.

Question

{question}

Options

{options}

Designated Correct Answer

{answer}

Instructions

Check if the question can be answered by simply picking the most visually salient or "obvious" object/action, without careful visual analysis.

Is the correct answer the most prominent or commonly expected element in such a scene?
Are distractor options obviously implausible or uncommon objects/actions for the described setting?
Could a student "game" this question by guessing the most visually dominant element?

If visual salience is an exploitable shortcut → verdict "fail". If the question requires careful visual discrimination → verdict "pass".

Response Format (strict JSON)

{{"verdict": "pass" or "fail", "reason": "brief explanation"}}

924 B Raw Blame History