Files

T

iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-07-11 23:30:13 -04:00

902 B

Raw Permalink Blame History

Leak Test: Option Length Shortcut (ENUMERATION Family)

You are detecting whether a question exploits option length patterns as a shortcut.

Question

{question}

Options

{options}

Designated Correct Answer

{answer}

Instructions

Check if the question can be answered by exploiting the length or specificity of options, without actually understanding the video content.

Is the correct answer notably longer or more specific than distractors?
Are distractor options clearly shorter, vaguer, or less detailed than the correct answer?
Could a student "game" this question by always picking the longest/most-detailed option?

If option length is an exploitable shortcut → verdict "fail". If options are roughly balanced in length and specificity → verdict "pass".

Response Format (strict JSON)

{{"verdict": "pass" or "fail", "reason": "brief explanation"}}

902 B Raw Permalink Blame History