Files

T

iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check

Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-07-11 23:30:13 -04:00

972 B

Raw Permalink Blame History

Leak Test: Spatial Default Shortcut (SPATIAL Family)

You are detecting whether a question exploits spatial default assumptions as a shortcut.

Question

{question}

Options

{options}

Designated Correct Answer

{answer}

Instructions

Check if the question can be answered by relying on common spatial assumptions (e.g., "center of frame", "left to right", "foreground") without actual visual understanding.

Is the correct answer the "default" spatial position people would assume (e.g., center, front)?
Are distractor options in positions that seem intuitively unlikely (e.g., extreme edges, behind)?
Could a student "game" this question by always picking the spatially default/expected option?

If spatial defaults are an exploitable shortcut → verdict "fail". If the question requires genuine spatial reasoning → verdict "pass".

Response Format (strict JSON)

{{"verdict": "pass" or "fail", "reason": "brief explanation"}}

972 B Raw Permalink Blame History