feat: loosen multi_true gate to qualifier-scoped correctness

This commit is contained in:
2026-07-14 14:33:22 -04:00
parent b1f15ddb3a
commit 76f719018c
2 changed files with 24 additions and 4 deletions
+11 -4
View File
@@ -16,10 +16,17 @@ You are checking whether a multiple-choice question has **more than one plausibl
## Instructions ## Instructions
1. Read the source material and the question carefully. 1. Read the source material and the question carefully. Note the question's
2. For each option, assess whether it could be considered a correct or plausible answer given the source. explicit qualifiers (subject, timing, manner, object).
3. If exactly ONE option is clearly correct → verdict "pass". 2. For each option, assess whether it is **fully correct under the question's
4. If TWO or more options are plausibly correct → verdict "fail". qualifiers** — not merely whether it has some partial or local support in
the source.
3. A good hard-negative distractor MAY have partial/local evidence in the video
yet be FALSE under the question's constraints. Such an option is NOT a
second correct answer — do NOT fail the question for it.
4. Verdict "fail" ONLY IF two or more options are each fully correct under the
question's qualifiers (a genuine ambiguity).
5. Otherwise verdict "pass".
## Response Format (strict JSON) ## Response Format (strict JSON)
+13
View File
@@ -0,0 +1,13 @@
"""multi_true rubric 已松绑为题干限定判据。"""
from pathlib import Path
_PROMPT = Path(__file__).resolve().parents[2] / "store" / "prompts" / "question_gen" / "gate_multi_true.md"
def test_rubric_uses_qualifier_constraint():
text = _PROMPT.read_text(encoding="utf-8")
# 新 rubric 必须提到"题干限定下同时为真才 fail"qualifier / under the question's constraints
assert "under the question" in text.lower() or "qualifier" in text.lower()
# 必须显式放行"有局部真实证据但在限定下为假"的干扰项
assert "partial" in text.lower() or "locally" in text.lower()