feat: loosen multi_true gate to qualifier-scoped correctness

This commit is contained in:
2026-07-14 14:33:22 -04:00
parent b1f15ddb3a
commit 76f719018c
2 changed files with 24 additions and 4 deletions
+11 -4
View File
@@ -16,10 +16,17 @@ You are checking whether a multiple-choice question has **more than one plausibl
## Instructions
1. Read the source material and the question carefully.
2. For each option, assess whether it could be considered a correct or plausible answer given the source.
3. If exactly ONE option is clearly correct → verdict "pass".
4. If TWO or more options are plausibly correct → verdict "fail".
1. Read the source material and the question carefully. Note the question's
explicit qualifiers (subject, timing, manner, object).
2. For each option, assess whether it is **fully correct under the question's
qualifiers** — not merely whether it has some partial or local support in
the source.
3. A good hard-negative distractor MAY have partial/local evidence in the video
yet be FALSE under the question's constraints. Such an option is NOT a
second correct answer — do NOT fail the question for it.
4. Verdict "fail" ONLY IF two or more options are each fully correct under the
question's qualifiers (a genuine ambiguity).
5. Otherwise verdict "pass".
## Response Format (strict JSON)
+13
View File
@@ -0,0 +1,13 @@
"""multi_true rubric 已松绑为题干限定判据。"""
from pathlib import Path
_PROMPT = Path(__file__).resolve().parents[2] / "store" / "prompts" / "question_gen" / "gate_multi_true.md"
def test_rubric_uses_qualifier_constraint():
text = _PROMPT.read_text(encoding="utf-8")
# 新 rubric 必须提到"题干限定下同时为真才 fail"qualifier / under the question's constraints
assert "under the question" in text.lower() or "qualifier" in text.lower()
# 必须显式放行"有局部真实证据但在限定下为假"的干扰项
assert "partial" in text.lower() or "locally" in text.lower()