From 1d2385291a742f23bcfebcda1700ef31f1f3136b Mon Sep 17 00:00:00 2001 From: iomgaa Date: Sat, 11 Jul 2026 09:33:53 -0400 Subject: [PATCH] docs(wiki): record Spec-1 harness-eval finding (all metrics pass) --- research-wiki/findings/eval-spec1check.md | 22 ++++++++++++++++++++++ research-wiki/graph/edges.json | 5 +++++ research-wiki/index.md | 5 +++-- research-wiki/log.md | 2 ++ 4 files changed, 32 insertions(+), 2 deletions(-) create mode 100644 research-wiki/findings/eval-spec1check.md diff --git a/research-wiki/findings/eval-spec1check.md b/research-wiki/findings/eval-spec1check.md new file mode 100644 index 0000000..7e6bb0b --- /dev/null +++ b/research-wiki/findings/eval-spec1check.md @@ -0,0 +1,22 @@ +--- +type: finding +node_id: finding:eval-spec1check +title: "Harness 评估: Spec-1 修复验证 (infer_spec1check)" +date: 2026-07-11 +--- + +# Harness 评估: Spec-1 修复验证 (infer_spec1check) + +**运行信息**: run_id=infer_spec1check（24 题抽样，复用已有运行）；对照基线 infer_adhoc（902 题）。评估时间 2026-07-11。 + +**硬性指标**（wiki 无注册 metric 实体，按 Spec-1 验收标准直查 SQL）： + +| 指标 | 基线 | 当前 | 阈值 | 判定 | +|------|------|------|------|------| +| 异常终止（parse_error+error+budget） | 5/902 (0.55%) | **0/24** | =0 | ✅ pass | +| view_node 输出含 [实体] 区块 | 0%（能力不存在） | **24/24 (100%)** | >0 | ✅ pass | +| 同题正确率 | 17/26 (65.4%，含 2 重复行) | 15/24 (62.5%) | 无回归信号（n=24 噪声范围） | ✅ pass（advisory） | + +**综合判定**: 通过。Spec-1 两个硬目标（T8 归零、实体可见性）直接验证成立；正确率差异在小样本噪声内。语义评估（LlmJudge）跳过——wiki 尚无注册 rubric。 + +**备注**: 本项目首次 harness-eval，schemas/metrics 实体体系尚未建立；Spec-3 出题管线的 structured-logging 阶段将补齐该体系。 diff --git a/research-wiki/graph/edges.json b/research-wiki/graph/edges.json index 829c1d8..488ecf1 100644 --- a/research-wiki/graph/edges.json +++ b/research-wiki/graph/edges.json @@ -100,6 +100,11 @@ "id": "plan:agent-runtime-fixes-plan", "label": "Spec-1 Agent 执行环境修复实现计划", "type": "plan" + }, + { + "id": "finding:eval-spec1check", + "label": "Harness 评估: Spec-1 修复验证 (infer_spec1check)", + "type": "finding" } ], "links": [ diff --git a/research-wiki/index.md b/research-wiki/index.md index fc69055..7cbcc10 100644 --- a/research-wiki/index.md +++ b/research-wiki/index.md @@ -1,6 +1,6 @@ # Research Wiki 索引 -> 自动生成，更新时间：2026-07-11 12:06 UTC +> 自动生成，更新时间：2026-07-11 13:33 UTC ## design (21) - [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design` @@ -25,9 +25,10 @@ - [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth` - [赛题生成工具设计（Question Generation Synthesis）](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design` -## finding (2) +## finding (3) - [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy` - [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis` +- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check` ## plan (19) - [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm` diff --git a/research-wiki/log.md b/research-wiki/log.md index 7f6d105..eb07d1e 100644 --- a/research-wiki/log.md +++ b/research-wiki/log.md @@ -45,3 +45,5 @@ - [2026-07-11 12:06 UTC] 新增 plan: Spec-1 Agent 执行环境修复实现计划 (plan:agent-runtime-fixes-plan) - [2026-07-11 12:06 UTC] 新增边: plan:agent-runtime-fixes-plan --implements--> design:agent-runtime-fixes - [2026-07-11 12:06 UTC] 重建索引: 42 篇页面 +- [2026-07-11 13:33 UTC] 新增 finding: Harness 评估: Spec-1 修复验证 (infer_spec1check) (finding:eval-spec1check) +- [2026-07-11 13:33 UTC] 重建索引: 43 篇页面