docs(wiki): record Spec-1 harness-eval finding (all metrics pass)
This commit is contained in:
@@ -0,0 +1,22 @@
|
|||||||
|
---
|
||||||
|
type: finding
|
||||||
|
node_id: finding:eval-spec1check
|
||||||
|
title: "Harness 评估: Spec-1 修复验证 (infer_spec1check)"
|
||||||
|
date: 2026-07-11
|
||||||
|
---
|
||||||
|
|
||||||
|
# Harness 评估: Spec-1 修复验证 (infer_spec1check)
|
||||||
|
|
||||||
|
**运行信息**: run_id=infer_spec1check(24 题抽样,复用已有运行);对照基线 infer_adhoc(902 题)。评估时间 2026-07-11。
|
||||||
|
|
||||||
|
**硬性指标**(wiki 无注册 metric 实体,按 Spec-1 验收标准直查 SQL):
|
||||||
|
|
||||||
|
| 指标 | 基线 | 当前 | 阈值 | 判定 |
|
||||||
|
|------|------|------|------|------|
|
||||||
|
| 异常终止(parse_error+error+budget) | 5/902 (0.55%) | **0/24** | =0 | ✅ pass |
|
||||||
|
| view_node 输出含 [实体] 区块 | 0%(能力不存在) | **24/24 (100%)** | >0 | ✅ pass |
|
||||||
|
| 同题正确率 | 17/26 (65.4%,含 2 重复行) | 15/24 (62.5%) | 无回归信号(n=24 噪声范围) | ✅ pass(advisory) |
|
||||||
|
|
||||||
|
**综合判定**: 通过。Spec-1 两个硬目标(T8 归零、实体可见性)直接验证成立;正确率差异在小样本噪声内。语义评估(LlmJudge)跳过——wiki 尚无注册 rubric。
|
||||||
|
|
||||||
|
**备注**: 本项目首次 harness-eval,schemas/metrics 实体体系尚未建立;Spec-3 出题管线的 structured-logging 阶段将补齐该体系。
|
||||||
@@ -100,6 +100,11 @@
|
|||||||
"id": "plan:agent-runtime-fixes-plan",
|
"id": "plan:agent-runtime-fixes-plan",
|
||||||
"label": "Spec-1 Agent 执行环境修复实现计划",
|
"label": "Spec-1 Agent 执行环境修复实现计划",
|
||||||
"type": "plan"
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "finding:eval-spec1check",
|
||||||
|
"label": "Harness 评估: Spec-1 修复验证 (infer_spec1check)",
|
||||||
|
"type": "finding"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"links": [
|
"links": [
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Research Wiki 索引
|
# Research Wiki 索引
|
||||||
|
|
||||||
> 自动生成,更新时间:2026-07-11 12:06 UTC
|
> 自动生成,更新时间:2026-07-11 13:33 UTC
|
||||||
|
|
||||||
## design (21)
|
## design (21)
|
||||||
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
||||||
@@ -25,9 +25,10 @@
|
|||||||
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
||||||
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
||||||
|
|
||||||
## finding (2)
|
## finding (3)
|
||||||
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
||||||
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
||||||
|
- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check`
|
||||||
|
|
||||||
## plan (19)
|
## plan (19)
|
||||||
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
||||||
|
|||||||
@@ -45,3 +45,5 @@
|
|||||||
- [2026-07-11 12:06 UTC] 新增 plan: Spec-1 Agent 执行环境修复实现计划 (plan:agent-runtime-fixes-plan)
|
- [2026-07-11 12:06 UTC] 新增 plan: Spec-1 Agent 执行环境修复实现计划 (plan:agent-runtime-fixes-plan)
|
||||||
- [2026-07-11 12:06 UTC] 新增边: plan:agent-runtime-fixes-plan --implements--> design:agent-runtime-fixes
|
- [2026-07-11 12:06 UTC] 新增边: plan:agent-runtime-fixes-plan --implements--> design:agent-runtime-fixes
|
||||||
- [2026-07-11 12:06 UTC] 重建索引: 42 篇页面
|
- [2026-07-11 12:06 UTC] 重建索引: 42 篇页面
|
||||||
|
- [2026-07-11 13:33 UTC] 新增 finding: Harness 评估: Spec-1 修复验证 (infer_spec1check) (finding:eval-spec1check)
|
||||||
|
- [2026-07-11 13:33 UTC] 重建索引: 43 篇页面
|
||||||
|
|||||||
Reference in New Issue
Block a user