eval: 准备正向采纳的真实运行恢复与质量评估
新增单份串行启动入口、首个正向运行杀进程恢复证据、逐候选审阅导出与账目统计；参考只读副本分数对照。456项免费测试通过，尚未启动三份真实验收。同步结构说明与当前工作状态。
This commit is contained in:
+1
-1
@@ -104,4 +104,4 @@
|
||||
|
||||
## 下一步
|
||||
|
||||
任务书 16（flow 第 3 部分：实现到采纳，含实验员、核验员提示词）已写，等用户另开会话执行。
|
||||
任务书 16 正在执行：第二块代码已完成独立审阅；第三块的 `eval/forward` 运行、导出、统计入口及结构文档已备好，尚未启动这三份真实研究。下一步由父会话逐份启动并监控，其中一份首个正向运行中杀控制进程、恢复同份；完成后导出逐候选审阅包、五项质量表和沿用任务书 15 的路线对照。真实验收未完成，不记入“已完成”。
|
||||
|
||||
+37
-4
@@ -3,6 +3,7 @@
|
||||
这份文件说明代码分成哪些模块、模块之间怎么依赖、流程按什么步骤推进、账本记哪些记录。研究方法本身以 `docs/方法.md` 为准；这里只说代码怎么组织来实现它。
|
||||
|
||||
来源：
|
||||
|
||||
- 用户 2026-10-01 认可"按负责一件事来切"、下面这套模块与可替换点，以及步骤清单和账本记录种类。
|
||||
- 用户 2026-10-01 同意：提交的字段定义放 vocab，submissions 只判收不收；vocab 可放规范序列化和哈希两个函数，上限调到 600 行；执行后实际 1030 行，用户同意上限改为 1100 行；任务书 09 加对号清单时用户 2026-10-03 同意改为 1150 行，vocab 只放对号清单，任务材料的其余类型放 materials。
|
||||
- 用户 2026-10-02 同意："全部只追加"：账本用一张只追加的表，全部记录都不能改。
|
||||
@@ -121,6 +122,7 @@
|
||||
| 4. 消融到收尾 | ⑬–⑱ 与研究结束 | 解释冻结早于任何一格启动；每格只结算一次 | 选格输入和 15-real-1、16-real-1 比；最后整份回放 16-real-1，当四部分合起来的验收 |
|
||||
|
||||
**怎么验证**（用户 2026-10-03 定）：一律用真 Jev、真 Docker，不做假的 Jev、Docker、任务，flow 不为此开替换口子。同样的输入真实跑几次，结果都符合要求就算通过；真实运行里没出现过的情况不专门造出来测。用户原话："我建议用一律用真 Jev、真 Docker。因为其实我们现在是在做科学研究，不是在开发程序……我们只需要重复运行多次"。模型一侧两种都做：
|
||||
|
||||
- 回放：用本机假模型服务，把参考项目真实记录里模型交的东西按原顺序转成新格式再交一遍，走真实的 Pi。新 Jev 的判断可能和当时不同，两边走向分开以后不再对照，按方法判断分开的地方谁对。
|
||||
- 真模型：用配置里的模型真实跑几次。
|
||||
|
||||
@@ -529,6 +531,7 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- 消融实现：这一格已有公开的消融核验（核验不通过、重做）时，和修复一样从上一次交的这一格代码接着改（用户 2026-10-03 定）：工作目录写"上一次交的这一格代码的副本"，能读的版本里加上它，完整代码照旧列出作对照；另放"退回意见"：最近一次核验里关错了的因子和核验员写的依据，写法同修复的退回意见。
|
||||
- 指标：冻结之后一律用全名"候选编号.短名"：已登记指标清单、对号清单的 `reusableMetrics`、实验员材料的"要实现的指标定义"（候选代码在结果文件里也用全名当键）。提案、回应、统一的对号清单里，已占用的指标编号只有任务的指标、`answer-path` 和 `safeguard-triggers` 两个保留键（用户 2026-10-03 定）；已登记的全名带 `.`，新登记的短名不带，不用列进来。
|
||||
- `Board`（类）：materials 读账本的那一份，构造参数是账本记录数组。flow 的"下一步做什么"是只拿得到记录数组的纯函数，它和 flow 的三个程序步骤都经这一份读账本，不另写一份（任务书 14 第 2.4 节）。flow 用到的：`stages`、`stage`（阶段的轮次、名单、关没关）、`published`（已公开的某种提交，最后一版为准）、`decision`（某轮某种决定）、`proposal`、`proposalsOf`、`authorsOf`（候选合并了哪些原提案、作者按编号排）、`answerOf`（冻结出候选的最后一份回应或统一）、`protocol`、`protocols`、`carried`（留池候选）、`baselineOf`（第 r 轮的本批基线）、`baselineScore`、`runStates`。任务书 14 写的是导出 `baselineOf`、`baselineScore` 两个函数，实际导出了整个类，原因是纯函数只拿得到记录数组，要用的远不止这两样（这一处偏离待用户确认）。
|
||||
- `protocolMetrics(board, protocol)`：把冻结协议自定义及引用的指标合成原登记定义，实验员材料与 flow 问 Jev、正式运行共用（来源：任务书 16 第 2.2、2.4 节）。
|
||||
- `finalEntries(ledger)`：按账本取的最终有效的入口（各人最后一版入口清单里的每个入口，带作者和正文）；flow 分配入口时用，和认领的对号清单是同一份（任务书 12 第三部分）。
|
||||
- `queryMaterial(material, id)`：只在这份材料的记录里按编号找，交回全文；找不到交回 null（agents 的查询回调据此答"看不到"）。
|
||||
- `readPrompts(dir)`：读 `roles/<角色>.md` 和 `tasks/<提交种类>.md`，交回文字和每个文件原始字节的 sha256；研究开始时 flow 把哈希冻结进研究配置记录（方法.md 4.1）。
|
||||
@@ -537,6 +540,8 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- `TaskRequest`、`StudyFacts`、`CodeChanges`、`Prompts`、`PromptFile`（类型）：flow 传进来的东西，全部必填；`CodeChanges` 与 workspace 的比较结果同形（照 jev `codeHunks` 的先例，不引用 workspace）。
|
||||
- 账本里只存成 Json 的内容的形状（用户 2026-10-03 决定 2：materials 定义，flow 照着写，ledger 不改）：`StudyRecord`（研究配置记录：配置、步骤清单、提示词每个文件的哈希、镜像编号、任务事实、种子；配置和任务事实存 Json，flow 读回时用 `checkConfig`、`checkTaskFacts` 再检查，任务书 12 第三部分）、`FrozenProtocol`（冻结协议：方法.md 4.3 ⑥ 列的因子、保障、指标、正文、实现条款，加轮次、出发的基线、合并了哪些原提案；不带答复，主体由 submissions 的 `protocolBody` 拼，指标编号是全名）、`LocationTable`（代码记录的位置表）、`DecisionKind`、`DECISION_KINDS`、`DecisionShapes`（六种决定：入口分配、判重合并、选批、采纳、选格、押的因子，输入输出用 rules 的类型；没采纳时也写一条采纳决定）、`DedupOutput`（判重合并的输出）、`Settlement`（结算）。
|
||||
- `freezeProtocol(body, candidate, round, baseline, proposals)`：flow 写冻结协议时调用。输入 submissions 交回的冻结协议主体、候选编号、轮次、基线、合并了哪些原提案，交回 `FrozenProtocol`；新登记的假设专用指标改用全名"候选编号.短名"（例如 `r2-c1.restart-gain`），引用的本来就是全名，照抄。加前缀只在这一处：同一轮的研究者互相看不到提案，会起同一个短名、定义却不同，候选编号判重合并之后才有，所以只能在冻结时加（用户 2026-10-03 定）。
|
||||
- `experimenterId(candidate)`、`verifierId(candidate)`：每个候选固定一位实验员、一位核验员，修复及再次核验沿用编号（来源：任务书 16 第 2.4 节）。
|
||||
- `FORWARD_NOT_RUN`、`forwardNotRunEvent(候选, 原因)`：正向候选修复用完仍未通过核验时的“不跑”事件，`objects` 只有该候选；不扣次（来源：任务书 16 第 2.2 节）。
|
||||
- 事件：`CELL_NOT_RUN`、`cellNotRunEvent(格, 原因)`：一格消融核验不通过、重做次数用完而不跑时，flow 用它写事件记录（`objects` 只有这一格），结算记证据不足（方法.md 4.3 ⑯、6.6）。
|
||||
- 编号规则：`ORIGINAL_BASELINE`、`researcherId`（第 i 位研究者，从 0 数，编号 `researcher-<i+1>`，任务书 12 第三部分）、`entryId`、`proposalId`、`critiqueId`、`candidateId`、`factorId`、`cellId`、`splitCell`、`explanationId`、`callId`，以及协议内的局部编号 `factorKey`（p1…）、`clauseKey`（c1…）。入口编号按作者一直往后编，讨论时整份替换也不复用旧编号。
|
||||
|
||||
@@ -581,7 +586,7 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
|
||||
### flow
|
||||
|
||||
下面是 `src/flow/index.ts` 导出的全部内容，以后 plugin 也只用它。flow 只经各模块的 `index.ts` 用它们；正式代码里没有为检验写的参数、环境变量、回放模式或假实现（CLAUDE.md"测试"）。这一节对应 flow 第 1、2 部分（任务书 12、14）：骨架、研究开始、第零轮，以及第 r 轮从开轮到选批；走到第 r 轮的 `implementation` 停下，原因"实现还没写（任务书 16）"。轮次按"第 r 轮"写，不写死 1；现在走得到的只有第 1 轮。
|
||||
下面是 `src/flow/index.ts` 导出的全部内容，以后 plugin 也只用它。flow 只经各模块的 `index.ts` 用它们；正式代码里没有为检验写的参数、环境变量、回放模式或假实现（CLAUDE.md"测试"）。这一节对应 flow 第 1–3 部分（任务书 12、14、16）：骨架、研究开始、第零轮，以及第 r 轮从开轮到采纳；采纳决定写完后走到 `listCells` 停下，原因"消融还没写（任务书 17）"。轮次按"第 r 轮"写，不写死 1；现在走得到的只有第 1 轮。
|
||||
|
||||
- `runStudy(input)`：推进一份研究，直到停下。输入三样，都必填：研究目录；配置（新研究交 `checkConfig` 检查过的配置，接着跑一份已有的研究写 null，以账本里冻结的那份为准；两者对不上就报错）；"停下"信号。每次启动走同一条路：打开账本；新研究先冻结配置，本机已有 `inforsi-<研究编号>` 的镜像就说明研究编号用过了，报错停下，要换一个编号（编号重复的新研究会把旧研究遗留的同名容器当成自己的运行）；已有研究的，核对冻结的配置、镜像还在、提示词哈希没变（开发入口开着时提示词可以变）；给开始了没结束的模型调用补写结束（出错、"进程中断"、墙钟时间测不到）；按容器名问 Docker 核对登记了没结果的正式运行（没启动的写"没启动"退还名额，启动了的补写确认启动、等它、判分、写结果，问不到的写"状态未知"事件并停下）；然后进主循环：读账本 → 问"下一步做什么" → 做这一步 → 回到开头。主循环里 `docker run` 没启动的，写完"没启动"结果（退还名额、记下 Docker 报错原文）就停下，停下原因带上报错原文，本进程不重跑；下次启动时照旧起新编号重跑。恢复时补写的"没启动"不停下。
|
||||
- `StudyInput`（类型）：上面三样输入。
|
||||
@@ -589,7 +594,7 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
|
||||
研究目录里放：`ledger.sqlite` 账本；`repo.git` 代码版本的裸仓库；`runs/<运行编号>/` 一次正式运行（取出的代码、运行目录、日志、判分目录、校验器日志）；`calls/<调用编号>/` 一次模型调用（工作目录、会话文件、能读的代码版本）。取出代码、建运行目录和判分目录时目标都是新建的空目录。容器名 `inforsi-<研究编号>-<运行编号>`，判分容器另加 `-judge`。同一份研究的全部运行用同一个种子：研究编号经 vocab 的 `hash` 取前 8 位十六进制转成整数，冻结进研究配置记录。
|
||||
|
||||
**每一步以哪条记录为准算做完**（任务书 12 第 2.2 节、任务书 14 第 2.2 节）。步骤的先后只写在 flow 的"下一步做什么"一个纯函数里；后面两部分每加一步，这张表加一行。做完的轮次整轮跳过，第一个没做完的轮次里第一个没做完的步骤就是下一步。
|
||||
**每一步以哪条记录为准算做完**（任务书 12 第 2.2 节、任务书 14 第 2.2 节、任务书 16 第 2.3 节）。步骤的先后只写在 flow 的"下一步做什么"一个纯函数里；后续每加一步，这张表加一行。做完的轮次整轮跳过，第一个没做完的轮次里第一个没做完的步骤就是下一步。
|
||||
|
||||
| 步骤（config 的 `STUDY_STEPS`、`ROUND_STEPS`） | 看到哪条记录算做完 |
|
||||
| --- | --- |
|
||||
@@ -602,11 +607,16 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
| `dedupe` 判重合并 | 本轮的判重合并决定（`decision`，种类 `dedup`） |
|
||||
| `freezeProtocol` 冻结协议 | 本轮每个新候选都有冻结协议记录（`protocol`）；全部候选在同一个事务里写 |
|
||||
| `selectBatch` 选批 | 本轮的选批决定（`decision`，种类 `batch`） |
|
||||
| `implementation` | 这一轮实现阶段的关闭记录 |
|
||||
| `verification`、`repair` | 每个入选候选最近一次核验通过，或修复次数用完且有“不跑”事件；中间每次核验、修复阶段各有开关记录 |
|
||||
| `forwardRun` | 每个通过核验的候选最近一次正向运行有结果，而且不是“没启动” |
|
||||
| `publish`、`informationAccount` | 不写记录：`forwardRun` 做完就算做完；公开靠本轮采纳决定，信息账从预测和结果现算 |
|
||||
| `adopt` | 本轮的采纳决定（`decision`，种类 `adopt`），没有过线者也写，`adopted` 为 null |
|
||||
| 按配置跳过的步骤 | 一条"按配置跳过"事件（`skippedByConfig`），写明步骤（第 r 轮的步骤写 `r<轮>-<步骤>`）和 config 给的原因 |
|
||||
|
||||
研究者阶段没开就开，名单是全部研究者（本轮名单就是冻结的配置）；每个任务都有被收下的提交之后才关。
|
||||
|
||||
**每个阶段谁做、对哪个对象**（任务书 14 第 2.3 节）。一个任务就是"一个人对一个对象"，交一种提交。同一阶段里，一个人的几个任务按下表列出的先后在一个接一个的会话里跑，不同的人同时跑（任务书 12 第 2.6 节）。一个任务做完，以"这个人在这个阶段对这个对象有一份被收下的提交"为准。
|
||||
**每个阶段谁做、对哪个对象**（任务书 14 第 2.3 节、任务书 16 第 2.2、2.4 节）。一个任务就是"一个人对一个对象"，交一种提交。同一阶段里，一个人的几个任务按下表列出的先后在一个接一个的会话里跑，不同的人同时跑（任务书 12 第 2.6 节）。一个任务做完，以"这个人在这个阶段对这个对象有一份被收下的提交"为准。
|
||||
|
||||
| 阶段 | 任务 | 对象 |
|
||||
| --- | --- | --- |
|
||||
@@ -616,10 +626,13 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
| 挑刺 | 每位研究者 × 本轮每条不是自己写的原提案（监督者定，照参考项目：方法没写谁挑谁） | 原提案 |
|
||||
| 回应与统一 | 每个新候选一个任务：只有一条原提案的，由作者交"回应"；几条合成的，由其中研究者编号最小的作者交"统一"（方法.md 4.3 ③⑥） | 候选 |
|
||||
| 第二次预测 | 每位研究者 × 候选池里每个候选（新候选加留池候选，方法.md 4.3 ①⑦） | 候选 |
|
||||
| 实现 | 每个入选候选一个实验员，从本批基线取工作目录，不同候选同时实现 | 候选 |
|
||||
| 核验 | 每个待核验候选一个核验员；首次核验本批全部，之后只核验刚修复的候选 | 候选 |
|
||||
| 修复 | 只给核验不通过的候选，交原实验员，从上次提交的代码接着改 | 候选 |
|
||||
|
||||
没做完的任务派不派，按这个任务最近一次调用怎么结束的来定（任务书 12 第 2.3 节；第 2 部分从"一个人"细到"一个人对一个对象"，任务书 14 第 2.3 节）。认是哪个任务，看"模型调用开始"里存的材料带的对象。没有调用、被用户停下、出错（含"进程中断"）都派；没交的，开发入口关着就停下（交不出合格提交按内部错误，方法.md 4.3 ②），开着就再派。能派的先全派。一次派发里有任务没交上（没交或出错），不再开新的会话，已经在跑的不打断，等这次派发的会话全部结束后停下。
|
||||
|
||||
一个任务交上来之后才知道的东西（submissions 的 `known`，任务书 14 第 2.3 节）：提案是本人在本阶段已经收下几条（替换时不算被替换的那份）；回应是作者那条原提案；其余种类没有。
|
||||
一个任务交上来之后才知道的东西（submissions 的 `known`，任务书 14 第 2.3 节）：提案是本人在本阶段已经收下几条（替换时不算被替换的那份）；回应是作者那条原提案；实现、修复交代码快照结果（见下文）；其余种类没有。
|
||||
|
||||
提案和挑刺一个任务可以交几份，每份一个编号：提案用 materials 的 `proposalId`（轮次、作者、第几份），挑刺用 `critiqueId`（目标原提案、作者、第几份），收下时由 flow 按这个任务已收下几份往后分配，回给模型的提交编号就是它（任务书 14 第 2.3 节）。只交一份的种类，提交编号是"阶段.本人.种类.对象"。不写 `replaces` 是交新的一份，只交一份的种类已经交过就退回；写了 `replaces`，必须是这个任务已收下的某一份，整份替换它。
|
||||
|
||||
@@ -641,6 +654,15 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- 种子用 rules 的 `decisionSeed`，由研究编号、轮次、版本算。同一轮里要随机的决定各用一个版本，选批用 1，选格预留 2，两次抽取不会用同一串随机数（方法.md 第九部分；选格由第 4 部分接上）。
|
||||
- 规则取步骤清单里 `selectBatch` 那一步的，调 `runBatchRule`。写一条决定：种类 `batch`，规则名、版本和参数取 `runBatchRule` 交回的。
|
||||
|
||||
**实现、核验、修复与采纳**（来源：任务书 16 第 2.2–2.4 节）：
|
||||
|
||||
- 阶段编号：`r<轮>-implementation`、`r<轮>-verification-1`、`r<轮>-repair-1`、`r<轮>-verification-2`……第几次从账本数，不另存。实现阶段关后才核验，核验阶段关后才派修复。修复次数用完仍不通过，用 `forwardNotRunEvent` 写清原因，不登记运行、不扣次。
|
||||
- 角色各取配置中的模型及自己的角色提示词。实验员工作目录先取版本，提交时用 workspace 存版本；相对本批基线比较、用 `codeHunks` 切块、查 `protectedChanges`，拼成 submissions 的 `known.code`。收下的提交和代码版本、位置表在同一事务写入；指标引用用 `protocolMetrics` 换成原登记定义。
|
||||
- 核验及修复材料的 `changes` 都相对本批基线：修复看基线到上次提交，修后核验看基线到修复结果；不新增上次到此次的材料。核验通过与否用 `verificationOutcome` 判。
|
||||
- 实验员的轻量检查接 `SessionTask.lightCheck`：tasks 给定义、runner 执行、tasks 读结果；每次模型调用分别按 `lightChecks.maxCount` 计次、按 `seconds` 限时，会话保留记录，不记入正式运行账本。
|
||||
- 通过核验的候选用既有 `formalRun`，用途 `forward`；先登记占名额，确认启动才扣次，再独立判分记结果。**一批内一个接一个跑**，避免抢 CPU 改变按秒停下的程序成绩。没启动的处理同基线：本进程停下，下次启动起新编号。
|
||||
- 本批全部有结果或“不跑”事件后，用 rules 的 `adopt`（内部用 `adoptionLine`），只传有效实测分数、本批基线分数和任务的方向、ε、已知最好；写种类 `adopt` 的决定，即使 `adopted` 为 null 也写。materials 据此公开本轮正向运行。
|
||||
|
||||
检验命令（`eval/`，真实运行，要 Docker；带 `*` 的会花模型或 Jev 的钱。输出都在仓库外 `/home/iomgaa/inforsi-eval/<日期-名字>/`，名字也当研究编号用，只能小写、不能重复）：
|
||||
|
||||
| 命令 | 用途 |
|
||||
@@ -659,3 +681,14 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
| `npm run eval:r1:jevcheck -- <检验目录>` * | 对每次导出的运行问 Jev 三类问题（任务书 14 第 3.2 节第 3 步）：判重对不对、挑刺有没有被回应、提案对应研究路径的哪一步；原始概率写进 `jevcheck.json` |
|
||||
| `npm run eval:r1:stats -- <检验目录>...` | 第 1 轮的数字和过程（任务书 14 第 3.2 节第 4 步），另查阶段开关成对、冻结协议写在回应阶段关闭之后、停下原因是"实现还没写"；每个目录写一份 `stats.json` |
|
||||
|
||||
任务书 16 新增检验命令（来源：任务书 16 第三、四、五部分）：
|
||||
|
||||
| 命令 | 用途 |
|
||||
| --- | --- |
|
||||
| `npm run eval:forward:run -- new <研究编号> <全新绝对输出目录>` * | 配置照 `eval/round-zero/research.json`，只换编号；一份研究跑到采纳后停下，记录进程起止、账本观察时间及机器快照；异常保留现场并停下 |
|
||||
| `npm run eval:forward:run -- kill <研究编号> <全新绝对输出目录>` * | 同上；首个正向 `runStarted` 后、无结果且容器仍运行时杀控制进程，保存容器 ID 和启动时间，恢复同份一次，不另开研究 |
|
||||
| `npm run eval:forward:run -- resume <研究编号> <原输出目录>` * | 查明服务错误后手动续跑同份，追加进程/观察记录；也可用 `eval:study -- --resume --dir <输出目录>/study` |
|
||||
| `npm run eval:forward:export -- <研究目录> <全新绝对审阅目录>` | 候选协议全文、引用指标定义、基线/最终/各次代码、位置表及 concerns、每次核验/修复、退回与调用材料、会话位置 |
|
||||
| `npm run eval:forward:stats -- <全新绝对统计目录> <参考副本根目录> <检验目录>...` | 五项表中的可计算字段：预测与实测、参考首轮成绩、核验/修复次数、账目、轻量检查、杀进程三项检查；不自动作质量判断 |
|
||||
|
||||
三份研究须逐个启动，上一份结束并核查完再起下一份，同机不与别的研究并跑；进程起止见 `launches.jsonl`，机器观察见 `machine.jsonl`，杀进程前后见 `container-*.json`，扣次与后续结果见统计的 `kill`。快照只是观察证据，不声称追踪任意进程。结果放仓库外，不提交。独立审阅按候选分两步：先评实现忠实度，再给历史核验及其当时代码评误判和修复范围。路线分类、完整路线的实测减合并 q5、A–J 表由阅读后填写；复用 `eval/route/` 的导出、条目、思考、分位数核查及 Jev 四问，不另造问卷或判别器。参考项目五份账本用已有只读副本，保留无效数值和重复尝试，并注明参考提示词含题目做法。
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import {
|
||||
Board,
|
||||
protocolMetrics,
|
||||
type TaskMaterial,
|
||||
} from "../../src/materials/index.ts";
|
||||
import { checkoutVersion, compareVersions } from "../../src/workspace/index.ts";
|
||||
import { candidateHistory, jsonFile } from "../lib/forward.ts";
|
||||
import { ofKind, readLedger } from "../lib/ledger.ts";
|
||||
|
||||
// 任务书16五项质量评估的候选审阅材料；不判路线、不代替全新上下文逐条阅读。
|
||||
// node eval/forward/export.ts <研究目录> <全新绝对输出目录>
|
||||
const [study, output, ...extra] = process.argv.slice(2);
|
||||
if (!study || !output || extra.length || output !== resolve(output))
|
||||
throw new Error(
|
||||
"用法：node eval/forward/export.ts <研究目录> <全新绝对输出目录>",
|
||||
);
|
||||
mkdirSync(output); // 不覆盖以前的审阅材料。
|
||||
const records = readLedger(study);
|
||||
const board = new Board(records);
|
||||
const repo = join(study, "repo.git");
|
||||
const batch = board.decision("batch", 1)!;
|
||||
const baseline = board.code(board.baselineOf(1));
|
||||
const plans = ofKind(records, "runPlanned").filter(
|
||||
(p) => p.content.round === 1 && p.content.purpose === "forward",
|
||||
);
|
||||
const index = [];
|
||||
for (const candidate of batch.output.chosen) {
|
||||
const dir = join(output, candidate);
|
||||
mkdirSync(dir);
|
||||
const history = candidateHistory(records, candidate);
|
||||
const candidatePlans = plans.filter((p) => p.content.object === candidate);
|
||||
const started = candidatePlans.filter((p) =>
|
||||
ofKind(records, "runStarted").some((s) => s.content.run === p.content.run),
|
||||
);
|
||||
const protocol = ofKind(records, "protocol")
|
||||
.filter((p) => p.content.candidate === candidate)
|
||||
.at(-1)!;
|
||||
jsonFile(join(dir, "protocol.json"), {
|
||||
record: protocol,
|
||||
resolvedMetrics: protocolMetrics(board, board.protocol(candidate)),
|
||||
});
|
||||
mkdirSync(join(dir, "baseline"));
|
||||
checkoutVersion(repo, baseline.version, join(dir, "baseline"));
|
||||
for (const attempt of history.attempts) {
|
||||
const path = join(dir, `code-${attempt.code.seq}`);
|
||||
mkdirSync(path);
|
||||
checkoutVersion(repo, attempt.code.content.version, path);
|
||||
jsonFile(
|
||||
join(dir, `changes-${attempt.code.seq}.json`),
|
||||
compareVersions(repo, baseline.version, attempt.code.content.version),
|
||||
);
|
||||
}
|
||||
// 首轮里每个候选只正式跑一个已通过版本；无启动的候选仍导出历史，但不冒称有最终通过代码。
|
||||
const firstRun = started[0];
|
||||
const finalCode =
|
||||
firstRun === undefined
|
||||
? null
|
||||
: ofKind(records, "code")
|
||||
.filter((c) => c.content.object === candidate && c.seq < firstRun.seq)
|
||||
.at(-1)!;
|
||||
if (finalCode !== null) {
|
||||
mkdirSync(join(dir, "final"));
|
||||
checkoutVersion(repo, finalCode.content.version, join(dir, "final"));
|
||||
}
|
||||
const finalAttempt =
|
||||
finalCode === null
|
||||
? null
|
||||
: history.attempts.find((a) => a.code.seq === finalCode.seq)!;
|
||||
jsonFile(join(dir, "final-implementation.json"), finalAttempt);
|
||||
jsonFile(join(dir, "attempts.json"), history.attempts);
|
||||
jsonFile(join(dir, "verifications.json"), history.verifications);
|
||||
const calls = ofKind(records, "callStarted").filter((c) => {
|
||||
const material = c.content.materials as TaskMaterial;
|
||||
return (
|
||||
material.object === candidate &&
|
||||
["implementation", "repair", "verification"].includes(material.kind)
|
||||
);
|
||||
});
|
||||
jsonFile(
|
||||
join(dir, "calls.json"),
|
||||
calls.map((c) => ({
|
||||
record: c,
|
||||
sessions: join(study, "calls", c.content.call, "sessions"),
|
||||
})),
|
||||
);
|
||||
jsonFile(
|
||||
join(dir, "rejections.json"),
|
||||
ofKind(records, "rejection").filter(
|
||||
(r) =>
|
||||
r.content.target === candidate &&
|
||||
["implementation", "repair", "verification"].includes(r.content.kind),
|
||||
),
|
||||
);
|
||||
const manifest = {
|
||||
study,
|
||||
candidate,
|
||||
protocolSeq: protocol.seq,
|
||||
baselineVersion: baseline.version,
|
||||
finalCode,
|
||||
ran: started.length > 0,
|
||||
runs: candidatePlans.map((p) => ({
|
||||
plan: p,
|
||||
result:
|
||||
ofKind(records, "runResult").find(
|
||||
(r) => r.content.run === p.content.run,
|
||||
) ?? null,
|
||||
artifacts: join(study, "runs", p.content.run),
|
||||
})),
|
||||
};
|
||||
jsonFile(join(dir, "manifest.json"), manifest);
|
||||
writeFileSync(
|
||||
join(dir, "review.txt"),
|
||||
`任务书16第3.2节：只读本候选，分两步评估，中文直接回复，不写文件。\n` +
|
||||
`第一步只读 protocol.json（冻结协议全文及引用指标原定义）、baseline/、final/、final-implementation.json（正文、位置表、concerns）。\n` +
|
||||
`逐条实现条款、因子、保障检查漏做、协议之外的改进、位置表是否准确，每条附代码文件和行号。先形成独立结论，再打开核验历史。\n` +
|
||||
`第二步读 verifications.json、attempts.json、code-<序号>/、changes-<序号>.json；每次核验对照其中code指向的那一版，不拿最终修好版本倒判历史。判断该拒却通过、不该拒却拒绝；修复是否只改指出处。\n` +
|
||||
`需要时读 calls.json 和 rejections.json；模型材料和原始会话都可追溯。不把提交格式退回当成核验失败。\n` +
|
||||
`答案路径只核对每个可能交出最终答案分支的标签，不预测实际走哪个分支。报告先写实现忠实度，再写逐次核验误判和修复范围。\n` +
|
||||
`按任务书15附录一路线代号归类冻结协议，另标是否完整高上限路线（起点、连续改进全部圆、多起点或扰动、合法收尾），附协议原文位置。\n` +
|
||||
`本候选${manifest.ran ? "已启动正式运行" : "未启动正式运行；没有final/，只评历史，不列成跑过的候选"}。\n`,
|
||||
);
|
||||
index.push({
|
||||
candidate,
|
||||
ran: manifest.ran,
|
||||
directory: dir,
|
||||
finalVersion: finalCode?.content.version ?? null,
|
||||
});
|
||||
}
|
||||
jsonFile(join(output, "index.json"), index);
|
||||
console.log(`候选审阅包：${output}`);
|
||||
@@ -0,0 +1,259 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { appendFileSync, existsSync, mkdirSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import { forwardAccounts, frozenConfig, jsonFile } from "../lib/forward.ts";
|
||||
import { ofKind, readLedger } from "../lib/ledger.ts";
|
||||
import { launch, sleep, waitForLedger, writeConfig } from "../lib/study.ts";
|
||||
import { watchTimeline } from "../lib/timeline.ts";
|
||||
|
||||
// 任务书16：一次只启动一份；三次由父会话逐个执行。kill 只打首个正向运行，随后恢复同一研究一次。
|
||||
// node eval/forward/run.ts <new|kill|resume> <研究编号> <绝对输出目录>
|
||||
const [mode, name, output, ...extra] = process.argv.slice(2);
|
||||
if (
|
||||
!mode ||
|
||||
!name ||
|
||||
!output ||
|
||||
extra.length ||
|
||||
!["new", "kill", "resume"].includes(mode) ||
|
||||
output !== resolve(output)
|
||||
) {
|
||||
throw new Error(
|
||||
"用法：node eval/forward/run.ts <new|kill|resume> <研究编号> <绝对输出目录>",
|
||||
);
|
||||
}
|
||||
const out = output;
|
||||
const studyDir = join(out, "study");
|
||||
if (mode === "resume") {
|
||||
if (frozenConfig(readLedger(studyDir)).studyId !== name)
|
||||
throw new Error("研究编号与账本不符");
|
||||
} else {
|
||||
if (existsSync(out))
|
||||
throw new Error(`输出目录已存在，不覆盖现场：${out}`);
|
||||
mkdirSync(out, { recursive: true });
|
||||
}
|
||||
const config =
|
||||
mode === "resume"
|
||||
? null
|
||||
: writeConfig(
|
||||
"eval/round-zero/research.json",
|
||||
join(out, "research.json"),
|
||||
name,
|
||||
null,
|
||||
);
|
||||
const inspect = (container: string) =>
|
||||
JSON.parse(
|
||||
execFileSync("docker", ["inspect", container], {
|
||||
encoding: "utf8",
|
||||
}),
|
||||
)[0] as {
|
||||
Id: string;
|
||||
Name: string;
|
||||
Created: string;
|
||||
State: {
|
||||
Running: boolean;
|
||||
StartedAt: string;
|
||||
FinishedAt: string;
|
||||
Status: string;
|
||||
};
|
||||
};
|
||||
function observe(label: string) {
|
||||
appendFileSync(
|
||||
join(out, "machine.jsonl"),
|
||||
`${JSON.stringify({
|
||||
at: new Date().toISOString(),
|
||||
label,
|
||||
processes: execFileSync(
|
||||
"ps",
|
||||
["-eo", "pid,ppid,lstart,args"],
|
||||
{ encoding: "utf8" },
|
||||
),
|
||||
containers: execFileSync(
|
||||
"docker",
|
||||
["ps", "--no-trunc"],
|
||||
{ encoding: "utf8" },
|
||||
),
|
||||
})}\n`,
|
||||
);
|
||||
}
|
||||
function start(path: string | null) {
|
||||
observe(path === null ? "resume" : "start");
|
||||
const study = launch(path, studyDir, {});
|
||||
appendFileSync(
|
||||
join(out, "launches.jsonl"),
|
||||
`${JSON.stringify({ event: "start", pid: study.child.pid, at: new Date().toISOString(), config: path })}\n`,
|
||||
);
|
||||
const interrupt = () => study.child.kill("SIGINT");
|
||||
process.on("SIGINT", interrupt);
|
||||
study.done.then((end) => {
|
||||
process.off("SIGINT", interrupt);
|
||||
appendFileSync(
|
||||
join(out, "launches.jsonl"),
|
||||
`${JSON.stringify({ event: "end", pid: study.child.pid, at: new Date().toISOString(), ...end })}\n`,
|
||||
);
|
||||
});
|
||||
return study;
|
||||
}
|
||||
let active = start(config);
|
||||
try {
|
||||
await waitForLedger(studyDir, active.child);
|
||||
let watching = watchTimeline(
|
||||
studyDir,
|
||||
active.child,
|
||||
join(out, "timeline.jsonl"),
|
||||
);
|
||||
if (mode === "kill") {
|
||||
let hit = false;
|
||||
while (
|
||||
active.child.exitCode === null &&
|
||||
active.child.signalCode === null
|
||||
) {
|
||||
const records = readLedger(studyDir);
|
||||
const plan = ofKind(records, "runPlanned").find(
|
||||
(r) =>
|
||||
r.content.purpose === "forward" &&
|
||||
r.content.round === 1,
|
||||
);
|
||||
if (plan !== undefined) {
|
||||
const run = plan.content.run;
|
||||
if (
|
||||
ofKind(records, "runResult").some(
|
||||
(r) => r.content.run === run,
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
"首个正向运行已经结束，未打中；保留现场，不改杀别的候选",
|
||||
);
|
||||
const started = ofKind(
|
||||
records,
|
||||
"runStarted",
|
||||
).find((r) => r.content.run === run);
|
||||
if (started !== undefined) {
|
||||
// 容器名是正式入口记录在结构文档里的约定，不 import flow 内部文件。
|
||||
const container = `inforsi-${name}-${run}`;
|
||||
const before = inspect(container);
|
||||
jsonFile(
|
||||
join(
|
||||
out,
|
||||
"container-before.json",
|
||||
),
|
||||
before,
|
||||
);
|
||||
if (!before.State.Running)
|
||||
throw new Error(
|
||||
"首个正向容器已不在运行，未打中",
|
||||
);
|
||||
if (
|
||||
ofKind(
|
||||
readLedger(studyDir),
|
||||
"runResult",
|
||||
).some(
|
||||
(r) =>
|
||||
r.content
|
||||
.run ===
|
||||
run,
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
"杀前已有结果，未打中",
|
||||
);
|
||||
jsonFile(join(out, "kill.json"), {
|
||||
run,
|
||||
candidate: plan.content.object,
|
||||
planSeq: plan.seq,
|
||||
startSeq: started.seq,
|
||||
beforeKillSeq:
|
||||
records.at(-1)!.seq,
|
||||
pid: active.child.pid,
|
||||
at: new Date().toISOString(),
|
||||
container,
|
||||
});
|
||||
if (!active.child.kill("SIGKILL"))
|
||||
throw new Error(
|
||||
"未发出 SIGKILL",
|
||||
);
|
||||
const killed = await active.done;
|
||||
await watching;
|
||||
if (killed.signal !== "SIGKILL")
|
||||
throw new Error(
|
||||
"控制进程不是以 SIGKILL 结束",
|
||||
);
|
||||
const atKill = readLedger(studyDir);
|
||||
jsonFile(
|
||||
join(
|
||||
out,
|
||||
"ledger-at-kill.json",
|
||||
),
|
||||
atKill,
|
||||
);
|
||||
if (
|
||||
ofKind(
|
||||
atKill,
|
||||
"runResult",
|
||||
).some(
|
||||
(r) =>
|
||||
r.content
|
||||
.run ===
|
||||
run,
|
||||
)
|
||||
)
|
||||
throw new Error(
|
||||
"杀后核对发现目标运行已经写结果，未打中；保留现场",
|
||||
);
|
||||
const afterKill = inspect(container);
|
||||
jsonFile(
|
||||
join(
|
||||
out,
|
||||
"container-after-kill.json",
|
||||
),
|
||||
afterKill,
|
||||
);
|
||||
observe("after-kill");
|
||||
active = start(null);
|
||||
const afterResume = inspect(container);
|
||||
jsonFile(
|
||||
join(
|
||||
out,
|
||||
"container-after-resume.json",
|
||||
),
|
||||
afterResume,
|
||||
);
|
||||
watching = watchTimeline(
|
||||
studyDir,
|
||||
active.child,
|
||||
join(out, "timeline.jsonl"),
|
||||
);
|
||||
hit = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
await sleep(100);
|
||||
}
|
||||
if (!hit)
|
||||
throw new Error(
|
||||
`没有打中首个正向运行：${JSON.stringify(await active.done)}`,
|
||||
);
|
||||
}
|
||||
await watching;
|
||||
const end = await active.done;
|
||||
jsonFile(join(out, "outcome.json"), end);
|
||||
observe("end");
|
||||
const accounts = forwardAccounts(readLedger(studyDir));
|
||||
jsonFile(join(out, "accounts.json"), accounts);
|
||||
// 杀进程报告由 stats 读原始容器观察和账本重算；即使外部服务失败后人工恢复，也不覆盖原杀进程证据。
|
||||
if (end.code !== 0 || !Object.values(accounts.checks).every(Boolean)) {
|
||||
throw new Error(
|
||||
`没有完成采纳验收，保留现场；查明原因后才能手动 resume：${JSON.stringify(end)}`,
|
||||
);
|
||||
}
|
||||
console.log(
|
||||
`已到采纳后停点：${studyDir}。仍需 export、stats 和独立质量审阅。`,
|
||||
);
|
||||
} finally {
|
||||
if (
|
||||
active.child.exitCode === null &&
|
||||
active.child.signalCode === null
|
||||
) {
|
||||
active.child.kill("SIGINT");
|
||||
await active.done;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
import { existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import type { LedgerRecord } from "../../src/ledger/index.ts";
|
||||
import {
|
||||
Board,
|
||||
validScore,
|
||||
type TaskMaterial,
|
||||
} from "../../src/materials/index.ts";
|
||||
import {
|
||||
forwardAccounts,
|
||||
frozenConfig,
|
||||
jsonFile,
|
||||
scoreBand,
|
||||
} from "../lib/forward.ts";
|
||||
import { finalSubmissions, ofKind, readLedger } from "../lib/ledger.ts";
|
||||
import { lightCheckUsage } from "../lib/light-checks.ts";
|
||||
import { referenceScores } from "../lib/reference-scores.ts";
|
||||
|
||||
// 五项表中可计算的字段。路线与审阅结论故意不自动猜，由父会话按候选审阅包填写报告。
|
||||
// node eval/forward/stats.ts <全新绝对输出目录> <五份参考副本根目录> <检验目录>...
|
||||
const [output, referenceRoot, ...dirs] = process.argv.slice(2);
|
||||
if (
|
||||
!output ||
|
||||
!referenceRoot ||
|
||||
dirs.length === 0 ||
|
||||
output !== resolve(output)
|
||||
)
|
||||
throw new Error(
|
||||
"用法：node eval/forward/stats.ts <全新绝对输出目录> <参考副本根目录> <检验目录>...",
|
||||
);
|
||||
mkdirSync(output);
|
||||
const summaries = dirs.map((dir) => {
|
||||
const studyDir = join(dir, "study");
|
||||
const records = readLedger(studyDir);
|
||||
const board = new Board(records);
|
||||
const config = frozenConfig(records);
|
||||
const accounts = forwardAccounts(records);
|
||||
const batch = board.decision("batch", 1)!;
|
||||
const finals = finalSubmissions(records);
|
||||
const scores = accounts.candidates.map((c) => {
|
||||
const p = board.protocol(c.candidate);
|
||||
const merged = batch.output.candidates.find(
|
||||
(b) => b.id === c.candidate,
|
||||
)!.merged;
|
||||
const secondPredictions = finals.flatMap((s) =>
|
||||
s.content.kind === "secondPrediction" &&
|
||||
s.content.target === c.candidate &&
|
||||
board.stage(s.content.stageId).round === 1
|
||||
? [
|
||||
{
|
||||
researcher: s.content.author,
|
||||
q5: s.content.content.range.q5,
|
||||
seq: s.seq,
|
||||
},
|
||||
]
|
||||
: [],
|
||||
);
|
||||
const authorEstimates = p.proposals.map((id) => {
|
||||
const proposal = finals.find(
|
||||
(s) => s.content.kind === "proposal" && s.content.item === id,
|
||||
)!;
|
||||
const seal = ofKind(records, "seal").find(
|
||||
(s) => s.content.proposal === proposal.seq,
|
||||
)!;
|
||||
return {
|
||||
proposal: id,
|
||||
author: proposal.content.author,
|
||||
range: seal.content.range,
|
||||
proposalSeq: proposal.seq,
|
||||
sealSeq: seal.seq,
|
||||
};
|
||||
});
|
||||
const runs = c.runs.map((r) => {
|
||||
const result = r.results[0] ?? null;
|
||||
const score = validScore(result?.content.judged ?? null);
|
||||
return {
|
||||
run: r.plan.content.run,
|
||||
planSeq: r.plan.seq,
|
||||
started: r.starts.length > 0,
|
||||
result,
|
||||
score,
|
||||
band: scoreBand(score, merged),
|
||||
measuredMinusQ5: score === null ? null : score - merged.q5,
|
||||
};
|
||||
});
|
||||
return {
|
||||
study: config.studyId,
|
||||
candidate: c.candidate,
|
||||
name: p.factors.map((f) => f.name).join("；"),
|
||||
merged,
|
||||
secondPredictions,
|
||||
authorEstimates,
|
||||
runs,
|
||||
adopted: board.decision("adopt", 1)?.output.adopted === c.candidate,
|
||||
verificationCount: c.verificationCount,
|
||||
repairCount: c.repairCount,
|
||||
started: c.started,
|
||||
manualRequired:
|
||||
"路线按冻结协议归类；完整高上限路线单列 measuredMinusQ5；忠实度及误判见独立审阅回复",
|
||||
};
|
||||
});
|
||||
const lightChecks = ofKind(records, "callStarted")
|
||||
.filter((c) =>
|
||||
["implementation", "repair"].includes(
|
||||
(c.content.materials as TaskMaterial).kind,
|
||||
),
|
||||
)
|
||||
.map((c) => ({
|
||||
call: c.content.call,
|
||||
seq: c.seq,
|
||||
...lightCheckUsage(
|
||||
join(studyDir, "calls", c.content.call, "sessions"),
|
||||
config.lightChecks.maxCount,
|
||||
),
|
||||
}));
|
||||
let kill = null;
|
||||
if (existsSync(join(dir, "kill.json"))) {
|
||||
const evidence = JSON.parse(
|
||||
readFileSync(join(dir, "kill.json"), "utf8"),
|
||||
) as { run: string; candidate: string; beforeKillSeq: number };
|
||||
type Container = {
|
||||
Id: string;
|
||||
State: { StartedAt: string; Running: boolean };
|
||||
};
|
||||
const snapshots = ["before", "after-kill", "after-resume"].map(
|
||||
(part) =>
|
||||
JSON.parse(
|
||||
readFileSync(join(dir, `container-${part}.json`), "utf8"),
|
||||
) as Container,
|
||||
);
|
||||
const target = accounts.runs.find(
|
||||
(r) => r.plan.content.run === evidence.run,
|
||||
)!;
|
||||
const candidatePlans = accounts.runs.filter(
|
||||
(r) =>
|
||||
r.plan.content.purpose === "forward" &&
|
||||
r.plan.content.object === evidence.candidate,
|
||||
);
|
||||
const atKill = JSON.parse(
|
||||
readFileSync(join(dir, "ledger-at-kill.json"), "utf8"),
|
||||
) as LedgerRecord[];
|
||||
const killSeq = atKill.at(-1)!.seq;
|
||||
kill = {
|
||||
evidence,
|
||||
killSeq,
|
||||
snapshots,
|
||||
charged: target.starts.length,
|
||||
resultSeq: target.results[0]?.seq ?? null,
|
||||
checks: {
|
||||
sameContainer:
|
||||
snapshots[0]!.State.Running &&
|
||||
snapshots.every(
|
||||
(c) =>
|
||||
c.Id === snapshots[0]!.Id &&
|
||||
c.State.StartedAt === snapshots[0]!.State.StartedAt,
|
||||
),
|
||||
chargedOnceNoRerun:
|
||||
target.starts.length === 1 && candidatePlans.length === 1,
|
||||
resultAndContinued:
|
||||
!ofKind(atKill, "runResult").some(
|
||||
(r) => r.content.run === evidence.run,
|
||||
) &&
|
||||
target.results.length === 1 &&
|
||||
target.results[0]!.seq > killSeq &&
|
||||
accounts.checks.adoptedAndStopped,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
dir,
|
||||
study: config.studyId,
|
||||
scores,
|
||||
accounts,
|
||||
lightChecks,
|
||||
kill,
|
||||
pass:
|
||||
Object.values(accounts.checks).every(Boolean) &&
|
||||
lightChecks.every((c) => c.pass) &&
|
||||
(kill === null || Object.values(kill.checks).every(Boolean)),
|
||||
};
|
||||
});
|
||||
const references = [
|
||||
"16-real-1",
|
||||
"15-real-1",
|
||||
"15-tight-2",
|
||||
"14-real-1",
|
||||
"12-real-2",
|
||||
].flatMap((name) => referenceScores(join(referenceRoot, name, "board.sqlite")));
|
||||
jsonFile(join(output, "stats.json"), summaries);
|
||||
jsonFile(
|
||||
join(output, "predictions-scores.json"),
|
||||
summaries.flatMap((s) => s.scores),
|
||||
);
|
||||
jsonFile(join(output, "reference-scores.json"), references);
|
||||
jsonFile(join(output, "score-comparison.json"), {
|
||||
references,
|
||||
ours: summaries.flatMap((s) => s.scores),
|
||||
caveat:
|
||||
"参考项目提示词带题目具体做法、方法版本也不同；保留无效成绩及重复尝试，不作公平因果比较",
|
||||
});
|
||||
jsonFile(join(output, "manual-review.json"), {
|
||||
implementation:
|
||||
"每个已启动候选一个全新上下文：协议逐条、基线、最终代码、concerns、位置表；逐项漏做/额外改进/位置表准确性和代码位置",
|
||||
verification:
|
||||
"同一审阅者再读各次核验及对应版本：误放行/误拒、修复是否仅改指出处；保存原始回复",
|
||||
route:
|
||||
"逐候选按任务书15附录一路线归类，并单列完整高上限路线的实测-q5；不自动按关键词贴标签",
|
||||
comparison:
|
||||
"复用eval/route导出、items、thinking、verify-ranges、原四问Jev及原阅读问卷，填A–J；与任务15三次历史结果并排，标历史批次",
|
||||
serial:
|
||||
"核对各目录launches.jsonl、machine.jsonl和父会话观察；上一份全部结束后才起下一份；不宣称快照排除任意外部进程",
|
||||
});
|
||||
console.log(
|
||||
`统计：${output}；程序检查${summaries.every((s) => s.pass) ? "通过" : "未通过"}；质量判断仍待独立阅读。`,
|
||||
);
|
||||
if (!summaries.every((s) => s.pass)) process.exitCode = 1;
|
||||
@@ -0,0 +1,171 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { checkConfig } from "../../src/config/index.ts";
|
||||
import type { LedgerRecord } from "../../src/ledger/index.ts";
|
||||
import {
|
||||
Board,
|
||||
FORWARD_NOT_RUN,
|
||||
type StudyRecord,
|
||||
} from "../../src/materials/index.ts";
|
||||
import { placement, runBalance } from "../../src/rules/index.ts";
|
||||
import { verificationOutcome } from "../../src/submissions/index.ts";
|
||||
import type { Range } from "../../src/vocab/index.ts";
|
||||
import { finalSubmissions, ofKind } from "./ledger.ts";
|
||||
|
||||
export const FORWARD_STOP = "消融还没写（任务书 17）";
|
||||
export const jsonFile = (path: string, value: unknown): void =>
|
||||
writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`);
|
||||
export const frozenConfig = (records: LedgerRecord[]) =>
|
||||
checkConfig((ofKind(records, "config")[0]!.content as StudyRecord).config);
|
||||
|
||||
/** 区间边界只是报告口径，不参与研究决定；失败或测不到分数保持 null。 */
|
||||
export function scoreBand(score: number | null, range: Range): string | null {
|
||||
if (score === null) return null;
|
||||
return {
|
||||
belowQ1: "<q1",
|
||||
q1ToQ5: "[q1,q5)",
|
||||
q5ToQ9: "[q5,q9]",
|
||||
aboveQ9: ">q9",
|
||||
}[placement(range, score)];
|
||||
}
|
||||
|
||||
/** 核验要对应当时那版代码，不能拿最终修好后的代码替代历史被审代码。 */
|
||||
export function candidateHistory(records: LedgerRecord[], candidate: string) {
|
||||
const code = ofKind(records, "code").filter(
|
||||
(r) => r.content.object === candidate,
|
||||
);
|
||||
const submissions = ofKind(records, "submission").filter(
|
||||
(r) => r.content.target === candidate,
|
||||
);
|
||||
const attempts = submissions.flatMap((r) => {
|
||||
if (r.content.kind !== "implementation" && r.content.kind !== "repair")
|
||||
return [];
|
||||
// flow 在同一事务里先写 submission，紧接着写 code。
|
||||
return [{ submission: r, code: code.find((c) => c.seq === r.seq + 1)! }];
|
||||
});
|
||||
const verifications = submissions.flatMap((r) =>
|
||||
r.content.kind === "verification"
|
||||
? [
|
||||
{
|
||||
submission: r,
|
||||
code: code.filter((c) => c.seq < r.seq).at(-1)!,
|
||||
outcome: verificationOutcome(r.content.content),
|
||||
},
|
||||
]
|
||||
: [],
|
||||
);
|
||||
return { attempts, verifications };
|
||||
}
|
||||
|
||||
/** 程序验收只读记录重算；基线也算正式运行，没启动的结果不扣次。 */
|
||||
export function forwardAccounts(records: LedgerRecord[]) {
|
||||
const board = new Board(records);
|
||||
const config = frozenConfig(records);
|
||||
const opens = ofKind(records, "stageOpen");
|
||||
const closes = ofKind(records, "stageClose");
|
||||
const stages = opens.map((r) => {
|
||||
const closing = closes.filter(
|
||||
(c) => c.content.stageId === r.content.stageId,
|
||||
);
|
||||
return {
|
||||
...r.content,
|
||||
openSeq: r.seq,
|
||||
closeSeqs: closing.map((c) => c.seq),
|
||||
pass: closing.length === 1 && closing[0]!.seq > r.seq,
|
||||
};
|
||||
});
|
||||
const orphanCloses = closes.filter(
|
||||
(c) => !opens.some((o) => o.content.stageId === c.content.stageId),
|
||||
);
|
||||
const started = ofKind(records, "runStarted");
|
||||
const balance = runBalance(config.runs, board.runStates());
|
||||
const runs = ofKind(records, "runPlanned").map((p) => ({
|
||||
plan: p,
|
||||
starts: started.filter((s) => s.content.run === p.content.run),
|
||||
results: ofKind(records, "runResult").filter(
|
||||
(r) => r.content.run === p.content.run,
|
||||
),
|
||||
}));
|
||||
const batch = board.decision("batch", 1);
|
||||
const finals = finalSubmissions(records);
|
||||
const candidates = (batch?.output.chosen ?? []).map((candidate) => {
|
||||
const history = candidateHistory(records, candidate);
|
||||
const events = ofKind(records, "event").filter(
|
||||
(r) =>
|
||||
r.content.event === FORWARD_NOT_RUN &&
|
||||
r.content.objects.includes(candidate),
|
||||
);
|
||||
const candidateRuns = runs.filter(
|
||||
(r) =>
|
||||
r.plan.content.purpose === "forward" &&
|
||||
r.plan.content.round === 1 &&
|
||||
r.plan.content.object === candidate,
|
||||
);
|
||||
const verificationCount = finals.filter(
|
||||
(r) =>
|
||||
r.content.target === candidate && r.content.kind === "verification",
|
||||
).length;
|
||||
const repairCount = finals.filter(
|
||||
(r) => r.content.target === candidate && r.content.kind === "repair",
|
||||
).length;
|
||||
const lastPassed = history.verifications.at(-1)?.outcome.passed ?? null;
|
||||
const exhausted = lastPassed === false && repairCount >= config.maxRepairs;
|
||||
return {
|
||||
candidate,
|
||||
verificationCount,
|
||||
repairCount,
|
||||
lastPassed,
|
||||
exhausted,
|
||||
notRunEvents: events,
|
||||
started: candidateRuns.some((r) => r.starts.length > 0),
|
||||
runs: candidateRuns,
|
||||
pass: exhausted
|
||||
? events.length === 1 && candidateRuns.length === 0
|
||||
: lastPassed === true &&
|
||||
candidateRuns.some(
|
||||
(r) => r.starts.length === 1 && r.results.length === 1,
|
||||
),
|
||||
};
|
||||
});
|
||||
const adoption =
|
||||
ofKind(records, "decision").find(
|
||||
(r) => r.content.decision === "adopt" && r.content.round === 1,
|
||||
) ?? null;
|
||||
const halt =
|
||||
ofKind(records, "event")
|
||||
.filter((r) => r.content.event === "halted")
|
||||
.at(-1) ?? null;
|
||||
const checks = {
|
||||
stagesPaired: stages.every((s) => s.pass) && orphanCloses.length === 0,
|
||||
chargedOnce:
|
||||
balance.charged === started.length &&
|
||||
config.runs - balance.balance === started.length &&
|
||||
balance.reserved === 0,
|
||||
forwardSerial: runs
|
||||
.filter((r) => r.plan.content.purpose === "forward")
|
||||
.every(
|
||||
(r, i, forwards) =>
|
||||
i === 0 ||
|
||||
(forwards[i - 1]!.results.length === 1 &&
|
||||
forwards[i - 1]!.results[0]!.seq < r.plan.seq),
|
||||
),
|
||||
runsFinished: runs.length > 0 && runs.every((r) => r.results.length === 1),
|
||||
candidatesFinished:
|
||||
candidates.length > 0 && candidates.every((c) => c.pass),
|
||||
adoptedAndStopped:
|
||||
adoption !== null &&
|
||||
halt?.content.reason === FORWARD_STOP &&
|
||||
halt.seq === records.at(-1)!.seq,
|
||||
};
|
||||
return {
|
||||
initial: config.runs,
|
||||
confirmedStarts: started.length,
|
||||
balance,
|
||||
runs,
|
||||
candidates,
|
||||
stages,
|
||||
orphanCloses,
|
||||
adoption,
|
||||
halt,
|
||||
checks,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
import { readdirSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
/** Pi 会话里的 research_check：请求和结果按 toolCallId 配对，不把超限请求算成执行。 */
|
||||
export function lightCheckUsage(sessionDir: string, maxCount: number) {
|
||||
type Part = { type: string; id?: string; name?: string; text?: string };
|
||||
type Message = {
|
||||
role: string;
|
||||
toolCallId?: string;
|
||||
toolName?: string;
|
||||
isError?: boolean;
|
||||
content: Part[];
|
||||
};
|
||||
const requests: Array<{ id: string; file: string; line: number }> = [];
|
||||
const results = new Map<
|
||||
string,
|
||||
{ file: string; line: number; message: Message }
|
||||
>();
|
||||
const files = readdirSync(sessionDir)
|
||||
.filter((f) => f.endsWith(".jsonl"))
|
||||
.sort();
|
||||
if (files.length === 0) throw new Error(`没有会话文件：${sessionDir}`);
|
||||
for (const file of files) {
|
||||
readFileSync(join(sessionDir, file), "utf8")
|
||||
.split("\n")
|
||||
.forEach((line, index) => {
|
||||
if (!line.trim()) return;
|
||||
const entry = JSON.parse(line) as { type: string; message?: Message };
|
||||
const m = entry.message;
|
||||
if (entry.type !== "message" || m === undefined) return;
|
||||
if (m.role === "assistant") {
|
||||
for (const p of m.content)
|
||||
if (p.type === "toolCall" && p.name === "research_check")
|
||||
requests.push({ id: p.id!, file, line: index + 1 });
|
||||
} else if (m.role === "toolResult" && m.toolName === "research_check")
|
||||
results.set(m.toolCallId!, { file, line: index + 1, message: m });
|
||||
});
|
||||
}
|
||||
const checks = requests.map((request) => {
|
||||
const result = results.get(request.id);
|
||||
if (result === undefined)
|
||||
return { request, result: null, status: "missingResult", used: null };
|
||||
const text = result.message.content
|
||||
.filter((p) => p.type === "text")
|
||||
.map((p) => p.text)
|
||||
.join("\n");
|
||||
const source = { file: result.file, line: result.line, text };
|
||||
if (result.message.isError)
|
||||
return { request, result: source, status: "toolError", used: null };
|
||||
if (text.startsWith("本次调用的轻量检查已用完"))
|
||||
return { request, result: source, status: "deniedLimit", used: null };
|
||||
const value = JSON.parse(text) as {
|
||||
used: number;
|
||||
maxCount: number;
|
||||
seconds: number;
|
||||
result: unknown;
|
||||
};
|
||||
return { request, result: source, status: "executed", used: value.used };
|
||||
});
|
||||
const used = checks.flatMap((c) => (c.used === null ? [] : [c.used]));
|
||||
const unresolved = checks.some(
|
||||
(c) => c.status === "missingResult" || c.status === "toolError",
|
||||
);
|
||||
return {
|
||||
sessionDir,
|
||||
maxCount,
|
||||
requested: checks.length,
|
||||
executed: used.length,
|
||||
deniedLimit: checks.filter((c) => c.status === "deniedLimit").length,
|
||||
// 工具报错或缺结果时不能猜实际执行量，也不当作 0 次通过。
|
||||
pass:
|
||||
!unresolved &&
|
||||
used.length <= maxCount &&
|
||||
used.every((n, i) => n === i + 1),
|
||||
checks,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { basename, dirname } from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
|
||||
/** 只读已有副本；保留 incomplete 的数值及重复尝试，不冒充有效成绩。 */
|
||||
export function referenceScores(path: string) {
|
||||
const study = basename(dirname(path));
|
||||
const db = new DatabaseSync(path, { readOnly: true });
|
||||
try {
|
||||
type Fact<T> = { seq: number; content: T };
|
||||
const facts = <T>(kind: string): Fact<T>[] =>
|
||||
(
|
||||
db
|
||||
.prepare(
|
||||
"SELECT created_seq, payload FROM facts WHERE kind = ? ORDER BY created_seq",
|
||||
)
|
||||
.all(kind) as Array<{ created_seq: number; payload: string }>
|
||||
).map((r) => ({
|
||||
seq: r.created_seq,
|
||||
content: JSON.parse(r.payload) as T,
|
||||
}));
|
||||
const links = facts<{
|
||||
purpose: string;
|
||||
roundId: string;
|
||||
runId: string;
|
||||
candidateId: string;
|
||||
}>("run-link");
|
||||
const observations = facts<{
|
||||
runId: string;
|
||||
status: string;
|
||||
finalMetric: { value: number | null };
|
||||
reasons: string[];
|
||||
}>("run-observation");
|
||||
const adoption = facts<{
|
||||
roundId: string;
|
||||
adoptedRunId: string | null;
|
||||
newBaselineId: string;
|
||||
}>("adoption").find((r) => r.content.roundId === `${study}.r1`)!;
|
||||
return links
|
||||
.filter(
|
||||
(r) =>
|
||||
r.content.purpose === "forward" &&
|
||||
r.content.roundId === `${study}.r1`,
|
||||
)
|
||||
.map((link) => {
|
||||
const observation = observations.find(
|
||||
(r) => r.content.runId === link.content.runId,
|
||||
)!;
|
||||
return {
|
||||
source: path,
|
||||
study,
|
||||
candidate: link.content.candidateId,
|
||||
run: link.content.runId,
|
||||
status: observation.content.status,
|
||||
rawScore: observation.content.finalMetric.value,
|
||||
validScore:
|
||||
observation.content.status === "valid"
|
||||
? observation.content.finalMetric.value
|
||||
: null,
|
||||
adopted: adoption.content.adoptedRunId === link.content.runId,
|
||||
newBaseline: adoption.content.newBaselineId,
|
||||
linkSeq: link.seq,
|
||||
observationSeq: observation.seq,
|
||||
adoptionSeq: adoption.seq,
|
||||
reasons: observation.content.reasons,
|
||||
};
|
||||
});
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
+4
-1
@@ -27,7 +27,10 @@
|
||||
"eval:route:extract": "node eval/route/extract.ts",
|
||||
"eval:route:items": "node eval/route/items.ts",
|
||||
"eval:route:jev": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/route/jevroutes.ts",
|
||||
"eval:route:thinking": "node eval/route/thinking.ts"
|
||||
"eval:route:thinking": "node eval/route/thinking.ts",
|
||||
"eval:forward:run": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/forward/run.ts",
|
||||
"eval:forward:export": "node eval/forward/export.ts",
|
||||
"eval:forward:stats": "node eval/forward/stats.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-ai": "0.85.1",
|
||||
|
||||
@@ -0,0 +1,354 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { test } from "node:test";
|
||||
import {
|
||||
candidateHistory,
|
||||
forwardAccounts,
|
||||
FORWARD_STOP,
|
||||
scoreBand,
|
||||
} from "../eval/lib/forward.ts";
|
||||
import { lightCheckUsage } from "../eval/lib/light-checks.ts";
|
||||
import { referenceScores } from "../eval/lib/reference-scores.ts";
|
||||
import { Ledger } from "../src/ledger/index.ts";
|
||||
import { forwardNotRunEvent } from "../src/materials/index.ts";
|
||||
import { createRepository, saveVersion } from "../src/workspace/index.ts";
|
||||
import { fullConfig } from "./config-helpers.ts";
|
||||
import {
|
||||
definition,
|
||||
exited,
|
||||
implementation,
|
||||
notStarted,
|
||||
proposal,
|
||||
tempDir,
|
||||
} from "./ledger-helpers.ts";
|
||||
import { judged, protocolOf, Writer } from "./materials-helpers.ts";
|
||||
|
||||
const verification = (found: boolean) => ({
|
||||
clauses: [{ clause: "c1", found, where: "solve.py:1" }],
|
||||
factors: [{ factor: "p1", found, where: "solve.py:1" }],
|
||||
safeguards: [],
|
||||
unassigned: [],
|
||||
body: "逐项核对",
|
||||
});
|
||||
|
||||
test("forward 导出/统计：历史代码配对、两次核验一次修复、封存自评与真实分数保持可追溯", (t) => {
|
||||
const root = tempDir(t),
|
||||
dir = join(root, "run"),
|
||||
study = join(dir, "study");
|
||||
mkdirSync(study, { recursive: true });
|
||||
const ledger = Ledger.open(join(study, "ledger.sqlite"));
|
||||
t.after(() => ledger.close());
|
||||
const w = new Writer(ledger),
|
||||
config = fullConfig();
|
||||
ledger.append("config", { config });
|
||||
const repo = join(study, "repo.git"),
|
||||
work = join(root, "work");
|
||||
createRepository(repo);
|
||||
mkdirSync(work);
|
||||
const version = (text: string) => {
|
||||
writeFileSync(join(work, "solve.py"), text);
|
||||
const result = saveVersion(repo, work, []);
|
||||
assert.ok(result.saved);
|
||||
return result.version;
|
||||
};
|
||||
ledger.append("code", {
|
||||
object: "base",
|
||||
version: version("value = 0\n"),
|
||||
locations: null,
|
||||
});
|
||||
w.run("base-1", "baseline", 0, null, judged(1, "main"));
|
||||
w.open("proposal", "proposal", 1, ["a"]);
|
||||
const p = proposal({ q1: 1, q5: 2, q9: 3 }, "协议正文");
|
||||
w.sub("proposal", "a", "proposal", null, "p", p);
|
||||
w.close("proposal");
|
||||
w.protocol("r1-c1", protocolOf("r1-c1", p, 1, "base", ["p"], ["改value"]));
|
||||
w.open("second", "secondPrediction", 1, ["a"]);
|
||||
w.sub("second", "a", "secondPrediction", "r1-c1", null, {
|
||||
range: { q1: 1, q5: 2, q9: 3 },
|
||||
body: "估计",
|
||||
});
|
||||
w.close("second");
|
||||
w.decision("batch", 1, null, {
|
||||
chosen: ["r1-c1"],
|
||||
candidates: [{ id: "r1-c1", merged: { q1: 1, q5: 2, q9: 3 } }],
|
||||
});
|
||||
const v1 = version("value = 1\n"),
|
||||
v2 = version("value = 2\n");
|
||||
w.open("impl", "implementation", 1, ["x"]);
|
||||
w.sub("impl", "x", "implementation", "r1-c1", null, implementation("首稿"));
|
||||
ledger.append("code", { object: "r1-c1", version: v1, locations: null });
|
||||
w.close("impl");
|
||||
w.open("verify1", "verification", 1, ["v"]);
|
||||
w.sub("verify1", "v", "verification", "r1-c1", null, verification(false));
|
||||
w.close("verify1");
|
||||
w.open("repair", "implementation", 1, ["x"]);
|
||||
w.sub("repair", "x", "repair", "r1-c1", null, implementation("只修value"));
|
||||
ledger.append("code", { object: "r1-c1", version: v2, locations: null });
|
||||
w.close("repair");
|
||||
w.open("verify2", "verification", 1, ["v"]);
|
||||
w.sub("verify2", "v", "verification", "r1-c1", null, verification(true));
|
||||
w.close("verify2");
|
||||
ledger.append("runPlanned", {
|
||||
run: "forward-1",
|
||||
purpose: "forward",
|
||||
round: 1,
|
||||
object: "r1-c1",
|
||||
definition,
|
||||
});
|
||||
ledger.append("runStarted", { run: "forward-1" });
|
||||
const atKill = ledger.records();
|
||||
writeFileSync(join(dir, "ledger-at-kill.json"), JSON.stringify(atKill));
|
||||
writeFileSync(
|
||||
join(dir, "kill.json"),
|
||||
JSON.stringify({
|
||||
run: "forward-1",
|
||||
candidate: "r1-c1",
|
||||
beforeKillSeq: atKill.at(-1)!.seq,
|
||||
}),
|
||||
);
|
||||
for (const part of ["before", "after-kill", "after-resume"])
|
||||
writeFileSync(
|
||||
join(dir, `container-${part}.json`),
|
||||
JSON.stringify({
|
||||
Id: "same-container",
|
||||
State: { Running: true, StartedAt: "2026-10-04T00:00:00Z" },
|
||||
}),
|
||||
);
|
||||
ledger.append("runResult", {
|
||||
run: "forward-1",
|
||||
process: exited,
|
||||
judged: judged(2.5, "main"),
|
||||
});
|
||||
w.decision("adopt", 1, null, { adopted: "r1-c1" });
|
||||
ledger.append("event", {
|
||||
event: "halted",
|
||||
objects: [],
|
||||
reason: FORWARD_STOP,
|
||||
});
|
||||
const records = ledger.records(),
|
||||
accounts = forwardAccounts(records);
|
||||
assert.deepEqual(accounts.balance, {
|
||||
charged: 2,
|
||||
reserved: 0,
|
||||
balance: config.runs - 2,
|
||||
});
|
||||
assert.ok(Object.values(accounts.checks).every(Boolean));
|
||||
assert.equal(accounts.candidates[0]!.verificationCount, 2);
|
||||
assert.equal(accounts.candidates[0]!.repairCount, 1);
|
||||
assert.deepEqual(
|
||||
candidateHistory(records, "r1-c1").verifications.map(
|
||||
(v) => v.code.content.version,
|
||||
),
|
||||
[v1, v2],
|
||||
);
|
||||
const exported = join(root, "export");
|
||||
const exportResult = spawnSync(
|
||||
process.execPath,
|
||||
["eval/forward/export.ts", study, exported],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
assert.equal(exportResult.status, 0, exportResult.stderr);
|
||||
assert.equal(
|
||||
readFileSync(join(exported, "r1-c1/final/solve.py"), "utf8"),
|
||||
"value = 2\n",
|
||||
);
|
||||
assert.match(
|
||||
readFileSync(join(exported, "r1-c1/attempts.json"), "utf8"),
|
||||
/只修value/,
|
||||
);
|
||||
assert.match(
|
||||
readFileSync(join(exported, "r1-c1/final-implementation.json"), "utf8"),
|
||||
/concerns/,
|
||||
);
|
||||
const refs = join(root, "refs");
|
||||
mkdirSync(refs);
|
||||
for (const name of [
|
||||
"16-real-1",
|
||||
"15-real-1",
|
||||
"15-tight-2",
|
||||
"14-real-1",
|
||||
"12-real-2",
|
||||
])
|
||||
referenceFixture(join(refs, name));
|
||||
const statsOut = join(root, "stats");
|
||||
const statsResult = spawnSync(
|
||||
process.execPath,
|
||||
["eval/forward/stats.ts", statsOut, refs, dir],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
assert.equal(statsResult.status, 0, statsResult.stderr);
|
||||
const scores = JSON.parse(
|
||||
readFileSync(join(statsOut, "predictions-scores.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(scores[0].authorEstimates[0].range, { q1: 1, q5: 2, q9: 3 });
|
||||
assert.equal(scores[0].runs[0].measuredMinusQ5, 0.5);
|
||||
assert.equal(scores[0].runs[0].band, "[q5,q9]");
|
||||
const statistics = JSON.parse(
|
||||
readFileSync(join(statsOut, "stats.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(statistics[0].kill.checks, {
|
||||
sameContainer: true,
|
||||
chargedOnceNoRerun: true,
|
||||
resultAndContinued: true,
|
||||
});
|
||||
writeFileSync(
|
||||
join(dir, "container-after-resume.json"),
|
||||
JSON.stringify({
|
||||
Id: "replacement",
|
||||
State: { Running: true, StartedAt: "2026-10-04T00:01:00Z" },
|
||||
}),
|
||||
);
|
||||
const badOut = join(root, "bad-stats");
|
||||
const bad = spawnSync(
|
||||
process.execPath,
|
||||
["eval/forward/stats.ts", badOut, refs, dir],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
assert.equal(bad.status, 1, bad.stderr);
|
||||
assert.equal(
|
||||
JSON.parse(readFileSync(join(badOut, "stats.json"), "utf8"))[0].kill.checks
|
||||
.sameContainer,
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
test("forward 账目：没启动退还、修复用尽不扣次；缺不跑事件或结果不能通过", (t) => {
|
||||
const ledger = Ledger.open(join(tempDir(t), "ledger.sqlite"));
|
||||
t.after(() => ledger.close());
|
||||
const w = new Writer(ledger),
|
||||
config = { ...fullConfig(), maxRepairs: 0 };
|
||||
ledger.append("config", { config });
|
||||
ledger.append("runPlanned", {
|
||||
run: "base-0",
|
||||
purpose: "baseline",
|
||||
round: 0,
|
||||
object: null,
|
||||
definition,
|
||||
});
|
||||
ledger.append("runResult", {
|
||||
run: "base-0",
|
||||
process: notStarted,
|
||||
judged: null,
|
||||
});
|
||||
w.run("base-1", "baseline", 0, null, judged(1, "main"));
|
||||
w.decision("batch", 1, null, { chosen: ["c"] });
|
||||
ledger.append("code", { object: "c", version: "v", locations: null });
|
||||
w.open("v", "verification", 1, ["v"]);
|
||||
w.sub("v", "v", "verification", "c", null, verification(false));
|
||||
w.close("v");
|
||||
assert.equal(forwardAccounts(ledger.records()).candidates[0]!.pass, false);
|
||||
ledger.append("event", forwardNotRunEvent("c", "修复用尽"));
|
||||
w.decision("adopt", 1, null, { adopted: null });
|
||||
ledger.append("event", {
|
||||
event: "halted",
|
||||
reason: FORWARD_STOP,
|
||||
objects: [],
|
||||
});
|
||||
const accounts = forwardAccounts(ledger.records());
|
||||
assert.deepEqual(accounts.balance, {
|
||||
charged: 1,
|
||||
reserved: 0,
|
||||
balance: config.runs - 1,
|
||||
});
|
||||
assert.ok(Object.values(accounts.checks).every(Boolean));
|
||||
ledger.append("runPlanned", {
|
||||
run: "unfinished",
|
||||
purpose: "forward",
|
||||
round: 1,
|
||||
object: "c",
|
||||
definition,
|
||||
});
|
||||
ledger.append("runStarted", { run: "unfinished" });
|
||||
assert.equal(forwardAccounts(ledger.records()).checks.runsFinished, false);
|
||||
assert.equal(forwardAccounts(ledger.records()).candidates[0]!.pass, false);
|
||||
});
|
||||
|
||||
test("forward 分位区间不把无效分数当零，三条边界按既有placement", () => {
|
||||
assert.deepEqual(
|
||||
[null, 0, 1, 2, 3, 4].map((s) => scoreBand(s, { q1: 1, q5: 2, q9: 3 })),
|
||||
[null, "<q1", "[q1,q5)", "[q5,q9]", "[q5,q9]", ">q9"],
|
||||
);
|
||||
});
|
||||
|
||||
test("forward 轻量检查：跨会话按工具编号配对，拒绝超限不算执行，缺结果不伪装零次", (t) => {
|
||||
const dir = tempDir(t);
|
||||
const request = (id: string) => ({
|
||||
type: "message",
|
||||
message: {
|
||||
role: "assistant",
|
||||
content: [{ type: "toolCall", name: "research_check", id }],
|
||||
},
|
||||
});
|
||||
const result = (id: string, text: string) => ({
|
||||
type: "message",
|
||||
message: {
|
||||
role: "toolResult",
|
||||
toolName: "research_check",
|
||||
toolCallId: id,
|
||||
content: [{ type: "text", text }],
|
||||
},
|
||||
});
|
||||
writeFileSync(join(dir, "1.jsonl"), JSON.stringify(request("a")) + "\n");
|
||||
writeFileSync(
|
||||
join(dir, "2.jsonl"),
|
||||
[
|
||||
result(
|
||||
"a",
|
||||
JSON.stringify({ used: 1, maxCount: 1, seconds: 5, result: {} }),
|
||||
),
|
||||
request("b"),
|
||||
result("b", "本次调用的轻量检查已用完（上限 1 次），不能再查。"),
|
||||
]
|
||||
.map((r) => JSON.stringify(r))
|
||||
.join("\n"),
|
||||
);
|
||||
const usage = lightCheckUsage(dir, 1);
|
||||
assert.equal(usage.requested, 2);
|
||||
assert.equal(usage.executed, 1);
|
||||
assert.equal(usage.deniedLimit, 1);
|
||||
assert.equal(usage.pass, true);
|
||||
writeFileSync(join(dir, "3.jsonl"), JSON.stringify(request("c")));
|
||||
assert.equal(lightCheckUsage(dir, 1).pass, false);
|
||||
});
|
||||
|
||||
function referenceFixture(dir: string) {
|
||||
mkdirSync(dir);
|
||||
const db = new DatabaseSync(join(dir, "board.sqlite"));
|
||||
db.exec("CREATE TABLE facts (created_seq INTEGER, kind TEXT, payload TEXT)");
|
||||
const study = dir.split("/").at(-1)!;
|
||||
const insert = (seq: number, kind: string, payload: unknown) =>
|
||||
db
|
||||
.prepare("INSERT INTO facts VALUES (?,?,?)")
|
||||
.run(seq, kind, JSON.stringify(payload));
|
||||
insert(1, "run-link", {
|
||||
purpose: "forward",
|
||||
roundId: `${study}.r1`,
|
||||
runId: "r",
|
||||
candidateId: "c",
|
||||
});
|
||||
insert(2, "run-observation", {
|
||||
runId: "r",
|
||||
status: "incomplete",
|
||||
finalMetric: { value: 99 },
|
||||
reasons: ["无效"],
|
||||
});
|
||||
insert(3, "adoption", {
|
||||
roundId: `${study}.r1`,
|
||||
adoptedRunId: null,
|
||||
newBaselineId: "B0",
|
||||
});
|
||||
db.close();
|
||||
}
|
||||
|
||||
test("forward 参考分数：incomplete 原值保留但有效分数为空", (t) => {
|
||||
const dir = join(tempDir(t), "reference");
|
||||
referenceFixture(dir);
|
||||
const rows = referenceScores(join(dir, "board.sqlite"));
|
||||
assert.equal(rows[0]!.rawScore, 99);
|
||||
assert.equal(rows[0]!.validScore, null);
|
||||
assert.equal(rows[0]!.adopted, false);
|
||||
assert.equal(rows[0]!.observationSeq, 2);
|
||||
});
|
||||
Reference in New Issue
Block a user