diff --git a/WORK_LOG.md b/WORK_LOG.md index 1d92c17..b25b02a 100644 --- a/WORK_LOG.md +++ b/WORK_LOG.md @@ -104,4 +104,4 @@ ## 下一步 -任务书 16(flow 第 3 部分:实现到采纳,含实验员、核验员提示词)已写,等用户另开会话执行。 +任务书 16 正在执行:第二块代码已完成独立审阅;第三块的 `eval/forward` 运行、导出、统计入口及结构文档已备好,尚未启动这三份真实研究。下一步由父会话逐份启动并监控,其中一份首个正向运行中杀控制进程、恢复同份;完成后导出逐候选审阅包、五项质量表和沿用任务书 15 的路线对照。真实验收未完成,不记入“已完成”。 diff --git a/docs/结构.md b/docs/结构.md index 4d687c5..1105996 100644 --- a/docs/结构.md +++ b/docs/结构.md @@ -3,6 +3,7 @@ 这份文件说明代码分成哪些模块、模块之间怎么依赖、流程按什么步骤推进、账本记哪些记录。研究方法本身以 `docs/方法.md` 为准;这里只说代码怎么组织来实现它。 来源: + - 用户 2026-10-01 认可"按负责一件事来切"、下面这套模块与可替换点,以及步骤清单和账本记录种类。 - 用户 2026-10-01 同意:提交的字段定义放 vocab,submissions 只判收不收;vocab 可放规范序列化和哈希两个函数,上限调到 600 行;执行后实际 1030 行,用户同意上限改为 1100 行;任务书 09 加对号清单时用户 2026-10-03 同意改为 1150 行,vocab 只放对号清单,任务材料的其余类型放 materials。 - 用户 2026-10-02 同意:"全部只追加":账本用一张只追加的表,全部记录都不能改。 @@ -32,7 +33,7 @@ ## 二、模块 | 层 | 模块 | 藏住的决定 | 依赖 | -|---|---|---|---| +| --- | --- | --- | --- | | 入口 | plugin(插件入口) | 用户在 Pi 里怎么操作:命令、状态显示、配置对话;文件锁;把各模块装配起来 | 装配时用到各模块 | | 组织 | flow(流程) | 按步骤清单推进研究;每次启动走同一条恢复路径 | ledger、rules、materials、submissions、agents、runner、tasks、workspace、config、jev | | 组织 | export(导出) | 导出给人看的格式 | ledger、rules、materials | @@ -114,13 +115,14 @@ 用户 2026-10-03 定:参考项目的 flow 太长、很难验证(用户原话:"我觉得我们之前其实很多问题都是 flow 没有做好"),所以拆成四部分,一部分验证通过后再写下一部分,每部分一份任务书。 | 部分 | 管哪些步骤 | 自己的验证目标 | 拿哪份真实记录对照 | -|---|---|---|---| +| --- | --- | --- | --- | | 1. 骨架、研究开始、第零轮 | 从账本算出现在到哪;"下一步做什么"的纯函数(步骤先后只写这一处);主循环;恢复入口;派一个模型任务、跑一次正式运行两个共用动作;冻结配置、建镜像、基线;第零轮 | 第零轮的研究质量六问(任务书 12):入口里有没有关键想法;分到的入口分没分开;好方向分出去没有;入口是不是改代码的路线;入口有没有写答案;讨论有没有用。另杀进程再启动,不重跑、不重扣、不漏写 | 关键想法和死路清单取自同类系统与参考项目高分运行的研究路径;16-real-1 第零轮回放,分配和参考项目相同,并用同样六问打分当对照 | | 2. 开轮到选批 | 4.3 ①–⑧(选批由用户 2026-10-03 定并进来) | 第 1 轮的整体研究质量(任务书 14):提案在研究路径上处在哪一步、多样性与判重、提案质量、挑刺与回应、预测是否合理、选批、过程健康;同一轮的提示词按调研结论先诊断再改 | 不回放(用户 2026-10-03 定)。参考项目高分运行(12-real-2、14-real-1、15-real-1、15-tight-2、16-real-1)第 1 轮的文字和我们的 3 次运行一起去掉来历,交全新上下文的审阅者按同一套方面打分比较 | | 3. 实现到采纳 | ⑨–⑫ | 扣掉的次数等于确认启动的运行数;修复用完不跑、不扣、记事件 | 15-tight-2 第 1 轮的修复 | | 4. 消融到收尾 | ⑬–⑱ 与研究结束 | 解释冻结早于任何一格启动;每格只结算一次 | 选格输入和 15-real-1、16-real-1 比;最后整份回放 16-real-1,当四部分合起来的验收 | **怎么验证**(用户 2026-10-03 定):一律用真 Jev、真 Docker,不做假的 Jev、Docker、任务,flow 不为此开替换口子。同样的输入真实跑几次,结果都符合要求就算通过;真实运行里没出现过的情况不专门造出来测。用户原话:"我建议用一律用真 Jev、真 Docker。因为其实我们现在是在做科学研究,不是在开发程序……我们只需要重复运行多次"。模型一侧两种都做: + - 回放:用本机假模型服务,把参考项目真实记录里模型交的东西按原顺序转成新格式再交一遍,走真实的 Pi。新 Jev 的判断可能和当时不同,两边走向分开以后不再对照,按方法判断分开的地方谁对。 - 真模型:用配置里的模型真实跑几次。 @@ -137,7 +139,7 @@ 恢复时只核对、不重做写入(CLAUDE.md"中断后怎么恢复"),所以记录照写就是,不需要通用的去重。有自己编号、按定义只发生一次的记录,按编号加数据库唯一约束,重复了就报错:阶段开、阶段关、运行登记、确认启动、运行结果、模型调用开始、模型调用结束、消融结算(按轮次:每轮最多采纳一个改法,所以最多一次消融)。没有这种编号的记录(提交的各个版本、事件、用量)照写。前后顺序(比如启动前要先登记)由 flow 的同一个入口保证,ledger 不查。运行编号由 flow 起;确认没启动、退还名额后要重试,就起一个新编号。 | # | 记录 | 内容 | 谁写入 | -|---|---|---|---| +| --- | --- | --- | --- | | 1 | 研究配置 | 冻结的配置、步骤清单、每个决策点用哪条规则、方法参数、提示词哈希 | flow(内容由 config 生成) | | 2 | 阶段 | 每个阶段开一条、关一条;关闭即公开 | flow | | 3 | 提交 | 各种提交的每一版:入口、讨论、志愿、提案、两次预测、挑刺、回应、解释、收尾、实现、核验:阶段关闭后以最后一版为准,此后由 ledger 拒绝再写。私人笔记的每一条:不属于任何阶段,以最后写入的一条为准。每一版带收下时用到的 Jev 判断(含原始概率和结论,没用到 Jev 的为空)。被退回的提交也记,附对象(同提交的 target:候选、原提案或消融格,没有对象的为空)、退回理由和用到的 Jev 判断 | flow(经 submissions 判定后) | @@ -161,7 +163,7 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 论文实验要换掉的部件,按种类用最轻的办法换: | 种类 | 例子 | 换法 | -|---|---|---| +| --- | --- | --- | | 外部服务 | Pi 会话、Docker 运行器、任务、Jev | 接口 | | 决策规则 | 选批:批内最好期望、均匀随机。选格:信息价值减成本、全测、随机同格数、都不测。以后可能加合并方式 | 纯函数,登记在 rules 里的"名字 → 函数"表里,配置写名字 | | 方法参数 | N、M、批量、运行次数、因子数上限、u、衰减、开放预留 | 配置值,不写成常数,也不用环境变量,每个都必须写在配置文件里。分两层:用户配置(配置对话逐项问);方法参数(做实验时才改,配置对话按方法.md 6.11 的现行值写进文件、给用户看过) | @@ -198,7 +200,7 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 下表是粗估。动手写某个模块之前,由用户逐个确认;写的过程中超过预期一倍,就停下来检查。 | 模块 | 预期行数 | -|---|---| +| --- | --- | | plugin | 600–1000 | | config | 300–600 | | flow | 2000–3500 | @@ -529,6 +531,7 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 - 消融实现:这一格已有公开的消融核验(核验不通过、重做)时,和修复一样从上一次交的这一格代码接着改(用户 2026-10-03 定):工作目录写"上一次交的这一格代码的副本",能读的版本里加上它,完整代码照旧列出作对照;另放"退回意见":最近一次核验里关错了的因子和核验员写的依据,写法同修复的退回意见。 - 指标:冻结之后一律用全名"候选编号.短名":已登记指标清单、对号清单的 `reusableMetrics`、实验员材料的"要实现的指标定义"(候选代码在结果文件里也用全名当键)。提案、回应、统一的对号清单里,已占用的指标编号只有任务的指标、`answer-path` 和 `safeguard-triggers` 两个保留键(用户 2026-10-03 定);已登记的全名带 `.`,新登记的短名不带,不用列进来。 - `Board`(类):materials 读账本的那一份,构造参数是账本记录数组。flow 的"下一步做什么"是只拿得到记录数组的纯函数,它和 flow 的三个程序步骤都经这一份读账本,不另写一份(任务书 14 第 2.4 节)。flow 用到的:`stages`、`stage`(阶段的轮次、名单、关没关)、`published`(已公开的某种提交,最后一版为准)、`decision`(某轮某种决定)、`proposal`、`proposalsOf`、`authorsOf`(候选合并了哪些原提案、作者按编号排)、`answerOf`(冻结出候选的最后一份回应或统一)、`protocol`、`protocols`、`carried`(留池候选)、`baselineOf`(第 r 轮的本批基线)、`baselineScore`、`runStates`。任务书 14 写的是导出 `baselineOf`、`baselineScore` 两个函数,实际导出了整个类,原因是纯函数只拿得到记录数组,要用的远不止这两样(这一处偏离待用户确认)。 +- `protocolMetrics(board, protocol)`:把冻结协议自定义及引用的指标合成原登记定义,实验员材料与 flow 问 Jev、正式运行共用(来源:任务书 16 第 2.2、2.4 节)。 - `finalEntries(ledger)`:按账本取的最终有效的入口(各人最后一版入口清单里的每个入口,带作者和正文);flow 分配入口时用,和认领的对号清单是同一份(任务书 12 第三部分)。 - `queryMaterial(material, id)`:只在这份材料的记录里按编号找,交回全文;找不到交回 null(agents 的查询回调据此答"看不到")。 - `readPrompts(dir)`:读 `roles/<角色>.md` 和 `tasks/<提交种类>.md`,交回文字和每个文件原始字节的 sha256;研究开始时 flow 把哈希冻结进研究配置记录(方法.md 4.1)。 @@ -537,6 +540,8 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 - `TaskRequest`、`StudyFacts`、`CodeChanges`、`Prompts`、`PromptFile`(类型):flow 传进来的东西,全部必填;`CodeChanges` 与 workspace 的比较结果同形(照 jev `codeHunks` 的先例,不引用 workspace)。 - 账本里只存成 Json 的内容的形状(用户 2026-10-03 决定 2:materials 定义,flow 照着写,ledger 不改):`StudyRecord`(研究配置记录:配置、步骤清单、提示词每个文件的哈希、镜像编号、任务事实、种子;配置和任务事实存 Json,flow 读回时用 `checkConfig`、`checkTaskFacts` 再检查,任务书 12 第三部分)、`FrozenProtocol`(冻结协议:方法.md 4.3 ⑥ 列的因子、保障、指标、正文、实现条款,加轮次、出发的基线、合并了哪些原提案;不带答复,主体由 submissions 的 `protocolBody` 拼,指标编号是全名)、`LocationTable`(代码记录的位置表)、`DecisionKind`、`DECISION_KINDS`、`DecisionShapes`(六种决定:入口分配、判重合并、选批、采纳、选格、押的因子,输入输出用 rules 的类型;没采纳时也写一条采纳决定)、`DedupOutput`(判重合并的输出)、`Settlement`(结算)。 - `freezeProtocol(body, candidate, round, baseline, proposals)`:flow 写冻结协议时调用。输入 submissions 交回的冻结协议主体、候选编号、轮次、基线、合并了哪些原提案,交回 `FrozenProtocol`;新登记的假设专用指标改用全名"候选编号.短名"(例如 `r2-c1.restart-gain`),引用的本来就是全名,照抄。加前缀只在这一处:同一轮的研究者互相看不到提案,会起同一个短名、定义却不同,候选编号判重合并之后才有,所以只能在冻结时加(用户 2026-10-03 定)。 +- `experimenterId(candidate)`、`verifierId(candidate)`:每个候选固定一位实验员、一位核验员,修复及再次核验沿用编号(来源:任务书 16 第 2.4 节)。 +- `FORWARD_NOT_RUN`、`forwardNotRunEvent(候选, 原因)`:正向候选修复用完仍未通过核验时的“不跑”事件,`objects` 只有该候选;不扣次(来源:任务书 16 第 2.2 节)。 - 事件:`CELL_NOT_RUN`、`cellNotRunEvent(格, 原因)`:一格消融核验不通过、重做次数用完而不跑时,flow 用它写事件记录(`objects` 只有这一格),结算记证据不足(方法.md 4.3 ⑯、6.6)。 - 编号规则:`ORIGINAL_BASELINE`、`researcherId`(第 i 位研究者,从 0 数,编号 `researcher-`,任务书 12 第三部分)、`entryId`、`proposalId`、`critiqueId`、`candidateId`、`factorId`、`cellId`、`splitCell`、`explanationId`、`callId`,以及协议内的局部编号 `factorKey`(p1…)、`clauseKey`(c1…)。入口编号按作者一直往后编,讨论时整份替换也不复用旧编号。 @@ -581,7 +586,7 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 ### flow -下面是 `src/flow/index.ts` 导出的全部内容,以后 plugin 也只用它。flow 只经各模块的 `index.ts` 用它们;正式代码里没有为检验写的参数、环境变量、回放模式或假实现(CLAUDE.md"测试")。这一节对应 flow 第 1、2 部分(任务书 12、14):骨架、研究开始、第零轮,以及第 r 轮从开轮到选批;走到第 r 轮的 `implementation` 停下,原因"实现还没写(任务书 16)"。轮次按"第 r 轮"写,不写死 1;现在走得到的只有第 1 轮。 +下面是 `src/flow/index.ts` 导出的全部内容,以后 plugin 也只用它。flow 只经各模块的 `index.ts` 用它们;正式代码里没有为检验写的参数、环境变量、回放模式或假实现(CLAUDE.md"测试")。这一节对应 flow 第 1–3 部分(任务书 12、14、16):骨架、研究开始、第零轮,以及第 r 轮从开轮到采纳;采纳决定写完后走到 `listCells` 停下,原因"消融还没写(任务书 17)"。轮次按"第 r 轮"写,不写死 1;现在走得到的只有第 1 轮。 - `runStudy(input)`:推进一份研究,直到停下。输入三样,都必填:研究目录;配置(新研究交 `checkConfig` 检查过的配置,接着跑一份已有的研究写 null,以账本里冻结的那份为准;两者对不上就报错);"停下"信号。每次启动走同一条路:打开账本;新研究先冻结配置,本机已有 `inforsi-<研究编号>` 的镜像就说明研究编号用过了,报错停下,要换一个编号(编号重复的新研究会把旧研究遗留的同名容器当成自己的运行);已有研究的,核对冻结的配置、镜像还在、提示词哈希没变(开发入口开着时提示词可以变);给开始了没结束的模型调用补写结束(出错、"进程中断"、墙钟时间测不到);按容器名问 Docker 核对登记了没结果的正式运行(没启动的写"没启动"退还名额,启动了的补写确认启动、等它、判分、写结果,问不到的写"状态未知"事件并停下);然后进主循环:读账本 → 问"下一步做什么" → 做这一步 → 回到开头。主循环里 `docker run` 没启动的,写完"没启动"结果(退还名额、记下 Docker 报错原文)就停下,停下原因带上报错原文,本进程不重跑;下次启动时照旧起新编号重跑。恢复时补写的"没启动"不停下。 - `StudyInput`(类型):上面三样输入。 @@ -589,10 +594,10 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 研究目录里放:`ledger.sqlite` 账本;`repo.git` 代码版本的裸仓库;`runs/<运行编号>/` 一次正式运行(取出的代码、运行目录、日志、判分目录、校验器日志);`calls/<调用编号>/` 一次模型调用(工作目录、会话文件、能读的代码版本)。取出代码、建运行目录和判分目录时目标都是新建的空目录。容器名 `inforsi-<研究编号>-<运行编号>`,判分容器另加 `-judge`。同一份研究的全部运行用同一个种子:研究编号经 vocab 的 `hash` 取前 8 位十六进制转成整数,冻结进研究配置记录。 -**每一步以哪条记录为准算做完**(任务书 12 第 2.2 节、任务书 14 第 2.2 节)。步骤的先后只写在 flow 的"下一步做什么"一个纯函数里;后面两部分每加一步,这张表加一行。做完的轮次整轮跳过,第一个没做完的轮次里第一个没做完的步骤就是下一步。 +**每一步以哪条记录为准算做完**(任务书 12 第 2.2 节、任务书 14 第 2.2 节、任务书 16 第 2.3 节)。步骤的先后只写在 flow 的"下一步做什么"一个纯函数里;后续每加一步,这张表加一行。做完的轮次整轮跳过,第一个没做完的轮次里第一个没做完的步骤就是下一步。 | 步骤(config 的 `STUDY_STEPS`、`ROUND_STEPS`) | 看到哪条记录算做完 | -|---|---| +| --- | --- | | `freezeConfig` 冻结配置 | 研究配置记录(`config`) | | `baselineRun` 基线 | 基线代码记录(`code`,对象是 materials 的 `ORIGINAL_BASELINE`),而且最近一次基线运行的结果不是"没启动";"没启动"的下次启动时起新编号(`base-2`……)重跑,判为不算数的停下 | | `entryDraft`、`discussion1`、`discussion2`、`claim` | 这个阶段的关闭记录(`stageClose`);阶段编号 `r0-<步骤>` | @@ -602,24 +607,32 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 | `dedupe` 判重合并 | 本轮的判重合并决定(`decision`,种类 `dedup`) | | `freezeProtocol` 冻结协议 | 本轮每个新候选都有冻结协议记录(`protocol`);全部候选在同一个事务里写 | | `selectBatch` 选批 | 本轮的选批决定(`decision`,种类 `batch`) | +| `implementation` | 这一轮实现阶段的关闭记录 | +| `verification`、`repair` | 每个入选候选最近一次核验通过,或修复次数用完且有“不跑”事件;中间每次核验、修复阶段各有开关记录 | +| `forwardRun` | 每个通过核验的候选最近一次正向运行有结果,而且不是“没启动” | +| `publish`、`informationAccount` | 不写记录:`forwardRun` 做完就算做完;公开靠本轮采纳决定,信息账从预测和结果现算 | +| `adopt` | 本轮的采纳决定(`decision`,种类 `adopt`),没有过线者也写,`adopted` 为 null | | 按配置跳过的步骤 | 一条"按配置跳过"事件(`skippedByConfig`),写明步骤(第 r 轮的步骤写 `r<轮>-<步骤>`)和 config 给的原因 | 研究者阶段没开就开,名单是全部研究者(本轮名单就是冻结的配置);每个任务都有被收下的提交之后才关。 -**每个阶段谁做、对哪个对象**(任务书 14 第 2.3 节)。一个任务就是"一个人对一个对象",交一种提交。同一阶段里,一个人的几个任务按下表列出的先后在一个接一个的会话里跑,不同的人同时跑(任务书 12 第 2.6 节)。一个任务做完,以"这个人在这个阶段对这个对象有一份被收下的提交"为准。 +**每个阶段谁做、对哪个对象**(任务书 14 第 2.3 节、任务书 16 第 2.2、2.4 节)。一个任务就是"一个人对一个对象",交一种提交。同一阶段里,一个人的几个任务按下表列出的先后在一个接一个的会话里跑,不同的人同时跑(任务书 12 第 2.6 节)。一个任务做完,以"这个人在这个阶段对这个对象有一份被收下的提交"为准。 | 阶段 | 任务 | 对象 | -|---|---|---| +| --- | --- | --- | | 第零轮的写入口、两次讨论、认领 | 每位研究者一个任务 | 无 | | 提案 | 每位研究者一个任务,一个会话里交 1 到 M 条 | 无 | | 第一次预测 | 每位研究者 × 本轮每个新候选(含自己的;留池候选不做,方法.md 4.3 ④) | 候选 | | 挑刺 | 每位研究者 × 本轮每条不是自己写的原提案(监督者定,照参考项目:方法没写谁挑谁) | 原提案 | | 回应与统一 | 每个新候选一个任务:只有一条原提案的,由作者交"回应";几条合成的,由其中研究者编号最小的作者交"统一"(方法.md 4.3 ③⑥) | 候选 | | 第二次预测 | 每位研究者 × 候选池里每个候选(新候选加留池候选,方法.md 4.3 ①⑦) | 候选 | +| 实现 | 每个入选候选一个实验员,从本批基线取工作目录,不同候选同时实现 | 候选 | +| 核验 | 每个待核验候选一个核验员;首次核验本批全部,之后只核验刚修复的候选 | 候选 | +| 修复 | 只给核验不通过的候选,交原实验员,从上次提交的代码接着改 | 候选 | 没做完的任务派不派,按这个任务最近一次调用怎么结束的来定(任务书 12 第 2.3 节;第 2 部分从"一个人"细到"一个人对一个对象",任务书 14 第 2.3 节)。认是哪个任务,看"模型调用开始"里存的材料带的对象。没有调用、被用户停下、出错(含"进程中断")都派;没交的,开发入口关着就停下(交不出合格提交按内部错误,方法.md 4.3 ②),开着就再派。能派的先全派。一次派发里有任务没交上(没交或出错),不再开新的会话,已经在跑的不打断,等这次派发的会话全部结束后停下。 -一个任务交上来之后才知道的东西(submissions 的 `known`,任务书 14 第 2.3 节):提案是本人在本阶段已经收下几条(替换时不算被替换的那份);回应是作者那条原提案;其余种类没有。 +一个任务交上来之后才知道的东西(submissions 的 `known`,任务书 14 第 2.3 节):提案是本人在本阶段已经收下几条(替换时不算被替换的那份);回应是作者那条原提案;实现、修复交代码快照结果(见下文);其余种类没有。 提案和挑刺一个任务可以交几份,每份一个编号:提案用 materials 的 `proposalId`(轮次、作者、第几份),挑刺用 `critiqueId`(目标原提案、作者、第几份),收下时由 flow 按这个任务已收下几份往后分配,回给模型的提交编号就是它(任务书 14 第 2.3 节)。只交一份的种类,提交编号是"阶段.本人.种类.对象"。不写 `replaces` 是交新的一份,只交一份的种类已经交过就退回;写了 `replaces`,必须是这个任务已收下的某一份,整份替换它。 @@ -641,10 +654,19 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 - 种子用 rules 的 `decisionSeed`,由研究编号、轮次、版本算。同一轮里要随机的决定各用一个版本,选批用 1,选格预留 2,两次抽取不会用同一串随机数(方法.md 第九部分;选格由第 4 部分接上)。 - 规则取步骤清单里 `selectBatch` 那一步的,调 `runBatchRule`。写一条决定:种类 `batch`,规则名、版本和参数取 `runBatchRule` 交回的。 +**实现、核验、修复与采纳**(来源:任务书 16 第 2.2–2.4 节): + +- 阶段编号:`r<轮>-implementation`、`r<轮>-verification-1`、`r<轮>-repair-1`、`r<轮>-verification-2`……第几次从账本数,不另存。实现阶段关后才核验,核验阶段关后才派修复。修复次数用完仍不通过,用 `forwardNotRunEvent` 写清原因,不登记运行、不扣次。 +- 角色各取配置中的模型及自己的角色提示词。实验员工作目录先取版本,提交时用 workspace 存版本;相对本批基线比较、用 `codeHunks` 切块、查 `protectedChanges`,拼成 submissions 的 `known.code`。收下的提交和代码版本、位置表在同一事务写入;指标引用用 `protocolMetrics` 换成原登记定义。 +- 核验及修复材料的 `changes` 都相对本批基线:修复看基线到上次提交,修后核验看基线到修复结果;不新增上次到此次的材料。核验通过与否用 `verificationOutcome` 判。 +- 实验员的轻量检查接 `SessionTask.lightCheck`:tasks 给定义、runner 执行、tasks 读结果;每次模型调用分别按 `lightChecks.maxCount` 计次、按 `seconds` 限时,会话保留记录,不记入正式运行账本。 +- 通过核验的候选用既有 `formalRun`,用途 `forward`;先登记占名额,确认启动才扣次,再独立判分记结果。**一批内一个接一个跑**,避免抢 CPU 改变按秒停下的程序成绩。没启动的处理同基线:本进程停下,下次启动起新编号。 +- 本批全部有结果或“不跑”事件后,用 rules 的 `adopt`(内部用 `adoptionLine`),只传有效实测分数、本批基线分数和任务的方向、ε、已知最好;写种类 `adopt` 的决定,即使 `adopted` 为 null 也写。materials 据此公开本轮正向运行。 + 检验命令(`eval/`,真实运行,要 Docker;带 `*` 的会花模型或 Jev 的钱。输出都在仓库外 `/home/iomgaa/inforsi-eval/<日期-名字>/`,名字也当研究编号用,只能小写、不能重复): | 命令 | 用途 | -|---|---| +| --- | --- | | `npm run eval:study -- --config <文件> --dir <目录>`、`-- --resume --dir <目录>` * | 起一份研究或接着跑,Ctrl-C 发停下信号 | | `npm run eval:r0:run -- <名字>` * | 用 `eval/round-zero/research.json` 真实跑一次第零轮 | | `npm run eval:r0:quality -- <检验目录>` * | 对第零轮产出问六问(任务书 12 第 5.2 节),原始概率写进 `<检验目录>/quality.json` | @@ -659,3 +681,14 @@ Jev 是一个判别器,和程序检查地位相同,不单独成一种记录 | `npm run eval:r1:jevcheck -- <检验目录>` * | 对每次导出的运行问 Jev 三类问题(任务书 14 第 3.2 节第 3 步):判重对不对、挑刺有没有被回应、提案对应研究路径的哪一步;原始概率写进 `jevcheck.json` | | `npm run eval:r1:stats -- <检验目录>...` | 第 1 轮的数字和过程(任务书 14 第 3.2 节第 4 步),另查阶段开关成对、冻结协议写在回应阶段关闭之后、停下原因是"实现还没写";每个目录写一份 `stats.json` | +任务书 16 新增检验命令(来源:任务书 16 第三、四、五部分): + +| 命令 | 用途 | +| --- | --- | +| `npm run eval:forward:run -- new <研究编号> <全新绝对输出目录>` * | 配置照 `eval/round-zero/research.json`,只换编号;一份研究跑到采纳后停下,记录进程起止、账本观察时间及机器快照;异常保留现场并停下 | +| `npm run eval:forward:run -- kill <研究编号> <全新绝对输出目录>` * | 同上;首个正向 `runStarted` 后、无结果且容器仍运行时杀控制进程,保存容器 ID 和启动时间,恢复同份一次,不另开研究 | +| `npm run eval:forward:run -- resume <研究编号> <原输出目录>` * | 查明服务错误后手动续跑同份,追加进程/观察记录;也可用 `eval:study -- --resume --dir <输出目录>/study` | +| `npm run eval:forward:export -- <研究目录> <全新绝对审阅目录>` | 候选协议全文、引用指标定义、基线/最终/各次代码、位置表及 concerns、每次核验/修复、退回与调用材料、会话位置 | +| `npm run eval:forward:stats -- <全新绝对统计目录> <参考副本根目录> <检验目录>...` | 五项表中的可计算字段:预测与实测、参考首轮成绩、核验/修复次数、账目、轻量检查、杀进程三项检查;不自动作质量判断 | + +三份研究须逐个启动,上一份结束并核查完再起下一份,同机不与别的研究并跑;进程起止见 `launches.jsonl`,机器观察见 `machine.jsonl`,杀进程前后见 `container-*.json`,扣次与后续结果见统计的 `kill`。快照只是观察证据,不声称追踪任意进程。结果放仓库外,不提交。独立审阅按候选分两步:先评实现忠实度,再给历史核验及其当时代码评误判和修复范围。路线分类、完整路线的实测减合并 q5、A–J 表由阅读后填写;复用 `eval/route/` 的导出、条目、思考、分位数核查及 Jev 四问,不另造问卷或判别器。参考项目五份账本用已有只读副本,保留无效数值和重复尝试,并注明参考提示词含题目做法。 diff --git a/eval/forward/export.ts b/eval/forward/export.ts new file mode 100644 index 0000000..2d598e0 --- /dev/null +++ b/eval/forward/export.ts @@ -0,0 +1,132 @@ +import { mkdirSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { + Board, + protocolMetrics, + type TaskMaterial, +} from "../../src/materials/index.ts"; +import { checkoutVersion, compareVersions } from "../../src/workspace/index.ts"; +import { candidateHistory, jsonFile } from "../lib/forward.ts"; +import { ofKind, readLedger } from "../lib/ledger.ts"; + +// 任务书16五项质量评估的候选审阅材料;不判路线、不代替全新上下文逐条阅读。 +// node eval/forward/export.ts <研究目录> <全新绝对输出目录> +const [study, output, ...extra] = process.argv.slice(2); +if (!study || !output || extra.length || output !== resolve(output)) + throw new Error( + "用法:node eval/forward/export.ts <研究目录> <全新绝对输出目录>", + ); +mkdirSync(output); // 不覆盖以前的审阅材料。 +const records = readLedger(study); +const board = new Board(records); +const repo = join(study, "repo.git"); +const batch = board.decision("batch", 1)!; +const baseline = board.code(board.baselineOf(1)); +const plans = ofKind(records, "runPlanned").filter( + (p) => p.content.round === 1 && p.content.purpose === "forward", +); +const index = []; +for (const candidate of batch.output.chosen) { + const dir = join(output, candidate); + mkdirSync(dir); + const history = candidateHistory(records, candidate); + const candidatePlans = plans.filter((p) => p.content.object === candidate); + const started = candidatePlans.filter((p) => + ofKind(records, "runStarted").some((s) => s.content.run === p.content.run), + ); + const protocol = ofKind(records, "protocol") + .filter((p) => p.content.candidate === candidate) + .at(-1)!; + jsonFile(join(dir, "protocol.json"), { + record: protocol, + resolvedMetrics: protocolMetrics(board, board.protocol(candidate)), + }); + mkdirSync(join(dir, "baseline")); + checkoutVersion(repo, baseline.version, join(dir, "baseline")); + for (const attempt of history.attempts) { + const path = join(dir, `code-${attempt.code.seq}`); + mkdirSync(path); + checkoutVersion(repo, attempt.code.content.version, path); + jsonFile( + join(dir, `changes-${attempt.code.seq}.json`), + compareVersions(repo, baseline.version, attempt.code.content.version), + ); + } + // 首轮里每个候选只正式跑一个已通过版本;无启动的候选仍导出历史,但不冒称有最终通过代码。 + const firstRun = started[0]; + const finalCode = + firstRun === undefined + ? null + : ofKind(records, "code") + .filter((c) => c.content.object === candidate && c.seq < firstRun.seq) + .at(-1)!; + if (finalCode !== null) { + mkdirSync(join(dir, "final")); + checkoutVersion(repo, finalCode.content.version, join(dir, "final")); + } + const finalAttempt = + finalCode === null + ? null + : history.attempts.find((a) => a.code.seq === finalCode.seq)!; + jsonFile(join(dir, "final-implementation.json"), finalAttempt); + jsonFile(join(dir, "attempts.json"), history.attempts); + jsonFile(join(dir, "verifications.json"), history.verifications); + const calls = ofKind(records, "callStarted").filter((c) => { + const material = c.content.materials as TaskMaterial; + return ( + material.object === candidate && + ["implementation", "repair", "verification"].includes(material.kind) + ); + }); + jsonFile( + join(dir, "calls.json"), + calls.map((c) => ({ + record: c, + sessions: join(study, "calls", c.content.call, "sessions"), + })), + ); + jsonFile( + join(dir, "rejections.json"), + ofKind(records, "rejection").filter( + (r) => + r.content.target === candidate && + ["implementation", "repair", "verification"].includes(r.content.kind), + ), + ); + const manifest = { + study, + candidate, + protocolSeq: protocol.seq, + baselineVersion: baseline.version, + finalCode, + ran: started.length > 0, + runs: candidatePlans.map((p) => ({ + plan: p, + result: + ofKind(records, "runResult").find( + (r) => r.content.run === p.content.run, + ) ?? null, + artifacts: join(study, "runs", p.content.run), + })), + }; + jsonFile(join(dir, "manifest.json"), manifest); + writeFileSync( + join(dir, "review.txt"), + `任务书16第3.2节:只读本候选,分两步评估,中文直接回复,不写文件。\n` + + `第一步只读 protocol.json(冻结协议全文及引用指标原定义)、baseline/、final/、final-implementation.json(正文、位置表、concerns)。\n` + + `逐条实现条款、因子、保障检查漏做、协议之外的改进、位置表是否准确,每条附代码文件和行号。先形成独立结论,再打开核验历史。\n` + + `第二步读 verifications.json、attempts.json、code-<序号>/、changes-<序号>.json;每次核验对照其中code指向的那一版,不拿最终修好版本倒判历史。判断该拒却通过、不该拒却拒绝;修复是否只改指出处。\n` + + `需要时读 calls.json 和 rejections.json;模型材料和原始会话都可追溯。不把提交格式退回当成核验失败。\n` + + `答案路径只核对每个可能交出最终答案分支的标签,不预测实际走哪个分支。报告先写实现忠实度,再写逐次核验误判和修复范围。\n` + + `按任务书15附录一路线代号归类冻结协议,另标是否完整高上限路线(起点、连续改进全部圆、多起点或扰动、合法收尾),附协议原文位置。\n` + + `本候选${manifest.ran ? "已启动正式运行" : "未启动正式运行;没有final/,只评历史,不列成跑过的候选"}。\n`, + ); + index.push({ + candidate, + ran: manifest.ran, + directory: dir, + finalVersion: finalCode?.content.version ?? null, + }); +} +jsonFile(join(output, "index.json"), index); +console.log(`候选审阅包:${output}`); diff --git a/eval/forward/run.ts b/eval/forward/run.ts new file mode 100644 index 0000000..7515e5d --- /dev/null +++ b/eval/forward/run.ts @@ -0,0 +1,259 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync, existsSync, mkdirSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { forwardAccounts, frozenConfig, jsonFile } from "../lib/forward.ts"; +import { ofKind, readLedger } from "../lib/ledger.ts"; +import { launch, sleep, waitForLedger, writeConfig } from "../lib/study.ts"; +import { watchTimeline } from "../lib/timeline.ts"; + +// 任务书16:一次只启动一份;三次由父会话逐个执行。kill 只打首个正向运行,随后恢复同一研究一次。 +// node eval/forward/run.ts <研究编号> <绝对输出目录> +const [mode, name, output, ...extra] = process.argv.slice(2); +if ( + !mode || + !name || + !output || + extra.length || + !["new", "kill", "resume"].includes(mode) || + output !== resolve(output) +) { + throw new Error( + "用法:node eval/forward/run.ts <研究编号> <绝对输出目录>", + ); +} +const out = output; +const studyDir = join(out, "study"); +if (mode === "resume") { + if (frozenConfig(readLedger(studyDir)).studyId !== name) + throw new Error("研究编号与账本不符"); +} else { + if (existsSync(out)) + throw new Error(`输出目录已存在,不覆盖现场:${out}`); + mkdirSync(out, { recursive: true }); +} +const config = + mode === "resume" + ? null + : writeConfig( + "eval/round-zero/research.json", + join(out, "research.json"), + name, + null, + ); +const inspect = (container: string) => + JSON.parse( + execFileSync("docker", ["inspect", container], { + encoding: "utf8", + }), + )[0] as { + Id: string; + Name: string; + Created: string; + State: { + Running: boolean; + StartedAt: string; + FinishedAt: string; + Status: string; + }; + }; +function observe(label: string) { + appendFileSync( + join(out, "machine.jsonl"), + `${JSON.stringify({ + at: new Date().toISOString(), + label, + processes: execFileSync( + "ps", + ["-eo", "pid,ppid,lstart,args"], + { encoding: "utf8" }, + ), + containers: execFileSync( + "docker", + ["ps", "--no-trunc"], + { encoding: "utf8" }, + ), + })}\n`, + ); +} +function start(path: string | null) { + observe(path === null ? "resume" : "start"); + const study = launch(path, studyDir, {}); + appendFileSync( + join(out, "launches.jsonl"), + `${JSON.stringify({ event: "start", pid: study.child.pid, at: new Date().toISOString(), config: path })}\n`, + ); + const interrupt = () => study.child.kill("SIGINT"); + process.on("SIGINT", interrupt); + study.done.then((end) => { + process.off("SIGINT", interrupt); + appendFileSync( + join(out, "launches.jsonl"), + `${JSON.stringify({ event: "end", pid: study.child.pid, at: new Date().toISOString(), ...end })}\n`, + ); + }); + return study; +} +let active = start(config); +try { + await waitForLedger(studyDir, active.child); + let watching = watchTimeline( + studyDir, + active.child, + join(out, "timeline.jsonl"), + ); + if (mode === "kill") { + let hit = false; + while ( + active.child.exitCode === null && + active.child.signalCode === null + ) { + const records = readLedger(studyDir); + const plan = ofKind(records, "runPlanned").find( + (r) => + r.content.purpose === "forward" && + r.content.round === 1, + ); + if (plan !== undefined) { + const run = plan.content.run; + if ( + ofKind(records, "runResult").some( + (r) => r.content.run === run, + ) + ) + throw new Error( + "首个正向运行已经结束,未打中;保留现场,不改杀别的候选", + ); + const started = ofKind( + records, + "runStarted", + ).find((r) => r.content.run === run); + if (started !== undefined) { + // 容器名是正式入口记录在结构文档里的约定,不 import flow 内部文件。 + const container = `inforsi-${name}-${run}`; + const before = inspect(container); + jsonFile( + join( + out, + "container-before.json", + ), + before, + ); + if (!before.State.Running) + throw new Error( + "首个正向容器已不在运行,未打中", + ); + if ( + ofKind( + readLedger(studyDir), + "runResult", + ).some( + (r) => + r.content + .run === + run, + ) + ) + throw new Error( + "杀前已有结果,未打中", + ); + jsonFile(join(out, "kill.json"), { + run, + candidate: plan.content.object, + planSeq: plan.seq, + startSeq: started.seq, + beforeKillSeq: + records.at(-1)!.seq, + pid: active.child.pid, + at: new Date().toISOString(), + container, + }); + if (!active.child.kill("SIGKILL")) + throw new Error( + "未发出 SIGKILL", + ); + const killed = await active.done; + await watching; + if (killed.signal !== "SIGKILL") + throw new Error( + "控制进程不是以 SIGKILL 结束", + ); + const atKill = readLedger(studyDir); + jsonFile( + join( + out, + "ledger-at-kill.json", + ), + atKill, + ); + if ( + ofKind( + atKill, + "runResult", + ).some( + (r) => + r.content + .run === + run, + ) + ) + throw new Error( + "杀后核对发现目标运行已经写结果,未打中;保留现场", + ); + const afterKill = inspect(container); + jsonFile( + join( + out, + "container-after-kill.json", + ), + afterKill, + ); + observe("after-kill"); + active = start(null); + const afterResume = inspect(container); + jsonFile( + join( + out, + "container-after-resume.json", + ), + afterResume, + ); + watching = watchTimeline( + studyDir, + active.child, + join(out, "timeline.jsonl"), + ); + hit = true; + break; + } + } + await sleep(100); + } + if (!hit) + throw new Error( + `没有打中首个正向运行:${JSON.stringify(await active.done)}`, + ); + } + await watching; + const end = await active.done; + jsonFile(join(out, "outcome.json"), end); + observe("end"); + const accounts = forwardAccounts(readLedger(studyDir)); + jsonFile(join(out, "accounts.json"), accounts); + // 杀进程报告由 stats 读原始容器观察和账本重算;即使外部服务失败后人工恢复,也不覆盖原杀进程证据。 + if (end.code !== 0 || !Object.values(accounts.checks).every(Boolean)) { + throw new Error( + `没有完成采纳验收,保留现场;查明原因后才能手动 resume:${JSON.stringify(end)}`, + ); + } + console.log( + `已到采纳后停点:${studyDir}。仍需 export、stats 和独立质量审阅。`, + ); +} finally { + if ( + active.child.exitCode === null && + active.child.signalCode === null + ) { + active.child.kill("SIGINT"); + await active.done; + } +} diff --git a/eval/forward/stats.ts b/eval/forward/stats.ts new file mode 100644 index 0000000..94364d5 --- /dev/null +++ b/eval/forward/stats.ts @@ -0,0 +1,216 @@ +import { existsSync, mkdirSync, readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import type { LedgerRecord } from "../../src/ledger/index.ts"; +import { + Board, + validScore, + type TaskMaterial, +} from "../../src/materials/index.ts"; +import { + forwardAccounts, + frozenConfig, + jsonFile, + scoreBand, +} from "../lib/forward.ts"; +import { finalSubmissions, ofKind, readLedger } from "../lib/ledger.ts"; +import { lightCheckUsage } from "../lib/light-checks.ts"; +import { referenceScores } from "../lib/reference-scores.ts"; + +// 五项表中可计算的字段。路线与审阅结论故意不自动猜,由父会话按候选审阅包填写报告。 +// node eval/forward/stats.ts <全新绝对输出目录> <五份参考副本根目录> <检验目录>... +const [output, referenceRoot, ...dirs] = process.argv.slice(2); +if ( + !output || + !referenceRoot || + dirs.length === 0 || + output !== resolve(output) +) + throw new Error( + "用法:node eval/forward/stats.ts <全新绝对输出目录> <参考副本根目录> <检验目录>...", + ); +mkdirSync(output); +const summaries = dirs.map((dir) => { + const studyDir = join(dir, "study"); + const records = readLedger(studyDir); + const board = new Board(records); + const config = frozenConfig(records); + const accounts = forwardAccounts(records); + const batch = board.decision("batch", 1)!; + const finals = finalSubmissions(records); + const scores = accounts.candidates.map((c) => { + const p = board.protocol(c.candidate); + const merged = batch.output.candidates.find( + (b) => b.id === c.candidate, + )!.merged; + const secondPredictions = finals.flatMap((s) => + s.content.kind === "secondPrediction" && + s.content.target === c.candidate && + board.stage(s.content.stageId).round === 1 + ? [ + { + researcher: s.content.author, + q5: s.content.content.range.q5, + seq: s.seq, + }, + ] + : [], + ); + const authorEstimates = p.proposals.map((id) => { + const proposal = finals.find( + (s) => s.content.kind === "proposal" && s.content.item === id, + )!; + const seal = ofKind(records, "seal").find( + (s) => s.content.proposal === proposal.seq, + )!; + return { + proposal: id, + author: proposal.content.author, + range: seal.content.range, + proposalSeq: proposal.seq, + sealSeq: seal.seq, + }; + }); + const runs = c.runs.map((r) => { + const result = r.results[0] ?? null; + const score = validScore(result?.content.judged ?? null); + return { + run: r.plan.content.run, + planSeq: r.plan.seq, + started: r.starts.length > 0, + result, + score, + band: scoreBand(score, merged), + measuredMinusQ5: score === null ? null : score - merged.q5, + }; + }); + return { + study: config.studyId, + candidate: c.candidate, + name: p.factors.map((f) => f.name).join(";"), + merged, + secondPredictions, + authorEstimates, + runs, + adopted: board.decision("adopt", 1)?.output.adopted === c.candidate, + verificationCount: c.verificationCount, + repairCount: c.repairCount, + started: c.started, + manualRequired: + "路线按冻结协议归类;完整高上限路线单列 measuredMinusQ5;忠实度及误判见独立审阅回复", + }; + }); + const lightChecks = ofKind(records, "callStarted") + .filter((c) => + ["implementation", "repair"].includes( + (c.content.materials as TaskMaterial).kind, + ), + ) + .map((c) => ({ + call: c.content.call, + seq: c.seq, + ...lightCheckUsage( + join(studyDir, "calls", c.content.call, "sessions"), + config.lightChecks.maxCount, + ), + })); + let kill = null; + if (existsSync(join(dir, "kill.json"))) { + const evidence = JSON.parse( + readFileSync(join(dir, "kill.json"), "utf8"), + ) as { run: string; candidate: string; beforeKillSeq: number }; + type Container = { + Id: string; + State: { StartedAt: string; Running: boolean }; + }; + const snapshots = ["before", "after-kill", "after-resume"].map( + (part) => + JSON.parse( + readFileSync(join(dir, `container-${part}.json`), "utf8"), + ) as Container, + ); + const target = accounts.runs.find( + (r) => r.plan.content.run === evidence.run, + )!; + const candidatePlans = accounts.runs.filter( + (r) => + r.plan.content.purpose === "forward" && + r.plan.content.object === evidence.candidate, + ); + const atKill = JSON.parse( + readFileSync(join(dir, "ledger-at-kill.json"), "utf8"), + ) as LedgerRecord[]; + const killSeq = atKill.at(-1)!.seq; + kill = { + evidence, + killSeq, + snapshots, + charged: target.starts.length, + resultSeq: target.results[0]?.seq ?? null, + checks: { + sameContainer: + snapshots[0]!.State.Running && + snapshots.every( + (c) => + c.Id === snapshots[0]!.Id && + c.State.StartedAt === snapshots[0]!.State.StartedAt, + ), + chargedOnceNoRerun: + target.starts.length === 1 && candidatePlans.length === 1, + resultAndContinued: + !ofKind(atKill, "runResult").some( + (r) => r.content.run === evidence.run, + ) && + target.results.length === 1 && + target.results[0]!.seq > killSeq && + accounts.checks.adoptedAndStopped, + }, + }; + } + return { + dir, + study: config.studyId, + scores, + accounts, + lightChecks, + kill, + pass: + Object.values(accounts.checks).every(Boolean) && + lightChecks.every((c) => c.pass) && + (kill === null || Object.values(kill.checks).every(Boolean)), + }; +}); +const references = [ + "16-real-1", + "15-real-1", + "15-tight-2", + "14-real-1", + "12-real-2", +].flatMap((name) => referenceScores(join(referenceRoot, name, "board.sqlite"))); +jsonFile(join(output, "stats.json"), summaries); +jsonFile( + join(output, "predictions-scores.json"), + summaries.flatMap((s) => s.scores), +); +jsonFile(join(output, "reference-scores.json"), references); +jsonFile(join(output, "score-comparison.json"), { + references, + ours: summaries.flatMap((s) => s.scores), + caveat: + "参考项目提示词带题目具体做法、方法版本也不同;保留无效成绩及重复尝试,不作公平因果比较", +}); +jsonFile(join(output, "manual-review.json"), { + implementation: + "每个已启动候选一个全新上下文:协议逐条、基线、最终代码、concerns、位置表;逐项漏做/额外改进/位置表准确性和代码位置", + verification: + "同一审阅者再读各次核验及对应版本:误放行/误拒、修复是否仅改指出处;保存原始回复", + route: + "逐候选按任务书15附录一路线归类,并单列完整高上限路线的实测-q5;不自动按关键词贴标签", + comparison: + "复用eval/route导出、items、thinking、verify-ranges、原四问Jev及原阅读问卷,填A–J;与任务15三次历史结果并排,标历史批次", + serial: + "核对各目录launches.jsonl、machine.jsonl和父会话观察;上一份全部结束后才起下一份;不宣称快照排除任意外部进程", +}); +console.log( + `统计:${output};程序检查${summaries.every((s) => s.pass) ? "通过" : "未通过"};质量判断仍待独立阅读。`, +); +if (!summaries.every((s) => s.pass)) process.exitCode = 1; diff --git a/eval/lib/forward.ts b/eval/lib/forward.ts new file mode 100644 index 0000000..57c2158 --- /dev/null +++ b/eval/lib/forward.ts @@ -0,0 +1,171 @@ +import { writeFileSync } from "node:fs"; +import { checkConfig } from "../../src/config/index.ts"; +import type { LedgerRecord } from "../../src/ledger/index.ts"; +import { + Board, + FORWARD_NOT_RUN, + type StudyRecord, +} from "../../src/materials/index.ts"; +import { placement, runBalance } from "../../src/rules/index.ts"; +import { verificationOutcome } from "../../src/submissions/index.ts"; +import type { Range } from "../../src/vocab/index.ts"; +import { finalSubmissions, ofKind } from "./ledger.ts"; + +export const FORWARD_STOP = "消融还没写(任务书 17)"; +export const jsonFile = (path: string, value: unknown): void => + writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`); +export const frozenConfig = (records: LedgerRecord[]) => + checkConfig((ofKind(records, "config")[0]!.content as StudyRecord).config); + +/** 区间边界只是报告口径,不参与研究决定;失败或测不到分数保持 null。 */ +export function scoreBand(score: number | null, range: Range): string | null { + if (score === null) return null; + return { + belowQ1: "q9", + }[placement(range, score)]; +} + +/** 核验要对应当时那版代码,不能拿最终修好后的代码替代历史被审代码。 */ +export function candidateHistory(records: LedgerRecord[], candidate: string) { + const code = ofKind(records, "code").filter( + (r) => r.content.object === candidate, + ); + const submissions = ofKind(records, "submission").filter( + (r) => r.content.target === candidate, + ); + const attempts = submissions.flatMap((r) => { + if (r.content.kind !== "implementation" && r.content.kind !== "repair") + return []; + // flow 在同一事务里先写 submission,紧接着写 code。 + return [{ submission: r, code: code.find((c) => c.seq === r.seq + 1)! }]; + }); + const verifications = submissions.flatMap((r) => + r.content.kind === "verification" + ? [ + { + submission: r, + code: code.filter((c) => c.seq < r.seq).at(-1)!, + outcome: verificationOutcome(r.content.content), + }, + ] + : [], + ); + return { attempts, verifications }; +} + +/** 程序验收只读记录重算;基线也算正式运行,没启动的结果不扣次。 */ +export function forwardAccounts(records: LedgerRecord[]) { + const board = new Board(records); + const config = frozenConfig(records); + const opens = ofKind(records, "stageOpen"); + const closes = ofKind(records, "stageClose"); + const stages = opens.map((r) => { + const closing = closes.filter( + (c) => c.content.stageId === r.content.stageId, + ); + return { + ...r.content, + openSeq: r.seq, + closeSeqs: closing.map((c) => c.seq), + pass: closing.length === 1 && closing[0]!.seq > r.seq, + }; + }); + const orphanCloses = closes.filter( + (c) => !opens.some((o) => o.content.stageId === c.content.stageId), + ); + const started = ofKind(records, "runStarted"); + const balance = runBalance(config.runs, board.runStates()); + const runs = ofKind(records, "runPlanned").map((p) => ({ + plan: p, + starts: started.filter((s) => s.content.run === p.content.run), + results: ofKind(records, "runResult").filter( + (r) => r.content.run === p.content.run, + ), + })); + const batch = board.decision("batch", 1); + const finals = finalSubmissions(records); + const candidates = (batch?.output.chosen ?? []).map((candidate) => { + const history = candidateHistory(records, candidate); + const events = ofKind(records, "event").filter( + (r) => + r.content.event === FORWARD_NOT_RUN && + r.content.objects.includes(candidate), + ); + const candidateRuns = runs.filter( + (r) => + r.plan.content.purpose === "forward" && + r.plan.content.round === 1 && + r.plan.content.object === candidate, + ); + const verificationCount = finals.filter( + (r) => + r.content.target === candidate && r.content.kind === "verification", + ).length; + const repairCount = finals.filter( + (r) => r.content.target === candidate && r.content.kind === "repair", + ).length; + const lastPassed = history.verifications.at(-1)?.outcome.passed ?? null; + const exhausted = lastPassed === false && repairCount >= config.maxRepairs; + return { + candidate, + verificationCount, + repairCount, + lastPassed, + exhausted, + notRunEvents: events, + started: candidateRuns.some((r) => r.starts.length > 0), + runs: candidateRuns, + pass: exhausted + ? events.length === 1 && candidateRuns.length === 0 + : lastPassed === true && + candidateRuns.some( + (r) => r.starts.length === 1 && r.results.length === 1, + ), + }; + }); + const adoption = + ofKind(records, "decision").find( + (r) => r.content.decision === "adopt" && r.content.round === 1, + ) ?? null; + const halt = + ofKind(records, "event") + .filter((r) => r.content.event === "halted") + .at(-1) ?? null; + const checks = { + stagesPaired: stages.every((s) => s.pass) && orphanCloses.length === 0, + chargedOnce: + balance.charged === started.length && + config.runs - balance.balance === started.length && + balance.reserved === 0, + forwardSerial: runs + .filter((r) => r.plan.content.purpose === "forward") + .every( + (r, i, forwards) => + i === 0 || + (forwards[i - 1]!.results.length === 1 && + forwards[i - 1]!.results[0]!.seq < r.plan.seq), + ), + runsFinished: runs.length > 0 && runs.every((r) => r.results.length === 1), + candidatesFinished: + candidates.length > 0 && candidates.every((c) => c.pass), + adoptedAndStopped: + adoption !== null && + halt?.content.reason === FORWARD_STOP && + halt.seq === records.at(-1)!.seq, + }; + return { + initial: config.runs, + confirmedStarts: started.length, + balance, + runs, + candidates, + stages, + orphanCloses, + adoption, + halt, + checks, + }; +} diff --git a/eval/lib/light-checks.ts b/eval/lib/light-checks.ts new file mode 100644 index 0000000..049bb77 --- /dev/null +++ b/eval/lib/light-checks.ts @@ -0,0 +1,77 @@ +import { readdirSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +/** Pi 会话里的 research_check:请求和结果按 toolCallId 配对,不把超限请求算成执行。 */ +export function lightCheckUsage(sessionDir: string, maxCount: number) { + type Part = { type: string; id?: string; name?: string; text?: string }; + type Message = { + role: string; + toolCallId?: string; + toolName?: string; + isError?: boolean; + content: Part[]; + }; + const requests: Array<{ id: string; file: string; line: number }> = []; + const results = new Map< + string, + { file: string; line: number; message: Message } + >(); + const files = readdirSync(sessionDir) + .filter((f) => f.endsWith(".jsonl")) + .sort(); + if (files.length === 0) throw new Error(`没有会话文件:${sessionDir}`); + for (const file of files) { + readFileSync(join(sessionDir, file), "utf8") + .split("\n") + .forEach((line, index) => { + if (!line.trim()) return; + const entry = JSON.parse(line) as { type: string; message?: Message }; + const m = entry.message; + if (entry.type !== "message" || m === undefined) return; + if (m.role === "assistant") { + for (const p of m.content) + if (p.type === "toolCall" && p.name === "research_check") + requests.push({ id: p.id!, file, line: index + 1 }); + } else if (m.role === "toolResult" && m.toolName === "research_check") + results.set(m.toolCallId!, { file, line: index + 1, message: m }); + }); + } + const checks = requests.map((request) => { + const result = results.get(request.id); + if (result === undefined) + return { request, result: null, status: "missingResult", used: null }; + const text = result.message.content + .filter((p) => p.type === "text") + .map((p) => p.text) + .join("\n"); + const source = { file: result.file, line: result.line, text }; + if (result.message.isError) + return { request, result: source, status: "toolError", used: null }; + if (text.startsWith("本次调用的轻量检查已用完")) + return { request, result: source, status: "deniedLimit", used: null }; + const value = JSON.parse(text) as { + used: number; + maxCount: number; + seconds: number; + result: unknown; + }; + return { request, result: source, status: "executed", used: value.used }; + }); + const used = checks.flatMap((c) => (c.used === null ? [] : [c.used])); + const unresolved = checks.some( + (c) => c.status === "missingResult" || c.status === "toolError", + ); + return { + sessionDir, + maxCount, + requested: checks.length, + executed: used.length, + deniedLimit: checks.filter((c) => c.status === "deniedLimit").length, + // 工具报错或缺结果时不能猜实际执行量,也不当作 0 次通过。 + pass: + !unresolved && + used.length <= maxCount && + used.every((n, i) => n === i + 1), + checks, + }; +} diff --git a/eval/lib/reference-scores.ts b/eval/lib/reference-scores.ts new file mode 100644 index 0000000..4f5d018 --- /dev/null +++ b/eval/lib/reference-scores.ts @@ -0,0 +1,70 @@ +import { basename, dirname } from "node:path"; +import { DatabaseSync } from "node:sqlite"; + +/** 只读已有副本;保留 incomplete 的数值及重复尝试,不冒充有效成绩。 */ +export function referenceScores(path: string) { + const study = basename(dirname(path)); + const db = new DatabaseSync(path, { readOnly: true }); + try { + type Fact = { seq: number; content: T }; + const facts = (kind: string): Fact[] => + ( + db + .prepare( + "SELECT created_seq, payload FROM facts WHERE kind = ? ORDER BY created_seq", + ) + .all(kind) as Array<{ created_seq: number; payload: string }> + ).map((r) => ({ + seq: r.created_seq, + content: JSON.parse(r.payload) as T, + })); + const links = facts<{ + purpose: string; + roundId: string; + runId: string; + candidateId: string; + }>("run-link"); + const observations = facts<{ + runId: string; + status: string; + finalMetric: { value: number | null }; + reasons: string[]; + }>("run-observation"); + const adoption = facts<{ + roundId: string; + adoptedRunId: string | null; + newBaselineId: string; + }>("adoption").find((r) => r.content.roundId === `${study}.r1`)!; + return links + .filter( + (r) => + r.content.purpose === "forward" && + r.content.roundId === `${study}.r1`, + ) + .map((link) => { + const observation = observations.find( + (r) => r.content.runId === link.content.runId, + )!; + return { + source: path, + study, + candidate: link.content.candidateId, + run: link.content.runId, + status: observation.content.status, + rawScore: observation.content.finalMetric.value, + validScore: + observation.content.status === "valid" + ? observation.content.finalMetric.value + : null, + adopted: adoption.content.adoptedRunId === link.content.runId, + newBaseline: adoption.content.newBaselineId, + linkSeq: link.seq, + observationSeq: observation.seq, + adoptionSeq: adoption.seq, + reasons: observation.content.reasons, + }; + }); + } finally { + db.close(); + } +} diff --git a/package.json b/package.json index 9b274cd..2538b31 100644 --- a/package.json +++ b/package.json @@ -27,7 +27,10 @@ "eval:route:extract": "node eval/route/extract.ts", "eval:route:items": "node eval/route/items.ts", "eval:route:jev": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/route/jevroutes.ts", - "eval:route:thinking": "node eval/route/thinking.ts" + "eval:route:thinking": "node eval/route/thinking.ts", + "eval:forward:run": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/forward/run.ts", + "eval:forward:export": "node eval/forward/export.ts", + "eval:forward:stats": "node eval/forward/stats.ts" }, "dependencies": { "@earendil-works/pi-ai": "0.85.1", diff --git a/tests/forward-eval.test.ts b/tests/forward-eval.test.ts new file mode 100644 index 0000000..c47667f --- /dev/null +++ b/tests/forward-eval.test.ts @@ -0,0 +1,354 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { test } from "node:test"; +import { + candidateHistory, + forwardAccounts, + FORWARD_STOP, + scoreBand, +} from "../eval/lib/forward.ts"; +import { lightCheckUsage } from "../eval/lib/light-checks.ts"; +import { referenceScores } from "../eval/lib/reference-scores.ts"; +import { Ledger } from "../src/ledger/index.ts"; +import { forwardNotRunEvent } from "../src/materials/index.ts"; +import { createRepository, saveVersion } from "../src/workspace/index.ts"; +import { fullConfig } from "./config-helpers.ts"; +import { + definition, + exited, + implementation, + notStarted, + proposal, + tempDir, +} from "./ledger-helpers.ts"; +import { judged, protocolOf, Writer } from "./materials-helpers.ts"; + +const verification = (found: boolean) => ({ + clauses: [{ clause: "c1", found, where: "solve.py:1" }], + factors: [{ factor: "p1", found, where: "solve.py:1" }], + safeguards: [], + unassigned: [], + body: "逐项核对", +}); + +test("forward 导出/统计:历史代码配对、两次核验一次修复、封存自评与真实分数保持可追溯", (t) => { + const root = tempDir(t), + dir = join(root, "run"), + study = join(dir, "study"); + mkdirSync(study, { recursive: true }); + const ledger = Ledger.open(join(study, "ledger.sqlite")); + t.after(() => ledger.close()); + const w = new Writer(ledger), + config = fullConfig(); + ledger.append("config", { config }); + const repo = join(study, "repo.git"), + work = join(root, "work"); + createRepository(repo); + mkdirSync(work); + const version = (text: string) => { + writeFileSync(join(work, "solve.py"), text); + const result = saveVersion(repo, work, []); + assert.ok(result.saved); + return result.version; + }; + ledger.append("code", { + object: "base", + version: version("value = 0\n"), + locations: null, + }); + w.run("base-1", "baseline", 0, null, judged(1, "main")); + w.open("proposal", "proposal", 1, ["a"]); + const p = proposal({ q1: 1, q5: 2, q9: 3 }, "协议正文"); + w.sub("proposal", "a", "proposal", null, "p", p); + w.close("proposal"); + w.protocol("r1-c1", protocolOf("r1-c1", p, 1, "base", ["p"], ["改value"])); + w.open("second", "secondPrediction", 1, ["a"]); + w.sub("second", "a", "secondPrediction", "r1-c1", null, { + range: { q1: 1, q5: 2, q9: 3 }, + body: "估计", + }); + w.close("second"); + w.decision("batch", 1, null, { + chosen: ["r1-c1"], + candidates: [{ id: "r1-c1", merged: { q1: 1, q5: 2, q9: 3 } }], + }); + const v1 = version("value = 1\n"), + v2 = version("value = 2\n"); + w.open("impl", "implementation", 1, ["x"]); + w.sub("impl", "x", "implementation", "r1-c1", null, implementation("首稿")); + ledger.append("code", { object: "r1-c1", version: v1, locations: null }); + w.close("impl"); + w.open("verify1", "verification", 1, ["v"]); + w.sub("verify1", "v", "verification", "r1-c1", null, verification(false)); + w.close("verify1"); + w.open("repair", "implementation", 1, ["x"]); + w.sub("repair", "x", "repair", "r1-c1", null, implementation("只修value")); + ledger.append("code", { object: "r1-c1", version: v2, locations: null }); + w.close("repair"); + w.open("verify2", "verification", 1, ["v"]); + w.sub("verify2", "v", "verification", "r1-c1", null, verification(true)); + w.close("verify2"); + ledger.append("runPlanned", { + run: "forward-1", + purpose: "forward", + round: 1, + object: "r1-c1", + definition, + }); + ledger.append("runStarted", { run: "forward-1" }); + const atKill = ledger.records(); + writeFileSync(join(dir, "ledger-at-kill.json"), JSON.stringify(atKill)); + writeFileSync( + join(dir, "kill.json"), + JSON.stringify({ + run: "forward-1", + candidate: "r1-c1", + beforeKillSeq: atKill.at(-1)!.seq, + }), + ); + for (const part of ["before", "after-kill", "after-resume"]) + writeFileSync( + join(dir, `container-${part}.json`), + JSON.stringify({ + Id: "same-container", + State: { Running: true, StartedAt: "2026-10-04T00:00:00Z" }, + }), + ); + ledger.append("runResult", { + run: "forward-1", + process: exited, + judged: judged(2.5, "main"), + }); + w.decision("adopt", 1, null, { adopted: "r1-c1" }); + ledger.append("event", { + event: "halted", + objects: [], + reason: FORWARD_STOP, + }); + const records = ledger.records(), + accounts = forwardAccounts(records); + assert.deepEqual(accounts.balance, { + charged: 2, + reserved: 0, + balance: config.runs - 2, + }); + assert.ok(Object.values(accounts.checks).every(Boolean)); + assert.equal(accounts.candidates[0]!.verificationCount, 2); + assert.equal(accounts.candidates[0]!.repairCount, 1); + assert.deepEqual( + candidateHistory(records, "r1-c1").verifications.map( + (v) => v.code.content.version, + ), + [v1, v2], + ); + const exported = join(root, "export"); + const exportResult = spawnSync( + process.execPath, + ["eval/forward/export.ts", study, exported], + { encoding: "utf8" }, + ); + assert.equal(exportResult.status, 0, exportResult.stderr); + assert.equal( + readFileSync(join(exported, "r1-c1/final/solve.py"), "utf8"), + "value = 2\n", + ); + assert.match( + readFileSync(join(exported, "r1-c1/attempts.json"), "utf8"), + /只修value/, + ); + assert.match( + readFileSync(join(exported, "r1-c1/final-implementation.json"), "utf8"), + /concerns/, + ); + const refs = join(root, "refs"); + mkdirSync(refs); + for (const name of [ + "16-real-1", + "15-real-1", + "15-tight-2", + "14-real-1", + "12-real-2", + ]) + referenceFixture(join(refs, name)); + const statsOut = join(root, "stats"); + const statsResult = spawnSync( + process.execPath, + ["eval/forward/stats.ts", statsOut, refs, dir], + { encoding: "utf8" }, + ); + assert.equal(statsResult.status, 0, statsResult.stderr); + const scores = JSON.parse( + readFileSync(join(statsOut, "predictions-scores.json"), "utf8"), + ); + assert.deepEqual(scores[0].authorEstimates[0].range, { q1: 1, q5: 2, q9: 3 }); + assert.equal(scores[0].runs[0].measuredMinusQ5, 0.5); + assert.equal(scores[0].runs[0].band, "[q5,q9]"); + const statistics = JSON.parse( + readFileSync(join(statsOut, "stats.json"), "utf8"), + ); + assert.deepEqual(statistics[0].kill.checks, { + sameContainer: true, + chargedOnceNoRerun: true, + resultAndContinued: true, + }); + writeFileSync( + join(dir, "container-after-resume.json"), + JSON.stringify({ + Id: "replacement", + State: { Running: true, StartedAt: "2026-10-04T00:01:00Z" }, + }), + ); + const badOut = join(root, "bad-stats"); + const bad = spawnSync( + process.execPath, + ["eval/forward/stats.ts", badOut, refs, dir], + { encoding: "utf8" }, + ); + assert.equal(bad.status, 1, bad.stderr); + assert.equal( + JSON.parse(readFileSync(join(badOut, "stats.json"), "utf8"))[0].kill.checks + .sameContainer, + false, + ); +}); + +test("forward 账目:没启动退还、修复用尽不扣次;缺不跑事件或结果不能通过", (t) => { + const ledger = Ledger.open(join(tempDir(t), "ledger.sqlite")); + t.after(() => ledger.close()); + const w = new Writer(ledger), + config = { ...fullConfig(), maxRepairs: 0 }; + ledger.append("config", { config }); + ledger.append("runPlanned", { + run: "base-0", + purpose: "baseline", + round: 0, + object: null, + definition, + }); + ledger.append("runResult", { + run: "base-0", + process: notStarted, + judged: null, + }); + w.run("base-1", "baseline", 0, null, judged(1, "main")); + w.decision("batch", 1, null, { chosen: ["c"] }); + ledger.append("code", { object: "c", version: "v", locations: null }); + w.open("v", "verification", 1, ["v"]); + w.sub("v", "v", "verification", "c", null, verification(false)); + w.close("v"); + assert.equal(forwardAccounts(ledger.records()).candidates[0]!.pass, false); + ledger.append("event", forwardNotRunEvent("c", "修复用尽")); + w.decision("adopt", 1, null, { adopted: null }); + ledger.append("event", { + event: "halted", + reason: FORWARD_STOP, + objects: [], + }); + const accounts = forwardAccounts(ledger.records()); + assert.deepEqual(accounts.balance, { + charged: 1, + reserved: 0, + balance: config.runs - 1, + }); + assert.ok(Object.values(accounts.checks).every(Boolean)); + ledger.append("runPlanned", { + run: "unfinished", + purpose: "forward", + round: 1, + object: "c", + definition, + }); + ledger.append("runStarted", { run: "unfinished" }); + assert.equal(forwardAccounts(ledger.records()).checks.runsFinished, false); + assert.equal(forwardAccounts(ledger.records()).candidates[0]!.pass, false); +}); + +test("forward 分位区间不把无效分数当零,三条边界按既有placement", () => { + assert.deepEqual( + [null, 0, 1, 2, 3, 4].map((s) => scoreBand(s, { q1: 1, q5: 2, q9: 3 })), + [null, "q9"], + ); +}); + +test("forward 轻量检查:跨会话按工具编号配对,拒绝超限不算执行,缺结果不伪装零次", (t) => { + const dir = tempDir(t); + const request = (id: string) => ({ + type: "message", + message: { + role: "assistant", + content: [{ type: "toolCall", name: "research_check", id }], + }, + }); + const result = (id: string, text: string) => ({ + type: "message", + message: { + role: "toolResult", + toolName: "research_check", + toolCallId: id, + content: [{ type: "text", text }], + }, + }); + writeFileSync(join(dir, "1.jsonl"), JSON.stringify(request("a")) + "\n"); + writeFileSync( + join(dir, "2.jsonl"), + [ + result( + "a", + JSON.stringify({ used: 1, maxCount: 1, seconds: 5, result: {} }), + ), + request("b"), + result("b", "本次调用的轻量检查已用完(上限 1 次),不能再查。"), + ] + .map((r) => JSON.stringify(r)) + .join("\n"), + ); + const usage = lightCheckUsage(dir, 1); + assert.equal(usage.requested, 2); + assert.equal(usage.executed, 1); + assert.equal(usage.deniedLimit, 1); + assert.equal(usage.pass, true); + writeFileSync(join(dir, "3.jsonl"), JSON.stringify(request("c"))); + assert.equal(lightCheckUsage(dir, 1).pass, false); +}); + +function referenceFixture(dir: string) { + mkdirSync(dir); + const db = new DatabaseSync(join(dir, "board.sqlite")); + db.exec("CREATE TABLE facts (created_seq INTEGER, kind TEXT, payload TEXT)"); + const study = dir.split("/").at(-1)!; + const insert = (seq: number, kind: string, payload: unknown) => + db + .prepare("INSERT INTO facts VALUES (?,?,?)") + .run(seq, kind, JSON.stringify(payload)); + insert(1, "run-link", { + purpose: "forward", + roundId: `${study}.r1`, + runId: "r", + candidateId: "c", + }); + insert(2, "run-observation", { + runId: "r", + status: "incomplete", + finalMetric: { value: 99 }, + reasons: ["无效"], + }); + insert(3, "adoption", { + roundId: `${study}.r1`, + adoptedRunId: null, + newBaselineId: "B0", + }); + db.close(); +} + +test("forward 参考分数:incomplete 原值保留但有效分数为空", (t) => { + const dir = join(tempDir(t), "reference"); + referenceFixture(dir); + const rows = referenceScores(join(dir, "board.sqlite")); + assert.equal(rows[0]!.rawScore, 99); + assert.equal(rows[0]!.validScore, null); + assert.equal(rows[0]!.adopted, false); + assert.equal(rows[0]!.observationSeq, 2); +});