feat: flow 第 1 部分（骨架、研究开始、第零轮）与第零轮检验
按任务书 12 实现 src/flow：从账本现算下一步的纯函数、主循环、恢复入口、派模型任务和跑正式运行两个共用动作、冻结配置、基线、第零轮三个研究者阶段和入口分配，走到第 1 轮时停下。materials 导出 finalEntries、加 researcherId 和 StudyRecord。 审阅采纳两条：研究编号本机已用过时冻结前报错；docker run 没启动时写完结果就停下，不在本进程里无限起新编号。 eval/ 新增第零轮检验：起研究命令行、回放假模型、检验用 Jev 客户端、六问质量评估、16-real-1 回放、杀进程与两份同时跑。tsc 与 depcruise 也检查 eval/，新增 eval 只经 index.ts 用正式代码的规则。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+10
-1
@@ -1,5 +1,6 @@
|
||||
// 模块之间的依赖规则，来源是 docs/结构.md 第二部分（模块表）和第六部分（检查规则）。
|
||||
// 模块按 src/<模块名>/ 认；新建模块目录后自动受这些规则检查。规则只管 src 下的模块，tests/ 不受"只有某模块能用某外部包"的限制。
|
||||
// 模块按 src/<模块名>/ 认；新建模块目录后自动受这些规则检查。规则主要管 src 下的模块，tests/、eval/ 不受"只有某模块能用某外部包"的限制；
|
||||
// eval/ 另有一条：只能经 src/<模块>/index.ts 用正式代码（任务书 12 第三部分第 4 条）。
|
||||
|
||||
// 模块表"依赖"一列：每个模块除了自己和 vocab 以外，还能引用哪些模块。plugin 装配时可用各模块。
|
||||
const ALLOWED = {
|
||||
@@ -76,6 +77,14 @@ module.exports = {
|
||||
from: { path: "^src/" },
|
||||
to: { pathNot: ["^src/", "^node_modules/"], dependencyTypesNot: ["core"] },
|
||||
},
|
||||
{
|
||||
name: "eval-through-index",
|
||||
comment:
|
||||
"eval/ 像用户一样用正式代码，只能经 src/<模块>/index.ts 引用（CLAUDE.md\"测试\"：正式代码和检验代码完全分开）",
|
||||
severity: "error",
|
||||
from: { path: "^eval/" },
|
||||
to: { path: "^src/", pathNot: "^src/[^/]+/index\\.ts$" },
|
||||
},
|
||||
{
|
||||
name: "pure-allowlist",
|
||||
comment:
|
||||
|
||||
+4
-6
@@ -22,6 +22,7 @@
|
||||
- 任务书 09（materials）：src/materials 12 个文件、1831 行（预期 1200–1800；超出部分主要是用户决定 2 新加的 shapes.ts 116 行，复审判定没有专门路径增多），vocab 加 `Checklist`（1105 行，上限 1150）。19 份提示词以参考项目为底稿只做必要改动搬进 `prompts/`；TASK.md 补了 `unversionedNames` 不存进版本。`npm test` 356 条通过（materials 39 条）。全新上下文审阅三轮（可见性一轮、规模与正确性一轮、修复复审一轮），materials 内部没有发现泄露；要靠 flow 写入时机保证的几条记进"未完成"的"给 flow"。
|
||||
- 任务书 10（submissions）：先改 materials 第一部分 7 处，另加两处小改动（jev 判重材料带关键设定、核验 `body` 字段说明补同行提醒）。src/submissions 6 个文件、391 行（预期 400–600）；materials 1850 行；vocab 1107 行；ledger 退回记录加对象字段，表结构版本升到 4。`npm test` 400 条通过；`npm run test:jev` 新增三个真实用例：写得好的回应收下（三问 0.06 / 0.04 / 0.04），改评测的回应退回（changes_measurement 0.91，阈值 0.21），实现 3 块 15 条判断全否。全新上下文审阅一轮：采纳消融实现退回文字前后矛盾、两处测试加强；另发现同轮指标重名，用户定的改法见"未完成"。结构.md 第七、八部分已更新。
|
||||
- 任务书 11（收尾，提交 13a7e60）：假设专用指标冻结时加候选前缀，全名"候选编号.短名"，只在 materials 的 `freezeProtocol` 加；短名恢复不能含 `.`（最长 64），已占用清单只剩任务的指标和两个保留键。消融格核验不通过重做时，材料写"工作目录是上一次交的这一格代码的副本"，能读的版本加上它，完整代码照旧列出；消融实现提示词同步改。jev 判重的材料说明补 `keySettings` 一句。materials 1868 行（上限 1850，多出主要是 `freezeProtocol`），submissions 391 行，vocab 1118 行。`npm test` 405 条通过；`npm run test:jev -- jev-latest` 判重一问照常拿回概率。结构.md 第八部分 vocab、materials、jev 已更新。
|
||||
- 任务书 12（flow 第 1 部分：骨架、研究开始、第零轮）：src/flow 11 个文件、775 行（预期 900–1200；实验员、核验员会话接法和多份提交的编号要用到第 2、3 部分才写），materials 1897 行（+29：`finalEntries`、`researcherId`、`StudyRecord`），eval/ 1006 行。`npm test` 418 条通过（新增 flow 下一步 11 条、真 Docker 2 条：研究编号重复报错、docker run 没启动后停下不在本进程重跑）。全新上下文审阅 2.5、2.7 一轮，采纳两条（同上两条测试对应的修复）。真实检验：16-real-1 回放分配与当年完全相同、0 退回；三处杀进程、两份同时跑全部通过。**研究质量未通过**：3 次真实运行（`/home/iomgaa/inforsi-eval/2026-10-03-r0-real-1`、`-2`、`-3`，2、3 被内存回收杀过一次、按正式恢复路径续跑完）第 1、3 问都不通过，入口全体没有 K1（联合优化）；材料核对符合方法.md 4.2，交用户判断。
|
||||
|
||||
## 关键决定（用户原话见各提交说明）
|
||||
|
||||
@@ -69,15 +70,12 @@
|
||||
- `inbox/调研.md`：要迁进 `docs/调研/` 的调研结论。
|
||||
- `docs/方法.md` 里还有 7 条标〔自定〕的规则，待用户过目。
|
||||
- 给 flow：同一轮里选批和选格都用随机规则时，传不同的"版本"算种子，否则两次抽取用同一串随机数。
|
||||
- 给 flow：确认启动前要先登记运行；模型调用要先开始再结束；阶段要先开再关；ledger 都不查。
|
||||
- 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。
|
||||
- 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。
|
||||
- 给 flow：容器名要在本机所有研究之间唯一（例如带研究编号）；`docker run` 因重名失败时 runner 按"启动了"处理。日志目录每次运行一个新的。
|
||||
- 给 flow：取出时目标目录必须是空的，workspace 不查。
|
||||
- 给 flow（materials 审阅）：冻结协议只在回应阶段关闭后写进账本，阶段内给 Jev 查的协议只放内存；解封只在本轮收尾阶段关闭后写；阶段关闭只在名单上的人交齐后写。materials 把这几条记录的出现当作"已公开"，提前写会泄露（审阅者在临时账本里实测过）。
|
||||
- 给 flow（materials 审阅）：冻结协议只在回应阶段关闭后写进账本，阶段内给 Jev 查的协议只放内存；解封只在本轮收尾阶段关闭后写。materials 把这几条记录的出现当作"已公开"，提前写会泄露（审阅者在临时账本里实测过）。
|
||||
- 给 flow：没采纳时也写一条采纳决定（adopted 为 null），本轮正向运行靠它公开；修复任务要在核验阶段关闭后再发。
|
||||
- 给 flow：消融格核验不通过、修复次数用完时写 `cellNotRunEvent(格, 原因)` 事件，结算记 insufficient。flow 重做时，把上一次这一格的提交取出来当工作目录（materials 的消融实现材料就是这样写的）。
|
||||
- 给 flow：修复的差异块按本批基线算（任务书 10 第二部分第 2 步），切好交给 submissions；形状不合格的提交由 agents 报给提交回调，flow 直接记退回，不经 submissions。
|
||||
- 给 flow：修复的差异块按本批基线算（任务书 10 第二部分第 2 步），切好交给 submissions。
|
||||
- 给 flow：消融实现交了 `cannotRun` 时 `known.code` 传 null，否则传存版本的结果。
|
||||
- 给 flow：问 Jev 查代码改动时，冻结协议里引用的指标要换成原登记的定义（materials 的实验员材料里已有这一步，写 flow 时把它导出共用，不另写一份）。
|
||||
- 给以后换任务镜像：镜像里必须有 `setpriv`（runner 用它把命令降到当前用户）；容器里不要用 `python -I`，它会忽略 `PYTHON_CPU_COUNT`。
|
||||
@@ -93,4 +91,4 @@
|
||||
|
||||
## 下一步
|
||||
|
||||
已完成模块收尾完毕，现在由用户和监督者一起过 flow。flow 分四部分写、各自验证，拆分、验证办法和对照用的记录见 `docs/结构.md` 第三部分"flow 分四部分写，各自验证"。用户还要求每部分也拿参考项目真实验收运行的记录（`/home/iomgaa/rb-acceptance/*/board.sqlite`）当对照，原话："我们也可以把我们的记录作为一个标杆，然后去匹配"。部分 1 已写成任务书 12（`任务书/12-flow-1-骨架与第零轮.md`），等用户另开会话执行，之后监督者核对。
|
||||
任务书 12 已做完并提交，等监督者核对。第零轮研究质量 3 次都没通过（入口里没有联合优化），是否改提示词或方法由用户和监督者定；定之前不写任务书 13。
|
||||
|
||||
+38
-2
@@ -525,15 +525,16 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- 核验：不放 Jev 对这份实现的判断。能走到核验说明 Jev 已判过，Jev 是判别条件，不是给核验员的提示。
|
||||
- 消融实现：这一格已有公开的消融核验（核验不通过、重做）时，和修复一样从上一次交的这一格代码接着改（用户 2026-10-03 定）：工作目录写"上一次交的这一格代码的副本"，能读的版本里加上它，完整代码照旧列出作对照；另放"退回意见"：最近一次核验里关错了的因子和核验员写的依据，写法同修复的退回意见。
|
||||
- 指标：冻结之后一律用全名"候选编号.短名"：已登记指标清单、对号清单的 `reusableMetrics`、实验员材料的"要实现的指标定义"（候选代码在结果文件里也用全名当键）。提案、回应、统一的对号清单里，已占用的指标编号只有任务的指标、`answer-path` 和 `safeguard-triggers` 两个保留键（用户 2026-10-03 定）；已登记的全名带 `.`，新登记的短名不带，不用列进来。
|
||||
- `finalEntries(ledger)`：按账本取的最终有效的入口（各人最后一版入口清单里的每个入口，带作者和正文）；flow 分配入口时用，和认领的对号清单是同一份（任务书 12 第三部分）。
|
||||
- `queryMaterial(material, id)`：只在这份材料的记录里按编号找，交回全文；找不到交回 null（agents 的查询回调据此答"看不到"）。
|
||||
- `readPrompts(dir)`：读 `roles/<角色>.md` 和 `tasks/<提交种类>.md`，交回文字和每个文件原始字节的 sha256；研究开始时 flow 把哈希冻结进研究配置记录（方法.md 4.1）。
|
||||
- `beliefBook(ledger, study, round)`：信念本全文（方法.md 6.10 五条），数字全部用 rules 现算，只写已公开的记录；`round` 是读者所在轮次，导出时写 null。选批历史的每次运行后面原样写出任务的附加记录（判分结果的 `extra`，合法化扣掉的分在里面，字段由任务定）。消融格记"证据不足"时，有运行的原因取判分，没有运行的（核验不通过、重做次数用完）原因取这一格的 `CELL_NOT_RUN` 事件。
|
||||
- `TaskMaterial`、`MaterialSection`、`MaterialRecord`、`ReadableVersion`（类型）：任务材料，以及其中的一节、一条可查询记录（编号、标题、全文）、一个能读的代码版本（版本号、说明、取出目录）。
|
||||
- `TaskRequest`、`StudyFacts`、`CodeChanges`、`Prompts`、`PromptFile`（类型）：flow 传进来的东西，全部必填；`CodeChanges` 与 workspace 的比较结果同形（照 jev `codeHunks` 的先例，不引用 workspace）。
|
||||
- 账本里只存成 Json 的内容的形状（用户 2026-10-03 决定 2：materials 定义，flow 照着写，ledger 不改）：`FrozenProtocol`（冻结协议：方法.md 4.3 ⑥ 列的因子、保障、指标、正文、实现条款，加轮次、出发的基线、合并了哪些原提案；不带答复，主体由 submissions 的 `protocolBody` 拼，指标编号是全名）、`LocationTable`（代码记录的位置表）、`DecisionKind`、`DECISION_KINDS`、`DecisionShapes`（六种决定：入口分配、判重合并、选批、采纳、选格、押的因子，输入输出用 rules 的类型；没采纳时也写一条采纳决定）、`DedupOutput`（判重合并的输出）、`Settlement`（结算）。
|
||||
- 账本里只存成 Json 的内容的形状（用户 2026-10-03 决定 2：materials 定义，flow 照着写，ledger 不改）：`StudyRecord`（研究配置记录：配置、步骤清单、提示词每个文件的哈希、镜像编号、任务事实、种子；配置和任务事实存 Json，flow 读回时用 `checkConfig`、`checkTaskFacts` 再检查，任务书 12 第三部分）、`FrozenProtocol`（冻结协议：方法.md 4.3 ⑥ 列的因子、保障、指标、正文、实现条款，加轮次、出发的基线、合并了哪些原提案；不带答复，主体由 submissions 的 `protocolBody` 拼，指标编号是全名）、`LocationTable`（代码记录的位置表）、`DecisionKind`、`DECISION_KINDS`、`DecisionShapes`（六种决定：入口分配、判重合并、选批、采纳、选格、押的因子，输入输出用 rules 的类型；没采纳时也写一条采纳决定）、`DedupOutput`（判重合并的输出）、`Settlement`（结算）。
|
||||
- `freezeProtocol(body, candidate, round, baseline, proposals)`：flow 写冻结协议时调用。输入 submissions 交回的冻结协议主体、候选编号、轮次、基线、合并了哪些原提案，交回 `FrozenProtocol`；新登记的假设专用指标改用全名"候选编号.短名"（例如 `r2-c1.restart-gain`），引用的本来就是全名，照抄。加前缀只在这一处：同一轮的研究者互相看不到提案，会起同一个短名、定义却不同，候选编号判重合并之后才有，所以只能在冻结时加（用户 2026-10-03 定）。
|
||||
- 事件：`CELL_NOT_RUN`、`cellNotRunEvent(格, 原因)`：一格消融核验不通过、重做次数用完而不跑时，flow 用它写事件记录（`objects` 只有这一格），结算记证据不足（方法.md 4.3 ⑯、6.6）。
|
||||
- 编号规则：`ORIGINAL_BASELINE`、`entryId`、`proposalId`、`critiqueId`、`candidateId`、`factorId`、`cellId`、`splitCell`、`explanationId`、`callId`，以及协议内的局部编号 `factorKey`（p1…）、`clauseKey`（c1…）。入口编号按作者一直往后编，讨论时整份替换也不复用旧编号。
|
||||
- 编号规则：`ORIGINAL_BASELINE`、`researcherId`（第 i 位研究者，从 0 数，编号 `researcher-<i+1>`，任务书 12 第三部分）、`entryId`、`proposalId`、`critiqueId`、`candidateId`、`factorId`、`cellId`、`splitCell`、`explanationId`、`callId`，以及协议内的局部编号 `factorKey`（p1…）、`clauseKey`（c1…）。入口编号按作者一直往后编，讨论时整份替换也不复用旧编号。
|
||||
|
||||
### agents
|
||||
|
||||
@@ -573,3 +574,38 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- `OriginalProposal`（类型）：回应拼协议时叠在下面的原提案：因子、保障、指标、正文，作者自评不进协议。
|
||||
- `ProtocolBody`（类型）：冻结协议的主体，即方法.md 4.3 ⑥ 列的因子、保障、指标、正文、实现条款；flow 用 materials 的 `freezeProtocol` 补上轮次、基线、合并了哪些原提案，写成 `FrozenProtocol`。
|
||||
- `Verdict`（类型）：判的结论：收下（带用到的全部 Jev 判断，没问 Jev 为 null；回应、统一另带拼好的协议主体，其余为 null），或退回（带原因和用到的 Jev 判断）。
|
||||
|
||||
### flow
|
||||
|
||||
下面是 `src/flow/index.ts` 导出的全部内容，以后 plugin 也只用它。flow 只经各模块的 `index.ts` 用它们；正式代码里没有为检验写的参数、环境变量、回放模式或假实现（CLAUDE.md"测试"）。这一节对应 flow 第 1 部分（任务书 12）：骨架、研究开始、第零轮，走到"开第 1 轮"停下。
|
||||
|
||||
- `runStudy(input)`：推进一份研究，直到停下。输入三样，都必填：研究目录；配置（新研究交 `checkConfig` 检查过的配置，接着跑一份已有的研究写 null，以账本里冻结的那份为准；两者对不上就报错）；"停下"信号。每次启动走同一条路：打开账本；新研究先冻结配置，本机已有 `inforsi-<研究编号>` 的镜像就说明研究编号用过了，报错停下，要换一个编号（编号重复的新研究会把旧研究遗留的同名容器当成自己的运行）；已有研究的，核对冻结的配置、镜像还在、提示词哈希没变（开发入口开着时提示词可以变）；给开始了没结束的模型调用补写结束（出错、"进程中断"、墙钟时间测不到）；按容器名问 Docker 核对登记了没结果的正式运行（没启动的写"没启动"退还名额，启动了的补写确认启动、等它、判分、写结果，问不到的写"状态未知"事件并停下）；然后进主循环：读账本 → 问"下一步做什么" → 做这一步 → 回到开头。主循环里 `docker run` 没启动的，写完"没启动"结果（退还名额、记下 Docker 报错原文）就停下，停下原因带上报错原文，本进程不重跑；下次启动时照旧起新编号重跑。恢复时补写的"没启动"不停下。
|
||||
- `StudyInput`（类型）：上面三样输入。
|
||||
- `StudyEnding`（类型）：两种结局之一：停下（附原因，同时写一条停下事件），或被用户停下（正在跑的模型会话立即停，正在跑的容器不管，下次启动由恢复入口接着等）。"研究结束"第 4 部分才有。
|
||||
|
||||
研究目录里放：`ledger.sqlite` 账本；`repo.git` 代码版本的裸仓库；`runs/<运行编号>/` 一次正式运行（取出的代码、运行目录、日志、判分目录、校验器日志）；`calls/<调用编号>/` 一次模型调用（工作目录、会话文件、能读的代码版本）。取出代码、建运行目录和判分目录时目标都是新建的空目录。容器名 `inforsi-<研究编号>-<运行编号>`，判分容器另加 `-judge`。同一份研究的全部运行用同一个种子：研究编号经 vocab 的 `hash` 取前 8 位十六进制转成整数，冻结进研究配置记录。
|
||||
|
||||
**每一步以哪条记录为准算做完**（任务书 12 第 2.2 节）。步骤的先后只写在 flow 的"下一步做什么"一个纯函数里；后面三部分每加一步，这张表加一行。
|
||||
|
||||
| 步骤（config 的 `STUDY_STEPS`） | 看到哪条记录算做完 |
|
||||
|---|---|
|
||||
| `freezeConfig` 冻结配置 | 研究配置记录（`config`） |
|
||||
| `baselineRun` 基线 | 基线代码记录（`code`，对象是 materials 的 `ORIGINAL_BASELINE`），而且最近一次基线运行的结果不是"没启动"；"没启动"的下次启动时起新编号（`base-2`……）重跑，判为不算数的停下 |
|
||||
| `entryDraft`、`discussion1`、`discussion2`、`claim` | 这个阶段的关闭记录（`stageClose`）；阶段编号 `r0-<步骤>`，只在名单上的人都有被收下的提交之后才关 |
|
||||
| `assignEntries` 分配 | 第 0 轮的入口分配决定（`decision`，种类 `entries`） |
|
||||
| 按配置跳过的步骤 | 一条"按配置跳过"事件（`skippedByConfig`），写明步骤和 config 给的原因 |
|
||||
|
||||
研究者阶段里没交上的人，按他在这个阶段最近一次调用怎么结束的来定：没有调用、被用户停下、出错（含"进程中断"）都派；没交的，开发入口关着就停下（交不出合格提交按内部错误，方法.md 4.3 ②），开着就再派。一次派发里有人没交上，等这次派发的会话全部结束后停下。
|
||||
|
||||
检验命令（`eval/`，真实运行，要 Docker；带 `*` 的会花模型或 Jev 的钱。输出都在仓库外 `/home/iomgaa/inforsi-eval/<日期-名字>/`，名字也当研究编号用，只能小写、不能重复）：
|
||||
|
||||
| 命令 | 用途 |
|
||||
|---|---|
|
||||
| `npm run eval:study -- --config <文件> --dir <目录>`、`-- --resume --dir <目录>` * | 起一份研究或接着跑，Ctrl-C 发停下信号 |
|
||||
| `npm run eval:r0:run -- <名字>` * | 用 `eval/round-zero/research.json` 真实跑一次第零轮 |
|
||||
| `npm run eval:r0:quality -- <检验目录>` * | 对第零轮产出问六问（任务书 12 第 5.2 节），原始概率写进 `<检验目录>/quality.json` |
|
||||
| `npm run eval:r0:extract` | 从参考项目 16-real-1 的账本重新提取第零轮提交，生成回放数据 |
|
||||
| `npm run eval:r0:replay -- <名字>` | 回放 16-real-1 的第零轮，比对分配结果 |
|
||||
| `npm run eval:r0:kill -- baseline\|drafting\|stageFull` | 在三处之一杀控制进程、重启跑完，逐条检查（用回放，不花钱） |
|
||||
| `npm run eval:r0:concurrent` | 两份研究同时跑到基线结束，检查各扣各的、容器不撞名 |
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import { parseArgs } from "node:util";
|
||||
import { loadConfig } from "../../src/config/index.ts";
|
||||
import { runStudy } from "../../src/flow/index.ts";
|
||||
|
||||
// 起研究的命令行入口：像用户一样给配置、指定研究目录、调 runStudy，按 Ctrl-C（SIGINT）时发"停下"信号。
|
||||
// 新研究：node eval/lib/cli.ts --config <research.json> --dir <研究目录>
|
||||
// 接着跑：node eval/lib/cli.ts --resume --dir <研究目录>（配置写 null，以账本里冻结的那份为准）
|
||||
// 结局打印成一行 "RESULT <json>"，检验脚本按这一行读。
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: { config: { type: "string" }, resume: { type: "boolean" }, dir: { type: "string" } },
|
||||
strict: true,
|
||||
});
|
||||
if (values.dir === undefined) throw new Error("要写 --dir <研究目录>");
|
||||
if ((values.config === undefined) === (values.resume !== true)) {
|
||||
throw new Error("--config <research.json> 和 --resume 二选一，必须写一个");
|
||||
}
|
||||
|
||||
const config = values.config === undefined ? null : loadConfig(values.config);
|
||||
const stop = new AbortController();
|
||||
process.on("SIGINT", () => {
|
||||
console.error("收到 SIGINT，发停下信号");
|
||||
stop.abort();
|
||||
});
|
||||
|
||||
const outcome = await runStudy({ studyDir: values.dir, config, stop: stop.signal });
|
||||
console.log(`RESULT ${JSON.stringify(outcome)}`);
|
||||
@@ -0,0 +1,71 @@
|
||||
// 检验自己用的小 Jev 客户端（任务书 12 第四部分第 6 条）：不经 src/jev，也不改它。
|
||||
// 请求格式参照 src/jev/jev.ts 和 RivalBits research-wiki/参考-Jev使用指南.md 第二部分：
|
||||
// POST {model, state, questions}，回答里 answers.<问题名>.noul 是 0 到 1 的概率。
|
||||
// 检验的问题都没有校准过阈值，一律以概率大于 0.5 判"是"，原始概率照存（任务书 12 第 5.2 节）。
|
||||
|
||||
const ENDPOINT = "https://api.typesafe.ai/v1/systemone";
|
||||
/** 检验用的 Jev 模型名，和研究配置的 models.jev 同一个。 */
|
||||
export const JEV_MODEL = "jev-latest";
|
||||
export const THRESHOLD = 0.5;
|
||||
export const THRESHOLD_NOTE = "阈值未校准：一律以概率大于 0.5 判\"是\"";
|
||||
|
||||
/** 一条是非题，写法照 src/jev/questions.ts：问题、关注点、判"是"判"否"的说明和反例。 */
|
||||
export interface Noul {
|
||||
type: "noul";
|
||||
instructions: { question: string; focus: string };
|
||||
criteria: {
|
||||
true: { what: string; examples: string[] };
|
||||
false: { what: string; not_for: string; examples: string[] };
|
||||
};
|
||||
}
|
||||
|
||||
export interface Answer {
|
||||
question: string;
|
||||
probability: number;
|
||||
yes: boolean;
|
||||
requestId: string;
|
||||
}
|
||||
|
||||
/** 调不通时重试前等多久：只重试连不上、超时、429 和 5xx（原因在我们控制之外），一共最多 4 次。 */
|
||||
const RETRY_WAITS_MS = [2_000, 10_000, 30_000];
|
||||
|
||||
/** 就一份材料问几问，一次请求。 */
|
||||
export async function ask(state: Record<string, unknown>, questions: Record<string, Noul>): Promise<Answer[]> {
|
||||
const key = process.env["TYPESAFE_API_KEY"];
|
||||
if (key === undefined || key.trim() === "") throw new Error("环境变量 TYPESAFE_API_KEY 没有设置");
|
||||
const body = JSON.stringify({ model: JEV_MODEL, state, questions });
|
||||
const failures: string[] = [];
|
||||
for (let attempt = 0; attempt <= RETRY_WAITS_MS.length; attempt++) {
|
||||
if (attempt > 0) await new Promise((ok) => setTimeout(ok, RETRY_WAITS_MS[attempt - 1]));
|
||||
let response: Response;
|
||||
let text: string;
|
||||
try {
|
||||
response = await fetch(ENDPOINT, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
|
||||
body,
|
||||
signal: AbortSignal.timeout(60_000),
|
||||
});
|
||||
text = await response.text();
|
||||
} catch (error) {
|
||||
failures.push((error as Error).message);
|
||||
continue;
|
||||
}
|
||||
const requestId = response.headers.get("x-typesafe-request-id") ?? "无";
|
||||
if (!response.ok) {
|
||||
const what = `HTTP ${response.status}（request-id: ${requestId}）：${text.slice(0, 300)}`;
|
||||
if (response.status !== 429 && response.status < 500) throw new Error(`Jev 拒绝了请求：${what}`);
|
||||
failures.push(what);
|
||||
continue;
|
||||
}
|
||||
const answers = (JSON.parse(text) as { answers?: Record<string, { noul?: unknown }> }).answers;
|
||||
return Object.keys(questions).map((question) => {
|
||||
const probability = answers?.[question]?.noul;
|
||||
if (typeof probability !== "number" || !(probability >= 0 && probability <= 1)) {
|
||||
throw new Error(`Jev 的回答里问题 ${question} 没有 0 到 1 之间的概率（request-id: ${requestId}）：${text.slice(0, 300)}`);
|
||||
}
|
||||
return { question, probability, yes: probability > THRESHOLD, requestId };
|
||||
});
|
||||
}
|
||||
throw new Error(`Jev 调不通，发了 ${failures.length} 次：${failures.join("；")}`);
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Ledger, type LedgerRecord } from "../../src/ledger/index.ts";
|
||||
import { entryId, type DecisionShapes } from "../../src/materials/index.ts";
|
||||
|
||||
// 读账本的小工具：只经 src/ledger/index.ts 打开、只读记录，不写。
|
||||
// 入口编号用 materials 的 entryId，和正式代码同一条编号规则。
|
||||
|
||||
/** 研究目录里账本的全部记录。账本还没有时报错：Ledger.open 会新建一个空账本，检验不能替 flow 建。 */
|
||||
export function readLedger(studyDir: string): LedgerRecord[] {
|
||||
const path = join(studyDir, "ledger.sqlite");
|
||||
if (!existsSync(path)) throw new Error(`${path} 还不存在`);
|
||||
const ledger = Ledger.open(path);
|
||||
try {
|
||||
return ledger.records();
|
||||
} finally {
|
||||
ledger.close();
|
||||
}
|
||||
}
|
||||
|
||||
export type Of<K extends LedgerRecord["kind"]> = Extract<LedgerRecord, { kind: K }>;
|
||||
export function ofKind<K extends LedgerRecord["kind"]>(records: LedgerRecord[], kind: K): Array<Of<K>> {
|
||||
return records.filter((r): r is Of<K> => r.kind === kind);
|
||||
}
|
||||
|
||||
export interface StageInfo {
|
||||
stageId: string;
|
||||
stage: string;
|
||||
round: number;
|
||||
roster: string[];
|
||||
closed: boolean;
|
||||
}
|
||||
|
||||
export function stages(records: LedgerRecord[]): StageInfo[] {
|
||||
const closed = new Set(ofKind(records, "stageClose").map((r) => r.content.stageId));
|
||||
return ofKind(records, "stageOpen").map(({ content: c }) => ({ ...c, closed: closed.has(c.stageId) }));
|
||||
}
|
||||
|
||||
/** 每份提交（阶段、作者、种类、对象、编号）的最新一版，按这一版的序号排。 */
|
||||
export function finalSubmissions(records: LedgerRecord[]): Array<Of<"submission">> {
|
||||
const latest = new Map<string, Of<"submission">>();
|
||||
for (const r of ofKind(records, "submission")) {
|
||||
const c = r.content;
|
||||
const slot = JSON.stringify([c.stageId, c.author, c.kind, c.target, c.item]);
|
||||
const seen = latest.get(slot);
|
||||
if (seen === undefined || seen.content.version < c.version) latest.set(slot, r);
|
||||
}
|
||||
return [...latest.values()].sort((a, b) => a.seq - b.seq);
|
||||
}
|
||||
|
||||
export interface Entry {
|
||||
id: string;
|
||||
author: string;
|
||||
name: string;
|
||||
body: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* 只看给定阶段里的提交，各人最后一版入口清单里的入口（方法.md 4.2：讨论时整份替换的新清单接着往后编号）。
|
||||
* 给第零轮全部阶段就是分配时的"入口全体"；只给写入口阶段就是闭门写完时的入口全体。
|
||||
*/
|
||||
export function entriesIn(records: LedgerRecord[], stageIds: ReadonlySet<string>): Entry[] {
|
||||
const lists = new Map<string, Entry[]>();
|
||||
const used = new Map<string, number>();
|
||||
for (const s of finalSubmissions(records)) {
|
||||
const c = s.content;
|
||||
if (!stageIds.has(c.stageId) || (c.kind !== "entry" && c.kind !== "discussion")) continue;
|
||||
const entries = (c.content as { entries?: Array<{ name: string; body: string }> }).entries;
|
||||
if (entries === undefined) continue;
|
||||
const n = used.get(c.author) ?? 0;
|
||||
lists.set(c.author, entries.map((e, i) => ({ id: entryId(c.author, n + i + 1), author: c.author, ...e })));
|
||||
used.set(c.author, n + entries.length);
|
||||
}
|
||||
return [...lists.values()].flat();
|
||||
}
|
||||
|
||||
export type Assignment = DecisionShapes["entries"]["output"][number];
|
||||
|
||||
/** 第 0 轮的入口分配决定：输入（全部志愿和入口）和输出（分配结果）；没有就是 undefined。 */
|
||||
export function entryDecision(records: LedgerRecord[]): { input: { entries: Array<{ id: string; author: string }> }; output: Assignment[] } | undefined {
|
||||
const d = ofKind(records, "decision").find((r) => r.content.decision === "entries" && r.content.round === 0);
|
||||
return d === undefined ? undefined : (d.content as unknown as { input: { entries: Array<{ id: string; author: string }> }; output: Assignment[] });
|
||||
}
|
||||
|
||||
/** 两份分配是否相同：每人分到的入口和第几志愿都一样，不看先后和对象里键的顺序。 */
|
||||
export function sameAssignment(a: Assignment[], b: Assignment[]): boolean {
|
||||
const key = (xs: Assignment[]) => xs.map((x) => `${x.researcher} ${x.entry} ${x.rank}`).sort().join("\n");
|
||||
return key(a) === key(b);
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
import { createServer, type ServerResponse } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
// 回放用的假模型服务（任务书 12 第四部分第 3 条）：本机起一个兼容 OpenAI 的流式接口，
|
||||
// 用 Pi 自己的 PI_CODING_AGENT_DIR 指一份 models.json 过来，正式代码不知道对面是假模型。
|
||||
// 每个模型名一份按顺序交出的提交清单：每次被派任务（会话的第一次请求），交出下一份；之后的请求回一句话结束。
|
||||
// 交完了就不再交，只回一句话：不为凑结果改内容。被退回时模型没有别的可交，会话就以"没交"结束。
|
||||
|
||||
/** 一份要交的提交：research_submit 的参数原样（payload，以及可能的 replaces），from 写它从哪来，只用于报告。 */
|
||||
export interface ReplayItem {
|
||||
from: string;
|
||||
arguments: { payload: unknown; replaces?: string };
|
||||
}
|
||||
|
||||
/** 回放脚本：模型名 → 按顺序交出的提交。没有要交的模型（实验员、核验员）写空数组。 */
|
||||
export interface ReplayScript {
|
||||
source: string;
|
||||
models: Record<string, ReplayItem[]>;
|
||||
}
|
||||
|
||||
export interface Replay {
|
||||
/** 起研究的子进程要带的环境变量。 */
|
||||
env: Record<string, string>;
|
||||
/** 每个模型名已经交出的提交（按交出先后）。 */
|
||||
delivered: Record<string, string[]>;
|
||||
/** 每个模型名还没交出的提交。 */
|
||||
remaining: () => Record<string, string[]>;
|
||||
close: () => Promise<void>;
|
||||
}
|
||||
|
||||
export const REPLAY_PROVIDER = "replay";
|
||||
|
||||
/**
|
||||
* 起服务并写好 Pi 的 models.json、auth.json。delayMs 是每次回复前等多久（杀进程检验要一段"正在写"的窗口，0 就是不等）。
|
||||
* 一份提交只有在回复完整写出后才算交出：回复前对面断开（控制进程被杀），这一份放回清单最前面，下次被派任务时还交它。
|
||||
*/
|
||||
export async function startReplay(script: ReplayScript, agentDir: string, delayMs: number): Promise<Replay> {
|
||||
const queues = new Map(Object.entries(script.models).map(([model, items]) => [model, [...items]]));
|
||||
const delivered: Record<string, string[]> = Object.fromEntries([...queues.keys()].map((m) => [m, []]));
|
||||
|
||||
const server = createServer((req, res) => {
|
||||
const chunks: Buffer[] = [];
|
||||
req.on("data", (c: Buffer) => chunks.push(c));
|
||||
req.on("end", () => void answer(JSON.parse(Buffer.concat(chunks).toString()), res));
|
||||
});
|
||||
|
||||
async function answer(body: { model: string; messages: Array<{ role: string }> }, res: ServerResponse): Promise<void> {
|
||||
const queue = queues.get(body.model);
|
||||
if (queue === undefined) throw new Error(`回放脚本里没有模型 ${body.model}`);
|
||||
// 会话的第一次请求里还没有助手的回复：这是新派的任务。
|
||||
const item = body.messages.some((m) => m.role === "assistant") ? undefined : queue.shift();
|
||||
res.on("close", () => {
|
||||
if (!res.writableFinished && item !== undefined) queue.unshift(item);
|
||||
});
|
||||
await new Promise((ok) => setTimeout(ok, delayMs));
|
||||
if (res.destroyed) return;
|
||||
const send = (delta: unknown, finish: string | null) =>
|
||||
res.write(`data: ${JSON.stringify({ id: "r", object: "chat.completion.chunk", created: 1, model: body.model, choices: [{ index: 0, delta, finish_reason: finish }] })}\n\n`);
|
||||
res.writeHead(200, { "content-type": "text/event-stream" });
|
||||
if (item === undefined) {
|
||||
send({ role: "assistant", content: "好的，我这次没有别的要交了。" }, null);
|
||||
send({}, "stop");
|
||||
} else {
|
||||
const args = JSON.stringify(item.arguments);
|
||||
send({ role: "assistant", tool_calls: [{ index: 0, id: "call-1", type: "function", function: { name: "research_submit", arguments: args } }] }, null);
|
||||
send({}, "tool_calls");
|
||||
}
|
||||
res.write(`data: ${JSON.stringify({ id: "r", object: "chat.completion.chunk", created: 1, model: body.model, choices: [], usage: { prompt_tokens: 100, completion_tokens: 10 } })}\n\n`);
|
||||
res.end("data: [DONE]\n\n");
|
||||
if (item !== undefined) delivered[body.model]!.push(item.from);
|
||||
}
|
||||
|
||||
await new Promise<void>((ok) => server.listen(0, "127.0.0.1", ok));
|
||||
const baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1`;
|
||||
const models = [...queues.keys()].map((id) => ({
|
||||
id,
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
contextWindow: 1_000_000,
|
||||
maxTokens: 32_000,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
}));
|
||||
mkdirSync(agentDir, { recursive: true });
|
||||
writeFileSync(join(agentDir, "models.json"), JSON.stringify({ providers: { [REPLAY_PROVIDER]: { baseUrl, api: "openai-completions", apiKey: "k", models } } }, null, 2));
|
||||
writeFileSync(join(agentDir, "auth.json"), "{}");
|
||||
return {
|
||||
env: { PI_CODING_AGENT_DIR: agentDir },
|
||||
delivered,
|
||||
remaining: () => Object.fromEntries([...queues].map(([m, q]) => [m, q.map((i) => i.from)])),
|
||||
close: () => new Promise<void>((ok) => server.close(() => ok())),
|
||||
};
|
||||
}
|
||||
|
||||
/** 回放用的各角色模型：研究者按脚本里的模型名，实验员、核验员各一个空清单的名字。 */
|
||||
export function replayModels(researchers: string[], jev: string): unknown {
|
||||
const m = (model: string) => ({ provider: REPLAY_PROVIDER, model, thinking: "off" });
|
||||
return { researchers: researchers.map(m), experimenter: m("experimenter"), verifier: m("verifier"), jev };
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
import { execFileSync, spawn, type ChildProcess } from "node:child_process";
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
// 检验脚本共用的起研究工具：检验目录放在仓库外，配置照用户的写法写一份，研究在子进程里跑（要能被 SIGKILL）。
|
||||
|
||||
/** 检验输出一律放在仓库外（任务书 12 第四部分第 4 条），不提交。 */
|
||||
export const EVAL_ROOT = "/home/iomgaa/inforsi-eval";
|
||||
|
||||
/** 新建一个检验目录 <日期-名字>；已经有了就报错，不把两次检验的产物混在一起。 */
|
||||
export function newEvalDir(name: string): string {
|
||||
const dir = join(EVAL_ROOT, `${new Date().toISOString().slice(0, 10)}-${name}`);
|
||||
if (existsSync(dir)) throw new Error(`检验目录 ${dir} 已经有了，换个名字或先挪走`);
|
||||
mkdirSync(dir, { recursive: true });
|
||||
return dir;
|
||||
}
|
||||
|
||||
/**
|
||||
* 照一份底稿写这次检验用的 research.json：换研究编号（容器名、镜像标签都带它），需要时换各角色模型；
|
||||
* 任务目录、提示词目录写成绝对路径（底稿里相对仓库根目录），这样配置文件放在仓库外也指得对。
|
||||
*/
|
||||
export function writeConfig(base: string, path: string, studyId: string, models: unknown): string {
|
||||
const config = JSON.parse(readFileSync(base, "utf8")) as Record<string, unknown>;
|
||||
config["studyId"] = studyId;
|
||||
config["task"] = resolve(config["task"] as string);
|
||||
config["promptDir"] = resolve(config["promptDir"] as string);
|
||||
if (models !== null) config["models"] = models;
|
||||
writeFileSync(path, `${JSON.stringify(config, null, 2)}\n`);
|
||||
return path;
|
||||
}
|
||||
|
||||
export interface StudyProcess {
|
||||
child: ChildProcess;
|
||||
/** 子进程退出：退出码、信号，以及它打印的结局（被杀掉时没有，是 null）。 */
|
||||
done: Promise<{ code: number | null; signal: NodeJS.Signals | null; outcome: unknown }>;
|
||||
}
|
||||
|
||||
/** 在子进程里起 eval/lib/cli.ts。config 为 null 是接着跑。子进程的输出原样转到本进程，并记进 <研究目录>.log。 */
|
||||
export function launch(config: string | null, studyDir: string, env: Record<string, string>): StudyProcess {
|
||||
const args = ["eval/lib/cli.ts", ...(config === null ? ["--resume"] : ["--config", config]), "--dir", studyDir];
|
||||
const child = spawn(process.execPath, args, { env: { ...process.env, ...env }, stdio: ["ignore", "pipe", "pipe"] });
|
||||
const log: string[] = [];
|
||||
let stdout = "";
|
||||
child.stdout!.on("data", (chunk: Buffer) => {
|
||||
stdout += chunk.toString();
|
||||
log.push(chunk.toString());
|
||||
process.stdout.write(chunk);
|
||||
});
|
||||
child.stderr!.on("data", (chunk: Buffer) => {
|
||||
log.push(chunk.toString());
|
||||
process.stderr.write(chunk);
|
||||
});
|
||||
const done = new Promise<{ code: number | null; signal: NodeJS.Signals | null; outcome: unknown }>((ok) =>
|
||||
child.on("close", (code, signal) => {
|
||||
writeFileSync(`${studyDir}.log`, log.join(""), { flag: "a" });
|
||||
const line = stdout.split("\n").find((l) => l.startsWith("RESULT "));
|
||||
ok({ code, signal, outcome: line === undefined ? null : JSON.parse(line.slice("RESULT ".length)) });
|
||||
}),
|
||||
);
|
||||
return { child, done };
|
||||
}
|
||||
|
||||
export function sleep(ms: number): Promise<void> {
|
||||
return new Promise((ok) => setTimeout(ok, ms));
|
||||
}
|
||||
|
||||
/** 本机名字以 prefix 开头的容器（含已停止的）。检验直接问 Docker，不经正式代码。 */
|
||||
export function containers(prefix: string): string[] {
|
||||
const names = execFileSync("docker", ["ps", "-a", "--format", "{{.Names}}"], { encoding: "utf8" });
|
||||
return names.split("\n").filter((n) => n.startsWith(prefix));
|
||||
}
|
||||
|
||||
/**
|
||||
* 等 flow 把账本的表建好，检验才第一次打开账本：Ledger.open 遇到空文件会自己建表，和 flow 同时建会冲突。
|
||||
* 信号是研究目录里出现 repo.git：flow 先 Ledger.open（建表），冻结配置时才建裸仓库（src/flow/loop.ts、study.ts 的 freezeConfig）。
|
||||
*/
|
||||
export async function waitForLedger(studyDir: string, child: ChildProcess): Promise<void> {
|
||||
while (!existsSync(join(studyDir, "repo.git"))) {
|
||||
if (child.exitCode !== null) throw new Error(`研究进程没建出 repo.git 就退出了，退出码 ${child.exitCode}`);
|
||||
await sleep(50);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { ofKind, readLedger } from "../lib/ledger.ts";
|
||||
import { replayModels, startReplay } from "../lib/replay.ts";
|
||||
import { containers, launch, newEvalDir, sleep, writeConfig } from "../lib/study.ts";
|
||||
|
||||
// 两份研究同时跑（任务书 12 第 5.4 节）：两个研究编号同时启动，都跑到基线结束，查两边各扣各的、容器不撞名。
|
||||
// node eval/round-zero/concurrent.ts
|
||||
// 研究者用一份空清单的回放：基线跑完、派写入口任务时模型一份不交，flow 按"交不出合格提交"停下，不花钱。
|
||||
|
||||
const out = newEvalDir("r0-concurrent");
|
||||
const ids = ["r0-conc-a", "r0-conc-b"];
|
||||
const researchers = ["researcher-1", "researcher-2", "researcher-3"];
|
||||
const replay = await startReplay({ source: "空清单", models: Object.fromEntries([...researchers, "experimenter", "verifier"].map((m) => [m, []])) }, join(out, "pi-agent"), 0);
|
||||
|
||||
const studies = ids.map((id) => {
|
||||
const dir = join(out, id);
|
||||
const config = writeConfig("eval/round-zero/research.json", join(out, `${id}.research.json`), id, replayModels(researchers, "jev-latest"));
|
||||
return { id, dir, run: launch(config, dir, replay.env) };
|
||||
});
|
||||
|
||||
// 跑的时候每 100 毫秒看一次本机 inforsi- 开头的容器，记下见过的名字。
|
||||
const seen = new Set<string>();
|
||||
let running = true;
|
||||
const watcher = (async () => {
|
||||
while (running) {
|
||||
for (const n of containers("inforsi-")) seen.add(n);
|
||||
await sleep(100);
|
||||
}
|
||||
})();
|
||||
const results = await Promise.all(studies.map((s) => s.run.done));
|
||||
running = false;
|
||||
await watcher;
|
||||
await replay.close();
|
||||
|
||||
const report = studies.map((s, i) => {
|
||||
const r = readLedger(s.dir);
|
||||
const base = new Set(ofKind(r, "runPlanned").filter((p) => p.content.purpose === "baseline").map((p) => p.content.run));
|
||||
return {
|
||||
id: s.id,
|
||||
outcome: results[i]!.outcome,
|
||||
baseStarted: ofKind(r, "runStarted").filter((x) => base.has(x.content.run)).length,
|
||||
baseResults: ofKind(r, "runResult").filter((x) => base.has(x.content.run)).map((x) => x.content),
|
||||
containersSeen: [...seen].filter((n) => n.startsWith(`inforsi-${s.id}-`)),
|
||||
};
|
||||
});
|
||||
const shared = report[0]!.containersSeen.filter((n) => report[1]!.containersSeen.includes(n));
|
||||
const pass = report.every((s) => s.baseStarted === 1 && s.baseResults.length === 1 && s.baseResults[0]!.judged !== null) && shared.length === 0;
|
||||
writeFileSync(join(out, "concurrent-report.json"), `${JSON.stringify({ pass, report, allContainersSeen: [...seen], shared }, null, 2)}\n`);
|
||||
for (const s of report) {
|
||||
console.log(`${s.id}：基线确认启动 ${s.baseStarted} 次；结果 ${JSON.stringify(s.baseResults.map((x) => x.judged))}；见过的容器 ${s.containersSeen.join("、") || "（没看到）"}；结局 ${JSON.stringify(s.outcome)}`);
|
||||
}
|
||||
console.log(`两边共用的容器名：${shared.join("、") || "没有"}；${pass ? "通过" : "不通过"}`);
|
||||
console.log(`报告：${join(out, "concurrent-report.json")}`);
|
||||
@@ -0,0 +1,71 @@
|
||||
import { copyFileSync, mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { entryId } from "../../src/materials/index.ts";
|
||||
import type { ReplayItem, ReplayScript } from "../lib/replay.ts";
|
||||
|
||||
// 从参考项目 16-real-1 的账本取第零轮的提交，按任务书 12 第 5.3 节转成新项目的格式，写成回放脚本。
|
||||
// 原账本只读：先复制到临时目录，再以只读方式打开复制件，原目录里不写任何东西。
|
||||
// node eval/round-zero/extract-16.ts
|
||||
|
||||
const SOURCE = "/home/iomgaa/rb-acceptance/16-real-1/board.sqlite";
|
||||
export const REPLAY_FILE = "eval/round-zero/replay-16-real-1.json";
|
||||
const PHASE_STAGE: Record<string, string> = { "entry-draft": "entryDraft", "entry-discuss": "discussion", "entry-claim": "claim" };
|
||||
|
||||
const copy = join(mkdtempSync(join(tmpdir(), "inforsi-rb16-")), "board.sqlite");
|
||||
copyFileSync(SOURCE, copy);
|
||||
const db = new DatabaseSync(copy, { readOnly: true });
|
||||
|
||||
type OldEntry = { name: string; startFrom: string; whyWorth: string };
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT st.phase, st.id AS stage, s.actor, s.version, s.created_seq AS seq, s.payload
|
||||
FROM submissions s JOIN stages st ON st.id = s.stage_id
|
||||
WHERE st.phase IN ('entry-draft', 'entry-discuss', 'entry-claim') ORDER BY s.created_seq`,
|
||||
)
|
||||
.all() as Array<{ phase: string; stage: string; actor: string; version: number; seq: number; payload: string }>;
|
||||
const fact = db.prepare("SELECT payload FROM facts WHERE kind = 'entry-assignment'").all() as Array<{ payload: string }>;
|
||||
db.close();
|
||||
if (fact.length !== 1) throw new Error(`entry-assignment 应有 1 条，实有 ${fact.length} 条`);
|
||||
|
||||
/** 当年入口 {name, startFrom, whyWorth} → 新项目 {name, body}。 */
|
||||
const entry = (e: OldEntry) => ({ name: e.name, body: `从哪里开始看：${e.startFrom}\n\n为什么值得走：${e.whyWorth}` });
|
||||
/** 当年入口编号 researcher-1-e4 → 新项目 e-researcher-1-4。 */
|
||||
function newId(old: string): string {
|
||||
const m = /^(researcher-\d+)-e(\d+)$/.exec(old);
|
||||
if (m === null) throw new Error(`入口编号 ${old} 不是 researcher-<i>-e<n> 的写法`);
|
||||
return entryId(m[1]!, Number(m[2]));
|
||||
}
|
||||
|
||||
const models: Record<string, ReplayItem[]> = {};
|
||||
for (const r of rows) {
|
||||
const p = JSON.parse(r.payload) as { entries?: OldEntry[]; body?: string; choices?: string[] };
|
||||
const payload =
|
||||
r.phase === "entry-draft"
|
||||
? { entries: p.entries!.map(entry) }
|
||||
: r.phase === "entry-discuss"
|
||||
? { body: p.body!, ...(p.entries === undefined ? {} : { entries: p.entries.map(entry) }) }
|
||||
: { choices: p.choices!.map(newId) };
|
||||
(models[r.actor] ??= []).push({ from: `${r.stage} ${PHASE_STAGE[r.phase]} seq ${r.seq} v${r.version}`, arguments: { payload } });
|
||||
}
|
||||
|
||||
const old = JSON.parse(fact[0]!.payload) as { assignments: Array<{ researcher: string; entryId: string; rank: number | null }>; entries: Array<{ entryId: string; author: string }> };
|
||||
const expected = old.assignments.map((a) => ({ researcher: a.researcher, entry: newId(a.entryId), rank: a.rank }));
|
||||
const script: ReplayScript & { expected: typeof expected; expectedEntries: string[] } = {
|
||||
source: `${SOURCE}（第零轮 submissions，created_seq ${rows[0]!.seq}–${rows.at(-1)!.seq}）`,
|
||||
models,
|
||||
expected,
|
||||
expectedEntries: old.entries.map((e) => newId(e.entryId)),
|
||||
};
|
||||
writeFileSync(REPLAY_FILE, `${JSON.stringify(script, null, 2)}\n`);
|
||||
|
||||
// 核对：三阶段各几份、作者、版本，以及当年的分配。
|
||||
console.log(`共 ${rows.length} 份提交，序号 ${rows[0]!.seq}–${rows.at(-1)!.seq}`);
|
||||
for (const phase of Object.keys(PHASE_STAGE)) {
|
||||
const these = rows.filter((r) => r.phase === phase);
|
||||
console.log(`${phase}：${these.length} 份；${these.map((r) => `${r.actor}@${r.seq}(v${r.version}${JSON.parse(r.payload).entries ? `，${JSON.parse(r.payload).entries.length} 个入口` : ""})`).join("、")}`);
|
||||
}
|
||||
console.log(`当年分配：${expected.map((a) => `${a.researcher} → ${a.entry}（第 ${a.rank} 志愿）`).join("；")}`);
|
||||
console.log(`当年最终入口：${script.expectedEntries.join("、")}`);
|
||||
console.log(`已写 ${REPLAY_FILE}`);
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"note": "任务书 12 第 5.1 节的关键想法清单和死路清单，出处照抄。英文说明给 Jev 看。",
|
||||
"key": [
|
||||
{
|
||||
"id": "K1",
|
||||
"name": "联合优化",
|
||||
"description": "Optimize the centers and radii of all 26 circles together (78 numbers) with a constrained continuous optimizer such as scipy SLSQP, with constraints that every circle stays inside the square and no two circles overlap.",
|
||||
"source": "最大的一跳：OpenEvolve 2.377→2.634；AdaEvolve 2.541→2.610；参考项目 12-real-1、12-real-2 第 1 轮采纳；16-real-1 关掉它少 0.132"
|
||||
},
|
||||
{
|
||||
"id": "K2",
|
||||
"name": "扰动重优化",
|
||||
"description": "Starting from the current best solution, perturb the circle centers by a moderate amount (about 0.01 to 0.06), run the joint optimization again, and keep the result only if it is better; this includes annealing-style acceptance and larger perturbations after the search stagnates.",
|
||||
"source": "AlphaEvolve、ShinkaEvolve、Discovery Loop；参考项目 12-real-2 第 2 轮靠它从 2.6155 到 2.635983，15-tight-2 也到了 2.635983"
|
||||
},
|
||||
{
|
||||
"id": "K3",
|
||||
"name": "结构化起点与多起点",
|
||||
"description": "Start from structured layouts such as hexagonal or staggered rows, or from many random starting layouts, run the optimization from each start, and keep the best result.",
|
||||
"source": "ThetaEvolve、TTT-Discover 只靠\"错排起点 + 一次 SLSQP\"就到 2.635983；CodeEvolve 10 次重启"
|
||||
},
|
||||
{
|
||||
"id": "K4",
|
||||
"name": "固定圆心求最优半径",
|
||||
"description": "Keep the circle centers fixed and solve a linear program for the radii that maximizes their sum.",
|
||||
"source": "ShinkaEvolve 容差 0 那次的第一步、Discovery Loop 的种子；16-real-1 消融撑 28%"
|
||||
},
|
||||
{
|
||||
"id": "K5",
|
||||
"name": "严格合法收尾",
|
||||
"description": "As a final step, shrink all radii by a tiny amount or refine the solution so that it is strictly valid under zero tolerance.",
|
||||
"source": "ShinkaEvolve、ThetaEvolve 严格版；本任务容差为 0（TASK.md）"
|
||||
}
|
||||
],
|
||||
"deadEnds": [
|
||||
{
|
||||
"id": "D1",
|
||||
"name": "只做几何构造",
|
||||
"description": "Place the circle centers by a hand-designed rule (hexagonal grid, square grid, concentric rings) and only tune the arrangement and its parameters, without using continuous optimization to move the centers.",
|
||||
"source": "各家卡在 2.2–2.54；等半径六角的上限是 2.438966；OpenEvolve 第一阶段提示词要求只写构造，卡在 2.377；参考项目 9a、9b-real-2 卡在 2.517、2.541"
|
||||
},
|
||||
{
|
||||
"id": "D2",
|
||||
"name": "只优化半径",
|
||||
"description": "Keep the circle centers fixed and only change how the radii are allocated, as the whole route rather than as a helper step.",
|
||||
"source": "上限受布局限制，2.5414 是反复出现的平台"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import type { LedgerRecord } from "../../src/ledger/index.ts";
|
||||
import { entryDecision, finalSubmissions, ofKind, readLedger, sameAssignment, stages, type Assignment } from "../lib/ledger.ts";
|
||||
import { replayModels, startReplay, type ReplayScript } from "../lib/replay.ts";
|
||||
import { containers, launch, newEvalDir, sleep, waitForLedger, writeConfig } from "../lib/study.ts";
|
||||
|
||||
// 杀进程检验（任务书 12 第 5.4 节）：在一处 SIGKILL 控制进程，再启动让它跑完第零轮，逐条查五件事。
|
||||
// node eval/round-zero/kill.ts baseline|drafting|stageFull
|
||||
// 模型一侧用 16-real-1 的回放（不花钱、结果确定，分配应和当年相同）；基线照样用真 Docker。
|
||||
// 回放每次回复前等 DELAY_MS，留出"研究者正在写""最后一份刚收下、阶段还没关"的窗口。
|
||||
|
||||
const DELAY_MS = 3000;
|
||||
type Point = "baseline" | "drafting" | "stageFull";
|
||||
const point = process.argv[2] as Point;
|
||||
|
||||
/** 三处杀点：看到账本里出现这种状态就杀。 */
|
||||
const HITS: Record<Point, (r: LedgerRecord[]) => boolean> = {
|
||||
// 基线容器正在跑：基线运行确认启动了、还没有结果。
|
||||
baseline: (r) => {
|
||||
const base = new Set(ofKind(r, "runPlanned").filter((p) => p.content.purpose === "baseline").map((p) => p.content.run));
|
||||
const done = new Set(ofKind(r, "runResult").map((x) => x.content.run));
|
||||
return ofKind(r, "runStarted").some((s) => base.has(s.content.run) && !done.has(s.content.run));
|
||||
},
|
||||
// 研究者 2 正在写入口：他在写入口阶段的调用开始了、还没结束。
|
||||
drafting: (r) => {
|
||||
const draft = new Set(stages(r).filter((s) => s.stage === "entryDraft").map((s) => s.stageId));
|
||||
return openCalls(r).some((c) => c.content.actor === "researcher-2" && draft.has(c.content.stageId ?? ""));
|
||||
},
|
||||
// 某阶段最后一份提交刚收下、阶段还没关：名单上的人都交了，还没有关闭记录。
|
||||
stageFull: (r) => stages(r).some((s) => !s.closed && s.roster.every((a) => submitted(r, s.stageId, a))),
|
||||
};
|
||||
if (!(point in HITS)) throw new Error(`杀点只能是 ${Object.keys(HITS).join("、")} 之一`);
|
||||
|
||||
function openCalls(r: LedgerRecord[]): Array<LedgerRecord<"callStarted">> {
|
||||
const ended = new Set(ofKind(r, "callEnded").map((e) => e.content.call));
|
||||
return ofKind(r, "callStarted").filter((c) => !ended.has(c.content.call));
|
||||
}
|
||||
const submitted = (r: LedgerRecord[], stageId: string, author: string) =>
|
||||
ofKind(r, "submission").some((s) => s.content.stageId === stageId && s.content.author === author);
|
||||
|
||||
const script = JSON.parse(readFileSync("eval/round-zero/replay-16-real-1.json", "utf8")) as ReplayScript & { expected: Assignment[] };
|
||||
// 研究编号也是镜像标签的一部分，Docker 镜像名只许小写，所以杀点名转成小写。
|
||||
const name = `r0-kill-${point.toLowerCase()}`;
|
||||
const out = newEvalDir(name);
|
||||
const replay = await startReplay(script, join(out, "pi-agent"), DELAY_MS);
|
||||
const config = writeConfig("eval/round-zero/research.json", join(out, "research.json"), name, replayModels(Object.keys(script.models).sort(), "jev-latest"));
|
||||
const studyDir = join(out, "study");
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 第一次启动，打中就杀
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const first = launch(config, studyDir, replay.env);
|
||||
await waitForLedger(studyDir, first.child);
|
||||
let triggered: LedgerRecord | undefined;
|
||||
while (first.child.exitCode === null) {
|
||||
const r = readLedger(studyDir);
|
||||
if (HITS[point](r)) {
|
||||
first.child.kill("SIGKILL");
|
||||
triggered = r.at(-1);
|
||||
break;
|
||||
}
|
||||
await sleep(20);
|
||||
}
|
||||
await first.done;
|
||||
if (triggered === undefined) throw new Error(`没打中 ${point}：研究进程自己结束了，账本最后一条是 ${JSON.stringify(readLedger(studyDir).at(-1))}`);
|
||||
// 进程已死，此刻的账本就是被杀时的样子。
|
||||
const atKill = readLedger(studyDir);
|
||||
const killSeq = atKill.at(-1)!.seq;
|
||||
const killedCalls = openCalls(atKill);
|
||||
const openStages = stages(atKill).filter((s) => !s.closed);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 再启动，跑完第零轮
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const second = await launch(null, studyDir, replay.env).done;
|
||||
await replay.close();
|
||||
const r = readLedger(studyDir);
|
||||
const studyId = name;
|
||||
|
||||
const baseRuns = new Set(ofKind(r, "runPlanned").filter((p) => p.content.purpose === "baseline").map((p) => p.content.run));
|
||||
const baseStarted = ofKind(r, "runStarted").filter((s) => baseRuns.has(s.content.run));
|
||||
const leftover = containers(`inforsi-${studyId}-`);
|
||||
const callEnds = killedCalls.map((c) => ({ call: c.content.call, end: ofKind(r, "callEnded").find((e) => e.content.call === c.content.call)?.content ?? null }));
|
||||
const redispatch = openStages.flatMap((s) =>
|
||||
s.roster.map((actor) => {
|
||||
const before = submitted(atKill, s.stageId, actor);
|
||||
const again = ofKind(r, "callStarted").some((c) => c.seq > killSeq && c.content.stageId === s.stageId && c.content.actor === actor);
|
||||
return { stageId: s.stageId, actor, submittedBeforeKill: before, dispatchedAfterRestart: again, ok: before !== again };
|
||||
}),
|
||||
);
|
||||
const finals = finalSubmissions(r);
|
||||
const perStage = stages(r).map((s) => ({
|
||||
stageId: s.stageId,
|
||||
closed: s.closed,
|
||||
finalPerPerson: Object.fromEntries(s.roster.map((a) => [a, finals.filter((f) => f.content.stageId === s.stageId && f.content.author === a).length])),
|
||||
}));
|
||||
const decision = entryDecision(r);
|
||||
const assignment = decision === undefined ? null : decision.output;
|
||||
|
||||
const checks = {
|
||||
"1 基线只扣了 1 次，Docker 里没有留下基线容器": { pass: baseStarted.length === 1 && leftover.length === 0, baseRunsPlanned: [...baseRuns], baseStarted: baseStarted.length, leftoverContainers: leftover },
|
||||
"2 被杀那次调用有结束记录，结局出错、原因进程中断": {
|
||||
pass: callEnds.every((c) => c.end !== null && c.end.status === "failed" && c.end.reason.includes("进程中断")),
|
||||
calls: callEnds,
|
||||
},
|
||||
"3 没交上的人被重新派了，交过的没被重派": { pass: redispatch.every((x) => x.ok), detail: redispatch },
|
||||
"4 每个阶段都关上了，每人每阶段只有一份最终收下的提交": {
|
||||
pass: perStage.every((s) => s.closed && Object.values(s.finalPerPerson).every((n) => n === 1)),
|
||||
detail: perStage,
|
||||
},
|
||||
"5 分配照常写出": { pass: assignment !== null, assignment, sameAsReference: assignment !== null && sameAssignment(assignment, script.expected) },
|
||||
};
|
||||
const report = { point, triggeredAt: triggered, lastRecordAtKill: atKill.at(-1), outcomeAfterRestart: second.outcome, checks };
|
||||
writeFileSync(join(out, "kill-report.json"), `${JSON.stringify(report, null, 2)}\n`);
|
||||
console.log(`\n杀点 ${point}：触发时账本最后一条 #${triggered.seq} ${triggered.kind}；被杀时最后一条 #${killSeq} ${atKill.at(-1)!.kind} ${JSON.stringify(atKill.at(-1)!.content).slice(0, 200)}`);
|
||||
console.log(`再启动后的结局：${JSON.stringify(second.outcome)}`);
|
||||
for (const [k, v] of Object.entries(checks)) console.log(`${v.pass ? "通过" : "不通过"} ${k}`);
|
||||
console.log(`报告：${join(out, "kill-report.json")}`);
|
||||
@@ -0,0 +1,148 @@
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { ask, THRESHOLD_NOTE, type Answer, type Noul } from "../lib/jev.ts";
|
||||
import { entriesIn, entryDecision, readLedger, stages, type Entry } from "../lib/ledger.ts";
|
||||
import { ENTRY_ONLY, LEADS_TO_IDEA, SAME_KIND, STOPS_AT_DEAD_END, TASKS } from "./questions.ts";
|
||||
|
||||
// 第零轮质量评估（任务书 12 第 5.2 节六问）。只读账本，问 Jev，原始概率全部存进 <检验目录>/quality.json。
|
||||
// node eval/round-zero/quality.ts <检验目录>（研究在 <检验目录>/study）
|
||||
// "入口全体"是分配时最终有效的全部入口，"分到的入口"是分配结果里每人分到的那个。
|
||||
|
||||
type Idea = { id: string; name: string; description: string };
|
||||
const ideas = JSON.parse(readFileSync("eval/round-zero/ideas.json", "utf8")) as { key: Idea[]; deadEnds: Idea[] };
|
||||
|
||||
const out = process.argv[2];
|
||||
if (out === undefined) throw new Error("要写检验目录，研究在它下面的 study 里");
|
||||
const records = readLedger(join(out, "study"));
|
||||
const decision = entryDecision(records);
|
||||
if (decision === undefined) throw new Error("账本里还没有第 0 轮的入口分配决定，第零轮没跑完");
|
||||
const round0 = stages(records).filter((s) => s.round === 0);
|
||||
const final = entriesIn(records, new Set(round0.map((s) => s.stageId)));
|
||||
const draft = entriesIn(records, new Set(round0.filter((s) => s.stage === "entryDraft").map((s) => s.stageId)));
|
||||
// 自己按编号规则算出的入口全体，要和 flow 分配时用的那份一致；对不上说明 flow 或这里有一边取错了。
|
||||
const decided = decision.input.entries;
|
||||
if (JSON.stringify(decided.map((e) => e.id).sort()) !== JSON.stringify(final.map((e) => e.id).sort())) {
|
||||
throw new Error(`分配决定里的入口 ${decided.map((e) => e.id).join("、")} 和从提交算出的入口全体 ${final.map((e) => e.id).join("、")} 对不上`);
|
||||
}
|
||||
const assignment = decision.output;
|
||||
const assigned = assignment.map((a) => final.find((e) => e.id === a.entry)!);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 问 Jev：同样的材料只问一次；每次请求的材料、问题、原始概率都记下
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const raw: Array<{ site: string; ids: string[]; answers: Answer[] }> = [];
|
||||
const cache = new Map<string, Promise<Answer[]>>();
|
||||
/** 同时最多几个请求在路上：一下发上百个容易被限流。 */
|
||||
const PARALLEL = 6;
|
||||
let running = 0;
|
||||
const waiting: Array<() => void> = [];
|
||||
async function limited<T>(fn: () => Promise<T>): Promise<T> {
|
||||
if (running >= PARALLEL) await new Promise<void>((ok) => waiting.push(ok));
|
||||
running++;
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
running--;
|
||||
waiting.shift()?.();
|
||||
}
|
||||
}
|
||||
const show = (e: Entry) => ({ name: e.name, body: e.body });
|
||||
/** 同样的材料、同样的问题只问一次（写入口阶段没换的入口在讨论后还是同一段文字）。 */
|
||||
function query(site: string, ids: string[], state: Record<string, unknown>, questions: Record<string, Noul>): Promise<Answer[]> {
|
||||
const key = JSON.stringify([site, state]);
|
||||
if (!cache.has(key)) {
|
||||
cache.set(key, limited(() => ask(state, questions)).then((answers) => (raw.push({ site, ids, answers }), answers)));
|
||||
}
|
||||
return cache.get(key)!;
|
||||
}
|
||||
const yes = async (p: Promise<Answer[]>, question: string) => (await p).find((a) => a.question === question)!.yes;
|
||||
|
||||
const ideaState = (e: Entry, idea: Idea) => ({ task: TASKS.idea, entry: show(e), idea: { description: idea.description } });
|
||||
const leadsTo = (e: Entry, idea: Idea) => yes(query("leads", [e.id, idea.id], ideaState(e, idea), LEADS_TO_IDEA), "leads_to_idea");
|
||||
const staysIn = (e: Entry, idea: Idea) => yes(query("stays", [e.id, idea.id], ideaState(e, idea), STOPS_AT_DEAD_END), "stays_within_idea");
|
||||
const entryOnly = (e: Entry, question: string) => yes(query("entry", [e.id], { task: TASKS.entry, entry: show(e) }, ENTRY_ONLY), question);
|
||||
const sameKind = (a: Entry, b: Entry) =>
|
||||
yes(query("pair", [a.id, b.id], { task: TASKS.pair, entry_a: show(a), entry_b: show(b) }, SAME_KIND), "same_kind_of_change");
|
||||
|
||||
async function coverage(list: Entry[]): Promise<Record<string, string[]>> {
|
||||
const out: Record<string, string[]> = {};
|
||||
for (const idea of ideas.key) {
|
||||
const hits = await Promise.all(list.map((e) => leadsTo(e, idea)));
|
||||
out[idea.id] = list.filter((_, i) => hits[i]).map((e) => e.id);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
async function overlap(list: Entry[]): Promise<{ pairs: number; same: string[][]; ratio: number | null }> {
|
||||
const pairs = list.flatMap((a, i) => list.slice(i + 1).filter((b) => b.author !== a.author).map((b) => [a, b] as const));
|
||||
const hits = await Promise.all(pairs.map(([a, b]) => sameKind(a, b)));
|
||||
const same = pairs.filter((_, i) => hits[i]).map(([a, b]) => [a.id, b.id]);
|
||||
return { pairs: pairs.length, same, ratio: pairs.length === 0 ? null : same.length / pairs.length };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 六问
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const k1 = ideas.key.find((i) => i.id === "K1")!;
|
||||
const covFinal = await coverage(final);
|
||||
const q1 = { coverage: covFinal, pass: covFinal["K1"]!.length > 0 };
|
||||
|
||||
const assignedPairs = assigned.flatMap((a, i) => assigned.slice(i + 1).map((b) => [a, b] as const));
|
||||
const pairHits = await Promise.all(assignedPairs.map(([a, b]) => sameKind(a, b)));
|
||||
const q2 = { same: assignedPairs.filter((_, i) => pairHits[i]).map(([a, b]) => [a.id, b.id]), pass: !pairHits.some(Boolean) };
|
||||
|
||||
const assignedK1 = await Promise.all(assigned.map((e) => leadsTo(e, k1)));
|
||||
const deadEnd = await Promise.all(
|
||||
assigned.map(async (e) => {
|
||||
const hits = await Promise.all(ideas.deadEnds.map((d) => staysIn(e, d)));
|
||||
return ideas.deadEnds.filter((_, i) => hits[i]).map((d) => d.id);
|
||||
}),
|
||||
);
|
||||
const q3 = {
|
||||
k1: assigned.filter((_, i) => assignedK1[i]).map((e) => e.id),
|
||||
deadEnds: Object.fromEntries(assigned.map((e, i) => [e.id, deadEnd[i]])),
|
||||
pass: assignedK1.some(Boolean) && deadEnd.every((d) => d.length === 0),
|
||||
};
|
||||
|
||||
const perEntry = async (list: Entry[], question: string) => (await Promise.all(list.map((e) => entryOnly(e, question))));
|
||||
const changesCode = await perEntry(assigned, "changes_code");
|
||||
const q4 = { notCodeRoute: assigned.filter((_, i) => !changesCode[i]).map((e) => e.id), pass: changesCode.every(Boolean) };
|
||||
|
||||
const writesAnswer = async (list: Entry[]) => {
|
||||
const [code, score] = [await perEntry(list, "names_code_change"), await perEntry(list, "states_expected_score")];
|
||||
return list.filter((_, i) => code[i] || score[i]).map((e) => e.id);
|
||||
};
|
||||
const q5 = { assignedWritingAnswer: await writesAnswer(assigned), allWritingAnswer: await writesAnswer(final) };
|
||||
const q5pass = q5.assignedWritingAnswer.length === 0;
|
||||
|
||||
const q6 = {
|
||||
beforeDiscussion: { overlap: await overlap(draft), coverage: await coverage(draft) },
|
||||
afterDiscussion: { overlap: await overlap(final), coverage: covFinal },
|
||||
};
|
||||
|
||||
const report = {
|
||||
study: join(out, "study"),
|
||||
thresholdNote: THRESHOLD_NOTE,
|
||||
entries: final.map((e) => ({ id: e.id, author: e.author, name: e.name })),
|
||||
draftEntries: draft.map((e) => ({ id: e.id, author: e.author, name: e.name })),
|
||||
assignment: assignment.map((a, i) => ({ ...a, name: assigned[i]!.name })),
|
||||
results: { q1, q2, q3, q4, q5: { ...q5, pass: q5pass }, q6 },
|
||||
passed: { q1: q1.pass, q2: q2.pass, q3: q3.pass, q4: q4.pass, q5: q5pass },
|
||||
raw,
|
||||
};
|
||||
writeFileSync(join(out, "quality.json"), `${JSON.stringify(report, null, 2)}\n`);
|
||||
|
||||
const mark = (b: boolean) => (b ? "通过" : "不通过");
|
||||
const ratio = (o: { pairs: number; same: string[][]; ratio: number | null }) => `${o.same.length}/${o.pairs}${o.ratio === null ? "" : `（${(o.ratio * 100).toFixed(0)}%）`}`;
|
||||
const count = (c: Record<string, string[]>) => Object.entries(c).map(([k, v]) => `${k}:${v.length}`).join(" ");
|
||||
console.log(`（${THRESHOLD_NOTE}）`);
|
||||
console.log(`入口全体：${final.map((e) => `${e.id} ${e.name}`).join("\n ")}`);
|
||||
console.log(`分到的入口：${report.assignment.map((a) => `${a.researcher} → ${a.entry}（第 ${a.rank} 志愿）${a.name}`).join("\n ")}`);
|
||||
console.log(`1 方向对不对：${mark(q1.pass)}；覆盖 ${count(covFinal)}`);
|
||||
console.log(`2 分没分开：${mark(q2.pass)}；判"是"的对 ${JSON.stringify(q2.same)}`);
|
||||
console.log(`3 好方向分出去没有：${mark(q3.pass)}；指向 K1 的 ${JSON.stringify(q3.k1)}，停在死路的 ${JSON.stringify(q3.deadEnds)}`);
|
||||
console.log(`4 改代码的路线：${mark(q4.pass)}；不是的 ${JSON.stringify(q4.notCodeRoute)}`);
|
||||
console.log(`5 有没有写答案：${mark(q5pass)}；分到的里写了的 ${JSON.stringify(q5.assignedWritingAnswer)}，入口全体里写了的 ${JSON.stringify(q5.allWritingAnswer)}`);
|
||||
console.log(`6 讨论有没有用：讨论前重合 ${ratio(q6.beforeDiscussion.overlap)}、覆盖 ${count(q6.beforeDiscussion.coverage)}；讨论后重合 ${ratio(q6.afterDiscussion.overlap)}、覆盖 ${count(covFinal)}`);
|
||||
console.log(`原始概率：${join(out, "quality.json")}`);
|
||||
@@ -0,0 +1,176 @@
|
||||
import type { Noul } from "../lib/jev.ts";
|
||||
|
||||
// 第零轮质量评估问 Jev 的问题（任务书 12 第 5.2 节"问 Jev 的规矩"）：英文、一问只问一件事、
|
||||
// 带材料说明、判"是"判"否"的说明和反例；写法照 src/jev/questions.ts。这些是检验自己的问题，不进 src/jev。
|
||||
// 任务书里两处一句话里有两个条件（第 3 问"D1 或 D2"、第 5 问"写出改成什么，或者预计分数"），按"一问一事"拆成两问，
|
||||
// 在 quality.ts 里用"任一判是"合起来。
|
||||
|
||||
const TASK =
|
||||
"Round zero of a study that improves a program for the circle packing task: place 26 non-overlapping circles inside the unit square " +
|
||||
"so that the sum of their radii is as large as possible (no tolerance: any overlap or circle outside the square is invalid). " +
|
||||
"The baseline program puts the circles on fixed rings around the center and then shrinks radii pair by pair to remove overlaps; " +
|
||||
"it scores about 0.96 and the best known score is about 2.636. " +
|
||||
"In round zero each researcher writes entries: starting points for a line of work that one researcher will follow in the next rounds. " +
|
||||
"An entry has a `name` and a `body` (in Chinese) saying where to start looking and why the route is worth walking. " +
|
||||
"Entries are meant to name a starting point only, not the finished code change or an expected score.";
|
||||
|
||||
export const TASKS = {
|
||||
idea: `${TASK} \`idea\` is a technique from the literature on this task, described in English.`,
|
||||
entry: TASK,
|
||||
pair: `${TASK} \`entry_a\` and \`entry_b\` were written by two different researchers.`,
|
||||
};
|
||||
|
||||
/** 第 1 问：按这个入口走，会直接走到这个想法吗（idea 是 K1–K5 之一）。 */
|
||||
export const LEADS_TO_IDEA: Record<string, Noul> = {
|
||||
leads_to_idea: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Will a researcher who follows `entry` arrive directly at the technique described in `idea.description`?",
|
||||
focus:
|
||||
"Read `entry.body` for what the researcher will look at first and what they intend to try or measure. " +
|
||||
"Answer yes if the entry names this technique, or if this technique is the natural first code change on the route the entry points to. " +
|
||||
"The entry may describe the technique in other words, in Chinese. Sharing the goal of a higher score, or reading the same function, is not enough.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The entry names this technique, or the route it points to leads straight to this technique as its first code change",
|
||||
examples: [
|
||||
"idea: optimize centers and radii together with a constrained optimizer; entry: 把 26 个圆心和半径一起交给 scipy 的约束优化器",
|
||||
"idea: solve a linear program for the radii with fixed centers; entry: 先问给定圆心下半径和的最大值怎么算，它是不是一个线性规划",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "The entry heads somewhere else: a different technique, only reading rules, or only measuring, so this technique would not be its next step",
|
||||
not_for: "Entries that only share the goal of raising the score, or only read the code this technique would change, without heading for the technique",
|
||||
examples: [
|
||||
"idea: optimize centers and radii together with a constrained optimizer; entry: 只读评测脚本，弄清容差和合法化规则",
|
||||
"idea: solve a linear program for the radii with fixed centers; entry: 把三环布局换成六角网格",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/** 第 3 问后半：这个入口的路线是不是只停在这条死路上（idea 是 D1 或 D2 之一）。 */
|
||||
export const STOPS_AT_DEAD_END: Record<string, Noul> = {
|
||||
stays_within_idea: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Does the whole route in `entry` stay within the approach described in `idea.description`, with no plan to go beyond it?",
|
||||
focus:
|
||||
"Answer yes only if everything the entry plans to change falls inside this approach. " +
|
||||
"If the entry also plans a continuous optimization that moves the circle centers, or uses the approach only as a helper step " +
|
||||
"(a starting layout, or a way to score layouts inside a larger search), answer no.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Everything the entry plans to change is inside this approach; it is the whole route",
|
||||
examples: [
|
||||
"idea: hand-designed placement of centers only; entry: 把三环布局换成六角网格，调行数和间距",
|
||||
"idea: change only the radii with fixed centers; entry: 圆心不动，只改半径的分配方法，这就是整条路线",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "The entry goes beyond this approach, or uses it only as one step of a larger route",
|
||||
not_for: "Entries that use this approach as a starting layout or as a scoring helper inside a search that moves the centers",
|
||||
examples: [
|
||||
"idea: hand-designed placement of centers only; entry: 从六角起点出发，再用约束优化同时移动圆心和半径",
|
||||
"idea: change only the radii with fixed centers; entry: 先用精确的半径求解当打分器，再在外层搜索圆心位置",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/** 第 4、5 问：只看一个入口。第 5 问拆成两问。 */
|
||||
export const ENTRY_ONLY: Record<string, Noul> = {
|
||||
changes_code: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Is `entry` a route that changes the program's code to raise the score, rather than one that only reads rules, measures, or analyzes?",
|
||||
focus:
|
||||
"A route may begin by reading code or measuring something, as long as the reading or measuring is meant to lead to a code change that raises the score. " +
|
||||
"Answer no only if the entry's purpose ends at understanding, measuring, or checking rules, with no plan to change the code.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The entry intends to change the program, possibly after a first step of reading or measuring",
|
||||
examples: ["先读半径的计算方法，量它离最优差多少，然后把它换掉", "从错排起点出发，跑一次约束优化"],
|
||||
},
|
||||
false: {
|
||||
what: "The entry only reads rules, measures, or analyzes, and does not plan any change to the code",
|
||||
not_for: "Entries that begin with a diagnosis but clearly intend to change the code afterwards",
|
||||
examples: ["只读评测脚本，列出评测接受和拒绝哪些解", "统计基线产物里有多少个圆贴着边，写成一份报告"],
|
||||
},
|
||||
},
|
||||
},
|
||||
names_code_change: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Does `entry` spell out the finished code change to make, such as the exact algorithm with its settings or the replacement code, instead of only a starting point?",
|
||||
focus:
|
||||
"Entries are meant to say where to start looking and why. Mentioning a technique as a question to explore, or as a direction to look in, is a starting point. " +
|
||||
"Stating the change to make with the method and its settings already decided spells out the answer.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The entry states what the code should be changed into, with the method and its settings decided",
|
||||
examples: [
|
||||
"把布局函数换成 SLSQP，同时优化 78 个变量，起点用 5 行错排，迭代 500 次",
|
||||
"半径计算改成用 scipy linprog 求解，约束是 r_i + r_j ≤ d_ij 和到四边的距离",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "The entry names where to look and what to question, leaving the change itself to be worked out",
|
||||
not_for: "Entries that name a technique only as a question to check or a direction to explore",
|
||||
examples: ["先读半径的计算方法，问它离给定圆心下的最优半径和差多少", "从基线的布局出发，看哪些圆被挤到了边上"],
|
||||
},
|
||||
},
|
||||
},
|
||||
states_expected_score: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Does `entry` state a score, or an improvement of the score, that following this route is expected to reach?",
|
||||
focus:
|
||||
"Look for predictions of the result of this route. Quoting the baseline score, the best known score, or measured facts about the current program is not a prediction.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The entry predicts the score or the gain that this route will bring",
|
||||
examples: ["这条路预计能到 2.6 以上", "改完能提升约 0.5"],
|
||||
},
|
||||
false: {
|
||||
what: "The entry predicts no score or gain for this route",
|
||||
not_for: "Quoting the current score, the best known score, or the size of the current gap without predicting what the route reaches",
|
||||
examples: ["基线现在是 0.96，已知最好是 2.636", "现在离已知最好还差 1.68"],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/** 第 2、6 问：两位研究者从这两个入口出发，多半会做同一类代码改动吗。 */
|
||||
export const SAME_KIND: Record<string, Noul> = {
|
||||
same_kind_of_change: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Would two researchers, one starting from `entry_a` and one from `entry_b`, most likely make the same kind of code change?",
|
||||
focus:
|
||||
"Imagine each researcher follows their entry and writes one code change. Compare which part of the program they would change and by what technique. " +
|
||||
"Different wording, a different first file to read, or a different motivation does not make the changes different if both would change the same thing in the same way.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Both would most likely change the same part of the program with the same technique",
|
||||
examples: [
|
||||
"a: 用约束优化器同时调整圆心和半径; b: 把 78 个变量一起交给 SLSQP",
|
||||
"a: 换成六角网格布局; b: 把圆心改成错排的行，调行数",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "They would most likely change different parts of the program, or the same part with a different technique",
|
||||
not_for: "Entries that merely share the goal of a higher score, or read the same function, but would change different things",
|
||||
examples: ["a: 圆心固定，用线性规划求半径; b: 用约束优化器同时调整圆心和半径", "a: 换成六角网格布局; b: 只读评测规则，弄清容差"],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,51 @@
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { entriesIn, entryDecision, ofKind, readLedger, sameAssignment, stages, type Assignment } from "../lib/ledger.ts";
|
||||
import { replayModels, startReplay, type ReplayScript } from "../lib/replay.ts";
|
||||
import { launch, newEvalDir, writeConfig } from "../lib/study.ts";
|
||||
|
||||
// 把参考项目 16-real-1 的第零轮原样交给新 flow 回放一遍（任务书 12 第 5.3 节）：基线照样用真 Docker 跑，
|
||||
// 三位研究者由回放假模型按当年的先后交提交。跑完核对分配是否和当年完全相同，列出被退回的提交和没交出去的提交。
|
||||
// node eval/round-zero/replay.ts <名字>
|
||||
// 研究在 <检验目录>/study；之后用 quality.ts 打分。
|
||||
|
||||
type Script = ReplayScript & { expected: Assignment[]; expectedEntries: string[] };
|
||||
|
||||
const name = process.argv[2];
|
||||
if (name === undefined) throw new Error("要写检验名字，例如 r0-replay-16");
|
||||
const script = JSON.parse(readFileSync("eval/round-zero/replay-16-real-1.json", "utf8")) as Script;
|
||||
const out = newEvalDir(name);
|
||||
const replay = await startReplay(script, join(out, "pi-agent"), 0);
|
||||
const researchers = Object.keys(script.models).sort();
|
||||
const config = writeConfig("eval/round-zero/research.json", join(out, "research.json"), name, replayModels(researchers, "jev-latest"));
|
||||
const studyDir = join(out, "study");
|
||||
const { outcome } = await launch(config, studyDir, replay.env).done;
|
||||
await replay.close();
|
||||
|
||||
const records = readLedger(studyDir);
|
||||
const decision = entryDecision(records);
|
||||
const got = decision === undefined ? null : decision.output;
|
||||
const entries = entriesIn(records, new Set(stages(records).filter((s) => s.round === 0).map((s) => s.stageId))).map((e) => e.id);
|
||||
// 入口全体按编号比（账本里按各人最后一版清单的先后排，当年的清单按编号排）；分配按研究者、入口、志愿三样比，不看对象里键的先后。
|
||||
const sameEntries = JSON.stringify([...entries].sort()) === JSON.stringify([...script.expectedEntries].sort());
|
||||
const same = got !== null && sameAssignment(got, script.expected);
|
||||
const report = {
|
||||
outcome,
|
||||
assignment: got,
|
||||
expected: script.expected,
|
||||
sameAsReference: same,
|
||||
entries,
|
||||
expectedEntries: script.expectedEntries,
|
||||
sameEntriesAsReference: sameEntries,
|
||||
rejections: ofKind(records, "rejection").map((r) => ({ seq: r.seq, stageId: r.content.stageId, author: r.content.author, kind: r.content.kind, reason: r.content.reason })),
|
||||
delivered: replay.delivered,
|
||||
notDelivered: replay.remaining(),
|
||||
};
|
||||
writeFileSync(join(out, "replay-report.json"), `${JSON.stringify(report, null, 2)}\n`);
|
||||
console.log(`\n结局：${JSON.stringify(outcome)}`);
|
||||
console.log(`分配：${got === null ? "没有写出分配决定" : got.map((a) => `${a.researcher} → ${a.entry}（第 ${a.rank} 志愿）`).join("；")}`);
|
||||
console.log(`当年：${script.expected.map((a) => `${a.researcher} → ${a.entry}（第 ${a.rank} 志愿）`).join("；")}`);
|
||||
console.log(`分配和当年相同：${same ? "是" : "否"}；入口全体和当年相同：${sameEntries ? "是" : "否"}（${entries.join("、")}）`);
|
||||
console.log(`退回 ${report.rejections.length} 份${report.rejections.map((r) => `\n ${r.author} ${r.stageId} ${r.kind}：${r.reason}`).join("")}`);
|
||||
console.log(`没交出去的：${JSON.stringify(report.notDelivered)}`);
|
||||
console.log(`报告：${join(out, "replay-report.json")}；研究目录：${studyDir}`);
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"studyId": "r0-eval",
|
||||
"task": "tasks/circle_packing",
|
||||
"models": {
|
||||
"researchers": [
|
||||
{ "provider": "iomgaa", "model": "glm-5.3", "thinking": "medium" },
|
||||
{ "provider": "iomgaa", "model": "glm-5.3", "thinking": "medium" },
|
||||
{ "provider": "iomgaa", "model": "glm-5.3", "thinking": "medium" }
|
||||
],
|
||||
"experimenter": { "provider": "iomgaa", "model": "glm-5.3", "thinking": "medium" },
|
||||
"verifier": { "provider": "iomgaa", "model": "glm-5.3", "thinking": "medium" },
|
||||
"jev": "jev-latest"
|
||||
},
|
||||
"proposalLimit": 1,
|
||||
"batchLimit": 2,
|
||||
"runs": 13,
|
||||
"factorLimit": 3,
|
||||
"maxRepairs": 1,
|
||||
"lightChecks": { "maxCount": 2, "seconds": 5 },
|
||||
"promptDir": "prompts",
|
||||
"devEntry": false,
|
||||
"flow": {
|
||||
"preset": "full",
|
||||
"roundZero": { "enabled": true, "discussions": 2 },
|
||||
"firstPrediction": true,
|
||||
"critique": true,
|
||||
"ablation": { "enabled": true, "cellRule": "valueMinusCost" },
|
||||
"batchRule": "bestExpected",
|
||||
"openReserve": 0.1,
|
||||
"knowledgeUse": 0.5,
|
||||
"decay": 0.5
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { join } from "node:path";
|
||||
import { launch, newEvalDir, writeConfig } from "../lib/study.ts";
|
||||
|
||||
// 用真模型、真 Docker 跑一次第零轮（任务书 12 第 5.2 节"同样的配置真实跑 3 次"）：配置照 eval/round-zero/research.json，
|
||||
// 研究编号换成检验名字。研究在 <检验目录>/study，跑完用 quality.ts 打分。要花钱。
|
||||
// node eval/round-zero/run.ts <名字>
|
||||
// 中途按 Ctrl-C 会把停下信号传给研究进程；接着跑用 node eval/lib/cli.ts --resume --dir <检验目录>/study。
|
||||
|
||||
const name = process.argv[2];
|
||||
if (name === undefined) throw new Error("要写检验名字，例如 r0-real-1");
|
||||
const out = newEvalDir(name);
|
||||
const config = writeConfig("eval/round-zero/research.json", join(out, "research.json"), name, null);
|
||||
const study = launch(config, join(out, "study"), {});
|
||||
process.on("SIGINT", () => study.child.kill("SIGINT"));
|
||||
const { outcome } = await study.done;
|
||||
console.log(`\n结局：${JSON.stringify(outcome)}\n研究目录：${join(out, "study")}`);
|
||||
+9
-2
@@ -7,10 +7,17 @@
|
||||
"node": ">=22.19.0"
|
||||
},
|
||||
"scripts": {
|
||||
"check": "tsc --noEmit -p tsconfig.json && depcruise src tests --config .dependency-cruiser.cjs",
|
||||
"check": "tsc --noEmit -p tsconfig.json && depcruise src tests eval --config .dependency-cruiser.cjs",
|
||||
"test": "node --test \"tests/**/*.test.ts\"",
|
||||
"test:pi": "node tests/agents-pi.ts",
|
||||
"test:jev": "node tests/jev-real.ts"
|
||||
"test:jev": "node tests/jev-real.ts",
|
||||
"eval:study": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/lib/cli.ts",
|
||||
"eval:r0:run": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/round-zero/run.ts",
|
||||
"eval:r0:quality": "set -a && . /home/iomgaa/Projects/RivalBits/.env && set +a && node eval/round-zero/quality.ts",
|
||||
"eval:r0:extract": "node eval/round-zero/extract-16.ts",
|
||||
"eval:r0:replay": "node eval/round-zero/replay.ts",
|
||||
"eval:r0:kill": "node eval/round-zero/kill.ts",
|
||||
"eval:r0:concurrent": "node eval/round-zero/concurrent.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-ai": "0.85.1",
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import { rmSync } from "node:fs";
|
||||
import type { Content } from "../ledger/index.ts";
|
||||
import { formalRun as runDefinition, judge, prepareJudging, prepareRun } from "../tasks/index.ts";
|
||||
import type { HypothesisMetric, JudgedResult, ProcessResult } from "../vocab/index.ts";
|
||||
import { checkoutVersion } from "../workspace/index.ts";
|
||||
import { containerName, freshDir, judgeContainerName, runPaths, workspaceRepo } from "./paths.ts";
|
||||
import type { Study } from "./study.ts";
|
||||
|
||||
// 共用动作二：跑一次正式运行（任务书 12 第 2.7 节）。先登记、再启动、再写结果，前后顺序由这个入口保证
|
||||
// （方法.md 第七、九部分）：登记占名额，确认启动才扣次，确认没启动的写结果"没启动"、退还名额。
|
||||
// 判分不扣次数，不登记。恢复时接着等一次已经启动的运行，走同一个 finishRun。
|
||||
|
||||
export type RunPlan = Omit<Content<"runPlanned">, "definition">;
|
||||
|
||||
/**
|
||||
* 一次正式运行怎么结束：写完结果；被用户停下；或 docker run 没启动，附 Docker 的报错原文。
|
||||
* 没启动是外部错误，本进程不重试（CLAUDE.md"错误怎么处理"，重试 0 次）：调用方写停下事件，
|
||||
* 下次启动时"下一步做什么"照旧起新编号重跑。
|
||||
*/
|
||||
export type RunOutcome = "done" | "stopped" | { notStarted: string };
|
||||
|
||||
/** 跑一次正式运行：取出 version 的代码，在任务镜像里跑，判分，写结果。用户停下时不等容器，交回 "stopped"。 */
|
||||
export async function formalRun(s: Study, plan: RunPlan, version: string, metrics: HypothesisMetric[]): Promise<RunOutcome> {
|
||||
const definition = runDefinition(s.task, s.frozen.image, String(s.frozen.seed));
|
||||
s.ledger.append("runPlanned", { ...plan, definition });
|
||||
const paths = runPaths(s.dir, plan.run);
|
||||
freshDir(paths.base);
|
||||
checkoutVersion(workspaceRepo(s.dir), version, freshDir(paths.code));
|
||||
prepareRun(s.task, paths.code, freshDir(paths.run), metrics);
|
||||
const started = await s.runner.start(containerName(s.frozen.config.studyId, plan.run), definition, paths.run);
|
||||
if (!started.started) {
|
||||
s.ledger.append("runResult", { run: plan.run, process: { ending: "notStarted", reason: started.reason }, judged: null });
|
||||
return { notStarted: started.reason };
|
||||
}
|
||||
s.ledger.append("runStarted", { run: plan.run });
|
||||
return finishRun(s, plan.run, metrics);
|
||||
}
|
||||
|
||||
/**
|
||||
* 等一次已经确认启动的运行结束，判分，写结果，删掉容器。日志写进这次运行自己的日志目录。
|
||||
* 收到"停下"信号就不等了，直接交回 "stopped"；容器不管它，下次启动时恢复入口再来接着等。
|
||||
*/
|
||||
export async function finishRun(s: Study, run: string, metrics: HypothesisMetric[]): Promise<"done" | "stopped"> {
|
||||
const name = containerName(s.frozen.config.studyId, run);
|
||||
const paths = runPaths(s.dir, run);
|
||||
const waited = await untilStopped(s.stop, s.runner.wait(name, emptyDir(paths.logs)));
|
||||
if (waited === "stopped") return "stopped";
|
||||
const judged = await judgeRun(s, run, waited, metrics);
|
||||
s.ledger.append("runResult", { run, process: waited, judged });
|
||||
await s.runner.remove(name);
|
||||
return "done";
|
||||
}
|
||||
|
||||
/**
|
||||
* 判分：tasks 的 prepareJudging，要跑校验器就让 runner 在镜像里跑，再交给 judge。
|
||||
* 判分目录每次重新建空的（恢复时会再判一次）；判分容器还在就先删掉再用。
|
||||
*/
|
||||
async function judgeRun(s: Study, run: string, formal: ProcessResult, metrics: HypothesisMetric[]): Promise<JudgedResult> {
|
||||
const paths = runPaths(s.dir, run);
|
||||
const step = prepareJudging(s.task, paths.run, emptyDir(paths.judge), s.frozen.image, formal, metrics);
|
||||
if (step.status === "judged") return step.result;
|
||||
const name = judgeContainerName(s.frozen.config.studyId, run);
|
||||
if ((await s.runner.status(name)) !== "absent") await s.runner.remove(name);
|
||||
const started = await s.runner.start(name, step.definition, paths.judge);
|
||||
if (!started.started) throw new Error(`运行 ${run} 的校验器容器没有启动：${started.reason}`);
|
||||
const validator = await s.runner.wait(name, emptyDir(paths.judgeLogs));
|
||||
await s.runner.remove(name);
|
||||
return judge(s.task, paths.run, paths.judge, formal, validator, metrics);
|
||||
}
|
||||
|
||||
/** 清空重建一个目录：恢复时会再等一次、再判一次，上一次没写完的日志和判分目录不留。 */
|
||||
function emptyDir(dir: string): string {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
return freshDir(dir);
|
||||
}
|
||||
|
||||
/** 等一件事做完，或者收到"停下"信号先交回 "stopped"。被放下的那个 Promise 之后的结果没人用。 */
|
||||
async function untilStopped<T>(stop: AbortSignal, work: Promise<T>): Promise<T | "stopped"> {
|
||||
let onStop = (): void => undefined;
|
||||
const stopped = new Promise<"stopped">((resolve) => {
|
||||
onStop = () => resolve("stopped");
|
||||
if (stop.aborted) onStop();
|
||||
else stop.addEventListener("abort", onStop, { once: true });
|
||||
});
|
||||
try {
|
||||
return await Promise.race([work, stopped]);
|
||||
} finally {
|
||||
stop.removeEventListener("abort", onStop);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import { checkConfig, type ResearchConfig, type StepList } from "../config/index.ts";
|
||||
import type { LedgerRecord } from "../ledger/index.ts";
|
||||
import type { StudyRecord } from "../materials/index.ts";
|
||||
import { checkTaskFacts, type TaskFacts } from "../tasks/index.ts";
|
||||
|
||||
// 从账本读回冻结的研究配置（方法.md 4.1：之后的流程只读这份冻结的配置）。纯函数，"下一步做什么"也用它。
|
||||
|
||||
/** 完整流程的配置。朴素循环和步骤开关以后再做（docs/结构.md 第五部分），flow 现在只走完整流程。 */
|
||||
export type FullConfig = ResearchConfig & { flow: Extract<ResearchConfig["flow"], { preset: "full" }> };
|
||||
|
||||
export function fullConfig(config: ResearchConfig): FullConfig {
|
||||
if (config.flow.preset !== "full") {
|
||||
throw new Error("flow 现在只实现完整流程，朴素循环还没写（docs/结构.md 第五部分\"步骤开关和对照组\"）");
|
||||
}
|
||||
return config as FullConfig;
|
||||
}
|
||||
|
||||
/** 冻结的研究配置：配置、任务事实再检查过一遍；步骤清单是 config 的 stepList 写下的，照原样用。 */
|
||||
export type Frozen = {
|
||||
config: FullConfig;
|
||||
steps: StepList;
|
||||
facts: TaskFacts;
|
||||
prompts: StudyRecord["prompts"];
|
||||
image: string;
|
||||
seed: number;
|
||||
};
|
||||
|
||||
/** 账本里还没有研究配置记录就是新研究，交回 null。 */
|
||||
export function readFrozen(records: readonly LedgerRecord[]): Frozen | null {
|
||||
const record = records.find((r): r is LedgerRecord<"config"> => r.kind === "config");
|
||||
if (record === undefined) return null;
|
||||
const c = record.content as unknown as StudyRecord;
|
||||
return {
|
||||
config: fullConfig(checkConfig(c.config)),
|
||||
steps: c.steps as unknown as StepList,
|
||||
facts: checkTaskFacts(c.task),
|
||||
prompts: c.prompts,
|
||||
image: c.image,
|
||||
seed: c.seed,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
// flow：按冻结的步骤清单推进一份研究；每次启动走同一条恢复路径（docs/结构.md 第二部分）。
|
||||
// 对外只有 runStudy，以后 plugin 也用它。
|
||||
export { runStudy, type StudyInput } from "./loop.ts";
|
||||
export type { StudyEnding } from "./study.ts";
|
||||
@@ -0,0 +1,99 @@
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import type { ResearchConfig } from "../config/index.ts";
|
||||
import { Ledger } from "../ledger/index.ts";
|
||||
import { Runner } from "../runner/index.ts";
|
||||
import { fullConfig, readFrozen } from "./frozen.ts";
|
||||
import { EVENTS, nextStep, type Next } from "./next-step.ts";
|
||||
import { ledgerFile } from "./paths.ts";
|
||||
import { recover } from "./recover.ts";
|
||||
import { baselineRun, saveBaseline } from "./steps/baseline.ts";
|
||||
import { assignEntries, dispatchStage } from "./steps/round-zero.ts";
|
||||
import { freezeConfig, halt, openStudy, USER_STOPPED, type Study, type StudyEnding } from "./study.ts";
|
||||
|
||||
// 主循环：读账本 → 问下一步 → 做这一步 → 回到开头（任务书 12 第 2.4 节）。每做完一步都重新读账本。
|
||||
// 每次启动走同一条路：新研究先冻结配置；之后把研究接上（核对镜像和提示词）、核对未完成的动作，再进入主循环。
|
||||
|
||||
export type StudyInput = {
|
||||
/** 研究目录：账本、代码仓库、每次运行和每次模型调用的目录都在这里（任务书 12 第 2.1 节）。 */
|
||||
studyDir: string;
|
||||
/** 新研究交 checkConfig 检查过的配置；接着跑一份已有的研究写 null，以账本里冻结的那份为准。 */
|
||||
config: ResearchConfig | null;
|
||||
/** 用户停下研究：正在跑的模型会话立即停，正在跑的容器不管它，下次启动时由恢复入口接着等。 */
|
||||
stop: AbortSignal;
|
||||
};
|
||||
|
||||
export async function runStudy(input: StudyInput): Promise<StudyEnding> {
|
||||
const dir = resolve(input.studyDir);
|
||||
const config = input.config === null ? null : fullConfig(input.config);
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const ledger = Ledger.open(ledgerFile(dir));
|
||||
try {
|
||||
const frozen = readFrozen(ledger.records()) !== null;
|
||||
if (!frozen && config === null) throw new Error(`研究目录 ${dir} 里还没有研究，配置不能写 null`);
|
||||
if (frozen && config !== null) throw new Error(`研究目录 ${dir} 里已经有冻结的配置：接着跑时配置写 null，以账本里冻结的那份为准`);
|
||||
// Docker 守护进程的地址交给 docker 命令按自己的环境变量和 context 决定。
|
||||
const runner = new Runner(null);
|
||||
// 恢复入口：把研究接上，核对未完成的动作，之后才问下一步（任务书 12 第 2.5 节）。新研究冻结配置之后走同一条路。
|
||||
const begin = async (): Promise<Study | StudyEnding> => {
|
||||
const opened = await openStudy(dir, ledger, runner, input.stop);
|
||||
if ("halt" in opened) return halt(ledger, opened.halt, []);
|
||||
const recovered = await recover(opened);
|
||||
if (recovered === "stopped") return USER_STOPPED;
|
||||
if (recovered !== null) return { status: "halted", reason: recovered.halt };
|
||||
return opened;
|
||||
};
|
||||
let study: Study | null = null;
|
||||
if (frozen) {
|
||||
const begun = await begin();
|
||||
if ("status" in begun) return begun;
|
||||
study = begun;
|
||||
}
|
||||
for (;;) {
|
||||
if (input.stop.aborted) return USER_STOPPED;
|
||||
const next = nextStep(ledger.records());
|
||||
if (next.do === "freezeConfig") {
|
||||
await freezeConfig(dir, ledger, runner, config!);
|
||||
const begun = await begin();
|
||||
if ("status" in begun) return begun;
|
||||
study = begun;
|
||||
continue;
|
||||
}
|
||||
const ended = await doStep(study!, next);
|
||||
if (ended !== null) return ended;
|
||||
}
|
||||
} finally {
|
||||
ledger.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** 做一步。交回 null 表示接着走下一步。 */
|
||||
async function doStep(s: Study, next: Exclude<Next, { do: "freezeConfig" }>): Promise<StudyEnding | null> {
|
||||
switch (next.do) {
|
||||
case "saveBaseline":
|
||||
saveBaseline(s);
|
||||
return null;
|
||||
case "baselineRun": {
|
||||
const ran = await baselineRun(s, next.run);
|
||||
if (ran === "stopped") return USER_STOPPED;
|
||||
if (ran === "done") return null;
|
||||
return halt(s.ledger, `运行 ${next.run} 没有启动，名额已退还；本进程不重跑，下次启动时起新编号重跑。Docker 的报错：${ran.notStarted}`, [next.run]);
|
||||
}
|
||||
case "skip":
|
||||
s.ledger.append("event", { event: EVENTS.skipped, reason: next.reason, objects: [next.step] });
|
||||
return null;
|
||||
case "openStage":
|
||||
s.ledger.append("stageOpen", { stageId: next.stageId, stage: next.stage, round: 0, roster: next.roster });
|
||||
return null;
|
||||
case "closeStage":
|
||||
s.ledger.append("stageClose", { stageId: next.stageId });
|
||||
return null;
|
||||
case "dispatch":
|
||||
return dispatchStage(s, next);
|
||||
case "assignEntries":
|
||||
assignEntries(s);
|
||||
return null;
|
||||
case "halt":
|
||||
return halt(s.ledger, next.reason, next.objects);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
import { runSession, type KindOf, type SessionEnding, type SubmitAttempt, type SubmitReceipt } from "../agents/index.ts";
|
||||
import { buildMaterial, callId, queryMaterial, researcherId, type TaskMaterial } from "../materials/index.ts";
|
||||
import { receive, type Incoming, type KnownOf } from "../submissions/index.ts";
|
||||
import type { JevJudgment, Json, Submission } from "../vocab/index.ts";
|
||||
import { checkoutVersion } from "../workspace/index.ts";
|
||||
import { callPaths, freshDir, workspaceRepo } from "./paths.ts";
|
||||
import type { Study } from "./study.ts";
|
||||
|
||||
// 共用动作一：派一个模型任务（任务书 12 第 2.6 节）。先写"模型调用开始"（材料整份存进去），再开会话，
|
||||
// 最后写"模型调用结束"，前后顺序由这个入口保证（方法.md 第九部分"先登记、再执行、再登记结果"）。
|
||||
// 提交收不收只由 submissions 判；形状不合格的由 agents 报上来，这里直接记退回（"给 flow"）。
|
||||
|
||||
/** 一个研究者任务：哪种提交、谁、哪个阶段、对象；known 是交上来之后才知道的东西（submissions 的 KnownOf）。 */
|
||||
export type ModelTask<K extends KindOf<"researcher">> = {
|
||||
kind: K;
|
||||
actor: string;
|
||||
stageId: string;
|
||||
object: string | null;
|
||||
known(content: Submission<K>): KnownOf<K>;
|
||||
};
|
||||
|
||||
/** 派一次任务，等会话结束，交回调用编号和结局。 */
|
||||
export async function runModelTask<K extends KindOf<"researcher">>(s: Study, t: ModelTask<K>): Promise<{ call: string; ending: SessionEnding }> {
|
||||
const { ledger } = s;
|
||||
const started = new Set(ledger.records().flatMap((r) => (r.kind === "callStarted" ? [r.content.call] : [])));
|
||||
let attempt = 1;
|
||||
while (started.has(callId(t.stageId, t.actor, t.kind, t.object, attempt))) attempt++;
|
||||
const call = callId(t.stageId, t.actor, t.kind, t.object, attempt);
|
||||
const paths = callPaths(s.dir, call);
|
||||
const material = buildMaterial({
|
||||
kind: t.kind,
|
||||
actor: t.actor,
|
||||
stageId: t.stageId,
|
||||
object: t.object,
|
||||
ledger,
|
||||
study: s.facts,
|
||||
prompts: s.prompts,
|
||||
codeRoot: paths.code,
|
||||
changes: null,
|
||||
artifacts: null,
|
||||
});
|
||||
const model = researcherModel(s, t.actor);
|
||||
// 先登记、后建目录：调用编号只在登记之后才算用掉，被杀在两者之间时下次的编号不会撞上旧目录。
|
||||
ledger.append("callStarted", {
|
||||
call,
|
||||
actor: t.actor,
|
||||
stageId: t.stageId,
|
||||
model: `${model.provider}/${model.model}（思考档位 ${model.thinking}）`,
|
||||
materials: material as unknown as Json,
|
||||
});
|
||||
freshDir(paths.base);
|
||||
freshDir(paths.code);
|
||||
for (const r of material.readable) checkoutVersion(workspaceRepo(s.dir), r.version, freshDir(r.dir));
|
||||
let reply = 0;
|
||||
const outcome = await runSession({
|
||||
role: "researcher",
|
||||
model,
|
||||
systemPrompt: s.prompts.roles.researcher.text,
|
||||
firstMessage: material.firstMessage,
|
||||
workDir: freshDir(paths.work),
|
||||
sessionDir: freshDir(paths.sessions),
|
||||
kind: t.kind,
|
||||
readableDirs: material.readable.map((r) => r.dir),
|
||||
query: (ref) => queryMaterial(material, ref),
|
||||
submit: (attempt) => submit(s, t, material, attempt as SubmitAttempt<K>),
|
||||
usage: (usage) => void ledger.append("callUsage", { call, reply: ++reply, usage }),
|
||||
writeNote: (body) => void ledger.append("note", { author: t.actor, body }),
|
||||
stop: s.stop,
|
||||
});
|
||||
// 会话文件就在这次调用的 sessions 目录里，不另记。
|
||||
const { wallSeconds, sessionFile: _sessionFile, ...ending } = outcome;
|
||||
ledger.append("callEnded", { call, wallSeconds: { status: "measured", value: wallSeconds }, ...ending });
|
||||
return { call, ending };
|
||||
}
|
||||
|
||||
/** 研究者用的模型：第 i 位研究者（researcherId(i)）用配置里第 i 个。 */
|
||||
function researcherModel(s: Study, actor: string) {
|
||||
const models = s.frozen.config.models.researchers;
|
||||
const i = models.findIndex((_, k) => researcherId(k) === actor);
|
||||
if (i < 0) throw new Error(`${actor} 不是这份研究的研究者`);
|
||||
return models[i]!;
|
||||
}
|
||||
|
||||
/**
|
||||
* 一份提交收不收。这个任务只有一份提交（第零轮每人每阶段一个任务、一份提交）：
|
||||
* 回给模型的提交编号就是这一份的位置，不分版本；替换时写它，接在最新一版后面。
|
||||
* 第一次交以后不写 replaces 又交，或 replaces 写的不是这一份，都退回并说明该怎么写。
|
||||
*/
|
||||
async function submit<K extends KindOf<"researcher">>(s: Study, t: ModelTask<K>, material: TaskMaterial, attempt: SubmitAttempt<K>): Promise<SubmitReceipt> {
|
||||
const { ledger } = s;
|
||||
const slot = { stageId: t.stageId, author: t.actor, kind: t.kind, target: t.object, item: null };
|
||||
const reject = (submitted: Json, reason: string, jev: JevJudgment[] | null): SubmitReceipt => {
|
||||
ledger.append("rejection", { stageId: t.stageId, author: t.actor, kind: t.kind, target: t.object, submitted, reason, jev });
|
||||
return { accepted: false, reason };
|
||||
};
|
||||
if (attempt.shape === "invalid") return reject(attempt.raw, attempt.reason, null);
|
||||
const id = receiptId(t);
|
||||
const latest = ledger.latestSubmission(slot)?.content.version ?? 0;
|
||||
const content = attempt.content as unknown as Json;
|
||||
if (attempt.replaces === null && latest > 0) {
|
||||
return reject(content, `本阶段你已经交过这一份（提交编号 ${id}）。要改就在 replaces 里写 ${id} 整份替换，不要另交`, null);
|
||||
}
|
||||
if (attempt.replaces !== null && (attempt.replaces !== id || latest === 0)) {
|
||||
const which = latest === 0 ? "你本阶段还没有收下的提交，第一次交时不要带 replaces" : `要替换就写 ${id}`;
|
||||
return reject(content, `replaces 写的 ${attempt.replaces} 不是你本阶段交过的提交：${which}`, null);
|
||||
}
|
||||
const verdict = await receive({
|
||||
kind: t.kind,
|
||||
content: attempt.content,
|
||||
checklist: material.checklist,
|
||||
known: t.known(attempt.content),
|
||||
jevModel: s.frozen.config.models.jev,
|
||||
} as Incoming);
|
||||
if (!verdict.accepted) return reject(content, verdict.reason, verdict.jev);
|
||||
ledger.append("submission", { ...slot, version: latest + 1, content: attempt.content, jev: verdict.jev } as never);
|
||||
return { accepted: true, id };
|
||||
}
|
||||
|
||||
/** 回给模型的提交编号：阶段、本人、提交种类、对象。 */
|
||||
const receiptId = (t: { stageId: string; actor: string; kind: string; object: string | null }): string =>
|
||||
`${t.stageId}.${t.actor}.${t.kind}.${t.object ?? "none"}`;
|
||||
@@ -0,0 +1,160 @@
|
||||
import type { StudyStepId } from "../config/index.ts";
|
||||
import type { Content, LedgerRecord } from "../ledger/index.ts";
|
||||
import { ORIGINAL_BASELINE, researcherId } from "../materials/index.ts";
|
||||
import type { Stage } from "../vocab/index.ts";
|
||||
import { readFrozen, type Frozen } from "./frozen.ts";
|
||||
|
||||
// "下一步做什么"（docs/结构.md 第二部分"几条边界"）：步骤的先后只写在这里。
|
||||
// 纯函数：输入账本记录（冻结的步骤清单就在其中的研究配置记录里），输出下一步要做的一件事；
|
||||
// 不读文件、不调模型，同样的记录永远得到同样的下一步。账本不存"进行到第几步"，每次从记录现算。
|
||||
//
|
||||
// 每一步做完以哪条记录为准，见 docs/结构.md 第八部分 flow 一节的表；下面每个步骤的分支各注一行。
|
||||
// 没写成那条记录的步骤，下次会被重做或报错，不靠"这条记录公开了没有"这类隐含约定去猜。
|
||||
|
||||
/** 第零轮的研究者阶段：步骤 → 阶段名（vocab STAGES）和这个阶段交的提交种类。 */
|
||||
const ROUND_ZERO_STAGES = {
|
||||
entryDraft: { stage: "entryDraft", kind: "entry" },
|
||||
discussion1: { stage: "discussion", kind: "discussion" },
|
||||
discussion2: { stage: "discussion", kind: "discussion" },
|
||||
claim: { stage: "claim", kind: "claim" },
|
||||
} as const satisfies Partial<Record<StudyStepId, { stage: Stage; kind: string }>>;
|
||||
type RoundZeroStep = keyof typeof ROUND_ZERO_STAGES;
|
||||
export type RoundZeroKind = (typeof ROUND_ZERO_STAGES)[RoundZeroStep]["kind"];
|
||||
|
||||
/** 阶段编号由 flow 起：第零轮是 r0-<步骤>，一个步骤一个阶段。 */
|
||||
export const stageIdOf = (step: RoundZeroStep): string => `r0-${step}`;
|
||||
/** 基线运行编号：没启动就起下一个编号重跑（docs/结构.md 第四部分）。 */
|
||||
export const baselineRunId = (n: number): string => `base-${n}`;
|
||||
|
||||
/** flow 写的事件种类（账本记录 event）。 */
|
||||
export const EVENTS = {
|
||||
/** 按配置跳过的步骤，objects 是步骤编号，reason 是 config 给的原因。 */
|
||||
skipped: "skippedByConfig",
|
||||
/** 研究停下，reason 是停下的原因，objects 是相关的调用或运行。 */
|
||||
halted: "halted",
|
||||
/** 恢复时问不到 Docker，不知道一次运行有没有启动（方法.md 第七部分"状态未知"），objects 是运行编号。 */
|
||||
runStatusUnknown: "runStatusUnknown",
|
||||
} as const;
|
||||
|
||||
export type Next =
|
||||
| { do: "freezeConfig" }
|
||||
| { do: "saveBaseline" }
|
||||
| { do: "baselineRun"; run: string }
|
||||
| { do: "skip"; step: StudyStepId; reason: string }
|
||||
| { do: "openStage"; stageId: string; stage: Stage; roster: string[] }
|
||||
| { do: "closeStage"; stageId: string }
|
||||
| { do: "dispatch"; stageId: string; kind: RoundZeroKind; actors: string[] }
|
||||
| { do: "assignEntries" }
|
||||
| { do: "halt"; reason: string; objects: string[] };
|
||||
|
||||
export function nextStep(records: readonly LedgerRecord[]): Next {
|
||||
// 冻结配置 → 研究配置记录。
|
||||
const frozen = readFrozen(records);
|
||||
if (frozen === null) return { do: "freezeConfig" };
|
||||
const view = new View(records);
|
||||
for (const step of frozen.steps.study) {
|
||||
if (step.status === "skip") {
|
||||
// 按配置跳过的步骤 → 一条"按配置跳过"事件，写明步骤和原因。
|
||||
if (!view.events(EVENTS.skipped).some((e) => e.objects[0] === step.id)) return { do: "skip", step: step.id, reason: step.reason };
|
||||
continue;
|
||||
}
|
||||
const next = stepNext(step.id, view, frozen);
|
||||
if (next !== null) return next;
|
||||
}
|
||||
throw new Error("步骤清单走完了却没有停下：rounds 一步应当停下");
|
||||
}
|
||||
|
||||
/** 一个要执行的步骤还要做什么；做完了交回 null。 */
|
||||
function stepNext(step: StudyStepId, view: View, frozen: Frozen): Next | null {
|
||||
switch (step) {
|
||||
case "freezeConfig":
|
||||
return null;
|
||||
case "baselineRun":
|
||||
return baselineNext(view);
|
||||
case "entryDraft":
|
||||
case "discussion1":
|
||||
case "discussion2":
|
||||
case "claim":
|
||||
return stageNext(step, view, frozen);
|
||||
case "assignEntries":
|
||||
// 分配 → 第 0 轮的入口分配决定（decision，种类 entries）。
|
||||
return view.of("decision").some((d) => d.decision === "entries" && d.round === 0) ? null : { do: "assignEntries" };
|
||||
case "rounds":
|
||||
return { do: "halt", reason: "第 1 轮还没写（任务书 13）", objects: [] };
|
||||
case "export":
|
||||
throw new Error("走到了导出：rounds 一步应当先停下");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 基线 → 基线代码记录（对象 ORIGINAL_BASELINE），而且最近一次基线运行的结果不是"没启动"。
|
||||
* 没启动的起新编号重跑；判为不算数的停下（方法.md 4.1）。
|
||||
*/
|
||||
function baselineNext(view: View): Next | null {
|
||||
if (!view.of("code").some((c) => c.object === ORIGINAL_BASELINE)) return { do: "saveBaseline" };
|
||||
const plans = view.of("runPlanned").filter((p) => p.purpose === "baseline");
|
||||
const last = plans.at(-1);
|
||||
if (last === undefined) return { do: "baselineRun", run: baselineRunId(1) };
|
||||
const result = view.of("runResult").find((r) => r.run === last.run);
|
||||
if (result === undefined) throw new Error(`基线运行 ${last.run} 登记了还没有结果：恢复入口应当已经核对过`);
|
||||
if (result.process.ending === "notStarted") return { do: "baselineRun", run: baselineRunId(plans.length + 1) };
|
||||
if (result.judged?.verdict === "valid") return null;
|
||||
const why = result.judged === null ? result.process.ending : result.judged.verdictReason;
|
||||
return { do: "halt", reason: `基线运行 ${last.run} 判为不算数（${why}），不能当基线`, objects: [last.run] };
|
||||
}
|
||||
|
||||
/**
|
||||
* 研究者阶段 → 这个阶段的关闭记录（stageClose）。没开就开；名单上的人都有被收下的提交才关；
|
||||
* 没交上的人按最近一次调用怎么结束的来定派不派（任务书 12 第 2.3 节的表）。能派的先全派，派完还有人停在"没交"才停下。
|
||||
*/
|
||||
function stageNext(step: RoundZeroStep, view: View, frozen: Frozen): Next | null {
|
||||
const stageId = stageIdOf(step);
|
||||
const { stage, kind } = ROUND_ZERO_STAGES[step];
|
||||
const open = view.of("stageOpen").find((s) => s.stageId === stageId);
|
||||
if (open === undefined) {
|
||||
return { do: "openStage", stageId, stage, roster: frozen.config.models.researchers.map((_, i) => researcherId(i)) };
|
||||
}
|
||||
if (view.of("stageClose").some((s) => s.stageId === stageId)) return null;
|
||||
const missing = open.roster.filter((a) => !view.of("submission").some((s) => s.stageId === stageId && s.author === a));
|
||||
if (missing.length === 0) return { do: "closeStage", stageId };
|
||||
const actors: string[] = [];
|
||||
const stuck: Array<{ call: string; reason: string }> = [];
|
||||
for (const actor of missing) {
|
||||
const last = view.lastCall(stageId, actor);
|
||||
// 没有调用、被用户停下、出错（含恢复时补写的"进程中断"）都派；没交的只在开发入口开着时派（方法.md 第九部分"开发入口"）。
|
||||
if (last?.status === "notSubmitted" && !frozen.config.devEntry) stuck.push({ call: last.call, reason: last.reason });
|
||||
else actors.push(actor);
|
||||
}
|
||||
if (actors.length > 0) return { do: "dispatch", stageId, kind, actors };
|
||||
return {
|
||||
do: "halt",
|
||||
reason: `交不出合格提交（方法.md 4.3 ②：按内部错误停下）：${stuck.map((s) => `调用 ${s.call}，${s.reason}`).join("；")}`,
|
||||
objects: stuck.map((s) => s.call),
|
||||
};
|
||||
}
|
||||
|
||||
/** 按种类取记录内容的小工具。 */
|
||||
class View {
|
||||
readonly #records: readonly LedgerRecord[];
|
||||
|
||||
constructor(records: readonly LedgerRecord[]) {
|
||||
this.#records = records;
|
||||
}
|
||||
|
||||
of<K extends LedgerRecord["kind"]>(kind: K): Array<Content<K>> {
|
||||
return (this.#records.filter((r) => r.kind === kind) as Array<LedgerRecord<K>>).map((r) => r.content);
|
||||
}
|
||||
|
||||
events(event: string): Array<Content<"event">> {
|
||||
return this.of("event").filter((e) => e.event === event);
|
||||
}
|
||||
|
||||
/** 某人在某阶段最近一次模型调用的结束记录，没有调用为 null。开始了没结束的，恢复入口应当已经补写。 */
|
||||
lastCall(stageId: string, actor: string): Content<"callEnded"> | null {
|
||||
const started = this.of("callStarted").filter((c) => c.stageId === stageId && c.actor === actor).at(-1);
|
||||
if (started === undefined) return null;
|
||||
const ended = this.of("callEnded").find((e) => e.call === started.call);
|
||||
if (ended === undefined) throw new Error(`模型调用 ${started.call} 开始了还没有结束记录：恢复入口应当已经补写`);
|
||||
return ended;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
// 研究目录里放什么（任务书 12 第 2.1 节），全由 flow 管：
|
||||
// ledger.sqlite 账本
|
||||
// repo.git workspace 的裸仓库
|
||||
// runs/<运行编号>/ 一次正式运行：code 取出的代码、run 运行目录、logs 日志、judge 判分目录、judge-logs 校验器日志
|
||||
// calls/<调用编号>/ 一次模型调用：work 工作目录、sessions 会话文件、code 能读的代码版本（materials 的 codeRoot）
|
||||
|
||||
export const ledgerFile = (dir: string): string => join(dir, "ledger.sqlite");
|
||||
export const workspaceRepo = (dir: string): string => join(dir, "repo.git");
|
||||
|
||||
export function runPaths(dir: string, run: string) {
|
||||
const base = join(dir, "runs", run);
|
||||
return { base, code: join(base, "code"), run: join(base, "run"), logs: join(base, "logs"), judge: join(base, "judge"), judgeLogs: join(base, "judge-logs") };
|
||||
}
|
||||
|
||||
export function callPaths(dir: string, call: string) {
|
||||
const base = join(dir, "calls", call);
|
||||
return { base, work: join(base, "work"), sessions: join(base, "sessions"), code: join(base, "code") };
|
||||
}
|
||||
|
||||
/** 容器名带研究编号，在本机所有研究之间唯一（参考项目 9b 两份研究撞过名）；判分容器另起名字。 */
|
||||
export const containerName = (studyId: string, run: string): string => `inforsi-${studyId}-${run}`;
|
||||
export const judgeContainerName = (studyId: string, run: string): string => `inforsi-${studyId}-${run}-judge`;
|
||||
|
||||
/**
|
||||
* 新建一个空目录：上一层没有就建，目录本身已经存在就报错。取出代码、建运行目录、判分目录时目标都必须是空的
|
||||
* （workspace、tasks 不查）；每次运行、每次调用都有新编号，所以不会撞上旧目录。
|
||||
*/
|
||||
export function freshDir(path: string): string {
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
mkdirSync(path);
|
||||
return path;
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import type { Content } from "../ledger/index.ts";
|
||||
import type { HypothesisMetric } from "../vocab/index.ts";
|
||||
import { finishRun } from "./formal-run.ts";
|
||||
import { EVENTS } from "./next-step.ts";
|
||||
import { containerName } from "./paths.ts";
|
||||
import type { Study } from "./study.ts";
|
||||
|
||||
// 恢复入口第 3、4 步（任务书 12 第 2.5 节）：每次启动都走这里，不分第一次启动、正常重启还是被杀后重启。
|
||||
// 从账本找出已登记要做、还没有结果的动作，逐个核对，只核对不重做（CLAUDE.md"中断后怎么恢复"）。
|
||||
|
||||
/** 核对完交回 null；问不到 Docker 时写"状态未知"事件，交回停下的原因；等运行时被用户停下交回 "stopped"。 */
|
||||
export async function recover(s: Study): Promise<{ halt: string } | "stopped" | null> {
|
||||
const pending = s.ledger.pending();
|
||||
// 开始了还没结束的模型调用：控制进程中断了，补写结束记录，墙钟时间测不到（docs/结构.md 第四部分第 10 种）。
|
||||
for (const c of pending.calls) {
|
||||
const unmeasured = { status: "unmeasured", reason: "进程中断" } as const;
|
||||
s.ledger.append("callEnded", { call: c.content.call, wallSeconds: unmeasured, status: "failed", reason: "进程中断" });
|
||||
}
|
||||
// 登记了还没结果的正式运行：按容器名问 Docker（方法.md 第七部分"不确定有没有启动时，先核对"）。
|
||||
for (const { plan, started } of pending.runs) {
|
||||
const { run } = plan.content;
|
||||
const name = containerName(s.frozen.config.studyId, run);
|
||||
const status = await s.runner.status(name);
|
||||
if (status === "unreachable") {
|
||||
const reason = `恢复时问不到 Docker，不知道运行 ${run}（容器 ${name}）有没有启动：状态未知，等用户核对（方法.md 第七部分）`;
|
||||
s.ledger.append("event", { event: EVENTS.runStatusUnknown, reason, objects: [run] });
|
||||
return { halt: reason };
|
||||
}
|
||||
if (!started && (status === "absent" || status === "created")) {
|
||||
// 确认没启动：写结果"没启动"，退还名额；建了没启动的删掉。
|
||||
if (status === "created") await s.runner.remove(name);
|
||||
const reason = `恢复时核对：容器 ${name} ${status === "absent" ? "不存在" : "建了没有启动，已删掉"}，这次运行没有启动`;
|
||||
s.ledger.append("runResult", { run, process: { ending: "notStarted", reason }, judged: null });
|
||||
continue;
|
||||
}
|
||||
// 在跑或已结束：没写"确认启动"的补写（扣次），然后等它、判分、写结果。
|
||||
// 已经确认启动、容器却不见了，真实运行里没出现过，不专门写：runner.wait 会报错，按内部错误停下。
|
||||
if (!started) s.ledger.append("runStarted", { run });
|
||||
if ((await finishRun(s, run, metricsOf(plan.content))) === "stopped") return "stopped";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** 一次运行要测的假设专用指标。第 1 部分只有基线运行，它不测假设专用指标；正向、消融运行由第 3、4 部分补。 */
|
||||
function metricsOf(plan: Content<"runPlanned">): HypothesisMetric[] {
|
||||
if (plan.purpose !== "baseline") throw new Error(`运行 ${plan.run} 是 ${plan.purpose} 运行，flow 第 1 部分还不会恢复它`);
|
||||
return [];
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { ORIGINAL_BASELINE } from "../../materials/index.ts";
|
||||
import { saveVersion } from "../../workspace/index.ts";
|
||||
import { formalRun, type RunOutcome } from "../formal-run.ts";
|
||||
import { workspaceRepo } from "../paths.ts";
|
||||
import type { Study } from "../study.ts";
|
||||
|
||||
// 基线（方法.md 4.1）：存任务的基线代码，写代码记录，再用共用动作二跑一次；每份研究都跑，计入运行次数。
|
||||
|
||||
/** 存任务的基线目录（"不存"清单取冻结的任务事实），写代码记录：对象 ORIGINAL_BASELINE，位置表 null。 */
|
||||
export function saveBaseline(s: Study): void {
|
||||
const saved = saveVersion(workspaceRepo(s.dir), s.task.baselineDir, s.frozen.facts.unversionedNames);
|
||||
if (!saved.saved) throw new Error(`任务的基线目录 ${s.task.baselineDir} 存不成版本：${saved.path}，${saved.reason}`);
|
||||
s.ledger.append("code", { object: ORIGINAL_BASELINE, version: saved.version, locations: null });
|
||||
}
|
||||
|
||||
/** 跑一次基线运行。基线不测假设专用指标。 */
|
||||
export function baselineRun(s: Study, run: string): Promise<RunOutcome> {
|
||||
const code = s.ledger.records().findLast((r) => r.kind === "code" && r.content.object === ORIGINAL_BASELINE);
|
||||
const version = (code!.content as { version: string }).version;
|
||||
return formalRun(s, { run, purpose: "baseline", round: 0, object: null }, version, []);
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
import { finalEntries } from "../../materials/index.ts";
|
||||
import { assignEntries as assign } from "../../rules/index.ts";
|
||||
import type { Json } from "../../vocab/index.ts";
|
||||
import { runModelTask } from "../model-task.ts";
|
||||
import type { Next } from "../next-step.ts";
|
||||
import { halt, USER_STOPPED, type Study, type StudyEnding } from "../study.ts";
|
||||
|
||||
// 第零轮（方法.md 4.2）：三个研究者阶段派任务，认领之后由程序分配入口。
|
||||
|
||||
/**
|
||||
* 派一个研究者阶段里要派的人，同时跑，各自一个会话（任务书 12 第 2.6 节）。第零轮的提交种类交上来之后没有要另算的东西。
|
||||
* 有人没交上（没交或出错），等这次派发的会话全部结束后停下，不打断别人（第 2.4 节）。
|
||||
* 某个会话抛出内部错误时，也等别人结束再把它抛出去。
|
||||
*/
|
||||
export async function dispatchStage(s: Study, next: Extract<Next, { do: "dispatch" }>): Promise<StudyEnding | null> {
|
||||
const settled = await Promise.allSettled(
|
||||
next.actors.map((actor) => runModelTask(s, { kind: next.kind, actor, stageId: next.stageId, object: null, known: () => null })),
|
||||
);
|
||||
const failure = settled.find((r) => r.status === "rejected");
|
||||
if (failure !== undefined) throw failure.reason;
|
||||
if (s.stop.aborted) return USER_STOPPED;
|
||||
const results = settled.map((r) => (r as PromiseFulfilledResult<Awaited<ReturnType<typeof runModelTask>>>).value);
|
||||
const missed = results.flatMap(({ call, ending }) =>
|
||||
ending.status === "notSubmitted" || ending.status === "failed" ? [{ call, text: `${ending.status === "failed" ? "出错" : "没交"}：${ending.reason}` }] : [],
|
||||
);
|
||||
if (missed.length === 0) return null;
|
||||
return halt(s.ledger, `阶段 ${next.stageId} 有人没交上，等这次派发的会话全部结束后停下：${missed.map((m) => `调用 ${m.call}，${m.text}`).join("；")}`, missed.map((m) => m.call));
|
||||
}
|
||||
|
||||
/**
|
||||
* 分配（方法.md 4.2"分配规则"）：每人最后一份认领和最终有效的入口交给 rules 的 assignEntries，写一条入口分配决定。
|
||||
* assignEntries 不在规则登记表里，规则名就写它，版本 1；没有方法参数，也不用随机。
|
||||
*/
|
||||
export function assignEntries(s: Study): void {
|
||||
const roster = s.ledger.records().find((r) => r.kind === "stageOpen" && r.content.stage === "claim" && r.content.round === 0);
|
||||
const claim = roster!.content as { stageId: string; roster: string[] };
|
||||
const researchers = claim.roster.map((id) => {
|
||||
const last = s.ledger.latestSubmission({ stageId: claim.stageId, author: id, kind: "claim", target: null, item: null });
|
||||
return { id, choices: (last!.content.content as { choices: string[] }).choices };
|
||||
});
|
||||
const entries = finalEntries(s.ledger).map((e) => ({ id: e.id, author: e.author }));
|
||||
const input = { researchers, entries };
|
||||
s.ledger.append("decision", { decision: "entries", round: 0, rule: "assignEntries", ruleVersion: 1, params: null, seed: null, input, output: assign(input) as unknown as Json });
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
import { stepList } from "../config/index.ts";
|
||||
import type { Ledger } from "../ledger/index.ts";
|
||||
import { readPrompts, type Prompts, type StudyFacts, type StudyRecord } from "../materials/index.ts";
|
||||
import type { Runner } from "../runner/index.ts";
|
||||
import { loadTask, metricDefinitions, type Task } from "../tasks/index.ts";
|
||||
import { hash, ROLES, SUBMISSION_KINDS, type Json, type SubmissionKind } from "../vocab/index.ts";
|
||||
import { createRepository } from "../workspace/index.ts";
|
||||
import { readFrozen, type FullConfig, type Frozen } from "./frozen.ts";
|
||||
import { EVENTS } from "./next-step.ts";
|
||||
import { workspaceRepo } from "./paths.ts";
|
||||
|
||||
// 一份研究跑起来要的东西：研究目录、账本、Docker、冻结的配置，以及按冻结配置读进来的任务和提示词。
|
||||
// 研究开始时冻结配置（方法.md 4.1）；之后每次启动都从账本里那份冻结的配置重建，不再读配置文件和 task.json。
|
||||
|
||||
export type Study = {
|
||||
dir: string;
|
||||
ledger: Ledger;
|
||||
runner: Runner;
|
||||
stop: AbortSignal;
|
||||
frozen: Frozen;
|
||||
task: Task;
|
||||
prompts: Prompts;
|
||||
facts: StudyFacts;
|
||||
};
|
||||
|
||||
/**
|
||||
* 研究开始：核对研究编号没被用过，读任务和提示词、建镜像、建裸仓库，一次写进研究配置记录（任务书 12 第 2.8 节）。
|
||||
* 研究编号来自用户配置，是边界：容器名是 inforsi-<研究编号>-<运行编号>，编号重复的新研究会把旧研究遗留的容器
|
||||
* 当成自己的运行（docker run 重名失败后按名字查到它，算作启动了），扣次并记下别人的结果。
|
||||
* 每份研究冻结配置时都建 inforsi-<研究编号> 的镜像，本机已有这个标签就说明编号用过了，报错停下。
|
||||
*/
|
||||
export async function freezeConfig(dir: string, ledger: Ledger, runner: Runner, config: FullConfig): Promise<void> {
|
||||
const tag = `inforsi-${config.studyId}`;
|
||||
if (await runner.hasImage(tag)) {
|
||||
throw new Error(`研究编号 ${config.studyId} 已被本机用过（已有镜像 ${tag}），换一个研究编号再开新研究`);
|
||||
}
|
||||
const task = loadTask(config.task);
|
||||
const prompts = readPrompts(config.promptDir);
|
||||
// 冻结的是镜像编号，标签以后被别的构建挪走也不影响（docs/结构.md"依赖跟着任务走"）。
|
||||
const image = await runner.buildImage(task.dir, tag);
|
||||
createRepository(workspaceRepo(dir));
|
||||
const record: StudyRecord = {
|
||||
config: config as unknown as Json,
|
||||
steps: stepList(config) as unknown as Json,
|
||||
prompts: promptHashes(prompts),
|
||||
image,
|
||||
task: task.facts as unknown as Json,
|
||||
// 同一份研究的全部运行用同一个种子：研究编号的哈希取前 8 位十六进制（任务书 12 第 2.7 节）。
|
||||
seed: Number.parseInt(hash(config.studyId).slice(0, 8), 16),
|
||||
};
|
||||
ledger.append("config", record as unknown as Json);
|
||||
}
|
||||
|
||||
/**
|
||||
* 按账本里冻结的配置把研究接上（恢复入口第 2 步）：镜像还在、提示词没变，才往下走。
|
||||
* 镜像不在就停下，不重建（docs/结构.md"依赖跟着任务走"）；提示词变了而开发入口关着也停下（方法.md 第九部分）。
|
||||
* 交回研究，或停下的原因。
|
||||
*/
|
||||
export async function openStudy(dir: string, ledger: Ledger, runner: Runner, stop: AbortSignal): Promise<Study | { halt: string }> {
|
||||
const frozen = readFrozen(ledger.records())!;
|
||||
if (!(await runner.hasImage(frozen.image))) {
|
||||
return { halt: `本机没有冻结的任务镜像 ${frozen.image}：一份研究不能自己改依赖，不重建（docs/结构.md 第二部分"依赖跟着任务走"）` };
|
||||
}
|
||||
const prompts = readPrompts(frozen.config.promptDir);
|
||||
const now = promptHashes(prompts);
|
||||
const changed = [
|
||||
...ROLES.filter((r) => now.roles[r] !== frozen.prompts.roles[r]).map((r) => `roles/${r}.md`),
|
||||
...KINDS.filter((k) => now.tasks[k] !== frozen.prompts.tasks[k]).map((k) => `tasks/${k}.md`),
|
||||
];
|
||||
if (changed.length > 0 && !frozen.config.devEntry) {
|
||||
return { halt: `提示词和冻结时不一样：${changed.join("、")}；开发入口关着，不能换提示词（方法.md 第九部分）` };
|
||||
}
|
||||
// 任务事实只认冻结的那份；任务目录里的基线、harness、校验器和 TASK.md 照目录读。
|
||||
const task: Task = { ...loadTask(frozen.config.task), facts: frozen.facts };
|
||||
return { dir, ledger, runner, stop, frozen, task, prompts, facts: studyFacts(frozen, task) };
|
||||
}
|
||||
|
||||
const KINDS = Object.keys(SUBMISSION_KINDS) as SubmissionKind[];
|
||||
|
||||
function promptHashes(prompts: Prompts): StudyRecord["prompts"] {
|
||||
const pick = <K extends string>(files: Record<K, { hash: string }>) =>
|
||||
Object.fromEntries(Object.entries<{ hash: string }>(files).map(([k, f]) => [k, f.hash])) as Record<K, string>;
|
||||
return { roles: pick(prompts.roles), tasks: pick(prompts.tasks) };
|
||||
}
|
||||
|
||||
/** materials 要的研究事实，取自冻结的配置和任务。 */
|
||||
function studyFacts(frozen: Frozen, task: Task): StudyFacts {
|
||||
const { config, facts } = frozen;
|
||||
return {
|
||||
finalMetric: facts.finalMetric.id,
|
||||
direction: facts.finalMetric.direction,
|
||||
epsilon: facts.epsilon,
|
||||
knownBest: facts.knownBest,
|
||||
lambda: config.flow.openReserve,
|
||||
totalRuns: config.runs,
|
||||
proposalLimit: config.proposalLimit,
|
||||
factorLimit: config.factorLimit,
|
||||
metrics: metricDefinitions(task),
|
||||
taskText: task.guide,
|
||||
wallSeconds: facts.run.wallSeconds,
|
||||
};
|
||||
}
|
||||
|
||||
/** runStudy 的两种结局：停下（附原因），或被用户停下。"研究结束"第 4 部分才走得到。 */
|
||||
export type StudyEnding = { status: "halted"; reason: string } | { status: "userStopped" };
|
||||
export const USER_STOPPED: StudyEnding = { status: "userStopped" };
|
||||
|
||||
/** 停下：写一条停下事件，交回停下的结局。 */
|
||||
export function halt(ledger: Ledger, reason: string, objects: string[]): StudyEnding {
|
||||
ledger.append("event", { event: EVENTS.halted, reason, objects });
|
||||
return { status: "halted", reason };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { Ledger } from "../ledger/index.ts";
|
||||
import { adoptionLine, compareIds, runBalance } from "../rules/index.ts";
|
||||
import type { Checklist, SubmissionKind } from "../vocab/index.ts";
|
||||
import { beliefBookOf } from "./belief-book.ts";
|
||||
@@ -243,6 +244,11 @@ export function validEntries(board: Board): Array<{ id: string; author: string;
|
||||
});
|
||||
}
|
||||
|
||||
/** 按账本取的有效入口：flow 分配入口时用，和认领的对号清单是同一份（方法.md 4.2）。 */
|
||||
export function finalEntries(ledger: Ledger): Array<{ id: string; author: string; name: string; body: string }> {
|
||||
return validEntries(new Board(ledger));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// "你自己的"（方法.md 第三部分"给研究者的材料顺序"第 2 条）
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// materials：每个任务能看到哪些材料、第一条消息怎么写、查询在材料范围内回答什么（docs/结构.md 第二部分），
|
||||
// 以及账本里只存成 Json 的几样内容的形状和编号规则（用户 2026-10-03 决定 2）。
|
||||
export { beliefBook } from "./belief-book.ts";
|
||||
export { type MaterialRecord, type MaterialSection, type ReadableVersion, type TaskMaterial } from "./blocks.ts";
|
||||
export { finalEntries, type MaterialRecord, type MaterialSection, type ReadableVersion, type TaskMaterial } from "./blocks.ts";
|
||||
export { buildMaterial } from "./build.ts";
|
||||
export { type CodeChanges, type PromptFile, type Prompts, type StudyFacts, type TaskRequest } from "./inputs.ts";
|
||||
export { queryMaterial, readPrompts } from "./prompts.ts";
|
||||
@@ -21,6 +21,7 @@ export {
|
||||
freezeProtocol,
|
||||
ORIGINAL_BASELINE,
|
||||
proposalId,
|
||||
researcherId,
|
||||
splitCell,
|
||||
type DecisionKind,
|
||||
type DecisionShapes,
|
||||
@@ -28,4 +29,5 @@ export {
|
||||
type FrozenProtocol,
|
||||
type LocationTable,
|
||||
type Settlement,
|
||||
type StudyRecord,
|
||||
} from "./shapes.ts";
|
||||
|
||||
+22
-1
@@ -11,7 +11,7 @@ import type {
|
||||
settleAblation,
|
||||
} from "../rules/index.ts";
|
||||
import type { Content } from "../ledger/index.ts";
|
||||
import type { Submission } from "../vocab/index.ts";
|
||||
import type { Json, Role, Submission, SubmissionKind } from "../vocab/index.ts";
|
||||
|
||||
// 账本里只存成 Json 的几样内容长什么样、编号怎么起（用户 2026-10-03 决定 2）：materials 读它们，flow 照着写。
|
||||
// 决定和结算的输出直接用 rules 的返回类型；ledger 不认识这些类型，只存 Json。
|
||||
@@ -23,6 +23,8 @@ import type { Submission } from "../vocab/index.ts";
|
||||
/** 原始基线（任务给的代码）的对象编号：它的冻结代码记录、基线运行都挂在这个名字下。 */
|
||||
export const ORIGINAL_BASELINE = "base";
|
||||
|
||||
/** 研究者：第 i 位（从 0 数）是 researcher-(i+1)，与参考项目的写法相同，回放时不用改编号。 */
|
||||
export const researcherId = (i: number): string => `researcher-${i + 1}`;
|
||||
/** 入口：作者在第零轮交的第 n 个入口，讨论时整份替换的新清单接着往后编（方法.md 4.2）。 */
|
||||
export const entryId = (author: string, n: number): string => `e-${author}-${n}`;
|
||||
/** 提案：第 round 轮作者交的第 n 条，也是提交记录的 item。 */
|
||||
@@ -50,6 +52,25 @@ export const explanationId = (round: number, author: string, i: number): string
|
||||
export const callId = (stageId: string, actor: string, kind: string, object: string | null, attempt: number): string =>
|
||||
`${stageId}.${actor}.${kind}.${object ?? "none"}.${attempt}`;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 研究配置（账本记录 config，方法.md 4.1）
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* 研究配置记录：研究开始时 flow 一次写入，之后只认这一份，不再读配置文件和 task.json。
|
||||
* materials 不能引用 config、tasks，所以配置（checkConfig 过的）、步骤清单（config 的 stepList）、任务事实（task.json）
|
||||
* 存成 Json，flow 读回时用 checkConfig、checkTaskFacts 再检查。prompts 是提示词目录里每个文件原始字节的 sha256，
|
||||
* image 是任务镜像编号，seed 是这份研究全部运行共用的种子。
|
||||
*/
|
||||
export type StudyRecord = {
|
||||
config: Json;
|
||||
steps: Json;
|
||||
prompts: { roles: Record<Role, string>; tasks: Record<SubmissionKind, string> };
|
||||
image: string;
|
||||
task: Json;
|
||||
seed: number;
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 冻结协议（账本记录 protocol.protocol，方法.md 4.3 ⑥）
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { test, type TestContext } from "node:test";
|
||||
import { runStudy } from "../src/flow/index.ts";
|
||||
import { Ledger } from "../src/ledger/index.ts";
|
||||
import { Runner } from "../src/runner/index.ts";
|
||||
import { fullConfig } from "./config-helpers.ts";
|
||||
|
||||
// flow 碰到 Docker 的两处边界，用真实 Docker 测，不花模型的钱：走不到派模型任务就停下。
|
||||
// 镜像一律 inforsi-test- 开头的固定名字；本机还有别的项目在建删容器和镜像，只动自己前缀的。
|
||||
|
||||
const TASK_DIR = join(import.meta.dirname, "..", "tasks", "circle_packing");
|
||||
|
||||
function docker(args: string[]): { status: number | null; stdout: string; stderr: string } {
|
||||
return spawnSync("docker", args, { encoding: "utf8" });
|
||||
}
|
||||
|
||||
/** 删掉测试自己的镜像标签；不存在不算错，别的错误照样报。 */
|
||||
function removeImage(tag: string): void {
|
||||
const removed = docker(["rmi", tag]);
|
||||
if (removed.status !== 0 && !removed.stderr.includes("No such image")) throw new Error(removed.stderr);
|
||||
}
|
||||
|
||||
/** 一个测试用的研究编号：先清掉上次没删干净的同名镜像，测试结束后再删。 */
|
||||
function studyId(t: TestContext, suffix: string): string {
|
||||
const id = `test-flow-${suffix}`;
|
||||
removeImage(`inforsi-${id}`);
|
||||
t.after(() => removeImage(`inforsi-${id}`));
|
||||
return id;
|
||||
}
|
||||
|
||||
function tempDir(t: TestContext): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "inforsi-flow-"));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
return dir;
|
||||
}
|
||||
|
||||
function ledgerKinds(studyDir: string): string[] {
|
||||
const ledger = Ledger.open(join(studyDir, "ledger.sqlite"));
|
||||
try {
|
||||
return ledger.records().map((r) => r.kind);
|
||||
} finally {
|
||||
ledger.close();
|
||||
}
|
||||
}
|
||||
|
||||
test("研究编号本机用过（已有 inforsi-<研究编号> 镜像）：冻结配置前报错停下，不建镜像、不写研究配置记录", async (t) => {
|
||||
const id = studyId(t, "dup");
|
||||
// 一份旧研究冻结配置时建下的镜像。
|
||||
await new Runner(null).buildImage(TASK_DIR, `inforsi-${id}`);
|
||||
const dir = tempDir(t);
|
||||
await assert.rejects(
|
||||
runStudy({ studyDir: dir, config: { ...fullConfig(), studyId: id }, stop: new AbortController().signal }),
|
||||
new RegExp(`研究编号 ${id} 已被本机用过（已有镜像 inforsi-${id}）`),
|
||||
);
|
||||
assert.deepEqual(ledgerKinds(dir), []);
|
||||
});
|
||||
|
||||
test("docker run 没启动：写结果\"没启动\"后停下，原因带 Docker 报错原文，不在本进程里起新编号重跑", { timeout: 120_000 }, async (t) => {
|
||||
// 任务事实里写一个本机没有的 Docker 运行时，docker run 每次都失败，也不建容器。
|
||||
const task = join(tempDir(t), "task");
|
||||
cpSync(TASK_DIR, task, { recursive: true });
|
||||
const facts = JSON.parse(readFileSync(join(task, "task.json"), "utf8")) as { run: { runtime: string } };
|
||||
facts.run.runtime = "inforsi-test-no-such-runtime";
|
||||
writeFileSync(join(task, "task.json"), JSON.stringify(facts));
|
||||
const dir = tempDir(t);
|
||||
const ending = await runStudy({
|
||||
studyDir: dir,
|
||||
config: { ...fullConfig(), studyId: studyId(t, "not-started"), task },
|
||||
stop: new AbortController().signal,
|
||||
});
|
||||
assert.equal(ending.status, "halted");
|
||||
assert.match((ending as { reason: string }).reason, /运行 base-1 没有启动.*unknown or invalid runtime name: inforsi-test-no-such-runtime/);
|
||||
assert.deepEqual(ledgerKinds(dir), ["config", "code", "runPlanned", "runResult", "event"]);
|
||||
});
|
||||
@@ -0,0 +1,205 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { test } from "node:test";
|
||||
import { stepList, type ResearchConfig } from "../src/config/index.ts";
|
||||
import type { AppendKind, Content, LedgerRecord } from "../src/ledger/index.ts";
|
||||
import type { StudyRecord } from "../src/materials/index.ts";
|
||||
import { nextStep } from "../src/flow/next-step.ts";
|
||||
import type { Json } from "../src/vocab/index.ts";
|
||||
import { fullConfig } from "./config-helpers.ts";
|
||||
|
||||
// flow 的"下一步做什么"（任务书 12 第六部分）：纯函数，手写账本记录，看它给出的下一步。
|
||||
// 其余都由真实运行证明（任务书 12 第五部分），这里不写假模型、假 Docker。
|
||||
|
||||
const TASK_FACTS = JSON.parse(readFileSync(join(import.meta.dirname, "..", "tasks", "circle_packing", "task.json"), "utf8")) as Json;
|
||||
const HASH = "0".repeat(64);
|
||||
const measured = (value: number) => ({ status: "measured", value }) as const;
|
||||
|
||||
/** 调用结束记录去掉调用编号（四种结局各自去掉）。 */
|
||||
type Ended = Content<"callEnded"> extends infer E ? (E extends unknown ? Omit<E, "call"> : never) : never;
|
||||
|
||||
/** 一本手写的账本：按写入先后编序号。内容照账本的形状写，nextStep 不再检查。 */
|
||||
class Book {
|
||||
readonly records: LedgerRecord[] = [];
|
||||
|
||||
add<K extends AppendKind>(kind: K, content: Content<K>): this {
|
||||
this.records.push({ seq: this.records.length + 1, kind, hash: HASH, content } as LedgerRecord);
|
||||
return this;
|
||||
}
|
||||
|
||||
config(config: ResearchConfig): this {
|
||||
const record: StudyRecord = {
|
||||
config: config as unknown as Json,
|
||||
steps: stepList(config) as unknown as Json,
|
||||
prompts: { roles: {} as StudyRecord["prompts"]["roles"], tasks: {} as StudyRecord["prompts"]["tasks"] },
|
||||
image: `sha256:${HASH}`,
|
||||
task: TASK_FACTS,
|
||||
seed: 1,
|
||||
};
|
||||
return this.add("config", record as unknown as Json);
|
||||
}
|
||||
|
||||
baselineCode(): this {
|
||||
return this.add("code", { object: "base", version: "v-base", locations: null });
|
||||
}
|
||||
|
||||
/** 一次基线运行：没启动，或启动了、判分算数或不算数。 */
|
||||
baselineRun(run: string, outcome: "notStarted" | "valid" | "invalid"): this {
|
||||
const definition: Content<"runPlanned">["definition"] = {
|
||||
command: ["python", "harness.py"],
|
||||
workdir: "/work",
|
||||
image: { digest: `sha256:${HASH}`, runtime: "runc" },
|
||||
wallSeconds: 120,
|
||||
cores: 1,
|
||||
inheritEnv: false,
|
||||
env: {},
|
||||
};
|
||||
this.add("runPlanned", { run, purpose: "baseline", round: 0, object: null, definition });
|
||||
if (outcome === "notStarted") {
|
||||
return this.add("runResult", { run, process: { ending: "notStarted", reason: "docker run 失败" }, judged: null });
|
||||
}
|
||||
this.add("runStarted", { run });
|
||||
const process = { ending: "exited", exitCode: 0, cpuSeconds: measured(10), wallSeconds: measured(10), peakMemoryBytes: measured(1) } as const;
|
||||
const judged = {
|
||||
verdict: outcome,
|
||||
verdictReason: outcome === "valid" ? "合法" : "不合法：圆重叠",
|
||||
score: measured(0.9597642169962064),
|
||||
metrics: {},
|
||||
answerPath: { status: "unmeasured", reason: "基线不写" },
|
||||
safeguardTriggers: { status: "unmeasured", reason: "基线不写" },
|
||||
extra: null,
|
||||
} as unknown as Content<"runResult">["judged"];
|
||||
return this.add("runResult", { run, process, judged });
|
||||
}
|
||||
|
||||
open(stageId: string, stage: Content<"stageOpen">["stage"], roster: string[]): this {
|
||||
return this.add("stageOpen", { stageId, stage, round: 0, roster });
|
||||
}
|
||||
|
||||
/** 一份被收下的提交。nextStep 只看这个人在这个阶段有没有被收下的提交，种类一律写成入口。 */
|
||||
entry(stageId: string, author: string): this {
|
||||
const content = { entries: [{ name: "入口一", body: "从这里开始看" }, { name: "入口二", body: "从那里开始看" }] };
|
||||
return this.add("submission", { stageId, author, target: null, item: null, version: 1, kind: "entry", content, jev: null });
|
||||
}
|
||||
|
||||
/** 一次模型调用：开始和结束。 */
|
||||
call(stageId: string, actor: string, n: number, ended: Ended): this {
|
||||
const call = `${stageId}.${actor}.entry.none.${n}`;
|
||||
this.add("callStarted", { call, actor, stageId, model: "iomgaa/glm-5.3", materials: null });
|
||||
return this.add("callEnded", { call, ...ended } as Content<"callEnded">);
|
||||
}
|
||||
}
|
||||
|
||||
const R = ["researcher-1", "researcher-2", "researcher-3"];
|
||||
|
||||
/** 冻结了配置、基线跑完算数的账本。 */
|
||||
function afterBaseline(config: ResearchConfig = fullConfig()): Book {
|
||||
return new Book().config(config).baselineCode().baselineRun("base-1", "valid");
|
||||
}
|
||||
|
||||
test("空账本：冻结配置", () => {
|
||||
assert.deepEqual(nextStep([]), { do: "freezeConfig" });
|
||||
});
|
||||
|
||||
test("有配置、没有基线：先存基线代码，再用 base-1 跑基线", () => {
|
||||
const book = new Book().config(fullConfig());
|
||||
assert.deepEqual(nextStep(book.records), { do: "saveBaseline" });
|
||||
book.baselineCode();
|
||||
assert.deepEqual(nextStep(book.records), { do: "baselineRun", run: "base-1" });
|
||||
});
|
||||
|
||||
test("基线最近一次是\"没启动\"：起新编号再跑", () => {
|
||||
const book = new Book().config(fullConfig()).baselineCode().baselineRun("base-1", "notStarted");
|
||||
assert.deepEqual(nextStep(book.records), { do: "baselineRun", run: "base-2" });
|
||||
book.baselineRun("base-2", "notStarted");
|
||||
assert.deepEqual(nextStep(book.records), { do: "baselineRun", run: "base-3" });
|
||||
});
|
||||
|
||||
test("基线判为不算数：停下，原因写明", () => {
|
||||
const book = new Book().config(fullConfig()).baselineCode().baselineRun("base-1", "invalid");
|
||||
const next = nextStep(book.records);
|
||||
assert.equal(next.do, "halt");
|
||||
assert.match((next as { reason: string }).reason, /base-1 判为不算数（不合法：圆重叠）/);
|
||||
});
|
||||
|
||||
test("基线算数之后：开写入口阶段，名单是全部研究者", () => {
|
||||
assert.deepEqual(nextStep(afterBaseline().records), { do: "openStage", stageId: "r0-entryDraft", stage: "entryDraft", roster: R });
|
||||
});
|
||||
|
||||
test("阶段开着、一人已交、两人没交：派那两人", () => {
|
||||
const book = afterBaseline().open("r0-entryDraft", "entryDraft", R).entry("r0-entryDraft", "researcher-1");
|
||||
assert.deepEqual(nextStep(book.records), { do: "dispatch", stageId: "r0-entryDraft", kind: "entry", actors: ["researcher-2", "researcher-3"] });
|
||||
});
|
||||
|
||||
test("都交了：关阶段；关了以后开第 1 次讨论", () => {
|
||||
const book = afterBaseline().open("r0-entryDraft", "entryDraft", R);
|
||||
for (const r of R) book.entry("r0-entryDraft", r);
|
||||
assert.deepEqual(nextStep(book.records), { do: "closeStage", stageId: "r0-entryDraft" });
|
||||
book.add("stageClose", { stageId: "r0-entryDraft" });
|
||||
assert.deepEqual(nextStep(book.records), { do: "openStage", stageId: "r0-discussion1", stage: "discussion", roster: R });
|
||||
});
|
||||
|
||||
test("有人最近一次是\"没交\"：开发入口关着先派能派的、再停下；开着重新派", () => {
|
||||
const stuck = (devEntry: boolean) =>
|
||||
new Book()
|
||||
.config({ ...fullConfig(), devEntry })
|
||||
.baselineCode()
|
||||
.baselineRun("base-1", "valid")
|
||||
.open("r0-entryDraft", "entryDraft", R)
|
||||
.entry("r0-entryDraft", "researcher-1")
|
||||
.call("r0-entryDraft", "researcher-2", 1, { wallSeconds: measured(30), status: "notSubmitted", reason: "模型结束了回复，没有一份提交被收下" });
|
||||
// researcher-3 还没派过，先派他。
|
||||
assert.deepEqual(nextStep(stuck(false).records), { do: "dispatch", stageId: "r0-entryDraft", kind: "entry", actors: ["researcher-3"] });
|
||||
// researcher-3 交上以后只剩"没交"的 researcher-2：停下，附调用编号。
|
||||
const closed = stuck(false).entry("r0-entryDraft", "researcher-3");
|
||||
const next = nextStep(closed.records);
|
||||
assert.equal(next.do, "halt");
|
||||
assert.match((next as { reason: string }).reason, /^交不出合格提交/);
|
||||
assert.deepEqual((next as { objects: string[] }).objects, ["r0-entryDraft.researcher-2.entry.none.1"]);
|
||||
// 开发入口开着：重新派。
|
||||
const dev = stuck(true).entry("r0-entryDraft", "researcher-3");
|
||||
assert.deepEqual(nextStep(dev.records), { do: "dispatch", stageId: "r0-entryDraft", kind: "entry", actors: ["researcher-2"] });
|
||||
});
|
||||
|
||||
test("有人最近一次是\"出错\"或\"进程中断\"：重新派；只看最近一次调用", () => {
|
||||
const book = afterBaseline()
|
||||
.open("r0-entryDraft", "entryDraft", R)
|
||||
.entry("r0-entryDraft", "researcher-1")
|
||||
.call("r0-entryDraft", "researcher-2", 1, { wallSeconds: measured(5), status: "notSubmitted", reason: "没交" })
|
||||
.call("r0-entryDraft", "researcher-2", 2, { wallSeconds: measured(5), status: "failed", reason: "模型服务报错" })
|
||||
.call("r0-entryDraft", "researcher-3", 1, { wallSeconds: { status: "unmeasured", reason: "进程中断" }, status: "failed", reason: "进程中断" });
|
||||
assert.deepEqual(nextStep(book.records), { do: "dispatch", stageId: "r0-entryDraft", kind: "entry", actors: ["researcher-2", "researcher-3"] });
|
||||
});
|
||||
|
||||
test("只有一位研究者：两次讨论各写一条按配置跳过，接着开认领", () => {
|
||||
const one = fullConfig();
|
||||
one.models.researchers = [one.models.researchers[0]!];
|
||||
const book = afterBaseline(one).open("r0-entryDraft", "entryDraft", ["researcher-1"]).entry("r0-entryDraft", "researcher-1");
|
||||
book.add("stageClose", { stageId: "r0-entryDraft" });
|
||||
const reason = "只有一位研究者，第零轮不讨论（方法.md 4.2）";
|
||||
assert.deepEqual(nextStep(book.records), { do: "skip", step: "discussion1", reason });
|
||||
book.add("event", { event: "skippedByConfig", reason, objects: ["discussion1"] });
|
||||
assert.deepEqual(nextStep(book.records), { do: "skip", step: "discussion2", reason });
|
||||
book.add("event", { event: "skippedByConfig", reason, objects: ["discussion2"] });
|
||||
assert.deepEqual(nextStep(book.records), { do: "openStage", stageId: "r0-claim", stage: "claim", roster: ["researcher-1"] });
|
||||
});
|
||||
|
||||
test("认领关了：分配；分配做完：停下，原因\"第 1 轮还没写\"", () => {
|
||||
const book = afterBaseline();
|
||||
const stages = [
|
||||
["r0-entryDraft", "entryDraft"],
|
||||
["r0-discussion1", "discussion"],
|
||||
["r0-discussion2", "discussion"],
|
||||
["r0-claim", "claim"],
|
||||
] as const;
|
||||
for (const [stageId, stage] of stages) {
|
||||
book.open(stageId, stage, R);
|
||||
for (const r of R) book.entry(stageId, r);
|
||||
book.add("stageClose", { stageId });
|
||||
}
|
||||
assert.deepEqual(nextStep(book.records), { do: "assignEntries" });
|
||||
const output = R.map((researcher, i) => ({ researcher, entry: `e-${researcher}-1`, rank: i + 1 }));
|
||||
book.add("decision", { decision: "entries", round: 0, rule: "assignEntries", ruleVersion: 1, params: null, seed: null, input: null, output });
|
||||
assert.deepEqual(nextStep(book.records), { do: "halt", reason: "第 1 轮还没写（任务书 13）", objects: [] });
|
||||
});
|
||||
+1
-1
@@ -14,5 +14,5 @@
|
||||
"verbatimModuleSyntax": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src/**/*.ts", "tests/**/*.ts"]
|
||||
"include": ["src/**/*.ts", "tests/**/*.ts", "eval/**/*.ts"]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user