feat: 任务书 08，jev 模块与账本记 Jev 判断
src/jev 3 个文件 515 行：判重、查冻结条款、查代码改动三个函数，九问问法和阈值照参考项目，只改说法和字段名，阈值注明未在新项目材料上校准，等真实数据。 新增 npm run test:jev（真实 Jev 三处各问一次）。vocab 加 JevJudgment。 ledger 表结构版本 3：callEnded 四种结局对上 agents，墙钟时间用 Measured（恢复时补写出错、原因"进程中断"、测不到）；退回、提交记录都带 jev。 config 加必填的 models.jev。 用户原话："这次就给提交记录加一个和退回记录同样的 Jev 判断字段"；"墙钟时间改用 Measured，恢复时写出错加测不到（原因'进程中断'）"。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+3
-2
@@ -18,6 +18,7 @@
|
||||
- 任务书 05（tasks 与 runner）：src/runner 2 个文件、271 行，src/tasks 6 个文件、691 行，`npm test` 274 条通过（runner 12 条、tasks 16 条，真实 Docker，整套约 12 秒）。任务目录搬到 `tasks/circle_packing/`，新增 task.json；`VERDICTS` 删掉"判不了"。基线从头到尾判为算数，分数 0.9597642169962064，两次逐位相同。全新上下文审阅两轮：第一轮发现候选能伪造计量结果（把超时记成正常退出）、运行目录里旧 metrics.json 会被当成本次产物，都已修（计量 shell 以容器内 root 跑、命令用 setpriv 降权、结果用 docker cp 取；harness 开跑前删旧输出）；第二轮只审这次修复，另采纳一条：容器加 no-new-privileges，禁止候选靠 setuid 程序重新拿到 root。结构.md 第八部分补了 runner、tasks。
|
||||
- 任务书 06（workspace）：src/workspace 2 个文件、205 行，`npm test` 284 条通过（workspace 10 条，真实 git）。版本号是目录树编号，存版本自己列文件加 hash-object，取出用 ls-tree 加 cat-file（实测 `git archive` 会按树里的 `.gitattributes` 丢文件、改换行符），git 子进程另关掉系统和用户的 attributes 文件（实测只关配置时 `~/.config/git/attributes` 仍会被读）。tasks 的 task.json 加必填的 `unversionedNames`，`protectedFiles` 删掉 solution.json。全新上下文审阅一轮：两处偏离都实测成立；采纳一条（写明空文件增删、只改可执行位没有块），其余不改。结构.md 第八部分补了 workspace。
|
||||
- 任务书 07（agents）：src/agents 4 个文件、681 行（预期按用户决定改为 600–700，超出部分是任务书第四部分要保留的三处提交处理、读代码分段和会话核对，没有新增专门路径），`npm test` 293 条通过（agents 9 条，其中 1 条经真实 Pi 会话连本机假模型服务）。package.json 固定 `@earendil-works/pi-coding-agent`、`pi-ai` 0.85.1 并加 overrides（同参考项目）；新增 `npm run test:pi`（真实 glm-5.3，研究者先退回再收下、实验员写文件跑 shell 再交）。实测：glm-5.3 经 iomgaa 服务时，正文里没转义的半角双引号会在服务那一侧把参数截成合法但半截的 JSON，Pi 收到的原文本身合法，JSON 检查查不出（抓包确认）；提交工具说明里加了一句"引号用「」"，之后 12 份提交没再出现。结构.md 第七、八部分已更新。
|
||||
- 任务书 08（jev）：src/jev 3 个文件、515 行，九问的问法和阈值照参考项目，只改说法和字段名，阈值注明未在新材料上校准；判重一对一次请求、冻结条款三问各一次、代码改动一块一次（五问同材料）；`npm run test:jev` 三处各问一次真实 Jev，九问都拿回 0 到 1 的概率和请求编号。vocab 加 `JevJudgment`。ledger 表结构版本升到 3：`callEnded` 四种结局对上 agents、墙钟时间用 `Measured`（中断恢复时补写出错、原因"进程中断"、时间测不到）；退回和提交记录都带 `jev`（判断数组或 null）。config 加必填的 `models.jev`。`npm test` 316 条通过。账本改动全新上下文审阅两轮。
|
||||
|
||||
## 关键决定（用户原话见各提交说明）
|
||||
|
||||
@@ -44,6 +45,7 @@
|
||||
- 实验员会话里的 shell 命令不测 CPU，只记每次模型调用的墙钟时间；正式运行的资源用量照记。用户原话："那要不就不测了？本来cpu就不重要，现在又可能测不准"。
|
||||
- agents（任务书 07）：只收拼好的提示词和第一条消息（读提示词归 materials），不碰账本；提交工具去掉 slot；保留参考项目两处有真实证据的提交特殊处理（非法 JSON 被截半、null 变 "null"）和子会话加载核对；删 Jev 审命令、上下文估算、输出上限配置；预期 400–600 行。用户原话："同意，写任务书 07 吧"。
|
||||
- jev（任务书 08）：方法里三处 Jev 判别共九问（判重 1、冻结条款 3、实现改动 5），比原方法多出的"条款太含糊""算指标方式与定义不符""比登记的停止条件更早结束"三问已补进方法.md；这次不校准：问法以参考项目为基础只改说法，阈值暂用参考项目的值并注明未校准；账本存下每次 Jev 判断（退回、收下、判重都存）；Jev 模型名进 config；不做"永远判不同"的空实现。用户原话："九个都问，并把多出的三个补进 docs/方法.md"；后来改为不校准："我们到时候应该是通过真实的运行收集一批真实的数据，然后由claude或者gpt亲自打标签作为集合……jev这次的验收应该也不以效果为主，因为还没有数据校准，prompt还是用原本的prompt作为基础就好"。
|
||||
- 中断的模型调用，恢复时补写"出错、原因'进程中断'、墙钟时间测不到"（`callEnded` 的墙钟时间用 `Measured`）。用户原话："墙钟时间改用 Measured，恢复时写出错加测不到（原因'进程中断'）"。
|
||||
|
||||
## 未完成
|
||||
|
||||
@@ -70,7 +72,6 @@
|
||||
- 给 flow：同一轮里选批和选格都用随机规则时，传不同的"版本"算种子，否则两次抽取用同一串随机数。
|
||||
- 给 flow：确认启动前要先登记运行；模型调用要先开始再结束；阶段要先开再关；ledger 都不查。
|
||||
- 给 materials：别人的"退回"记录不放进任何研究者的材料（退回的提案原样带着作者自评，不另拆封存）。
|
||||
- ledger 的 `callEnded` 加墙钟时间字段（表结构版本号升级，改完派全新上下文审阅），并入下一份任务书做；同时让 `callEnded` 的结束方式对上 agents 的四种结局（交了、没交、出错、被停下），现在只有 completed / failed 两种。结构.md 第三部分第 10 种和任务书 07 都要求 flow 把墙钟时间写进"模型调用结束"。用户原话："问题 1 按 (a)，但并入下一份任务书做"。
|
||||
- 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。
|
||||
- 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。
|
||||
- 给 submissions：研究者登记的假设专用指标编号不能和常规辅助指标重名（`metricDefinitions` 里的 7 个），否则候选自报的值会顶替宿主测的值（任务书 05 审阅意见）。
|
||||
@@ -91,4 +92,4 @@
|
||||
|
||||
## 下一步
|
||||
|
||||
任务书 08（jev，`任务书/08-jev.md`）已写好，等用户另开新 Agent 执行；完成后监督者核对。
|
||||
任务书 08 已完成，等监督者核对。之后：用真实研究的数据校准 Jev 九问。
|
||||
|
||||
+21
-6
@@ -123,17 +123,17 @@
|
||||
|---|---|---|---|
|
||||
| 1 | 研究配置 | 冻结的配置、步骤清单、每个决策点用哪条规则、方法参数、提示词哈希 | flow（内容由 config 生成） |
|
||||
| 2 | 阶段 | 每个阶段开一条、关一条；关闭即公开 | flow |
|
||||
| 3 | 提交 | 各种提交的每一版：入口、讨论、志愿、提案、两次预测、挑刺、回应、解释、收尾、实现、核验：阶段关闭后以最后一版为准，此后由 ledger 拒绝再写。私人笔记的每一条：不属于任何阶段，以最后写入的一条为准。被退回的提交也记，附退回理由（含 Jev 的原始概率和结论） | flow（经 submissions 判定后） |
|
||||
| 3 | 提交 | 各种提交的每一版：入口、讨论、志愿、提案、两次预测、挑刺、回应、解释、收尾、实现、核验：阶段关闭后以最后一版为准，此后由 ledger 拒绝再写。私人笔记的每一条：不属于任何阶段，以最后写入的一条为准。每一版带收下时用到的 Jev 判断（含原始概率和结论，没用到 Jev 的为空）。被退回的提交也记，附退回理由和用到的 Jev 判断 | flow（经 submissions 判定后） |
|
||||
| 4 | 冻结协议 | 每个候选的冻结版本：因子清单、保障清单、实现条款、指标登记 | flow |
|
||||
| 5 | 作者自评 | 封存时的三个数；解封时追加一条解封记录 | flow |
|
||||
| 6 | 冻结代码 | 每份候选代码和消融格代码的版本号，以及因子位置表 | flow（版本号由 workspace 给出） |
|
||||
| 7 | 运行 | 每次正式运行三条：登记要跑（占名额）；确认启动（扣次）；结果（分数、合法化、指标、答案路径、资源） | flow（结果由 runner 与 tasks 给出） |
|
||||
| 8 | 决定 | 程序做的每个决定：选批、采纳、选格、押的因子、入口分配、判重合并。每条存规则名、规则版本、方法参数、种子、全部输入、输出；用到 Jev 的（判重合并），输入里带 Jev 的原始概率 | flow（由 rules 计算） |
|
||||
| 9 | 结算 | 每格的结局、撑几成、配合关系标签、每条解释的确认或否定、世界比特、惊讶 | flow（由 rules 计算） |
|
||||
| 10 | 用量 | 模型调用三条：开始（附这次任务的全部材料）、每次回复的用量、结束（带这次调用的墙钟时间）。正式运行的资源用量记在运行结果里。只记录 | flow（用量由 agents 交出） |
|
||||
| 10 | 用量 | 模型调用三条：开始（附这次任务的全部材料）、每次回复的用量、结束（带这次调用的墙钟时间；中断后恢复时补写的结束记录，结局记出错、原因写"进程中断"，墙钟时间记测不到）。正式运行的资源用量记在运行结果里。只记录 | flow（用量由 agents 交出） |
|
||||
| 11 | 事件 | 违规记录、"按配置跳过"、"没测"及原因、停下的原因 | flow |
|
||||
|
||||
Jev 是一个判别器，和程序检查地位相同，不单独成一种记录：它的原始概率和结论，记在它判断的那件事里（第 3 种的退回理由、第 8 种的判重合并）。
|
||||
Jev 是一个判别器，和程序检查地位相同，不单独成一种记录：它的原始概率和结论，记在它判断的那件事里（第 3 种的提交和退回、第 8 种的判重合并）。
|
||||
|
||||
**写入时由 ledger 拒绝的几种情况**：阶段已关闭还要写提交；要替换的那一份不是本人在这个阶段刚交的；没有"确认启动"就写运行结果（"确认没启动"这种结果除外）；这次消融已经有格启动了才冻结解释。
|
||||
|
||||
@@ -250,6 +250,10 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- `SUBMISSION_SCHEMAS`：每种提交的字段定义，键与 `SUBMISSION_KINDS` 相同；同一份定义是类型、收件格式检查和提交工具参数。
|
||||
- `Submission<K>`（类型）：某种提交的内容类型，例如 `Submission<"proposal">`。
|
||||
|
||||
**Jev**
|
||||
|
||||
- `JevJudgment`（字段定义）：Jev 对一个问题的一次判断：问题名、问题全文的哈希（措辞、示例、阈值一起算）、阈值、原始概率、结论（概率是否超过阈值）、Jev 返回的请求编号。jev 交出它，ledger 在提交和退回记录里原样存它，两个模块互不引用，所以放在这里。
|
||||
|
||||
### rules
|
||||
|
||||
下面是 `src/rules/index.ts` 导出的全部内容。rules 是纯计算：输入数字，输出数字，不碰文件、数据库和模型，也不用全局随机数。λ、ε、已知最好、每批上限、余额这些参数都由调用方传进来，rules 里只写死方法.md 6.11 标"固定"的几个常数。三个数一律用 vocab 的 `Range`。编号（候选、研究者、入口、因子、解释）一律按自然序比较：数字部分按数值大小比，所以 c2 排在 c10 前面。
|
||||
@@ -407,13 +411,13 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
|
||||
**类型**
|
||||
|
||||
- `Kind`：记录的种类，共 19 种：`config`（研究配置）、`stageOpen` / `stageClose`（阶段开、关）、`submission`（提交的每一版）、`rejection`（退回，带原样内容和理由，理由来自 Jev 的带原始概率和结论）、`note`（私人笔记，每次修改追加一条，以最后写入的一条为准，不带版本号）、`protocol`（冻结协议）、`seal` / `unseal`（作者自评的封存、解封，都用记录序号指明是哪一条）、`code`（冻结代码）、`runPlanned` / `runStarted` / `runResult`（运行的登记要跑、确认启动、结果）、`decision`（决定）、`settlement`（一轮的消融结算，按轮次唯一）、`callStarted` / `callUsage` / `callEnded`（模型调用的开始，附这次任务的全部材料；每次回复的用量；结束）、`event`（事件）。
|
||||
- `Kind`：记录的种类，共 19 种：`config`（研究配置）、`stageOpen` / `stageClose`（阶段开、关）、`submission`（提交的每一版，外层带收下时用到的 `JevJudgment` 数组，没用到 Jev 的为 null）、`rejection`（退回，带原样内容和理由；Jev 退回的，原样存这次用到的一条或几条 `JevJudgment`，程序检查退回的这一项写 null）、`note`（私人笔记，每次修改追加一条，以最后写入的一条为准，不带版本号）、`protocol`（冻结协议）、`seal` / `unseal`（作者自评的封存、解封，都用记录序号指明是哪一条）、`code`（冻结代码）、`runPlanned` / `runStarted` / `runResult`（运行的登记要跑、确认启动、结果）、`decision`（决定）、`settlement`（一轮的消融结算，按轮次唯一）、`callStarted` / `callUsage` / `callEnded`（模型调用的开始，附这次任务的全部材料；每次回复的用量；结束，带墙钟时间（秒，`Measured`：中断后恢复时补写的结束记录，结局记出错、原因写"进程中断"，墙钟时间记测不到），结局与 agents 的会话结局同名：交了、没交（带原因）、出错（带原因）、被停下）、`event`（事件）。
|
||||
- `AppendKind`：调用方能追加的种类，除了 `seal`（封存只由 ledger 在写提案时一并写）。
|
||||
- `Content<K>`：某种记录存下的内容。
|
||||
- `Input<K>`：追加时交的内容；只有提交和存下的不同：提案带着作者自评。
|
||||
- `LedgerRecord<K>`：一条记录：序号、种类、内容的哈希、内容。
|
||||
- `SubmissionSlot`（类型）：一份提交不论第几版：阶段、作者、提交种类（版本号只在同一种提交里往下接）、针对谁（候选或格，没有就空）、这一份的编号（只有提案、挑刺这两种一人能交好几份的才有，其余必须为空）。
|
||||
- `StoredSubmission`、`SubmissionInput`（类型）：存下的提交和追加时交的提交，按提交种类区分内容类型；版本号从 1 起，替换时加 1，被替换的就是上一版。
|
||||
- `StoredSubmission`、`SubmissionInput`（类型）：存下的提交和追加时交的提交，按提交种类区分内容类型；版本号从 1 起，替换时加 1，被替换的就是上一版；`jev` 和 `content` 并列，存这一版收下时用到的 Jev 判断，不进 `content`。
|
||||
|
||||
### config
|
||||
|
||||
@@ -426,7 +430,7 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
|
||||
**字段定义**
|
||||
|
||||
- `ResearchConfig`（字段定义）：research.json 的全部字段，全部必填，不认识的字段报错。每个字段带中文说明（`description`）和类别（`category`：用户配置或方法参数），配置对话照着它问，不另写一份。流程是两选一：完整流程写四个开关、选批规则、它才用得到的 λ、u、衰减；朴素循环只写 `preset`，别的都写不进去。
|
||||
- `ResearchConfig`（字段定义）：research.json 的全部字段，全部必填，不认识的字段报错。每个字段带中文说明（`description`）和类别（`category`：用户配置或方法参数），配置对话照着它问，不另写一份。`models` 里除了各角色的模型，还有 Jev 用的模型名（`models.jev`，用户配置）；Jev 的密钥不进配置，从环境变量读。流程是两选一：完整流程写四个开关、选批规则、它才用得到的 λ、u、衰减；朴素循环只写 `preset`，别的都写不进去。
|
||||
- `FIELD_CATEGORIES`、`FieldCategory`：字段的两种类别，用户配置、方法参数。
|
||||
- `THINKING_LEVELS`、`ThinkingLevel`：Pi 0.85.1 接受的七个思考档位，出处写在代码注释里；config 不能引用 Pi，所以照抄一份。
|
||||
|
||||
@@ -505,3 +509,14 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- `SubmitAttempt<K>`（类型）：交给提交回调的一次提交：形状合格的内容和"替换哪一份"（只有研究者的提交能替换），或形状不合格的原样参数和原因（参数原文不是合法 JSON、必填字段交了 null、Pi 的形状校验不通过），后者回调只需记一条退回。
|
||||
- `SubmitReceipt`（类型）：提交回调的回答，收下和编号，或退回和原因，agents 原样回给模型；收下时另回显收到的字数和结尾。
|
||||
- `SessionEnding`、`SessionOutcome`（类型）：交了、没交（附原因，例如最后一条回复被输出上限截断）、出错（附原因）、被停下四种，都带墙钟时间（秒）和会话文件路径（Pi 没写出会话文件时为 null）。
|
||||
|
||||
### jev
|
||||
|
||||
下面是 `src/jev/index.ts` 导出的全部内容。九问的问法和阈值照参考项目，只改了说法和字段名；阈值没有在新项目的材料上校准过，等真实研究的数据（任务书 08 第三部分）。模型名由调用方从 config 的 `models.jev` 传入，密钥从环境变量 `TYPESAFE_API_KEY` 读进请求头。调不通时在发请求那一处有限次重试，仍不通就报错；回答读不懂报错并带请求编号。
|
||||
|
||||
- `judgeSameChange(model, a, b)`：③ 判重，两条提案的因子清单是不是同一个改法，一对一次请求，交回一条 `JevJudgment`。
|
||||
- `checkProtocol(model, protocol)`：⑥ 查冻结条款，三问（改动了评测、自相矛盾、太含糊）各一次请求，交回 `JevCheck`。
|
||||
- `checkHunk(model, hunk, protocol)`：⑨ 查一段代码改动，五问（不固定种子的随机数、没登记就读时钟、算登记指标的方式与定义不符、比登记的停止条件更早结束、测不到的量写成 0）同一份材料一次请求，交回 `JevCheck`。
|
||||
- `codeHunks(comparison)`：把 workspace 的比较结果（文件清单和差异原文）切成差异块；没有 `@@` 行的改动（二进制、只改可执行位、空文件增删）不出块，由程序检查。
|
||||
- `DedupFactor`、`FrozenProtocol`、`ProtocolMetric`、`ProtocolRules`、`CodeHunk`（类型）：三个函数收的带类型输入，取自 vocab 的字段定义；材料由 jev 自己拼。
|
||||
- `JevCheck`（类型）：每一问的判断，以及有没有任何一问判"是"。
|
||||
|
||||
+2
-1
@@ -9,7 +9,8 @@
|
||||
"scripts": {
|
||||
"check": "tsc --noEmit -p tsconfig.json && depcruise src tests --config .dependency-cruiser.cjs",
|
||||
"test": "node --test \"tests/**/*.test.ts\"",
|
||||
"test:pi": "node tests/agents-pi.ts"
|
||||
"test:pi": "node tests/agents-pi.ts",
|
||||
"test:jev": "node tests/jev-real.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-ai": "0.85.1",
|
||||
|
||||
@@ -138,10 +138,15 @@ export const ResearchConfig = Type.Object(
|
||||
),
|
||||
experimenter: user(Model, "实验员用的模型"),
|
||||
verifier: user(Model, "核验员用的模型"),
|
||||
jev: user(
|
||||
Type.String(TEXT),
|
||||
"Jev 判别器用的模型名，原样放进发给 Jev 服务的请求（参考项目用的是 jev-latest）。" +
|
||||
"Jev 不经过 Pi，所以只写模型名；密钥不写在配置里，从环境变量 TYPESAFE_API_KEY 读",
|
||||
),
|
||||
},
|
||||
CLOSED,
|
||||
),
|
||||
"各角色用的模型",
|
||||
"各角色用的模型，以及判重、查冻结条款、查代码改动用的 Jev 模型",
|
||||
),
|
||||
proposalLimit: user(PositiveInteger, "每人每轮提案上限 M（方法.md 4.3 ②）"),
|
||||
batchLimit: user(PositiveInteger, "每批上限：一轮正向最多跑几个候选，本轮批量 = min(每批上限, 余额)（方法.md 第七部分）"),
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
// jev：问 Jev 的问法和阈值（docs/结构.md 第二部分）。判断的类型 JevJudgment 在 vocab。
|
||||
export {
|
||||
checkHunk,
|
||||
checkProtocol,
|
||||
codeHunks,
|
||||
judgeSameChange,
|
||||
type CodeHunk,
|
||||
type DedupFactor,
|
||||
type FrozenProtocol,
|
||||
type JevCheck,
|
||||
type ProtocolMetric,
|
||||
type ProtocolRules,
|
||||
} from "./jev.ts";
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
import type { HypothesisMetric, JevJudgment, Json, Submission } from "../vocab/index.ts";
|
||||
import { questionsOf, taskOf, type Question } from "./questions.ts";
|
||||
|
||||
// 问 Jev：从带类型的输入拼材料、发请求、读回答（docs/方法.md 4.3 ③⑥⑨）。
|
||||
// 材料里放什么、字段叫什么只在这里定。分请求照参考项目 src/judge/judge.ts：
|
||||
// 判重一对一次请求；冻结条款三问每问单独一次请求，材料相同；代码改动一个差异块一次请求，五问同在。
|
||||
|
||||
type Factor = Submission<"proposal">["factors"][number];
|
||||
|
||||
/** 判重看的因子：名称和做法。关掉时换成什么、延续哪个因子不放：它们不改变"这条提案改什么"。 */
|
||||
export type DedupFactor = Pick<Factor, "name" | "method">;
|
||||
|
||||
/** 一份冻结协议的条款（docs/方法.md 4.3 ⑥"冻结协议包括"），字段与统一提交相同。 */
|
||||
export type FrozenProtocol = Pick<Submission<"unification">, "factors" | "safeguards" | "metrics" | "body" | "clauses">;
|
||||
|
||||
/** 协议要求候选代码采集的一条指标（新登记的和引用的都在内，由调用方查全）。 */
|
||||
export type ProtocolMetric = Pick<HypothesisMetric, "id" | "howMeasured" | "missing">;
|
||||
|
||||
/** 查代码改动时对照的那部分协议：协议正文和要采集的指标。没有要采集的指标时是空数组。 */
|
||||
export interface ProtocolRules {
|
||||
body: string;
|
||||
metrics: readonly ProtocolMetric[];
|
||||
}
|
||||
|
||||
/** 一个差异块：哪个文件、文件是新增删除还是修改，以及带 3 行上下文的块正文（从 @@ 行起）。 */
|
||||
export interface CodeHunk {
|
||||
path: string;
|
||||
change: "added" | "deleted" | "modified";
|
||||
text: string;
|
||||
}
|
||||
|
||||
/** 查冻结条款、查代码改动的结论：每一问的判断，以及有没有任何一问判"是"。 */
|
||||
export interface JevCheck {
|
||||
flagged: boolean;
|
||||
judgments: JevJudgment[];
|
||||
}
|
||||
|
||||
/** ③ 判重：两条提案的因子清单是不是同一个改法。 */
|
||||
export async function judgeSameChange(model: string, a: readonly DedupFactor[], b: readonly DedupFactor[]): Promise<JevJudgment> {
|
||||
const state = {
|
||||
task: taskOf("dedup"),
|
||||
proposal_a: { factors: a.map(dedupFactor) },
|
||||
proposal_b: { factors: b.map(dedupFactor) },
|
||||
};
|
||||
const [judgment] = await ask(model, state, questionsOf("dedup"));
|
||||
return judgment as JevJudgment;
|
||||
}
|
||||
|
||||
/** ⑥ 查一份冻结协议的条款：改动了评测、自相矛盾、写得太含糊。三问逐个发，材料相同。 */
|
||||
export async function checkProtocol(model: string, protocol: FrozenProtocol): Promise<JevCheck> {
|
||||
const state = {
|
||||
task: taskOf("freeze"),
|
||||
protocol: {
|
||||
factors: protocol.factors,
|
||||
safeguards: protocol.safeguards,
|
||||
body: protocol.body,
|
||||
metrics: protocol.metrics,
|
||||
clauses: protocol.clauses,
|
||||
},
|
||||
};
|
||||
const judgments: JevJudgment[] = [];
|
||||
for (const question of questionsOf("freeze")) judgments.push(...(await ask(model, state, [question])));
|
||||
return check(judgments);
|
||||
}
|
||||
|
||||
/** ⑨ 查一个差异块：五问一次请求，对照协议正文和要采集的指标。 */
|
||||
export async function checkHunk(model: string, hunk: CodeHunk, protocol: ProtocolRules): Promise<JevCheck> {
|
||||
const state = {
|
||||
task: taskOf("diff"),
|
||||
protocol: {
|
||||
body: protocol.body,
|
||||
metrics: protocol.metrics.map((m) => ({ id: m.id, howMeasured: m.howMeasured, missing: m.missing })),
|
||||
},
|
||||
change: { path: hunk.path, change: hunk.change, diff: hunk.text },
|
||||
};
|
||||
return check(await ask(model, state, questionsOf("diff")));
|
||||
}
|
||||
|
||||
function dedupFactor(f: DedupFactor): { [key: string]: Json } {
|
||||
return { name: f.name, method: f.method };
|
||||
}
|
||||
|
||||
function check(judgments: JevJudgment[]): JevCheck {
|
||||
return { flagged: judgments.some((j) => j.yes), judgments };
|
||||
}
|
||||
|
||||
/**
|
||||
* 把 workspace 比较两个版本的结果切成差异块，每块一次请求（一次放多块有位置效应）。
|
||||
* 差异原文里每个文件一段（以 `diff --git ` 开头），与文件清单同序，按位置对上；段里每个 `@@` 行起一块。
|
||||
* 二进制文件、只改可执行位、空文件增删没有 `@@` 行，不出块：这些改动 Jev 看不到内容，由程序检查。
|
||||
*/
|
||||
export function codeHunks(comparison: {
|
||||
files: ReadonlyArray<{ path: string; change: CodeHunk["change"] }>;
|
||||
diff: string;
|
||||
}): CodeHunk[] {
|
||||
// 内容行都以 +、-、空格开头，行首的 `diff --git `、`@@ ` 只会是 git 写的。
|
||||
const sections = comparison.diff.split(/^diff --git /m).slice(1);
|
||||
if (sections.length !== comparison.files.length) {
|
||||
throw new Error(`差异原文有 ${sections.length} 段，文件清单有 ${comparison.files.length} 个，对不上`);
|
||||
}
|
||||
return comparison.files.flatMap((file, i) =>
|
||||
(sections[i] as string)
|
||||
.split(/^(?=@@ )/m)
|
||||
.slice(1)
|
||||
.map((text) => ({ path: file.path, change: file.change, text: text.replace(/\n$/, "") })),
|
||||
);
|
||||
}
|
||||
|
||||
/** 就一份材料问几问，一次请求，交回每一问的判断。 */
|
||||
async function ask(model: string, state: { [key: string]: Json }, questions: readonly Question[]): Promise<JevJudgment[]> {
|
||||
const reply = await post(
|
||||
JSON.stringify({ model, state, questions: Object.fromEntries(questions.map((q) => [q.name, q.noul])) }),
|
||||
);
|
||||
// 回答是外部服务的返回，读不懂就报错停下，带上请求编号（CLAUDE.md"罕见情况怎么办"第 2 条）。
|
||||
if (reply.requestId === null) throw new Error(`Jev 的回答没带请求编号（x-typesafe-request-id），正文：${reply.text.slice(0, 500)}`);
|
||||
const requestId = reply.requestId;
|
||||
const where = `（request-id: ${requestId}）`;
|
||||
let response: unknown;
|
||||
try {
|
||||
response = JSON.parse(reply.text);
|
||||
} catch (error) {
|
||||
throw new Error(`Jev 的回答不是 JSON${where}：${reply.text.slice(0, 500)}`, { cause: error });
|
||||
}
|
||||
const answers = (response as { answers?: Record<string, { noul?: unknown } | undefined> } | null)?.answers;
|
||||
return questions.map((q) => {
|
||||
const probability = answers?.[q.name]?.noul;
|
||||
if (typeof probability !== "number" || !(probability >= 0 && probability <= 1)) {
|
||||
throw new Error(`Jev 的回答里问题 ${q.name} 没有 0 到 1 之间的概率${where}：${reply.text.slice(0, 500)}`);
|
||||
}
|
||||
return { question: q.name, questionHash: q.hash, threshold: q.threshold, probability, yes: probability > q.threshold, requestId };
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 发请求
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const ENDPOINT = "https://api.typesafe.ai/v1/systemone";
|
||||
|
||||
/** 一次请求最多等多久。Jev 一般零点几秒就答，等一分钟还没回就当这一次失败。 */
|
||||
const TIMEOUT_MS = 60_000;
|
||||
|
||||
/**
|
||||
* 调不通时第几次重试前等多久，一共最多发 4 次。
|
||||
* 网络抖动、服务短暂过载、限流（HTTP 429）多在几秒到几十秒内恢复，所以先等 2 秒，再拉长到 10 秒、30 秒；
|
||||
* 合计等 42 秒还不通，就不是一时的问题，报错停下由用户决定，不把"没判"当成"没问题"。
|
||||
*/
|
||||
const RETRY_WAITS_MS = [2_000, 10_000, 30_000];
|
||||
|
||||
/**
|
||||
* 发一次请求，拿回正文和请求编号。只有原因在我们控制之外的失败才重试（CLAUDE.md"错误怎么处理"）：
|
||||
* 连不上、超时、读正文失败、HTTP 429 和 5xx。其余 4xx 说明请求本身或密钥有问题，重试也一样，直接报错。
|
||||
* 密钥只从环境变量读进请求头，不进正文。
|
||||
*/
|
||||
async function post(body: string): Promise<{ requestId: string | null; text: string }> {
|
||||
const key = process.env["TYPESAFE_API_KEY"];
|
||||
if (key === undefined || key.trim() === "") {
|
||||
throw new Error("环境变量 TYPESAFE_API_KEY 没有设置，Jev 调不通");
|
||||
}
|
||||
const failures: string[] = [];
|
||||
for (let attempt = 0; attempt <= RETRY_WAITS_MS.length; attempt += 1) {
|
||||
if (attempt > 0) await wait(RETRY_WAITS_MS[attempt - 1] as number);
|
||||
let response: Response;
|
||||
let text: string;
|
||||
try {
|
||||
response = await fetch(ENDPOINT, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
|
||||
body,
|
||||
signal: AbortSignal.timeout(TIMEOUT_MS),
|
||||
});
|
||||
text = await response.text();
|
||||
} catch (error) {
|
||||
failures.push(`第 ${attempt + 1} 次：${error instanceof Error ? error.message : String(error)}`);
|
||||
continue;
|
||||
}
|
||||
const requestId = response.headers.get("x-typesafe-request-id");
|
||||
if (response.ok) return { requestId, text };
|
||||
const what = `HTTP ${response.status}（request-id: ${requestId ?? "无"}）：${text.slice(0, 500)}`;
|
||||
if (response.status !== 429 && response.status < 500) throw new Error(`Jev 拒绝了请求，不重试：${what}`);
|
||||
failures.push(`第 ${attempt + 1} 次：${what}`);
|
||||
}
|
||||
throw new Error(`Jev 调不通，发了 ${failures.length} 次都失败：${failures.join("；")}`);
|
||||
}
|
||||
|
||||
function wait(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
import { hash, type Json } from "../vocab/index.ts";
|
||||
|
||||
// Jev 的九问（docs/方法.md 4.3 ③⑥⑨）：每一处的材料说明、问题原文、阈值。
|
||||
// 措辞、判"是"判"否"的说明和示例句取自参考项目 src/judge/questions.ts（填圆那一套），
|
||||
// 只改了必须改的：点 → 因子，字段名对上 vocab 的提案和冻结协议，改动对上 workspace 的比较结果。
|
||||
// 写法照参考项目 research-wiki/参考-Jev使用指南.md：问题用英文、一问只问一件事、
|
||||
// 统一问坏的一面（概率高 = 有问题或"是同一个"）、判据带 not_for 和例子、问题里用反引号点到材料路径。
|
||||
|
||||
/** 用到 Jev 的三处：判重、冻结条款、一段代码改动。 */
|
||||
export type Site = "dedup" | "freeze" | "diff";
|
||||
|
||||
/** 一条是非题（Jev 的 noul）。 */
|
||||
export interface Noul {
|
||||
type: "noul";
|
||||
instructions: Json;
|
||||
criteria: Json;
|
||||
}
|
||||
|
||||
/**
|
||||
* 每一问的阈值：概率超过阈值判"是"。
|
||||
* 这是参考项目在它自己的材料上校准的值（JEV_THRESHOLDS 填圆那一套），没有在新项目的材料上校准过，等真实数据。
|
||||
*/
|
||||
const THRESHOLDS: Readonly<Record<Site, Readonly<Record<string, number>>>> = {
|
||||
dedup: { same_change: 0.3 },
|
||||
freeze: { self_contradictory: 0.5, changes_measurement: 0.21, too_vague: 0.78 },
|
||||
diff: { unseeded_randomness: 0.55, reads_clock: 0.55, touches_measurement: 0.65, early_exit: 0.63, zero_for_unmeasured: 0.62 },
|
||||
};
|
||||
|
||||
/** 放进材料 `task` 字段的说明：这是什么材料、各部分是什么。 */
|
||||
const TASKS: Readonly<Record<Site, string>> = {
|
||||
dedup:
|
||||
"Two research proposals from the same round of a study that improves a program by changing its code. " +
|
||||
"Each proposal lists its change as `factors`: one or more separate edits, each with a short `name` and a `method` saying what to change and how. " +
|
||||
"Together the factors of one proposal are the whole change that one implementation would carry out. " +
|
||||
"The proposals were written by different researchers, in Chinese or English.",
|
||||
freeze:
|
||||
"The frozen protocol of one candidate in a study that improves a program by changing its code. " +
|
||||
"An implementer will edit the code exactly as `protocol` says, and the unchanged task harness will then run it once and measure the final metric. " +
|
||||
"The implementer may not edit the task's harness or validator files. `protocol.clauses` lists the clauses the implementer must satisfy one by one.",
|
||||
diff:
|
||||
"One hunk of a code change made by an implementer to a candidate program, in unified diff format (`change.diff`; lines starting with + are added, - are removed). " +
|
||||
"`protocol.body` is the frozen protocol the implementer must follow, in Markdown with headings, among them: what must not change, randomness, when to stop, and which runs do not count. " +
|
||||
"`protocol.metrics` lists the metrics the protocol requires the implementer " +
|
||||
"to compute in the candidate code and write to metrics.json (each with its `howMeasured` and `missing` rule; an empty list means none). " +
|
||||
"Only this hunk is shown; judge only what the hunk itself adds or changes.",
|
||||
};
|
||||
|
||||
const QUESTIONS: Readonly<Record<Site, Readonly<Record<string, Noul>>>> = {
|
||||
// ③ 判重：一对提案一次请求。
|
||||
dedup: {
|
||||
same_change: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Do `proposal_a` and `proposal_b` propose the same code change, so that one implementation would carry out both?",
|
||||
focus:
|
||||
"Compare the whole change of each proposal: which parts of the program are changed and by what method, reading every factor's `method`. " +
|
||||
"Ignore wording, language, level of detail, factor names, and how the change is divided into factors: one proposal may describe as two factors what the other describes as one. " +
|
||||
"The same method with different parameter values or sizes is NOT the same change: each setting would need its own run.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Both describe the same algorithmic change to the same part of the program with the same settings; they differ only in wording, language, detail or how the change is split into factors",
|
||||
examples: [
|
||||
"'sort items by weight before packing' vs 'process items in decreasing order of weight'",
|
||||
"'replace SGD with Adam, lr 1e-3' vs 'switch the optimizer to Adam at learning rate 0.001'",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "They change different parts of the program, change the same part by a different method, or use the same method with different parameter values",
|
||||
not_for: "Two proposals that merely share the same goal, or merely edit the same function in different ways",
|
||||
examples: [
|
||||
"'replace SGD with Adam' vs 'add a learning-rate warmup to SGD'",
|
||||
"'a 6-row hexagonal grid' vs 'a 5-row hexagonal grid'",
|
||||
"'learning rate 1e-3' vs 'learning rate 1e-2' with the same optimizer",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
// ⑥ 冻结条款：每一问单独一次请求，材料相同。
|
||||
freeze: {
|
||||
self_contradictory: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question:
|
||||
"Does the change that `protocol` asks for break a restriction stated elsewhere in `protocol`, or do two of its statements exclude each other, so that no implementation can satisfy all of them?",
|
||||
focus:
|
||||
"Take the change described in `factors` (each factor's `method`), `safeguards` and `clauses`, then check it against every restriction written in `protocol.body` (what must not change, randomness, when to stop). " +
|
||||
"A restriction may name code that must not be edited, or things that must stay as in the baseline, such as positions, sizes or constants; carrying out the change may alter those things even if the change is described in other words. " +
|
||||
"A statement that is merely unusual, ambitious or likely to fail is not a contradiction.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Carrying out the requested change necessarily violates a stated restriction, or two statements require things that cannot both hold",
|
||||
examples: [
|
||||
"`clauses` says rewrite compute_max_radii while `protocol` says compute_max_radii must not change",
|
||||
"a factor's `method` uses random restarts while `protocol` says the program uses no randomness",
|
||||
"a factor's `method` doubles the step size while `protocol` says every numeric constant keeps its baseline value",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "All statements can hold at the same time in one implementation",
|
||||
not_for: "Plans that may not improve the metric, or statements that are vague but not conflicting",
|
||||
examples: ["rewrite construct_packing and keep compute_max_radii unchanged", "use seed 42 for 10 random restarts and keep the best result"],
|
||||
},
|
||||
},
|
||||
},
|
||||
changes_measurement: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question:
|
||||
"Does `protocol` require changing how the final metric is measured, computed or reported, rather than changing the solution that gets measured?",
|
||||
focus:
|
||||
"The final metric is measured by the task's harness from what the program returns. " +
|
||||
"Changing the algorithm that produces the solution is allowed. Changing how the returned value is summed, scored, validated, rounded or reported is a change of measurement. " +
|
||||
"The program's own handling of its candidate solutions — legality checks, repairs, safety margins such as subtracting 1e-9 from every radius before checking, fallbacks, and rules for picking the best result — changes the answer it returns, not how that answer is measured, even when it copies the validator's formula. " +
|
||||
"Likewise, rules for the protocol's own diagnostic metrics (what they count, how they are aggregated, when they are written) do not change the final metric.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The protocol asks to alter the scoring or validation: the formula of the metric, what gets counted, tolerances, the harness or validator, or the value reported instead of the real one",
|
||||
examples: [
|
||||
"report the sum of diameters instead of the sum of radii",
|
||||
"count only circles with radius above 0.01 when summing",
|
||||
"raise the overlap tolerance in validate.py to 1e-6",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "The protocol only changes how the solution is built or searched; the metric is computed from its output as before",
|
||||
not_for: "Adding new diagnostic metrics that are recorded next to the final metric without replacing it",
|
||||
examples: ["replace the ring layout with a hexagonal grid", "compute radii with a linear program"],
|
||||
},
|
||||
},
|
||||
},
|
||||
too_vague: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question:
|
||||
"Does `protocol` leave the implementer to guess what to change, for example which function to edit, which method to use, or when an iterative search stops?",
|
||||
focus:
|
||||
"Read `factors` (each factor's `method`), `clauses` and the stopping rule in `protocol.body` as an implementer who must follow them without asking questions. " +
|
||||
"Brief text is fine when it names the function, the method and its settings. " +
|
||||
"Judge only whether each statement is specific. Whether two statements conflict with each other is checked separately; a protocol whose statements are concrete but conflicting is not vague.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "An implementer would have to invent a key part of the change: the target code, the method, its main parameters, or the stopping rule of a loop or search",
|
||||
examples: ["improve the layout of the circles somehow", "optimize the radii better", "run a local search until it is good enough"],
|
||||
},
|
||||
false: {
|
||||
what: "The target code, the method and its settings are stated concretely enough to implement directly",
|
||||
not_for:
|
||||
"Protocols that leave only minor coding details, such as variable names, to the implementer, or protocols whose concrete statements contradict each other",
|
||||
examples: [
|
||||
"in construct_packing, change the outer ring radius constant from 0.7 to 0.45",
|
||||
"replace compute_max_radii with an LP solved by a numpy simplex; stop when no pivot improves",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
// ⑨ 代码改动：一个差异块一次请求，五问同在。
|
||||
diff: {
|
||||
unseeded_randomness: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Do the added lines in `change.diff` use random numbers without the fixed seed that the randomness section of `protocol.body` requires?",
|
||||
focus:
|
||||
"Look for random number generators and random sampling in added lines. " +
|
||||
"Randomness that is seeded exactly as the randomness section of `protocol.body` describes is fine.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Added code draws random numbers from an unseeded generator, a time-based seed, or a seed that differs from the protocol",
|
||||
examples: ["np.random.rand(26, 2)", "rng = np.random.default_rng()", "random.seed(time.time())"],
|
||||
},
|
||||
false: {
|
||||
what: "Added code uses no random numbers, or seeds them as the protocol says",
|
||||
not_for: "Deterministic code that merely contains the word random in a name or comment",
|
||||
examples: ["rng = np.random.default_rng(42) when the protocol registers seed 42", "centers[i] = [x0 + d * k, y0]"],
|
||||
},
|
||||
},
|
||||
},
|
||||
reads_clock: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question:
|
||||
"Do the added lines in `change.diff` make the program's result depend on the clock in a way that the stopping section of `protocol.body` does not register?",
|
||||
focus:
|
||||
"Look for time.time, time.perf_counter, time.monotonic, datetime.now and deadlines built from them. " +
|
||||
"A wall-clock time limit that the stopping section of `protocol.body` itself describes, implemented with the same limit to stop the same loop, is required by the protocol and is fine. " +
|
||||
"Any other use of the clock that changes what the program computes counts, including a time limit different from the registered one " +
|
||||
"and the registered time used for something other than stopping. Reading the clock only to print or log timing does not make the result depend on it.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Added code reads the clock and uses it to decide how long to search, when to stop, or which values to use, and the stopping section of `protocol.body` does not register that use or registers a different time limit",
|
||||
examples: [
|
||||
"while time.time() - t0 < 5: # protocol says a fixed 2000 steps",
|
||||
"step = 0.02 if time.time() - t0 < 3 else 0.005",
|
||||
"seed = int(datetime.now().timestamp())",
|
||||
"while time.monotonic() - t0 < 60.0: # protocol registers a 120 s limit",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "Added code does not read the clock, reads it only for printing or logging, or stops a loop on elapsed time exactly as the stopping section of `protocol.body` registers",
|
||||
not_for: "Loops bounded by an iteration count",
|
||||
examples: [
|
||||
"for it in range(2000):",
|
||||
"print(f'took {time.time() - t0:.2f}s')",
|
||||
"while time.monotonic() - t0 < 120.0 and it < 2000: # protocol: stop after 120 s of wall time or 2000 iterations",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
touches_measurement: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question:
|
||||
"Do the added or removed lines in `change.diff` change how the program computes or reports its score, or compute a metric from `protocol.metrics` differently from its `howMeasured`?",
|
||||
focus:
|
||||
"The score is computed from the returned solution, such as summing the radii in run_packing. " +
|
||||
"Changing how circles are placed or how radii are chosen changes the solution, not the score computation. " +
|
||||
"Code that computes a metric listed in `protocol.metrics` the way its `howMeasured` says and writes it to metrics.json is required by the protocol; it is not a change to the score.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "The hunk changes the score formula, scales or rounds the reported value, reports a value other than the one computed from the solution, edits the code that checks validity, or computes a listed metric with a formula or data different from its `howMeasured`",
|
||||
examples: [
|
||||
"sum_radii = np.sum(radii) * 1.01",
|
||||
"sum_radii = 2 * np.sum(radii)",
|
||||
"return centers, radii, 2.7",
|
||||
"val = 1.0 # when the metric's howMeasured gives a formula",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "The hunk only changes how centers or radii are produced, or computes and writes a metric from `protocol.metrics` as its `howMeasured` describes; the score is still computed as before",
|
||||
not_for: "Recording extra diagnostic values next to the score",
|
||||
examples: [
|
||||
"radii = solve_lp(centers)",
|
||||
"centers[i + 9] = [0.5 + 0.45 * np.cos(a), 0.5 + 0.45 * np.sin(a)]",
|
||||
"json.dump({metric_id: float(np.min(gaps))}, f) # when the metric's howMeasured says min gap",
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
early_exit: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Do the added lines in `change.diff` end a loop, a search or the program earlier than the stopping section of `protocol.body` says?",
|
||||
focus:
|
||||
"Look for added break, return, sys.exit or raise statements and added limits on iterations. " +
|
||||
"A stop that the stopping section of `protocol.body` itself describes is fine.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "Added code stops work before the protocol's stopping rule is met, or adds a stopping rule the protocol does not mention",
|
||||
examples: ["if it > 100: break # protocol says 2000 iterations", "if total > 2.0: return centers, radii", "sys.exit(0)"],
|
||||
},
|
||||
false: {
|
||||
what: "Added code has no new exits, or exits exactly as the protocol's stopping rule says",
|
||||
not_for: "Returning the final result at the normal end of a function",
|
||||
examples: ["if no_improvement_for >= 100: break # protocol: stop after 100 idle iterations", "return centers, radii"],
|
||||
},
|
||||
},
|
||||
},
|
||||
zero_for_unmeasured: {
|
||||
type: "noul",
|
||||
instructions: {
|
||||
question: "Do the added lines in `change.diff` record 0 or another fixed number when a value could not be measured or computed?",
|
||||
focus:
|
||||
"Look at exception handlers, missing-value branches and default values of recorded metrics. " +
|
||||
"Recording an explicit 'unavailable' marker or leaving the value out is the correct way; writing 0 hides the failure.",
|
||||
},
|
||||
criteria: {
|
||||
true: {
|
||||
what: "When a computation fails or a value is missing, added code stores or returns 0 or another made-up number in its place",
|
||||
examples: [
|
||||
"except Exception: min_gap = 0.0",
|
||||
"metrics['clipped-count'] = count if ok else 0",
|
||||
"except ValueError: return centers, np.zeros(n)",
|
||||
],
|
||||
},
|
||||
false: {
|
||||
what: "Added code records failures as unavailable, raises an error, or does not deal with missing values at all",
|
||||
not_for: "Initializing an accumulator to 0 before a loop that always fills it",
|
||||
examples: ["except Exception: metrics['min-gap'] = None", "total = 0.0\nfor r in radii: total += r"],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/** 一问的全部内容：问题原文、阈值，以及材料说明、问题原文、阈值三者合起来的哈希。 */
|
||||
export interface Question {
|
||||
name: string;
|
||||
noul: Noul;
|
||||
threshold: number;
|
||||
hash: string;
|
||||
}
|
||||
|
||||
/** 一处的全部问题，按上面写的顺序。 */
|
||||
export function questionsOf(site: Site): Question[] {
|
||||
return Object.entries(QUESTIONS[site]).map(([name, noul]) => {
|
||||
const threshold = THRESHOLDS[site][name] as number;
|
||||
return { name, noul, threshold, hash: hash({ task: TASKS[site], name, noul: noul as unknown as Json, threshold }) };
|
||||
});
|
||||
}
|
||||
|
||||
export function taskOf(site: Site): string {
|
||||
return TASKS[site];
|
||||
}
|
||||
+31
-8
@@ -2,7 +2,9 @@ import { Type, type Static, type TSchema } from "typebox";
|
||||
import { Value } from "typebox/value";
|
||||
import {
|
||||
Id,
|
||||
JevJudgment,
|
||||
JudgedResult,
|
||||
Measured,
|
||||
ModelUsage,
|
||||
ProcessResult,
|
||||
Range,
|
||||
@@ -26,6 +28,11 @@ const Round = Type.Integer({ minimum: 0 });
|
||||
const Version = Type.Integer({ minimum: 1 });
|
||||
const Seq = Type.Integer({ minimum: 1 });
|
||||
const OptionalId = Type.Union([Id, Type.Null()]);
|
||||
/**
|
||||
* 这件事用到的 Jev 判断，原样存（docs/结构.md 第四部分：Jev 不单独成记录，记在它判断的那件事里）。
|
||||
* 一次可能问好几问（如冻结条款三问、代码改动五问），所以是数组，至少一条；没用到 Jev 的写 null。
|
||||
*/
|
||||
const JevUsed = Type.Union([Type.Array(JevJudgment, { minItems: 1 }), Type.Null()]);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 提交（记录 3）
|
||||
@@ -64,16 +71,24 @@ type StoredBody<K extends SubmissionKind> = K extends "proposal"
|
||||
? Omit<Submission<"proposal">, "authorEstimate">
|
||||
: Submission<K>;
|
||||
|
||||
/** 一份提交的一版。version 从 1 起，替换时加 1，被替换的就是上一版。 */
|
||||
/**
|
||||
* 一份提交的一版。version 从 1 起，替换时加 1，被替换的就是上一版。
|
||||
* jev 是收下这一版时用到的 Jev 判断（判"否"才收下，原始概率也存），没用到 Jev 的为 null。
|
||||
*/
|
||||
type SubmissionOf<Body extends Record<SubmissionKind, unknown>> = {
|
||||
[K in SubmissionKind]: Omit<SubmissionSlot, "kind"> & { version: number; kind: K; content: Body[K] };
|
||||
[K in SubmissionKind]: Omit<SubmissionSlot, "kind"> & {
|
||||
version: number;
|
||||
kind: K;
|
||||
content: Body[K];
|
||||
jev: Static<typeof JevUsed>;
|
||||
};
|
||||
}[SubmissionKind];
|
||||
|
||||
export type StoredSubmission = SubmissionOf<{ [K in SubmissionKind]: StoredBody<K> }>;
|
||||
/** 写入时交给 ledger 的提交：提案带着作者自评。 */
|
||||
export type SubmissionInput = SubmissionOf<{ [K in SubmissionKind]: Submission<K> }>;
|
||||
|
||||
/** 某种提交一版的完整字段定义：外层的位置和版本号，加上按提交种类检查的 content。 */
|
||||
/** 某种提交一版的完整字段定义：外层的位置、版本号和 Jev 判断，加上按提交种类检查的 content。 */
|
||||
function submissionSchema(kind: SubmissionKind, body: TSchema): TSchema {
|
||||
return Type.Object(
|
||||
{
|
||||
@@ -84,6 +99,7 @@ function submissionSchema(kind: SubmissionKind, body: TSchema): TSchema {
|
||||
version: Version,
|
||||
kind: Type.Literal(kind),
|
||||
content: body,
|
||||
jev: JevUsed,
|
||||
},
|
||||
CLOSED,
|
||||
);
|
||||
@@ -115,7 +131,7 @@ export const KINDS = {
|
||||
submission: Type.Unsafe<StoredSubmission>({}),
|
||||
/**
|
||||
* 3 退回：被退回的那份内容原样存（它可能正是因为不合格式才被退回），连同退回理由。
|
||||
* 理由来自 Jev 的，jev 里带原始概率和结论；程序检查退回的，jev 为空。
|
||||
* jev 是这次退回用到的 Jev 判断（见 JevUsed），判"否"的几问也一起存；没用到 Jev 的写 null。
|
||||
*/
|
||||
rejection: Type.Object(
|
||||
{
|
||||
@@ -124,7 +140,7 @@ export const KINDS = {
|
||||
kind: Type.Enum(SUBMISSION_KIND_NAMES, { type: "string" }),
|
||||
submitted: AnyJson,
|
||||
reason: Text,
|
||||
jev: Type.Union([Type.Object({ probabilities: AnyJson, conclusion: AnyJson }, CLOSED), Type.Null()]),
|
||||
jev: JevUsed,
|
||||
},
|
||||
CLOSED,
|
||||
),
|
||||
@@ -183,10 +199,17 @@ export const KINDS = {
|
||||
callStarted: Type.Object({ call: Id, actor: Id, stageId: OptionalId, model: Text, materials: AnyJson }, CLOSED),
|
||||
/** 10 模型调用每次回复的用量，reply 从 1 起。 */
|
||||
callUsage: Type.Object({ call: Id, reply: Type.Integer({ minimum: 1 }), usage: ModelUsage }, CLOSED),
|
||||
/** 10 模型调用结束：正常结束，或失败并写明原因。一次调用只有一条。 */
|
||||
/**
|
||||
* 10 模型调用结束，一次调用只有一条。status 与 agents 的会话结局（SessionEnding）同名同义：
|
||||
* submitted 至少有一份被收下；notSubmitted 模型自己停下、一份都没收下；failed 模型服务报错等；stopped 用户停下。
|
||||
* wallSeconds 是这次调用的墙钟时间（秒），四种结局都有：正常结束时由 agents 测出；
|
||||
* 控制进程中断、恢复时补写的结束记录量不到，写 failed、原因"进程中断"，墙钟时间记测不到（docs/结构.md 第四部分，CLAUDE.md"中断后怎么恢复"）。
|
||||
*/
|
||||
callEnded: Type.Union([
|
||||
Type.Object({ call: Id, ending: Type.Literal("completed") }, CLOSED),
|
||||
Type.Object({ call: Id, ending: Type.Literal("failed"), reason: Text }, CLOSED),
|
||||
Type.Object({ call: Id, wallSeconds: Measured, status: Type.Literal("submitted") }, CLOSED),
|
||||
Type.Object({ call: Id, wallSeconds: Measured, status: Type.Literal("notSubmitted"), reason: Text }, CLOSED),
|
||||
Type.Object({ call: Id, wallSeconds: Measured, status: Type.Literal("failed"), reason: Text }, CLOSED),
|
||||
Type.Object({ call: Id, wallSeconds: Measured, status: Type.Literal("stopped") }, CLOSED),
|
||||
]),
|
||||
|
||||
/** 11 事件：违规、"按配置跳过"、"没测"及原因、停下的原因、运行状态未知等。event 是事件种类。 */
|
||||
|
||||
@@ -3,7 +3,7 @@ import type { DatabaseSync } from "node:sqlite";
|
||||
// 账本的表结构（docs/结构.md 第四部分"全部只追加"）：只有一张表，每一行是序号、种类、内容、内容的哈希。
|
||||
// 表结构一改就把版本号加 1；旧文件版本号对不上时打开报错，不迁移，也不逐字核对建表语句。
|
||||
|
||||
export const SCHEMA_VERSION = 2;
|
||||
export const SCHEMA_VERSION = 3;
|
||||
|
||||
/** 新文件要执行的建表语句，由 Ledger 在一个事务里执行，最后写入版本号。 */
|
||||
export const CREATE_SCHEMA = [
|
||||
|
||||
@@ -27,4 +27,5 @@ export {
|
||||
type MissingPolicy,
|
||||
type Verdict,
|
||||
} from "./runs.ts";
|
||||
export { JevJudgment } from "./jev.ts";
|
||||
export { Range, SUBMISSION_SCHEMAS, type Submission } from "./submissions.ts";
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { Type, type Static } from "typebox";
|
||||
|
||||
/**
|
||||
* Jev 对一个问题的一次判断（docs/方法.md 4.3 ③⑥⑨）。jev 交出它，ledger 在退回、判重合并等记录里原样存它，
|
||||
* 两个模块互不引用，所以类型放在这里。
|
||||
* - question：问题名，如 same_change；
|
||||
* - questionHash：问题全文（措辞、判"是"判"否"的说明与示例、阈值）的哈希，改了一个字就能从记录里看出来；
|
||||
* - probability：Jev 返回的原始概率；yes：probability 是否超过 threshold（概率高 = 有问题，或"是同一个"）；
|
||||
* - requestId：Jev 返回的请求编号，用来到服务那边核对。
|
||||
*/
|
||||
export const JevJudgment = Type.Object(
|
||||
{
|
||||
question: Type.String({ pattern: "^[a-z][a-z_]*$" }),
|
||||
questionHash: Type.String({ pattern: "^[0-9a-f]{64}$" }),
|
||||
threshold: Type.Number({ minimum: 0, maximum: 1 }),
|
||||
probability: Type.Number({ minimum: 0, maximum: 1 }),
|
||||
yes: Type.Boolean(),
|
||||
requestId: Type.String({ minLength: 1 }),
|
||||
},
|
||||
{ additionalProperties: false },
|
||||
);
|
||||
export type JevJudgment = Static<typeof JevJudgment>;
|
||||
@@ -91,6 +91,21 @@ test("类型或取值不对报错，带字段路径", () => {
|
||||
}
|
||||
});
|
||||
|
||||
test("Jev 模型必填：少了、写 null、写空白都报错，不补 jev-latest", () => {
|
||||
const missing = fullConfig() as unknown as Obj;
|
||||
delete (missing.models as Obj).jev;
|
||||
assert.match(errorOf(missing), /^\/models\/jev：缺少这一项/m);
|
||||
const nulled = fullConfig() as unknown as Obj;
|
||||
(nulled.models as Obj).jev = null;
|
||||
assert.match(errorOf(nulled), /^\/models\/jev：应为字符串$/m);
|
||||
const blank = fullConfig() as unknown as Obj;
|
||||
(blank.models as Obj).jev = " ";
|
||||
assert.match(errorOf(blank), /^\/models\/jev：不能为空，也不能只有空白$/m);
|
||||
const naive = naiveConfig() as unknown as Obj;
|
||||
delete (naive.models as Obj).jev;
|
||||
assert.match(errorOf(naive), /^\/models\/jev：缺少这一项/m);
|
||||
});
|
||||
|
||||
test("一次列出多处错误，超过 typebox 默认的 8 条、错在 Union 分支里的也都列出", () => {
|
||||
const config = fullConfig() as unknown as Obj;
|
||||
config.runs = -1;
|
||||
@@ -196,6 +211,7 @@ test("字段定义里每个字段都有中文说明，并标明用户配置还
|
||||
assert.equal(category(["runs"]), "用户配置");
|
||||
assert.equal(category(["factorLimit"]), "方法参数");
|
||||
assert.equal(category(["flow"]), "方法参数");
|
||||
assert.equal(category(["models", "jev"]), "用户配置");
|
||||
assert.ok(seen.includes("/models/researchers/0/thinking"));
|
||||
assert.ok(seen.includes("/flow/ablation/cellRule"));
|
||||
});
|
||||
|
||||
@@ -13,6 +13,7 @@ export function fullConfig(): ResearchConfig {
|
||||
researchers: [{ ...glm }, { ...glm }, { ...glm }],
|
||||
experimenter: { ...glm },
|
||||
verifier: { ...glm },
|
||||
jev: "jev-latest",
|
||||
},
|
||||
proposalLimit: 1,
|
||||
batchLimit: 2,
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { checkHunk, checkProtocol, judgeSameChange } from "../src/jev/index.ts";
|
||||
import type { JevJudgment } from "../src/vocab/index.ts";
|
||||
|
||||
// Jev 的真实接线测试（任务书 08 第七部分），npm run test:jev -- <模型名>：要联网、要密钥 TYPESAFE_API_KEY，不放进 npm test。
|
||||
// 三处各问一次真实 Jev，材料是随手写的简单填圆材料。只确认请求发得出去、九问都拿回 0 到 1 之间的概率、
|
||||
// 请求编号读得到；不判断概率对不对。
|
||||
|
||||
const model = process.argv[2];
|
||||
if (model === undefined || process.argv.length !== 3) throw new Error("用法：npm run test:jev -- <Jev 模型名>，例如 jev-latest");
|
||||
|
||||
const body =
|
||||
"### 不许改什么\n\nharness.py、validate.py 不改。\n\n### 随机性\n\n不用随机数。\n\n" +
|
||||
"### 何时停\n\n程序跑完一次 construct_packing 就停。\n\n### 什么运行不算数\n\n校验不通过的运行不算数。";
|
||||
|
||||
const factor = {
|
||||
name: "外圈环半径缩到 0.45",
|
||||
method: "把 construct_packing 里外圈环半径常数 0.7 改为 0.45。",
|
||||
keySettings: "环半径 0.45",
|
||||
continues: null,
|
||||
};
|
||||
|
||||
const all: Array<[string, JevJudgment]> = [];
|
||||
|
||||
const same = await judgeSameChange(model, [factor], [{ name: "外圈收紧", method: "construct_packing 中外圈 8 个圆所在环的半径从 0.7 改成 0.45。" }]);
|
||||
all.push(["判重", same]);
|
||||
|
||||
const frozen = await checkProtocol(model, {
|
||||
factors: [factor],
|
||||
safeguards: [],
|
||||
metrics: { defined: [], reused: [] },
|
||||
body,
|
||||
clauses: ["initial.py 的 construct_packing 里，外圈环半径常数 0.7 改为 0.45"],
|
||||
});
|
||||
for (const j of frozen.judgments) all.push(["冻结条款", j]);
|
||||
|
||||
const hunk = await checkHunk(
|
||||
model,
|
||||
{
|
||||
path: "initial.py",
|
||||
change: "modified",
|
||||
text:
|
||||
"@@ -33,7 +33,7 @@ def construct_packing():\n" +
|
||||
" for i in range(8):\n" +
|
||||
" angle = 2 * np.pi * i / 8\n" +
|
||||
"- centers[i + 1] = [0.5 + 0.7 * np.cos(angle), 0.5 + 0.7 * np.sin(angle)]\n" +
|
||||
"+ centers[i + 1] = [0.5 + 0.45 * np.cos(angle), 0.5 + 0.45 * np.sin(angle)]\n" +
|
||||
" \n" +
|
||||
" for i in range(16):\n" +
|
||||
" angle = 2 * np.pi * i / 16",
|
||||
},
|
||||
{ body, metrics: [] },
|
||||
);
|
||||
for (const j of hunk.judgments) all.push(["代码改动", j]);
|
||||
|
||||
for (const [site, j] of all) {
|
||||
console.log(`${site}\t${j.question}\t概率 ${j.probability}\t阈值 ${j.threshold}\t${j.yes ? "是" : "否"}\t请求编号 ${j.requestId}`);
|
||||
}
|
||||
console.log(`冻结条款有问题：${frozen.flagged}；代码改动有问题：${hunk.flagged}`);
|
||||
|
||||
assert.deepEqual(
|
||||
all.map(([, j]) => j.question),
|
||||
["same_change", "self_contradictory", "changes_measurement", "too_vague", "unseeded_randomness", "reads_clock", "touches_measurement", "early_exit", "zero_for_unmeasured"],
|
||||
);
|
||||
for (const [, j] of all) {
|
||||
assert.ok(j.probability >= 0 && j.probability <= 1, `${j.question} 的概率不在 0 到 1 之间`);
|
||||
assert.ok(j.requestId.length > 0, `${j.question} 没有请求编号`);
|
||||
}
|
||||
// 冻结三问各发一次请求，请求编号各不相同；代码改动五问一次请求，编号相同。
|
||||
assert.equal(new Set(frozen.judgments.map((j) => j.requestId)).size, 3);
|
||||
assert.equal(new Set(hunk.judgments.map((j) => j.requestId)).size, 1);
|
||||
console.log("九问都拿回了 0 到 1 之间的概率和请求编号");
|
||||
@@ -0,0 +1,308 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { test, type TestContext } from "node:test";
|
||||
import { checkHunk, checkProtocol, codeHunks, judgeSameChange, type FrozenProtocol, type ProtocolRules } from "../src/jev/index.ts";
|
||||
import { hash, type Json } from "../src/vocab/index.ts";
|
||||
import { compareVersions, createRepository, saveVersion } from "../src/workspace/index.ts";
|
||||
|
||||
// jev 的测试（任务书 08 第七部分），不花钱：假 Jev 在测试自己的环境里造——把全局 fetch 换成假的，
|
||||
// 产品代码照常向真实地址发请求，拿到的是这里造的回答。重试的等待把全局 setTimeout 换成立刻回调，并记下要等多久。
|
||||
|
||||
type Reply = { status: number; body: string; requestId: string | null };
|
||||
type Sent = { url: string; auth: string | null; body: { model: string; state: any; questions: Record<string, any> } };
|
||||
|
||||
/** 装一个假 Jev：每次请求交给 answer 决定回什么，并记下发出去的请求。 */
|
||||
function fakeJev(t: TestContext, answer: (body: Sent["body"], n: number) => Reply | Error): Sent[] {
|
||||
const sent: Sent[] = [];
|
||||
const key = process.env["TYPESAFE_API_KEY"];
|
||||
process.env["TYPESAFE_API_KEY"] = "test-key";
|
||||
t.after(() => {
|
||||
if (key === undefined) delete process.env["TYPESAFE_API_KEY"];
|
||||
else process.env["TYPESAFE_API_KEY"] = key;
|
||||
});
|
||||
t.mock.method(globalThis, "fetch", async (url: string, init: RequestInit) => {
|
||||
const body = JSON.parse(init.body as string);
|
||||
sent.push({ url, auth: new Headers(init.headers).get("authorization"), body });
|
||||
const reply = answer(body, sent.length);
|
||||
if (reply instanceof Error) throw reply;
|
||||
const headers = reply.requestId === null ? {} : { "x-typesafe-request-id": reply.requestId };
|
||||
return new Response(reply.body, { status: reply.status, headers });
|
||||
});
|
||||
return sent;
|
||||
}
|
||||
|
||||
/** 正常回答：每一问的概率由 p 给，请求编号是 req_<第几次请求>。 */
|
||||
function answerAll(p: (name: string) => number): (body: Sent["body"], n: number) => Reply {
|
||||
return (body, n) => ({
|
||||
status: 200,
|
||||
requestId: `req_${n}`,
|
||||
body: JSON.stringify({
|
||||
model: "jev-test",
|
||||
answers: Object.fromEntries(Object.keys(body.questions).map((name) => [name, { type: "noul", noul: p(name) }])),
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
/** 把重试前的等待换成立刻继续，记下每次要等的毫秒数。 */
|
||||
function instantWaits(t: TestContext): number[] {
|
||||
const waits: number[] = [];
|
||||
t.mock.method(globalThis, "setTimeout", (fn: () => void, ms: number) => {
|
||||
waits.push(ms);
|
||||
setImmediate(fn);
|
||||
});
|
||||
return waits;
|
||||
}
|
||||
|
||||
const PROTOCOL: FrozenProtocol = {
|
||||
factors: [
|
||||
{
|
||||
name: "外圈环半径缩到 0.45",
|
||||
method: "把 construct_packing 里外圈环半径常数 0.7 改为 0.45。",
|
||||
keySettings: "环半径 0.45",
|
||||
offBehavior: "回到 0.7",
|
||||
continues: null,
|
||||
},
|
||||
],
|
||||
safeguards: ["clip-radii"],
|
||||
metrics: {
|
||||
defined: [
|
||||
{ id: "min-gap", meaning: "最小间隙", unit: "无", direction: "maximize", howMeasured: "最终解上所有圆对的最小间隙，测一次", missing: "recordMissing" },
|
||||
],
|
||||
reused: ["c1.overlap"],
|
||||
},
|
||||
body: "### 不许改什么\n\nharness.py 不改。",
|
||||
clauses: ["外圈环半径常数改为 0.45（initial.py:36）"],
|
||||
};
|
||||
|
||||
const RULES: ProtocolRules = {
|
||||
body: PROTOCOL.body,
|
||||
metrics: [{ id: "min-gap", howMeasured: "最终解上所有圆对的最小间隙，测一次", missing: "recordMissing" }],
|
||||
};
|
||||
|
||||
const HUNK = { path: "initial.py", change: "modified", text: "@@ -1 +1 @@\n-a\n+b" } as const;
|
||||
|
||||
/** 参考项目 src/judge/questions.ts 的 JEV_THRESHOLDS 填圆那一套里这九问的值，抄自参考项目源码。 */
|
||||
const REFERENCE_THRESHOLDS: Record<string, number> = {
|
||||
same_change: 0.3,
|
||||
self_contradictory: 0.5,
|
||||
changes_measurement: 0.21,
|
||||
too_vague: 0.78,
|
||||
unseeded_randomness: 0.55,
|
||||
reads_clock: 0.55,
|
||||
touches_measurement: 0.65,
|
||||
early_exit: 0.63,
|
||||
zero_for_unmeasured: 0.62,
|
||||
};
|
||||
|
||||
test("概率超过阈值判是，等于或低于阈值判否；阈值是参考项目的值", async (t) => {
|
||||
// 冻结三问：第一问比阈值高 0.01，第二问正好等于阈值，第三问低 0.01。
|
||||
const offsets: Record<string, number> = { self_contradictory: 0.01, changes_measurement: 0, too_vague: -0.01 };
|
||||
const sent = fakeJev(t, answerAll((name) => Math.round(((REFERENCE_THRESHOLDS[name] as number) + (offsets[name] as number)) * 100) / 100));
|
||||
const result = await checkProtocol("jev-x", PROTOCOL);
|
||||
assert.deepEqual(
|
||||
result.judgments.map((j) => [j.question, j.threshold, j.yes, j.requestId]),
|
||||
[
|
||||
["self_contradictory", 0.5, true, "req_1"],
|
||||
["changes_measurement", 0.21, false, "req_2"],
|
||||
["too_vague", 0.78, false, "req_3"],
|
||||
],
|
||||
);
|
||||
assert.equal(result.flagged, true);
|
||||
// 冻结三问每问单独一次请求，材料相同；模型名照传；密钥只在请求头里。
|
||||
assert.deepEqual(
|
||||
sent.map((s) => Object.keys(s.body.questions)),
|
||||
[["self_contradictory"], ["changes_measurement"], ["too_vague"]],
|
||||
);
|
||||
for (const s of sent) {
|
||||
assert.deepEqual(s.body.state, sent[0]?.body.state);
|
||||
assert.equal(s.body.model, "jev-x");
|
||||
assert.equal(s.auth, "Bearer test-key");
|
||||
assert.equal(s.url, "https://api.typesafe.ai/v1/systemone");
|
||||
assert.ok(!JSON.stringify(s.body).includes("test-key"));
|
||||
}
|
||||
});
|
||||
|
||||
test("一个差异块五问一次请求，判重一对一次请求；阈值都是参考项目的值", async (t) => {
|
||||
const sent = fakeJev(t, answerAll((name) => (name === "early_exit" ? 0.64 : 0.2)));
|
||||
const hunk = await checkHunk("jev-x", HUNK, RULES);
|
||||
assert.equal(sent.length, 1);
|
||||
assert.equal(hunk.flagged, true);
|
||||
assert.deepEqual(
|
||||
hunk.judgments.map((j) => [j.question, j.threshold, j.yes]),
|
||||
[
|
||||
["unseeded_randomness", 0.55, false],
|
||||
["reads_clock", 0.55, false],
|
||||
["touches_measurement", 0.65, false],
|
||||
["early_exit", 0.63, true],
|
||||
["zero_for_unmeasured", 0.62, false],
|
||||
],
|
||||
);
|
||||
const same = await judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors);
|
||||
assert.equal(sent.length, 2);
|
||||
assert.deepEqual(same, { question: "same_change", questionHash: same.questionHash, threshold: 0.3, probability: 0.2, yes: false, requestId: "req_2" });
|
||||
});
|
||||
|
||||
test("回答读不懂就报错，带上请求编号", async (t) => {
|
||||
const cases: Array<[string, Reply, RegExp]> = [
|
||||
["没有概率", { status: 200, requestId: "req_a", body: JSON.stringify({ answers: {} }) }, /same_change 没有 0 到 1 之间的概率（request-id: req_a）/],
|
||||
[
|
||||
"概率大于 1",
|
||||
{ status: 200, requestId: "req_b", body: JSON.stringify({ answers: { same_change: { noul: 1.2 } } }) },
|
||||
/same_change 没有 0 到 1 之间的概率（request-id: req_b）/,
|
||||
],
|
||||
[
|
||||
"概率不是数",
|
||||
{ status: 200, requestId: "req_c", body: JSON.stringify({ answers: { same_change: { noul: "0.4" } } }) },
|
||||
/same_change 没有 0 到 1 之间的概率（request-id: req_c）/,
|
||||
],
|
||||
["不是 JSON", { status: 200, requestId: "req_d", body: "<html>" }, /不是 JSON（request-id: req_d）/],
|
||||
["没带请求编号", { status: 200, requestId: null, body: JSON.stringify({ answers: { same_change: { noul: 0.1 } } }) }, /没带请求编号/],
|
||||
];
|
||||
for (const [what, reply, message] of cases) {
|
||||
await t.test(what, async (t) => {
|
||||
fakeJev(t, () => reply);
|
||||
await assert.rejects(judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors), message);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("调不通时按 2、10、30 秒重试，发满 4 次再报错", async (t) => {
|
||||
const waits = instantWaits(t);
|
||||
const sent = fakeJev(t, () => new TypeError("fetch failed"));
|
||||
await assert.rejects(checkHunk("jev-x", HUNK, RULES), /发了 4 次都失败/);
|
||||
assert.equal(sent.length, 4);
|
||||
assert.deepEqual(waits, [2_000, 10_000, 30_000]);
|
||||
});
|
||||
|
||||
test("服务暂时出错后恢复就照常判；请求本身被拒不重试；没有密钥直接报错", async (t) => {
|
||||
await t.test("503 后恢复", async (t) => {
|
||||
const waits = instantWaits(t);
|
||||
const ok = answerAll(() => 0.9);
|
||||
const sent = fakeJev(t, (body, n) => (n === 1 ? { status: 503, requestId: "req_x", body: "busy" } : ok(body, n)));
|
||||
const j = await judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors);
|
||||
assert.deepEqual([j.yes, j.requestId], [true, "req_2"]);
|
||||
assert.equal(sent.length, 2);
|
||||
assert.deepEqual(waits, [2_000]);
|
||||
});
|
||||
await t.test("422 直接报错", async (t) => {
|
||||
const sent = fakeJev(t, () => ({ status: 422, requestId: "req_y", body: "bad" }));
|
||||
await assert.rejects(judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors), /拒绝了请求，不重试：HTTP 422（request-id: req_y）/);
|
||||
assert.equal(sent.length, 1);
|
||||
});
|
||||
await t.test("没有密钥", async (t) => {
|
||||
const sent = fakeJev(t, answerAll(() => 0.1));
|
||||
delete process.env["TYPESAFE_API_KEY"];
|
||||
await assert.rejects(judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors), /TYPESAFE_API_KEY/);
|
||||
assert.equal(sent.length, 0);
|
||||
});
|
||||
});
|
||||
|
||||
test("问题哈希把材料说明、措辞、示例、阈值一起算，改一个字就变", async (t) => {
|
||||
const sent = fakeJev(t, answerAll(() => 0.1));
|
||||
const judgments = [
|
||||
await judgeSameChange("jev-x", PROTOCOL.factors, PROTOCOL.factors),
|
||||
...(await checkProtocol("jev-x", PROTOCOL)).judgments,
|
||||
...(await checkHunk("jev-x", HUNK, RULES)).judgments,
|
||||
];
|
||||
assert.equal(judgments.length, 9);
|
||||
assert.equal(new Set(judgments.map((j) => j.questionHash)).size, 9);
|
||||
// 从实际发出去的请求里取回每一问的材料说明和问题原文。
|
||||
const asked = new Map(sent.flatMap((s) => Object.entries(s.body.questions).map(([name, noul]) => [name, { task: s.body.state.task, noul }])));
|
||||
for (const j of judgments) {
|
||||
const { task, noul } = asked.get(j.question) as { task: string; noul: Json };
|
||||
const content = { task, name: j.question, noul, threshold: j.threshold };
|
||||
assert.equal(j.questionHash, hash(content));
|
||||
const changed = JSON.parse(JSON.stringify(content));
|
||||
changed.noul.criteria.true.examples[0] = `${changed.noul.criteria.true.examples[0]}x`;
|
||||
assert.notEqual(hash(changed), j.questionHash);
|
||||
assert.notEqual(hash({ ...content, threshold: j.threshold + 0.01 }), j.questionHash);
|
||||
assert.notEqual(hash({ ...content, task: `${task} ` }), j.questionHash);
|
||||
}
|
||||
});
|
||||
|
||||
test("材料：给定带类型的输入，发出去的材料与手写预期一致", async (t) => {
|
||||
const sent = fakeJev(t, answerAll(() => 0.1));
|
||||
const other = [{ name: "六行六边形网格", method: "用 6 行六边形网格摆圆心。", keySettings: "6 行", continues: "f2" }];
|
||||
await judgeSameChange("jev-x", PROTOCOL.factors, other);
|
||||
await checkProtocol("jev-x", PROTOCOL);
|
||||
await checkHunk("jev-x", { path: "a.py", change: "added", text: "@@ -0,0 +1 @@\n+x" }, RULES);
|
||||
const states = sent.map((s) => s.body.state);
|
||||
const [dedup, freeze, , , diff] = states;
|
||||
for (const state of states) assert.equal(typeof state.task, "string");
|
||||
// 判重只放名称和做法：关键设定、关掉时换成什么、延续哪个因子都不放。
|
||||
assert.deepEqual(dedup, {
|
||||
task: dedup.task,
|
||||
proposal_a: { factors: [{ name: "外圈环半径缩到 0.45", method: "把 construct_packing 里外圈环半径常数 0.7 改为 0.45。" }] },
|
||||
proposal_b: { factors: [{ name: "六行六边形网格", method: "用 6 行六边形网格摆圆心。" }] },
|
||||
});
|
||||
assert.deepEqual(freeze, {
|
||||
task: freeze.task,
|
||||
protocol: {
|
||||
factors: [
|
||||
{
|
||||
name: "外圈环半径缩到 0.45",
|
||||
method: "把 construct_packing 里外圈环半径常数 0.7 改为 0.45。",
|
||||
keySettings: "环半径 0.45",
|
||||
offBehavior: "回到 0.7",
|
||||
continues: null,
|
||||
},
|
||||
],
|
||||
safeguards: ["clip-radii"],
|
||||
body: "### 不许改什么\n\nharness.py 不改。",
|
||||
metrics: {
|
||||
defined: [
|
||||
{ id: "min-gap", meaning: "最小间隙", unit: "无", direction: "maximize", howMeasured: "最终解上所有圆对的最小间隙，测一次", missing: "recordMissing" },
|
||||
],
|
||||
reused: ["c1.overlap"],
|
||||
},
|
||||
clauses: ["外圈环半径常数改为 0.45（initial.py:36）"],
|
||||
},
|
||||
});
|
||||
assert.deepEqual(diff, {
|
||||
task: diff.task,
|
||||
protocol: {
|
||||
body: "### 不许改什么\n\nharness.py 不改。",
|
||||
metrics: [{ id: "min-gap", howMeasured: "最终解上所有圆对的最小间隙，测一次", missing: "recordMissing" }],
|
||||
},
|
||||
change: { path: "a.py", change: "added", diff: "@@ -0,0 +1 @@\n+x" },
|
||||
});
|
||||
assert.equal(new Set(states.map((s) => s.task)).size, 3);
|
||||
});
|
||||
|
||||
function tempDir(t: TestContext): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "inforsi-jev-"));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
return dir;
|
||||
}
|
||||
|
||||
test("把 workspace 的比较结果切成差异块，每块从 @@ 行起", (t) => {
|
||||
const root = tempDir(t);
|
||||
const repo = join(root, "v.git");
|
||||
createRepository(repo);
|
||||
const dir = join(root, "code");
|
||||
mkdirSync(dir);
|
||||
const lines = Array.from({ length: 30 }, (_, i) => `line ${i + 1}`);
|
||||
writeFileSync(join(dir, "a.py"), `${lines.join("\n")}\n`);
|
||||
writeFileSync(join(dir, "bin.dat"), Buffer.from([0, 1, 2]));
|
||||
const v1 = saveVersion(repo, dir, []);
|
||||
lines[1] = "changed 2";
|
||||
lines[25] = "changed 26";
|
||||
writeFileSync(join(dir, "a.py"), `${lines.join("\n")}\n`);
|
||||
writeFileSync(join(dir, "bin.dat"), Buffer.from([0, 9, 2]));
|
||||
writeFileSync(join(dir, "new.py"), "x = 1\n");
|
||||
const v2 = saveVersion(repo, dir, []);
|
||||
assert.ok(v1.saved && v2.saved);
|
||||
const hunks = codeHunks(compareVersions(repo, v1.version, v2.version));
|
||||
assert.deepEqual(
|
||||
hunks.map((h) => [h.path, h.change, h.text.split("\n")[0]]),
|
||||
[
|
||||
["a.py", "modified", "@@ -1,5 +1,5 @@"],
|
||||
["a.py", "modified", "@@ -23,7 +23,7 @@ line 22"],
|
||||
["new.py", "added", "@@ -0,0 +1 @@"],
|
||||
],
|
||||
);
|
||||
assert.equal(hunks[0]?.text, "@@ -1,5 +1,5 @@\n line 1\n-line 2\n+changed 2\n line 3\n line 4\n line 5");
|
||||
assert.equal(hunks[2]?.text, "@@ -0,0 +1 @@\n+x = 1");
|
||||
});
|
||||
@@ -33,14 +33,38 @@ test("模型调用：开始了没结束的被找出来，带着这次任务的
|
||||
ledger.append("callStarted", callStart("k1", "r1", "s1"));
|
||||
ledger.append("callStarted", callStart("k2", "e1"));
|
||||
ledger.append("callUsage", { call: "k1", reply: 1, usage });
|
||||
ledger.append("callEnded", { call: "k2", ending: "failed", reason: "接口超时" });
|
||||
ledger.append("callEnded", { call: "k2", wallSeconds: { status: "measured", value: 30 }, status: "failed", reason: "接口超时" });
|
||||
const calls = ledger.pending().calls;
|
||||
assert.deepEqual(calls.map((c) => c.content.call), ["k1"]);
|
||||
assert.deepEqual(calls[0]!.content.materials, callStart("k1").materials);
|
||||
ledger.append("callEnded", { call: "k1", ending: "completed" });
|
||||
ledger.append("callEnded", { call: "k1", wallSeconds: { status: "measured", value: 600 }, status: "notSubmitted", reason: "模型自己停下了" });
|
||||
assert.deepEqual(ledger.pending(), { runs: [], calls: [] });
|
||||
});
|
||||
|
||||
test("中断后恢复：给没结束的旧调用补写出错加测不到（进程中断），补写后不再算未完成", (t) => {
|
||||
const path = join(tempDir(t), "ledger.db");
|
||||
const ledger = Ledger.open(path);
|
||||
ledger.append("callStarted", callStart("k1", "r1", "s1"));
|
||||
ledger.append("callUsage", { call: "k1", reply: 1, usage });
|
||||
ledger.close(); // 控制进程被系统杀掉
|
||||
const reopened = Ledger.open(path);
|
||||
t.after(() => reopened.close());
|
||||
assert.deepEqual(reopened.pending().calls.map((c) => c.content.call), ["k1"]);
|
||||
reopened.append("callEnded", {
|
||||
call: "k1",
|
||||
wallSeconds: { status: "unmeasured", reason: "进程中断" },
|
||||
status: "failed",
|
||||
reason: "进程中断",
|
||||
});
|
||||
assert.deepEqual(reopened.pending(), { runs: [], calls: [] });
|
||||
assert.deepEqual(reopened.records().at(-1)!.content, {
|
||||
call: "k1",
|
||||
wallSeconds: { status: "unmeasured", reason: "进程中断" },
|
||||
status: "failed",
|
||||
reason: "进程中断",
|
||||
});
|
||||
});
|
||||
|
||||
test("关掉再打开：未完成的动作照样找得出来", (t) => {
|
||||
const path = join(tempDir(t), "ledger.db");
|
||||
const ledger = Ledger.open(path);
|
||||
|
||||
@@ -24,7 +24,7 @@ import {
|
||||
// 4. 这一轮消融已经有格确认启动了，才来关这一轮的解释阶段 → 拒收（方法.md 4.3 ⑭）。
|
||||
|
||||
function submitWrapUp(ledger: Ledger, stageId: string, author: string, version: number, body = "收尾") {
|
||||
return ledger.append("submission", { stageId, author, target: null, item: null, version, kind: "wrapUp", content: wrapUp(body) });
|
||||
return ledger.append("submission", { stageId, author, target: null, item: null, version, jev: null, kind: "wrapUp", content: wrapUp(body) });
|
||||
}
|
||||
|
||||
test("拒收 1：只收开着的阶段的提交：没开过的不收，关了以后不收、替换也不收；开着时收", (t) => {
|
||||
@@ -42,9 +42,9 @@ test("拒收 2 的正常情况：替换本人在这个阶段、对同一个对
|
||||
const { ledger } = freshLedger(t);
|
||||
openStage(ledger, "s1", "firstPrediction", 1);
|
||||
const slot = { stageId: "s1", author: "r1", target: "c1", item: null, kind: "firstPrediction" as const };
|
||||
ledger.append("submission", { ...slot, version: 1, content: prediction(range(1, 2, 3)) });
|
||||
ledger.append("submission", { ...slot, version: 2, content: prediction(range(1, 2, 4)) });
|
||||
ledger.append("submission", { ...slot, version: 3, content: prediction(range(1, 2, 5)) });
|
||||
ledger.append("submission", { ...slot, version: 1, jev: null, content: prediction(range(1, 2, 3)) });
|
||||
ledger.append("submission", { ...slot, version: 2, jev: null, content: prediction(range(1, 2, 4)) });
|
||||
ledger.append("submission", { ...slot, version: 3, jev: null, content: prediction(range(1, 2, 5)) });
|
||||
assert.equal(ledger.latestSubmission(slot)?.content.version, 3);
|
||||
});
|
||||
|
||||
@@ -53,13 +53,13 @@ test("拒收 2：要替换的那一份不是最新一版、不是本人的、不
|
||||
openStage(ledger, "s1", "firstPrediction", 1);
|
||||
openStage(ledger, "s2", "firstPrediction", 1);
|
||||
const slot = { stageId: "s1", author: "r1", target: "c1", item: null, kind: "firstPrediction" as const };
|
||||
ledger.append("submission", { ...slot, version: 1, content: prediction(range(1, 2, 3)) });
|
||||
ledger.append("submission", { ...slot, version: 2, content: prediction(range(1, 2, 4)) });
|
||||
const next = { ...slot, version: 3, content: prediction(range(1, 2, 6)) };
|
||||
ledger.append("submission", { ...slot, version: 1, jev: null, content: prediction(range(1, 2, 3)) });
|
||||
ledger.append("submission", { ...slot, version: 2, jev: null, content: prediction(range(1, 2, 4)) });
|
||||
const next = { ...slot, version: 3, jev: null, content: prediction(range(1, 2, 6)) };
|
||||
// 想替换第 1 版，但最新是第 2 版。
|
||||
assert.throws(() => ledger.append("submission", { ...slot, version: 2, content: prediction(range(0, 2, 4)) }), /最新是第 2 版/);
|
||||
assert.throws(() => ledger.append("submission", { ...slot, version: 2, jev: null, content: prediction(range(0, 2, 4)) }), /最新是第 2 版/);
|
||||
// 要替换的第 3 版不存在。
|
||||
assert.throws(() => ledger.append("submission", { ...slot, version: 4, content: prediction(range(1, 2, 6)) }), /最新是第 2 版/);
|
||||
assert.throws(() => ledger.append("submission", { ...slot, version: 4, jev: null, content: prediction(range(1, 2, 6)) }), /最新是第 2 版/);
|
||||
// 不是本人的、不是同一个对象的、不在这个阶段的：本人在那里还没有交过。
|
||||
assert.throws(() => ledger.append("submission", { ...next, author: "r2" }), /还没有交过/);
|
||||
assert.throws(() => ledger.append("submission", { ...next, target: "c2" }), /还没有交过/);
|
||||
@@ -71,10 +71,10 @@ test("拒收 2：同一个人对同一个候选的几份挑刺按各自的编号
|
||||
const { ledger } = freshLedger(t);
|
||||
openStage(ledger, "s1", "critique", 1);
|
||||
const slot = { stageId: "s1", author: "r1", target: "c1", kind: "critique" as const };
|
||||
ledger.append("submission", { ...slot, item: "k1", version: 1, content: critique("第一份") });
|
||||
ledger.append("submission", { ...slot, item: "k2", version: 1, content: critique("第二份") });
|
||||
ledger.append("submission", { ...slot, item: "k1", version: 2, content: critique("第一份改") });
|
||||
assert.throws(() => ledger.append("submission", { ...slot, item: "k3", version: 2, content: critique("x") }), /还没有交过/);
|
||||
ledger.append("submission", { ...slot, item: "k1", version: 1, jev: null, content: critique("第一份") });
|
||||
ledger.append("submission", { ...slot, item: "k2", version: 1, jev: null, content: critique("第二份") });
|
||||
ledger.append("submission", { ...slot, item: "k1", version: 2, jev: null, content: critique("第一份改") });
|
||||
assert.throws(() => ledger.append("submission", { ...slot, item: "k3", version: 2, jev: null, content: critique("x") }), /还没有交过/);
|
||||
});
|
||||
|
||||
test("拒收 2：实现、修复、核验、消融实现、消融核验只收第一版，不能替换", (t) => {
|
||||
@@ -95,8 +95,8 @@ test("拒收 2：实现、修复、核验、消融实现、消融核验只收第
|
||||
{ ...base, author: "v1", stageId: "aver", target: "cell1", kind: "ablationVerification" as const, content: { factors: [] } },
|
||||
];
|
||||
for (const c of cases) {
|
||||
ledger.append("submission", { ...c, version: 1 });
|
||||
assert.throws(() => ledger.append("submission", { ...c, version: 2 }), /不能替换/, c.kind);
|
||||
ledger.append("submission", { ...c, version: 1, jev: null });
|
||||
assert.throws(() => ledger.append("submission", { ...c, version: 2, jev: null }), /不能替换/, c.kind);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -105,14 +105,14 @@ test("拒收 2：版本号只在同一种提交里接：不能替换的实现不
|
||||
const { ledger } = freshLedger(t);
|
||||
openStage(ledger, "impl", "implementation", 1, ["e1"]);
|
||||
const implSlot = { stageId: "impl", author: "e1", target: "c1", item: null };
|
||||
ledger.append("submission", { ...implSlot, version: 1, kind: "implementation", content: implementation() });
|
||||
assert.throws(() => ledger.append("submission", { ...implSlot, version: 2, kind: "wrapUp", content: wrapUp() }), /还没有交过/);
|
||||
ledger.append("submission", { ...implSlot, version: 1, jev: null, kind: "implementation", content: implementation() });
|
||||
assert.throws(() => ledger.append("submission", { ...implSlot, version: 2, jev: null, kind: "wrapUp", content: wrapUp() }), /还没有交过/);
|
||||
assert.equal(ledger.latestSubmission({ ...implSlot, kind: "implementation" })?.content.kind, "implementation");
|
||||
|
||||
openStage(ledger, "mix", "proposal", 1);
|
||||
const mixSlot = { stageId: "mix", author: "r1", target: null, item: "x1" };
|
||||
ledger.append("submission", { ...mixSlot, version: 1, kind: "proposal", content: proposal(range(1, 2, 3)) });
|
||||
assert.throws(() => ledger.append("submission", { ...mixSlot, version: 2, kind: "critique", content: critique() }), /还没有交过/);
|
||||
ledger.append("submission", { ...mixSlot, version: 1, jev: null, kind: "proposal", content: proposal(range(1, 2, 3)) });
|
||||
assert.throws(() => ledger.append("submission", { ...mixSlot, version: 2, jev: null, kind: "critique", content: critique() }), /还没有交过/);
|
||||
assert.equal(ledger.latestSubmission({ ...mixSlot, kind: "critique" }), undefined);
|
||||
});
|
||||
|
||||
|
||||
+184
-17
@@ -3,7 +3,7 @@ import { join } from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { test } from "node:test";
|
||||
import { Ledger, type LedgerRecord } from "../src/ledger/index.ts";
|
||||
import { hash } from "../src/vocab/index.ts";
|
||||
import { hash, type JevJudgment } from "../src/vocab/index.ts";
|
||||
import {
|
||||
callStart,
|
||||
closeStage,
|
||||
@@ -111,8 +111,11 @@ test("唯一约束：同一次模型调用开始两次报错", (t) => {
|
||||
test("唯一约束：同一次模型调用结束两次报错", (t) => {
|
||||
const { ledger, path } = freshLedger(t);
|
||||
ledger.append("callStarted", callStart("k1"));
|
||||
ledger.append("callEnded", { call: "k1", ending: "completed" });
|
||||
assert.throws(() => ledger.append("callEnded", { call: "k1", ending: "failed", reason: "又报了一次" }), /call_ended_once/);
|
||||
ledger.append("callEnded", { call: "k1", wallSeconds: { status: "measured", value: 12.5 }, status: "submitted" });
|
||||
assert.throws(
|
||||
() => ledger.append("callEnded", { call: "k1", wallSeconds: { status: "measured", value: 3 }, status: "failed", reason: "又报了一次" }),
|
||||
/call_ended_once/,
|
||||
);
|
||||
assert.equal(countRows(path), 2);
|
||||
});
|
||||
|
||||
@@ -162,19 +165,19 @@ test("内容不符合定义时不写入：多出的字段、提交内容不合
|
||||
openStage(ledger, "s1", "wrapUp", 1);
|
||||
assert.throws(() => ledger.append("runStarted", { run: "r1", extra: 1 } as never), /runStarted/);
|
||||
assert.throws(
|
||||
() => ledger.append("submission", { stageId: "s1", author: "r1", target: null, item: null, version: 1, kind: "wrapUp", content: { body: "" } }),
|
||||
() => ledger.append("submission", { stageId: "s1", author: "r1", target: null, item: null, version: 1, jev: null, kind: "wrapUp", content: { body: "" } }),
|
||||
/wrapUp/,
|
||||
);
|
||||
// 提案、挑刺一人能交好几份，必须带编号；实现一人对一个候选只交一份，编号必须为空，
|
||||
// 否则另起一个编号就能绕开"实现不能替换"。
|
||||
openStage(ledger, "prop", "proposal", 1);
|
||||
assert.throws(
|
||||
() => ledger.append("submission", { stageId: "prop", author: "r1", target: null, item: null as never, version: 1, kind: "proposal", content: proposal(range(1, 2, 3)) }),
|
||||
() => ledger.append("submission", { stageId: "prop", author: "r1", target: null, item: null as never, version: 1, jev: null, kind: "proposal", content: proposal(range(1, 2, 3)) }),
|
||||
/proposal/,
|
||||
);
|
||||
openStage(ledger, "impl", "implementation", 1, ["e1"]);
|
||||
assert.throws(
|
||||
() => ledger.append("submission", { stageId: "impl", author: "e1", target: "c1", item: "x" as never, version: 1, kind: "implementation", content: implementation() }),
|
||||
() => ledger.append("submission", { stageId: "impl", author: "e1", target: "c1", item: "x" as never, version: 1, jev: null, kind: "implementation", content: implementation() }),
|
||||
/implementation/,
|
||||
);
|
||||
assert.equal(countRows(path), 3); // 三条开阶段
|
||||
@@ -184,7 +187,7 @@ test("写提案：作者自评拆成一条封存，提案记录里不带它", (t
|
||||
const { ledger } = freshLedger(t);
|
||||
openStage(ledger, "s1", "proposal", 1);
|
||||
const record = ledger.append("submission", {
|
||||
stageId: "s1", author: "r1", target: null, item: "r1-1", version: 1, kind: "proposal", content: proposal(range(1, 2, 3)),
|
||||
stageId: "s1", author: "r1", target: null, item: "r1-1", version: 1, kind: "proposal", content: proposal(range(1, 2, 3)), jev: null,
|
||||
});
|
||||
const all = ledger.records();
|
||||
const stored = all.find((r) => r.kind === "submission")!;
|
||||
@@ -197,17 +200,174 @@ test("写提案：作者自评拆成一条封存，提案记录里不带它", (t
|
||||
assert.equal(seal.seq, record.seq + 1);
|
||||
});
|
||||
|
||||
test("退回：被退回的内容原样存，带退回理由；来自 Jev 的带原始概率和结论", (t) => {
|
||||
const { ledger } = freshLedger(t);
|
||||
// Jev 判断的样子照 src/vocab/jev.ts 的 JevJudgment 手写：问题名、问题全文哈希（64 位十六进制）、阈值、原始概率、结论、请求编号。
|
||||
const judgment = (question: string, probability: number, threshold: number, requestId: string): JevJudgment => ({
|
||||
question,
|
||||
questionHash: "ab".repeat(32),
|
||||
threshold,
|
||||
probability,
|
||||
yes: probability > threshold,
|
||||
requestId,
|
||||
});
|
||||
|
||||
test("退回：被退回的内容原样存；程序检查退回的 jev 为 null，Jev 退回的原样存它用到的几条判断", (t) => {
|
||||
const path = join(tempDir(t), "ledger.db");
|
||||
const ledger = Ledger.open(path);
|
||||
const malformed = { factors: "不是数组" };
|
||||
// 冻结条款一次问三问，退回时三条判断都存，包括判"否"的两条。
|
||||
const clauses = [
|
||||
judgment("changes_measurement", 0.97, 0.5, "req-1"),
|
||||
judgment("self_contradictory", 0.02, 0.6, "req-2"),
|
||||
judgment("too_vague", 0.1, 0.7, "req-3"),
|
||||
];
|
||||
ledger.append("rejection", { stageId: "s1", author: "r1", kind: "proposal", submitted: malformed, reason: "因子清单格式不对", jev: null });
|
||||
ledger.append("rejection", {
|
||||
stageId: "s6", author: "r2", kind: "response", submitted: { body: "改了评测" }, reason: "冻结条款改动了评测",
|
||||
jev: { probabilities: { yes: 0.97, no: 0.03 }, conclusion: "changesEvaluation" },
|
||||
stageId: "s6", author: "r2", kind: "response", submitted: { body: "改了评测" }, reason: "冻结条款改动了评测", jev: clauses,
|
||||
});
|
||||
const rejections = ledger.records().filter((r) => r.kind === "rejection");
|
||||
ledger.close();
|
||||
const reopened = Ledger.open(path);
|
||||
t.after(() => reopened.close());
|
||||
const rejections = reopened.records().filter((r) => r.kind === "rejection");
|
||||
assert.deepEqual(rejections.map((r) => r.content.submitted), [malformed, { body: "改了评测" }]);
|
||||
assert.deepEqual(rejections[1]!.content.jev, { probabilities: { yes: 0.97, no: 0.03 }, conclusion: "changesEvaluation" });
|
||||
assert.equal(rejections[0]!.content.jev, null);
|
||||
assert.deepEqual(rejections[1]!.content.jev, [
|
||||
{ question: "changes_measurement", questionHash: "ab".repeat(32), threshold: 0.5, probability: 0.97, yes: true, requestId: "req-1" },
|
||||
{ question: "self_contradictory", questionHash: "ab".repeat(32), threshold: 0.6, probability: 0.02, yes: false, requestId: "req-2" },
|
||||
{ question: "too_vague", questionHash: "ab".repeat(32), threshold: 0.7, probability: 0.1, yes: false, requestId: "req-3" },
|
||||
]);
|
||||
});
|
||||
|
||||
test("退回：jev 不合 JevJudgment 的拒收，一条都不写", (t) => {
|
||||
const { ledger, path } = freshLedger(t);
|
||||
const good = judgment("same_change", 0.9, 0.5, "req-1");
|
||||
const { requestId: _, ...withoutRequest } = good;
|
||||
const bad: unknown[] = [
|
||||
[], // 空数组：Jev 退回至少有一条判断，程序退回写 null
|
||||
{ probabilities: { yes: 0.97, no: 0.03 }, conclusion: "changesEvaluation" }, // 旧写法：不是数组
|
||||
good, // 一条判断也要放进数组
|
||||
[{ ...good, probability: 1.2 }], // 概率超出 0 到 1
|
||||
[{ ...good, threshold: -0.1 }], // 阈值超出 0 到 1
|
||||
[withoutRequest], // 缺请求编号
|
||||
[{ ...good, requestId: "" }], // 请求编号为空
|
||||
[{ ...good, questionHash: "abc" }], // 哈希不是 64 位十六进制
|
||||
[{ ...good, question: "Same Change" }], // 问题名格式不对
|
||||
[{ ...good, yes: "true" }], // 结论不是布尔值
|
||||
[{ ...good, note: "多写一项" }], // 不认识的字段
|
||||
];
|
||||
for (const jev of bad) {
|
||||
assert.throws(
|
||||
() => ledger.append("rejection", { stageId: "s1", author: "r1", kind: "proposal", submitted: {}, reason: "判为同一个改法", jev } as never),
|
||||
/账本记录"rejection"的内容不符合定义/,
|
||||
JSON.stringify(jev),
|
||||
);
|
||||
}
|
||||
assert.equal(countRows(path), 0);
|
||||
});
|
||||
|
||||
test("模型调用结束：四种结局各存一条，墙钟时间（秒）测到带值、测不到带原因，关掉再打开读出不变", (t) => {
|
||||
const path = join(tempDir(t), "ledger.db");
|
||||
const ledger = Ledger.open(path);
|
||||
const endings = [
|
||||
{ call: "k1", wallSeconds: { status: "measured", value: 812.25 }, status: "submitted" },
|
||||
{ call: "k2", wallSeconds: { status: "measured", value: 95 }, status: "notSubmitted", reason: "模型自己停下了，一份都没交" },
|
||||
{ call: "k3", wallSeconds: { status: "unmeasured", reason: "进程中断" }, status: "failed", reason: "进程中断，恢复时补写" },
|
||||
{ call: "k4", wallSeconds: { status: "measured", value: 0 }, status: "stopped" },
|
||||
] as const;
|
||||
for (const e of endings) {
|
||||
ledger.append("callStarted", callStart(e.call));
|
||||
ledger.append("callEnded", e);
|
||||
}
|
||||
ledger.close();
|
||||
const reopened = Ledger.open(path);
|
||||
t.after(() => reopened.close());
|
||||
const ended = reopened.records().filter((r) => r.kind === "callEnded").map((r) => r.content);
|
||||
assert.deepEqual(ended, [
|
||||
{ call: "k1", wallSeconds: { status: "measured", value: 812.25 }, status: "submitted" },
|
||||
{ call: "k2", wallSeconds: { status: "measured", value: 95 }, status: "notSubmitted", reason: "模型自己停下了，一份都没交" },
|
||||
{ call: "k3", wallSeconds: { status: "unmeasured", reason: "进程中断" }, status: "failed", reason: "进程中断，恢复时补写" },
|
||||
{ call: "k4", wallSeconds: { status: "measured", value: 0 }, status: "stopped" },
|
||||
]);
|
||||
assert.deepEqual(reopened.pending().calls, []);
|
||||
});
|
||||
|
||||
test("模型调用结束：结局和原因对不上、墙钟时间缺失或不合 Measured、测到的值为负，都拒收", (t) => {
|
||||
const { ledger, path } = freshLedger(t);
|
||||
const s3 = { status: "measured", value: 3 };
|
||||
const bad: unknown[] = [
|
||||
{ call: "k1", wallSeconds: s3, status: "notSubmitted" }, // 没交要带原因
|
||||
{ call: "k1", wallSeconds: s3, status: "failed" }, // 出错要带原因
|
||||
{ call: "k1", wallSeconds: s3, status: "failed", reason: "" }, // 原因不能为空
|
||||
{ call: "k1", wallSeconds: s3, status: "submitted", reason: "交了" }, // 交了不带原因
|
||||
{ call: "k1", wallSeconds: s3, status: "stopped", reason: "用户停下" }, // 被停下不带原因
|
||||
{ call: "k1", wallSeconds: s3, status: "completed" }, // 旧写法
|
||||
{ call: "k1", status: "submitted" }, // 缺墙钟时间
|
||||
{ call: "k1", wallSeconds: 3, status: "submitted" }, // 裸数字，不是 Measured
|
||||
{ call: "k1", wallSeconds: null, status: "submitted" }, // 测不到不能写 null
|
||||
{ call: "k1", wallSeconds: { status: "measured", value: "3" }, status: "submitted" }, // 值不是数
|
||||
{ call: "k1", wallSeconds: { status: "measured" }, status: "submitted" }, // 测到却没带值
|
||||
{ call: "k1", wallSeconds: { status: "unmeasured" }, status: "failed", reason: "进程中断" }, // 测不到却没带原因
|
||||
{ call: "k1", wallSeconds: { status: "unmeasured", reason: "" }, status: "failed", reason: "进程中断" }, // 原因为空
|
||||
{ call: "k1", wallSeconds: { status: "unmeasured", reason: "进程中断", value: 0 }, status: "failed", reason: "进程中断" }, // 测不到又带值
|
||||
];
|
||||
for (const content of bad) {
|
||||
assert.throws(() => ledger.append("callEnded", content as never), /账本记录"callEnded"的内容不符合定义/, JSON.stringify(content));
|
||||
}
|
||||
assert.equal(countRows(path), 0);
|
||||
});
|
||||
|
||||
test("提交：jev 在外层与 content 并列，null 或 JevJudgment 数组都能存能读；提案拆封存不受影响", (t) => {
|
||||
const path = join(tempDir(t), "ledger.db");
|
||||
const ledger = Ledger.open(path);
|
||||
openStage(ledger, "s1", "proposal", 1);
|
||||
openStage(ledger, "s9", "implementation", 1);
|
||||
// 代码改动五问都判"否"才收下，五条判断连同原始概率都存。
|
||||
const checks = [
|
||||
judgment("unseeded_randomness", 0.01, 0.5, "req-1"),
|
||||
judgment("reads_clock", 0.02, 0.5, "req-2"),
|
||||
judgment("touches_measurement", 0.2, 0.6, "req-3"),
|
||||
judgment("early_exit", 0.05, 0.5, "req-4"),
|
||||
judgment("zero_for_unmeasured", 0.3, 0.7, "req-5"),
|
||||
];
|
||||
const p = ledger.append("submission", { stageId: "s1", author: "r1", target: null, item: "r1-1", version: 1, kind: "proposal", content: proposal(range(1, 2, 3)), jev: null });
|
||||
ledger.append("submission", { stageId: "s9", author: "e1", target: "c1", item: null, version: 1, kind: "implementation", content: implementation(), jev: checks });
|
||||
ledger.close();
|
||||
const reopened = Ledger.open(path);
|
||||
t.after(() => reopened.close());
|
||||
const all = reopened.records();
|
||||
const subs = all.filter((r) => r.kind === "submission");
|
||||
assert.deepEqual(subs.map((r) => r.content.jev), [
|
||||
null,
|
||||
[
|
||||
{ question: "unseeded_randomness", questionHash: "ab".repeat(32), threshold: 0.5, probability: 0.01, yes: false, requestId: "req-1" },
|
||||
{ question: "reads_clock", questionHash: "ab".repeat(32), threshold: 0.5, probability: 0.02, yes: false, requestId: "req-2" },
|
||||
{ question: "touches_measurement", questionHash: "ab".repeat(32), threshold: 0.6, probability: 0.2, yes: false, requestId: "req-3" },
|
||||
{ question: "early_exit", questionHash: "ab".repeat(32), threshold: 0.5, probability: 0.05, yes: false, requestId: "req-4" },
|
||||
{ question: "zero_for_unmeasured", questionHash: "ab".repeat(32), threshold: 0.7, probability: 0.3, yes: false, requestId: "req-5" },
|
||||
],
|
||||
]);
|
||||
assert.equal("jev" in (subs[1]!.content.content as object), false);
|
||||
const seal = all.find((r) => r.kind === "seal")!;
|
||||
assert.deepEqual(seal.content, { proposal: p.seq, range: range(1, 2, 3) });
|
||||
assert.equal(seal.seq, p.seq + 1);
|
||||
});
|
||||
|
||||
test("提交：缺 jev、jev 不合 JevJudgment 数组、jev 写进 content 里，都拒收", (t) => {
|
||||
const { ledger, path } = freshLedger(t);
|
||||
openStage(ledger, "s1", "wrapUp", 1);
|
||||
const good = judgment("same_change", 0.1, 0.5, "req-1");
|
||||
const base = { stageId: "s1", author: "r1", target: null, item: null, version: 1, kind: "wrapUp", content: { body: "收尾" } };
|
||||
const bad: unknown[] = [
|
||||
base, // 缺 jev：没用到 Jev 要写 null，不靠省略
|
||||
{ ...base, jev: [] }, // 空数组
|
||||
{ ...base, jev: good }, // 一条也要放进数组
|
||||
{ ...base, jev: [{ ...good, probability: -0.1 }] }, // 概率超出 0 到 1
|
||||
{ ...base, jev: [{ ...good, extra: 1 }] }, // 不认识的字段
|
||||
{ ...base, jev: null, content: { body: "收尾", jev: null } }, // jev 不进 content
|
||||
];
|
||||
for (const content of bad) {
|
||||
assert.throws(() => ledger.append("submission", content as never), /账本记录"submission（提交种类 wrapUp）"的内容不符合定义/, JSON.stringify(content));
|
||||
}
|
||||
assert.equal(countRows(path), 1);
|
||||
});
|
||||
|
||||
test("事务：中途出错，一条都没写进去；中途被拒收，前面写的也撤销；正常结束则全部写进去", (t) => {
|
||||
@@ -253,7 +413,7 @@ test("关掉再打开：读出的记录不变，序号接着往下排", (t) => {
|
||||
const ledger = Ledger.open(path);
|
||||
ledger.append("config", { researchers: ["r1", "r2"] });
|
||||
openStage(ledger, "s1", "proposal", 1);
|
||||
ledger.append("submission", { stageId: "s1", author: "r1", target: null, item: "r1-1", version: 1, kind: "proposal", content: proposal(range(1, 2, 3)) });
|
||||
ledger.append("submission", { stageId: "s1", author: "r1", target: null, item: "r1-1", version: 1, jev: null, kind: "proposal", content: proposal(range(1, 2, 3)) });
|
||||
ledger.append("note", { author: "r2", body: "笔记" });
|
||||
closeStage(ledger, "s1");
|
||||
planRun(ledger, "run1", "forward", 1);
|
||||
@@ -276,6 +436,13 @@ test("打开：表结构版本不对报错；不是空文件也不是账本的
|
||||
db.close();
|
||||
assert.throws(() => Ledger.open(wrongVersion), /版本是 99/);
|
||||
|
||||
// 版本 2 是"模型调用结束"还没有墙钟时间、退回还没有 Jev 判断数组时的账本，不迁移，打开报错。
|
||||
const previous = join(dir, "v2.db");
|
||||
const v2 = new DatabaseSync(previous);
|
||||
v2.exec("PRAGMA user_version = 2");
|
||||
v2.close();
|
||||
assert.throws(() => Ledger.open(previous), /版本是 2.*只认版本 3/s);
|
||||
|
||||
const foreign = join(dir, "foreign.db");
|
||||
const other = new DatabaseSync(foreign);
|
||||
other.exec("CREATE TABLE t (x INTEGER)");
|
||||
@@ -311,9 +478,9 @@ test("查询：阶段关了没有、某份提交的最新一版（同一人对
|
||||
assert.equal(ledger.stageClosed("s1"), false);
|
||||
const slot = { stageId: "s1", author: "r1", kind: "critique" as const, target: "c1", item: "k1" };
|
||||
assert.equal(ledger.latestSubmission(slot), undefined);
|
||||
ledger.append("submission", { ...slot, version: 1, content: { target: "whole", body: "第一版" } });
|
||||
const v2 = ledger.append("submission", { ...slot, version: 2, content: { target: "whole", body: "第二版" } });
|
||||
ledger.append("submission", { ...slot, item: "k2", version: 1, content: { target: "p1", body: "另一份" } });
|
||||
ledger.append("submission", { ...slot, version: 1, jev: null, content: { target: "whole", body: "第一版" } });
|
||||
const v2 = ledger.append("submission", { ...slot, version: 2, jev: null, content: { target: "whole", body: "第二版" } });
|
||||
ledger.append("submission", { ...slot, item: "k2", version: 1, jev: null, content: { target: "p1", body: "另一份" } });
|
||||
assert.deepEqual(ledger.latestSubmission(slot), v2);
|
||||
closeStage(ledger, "s1");
|
||||
assert.equal(ledger.stageClosed("s1"), true);
|
||||
|
||||
Reference in New Issue
Block a user