feat: 任务书 05，runner 与 tasks 两个模块
runner 用 Docker 跑一条命令并交回结局和资源用量；tasks 读 task.json、准备运行目录、给出正式运行/校验器/轻量检查的运行定义，并从原始产物独立判分。任务目录从参考项目搬到 tasks/circle_packing/，新增 task.json；vocab 的 VERDICTS 删掉"判不了"，ProcessResult 的 cancelled 改为"容器从外面被杀了"。 全新上下文审阅后的两处修复：计量 shell 以容器内 root 运行，命令经 setpriv 降权，结果写在容器自己的文件层、结束后 docker cp 取出，并加 no-new-privileges，候选无法伪造计量结果把超时记成正常退出；harness 开跑前删掉旧的 solution.json、metrics.json，旧产物不会被当成本次的值。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+6
-1
@@ -15,6 +15,7 @@
|
|||||||
- 任务书 02（rules）：src/rules 13 个文件、1761 行，`npm test` 192 条通过。预期值来源有三个：方法.md 的手算例；`tests/fixtures/rules/` 下不看实现写的 Python 脚本（重跑后 JSON 逐字节相同）；参考项目 calc15、calc14b 里与现行方法一致的部分。审阅做了两轮，每轮都换一个全新上下文的子 Agent。结构.md 第八部分补了 rules 对外内容。
|
- 任务书 02（rules）：src/rules 13 个文件、1761 行，`npm test` 192 条通过。预期值来源有三个：方法.md 的手算例；`tests/fixtures/rules/` 下不看实现写的 Python 脚本（重跑后 JSON 逐字节相同）；参考项目 calc15、calc14b 里与现行方法一致的部分。审阅做了两轮，每轮都换一个全新上下文的子 Agent。结构.md 第八部分补了 rules 对外内容。
|
||||||
- 任务书 03（ledger）：src/ledger 5 个文件、584 行，`npm test` 226 条通过（ledger 34 条）。一张只追加的表，两条禁改禁删触发器，8 处唯一约束（阶段开、阶段关、运行登记、确认启动或确认没启动、运行结果、模型调用开始、结束、消融结算），写入时只查四种拒收，提案的作者自评拆成封存另存。按第一版写过可见性和通用唯一键，按新版全部清掉了。全新上下文的审阅做了三轮。结构.md 第八部分补了 ledger 对外内容。
|
- 任务书 03（ledger）：src/ledger 5 个文件、584 行，`npm test` 226 条通过（ledger 34 条）。一张只追加的表，两条禁改禁删触发器，8 处唯一约束（阶段开、阶段关、运行登记、确认启动或确认没启动、运行结果、模型调用开始、结束、消融结算），写入时只查四种拒收，提案的作者自评拆成封存另存。按第一版写过可见性和通用唯一键，按新版全部清掉了。全新上下文的审阅做了三轮。结构.md 第八部分补了 ledger 对外内容。
|
||||||
- 任务书 04（config）：src/config 4 个文件、453 行，`npm test` 246 条通过（config 19 条）。流程用联合类型（完整流程 / 朴素循环），规则名取自 rules 登记表，config 改为依赖 rules。ledger 顺手两处：结算按轮次唯一、笔记去掉版本号，表结构版本号升到 2，全新上下文审阅一轮无问题。结构.md 第八部分补了 config 对外内容。
|
- 任务书 04（config）：src/config 4 个文件、453 行，`npm test` 246 条通过（config 19 条）。流程用联合类型（完整流程 / 朴素循环），规则名取自 rules 登记表，config 改为依赖 rules。ledger 顺手两处：结算按轮次唯一、笔记去掉版本号，表结构版本号升到 2，全新上下文审阅一轮无问题。结构.md 第八部分补了 config 对外内容。
|
||||||
|
- 任务书 05（tasks 与 runner）：src/runner 2 个文件、271 行，src/tasks 6 个文件、691 行，`npm test` 274 条通过（runner 12 条、tasks 16 条，真实 Docker，整套约 12 秒）。任务目录搬到 `tasks/circle_packing/`，新增 task.json；`VERDICTS` 删掉"判不了"。基线从头到尾判为算数，分数 0.9597642169962064，两次逐位相同。全新上下文审阅两轮：第一轮发现候选能伪造计量结果（把超时记成正常退出）、运行目录里旧 metrics.json 会被当成本次产物，都已修（计量 shell 以容器内 root 跑、命令用 setpriv 降权、结果用 docker cp 取；harness 开跑前删旧输出）；第二轮只审这次修复，另采纳一条：容器加 no-new-privileges，禁止候选靠 setuid 程序重新拿到 root。结构.md 第八部分补了 runner、tasks。
|
||||||
|
|
||||||
## 关键决定（用户原话见各提交说明）
|
## 关键决定（用户原话见各提交说明）
|
||||||
|
|
||||||
@@ -66,6 +67,10 @@
|
|||||||
- 给 agents、runner："整个环境的用量"（实验员会话的 CPU、内存等）写这两个模块时再定记录种类。
|
- 给 agents、runner："整个环境的用量"（实验员会话的 CPU、内存等）写这两个模块时再定记录种类。
|
||||||
- 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。
|
- 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。
|
||||||
- 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。
|
- 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。
|
||||||
|
- 给 submissions：研究者登记的假设专用指标编号不能和常规辅助指标重名（`metricDefinitions` 里的 7 个），否则候选自报的值会顶替宿主测的值（任务书 05 审阅意见）。
|
||||||
|
- 给 flow：容器名要在本机所有研究之间唯一（例如带研究编号）；`docker run` 因重名失败时 runner 按"启动了"处理。日志目录每次运行一个新的。
|
||||||
|
- 给以后换任务镜像：镜像里必须有 `setpriv`（runner 用它把命令降到当前用户）；容器里不要用 `python -I`，它会忽略 `PYTHON_CPU_COUNT`。
|
||||||
|
- 未处理的可能情况（没有真实证据，只列出）：Docker 守护进程卡住不应答时 docker 命令没有超时，会一直等；候选写出大到算术溢出的有限数时，validate.py 会异常退出，被当成宿主工具坏了而停下。
|
||||||
|
|
||||||
## 工作方式（用户 2026-10-01 定）
|
## 工作方式（用户 2026-10-01 定）
|
||||||
|
|
||||||
@@ -78,4 +83,4 @@
|
|||||||
|
|
||||||
## 下一步
|
## 下一步
|
||||||
|
|
||||||
任务书 05（tasks 与 runner，`任务书/05-tasks-runner.md`）已写好，等用户另开新 Agent 执行；完成后监督者核对。
|
任务书 05 已完成，等监督者核对。
|
||||||
|
|||||||
+45
-2
@@ -235,8 +235,8 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
|||||||
**运行**
|
**运行**
|
||||||
|
|
||||||
- `RunDefinition`（字段定义）：tasks 交给 runner 的运行定义，写明命令、工作目录、按 sha256 钉死的镜像和运行时、时限、核数和环境变量；候选进程不继承宿主的环境变量。
|
- `RunDefinition`（字段定义）：tasks 交给 runner 的运行定义，写明命令、工作目录、按 sha256 钉死的镜像和运行时、时限、核数和环境变量；候选进程不继承宿主的环境变量。
|
||||||
- `ProcessResult`（字段定义）：runner 交出的进程层面结果，分没启动、正常退出、超时、被取消四种；启动过的带资源用量。
|
- `ProcessResult`（字段定义）：runner 交出的进程层面结果，分没启动、正常退出、超时、从外面被杀四种；启动过的带资源用量，从外面被杀的三项资源都记测不到。
|
||||||
- `VERDICTS`、`Verdict`：一次运行的判定，算数、不算数、判不了三选一。
|
- `VERDICTS`、`Verdict`：一次运行的判定，算数、不算数二选一。
|
||||||
- `ANSWER_PATH_KEY`、`SAFEGUARD_TRIGGERS_KEY`：候选在指标文件里多写的两个保留键名，`answer-path` 和 `safeguard-triggers`。
|
- `ANSWER_PATH_KEY`、`SAFEGUARD_TRIGGERS_KEY`：候选在指标文件里多写的两个保留键名，`answer-path` 和 `safeguard-triggers`。
|
||||||
- `AnswerPath`（字段定义）：读到的答案路径原样字符串，或读不到及原因；在不在位置表里由拿得到位置表的模块现算。
|
- `AnswerPath`（字段定义）：读到的答案路径原样字符串，或读不到及原因；在不在位置表里由拿得到位置表的模块现算。
|
||||||
- `SafeguardTriggers`（字段定义）：读到的"保障短名 → 动手次数"原样内容，或读不到及原因。
|
- `SafeguardTriggers`（字段定义）：读到的"保障短名 → 动手次数"原样内容，或读不到及原因。
|
||||||
@@ -437,3 +437,46 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
|||||||
- `RuleRef`（类型）：决策点用的规则，名字和版本。
|
- `RuleRef`（类型）：决策点用的规则，名字和版本。
|
||||||
- `STUDY_STEPS`、`StudyStepId`：整份研究的步骤，按先后排。
|
- `STUDY_STEPS`、`StudyStepId`：整份研究的步骤，按先后排。
|
||||||
- `ROUND_STEPS`、`RoundStepId`：每一轮的步骤，按先后排；比第三部分拆得更细（例如"回应与统一"和"协议冻结"分成两步，挑刺与回应关掉时协议照样冻结）。
|
- `ROUND_STEPS`、`RoundStepId`：每一轮的步骤，按先后排；比第三部分拆得更细（例如"回应与统一"和"协议冻结"分成两步，挑刺与回应关掉时协议照样冻结）。
|
||||||
|
|
||||||
|
### runner
|
||||||
|
|
||||||
|
下面是 `src/runner/index.ts` 导出的全部内容。runner 用 docker 命令行做事，不碰账本，自己也不存状态：依据都在 Docker 里，所以新进程里新建的 `Runner` 能接着等早先启动的容器。Docker 不应答时，查状态答"问不到"，其余函数直接报错，都不重试。
|
||||||
|
|
||||||
|
- `new Runner(dockerHost)`：Docker 守护进程的地址必须给出；写 null 表示不加 `-H`，由 docker 命令按自己的环境变量和 context 决定。
|
||||||
|
- `runner.buildImage(contextDir, tag)`：用目录里的 Dockerfile 建镜像，交回镜像编号（`docker image inspect -f '{{.Id}}'` 的值）。
|
||||||
|
- `runner.hasImage(digest)`：本机有没有这个镜像编号。
|
||||||
|
- `runner.start(name, definition, runDir)`：起一个容器，运行目录挂到运行定义的工作目录；容器里以 root 运行的计量 shell 把命令降到宿主当前用户再跑，结局、退出码、墙钟、CPU 秒、峰值内存写在容器内只有 root 可写的目录里，命令改不了。`docker run` 失败时按名字再查一次，容器不存在或建了没启动算没启动（建了的删掉），在跑或已结束算启动了。
|
||||||
|
- `runner.wait(name, logDir)`：等容器结束，交回 `ProcessResult`，把标准输出、标准错误和从容器里拷出的结果文件存成日志目录里的 `stdout.log`、`stderr.log`、`usage`。有结果文件按文件定正常退出或超时；没有就是从外面被杀。
|
||||||
|
- `runner.status(name)`：没有这个容器、建了没启动、在跑、已结束、问不到，五种之一。
|
||||||
|
- `runner.remove(name)`：删掉一个已结束或没启动的容器，flow 写完结果后调用。
|
||||||
|
- `ContainerStatus`、`StartOutcome`（类型）：查状态的五种回答；启动了，或没启动和 Docker 的报错原文。
|
||||||
|
|
||||||
|
容器名由调用方给，必须是确定的（恢复时按名字去问）；runner 只检查它符合 Docker 的命名要求。没有取消函数：方法里没有程序主动取消一次已启动运行的场合。
|
||||||
|
|
||||||
|
### tasks
|
||||||
|
|
||||||
|
下面是 `src/tasks/index.ts` 导出的全部内容。tasks 不起子进程：凡是要执行任务代码的事（正式运行、判分时跑校验器、轻量检查）都只交出 `RunDefinition`，由调用方交给 runner 在任务镜像里跑，再把产物交回这里解读。任务种类到实现是 `task.ts` 里的一张对照表，现在只有填圆。各目录（运行目录、判分目录、检查目录）都由调用方建好、传空目录进来。
|
||||||
|
|
||||||
|
**任务事实**
|
||||||
|
|
||||||
|
- `TASK_KINDS`、`TaskKind`：任务种类名单，现在只有 `circle_packing`。
|
||||||
|
- `TaskFacts`（字段定义）：task.json 的字段：任务种类、最终指标的编号和方向、ε、已知最好（没有写 null）、单次时限、核数、Docker 运行时、不许改的文件名单。全部必填，不认识的字段报错。
|
||||||
|
- `checkTaskFacts(value)`：严格检查一份已解析的 task.json，一次列出全部错误，每条写成"字段路径：原因"。
|
||||||
|
- `Task`（类型）：读进来的一个任务：task.json 的内容、任务目录（Dockerfile 所在，给 runner 建镜像）、基线目录、TASK.md 全文。
|
||||||
|
- `loadTask(taskDir)`：读任务目录，检查 task.json，交回 `Task`。
|
||||||
|
- `metricDefinitions(task)`：最终指标和常规辅助指标的定义（墙钟、CPU 秒、峰值内存、最小间隙、原分数、原违反量、扣掉的分）；辅助指标的定义写在填圆的代码里。
|
||||||
|
- `protectedChanges(task, changedFiles)`：改动过的文件里属于"不许改"名单的那些。
|
||||||
|
|
||||||
|
**运行**
|
||||||
|
|
||||||
|
- `prepareRun(task, candidateDir, runDir, hypothesisMetrics)`：把候选代码、任务的 harness 和这次要测的假设专用指标说明（metrics-spec.json）放进运行目录。
|
||||||
|
- `formalRun(task, imageDigest, seed)`：正式运行的运行定义；种子由调用方给，同一份研究的全部运行用同一个，经环境变量交给 harness。
|
||||||
|
- `lightCheck(task, candidateDir, checkDir, imageDigest, seconds)`：把候选代码和检查脚本放进检查目录，交回轻量检查的运行定义；只看语法和入口函数，不执行候选代码，次数上限不归这里管。
|
||||||
|
- `readLightCheck(task, checkDir, result)`：解读轻量检查：通过，或不通过及原因；检查脚本自己没正常跑完时报错停下。
|
||||||
|
- `LightCheckOutcome`（类型）：轻量检查的结论。
|
||||||
|
|
||||||
|
**判分**
|
||||||
|
|
||||||
|
- `prepareJudging(task, runDir, judgeDir, imageDigest, formal, hypothesisMetrics)`：判分第一步。正式运行超时、从外面被杀、退出码非 0，或者没有产物，直接交回不算数的判分结果；否则把产物和宿主保管的校验器复制进判分目录，交回校验器的运行定义。
|
||||||
|
- `judge(task, runDir, judgeDir, formal, validator, hypothesisMetrics)`：判分第二步。读校验器写的判定和候选写的指标文件，加上正式运行的结果，交回 `JudgedResult`：校验器判合法（含合法化后合法）算数；判不合法、或"缺了整次作废"的假设专用指标测不到，不算数。合法化的三个数既进 `extra`，也是常规辅助指标的取值；测不到的指标记测不到并附原因。校验器自己没正常跑完、判定文件认不出、修后分数高于原分数时报错停下，不当成候选的失败。
|
||||||
|
- `JudgingStep`（类型）：判分第一步的结果，已判定或要跑校验器。
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
// runner：用 Docker 把一条命令跑完，交回它怎么结束的、用了多少资源（docs/结构.md 第二部分）。
|
||||||
|
export { Runner, type ContainerStatus, type StartOutcome } from "./runner.ts";
|
||||||
@@ -0,0 +1,269 @@
|
|||||||
|
import { spawn } from "node:child_process";
|
||||||
|
import { closeSync, openSync, readFileSync } from "node:fs";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import type { Measured, ProcessResult, RunDefinition } from "../vocab/index.ts";
|
||||||
|
|
||||||
|
// 用 Docker 把一条命令跑完：建镜像、起容器、等结束、查状态、删容器（docs/结构.md 第二部分）。
|
||||||
|
// runner 不碰账本：登记要跑、确认启动、写结果都由 flow 做（方法.md 第九部分）。
|
||||||
|
// 本身不存状态，全部依据都在 Docker 里，所以另一个进程里新建的 Runner 也能接着等一个早先启动的容器。
|
||||||
|
|
||||||
|
/** 查状态的五种回答。unreachable 是 Docker 不应答，flow 据此记"状态未知"并停下（方法.md 第七、第九部分）。 */
|
||||||
|
export type ContainerStatus = "absent" | "created" | "running" | "exited" | "unreachable";
|
||||||
|
|
||||||
|
/** 启动的结果：启动了，或者没启动和 Docker 的报错原文。 */
|
||||||
|
export type StartOutcome = { started: true } | { started: false; reason: string };
|
||||||
|
|
||||||
|
/** Docker 对容器名的要求（与 Docker 源码里的 RestrictedNamePattern 相同）。 */
|
||||||
|
const CONTAINER_NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]+$/;
|
||||||
|
|
||||||
|
/** 容器里的固定主机名：默认主机名是容器编号前 12 位，每次不同，镜像编号钉不住它。 */
|
||||||
|
const HOSTNAME = "inforsi";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 计量结果在容器自己文件层里的位置，不挂到宿主。目录由容器里的 root 建、只有 root 可写，
|
||||||
|
* 降权后的命令既写不进去也改不了权限，伪造不了结果。
|
||||||
|
*/
|
||||||
|
const RECORD_DIR = "/inforsi-record";
|
||||||
|
const USAGE_FILE = "usage";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 容器里的第一个进程，以容器内 root 身份运行。参数依次是时限（纳秒）、时限（秒，交给 timeout）、
|
||||||
|
* 降权到的用户编号和组编号、真正的命令。
|
||||||
|
* - timeout 和这段 shell 是 root，被 timeout 包着的 setpriv 降到宿主当前用户再执行命令：
|
||||||
|
* 命令写进运行目录的文件在宿主上归当前用户，宿主能读产物和清理；命令也没法向 root 进程发信号。
|
||||||
|
* setpriv 不改环境变量，命令的环境仍只有镜像的和 docker run 给的。
|
||||||
|
* - 用 KILL 结束超时的命令：它不能被捕获，候选不能拖延；timeout 此时的退出码是 128+9=137。
|
||||||
|
* - 超时由这段 shell 自己确认：退出码 137 且用时不短于时限。命令自己以 124 或 137 结束时，
|
||||||
|
* 用时不到时限，不会被当成超时。
|
||||||
|
* - 资源用量读容器自己的 cgroup v2（cgroup 命名空间下 /sys/fs/cgroup 就是本容器），
|
||||||
|
* 命令被 timeout 杀掉之后照样读得到；从外面杀掉整个容器时这段 shell 也一起没了，结果文件不会出现。
|
||||||
|
* - 先写临时文件再改名：结果文件要么完整，要么不存在。
|
||||||
|
*/
|
||||||
|
const MEASURE_SCRIPT = `limit_ns=$1; limit=$2; uid=$3; gid=$4; shift 4
|
||||||
|
mkdir -m 700 ${RECORD_DIR}
|
||||||
|
start=$(date +%s%N)
|
||||||
|
timeout -s KILL "$limit" setpriv --reuid="$uid" --regid="$gid" --clear-groups -- "$@"
|
||||||
|
code=$?
|
||||||
|
end=$(date +%s%N)
|
||||||
|
wall=$((end - start))
|
||||||
|
ending=exited
|
||||||
|
if [ "$code" -eq 137 ] && [ "$wall" -ge "$limit_ns" ]; then ending=timeout; fi
|
||||||
|
cpu=$(sed -n 's/^usage_usec //p' /sys/fs/cgroup/cpu.stat)
|
||||||
|
mem=$(cat /sys/fs/cgroup/memory.peak)
|
||||||
|
echo "$ending $code $wall $cpu $mem" > ${RECORD_DIR}/${USAGE_FILE}.tmp
|
||||||
|
mv ${RECORD_DIR}/${USAGE_FILE}.tmp ${RECORD_DIR}/${USAGE_FILE}
|
||||||
|
`;
|
||||||
|
|
||||||
|
/** 结果文件的格式：结局、退出码、墙钟纳秒、CPU 微秒、峰值内存字节，空格分隔，一行。 */
|
||||||
|
const USAGE_LINE = /^(exited|timeout) (\d+) (\d+) (\d+) (\d+)\n$/;
|
||||||
|
|
||||||
|
/** docker cp 找不到容器里的文件时的报错。 */
|
||||||
|
const NO_SUCH_FILE = "Could not find the file";
|
||||||
|
|
||||||
|
/** 认得出的"Docker 不应答"报错：本地套接字连不上、TCP 地址连不上两种写法。 */
|
||||||
|
const UNREACHABLE = /failed to connect to the docker API|Cannot connect to the Docker daemon/;
|
||||||
|
|
||||||
|
interface Done {
|
||||||
|
code: number;
|
||||||
|
stdout: string;
|
||||||
|
stderr: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class Runner {
|
||||||
|
readonly #hostArgs: string[];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* dockerHost：Docker 守护进程的地址，例如 unix:///var/run/docker.sock；
|
||||||
|
* null 表示不加 -H，由 docker 命令自己按 DOCKER_HOST 环境变量和当前 context 决定。
|
||||||
|
*/
|
||||||
|
constructor(dockerHost: string | null) {
|
||||||
|
this.#hostArgs = dockerHost === null ? [] : ["-H", dockerHost];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 用目录里的 Dockerfile 建镜像，打上 tag，交回镜像编号（`docker image inspect -f '{{.Id}}'` 的值）。 */
|
||||||
|
async buildImage(contextDir: string, tag: string): Promise<string> {
|
||||||
|
const built = await this.#docker(["build", "-t", tag, contextDir]);
|
||||||
|
if (built.code !== 0) throw new Error(`docker build ${contextDir} 失败：\n${built.stderr}`);
|
||||||
|
const inspected = await this.#docker(["image", "inspect", "--format", "{{.Id}}", tag]);
|
||||||
|
if (inspected.code !== 0) throw new Error(`docker image inspect ${tag} 失败：${inspected.stderr}`);
|
||||||
|
const digest = inspected.stdout.trim();
|
||||||
|
if (!/^sha256:[0-9a-f]{64}$/.test(digest)) {
|
||||||
|
throw new Error(`docker image inspect ${tag} 交回的不是镜像编号：${inspected.stdout}`);
|
||||||
|
}
|
||||||
|
return digest;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 本机有没有这个镜像编号。Docker 不应答或报别的错，直接报错。 */
|
||||||
|
async hasImage(digest: string): Promise<boolean> {
|
||||||
|
const inspected = await this.#docker(["image", "inspect", "--format", "{{.Id}}", digest]);
|
||||||
|
if (inspected.code === 0) return true;
|
||||||
|
if (inspected.stderr.includes("No such image")) return false;
|
||||||
|
throw new Error(`docker image inspect ${digest} 失败：${inspected.stderr}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 起一个容器跑运行定义里的命令，runDir 挂到 definition.workdir。
|
||||||
|
* docker run 失败时按名字再查一次（方法.md 第七部分"不确定有没有启动时，先核对"）：
|
||||||
|
* 容器不存在或建了没启动，算没启动（建了的顺手删掉）；在跑或已结束，算启动了。
|
||||||
|
*/
|
||||||
|
async start(name: string, definition: RunDefinition, runDir: string): Promise<StartOutcome> {
|
||||||
|
checkName(name);
|
||||||
|
const getuid = process.getuid;
|
||||||
|
const getgid = process.getgid;
|
||||||
|
if (getuid === undefined || getgid === undefined) throw new Error("runner 只能在 POSIX 系统上用：读不到当前用户编号");
|
||||||
|
const wallNs = Math.ceil(definition.wallSeconds * 1e9);
|
||||||
|
const args = [
|
||||||
|
"run",
|
||||||
|
"--detach",
|
||||||
|
"--name",
|
||||||
|
name,
|
||||||
|
// 本机 Docker 的默认运行时不一定是 runc（这台机器是 nvidia），跟着默认走，换台机器环境就悄悄变了。
|
||||||
|
"--runtime",
|
||||||
|
definition.image.runtime,
|
||||||
|
// 镜像只按编号用本机已有的，不去仓库拉：拉到的可能是另一份内容。
|
||||||
|
"--pull",
|
||||||
|
"never",
|
||||||
|
// 第一个进程是 root，好让计量结果不归命令管；命令本身由计量 shell 降到宿主当前用户再跑。
|
||||||
|
"--user",
|
||||||
|
"0:0",
|
||||||
|
// 镜像里有 su、mount 这类 setuid 程序；禁止靠它们重新拿到 root，降权后的命令就碰不到计量结果。
|
||||||
|
// setpriv 降权是 root 主动换身份，不受这条限制。
|
||||||
|
"--security-opt",
|
||||||
|
"no-new-privileges",
|
||||||
|
"--hostname",
|
||||||
|
HOSTNAME,
|
||||||
|
// 方法要求运行时不联网。
|
||||||
|
"--network",
|
||||||
|
"none",
|
||||||
|
// --cpus 限总量，不固定到某个核。PYTHON_CPU_COUNT 让 os.cpu_count() 报这个核数（Python 3.13 起），
|
||||||
|
// 否则它报宿主核数，候选会开一堆进程抢一个核。写在 env 之后：同名时 docker 取后一个。
|
||||||
|
`--cpus=${definition.cores}`,
|
||||||
|
// 降权后的用户在镜像里没有家目录；HOME 不写明的话会是 root 的 /root。写在 env 之前，运行定义可以改。
|
||||||
|
"--env",
|
||||||
|
"HOME=/",
|
||||||
|
...Object.entries(definition.env).flatMap(([key, value]) => ["--env", `${key}=${value}`]),
|
||||||
|
"--env",
|
||||||
|
`PYTHON_CPU_COUNT=${definition.cores}`,
|
||||||
|
// 用 --mount 而不是 -v：-v 遇到相对路径会悄悄建一个同名数据卷，--mount 直接报错。
|
||||||
|
"--mount",
|
||||||
|
`type=bind,source=${runDir},target=${definition.workdir}`,
|
||||||
|
"--workdir",
|
||||||
|
definition.workdir,
|
||||||
|
// 第一个进程固定是计量 shell，不受镜像 ENTRYPOINT 影响。
|
||||||
|
"--entrypoint",
|
||||||
|
"/bin/sh",
|
||||||
|
// 不加 --rm：中断恢复时要按名字找到这个容器。
|
||||||
|
definition.image.digest,
|
||||||
|
"-c",
|
||||||
|
MEASURE_SCRIPT,
|
||||||
|
"inforsi-measure",
|
||||||
|
String(wallNs),
|
||||||
|
String(definition.wallSeconds),
|
||||||
|
String(getuid.call(process)),
|
||||||
|
String(getgid.call(process)),
|
||||||
|
...definition.command,
|
||||||
|
];
|
||||||
|
const run = await this.#docker(args);
|
||||||
|
if (run.code === 0) return { started: true };
|
||||||
|
const status = await this.status(name);
|
||||||
|
if (status === "running" || status === "exited") return { started: true };
|
||||||
|
if (status === "unreachable") {
|
||||||
|
throw new Error(`docker run ${name} 失败，之后查状态时 Docker 不应答，不知道有没有启动。docker run 的报错：${run.stderr}`);
|
||||||
|
}
|
||||||
|
if (status === "created") await this.remove(name);
|
||||||
|
return { started: false, reason: run.stderr.trim() };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 等容器结束，交回进程层面的结果。标准输出、标准错误存成 logDir 下的 stdout.log、stderr.log，
|
||||||
|
* 计量结果文件从容器里拷出来存成 logDir 下的 usage（容器结束后 docker cp 照样能拷）。
|
||||||
|
* 只靠名字，能接上别的进程启动的容器；容器早已结束时 docker wait 立刻返回。
|
||||||
|
*/
|
||||||
|
async wait(name: string, logDir: string): Promise<ProcessResult> {
|
||||||
|
checkName(name);
|
||||||
|
const waited = await this.#docker(["wait", name]);
|
||||||
|
if (waited.code !== 0) throw new Error(`docker wait ${name} 失败：${waited.stderr}`);
|
||||||
|
await this.#saveLogs(name, logDir);
|
||||||
|
const usagePath = join(logDir, USAGE_FILE);
|
||||||
|
const copied = await this.#docker(["cp", `${name}:${RECORD_DIR}/${USAGE_FILE}`, usagePath]);
|
||||||
|
if (copied.code !== 0) {
|
||||||
|
if (!copied.stderr.includes(NO_SUCH_FILE)) throw new Error(`docker cp ${name} 的计量结果失败：${copied.stderr}`);
|
||||||
|
const reason = `容器从外面被结束（容器退出码 ${waited.stdout.trim()}），计量 shell 没有写出结果文件`;
|
||||||
|
const unmeasured: Measured = { status: "unmeasured", reason };
|
||||||
|
return { ending: "cancelled", cpuSeconds: unmeasured, wallSeconds: unmeasured, peakMemoryBytes: unmeasured };
|
||||||
|
}
|
||||||
|
const text = readFileSync(usagePath, "utf8");
|
||||||
|
const match = USAGE_LINE.exec(text);
|
||||||
|
if (match === null) throw new Error(`${usagePath} 的格式认不出：${JSON.stringify(text)}`);
|
||||||
|
const [, ending, code, wallNs, cpuUsec, memBytes] = match as unknown as [string, string, string, string, string, string];
|
||||||
|
const usage = {
|
||||||
|
cpuSeconds: measured(Number(cpuUsec) / 1e6),
|
||||||
|
wallSeconds: measured(Number(wallNs) / 1e9),
|
||||||
|
peakMemoryBytes: measured(Number(memBytes)),
|
||||||
|
};
|
||||||
|
if (ending === "timeout") return { ending: "timeout", ...usage };
|
||||||
|
return { ending: "exited", exitCode: Number(code), ...usage };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 按名字查容器的状态。Docker 报出这五种以外的状态（paused、restarting、dead 等）时报错。 */
|
||||||
|
async status(name: string): Promise<ContainerStatus> {
|
||||||
|
checkName(name);
|
||||||
|
const inspected = await this.#docker(["container", "inspect", "--format", "{{.State.Status}}", name]);
|
||||||
|
if (inspected.code !== 0) {
|
||||||
|
if (inspected.stderr.includes("No such container")) return "absent";
|
||||||
|
if (UNREACHABLE.test(inspected.stderr)) return "unreachable";
|
||||||
|
throw new Error(`docker container inspect ${name} 失败：${inspected.stderr}`);
|
||||||
|
}
|
||||||
|
const state = inspected.stdout.trim();
|
||||||
|
if (state === "created" || state === "running" || state === "exited") return state;
|
||||||
|
throw new Error(`容器 ${name} 处于认不出的状态：${state}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 删掉一个已结束或没启动的容器。还在跑的容器 docker 拒绝删，这里照样报错。 */
|
||||||
|
async remove(name: string): Promise<void> {
|
||||||
|
checkName(name);
|
||||||
|
const removed = await this.#docker(["rm", name]);
|
||||||
|
if (removed.code !== 0) throw new Error(`docker rm ${name} 失败：${removed.stderr}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 两条流直接写进文件，不经过内存：候选可能打印很多内容。恢复时可能再等一次，所以覆盖已有的文件。 */
|
||||||
|
async #saveLogs(name: string, logDir: string): Promise<void> {
|
||||||
|
const out = openSync(join(logDir, "stdout.log"), "w");
|
||||||
|
const err = openSync(join(logDir, "stderr.log"), "w");
|
||||||
|
try {
|
||||||
|
const code = await new Promise<number | null>((resolve, reject) => {
|
||||||
|
const child = spawn("docker", [...this.#hostArgs, "logs", name], { stdio: ["ignore", out, err] });
|
||||||
|
child.on("error", reject);
|
||||||
|
child.on("close", resolve);
|
||||||
|
});
|
||||||
|
if (code !== 0) throw new Error(`docker logs ${name} 失败，退出码 ${code}，详情见 ${logDir}/stderr.log`);
|
||||||
|
} finally {
|
||||||
|
closeSync(out);
|
||||||
|
closeSync(err);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 调一次 docker 命令，收齐输出。docker 命令本身起不来或被信号杀掉时报错。 */
|
||||||
|
#docker(args: string[]): Promise<Done> {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const child = spawn("docker", [...this.#hostArgs, ...args], { stdio: ["ignore", "pipe", "pipe"] });
|
||||||
|
let stdout = "";
|
||||||
|
let stderr = "";
|
||||||
|
child.stdout.setEncoding("utf8").on("data", (chunk: string) => (stdout += chunk));
|
||||||
|
child.stderr.setEncoding("utf8").on("data", (chunk: string) => (stderr += chunk));
|
||||||
|
child.on("error", reject);
|
||||||
|
child.on("close", (code, signal) => {
|
||||||
|
if (code === null) reject(new Error(`docker ${args[0]} 被信号 ${signal} 结束`));
|
||||||
|
else resolve({ code, stdout, stderr });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function checkName(name: string): void {
|
||||||
|
if (!CONTAINER_NAME.test(name)) throw new Error(`容器名 ${JSON.stringify(name)} 不符合 Docker 的要求 ${CONTAINER_NAME}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function measured(value: number): Measured {
|
||||||
|
return { status: "measured", value };
|
||||||
|
}
|
||||||
@@ -0,0 +1,342 @@
|
|||||||
|
import { copyFileSync, cpSync, existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { Type, type Static } from "typebox";
|
||||||
|
import { Value } from "typebox/value";
|
||||||
|
import type {
|
||||||
|
HypothesisMetric,
|
||||||
|
Json,
|
||||||
|
JudgedResult,
|
||||||
|
Measured,
|
||||||
|
MetricDefinition,
|
||||||
|
ProcessResult,
|
||||||
|
RunDefinition,
|
||||||
|
} from "../vocab/index.ts";
|
||||||
|
import type { TaskFacts } from "./facts.ts";
|
||||||
|
import { measured, readMetricsFile, unmeasured } from "./metrics-file.ts";
|
||||||
|
import type { LightCheckOutcome, Task, TaskImplementation } from "./types.ts";
|
||||||
|
|
||||||
|
// 填圆：在单位正方形里放 26 个圆，最大化半径之和。任务说明见 tasks/circle_packing/TASK.md。
|
||||||
|
// 凡是要执行任务代码的事（正式运行、校验器、轻量检查）都只交出运行定义，由调用方交给 runner 在任务镜像里跑；
|
||||||
|
// 这里只读写文件、解读产物（docs/结构.md 第二部分"runner 只管怎么跑，tasks 只管怎么判"）。
|
||||||
|
|
||||||
|
const CIRCLES = 26;
|
||||||
|
const ENTRY = "initial.py";
|
||||||
|
const HARNESS = "harness.py";
|
||||||
|
const VALIDATOR = "validate.py";
|
||||||
|
const CHECK_SCRIPT = "check-candidate.py";
|
||||||
|
const METRICS_SPEC = "metrics-spec.json";
|
||||||
|
const METRICS = "metrics.json";
|
||||||
|
const SOLUTION = "solution.json";
|
||||||
|
const VERDICT = "verdict.json";
|
||||||
|
const CHECK_RESULT = "check.json";
|
||||||
|
/** 轻量检查目录里放候选代码的子目录；检查脚本放在它的上一层，免得扫到自己。 */
|
||||||
|
const CHECK_CANDIDATE = "candidate";
|
||||||
|
/** 运行目录挂进容器的位置。 */
|
||||||
|
const WORKDIR = "/work";
|
||||||
|
/** 校验器的时限。26 个圆两两比较加合法化，实测一秒以内；它超时说明宿主的工具出了问题。 */
|
||||||
|
const VALIDATOR_SECONDS = 60;
|
||||||
|
/** 合法化前先按容差 0 判原产物（方法.md 第八部分"合法化"第 1 条）。 */
|
||||||
|
const TOLERANCE = "0";
|
||||||
|
|
||||||
|
// 常规辅助指标的编号（方法.md 第五部分"指标"）。前三个来自 runner 的资源用量，后四个来自校验器。
|
||||||
|
const WALL = "wall-seconds";
|
||||||
|
const CPU = "cpu-seconds";
|
||||||
|
const MEMORY = "peak-memory-bytes";
|
||||||
|
const CLEARANCE = "min-pairwise-clearance";
|
||||||
|
const RAW_SCORE = "raw-score";
|
||||||
|
const RAW_VIOLATION = "raw-violation";
|
||||||
|
const DEDUCTION = "legalization-deduction";
|
||||||
|
const FROM_VALIDATOR = [CLEARANCE, RAW_SCORE, RAW_VIOLATION, DEDUCTION];
|
||||||
|
|
||||||
|
const SIDE = "单位正方形的边长";
|
||||||
|
const PER_RUN = "每次运行结束后由宿主读一次，直接取值";
|
||||||
|
|
||||||
|
function auxiliary(
|
||||||
|
id: string,
|
||||||
|
meaning: string,
|
||||||
|
unit: string,
|
||||||
|
direction: "maximize" | "minimize",
|
||||||
|
howMeasured: string,
|
||||||
|
): MetricDefinition {
|
||||||
|
return { id, layer: "auxiliary", meaning, unit, direction, howMeasured, missing: "recordMissing" };
|
||||||
|
}
|
||||||
|
|
||||||
|
function metricDefinitions(facts: TaskFacts): MetricDefinition[] {
|
||||||
|
const validator = `宿主的校验器从 ${SOLUTION} 重算，${PER_RUN}`;
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
id: facts.finalMetric.id,
|
||||||
|
layer: "final",
|
||||||
|
meaning:
|
||||||
|
"合法摆法的半径之和，由宿主的独立校验器从实际产物重算；原产物差一点不合法时，先按 TASK.md 的只缩不移规则修，再按修后的半径算",
|
||||||
|
unit: SIDE,
|
||||||
|
direction: facts.finalMetric.direction,
|
||||||
|
howMeasured: validator,
|
||||||
|
missing: "invalidateRun",
|
||||||
|
},
|
||||||
|
auxiliary(WALL, "正式运行的墙钟时间", "秒", "minimize", `容器里的计量程序记录，${PER_RUN}`),
|
||||||
|
auxiliary(CPU, "正式运行用掉的 CPU 秒（容器 cgroup 的 usage_usec）", "秒", "minimize", `容器里的计量程序记录，${PER_RUN}`),
|
||||||
|
auxiliary(MEMORY, "正式运行的峰值内存（容器 cgroup 的 memory.peak）", "字节", "minimize", `容器里的计量程序记录，${PER_RUN}`),
|
||||||
|
auxiliary(CLEARANCE, "所有圆对中「圆心距减两半径之和」的最小值，大于等于 0 表示不重叠；修过的取修后的摆法", SIDE, "maximize", validator),
|
||||||
|
auxiliary(RAW_SCORE, "合法化之前原产物的半径之和", SIDE, "maximize", validator),
|
||||||
|
auxiliary(RAW_VIOLATION, "原产物的最大违反量：两圆重叠的量和圆出边的量里最大的一个，0 表示没有违反", SIDE, "minimize", validator),
|
||||||
|
auxiliary(DEDUCTION, "只缩不移合法化扣掉的分：原分数减最终成绩，原产物合法时是 0", SIDE, "minimize", validator),
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 正式运行、校验器、轻量检查共用的运行定义。数学库的线程数和核数一致，否则会开多个线程抢核。 */
|
||||||
|
function definition(task: Task, imageDigest: string, command: string[], wallSeconds: number, env: Record<string, string>): RunDefinition {
|
||||||
|
const cores = String(task.facts.run.cores);
|
||||||
|
return {
|
||||||
|
command,
|
||||||
|
workdir: WORKDIR,
|
||||||
|
image: { digest: imageDigest, runtime: task.facts.run.runtime },
|
||||||
|
wallSeconds,
|
||||||
|
cores: task.facts.run.cores,
|
||||||
|
inheritEnv: false,
|
||||||
|
env: { OMP_NUM_THREADS: cores, OPENBLAS_NUM_THREADS: cores, MKL_NUM_THREADS: cores, ...env },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// 解释器不加 -I：-I 会忽略全部 PYTHON* 环境变量，连同 runner 设的 PYTHON_CPU_COUNT，
|
||||||
|
// 于是 os.cpu_count() 报回宿主核数（本机实测 16）。容器里的环境变量全部由运行定义给出，用不着 -I。
|
||||||
|
// -B：不写 __pycache__，不往运行目录里添文件。
|
||||||
|
|
||||||
|
function prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void {
|
||||||
|
cpSync(candidateDir, runDir, { recursive: true });
|
||||||
|
// harness 和指标说明以宿主的为准，候选目录里有同名文件也被盖掉（这两个名字在 task.json 的不许改名单里）。
|
||||||
|
copyFileSync(join(task.dir, HARNESS), join(runDir, HARNESS));
|
||||||
|
writeFileSync(join(runDir, METRICS_SPEC), `${JSON.stringify(hypothesisMetrics, null, 2)}\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function formalRun(task: Task, imageDigest: string, seed: string): RunDefinition {
|
||||||
|
const command = ["python", "-B", `${WORKDIR}/${HARNESS}`, "--program", ENTRY, "--output", SOLUTION];
|
||||||
|
return definition(task, imageDigest, command, task.facts.run.wallSeconds, { INFORSI_SEED: seed });
|
||||||
|
}
|
||||||
|
|
||||||
|
function lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition {
|
||||||
|
cpSync(candidateDir, join(checkDir, CHECK_CANDIDATE), { recursive: true });
|
||||||
|
copyFileSync(join(task.dir, CHECK_SCRIPT), join(checkDir, CHECK_SCRIPT));
|
||||||
|
const command = ["python", "-B", CHECK_SCRIPT, CHECK_CANDIDATE, ENTRY, CHECK_RESULT];
|
||||||
|
return definition(task, imageDigest, command, seconds, {});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 检查脚本写的结果文件。valid 与 reasons 是否为空必须一致。 */
|
||||||
|
const CheckResult = Type.Object({ valid: Type.Boolean(), reasons: Type.Array(Type.String()) });
|
||||||
|
|
||||||
|
/** 宿主的工具（检查脚本、校验器）没正常跑完时报错停下：这不是候选的失败（任务书 05 第三部分）。 */
|
||||||
|
function requireExit(what: string, result: ProcessResult, codes: number[]): number {
|
||||||
|
if (result.ending === "exited" && codes.includes(result.exitCode)) return result.exitCode;
|
||||||
|
const how = result.ending === "notStarted" ? `没启动：${result.reason}` : result.ending === "exited" ? `退出码 ${result.exitCode}` : result.ending;
|
||||||
|
throw new Error(`${what}没有正常跑完（${how}），这是宿主一侧的工具出了问题，不能当成候选的失败`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function readJson(path: string): unknown {
|
||||||
|
try {
|
||||||
|
return JSON.parse(readFileSync(path, "utf8"));
|
||||||
|
} catch (error) {
|
||||||
|
throw new Error(`读不出 ${path}：${(error as Error).message}`, { cause: error });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function readLightCheck(checkDir: string, result: ProcessResult): LightCheckOutcome {
|
||||||
|
requireExit("轻量检查脚本", result, [0]);
|
||||||
|
const path = join(checkDir, CHECK_RESULT);
|
||||||
|
const check = readJson(path);
|
||||||
|
if (!Value.Check(CheckResult, check) || check.valid !== (check.reasons.length === 0)) {
|
||||||
|
throw new Error(`${path} 的内容认不出：${JSON.stringify(check)}`);
|
||||||
|
}
|
||||||
|
return check.valid ? { passed: true } : { passed: false, reasons: check.reasons };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 正式运行没正常结束的原因；正常退出（退出码 0）时为 null。 */
|
||||||
|
function processFailure(task: Task, formal: Exclude<ProcessResult, { ending: "notStarted" }>): string | null {
|
||||||
|
if (formal.ending === "timeout") return `超过单次时限 ${task.facts.run.wallSeconds} 秒，被结束`;
|
||||||
|
if (formal.ending === "cancelled") return "容器从外面被结束（关机、手动 docker kill 等），没有留下计量结果";
|
||||||
|
if (formal.exitCode !== 0) return `正式运行以退出码 ${formal.exitCode} 结束`;
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function started(formal: ProcessResult): Exclude<ProcessResult, { ending: "notStarted" }> {
|
||||||
|
if (formal.ending === "notStarted") throw new Error(`没启动的运行不判分：${formal.reason}`);
|
||||||
|
return formal;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 拼出 JudgedResult：资源用量来自正式运行，假设专用指标和两个保留键来自候选写的 metrics.json。
|
||||||
|
* 有"缺了整次作废"的假设专用指标测不到时，原本算数的也改判不算数（方法.md 第五部分"指标"）。
|
||||||
|
*/
|
||||||
|
function assemble(
|
||||||
|
runDir: string,
|
||||||
|
formal: Exclude<ProcessResult, { ending: "notStarted" }>,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
verdict: { valid: boolean; reason: string },
|
||||||
|
score: Measured,
|
||||||
|
fromValidator: Record<string, Measured>,
|
||||||
|
extra: Json,
|
||||||
|
): JudgedResult {
|
||||||
|
const file = readMetricsFile(join(runDir, METRICS), METRICS, hypothesisMetrics);
|
||||||
|
const voided = hypothesisMetrics.flatMap((m) => {
|
||||||
|
const value = file.hypothesis[m.id]!;
|
||||||
|
return m.missing === "invalidateRun" && value.status === "unmeasured" ? [`${m.id}（${value.reason}）`] : [];
|
||||||
|
});
|
||||||
|
const result = verdict.valid && voided.length > 0
|
||||||
|
? { valid: false, reason: `登记为"缺了整次作废"的假设专用指标测不到：${voided.join("；")}` }
|
||||||
|
: verdict;
|
||||||
|
return {
|
||||||
|
verdict: result.valid ? "valid" : "invalid",
|
||||||
|
verdictReason: result.reason,
|
||||||
|
score,
|
||||||
|
metrics: {
|
||||||
|
[WALL]: formal.wallSeconds,
|
||||||
|
[CPU]: formal.cpuSeconds,
|
||||||
|
[MEMORY]: formal.peakMemoryBytes,
|
||||||
|
...fromValidator,
|
||||||
|
...file.hypothesis,
|
||||||
|
},
|
||||||
|
answerPath: file.answerPath,
|
||||||
|
safeguardTriggers: file.safeguardTriggers,
|
||||||
|
extra,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function prepareJudging(
|
||||||
|
task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
imageDigest: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): ReturnType<TaskImplementation["prepareJudging"]> {
|
||||||
|
const run = started(formal);
|
||||||
|
const solution = join(runDir, SOLUTION);
|
||||||
|
const failure = processFailure(task, run) ?? (existsSync(solution) ? null : `没有产物：运行目录里没有 ${SOLUTION}`);
|
||||||
|
if (failure !== null) {
|
||||||
|
const notValidated = Object.fromEntries(FROM_VALIDATOR.map((id) => [id, unmeasured(`没有跑校验器：${failure}`)]));
|
||||||
|
const result = assemble(runDir, run, hypothesisMetrics, { valid: false, reason: failure }, unmeasured(failure), notValidated, {
|
||||||
|
legalization: null,
|
||||||
|
});
|
||||||
|
return { status: "judged", result };
|
||||||
|
}
|
||||||
|
copyFileSync(solution, join(judgeDir, SOLUTION));
|
||||||
|
copyFileSync(join(task.dir, VALIDATOR), join(judgeDir, VALIDATOR));
|
||||||
|
const command = [
|
||||||
|
"python",
|
||||||
|
"-B",
|
||||||
|
VALIDATOR,
|
||||||
|
"--circles",
|
||||||
|
String(CIRCLES),
|
||||||
|
"--solution",
|
||||||
|
SOLUTION,
|
||||||
|
"--atol",
|
||||||
|
TOLERANCE,
|
||||||
|
"--legalize",
|
||||||
|
"--verdict",
|
||||||
|
VERDICT,
|
||||||
|
];
|
||||||
|
return { status: "validate", definition: definition(task, imageDigest, command, VALIDATOR_SECONDS, {}) };
|
||||||
|
}
|
||||||
|
|
||||||
|
const NumberOrNull = Type.Union([Type.Number(), Type.Null()]);
|
||||||
|
|
||||||
|
/** validate.py --legalize 写的判定里，这里要读的部分；其余键（evidence、legal_radii 等）不读。 */
|
||||||
|
const ValidatorVerdict = Type.Object({
|
||||||
|
valid: Type.Boolean(),
|
||||||
|
computed_sum: NumberOrNull,
|
||||||
|
reasons: Type.Array(Type.String()),
|
||||||
|
diagnostics: Type.Object({ min_pairwise_clearance: NumberOrNull }),
|
||||||
|
legalization: Type.Object({
|
||||||
|
raw_valid: Type.Boolean(),
|
||||||
|
raw_score: NumberOrNull,
|
||||||
|
raw_violation: NumberOrNull,
|
||||||
|
applied: Type.Boolean(),
|
||||||
|
repaired: Type.Boolean(),
|
||||||
|
shrunk_circles: Type.Array(Type.Integer()),
|
||||||
|
reasons: Type.Array(Type.String()),
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
type ValidatorVerdict = Static<typeof ValidatorVerdict>;
|
||||||
|
|
||||||
|
/** 校验器在形状不对、含非有限数或负半径时提前返回，几何量没算。 */
|
||||||
|
const NOT_COMPUTED = "原产物形状不对、含非有限数或负半径，校验器没算这个量";
|
||||||
|
|
||||||
|
function orUnmeasured(value: number | null, reason: string): Measured {
|
||||||
|
return value === null ? unmeasured(reason) : measured(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
function readVerdict(judgeDir: string, exitCode: number): ValidatorVerdict {
|
||||||
|
const path = join(judgeDir, VERDICT);
|
||||||
|
const verdict = readJson(path);
|
||||||
|
if (!Value.Check(ValidatorVerdict, verdict)) throw new Error(`${path} 的内容认不出：${JSON.stringify(verdict)}`);
|
||||||
|
// 退出码 0 = 合法、1 = 不合法；合法化之后的结论就是最终判定。对不上说明跑的不是这份校验器。
|
||||||
|
const consistent =
|
||||||
|
verdict.valid === (exitCode === 0) &&
|
||||||
|
verdict.legalization.repaired === verdict.valid &&
|
||||||
|
(!verdict.valid || verdict.computed_sum !== null);
|
||||||
|
if (!consistent) throw new Error(`${path} 与退出码 ${exitCode} 或自身前后对不上：${JSON.stringify(verdict)}`);
|
||||||
|
return verdict;
|
||||||
|
}
|
||||||
|
|
||||||
|
function judge(
|
||||||
|
_task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
validator: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): JudgedResult {
|
||||||
|
const run = started(formal);
|
||||||
|
const verdict = readVerdict(judgeDir, requireExit("校验器", validator, [0, 1]));
|
||||||
|
const legal = verdict.legalization;
|
||||||
|
let deduction: Measured;
|
||||||
|
let score: Measured;
|
||||||
|
if (verdict.valid) {
|
||||||
|
const final = verdict.computed_sum!;
|
||||||
|
score = measured(final);
|
||||||
|
// legal.repaired 为真时 raw_score 一定有值：原产物能算出分数，才可能合法或被修好。
|
||||||
|
const raw = legal.raw_score!;
|
||||||
|
// 只缩半径只会降分；修后反而更高，说明修法写错了（方法.md 第八部分"合法化"第 3 条）。
|
||||||
|
if (final > raw) throw new Error(`合法化后分数 ${final} 高于原分数 ${raw}：只缩半径只会降分，修法出错了`);
|
||||||
|
deduction = measured(raw - final);
|
||||||
|
} else {
|
||||||
|
const reason = `不算数：${verdict.reasons.concat(legal.reasons).join("；")}`;
|
||||||
|
score = unmeasured(reason);
|
||||||
|
deduction = unmeasured(`修不了，没有修后分数：${legal.reasons.join("；")}`);
|
||||||
|
}
|
||||||
|
const rawScore = orUnmeasured(legal.raw_score, NOT_COMPUTED);
|
||||||
|
const rawViolation = orUnmeasured(legal.raw_violation, NOT_COMPUTED);
|
||||||
|
const fromValidator = {
|
||||||
|
[CLEARANCE]: orUnmeasured(verdict.diagnostics.min_pairwise_clearance, NOT_COMPUTED),
|
||||||
|
[RAW_SCORE]: rawScore,
|
||||||
|
[RAW_VIOLATION]: rawViolation,
|
||||||
|
[DEDUCTION]: deduction,
|
||||||
|
};
|
||||||
|
const reason = verdict.valid
|
||||||
|
? legal.raw_valid
|
||||||
|
? "校验器按容差 0 判原产物合法"
|
||||||
|
: `原产物不合法，按只缩不移规则修好后合法（缩了 ${legal.shrunk_circles.length} 个圆）`
|
||||||
|
: `校验器判不合法：${verdict.reasons.concat(legal.reasons).join("；")}`;
|
||||||
|
const extra = {
|
||||||
|
legalization: {
|
||||||
|
rawValid: legal.raw_valid,
|
||||||
|
applied: legal.applied,
|
||||||
|
repaired: legal.repaired,
|
||||||
|
shrunkCircles: legal.shrunk_circles,
|
||||||
|
reasons: legal.reasons,
|
||||||
|
rawScore,
|
||||||
|
rawViolation,
|
||||||
|
deduction,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return assemble(runDir, run, hypothesisMetrics, { valid: verdict.valid, reason }, score, fromValidator, extra);
|
||||||
|
}
|
||||||
|
|
||||||
|
export const circlePacking: TaskImplementation = {
|
||||||
|
metricDefinitions,
|
||||||
|
prepareRun,
|
||||||
|
formalRun,
|
||||||
|
lightCheck,
|
||||||
|
readLightCheck,
|
||||||
|
prepareJudging,
|
||||||
|
judge,
|
||||||
|
};
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
import { Type, type Static } from "typebox";
|
||||||
|
import { Settings } from "typebox/system";
|
||||||
|
import { Value } from "typebox/value";
|
||||||
|
import { Direction, Id } from "../vocab/index.ts";
|
||||||
|
|
||||||
|
// task.json 的字段定义与检查。task.json 是用户提供的东西，属于系统边界（CLAUDE.md"罕见情况怎么办"），
|
||||||
|
// 所以每个字段必填、不认识的字段报错，一次列出全部错误，每条写明字段路径。可以没有的值写 null。
|
||||||
|
|
||||||
|
/** 任务种类。每一种在 task.ts 的对照表里对应一份实现；nanochat、AI4AI 以后按同样的形状加。 */
|
||||||
|
export const TASK_KINDS = ["circle_packing"] as const;
|
||||||
|
export type TaskKind = (typeof TASK_KINDS)[number];
|
||||||
|
|
||||||
|
const CLOSED = { additionalProperties: false } as const;
|
||||||
|
|
||||||
|
export const TaskFacts = Type.Object(
|
||||||
|
{
|
||||||
|
kind: Type.Enum(TASK_KINDS, { type: "string", description: "任务种类，决定用哪一份实现" }),
|
||||||
|
finalMetric: Type.Object(
|
||||||
|
{
|
||||||
|
id: Id,
|
||||||
|
direction: Direction,
|
||||||
|
},
|
||||||
|
{ ...CLOSED, description: "最终指标的编号和方向（方法.md 第十部分第 1 条）" },
|
||||||
|
),
|
||||||
|
epsilon: Type.Number({ exclusiveMinimum: 0, description: "精度 ε（方法.md 6.11）" }),
|
||||||
|
knownBest: Type.Union([Type.Number(), Type.Null()], {
|
||||||
|
description: "已知最好成绩；没有就写 null（方法.md 6.11）",
|
||||||
|
}),
|
||||||
|
run: Type.Object(
|
||||||
|
{
|
||||||
|
wallSeconds: Type.Number({ exclusiveMinimum: 0, description: "单次正式运行的墙钟时限（秒）" }),
|
||||||
|
cores: Type.Integer({ minimum: 1, description: "单次正式运行的核数" }),
|
||||||
|
runtime: Type.String({ minLength: 1, description: "Docker 运行时，必须写明，例如 runc" }),
|
||||||
|
},
|
||||||
|
{ ...CLOSED, description: "同批候选共用的比较条件（方法.md 第八部分）" },
|
||||||
|
),
|
||||||
|
protectedFiles: Type.Array(Type.String({ minLength: 1 }), {
|
||||||
|
description: "不许改的文件，路径相对候选代码目录（方法.md 第十部分第 2 条）",
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
CLOSED,
|
||||||
|
);
|
||||||
|
export type TaskFacts = Static<typeof TaskFacts>;
|
||||||
|
|
||||||
|
type SchemaError = ReturnType<typeof Value.Errors>[number];
|
||||||
|
|
||||||
|
const TYPES: Readonly<Record<string, string>> = { number: "数字", integer: "整数", string: "字符串", object: "对象", array: "数组" };
|
||||||
|
|
||||||
|
/** 本文件的字段用得到的几种错误，写成中文；其余照 typebox 的原文。 */
|
||||||
|
const REASONS: Readonly<Record<string, (p: Record<string, unknown>) => string>> = {
|
||||||
|
type: (p) => `应为${TYPES[String(p.type)] ?? String(p.type)}`,
|
||||||
|
enum: (p) => `只能是 ${(p.allowedValues as unknown[]).map((v) => JSON.stringify(v)).join("、")} 之一`,
|
||||||
|
exclusiveMinimum: (p) => `必须大于 ${String(p.limit)}`,
|
||||||
|
minimum: (p) => `不能小于 ${String(p.limit)}`,
|
||||||
|
minLength: () => "不能为空",
|
||||||
|
pattern: (p) => `格式不对，应符合 ${String(p.pattern)}`,
|
||||||
|
};
|
||||||
|
|
||||||
|
function errorLines(errors: SchemaError[]): string[] {
|
||||||
|
const lines: string[] = [];
|
||||||
|
for (const e of errors) {
|
||||||
|
const p = e.params as Record<string, unknown>;
|
||||||
|
const at = e.instancePath || "/";
|
||||||
|
// 唯一的几选一是 knownBest（数字或 null）；各分支自己的错误不再逐条列。
|
||||||
|
if (e.schemaPath.includes("/anyOf/")) continue;
|
||||||
|
if (e.keyword === "anyOf") lines.push(`${at}：应为数字，没有就写 null`);
|
||||||
|
else if (e.keyword === "required") {
|
||||||
|
for (const name of p.requiredProperties as string[]) lines.push(`${e.instancePath}/${name}：缺少这一项（每个字段都必填）`);
|
||||||
|
} else if (e.keyword === "additionalProperties") {
|
||||||
|
for (const name of p.additionalProperties as string[]) lines.push(`${e.instancePath}/${name}：不认识的字段`);
|
||||||
|
} else lines.push(`${at}：${REASONS[e.keyword]?.(p) ?? e.message}`);
|
||||||
|
}
|
||||||
|
return lines;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 检查一份已经解析好的 task.json；合格就原样返回，不合格就报错，列出全部错误。 */
|
||||||
|
export function checkTaskFacts(value: unknown): TaskFacts {
|
||||||
|
if (Value.Check(TaskFacts, value)) return value;
|
||||||
|
// typebox 默认最多收 8 条错误，不放开就会漏报；放开只在这一次调用里。
|
||||||
|
const cap = Settings.Get().maxErrors;
|
||||||
|
Settings.Set({ maxErrors: Infinity });
|
||||||
|
let errors: SchemaError[];
|
||||||
|
try {
|
||||||
|
errors = [...Value.Errors(TaskFacts, value)];
|
||||||
|
} finally {
|
||||||
|
Settings.Set({ maxErrors: cap });
|
||||||
|
}
|
||||||
|
const lines = errorLines(errors);
|
||||||
|
throw new Error(`task.json 不合格，共 ${lines.length} 处：\n${lines.join("\n")}`);
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
// tasks：任务目录里的事实和文件、怎么准备一次运行、怎么从原始产物独立判分（docs/结构.md 第二部分）。
|
||||||
|
// 不起子进程：要执行任务代码的事都只交出运行定义，由调用方交给 runner 在任务镜像里跑。
|
||||||
|
export { checkTaskFacts, TASK_KINDS, TaskFacts, type TaskKind } from "./facts.ts";
|
||||||
|
export {
|
||||||
|
formalRun,
|
||||||
|
judge,
|
||||||
|
lightCheck,
|
||||||
|
loadTask,
|
||||||
|
metricDefinitions,
|
||||||
|
prepareJudging,
|
||||||
|
prepareRun,
|
||||||
|
protectedChanges,
|
||||||
|
readLightCheck,
|
||||||
|
} from "./task.ts";
|
||||||
|
export type { JudgingStep, LightCheckOutcome, Task } from "./types.ts";
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
import { existsSync, readFileSync } from "node:fs";
|
||||||
|
import {
|
||||||
|
ANSWER_PATH_KEY,
|
||||||
|
SAFEGUARD_TRIGGERS_KEY,
|
||||||
|
type AnswerPath,
|
||||||
|
type HypothesisMetric,
|
||||||
|
type Measured,
|
||||||
|
type SafeguardTriggers,
|
||||||
|
} from "../vocab/index.ts";
|
||||||
|
|
||||||
|
// 读候选写的指标文件：假设专用指标的取值，加两个保留键（方法.md 第五部分"指标""答案路径"）。
|
||||||
|
// 这个文件是候选的产物，读不到、写错了都记成明确的状态并附原因，不补 0、不补空对象，也不报错停下。
|
||||||
|
|
||||||
|
export interface MetricsFileReading {
|
||||||
|
/** 每条登记的假设专用指标一项，键是指标编号。 */
|
||||||
|
hypothesis: Record<string, Measured>;
|
||||||
|
answerPath: AnswerPath;
|
||||||
|
safeguardTriggers: SafeguardTriggers;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function unmeasured(reason: string): Measured {
|
||||||
|
return { status: "unmeasured", reason };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function measured(value: number): Measured {
|
||||||
|
return { status: "measured", value };
|
||||||
|
}
|
||||||
|
|
||||||
|
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||||
|
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 整个文件都读不到时，每一项都记同一个原因。 */
|
||||||
|
function allUnreadable(reason: string, metrics: HypothesisMetric[]): MetricsFileReading {
|
||||||
|
return {
|
||||||
|
hypothesis: Object.fromEntries(metrics.map((m) => [m.id, unmeasured(reason)])),
|
||||||
|
answerPath: { status: "unreadable", reason },
|
||||||
|
safeguardTriggers: { status: "unreadable", reason },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function readMetric(values: Record<string, unknown>, id: string, name: string): Measured {
|
||||||
|
const value = values[id];
|
||||||
|
if (value === undefined) return unmeasured(`${name} 里没有指标 ${id}`);
|
||||||
|
if (value === null) return unmeasured(`候选写的是 null：这一次没测到 ${id}`);
|
||||||
|
// JSON.parse 会把 1e999 读成 Infinity，所以要查有限。
|
||||||
|
if (typeof value !== "number" || !Number.isFinite(value)) {
|
||||||
|
return unmeasured(`${name} 里指标 ${id} 的值不是有限数：${JSON.stringify(value)}`);
|
||||||
|
}
|
||||||
|
return measured(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
function readAnswerPath(values: Record<string, unknown>, name: string): AnswerPath {
|
||||||
|
const value = values[ANSWER_PATH_KEY];
|
||||||
|
if (value === undefined) return { status: "unreadable", reason: `${name} 里没写 ${ANSWER_PATH_KEY}` };
|
||||||
|
if (typeof value !== "string") {
|
||||||
|
return { status: "unreadable", reason: `${name} 里 ${ANSWER_PATH_KEY} 不是字符串：${JSON.stringify(value)}` };
|
||||||
|
}
|
||||||
|
return { status: "read", value };
|
||||||
|
}
|
||||||
|
|
||||||
|
function readTriggers(values: Record<string, unknown>, name: string): SafeguardTriggers {
|
||||||
|
const value = values[SAFEGUARD_TRIGGERS_KEY];
|
||||||
|
if (value === undefined) return { status: "unreadable", reason: `${name} 里没写 ${SAFEGUARD_TRIGGERS_KEY}` };
|
||||||
|
const counts = isRecord(value) ? Object.entries(value) : null;
|
||||||
|
if (counts === null || counts.some(([, n]) => !Number.isInteger(n) || (n as number) < 0)) {
|
||||||
|
const reason = `${name} 里 ${SAFEGUARD_TRIGGERS_KEY} 应为"保障短名 → 非负整数"的对象，实际是 ${JSON.stringify(value)}`;
|
||||||
|
return { status: "unreadable", reason };
|
||||||
|
}
|
||||||
|
return { status: "read", counts: Object.fromEntries(counts) as Record<string, number> };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 读指标文件。metrics 是这次登记的假设专用指标，只按它们的编号取值，多写的键不读。 */
|
||||||
|
export function readMetricsFile(path: string, name: string, metrics: HypothesisMetric[]): MetricsFileReading {
|
||||||
|
if (!existsSync(path)) return allUnreadable(`${name} 不存在`, metrics);
|
||||||
|
let values: unknown;
|
||||||
|
try {
|
||||||
|
values = JSON.parse(readFileSync(path, "utf8"));
|
||||||
|
} catch (error) {
|
||||||
|
return allUnreadable(`${name} 不是合法的 JSON：${(error as Error).message}`, metrics);
|
||||||
|
}
|
||||||
|
if (!isRecord(values)) return allUnreadable(`${name} 应为 JSON 对象（指标编号 → 值）`, metrics);
|
||||||
|
return {
|
||||||
|
hypothesis: Object.fromEntries(metrics.map((m) => [m.id, readMetric(values, m.id, name)])),
|
||||||
|
answerPath: readAnswerPath(values, name),
|
||||||
|
safeguardTriggers: readTriggers(values, name),
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
import { readFileSync } from "node:fs";
|
||||||
|
import { join, resolve } from "node:path";
|
||||||
|
import type { HypothesisMetric, JudgedResult, MetricDefinition, ProcessResult, RunDefinition } from "../vocab/index.ts";
|
||||||
|
import { circlePacking } from "./circle-packing.ts";
|
||||||
|
import { checkTaskFacts, type TaskKind } from "./facts.ts";
|
||||||
|
import type { JudgingStep, LightCheckOutcome, Task, TaskImplementation } from "./types.ts";
|
||||||
|
|
||||||
|
// tasks 对外的函数。任务目录的约定：task.json（任务事实）、TASK.md（给实验员的说明）、
|
||||||
|
// Dockerfile 与依赖清单（建镜像）、baseline/（基线代码），以及各任务自己的 Python 工具。
|
||||||
|
// 每个函数按 task.facts.kind 从对照表里取出那种任务的实现来做。
|
||||||
|
|
||||||
|
/** 任务种类 → 实现。加一种任务就在这里加一行。 */
|
||||||
|
const IMPLEMENTATIONS: Readonly<Record<TaskKind, TaskImplementation>> = {
|
||||||
|
circle_packing: circlePacking,
|
||||||
|
};
|
||||||
|
|
||||||
|
function implementation(task: Task): TaskImplementation {
|
||||||
|
return IMPLEMENTATIONS[task.facts.kind];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 读一个任务目录：严格检查 task.json，同时交回 Dockerfile 所在目录、基线目录和 TASK.md 全文。 */
|
||||||
|
export function loadTask(taskDir: string): Task {
|
||||||
|
const dir = resolve(taskDir);
|
||||||
|
const path = join(dir, "task.json");
|
||||||
|
let value: unknown;
|
||||||
|
try {
|
||||||
|
value = JSON.parse(readFileSync(path, "utf8"));
|
||||||
|
} catch (error) {
|
||||||
|
throw new Error(`${path} 读不出来或不是合法的 JSON：${(error as Error).message}`, { cause: error });
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
facts: checkTaskFacts(value),
|
||||||
|
dir,
|
||||||
|
baselineDir: join(dir, "baseline"),
|
||||||
|
guide: readFileSync(join(dir, "TASK.md"), "utf8"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 最终指标与常规辅助指标的定义。最终指标的编号和方向取自 task.json。 */
|
||||||
|
export function metricDefinitions(task: Task): MetricDefinition[] {
|
||||||
|
return implementation(task).metricDefinitions(task.facts);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 准备一次正式运行的运行目录：把候选代码、任务的 harness 和这次要测的假设专用指标说明
|
||||||
|
* （metrics-spec.json）放进去。runDir 由调用方建好，应为空目录。
|
||||||
|
* 消融格里传进来的指标应当已经全部改成"缺了只记缺失"（vocab 指标说明），这里不区分。
|
||||||
|
*/
|
||||||
|
export function prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void {
|
||||||
|
implementation(task).prepareRun(task, candidateDir, runDir, hypothesisMetrics);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 正式运行的运行定义。种子由调用方给，同一份研究的全部运行用同一个种子：
|
||||||
|
* 评测要确定，消融不重跑基线也靠这一点（方法.md 第十部分第 9 条）。
|
||||||
|
*/
|
||||||
|
export function formalRun(task: Task, imageDigest: string, seed: string): RunDefinition {
|
||||||
|
return implementation(task).formalRun(task, imageDigest, seed);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 轻量检查：把候选代码和检查脚本放进 checkDir（调用方建好的空目录），交回运行定义。
|
||||||
|
* 只看语法和入口函数，不执行候选代码。seconds 来自配置；次数上限不归这里管。
|
||||||
|
*/
|
||||||
|
export function lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition {
|
||||||
|
return implementation(task).lightCheck(task, candidateDir, checkDir, imageDigest, seconds);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解读轻量检查：通过，或者不通过及原因。检查脚本自己没正常跑完时报错停下。 */
|
||||||
|
export function readLightCheck(task: Task, checkDir: string, result: ProcessResult): LightCheckOutcome {
|
||||||
|
return implementation(task).readLightCheck(checkDir, result);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 判分第一步。正式运行没正常结束、或者没有产物时直接判不算数；否则把产物和宿主保管的校验器
|
||||||
|
* 复制进 judgeDir（调用方建好的空目录），交回校验器的运行定义。
|
||||||
|
*/
|
||||||
|
export function prepareJudging(
|
||||||
|
task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
imageDigest: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): JudgingStep {
|
||||||
|
return implementation(task).prepareJudging(task, runDir, judgeDir, imageDigest, formal, hypothesisMetrics);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 判分第二步：读校验器写的判定、运行目录里候选写的指标文件，加上正式运行的结果，交回判分结果。
|
||||||
|
* 校验器自己没正常跑完、判定文件认不出、修后分数高于原分数时报错停下：这些是宿主一侧的问题。
|
||||||
|
*/
|
||||||
|
export function judge(
|
||||||
|
task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
validator: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): JudgedResult {
|
||||||
|
return implementation(task).judge(task, runDir, judgeDir, formal, validator, hypothesisMetrics);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 改动过的文件里，属于 task.json"不许改"名单的那些。路径都相对候选代码目录。 */
|
||||||
|
export function protectedChanges(task: Task, changedFiles: string[]): string[] {
|
||||||
|
return changedFiles.filter((file) => task.facts.protectedFiles.includes(file));
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import type { HypothesisMetric, JudgedResult, MetricDefinition, ProcessResult, RunDefinition } from "../vocab/index.ts";
|
||||||
|
import type { TaskFacts } from "./facts.ts";
|
||||||
|
|
||||||
|
/** 读进来的一个任务。 */
|
||||||
|
export interface Task {
|
||||||
|
/** task.json 的内容，已严格检查。 */
|
||||||
|
facts: TaskFacts;
|
||||||
|
/** 任务目录的绝对路径，里面有 Dockerfile 和依赖清单，runner 拿它建镜像。 */
|
||||||
|
dir: string;
|
||||||
|
/** 基线代码目录。 */
|
||||||
|
baselineDir: string;
|
||||||
|
/** TASK.md 全文，给实验员看的任务说明。 */
|
||||||
|
guide: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 轻量检查的结论：通过，或者不通过及原因。 */
|
||||||
|
export type LightCheckOutcome = { passed: true } | { passed: false; reasons: string[] };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 判分第一步的结果：要么不用跑校验器就已判定（正式运行没正常结束、没有产物），
|
||||||
|
* 要么交回校验器的运行定义，由调用方交给 runner 去跑。
|
||||||
|
*/
|
||||||
|
export type JudgingStep = { status: "judged"; result: JudgedResult } | { status: "validate"; definition: RunDefinition };
|
||||||
|
|
||||||
|
/** 一种任务的实现。对外函数按 task.facts.kind 从对照表里取出它来调。参数的含义见 task.ts 里同名的对外函数。 */
|
||||||
|
export interface TaskImplementation {
|
||||||
|
/** 最终指标（编号和方向取自 task.json）与常规辅助指标的定义。 */
|
||||||
|
metricDefinitions(facts: TaskFacts): MetricDefinition[];
|
||||||
|
prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void;
|
||||||
|
formalRun(task: Task, imageDigest: string, seed: string): RunDefinition;
|
||||||
|
lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition;
|
||||||
|
readLightCheck(checkDir: string, result: ProcessResult): LightCheckOutcome;
|
||||||
|
prepareJudging(
|
||||||
|
task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
imageDigest: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): JudgingStep;
|
||||||
|
judge(
|
||||||
|
task: Task,
|
||||||
|
runDir: string,
|
||||||
|
judgeDir: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
validator: ProcessResult,
|
||||||
|
hypothesisMetrics: HypothesisMetric[],
|
||||||
|
): JudgedResult;
|
||||||
|
}
|
||||||
+4
-3
@@ -128,7 +128,8 @@ const RESOURCE_USAGE = {
|
|||||||
/**
|
/**
|
||||||
* 以什么方式结束。有没有启动由它决定：只有 notStarted 是没启动，
|
* 以什么方式结束。有没有启动由它决定：只有 notStarted 是没启动，
|
||||||
* 其余三种都启动过、要扣次数（方法.md 第七部分）。
|
* 其余三种都启动过、要扣次数（方法.md 第七部分）。
|
||||||
* 启动前就取消的记 notStarted；启动后取消的记 cancelled。
|
* 启动前就取消的记 notStarted；cancelled 是容器从外面被杀了（关机、手动 docker kill、
|
||||||
|
* 计量的 shell 自己被杀），没有留下计量结果，三项资源都记测不到。
|
||||||
* 退出码只有正常退出才有，资源用量只有启动过才有，所以分成四种形状。
|
* 退出码只有正常退出才有，资源用量只有启动过才有，所以分成四种形状。
|
||||||
*/
|
*/
|
||||||
export const ProcessResult = Type.Union([
|
export const ProcessResult = Type.Union([
|
||||||
@@ -168,8 +169,8 @@ export type ProcessResult = Static<typeof ProcessResult>;
|
|||||||
// 运行结果第二层：tasks 从原始产物独立判分
|
// 运行结果第二层：tasks 从原始产物独立判分
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
/** 判定三选一：算数、不算数、判不了（判定所需的证据没拿到）。 */
|
/** 判定二选一：算数、不算数。 */
|
||||||
export const VERDICTS = ["valid", "invalid", "undecidable"] as const;
|
export const VERDICTS = ["valid", "invalid"] as const;
|
||||||
export type Verdict = (typeof VERDICTS)[number];
|
export type Verdict = (typeof VERDICTS)[number];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# 基础镜像按 digest 钉死，不用 tag——tag 会被重新指向，digest 不会。
|
||||||
|
# 这里写全仓库名（dockerproxy.net/library/python），因为本机没配 registry mirror，
|
||||||
|
# 只写 python 会去 docker.io 拉。
|
||||||
|
FROM dockerproxy.net/library/python:3.13-slim@sha256:8d9d0b8bcf6506481eae4907c18f5e3e7902e629f5f6d684f9e7c32e85e3ddf0
|
||||||
|
# 每次运行只给一个核。numba 不设这个变量时 parallel=True 会按宿主核数开线程
|
||||||
|
# （调研_填圆运行环境依赖.md 3.3 实测开 16 个）；写在镜像里，由镜像编号钉住，
|
||||||
|
# 不依赖每份配置记得写。数学库的线程数由 task.env 的 OMP_NUM_THREADS 管。
|
||||||
|
ENV NUMBA_NUM_THREADS=1
|
||||||
|
COPY requirements.txt /req.txt
|
||||||
|
RUN pip install --no-cache-dir -r /req.txt
|
||||||
@@ -0,0 +1,295 @@
|
|||||||
|
# 填圆任务（n=26，单位正方形）
|
||||||
|
|
||||||
|
这份文件是给实验员看的任务契约：要做什么、产物写在哪、键名怎么写。
|
||||||
|
读它的时候假定你手上已经有一份冻结的提案，本文件不重复提案里的内容。
|
||||||
|
|
||||||
|
## 目标
|
||||||
|
|
||||||
|
在单位正方形 `[0,1]²` 内放置 **26 个互不重叠、不越过边界**的圆，**最大化半径之和**。
|
||||||
|
|
||||||
|
最终指标：`sum-of-radii`，越大越好。
|
||||||
|
|
||||||
|
量级参照：容差 0 的口径下，26 个圆公开的最好半径和约 **2.635983**（AlphaEvolve 续作公开的构造，
|
||||||
|
按它自己的严格校验在本机重算得 2.6359830849）。常见的"2.635"是白皮书截到三位小数的数，不是另一个纪录。
|
||||||
|
别处报告的更高的数要先看容差：容差放到 1e-6，把每个半径加 5e-7 也判合格，和就能到 2.635996——那是口径不同，不是更好的摆法。
|
||||||
|
|
||||||
|
**成绩是从你交出来的产物算的，不是从你返回的那个数读的。**
|
||||||
|
`validate.py` 读 `solution.json` 里的 26 个半径，用 `math.fsum` 求和，那就是成绩。
|
||||||
|
|
||||||
|
你在 `run_packing()` 里返回的第三个值（`sum_of_radii`）**不是成绩**。
|
||||||
|
`harness.py` 照常把它写进 `solution.json` 的 `reported_sum` 字段，但
|
||||||
|
`validate.py` 根本不读这个字段：它不进判定、不进诊断，也不跟任何数对账。
|
||||||
|
所以这个数写过期了、跟半径对不上，产物照样合法，成绩照样是算出来的那一个。
|
||||||
|
|
||||||
|
只有一件事要注意，而且跟对账无关：`harness.py` 落盘时不接受非有限数，
|
||||||
|
所以这个值要是 `NaN` 或无穷大，整份 `solution.json` 根本写不出来，
|
||||||
|
这次运行连产物都没有。返回一个普通的有限数就行，哪怕它是错的。
|
||||||
|
|
||||||
|
## 评测会收下什么
|
||||||
|
|
||||||
|
判断代码里一段检查、修复或兜底"挡掉了谁、改了什么"，要先知道评测本身会收下什么。
|
||||||
|
**评测收下一次运行，按下面的次序：**
|
||||||
|
|
||||||
|
1. 你交的 `solution.json` 形状对（26 个圆心、26 个半径）、全是有限数、半径不为负、圆心都在正方形里。
|
||||||
|
这几条任何一条不成立，评测**不收**，也不修。
|
||||||
|
2. 原产物按**容差 0** 判：任意两圆不重叠、每个圆不越出正方形边。成立就收下，成绩是原产物的半径和。
|
||||||
|
3. 不成立时，先按下一节"只缩不移合法化"修一次，修后再按容差 0 判。修得好就收下，成绩是修后的半径和
|
||||||
|
（只比原产物低一点）；修不好才不收。
|
||||||
|
|
||||||
|
所以"差一点不合法"（重叠或越界只有浮点零头那么大）的产物，**评测本来会收**，只扣掉零头。
|
||||||
|
候选代码里比这更严的检查——例如容差 0 判不合法就把这个起点整个丢掉、或换成另一份解——挡掉的正是评测会收的产物，
|
||||||
|
它改的是交出去的答案，不是"让代码合法"。反过来，只在评测确实不收的情形（形状不对、非有限数、圆心出界）
|
||||||
|
才动手修或换的代码，在评测会收的运行上不改答案。
|
||||||
|
|
||||||
|
## 差一点不合法时：只缩不移合法化
|
||||||
|
|
||||||
|
校验器先按容差 0 判你交的原产物。原产物不合法时，宿主按下面这条写死的规则修一次，
|
||||||
|
**只缩半径、不挪圆心**，修好了就用修后的半径和当成绩：
|
||||||
|
|
||||||
|
1. **只缩参与违反的圆。**每个圆的半径减去它参与的最大违反量——跟别的圆重叠的量、
|
||||||
|
越出正方形边的量，取最大的那个。没参与违反的圆一个都不动。
|
||||||
|
2. 缩完仍不合法（浮点舍入留下的零头），把这些圆的半径各往 0 挪一个最小的浮点单位，
|
||||||
|
再判，直到合法。
|
||||||
|
3. 同一个校验器按容差 0 重判修后的产物。
|
||||||
|
4. **修不了**：有圆心在正方形外面；参与违反的圆半径都缩到 0 仍不合法；
|
||||||
|
形状不对、含非有限值或负半径（这些不是"差一点"，缩半径解决不了）。
|
||||||
|
修不了的运行照旧记失败，运行次数照扣——跟跑崩、超时、没产物一样。
|
||||||
|
|
||||||
|
**缩小只会降分**，所以这条规则不会让不合法的摆法比合法的更高分。它管的是"浮点上差一点"：
|
||||||
|
例如两个圆重叠 2.8e-17，修后扣掉约 2.2e-16。要一分不扣，就在代码里给每个圆留一点空隙。
|
||||||
|
|
||||||
|
每次运行多记三个数，跟成绩一起进运行记录：原违反量（`raw-violation`）、
|
||||||
|
原分数（`raw-score`）、合法化扣掉的分（`legalization-deduction`，原产物合法时是 0）。
|
||||||
|
|
||||||
|
## 什么样的结果才算数
|
||||||
|
|
||||||
|
下面五条不是禁令，是**这次实验要成立需要的条件**。写出来是为了让你知道该往哪边走，
|
||||||
|
不是为了防谁。
|
||||||
|
|
||||||
|
### 一 · 成绩必须是算出来的，不能是写死的
|
||||||
|
|
||||||
|
代码里可以有常数：半径比例、松弛步长、网格怎么划分、迭代多少轮，都可以写死。
|
||||||
|
**不能有的是一张 26 行的答案表**——把某次跑出来的圆心和半径直接抄进源码返回。
|
||||||
|
|
||||||
|
*为什么*：研究要找的是方法，不是一个数。一份写死的答案没有机制可以解释——
|
||||||
|
问它"为什么这个改法有效"，答案是"因为答案是抄的"，而这句话不会出现在提案里。
|
||||||
|
一个没有方法的高分会让整轮研究的结论是假的：后面几轮会建立在
|
||||||
|
"某某方向有效"这个不存在的发现之上。
|
||||||
|
|
||||||
|
### 二 · 只读工作目录里的东西，不联网
|
||||||
|
|
||||||
|
你可以带自己的数据文件，它们会跟代码一起冻结、一起放进工作目录。
|
||||||
|
但运行时不要去读工作目录外面的文件。正式运行的容器是断网的，网络请求一律失败。
|
||||||
|
|
||||||
|
*为什么*：冻结的代码要能完整说清"这次跑的是什么"。
|
||||||
|
从网上取一份东西进来，那份东西明天可能就变了，这次运行就再也重现不了；
|
||||||
|
读工作目录外面的文件也一样，那个文件不在冻结的代码里。
|
||||||
|
|
||||||
|
### 三 · 在执行限额之内跑完
|
||||||
|
|
||||||
|
每次正式运行的墙钟时限是 120 秒、1 核，同一批候选都一样。到时限还没结束，进程被直接杀掉，
|
||||||
|
这次运行记为超时失败，运行次数照扣。
|
||||||
|
|
||||||
|
*为什么*：跟基线比的前提是同样的条件。用一百倍时间换来的成绩，
|
||||||
|
说明的是算力，不是方法。
|
||||||
|
|
||||||
|
### 四 · 用了多少算力要说出来
|
||||||
|
|
||||||
|
每次运行都记墙钟时间（`wall-seconds`）、CPU 秒（`cpu-seconds`）和峰值内存（`peak-memory-bytes`），
|
||||||
|
材料里同时写"是基线的几倍"和"占单次时限的几成"。这些只记录，不影响采纳。
|
||||||
|
|
||||||
|
**加大搜索预算本身是正当的做法**，不需要藏着。但用远多于基线的算力换来的提升，
|
||||||
|
说明的是算力，不是方法——所以要在提案里写明加了多少、为什么值得。
|
||||||
|
|
||||||
|
*为什么*：跟基线比的前提是同样的条件。一次多花一百五十倍时间、换来百分之七提升的运行，
|
||||||
|
如果被记成"某某改法有效"，后面几轮会去改那个改法，而真正起作用的是预算。
|
||||||
|
|
||||||
|
### 五 · 说了什么就要真的做什么
|
||||||
|
|
||||||
|
代码和注释里写了"迭代到收敛"，就要真的迭代到收敛；写了"最多 1000 步"，
|
||||||
|
就不能实际只跑 2 步。
|
||||||
|
|
||||||
|
*为什么*：这一条跟前面几条不一样，**它坏的不是成绩，是解释**。
|
||||||
|
成绩可能完全是真的、可复现的，但"松弛到收敛有效"这条结论会进记录，
|
||||||
|
下一轮建立在一个没有发生过的机制之上。对不上的时候，
|
||||||
|
该作废的是那条结论，不是这次运行。
|
||||||
|
|
||||||
|
这也是为什么第一条说"没有方法的高分会让整轮研究的结论是假的"——
|
||||||
|
证据链上的错，后面每一轮都看不见。
|
||||||
|
|
||||||
|
## 运行方法
|
||||||
|
|
||||||
|
候选可以是多文件 Python 项目，普通相对数据资源会保留；入口 `initial.py` 必须定义：
|
||||||
|
|
||||||
|
```python
|
||||||
|
def run_packing():
|
||||||
|
...
|
||||||
|
return centers, radii, sum_of_radii # (26,2)、(26,)、float
|
||||||
|
```
|
||||||
|
|
||||||
|
正式运行由程序启动，在任务镜像里、断网、1 核、120 秒时限下执行（工作目录就是你的代码目录）：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -B harness.py --program initial.py --output solution.json
|
||||||
|
```
|
||||||
|
|
||||||
|
跑完以后，宿主把 `solution.json` 复制到另一个目录，用自己保管的校验器判分：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -B validate.py --circles 26 --solution solution.json --atol 0 --legalize --verdict verdict.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`harness.py` 在 import 候选前，用环境变量 `INFORSI_SEED` 给的种子字符串（取 SHA256 前 4 字节大端整数）
|
||||||
|
初始化 Python 的 `random` 和 numpy 的 `random`，执行候选并原子落盘原始解，不做任何有效性判断。
|
||||||
|
import 之前它还会删掉工作目录里已有的 `solution.json` 和 `metrics.json`，宿主读到的只会是这一次写的。
|
||||||
|
你要用随机，用这两个。同一份研究的全部运行用同一个种子，所以同一份代码跑两次，结果逐位相同。
|
||||||
|
|
||||||
|
`validate.py` 是独立校验器，重算半径和并检查形状 / 有限 / 非负 / 边界 / 重叠；
|
||||||
|
加 `--legalize` 时按上面"只缩不移合法化"一节修差一点不合法的产物（正式评测就是这样调的）。
|
||||||
|
|
||||||
|
写代码的过程中可以做**轻量检查**：由宿主在同一个镜像里跑，只看语法和入口函数，
|
||||||
|
不执行候选的任何一行代码。它有次数和时长上限（每次几秒），不算一次正式运行。
|
||||||
|
|
||||||
|
**不要自己跑正式规模的求解。** 正式运行要先登记、再启动、再扣次数，由程序统一做；
|
||||||
|
正式规模的求解哪怕叫"调试"，也是正式运行，不能借轻量检查或自己的 shell 绕开预算。
|
||||||
|
|
||||||
|
## 指标怎么交给宿主（这一节最容易写错，先读完再动手）
|
||||||
|
|
||||||
|
工作目录里有两个指标文件，一进一出。**路径都相对工作目录**，也就是 `harness.py`
|
||||||
|
运行时的 cwd，不要写绝对路径。
|
||||||
|
|
||||||
|
### `metrics-spec.json`：宿主写给你的
|
||||||
|
|
||||||
|
宿主在准备运行目录时写好它，内容是这次运行登记的假设专用指标定义，
|
||||||
|
一个 JSON 数组，每个元素是一条指标：
|
||||||
|
|
||||||
|
| 字段 | 含义 |
|
||||||
|
| --- | --- |
|
||||||
|
| `id` | 指标编号，宿主按它取值 |
|
||||||
|
| `meaning` | 这个量测的是什么 |
|
||||||
|
| `unit` | 单位 |
|
||||||
|
| `direction` | `maximize` 越大越好，`minimize` 越小越好 |
|
||||||
|
| `howMeasured` | 研究者写的测法：在什么时刻测、测几次、怎么合成一个数。**你照着它实现** |
|
||||||
|
| `missing` | 测不到时这次运行怎么算：`recordMissing` 只记缺失、成绩照常；`invalidateRun` 整次作废 |
|
||||||
|
|
||||||
|
没有假设专用指标时，这个文件是一个空数组 `[]`。
|
||||||
|
|
||||||
|
### `metrics.json`：你写给宿主的
|
||||||
|
|
||||||
|
一个 JSON 对象：**键必须是 `metrics-spec.json` 里那条指标的 `id` 字段原文**，
|
||||||
|
值是有限数或 `null`。另外**每次都要写两个保留键**（下面"两个保留键"一节），它们不是指标、值不是数。
|
||||||
|
|
||||||
|
> 编号从 `metrics-spec.json` 里**照抄**，不要自己拼。宿主按编号取值，键对不上，
|
||||||
|
> 这条指标就记成"测不到"加一句原因——**而成绩完全正常**，报告上看不出任何异常，
|
||||||
|
> 只是研究者分析机制时悄悄少了一块证据。
|
||||||
|
|
||||||
|
四条硬要求：
|
||||||
|
|
||||||
|
1. **键照抄 `id`**，见上。
|
||||||
|
2. **值是有限数或 `null`。测不到就写 `null`，不要写 0。**
|
||||||
|
写 0 之后宿主分不出"测到了一个 0"和"没测到"，而这种错事后从证据里查不出来。
|
||||||
|
算出 `NaN` / `inf`（退化布局里零半径参与的比值最常见）也一样，那是"没测到"。
|
||||||
|
harness 会把非有限值原样改写成 `null` 并在 stderr 记一行，退出码仍是 0——
|
||||||
|
成绩不受影响，但那条指标这一次就是没有数据。
|
||||||
|
3. **必须在 `run_packing()` 返回之前写完。** harness 在候选返回后立刻整理这个文件；
|
||||||
|
之后再写进去的内容不会被整理（非有限值不会变成 `null`，裸 `NaN` 会让宿主整份读不出来，
|
||||||
|
**这一次所有假设专用指标一起丢**，不只是坏的那一条）。
|
||||||
|
4. 键多了无所谓，宿主只按 `metrics-spec.json` 里的编号取值；键少了，缺的那条记"测不到"加一句原因。
|
||||||
|
|
||||||
|
文件不存在表示这一次的假设专用指标一个都没测到——这是一个明确状态，不是错误，
|
||||||
|
但它和"写错了键名"在报告里看起来一样，所以别靠它。
|
||||||
|
|
||||||
|
**测不到的后果看 `missing`**：选了 `invalidateRun` 的指标测不到，这次运行整次作废、不能被采纳，
|
||||||
|
运行次数照扣；选了 `recordMissing` 的只记缺失。
|
||||||
|
|
||||||
|
### 两个保留键：答案出自哪条路径
|
||||||
|
|
||||||
|
研究者不能运行代码，"最终交的答案到底是哪段代码给的"只能读代码猜，猜错了没人发现。
|
||||||
|
所以每次运行由你的代码自己说出来，写在同一个 `metrics.json` 里：
|
||||||
|
|
||||||
|
| 键 | 值 | 写什么 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `answer-path` | 字符串 | 最终交的答案出自哪里：`main`（正常主流程），或冻结协议因子清单里某个因子的编号（`p1`、`p2`……），或保障清单里某条保障的短名 |
|
||||||
|
| `safeguard-triggers` | 对象 | 键是冻结协议保障清单里的**每一条**保障短名，值是这次运行里它动手了几次（非负整数，没动手写 0）；保障写"无"时写 `{}` |
|
||||||
|
|
||||||
|
- 这两个键**不是指标**，不参与成绩判定；写错、没写都不作废运行，宿主照实记"候选没写答案路径"
|
||||||
|
或"不认识的路径：原文"。但研究者下一阶段就靠它看"这一格量到的是因子还是保障"，所以要如实写。
|
||||||
|
- 消融格（关掉某个因子的版本）同样要写：关掉的因子不会是答案路径；如果关掉之后答案改由某条保障给出，就写那条保障。
|
||||||
|
- harness 不改写这两个键（指标值才会把非有限数改写成 `null`）。同样要在 `run_packing()` 返回之前写完。
|
||||||
|
|
||||||
|
### 一小段示例
|
||||||
|
|
||||||
|
```python
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
|
||||||
|
SPEC_FILE = "metrics-spec.json"
|
||||||
|
METRICS_FILE = "metrics.json"
|
||||||
|
|
||||||
|
|
||||||
|
def run_packing():
|
||||||
|
with open(SPEC_FILE, encoding="utf-8") as handle:
|
||||||
|
spec = json.load(handle)
|
||||||
|
# spec[i]["id"] 就是宿主认的编号，spec[i]["howMeasured"] 是照着实现的测法。
|
||||||
|
# 这里按这次实际条数对号入座，下面只演示一条。
|
||||||
|
spread_id = spec[0]["id"]
|
||||||
|
|
||||||
|
centers, radii, reported_sum = solve()
|
||||||
|
|
||||||
|
spread = float(max(radii) - min(radii))
|
||||||
|
measured = {
|
||||||
|
spread_id: spread if math.isfinite(spread) else None,
|
||||||
|
# 两个保留键：答案出自主流程；协议里的一条保障 "shrink-repair" 这次没动手。
|
||||||
|
"answer-path": "main",
|
||||||
|
"safeguard-triggers": {"shrink-repair": 0},
|
||||||
|
}
|
||||||
|
|
||||||
|
# 在 return 之前写完，路径相对工作目录。
|
||||||
|
with open(METRICS_FILE, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(measured, handle)
|
||||||
|
|
||||||
|
return centers, radii, reported_sum
|
||||||
|
```
|
||||||
|
|
||||||
|
最终指标 `sum-of-radii` 和合法性判定**不经过这条链**：它们由 `validate.py`
|
||||||
|
从 `solution.json` 独立重算，你不用也不该在 `metrics.json` 里写它们。
|
||||||
|
|
||||||
|
## 允许修改范围
|
||||||
|
|
||||||
|
候选只能修改自己代码目录里的 `*.py` 和自带的数据文件（基线的入口是 `initial.py`，其中
|
||||||
|
`EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` 之间是官方预留的进化区域）。
|
||||||
|
不许改的文件见同目录 `task.json` 的 `protectedFiles`；程序会检查有没有改它们。
|
||||||
|
`harness.py` 和 `metrics-spec.json` 在准备运行目录时由宿主放进去，你目录里的同名文件会被盖掉；
|
||||||
|
校验器和轻量检查脚本由宿主保管，不在你的目录里。也不要改校验容差或执行限额。
|
||||||
|
|
||||||
|
## 基线与许可
|
||||||
|
|
||||||
|
`baseline/` 为 SakanaAI/ShinkaEvolve 官方代码，Apache-2.0（见 `baseline/LICENSE`），
|
||||||
|
固定提交与 sha256 见 `baseline/SOURCES.md`。基线的嵌套缩半径循环与退化布局是
|
||||||
|
刻意保留的原始事实，不得"修复"——任何改进必须作为正式候选经过研究流程。
|
||||||
|
|
||||||
|
## 依赖
|
||||||
|
|
||||||
|
Python 3.13。正式实验环境里装的第三方库**只有下面这些**，版本与 `requirements.txt` 逐行一致：
|
||||||
|
|
||||||
|
| 包 | 版本 | 用途 |
|
||||||
|
|---|---|---|
|
||||||
|
| numpy | 2.3.1 | 数组与矩阵运算 |
|
||||||
|
| scipy | 1.18.1 | 数值优化、线性规划、空间距离计算等 |
|
||||||
|
| numba | 0.67.0 | 把 Python 循环编译成机器码 |
|
||||||
|
| llvmlite | 0.49.0 | numba 的编译后端 |
|
||||||
|
| sympy | 1.14.0 | 符号与精确计算 |
|
||||||
|
| mpmath | 1.3.0 | 任意精度浮点运算（sympy 依赖它） |
|
||||||
|
| networkx | 3.7 | 图结构分析 |
|
||||||
|
|
||||||
|
**依赖只有镜像里的这几个包。**运行时不能再装别的包：正式运行在固定的镜像里跑、容器断网，
|
||||||
|
你在工作目录里装的包进不了正式运行。
|
||||||
|
|
||||||
|
**每次运行只有 1 核。**容器按 `--cpus=1` 限总量，环境里 `PYTHON_CPU_COUNT=1`，
|
||||||
|
所以 `os.cpu_count()`、`multiprocessing.cpu_count()` 都报 1。多开进程或线程不会更快，
|
||||||
|
只会互相抢这 1 个核、少干活。数学库（`OMP_NUM_THREADS`）和 numba（`NUMBA_NUM_THREADS`）的线程数都固定为 1。numba 首次编译函数的时间算在这次运行的墙钟里。
|
||||||
|
|
||||||
|
**环境由镜像钉死。**任务跑在 `Dockerfile` 建出来的镜像里，这些库都烧在那个镜像里。
|
||||||
|
镜像在研究开始时建一次，镜像编号冻结进账本；一份研究不能自己改依赖。
|
||||||
|
|
||||||
|
（给维护者：`Dockerfile` 的基础镜像经 `dockerproxy.net` 拉取，换机器时可能要改这一行。）
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
|
1. Definitions.
|
||||||
|
|
||||||
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
|
the copyright owner that is granting the License.
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright 2025 Sakana AI
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# 官方基线来源声明
|
||||||
|
|
||||||
|
本目录文件来自 SakanaAI/ShinkaEvolve，Apache-2.0 许可（见同目录 `LICENSE`）。
|
||||||
|
|
||||||
|
- 固定提交：`9912af12d423504b8d580f4179fd15f5f88b8c50`
|
||||||
|
- 固定日期：2026-09-17（由项目负责人原字节下载并哈希固定在 `/tmp/rivalbits2-shinka-18k23a46/manifest.json`）
|
||||||
|
|
||||||
|
| 文件 | 原始 URL | sha256 | 修改 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `initial.py` | <https://raw.githubusercontent.com/SakanaAI/ShinkaEvolve/9912af12d423504b8d580f4179fd15f5f88b8c50/examples/circle_packing/initial.py> | `09da8428a85792a94a9c6cc2193b6a0db1090ce598a17b75bc0db48d0953d098` | 无（逐字节复制） |
|
||||||
|
| `evaluate.py` | <https://raw.githubusercontent.com/SakanaAI/ShinkaEvolve/9912af12d423504b8d580f4179fd15f5f88b8c50/examples/circle_packing/evaluate.py> | `eb8b9787dc25dee24a5f5305d1e3bd876e5c336cded1c9bab7fb204612928b4d` | 无（逐字节复制） |
|
||||||
|
| `LICENSE` | <https://raw.githubusercontent.com/SakanaAI/ShinkaEvolve/9912af12d423504b8d580f4179fd15f5f88b8c50/LICENSE> | `c3e5b1c26607caf7208d5a105c113fe240630bb873fdb8ca5b4c2e32c721ddaf` | 无 |
|
||||||
|
|
||||||
|
上表三个 sha256 是基线"没被改过"的唯一判据，由本项目的 `tests/tasks.test.ts`
|
||||||
|
（"基线三个文件的 sha256 等于 SOURCES.md 表上的值"）逐个断言。
|
||||||
|
|
||||||
|
它防的是**别的测试看不见的改动**：给 `initial.py` 加一行注释、或者动一个字节的
|
||||||
|
`evaluate.py` / `LICENSE`，成绩一位不变，别的测试都不会挂。`evaluate.py` 和 `LICENSE`
|
||||||
|
根本不在运行路径上。
|
||||||
|
|
||||||
|
一个实测过的事实（参考项目里跑出来的）：把 `compute_max_radii` 那个白跑 105625 次的冗余双循环删掉，
|
||||||
|
半径和一位不差（都是 0.9597642169962064），但第 (7, 8) 对圆会重叠 2.7755575615628914e-17，
|
||||||
|
容差 0 下判不合法，要经合法化才收下。那个冗余循环仍然是本研究的改进起点。
|
||||||
|
|
||||||
|
## 已知的基线事实（刻意保留，不是缺陷）
|
||||||
|
|
||||||
|
- `initial.py` 的 `compute_max_radii` 含嵌套的冗余缩半径循环（外层 `for i/for j`
|
||||||
|
包了一层重复的 `for i/for j`），且外环布局（0.7 半径环经 clip 到 0.99）是退化布局；
|
||||||
|
圆心 clip 后可能出现零半径。这些是官方基线的原样事实，是本研究的改进起点，
|
||||||
|
**不得“修复”**——任何改进必须作为正式候选经过研究流程。
|
||||||
|
- `evaluate.py` 依赖 `shinka.core.run_shinka_eval`（ShinkaEvolve 框架内部），
|
||||||
|
且其聚合以候选自报的 `reported_sum` 作为 `combined_score`。本项目**不信任**
|
||||||
|
自报分数：独立校验器 `../validate.py` 不 import 本文件，自行重算半径和。
|
||||||
|
本文件仅作为官方协议参考保留，不在本项目的运行路径上执行。
|
||||||
@@ -0,0 +1,197 @@
|
|||||||
|
"""
|
||||||
|
Evaluator for circle packing example (n=26) with improved timeout handling
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import argparse
|
||||||
|
import numpy as np
|
||||||
|
from typing import Tuple, Optional, List, Dict, Any
|
||||||
|
|
||||||
|
from shinka.core import run_shinka_eval
|
||||||
|
|
||||||
|
|
||||||
|
def format_centers_string(centers: np.ndarray) -> str:
|
||||||
|
"""Formats circle centers into a multi-line string for display."""
|
||||||
|
return "\n".join(
|
||||||
|
[
|
||||||
|
f" centers[{i}] = ({x_coord:.4f}, {y_coord:.4f})"
|
||||||
|
for i, (x_coord, y_coord) in enumerate(centers)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def adapted_validate_packing(
|
||||||
|
run_output: Tuple[np.ndarray, np.ndarray, float],
|
||||||
|
atol=0.0,
|
||||||
|
) -> Tuple[bool, Optional[str]]:
|
||||||
|
"""
|
||||||
|
Validates circle packing results based on the output of 'run_packing'.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
run_output: Tuple (centers, radii, reported_sum) from run_packing.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(is_valid: bool, error_message: Optional[str])
|
||||||
|
"""
|
||||||
|
centers, radii, reported_sum = run_output
|
||||||
|
msg = "The circles are placed correctly. There are no overlaps or any circles outside the unit square."
|
||||||
|
if not isinstance(centers, np.ndarray):
|
||||||
|
centers = np.array(centers)
|
||||||
|
if not isinstance(radii, np.ndarray):
|
||||||
|
radii = np.array(radii)
|
||||||
|
|
||||||
|
n_expected = 26
|
||||||
|
if centers.shape != (n_expected, 2):
|
||||||
|
msg = (
|
||||||
|
f"Centers shape incorrect. Expected ({n_expected}, 2), got {centers.shape}"
|
||||||
|
)
|
||||||
|
return False, msg
|
||||||
|
if radii.shape != (n_expected,):
|
||||||
|
msg = f"Radii shape incorrect. Expected ({n_expected},), got {radii.shape}"
|
||||||
|
return False, msg
|
||||||
|
|
||||||
|
if (
|
||||||
|
not np.all(np.isfinite(centers))
|
||||||
|
or not np.all(np.isfinite(radii))
|
||||||
|
or not np.isfinite(reported_sum)
|
||||||
|
):
|
||||||
|
msg = "Non-finite values found in centers, radii, or reported_sum."
|
||||||
|
return False, msg
|
||||||
|
|
||||||
|
if np.any(radii < 0):
|
||||||
|
negative_indices = np.where(radii < 0)[0]
|
||||||
|
msg = f"Negative radii found for circles at indices: {negative_indices}"
|
||||||
|
return False, msg
|
||||||
|
|
||||||
|
if not np.isclose(np.sum(radii), reported_sum, atol=atol):
|
||||||
|
msg = (
|
||||||
|
f"Sum of radii ({np.sum(radii):.6f}) does not match "
|
||||||
|
f"reported ({reported_sum:.6f})"
|
||||||
|
)
|
||||||
|
return False, msg
|
||||||
|
|
||||||
|
for i in range(n_expected):
|
||||||
|
x, y = centers[i]
|
||||||
|
r = radii[i]
|
||||||
|
is_outside = (
|
||||||
|
x - r < -atol or x + r > 1 + atol or y - r < -atol or y + r > 1 + atol
|
||||||
|
)
|
||||||
|
if is_outside:
|
||||||
|
msg = (
|
||||||
|
f"Circle {i} (x={x:.4f}, y={y:.4f}, r={r:.4f}) is outside unit square."
|
||||||
|
)
|
||||||
|
return False, msg
|
||||||
|
|
||||||
|
for i in range(n_expected):
|
||||||
|
for j in range(i + 1, n_expected):
|
||||||
|
dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2))
|
||||||
|
if dist < radii[i] + radii[j] - atol:
|
||||||
|
msg = (
|
||||||
|
f"Circles {i} & {j} overlap. Dist: {dist:.4f}, "
|
||||||
|
f"Sum Radii: {(radii[i] + radii[j]):.4f}"
|
||||||
|
)
|
||||||
|
return False, msg
|
||||||
|
return True, msg
|
||||||
|
|
||||||
|
|
||||||
|
def get_circle_packing_kwargs(run_index: int) -> Dict[str, Any]:
|
||||||
|
"""Provides keyword arguments for circle packing runs (none needed)."""
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def aggregate_circle_packing_metrics(
|
||||||
|
results: List[Tuple[np.ndarray, np.ndarray, float]], results_dir: str
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Aggregates metrics for circle packing. Assumes num_runs=1.
|
||||||
|
Saves extra.npz with detailed packing information.
|
||||||
|
"""
|
||||||
|
if not results:
|
||||||
|
return {"combined_score": 0.0, "error": "No results to aggregate"}
|
||||||
|
|
||||||
|
centers, radii, reported_sum = results[0]
|
||||||
|
|
||||||
|
public_metrics = {
|
||||||
|
"centers_str": format_centers_string(centers),
|
||||||
|
"num_circles": centers.shape[0],
|
||||||
|
}
|
||||||
|
private_metrics = {
|
||||||
|
"reported_sum_of_radii": float(reported_sum),
|
||||||
|
}
|
||||||
|
metrics = {
|
||||||
|
"combined_score": float(reported_sum),
|
||||||
|
"public": public_metrics,
|
||||||
|
"private": private_metrics,
|
||||||
|
}
|
||||||
|
|
||||||
|
extra_file = os.path.join(results_dir, "extra.npz")
|
||||||
|
try:
|
||||||
|
np.savez(
|
||||||
|
extra_file,
|
||||||
|
centers=centers,
|
||||||
|
radii=radii,
|
||||||
|
reported_sum=reported_sum,
|
||||||
|
)
|
||||||
|
print(f"Detailed packing data saved to {extra_file}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error saving extra.npz: {e}")
|
||||||
|
metrics["extra_npz_save_error"] = str(e)
|
||||||
|
|
||||||
|
return metrics
|
||||||
|
|
||||||
|
|
||||||
|
def main(program_path: str, results_dir: str):
|
||||||
|
"""Runs the circle packing evaluation using shinka.eval."""
|
||||||
|
print(f"Evaluating program: {program_path}")
|
||||||
|
print(f"Saving results to: {results_dir}")
|
||||||
|
os.makedirs(results_dir, exist_ok=True)
|
||||||
|
|
||||||
|
num_experiment_runs = 1
|
||||||
|
|
||||||
|
# Define a nested function to pass results_dir to the aggregator
|
||||||
|
def _aggregator_with_context(
|
||||||
|
r: List[Tuple[np.ndarray, np.ndarray, float]],
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
return aggregate_circle_packing_metrics(r, results_dir)
|
||||||
|
|
||||||
|
metrics, correct, error_msg = run_shinka_eval(
|
||||||
|
program_path=program_path,
|
||||||
|
results_dir=results_dir,
|
||||||
|
experiment_fn_name="run_packing",
|
||||||
|
num_runs=num_experiment_runs,
|
||||||
|
get_experiment_kwargs=get_circle_packing_kwargs,
|
||||||
|
validate_fn=adapted_validate_packing,
|
||||||
|
aggregate_metrics_fn=_aggregator_with_context,
|
||||||
|
)
|
||||||
|
|
||||||
|
if correct:
|
||||||
|
print("Evaluation and Validation completed successfully.")
|
||||||
|
else:
|
||||||
|
print(f"Evaluation or Validation failed: {error_msg}")
|
||||||
|
|
||||||
|
print("Metrics:")
|
||||||
|
for key, value in metrics.items():
|
||||||
|
if isinstance(value, str) and len(value) > 100:
|
||||||
|
print(f" {key}: <string_too_long_to_display>")
|
||||||
|
else:
|
||||||
|
print(f" {key}: {value}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Circle packing evaluator using shinka.eval"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--program_path",
|
||||||
|
type=str,
|
||||||
|
default="initial.py",
|
||||||
|
help="Path to program to evaluate (must contain 'run_packing')",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--results_dir",
|
||||||
|
type=str,
|
||||||
|
default="results",
|
||||||
|
help="Dir to save results (metrics.json, correct.json, extra.npz)",
|
||||||
|
)
|
||||||
|
parsed_args = parser.parse_args()
|
||||||
|
main(parsed_args.program_path, parsed_args.results_dir)
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# EVOLVE-BLOCK-START
|
||||||
|
"""Constructor-based circle packing for n=26 circles"""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
def construct_packing():
|
||||||
|
"""
|
||||||
|
Construct a specific arrangement of 26 circles in a unit square
|
||||||
|
that attempts to maximize the sum of their radii.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (centers, radii, sum_of_radii)
|
||||||
|
centers: np.array of shape (26, 2) with (x, y) coordinates
|
||||||
|
radii: np.array of shape (26) with radius of each circle
|
||||||
|
sum_of_radii: Sum of all radii
|
||||||
|
"""
|
||||||
|
# Initialize arrays for 26 circles
|
||||||
|
n = 26
|
||||||
|
centers = np.zeros((n, 2))
|
||||||
|
|
||||||
|
# Place circles in a structured pattern
|
||||||
|
# This is a simple pattern - evolution will improve this
|
||||||
|
|
||||||
|
# First, place a large circle in the center
|
||||||
|
centers[0] = [0.5, 0.5]
|
||||||
|
|
||||||
|
# Place 8 circles around it in a ring
|
||||||
|
for i in range(8):
|
||||||
|
angle = 2 * np.pi * i / 8
|
||||||
|
centers[i + 1] = [0.5 + 0.3 * np.cos(angle), 0.5 + 0.3 * np.sin(angle)]
|
||||||
|
|
||||||
|
# Place 16 more circles in an outer ring
|
||||||
|
for i in range(16):
|
||||||
|
angle = 2 * np.pi * i / 16
|
||||||
|
centers[i + 9] = [0.5 + 0.7 * np.cos(angle), 0.5 + 0.7 * np.sin(angle)]
|
||||||
|
|
||||||
|
# Additional positioning adjustment to make sure all circles
|
||||||
|
# are inside the square and don't overlap
|
||||||
|
# Clip to ensure everything is inside the unit square
|
||||||
|
centers = np.clip(centers, 0.01, 0.99)
|
||||||
|
|
||||||
|
# Compute maximum valid radii for this configuration
|
||||||
|
radii = compute_max_radii(centers)
|
||||||
|
return centers, radii
|
||||||
|
|
||||||
|
|
||||||
|
def compute_max_radii(centers):
|
||||||
|
"""
|
||||||
|
Compute the maximum possible radii for each circle position
|
||||||
|
such that they don't overlap and stay within the unit square.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
centers: np.array of shape (n, 2) with (x, y) coordinates
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
np.array of shape (n) with radius of each circle
|
||||||
|
"""
|
||||||
|
n = centers.shape[0]
|
||||||
|
radii = np.ones(n)
|
||||||
|
|
||||||
|
# First, limit by distance to square borders
|
||||||
|
for i in range(n):
|
||||||
|
x, y = centers[i]
|
||||||
|
# Distance to borders
|
||||||
|
radii[i] = min(x, y, 1 - x, 1 - y)
|
||||||
|
|
||||||
|
for i in range(n):
|
||||||
|
for j in range(i + 1, n):
|
||||||
|
dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2))
|
||||||
|
|
||||||
|
# Then, limit by distance to other circles
|
||||||
|
# Each pair of circles with centers at distance d can have
|
||||||
|
# sum of radii at most d to avoid overlap
|
||||||
|
for i in range(n):
|
||||||
|
for j in range(i + 1, n):
|
||||||
|
dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2))
|
||||||
|
|
||||||
|
# If current radii would cause overlap
|
||||||
|
if radii[i] + radii[j] > dist:
|
||||||
|
# Scale both radii proportionally
|
||||||
|
scale = dist / (radii[i] + radii[j])
|
||||||
|
radii[i] *= scale
|
||||||
|
radii[j] *= scale
|
||||||
|
|
||||||
|
# Ensure no negative radii due to numerical issues
|
||||||
|
radii = np.maximum(radii, 0.0)
|
||||||
|
return radii
|
||||||
|
|
||||||
|
|
||||||
|
# EVOLVE-BLOCK-END
|
||||||
|
|
||||||
|
|
||||||
|
# This part remains fixed (not evolved)
|
||||||
|
def run_packing():
|
||||||
|
"""Run the circle packing constructor for n=26"""
|
||||||
|
centers, radii = construct_packing()
|
||||||
|
# Calculate the sum of radii
|
||||||
|
sum_radii = np.sum(radii)
|
||||||
|
return centers, radii, sum_radii
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
"""候选代码的轻量静态检查（不执行候选的任何一行代码）。
|
||||||
|
|
||||||
|
实验员需要一个便宜的办法确认代码能跑；不给他这个办法，他就会自己跑一遍完整实验
|
||||||
|
去验证。这个脚本就是那个便宜的办法：只做三件事，全部基于抽象语法树。
|
||||||
|
|
||||||
|
1. 入口文件在不在
|
||||||
|
2. 候选目录下每个 ``*.py`` 能不能 ``ast.parse``
|
||||||
|
3. 只对入口文件：顶层恰好有一个叫 ``run_packing`` 的函数，且零个必填参数
|
||||||
|
|
||||||
|
用法：``python -B check-candidate.py <候选目录> <入口文件相对候选目录的路径> <结果文件>``。
|
||||||
|
结果写进 <结果文件>，一个 JSON 对象：``{"valid": bool, "reasons": [...]}``。
|
||||||
|
对候选代码的每一种意见都走"退出码 0 加 reasons"；非零退出说明这个脚本自己没跑成。
|
||||||
|
|
||||||
|
两条调用约定，改的时候不要破坏：
|
||||||
|
|
||||||
|
- **本文件不能放进候选目录。** 第 2 步从候选目录往下扫 ``*.py``，
|
||||||
|
脚本自己在里面就会把自己也扫进去。宿主把它放在候选目录的上一层。
|
||||||
|
- **只用标准库，``-B`` 要保留。** 不写 ``__pycache__``：检查本身不该往候选目录里添文件。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import sys
|
||||||
|
|
||||||
|
ENTRY_FUNCTION = "run_packing"
|
||||||
|
|
||||||
|
# 两种定义形式都要认，否则 `async def run_packing()` 会被判成"不存在"。
|
||||||
|
FUNCTION_NODES = (ast.FunctionDef, ast.AsyncFunctionDef)
|
||||||
|
|
||||||
|
|
||||||
|
def has_zero_required_arguments(node) -> bool:
|
||||||
|
"""判断函数是不是可以零参数调用。
|
||||||
|
|
||||||
|
位置参数（含仅位置参数）里没有默认值的那些必须为 0 个；
|
||||||
|
仅关键字参数里 kw_defaults 为 None 的表示没有默认值，一个都不能有。
|
||||||
|
"""
|
||||||
|
positional = len(node.args.posonlyargs) + len(node.args.args)
|
||||||
|
required_positional = positional - len(node.args.defaults)
|
||||||
|
keyword_without_default = any(
|
||||||
|
default is None for default in node.args.kw_defaults
|
||||||
|
)
|
||||||
|
return required_positional <= 0 and not keyword_without_default
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
if len(sys.argv) != 4:
|
||||||
|
print(
|
||||||
|
"用法: check-candidate.py <候选目录> <入口文件相对路径> <结果文件>",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
root = pathlib.Path(sys.argv[1])
|
||||||
|
entry_point = sys.argv[2]
|
||||||
|
reasons: list[str] = []
|
||||||
|
|
||||||
|
if not (root / entry_point).is_file():
|
||||||
|
reasons.append(f"入口文件不存在: {entry_point}")
|
||||||
|
|
||||||
|
# 第 3 步挂在第 2 步的 rglob("*.py") 上：入口文件名不以 .py 结尾，
|
||||||
|
# 循环扫不到它，入口函数那条判定会整个被跳过，却照样报通过。
|
||||||
|
if not entry_point.endswith(".py"):
|
||||||
|
reasons.append(
|
||||||
|
f"入口文件必须以 .py 结尾，否则入口函数检查扫不到它: {entry_point}"
|
||||||
|
)
|
||||||
|
|
||||||
|
for path in root.rglob("*.py"):
|
||||||
|
relative = str(path.relative_to(root))
|
||||||
|
try:
|
||||||
|
tree = ast.parse(path.read_text(encoding="utf-8"), filename=relative)
|
||||||
|
except (SyntaxError, ValueError) as error:
|
||||||
|
# SyntaxError：语法错；ValueError：含空字节或不是 UTF-8（UnicodeDecodeError 是它的子类）。
|
||||||
|
reasons.append(f"{relative} 解析失败: {error}")
|
||||||
|
continue
|
||||||
|
if relative != entry_point:
|
||||||
|
continue
|
||||||
|
candidates = [
|
||||||
|
node
|
||||||
|
for node in tree.body
|
||||||
|
if isinstance(node, FUNCTION_NODES) and node.name == ENTRY_FUNCTION
|
||||||
|
]
|
||||||
|
if len(candidates) != 1:
|
||||||
|
reasons.append(
|
||||||
|
f"{entry_point} 顶层的 {ENTRY_FUNCTION}() 应当恰好有一个定义，"
|
||||||
|
f"实际有 {len(candidates)} 个"
|
||||||
|
)
|
||||||
|
elif not has_zero_required_arguments(candidates[0]):
|
||||||
|
reasons.append(
|
||||||
|
f"{entry_point} 的 {ENTRY_FUNCTION}() 必须能零参数调用，"
|
||||||
|
f"当前有必填参数"
|
||||||
|
)
|
||||||
|
|
||||||
|
with open(sys.argv[3], "w", encoding="utf-8") as handle:
|
||||||
|
json.dump({"valid": not reasons, "reasons": reasons}, handle)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
"""填圆任务的运行 harness（宿主的固定工具，候选不得修改）。
|
||||||
|
|
||||||
|
它按文件路径导入候选程序、调用候选定义的 ``run_packing()``、把原始解原子落盘。
|
||||||
|
**它不判定解是否合法**：成绩只由独立校验器 ``validate.py`` 事后从实际产物重算。
|
||||||
|
|
||||||
|
指标的分工（docs/方法.md 第五部分"指标"）：
|
||||||
|
- 最终指标和合法性判定归 ``validate.py``，跟候选无关；
|
||||||
|
- 假设专用指标由研究者在协议里登记、实验员按定义在候选代码里实现并写出
|
||||||
|
``metrics.json``，宿主直接采信。harness 不内置任何指标算法。
|
||||||
|
|
||||||
|
命令行：python -B harness.py --program <program.py> --output <solution.json>
|
||||||
|
种子从环境变量 INFORSI_SEED 读，必须给出。
|
||||||
|
成功退出 0；任何失败都带 traceback 写 stderr 并非零退出。
|
||||||
|
|
||||||
|
工作目录里的三个文件（给实验员看的完整契约在同目录 ``TASK.md`` 的"指标怎么交给宿主"）：
|
||||||
|
- ``metrics-spec.json``：本次运行登记的假设专用指标定义，由宿主在准备运行目录时写好，
|
||||||
|
harness 不碰它。
|
||||||
|
- ``solution.json``（``--output`` 指定）：候选返回的原始解。
|
||||||
|
- ``metrics.json``：实验员的代码写出的假设专用指标取值，**键必须是 metrics-spec.json 里
|
||||||
|
那条指标的 ``id`` 原文**，值是有限数或 null，路径相对工作目录，
|
||||||
|
且必须在 ``run_packing()`` 返回之前写完。没有这个文件就表示这一次的假设专用指标一个
|
||||||
|
都没测到。另有两个保留键（docs/方法.md 第五部分"答案路径"）：``answer-path``（字符串，
|
||||||
|
最终交的答案出自哪条路径）和 ``safeguard-triggers``（对象，每条保障动手几次）。
|
||||||
|
它们不是指标、不是数，harness 原样保留，宿主读成辅助记录。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import sys
|
||||||
|
from typing import Any, cast
|
||||||
|
|
||||||
|
METRICS_FILE = "metrics.json"
|
||||||
|
# metrics.json 里不是数的两个保留键（docs/方法.md 第五部分"答案路径"）。finalize_metrics 只改写指标值，这两个原样留着：
|
||||||
|
# 改写成 null 的话，候选写对了的答案路径到宿主那里就成了"不认识的路径：null"，出错后无法察觉。
|
||||||
|
RESERVED_KEYS = ("answer-path", "safeguard-triggers")
|
||||||
|
|
||||||
|
|
||||||
|
def load_program(program_path: str):
|
||||||
|
"""按显式文件路径导入候选模块（不假定固定的 main.py 模式）。"""
|
||||||
|
spec = importlib.util.spec_from_file_location("candidate_program", program_path)
|
||||||
|
if spec is None or spec.loader is None:
|
||||||
|
raise RuntimeError(f"无法从 {program_path} 加载候选模块")
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def write_json_atomically(path: str, payload) -> None:
|
||||||
|
"""先写临时文件再 os.replace，让崩溃的运行永远不会留下半截产物。"""
|
||||||
|
temporary = path + ".tmp"
|
||||||
|
with open(temporary, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(payload, handle, allow_nan=False)
|
||||||
|
handle.flush()
|
||||||
|
os.fsync(handle.fileno())
|
||||||
|
os.replace(temporary, path)
|
||||||
|
directory_fd = os.open(
|
||||||
|
os.path.dirname(os.path.abspath(path)), os.O_RDONLY | os.O_DIRECTORY
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
os.fsync(directory_fd)
|
||||||
|
finally:
|
||||||
|
os.close(directory_fd)
|
||||||
|
|
||||||
|
|
||||||
|
def is_finite_number(value) -> bool:
|
||||||
|
"""有限数的定义：不是布尔值的 int/float，且 math.isfinite。
|
||||||
|
|
||||||
|
布尔值排除在外，否则 ``true`` 会被当成 1 记进证据。
|
||||||
|
超出浮点范围的大整数 math.isfinite 会抛 OverflowError，一并算作非有限。
|
||||||
|
"""
|
||||||
|
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
return math.isfinite(value)
|
||||||
|
except OverflowError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def finalize_metrics() -> None:
|
||||||
|
"""把实验员写出的 metrics.json 整理成宿主读得进去的形状，然后原子落盘。
|
||||||
|
|
||||||
|
只做一件事：指标的非有限值（NaN、±inf、布尔、非数值）**原样改写成 null**，其余照抄；
|
||||||
|
两个保留键（RESERVED_KEYS）不是指标，原样保留。
|
||||||
|
这正是"测不到就说测不到"。被改写的编号在 stderr 记一行，**退出码保持 0**。
|
||||||
|
|
||||||
|
为什么不抛错——这里抛错会让情况变坏，不是变好：
|
||||||
|
1. 它防不住自述要防的那件事。实验员把测不到的量写成 0.0 照样放行。
|
||||||
|
2. 它实际拦的几种情况宿主已经逐条拦过，而且处理温和得多：那一条记 null
|
||||||
|
加一句原因，成绩照常。
|
||||||
|
3. 抛错时 solution.json 已经完整落盘、解合法、成绩算得出来，但非零退出码
|
||||||
|
会让宿主判定这次运行没干净结束，最终指标也记成 null——一条坏指标白白
|
||||||
|
作废掉整次运行，而运行名额照扣。更糟的是那份裸 NaN 的 metrics.json 会
|
||||||
|
留在盘上，JS 的 JSON.parse 不接受裸 NaN，于是**这一轮所有候选指标一起
|
||||||
|
丢**，不只是坏的那一条。
|
||||||
|
|
||||||
|
它防的是"模型正常工作时就会做"的那类失手：实验员算某条比值时碰上退化布局
|
||||||
|
（分母是零半径）拿到 nan 没接住。改写成 null 之后，坏的那条变成 null 加原因，
|
||||||
|
好的那些照常，成绩和运行名额都不受影响。
|
||||||
|
|
||||||
|
**不**校验键必须是 metrics-spec.json 里声明过的那些：宿主只按登记的编号取值，
|
||||||
|
多出来的键不读。
|
||||||
|
|
||||||
|
文件不存在表示候选指标一个都没测到，不补零也不补空文件。
|
||||||
|
"""
|
||||||
|
if not os.path.exists(METRICS_FILE):
|
||||||
|
return
|
||||||
|
with open(METRICS_FILE, encoding="utf-8") as handle:
|
||||||
|
try:
|
||||||
|
measured = json.load(handle)
|
||||||
|
except ValueError as error:
|
||||||
|
# 同样不抛错，理由同上：整份读不出来是宿主那边看得见的一个状态
|
||||||
|
# （它会记一句"metrics.json 读不出来"），不该连累已经算出来的成绩。
|
||||||
|
print(f"{METRICS_FILE} 不是合法 JSON，原样留在盘上: {error}", file=sys.stderr)
|
||||||
|
return
|
||||||
|
if not isinstance(measured, dict):
|
||||||
|
print(
|
||||||
|
f"{METRICS_FILE} 必须是 JSON 对象（指标编号 -> 值），实际是 "
|
||||||
|
f"{type(measured).__name__}，原样留在盘上",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
rewritten: dict = {}
|
||||||
|
replaced: list[str] = []
|
||||||
|
for metric_id, value in measured.items():
|
||||||
|
if metric_id in RESERVED_KEYS or value is None or is_finite_number(value):
|
||||||
|
rewritten[metric_id] = value
|
||||||
|
else:
|
||||||
|
rewritten[metric_id] = None
|
||||||
|
replaced.append(f"{metric_id}={value!r}")
|
||||||
|
if replaced:
|
||||||
|
print(
|
||||||
|
f"{METRICS_FILE} 里这些指标的值不是有限数，已按'测不到就写 null'改写成 "
|
||||||
|
f"null: {', '.join(replaced)}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
write_json_atomically(METRICS_FILE, rewritten)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="运行填圆候选程序并落盘它返回的原始解。"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--program",
|
||||||
|
required=True,
|
||||||
|
help="定义了 run_packing() 的候选程序文件路径",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output",
|
||||||
|
required=True,
|
||||||
|
help="原始解 JSON 的写出路径（centers、radii、reported_sum）",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# 运行目录是候选目录整个复制来的，里面可能留着以前某次运行写的这两个输出文件。
|
||||||
|
# 候选这次没写时，宿主会把旧值当成这次的产物读走，所以先删掉，让读到的只可能是这次写的。
|
||||||
|
for leftover in (args.output, METRICS_FILE):
|
||||||
|
if os.path.exists(leftover):
|
||||||
|
os.remove(leftover)
|
||||||
|
|
||||||
|
# 在导入候选模块之前设种子。宿主给的是一个种子字符串（同一份研究的全部运行共用，
|
||||||
|
# docs/方法.md 第十部分要求评测确定），取它的 sha256 前 4 字节大端整数。
|
||||||
|
# 没给就报 KeyError 停下，不替它补一个。
|
||||||
|
np = importlib.import_module("numpy")
|
||||||
|
|
||||||
|
seed = int.from_bytes(
|
||||||
|
hashlib.sha256(os.environ["INFORSI_SEED"].encode()).digest()[:4],
|
||||||
|
"big",
|
||||||
|
)
|
||||||
|
random.seed(seed)
|
||||||
|
np.random.seed(seed)
|
||||||
|
|
||||||
|
module = load_program(args.program)
|
||||||
|
run_packing = getattr(module, "run_packing", None)
|
||||||
|
if not callable(run_packing):
|
||||||
|
raise RuntimeError(f"{args.program} 没有定义可调用的 run_packing()")
|
||||||
|
|
||||||
|
# 候选模块是动态加载的，下面的转换就是协议本身的强制点。
|
||||||
|
centers, radii, reported_sum = cast(Any, run_packing())
|
||||||
|
solution = {
|
||||||
|
"centers": [[float(x), float(y)] for x, y in centers],
|
||||||
|
"radii": [float(r) for r in radii],
|
||||||
|
"reported_sum": float(reported_sum),
|
||||||
|
}
|
||||||
|
write_json_atomically(args.output, solution)
|
||||||
|
|
||||||
|
finalize_metrics()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
numpy==2.3.1 # 数组与矩阵运算
|
||||||
|
scipy==1.18.1 # 数值优化、线性规划、空间距离计算等
|
||||||
|
numba==0.67.0 # 把 Python 循环编译成机器码
|
||||||
|
llvmlite==0.49.0 # numba 的编译后端
|
||||||
|
sympy==1.14.0 # 符号与精确计算
|
||||||
|
mpmath==1.3.0 # 任意精度浮点运算（sympy 依赖它）
|
||||||
|
networkx==3.7 # 图结构分析
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"kind": "circle_packing",
|
||||||
|
"finalMetric": { "id": "sum-of-radii", "direction": "maximize" },
|
||||||
|
"epsilon": 0.01,
|
||||||
|
"knownBest": 2.635983,
|
||||||
|
"run": { "wallSeconds": 120, "cores": 1, "runtime": "runc" },
|
||||||
|
"protectedFiles": ["harness.py", "validate.py", "check-candidate.py", "metrics-spec.json", "solution.json", "evaluate.py", "LICENSE", "SOURCES.md"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,353 @@
|
|||||||
|
"""填圆的独立校验器（宿主保管，候选不得修改）。
|
||||||
|
|
||||||
|
从实际产物重算一切。不导入官方 ShinkaEvolve 的 evaluate.py，也不读候选自报的分数。
|
||||||
|
|
||||||
|
成绩的定义：成绩就是下面那一句 ``math.fsum(radius_values)``。
|
||||||
|
只有这一处算它，所以不存在"两个成绩哪个算数"的问题。
|
||||||
|
|
||||||
|
**产物里的 ``reported_sum`` 这个字段照旧允许候选写**（``harness.py`` 就在写它），
|
||||||
|
校验器只是**根本不读它**：它不进判定、不进 diagnostics、不影响 ``valid``。
|
||||||
|
|
||||||
|
检查项（n = --circles，单位正方形 [0,1]^2）：
|
||||||
|
- centers 形状 (n, 2)，radii 形状 (n,)
|
||||||
|
- 全是有限数
|
||||||
|
- 半径不为负
|
||||||
|
- 每个圆在正方形内（容差 atol）
|
||||||
|
- 两两不重叠（容差 atol）
|
||||||
|
|
||||||
|
圆数、容差都没有默认值：忘了传就静默按旧值判，正是"改一处忘一处、没有任何东西会发现"
|
||||||
|
的那一类问题。所以 ``--circles``、``--atol`` 必填。
|
||||||
|
|
||||||
|
命令行：python -B validate.py --circles <n> --solution <solution.json> --atol <容差>
|
||||||
|
[--legalize] --verdict <verdict.json>
|
||||||
|
判定写进 ``--verdict`` 指定的文件（一个 JSON 对象），宿主读这个文件，不读标准输出。
|
||||||
|
退出码：0 = 合法，1 = 不合法，2 = 用法或读取出错（这时不写判定文件）。
|
||||||
|
校验器自己出错（例如算术异常）不兜住，带 traceback 非零退出、不写判定文件：
|
||||||
|
那是宿主的工具坏了，不能当成候选的失败。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# 半径分布那一行最多列几段；超了就中间省略，省略号让"截断了"这件事看得见。
|
||||||
|
PROFILE_MAX_GROUPS = 8
|
||||||
|
|
||||||
|
|
||||||
|
def _is_finite_number(value) -> bool:
|
||||||
|
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
return math.isfinite(value)
|
||||||
|
except OverflowError:
|
||||||
|
# JSON integers can exceed the range of the float geometry protocol.
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def render_radius_profile(radius_values) -> str:
|
||||||
|
"""把半径分布写成一行文本（判定里的 ``evidence.radius_profile``）。
|
||||||
|
|
||||||
|
降序排列，**数值完全相同**的连成一段写作 ``值 x个数``；只是四舍五入到小数点后
|
||||||
|
四位之后看着一样、实际不同的两个数仍然各占一段（所以会出现
|
||||||
|
``0.1580, 0.1580`` 这种相邻重复，它说明这是两个不同的数）。
|
||||||
|
|
||||||
|
例：``3 radii, desc: 0.2500 x2, 0.0000``
|
||||||
|
``26 radii, desc: 0.5000, 0.0000 x25``
|
||||||
|
"""
|
||||||
|
groups: list[list] = []
|
||||||
|
for value in sorted(radius_values, reverse=True):
|
||||||
|
if groups and groups[-1][0] == value:
|
||||||
|
groups[-1][1] += 1
|
||||||
|
else:
|
||||||
|
groups.append([value, 1])
|
||||||
|
rendered = [
|
||||||
|
f"{value:.4f}" if count == 1 else f"{value:.4f} x{count}"
|
||||||
|
for value, count in groups
|
||||||
|
]
|
||||||
|
if len(rendered) > PROFILE_MAX_GROUPS:
|
||||||
|
rendered = rendered[: PROFILE_MAX_GROUPS - 2] + ["...", rendered[-1]]
|
||||||
|
return f"{len(radius_values)} radii, desc: " + ", ".join(rendered)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_packing(centers, radii, circles, atol) -> dict:
|
||||||
|
"""Validate a packing and recompute its score independently.
|
||||||
|
|
||||||
|
``circles`` 是任务声明的圆数（调用方必须给，没有默认值）。
|
||||||
|
|
||||||
|
Returns a structured verdict dict. All applicable reasons are collected
|
||||||
|
(not just the first failure) so diagnostics stay informative.
|
||||||
|
"""
|
||||||
|
reasons: list[str] = []
|
||||||
|
diagnostics: dict = {
|
||||||
|
"declared_circles": circles,
|
||||||
|
"atol": atol,
|
||||||
|
"num_circles": None,
|
||||||
|
"max_boundary_violation": None,
|
||||||
|
"max_overlap_violation": None,
|
||||||
|
"min_pairwise_clearance": None,
|
||||||
|
"min_radius": None,
|
||||||
|
"max_radius": None,
|
||||||
|
"zero_radius_count": None,
|
||||||
|
"all_radii_finite": None,
|
||||||
|
}
|
||||||
|
evidence: dict = {"radius_profile": None}
|
||||||
|
computed_sum = None
|
||||||
|
|
||||||
|
def verdict() -> dict:
|
||||||
|
return {
|
||||||
|
"valid": len(reasons) == 0,
|
||||||
|
"computed_sum": computed_sum,
|
||||||
|
"reasons": reasons,
|
||||||
|
"diagnostics": diagnostics,
|
||||||
|
"evidence": evidence,
|
||||||
|
}
|
||||||
|
|
||||||
|
# 候选实际返回了几个圆：只要 centers 是个列表，这个数就是事实，先记下来。
|
||||||
|
# 数量不对时它正好说明错在哪里，记成 null 就看不出来了。
|
||||||
|
if isinstance(centers, list):
|
||||||
|
diagnostics["num_circles"] = len(centers)
|
||||||
|
|
||||||
|
# Shape and element checks.
|
||||||
|
shape_ok = (
|
||||||
|
isinstance(centers, list)
|
||||||
|
and len(centers) == circles
|
||||||
|
and all(isinstance(pair, list) and len(pair) == 2 for pair in centers)
|
||||||
|
and isinstance(radii, list)
|
||||||
|
and len(radii) == circles
|
||||||
|
)
|
||||||
|
if not shape_ok:
|
||||||
|
reasons.append(
|
||||||
|
f"shape incorrect: expected centers ({circles}, 2) and radii "
|
||||||
|
f"({circles},), got centers "
|
||||||
|
f"{(len(centers), [len(p) if isinstance(p, list) else type(p).__name__ for p in centers[:3]]) if isinstance(centers, list) else type(centers).__name__} "
|
||||||
|
f"radii {len(radii) if isinstance(radii, list) else type(radii).__name__}"
|
||||||
|
)
|
||||||
|
return verdict()
|
||||||
|
|
||||||
|
# 半径那几个诊断量只跟半径有关，圆心坏了不影响它们，所以先算。
|
||||||
|
radii_finite = all(_is_finite_number(r) for r in radii)
|
||||||
|
diagnostics["all_radii_finite"] = radii_finite
|
||||||
|
radius_values: list[float] | None = None
|
||||||
|
if radii_finite:
|
||||||
|
radius_values = [float(r) for r in radii]
|
||||||
|
diagnostics["min_radius"] = min(radius_values)
|
||||||
|
diagnostics["max_radius"] = max(radius_values)
|
||||||
|
diagnostics["zero_radius_count"] = sum(1 for r in radius_values if r == 0.0)
|
||||||
|
evidence["radius_profile"] = render_radius_profile(radius_values)
|
||||||
|
|
||||||
|
centers_finite = all(_is_finite_number(v) for pair in centers for v in pair)
|
||||||
|
if not (centers_finite and radii_finite):
|
||||||
|
reasons.append("non-finite values found in centers or radii")
|
||||||
|
# Cannot trust geometry with non-finite or non-numeric values; stop here.
|
||||||
|
return verdict()
|
||||||
|
|
||||||
|
# 走到这里 radii_finite 必然为真，radius_values 已经在上面算好。
|
||||||
|
points = [(float(pair[0]), float(pair[1])) for pair in centers]
|
||||||
|
negative = [i for i, r in enumerate(radius_values) if r < 0]
|
||||||
|
if negative:
|
||||||
|
reasons.append(f"negative radii at indices: {negative}")
|
||||||
|
# Negative radii make geometry checks meaningless; stop here.
|
||||||
|
return verdict()
|
||||||
|
|
||||||
|
# Independent recomputation of the score.
|
||||||
|
computed_sum = math.fsum(radius_values)
|
||||||
|
|
||||||
|
# Boundary check: circle i must lie inside [0,1]^2 within atol.
|
||||||
|
max_boundary = 0.0
|
||||||
|
boundary_indices = []
|
||||||
|
for i, ((x, y), r) in enumerate(zip(points, radius_values, strict=True)):
|
||||||
|
violation = max(r - x, x + r - 1.0, r - y, y + r - 1.0)
|
||||||
|
max_boundary = max(max_boundary, violation)
|
||||||
|
if violation > atol:
|
||||||
|
boundary_indices.append(i)
|
||||||
|
diagnostics["max_boundary_violation"] = max_boundary
|
||||||
|
if boundary_indices:
|
||||||
|
reasons.append(
|
||||||
|
f"circles outside unit square at indices: {boundary_indices} "
|
||||||
|
f"(max violation {max_boundary!r}, atol={atol})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Pairwise overlap check.
|
||||||
|
max_overlap = 0.0
|
||||||
|
min_clearance = math.inf
|
||||||
|
overlap_pairs = []
|
||||||
|
for i in range(circles):
|
||||||
|
xi, yi = points[i]
|
||||||
|
for j in range(i + 1, circles):
|
||||||
|
xj, yj = points[j]
|
||||||
|
dist = math.hypot(xi - xj, yi - yj)
|
||||||
|
clearance = dist - (radius_values[i] + radius_values[j])
|
||||||
|
min_clearance = min(min_clearance, clearance)
|
||||||
|
violation = -clearance
|
||||||
|
max_overlap = max(max_overlap, violation)
|
||||||
|
if violation > atol:
|
||||||
|
overlap_pairs.append((i, j))
|
||||||
|
diagnostics["max_overlap_violation"] = max_overlap
|
||||||
|
diagnostics["min_pairwise_clearance"] = (
|
||||||
|
min_clearance if math.isfinite(min_clearance) else None
|
||||||
|
)
|
||||||
|
if overlap_pairs:
|
||||||
|
reasons.append(
|
||||||
|
f"overlapping circle pairs: {overlap_pairs[:5]}"
|
||||||
|
f"{' ...' if len(overlap_pairs) > 5 else ''} "
|
||||||
|
f"(max violation {max_overlap!r}, atol={atol})"
|
||||||
|
)
|
||||||
|
|
||||||
|
return verdict()
|
||||||
|
|
||||||
|
|
||||||
|
def legalize_packing(centers, radii, circles, atol) -> dict:
|
||||||
|
"""只缩不移合法化（docs/方法.md 第八部分"合法化"，规则原文写在同目录 TASK.md）。
|
||||||
|
|
||||||
|
1. 先按 ``atol`` 判原产物。合法就原样返回，扣分 0。
|
||||||
|
2. 不合法时只缩半径、不挪圆心：**只缩参与违反的圆**——每个圆的半径减去它参与的
|
||||||
|
最大违反量（跟别的圆重叠的量、出边的量取最大）。
|
||||||
|
3. 缩完仍不合法（浮点舍入留下的零头），把这些圆的半径各往 0 挪一个最小浮点单位，
|
||||||
|
再判，直到合法。
|
||||||
|
4. 修不了：形状不对、含非有限值或负半径（这些不是"差一点"，缩半径解决不了）；
|
||||||
|
有圆心在正方形外面；这些圆的半径都缩到 0 仍不合法。
|
||||||
|
|
||||||
|
返回值顶层是**最后被判的那份摆法**的裁决（修好了就是修后的，修不了就是原来的），
|
||||||
|
外加 ``legalization`` 一节：原产物合不合法、原分数、原违反量、缩了哪些圆、修后半径。
|
||||||
|
扣掉多少分由宿主拿原分数减修后分数算，这里不另报一个数。
|
||||||
|
"""
|
||||||
|
raw = validate_packing(centers, radii, circles, atol=atol)
|
||||||
|
diagnostics = raw["diagnostics"]
|
||||||
|
violations = [
|
||||||
|
v
|
||||||
|
for v in (
|
||||||
|
diagnostics.get("max_overlap_violation"),
|
||||||
|
diagnostics.get("max_boundary_violation"),
|
||||||
|
)
|
||||||
|
if v is not None
|
||||||
|
]
|
||||||
|
record: dict = {
|
||||||
|
"raw_valid": raw["valid"],
|
||||||
|
"raw_score": raw["computed_sum"],
|
||||||
|
# 形状、有限性、负半径那几步提前返回时几何检查没做，原违反量量不到，记 null。
|
||||||
|
"raw_violation": max(violations) if len(violations) == 2 else None,
|
||||||
|
"applied": False,
|
||||||
|
"repaired": raw["valid"],
|
||||||
|
"shrunk_circles": [],
|
||||||
|
"legal_radii": None,
|
||||||
|
"reasons": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
def finish(verdict: dict) -> dict:
|
||||||
|
return {**verdict, "legalization": record}
|
||||||
|
|
||||||
|
if raw["valid"]:
|
||||||
|
return finish(raw)
|
||||||
|
if raw["computed_sum"] is None:
|
||||||
|
record["reasons"].append(
|
||||||
|
"形状不对、含非有限值或有负半径：这不是差一点的违规，缩半径修不了"
|
||||||
|
)
|
||||||
|
return finish(raw)
|
||||||
|
|
||||||
|
points = [(float(x), float(y)) for x, y in centers]
|
||||||
|
outside = [i for i, (x, y) in enumerate(points) if not (0.0 <= x <= 1.0 and 0.0 <= y <= 1.0)]
|
||||||
|
if outside:
|
||||||
|
record["reasons"].append(
|
||||||
|
f"圆心在正方形外面（编号 {outside}）：只缩半径不挪圆心，修不了"
|
||||||
|
)
|
||||||
|
return finish(raw)
|
||||||
|
|
||||||
|
radius_values = [float(r) for r in radii]
|
||||||
|
# 每个圆参与的最大违反量。只有超过 atol 的才算违反，减的是违反量本身。
|
||||||
|
shrink = [0.0] * circles
|
||||||
|
for i, ((x, y), r) in enumerate(zip(points, radius_values, strict=True)):
|
||||||
|
boundary = max(r - x, x + r - 1.0, r - y, y + r - 1.0)
|
||||||
|
if boundary > atol:
|
||||||
|
shrink[i] = boundary
|
||||||
|
for i in range(circles):
|
||||||
|
xi, yi = points[i]
|
||||||
|
for j in range(i + 1, circles):
|
||||||
|
xj, yj = points[j]
|
||||||
|
overlap = radius_values[i] + radius_values[j] - math.hypot(xi - xj, yi - yj)
|
||||||
|
if overlap > atol:
|
||||||
|
shrink[i] = max(shrink[i], overlap)
|
||||||
|
shrink[j] = max(shrink[j], overlap)
|
||||||
|
involved = [i for i in range(circles) if shrink[i] > 0.0]
|
||||||
|
record["applied"] = True
|
||||||
|
record["shrunk_circles"] = involved
|
||||||
|
|
||||||
|
legal = [max(r - s, 0.0) for r, s in zip(radius_values, shrink, strict=True)]
|
||||||
|
while True:
|
||||||
|
verdict = validate_packing(centers, legal, circles, atol=atol)
|
||||||
|
if verdict["valid"]:
|
||||||
|
record["repaired"] = True
|
||||||
|
record["legal_radii"] = legal
|
||||||
|
return finish(verdict)
|
||||||
|
nudged = [
|
||||||
|
math.nextafter(legal[i], 0.0) if shrink[i] > 0.0 else legal[i]
|
||||||
|
for i in range(circles)
|
||||||
|
]
|
||||||
|
if nudged == legal:
|
||||||
|
record["reasons"].append(
|
||||||
|
f"参与违反的圆（编号 {involved}）半径都缩到 0 仍不合法，修不了"
|
||||||
|
)
|
||||||
|
return finish(raw)
|
||||||
|
legal = nudged
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Independent circle-packing validator (unit square)."
|
||||||
|
)
|
||||||
|
parser.add_argument("--solution", required=True, help="solution JSON path")
|
||||||
|
parser.add_argument(
|
||||||
|
"--circles",
|
||||||
|
type=int,
|
||||||
|
required=True,
|
||||||
|
help="how many circles the task declares (positive integer)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--atol",
|
||||||
|
type=float,
|
||||||
|
required=True,
|
||||||
|
help="absolute tolerance for boundary/overlap checks",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--legalize",
|
||||||
|
action="store_true",
|
||||||
|
help=(
|
||||||
|
"原产物不合法时按只缩不移规则修（见 legalize_packing），"
|
||||||
|
"输出多一节 legalization；不加时只判不修"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--verdict", required=True, help="判定写进这个文件（JSON 对象）"
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
if args.circles < 1:
|
||||||
|
print("--circles must be a positive integer", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
if not (math.isfinite(args.atol) and args.atol >= 0):
|
||||||
|
print("--atol must be finite and non-negative", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(args.solution, encoding="utf-8") as handle:
|
||||||
|
solution = json.load(handle)
|
||||||
|
except (OSError, ValueError, RecursionError) as error:
|
||||||
|
print(f"cannot read solution: {error}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
if not isinstance(solution, dict):
|
||||||
|
print("solution must be a JSON object", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
judge = legalize_packing if args.legalize else validate_packing
|
||||||
|
verdict = judge(
|
||||||
|
solution.get("centers"),
|
||||||
|
solution.get("radii"),
|
||||||
|
args.circles,
|
||||||
|
atol=args.atol,
|
||||||
|
)
|
||||||
|
with open(args.verdict, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(verdict, handle, allow_nan=False)
|
||||||
|
return 0 if verdict["valid"] else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"centers": [[0.100484056446833, 0.07485776462601139], [0.10728235286470139, 0.2738532442085582], [0.086108914434533, 0.5223893812779171], [0.10580193867928778, 0.6806936405144427], [0.0856033942977713, 0.9222111281543698], [0.3120328263981882, 0.10480425665863033], [0.32716502959347243, 0.3067558057750085], [0.2897009711447359, 0.47507350932200926], [0.2935854248884542, 0.7104275384094612], [0.27237639359908744, 0.8951262487554578], [0.4914744630822173, 0.08989291879330835], [0.5100326585018874, 0.29884106286561407], [0.5213978072978597, 0.4946336955389669], [0.5151435556465788, 0.7026215638204316], [0.49451516606288104, 0.901030906906145], [0.721963590658637, 0.11913688062063861], [0.7296948042290226, 0.2744414856202059], [0.7002013752356513, 0.5129578551833195], [0.7069421474478524, 0.7067135961639011], [0.7178150742617755, 0.9143740466217667], [0.8917011179459332, 0.11647248318508979], [0.9275227999594169, 0.2848229492760654], [0.9028755468513529, 0.5143032303369243], [0.9178606846823245, 0.681413920131626], [0.9136434762753853, 0.8933655911924985], [0.18562920767437355, 0.7769807813776922]], "radii": [0.06361218681134884, 0.0441172439828679, 0.03913703570214593, 0.027074375269719862, 0.07778887184563021, 0.05730567971921061, 0.029286768433871176, 0.058739899233888004, 0.050016382123936116, 0.07485971141413426, 0.07105090920173604, 0.04742662315759923, 0.039360492244268, 0.05993669686613301, 0.0893218390023967, 0.06428677645874871, 0.08948690383353296, 0.09702962465194793, 0.09684333677627958, 0.08562595337823331, 0.1020164679124565, 0.07010288699954595, 0.09090324918001258, 0.07687796847999327, 0.08635652372461466, 0.07171254287625847], "reported_sum": 1.76027694928051}
|
||||||
@@ -0,0 +1,224 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { spawnSync } from "node:child_process";
|
||||||
|
import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { after, test, type TestContext } from "node:test";
|
||||||
|
import { Runner } from "../src/runner/index.ts";
|
||||||
|
import type { ProcessResult, RunDefinition } from "../src/vocab/index.ts";
|
||||||
|
|
||||||
|
// 用真实 Docker 测（任务书 05 第五部分）：没有 Docker 时整个文件失败，不跳过。
|
||||||
|
// 镜像、容器名一律 inforsi-test- 开头的固定名字；本机还有别的项目在建删容器，只动自己前缀的。
|
||||||
|
|
||||||
|
const TASK_DIR = join(import.meta.dirname, "..", "tasks", "circle_packing");
|
||||||
|
const IMAGE_TAG = "inforsi-test-circle";
|
||||||
|
const MISSING_DIGEST = `sha256:${"0".repeat(64)}`;
|
||||||
|
|
||||||
|
const runner = new Runner(null);
|
||||||
|
const digest = await runner.buildImage(TASK_DIR, IMAGE_TAG);
|
||||||
|
after(() => docker(["rmi", IMAGE_TAG]));
|
||||||
|
|
||||||
|
function docker(args: string[]): { status: number | null; stdout: string; stderr: string } {
|
||||||
|
return spawnSync("docker", args, { encoding: "utf8" });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 删掉测试自己的容器；不存在不算错，别的错误照样报。 */
|
||||||
|
function forceRemove(name: string): void {
|
||||||
|
const removed = docker(["rm", "-f", name]);
|
||||||
|
if (removed.status !== 0 && !removed.stderr.includes("No such container")) throw new Error(removed.stderr);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 一个测试用的容器名：先清掉上次没删干净的同名容器，测试结束后再删。 */
|
||||||
|
function containerName(t: TestContext, suffix: string): string {
|
||||||
|
const name = `inforsi-test-${suffix}`;
|
||||||
|
forceRemove(name);
|
||||||
|
t.after(() => forceRemove(name));
|
||||||
|
return name;
|
||||||
|
}
|
||||||
|
|
||||||
|
function dirs(t: TestContext): { runDir: string; logDir: string } {
|
||||||
|
const root = mkdtempSync(join(tmpdir(), "inforsi-runner-"));
|
||||||
|
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||||
|
const runDir = join(root, "run");
|
||||||
|
const logDir = join(root, "log");
|
||||||
|
mkdirSync(runDir);
|
||||||
|
mkdirSync(logDir);
|
||||||
|
return { runDir, logDir };
|
||||||
|
}
|
||||||
|
|
||||||
|
function definition(command: string[], wallSeconds: number, image: string): RunDefinition {
|
||||||
|
return {
|
||||||
|
command,
|
||||||
|
workdir: "/work",
|
||||||
|
image: { digest: image, runtime: "runc" },
|
||||||
|
wallSeconds,
|
||||||
|
cores: 1,
|
||||||
|
inheritEnv: false,
|
||||||
|
env: {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const python = (code: string): string[] => ["python3", "-c", code];
|
||||||
|
|
||||||
|
/** 起一个容器并等它结束，交回结果和日志目录。 */
|
||||||
|
async function runToEnd(t: TestContext, suffix: string, command: string[], wallSeconds: number) {
|
||||||
|
const name = containerName(t, suffix);
|
||||||
|
const { runDir, logDir } = dirs(t);
|
||||||
|
const outcome = await runner.start(name, definition(command, wallSeconds, digest), runDir);
|
||||||
|
assert.deepEqual(outcome, { started: true });
|
||||||
|
const result = await runner.wait(name, logDir);
|
||||||
|
return { name, result, logDir };
|
||||||
|
}
|
||||||
|
|
||||||
|
function value(result: ProcessResult, key: "cpuSeconds" | "wallSeconds" | "peakMemoryBytes"): number {
|
||||||
|
assert.notEqual(result.ending, "notStarted");
|
||||||
|
if (result.ending === "notStarted") throw new Error("unreachable");
|
||||||
|
const item = result[key];
|
||||||
|
assert.equal(item.status, "measured", `${key} 应当测到`);
|
||||||
|
if (item.status !== "measured") throw new Error("unreachable");
|
||||||
|
return item.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
test("建镜像命中参考项目的缓存；查镜像：有的答有，没有的答没有", async () => {
|
||||||
|
assert.match(digest, /^sha256:7ed52793365c/);
|
||||||
|
assert.equal(await runner.hasImage(digest), true);
|
||||||
|
assert.equal(await runner.hasImage(MISSING_DIGEST), false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("正常退出：带退出码和三项资源用量，标准输出、标准错误存成两个文件", async (t) => {
|
||||||
|
const code = "import sys; print('out'); print('err', file=sys.stderr); sys.exit(3)";
|
||||||
|
const { result, logDir } = await runToEnd(t, "exit", python(code), 30);
|
||||||
|
assert.equal(result.ending, "exited");
|
||||||
|
assert.equal(result.ending === "exited" && result.exitCode, 3);
|
||||||
|
assert.ok(value(result, "cpuSeconds") > 0);
|
||||||
|
assert.ok(value(result, "wallSeconds") > 0 && value(result, "wallSeconds") < 30);
|
||||||
|
assert.ok(value(result, "peakMemoryBytes") > 0);
|
||||||
|
assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "out\n");
|
||||||
|
assert.equal(readFileSync(join(logDir, "stderr.log"), "utf8"), "err\n");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("超时：结局 timeout，CPU 秒接近时限，墙钟不短于时限", async (t) => {
|
||||||
|
const { result } = await runToEnd(t, "timeout", python("while True: pass"), 2);
|
||||||
|
assert.equal(result.ending, "timeout");
|
||||||
|
const cpu = value(result, "cpuSeconds");
|
||||||
|
assert.ok(cpu > 1.7 && cpu < 2.3, `CPU 秒 ${cpu}`);
|
||||||
|
assert.ok(value(result, "wallSeconds") >= 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("候选伪造计量结果再死循环：伪造不成，结局仍是 timeout，资源用量是真实测到的；写出的文件归当前用户", async (t) => {
|
||||||
|
// 审阅时实测过的手法：先写一份假的 usage，再把目录改成只读，然后死循环到超时。
|
||||||
|
const code = [
|
||||||
|
"import os",
|
||||||
|
"for attempt in (lambda: open('/inforsi-record/usage', 'w').write('exited 0 1 1 1\\n'), lambda: os.chmod('/inforsi-record', 0o555)):",
|
||||||
|
" try:",
|
||||||
|
" attempt()",
|
||||||
|
" except OSError as error:",
|
||||||
|
" print(type(error).__name__, flush=True)",
|
||||||
|
"open('out.txt', 'w').write('x')",
|
||||||
|
"while True: pass",
|
||||||
|
].join("\n");
|
||||||
|
const name = containerName(t, "forge");
|
||||||
|
const { runDir, logDir } = dirs(t);
|
||||||
|
assert.deepEqual(await runner.start(name, definition(python(code), 2, digest), runDir), { started: true });
|
||||||
|
const result = await runner.wait(name, logDir);
|
||||||
|
assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "PermissionError\nPermissionError\n");
|
||||||
|
assert.equal(result.ending, "timeout");
|
||||||
|
const cpu = value(result, "cpuSeconds");
|
||||||
|
t.diagnostic(`结局 ${result.ending}，CPU 秒 ${cpu}，墙钟秒 ${value(result, "wallSeconds")}`);
|
||||||
|
assert.ok(cpu > 1.7 && cpu < 2.3, `CPU 秒 ${cpu}`);
|
||||||
|
assert.ok(value(result, "wallSeconds") >= 2);
|
||||||
|
assert.equal(statSync(join(runDir, "out.txt")).uid, process.getuid?.());
|
||||||
|
});
|
||||||
|
|
||||||
|
test("命令自己以 124 或 137 退出，不判成超时", async (t) => {
|
||||||
|
const own124 = await runToEnd(t, "exit124", python("import sys; sys.exit(124)"), 30);
|
||||||
|
assert.equal(own124.result.ending, "exited");
|
||||||
|
assert.equal(own124.result.ending === "exited" && own124.result.exitCode, 124);
|
||||||
|
const own137 = await runToEnd(t, "exit137", python("import os; os.kill(os.getpid(), 9)"), 30);
|
||||||
|
assert.equal(own137.result.ending, "exited");
|
||||||
|
assert.equal(own137.result.ending === "exited" && own137.result.exitCode, 137);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("从外面 docker kill：结局 cancelled，三项资源都测不到并写明原因", async (t) => {
|
||||||
|
const name = containerName(t, "killed");
|
||||||
|
const { runDir, logDir } = dirs(t);
|
||||||
|
const outcome = await runner.start(name, definition(python("import time; time.sleep(60)"), 120, digest), runDir);
|
||||||
|
assert.deepEqual(outcome, { started: true });
|
||||||
|
assert.equal(await runner.status(name), "running");
|
||||||
|
const killed = docker(["kill", name]);
|
||||||
|
assert.equal(killed.status, 0, killed.stderr);
|
||||||
|
const result = await runner.wait(name, logDir);
|
||||||
|
assert.equal(result.ending, "cancelled");
|
||||||
|
if (result.ending !== "cancelled") return;
|
||||||
|
for (const item of [result.cpuSeconds, result.wallSeconds, result.peakMemoryBytes]) {
|
||||||
|
assert.equal(item.status, "unmeasured");
|
||||||
|
assert.match(item.status === "unmeasured" ? item.reason : "", /从外面被结束/);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("镜像编号不存在：没启动，原因是 Docker 的报错原文，也没留下容器", async (t) => {
|
||||||
|
const name = containerName(t, "noimage");
|
||||||
|
const { runDir } = dirs(t);
|
||||||
|
const outcome = await runner.start(name, definition(python("pass"), 30, MISSING_DIGEST), runDir);
|
||||||
|
assert.equal(outcome.started, false);
|
||||||
|
assert.match(outcome.started ? "" : outcome.reason, /No such image/);
|
||||||
|
assert.equal(await runner.status(name), "absent");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("断网：容器里连不上外网", async (t) => {
|
||||||
|
const code = [
|
||||||
|
"import socket, sys",
|
||||||
|
"try:",
|
||||||
|
" socket.create_connection(('1.1.1.1', 53), timeout=3)",
|
||||||
|
"except OSError as error:",
|
||||||
|
" print(type(error).__name__); sys.exit(0)",
|
||||||
|
"sys.exit(1)",
|
||||||
|
].join("\n");
|
||||||
|
const { result } = await runToEnd(t, "network", python(code), 30);
|
||||||
|
assert.equal(result.ending === "exited" && result.exitCode, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("核数：容器里 os.cpu_count() 报 1", async (t) => {
|
||||||
|
const { logDir } = await runToEnd(t, "cores", python("import os; print(os.cpu_count())"), 30);
|
||||||
|
assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "1\n");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("查状态：没有这个容器、建了没启动、在跑、已结束、问不到", async (t) => {
|
||||||
|
assert.equal(await runner.status(containerName(t, "status-absent")), "absent");
|
||||||
|
|
||||||
|
const created = containerName(t, "status-created");
|
||||||
|
const create = docker(["create", "--name", created, digest, "true"]);
|
||||||
|
assert.equal(create.status, 0, create.stderr);
|
||||||
|
assert.equal(await runner.status(created), "created");
|
||||||
|
|
||||||
|
const name = containerName(t, "status-running");
|
||||||
|
const { runDir, logDir } = dirs(t);
|
||||||
|
await runner.start(name, definition(python("import time; time.sleep(3)"), 30, digest), runDir);
|
||||||
|
assert.equal(await runner.status(name), "running");
|
||||||
|
await runner.wait(name, logDir);
|
||||||
|
assert.equal(await runner.status(name), "exited");
|
||||||
|
|
||||||
|
const nowhere = new Runner("unix:///nonexistent/docker.sock");
|
||||||
|
assert.equal(await nowhere.status(name), "unreachable");
|
||||||
|
|
||||||
|
await runner.remove(name);
|
||||||
|
assert.equal(await runner.status(name), "absent");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("等结束能接上另一个 Runner 实例启动的容器；已结束的容器再等一次，结果相同", async (t) => {
|
||||||
|
const name = containerName(t, "resume");
|
||||||
|
const { runDir, logDir } = dirs(t);
|
||||||
|
const starter = new Runner(null);
|
||||||
|
await starter.start(name, definition(python("import time; time.sleep(2); print('done')"), 30, digest), runDir);
|
||||||
|
const resumed = new Runner(null);
|
||||||
|
const first = await resumed.wait(name, logDir);
|
||||||
|
assert.equal(first.ending === "exited" && first.exitCode, 0);
|
||||||
|
assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "done\n");
|
||||||
|
const again = await new Runner(null).wait(name, logDir);
|
||||||
|
assert.deepEqual(again, first);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("容器名不符合 Docker 的要求时报错", async () => {
|
||||||
|
await assert.rejects(runner.status("inforsi test"), /不符合 Docker 的要求/);
|
||||||
|
await assert.rejects(runner.status("-x"), /不符合 Docker 的要求/);
|
||||||
|
});
|
||||||
@@ -0,0 +1,460 @@
|
|||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { spawnSync } from "node:child_process";
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { cpSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { after, test, type TestContext } from "node:test";
|
||||||
|
import { Runner } from "../src/runner/index.ts";
|
||||||
|
import {
|
||||||
|
checkTaskFacts,
|
||||||
|
formalRun,
|
||||||
|
judge,
|
||||||
|
lightCheck,
|
||||||
|
loadTask,
|
||||||
|
metricDefinitions,
|
||||||
|
prepareJudging,
|
||||||
|
prepareRun,
|
||||||
|
protectedChanges,
|
||||||
|
readLightCheck,
|
||||||
|
} from "../src/tasks/index.ts";
|
||||||
|
import type { HypothesisMetric, JudgedResult, Measured, ProcessResult, RunDefinition } from "../src/vocab/index.ts";
|
||||||
|
|
||||||
|
// tasks 的测试（任务书 05 第五部分）。凡是要执行任务代码的，都经真实 Docker 在任务镜像里跑：
|
||||||
|
// 没有 Docker 时整个文件失败，不跳过。镜像和容器一律用 inforsi-test-tasks- 开头的固定名字，
|
||||||
|
// 与 runner 的测试（inforsi-test-circle、inforsi-test-<用途>）错开，两个文件并行跑也不互相删。
|
||||||
|
// 判分的预期值是手算的 2 的幂有理数布局（借自参考项目 tests/circle-validator.test.ts），
|
||||||
|
// 加一份参考项目真实运行里差一点不合法的产物（tests/fixtures/legalization/）。
|
||||||
|
|
||||||
|
const TASK_DIR = join(import.meta.dirname, "..", "tasks", "circle_packing");
|
||||||
|
const IMAGE_TAG = "inforsi-test-tasks";
|
||||||
|
const FIXTURE = join(import.meta.dirname, "fixtures", "legalization", "14-real-1-keep-grid-start.json");
|
||||||
|
|
||||||
|
const task = loadTask(TASK_DIR);
|
||||||
|
const runner = new Runner(null);
|
||||||
|
const digest = await runner.buildImage(task.dir, IMAGE_TAG);
|
||||||
|
after(() => docker(["rmi", IMAGE_TAG]));
|
||||||
|
|
||||||
|
function docker(args: string[]): { status: number | null; stderr: string } {
|
||||||
|
return spawnSync("docker", args, { encoding: "utf8" });
|
||||||
|
}
|
||||||
|
|
||||||
|
function forceRemove(name: string): void {
|
||||||
|
const removed = docker(["rm", "-f", name]);
|
||||||
|
if (removed.status !== 0 && !removed.stderr.includes("No such container")) throw new Error(removed.stderr);
|
||||||
|
}
|
||||||
|
|
||||||
|
function tempDir(t: TestContext, label: string): string {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), `inforsi-tasks-${label}-`));
|
||||||
|
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 让 runner 在任务镜像里跑一份运行定义，交回结果。 */
|
||||||
|
async function run(t: TestContext, suffix: string, definition: RunDefinition, dir: string): Promise<ProcessResult> {
|
||||||
|
const name = `inforsi-test-tasks-${suffix}`;
|
||||||
|
forceRemove(name);
|
||||||
|
t.after(() => forceRemove(name));
|
||||||
|
const logDir = tempDir(t, "log");
|
||||||
|
const outcome = await runner.start(name, definition, dir);
|
||||||
|
assert.deepEqual(outcome, { started: true });
|
||||||
|
return runner.wait(name, logDir);
|
||||||
|
}
|
||||||
|
|
||||||
|
const m = (value: number): Measured => ({ status: "measured", value });
|
||||||
|
|
||||||
|
/** 一次正常结束的正式运行；资源用量是随手写的，判分只是原样转记。 */
|
||||||
|
const EXITED_OK: ProcessResult = { ending: "exited", exitCode: 0, cpuSeconds: m(1.5), wallSeconds: m(2.5), peakMemoryBytes: m(1024) };
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 手算布局（2 的幂的有理数，双精度里精确）
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
interface Solution {
|
||||||
|
centers: number[][];
|
||||||
|
radii: number[];
|
||||||
|
reported_sum: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 一行等间距圆：圆心 (firstX + i·spacing, 0.5)，半径统一。 */
|
||||||
|
function rowLayout(radius: number, spacing: number, firstX: number): Solution {
|
||||||
|
const centers: number[][] = [];
|
||||||
|
const radii: number[] = [];
|
||||||
|
for (let i = 0; i < 26; i += 1) {
|
||||||
|
centers.push([firstX + i * spacing, 0.5]);
|
||||||
|
radii.push(radius);
|
||||||
|
}
|
||||||
|
return { centers, radii, reported_sum: 99 };
|
||||||
|
}
|
||||||
|
|
||||||
|
// 宽松布局：半径 1/128，间距 1/32，首个圆心 7/64。相邻间隙 1/32 − 1/64 = 1/64 > 0；
|
||||||
|
// 最右 7/64 + 25/32 + 1/128 = 0.8984375 < 1。半径和 26/128 = 0.203125。
|
||||||
|
const loose = (): Solution => rowLayout(1 / 128, 1 / 32, 7 / 64);
|
||||||
|
// 相切布局：半径 1/64，间距 1/32 恰好等于两半径之和，间隙恰好 0。半径和 26/64 = 0.40625。
|
||||||
|
const tangent = (): Solution => rowLayout(1 / 64, 1 / 32, 7 / 64);
|
||||||
|
|
||||||
|
/** 判分走完两步：有产物就在镜像里跑校验器。runDir 里放好 solution.json（或不放）。 */
|
||||||
|
async function judgeRun(
|
||||||
|
t: TestContext,
|
||||||
|
suffix: string,
|
||||||
|
runDir: string,
|
||||||
|
formal: ProcessResult,
|
||||||
|
metrics: HypothesisMetric[],
|
||||||
|
): Promise<JudgedResult> {
|
||||||
|
const judgeDir = tempDir(t, "judge");
|
||||||
|
const step = prepareJudging(task, runDir, judgeDir, digest, formal, metrics);
|
||||||
|
if (step.status === "judged") return step.result;
|
||||||
|
const validator = await run(t, suffix, step.definition, judgeDir);
|
||||||
|
return judge(task, runDir, judgeDir, formal, validator, metrics);
|
||||||
|
}
|
||||||
|
|
||||||
|
function runDirWith(t: TestContext, solution: Solution | string | null): string {
|
||||||
|
const dir = tempDir(t, "run");
|
||||||
|
if (solution !== null) {
|
||||||
|
writeFileSync(join(dir, "solution.json"), typeof solution === "string" ? solution : JSON.stringify(solution));
|
||||||
|
}
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
function value(item: Measured | undefined): number {
|
||||||
|
assert.equal(item?.status, "measured", JSON.stringify(item));
|
||||||
|
return (item as { value: number }).value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function reason(item: Measured | undefined): string {
|
||||||
|
assert.equal(item?.status, "unmeasured", JSON.stringify(item));
|
||||||
|
return (item as { reason: string }).reason;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 任务目录与 task.json
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test("基线三个文件的 sha256 等于 SOURCES.md 表上的值", () => {
|
||||||
|
// 期望值从 baseline/SOURCES.md 的表格手抄，不调项目里的哈希函数。
|
||||||
|
const expected: Array<[string, string]> = [
|
||||||
|
["initial.py", "09da8428a85792a94a9c6cc2193b6a0db1090ce598a17b75bc0db48d0953d098"],
|
||||||
|
["evaluate.py", "eb8b9787dc25dee24a5f5305d1e3bd876e5c336cded1c9bab7fb204612928b4d"],
|
||||||
|
["LICENSE", "c3e5b1c26607caf7208d5a105c113fe240630bb873fdb8ca5b4c2e32c721ddaf"],
|
||||||
|
];
|
||||||
|
for (const [name, sha] of expected) {
|
||||||
|
assert.equal(createHash("sha256").update(readFileSync(join(task.baselineDir, name))).digest("hex"), sha, name);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("读任务：task.json 的值、Dockerfile 所在目录、TASK.md 全文；最终指标和辅助指标的定义", () => {
|
||||||
|
assert.deepEqual(task.facts, JSON.parse(readFileSync(join(TASK_DIR, "task.json"), "utf8")));
|
||||||
|
assert.equal(task.facts.finalMetric.id, "sum-of-radii");
|
||||||
|
assert.equal(task.facts.epsilon, 0.01);
|
||||||
|
assert.equal(task.facts.knownBest, 2.635983);
|
||||||
|
assert.deepEqual(task.facts.run, { wallSeconds: 120, cores: 1, runtime: "runc" });
|
||||||
|
assert.ok(existsSync(join(task.dir, "Dockerfile")));
|
||||||
|
assert.match(task.guide, /^# 填圆任务/);
|
||||||
|
const ids = metricDefinitions(task).map((d) => `${d.layer}:${d.id}`);
|
||||||
|
assert.deepEqual(ids, [
|
||||||
|
"final:sum-of-radii",
|
||||||
|
"auxiliary:wall-seconds",
|
||||||
|
"auxiliary:cpu-seconds",
|
||||||
|
"auxiliary:peak-memory-bytes",
|
||||||
|
"auxiliary:min-pairwise-clearance",
|
||||||
|
"auxiliary:raw-score",
|
||||||
|
"auxiliary:raw-violation",
|
||||||
|
"auxiliary:legalization-deduction",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("task.json：缺字段、多字段、类型不对，一次全部列出，带字段路径", () => {
|
||||||
|
const good = JSON.parse(readFileSync(join(TASK_DIR, "task.json"), "utf8")) as Record<string, unknown>;
|
||||||
|
const bad = {
|
||||||
|
...good,
|
||||||
|
epsilon: "0.01",
|
||||||
|
knownBest: "unknown",
|
||||||
|
run: { wallSeconds: 120, cores: 1.5 },
|
||||||
|
extra: true,
|
||||||
|
} as Record<string, unknown>;
|
||||||
|
delete bad.protectedFiles;
|
||||||
|
assert.throws(
|
||||||
|
() => checkTaskFacts(bad),
|
||||||
|
(error: Error) => {
|
||||||
|
const lines = error.message.split("\n").slice(1);
|
||||||
|
assert.match(error.message, /^task\.json 不合格，共 6 处/);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/protectedFiles：缺少")), error.message);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/extra：不认识的字段")), error.message);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/epsilon：")), error.message);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/knownBest：应为数字，没有就写 null")), error.message);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/run/cores：")), error.message);
|
||||||
|
assert.ok(lines.some((l) => l.startsWith("/run/runtime：缺少")), error.message);
|
||||||
|
return true;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
assert.deepEqual(checkTaskFacts({ ...good, knownBest: null }).knownBest, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("读任务：task.json 不是合法 JSON 时报错带文件路径", (t) => {
|
||||||
|
const dir = tempDir(t, "badjson");
|
||||||
|
writeFileSync(join(dir, "task.json"), "{ not json");
|
||||||
|
assert.throws(() => loadTask(dir), /task\.json 读不出来或不是合法的 JSON/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("不许改的文件：只交回改动名单里属于 task.json 名单的那些", () => {
|
||||||
|
assert.deepEqual(protectedChanges(task, ["initial.py", "harness.py", "lib/util.py", "validate.py"]), ["harness.py", "validate.py"]);
|
||||||
|
assert.deepEqual(protectedChanges(task, ["initial.py"]), []);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 判定表（任务书 05 第三部分"判定怎么定"）
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test("判定表 1：正式运行超时、被从外面结束、退出码非 0，不算数，不跑校验器，资源用量照记", async (t) => {
|
||||||
|
const un: Measured = { status: "unmeasured", reason: "容器从外面被结束" };
|
||||||
|
const cases: Array<[ProcessResult, RegExp]> = [
|
||||||
|
[{ ending: "timeout", cpuSeconds: m(119), wallSeconds: m(120), peakMemoryBytes: m(5) }, /超过单次时限 120 秒/],
|
||||||
|
[{ ending: "cancelled", cpuSeconds: un, wallSeconds: un, peakMemoryBytes: un }, /容器从外面被结束/],
|
||||||
|
[{ ...EXITED_OK, exitCode: 1 }, /退出码 1/],
|
||||||
|
];
|
||||||
|
for (const [formal, why] of cases) {
|
||||||
|
// 有一份合法的产物也不跑校验器：运行没正常结束就不算数。
|
||||||
|
const result = await judgeRun(t, "unused", runDirWith(t, loose()), formal, []);
|
||||||
|
assert.equal(result.verdict, "invalid");
|
||||||
|
assert.match(result.verdictReason, why);
|
||||||
|
assert.match(reason(result.score), why);
|
||||||
|
assert.match(reason(result.metrics["raw-score"]), /没有跑校验器/);
|
||||||
|
assert.deepEqual(result.extra, { legalization: null });
|
||||||
|
if (formal.ending !== "notStarted") assert.deepEqual(result.metrics["cpu-seconds"], formal.cpuSeconds);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("判定表 2：没有 solution.json，不算数：没有产物", async (t) => {
|
||||||
|
const result = await judgeRun(t, "unused", runDirWith(t, null), EXITED_OK, []);
|
||||||
|
assert.equal(result.verdict, "invalid");
|
||||||
|
assert.match(result.verdictReason, /^没有产物/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("判定表 3：校验器判不合法（形状不对、非有限数、圆心出界修不了），不算数，带校验器的原因", async (t) => {
|
||||||
|
const short = loose();
|
||||||
|
short.centers.pop();
|
||||||
|
short.radii.pop();
|
||||||
|
const shape = await judgeRun(t, "shape", runDirWith(t, short), EXITED_OK, []);
|
||||||
|
assert.equal(shape.verdict, "invalid");
|
||||||
|
assert.match(shape.verdictReason, /shape incorrect/);
|
||||||
|
assert.match(reason(shape.metrics["raw-score"]), /形状不对/);
|
||||||
|
assert.match(reason(shape.metrics["min-pairwise-clearance"]), /形状不对/);
|
||||||
|
|
||||||
|
const nan = JSON.stringify(loose()).replace('"radii":[0.0078125', '"radii":[NaN');
|
||||||
|
const nonFinite = await judgeRun(t, "nan", runDirWith(t, nan), EXITED_OK, []);
|
||||||
|
assert.equal(nonFinite.verdict, "invalid");
|
||||||
|
assert.match(nonFinite.verdictReason, /non-finite/);
|
||||||
|
|
||||||
|
// 0 号圆心挪到 x = −1/64：圆心在正方形外，只缩半径修不了。
|
||||||
|
const outside = loose();
|
||||||
|
outside.centers[0] = [-1 / 64, 0.5];
|
||||||
|
const result = await judgeRun(t, "outside", runDirWith(t, outside), EXITED_OK, []);
|
||||||
|
assert.equal(result.verdict, "invalid");
|
||||||
|
assert.match(result.verdictReason, /圆心在正方形外面/);
|
||||||
|
assert.match(reason(result.score), /圆心在正方形外面/);
|
||||||
|
assert.match(reason(result.metrics["legalization-deduction"]), /修不了/);
|
||||||
|
// 原分数照样算得出：26 个圆半径都没动，0.203125。
|
||||||
|
assert.equal(value(result.metrics["raw-score"]), 0.203125);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("判定表 4：校验器判合法，算数，分数取校验器算的（宽松 0.203125、相切 0.40625，扣 0）", async (t) => {
|
||||||
|
const result = await judgeRun(t, "loose", runDirWith(t, loose()), EXITED_OK, []);
|
||||||
|
assert.equal(result.verdict, "valid");
|
||||||
|
assert.equal(value(result.score), 0.203125);
|
||||||
|
assert.equal(value(result.metrics["min-pairwise-clearance"]), 0.015625);
|
||||||
|
assert.equal(value(result.metrics["raw-score"]), 0.203125);
|
||||||
|
assert.equal(value(result.metrics["raw-violation"]), 0);
|
||||||
|
assert.equal(value(result.metrics["legalization-deduction"]), 0);
|
||||||
|
// 资源用量原样来自正式运行。
|
||||||
|
assert.equal(value(result.metrics["wall-seconds"]), 2.5);
|
||||||
|
assert.equal(value(result.metrics["peak-memory-bytes"]), 1024);
|
||||||
|
|
||||||
|
const touching = await judgeRun(t, "tangent", runDirWith(t, tangent()), EXITED_OK, []);
|
||||||
|
assert.equal(touching.verdict, "valid");
|
||||||
|
assert.equal(value(touching.score), 0.40625);
|
||||||
|
assert.equal(value(touching.metrics["min-pairwise-clearance"]), 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("判定表 4（合法化）：差一点不合法的修好后算数，修后分数不高于原分数，扣分等于原分数减最终分数", async (t) => {
|
||||||
|
// 0、1 号半径放大到 3/128：重叠 3/64 − 1/32 = 1/64；1、2 号间隙恰好 0。原分数 30/128 = 0.234375。
|
||||||
|
// 只缩参与违反的圆：0、1 号各减 1/64 回到 1/128，修后 0.203125，扣 1/32 = 0.03125。
|
||||||
|
const overlap = loose();
|
||||||
|
overlap.radii[0] = 3 / 128;
|
||||||
|
overlap.radii[1] = 3 / 128;
|
||||||
|
const shrunk = await judgeRun(t, "overlap", runDirWith(t, overlap), EXITED_OK, []);
|
||||||
|
assert.equal(shrunk.verdict, "valid");
|
||||||
|
assert.equal(value(shrunk.score), 0.203125);
|
||||||
|
assert.equal(value(shrunk.metrics["raw-score"]), 0.234375);
|
||||||
|
assert.equal(value(shrunk.metrics["raw-violation"]), 0.015625);
|
||||||
|
assert.equal(value(shrunk.metrics["legalization-deduction"]), 0.03125);
|
||||||
|
const extra = shrunk.extra as { legalization: { shrunkCircles: number[]; rawValid: boolean; deduction: Measured } };
|
||||||
|
assert.deepEqual(extra.legalization.shrunkCircles, [0, 1]);
|
||||||
|
assert.equal(extra.legalization.rawValid, false);
|
||||||
|
assert.equal(value(extra.legalization.deduction), 0.03125);
|
||||||
|
|
||||||
|
// 0 号圆心挪到 x = 1/256，出左边 1/128 − 1/256 = 1/256；修后半径 1/256，和 51/256 = 0.19921875。
|
||||||
|
const edge = loose();
|
||||||
|
edge.centers[0] = [1 / 256, 0.5];
|
||||||
|
const edged = await judgeRun(t, "edge", runDirWith(t, edge), EXITED_OK, []);
|
||||||
|
assert.equal(value(edged.score), 0.19921875);
|
||||||
|
assert.equal(value(edged.metrics["raw-violation"]), 1 / 256);
|
||||||
|
assert.equal(value(edged.metrics["legalization-deduction"]), 0.203125 - 0.19921875);
|
||||||
|
|
||||||
|
// 参考项目真实运行 14-real-1 的产物：重叠约 2.78e-17，修后 1.7602769492805097（参考项目设计-15 7.1 节记的数）。
|
||||||
|
const real = await judgeRun(t, "real", runDirWith(t, readFileSync(FIXTURE, "utf8")), EXITED_OK, []);
|
||||||
|
assert.equal(real.verdict, "valid");
|
||||||
|
assert.equal(value(real.score), 1.7602769492805097);
|
||||||
|
const raw = value(real.metrics["raw-score"]);
|
||||||
|
assert.equal(raw, 1.76027694928051);
|
||||||
|
assert.ok(value(real.score) <= raw);
|
||||||
|
assert.equal(value(real.metrics["legalization-deduction"]), raw - value(real.score));
|
||||||
|
assert.ok(Math.abs(value(real.metrics["raw-violation"]) - 2.78e-17) < 0.01e-17);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("判定表 5：缺了整次作废的假设专用指标测不到，不算数；缺了只记缺失的照常算数", async (t) => {
|
||||||
|
const spec = (id: string, missing: HypothesisMetric["missing"]): HypothesisMetric => ({
|
||||||
|
id,
|
||||||
|
meaning: "测试用",
|
||||||
|
unit: "无",
|
||||||
|
direction: "maximize",
|
||||||
|
howMeasured: "测试用",
|
||||||
|
missing,
|
||||||
|
});
|
||||||
|
const runDir = runDirWith(t, loose());
|
||||||
|
writeFileSync(
|
||||||
|
join(runDir, "metrics.json"),
|
||||||
|
JSON.stringify({ spread: 0.5, gap: null, "answer-path": "p1", "safeguard-triggers": { "shrink-repair": 2 } }),
|
||||||
|
);
|
||||||
|
const kept = await judgeRun(t, "keep", runDir, EXITED_OK, [spec("spread", "invalidateRun"), spec("gap", "recordMissing")]);
|
||||||
|
assert.equal(kept.verdict, "valid");
|
||||||
|
assert.equal(value(kept.metrics.spread), 0.5);
|
||||||
|
assert.match(reason(kept.metrics.gap), /null/);
|
||||||
|
assert.deepEqual(kept.answerPath, { status: "read", value: "p1" });
|
||||||
|
assert.deepEqual(kept.safeguardTriggers, { status: "read", counts: { "shrink-repair": 2 } });
|
||||||
|
|
||||||
|
const voided = await judgeRun(t, "void", runDir, EXITED_OK, [spec("gap", "invalidateRun")]);
|
||||||
|
assert.equal(voided.verdict, "invalid");
|
||||||
|
assert.match(voided.verdictReason, /缺了整次作废.*gap/);
|
||||||
|
// 成绩照样算出来记下，只是这次运行不算数。
|
||||||
|
assert.equal(value(voided.score), 0.203125);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("答案路径、保障动手次数：读不到时记读不到及原因，不作废运行", async (t) => {
|
||||||
|
const missing = await judgeRun(t, "nometrics", runDirWith(t, loose()), EXITED_OK, []);
|
||||||
|
assert.equal(missing.verdict, "valid");
|
||||||
|
assert.deepEqual(missing.answerPath, { status: "unreadable", reason: "metrics.json 不存在" });
|
||||||
|
|
||||||
|
const runDir = runDirWith(t, loose());
|
||||||
|
writeFileSync(join(runDir, "metrics.json"), JSON.stringify({ "answer-path": 3, "safeguard-triggers": { a: -1 } }));
|
||||||
|
const wrong = await judgeRun(t, "wrongkeys", runDir, EXITED_OK, []);
|
||||||
|
assert.equal(wrong.verdict, "valid");
|
||||||
|
assert.equal(wrong.answerPath.status, "unreadable");
|
||||||
|
assert.equal(wrong.safeguardTriggers.status, "unreadable");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("校验器自己没正常跑完、判定认不出、修后分数高于原分数：报错停下，不当成候选的失败", (t) => {
|
||||||
|
const runDir = runDirWith(t, loose());
|
||||||
|
const judgeDir = tempDir(t, "judge");
|
||||||
|
const un: Measured = { status: "unmeasured", reason: "x" };
|
||||||
|
const timeout: ProcessResult = { ending: "timeout", cpuSeconds: un, wallSeconds: un, peakMemoryBytes: un };
|
||||||
|
assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, timeout, []), /校验器没有正常跑完（timeout）/);
|
||||||
|
// 退出码 2 是用法或读取出错，这时校验器不写判定文件。
|
||||||
|
assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, { ...EXITED_OK, exitCode: 2 }, []), /校验器没有正常跑完（退出码 2）/);
|
||||||
|
assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, EXITED_OK, []), /读不出 .*verdict\.json/);
|
||||||
|
|
||||||
|
const verdict = {
|
||||||
|
valid: true,
|
||||||
|
computed_sum: 0.3,
|
||||||
|
reasons: [],
|
||||||
|
diagnostics: { min_pairwise_clearance: 0 },
|
||||||
|
legalization: { raw_valid: false, raw_score: 0.2, raw_violation: 0.1, applied: true, repaired: true, shrunk_circles: [0], reasons: [] },
|
||||||
|
};
|
||||||
|
writeFileSync(join(judgeDir, "verdict.json"), JSON.stringify(verdict));
|
||||||
|
assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, EXITED_OK, []), /合法化后分数 0\.3 高于原分数 0\.2/);
|
||||||
|
// 退出码说不合法、判定说合法：对不上。
|
||||||
|
assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, { ...EXITED_OK, exitCode: 1 }, []), /对不上/);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 轻量检查
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function check(t: TestContext, suffix: string, files: Record<string, string>) {
|
||||||
|
const candidate = tempDir(t, "candidate");
|
||||||
|
for (const [name, text] of Object.entries(files)) writeFileSync(join(candidate, name), text);
|
||||||
|
const checkDir = tempDir(t, "check");
|
||||||
|
const definition = lightCheck(task, candidate, checkDir, digest, 5);
|
||||||
|
return readLightCheck(task, checkDir, await run(t, suffix, definition, checkDir));
|
||||||
|
}
|
||||||
|
|
||||||
|
test("轻量检查：语法错的、没有入口函数的不通过，正常的通过", async (t) => {
|
||||||
|
const syntax = await check(t, "check-syntax", { "initial.py": "def run_packing(:\n pass\n" });
|
||||||
|
assert.equal(syntax.passed, false);
|
||||||
|
assert.match(syntax.passed ? "" : syntax.reasons.join(), /initial\.py 解析失败/);
|
||||||
|
|
||||||
|
const noEntry = await check(t, "check-noentry", { "initial.py": "def solve():\n return [], [], 0.0\n" });
|
||||||
|
assert.equal(noEntry.passed, false);
|
||||||
|
assert.match(noEntry.passed ? "" : noEntry.reasons.join(), /run_packing\(\) 应当恰好有一个定义，实际有 0 个/);
|
||||||
|
|
||||||
|
const baseline = tempDir(t, "baseline");
|
||||||
|
cpSync(task.baselineDir, baseline, { recursive: true });
|
||||||
|
const checkDir = tempDir(t, "check");
|
||||||
|
const ok = readLightCheck(task, checkDir, await run(t, "check-ok", lightCheck(task, baseline, checkDir, digest, 5), checkDir));
|
||||||
|
assert.deepEqual(ok, { passed: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 从头到尾
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 基线 initial.py 准备运行目录 → 正式运行 → 跑校验器 → 判分。 */
|
||||||
|
async function endToEnd(t: TestContext, suffix: string): Promise<JudgedResult> {
|
||||||
|
const runDir = tempDir(t, "e2e");
|
||||||
|
prepareRun(task, task.baselineDir, runDir, []);
|
||||||
|
assert.equal(readFileSync(join(runDir, "metrics-spec.json"), "utf8"), "[]\n");
|
||||||
|
assert.ok(existsSync(join(runDir, "harness.py")) && existsSync(join(runDir, "initial.py")));
|
||||||
|
const formal = await run(t, `e2e-${suffix}`, formalRun(task, digest, "inforsi-test-seed"), runDir);
|
||||||
|
assert.equal(formal.ending, "exited", JSON.stringify(formal));
|
||||||
|
return judgeRun(t, `e2e-${suffix}-judge`, runDir, formal, []);
|
||||||
|
}
|
||||||
|
|
||||||
|
test("从头到尾：基线跑两次都判为算数，分数逐位相同", async (t) => {
|
||||||
|
const first = await endToEnd(t, "1");
|
||||||
|
const second = await endToEnd(t, "2");
|
||||||
|
for (const result of [first, second]) {
|
||||||
|
assert.equal(result.verdict, "valid", result.verdictReason);
|
||||||
|
assert.equal(result.score.status, "measured");
|
||||||
|
}
|
||||||
|
assert.ok(Object.is(value(first.score), value(second.score)), `${value(first.score)} 与 ${value(second.score)}`);
|
||||||
|
for (const [label, result] of [["第一次", first], ["第二次", second]] as const) {
|
||||||
|
t.diagnostic(
|
||||||
|
`${label}：verdict=${result.verdict}（${result.verdictReason}） score=${value(result.score)} ` +
|
||||||
|
`cpu=${JSON.stringify(result.metrics["cpu-seconds"])} wall=${JSON.stringify(result.metrics["wall-seconds"])} ` +
|
||||||
|
`mem=${JSON.stringify(result.metrics["peak-memory-bytes"])} deduction=${JSON.stringify(result.metrics["legalization-deduction"])}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("候选目录里遗留的旧 metrics.json 不会被当成这次的产物：整次作废的指标这次没写，判为不算数", async (t) => {
|
||||||
|
const candidate = tempDir(t, "candidate");
|
||||||
|
cpSync(task.baselineDir, candidate, { recursive: true });
|
||||||
|
// 基线代码不写 metrics.json；这份是以前某次运行留下的。
|
||||||
|
writeFileSync(join(candidate, "metrics.json"), JSON.stringify({ spread: 0.5, "answer-path": "main", "safeguard-triggers": {} }));
|
||||||
|
const spread: HypothesisMetric = {
|
||||||
|
id: "spread",
|
||||||
|
meaning: "测试用",
|
||||||
|
unit: "无",
|
||||||
|
direction: "maximize",
|
||||||
|
howMeasured: "测试用",
|
||||||
|
missing: "invalidateRun",
|
||||||
|
};
|
||||||
|
const runDir = tempDir(t, "stale");
|
||||||
|
prepareRun(task, candidate, runDir, [spread]);
|
||||||
|
const formal = await run(t, "stale", formalRun(task, digest, "inforsi-test-seed"), runDir);
|
||||||
|
assert.equal(formal.ending, "exited", JSON.stringify(formal));
|
||||||
|
const result = await judgeRun(t, "stale-judge", runDir, formal, [spread]);
|
||||||
|
t.diagnostic(`verdict=${result.verdict}（${result.verdictReason}） spread=${JSON.stringify(result.metrics.spread)}`);
|
||||||
|
assert.equal(result.metrics.spread?.status, "unmeasured");
|
||||||
|
assert.equal(result.verdict, "invalid");
|
||||||
|
assert.match(result.verdictReason, /缺了整次作废.*spread/);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user