diff --git a/WORK_LOG.md b/WORK_LOG.md index 73a956c..5fbeddd 100644 --- a/WORK_LOG.md +++ b/WORK_LOG.md @@ -15,6 +15,7 @@ - 任务书 02（rules）：src/rules 13 个文件、1761 行，`npm test` 192 条通过。预期值来源有三个：方法.md 的手算例；`tests/fixtures/rules/` 下不看实现写的 Python 脚本（重跑后 JSON 逐字节相同）；参考项目 calc15、calc14b 里与现行方法一致的部分。审阅做了两轮，每轮都换一个全新上下文的子 Agent。结构.md 第八部分补了 rules 对外内容。 - 任务书 03（ledger）：src/ledger 5 个文件、584 行，`npm test` 226 条通过（ledger 34 条）。一张只追加的表，两条禁改禁删触发器，8 处唯一约束（阶段开、阶段关、运行登记、确认启动或确认没启动、运行结果、模型调用开始、结束、消融结算），写入时只查四种拒收，提案的作者自评拆成封存另存。按第一版写过可见性和通用唯一键，按新版全部清掉了。全新上下文的审阅做了三轮。结构.md 第八部分补了 ledger 对外内容。 - 任务书 04（config）：src/config 4 个文件、453 行，`npm test` 246 条通过（config 19 条）。流程用联合类型（完整流程 / 朴素循环），规则名取自 rules 登记表，config 改为依赖 rules。ledger 顺手两处：结算按轮次唯一、笔记去掉版本号，表结构版本号升到 2，全新上下文审阅一轮无问题。结构.md 第八部分补了 config 对外内容。 +- 任务书 05（tasks 与 runner）：src/runner 2 个文件、271 行，src/tasks 6 个文件、691 行，`npm test` 274 条通过（runner 12 条、tasks 16 条，真实 Docker，整套约 12 秒）。任务目录搬到 `tasks/circle_packing/`，新增 task.json；`VERDICTS` 删掉"判不了"。基线从头到尾判为算数，分数 0.9597642169962064，两次逐位相同。全新上下文审阅两轮：第一轮发现候选能伪造计量结果（把超时记成正常退出）、运行目录里旧 metrics.json 会被当成本次产物，都已修（计量 shell 以容器内 root 跑、命令用 setpriv 降权、结果用 docker cp 取；harness 开跑前删旧输出）；第二轮只审这次修复，另采纳一条：容器加 no-new-privileges，禁止候选靠 setuid 程序重新拿到 root。结构.md 第八部分补了 runner、tasks。 ## 关键决定（用户原话见各提交说明） @@ -66,6 +67,10 @@ - 给 agents、runner："整个环境的用量"（实验员会话的 CPU、内存等）写这两个模块时再定记录种类。 - 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。 - 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。 +- 给 submissions：研究者登记的假设专用指标编号不能和常规辅助指标重名（`metricDefinitions` 里的 7 个），否则候选自报的值会顶替宿主测的值（任务书 05 审阅意见）。 +- 给 flow：容器名要在本机所有研究之间唯一（例如带研究编号）；`docker run` 因重名失败时 runner 按"启动了"处理。日志目录每次运行一个新的。 +- 给以后换任务镜像：镜像里必须有 `setpriv`（runner 用它把命令降到当前用户）；容器里不要用 `python -I`，它会忽略 `PYTHON_CPU_COUNT`。 +- 未处理的可能情况（没有真实证据，只列出）：Docker 守护进程卡住不应答时 docker 命令没有超时，会一直等；候选写出大到算术溢出的有限数时，validate.py 会异常退出，被当成宿主工具坏了而停下。 ## 工作方式（用户 2026-10-01 定） @@ -78,4 +83,4 @@ ## 下一步 -任务书 05（tasks 与 runner，`任务书/05-tasks-runner.md`）已写好，等用户另开新 Agent 执行；完成后监督者核对。 +任务书 05 已完成，等监督者核对。 diff --git a/docs/结构.md b/docs/结构.md index 8bb2f8d..158775e 100644 --- a/docs/结构.md +++ b/docs/结构.md @@ -235,8 +235,8 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录 **运行** - `RunDefinition`（字段定义）：tasks 交给 runner 的运行定义，写明命令、工作目录、按 sha256 钉死的镜像和运行时、时限、核数和环境变量；候选进程不继承宿主的环境变量。 -- `ProcessResult`（字段定义）：runner 交出的进程层面结果，分没启动、正常退出、超时、被取消四种；启动过的带资源用量。 -- `VERDICTS`、`Verdict`：一次运行的判定，算数、不算数、判不了三选一。 +- `ProcessResult`（字段定义）：runner 交出的进程层面结果，分没启动、正常退出、超时、从外面被杀四种；启动过的带资源用量，从外面被杀的三项资源都记测不到。 +- `VERDICTS`、`Verdict`：一次运行的判定，算数、不算数二选一。 - `ANSWER_PATH_KEY`、`SAFEGUARD_TRIGGERS_KEY`：候选在指标文件里多写的两个保留键名，`answer-path` 和 `safeguard-triggers`。 - `AnswerPath`（字段定义）：读到的答案路径原样字符串，或读不到及原因；在不在位置表里由拿得到位置表的模块现算。 - `SafeguardTriggers`（字段定义）：读到的"保障短名 → 动手次数"原样内容，或读不到及原因。 @@ -437,3 +437,46 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录 - `RuleRef`（类型）：决策点用的规则，名字和版本。 - `STUDY_STEPS`、`StudyStepId`：整份研究的步骤，按先后排。 - `ROUND_STEPS`、`RoundStepId`：每一轮的步骤，按先后排；比第三部分拆得更细（例如"回应与统一"和"协议冻结"分成两步，挑刺与回应关掉时协议照样冻结）。 + +### runner + +下面是 `src/runner/index.ts` 导出的全部内容。runner 用 docker 命令行做事，不碰账本，自己也不存状态：依据都在 Docker 里，所以新进程里新建的 `Runner` 能接着等早先启动的容器。Docker 不应答时，查状态答"问不到"，其余函数直接报错，都不重试。 + +- `new Runner(dockerHost)`：Docker 守护进程的地址必须给出；写 null 表示不加 `-H`，由 docker 命令按自己的环境变量和 context 决定。 +- `runner.buildImage(contextDir, tag)`：用目录里的 Dockerfile 建镜像，交回镜像编号（`docker image inspect -f '{{.Id}}'` 的值）。 +- `runner.hasImage(digest)`：本机有没有这个镜像编号。 +- `runner.start(name, definition, runDir)`：起一个容器，运行目录挂到运行定义的工作目录；容器里以 root 运行的计量 shell 把命令降到宿主当前用户再跑，结局、退出码、墙钟、CPU 秒、峰值内存写在容器内只有 root 可写的目录里，命令改不了。`docker run` 失败时按名字再查一次，容器不存在或建了没启动算没启动（建了的删掉），在跑或已结束算启动了。 +- `runner.wait(name, logDir)`：等容器结束，交回 `ProcessResult`，把标准输出、标准错误和从容器里拷出的结果文件存成日志目录里的 `stdout.log`、`stderr.log`、`usage`。有结果文件按文件定正常退出或超时；没有就是从外面被杀。 +- `runner.status(name)`：没有这个容器、建了没启动、在跑、已结束、问不到，五种之一。 +- `runner.remove(name)`：删掉一个已结束或没启动的容器，flow 写完结果后调用。 +- `ContainerStatus`、`StartOutcome`（类型）：查状态的五种回答；启动了，或没启动和 Docker 的报错原文。 + +容器名由调用方给，必须是确定的（恢复时按名字去问）；runner 只检查它符合 Docker 的命名要求。没有取消函数：方法里没有程序主动取消一次已启动运行的场合。 + +### tasks + +下面是 `src/tasks/index.ts` 导出的全部内容。tasks 不起子进程：凡是要执行任务代码的事（正式运行、判分时跑校验器、轻量检查）都只交出 `RunDefinition`，由调用方交给 runner 在任务镜像里跑，再把产物交回这里解读。任务种类到实现是 `task.ts` 里的一张对照表，现在只有填圆。各目录（运行目录、判分目录、检查目录）都由调用方建好、传空目录进来。 + +**任务事实** + +- `TASK_KINDS`、`TaskKind`：任务种类名单，现在只有 `circle_packing`。 +- `TaskFacts`（字段定义）：task.json 的字段：任务种类、最终指标的编号和方向、ε、已知最好（没有写 null）、单次时限、核数、Docker 运行时、不许改的文件名单。全部必填，不认识的字段报错。 +- `checkTaskFacts(value)`：严格检查一份已解析的 task.json，一次列出全部错误，每条写成"字段路径：原因"。 +- `Task`（类型）：读进来的一个任务：task.json 的内容、任务目录（Dockerfile 所在，给 runner 建镜像）、基线目录、TASK.md 全文。 +- `loadTask(taskDir)`：读任务目录，检查 task.json，交回 `Task`。 +- `metricDefinitions(task)`：最终指标和常规辅助指标的定义（墙钟、CPU 秒、峰值内存、最小间隙、原分数、原违反量、扣掉的分）；辅助指标的定义写在填圆的代码里。 +- `protectedChanges(task, changedFiles)`：改动过的文件里属于"不许改"名单的那些。 + +**运行** + +- `prepareRun(task, candidateDir, runDir, hypothesisMetrics)`：把候选代码、任务的 harness 和这次要测的假设专用指标说明（metrics-spec.json）放进运行目录。 +- `formalRun(task, imageDigest, seed)`：正式运行的运行定义；种子由调用方给，同一份研究的全部运行用同一个，经环境变量交给 harness。 +- `lightCheck(task, candidateDir, checkDir, imageDigest, seconds)`：把候选代码和检查脚本放进检查目录，交回轻量检查的运行定义；只看语法和入口函数，不执行候选代码，次数上限不归这里管。 +- `readLightCheck(task, checkDir, result)`：解读轻量检查：通过，或不通过及原因；检查脚本自己没正常跑完时报错停下。 +- `LightCheckOutcome`（类型）：轻量检查的结论。 + +**判分** + +- `prepareJudging(task, runDir, judgeDir, imageDigest, formal, hypothesisMetrics)`：判分第一步。正式运行超时、从外面被杀、退出码非 0，或者没有产物，直接交回不算数的判分结果；否则把产物和宿主保管的校验器复制进判分目录，交回校验器的运行定义。 +- `judge(task, runDir, judgeDir, formal, validator, hypothesisMetrics)`：判分第二步。读校验器写的判定和候选写的指标文件，加上正式运行的结果，交回 `JudgedResult`：校验器判合法（含合法化后合法）算数；判不合法、或"缺了整次作废"的假设专用指标测不到，不算数。合法化的三个数既进 `extra`，也是常规辅助指标的取值；测不到的指标记测不到并附原因。校验器自己没正常跑完、判定文件认不出、修后分数高于原分数时报错停下，不当成候选的失败。 +- `JudgingStep`（类型）：判分第一步的结果，已判定或要跑校验器。 diff --git a/src/runner/index.ts b/src/runner/index.ts new file mode 100644 index 0000000..8ac16ed --- /dev/null +++ b/src/runner/index.ts @@ -0,0 +1,2 @@ +// runner：用 Docker 把一条命令跑完，交回它怎么结束的、用了多少资源（docs/结构.md 第二部分）。 +export { Runner, type ContainerStatus, type StartOutcome } from "./runner.ts"; diff --git a/src/runner/runner.ts b/src/runner/runner.ts new file mode 100644 index 0000000..6c4e69d --- /dev/null +++ b/src/runner/runner.ts @@ -0,0 +1,269 @@ +import { spawn } from "node:child_process"; +import { closeSync, openSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import type { Measured, ProcessResult, RunDefinition } from "../vocab/index.ts"; + +// 用 Docker 把一条命令跑完：建镜像、起容器、等结束、查状态、删容器（docs/结构.md 第二部分）。 +// runner 不碰账本：登记要跑、确认启动、写结果都由 flow 做（方法.md 第九部分）。 +// 本身不存状态，全部依据都在 Docker 里，所以另一个进程里新建的 Runner 也能接着等一个早先启动的容器。 + +/** 查状态的五种回答。unreachable 是 Docker 不应答，flow 据此记"状态未知"并停下（方法.md 第七、第九部分）。 */ +export type ContainerStatus = "absent" | "created" | "running" | "exited" | "unreachable"; + +/** 启动的结果：启动了，或者没启动和 Docker 的报错原文。 */ +export type StartOutcome = { started: true } | { started: false; reason: string }; + +/** Docker 对容器名的要求（与 Docker 源码里的 RestrictedNamePattern 相同）。 */ +const CONTAINER_NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]+$/; + +/** 容器里的固定主机名：默认主机名是容器编号前 12 位，每次不同，镜像编号钉不住它。 */ +const HOSTNAME = "inforsi"; + +/** + * 计量结果在容器自己文件层里的位置，不挂到宿主。目录由容器里的 root 建、只有 root 可写， + * 降权后的命令既写不进去也改不了权限，伪造不了结果。 + */ +const RECORD_DIR = "/inforsi-record"; +const USAGE_FILE = "usage"; + +/** + * 容器里的第一个进程，以容器内 root 身份运行。参数依次是时限（纳秒）、时限（秒，交给 timeout）、 + * 降权到的用户编号和组编号、真正的命令。 + * - timeout 和这段 shell 是 root，被 timeout 包着的 setpriv 降到宿主当前用户再执行命令： + * 命令写进运行目录的文件在宿主上归当前用户，宿主能读产物和清理；命令也没法向 root 进程发信号。 + * setpriv 不改环境变量，命令的环境仍只有镜像的和 docker run 给的。 + * - 用 KILL 结束超时的命令：它不能被捕获，候选不能拖延；timeout 此时的退出码是 128+9=137。 + * - 超时由这段 shell 自己确认：退出码 137 且用时不短于时限。命令自己以 124 或 137 结束时， + * 用时不到时限，不会被当成超时。 + * - 资源用量读容器自己的 cgroup v2（cgroup 命名空间下 /sys/fs/cgroup 就是本容器）， + * 命令被 timeout 杀掉之后照样读得到；从外面杀掉整个容器时这段 shell 也一起没了，结果文件不会出现。 + * - 先写临时文件再改名：结果文件要么完整，要么不存在。 + */ +const MEASURE_SCRIPT = `limit_ns=$1; limit=$2; uid=$3; gid=$4; shift 4 +mkdir -m 700 ${RECORD_DIR} +start=$(date +%s%N) +timeout -s KILL "$limit" setpriv --reuid="$uid" --regid="$gid" --clear-groups -- "$@" +code=$? +end=$(date +%s%N) +wall=$((end - start)) +ending=exited +if [ "$code" -eq 137 ] && [ "$wall" -ge "$limit_ns" ]; then ending=timeout; fi +cpu=$(sed -n 's/^usage_usec //p' /sys/fs/cgroup/cpu.stat) +mem=$(cat /sys/fs/cgroup/memory.peak) +echo "$ending $code $wall $cpu $mem" > ${RECORD_DIR}/${USAGE_FILE}.tmp +mv ${RECORD_DIR}/${USAGE_FILE}.tmp ${RECORD_DIR}/${USAGE_FILE} +`; + +/** 结果文件的格式：结局、退出码、墙钟纳秒、CPU 微秒、峰值内存字节，空格分隔，一行。 */ +const USAGE_LINE = /^(exited|timeout) (\d+) (\d+) (\d+) (\d+)\n$/; + +/** docker cp 找不到容器里的文件时的报错。 */ +const NO_SUCH_FILE = "Could not find the file"; + +/** 认得出的"Docker 不应答"报错：本地套接字连不上、TCP 地址连不上两种写法。 */ +const UNREACHABLE = /failed to connect to the docker API|Cannot connect to the Docker daemon/; + +interface Done { + code: number; + stdout: string; + stderr: string; +} + +export class Runner { + readonly #hostArgs: string[]; + + /** + * dockerHost：Docker 守护进程的地址，例如 unix:///var/run/docker.sock； + * null 表示不加 -H，由 docker 命令自己按 DOCKER_HOST 环境变量和当前 context 决定。 + */ + constructor(dockerHost: string | null) { + this.#hostArgs = dockerHost === null ? [] : ["-H", dockerHost]; + } + + /** 用目录里的 Dockerfile 建镜像，打上 tag，交回镜像编号（`docker image inspect -f '{{.Id}}'` 的值）。 */ + async buildImage(contextDir: string, tag: string): Promise { + const built = await this.#docker(["build", "-t", tag, contextDir]); + if (built.code !== 0) throw new Error(`docker build ${contextDir} 失败：\n${built.stderr}`); + const inspected = await this.#docker(["image", "inspect", "--format", "{{.Id}}", tag]); + if (inspected.code !== 0) throw new Error(`docker image inspect ${tag} 失败：${inspected.stderr}`); + const digest = inspected.stdout.trim(); + if (!/^sha256:[0-9a-f]{64}$/.test(digest)) { + throw new Error(`docker image inspect ${tag} 交回的不是镜像编号：${inspected.stdout}`); + } + return digest; + } + + /** 本机有没有这个镜像编号。Docker 不应答或报别的错，直接报错。 */ + async hasImage(digest: string): Promise { + const inspected = await this.#docker(["image", "inspect", "--format", "{{.Id}}", digest]); + if (inspected.code === 0) return true; + if (inspected.stderr.includes("No such image")) return false; + throw new Error(`docker image inspect ${digest} 失败：${inspected.stderr}`); + } + + /** + * 起一个容器跑运行定义里的命令，runDir 挂到 definition.workdir。 + * docker run 失败时按名字再查一次（方法.md 第七部分"不确定有没有启动时，先核对"）： + * 容器不存在或建了没启动，算没启动（建了的顺手删掉）；在跑或已结束，算启动了。 + */ + async start(name: string, definition: RunDefinition, runDir: string): Promise { + checkName(name); + const getuid = process.getuid; + const getgid = process.getgid; + if (getuid === undefined || getgid === undefined) throw new Error("runner 只能在 POSIX 系统上用：读不到当前用户编号"); + const wallNs = Math.ceil(definition.wallSeconds * 1e9); + const args = [ + "run", + "--detach", + "--name", + name, + // 本机 Docker 的默认运行时不一定是 runc（这台机器是 nvidia），跟着默认走，换台机器环境就悄悄变了。 + "--runtime", + definition.image.runtime, + // 镜像只按编号用本机已有的，不去仓库拉：拉到的可能是另一份内容。 + "--pull", + "never", + // 第一个进程是 root，好让计量结果不归命令管；命令本身由计量 shell 降到宿主当前用户再跑。 + "--user", + "0:0", + // 镜像里有 su、mount 这类 setuid 程序；禁止靠它们重新拿到 root，降权后的命令就碰不到计量结果。 + // setpriv 降权是 root 主动换身份，不受这条限制。 + "--security-opt", + "no-new-privileges", + "--hostname", + HOSTNAME, + // 方法要求运行时不联网。 + "--network", + "none", + // --cpus 限总量，不固定到某个核。PYTHON_CPU_COUNT 让 os.cpu_count() 报这个核数（Python 3.13 起）， + // 否则它报宿主核数，候选会开一堆进程抢一个核。写在 env 之后：同名时 docker 取后一个。 + `--cpus=${definition.cores}`, + // 降权后的用户在镜像里没有家目录；HOME 不写明的话会是 root 的 /root。写在 env 之前，运行定义可以改。 + "--env", + "HOME=/", + ...Object.entries(definition.env).flatMap(([key, value]) => ["--env", `${key}=${value}`]), + "--env", + `PYTHON_CPU_COUNT=${definition.cores}`, + // 用 --mount 而不是 -v：-v 遇到相对路径会悄悄建一个同名数据卷，--mount 直接报错。 + "--mount", + `type=bind,source=${runDir},target=${definition.workdir}`, + "--workdir", + definition.workdir, + // 第一个进程固定是计量 shell，不受镜像 ENTRYPOINT 影响。 + "--entrypoint", + "/bin/sh", + // 不加 --rm：中断恢复时要按名字找到这个容器。 + definition.image.digest, + "-c", + MEASURE_SCRIPT, + "inforsi-measure", + String(wallNs), + String(definition.wallSeconds), + String(getuid.call(process)), + String(getgid.call(process)), + ...definition.command, + ]; + const run = await this.#docker(args); + if (run.code === 0) return { started: true }; + const status = await this.status(name); + if (status === "running" || status === "exited") return { started: true }; + if (status === "unreachable") { + throw new Error(`docker run ${name} 失败，之后查状态时 Docker 不应答，不知道有没有启动。docker run 的报错：${run.stderr}`); + } + if (status === "created") await this.remove(name); + return { started: false, reason: run.stderr.trim() }; + } + + /** + * 等容器结束，交回进程层面的结果。标准输出、标准错误存成 logDir 下的 stdout.log、stderr.log， + * 计量结果文件从容器里拷出来存成 logDir 下的 usage（容器结束后 docker cp 照样能拷）。 + * 只靠名字，能接上别的进程启动的容器；容器早已结束时 docker wait 立刻返回。 + */ + async wait(name: string, logDir: string): Promise { + checkName(name); + const waited = await this.#docker(["wait", name]); + if (waited.code !== 0) throw new Error(`docker wait ${name} 失败：${waited.stderr}`); + await this.#saveLogs(name, logDir); + const usagePath = join(logDir, USAGE_FILE); + const copied = await this.#docker(["cp", `${name}:${RECORD_DIR}/${USAGE_FILE}`, usagePath]); + if (copied.code !== 0) { + if (!copied.stderr.includes(NO_SUCH_FILE)) throw new Error(`docker cp ${name} 的计量结果失败：${copied.stderr}`); + const reason = `容器从外面被结束（容器退出码 ${waited.stdout.trim()}），计量 shell 没有写出结果文件`; + const unmeasured: Measured = { status: "unmeasured", reason }; + return { ending: "cancelled", cpuSeconds: unmeasured, wallSeconds: unmeasured, peakMemoryBytes: unmeasured }; + } + const text = readFileSync(usagePath, "utf8"); + const match = USAGE_LINE.exec(text); + if (match === null) throw new Error(`${usagePath} 的格式认不出：${JSON.stringify(text)}`); + const [, ending, code, wallNs, cpuUsec, memBytes] = match as unknown as [string, string, string, string, string, string]; + const usage = { + cpuSeconds: measured(Number(cpuUsec) / 1e6), + wallSeconds: measured(Number(wallNs) / 1e9), + peakMemoryBytes: measured(Number(memBytes)), + }; + if (ending === "timeout") return { ending: "timeout", ...usage }; + return { ending: "exited", exitCode: Number(code), ...usage }; + } + + /** 按名字查容器的状态。Docker 报出这五种以外的状态（paused、restarting、dead 等）时报错。 */ + async status(name: string): Promise { + checkName(name); + const inspected = await this.#docker(["container", "inspect", "--format", "{{.State.Status}}", name]); + if (inspected.code !== 0) { + if (inspected.stderr.includes("No such container")) return "absent"; + if (UNREACHABLE.test(inspected.stderr)) return "unreachable"; + throw new Error(`docker container inspect ${name} 失败：${inspected.stderr}`); + } + const state = inspected.stdout.trim(); + if (state === "created" || state === "running" || state === "exited") return state; + throw new Error(`容器 ${name} 处于认不出的状态：${state}`); + } + + /** 删掉一个已结束或没启动的容器。还在跑的容器 docker 拒绝删，这里照样报错。 */ + async remove(name: string): Promise { + checkName(name); + const removed = await this.#docker(["rm", name]); + if (removed.code !== 0) throw new Error(`docker rm ${name} 失败：${removed.stderr}`); + } + + /** 两条流直接写进文件，不经过内存：候选可能打印很多内容。恢复时可能再等一次，所以覆盖已有的文件。 */ + async #saveLogs(name: string, logDir: string): Promise { + const out = openSync(join(logDir, "stdout.log"), "w"); + const err = openSync(join(logDir, "stderr.log"), "w"); + try { + const code = await new Promise((resolve, reject) => { + const child = spawn("docker", [...this.#hostArgs, "logs", name], { stdio: ["ignore", out, err] }); + child.on("error", reject); + child.on("close", resolve); + }); + if (code !== 0) throw new Error(`docker logs ${name} 失败，退出码 ${code}，详情见 ${logDir}/stderr.log`); + } finally { + closeSync(out); + closeSync(err); + } + } + + /** 调一次 docker 命令，收齐输出。docker 命令本身起不来或被信号杀掉时报错。 */ + #docker(args: string[]): Promise { + return new Promise((resolve, reject) => { + const child = spawn("docker", [...this.#hostArgs, ...args], { stdio: ["ignore", "pipe", "pipe"] }); + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf8").on("data", (chunk: string) => (stdout += chunk)); + child.stderr.setEncoding("utf8").on("data", (chunk: string) => (stderr += chunk)); + child.on("error", reject); + child.on("close", (code, signal) => { + if (code === null) reject(new Error(`docker ${args[0]} 被信号 ${signal} 结束`)); + else resolve({ code, stdout, stderr }); + }); + }); + } +} + +function checkName(name: string): void { + if (!CONTAINER_NAME.test(name)) throw new Error(`容器名 ${JSON.stringify(name)} 不符合 Docker 的要求 ${CONTAINER_NAME}`); +} + +function measured(value: number): Measured { + return { status: "measured", value }; +} diff --git a/src/tasks/circle-packing.ts b/src/tasks/circle-packing.ts new file mode 100644 index 0000000..5cc4b96 --- /dev/null +++ b/src/tasks/circle-packing.ts @@ -0,0 +1,342 @@ +import { copyFileSync, cpSync, existsSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { Type, type Static } from "typebox"; +import { Value } from "typebox/value"; +import type { + HypothesisMetric, + Json, + JudgedResult, + Measured, + MetricDefinition, + ProcessResult, + RunDefinition, +} from "../vocab/index.ts"; +import type { TaskFacts } from "./facts.ts"; +import { measured, readMetricsFile, unmeasured } from "./metrics-file.ts"; +import type { LightCheckOutcome, Task, TaskImplementation } from "./types.ts"; + +// 填圆：在单位正方形里放 26 个圆，最大化半径之和。任务说明见 tasks/circle_packing/TASK.md。 +// 凡是要执行任务代码的事（正式运行、校验器、轻量检查）都只交出运行定义，由调用方交给 runner 在任务镜像里跑； +// 这里只读写文件、解读产物（docs/结构.md 第二部分"runner 只管怎么跑，tasks 只管怎么判"）。 + +const CIRCLES = 26; +const ENTRY = "initial.py"; +const HARNESS = "harness.py"; +const VALIDATOR = "validate.py"; +const CHECK_SCRIPT = "check-candidate.py"; +const METRICS_SPEC = "metrics-spec.json"; +const METRICS = "metrics.json"; +const SOLUTION = "solution.json"; +const VERDICT = "verdict.json"; +const CHECK_RESULT = "check.json"; +/** 轻量检查目录里放候选代码的子目录；检查脚本放在它的上一层，免得扫到自己。 */ +const CHECK_CANDIDATE = "candidate"; +/** 运行目录挂进容器的位置。 */ +const WORKDIR = "/work"; +/** 校验器的时限。26 个圆两两比较加合法化，实测一秒以内；它超时说明宿主的工具出了问题。 */ +const VALIDATOR_SECONDS = 60; +/** 合法化前先按容差 0 判原产物（方法.md 第八部分"合法化"第 1 条）。 */ +const TOLERANCE = "0"; + +// 常规辅助指标的编号（方法.md 第五部分"指标"）。前三个来自 runner 的资源用量，后四个来自校验器。 +const WALL = "wall-seconds"; +const CPU = "cpu-seconds"; +const MEMORY = "peak-memory-bytes"; +const CLEARANCE = "min-pairwise-clearance"; +const RAW_SCORE = "raw-score"; +const RAW_VIOLATION = "raw-violation"; +const DEDUCTION = "legalization-deduction"; +const FROM_VALIDATOR = [CLEARANCE, RAW_SCORE, RAW_VIOLATION, DEDUCTION]; + +const SIDE = "单位正方形的边长"; +const PER_RUN = "每次运行结束后由宿主读一次，直接取值"; + +function auxiliary( + id: string, + meaning: string, + unit: string, + direction: "maximize" | "minimize", + howMeasured: string, +): MetricDefinition { + return { id, layer: "auxiliary", meaning, unit, direction, howMeasured, missing: "recordMissing" }; +} + +function metricDefinitions(facts: TaskFacts): MetricDefinition[] { + const validator = `宿主的校验器从 ${SOLUTION} 重算，${PER_RUN}`; + return [ + { + id: facts.finalMetric.id, + layer: "final", + meaning: + "合法摆法的半径之和，由宿主的独立校验器从实际产物重算；原产物差一点不合法时，先按 TASK.md 的只缩不移规则修，再按修后的半径算", + unit: SIDE, + direction: facts.finalMetric.direction, + howMeasured: validator, + missing: "invalidateRun", + }, + auxiliary(WALL, "正式运行的墙钟时间", "秒", "minimize", `容器里的计量程序记录，${PER_RUN}`), + auxiliary(CPU, "正式运行用掉的 CPU 秒（容器 cgroup 的 usage_usec）", "秒", "minimize", `容器里的计量程序记录，${PER_RUN}`), + auxiliary(MEMORY, "正式运行的峰值内存（容器 cgroup 的 memory.peak）", "字节", "minimize", `容器里的计量程序记录，${PER_RUN}`), + auxiliary(CLEARANCE, "所有圆对中「圆心距减两半径之和」的最小值，大于等于 0 表示不重叠；修过的取修后的摆法", SIDE, "maximize", validator), + auxiliary(RAW_SCORE, "合法化之前原产物的半径之和", SIDE, "maximize", validator), + auxiliary(RAW_VIOLATION, "原产物的最大违反量：两圆重叠的量和圆出边的量里最大的一个，0 表示没有违反", SIDE, "minimize", validator), + auxiliary(DEDUCTION, "只缩不移合法化扣掉的分：原分数减最终成绩，原产物合法时是 0", SIDE, "minimize", validator), + ]; +} + +/** 正式运行、校验器、轻量检查共用的运行定义。数学库的线程数和核数一致，否则会开多个线程抢核。 */ +function definition(task: Task, imageDigest: string, command: string[], wallSeconds: number, env: Record): RunDefinition { + const cores = String(task.facts.run.cores); + return { + command, + workdir: WORKDIR, + image: { digest: imageDigest, runtime: task.facts.run.runtime }, + wallSeconds, + cores: task.facts.run.cores, + inheritEnv: false, + env: { OMP_NUM_THREADS: cores, OPENBLAS_NUM_THREADS: cores, MKL_NUM_THREADS: cores, ...env }, + }; +} + +// 解释器不加 -I：-I 会忽略全部 PYTHON* 环境变量，连同 runner 设的 PYTHON_CPU_COUNT， +// 于是 os.cpu_count() 报回宿主核数（本机实测 16）。容器里的环境变量全部由运行定义给出，用不着 -I。 +// -B：不写 __pycache__，不往运行目录里添文件。 + +function prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void { + cpSync(candidateDir, runDir, { recursive: true }); + // harness 和指标说明以宿主的为准，候选目录里有同名文件也被盖掉（这两个名字在 task.json 的不许改名单里）。 + copyFileSync(join(task.dir, HARNESS), join(runDir, HARNESS)); + writeFileSync(join(runDir, METRICS_SPEC), `${JSON.stringify(hypothesisMetrics, null, 2)}\n`); +} + +function formalRun(task: Task, imageDigest: string, seed: string): RunDefinition { + const command = ["python", "-B", `${WORKDIR}/${HARNESS}`, "--program", ENTRY, "--output", SOLUTION]; + return definition(task, imageDigest, command, task.facts.run.wallSeconds, { INFORSI_SEED: seed }); +} + +function lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition { + cpSync(candidateDir, join(checkDir, CHECK_CANDIDATE), { recursive: true }); + copyFileSync(join(task.dir, CHECK_SCRIPT), join(checkDir, CHECK_SCRIPT)); + const command = ["python", "-B", CHECK_SCRIPT, CHECK_CANDIDATE, ENTRY, CHECK_RESULT]; + return definition(task, imageDigest, command, seconds, {}); +} + +/** 检查脚本写的结果文件。valid 与 reasons 是否为空必须一致。 */ +const CheckResult = Type.Object({ valid: Type.Boolean(), reasons: Type.Array(Type.String()) }); + +/** 宿主的工具（检查脚本、校验器）没正常跑完时报错停下：这不是候选的失败（任务书 05 第三部分）。 */ +function requireExit(what: string, result: ProcessResult, codes: number[]): number { + if (result.ending === "exited" && codes.includes(result.exitCode)) return result.exitCode; + const how = result.ending === "notStarted" ? `没启动：${result.reason}` : result.ending === "exited" ? `退出码 ${result.exitCode}` : result.ending; + throw new Error(`${what}没有正常跑完（${how}），这是宿主一侧的工具出了问题，不能当成候选的失败`); +} + +function readJson(path: string): unknown { + try { + return JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + throw new Error(`读不出 ${path}：${(error as Error).message}`, { cause: error }); + } +} + +function readLightCheck(checkDir: string, result: ProcessResult): LightCheckOutcome { + requireExit("轻量检查脚本", result, [0]); + const path = join(checkDir, CHECK_RESULT); + const check = readJson(path); + if (!Value.Check(CheckResult, check) || check.valid !== (check.reasons.length === 0)) { + throw new Error(`${path} 的内容认不出：${JSON.stringify(check)}`); + } + return check.valid ? { passed: true } : { passed: false, reasons: check.reasons }; +} + +/** 正式运行没正常结束的原因；正常退出（退出码 0）时为 null。 */ +function processFailure(task: Task, formal: Exclude): string | null { + if (formal.ending === "timeout") return `超过单次时限 ${task.facts.run.wallSeconds} 秒，被结束`; + if (formal.ending === "cancelled") return "容器从外面被结束（关机、手动 docker kill 等），没有留下计量结果"; + if (formal.exitCode !== 0) return `正式运行以退出码 ${formal.exitCode} 结束`; + return null; +} + +function started(formal: ProcessResult): Exclude { + if (formal.ending === "notStarted") throw new Error(`没启动的运行不判分：${formal.reason}`); + return formal; +} + +/** + * 拼出 JudgedResult：资源用量来自正式运行，假设专用指标和两个保留键来自候选写的 metrics.json。 + * 有"缺了整次作废"的假设专用指标测不到时，原本算数的也改判不算数（方法.md 第五部分"指标"）。 + */ +function assemble( + runDir: string, + formal: Exclude, + hypothesisMetrics: HypothesisMetric[], + verdict: { valid: boolean; reason: string }, + score: Measured, + fromValidator: Record, + extra: Json, +): JudgedResult { + const file = readMetricsFile(join(runDir, METRICS), METRICS, hypothesisMetrics); + const voided = hypothesisMetrics.flatMap((m) => { + const value = file.hypothesis[m.id]!; + return m.missing === "invalidateRun" && value.status === "unmeasured" ? [`${m.id}（${value.reason}）`] : []; + }); + const result = verdict.valid && voided.length > 0 + ? { valid: false, reason: `登记为"缺了整次作废"的假设专用指标测不到：${voided.join("；")}` } + : verdict; + return { + verdict: result.valid ? "valid" : "invalid", + verdictReason: result.reason, + score, + metrics: { + [WALL]: formal.wallSeconds, + [CPU]: formal.cpuSeconds, + [MEMORY]: formal.peakMemoryBytes, + ...fromValidator, + ...file.hypothesis, + }, + answerPath: file.answerPath, + safeguardTriggers: file.safeguardTriggers, + extra, + }; +} + +function prepareJudging( + task: Task, + runDir: string, + judgeDir: string, + imageDigest: string, + formal: ProcessResult, + hypothesisMetrics: HypothesisMetric[], +): ReturnType { + const run = started(formal); + const solution = join(runDir, SOLUTION); + const failure = processFailure(task, run) ?? (existsSync(solution) ? null : `没有产物：运行目录里没有 ${SOLUTION}`); + if (failure !== null) { + const notValidated = Object.fromEntries(FROM_VALIDATOR.map((id) => [id, unmeasured(`没有跑校验器：${failure}`)])); + const result = assemble(runDir, run, hypothesisMetrics, { valid: false, reason: failure }, unmeasured(failure), notValidated, { + legalization: null, + }); + return { status: "judged", result }; + } + copyFileSync(solution, join(judgeDir, SOLUTION)); + copyFileSync(join(task.dir, VALIDATOR), join(judgeDir, VALIDATOR)); + const command = [ + "python", + "-B", + VALIDATOR, + "--circles", + String(CIRCLES), + "--solution", + SOLUTION, + "--atol", + TOLERANCE, + "--legalize", + "--verdict", + VERDICT, + ]; + return { status: "validate", definition: definition(task, imageDigest, command, VALIDATOR_SECONDS, {}) }; +} + +const NumberOrNull = Type.Union([Type.Number(), Type.Null()]); + +/** validate.py --legalize 写的判定里，这里要读的部分；其余键（evidence、legal_radii 等）不读。 */ +const ValidatorVerdict = Type.Object({ + valid: Type.Boolean(), + computed_sum: NumberOrNull, + reasons: Type.Array(Type.String()), + diagnostics: Type.Object({ min_pairwise_clearance: NumberOrNull }), + legalization: Type.Object({ + raw_valid: Type.Boolean(), + raw_score: NumberOrNull, + raw_violation: NumberOrNull, + applied: Type.Boolean(), + repaired: Type.Boolean(), + shrunk_circles: Type.Array(Type.Integer()), + reasons: Type.Array(Type.String()), + }), +}); +type ValidatorVerdict = Static; + +/** 校验器在形状不对、含非有限数或负半径时提前返回，几何量没算。 */ +const NOT_COMPUTED = "原产物形状不对、含非有限数或负半径，校验器没算这个量"; + +function orUnmeasured(value: number | null, reason: string): Measured { + return value === null ? unmeasured(reason) : measured(value); +} + +function readVerdict(judgeDir: string, exitCode: number): ValidatorVerdict { + const path = join(judgeDir, VERDICT); + const verdict = readJson(path); + if (!Value.Check(ValidatorVerdict, verdict)) throw new Error(`${path} 的内容认不出：${JSON.stringify(verdict)}`); + // 退出码 0 = 合法、1 = 不合法；合法化之后的结论就是最终判定。对不上说明跑的不是这份校验器。 + const consistent = + verdict.valid === (exitCode === 0) && + verdict.legalization.repaired === verdict.valid && + (!verdict.valid || verdict.computed_sum !== null); + if (!consistent) throw new Error(`${path} 与退出码 ${exitCode} 或自身前后对不上：${JSON.stringify(verdict)}`); + return verdict; +} + +function judge( + _task: Task, + runDir: string, + judgeDir: string, + formal: ProcessResult, + validator: ProcessResult, + hypothesisMetrics: HypothesisMetric[], +): JudgedResult { + const run = started(formal); + const verdict = readVerdict(judgeDir, requireExit("校验器", validator, [0, 1])); + const legal = verdict.legalization; + let deduction: Measured; + let score: Measured; + if (verdict.valid) { + const final = verdict.computed_sum!; + score = measured(final); + // legal.repaired 为真时 raw_score 一定有值：原产物能算出分数，才可能合法或被修好。 + const raw = legal.raw_score!; + // 只缩半径只会降分；修后反而更高，说明修法写错了（方法.md 第八部分"合法化"第 3 条）。 + if (final > raw) throw new Error(`合法化后分数 ${final} 高于原分数 ${raw}：只缩半径只会降分，修法出错了`); + deduction = measured(raw - final); + } else { + const reason = `不算数：${verdict.reasons.concat(legal.reasons).join("；")}`; + score = unmeasured(reason); + deduction = unmeasured(`修不了，没有修后分数：${legal.reasons.join("；")}`); + } + const rawScore = orUnmeasured(legal.raw_score, NOT_COMPUTED); + const rawViolation = orUnmeasured(legal.raw_violation, NOT_COMPUTED); + const fromValidator = { + [CLEARANCE]: orUnmeasured(verdict.diagnostics.min_pairwise_clearance, NOT_COMPUTED), + [RAW_SCORE]: rawScore, + [RAW_VIOLATION]: rawViolation, + [DEDUCTION]: deduction, + }; + const reason = verdict.valid + ? legal.raw_valid + ? "校验器按容差 0 判原产物合法" + : `原产物不合法，按只缩不移规则修好后合法（缩了 ${legal.shrunk_circles.length} 个圆）` + : `校验器判不合法：${verdict.reasons.concat(legal.reasons).join("；")}`; + const extra = { + legalization: { + rawValid: legal.raw_valid, + applied: legal.applied, + repaired: legal.repaired, + shrunkCircles: legal.shrunk_circles, + reasons: legal.reasons, + rawScore, + rawViolation, + deduction, + }, + }; + return assemble(runDir, run, hypothesisMetrics, { valid: verdict.valid, reason }, score, fromValidator, extra); +} + +export const circlePacking: TaskImplementation = { + metricDefinitions, + prepareRun, + formalRun, + lightCheck, + readLightCheck, + prepareJudging, + judge, +}; diff --git a/src/tasks/facts.ts b/src/tasks/facts.ts new file mode 100644 index 0000000..d7b51ff --- /dev/null +++ b/src/tasks/facts.ts @@ -0,0 +1,90 @@ +import { Type, type Static } from "typebox"; +import { Settings } from "typebox/system"; +import { Value } from "typebox/value"; +import { Direction, Id } from "../vocab/index.ts"; + +// task.json 的字段定义与检查。task.json 是用户提供的东西，属于系统边界（CLAUDE.md"罕见情况怎么办"）， +// 所以每个字段必填、不认识的字段报错，一次列出全部错误，每条写明字段路径。可以没有的值写 null。 + +/** 任务种类。每一种在 task.ts 的对照表里对应一份实现；nanochat、AI4AI 以后按同样的形状加。 */ +export const TASK_KINDS = ["circle_packing"] as const; +export type TaskKind = (typeof TASK_KINDS)[number]; + +const CLOSED = { additionalProperties: false } as const; + +export const TaskFacts = Type.Object( + { + kind: Type.Enum(TASK_KINDS, { type: "string", description: "任务种类，决定用哪一份实现" }), + finalMetric: Type.Object( + { + id: Id, + direction: Direction, + }, + { ...CLOSED, description: "最终指标的编号和方向（方法.md 第十部分第 1 条）" }, + ), + epsilon: Type.Number({ exclusiveMinimum: 0, description: "精度 ε（方法.md 6.11）" }), + knownBest: Type.Union([Type.Number(), Type.Null()], { + description: "已知最好成绩；没有就写 null（方法.md 6.11）", + }), + run: Type.Object( + { + wallSeconds: Type.Number({ exclusiveMinimum: 0, description: "单次正式运行的墙钟时限（秒）" }), + cores: Type.Integer({ minimum: 1, description: "单次正式运行的核数" }), + runtime: Type.String({ minLength: 1, description: "Docker 运行时，必须写明，例如 runc" }), + }, + { ...CLOSED, description: "同批候选共用的比较条件（方法.md 第八部分）" }, + ), + protectedFiles: Type.Array(Type.String({ minLength: 1 }), { + description: "不许改的文件，路径相对候选代码目录（方法.md 第十部分第 2 条）", + }), + }, + CLOSED, +); +export type TaskFacts = Static; + +type SchemaError = ReturnType[number]; + +const TYPES: Readonly> = { number: "数字", integer: "整数", string: "字符串", object: "对象", array: "数组" }; + +/** 本文件的字段用得到的几种错误，写成中文；其余照 typebox 的原文。 */ +const REASONS: Readonly) => string>> = { + type: (p) => `应为${TYPES[String(p.type)] ?? String(p.type)}`, + enum: (p) => `只能是 ${(p.allowedValues as unknown[]).map((v) => JSON.stringify(v)).join("、")} 之一`, + exclusiveMinimum: (p) => `必须大于 ${String(p.limit)}`, + minimum: (p) => `不能小于 ${String(p.limit)}`, + minLength: () => "不能为空", + pattern: (p) => `格式不对，应符合 ${String(p.pattern)}`, +}; + +function errorLines(errors: SchemaError[]): string[] { + const lines: string[] = []; + for (const e of errors) { + const p = e.params as Record; + const at = e.instancePath || "/"; + // 唯一的几选一是 knownBest（数字或 null）；各分支自己的错误不再逐条列。 + if (e.schemaPath.includes("/anyOf/")) continue; + if (e.keyword === "anyOf") lines.push(`${at}：应为数字，没有就写 null`); + else if (e.keyword === "required") { + for (const name of p.requiredProperties as string[]) lines.push(`${e.instancePath}/${name}：缺少这一项（每个字段都必填）`); + } else if (e.keyword === "additionalProperties") { + for (const name of p.additionalProperties as string[]) lines.push(`${e.instancePath}/${name}：不认识的字段`); + } else lines.push(`${at}：${REASONS[e.keyword]?.(p) ?? e.message}`); + } + return lines; +} + +/** 检查一份已经解析好的 task.json；合格就原样返回，不合格就报错，列出全部错误。 */ +export function checkTaskFacts(value: unknown): TaskFacts { + if (Value.Check(TaskFacts, value)) return value; + // typebox 默认最多收 8 条错误，不放开就会漏报；放开只在这一次调用里。 + const cap = Settings.Get().maxErrors; + Settings.Set({ maxErrors: Infinity }); + let errors: SchemaError[]; + try { + errors = [...Value.Errors(TaskFacts, value)]; + } finally { + Settings.Set({ maxErrors: cap }); + } + const lines = errorLines(errors); + throw new Error(`task.json 不合格，共 ${lines.length} 处：\n${lines.join("\n")}`); +} diff --git a/src/tasks/index.ts b/src/tasks/index.ts new file mode 100644 index 0000000..a1fbc6e --- /dev/null +++ b/src/tasks/index.ts @@ -0,0 +1,15 @@ +// tasks：任务目录里的事实和文件、怎么准备一次运行、怎么从原始产物独立判分（docs/结构.md 第二部分）。 +// 不起子进程：要执行任务代码的事都只交出运行定义，由调用方交给 runner 在任务镜像里跑。 +export { checkTaskFacts, TASK_KINDS, TaskFacts, type TaskKind } from "./facts.ts"; +export { + formalRun, + judge, + lightCheck, + loadTask, + metricDefinitions, + prepareJudging, + prepareRun, + protectedChanges, + readLightCheck, +} from "./task.ts"; +export type { JudgingStep, LightCheckOutcome, Task } from "./types.ts"; diff --git a/src/tasks/metrics-file.ts b/src/tasks/metrics-file.ts new file mode 100644 index 0000000..0112de9 --- /dev/null +++ b/src/tasks/metrics-file.ts @@ -0,0 +1,88 @@ +import { existsSync, readFileSync } from "node:fs"; +import { + ANSWER_PATH_KEY, + SAFEGUARD_TRIGGERS_KEY, + type AnswerPath, + type HypothesisMetric, + type Measured, + type SafeguardTriggers, +} from "../vocab/index.ts"; + +// 读候选写的指标文件：假设专用指标的取值，加两个保留键（方法.md 第五部分"指标""答案路径"）。 +// 这个文件是候选的产物，读不到、写错了都记成明确的状态并附原因，不补 0、不补空对象，也不报错停下。 + +export interface MetricsFileReading { + /** 每条登记的假设专用指标一项，键是指标编号。 */ + hypothesis: Record; + answerPath: AnswerPath; + safeguardTriggers: SafeguardTriggers; +} + +export function unmeasured(reason: string): Measured { + return { status: "unmeasured", reason }; +} + +export function measured(value: number): Measured { + return { status: "measured", value }; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** 整个文件都读不到时，每一项都记同一个原因。 */ +function allUnreadable(reason: string, metrics: HypothesisMetric[]): MetricsFileReading { + return { + hypothesis: Object.fromEntries(metrics.map((m) => [m.id, unmeasured(reason)])), + answerPath: { status: "unreadable", reason }, + safeguardTriggers: { status: "unreadable", reason }, + }; +} + +function readMetric(values: Record, id: string, name: string): Measured { + const value = values[id]; + if (value === undefined) return unmeasured(`${name} 里没有指标 ${id}`); + if (value === null) return unmeasured(`候选写的是 null：这一次没测到 ${id}`); + // JSON.parse 会把 1e999 读成 Infinity，所以要查有限。 + if (typeof value !== "number" || !Number.isFinite(value)) { + return unmeasured(`${name} 里指标 ${id} 的值不是有限数：${JSON.stringify(value)}`); + } + return measured(value); +} + +function readAnswerPath(values: Record, name: string): AnswerPath { + const value = values[ANSWER_PATH_KEY]; + if (value === undefined) return { status: "unreadable", reason: `${name} 里没写 ${ANSWER_PATH_KEY}` }; + if (typeof value !== "string") { + return { status: "unreadable", reason: `${name} 里 ${ANSWER_PATH_KEY} 不是字符串：${JSON.stringify(value)}` }; + } + return { status: "read", value }; +} + +function readTriggers(values: Record, name: string): SafeguardTriggers { + const value = values[SAFEGUARD_TRIGGERS_KEY]; + if (value === undefined) return { status: "unreadable", reason: `${name} 里没写 ${SAFEGUARD_TRIGGERS_KEY}` }; + const counts = isRecord(value) ? Object.entries(value) : null; + if (counts === null || counts.some(([, n]) => !Number.isInteger(n) || (n as number) < 0)) { + const reason = `${name} 里 ${SAFEGUARD_TRIGGERS_KEY} 应为"保障短名 → 非负整数"的对象，实际是 ${JSON.stringify(value)}`; + return { status: "unreadable", reason }; + } + return { status: "read", counts: Object.fromEntries(counts) as Record }; +} + +/** 读指标文件。metrics 是这次登记的假设专用指标，只按它们的编号取值，多写的键不读。 */ +export function readMetricsFile(path: string, name: string, metrics: HypothesisMetric[]): MetricsFileReading { + if (!existsSync(path)) return allUnreadable(`${name} 不存在`, metrics); + let values: unknown; + try { + values = JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + return allUnreadable(`${name} 不是合法的 JSON：${(error as Error).message}`, metrics); + } + if (!isRecord(values)) return allUnreadable(`${name} 应为 JSON 对象（指标编号 → 值）`, metrics); + return { + hypothesis: Object.fromEntries(metrics.map((m) => [m.id, readMetric(values, m.id, name)])), + answerPath: readAnswerPath(values, name), + safeguardTriggers: readTriggers(values, name), + }; +} diff --git a/src/tasks/task.ts b/src/tasks/task.ts new file mode 100644 index 0000000..8551354 --- /dev/null +++ b/src/tasks/task.ts @@ -0,0 +1,107 @@ +import { readFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import type { HypothesisMetric, JudgedResult, MetricDefinition, ProcessResult, RunDefinition } from "../vocab/index.ts"; +import { circlePacking } from "./circle-packing.ts"; +import { checkTaskFacts, type TaskKind } from "./facts.ts"; +import type { JudgingStep, LightCheckOutcome, Task, TaskImplementation } from "./types.ts"; + +// tasks 对外的函数。任务目录的约定：task.json（任务事实）、TASK.md（给实验员的说明）、 +// Dockerfile 与依赖清单（建镜像）、baseline/（基线代码），以及各任务自己的 Python 工具。 +// 每个函数按 task.facts.kind 从对照表里取出那种任务的实现来做。 + +/** 任务种类 → 实现。加一种任务就在这里加一行。 */ +const IMPLEMENTATIONS: Readonly> = { + circle_packing: circlePacking, +}; + +function implementation(task: Task): TaskImplementation { + return IMPLEMENTATIONS[task.facts.kind]; +} + +/** 读一个任务目录：严格检查 task.json，同时交回 Dockerfile 所在目录、基线目录和 TASK.md 全文。 */ +export function loadTask(taskDir: string): Task { + const dir = resolve(taskDir); + const path = join(dir, "task.json"); + let value: unknown; + try { + value = JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + throw new Error(`${path} 读不出来或不是合法的 JSON：${(error as Error).message}`, { cause: error }); + } + return { + facts: checkTaskFacts(value), + dir, + baselineDir: join(dir, "baseline"), + guide: readFileSync(join(dir, "TASK.md"), "utf8"), + }; +} + +/** 最终指标与常规辅助指标的定义。最终指标的编号和方向取自 task.json。 */ +export function metricDefinitions(task: Task): MetricDefinition[] { + return implementation(task).metricDefinitions(task.facts); +} + +/** + * 准备一次正式运行的运行目录：把候选代码、任务的 harness 和这次要测的假设专用指标说明 + * （metrics-spec.json）放进去。runDir 由调用方建好，应为空目录。 + * 消融格里传进来的指标应当已经全部改成"缺了只记缺失"（vocab 指标说明），这里不区分。 + */ +export function prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void { + implementation(task).prepareRun(task, candidateDir, runDir, hypothesisMetrics); +} + +/** + * 正式运行的运行定义。种子由调用方给，同一份研究的全部运行用同一个种子： + * 评测要确定，消融不重跑基线也靠这一点（方法.md 第十部分第 9 条）。 + */ +export function formalRun(task: Task, imageDigest: string, seed: string): RunDefinition { + return implementation(task).formalRun(task, imageDigest, seed); +} + +/** + * 轻量检查：把候选代码和检查脚本放进 checkDir（调用方建好的空目录），交回运行定义。 + * 只看语法和入口函数，不执行候选代码。seconds 来自配置；次数上限不归这里管。 + */ +export function lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition { + return implementation(task).lightCheck(task, candidateDir, checkDir, imageDigest, seconds); +} + +/** 解读轻量检查：通过，或者不通过及原因。检查脚本自己没正常跑完时报错停下。 */ +export function readLightCheck(task: Task, checkDir: string, result: ProcessResult): LightCheckOutcome { + return implementation(task).readLightCheck(checkDir, result); +} + +/** + * 判分第一步。正式运行没正常结束、或者没有产物时直接判不算数；否则把产物和宿主保管的校验器 + * 复制进 judgeDir（调用方建好的空目录），交回校验器的运行定义。 + */ +export function prepareJudging( + task: Task, + runDir: string, + judgeDir: string, + imageDigest: string, + formal: ProcessResult, + hypothesisMetrics: HypothesisMetric[], +): JudgingStep { + return implementation(task).prepareJudging(task, runDir, judgeDir, imageDigest, formal, hypothesisMetrics); +} + +/** + * 判分第二步：读校验器写的判定、运行目录里候选写的指标文件，加上正式运行的结果，交回判分结果。 + * 校验器自己没正常跑完、判定文件认不出、修后分数高于原分数时报错停下：这些是宿主一侧的问题。 + */ +export function judge( + task: Task, + runDir: string, + judgeDir: string, + formal: ProcessResult, + validator: ProcessResult, + hypothesisMetrics: HypothesisMetric[], +): JudgedResult { + return implementation(task).judge(task, runDir, judgeDir, formal, validator, hypothesisMetrics); +} + +/** 改动过的文件里，属于 task.json"不许改"名单的那些。路径都相对候选代码目录。 */ +export function protectedChanges(task: Task, changedFiles: string[]): string[] { + return changedFiles.filter((file) => task.facts.protectedFiles.includes(file)); +} diff --git a/src/tasks/types.ts b/src/tasks/types.ts new file mode 100644 index 0000000..1645ecf --- /dev/null +++ b/src/tasks/types.ts @@ -0,0 +1,49 @@ +import type { HypothesisMetric, JudgedResult, MetricDefinition, ProcessResult, RunDefinition } from "../vocab/index.ts"; +import type { TaskFacts } from "./facts.ts"; + +/** 读进来的一个任务。 */ +export interface Task { + /** task.json 的内容，已严格检查。 */ + facts: TaskFacts; + /** 任务目录的绝对路径，里面有 Dockerfile 和依赖清单，runner 拿它建镜像。 */ + dir: string; + /** 基线代码目录。 */ + baselineDir: string; + /** TASK.md 全文，给实验员看的任务说明。 */ + guide: string; +} + +/** 轻量检查的结论：通过，或者不通过及原因。 */ +export type LightCheckOutcome = { passed: true } | { passed: false; reasons: string[] }; + +/** + * 判分第一步的结果：要么不用跑校验器就已判定（正式运行没正常结束、没有产物）， + * 要么交回校验器的运行定义，由调用方交给 runner 去跑。 + */ +export type JudgingStep = { status: "judged"; result: JudgedResult } | { status: "validate"; definition: RunDefinition }; + +/** 一种任务的实现。对外函数按 task.facts.kind 从对照表里取出它来调。参数的含义见 task.ts 里同名的对外函数。 */ +export interface TaskImplementation { + /** 最终指标（编号和方向取自 task.json）与常规辅助指标的定义。 */ + metricDefinitions(facts: TaskFacts): MetricDefinition[]; + prepareRun(task: Task, candidateDir: string, runDir: string, hypothesisMetrics: HypothesisMetric[]): void; + formalRun(task: Task, imageDigest: string, seed: string): RunDefinition; + lightCheck(task: Task, candidateDir: string, checkDir: string, imageDigest: string, seconds: number): RunDefinition; + readLightCheck(checkDir: string, result: ProcessResult): LightCheckOutcome; + prepareJudging( + task: Task, + runDir: string, + judgeDir: string, + imageDigest: string, + formal: ProcessResult, + hypothesisMetrics: HypothesisMetric[], + ): JudgingStep; + judge( + task: Task, + runDir: string, + judgeDir: string, + formal: ProcessResult, + validator: ProcessResult, + hypothesisMetrics: HypothesisMetric[], + ): JudgedResult; +} diff --git a/src/vocab/runs.ts b/src/vocab/runs.ts index c9f9b4a..50e6c93 100644 --- a/src/vocab/runs.ts +++ b/src/vocab/runs.ts @@ -128,7 +128,8 @@ const RESOURCE_USAGE = { /** * 以什么方式结束。有没有启动由它决定：只有 notStarted 是没启动， * 其余三种都启动过、要扣次数（方法.md 第七部分）。 - * 启动前就取消的记 notStarted；启动后取消的记 cancelled。 + * 启动前就取消的记 notStarted；cancelled 是容器从外面被杀了（关机、手动 docker kill、 + * 计量的 shell 自己被杀），没有留下计量结果，三项资源都记测不到。 * 退出码只有正常退出才有，资源用量只有启动过才有，所以分成四种形状。 */ export const ProcessResult = Type.Union([ @@ -168,8 +169,8 @@ export type ProcessResult = Static; // 运行结果第二层：tasks 从原始产物独立判分 // --------------------------------------------------------------------------- -/** 判定三选一：算数、不算数、判不了（判定所需的证据没拿到）。 */ -export const VERDICTS = ["valid", "invalid", "undecidable"] as const; +/** 判定二选一：算数、不算数。 */ +export const VERDICTS = ["valid", "invalid"] as const; export type Verdict = (typeof VERDICTS)[number]; /** diff --git a/tasks/circle_packing/Dockerfile b/tasks/circle_packing/Dockerfile new file mode 100644 index 0000000..7f9eff3 --- /dev/null +++ b/tasks/circle_packing/Dockerfile @@ -0,0 +1,10 @@ +# 基础镜像按 digest 钉死，不用 tag——tag 会被重新指向，digest 不会。 +# 这里写全仓库名（dockerproxy.net/library/python），因为本机没配 registry mirror， +# 只写 python 会去 docker.io 拉。 +FROM dockerproxy.net/library/python:3.13-slim@sha256:8d9d0b8bcf6506481eae4907c18f5e3e7902e629f5f6d684f9e7c32e85e3ddf0 +# 每次运行只给一个核。numba 不设这个变量时 parallel=True 会按宿主核数开线程 +# （调研_填圆运行环境依赖.md 3.3 实测开 16 个）；写在镜像里，由镜像编号钉住， +# 不依赖每份配置记得写。数学库的线程数由 task.env 的 OMP_NUM_THREADS 管。 +ENV NUMBA_NUM_THREADS=1 +COPY requirements.txt /req.txt +RUN pip install --no-cache-dir -r /req.txt diff --git a/tasks/circle_packing/TASK.md b/tasks/circle_packing/TASK.md new file mode 100644 index 0000000..711eb95 --- /dev/null +++ b/tasks/circle_packing/TASK.md @@ -0,0 +1,295 @@ +# 填圆任务（n=26，单位正方形） + +这份文件是给实验员看的任务契约：要做什么、产物写在哪、键名怎么写。 +读它的时候假定你手上已经有一份冻结的提案，本文件不重复提案里的内容。 + +## 目标 + +在单位正方形 `[0,1]²` 内放置 **26 个互不重叠、不越过边界**的圆，**最大化半径之和**。 + +最终指标：`sum-of-radii`，越大越好。 + +量级参照：容差 0 的口径下，26 个圆公开的最好半径和约 **2.635983**（AlphaEvolve 续作公开的构造， +按它自己的严格校验在本机重算得 2.6359830849）。常见的"2.635"是白皮书截到三位小数的数，不是另一个纪录。 +别处报告的更高的数要先看容差：容差放到 1e-6，把每个半径加 5e-7 也判合格，和就能到 2.635996——那是口径不同，不是更好的摆法。 + +**成绩是从你交出来的产物算的，不是从你返回的那个数读的。** +`validate.py` 读 `solution.json` 里的 26 个半径，用 `math.fsum` 求和，那就是成绩。 + +你在 `run_packing()` 里返回的第三个值（`sum_of_radii`）**不是成绩**。 +`harness.py` 照常把它写进 `solution.json` 的 `reported_sum` 字段，但 +`validate.py` 根本不读这个字段：它不进判定、不进诊断，也不跟任何数对账。 +所以这个数写过期了、跟半径对不上，产物照样合法，成绩照样是算出来的那一个。 + +只有一件事要注意，而且跟对账无关：`harness.py` 落盘时不接受非有限数， +所以这个值要是 `NaN` 或无穷大，整份 `solution.json` 根本写不出来， +这次运行连产物都没有。返回一个普通的有限数就行，哪怕它是错的。 + +## 评测会收下什么 + +判断代码里一段检查、修复或兜底"挡掉了谁、改了什么"，要先知道评测本身会收下什么。 +**评测收下一次运行，按下面的次序：** + +1. 你交的 `solution.json` 形状对（26 个圆心、26 个半径）、全是有限数、半径不为负、圆心都在正方形里。 + 这几条任何一条不成立，评测**不收**，也不修。 +2. 原产物按**容差 0** 判：任意两圆不重叠、每个圆不越出正方形边。成立就收下，成绩是原产物的半径和。 +3. 不成立时，先按下一节"只缩不移合法化"修一次，修后再按容差 0 判。修得好就收下，成绩是修后的半径和 + （只比原产物低一点）；修不好才不收。 + +所以"差一点不合法"（重叠或越界只有浮点零头那么大）的产物，**评测本来会收**，只扣掉零头。 +候选代码里比这更严的检查——例如容差 0 判不合法就把这个起点整个丢掉、或换成另一份解——挡掉的正是评测会收的产物， +它改的是交出去的答案，不是"让代码合法"。反过来，只在评测确实不收的情形（形状不对、非有限数、圆心出界） +才动手修或换的代码，在评测会收的运行上不改答案。 + +## 差一点不合法时：只缩不移合法化 + +校验器先按容差 0 判你交的原产物。原产物不合法时，宿主按下面这条写死的规则修一次， +**只缩半径、不挪圆心**，修好了就用修后的半径和当成绩： + +1. **只缩参与违反的圆。**每个圆的半径减去它参与的最大违反量——跟别的圆重叠的量、 + 越出正方形边的量，取最大的那个。没参与违反的圆一个都不动。 +2. 缩完仍不合法（浮点舍入留下的零头），把这些圆的半径各往 0 挪一个最小的浮点单位， + 再判，直到合法。 +3. 同一个校验器按容差 0 重判修后的产物。 +4. **修不了**：有圆心在正方形外面；参与违反的圆半径都缩到 0 仍不合法； + 形状不对、含非有限值或负半径（这些不是"差一点"，缩半径解决不了）。 + 修不了的运行照旧记失败，运行次数照扣——跟跑崩、超时、没产物一样。 + +**缩小只会降分**，所以这条规则不会让不合法的摆法比合法的更高分。它管的是"浮点上差一点"： +例如两个圆重叠 2.8e-17，修后扣掉约 2.2e-16。要一分不扣，就在代码里给每个圆留一点空隙。 + +每次运行多记三个数，跟成绩一起进运行记录：原违反量（`raw-violation`）、 +原分数（`raw-score`）、合法化扣掉的分（`legalization-deduction`，原产物合法时是 0）。 + +## 什么样的结果才算数 + +下面五条不是禁令，是**这次实验要成立需要的条件**。写出来是为了让你知道该往哪边走， +不是为了防谁。 + +### 一 · 成绩必须是算出来的，不能是写死的 + +代码里可以有常数：半径比例、松弛步长、网格怎么划分、迭代多少轮，都可以写死。 +**不能有的是一张 26 行的答案表**——把某次跑出来的圆心和半径直接抄进源码返回。 + +*为什么*：研究要找的是方法，不是一个数。一份写死的答案没有机制可以解释—— +问它"为什么这个改法有效"，答案是"因为答案是抄的"，而这句话不会出现在提案里。 +一个没有方法的高分会让整轮研究的结论是假的：后面几轮会建立在 +"某某方向有效"这个不存在的发现之上。 + +### 二 · 只读工作目录里的东西，不联网 + +你可以带自己的数据文件，它们会跟代码一起冻结、一起放进工作目录。 +但运行时不要去读工作目录外面的文件。正式运行的容器是断网的，网络请求一律失败。 + +*为什么*：冻结的代码要能完整说清"这次跑的是什么"。 +从网上取一份东西进来，那份东西明天可能就变了，这次运行就再也重现不了； +读工作目录外面的文件也一样，那个文件不在冻结的代码里。 + +### 三 · 在执行限额之内跑完 + +每次正式运行的墙钟时限是 120 秒、1 核，同一批候选都一样。到时限还没结束，进程被直接杀掉， +这次运行记为超时失败，运行次数照扣。 + +*为什么*：跟基线比的前提是同样的条件。用一百倍时间换来的成绩， +说明的是算力，不是方法。 + +### 四 · 用了多少算力要说出来 + +每次运行都记墙钟时间（`wall-seconds`）、CPU 秒（`cpu-seconds`）和峰值内存（`peak-memory-bytes`）， +材料里同时写"是基线的几倍"和"占单次时限的几成"。这些只记录，不影响采纳。 + +**加大搜索预算本身是正当的做法**，不需要藏着。但用远多于基线的算力换来的提升， +说明的是算力，不是方法——所以要在提案里写明加了多少、为什么值得。 + +*为什么*：跟基线比的前提是同样的条件。一次多花一百五十倍时间、换来百分之七提升的运行， +如果被记成"某某改法有效"，后面几轮会去改那个改法，而真正起作用的是预算。 + +### 五 · 说了什么就要真的做什么 + +代码和注释里写了"迭代到收敛"，就要真的迭代到收敛；写了"最多 1000 步"， +就不能实际只跑 2 步。 + +*为什么*：这一条跟前面几条不一样，**它坏的不是成绩，是解释**。 +成绩可能完全是真的、可复现的，但"松弛到收敛有效"这条结论会进记录， +下一轮建立在一个没有发生过的机制之上。对不上的时候， +该作废的是那条结论，不是这次运行。 + +这也是为什么第一条说"没有方法的高分会让整轮研究的结论是假的"—— +证据链上的错，后面每一轮都看不见。 + +## 运行方法 + +候选可以是多文件 Python 项目，普通相对数据资源会保留；入口 `initial.py` 必须定义： + +```python +def run_packing(): + ... + return centers, radii, sum_of_radii # (26,2)、(26,)、float +``` + +正式运行由程序启动，在任务镜像里、断网、1 核、120 秒时限下执行（工作目录就是你的代码目录）： + +```bash +python -B harness.py --program initial.py --output solution.json +``` + +跑完以后，宿主把 `solution.json` 复制到另一个目录，用自己保管的校验器判分： + +```bash +python -B validate.py --circles 26 --solution solution.json --atol 0 --legalize --verdict verdict.json +``` + +`harness.py` 在 import 候选前，用环境变量 `INFORSI_SEED` 给的种子字符串（取 SHA256 前 4 字节大端整数） +初始化 Python 的 `random` 和 numpy 的 `random`，执行候选并原子落盘原始解，不做任何有效性判断。 +import 之前它还会删掉工作目录里已有的 `solution.json` 和 `metrics.json`，宿主读到的只会是这一次写的。 +你要用随机，用这两个。同一份研究的全部运行用同一个种子，所以同一份代码跑两次，结果逐位相同。 + +`validate.py` 是独立校验器，重算半径和并检查形状 / 有限 / 非负 / 边界 / 重叠； +加 `--legalize` 时按上面"只缩不移合法化"一节修差一点不合法的产物（正式评测就是这样调的）。 + +写代码的过程中可以做**轻量检查**：由宿主在同一个镜像里跑，只看语法和入口函数， +不执行候选的任何一行代码。它有次数和时长上限（每次几秒），不算一次正式运行。 + +**不要自己跑正式规模的求解。** 正式运行要先登记、再启动、再扣次数，由程序统一做； +正式规模的求解哪怕叫"调试"，也是正式运行，不能借轻量检查或自己的 shell 绕开预算。 + +## 指标怎么交给宿主（这一节最容易写错，先读完再动手） + +工作目录里有两个指标文件，一进一出。**路径都相对工作目录**，也就是 `harness.py` +运行时的 cwd，不要写绝对路径。 + +### `metrics-spec.json`：宿主写给你的 + +宿主在准备运行目录时写好它，内容是这次运行登记的假设专用指标定义， +一个 JSON 数组，每个元素是一条指标： + +| 字段 | 含义 | +| --- | --- | +| `id` | 指标编号，宿主按它取值 | +| `meaning` | 这个量测的是什么 | +| `unit` | 单位 | +| `direction` | `maximize` 越大越好，`minimize` 越小越好 | +| `howMeasured` | 研究者写的测法：在什么时刻测、测几次、怎么合成一个数。**你照着它实现** | +| `missing` | 测不到时这次运行怎么算：`recordMissing` 只记缺失、成绩照常；`invalidateRun` 整次作废 | + +没有假设专用指标时，这个文件是一个空数组 `[]`。 + +### `metrics.json`：你写给宿主的 + +一个 JSON 对象：**键必须是 `metrics-spec.json` 里那条指标的 `id` 字段原文**， +值是有限数或 `null`。另外**每次都要写两个保留键**（下面"两个保留键"一节），它们不是指标、值不是数。 + +> 编号从 `metrics-spec.json` 里**照抄**，不要自己拼。宿主按编号取值，键对不上， +> 这条指标就记成"测不到"加一句原因——**而成绩完全正常**，报告上看不出任何异常， +> 只是研究者分析机制时悄悄少了一块证据。 + +四条硬要求： + +1. **键照抄 `id`**，见上。 +2. **值是有限数或 `null`。测不到就写 `null`，不要写 0。** + 写 0 之后宿主分不出"测到了一个 0"和"没测到"，而这种错事后从证据里查不出来。 + 算出 `NaN` / `inf`（退化布局里零半径参与的比值最常见）也一样，那是"没测到"。 + harness 会把非有限值原样改写成 `null` 并在 stderr 记一行，退出码仍是 0—— + 成绩不受影响，但那条指标这一次就是没有数据。 +3. **必须在 `run_packing()` 返回之前写完。** harness 在候选返回后立刻整理这个文件； + 之后再写进去的内容不会被整理（非有限值不会变成 `null`，裸 `NaN` 会让宿主整份读不出来， + **这一次所有假设专用指标一起丢**，不只是坏的那一条）。 +4. 键多了无所谓，宿主只按 `metrics-spec.json` 里的编号取值；键少了，缺的那条记"测不到"加一句原因。 + +文件不存在表示这一次的假设专用指标一个都没测到——这是一个明确状态，不是错误， +但它和"写错了键名"在报告里看起来一样，所以别靠它。 + +**测不到的后果看 `missing`**：选了 `invalidateRun` 的指标测不到，这次运行整次作废、不能被采纳， +运行次数照扣；选了 `recordMissing` 的只记缺失。 + +### 两个保留键：答案出自哪条路径 + +研究者不能运行代码，"最终交的答案到底是哪段代码给的"只能读代码猜，猜错了没人发现。 +所以每次运行由你的代码自己说出来，写在同一个 `metrics.json` 里： + +| 键 | 值 | 写什么 | +| --- | --- | --- | +| `answer-path` | 字符串 | 最终交的答案出自哪里：`main`（正常主流程），或冻结协议因子清单里某个因子的编号（`p1`、`p2`……），或保障清单里某条保障的短名 | +| `safeguard-triggers` | 对象 | 键是冻结协议保障清单里的**每一条**保障短名，值是这次运行里它动手了几次（非负整数，没动手写 0）；保障写"无"时写 `{}` | + +- 这两个键**不是指标**，不参与成绩判定；写错、没写都不作废运行，宿主照实记"候选没写答案路径" + 或"不认识的路径：原文"。但研究者下一阶段就靠它看"这一格量到的是因子还是保障"，所以要如实写。 +- 消融格（关掉某个因子的版本）同样要写：关掉的因子不会是答案路径；如果关掉之后答案改由某条保障给出，就写那条保障。 +- harness 不改写这两个键（指标值才会把非有限数改写成 `null`）。同样要在 `run_packing()` 返回之前写完。 + +### 一小段示例 + +```python +import json +import math + +SPEC_FILE = "metrics-spec.json" +METRICS_FILE = "metrics.json" + + +def run_packing(): + with open(SPEC_FILE, encoding="utf-8") as handle: + spec = json.load(handle) + # spec[i]["id"] 就是宿主认的编号，spec[i]["howMeasured"] 是照着实现的测法。 + # 这里按这次实际条数对号入座，下面只演示一条。 + spread_id = spec[0]["id"] + + centers, radii, reported_sum = solve() + + spread = float(max(radii) - min(radii)) + measured = { + spread_id: spread if math.isfinite(spread) else None, + # 两个保留键：答案出自主流程；协议里的一条保障 "shrink-repair" 这次没动手。 + "answer-path": "main", + "safeguard-triggers": {"shrink-repair": 0}, + } + + # 在 return 之前写完，路径相对工作目录。 + with open(METRICS_FILE, "w", encoding="utf-8") as handle: + json.dump(measured, handle) + + return centers, radii, reported_sum +``` + +最终指标 `sum-of-radii` 和合法性判定**不经过这条链**：它们由 `validate.py` +从 `solution.json` 独立重算，你不用也不该在 `metrics.json` 里写它们。 + +## 允许修改范围 + +候选只能修改自己代码目录里的 `*.py` 和自带的数据文件（基线的入口是 `initial.py`，其中 +`EVOLVE-BLOCK-START` / `EVOLVE-BLOCK-END` 之间是官方预留的进化区域）。 +不许改的文件见同目录 `task.json` 的 `protectedFiles`；程序会检查有没有改它们。 +`harness.py` 和 `metrics-spec.json` 在准备运行目录时由宿主放进去，你目录里的同名文件会被盖掉； +校验器和轻量检查脚本由宿主保管，不在你的目录里。也不要改校验容差或执行限额。 + +## 基线与许可 + +`baseline/` 为 SakanaAI/ShinkaEvolve 官方代码，Apache-2.0（见 `baseline/LICENSE`）， +固定提交与 sha256 见 `baseline/SOURCES.md`。基线的嵌套缩半径循环与退化布局是 +刻意保留的原始事实，不得"修复"——任何改进必须作为正式候选经过研究流程。 + +## 依赖 + +Python 3.13。正式实验环境里装的第三方库**只有下面这些**，版本与 `requirements.txt` 逐行一致： + +| 包 | 版本 | 用途 | +|---|---|---| +| numpy | 2.3.1 | 数组与矩阵运算 | +| scipy | 1.18.1 | 数值优化、线性规划、空间距离计算等 | +| numba | 0.67.0 | 把 Python 循环编译成机器码 | +| llvmlite | 0.49.0 | numba 的编译后端 | +| sympy | 1.14.0 | 符号与精确计算 | +| mpmath | 1.3.0 | 任意精度浮点运算（sympy 依赖它） | +| networkx | 3.7 | 图结构分析 | + +**依赖只有镜像里的这几个包。**运行时不能再装别的包：正式运行在固定的镜像里跑、容器断网， +你在工作目录里装的包进不了正式运行。 + +**每次运行只有 1 核。**容器按 `--cpus=1` 限总量，环境里 `PYTHON_CPU_COUNT=1`， +所以 `os.cpu_count()`、`multiprocessing.cpu_count()` 都报 1。多开进程或线程不会更快， +只会互相抢这 1 个核、少干活。数学库（`OMP_NUM_THREADS`）和 numba（`NUMBA_NUM_THREADS`）的线程数都固定为 1。numba 首次编译函数的时间算在这次运行的墙钟里。 + +**环境由镜像钉死。**任务跑在 `Dockerfile` 建出来的镜像里，这些库都烧在那个镜像里。 +镜像在研究开始时建一次，镜像编号冻结进账本；一份研究不能自己改依赖。 + +（给维护者：`Dockerfile` 的基础镜像经 `dockerproxy.net` 拉取，换机器时可能要改这一行。） diff --git a/tasks/circle_packing/baseline/LICENSE b/tasks/circle_packing/baseline/LICENSE new file mode 100644 index 0000000..1684699 --- /dev/null +++ b/tasks/circle_packing/baseline/LICENSE @@ -0,0 +1,201 @@ +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, +and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by +the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all +other entities that control, are controlled by, or are under common +control with that entity. For the purposes of this definition, +"control" means (i) the power, direct or indirect, to cause the +direction or management of such entity, whether by contract or +otherwise, or (ii) ownership of fifty percent (50%) or more of the +outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity +exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, +including but not limited to software source code, documentation +source, and configuration files. + +"Object" form shall mean any form resulting from mechanical +transformation or translation of a Source form, including but +not limited to compiled object code, generated documentation, +and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or +Object form, made available under the License, as indicated by a +copyright notice that is included in or attached to the work +(an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object +form, that is based on (or derived from) the Work and for which the +editorial revisions, annotations, elaborations, or other modifications +represent, as a whole, an original work of authorship. For the purposes +of this License, Derivative Works shall not include works that remain +separable from, or merely link (or bind by name) to the interfaces of, +the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including +the original version of the Work and any modifications or additions +to that Work or Derivative Works thereof, that is intentionally +submitted to Licensor for inclusion in the Work by the copyright owner +or by an individual or Legal Entity authorized to submit on behalf of +the copyright owner. For the purposes of this definition, "submitted" +means any form of electronic, verbal, or written communication sent +to the Licensor or its representatives, including but not limited to +communication on electronic mailing lists, source code control systems, +and issue tracking systems that are managed by, or on behalf of, the +Licensor for the purpose of discussing and improving the Work, but +excluding communication that is conspicuously marked or otherwise +designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity +on behalf of whom a Contribution has been received by Licensor and +subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +copyright license to reproduce, prepare Derivative Works of, +publicly display, publicly perform, sublicense, and distribute the +Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of +this License, each Contributor hereby grants to You a perpetual, +worldwide, non-exclusive, no-charge, royalty-free, irrevocable +(except as stated in this section) patent license to make, have made, +use, offer to sell, sell, import, and otherwise transfer the Work, +where such license applies only to those patent claims licensable +by such Contributor that are necessarily infringed by their +Contribution(s) alone or by combination of their Contribution(s) +with the Work to which such Contribution(s) was submitted. If You +institute patent litigation against any entity (including a +cross-claim or counterclaim in a lawsuit) alleging that the Work +or a Contribution incorporated within the Work constitutes direct +or contributory patent infringement, then any patent licenses +granted to You under this License for that Work shall terminate +as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the +Work or Derivative Works thereof in any medium, with or without +modifications, and in Source or Object form, provided that You +meet the following conditions: + +(a) You must give any other recipients of the Work or +Derivative Works a copy of this License; and + +(b) You must cause any modified files to carry prominent notices +stating that You changed the files; and + +(c) You must retain, in the Source form of any Derivative Works +that You distribute, all copyright, patent, trademark, and +attribution notices from the Source form of the Work, +excluding those notices that do not pertain to any part of +the Derivative Works; and + +(d) If the Work includes a "NOTICE" text file as part of its +distribution, then any Derivative Works that You distribute must +include a readable copy of the attribution notices contained +within such NOTICE file, excluding those notices that do not +pertain to any part of the Derivative Works, in at least one +of the following places: within a NOTICE text file distributed +as part of the Derivative Works; within the Source form or +documentation, if provided along with the Derivative Works; or, +within a display generated by the Derivative Works, if and +wherever such third-party notices normally appear. The contents +of the NOTICE file are for informational purposes only and +do not modify the License. You may add Your own attribution +notices within Derivative Works that You distribute, alongside +or as an addendum to the NOTICE text from the Work, provided +that such additional attribution notices cannot be construed +as modifying the License. + +You may add Your own copyright statement to Your modifications and +may provide additional or different license terms and conditions +for use, reproduction, or distribution of Your modifications, or +for any such Derivative Works as a whole, provided Your use, +reproduction, and distribution of the Work otherwise complies with +the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, +any Contribution intentionally submitted for inclusion in the Work +by You to the Licensor shall be under the terms and conditions of +this License, without any additional terms or conditions. +Notwithstanding the above, nothing herein shall supersede or modify +the terms of any separate license agreement you may have executed +with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade +names, trademarks, service marks, or product names of the Licensor, +except as required for reasonable and customary use in describing the +origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or +agreed to in writing, Licensor provides the Work (and each +Contributor provides its Contributions) on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +implied, including, without limitation, any warranties or conditions +of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A +PARTICULAR PURPOSE. You are solely responsible for determining the +appropriateness of using or redistributing the Work and assume any +risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, +whether in tort (including negligence), contract, or otherwise, +unless required by applicable law (such as deliberate and grossly +negligent acts) or agreed to in writing, shall any Contributor be +liable to You for damages, including any direct, indirect, special, +incidental, or consequential damages of any character arising as a +result of this License or out of the use or inability to use the +Work (including but not limited to damages for loss of goodwill, +work stoppage, computer failure or malfunction, or any and all +other commercial damages or losses), even if such Contributor +has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing +the Work or Derivative Works thereof, You may choose to offer, +and charge a fee for, acceptance of support, warranty, indemnity, +or other liability obligations and/or rights consistent with this +License. However, in accepting such obligations, You may act only +on Your own behalf and on Your sole responsibility, not on behalf +of any other Contributor, and only if You agree to indemnify, +defend, and hold each Contributor harmless for any liability +incurred by, or claims asserted against, such Contributor by reason +of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + +To apply the Apache License to your work, attach the following +boilerplate notice, with the fields enclosed by brackets "[]" +replaced with your own identifying information. (Don't include +the brackets!) The text should be enclosed in the appropriate +comment syntax for the file format. We also recommend that a +file or class name and description of purpose be included on the +same "printed page" as the copyright notice for easier +identification within third-party archives. + +Copyright 2025 Sakana AI + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/tasks/circle_packing/baseline/SOURCES.md b/tasks/circle_packing/baseline/SOURCES.md new file mode 100644 index 0000000..f05f527 --- /dev/null +++ b/tasks/circle_packing/baseline/SOURCES.md @@ -0,0 +1,34 @@ +# 官方基线来源声明 + +本目录文件来自 SakanaAI/ShinkaEvolve，Apache-2.0 许可（见同目录 `LICENSE`）。 + +- 固定提交：`9912af12d423504b8d580f4179fd15f5f88b8c50` +- 固定日期：2026-09-17（由项目负责人原字节下载并哈希固定在 `/tmp/rivalbits2-shinka-18k23a46/manifest.json`） + +| 文件 | 原始 URL | sha256 | 修改 | +| --- | --- | --- | --- | +| `initial.py` | | `09da8428a85792a94a9c6cc2193b6a0db1090ce598a17b75bc0db48d0953d098` | 无（逐字节复制） | +| `evaluate.py` | | `eb8b9787dc25dee24a5f5305d1e3bd876e5c336cded1c9bab7fb204612928b4d` | 无（逐字节复制） | +| `LICENSE` | | `c3e5b1c26607caf7208d5a105c113fe240630bb873fdb8ca5b4c2e32c721ddaf` | 无 | + +上表三个 sha256 是基线"没被改过"的唯一判据，由本项目的 `tests/tasks.test.ts` +（"基线三个文件的 sha256 等于 SOURCES.md 表上的值"）逐个断言。 + +它防的是**别的测试看不见的改动**：给 `initial.py` 加一行注释、或者动一个字节的 +`evaluate.py` / `LICENSE`，成绩一位不变，别的测试都不会挂。`evaluate.py` 和 `LICENSE` +根本不在运行路径上。 + +一个实测过的事实（参考项目里跑出来的）：把 `compute_max_radii` 那个白跑 105625 次的冗余双循环删掉， +半径和一位不差（都是 0.9597642169962064），但第 (7, 8) 对圆会重叠 2.7755575615628914e-17， +容差 0 下判不合法，要经合法化才收下。那个冗余循环仍然是本研究的改进起点。 + +## 已知的基线事实（刻意保留，不是缺陷） + +- `initial.py` 的 `compute_max_radii` 含嵌套的冗余缩半径循环（外层 `for i/for j` + 包了一层重复的 `for i/for j`），且外环布局（0.7 半径环经 clip 到 0.99）是退化布局； + 圆心 clip 后可能出现零半径。这些是官方基线的原样事实，是本研究的改进起点， + **不得“修复”**——任何改进必须作为正式候选经过研究流程。 +- `evaluate.py` 依赖 `shinka.core.run_shinka_eval`（ShinkaEvolve 框架内部）， + 且其聚合以候选自报的 `reported_sum` 作为 `combined_score`。本项目**不信任** + 自报分数：独立校验器 `../validate.py` 不 import 本文件，自行重算半径和。 + 本文件仅作为官方协议参考保留，不在本项目的运行路径上执行。 diff --git a/tasks/circle_packing/baseline/evaluate.py b/tasks/circle_packing/baseline/evaluate.py new file mode 100644 index 0000000..4af2a96 --- /dev/null +++ b/tasks/circle_packing/baseline/evaluate.py @@ -0,0 +1,197 @@ +""" +Evaluator for circle packing example (n=26) with improved timeout handling +""" + +import os +import argparse +import numpy as np +from typing import Tuple, Optional, List, Dict, Any + +from shinka.core import run_shinka_eval + + +def format_centers_string(centers: np.ndarray) -> str: + """Formats circle centers into a multi-line string for display.""" + return "\n".join( + [ + f" centers[{i}] = ({x_coord:.4f}, {y_coord:.4f})" + for i, (x_coord, y_coord) in enumerate(centers) + ] + ) + + +def adapted_validate_packing( + run_output: Tuple[np.ndarray, np.ndarray, float], + atol=0.0, +) -> Tuple[bool, Optional[str]]: + """ + Validates circle packing results based on the output of 'run_packing'. + + Args: + run_output: Tuple (centers, radii, reported_sum) from run_packing. + + Returns: + (is_valid: bool, error_message: Optional[str]) + """ + centers, radii, reported_sum = run_output + msg = "The circles are placed correctly. There are no overlaps or any circles outside the unit square." + if not isinstance(centers, np.ndarray): + centers = np.array(centers) + if not isinstance(radii, np.ndarray): + radii = np.array(radii) + + n_expected = 26 + if centers.shape != (n_expected, 2): + msg = ( + f"Centers shape incorrect. Expected ({n_expected}, 2), got {centers.shape}" + ) + return False, msg + if radii.shape != (n_expected,): + msg = f"Radii shape incorrect. Expected ({n_expected},), got {radii.shape}" + return False, msg + + if ( + not np.all(np.isfinite(centers)) + or not np.all(np.isfinite(radii)) + or not np.isfinite(reported_sum) + ): + msg = "Non-finite values found in centers, radii, or reported_sum." + return False, msg + + if np.any(radii < 0): + negative_indices = np.where(radii < 0)[0] + msg = f"Negative radii found for circles at indices: {negative_indices}" + return False, msg + + if not np.isclose(np.sum(radii), reported_sum, atol=atol): + msg = ( + f"Sum of radii ({np.sum(radii):.6f}) does not match " + f"reported ({reported_sum:.6f})" + ) + return False, msg + + for i in range(n_expected): + x, y = centers[i] + r = radii[i] + is_outside = ( + x - r < -atol or x + r > 1 + atol or y - r < -atol or y + r > 1 + atol + ) + if is_outside: + msg = ( + f"Circle {i} (x={x:.4f}, y={y:.4f}, r={r:.4f}) is outside unit square." + ) + return False, msg + + for i in range(n_expected): + for j in range(i + 1, n_expected): + dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2)) + if dist < radii[i] + radii[j] - atol: + msg = ( + f"Circles {i} & {j} overlap. Dist: {dist:.4f}, " + f"Sum Radii: {(radii[i] + radii[j]):.4f}" + ) + return False, msg + return True, msg + + +def get_circle_packing_kwargs(run_index: int) -> Dict[str, Any]: + """Provides keyword arguments for circle packing runs (none needed).""" + return {} + + +def aggregate_circle_packing_metrics( + results: List[Tuple[np.ndarray, np.ndarray, float]], results_dir: str +) -> Dict[str, Any]: + """ + Aggregates metrics for circle packing. Assumes num_runs=1. + Saves extra.npz with detailed packing information. + """ + if not results: + return {"combined_score": 0.0, "error": "No results to aggregate"} + + centers, radii, reported_sum = results[0] + + public_metrics = { + "centers_str": format_centers_string(centers), + "num_circles": centers.shape[0], + } + private_metrics = { + "reported_sum_of_radii": float(reported_sum), + } + metrics = { + "combined_score": float(reported_sum), + "public": public_metrics, + "private": private_metrics, + } + + extra_file = os.path.join(results_dir, "extra.npz") + try: + np.savez( + extra_file, + centers=centers, + radii=radii, + reported_sum=reported_sum, + ) + print(f"Detailed packing data saved to {extra_file}") + except Exception as e: + print(f"Error saving extra.npz: {e}") + metrics["extra_npz_save_error"] = str(e) + + return metrics + + +def main(program_path: str, results_dir: str): + """Runs the circle packing evaluation using shinka.eval.""" + print(f"Evaluating program: {program_path}") + print(f"Saving results to: {results_dir}") + os.makedirs(results_dir, exist_ok=True) + + num_experiment_runs = 1 + + # Define a nested function to pass results_dir to the aggregator + def _aggregator_with_context( + r: List[Tuple[np.ndarray, np.ndarray, float]], + ) -> Dict[str, Any]: + return aggregate_circle_packing_metrics(r, results_dir) + + metrics, correct, error_msg = run_shinka_eval( + program_path=program_path, + results_dir=results_dir, + experiment_fn_name="run_packing", + num_runs=num_experiment_runs, + get_experiment_kwargs=get_circle_packing_kwargs, + validate_fn=adapted_validate_packing, + aggregate_metrics_fn=_aggregator_with_context, + ) + + if correct: + print("Evaluation and Validation completed successfully.") + else: + print(f"Evaluation or Validation failed: {error_msg}") + + print("Metrics:") + for key, value in metrics.items(): + if isinstance(value, str) and len(value) > 100: + print(f" {key}: ") + else: + print(f" {key}: {value}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Circle packing evaluator using shinka.eval" + ) + parser.add_argument( + "--program_path", + type=str, + default="initial.py", + help="Path to program to evaluate (must contain 'run_packing')", + ) + parser.add_argument( + "--results_dir", + type=str, + default="results", + help="Dir to save results (metrics.json, correct.json, extra.npz)", + ) + parsed_args = parser.parse_args() + main(parsed_args.program_path, parsed_args.results_dir) diff --git a/tasks/circle_packing/baseline/initial.py b/tasks/circle_packing/baseline/initial.py new file mode 100644 index 0000000..0b55e5b --- /dev/null +++ b/tasks/circle_packing/baseline/initial.py @@ -0,0 +1,100 @@ +# EVOLVE-BLOCK-START +"""Constructor-based circle packing for n=26 circles""" + +import numpy as np + + +def construct_packing(): + """ + Construct a specific arrangement of 26 circles in a unit square + that attempts to maximize the sum of their radii. + + Returns: + Tuple of (centers, radii, sum_of_radii) + centers: np.array of shape (26, 2) with (x, y) coordinates + radii: np.array of shape (26) with radius of each circle + sum_of_radii: Sum of all radii + """ + # Initialize arrays for 26 circles + n = 26 + centers = np.zeros((n, 2)) + + # Place circles in a structured pattern + # This is a simple pattern - evolution will improve this + + # First, place a large circle in the center + centers[0] = [0.5, 0.5] + + # Place 8 circles around it in a ring + for i in range(8): + angle = 2 * np.pi * i / 8 + centers[i + 1] = [0.5 + 0.3 * np.cos(angle), 0.5 + 0.3 * np.sin(angle)] + + # Place 16 more circles in an outer ring + for i in range(16): + angle = 2 * np.pi * i / 16 + centers[i + 9] = [0.5 + 0.7 * np.cos(angle), 0.5 + 0.7 * np.sin(angle)] + + # Additional positioning adjustment to make sure all circles + # are inside the square and don't overlap + # Clip to ensure everything is inside the unit square + centers = np.clip(centers, 0.01, 0.99) + + # Compute maximum valid radii for this configuration + radii = compute_max_radii(centers) + return centers, radii + + +def compute_max_radii(centers): + """ + Compute the maximum possible radii for each circle position + such that they don't overlap and stay within the unit square. + + Args: + centers: np.array of shape (n, 2) with (x, y) coordinates + + Returns: + np.array of shape (n) with radius of each circle + """ + n = centers.shape[0] + radii = np.ones(n) + + # First, limit by distance to square borders + for i in range(n): + x, y = centers[i] + # Distance to borders + radii[i] = min(x, y, 1 - x, 1 - y) + + for i in range(n): + for j in range(i + 1, n): + dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2)) + + # Then, limit by distance to other circles + # Each pair of circles with centers at distance d can have + # sum of radii at most d to avoid overlap + for i in range(n): + for j in range(i + 1, n): + dist = np.sqrt(np.sum((centers[i] - centers[j]) ** 2)) + + # If current radii would cause overlap + if radii[i] + radii[j] > dist: + # Scale both radii proportionally + scale = dist / (radii[i] + radii[j]) + radii[i] *= scale + radii[j] *= scale + + # Ensure no negative radii due to numerical issues + radii = np.maximum(radii, 0.0) + return radii + + +# EVOLVE-BLOCK-END + + +# This part remains fixed (not evolved) +def run_packing(): + """Run the circle packing constructor for n=26""" + centers, radii = construct_packing() + # Calculate the sum of radii + sum_radii = np.sum(radii) + return centers, radii, sum_radii diff --git a/tasks/circle_packing/check-candidate.py b/tasks/circle_packing/check-candidate.py new file mode 100644 index 0000000..4e6cfcc --- /dev/null +++ b/tasks/circle_packing/check-candidate.py @@ -0,0 +1,100 @@ +"""候选代码的轻量静态检查（不执行候选的任何一行代码）。 + +实验员需要一个便宜的办法确认代码能跑；不给他这个办法，他就会自己跑一遍完整实验 +去验证。这个脚本就是那个便宜的办法：只做三件事，全部基于抽象语法树。 + + 1. 入口文件在不在 + 2. 候选目录下每个 ``*.py`` 能不能 ``ast.parse`` + 3. 只对入口文件：顶层恰好有一个叫 ``run_packing`` 的函数，且零个必填参数 + +用法：``python -B check-candidate.py <候选目录> <入口文件相对候选目录的路径> <结果文件>``。 +结果写进 <结果文件>，一个 JSON 对象：``{"valid": bool, "reasons": [...]}``。 +对候选代码的每一种意见都走"退出码 0 加 reasons"；非零退出说明这个脚本自己没跑成。 + +两条调用约定，改的时候不要破坏： + + - **本文件不能放进候选目录。** 第 2 步从候选目录往下扫 ``*.py``， + 脚本自己在里面就会把自己也扫进去。宿主把它放在候选目录的上一层。 + - **只用标准库，``-B`` 要保留。** 不写 ``__pycache__``：检查本身不该往候选目录里添文件。 +""" + +import ast +import json +import pathlib +import sys + +ENTRY_FUNCTION = "run_packing" + +# 两种定义形式都要认，否则 `async def run_packing()` 会被判成"不存在"。 +FUNCTION_NODES = (ast.FunctionDef, ast.AsyncFunctionDef) + + +def has_zero_required_arguments(node) -> bool: + """判断函数是不是可以零参数调用。 + + 位置参数（含仅位置参数）里没有默认值的那些必须为 0 个； + 仅关键字参数里 kw_defaults 为 None 的表示没有默认值，一个都不能有。 + """ + positional = len(node.args.posonlyargs) + len(node.args.args) + required_positional = positional - len(node.args.defaults) + keyword_without_default = any( + default is None for default in node.args.kw_defaults + ) + return required_positional <= 0 and not keyword_without_default + + +def main() -> int: + if len(sys.argv) != 4: + print( + "用法: check-candidate.py <候选目录> <入口文件相对路径> <结果文件>", + file=sys.stderr, + ) + return 2 + + root = pathlib.Path(sys.argv[1]) + entry_point = sys.argv[2] + reasons: list[str] = [] + + if not (root / entry_point).is_file(): + reasons.append(f"入口文件不存在: {entry_point}") + + # 第 3 步挂在第 2 步的 rglob("*.py") 上：入口文件名不以 .py 结尾， + # 循环扫不到它，入口函数那条判定会整个被跳过，却照样报通过。 + if not entry_point.endswith(".py"): + reasons.append( + f"入口文件必须以 .py 结尾，否则入口函数检查扫不到它: {entry_point}" + ) + + for path in root.rglob("*.py"): + relative = str(path.relative_to(root)) + try: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=relative) + except (SyntaxError, ValueError) as error: + # SyntaxError：语法错；ValueError：含空字节或不是 UTF-8（UnicodeDecodeError 是它的子类）。 + reasons.append(f"{relative} 解析失败: {error}") + continue + if relative != entry_point: + continue + candidates = [ + node + for node in tree.body + if isinstance(node, FUNCTION_NODES) and node.name == ENTRY_FUNCTION + ] + if len(candidates) != 1: + reasons.append( + f"{entry_point} 顶层的 {ENTRY_FUNCTION}() 应当恰好有一个定义，" + f"实际有 {len(candidates)} 个" + ) + elif not has_zero_required_arguments(candidates[0]): + reasons.append( + f"{entry_point} 的 {ENTRY_FUNCTION}() 必须能零参数调用，" + f"当前有必填参数" + ) + + with open(sys.argv[3], "w", encoding="utf-8") as handle: + json.dump({"valid": not reasons, "reasons": reasons}, handle) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tasks/circle_packing/harness.py b/tasks/circle_packing/harness.py new file mode 100644 index 0000000..ff0718f --- /dev/null +++ b/tasks/circle_packing/harness.py @@ -0,0 +1,198 @@ +"""填圆任务的运行 harness（宿主的固定工具，候选不得修改）。 + +它按文件路径导入候选程序、调用候选定义的 ``run_packing()``、把原始解原子落盘。 +**它不判定解是否合法**：成绩只由独立校验器 ``validate.py`` 事后从实际产物重算。 + +指标的分工（docs/方法.md 第五部分"指标"）： + - 最终指标和合法性判定归 ``validate.py``，跟候选无关； + - 假设专用指标由研究者在协议里登记、实验员按定义在候选代码里实现并写出 + ``metrics.json``，宿主直接采信。harness 不内置任何指标算法。 + +命令行：python -B harness.py --program --output +种子从环境变量 INFORSI_SEED 读，必须给出。 +成功退出 0；任何失败都带 traceback 写 stderr 并非零退出。 + +工作目录里的三个文件（给实验员看的完整契约在同目录 ``TASK.md`` 的"指标怎么交给宿主"）： + - ``metrics-spec.json``：本次运行登记的假设专用指标定义，由宿主在准备运行目录时写好， + harness 不碰它。 + - ``solution.json``（``--output`` 指定）：候选返回的原始解。 + - ``metrics.json``：实验员的代码写出的假设专用指标取值，**键必须是 metrics-spec.json 里 + 那条指标的 ``id`` 原文**，值是有限数或 null，路径相对工作目录， + 且必须在 ``run_packing()`` 返回之前写完。没有这个文件就表示这一次的假设专用指标一个 + 都没测到。另有两个保留键（docs/方法.md 第五部分"答案路径"）：``answer-path``（字符串， + 最终交的答案出自哪条路径）和 ``safeguard-triggers``（对象，每条保障动手几次）。 + 它们不是指标、不是数，harness 原样保留，宿主读成辅助记录。 +""" + +import argparse +import hashlib +import importlib.util +import json +import math +import os +import random +import sys +from typing import Any, cast + +METRICS_FILE = "metrics.json" +# metrics.json 里不是数的两个保留键（docs/方法.md 第五部分"答案路径"）。finalize_metrics 只改写指标值，这两个原样留着： +# 改写成 null 的话，候选写对了的答案路径到宿主那里就成了"不认识的路径：null"，出错后无法察觉。 +RESERVED_KEYS = ("answer-path", "safeguard-triggers") + + +def load_program(program_path: str): + """按显式文件路径导入候选模块（不假定固定的 main.py 模式）。""" + spec = importlib.util.spec_from_file_location("candidate_program", program_path) + if spec is None or spec.loader is None: + raise RuntimeError(f"无法从 {program_path} 加载候选模块") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def write_json_atomically(path: str, payload) -> None: + """先写临时文件再 os.replace，让崩溃的运行永远不会留下半截产物。""" + temporary = path + ".tmp" + with open(temporary, "w", encoding="utf-8") as handle: + json.dump(payload, handle, allow_nan=False) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, path) + directory_fd = os.open( + os.path.dirname(os.path.abspath(path)), os.O_RDONLY | os.O_DIRECTORY + ) + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + + +def is_finite_number(value) -> bool: + """有限数的定义：不是布尔值的 int/float，且 math.isfinite。 + + 布尔值排除在外，否则 ``true`` 会被当成 1 记进证据。 + 超出浮点范围的大整数 math.isfinite 会抛 OverflowError，一并算作非有限。 + """ + if not isinstance(value, (int, float)) or isinstance(value, bool): + return False + try: + return math.isfinite(value) + except OverflowError: + return False + + +def finalize_metrics() -> None: + """把实验员写出的 metrics.json 整理成宿主读得进去的形状，然后原子落盘。 + + 只做一件事：指标的非有限值（NaN、±inf、布尔、非数值）**原样改写成 null**，其余照抄； + 两个保留键（RESERVED_KEYS）不是指标，原样保留。 + 这正是"测不到就说测不到"。被改写的编号在 stderr 记一行，**退出码保持 0**。 + + 为什么不抛错——这里抛错会让情况变坏，不是变好： + 1. 它防不住自述要防的那件事。实验员把测不到的量写成 0.0 照样放行。 + 2. 它实际拦的几种情况宿主已经逐条拦过，而且处理温和得多：那一条记 null + 加一句原因，成绩照常。 + 3. 抛错时 solution.json 已经完整落盘、解合法、成绩算得出来，但非零退出码 + 会让宿主判定这次运行没干净结束，最终指标也记成 null——一条坏指标白白 + 作废掉整次运行，而运行名额照扣。更糟的是那份裸 NaN 的 metrics.json 会 + 留在盘上，JS 的 JSON.parse 不接受裸 NaN，于是**这一轮所有候选指标一起 + 丢**，不只是坏的那一条。 + + 它防的是"模型正常工作时就会做"的那类失手：实验员算某条比值时碰上退化布局 + （分母是零半径）拿到 nan 没接住。改写成 null 之后，坏的那条变成 null 加原因， + 好的那些照常，成绩和运行名额都不受影响。 + + **不**校验键必须是 metrics-spec.json 里声明过的那些：宿主只按登记的编号取值， + 多出来的键不读。 + + 文件不存在表示候选指标一个都没测到，不补零也不补空文件。 + """ + if not os.path.exists(METRICS_FILE): + return + with open(METRICS_FILE, encoding="utf-8") as handle: + try: + measured = json.load(handle) + except ValueError as error: + # 同样不抛错，理由同上：整份读不出来是宿主那边看得见的一个状态 + # （它会记一句"metrics.json 读不出来"），不该连累已经算出来的成绩。 + print(f"{METRICS_FILE} 不是合法 JSON，原样留在盘上: {error}", file=sys.stderr) + return + if not isinstance(measured, dict): + print( + f"{METRICS_FILE} 必须是 JSON 对象（指标编号 -> 值），实际是 " + f"{type(measured).__name__}，原样留在盘上", + file=sys.stderr, + ) + return + + rewritten: dict = {} + replaced: list[str] = [] + for metric_id, value in measured.items(): + if metric_id in RESERVED_KEYS or value is None or is_finite_number(value): + rewritten[metric_id] = value + else: + rewritten[metric_id] = None + replaced.append(f"{metric_id}={value!r}") + if replaced: + print( + f"{METRICS_FILE} 里这些指标的值不是有限数，已按'测不到就写 null'改写成 " + f"null: {', '.join(replaced)}", + file=sys.stderr, + ) + write_json_atomically(METRICS_FILE, rewritten) + + +def main() -> int: + parser = argparse.ArgumentParser( + description="运行填圆候选程序并落盘它返回的原始解。" + ) + parser.add_argument( + "--program", + required=True, + help="定义了 run_packing() 的候选程序文件路径", + ) + parser.add_argument( + "--output", + required=True, + help="原始解 JSON 的写出路径（centers、radii、reported_sum）", + ) + args = parser.parse_args() + + # 运行目录是候选目录整个复制来的，里面可能留着以前某次运行写的这两个输出文件。 + # 候选这次没写时，宿主会把旧值当成这次的产物读走，所以先删掉，让读到的只可能是这次写的。 + for leftover in (args.output, METRICS_FILE): + if os.path.exists(leftover): + os.remove(leftover) + + # 在导入候选模块之前设种子。宿主给的是一个种子字符串（同一份研究的全部运行共用， + # docs/方法.md 第十部分要求评测确定），取它的 sha256 前 4 字节大端整数。 + # 没给就报 KeyError 停下，不替它补一个。 + np = importlib.import_module("numpy") + + seed = int.from_bytes( + hashlib.sha256(os.environ["INFORSI_SEED"].encode()).digest()[:4], + "big", + ) + random.seed(seed) + np.random.seed(seed) + + module = load_program(args.program) + run_packing = getattr(module, "run_packing", None) + if not callable(run_packing): + raise RuntimeError(f"{args.program} 没有定义可调用的 run_packing()") + + # 候选模块是动态加载的，下面的转换就是协议本身的强制点。 + centers, radii, reported_sum = cast(Any, run_packing()) + solution = { + "centers": [[float(x), float(y)] for x, y in centers], + "radii": [float(r) for r in radii], + "reported_sum": float(reported_sum), + } + write_json_atomically(args.output, solution) + + finalize_metrics() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tasks/circle_packing/requirements.txt b/tasks/circle_packing/requirements.txt new file mode 100644 index 0000000..3afb371 --- /dev/null +++ b/tasks/circle_packing/requirements.txt @@ -0,0 +1,7 @@ +numpy==2.3.1 # 数组与矩阵运算 +scipy==1.18.1 # 数值优化、线性规划、空间距离计算等 +numba==0.67.0 # 把 Python 循环编译成机器码 +llvmlite==0.49.0 # numba 的编译后端 +sympy==1.14.0 # 符号与精确计算 +mpmath==1.3.0 # 任意精度浮点运算（sympy 依赖它） +networkx==3.7 # 图结构分析 diff --git a/tasks/circle_packing/task.json b/tasks/circle_packing/task.json new file mode 100644 index 0000000..bee4aad --- /dev/null +++ b/tasks/circle_packing/task.json @@ -0,0 +1,8 @@ +{ + "kind": "circle_packing", + "finalMetric": { "id": "sum-of-radii", "direction": "maximize" }, + "epsilon": 0.01, + "knownBest": 2.635983, + "run": { "wallSeconds": 120, "cores": 1, "runtime": "runc" }, + "protectedFiles": ["harness.py", "validate.py", "check-candidate.py", "metrics-spec.json", "solution.json", "evaluate.py", "LICENSE", "SOURCES.md"] +} diff --git a/tasks/circle_packing/validate.py b/tasks/circle_packing/validate.py new file mode 100644 index 0000000..9e82400 --- /dev/null +++ b/tasks/circle_packing/validate.py @@ -0,0 +1,353 @@ +"""填圆的独立校验器（宿主保管，候选不得修改）。 + +从实际产物重算一切。不导入官方 ShinkaEvolve 的 evaluate.py，也不读候选自报的分数。 + +成绩的定义：成绩就是下面那一句 ``math.fsum(radius_values)``。 +只有这一处算它，所以不存在"两个成绩哪个算数"的问题。 + +**产物里的 ``reported_sum`` 这个字段照旧允许候选写**（``harness.py`` 就在写它）， +校验器只是**根本不读它**：它不进判定、不进 diagnostics、不影响 ``valid``。 + +检查项（n = --circles，单位正方形 [0,1]^2）： + - centers 形状 (n, 2)，radii 形状 (n,) + - 全是有限数 + - 半径不为负 + - 每个圆在正方形内（容差 atol） + - 两两不重叠（容差 atol） + +圆数、容差都没有默认值：忘了传就静默按旧值判，正是"改一处忘一处、没有任何东西会发现" +的那一类问题。所以 ``--circles``、``--atol`` 必填。 + +命令行：python -B validate.py --circles --solution --atol <容差> + [--legalize] --verdict +判定写进 ``--verdict`` 指定的文件（一个 JSON 对象），宿主读这个文件，不读标准输出。 +退出码：0 = 合法，1 = 不合法，2 = 用法或读取出错（这时不写判定文件）。 +校验器自己出错（例如算术异常）不兜住，带 traceback 非零退出、不写判定文件： +那是宿主的工具坏了，不能当成候选的失败。 +""" + +import argparse +import json +import math +import sys + +# 半径分布那一行最多列几段；超了就中间省略，省略号让"截断了"这件事看得见。 +PROFILE_MAX_GROUPS = 8 + + +def _is_finite_number(value) -> bool: + if not isinstance(value, (int, float)) or isinstance(value, bool): + return False + try: + return math.isfinite(value) + except OverflowError: + # JSON integers can exceed the range of the float geometry protocol. + return False + + +def render_radius_profile(radius_values) -> str: + """把半径分布写成一行文本（判定里的 ``evidence.radius_profile``）。 + + 降序排列，**数值完全相同**的连成一段写作 ``值 x个数``；只是四舍五入到小数点后 + 四位之后看着一样、实际不同的两个数仍然各占一段（所以会出现 + ``0.1580, 0.1580`` 这种相邻重复，它说明这是两个不同的数）。 + + 例：``3 radii, desc: 0.2500 x2, 0.0000`` + ``26 radii, desc: 0.5000, 0.0000 x25`` + """ + groups: list[list] = [] + for value in sorted(radius_values, reverse=True): + if groups and groups[-1][0] == value: + groups[-1][1] += 1 + else: + groups.append([value, 1]) + rendered = [ + f"{value:.4f}" if count == 1 else f"{value:.4f} x{count}" + for value, count in groups + ] + if len(rendered) > PROFILE_MAX_GROUPS: + rendered = rendered[: PROFILE_MAX_GROUPS - 2] + ["...", rendered[-1]] + return f"{len(radius_values)} radii, desc: " + ", ".join(rendered) + + +def validate_packing(centers, radii, circles, atol) -> dict: + """Validate a packing and recompute its score independently. + + ``circles`` 是任务声明的圆数（调用方必须给，没有默认值）。 + + Returns a structured verdict dict. All applicable reasons are collected + (not just the first failure) so diagnostics stay informative. + """ + reasons: list[str] = [] + diagnostics: dict = { + "declared_circles": circles, + "atol": atol, + "num_circles": None, + "max_boundary_violation": None, + "max_overlap_violation": None, + "min_pairwise_clearance": None, + "min_radius": None, + "max_radius": None, + "zero_radius_count": None, + "all_radii_finite": None, + } + evidence: dict = {"radius_profile": None} + computed_sum = None + + def verdict() -> dict: + return { + "valid": len(reasons) == 0, + "computed_sum": computed_sum, + "reasons": reasons, + "diagnostics": diagnostics, + "evidence": evidence, + } + + # 候选实际返回了几个圆：只要 centers 是个列表，这个数就是事实，先记下来。 + # 数量不对时它正好说明错在哪里，记成 null 就看不出来了。 + if isinstance(centers, list): + diagnostics["num_circles"] = len(centers) + + # Shape and element checks. + shape_ok = ( + isinstance(centers, list) + and len(centers) == circles + and all(isinstance(pair, list) and len(pair) == 2 for pair in centers) + and isinstance(radii, list) + and len(radii) == circles + ) + if not shape_ok: + reasons.append( + f"shape incorrect: expected centers ({circles}, 2) and radii " + f"({circles},), got centers " + f"{(len(centers), [len(p) if isinstance(p, list) else type(p).__name__ for p in centers[:3]]) if isinstance(centers, list) else type(centers).__name__} " + f"radii {len(radii) if isinstance(radii, list) else type(radii).__name__}" + ) + return verdict() + + # 半径那几个诊断量只跟半径有关，圆心坏了不影响它们，所以先算。 + radii_finite = all(_is_finite_number(r) for r in radii) + diagnostics["all_radii_finite"] = radii_finite + radius_values: list[float] | None = None + if radii_finite: + radius_values = [float(r) for r in radii] + diagnostics["min_radius"] = min(radius_values) + diagnostics["max_radius"] = max(radius_values) + diagnostics["zero_radius_count"] = sum(1 for r in radius_values if r == 0.0) + evidence["radius_profile"] = render_radius_profile(radius_values) + + centers_finite = all(_is_finite_number(v) for pair in centers for v in pair) + if not (centers_finite and radii_finite): + reasons.append("non-finite values found in centers or radii") + # Cannot trust geometry with non-finite or non-numeric values; stop here. + return verdict() + + # 走到这里 radii_finite 必然为真，radius_values 已经在上面算好。 + points = [(float(pair[0]), float(pair[1])) for pair in centers] + negative = [i for i, r in enumerate(radius_values) if r < 0] + if negative: + reasons.append(f"negative radii at indices: {negative}") + # Negative radii make geometry checks meaningless; stop here. + return verdict() + + # Independent recomputation of the score. + computed_sum = math.fsum(radius_values) + + # Boundary check: circle i must lie inside [0,1]^2 within atol. + max_boundary = 0.0 + boundary_indices = [] + for i, ((x, y), r) in enumerate(zip(points, radius_values, strict=True)): + violation = max(r - x, x + r - 1.0, r - y, y + r - 1.0) + max_boundary = max(max_boundary, violation) + if violation > atol: + boundary_indices.append(i) + diagnostics["max_boundary_violation"] = max_boundary + if boundary_indices: + reasons.append( + f"circles outside unit square at indices: {boundary_indices} " + f"(max violation {max_boundary!r}, atol={atol})" + ) + + # Pairwise overlap check. + max_overlap = 0.0 + min_clearance = math.inf + overlap_pairs = [] + for i in range(circles): + xi, yi = points[i] + for j in range(i + 1, circles): + xj, yj = points[j] + dist = math.hypot(xi - xj, yi - yj) + clearance = dist - (radius_values[i] + radius_values[j]) + min_clearance = min(min_clearance, clearance) + violation = -clearance + max_overlap = max(max_overlap, violation) + if violation > atol: + overlap_pairs.append((i, j)) + diagnostics["max_overlap_violation"] = max_overlap + diagnostics["min_pairwise_clearance"] = ( + min_clearance if math.isfinite(min_clearance) else None + ) + if overlap_pairs: + reasons.append( + f"overlapping circle pairs: {overlap_pairs[:5]}" + f"{' ...' if len(overlap_pairs) > 5 else ''} " + f"(max violation {max_overlap!r}, atol={atol})" + ) + + return verdict() + + +def legalize_packing(centers, radii, circles, atol) -> dict: + """只缩不移合法化（docs/方法.md 第八部分"合法化"，规则原文写在同目录 TASK.md）。 + + 1. 先按 ``atol`` 判原产物。合法就原样返回，扣分 0。 + 2. 不合法时只缩半径、不挪圆心：**只缩参与违反的圆**——每个圆的半径减去它参与的 + 最大违反量（跟别的圆重叠的量、出边的量取最大）。 + 3. 缩完仍不合法（浮点舍入留下的零头），把这些圆的半径各往 0 挪一个最小浮点单位， + 再判，直到合法。 + 4. 修不了：形状不对、含非有限值或负半径（这些不是"差一点"，缩半径解决不了）； + 有圆心在正方形外面；这些圆的半径都缩到 0 仍不合法。 + + 返回值顶层是**最后被判的那份摆法**的裁决（修好了就是修后的，修不了就是原来的）， + 外加 ``legalization`` 一节：原产物合不合法、原分数、原违反量、缩了哪些圆、修后半径。 + 扣掉多少分由宿主拿原分数减修后分数算，这里不另报一个数。 + """ + raw = validate_packing(centers, radii, circles, atol=atol) + diagnostics = raw["diagnostics"] + violations = [ + v + for v in ( + diagnostics.get("max_overlap_violation"), + diagnostics.get("max_boundary_violation"), + ) + if v is not None + ] + record: dict = { + "raw_valid": raw["valid"], + "raw_score": raw["computed_sum"], + # 形状、有限性、负半径那几步提前返回时几何检查没做，原违反量量不到，记 null。 + "raw_violation": max(violations) if len(violations) == 2 else None, + "applied": False, + "repaired": raw["valid"], + "shrunk_circles": [], + "legal_radii": None, + "reasons": [], + } + + def finish(verdict: dict) -> dict: + return {**verdict, "legalization": record} + + if raw["valid"]: + return finish(raw) + if raw["computed_sum"] is None: + record["reasons"].append( + "形状不对、含非有限值或有负半径：这不是差一点的违规，缩半径修不了" + ) + return finish(raw) + + points = [(float(x), float(y)) for x, y in centers] + outside = [i for i, (x, y) in enumerate(points) if not (0.0 <= x <= 1.0 and 0.0 <= y <= 1.0)] + if outside: + record["reasons"].append( + f"圆心在正方形外面（编号 {outside}）：只缩半径不挪圆心，修不了" + ) + return finish(raw) + + radius_values = [float(r) for r in radii] + # 每个圆参与的最大违反量。只有超过 atol 的才算违反，减的是违反量本身。 + shrink = [0.0] * circles + for i, ((x, y), r) in enumerate(zip(points, radius_values, strict=True)): + boundary = max(r - x, x + r - 1.0, r - y, y + r - 1.0) + if boundary > atol: + shrink[i] = boundary + for i in range(circles): + xi, yi = points[i] + for j in range(i + 1, circles): + xj, yj = points[j] + overlap = radius_values[i] + radius_values[j] - math.hypot(xi - xj, yi - yj) + if overlap > atol: + shrink[i] = max(shrink[i], overlap) + shrink[j] = max(shrink[j], overlap) + involved = [i for i in range(circles) if shrink[i] > 0.0] + record["applied"] = True + record["shrunk_circles"] = involved + + legal = [max(r - s, 0.0) for r, s in zip(radius_values, shrink, strict=True)] + while True: + verdict = validate_packing(centers, legal, circles, atol=atol) + if verdict["valid"]: + record["repaired"] = True + record["legal_radii"] = legal + return finish(verdict) + nudged = [ + math.nextafter(legal[i], 0.0) if shrink[i] > 0.0 else legal[i] + for i in range(circles) + ] + if nudged == legal: + record["reasons"].append( + f"参与违反的圆（编号 {involved}）半径都缩到 0 仍不合法，修不了" + ) + return finish(raw) + legal = nudged + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Independent circle-packing validator (unit square)." + ) + parser.add_argument("--solution", required=True, help="solution JSON path") + parser.add_argument( + "--circles", + type=int, + required=True, + help="how many circles the task declares (positive integer)", + ) + parser.add_argument( + "--atol", + type=float, + required=True, + help="absolute tolerance for boundary/overlap checks", + ) + parser.add_argument( + "--legalize", + action="store_true", + help=( + "原产物不合法时按只缩不移规则修（见 legalize_packing），" + "输出多一节 legalization；不加时只判不修" + ), + ) + parser.add_argument( + "--verdict", required=True, help="判定写进这个文件（JSON 对象）" + ) + args = parser.parse_args() + if args.circles < 1: + print("--circles must be a positive integer", file=sys.stderr) + return 2 + if not (math.isfinite(args.atol) and args.atol >= 0): + print("--atol must be finite and non-negative", file=sys.stderr) + return 2 + + try: + with open(args.solution, encoding="utf-8") as handle: + solution = json.load(handle) + except (OSError, ValueError, RecursionError) as error: + print(f"cannot read solution: {error}", file=sys.stderr) + return 2 + if not isinstance(solution, dict): + print("solution must be a JSON object", file=sys.stderr) + return 2 + + judge = legalize_packing if args.legalize else validate_packing + verdict = judge( + solution.get("centers"), + solution.get("radii"), + args.circles, + atol=args.atol, + ) + with open(args.verdict, "w", encoding="utf-8") as handle: + json.dump(verdict, handle, allow_nan=False) + return 0 if verdict["valid"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/fixtures/legalization/14-real-1-keep-grid-start.json b/tests/fixtures/legalization/14-real-1-keep-grid-start.json new file mode 100644 index 0000000..99a96b6 --- /dev/null +++ b/tests/fixtures/legalization/14-real-1-keep-grid-start.json @@ -0,0 +1 @@ +{"centers": [[0.100484056446833, 0.07485776462601139], [0.10728235286470139, 0.2738532442085582], [0.086108914434533, 0.5223893812779171], [0.10580193867928778, 0.6806936405144427], [0.0856033942977713, 0.9222111281543698], [0.3120328263981882, 0.10480425665863033], [0.32716502959347243, 0.3067558057750085], [0.2897009711447359, 0.47507350932200926], [0.2935854248884542, 0.7104275384094612], [0.27237639359908744, 0.8951262487554578], [0.4914744630822173, 0.08989291879330835], [0.5100326585018874, 0.29884106286561407], [0.5213978072978597, 0.4946336955389669], [0.5151435556465788, 0.7026215638204316], [0.49451516606288104, 0.901030906906145], [0.721963590658637, 0.11913688062063861], [0.7296948042290226, 0.2744414856202059], [0.7002013752356513, 0.5129578551833195], [0.7069421474478524, 0.7067135961639011], [0.7178150742617755, 0.9143740466217667], [0.8917011179459332, 0.11647248318508979], [0.9275227999594169, 0.2848229492760654], [0.9028755468513529, 0.5143032303369243], [0.9178606846823245, 0.681413920131626], [0.9136434762753853, 0.8933655911924985], [0.18562920767437355, 0.7769807813776922]], "radii": [0.06361218681134884, 0.0441172439828679, 0.03913703570214593, 0.027074375269719862, 0.07778887184563021, 0.05730567971921061, 0.029286768433871176, 0.058739899233888004, 0.050016382123936116, 0.07485971141413426, 0.07105090920173604, 0.04742662315759923, 0.039360492244268, 0.05993669686613301, 0.0893218390023967, 0.06428677645874871, 0.08948690383353296, 0.09702962465194793, 0.09684333677627958, 0.08562595337823331, 0.1020164679124565, 0.07010288699954595, 0.09090324918001258, 0.07687796847999327, 0.08635652372461466, 0.07171254287625847], "reported_sum": 1.76027694928051} \ No newline at end of file diff --git a/tests/runner.test.ts b/tests/runner.test.ts new file mode 100644 index 0000000..d14c42b --- /dev/null +++ b/tests/runner.test.ts @@ -0,0 +1,224 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { after, test, type TestContext } from "node:test"; +import { Runner } from "../src/runner/index.ts"; +import type { ProcessResult, RunDefinition } from "../src/vocab/index.ts"; + +// 用真实 Docker 测（任务书 05 第五部分）：没有 Docker 时整个文件失败，不跳过。 +// 镜像、容器名一律 inforsi-test- 开头的固定名字；本机还有别的项目在建删容器，只动自己前缀的。 + +const TASK_DIR = join(import.meta.dirname, "..", "tasks", "circle_packing"); +const IMAGE_TAG = "inforsi-test-circle"; +const MISSING_DIGEST = `sha256:${"0".repeat(64)}`; + +const runner = new Runner(null); +const digest = await runner.buildImage(TASK_DIR, IMAGE_TAG); +after(() => docker(["rmi", IMAGE_TAG])); + +function docker(args: string[]): { status: number | null; stdout: string; stderr: string } { + return spawnSync("docker", args, { encoding: "utf8" }); +} + +/** 删掉测试自己的容器；不存在不算错，别的错误照样报。 */ +function forceRemove(name: string): void { + const removed = docker(["rm", "-f", name]); + if (removed.status !== 0 && !removed.stderr.includes("No such container")) throw new Error(removed.stderr); +} + +/** 一个测试用的容器名：先清掉上次没删干净的同名容器，测试结束后再删。 */ +function containerName(t: TestContext, suffix: string): string { + const name = `inforsi-test-${suffix}`; + forceRemove(name); + t.after(() => forceRemove(name)); + return name; +} + +function dirs(t: TestContext): { runDir: string; logDir: string } { + const root = mkdtempSync(join(tmpdir(), "inforsi-runner-")); + t.after(() => rmSync(root, { recursive: true, force: true })); + const runDir = join(root, "run"); + const logDir = join(root, "log"); + mkdirSync(runDir); + mkdirSync(logDir); + return { runDir, logDir }; +} + +function definition(command: string[], wallSeconds: number, image: string): RunDefinition { + return { + command, + workdir: "/work", + image: { digest: image, runtime: "runc" }, + wallSeconds, + cores: 1, + inheritEnv: false, + env: {}, + }; +} + +const python = (code: string): string[] => ["python3", "-c", code]; + +/** 起一个容器并等它结束，交回结果和日志目录。 */ +async function runToEnd(t: TestContext, suffix: string, command: string[], wallSeconds: number) { + const name = containerName(t, suffix); + const { runDir, logDir } = dirs(t); + const outcome = await runner.start(name, definition(command, wallSeconds, digest), runDir); + assert.deepEqual(outcome, { started: true }); + const result = await runner.wait(name, logDir); + return { name, result, logDir }; +} + +function value(result: ProcessResult, key: "cpuSeconds" | "wallSeconds" | "peakMemoryBytes"): number { + assert.notEqual(result.ending, "notStarted"); + if (result.ending === "notStarted") throw new Error("unreachable"); + const item = result[key]; + assert.equal(item.status, "measured", `${key} 应当测到`); + if (item.status !== "measured") throw new Error("unreachable"); + return item.value; +} + +test("建镜像命中参考项目的缓存；查镜像：有的答有，没有的答没有", async () => { + assert.match(digest, /^sha256:7ed52793365c/); + assert.equal(await runner.hasImage(digest), true); + assert.equal(await runner.hasImage(MISSING_DIGEST), false); +}); + +test("正常退出：带退出码和三项资源用量，标准输出、标准错误存成两个文件", async (t) => { + const code = "import sys; print('out'); print('err', file=sys.stderr); sys.exit(3)"; + const { result, logDir } = await runToEnd(t, "exit", python(code), 30); + assert.equal(result.ending, "exited"); + assert.equal(result.ending === "exited" && result.exitCode, 3); + assert.ok(value(result, "cpuSeconds") > 0); + assert.ok(value(result, "wallSeconds") > 0 && value(result, "wallSeconds") < 30); + assert.ok(value(result, "peakMemoryBytes") > 0); + assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "out\n"); + assert.equal(readFileSync(join(logDir, "stderr.log"), "utf8"), "err\n"); +}); + +test("超时：结局 timeout，CPU 秒接近时限，墙钟不短于时限", async (t) => { + const { result } = await runToEnd(t, "timeout", python("while True: pass"), 2); + assert.equal(result.ending, "timeout"); + const cpu = value(result, "cpuSeconds"); + assert.ok(cpu > 1.7 && cpu < 2.3, `CPU 秒 ${cpu}`); + assert.ok(value(result, "wallSeconds") >= 2); +}); + +test("候选伪造计量结果再死循环：伪造不成，结局仍是 timeout，资源用量是真实测到的；写出的文件归当前用户", async (t) => { + // 审阅时实测过的手法：先写一份假的 usage，再把目录改成只读，然后死循环到超时。 + const code = [ + "import os", + "for attempt in (lambda: open('/inforsi-record/usage', 'w').write('exited 0 1 1 1\\n'), lambda: os.chmod('/inforsi-record', 0o555)):", + " try:", + " attempt()", + " except OSError as error:", + " print(type(error).__name__, flush=True)", + "open('out.txt', 'w').write('x')", + "while True: pass", + ].join("\n"); + const name = containerName(t, "forge"); + const { runDir, logDir } = dirs(t); + assert.deepEqual(await runner.start(name, definition(python(code), 2, digest), runDir), { started: true }); + const result = await runner.wait(name, logDir); + assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "PermissionError\nPermissionError\n"); + assert.equal(result.ending, "timeout"); + const cpu = value(result, "cpuSeconds"); + t.diagnostic(`结局 ${result.ending}，CPU 秒 ${cpu}，墙钟秒 ${value(result, "wallSeconds")}`); + assert.ok(cpu > 1.7 && cpu < 2.3, `CPU 秒 ${cpu}`); + assert.ok(value(result, "wallSeconds") >= 2); + assert.equal(statSync(join(runDir, "out.txt")).uid, process.getuid?.()); +}); + +test("命令自己以 124 或 137 退出，不判成超时", async (t) => { + const own124 = await runToEnd(t, "exit124", python("import sys; sys.exit(124)"), 30); + assert.equal(own124.result.ending, "exited"); + assert.equal(own124.result.ending === "exited" && own124.result.exitCode, 124); + const own137 = await runToEnd(t, "exit137", python("import os; os.kill(os.getpid(), 9)"), 30); + assert.equal(own137.result.ending, "exited"); + assert.equal(own137.result.ending === "exited" && own137.result.exitCode, 137); +}); + +test("从外面 docker kill：结局 cancelled，三项资源都测不到并写明原因", async (t) => { + const name = containerName(t, "killed"); + const { runDir, logDir } = dirs(t); + const outcome = await runner.start(name, definition(python("import time; time.sleep(60)"), 120, digest), runDir); + assert.deepEqual(outcome, { started: true }); + assert.equal(await runner.status(name), "running"); + const killed = docker(["kill", name]); + assert.equal(killed.status, 0, killed.stderr); + const result = await runner.wait(name, logDir); + assert.equal(result.ending, "cancelled"); + if (result.ending !== "cancelled") return; + for (const item of [result.cpuSeconds, result.wallSeconds, result.peakMemoryBytes]) { + assert.equal(item.status, "unmeasured"); + assert.match(item.status === "unmeasured" ? item.reason : "", /从外面被结束/); + } +}); + +test("镜像编号不存在：没启动，原因是 Docker 的报错原文，也没留下容器", async (t) => { + const name = containerName(t, "noimage"); + const { runDir } = dirs(t); + const outcome = await runner.start(name, definition(python("pass"), 30, MISSING_DIGEST), runDir); + assert.equal(outcome.started, false); + assert.match(outcome.started ? "" : outcome.reason, /No such image/); + assert.equal(await runner.status(name), "absent"); +}); + +test("断网：容器里连不上外网", async (t) => { + const code = [ + "import socket, sys", + "try:", + " socket.create_connection(('1.1.1.1', 53), timeout=3)", + "except OSError as error:", + " print(type(error).__name__); sys.exit(0)", + "sys.exit(1)", + ].join("\n"); + const { result } = await runToEnd(t, "network", python(code), 30); + assert.equal(result.ending === "exited" && result.exitCode, 0); +}); + +test("核数：容器里 os.cpu_count() 报 1", async (t) => { + const { logDir } = await runToEnd(t, "cores", python("import os; print(os.cpu_count())"), 30); + assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "1\n"); +}); + +test("查状态：没有这个容器、建了没启动、在跑、已结束、问不到", async (t) => { + assert.equal(await runner.status(containerName(t, "status-absent")), "absent"); + + const created = containerName(t, "status-created"); + const create = docker(["create", "--name", created, digest, "true"]); + assert.equal(create.status, 0, create.stderr); + assert.equal(await runner.status(created), "created"); + + const name = containerName(t, "status-running"); + const { runDir, logDir } = dirs(t); + await runner.start(name, definition(python("import time; time.sleep(3)"), 30, digest), runDir); + assert.equal(await runner.status(name), "running"); + await runner.wait(name, logDir); + assert.equal(await runner.status(name), "exited"); + + const nowhere = new Runner("unix:///nonexistent/docker.sock"); + assert.equal(await nowhere.status(name), "unreachable"); + + await runner.remove(name); + assert.equal(await runner.status(name), "absent"); +}); + +test("等结束能接上另一个 Runner 实例启动的容器；已结束的容器再等一次，结果相同", async (t) => { + const name = containerName(t, "resume"); + const { runDir, logDir } = dirs(t); + const starter = new Runner(null); + await starter.start(name, definition(python("import time; time.sleep(2); print('done')"), 30, digest), runDir); + const resumed = new Runner(null); + const first = await resumed.wait(name, logDir); + assert.equal(first.ending === "exited" && first.exitCode, 0); + assert.equal(readFileSync(join(logDir, "stdout.log"), "utf8"), "done\n"); + const again = await new Runner(null).wait(name, logDir); + assert.deepEqual(again, first); +}); + +test("容器名不符合 Docker 的要求时报错", async () => { + await assert.rejects(runner.status("inforsi test"), /不符合 Docker 的要求/); + await assert.rejects(runner.status("-x"), /不符合 Docker 的要求/); +}); diff --git a/tests/tasks.test.ts b/tests/tasks.test.ts new file mode 100644 index 0000000..c294709 --- /dev/null +++ b/tests/tasks.test.ts @@ -0,0 +1,460 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { cpSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { after, test, type TestContext } from "node:test"; +import { Runner } from "../src/runner/index.ts"; +import { + checkTaskFacts, + formalRun, + judge, + lightCheck, + loadTask, + metricDefinitions, + prepareJudging, + prepareRun, + protectedChanges, + readLightCheck, +} from "../src/tasks/index.ts"; +import type { HypothesisMetric, JudgedResult, Measured, ProcessResult, RunDefinition } from "../src/vocab/index.ts"; + +// tasks 的测试（任务书 05 第五部分）。凡是要执行任务代码的，都经真实 Docker 在任务镜像里跑： +// 没有 Docker 时整个文件失败，不跳过。镜像和容器一律用 inforsi-test-tasks- 开头的固定名字， +// 与 runner 的测试（inforsi-test-circle、inforsi-test-<用途>）错开，两个文件并行跑也不互相删。 +// 判分的预期值是手算的 2 的幂有理数布局（借自参考项目 tests/circle-validator.test.ts）， +// 加一份参考项目真实运行里差一点不合法的产物（tests/fixtures/legalization/）。 + +const TASK_DIR = join(import.meta.dirname, "..", "tasks", "circle_packing"); +const IMAGE_TAG = "inforsi-test-tasks"; +const FIXTURE = join(import.meta.dirname, "fixtures", "legalization", "14-real-1-keep-grid-start.json"); + +const task = loadTask(TASK_DIR); +const runner = new Runner(null); +const digest = await runner.buildImage(task.dir, IMAGE_TAG); +after(() => docker(["rmi", IMAGE_TAG])); + +function docker(args: string[]): { status: number | null; stderr: string } { + return spawnSync("docker", args, { encoding: "utf8" }); +} + +function forceRemove(name: string): void { + const removed = docker(["rm", "-f", name]); + if (removed.status !== 0 && !removed.stderr.includes("No such container")) throw new Error(removed.stderr); +} + +function tempDir(t: TestContext, label: string): string { + const dir = mkdtempSync(join(tmpdir(), `inforsi-tasks-${label}-`)); + t.after(() => rmSync(dir, { recursive: true, force: true })); + return dir; +} + +/** 让 runner 在任务镜像里跑一份运行定义，交回结果。 */ +async function run(t: TestContext, suffix: string, definition: RunDefinition, dir: string): Promise { + const name = `inforsi-test-tasks-${suffix}`; + forceRemove(name); + t.after(() => forceRemove(name)); + const logDir = tempDir(t, "log"); + const outcome = await runner.start(name, definition, dir); + assert.deepEqual(outcome, { started: true }); + return runner.wait(name, logDir); +} + +const m = (value: number): Measured => ({ status: "measured", value }); + +/** 一次正常结束的正式运行；资源用量是随手写的，判分只是原样转记。 */ +const EXITED_OK: ProcessResult = { ending: "exited", exitCode: 0, cpuSeconds: m(1.5), wallSeconds: m(2.5), peakMemoryBytes: m(1024) }; + +// --------------------------------------------------------------------------- +// 手算布局（2 的幂的有理数，双精度里精确） +// --------------------------------------------------------------------------- + +interface Solution { + centers: number[][]; + radii: number[]; + reported_sum: number; +} + +/** 一行等间距圆：圆心 (firstX + i·spacing, 0.5)，半径统一。 */ +function rowLayout(radius: number, spacing: number, firstX: number): Solution { + const centers: number[][] = []; + const radii: number[] = []; + for (let i = 0; i < 26; i += 1) { + centers.push([firstX + i * spacing, 0.5]); + radii.push(radius); + } + return { centers, radii, reported_sum: 99 }; +} + +// 宽松布局：半径 1/128，间距 1/32，首个圆心 7/64。相邻间隙 1/32 − 1/64 = 1/64 > 0； +// 最右 7/64 + 25/32 + 1/128 = 0.8984375 < 1。半径和 26/128 = 0.203125。 +const loose = (): Solution => rowLayout(1 / 128, 1 / 32, 7 / 64); +// 相切布局：半径 1/64，间距 1/32 恰好等于两半径之和，间隙恰好 0。半径和 26/64 = 0.40625。 +const tangent = (): Solution => rowLayout(1 / 64, 1 / 32, 7 / 64); + +/** 判分走完两步：有产物就在镜像里跑校验器。runDir 里放好 solution.json（或不放）。 */ +async function judgeRun( + t: TestContext, + suffix: string, + runDir: string, + formal: ProcessResult, + metrics: HypothesisMetric[], +): Promise { + const judgeDir = tempDir(t, "judge"); + const step = prepareJudging(task, runDir, judgeDir, digest, formal, metrics); + if (step.status === "judged") return step.result; + const validator = await run(t, suffix, step.definition, judgeDir); + return judge(task, runDir, judgeDir, formal, validator, metrics); +} + +function runDirWith(t: TestContext, solution: Solution | string | null): string { + const dir = tempDir(t, "run"); + if (solution !== null) { + writeFileSync(join(dir, "solution.json"), typeof solution === "string" ? solution : JSON.stringify(solution)); + } + return dir; +} + +function value(item: Measured | undefined): number { + assert.equal(item?.status, "measured", JSON.stringify(item)); + return (item as { value: number }).value; +} + +function reason(item: Measured | undefined): string { + assert.equal(item?.status, "unmeasured", JSON.stringify(item)); + return (item as { reason: string }).reason; +} + +// --------------------------------------------------------------------------- +// 任务目录与 task.json +// --------------------------------------------------------------------------- + +test("基线三个文件的 sha256 等于 SOURCES.md 表上的值", () => { + // 期望值从 baseline/SOURCES.md 的表格手抄，不调项目里的哈希函数。 + const expected: Array<[string, string]> = [ + ["initial.py", "09da8428a85792a94a9c6cc2193b6a0db1090ce598a17b75bc0db48d0953d098"], + ["evaluate.py", "eb8b9787dc25dee24a5f5305d1e3bd876e5c336cded1c9bab7fb204612928b4d"], + ["LICENSE", "c3e5b1c26607caf7208d5a105c113fe240630bb873fdb8ca5b4c2e32c721ddaf"], + ]; + for (const [name, sha] of expected) { + assert.equal(createHash("sha256").update(readFileSync(join(task.baselineDir, name))).digest("hex"), sha, name); + } +}); + +test("读任务：task.json 的值、Dockerfile 所在目录、TASK.md 全文；最终指标和辅助指标的定义", () => { + assert.deepEqual(task.facts, JSON.parse(readFileSync(join(TASK_DIR, "task.json"), "utf8"))); + assert.equal(task.facts.finalMetric.id, "sum-of-radii"); + assert.equal(task.facts.epsilon, 0.01); + assert.equal(task.facts.knownBest, 2.635983); + assert.deepEqual(task.facts.run, { wallSeconds: 120, cores: 1, runtime: "runc" }); + assert.ok(existsSync(join(task.dir, "Dockerfile"))); + assert.match(task.guide, /^# 填圆任务/); + const ids = metricDefinitions(task).map((d) => `${d.layer}:${d.id}`); + assert.deepEqual(ids, [ + "final:sum-of-radii", + "auxiliary:wall-seconds", + "auxiliary:cpu-seconds", + "auxiliary:peak-memory-bytes", + "auxiliary:min-pairwise-clearance", + "auxiliary:raw-score", + "auxiliary:raw-violation", + "auxiliary:legalization-deduction", + ]); +}); + +test("task.json：缺字段、多字段、类型不对，一次全部列出，带字段路径", () => { + const good = JSON.parse(readFileSync(join(TASK_DIR, "task.json"), "utf8")) as Record; + const bad = { + ...good, + epsilon: "0.01", + knownBest: "unknown", + run: { wallSeconds: 120, cores: 1.5 }, + extra: true, + } as Record; + delete bad.protectedFiles; + assert.throws( + () => checkTaskFacts(bad), + (error: Error) => { + const lines = error.message.split("\n").slice(1); + assert.match(error.message, /^task\.json 不合格，共 6 处/); + assert.ok(lines.some((l) => l.startsWith("/protectedFiles：缺少")), error.message); + assert.ok(lines.some((l) => l.startsWith("/extra：不认识的字段")), error.message); + assert.ok(lines.some((l) => l.startsWith("/epsilon：")), error.message); + assert.ok(lines.some((l) => l.startsWith("/knownBest：应为数字，没有就写 null")), error.message); + assert.ok(lines.some((l) => l.startsWith("/run/cores：")), error.message); + assert.ok(lines.some((l) => l.startsWith("/run/runtime：缺少")), error.message); + return true; + }, + ); + assert.deepEqual(checkTaskFacts({ ...good, knownBest: null }).knownBest, null); +}); + +test("读任务：task.json 不是合法 JSON 时报错带文件路径", (t) => { + const dir = tempDir(t, "badjson"); + writeFileSync(join(dir, "task.json"), "{ not json"); + assert.throws(() => loadTask(dir), /task\.json 读不出来或不是合法的 JSON/); +}); + +test("不许改的文件：只交回改动名单里属于 task.json 名单的那些", () => { + assert.deepEqual(protectedChanges(task, ["initial.py", "harness.py", "lib/util.py", "validate.py"]), ["harness.py", "validate.py"]); + assert.deepEqual(protectedChanges(task, ["initial.py"]), []); +}); + +// --------------------------------------------------------------------------- +// 判定表（任务书 05 第三部分"判定怎么定"） +// --------------------------------------------------------------------------- + +test("判定表 1：正式运行超时、被从外面结束、退出码非 0，不算数，不跑校验器，资源用量照记", async (t) => { + const un: Measured = { status: "unmeasured", reason: "容器从外面被结束" }; + const cases: Array<[ProcessResult, RegExp]> = [ + [{ ending: "timeout", cpuSeconds: m(119), wallSeconds: m(120), peakMemoryBytes: m(5) }, /超过单次时限 120 秒/], + [{ ending: "cancelled", cpuSeconds: un, wallSeconds: un, peakMemoryBytes: un }, /容器从外面被结束/], + [{ ...EXITED_OK, exitCode: 1 }, /退出码 1/], + ]; + for (const [formal, why] of cases) { + // 有一份合法的产物也不跑校验器：运行没正常结束就不算数。 + const result = await judgeRun(t, "unused", runDirWith(t, loose()), formal, []); + assert.equal(result.verdict, "invalid"); + assert.match(result.verdictReason, why); + assert.match(reason(result.score), why); + assert.match(reason(result.metrics["raw-score"]), /没有跑校验器/); + assert.deepEqual(result.extra, { legalization: null }); + if (formal.ending !== "notStarted") assert.deepEqual(result.metrics["cpu-seconds"], formal.cpuSeconds); + } +}); + +test("判定表 2：没有 solution.json，不算数：没有产物", async (t) => { + const result = await judgeRun(t, "unused", runDirWith(t, null), EXITED_OK, []); + assert.equal(result.verdict, "invalid"); + assert.match(result.verdictReason, /^没有产物/); +}); + +test("判定表 3：校验器判不合法（形状不对、非有限数、圆心出界修不了），不算数，带校验器的原因", async (t) => { + const short = loose(); + short.centers.pop(); + short.radii.pop(); + const shape = await judgeRun(t, "shape", runDirWith(t, short), EXITED_OK, []); + assert.equal(shape.verdict, "invalid"); + assert.match(shape.verdictReason, /shape incorrect/); + assert.match(reason(shape.metrics["raw-score"]), /形状不对/); + assert.match(reason(shape.metrics["min-pairwise-clearance"]), /形状不对/); + + const nan = JSON.stringify(loose()).replace('"radii":[0.0078125', '"radii":[NaN'); + const nonFinite = await judgeRun(t, "nan", runDirWith(t, nan), EXITED_OK, []); + assert.equal(nonFinite.verdict, "invalid"); + assert.match(nonFinite.verdictReason, /non-finite/); + + // 0 号圆心挪到 x = −1/64：圆心在正方形外，只缩半径修不了。 + const outside = loose(); + outside.centers[0] = [-1 / 64, 0.5]; + const result = await judgeRun(t, "outside", runDirWith(t, outside), EXITED_OK, []); + assert.equal(result.verdict, "invalid"); + assert.match(result.verdictReason, /圆心在正方形外面/); + assert.match(reason(result.score), /圆心在正方形外面/); + assert.match(reason(result.metrics["legalization-deduction"]), /修不了/); + // 原分数照样算得出：26 个圆半径都没动，0.203125。 + assert.equal(value(result.metrics["raw-score"]), 0.203125); +}); + +test("判定表 4：校验器判合法，算数，分数取校验器算的（宽松 0.203125、相切 0.40625，扣 0）", async (t) => { + const result = await judgeRun(t, "loose", runDirWith(t, loose()), EXITED_OK, []); + assert.equal(result.verdict, "valid"); + assert.equal(value(result.score), 0.203125); + assert.equal(value(result.metrics["min-pairwise-clearance"]), 0.015625); + assert.equal(value(result.metrics["raw-score"]), 0.203125); + assert.equal(value(result.metrics["raw-violation"]), 0); + assert.equal(value(result.metrics["legalization-deduction"]), 0); + // 资源用量原样来自正式运行。 + assert.equal(value(result.metrics["wall-seconds"]), 2.5); + assert.equal(value(result.metrics["peak-memory-bytes"]), 1024); + + const touching = await judgeRun(t, "tangent", runDirWith(t, tangent()), EXITED_OK, []); + assert.equal(touching.verdict, "valid"); + assert.equal(value(touching.score), 0.40625); + assert.equal(value(touching.metrics["min-pairwise-clearance"]), 0); +}); + +test("判定表 4（合法化）：差一点不合法的修好后算数，修后分数不高于原分数，扣分等于原分数减最终分数", async (t) => { + // 0、1 号半径放大到 3/128：重叠 3/64 − 1/32 = 1/64；1、2 号间隙恰好 0。原分数 30/128 = 0.234375。 + // 只缩参与违反的圆：0、1 号各减 1/64 回到 1/128，修后 0.203125，扣 1/32 = 0.03125。 + const overlap = loose(); + overlap.radii[0] = 3 / 128; + overlap.radii[1] = 3 / 128; + const shrunk = await judgeRun(t, "overlap", runDirWith(t, overlap), EXITED_OK, []); + assert.equal(shrunk.verdict, "valid"); + assert.equal(value(shrunk.score), 0.203125); + assert.equal(value(shrunk.metrics["raw-score"]), 0.234375); + assert.equal(value(shrunk.metrics["raw-violation"]), 0.015625); + assert.equal(value(shrunk.metrics["legalization-deduction"]), 0.03125); + const extra = shrunk.extra as { legalization: { shrunkCircles: number[]; rawValid: boolean; deduction: Measured } }; + assert.deepEqual(extra.legalization.shrunkCircles, [0, 1]); + assert.equal(extra.legalization.rawValid, false); + assert.equal(value(extra.legalization.deduction), 0.03125); + + // 0 号圆心挪到 x = 1/256，出左边 1/128 − 1/256 = 1/256；修后半径 1/256，和 51/256 = 0.19921875。 + const edge = loose(); + edge.centers[0] = [1 / 256, 0.5]; + const edged = await judgeRun(t, "edge", runDirWith(t, edge), EXITED_OK, []); + assert.equal(value(edged.score), 0.19921875); + assert.equal(value(edged.metrics["raw-violation"]), 1 / 256); + assert.equal(value(edged.metrics["legalization-deduction"]), 0.203125 - 0.19921875); + + // 参考项目真实运行 14-real-1 的产物：重叠约 2.78e-17，修后 1.7602769492805097（参考项目设计-15 7.1 节记的数）。 + const real = await judgeRun(t, "real", runDirWith(t, readFileSync(FIXTURE, "utf8")), EXITED_OK, []); + assert.equal(real.verdict, "valid"); + assert.equal(value(real.score), 1.7602769492805097); + const raw = value(real.metrics["raw-score"]); + assert.equal(raw, 1.76027694928051); + assert.ok(value(real.score) <= raw); + assert.equal(value(real.metrics["legalization-deduction"]), raw - value(real.score)); + assert.ok(Math.abs(value(real.metrics["raw-violation"]) - 2.78e-17) < 0.01e-17); +}); + +test("判定表 5：缺了整次作废的假设专用指标测不到，不算数；缺了只记缺失的照常算数", async (t) => { + const spec = (id: string, missing: HypothesisMetric["missing"]): HypothesisMetric => ({ + id, + meaning: "测试用", + unit: "无", + direction: "maximize", + howMeasured: "测试用", + missing, + }); + const runDir = runDirWith(t, loose()); + writeFileSync( + join(runDir, "metrics.json"), + JSON.stringify({ spread: 0.5, gap: null, "answer-path": "p1", "safeguard-triggers": { "shrink-repair": 2 } }), + ); + const kept = await judgeRun(t, "keep", runDir, EXITED_OK, [spec("spread", "invalidateRun"), spec("gap", "recordMissing")]); + assert.equal(kept.verdict, "valid"); + assert.equal(value(kept.metrics.spread), 0.5); + assert.match(reason(kept.metrics.gap), /null/); + assert.deepEqual(kept.answerPath, { status: "read", value: "p1" }); + assert.deepEqual(kept.safeguardTriggers, { status: "read", counts: { "shrink-repair": 2 } }); + + const voided = await judgeRun(t, "void", runDir, EXITED_OK, [spec("gap", "invalidateRun")]); + assert.equal(voided.verdict, "invalid"); + assert.match(voided.verdictReason, /缺了整次作废.*gap/); + // 成绩照样算出来记下，只是这次运行不算数。 + assert.equal(value(voided.score), 0.203125); +}); + +test("答案路径、保障动手次数：读不到时记读不到及原因，不作废运行", async (t) => { + const missing = await judgeRun(t, "nometrics", runDirWith(t, loose()), EXITED_OK, []); + assert.equal(missing.verdict, "valid"); + assert.deepEqual(missing.answerPath, { status: "unreadable", reason: "metrics.json 不存在" }); + + const runDir = runDirWith(t, loose()); + writeFileSync(join(runDir, "metrics.json"), JSON.stringify({ "answer-path": 3, "safeguard-triggers": { a: -1 } })); + const wrong = await judgeRun(t, "wrongkeys", runDir, EXITED_OK, []); + assert.equal(wrong.verdict, "valid"); + assert.equal(wrong.answerPath.status, "unreadable"); + assert.equal(wrong.safeguardTriggers.status, "unreadable"); +}); + +test("校验器自己没正常跑完、判定认不出、修后分数高于原分数：报错停下，不当成候选的失败", (t) => { + const runDir = runDirWith(t, loose()); + const judgeDir = tempDir(t, "judge"); + const un: Measured = { status: "unmeasured", reason: "x" }; + const timeout: ProcessResult = { ending: "timeout", cpuSeconds: un, wallSeconds: un, peakMemoryBytes: un }; + assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, timeout, []), /校验器没有正常跑完（timeout）/); + // 退出码 2 是用法或读取出错，这时校验器不写判定文件。 + assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, { ...EXITED_OK, exitCode: 2 }, []), /校验器没有正常跑完（退出码 2）/); + assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, EXITED_OK, []), /读不出 .*verdict\.json/); + + const verdict = { + valid: true, + computed_sum: 0.3, + reasons: [], + diagnostics: { min_pairwise_clearance: 0 }, + legalization: { raw_valid: false, raw_score: 0.2, raw_violation: 0.1, applied: true, repaired: true, shrunk_circles: [0], reasons: [] }, + }; + writeFileSync(join(judgeDir, "verdict.json"), JSON.stringify(verdict)); + assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, EXITED_OK, []), /合法化后分数 0\.3 高于原分数 0\.2/); + // 退出码说不合法、判定说合法：对不上。 + assert.throws(() => judge(task, runDir, judgeDir, EXITED_OK, { ...EXITED_OK, exitCode: 1 }, []), /对不上/); +}); + +// --------------------------------------------------------------------------- +// 轻量检查 +// --------------------------------------------------------------------------- + +async function check(t: TestContext, suffix: string, files: Record) { + const candidate = tempDir(t, "candidate"); + for (const [name, text] of Object.entries(files)) writeFileSync(join(candidate, name), text); + const checkDir = tempDir(t, "check"); + const definition = lightCheck(task, candidate, checkDir, digest, 5); + return readLightCheck(task, checkDir, await run(t, suffix, definition, checkDir)); +} + +test("轻量检查：语法错的、没有入口函数的不通过，正常的通过", async (t) => { + const syntax = await check(t, "check-syntax", { "initial.py": "def run_packing(:\n pass\n" }); + assert.equal(syntax.passed, false); + assert.match(syntax.passed ? "" : syntax.reasons.join(), /initial\.py 解析失败/); + + const noEntry = await check(t, "check-noentry", { "initial.py": "def solve():\n return [], [], 0.0\n" }); + assert.equal(noEntry.passed, false); + assert.match(noEntry.passed ? "" : noEntry.reasons.join(), /run_packing\(\) 应当恰好有一个定义，实际有 0 个/); + + const baseline = tempDir(t, "baseline"); + cpSync(task.baselineDir, baseline, { recursive: true }); + const checkDir = tempDir(t, "check"); + const ok = readLightCheck(task, checkDir, await run(t, "check-ok", lightCheck(task, baseline, checkDir, digest, 5), checkDir)); + assert.deepEqual(ok, { passed: true }); +}); + +// --------------------------------------------------------------------------- +// 从头到尾 +// --------------------------------------------------------------------------- + +/** 基线 initial.py 准备运行目录 → 正式运行 → 跑校验器 → 判分。 */ +async function endToEnd(t: TestContext, suffix: string): Promise { + const runDir = tempDir(t, "e2e"); + prepareRun(task, task.baselineDir, runDir, []); + assert.equal(readFileSync(join(runDir, "metrics-spec.json"), "utf8"), "[]\n"); + assert.ok(existsSync(join(runDir, "harness.py")) && existsSync(join(runDir, "initial.py"))); + const formal = await run(t, `e2e-${suffix}`, formalRun(task, digest, "inforsi-test-seed"), runDir); + assert.equal(formal.ending, "exited", JSON.stringify(formal)); + return judgeRun(t, `e2e-${suffix}-judge`, runDir, formal, []); +} + +test("从头到尾：基线跑两次都判为算数，分数逐位相同", async (t) => { + const first = await endToEnd(t, "1"); + const second = await endToEnd(t, "2"); + for (const result of [first, second]) { + assert.equal(result.verdict, "valid", result.verdictReason); + assert.equal(result.score.status, "measured"); + } + assert.ok(Object.is(value(first.score), value(second.score)), `${value(first.score)} 与 ${value(second.score)}`); + for (const [label, result] of [["第一次", first], ["第二次", second]] as const) { + t.diagnostic( + `${label}：verdict=${result.verdict}（${result.verdictReason}） score=${value(result.score)} ` + + `cpu=${JSON.stringify(result.metrics["cpu-seconds"])} wall=${JSON.stringify(result.metrics["wall-seconds"])} ` + + `mem=${JSON.stringify(result.metrics["peak-memory-bytes"])} deduction=${JSON.stringify(result.metrics["legalization-deduction"])}`, + ); + } +}); + +test("候选目录里遗留的旧 metrics.json 不会被当成这次的产物：整次作废的指标这次没写，判为不算数", async (t) => { + const candidate = tempDir(t, "candidate"); + cpSync(task.baselineDir, candidate, { recursive: true }); + // 基线代码不写 metrics.json；这份是以前某次运行留下的。 + writeFileSync(join(candidate, "metrics.json"), JSON.stringify({ spread: 0.5, "answer-path": "main", "safeguard-triggers": {} })); + const spread: HypothesisMetric = { + id: "spread", + meaning: "测试用", + unit: "无", + direction: "maximize", + howMeasured: "测试用", + missing: "invalidateRun", + }; + const runDir = tempDir(t, "stale"); + prepareRun(task, candidate, runDir, [spread]); + const formal = await run(t, "stale", formalRun(task, digest, "inforsi-test-seed"), runDir); + assert.equal(formal.ending, "exited", JSON.stringify(formal)); + const result = await judgeRun(t, "stale-judge", runDir, formal, [spread]); + t.diagnostic(`verdict=${result.verdict}（${result.verdictReason}） spread=${JSON.stringify(result.metrics.spread)}`); + assert.equal(result.metrics.spread?.status, "unmeasured"); + assert.equal(result.verdict, "invalid"); + assert.match(result.verdictReason, /缺了整次作废.*spread/); +});