feat: 任务书 07，agents 模块与真实模型测试
src/agents 4 个文件 681 行：runSession 按角色开 Pi 子会话、发工具、核对实际加载内容， 提交参数非法 JSON、必填字段交 null、Pi 形状校验不通过都经提交回调记退回。 Pi 依赖固定 0.85.1；新增 npm run test:pi（glm-5.3 真实会话）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+3
-2
@@ -17,6 +17,7 @@
|
||||
- 任务书 04（config）：src/config 4 个文件、453 行，`npm test` 246 条通过（config 19 条）。流程用联合类型（完整流程 / 朴素循环），规则名取自 rules 登记表，config 改为依赖 rules。ledger 顺手两处：结算按轮次唯一、笔记去掉版本号，表结构版本号升到 2，全新上下文审阅一轮无问题。结构.md 第八部分补了 config 对外内容。
|
||||
- 任务书 05（tasks 与 runner）：src/runner 2 个文件、271 行，src/tasks 6 个文件、691 行，`npm test` 274 条通过（runner 12 条、tasks 16 条，真实 Docker，整套约 12 秒）。任务目录搬到 `tasks/circle_packing/`，新增 task.json；`VERDICTS` 删掉"判不了"。基线从头到尾判为算数，分数 0.9597642169962064，两次逐位相同。全新上下文审阅两轮：第一轮发现候选能伪造计量结果（把超时记成正常退出）、运行目录里旧 metrics.json 会被当成本次产物，都已修（计量 shell 以容器内 root 跑、命令用 setpriv 降权、结果用 docker cp 取；harness 开跑前删旧输出）；第二轮只审这次修复，另采纳一条：容器加 no-new-privileges，禁止候选靠 setuid 程序重新拿到 root。结构.md 第八部分补了 runner、tasks。
|
||||
- 任务书 06（workspace）：src/workspace 2 个文件、205 行，`npm test` 284 条通过（workspace 10 条，真实 git）。版本号是目录树编号，存版本自己列文件加 hash-object，取出用 ls-tree 加 cat-file（实测 `git archive` 会按树里的 `.gitattributes` 丢文件、改换行符），git 子进程另关掉系统和用户的 attributes 文件（实测只关配置时 `~/.config/git/attributes` 仍会被读）。tasks 的 task.json 加必填的 `unversionedNames`，`protectedFiles` 删掉 solution.json。全新上下文审阅一轮：两处偏离都实测成立；采纳一条（写明空文件增删、只改可执行位没有块），其余不改。结构.md 第八部分补了 workspace。
|
||||
- 任务书 07（agents）：src/agents 4 个文件、681 行（预期 400–600，超出部分是任务书第四部分要保留的三处提交处理、读代码分段和会话核对，没有新增专门路径），`npm test` 293 条通过（agents 9 条，其中 1 条经真实 Pi 会话连本机假模型服务）。package.json 固定 `@earendil-works/pi-coding-agent`、`pi-ai` 0.85.1 并加 overrides（同参考项目）；新增 `npm run test:pi`（真实 glm-5.3，研究者先退回再收下、实验员写文件跑 shell 再交）。实测：glm-5.3 经 iomgaa 服务时，正文里没转义的半角双引号会在服务那一侧把参数截成合法但半截的 JSON，Pi 收到的原文本身合法，JSON 检查查不出（抓包确认）；提交工具说明里加了一句"引号用「」"，之后 12 份提交没再出现。结构.md 第七、八部分已更新。
|
||||
|
||||
## 关键决定（用户原话见各提交说明）
|
||||
|
||||
@@ -67,7 +68,7 @@
|
||||
- 给 flow：同一轮里选批和选格都用随机规则时，传不同的"版本"算种子，否则两次抽取用同一串随机数。
|
||||
- 给 flow：确认启动前要先登记运行；模型调用要先开始再结束；阶段要先开再关；ledger 都不查。
|
||||
- 给 materials：别人的"退回"记录不放进任何研究者的材料（退回的提案原样带着作者自评，不另拆封存）。
|
||||
- 给 agents：模型的思考档位必须写明，不用参考项目"写 null 表示用 Pi 自己的默认"那种写法。
|
||||
- ledger 的 `callEnded` 没有墙钟时间字段，结构.md 第三部分第 10 种和任务书 07 都要求 flow 把墙钟时间写进"模型调用结束"（任务书 07 执行时发现）。
|
||||
- 给 flow：步骤清单没有单列"轮末动作"（解封作者自评、入选未跑的候选出池），flow 要做这些。
|
||||
- 写 plugin 时实测：源码导入路径带 `.ts` 扩展名，Pi 加载插件能否接受。
|
||||
- 给 submissions：研究者登记的假设专用指标编号不能和常规辅助指标重名（`metricDefinitions` 里的 7 个），否则候选自报的值会顶替宿主测的值（任务书 05 审阅意见）。
|
||||
@@ -88,4 +89,4 @@
|
||||
|
||||
## 下一步
|
||||
|
||||
任务书 07（agents，`任务书/07-agents.md`）已写好，等用户另开新 Agent 执行；完成后监督者核对。
|
||||
任务书 07（agents）已提交，等监督者核对，并等用户定：ledger 的"模型调用结束"加不加墙钟时间字段；agents 行数上限是否放宽。
|
||||
|
||||
+13
-1
@@ -190,7 +190,7 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
| submissions | 600–1000 |
|
||||
| jev | 300–500 |
|
||||
| materials | 1000–1800 |
|
||||
| agents | 800–1300 |
|
||||
| agents | 400–600 |
|
||||
| workspace | 200–300 |
|
||||
| runner | 250–450 |
|
||||
| tasks | 公共部分与填圆合计 400–700，另有任务目录里的 Python 文件 |
|
||||
@@ -493,3 +493,15 @@ Jev 是一个判别器，和程序检查地位相同，不单独成一种记录
|
||||
- `SaveOutcome`（类型）：存好了和版本号，或者存不了、路径和原因。
|
||||
- `LineRange`、`Hunk`（类型）：一块改动在一侧是"第几行到第几行"；这一侧没有行时写成"在第几行之后"（0 表示文件开头），同 `git diff -U0` 的写法。
|
||||
- `FileChange`、`Comparison`（类型）：一个改动过的文件，二进制文件不拆块；比较的全部结果。块可以为空：空文件的新增、删除和只改可执行位都没有块，按整个文件改了算。
|
||||
|
||||
### agents
|
||||
|
||||
下面是 `src/agents/index.ts` 导出的全部内容。agents 是程序里唯一直接用 Pi 的模块：开一个子会话，按角色发工具，把模型的查询、提交、笔记、轻量检查和每次回复的用量转给调用方的回调，等会话结束交回结局。它不碰账本，不决定给模型看什么，也不判断交上来的东西好不好。Pi 的设置不读用户和工作目录里的 settings.json，用到的几项（重试、压缩上下文、HTTP 超时）写在 `session.ts` 里；模型服务配置照常用本机 Pi 的 models.json 和 auth.json。
|
||||
|
||||
- `runSession(task)`：跑一次会话，交回结局。建好会话后先核对实际装了什么（模型和思考档位没被换、没有扩展、技能、提示模板和上下文文件、系统提示词只多了 Pi 自己加的工作目录一行、工具表正好是该给的那几个且是给的实现），不通过就一个请求都不发，交回出错；回调抛错按调用方的内部错误处理，会话立即停下，错误原样抛出。
|
||||
- `SessionTask`（类型）：跑一次会话的输入，按角色区分，全部必填：模型、角色提示词和第一条消息（materials 拼好的文字）、工作目录、会话文件目录、提交种类、查询、提交、用量三个回调和"停下"信号；研究者另有能读的目录和写笔记，核验员另有能读的目录，实验员另有轻量检查；研究者交上一份被收下后会话不结束，实验员、核验员交上就结束。
|
||||
- `RoleModel`、`ThinkingLevel`（类型）：provider、模型名和思考档位，档位必须写明，名单与 config 的 `THINKING_LEVELS` 相同。
|
||||
- `KindOf<R>`（类型）：某个角色交的提交种类，取自 vocab 的 `SUBMISSION_KINDS`。
|
||||
- `SubmitAttempt<K>`（类型）：交给提交回调的一次提交：形状合格的内容和"替换哪一份"（只有研究者的提交能替换），或形状不合格的原样参数和原因（参数原文不是合法 JSON、必填字段交了 null、Pi 的形状校验不通过），后者回调只需记一条退回。
|
||||
- `SubmitReceipt`（类型）：提交回调的回答，收下和编号，或退回和原因，agents 原样回给模型；收下时另回显收到的字数和结尾。
|
||||
- `SessionEnding`、`SessionOutcome`（类型）：交了、没交（附原因，例如最后一条回复被输出上限截断）、出错（附原因）、被停下四种，都带墙钟时间（秒）和会话文件路径（Pi 没写出会话文件时为 null）。
|
||||
|
||||
Generated
+3384
-11
File diff suppressed because it is too large
Load Diff
+10
-1
@@ -8,14 +8,23 @@
|
||||
},
|
||||
"scripts": {
|
||||
"check": "tsc --noEmit -p tsconfig.json && depcruise src tests --config .dependency-cruiser.cjs",
|
||||
"test": "node --test \"tests/**/*.test.ts\""
|
||||
"test": "node --test \"tests/**/*.test.ts\"",
|
||||
"test:pi": "node tests/agents-pi.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@earendil-works/pi-ai": "0.85.1",
|
||||
"@earendil-works/pi-coding-agent": "0.85.1",
|
||||
"typebox": "1.3.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "22.19.19",
|
||||
"dependency-cruiser": "18.5.0",
|
||||
"typescript": "5.9.3"
|
||||
},
|
||||
"overrides": {
|
||||
"@earendil-works/pi-agent-core": "0.85.1",
|
||||
"@earendil-works/pi-ai": "0.85.1",
|
||||
"@earendil-works/pi-tui": "0.85.1",
|
||||
"@earendil-works/chord": "0.85.1"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
// agents：把一个任务交给 Pi 里的一个模型去做，等它交上来（docs/结构.md 第二部分）。
|
||||
// 程序里只有它直接跟 Pi 打交道；不碰账本，不决定给模型看什么，也不判断交上来的东西好不好。
|
||||
export { runSession } from "./session.ts";
|
||||
export type {
|
||||
KindOf,
|
||||
RoleModel,
|
||||
SessionEnding,
|
||||
SessionOutcome,
|
||||
SessionTask,
|
||||
SubmitAttempt,
|
||||
SubmitReceipt,
|
||||
ThinkingLevel,
|
||||
} from "./types.ts";
|
||||
@@ -0,0 +1,225 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import type { Usage } from "@earendil-works/pi-ai";
|
||||
import {
|
||||
createAgentSession,
|
||||
DefaultResourceLoader,
|
||||
getAgentDir,
|
||||
ModelRuntime,
|
||||
SessionManager,
|
||||
SettingsManager,
|
||||
type AgentSession,
|
||||
type AgentSessionEvent,
|
||||
type CreateAgentSessionResult,
|
||||
type ToolDefinition,
|
||||
} from "@earendil-works/pi-coding-agent";
|
||||
import type { ModelUsage } from "../vocab/index.ts";
|
||||
import { buildTools, newSubmitState, toolNames, watchSubmitArguments, type Host } from "./tools.ts";
|
||||
import type { SessionEnding, SessionOutcome, SessionTask } from "./types.ts";
|
||||
|
||||
// 跑一次角色会话：新建一个 Pi 子会话，按角色发工具，发第一条消息，等模型干完，交回结局（任务书 07 第一部分）。
|
||||
// 只用 Pi 0.85.1 的公开接口；用量只从公开事件 message_end、compaction_end 取。
|
||||
|
||||
/**
|
||||
* 子会话用的 Pi 设置。不读用户和工作目录里的 settings.json（本机那份改了压缩的保留量、列了一串插件包），
|
||||
* 用到的每一项都写在这里，值与 Pi 0.85.1 自己的默认相同（dist/core/settings-manager.js）：
|
||||
* - 重试：模型服务报可重试的错（过载、限流、服务端错误）时由 Pi 自己重试 3 次，等待从 2 秒起每次翻倍
|
||||
* （agent-session.js _prepareRetry）；服务要求等待超过 60 秒的直接算失败。HTTP 客户端那一层不再另外重试
|
||||
* （provider.maxRetries 为 0，openai-completions 走 pi-ai utils/provider-retry.js，0 即不重试）。
|
||||
* 每次失败的请求照样交一份"测不到"的用量。
|
||||
* - 压缩上下文：开着，离上下文上限不到 16384 个 token 时把较早的对话压成摘要，保留最近 20000 个 token。
|
||||
* 写摘要也是一次模型调用，用量从 compaction_end 交出。
|
||||
* - HTTP 空闲超时 300 秒。
|
||||
*/
|
||||
const PI_SETTINGS = {
|
||||
retry: { enabled: true, maxRetries: 3, baseDelayMs: 2000, provider: { maxRetries: 0, maxRetryDelayMs: 60_000 } },
|
||||
compaction: { enabled: true, reserveTokens: 16384, keepRecentTokens: 20000 },
|
||||
httpIdleTimeoutMs: 300_000,
|
||||
};
|
||||
|
||||
const COST_UNKNOWN = { status: "unmeasured", reason: "Pi 的价格表不是账单，不据此记费用" } as const;
|
||||
|
||||
/**
|
||||
* 一次回复的用量。一次成功的回复至少要读系统提示词，输入不可能是 0；输入、缓存读、缓存写全是 0，
|
||||
* 说明模型服务没报用量（出错、被中止的请求就是这样），四项都记测不到，不记 0（方法.md 第七部分）。
|
||||
*/
|
||||
function toModelUsage(usage: Usage, stopReason: string): ModelUsage {
|
||||
if (usage.input + usage.cacheRead + usage.cacheWrite === 0) {
|
||||
const reason = `模型服务没有报这次回复的用量（这次回复的结束原因：${stopReason}）`;
|
||||
const unmeasured = { status: "unmeasured", reason } as const;
|
||||
return { inputTokens: unmeasured, cacheReadTokens: unmeasured, cacheWriteTokens: unmeasured, outputTokens: unmeasured, cost: COST_UNKNOWN };
|
||||
}
|
||||
const measured = (value: number) => ({ status: "measured", value }) as const;
|
||||
return {
|
||||
inputTokens: measured(usage.input),
|
||||
cacheReadTokens: measured(usage.cacheRead),
|
||||
cacheWriteTokens: measured(usage.cacheWrite),
|
||||
outputTokens: measured(usage.output),
|
||||
cost: COST_UNKNOWN,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 跑一次会话，交回结局。回调抛出的错误是调用方自己的内部错误：会话立即停下，错误原样抛出，不变成结局。
|
||||
* 模型和凭据由 Pi 的模型服务配置给（~/.pi/agent 下的 models.json、auth.json）。
|
||||
*/
|
||||
export async function runSession(task: SessionTask): Promise<SessionOutcome> {
|
||||
const started = performance.now();
|
||||
const runtime = await ModelRuntime.create();
|
||||
const { provider, model: modelId, thinking } = task.model;
|
||||
const model = runtime.getModel(provider, modelId);
|
||||
const end = (session: AgentSession | null, ending: SessionEnding): SessionOutcome => {
|
||||
const file = session?.sessionFile;
|
||||
const sessionFile = file !== undefined && existsSync(file) ? file : null;
|
||||
return { ...ending, wallSeconds: (performance.now() - started) / 1000, sessionFile };
|
||||
};
|
||||
if (model === undefined) return end(null, { status: "failed", reason: `Pi 的模型服务配置里没有 ${provider}/${modelId}，不换用别的模型` });
|
||||
|
||||
// 资源：五个自动发现全关，系统提示词直接给，.pi/APPEND_SYSTEM.md 的发现也关掉；建好后逐项核对。
|
||||
const settingsManager = SettingsManager.inMemory(PI_SETTINGS);
|
||||
const loader = new DefaultResourceLoader({
|
||||
cwd: task.workDir,
|
||||
agentDir: getAgentDir(),
|
||||
settingsManager,
|
||||
noExtensions: true,
|
||||
noSkills: true,
|
||||
noPromptTemplates: true,
|
||||
noThemes: true,
|
||||
noContextFiles: true,
|
||||
systemPromptOverride: () => task.systemPrompt,
|
||||
appendSystemPromptOverride: () => [],
|
||||
});
|
||||
await loader.reload();
|
||||
|
||||
let session: AgentSession | null = null;
|
||||
const failures: unknown[] = [];
|
||||
const host: Host = {
|
||||
fail: (error) => {
|
||||
failures.push(error);
|
||||
void session?.abort();
|
||||
},
|
||||
guard: (run) => {
|
||||
try {
|
||||
const result = run();
|
||||
return (result instanceof Promise ? result.catch((error) => { host.fail(error); throw error; }) : result) as typeof result;
|
||||
} catch (error) {
|
||||
host.fail(error);
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
};
|
||||
const state = newSubmitState();
|
||||
const customTools = buildTools(task, state, host);
|
||||
const expected = toolNames(task.role);
|
||||
const created = await createAgentSession({
|
||||
cwd: task.workDir,
|
||||
model,
|
||||
thinkingLevel: thinking,
|
||||
modelRuntime: runtime,
|
||||
resourceLoader: loader,
|
||||
sessionManager: SessionManager.create(task.workDir, task.sessionDir),
|
||||
settingsManager,
|
||||
tools: expected,
|
||||
customTools,
|
||||
});
|
||||
session = created.session;
|
||||
const live = session;
|
||||
|
||||
let stopped = false;
|
||||
const onStop = () => {
|
||||
stopped = true;
|
||||
void live.abort();
|
||||
};
|
||||
const unsubscribe = live.subscribe((event: AgentSessionEvent) => {
|
||||
if (event.type === "message_update") watchSubmitArguments(state, event.assistantMessageEvent);
|
||||
else if (event.type === "message_end" && event.message.role === "assistant") {
|
||||
report(toModelUsage(event.message.usage, event.message.stopReason));
|
||||
} else if (event.type === "compaction_end" && event.result?.usage !== undefined) {
|
||||
report(toModelUsage(event.result.usage, "compaction"));
|
||||
}
|
||||
});
|
||||
// 用量回调在 Pi 的事件监听里调，出错不能抛回 Pi，交给 host.fail 停下会话、事后抛出。
|
||||
function report(usage: ModelUsage): void {
|
||||
try {
|
||||
task.usage(usage);
|
||||
} catch (error) {
|
||||
host.fail(error);
|
||||
}
|
||||
}
|
||||
try {
|
||||
const problems = verify(task, created, loader, expected, customTools);
|
||||
if (problems.length > 0) return end(live, { status: "failed", reason: `子会话核对不通过，没有发任何请求：${problems.join("；")}` });
|
||||
task.stop.addEventListener("abort", onStop, { once: true });
|
||||
if (task.stop.aborted) return end(live, { status: "stopped" });
|
||||
|
||||
// expandPromptTemplates: false：第一条消息以 "/" 开头时不被当成命令或提示模板展开。
|
||||
// prompt 只在发请求之前的检查里抛错（例如这个 provider 没有凭据），那是配置问题，原样抛给调用方。
|
||||
await live.prompt(task.firstMessage, { expandPromptTemplates: false });
|
||||
await Promise.all(state.pending);
|
||||
if (failures.length > 0) throw failures[0];
|
||||
if (stopped) return end(live, { status: "stopped" });
|
||||
|
||||
const last = live.messages.findLast((m) => m.role === "assistant");
|
||||
if (last?.role === "assistant" && (last.stopReason === "error" || last.stopReason === "aborted")) {
|
||||
return end(live, { status: "failed", reason: `模型服务报错（Pi 已按设置重试）：${last.errorMessage ?? last.stopReason}` });
|
||||
}
|
||||
if (state.accepted > 0) return end(live, { status: "submitted" });
|
||||
const reason =
|
||||
last?.role === "assistant" && last.stopReason === "length"
|
||||
? `最后一条回复被输出上限截断（输出 ${last.usage.output} 个 token），没有被收下的提交`
|
||||
: "模型结束了回复，没有一份提交被收下";
|
||||
return end(live, { status: "notSubmitted", reason });
|
||||
} finally {
|
||||
task.stop.removeEventListener("abort", onStop);
|
||||
unsubscribe();
|
||||
live.dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 建好会话后、发任何请求前，逐项核对子会话里实际装了什么（CLAUDE.md"Pi 与三个角色"最后一条）。
|
||||
* 本机 ~/.pi/agent 下有扩展和技能，开关一旦失效，子会话会悄悄多出东西，没人操作也会发生。
|
||||
*/
|
||||
function verify(
|
||||
task: SessionTask,
|
||||
created: CreateAgentSessionResult,
|
||||
loader: DefaultResourceLoader,
|
||||
expected: string[],
|
||||
customTools: ToolDefinition[],
|
||||
): string[] {
|
||||
const { session } = created;
|
||||
const problems: string[] = [];
|
||||
const want = `${task.model.provider}/${task.model.model}`;
|
||||
if (created.modelFallbackMessage !== undefined) problems.push(`Pi 换了模型：${created.modelFallbackMessage}`);
|
||||
if (`${session.model?.provider}/${session.model?.id}` !== want) {
|
||||
problems.push(`会话模型是 ${session.model?.provider}/${session.model?.id}，不是 ${want}`);
|
||||
}
|
||||
// Pi 会按模型能力把档位悄悄往下调（pi-ai clampThinkingLevel）。
|
||||
if (session.thinkingLevel !== task.model.thinking) {
|
||||
problems.push(`思考档位是 ${session.thinkingLevel}，不是 ${task.model.thinking}（模型不支持这一档）`);
|
||||
}
|
||||
const extensions = created.extensionsResult.extensions.map((e) => e.path);
|
||||
if (extensions.length > 0) problems.push(`加载了扩展：${extensions.join(", ")}`);
|
||||
const extensionErrors = created.extensionsResult.errors.map((e) => e.path);
|
||||
if (extensionErrors.length > 0) problems.push(`扩展加载报错：${extensionErrors.join(", ")}`);
|
||||
const skills = loader.getSkills().skills.map((s) => s.name);
|
||||
if (skills.length > 0) problems.push(`加载了技能：${skills.join(", ")}`);
|
||||
const prompts = loader.getPrompts().prompts.map((p) => p.name);
|
||||
if (prompts.length > 0) problems.push(`加载了提示模板：${prompts.join(", ")}`);
|
||||
const contextFiles = loader.getAgentsFiles().agentsFiles.map((f) => f.path);
|
||||
if (contextFiles.length > 0) problems.push(`加载了上下文文件：${contextFiles.join(", ")}`);
|
||||
// 给了自定义系统提示词时，Pi 只在末尾加一行工作目录（dist/core/system-prompt.js buildSystemPrompt），
|
||||
// 追加内容、上下文文件、技能清单都会接在它后面；实际发出去的必须正好是这样。
|
||||
if (session.systemPrompt !== `${task.systemPrompt}\nCurrent working directory: ${resolve(task.workDir)}\n`) {
|
||||
problems.push("系统提示词不是给的那一份加工作目录一行，被改动或追加了内容");
|
||||
}
|
||||
const actual = [...session.getActiveToolNames()].sort();
|
||||
if (actual.join(",") !== [...expected].sort().join(",")) {
|
||||
problems.push(`工具表是 [${actual.join(", ")}]，应为 [${[...expected].sort().join(", ")}]`);
|
||||
}
|
||||
// 自定义工具（含实验员换过环境的 bash）必须就是给的那个对象，没被同名的别的定义顶掉。
|
||||
for (const tool of customTools) {
|
||||
if (session.getToolDefinition(tool.name) !== tool) problems.push(`工具 ${tool.name} 不是给的实现`);
|
||||
}
|
||||
return problems;
|
||||
}
|
||||
@@ -0,0 +1,355 @@
|
||||
import { readdirSync, readFileSync, realpathSync, statSync } from "node:fs";
|
||||
import { isAbsolute, sep } from "node:path";
|
||||
import { parseJsonWithRepair, validateToolArguments, type AssistantMessageEvent } from "@earendil-works/pi-ai";
|
||||
import { createBashToolDefinition, defineTool, type ToolDefinition } from "@earendil-works/pi-coding-agent";
|
||||
import { Type, type TSchema } from "typebox";
|
||||
import { Id, SUBMISSION_KINDS, SUBMISSION_SCHEMAS, type Json, type Role } from "../vocab/index.ts";
|
||||
import type { SessionTask } from "./types.ts";
|
||||
|
||||
// 按角色发的工具（任务书 07 第三部分）。Pi 的 tools 是严格白名单，没列的工具连注册都不注册，
|
||||
// 所以"研究者没有 bash"靠的是名单里不写它；会话建好后 session.ts 再逐个核对一遍。
|
||||
|
||||
/** Pi 自带、只发给实验员的工具。 */
|
||||
const BUILTINS = ["read", "write", "edit", "bash", "grep", "find", "ls"] as const;
|
||||
|
||||
/** 每个角色应当拿到的工具名。会话建好后，实际工具表必须正好是这几个。 */
|
||||
export function toolNames(role: Role): string[] {
|
||||
switch (role) {
|
||||
case "researcher":
|
||||
return ["research_query", "read_source", "research_submit", "write_notes"];
|
||||
case "verifier":
|
||||
return ["research_query", "read_source", "research_submit"];
|
||||
case "experimenter":
|
||||
return ["research_query", "research_submit", "research_check", ...BUILTINS];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 回调出错时的处理，由 session.ts 给：记下错误、停下会话。
|
||||
* guard 包住一次回调，出错时处理完再把错误抛回 Pi；fail 只处理不抛，用在不能抛的地方。
|
||||
*/
|
||||
export interface Host {
|
||||
guard<T>(run: () => T): T;
|
||||
fail(error: unknown): void;
|
||||
}
|
||||
|
||||
/** 一次会话里提交工具的状态。 */
|
||||
export interface SubmitState {
|
||||
accepted: number;
|
||||
/** 参数原文不是合法 JSON 的提交调用：键是 Pi 解析出的参数对象，值是原文字数。 */
|
||||
malformed: WeakMap<object, number>;
|
||||
/** 正在流式输出的工具调用原文，按在回复里的位置累计，到 toolcall_end 取出删掉。 */
|
||||
streaming: Map<number, string>;
|
||||
/** 形状不合格、已交给提交回调记退回但还没写完的那几次。下一次提交和会话结束前都先等它们写完。 */
|
||||
pending: Promise<void>[];
|
||||
}
|
||||
|
||||
export function newSubmitState(): SubmitState {
|
||||
return { accepted: 0, malformed: new WeakMap(), streaming: new Map(), pending: [] };
|
||||
}
|
||||
|
||||
/**
|
||||
* 参数原文不是合法 JSON 时，Pi 0.85.1 不报错，而是退到 partial-json 只取能认出的前半截
|
||||
* （pi-ai utils/json-parse.js parseStreamingJson）。最常见的是正文里有没转义的半角双引号，
|
||||
* 后半段整段丢掉、形状照样合格（参考项目 15-real-1：一份收尾被截成 214 字收下）。
|
||||
* Pi 解析完就删了原文，但流式输出时每一片原文都以 toolcall_delta 发出来；这里按位置拼回，
|
||||
* 到 toolcall_end 用 Pi 自己的严格解析（只修控制字符和非法转义，不丢内容）再解析一次，失败就登记。
|
||||
* Pi 等会话监听处理完事件、整条回复结束后才执行工具，所以登记一定在提交之前。
|
||||
*/
|
||||
export function watchSubmitArguments(state: SubmitState, event: AssistantMessageEvent): void {
|
||||
if (event.type === "toolcall_start") state.streaming.set(event.contentIndex, "");
|
||||
else if (event.type === "toolcall_delta") {
|
||||
state.streaming.set(event.contentIndex, (state.streaming.get(event.contentIndex) ?? "") + event.delta);
|
||||
} else if (event.type === "toolcall_end") {
|
||||
const raw = state.streaming.get(event.contentIndex) ?? "";
|
||||
state.streaming.delete(event.contentIndex);
|
||||
if (event.toolCall.name !== "research_submit") return;
|
||||
try {
|
||||
parseJsonWithRepair(raw);
|
||||
} catch {
|
||||
state.malformed.set(event.toolCall.arguments, [...raw].length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 必填的文字、数字、真假值字段交了 null 的位置（形如 payload.responses.0.reply）。
|
||||
* Pi 0.85.1 校验前先按形状转换类型（pi-ai validateToolArguments 里的 Value.Convert）：文字位置上的 null
|
||||
* 变成字符串 "null"，数字变成 0，真假值变成 false，之后的形状检查照单收下（参考项目 14-real-1）。
|
||||
* 所以在转换之前查原始参数。可选字段交 null 时 Pi 当没填处理；本来就收 null 的（二选一里有 null）不算。
|
||||
*/
|
||||
const SCALARS = new Set(["string", "number", "integer", "boolean"]);
|
||||
export function nullFieldPaths(schema: TSchema, value: unknown, path: string, out: string[]): void {
|
||||
const s = schema as { type?: unknown; properties?: Record<string, TSchema>; required?: string[]; items?: TSchema; anyOf?: TSchema[] };
|
||||
if (s.anyOf) {
|
||||
// 二选一：null 由它自己的成员决定收不收；交了对象就按其中的对象成员往下查。
|
||||
const isObject = value !== null && typeof value === "object" && !Array.isArray(value);
|
||||
const member = isObject ? s.anyOf.find((m) => (m as { type?: unknown }).type === "object") : undefined;
|
||||
if (member) nullFieldPaths(member, value, path, out);
|
||||
} else if (typeof s.type === "string" && SCALARS.has(s.type)) {
|
||||
if (value === null) out.push(path);
|
||||
} else if (s.type === "array" && s.items && Array.isArray(value)) {
|
||||
value.forEach((item, i) => nullFieldPaths(s.items!, item, `${path}.${i}`, out));
|
||||
} else if (s.type === "object" && s.properties && typeof value === "object" && value !== null && !Array.isArray(value)) {
|
||||
const required = new Set(s.required ?? []);
|
||||
for (const [key, sub] of Object.entries(s.properties)) {
|
||||
const v = (value as Record<string, unknown>)[key];
|
||||
if (v === null && !required.has(key)) continue;
|
||||
nullFieldPaths(sub, v, path === "" ? key : `${path}.${key}`, out);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** 回执里回显结尾的字数。 */
|
||||
const TAIL_CHARS = 40;
|
||||
|
||||
/**
|
||||
* 提交回执回显程序收到了什么（方法.md 第五部分"提交回执"）：文字共多少字、最后一个文字字段的结尾。
|
||||
* 截断丢的都是参数的后半段，所以按模型写的顺序取最后一个文字字段，模型对照自己写的结尾就看得出。
|
||||
* 字数按 Unicode 码点数。
|
||||
*/
|
||||
export function receiptEcho(payload: Json): string {
|
||||
const leaves: Array<{ path: string; text: string }> = [];
|
||||
const walk = (value: Json, path: string): void => {
|
||||
if (typeof value === "string") leaves.push({ path, text: value });
|
||||
else if (Array.isArray(value)) value.forEach((v, i) => walk(v, `${path}[${i}]`));
|
||||
else if (value !== null && typeof value === "object") {
|
||||
for (const [k, v] of Object.entries(value)) walk(v, path === "" ? k : `${path}.${k}`);
|
||||
}
|
||||
};
|
||||
walk(payload, "");
|
||||
const last = leaves.at(-1);
|
||||
if (last === undefined) return "收到的内容里没有文字字段。";
|
||||
const total = leaves.reduce((n, l) => n + [...l.text].length, 0);
|
||||
const chars = [...last.text];
|
||||
const tail = chars.length <= TAIL_CHARS ? last.text : `……${chars.slice(-TAIL_CHARS).join("")}`;
|
||||
return `收到文字共 ${total} 字；最后一个文字字段 ${last.path} 共 ${chars.length} 字，结尾是：「${tail}」。跟你写的对一下，结尾不对说明没有完整收到。`;
|
||||
}
|
||||
|
||||
/** 读代码单次最多返回多少，与 Pi 自带 read 工具的上限一致（2000 行或 50KB，先到为准）。 */
|
||||
const READ_MAX_LINES = 2000;
|
||||
const READ_MAX_BYTES = 50 * 1024;
|
||||
|
||||
/**
|
||||
* 读一个文件或列一个目录。路径必须是绝对路径，按真实路径（解析 .. 和符号链接之后）判断在不在可读目录里：
|
||||
* 模型正常探索时就会去 ls 上一层，看到别人还没公开的东西。不合格就抛错，Pi 把错误回给模型。
|
||||
*/
|
||||
export function readSource(roots: readonly string[], path: string, offset: number): string {
|
||||
const allowed = roots.map((r) => realpathSync(r));
|
||||
const listing = allowed.map((r) => ` ${r}`).join("\n");
|
||||
if (!isAbsolute(path)) throw new Error(`read_source 只接受绝对路径，收到的是 "${path}"。可读的目录：\n${listing}`);
|
||||
let real: string;
|
||||
try {
|
||||
real = realpathSync(path);
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") throw new Error(`路径不存在：${path}`);
|
||||
throw error;
|
||||
}
|
||||
if (!allowed.some((root) => real === root || real.startsWith(root.endsWith(sep) ? root : root + sep))) {
|
||||
const via = real === path ? "" : `（真实路径是 ${real}）`;
|
||||
throw new Error(`不允许读取 ${path}${via}：它不在可读的目录里。可读的目录：\n${listing}`);
|
||||
}
|
||||
let lines: string[];
|
||||
let what: string;
|
||||
if (statSync(real).isDirectory()) {
|
||||
const entries = readdirSync(real, { withFileTypes: true }).map((e) => (e.isDirectory() ? `${e.name}/` : e.name)).sort();
|
||||
lines = entries.length > 0 ? entries : ["（空目录）"];
|
||||
what = `目录 ${path}，共 ${entries.length} 项`;
|
||||
} else {
|
||||
const text = readFileSync(real, "utf8");
|
||||
lines = text.split("\n");
|
||||
what = `文件 ${path}，共 ${lines.length} 行、${Buffer.byteLength(text)} 字节`;
|
||||
}
|
||||
if (offset > lines.length) throw new Error(`offset=${offset} 超出范围：${what}，offset 取 1 到 ${lines.length}。`);
|
||||
const shown: string[] = [];
|
||||
let bytes = 0;
|
||||
for (let i = offset - 1; i < lines.length && shown.length < READ_MAX_LINES; i++) {
|
||||
const line = lines[i] as string;
|
||||
bytes += Buffer.byteLength(line) + 1;
|
||||
if (bytes > READ_MAX_BYTES) {
|
||||
// 单独一行就超过上限：只给这一行的开头，并明说。
|
||||
if (shown.length === 0) {
|
||||
const head = Buffer.from(line).subarray(0, READ_MAX_BYTES).toString("utf8");
|
||||
return `${head}\n\n[截断：第 ${offset} 行超过 ${READ_MAX_BYTES} 字节，只显示了它的开头。${what}。]`;
|
||||
}
|
||||
break;
|
||||
}
|
||||
shown.push(line);
|
||||
}
|
||||
const last = offset + shown.length - 1;
|
||||
if (offset === 1 && last === lines.length) return shown.join("\n");
|
||||
const next = last < lines.length ? `要读后面，用 offset=${last + 1} 再调一次。` : "已读到结尾。";
|
||||
return `${shown.join("\n")}\n\n[截断：本次显示第 ${offset}–${last} 行（单次最多 ${READ_MAX_LINES} 行或 ${READ_MAX_BYTES} 字节）。${what}。${next}]`;
|
||||
}
|
||||
|
||||
/**
|
||||
* 实验员 shell 的环境变量：只带这几样，其余一律不给（插件进程的环境里可能有模型密钥）。
|
||||
* Pi 0.85.1 的 bash 工具先用 process.env 拼出环境（PATH 前面加上 Pi 自己的 bin 目录），
|
||||
* 再交给 spawnHook，用它的返回值去起进程（dist/core/tools/bash.js resolveSpawnContext）。
|
||||
*/
|
||||
const PASSED_THROUGH = ["PATH", "HOME", "LANG", "LC_ALL", "TMPDIR"] as const;
|
||||
export function shellEnvironment(base: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
|
||||
const env: NodeJS.ProcessEnv = {};
|
||||
for (const key of PASSED_THROUGH) if (base[key] !== undefined) env[key] = base[key];
|
||||
return env;
|
||||
}
|
||||
|
||||
function text(value: string) {
|
||||
return { content: [{ type: "text" as const, text: value }], details: {} };
|
||||
}
|
||||
|
||||
const OFFSET = Type.Optional(Type.Integer({ minimum: 1, description: "从第几行开始读，默认 1" }));
|
||||
|
||||
/** 这个任务的自定义工具（实验员还有换过环境的 bash）。Pi 自带的其余几个由 Pi 按白名单自己装。 */
|
||||
export function buildTools(task: SessionTask, state: SubmitState, host: Host): ToolDefinition[] {
|
||||
const tools: ToolDefinition[] = [];
|
||||
|
||||
tools.push(defineTool({
|
||||
name: "research_query",
|
||||
label: "查一条记录的全文",
|
||||
description:
|
||||
"本阶段材料已经在第一条消息里。要看材料里某条记录的全文，把它的编号填进 ref。本阶段看不到的编号会返回错误。",
|
||||
parameters: Type.Object({ ref: Type.String({ description: "记录的编号" }) }, { additionalProperties: false }),
|
||||
execute: async (_id, params) => {
|
||||
const found = host.guard(() => task.query(params.ref));
|
||||
if (found === null) {
|
||||
throw new Error(
|
||||
`本阶段看不到编号为 ${params.ref} 的记录。只能查本阶段材料里出现、而且本阶段看得到的记录；` +
|
||||
"材料里只写了编号、没带状态的，可能就是本阶段看不到的。",
|
||||
);
|
||||
}
|
||||
return text(found);
|
||||
},
|
||||
}));
|
||||
|
||||
// 只有研究者的提交能替换（方法.md 第五部分）；其余几种的参数里就没有 replaces。
|
||||
const replaceable = SUBMISSION_KINDS[task.kind].role === "researcher";
|
||||
const parameters = Type.Object(
|
||||
{
|
||||
payload: SUBMISSION_SCHEMAS[task.kind],
|
||||
...(replaceable
|
||||
? {
|
||||
replaces: Type.Optional({
|
||||
...Id,
|
||||
description:
|
||||
"替换你本阶段已经交过的一份时，写它的提交编号（收下时回给你的编号），只按最后一版算；" +
|
||||
"第一次交、不替换时整个字段都不要带",
|
||||
}),
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
{ additionalProperties: false },
|
||||
);
|
||||
/** 形状不合格：交给提交回调记一条退回（不等它写完，下一次提交前再等），再把原因抛给 Pi 回给模型。 */
|
||||
const refuse = (raw: unknown, reason: string): never => {
|
||||
const attempt = { shape: "invalid", raw: raw as Json, reason } as const;
|
||||
state.pending.push(task.submit(attempt).then(() => undefined, host.fail));
|
||||
throw new Error(reason);
|
||||
};
|
||||
tools.push(defineTool({
|
||||
name: "research_submit",
|
||||
label: "正式提交",
|
||||
description:
|
||||
"正式提交本任务的成果。只有调用这个工具才算提交；在回复里说\"我已提交\"不算数。payload 是内容。" +
|
||||
"文字里要加引号时用「」或中文引号“”，不要用半角双引号 \"：模型服务在没转义的半角双引号处会把文字截断。" +
|
||||
"程序会检查，不合格会返回原因，改了可以再交。" +
|
||||
(replaceable ? "阶段结束前要改已经收下的一份，用 replaces 写它的提交编号来替换。" : "交上的不能替换。"),
|
||||
parameters,
|
||||
// Pi 0.85.1 在形状校验之前调 prepareArguments（pi-agent-core agent-loop.js prepareToolCall），
|
||||
// 形状不合格时根本不调 execute，提交回调就收不到。所以在这里先查：参数原文不是合法 JSON、
|
||||
// 必填字段交了 null、Pi 自己的形状校验不通过，三种都交给提交回调记退回。
|
||||
prepareArguments: (args) => {
|
||||
const rawChars = state.malformed.get(args as object);
|
||||
if (rawChars !== undefined) {
|
||||
refuse(
|
||||
args,
|
||||
`research_submit 的参数原文（${rawChars} 字）不是合法的 JSON，程序只能认出前一部分，没有收下。` +
|
||||
"最常见的原因是正文里有没转义的半角双引号 \"：解析在那里就断了，后面的内容全部丢失。" +
|
||||
"整份重交：正文里的引号改用「」或中文引号“”，或写成 \\\"。",
|
||||
);
|
||||
}
|
||||
const nulls: string[] = [];
|
||||
nullFieldPaths(parameters, args, "", nulls);
|
||||
if (nulls.length > 0) {
|
||||
refuse(
|
||||
args,
|
||||
`research_submit 的这些必填字段交的是 null：${nulls.join("、")}。` +
|
||||
"这些字段要写实际内容（文字写文字，数字写数），不能交 null，文字也不能是空串或只有空白；改好后整份重交。",
|
||||
);
|
||||
}
|
||||
try {
|
||||
validateToolArguments(
|
||||
{ name: "research_submit", description: "", parameters },
|
||||
{ type: "toolCall", id: "", name: "research_submit", arguments: args as Record<string, unknown> },
|
||||
);
|
||||
} catch (error) {
|
||||
refuse(args, (error as Error).message);
|
||||
}
|
||||
return args as never;
|
||||
},
|
||||
// 同一次回复里调两次提交时，Pi 默认并行执行；声明 sequential 后整批按顺序执行，回调不会同时进来。
|
||||
executionMode: "sequential",
|
||||
execute: async (_id, params) => {
|
||||
await Promise.all(state.pending);
|
||||
const { payload, replaces } = params as { payload: Json; replaces?: string };
|
||||
const attempt = { shape: "valid", content: payload as never, replaces: replaces ?? null } as const;
|
||||
const receipt = await host.guard(() => task.submit(attempt));
|
||||
if (!receipt.accepted) throw new Error(`没有收下：${receipt.reason}`);
|
||||
state.accepted++;
|
||||
const how = replaceable ? "阶段公开前要改，用 replaces 写这个编号整份重交。" : "";
|
||||
return {
|
||||
...text(`已收下，提交编号 ${receipt.id}。${receiptEcho(payload)}${how}`),
|
||||
// 实验员、核验员交上就结束：同一批工具结果都带 terminate 时，Pi 不再发下一次请求。
|
||||
terminate: task.role !== "researcher",
|
||||
};
|
||||
},
|
||||
}));
|
||||
|
||||
if (task.role === "researcher" || task.role === "verifier") {
|
||||
const roots = task.readableDirs;
|
||||
tools.push(defineTool({
|
||||
name: "read_source",
|
||||
label: "读代码与资料",
|
||||
description:
|
||||
`读取代码或资料文件，或列出一个目录。path 用绝对路径，必须在这些目录里：${roots.join("、")}。` +
|
||||
`单次最多 ${READ_MAX_LINES} 行或 ${READ_MAX_BYTES} 字节，超出时会告诉你用 offset 从第几行接着读。`,
|
||||
parameters: Type.Object({ path: Type.String({ description: "绝对路径" }), offset: OFFSET }, { additionalProperties: false }),
|
||||
execute: async (_id, params) => text(readSource(roots, params.path, params.offset ?? 1)),
|
||||
}));
|
||||
}
|
||||
|
||||
if (task.role === "researcher") {
|
||||
const writeNote = task.writeNote;
|
||||
tools.push(defineTool({
|
||||
name: "write_notes",
|
||||
label: "写私人笔记",
|
||||
description:
|
||||
"把你的私人笔记整份覆盖写入（不是追加）。要改就把现有笔记连同修改一起完整写回。" +
|
||||
"笔记只有你自己看得到，不是正式提交：写在里面的计划和结论都不算数。",
|
||||
parameters: Type.Object({ content: Type.String({ pattern: "\\S", description: "笔记全文" }) }, { additionalProperties: false }),
|
||||
execute: async (_id, params) => {
|
||||
host.guard(() => writeNote(params.content));
|
||||
return text(`已写入笔记，共 ${[...params.content].length} 字。`);
|
||||
},
|
||||
}));
|
||||
}
|
||||
|
||||
if (task.role === "experimenter") {
|
||||
const lightCheck = task.lightCheck;
|
||||
tools.push(defineTool({
|
||||
name: "research_check",
|
||||
label: "轻量检查",
|
||||
description:
|
||||
"检查你当前工作目录里的代码能不能被正式实验接上（只查语法和接口形状，不执行求解）。" +
|
||||
"想确认代码能不能跑，用这个，不要自己在 shell 里跑完整求解。无参数；次数有上限，用完会报错。",
|
||||
parameters: Type.Object({}, { additionalProperties: false }),
|
||||
execute: async () => text(await host.guard(() => lightCheck())),
|
||||
}));
|
||||
tools.push(createBashToolDefinition(task.workDir, {
|
||||
// 不把 PI_SESSION_FILE 等会话信息放进环境：实验员不需要，会话文件路径也不该给它。
|
||||
exposeSessionEnvironment: false,
|
||||
spawnHook: (context) => ({ ...context, env: shellEnvironment(context.env) }),
|
||||
}) as unknown as ToolDefinition);
|
||||
}
|
||||
return tools;
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
import type { CreateAgentSessionOptions } from "@earendil-works/pi-coding-agent";
|
||||
import type { Json, ModelUsage, Role, Submission, SUBMISSION_KINDS, SubmissionKind } from "../vocab/index.ts";
|
||||
|
||||
// agents 对外的约定：flow 交给它什么、拿回什么（任务书 07 第二部分）。
|
||||
// agents 不碰账本、不决定给模型看什么、不判断交上来的东西好不好，这些都经下面的回调交给 flow。
|
||||
|
||||
/** Pi 0.85.1 的思考档位，与 config 的 THINKING_LEVELS 是同一份名单。 */
|
||||
export type ThinkingLevel = NonNullable<CreateAgentSessionOptions["thinkingLevel"]>;
|
||||
|
||||
/** 用哪个模型：provider、模型名与 Pi 的模型服务配置里的写法相同；思考档位必须写明。 */
|
||||
export interface RoleModel {
|
||||
provider: string;
|
||||
model: string;
|
||||
thinking: ThinkingLevel;
|
||||
}
|
||||
|
||||
/** 某个角色交的提交种类，取自 vocab 的 SUBMISSION_KINDS。 */
|
||||
export type KindOf<R extends Role> = {
|
||||
[K in SubmissionKind]: (typeof SUBMISSION_KINDS)[K]["role"] extends R ? K : never;
|
||||
}[SubmissionKind];
|
||||
|
||||
/**
|
||||
* 模型调一次提交工具，交给提交回调的东西：
|
||||
* - 形状合格：内容和"替换哪一份"（只有研究者的提交能替换，其余永远是 null）；
|
||||
* - 形状不合格：Pi 解析出的参数原样、退回原因。agents 已经把这个原因回给模型，
|
||||
* 回调只需记一条退回（方法.md 第三部分：被退回的提交也记），它交回的回执不再回给模型。
|
||||
*/
|
||||
export type SubmitAttempt<K extends SubmissionKind> =
|
||||
| { shape: "valid"; content: Submission<K>; replaces: string | null }
|
||||
| { shape: "invalid"; raw: Json; reason: string };
|
||||
|
||||
/** 提交回调的回答：收下和编号，或退回和原因。agents 原样回给模型。 */
|
||||
export type SubmitReceipt = { accepted: true; id: string } | { accepted: false; reason: string };
|
||||
|
||||
/** 三种角色共有的输入。回调抛出的错误按内部错误处理：会话立即停下，错误原样抛给调用方。 */
|
||||
interface Common<K extends SubmissionKind> {
|
||||
model: RoleModel;
|
||||
/** 角色提示词、第一条消息：materials 拼好的文字。 */
|
||||
systemPrompt: string;
|
||||
firstMessage: string;
|
||||
/** 会话的工作目录、Pi 会话文件存放的目录，都由调用方建好。 */
|
||||
workDir: string;
|
||||
sessionDir: string;
|
||||
/** 这次要交的提交种类，提交工具的参数形状就是 vocab 里它的字段定义。 */
|
||||
kind: K;
|
||||
/** 查一条材料：交回文字，或 null 表示看不到（不存在和看不到故意不区分）。 */
|
||||
query(ref: string): string | null;
|
||||
submit(attempt: SubmitAttempt<K>): Promise<SubmitReceipt>;
|
||||
/** 每次模型回复（包括出错重试的那几次、压缩上下文时写摘要的那次）交一份用量。 */
|
||||
usage(usage: ModelUsage): void;
|
||||
/** 用户中途停下研究时触发。 */
|
||||
stop: AbortSignal;
|
||||
}
|
||||
|
||||
/**
|
||||
* 跑一次会话的输入，按角色区分（任务书 07 第二、三部分）。
|
||||
* 研究者交上一份被收下后会话不结束：可能要交几份，或看了回执发现被截断要替换；实验员、核验员交上就结束。
|
||||
*/
|
||||
export type SessionTask =
|
||||
| (Common<KindOf<"researcher">> & {
|
||||
role: "researcher";
|
||||
/** 读代码工具能读的目录。 */
|
||||
readableDirs: readonly string[];
|
||||
/** 写私人笔记：整份覆盖，笔记是账本记录，由回调写。 */
|
||||
writeNote(body: string): void;
|
||||
})
|
||||
| (Common<KindOf<"verifier">> & { role: "verifier"; readableDirs: readonly string[] })
|
||||
| (Common<KindOf<"experimenter">> & {
|
||||
role: "experimenter";
|
||||
/** 轻量检查：flow 用 tasks 和 runner 接好，交回给模型看的结果文字。 */
|
||||
lightCheck(): Promise<string>;
|
||||
});
|
||||
|
||||
/**
|
||||
* 会话怎么结束的：
|
||||
* - submitted：至少有一份被收下；
|
||||
* - notSubmitted：模型自己停下了，一份都没被收下；
|
||||
* - failed：模型服务报错、子会话核对不通过等；
|
||||
* - stopped：用户停下。
|
||||
*/
|
||||
export type SessionEnding =
|
||||
| { status: "submitted" }
|
||||
| { status: "notSubmitted"; reason: string }
|
||||
| { status: "failed"; reason: string }
|
||||
| { status: "stopped" };
|
||||
|
||||
/** 会话的结局，带墙钟时间（秒）和会话文件路径；Pi 在第一条回复之后才写会话文件，没写出来是 null。 */
|
||||
export type SessionOutcome = SessionEnding & { wallSeconds: number; sessionFile: string | null };
|
||||
@@ -0,0 +1,145 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { runSession, type SessionOutcome, type SessionTask, type SubmitAttempt } from "../src/agents/index.ts";
|
||||
import type { ModelUsage } from "../src/vocab/index.ts";
|
||||
import { fullConfig } from "./config-helpers.ts";
|
||||
|
||||
// agents 的真实模型测试（任务书 07 第六部分），npm run test:pi：要花钱、要联网，不放进 npm test。
|
||||
// 用本机真实的 ~/.pi/agent（里面有扩展、技能、插件包），模型取测试配置里的 glm-5.3。
|
||||
// 每次会话的结局、每次回复的用量、墙钟时间都完整打印；会话目录留在临时目录里，路径也打印出来。
|
||||
|
||||
const models = fullConfig().models;
|
||||
const root = mkdtempSync(join(tmpdir(), "inforsi-agents-pi-"));
|
||||
console.log(`临时目录：${root}`);
|
||||
const agentDir = join(homedir(), ".pi", "agent");
|
||||
for (const sub of ["extensions", "skills"]) {
|
||||
const dir = join(agentDir, sub);
|
||||
console.log(`本机 ${dir}：${existsSync(dir) ? readdirSync(dir).join(", ") : "（没有）"}`);
|
||||
}
|
||||
const settings = JSON.parse(readFileSync(join(agentDir, "settings.json"), "utf8")) as { packages?: unknown[]; compaction?: unknown };
|
||||
console.log(`本机 settings.json：packages ${JSON.stringify(settings.packages)}；compaction ${JSON.stringify(settings.compaction)}`);
|
||||
|
||||
function dirs(name: string): { workDir: string; sessionDir: string } {
|
||||
const workDir = join(root, name, "work");
|
||||
const sessionDir = join(root, name, "sessions");
|
||||
mkdirSync(workDir, { recursive: true });
|
||||
mkdirSync(sessionDir, { recursive: true });
|
||||
return { workDir, sessionDir };
|
||||
}
|
||||
|
||||
function show(usage: ModelUsage): string {
|
||||
const v = (m: ModelUsage["inputTokens"]) => (m.status === "measured" ? String(m.value) : `测不到（${m.reason}）`);
|
||||
return `输入 ${v(usage.inputTokens)}，缓存读 ${v(usage.cacheReadTokens)}，缓存写 ${v(usage.cacheWriteTokens)}，输出 ${v(usage.outputTokens)}，价格 ${v(usage.cost)}`;
|
||||
}
|
||||
|
||||
/** 会话文件里助手回复的条数（Pi 的会话文件每行一条 JSON）。 */
|
||||
function assistantReplies(sessionFile: string): number {
|
||||
return readFileSync(sessionFile, "utf8")
|
||||
.split("\n")
|
||||
.filter((line) => line.trim() !== "")
|
||||
.map((line) => JSON.parse(line) as { type?: string; message?: { role?: string } })
|
||||
.filter((entry) => entry.type === "message" && entry.message?.role === "assistant").length;
|
||||
}
|
||||
|
||||
function report(name: string, outcome: SessionOutcome, usages: ModelUsage[], attempts: string[]): void {
|
||||
console.log(`\n===== ${name} =====`);
|
||||
console.log(`结局：${JSON.stringify(outcome)}`);
|
||||
console.log(`墙钟时间：${outcome.wallSeconds.toFixed(1)} 秒`);
|
||||
attempts.forEach((a, i) => console.log(`提交回调第 ${i + 1} 次：${a}`));
|
||||
usages.forEach((u, i) => console.log(`第 ${i + 1} 次回复的用量：${show(u)}`));
|
||||
assert.ok(outcome.sessionFile !== null, "会话文件写出来了");
|
||||
const replies = assistantReplies(outcome.sessionFile);
|
||||
console.log(`会话文件里助手回复 ${replies} 条，用量回调 ${usages.length} 次`);
|
||||
assert.equal(usages.length, replies, "每次回复都交了一份用量");
|
||||
assert.ok(usages.every((u) => u.inputTokens.status === "measured" && u.outputTokens.status === "measured"), "模型服务每次都报了用量");
|
||||
}
|
||||
|
||||
const stop = new AbortController().signal;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 研究者：交一份收尾，第一次故意退回，改了再交，收下
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
{
|
||||
const { workDir, sessionDir } = dirs("researcher");
|
||||
const readable = join(root, "researcher", "task");
|
||||
mkdirSync(readable);
|
||||
writeFileSync(join(readable, "README.md"), "填圆任务：在单位正方形里放 26 个圆，半径之和越大越好。\n");
|
||||
const usages: ModelUsage[] = [];
|
||||
const attempts: string[] = [];
|
||||
const task: SessionTask = {
|
||||
role: "researcher",
|
||||
model: models.researchers[0]!,
|
||||
systemPrompt: "你是一个研究小组里的研究者。你只能用给你的工具读资料和提交；只有调用 research_submit 才算交上了。",
|
||||
firstMessage: [
|
||||
"这是一次接口演练。请交一份收尾（research_submit，payload 里只有 body 一个字段）。",
|
||||
`body 写两三句话，说说这一轮的情况：可以先用 read_source 读一下 ${readable}/README.md 了解任务。`,
|
||||
"程序如果退回，按退回原因改好再交。被收下以后就结束，不用再做别的。",
|
||||
].join("\n"),
|
||||
...{ workDir, sessionDir },
|
||||
kind: "wrapUp",
|
||||
readableDirs: [readable],
|
||||
query: () => null,
|
||||
writeNote: (body) => attempts.push(`（写笔记 ${body.length} 字）`),
|
||||
usage: (u) => usages.push(u),
|
||||
stop,
|
||||
submit: async (attempt: SubmitAttempt<"wrapUp">) => {
|
||||
attempts.push(JSON.stringify(attempt));
|
||||
if (attempt.shape === "invalid") return { accepted: false, reason: attempt.reason };
|
||||
if (attempts.filter((a) => a.includes('"shape":"valid"')).length === 1) {
|
||||
return { accepted: false, reason: "收尾的最后一行必须是「下一步：继续在 p1 上试」，原样写上后整份重交" };
|
||||
}
|
||||
return { accepted: true, id: "s2" };
|
||||
},
|
||||
};
|
||||
const outcome = await runSession(task);
|
||||
report("研究者：先退回再收下", outcome, usages, attempts);
|
||||
assert.equal(outcome.status, "submitted");
|
||||
assert.ok(attempts.filter((a) => a.includes('"shape":"valid"')).length >= 2, "至少交了两次：一次被退回，一次收下");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 实验员：在工作目录里写一个文件、跑一条 shell 命令，再交
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
{
|
||||
const { workDir, sessionDir } = dirs("experimenter");
|
||||
const usages: ModelUsage[] = [];
|
||||
const attempts: string[] = [];
|
||||
let checks = 0;
|
||||
const task: SessionTask = {
|
||||
role: "experimenter",
|
||||
model: models.experimenter,
|
||||
systemPrompt: "你是一个研究小组里的实验员，在自己的工作目录里写代码、跑代码。只有调用 research_submit 才算交上了。",
|
||||
firstMessage: [
|
||||
"这是一次接口演练，按顺序做三件事：",
|
||||
"1. 在工作目录里写一个 hello.py，运行时打印 hello inforsi。",
|
||||
"2. 用 bash 运行 python3 hello.py，确认输出。",
|
||||
"3. 用 research_submit 提交：payload 里 alsoOff 交空数组，cannotRun 交 null，concerns 写一句你运行时看到的输出。",
|
||||
].join("\n"),
|
||||
...{ workDir, sessionDir },
|
||||
kind: "ablationImplementation",
|
||||
query: () => null,
|
||||
lightCheck: async () => `第 ${++checks} 次轻量检查：通过`,
|
||||
usage: (u) => usages.push(u),
|
||||
stop,
|
||||
submit: async (attempt: SubmitAttempt<"ablationImplementation">) => {
|
||||
attempts.push(JSON.stringify(attempt));
|
||||
if (attempt.shape === "invalid") return { accepted: false, reason: attempt.reason };
|
||||
return existsSync(join(workDir, "hello.py"))
|
||||
? { accepted: true, id: "s1" }
|
||||
: { accepted: false, reason: "工作目录里没有 hello.py" };
|
||||
},
|
||||
};
|
||||
const outcome = await runSession(task);
|
||||
report("实验员：写文件、跑 shell、再交", outcome, usages, attempts);
|
||||
assert.equal(outcome.status, "submitted");
|
||||
console.log(`hello.py：${JSON.stringify(readFileSync(join(workDir, "hello.py"), "utf8"))}`);
|
||||
const log = readFileSync(outcome.sessionFile!, "utf8");
|
||||
assert.match(log, /"name":"bash"/, "会话里调过 bash");
|
||||
assert.match(log, /hello inforsi/, "bash 的输出进了会话");
|
||||
}
|
||||
|
||||
console.log("\n全部通过：两次会话的子会话核对都通过（没有加载扩展、技能、提示模板和上下文文件），都交上了，每次回复都有用量。");
|
||||
@@ -0,0 +1,293 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { test, type TestContext } from "node:test";
|
||||
import { parseStreamingJson, validateToolArguments, type AssistantMessageEvent } from "@earendil-works/pi-ai";
|
||||
import type { ToolDefinition } from "@earendil-works/pi-coding-agent";
|
||||
import { buildTools, newSubmitState, readSource, toolNames, watchSubmitArguments, type Host, type SubmitState } from "../src/agents/tools.ts";
|
||||
import { runSession, type KindOf, type SessionTask, type SubmitAttempt } from "../src/agents/index.ts";
|
||||
import type { ModelUsage } from "../src/vocab/index.ts";
|
||||
|
||||
// agents 不花钱的测试（任务书 07 第六部分）：提交参数的两处特殊处理、读代码的目录限制、
|
||||
// 实验员 shell 的环境变量、三种角色的工具表。真实模型链路由 npm run test:pi 证明。
|
||||
// 这里直接用 tools.ts 搭工具，回调是测试自己造的任务对象，不经 Pi 会话。
|
||||
|
||||
function tempDir(t: TestContext, name: string): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), `inforsi-agents-${name}-`));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
return dir;
|
||||
}
|
||||
|
||||
/** 回调出错直接抛，测试里不应发生。 */
|
||||
const HOST: Host = {
|
||||
guard: (run) => run(),
|
||||
fail: (error) => {
|
||||
throw error;
|
||||
},
|
||||
};
|
||||
|
||||
const signal = new AbortController().signal;
|
||||
const common = {
|
||||
model: { provider: "p", model: "m", thinking: "high" },
|
||||
systemPrompt: "角色提示词",
|
||||
firstMessage: "第一条消息",
|
||||
sessionDir: "/nonexistent-sessions",
|
||||
query: () => null,
|
||||
usage: () => {},
|
||||
stop: signal,
|
||||
} as const;
|
||||
|
||||
/** 造一个任务，记下每次提交回调收到了什么；回调一律收下，编号 s1、s2…… */
|
||||
function setup(task: SessionTask): { tools: ToolDefinition[]; state: SubmitState; attempts: Array<SubmitAttempt<never>> } {
|
||||
const attempts: Array<SubmitAttempt<never>> = [];
|
||||
const recorded = {
|
||||
...task,
|
||||
submit: async (attempt: SubmitAttempt<never>) => {
|
||||
attempts.push(attempt);
|
||||
return { accepted: true as const, id: `s${attempts.length}` };
|
||||
},
|
||||
} as SessionTask;
|
||||
const state = newSubmitState();
|
||||
return { tools: buildTools(recorded, state, HOST), state, attempts };
|
||||
}
|
||||
|
||||
function researcher(kind: "wrapUp" | "firstPrediction", workDir: string, readableDirs: string[]): SessionTask {
|
||||
return { ...common, role: "researcher", kind, workDir, readableDirs, writeNote: () => {}, submit: async () => ({ accepted: true, id: "x" }) };
|
||||
}
|
||||
|
||||
function tool(tools: ToolDefinition[], name: string): ToolDefinition {
|
||||
const found = tools.find((t) => t.name === name);
|
||||
assert.ok(found, `没有工具 ${name}`);
|
||||
return found;
|
||||
}
|
||||
|
||||
/** 按 Pi 流式输出的样子把一段参数原文喂给监听：原文 40 字一片，结尾的参数对象用 Pi 自己的解析函数得到。 */
|
||||
function stream(state: SubmitState, raw: string): Record<string, unknown> {
|
||||
const args = parseStreamingJson(raw) as Record<string, unknown>;
|
||||
const events: unknown[] = [{ type: "toolcall_start", contentIndex: 0 }];
|
||||
for (let i = 0; i < raw.length; i += 40) events.push({ type: "toolcall_delta", contentIndex: 0, delta: raw.slice(i, i + 40) });
|
||||
events.push({ type: "toolcall_end", contentIndex: 0, toolCall: { type: "toolCall", id: "c1", name: "research_submit", arguments: args } });
|
||||
for (const event of events) watchSubmitArguments(state, event as AssistantMessageEvent);
|
||||
return args;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 提交参数原文不是合法 JSON（任务书第四部分第 1 条，参考项目 15-real-1、提交 bcc422e）
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("提交参数原文里半角双引号没转义：退回并说明原因，不收前半截；replaces 在前、在后都一样", async (t) => {
|
||||
const head = "## 什么被支持了\n因子 p1 的效果被确认。";
|
||||
const rest = '下一步想试 "更紧的排列" 这个说法，然后继续。';
|
||||
// 模型正确转义的部分按 JSON 规则写；后半段的半角双引号没转义。
|
||||
const esc = (s: string) => JSON.stringify(s).slice(1, -1);
|
||||
const raws = {
|
||||
replacesFirst: `{"replaces": "s7", "payload": {"body": "${esc(head)}${rest}"}}`,
|
||||
payloadFirst: `{"payload": {"body": "${esc(head)}${rest}"}, "replaces": "s7"}`,
|
||||
};
|
||||
for (const [name, raw] of Object.entries(raws)) {
|
||||
assert.throws(() => JSON.parse(raw), SyntaxError, `${name}：原文本身就不是合法 JSON`);
|
||||
const { tools, state, attempts } = setup(researcher("wrapUp", tempDir(t, "work"), []));
|
||||
const submit = tool(tools, "research_submit");
|
||||
const args = stream(state, raw);
|
||||
if (name === "replacesFirst") {
|
||||
// 不拦的话：Pi 解析出的前半截形状照样合格，正文停在那个引号前。
|
||||
const passed = validateToolArguments(
|
||||
{ name: "research_submit", description: "", parameters: submit.parameters },
|
||||
{ type: "toolCall", id: "", name: "research_submit", arguments: args },
|
||||
) as { payload: { body: string } };
|
||||
assert.equal(passed.payload.body, `${head}下一步想试 `);
|
||||
}
|
||||
assert.throws(() => submit.prepareArguments!(args), /不是合法的 JSON[\s\S]*半角双引号/, name);
|
||||
await Promise.all(state.pending);
|
||||
assert.equal(attempts.length, 1, `${name}：交给提交回调记一条退回`);
|
||||
const attempt = attempts[0]!;
|
||||
assert.equal(attempt.shape, "invalid");
|
||||
assert.ok(attempt.shape === "invalid" && /不是合法的 JSON/.test(attempt.reason));
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* 本机假模型服务：OpenAI 兼容的流式接口，第一次回复一个提交调用（参数原文按 7 字一片发出），之后回一句话。
|
||||
* Pi 经测试自己的 PI_CODING_AGENT_DIR（里面的 models.json 指向这个服务）连过来，不碰本机的 ~/.pi/agent。
|
||||
*/
|
||||
async function fakeModel(t: TestContext, rawArguments: string): Promise<void> {
|
||||
let replies = 0;
|
||||
const server = createServer((req, res) => {
|
||||
req.resume();
|
||||
req.on("end", () => {
|
||||
const send = (delta: unknown, finish: string | null) =>
|
||||
res.write(`data: ${JSON.stringify({ id: "x", object: "chat.completion.chunk", created: 1, model: "m", choices: [{ index: 0, delta, finish_reason: finish }] })}\n\n`);
|
||||
res.writeHead(200, { "content-type": "text/event-stream" });
|
||||
if (replies++ === 0) {
|
||||
send({ role: "assistant", tool_calls: [{ index: 0, id: "call-1", type: "function", function: { name: "research_submit", arguments: "" } }] }, null);
|
||||
for (let i = 0; i < rawArguments.length; i += 7) send({ tool_calls: [{ index: 0, function: { arguments: rawArguments.slice(i, i + 7) } }] }, null);
|
||||
send({}, "tool_calls");
|
||||
} else {
|
||||
send({ role: "assistant", content: "知道了。" }, null);
|
||||
send({}, "stop");
|
||||
}
|
||||
res.write(`data: ${JSON.stringify({ id: "x", object: "chat.completion.chunk", created: 1, model: "m", choices: [], usage: { prompt_tokens: 100, completion_tokens: 5 } })}\n\n`);
|
||||
res.end("data: [DONE]\n\n");
|
||||
});
|
||||
});
|
||||
await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));
|
||||
t.after(() => server.close());
|
||||
const agentDir = tempDir(t, "pi-agent");
|
||||
const model = { id: "m", reasoning: true, input: ["text"], contextWindow: 100000, maxTokens: 1000, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 } };
|
||||
const baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1`;
|
||||
writeFileSync(join(agentDir, "models.json"), JSON.stringify({ providers: { fake: { baseUrl, api: "openai-completions", apiKey: "k", models: [model] } } }));
|
||||
writeFileSync(join(agentDir, "auth.json"), "{}");
|
||||
process.env.PI_CODING_AGENT_DIR = agentDir;
|
||||
t.after(() => delete process.env.PI_CODING_AGENT_DIR);
|
||||
}
|
||||
|
||||
test("经过真实的 Pi 会话：参数原文不是合法 JSON 的提交交给提交回调记退回，原因回给模型", async (t) => {
|
||||
await fakeModel(t, '{"payload": {"body": "前半段，然后"引号"后半段"}}');
|
||||
const work = tempDir(t, "session");
|
||||
mkdirSync(join(work, "sessions"));
|
||||
const attempts: Array<SubmitAttempt<KindOf<"researcher">>> = [];
|
||||
const usages: ModelUsage[] = [];
|
||||
const outcome = await runSession({
|
||||
...common,
|
||||
model: { provider: "fake", model: "m", thinking: "medium" },
|
||||
role: "researcher",
|
||||
kind: "wrapUp",
|
||||
workDir: work,
|
||||
sessionDir: join(work, "sessions"),
|
||||
readableDirs: [],
|
||||
writeNote: () => {},
|
||||
usage: (u) => usages.push(u),
|
||||
submit: async (attempt) => {
|
||||
attempts.push(attempt);
|
||||
return { accepted: true, id: "s1" };
|
||||
},
|
||||
});
|
||||
assert.equal(outcome.status, "notSubmitted", JSON.stringify(outcome));
|
||||
assert.equal(attempts.length, 1);
|
||||
assert.ok(attempts[0]!.shape === "invalid" && /不是合法的 JSON/.test(attempts[0]!.reason), JSON.stringify(attempts));
|
||||
assert.deepEqual(attempts[0]!.shape === "invalid" && attempts[0]!.raw, { payload: { body: "前半段，然后" } });
|
||||
assert.equal(usages.length, 2, "两次回复各一份用量");
|
||||
assert.ok(usages.every((u) => u.inputTokens.status === "measured" && u.inputTokens.value === 100));
|
||||
});
|
||||
|
||||
test("参数原文是合法 JSON 时照常往下走，不登记", (t) => {
|
||||
const { tools, state, attempts } = setup(researcher("wrapUp", tempDir(t, "work"), []));
|
||||
const args = stream(state, JSON.stringify({ payload: { body: '正文里有 "引号"，正确转义了' } }));
|
||||
assert.equal(tool(tools, "research_submit").prepareArguments!(args), args);
|
||||
assert.equal(attempts.length, 0);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 必填字段交 null（任务书第四部分第 2 条，参考项目 14-real-1、提交 fd73b78）
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("必填的数字、文字交 null：退回并报出字段位置，不让 Pi 转成 0 或 \"null\"", async (t) => {
|
||||
const { tools, state, attempts } = setup(researcher("firstPrediction", tempDir(t, "work"), []));
|
||||
const submit = tool(tools, "research_submit");
|
||||
const args = { payload: { range: { q1: 0.1, q5: null, q9: 0.3 }, body: null } };
|
||||
assert.throws(() => submit.prepareArguments!(args), /交的是 null：payload\.range\.q5、payload\.body。/);
|
||||
await Promise.all(state.pending);
|
||||
assert.equal(attempts.length, 1);
|
||||
assert.ok(attempts[0]!.shape === "invalid" && attempts[0]!.reason.includes("payload.range.q5、payload.body"));
|
||||
});
|
||||
|
||||
test("二选一里本来收 null 的字段交 null 照常往下走；Pi 的形状校验不通过也交给提交回调", async (t) => {
|
||||
const work = tempDir(t, "work");
|
||||
const task: SessionTask = { ...common, role: "experimenter", kind: "ablationImplementation", workDir: work, lightCheck: async () => "", submit: async () => ({ accepted: true, id: "x" }) };
|
||||
const { tools, state, attempts } = setup(task);
|
||||
const submit = tool(tools, "research_submit");
|
||||
const ok = { payload: { alsoOff: [], cannotRun: null, concerns: "无" } };
|
||||
assert.equal(submit.prepareArguments!(ok), ok);
|
||||
assert.throws(() => submit.prepareArguments!({ payload: { alsoOff: ["x1"], cannotRun: null, concerns: "无" } }), /alsoOff/);
|
||||
// 实验员的提交不能替换：参数里根本没有 replaces。
|
||||
assert.throws(() => submit.prepareArguments!({ ...ok, replaces: "s1" }), /replaces|additional/i);
|
||||
await Promise.all(state.pending);
|
||||
assert.deepEqual(attempts.map((a) => a.shape), ["invalid", "invalid"]);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 读代码：只能读给定目录里的东西
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("读代码：目录里的文件和目录读得到；目录外的读不到，用 .. 和符号链接绕出去的也读不到", (t) => {
|
||||
const base = realpathSync(tempDir(t, "read"));
|
||||
const root = join(base, "task");
|
||||
const secret = join(base, "other-researcher");
|
||||
mkdirSync(join(root, "src"), { recursive: true });
|
||||
mkdirSync(secret);
|
||||
writeFileSync(join(root, "src", "a.py"), "line1\nline2\n");
|
||||
writeFileSync(join(secret, "proposal.md"), "还没公开的提案");
|
||||
symlinkSync(secret, join(root, "link-dir"));
|
||||
symlinkSync(join(secret, "proposal.md"), join(root, "link-file"));
|
||||
|
||||
assert.equal(readSource([root], join(root, "src", "a.py"), 1), "line1\nline2\n");
|
||||
assert.equal(readSource([root], root, 1), "link-dir\nlink-file\nsrc/");
|
||||
for (const path of [base, secret, join(secret, "proposal.md"), join(root, "..", "other-researcher", "proposal.md"), join(root, "link-dir"), join(root, "link-file"), join(root, "link-dir", "proposal.md")]) {
|
||||
assert.throws(() => readSource([root], path, 1), /不在可读的目录里/, path);
|
||||
}
|
||||
assert.throws(() => readSource([root], "src/a.py", 1), /只接受绝对路径/);
|
||||
// 前缀相同但不是子目录的也不算在里面。
|
||||
mkdirSync(`${root}-copy`);
|
||||
assert.throws(() => readSource([root], `${root}-copy`, 1), /不在可读的目录里/);
|
||||
});
|
||||
|
||||
test("读代码：超过单次上限时分段，并写明从第几行接着读", (t) => {
|
||||
const root = realpathSync(tempDir(t, "page"));
|
||||
const lines = Array.from({ length: 2500 }, (_, i) => `第 ${i + 1} 行`);
|
||||
writeFileSync(join(root, "big.txt"), lines.join("\n"));
|
||||
const first = readSource([root], join(root, "big.txt"), 1);
|
||||
assert.ok(first.startsWith("第 1 行\n"));
|
||||
assert.match(first, /本次显示第 1–2000 行[\s\S]*用 offset=2001 再调一次/);
|
||||
const second = readSource([root], join(root, "big.txt"), 2001);
|
||||
assert.match(second, /^第 2001 行\n/);
|
||||
assert.match(second, /本次显示第 2001–2500 行[\s\S]*已读到结尾/);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 实验员的 bash：环境变量只放 PATH、HOME、LANG、LC_ALL、TMPDIR
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("实验员 shell 里看不到插件进程环境里的密钥，也没有 Pi 的会话信息", async (t) => {
|
||||
const work = tempDir(t, "bash");
|
||||
process.env.INFORSI_FAKE_API_KEY = "sk-inforsi-fake-0123456789";
|
||||
t.after(() => delete process.env.INFORSI_FAKE_API_KEY);
|
||||
const task: SessionTask = { ...common, role: "experimenter", kind: "implementation", workDir: work, lightCheck: async () => "", submit: async () => ({ accepted: true, id: "x" }) };
|
||||
const bash = tool(setup(task).tools, "bash");
|
||||
const result = await bash.execute("call-1", { command: "env; pwd" }, signal, () => {}, undefined as never);
|
||||
const output = result.content.map((c) => (c.type === "text" ? c.text : "")).join("");
|
||||
assert.ok(!output.includes("sk-inforsi-fake-0123456789"), output);
|
||||
const names = output.split("\n").filter((l) => /^[A-Za-z_][A-Za-z0-9_]*=/.test(l)).map((l) => l.slice(0, l.indexOf("=")));
|
||||
const allowed = new Set(["PATH", "HOME", "LANG", "LC_ALL", "TMPDIR", "PWD", "SHLVL", "_", "OLDPWD"]);
|
||||
assert.deepEqual(names.filter((n) => !allowed.has(n)), [], `shell 里多出的变量：${names.join(", ")}`);
|
||||
assert.ok(names.includes("PATH"));
|
||||
assert.ok(output.includes(realpathSync(work)), "命令在工作目录里跑");
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 工具表
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test("三种角色各自拿到的工具名正好是任务书第三部分那几个", (t) => {
|
||||
const work = tempDir(t, "tools");
|
||||
const expected = {
|
||||
researcher: ["read_source", "research_query", "research_submit", "write_notes"],
|
||||
verifier: ["read_source", "research_query", "research_submit"],
|
||||
experimenter: ["bash", "edit", "find", "grep", "ls", "read", "research_check", "research_query", "research_submit", "write"],
|
||||
};
|
||||
const tasks: SessionTask[] = [
|
||||
researcher("wrapUp", work, [work]),
|
||||
{ ...common, role: "verifier", kind: "verification", workDir: work, readableDirs: [work], submit: async () => ({ accepted: true, id: "x" }) },
|
||||
{ ...common, role: "experimenter", kind: "implementation", workDir: work, lightCheck: async () => "", submit: async () => ({ accepted: true, id: "x" }) },
|
||||
];
|
||||
for (const task of tasks) {
|
||||
assert.deepEqual([...toolNames(task.role)].sort(), expected[task.role], task.role);
|
||||
// 自定义工具都在名单里；名单里其余的是 Pi 自带、由 Pi 按白名单装的那几个，只有实验员有。
|
||||
const custom = setup(task).tools.map((t) => t.name);
|
||||
assert.ok(custom.every((name) => toolNames(task.role).includes(name)), task.role);
|
||||
const builtins = toolNames(task.role).filter((name) => !custom.includes(name)).sort();
|
||||
assert.deepEqual(builtins, task.role === "experimenter" ? ["edit", "find", "grep", "ls", "read", "write"] : [], task.role);
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user