iomgaa
11107f5758
方案 A 平铺模块：prompt.py / skills.py / tools.py / vision.py 新增 OCRProvider Protocol + adapters/ocr.py Prompt 从 TRM4 store/prompts/v2/ 原封不动复制
314 lines
10 KiB
Markdown
314 lines
10 KiB
Markdown
---
|
||
type: design
|
||
node_id: design:2026-07-07-search-module-design
|
||
title: "搜索 Agent 装配层设计（app/search/）"
|
||
date: 2026-07-07
|
||
---
|
||
|
||
# 搜索 Agent 装配层设计（app/search/）
|
||
|
||
**日期** 2026-07-07 · **状态** 已批准 · **关联** TRM4 `core/search/` + `core/tree/tools.py` + `core/tree/vision.py`
|
||
|
||
---
|
||
|
||
## §1 定位
|
||
|
||
`app/search/` 是搜索 Agent 的"装配层"——为 `core/agent/loop.py` AgentLoop 提供 **prompt 组装**、**skill 管理**、**工具定义/分发** 和 **视觉观察**。它不控制推理循环，只被 AgentLoop 调用；编排责任在 `app/harness/inference.py`。
|
||
|
||
### 与 TRM4 的映射
|
||
|
||
| TRM4 | TRM5 | 变更类型 |
|
||
|------|------|---------|
|
||
| `core/search/prompt.py` | `app/search/prompt.py` | 保真迁移 + P4 显式参数 |
|
||
| `core/search/skills.py` | `app/search/skills.py` | 保真迁移 |
|
||
| `core/tree/tools.py` | `app/search/tools.py` | 重组为 `SearchToolDispatcher` 类 |
|
||
| `core/tree/vision.py` | `app/search/vision.py` | 异步化 + Protocol 注入 |
|
||
| `core/tree/ocr.py` | `adapters/ocr.py` | 异步化 + OCRProvider Protocol |
|
||
|
||
---
|
||
|
||
## §2 模块结构
|
||
|
||
```
|
||
app/search/
|
||
├── __init__.py # 公开 API 重导出
|
||
├── prompt.py # PromptManager — prompt 加载与拼装
|
||
├── skills.py # SkillRegistry + discover_skills — skill 扫描与注册
|
||
├── tools.py # SearchToolDispatcher（实现 ToolDispatcher Protocol）
|
||
└── vision.py # observe_frame（VLM 两轮 + OCR 注入）
|
||
|
||
adapters/
|
||
└── ocr.py # MonkeyOCRClient（实现 OCRProvider Protocol）
|
||
|
||
core/protocols.py # 新增 OCRProvider Protocol
|
||
|
||
store/prompts/ # 初始种子（从 TRM4 v2 直接复制，不修改）
|
||
├── system.md
|
||
├── observe_frame_extract.md
|
||
└── observe_frame_verify.md
|
||
```
|
||
|
||
---
|
||
|
||
## §3 依赖方向
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph adapters
|
||
OCR_IMPL["adapters/ocr.py\nMonkeyOCRClient"]
|
||
end
|
||
|
||
subgraph core
|
||
PROTO["core/protocols.py\nOCRProvider Protocol"]
|
||
AGENT_PROTO["core/agent/protocols.py\nToolDispatcher Protocol"]
|
||
end
|
||
|
||
subgraph app/search
|
||
PROMPT["prompt.py\nPromptManager"]
|
||
SKILLS["skills.py\nSkillRegistry"]
|
||
TOOLS["tools.py\nSearchToolDispatcher"]
|
||
VISION["vision.py\nobserve_frame"]
|
||
end
|
||
|
||
subgraph app/tree
|
||
ENV["environment.py\nTreeEnvironment"]
|
||
end
|
||
|
||
OCR_IMPL -->|实现| PROTO
|
||
TOOLS -->|实现| AGENT_PROTO
|
||
TOOLS --> ENV
|
||
TOOLS --> SKILLS
|
||
TOOLS --> VISION
|
||
VISION -->|依赖| PROTO
|
||
PROMPT --> SKILLS
|
||
PROMPT -.->|读取| STORE["store/prompts/*.md"]
|
||
```
|
||
|
||
依赖只向内或同层，`core/` 不认识 `app/search/`。
|
||
|
||
---
|
||
|
||
## §4 公开 API
|
||
|
||
### 4.1 PromptManager（prompt.py）
|
||
|
||
```python
|
||
class PromptManager:
|
||
def __init__(self, prompts_dir: Path) -> None: ...
|
||
def build_inference_prompt(
|
||
self,
|
||
skill_mode: str,
|
||
task_type: str,
|
||
always_skills_text: str,
|
||
task_skill_map: dict[str, str],
|
||
catalog_text: str,
|
||
) -> str: ...
|
||
def format_user_prompt(
|
||
self,
|
||
question: str,
|
||
options: list[str],
|
||
l1_node_ids: list[str],
|
||
task_type: str | None = None,
|
||
) -> str: ...
|
||
def load(self, name: str) -> str: ...
|
||
```
|
||
|
||
**与 TRM4 有意变更**：
|
||
- 工具描述从 `app/search/tools.py` 的 `get_tool_descriptions()` 获取（职责归属修正）
|
||
- `format_user_prompt` 参数从 `dict` 改为显式 `question` / `options` / `l1_node_ids`（P4）
|
||
|
||
### 4.2 SkillRegistry + discover_skills（skills.py）
|
||
|
||
```python
|
||
def parse_frontmatter(text: str) -> dict[str, str]: ...
|
||
def strip_frontmatter(text: str) -> str: ...
|
||
|
||
class SkillRegistry:
|
||
def set_paths(self, mapping: dict[str, Path]) -> None: ...
|
||
def read(self, name: str) -> str: ...
|
||
|
||
def discover_skills(skills_dir: Path) -> tuple[str, dict[str, str], str, SkillRegistry]: ...
|
||
```
|
||
|
||
与 TRM4 逻辑完全一致，无有意变更。
|
||
|
||
### 4.3 SearchToolDispatcher（tools.py）
|
||
|
||
```python
|
||
def get_tool_descriptions(include_read_skill: bool = False) -> str: ...
|
||
|
||
class SearchToolDispatcher:
|
||
"""实现 core/agent/protocols.ToolDispatcher。"""
|
||
def __init__(
|
||
self,
|
||
env: TreeEnvironment,
|
||
vlm: VLMProvider,
|
||
ocr: OCRProvider | None,
|
||
prompts_dir: Path,
|
||
skills: SkillRegistry | None,
|
||
*,
|
||
embed_fn: Callable[[str | list[str]], np.ndarray],
|
||
verify_vision: bool = True,
|
||
stats_sink: Callable[[dict[str, int]], None] | None = None,
|
||
) -> None: ...
|
||
|
||
async def dispatch(
|
||
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
|
||
) -> str: ...
|
||
```
|
||
|
||
| 工具 | 实现路径 |
|
||
|------|---------|
|
||
| `view_node` | → `env.view_node(node_id)` |
|
||
| `search_similar` | → `env.search_similar(query, top_k, embed_fn=...)` + 格式化 |
|
||
| `observe_frame` | → `env.resolve_frame_paths(...)` + `vision.observe_frame(...)` |
|
||
| `submit_answer` | → 返回确认文本 |
|
||
| `read_skill` | → `skills.read(name)` |
|
||
| 未知工具 | → `raise ValueError`（AgentLoop 捕获，不计步） |
|
||
|
||
**与 TRM4 有意变更**：
|
||
- 自由函数 + 大量位置参数 → 类封装（构造时注入依赖）
|
||
- 工具描述 `get_tool_descriptions()` 移入此文件
|
||
- `search_similar` 结果格式化由 dispatcher 负责（env 返回 `list[tuple[str, float]]`）
|
||
|
||
### 4.4 observe_frame（vision.py）
|
||
|
||
```python
|
||
async def observe_frame(
|
||
vlm: VLMProvider,
|
||
frame_paths: list[Path],
|
||
question: str,
|
||
prompts_dir: Path,
|
||
*,
|
||
ocr: OCRProvider | None,
|
||
stats_sink: Callable[[dict[str, int]], None] | None = None,
|
||
verify: bool = True,
|
||
) -> str: ...
|
||
```
|
||
|
||
两轮 VLM 调用保真：
|
||
|
||
```
|
||
1. [可选] OCR 转录 → 事前并置到 user_content
|
||
2. 提取轮: VLM + observe_frame_extract.md
|
||
3. [可选] 验证轮: VLM + observe_frame_verify.md
|
||
4. 返回 "[视觉观察] {证据}\n[验证] {核实结果}"
|
||
```
|
||
|
||
**与 TRM4 有意变更**：
|
||
|
||
| 项目 | TRM4 | TRM5 |
|
||
|------|------|------|
|
||
| 异步 | `_call_vl` 同步 | `await vlm.chat_with_images()` |
|
||
| VLM 接口 | 裸 LLMClient + 手动 base64 | VLMProvider Protocol，images 传 Path |
|
||
| OCR 接口 | `Callable[[list[Path]], str]` | `OCRProvider` Protocol（async） |
|
||
| Prompt 内容 | store/prompts/v2/ | 原封不动复制 |
|
||
|
||
### 4.5 OCRProvider Protocol（core/protocols.py 新增）
|
||
|
||
```python
|
||
@runtime_checkable
|
||
class OCRProvider(Protocol):
|
||
"""帧文字转录端口。"""
|
||
async def transcribe_frames(self, frame_paths: list[Path]) -> str: ...
|
||
```
|
||
|
||
### 4.6 MonkeyOCRClient（adapters/ocr.py）
|
||
|
||
```python
|
||
class MonkeyOCRClient:
|
||
"""实现 OCRProvider Protocol。多端点轮询 + 单帧降级。"""
|
||
def __init__(self, urls: list[str]) -> None: ...
|
||
async def check_health(self) -> None: ...
|
||
async def transcribe_frames(self, frame_paths: list[Path]) -> str: ...
|
||
```
|
||
|
||
内部同步 HTTP 调用通过 `asyncio.to_thread` 包装。端点轮询 + 线程安全 Session 保留 TRM4 逻辑。
|
||
|
||
---
|
||
|
||
## §5 交互流程
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant H as harness/inference
|
||
participant PM as PromptManager
|
||
participant SK as discover_skills
|
||
participant AL as AgentLoop
|
||
participant TD as SearchToolDispatcher
|
||
participant ENV as TreeEnvironment
|
||
participant V as vision.observe_frame
|
||
participant VLM as VLMProvider
|
||
participant OCR as OCRProvider
|
||
|
||
H->>SK: discover_skills(skills_dir)
|
||
SK-->>H: (always_text, task_skill_map, catalog_text, registry)
|
||
H->>PM: build_inference_prompt(...)
|
||
PM-->>H: system_prompt
|
||
H->>PM: format_user_prompt(question, options, l1_ids)
|
||
PM-->>H: user_prompt
|
||
H->>TD: 构造(env, vlm, ocr, prompts_dir, registry, embed_fn)
|
||
H->>AL: run(system_prompt, user_prompt, tool_dispatcher)
|
||
|
||
loop AgentLoop 每步推理
|
||
AL->>TD: dispatch("view_node", args, context)
|
||
TD->>ENV: view_node(node_id)
|
||
ENV-->>TD: 节点文本
|
||
TD-->>AL: 输出
|
||
|
||
AL->>TD: dispatch("observe_frame", args, context)
|
||
TD->>ENV: resolve_frame_paths(node_ids)
|
||
ENV-->>TD: list[Path]
|
||
TD->>V: observe_frame(vlm, paths, question, ...)
|
||
V->>OCR: transcribe_frames(paths)
|
||
OCR-->>V: ocr_text
|
||
V->>VLM: chat_with_images(extract prompt)
|
||
VLM-->>V: raw_evidence
|
||
V->>VLM: chat_with_images(verify prompt)
|
||
VLM-->>V: verify_result
|
||
V-->>TD: "[视觉观察] ...\n[验证] ..."
|
||
TD-->>AL: 输出
|
||
|
||
AL->>TD: dispatch("submit_answer", args, context)
|
||
TD-->>AL: "[ok] 答案已提交"
|
||
end
|
||
AL-->>H: LoopResult
|
||
```
|
||
|
||
---
|
||
|
||
## §6 错误处理
|
||
|
||
| 场景 | 处理 | 与 TRM4 一致性 |
|
||
|------|------|---------------|
|
||
| 节点不存在 | env 抛 KeyError，dispatcher 捕获返回错误文本 | 一致 |
|
||
| 帧文件不存在 | FileNotFoundError，vision 返回 `[VL错误]` | 一致 |
|
||
| VLM 提取轮失败 | 捕获 Exception，返回 `[VL错误]` | 一致 |
|
||
| VLM 验证轮失败 | 降级返回 `[验证] 跳过（调用失败）` | 一致 |
|
||
| OCR 失败 | 降级不注入，stats `ocr_failed=1` | 一致 |
|
||
| 未知工具名 | raise ValueError，AgentLoop 不计步 | 一致 |
|
||
| read_skill 未注册 | KeyError 透传，dispatcher 捕获返回错误文本 | 一致 |
|
||
|
||
原则：工具执行错误不中断 AgentLoop，所有异常在 dispatcher 层兜底为错误文本。
|
||
|
||
---
|
||
|
||
## §7 测试策略
|
||
|
||
| 测试文件 | 覆盖 | 方法 |
|
||
|----------|------|------|
|
||
| `test_search_prompt.py` | PromptManager 加载/拼装/格式化 | 临时目录写 prompt 文件 |
|
||
| `test_search_skills.py` | frontmatter 解析、discover_skills 分类 | 临时目录写 .md |
|
||
| `test_search_tools.py` | SearchToolDispatcher 5 个工具分发 | 假 env/VLM/OCR 通过 Protocol 注入 |
|
||
| `test_search_vision.py` | observe_frame 两轮、OCR 注入/降级、stats | 假 VLMProvider + 假 OCRProvider |
|
||
| `test_ocr_adapter.py` | MonkeyOCRClient 健康检查/轮询/降级 | responses 库 mock HTTP |
|
||
|
||
---
|
||
|
||
## §8 被否决的方案
|
||
|
||
| 方案 | 否决理由 |
|
||
|------|---------|
|
||
| vision.py 放 app/tree/ | observe_frame 是搜索工具实现，不是建树管线；tree/ 是离线预处理模块 |
|
||
| tools/ 子包 | 当前仅 5 个工具，子包过度组织 |
|