refactor: remove deprecated retriever module
RecursiveRetriever was a failed approach, not carried into TRM5. - delete app/retriever/ (empty placeholder) - drop retriever + train blocks from config/default.yaml - renumber fidelity checklist 13->12 items (drop #4, shift up) - sync core-goal text, dir tree, module-interaction diagrams across CLAUDE.md, ARCHITECTURE.md, overview.md, README.md - reference/ kept intact as historical code
This commit is contained in:
@@ -8,7 +8,7 @@
|
||||
|
||||
## §1 核心定位
|
||||
|
||||
**项目目标**:在层次化视频树上构建可自我进化的搜索 Agent + 可训练的递归检索器,实现长视频理解。目标会议 EMNLP 2026。
|
||||
**项目目标**:在层次化视频树上构建可自我进化的搜索 Agent,实现长视频理解。目标会议 EMNLP 2026。
|
||||
|
||||
**系统类比**——自进化循环对标 PyTorch 训练:
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
| 模块 | 目录 | 一句话定义 |
|
||||
|------|------|-----------|
|
||||
| 建树 | `app/tree/` | 离线预处理——VLM 生成三层 TreeIndex(L1段落→L2片段→L3帧),支持字幕注入和后增强 |
|
||||
| 训练 | `app/harness/` + `core/` | 自进化循环(推理→诊断→进化)+ RecursiveRetriever 参数训练 |
|
||||
| 训练 | `app/harness/` + `core/` | 自进化循环(推理→诊断→进化)+ CE-Gate 信息阶梯 |
|
||||
| 新题构建 | `app/question_gen/` | 生成 Video-MME 风格训练题,原始 benchmark 作 held-out 泛化评测 |
|
||||
|
||||
---
|
||||
@@ -55,7 +55,6 @@ flowchart TB
|
||||
HARNESS["app/harness/\n训练循环"]
|
||||
QGEN["app/question_gen/\n新题构建"]
|
||||
SEARCH["app/search/\nAgent 装配"]
|
||||
RET["app/retriever/\n可训练检索器"]
|
||||
PORTS["app/ports.py"]
|
||||
end
|
||||
|
||||
@@ -68,7 +67,7 @@ flowchart TB
|
||||
AD_LLM & AD_VLM & AD_EMB & AD_CACHE & AD_TEL -->|实现| CPROTO & PORTS
|
||||
HARNESS --> AGENT & EVO
|
||||
SEARCH --> AGENT
|
||||
TREE & RET & QGEN --> PORTS
|
||||
TREE & QGEN --> PORTS
|
||||
AGENT & EVO -->|定义| CPROTO
|
||||
```
|
||||
|
||||
@@ -81,7 +80,6 @@ flowchart TD
|
||||
CLI["main.py CLI"] --> RUNNER["app/harness/runner.py\n训练循环编排"]
|
||||
CLI --> BUILD["app/tree/video_builder.py\n建树"]
|
||||
CLI --> QGEN["app/question_gen/loader.py\n新题构建"]
|
||||
CLI --> TRAIN_RET["app/retriever/train.py\n检索器训练"]
|
||||
|
||||
RUNNER --> INF["app/harness/inference.py\n推理 step"]
|
||||
RUNNER --> DIAG["core/evolution/diagnose.py\n诊断"]
|
||||
@@ -147,10 +145,6 @@ project_root/
|
||||
│ │ ├── summarizer.py # 两轮 LLM 摘要(view_node / search_similar 用)
|
||||
│ │ ├── vision.py # observe_frame(VLM 两轮 + OCR 注入)
|
||||
│ │ └── tools.py # SearchToolDispatcher(实现 ToolDispatcher)
|
||||
│ ├── retriever/ # 可训练检索器
|
||||
│ │ ├── recursive.py # RecursiveRetriever (CrossAttention+ACT)
|
||||
│ │ ├── losses.py # NavigationLoss + ACTLoss
|
||||
│ │ └── train.py # 两阶段训练入口
|
||||
│ └── ports.py # 应用层特有端口
|
||||
│
|
||||
├── adapters/ # 外部实现层
|
||||
@@ -316,23 +310,22 @@ chat(messages) →
|
||||
|
||||
## §6 核心算法保真清单
|
||||
|
||||
迁移时逐一比对参考代码,不可简化。建树 4 项 + 训练 9 项 = 13 项:
|
||||
迁移时逐一比对参考代码,不可简化。建树 4 项 + 训练 8 项 = 12 项:
|
||||
|
||||
| # | 算法 | 参考文件 | 核心逻辑 |
|
||||
|---|------|---------|---------|
|
||||
| 1 | L2 轴心建树策略 | `reference/video_tree_trm/video_tree_builder.py` | L2 先行→L3 向下→L1 向上,asyncio 链式并发 |
|
||||
| 2 | VLM 批量帧描述 + JSON fallback | `reference/video_tree_trm/video_tree_builder.py` | `_L3_BATCH_SIZE=5` 批量调用,解析失败逐帧 fallback |
|
||||
| 3 | 断点续跑机制 | `reference/video_tree_trm/video_tree_builder.py` | `progress.json` + L1 中间 JSON,按段恢复 |
|
||||
| 4 | RecursiveRetriever | `reference/docs/architecture.md §5` | Cross-Attention 选择器 + ACT halt + z 状态累积 |
|
||||
| 5 | CE-Gate e-process | TRM4 `core/harness/eprocess.py` | 截断 Beta 混合、四出口门控 |
|
||||
| 6 | 信息阶梯 | TRM4 `core/harness/gate_ladder.py` | 冷启动 2:1、gamma-EMA、反泄漏 |
|
||||
| 7 | 块顺序验证 | TRM4 `core/harness/validate.py` | 基线缓存、INFRA 护栏、配对翻转 |
|
||||
| 8 | 诊断瀑布 | TRM4 `core/harness/diagnose.py` | 错误归因级联、缺陷 vs 失误、D1-D5 |
|
||||
| 9 | 进化 patch 引擎 | TRM4 `core/harness/evolve.py` + `patch.py` | 保护跨度、rank-and-clip、附录/动量 |
|
||||
| 10 | Mini-batch 构建 | TRM4 `core/harness/batching.py` | FFD + round-robin + 正确率混合 |
|
||||
| 11 | Agent Loop | TRM4 `core/loop.py` | Thinking+JSON、json_repair、pluggy hook |
|
||||
| 12 | 树环境语义搜索 | TRM4 `core/tree/environment.py` | 分块 embedding、祖先去重、锚定验证 |
|
||||
| 13 | 训练循环编排 | TRM4 `core/harness/runner.py` | 三级嵌套、慢更新10步、断点续训 |
|
||||
| 4 | CE-Gate e-process | TRM4 `core/harness/eprocess.py` | 截断 Beta 混合、四出口门控 |
|
||||
| 5 | 信息阶梯 | TRM4 `core/harness/gate_ladder.py` | 冷启动 2:1、gamma-EMA、反泄漏 |
|
||||
| 6 | 块顺序验证 | TRM4 `core/harness/validate.py` | 基线缓存、INFRA 护栏、配对翻转 |
|
||||
| 7 | 诊断瀑布 | TRM4 `core/harness/diagnose.py` | 错误归因级联、缺陷 vs 失误、D1-D5 |
|
||||
| 8 | 进化 patch 引擎 | TRM4 `core/harness/evolve.py` + `patch.py` | 保护跨度、rank-and-clip、附录/动量 |
|
||||
| 9 | Mini-batch 构建 | TRM4 `core/harness/batching.py` | FFD + round-robin + 正确率混合 |
|
||||
| 10 | Agent Loop | TRM4 `core/loop.py` | Thinking+JSON、json_repair、pluggy hook |
|
||||
| 11 | 树环境语义搜索 | TRM4 `core/tree/environment.py` | 分块 embedding、祖先去重、锚定验证 |
|
||||
| 12 | 训练循环编排 | TRM4 `core/harness/runner.py` | 三级嵌套、慢更新10步、断点续训 |
|
||||
|
||||
> **TRM4** 指 `/home/iomgaa/Projects/Video-Tree-TRM4/`,**reference** 指 `/home/iomgaa/Projects/Video-Tree-TRM5/reference/`。
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ status: approved
|
||||
|
||||
## 1. 背景与动机
|
||||
|
||||
TRM5 的三大模块(建树、训练 harness、新题构建)中,建树是一切的地基——搜索 Agent、训练循环、检索器全部依赖树结构。当前 `app/tree/` 目录为空,需要从 reference 代码和 TRM4 迁移建树能力。
|
||||
TRM5 的三大模块(建树、训练 harness、新题构建)中,建树是一切的地基——搜索 Agent、训练循环全部依赖树结构。当前 `app/tree/` 目录为空,需要从 reference 代码和 TRM4 迁移建树能力。
|
||||
|
||||
### 1.1 现状
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 系统总览 (Overview)
|
||||
|
||||
> Video-Tree-TRM5:在层次化视频树上构建可自我进化的搜索 Agent + 可训练递归检索器,通过 Harness Engineering 持续改进实现长视频理解。目标会议 EMNLP 2026。
|
||||
> Video-Tree-TRM5:在层次化视频树上构建可自我进化的搜索 Agent,通过 Harness Engineering 持续改进实现长视频理解。目标会议 EMNLP 2026。
|
||||
|
||||
## 1. 核心思想:自进化循环对标 PyTorch 训练
|
||||
|
||||
@@ -25,7 +25,6 @@ flowchart TD
|
||||
main[main.py CLI 入口] --> runner[app/harness/runner.py 训练循环]
|
||||
main --> build[app/tree/video_builder.py 建树]
|
||||
main --> qgen[app/question_gen/generator.py 新题构建]
|
||||
main --> train_ret[app/retriever/train.py 检索器训练]
|
||||
|
||||
runner --> inf[app/harness/inference.py 推理]
|
||||
runner --> diag[core/evolution/diagnose.py 诊断]
|
||||
@@ -46,7 +45,6 @@ flowchart TD
|
||||
| `app/harness/` | 训练 harness:runner 循环编排、推理 step、mini-batch、信息阶梯、workspace 版本管理 |
|
||||
| `app/question_gen/` | 新题构建:题目生成、基线校准、去重 |
|
||||
| `app/search/` | 搜索 Agent 装配:PromptManager + SkillRegistry |
|
||||
| `app/retriever/` | 可训练检索器:RecursiveRetriever(CrossAttention+ACT)、两阶段训练 |
|
||||
| `core/agent/` | AgentLoop 引擎:Thinking+JSON 推理循环,pluggy hook 驱动 |
|
||||
| `core/evolution/` | 诊断+进化引擎:两阶段诊断、patch/rewrite 进化、CE-Gate e-process |
|
||||
| `adapters/` | 外部实现层:GovernedLLMClient(遥测+熔断+缓存)、VLM、Embedding、ASR、OCR |
|
||||
|
||||
Reference in New Issue
Block a user