diff --git a/research-wiki/plans/2026-07-09-main-inference-entry.md b/research-wiki/plans/2026-07-09-main-inference-entry.md index 28e3094..7db58cb 100644 --- a/research-wiki/plans/2026-07-09-main-inference-entry.md +++ b/research-wiki/plans/2026-07-09-main-inference-entry.md @@ -16,11 +16,23 @@ | 操作 | 文件 | 职责 | |------|------|------| +| **Task 0: subtitle 迁入 Card** | | | +| Modify | `app/tree/index.py:73-194` | L3Card/L2Card 加 subtitle 字段;L3Node 去 subtitle 字段 | +| Modify | `app/tree/index.py:267-390` | to_dict/from_dict 序列化适配 | +| Modify | `app/tree/subtitle.py:239-308` | Voronoi 写入 Card.subtitle 而非 Node.subtitle | +| Modify | `app/tree/environment.py:448-489` | _node_full_text/_node_anchored_text 简化 | +| Modify | `app/tree/environment.py:367-381` | get_subtitle 读 Card.subtitle | +| Modify | `app/tree/video_builder.py:475-486` | Phase 7 保留但改写 Card.subtitle | +| Modify | `app/tree/verify.py` | `l3.subtitle` → `l3.card.subtitle` | +| Modify | `app/question_gen/synthesizer.py` | `l3.subtitle` / `l2.subtitle` → `card.subtitle` | +| Create | `tools/migrate_subtitle_to_card.py` | 迁移脚本:300 棵树 subtitle 从 Node 移入 Card | +| Modify | 9 个测试文件 | Card 构造加 subtitle 参数 | +| **Task 1-8: 原有任务** | | | | Create | `app/harness/deps_router.py` | 按 video_id 懒加载 InferenceDeps 并路由 dispatch/prompt_builder | -| Modify | `app/ports.py:76` | 新增 4 个 Protocol(ToolDispatchFn, ToolDispatchFactory, PromptBuilderFn, PromptBuilderFactory) | +| Modify | `app/ports.py` (文件末尾追加) | 新增 4 个 Protocol | | Modify | `app/harness/runner.py:454-469` | __init__ 增加 2 个 factory 参数 + fail-fast 校验 | | Modify | `app/harness/runner.py:2064-2082` | _make_* 方法优先用注入值 | -| Create | `main.py` | Composition Root:argparse + InfraSettings + 适配器构建 + Runner 组装 | +| Create | `main.py` | Composition Root | | Move | `store/prompts/*.md` → `store/prompts/v1/` | 版本化目录重组 | | Create | `store/skills/v1/` (13 files) | 从 TRM4 v1 精简 + 注入 TRM5 card 字段 | | Modify | `config/default.yaml:29,31` | concurrency=24, max_steps=40 | @@ -29,6 +41,426 @@ --- +### Task 0: subtitle 迁入 Card + 建树管线修正 + 300 棵树迁移 + +**Files:** +- Modify: `app/tree/index.py` (L3Card, L2Card, L3Node, to_dict, from_dict) +- Modify: `app/tree/subtitle.py:239-308` (assign_subtitles_voronoi) +- Modify: `app/tree/environment.py:367-381,448-489` (_node_full_text, get_subtitle) +- Modify: `app/tree/video_builder.py:475-486,902-996` (builder pipeline) +- Create: `tools/migrate_subtitle_to_card.py` +- Modify: 9 个测试文件 (Card 构造适配) + +**设计决策:** +- L3Card/L2Card 加 `subtitle: str = ""` 字段(放在字段列表末尾,`_collect_card_strings` 自动收集) +- L1Card 不加 subtitle(用户确认) +- L3Node 移除 `subtitle: str | None` 字段(数据迁入 Card) +- Card 保持 `frozen=True`,建树时在 Card 创建前计算好 subtitle 传入构造函数;迁移脚本用 `dataclasses.replace()` 创建新 Card +- 建树管线中 L2/L3 直接在 Card 构造时注入字幕,移除 Phase 7 Voronoi 后处理 +- L2 subtitle = `extract_subtitle_for_range(srt_entries, l2_time_range)` 在 `_build_l2_video_async` 中计算 +- L3 subtitle 仍用 Voronoi 逻辑(精确分配帧级字幕),但写入 Card 而非 Node + +- [ ] **Step 1: 修改 Card dataclass** + +在 `app/tree/index.py` 中: + +**L3Card** (第 73-92 行) — 末尾加 `subtitle: str = ""`: + +```python +@dataclass(frozen=True) +class L3Card: + """L3 帧级语义卡片(不可变)。""" + + frame_summary: str + visible_entities: list[str] + ongoing_actions: list[str] + visible_text: list[str] + spatial_layout: str + visual_attributes: dict[str, Any] + subtitle: str = "" +``` + +**L2Card** (第 96-117 行) — 末尾加 `subtitle: str = ""`: + +```python +@dataclass(frozen=True) +class L2Card: + """L2 事件级语义卡片(不可变)。""" + + event_description: str + entities: list[str] + actions: list[str] + action_subjects: list[str] + visible_text: list[str] + spatial_relations: str + state_changes: str | None + subtitle: str = "" +``` + +**L3Node** (第 176-194 行) — 移除 `subtitle` 字段: + +```python +@dataclass +class L3Node: + """L3 帧级语义节点(叶层)。""" + + id: str + card: L3Card + embedding: np.ndarray | None = None + timestamp: float | None = None + frame_path: str | None = None + # subtitle 已迁入 L3Card,此处不再保留 +``` + +- [ ] **Step 2: 修改序列化/反序列化** + +**to_dict** — L3 `l3_to_dict` (第 267-284 行): subtitle 从 card 输出,移除节点级 subtitle: + +```python +def l3_to_dict(n: L3Node) -> dict[str, Any]: + d: dict[str, Any] = { + "id": n.id, + "card": { + "frame_summary": n.card.frame_summary, + "visible_entities": n.card.visible_entities, + "ongoing_actions": n.card.ongoing_actions, + "visible_text": n.card.visible_text, + "spatial_layout": n.card.spatial_layout, + "visual_attributes": n.card.visual_attributes, + "subtitle": n.card.subtitle, + }, + "timestamp": n.timestamp, + "frame_path": n.frame_path, + } + if include_embedding: + d["embedding"] = _embed_to_str(n.embedding) + return d +``` + +**to_dict** — L2 `l2_to_dict` (第 286-303 行): card 字典加 subtitle: + +```python +def l2_to_dict(n: L2Node) -> dict[str, Any]: + d: dict[str, Any] = { + "id": n.id, + "card": { + "event_description": n.card.event_description, + "entities": n.card.entities, + "actions": n.card.actions, + "action_subjects": n.card.action_subjects, + "visible_text": n.card.visible_text, + "spatial_relations": n.card.spatial_relations, + "state_changes": n.card.state_changes, + "subtitle": n.card.subtitle, + }, + "time_range": list(n.time_range) if n.time_range else None, + "children": [l3_to_dict(c) for c in n.children], + } + if include_embedding: + d["embedding"] = _embed_to_str(n.embedding) + return d +``` + +**from_dict** — L3 反序列化 (第 337-354 行): subtitle 从 card 读取,兼容旧格式从节点级读取: + +```python +l3_card = L3Card( + frame_summary=l3d["card"]["frame_summary"], + visible_entities=l3d["card"]["visible_entities"], + ongoing_actions=l3d["card"]["ongoing_actions"], + visible_text=l3d["card"]["visible_text"], + spatial_layout=l3d["card"]["spatial_layout"], + visual_attributes=l3d["card"]["visual_attributes"], + subtitle=l3d["card"].get("subtitle", "") or l3d.get("subtitle", "") or "", +) +l3_nodes.append( + L3Node( + id=l3d["id"], + card=l3_card, + embedding=_embed_from_str(l3d.get("embedding")), + timestamp=l3d.get("timestamp"), + frame_path=l3d.get("frame_path"), + ) +) +``` + +**from_dict** — L2 反序列化 (第 355-363 行): 加 subtitle: + +```python +l2_card = L2Card( + event_description=l2d["card"]["event_description"], + entities=l2d["card"]["entities"], + actions=l2d["card"]["actions"], + action_subjects=l2d["card"]["action_subjects"], + visible_text=l2d["card"]["visible_text"], + spatial_relations=l2d["card"]["spatial_relations"], + state_changes=l2d["card"]["state_changes"], + subtitle=l2d["card"].get("subtitle", ""), +) +``` + +- [ ] **Step 3: 修改 environment.py** + +**`get_subtitle`** (第 367-381 行) — 读 Card.subtitle,L1/L2/L3 均支持: + +```python +def get_subtitle(self, node_id: str) -> str: + """返回节点字幕文本。L2/L3 从 Card 读取,L1 返回空串。""" + node = self._id_to_node.get(node_id) + if node is None: + return "" + if isinstance(node, (L2Node, L3Node)): + return node.card.subtitle or "" + return "" +``` + +**`_collect_card_strings`** (第 67-78 行) — 新增 `skip_fields` 参数,排除 subtitle(subtitle 需单独标签/锚标处理): + +```python +def _collect_card_strings(node: AnyNode, *, skip_fields: frozenset[str] = frozenset()) -> list[str]: + """从节点 card 中递归收集所有非空字符串字段,可排除指定字段。""" + result: list[str] = [] + _collect_from_obj(node.card, result, skip_fields=skip_fields) + return result +``` + +`_collect_from_obj` (第 81-101 行) — 在 dataclass 分支中跳过 `skip_fields`: + +```python +def _collect_from_obj(obj: object, out: list[str], *, skip_fields: frozenset[str] = frozenset()) -> None: + if isinstance(obj, str): + stripped = obj.strip() + if stripped: + out.append(stripped) + elif isinstance(obj, dict): + for v in obj.values(): + _collect_from_obj(v, out, skip_fields=skip_fields) + elif isinstance(obj, (list, tuple)): + for item in obj: + _collect_from_obj(item, out, skip_fields=skip_fields) + elif hasattr(obj, "__dataclass_fields__"): + for field_name in obj.__dataclass_fields__: + if field_name in skip_fields: + continue + _collect_from_obj(getattr(obj, field_name), out, skip_fields=skip_fields) +``` + +**`_node_full_text`** (第 448-461 行) — 保持"字幕:"标签,改为读 `card.subtitle`: + +```python +_SUBTITLE_SKIP = frozenset({"subtitle"}) + +def _node_full_text(self, node: AnyNode) -> str: + """获取节点完整文本(card 非 subtitle 字段 + 带标签的字幕)。""" + card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP) + text = "\n".join(card_strings) + if isinstance(node, (L2Node, L3Node)) and node.card.subtitle: + text += f"\n字幕: {node.card.subtitle}" + return text +``` + +**`_node_anchored_text`** (第 463-489 行) — 保持 [cN]/[sN] 双锚语义,改为读 `card.subtitle`: + +```python +def _node_anchored_text(self, node: AnyNode) -> str: + """获取带行号锚的节点文本。card 字段 [cN],字幕 [sN]。""" + card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP) + card_lines: list[str] = [] + for s in card_strings: + card_lines.extend(ln for ln in s.splitlines() if ln.strip()) + + sub_lines: list[str] = [] + if isinstance(node, (L2Node, L3Node)) and node.card.subtitle: + sub_lines = [ln for ln in node.card.subtitle.splitlines() if ln.strip()] + + anchored: list[str] = [] + for i, line in enumerate(card_lines, 1): + anchored.append(f"[c{i}] {line}") + for i, line in enumerate(sub_lines, 1): + anchored.append(f"[s{i}] {line}") + return "\n".join(anchored) +``` + +- [ ] **Step 4: 修改 subtitle.py — Voronoi 写入 Card** + +`assign_subtitles_voronoi` (第 239-308 行) — 用 `dataclasses.replace()` 写入 Card.subtitle: + +在函数顶部加 `import dataclasses`,然后将第 302 行: +```python +# 原: l3.subtitle = subtitle_text if subtitle_text else None +# 改: +l3.card = dataclasses.replace(l3.card, subtitle=subtitle_text or "") +``` + +同时在函数末尾为每个 L2 聚合子节点字幕: + +```python +# L2 字幕聚合(在 L3 分配完成后) +for l1 in index.roots: + for l2 in l1.children: + l2_sub_parts = [c.card.subtitle for c in l2.children if c.card.subtitle] + if l2_sub_parts: + l2.card = dataclasses.replace(l2.card, subtitle="\n".join(l2_sub_parts)) +``` + +- [ ] **Step 5: 修改 video_builder.py — 建树管线** + +Phase 7 保留 `assign_subtitles_voronoi`(它现在写 Card.subtitle + L2 聚合),不在 `_build_l2_video_async` 中注入字幕(避免双写)。仅更新注释: + +```python +# Phase 7: 字幕注入 Card(L3 Voronoi 分配 + L2 聚合) +if srt_entries: + assign_subtitles_voronoi(index, srt_entries) + logger.info("字幕已注入 Card: L3 Voronoi + L2 聚合", n_entries=len(srt_entries)) +``` + +`_build_l2_video_async` 和 `_build_l3_video_async` 不做字幕注入改动。字幕注入统一由 Phase 7 `assign_subtitles_voronoi` 负责。 + +- [ ] **Step 5.5: 修改 verify.py 和 synthesizer.py 中的 subtitle 访问** + +搜索并替换所有 `l3.subtitle` → `l3.card.subtitle`,`l2.subtitle` → `l2.card.subtitle`(若存在): + +```bash +grep -rn "\.subtitle" app/tree/verify.py app/question_gen/synthesizer.py +``` + +对每处命中做替换,例如: +- `app/tree/verify.py`: `node.subtitle` → `node.card.subtitle`(仅 L3/L2 节点) +- `app/question_gen/synthesizer.py`: `l3.subtitle` → `l3.card.subtitle` + +- [ ] **Step 6: 迁移脚本** + +```python +# tools/migrate_subtitle_to_card.py +"""迁移 300 棵树:L3 subtitle 从 Node 级移入 Card 级,L2 聚合子节点字幕。""" + +import dataclasses +import json +import sys +from pathlib import Path + +from loguru import logger + + +def migrate_tree(tree_path: Path) -> bool: + """迁移单棵树,返回是否有变更。幂等:已迁移的树不会被修改。""" + data = json.loads(tree_path.read_text(encoding="utf-8")) + changed = False + + for l1 in data.get("roots", []): + for l2 in l1.get("children", []): + l2_sub_parts: list[str] = [] + + for l3 in l2.get("children", []): + # L3: node 级 subtitle 迁入 card(幂等:已有 card subtitle 则跳过) + card_sub = l3.get("card", {}).get("subtitle", "") + node_sub = l3.get("subtitle") # None = 从未有字幕 + + if node_sub is not None and "subtitle" in l3: + # 旧格式:有 node 级 subtitle + final_sub = card_sub or (node_sub if node_sub else "") + l3["card"]["subtitle"] = final_sub + del l3["subtitle"] + changed = True + elif "subtitle" not in l3.get("card", {}): + # 无任何 subtitle 数据 + l3.setdefault("card", {})["subtitle"] = "" + changed = True + + # 收集 L3 字幕用于 L2 聚合 + effective_sub = l3.get("card", {}).get("subtitle", "") + if effective_sub: + l2_sub_parts.append(effective_sub) + + # L2: 聚合 L3 字幕(幂等:已有 card subtitle 则跳过) + l2_card = l2.setdefault("card", {}) + if "subtitle" not in l2_card: + l2_card["subtitle"] = "\n".join(l2_sub_parts) if l2_sub_parts else "" + changed = True + + if changed: + tree_path.write_text( + json.dumps(data, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + return changed + + +def main() -> None: + """迁移 store/videos/ 下所有 tree.json。""" + videos_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("store/videos") + tree_files = sorted(videos_dir.glob("*/tree.json")) + logger.info("发现 {} 棵树待迁移", len(tree_files)) + + migrated = 0 + for tf in tree_files: + if migrate_tree(tf): + migrated += 1 + logger.info("迁移完成: {}/{} 棵树有变更", migrated, len(tree_files)) + + +if __name__ == "__main__": + main() +``` + +Run: `conda run -n Video-Tree-TRM python tools/migrate_subtitle_to_card.py store/videos` + +验证: +```bash +python3 -c " +import json; d=json.loads(open('store/videos/068rdc75mHM/tree.json').read()) +l1=d['roots'][0]; l2=l1['children'][0]; l3=l2['children'][0] +print('L3 card subtitle:', bool(l3['card'].get('subtitle'))) +print('L3 node subtitle:', 'subtitle' in l3 and l3.get('subtitle') is not None) +print('L2 card subtitle:', bool(l2['card'].get('subtitle'))) +" +``` +Expected: L3 card subtitle: True, L3 node subtitle: False, L2 card subtitle: True + +- [ ] **Step 7: 更新测试文件** + +需要更新的 9 个文件中所有 `L3Card(...)` 和 `L2Card(...)` 构造调用。由于 `subtitle` 有默认值 `""`,大多数测试不需要改动(默认空串即可)。但 `L3Node(...)` 构造中如果传了 `subtitle=` 参数需要移除。 + +需要检查并修改的具体文件: + +``` +tests/integration/test_tree_build_e2e.py +tests/unit/test_repair_detector.py +tests/unit/test_repair_regenerator.py +tests/unit/test_search_tools.py +tests/unit/test_subtitle.py +tests/unit/test_tree_environment.py +tests/unit/test_tree_index.py +tests/unit/test_verify.py +tests/unit/test_video_builder.py +``` + +对每个文件: +1. `L3Node(..., subtitle=xxx)` → 移除 `subtitle=` 参数,改为在 L3Card 构造时传入 `subtitle=xxx` +2. `L3Card(a, b, c, d, e, f)` 位置参数调用 → 不影响(subtitle 有默认值) +3. 确认 `L2Card(...)` 构造不受影响(subtitle 有默认值) + +- [ ] **Step 8: 运行全量测试** + +Run: `conda run -n Video-Tree-TRM pytest tests/ -x -q` +Expected: ALL PASSED + +- [ ] **Step 9: Commit** + +```bash +git add app/tree/index.py app/tree/subtitle.py app/tree/environment.py app/tree/video_builder.py tools/migrate_subtitle_to_card.py tests/ +git commit -m "refactor(tree): subtitle 迁入 L3Card/L2Card + 建树管线修正 + 300 棵树迁移 + +- L3Card/L2Card 新增 subtitle: str 字段 +- L3Node 移除 subtitle 字段(数据迁入 Card) +- assign_subtitles_voronoi 改写 Card.subtitle +- _node_full_text 简化(Card 已含 subtitle) +- 建树管线 L2 直接注入字幕到 Card +- 迁移脚本处理 300 棵现有树" +``` + +--- + ### Task 1: store/ 目录重组 **Files:** @@ -56,11 +488,18 @@ git mv store/prompts/view_node_children_verify.md store/prompts/v1/ mkdir -p store/skills/v1 ``` -- [ ] **Step 3: 验证目录结构** +- [ ] **Step 3: 修正 system.md — L1 去掉 subtitle 描述** + +编辑 `store/prompts/v1/system.md`,在 L1 字段表中删除 `| subtitle | 完整字幕(较长) |` 行。在信任层级段落末尾将"三个层级都包含 visible_text 和 subtitle 字段"改为"三个层级都包含 visible_text 字段,L2 和 L3 额外包含 subtitle 字段"。 + +- [ ] **Step 4: 验证目录结构** Run: `ls store/prompts/v1/ && ls store/skills/v1/` Expected: 9 个 .md 文件在 prompts/v1/ 下,skills/v1/ 为空目录。 +Run: `grep subtitle store/prompts/v1/system.md` +Expected: 仅 L2 和 L3 部分出现 subtitle。 + - [ ] **Step 4: Commit** ```bash @@ -121,8 +560,8 @@ git commit -m "refactor(store): prompts 版本化目录重组 + skills/v1 骨架 | L3 | ongoing_actions | 正在发生的动作 | | L3 | spatial_layout | 精确空间位置 | | L3 | visual_attributes | 光照、色调、机位 | +| L2/L3 | subtitle | 字幕转写(L1 无此字段) | | 全层 | visible_text | 画面文字(OCR) | -| 全层 | subtitle | 字幕转写 | ``` **TRM4 v1 源文件路径:** `/home/iomgaa/Projects/Video-Tree-TRM4/store/skills/v1/` @@ -1307,10 +1746,11 @@ git commit -m "test: 冒烟测试通过,900 题推理管线就绪" ## 核心算法保真校验 -本计划不涉及核心算法迁移。所有 13 项核心算法(L2 轴心建树、CE-Gate e-process、Agent Loop 等)在 TRM5 中已有完整实现。本次工作仅涉及: -- 依赖注入的架构改进(Runner factory 注入) -- 新增基础设施模块(InferenceDepsRouter、main.py) -- 内容准备(skills/v1、prompts/v1 目录重组) -- 配置变更 +Task 0 涉及 Voronoi 字幕分配(算法清单中非独立项,属于建树模块的后处理步骤)。改动范围: -保真校验不适用。 +| 算法 | 改动性质 | 保真判定 | +|------|---------|---------| +| Voronoi 字幕分配 (`subtitle.py`) | 写入目标从 `L3Node.subtitle` 改为 `L3Card.subtitle` + 新增 L2 聚合 | 核心逻辑(中点计算、范围扩展)不变,仅写入目标变更。**保真**。 | +| L2 建树 (`video_builder.py`) | Card 构造后注入 subtitle | 新增逻辑(原来不存在 L2 subtitle),不涉及算法简化。**不适用**。 | + +其余 Task 1-8 不涉及核心算法迁移。