diff --git a/research-wiki/plans/2026-07-07-core-evolution.md b/research-wiki/plans/2026-07-07-core-evolution.md new file mode 100644 index 0000000..beb7b14 --- /dev/null +++ b/research-wiki/plans/2026-07-07-core-evolution.md @@ -0,0 +1,1145 @@ +# core/evolution/ Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Migrate TRM4 evolution engine into a Clean Architecture extractable kernel (`core/evolution/`). + +**Architecture:** 7 files, dependency-ordered. Pure decision logic only — no DB writes, no filesystem versioning, no inference orchestration. All external I/O through 3 Protocols + 1 existing LLMProvider. Async-first (asyncio.gather + Semaphore). + +**Tech Stack:** Python 3.11, asyncio, scipy.special, json_repair, loguru, pluggy (already in project) + +**Design doc:** `research-wiki/designs/2026-07-07-core-evolution-design.md` + +**TRM4 source:** `/home/iomgaa/Projects/Video-Tree-TRM4/core/harness/` + +--- + +## File Structure + +| File | Lines (est.) | Creates | Depends on | +|------|-------------|---------|-----------| +| `core/evolution/protocols.py` | 60 | New | `core/evolution/types.py` (TYPE_CHECKING) | +| `core/evolution/types.py` | 280 | New | — | +| `core/evolution/gate.py` | 170 | New | `types.py` | +| `core/evolution/patch.py` | 440 | New | — (loguru only) | +| `core/evolution/validate.py` | 70 | New | `gate.py`, `types.py` | +| `core/evolution/diagnose.py` | 1200 | New | `types.py`, `protocols.py`, `core/protocols.py` | +| `core/evolution/evolve.py` | 900 | New | `types.py`, `protocols.py`, `patch.py`, `core/protocols.py` | +| `core/evolution/__init__.py` | 30 | Modify | all above | +| `tests/unit/test_gate.py` | 200 | New | — | +| `tests/unit/test_patch.py` | 350 | New | — | +| `tests/unit/test_validate.py` | 120 | New | — | +| `tests/unit/test_diagnose.py` | 400 | New | — | +| `tests/unit/test_evolve.py` | 400 | New | — | + +--- + +### Task 1: protocols.py + types.py（基础层） + +**Files:** +- Create: `core/evolution/protocols.py` +- Create: `core/evolution/types.py` +- Test: `tests/unit/test_evolution_types.py` + +- [ ] **Step 1: Write type construction tests** + +```python +"""tests/unit/test_evolution_types.py""" +from core.evolution.types import ( + GateParams, GateVerdict, SpanMetrics, SkillStepAdherence, + QuestionMetrics, ErrorAttribution, CaseSample, + SkillCasePack, SystemCasePack, ToolCasePack, DiagnosisResult, + EvolutionRecord, RejectedEdit, EvolutionResult, + PairResult, QuadrantClassification, + DiagnosePrompts, EvolvePrompts, +) + +def test_gate_params_frozen(): + p = GateParams(e_confirm=20.0, e_provisional=3.0, w_net_min=2, + delta_min=0.02, lambda_dir=-0.642, e_rollback=10.0) + assert p.e_confirm == 20.0 + import pytest + with pytest.raises(AttributeError): + p.e_confirm = 1.0 + +def test_evolution_record_mutable(): + r = EvolutionRecord( + target_file="test.md", target_type="skill", + original_content="a", evolved_content="b", + reason="test", status="accepted", source_version="v1", + suggestions=[], edits=[], apply_report=[], clip_info={}, + ) + r.status = "rejected" + assert r.status == "rejected" + +def test_diagnose_prompts_frozen(): + dp = DiagnosePrompts( + defect_vs_lapse="p1", reasoning_sub="p2", + span_eval_system="p3", span_eval_user="p4", + missed_nodes="p5", skill_adherence="p6", + confirmation_bias="p7", evidence_sufficiency="p8", + ) + assert dp.defect_vs_lapse == "p1" +``` + +- [ ] **Step 2: Run test — expect FAIL (ImportError)** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolution_types.py -v` + +- [ ] **Step 3: Implement types.py** + +从 TRM4 迁移所有 dataclass。关键变更： + +| TRM4 位置 | TRM5 变更 | +|-----------|---------| +| `eprocess.py::GateParams/GateVerdict` | 原样迁移 | +| `diagnose.py::SpanMetrics` 等 9 个 | 全部标 `frozen=True`（TRM4 中 QuestionMetrics 非 frozen，TRM5 一次性构造） | +| `evolve.py::EvolutionRecord` | 保持 mutable，新增 `result_version: str | None = None` 字段 | +| `evolve.py::RejectedEdit` | 原样迁移，frozen | +| `evolve.py::EvolutionResult` | 移除 `skills_version`/`prompts_version`（app/ 职责） | +| `validate.py::ValidationOutcome/Probation/InferenceRunConfig` | 不迁——属 app/harness/ | +| 新增 `PairResult`/`QuadrantClassification` | 块验证纯决策输出 | +| 新增 `DiagnosePrompts`/`EvolvePrompts` | 模板束 | + +**保真校验点**：逐字段对比 TRM4 dataclass，确保无遗漏字段。特别注意 `DiagnosisResult` 的完整字段列表（约 20 个字段）。 + +- [ ] **Step 4: Implement protocols.py** + +```python +"""core/evolution/protocols.py — 3 个只读 Protocol""" +from __future__ import annotations +from typing import Any, Protocol, runtime_checkable + +@runtime_checkable +class SkillStore(Protocol): + def read_skill(self, filename: str) -> str: ... + def list_skill_files(self) -> list[str]: ... + +@runtime_checkable +class PromptStore(Protocol): + def read_prompt(self, filename: str) -> str: ... + def list_prompt_files(self) -> list[str]: ... + +@runtime_checkable +class RunLog(Protocol): + async def get_predictions( + self, run_id: str, *, question_ids: list[str] | None = None, + ) -> list[dict[str, Any]]: ... + async def get_traces( + self, run_id: str, *, question_ids: list[str] | None = None, + ) -> list[dict[str, Any]]: ... +``` + +- [ ] **Step 5: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolution_types.py -v` + +- [ ] **Step 6: Commit** + +``` +git add core/evolution/types.py core/evolution/protocols.py tests/unit/test_evolution_types.py +git commit -m "feat(evolution): types.py + protocols.py — foundation dataclasses and Protocol ports" +``` + +--- + +### Task 2: gate.py — CE-Gate e-process（算法 #5） + +**Files:** +- Create: `core/evolution/gate.py` +- Test: `tests/unit/test_gate.py` +- Source: TRM4 `eprocess.py` (163 行，原样迁移，零有意变更) + +- [ ] **Step 1: Write gate tests** + +```python +"""tests/unit/test_gate.py""" +import math +import pytest +from core.evolution.gate import compute_e_value, gate_decision, probation_verdict +from core.evolution.types import GateParams, GateVerdict + +_PARAMS = GateParams( + e_confirm=20.0, e_provisional=3.0, w_net_min=2, + delta_min=0.02, lambda_dir=-0.642, e_rollback=10.0, +) + +class TestComputeEValue: + def test_zero_zero_returns_one(self): + assert compute_e_value(0, 0) == pytest.approx(1.0) + + def test_negative_w_raises(self): + with pytest.raises(ValueError): + compute_e_value(-1, 0) + + def test_negative_l_raises(self): + with pytest.raises(ValueError): + compute_e_value(0, -1) + + def test_heavy_loss_returns_near_zero(self): + assert compute_e_value(0, 20) < 0.01 + + def test_heavy_win_returns_large(self): + assert compute_e_value(10, 0) > 100 + + def test_symmetric(self): + e_5_3 = compute_e_value(5, 3) + e_3_5 = compute_e_value(3, 5) + assert e_5_3 > e_3_5 + +class TestGateDecision: + def test_confirmed_needs_both_e_and_delta(self): + v = gate_decision(10, 0, 10, 10, params=_PARAMS) + assert v.decision == "accept_confirmed" + + def test_continue_on_balanced(self): + v = gate_decision(3, 3, 6, 20, params=_PARAMS) + assert v.decision == "continue" + + def test_reject_inertia_on_exhaustion(self): + v = gate_decision(1, 1, 2, 0, params=_PARAMS) + assert v.decision == "reject_inertia" + + def test_n_used_zero_raises(self): + with pytest.raises(ValueError): + gate_decision(0, 0, 0, 10, params=_PARAMS) + + def test_n_remaining_negative_raises(self): + with pytest.raises(ValueError): + gate_decision(1, 0, 1, -1, params=_PARAMS) + +class TestProbationVerdict: + def test_strong_win_confirmed(self): + assert probation_verdict(10, 0, params=_PARAMS) == "confirmed" + + def test_strong_loss_rollback(self): + assert probation_verdict(0, 10, params=_PARAMS) == "rollback" + + def test_balanced_unverified(self): + assert probation_verdict(3, 3, params=_PARAMS) == "unverified" +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py -v` + +- [ ] **Step 3: Implement gate.py** + +从 TRM4 `eprocess.py` 原样迁移全部代码（163 行）。变更仅限： +- import 路径：`core.harness.eprocess` → `core.evolution.gate` +- `GateParams`/`GateVerdict` 从 `core.evolution.types` 导入（不在 gate.py 定义） + +**保真校验**：逐行比对 TRM4 `eprocess.py`，确保 `_WALD_WIN`/`_WALD_LOSS`/`_SHRINK_PSEUDO` 常量值、log 空间公式、对称性技巧、四出口优先级链、futility best-case 检查全部保留。 + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/gate.py tests/unit/test_gate.py +git commit -m "feat(evolution): gate.py — CE-Gate e-process (#5)" +``` + +--- + +### Task 3: patch.py — 补丁引擎（算法 #9 局部） + +**Files:** +- Create: `core/evolution/patch.py` +- Test: `tests/unit/test_patch.py` +- Source: TRM4 `patch.py` (427 行，原样迁移，零有意变更) + +- [ ] **Step 1: Write patch tests** + +```python +"""tests/unit/test_patch.py""" +import pytest +from core.evolution.patch import ( + APPENDIX_START, APPENDIX_END, MOMENTUM_START, MOMENTUM_END, + appendix_region_bounds, momentum_region_bounds, momentum_inner, + append_to_appendix, extract_appendix_notes, replace_appendix_notes, + replace_momentum, apply_patch_with_report, +) + +class TestRegionBounds: + def test_no_markers_returns_none(self): + assert appendix_region_bounds("hello") is None + assert momentum_region_bounds("hello") is None + + def test_both_markers_returns_range(self): + text = f"head\n{APPENDIX_START}\nbody\n{APPENDIX_END}\ntail" + start, end = appendix_region_bounds(text) + assert text[start:end].startswith(APPENDIX_START) + assert text[start:end].endswith(APPENDIX_END) + + def test_single_marker_raises(self): + with pytest.raises(ValueError): + appendix_region_bounds(f"head\n{APPENDIX_START}\nbody") + with pytest.raises(ValueError): + momentum_region_bounds(f"head\n{MOMENTUM_END}\nbody") + +class TestAppendix: + def test_append_creates_region(self): + result = append_to_appendix("content", ["note1"]) + assert APPENDIX_START in result + assert "- note1" in result + + def test_extract_notes(self): + text = f"{APPENDIX_START}\n- a\n- b\n{APPENDIX_END}" + assert extract_appendix_notes(text) == ["a", "b"] + + def test_replace_empty_deletes_region(self): + text = f"head\n{APPENDIX_START}\n- old\n{APPENDIX_END}\ntail" + result = replace_appendix_notes(text, []) + assert APPENDIX_START not in result + +class TestMomentum: + def test_replace_creates_region(self): + result = replace_momentum("content", "guidance text") + assert MOMENTUM_START in result + assert "guidance text" in result + + def test_marker_injection_raises(self): + with pytest.raises(ValueError): + replace_momentum("content", f"evil {MOMENTUM_START}") + + def test_empty_guidance_clears(self): + text = replace_momentum("content", "old") + result = replace_momentum(text, "") + inner = momentum_inner(result) + assert inner == "" + +class TestApplyPatch: + def test_append_before_protected(self): + content = f"body\n{APPENDIX_START}\nprotected\n{APPENDIX_END}" + edits = [{"op": "append", "target": "", "content": "new line"}] + new, report = apply_patch_with_report(content, edits, [f"{APPENDIX_START}\nprotected\n{APPENDIX_END}"]) + assert report[0]["status"].startswith("applied") + assert new.index("new line") < new.index(APPENDIX_START) + + def test_replace_in_protected_skipped(self): + protected = f"{APPENDIX_START}\nprotected\n{APPENDIX_END}" + content = f"body\n{protected}" + edits = [{"op": "replace", "target": "protected", "content": "replaced"}] + new, report = apply_patch_with_report(content, edits, [protected]) + assert report[0]["status"] == "skipped_protected" + assert "protected" in new + + def test_insert_after_fallback(self): + content = "line1\nline2" + edits = [{"op": "insert_after", "target": "nonexistent", "content": "new"}] + new, report = apply_patch_with_report(content, edits) + assert "applied_insert_after_fallback" in report[0]["status"] + + def test_delete_first_occurrence(self): + content = "a\nb\na\nc" + edits = [{"op": "delete", "target": "a", "content": ""}] + new, report = apply_patch_with_report(content, edits) + assert new.count("a") == 1 +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_patch.py -v` + +- [ ] **Step 3: Implement patch.py** + +从 TRM4 `patch.py` 原样迁移全部代码（427 行）。零有意变更——import 路径调整除外。 + +**保真校验**： +- 7 个常量值完全一致 +- `_protected_ranges` 半开区间语义 `[start, end)` +- `_in_ranges` 用 `start <= pos < end` +- append op 的 `start > 0` 过滤（跳过 frontmatter） +- insert_after 三结果（成功 / 降级 append / skip） +- target 不 strip，payload strip +- 每条 edit 前重算 ranges +- report 字段 truncation（target[:200], content[:200]） + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_patch.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/patch.py tests/unit/test_patch.py +git commit -m "feat(evolution): patch.py — patch engine with protected regions (#9)" +``` + +--- + +### Task 4: validate.py — 纯决策函数（算法 #7 局部） + +**Files:** +- Create: `core/evolution/validate.py` +- Test: `tests/unit/test_validate.py` + +- [ ] **Step 1: Write validate tests** + +```python +"""tests/unit/test_validate.py""" +from core.evolution.validate import pair_block, classify_quadrants, compute_accuracy + +class TestPairBlock: + def test_basic_flips(self): + baseline = {"q1": False, "q2": True, "q3": True} + candidate = {"q1": True, "q2": False, "q3": True} + result = pair_block(baseline, candidate, ["q1", "q2", "q3"]) + assert result.w == 1 # q1: wrong→right + assert result.l == 1 # q2: right→wrong + assert result.observed == { + "q1": (False, True), "q2": (True, False), "q3": (True, True) + } + + def test_empty(self): + result = pair_block({}, {}, []) + assert result.w == 0 and result.l == 0 + +class TestClassifyQuadrants: + def test_all_four(self): + observed = { + "q1": (False, True), # improved + "q2": (True, False), # regressed + "q3": (False, False), # persistent_fail + "q4": (True, True), # stable_success + } + qc = classify_quadrants(observed) + assert qc.improvements == ["q1"] + assert qc.regressions == ["q2"] + assert qc.persistent_fails == ["q3"] + assert qc.stable_successes == ["q4"] + + def test_sorted_within_quadrant(self): + observed = {"z": (False, True), "a": (False, True)} + qc = classify_quadrants(observed) + assert qc.improvements == ["a", "z"] + +class TestComputeAccuracy: + def test_basic(self): + assert compute_accuracy({"q1": True, "q2": False}, ["q1", "q2"]) == 0.5 + + def test_empty_raises(self): + import pytest + with pytest.raises(ZeroDivisionError): + compute_accuracy({}, []) +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_validate.py -v` + +- [ ] **Step 3: Implement validate.py** + +三个纯函数，约 70 行。从 TRM4 `validate.py` 的 `_pair_block` 和 `_classify_quadrants` 提取纯逻辑。 + +```python +"""core/evolution/validate.py — 块验证纯决策函数。""" +from core.evolution.types import PairResult, QuadrantClassification + +def pair_block( + baseline: dict[str, bool], + candidate: dict[str, bool], + question_ids: list[str], +) -> PairResult: + """逐题比对基线与候选对错，统计翻转。""" + w = l = 0 + observed: dict[str, tuple[bool, bool]] = {} + for qid in question_ids: + b, c = baseline[qid], candidate[qid] + observed[qid] = (b, c) + if not b and c: + w += 1 + elif b and not c: + l += 1 + return PairResult(w=w, l=l, observed=observed) + +def classify_quadrants( + observed: dict[str, tuple[bool, bool]], +) -> QuadrantClassification: + """按 (baseline, candidate) 四组分类，各组内 sorted。""" + improvements, regressions, persistent_fails, stable_successes = [], [], [], [] + for qid, (prev, curr) in observed.items(): + if not prev and curr: + improvements.append(qid) + elif prev and not curr: + regressions.append(qid) + elif not prev and not curr: + persistent_fails.append(qid) + else: + stable_successes.append(qid) + return QuadrantClassification( + improvements=sorted(improvements), + regressions=sorted(regressions), + persistent_fails=sorted(persistent_fails), + stable_successes=sorted(stable_successes), + ) + +def compute_accuracy( + correctness: dict[str, bool], + question_ids: list[str], +) -> float: + """纯算术：sum(correct) / len(ids)。""" + return sum(correctness[qid] for qid in question_ids) / len(question_ids) +``` + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_validate.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/validate.py tests/unit/test_validate.py +git commit -m "feat(evolution): validate.py — pure block validation decision functions (#7)" +``` + +--- + +### Task 5: diagnose.py — 指标计算与 judge 辅助函数 + +**Files:** +- Create: `core/evolution/diagnose.py`（本 Task 写 metrics 部分，约 500 行） +- Test: `tests/unit/test_diagnose.py`（本 Task 写 metrics 测试） +- Source: TRM4 `metrics.py` + `diagnose.py` 的 `attribute_error`/`classify_defect_vs_lapse` + +- [ ] **Step 1: Write metrics + attribution tests** + +```python +"""tests/unit/test_diagnose.py""" +import pytest +from core.evolution.diagnose import ( + calc_format_compliance, calc_budget_usage, + calc_confidence_calibration, calc_repeat_visit_rate, + calc_search_keyword_repetition, calc_level_jump_pattern, + calc_tool_usage, extract_json_from_response, + attribute_error, question_soft_score, aggregate_soft, +) + +class TestRuleMetrics: + def test_format_compliance_empty_returns_one(self): + assert calc_format_compliance([]) == 1.0 + + def test_budget_usage(self): + assert calc_budget_usage(5, 15) == pytest.approx(1/3) + + def test_confidence_calibration(self): + assert calc_confidence_calibration(0.9, False) == "high_conf_wrong" + assert calc_confidence_calibration(0.3, True) == "low_conf_right" + assert calc_confidence_calibration(0.6, True) == "calibrated" + + def test_repeat_visit_empty(self): + assert calc_repeat_visit_rate([]) == 0.0 + + def test_repeat_visit_all_unique(self): + assert calc_repeat_visit_rate(["a", "b", "c"]) == 0.0 + + def test_repeat_visit_all_same(self): + assert calc_repeat_visit_rate(["a", "a", "a"]) == pytest.approx(2/3) + + def test_keyword_repetition_lt2(self): + assert calc_search_keyword_repetition(["one"]) == 0.0 + + def test_keyword_repetition_max_jaccard(self): + val = calc_search_keyword_repetition(["abcdef", "abcxyz"]) + assert 0.0 < val < 1.0 + + def test_level_jump_pattern(self): + assert "L1" in calc_level_jump_pattern(["seg_L1_000", "seg_L2_001"]) + + def test_tool_usage_counts(self): + assert calc_tool_usage(["view_node", "view_node", "search_similar"]) == { + "view_node": 2, "search_similar": 1, + } + +class TestJsonExtraction: + def test_fenced_block(self): + raw = '```json\n{"key": "val"}\n```' + assert extract_json_from_response(raw) == {"key": "val"} + + def test_outermost_braces(self): + raw = 'prefix {"key": 1} suffix' + assert extract_json_from_response(raw) == {"key": 1} + + def test_non_dict_raises(self): + with pytest.raises(ValueError): + extract_json_from_response("[1,2,3]") + + def test_garbage_raises(self): + with pytest.raises(ValueError): + extract_json_from_response("not json at all") + +class TestAttributeError: + def test_extraction_failure(self): + from core.evolution.types import QuestionMetrics, SpanMetrics + span = SpanMetrics(step=1, tool_name="view_node", + extraction_completeness=0.3, hallucination_rate=0.0, + missed_info_tags=[], hallucination_tags=[]) + qm = _make_qm(correct=False, span_metrics=[span], missed_nodes=[], + evidence_sufficient=True) + ea = attribute_error(qm) + assert ea.error_type == "extraction_failure" + + def test_search_failure(self): + qm = _make_qm(correct=False, span_metrics=[], missed_nodes=["L2_001"], + evidence_sufficient=False) + ea = attribute_error(qm) + assert ea.error_type == "search_failure" + + def test_reasoning_failure(self): + qm = _make_qm(correct=False, span_metrics=[], missed_nodes=[], + evidence_sufficient=True) + ea = attribute_error(qm) + assert ea.error_type == "reasoning_failure" + + def test_mixed_fallback(self): + qm = _make_qm(correct=False, span_metrics=[], missed_nodes=[], + evidence_sufficient=False) + ea = attribute_error(qm) + assert ea.error_type == "mixed" + +class TestSoftScore: + def test_no_spans_returns_none(self): + assert question_soft_score([]) is None + + def test_aggregate_skips_none(self): + assert aggregate_soft([0.8, None, 0.6]) == pytest.approx(0.7) + + def test_aggregate_all_none(self): + assert aggregate_soft([None, None]) is None +``` + +注：`_make_qm` 是测试辅助工厂函数，构造 `QuestionMetrics` 并为非关键字段填充合理默认值。 + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v` + +- [ ] **Step 3: Implement diagnose.py metrics 部分** + +从 TRM4 `metrics.py` 迁移以下函数组： +- `extract_rule_metrics`（从 prediction dict + raw_contents 提取 7 个规则指标；confidence 优先取末步 JSON 的 `reflect.confidence`，否则 `prediction["answer_confidence"]` 默认 0.5） +- 7 个规则指标函数（`calc_format_compliance` 等）+ `_trigrams` + `_parse_json_object` + `_extract_last_confidence` +- `extract_json_from_response`（三级解析：fenced → outermost `{}` → `json_repair`） +- `_call_judge`（async 化，max_retries=2 即共 3 次，仅 ValueError 重试，API 错误直传） +- `question_soft_score` + `aggregate_soft` +- 5 个 judge 函数（`evaluate_span` 等，async 化），prompt 文件名：`diagnose_span.md`/`diagnose_missed_nodes.md`/`diagnose_skill_adherence.md`/`diagnose_confirmation_bias.md`/`diagnose_evidence_sufficiency.md` +- `compute_question_metrics`（async 化） +- `_format_trace_text`（metrics 版：thought[:100], output[:200]） + +从 TRM4 `diagnose.py` 迁移： +- `attribute_error`（归因瀑布，纯函数） +- `classify_defect_vs_lapse`（async 化，LLMProvider 替代 LLMClient） +- `_make_degraded_metrics`（worker 抛 ValueError 时生成 degraded=True 的 QuestionMetrics，judge 字段 None/空列表；其他异常直传） + +**关键变更**： +- `LLMClient` → `LLMProvider`；`response.choices[0].message.content` → `response.content` +- `_call_judge` 变 async：`await llm.chat(messages)` +- judge 函数均变 async +- `load_diagnose_prompt(prompts_dir, filename)` → 直接从 `DiagnosePrompts` 束取属性 + +**保真校验**： +- `_SPAN_EVAL_TOOLS = {"view_node", "search_similar", "observe_frame"}` +- trigram 是字符级，取 MAX（非 mean） +- `calc_format_compliance` 空返回 1.0；`calc_budget_usage` 无除零 guard（P5） +- confidence 阈值：`>=0.7` 且错 → high_conf_wrong，`<0.5` 且对 → low_conf_right +- `calc_level_jump_pattern` regex `r"_L(\d+)_"`，用 `→` 连接 +- `_call_judge` max_retries=2（共 3 次），API 错误直传 +- 归因瀑布精确顺序：extraction → search → reasoning → mixed +- defect_vs_lapse 解析失败降级 "lapse" +- `_extract_last_confidence` 任意异常返回 0.5 +- judge 返回值默认：span completeness/hallucination 默认 0.0，tags 用 `list()`，missed_nodes 非 list 返 `[]` + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/diagnose.py tests/unit/test_diagnose.py +git commit -m "feat(evolution): diagnose.py metrics + attribution (Stage 1)" +``` + +--- + +### Task 6: diagnose.py — 聚合 + 案例包 + 入口 + +**Files:** +- Modify: `core/evolution/diagnose.py`（追加 ~700 行） +- Modify: `tests/unit/test_diagnose.py`（追加聚合 + 入口测试） + +- [ ] **Step 1: Write aggregation + case pack tests** + +```python +# 追加到 tests/unit/test_diagnose.py +from unittest.mock import AsyncMock +from core.evolution.types import ( + QuestionMetrics, ErrorAttribution, SpanMetrics, + SkillCasePack, SystemCasePack, ToolCasePack, DiagnosisResult, +) +from core.evolution.diagnose import ( + aggregate_d2, aggregate_d3, aggregate_d4, aggregate_d5, + merge_system_packs, merge_tool_packs, run_diagnosis, +) + +class TestAggregation: + def test_d2_empty(self): + assert aggregate_d2([]) == {} + + def test_d5_empty_returns_zero_structure(self): + result = aggregate_d5([]) + assert "early_submit_rate" in result + assert result["early_submit_rate"] == 0.0 + +class TestMerge: + def test_merge_system_packs_none_on_empty(self): + assert merge_system_packs([]) is None + + def test_merge_system_packs_wraps_stats(self): + pack = SystemCasePack( + stats={"a": 1}, failure_cases=[], success_cases=[], + ) + merged = merge_system_packs([pack, pack]) + assert "per_step" in merged.stats + assert len(merged.stats["per_step"]) == 2 + +class TestRunDiagnosis: + def test_empty_predictions_returns_empty_result(self): + import asyncio + from core.evolution.types import DiagnosePrompts + mock_log = AsyncMock() + mock_log.get_predictions.return_value = [] + mock_log.get_traces.return_value = [] + mock_llm = AsyncMock() + mock_store = MagicMock() + mock_store.list_skill_files.return_value = [] + prompts = DiagnosePrompts( + defect_vs_lapse="", reasoning_sub="", + span_eval_system="", span_eval_user="", + missed_nodes="", skill_adherence="", + confirmation_bias="", evidence_sufficiency="", + ) + result = asyncio.run(run_diagnosis( + "run1", [], {}, mock_llm, mock_log, mock_store, prompts, + concurrency=1, + )) + assert isinstance(result, DiagnosisResult) +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py::TestAggregation tests/unit/test_diagnose.py::TestMerge tests/unit/test_diagnose.py::TestRunDiagnosis -v` + +- [ ] **Step 3: Implement aggregation + case packs + run_diagnosis** + +从 TRM4 `diagnose.py` 迁移： +- `_mean`, `_percentile` 辅助函数 +- `aggregate_d2/d3/d4/d5` +- `_build_skill_case_packs`（含 severity 函数、C3 lapse routing、single-failure fallback；成功案例 `n_success=max(2, len(failures)//2)`，acc≤0.3 按 budget 升序否则按 adherence 降序） +- `_build_system_case_pack`（`_MIN_PATTERN_COUNT=3`，3 种行为模式；成功案例要求 correct+calibrated+no_bias+0.3≤budget≤0.8，按 abs(budget-0.5) 排序） +- `_build_tool_case_packs`（`_TOOL_TARGET_FILES` 映射；低 completeness 先选最多 4 条，高 hallucination 补到总数 4 上限；成功 span 要求 completeness≥0.9 且 hallucination==0.0） +- `merge_system_packs`/`merge_tool_packs`（stats 用 `{"per_step": [...]}` 包裹） +- `_classify_reasoning_failure`（串行 pass，prompt `diagnose_reasoning_failure.md`，JSON key `type`，解析失败 → `reasoning_failure_type=None` 不中断） +- `run_diagnosis` 入口（async，Semaphore 限并发，reasoning_failure 串行 pass） + +注：`resolve_skill_file` 定义在 evolve.py（Task 7），diagnose.py 从 evolve 导入。 + +**关键变更**： +- `ThreadPoolExecutor` → `asyncio.gather` + `Semaphore(concurrency)` +- `HarnessLog` → `RunLog` Protocol（`get_predictions`/`get_traces`） +- 不写 DB（`_ensure_diagnosis_tables`/`_clear_existing`/`_insert_*` 全部移除） +- 不写 JSON 文件（`write analyses/...` 移除） +- 树数据从参数传入（非 `_load_tree_cache` 文件读） +- skill 内容从 `SkillStore` 读 +- INFRA 统计按 task/video/question 过滤范围重算，不受 stop_reason 过滤影响 + +**保真校验**： +- `_INFRA_STOP_REASONS = frozenset({"error", "parse_error"})` +- 案例包选择规则（见上述各函数描述） +- single-failure fallback：1 个 defect → lapse_note（fallback 文本 `"复核该类已有规则，避免重复此类单例失败"`） +- lapse_note 空白过滤（strip 后空则丢弃） +- `_format_trace_text`（diagnose 版不截断，与 metrics 版不同！） +- D3 `avg_steps` key 实际存 budget_usage mean（TRM4 命名不一致，保留） +- `_make_case_sample` metrics 子字典固定 key：correct/error_type/budget_usage/confidence_calibration/repeat_visit_rate/tool_usage/missed_nodes/adherence_rate/confirmation_bias/evidence_sufficient + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/diagnose.py tests/unit/test_diagnose.py +git commit -m "feat(evolution): diagnose.py aggregation + case packs + run_diagnosis (#8)" +``` + +--- + +### Task 7: evolve.py — 验证 + 辅助函数 + +**Files:** +- Create: `core/evolution/evolve.py`（本 Task 写验证 + 辅助部分，约 400 行） +- Test: `tests/unit/test_evolve.py` +- Source: TRM4 `evolve.py` 的 `validate_*`/`rank_and_clip`/`edit_budget_at`/`_resolve_skill_file` 等 + +- [ ] **Step 1: Write evolve validation + helpers tests** + +```python +"""tests/unit/test_evolve.py""" +import pytest +from core.evolution.evolve import ( + validate_skill, validate_system, validate_tool, + edit_budget_at, resolve_skill_file, +) + +class TestValidateSkill: + def test_identical_passes(self): + content = "---\nname: test\ndescription: d\ntask_type: t\n---\nbody" + result = validate_skill(content, content) + assert result.passed + + def test_changed_frontmatter_fails(self): + orig = "---\nname: a\ndescription: d\ntask_type: t\n---\nbody" + evol = "---\nname: b\ndescription: d\ntask_type: t\n---\nbody" + result = validate_skill(orig, evol) + assert not result.passed + + def test_length_ratio_too_short_fails(self): + orig = "---\nname: a\ndescription: d\ntask_type: t\n---\n" + "x" * 1000 + evol = "---\nname: a\ndescription: d\ntask_type: t\n---\nshort" + result = validate_skill(orig, evol) + assert not result.passed + +class TestValidateSystem: + def test_identical_passes(self): + content = "intro\n## 能力边界\nfrozen\n## 输出格式\nfrozen2\n## other\nbody" + result = validate_system(content, content) + assert result.passed + + def test_changed_frozen_section_fails(self): + orig = "intro\n## 能力边界\noriginal\n## other\nbody" + evol = "intro\n## 能力边界\nchanged\n## other\nbody" + result = validate_system(orig, evol) + assert not result.passed + +class TestValidateTool: + def test_identical_passes(self): + extract = "## 输出格式\nfixed\n## other\nbody" + verify = "## 输出格式\nfixed2\n## other\nbody2" + result = validate_tool(extract, extract, verify, verify) + assert result.passed + + def test_no_code_block_check(self): + extract = "## 输出格式\nfixed\n```\nunclosed" + result = validate_tool(extract, extract, "v", "v") + assert result.passed # tool 不检查代码块闭合 + +class TestEditBudget: + def test_start_at_zero(self): + assert edit_budget_at(0, 100, 5, 2) == 5 + + def test_end_at_total(self): + assert edit_budget_at(100, 100, 5, 2) == 2 + + def test_total_steps_one(self): + assert edit_budget_at(0, 1, 5, 2) == 5 + + def test_start_less_than_end_asserts(self): + with pytest.raises(AssertionError): + edit_budget_at(0, 100, 2, 5) + +class TestResolveSkillFile: + def test_direct_match(self): + class FakeStore: + def list_skill_files(self): return ["action-reasoning.md", "default-strategy.md"] + def read_skill(self, f): return "" + assert resolve_skill_file(FakeStore(), "Action Reasoning") == "action-reasoning.md" + + def test_fallback_to_default(self): + class FakeStore: + def list_skill_files(self): return ["default-strategy.md"] + def read_skill(self, f): return "" + assert resolve_skill_file(FakeStore(), "Unknown Type") == "default-strategy.md" +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v` + +- [ ] **Step 3: Implement evolve.py validation + helpers** + +从 TRM4 `evolve.py` 迁移： +- `_parse_frontmatter`、`_strip_appendix_region`（宽容版）、`_strip_momentum_region`（严格版）、`_strip_protected_regions` +- `_check_length`（去 appendix+momentum 后比较，ratio [0.3, 2.0]） +- `_check_code_blocks` +- `_extract_section` +- `_skill_protected_spans`/`_system_protected_spans`/`_tool_protected_spans` +- `validate_skill`/`validate_system`/`validate_tool` → 返回 `ValidationResult`（定义在此文件内部，非 types.py） +- `edit_budget_at`（纯数学，保持 TRM4 断言和 banker's rounding） +- `rank_and_clip`（async 化，`type(idx) is int` 排除 bool） +- `_select_top_edits` +- `_parse_llm_json`（两级：fenced → json.loads，失败返回 None） +- `resolve_skill_file`（接受 `SkillStore` 而非 `Path`） +- `_format_case_samples`（tool_output[:500] 截断） +- `_format_spans`（tool_output[:500] 截断） +- `_format_rejected_edits` + +**关键变更**： +- `LLMClient` → `LLMProvider` +- `_resolve_skill_file(skills_dir: Path, ...)` → `resolve_skill_file(skill_store: SkillStore, ...)` +- `rank_and_clip` 变 async + +**保真校验**： +- 冻结区配置：Skill(frontmatter+appendix+momentum)、System(3 sections+appendix)、Tool(输出格式+appendix) +- frontmatter 三字段：name/description/task_type +- `_parse_frontmatter` regex 必须从文件开头匹配 `^---\n...\n---`，`yaml.safe_load` 失败返回 None +- `_strip_appendix_region` 宽容 vs `_strip_momentum_region` 严格（不对称保留） +- `_parse_llm_json`：只匹配 ` ```json ` fenced block（非 ``` 不带 json）→ `json.loads`，失败返回 None（与 metrics 的三级不同！） +- validate_tool 不检查代码块闭合（与 skill/system 不同） +- `type(idx) is int`（非 isinstance） +- `rank_and_clip`/`_request_rank_indices`：rank LLM ValueError 降级，API 异常不捕获 +- `rewrite_from_suggestions`：prompt `evolve_rewrite.md`，JSON key `rewritten`，重写不得长于原文，只捕 ValueError/KeyError/TypeError/AttributeError +- `_format_case_samples` tool_output[:500] 截断 +- `_format_rejected_edits` gate 证据格式 `W=... L=... E={:.2f} δ̂={:+.3f}` + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/evolve.py tests/unit/test_evolve.py +git commit -m "feat(evolution): evolve.py validation + helpers" +``` + +--- + +### Task 8: evolve.py — per-target 进化 + +**Files:** +- Modify: `core/evolution/evolve.py`（追加 ~500 行） +- Modify: `tests/unit/test_evolve.py`（追加进化测试） + +- [ ] **Step 1: Write evolution loop tests** + +```python +# 追加到 tests/unit/test_evolve.py +import asyncio +from unittest.mock import AsyncMock, MagicMock +from core.evolution.types import ( + SkillCasePack, SystemCasePack, ToolCasePack, + EvolutionRecord, EvolvePrompts, +) +from core.evolution.evolve import ( + evolve_single_skill, evolve_system_prompt, evolve_single_tool, + consolidate_appendix, +) + +_PROMPTS = EvolvePrompts( + evolve_skill="sk", evolve_system="sys", evolve_tool="tool", + evolve_rank="rank", consolidate_system="cons", +) + +def _make_fake_llm(response_content: str): + """构造返回固定内容的假 LLMProvider。""" + from core.types import LLMResponse + mock = AsyncMock() + mock.chat.return_value = LLMResponse( + content=response_content, thinking="", model="test", + provider="test", prompt_tokens=0, completion_tokens=0, + latency_ms=0, ttft_ms=None, max_inter_token_ms=None, + cache_hit=False, call_id="test-id", + ) + return mock + +class TestEvolveSingleSkill: + def test_empty_pack_skipped(self): + pack = SkillCasePack( + task_type="test", target_file="test.md", + stats={}, failure_cases=[], success_cases=[], lapse_notes=[], + ) + store = MagicMock() + store.read_skill.return_value = "---\nname: t\ndescription: d\ntask_type: t\n---\nbody" + store.list_skill_files.return_value = ["test.md"] + llm = _make_fake_llm('{"suggestions":[],"edits":[]}') + record = asyncio.run(evolve_single_skill( + llm, pack, store, _PROMPTS, "v1", 5, 6, + )) + assert record.status in ("rejected", "skipped") + +class TestEvolveSystemPrompt: + def test_no_failures_returns_skipped(self): + pack = SystemCasePack(stats={}, failure_cases=[], success_cases=[]) + store = MagicMock() + store.read_prompt.return_value = "## 能力边界\nfixed\n## 输出格式\nfixed\n## 视频树结构\nfixed\nbody" + llm = _make_fake_llm('{"suggestions":[],"edits":[]}') + record = asyncio.run(evolve_system_prompt( + llm, pack, store, _PROMPTS, "v1", 5, + )) + assert record.status in ("rejected", "skipped") + +class TestEvolveSingleTool: + def test_evolved_content_is_json(self): + pack = ToolCasePack( + tool_name="view_node", + target_files=["view_node_extract.md", "view_node_verify.md"], + stats={}, failure_spans=[], success_spans=[], + ) + store = MagicMock() + store.read_prompt.return_value = "## 输出格式\nfixed\nbody" + llm = _make_fake_llm('{"suggestions":[],"edits":[]}') + record = asyncio.run(evolve_single_tool( + llm, pack, store, _PROMPTS, "v1", 5, + )) + import json + parsed = json.loads(record.evolved_content) + assert "extract" in parsed and "verify" in parsed + +class TestConsolidateAppendix: + def test_single_note_passthrough(self): + llm = _make_fake_llm("") + result = asyncio.run(consolidate_appendix(llm, ["note1"])) + assert result == ["note1"] + + def test_exception_returns_original(self): + llm = AsyncMock() + llm.chat.side_effect = RuntimeError("boom") + result = asyncio.run(consolidate_appendix(llm, ["a", "b", "c"])) + assert result == ["a", "b", "c"] +``` + +- [ ] **Step 2: Run test — expect FAIL** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py::TestEvolveSingleSkill tests/unit/test_evolve.py::TestEvolveSystemPrompt tests/unit/test_evolve.py::TestEvolveSingleTool tests/unit/test_evolve.py::TestConsolidateAppendix -v` + +- [ ] **Step 3: Implement per-target evolution** + +从 TRM4 `evolve.py` 迁移： +- `_run_patch_evolution_loop`（async 化，`range(2)` 两轮，三种失败反馈） +- `_build_lapse_only_attempt`（合成 `applied_append` report） +- `evolve_single_skill`（三分支：lapse-only/rewrite/patch + appendix 追加 + consolidation） +- `evolve_system_prompt`（无 lapse、无 rewrite、无 appendix） +- `evolve_single_tool`（extract+verify 合池 `_src` 标记、shared budget、JSON evolved_content） +- `consolidate_appendix`（async 化，四守卫） +- `rewrite_from_suggestions`（async 化，3 个拒绝条件） +- `_append_lapse_with_consolidation`（`>= threshold` 触发、G4 `>=` 拒绝等长） + +**关键变更**： +- 全部 `client.chat()` → `await llm.chat(messages)` +- `response.choices[0].message.content` → `response.content` +- `skills_dir / target_file` → `skill_store.read_skill(target_file)` +- `prompts_dir / filename` → `prompt_store.read_prompt(filename)` +- 版本写入（`advance_version`/copytree）全部移除——返回 `EvolutionRecord` +- `run_evolution` 编排移除——per-target 函数是最高粒度 + +**保真校验**： +- Skill 三分支精确条件和行为 +- `_build_lapse_only_attempt` 合成 `applied_append` 状态 +- rank_and_clip 三级降级 +- Tool `_src` 标记合池/拆回 +- consolidate 四守卫（G4 在调用方） +- rewrite 长度限制（重写不得长于原文）、异常类型（仅捕 ValueError/KeyError/TypeError/AttributeError） +- 两轮重试反馈文本 + +- [ ] **Step 4: Run test — expect PASS** + +Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v` + +- [ ] **Step 5: Commit** + +``` +git add core/evolution/evolve.py tests/unit/test_evolve.py +git commit -m "feat(evolution): evolve.py per-target evolution — skill/system/tool (#9)" +``` + +--- + +### Task 9: __init__.py + 集成 + lint + +**Files:** +- Modify: `core/evolution/__init__.py` +- Run: lint + dependency check + +- [ ] **Step 1: Write __init__.py public API** + +```python +"""core/evolution/ — 自进化循环决策内核。""" +from core.evolution.gate import compute_e_value, gate_decision, probation_verdict +from core.evolution.patch import ( + apply_patch_with_report, append_to_appendix, + extract_appendix_notes, replace_appendix_notes, + replace_momentum, momentum_inner, +) +from core.evolution.validate import pair_block, classify_quadrants, compute_accuracy +from core.evolution.diagnose import run_diagnosis +from core.evolution.evolve import ( + evolve_single_skill, evolve_system_prompt, evolve_single_tool, + edit_budget_at, resolve_skill_file, +) + +__all__ = [ + "compute_e_value", "gate_decision", "probation_verdict", + "apply_patch_with_report", "append_to_appendix", + "extract_appendix_notes", "replace_appendix_notes", + "replace_momentum", "momentum_inner", + "pair_block", "classify_quadrants", "compute_accuracy", + "run_diagnosis", + "evolve_single_skill", "evolve_system_prompt", "evolve_single_tool", + "edit_budget_at", "resolve_skill_file", +] +``` + +- [ ] **Step 2: Run lint** + +```bash +conda activate Video-Tree-TRM & ruff check core/evolution/ --fix +conda activate Video-Tree-TRM & ruff format core/evolution/ +``` + +- [ ] **Step 3: Dependency direction check** + +```bash +# core/evolution/ 不得 import app/ 或 adapters/ +grep -rn "from app\." core/evolution/ && echo "VIOLATION" || echo "OK" +grep -rn "from adapters\." core/evolution/ && echo "VIOLATION" || echo "OK" +grep -rn "import app\." core/evolution/ && echo "VIOLATION" || echo "OK" +grep -rn "import adapters\." core/evolution/ && echo "VIOLATION" || echo "OK" +``` + +Expected: 全部 OK + +- [ ] **Step 4: Update ARCHITECTURE.md §3.1** + +将 SkillStore/PromptStore/RunLog 的 Protocol 定义从"含写方法"修订为"core/ 只读，写方法在 app/ 实现类"。同步设计文档 §3 的修订说明。 + +- [ ] **Step 5: Run full test suite** + +```bash +conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py tests/unit/test_patch.py tests/unit/test_validate.py tests/unit/test_diagnose.py tests/unit/test_evolve.py tests/unit/test_evolution_types.py -v --tb=short +``` + +- [ ] **Step 6: Commit** + +``` +git add core/evolution/__init__.py research-wiki/ARCHITECTURE.md +git commit -m "feat(evolution): __init__.py public API + ARCHITECTURE.md Protocol update" +``` + +--- + +## Algorithm Fidelity Check + +本计划涉及 4 项核心算法迁移： + +| # | 算法 | 计划 Task | 保真措施 | +|---|------|----------|---------| +| 5 | CE-Gate e-process | Task 2 | 原样迁移 163 行，零有意变更；测试覆盖边界（W=L=0、负值、重 win/loss） | +| 7 | 块顺序验证 | Task 4 | 纯决策函数提取；编排留 app/harness/（Design B 约定） | +| 8 | 诊断瀑布 | Task 5-6 | 归因瀑布顺序、defect/lapse 分类、案例包选择规则逐条保真 | +| 9 | 进化 patch 引擎 | Task 3, 7-8 | patch.py 原样迁移；evolve.py 三分支/rank_clip/consolidate 四守卫逐条保真 | + +不涉及的算法：#1-4（建树/检索器）、#6（信息阶梯，app/harness/）、#10（mini-batch，app/harness/）、#11（Agent Loop，已迁移）、#12（树环境语义搜索，已迁移）、#13（训练循环编排，app/harness/）。