iomgaa/Video-Tree-TRM5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4e241aa5819b8ab33bc669cdf391b7266025fbb7
Video-Tree-TRM5/research-wiki/plans/2026-07-07-core-evolution.md
T

1146 lines
44 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# core/evolution/ Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Migrate TRM4 evolution engine into a Clean Architecture extractable kernel (`core/evolution/`).
**Architecture:** 7 files, dependency-ordered. Pure decision logic only — no DB writes, no filesystem versioning, no inference orchestration. All external I/O through 3 Protocols + 1 existing LLMProvider. Async-first (asyncio.gather + Semaphore).
**Tech Stack:** Python 3.11, asyncio, scipy.special, json_repair, loguru, pluggy (already in project)
**Design doc:** `research-wiki/designs/2026-07-07-core-evolution-design.md`
**TRM4 source:** `/home/iomgaa/Projects/Video-Tree-TRM4/core/harness/`
---
## File Structure
| File | Lines (est.) | Creates | Depends on |
|------|-------------|---------|-----------|
| `core/evolution/protocols.py` | 60 | New | `core/evolution/types.py` (TYPE_CHECKING) |
| `core/evolution/types.py` | 280 | New | — |
| `core/evolution/gate.py` | 170 | New | `types.py` |
| `core/evolution/patch.py` | 440 | New | — (loguru only) |
| `core/evolution/validate.py` | 70 | New | `gate.py`, `types.py` |
| `core/evolution/diagnose.py` | 1200 | New | `types.py`, `protocols.py`, `core/protocols.py` |
| `core/evolution/evolve.py` | 900 | New | `types.py`, `protocols.py`, `patch.py`, `core/protocols.py` |
| `core/evolution/__init__.py` | 30 | Modify | all above |
| `tests/unit/test_gate.py` | 200 | New | — |
| `tests/unit/test_patch.py` | 350 | New | — |
| `tests/unit/test_validate.py` | 120 | New | — |
| `tests/unit/test_diagnose.py` | 400 | New | — |
| `tests/unit/test_evolve.py` | 400 | New | — |
---
### Task 1: protocols.py + types.py（基础层）
**Files:**
- Create: `core/evolution/protocols.py`
- Create: `core/evolution/types.py`
- Test: `tests/unit/test_evolution_types.py`
- [ ] **Step 1: Write type construction tests**
```python
"""tests/unit/test_evolution_types.py"""
from core.evolution.types import (
GateParams, GateVerdict, SpanMetrics, SkillStepAdherence,
QuestionMetrics, ErrorAttribution, CaseSample,
SkillCasePack, SystemCasePack, ToolCasePack, DiagnosisResult,
EvolutionRecord, RejectedEdit, EvolutionResult,
PairResult, QuadrantClassification,
DiagnosePrompts, EvolvePrompts,
)
def test_gate_params_frozen():
p = GateParams(e_confirm=20.0, e_provisional=3.0, w_net_min=2,
delta_min=0.02, lambda_dir=-0.642, e_rollback=10.0)
assert p.e_confirm == 20.0
import pytest
with pytest.raises(AttributeError):
p.e_confirm = 1.0
def test_evolution_record_mutable():
r = EvolutionRecord(
target_file="test.md", target_type="skill",
original_content="a", evolved_content="b",
reason="test", status="accepted", source_version="v1",
suggestions=[], edits=[], apply_report=[], clip_info={},
)
r.status = "rejected"
assert r.status == "rejected"
def test_diagnose_prompts_frozen():
dp = DiagnosePrompts(
defect_vs_lapse="p1", reasoning_sub="p2",
span_eval_system="p3", span_eval_user="p4",
missed_nodes="p5", skill_adherence="p6",
confirmation_bias="p7", evidence_sufficiency="p8",
)
assert dp.defect_vs_lapse == "p1"
```
- [ ] **Step 2: Run test — expect FAIL (ImportError)**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolution_types.py -v`
- [ ] **Step 3: Implement types.py**
从 TRM4 迁移所有 dataclass。关键变更：
| TRM4 位置 | TRM5 变更 |
|-----------|---------|
| `eprocess.py::GateParams/GateVerdict` | 原样迁移 |
| `diagnose.py::SpanMetrics` 等 9 个 | 全部标 `frozen=True`（TRM4 中 QuestionMetrics 非 frozen，TRM5 一次性构造） |
| `evolve.py::EvolutionRecord` | 保持 mutable，新增 `result_version: str | None = None` 字段 |
| `evolve.py::RejectedEdit` | 原样迁移，frozen |
| `evolve.py::EvolutionResult` | 移除 `skills_version`/`prompts_version`（app/ 职责） |
| `validate.py::ValidationOutcome/Probation/InferenceRunConfig` | 不迁——属 app/harness/ |
| 新增 `PairResult`/`QuadrantClassification` | 块验证纯决策输出 |
| 新增 `DiagnosePrompts`/`EvolvePrompts` | 模板束 |
**保真校验点**：逐字段对比 TRM4 dataclass，确保无遗漏字段。特别注意 `DiagnosisResult` 的完整字段列表（约 20 个字段）。
- [ ] **Step 4: Implement protocols.py**
```python
"""core/evolution/protocols.py — 3 个只读 Protocol"""
from __future__ import annotations
from typing import Any, Protocol, runtime_checkable
@runtime_checkable
class SkillStore(Protocol):
def read_skill(self, filename: str) -> str: ...
def list_skill_files(self) -> list[str]: ...
@runtime_checkable
class PromptStore(Protocol):
def read_prompt(self, filename: str) -> str: ...
def list_prompt_files(self) -> list[str]: ...
@runtime_checkable
class RunLog(Protocol):
async def get_predictions(
self, run_id: str, *, question_ids: list[str] | None = None,
) -> list[dict[str, Any]]: ...
async def get_traces(
self, run_id: str, *, question_ids: list[str] | None = None,
) -> list[dict[str, Any]]: ...
```
- [ ] **Step 5: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolution_types.py -v`
- [ ] **Step 6: Commit**
```
git add core/evolution/types.py core/evolution/protocols.py tests/unit/test_evolution_types.py
git commit -m "feat(evolution): types.py + protocols.py — foundation dataclasses and Protocol ports"
```
---
### Task 2: gate.py — CE-Gate e-process（算法 #5）
**Files:**
- Create: `core/evolution/gate.py`
- Test: `tests/unit/test_gate.py`
- Source: TRM4 `eprocess.py` (163 行，原样迁移，零有意变更)
- [ ] **Step 1: Write gate tests**
```python
"""tests/unit/test_gate.py"""
import math
import pytest
from core.evolution.gate import compute_e_value, gate_decision, probation_verdict
from core.evolution.types import GateParams, GateVerdict
_PARAMS = GateParams(
e_confirm=20.0, e_provisional=3.0, w_net_min=2,
delta_min=0.02, lambda_dir=-0.642, e_rollback=10.0,
)
class TestComputeEValue:
def test_zero_zero_returns_one(self):
assert compute_e_value(0, 0) == pytest.approx(1.0)
def test_negative_w_raises(self):
with pytest.raises(ValueError):
compute_e_value(-1, 0)
def test_negative_l_raises(self):
with pytest.raises(ValueError):
compute_e_value(0, -1)
def test_heavy_loss_returns_near_zero(self):
assert compute_e_value(0, 20) < 0.01
def test_heavy_win_returns_large(self):
assert compute_e_value(10, 0) > 100
def test_symmetric(self):
e_5_3 = compute_e_value(5, 3)
e_3_5 = compute_e_value(3, 5)
assert e_5_3 > e_3_5
class TestGateDecision:
def test_confirmed_needs_both_e_and_delta(self):
v = gate_decision(10, 0, 10, 10, params=_PARAMS)
assert v.decision == "accept_confirmed"
def test_continue_on_balanced(self):
v = gate_decision(3, 3, 6, 20, params=_PARAMS)
assert v.decision == "continue"
def test_reject_inertia_on_exhaustion(self):
v = gate_decision(1, 1, 2, 0, params=_PARAMS)
assert v.decision == "reject_inertia"
def test_n_used_zero_raises(self):
with pytest.raises(ValueError):
gate_decision(0, 0, 0, 10, params=_PARAMS)
def test_n_remaining_negative_raises(self):
with pytest.raises(ValueError):
gate_decision(1, 0, 1, -1, params=_PARAMS)
class TestProbationVerdict:
def test_strong_win_confirmed(self):
assert probation_verdict(10, 0, params=_PARAMS) == "confirmed"
def test_strong_loss_rollback(self):
assert probation_verdict(0, 10, params=_PARAMS) == "rollback"
def test_balanced_unverified(self):
assert probation_verdict(3, 3, params=_PARAMS) == "unverified"
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py -v`
- [ ] **Step 3: Implement gate.py**
从 TRM4 `eprocess.py` 原样迁移全部代码（163 行）。变更仅限：
- import 路径：`core.harness.eprocess` → `core.evolution.gate`
- `GateParams`/`GateVerdict` 从 `core.evolution.types` 导入（不在 gate.py 定义）
**保真校验**：逐行比对 TRM4 `eprocess.py`，确保 `_WALD_WIN`/`_WALD_LOSS`/`_SHRINK_PSEUDO` 常量值、log 空间公式、对称性技巧、四出口优先级链、futility best-case 检查全部保留。
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/gate.py tests/unit/test_gate.py
git commit -m "feat(evolution): gate.py — CE-Gate e-process (#5)"
```
---
### Task 3: patch.py — 补丁引擎（算法 #9 局部）
**Files:**
- Create: `core/evolution/patch.py`
- Test: `tests/unit/test_patch.py`
- Source: TRM4 `patch.py` (427 行，原样迁移，零有意变更)
- [ ] **Step 1: Write patch tests**
```python
"""tests/unit/test_patch.py"""
import pytest
from core.evolution.patch import (
APPENDIX_START, APPENDIX_END, MOMENTUM_START, MOMENTUM_END,
appendix_region_bounds, momentum_region_bounds, momentum_inner,
append_to_appendix, extract_appendix_notes, replace_appendix_notes,
replace_momentum, apply_patch_with_report,
)
class TestRegionBounds:
def test_no_markers_returns_none(self):
assert appendix_region_bounds("hello") is None
assert momentum_region_bounds("hello") is None
def test_both_markers_returns_range(self):
text = f"head\n{APPENDIX_START}\nbody\n{APPENDIX_END}\ntail"
start, end = appendix_region_bounds(text)
assert text[start:end].startswith(APPENDIX_START)
assert text[start:end].endswith(APPENDIX_END)
def test_single_marker_raises(self):
with pytest.raises(ValueError):
appendix_region_bounds(f"head\n{APPENDIX_START}\nbody")
with pytest.raises(ValueError):
momentum_region_bounds(f"head\n{MOMENTUM_END}\nbody")
class TestAppendix:
def test_append_creates_region(self):
result = append_to_appendix("content", ["note1"])
assert APPENDIX_START in result
assert "- note1" in result
def test_extract_notes(self):
text = f"{APPENDIX_START}\n- a\n- b\n{APPENDIX_END}"
assert extract_appendix_notes(text) == ["a", "b"]
def test_replace_empty_deletes_region(self):
text = f"head\n{APPENDIX_START}\n- old\n{APPENDIX_END}\ntail"
result = replace_appendix_notes(text, [])
assert APPENDIX_START not in result
class TestMomentum:
def test_replace_creates_region(self):
result = replace_momentum("content", "guidance text")
assert MOMENTUM_START in result
assert "guidance text" in result
def test_marker_injection_raises(self):
with pytest.raises(ValueError):
replace_momentum("content", f"evil {MOMENTUM_START}")
def test_empty_guidance_clears(self):
text = replace_momentum("content", "old")
result = replace_momentum(text, "")
inner = momentum_inner(result)
assert inner == ""
class TestApplyPatch:
def test_append_before_protected(self):
content = f"body\n{APPENDIX_START}\nprotected\n{APPENDIX_END}"
edits = [{"op": "append", "target": "", "content": "new line"}]
new, report = apply_patch_with_report(content, edits, [f"{APPENDIX_START}\nprotected\n{APPENDIX_END}"])
assert report[0]["status"].startswith("applied")
assert new.index("new line") < new.index(APPENDIX_START)
def test_replace_in_protected_skipped(self):
protected = f"{APPENDIX_START}\nprotected\n{APPENDIX_END}"
content = f"body\n{protected}"
edits = [{"op": "replace", "target": "protected", "content": "replaced"}]
new, report = apply_patch_with_report(content, edits, [protected])
assert report[0]["status"] == "skipped_protected"
assert "protected" in new
def test_insert_after_fallback(self):
content = "line1\nline2"
edits = [{"op": "insert_after", "target": "nonexistent", "content": "new"}]
new, report = apply_patch_with_report(content, edits)
assert "applied_insert_after_fallback" in report[0]["status"]
def test_delete_first_occurrence(self):
content = "a\nb\na\nc"
edits = [{"op": "delete", "target": "a", "content": ""}]
new, report = apply_patch_with_report(content, edits)
assert new.count("a") == 1
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_patch.py -v`
- [ ] **Step 3: Implement patch.py**
从 TRM4 `patch.py` 原样迁移全部代码（427 行）。零有意变更——import 路径调整除外。
**保真校验**：
- 7 个常量值完全一致
- `_protected_ranges` 半开区间语义 `[start, end)`
- `_in_ranges` 用 `start <= pos < end`
- append op 的 `start > 0` 过滤（跳过 frontmatter）
- insert_after 三结果（成功 / 降级 append / skip）
- target 不 strip，payload strip
- 每条 edit 前重算 ranges
- report 字段 truncation（target[:200], content[:200]）
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_patch.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/patch.py tests/unit/test_patch.py
git commit -m "feat(evolution): patch.py — patch engine with protected regions (#9)"
```
---
### Task 4: validate.py — 纯决策函数（算法 #7 局部）
**Files:**
- Create: `core/evolution/validate.py`
- Test: `tests/unit/test_validate.py`
- [ ] **Step 1: Write validate tests**
```python
"""tests/unit/test_validate.py"""
from core.evolution.validate import pair_block, classify_quadrants, compute_accuracy
class TestPairBlock:
def test_basic_flips(self):
baseline = {"q1": False, "q2": True, "q3": True}
candidate = {"q1": True, "q2": False, "q3": True}
result = pair_block(baseline, candidate, ["q1", "q2", "q3"])
assert result.w == 1 # q1: wrong→right
assert result.l == 1 # q2: right→wrong
assert result.observed == {
"q1": (False, True), "q2": (True, False), "q3": (True, True)
}
def test_empty(self):
result = pair_block({}, {}, [])
assert result.w == 0 and result.l == 0
class TestClassifyQuadrants:
def test_all_four(self):
observed = {
"q1": (False, True), # improved
"q2": (True, False), # regressed
"q3": (False, False), # persistent_fail
"q4": (True, True), # stable_success
}
qc = classify_quadrants(observed)
assert qc.improvements == ["q1"]
assert qc.regressions == ["q2"]
assert qc.persistent_fails == ["q3"]
assert qc.stable_successes == ["q4"]
def test_sorted_within_quadrant(self):
observed = {"z": (False, True), "a": (False, True)}
qc = classify_quadrants(observed)
assert qc.improvements == ["a", "z"]
class TestComputeAccuracy:
def test_basic(self):
assert compute_accuracy({"q1": True, "q2": False}, ["q1", "q2"]) == 0.5
def test_empty_raises(self):
import pytest
with pytest.raises(ZeroDivisionError):
compute_accuracy({}, [])
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_validate.py -v`
- [ ] **Step 3: Implement validate.py**
三个纯函数，约 70 行。从 TRM4 `validate.py` 的 `_pair_block` 和 `_classify_quadrants` 提取纯逻辑。
```python
"""core/evolution/validate.py — 块验证纯决策函数。"""
from core.evolution.types import PairResult, QuadrantClassification
def pair_block(
baseline: dict[str, bool],
candidate: dict[str, bool],
question_ids: list[str],
) -> PairResult:
"""逐题比对基线与候选对错，统计翻转。"""
w = l = 0
observed: dict[str, tuple[bool, bool]] = {}
for qid in question_ids:
b, c = baseline[qid], candidate[qid]
observed[qid] = (b, c)
if not b and c:
w += 1
elif b and not c:
l += 1
return PairResult(w=w, l=l, observed=observed)
def classify_quadrants(
observed: dict[str, tuple[bool, bool]],
) -> QuadrantClassification:
"""按 (baseline, candidate) 四组分类，各组内 sorted。"""
improvements, regressions, persistent_fails, stable_successes = [], [], [], []
for qid, (prev, curr) in observed.items():
if not prev and curr:
improvements.append(qid)
elif prev and not curr:
regressions.append(qid)
elif not prev and not curr:
persistent_fails.append(qid)
else:
stable_successes.append(qid)
return QuadrantClassification(
improvements=sorted(improvements),
regressions=sorted(regressions),
persistent_fails=sorted(persistent_fails),
stable_successes=sorted(stable_successes),
)
def compute_accuracy(
correctness: dict[str, bool],
question_ids: list[str],
) -> float:
"""纯算术：sum(correct) / len(ids)。"""
return sum(correctness[qid] for qid in question_ids) / len(question_ids)
```
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_validate.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/validate.py tests/unit/test_validate.py
git commit -m "feat(evolution): validate.py — pure block validation decision functions (#7)"
```
---
### Task 5: diagnose.py — 指标计算与 judge 辅助函数
**Files:**
- Create: `core/evolution/diagnose.py`（本 Task 写 metrics 部分，约 500 行）
- Test: `tests/unit/test_diagnose.py`（本 Task 写 metrics 测试）
- Source: TRM4 `metrics.py` + `diagnose.py` 的 `attribute_error`/`classify_defect_vs_lapse`
- [ ] **Step 1: Write metrics + attribution tests**
```python
"""tests/unit/test_diagnose.py"""
import pytest
from core.evolution.diagnose import (
calc_format_compliance, calc_budget_usage,
calc_confidence_calibration, calc_repeat_visit_rate,
calc_search_keyword_repetition, calc_level_jump_pattern,
calc_tool_usage, extract_json_from_response,
attribute_error, question_soft_score, aggregate_soft,
)
class TestRuleMetrics:
def test_format_compliance_empty_returns_one(self):
assert calc_format_compliance([]) == 1.0
def test_budget_usage(self):
assert calc_budget_usage(5, 15) == pytest.approx(1/3)
def test_confidence_calibration(self):
assert calc_confidence_calibration(0.9, False) == "high_conf_wrong"
assert calc_confidence_calibration(0.3, True) == "low_conf_right"
assert calc_confidence_calibration(0.6, True) == "calibrated"
def test_repeat_visit_empty(self):
assert calc_repeat_visit_rate([]) == 0.0
def test_repeat_visit_all_unique(self):
assert calc_repeat_visit_rate(["a", "b", "c"]) == 0.0
def test_repeat_visit_all_same(self):
assert calc_repeat_visit_rate(["a", "a", "a"]) == pytest.approx(2/3)
def test_keyword_repetition_lt2(self):
assert calc_search_keyword_repetition(["one"]) == 0.0
def test_keyword_repetition_max_jaccard(self):
val = calc_search_keyword_repetition(["abcdef", "abcxyz"])
assert 0.0 < val < 1.0
def test_level_jump_pattern(self):
assert "L1" in calc_level_jump_pattern(["seg_L1_000", "seg_L2_001"])
def test_tool_usage_counts(self):
assert calc_tool_usage(["view_node", "view_node", "search_similar"]) == {
"view_node": 2, "search_similar": 1,
}
class TestJsonExtraction:
def test_fenced_block(self):
raw = '```json\n{"key": "val"}\n```'
assert extract_json_from_response(raw) == {"key": "val"}
def test_outermost_braces(self):
raw = 'prefix {"key": 1} suffix'
assert extract_json_from_response(raw) == {"key": 1}
def test_non_dict_raises(self):
with pytest.raises(ValueError):
extract_json_from_response("[1,2,3]")
def test_garbage_raises(self):
with pytest.raises(ValueError):
extract_json_from_response("not json at all")
class TestAttributeError:
def test_extraction_failure(self):
from core.evolution.types import QuestionMetrics, SpanMetrics
span = SpanMetrics(step=1, tool_name="view_node",
extraction_completeness=0.3, hallucination_rate=0.0,
missed_info_tags=[], hallucination_tags=[])
qm = _make_qm(correct=False, span_metrics=[span], missed_nodes=[],
evidence_sufficient=True)
ea = attribute_error(qm)
assert ea.error_type == "extraction_failure"
def test_search_failure(self):
qm = _make_qm(correct=False, span_metrics=[], missed_nodes=["L2_001"],
evidence_sufficient=False)
ea = attribute_error(qm)
assert ea.error_type == "search_failure"
def test_reasoning_failure(self):
qm = _make_qm(correct=False, span_metrics=[], missed_nodes=[],
evidence_sufficient=True)
ea = attribute_error(qm)
assert ea.error_type == "reasoning_failure"
def test_mixed_fallback(self):
qm = _make_qm(correct=False, span_metrics=[], missed_nodes=[],
evidence_sufficient=False)
ea = attribute_error(qm)
assert ea.error_type == "mixed"
class TestSoftScore:
def test_no_spans_returns_none(self):
assert question_soft_score([]) is None
def test_aggregate_skips_none(self):
assert aggregate_soft([0.8, None, 0.6]) == pytest.approx(0.7)
def test_aggregate_all_none(self):
assert aggregate_soft([None, None]) is None
```
注：`_make_qm` 是测试辅助工厂函数，构造 `QuestionMetrics` 并为非关键字段填充合理默认值。
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v`
- [ ] **Step 3: Implement diagnose.py metrics 部分**
从 TRM4 `metrics.py` 迁移以下函数组：
- `extract_rule_metrics`（从 prediction dict + raw_contents 提取 7 个规则指标；confidence 优先取末步 JSON 的 `reflect.confidence`，否则 `prediction["answer_confidence"]` 默认 0.5）
- 7 个规则指标函数（`calc_format_compliance` 等）+ `_trigrams` + `_parse_json_object` + `_extract_last_confidence`
- `extract_json_from_response`（三级解析：fenced → outermost `{}` → `json_repair`）
- `_call_judge`（async 化，max_retries=2 即共 3 次，仅 ValueError 重试，API 错误直传）
- `question_soft_score` + `aggregate_soft`
- 5 个 judge 函数（`evaluate_span` 等，async 化），prompt 文件名：`diagnose_span.md`/`diagnose_missed_nodes.md`/`diagnose_skill_adherence.md`/`diagnose_confirmation_bias.md`/`diagnose_evidence_sufficiency.md`
- `compute_question_metrics`（async 化）
- `_format_trace_text`（metrics 版：thought[:100], output[:200]）
从 TRM4 `diagnose.py` 迁移：
- `attribute_error`（归因瀑布，纯函数）
- `classify_defect_vs_lapse`（async 化，LLMProvider 替代 LLMClient）
- `_make_degraded_metrics`（worker 抛 ValueError 时生成 degraded=True 的 QuestionMetrics，judge 字段 None/空列表；其他异常直传）
**关键变更**：
- `LLMClient` → `LLMProvider`；`response.choices[0].message.content` → `response.content`
- `_call_judge` 变 async：`await llm.chat(messages)`
- judge 函数均变 async
- `load_diagnose_prompt(prompts_dir, filename)` → 直接从 `DiagnosePrompts` 束取属性
**保真校验**：
- `_SPAN_EVAL_TOOLS = {"view_node", "search_similar", "observe_frame"}`
- trigram 是字符级，取 MAX（非 mean）
- `calc_format_compliance` 空返回 1.0；`calc_budget_usage` 无除零 guard（P5）
- confidence 阈值：`>=0.7` 且错 → high_conf_wrong，`<0.5` 且对 → low_conf_right
- `calc_level_jump_pattern` regex `r"_L(\d+)_"`，用 `→` 连接
- `_call_judge` max_retries=2（共 3 次），API 错误直传
- 归因瀑布精确顺序：extraction → search → reasoning → mixed
- defect_vs_lapse 解析失败降级 "lapse"
- `_extract_last_confidence` 任意异常返回 0.5
- judge 返回值默认：span completeness/hallucination 默认 0.0，tags 用 `list()`，missed_nodes 非 list 返 `[]`
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/diagnose.py tests/unit/test_diagnose.py
git commit -m "feat(evolution): diagnose.py metrics + attribution (Stage 1)"
```
---
### Task 6: diagnose.py — 聚合 + 案例包 + 入口
**Files:**
- Modify: `core/evolution/diagnose.py`（追加 ~700 行）
- Modify: `tests/unit/test_diagnose.py`（追加聚合 + 入口测试）
- [ ] **Step 1: Write aggregation + case pack tests**
```python
# 追加到 tests/unit/test_diagnose.py
from unittest.mock import AsyncMock
from core.evolution.types import (
QuestionMetrics, ErrorAttribution, SpanMetrics,
SkillCasePack, SystemCasePack, ToolCasePack, DiagnosisResult,
)
from core.evolution.diagnose import (
aggregate_d2, aggregate_d3, aggregate_d4, aggregate_d5,
merge_system_packs, merge_tool_packs, run_diagnosis,
)
class TestAggregation:
def test_d2_empty(self):
assert aggregate_d2([]) == {}
def test_d5_empty_returns_zero_structure(self):
result = aggregate_d5([])
assert "early_submit_rate" in result
assert result["early_submit_rate"] == 0.0
class TestMerge:
def test_merge_system_packs_none_on_empty(self):
assert merge_system_packs([]) is None
def test_merge_system_packs_wraps_stats(self):
pack = SystemCasePack(
stats={"a": 1}, failure_cases=[], success_cases=[],
)
merged = merge_system_packs([pack, pack])
assert "per_step" in merged.stats
assert len(merged.stats["per_step"]) == 2
class TestRunDiagnosis:
def test_empty_predictions_returns_empty_result(self):
import asyncio
from core.evolution.types import DiagnosePrompts
mock_log = AsyncMock()
mock_log.get_predictions.return_value = []
mock_log.get_traces.return_value = []
mock_llm = AsyncMock()
mock_store = MagicMock()
mock_store.list_skill_files.return_value = []
prompts = DiagnosePrompts(
defect_vs_lapse="", reasoning_sub="",
span_eval_system="", span_eval_user="",
missed_nodes="", skill_adherence="",
confirmation_bias="", evidence_sufficiency="",
)
result = asyncio.run(run_diagnosis(
"run1", [], {}, mock_llm, mock_log, mock_store, prompts,
concurrency=1,
))
assert isinstance(result, DiagnosisResult)
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py::TestAggregation tests/unit/test_diagnose.py::TestMerge tests/unit/test_diagnose.py::TestRunDiagnosis -v`
- [ ] **Step 3: Implement aggregation + case packs + run_diagnosis**
从 TRM4 `diagnose.py` 迁移：
- `_mean`, `_percentile` 辅助函数
- `aggregate_d2/d3/d4/d5`
- `_build_skill_case_packs`（含 severity 函数、C3 lapse routing、single-failure fallback；成功案例 `n_success=max(2, len(failures)//2)`，acc≤0.3 按 budget 升序否则按 adherence 降序）
- `_build_system_case_pack`（`_MIN_PATTERN_COUNT=3`，3 种行为模式；成功案例要求 correct+calibrated+no_bias+0.3≤budget≤0.8，按 abs(budget-0.5) 排序）
- `_build_tool_case_packs`（`_TOOL_TARGET_FILES` 映射；低 completeness 先选最多 4 条，高 hallucination 补到总数 4 上限；成功 span 要求 completeness≥0.9 且 hallucination==0.0）
- `merge_system_packs`/`merge_tool_packs`（stats 用 `{"per_step": [...]}` 包裹）
- `_classify_reasoning_failure`（串行 pass，prompt `diagnose_reasoning_failure.md`，JSON key `type`，解析失败 → `reasoning_failure_type=None` 不中断）
- `run_diagnosis` 入口（async，Semaphore 限并发，reasoning_failure 串行 pass）
注：`resolve_skill_file` 定义在 evolve.py（Task 7），diagnose.py 从 evolve 导入。
**关键变更**：
- `ThreadPoolExecutor` → `asyncio.gather` + `Semaphore(concurrency)`
- `HarnessLog` → `RunLog` Protocol（`get_predictions`/`get_traces`）
- 不写 DB（`_ensure_diagnosis_tables`/`_clear_existing`/`_insert_*` 全部移除）
- 不写 JSON 文件（`write analyses/...` 移除）
- 树数据从参数传入（非 `_load_tree_cache` 文件读）
- skill 内容从 `SkillStore` 读
- INFRA 统计按 task/video/question 过滤范围重算，不受 stop_reason 过滤影响
**保真校验**：
- `_INFRA_STOP_REASONS = frozenset({"error", "parse_error"})`
- 案例包选择规则（见上述各函数描述）
- single-failure fallback：1 个 defect → lapse_note（fallback 文本 `"复核该类已有规则，避免重复此类单例失败"`）
- lapse_note 空白过滤（strip 后空则丢弃）
- `_format_trace_text`（diagnose 版不截断，与 metrics 版不同！）
- D3 `avg_steps` key 实际存 budget_usage mean（TRM4 命名不一致，保留）
- `_make_case_sample` metrics 子字典固定 key：correct/error_type/budget_usage/confidence_calibration/repeat_visit_rate/tool_usage/missed_nodes/adherence_rate/confirmation_bias/evidence_sufficient
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_diagnose.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/diagnose.py tests/unit/test_diagnose.py
git commit -m "feat(evolution): diagnose.py aggregation + case packs + run_diagnosis (#8)"
```
---
### Task 7: evolve.py — 验证 + 辅助函数
**Files:**
- Create: `core/evolution/evolve.py`（本 Task 写验证 + 辅助部分，约 400 行）
- Test: `tests/unit/test_evolve.py`
- Source: TRM4 `evolve.py` 的 `validate_*`/`rank_and_clip`/`edit_budget_at`/`_resolve_skill_file` 等
- [ ] **Step 1: Write evolve validation + helpers tests**
```python
"""tests/unit/test_evolve.py"""
import pytest
from core.evolution.evolve import (
validate_skill, validate_system, validate_tool,
edit_budget_at, resolve_skill_file,
)
class TestValidateSkill:
def test_identical_passes(self):
content = "---\nname: test\ndescription: d\ntask_type: t\n---\nbody"
result = validate_skill(content, content)
assert result.passed
def test_changed_frontmatter_fails(self):
orig = "---\nname: a\ndescription: d\ntask_type: t\n---\nbody"
evol = "---\nname: b\ndescription: d\ntask_type: t\n---\nbody"
result = validate_skill(orig, evol)
assert not result.passed
def test_length_ratio_too_short_fails(self):
orig = "---\nname: a\ndescription: d\ntask_type: t\n---\n" + "x" * 1000
evol = "---\nname: a\ndescription: d\ntask_type: t\n---\nshort"
result = validate_skill(orig, evol)
assert not result.passed
class TestValidateSystem:
def test_identical_passes(self):
content = "intro\n## 能力边界\nfrozen\n## 输出格式\nfrozen2\n## other\nbody"
result = validate_system(content, content)
assert result.passed
def test_changed_frozen_section_fails(self):
orig = "intro\n## 能力边界\noriginal\n## other\nbody"
evol = "intro\n## 能力边界\nchanged\n## other\nbody"
result = validate_system(orig, evol)
assert not result.passed
class TestValidateTool:
def test_identical_passes(self):
extract = "## 输出格式\nfixed\n## other\nbody"
verify = "## 输出格式\nfixed2\n## other\nbody2"
result = validate_tool(extract, extract, verify, verify)
assert result.passed
def test_no_code_block_check(self):
extract = "## 输出格式\nfixed\n```\nunclosed"
result = validate_tool(extract, extract, "v", "v")
assert result.passed # tool 不检查代码块闭合
class TestEditBudget:
def test_start_at_zero(self):
assert edit_budget_at(0, 100, 5, 2) == 5
def test_end_at_total(self):
assert edit_budget_at(100, 100, 5, 2) == 2
def test_total_steps_one(self):
assert edit_budget_at(0, 1, 5, 2) == 5
def test_start_less_than_end_asserts(self):
with pytest.raises(AssertionError):
edit_budget_at(0, 100, 2, 5)
class TestResolveSkillFile:
def test_direct_match(self):
class FakeStore:
def list_skill_files(self): return ["action-reasoning.md", "default-strategy.md"]
def read_skill(self, f): return ""
assert resolve_skill_file(FakeStore(), "Action Reasoning") == "action-reasoning.md"
def test_fallback_to_default(self):
class FakeStore:
def list_skill_files(self): return ["default-strategy.md"]
def read_skill(self, f): return ""
assert resolve_skill_file(FakeStore(), "Unknown Type") == "default-strategy.md"
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v`
- [ ] **Step 3: Implement evolve.py validation + helpers**
从 TRM4 `evolve.py` 迁移：
- `_parse_frontmatter`、`_strip_appendix_region`（宽容版）、`_strip_momentum_region`（严格版）、`_strip_protected_regions`
- `_check_length`（去 appendix+momentum 后比较，ratio [0.3, 2.0]）
- `_check_code_blocks`
- `_extract_section`
- `_skill_protected_spans`/`_system_protected_spans`/`_tool_protected_spans`
- `validate_skill`/`validate_system`/`validate_tool` → 返回 `ValidationResult`（定义在此文件内部，非 types.py）
- `edit_budget_at`（纯数学，保持 TRM4 断言和 banker's rounding）
- `rank_and_clip`（async 化，`type(idx) is int` 排除 bool）
- `_select_top_edits`
- `_parse_llm_json`（两级：fenced → json.loads，失败返回 None）
- `resolve_skill_file`（接受 `SkillStore` 而非 `Path`）
- `_format_case_samples`（tool_output[:500] 截断）
- `_format_spans`（tool_output[:500] 截断）
- `_format_rejected_edits`
**关键变更**：
- `LLMClient` → `LLMProvider`
- `_resolve_skill_file(skills_dir: Path, ...)` → `resolve_skill_file(skill_store: SkillStore, ...)`
- `rank_and_clip` 变 async
**保真校验**：
- 冻结区配置：Skill(frontmatter+appendix+momentum)、System(3 sections+appendix)、Tool(输出格式+appendix)
- frontmatter 三字段：name/description/task_type
- `_parse_frontmatter` regex 必须从文件开头匹配 `^---\n...\n---`，`yaml.safe_load` 失败返回 None
- `_strip_appendix_region` 宽容 vs `_strip_momentum_region` 严格（不对称保留）
- `_parse_llm_json`：只匹配 ` ```json ` fenced block（非 ``` 不带 json）→ `json.loads`，失败返回 None（与 metrics 的三级不同！）
- validate_tool 不检查代码块闭合（与 skill/system 不同）
- `type(idx) is int`（非 isinstance）
- `rank_and_clip`/`_request_rank_indices`：rank LLM ValueError 降级，API 异常不捕获
- `rewrite_from_suggestions`：prompt `evolve_rewrite.md`，JSON key `rewritten`，重写不得长于原文，只捕 ValueError/KeyError/TypeError/AttributeError
- `_format_case_samples` tool_output[:500] 截断
- `_format_rejected_edits` gate 证据格式 `W=... L=... E={:.2f} δ̂={:+.3f}`
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/evolve.py tests/unit/test_evolve.py
git commit -m "feat(evolution): evolve.py validation + helpers"
```
---
### Task 8: evolve.py — per-target 进化
**Files:**
- Modify: `core/evolution/evolve.py`（追加 ~500 行）
- Modify: `tests/unit/test_evolve.py`（追加进化测试）
- [ ] **Step 1: Write evolution loop tests**
```python
# 追加到 tests/unit/test_evolve.py
import asyncio
from unittest.mock import AsyncMock, MagicMock
from core.evolution.types import (
SkillCasePack, SystemCasePack, ToolCasePack,
EvolutionRecord, EvolvePrompts,
)
from core.evolution.evolve import (
evolve_single_skill, evolve_system_prompt, evolve_single_tool,
consolidate_appendix,
)
_PROMPTS = EvolvePrompts(
evolve_skill="sk", evolve_system="sys", evolve_tool="tool",
evolve_rank="rank", consolidate_system="cons",
)
def _make_fake_llm(response_content: str):
"""构造返回固定内容的假 LLMProvider。"""
from core.types import LLMResponse
mock = AsyncMock()
mock.chat.return_value = LLMResponse(
content=response_content, thinking="", model="test",
provider="test", prompt_tokens=0, completion_tokens=0,
latency_ms=0, ttft_ms=None, max_inter_token_ms=None,
cache_hit=False, call_id="test-id",
)
return mock
class TestEvolveSingleSkill:
def test_empty_pack_skipped(self):
pack = SkillCasePack(
task_type="test", target_file="test.md",
stats={}, failure_cases=[], success_cases=[], lapse_notes=[],
)
store = MagicMock()
store.read_skill.return_value = "---\nname: t\ndescription: d\ntask_type: t\n---\nbody"
store.list_skill_files.return_value = ["test.md"]
llm = _make_fake_llm('{"suggestions":[],"edits":[]}')
record = asyncio.run(evolve_single_skill(
llm, pack, store, _PROMPTS, "v1", 5, 6,
))
assert record.status in ("rejected", "skipped")
class TestEvolveSystemPrompt:
def test_no_failures_returns_skipped(self):
pack = SystemCasePack(stats={}, failure_cases=[], success_cases=[])
store = MagicMock()
store.read_prompt.return_value = "## 能力边界\nfixed\n## 输出格式\nfixed\n## 视频树结构\nfixed\nbody"
llm = _make_fake_llm('{"suggestions":[],"edits":[]}')
record = asyncio.run(evolve_system_prompt(
llm, pack, store, _PROMPTS, "v1", 5,
))
assert record.status in ("rejected", "skipped")
class TestEvolveSingleTool:
def test_evolved_content_is_json(self):
pack = ToolCasePack(
tool_name="view_node",
target_files=["view_node_extract.md", "view_node_verify.md"],
stats={}, failure_spans=[], success_spans=[],
)
store = MagicMock()
store.read_prompt.return_value = "## 输出格式\nfixed\nbody"
llm = _make_fake_llm('{"suggestions":[],"edits":[]}')
record = asyncio.run(evolve_single_tool(
llm, pack, store, _PROMPTS, "v1", 5,
))
import json
parsed = json.loads(record.evolved_content)
assert "extract" in parsed and "verify" in parsed
class TestConsolidateAppendix:
def test_single_note_passthrough(self):
llm = _make_fake_llm("")
result = asyncio.run(consolidate_appendix(llm, ["note1"]))
assert result == ["note1"]
def test_exception_returns_original(self):
llm = AsyncMock()
llm.chat.side_effect = RuntimeError("boom")
result = asyncio.run(consolidate_appendix(llm, ["a", "b", "c"]))
assert result == ["a", "b", "c"]
```
- [ ] **Step 2: Run test — expect FAIL**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py::TestEvolveSingleSkill tests/unit/test_evolve.py::TestEvolveSystemPrompt tests/unit/test_evolve.py::TestEvolveSingleTool tests/unit/test_evolve.py::TestConsolidateAppendix -v`
- [ ] **Step 3: Implement per-target evolution**
从 TRM4 `evolve.py` 迁移：
- `_run_patch_evolution_loop`（async 化，`range(2)` 两轮，三种失败反馈）
- `_build_lapse_only_attempt`（合成 `applied_append` report）
- `evolve_single_skill`（三分支：lapse-only/rewrite/patch + appendix 追加 + consolidation）
- `evolve_system_prompt`（无 lapse、无 rewrite、无 appendix）
- `evolve_single_tool`（extract+verify 合池 `_src` 标记、shared budget、JSON evolved_content）
- `consolidate_appendix`（async 化，四守卫）
- `rewrite_from_suggestions`（async 化，3 个拒绝条件）
- `_append_lapse_with_consolidation`（`>= threshold` 触发、G4 `>=` 拒绝等长）
**关键变更**：
- 全部 `client.chat()` → `await llm.chat(messages)`
- `response.choices[0].message.content` → `response.content`
- `skills_dir / target_file` → `skill_store.read_skill(target_file)`
- `prompts_dir / filename` → `prompt_store.read_prompt(filename)`
- 版本写入（`advance_version`/copytree）全部移除——返回 `EvolutionRecord`
- `run_evolution` 编排移除——per-target 函数是最高粒度
**保真校验**：
- Skill 三分支精确条件和行为
- `_build_lapse_only_attempt` 合成 `applied_append` 状态
- rank_and_clip 三级降级
- Tool `_src` 标记合池/拆回
- consolidate 四守卫（G4 在调用方）
- rewrite 长度限制（重写不得长于原文）、异常类型（仅捕 ValueError/KeyError/TypeError/AttributeError）
- 两轮重试反馈文本
- [ ] **Step 4: Run test — expect PASS**
Run: `conda activate Video-Tree-TRM & pytest tests/unit/test_evolve.py -v`
- [ ] **Step 5: Commit**
```
git add core/evolution/evolve.py tests/unit/test_evolve.py
git commit -m "feat(evolution): evolve.py per-target evolution — skill/system/tool (#9)"
```
---
### Task 9: __init__.py + 集成 + lint
**Files:**
- Modify: `core/evolution/__init__.py`
- Run: lint + dependency check
- [ ] **Step 1: Write __init__.py public API**
```python
"""core/evolution/ — 自进化循环决策内核。"""
from core.evolution.gate import compute_e_value, gate_decision, probation_verdict
from core.evolution.patch import (
apply_patch_with_report, append_to_appendix,
extract_appendix_notes, replace_appendix_notes,
replace_momentum, momentum_inner,
)
from core.evolution.validate import pair_block, classify_quadrants, compute_accuracy
from core.evolution.diagnose import run_diagnosis
from core.evolution.evolve import (
evolve_single_skill, evolve_system_prompt, evolve_single_tool,
edit_budget_at, resolve_skill_file,
)
__all__ = [
"compute_e_value", "gate_decision", "probation_verdict",
"apply_patch_with_report", "append_to_appendix",
"extract_appendix_notes", "replace_appendix_notes",
"replace_momentum", "momentum_inner",
"pair_block", "classify_quadrants", "compute_accuracy",
"run_diagnosis",
"evolve_single_skill", "evolve_system_prompt", "evolve_single_tool",
"edit_budget_at", "resolve_skill_file",
]
```
- [ ] **Step 2: Run lint**
```bash
conda activate Video-Tree-TRM & ruff check core/evolution/ --fix
conda activate Video-Tree-TRM & ruff format core/evolution/
```
- [ ] **Step 3: Dependency direction check**
```bash
# core/evolution/ 不得 import app/ 或 adapters/
grep -rn "from app\." core/evolution/ && echo "VIOLATION" || echo "OK"
grep -rn "from adapters\." core/evolution/ && echo "VIOLATION" || echo "OK"
grep -rn "import app\." core/evolution/ && echo "VIOLATION" || echo "OK"
grep -rn "import adapters\." core/evolution/ && echo "VIOLATION" || echo "OK"
```
Expected: 全部 OK
- [ ] **Step 4: Update ARCHITECTURE.md §3.1**
将 SkillStore/PromptStore/RunLog 的 Protocol 定义从"含写方法"修订为"core/ 只读，写方法在 app/ 实现类"。同步设计文档 §3 的修订说明。
- [ ] **Step 5: Run full test suite**
```bash
conda activate Video-Tree-TRM & pytest tests/unit/test_gate.py tests/unit/test_patch.py tests/unit/test_validate.py tests/unit/test_diagnose.py tests/unit/test_evolve.py tests/unit/test_evolution_types.py -v --tb=short
```
- [ ] **Step 6: Commit**
```
git add core/evolution/__init__.py research-wiki/ARCHITECTURE.md
git commit -m "feat(evolution): __init__.py public API + ARCHITECTURE.md Protocol update"
```
---
## Algorithm Fidelity Check
本计划涉及 4 项核心算法迁移：
| # | 算法 | 计划 Task | 保真措施 |
|---|------|----------|---------|
| 5 | CE-Gate e-process | Task 2 | 原样迁移 163 行，零有意变更；测试覆盖边界（W=L=0、负值、重 win/loss） |
| 7 | 块顺序验证 | Task 4 | 纯决策函数提取；编排留 app/harness/（Design B 约定） |
| 8 | 诊断瀑布 | Task 5-6 | 归因瀑布顺序、defect/lapse 分类、案例包选择规则逐条保真 |
| 9 | 进化 patch 引擎 | Task 3, 7-8 | patch.py 原样迁移；evolve.py 三分支/rank_clip/consolidate 四守卫逐条保真 |
不涉及的算法：#1-4（建树/检索器）、#6（信息阶梯，app/harness/）、#10（mini-batch，app/harness/）、#11（Agent Loop，已迁移）、#12（树环境语义搜索，已迁移）、#13（训练循环编排，app/harness/）。
Reference in New Issue View Git Blame Copy Permalink