merge: question-gen v3 + preflight fixes + continuous concurrent gate
This commit is contained in:
@@ -22,6 +22,7 @@ Every project goes through this process. A todo list, a single-function utility,
|
|||||||
You MUST create a task for each of these items and complete them in order:
|
You MUST create a task for each of these items and complete them in order:
|
||||||
|
|
||||||
1. **Explore project context** — check files, docs, recent commits
|
1. **Explore project context** — check files, docs, recent commits
|
||||||
|
1.5. **Prior-version audit (mandatory for rewrites/refactors)** — if the task replaces or rewrites an existing module, list every behavior of the old version (including persistence, crash recovery, idempotency, resume) and confirm each is kept, replaced, or deliberately dropped. Undocumented implicit drops = bugs.
|
||||||
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
|
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
|
||||||
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
|
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
|
||||||
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
|
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
|
||||||
@@ -92,7 +93,12 @@ digraph brainstorming {
|
|||||||
- Once you believe you understand what you're building, present the design
|
- Once you believe you understand what you're building, present the design
|
||||||
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
|
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
|
||||||
- Ask after each section whether it looks right so far
|
- Ask after each section whether it looks right so far
|
||||||
- Cover: architecture, components, data flow, error handling, testing
|
- Cover: architecture, components, data flow, error handling, testing, **non-functional requirements** (see below)
|
||||||
|
- **Non-functional requirements (mandatory section):** Every design MUST explicitly address these four dimensions — even if the answer is "not applicable":
|
||||||
|
- **Persistence strategy:** When does data hit disk? How much is lost on crash? Overwrite or append?
|
||||||
|
- **Idempotency:** Is the same operation safe to repeat? Does it produce the same result?
|
||||||
|
- **Resume/checkpoint:** Can the process recover from interruption? How is progress persisted?
|
||||||
|
- **Atomicity:** Are writes atomic? Can a partial write corrupt data?
|
||||||
- Be ready to go back and clarify if something doesn't make sense
|
- Be ready to go back and clarify if something doesn't make sense
|
||||||
|
|
||||||
**Design for isolation and clarity:**
|
**Design for isolation and clarity:**
|
||||||
@@ -124,6 +130,8 @@ After writing the spec document, look at it with fresh eyes:
|
|||||||
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
|
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
|
||||||
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
|
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
|
||||||
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
|
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
|
||||||
|
5. **Non-functional coverage:** Does the design explicitly address persistence, idempotency, resume, and atomicity? If any dimension is missing, add it now — even if the answer is "not applicable."
|
||||||
|
6. **Prior-version regression check (rewrites only):** If this replaces an existing module, confirm every behavior from the prior-version audit (step 1.5) is accounted for in the design. Any gap = a spec bug.
|
||||||
|
|
||||||
Fix any issues inline. No need to re-review — just fix and move on.
|
Fix any issues inline. No need to re-review — just fix and move on.
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -46,7 +46,8 @@ LLM_CIRCUIT_BREAKER_COOLDOWN=60
|
|||||||
LLM_TTFT_TIMEOUT=30
|
LLM_TTFT_TIMEOUT=30
|
||||||
LLM_INTER_TOKEN_TIMEOUT=15
|
LLM_INTER_TOKEN_TIMEOUT=15
|
||||||
LLM_RETRY_MAX_DELAY=30.0
|
LLM_RETRY_MAX_DELAY=30.0
|
||||||
REDIS_CACHE_TTL=86400
|
# 正整数秒,禁止 0(0 会被拒绝启动);训练场景建议 >= 单次训练时长
|
||||||
|
REDIS_CACHE_TTL=604800
|
||||||
|
|
||||||
# 建树批量并行:全局 VLM/LLM 在途调用上限(Spec-2 工程配置)
|
# 建树批量并行:全局 VLM/LLM 在途调用上限(Spec-2 工程配置)
|
||||||
TREE_BUILD_API_CONCURRENCY=16
|
TREE_BUILD_API_CONCURRENCY=16
|
||||||
|
|||||||
@@ -96,9 +96,10 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
|
|||||||
### Phase 1: 规划与设计 (Planning)
|
### Phase 1: 规划与设计 (Planning)
|
||||||
1. **需求探索**: 涉及创建新功能、新组件、修改行为时,**必须**先调用 `brainstorming` skill 进行需求探索与设计。无论用户的指令多么具体、改动多么简单,都不得跳过此步骤(除非用户显式说"跳过 brainstorming")。
|
1. **需求探索**: 涉及创建新功能、新组件、修改行为时,**必须**先调用 `brainstorming` skill 进行需求探索与设计。无论用户的指令多么具体、改动多么简单,都不得跳过此步骤(除非用户显式说"跳过 brainstorming")。
|
||||||
2. **查阅规格 & 讨论**: 仔细阅读 `research-wiki/`(单一事实源)下对应的文档,了解项目最新情况。对于不理解的地方请与人类进行多轮讨论,确保理解人类的设计意图。
|
2. **查阅规格 & 讨论**: 仔细阅读 `research-wiki/`(单一事实源)下对应的文档,了解项目最新情况。对于不理解的地方请与人类进行多轮讨论,确保理解人类的设计意图。
|
||||||
3. **日志方案设计**: 功能会产生运行时数据时,**必须**调用 `structured-logging` skill 设计日志方案。
|
3. **前序版本对照(重写/重构时强制)**: 当任务涉及重写或重构已有模块时,**必须**列出前序版本的所有行为(包括持久化策略、崩溃恢复、幂等性、断点续跑等非功能性行为),逐一确认新版本是保留、替代、还是删除。未经确认的隐式删除 = bug。
|
||||||
4. **撰写计划**: 正式编码前,**必须**调用 `writing-plans` skill 撰写实现计划。
|
4. **日志方案设计**: 功能会产生运行时数据时,**必须**调用 `structured-logging` skill 设计日志方案。
|
||||||
5. **审核门控(差异化)**:
|
5. **撰写计划**: 正式编码前,**必须**调用 `writing-plans` skill 撰写实现计划。
|
||||||
|
6. **审核门控(差异化)**:
|
||||||
- **design:Claude 自审 → Codex 审 → 人类审**(保留人类门,批准后方可进入计划阶段)。
|
- **design:Claude 自审 → Codex 审 → 人类审**(保留人类门,批准后方可进入计划阶段)。
|
||||||
- **plan:Claude 自审 → Codex 审 → 直接执行**(无 plan 人类门);plan 经 Claude 自审 + Codex 审通过后直接进入 Phase 2 执行。
|
- **plan:Claude 自审 → Codex 审 → 直接执行**(无 plan 人类门);plan 经 Claude 自审 + Codex 审通过后直接进入 Phase 2 执行。
|
||||||
|
|
||||||
@@ -159,6 +160,19 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
|
|||||||
- **功能修改**:
|
- **功能修改**:
|
||||||
- **必须** 不考虑向后兼容,直接修改原文件。代码简洁性优先。
|
- **必须** 不考虑向后兼容,直接修改原文件。代码简洁性优先。
|
||||||
|
|
||||||
|
### 4.2.1 设计文档非功能性需求覆盖(强制)
|
||||||
|
|
||||||
|
> **教训来源**: v2 出题管线重写时未继承 v1 的逐题追加持久化策略,导致多次 run 的题目丢失。
|
||||||
|
|
||||||
|
设计文档**必须**显式覆盖以下非功能性维度(即使答案是"不适用"也要写明):
|
||||||
|
|
||||||
|
| 维度 | 必答问题 |
|
||||||
|
|------|---------|
|
||||||
|
| **持久化策略** | 数据何时落盘?崩溃时最多丢多少?是覆盖写还是追加? |
|
||||||
|
| **幂等性** | 同一操作重复执行是否安全?结果是否一致? |
|
||||||
|
| **断点续跑** | 中断后重启能否从断点恢复?进度如何持久化? |
|
||||||
|
| **原子性** | 写操作是否原子?部分写入是否会损坏数据? |
|
||||||
|
|
||||||
### 4.3 Git 工作流规范
|
### 4.3 Git 工作流规范
|
||||||
- **Feature Branch**: 所有开发工作在 feature 分支上进行,**严禁**直接在 main/master 上修改。
|
- **Feature Branch**: 所有开发工作在 feature 分支上进行,**严禁**直接在 main/master 上修改。
|
||||||
- **增量提交**: 频繁提交,每个提交有明确的语义。
|
- **增量提交**: 频繁提交,每个提交有明确的语义。
|
||||||
|
|||||||
@@ -0,0 +1,183 @@
|
|||||||
|
"""SqliteDiagnosisSignalStore:baseline 逐题诊断信号的 SQLite 持久化适配器。
|
||||||
|
|
||||||
|
实现 core/evolution/protocols.py::DiagnosisSignalStore 端口。信号行以
|
||||||
|
(question_id, baseline_run_id, diag_fingerprint) 为主键,INSERT OR REPLACE
|
||||||
|
保证逐题幂等 upsert;bool 字段以 0/1 存储,可空字段以 NULL 存储。表建在
|
||||||
|
harness.db,供离线诊断编排写入、视频级切分选择器读取。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from core.evolution.types import DiagnosisSignalRow
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from types import TracebackType
|
||||||
|
|
||||||
|
# 表列顺序即 DiagnosisSignalRow 字段顺序(question_id..session_id),
|
||||||
|
# upsert 写入与 load 还原共用,避免手写列名两处漂移。
|
||||||
|
_COLUMNS: tuple[str, ...] = (
|
||||||
|
"question_id",
|
||||||
|
"video_id",
|
||||||
|
"baseline_run_id",
|
||||||
|
"diag_fingerprint",
|
||||||
|
"task_type",
|
||||||
|
"error_type",
|
||||||
|
"cause_category",
|
||||||
|
"tier",
|
||||||
|
"evolution_target",
|
||||||
|
"degraded",
|
||||||
|
"infra",
|
||||||
|
"session_id",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SqliteDiagnosisSignalStore:
|
||||||
|
"""逐题诊断信号的 SQLite 存储实现。
|
||||||
|
|
||||||
|
构造时按需建表(CREATE TABLE IF NOT EXISTS),主键
|
||||||
|
(question_id, baseline_run_id, diag_fingerprint) 保证同键覆盖。
|
||||||
|
每次写操作单事务 commit,保证原子落盘与断点续跑。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
db_path: SQLite 数据库文件路径(通常为 harness.db)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, db_path: str) -> None:
|
||||||
|
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._conn = sqlite3.connect(db_path)
|
||||||
|
self._conn.row_factory = sqlite3.Row
|
||||||
|
self._init_table()
|
||||||
|
|
||||||
|
def _init_table(self) -> None:
|
||||||
|
"""创建 baseline_diagnosis 表(若不存在)。"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS baseline_diagnosis (
|
||||||
|
question_id TEXT NOT NULL,
|
||||||
|
video_id TEXT NOT NULL,
|
||||||
|
baseline_run_id TEXT NOT NULL,
|
||||||
|
diag_fingerprint TEXT NOT NULL,
|
||||||
|
task_type TEXT NOT NULL,
|
||||||
|
error_type TEXT,
|
||||||
|
cause_category TEXT,
|
||||||
|
tier TEXT NOT NULL,
|
||||||
|
evolution_target TEXT,
|
||||||
|
degraded INTEGER NOT NULL,
|
||||||
|
infra INTEGER NOT NULL,
|
||||||
|
session_id TEXT,
|
||||||
|
PRIMARY KEY (question_id, baseline_run_id, diag_fingerprint)
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
|
def upsert(self, row: DiagnosisSignalRow) -> None:
|
||||||
|
"""写入或覆盖单题诊断信号(按主键幂等,单事务)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
row: 待持久化的诊断信号行。bool 字段转 0/1,None 存 NULL。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
用 INSERT OR REPLACE 按主键覆盖,避免重复行;commit 保证原子。
|
||||||
|
"""
|
||||||
|
placeholders = ", ".join("?" for _ in _COLUMNS)
|
||||||
|
col_names = ", ".join(_COLUMNS)
|
||||||
|
values = (
|
||||||
|
row.question_id,
|
||||||
|
row.video_id,
|
||||||
|
row.baseline_run_id,
|
||||||
|
row.diag_fingerprint,
|
||||||
|
row.task_type,
|
||||||
|
row.error_type,
|
||||||
|
row.cause_category,
|
||||||
|
row.tier,
|
||||||
|
row.evolution_target,
|
||||||
|
int(row.degraded),
|
||||||
|
int(row.infra),
|
||||||
|
row.session_id,
|
||||||
|
)
|
||||||
|
self._conn.execute(
|
||||||
|
f"INSERT OR REPLACE INTO baseline_diagnosis ({col_names}) VALUES ({placeholders})",
|
||||||
|
values,
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
|
||||||
|
def done_question_ids(
|
||||||
|
self,
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
*,
|
||||||
|
retry_uncertain: bool = False,
|
||||||
|
) -> set[str]:
|
||||||
|
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
baseline_run_id: baseline run 标识。
|
||||||
|
diag_fingerprint: 诊断口径指纹。
|
||||||
|
retry_uncertain: True 时追加 `AND tier != 'uncertain'`,把 uncertain
|
||||||
|
(信号不可信降级)题排除出已完成集,令其被重新诊断;默认 False。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
已落盘信号的 question_id 集合;无匹配时为空集,供断点续跑跳过。
|
||||||
|
"""
|
||||||
|
sql = (
|
||||||
|
"SELECT DISTINCT question_id FROM baseline_diagnosis"
|
||||||
|
" WHERE baseline_run_id = ? AND diag_fingerprint = ?"
|
||||||
|
)
|
||||||
|
if retry_uncertain:
|
||||||
|
sql += " AND tier != 'uncertain'"
|
||||||
|
cursor = self._conn.execute(sql, (baseline_run_id, diag_fingerprint))
|
||||||
|
return {r["question_id"] for r in cursor.fetchall()}
|
||||||
|
|
||||||
|
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
|
||||||
|
"""加载指定 run 与诊断指纹下的全部诊断信号行。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
baseline_run_id: baseline run 标识。
|
||||||
|
diag_fingerprint: 诊断口径指纹。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
还原后的 DiagnosisSignalRow 列表(0/1→bool,NULL→None)。
|
||||||
|
"""
|
||||||
|
col_names = ", ".join(_COLUMNS)
|
||||||
|
cursor = self._conn.execute(
|
||||||
|
f"SELECT {col_names} FROM baseline_diagnosis"
|
||||||
|
" WHERE baseline_run_id = ? AND diag_fingerprint = ?",
|
||||||
|
(baseline_run_id, diag_fingerprint),
|
||||||
|
)
|
||||||
|
return [
|
||||||
|
DiagnosisSignalRow(
|
||||||
|
question_id=r["question_id"],
|
||||||
|
video_id=r["video_id"],
|
||||||
|
baseline_run_id=r["baseline_run_id"],
|
||||||
|
diag_fingerprint=r["diag_fingerprint"],
|
||||||
|
task_type=r["task_type"],
|
||||||
|
error_type=r["error_type"],
|
||||||
|
cause_category=r["cause_category"],
|
||||||
|
tier=r["tier"],
|
||||||
|
evolution_target=r["evolution_target"],
|
||||||
|
degraded=bool(r["degraded"]),
|
||||||
|
infra=bool(r["infra"]),
|
||||||
|
session_id=r["session_id"],
|
||||||
|
)
|
||||||
|
for r in cursor.fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""关闭底层 SQLite 连接,释放文件描述符与锁。"""
|
||||||
|
self._conn.close()
|
||||||
|
|
||||||
|
def __enter__(self) -> SqliteDiagnosisSignalStore:
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(
|
||||||
|
self,
|
||||||
|
exc_type: type[BaseException] | None,
|
||||||
|
exc_val: BaseException | None,
|
||||||
|
exc_tb: TracebackType | None,
|
||||||
|
) -> None:
|
||||||
|
self.close()
|
||||||
+18
-3
@@ -20,21 +20,32 @@ class CircuitBreaker:
|
|||||||
self._cooldown_s = cooldown_s
|
self._cooldown_s = cooldown_s
|
||||||
self._fails: dict[str, int] = {}
|
self._fails: dict[str, int] = {}
|
||||||
self._open_until: dict[str, float] = {}
|
self._open_until: dict[str, float] = {}
|
||||||
|
self._half_open_inflight: dict[str, bool] = {}
|
||||||
|
|
||||||
def is_open(self, source_name: str, now: float) -> bool:
|
def is_open(self, source_name: str, now: float) -> bool:
|
||||||
"""判断指定源是否处于开路状态。
|
"""判断指定源是否处于开路状态。
|
||||||
|
|
||||||
冷却截止时刻之前为开路;到期返回 False(放行一个试探,即半开)。
|
冷却截止时刻之前为开路;到期进入半开,**只放行一个探针**(其余仍被挡),
|
||||||
|
避免冷却到期瞬间惊群重连再次压垮上游。"检查+标记探针"在 asyncio 单线程内
|
||||||
|
同步执行,天然原子无竞态。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
source_name: 被熔断的源标识。
|
source_name: 被熔断的源标识。
|
||||||
now: 当前时刻(秒级时间戳),由调用方注入。
|
now: 当前时刻(秒级时间戳),由调用方注入。
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
True 表示开路(拒绝请求),False 表示关闭或半开(放行)。
|
True 表示开路(拒绝请求),False 表示关闭或半开放行探针。
|
||||||
"""
|
"""
|
||||||
until = self._open_until.get(source_name)
|
until = self._open_until.get(source_name)
|
||||||
return until is not None and now < until
|
if until is None:
|
||||||
|
return False
|
||||||
|
if now < until:
|
||||||
|
return True # 冷却中,全挡
|
||||||
|
# 冷却到期:half-open,只放行一个探针
|
||||||
|
if self._half_open_inflight.get(source_name):
|
||||||
|
return True # 已有探针在途,继续挡
|
||||||
|
self._half_open_inflight[source_name] = True
|
||||||
|
return False
|
||||||
|
|
||||||
def record_failure(self, source_name: str, now: float) -> None:
|
def record_failure(self, source_name: str, now: float) -> None:
|
||||||
"""记录一次失败;累计达阈值则开路至 now + cooldown。
|
"""记录一次失败;累计达阈值则开路至 now + cooldown。
|
||||||
@@ -47,6 +58,8 @@ class CircuitBreaker:
|
|||||||
self._fails[source_name] = count
|
self._fails[source_name] = count
|
||||||
if count >= self._fail_threshold:
|
if count >= self._fail_threshold:
|
||||||
self._open_until[source_name] = now + self._cooldown_s
|
self._open_until[source_name] = now + self._cooldown_s
|
||||||
|
# 探针失败清在途标记,使下一轮 cooldown 到期后可再放行探针
|
||||||
|
self._half_open_inflight.pop(source_name, None)
|
||||||
|
|
||||||
def force_open(self, source_name: str, now: float) -> None:
|
def force_open(self, source_name: str, now: float) -> None:
|
||||||
"""强制开路(用于 401/403 等不可恢复错误),一次即熔断。
|
"""强制开路(用于 401/403 等不可恢复错误),一次即熔断。
|
||||||
@@ -59,6 +72,7 @@ class CircuitBreaker:
|
|||||||
"""
|
"""
|
||||||
self._fails[source_name] = self._fail_threshold
|
self._fails[source_name] = self._fail_threshold
|
||||||
self._open_until[source_name] = now + self._cooldown_s
|
self._open_until[source_name] = now + self._cooldown_s
|
||||||
|
self._half_open_inflight.pop(source_name, None)
|
||||||
|
|
||||||
def record_success(self, source_name: str) -> None:
|
def record_success(self, source_name: str) -> None:
|
||||||
"""记录一次成功;清零失败计数与开路状态(关闭熔断器)。
|
"""记录一次成功;清零失败计数与开路状态(关闭熔断器)。
|
||||||
@@ -68,3 +82,4 @@ class CircuitBreaker:
|
|||||||
"""
|
"""
|
||||||
self._fails.pop(source_name, None)
|
self._fails.pop(source_name, None)
|
||||||
self._open_until.pop(source_name, None)
|
self._open_until.pop(source_name, None)
|
||||||
|
self._half_open_inflight.pop(source_name, None)
|
||||||
|
|||||||
+16
-3
@@ -179,7 +179,10 @@ def _is_transient_error(exc: Exception) -> bool:
|
|||||||
返回:
|
返回:
|
||||||
True 表示可重试,False 表示不可重试。
|
True 表示可重试,False 表示不可重试。
|
||||||
"""
|
"""
|
||||||
if isinstance(exc, (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout)):
|
# 两族基类覆盖断连族:TimeoutException(ConnectTimeout/ReadTimeout/WriteTimeout/PoolTimeout)
|
||||||
|
# 与 TransportError(ConnectError/ReadError/RemoteProtocolError 等)。
|
||||||
|
# 注意 HTTPStatusError 非 TransportError 子类,401/403 致命分支不受影响。
|
||||||
|
if isinstance(exc, (httpx.TimeoutException, httpx.TransportError)):
|
||||||
return True
|
return True
|
||||||
if isinstance(exc, httpx.HTTPStatusError):
|
if isinstance(exc, httpx.HTTPStatusError):
|
||||||
return exc.response.status_code in _TRANSIENT_STATUS_CODES
|
return exc.response.status_code in _TRANSIENT_STATUS_CODES
|
||||||
@@ -274,6 +277,7 @@ class GovernedLLMClient:
|
|||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
parent_call_id: str | None = None,
|
parent_call_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse:
|
) -> LLMResponse:
|
||||||
"""发起 LLM 调用,经四层治理栈:熔断 → 缓存 → 重试+流式 → 遥测。
|
"""发起 LLM 调用,经四层治理栈:熔断 → 缓存 → 重试+流式 → 遥测。
|
||||||
|
|
||||||
@@ -281,6 +285,7 @@ class GovernedLLMClient:
|
|||||||
messages: OpenAI 格式消息列表。
|
messages: OpenAI 格式消息列表。
|
||||||
session_id: 会话 ID(传递到遥测)。
|
session_id: 会话 ID(传递到遥测)。
|
||||||
parent_call_id: 父调用 ID(传递到遥测)。
|
parent_call_id: 父调用 ID(传递到遥测)。
|
||||||
|
cache_salt: 可选缓存盐,透传到 Redis 缓存键(如跨 epoch 重采样)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
LLMResponse 统一响应。
|
LLMResponse 统一响应。
|
||||||
@@ -296,7 +301,11 @@ class GovernedLLMClient:
|
|||||||
raise CircuitOpenError(f"熔断器已开启,拒绝调用 provider={self._provider}")
|
raise CircuitOpenError(f"熔断器已开启,拒绝调用 provider={self._provider}")
|
||||||
|
|
||||||
# ② 缓存查询(cache 为 None 时跳过)— call_id 在缓存路径独立生成
|
# ② 缓存查询(cache 为 None 时跳过)— call_id 在缓存路径独立生成
|
||||||
cached = await self._cache.get(self._model, messages) if self._cache is not None else None
|
cached = (
|
||||||
|
await self._cache.get(self._model, messages, cache_salt)
|
||||||
|
if self._cache is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
if cached is not None:
|
if cached is not None:
|
||||||
cache_call_id = str(uuid4())
|
cache_call_id = str(uuid4())
|
||||||
response = LLMResponse(
|
response = LLMResponse(
|
||||||
@@ -370,7 +379,7 @@ class GovernedLLMClient:
|
|||||||
|
|
||||||
# ④ 写缓存(cache 为 None 时跳过)
|
# ④ 写缓存(cache 为 None 时跳过)
|
||||||
if self._cache is not None:
|
if self._cache is not None:
|
||||||
await self._cache.set(self._model, messages, response)
|
await self._cache.set(self._model, messages, response, cache_salt)
|
||||||
|
|
||||||
# ⑤ 遥测
|
# ⑤ 遥测
|
||||||
await self._telemetry.record_llm_call(
|
await self._telemetry.record_llm_call(
|
||||||
@@ -572,6 +581,10 @@ class GovernedLLMClient:
|
|||||||
else:
|
else:
|
||||||
thinking_parts.append(text)
|
thinking_parts.append(text)
|
||||||
|
|
||||||
|
# 流耗尽但未收 [DONE] → 服务端截断,视为可重试的 SSE 异常(不写缓存/不当成功)
|
||||||
|
if not usage_sink.get("done"):
|
||||||
|
raise _SseAnomaly("truncated_no_done")
|
||||||
|
|
||||||
content = "".join(content_parts)
|
content = "".join(content_parts)
|
||||||
thinking = "".join(thinking_parts)
|
thinking = "".join(thinking_parts)
|
||||||
usage = usage_sink.get("usage", {})
|
usage = usage_sink.get("usage", {})
|
||||||
|
|||||||
+43
-9
@@ -12,6 +12,26 @@ from loguru import logger
|
|||||||
from core.types import LLMResponse
|
from core.types import LLMResponse
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_cache_ttl(ttl: int) -> int:
|
||||||
|
"""校验 Redis 缓存 TTL:必须为正整数(消灭 0=永不过期 的隐式语义)。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ttl: 待校验的 TTL 秒数。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
校验通过的正整数 TTL。
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: ttl <= 0。
|
||||||
|
"""
|
||||||
|
if ttl <= 0:
|
||||||
|
raise ValueError(
|
||||||
|
f"REDIS_CACHE_TTL 必须为正整数秒,实际 {ttl}。"
|
||||||
|
"训练场景建议 >= 单次训练时长(如 86400)。"
|
||||||
|
)
|
||||||
|
return ttl
|
||||||
|
|
||||||
|
|
||||||
class RedisResponseCache:
|
class RedisResponseCache:
|
||||||
"""基于 Redis 的 LLM 响应缓存。
|
"""基于 Redis 的 LLM 响应缓存。
|
||||||
|
|
||||||
@@ -29,36 +49,48 @@ class RedisResponseCache:
|
|||||||
self._redis = redis
|
self._redis = redis
|
||||||
self._ttl_s = ttl_s
|
self._ttl_s = ttl_s
|
||||||
|
|
||||||
def _build_key(self, model: str, messages: list[dict[str, str]]) -> str:
|
def _build_key(
|
||||||
|
self,
|
||||||
|
model: str,
|
||||||
|
messages: list[dict[str, str]],
|
||||||
|
cache_salt: str | None = None,
|
||||||
|
) -> str:
|
||||||
"""构造 content-addressed 缓存键。
|
"""构造 content-addressed 缓存键。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
model: 模型名称。
|
model: 模型名称。
|
||||||
messages: 消息列表。
|
messages: 消息列表。
|
||||||
|
cache_salt: 可选缓存盐(如跨 epoch 强制重采样)。仅当非 None 时才加入
|
||||||
|
键 payload,保证默认 None 时键结构与旧缓存一字节不差、旧键不失效。
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
sha256 哈希字符串作为 Redis 键。
|
sha256 哈希字符串作为 Redis 键。
|
||||||
"""
|
"""
|
||||||
payload = json.dumps(
|
key_obj: dict[str, Any] = {"model": model, "messages": messages}
|
||||||
{"model": model, "messages": messages},
|
if cache_salt is not None:
|
||||||
sort_keys=True,
|
key_obj["salt"] = cache_salt
|
||||||
ensure_ascii=False,
|
payload = json.dumps(key_obj, sort_keys=True, ensure_ascii=False)
|
||||||
)
|
|
||||||
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||||
return f"llm_cache:{digest}"
|
return f"llm_cache:{digest}"
|
||||||
|
|
||||||
async def get(self, model: str, messages: list[dict[str, str]]) -> LLMResponse | None:
|
async def get(
|
||||||
|
self,
|
||||||
|
model: str,
|
||||||
|
messages: list[dict[str, str]],
|
||||||
|
cache_salt: str | None = None,
|
||||||
|
) -> LLMResponse | None:
|
||||||
"""从缓存读取 LLM 响应。
|
"""从缓存读取 LLM 响应。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
model: 模型名称。
|
model: 模型名称。
|
||||||
messages: 消息列表。
|
messages: 消息列表。
|
||||||
|
cache_salt: 可选缓存盐,透传到键构造。
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
缓存命中时返回 LLMResponse,未命中或 Redis 异常时返回 None。
|
缓存命中时返回 LLMResponse,未命中或 Redis 异常时返回 None。
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
key = self._build_key(model, messages)
|
key = self._build_key(model, messages, cache_salt)
|
||||||
raw = await self._redis.get(key)
|
raw = await self._redis.get(key)
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Redis 缓存读取失败,降级为未命中")
|
logger.warning("Redis 缓存读取失败,降级为未命中")
|
||||||
@@ -75,6 +107,7 @@ class RedisResponseCache:
|
|||||||
model: str,
|
model: str,
|
||||||
messages: list[dict[str, str]],
|
messages: list[dict[str, str]],
|
||||||
response: LLMResponse,
|
response: LLMResponse,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""将 LLM 响应写入缓存。
|
"""将 LLM 响应写入缓存。
|
||||||
|
|
||||||
@@ -82,9 +115,10 @@ class RedisResponseCache:
|
|||||||
model: 模型名称。
|
model: 模型名称。
|
||||||
messages: 消息列表。
|
messages: 消息列表。
|
||||||
response: 待缓存的 LLMResponse。
|
response: 待缓存的 LLMResponse。
|
||||||
|
cache_salt: 可选缓存盐,透传到键构造。
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
key = self._build_key(model, messages)
|
key = self._build_key(model, messages, cache_salt)
|
||||||
value = json.dumps(dataclasses.asdict(response), ensure_ascii=False)
|
value = json.dumps(dataclasses.asdict(response), ensure_ascii=False)
|
||||||
if self._ttl_s:
|
if self._ttl_s:
|
||||||
await self._redis.set(key, value, ex=self._ttl_s)
|
await self._redis.set(key, value, ex=self._ttl_s)
|
||||||
|
|||||||
+53
-27
@@ -1,20 +1,22 @@
|
|||||||
"""SQLite 遥测记录器 — TelemetryRecorder Protocol 的生产实现。
|
"""SQLite 遥测记录器 — TelemetryRecorder Protocol 的生产实现。
|
||||||
|
|
||||||
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,
|
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,确保事件循环不被阻塞。
|
||||||
确保事件循环不被阻塞。表在首次写入时懒初始化。
|
构造时建单持久连接 + 建表(对齐 app/harness/log.py:HarnessLog 的并发写模式),
|
||||||
|
写入经进程内 threading.Lock 串行化,消除多连接并发写的 database is locked。
|
||||||
|
|
||||||
|
零丢失保证范围 = 单进程、单 recorder 实例(当前 main.py / video_split_cli 均单实例
|
||||||
|
注入)。同进程多个 recorder 指向同一 db 会退回跨连接竞争——本实现不支持该场景。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import sqlite3
|
import sqlite3
|
||||||
from typing import TYPE_CHECKING
|
import threading
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
class SQLiteTelemetryRecorder:
|
class SQLiteTelemetryRecorder:
|
||||||
"""基于 SQLite 的 LLM 调用遥测记录器。
|
"""基于 SQLite 的 LLM 调用遥测记录器。
|
||||||
@@ -57,16 +59,43 @@ class SQLiteTelemetryRecorder:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, db_path: Path) -> None:
|
def __init__(self, db_path: Path) -> None:
|
||||||
self._db_path = db_path
|
"""建单持久连接 + 进程内 Lock(对齐 app/harness/log.py:HarnessLog 并发写模式)。
|
||||||
self._table_ready = False
|
|
||||||
|
|
||||||
def _ensure_table(self, conn: sqlite3.Connection) -> None:
|
把并发控制拉到进程内(threading.Lock 串行化写),消除"每次新连接并发写同一
|
||||||
"""懒初始化:首次写入时创建 llm_calls 表。"""
|
db、靠 SQLite busy_timeout 跨连接协调"在高频下撑爆 timeout → database is locked
|
||||||
if self._table_ready:
|
的根因。check_same_thread=False:record_llm_call 经 asyncio.to_thread 在线程池
|
||||||
return
|
不同线程调用,共享连接跨线程访问需此 flag,串行性由 self._lock 保证。
|
||||||
|
|
||||||
|
遥测哲学(P5):连接初始化失败降级不冒泡(self._conn=None,写入直接丢弃 warning),
|
||||||
|
绝不因遥测故障拖垮 LLM 调用 / 训练。
|
||||||
|
"""
|
||||||
|
self._db_path = db_path
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
self._conn: sqlite3.Connection | None = None
|
||||||
|
# mkdir / connect / PRAGMA / 建表统一纳入降级边界:任一失败(OSError 含
|
||||||
|
# PermissionError、sqlite3.Error)都降级为 self._conn=None,绝不冒泡拖垮初始化。
|
||||||
|
try:
|
||||||
|
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
conn = sqlite3.connect(str(db_path), check_same_thread=False)
|
||||||
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
|
conn.execute("PRAGMA busy_timeout=5000")
|
||||||
conn.execute(self._CREATE_TABLE_SQL)
|
conn.execute(self._CREATE_TABLE_SQL)
|
||||||
conn.commit()
|
conn.commit()
|
||||||
self._table_ready = True
|
self._conn = conn
|
||||||
|
except (OSError, sqlite3.Error) as exc:
|
||||||
|
logger.warning("遥测连接初始化失败(已降级,后续写入丢弃): {}", exc)
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""幂等关闭持久连接(对齐 HarnessLog;进程退出前可选调以释放 fd)。
|
||||||
|
|
||||||
|
不调也不丢数据——每次 _write 已 commit 落 WAL,进程退出 OS 回收 fd、
|
||||||
|
WAL 已提交内容下次打开自动 checkpoint 恢复。
|
||||||
|
"""
|
||||||
|
with self._lock:
|
||||||
|
if self._conn is not None:
|
||||||
|
self._conn.close()
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
def _write(
|
def _write(
|
||||||
self,
|
self,
|
||||||
@@ -87,20 +116,19 @@ class SQLiteTelemetryRecorder:
|
|||||||
cache_hit: bool,
|
cache_hit: bool,
|
||||||
error: str | None,
|
error: str | None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""同步写入一条 LLM 调用记录到 SQLite。
|
"""同步写入一条 LLM 调用记录(单持久连接 + Lock 串行化,对齐 HarnessLog)。
|
||||||
|
|
||||||
三层防御加固:
|
三层防御:
|
||||||
1. INSERT OR IGNORE — 主键冲突静默忽略
|
1. INSERT OR IGNORE — call_id 主键冲突静默忽略(幂等)
|
||||||
2. WAL + busy_timeout — 并发写锁容忍
|
2. 进程内 threading.Lock 串行化写 — 消除并发锁竞争(非依赖 SQLite busy_timeout)
|
||||||
3. try/except sqlite3.Error — DB 错误不冒泡到调用方
|
3. try/except sqlite3.Error — DB 错误降级不冒泡,遥测失败绝不拖垮 LLM 调用
|
||||||
"""
|
"""
|
||||||
|
if self._conn is None:
|
||||||
|
logger.warning("遥测连接不可用(已降级),丢弃 call_id={}", call_id)
|
||||||
|
return
|
||||||
try:
|
try:
|
||||||
conn = sqlite3.connect(str(self._db_path), timeout=10.0)
|
with self._lock:
|
||||||
try:
|
self._conn.execute(
|
||||||
conn.execute("PRAGMA journal_mode=WAL")
|
|
||||||
conn.execute("PRAGMA busy_timeout=5000")
|
|
||||||
self._ensure_table(conn)
|
|
||||||
conn.execute(
|
|
||||||
self._INSERT_SQL,
|
self._INSERT_SQL,
|
||||||
(
|
(
|
||||||
call_id,
|
call_id,
|
||||||
@@ -120,9 +148,7 @@ class SQLiteTelemetryRecorder:
|
|||||||
error,
|
error,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
conn.commit()
|
self._conn.commit()
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
except sqlite3.Error as exc:
|
except sqlite3.Error as exc:
|
||||||
logger.warning("遥测写入失败(已降级),call_id={}: {}", call_id, exc)
|
logger.warning("遥测写入失败(已降级),call_id={}: {}", call_id, exc)
|
||||||
|
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ class GovernedVLMClient:
|
|||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
parent_call_id: str | None = None,
|
parent_call_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse:
|
) -> LLMResponse:
|
||||||
"""图文调用:将图片编码为 base64 嵌入 messages,委托给 LLM 客户端。
|
"""图文调用:将图片编码为 base64 嵌入 messages,委托给 LLM 客户端。
|
||||||
|
|
||||||
@@ -45,6 +46,7 @@ class GovernedVLMClient:
|
|||||||
images: 图片文件路径列表。
|
images: 图片文件路径列表。
|
||||||
session_id: 会话 ID(遥测用)。
|
session_id: 会话 ID(遥测用)。
|
||||||
parent_call_id: 父调用 ID(遥测用)。
|
parent_call_id: 父调用 ID(遥测用)。
|
||||||
|
cache_salt: 可选缓存盐,透传到底层 LLM 缓存键。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
LLMResponse。
|
LLMResponse。
|
||||||
@@ -54,6 +56,7 @@ class GovernedVLMClient:
|
|||||||
vision_messages,
|
vision_messages,
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
parent_call_id=parent_call_id,
|
parent_call_id=parent_call_id,
|
||||||
|
cache_salt=cache_salt,
|
||||||
)
|
)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
|
|||||||
@@ -0,0 +1,266 @@
|
|||||||
|
"""离线诊断编排:把 baseline run 的错题诊断投影为逐题信号行并断点续跑落库。
|
||||||
|
|
||||||
|
"结果驱动视频级切分"离线管线的诊断步。给定一批可诊断错题:
|
||||||
|
1. 算 remaining(跳过 store 已完成题)实现续跑幂等;
|
||||||
|
2. 对剩余错题调 core.evolution.diagnose.run_diagnosis(经 StepsJsonRunLog
|
||||||
|
包装内层 RunLog,兼容 traces 未落表的历史 run);
|
||||||
|
3. 把 error_attributions / infra / degraded 三类产物确定性投影为
|
||||||
|
DiagnosisSignalRow(tier 由 split_selection.score_signal 判定);
|
||||||
|
4. run 末(Phase 3)逐行 store.upsert 落库——诊断在 Phase 2 全部跑完后才落库,
|
||||||
|
故崩溃丢本次 run 未落库的全部结果(不是"仅一行");靠 GovernedLLMClient 的
|
||||||
|
Redis 缓存缓解重跑时的 LLM 重烧,下次调用命中缓存直接续。
|
||||||
|
|
||||||
|
错误处理诚实标注(不谎称全传播):
|
||||||
|
- run_diagnosis 的 C1/C2 阶段(指标计算、错误归因)网络/API 失败经
|
||||||
|
GovernedLLMClient 重试栈后仍失败会向上抛出,本编排不捕获、不掩盖,
|
||||||
|
直接冒泡给调用方。
|
||||||
|
- 但 C3 阶段(defect/lapse judge)的调用整体包在 `except Exception` 内
|
||||||
|
(core/evolution/diagnose.py:2186),故 C3 judge 的**全部异常(含网络/API
|
||||||
|
失败)都被吞并→warning→默认归为 lapse**,不会向上抛;judge 语义歧义同样
|
||||||
|
按此保护性 fallback 处理。本编排原样接受该判定,不二次兜底、也不谎称
|
||||||
|
C3 阶段网络失败会传播。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.baseline_run_log import StepsJsonRunLog
|
||||||
|
from app.harness.split_selection import evolution_target_of, score_signal
|
||||||
|
from core.evolution.diagnose import run_diagnosis
|
||||||
|
from core.evolution.types import DiagnosisSignalRow
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from core.evolution.protocols import DiagnosisSignalStore
|
||||||
|
from core.evolution.types import DiagnosisResult
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DiagnosisDeps:
|
||||||
|
"""离线诊断编排的依赖束(一次编排的全部外部端口 + 运行参数)。
|
||||||
|
|
||||||
|
frozen 保证一次编排内依赖不可变;LLM/RunLog/SkillStore/prompts 走 Protocol
|
||||||
|
注入,便于测试替换成假实现。
|
||||||
|
|
||||||
|
属性:
|
||||||
|
run_log: 内层 RunLog 实现(提供 get_predictions/get_traces),
|
||||||
|
编排内部再用 StepsJsonRunLog 包装以兼容 traces 未落表的 run。
|
||||||
|
llm: LLM 调用端口(治理后的 GovernedLLMClient)。
|
||||||
|
skill_store: 技能文件读取端口。
|
||||||
|
prompts: 诊断模板束(DiagnosePrompts)。
|
||||||
|
tree_data: 树结构字典(多视频 {video_id: tree} 或单棵树),透传给 run_diagnosis。
|
||||||
|
concurrency: 诊断并发上限。
|
||||||
|
"""
|
||||||
|
|
||||||
|
run_log: Any
|
||||||
|
llm: Any
|
||||||
|
skill_store: Any
|
||||||
|
prompts: Any
|
||||||
|
tree_data: dict[str, Any]
|
||||||
|
concurrency: int
|
||||||
|
|
||||||
|
|
||||||
|
async def run_baseline_diagnosis(
|
||||||
|
*,
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
wrong_ids: list[str],
|
||||||
|
questions: dict[str, GeneratedQuestion],
|
||||||
|
store: DiagnosisSignalStore,
|
||||||
|
deps: DiagnosisDeps,
|
||||||
|
retry_uncertain: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""对 baseline run 的错题跑离线诊断并把信号落库(断点续跑幂等)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
baseline_run_id: baseline run 标识(如 "infer_adhoc"),信号行主键之一。
|
||||||
|
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号,主键之一。
|
||||||
|
wrong_ids: 本次待诊断的可诊断错题 question_id 列表(保序)。
|
||||||
|
questions: question_id → GeneratedQuestion 映射,需覆盖 wrong_ids 全部题
|
||||||
|
及 run_diagnosis 返回的所有 infra/degraded 题(用于取 video_id/task_type)。
|
||||||
|
store: 诊断信号存储端口,upsert 落盘并提供 done_question_ids 续跑查询。
|
||||||
|
deps: 外部依赖束(见 DiagnosisDeps)。
|
||||||
|
retry_uncertain: True 时把已落 tier='uncertain'(信号不可信降级)的题也纳入
|
||||||
|
remaining 重新诊断,透传给 store.done_question_ids;默认 False。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
None。副作用为把逐题 DiagnosisSignalRow 写入 store。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
- remaining = wrong_ids 去除 store 已完成题;空则直接 return(续跑幂等,
|
||||||
|
重复调用零副作用)。
|
||||||
|
- run_diagnosis 只诊断 remaining,避免重复 LLM 调用浪费。
|
||||||
|
- 三类产物投影互斥落库:error_attributions(defect/lapse)、infra_question_ids
|
||||||
|
(T0)、degraded_question_ids(uncertain)。
|
||||||
|
"""
|
||||||
|
# Phase 1: 算 remaining(续跑幂等)
|
||||||
|
done = store.done_question_ids(
|
||||||
|
baseline_run_id, diag_fingerprint, retry_uncertain=retry_uncertain
|
||||||
|
)
|
||||||
|
remaining = [qid for qid in wrong_ids if qid not in done]
|
||||||
|
if not remaining:
|
||||||
|
logger.info(
|
||||||
|
"离线诊断续跑:baseline={} fingerprint={} 无剩余错题(已完成 {} 题),跳过。",
|
||||||
|
baseline_run_id,
|
||||||
|
diag_fingerprint,
|
||||||
|
len(done),
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"离线诊断开始:baseline={} fingerprint={} 剩余 {}/{} 题待诊断。",
|
||||||
|
baseline_run_id,
|
||||||
|
diag_fingerprint,
|
||||||
|
len(remaining),
|
||||||
|
len(wrong_ids),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 2: 对剩余错题跑诊断(StepsJsonRunLog 兼容 traces 未落表的历史 run)
|
||||||
|
result = await run_diagnosis(
|
||||||
|
baseline_run_id,
|
||||||
|
[questions[qid] for qid in remaining],
|
||||||
|
deps.tree_data,
|
||||||
|
deps.llm,
|
||||||
|
StepsJsonRunLog(deps.run_log),
|
||||||
|
deps.skill_store,
|
||||||
|
deps.prompts,
|
||||||
|
concurrency=deps.concurrency,
|
||||||
|
question_ids=list(remaining),
|
||||||
|
only_incorrect=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 3: 投影落库
|
||||||
|
counts = _project_and_persist(
|
||||||
|
result=result,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
questions=questions,
|
||||||
|
store=store,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"离线诊断落库完成:baseline={} fingerprint={} "
|
||||||
|
"T2={} T1={} T0(infra)={} uncertain(degraded)={} 共 {} 行。",
|
||||||
|
baseline_run_id,
|
||||||
|
diag_fingerprint,
|
||||||
|
counts["T2"],
|
||||||
|
counts["T1"],
|
||||||
|
counts["T0"],
|
||||||
|
counts["uncertain"],
|
||||||
|
sum(counts.values()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _project_and_persist(
|
||||||
|
*,
|
||||||
|
result: DiagnosisResult,
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
questions: dict[str, GeneratedQuestion],
|
||||||
|
store: DiagnosisSignalStore,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""把 DiagnosisResult 三类产物投影为信号行并逐行 upsert,返回各 tier 计数。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
result: run_diagnosis 的返回,含 error_attributions/infra/degraded 三类产物。
|
||||||
|
baseline_run_id: 信号行主键之一。
|
||||||
|
diag_fingerprint: 信号行主键之一。
|
||||||
|
questions: question_id → GeneratedQuestion,用于取 video_id/task_type。
|
||||||
|
store: 诊断信号存储端口。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{tier: 行数} 计数字典(T2/T1/T0/uncertain),供上层日志与 manifest。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
本函数在 run 末(Phase 3)逐行 upsert(单行单事务);诊断已在 Phase 2 全部
|
||||||
|
跑完,故本阶段中途崩溃丢本次 run 未落库的余下行。三桶**非互斥**:
|
||||||
|
同一 degraded 错题可能同时出现在 error_attributions(judge 解析失败仍建
|
||||||
|
attribution)里,故按 **degraded > infra > attribution** 优先级去重——先落
|
||||||
|
degraded/infra,再在 attribution 循环跳过已落题,保证**每题恰写一行、
|
||||||
|
counts 恰计一次**(否则同 PK 覆盖会导致 counts 双计且分层错乱)。
|
||||||
|
"""
|
||||||
|
counts = {"T2": 0, "T1": 0, "T0": 0, "uncertain": 0}
|
||||||
|
# 优先级去重:degraded > infra > attribution。先记录高优先集合,
|
||||||
|
# attribution 循环遇到已落题即跳过,确保每题唯一落库。
|
||||||
|
persisted: set[str] = set()
|
||||||
|
|
||||||
|
# degraded_question_ids(最高优先):judge 解析失败降级 → uncertain,信号不可信排除出 T2
|
||||||
|
for qid in result.degraded_question_ids:
|
||||||
|
q = questions[qid]
|
||||||
|
store.upsert(
|
||||||
|
DiagnosisSignalRow(
|
||||||
|
question_id=qid,
|
||||||
|
video_id=q.video_id,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
task_type=q.task_type,
|
||||||
|
error_type=None,
|
||||||
|
cause_category=None,
|
||||||
|
tier="uncertain",
|
||||||
|
evolution_target=None,
|
||||||
|
degraded=True,
|
||||||
|
infra=False,
|
||||||
|
session_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
persisted.add(qid)
|
||||||
|
counts["uncertain"] += 1
|
||||||
|
|
||||||
|
# infra_question_ids:基础设施失败护栏排除 → T0,不参与训练主体
|
||||||
|
# (防御性跳过已落 degraded 题,虽 infra 通常已在诊断前过滤不重叠)
|
||||||
|
for qid in result.infra_question_ids:
|
||||||
|
if qid in persisted:
|
||||||
|
continue
|
||||||
|
q = questions[qid]
|
||||||
|
store.upsert(
|
||||||
|
DiagnosisSignalRow(
|
||||||
|
question_id=qid,
|
||||||
|
video_id=q.video_id,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
task_type=q.task_type,
|
||||||
|
error_type=None,
|
||||||
|
cause_category=None,
|
||||||
|
tier="T0",
|
||||||
|
evolution_target=None,
|
||||||
|
degraded=False,
|
||||||
|
infra=True,
|
||||||
|
session_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
persisted.add(qid)
|
||||||
|
counts["T0"] += 1
|
||||||
|
|
||||||
|
# error_attributions(最低优先):defect→T2 / lapse→T1 / 其它→uncertain(由 score_signal 判定)
|
||||||
|
# 跳过已作为 degraded/infra 落库的题,避免同 PK 覆盖与 counts 双计。
|
||||||
|
for ea in result.error_attributions:
|
||||||
|
if ea.question_id in persisted:
|
||||||
|
continue
|
||||||
|
q = questions[ea.question_id]
|
||||||
|
tier = score_signal(cause_category=ea.cause_category, infra=False, degraded=False).tier
|
||||||
|
# error_type 是 ErrorAttribution 必填字段(永远已知),确定性派生进化目标。
|
||||||
|
evolution_target = evolution_target_of(ea.error_type)
|
||||||
|
store.upsert(
|
||||||
|
DiagnosisSignalRow(
|
||||||
|
question_id=ea.question_id,
|
||||||
|
video_id=q.video_id,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
task_type=q.task_type,
|
||||||
|
error_type=ea.error_type,
|
||||||
|
cause_category=ea.cause_category,
|
||||||
|
tier=tier,
|
||||||
|
evolution_target=evolution_target,
|
||||||
|
degraded=False,
|
||||||
|
infra=False,
|
||||||
|
session_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
persisted.add(ea.question_id)
|
||||||
|
counts[tier] = counts.get(tier, 0) + 1
|
||||||
|
|
||||||
|
return counts
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""RunLog 包装器:traces 表空时从 predictions.steps_json 重建轨迹。
|
||||||
|
|
||||||
|
用于对 infer_adhoc 这类 traces 未落表、轨迹在 steps_json 的历史 run 跑离线诊断。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||||||
|
|
||||||
|
|
||||||
|
class StepsJsonRunLog:
|
||||||
|
"""委托内层 RunLog;get_traces 空表时回退 steps_json。
|
||||||
|
|
||||||
|
实现 core/evolution/protocols.py 的 RunLog Protocol(duck-typing)。
|
||||||
|
对 traces 已落表的正常 run 完全透传;仅当底层 traces 为空时,
|
||||||
|
才从 predictions.steps_json 经 steps_json_to_trace_rows 重建轨迹行。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, inner: Any) -> None:
|
||||||
|
"""构造包装器。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
inner: 内层 RunLog 实现(如 app/harness/log.py::RunLogImpl),
|
||||||
|
需提供 get_predictions / get_traces 两个 async 方法。
|
||||||
|
"""
|
||||||
|
self._inner = inner
|
||||||
|
|
||||||
|
async def get_predictions(
|
||||||
|
self, run_id: str, *, question_ids: list[str] | None = None
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""透传内层预测查询。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
run_id: 运行标识。
|
||||||
|
question_ids: 可选的题目 ID 过滤列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
内层返回的预测记录字典列表,原样透传。
|
||||||
|
"""
|
||||||
|
return await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||||
|
|
||||||
|
async def get_traces(
|
||||||
|
self, run_id: str, *, question_ids: list[str] | None = None
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""查询轨迹;底层 traces 表空时从 steps_json 回退重建。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
run_id: 运行标识。
|
||||||
|
question_ids: 可选的题目 ID 过滤列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
轨迹行字典列表。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
- 内层 traces 非空 → 原样返回,不触发回退(正常 run 路径)。
|
||||||
|
- 内层 traces 为空 → 拉取同一过滤条件下的 predictions,
|
||||||
|
逐题经 steps_json_to_trace_rows 展开为轨迹行并拼接。
|
||||||
|
- steps_json 缺失时以空串传入,由下游确定性返回 []。
|
||||||
|
"""
|
||||||
|
inner_rows = await self._inner.get_traces(run_id, question_ids=question_ids)
|
||||||
|
if inner_rows:
|
||||||
|
return inner_rows
|
||||||
|
preds = await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for p in preds:
|
||||||
|
rows.extend(
|
||||||
|
steps_json_to_trace_rows(p["video_id"], p["question_id"], p.get("steps_json") or "")
|
||||||
|
)
|
||||||
|
return rows
|
||||||
+214
-103
@@ -1,13 +1,37 @@
|
|||||||
"""混合 mini-batch 切分:大类打散、小类整锁,供 runner 每 step 处理一个 batch。"""
|
"""混合 mini-batch 切分:以 QuestionUnit 为最小调度粒度,大类打散、小类整锁。
|
||||||
|
|
||||||
|
供 runner 每 step 处理一个 batch。孪生对(AR pair)作为 2 题单元整锁不拆、按单元级
|
||||||
|
正确性分桶;非 AR single 单元的抽样/洗牌 draw 流与"引入 QuestionUnit 前"的旧逐题算法
|
||||||
|
逐字节一致(AR 折叠不干扰非 AR draw 流)。
|
||||||
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
import math
|
import math
|
||||||
import random
|
import random
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from app.harness.question_units import build_units, flatten_units, unit_correctness
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from core.types import GeneratedQuestion
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
|
||||||
|
def _rng_ns(seed: int, ns: str) -> random.Random:
|
||||||
|
"""由 (seed, 命名空间) 稳定派生独立随机数发生器。
|
||||||
|
|
||||||
|
用 SHA-256 派生而非 Python 内置 ``hash()``——后者受 hash randomization 影响,
|
||||||
|
跨进程不可复现。不同命名空间的 draw 流互不干扰,使 AR 单元折叠不扰动非 AR 抽样。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
seed: 实验随机种子。
|
||||||
|
ns: 命名空间标签(如 "AR")。
|
||||||
|
返回:
|
||||||
|
以 SHA-256(f"{ns}:{seed}") 前 8 字节为种子的 ``random.Random``。
|
||||||
|
"""
|
||||||
|
digest = hashlib.sha256(f"{ns}:{seed}".encode()).digest()
|
||||||
|
return random.Random(int.from_bytes(digest[:8], "big"))
|
||||||
|
|
||||||
|
|
||||||
def build_batches(
|
def build_batches(
|
||||||
@@ -18,55 +42,59 @@ def build_batches(
|
|||||||
seed: int,
|
seed: int,
|
||||||
correct_ratio: float = 0.0,
|
correct_ratio: float = 0.0,
|
||||||
) -> tuple[list[list[GeneratedQuestion]], int]:
|
) -> tuple[list[list[GeneratedQuestion]], int]:
|
||||||
"""把诊断池里的题目切成多个混合 mini-batch。
|
"""把诊断池里的题目切成多个混合 mini-batch(以 QuestionUnit 为原子调度单元)。
|
||||||
|
|
||||||
当 ``correct_ratio > 0`` 时,按题型为每组错题配比一定数量的正确题,使 batch
|
single 题为 1 题单元,AR pair 孪生对为 2 题单元;同一 pair 的两题整锁进同一 batch,
|
||||||
包含正误混合样本("动量"机制);``correct_ratio <= 0`` 时退化为纯错题模式。
|
按单元级正确性(双向 AND)分桶。当 ``correct_ratio > 0`` 时,按题型为每组错误单元配比
|
||||||
|
一定数量的正确单元("动量"机制);``correct_ratio <= 0`` 时退化为纯错误单元模式。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
items: 候选题目全集。
|
items: 候选题目全集(可混含 single 与孪生对成员)。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对。
|
||||||
batch_size: 单个 batch 的样本数上限(> 0)。
|
batch_size: 单个 batch 的题目数上限(> 0,pair 占 2)。
|
||||||
min_class_per_batch: 小类判定阈值——题目数 ≤ 此值的题型整组锁进单一
|
min_class_per_batch: 小类判定阈值——单元题目总数 ≤ 此值的题型整组锁进单一
|
||||||
batch(> 0)。
|
batch(> 0)。
|
||||||
seed: 随机种子,保证相同输入产出完全一致的切分。
|
seed: 随机种子,保证相同输入产出完全一致的切分。
|
||||||
correct_ratio: 正确题占比(0.0 ~ 1.0)。0.0 = 纯错题;0.5 = 错题:正确题 = 1:1。
|
correct_ratio: 正确题占比(0.0 ~ 1.0)。0.0 = 纯错误单元;0.5 = 错:正 = 1:1。
|
||||||
返回:
|
返回:
|
||||||
(非空 mini-batch 列表, selected_count);无错题时返回 ([], 0)。
|
(非空 mini-batch 列表, selected_count);无错误单元时返回 ([], 0)。
|
||||||
selected_count 是所有 batch 中题目总数。
|
selected_count 是所有 batch 中题目(展开后)总数。
|
||||||
异常:
|
异常:
|
||||||
ValueError: batch_size 或 min_class_per_batch < 1, 或
|
ValueError: batch_size 或 min_class_per_batch < 1, 或
|
||||||
min_class_per_batch >= batch_size(破坏小类整组装箱不超容的前提)。
|
min_class_per_batch >= batch_size(破坏小类整组装箱不超容的前提)。
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
装箱顺序为「先小类后大类」。小类整组用 first-fit-decreasing 装箱:按组大小
|
非 AR(single)与 AR(pair)各用独立稳定派生的 rng:非 AR 用 ``random.Random(seed)``
|
||||||
降序处理(同大小再按 task_type 排序保证确定性),每组放进第一个剩余容量足够
|
(复现旧逐题算法的确切 draw 序列,保证纯非 AR 输入逐字节一致),AR 用
|
||||||
的 batch;若现有 batch 都装不下就新开一个空 batch——因小类组大小
|
``_rng_ns(seed, "AR")``;二者 draw 流互不干扰,故加入/移除 pair 不改变非 AR 的
|
||||||
≤ min_class_per_batch < batch_size,新空 batch 必能容纳,故小类装箱永不抛
|
抽样/洗牌序列。抽样在合并前按流分别进行(``_select_mixed_by_task_type`` 各跑一次),
|
||||||
ValueError,且保证整组不拆。再把大类样本(seed 确定性 shuffle 后)round-robin
|
大类洗牌按单元 kind 拆分后各用对应流。装箱顺序「先小类后大类」:小类整组
|
||||||
分发到所有现存 batch 填充剩余容量。这样小类聚集于单 batch、大类散布多 batch
|
first-fit-decreasing(容量按单元 ``size`` 计,pair 占 2)装入首个容得下的 batch,
|
||||||
且与小类共箱,自然产生多类混合 batch(纯类切片会被 multiclass 断言拒绝)。
|
装不下新开 bin;大类洗牌后 round-robin 分发,遇碎片(size-2 单元放不进任一现存
|
||||||
nb = ceil(总题数/batch_size) 是初始 batch 数下界估计而非硬上限:小类装箱可能
|
batch 的剩余容量)新开 bin 兜底而非报错。最终每个 batch 展开回题目列表。
|
||||||
新开 bin 使实际 batch 数超过 nb。每次新开 bin 都意味着总容量随之增加,故总容量
|
题型按名称排序处理以保证跨运行确定性。
|
||||||
恒 ≥ 总题数,大类 round-robin 跳过满箱后仍能放下全部样本,不会违反 batch_size
|
|
||||||
上限。题型按名称排序处理以保证跨运行确定性,不依赖 dict 遍历顺序。
|
|
||||||
"""
|
"""
|
||||||
_validate_params(batch_size, min_class_per_batch)
|
_validate_params(batch_size, min_class_per_batch)
|
||||||
|
|
||||||
rng = random.Random(seed)
|
# 非 AR 复现旧版 random.Random(seed) 的确切序列以满足黄金 byte-identity;
|
||||||
grouped = _select_mixed_by_task_type(items, correctness, correct_ratio, rng)
|
# AR 走独立命名空间派生流,二者互不干扰。
|
||||||
total = sum(len(g) for g in grouped.values())
|
rng_nonar = random.Random(seed)
|
||||||
|
rng_ar = _rng_ns(seed, "AR")
|
||||||
|
|
||||||
|
grouped = _group_units_by_task_type(items, correctness, correct_ratio, rng_nonar, rng_ar)
|
||||||
|
|
||||||
|
total = sum(_group_load(g) for g in grouped.values())
|
||||||
if total == 0:
|
if total == 0:
|
||||||
return [], 0
|
return [], 0
|
||||||
|
|
||||||
nb = max(1, math.ceil(total / batch_size))
|
nb = max(1, math.ceil(total / batch_size))
|
||||||
batches: list[list[GeneratedQuestion]] = [[] for _ in range(nb)]
|
batches: list[list[QuestionUnit]] = [[] for _ in range(nb)]
|
||||||
|
|
||||||
small, large = _split_by_size(grouped, min_class_per_batch)
|
small, large = _split_by_size(grouped, min_class_per_batch)
|
||||||
for group in _small_groups_decreasing(small):
|
for group in _small_groups_decreasing(small):
|
||||||
_pack_small_class(batches, group, batch_size)
|
_pack_small_class(batches, group, batch_size)
|
||||||
_distribute_large_classes(batches, large, batch_size, rng)
|
_distribute_large_classes(batches, large, batch_size, rng_nonar, rng_ar)
|
||||||
|
|
||||||
result = [b for b in batches if b]
|
result = [flatten_units(b) for b in batches if b]
|
||||||
selected_count = sum(len(b) for b in result)
|
selected_count = sum(len(b) for b in result)
|
||||||
return result, selected_count
|
return result, selected_count
|
||||||
|
|
||||||
@@ -74,7 +102,7 @@ def build_batches(
|
|||||||
def _validate_params(batch_size: int, min_class_per_batch: int) -> None:
|
def _validate_params(batch_size: int, min_class_per_batch: int) -> None:
|
||||||
"""校验切分参数,非法值直接报错而非用默认值掩盖。
|
"""校验切分参数,非法值直接报错而非用默认值掩盖。
|
||||||
|
|
||||||
除各自 >= 1 外,强制 min_class_per_batch < batch_size:小类组大小 ≤
|
除各自 >= 1 外,强制 min_class_per_batch < batch_size:小类组题目总数 ≤
|
||||||
min_class_per_batch,唯有此前提成立才能保证小类整组放入单一 batch 而不超容;否则
|
min_class_per_batch,唯有此前提成立才能保证小类整组放入单一 batch 而不超容;否则
|
||||||
_pack_small_class 新开的 bin 会装入超 batch_size 的整组,静默违反容量合约。此约束
|
_pack_small_class 新开的 bin 会装入超 batch_size 的整组,静默违反容量合约。此约束
|
||||||
与 config._validate_minibatch 一致,是 build_batches 对自身前提的防御性自校验(P5)。
|
与 config._validate_minibatch 一致,是 build_batches 对自身前提的防御性自校验(P5)。
|
||||||
@@ -91,52 +119,129 @@ def _validate_params(batch_size: int, min_class_per_batch: int) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _split_by_size(
|
def _group_units_by_task_type(
|
||||||
grouped: dict[str, list[GeneratedQuestion]],
|
|
||||||
min_class_per_batch: int,
|
|
||||||
) -> tuple[dict[str, list[GeneratedQuestion]], dict[str, list[GeneratedQuestion]]]:
|
|
||||||
"""按错题数把题型分为小类(≤ 阈值)与大类(> 阈值)两组。"""
|
|
||||||
small = {t: g for t, g in grouped.items() if len(g) <= min_class_per_batch}
|
|
||||||
large = {t: g for t, g in grouped.items() if len(g) > min_class_per_batch}
|
|
||||||
return small, large
|
|
||||||
|
|
||||||
|
|
||||||
def _select_mixed_by_task_type(
|
|
||||||
items: list[GeneratedQuestion],
|
items: list[GeneratedQuestion],
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
correct_ratio: float,
|
correct_ratio: float,
|
||||||
rng: random.Random,
|
rng_nonar: random.Random,
|
||||||
) -> dict[str, list[GeneratedQuestion]]:
|
rng_ar: random.Random,
|
||||||
"""按题型分组,为每组错题按比例采样正确题混入。
|
) -> dict[str, list[QuestionUnit]]:
|
||||||
|
"""把题目聚合为单元并按题型分组:非 AR 与 AR 各走独立 draw 流后合并。
|
||||||
只对有错题的题型做混合——无错题的题型不进 batch,即使有正确题。
|
|
||||||
``correct_ratio <= 0`` 时退化为纯错题模式(向后兼容)。
|
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
items: 候选题目全集。
|
items: 候选题目全集。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对。
|
||||||
correct_ratio: 正确题占比(0.0 ~ 1.0)。
|
correct_ratio: 正确题占比。
|
||||||
rng: 随机数发生器,用于采样正确题。
|
rng_nonar: 非 AR(single 单元)抽样用 rng。
|
||||||
|
rng_ar: AR(pair 单元)抽样用 rng。
|
||||||
返回:
|
返回:
|
||||||
task_type -> 该题型的混合题目列表(错题全部 + 按比例采样的正确题)。
|
task_type -> 混合后的单元列表(single 单元在前、pair 单元在后)。
|
||||||
"""
|
"""
|
||||||
errors_by_type: dict[str, list[GeneratedQuestion]] = {}
|
units = build_units(items)
|
||||||
correct_by_type: dict[str, list[GeneratedQuestion]] = {}
|
singles = [u for u in units if u.kind == "single"]
|
||||||
for q in items:
|
pairs = [u for u in units if u.kind == "pair"]
|
||||||
qid = q.question_id
|
grouped_nonar = _select_mixed_by_task_type(singles, correctness, correct_ratio, rng_nonar)
|
||||||
if correctness.get(qid) is False:
|
grouped_ar = _select_mixed_by_task_type(pairs, correctness, correct_ratio, rng_ar)
|
||||||
errors_by_type.setdefault(q.task_type, []).append(q)
|
return _merge_grouped(grouped_nonar, grouped_ar)
|
||||||
elif correctness.get(qid, False):
|
|
||||||
correct_by_type.setdefault(q.task_type, []).append(q)
|
|
||||||
|
def _group_load(group: list[QuestionUnit]) -> int:
|
||||||
|
"""一组单元展开后的题目总数(single 计 1,pair 计 2),即占用的 batch 容量。"""
|
||||||
|
return sum(u.size for u in group)
|
||||||
|
|
||||||
|
|
||||||
|
def _batch_load(batch: list[QuestionUnit]) -> int:
|
||||||
|
"""一个 batch 内单元展开后的题目总数,用于容量判断。"""
|
||||||
|
return sum(u.size for u in batch)
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_grouped(
|
||||||
|
grouped_nonar: dict[str, list[QuestionUnit]],
|
||||||
|
grouped_ar: dict[str, list[QuestionUnit]],
|
||||||
|
) -> dict[str, list[QuestionUnit]]:
|
||||||
|
"""按 task_type 合并非 AR 与 AR 两条流的分组(single 在前、pair 在后)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
grouped_nonar: 非 AR(single 单元)分组。
|
||||||
|
grouped_ar: AR(pair 单元)分组。
|
||||||
|
返回:
|
||||||
|
task_type -> 合并后的单元列表;每类 single 单元在前、pair 单元在后,顺序稳定。
|
||||||
|
"""
|
||||||
|
merged: dict[str, list[QuestionUnit]] = {}
|
||||||
|
for task_type in sorted({*grouped_nonar, *grouped_ar}):
|
||||||
|
merged[task_type] = grouped_nonar.get(task_type, []) + grouped_ar.get(task_type, [])
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
def _split_by_size(
|
||||||
|
grouped: dict[str, list[QuestionUnit]],
|
||||||
|
min_class_per_batch: int,
|
||||||
|
) -> tuple[dict[str, list[QuestionUnit]], dict[str, list[QuestionUnit]]]:
|
||||||
|
"""按题目总数(单元展开)把题型分为小类(≤ 阈值)与大类(> 阈值)两组。"""
|
||||||
|
small = {t: g for t, g in grouped.items() if _group_load(g) <= min_class_per_batch}
|
||||||
|
large = {t: g for t, g in grouped.items() if _group_load(g) > min_class_per_batch}
|
||||||
|
return small, large
|
||||||
|
|
||||||
|
|
||||||
|
def _classify_unit(unit: QuestionUnit, correctness: dict[str, bool]) -> str | None:
|
||||||
|
"""判定单元落入哪个桶:error / correct / None(未知,跳过)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
unit: 目标单元。
|
||||||
|
correctness: question_id -> 是否答对(缺键视为未知)。
|
||||||
|
返回:
|
||||||
|
"error"(单元级正确性为 False)、"correct"(双向 AND 为 True);单元内任一题
|
||||||
|
未知(correctness 缺该键)返回 None,与旧逐题算法把未知题排除在错/对两桶之外
|
||||||
|
的语义一致。
|
||||||
|
关键实现:
|
||||||
|
先探测是否有未知题(get 返回 None ⟺ 键缺失,因 correctness 值恒为 bool),
|
||||||
|
全部已知后交由 unit_correctness 计双向 AND(此时 KeyError 不可达)。
|
||||||
|
"""
|
||||||
|
if any(correctness.get(q.question_id) is None for q in unit.questions):
|
||||||
|
return None
|
||||||
|
return "correct" if unit_correctness(unit, correctness) else "error"
|
||||||
|
|
||||||
|
|
||||||
|
def _select_mixed_by_task_type(
|
||||||
|
units: list[QuestionUnit],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
correct_ratio: float,
|
||||||
|
rng: random.Random,
|
||||||
|
) -> dict[str, list[QuestionUnit]]:
|
||||||
|
"""按题型分组,为每组错误单元按比例采样正确单元混入(单元粒度)。
|
||||||
|
|
||||||
|
只对有错误单元的题型做混合——无错误单元的题型不进 batch,即使有正确单元。
|
||||||
|
``correct_ratio <= 0`` 时退化为纯错误单元模式。本函数只处理单一 draw 流(全 single
|
||||||
|
或全 pair),使非 AR 与 AR 的抽样互不干扰。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
units: 同一流的候选单元(全 single 或全 pair)。
|
||||||
|
correctness: question_id -> 基线是否答对。
|
||||||
|
correct_ratio: 正确题占比(0.0 ~ 1.0)。
|
||||||
|
rng: 本流专用随机数发生器,用于采样正确单元。
|
||||||
|
返回:
|
||||||
|
task_type -> 该题型的混合单元列表(错误单元全部 + 按比例采样的正确单元)。
|
||||||
|
关键实现:
|
||||||
|
n_correct 按错误单元「题目总数」而非单元数计,与旧逐题语义对齐(纯 single 时
|
||||||
|
单元数 == 题目数,采样序列逐字节一致)。
|
||||||
|
"""
|
||||||
|
errors_by_type: dict[str, list[QuestionUnit]] = {}
|
||||||
|
correct_by_type: dict[str, list[QuestionUnit]] = {}
|
||||||
|
for unit in units:
|
||||||
|
bucket = _classify_unit(unit, correctness)
|
||||||
|
if bucket == "error":
|
||||||
|
errors_by_type.setdefault(unit.task_type, []).append(unit)
|
||||||
|
elif bucket == "correct":
|
||||||
|
correct_by_type.setdefault(unit.task_type, []).append(unit)
|
||||||
|
|
||||||
if correct_ratio <= 0:
|
if correct_ratio <= 0:
|
||||||
return errors_by_type
|
return errors_by_type
|
||||||
|
|
||||||
# 为每个有错题的 task_type 混入正确题
|
grouped: dict[str, list[QuestionUnit]] = {}
|
||||||
grouped: dict[str, list[GeneratedQuestion]] = {}
|
|
||||||
for task_type in sorted(errors_by_type):
|
for task_type in sorted(errors_by_type):
|
||||||
errs = errors_by_type[task_type]
|
errs = errors_by_type[task_type]
|
||||||
n_correct = round(len(errs) * correct_ratio / (1 - correct_ratio))
|
n_err = _group_load(errs)
|
||||||
|
n_correct = round(n_err * correct_ratio / (1 - correct_ratio))
|
||||||
available = correct_by_type.get(task_type, [])
|
available = correct_by_type.get(task_type, [])
|
||||||
sampled = (
|
sampled = (
|
||||||
list(available) if len(available) <= n_correct else rng.sample(available, n_correct)
|
list(available) if len(available) <= n_correct else rng.sample(available, n_correct)
|
||||||
@@ -147,94 +252,100 @@ def _select_mixed_by_task_type(
|
|||||||
|
|
||||||
|
|
||||||
def _small_groups_decreasing(
|
def _small_groups_decreasing(
|
||||||
small: dict[str, list[GeneratedQuestion]],
|
small: dict[str, list[QuestionUnit]],
|
||||||
) -> list[list[GeneratedQuestion]]:
|
) -> list[list[QuestionUnit]]:
|
||||||
"""按组大小降序、同大小按 task_type 升序排出小类组(first-fit-decreasing 顺序)。
|
"""按组题目总数降序、同大小按 task_type 升序排出小类组(first-fit-decreasing 顺序)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
small: task_type -> 小类错题列表。
|
small: task_type -> 小类单元列表。
|
||||||
返回:
|
返回:
|
||||||
排好序的小类组列表;降序处理可降低碎片,确定性 tie-break 保证跨运行一致。
|
排好序的小类组列表;降序处理可降低碎片,确定性 tie-break 保证跨运行一致。
|
||||||
"""
|
"""
|
||||||
return [small[t] for t in sorted(small, key=lambda t: (-len(small[t]), t))]
|
return [small[t] for t in sorted(small, key=lambda t: (-_group_load(small[t]), t))]
|
||||||
|
|
||||||
|
|
||||||
def _pack_small_class(
|
def _pack_small_class(
|
||||||
batches: list[list[GeneratedQuestion]],
|
batches: list[list[QuestionUnit]],
|
||||||
group: list[GeneratedQuestion],
|
group: list[QuestionUnit],
|
||||||
batch_size: int,
|
batch_size: int,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""用 first-fit 把一个小类整组放入首个容得下的 batch,装不下则新开 bin(就地修改)。
|
"""用 first-fit 把一个小类整组放入首个容得下的 batch,装不下则新开 bin(就地修改)。
|
||||||
|
|
||||||
因小类组大小 ≤ min_class_per_batch < batch_size,新开的空 batch 必能容纳整组,
|
因小类组题目总数 ≤ min_class_per_batch < batch_size,新开的空 batch 必能容纳整组,
|
||||||
故此函数永不抛 ValueError,且整组不拆。
|
故此函数永不抛 ValueError,且整组(含内部 pair 单元)不拆。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
batches: 当前各 batch(就地追加,必要时 append 新空 batch)。
|
batches: 当前各 batch(就地追加,必要时 append 新空 batch)。
|
||||||
group: 待锁定的小类错题(整组不拆)。
|
group: 待锁定的小类单元组(整组不拆)。
|
||||||
batch_size: 单 batch 容量上限。
|
batch_size: 单 batch 题目容量上限。
|
||||||
"""
|
"""
|
||||||
|
load = _group_load(group)
|
||||||
for b in batches:
|
for b in batches:
|
||||||
if len(b) + len(group) <= batch_size:
|
if _batch_load(b) + load <= batch_size:
|
||||||
b.extend(group)
|
b.extend(group)
|
||||||
return
|
return
|
||||||
batches.append(list(group))
|
batches.append(list(group))
|
||||||
|
|
||||||
|
|
||||||
def _distribute_large_classes(
|
def _distribute_large_classes(
|
||||||
batches: list[list[GeneratedQuestion]],
|
batches: list[list[QuestionUnit]],
|
||||||
large: dict[str, list[GeneratedQuestion]],
|
large: dict[str, list[QuestionUnit]],
|
||||||
batch_size: int,
|
batch_size: int,
|
||||||
rng: random.Random,
|
rng_nonar: random.Random,
|
||||||
|
rng_ar: random.Random,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""将各大类样本 shuffle 后 round-robin 分发到所有现存 batch(就地修改)。
|
"""将各大类单元洗牌后 round-robin 分发到所有现存 batch(就地修改)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
batches: 当前各 batch(含小类装箱可能新开的 bin,就地追加)。
|
batches: 当前各 batch(含小类装箱可能新开的 bin,就地追加)。
|
||||||
large: task_type -> 大类错题列表。
|
large: task_type -> 大类单元列表。
|
||||||
batch_size: 单 batch 容量上限。
|
batch_size: 单 batch 题目容量上限。
|
||||||
rng: 复用的随机数发生器,保证 shuffle 确定性。
|
rng_nonar: 非 AR(single 单元)洗牌用 rng。
|
||||||
异常:
|
rng_ar: AR(pair 单元)洗牌用 rng。
|
||||||
ValueError: 所有 batch 均满仍有样本未放置(总容量估算异常,合法输入不可达)。
|
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
轮转范围是「所有现存 batch」而非固定 nb 个——小类装箱新开的 bin 也参与分发。
|
每组按单元 kind 拆成 single 子列与 pair 子列,分别用 rng_nonar / rng_ar 洗牌后
|
||||||
总容量 = 现存 batch 数 × batch_size,每次新开 bin 都同步抬高总容量,故总容量恒
|
拼接(single 在前),使非 AR 洗牌 draw 流不受 pair 存在与否影响(纯 single 时
|
||||||
≥ 总错题数,防御性 ValueError 在合法输入下不可达。全局指针在所有大类样本间持续
|
single 子列即整组,复现旧版单一 rng.shuffle 的序列)。全局指针在所有大类单元间
|
||||||
轮转(不为每类重置),满箱即跳过,使大类充分散布并与已锁定的小类共箱。题型按名称
|
持续轮转,遇满箱跳过、遇碎片新开 bin。题型按名称排序以保证分发顺序确定。
|
||||||
排序以保证分发顺序确定。
|
|
||||||
"""
|
"""
|
||||||
nb = len(batches)
|
|
||||||
pointer = 0
|
pointer = 0
|
||||||
for task_type in sorted(large):
|
for task_type in sorted(large):
|
||||||
group = list(large[task_type])
|
group = large[task_type]
|
||||||
rng.shuffle(group)
|
singles = [u for u in group if u.kind == "single"]
|
||||||
for q in group:
|
pairs = [u for u in group if u.kind == "pair"]
|
||||||
pointer = _place_round_robin(batches, q, pointer, batch_size, nb)
|
rng_nonar.shuffle(singles)
|
||||||
|
rng_ar.shuffle(pairs)
|
||||||
|
for unit in singles + pairs:
|
||||||
|
pointer = _place_round_robin(batches, unit, pointer, batch_size)
|
||||||
|
|
||||||
|
|
||||||
def _place_round_robin(
|
def _place_round_robin(
|
||||||
batches: list[list[GeneratedQuestion]],
|
batches: list[list[QuestionUnit]],
|
||||||
q: GeneratedQuestion,
|
unit: QuestionUnit,
|
||||||
pointer: int,
|
pointer: int,
|
||||||
batch_size: int,
|
batch_size: int,
|
||||||
nb: int,
|
|
||||||
) -> int:
|
) -> int:
|
||||||
"""从 pointer 起找第一个未满 batch 放入 q,返回下一次起始指针。
|
"""从 pointer 起找第一个容量够放 unit 的 batch 放入,返回下一次起始指针。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
batches: 当前各 batch(就地追加)。
|
batches: 当前各 batch(就地追加)。
|
||||||
q: 待放置的样本。
|
unit: 待放置的单元(占用 unit.size 个容量)。
|
||||||
pointer: 本次轮转起始 batch 下标。
|
pointer: 本次轮转起始 batch 下标。
|
||||||
batch_size: 单 batch 容量上限。
|
batch_size: 单 batch 题目容量上限。
|
||||||
nb: batch 总数。
|
|
||||||
返回:
|
返回:
|
||||||
下一次轮转的起始指针(已前移一位)。
|
下一次轮转的起始指针(已前移一位)。
|
||||||
异常:
|
关键实现:
|
||||||
ValueError: 扫描一轮所有 batch 均满(总容量估算异常)。
|
单个单元容量 ≤ batch_size 是前提(pair 占 2,而 batch_size > min_class ≥ 1 ⇒
|
||||||
|
batch_size ≥ 2),故此处断言防御。扫描一轮所有现存 batch 都放不下(size-2 单元
|
||||||
|
遇满地碎片)时新开 bin 兜底而非报错——聚合容量足够但单箱剩余不足是合法碎片场景。
|
||||||
|
纯 single(size 1)永不触发新开分支,故与旧逐题 round-robin 逐字节一致。
|
||||||
"""
|
"""
|
||||||
|
assert unit.size <= batch_size, f"单元 size={unit.size} 超过 batch_size={batch_size}"
|
||||||
|
nb = len(batches)
|
||||||
for offset in range(nb):
|
for offset in range(nb):
|
||||||
idx = (pointer + offset) % nb
|
idx = (pointer + offset) % nb
|
||||||
if len(batches[idx]) < batch_size:
|
if _batch_load(batches[idx]) + unit.size <= batch_size:
|
||||||
batches[idx].append(q)
|
batches[idx].append(unit)
|
||||||
return (idx + 1) % nb
|
return (idx + 1) % nb
|
||||||
raise ValueError("所有 batch 均满仍有样本待放置, 总容量估算异常")
|
batches.append([unit])
|
||||||
|
return len(batches) % len(batches)
|
||||||
|
|||||||
@@ -0,0 +1,634 @@
|
|||||||
|
"""结果驱动视频级切分的顶层编排:诊断信号 → 冻结 pools.json + manifest。
|
||||||
|
|
||||||
|
把已实现的组件串成 capstone 管线:从 harness.db 读 canonical 基线预测、从
|
||||||
|
DiagnosisSignalStore 读逐题诊断信号,构建全视频画像、贪心联合约束选择 trainval /
|
||||||
|
test,再以视频组为原子切出诊断 / 验证池,原子冻结 pools.json 并写溯源 manifest。
|
||||||
|
全程带六条防御断言(P5,任一不满足即 fail-fast,绝不静默兜底)。
|
||||||
|
|
||||||
|
只有基线推理与诊断是上游产物;本模块纯 code-controlled,不发起任何 LLM 调用,
|
||||||
|
读预测走只读连接,不改动 harness.db。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
from dataclasses import asdict, dataclass, field
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.pools import save_pools, split_by_video_assignment
|
||||||
|
from app.harness.split_manifest import write_manifest
|
||||||
|
from app.harness.split_selection import (
|
||||||
|
SelectConfig,
|
||||||
|
build_video_records,
|
||||||
|
derive_reportable_types,
|
||||||
|
select_split,
|
||||||
|
)
|
||||||
|
from app.question_gen.loader import load_benchmark
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app.harness.pools import Pools
|
||||||
|
from app.harness.split_selection import SplitAssignment, VideoRecord
|
||||||
|
from core.evolution.protocols import DiagnosisSignalStore
|
||||||
|
from core.evolution.types import DiagnosisSignalRow
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
# 多样性主格子 = 12 题型 × 4 错误类别 = 48 格,覆盖报告以此为分母。
|
||||||
|
_DIVERSITY_GRID_TOTAL = 48
|
||||||
|
_QUESTIONS_PER_VIDEO = 3
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SplitBuildConfig:
|
||||||
|
"""结果驱动切分的旋钮快照(科研配置,随实验扫动)。
|
||||||
|
|
||||||
|
承载贪心选择器与视频组题级切分的全部可扫参数;asdict 后直接写入 manifest 的
|
||||||
|
config 快照,保证复现时可比对。floor_k 为不可哈希容器,标 hash=False 排除出
|
||||||
|
自动 __hash__,避免 frozen dataclass 被哈希时报错(本类不作字典键,仅承载配置)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
n_trainval: trainval 目标视频数(多样性阶段填充上限)。
|
||||||
|
floor_k: 各高信号 task_type 的 T2 defect 下限(select_split 硬约束)。
|
||||||
|
epsilon: test 相对全局的最大允许分布偏差(题型 / 难度两维)。
|
||||||
|
report_floor: per-type 报告门限,题数 ≥ 此值的 task_type 才入 ε 约束。
|
||||||
|
select_seed: 贪心选择器预洗牌种子(打破等增益平局)。
|
||||||
|
val_ratio: validation 占 trainval 视频组总数的比例。
|
||||||
|
split_seed: 视频组题级切分的洗牌种子。
|
||||||
|
val_wrong_min: validation 池最少错题数,切分时保证功效(不足则从 diag 换入
|
||||||
|
低 T2 错题组补足,耗尽 fail-loud)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
n_trainval: int
|
||||||
|
floor_k: dict[str, int] = field(hash=False)
|
||||||
|
epsilon: float
|
||||||
|
report_floor: int
|
||||||
|
select_seed: int
|
||||||
|
val_ratio: float
|
||||||
|
split_seed: int
|
||||||
|
val_wrong_min: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SplitBuildResult:
|
||||||
|
"""build_split 的返回结果(冻结产物 + 溯源)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
pools: 冻结的三池(diagnosis / validation / test)。
|
||||||
|
manifest: 写入 split_manifest.json 的溯源字典(含 pools_sha256)。
|
||||||
|
assignment: video_id -> "trainval" | "test" 归属字典。
|
||||||
|
"""
|
||||||
|
|
||||||
|
pools: Pools
|
||||||
|
manifest: dict
|
||||||
|
assignment: dict[str, str]
|
||||||
|
|
||||||
|
def __getitem__(self, key: str) -> object:
|
||||||
|
"""兼容字典式访问(result["pools"] / ["manifest"] / ["assignment"])。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
key: 字段名,取值 pools / manifest / assignment。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
对应字段值。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
KeyError: key 非上述三者之一。
|
||||||
|
"""
|
||||||
|
if key not in {"pools", "manifest", "assignment"}:
|
||||||
|
raise KeyError(f"未知字段: {key}")
|
||||||
|
return getattr(self, key)
|
||||||
|
|
||||||
|
|
||||||
|
def _unique_backup_path(path: Path, suffix: str) -> Path:
|
||||||
|
"""求 path 的唯一 .bak.<suffix> 备份路径,已存在则追加递增序号避免覆盖。
|
||||||
|
|
||||||
|
首选 ``<name>.bak.<suffix>``;若已存在,退化为 ``<name>.bak.<suffix>.2``、
|
||||||
|
``.3`` … 直到找到不存在的名字。保证连续 forced freeze(同 suffix 或都缺
|
||||||
|
manifest 用 'prev')不会静默覆盖此前保留的备份。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
path: 待备份的原文件路径。
|
||||||
|
suffix: 备份后缀(旧 pools_sha256 前 8 位或 'prev')。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
目录内唯一、尚不存在的备份路径。
|
||||||
|
"""
|
||||||
|
candidate = path.with_name(f"{path.name}.bak.{suffix}")
|
||||||
|
counter = 2
|
||||||
|
while candidate.exists():
|
||||||
|
candidate = path.with_name(f"{path.name}.bak.{suffix}.{counter}")
|
||||||
|
counter += 1
|
||||||
|
return candidate
|
||||||
|
|
||||||
|
|
||||||
|
def _guard_frozen_products(out_path: Path, manifest_path: Path, *, force: bool) -> None:
|
||||||
|
"""冻结前的覆盖保护:产物已存在时按 force 决定报错或备份。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
out_path: 目标 pools.json 路径。
|
||||||
|
manifest_path: 目标 split_manifest.json 路径。
|
||||||
|
force: False 时已存在即 FileExistsError;True 时把旧产物重命名为唯一的
|
||||||
|
.bak.<旧 pools_sha256 前 8 位或 'prev'>(同名已存在则追加递增序号)再放行。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
FileExistsError: force=False 且产物已存在(防静默覆盖冻结锚点)。
|
||||||
|
OSError: 备份 rename 失败;已备份的文件先 rollback 回原名再抛出,保证
|
||||||
|
要么两文件都备份、要么都不动(原子性,不留半备份的不一致目录)。
|
||||||
|
"""
|
||||||
|
if not out_path.exists() and not manifest_path.exists():
|
||||||
|
return
|
||||||
|
if not force:
|
||||||
|
raise FileExistsError(
|
||||||
|
f"已存在冻结产物 {out_path}(或其 manifest)。重跑切分会覆盖训练依赖的"
|
||||||
|
"冻结锚点——确认要替换请加 --force(旧产物将备份为 .bak.*)。"
|
||||||
|
)
|
||||||
|
# 备份后缀取旧 manifest 的 pools_sha256 前 8 位,无则用 'prev'
|
||||||
|
suffix = "prev"
|
||||||
|
if manifest_path.exists():
|
||||||
|
try:
|
||||||
|
old = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
suffix = str(old.get("pools_sha256", "prev"))[:8] or "prev"
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
suffix = "prev"
|
||||||
|
# 先为每个存在的文件求唯一备份路径(互不冲突),再逐个 rename;
|
||||||
|
# 中途失败则把已备份的 rollback 回原名,保证原子性。
|
||||||
|
to_backup = [p for p in (out_path, manifest_path) if p.exists()]
|
||||||
|
done: list[tuple[Path, Path]] = [] # (备份路径, 原路径),供 rollback
|
||||||
|
try:
|
||||||
|
for p in to_backup:
|
||||||
|
dst = _unique_backup_path(p, suffix)
|
||||||
|
p.rename(dst)
|
||||||
|
done.append((dst, p))
|
||||||
|
except OSError:
|
||||||
|
for backup_path, original in reversed(done):
|
||||||
|
backup_path.rename(original)
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def build_split(
|
||||||
|
*,
|
||||||
|
db_path: Path,
|
||||||
|
baseline_run_id: str,
|
||||||
|
signal_store: DiagnosisSignalStore,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
questions_dir: Path,
|
||||||
|
config: SplitBuildConfig,
|
||||||
|
out_path: Path,
|
||||||
|
manifest_path: Path,
|
||||||
|
generated_at: str,
|
||||||
|
force: bool = False,
|
||||||
|
) -> SplitBuildResult:
|
||||||
|
"""顶层编排结果驱动视频级切分,冻结 pools.json + manifest 并跑防御断言。
|
||||||
|
|
||||||
|
步骤:读 canonical 基线预测 → 读诊断信号 → 构建全视频画像 → 贪心选择 trainval /
|
||||||
|
test → 加载题库并以视频归属切三池 → 原子冻结 pools.json → 写溯源 manifest →
|
||||||
|
六条防御断言 fail-fast 校验。
|
||||||
|
|
||||||
|
契约(Task 11):val_wrong_min 前置到切分内保证功效——build_split 计算
|
||||||
|
wrong_tier_by_video 并连同 config.val_wrong_min 传入 split_by_video_assignment,
|
||||||
|
切分时若 val 错题不足即从 diag 换入低 T2 错题组补足(耗尽 fail-loud)。CLI 的
|
||||||
|
check_mcnemar_power 作切分冻结后的冗余最终确认。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
db_path: harness.db 路径(只读读取 predictions,不改动)。
|
||||||
|
baseline_run_id: 基线 run 标识(如 "infer_adhoc")。
|
||||||
|
signal_store: 逐题诊断信号存储端口,读 (run, fingerprint) 下全部信号行。
|
||||||
|
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号。
|
||||||
|
questions_dir: benchmark 题库目录,加载 GeneratedQuestion。
|
||||||
|
config: 切分旋钮快照。
|
||||||
|
out_path: 冻结 pools.json 目标路径(原子写)。
|
||||||
|
manifest_path: 溯源 manifest 目标路径(原子写)。
|
||||||
|
generated_at: 生成时间戳(ISO 字符串),由调用方传入以保证可复现。
|
||||||
|
force: 覆盖保护开关。False(默认)时若 out_path/manifest_path 已存在即
|
||||||
|
FileExistsError(防静默覆盖训练依赖的冻结锚点);True 时先把旧产物备份为
|
||||||
|
.bak.* 再放行覆盖。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
SplitBuildResult,含 pools / manifest / assignment,支持字典式访问。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
AssertionError: 六条防御断言任一不满足(fail-fast,不静默)。
|
||||||
|
ValueError: 上游依赖校验失败(如 correctness 缺题、assignment 非法)。
|
||||||
|
"""
|
||||||
|
# Phase 1: canonical 基线预测 + 诊断信号。
|
||||||
|
preds = load_canonical_predictions(db_path, baseline_run_id)
|
||||||
|
signal_rows_raw = signal_store.load(baseline_run_id, diag_fingerprint)
|
||||||
|
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint)
|
||||||
|
signal_rows = [
|
||||||
|
{
|
||||||
|
"question_id": row.question_id,
|
||||||
|
"task_type": row.task_type,
|
||||||
|
"error_type": row.error_type,
|
||||||
|
"tier": row.tier,
|
||||||
|
}
|
||||||
|
for row in signal_rows_raw
|
||||||
|
]
|
||||||
|
|
||||||
|
# Phase 2: 全视频画像 + 贪心联合约束选择。
|
||||||
|
videos = build_video_records(preds, signal_rows)
|
||||||
|
total_by_type = Counter(pred["task_type"] for pred in preds)
|
||||||
|
reportable_types = derive_reportable_types(dict(total_by_type), config.report_floor)
|
||||||
|
assignment_obj = select_split(
|
||||||
|
videos,
|
||||||
|
config=SelectConfig(
|
||||||
|
n_trainval=config.n_trainval,
|
||||||
|
floor_k=config.floor_k,
|
||||||
|
epsilon=config.epsilon,
|
||||||
|
reportable_types=reportable_types,
|
||||||
|
seed=config.select_seed,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assignment = _assignment_to_dict(assignment_obj)
|
||||||
|
logger.info(
|
||||||
|
"视频级切分完成: trainval={} test={} (总 {} 视频)",
|
||||||
|
len(assignment_obj.trainval),
|
||||||
|
len(assignment_obj.test),
|
||||||
|
len(videos),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 3: 加载题库 + 视频归属切三池 + 原子冻结。
|
||||||
|
questions = load_benchmark(questions_dir)
|
||||||
|
correctness = {pred["question_id"]: pred["correct"] for pred in preds}
|
||||||
|
tier_by_q = {row["question_id"]: row["tier"] for row in signal_rows}
|
||||||
|
wrong_tier_by_video: dict[str, int] = defaultdict(int)
|
||||||
|
for pred in preds:
|
||||||
|
if not pred["correct"] and tier_by_q.get(pred["question_id"]) == "T2":
|
||||||
|
wrong_tier_by_video[pred["video_id"]] += 1
|
||||||
|
pools = split_by_video_assignment(
|
||||||
|
questions,
|
||||||
|
assignment,
|
||||||
|
correctness,
|
||||||
|
config.val_ratio,
|
||||||
|
config.split_seed,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
val_wrong_min=config.val_wrong_min,
|
||||||
|
wrong_tier_by_video=dict(wrong_tier_by_video),
|
||||||
|
)
|
||||||
|
_guard_frozen_products(out_path, manifest_path, force=force)
|
||||||
|
save_pools(pools, out_path)
|
||||||
|
|
||||||
|
# Phase 4: 溯源 manifest(pools_sha256 锚定冻结内容)。
|
||||||
|
coverage_report = _build_coverage_report(
|
||||||
|
videos, assignment_obj, signal_rows_raw, reportable_types, config
|
||||||
|
)
|
||||||
|
manifest = write_manifest(
|
||||||
|
manifest_path,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
seed=config.split_seed,
|
||||||
|
config=asdict(config),
|
||||||
|
pools_json_text=out_path.read_text(encoding="utf-8"),
|
||||||
|
coverage_report=coverage_report,
|
||||||
|
generated_at=generated_at,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 5: 防御断言 fail-fast。
|
||||||
|
_assert_split_invariants(
|
||||||
|
pools=pools,
|
||||||
|
expected_question_ids={q.question_id for q in questions},
|
||||||
|
signal_rows_raw=signal_rows_raw,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
out_path=out_path,
|
||||||
|
manifest=manifest,
|
||||||
|
)
|
||||||
|
|
||||||
|
return SplitBuildResult(pools=pools, manifest=manifest, assignment=assignment)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_choice(choice: str | None) -> str:
|
||||||
|
"""选项归一:strip → 大写 → 取首字母,None 归一为空串。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
choice: 原始选项文本(预测或答案),可为 None。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
归一后的单字母(无内容时为空串)。
|
||||||
|
"""
|
||||||
|
return (choice or "").strip().upper()[:1]
|
||||||
|
|
||||||
|
|
||||||
|
def load_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
|
||||||
|
"""从 harness.db 只读取指定 run 每题首行(ORDER BY rowid)为 canonical 预测。
|
||||||
|
|
||||||
|
共享口径 helper:CLI(可诊断错题筛选 + INFRA T0 补录)与 build_split(切分)
|
||||||
|
共用同一"每 qid 取 rowid 最小首行 + 归一化 correct 判定"口径,消除两处重复实现。
|
||||||
|
同一 question_id 可能有多行(重跑 / 补测),canonical 口径取 rowid 最小的首行,
|
||||||
|
保证 distinct question 计数与对错判定确定。correct = 预测与答案各自归一
|
||||||
|
(strip → 大写 → 取首字母)后逐字符相等。
|
||||||
|
|
||||||
|
口径边界:旧 build_or_load_pools 的 legacy 池构建路径(app/harness/pools.py)是
|
||||||
|
另一条独立既有链路,不共用本 helper,两者刻意不统一(本次不动 legacy 路径)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
db_path: harness.db 路径(URI mode=ro 只读打开,绝不改动基线 db)。
|
||||||
|
baseline_run_id: 基线 run 标识。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
canonical 预测行列表,每行含 question_id / video_id / task_type /
|
||||||
|
prediction / answer / stop_reason / correct(bool)。按 rowid 升序去重,
|
||||||
|
每 qid 保留首行。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: 该 run 无任何预测行(fail-fast,不返回空切分)。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
按 rowid 升序遍历,首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
|
||||||
|
"""
|
||||||
|
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
try:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT question_id, video_id, task_type, prediction, answer, stop_reason "
|
||||||
|
"FROM predictions WHERE run_id = ? ORDER BY rowid",
|
||||||
|
(baseline_run_id,),
|
||||||
|
).fetchall()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
canonical: dict[str, dict] = {}
|
||||||
|
for row in rows:
|
||||||
|
qid = row["question_id"]
|
||||||
|
if qid in canonical:
|
||||||
|
continue
|
||||||
|
canonical[qid] = {
|
||||||
|
"question_id": qid,
|
||||||
|
"video_id": row["video_id"],
|
||||||
|
"task_type": row["task_type"],
|
||||||
|
"prediction": row["prediction"],
|
||||||
|
"answer": row["answer"],
|
||||||
|
"stop_reason": row["stop_reason"],
|
||||||
|
"correct": _normalize_choice(row["prediction"]) == _normalize_choice(row["answer"]),
|
||||||
|
}
|
||||||
|
if not canonical:
|
||||||
|
raise ValueError(f"run_id={baseline_run_id} 无任何预测行,无法切分")
|
||||||
|
return list(canonical.values())
|
||||||
|
|
||||||
|
|
||||||
|
def _assignment_to_dict(assignment_obj: SplitAssignment) -> dict[str, str]:
|
||||||
|
"""把 SplitAssignment 展平为 video_id -> "trainval" | "test" 归属字典。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
assignment_obj: 贪心选择器产出的切分归属。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
全部视频的归属字典(trainval 与 test 并集,键互斥)。
|
||||||
|
"""
|
||||||
|
assignment = dict.fromkeys(assignment_obj.trainval, "trainval")
|
||||||
|
for vid in assignment_obj.test:
|
||||||
|
assignment[vid] = "test"
|
||||||
|
return assignment
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_fingerprint_consistent(
|
||||||
|
signal_rows_raw: list[DiagnosisSignalRow],
|
||||||
|
diag_fingerprint: str,
|
||||||
|
) -> None:
|
||||||
|
"""防御④:全部诊断信号行的 diag_fingerprint 必须与传入指纹一致。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
signal_rows_raw: store 读回的诊断信号行。
|
||||||
|
diag_fingerprint: 期望的诊断口径指纹。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
AssertionError: 存在指纹不一致的信号行(store 未正确按指纹过滤)。
|
||||||
|
"""
|
||||||
|
mismatched = [
|
||||||
|
row.question_id for row in signal_rows_raw if row.diag_fingerprint != diag_fingerprint
|
||||||
|
]
|
||||||
|
if mismatched:
|
||||||
|
raise AssertionError(
|
||||||
|
f"诊断信号指纹不一致 {len(mismatched)} 行,期望 {diag_fingerprint}: {mismatched[:5]}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fraction_by_type(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
|
||||||
|
"""各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: 视频记录子集。
|
||||||
|
keys: 需计算占比的 task_type 键集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{task_type: 占比};records 为空时全部记 0.0。
|
||||||
|
"""
|
||||||
|
total = len(records)
|
||||||
|
if total == 0:
|
||||||
|
return dict.fromkeys(keys, 0.0)
|
||||||
|
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
|
||||||
|
|
||||||
|
|
||||||
|
def _fraction_by_difficulty(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
|
||||||
|
"""各难度桶(错题数)在给定视频集中的占比。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: 视频记录子集。
|
||||||
|
buckets: 难度桶键集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{难度桶: 占比};records 为空时全部记 0.0。
|
||||||
|
"""
|
||||||
|
total = len(records)
|
||||||
|
if total == 0:
|
||||||
|
return dict.fromkeys(buckets, 0.0)
|
||||||
|
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
|
||||||
|
|
||||||
|
|
||||||
|
def _max_dev(global_dist: dict, subset_dist: dict, keys: set) -> float:
|
||||||
|
"""逐键取全局与子集分布的最大绝对偏差(键集为空约定 0.0)。"""
|
||||||
|
if not keys:
|
||||||
|
return 0.0
|
||||||
|
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_coverage_report(
|
||||||
|
videos: list[VideoRecord],
|
||||||
|
assignment_obj: SplitAssignment,
|
||||||
|
signal_rows_raw: list[DiagnosisSignalRow],
|
||||||
|
reportable_types: set[str],
|
||||||
|
config: SplitBuildConfig,
|
||||||
|
) -> dict:
|
||||||
|
"""组装 manifest 覆盖报告:48 格覆盖 / floor 达标 / test 代表性偏差 / tier 占比。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
videos: 全视频画像记录。
|
||||||
|
assignment_obj: 切分归属(trainval / test)。
|
||||||
|
signal_rows_raw: 诊断信号行(统计 tier 占比)。
|
||||||
|
reportable_types: 参与 ε 代表性校验的题型集。
|
||||||
|
config: 切分旋钮(floor_k / epsilon)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
覆盖报告字典,含 cells_covered / grid_total / floor_satisfied /
|
||||||
|
test_representativeness_deviation / tier_distribution /
|
||||||
|
evolution_target_distribution。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
evolution_target_distribution 只统计 T2 信号(可训练缺陷),按
|
||||||
|
tool / skill / system 计数,报告"哪层参数组拿到梯度";T0/T1/uncertain 行
|
||||||
|
evolution_target 恒为 None,不入该分布。
|
||||||
|
"""
|
||||||
|
by_id = {v.video_id: v for v in videos}
|
||||||
|
trainval = [by_id[vid] for vid in assignment_obj.trainval]
|
||||||
|
test = [by_id[vid] for vid in assignment_obj.test]
|
||||||
|
|
||||||
|
covered_cells: set[tuple[str, str]] = set()
|
||||||
|
trainval_wrong: Counter[str] = Counter()
|
||||||
|
for video in trainval:
|
||||||
|
covered_cells |= set(video.cells)
|
||||||
|
trainval_wrong.update(video.wrong_by_type)
|
||||||
|
floor_satisfied = {
|
||||||
|
task_type: trainval_wrong.get(task_type, 0) >= floor
|
||||||
|
for task_type, floor in config.floor_k.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
type_dev = _max_dev(
|
||||||
|
_fraction_by_type(videos, reportable_types),
|
||||||
|
_fraction_by_type(test, reportable_types),
|
||||||
|
reportable_types,
|
||||||
|
)
|
||||||
|
diff_buckets = {r.difficulty for r in videos}
|
||||||
|
diff_dev = _max_dev(
|
||||||
|
_fraction_by_difficulty(videos, diff_buckets),
|
||||||
|
_fraction_by_difficulty(test, diff_buckets),
|
||||||
|
diff_buckets,
|
||||||
|
)
|
||||||
|
|
||||||
|
tier_distribution, evolution_target_distribution = _signal_distributions(signal_rows_raw)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"cells_covered": len(covered_cells),
|
||||||
|
"grid_total": _DIVERSITY_GRID_TOTAL,
|
||||||
|
"floor_satisfied": floor_satisfied,
|
||||||
|
"test_representativeness_deviation": {
|
||||||
|
"type_max": type_dev,
|
||||||
|
"difficulty_max": diff_dev,
|
||||||
|
"epsilon": config.epsilon,
|
||||||
|
},
|
||||||
|
"tier_distribution": tier_distribution,
|
||||||
|
"evolution_target_distribution": evolution_target_distribution,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _signal_distributions(
|
||||||
|
signal_rows_raw: list[DiagnosisSignalRow],
|
||||||
|
) -> tuple[dict[str, float], dict[str, int]]:
|
||||||
|
"""由诊断信号行算 tier 占比分布与 T2 进化目标计数分布。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
signal_rows_raw: 诊断信号行。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
(tier_distribution, evolution_target_distribution) 二元组:
|
||||||
|
- tier_distribution: {tier: 占比},无信号时为空 dict;
|
||||||
|
- evolution_target_distribution: 仅统计 T2(可训练缺陷)信号,按
|
||||||
|
tool / skill / system 计数,报告哪层参数组拿到梯度;T0/T1/uncertain 行
|
||||||
|
evolution_target 恒为 None,不入该分布。
|
||||||
|
"""
|
||||||
|
tier_counts = Counter(row.tier for row in signal_rows_raw)
|
||||||
|
total_signals = sum(tier_counts.values())
|
||||||
|
tier_distribution = (
|
||||||
|
{tier: count / total_signals for tier, count in tier_counts.items()}
|
||||||
|
if total_signals
|
||||||
|
else {}
|
||||||
|
)
|
||||||
|
evolution_target_distribution = dict(
|
||||||
|
Counter(
|
||||||
|
row.evolution_target
|
||||||
|
for row in signal_rows_raw
|
||||||
|
if row.tier == "T2" and row.evolution_target is not None
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tier_distribution, evolution_target_distribution
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_split_invariants(
|
||||||
|
*,
|
||||||
|
pools: Pools,
|
||||||
|
expected_question_ids: set[str],
|
||||||
|
signal_rows_raw: list[DiagnosisSignalRow],
|
||||||
|
diag_fingerprint: str,
|
||||||
|
out_path: Path,
|
||||||
|
manifest: dict,
|
||||||
|
) -> None:
|
||||||
|
"""六条防御断言 fail-fast:任一不满足即 AssertionError(P5,不静默不兜底)。
|
||||||
|
|
||||||
|
① 三池视频集两两不相交;② 三池覆盖全部题(按 distinct question);
|
||||||
|
③ 每 video 恰 3 题;④ 诊断信号指纹一致;⑤ manifest pools_sha256 == sha256(冻结内容);
|
||||||
|
⑥ question_id 全局唯一(无重复行)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
pools: 冻结的三池。
|
||||||
|
expected_question_ids: 加载题库的 question_id 全集(覆盖基准)。
|
||||||
|
signal_rows_raw: 诊断信号行(指纹校验)。
|
||||||
|
diag_fingerprint: 期望诊断指纹。
|
||||||
|
out_path: 冻结 pools.json 路径。
|
||||||
|
manifest: 已写入的 manifest 字典。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
AssertionError: 任一防御断言不满足。
|
||||||
|
"""
|
||||||
|
all_questions = pools.diagnosis + pools.validation + pools.test
|
||||||
|
_assert_pools_video_disjoint(pools) # ①
|
||||||
|
_assert_question_ids_unique(all_questions) # ⑥
|
||||||
|
_assert_question_coverage(all_questions, expected_question_ids) # ②
|
||||||
|
_assert_three_questions_per_video(all_questions) # ③
|
||||||
|
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint) # ④
|
||||||
|
_assert_pools_sha256(out_path, manifest) # ⑤
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_pools_video_disjoint(pools: Pools) -> None:
|
||||||
|
"""防御①:diagnosis / validation / test 三池视频集两两不相交。"""
|
||||||
|
diag_v = {q.video_id for q in pools.diagnosis}
|
||||||
|
val_v = {q.video_id for q in pools.validation}
|
||||||
|
test_v = {q.video_id for q in pools.test}
|
||||||
|
if diag_v & val_v or diag_v & test_v or val_v & test_v:
|
||||||
|
raise AssertionError(
|
||||||
|
f"三池视频集非互斥: diag∩val={diag_v & val_v}, "
|
||||||
|
f"diag∩test={diag_v & test_v}, val∩test={val_v & test_v}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_question_ids_unique(all_questions: list[GeneratedQuestion]) -> None:
|
||||||
|
"""防御⑥:三池合并后 question_id 全局唯一(无重复行)。"""
|
||||||
|
qids = [q.question_id for q in all_questions]
|
||||||
|
if len(qids) != len(set(qids)):
|
||||||
|
duplicates = [qid for qid, count in Counter(qids).items() if count > 1]
|
||||||
|
raise AssertionError(f"question_id 重复 {len(duplicates)} 个: {duplicates[:5]}")
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_question_coverage(
|
||||||
|
all_questions: list[GeneratedQuestion],
|
||||||
|
expected_question_ids: set[str],
|
||||||
|
) -> None:
|
||||||
|
"""防御②:三池覆盖题库全部题(按 distinct question,缺题 / 多题均 fail-fast)。"""
|
||||||
|
actual = {q.question_id for q in all_questions}
|
||||||
|
if actual != expected_question_ids:
|
||||||
|
missing = expected_question_ids - actual
|
||||||
|
extra = actual - expected_question_ids
|
||||||
|
raise AssertionError(
|
||||||
|
f"三池题目覆盖不完整: 缺 {len(missing)} 多 {len(extra)} (缺样例 {sorted(missing)[:5]})"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_three_questions_per_video(all_questions: list[GeneratedQuestion]) -> None:
|
||||||
|
"""防御③:每 video 恰 3 题(视频组原子切分不应劈裂视频的题)。"""
|
||||||
|
per_video = Counter(q.video_id for q in all_questions)
|
||||||
|
bad_videos = {vid: n for vid, n in per_video.items() if n != _QUESTIONS_PER_VIDEO}
|
||||||
|
if bad_videos:
|
||||||
|
raise AssertionError(
|
||||||
|
f"存在 video 题数 != {_QUESTIONS_PER_VIDEO}: {dict(list(bad_videos.items())[:5])}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_pools_sha256(out_path: Path, manifest: dict) -> None:
|
||||||
|
"""防御⑤:manifest 的 pools_sha256 == sha256(冻结 pools.json 内容)。"""
|
||||||
|
actual_sha = hashlib.sha256(out_path.read_text(encoding="utf-8").encode("utf-8")).hexdigest()
|
||||||
|
if actual_sha != manifest["pools_sha256"]:
|
||||||
|
raise AssertionError(
|
||||||
|
f"pools_sha256 不一致: manifest={manifest['pools_sha256']} 实际={actual_sha}"
|
||||||
|
)
|
||||||
@@ -41,6 +41,7 @@ _STRUCTURAL_KEYS = (
|
|||||||
"diag_size",
|
"diag_size",
|
||||||
"val_size",
|
"val_size",
|
||||||
"batch_correct_ratio",
|
"batch_correct_ratio",
|
||||||
|
"trainable_min_units",
|
||||||
)
|
)
|
||||||
|
|
||||||
_DECISION_KEYS = (
|
_DECISION_KEYS = (
|
||||||
@@ -57,7 +58,6 @@ _DECISION_KEYS = (
|
|||||||
"gate_delta_min",
|
"gate_delta_min",
|
||||||
"gate_lambda_dir",
|
"gate_lambda_dir",
|
||||||
"gate_e_rollback",
|
"gate_e_rollback",
|
||||||
"gate_block",
|
|
||||||
"gate_n_max",
|
"gate_n_max",
|
||||||
"gate_p_low",
|
"gate_p_low",
|
||||||
"gate_p_high",
|
"gate_p_high",
|
||||||
@@ -94,7 +94,7 @@ def serialize_state(state: Any) -> dict[str, Any]:
|
|||||||
"eval_prev_run_id": state.eval_prev_run_id,
|
"eval_prev_run_id": state.eval_prev_run_id,
|
||||||
"baseline_skills_version": state.baseline_skills_version,
|
"baseline_skills_version": state.baseline_skills_version,
|
||||||
"baseline_prompts_version": state.baseline_prompts_version,
|
"baseline_prompts_version": state.baseline_prompts_version,
|
||||||
"steps_since_best_improved": state.steps_since_best_improved,
|
"epochs_since_best_improved": state.epochs_since_best_improved,
|
||||||
"epoch_start_skills": state.epoch_start_skills,
|
"epoch_start_skills": state.epoch_start_skills,
|
||||||
"changed_task_types_this_epoch": sorted(state.changed_task_types_this_epoch),
|
"changed_task_types_this_epoch": sorted(state.changed_task_types_this_epoch),
|
||||||
"rejected_buffer": {k: [asdict(x) for x in v] for k, v in state.rejected_buffer.items()},
|
"rejected_buffer": {k: [asdict(x) for x in v] for k, v in state.rejected_buffer.items()},
|
||||||
@@ -145,7 +145,7 @@ def deserialize_state_fields(d: dict[str, Any]) -> dict[str, Any]:
|
|||||||
"eval_prev_run_id": d["eval_prev_run_id"],
|
"eval_prev_run_id": d["eval_prev_run_id"],
|
||||||
"baseline_skills_version": d["baseline_skills_version"],
|
"baseline_skills_version": d["baseline_skills_version"],
|
||||||
"baseline_prompts_version": d["baseline_prompts_version"],
|
"baseline_prompts_version": d["baseline_prompts_version"],
|
||||||
"steps_since_best_improved": d["steps_since_best_improved"],
|
"epochs_since_best_improved": d["epochs_since_best_improved"],
|
||||||
"epoch_start_skills": d["epoch_start_skills"],
|
"epoch_start_skills": d["epoch_start_skills"],
|
||||||
"changed_task_types_this_epoch": set(d["changed_task_types_this_epoch"]),
|
"changed_task_types_this_epoch": set(d["changed_task_types_this_epoch"]),
|
||||||
"rejected_buffer": {
|
"rejected_buffer": {
|
||||||
@@ -233,7 +233,8 @@ def write_checkpoint(
|
|||||||
global_step: 全局 step 序号。
|
global_step: 全局 step 序号。
|
||||||
total_steps: 全局总 step 数。
|
total_steps: 全局总 step 数。
|
||||||
version_snapshot: skills/prompts 版本快照。
|
version_snapshot: skills/prompts 版本快照。
|
||||||
epoch_batches: 本 epoch 的 batch 划分(question_id 列表的列表)。
|
epoch_batches: 本 epoch 的 batch 划分(unit_id 列表的列表,孪生对折叠为
|
||||||
|
单个 unit_id;纯非 AR 下 unit_id==question_id)。
|
||||||
config: 训练配置对象,用于计算 config_fingerprint。
|
config: 训练配置对象,用于计算 config_fingerprint。
|
||||||
|
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ class RunConfig:
|
|||||||
batch_size: mini-batch 单批题目数。
|
batch_size: mini-batch 单批题目数。
|
||||||
min_class_per_batch: 单批中每个任务类型至少保留的题目数(< batch_size)。
|
min_class_per_batch: 单批中每个任务类型至少保留的题目数(< batch_size)。
|
||||||
eval_min_per_class: 验证池中每个任务类型至少保底的题目数。
|
eval_min_per_class: 验证池中每个任务类型至少保底的题目数。
|
||||||
|
trainable_min_units: 可训练性预检:每题型 diag+val 单元数下限,低于则剔除该题型。
|
||||||
early_stop_patience: 全局 best 连续未提升的容忍轮数,达到即早停。
|
early_stop_patience: 全局 best 连续未提升的容忍轮数,达到即早停。
|
||||||
test_size: held-out 测试池题目数。
|
test_size: held-out 测试池题目数。
|
||||||
use_slow_momentum: 是否启用快慢双速进化中的慢速 momentum 更新。
|
use_slow_momentum: 是否启用快慢双速进化中的慢速 momentum 更新。
|
||||||
@@ -69,14 +70,13 @@ class RunConfig:
|
|||||||
gate_delta_min: 最小点估计效应量下限(承接旧 margin 语义)。
|
gate_delta_min: 最小点估计效应量下限(承接旧 margin 语义)。
|
||||||
gate_lambda_dir: Wald 方向拒绝的对数似然比阈值(必须为负)。
|
gate_lambda_dir: Wald 方向拒绝的对数似然比阈值(必须为负)。
|
||||||
gate_e_rollback: 试用期对称回滚门(回滚 e 值门槛)。
|
gate_e_rollback: 试用期对称回滚门(回滚 e 值门槛)。
|
||||||
gate_block: 块序贯验证的块大小(=推理并发度,块内跑满)。
|
|
||||||
gate_n_max: 单次 gate 消耗的题数上限。
|
gate_n_max: 单次 gate 消耗的题数上限。
|
||||||
gate_p_low: 信息量阶梯 p-hat 保留区间下界(剔除必错零信息题)。
|
gate_p_low: 信息量阶梯 p-hat 保留区间下界(剔除必错零信息题)。
|
||||||
gate_p_high: 信息量阶梯 p-hat 保留区间上界(剔除必对零信息题)。
|
gate_p_high: 信息量阶梯 p-hat 保留区间上界(剔除必对零信息题)。
|
||||||
gate_probe_quota: 冷启动探针集比例(全错题中插尾的比例)。
|
gate_probe_quota: 冷启动探针集比例(全错题中插尾的比例)。
|
||||||
gate_gamma_decay: 逐题正确率估计 p-hat 的 EMA 衰减系数。
|
gate_gamma_decay: 逐题正确率估计 p-hat 的 EMA 衰减系数。
|
||||||
gate_cooldown_steps: 回滚后该题型跳过进化的冷却 step 数。
|
gate_cooldown_steps: 回滚后该题型跳过进化的冷却 step 数。
|
||||||
gate_guard_err: gate 内跨块累计 INFRA 错误率护栏。
|
gate_guard_err: gate 内累计 INFRA 错误率护栏。
|
||||||
skill_update_mode: skill 进化模式,"patch"(局部 edit)/ "rewrite"(整篇重写)。
|
skill_update_mode: skill 进化模式,"patch"(局部 edit)/ "rewrite"(整篇重写)。
|
||||||
appendix_consolidate_threshold: appendix note 条数达此值触发 LLM consolidation。
|
appendix_consolidate_threshold: appendix note 条数达此值触发 LLM consolidation。
|
||||||
run_id: diagnose/evolve 模式要分析的运行 ID,默认空字符串。
|
run_id: diagnose/evolve 模式要分析的运行 ID,默认空字符串。
|
||||||
@@ -114,6 +114,7 @@ class RunConfig:
|
|||||||
batch_size: int
|
batch_size: int
|
||||||
min_class_per_batch: int
|
min_class_per_batch: int
|
||||||
eval_min_per_class: int
|
eval_min_per_class: int
|
||||||
|
trainable_min_units: int
|
||||||
early_stop_patience: int
|
early_stop_patience: int
|
||||||
test_size: int
|
test_size: int
|
||||||
use_slow_momentum: bool
|
use_slow_momentum: bool
|
||||||
@@ -123,7 +124,6 @@ class RunConfig:
|
|||||||
gate_delta_min: float
|
gate_delta_min: float
|
||||||
gate_lambda_dir: float
|
gate_lambda_dir: float
|
||||||
gate_e_rollback: float
|
gate_e_rollback: float
|
||||||
gate_block: int
|
|
||||||
gate_n_max: int
|
gate_n_max: int
|
||||||
gate_p_low: float
|
gate_p_low: float
|
||||||
gate_p_high: float
|
gate_p_high: float
|
||||||
@@ -297,6 +297,8 @@ def _validate_minibatch(config: RunConfig) -> None:
|
|||||||
)
|
)
|
||||||
if config.eval_min_per_class < 1:
|
if config.eval_min_per_class < 1:
|
||||||
raise ValueError(f"eval_min_per_class 必须 >= 1,实际: {config.eval_min_per_class}")
|
raise ValueError(f"eval_min_per_class 必须 >= 1,实际: {config.eval_min_per_class}")
|
||||||
|
if config.trainable_min_units < 1:
|
||||||
|
raise ValueError(f"trainable_min_units 必须 >= 1,实际: {config.trainable_min_units}")
|
||||||
if config.pool_split_mode != "per_category":
|
if config.pool_split_mode != "per_category":
|
||||||
floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT
|
floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT
|
||||||
if config.val_size < floor:
|
if config.val_size < floor:
|
||||||
@@ -357,7 +359,7 @@ def _validate_gate_thresholds(config: RunConfig) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def _validate_gate_ladder(config: RunConfig) -> None:
|
def _validate_gate_ladder(config: RunConfig) -> None:
|
||||||
"""校验 CE-Gate 信息量阶梯与块序贯参数。
|
"""校验 CE-Gate 信息量阶梯参数。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
config: 待校验的配置实例。
|
config: 待校验的配置实例。
|
||||||
@@ -365,11 +367,8 @@ def _validate_gate_ladder(config: RunConfig) -> None:
|
|||||||
异常:
|
异常:
|
||||||
ValueError: 任一阶梯参数不合法。
|
ValueError: 任一阶梯参数不合法。
|
||||||
"""
|
"""
|
||||||
if config.gate_block <= 0 or config.gate_n_max < config.gate_block:
|
if config.gate_n_max <= 0:
|
||||||
raise ValueError(
|
raise ValueError(f"需 gate_n_max > 0,实际: n_max={config.gate_n_max}")
|
||||||
f"需 0 < gate_block <= gate_n_max,"
|
|
||||||
f"实际: block={config.gate_block}, n_max={config.gate_n_max}"
|
|
||||||
)
|
|
||||||
if not (0 <= config.gate_p_low < config.gate_p_high <= 1):
|
if not (0 <= config.gate_p_low < config.gate_p_high <= 1):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"需 0 <= gate_p_low < gate_p_high <= 1,"
|
f"需 0 <= gate_p_low < gate_p_high <= 1,"
|
||||||
|
|||||||
+131
-62
@@ -1,13 +1,18 @@
|
|||||||
"""CE-Gate 信息量阶梯与基线缓存。
|
"""CE-Gate 信息量阶梯与基线缓存(unit 粒度,核心算法保真 #5)。
|
||||||
|
|
||||||
阶梯(每题型一条):gate 的出题顺序表。冷启动(FRESH)用种子基线对错
|
阶梯(每题型一条):gate 的出题顺序表,键为 **unit_id**(single 题 unit_id
|
||||||
两档粗排(错题高优先 2:1 交错 + 全错题 probe_quota 探针插尾);
|
等于 question_id,AR pair 折叠为一个单元、unit_id 等于共享 pair_id)。冷启动
|
||||||
epoch >=1 用非 gate run 观测做 gamma-EMA 更新 p_hat,按信息量 p_hat(1-p_hat) 降序、
|
(FRESH)用种子基线的**单元级**对错两档粗排(错 unit 高优先 2:1 交错 + 全错
|
||||||
剔 p_hat 不在 [p_low, p_high]。防泄露铁律:gate 内 rollout 永不回流 p_hat
|
unit 的 probe_quota 探针插尾);epoch >=1 用非 gate run 观测**折叠成单元观测**后做
|
||||||
(调用方以 run_id 含 "_gate_" 过滤观测源)。
|
gamma-EMA 更新 p_hat,按信息量 p_hat(1-p_hat) 降序、剔 p_hat 不在 [p_low, p_high]。
|
||||||
|
单元错 = 该单元任一成员错(AR pair 双向 AND)。防泄露铁律:gate 内 rollout 永不
|
||||||
|
回流 p_hat(调用方以 run_id 含 "_gate_" 过滤观测源),本迁移不改此过滤。
|
||||||
|
|
||||||
BaselineCache:基线侧逐题对错缓存,键 = (task_type, skill_hash,
|
持久化门控:gate_pools.json 带 schema_version(当前 = 2,unit 键)。旧版无
|
||||||
prompts_version, qid) 内容寻址、无显式失效。JSON 持久化到 workspace,
|
schema_version(v1、qid 键)加载时**直接报错**,拒绝静默混用 qid/unit 键。
|
||||||
|
|
||||||
|
BaselineCache:基线侧单元级对错缓存,键 = (task_type, skill_hash,
|
||||||
|
prompts_version, unit_id) 内容寻址、无显式失效。JSON 持久化到 workspace,
|
||||||
供 resume 后合法复用已冻结阶梯上的新鲜 draw。
|
供 resume 后合法复用已冻结阶梯上的新鲜 draw。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -22,10 +27,16 @@ from typing import TYPE_CHECKING
|
|||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.question_units import build_units, unit_correctness
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from core.types import GeneratedQuestion
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
# gate_pools.json 结构版本。v1(隐式、无此字段)为逐题 qid 键的存量格式;
|
||||||
|
# v2 起改为 unit_id 键。load 时严格校验,不匹配即报错(不静默迁移/混用)。
|
||||||
|
SCHEMA_VERSION = 2
|
||||||
|
|
||||||
|
|
||||||
def skill_hash(content: str) -> str:
|
def skill_hash(content: str) -> str:
|
||||||
@@ -42,50 +53,52 @@ def skill_hash(content: str) -> str:
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class LadderEntry:
|
class LadderEntry:
|
||||||
"""阶梯单元:题目与其估计答对率。
|
"""阶梯单元:题目单元与其估计答对率。
|
||||||
|
|
||||||
字段:
|
字段:
|
||||||
question_id: 题目唯一标识。
|
unit_id: 单元唯一标识(single 等于 question_id,AR pair 等于共享 pair_id)。
|
||||||
p_hat: 估计答对率。冷启动为 Beta(1,1) 平滑的单次观测后验均值
|
p_hat: 估计答对率。冷启动为 Beta(1,1) 平滑的单次观测后验均值
|
||||||
(错=1/3、对=2/3),此后经 gamma-EMA 更新。
|
(错=1/3、对=2/3),此后经 gamma-EMA 更新。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
question_id: str
|
unit_id: str
|
||||||
p_hat: float
|
p_hat: float
|
||||||
|
|
||||||
|
|
||||||
def build_cold_entries(
|
def build_cold_entries(
|
||||||
questions: list[GeneratedQuestion],
|
units: list[QuestionUnit],
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
probe_quota: float,
|
probe_quota: float,
|
||||||
seed: int,
|
seed: int,
|
||||||
) -> list[LadderEntry]:
|
) -> list[LadderEntry]:
|
||||||
"""冷启动排序:错题高优先 2:1 交错 + 全错题 probe_quota 探针插尾。
|
"""冷启动排序(unit 粒度):错 unit 高优先 2:1 交错 + 全错 unit 探针插尾。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
questions: 该题型的全部候选题(已排除 test 池)。
|
units: 该题型的全部候选单元(已排除 test 池;AR pair 已折叠成单元)。
|
||||||
correctness: question_id -> 种子基线是否答对(900 题全量对错)。
|
correctness: question_id -> 种子基线是否答对(900 题全量逐题对错)。
|
||||||
probe_quota: 从错题中随机抽出插到梯尾的探针比例(防"解锁新能力"盲区)。
|
单元级对错由 unit_correctness(strict=False) 折叠(任一成员错 → 单元错)。
|
||||||
|
probe_quota: 从错 unit 中随机抽出插到梯尾的探针比例(防"解锁新能力"盲区)。
|
||||||
seed: 洗牌种子,保证确定性重建。
|
seed: 洗牌种子,保证确定性重建。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
排序后的 LadderEntry 列表(p_hat 用 Beta(1,1) 平滑:错=1/3、对=2/3,
|
排序后的 LadderEntry 列表(键=unit_id;p_hat 用 Beta(1,1) 平滑:错=1/3、
|
||||||
与 warm 阶段 gamma-EMA / 信息量排序自然衔接)。
|
对=2/3,与 warm 阶段 gamma-EMA / 信息量排序自然衔接)。
|
||||||
|
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
错题、对题各自固定种子洗牌 -> 抽探针 -> 剩余按 错错对 2:1 交错
|
与逐题版**同公式、同比例、同顺序**,仅把调度粒度从题换成单元:错 unit、
|
||||||
(一方耗尽后顺排另一方)-> 探针追加尾部。
|
对 unit 各自固定种子洗牌 -> 按 probe_quota 从错 unit 抽探针 -> 剩余按
|
||||||
|
错错对 2:1 交错(一方耗尽后顺排另一方)-> 探针追加尾部。
|
||||||
"""
|
"""
|
||||||
rng = random.Random(seed)
|
rng = random.Random(seed)
|
||||||
wrong = [q for q in questions if not correctness.get(q.question_id, False)]
|
wrong = [u for u in units if not unit_correctness(u, correctness, strict=False)]
|
||||||
right = [q for q in questions if correctness.get(q.question_id, False)]
|
right = [u for u in units if unit_correctness(u, correctness, strict=False)]
|
||||||
rng.shuffle(wrong)
|
rng.shuffle(wrong)
|
||||||
rng.shuffle(right)
|
rng.shuffle(right)
|
||||||
|
|
||||||
n_probe = int(len(wrong) * probe_quota)
|
n_probe = int(len(wrong) * probe_quota)
|
||||||
probes, wrong_main = wrong[:n_probe], wrong[n_probe:]
|
probes, wrong_main = wrong[:n_probe], wrong[n_probe:]
|
||||||
|
|
||||||
interleaved: list[GeneratedQuestion] = []
|
interleaved: list[QuestionUnit] = []
|
||||||
wi, ri = 0, 0
|
wi, ri = 0, 0
|
||||||
while wi < len(wrong_main) or ri < len(right):
|
while wi < len(wrong_main) or ri < len(right):
|
||||||
for _ in range(2):
|
for _ in range(2):
|
||||||
@@ -97,10 +110,10 @@ def build_cold_entries(
|
|||||||
ri += 1
|
ri += 1
|
||||||
interleaved.extend(probes)
|
interleaved.extend(probes)
|
||||||
|
|
||||||
def _p0(q: GeneratedQuestion) -> float:
|
def _p0(u: QuestionUnit) -> float:
|
||||||
return 2 / 3 if correctness.get(q.question_id, False) else 1 / 3
|
return 2 / 3 if unit_correctness(u, correctness, strict=False) else 1 / 3
|
||||||
|
|
||||||
return [LadderEntry(q.question_id, _p0(q)) for q in interleaved]
|
return [LadderEntry(u.unit_id, _p0(u)) for u in interleaved]
|
||||||
|
|
||||||
|
|
||||||
def order_ladder(entries: list[LadderEntry], p_low: float, p_high: float) -> list[LadderEntry]:
|
def order_ladder(entries: list[LadderEntry], p_low: float, p_high: float) -> list[LadderEntry]:
|
||||||
@@ -135,23 +148,24 @@ class GatePools:
|
|||||||
def ladder_for(
|
def ladder_for(
|
||||||
self,
|
self,
|
||||||
task_type: str,
|
task_type: str,
|
||||||
exclude_qids: set[str],
|
exclude_units: set[str],
|
||||||
p_low: float,
|
p_low: float,
|
||||||
p_high: float,
|
p_high: float,
|
||||||
cold: bool,
|
cold: bool,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
"""取该题型的 gate 出题序(qid 列表),排除本 step 进化案例包题。
|
"""取该题型的 gate 出题序(unit_id 列表),排除本 step 进化案例包所在单元。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
task_type: 目标题型。
|
task_type: 目标题型。
|
||||||
exclude_qids: 本 step 案例包(failure/success cases)的题目 id,
|
exclude_units: 本 step 案例包(failure/success cases)所在单元的
|
||||||
防止在"刚学的那道题"上自测。
|
unit_id,防止在"刚学的那道题"上自测。按 **unit** 排除:命中单元
|
||||||
|
整体剔除,避免只排 AR pair 半个成员而向 gate 池灌入半个 pair。
|
||||||
p_low / p_high: warm 阶段的 p_hat 保留区间。
|
p_low / p_high: warm 阶段的 p_hat 保留区间。
|
||||||
cold: True 表示尚无 epoch 级观测(epoch 1),用冷启动存储序;
|
cold: True 表示尚无 epoch 级观测(epoch 1),用冷启动存储序;
|
||||||
False 走 order_ladder 信息量排序。
|
False 走 order_ladder 信息量排序。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
排除后的有序 question_id 列表。
|
排除后的有序 unit_id 列表。
|
||||||
|
|
||||||
异常:
|
异常:
|
||||||
ValueError: 该题型无阶梯(冷启动构建缺失),或该题型阶梯为空。
|
ValueError: 该题型无阶梯(冷启动构建缺失),或该题型阶梯为空。
|
||||||
@@ -162,33 +176,54 @@ class GatePools:
|
|||||||
if not pool:
|
if not pool:
|
||||||
raise ValueError(f"task_type={task_type} 阶梯为空,无可出题目")
|
raise ValueError(f"task_type={task_type} 阶梯为空,无可出题目")
|
||||||
ordered = pool if cold else order_ladder(pool, p_low, p_high)
|
ordered = pool if cold else order_ladder(pool, p_low, p_high)
|
||||||
return [e.question_id for e in ordered if e.question_id not in exclude_qids]
|
return [e.unit_id for e in ordered if e.unit_id not in exclude_units]
|
||||||
|
|
||||||
def update_probs(self, observations: dict[str, bool], gamma: float) -> None:
|
def update_probs(
|
||||||
"""gamma-EMA 更新 p_hat:p_hat <- gamma * p_hat + (1-gamma) * obs。只更新有新观测的题。
|
self,
|
||||||
|
per_q_observations: dict[str, bool],
|
||||||
|
units_by_id: dict[str, QuestionUnit],
|
||||||
|
gamma: float,
|
||||||
|
) -> None:
|
||||||
|
"""gamma-EMA 更新 p_hat:先把逐题观测折叠成单元观测,再按 unit_id 匹配更新。
|
||||||
|
|
||||||
|
p_hat <- gamma * p_hat + (1-gamma) * unit_obs。只更新"整个单元都被观测到"
|
||||||
|
的单元;单元观测 = 成员逐题对错的 AND(任一成员错 → 单元错)。折叠是必需的:
|
||||||
|
AR pair 的 unit_id 是 pair_id,若直接按 unit_id 去逐题观测里匹配将永不命中、
|
||||||
|
导致 gamma-EMA 停摆(核心算法保真 #5)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
observations: question_id -> 本 epoch 非 gate run 的最新对错。
|
per_q_observations: question_id -> 本 epoch 非 gate run 的最新逐题对错。
|
||||||
调用方必须已按 run_id 过滤掉 gate 内 rollout(防泄露铁律)。
|
调用方必须已按 run_id 过滤掉 gate 内 rollout(防泄露铁律)。
|
||||||
|
units_by_id: unit_id -> QuestionUnit,用于把逐题观测折叠成单元观测。
|
||||||
gamma: EMA 衰减系数。
|
gamma: EMA 衰减系数。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
单元只有在其**全部**成员都出现在 per_q_observations 时才更新;半观测
|
||||||
|
(AR pair 只见一半)跳过,避免用不完整证据污染 p_hat。
|
||||||
"""
|
"""
|
||||||
for entries in self.entries.values():
|
for entries in self.entries.values():
|
||||||
for e in entries:
|
for e in entries:
|
||||||
if e.question_id in observations:
|
unit = units_by_id.get(e.unit_id)
|
||||||
obs = 1.0 if observations[e.question_id] else 0.0
|
if unit is None:
|
||||||
|
continue
|
||||||
|
if not all(q.question_id in per_q_observations for q in unit.questions):
|
||||||
|
continue
|
||||||
|
unit_correct = all(per_q_observations[q.question_id] for q in unit.questions)
|
||||||
|
obs = 1.0 if unit_correct else 0.0
|
||||||
e.p_hat = gamma * e.p_hat + (1 - gamma) * obs
|
e.p_hat = gamma * e.p_hat + (1 - gamma) * obs
|
||||||
|
|
||||||
def save(self, path: Path) -> None:
|
def save(self, path: Path) -> None:
|
||||||
"""原子写 gate_pools.json(.tmp 再 replace)。
|
"""原子写 gate_pools.json(.tmp 再 replace),落 schema_version + unit_id 键。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
path: 目标 JSON 路径。
|
path: 目标 JSON 路径。
|
||||||
"""
|
"""
|
||||||
payload = {
|
payload = {
|
||||||
|
"schema_version": SCHEMA_VERSION,
|
||||||
"seed": self.seed,
|
"seed": self.seed,
|
||||||
"fingerprint": self.fingerprint,
|
"fingerprint": self.fingerprint,
|
||||||
"entries": {
|
"entries": {
|
||||||
t: [{"question_id": e.question_id, "p_hat": e.p_hat} for e in es]
|
t: [{"unit_id": e.unit_id, "p_hat": e.p_hat} for e in es]
|
||||||
for t, es in self.entries.items()
|
for t, es in self.entries.items()
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -198,18 +233,28 @@ class GatePools:
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def load(cls, path: Path) -> GatePools:
|
def load(cls, path: Path) -> GatePools:
|
||||||
"""从 gate_pools.json 恢复。
|
"""从 gate_pools.json 恢复;schema_version 不匹配直接报错(不静默混用)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
path: gate_pools.json 路径。
|
path: gate_pools.json 路径。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
复活的 GatePools。
|
复活的 GatePools。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
RuntimeError: 缺 schema_version(存量 v1、qid 键)或版本不等于
|
||||||
|
SCHEMA_VERSION——拒绝把 qid 键当 unit 键静默复用,须 FRESH 重建。
|
||||||
"""
|
"""
|
||||||
d = json.loads(path.read_text(encoding="utf-8"))
|
d = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
version = d.get("schema_version")
|
||||||
|
if version != SCHEMA_VERSION:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"gate_pools.json schema_version={version!r} 与当前 {SCHEMA_VERSION} 不符"
|
||||||
|
f"(存量 qid 键池不可当 unit 键复用),请删除后 FRESH 重建: {path}"
|
||||||
|
)
|
||||||
return cls(
|
return cls(
|
||||||
entries={
|
entries={
|
||||||
t: [LadderEntry(x["question_id"], x["p_hat"]) for x in es]
|
t: [LadderEntry(x["unit_id"], x["p_hat"]) for x in es]
|
||||||
for t, es in d["entries"].items()
|
for t, es in d["entries"].items()
|
||||||
},
|
},
|
||||||
seed=d["seed"],
|
seed=d["seed"],
|
||||||
@@ -264,22 +309,46 @@ def build_or_load_gate_pools(
|
|||||||
|
|
||||||
entries: dict[str, list[LadderEntry]] = {}
|
entries: dict[str, list[LadderEntry]] = {}
|
||||||
for t in task_types:
|
for t in task_types:
|
||||||
pool = [q for q in questions if q.task_type == t and q.question_id not in test_qids]
|
units = _task_units_excluding_test(questions, t, test_qids)
|
||||||
if not pool:
|
if not units:
|
||||||
raise ValueError(f"task_type={t} 无非 test 题,无法建阶梯")
|
raise ValueError(f"task_type={t} 无非 test 单元,无法建阶梯")
|
||||||
entries[t] = build_cold_entries(pool, baseline_correctness, probe_quota, seed)
|
entries[t] = build_cold_entries(units, baseline_correctness, probe_quota, seed)
|
||||||
logger.info("gate 阶梯[{}]: {} 题(冷启动)", t, len(entries[t]))
|
logger.info("gate 阶梯[{}]: {} 单元(冷启动)", t, len(entries[t]))
|
||||||
pools = GatePools(entries=entries, seed=seed, fingerprint=fingerprint)
|
pools = GatePools(entries=entries, seed=seed, fingerprint=fingerprint)
|
||||||
pools.save(path)
|
pools.save(path)
|
||||||
return pools
|
return pools
|
||||||
|
|
||||||
|
|
||||||
class BaselineCache:
|
def _task_units_excluding_test(
|
||||||
"""基线侧逐题对错缓存(内容寻址,JSON 持久化)。
|
questions: list[GeneratedQuestion], task_type: str, test_qids: set[str]
|
||||||
|
) -> list[QuestionUnit]:
|
||||||
|
"""取某题型的非 test 候选单元:先按 unit 折叠,再整体排除含 test 成员的单元。
|
||||||
|
|
||||||
键 = (task_type, skill_hash, prompts_version, qid):任何影响该题型
|
先折叠后排除保证 AR pair 不被拆半(否则半个 pair 交给下游会触发 build_units 的
|
||||||
|
孤儿 fail-fast);single 单元等价于逐题排除(核心算法保真 #5)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: benchmark 全量题。
|
||||||
|
task_type: 目标题型。
|
||||||
|
test_qids: held-out test 池题目 id。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
该题型下不含任何 test 成员的候选单元列表。
|
||||||
|
"""
|
||||||
|
pool = [q for q in questions if q.task_type == task_type]
|
||||||
|
return [
|
||||||
|
u for u in build_units(pool) if all(q.question_id not in test_qids for q in u.questions)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class BaselineCache:
|
||||||
|
"""基线侧单元级对错缓存(内容寻址,JSON 持久化)。
|
||||||
|
|
||||||
|
键 = (task_type, skill_hash, prompts_version, unit_id):任何影响该题型
|
||||||
有效 skill 的变化(含共享 default-strategy.md 被他类 accept 改写)
|
有效 skill 的变化(含共享 default-strategy.md 被他类 accept 改写)
|
||||||
都使 skill_hash 变化、缓存自然 miss;prompts 版本变化同理。
|
都使 skill_hash 变化、缓存自然 miss;prompts 版本变化同理。unit_id 维度
|
||||||
|
使 single 题以自身 question_id、AR pair 以共享 pair_id 寻址,缓存单元级
|
||||||
|
对错(pair 双向 AND 折叠后一个布尔)。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, path: Path) -> None:
|
def __init__(self, path: Path) -> None:
|
||||||
@@ -294,32 +363,32 @@ class BaselineCache:
|
|||||||
self._store = json.loads(path.read_text(encoding="utf-8"))
|
self._store = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _key(task_type: str, s_hash: str, prompts_version: str, qid: str) -> str:
|
def _key(task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> str:
|
||||||
"""拼缓存键(四维内容寻址)。"""
|
"""拼缓存键(四维内容寻址,第四维为 unit_id)。"""
|
||||||
return f"{task_type}|{s_hash}|{prompts_version}|{qid}"
|
return f"{task_type}|{s_hash}|{prompts_version}|{unit_id}"
|
||||||
|
|
||||||
def get(self, task_type: str, s_hash: str, prompts_version: str, qid: str) -> bool | None:
|
def get(self, task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> bool | None:
|
||||||
"""读缓存;未命中返回 None。
|
"""读缓存;未命中返回 None。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
task_type: 题型。
|
task_type: 题型。
|
||||||
s_hash: 基线侧生效 skill 文件的内容哈希。
|
s_hash: 基线侧生效 skill 文件的内容哈希。
|
||||||
prompts_version: 当前 prompts 版本。
|
prompts_version: 当前 prompts 版本。
|
||||||
qid: 题目 id。
|
unit_id: 单元 id(single=question_id,AR pair=pair_id)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
缓存的对错;未命中 None。
|
缓存的单元级对错;未命中 None。
|
||||||
"""
|
"""
|
||||||
return self._store.get(self._key(task_type, s_hash, prompts_version, qid))
|
return self._store.get(self._key(task_type, s_hash, prompts_version, unit_id))
|
||||||
|
|
||||||
def put(
|
def put(
|
||||||
self, task_type: str, s_hash: str, prompts_version: str, qid: str, correct: bool
|
self, task_type: str, s_hash: str, prompts_version: str, unit_id: str, correct: bool
|
||||||
) -> None:
|
) -> None:
|
||||||
"""写缓存并落盘(原子写,gate 频度低、全量重写成本可忽略)。
|
"""写缓存并落盘(原子写,gate 频度低、全量重写成本可忽略)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
task_type / s_hash / prompts_version / qid: 缓存键四维。
|
task_type / s_hash / prompts_version / unit_id: 缓存键四维。
|
||||||
correct: 基线侧该题对错。
|
correct: 基线侧该单元对错(AR pair 双向 AND 折叠后一个布尔)。
|
||||||
|
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
先盘后存:新条目先原子落盘(tmp 写 + os.replace)成功后才更新
|
先盘后存:新条目先原子落盘(tmp 写 + os.replace)成功后才更新
|
||||||
@@ -327,7 +396,7 @@ class BaselineCache:
|
|||||||
"""
|
"""
|
||||||
updated = {
|
updated = {
|
||||||
**self._store,
|
**self._store,
|
||||||
self._key(task_type, s_hash, prompts_version, qid): correct,
|
self._key(task_type, s_hash, prompts_version, unit_id): correct,
|
||||||
}
|
}
|
||||||
tmp = self._path.with_suffix(".json.tmp")
|
tmp = self._path.with_suffix(".json.tmp")
|
||||||
tmp.write_text(json.dumps(updated, ensure_ascii=False), encoding="utf-8")
|
tmp.write_text(json.dumps(updated, ensure_ascii=False), encoding="utf-8")
|
||||||
|
|||||||
+184
-35
@@ -14,12 +14,14 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
|
import sqlite3
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.question_units import build_units, unit_correctness
|
||||||
from core.agent.loop import AgentLoop
|
from core.agent.loop import AgentLoop
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
@@ -28,22 +30,22 @@ if TYPE_CHECKING:
|
|||||||
from app.harness.log import HarnessLog
|
from app.harness.log import HarnessLog
|
||||||
from core.agent.types import LoopResult
|
from core.agent.types import LoopResult
|
||||||
from core.protocols import LLMProvider
|
from core.protocols import LLMProvider
|
||||||
from core.types import GeneratedQuestion
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class InferenceResult:
|
class InferenceResult:
|
||||||
"""推理聚合结果。
|
"""推理聚合结果(正确率按 unit 粒度)。
|
||||||
|
|
||||||
属性:
|
属性:
|
||||||
run_id: 运行标识。
|
run_id: 运行标识。
|
||||||
accuracy: 总正确率。
|
accuracy: unit 级正确率(correct / total)。
|
||||||
total: 总题数。
|
total: unit 总数(single 数 + pair 数,孤儿 pair 已剔除不计入)。
|
||||||
correct: 正确题数。
|
correct: 正确 unit 数(single 单题正确;pair 走 original/mirror 双向 AND)。
|
||||||
per_task_type: 按题型分组的指标 {task_type: {accuracy, total, correct}}。
|
per_task_type: 按题型分组的 unit 级指标 {task_type: {accuracy, total, correct}}。
|
||||||
steps_mean: 平均步数。
|
steps_mean: 平均步数(record 粒度,逐题溯源)。
|
||||||
token_usage: token 总用量 {prompt_tokens, completion_tokens}。
|
token_usage: token 总用量 {prompt_tokens, completion_tokens}(record 粒度)。
|
||||||
stop_reason_counts: 终止原因计数 {reason: count}。
|
stop_reason_counts: 终止原因计数 {reason: count}(record 粒度)。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
run_id: str
|
run_id: str
|
||||||
@@ -161,6 +163,24 @@ def _to_text_field(value: Any) -> str:
|
|||||||
return json.dumps(value, ensure_ascii=False)
|
return json.dumps(value, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_prediction(answer: object) -> str | None:
|
||||||
|
"""归一化 prediction 落库值。
|
||||||
|
|
||||||
|
LLM 提交的 answer 有时是 list/dict(如 {'answer': ['B']}),sqlite 无法绑定
|
||||||
|
非标量类型直接入库会抛 ProgrammingError 击穿整轮 gather。None 保留(INFRA 空
|
||||||
|
预测语义,供正确率判定天然计错);str 原样;其余 JSON 序列化为文本。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
answer: LoopResult.result 中的 answer 原始值(可能是 None/str/list/dict)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
None(保留空预测语义)或可直接入库的字符串。
|
||||||
|
"""
|
||||||
|
if answer is None or isinstance(answer, str):
|
||||||
|
return answer
|
||||||
|
return _to_text_field(answer)
|
||||||
|
|
||||||
|
|
||||||
def _zero_result(run_id: str) -> InferenceResult:
|
def _zero_result(run_id: str) -> InferenceResult:
|
||||||
"""空记录时的零值 InferenceResult。
|
"""空记录时的零值 InferenceResult。
|
||||||
|
|
||||||
@@ -182,23 +202,25 @@ def _zero_result(run_id: str) -> InferenceResult:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _group_by_task_type(records: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
def _group_by_task_type(graded: list[tuple[QuestionUnit, bool]]) -> dict[str, dict[str, Any]]:
|
||||||
"""按 task_type 分组聚合正确率指标。
|
"""按 task_type 分组聚合 unit 级正确率指标。
|
||||||
|
|
||||||
|
pair 单元整体计 1 个 unit,归入其 task_type;single 单元计 1 个 unit。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
records: 预测记录列表。
|
graded: (单元, 该单元是否整体正确) 元组列表。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
{task_type: {accuracy, total, correct}} 映射。
|
{task_type: {accuracy, total, correct}} 映射(unit 粒度)。
|
||||||
"""
|
"""
|
||||||
task_groups: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
task_groups: dict[str, list[bool]] = defaultdict(list)
|
||||||
for r in records:
|
for unit, is_correct in graded:
|
||||||
task_groups[r["task_type"]].append(r)
|
task_groups[unit.task_type].append(is_correct)
|
||||||
|
|
||||||
per_task_type: dict[str, dict[str, Any]] = {}
|
per_task_type: dict[str, dict[str, Any]] = {}
|
||||||
for task_type, group in task_groups.items():
|
for task_type, verdicts in task_groups.items():
|
||||||
t_total = len(group)
|
t_total = len(verdicts)
|
||||||
t_correct = sum(1 for r in group if r["prediction"] == r["answer"])
|
t_correct = sum(verdicts)
|
||||||
per_task_type[task_type] = {
|
per_task_type[task_type] = {
|
||||||
"accuracy": t_correct / t_total,
|
"accuracy": t_correct / t_total,
|
||||||
"total": t_total,
|
"total": t_total,
|
||||||
@@ -207,35 +229,143 @@ def _group_by_task_type(records: list[dict[str, Any]]) -> dict[str, dict[str, An
|
|||||||
return per_task_type
|
return per_task_type
|
||||||
|
|
||||||
|
|
||||||
def _aggregate_results(records: list[dict[str, Any]], run_id: str) -> InferenceResult:
|
def _is_valid_pair(group: list[GeneratedQuestion]) -> bool:
|
||||||
"""从内存 records 聚合推理指标。
|
"""判定同一 pair_id 分组是否为合法孪生对(恰好 1 original + 1 mirror,无多余)。
|
||||||
|
|
||||||
TRM4 从 DB 回读 predictions 表聚合;TRM5 改为从内存直接聚合,
|
要求分组总数恰为 2 且角色齐备唯一;有额外非法 role 记录(total>2)或角色
|
||||||
避免 DB 回读的同步开销和额外依赖。
|
缺失/重复均视为非法,交由调用方剔除,防非法记录混入 build_units。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
records: _run_single_question 返回的 record 列表。
|
group: 归属同一 pair_id 的题目列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
合法孪生对为 True,否则 False。
|
||||||
|
"""
|
||||||
|
if len(group) != 2:
|
||||||
|
return False
|
||||||
|
originals = sum(1 for q in group if q.question_role == "pair_original")
|
||||||
|
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
|
||||||
|
return originals == 1 and mirrors == 1
|
||||||
|
|
||||||
|
|
||||||
|
def _drop_orphan_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||||
|
"""剔除收不齐 2 条 / 角色非法的孤儿 pair,告警不静默。
|
||||||
|
|
||||||
|
每条题目均会各答一次并逐题落库;能否合成 pair 单元仅取决于 questions
|
||||||
|
是否同时含该 pair_id 的 original + mirror(且无多余非法记录)。非法者告警并
|
||||||
|
整对剔除,使后续 build_units 只面对合法孪生对(不触发 fail-fast),孤儿 unit
|
||||||
|
不计入 total(对齐设计 §8 聚合入口的"告警 + 剔除")。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: 待聚合的题目列表(可混含 single 与孪生对成员)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
可安全交给 build_units 的题目列表(single 全保留,pair 仅保留合法成对者)。
|
||||||
|
"""
|
||||||
|
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||||
|
singles: list[GeneratedQuestion] = []
|
||||||
|
for q in questions:
|
||||||
|
if q.pair_id:
|
||||||
|
by_pair[q.pair_id].append(q)
|
||||||
|
else:
|
||||||
|
singles.append(q)
|
||||||
|
|
||||||
|
kept_pairs: list[GeneratedQuestion] = []
|
||||||
|
for pair_id, group in by_pair.items():
|
||||||
|
if _is_valid_pair(group):
|
||||||
|
kept_pairs.extend(group)
|
||||||
|
else:
|
||||||
|
logger.warning(
|
||||||
|
"孤儿 pair {}:非法配对(total={}),剔除该 unit 不计入 total",
|
||||||
|
pair_id,
|
||||||
|
len(group),
|
||||||
|
)
|
||||||
|
return singles + kept_pairs
|
||||||
|
|
||||||
|
|
||||||
|
def _per_question_correctness(records: list[dict[str, Any]]) -> dict[str, bool]:
|
||||||
|
"""由逐题 record 构造 question_id → 该题作答是否正确 的映射。
|
||||||
|
|
||||||
|
prediction 为 None(作答异常)时与 answer 不相等 → False,天然计错。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: _run_single_question 返回的逐题 record 列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{question_id: prediction == answer} 映射,供 unit_correctness 取值。
|
||||||
|
"""
|
||||||
|
return {r["question_id"]: r["prediction"] == r["answer"] for r in records}
|
||||||
|
|
||||||
|
|
||||||
|
def _grade_unit(unit: QuestionUnit, per_q: dict[str, bool]) -> bool:
|
||||||
|
"""判定单元整体正确性,缺 prediction 时 fail-loud(带上下文)。
|
||||||
|
|
||||||
|
_drop_orphan_pairs 已剔除孤儿/非法配对,正常情况下 unit 内每题都应有对应
|
||||||
|
record;若仍缺失说明聚合不变量被破坏(如 records 与 questions 不同源)。此处
|
||||||
|
显式抛带上下文的 ValueError(fail-loud,不 catch/不跳过/不兜底),而非放任
|
||||||
|
unit_correctness 抛裸 KeyError 丢失定位信息。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
unit: 待判定单元。
|
||||||
|
per_q: question_id → 该题是否作答正确 的映射。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
单元整体是否正确(single 即单题正确;pair 走双向 AND)。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: unit 内某 question_id 不在 per_q 中(聚合不变量被破坏)。
|
||||||
|
"""
|
||||||
|
missing = [q.question_id for q in unit.questions if q.question_id not in per_q]
|
||||||
|
if missing:
|
||||||
|
raise ValueError(
|
||||||
|
f"unit {unit.unit_id} 的 question {missing} 缺 prediction"
|
||||||
|
"(_drop_orphan_pairs 后不应发生,聚合不变量被破坏)"
|
||||||
|
)
|
||||||
|
return unit_correctness(unit, per_q)
|
||||||
|
|
||||||
|
|
||||||
|
def _aggregate_results(
|
||||||
|
records: list[dict[str, Any]],
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
run_id: str,
|
||||||
|
) -> InferenceResult:
|
||||||
|
"""从内存 records + 题目列表按 unit 粒度聚合推理指标。
|
||||||
|
|
||||||
|
逐题 record 保留逐题溯源(token/steps/stop_reason 诊断仍按 record 汇总);
|
||||||
|
正确率则按 unit 粒度计:single 计 1,AR pair 经 build_units 收齐 original +
|
||||||
|
mirror 后走 unit_correctness 的双向 AND 判定,整对计 1 个 unit。孤儿 pair
|
||||||
|
在 _drop_orphan_pairs 中告警 + 剔除,不计入 total。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: _run_single_question 返回的逐题 record 列表。
|
||||||
|
questions: 与 records 对应的题目列表(提供 pair_id/question_role 元数据)。
|
||||||
run_id: 当前运行标识。
|
run_id: 当前运行标识。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
InferenceResult 冻结实例。
|
InferenceResult 冻结实例(total/correct/per_task_type 为 unit 粒度)。
|
||||||
"""
|
"""
|
||||||
total = len(records)
|
if not records:
|
||||||
if total == 0:
|
|
||||||
return _zero_result(run_id)
|
return _zero_result(run_id)
|
||||||
|
|
||||||
correct = sum(1 for r in records if r["prediction"] == r["answer"])
|
per_q = _per_question_correctness(records)
|
||||||
|
units = build_units(_drop_orphan_pairs(questions))
|
||||||
|
graded = [(unit, _grade_unit(unit, per_q)) for unit in units]
|
||||||
|
|
||||||
|
total = len(graded)
|
||||||
|
correct = sum(1 for _, is_correct in graded if is_correct)
|
||||||
|
|
||||||
stop_counts: dict[str, int] = defaultdict(int)
|
stop_counts: dict[str, int] = defaultdict(int)
|
||||||
for r in records:
|
for r in records:
|
||||||
stop_counts[r["stop_reason"]] += 1
|
stop_counts[r["stop_reason"]] += 1
|
||||||
|
|
||||||
|
n_records = len(records)
|
||||||
return InferenceResult(
|
return InferenceResult(
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
accuracy=correct / total,
|
accuracy=correct / total if total else 0.0,
|
||||||
total=total,
|
total=total,
|
||||||
correct=correct,
|
correct=correct,
|
||||||
per_task_type=_group_by_task_type(records),
|
per_task_type=_group_by_task_type(graded),
|
||||||
steps_mean=sum(r["steps_used"] for r in records) / total,
|
steps_mean=sum(r["steps_used"] for r in records) / n_records,
|
||||||
token_usage={
|
token_usage={
|
||||||
"prompt_tokens": sum(r["prompt_tokens"] for r in records),
|
"prompt_tokens": sum(r["prompt_tokens"] for r in records),
|
||||||
"completion_tokens": sum(r["completion_tokens"] for r in records),
|
"completion_tokens": sum(r["completion_tokens"] for r in records),
|
||||||
@@ -258,6 +388,7 @@ async def _run_single_question(
|
|||||||
log: HarnessLog,
|
log: HarnessLog,
|
||||||
max_steps: int,
|
max_steps: int,
|
||||||
plugins: list[object],
|
plugins: list[object],
|
||||||
|
run_id: str,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""执行单道题目的 Agent 推理。
|
"""执行单道题目的 Agent 推理。
|
||||||
|
|
||||||
@@ -272,11 +403,17 @@ async def _run_single_question(
|
|||||||
log: HarnessLog 实例(线程安全)。
|
log: HarnessLog 实例(线程安全)。
|
||||||
max_steps: AgentLoop 最大步数。
|
max_steps: AgentLoop 最大步数。
|
||||||
plugins: pluggy 插件列表。
|
plugins: pluggy 插件列表。
|
||||||
|
run_id: 运行标识,用作 cache_salt——run_id 含 _e{epoch} 天然跨 epoch 重采样、
|
||||||
|
同 epoch 续跑命中缓存(算法 #10 透传)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
预测结果字典(含 video_id, question_id, prediction, answer 等)。
|
预测结果字典(含 video_id, question_id, prediction, answer 等)。
|
||||||
"""
|
"""
|
||||||
|
# run_id 必须显式入 record:HarnessLog.insert 缺省用**实例** run_id 填充,
|
||||||
|
# 连续并发 gate 共享单一 gate_log(实例 run_id 为 step 级)时,各臂行必须
|
||||||
|
# 落自己的臂 run_id,否则 validate 回读 _load_run_rows(臂 run_id) 为空。
|
||||||
record: dict[str, Any] = {
|
record: dict[str, Any] = {
|
||||||
|
"run_id": run_id,
|
||||||
"video_id": qa.video_id,
|
"video_id": qa.video_id,
|
||||||
"question_id": qa.question_id,
|
"question_id": qa.question_id,
|
||||||
"task_type": qa.task_type,
|
"task_type": qa.task_type,
|
||||||
@@ -301,6 +438,7 @@ async def _run_single_question(
|
|||||||
dispatcher,
|
dispatcher,
|
||||||
plugins=plugins,
|
plugins=plugins,
|
||||||
session_id=qa.question_id,
|
session_id=qa.question_id,
|
||||||
|
cache_salt=run_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
result_dict = loop_result.result if isinstance(loop_result.result, dict) else {}
|
result_dict = loop_result.result if isinstance(loop_result.result, dict) else {}
|
||||||
@@ -308,7 +446,7 @@ async def _run_single_question(
|
|||||||
reasoning = _to_text_field(result_dict.get("reasoning", ""))
|
reasoning = _to_text_field(result_dict.get("reasoning", ""))
|
||||||
record.update(
|
record.update(
|
||||||
{
|
{
|
||||||
"prediction": result_dict.get("answer"),
|
"prediction": _normalize_prediction(result_dict.get("answer")),
|
||||||
"evidence": evidence,
|
"evidence": evidence,
|
||||||
"reasoning": reasoning,
|
"reasoning": reasoning,
|
||||||
"steps_used": loop_result.steps_used,
|
"steps_used": loop_result.steps_used,
|
||||||
@@ -331,8 +469,18 @@ async def _run_single_question(
|
|||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("[{}] QA {} 执行异常", qa.video_id, qa.question_id)
|
logger.exception("[{}] QA {} 执行异常", qa.video_id, qa.question_id)
|
||||||
|
|
||||||
# prediction 必落库(try 外,无论成败)
|
# prediction 必落库(try 外,无论成败);绑定异常降级为最小 error 行,不击穿 gather
|
||||||
|
try:
|
||||||
await asyncio.to_thread(log.insert, "predictions", record)
|
await asyncio.to_thread(log.insert, "predictions", record)
|
||||||
|
except (sqlite3.InterfaceError, sqlite3.ProgrammingError):
|
||||||
|
logger.exception("[{}] QA {} 落库绑定异常,降级为 error 行", qa.video_id, qa.question_id)
|
||||||
|
record["prediction"] = None
|
||||||
|
record["stop_reason"] = "error"
|
||||||
|
await asyncio.to_thread(
|
||||||
|
log.insert,
|
||||||
|
"predictions",
|
||||||
|
{k: v for k, v in record.items() if isinstance(v, (str, int, float, type(None)))},
|
||||||
|
)
|
||||||
return record
|
return record
|
||||||
|
|
||||||
|
|
||||||
@@ -399,7 +547,7 @@ async def run_inference(
|
|||||||
|
|
||||||
if not questions:
|
if not questions:
|
||||||
logger.info("题目列表为空,返回零值 InferenceResult")
|
logger.info("题目列表为空,返回零值 InferenceResult")
|
||||||
return _aggregate_results([], run_id)
|
return _aggregate_results([], [], run_id)
|
||||||
|
|
||||||
sem = asyncio.Semaphore(concurrency)
|
sem = asyncio.Semaphore(concurrency)
|
||||||
total_count = len(questions)
|
total_count = len(questions)
|
||||||
@@ -418,6 +566,7 @@ async def run_inference(
|
|||||||
log=log,
|
log=log,
|
||||||
max_steps=max_steps,
|
max_steps=max_steps,
|
||||||
plugins=plugins,
|
plugins=plugins,
|
||||||
|
run_id=run_id,
|
||||||
)
|
)
|
||||||
logger.info(
|
logger.info(
|
||||||
"[{}/{}] {} QA {} 完成 (stop={})",
|
"[{}/{}] {} QA {} 完成 (stop={})",
|
||||||
@@ -431,7 +580,7 @@ async def run_inference(
|
|||||||
|
|
||||||
results = await asyncio.gather(*[_bounded(i, qa) for i, qa in enumerate(questions)])
|
results = await asyncio.gather(*[_bounded(i, qa) for i, qa in enumerate(questions)])
|
||||||
|
|
||||||
inference_result = _aggregate_results(list(results), run_id)
|
inference_result = _aggregate_results(list(results), questions, run_id)
|
||||||
logger.info(
|
logger.info(
|
||||||
"推理完成: accuracy={:.2%} ({}/{})",
|
"推理完成: accuracy={:.2%} ({}/{})",
|
||||||
inference_result.accuracy,
|
inference_result.accuracy,
|
||||||
|
|||||||
+17
-1
@@ -52,6 +52,9 @@ class HarnessLog:
|
|||||||
run_id: 本次运行的唯一标识。
|
run_id: 本次运行的唯一标识。
|
||||||
git_sha: 代码版本,默认自动获取。
|
git_sha: 代码版本,默认自动获取。
|
||||||
config_snapshot: 本次运行的配置快照。
|
config_snapshot: 本次运行的配置快照。
|
||||||
|
register_run: 是否注册运行(upsert _runs + 退出时同步 status)。默认 True;
|
||||||
|
只读查询已有 run(如基线预测回读)时传 False,避免把该 run 的
|
||||||
|
started_at/config/status 改写、把基线元数据污染成本次进程的运行状态。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -60,16 +63,24 @@ class HarnessLog:
|
|||||||
run_id: str,
|
run_id: str,
|
||||||
git_sha: str | None = None,
|
git_sha: str | None = None,
|
||||||
config_snapshot: dict[str, Any] | None = None,
|
config_snapshot: dict[str, Any] | None = None,
|
||||||
|
*,
|
||||||
|
register_run: bool = True,
|
||||||
) -> None:
|
) -> None:
|
||||||
self._run_id = run_id
|
self._run_id = run_id
|
||||||
|
self._register_run = register_run
|
||||||
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
# 单持久连接 + 进程内 Lock 串行化写:把并发控制拉到进程内,消除多连接争
|
||||||
|
# SQLite 写锁。同款模式复用于 adapters/telemetry.py:SQLiteTelemetryRecorder。
|
||||||
self._conn = sqlite3.connect(db_path, check_same_thread=False)
|
self._conn = sqlite3.connect(db_path, check_same_thread=False)
|
||||||
self._lock = threading.Lock()
|
self._lock = threading.Lock()
|
||||||
self._conn.row_factory = sqlite3.Row
|
self._conn.row_factory = sqlite3.Row
|
||||||
self._conn.execute("PRAGMA journal_mode=WAL")
|
self._conn.execute("PRAGMA journal_mode=WAL")
|
||||||
self._init_fixed_tables()
|
self._init_fixed_tables()
|
||||||
|
if register_run:
|
||||||
resolved_sha = git_sha or _get_git_sha()
|
resolved_sha = git_sha or _get_git_sha()
|
||||||
config_json = json.dumps(config_snapshot, ensure_ascii=False) if config_snapshot else None
|
config_json = (
|
||||||
|
json.dumps(config_snapshot, ensure_ascii=False) if config_snapshot else None
|
||||||
|
)
|
||||||
self._conn.execute(
|
self._conn.execute(
|
||||||
"INSERT INTO _runs"
|
"INSERT INTO _runs"
|
||||||
" (run_id, git_sha, started_at, config, status)"
|
" (run_id, git_sha, started_at, config, status)"
|
||||||
@@ -217,8 +228,13 @@ class HarnessLog:
|
|||||||
|
|
||||||
参数:
|
参数:
|
||||||
status: 最终状态,"completed" 或 "failed"。
|
status: 最终状态,"completed" 或 "failed"。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
register_run=False(只读打开)时跳过 status 更新,仅关闭连接,
|
||||||
|
避免只读回读把已有 run 的 finished_at/status 改写。
|
||||||
"""
|
"""
|
||||||
with self._lock:
|
with self._lock:
|
||||||
|
if self._register_run:
|
||||||
self._conn.execute(
|
self._conn.execute(
|
||||||
"UPDATE _runs SET finished_at = ?, status = ? WHERE run_id = ?",
|
"UPDATE _runs SET finished_at = ?, status = ? WHERE run_id = ?",
|
||||||
(_now_iso(), status, self._run_id),
|
(_now_iso(), status, self._run_id),
|
||||||
|
|||||||
@@ -87,6 +87,8 @@ _HOLDOUT_COLS: dict[str, str] = {
|
|||||||
_QUADRANT_COLS: dict[str, str] = {
|
_QUADRANT_COLS: dict[str, str] = {
|
||||||
"epoch": "INTEGER",
|
"epoch": "INTEGER",
|
||||||
"step": "INTEGER",
|
"step": "INTEGER",
|
||||||
|
# question_id 列承载 unit_id(single=question_id,pair=pair_id);
|
||||||
|
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||||
"question_id": "TEXT",
|
"question_id": "TEXT",
|
||||||
"task_type": "TEXT",
|
"task_type": "TEXT",
|
||||||
"prev_correct": "INTEGER",
|
"prev_correct": "INTEGER",
|
||||||
@@ -98,8 +100,10 @@ _GATE_EVIDENCE_COLS: dict[str, str] = {
|
|||||||
"epoch": "INTEGER",
|
"epoch": "INTEGER",
|
||||||
"step": "INTEGER",
|
"step": "INTEGER",
|
||||||
"task_type": "TEXT",
|
"task_type": "TEXT",
|
||||||
|
# question_id 列承载 unit_id(single=question_id,pair=pair_id);
|
||||||
|
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||||
"question_id": "TEXT",
|
"question_id": "TEXT",
|
||||||
"block_idx": "INTEGER",
|
"ladder_rank": "INTEGER",
|
||||||
"baseline_correct": "INTEGER",
|
"baseline_correct": "INTEGER",
|
||||||
"candidate_correct": "INTEGER",
|
"candidate_correct": "INTEGER",
|
||||||
"e_value": "REAL",
|
"e_value": "REAL",
|
||||||
@@ -131,7 +135,8 @@ def write_dual_metric(
|
|||||||
db_path: SQLite 路径。
|
db_path: SQLite 路径。
|
||||||
run_id: 训练 run ID。
|
run_id: 训练 run ID。
|
||||||
epoch: 轮次(1-based)。
|
epoch: 轮次(1-based)。
|
||||||
version_kind: baseline / best_hard / best_mixed / final。
|
version_kind: baseline / best_hard / best_mixed / final / slow_candidate
|
||||||
|
(slow_candidate = 慢更新 R2 可能被 revert 的候选,不占 epoch 终值 final 口径)。
|
||||||
skills_version / prompts_version: 评估的资源版本。
|
skills_version / prompts_version: 评估的资源版本。
|
||||||
pool: val / test。
|
pool: val / test。
|
||||||
hard_acc: hard 准确率。
|
hard_acc: hard 准确率。
|
||||||
@@ -275,7 +280,7 @@ def write_quadrant_pairs(
|
|||||||
step: int,
|
step: int,
|
||||||
pairs: list[dict[str, Any]],
|
pairs: list[dict[str, Any]],
|
||||||
) -> None:
|
) -> None:
|
||||||
"""落 quadrant_pair 多行:fast gate 后逐题四象限(prev/curr 翻转 + category)落库。
|
"""落 quadrant_pair 多行:fast gate 后按 **unit** 四象限(prev/curr 翻转 + category)落库。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
db_path: SQLite 路径。
|
db_path: SQLite 路径。
|
||||||
@@ -283,7 +288,10 @@ def write_quadrant_pairs(
|
|||||||
epoch: 轮次(1-based)。
|
epoch: 轮次(1-based)。
|
||||||
step: epoch 内 step 序号(0-based)。
|
step: epoch 内 step 序号(0-based)。
|
||||||
pairs: 每条含 question_id/task_type/prev_correct/curr_correct/category;
|
pairs: 每条含 question_id/task_type/prev_correct/curr_correct/category;
|
||||||
prev_correct/curr_correct 为 bool,写库前转 0/1。
|
question_id 字段承载 **unit_id**(single=question_id,pair=pair_id,
|
||||||
|
与 gate e-process 同粒度)——逐题明细在 predictions 表溯源,按 pair_id
|
||||||
|
join 真实 question 表会 join 不上;prev_correct/curr_correct 为 bool,
|
||||||
|
写库前转 0/1。
|
||||||
|
|
||||||
关键实现:
|
关键实现:
|
||||||
用 insert_many 批量落库;pairs 为空时只建表不插入(fast gate 无翻转的极端情况)。
|
用 insert_many 批量落库;pairs 为空时只建表不插入(fast gate 无翻转的极端情况)。
|
||||||
@@ -326,16 +334,22 @@ def write_gate_evidence(
|
|||||||
step: int,
|
step: int,
|
||||||
rows: list[dict[str, Any]],
|
rows: list[dict[str, Any]],
|
||||||
) -> None:
|
) -> None:
|
||||||
"""落 gate_evidence 逐题行:CE-Gate 每次决策的可回放审计记录。
|
"""落 gate_evidence 单元行:CE-Gate 每次决策的可回放审计记录(unit 口径)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
db_path: SQLite 路径。
|
db_path: SQLite 路径。
|
||||||
run_id: 训练 run ID。
|
run_id: 训练 run ID。
|
||||||
epoch: 该 gate 所属的轮次(1-based)。
|
epoch: 该 gate 所属的轮次(1-based)。
|
||||||
step: epoch 内 step 序号(0-based)。
|
step: epoch 内 step 序号(0-based)。
|
||||||
rows: 每题一行,含 question_id/task_type/block_idx/baseline_correct/
|
rows: 每 **单元** 一行,含 question_id/task_type/ladder_rank(阶梯序号,
|
||||||
candidate_correct/e_value(该题所在块判定后的累计 e 值)/
|
0-based)/baseline_correct/
|
||||||
stop_reason(仅最后一题携带最终 stop_reason,其余空串)。
|
candidate_correct/e_value(该单元判定后的累计 e 值)/
|
||||||
|
stop_reason(仅最后一单元携带最终 stop_reason,其余空串)。
|
||||||
|
question_id 字段承载 **unit_id**(single=question_id,pair=pair_id)——
|
||||||
|
逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
无。
|
||||||
|
|
||||||
关键实现:
|
关键实现:
|
||||||
逐行 insert(非 insert_many),保证每行独立事务。
|
逐行 insert(非 insert_many),保证每行独立事务。
|
||||||
@@ -344,6 +358,12 @@ def write_gate_evidence(
|
|||||||
|
|
||||||
with HarnessLog(db_path, run_id) as log:
|
with HarnessLog(db_path, run_id) as log:
|
||||||
log.create_table("gate_evidence", _GATE_EVIDENCE_COLS)
|
log.create_table("gate_evidence", _GATE_EVIDENCE_COLS)
|
||||||
|
# 幂等迁移(对齐 question_gen/run_store 先例):块序贯时代的旧表只有
|
||||||
|
# block_idx 列,CREATE TABLE IF NOT EXISTS 不补列,直接插 ladder_rank
|
||||||
|
# 会 OperationalError——为旧 workspace 复用补列,新表恒为 no-op。
|
||||||
|
cols = {r["name"] for r in log.query("PRAGMA table_info(gate_evidence)")}
|
||||||
|
if "ladder_rank" not in cols:
|
||||||
|
log.execute("ALTER TABLE gate_evidence ADD COLUMN ladder_rank INTEGER")
|
||||||
for row in rows:
|
for row in rows:
|
||||||
log.insert("gate_evidence", {"epoch": epoch, "step": step, **row})
|
log.insert("gate_evidence", {"epoch": epoch, "step": step, **row})
|
||||||
|
|
||||||
|
|||||||
+403
-58
@@ -2,14 +2,15 @@
|
|||||||
|
|
||||||
三池切分对应训练循环中的 DataLoader 阶段——从题目全集中按
|
三池切分对应训练循环中的 DataLoader 阶段——从题目全集中按
|
||||||
test -> validation -> diagnosis 的顺序 progressive exclusion,
|
test -> validation -> diagnosis 的顺序 progressive exclusion,
|
||||||
保证 question_id 互斥。test 池用自然分布(correct_ratio=None),
|
以 unit 为原子保证 unit_id 互斥(AR 孪生对两题永不被劈到不同池)。
|
||||||
验证池/诊断池按对错比例分层采样。
|
test 池用自然分布(correct_ratio=None),验证池/诊断池按对错比例分层采样。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import math
|
import math
|
||||||
|
import os
|
||||||
import random
|
import random
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
@@ -17,6 +18,7 @@ from typing import TYPE_CHECKING
|
|||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.question_units import build_units, flatten_units, unit_correctness
|
||||||
from app.question_gen import stratified_sample
|
from app.question_gen import stratified_sample
|
||||||
from core.types import GeneratedQuestion, PoolConfig
|
from core.types import GeneratedQuestion, PoolConfig
|
||||||
|
|
||||||
@@ -25,6 +27,7 @@ if TYPE_CHECKING:
|
|||||||
|
|
||||||
from app.harness.config import RunConfig
|
from app.harness.config import RunConfig
|
||||||
from app.ports import PoolStrategy
|
from app.ports import PoolStrategy
|
||||||
|
from core.types import QuestionUnit
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -70,11 +73,15 @@ def build_pools(
|
|||||||
冻结的三池 Pools。
|
冻结的三池 Pools。
|
||||||
|
|
||||||
关键实现细节:
|
关键实现细节:
|
||||||
切分顺序 test -> validation -> diagnosis;后两步从剩余题中采样以保证
|
切分顺序 test -> validation -> diagnosis;后两步从剩余单元中采样以保证
|
||||||
question_id 互斥。test 池用 correct_ratio=None 的自然分布采样。
|
unit_id 互斥。test 池用 correct_ratio=None 的自然分布采样。以 unit 为采样
|
||||||
|
原子(pair 计 1 个 unit),孪生对两题永不被劈到不同池;size/correct_ratio
|
||||||
|
按 unit 计数,single-only 输入下 unit 与 question 一一对应,行为完全不变。
|
||||||
"""
|
"""
|
||||||
|
units = build_units(questions)
|
||||||
|
|
||||||
test = _sample_excluding(
|
test = _sample_excluding(
|
||||||
questions,
|
units,
|
||||||
set(),
|
set(),
|
||||||
correctness,
|
correctness,
|
||||||
size=test_cfg["size"],
|
size=test_cfg["size"],
|
||||||
@@ -83,12 +90,12 @@ def build_pools(
|
|||||||
seed=test_cfg.get("seed", 0),
|
seed=test_cfg.get("seed", 0),
|
||||||
min_per_class=None,
|
min_per_class=None,
|
||||||
)
|
)
|
||||||
selected_ids = {q.question_id for q in test}
|
selected_units = {q.unit_id for q in test}
|
||||||
|
|
||||||
validation = _sample_excluding(questions, selected_ids, correctness, **val_cfg)
|
validation = _sample_excluding(units, selected_units, correctness, **val_cfg)
|
||||||
selected_ids |= {q.question_id for q in validation}
|
selected_units |= {q.unit_id for q in validation}
|
||||||
|
|
||||||
diagnosis = _sample_excluding(questions, selected_ids, correctness, **diag_cfg)
|
diagnosis = _sample_excluding(units, selected_units, correctness, **diag_cfg)
|
||||||
|
|
||||||
val_correct = sum(1 for q in validation if correctness.get(q.question_id))
|
val_correct = sum(1 for q in validation if correctness.get(q.question_id))
|
||||||
baseline_val_accuracy = val_correct / len(validation) if validation else 0.0
|
baseline_val_accuracy = val_correct / len(validation) if validation else 0.0
|
||||||
@@ -105,6 +112,267 @@ def build_pools(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_VIDEO_ASSIGNMENT_LABELS = ("trainval", "test")
|
||||||
|
|
||||||
|
|
||||||
|
class InsufficientValSignal(Exception): # noqa: N818 领域名「验证信号不足」,非通用错误后缀更贴切
|
||||||
|
"""validation 池错题数不足以支撑可靠验证信号(如 McNemar 检验功效)时抛出。
|
||||||
|
|
||||||
|
fail loud(P5):不静默兜底、不放宽阈值,直接暴露 val 池错题数与所需下限,
|
||||||
|
由调用方决定放大 val_ratio / 换 trainval 归属或调低 val_wrong_min。
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def split_by_video_assignment(
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
assignment: dict[str, str],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
val_ratio: float,
|
||||||
|
seed: int,
|
||||||
|
baseline_run_id: str = "",
|
||||||
|
val_wrong_min: int = 0,
|
||||||
|
wrong_tier_by_video: dict[str, int] | None = None,
|
||||||
|
) -> Pools:
|
||||||
|
"""按视频归属做原子切分:同一视频所有题绝不跨 trainval/test 池。
|
||||||
|
|
||||||
|
切分原子从 unit 提升为 **视频组**(同 video 的全部题同进同出),彻底杜绝
|
||||||
|
同视频多题散落不同池造成的内容泄漏。trainval 题集内部再以视频组为原子做
|
||||||
|
correctness 分层,切出 validation(占 val_ratio)与 diagnosis(其余)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: 题目全集。
|
||||||
|
assignment: video_id -> "trainval" | "test" 归属字典(由选择器上游产出)。
|
||||||
|
correctness: question_id -> 基线是否答对;trainval 分层与验证池准确率均依赖它。
|
||||||
|
val_ratio: validation 占 trainval 视频组总数的比例,[0.0, 1.0]。
|
||||||
|
seed: 随机种子,保证视频组 shuffle 可复现。
|
||||||
|
baseline_run_id: 基线 run 标识;离线切分阶段可留空,由调用方回填。
|
||||||
|
val_wrong_min: validation 池最少错题数(默认 0 = 不检查,保持既有调用契约)。
|
||||||
|
> 0 时切分时保证(不足则从 diag 换入低 T2 错题组补足,耗尽 fail-loud,
|
||||||
|
见 InsufficientValSignal)。
|
||||||
|
wrong_tier_by_video: video_id -> 该视频错题中 T2(defect) 数量;透传给
|
||||||
|
_split_trainval_by_video_group 做 tier 感知 diag/val 分配,None 时退化为
|
||||||
|
原随机 shuffle。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
冻结的三池 Pools:diagnosis/validation 仍是逐题 GeneratedQuestion 列表
|
||||||
|
(元素粒度不变,仅改变"哪些视频进哪个池"),test 为全部 test 题。
|
||||||
|
baseline_val_accuracy = validation 池正确率。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: assignment 缺失某题 video_id(fail-fast 不静默丢题)、
|
||||||
|
assignment 取值非法、correctness 缺失任一参与 Pools 的题(trainval 或
|
||||||
|
test)、或 val_ratio 越界。
|
||||||
|
InsufficientValSignal: val_wrong_min > 0 且 validation 池错题数 < val_wrong_min
|
||||||
|
(P5,验证信号不足以支撑可靠比较,直接报错而非静默放行)。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
视频组 correctness 取组内全部题的 AND(组内均答对才记为 correct 组),
|
||||||
|
据此在 trainval 内做与 _split_one_category 同构的比例分层,但原子是视频组。
|
||||||
|
val_ratio 决定 validation 组数:val_correct = floor(n_correct * n_val / n_total),
|
||||||
|
余额补 wrong 组,全 correct / 全 wrong 时退化为非分层随机划分。下游
|
||||||
|
gate_ladder(信息阶梯冷启动 2:1,核心算法保真 #5)消费的 unit 结构不变。
|
||||||
|
"""
|
||||||
|
if not 0.0 <= val_ratio <= 1.0:
|
||||||
|
raise ValueError(f"val_ratio 必须在 [0.0, 1.0],实际 {val_ratio}")
|
||||||
|
|
||||||
|
trainval_qs, test_qs = _partition_by_video_assignment(questions, assignment, correctness)
|
||||||
|
|
||||||
|
diagnosis, validation = _split_trainval_by_video_group(
|
||||||
|
trainval_qs, correctness, val_ratio, random.Random(seed),
|
||||||
|
wrong_tier_by_video=wrong_tier_by_video,
|
||||||
|
val_wrong_min=val_wrong_min,
|
||||||
|
)
|
||||||
|
|
||||||
|
val_correct = sum(1 for q in validation if correctness.get(q.question_id))
|
||||||
|
baseline_val_accuracy = val_correct / len(validation) if validation else 0.0
|
||||||
|
return Pools(
|
||||||
|
diagnosis=diagnosis,
|
||||||
|
validation=validation,
|
||||||
|
test=test_qs,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
baseline_val_accuracy=baseline_val_accuracy,
|
||||||
|
correctness={
|
||||||
|
q.question_id: correctness[q.question_id] for q in test_qs + validation + diagnosis
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _partition_by_video_assignment(
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
assignment: dict[str, str],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
) -> tuple[list[GeneratedQuestion], list[GeneratedQuestion]]:
|
||||||
|
"""校验归属字典并按 video 归属把题划成 (trainval_qs, test_qs)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: 题目全集。
|
||||||
|
assignment: video_id -> "trainval" | "test" 归属字典。
|
||||||
|
correctness: question_id -> 基线是否答对;对全部参与 Pools 的题(trainval
|
||||||
|
与 test 双侧)强制完整。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
(trainval_qs, test_qs) 逐题列表元组,划分依据每题的 video_id 归属。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: assignment 取值非法、缺失某题 video_id、或 correctness 缺失
|
||||||
|
任一参与 Pools 的题(trainval 或 test,fail-fast,不静默丢题)。
|
||||||
|
"""
|
||||||
|
_assert_valid_assignment(questions, assignment)
|
||||||
|
|
||||||
|
trainval_qs = [q for q in questions if assignment[q.video_id] == "trainval"]
|
||||||
|
test_qs = [q for q in questions if assignment[q.video_id] == "test"]
|
||||||
|
|
||||||
|
# Pools.correctness 会为 test + validation + diagnosis 全体写入基线对错,
|
||||||
|
# 故 test 侧同样必须有 correctness,缺失即报错而非静默兜底 False(P5)。
|
||||||
|
missing_correctness = [
|
||||||
|
q.question_id for q in trainval_qs + test_qs if q.question_id not in correctness
|
||||||
|
]
|
||||||
|
if missing_correctness:
|
||||||
|
raise ValueError(
|
||||||
|
f"correctness 缺失 {len(missing_correctness)} 道题: {missing_correctness[:5]}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return trainval_qs, test_qs
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_valid_assignment(
|
||||||
|
questions: list[GeneratedQuestion],
|
||||||
|
assignment: dict[str, str],
|
||||||
|
) -> None:
|
||||||
|
"""校验归属字典取值合法且覆盖全部题的 video_id,否则 fail-fast。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: 题目全集。
|
||||||
|
assignment: video_id -> "trainval" | "test" 归属字典。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: assignment 含非法取值,或缺失某题的 video_id。
|
||||||
|
"""
|
||||||
|
bad_labels = {v for v in assignment.values() if v not in _VIDEO_ASSIGNMENT_LABELS}
|
||||||
|
if bad_labels:
|
||||||
|
raise ValueError(
|
||||||
|
f"assignment 含非法归属值 {sorted(bad_labels)},仅允许 {_VIDEO_ASSIGNMENT_LABELS}"
|
||||||
|
)
|
||||||
|
|
||||||
|
missing_videos = sorted({q.video_id for q in questions if q.video_id not in assignment})
|
||||||
|
if missing_videos:
|
||||||
|
raise ValueError(f"assignment 缺失 {len(missing_videos)} 个 video_id: {missing_videos[:5]}")
|
||||||
|
|
||||||
|
|
||||||
|
def _partition_video_groups_by_correctness(
|
||||||
|
groups: dict[str, list[GeneratedQuestion]],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
) -> tuple[list[str], list[str]]:
|
||||||
|
"""按视频组正确性把 video_id 分成 (correct_vids, wrong_vids)。
|
||||||
|
|
||||||
|
组正确性取组内全部题的 AND(组内均答对才记为 correct 组),排序保证确定性。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
groups: video_id -> 该视频全部题列表。
|
||||||
|
correctness: question_id -> 基线是否答对(调用方已校验完整)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
(correct_vids, wrong_vids) 两个 video_id 列表,按 video_id 升序。
|
||||||
|
"""
|
||||||
|
correct_vids: list[str] = []
|
||||||
|
wrong_vids: list[str] = []
|
||||||
|
for vid in sorted(groups.keys()):
|
||||||
|
if all(correctness[q.question_id] for q in groups[vid]):
|
||||||
|
correct_vids.append(vid)
|
||||||
|
else:
|
||||||
|
wrong_vids.append(vid)
|
||||||
|
return correct_vids, wrong_vids
|
||||||
|
|
||||||
|
|
||||||
|
def _split_trainval_by_video_group(
|
||||||
|
trainval_qs: list[GeneratedQuestion],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
val_ratio: float,
|
||||||
|
rng: random.Random,
|
||||||
|
wrong_tier_by_video: dict[str, int] | None = None,
|
||||||
|
val_wrong_min: int = 0,
|
||||||
|
) -> tuple[list[GeneratedQuestion], list[GeneratedQuestion]]:
|
||||||
|
"""以视频组为原子对 trainval 题集做 correctness 分层,切出 (diagnosis, validation)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
trainval_qs: trainval 归属的全部题(correctness 已在调用方校验完整)。
|
||||||
|
correctness: question_id -> 基线是否答对;视频组正确性取组内全部题 AND。
|
||||||
|
val_ratio: validation 占视频组总数的比例。
|
||||||
|
rng: 随机数生成器,保证视频组 shuffle 可复现。
|
||||||
|
wrong_tier_by_video: video_id -> 该视频错题中 T2(defect) 的数量。提供时错题
|
||||||
|
视频组按 T2 含量升序进 val(T2 高的组保留在 diagnosis,把高价值缺陷信号
|
||||||
|
留给诊断),确定性排序取代随机 shuffle;None 时退化为原随机 shuffle。
|
||||||
|
val_wrong_min: validation 池最少错题数(切分时保证功效)。> 0 且初分 val 错题
|
||||||
|
不足时,从 diag 侧的错题组按 T2 升序换入 val 直到满足(每组至多移动一次),
|
||||||
|
耗尽仍不足则抛 InsufficientValSignal(fail loud,P5)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
(diagnosis, validation) 逐题列表元组;同一 video 的全部题整组落在同一侧,
|
||||||
|
两侧互斥且并集 == trainval_qs。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
与 _split_one_category 同构:先按视频组 correctness 分正确组/错误组,按比例
|
||||||
|
把 n_val 个组分层落入 validation(全正确退化为非分层随机划分;全错误时若有
|
||||||
|
wrong_tier_by_video 仍按 T2 升序分配,否则随机划分),
|
||||||
|
再把选中组内所有题展开。视频组按 video_id 排序后再 shuffle,保证确定性。
|
||||||
|
"""
|
||||||
|
groups: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||||
|
for q in trainval_qs:
|
||||||
|
groups[q.video_id].append(q)
|
||||||
|
|
||||||
|
video_ids = sorted(groups.keys())
|
||||||
|
n_total = len(video_ids)
|
||||||
|
n_val = round(n_total * val_ratio)
|
||||||
|
|
||||||
|
correct_vids, wrong_vids = _partition_video_groups_by_correctness(groups, correctness)
|
||||||
|
n_correct = len(correct_vids)
|
||||||
|
|
||||||
|
if n_correct == 0 and wrong_tier_by_video is not None:
|
||||||
|
# 全部错误 + 有 tier 信号:按 T2 升序,低 T2 组优先进 val(保留高 T2 在 diag)
|
||||||
|
wrong_vids.sort(key=lambda v: (wrong_tier_by_video.get(v, 0), v))
|
||||||
|
val_vids = set(wrong_vids[:n_val])
|
||||||
|
elif n_correct == 0 or n_correct == n_total:
|
||||||
|
label = "全部正确" if n_correct == n_total else "全部错误"
|
||||||
|
logger.warning("trainval 视频组 {} ({} 组),退化为非分层随机划分", label, n_total)
|
||||||
|
shuffled = list(video_ids)
|
||||||
|
rng.shuffle(shuffled)
|
||||||
|
val_vids = set(shuffled[:n_val])
|
||||||
|
else:
|
||||||
|
val_correct = math.floor(n_correct * n_val / n_total)
|
||||||
|
val_wrong = n_val - val_correct
|
||||||
|
rng.shuffle(correct_vids)
|
||||||
|
if wrong_tier_by_video is None:
|
||||||
|
rng.shuffle(wrong_vids)
|
||||||
|
else:
|
||||||
|
# T2 少的错题组优先进 val(保留 T2 高的组在 diag),确定性排序
|
||||||
|
wrong_vids.sort(key=lambda v: (wrong_tier_by_video.get(v, 0), v))
|
||||||
|
val_vids = set(correct_vids[:val_correct] + wrong_vids[:val_wrong])
|
||||||
|
|
||||||
|
if val_wrong_min > 0:
|
||||||
|
val_wrong_now = sum(
|
||||||
|
1 for v in val_vids for q in groups[v] if not correctness[q.question_id]
|
||||||
|
)
|
||||||
|
# diag 侧仍在的错题组,按 T2 升序(低价值优先移交 val)
|
||||||
|
diag_wrong_pool = sorted(
|
||||||
|
(v for v in wrong_vids if v not in val_vids),
|
||||||
|
key=lambda v: ((wrong_tier_by_video or {}).get(v, 0), v),
|
||||||
|
)
|
||||||
|
for v in diag_wrong_pool:
|
||||||
|
if val_wrong_now >= val_wrong_min:
|
||||||
|
break
|
||||||
|
val_vids.add(v)
|
||||||
|
val_wrong_now += sum(1 for q in groups[v] if not correctness[q.question_id])
|
||||||
|
if val_wrong_now < val_wrong_min:
|
||||||
|
raise InsufficientValSignal(
|
||||||
|
f"trainval 错题不足以让 val 达到 val_wrong_min={val_wrong_min}"
|
||||||
|
f"(修复后仅 {val_wrong_now}),请放大 val_ratio 或调整 trainval 归属。"
|
||||||
|
)
|
||||||
|
|
||||||
|
diagnosis = [q for q in trainval_qs if q.video_id not in val_vids]
|
||||||
|
validation = [q for q in trainval_qs if q.video_id in val_vids]
|
||||||
|
return diagnosis, validation
|
||||||
|
|
||||||
|
|
||||||
class GlobalPoolStrategy:
|
class GlobalPoolStrategy:
|
||||||
"""全局三分策略:test -> val -> diag progressive exclusion。
|
"""全局三分策略:test -> val -> diag progressive exclusion。
|
||||||
|
|
||||||
@@ -167,26 +435,52 @@ class GlobalPoolStrategy:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_correctness_complete(
|
||||||
|
units: list[QuestionUnit],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
) -> None:
|
||||||
|
"""校验 correctness 覆盖所有单元成员题(含 pair 两题),缺失即 fail-fast。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
units: 待校验单元列表。
|
||||||
|
correctness: question_id -> 基线是否答对。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: correctness 中缺少某些 question_id。
|
||||||
|
"""
|
||||||
|
missing = [
|
||||||
|
q.question_id for u in units for q in u.questions if q.question_id not in correctness
|
||||||
|
]
|
||||||
|
if missing:
|
||||||
|
raise ValueError(f"correctness 缺失 {len(missing)} 题: {missing[:5]}")
|
||||||
|
|
||||||
|
|
||||||
def _sample_excluding(
|
def _sample_excluding(
|
||||||
questions: list[GeneratedQuestion],
|
units: list[QuestionUnit],
|
||||||
exclude_ids: set[str],
|
exclude_unit_ids: set[str],
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
**cfg: object,
|
**cfg: object,
|
||||||
) -> list[GeneratedQuestion]:
|
) -> list[GeneratedQuestion]:
|
||||||
"""排除已选 question_id 后,按 cfg 对剩余题做分层采样。
|
"""排除已选 unit 后,以 unit 为原子按 cfg 分层采样,返回展开后的逐题列表。
|
||||||
|
|
||||||
|
候选单元展开为逐题列表后透传给 stratified_sample,后者内部重新 build_units
|
||||||
|
做单元原子采样:correct_ratio / size 按 unit 计数(pair 计 1 个 unit),单元级
|
||||||
|
正确性由 stratified_sample 内部对成员取 AND,命中的孪生对两题永远同进同出。
|
||||||
|
single-only 输入下 unit 与 question 一一对应、顺序不变,采样结果与逐题采样一致。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
questions: 题目全集。
|
units: 单元全集(single 单封、pair 成对聚合)。
|
||||||
exclude_ids: 已被其他池选走的 question_id,从候选中剔除以保证三池互斥。
|
exclude_unit_ids: 已被其他池选走的 unit_id,从候选中剔除以保证三池互斥。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对;单元级正确性由 stratified_sample
|
||||||
|
对成员取 AND(缺失按 False,宽松口径)。
|
||||||
cfg: 透传给 stratified_sample 的采样配置
|
cfg: 透传给 stratified_sample 的采样配置
|
||||||
(size/correct_ratio/task_types[/seed/min_per_class])。
|
(size/correct_ratio/task_types[/seed/min_per_class])。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
采样后的题目列表。
|
采样命中单元展开后的题目列表。
|
||||||
"""
|
"""
|
||||||
pool = [q for q in questions if q.question_id not in exclude_ids]
|
candidates = [u for u in units if u.unit_id not in exclude_unit_ids]
|
||||||
return stratified_sample(pool, correctness, **cfg)
|
return stratified_sample(flatten_units(candidates), correctness, **cfg)
|
||||||
|
|
||||||
|
|
||||||
def _q_to_dict(q: GeneratedQuestion) -> dict:
|
def _q_to_dict(q: GeneratedQuestion) -> dict:
|
||||||
@@ -197,6 +491,10 @@ def _q_to_dict(q: GeneratedQuestion) -> dict:
|
|||||||
|
|
||||||
返回:
|
返回:
|
||||||
包含全部字段的字典(options/source_nodes 从 tuple 转为 list)。
|
包含全部字段的字典(options/source_nodes 从 tuple 转为 list)。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
pair 四字段(pair_id/question_role/flip_axis/unit_id)必须写出——pools.json
|
||||||
|
是训练主回路读回题目的地方,漏写会让孪生对解冻后退化成孤儿 single。
|
||||||
"""
|
"""
|
||||||
return {
|
return {
|
||||||
"question_id": q.question_id,
|
"question_id": q.question_id,
|
||||||
@@ -210,6 +508,10 @@ def _q_to_dict(q: GeneratedQuestion) -> dict:
|
|||||||
"family": q.family,
|
"family": q.family,
|
||||||
"skill_target": q.skill_target,
|
"skill_target": q.skill_target,
|
||||||
"difficulty_steps": q.difficulty_steps,
|
"difficulty_steps": q.difficulty_steps,
|
||||||
|
"pair_id": q.pair_id,
|
||||||
|
"question_role": q.question_role,
|
||||||
|
"flip_axis": q.flip_axis,
|
||||||
|
"unit_id": q.unit_id,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -221,6 +523,11 @@ def _dict_to_q(d: dict) -> GeneratedQuestion:
|
|||||||
|
|
||||||
返回:
|
返回:
|
||||||
恢复的 GeneratedQuestion 实例(options/source_nodes 恢复为 tuple)。
|
恢复的 GeneratedQuestion 实例(options/source_nodes 恢复为 tuple)。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
pair 四字段用 .get 兼容旧 workspace 的 pools.json(无这些字段不崩,默认退化
|
||||||
|
为 single)——断点续跑铁律。unit_id 缺省时传 "",交给 GeneratedQuestion
|
||||||
|
的 __post_init__ 回填为 pair_id 或 question_id,避免孤儿 single。
|
||||||
"""
|
"""
|
||||||
return GeneratedQuestion(
|
return GeneratedQuestion(
|
||||||
question_id=d["question_id"],
|
question_id=d["question_id"],
|
||||||
@@ -234,9 +541,29 @@ def _dict_to_q(d: dict) -> GeneratedQuestion:
|
|||||||
family=d.get("family"),
|
family=d.get("family"),
|
||||||
skill_target=d.get("skill_target"),
|
skill_target=d.get("skill_target"),
|
||||||
difficulty_steps=d.get("difficulty_steps"),
|
difficulty_steps=d.get("difficulty_steps"),
|
||||||
|
pair_id=d.get("pair_id"),
|
||||||
|
question_role=d.get("question_role", "single"),
|
||||||
|
flip_axis=d.get("flip_axis"),
|
||||||
|
unit_id=d.get("unit_id", ""),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _atomic_write_json(path: Path, obj: object) -> None:
|
||||||
|
"""原子写 JSON:先写 <path>.tmp 再 os.replace,避免半截文件。
|
||||||
|
|
||||||
|
崩溃或并发写入时,直接 write_text 可能留下被截断的 JSON;本助手先把完整
|
||||||
|
内容写入同目录临时文件,再用同一文件系统上的原子 rename 替换目标,
|
||||||
|
保证读者只会看到旧完整文件或新完整文件。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
path: 目标 JSON 文件路径。
|
||||||
|
obj: 可 json 序列化对象。
|
||||||
|
"""
|
||||||
|
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||||
|
tmp.write_text(json.dumps(obj, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
def save_pools(
|
def save_pools(
|
||||||
pools: Pools,
|
pools: Pools,
|
||||||
path: Path,
|
path: Path,
|
||||||
@@ -289,10 +616,7 @@ def save_pools(
|
|||||||
data["train_ratio"] = config.train_ratio
|
data["train_ratio"] = config.train_ratio
|
||||||
data["test_source"] = str(config.test_questions_dir) if config.test_questions_dir else None
|
data["test_source"] = str(config.test_questions_dir) if config.test_questions_dir else None
|
||||||
|
|
||||||
path.write_text(
|
_atomic_write_json(path, data)
|
||||||
json.dumps(data, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def load_pools(path: Path) -> Pools:
|
def load_pools(path: Path) -> Pools:
|
||||||
@@ -473,7 +797,9 @@ def build_or_load_pools(
|
|||||||
# 增量构建新类别
|
# 增量构建新类别
|
||||||
paths = resolve_paths(config.workspace_dir)
|
paths = resolve_paths(config.workspace_dir)
|
||||||
questions = load_benchmark(paths.questions_dir)
|
questions = load_benchmark(paths.questions_dir)
|
||||||
with HarnessLog(str(db_path), baseline_run_id) as hlog:
|
with HarnessLog(
|
||||||
|
str(db_path), baseline_run_id, register_run=False
|
||||||
|
) as hlog:
|
||||||
rows = hlog.query(
|
rows = hlog.query(
|
||||||
"SELECT question_id, prediction, answer "
|
"SELECT question_id, prediction, answer "
|
||||||
"FROM predictions WHERE run_id=?",
|
"FROM predictions WHERE run_id=?",
|
||||||
@@ -514,22 +840,41 @@ def build_or_load_pools(
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
# 重新冻结
|
# 重新冻结
|
||||||
pools_path.write_text(
|
_atomic_write_json(pools_path, raw)
|
||||||
json.dumps(raw, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"per_category 增量追加 {} 个新类别: {}",
|
"per_category 增量追加 {} 个新类别: {}",
|
||||||
len(new_types),
|
len(new_types),
|
||||||
sorted(new_types),
|
sorted(new_types),
|
||||||
)
|
)
|
||||||
|
else:
|
||||||
|
# global:校验 baseline_run_id 与(若有)manifest 内容指纹,
|
||||||
|
# 拒绝静默加载与 seed 错配 / 被篡改的冻结切分(P5 fail loud)。
|
||||||
|
frozen_baseline = raw.get("baseline_run_id")
|
||||||
|
if frozen_baseline != baseline_run_id:
|
||||||
|
raise ValueError(
|
||||||
|
f"冻结 pools.json 的 baseline_run_id={frozen_baseline!r} 与 seed "
|
||||||
|
f"的 {baseline_run_id!r} 不一致,拒绝静默加载错配切分。"
|
||||||
|
)
|
||||||
|
manifest_path = config.workspace_dir / "split_manifest.json"
|
||||||
|
if manifest_path.exists():
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
actual_sha = hashlib.sha256(
|
||||||
|
pools_path.read_text(encoding="utf-8").encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
if manifest.get("pools_sha256") != actual_sha:
|
||||||
|
raise ValueError(
|
||||||
|
"pools.json 内容指纹与 split_manifest.pools_sha256 不符,"
|
||||||
|
"冻结产物疑被篡改,拒绝加载。"
|
||||||
|
)
|
||||||
|
|
||||||
return load_pools(pools_path)
|
return load_pools(pools_path)
|
||||||
|
|
||||||
# ── 全新构建 ──
|
# ── 全新构建 ──
|
||||||
paths = resolve_paths(config.workspace_dir)
|
paths = resolve_paths(config.workspace_dir)
|
||||||
questions = load_benchmark(paths.questions_dir)
|
questions = load_benchmark(paths.questions_dir)
|
||||||
with HarnessLog(str(db_path), baseline_run_id) as hlog:
|
with HarnessLog(str(db_path), baseline_run_id, register_run=False) as hlog:
|
||||||
rows = hlog.query(
|
rows = hlog.query(
|
||||||
"SELECT question_id, prediction, answer FROM predictions WHERE run_id=?",
|
"SELECT question_id, prediction, answer FROM predictions WHERE run_id=?",
|
||||||
(baseline_run_id,),
|
(baseline_run_id,),
|
||||||
@@ -604,14 +949,14 @@ class PerCategoryPoolStrategy:
|
|||||||
rng = random.Random(config.seed)
|
rng = random.Random(config.seed)
|
||||||
|
|
||||||
for task_type in sorted(groups.keys()):
|
for task_type in sorted(groups.keys()):
|
||||||
train, val = self._split_one_category(
|
train_units, val_units = self._split_one_category(
|
||||||
groups[task_type],
|
build_units(groups[task_type]),
|
||||||
correctness,
|
correctness,
|
||||||
config.train_ratio,
|
config.train_ratio,
|
||||||
rng,
|
rng,
|
||||||
)
|
)
|
||||||
all_train.extend(train)
|
all_train.extend(flatten_units(train_units))
|
||||||
all_val.extend(val)
|
all_val.extend(flatten_units(val_units))
|
||||||
|
|
||||||
# Phase 4: test 池(从外部目录加载,无则空;按 task_types 过滤)
|
# Phase 4: test 池(从外部目录加载,无则空;按 task_types 过滤)
|
||||||
test: list[GeneratedQuestion] = []
|
test: list[GeneratedQuestion] = []
|
||||||
@@ -759,48 +1104,48 @@ class PerCategoryPoolStrategy:
|
|||||||
|
|
||||||
def _split_one_category(
|
def _split_one_category(
|
||||||
self,
|
self,
|
||||||
questions: list[GeneratedQuestion],
|
units: list[QuestionUnit],
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
train_ratio: float,
|
train_ratio: float,
|
||||||
rng: random.Random,
|
rng: random.Random,
|
||||||
) -> tuple[list[GeneratedQuestion], list[GeneratedQuestion]]:
|
) -> tuple[list[QuestionUnit], list[QuestionUnit]]:
|
||||||
"""单类别 correctness 分层划分。
|
"""单类别 correctness 分层划分,以 unit 为原子(pair 计 1 个 unit)。
|
||||||
|
|
||||||
|
孪生对两题作为一个整体落入 train 或 val,绝不被拆散;single-only 输入下
|
||||||
|
unit 与 question 一一对应、rng 消耗量不变,划分结果与逐题划分完全一致。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
questions: 单类别全部题目。
|
units: 单类别全部单元。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对;单元级正确性取成员的 AND。
|
||||||
train_ratio: train 占总量的比例。
|
train_ratio: train 占单元总量的比例。
|
||||||
rng: 随机数生成器(保证跨类别可复现)。
|
rng: 随机数生成器(保证跨类别可复现)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
(train, val) 题目列表元组,两池互斥且总量 == len(questions)。
|
(train_units, val_units) 单元列表元组,两侧互斥且总量 == len(units)。
|
||||||
|
|
||||||
异常:
|
异常:
|
||||||
ValueError: correctness 中缺少某些 question_id。
|
ValueError: correctness 中缺少某些 question_id。
|
||||||
"""
|
"""
|
||||||
n_total = len(questions)
|
n_total = len(units)
|
||||||
n_train = round(n_total * train_ratio)
|
n_train = round(n_total * train_ratio)
|
||||||
n_val = n_total - n_train
|
n_val = n_total - n_train
|
||||||
|
|
||||||
# 校验 correctness 完整性
|
_assert_correctness_complete(units, correctness)
|
||||||
missing = [q.question_id for q in questions if q.question_id not in correctness]
|
|
||||||
if missing:
|
|
||||||
raise ValueError(f"correctness 缺失 {len(missing)} 题: {missing[:5]}")
|
|
||||||
|
|
||||||
correct_qs = [q for q in questions if correctness[q.question_id]]
|
correct_units = [u for u in units if unit_correctness(u, correctness, strict=False)]
|
||||||
wrong_qs = [q for q in questions if not correctness[q.question_id]]
|
wrong_units = [u for u in units if not unit_correctness(u, correctness, strict=False)]
|
||||||
n_correct = len(correct_qs)
|
n_correct = len(correct_units)
|
||||||
|
|
||||||
# 全 correct 或全 wrong -> 退化为非分层随机划分
|
# 全 correct 或全 wrong -> 退化为非分层随机划分
|
||||||
if n_correct == 0 or n_correct == n_total:
|
if n_correct == 0 or n_correct == n_total:
|
||||||
label = "全部正确" if n_correct == n_total else "全部错误"
|
label = "全部正确" if n_correct == n_total else "全部错误"
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"类别 {} {} ({} 题),退化为非分层随机划分",
|
"类别 {} {} ({} 单元),退化为非分层随机划分",
|
||||||
questions[0].task_type,
|
units[0].task_type,
|
||||||
label,
|
label,
|
||||||
n_total,
|
n_total,
|
||||||
)
|
)
|
||||||
shuffled = list(questions)
|
shuffled = list(units)
|
||||||
rng.shuffle(shuffled)
|
rng.shuffle(shuffled)
|
||||||
return shuffled[:n_train], shuffled[n_train:]
|
return shuffled[:n_train], shuffled[n_train:]
|
||||||
|
|
||||||
@@ -808,11 +1153,11 @@ class PerCategoryPoolStrategy:
|
|||||||
train_correct = math.floor(n_correct * n_train / n_total)
|
train_correct = math.floor(n_correct * n_train / n_total)
|
||||||
train_wrong = n_train - train_correct
|
train_wrong = n_train - train_correct
|
||||||
|
|
||||||
rng.shuffle(correct_qs)
|
rng.shuffle(correct_units)
|
||||||
rng.shuffle(wrong_qs)
|
rng.shuffle(wrong_units)
|
||||||
|
|
||||||
train = correct_qs[:train_correct] + wrong_qs[:train_wrong]
|
train = correct_units[:train_correct] + wrong_units[:train_wrong]
|
||||||
val = correct_qs[train_correct:] + wrong_qs[train_wrong:]
|
val = correct_units[train_correct:] + wrong_units[train_wrong:]
|
||||||
|
|
||||||
assert len(train) == n_train, f"train 数量不匹配: {len(train)} != {n_train}"
|
assert len(train) == n_train, f"train 数量不匹配: {len(train)} != {n_train}"
|
||||||
assert len(val) == n_val, f"val 数量不匹配: {len(val)} != {n_val}"
|
assert len(val) == n_val, f"val 数量不匹配: {len(val)} != {n_val}"
|
||||||
@@ -847,15 +1192,15 @@ class PerCategoryPoolStrategy:
|
|||||||
rng = random.Random(config.seed)
|
rng = random.Random(config.seed)
|
||||||
|
|
||||||
for task_type in sorted(groups.keys()):
|
for task_type in sorted(groups.keys()):
|
||||||
train, val = self._split_one_category(
|
train_units, val_units = self._split_one_category(
|
||||||
groups[task_type],
|
build_units(groups[task_type]),
|
||||||
correctness,
|
correctness,
|
||||||
config.train_ratio,
|
config.train_ratio,
|
||||||
rng,
|
rng,
|
||||||
)
|
)
|
||||||
result[task_type] = {
|
result[task_type] = {
|
||||||
"train": [q.question_id for q in train],
|
"train": [q.question_id for q in flatten_units(train_units)],
|
||||||
"val": [q.question_id for q in val],
|
"val": [q.question_id for q in flatten_units(val_units)],
|
||||||
}
|
}
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
"""QuestionUnit 组装/展开/校验/单元正确性——pair 契约的唯一入口。
|
||||||
|
|
||||||
|
pool 构建、批处理、推理、评测(Task 3+)均通过本模块聚合/展开孪生对,
|
||||||
|
保证 AR pair 的"两题作为整体调度"契约只在一处实现、fail-fast 暴露非法配对。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
|
||||||
|
def _assemble_pair(pair_id: str, group: list[GeneratedQuestion]) -> QuestionUnit:
|
||||||
|
"""校验单个 pair 分组的数量/角色并组装为 pair 单元(fail-fast)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
pair_id: 该分组共享的孪生对标识。
|
||||||
|
group: 归属同一 pair_id 的题目列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
kind="pair" 的 QuestionUnit。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
- 分组必须恰好 2 条,否则视为孤儿/超员,raise ValueError。
|
||||||
|
- 显式检查 pair_original / pair_mirror 角色齐备且唯一,缺失或重复
|
||||||
|
直接 raise ValueError(防 next(...) 静默 StopIteration)。
|
||||||
|
- 合法孪生对交由 QuestionUnit.from_pair 做 video_id/task_type/flip_axis
|
||||||
|
一致性断言。
|
||||||
|
"""
|
||||||
|
if len(group) != 2:
|
||||||
|
raise ValueError(f"pair {pair_id} 数量={len(group)}≠2(孤儿或超员)")
|
||||||
|
originals = [q for q in group if q.question_role == "pair_original"]
|
||||||
|
mirrors = [q for q in group if q.question_role == "pair_mirror"]
|
||||||
|
if len(originals) != 1 or len(mirrors) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
f"pair {pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)},"
|
||||||
|
"需各恰好 1 条"
|
||||||
|
)
|
||||||
|
return QuestionUnit.from_pair(originals[0], mirrors[0])
|
||||||
|
|
||||||
|
|
||||||
|
def build_units(questions: list[GeneratedQuestion]) -> list[QuestionUnit]:
|
||||||
|
"""将扁平题目列表聚合为单元列表:single 单封、pair 按 pair_id 成对聚合。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions: 待聚合的题目列表,可混含 single 与孪生对成员。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
单元列表,先 single 后 pair,顺序稳定(single 保留输入顺序,
|
||||||
|
pair 按首次出现的 pair_id 顺序)。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
pair_id 为空 → single 单元;非空 → 归入对应 pair 桶。各 pair 桶的
|
||||||
|
数量/角色校验与组装下沉到 _assemble_pair(fail-fast),本体只做分组
|
||||||
|
与派发。
|
||||||
|
"""
|
||||||
|
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||||
|
singles: list[QuestionUnit] = []
|
||||||
|
for q in questions:
|
||||||
|
if q.pair_id:
|
||||||
|
by_pair[q.pair_id].append(q)
|
||||||
|
else:
|
||||||
|
singles.append(QuestionUnit.from_single(q))
|
||||||
|
|
||||||
|
pairs = [_assemble_pair(pid, qs) for pid, qs in by_pair.items()]
|
||||||
|
return singles + pairs
|
||||||
|
|
||||||
|
|
||||||
|
def flatten_units(units: list[QuestionUnit]) -> list[GeneratedQuestion]:
|
||||||
|
"""将单元列表无损展开回扁平题目列表。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
units: 单元列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
展开后的题目列表,保持单元顺序及单元内题目顺序。
|
||||||
|
"""
|
||||||
|
return [q for u in units for q in u.questions]
|
||||||
|
|
||||||
|
|
||||||
|
def validate_units(units: list[QuestionUnit]) -> list[QuestionUnit]:
|
||||||
|
"""校验单元列表结构合法性,通过则原样返回(便于链式调用)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
units: 待校验单元列表。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
校验通过的原单元列表。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
pair 单元必须恰好含 2 题,否则 raise ValueError;single 单元无需额外
|
||||||
|
校验(构造时即为 1 题)。用于消费方在使用前做一道防御闸门。
|
||||||
|
"""
|
||||||
|
for u in units:
|
||||||
|
if u.kind == "pair" and u.size != 2:
|
||||||
|
raise ValueError(f"unit {u.unit_id} pair 不成对(size={u.size})")
|
||||||
|
return units
|
||||||
|
|
||||||
|
|
||||||
|
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool], *, strict: bool = True) -> bool:
|
||||||
|
"""计算单元级正确性:AR pair 走双向 AND,single 即单题正确性。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
unit: 目标单元。
|
||||||
|
per_q: 题目 question_id → 该题是否作答正确的映射。
|
||||||
|
strict: 缺键策略。True(默认)时以 per_q[q.question_id] 取值,缺任一题
|
||||||
|
触发 KeyError(防静默兜底,强制上游先补齐全部单题结果);False 时以
|
||||||
|
per_q.get(q.question_id, False) 取值,缺键计 False(宽松口径,供池
|
||||||
|
构建 / gate 冷启动 / 采样等"缺基线对错即视为未答对"的调用点复用)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
单元内所有题目均正确时为 True,否则 False。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
pool 构建(pools)、gate 冷启动(gate_ladder)、分层采样(loader)三处
|
||||||
|
原各自持有的 loose 版 _unit_correct 副本统一收敛到本函数 strict=False 分支,
|
||||||
|
消除重复逻辑与 missing-key 策略分叉。
|
||||||
|
"""
|
||||||
|
if strict:
|
||||||
|
return all(per_q[q.question_id] for q in unit.questions)
|
||||||
|
return all(per_q.get(q.question_id, False) for q in unit.questions)
|
||||||
|
|
||||||
|
|
||||||
|
def unit_correctness_view(
|
||||||
|
units: list[QuestionUnit], per_q: dict[str, bool], *, strict: bool = True
|
||||||
|
) -> dict[str, bool]:
|
||||||
|
"""把逐题对错折叠成单元级视图:unit_id → 单元是否整体正确。
|
||||||
|
|
||||||
|
进化引擎(gate e-process / quadrant / probation / pair_block / compute_accuracy)
|
||||||
|
统一消费此单元视图,保证 AR pair 双向 AND、非 AR single 单题,混格池中
|
||||||
|
孪生对折叠为一个单元、不被 P/Q 单题计分污染(核心算法保真 #5)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
units: 目标单元列表(single 或 pair)。
|
||||||
|
per_q: 题目 question_id → 该题是否作答正确(唯一逐题溯源来源)。
|
||||||
|
strict: 缺键策略,透传给 unit_correctness。True(默认)缺任一题 raise
|
||||||
|
KeyError;False 缺键计 False(宽松口径)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
unit_id → 单元级正确性。single 的 unit_id 等于其 question_id,
|
||||||
|
pair 的 unit_id 等于共享 pair_id。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
逐单元复用 unit_correctness(strict 透传),默认 strict 禁静默兜底、
|
||||||
|
强制上游先补齐全部单题结果。
|
||||||
|
"""
|
||||||
|
return {u.unit_id: unit_correctness(u, per_q, strict=strict) for u in units}
|
||||||
+701
-171
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,59 @@
|
|||||||
|
"""结果驱动视频级切分的冻结溯源 manifest。
|
||||||
|
|
||||||
|
冻结的 pools.json 是切分产物;manifest 记录产出这份切分的关键输入
|
||||||
|
(baseline_run_id、诊断指纹、随机种子、配置)与 pools.json 的内容指纹
|
||||||
|
(pools_sha256),供后续 build_split 写溯源、以及复现校验时比对。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from app.harness.pools import _atomic_write_json
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def write_manifest(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
seed: int,
|
||||||
|
config: dict,
|
||||||
|
pools_json_text: str,
|
||||||
|
coverage_report: dict,
|
||||||
|
generated_at: str,
|
||||||
|
) -> dict:
|
||||||
|
"""写切分冻结溯源 manifest(原子写),返回写入的 dict。
|
||||||
|
|
||||||
|
pools_sha256 = sha256(pools_json_text),供复现时校验冻结的 pools.json 内容
|
||||||
|
是否与本次切分一致。generated_at 由调用方传入(库内不用 datetime.now),
|
||||||
|
以保证相同输入产出相同 manifest,可复现。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
path: manifest 目标 JSON 文件路径。
|
||||||
|
baseline_run_id: 产出本次切分所依据的基线 run 标识。
|
||||||
|
diag_fingerprint: 诊断结果指纹(决定 train/val 归属的输入)。
|
||||||
|
seed: 切分使用的随机种子。
|
||||||
|
config: 切分相关配置快照(如 train_ratio 等)。
|
||||||
|
pools_json_text: 冻结的 pools.json 完整文本,用于计算内容指纹。
|
||||||
|
coverage_report: 各类别 train/val 覆盖统计报告。
|
||||||
|
generated_at: 生成时间戳(ISO 字符串),由调用方传入。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
写入 manifest 的 dict(与落盘内容一致)。
|
||||||
|
"""
|
||||||
|
manifest = {
|
||||||
|
"baseline_run_id": baseline_run_id,
|
||||||
|
"diag_fingerprint": diag_fingerprint,
|
||||||
|
"seed": seed,
|
||||||
|
"config": config,
|
||||||
|
"pools_sha256": hashlib.sha256(pools_json_text.encode("utf-8")).hexdigest(),
|
||||||
|
"coverage_report": coverage_report,
|
||||||
|
"generated_at": generated_at,
|
||||||
|
}
|
||||||
|
_atomic_write_json(path, manifest)
|
||||||
|
return manifest
|
||||||
@@ -0,0 +1,551 @@
|
|||||||
|
"""视频级切分选择:signal 分层、视频聚合、贪心联合约束选择(纯函数)。
|
||||||
|
|
||||||
|
结果驱动切分管线的核心:把诊断信号投影为多样性格子,供贪心选择器最大化覆盖。
|
||||||
|
本模块起步定义 evolution_target 派生与多样性格子;后续追加 score_signal /
|
||||||
|
build_video_records / select_split。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import random
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
|
||||||
|
def diag_fingerprint(prompt_version: str, model: str, code_version: str) -> str:
|
||||||
|
"""由 (诊断 prompt 版本, 模型名, 代码 git 短 SHA) 合成诊断口径指纹。
|
||||||
|
|
||||||
|
诊断信号以 (question_id, baseline_run_id, diag_fingerprint) 为主键持久化,
|
||||||
|
指纹隔离不同诊断配置的信号——换 prompt 版本 / 换模型 / 换代码实现都会得到
|
||||||
|
新指纹,从而 `--force` 用新指纹重跑诊断时**不覆盖旧记录**(旧指纹行仍在),
|
||||||
|
保证不同口径的诊断结果可并存、可回溯、可比对。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
prompt_version: 诊断 prompt 的版本标识(如 prompts/diagnose_*.md 的版本)。
|
||||||
|
model: 执行诊断的模型名(如 "deepseek-v4")。
|
||||||
|
code_version: 诊断代码的版本(约定为 git 短 SHA)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
16 位十六进制指纹(sha256 截断),对三分量任一变化敏感、对相同三元组确定。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
三分量用 "|" 分隔后 sha256,取前 16 位;分隔符防止 ("ab","c") 与 ("a","bc")
|
||||||
|
碰撞成同一指纹。纯函数,相同输入永远同输出,可安全用于主键。
|
||||||
|
"""
|
||||||
|
return hashlib.sha256("|".join([prompt_version, model, code_version]).encode()).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
_EVOLUTION_TARGET = {
|
||||||
|
"extraction_failure": "tool",
|
||||||
|
"search_failure": "skill",
|
||||||
|
"reasoning_failure": "skill",
|
||||||
|
"mixed": "system",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def evolution_target_of(error_type: str) -> str:
|
||||||
|
"""由 error_type 确定性派生进化目标(tool/skill/system)。
|
||||||
|
|
||||||
|
这是报告用的派生标注,非独立多样性轴(多样性主格子=task_type×error_type)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
error_type: 诊断瀑布归因的错误类别(extraction/search/reasoning/mixed_failure)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
进化目标字符串 tool / skill / system。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
ValueError: error_type 不在已知集合内(不静默兜底)。
|
||||||
|
"""
|
||||||
|
if error_type not in _EVOLUTION_TARGET:
|
||||||
|
raise ValueError(f"未知 error_type: {error_type}")
|
||||||
|
return _EVOLUTION_TARGET[error_type]
|
||||||
|
|
||||||
|
|
||||||
|
def cell_of(task_type: str, error_type: str) -> tuple[str, str]:
|
||||||
|
"""构造多样性主格子 = (task_type, error_type)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
task_type: 题型(12 类之一)。
|
||||||
|
error_type: 错误类别(4 类之一)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
(task_type, error_type) 二元组,作为覆盖计数的格子键。
|
||||||
|
"""
|
||||||
|
return (task_type, error_type)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SignalLabel:
|
||||||
|
"""诊断信号分层标签(DiagnosisResult 的确定性投影)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
tier: 信号层级,取值 T0 / T1 / T2 / uncertain(判据见 score_signal)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
tier: str
|
||||||
|
|
||||||
|
|
||||||
|
def score_signal(*, cause_category: str | None, infra: bool, degraded: bool) -> SignalLabel:
|
||||||
|
"""把诊断产物投影为信号分层 tier(不发明新分类,是确定性投影)。
|
||||||
|
|
||||||
|
分层优先级顺序固定(用早返回表达,不用魔法权重):
|
||||||
|
先判 INFRA,再判 degraded,然后 defect / lapse,最后兜底 uncertain。
|
||||||
|
|
||||||
|
各层判据来源:
|
||||||
|
T0 — infra=True,即诊断 INFRA 排除(stop_reason ∈ {error, parse_error}),
|
||||||
|
基础设施失败先于一切判定,排除出可训练主体。
|
||||||
|
uncertain — degraded=True(judge 解析失败)或 cause_category 落不到
|
||||||
|
defect/lapse 上(如为 None),信号不可信,排除出 T2。
|
||||||
|
T2 — cause_category == "defect",可训练核心,进多样性覆盖与训练主体。
|
||||||
|
T1 — cause_category == "lapse",低信号(含无解题),接受但不作训练主体。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
cause_category: 诊断的缺陷归因("defect" / "lapse" / None)。
|
||||||
|
infra: 是否被 INFRA 护栏排除(基础设施失败)。
|
||||||
|
degraded: judge 是否解析失败导致诊断降级。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
SignalLabel,其 tier 字段为上述四层之一。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
关键字参数强制传入,防止 infra / degraded 两个 bool 位置混淆。
|
||||||
|
"""
|
||||||
|
if infra:
|
||||||
|
return SignalLabel(tier="T0")
|
||||||
|
if degraded:
|
||||||
|
return SignalLabel(tier="uncertain")
|
||||||
|
if cause_category == "defect":
|
||||||
|
return SignalLabel(tier="T2")
|
||||||
|
if cause_category == "lapse":
|
||||||
|
return SignalLabel(tier="T1")
|
||||||
|
return SignalLabel(tier="uncertain")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class VideoRecord:
|
||||||
|
"""全视频画像单元(贪心选择器 Task 8 的输入单元)。
|
||||||
|
|
||||||
|
覆盖全部视频(含全对、零诊断信号的视频),既承载 test 代表性所需的难度/题型画像,
|
||||||
|
也叠加 T2 可训练缺陷的多样性格子,供选择器算覆盖与补集。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
video_id: 视频唯一标识。
|
||||||
|
type_set: 该视频所有题的 task_type 集合(去重,画像用)。
|
||||||
|
n_correct: 该视频答对题数。
|
||||||
|
difficulty: 难度画像桶 = 错题数 = 题数 - n_correct。
|
||||||
|
cells: 仅 tier=="T2" 信号行投影的 (task_type, error_type) 主格子并集(去重)。
|
||||||
|
wrong_by_type: 各 task_type 的 T2 计数,供选择器 floor 约束(普通 dict)。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
frozen 生成的 __hash__ 会遍历各字段;wrong_by_type 为不可哈希 dict,
|
||||||
|
故显式标注 hash=False 将其排除出哈希,避免 VideoRecord 入 set/dict 键时报错,
|
||||||
|
仍保留其参与相等性比较。
|
||||||
|
"""
|
||||||
|
|
||||||
|
video_id: str
|
||||||
|
type_set: frozenset[str]
|
||||||
|
n_correct: int
|
||||||
|
difficulty: int
|
||||||
|
cells: frozenset[tuple[str, str]]
|
||||||
|
wrong_by_type: dict[str, int] = field(hash=False)
|
||||||
|
|
||||||
|
|
||||||
|
def build_video_records(preds: list[dict], signal_rows: list[dict]) -> list[VideoRecord]:
|
||||||
|
"""由全量 predictions 与诊断信号行构建全视频 VideoRecord 列表。
|
||||||
|
|
||||||
|
先按 video_id 聚合全部 predictions(覆盖全对、零信号视频),再叠加仅 tier=="T2"
|
||||||
|
的诊断信号为多样性格子与 wrong_by_type 计数。非 T2 信号行(T0/T1/uncertain)
|
||||||
|
不计入格子与计数。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
preds: 全量预测行,每行含 video_id / question_id / task_type / correct。
|
||||||
|
每视频含其全部题(不限于错题),correct 为布尔答对标记。
|
||||||
|
signal_rows: 诊断信号行,每行含 question_id / task_type / error_type / tier。
|
||||||
|
诊断只覆盖错题子集,正确题无对应信号行属正常,不视为错误。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
全部视频的 VideoRecord 列表,按视频在 preds 中首次出现顺序排列。
|
||||||
|
无任何 T2 信号的视频其 cells 为空 frozenset、wrong_by_type 为空 dict。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
signal_rows 的 question_id 若不在 preds 中则忽略(诊断可能滞后于当前预测集,
|
||||||
|
非数据损坏),不 fail-fast;缺失必需键则按 KeyError 直接暴露(不静默兜底)。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
KeyError: preds 或 signal_rows 行缺少必需键(校验前置,防脏数据静默通过)。
|
||||||
|
"""
|
||||||
|
# Phase 1: 按 video_id 聚合 preds(保持首次出现顺序)。
|
||||||
|
signal_by_qid = {row["question_id"]: row for row in signal_rows}
|
||||||
|
aggregates: dict[str, dict] = {}
|
||||||
|
for pred in preds:
|
||||||
|
video_id = pred["video_id"]
|
||||||
|
bucket = aggregates.setdefault(
|
||||||
|
video_id, {"types": set(), "question_ids": [], "n_correct": 0}
|
||||||
|
)
|
||||||
|
bucket["types"].add(pred["task_type"])
|
||||||
|
bucket["question_ids"].append(pred["question_id"])
|
||||||
|
if pred["correct"]:
|
||||||
|
bucket["n_correct"] += 1
|
||||||
|
|
||||||
|
# Phase 2: 逐视频叠加 T2 信号为格子与 wrong_by_type。
|
||||||
|
records: list[VideoRecord] = []
|
||||||
|
for video_id, bucket in aggregates.items():
|
||||||
|
cells: set[tuple[str, str]] = set()
|
||||||
|
wrong_by_type: dict[str, int] = {}
|
||||||
|
for question_id in bucket["question_ids"]:
|
||||||
|
row = signal_by_qid.get(question_id)
|
||||||
|
if row is None or row["tier"] != "T2":
|
||||||
|
continue
|
||||||
|
task_type = row["task_type"]
|
||||||
|
cells.add(cell_of(task_type, row["error_type"]))
|
||||||
|
wrong_by_type[task_type] = wrong_by_type.get(task_type, 0) + 1
|
||||||
|
n_questions = len(bucket["question_ids"])
|
||||||
|
records.append(
|
||||||
|
VideoRecord(
|
||||||
|
video_id=video_id,
|
||||||
|
type_set=frozenset(bucket["types"]),
|
||||||
|
n_correct=bucket["n_correct"],
|
||||||
|
difficulty=n_questions - bucket["n_correct"],
|
||||||
|
cells=frozenset(cells),
|
||||||
|
wrong_by_type=wrong_by_type,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SelectConfig:
|
||||||
|
"""贪心联合约束选择器的实验配置(科研配置,随实验扫动)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
n_trainval: trainval 目标视频数(多样性阶段的填充上限)。
|
||||||
|
floor_k: 各高信号 task_type 的 T2 defect 数下限(硬约束,floor 阶段满足)。
|
||||||
|
epsilon: test 相对全局的最大允许分布偏差(题型占比 / 难度画像两维,逐桶)。
|
||||||
|
reportable_types: 参与 ε 题型代表性校验的 task_type 集(长尾类型不入约束)。
|
||||||
|
seed: 预洗牌随机种子,仅用于打破等增益平局,保证同 config 同 videos 同解。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
floor_k / reportable_types 为不可哈希容器,标 hash=False 排除出自动 __hash__,
|
||||||
|
避免 frozen dataclass 被哈希时报错(本类不作为字典键,仅承载配置)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
n_trainval: int
|
||||||
|
floor_k: dict[str, int] = field(hash=False)
|
||||||
|
epsilon: float
|
||||||
|
reportable_types: frozenset[str] | set[str] = field(hash=False)
|
||||||
|
seed: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SplitAssignment:
|
||||||
|
"""视频级切分归属结果(交给 split_by_video_assignment 做题级切分)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
trainval: 进入 trainval 的 video_id 元组(按选择顺序,确定性)。
|
||||||
|
test: 补集视频的 video_id 元组(按 videos 原始顺序)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
trainval: tuple[str, ...]
|
||||||
|
test: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
class InfeasibleSplitError(Exception):
|
||||||
|
"""floor 硬约束与 ε 守护死锁、无法在不破坏 test 代表性下满足 floor 时抛出。
|
||||||
|
|
||||||
|
fail loud(P5):不静默兜底、不随机塞题,直接暴露不可行并报告未达标类型。
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def derive_reportable_types(total_by_type: dict[str, int], report_floor: int) -> set[str]:
|
||||||
|
"""派生可 per-type 报告的 task_type 集(长尾处理:题数 ≥ report_floor 才报告)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
total_by_type: 各 task_type 的总题数(或代理承载数)。
|
||||||
|
report_floor: 报告门限,低于此的类型并入长尾、不单独报告也不入 ε 约束。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
总题数 ≥ report_floor 的 task_type 集合。
|
||||||
|
"""
|
||||||
|
return {task_type for task_type, total in total_by_type.items() if total >= report_floor}
|
||||||
|
|
||||||
|
|
||||||
|
def _type_membership_fraction(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
|
||||||
|
"""计算各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: 视频记录子集(非空,调用方保证)。
|
||||||
|
keys: 需计算占比的 task_type 键集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{task_type: 占比},占比 ∈ [0, 1]。
|
||||||
|
"""
|
||||||
|
total = len(records)
|
||||||
|
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
|
||||||
|
|
||||||
|
|
||||||
|
def _difficulty_fraction(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
|
||||||
|
"""计算各难度桶在给定视频集中的占比(难度 = 错题数)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
records: 视频记录子集(非空,调用方保证)。
|
||||||
|
buckets: 需计算占比的难度桶键集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{难度桶: 占比},占比 ∈ [0, 1]。
|
||||||
|
"""
|
||||||
|
total = len(records)
|
||||||
|
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
|
||||||
|
|
||||||
|
|
||||||
|
def _max_deviation(global_dist: dict, subset_dist: dict, keys: set) -> float:
|
||||||
|
"""逐键取全局与子集分布的最大绝对偏差(键集为空时约定为 0.0)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
global_dist: 全局分布(键 → 占比)。
|
||||||
|
subset_dist: 子集分布(键 → 占比)。
|
||||||
|
keys: 参与比较的键集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
逐键 |global - subset| 的最大值;keys 为空返回 0.0。
|
||||||
|
"""
|
||||||
|
if not keys:
|
||||||
|
return 0.0
|
||||||
|
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
|
||||||
|
|
||||||
|
|
||||||
|
def _epsilon_ok(
|
||||||
|
test_video_records: list[VideoRecord],
|
||||||
|
videos_all: list[VideoRecord],
|
||||||
|
config: SelectConfig,
|
||||||
|
) -> bool:
|
||||||
|
"""校验 test 子集相对全局在题型占比与难度画像两维的偏差是否均 ≤ epsilon。
|
||||||
|
|
||||||
|
test 越简单则 headline 越虚高,故 test 必须保持代表性:逐 reportable 题型、逐难度桶
|
||||||
|
比较 test 与全局占比,任一维超 epsilon 即判不合格。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
test_video_records: 候选 test 子集(trainval 补集)。
|
||||||
|
videos_all: 全部视频(全局分布基准)。
|
||||||
|
config: 选择配置,提供 epsilon 与 reportable_types。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
两维最大偏差均 ≤ epsilon 为 True;test 为空视为不合格返回 False。
|
||||||
|
"""
|
||||||
|
if not test_video_records:
|
||||||
|
return False
|
||||||
|
type_keys = set(config.reportable_types)
|
||||||
|
global_type = _type_membership_fraction(videos_all, type_keys)
|
||||||
|
subset_type = _type_membership_fraction(test_video_records, type_keys)
|
||||||
|
if _max_deviation(global_type, subset_type, type_keys) > config.epsilon:
|
||||||
|
return False
|
||||||
|
diff_keys = {r.difficulty for r in videos_all}
|
||||||
|
global_diff = _difficulty_fraction(videos_all, diff_keys)
|
||||||
|
subset_diff = _difficulty_fraction(test_video_records, diff_keys)
|
||||||
|
return _max_deviation(global_diff, subset_diff, diff_keys) <= config.epsilon
|
||||||
|
|
||||||
|
|
||||||
|
def _current_wrong_counts(selected: list[VideoRecord]) -> dict[str, int]:
|
||||||
|
"""聚合已选 trainval 视频的 T2 defect 计数(供 floor 达标判定)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
selected: 当前已进入 trainval 的视频记录。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{task_type: T2 defect 累计数}。
|
||||||
|
"""
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
for video in selected:
|
||||||
|
for task_type, wrong in video.wrong_by_type.items():
|
||||||
|
counts[task_type] = counts.get(task_type, 0) + wrong
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def _unmet_floors(selected: list[VideoRecord], floor_k: dict[str, int]) -> dict[str, int]:
|
||||||
|
"""计算尚未达标的 floor 类型及其缺口(已达标类型不返回)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
selected: 当前已进入 trainval 的视频记录。
|
||||||
|
floor_k: 各高信号 task_type 的 defect 下限。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{task_type: 缺口数},仅含 current < floor 的类型;全达标返回空 dict。
|
||||||
|
"""
|
||||||
|
counts = _current_wrong_counts(selected)
|
||||||
|
return {
|
||||||
|
task_type: floor - counts.get(task_type, 0)
|
||||||
|
for task_type, floor in floor_k.items()
|
||||||
|
if counts.get(task_type, 0) < floor
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _floor_fill_count(video: VideoRecord, deficits: dict[str, int]) -> int:
|
||||||
|
"""计算某视频能填补的 floor 缺口槽数(逐类型取 min(defect, 缺口) 求和)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
video: 候选视频记录。
|
||||||
|
deficits: 各未达标类型的缺口。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
该视频实际可填的槽数总和(0 表示对当前缺口无贡献)。
|
||||||
|
"""
|
||||||
|
return sum(
|
||||||
|
min(video.wrong_by_type.get(task_type, 0), deficit)
|
||||||
|
for task_type, deficit in deficits.items()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _marginal_gain(video: VideoRecord, current_cells: set[tuple[str, str]]) -> int:
|
||||||
|
"""计算把某视频移入 trainval 的边际覆盖增益(新开的 T2 格子数)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
video: 候选视频记录。
|
||||||
|
current_cells: 当前 trainval 的 T2 格子并集。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
video.cells 相对 current_cells 的新增格子数(去重)。
|
||||||
|
"""
|
||||||
|
return len(video.cells - current_cells)
|
||||||
|
|
||||||
|
|
||||||
|
def _prospective_test(pool: list[VideoRecord], candidate: VideoRecord) -> list[VideoRecord]:
|
||||||
|
"""构造"把候选移入 trainval 后"的 test 子集 = 当前剩余池去掉候选。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
pool: 当前尚未进入 trainval 的视频(即当前 test 补集)。
|
||||||
|
candidate: 拟移入 trainval 的候选视频。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
pool 去掉 candidate 后的视频列表。
|
||||||
|
"""
|
||||||
|
return [r for r in pool if r.video_id != candidate.video_id]
|
||||||
|
|
||||||
|
|
||||||
|
def _satisfy_floors(
|
||||||
|
selected: list[VideoRecord],
|
||||||
|
pool: list[VideoRecord],
|
||||||
|
videos_all: list[VideoRecord],
|
||||||
|
config: SelectConfig,
|
||||||
|
) -> None:
|
||||||
|
"""Floor 阶段:硬约束优先,逐步移入能填 floor 槽且不破 ε 的视频(就地改 selected/pool)。
|
||||||
|
|
||||||
|
每轮取未达标类型的缺口,候选 = 能填 ≥1 槽 且 移入后 test 仍满足 ε 的视频;候选为空即
|
||||||
|
死锁抛 InfeasibleSplitError;否则选填槽最多者(等槽数按预洗牌顺序取首个,确定性)。
|
||||||
|
|
||||||
|
n_trainval 是硬预算:floor 需求超出预算(尚有缺口却已达 n_trainval)也判不可行 fail loud,
|
||||||
|
保证返回的 trainval 永不超过 n_trainval(不因硬约束悄悄超额、挤占 test)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
selected: 当前 trainval(就地追加)。
|
||||||
|
pool: 当前剩余池 = test 补集(就地移除)。
|
||||||
|
videos_all: 全部视频(ε 全局基准)。
|
||||||
|
config: 选择配置。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
InfeasibleSplitError: 存在未达标类型但无候选可在不破 ε 下填补,
|
||||||
|
或 floor 需求超过 n_trainval 预算。
|
||||||
|
"""
|
||||||
|
while True:
|
||||||
|
deficits = _unmet_floors(selected, config.floor_k)
|
||||||
|
if not deficits:
|
||||||
|
return
|
||||||
|
if len(selected) >= config.n_trainval:
|
||||||
|
raise InfeasibleSplitError(
|
||||||
|
f"floor 需求超过 n_trainval={config.n_trainval} 预算,剩余缺口: {dict(deficits)}"
|
||||||
|
)
|
||||||
|
candidates = [
|
||||||
|
video
|
||||||
|
for video in pool
|
||||||
|
if _floor_fill_count(video, deficits) > 0
|
||||||
|
and _epsilon_ok(_prospective_test(pool, video), videos_all, config)
|
||||||
|
]
|
||||||
|
if not candidates:
|
||||||
|
raise InfeasibleSplitError(
|
||||||
|
f"floor 无法在 ε≤{config.epsilon} 下满足,未达标类型缺口: {dict(deficits)}"
|
||||||
|
)
|
||||||
|
pick = max(candidates, key=lambda video: _floor_fill_count(video, deficits))
|
||||||
|
selected.append(pick)
|
||||||
|
pool.remove(pick)
|
||||||
|
|
||||||
|
|
||||||
|
def _maximize_diversity(
|
||||||
|
selected: list[VideoRecord],
|
||||||
|
pool: list[VideoRecord],
|
||||||
|
videos_all: list[VideoRecord],
|
||||||
|
config: SelectConfig,
|
||||||
|
) -> None:
|
||||||
|
"""多样性阶段:submodular 贪心,按边际覆盖增益降序填至 n_trainval(就地改 selected/pool)。
|
||||||
|
|
||||||
|
每轮对剩余视频算新开格子数,按 -增益稳定排序(等增益按预洗牌顺序),取第一个移入后 test
|
||||||
|
仍满足 ε 的视频;若无任一视频可加而不破 ε,则停并记 warning(欠额,不静默不报错)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
selected: 当前 trainval(就地追加)。
|
||||||
|
pool: 当前剩余池 = test 补集(就地移除)。
|
||||||
|
videos_all: 全部视频(ε 全局基准)。
|
||||||
|
config: 选择配置。
|
||||||
|
"""
|
||||||
|
while len(selected) < config.n_trainval:
|
||||||
|
if not pool:
|
||||||
|
logger.warning(
|
||||||
|
"多样性阶段剩余池耗尽,trainval 欠额: {}/{}", len(selected), config.n_trainval
|
||||||
|
)
|
||||||
|
return
|
||||||
|
current_cells = set().union(*(v.cells for v in selected)) if selected else set()
|
||||||
|
ranked = sorted(pool, key=lambda video: -_marginal_gain(video, current_cells))
|
||||||
|
pick = next(
|
||||||
|
(
|
||||||
|
video
|
||||||
|
for video in ranked
|
||||||
|
if _epsilon_ok(_prospective_test(pool, video), videos_all, config)
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if pick is None:
|
||||||
|
logger.warning(
|
||||||
|
"多样性阶段 ε 守护阻断全部候选,trainval 欠额: {}/{}",
|
||||||
|
len(selected),
|
||||||
|
config.n_trainval,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
selected.append(pick)
|
||||||
|
pool.remove(pick)
|
||||||
|
|
||||||
|
|
||||||
|
def select_split(videos: list[VideoRecord], *, config: SelectConfig) -> SplitAssignment:
|
||||||
|
"""贪心联合约束视频级切分:floor 硬约束先满足、多样性覆盖后最大化、ε 守护 test 代表性。
|
||||||
|
|
||||||
|
核心洞察:全数据集错题总数固定,越把信号塞 trainval、test 越简单、headline 越虚高,
|
||||||
|
故 test 必须保持代表性(ε 约束),trainval 只靠 floor + 多样性覆盖富集,不从 test 偷难题。
|
||||||
|
|
||||||
|
两阶段贪心(均带 ε 守护):先 Floor 阶段满足各高信号类型 defect 下限(不可行 fail loud),
|
||||||
|
再多样性阶段按边际覆盖增益填至 n_trainval(欠额记 warning)。test = trainval 补集。
|
||||||
|
|
||||||
|
确定性:入场用 random.Random(seed) 对视频列表做一次预洗牌,此后 max / 稳定排序仅取首个,
|
||||||
|
seed 只打破等增益 / 等槽数平局;同 config 同 videos → 同结果。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
videos: 全部视频记录(Task 7 build_video_records 产物)。
|
||||||
|
config: 选择配置(关键字传入,含 n_trainval / floor_k / epsilon / reportable_types / seed)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
SplitAssignment,trainval 按选择顺序、test 按 videos 原始顺序。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
InfeasibleSplitError: videos 为空、floor 与 ε 死锁无法满足、
|
||||||
|
或 floor 需求超过 n_trainval 预算。
|
||||||
|
"""
|
||||||
|
if not videos:
|
||||||
|
raise InfeasibleSplitError("videos 为空,无法执行切分")
|
||||||
|
rng = random.Random(config.seed)
|
||||||
|
pool = list(videos)
|
||||||
|
rng.shuffle(pool)
|
||||||
|
selected: list[VideoRecord] = []
|
||||||
|
_satisfy_floors(selected, pool, videos, config)
|
||||||
|
_maximize_diversity(selected, pool, videos, config)
|
||||||
|
trainval_ids = {video.video_id for video in selected}
|
||||||
|
trainval = tuple(video.video_id for video in selected)
|
||||||
|
test = tuple(video.video_id for video in videos if video.video_id not in trainval_ids)
|
||||||
|
return SplitAssignment(trainval=trainval, test=test)
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
"""把 predictions.steps_json 转成 RunLog.get_traces 的行形。
|
||||||
|
|
||||||
|
infer_adhoc 的 traces 表为空,轨迹存于 steps_json({thought, tool_call, tool_output})。
|
||||||
|
诊断管线经 get_traces 消费轨迹,故需此确定性转换适配。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def steps_json_to_trace_rows(
|
||||||
|
video_id: str, question_id: str, steps_json: str
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""将单题 steps_json 解析为 trace 行列表(step 从 0 递增)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
video_id: 视频 ID。
|
||||||
|
question_id: 题 ID。
|
||||||
|
steps_json: predictions.steps_json 原文(JSON 数组字符串)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
行字典列表,字段对齐 traces 表 schema;空/空数组返回 []。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
- 工具名读 tool_call.tool(infer_adhoc 真实字段),对极少数历史
|
||||||
|
数据的 name 做 back-compat 回退。
|
||||||
|
- steps_json 非 JSON 数组时直接报错,不做兜底掩盖。
|
||||||
|
"""
|
||||||
|
if not steps_json or not steps_json.strip():
|
||||||
|
return []
|
||||||
|
steps = json.loads(steps_json)
|
||||||
|
if not isinstance(steps, list):
|
||||||
|
raise ValueError(f"steps_json 非数组: {question_id}")
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for i, s in enumerate(steps):
|
||||||
|
call = s.get("tool_call") or {}
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"video_id": video_id,
|
||||||
|
"question_id": question_id,
|
||||||
|
"step": i,
|
||||||
|
# infer_adhoc 用 "tool";back-compat 兼容极少数 "name"
|
||||||
|
"tool_name": call.get("tool", call.get("name")),
|
||||||
|
"tool_args": call.get("args", {}),
|
||||||
|
"tool_output": s.get("tool_output"),
|
||||||
|
"thought": s.get("thought"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return rows
|
||||||
+25
-1
@@ -190,6 +190,9 @@ def init_seed(
|
|||||||
baseline_run_id: str,
|
baseline_run_id: str,
|
||||||
parent: str | None,
|
parent: str | None,
|
||||||
description: str,
|
description: str,
|
||||||
|
*,
|
||||||
|
pools_json: Path | None = None,
|
||||||
|
split_manifest: Path | None = None,
|
||||||
) -> Path:
|
) -> Path:
|
||||||
"""在 store/seeds/<name> 写一个种子:权重 + baseline.db + seed.json。
|
"""在 store/seeds/<name> 写一个种子:权重 + baseline.db + seed.json。
|
||||||
|
|
||||||
@@ -202,6 +205,10 @@ def init_seed(
|
|||||||
baseline_run_id: 全量记录的 run_id,fresh 时注入 build_pools。
|
baseline_run_id: 全量记录的 run_id,fresh 时注入 build_pools。
|
||||||
parent: 来源(initial 为 None)。
|
parent: 来源(initial 为 None)。
|
||||||
description: 人类可读说明。
|
description: 人类可读说明。
|
||||||
|
pools_json: 可选,冻结切分 pools.json 源路径;提供时拷入 seed 目录,
|
||||||
|
供 fresh 训练时携带冻结切分进 workspace(见 init_workspace_from_seed)。
|
||||||
|
split_manifest: 可选,冻结切分 split_manifest.json 源路径;提供时拷入 seed
|
||||||
|
目录,供加载时校验 pools.json 内容指纹(pools_sha256)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
种子目录路径。
|
种子目录路径。
|
||||||
@@ -216,6 +223,10 @@ def init_seed(
|
|||||||
shutil.copytree(skills_dir, seed_dir / "skills")
|
shutil.copytree(skills_dir, seed_dir / "skills")
|
||||||
shutil.copytree(prompts_dir, seed_dir / "prompts")
|
shutil.copytree(prompts_dir, seed_dir / "prompts")
|
||||||
shutil.copy2(baseline_db, seed_dir / "baseline.db")
|
shutil.copy2(baseline_db, seed_dir / "baseline.db")
|
||||||
|
if pools_json is not None:
|
||||||
|
shutil.copy2(pools_json, seed_dir / "pools.json")
|
||||||
|
if split_manifest is not None:
|
||||||
|
shutil.copy2(split_manifest, seed_dir / "split_manifest.json")
|
||||||
(seed_dir / "seed.json").write_text(
|
(seed_dir / "seed.json").write_text(
|
||||||
json.dumps(
|
json.dumps(
|
||||||
{
|
{
|
||||||
@@ -266,7 +277,9 @@ def read_seed(store_dir: Path, name: str) -> dict:
|
|||||||
return json.loads(seed_json.read_text())
|
return json.loads(seed_json.read_text())
|
||||||
|
|
||||||
|
|
||||||
def extract_run_db(src_db: Path, dst_db: Path, run_id: str) -> None:
|
def extract_run_db(
|
||||||
|
src_db: Path, dst_db: Path, run_id: str, *, dedupe_per_question: bool = False
|
||||||
|
) -> None:
|
||||||
"""从 src_db 抽出某 run_id 的 _runs + predictions 行,写一个最小 db(种子 baseline.db)。
|
"""从 src_db 抽出某 run_id 的 _runs + predictions 行,写一个最小 db(种子 baseline.db)。
|
||||||
|
|
||||||
用源表的**原始 CREATE 语句**重建目标表,保留主键/列类型/约束——
|
用源表的**原始 CREATE 语句**重建目标表,保留主键/列类型/约束——
|
||||||
@@ -277,6 +290,9 @@ def extract_run_db(src_db: Path, dst_db: Path, run_id: str) -> None:
|
|||||||
src_db: 源 harness.db。
|
src_db: 源 harness.db。
|
||||||
dst_db: 目标 db(不得已存在)。
|
dst_db: 目标 db(不得已存在)。
|
||||||
run_id: 要抽取的 run。
|
run_id: 要抽取的 run。
|
||||||
|
dedupe_per_question: True 时 predictions 表每 question_id 仅保留 rowid 最小
|
||||||
|
的首行(对齐 canonical「每 question_id 取第一行 ORDER BY rowid」口径,
|
||||||
|
902→900)。_runs 表不受影响。
|
||||||
|
|
||||||
异常:
|
异常:
|
||||||
RuntimeError: 源中无该表或无该 run 的行。
|
RuntimeError: 源中无该表或无该 run 的行。
|
||||||
@@ -294,6 +310,14 @@ def extract_run_db(src_db: Path, dst_db: Path, run_id: str) -> None:
|
|||||||
dst.execute(create_sql[0])
|
dst.execute(create_sql[0])
|
||||||
cols = [r[1] for r in src.execute(f"PRAGMA table_info({table})")]
|
cols = [r[1] for r in src.execute(f"PRAGMA table_info({table})")]
|
||||||
col_sql = ", ".join(cols)
|
col_sql = ", ".join(cols)
|
||||||
|
if table == "predictions" and dedupe_per_question:
|
||||||
|
rows = src.execute(
|
||||||
|
f"SELECT {col_sql} FROM {table} WHERE run_id=? "
|
||||||
|
"AND rowid IN (SELECT MIN(rowid) FROM predictions "
|
||||||
|
"WHERE run_id=? GROUP BY question_id)",
|
||||||
|
(run_id, run_id),
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
rows = src.execute(
|
rows = src.execute(
|
||||||
f"SELECT {col_sql} FROM {table} WHERE run_id=?", (run_id,)
|
f"SELECT {col_sql} FROM {table} WHERE run_id=?", (run_id,)
|
||||||
).fetchall()
|
).fetchall()
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""诊断侧树读取适配:把嵌套 tree.json 展平成诊断消费的扁平 nodes dict。
|
||||||
|
|
||||||
|
诊断编排(core/evolution/diagnose.py)期望 tree_data 形如
|
||||||
|
{"nodes": {node_id: {card, level, time_range}}},但 TRM5 建树产物
|
||||||
|
store/videos/<vid>/tree.json 是嵌套 {"metadata","roots":[...]}。本模块递归展平,
|
||||||
|
接通 TRM4→TRM5 迁移时断掉的 ground_truth 加载环。
|
||||||
|
|
||||||
|
不走 TreeIndex 对象层:仅 L1Node 有 to_dict(app/tree/index.py:260),L2/L3 为其内部闭包,
|
||||||
|
且 to_dict 输出无 level、L3 用 timestamp 无 time_range。直接遍历 json 更省且零改建树模块。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def load_tree_nodes(store_dir: Path, video_id: str) -> dict[str, Any]:
|
||||||
|
"""加载单视频 tree.json 并展平成扁平 nodes dict。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
store_dir: store 根目录(含 videos/<video_id>/tree.json)。
|
||||||
|
video_id: 视频标识。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{"nodes": {node_id: {"card": dict, "level": int, "time_range": list}}}。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
FileNotFoundError: tree.json 不存在(沿用 factory.py fail-loud 先例)。
|
||||||
|
ValueError: roots 非 list 或为空、节点缺 id、或节点既无 time_range 又无 timestamp。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
level 由遍历深度赋值(root=1/child=2/孙=3),不解析 node_id——node_id 累积式
|
||||||
|
(..._L1_..._L2_..._L3_)用正则首匹配会把 L2/L3 误判成 1。
|
||||||
|
L3 无 time_range,用 timestamp 合成 [t, t]。
|
||||||
|
"""
|
||||||
|
tree_path = store_dir / "videos" / video_id / "tree.json"
|
||||||
|
if not tree_path.exists():
|
||||||
|
raise FileNotFoundError(f"树索引文件不存在: {tree_path}(诊断需真实树,P5 fail loud)")
|
||||||
|
tree = json.loads(tree_path.read_text(encoding="utf-8"))
|
||||||
|
roots = tree.get("roots")
|
||||||
|
if not isinstance(roots, list) or not roots:
|
||||||
|
raise ValueError(f"树无有效 roots: {tree_path}")
|
||||||
|
|
||||||
|
nodes: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def _walk(node: dict[str, Any], level: int) -> None:
|
||||||
|
node_id = node.get("id")
|
||||||
|
if not isinstance(node_id, str) or not node_id:
|
||||||
|
raise ValueError(f"节点缺 id: {tree_path}")
|
||||||
|
time_range = node.get("time_range")
|
||||||
|
if time_range is None:
|
||||||
|
ts = node.get("timestamp")
|
||||||
|
if ts is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"节点既无 time_range 又无 timestamp(树损坏): {node_id} in {tree_path}"
|
||||||
|
)
|
||||||
|
time_range = [ts, ts]
|
||||||
|
nodes[node_id] = {
|
||||||
|
"card": node.get("card", {}),
|
||||||
|
"level": level,
|
||||||
|
"time_range": time_range,
|
||||||
|
}
|
||||||
|
for child in node.get("children", []) or []:
|
||||||
|
_walk(child, level + 1)
|
||||||
|
|
||||||
|
for root in roots:
|
||||||
|
_walk(root, 1)
|
||||||
|
|
||||||
|
return {"nodes": nodes}
|
||||||
|
|
||||||
|
|
||||||
|
def load_tree_data_for_videos(store_dir: Path, video_ids: list[str]) -> dict[str, Any]:
|
||||||
|
"""按一组 video_id 去重加载展平树,供诊断按 video 注入。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
store_dir: store 根目录。
|
||||||
|
video_ids: 视频标识列表(可含重复,内部按首次出现顺序去重)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{video_id: {"nodes": {...}}}。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
同 load_tree_nodes(任一视频树缺失/无效即 fail-loud)。
|
||||||
|
"""
|
||||||
|
return {vid: load_tree_nodes(store_dir, vid) for vid in dict.fromkeys(video_ids)}
|
||||||
+606
-338
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,824 @@
|
|||||||
|
"""结果驱动视频级切分的自包含两阶段 CLI 入口。
|
||||||
|
|
||||||
|
把整条离线管线的编排从 shell 搬进 Python:一次调用内联串起
|
||||||
|
Phase 1 离线诊断(run_baseline_diagnosis,LLM 重活,断点续跑幂等)→
|
||||||
|
Phase 2 冻结切分(build_split,纯 code-controlled,产出 pools.json + manifest)→
|
||||||
|
McNemar 功效护栏(validation 池错题数达阈校验)。
|
||||||
|
|
||||||
|
复现锚点约定(C-2):
|
||||||
|
- pools.json 的内容(+ seed + diag_fingerprint)是切分的**复现锚点**——相同输入
|
||||||
|
产出字节级相同的 pools.json 与 pools_sha256。
|
||||||
|
- manifest 的 generated_at 是**溯源元数据**,非复现锚点:真实运行默认盖真实 UTC
|
||||||
|
now(记录本次切分何时产出),但可用 `--generated-at <ISO>` 显式固定,以对
|
||||||
|
manifest 做字节级复现比对。write_manifest 库内不调 datetime.now,时间戳一律由
|
||||||
|
本 CLI 传入。
|
||||||
|
|
||||||
|
设计要点:
|
||||||
|
- 诊断口径指纹 = (诊断 prompt 版本, 模型名, git 短 SHA) 三分量合成,隔离不同
|
||||||
|
诊断配置的信号;换 prompt / 模型 / 代码实现即换指纹,旧信号不被覆盖。
|
||||||
|
- 真实依赖组装参考 app/harness/runner.py::_run_diagnosis:GovernedLLMClient
|
||||||
|
(search llm, thinking=True) + RunLogImpl(harness.db) + VersionedSkillStore +
|
||||||
|
DiagnosePrompts(项目根 prompts/) + tree_data 按 wrong_ids 涉及 video 预加载
|
||||||
|
(store/videos/<vid>/tree.json 展平)。
|
||||||
|
- 缺 .env / config 关键项一律 fail loud(P5),绝不静默兜底。
|
||||||
|
- `--dry-run` 用假 deps 跑通两阶段 wiring 不真调 LLM,打印将执行的步骤 + 指纹,
|
||||||
|
用于校验装配正确性(对齐 CLAUDE.md §2.5 smoke test)。
|
||||||
|
|
||||||
|
编排函数(run_pipeline)通过依赖注入接收 DiagnosisDeps / signal_store / wrong_ids /
|
||||||
|
questions / canonical_preds,便于单测用假实现替换、不触真实 LLM 与 harness.db。
|
||||||
|
其中 canonical_preds 供 Phase 0 补 INFRA / 空预测错题的 T0 信号(这些题不进诊断)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import datetime
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from app.harness.baseline_diagnosis import DiagnosisDeps, run_baseline_diagnosis
|
||||||
|
from app.harness.build_split import (
|
||||||
|
SplitBuildConfig,
|
||||||
|
SplitBuildResult,
|
||||||
|
build_split,
|
||||||
|
load_canonical_predictions,
|
||||||
|
)
|
||||||
|
from app.harness.split_selection import diag_fingerprint
|
||||||
|
from app.question_gen.loader import load_benchmark
|
||||||
|
from core.evolution.types import DiagnosisSignalRow
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from app.harness.pools import Pools
|
||||||
|
from core.evolution.protocols import DiagnosisSignalStore
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
# 与 core.evolution.diagnose._INFRA_STOP_REASONS 对齐:执行/解析层失败排除出可诊断错题。
|
||||||
|
_INFRA_STOP_REASONS: frozenset[str] = frozenset({"error", "parse_error"})
|
||||||
|
|
||||||
|
# 工程路径默认值(少变;可经 CLI 单次覆盖)。诊断信号表建在 harness.db。
|
||||||
|
_DEFAULT_HARNESS_DB = Path("workspaces/default/harness.db")
|
||||||
|
_DEFAULT_QUESTIONS_DIR = Path("store/questions/benchmarks/Video-MME")
|
||||||
|
_DEFAULT_OUT_DIR = Path("workspaces/video-split")
|
||||||
|
_DEFAULT_STORE_DIR = Path("store") # tree.json 在 store/videos/<vid>/
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 配置解析(fail loud)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class VideoSplitConfig:
|
||||||
|
"""结果驱动视频级切分的科研旋钮快照(从 config/video_split.yaml 解析)。
|
||||||
|
|
||||||
|
字段:
|
||||||
|
baseline_run_id: 基线 run 标识(错题诊断与切分依据)。
|
||||||
|
n_trainval: trainval 目标视频数(多样性阶段填充上限)。
|
||||||
|
epsilon: test 相对全局最大允许分布偏差(题型 / 难度两维)。
|
||||||
|
report_floor: per-type 报告门限,题数 ≥ 此值的 task_type 才入 ε 约束。
|
||||||
|
val_wrong_min: validation 池最少错题数(McNemar 功效阈;0=不检查)。
|
||||||
|
val_ratio: validation 占 trainval 视频组总数的比例。
|
||||||
|
seed: 贪心选择器预洗牌 + 视频组题级切分种子。
|
||||||
|
floor_k: 各高信号 task_type 的 T2 defect 下限(硬约束)。
|
||||||
|
prompt_version: 诊断 prompt 版本标识(指纹分量)。
|
||||||
|
model: 执行诊断的模型名(指纹分量)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
baseline_run_id: str
|
||||||
|
n_trainval: int
|
||||||
|
epsilon: float
|
||||||
|
report_floor: int
|
||||||
|
val_wrong_min: int
|
||||||
|
val_ratio: float
|
||||||
|
seed: int
|
||||||
|
floor_k: dict[str, int]
|
||||||
|
prompt_version: str
|
||||||
|
model: str
|
||||||
|
|
||||||
|
|
||||||
|
def _require(section: dict[str, Any], keys: tuple[str, ...], where: str) -> None:
|
||||||
|
"""校验 section 含全部必填键,缺任一即 fail loud(P5,不静默兜底)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
section: 待校验的配置子字典。
|
||||||
|
keys: 必填键元组。
|
||||||
|
where: 出错信息中标注的段名(如 "video_split")。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: 存在缺失键。
|
||||||
|
"""
|
||||||
|
missing = [k for k in keys if k not in section]
|
||||||
|
if missing:
|
||||||
|
raise SystemExit(f"config {where} 段缺关键项 {missing},无法运行(P5 fail loud)")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_config(raw: dict[str, Any]) -> VideoSplitConfig:
|
||||||
|
"""把 yaml 原始字典解析为 VideoSplitConfig,缺关键项 fail loud。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
raw: yaml.safe_load 的顶层字典,需含 video_split / diag 两段。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
VideoSplitConfig 冻结快照。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: 缺 video_split / diag 段或段内关键项。
|
||||||
|
"""
|
||||||
|
if "video_split" not in raw or "diag" not in raw:
|
||||||
|
raise SystemExit("config 缺 video_split / diag 段,无法运行(P5 fail loud)")
|
||||||
|
vs = raw["video_split"]
|
||||||
|
dg = raw["diag"]
|
||||||
|
_require(
|
||||||
|
vs,
|
||||||
|
(
|
||||||
|
"baseline_run_id",
|
||||||
|
"n_trainval",
|
||||||
|
"epsilon",
|
||||||
|
"report_floor",
|
||||||
|
"val_wrong_min",
|
||||||
|
"val_ratio",
|
||||||
|
"seed",
|
||||||
|
"floor_k",
|
||||||
|
),
|
||||||
|
"video_split",
|
||||||
|
)
|
||||||
|
_require(dg, ("prompt_version", "model"), "diag")
|
||||||
|
return VideoSplitConfig(
|
||||||
|
baseline_run_id=vs["baseline_run_id"],
|
||||||
|
n_trainval=vs["n_trainval"],
|
||||||
|
epsilon=vs["epsilon"],
|
||||||
|
report_floor=vs["report_floor"],
|
||||||
|
val_wrong_min=vs["val_wrong_min"],
|
||||||
|
val_ratio=vs["val_ratio"],
|
||||||
|
seed=vs["seed"],
|
||||||
|
floor_k=dict(vs["floor_k"]),
|
||||||
|
prompt_version=dg["prompt_version"],
|
||||||
|
model=dg["model"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(config_path: Path) -> VideoSplitConfig:
|
||||||
|
"""读取并解析 video_split yaml 配置文件(缺文件 / 关键项 fail loud)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
config_path: yaml 配置路径。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
VideoSplitConfig。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: 文件不存在或缺关键项。
|
||||||
|
"""
|
||||||
|
if not config_path.exists():
|
||||||
|
raise SystemExit(f"config 文件不存在: {config_path}(P5 fail loud)")
|
||||||
|
raw = yaml.safe_load(config_path.read_text(encoding="utf-8"))
|
||||||
|
return parse_config(raw)
|
||||||
|
|
||||||
|
|
||||||
|
def git_short_sha() -> str:
|
||||||
|
"""取当前 git 短 SHA 作为诊断口径指纹的代码分量(诊断代码变则指纹变)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
git rev-parse --short HEAD 输出(去空白)。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: 非 git 仓库或 git 不可用(fail loud,指纹不可缺分量)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
out = subprocess.run(
|
||||||
|
["git", "rev-parse", "--short", "HEAD"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
except (subprocess.CalledProcessError, FileNotFoundError) as exc:
|
||||||
|
raise SystemExit(f"无法获取 git 短 SHA 作为诊断代码版本: {exc}(P5 fail loud)") from exc
|
||||||
|
sha = out.stdout.strip()
|
||||||
|
if not sha:
|
||||||
|
raise SystemExit("git rev-parse --short HEAD 返回空,诊断指纹缺代码分量(P5 fail loud)")
|
||||||
|
return sha
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 真实依赖组装(参考 runner.py::_run_diagnosis)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class _DiagLLMSettings:
|
||||||
|
"""诊断 LLM 的工程配置(从 .env 读取 search llm 凭证 + 韧性旋钮)。
|
||||||
|
|
||||||
|
仅承载诊断所需字段(搜索 LLM = 诊断 judge),不复用 main.InfraSettings 以免
|
||||||
|
构造整套适配器(embed / vlm)的重活;缺关键凭证 fail loud。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
class _Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||||
|
|
||||||
|
search_llm_model: str = ""
|
||||||
|
search_llm_base_url: str = ""
|
||||||
|
search_llm_api_key: str = ""
|
||||||
|
redis_url: str = ""
|
||||||
|
redis_cache_ttl: int = 86400
|
||||||
|
llm_timeout: float = 300.0
|
||||||
|
llm_max_retries: int = 3
|
||||||
|
llm_retry_base_delay: float = 20.0
|
||||||
|
llm_retry_max_delay: float = 120.0
|
||||||
|
llm_circuit_breaker_threshold: int = 48
|
||||||
|
llm_circuit_breaker_cooldown: float = 60.0
|
||||||
|
llm_ttft_timeout: float = 30.0
|
||||||
|
llm_inter_token_timeout: float = 15.0
|
||||||
|
|
||||||
|
self._s = _Settings()
|
||||||
|
|
||||||
|
def __getattr__(self, name: str) -> Any:
|
||||||
|
return getattr(self._s, name)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_redis_cache(settings: Any) -> Any | None:
|
||||||
|
"""按 .env redis_url 构建响应缓存(不可用则降级 None,与 main 一致)。"""
|
||||||
|
if not settings.redis_url:
|
||||||
|
return None
|
||||||
|
|
||||||
|
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
|
||||||
|
|
||||||
|
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
|
||||||
|
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
|
||||||
|
try:
|
||||||
|
import redis.asyncio as aioredis
|
||||||
|
|
||||||
|
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
|
||||||
|
return RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
|
||||||
|
except Exception:
|
||||||
|
logger.warning("Redis 缓存不可用,诊断降级为无缓存模式")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def build_diagnosis_deps(
|
||||||
|
*,
|
||||||
|
harness_db: Path,
|
||||||
|
store_dir: Path,
|
||||||
|
video_ids: list[str],
|
||||||
|
concurrency: int,
|
||||||
|
expected_model: str,
|
||||||
|
) -> DiagnosisDeps:
|
||||||
|
"""组装 Phase 1 诊断的真实依赖束(GovernedLLMClient + RunLogImpl + prompts)。
|
||||||
|
|
||||||
|
与 runner.py::_run_diagnosis 对齐:search LLM(thinking=True)作诊断 judge,
|
||||||
|
RunLogImpl 只读读取 harness.db 的 predictions/traces,VersionedSkillStore 读技能,
|
||||||
|
DiagnosePrompts 从项目根 prompts/ 加载,tree_data 按 video_ids 从
|
||||||
|
store/videos/<vid>/tree.json 展平预加载(诊断需真实树,缺失即 fail-loud)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
harness_db: harness.db 路径(诊断读预测 + 信号落库同库)。
|
||||||
|
store_dir: store 根目录(含 videos/<vid>/tree.json)。
|
||||||
|
video_ids: wrong_ids 涉及的 video 标识列表(可含重复,内部去重加载树)。
|
||||||
|
concurrency: 诊断并发上限。
|
||||||
|
expected_model: config.diag.model(诊断口径指纹的模型分量)。必须与 .env
|
||||||
|
SEARCH_LLM_MODEL 一致——指纹里的 model 与实际诊断所用 model 不一致会让
|
||||||
|
信号以错误模型指纹落库,破坏可复现 / resume / 口径隔离,故此处 fail loud。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
DiagnosisDeps 冻结依赖束。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: .env 缺 search LLM 凭证(model / base_url / api_key 任一为空),
|
||||||
|
或 config.diag.model 与 .env SEARCH_LLM_MODEL 不一致(指纹漂移防护)。
|
||||||
|
FileNotFoundError / ValueError: video_ids 中任一 video 的 tree.json 缺失或
|
||||||
|
无效(load_tree_data_for_videos fail-loud,诊断需真实树)。
|
||||||
|
"""
|
||||||
|
from adapters.breaker import CircuitBreaker
|
||||||
|
from adapters.llm import GovernedLLMClient
|
||||||
|
from adapters.telemetry import SQLiteTelemetryRecorder
|
||||||
|
from app.harness.log import RunLogImpl
|
||||||
|
from app.harness.workspace import VersionedSkillStore
|
||||||
|
|
||||||
|
settings = _DiagLLMSettings()
|
||||||
|
if not (
|
||||||
|
settings.search_llm_model and settings.search_llm_base_url and settings.search_llm_api_key
|
||||||
|
):
|
||||||
|
raise SystemExit(
|
||||||
|
"诊断 LLM 凭证缺失:.env 需配置 SEARCH_LLM_MODEL / SEARCH_LLM_BASE_URL / "
|
||||||
|
"SEARCH_LLM_API_KEY(P5 fail loud,不静默兜底)"
|
||||||
|
)
|
||||||
|
if expected_model != settings.search_llm_model:
|
||||||
|
raise SystemExit(
|
||||||
|
"诊断模型指纹漂移:config.diag.model="
|
||||||
|
f"{expected_model!r} 与 .env SEARCH_LLM_MODEL={settings.search_llm_model!r} "
|
||||||
|
"不一致;指纹里的 model 必须等于实际诊断所用 model(P5 fail loud,"
|
||||||
|
"请对齐 config/video_split.yaml diag.model 与 .env SEARCH_LLM_MODEL)"
|
||||||
|
)
|
||||||
|
|
||||||
|
telemetry_db = Path("logs/telemetry.db")
|
||||||
|
telemetry_db.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
telemetry = SQLiteTelemetryRecorder(telemetry_db)
|
||||||
|
breaker = CircuitBreaker(
|
||||||
|
fail_threshold=max(settings.llm_circuit_breaker_threshold, 1),
|
||||||
|
cooldown_s=settings.llm_circuit_breaker_cooldown,
|
||||||
|
)
|
||||||
|
llm = GovernedLLMClient(
|
||||||
|
model=settings.search_llm_model,
|
||||||
|
base_url=settings.search_llm_base_url,
|
||||||
|
api_key=settings.search_llm_api_key,
|
||||||
|
provider=settings.search_llm_model.split("-")[0],
|
||||||
|
thinking=True,
|
||||||
|
breaker=breaker,
|
||||||
|
cache=_build_redis_cache(settings),
|
||||||
|
telemetry=telemetry,
|
||||||
|
timeout_s=settings.llm_timeout,
|
||||||
|
ttft_timeout_s=settings.llm_ttft_timeout,
|
||||||
|
inter_token_timeout_s=settings.llm_inter_token_timeout,
|
||||||
|
max_retries=settings.llm_max_retries,
|
||||||
|
retry_base_delay_s=settings.llm_retry_base_delay,
|
||||||
|
retry_max_delay_s=settings.llm_retry_max_delay,
|
||||||
|
)
|
||||||
|
from app.harness.tree_nodes import load_tree_data_for_videos
|
||||||
|
|
||||||
|
return DiagnosisDeps(
|
||||||
|
run_log=RunLogImpl(str(harness_db)),
|
||||||
|
llm=llm,
|
||||||
|
skill_store=VersionedSkillStore(_diagnosis_skills_dir()),
|
||||||
|
prompts=_load_diagnose_prompts(),
|
||||||
|
tree_data=load_tree_data_for_videos(store_dir, video_ids),
|
||||||
|
concurrency=concurrency,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _diagnosis_skills_dir() -> Path:
|
||||||
|
"""诊断用技能目录:种子 store 的当前技能版本(诊断读技能遵从判定)。
|
||||||
|
|
||||||
|
诊断只读技能内容判断"是否遵从技能",用 store 种子 v1 即可(与基线 run 一致)。
|
||||||
|
"""
|
||||||
|
return Path("store/skills/v1")
|
||||||
|
|
||||||
|
|
||||||
|
def _load_diagnose_prompts() -> Any:
|
||||||
|
"""加载诊断模板束(从项目根 prompts/ 读取;与 runner._load_diagnose_prompts 一致)。"""
|
||||||
|
from core.evolution.types import DiagnosePrompts
|
||||||
|
|
||||||
|
def _read(name: str) -> str:
|
||||||
|
p = Path("prompts") / name
|
||||||
|
if not p.exists():
|
||||||
|
raise FileNotFoundError(f"缺进化/诊断模板: {p}(请从 TRM4 迁移或检查 prompts/)")
|
||||||
|
return p.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
return DiagnosePrompts(
|
||||||
|
defect_vs_lapse=_read("defect_vs_lapse.md"),
|
||||||
|
reasoning_sub=_read("reasoning_sub.md"),
|
||||||
|
span_eval_system=_read("span_eval_system.md"),
|
||||||
|
missed_nodes=_read("missed_nodes.md"),
|
||||||
|
skill_adherence=_read("skill_adherence.md"),
|
||||||
|
confirmation_bias=_read("confirmation_bias.md"),
|
||||||
|
evidence_sufficiency=_read("evidence_sufficiency.md"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def select_diagnosable_wrong_ids(preds: list[dict]) -> list[str]:
|
||||||
|
"""从 canonical 预测筛出可诊断错题 question_id(保序)。
|
||||||
|
|
||||||
|
可诊断错题判据:预测非空 且 stop_reason 非 INFRA(error / parse_error)且
|
||||||
|
归一后预测 != 答案。INFRA / 空预测错题不进 wrong_ids——它们改由
|
||||||
|
persist_infra_t0_rows 直接落 T0(run_diagnosis 内部也会二次排除同类题)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
preds: load_canonical_predictions 产出的 canonical 预测行(已按 qid 去重)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
可诊断错题 question_id 列表(保 preds 顺序)。
|
||||||
|
"""
|
||||||
|
wrong_ids: list[str] = []
|
||||||
|
for pred in preds:
|
||||||
|
prediction = (pred["prediction"] or "").strip()
|
||||||
|
if not prediction or pred["stop_reason"] in _INFRA_STOP_REASONS:
|
||||||
|
continue
|
||||||
|
if not pred["correct"]:
|
||||||
|
wrong_ids.append(pred["question_id"])
|
||||||
|
return wrong_ids
|
||||||
|
|
||||||
|
|
||||||
|
def persist_infra_t0_rows(
|
||||||
|
store: DiagnosisSignalStore,
|
||||||
|
preds: list[dict],
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
) -> int:
|
||||||
|
"""把非正确且 INFRA / 空预测的错题以 T0 信号行 upsert 落库(幂等)。
|
||||||
|
|
||||||
|
这些题(stop_reason ∈ {error, parse_error} 或预测为空)从不进入 run_diagnosis
|
||||||
|
(筛选时被前置排除),故其 T0 信号必须在此单独补齐——否则 signal store 缺这些行,
|
||||||
|
tier 分布 / manifest 不完整(计划要求 4 个 INFRA 空预测错题 → T0)。
|
||||||
|
|
||||||
|
投影口径与 baseline_diagnosis 的 INFRA 投影一致:infra=True、tier="T0"、
|
||||||
|
error_type / cause_category / evolution_target 均 None、degraded=False;
|
||||||
|
video_id / task_type 从 canonical 预测取。store.upsert 按主键
|
||||||
|
(question_id, baseline_run_id, diag_fingerprint) 幂等,重复调用零副作用。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
store: 诊断信号存储端口(与诊断落库同一 store)。
|
||||||
|
preds: load_canonical_predictions 产出的 canonical 预测行。
|
||||||
|
baseline_run_id: 基线 run 标识(信号行主键之一)。
|
||||||
|
diag_fingerprint: 诊断口径指纹(信号行主键之一)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
落库的 T0 行数(供日志)。
|
||||||
|
"""
|
||||||
|
count = 0
|
||||||
|
for pred in preds:
|
||||||
|
prediction = (pred["prediction"] or "").strip()
|
||||||
|
is_infra_or_empty = pred["stop_reason"] in _INFRA_STOP_REASONS or not prediction
|
||||||
|
if pred["correct"] or not is_infra_or_empty:
|
||||||
|
continue
|
||||||
|
store.upsert(
|
||||||
|
DiagnosisSignalRow(
|
||||||
|
question_id=pred["question_id"],
|
||||||
|
video_id=pred["video_id"],
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
diag_fingerprint=diag_fingerprint,
|
||||||
|
task_type=pred["task_type"],
|
||||||
|
error_type=None,
|
||||||
|
cause_category=None,
|
||||||
|
tier="T0",
|
||||||
|
evolution_target=None,
|
||||||
|
degraded=False,
|
||||||
|
infra=True,
|
||||||
|
session_id=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
count += 1
|
||||||
|
return count
|
||||||
|
|
||||||
|
|
||||||
|
def load_questions_by_id(questions_dir: Path) -> dict[str, GeneratedQuestion]:
|
||||||
|
"""加载 benchmark 全部题并建 question_id → GeneratedQuestion 映射。
|
||||||
|
|
||||||
|
覆盖 wrong_ids 与 run_diagnosis 返回的全部 infra/degraded 题(取 video_id/task_type)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
questions_dir: benchmark 题库目录。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
question_id → GeneratedQuestion 映射。
|
||||||
|
"""
|
||||||
|
return {q.question_id: q for q in load_benchmark(questions_dir)}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# McNemar 功效护栏
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def check_mcnemar_power(pools: Pools, val_wrong_min: int) -> int:
|
||||||
|
"""校验 validation 池错题数达 McNemar 功效阈,不足即 fail loud。
|
||||||
|
|
||||||
|
val_wrong_min 已前置到 build_split 内的切分保证功效(不足即从 diag 换入低 T2
|
||||||
|
错题组补足,耗尽 fail-loud);本函数作切分冻结后的冗余最终确认:val 错题数 < 阈
|
||||||
|
→ 验证信号不足以支撑可靠比较。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
pools: 冻结三池(含 validation 与 correctness)。
|
||||||
|
val_wrong_min: 最少错题数阈(0 = 不检查)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
validation 池实际错题数(供日志)。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
SystemExit: val_wrong_min > 0 且 val 错题数 < 阈(P5 fail loud,不静默放行)。
|
||||||
|
"""
|
||||||
|
val_wrong = sum(1 for q in pools.validation if not pools.correctness[q.question_id])
|
||||||
|
if val_wrong_min > 0 and val_wrong < val_wrong_min:
|
||||||
|
raise SystemExit(
|
||||||
|
f"validation 池错题数 {val_wrong} < val_wrong_min={val_wrong_min},"
|
||||||
|
"验证信号不足以支撑可靠比较(McNemar 检验功效不够)。"
|
||||||
|
"请放大 val_ratio / 调整旋钮后重跑,勿静默放行。"
|
||||||
|
)
|
||||||
|
return val_wrong
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 两阶段编排(依赖注入,便于单测)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def run_pipeline(
|
||||||
|
*,
|
||||||
|
config: VideoSplitConfig,
|
||||||
|
fingerprint: str,
|
||||||
|
diagnosis_deps: DiagnosisDeps,
|
||||||
|
signal_store: DiagnosisSignalStore,
|
||||||
|
wrong_ids: list[str],
|
||||||
|
questions: dict[str, GeneratedQuestion],
|
||||||
|
canonical_preds: list[dict],
|
||||||
|
harness_db: Path,
|
||||||
|
questions_dir: Path,
|
||||||
|
out_dir: Path,
|
||||||
|
generated_at: str,
|
||||||
|
force: bool = False,
|
||||||
|
retry_uncertain: bool = False,
|
||||||
|
) -> SplitBuildResult:
|
||||||
|
"""内联三阶段:Phase 0 INFRA T0 补录 → Phase 1 诊断 → Phase 2 冻结切分 → McNemar 护栏。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
config: 科研旋钮快照。
|
||||||
|
fingerprint: 诊断口径指纹(已合成,作诊断信号主键之一)。
|
||||||
|
diagnosis_deps: Phase 1 诊断依赖束(真实或假实现)。
|
||||||
|
signal_store: 诊断信号存储端口(Phase 0/1 写、Phase 2 读)。
|
||||||
|
wrong_ids: 待诊断的可诊断错题 question_id 列表。
|
||||||
|
questions: question_id → GeneratedQuestion 映射。
|
||||||
|
canonical_preds: canonical 预测行(Phase 0 从中筛 INFRA / 空预测错题补 T0)。
|
||||||
|
harness_db: harness.db 路径(Phase 2 读 canonical 预测)。
|
||||||
|
questions_dir: benchmark 题库目录(Phase 2 加载题库切池)。
|
||||||
|
out_dir: 冻结产物目录(pools.json + split_manifest.json)。
|
||||||
|
generated_at: 生成时间戳(ISO 字符串,由调用方传入;见模块 C-2 复现锚点约定)。
|
||||||
|
force: 覆盖已存在冻结产物开关,透传给 build_split(False 时已存在即报错)。
|
||||||
|
retry_uncertain: 透传给 run_baseline_diagnosis,令已落 uncertain 题被重新诊断。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
SplitBuildResult(冻结三池 + manifest + assignment)。
|
||||||
|
"""
|
||||||
|
# Phase 0: INFRA / 空预测错题补 T0(这些题不进诊断,须单独落库保证 tier 分布/manifest 完整)。
|
||||||
|
n_t0 = persist_infra_t0_rows(signal_store, canonical_preds, config.baseline_run_id, fingerprint)
|
||||||
|
logger.info("Phase 0 INFRA T0 补录:落库 {} 行(INFRA / 空预测错题不进诊断)", n_t0)
|
||||||
|
|
||||||
|
# Phase 1: 离线诊断(断点续跑幂等:done_question_ids 已完成题跳过)。
|
||||||
|
logger.info(
|
||||||
|
"Phase 1 离线诊断:baseline={} 待诊断错题 {} 题", config.baseline_run_id, len(wrong_ids)
|
||||||
|
)
|
||||||
|
await run_baseline_diagnosis(
|
||||||
|
baseline_run_id=config.baseline_run_id,
|
||||||
|
diag_fingerprint=fingerprint,
|
||||||
|
wrong_ids=wrong_ids,
|
||||||
|
questions=questions,
|
||||||
|
store=signal_store,
|
||||||
|
deps=diagnosis_deps,
|
||||||
|
retry_uncertain=retry_uncertain,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 2: 冻结切分(读诊断信号 → 贪心选择 → 视频组原子切三池 → 冻结 + 六条断言)。
|
||||||
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
logger.info("Phase 2 冻结切分:out={}", out_dir)
|
||||||
|
result = build_split(
|
||||||
|
db_path=harness_db,
|
||||||
|
baseline_run_id=config.baseline_run_id,
|
||||||
|
signal_store=signal_store,
|
||||||
|
diag_fingerprint=fingerprint,
|
||||||
|
questions_dir=questions_dir,
|
||||||
|
config=SplitBuildConfig(
|
||||||
|
n_trainval=config.n_trainval,
|
||||||
|
floor_k=config.floor_k,
|
||||||
|
epsilon=config.epsilon,
|
||||||
|
report_floor=config.report_floor,
|
||||||
|
select_seed=config.seed,
|
||||||
|
val_ratio=config.val_ratio,
|
||||||
|
split_seed=config.seed,
|
||||||
|
val_wrong_min=config.val_wrong_min,
|
||||||
|
),
|
||||||
|
out_path=out_dir / "pools.json",
|
||||||
|
manifest_path=out_dir / "split_manifest.json",
|
||||||
|
generated_at=generated_at,
|
||||||
|
force=force,
|
||||||
|
)
|
||||||
|
|
||||||
|
# McNemar 功效护栏(build_split 契约外的 capstone 层校验)。
|
||||||
|
val_wrong = check_mcnemar_power(result.pools, config.val_wrong_min)
|
||||||
|
logger.info(
|
||||||
|
"切分冻结完成:pools={} manifest={} val错题={}/{}(阈)",
|
||||||
|
out_dir / "pools.json",
|
||||||
|
out_dir / "split_manifest.json",
|
||||||
|
val_wrong,
|
||||||
|
config.val_wrong_min,
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 真实执行 / dry-run 入口
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_paths(args: argparse.Namespace) -> tuple[Path, Path, Path, Path]:
|
||||||
|
"""解析 harness_db / questions_dir / out_dir / store_dir(CLI 覆盖默认工程路径)。"""
|
||||||
|
harness_db = args.harness_db or _DEFAULT_HARNESS_DB
|
||||||
|
questions_dir = args.questions_dir or _DEFAULT_QUESTIONS_DIR
|
||||||
|
out_dir = args.out_dir or _DEFAULT_OUT_DIR
|
||||||
|
store_dir = args.store_dir or _DEFAULT_STORE_DIR
|
||||||
|
return harness_db, questions_dir, out_dir, store_dir
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_real(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
|
||||||
|
"""真实执行两阶段管线:组装真实 deps、读错题、跑诊断 + 冻结切分。"""
|
||||||
|
harness_db, questions_dir, out_dir, store_dir = _resolve_paths(args)
|
||||||
|
if not harness_db.exists():
|
||||||
|
raise SystemExit(f"harness.db 不存在: {harness_db}(P5 fail loud)")
|
||||||
|
canonical_preds = load_canonical_predictions(harness_db, config.baseline_run_id)
|
||||||
|
wrong_ids = select_diagnosable_wrong_ids(canonical_preds)
|
||||||
|
questions = load_questions_by_id(questions_dir)
|
||||||
|
video_ids: list[str] = []
|
||||||
|
for qid in wrong_ids:
|
||||||
|
q = questions.get(qid)
|
||||||
|
if q is None:
|
||||||
|
raise SystemExit(
|
||||||
|
f"wrong_id {qid!r} 不在 questions_dir 题库中"
|
||||||
|
"(baseline predictions 与题库不匹配,P5 fail loud)"
|
||||||
|
)
|
||||||
|
video_ids.append(q.video_id)
|
||||||
|
deps = build_diagnosis_deps(
|
||||||
|
harness_db=harness_db,
|
||||||
|
store_dir=store_dir,
|
||||||
|
video_ids=video_ids,
|
||||||
|
concurrency=args.concurrency,
|
||||||
|
expected_model=config.model,
|
||||||
|
)
|
||||||
|
|
||||||
|
# generated_at:默认盖真实 UTC now(溯源用),--generated-at 可显式固定以复现(C-2)。
|
||||||
|
generated_at = args.generated_at or datetime.datetime.now(datetime.UTC).isoformat()
|
||||||
|
|
||||||
|
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||||
|
|
||||||
|
store = SqliteDiagnosisSignalStore(str(harness_db))
|
||||||
|
try:
|
||||||
|
asyncio.run(
|
||||||
|
run_pipeline(
|
||||||
|
config=config,
|
||||||
|
fingerprint=fingerprint,
|
||||||
|
diagnosis_deps=deps,
|
||||||
|
signal_store=store,
|
||||||
|
wrong_ids=wrong_ids,
|
||||||
|
questions=questions,
|
||||||
|
canonical_preds=canonical_preds,
|
||||||
|
harness_db=harness_db,
|
||||||
|
questions_dir=questions_dir,
|
||||||
|
out_dir=out_dir,
|
||||||
|
generated_at=generated_at,
|
||||||
|
force=args.force,
|
||||||
|
retry_uncertain=args.retry_uncertain,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
store.close()
|
||||||
|
|
||||||
|
|
||||||
|
class _DryRunLLM:
|
||||||
|
"""dry-run 假 LLM:被真实调用即报错,保证不真调 LLM。"""
|
||||||
|
|
||||||
|
async def complete(self, *args: Any, **kwargs: Any) -> Any:
|
||||||
|
raise AssertionError("dry-run 不应真调 LLM.complete")
|
||||||
|
|
||||||
|
|
||||||
|
class _DryRunLog:
|
||||||
|
"""dry-run 假 RunLog:predictions/traces 均返回空,诊断不真正执行。"""
|
||||||
|
|
||||||
|
async def get_predictions(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
|
||||||
|
return []
|
||||||
|
|
||||||
|
async def get_traces(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_dry_run(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
|
||||||
|
"""dry-run:用假 deps 跑通 Phase 1 wiring(空错题 → 诊断早返回),打印步骤 + 指纹。
|
||||||
|
|
||||||
|
Phase 2 build_split 需真实诊断信号方能冻结,dry-run 不真实冻结,仅打印其计划;
|
||||||
|
Phase 1 用空 wrong_ids 走 run_baseline_diagnosis 早返回路径,验证装配可调用而不触 LLM。
|
||||||
|
"""
|
||||||
|
harness_db, questions_dir, out_dir, _store_dir = _resolve_paths(args)
|
||||||
|
logger.info("=== dry-run:校验两阶段装配(不真调 LLM / 不冻结产物)===")
|
||||||
|
logger.info(
|
||||||
|
"诊断口径指纹 diag_fingerprint={} (prompt={} model={})",
|
||||||
|
fingerprint,
|
||||||
|
config.prompt_version,
|
||||||
|
config.model,
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"解析路径:harness_db={} questions_dir={} out_dir={}", harness_db, questions_dir, out_dir
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"旋钮:n_trainval={} epsilon={} report_floor={} val_ratio={} seed={} "
|
||||||
|
"val_wrong_min={} floor_k={}",
|
||||||
|
config.n_trainval,
|
||||||
|
config.epsilon,
|
||||||
|
config.report_floor,
|
||||||
|
config.val_ratio,
|
||||||
|
config.seed,
|
||||||
|
config.val_wrong_min,
|
||||||
|
config.floor_k,
|
||||||
|
)
|
||||||
|
|
||||||
|
fake_deps = DiagnosisDeps(
|
||||||
|
run_log=_DryRunLog(),
|
||||||
|
llm=_DryRunLLM(),
|
||||||
|
skill_store=object(),
|
||||||
|
prompts=object(),
|
||||||
|
tree_data={},
|
||||||
|
concurrency=args.concurrency,
|
||||||
|
)
|
||||||
|
|
||||||
|
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||||
|
|
||||||
|
dry_db = out_dir / "_dry_run_signals.db"
|
||||||
|
dry_db.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
store = SqliteDiagnosisSignalStore(str(dry_db))
|
||||||
|
try:
|
||||||
|
# Phase 0 装配:用一条假 INFRA 空预测走通 persist_infra_t0_rows(不触 LLM)。
|
||||||
|
fake_infra_preds = [
|
||||||
|
{
|
||||||
|
"question_id": "_dry_infra",
|
||||||
|
"video_id": "_dry_v",
|
||||||
|
"task_type": "Counting Problem",
|
||||||
|
"prediction": "",
|
||||||
|
"answer": "A",
|
||||||
|
"stop_reason": "error",
|
||||||
|
"correct": False,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
n_t0 = persist_infra_t0_rows(store, fake_infra_preds, config.baseline_run_id, fingerprint)
|
||||||
|
logger.info("Phase 0 装配 OK:persist_infra_t0_rows 落 {} 行 INFRA T0(假数据)", n_t0)
|
||||||
|
logger.info("Phase 1 装配 OK:run_baseline_diagnosis 以空错题走早返回路径(不触 LLM)")
|
||||||
|
asyncio.run(
|
||||||
|
run_baseline_diagnosis(
|
||||||
|
baseline_run_id=config.baseline_run_id,
|
||||||
|
diag_fingerprint=fingerprint,
|
||||||
|
wrong_ids=[],
|
||||||
|
questions={},
|
||||||
|
store=store,
|
||||||
|
deps=fake_deps,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
store.close()
|
||||||
|
dry_db.unlink(missing_ok=True)
|
||||||
|
logger.info(
|
||||||
|
"Phase 2 装配 OK:真实执行将调 build_split 冻结 pools.json + manifest(dry-run 跳过)"
|
||||||
|
)
|
||||||
|
logger.info("=== dry-run 通过:两阶段装配可调用,指纹已算出 ===")
|
||||||
|
|
||||||
|
|
||||||
|
def build_arg_parser() -> argparse.ArgumentParser:
|
||||||
|
"""构建 CLI 参数解析器。"""
|
||||||
|
parser = argparse.ArgumentParser(description="结果驱动视频级切分两阶段 CLI(诊断 → 冻结切分)")
|
||||||
|
parser.add_argument("--config", type=Path, default=Path("config/video_split.yaml"))
|
||||||
|
parser.add_argument("--dry-run", action="store_true", dest="dry_run")
|
||||||
|
parser.add_argument("--gpu", type=str, default=None, help="可选:设置 CUDA_VISIBLE_DEVICES")
|
||||||
|
parser.add_argument("--concurrency", type=int, default=8, help="诊断并发上限")
|
||||||
|
parser.add_argument("--harness-db", type=Path, default=None, dest="harness_db")
|
||||||
|
parser.add_argument("--questions-dir", type=Path, default=None, dest="questions_dir")
|
||||||
|
parser.add_argument("--out-dir", type=Path, default=None, dest="out_dir")
|
||||||
|
parser.add_argument("--store-dir", type=Path, default=None, dest="store_dir")
|
||||||
|
parser.add_argument(
|
||||||
|
"--generated-at",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
dest="generated_at",
|
||||||
|
help=(
|
||||||
|
"manifest generated_at 时间戳(ISO 字符串);默认盖真实 UTC now(溯源元数据)。"
|
||||||
|
"复现锚点是 pools.json 内容 + seed + fingerprint;generated_at 可显式传入以"
|
||||||
|
"对 manifest 做字节级复现比对。"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--force",
|
||||||
|
action="store_true",
|
||||||
|
help="覆盖已存在的冻结 pools.json/manifest(旧产物备份为 .bak.*)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--retry-uncertain",
|
||||||
|
action="store_true",
|
||||||
|
dest="retry_uncertain",
|
||||||
|
help="把已落 tier='uncertain'(信号不可信降级)的题重新诊断,而非当作已完成跳过",
|
||||||
|
)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> None:
|
||||||
|
"""CLI 入口:解析参数 → 载配置 → 算指纹 → dry-run 或真实两阶段执行。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
argv: 可选参数列表(默认 sys.argv[1:]),便于测试注入。
|
||||||
|
"""
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
args = build_arg_parser().parse_args(argv)
|
||||||
|
if args.gpu is not None:
|
||||||
|
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu
|
||||||
|
logger.info("CUDA_VISIBLE_DEVICES={}", args.gpu)
|
||||||
|
|
||||||
|
config = load_config(args.config)
|
||||||
|
fingerprint = diag_fingerprint(config.prompt_version, config.model, git_short_sha())
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
_execute_dry_run(config, fingerprint, args)
|
||||||
|
return
|
||||||
|
_execute_real(config, fingerprint, args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -64,6 +64,21 @@ def _now_iso() -> str:
|
|||||||
return datetime.now(UTC).isoformat()
|
return datetime.now(UTC).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def _atomic_write_json(path: Path, data: dict) -> None:
|
||||||
|
"""原子写 JSON:tmp + os.replace(对齐 checkpoint.py 范式,防半截损坏)。
|
||||||
|
|
||||||
|
先写同目录临时文件,再 os.replace 原子替换目标;替换阶段崩溃不会留下半截
|
||||||
|
JSON,原文件保持完好。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
path: 目标 JSON 文件路径。
|
||||||
|
data: 待序列化的字典。
|
||||||
|
"""
|
||||||
|
tmp = path.with_name(path.name + ".tmp")
|
||||||
|
tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Workspace 核心函数
|
# Workspace 核心函数
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -108,7 +123,7 @@ def _scaffold_workspace(
|
|||||||
},
|
},
|
||||||
"history": [],
|
"history": [],
|
||||||
}
|
}
|
||||||
(workspace_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2))
|
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||||
|
|
||||||
|
|
||||||
def init_workspace(
|
def init_workspace(
|
||||||
@@ -196,6 +211,13 @@ def init_workspace_from_seed(
|
|||||||
shutil.copytree(seed_dir / "prompts", workspace_dir / "prompts" / "v1")
|
shutil.copytree(seed_dir / "prompts", workspace_dir / "prompts" / "v1")
|
||||||
shutil.copy2(seed_dir / "baseline.db", workspace_dir / "harness.db")
|
shutil.copy2(seed_dir / "baseline.db", workspace_dir / "harness.db")
|
||||||
|
|
||||||
|
seed_pools = seed_dir / "pools.json"
|
||||||
|
if seed_pools.exists():
|
||||||
|
shutil.copy2(seed_pools, workspace_dir / "pools.json")
|
||||||
|
seed_manifest = seed_dir / "split_manifest.json"
|
||||||
|
if seed_manifest.exists():
|
||||||
|
shutil.copy2(seed_manifest, workspace_dir / "split_manifest.json")
|
||||||
|
|
||||||
logger.info("Workspace 从种子 '{}' 初始化完成: {}", seed_name, workspace_dir)
|
logger.info("Workspace 从种子 '{}' 初始化完成: {}", seed_name, workspace_dir)
|
||||||
return meta["baseline_run_id"]
|
return meta["baseline_run_id"]
|
||||||
|
|
||||||
@@ -279,7 +301,7 @@ def update_manifest(workspace_dir: Path, **version_updates: str) -> None:
|
|||||||
raise KeyError(f"无效的 manifest current 字段: {invalid}")
|
raise KeyError(f"无效的 manifest current 字段: {invalid}")
|
||||||
manifest = load_manifest(workspace_dir)
|
manifest = load_manifest(workspace_dir)
|
||||||
manifest["current"].update(version_updates)
|
manifest["current"].update(version_updates)
|
||||||
(workspace_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2))
|
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||||
|
|
||||||
|
|
||||||
def record_run(workspace_dir: Path, run_id: str) -> Path:
|
def record_run(workspace_dir: Path, run_id: str) -> Path:
|
||||||
@@ -308,9 +330,7 @@ def record_run(workspace_dir: Path, run_id: str) -> Path:
|
|||||||
"questions": current["questions"],
|
"questions": current["questions"],
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
(workspace_dir / "manifest.json").write_text(
|
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||||
json.dumps(manifest, ensure_ascii=False, indent=2)
|
|
||||||
)
|
|
||||||
|
|
||||||
run_dir = workspace_dir / "runs" / run_id
|
run_dir = workspace_dir / "runs" / run_id
|
||||||
# exist_ok:同 run_id 重跑时 run 目录已存在不应崩溃
|
# exist_ok:同 run_id 重跑时 run 目录已存在不应崩溃
|
||||||
@@ -362,7 +382,7 @@ def update_best(
|
|||||||
"run_id": run_id,
|
"run_id": run_id,
|
||||||
"epoch": epoch,
|
"epoch": epoch,
|
||||||
}
|
}
|
||||||
(workspace_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2))
|
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||||
logger.info("Best 已更新: val_acc={}, run={}, epoch={}", val_acc, run_id, epoch)
|
logger.info("Best 已更新: val_acc={}, run={}, epoch={}", val_acc, run_id, epoch)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -81,18 +81,11 @@ RETRIEVAL_FAMILY = QuestionFamilySpec(
|
|||||||
),
|
),
|
||||||
legal_task_types=frozenset(
|
legal_task_types=frozenset(
|
||||||
[
|
[
|
||||||
"Action Recognition",
|
|
||||||
"Action Reasoning",
|
|
||||||
"Action Prediction",
|
|
||||||
"Action Sequence",
|
|
||||||
"Object Recognition",
|
"Object Recognition",
|
||||||
"Object Reasoning",
|
"Object Reasoning",
|
||||||
"Object Interaction",
|
"Action Recognition",
|
||||||
"Scene Understanding",
|
"Attribute Perception",
|
||||||
"Event Reasoning",
|
"OCR Problems",
|
||||||
"Causal Reasoning",
|
|
||||||
"Temporal Reasoning",
|
|
||||||
"Spatial Reasoning",
|
|
||||||
]
|
]
|
||||||
),
|
),
|
||||||
leak_profile=LeakTestProfile(
|
leak_profile=LeakTestProfile(
|
||||||
@@ -116,10 +109,7 @@ REASONING_FAMILY = QuestionFamilySpec(
|
|||||||
[
|
[
|
||||||
"Action Reasoning",
|
"Action Reasoning",
|
||||||
"Object Reasoning",
|
"Object Reasoning",
|
||||||
"Event Reasoning",
|
"Information Synopsis",
|
||||||
"Causal Reasoning",
|
|
||||||
"Temporal Reasoning",
|
|
||||||
"Spatial Reasoning",
|
|
||||||
]
|
]
|
||||||
),
|
),
|
||||||
leak_profile=LeakTestProfile(
|
leak_profile=LeakTestProfile(
|
||||||
@@ -141,10 +131,10 @@ ENUMERATION_FAMILY = QuestionFamilySpec(
|
|||||||
),
|
),
|
||||||
legal_task_types=frozenset(
|
legal_task_types=frozenset(
|
||||||
[
|
[
|
||||||
"Action Sequence",
|
"Counting Problem",
|
||||||
"Object Recognition",
|
"Temporal Reasoning",
|
||||||
"Object Interaction",
|
"Temporal Perception",
|
||||||
"Scene Understanding",
|
"Information Synopsis",
|
||||||
]
|
]
|
||||||
),
|
),
|
||||||
leak_profile=LeakTestProfile(
|
leak_profile=LeakTestProfile(
|
||||||
@@ -166,10 +156,10 @@ VISUAL_FAMILY = QuestionFamilySpec(
|
|||||||
),
|
),
|
||||||
legal_task_types=frozenset(
|
legal_task_types=frozenset(
|
||||||
[
|
[
|
||||||
"Object Recognition",
|
"Attribute Perception",
|
||||||
"Scene Understanding",
|
"Counting Problem",
|
||||||
|
"OCR Problems",
|
||||||
"Action Recognition",
|
"Action Recognition",
|
||||||
"Spatial Reasoning",
|
|
||||||
]
|
]
|
||||||
),
|
),
|
||||||
leak_profile=LeakTestProfile(
|
leak_profile=LeakTestProfile(
|
||||||
@@ -191,9 +181,8 @@ SPATIAL_FAMILY = QuestionFamilySpec(
|
|||||||
),
|
),
|
||||||
legal_task_types=frozenset(
|
legal_task_types=frozenset(
|
||||||
[
|
[
|
||||||
|
"Spatial Perception",
|
||||||
"Spatial Reasoning",
|
"Spatial Reasoning",
|
||||||
"Object Interaction",
|
|
||||||
"Scene Understanding",
|
|
||||||
]
|
]
|
||||||
),
|
),
|
||||||
leak_profile=LeakTestProfile(
|
leak_profile=LeakTestProfile(
|
||||||
|
|||||||
+73
-41
@@ -15,6 +15,8 @@ from core.types import GeneratedQuestion
|
|||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from core.types import QuestionUnit
|
||||||
|
|
||||||
_LEGACY_DEFAULT_DIFFICULTY = "medium"
|
_LEGACY_DEFAULT_DIFFICULTY = "medium"
|
||||||
|
|
||||||
|
|
||||||
@@ -26,6 +28,11 @@ def load_benchmark(questions_dir: Path) -> list[GeneratedQuestion]:
|
|||||||
video_id),v2 生成题把多视频题目合并在单个 JSON 中(每条记录自带
|
video_id),v2 生成题把多视频题目合并在单个 JSON 中(每条记录自带
|
||||||
``video_id``),两种格式均兼容。
|
``video_id``),两种格式均兼容。
|
||||||
|
|
||||||
|
pair 契约字段(``pair_id`` / ``question_role`` / ``flip_axis`` / ``unit_id``)
|
||||||
|
用 ``.get`` 读取:旧 benchmark 无这些键时退化为 single(``question_role``
|
||||||
|
默认 "single",``unit_id`` 留空由 __post_init__ 回填为 question_id),
|
||||||
|
保证历史题库可无缝加载。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
questions_dir: 包含 *.json 文件的目录路径。
|
questions_dir: 包含 *.json 文件的目录路径。
|
||||||
|
|
||||||
@@ -52,6 +59,12 @@ def load_benchmark(questions_dir: Path) -> list[GeneratedQuestion]:
|
|||||||
skill_target=qa.get("skill_target"),
|
skill_target=qa.get("skill_target"),
|
||||||
difficulty_steps=qa.get("difficulty_steps"),
|
difficulty_steps=qa.get("difficulty_steps"),
|
||||||
sub_pattern=qa.get("sub_pattern"),
|
sub_pattern=qa.get("sub_pattern"),
|
||||||
|
# pair 契约字段:旧 benchmark 无这些键时按 single 默认兜底,
|
||||||
|
# unit_id 留空交由 GeneratedQuestion.__post_init__ 回填。
|
||||||
|
pair_id=qa.get("pair_id"),
|
||||||
|
question_role=qa.get("question_role", "single"),
|
||||||
|
flip_axis=qa.get("flip_axis"),
|
||||||
|
unit_id=qa.get("unit_id", ""),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
return results
|
return results
|
||||||
@@ -66,62 +79,83 @@ def stratified_sample(
|
|||||||
seed: int,
|
seed: int,
|
||||||
min_per_class: int | None,
|
min_per_class: int | None,
|
||||||
) -> list[GeneratedQuestion]:
|
) -> list[GeneratedQuestion]:
|
||||||
"""按题型过滤后采样 size 道题,可选按对错比例分层并按题型保底。
|
"""按题型过滤后采样 size 个单元,可选按对错比例分层并按题型保底。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
questions: 候选题目全集。
|
questions: 候选题目全集(single 与孪生对成员可混含)。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对(单元级正确性取成员 AND)。
|
||||||
size: 采样总量。
|
size: 采样单元总量(single 计 1、pair 计 1)。
|
||||||
correct_ratio: 采样中"基线答对"题的占比;None 表示自然分布。
|
correct_ratio: 采样中"基线答对"单元的占比;None 表示自然分布。
|
||||||
task_types: 限定题型;None 表示不限。
|
task_types: 限定题型;None 表示不限。
|
||||||
seed: 随机种子,保证可复现。
|
seed: 随机种子,保证可复现。
|
||||||
min_per_class: 每个题型补足到的下限;None 表示不补足。
|
min_per_class: 每个题型补足到的单元下限;None 表示不补足。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
采样后的题目列表。
|
采样后的题目列表(pair 单元展开为原始的两道题)。
|
||||||
|
|
||||||
异常:
|
异常:
|
||||||
ValueError: 自然分布时池不足 size,或分层时某层题目不足。
|
ValueError: 自然分布时单元池不足 size,或分层时某层单元不足。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
以 **QuestionUnit 为采样原子**(single 计 1、pair 计 1),size /
|
||||||
|
correct_ratio / min_per_class 均按 unit 计数,孪生对两题永不被劈开。
|
||||||
|
采样完成后 flatten_units 展开回逐题列表。纯 single 输入时 build_units
|
||||||
|
与题目一一对应、顺序不变,rng 消耗与旧逐题实现完全一致(字节级回归)。
|
||||||
|
|
||||||
|
build_units / flatten_units 采用函数内延迟导入:loader 属 question_gen,
|
||||||
|
question_units 属 harness,而 harness 包初始化会反向 import question_gen,
|
||||||
|
模块级导入将触发循环依赖(沿用 adversarial_filter 的既有做法)。
|
||||||
"""
|
"""
|
||||||
|
from app.harness.question_units import build_units, flatten_units
|
||||||
|
|
||||||
rng = random.Random(seed)
|
rng = random.Random(seed)
|
||||||
pool = [q for q in questions if task_types is None or q.task_type in task_types]
|
units = build_units(questions)
|
||||||
|
pool = [u for u in units if task_types is None or u.task_type in task_types]
|
||||||
|
|
||||||
if correct_ratio is None:
|
if correct_ratio is None:
|
||||||
if len(pool) < size:
|
if len(pool) < size:
|
||||||
raise ValueError(f"自然分布采样不足: 需 {size} 道, 实有 {len(pool)} 道")
|
raise ValueError(f"自然分布采样不足: 需 {size} 个单元, 实有 {len(pool)} 个")
|
||||||
sampled = rng.sample(pool, size)
|
sampled = rng.sample(pool, size)
|
||||||
else:
|
else:
|
||||||
sampled = _ratio_stratified_sample(pool, correctness, size, correct_ratio, rng)
|
sampled = _ratio_stratified_sample(pool, correctness, size, correct_ratio, rng)
|
||||||
|
|
||||||
if min_per_class is not None:
|
if min_per_class is not None:
|
||||||
sampled = _backfill_per_class(sampled, pool, min_per_class, rng)
|
sampled = _backfill_per_class(sampled, pool, min_per_class, rng)
|
||||||
return sampled
|
return flatten_units(sampled)
|
||||||
|
|
||||||
|
|
||||||
def _ratio_stratified_sample(
|
def _ratio_stratified_sample(
|
||||||
pool: list[GeneratedQuestion],
|
pool: list[QuestionUnit],
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
size: int,
|
size: int,
|
||||||
correct_ratio: float,
|
correct_ratio: float,
|
||||||
rng: random.Random,
|
rng: random.Random,
|
||||||
) -> list[GeneratedQuestion]:
|
) -> list[QuestionUnit]:
|
||||||
"""按对错比例分层采样:对题占 correct_ratio,其余为错题。
|
"""按对错比例分层采样:对单元占 correct_ratio,其余为错单元。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
pool: 题型过滤后的候选题。
|
pool: 题型过滤后的候选单元。
|
||||||
correctness: question_id -> 基线是否答对。
|
correctness: question_id -> 基线是否答对。
|
||||||
size: 采样总量。
|
size: 采样单元总量。
|
||||||
correct_ratio: 对题占比。
|
correct_ratio: 对单元占比。
|
||||||
rng: 随机数发生器。
|
rng: 随机数发生器。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
采样后的题目列表(对题在前、错题在后)。
|
采样后的单元列表(对单元在前、错单元在后)。
|
||||||
|
|
||||||
异常:
|
异常:
|
||||||
ValueError: 对题或错题层不足。
|
ValueError: 对单元或错单元层不足。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
unit_correctness 采用函数内延迟导入:loader 属 question_gen,
|
||||||
|
question_units 属 harness,模块级导入将触发循环依赖(沿用 build_units /
|
||||||
|
flatten_units 的既有做法)。以 strict=False 保持"缺基线对错即视为未答对"的
|
||||||
|
原 loose 语义不变。
|
||||||
"""
|
"""
|
||||||
correct = [q for q in pool if correctness.get(q.question_id, False)]
|
from app.harness.question_units import unit_correctness
|
||||||
wrong = [q for q in pool if not correctness.get(q.question_id, False)]
|
|
||||||
|
correct = [u for u in pool if unit_correctness(u, correctness, strict=False)]
|
||||||
|
wrong = [u for u in pool if not unit_correctness(u, correctness, strict=False)]
|
||||||
n_correct = round(size * correct_ratio)
|
n_correct = round(size * correct_ratio)
|
||||||
n_wrong = size - n_correct
|
n_wrong = size - n_correct
|
||||||
if len(correct) < n_correct or len(wrong) < n_wrong:
|
if len(correct) < n_correct or len(wrong) < n_wrong:
|
||||||
@@ -132,42 +166,40 @@ def _ratio_stratified_sample(
|
|||||||
|
|
||||||
|
|
||||||
def _backfill_per_class(
|
def _backfill_per_class(
|
||||||
sampled: list[GeneratedQuestion],
|
sampled: list[QuestionUnit],
|
||||||
pool: list[GeneratedQuestion],
|
pool: list[QuestionUnit],
|
||||||
min_per_class: int,
|
min_per_class: int,
|
||||||
rng: random.Random,
|
rng: random.Random,
|
||||||
) -> list[GeneratedQuestion]:
|
) -> list[QuestionUnit]:
|
||||||
"""对候选池中出现的每个题型,将采样结果补足到 min_per_class 道。
|
"""对候选池中出现的每个题型,将采样单元补足到 min_per_class 个。
|
||||||
|
|
||||||
遍历对象是候选池 pool 里出现的全部题型(非仅 sampled 命中的),
|
遍历对象是候选池 pool 里出现的全部题型(非仅 sampled 命中的),
|
||||||
保证任意稀疏题型都能拿到足额样本。
|
保证任意稀疏题型都能拿到足额样本。补足以 unit 为原子,孪生对整进整出。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
sampled: 主采样结果(不修改,返回新列表)。
|
sampled: 主采样结果单元(不修改,返回新列表)。
|
||||||
pool: 候选题全集(补足来源 + 题型枚举来源)。
|
pool: 候选单元全集(补足来源 + 题型枚举来源)。
|
||||||
min_per_class: 每个题型的下限。
|
min_per_class: 每个题型的单元下限。
|
||||||
rng: 随机数发生器。
|
rng: 随机数发生器。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
补足后的题目列表。
|
补足后的单元列表。
|
||||||
"""
|
"""
|
||||||
selected_ids = {q.question_id for q in sampled}
|
selected_ids = {u.unit_id for u in sampled}
|
||||||
result = list(sampled)
|
result = list(sampled)
|
||||||
counts: dict[str, int] = {}
|
counts: dict[str, int] = {}
|
||||||
for q in sampled:
|
for u in sampled:
|
||||||
counts[q.task_type] = counts.get(q.task_type, 0) + 1
|
counts[u.task_type] = counts.get(u.task_type, 0) + 1
|
||||||
ordered_task_types: dict[str, None] = {}
|
ordered_task_types: dict[str, None] = {}
|
||||||
for q in pool:
|
for u in pool:
|
||||||
ordered_task_types.setdefault(q.task_type, None)
|
ordered_task_types.setdefault(u.task_type, None)
|
||||||
for task_type in ordered_task_types:
|
for task_type in ordered_task_types:
|
||||||
deficit = min_per_class - counts.get(task_type, 0)
|
deficit = min_per_class - counts.get(task_type, 0)
|
||||||
if deficit <= 0:
|
if deficit <= 0:
|
||||||
continue
|
continue
|
||||||
candidates = [
|
candidates = [u for u in pool if u.task_type == task_type and u.unit_id not in selected_ids]
|
||||||
q for q in pool if q.task_type == task_type and q.question_id not in selected_ids
|
|
||||||
]
|
|
||||||
take = rng.sample(candidates, min(deficit, len(candidates)))
|
take = rng.sample(candidates, min(deficit, len(candidates)))
|
||||||
for q in take:
|
for u in take:
|
||||||
selected_ids.add(q.question_id)
|
selected_ids.add(u.unit_id)
|
||||||
result.append(q)
|
result.append(u)
|
||||||
return result
|
return result
|
||||||
|
|||||||
@@ -0,0 +1,196 @@
|
|||||||
|
"""accepted 题库的 pair 原子成对落盘 helper(纯件,不依赖 pipeline)。
|
||||||
|
|
||||||
|
三件可复用纯件供 Phase 2 新 pipeline 的 on_accept 回调复用:``PairPendingBuffer``
|
||||||
|
(按 pair_id 收齐才 emit 单元)、``write_accepted``(tmp + os.replace 原子写)、
|
||||||
|
``read_accepted``(聚合成单元并剔除磁盘悬挂孤儿)。
|
||||||
|
|
||||||
|
wiring 归属:本模块**只是纯件**,不接任何生成侧回调;真正的 on_accept wiring
|
||||||
|
**见 Phase 2**。当前项目真实 accepted 写入点是 ``adversarial_filter.write_final_bank``
|
||||||
|
(Phase 2 待重建的旧代码),本模块沿用其 tmp + os.replace 模式但不 import/不改动它。
|
||||||
|
|
||||||
|
unit_hash 校验:``QuestionUnit.unit_hash`` 目前恒为 ""(填充是 Phase 2 的事),且
|
||||||
|
``GeneratedQuestion`` 不携带 unit_hash 字段。故把"unit_hash 不一致→拒"落地为**同
|
||||||
|
pair_id 两成员的绑定一致性校验**(video_id / task_type / flip_axis),不一致即
|
||||||
|
fail-fast raise,语义等价——只有 payload 绑定一致的孪生对才允许聚合。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from collections import defaultdict
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def _check_pair_binding(first: GeneratedQuestion, second: GeneratedQuestion) -> None:
|
||||||
|
"""校验同 pair_id 两成员的绑定一致性,不一致 fail-fast(unit_hash 语义代偿)。
|
||||||
|
|
||||||
|
参数 first/second 为先后到达的孪生对成员。video_id / task_type / flip_axis 任一
|
||||||
|
不一致即 raise ValueError——绑定不一致的两条题目不构成同一 payload 的孪生对,
|
||||||
|
拒绝聚合而非静默兜底。
|
||||||
|
"""
|
||||||
|
mismatches: list[str] = []
|
||||||
|
if first.video_id != second.video_id:
|
||||||
|
mismatches.append(f"video_id: {first.video_id} != {second.video_id}")
|
||||||
|
if first.task_type != second.task_type:
|
||||||
|
mismatches.append(f"task_type: {first.task_type} != {second.task_type}")
|
||||||
|
if first.flip_axis != second.flip_axis:
|
||||||
|
mismatches.append(f"flip_axis: {first.flip_axis} != {second.flip_axis}")
|
||||||
|
if mismatches:
|
||||||
|
raise ValueError(
|
||||||
|
f"pair {first.pair_id} 两成员绑定不一致(" + ";".join(mismatches) + "),拒绝聚合"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class PairPendingBuffer:
|
||||||
|
"""按 pair_id 收齐孪生对才 emit 单元的有状态缓冲器。
|
||||||
|
|
||||||
|
喂题接口 ``add`` 逐条消费题目:single 立即 emit ``QuestionUnit.from_single``;
|
||||||
|
pair 成员先缓存,等同一 pair_id 的 original+mirror 都到齐才 emit 一个 pair 单元
|
||||||
|
(复用 ``QuestionUnit.from_pair`` 走 fail-fast 校验)。批次末尾用
|
||||||
|
``pending_orphans`` 检测"只落 P 未落 Q"的悬挂项。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
"""pending 以 pair_id 索引首个到达的孪生对成员,等伙伴到齐再 emit。"""
|
||||||
|
self._pending: dict[str, GeneratedQuestion] = {}
|
||||||
|
|
||||||
|
def add(self, q: GeneratedQuestion) -> QuestionUnit | None:
|
||||||
|
"""喂入一条题目 q,返回本次凑齐的单元或 None(pair 尚未配齐)。
|
||||||
|
|
||||||
|
single 立即返回 kind="single" 单元;pair 首个成员缓存并返回 None,第二个
|
||||||
|
成员到齐后返回 kind="pair" 单元。同 pair_id 两成员绑定不一致或角色非法(如
|
||||||
|
两个 original)即 raise ValueError。
|
||||||
|
|
||||||
|
关键实现:配齐后**先校验、成功组装出 unit 才从 pending 删除**——若
|
||||||
|
``_check_pair_binding`` / ``_order_pair`` raise,首成员仍留在 pending,调用方
|
||||||
|
``pending_orphans`` 可取回被拒的悬挂成员。绑定校验用显式 ValueError(不依赖
|
||||||
|
会被 ``-O`` 剥除的 assert)。
|
||||||
|
"""
|
||||||
|
if not q.pair_id:
|
||||||
|
return QuestionUnit.from_single(q)
|
||||||
|
|
||||||
|
partner = self._pending.get(q.pair_id)
|
||||||
|
if partner is None:
|
||||||
|
self._pending[q.pair_id] = q
|
||||||
|
return None
|
||||||
|
|
||||||
|
_check_pair_binding(partner, q)
|
||||||
|
original, mirror = _order_pair(partner, q)
|
||||||
|
unit = QuestionUnit.from_pair(original, mirror)
|
||||||
|
del self._pending[q.pair_id]
|
||||||
|
return unit
|
||||||
|
|
||||||
|
def pending_orphans(self) -> list[GeneratedQuestion]:
|
||||||
|
"""返回仍未配齐的悬挂成员,供调用方在批次末尾检测"只落 P 未落 Q"。"""
|
||||||
|
return list(self._pending.values())
|
||||||
|
|
||||||
|
|
||||||
|
def _order_pair(
|
||||||
|
a: GeneratedQuestion, b: GeneratedQuestion
|
||||||
|
) -> tuple[GeneratedQuestion, GeneratedQuestion]:
|
||||||
|
"""按 question_role 把两成员 a/b 定序为 (original, mirror)。
|
||||||
|
|
||||||
|
两成员不构成恰好 1 original + 1 mirror(角色缺失或重复)即 raise ValueError,
|
||||||
|
防 next(...) 静默 StopIteration。
|
||||||
|
"""
|
||||||
|
originals = [q for q in (a, b) if q.question_role == "pair_original"]
|
||||||
|
mirrors = [q for q in (a, b) if q.question_role == "pair_mirror"]
|
||||||
|
if len(originals) != 1 or len(mirrors) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
f"pair {a.pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)},"
|
||||||
|
"需各恰好 1 条"
|
||||||
|
)
|
||||||
|
return originals[0], mirrors[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_disk_pair(pair_id: str, group: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||||
|
"""三态判定磁盘上同 pair_id 分组,区分悬挂孤儿(drop)与结构损坏(raise)。
|
||||||
|
|
||||||
|
- 恰好 1 original + 1 mirror:合法孪生对,额外做 ``_check_pair_binding`` 显式绑定
|
||||||
|
校验(video_id/task_type/flip_axis,-O 下仍生效),返回其两题。
|
||||||
|
- size==1(只落 P 未落 Q):业务上合法的悬挂孤儿,warn + drop,返回 []。
|
||||||
|
- 其余(size>2 超员、或 size==2 角色重复/缺角色):数据损坏/外部篡改,按 P5
|
||||||
|
fail-loud,raise ValueError(含 pair_id 与成员构成),绝不静默吞。
|
||||||
|
"""
|
||||||
|
originals = sum(1 for q in group if q.question_role == "pair_original")
|
||||||
|
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
|
||||||
|
if len(group) == 2 and originals == 1 and mirrors == 1:
|
||||||
|
_check_pair_binding(group[0], group[1])
|
||||||
|
return group
|
||||||
|
if len(group) == 1:
|
||||||
|
logger.warning("磁盘悬挂孤儿 pair {}:仅 1 成员(缺伙伴),warn+drop 该 unit", pair_id)
|
||||||
|
return []
|
||||||
|
raise ValueError(
|
||||||
|
f"磁盘 pair {pair_id} 结构损坏:成员数={len(group)}"
|
||||||
|
f"(original={originals} mirror={mirrors}),需恰好 1 original + 1 mirror"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _keep_complete_disk_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||||
|
"""筛选磁盘读回的题目:single 全保留、pair 按 ``_validate_disk_pair`` 三态处理。
|
||||||
|
|
||||||
|
悬挂孤儿 warn+drop、结构损坏/绑定不一致 raise ValueError、合法成对保留后交给
|
||||||
|
build_units(single 全保留)。
|
||||||
|
"""
|
||||||
|
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||||
|
singles: list[GeneratedQuestion] = []
|
||||||
|
for q in questions:
|
||||||
|
if q.pair_id:
|
||||||
|
by_pair[q.pair_id].append(q)
|
||||||
|
else:
|
||||||
|
singles.append(q)
|
||||||
|
|
||||||
|
kept_pairs = [q for pid, grp in by_pair.items() for q in _validate_disk_pair(pid, grp)]
|
||||||
|
return singles + kept_pairs
|
||||||
|
|
||||||
|
|
||||||
|
def write_accepted(path: Path, units: list[QuestionUnit]) -> None:
|
||||||
|
"""把 units 全量原子写到 path(tmp + os.replace),孪生对两题相邻落盘。
|
||||||
|
|
||||||
|
parent 不存在则自动创建;任一 pair 单元结构非法(size≠2)落盘前 fail-fast raise。
|
||||||
|
|
||||||
|
关键实现:沿用 ``write_final_bank`` 的原子写模式(先写同目录 ``.tmp`` 再
|
||||||
|
``os.replace`` 覆盖,保证读到的 JSON 恒完整)。序列化复用 T9 pools.py 的
|
||||||
|
``_q_to_dict``(唯一 GeneratedQuestion↔dict schema,含 pair 四字段),函数内
|
||||||
|
import 规避 app.question_gen↔app.harness 循环依赖。
|
||||||
|
"""
|
||||||
|
from app.harness.pools import _q_to_dict
|
||||||
|
from app.harness.question_units import flatten_units, validate_units
|
||||||
|
|
||||||
|
validate_units(units)
|
||||||
|
records = [_q_to_dict(q) for q in flatten_units(units)]
|
||||||
|
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = path.with_suffix(".tmp")
|
||||||
|
tmp.write_text(json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
os.replace(str(tmp), str(path))
|
||||||
|
logger.info("accepted 题库全量原子写: {} 单元 / {} 题 → {}", len(units), len(records), path)
|
||||||
|
|
||||||
|
|
||||||
|
def read_accepted(path: Path) -> list[QuestionUnit]:
|
||||||
|
"""读回 path 的 accepted JSON → 聚合为单元列表。
|
||||||
|
|
||||||
|
磁盘是外部输入,按 P5 全量校验后再用:``_keep_complete_disk_pairs`` 对 pair 分组三态处理
|
||||||
|
——"只落 P 未落 Q"的悬挂孤儿 warn+drop(不进结果、不 raise);结构损坏(超员/
|
||||||
|
角色重复)或绑定不一致(video_id/task_type/flip_axis)显式 raise ValueError(不
|
||||||
|
依赖 build_units 内会被 ``-O`` 剥除的 assert)。single 全保留。
|
||||||
|
|
||||||
|
关键实现:反序列化复用 T9 pools.py 的 ``_dict_to_q``(pair 四字段 .get 兼容),
|
||||||
|
函数内 import 规避循环依赖。sift 后 ``build_units`` 聚合、``validate_units`` 二次
|
||||||
|
防御闸门。
|
||||||
|
"""
|
||||||
|
from app.harness.pools import _dict_to_q
|
||||||
|
from app.harness.question_units import build_units, validate_units
|
||||||
|
|
||||||
|
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
questions = [_dict_to_q(d) for d in raw]
|
||||||
|
kept = _keep_complete_disk_pairs(questions)
|
||||||
|
return validate_units(build_units(kept))
|
||||||
@@ -91,6 +91,7 @@ class RunStats:
|
|||||||
# DDL
|
# DDL
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# DEPRECATED(v3): v2 生成落库,Phase 2 生成替换后删;过渡期与 v3 表并存
|
||||||
_DDL_RUNS = """
|
_DDL_RUNS = """
|
||||||
CREATE TABLE IF NOT EXISTS question_gen_runs (
|
CREATE TABLE IF NOT EXISTS question_gen_runs (
|
||||||
run_id TEXT PRIMARY KEY,
|
run_id TEXT PRIMARY KEY,
|
||||||
@@ -106,6 +107,7 @@ CREATE TABLE IF NOT EXISTS question_gen_runs (
|
|||||||
);
|
);
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
# DEPRECATED(v3): v2 生成落库,Phase 2 生成替换后删;过渡期与 v3 表并存
|
||||||
_DDL_ITEMS = """
|
_DDL_ITEMS = """
|
||||||
CREATE TABLE IF NOT EXISTS question_gen_items (
|
CREATE TABLE IF NOT EXISTS question_gen_items (
|
||||||
item_id TEXT PRIMARY KEY,
|
item_id TEXT PRIMARY KEY,
|
||||||
@@ -139,6 +141,7 @@ _DDL_INDEXES = [
|
|||||||
"CREATE INDEX IF NOT EXISTS idx_qgi_task_type ON question_gen_items(task_type);",
|
"CREATE INDEX IF NOT EXISTS idx_qgi_task_type ON question_gen_items(task_type);",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# DEPRECATED(v3): v2 生成落库,Phase 2 生成替换后删;过渡期与 v3 表并存
|
||||||
_DDL_VERDICTS = """
|
_DDL_VERDICTS = """
|
||||||
CREATE TABLE IF NOT EXISTS adversarial_verdicts (
|
CREATE TABLE IF NOT EXISTS adversarial_verdicts (
|
||||||
question_id TEXT NOT NULL,
|
question_id TEXT NOT NULL,
|
||||||
@@ -162,6 +165,102 @@ _DDL_VERDICTS_INDEXES = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# v3 出题领域观测表 DDL(facts/unit_verdict/collapse_metrics/quarantine/resume_state)
|
||||||
|
#
|
||||||
|
# 时间戳边界(Codex I-4):v3 新表的 ts 列一律由 insert 方法外部传入,DDL 中不设
|
||||||
|
# DEFAULT (datetime('now'))、方法体不调 datetime.now(),以保证观测幂等可复现——
|
||||||
|
# 断点续跑重放时不因进程内时钟漂移产生脏数据。resume_state 无 ts 列(schema 未列)。
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_DDL_FACTS = """
|
||||||
|
CREATE TABLE IF NOT EXISTS facts (
|
||||||
|
fact_id TEXT PRIMARY KEY,
|
||||||
|
video_id TEXT,
|
||||||
|
segment_id TEXT,
|
||||||
|
subject TEXT,
|
||||||
|
action TEXT,
|
||||||
|
object TEXT,
|
||||||
|
frame_ids TEXT,
|
||||||
|
polarity TEXT,
|
||||||
|
fact_type TEXT,
|
||||||
|
difficulty_tier INTEGER,
|
||||||
|
verifier_refs TEXT,
|
||||||
|
cross_agree INTEGER CHECK(cross_agree IN (0, 1)),
|
||||||
|
negative_at_target TEXT,
|
||||||
|
session_id TEXT,
|
||||||
|
ts TEXT
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
_DDL_UNIT_VERDICT = """
|
||||||
|
CREATE TABLE IF NOT EXISTS unit_verdict (
|
||||||
|
unit_id TEXT NOT NULL,
|
||||||
|
pair_id TEXT,
|
||||||
|
sub_pattern TEXT,
|
||||||
|
stage INTEGER NOT NULL CHECK(stage BETWEEN 1 AND 6),
|
||||||
|
verdict TEXT CHECK(verdict IN ('pass', 'fail', 'abstain')),
|
||||||
|
reason TEXT,
|
||||||
|
metric_value REAL,
|
||||||
|
model TEXT,
|
||||||
|
session_id TEXT,
|
||||||
|
ts TEXT,
|
||||||
|
PRIMARY KEY (unit_id, stage)
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
_DDL_COLLAPSE_METRICS = """
|
||||||
|
CREATE TABLE IF NOT EXISTS collapse_metrics (
|
||||||
|
pair_id TEXT PRIMARY KEY,
|
||||||
|
text_only_acc REAL,
|
||||||
|
single_frame_acc REAL,
|
||||||
|
placebo_drop REAL,
|
||||||
|
majority_vote_hit REAL,
|
||||||
|
slot_chi2 REAL,
|
||||||
|
distractor_min_dist REAL,
|
||||||
|
multiformat_consistency REAL,
|
||||||
|
subtitle_answerability REAL,
|
||||||
|
ts TEXT
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
_DDL_QUARANTINE = """
|
||||||
|
CREATE TABLE IF NOT EXISTS quarantine (
|
||||||
|
content_fingerprint TEXT PRIMARY KEY,
|
||||||
|
sub_pattern TEXT,
|
||||||
|
quarantine_reason TEXT,
|
||||||
|
round_no INTEGER,
|
||||||
|
ts TEXT
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
_DDL_RESUME_STATE = """
|
||||||
|
CREATE TABLE IF NOT EXISTS resume_state (
|
||||||
|
unit_id TEXT PRIMARY KEY,
|
||||||
|
status TEXT CHECK(status IN ('pending', 'accepted', 'rejected')),
|
||||||
|
config_fingerprint TEXT,
|
||||||
|
seq_offset INTEGER
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
_DDL_V3_TABLES = [
|
||||||
|
_DDL_FACTS,
|
||||||
|
_DDL_UNIT_VERDICT,
|
||||||
|
_DDL_COLLAPSE_METRICS,
|
||||||
|
_DDL_QUARANTINE,
|
||||||
|
_DDL_RESUME_STATE,
|
||||||
|
]
|
||||||
|
|
||||||
|
_DDL_V3_INDEXES = [
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_facts_video ON facts(video_id);",
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_facts_session ON facts(session_id);",
|
||||||
|
# unit_verdict(unit_id) 无需独立索引:主键 (unit_id, stage) 的左前缀已覆盖 unit_id 查询。
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_uv_sub_pattern ON unit_verdict(sub_pattern);",
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_quar_sub_pattern ON quarantine(sub_pattern);",
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_resume_status ON resume_state(status);",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Store 实现
|
# Store 实现
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -201,6 +300,10 @@ class QuestionGenStore:
|
|||||||
self._conn.execute(_DDL_VERDICTS)
|
self._conn.execute(_DDL_VERDICTS)
|
||||||
for idx_sql in _DDL_VERDICTS_INDEXES:
|
for idx_sql in _DDL_VERDICTS_INDEXES:
|
||||||
self._conn.execute(idx_sql)
|
self._conn.execute(idx_sql)
|
||||||
|
for ddl in _DDL_V3_TABLES:
|
||||||
|
self._conn.execute(ddl)
|
||||||
|
for idx_sql in _DDL_V3_INDEXES:
|
||||||
|
self._conn.execute(idx_sql)
|
||||||
self._conn.commit()
|
self._conn.commit()
|
||||||
|
|
||||||
# 幂等迁移:为已有表加 sub_pattern / selector_scores 列
|
# 幂等迁移:为已有表加 sub_pattern / selector_scores 列
|
||||||
@@ -648,6 +751,320 @@ class QuestionGenStore:
|
|||||||
|
|
||||||
return {row[0]: "accepted" for row in rows}
|
return {row[0]: "accepted" for row in rows}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# v3 出题领域观测写入(ts 一律外部传入,禁进程内 now,保幂等可复现)
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
def insert_fact(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
fact_id: str,
|
||||||
|
video_id: str,
|
||||||
|
segment_id: str,
|
||||||
|
subject: str,
|
||||||
|
action: str,
|
||||||
|
object: str, # noqa: A002 — 与设计列名 object 一致,仅 kwargs 传入无遮蔽风险
|
||||||
|
frame_ids: str,
|
||||||
|
polarity: str,
|
||||||
|
fact_type: str,
|
||||||
|
difficulty_tier: int,
|
||||||
|
verifier_refs: str,
|
||||||
|
cross_agree: int,
|
||||||
|
negative_at_target: str,
|
||||||
|
session_id: str,
|
||||||
|
ts: str,
|
||||||
|
) -> None:
|
||||||
|
"""写入一条帧感知抽取 Fact(facts 表,主键 fact_id)。
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
fact_id : str
|
||||||
|
Fact 唯一标识(UUID)。
|
||||||
|
video_id, segment_id : str
|
||||||
|
溯源:所属视频与片段。
|
||||||
|
subject, action, object : str
|
||||||
|
结构化绑定三元组。
|
||||||
|
frame_ids : str
|
||||||
|
感知所用帧 ID 的 JSON 数组字符串。
|
||||||
|
polarity : str
|
||||||
|
事实极性(真/假)。
|
||||||
|
fact_type : str
|
||||||
|
事实类型(binding/state/manner/order/evidence)。
|
||||||
|
difficulty_tier : int
|
||||||
|
感知难度分层。
|
||||||
|
verifier_refs : str
|
||||||
|
双 VLM(qwen/MiniMax)各自裁决的 JSON 字符串。
|
||||||
|
cross_agree : int
|
||||||
|
双 VLM 是否一致(0/1)。
|
||||||
|
negative_at_target : str
|
||||||
|
目标点为假的核实结果。
|
||||||
|
session_id : str
|
||||||
|
epoch/step 关联 ID。
|
||||||
|
ts : str
|
||||||
|
观测时间戳 ISO 字符串,由调用方外部传入(禁进程内 now)。
|
||||||
|
"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO facts
|
||||||
|
(fact_id, video_id, segment_id, subject, action, object, frame_ids,
|
||||||
|
polarity, fact_type, difficulty_tier, verifier_refs, cross_agree,
|
||||||
|
negative_at_target, session_id, ts)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
fact_id,
|
||||||
|
video_id,
|
||||||
|
segment_id,
|
||||||
|
subject,
|
||||||
|
action,
|
||||||
|
object,
|
||||||
|
frame_ids,
|
||||||
|
polarity,
|
||||||
|
fact_type,
|
||||||
|
difficulty_tier,
|
||||||
|
verifier_refs,
|
||||||
|
cross_agree,
|
||||||
|
negative_at_target,
|
||||||
|
session_id,
|
||||||
|
ts,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
logger.debug("facts 已写入: fact_id={}", fact_id)
|
||||||
|
|
||||||
|
def insert_unit_verdict(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
unit_id: str,
|
||||||
|
pair_id: str | None,
|
||||||
|
sub_pattern: str,
|
||||||
|
stage: int,
|
||||||
|
verdict: str,
|
||||||
|
reason: str,
|
||||||
|
metric_value: float | None,
|
||||||
|
model: str | None,
|
||||||
|
session_id: str,
|
||||||
|
ts: str,
|
||||||
|
) -> None:
|
||||||
|
"""写入六层验证某一层的裁决(unit_verdict 表,主键 (unit_id, stage))。
|
||||||
|
|
||||||
|
同一 (unit_id, stage) 重跑时以 upsert 覆盖旧行——每 unit 每层仅保留最新一条,
|
||||||
|
便于断点续跑重放而不残留过期裁决。
|
||||||
|
|
||||||
|
覆盖语义:ON CONFLICT DO UPDATE 用 ``excluded.*`` **整行覆盖**全部非键列
|
||||||
|
(含可空的 metric_value/model 及 ts)——只保留每层最新完整裁决,不支持部分字段
|
||||||
|
更新。调用方每次必须传完整行,否则会用 metric_value=None/model=None 误抹先前非空值。
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
unit_id : str
|
||||||
|
pair/single 单元标识。
|
||||||
|
pair_id : str | None
|
||||||
|
所属 pair 关联标识(single 单元可为 None)。
|
||||||
|
sub_pattern : str
|
||||||
|
6 子模式之一。
|
||||||
|
stage : int
|
||||||
|
验证层号(1-6)。
|
||||||
|
verdict : str
|
||||||
|
该层裁决(pass/fail/abstain)。
|
||||||
|
reason : str
|
||||||
|
拒因描述。
|
||||||
|
metric_value : float | None
|
||||||
|
该层量化值(无量化值时为 None)。
|
||||||
|
model : str | None
|
||||||
|
裁判模型名(无模型判定时为 None)。
|
||||||
|
session_id : str
|
||||||
|
epoch/step 关联 ID。
|
||||||
|
ts : str
|
||||||
|
观测时间戳 ISO 字符串,由调用方外部传入(禁进程内 now)。
|
||||||
|
"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO unit_verdict
|
||||||
|
(unit_id, pair_id, sub_pattern, stage, verdict, reason,
|
||||||
|
metric_value, model, session_id, ts)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(unit_id, stage) DO UPDATE SET
|
||||||
|
pair_id=excluded.pair_id,
|
||||||
|
sub_pattern=excluded.sub_pattern,
|
||||||
|
verdict=excluded.verdict,
|
||||||
|
reason=excluded.reason,
|
||||||
|
metric_value=excluded.metric_value,
|
||||||
|
model=excluded.model,
|
||||||
|
session_id=excluded.session_id,
|
||||||
|
ts=excluded.ts
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
unit_id,
|
||||||
|
pair_id,
|
||||||
|
sub_pattern,
|
||||||
|
stage,
|
||||||
|
verdict,
|
||||||
|
reason,
|
||||||
|
metric_value,
|
||||||
|
model,
|
||||||
|
session_id,
|
||||||
|
ts,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
logger.debug("unit_verdict 已写入: unit_id={}, stage={}", unit_id, stage)
|
||||||
|
|
||||||
|
def insert_collapse_metrics(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
pair_id: str,
|
||||||
|
text_only_acc: float,
|
||||||
|
single_frame_acc: float,
|
||||||
|
placebo_drop: float,
|
||||||
|
majority_vote_hit: float,
|
||||||
|
slot_chi2: float,
|
||||||
|
distractor_min_dist: float,
|
||||||
|
multiformat_consistency: float,
|
||||||
|
subtitle_answerability: float,
|
||||||
|
ts: str,
|
||||||
|
) -> None:
|
||||||
|
"""写入配对坍缩度量(collapse_metrics 表,主键 pair_id)。
|
||||||
|
|
||||||
|
同一 pair_id 重算时以 upsert 覆盖旧行——每 pair 仅保留最新一组度量。
|
||||||
|
|
||||||
|
覆盖语义:ON CONFLICT DO UPDATE 用 ``excluded.*`` **整行覆盖**全部非键列(含 ts),
|
||||||
|
不支持部分字段更新;调用方每次必须传完整度量行。
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
pair_id : str
|
||||||
|
配对唯一标识。
|
||||||
|
text_only_acc, single_frame_acc, placebo_drop : float
|
||||||
|
模型探针类度量。
|
||||||
|
majority_vote_hit, slot_chi2, distractor_min_dist : float
|
||||||
|
纯结构类度量。
|
||||||
|
multiformat_consistency, subtitle_answerability : float
|
||||||
|
探针类度量。
|
||||||
|
ts : str
|
||||||
|
观测时间戳 ISO 字符串,由调用方外部传入(禁进程内 now)。
|
||||||
|
"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO collapse_metrics
|
||||||
|
(pair_id, text_only_acc, single_frame_acc, placebo_drop,
|
||||||
|
majority_vote_hit, slot_chi2, distractor_min_dist,
|
||||||
|
multiformat_consistency, subtitle_answerability, ts)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(pair_id) DO UPDATE SET
|
||||||
|
text_only_acc=excluded.text_only_acc,
|
||||||
|
single_frame_acc=excluded.single_frame_acc,
|
||||||
|
placebo_drop=excluded.placebo_drop,
|
||||||
|
majority_vote_hit=excluded.majority_vote_hit,
|
||||||
|
slot_chi2=excluded.slot_chi2,
|
||||||
|
distractor_min_dist=excluded.distractor_min_dist,
|
||||||
|
multiformat_consistency=excluded.multiformat_consistency,
|
||||||
|
subtitle_answerability=excluded.subtitle_answerability,
|
||||||
|
ts=excluded.ts
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
pair_id,
|
||||||
|
text_only_acc,
|
||||||
|
single_frame_acc,
|
||||||
|
placebo_drop,
|
||||||
|
majority_vote_hit,
|
||||||
|
slot_chi2,
|
||||||
|
distractor_min_dist,
|
||||||
|
multiformat_consistency,
|
||||||
|
subtitle_answerability,
|
||||||
|
ts,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
logger.debug("collapse_metrics 已写入: pair_id={}", pair_id)
|
||||||
|
|
||||||
|
def quarantine(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
content_fingerprint: str,
|
||||||
|
sub_pattern: str,
|
||||||
|
quarantine_reason: str,
|
||||||
|
round_no: int,
|
||||||
|
ts: str,
|
||||||
|
) -> None:
|
||||||
|
"""将失败题的内容指纹写入隔离区黑名单(quarantine 表,主键 content_fingerprint)。
|
||||||
|
|
||||||
|
同一 content_fingerprint 重复调用以 upsert 覆盖——保证语义相同题面只占一行,
|
||||||
|
补构造前查此表当黑名单去重。覆盖语义:ON CONFLICT DO UPDATE 用 ``excluded.*``
|
||||||
|
**整行覆盖**全部非键列(含 ts),不支持部分字段更新。
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
content_fingerprint : str
|
||||||
|
题面语义内容指纹(去重键)。
|
||||||
|
sub_pattern : str
|
||||||
|
题目所属子模式。
|
||||||
|
quarantine_reason : str
|
||||||
|
隔离原因。
|
||||||
|
round_no : int
|
||||||
|
隔离发生的过滤轮次。
|
||||||
|
ts : str
|
||||||
|
观测时间戳 ISO 字符串,由调用方外部传入(禁进程内 now)。
|
||||||
|
"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO quarantine
|
||||||
|
(content_fingerprint, sub_pattern, quarantine_reason, round_no, ts)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(content_fingerprint) DO UPDATE SET
|
||||||
|
sub_pattern=excluded.sub_pattern,
|
||||||
|
quarantine_reason=excluded.quarantine_reason,
|
||||||
|
round_no=excluded.round_no,
|
||||||
|
ts=excluded.ts
|
||||||
|
""",
|
||||||
|
(content_fingerprint, sub_pattern, quarantine_reason, round_no, ts),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
logger.debug("quarantine 已写入: fingerprint={}", content_fingerprint)
|
||||||
|
|
||||||
|
def upsert_resume_state(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
unit_id: str,
|
||||||
|
status: str,
|
||||||
|
config_fingerprint: str,
|
||||||
|
seq_offset: int,
|
||||||
|
) -> None:
|
||||||
|
"""写入/更新断点续跑状态(resume_state 表,主键 unit_id)。
|
||||||
|
|
||||||
|
同一 unit_id 覆盖更新——每单元仅保留最新续跑状态。config_fingerprint 变化时
|
||||||
|
由调用方据此判定旧进度作废并重跑。本表无 ts 列(schema 未定义),但方法体仍禁
|
||||||
|
进程内 now,保持 v3 观测一致的幂等可复现语义。
|
||||||
|
|
||||||
|
覆盖语义:ON CONFLICT DO UPDATE 用 ``excluded.*`` **整行覆盖**全部非键列,不支持
|
||||||
|
部分字段更新;调用方每次必须传完整状态行。
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
unit_id : str
|
||||||
|
单元唯一标识。
|
||||||
|
status : str
|
||||||
|
续跑状态(pending/accepted/rejected)。
|
||||||
|
config_fingerprint : str
|
||||||
|
求解器/裁判 config 指纹,变更时旧进度作废。
|
||||||
|
seq_offset : int
|
||||||
|
补构造续编偏移,防止编号相撞。
|
||||||
|
"""
|
||||||
|
self._conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO resume_state
|
||||||
|
(unit_id, status, config_fingerprint, seq_offset)
|
||||||
|
VALUES (?, ?, ?, ?)
|
||||||
|
ON CONFLICT(unit_id) DO UPDATE SET
|
||||||
|
status=excluded.status,
|
||||||
|
config_fingerprint=excluded.config_fingerprint,
|
||||||
|
seq_offset=excluded.seq_offset
|
||||||
|
""",
|
||||||
|
(unit_id, status, config_fingerprint, seq_offset),
|
||||||
|
)
|
||||||
|
self._conn.commit()
|
||||||
|
logger.debug("resume_state 已写入: unit_id={}, status={}", unit_id, status)
|
||||||
|
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
"""关闭数据库连接。"""
|
"""关闭数据库连接。"""
|
||||||
self._conn.close()
|
self._conn.close()
|
||||||
|
|||||||
+1
-1
@@ -42,7 +42,6 @@ harness:
|
|||||||
gate_delta_min: 0.02
|
gate_delta_min: 0.02
|
||||||
gate_lambda_dir: -0.642
|
gate_lambda_dir: -0.642
|
||||||
gate_e_rollback: 10.0
|
gate_e_rollback: 10.0
|
||||||
gate_block: 8
|
|
||||||
gate_n_max: 40
|
gate_n_max: 40
|
||||||
gate_p_low: 0.05
|
gate_p_low: 0.05
|
||||||
gate_p_high: 0.95
|
gate_p_high: 0.95
|
||||||
@@ -67,6 +66,7 @@ harness:
|
|||||||
batch_correct_ratio: 0.5
|
batch_correct_ratio: 0.5
|
||||||
momentum_samples: 20
|
momentum_samples: 20
|
||||||
eval_min_per_class: 2
|
eval_min_per_class: 2
|
||||||
|
trainable_min_units: 8
|
||||||
early_stop_patience: 8
|
early_stop_patience: 8
|
||||||
use_slow_momentum: true
|
use_slow_momentum: true
|
||||||
# 池构建策略
|
# 池构建策略
|
||||||
|
|||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# 补生 Video-MME 6 类缺失题型(180 题)
|
||||||
|
# 原 360 题中有 6 类错误类型已归档,此配置只生成缺失的 6 类。
|
||||||
|
|
||||||
|
# ── 建树模块 ──(与 question_gen_360.yaml 一致)
|
||||||
|
tree:
|
||||||
|
max_paragraphs_per_l2: 5
|
||||||
|
l1_segment_duration: 600.0
|
||||||
|
l2_clip_duration: 60.0
|
||||||
|
l3_fps: 0.5
|
||||||
|
l2_representative_frames: 6
|
||||||
|
cache_dir: "cache/trees"
|
||||||
|
concurrency: 16
|
||||||
|
subtitle_inject: true
|
||||||
|
srt_window_sec: 5.0
|
||||||
|
|
||||||
|
# ── Embedding ──
|
||||||
|
embed:
|
||||||
|
backend: "local"
|
||||||
|
model_name: "BAAI/bge-base-zh-v1.5"
|
||||||
|
embed_dim: 768
|
||||||
|
device: "cuda"
|
||||||
|
|
||||||
|
# ── Harness ──(占位,出题不使用)
|
||||||
|
harness:
|
||||||
|
workspace_dir: "workspaces/default"
|
||||||
|
store_dir: store
|
||||||
|
mode: infer
|
||||||
|
concurrency: 24
|
||||||
|
max_steps: 40
|
||||||
|
skill_mode: auto
|
||||||
|
n_samples: 0
|
||||||
|
questions: "benchmarks/Video-MME"
|
||||||
|
skills_version: v1
|
||||||
|
prompts_version: v1
|
||||||
|
epochs: 1
|
||||||
|
gate_e_confirm: 20.0
|
||||||
|
gate_e_provisional: 3.0
|
||||||
|
gate_w_net_min: 2
|
||||||
|
gate_delta_min: 0.02
|
||||||
|
gate_lambda_dir: -0.642
|
||||||
|
gate_e_rollback: 10.0
|
||||||
|
gate_n_max: 40
|
||||||
|
gate_p_low: 0.05
|
||||||
|
gate_p_high: 0.95
|
||||||
|
gate_probe_quota: 0.2
|
||||||
|
gate_gamma_decay: 0.9
|
||||||
|
gate_cooldown_steps: 2
|
||||||
|
gate_guard_err: 0.10
|
||||||
|
edit_budget_start: 5
|
||||||
|
edit_budget_end: 2
|
||||||
|
skill_update_mode: patch
|
||||||
|
appendix_consolidate_threshold: 6
|
||||||
|
diag_size: 200
|
||||||
|
diag_correct_ratio: 0.5
|
||||||
|
val_size: 30
|
||||||
|
val_correct_ratio: 0.5
|
||||||
|
test_size: 60
|
||||||
|
batch_size: 15
|
||||||
|
min_class_per_batch: 2
|
||||||
|
batch_correct_ratio: 0.5
|
||||||
|
momentum_samples: 20
|
||||||
|
eval_min_per_class: 2
|
||||||
|
early_stop_patience: 8
|
||||||
|
use_slow_momentum: true
|
||||||
|
|
||||||
|
# ── 出题管线 v2 ──
|
||||||
|
question_gen_v2:
|
||||||
|
family_ratios:
|
||||||
|
retrieval: 0.30
|
||||||
|
reasoning: 0.25
|
||||||
|
enumeration: 0.20
|
||||||
|
visual: 0.15
|
||||||
|
spatial: 0.10
|
||||||
|
dedup_threshold: 0.85
|
||||||
|
retry_limit: 10
|
||||||
|
heavy_sample_rate: 0.15
|
||||||
|
output_dir: "store/questions/generated-v2-180补"
|
||||||
|
per_type: 30 # 6 类 x 30 = 180 题
|
||||||
|
concurrency: 24
|
||||||
|
seed: 43 # 不同于原始 seed=42,避免生成相同题目
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# config/default.yaml
|
||||||
|
# 科研实验配置默认值来源(会在实验中反复扫动/对比的参数)。
|
||||||
|
# 工程配置(少变、敏感)由 .env / pydantic-settings 管理,不在此文件。
|
||||||
|
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
|
||||||
|
|
||||||
|
# ── 建树模块 ──
|
||||||
|
tree:
|
||||||
|
max_paragraphs_per_l2: 5
|
||||||
|
l1_segment_duration: 600.0 # L1 段时长(秒)
|
||||||
|
l2_clip_duration: 60.0 # L2 clip 时长(秒)
|
||||||
|
l3_fps: 0.5 # L3 帧提取频率(帧/秒)
|
||||||
|
l2_representative_frames: 6 # L2 VLM 描述用的代表帧数
|
||||||
|
cache_dir: "cache/trees"
|
||||||
|
concurrency: 16 # asyncio Semaphore 上限
|
||||||
|
subtitle_inject: true # 建树时是否注入 SRT 字幕
|
||||||
|
srt_window_sec: 5.0 # 字幕匹配时间窗口(前后各 N 秒)
|
||||||
|
|
||||||
|
# ── Embedding ──
|
||||||
|
embed:
|
||||||
|
backend: "local"
|
||||||
|
model_name: "BAAI/bge-base-zh-v1.5"
|
||||||
|
embed_dim: 768
|
||||||
|
device: "cuda"
|
||||||
|
|
||||||
|
# ── Harness 自进化循环 ──
|
||||||
|
harness:
|
||||||
|
workspace_dir: "workspaces/default"
|
||||||
|
store_dir: store
|
||||||
|
mode: infer
|
||||||
|
concurrency: 24
|
||||||
|
max_steps: 40
|
||||||
|
skill_mode: auto
|
||||||
|
n_samples: 0
|
||||||
|
questions: "benchmarks/Video-MME"
|
||||||
|
skills_version: v1
|
||||||
|
prompts_version: v1
|
||||||
|
epochs: 1
|
||||||
|
# CE-Gate 参数
|
||||||
|
gate_e_confirm: 20.0
|
||||||
|
gate_e_provisional: 3.0
|
||||||
|
gate_w_net_min: 2
|
||||||
|
gate_delta_min: 0.02
|
||||||
|
gate_lambda_dir: -0.642
|
||||||
|
gate_e_rollback: 10.0
|
||||||
|
gate_n_max: 40
|
||||||
|
gate_p_low: 0.05
|
||||||
|
gate_p_high: 0.95
|
||||||
|
gate_probe_quota: 0.2
|
||||||
|
gate_gamma_decay: 0.9
|
||||||
|
gate_cooldown_steps: 2
|
||||||
|
gate_guard_err: 0.10
|
||||||
|
# 进化参数
|
||||||
|
edit_budget_start: 5
|
||||||
|
edit_budget_end: 2
|
||||||
|
skill_update_mode: patch
|
||||||
|
appendix_consolidate_threshold: 6
|
||||||
|
# 数据池
|
||||||
|
diag_size: 200
|
||||||
|
diag_correct_ratio: 0.5
|
||||||
|
val_size: 30
|
||||||
|
val_correct_ratio: 0.5
|
||||||
|
test_size: 60
|
||||||
|
# mini-batch
|
||||||
|
batch_size: 15
|
||||||
|
min_class_per_batch: 2
|
||||||
|
batch_correct_ratio: 0.5
|
||||||
|
momentum_samples: 20
|
||||||
|
eval_min_per_class: 2
|
||||||
|
early_stop_patience: 8
|
||||||
|
use_slow_momentum: true
|
||||||
|
|
||||||
|
# ── 出题管线 v2 ──
|
||||||
|
question_gen_v2:
|
||||||
|
family_ratios:
|
||||||
|
retrieval: 0.30
|
||||||
|
reasoning: 0.25
|
||||||
|
enumeration: 0.20
|
||||||
|
visual: 0.15
|
||||||
|
spatial: 0.10
|
||||||
|
dedup_threshold: 0.85
|
||||||
|
retry_limit: 10
|
||||||
|
heavy_sample_rate: 0.15
|
||||||
|
output_dir: "store/questions/generated-v2-360"
|
||||||
|
per_type: 30 # 12 类 x 30 = 360 题
|
||||||
|
concurrency: 24
|
||||||
|
seed: 42
|
||||||
@@ -22,7 +22,6 @@ harness:
|
|||||||
gate_delta_min: 0.02
|
gate_delta_min: 0.02
|
||||||
gate_lambda_dir: -0.642
|
gate_lambda_dir: -0.642
|
||||||
gate_e_rollback: 10.0
|
gate_e_rollback: 10.0
|
||||||
gate_block: 8
|
|
||||||
gate_n_max: 40
|
gate_n_max: 40
|
||||||
gate_p_low: 0.05
|
gate_p_low: 0.05
|
||||||
gate_p_high: 0.95
|
gate_p_high: 0.95
|
||||||
@@ -48,6 +47,7 @@ harness:
|
|||||||
batch_correct_ratio: 0.5
|
batch_correct_ratio: 0.5
|
||||||
momentum_samples: 20
|
momentum_samples: 20
|
||||||
eval_min_per_class: 2
|
eval_min_per_class: 2
|
||||||
|
trainable_min_units: 8
|
||||||
early_stop_patience: 4
|
early_stop_patience: 4
|
||||||
test_size: 63
|
test_size: 63
|
||||||
diag_size: 20
|
diag_size: 20
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# config/train_ar30.yaml
|
||||||
|
# Action Recognition 训练 — 基于 SubPattern 靶向生成的 30 题
|
||||||
|
# 对比基线: v2-360 的 AR 题(100% 单帧,训练无效)
|
||||||
|
# 本次: AR30 题(6 种失败子模式靶向,跨段时序)
|
||||||
|
|
||||||
|
harness:
|
||||||
|
workspace_dir: "workspaces/train-ar30"
|
||||||
|
store_dir: store
|
||||||
|
mode: train
|
||||||
|
run_id: train_ar30_v1
|
||||||
|
concurrency: 24
|
||||||
|
max_steps: 40
|
||||||
|
skill_mode: auto
|
||||||
|
n_samples: 0
|
||||||
|
questions: "generated-ar30"
|
||||||
|
skills_version: v1
|
||||||
|
prompts_version: v1
|
||||||
|
epochs: 3
|
||||||
|
# CE-Gate 参数(沿用 default.yaml)
|
||||||
|
gate_e_confirm: 20.0
|
||||||
|
gate_e_provisional: 3.0
|
||||||
|
gate_w_net_min: 2
|
||||||
|
gate_delta_min: 0.02
|
||||||
|
gate_lambda_dir: -0.642
|
||||||
|
gate_e_rollback: 10.0
|
||||||
|
gate_n_max: 40
|
||||||
|
gate_p_low: 0.05
|
||||||
|
gate_p_high: 0.95
|
||||||
|
gate_probe_quota: 0.2
|
||||||
|
gate_gamma_decay: 0.9
|
||||||
|
gate_cooldown_steps: 2
|
||||||
|
gate_guard_err: 0.10
|
||||||
|
# 进化参数
|
||||||
|
edit_budget_start: 5
|
||||||
|
edit_budget_end: 2
|
||||||
|
skill_update_mode: patch
|
||||||
|
appendix_consolidate_threshold: 6
|
||||||
|
# 池配置 — per_category 单题型
|
||||||
|
pool_split_mode: per_category
|
||||||
|
task_types:
|
||||||
|
- "Action Recognition"
|
||||||
|
train_ratio: 0.667
|
||||||
|
test_questions: "benchmarks/Video-MME"
|
||||||
|
run_holdout_eval: false
|
||||||
|
# mini-batch
|
||||||
|
batch_size: 10
|
||||||
|
min_class_per_batch: 2
|
||||||
|
batch_correct_ratio: 0.5
|
||||||
|
momentum_samples: 20
|
||||||
|
eval_min_per_class: 2
|
||||||
|
trainable_min_units: 8
|
||||||
|
early_stop_patience: 4
|
||||||
|
test_size: 63
|
||||||
|
diag_size: 20
|
||||||
|
diag_correct_ratio: 0.5
|
||||||
|
val_size: 10
|
||||||
|
val_correct_ratio: 0.5
|
||||||
|
use_slow_momentum: true
|
||||||
|
|
||||||
|
embed:
|
||||||
|
backend: "local"
|
||||||
|
model_name: "BAAI/bge-base-zh-v1.5"
|
||||||
|
embed_dim: 768
|
||||||
|
device: "cuda"
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# config/train_videomme.yaml
|
||||||
|
# Video-MME 900 题自进化训练 —— 消费 video-split 冻结切分(global 三池)
|
||||||
|
#
|
||||||
|
# 数据来源: workspaces/video-split/pools.json(tier 感知 diag/val + val 功效修复)
|
||||||
|
# 经 adhoc-baseline seed 携带进 workspace(WP2 接线)。
|
||||||
|
# 训练前置: .env REDIS_CACHE_TTL 须为正整数(WP4 fail-loud);见
|
||||||
|
# research-wiki/reviews/2026-07-16-preflight-final-review.md runbook。
|
||||||
|
|
||||||
|
harness:
|
||||||
|
workspace_dir: "workspaces/train-videomme"
|
||||||
|
store_dir: store
|
||||||
|
mode: train
|
||||||
|
run_id: train_videomme_v2
|
||||||
|
concurrency: 32
|
||||||
|
max_steps: 40
|
||||||
|
skill_mode: auto
|
||||||
|
n_samples: 0
|
||||||
|
questions: "benchmarks/Video-MME" # gate 指纹依赖加载全 900 题
|
||||||
|
skills_version: v1
|
||||||
|
prompts_version: v1
|
||||||
|
epochs: 3
|
||||||
|
# CE-Gate 参数(沿用 default.yaml)
|
||||||
|
gate_e_confirm: 20.0
|
||||||
|
gate_e_provisional: 3.0
|
||||||
|
gate_w_net_min: 2
|
||||||
|
gate_delta_min: 0.02
|
||||||
|
gate_lambda_dir: -0.642
|
||||||
|
gate_e_rollback: 10.0
|
||||||
|
gate_n_max: 40
|
||||||
|
gate_p_low: 0.05
|
||||||
|
gate_p_high: 0.95
|
||||||
|
gate_probe_quota: 0.2
|
||||||
|
gate_gamma_decay: 0.9
|
||||||
|
gate_cooldown_steps: 2
|
||||||
|
gate_guard_err: 0.10
|
||||||
|
# 进化参数
|
||||||
|
edit_budget_start: 5
|
||||||
|
edit_budget_end: 2
|
||||||
|
skill_update_mode: patch
|
||||||
|
appendix_consolidate_threshold: 6
|
||||||
|
# 池配置 —— global 冻结切分(diag/val/test 尺寸由 pools.json 冻结,以下采样旋钮加载时忽略)
|
||||||
|
# 实际冻结切分(0.4+tier,fingerprint d456ef4f): diag=180 val=120 test=600
|
||||||
|
pool_split_mode: global
|
||||||
|
diag_size: 180
|
||||||
|
diag_correct_ratio: 0.5
|
||||||
|
val_size: 120
|
||||||
|
val_correct_ratio: 0.5
|
||||||
|
test_size: 600
|
||||||
|
test_questions: "benchmarks/Video-MME"
|
||||||
|
# 可训练性预检(WP3):val 单元 < eval_min_per_class 或 非test单元 < trainable_min_units 的题型剔除
|
||||||
|
eval_min_per_class: 2
|
||||||
|
trainable_min_units: 8
|
||||||
|
# mini-batch —— 对齐 TRM4 正式实验 batch=40(sh --batch-size 40 覆盖 yaml 15 的最终生效值):
|
||||||
|
# 8 可训题型 × 每型约 5 题/step,保住题型级诊断信号;同时 steps/epoch 180/40≈5,
|
||||||
|
# 进化/gate 验证轮数比 batch=10 少 4 倍。
|
||||||
|
batch_size: 40
|
||||||
|
min_class_per_batch: 2
|
||||||
|
batch_correct_ratio: 0.5
|
||||||
|
momentum_samples: 20
|
||||||
|
early_stop_patience: 2 # epoch 粒度(WP3):连续 2 epoch 无 best 刷新即停
|
||||||
|
use_slow_momentum: true
|
||||||
|
run_holdout_eval: true # 逐 epoch test 四向评估(WP3 去重版:baseline 推导 + 版本备忘录)
|
||||||
|
# 全 12 题型(不指定 task_types 子集,避免 I-4 语义偏差;微型类由预检自动剔除)
|
||||||
|
|
||||||
|
embed:
|
||||||
|
backend: "local"
|
||||||
|
model_name: "BAAI/bge-base-zh-v1.5"
|
||||||
|
embed_dim: 768
|
||||||
|
device: "cuda"
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# config/video_split.yaml
|
||||||
|
# 结果驱动视频级切分 —— 科研旋钮快照(会在实验中被反复扫动 / 对比的参数)。
|
||||||
|
# 工程配置(harness.db 路径、LLM 凭证、超时)由 .env / pydantic-settings 管理,不在此文件。
|
||||||
|
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
|
||||||
|
#
|
||||||
|
# ⚠️ 下列阈值为「占位值」,必须在离线诊断跑完后用真实 T2 分布标定。
|
||||||
|
# 标定程序见 scripts/build_video_split.sh 顶部注释(诊断产物 → 逐旋钮定值)。
|
||||||
|
|
||||||
|
video_split:
|
||||||
|
baseline_run_id: infer_adhoc # 基线 run 标识(错题诊断与切分依据),对应 workspaces/default/harness.db
|
||||||
|
n_trainval: 100 # trainval 目标视频数(多样性阶段填充上限;标定取 ~100)
|
||||||
|
epsilon: 0.1 # test 相对全局最大允许分布偏差(题型占比 / 难度画像两维,逐桶)
|
||||||
|
report_floor: 27 # per-type 报告门限:题数 ≥ 此值的 task_type 才入 ε 代表性约束
|
||||||
|
val_wrong_min: 20 # validation 池最少错题数(McNemar 检验功效阈 ≈ 20,低于则信号不足)
|
||||||
|
val_ratio: 0.4 # validation 占 trainval 视频组总数的比例(0.3→0.4 提升整包终审功效,WP2)
|
||||||
|
seed: 7 # 贪心选择器预洗牌 + 视频组题级切分种子(打破等增益 / 等槽平局)
|
||||||
|
floor_k: # 各 task_type 的 T2 defect 下限(硬约束)—— 均衡覆盖全 11 类,标定于 1cb1c203 真实 T2 分布
|
||||||
|
Object Reasoning: 5 # T2=25
|
||||||
|
Information Synopsis: 5 # T2=13
|
||||||
|
Action Reasoning: 5 # T2=11
|
||||||
|
Counting Problem: 5 # T2=10
|
||||||
|
Temporal Reasoning: 2 # T2=5
|
||||||
|
Object Recognition: 2 # T2=5
|
||||||
|
Action Recognition: 2 # T2=5
|
||||||
|
Attribute Perception: 1 # T2=3
|
||||||
|
Temporal Perception: 1 # T2=2
|
||||||
|
OCR Problems: 1 # T2=2
|
||||||
|
Spatial Perception: 1 # T2=1
|
||||||
|
|
||||||
|
diag: # 诊断口径指纹三分量(隔离不同诊断配置的信号,参与主键)
|
||||||
|
prompt_version: diagnose_v1 # 诊断 prompt 版本标识(换 prompt 即换指纹,旧记录不被覆盖)
|
||||||
|
model: deepseek-v4-pro # 执行诊断的模型名(必须与 .env SEARCH_LLM_MODEL 一致,CLI 会 fail loud 校验)
|
||||||
|
# code_version 由 build_video_split.sh 注入 git 短 SHA,不写死在此(随代码变动)
|
||||||
+10
-3
@@ -108,6 +108,7 @@ class AgentLoop:
|
|||||||
plugins: list[object] | None = None,
|
plugins: list[object] | None = None,
|
||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LoopResult:
|
) -> LoopResult:
|
||||||
"""执行 Thinking+JSON 推理循环。
|
"""执行 Thinking+JSON 推理循环。
|
||||||
|
|
||||||
@@ -117,6 +118,7 @@ class AgentLoop:
|
|||||||
tool_dispatcher: 工具调度器,ToolDispatcher Protocol 实例。
|
tool_dispatcher: 工具调度器,ToolDispatcher Protocol 实例。
|
||||||
plugins: pluggy 插件列表。
|
plugins: pluggy 插件列表。
|
||||||
session_id: 会话 ID,透传给 LLMProvider。
|
session_id: 会话 ID,透传给 LLMProvider。
|
||||||
|
cache_salt: 缓存盐,透传给 LLMProvider(如训练用 run_id 跨 epoch 重采样)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
LoopResult 实例,包含推理步骤、token 用量、终止原因。
|
LoopResult 实例,包含推理步骤、token 用量、终止原因。
|
||||||
@@ -138,7 +140,7 @@ class AgentLoop:
|
|||||||
# Phase 1: LLM 调用(步级重试:防穿透 GovernedLLMClient 的瞬时异常)
|
# Phase 1: LLM 调用(步级重试:防穿透 GovernedLLMClient 的瞬时异常)
|
||||||
try:
|
try:
|
||||||
response = await self._call_llm_with_step_retry(
|
response = await self._call_llm_with_step_retry(
|
||||||
messages, token_usage, session_id=session_id
|
messages, token_usage, session_id=session_id, cache_salt=cache_salt
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error("LLM API 调用失败({}): {}", type(e).__name__, e)
|
logger.error("LLM API 调用失败({}): {}", type(e).__name__, e)
|
||||||
@@ -269,6 +271,7 @@ class AgentLoop:
|
|||||||
token_usage: dict[str, int],
|
token_usage: dict[str, int],
|
||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse:
|
) -> LLMResponse:
|
||||||
"""带步级重试的 LLM 调用,兜底穿透治理层重试栈的瞬时异常。
|
"""带步级重试的 LLM 调用,兜底穿透治理层重试栈的瞬时异常。
|
||||||
|
|
||||||
@@ -292,7 +295,9 @@ class AgentLoop:
|
|||||||
step_attempt = 0
|
step_attempt = 0
|
||||||
while True:
|
while True:
|
||||||
try:
|
try:
|
||||||
return await self._call_llm(messages, token_usage, session_id=session_id)
|
return await self._call_llm(
|
||||||
|
messages, token_usage, session_id=session_id, cache_salt=cache_salt
|
||||||
|
)
|
||||||
except self._retryable_exceptions as e:
|
except self._retryable_exceptions as e:
|
||||||
step_attempt += 1
|
step_attempt += 1
|
||||||
if step_attempt > self._step_retries:
|
if step_attempt > self._step_retries:
|
||||||
@@ -315,6 +320,7 @@ class AgentLoop:
|
|||||||
token_usage: dict[str, int],
|
token_usage: dict[str, int],
|
||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse:
|
) -> LLMResponse:
|
||||||
"""调用 LLMProvider 并累加 token 使用量。
|
"""调用 LLMProvider 并累加 token 使用量。
|
||||||
|
|
||||||
@@ -322,11 +328,12 @@ class AgentLoop:
|
|||||||
messages: 消息历史。
|
messages: 消息历史。
|
||||||
token_usage: 可变字典,就地累加。
|
token_usage: 可变字典,就地累加。
|
||||||
session_id: 会话 ID,透传给 LLMProvider。
|
session_id: 会话 ID,透传给 LLMProvider。
|
||||||
|
cache_salt: 缓存盐,透传给 LLMProvider(跨 epoch 重采样)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
LLMResponse 实例。
|
LLMResponse 实例。
|
||||||
"""
|
"""
|
||||||
response = await self._llm.chat(messages, session_id=session_id)
|
response = await self._llm.chat(messages, session_id=session_id, cache_salt=cache_salt)
|
||||||
token_usage["prompt_tokens"] += response.prompt_tokens
|
token_usage["prompt_tokens"] += response.prompt_tokens
|
||||||
token_usage["completion_tokens"] += response.completion_tokens
|
token_usage["completion_tokens"] += response.completion_tokens
|
||||||
return response
|
return response
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
只依赖 Protocol 接口和标准库,可搬到无 adapters 的环境用假实现原样运行。
|
只依赖 Protocol 接口和标准库,可搬到无 adapters 的环境用假实现原样运行。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from core.evolution.diagnose import run_diagnosis
|
from core.evolution.diagnose import INFRA_STOP_REASONS, run_diagnosis
|
||||||
from core.evolution.evolve import (
|
from core.evolution.evolve import (
|
||||||
edit_budget_at,
|
edit_budget_at,
|
||||||
evolve_single_skill,
|
evolve_single_skill,
|
||||||
@@ -44,6 +44,7 @@ from core.evolution.types import (
|
|||||||
from core.evolution.validate import classify_quadrants, compute_accuracy, pair_block
|
from core.evolution.validate import classify_quadrants, compute_accuracy, pair_block
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"INFRA_STOP_REASONS",
|
||||||
"CaseSample",
|
"CaseSample",
|
||||||
"DiagnosePrompts",
|
"DiagnosePrompts",
|
||||||
"DiagnosisResult",
|
"DiagnosisResult",
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ if TYPE_CHECKING:
|
|||||||
_SPAN_EVAL_TOOLS: frozenset[str] = frozenset({"view_node", "search_similar", "observe_frame"})
|
_SPAN_EVAL_TOOLS: frozenset[str] = frozenset({"view_node", "search_similar", "observe_frame"})
|
||||||
"""span 级评估涵盖的工具集合。"""
|
"""span 级评估涵盖的工具集合。"""
|
||||||
|
|
||||||
_INFRA_STOP_REASONS: frozenset[str] = frozenset({"error", "parse_error"})
|
INFRA_STOP_REASONS: frozenset[str] = frozenset({"error", "parse_error"})
|
||||||
"""执行/解析层失败导致排除的 stop_reason 集合。"""
|
"""执行/解析层失败导致排除的 stop_reason 集合。"""
|
||||||
|
|
||||||
|
|
||||||
@@ -1489,12 +1489,15 @@ def _build_skill_case_packs(
|
|||||||
if qm.correct:
|
if qm.correct:
|
||||||
continue
|
continue
|
||||||
attr = attribution_map.get(qm.question_id)
|
attr = attribution_map.get(qm.question_id)
|
||||||
if attr is not None and attr.cause_category == "lapse":
|
# 仅明确 defect 且非 degraded 才进正文进化路径;
|
||||||
if attr.lapse_note and attr.lapse_note.strip():
|
# lapse / cause_category=None(判别失败)/ degraded(judge 解析失败)一律保守走 lapse,
|
||||||
|
# 不以降级或未判定信号驱动错误进化。
|
||||||
|
is_defect = attr is not None and attr.cause_category == "defect" and not qm.degraded
|
||||||
|
if not is_defect:
|
||||||
|
if attr is not None and attr.lapse_note and attr.lapse_note.strip():
|
||||||
lapse_notes.append(attr.lapse_note)
|
lapse_notes.append(attr.lapse_note)
|
||||||
continue
|
continue
|
||||||
et = attr.error_type if attr else "mixed"
|
wrong_by_error[attr.error_type].append(qm)
|
||||||
wrong_by_error[et].append(qm)
|
|
||||||
|
|
||||||
# 单条 fallback
|
# 单条 fallback
|
||||||
n_body_failures = sum(len(group) for group in wrong_by_error.values())
|
n_body_failures = sum(len(group) for group in wrong_by_error.values())
|
||||||
@@ -2004,7 +2007,7 @@ def _count_infra_excluded(
|
|||||||
qids = [
|
qids = [
|
||||||
row["question_id"]
|
row["question_id"]
|
||||||
for row in prediction_rows
|
for row in prediction_rows
|
||||||
if row.get("stop_reason") in _INFRA_STOP_REASONS
|
if row.get("stop_reason") in INFRA_STOP_REASONS
|
||||||
]
|
]
|
||||||
return len(qids), qids
|
return len(qids), qids
|
||||||
|
|
||||||
@@ -2080,7 +2083,7 @@ async def run_diagnosis(
|
|||||||
|
|
||||||
for row in all_predictions:
|
for row in all_predictions:
|
||||||
stop_reason = row.get("stop_reason")
|
stop_reason = row.get("stop_reason")
|
||||||
if stop_reason in _INFRA_STOP_REASONS:
|
if stop_reason in INFRA_STOP_REASONS:
|
||||||
continue
|
continue
|
||||||
if task_type_filter and row.get("task_type") not in task_type_filter:
|
if task_type_filter and row.get("task_type") not in task_type_filter:
|
||||||
continue
|
continue
|
||||||
@@ -2142,7 +2145,15 @@ async def run_diagnosis(
|
|||||||
key = (prediction.get("video_id", ""), prediction.get("question_id", ""))
|
key = (prediction.get("video_id", ""), prediction.get("question_id", ""))
|
||||||
traces = traces_by_question.get(key, [])
|
traces = traces_by_question.get(key, [])
|
||||||
vid = prediction.get("video_id", "")
|
vid = prediction.get("video_id", "")
|
||||||
td = tree_data_by_video.get(vid, {})
|
if vid not in tree_data_by_video:
|
||||||
|
qid = prediction.get("question_id", "")
|
||||||
|
# P5 fail-loud:诊断需真实树,调用方须为每个诊断视频加载 tree_data;
|
||||||
|
# 静默回退空树会让 ground_truth 恒空、error_type 归因坍缩(本次修复的根因)。
|
||||||
|
raise ValueError(
|
||||||
|
f"诊断视频树未覆盖: video_id={vid!r} question_id={qid!r} 不在注入的 tree_data 中"
|
||||||
|
"(调用方须为每个诊断视频加载树,P5 fail loud)"
|
||||||
|
)
|
||||||
|
td = tree_data_by_video[vid]
|
||||||
skill_content = skill_cache.get(prediction.get("task_type", ""), "")
|
skill_content = skill_cache.get(prediction.get("task_type", ""), "")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -19,6 +19,8 @@ from loguru import logger
|
|||||||
from core.evolution.patch import (
|
from core.evolution.patch import (
|
||||||
APPENDIX_END,
|
APPENDIX_END,
|
||||||
APPENDIX_START,
|
APPENDIX_START,
|
||||||
|
MOMENTUM_END,
|
||||||
|
MOMENTUM_START,
|
||||||
append_to_appendix,
|
append_to_appendix,
|
||||||
apply_patch_with_report,
|
apply_patch_with_report,
|
||||||
extract_appendix_notes,
|
extract_appendix_notes,
|
||||||
@@ -293,10 +295,39 @@ def _tool_protected_spans(text: str) -> list[str]:
|
|||||||
# =========================================================================
|
# =========================================================================
|
||||||
|
|
||||||
|
|
||||||
|
def _check_marker_integrity(evolved: str) -> list[str]:
|
||||||
|
"""校验 evolved 中冻结区 marker 的完整性(成对、至多一对、START 先于 END)。
|
||||||
|
|
||||||
|
进化写入可能破坏 appendix/momentum marker 配对,破坏后 append_to_appendix /
|
||||||
|
replace_momentum 等下游会静默误拼或抛错。此处集中拦截:任一 marker 对违反
|
||||||
|
「START 数==END 数、各至多一对、START 在 END 前」即整体 reject。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
evolved: 改写后的全文。
|
||||||
|
返回:
|
||||||
|
错误信息列表(空列表表示 marker 完整)。
|
||||||
|
"""
|
||||||
|
errors: list[str] = []
|
||||||
|
for name, start_m, end_m in (
|
||||||
|
("APPENDIX", APPENDIX_START, APPENDIX_END),
|
||||||
|
("MOMENTUM", MOMENTUM_START, MOMENTUM_END),
|
||||||
|
):
|
||||||
|
s = evolved.count(start_m)
|
||||||
|
e = evolved.count(end_m)
|
||||||
|
if s != e:
|
||||||
|
errors.append(f"{name} marker 不配对:START={s} END={e}")
|
||||||
|
elif s > 1:
|
||||||
|
errors.append(f"{name} marker 出现多对({s}),至多一对")
|
||||||
|
elif s == 1 and evolved.index(start_m) > evolved.index(end_m):
|
||||||
|
errors.append(f"{name} marker 顺序错误:END 出现在 START 之前")
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
def validate_skill(original: str, evolved: str) -> ValidationResult:
|
def validate_skill(original: str, evolved: str) -> ValidationResult:
|
||||||
"""校验 Skill 改写结果。
|
"""校验 Skill 改写结果。
|
||||||
|
|
||||||
检查项: frontmatter 三字段保留(name / description / task_type)、
|
检查项: frontmatter 三字段保留(name / description / task_type)、
|
||||||
|
marker 完整性(appendix/momentum 成对且至多一对、顺序正确)、
|
||||||
长度比在 [0.3, 2.0]、代码块闭合。
|
长度比在 [0.3, 2.0]、代码块闭合。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
@@ -305,6 +336,11 @@ def validate_skill(original: str, evolved: str) -> ValidationResult:
|
|||||||
|
|
||||||
返回:
|
返回:
|
||||||
ValidationResult 实例。
|
ValidationResult 实例。
|
||||||
|
|
||||||
|
关键实现细节:
|
||||||
|
marker 完整性先于长度校验——长度校验经 _strip_protected_regions 调用
|
||||||
|
momentum_region_bounds,对损坏 marker 会抛 ValueError;故 marker 破坏时先
|
||||||
|
返回失败,避免异常穿透且明确 reject 该候选。
|
||||||
"""
|
"""
|
||||||
errors: list[str] = []
|
errors: list[str] = []
|
||||||
orig_fm = _parse_frontmatter(original)
|
orig_fm = _parse_frontmatter(original)
|
||||||
@@ -319,6 +355,10 @@ def validate_skill(original: str, evolved: str) -> ValidationResult:
|
|||||||
errors.append(
|
errors.append(
|
||||||
f"frontmatter 字段 {key} 被修改: {orig_fm.get(key)!r} → {evol_fm.get(key)!r}"
|
f"frontmatter 字段 {key} 被修改: {orig_fm.get(key)!r} → {evol_fm.get(key)!r}"
|
||||||
)
|
)
|
||||||
|
marker_errors = _check_marker_integrity(evolved)
|
||||||
|
if marker_errors:
|
||||||
|
errors.extend(marker_errors)
|
||||||
|
return ValidationResult(passed=False, errors=errors)
|
||||||
errors.extend(_check_length(original, evolved))
|
errors.extend(_check_length(original, evolved))
|
||||||
errors.extend(_check_code_blocks(evolved))
|
errors.extend(_check_code_blocks(evolved))
|
||||||
return ValidationResult(passed=len(errors) == 0, errors=errors)
|
return ValidationResult(passed=len(errors) == 0, errors=errors)
|
||||||
|
|||||||
+43
-7
@@ -282,9 +282,34 @@ def _protected_ranges(content: str, spans: list[str]) -> list[tuple[int, int]]:
|
|||||||
return ranges
|
return ranges
|
||||||
|
|
||||||
|
|
||||||
def _in_ranges(pos: int, ranges: list[tuple[int, int]]) -> bool:
|
def _span_overlaps_ranges(pos: int, length: int, ranges: list[tuple[int, int]]) -> bool:
|
||||||
"""判断位置 pos 是否落在任意冻结区间内。"""
|
"""判断 [pos, pos+length) 是否与任一冻结区间相交(不止起点)。
|
||||||
return any(start <= pos < end for start, end in ranges)
|
|
||||||
|
起点落在正文、末端伸入冻结区的 target 也须拦截,否则 replace/delete 会连带
|
||||||
|
改动冻结区(如破坏 appendix/momentum marker)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
pos: target 在正文中的起点。
|
||||||
|
length: target 长度。
|
||||||
|
ranges: 冻结区间 [start, end) 列表。
|
||||||
|
返回:
|
||||||
|
与任一区间相交返回 True。
|
||||||
|
"""
|
||||||
|
end = pos + length
|
||||||
|
return any(start < end and pos < r_end for start, r_end in ranges)
|
||||||
|
|
||||||
|
|
||||||
|
# 冻结区 marker 字面量:LLM 生成的 edit 不得注入这些字面量,否则破坏 marker 配对
|
||||||
|
_MARKER_LITERALS = (APPENDIX_START, APPENDIX_END, MOMENTUM_START, MOMENTUM_END)
|
||||||
|
|
||||||
|
|
||||||
|
def _edit_injects_marker(edit: dict) -> bool:
|
||||||
|
"""判断 edit 的 target/content 是否含冻结区 marker 字面量(注入拦截)。"""
|
||||||
|
for key in ("target", "content"):
|
||||||
|
value = edit.get(key)
|
||||||
|
if isinstance(value, str) and any(m in value for m in _MARKER_LITERALS):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def _append_at(content: str, ranges: list[tuple[int, int]]) -> int:
|
def _append_at(content: str, ranges: list[tuple[int, int]]) -> int:
|
||||||
@@ -317,8 +342,8 @@ def _do_insert_after(
|
|||||||
_insert_at(content, _append_at(content, ranges), payload),
|
_insert_at(content, _append_at(content, ranges), payload),
|
||||||
"applied_insert_after_fallback",
|
"applied_insert_after_fallback",
|
||||||
)
|
)
|
||||||
if _in_ranges(pos, ranges):
|
if _span_overlaps_ranges(pos, len(target), ranges):
|
||||||
logger.warning("insert_after 目标在冻结区,跳过 target={}", target[:80])
|
logger.warning("insert_after 目标跨入冻结区,跳过 target={}", target[:80])
|
||||||
return content, "skipped_protected"
|
return content, "skipped_protected"
|
||||||
at = pos + len(target)
|
at = pos + len(target)
|
||||||
nl = content.find("\n", at)
|
nl = content.find("\n", at)
|
||||||
@@ -340,8 +365,8 @@ def _do_replace_delete(
|
|||||||
if pos == -1:
|
if pos == -1:
|
||||||
logger.warning("{} 锚点缺失,跳过 target={}", op, target[:80])
|
logger.warning("{} 锚点缺失,跳过 target={}", op, target[:80])
|
||||||
return content, "skipped_target_not_found"
|
return content, "skipped_target_not_found"
|
||||||
if _in_ranges(pos, ranges):
|
if _span_overlaps_ranges(pos, len(target), ranges):
|
||||||
logger.warning("{} 目标在冻结区,跳过 target={}", op, target[:80])
|
logger.warning("{} 目标跨入冻结区,跳过 target={}", op, target[:80])
|
||||||
return content, "skipped_protected"
|
return content, "skipped_protected"
|
||||||
new_content = content.replace(target, payload if op == "replace" else "", 1)
|
new_content = content.replace(target, payload if op == "replace" else "", 1)
|
||||||
return new_content, "applied_" + op
|
return new_content, "applied_" + op
|
||||||
@@ -403,6 +428,17 @@ def apply_patch_with_report(
|
|||||||
reports: list[dict] = []
|
reports: list[dict] = []
|
||||||
for i, edit in enumerate(edits, 1):
|
for i, edit in enumerate(edits, 1):
|
||||||
try:
|
try:
|
||||||
|
if isinstance(edit, dict) and _edit_injects_marker(edit):
|
||||||
|
logger.warning("edit 含冻结区 marker 字面量,拒绝该 edit index={}", i)
|
||||||
|
report = {
|
||||||
|
"op": str(edit.get("op", "")),
|
||||||
|
"target": str(edit.get("target", "") or "")[:200],
|
||||||
|
"content_preview": str(edit.get("content", "") or "")[:200],
|
||||||
|
"status": "skipped_marker_injection",
|
||||||
|
}
|
||||||
|
report["index"] = i
|
||||||
|
reports.append(report)
|
||||||
|
continue
|
||||||
ranges = _protected_ranges(content, spans)
|
ranges = _protected_ranges(content, spans)
|
||||||
content, report = _apply_one(content, edit, ranges)
|
content, report = _apply_one(content, edit, ranges)
|
||||||
except (KeyError, TypeError, ValueError, AttributeError) as exc:
|
except (KeyError, TypeError, ValueError, AttributeError) as exc:
|
||||||
|
|||||||
@@ -1,13 +1,18 @@
|
|||||||
"""core/evolution/ 子包的只读 Protocol 定义。
|
"""core/evolution/ 子包的持久化 Protocol 定义。
|
||||||
|
|
||||||
三个 Protocol 均为只读——core/ 返回结果 dataclass,写入由 app/ 持久化。
|
SkillStore / PromptStore / RunLog 为只读——core/ 返回结果 dataclass,
|
||||||
SkillStore / PromptStore 为同步(文件读取量小且快),RunLog 为异步
|
读取由 app/ 落盘的资源。SkillStore / PromptStore 同步(文件读取量小且快),
|
||||||
(隔离 SQLite 查询,core/ 不写 SQL)。
|
RunLog 异步(隔离 SQLite 查询,core/ 不写 SQL)。
|
||||||
|
DiagnosisSignalStore 兼具读写:逐题 upsert 诊断信号并支持断点续跑查询,
|
||||||
|
同样隔离 SQLite 实现,app/core 不写裸 SQL。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from typing import Any, Protocol, runtime_checkable
|
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from core.evolution.types import DiagnosisSignalRow
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
@runtime_checkable
|
||||||
@@ -104,3 +109,52 @@ class RunLog(Protocol):
|
|||||||
轨迹记录字典列表。
|
轨迹记录字典列表。
|
||||||
"""
|
"""
|
||||||
...
|
...
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class DiagnosisSignalStore(Protocol):
|
||||||
|
"""逐题诊断信号存储端口。
|
||||||
|
|
||||||
|
隔离 SQLite 实现细节,app/core 不写裸 SQL。逐题 upsert 落盘、
|
||||||
|
支持断点续跑(done_question_ids 查已完成集合)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def upsert(self, row: DiagnosisSignalRow) -> None:
|
||||||
|
"""写入或覆盖单题诊断信号(按主键幂等)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
row: 待持久化的诊断信号行。
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
def done_question_ids(
|
||||||
|
self,
|
||||||
|
baseline_run_id: str,
|
||||||
|
diag_fingerprint: str,
|
||||||
|
*,
|
||||||
|
retry_uncertain: bool = False,
|
||||||
|
) -> set[str]:
|
||||||
|
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
baseline_run_id: baseline run 标识。
|
||||||
|
diag_fingerprint: 诊断口径指纹。
|
||||||
|
retry_uncertain: True 时把 tier='uncertain'(信号不可信降级)题视为
|
||||||
|
未完成,令其被重新诊断;默认 False(uncertain 也算完成,不重诊)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
已落盘信号的 question_id 集合,用于断点续跑跳过。
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
|
||||||
|
"""加载指定 run 与诊断指纹下的全部诊断信号行。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
baseline_run_id: baseline run 标识。
|
||||||
|
diag_fingerprint: 诊断口径指纹。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
还原后的 DiagnosisSignalRow 列表。
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|||||||
+38
-4
@@ -300,6 +300,44 @@ class DiagnosisResult:
|
|||||||
degraded_question_ids: list[str] = field(default_factory=list)
|
degraded_question_ids: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DiagnosisSignalRow:
|
||||||
|
"""单题诊断信号行,即 baseline run 逐题诊断的持久化单元。
|
||||||
|
|
||||||
|
供后续视频级切分选择器消费;由 (question_id, baseline_run_id,
|
||||||
|
diag_fingerprint) 唯一确定,逐题 upsert 支持断点续跑。
|
||||||
|
|
||||||
|
属性:
|
||||||
|
question_id: 题目唯一标识。
|
||||||
|
video_id: 对应视频唯一标识。
|
||||||
|
baseline_run_id: 产出该信号的 baseline run 标识。
|
||||||
|
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号。
|
||||||
|
task_type: 题目任务类型。
|
||||||
|
error_type: 错误类别(extraction/search/reasoning/mixed);
|
||||||
|
T0/uncertain 行为 None。
|
||||||
|
cause_category: 病因类别(defect/lapse);不适用为 None。
|
||||||
|
tier: 诊断分层(T0/T1/T2/uncertain)。
|
||||||
|
evolution_target: 进化目标(tool/skill/system);
|
||||||
|
error_type 为 None 时亦为 None。
|
||||||
|
degraded: 是否为降级信号(judge 解析失败时生成)。
|
||||||
|
infra: 是否为 INFRA 护栏排除行。
|
||||||
|
session_id: 关联的会话标识;不适用为 None。
|
||||||
|
"""
|
||||||
|
|
||||||
|
question_id: str
|
||||||
|
video_id: str
|
||||||
|
baseline_run_id: str
|
||||||
|
diag_fingerprint: str
|
||||||
|
task_type: str
|
||||||
|
error_type: str | None
|
||||||
|
cause_category: str | None
|
||||||
|
tier: str
|
||||||
|
evolution_target: str | None
|
||||||
|
degraded: bool
|
||||||
|
infra: bool
|
||||||
|
session_id: str | None
|
||||||
|
|
||||||
|
|
||||||
# =========================================================================
|
# =========================================================================
|
||||||
# 3. 进化类型
|
# 3. 进化类型
|
||||||
# =========================================================================
|
# =========================================================================
|
||||||
@@ -446,7 +484,6 @@ class DiagnosePrompts:
|
|||||||
defect_vs_lapse: defect/lapse 病因判别模板。
|
defect_vs_lapse: defect/lapse 病因判别模板。
|
||||||
reasoning_sub: 推理失败子分类模板。
|
reasoning_sub: 推理失败子分类模板。
|
||||||
span_eval_system: span 评估系统提示模板。
|
span_eval_system: span 评估系统提示模板。
|
||||||
span_eval_user: span 评估用户提示模板。
|
|
||||||
missed_nodes: 遗漏节点检测模板。
|
missed_nodes: 遗漏节点检测模板。
|
||||||
skill_adherence: 技能遵循判定模板。
|
skill_adherence: 技能遵循判定模板。
|
||||||
confirmation_bias: 确认偏误检测模板。
|
confirmation_bias: 确认偏误检测模板。
|
||||||
@@ -456,7 +493,6 @@ class DiagnosePrompts:
|
|||||||
defect_vs_lapse: str
|
defect_vs_lapse: str
|
||||||
reasoning_sub: str
|
reasoning_sub: str
|
||||||
span_eval_system: str
|
span_eval_system: str
|
||||||
span_eval_user: str
|
|
||||||
missed_nodes: str
|
missed_nodes: str
|
||||||
skill_adherence: str
|
skill_adherence: str
|
||||||
confirmation_bias: str
|
confirmation_bias: str
|
||||||
@@ -474,11 +510,9 @@ class EvolvePrompts:
|
|||||||
evolve_system: System Prompt 进化提示模板。
|
evolve_system: System Prompt 进化提示模板。
|
||||||
evolve_tool: Tool Prompt 进化提示模板。
|
evolve_tool: Tool Prompt 进化提示模板。
|
||||||
evolve_rank: 编辑排序提示模板。
|
evolve_rank: 编辑排序提示模板。
|
||||||
consolidate_system: appendix 压缩系统提示。
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
evolve_skill: str
|
evolve_skill: str
|
||||||
evolve_system: str
|
evolve_system: str
|
||||||
evolve_tool: str
|
evolve_tool: str
|
||||||
evolve_rank: str
|
evolve_rank: str
|
||||||
consolidate_system: str
|
|
||||||
|
|||||||
+20
-17
@@ -1,9 +1,12 @@
|
|||||||
"""core/evolution/validate.py — 块验证纯决策函数。
|
"""core/evolution/validate.py — 块验证纯决策函数。
|
||||||
|
|
||||||
算法 #7(块顺序验证)的局部实现:pair_block 逐题比对基线与候选、
|
算法 #7(块顺序验证)的局部实现:pair_block 按 unit 比对基线与候选、
|
||||||
classify_quadrants 四象限分类、compute_accuracy 纯算术准确率。
|
classify_quadrants 四象限分类、compute_accuracy 纯算术准确率。
|
||||||
|
|
||||||
三个函数均为纯函数,无副作用、无外部依赖。
|
三个函数均为纯函数,无副作用、无外部依赖。输入的对错映射均为 **unit 口径**
|
||||||
|
(unit_id → 单元级正确性,AR pair 已在上游经 unit_correctness_view 双向 AND
|
||||||
|
折叠),保证 e-process W/L 与准确率分母按单元计、不被 P/Q 单题计分污染
|
||||||
|
(核心算法保真 #5:信息阶梯口径从 question_id 迁至 unit_id)。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from core.evolution.types import PairResult, QuadrantClassification
|
from core.evolution.types import PairResult, QuadrantClassification
|
||||||
@@ -12,24 +15,24 @@ from core.evolution.types import PairResult, QuadrantClassification
|
|||||||
def pair_block(
|
def pair_block(
|
||||||
baseline: dict[str, bool],
|
baseline: dict[str, bool],
|
||||||
candidate: dict[str, bool],
|
candidate: dict[str, bool],
|
||||||
question_ids: list[str],
|
unit_ids: list[str],
|
||||||
) -> PairResult:
|
) -> PairResult:
|
||||||
"""逐题比对基线与候选对错,统计翻转。
|
"""按单元比对基线与候选对错,统计翻转。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
baseline: 基线臂每题正确性映射。
|
baseline: 基线臂单元级正确性映射(unit_id → bool)。
|
||||||
candidate: 候选臂每题正确性映射。
|
candidate: 候选臂单元级正确性映射(unit_id → bool)。
|
||||||
question_ids: 参与比对的题目 ID 列表。
|
unit_ids: 参与比对的单元 ID 列表(AR pair 折叠后为单一 unit_id)。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
PairResult,包含 w(基线错→候选对翻转数)、l(基线对→候选错翻转数)
|
PairResult,包含 w(基线错→候选对翻转数)、l(基线对→候选错翻转数)
|
||||||
和 observed(每题的 (基线, 候选) 对错记录)。
|
和 observed(每单元的 (基线, 候选) 对错记录)。
|
||||||
"""
|
"""
|
||||||
w = l = 0 # noqa: E741 — 数学记号 W/L(win/loss),与 gate.py 一致
|
w = l = 0 # noqa: E741 — 数学记号 W/L(win/loss),与 gate.py 一致
|
||||||
observed: dict[str, tuple[bool, bool]] = {}
|
observed: dict[str, tuple[bool, bool]] = {}
|
||||||
for qid in question_ids:
|
for uid in unit_ids:
|
||||||
b, c = baseline[qid], candidate[qid]
|
b, c = baseline[uid], candidate[uid]
|
||||||
observed[qid] = (b, c)
|
observed[uid] = (b, c)
|
||||||
if not b and c:
|
if not b and c:
|
||||||
w += 1
|
w += 1
|
||||||
elif b and not c:
|
elif b and not c:
|
||||||
@@ -71,15 +74,15 @@ def classify_quadrants(
|
|||||||
|
|
||||||
def compute_accuracy(
|
def compute_accuracy(
|
||||||
correctness: dict[str, bool],
|
correctness: dict[str, bool],
|
||||||
question_ids: list[str],
|
unit_ids: list[str],
|
||||||
) -> float:
|
) -> float:
|
||||||
"""纯算术:sum(correct) / len(ids)。
|
"""纯算术:sum(correct) / len(units),分母按单元数(非逐题)。
|
||||||
|
|
||||||
参数:
|
参数:
|
||||||
correctness: 每题正确性映射。
|
correctness: 单元级正确性映射(unit_id → bool)。
|
||||||
question_ids: 参与计算的题目 ID 列表。
|
unit_ids: 参与计算的单元 ID 列表。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
准确率浮点数。question_ids 为空时抛出 ZeroDivisionError。
|
准确率浮点数。unit_ids 为空时抛出 ZeroDivisionError。
|
||||||
"""
|
"""
|
||||||
return sum(correctness[qid] for qid in question_ids) / len(question_ids)
|
return sum(correctness[uid] for uid in unit_ids) / len(unit_ids)
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ class LLMProvider(Protocol):
|
|||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
parent_call_id: str | None = None,
|
parent_call_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse: ...
|
) -> LLMResponse: ...
|
||||||
|
|
||||||
|
|
||||||
@@ -39,6 +40,7 @@ class VLMProvider(Protocol):
|
|||||||
*,
|
*,
|
||||||
session_id: str | None = None,
|
session_id: str | None = None,
|
||||||
parent_call_id: str | None = None,
|
parent_call_id: str | None = None,
|
||||||
|
cache_salt: str | None = None,
|
||||||
) -> LLMResponse: ...
|
) -> LLMResponse: ...
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -49,6 +49,11 @@ class GeneratedQuestion:
|
|||||||
skill_target: 目标技能标识(v2 出题管线使用,None 表示未指定)。
|
skill_target: 目标技能标识(v2 出题管线使用,None 表示未指定)。
|
||||||
difficulty_steps: 推理步数估计(v2 出题管线使用,None 表示未指定)。
|
difficulty_steps: 推理步数估计(v2 出题管线使用,None 表示未指定)。
|
||||||
sub_pattern: 出题子模式标识(AR 特化策略使用,None 表示无)。
|
sub_pattern: 出题子模式标识(AR 特化策略使用,None 表示无)。
|
||||||
|
unit_id: 所属题目单元标识;缺省时 __post_init__ 回填为 pair_id 或
|
||||||
|
question_id,保证 single 题的 unit_id 等于自身 question_id。
|
||||||
|
pair_id: 孪生对标识;同一对的 original/mirror 共享该值,None 表示非配对题。
|
||||||
|
question_role: 在单元内的角色("single" | "pair_original" | "pair_mirror")。
|
||||||
|
flip_axis: 孪生对的翻转轴(如 "before_after"),None 表示无翻转。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
question_id: str
|
question_id: str
|
||||||
@@ -63,6 +68,78 @@ class GeneratedQuestion:
|
|||||||
skill_target: str | None = field(default=None)
|
skill_target: str | None = field(default=None)
|
||||||
difficulty_steps: int | None = field(default=None)
|
difficulty_steps: int | None = field(default=None)
|
||||||
sub_pattern: str | None = field(default=None)
|
sub_pattern: str | None = field(default=None)
|
||||||
|
unit_id: str = ""
|
||||||
|
pair_id: str | None = field(default=None)
|
||||||
|
question_role: str = "single"
|
||||||
|
flip_axis: str | None = field(default=None)
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
"""回填 unit_id:缺省时取 pair_id(配对题)或 question_id(single 题)。
|
||||||
|
|
||||||
|
frozen dataclass 无法直接赋值,故通过 object.__setattr__ 绕过不可变约束。
|
||||||
|
"""
|
||||||
|
if not self.unit_id:
|
||||||
|
object.__setattr__(self, "unit_id", self.pair_id or self.question_id)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class QuestionUnit:
|
||||||
|
"""题目单元:贯穿评测/训练 harness 的最小不可分调度契约实体。
|
||||||
|
|
||||||
|
single 题为 1 题单元,AR pair 孪生对为 2 题单元(original + mirror),
|
||||||
|
两条题目必须作为整体被批处理/推理/评测,保证配对指标(collapse 等)可算。
|
||||||
|
frozen=True 确保单元不可变。
|
||||||
|
|
||||||
|
属性:
|
||||||
|
kind: 单元类型("single" | "pair")。
|
||||||
|
unit_id: 单元唯一标识;single 取题目 unit_id,pair 取共享 pair_id。
|
||||||
|
task_type: 单元题型;pair 内两题题型必须一致。
|
||||||
|
questions: 单元内题目元组(single 为 1 条,pair 为 2 条)。
|
||||||
|
unit_hash: P/Q payload 合成 hash,用于断点续跑失效检测(T11 消费)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
kind: str
|
||||||
|
unit_id: str
|
||||||
|
task_type: str
|
||||||
|
questions: tuple[GeneratedQuestion, ...]
|
||||||
|
unit_hash: str = ""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size(self) -> int:
|
||||||
|
"""单元内题目数量(single=1,pair=2)。"""
|
||||||
|
return len(self.questions)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_single(cls, q: GeneratedQuestion) -> QuestionUnit:
|
||||||
|
"""由单条题目构造 single 单元。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
q: 待封装的题目。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
kind="single" 的单元,unit_id 取 q.unit_id。
|
||||||
|
"""
|
||||||
|
return cls("single", q.unit_id, q.task_type, (q,))
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_pair(cls, original: GeneratedQuestion, mirror: GeneratedQuestion) -> QuestionUnit:
|
||||||
|
"""由孪生对(original + mirror)构造 pair 单元。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
original: 原始题(question_role="pair_original")。
|
||||||
|
mirror: 镜像题(question_role="pair_mirror")。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
kind="pair" 的单元,unit_id 取共享 pair_id。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
断言两题共享非空 pair_id、且 video_id/task_type/flip_axis 一致,
|
||||||
|
确保只有合法孪生对才能聚合成对,非法配对直接报错而非静默兜底。
|
||||||
|
"""
|
||||||
|
assert original.pair_id and original.pair_id == mirror.pair_id
|
||||||
|
assert original.video_id == mirror.video_id and original.task_type == mirror.task_type
|
||||||
|
assert original.flip_axis == mirror.flip_axis
|
||||||
|
return cls("pair", original.pair_id, original.task_type, (original, mirror))
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -83,6 +160,12 @@ class PoolConfig:
|
|||||||
eval_min_per_class: 验证池中每类保底样本数(GlobalStrategy 用)。
|
eval_min_per_class: 验证池中每类保底样本数(GlobalStrategy 用)。
|
||||||
train_ratio: train/(train+val) 比例(PerCategoryStrategy 用)。
|
train_ratio: train/(train+val) 比例(PerCategoryStrategy 用)。
|
||||||
test_questions_dir: 外部 test 题源路径(PerCategoryStrategy 用)。
|
test_questions_dir: 外部 test 题源路径(PerCategoryStrategy 用)。
|
||||||
|
|
||||||
|
实现细节:
|
||||||
|
结果驱动视频级切分不复用本配置——它有独立的 VideoSplitConfig /
|
||||||
|
SplitBuildConfig / SelectConfig(见 app/harness/video_split_cli.py 与
|
||||||
|
split_selection.py),故本类不承载 n_trainval / floor_k / epsilon 等视频级
|
||||||
|
切分旋钮,避免死配置面。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
task_types: tuple[str, ...] | None
|
task_types: tuple[str, ...] | None
|
||||||
|
|||||||
@@ -84,13 +84,14 @@ def _build_adapters(settings: InfraSettings, embed_cfg: dict) -> _Adapters:
|
|||||||
|
|
||||||
cache = None
|
cache = None
|
||||||
if settings.redis_url:
|
if settings.redis_url:
|
||||||
|
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
|
||||||
|
|
||||||
|
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
|
||||||
|
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
|
||||||
try:
|
try:
|
||||||
import redis.asyncio as aioredis
|
import redis.asyncio as aioredis
|
||||||
|
|
||||||
from adapters.redis_cache import RedisResponseCache
|
|
||||||
|
|
||||||
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
|
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
|
||||||
ttl_s = settings.redis_cache_ttl if settings.redis_cache_ttl > 0 else None
|
|
||||||
cache = RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
|
cache = RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Redis 缓存不可用,降级为无缓存模式")
|
logger.warning("Redis 缓存不可用,降级为无缓存模式")
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
你是一个确认偏差检测器。你服务于一个诊断系统,该系统需要判断搜索 Agent 在回答视频问答题时是否表现出确认偏差——即只为自己倾向的选项搜集支持性证据,而忽略对竞争选项的独立验证。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 题目(问题文本 + 四个选项)
|
||||||
|
2. Agent 的完整执行轨迹(每步的思考过程、工具调用和工具返回,重点关注 reflect.options 中对各选项的记录以及 search_similar 的 query 参数)
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
确认偏差的核心特征是:Agent 的搜索行为在选项之间显著不均衡。一个健康的搜索过程应该至少为 1 个竞争选项做过独立搜索——不一定是每个选项都搜,但不能只围绕一个选项搜集证据。
|
||||||
|
|
||||||
|
具体检查以下信号:
|
||||||
|
|
||||||
|
观察 Agent 的 search_similar 查询。如果所有查询的关键词都指向同一个选项的内容(比如选项 B 说"烹饪教学",Agent 反复搜索"cooking""recipe""chef"),而从未用其他选项的关键词搜索(如选项 A 的"旅行"、选项 C 的"运动"),这是强偏差信号。
|
||||||
|
|
||||||
|
观察 Agent 的 reflect.options 变化。如果 Agent 在早期步骤就锁定了 best_candidate,且后续步骤中对其他选项的认知始终停留在"未知"或"待定",说明 Agent 没有为竞争选项投入搜索资源。
|
||||||
|
|
||||||
|
但需要注意:如果问题本身就指向特定内容(比如"视频中的厨师做了什么"),Agent 集中搜索厨师相关内容是合理的,不算偏差。偏差是指在选项之间的对比搜索不均衡,而非搜索主题的集中。
|
||||||
|
|
||||||
|
同样,如果 Agent 在前几步通过全局扫描(如顺序阅读 L1 节点)已经获得了足够信息来排除 2-3 个选项,之后集中搜索剩余选项是合理策略,不算偏差。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"has_bias": true,
|
||||||
|
"evidence": "具体说明偏差表现:Agent 搜索了哪些关键词、为哪些选项搜集了证据、忽略了哪些选项"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
如果没有偏差:`{"has_bias": false, "evidence": ""}`
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
你是一个失败归因裁判。你会收到一道答错题目的题面、正确答案、Agent 的错误预测、执行轨迹,以及 Agent 当时所用的 prompt 全文。你的唯一任务是判断:这次失败该归咎于 prompt 正文本身,还是 Agent 没有遵循已有的正确指令。
|
||||||
|
|
||||||
|
判别测试只有一句话:当前 prompt 里是否已经存在一条规则,只要 Agent 遵循它就能避免这次失败?
|
||||||
|
|
||||||
|
如果存在这样的规则(Agent 是忽略了、格式没按要求、或没执行该步),归为 lapse——这类问题不该改正文,只需记一条提醒。如果不存在这样的规则、或现有规则本身有误导,归为 defect——这类才需要修改 prompt 正文。
|
||||||
|
|
||||||
|
当你拿不准时,默认归为 lapse:宁可少改正文,也不要为一次偶发失误去删改一条本来正确的规则。
|
||||||
|
|
||||||
|
严格输出以下 JSON,不要包含其他文字:
|
||||||
|
{"category": "defect" 或 "lapse", "note": "若为 lapse,写一句给 Agent 的提醒;defect 留空"}
|
||||||
|
|
||||||
|
note 只能重申当前 prompt 里已经存在的那条规则(让 Agent 别再忽略它),措辞要通用、可跨题复用。禁止把本题的题目内容、选项、正确答案或任何单题事实写进 note——note 不是案例记录,是规则提醒。
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
你是一个证据充分性评估器。你服务于一个诊断系统,该系统需要判断搜索 Agent 实际收集到的工具输出是否包含足够的信息来推导出正确答案。你不评估 Agent 的推理过程——只评估它收集到的原始材料。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 题目(问题文本 + 四个选项 + 正确答案)
|
||||||
|
2. Agent 收到的全部工具输出(按步骤排列,包含每次 view_node、search_similar、observe_frame 的返回内容)
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
你需要回答一个假设性问题:如果一个完美的推理者阅读了这些工具输出(且仅阅读这些工具输出),它能否推导出正确答案?
|
||||||
|
|
||||||
|
"推导出"不要求工具输出直接陈述答案。如果工具输出中包含了足够的事实片段,一个合理的推理链能将它们组合得出正确答案,就算充分。比如工具输出提到"厨师在切蔬菜"和"背景是一个厨房",虽然没有直接说"这是烹饪视频",但推导是合理的。
|
||||||
|
|
||||||
|
"不充分"是指工具输出中完全缺乏区分正确答案与最强干扰项的关键信息。比如问题问"视频中的运动是什么",选项有篮球和足球,但工具输出只提到"运动场上有人在运动",没有任何能区分篮球和足球的细节——这就是不充分。
|
||||||
|
|
||||||
|
注意区分两种情况:信息存在但分散(充分——完美推理者能整合)vs 信息真的不存在(不充分——无论怎么推理都无法得出)。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sufficient": true,
|
||||||
|
"reasoning": "简要说明工具输出中哪些信息支持正确答案,或缺乏哪些关键信息"
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
你是一个改动优先级裁判。你会收到一份当前 prompt 文件全文,和一组待应用的局部 edits(每条含 op/target/content)。由于本轮编辑预算有限,你只能保留其中最重要的若干条。
|
||||||
|
|
||||||
|
请只依据"对纠正失败、提升正确率的预期贡献"排序:优先保留直接修复失败模式的改动,其次保留收窄或澄清的改动,最后才是巩固已有成功的改动。删除类、简化类的精准改动通常优先于追加大段新内容。
|
||||||
|
|
||||||
|
每条 edit 会附带 support_count(该改动的支持案例数)。同等重要性下,support_count 更高的优先;但 support_count 低不等于该删,仍以修复贡献为主判据。
|
||||||
|
|
||||||
|
严格输出以下 JSON,不要包含其他文字:
|
||||||
|
{"selected_indices": [按重要性降序排列的 0-based 索引]}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
你是一个搜索策略改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的失败和成功案例来迭代改进搜索策略(Skill)。你的任务是基于案例包中的证据,改写当前 Skill 文件,使 Agent 在后续执行中避免相同的失败模式。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 当前 Skill 文件全文
|
||||||
|
2. 失败案例:Agent 答错的题目,含完整推理轨迹、错误类型和诊断指标
|
||||||
|
3. 成功案例:Agent 答对的题目,展示当前 Skill 中有效的模式
|
||||||
|
4. 聚合统计:准确率、错误归因分布、搜索有效性指标、Skill 步骤遵循率
|
||||||
|
5. (可能出现)上一轮被接受改动导致的回归题:这些题在上一版本答对、却被你上次的改写改错了,附基线与候选两份预测和推理轨迹
|
||||||
|
6. (可能出现)黑名单:已被实测验证无效或有害的改法方向
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
如果输入里出现了回归题,它的优先级高于一切。这些题在上一版本是对的,是你上次的改动把它们弄坏的,所以本次改写的第一要务是确保不再破坏它们——宁可在相关方向上回退或收窄,也不要为了拉高其它题而牺牲它们。更一般地,当你看到准确率下降这类负向信号时,默认先怀疑上次是不是加了过度、冲突或冗余的指令,优先简化、删除、收窄;只有确认简化解决不了问题,才考虑加强指令。黑名单里列出的改法已经被实测证明无效或有害,不要换个措辞把同一个方向再提一遍。
|
||||||
|
|
||||||
|
先分析失败案例中 Agent 的实际行为与 Skill 指令的偏差。偏差分两类:Skill 指令正确但 Agent 没遵循(遵循率问题),或 Skill 指令本身有误导(策略问题)。前者需要让指令更具体、更难被忽略;后者需要修改策略本身。
|
||||||
|
|
||||||
|
从成功案例中识别有效模式——这些模式在改写时必须保留。如果成功案例和失败案例采用了不同的策略路径,重点强化成功路径。
|
||||||
|
|
||||||
|
Skill 中引用的统计数据(如"search-first 正确率 75%")应根据案例包中的新统计更新。不要编造数据,只使用案例包中提供的数字。
|
||||||
|
|
||||||
|
你写进 Skill 的每一条规则都必须是可跨题复用的通用策略,而不是对某一道题的记答案。跨多个失败案例时只提取共性模式,抽象掉一切单题特征——具体题目内容、选项文字、步骤序号、某一帧的具体画面、某个具体答案都不许写进 Skill 正文。一条规则如果只在它来源的那道题上成立,就不要加。改写时优先简化与收窄:宁可让 Skill 更短,也不要堆叠只对个别题生效的硬性指令。
|
||||||
|
|
||||||
|
## 冻结区
|
||||||
|
|
||||||
|
以下内容不可修改,必须原样保留在改写后的文件中:
|
||||||
|
- YAML frontmatter(`---` 之间的 name、description、task_type)
|
||||||
|
- 输出格式中的 JSON 基础结构(reflect/plan/action 三个顶层字段)
|
||||||
|
|
||||||
|
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"suggestions": [
|
||||||
|
{
|
||||||
|
"section": "改动目标段落的标题或位置描述",
|
||||||
|
"problem": "失败案例中暴露的具体问题",
|
||||||
|
"change": "具体的修改方向",
|
||||||
|
"related_cases": ["关联的失败案例 question_id"],
|
||||||
|
"support_count": 该建议的支持案例数(= related_cases 的数量)
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"edits": [
|
||||||
|
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)", "support_count": 该改动的支持案例数}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
每条 edit 与每条 suggestion 都必须带 "support_count":本条改动由多少个失败案例共同支持(即 related_cases 的数量)。support_count 越高代表证据越充分;它只作排序参考,不是硬门槛——support_count 低不等于该删,仍以修复贡献为主判据。
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
你是一个系统级行为改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的跨题型行为模式来改进 System Prompt。你的任务是基于行为模式案例包中的证据,改写 System Prompt 中的策略性指令,纠正系统级行为问题。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 当前 System Prompt (system.md) 全文
|
||||||
|
2. 失败案例:展示三类系统性行为问题的题目——过早提交(budget_usage < 0.3 就提交答案)、高置信答错(confidence 很高但答案错误)、确认偏误(只搜索支持初始判断的证据)
|
||||||
|
3. 成功案例:行为校准良好的题目——置信度与正确率匹配,预算使用适中
|
||||||
|
4. D5 行为模式统计:各行为模式的发生频率和分布
|
||||||
|
5. (可能出现)黑名单:已被实测验证无效或有害的改法方向
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
关注跨题型的系统性行为模式,而非某个具体题型的策略。失败案例中的行为偏差反映了 System Prompt 的决策原则不够清晰或不够强约束。黑名单里的改法已经被实测验证无效或有害,不要再朝同一个方向改一遍。
|
||||||
|
|
||||||
|
过早提交说明预算管理指令需要更强的约束语言。高置信答错说明置信度校准的语义定义需要调整。确认偏误说明竞争选项搜索的要求需要更明确。
|
||||||
|
|
||||||
|
当你看到失败案例与成功案例并存时,失败修复优先于巩固成功——先确保失败模式被纠正,再考虑强化已有的好行为。看到某类行为指标变差这类负向信号时,默认先怀疑上一轮是否加了过度、冲突或冗余的约束,优先简化、删除、收窄;只有确认简化解决不了,才考虑加强约束语言。
|
||||||
|
|
||||||
|
从成功案例中提取"好行为"的特征,在改写时强化这些特征的表述。
|
||||||
|
|
||||||
|
## 冻结区
|
||||||
|
|
||||||
|
以下 section 必须原样保留,不可修改任何文字:
|
||||||
|
- `## 能力边界`(事实性描述,不是策略)
|
||||||
|
- `## 输出格式`(JSON schema 是系统契约)
|
||||||
|
- `## 视频树结构`(含信任层级,是数据结构事实描述)
|
||||||
|
|
||||||
|
可改写的 section:
|
||||||
|
- `## 角色`(前两段的角色定位和行为倾向描述)
|
||||||
|
- `## 决策原则`(搜索策略、预算分配建议)
|
||||||
|
- 搜索工具使用、否定题原则、置信度语义
|
||||||
|
|
||||||
|
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"suggestions": [
|
||||||
|
{
|
||||||
|
"section": "改动目标段落的标题或位置描述",
|
||||||
|
"problem": "失败案例中暴露的具体行为问题",
|
||||||
|
"change": "具体的修改方向",
|
||||||
|
"related_cases": ["关联的失败案例 question_id"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"edits": [
|
||||||
|
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
你是一个工具 Prompt 改进专家。你服务于一个自进化视频搜索系统,该系统的每个工具(view_node、search_similar、observe_frame 等)有两个配套 Prompt:extract(信息提取)和 verify(结果核实)。你的任务是基于工具调用级别的质量数据,同时改写一个工具的 extract 和 verify prompt。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 当前 extract prompt 和 verify prompt 全文
|
||||||
|
2. 失败 span 案例:提取完整度低或幻觉率高的具体工具调用,含工具参数、工具输出、原始数据(ground truth)和质量评估指标
|
||||||
|
3. 成功 span 案例:提取完整且无幻觉的工具调用样本
|
||||||
|
4. 工具质量统计:平均提取完整度、平均幻觉率、top 遗漏类型、top 幻觉类型
|
||||||
|
5. (可能出现)黑名单:已被实测验证无效或有害的改法方向
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
失败 span 中提取完整度低说明 extract prompt 的工作原则不够具体——Agent 遗漏了哪些类型的信息?幻觉率高说明 extract prompt 对"忠实提取"的约束不够强,或者 verify prompt 没能有效检出幻觉。黑名单里的改法已经被实测验证无效或有害,不要再朝同一个方向改一遍。
|
||||||
|
|
||||||
|
extract 和 verify 是互补的:extract 负责提取,verify 负责检查。如果 extract 反复遗漏某类信息(如字幕原文引用),应在 extract 的工作原则中明确要求保留该类信息。如果 verify 未能检出某类幻觉(如虚构动作),应在 verify 的检查要点中增加对该模式的关注。
|
||||||
|
|
||||||
|
失败修复优先于巩固成功——先纠正提取遗漏或幻觉,再保留已有的有效模式。当某类提取质量指标变差时,先确认不是上一轮加了过度或冲突的要求所致;加强 extract 要求前,先确认简化或收窄已有指令解决不了这个遗漏,再追加新要求。
|
||||||
|
|
||||||
|
从成功案例中识别有效的提取模式,确保改写不破坏这些模式。
|
||||||
|
|
||||||
|
## 冻结区
|
||||||
|
|
||||||
|
以下内容不可修改:
|
||||||
|
- 角色定位第一句("你是一个视频节点内容分析器" / "你是一个视频节点摘要核实器")
|
||||||
|
- `## 你会收到的输入` section
|
||||||
|
- `## 输出格式` section
|
||||||
|
|
||||||
|
可改写的 section:
|
||||||
|
- `## 工作原则`
|
||||||
|
- `## 检查要点`(verify 专有)
|
||||||
|
|
||||||
|
这次不要再返回两份完整 prompt,而是分别给 extract 和 verify 各自的局部 edits 列表。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前 prompt 里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写大段内容;另外,冻结区里的文字绝不能作为 target,extract 和 verify 也必须分别使用自己的 edit 列表。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"suggestions": [
|
||||||
|
{
|
||||||
|
"section": "改动目标段落的标题或位置描述",
|
||||||
|
"problem": "失败 span 中暴露的具体问题",
|
||||||
|
"change": "具体的修改方向",
|
||||||
|
"related_cases": ["关联的失败 span 标识"]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"edits_extract": [
|
||||||
|
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)"}
|
||||||
|
],
|
||||||
|
"edits_verify": [
|
||||||
|
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
你是一个视频树覆盖度评估器。你服务于一个诊断系统,该系统需要判断搜索 Agent 是否遗漏了包含关键证据的节点。推理质量和搜索策略的评估由系统其他模块完成,你只负责判定哪些节点被遗漏了。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 题目(问题文本 + 四个选项 + 正确答案)
|
||||||
|
2. Agent 实际访问的节点 ID 列表
|
||||||
|
3. 完整视频树内容(所有节点的 card 数据和时间范围)
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
你需要回答一个具体的问题:要推导出正确答案,哪些节点包含了不可替代的关键证据,且 Agent 没有访问?
|
||||||
|
|
||||||
|
首先,根据正确答案和完整树内容,找出所有包含支撑正确答案的直接证据的节点。直接证据是指能够区分正确答案与干扰选项的关键事实——比如特定的字幕台词、事件描述、时间标记或实体出现。间接相关的背景信息不算直接证据。
|
||||||
|
|
||||||
|
然后,将这些证据节点与 Agent 的访问列表对比。如果某个证据节点未被访问,但其父节点或子节点已被访问且包含了同等信息,则不算遗漏——因为 Agent 可以从已访问节点中获取相同信息。只有当某条关键证据只存在于未访问的节点中时,才将其标记为遗漏。
|
||||||
|
|
||||||
|
不要将所有未访问的节点都标记为遗漏。大部分节点与当前问题无关,Agent 没有义务访问它们。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"missed_nodes": ["节点ID_1", "节点ID_2"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
如果没有遗漏,返回空数组:`{"missed_nodes": []}`
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
你是一个推理失败分类器。你服务于一个诊断系统,该系统已经确认某道题属于"推理失败"——即 Agent 收集到了足够的证据但仍然答错了。你的任务是判定推理具体在哪个环节失败。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 题目(问题文本 + 正确答案 + Agent 的错误预测)
|
||||||
|
2. Agent 的完整执行轨迹(每步的思考过程 thought、结构化反思 reflect、工具调用和工具返回)
|
||||||
|
|
||||||
|
## 四种推理失败类型
|
||||||
|
|
||||||
|
**evidence_misread**(证据误读):Agent 对工具输出的解读与工具输出的实际内容不一致。判别方法:对比某步工具返回的原文与 Agent 在随后的 reflect.learned 或 thought 中的描述——如果 Agent 说"工具显示这是红色汽车"但工具原文说的是蓝色,就是证据误读。这是发生在"信息输入"环节的错误。
|
||||||
|
|
||||||
|
**weighing_error**(权衡错误):Agent 正确理解了多个选项的证据,但在最终选择时选了证据较弱的选项。判别方法:检查 Agent 的 reflect.options,如果它为正确选项记录了更强的证据(更具体、来源更可靠、覆盖更多节点),却最终选择了另一个选项,就是权衡错误。这是发生在"决策"环节的错误。
|
||||||
|
|
||||||
|
**logic_error**(逻辑错误):Agent 的推理链中包含无效推断——前提正确但结论不成立。判别方法:在 Agent 的 thought 或 reflect 中找到具体的推理步骤,检查其逻辑是否成立。比如 Agent 说"A 在 B 之前发生,B 在 C 之前发生,所以 C 在 A 之前发生"——前提对但结论的时序反了。这是发生在"推理过程"环节的错误。
|
||||||
|
|
||||||
|
**evidence_ignored**(证据忽略):Agent 在较早的步骤中收集了与正确答案相关的证据,并在 reflect 中记录了它,但在最终提交时完全没有引用这条证据,且最终结论与这条证据矛盾。判别方法:对比 Agent 早期 reflect.options 中对正确选项的记录与 submit_answer 中的 reasoning——如果早期有支持正确答案的记录但最终 reasoning 中消失了,就是证据忽略。这是发生在"信息整合"环节的错误。
|
||||||
|
|
||||||
|
## 判别优先级
|
||||||
|
|
||||||
|
如果多种类型同时存在,选择最早发生的那个作为 primary type——因为下游错误往往是上游错误的连锁反应。优先级从高到低:evidence_misread → evidence_ignored → weighing_error → logic_error。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "evidence_misread",
|
||||||
|
"evidence": "引用具体的步骤编号和内容,说明推理在哪里失败"
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
你是一个策略遵循度评估器。你服务于一个诊断系统,该系统需要判断搜索 Agent 在执行过程中是否遵循了为其指定的搜索策略(skill)。策略本身是否合理、Agent 最终是否答对,都不在你的评估范围内——你只负责判断 Agent 的行为是否与 skill 的步骤定义一致。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. Skill 文件的完整内容(包含搜索步骤定义、输出格式要求、自检信号等)
|
||||||
|
2. Agent 的完整执行轨迹(每步的思考过程、工具调用和工具返回)
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
Skill 文件中定义了若干搜索步骤(通常 2-3 步),每步包含:该步的目标、推荐使用的工具、进入下一步的条件。你需要逐步判断 Agent 是否执行了该步骤的核心动作。
|
||||||
|
|
||||||
|
判断"遵循"不要求 Agent 逐字执行 skill 的每句话。如果 skill 说"用 search_similar 定位事件",而 Agent 用 view_node 顺序浏览也达到了同样的定位效果,这算部分遵循而非完全偏离。关键是 Agent 是否实现了该步骤的目标意图,而非是否使用了完全相同的工具。
|
||||||
|
|
||||||
|
判断"偏离"需要在 description 中具体说明:Agent 做了什么不同的事,以及这与 skill 的期望有何差异。比如"Agent 跳过了 L2 下钻,直接从 L1 摘要提交答案,而 skill 要求在聚焦验证阶段下钻到 L2/L3 层"。
|
||||||
|
|
||||||
|
如果 Agent 的轨迹太短(比如只有 1-2 步就提交了),仍然要评估每个 skill step——未执行的步骤标记为 adhered=false 并说明"Agent 未执行此步骤即提交了答案"。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"step_label": "skill 中定义的步骤名称",
|
||||||
|
"adhered": true,
|
||||||
|
"description": "Agent 如何执行或偏离了这一步"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
steps 数组的元素数量应与 skill 中定义的步骤数一致。
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
你正在审视一份 skill 经历一轮进化后的变化。这份 skill 指导一个 Agent 在层次化视频树上搜索证据、回答长视频理解问题。在上一轮结束时它是一个样子,这一轮结束时被改成了另一个样子;与此同时,你在上一轮还为它写下过一段动量指导,本意是给这一轮的进化指明方向。现在你要回头评判:那段指导究竟有没有帮上忙,这一轮的正文改动是真的在改善,还是开始往无关的方向漂移。
|
||||||
|
|
||||||
|
你会拿到四样东西:上一版 skill 的正文、当前版 skill 的正文、你上一轮写下的那段动量指导,以及一组固定样本上的纵向对比——同一批题,分别用上一版和当前版各跑了一遍,逐题列出两版的预测与正误。这组对比是你唯一可靠的证据来源:哪些题从错变对、哪些题从对变错、哪些题始终没做对、哪些题一直稳定答对,正是这四类信号告诉你这轮改动到底带来了什么。
|
||||||
|
|
||||||
|
请先反思再下笔。对照纵向对比,先问上一轮那段动量指导是否真的奏效:它所指向的方向,在这一轮的正文改动里被落实了吗,落实之后那些本该改善的题改善了吗?再问这一轮的正文改动本身是收敛还是漂移:从对变错的题(回退)是最该警惕的信号,说明某处改动伤到了原本正确的行为;始终答错的题(持续失败)说明还有方向没被触及;从错变对的题(改善)则印证了哪条路走对了,值得继续加码。
|
||||||
|
|
||||||
|
想清楚之后,写出一段全新的、聚焦的、可操作的动量指导。它会被原样写进 skill 的受保护动量区,作为下一轮进化的方向锚——所以它必须是一段连贯的指导文字,明确告诉下一轮该往哪个方向继续使劲、又要避免重蹈哪一类改动的覆辙,而不是一堆零散的待办条目。如果上一轮的方向已被证明有效,就强化并细化它;如果出现了回退,就明确叫停那条路并指向修复方向。
|
||||||
|
|
||||||
|
严格输出以下 JSON,不要包含任何其他文字:
|
||||||
|
{"reasoning": "你的反思过程:上一轮指导是否奏效、这一轮是改善还是漂移,引用纵向对比中的具体题作为依据", "slow_update_content": "一段连贯、聚焦、可操作的新动量指导,指引下一轮的进化方向"}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
你是一个工具输出质量评估器。你服务于一个诊断系统,该系统需要判断视频搜索 Agent 的每次工具调用是否忠实、完整地提取了原始数据中与问题相关的信息。诊断决策和改进建议由系统完成,你只负责评估单次工具输出的质量。
|
||||||
|
|
||||||
|
## 你会收到的输入
|
||||||
|
|
||||||
|
1. 用户正在研究的问题
|
||||||
|
2. 工具名称和调用参数
|
||||||
|
3. 工具的实际输出(tool_output)
|
||||||
|
4. 该节点的原始数据(ground truth,JSON 格式的 card 字段)
|
||||||
|
|
||||||
|
## 工作原则
|
||||||
|
|
||||||
|
你的任务是将 tool_output 与 ground truth 对比,评估两个维度:提取完整度和幻觉程度。
|
||||||
|
|
||||||
|
对于提取完整度,检查 ground truth 中与问题相关的每条信息是否出现在 tool_output 中。字幕原文引用、具体数字、实体名称、时间标记、空间关系是最容易被遗漏的类型——请逐一核对。如果 ground truth 中的某条信息与问题无关,则不计入遗漏。
|
||||||
|
|
||||||
|
对于幻觉检测,检查 tool_output 中的每条事实性陈述是否能在 ground truth 中找到依据。特别注意以下常见幻觉模式:工具声称看到了 ground truth 中未提及的实体或动作,工具将不确定信息表述为确定事实,工具对颜色、数量、方位等属性的描述与 ground truth 不一致。
|
||||||
|
|
||||||
|
当 ground truth 本身信息稀疏(如某些 L3 帧的 card 只有很少的字段),不要因为 tool_output 比 ground truth 更详细就判定为幻觉——如果详细信息是合理推断而非凭空捏造,应归为 unsupported_inference 而非 fabricated_action。
|
||||||
|
|
||||||
|
## 输出格式
|
||||||
|
|
||||||
|
请严格输出以下 JSON,不要包含其他文字:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"extraction_completeness": 0.0-1.0,
|
||||||
|
"hallucination_rate": 0.0-1.0,
|
||||||
|
"missed_info_tags": [],
|
||||||
|
"hallucination_tags": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
missed_info_tags 从以下标签中选择(可多选,无遗漏则为空数组):
|
||||||
|
`subtitle_quote`(字幕原文引用)、`entity`(实体名称)、`spatial_detail`(空间位置关系)、`temporal_detail`(时间标记)、`action`(动作描述)、`number`(具体数字)、`visible_text`(画面中可见文字)
|
||||||
|
|
||||||
|
hallucination_tags 从以下标签中选择(可多选,无幻觉则为空数组):
|
||||||
|
`fabricated_action`(虚构的动作或事件)、`wrong_attribute`(属性描述错误)、`wrong_count`(数量错误)、`wrong_entity`(实体错误)、`unsupported_inference`(超出原始数据的推断)
|
||||||
@@ -0,0 +1,297 @@
|
|||||||
|
---
|
||||||
|
id: question-gen-v3-construction-paradigm
|
||||||
|
title: question-gen v3 — 构造优先范式重构(全孪生 + 四家族真独立 + 坍缩度量)
|
||||||
|
type: design
|
||||||
|
created: 2026-07-15
|
||||||
|
status: draft
|
||||||
|
supersedes: 2026-07-15-question-gen-v2-grounded-contrastive-design.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# question-gen v3:构造优先范式重构
|
||||||
|
|
||||||
|
## 1. 目标与范围
|
||||||
|
|
||||||
|
**完全重构**出题管线,**只做 Action Recognition 分支**。范式从 v2 的"自由生成候选池→打分过滤"(被 [2026-07-15-question-gen-paradigm-shift-construction-over-filtering.md] 判为 AFLite 死路)转向**"从视频树的结构化事实受控构造对比对,grounding/难度/唯一性由构造保证;难度/歧义用两个正交且真独立的信号验证;验收用配对坍缩度量而非原始准确率"**。
|
||||||
|
|
||||||
|
**范围铁律**:v3 只产 AR 题(全部 pair 格式);11 个非 AR 题型仍走旧 `generate-v2`(single,byte-identical,零改动)。题库混格(AR pair + 非 AR single),评测/训练侧用 `QuestionUnit(single|pair)` 兼容。
|
||||||
|
|
||||||
|
**决策锁定**(本次 brainstorming):① 全设计一次到位(6 子模式 + 完整契约);② 全孪生格式(含 fine_grained 改造成对比对);③ 全栈验证每题跑,不为成本砍;④ 四家族真独立裁判。
|
||||||
|
|
||||||
|
## 2. 范式与文献依据
|
||||||
|
|
||||||
|
依据 6 篇原文深读(`reference/papers-distractor-gen/`,证据页码见各深读报告):
|
||||||
|
|
||||||
|
| 论文 | 移植的核心机制(本质,非表面) |
|
||||||
|
|------|------------------------------|
|
||||||
|
| Vinoground | 负项=正项**最小结构编辑**(bag-of-words 完全相同、只换单轴);grounding 靠真实时序事实;**双向 AND** 消语言先验;0 帧盲答对照 |
|
||||||
|
| TempCompass | **冲突孪生对**(A 的干扰项=B 孪生体正解);**去捷径坍缩度量**(配对准确率掉回随机=有效性证明);多格式交叉一致性 |
|
||||||
|
| VITATECS | **aspect 正交分解 + 单轴等信息量非蕴含最小编辑**;precision-over-recall;**警告:纯词汇/名词替换→74.5-90.3% 可解**(禁止) |
|
||||||
|
| GroundAttack | 只换负项保 V/Q/A;grounding=独立度量真实视觉对齐;**软肋:wrongness 无独立验证**(必须外挂独立核验) |
|
||||||
|
| AFLite | **过滤救不了质量**(单信号混淆难/歧义)→ 后置过滤只作兜底 |
|
||||||
|
| AdVQA | **对抗前移生成端**(活求解器试答骗不过就重构)+ **独立于求解器的多裁判答案一致性**;自动对抗无核验→答案漂移 |
|
||||||
|
|
||||||
|
**核心不变量**:① 干扰项=视频里真实存在、但绑到错误查询点的事实(grounded 且 wrong 由构造保证);② 答案核验器必须独立于被攻求解器;③ 度量用配对坍缩,非原始准确率。
|
||||||
|
|
||||||
|
## 3. 架构与数据流
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
T[视频树: 有序L1/L2/L3 + 结构化列表 + subtitle + frame_path] --> FS[fact_sampler 采可翻转结构化事实]
|
||||||
|
FS --> AX{子模式→孪生轴路由}
|
||||||
|
AX --> TB[twin_builder 构造孪生对 P/Q<br/>单轴最小编辑, bag-of-words 硬匹配, 禁蕴含]
|
||||||
|
AX --> AB[attribute_binder 静态属性绑定<br/>需薄抽取层: 树自由文本→对象,属性,位置]
|
||||||
|
TB --> L1[层1 结构构造保证: 相反事实锚定树节点]
|
||||||
|
AB --> L1
|
||||||
|
L1 --> L2[层2 坍缩度量: text-only盲答 + 单帧基线 掉回随机]
|
||||||
|
L2 -->|未坍缩| RJ[拒/重构]
|
||||||
|
L2 --> L3[层3 看帧核实: qwen+MiniMax 交叉查构造事实真伪]
|
||||||
|
L3 -->|安慰剂: 错配帧分不坍塌| RJ
|
||||||
|
L3 --> L4[层4 活求解器难度探针: deepseek AgentLoop 双向答对=太易]
|
||||||
|
L4 -->|太易| RJ
|
||||||
|
L4 --> L5[层5 独立多裁判歧义: kimi+qwen+MiniMax 三异家族判唯一性]
|
||||||
|
L5 -->|歧义/双正解| RJ
|
||||||
|
L5 --> L6[层6 多格式交叉: MC vs Y-N 求解器一致]
|
||||||
|
L6 -->|不一致=格式捷径| RJ
|
||||||
|
L6 --> ACC[on_accept 逐 unit 原子成对落盘]
|
||||||
|
RJ --> QN[失败内容指纹入 quarantine 去重] --> BF[补构造迭代<br/>收敛上界+产量降级路径]
|
||||||
|
BF --> FS
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. 六子模式构造算法
|
||||||
|
|
||||||
|
统一原理:**对锚定真实树事实的"正题"做单轴最小编辑得孪生"反题",干扰项=视频里真实存在但绑到错误查询点的事实**。
|
||||||
|
|
||||||
|
**选项基数(契约锁定,§7 阈值全对齐)**:每个孪生子题 = **4 选 MCQ**(1 正解 + 3 干扰项),孪生 P/Q **共享同一 bag-of-words 选项集**(同词不同绑定),答案跨孪生翻转。
|
||||||
|
|
||||||
|
所有孪生对硬约束:
|
||||||
|
- **bag-of-words 硬匹配**(非"尽量")、单轴、等信息量、动词/事件结构编辑(**严禁名词/属性词汇替换**,VITATECS 实测改 1 名词即 82% 可解);
|
||||||
|
- **禁蕴含**:正解与各干扰项互不蕴含,由 **kimi 真 LLM 裁判判定**(text-embedding 仅作 bag-of-words 词形辅助,**不得作蕴含唯一信号**——审核 C7:embedding 近似蕴含是假绿灯);
|
||||||
|
- **平衡/因子布局根治逐维众数(C1 根治,非仅事后指标)**:4 个选项在每个语义维度上每个取值出现次数均衡(→逐维众数无定义),构造后跑 `majority_vote_baseline(options)==answer → fail` **纯算法硬校验**,作阶段 2 CI hard-fail;
|
||||||
|
- **槽位强制反置**(防 C6 槽位偏置)+ **干扰项多样性**(3 个干扰项两两 embedding 距离须过下界门,防 GroundAttack distractor-similarity 扎堆一起被排除);
|
||||||
|
- **题干模板对称(C8,防翻转轴泄露)**:before/after、first/last 题干本身会暴露翻转轴 → 题干须模板对称(不靠词面差异指示答案);**pair 关系可解率(P+Q 盲答+规则解法)作构造期 hard-fail**,不只事后验收指标。
|
||||||
|
|
||||||
|
| 子模式 | 孪生轴 | 锚定树的什么 | 正/反题构造 | 干扰项来源 |
|
||||||
|
|--------|--------|-------------|------------|-----------|
|
||||||
|
| **temporal** | 时序换序 | L1 下有序 L2 序列或 L2 下有序 L3(`time_range`/`timestamp`,`children` 顺序=时序) | 正"X 之后是什么"→答真实后继;反 before↔after 翻转→答真实前驱 | 同视频真实存在但时序位置错的事件 |
|
||||||
|
| **cross_segment** | 主宾互换/first-last | `action_subjects`+`actions` 跨 L2 段 | 正"谁**先**做 X"、反"谁**后**做 X",选项同集 | 别的段真实做该动作的主体 |
|
||||||
|
| **premature** | 早/晚段锚点 | 段位**随机**(early/mid/late/final 各占,不固定 final) | 正问最终态、反问指定中间态 | 过早锚定的早段真实读数 |
|
||||||
|
| **evidence_gap** | 诚实↔虚构因果 | 事件链 + `state_changes`(注意常 null,不可依赖) | 正支证据在场答=事实;反支证据缺席答="不可确定" | 树中未展示的合理因果延续(虚构但貌真) |
|
||||||
|
| **semantic_rigidity** | 同义↔字幕陷阱 | 树的 `subtitle` 字段 + 视觉 `event_description` | 正题字幕与视觉真相一致;反题字幕是陷阱(匹配干扰项) | 字幕字面项(与视觉真相冲突) |
|
||||||
|
| **fine_grained** | 同动作不同方式 | `ongoing_actions`+`visual_attributes`+看帧(需薄抽取) | 同一动作两个真实时刻的不同执行方式做对比 | 另一时刻该动作的真实不同方式 |
|
||||||
|
|
||||||
|
> **事实来源澄清(对齐 §9)**:上表"锚定树的什么"列对 temporal 是**树结构直取**;对其余 5 子模式**仅为索引/候选提示**——真正的 grounded 事实由 **§9 帧感知抽取层从真实帧感知 + 双 VLM 交叉核实**得出(树的 `state_changes` 常 null、`actions/action_subjects` 无绑定等有损,均由帧感知绕过)。
|
||||||
|
|
||||||
|
> **VITATECS 硬警告落地**:编辑必须落在**动词/事件结构**(时序/主宾/次数/方式),**严禁名词/属性词汇替换**(原文实测改 1 名词即 82% 可解)。每题构造后跑纯算法门校验单轴性 + bag-of-words 相等 + 逐维众数无定义。
|
||||||
|
|
||||||
|
> **非 temporal 子模式的范式合法性论证 + 必达归属**(回应 finding §5 与 VITATECS 静态退化警告;构造机制见 §9 帧核实抽取层):
|
||||||
|
> - **cross_segment / premature / semantic_rigidity = 必达**(§15 阶段2)。合法性——编辑轴均落在**事件结构**(主体-动作绑定 / 段位状态 / 视觉真相 vs 字幕字面),**非静态名词替换**,不落入 VITATECS 退化区;grounding 由 §9 帧核实抽取保证。semantic_rigidity 作为必达静态类,编辑轴=视觉真相 vs 字幕字面。
|
||||||
|
> - **fine_grained = 实现+目标必达 + 运行时门兜底**:编辑轴是**动作执行方式(manner,属动词/事件结构轴:快↔慢、左手↔右手、单次↔重复)非静态名词**;grounding 由"两个真实时刻的方式对比"锚定。风险=未必总能找到两个真实对比时刻(产量)+ 方式差异是否够单轴等信息量。运行时合法性门不过则退 finding B 路线。
|
||||||
|
> - **evidence_gap = 实现+目标必达 + 运行时门兜底**:"不可确定"选项**必须在正/反题对称出现**(防审核 C3"恒定文本靶");编辑轴=证据在场↔缺席(事件结构);幻觉风险最高,运行时门不过退保守构造。
|
||||||
|
|
||||||
|
## 5. 四家族模型分工
|
||||||
|
|
||||||
|
| 角色 | 模型 | .env 变量 | 家族 |
|
||||||
|
|------|------|----------|------|
|
||||||
|
| 活求解器难度探针(复用 `core/agent` AgentLoop) | deepseek-v4-pro | SEARCH_LLM | A |
|
||||||
|
| 构造表面实现 + 看帧核实 | qwen3.6-plus | VL_LLM | B |
|
||||||
|
| 独立看帧核实(异家族交叉) | MiniMax-M3 | M3_VL_LLM | C |
|
||||||
|
| 独立歧义/唯一性 + **禁蕴含真裁判** | **kimi-for-coding** | **新增 JUDGE_PANEL_LLM** | D |
|
||||||
|
| **仅 bag-of-words 词形辅助**(非蕴含判定) | text-embedding-v4 | TEXT_EMBEDDING | 纯算法 |
|
||||||
|
|
||||||
|
> **禁蕴含/语义等价判定用 kimi 真 LLM 裁判**,text-embedding 只做"选项词袋是否相等"的词形辅助——**embedding 余弦不得作蕴含唯一信号**(审核 C7:测不出语义泛化蕴含,是假绿灯)。
|
||||||
|
|
||||||
|
> 图像嵌入 `qwen3-vl-embedding` 暂不可用(几何跨模态 grounding 推迟);全孪生构造范式不依赖它。`kimi-for-coding` 需新增 .env 配置并**先测网关 newapi 是否服务**(实现第一步验证)。所有调用经 `GovernedLLMClient`(`adapters/llm.py`)治理,禁裸调。
|
||||||
|
|
||||||
|
## 6. 六层验证栈(每题全跑,机制种类各异,不共享失效模式)
|
||||||
|
|
||||||
|
| 层 | 机制种类 | 判据 | 失败处置 |
|
||||||
|
|----|---------|------|---------|
|
||||||
|
| **1 结构构造保证** | 符号 | 孪生答案=锚定树节点的相反事实,构造时即成立 | 构造非法→重构 |
|
||||||
|
| **2 坍缩度量** | **模型探针→统计判据**(非纯算法,见 §7 分类) | text-only 盲答(deepseek,**输入含 subtitle** 以捕字幕泄答案)+ 单帧基线(VLM)正确率**必须掉回随机**(4 选≈0.25;pair 双向≈0.0625);超容差=可纯文本/单帧/字幕解→拒 | 拒+指纹去重 |
|
||||||
|
| **3 看帧核实 + 帧错配安慰剂** | 视觉感知 | qwen+MiniMax 交叉查"构造相反事实真成立、干扰项在查询点真不成立";**安慰剂:喂错配帧分数须坍塌**(不坍塌=没真看帧)→拒 | 拒+重构 |
|
||||||
|
| **4 活求解器难度探针** | 行为/对抗 | deepseek AgentLoop 对 pair 双向作答;双向都对=太易→拒(对抗前移,AdVQA) | 拒+补构造 |
|
||||||
|
| **5 独立多裁判歧义核验** | 语义 | **多裁判 = kimi(D 文本)+ qwen(B 看帧)+ MiniMax(C 看帧)三异家族**,均独立于求解器(deepseek A);判是否有干扰项也成立/双正解,AdVQA 式一致性(无共识/多数判双真→拒)。text-embedding **不算裁判** | 拒+指纹去重 |
|
||||||
|
| **6 多格式交叉一致性** | 跨格式 | 同一构造事实生成 **MC + Y/N 两格式**,求解器跨格式作答须一致;不一致=格式捷径而非真理解→拒(TempCompass 多格式交叉) | 拒+重构 |
|
||||||
|
|
||||||
|
> **ABSTAIN 处置明确 + 终态闭环(防旧非对称谬误 + Codex I5)**:任一 VLM 探针"无法给有效帧引用"→记 `ABSTAIN` 入**分层人工抽检队列**,既不自动放行(防"引不出=放过")也不自动过杀。**必须有终态**:人工结论回写 `unit_verdict`(accept/reject);**超 SLA 阈值未标注按 fail 处理**(防无限 pending 吞产量)。
|
||||||
|
> **难度/歧义两口径正交不复用**:难度=求解器答错(层4);歧义=独立裁判判唯一性(层5)。永不用同一次评分兼两职。
|
||||||
|
|
||||||
|
## 7. 坍缩度量与反自证验收指标
|
||||||
|
|
||||||
|
**主验收=配对坍缩**(TempCompass 度量革命):有效性证明不是原始准确率高,而是去捷径后掉回随机的幅度。
|
||||||
|
|
||||||
|
> **指标三分类(I3,防 R4 自证误分类)**:**纯结构指标**(逐维众数命中率、答案槽位卡方、pair 关系规则解法、distractor 距离)——不依赖任何模型,是真正的算法地锚;**模型探针指标**(text-only 盲答、单帧基线、帧错配安慰剂、双正解率+κ、多格式一致率)——依赖模型输出的统计判据,**不得当"纯算法"背书**;**人工/抽检指标**(难/烂混淆矩阵)——分层小样本人工双标签。三类须分别标注,验收结论以纯结构 + 人工为硬地锚,模型探针为辅助信号。
|
||||||
|
|
||||||
|
| 指标 | 靶向 | 手段(外部/纯算法/独立) |
|
||||||
|
|------|------|------------------------|
|
||||||
|
| 逐维众数命中率 | ②结构捷径 | 纯算法机械投票,阈值≈随机 |
|
||||||
|
| 答案槽位卡方 + pair 槽位反置率 | ②C6 | 纯统计 |
|
||||||
|
| text-only 盲答 / 单帧基线 掉回随机幅度 | 语言先验/单帧 | **模型探针**(deepseek 盲答 + VLM 单帧,非纯算法地锚,见上分类)|
|
||||||
|
| 帧错配安慰剂坍塌幅度 | ①虚假 grounding | 错配帧分数须显著坍塌 |
|
||||||
|
| pair 关系可解率 | ②C2 | P+Q 盲答+时序排序规则,须≈1/16 |
|
||||||
|
| **subtitle_answerability_rate** | ②字幕泄答案 | 仅喂 subtitle+选项盲答,须≈随机 |
|
||||||
|
| **distractor_similarity(多样性)** | GroundAttack 扎堆 | 3 干扰项两两 embedding 距离须过下界 |
|
||||||
|
| **多格式交叉一致率** | 格式捷径 | MC vs Y/N 求解器答案一致率(低=格式捷径) |
|
||||||
|
| 双正解率 + 裁判 κ | ③歧义 | kimi vs qwen/MiniMax(异家族),基线 33%→阈值≤5% |
|
||||||
|
| 难/烂混淆矩阵 | ④/R4 | 分层人工抽检小样本双标签(可选锚点) |
|
||||||
|
| quarantine_rate / backfill_yield_by_round / judge_disagreement_by_subpattern | R5/系统 | 观测落 run_store |
|
||||||
|
|
||||||
|
阈值全部进科研 YAML + harness run 快照(可复现、可扫动);kimi/qwen endpoint 属工程 `.env`。
|
||||||
|
|
||||||
|
## 8. QuestionUnit 契约(贯穿采样/分批/聚合,混格题库)
|
||||||
|
|
||||||
|
`core/types.py` 新增 `QuestionUnit(single|pair)`:pair 载 `pair_id`/`twin_a`/`twin_b`/`flip_axis`/`unit_hash`/`collapse_metric`;`GeneratedQuestion` 加 `unit_id`/`pair_id`/`question_role`/`flip_axis`(默认值保非 AR single 不变)。
|
||||||
|
|
||||||
|
> **选项基数契约(锁定,与 §4/§7 对齐)**:每个 `GeneratedQuestion`(含 twin 的 P/Q 子题)**仍是 4 选 MCQ + 字母答案**,AgentLoop/harness 的 MCQ 作答契约**不变**;pair 语义只是"P、Q 共享 bag-of-words 选项集、答案翻转、双向 AND 聚合"。多格式的 Y/N 子题作为**验证探针**(层6)——**独立类型、独立存储表、硬校验其不进 benchmark/pools/predictions、不复用 `GeneratedQuestion`**(M1),仅在构造期核验一致性后即抛弃,不改 MCQ 契约。
|
||||||
|
|
||||||
|
**≥13 处改造入口**(复用审计 + 契约审核全表,v2 只覆盖 7 处,遗漏必崩):
|
||||||
|
|
||||||
|
| 入口 | 文件 | 改造 |
|
||||||
|
|------|------|------|
|
||||||
|
| Global 三池切分 | `pools.py::build_pools`/`_sample_excluding` | 互斥单元 qid→pair_id,pair 同池 |
|
||||||
|
| per-category 切分 | `pools.py::_split_one_category` | 输入改 unit,pair 不跨 train/val |
|
||||||
|
| 分层采样 | `loader.py::stratified_sample` | 按 unit 计数,flatten 前 pair 不拆 |
|
||||||
|
| batching 对错桶 | `batching.py::_select_mixed_by_task_type` | 按 **unit correctness(双向 AND)** 分桶 |
|
||||||
|
| batching 整锁 | `batching.py::build_batches` | pair 像小类整组装箱,**同 batch 不拆**(否则坍缩算不出) |
|
||||||
|
| 聚合 | `inference.py::run_inference` | pair 按 pair_id 收齐→pair-level record 双向 AND;unit 粒度 total/correct |
|
||||||
|
| 序列化冻结/解冻 | `pools.py::_q_to_dict`/`_dict_to_q` + `save_pools.categories` | 显式写读 4 pair 字段 + 以 unit 记 train/val |
|
||||||
|
| gate 阶梯 | `gate_ladder.py::build_cold_entries`/`ladder_for`/`update_probs` | entry 迁 unit_id + **schema_version + 迁移脚本**(存量 gate_pools.json 失效防硬崩) + 冷启动 2:1/gamma-EMA/Beta 先验按 unit 重定义 |
|
||||||
|
| e-process | `core/evolution/validate.py::pair_block` | McNemar 臂配对按 unit(P AND Q 折叠后比对) |
|
||||||
|
| correctness 消费(C3) | `runner.py`(rollout 回写 163-190 / accept 合并 1125-1142 / val 回写 1442-1454 / quadrant 分桶 / probation / momentum 1719-1760)+ `core/evolution/validate.py`(`pair_block` 12-37 / `compute_accuracy` 72-85) | 见下"correctness API 规格"——**逐一列出并改写这 8+ 调用点**,非仅"新增视图" |
|
||||||
|
| **checkpoint/resume(C2,新增)** | `runner.py::epoch_batches`/`_batch_from_ids`(199-210/711-725) | 保存 **unit_id 序列**(非逐题 qid),恢复时展开完整 unit,防断点续跑后拆 pair |
|
||||||
|
| **gate BaselineCache(C2,新增)** | `gate_ladder.py::BaselineCache`(277-335) | 键从 qid 改 unit_id,基线缓存按 unit 记忆,防 P/Q 分开缓存污染双向 AND |
|
||||||
|
| 作弊门 | `adversarial_filter`→v3 难度探针 | pair 粒度判太易,禁半剔 |
|
||||||
|
| 读回 benchmark | `loader.py::load_benchmark` | 补读 4 pair 字段(`.get` 兼容旧 JSON) |
|
||||||
|
| 非 AR rng 隔离(I4) | `loader.py`(86-98/123-131/154-172 的 `rng.sample`)+ `pools.py`(604-612/811-815 的 split/shuffle) | **实现前先枚举所有 `rng.sample/shuffle` 调用点**;非 AR 抽样输入与 RNG 流固定,AR pair 折叠用**独立 seed namespace**,二者 draw 互不干扰 |
|
||||||
|
|
||||||
|
**correctness API 规格(C3,三对象转换边界,非口号)**:显式定义三个对象及转换——
|
||||||
|
- **逐题 predictions**(`question_id → prediction`):唯一溯源真相,rollout 逐题写。
|
||||||
|
- **unit correctness**(`unit_id → bool`):AR pair = P.pred==P.ans **AND** Q.pred==Q.ans;非 AR = 单题。由 predictions 折叠得出,供进化/gate/quadrant/momentum/probation **统一消费**。
|
||||||
|
- **pair collapse metric**(`pair_id → 坍缩指标`):验收用,不进 correctness。
|
||||||
|
|
||||||
|
上表 8+ 调用点**逐一改写为消费 unit correctness**(含 `validate.py::compute_accuracy` 分母改 unit 数、`pair_block` McNemar 臂按 unit 折叠、quadrant 分桶按 unit_id);e-process 的 delta 信号在 unit 粒度计算,防 P/Q 单题计分污染进化。**先写此 API 规格 + 调用点替换清单,再动实现。**
|
||||||
|
|
||||||
|
## 9. 帧感知 grounded 事实抽取层(一等公民,支撑全部 6 子模式)
|
||||||
|
|
||||||
|
**这是让 6 子模式全部进入必达、又不滑回 v2 生成-过滤陷阱的关键机制。核心原则:树只是有损索引,真正的 ground truth 是帧本身——抽取直接喂真实帧,不依赖树的有损自由文本。**
|
||||||
|
|
||||||
|
盘上每视频有真实关键帧(5 帧/L2 段,`store/videos/<id>/frames/`,L3 `frame_path` 定位);需更密帧时按 video id 重下原视频补采(有界局限,见 §16)。`grounded_fact_extractor`(新建一等公民组件):
|
||||||
|
|
||||||
|
1. **帧感知抽取(VLM 对真实帧做感知,非对文本做解读)**:把目标段/时刻的真实帧喂 VLM,直接感知结构化事实——cross_segment 感知 `(主体,动作,段)` 绑定;premature 感知"各段状态读数";semantic_rigidity 感知"视觉真相"再与 `subtitle` 比;fine_grained 感知"动作执行方式"(左右手/快慢/次数);evidence_gap 感知"哪些证据视觉在场"。树的 `event_description`/`entities` 仅作**索引与候选提示**,不作事实来源。
|
||||||
|
2. **双 VLM 交叉核实 + Fact schema(Codex C6)**:每条感知事实由 **qwen + MiniMax 两异家族 VLM 对同帧交叉核实**,落 **Fact schema**:`fact_id, subject, action, object, segment_id, frame_ids, polarity(真/假), verifier_refs, cross_agree, fact_type, difficulty_tier, negative_at_target`。**只有两 VLM 一致 + 多帧核实通过 + 绑定唯一的 fact 才进构造**;不一致/不唯一→丢弃。
|
||||||
|
|
||||||
|
### 9.1 构造合法性硬约束(Codex 三审 C3/C5/C1/C4,缺一即滑回歧义/伪难题)
|
||||||
|
|
||||||
|
- **判别性 + `negative_at_target` 排他(C5,最关键)**:干扰项**不是"任何真实事实绑错点"**——错绑定 ≠ 逻辑否定。必须:(a) 只选**判别性事实**(跨查询点/翻转轴会变的,非"表演者在台上"这类跨段持续/复现的事实);(b) 每个干扰项显式 `negative_at_target`——**帧核实它在目标查询点确为假**(闭世界证据或翻转轴互斥证明),不过则**判双正解 hard-fail**。重复场景视频(同一主体贯穿)大量事实非判别,须按此剔除(产量由 §10 降级兜底)。
|
||||||
|
- **按 fact_type 的最小采样密度(C3)**:凡涉及**顺序/变化/次数/速度**的 fact(cross_segment 先后、premature 状态变化、evidence_gap 在场缺席、fine_grained 方式)——5 帧/段不足以稳定感知 → **触发密帧重采(按 video id 重下补帧)或降级**;不止 fine_grained。§9 为每个 fact_type 定最小密度。
|
||||||
|
- **fact_type 难度分层(C1)**:粗粒度物体/姿态感知可靠,但**主体-动作绑定/快慢/计数/证据缺席是高阶判断**——这些 fact_type 打高 `difficulty_tier`,走**更严核实**(更多帧 + 人工小样本校准),禁止当普通感知放行。
|
||||||
|
- **唯一绑定粒度 + 不唯一拆分(C4,防产量崩)**:定义唯一性粒度(同 segment/同 frame/同 action class/同 option set);多主体多动作段不唯一时**给拆分策略**(拆到可唯一的粒度再构造),而非一律丢弃。
|
||||||
|
- **双 VLM 共享盲区校准(C2)**:两 VLM 一致**只降随机噪声、不排除同向幻觉**;加错配/遮挡/反提示一致性校准 + **人工小样本估 shared-error rate** 落验收面板;一致不当真值背书。
|
||||||
|
|
||||||
|
### 9.2 实证验证状态(小样本 spike,见 [2026-07-15-v3-frame-perception-spike-validation.md])
|
||||||
|
|
||||||
|
5 视频×3 段、94 次 VLM 调用 + Claude 亲自看帧核查,**机制已验证站得住**:可抽取率 100%、判别性 ~83%、negative_at_target 实证有效(正确抓双正解)、双 VLM 一致 87%、粗时序 5 帧可感知、抽样无双 VLM 一致幻觉。据实证细化 5 条(须折进实现):
|
||||||
|
1. **fact_sampler 加"动作性"前置筛**(样本含幻灯片/录屏非动作视频,抽取虽成功但对 AR 无意义)。
|
||||||
|
2. **fine_grained 密帧从"运行时兜底"升为"硬前置"**(实证:manner 5 帧确实不够,必须密帧重采或走 B 路线,不等运行时才发现)。
|
||||||
|
3. **manner 类 fact 不靠双 VLM 交叉核**(实证一致性低)→ 走更严核实(更多帧+人工小样本),对齐 §9.1 C1 难度分层。
|
||||||
|
4. **抽取主用详细但准确的 VLM(MiniMax 风格),保守 VLM(qwen)作交叉核**。
|
||||||
|
5. 多主体绑定未被 spike 压测(样本多单主体)→ 见 §16 开放风险。
|
||||||
|
|
||||||
|
**为何这不是 v2 陷阱、且比"从树文本抽取"更强**:v2 的失败是"VLM **判分**自由发明的干扰项"(判断题,不可靠);这里是"VLM **感知**真实帧里有什么"(感知/描述题,可靠得多)——干扰项始终是**从真实帧感知到的真实事实绑到错误查询点**,从不自由发明、从不循环打分。wrongness 由构造保证(真事实、错绑定),grounding 由**真实帧感知 + 双 VLM 交叉**保证(短于人工标注下最强)。
|
||||||
|
|
||||||
|
**做不了**:bbox/像素坐标/精确计数/说话人 ID——放弃,不硬凑。
|
||||||
|
|
||||||
|
**grounding 强度诚实分档**(不抹平):temporal = **结构保证**(树顺序,零模型依赖);其余 5 = **帧感知验证保证**(真实帧 + 双 VLM 交叉 + Fact schema,模型依赖但锚在真实视觉证据,非自由发明)。两档都是合法 grounded,来源不同,须在验收报告分别标注。
|
||||||
|
|
||||||
|
## 10. 对抗前移、产量与续跑
|
||||||
|
|
||||||
|
- **难度探针前移**:构造后、落盘前即用 deepseek AgentLoop 试答(对抗前移),不是事后过滤。
|
||||||
|
- **失败题内容指纹去重**(R5 真正对症):对**题面语义内容**(非 question_id)算指纹,`拒/太易/歧义` 的内容指纹入 quarantine 黑名单;补构造命中黑名单即丢(堵"等价题换皮重试穿门")。
|
||||||
|
- **收敛上界**:`max_total_constructed`/`max_backfill_per_slot`/`min_pass_yield`;连续 N 轮低产即停并报告。
|
||||||
|
- **产量降级路径**(解产量危机,三选一写进配置):欠产(如 22/30)作为**带质量标签的正式交付物**;或 min_pass_yield 触发转分层人工抽检;明确**质量优先于数量**,`generate_ar30.sh` 的 TARGET 变软。
|
||||||
|
|
||||||
|
## 11. 模块结构(Clean Architecture 四层)
|
||||||
|
|
||||||
|
**在现有 `app/question_gen/` 内按领域模块替换/重组**(对齐 CLAUDE.md §4.2"直接改原文件、不留向后兼容"与 §5"禁版本化并行目录"——**不建 `app/question_gen_v3/` 并行目录**)。废弃模块(generator_v2/synthesizer/distractor_selector/gates/pipeline_v2)**原地删除或改写**,新增领域模块同目录落位:`fact_sampler`(采可翻转事实)、`constructor`(子模式→孪生轴路由)、`twin_builder`(孪生构造)、`grounded_fact_extractor`(§9 帧核实事实抽取,喂 cross_segment/premature/semantic_rigidity/fine_grained/evidence_gap)、`difficulty_prober`(活求解器,**不兼答案裁判**)、`ambiguity_verifier`(kimi 独立多裁判)、`collapse_metric`(结构/探针/人工三类验收,见 §7)、`pipeline`(主编排,替换 pipeline_v2);`run_store` 原地扩 pair 感知(不新建 run_store_v3)。
|
||||||
|
|
||||||
|
**复用地基**:`adapters/*` 全部(GovernedLLMClient/VLM/telemetry/熔断/缓存)、`core/protocols`(扩 `JudgePanelProvider`)、`core/types`(扩 QuestionUnit)、`app/harness/*`(pair 契约适配,见 §8)、`run_store.py`/`loader.py`(续跑/加载基建)。
|
||||||
|
|
||||||
|
**废弃**:`generator_v2`/`synthesizer`/`distractor_selector`/`gates`/`pipeline_v2` 主编排 + 对应 prompts/config。**拆解复用**:`adversarial_filter` 的孪生翻转构造 + 活求解器探针 + verdicts 续跑(去后置过滤外壳,提升为构造/核验主路径)。**借鉴语义**:`strategy_action_recognition` 的 6 skill 靶点 + flip_axis 标注。
|
||||||
|
|
||||||
|
## 12. 非功能性需求(重写必须继承,防隐式丢失)
|
||||||
|
|
||||||
|
| 维度 | 设计 |
|
||||||
|
|------|------|
|
||||||
|
| **持久化** | 逐 **unit** 原子成对落盘:pair 在内存 pending buffer 收齐 P+Q 后一次性进 accepted;全量 tmp + `os.replace`,**pair 同落或都不落**,崩溃不留半对 |
|
||||||
|
| **幂等性** | 同 seed→同构造;孪生由 flip_axis 确定性构造;VLM 非确定为既有属性;`unit_hash` 检测续跑失效 |
|
||||||
|
| **断点续跑** | progress(slot→status);pair 以"整对完成"为 accepted 单位;verdicts 表按 (question_id/hash/config 指纹) 三元组续跑;config 指纹失效作废脏续跑 |
|
||||||
|
| **原子性** | 逐 unit 落盘;读回校验 pair 成对完整、剔孤儿;补构造续跑三件套(seq_offset/used_node_ids/embed_pool) |
|
||||||
|
| **治理/遥测** | 所有 LLM/VLM 经 GovernedLLMClient 五层栈;session_id 透传每次调用必录 telemetry |
|
||||||
|
| **并发/降级** | asyncio.Semaphore 限流;JSON 解析失败区分"契约违反 raise"vs"可降级 warning",不静默 |
|
||||||
|
|
||||||
|
### 12.1 日志与可观测方案(须走 structured-logging skill)
|
||||||
|
|
||||||
|
v3 产生大量运行时数据。设计阶段先给出**表结构骨架**(对齐 CLAUDE.md §3 Phase 1.4"设计阶段强制项",细节仍由 `structured-logging` skill 在 writing-plans 前定稿 + Wiki schema 注册):
|
||||||
|
|
||||||
|
| 表 | 关键列 |
|
||||||
|
|----|--------|
|
||||||
|
| `unit_verdict` | unit_id, pair_id, sub_pattern, stage(层1-6), verdict(pass/fail), reason, metric_value, model, session_id |
|
||||||
|
| `collapse_metrics` | pair_id, text_only_acc, single_frame_acc, placebo_drop, majority_vote_hit, slot_chi2, distractor_min_dist, multiformat_consistency |
|
||||||
|
| `quarantine` | content_fingerprint, sub_pattern, quarantine_reason, round_no, ts |
|
||||||
|
| `resume_state` | slot_id/unit_id, status, config_fingerprint(断点恢复+失效检测), seq_offset |
|
||||||
|
| telemetry | 每次 LLM/VLM 调用(GovernedLLMClient 自动录,session_id 透传) |
|
||||||
|
|
||||||
|
基线指标(各门通过率、backfill_yield_by_round、judge_disagreement_by_subpattern)定义与阈值进科研 YAML + run 快照。
|
||||||
|
|
||||||
|
## 13. 错误处理
|
||||||
|
|
||||||
|
孪生构造失败(找不到真实相反事实)→该 slot 重构或走产量降级;VLM/裁判异常→slot 级隔离宽异常重出不崩整批;kimi/NLI 裁判不可用→明确报错不静默跳过(防漏歧义);安慰剂/坍缩门模型不可用→报错。
|
||||||
|
|
||||||
|
## 14. 测试策略
|
||||||
|
|
||||||
|
- 单元:6 子模式孪生构造分流;bag-of-words 硬匹配校验;禁蕴含 NLI;逐维众数纯算法门;坍缩度量;pair 原子写/半写检测;双向 AND。
|
||||||
|
- 集成:AR 端到端(mock 四家族)→ pair 成对落盘、**六层验证全部生效(含层6 MC vs Y/N 多格式交叉的 mock 与失败路径)**、坍缩验收。
|
||||||
|
- 回归(pair 不拆):`stratified_sample`/`build_batches`/`run_inference`/`pools.build_pools` 四处 pair 同进同出、同批、配对聚合、孤儿剔除;gate_ladder 迁移续跑不崩。
|
||||||
|
- 回归(非 AR byte-identical):11 非 AR generate-v2 字节级不变,含 rng 隔离验证。
|
||||||
|
- Agent 类测试产出 MD(CLAUDE.md §4.6)。
|
||||||
|
|
||||||
|
## 15. 分期实现(一个设计,writing-plans 分阶段)
|
||||||
|
|
||||||
|
1. **契约地基**:QuestionUnit + ≥13 入口最小 pair 支持 + gate_ladder schema_version 迁移 + 非 AR rng 隔离 + pair 原子落盘。
|
||||||
|
2. **帧感知事实抽取层(§9)+ 构造器 + fact_sampler**。**v3 必达 = 全部 6 子模式**(用户决策:接受帧感知抽取风险):
|
||||||
|
- **temporal** grounding=树结构保证(零模型);**其余 5**(cross_segment/premature/semantic_rigidity/fine_grained/evidence_gap)grounding=**§9 帧感知验证保证**(喂真实帧 + 双 VLM 交叉核 + Fact schema,见 §9);
|
||||||
|
- 六者共用 bag-of-words/禁蕴含/单轴/逐维众数平衡纯算法门 + 六层验证 + §9 题干模板对称 hard-fail(C8)。**帧感知抽取层是本阶段核心交付。**
|
||||||
|
- **运行时合法性门兜底**:fine_grained(需两个真实对比时刻,可能超 5 帧/段密度→重下密帧)、evidence_gap(幻觉风险最高)各挂运行时门(帧核实+坍缩+小样本人工抽检),门不过该 pair 走 fallback(fine_grained 退 finding B 路线、evidence_gap 退保守构造)——**兜底是质量安全网,不是把子模式踢出必达**。
|
||||||
|
3. **六层验证栈**:坍缩度量(含 subtitle 输入)+ 看帧核实 + 帧错配安慰剂 + 难度探针(deepseek AgentLoop)+ kimi 独立歧义多裁判 + **多格式交叉一致(MC+Y/N)**。
|
||||||
|
4. **对抗前移 + 产量**:失败指纹去重 + quarantine + 收敛上界 + 产量降级路径 + 补构造迭代。
|
||||||
|
5. **验收面板 + 混格全链路**:反自证指标面板 + AR pair/非 AR single 端到端评分正确性验收。
|
||||||
|
|
||||||
|
> 质量闸门前移:§7 纯算法指标(逐维众数/坍缩/槽位卡方)在阶段 2 即作 CI hard-fail;难度探针在阶段 3 出早期难度信号——不等阶段 5 才验质量(对治上一版盲飞根因)。
|
||||||
|
|
||||||
|
## 16. 风险与回退
|
||||||
|
|
||||||
|
| 风险 | 回退 |
|
||||||
|
|------|------|
|
||||||
|
| kimi 网关不可用 | 实现第一步验证;退回 qwen+MiniMax 异家族双看帧裁判并标注局限 |
|
||||||
|
| **帧感知抽取风险(按 fact_type 的 failure taxonomy,C6)** | 逐类列失败路径 + 失败率进验收面板:稀疏帧漏检(顺序/变化/次数/速度类)→密帧重采/降级;双 VLM 共享盲区→错配/遮挡校准+人工 shared-error 抽样;**负事实不可证/双正解**→`negative_at_target` hard-fail(§9.1);事实过宽跨段复用→判别性筛;ABSTAIN 吞吐→SLA 终态(§6)|
|
||||||
|
| **重复场景视频(同主体贯穿)大量事实非判别** | §9.1 判别性筛 + `negative_at_target` 剔双正解;产量由 §10 降级兜底(宁少勿歧义)。**spike 实证:连 GsYi 重复场景判别性仍 ~83%,双正解剔除率 ~17% 可控** |
|
||||||
|
| **多主体繁忙场景的唯一绑定(Codex C4,spike 未压测)** | 样本多单主体,唯一绑定风险未验证 → writing-plans/实现早期补一次多主体段实测;不唯一时按 §9.1 拆分粒度处理 |
|
||||||
|
| **fine_grained 产量/合法性**("两时刻不同方式"未必总能找到、未必构成合法单轴编辑) | 运行时合法性门(帧核实+坍缩+人工抽检);不过退 finding B 路线(生成候选+独立 wrongness 帧核验);产量降级兜底 |
|
||||||
|
| **evidence_gap 虚构因果幻觉**(幻觉风险最高) | 运行时合法性门 + 保守构造 fallback;"不可确定"选项正/反题对称(防 C3 文本靶)|
|
||||||
|
| **semantic_rigidity(必达静态类)** | 编辑轴=视觉真相 vs 字幕字面(非名词替换,避 VITATECS 静态退化);视觉真相由帧核实确证 |
|
||||||
|
| fine_grained/静态类产量低(找不到真实对比时刻) | 产量降级路径;必要时该子模式先缓 |
|
||||||
|
| 属性抽取层不准 | 抽取后 VLM 看帧复核;不准即弃该题 |
|
||||||
|
| gate_ladder 迁移破坏冷启动算法保真 | schema_version + 迁移脚本 + 2:1/gamma-EMA/Beta 按 unit 重定义并写入保真清单 |
|
||||||
|
| 混格题库拖累训练 | 阶段 5 先验评分正确性;必要时 AR pair 与非 AR 分池 |
|
||||||
|
|
||||||
|
## 17. 核心算法保真
|
||||||
|
|
||||||
|
v3 触及算法保真清单(`ARCHITECTURE.md §6`)第 5 项(信息阶梯冷启动 2:1/gamma-EMA/反泄漏)——因 gate_ladder 迁 unit_id,须逐条比对按 unit 重定义、不简化,并在 commit 标注。其余 11 项不涉及。
|
||||||
|
|
||||||
|
> **gate-unit 迁移 ADR(C4,阶段 1 强制交付,不得占位进实现)**:以下每项须有**可测试**的明确定义——① unit 的冷启动"对/错"如何从 P/Q 得出(建议 unit 错=P 或 Q 任一错);② 2:1 错优先交错在 unit 上如何保持;③ unit `p_hat` 初值(Beta 先验参数);④ gamma-EMA 观测来源(predictions 折叠成 unit 观测再更新,防 `update_probs` 按 qid 匹配失效致 EMA 停摆);⑤ probe_quota 按 unit 还是按题抽;⑥ 存量 `gate_pools.json`(无 schema_version、qid 键)的处理:加 `schema_version` + 拒绝或一次性迁移脚本;⑦ 反泄漏(run_id 含 `_gate_` 过滤)不受影响的确认。
|
||||||
|
|
||||||
|
## 相关
|
||||||
|
- 范式依据:[2026-07-15-question-gen-paradigm-shift-construction-over-filtering.md]
|
||||||
|
- 前序审核:[2026-07-15-question-gen-v2-adversarial-audit.md]
|
||||||
|
- 被取代:[2026-07-15-question-gen-v2-grounded-contrastive-design.md]
|
||||||
|
- 论文:`reference/papers-distractor-gen/`(6 篇深读)
|
||||||
@@ -0,0 +1,210 @@
|
|||||||
|
---
|
||||||
|
id: results-driven-video-split
|
||||||
|
title: 结果驱动的视频级 train/val/test 切分(替代自造题模块)
|
||||||
|
type: design
|
||||||
|
created: 2026-07-15
|
||||||
|
status: draft
|
||||||
|
---
|
||||||
|
|
||||||
|
# 结果驱动的视频级切分设计
|
||||||
|
|
||||||
|
## 1. 目标与范围
|
||||||
|
|
||||||
|
**目标**:用一条**离线、结果驱动的视频级切分管线**,替代"自造 Video-MME 同质量训练题"这一在 AAAI 截止前风险过高的模块。基于已有的 `infer_adhoc` baseline(900 题 / 300 视频 / 660 对 240 错 / 73.3%),把 300 视频按**视频原子**切成 train/val/test,产出一个冻结的 `pools.json` 供 harness run 消费。
|
||||||
|
|
||||||
|
**范围边界**:
|
||||||
|
|
||||||
|
| 在范围内 | 不在范围内 |
|
||||||
|
|---------|-----------|
|
||||||
|
| 诊断 240 错题 → signal 分层 | 改诊断模块内部(只**调用** `core/evolution/diagnose`) |
|
||||||
|
| signal_scorer + 视频级聚合 + 贪心联合选择器(新增) | 改进化循环内部 |
|
||||||
|
| `pools.py` 切分原子 unit→**video** + 信号感知 | AR 出题 v3 管线(另行废弃,本设计外) |
|
||||||
|
| `loader` 指向 Video-MME 900(已如此) | 重跑推理(复用现有 baseline) |
|
||||||
|
| — | **不改 `gate_ladder.py` / 信息阶梯冷启动 2:1**(Codex M-1;算法保真 §4.7 第 5 项不受本切分波及,切分只改 pools 归属不改冷启动比例) |
|
||||||
|
|
||||||
|
**故事重定位**:论文核心贡献是"视频树上可自进化的搜索 Agent(Harness Engineering)",出题从来不是卖点。PyTorch 类比中 DataLoader **本就只加载已有数据集 + 切分、从不合成数据**——所以砍掉自造题、改为对 Video-MME 做结果驱动切分,类比反而更忠实。头号数字诚实、可比。
|
||||||
|
|
||||||
|
## 2. 背景:核心洞察(耦合)
|
||||||
|
|
||||||
|
全数据集错题总数固定 = 240。**守恒式为 `train错题 + val错题 + test错题 = 240`**(选择器在 **train+val 并集**上优化覆盖,再内部切 train/val):越把信号往 train+val 塞,test 越简单、headline 越虚高。因此"把最难视频塞进 train+val"是有害的(偏置方向恰好注水 test)。**val 会分走部分 defect 视频——这是可接受的**:gate/e-process 本就需要错题才有功效(§8),但 val 的抽取受 §8 的 correctness 分层约束,不破坏 train 的多样性目标。
|
||||||
|
|
||||||
|
**第一原则**:**test 神圣(代表性 + 冻结)**;train 的信号富集只能靠三条合法途径——① 每类型 floor(硬约束);② 调度层信息阶梯加权(运行时,本设计不涉及);③ 同 profile 视频交换自由度。ε 代表性约束是耦合的算法化身,天然挡住"偷 test 难题"。
|
||||||
|
|
||||||
|
信号地图(baseline 逐类型错题):Counting 24/48(0.50)、Action Recognition 22/63(0.65)、OCR 5/14、Object Reasoning 68/240、Action Reasoning 48/180、Temporal Reasoning 22/91 为高信号;Spatial Reasoning 0/11(全对,零信号);Temporal/Spatial Perception 量微。视频级:125 视频全对(零信号)、115 一错、55 两错、5 全错。
|
||||||
|
|
||||||
|
## 3. 架构与数据流
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
P[infer_adhoc predictions<br/>900题/300视频] --> S{对错拆分}
|
||||||
|
S -->|660 对| CG[correct 视频画像<br/>供 ε 代表性/难度约束]
|
||||||
|
S -->|240 错| DG[诊断管线<br/>调用 core/evolution/diagnose]
|
||||||
|
DG --> QA[每题: error_type + cause_category<br/>+ QuestionMetrics]
|
||||||
|
QA --> SC[signal_scorer<br/>→ tier T0/T1/T2 + 多样性格子]
|
||||||
|
SC --> VA[视频级聚合<br/>每视频: 覆盖格子集 + tier 构成]
|
||||||
|
CG --> GS
|
||||||
|
VA --> GS[贪心联合约束选择器<br/>max 多样性覆盖 s.t. test ε + floor]
|
||||||
|
GS --> TV[选中 ~100 视频 train+val]
|
||||||
|
GS --> TE[补集 ~200 视频 test]
|
||||||
|
TV --> SP[correctness 视频原子切 train/val<br/>val 按 McNemar 功效定尺寸]
|
||||||
|
TE --> FZ
|
||||||
|
SP --> FZ[save_pools → 冻结 pools.json(原子写)]
|
||||||
|
FZ --> RUN[harness run 消费]
|
||||||
|
```
|
||||||
|
|
||||||
|
**关键不变量**:① 视频为原子,三池视频集互斥、绝不共享视频 → 零内容泄漏;② test = train+val 的补集,代表性由 ε 约束保证(非事后随机);③ 全程离线、可复现(固定 seed + baseline run_id 溯源),不触碰在线进化。
|
||||||
|
|
||||||
|
## 4. 组件落位(Clean Architecture 四层)
|
||||||
|
|
||||||
|
| 模块 | 职责 | 新增/改动 | 层 |
|
||||||
|
|------|------|----------|----|
|
||||||
|
| `core/evolution/diagnose.py` | 诊断 240 错题 | 不动(只调用) | core |
|
||||||
|
| `app/harness/baseline_diagnosis.py` | 编排离线诊断:读 baseline 错题预测+轨迹 → 调 diagnose → 落库 | 新增 | app |
|
||||||
|
| `app/harness/split_selection.py` | 纯函数:`score_signal`(→tier+格子) / `aggregate_video_signal` / `select_split`(贪心) | 新增 | app |
|
||||||
|
| `app/harness/pools.py` | 切分原子 unit→video;`build_pools` 接选择器给的视频三分,做 video 原子 correctness train/val 切分 + 原子冻结 | 改动 | app |
|
||||||
|
| `app/question_gen/loader.py` | 指向 Video-MME 900 | 已如此 | app |
|
||||||
|
| `baseline_diagnosis` 落库端口 + SQLite 适配 | 每题 error_type/cause_category/tier/(派生)evolution_target/degraded/diag_fingerprint | 新端口(app/ports 或 core/protocols)+ adapters 实现 | ports+adapters |
|
||||||
|
|
||||||
|
`split_selection.py` 全为纯函数(相同输入→相同输出),诊断 I/O 隔离在 `baseline_diagnosis.py`(P6 可测试性)。**落库走端口/适配层,不在 app 拼裸 SQL**(Codex I-5,对齐 §4.8 遥测规范与依赖方向)。
|
||||||
|
|
||||||
|
**视频原子改造要点(Codex I-3,防实现复用 unit 逻辑漏泄漏)**:`build_units` 保留(pair 聚合仍需),但**新增 video 分组层**——`build_pools` 接选择器给定的 video→池 归属,`_split_one_category` 的采样原子从 `QuestionUnit` 提升为 **video 组**(同 video 全部 unit 同进同出)。改造后须由防御断言①验证三池视频不相交。
|
||||||
|
|
||||||
|
## 5. signal_score 与多样性定义
|
||||||
|
|
||||||
|
### 5.1 signal_score = DiagnosisResult 的确定性投影(不发明新分类)
|
||||||
|
|
||||||
|
`DiagnosisResult` 由现有两阶段管线产出:阶段1 规则指标(纯函数) + 5 个 LLM judge → `QuestionMetrics`;阶段2 瀑布归因出 **4 类 error_type**(`extraction_failure`→`search_failure`→`reasoning_failure`→`mixed`,代码 928–935),再由 `classify_defect_vs_lapse`(LLM judge) 判 `cause_category`;INFRA(`stop_reason∈{error,parse_error}`) 阶段1前排除。
|
||||||
|
|
||||||
|
signal_score 主要是**分层标签**(非脆弱连续加权分,避 P5 魔法数):
|
||||||
|
|
||||||
|
| Tier | 判据(全来自诊断) | 训练处置 |
|
||||||
|
|------|------------------|---------|
|
||||||
|
| **T0 排除** | INFRA(`stop_reason∈{error,parse_error}`) | signal=0,永不进 train |
|
||||||
|
| **T1 低信号** | `cause_category='lapse'`(**接受此桶混着"真·无解/标注错"**,不做三分类) | 低权重,对照/appendix,不作主训练 |
|
||||||
|
| **T2 高信号** | `cause_category='defect'` 且映射到可执行进化目标 | 核心训练集,内部按多样性排序 |
|
||||||
|
| (uncertain) | `degraded`(judge 解析失败) 或诊断硬失败 | 排除出 T2 + 计数上报,不静默丢 |
|
||||||
|
|
||||||
|
> lapse 为低信号是诊断自身的判断(lapse 路由到受保护 appendix、不重写 skill),非本设计新增。
|
||||||
|
|
||||||
|
### 5.2 多样性 = 对诊断逐题一等字段的覆盖(非错题数量)
|
||||||
|
|
||||||
|
一个 train 集"多样"当且仅当它张成"可修复失败模式"的空间,使进化对每块参数面都拿到梯度。
|
||||||
|
|
||||||
|
**主格子 = `(task_type × error_type)` = 12 × 4 = 48 格**——两者都是**逐题一等字段**(task_type 来自题、error_type 来自 `attribute_error`),无歧义、可确定性计算:
|
||||||
|
|
||||||
|
| 轴 | 取值空间 | 来源 |
|
||||||
|
|----|---------|------|
|
||||||
|
| task_type | 12 类 | 题目字段 |
|
||||||
|
| error_type | extraction/search/reasoning/mixed(4 值) | `ErrorAttribution.error_type`(逐题一等字段) |
|
||||||
|
|
||||||
|
**`evolution_target` 是由 error_type 确定性派生的标注,不是独立第三轴**(Codex C-1:`ErrorAttribution` 无 evolution_target 字段,逐题 CasePack 路由非一等产物)。设计声明一张**固定映射表**(科研 config,可测试),用于报告"哪层参数组拿到梯度",不增加维度:
|
||||||
|
|
||||||
|
| error_type | evolution_target | 理由 |
|
||||||
|
|-----------|-----------------|------|
|
||||||
|
| extraction_failure | tool | 抽取发生在 view_node/observe_frame 工具 prompt → ToolCasePack |
|
||||||
|
| search_failure | skill | 搜索策略是 skill → SkillCasePack |
|
||||||
|
| reasoning_failure | skill | 推理是 skill → SkillCasePack |
|
||||||
|
| mixed | system | 跨切面 → SystemCasePack |
|
||||||
|
|
||||||
|
**`behavioral mode` 降为 tie-breaker,不进主格子**(Codex I-2:system pack 实际只有 `{early_submit, high_conf_wrong, confirmation_bias}` 三类)。平局时用这三类的真实取值打破,不臆造名字。
|
||||||
|
|
||||||
|
**度量 = 对 48 格 `(task_type × error_type)` 的覆盖数(set-cover)**,submodular。**视频级计数规则(Codex I-1)**:一个视频贡献其全部 T2 错题所覆盖格子的**并集(去重集合)**,每格全局只计一次 → 多错题视频不会虚高覆盖增益。奖励"开新格子",惩罚"同格子第 10 道重复错题"。对比"最大化错题数"——既有害(偷 test 难题)又冗余。
|
||||||
|
|
||||||
|
## 6. 贪心联合约束选择器(`select_split`)
|
||||||
|
|
||||||
|
**floor = 硬约束(先满足,取值克制);多样性 = 目标(floor 后最大化)。** floor 违反 = 某高信号类型零梯度(灾难),少覆盖一格子仅边际损失。
|
||||||
|
|
||||||
|
```
|
||||||
|
输入: videos(每个带 其题对错 + 错题 tier/格子), 全局目标(类型比例,难度画像),
|
||||||
|
config(N_trainval, floor_K[type], ε, reportable_types, seed)
|
||||||
|
1. trainval=∅; test=all
|
||||||
|
2. # Floor 阶段(硬约束, 同样受 ε 守护 —— Codex C-2):
|
||||||
|
while 存在未达 floor 的高信号类型:
|
||||||
|
cand = 能填未满 floor 槽 且 "移走后 test 仍满足 ε" 的视频
|
||||||
|
若 cand 非空: 选填槽最多者 → 移入 trainval
|
||||||
|
否则: fail loud(floor 与 ε 死锁, 报是哪个类型的 floor 无法在不破 ε 下满足)
|
||||||
|
3. # 多样性阶段(submodular 贪心):
|
||||||
|
while |trainval| < N_trainval:
|
||||||
|
cand = 各候选视频移入 trainval 的"新开格子数"(并集去重边际增益)
|
||||||
|
按增益降序试: 取增益最大且"移走后 test 仍满足 ε"的视频 → 移入
|
||||||
|
若无任一视频可加而不破 ε → 停(报欠额, 不静默)
|
||||||
|
4. 返回 (trainval, test=补集)
|
||||||
|
```
|
||||||
|
|
||||||
|
- **submodular → 1−1/e 保证**;**ε 可行性检查在两个阶段都生效 = 耦合的算法化身**(拉太多难视频破坏 test 难度画像→被拒)。floor 阶段先前漏了 ε 检查(Codex C-2),已补:floor 视频移动同样必须保 test ε。
|
||||||
|
- **可行性冲突 fail loud**:floor 与 ε 死锁(某类型 floor 只能靠"会破 ε 的视频"填)/欠额时明确报哪条约束差多少,人放松旋钮,**绝不静默退随机**。
|
||||||
|
- 平局与遍历顺序由固定 seed 决定 → 可复现。
|
||||||
|
|
||||||
|
## 7. 长尾类型处理(report_floor 拟定 = 总题数 ≥ 27)
|
||||||
|
|
||||||
|
| 类型(总题) | per-type 报告 | train floor | 归属 |
|
||||||
|
|------|:---:|:---:|------|
|
||||||
|
| 8 类 ≥27(Object/Action Reasoning, Info Synopsis, Temporal Reasoning, Action/Object Recognition, Counting, Attribute Perception) | ✅ | ✅ | 参与 ε + floor 约束 |
|
||||||
|
| OCR(14) | ❌ 折进 overall | 🟡 无硬 floor,defect 机会性进多样性池 | 随视频落 |
|
||||||
|
| Spatial Reasoning(11) | ❌ | ⛔ 全对零 defect | 随视频落 test |
|
||||||
|
| Temporal Perception(6)/Spatial Perception(3) | ❌ | ⛔ | 随视频落,披露计数 |
|
||||||
|
|
||||||
|
非报告类型不进 ε 代表性约束(太少无法分层),但其视频仍参与选择(每视频跨~3 类,长尾题搭车跟随其视频)。报告折进 overall + 一行"rare types (aggregate)",披露计数。
|
||||||
|
|
||||||
|
## 8. val 尺寸(McNemar 功效)
|
||||||
|
|
||||||
|
选中 100 视频内部用**现有 `_split_one_category`(改视频原子)** 做 correctness 分层切 train/val:val 取较小片但带 `eval_min_per_class` 保底,且**含足够错题使 McNemar 有功效**(val 期望错题数 ≥ 功效阈值,具体值诊断后标定);train 取大头(defect 更密,供 rollout)。
|
||||||
|
|
||||||
|
## 9. 非功能性需求(强制四维)
|
||||||
|
|
||||||
|
离线一次性管线,但诊断阶段有 240 次 LLM judge 调用,必须可续跑。
|
||||||
|
|
||||||
|
| 维度 | 设计 |
|
||||||
|
|------|------|
|
||||||
|
| **持久化** | 诊断结果**逐题** upsert 入 `baseline_diagnosis` 表(崩溃最多丢在飞那题);signal/选择纯内存从表重算;最终 `pools.json` 冻结快照 |
|
||||||
|
| **幂等性** | 诊断表主键 = `(question_id, baseline_run_id, diag_fingerprint)`,其中 `diag_fingerprint = hash(诊断 prompt 版本 + model + 诊断代码版本)`(Codex C-4:仅 question_id upsert 不足以约束 judge 非确定/prompt/model 漂移);`--force` 在**新 fingerprint** 下写、**不覆盖**旧记录;冻结的 `pools.json` 记录其构建所依据的 `diag_fingerprint`,重建时 fingerprint 不匹配 → **fail loud**(防脏续跑污染已冻结切分)。选择=纯函数(表切片,全局统计,config,seed)→同输入必同切分 |
|
||||||
|
| **断点续跑** | 诊断重启查已完成集 → 跳过(镜像建树逐项续跑);崩在第 150 题从 150 续;选择/冻结秒级重跑即可 |
|
||||||
|
| **原子性** | 逐题写=SQLite 单行事务原子;**pools.json 补原子写**——现有 `save_pools`(341) 与 per_category 增量路径(566) **都用 `path.write_text`(非原子, Codex I-4)**,抽出共用冻结助手统一改 tmp + `os.replace`,两条路径同受益 |
|
||||||
|
|
||||||
|
## 10. 错误处理(P5:不静默、不兜底)
|
||||||
|
|
||||||
|
| 情形 | 处置 |
|
||||||
|
|------|------|
|
||||||
|
| 诊断 LLM 基础设施失败 | GovernedLLMClient 重试栈后仍失败 → 传播报错,不掩盖 |
|
||||||
|
| 单题诊断硬失败 | slot 级隔离:记 `error`/`degraded` 入表(带错误)+计入报告,不静默跳;该题→uncertain→不进 T2 |
|
||||||
|
| judge 解析失败(degraded) | 现有 `degraded` → uncertain → 排除 T2 + 计数上报 |
|
||||||
|
| 选择器不可行(floor 与 ε 冲突/欠额) | **fail loud**:报哪条约束差多少,人放松旋钮,绝不静默退随机 |
|
||||||
|
| correctness 缺失 | 现有 `_assert_correctness_complete` fail-fast 保留 |
|
||||||
|
| judge 判不准默认 lapse | 该 fallback **仅用于语义歧义**(judge 回复无法解析),基础设施失败照常传播(`classify_defect_vs_lapse` 994-996);默认 lapse 的题**计数上报**,不静默(Codex M-2) |
|
||||||
|
|
||||||
|
**防御性断言清单(P5,Codex I-6,实现须逐条落地)**:① 三池视频集两两不相交断言;② baseline predictions 覆盖全 900 题(缺题 fail-fast);③ 每 video 恰 3 题完整性;④ 诊断表 `diag_fingerprint` 与冻结 pools 记录一致;⑤ 冻结产物写 manifest(含 baseline_run_id/diag_fingerprint/config/seed/文件 hash)供复现校验;⑥ question_id/video_id 唯一性。
|
||||||
|
|
||||||
|
## 11. 测试策略
|
||||||
|
|
||||||
|
- **单元**:`score_signal`(INFRA/lapse/defect→正确 tier+格子);`aggregate_video_signal`(混 tier 视频→正确覆盖格子集);`select_split`(floor 满足 / **视频不跨池** / test ε 代表性 / 贪心覆盖下界 / **构造冲突→抛错** / 同 seed→同切分);video 原子 `_split_one_category`。
|
||||||
|
- **集成**:真实 infer_adhoc 240 错题 + 缓存真实诊断端到端 → 产出合法冻结 pools.json(视频互斥、floor 达标、test 代表)。**续跑**:诊断中途 kill→重启跳过已完成→同一最终结果。
|
||||||
|
- **LLM 类测试产出 MD**(§4.6):诊断跑涉及 LLM → 结构化 MD 报告。
|
||||||
|
- **回归**:现有 pools/loader 测试在视频原子改造后仍过;非 AR/pair 行为保留。
|
||||||
|
|
||||||
|
## 12. 前序版本继承(step 1.5 审计逐条落实)
|
||||||
|
|
||||||
|
| 前序行为 | 处置 |
|
||||||
|
|---------|------|
|
||||||
|
| 三池 / 逐步排除互斥 / pair 原子 / baseline_val_accuracy / correctness dict / 复现 | 保留 |
|
||||||
|
| test 自然分布 | 升级为 ε 代表性约束(更强) |
|
||||||
|
| val correctness 分层 + min_per_class | 保留 |
|
||||||
|
| pools.json 持久化 | 保留 + 补原子写 |
|
||||||
|
| **per_category 增量模式** | **保留但闲置(选项 A)**——AR 废弃在本设计外,新流程只走 global 视频感知;per_category 删除留给将来 AR 清理 |
|
||||||
|
|
||||||
|
## 13. 待标定旋钮(诊断跑完后用真实分布定,写入科研 YAML + run 快照)
|
||||||
|
|
||||||
|
`N_trainval`(~100)、`floor_K[type]`、代表性容差 `ε`、`report_floor`(≈27)、val 尺寸(McNemar 功效)、`seed`。设计固定**算法与约束结构**,数值不臆造。
|
||||||
|
|
||||||
|
## 14. 风险与回退
|
||||||
|
|
||||||
|
| 风险 | 回退 |
|
||||||
|
|------|------|
|
||||||
|
| 100 训练视频信号不足、进化曲线平 | 切方案 B:外部 benchmark 建树当训练集,全 900 Video-MME 留 held-out(A 打底 B 升级) |
|
||||||
|
| floor 与 ε 联合不可行 | 选择器 fail loud 报冲突 → 放松 floor_K 或 ε 或 N_trainval |
|
||||||
|
| 诊断 LLM 成本/耗时(240 题) | Redis 缓存 + 逐题续跑;一次性离线 |
|
||||||
|
| test 长尾类型不可报 | 折进 overall + rare-aggregate 行,披露计数 |
|
||||||
|
|
||||||
|
## 相关
|
||||||
|
- baseline 结果:`workspaces/default/harness.db` run_id=`infer_adhoc`
|
||||||
|
- 诊断模块:`core/evolution/diagnose.py`、`core/evolution/types.py`
|
||||||
|
- 被替代:自造题 v3(`2026-07-15-question-gen-v3-construction-paradigm-design.md`)
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
# gate 验证提速设计(v3):连续并发 gate + Redis 复用
|
||||||
|
|
||||||
|
> 2026-07-16。背景:Video-MME 900 训练实测,epoch 1 单题型 gate 双臂验证 ~3h(两臂串行 × 块串行 × 题型串行三重串行,~6 题型/step)→ step ~18h,3 epochs 不可行。v1 设计的"预灌 BaselineCache"经 Codex 审查发现统计致命伤已废弃(见 §7)。本版方案:**②″ 连续并发 gate**(题型并行 + 双臂并行 + 逐对连续早停,一次调度重构)+ **③ Redis 复用保障**。统计内核(e-值公式、四出口、配对翻转、信息量阶梯、题尽/试用期通道)一行不动。
|
||||||
|
|
||||||
|
## 1. 统计合法性(②″ 的前提)
|
||||||
|
|
||||||
|
| 论点 | 依据 |
|
||||||
|
|---|---|
|
||||||
|
| 块=8 不是统计需要 | TRM4 注释原文"块大小=推理并发度,块内跑满"——并发 8 时代的工程遗迹。e-process 为 anytime-valid 上鞅(Ville 不等式),在**预先声明的样本顺序**下任意时刻停下判定假阳率仍 ≤ 1/e_confirm;逐对判定是比逐块更细的合法 optional stopping |
|
||||||
|
| **统计消费必须按固定阶梯序前缀,不得按完成到达序**(Codex v2 复审 C1) | 到达序消费不合法:base 臂可缓存命中瞬间返回、cand 臂必新鲜跑,两臂延迟不对称,配对完成时间由 cand 延迟主导;若 cand 延迟与对错相关(走满 40 步的慢轨迹更易错),早到翻转对系统性偏向 W 型 → e-值虚高 → 假接受。**修法**:推理全并发乱序执行,但 (W,L) 更新与 gate_decision 只在"阶梯序最长已配齐前缀"延伸时推进——判定顺序回到预先声明的阶梯序,无条件合法;INFRA 单元视为"已解决(剔除)"不阻塞前缀推进 |
|
||||||
|
| 冻结后 in-flight 结果丢弃合法 | 在固定前缀消费下,停止时刻 τ 之后的样本不进入统计是标准 optional stopping;丢弃不引入偏差(到达序消费下则不成立,故必须配合上一条) |
|
||||||
|
| 题型间并行无实质依赖 | 各题型 agent 只加载自己的 skill 文件(12 题型各有专属 .md),A 型 accept 改 A 的文件,对 B 型推理内容零影响。现行字母序滚动版本只是记账先后,非实质依赖。唯一共享文件 default-strategy.md 仅在题型缺专属文件时 fallback——本次 12 型全有专属文件,不触发;**启动时 fail-fast 断言:同 step 内多个题型不得映射同一 target_file**,违反即中止(RuntimeError,防未来配置漂移) |
|
||||||
|
|
||||||
|
## 2. 改动 ②″:连续并发 gate 调度器
|
||||||
|
|
||||||
|
### 2.1 新流程(替换 `_gate_batch_skills` 串行 for + `validate.py` 块循环)
|
||||||
|
|
||||||
|
```text
|
||||||
|
step 内 gate 阶段:
|
||||||
|
Phase A. 并行进化:全部案例包题型(剔除 cooldown)gather 调 evolve_single_skill
|
||||||
|
无真实改动的题型照旧写 skipped step_report 退出
|
||||||
|
Phase B. 装配:每题型 阶梯出题(排除本 step 案例单元, 截断 n_max=40) + 物化候选目录
|
||||||
|
Phase C. 连续监控调度(推理乱序并发 × 统计前缀有序):
|
||||||
|
所有 (题型, 单元, 臂) 任务按「题型 round-robin × 题型内阶梯序」交错压入共享并发池(32)
|
||||||
|
base 臂任务:BaselineCache 命中 → 立即完成;miss → 新鲜跑基线版本,非 INFRA 回写缓存
|
||||||
|
cand 臂任务:跑该题型候选目录
|
||||||
|
事件循环(每题型持有前缀指针,初始 0):
|
||||||
|
单元两臂齐 → 标记该单元"已解决"(配对翻转 / 任一臂 INFRA 则剔除,护栏计数 §2.3)
|
||||||
|
若阶梯序前缀因此延伸 → 沿前缀逐单元消费:更新 (W,L,n_used) → gate_decision
|
||||||
|
判定 ≠ continue → 冻结该题型:排队未启任务撤销(启动前查冻结标志),
|
||||||
|
in-flight 任务跑完落库但不计入(τ 之后样本,合法丢弃)
|
||||||
|
前缀之外已完成的单元暂存,等前缀推进到它时才消费(统计顺序 = 预声明阶梯序)
|
||||||
|
某题型前缀消费完全部单元仍 continue → n_remaining=0 → 题尽第四出口(provisional/inertia)
|
||||||
|
Phase D. 汇总:按字母序对已判定题型依次执行现有 _accept_skill / _record_rejected_skill
|
||||||
|
(文件不相交,顺序仅为确定性;gate_evidence/step_report/quadrant_pairs 照写)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 保留与消解
|
||||||
|
|
||||||
|
| 项 | 处置 |
|
||||||
|
|---|---|
|
||||||
|
| e-值公式 / 四出口 / w_net_min / delta_min / lambda_dir / futility | **不动**(判定函数 `gate_decision` 原样,只是调用时机从"每块末"变"每对完成") |
|
||||||
|
| 信息量阶梯出题 + 案例单元排除 + n_max=40 | **不动**(发射顺序即阶梯序) |
|
||||||
|
| BaselineCache(miss 新鲜跑、INFRA 不写、内容寻址) | **不动**(无预灌;epoch 1 冷缓存,base 臂新鲜跑但与所有臂共享并发池) |
|
||||||
|
| provisional → probation 试用 / epoch 末结算回滚 / cooldown / 黑名单 | **不动** |
|
||||||
|
| `gate_block=8` 分块 | **消解删除**(config 键与 validate 块循环一并移除;CLAUDE.md §4.7 #6 需在 commit 标注本次语义修订:块序贯 → 阶梯序前缀逐对序贯,判据不变) |
|
||||||
|
| 每块两个 run_id(`_b{i}_base/_cand`) | **替换**为每臂一个 run_id(`..._gate_{slug}_base` / `_cand`)。溯源口径变化:predictions 不再携带块信息,gate_evidence 行以 **ladder_rank(阶梯序号)替代 block_idx** 补足调度溯源 |
|
||||||
|
| step 重跑幂等(**修复现行潜伏 bug**) | 现行 `_run_step` 只清 rollout run_id,gate 派生 run_id 的旧行从不清理,崩溃重跑会累积重复 predictions(HarnessLog 无主键去重)。**新增**:step 开始时一并清理 `{step_run_id}_gate_%` 的 predictions/traces 及该 (epoch, step) 的 gate_evidence / quadrant_pairs / step_report 旧行 |
|
||||||
|
| 断点粒度 | **明确声明**:gate 内无断点(与现行块方案一致),checkpoint 仍为 step 粒度;崩溃重跑该 step 时由上一行的清理保证干净重放,Redis/BaselineCache 命中使重放近零成本 |
|
||||||
|
| n_remaining 口径 | 等价迁移:= 阶梯计划中前缀尚未消费的单元数;题尽判定与现行一致 |
|
||||||
|
|
||||||
|
### 2.3 INFRA 护栏(并行下的等价迁移)
|
||||||
|
|
||||||
|
现行:跨块累计 error 率,分母 ≥10 且 > gate_guard_err(0.10) → 中止训练。迁移为**每题型运行时计数器**:分子=该题型 INFRA 单元数(任一臂),分母=该题型已完成推理的单元次数(两臂各计,缓存命中不计),阈值与中止行为不变。判定在每次单元完成事件时检查,先于配对更新。
|
||||||
|
|
||||||
|
### 2.4 并发与调度事实
|
||||||
|
|
||||||
|
- **实现级约束**(Codex I1):并发控制为 gate 调度器持有的单一共享 `asyncio.Semaphore(concurrency)`,单元级推理任务在该信号量下执行;不得沿用"每次 run_inference 各建信号量"(否则并行臂叠加超限)。峰值在飞请求恒 ≤32
|
||||||
|
- **公平调度**(Codex I2):任务发射严格按题型 round-robin 轮转(题型内按阶梯序),防止大题型占满槽饿死小题型;冻结题型立即停止补发
|
||||||
|
- **进化并行的并发契约**(Codex I3):`GovernedLLMClient` 已在 rollout 中承受 32 路并发调用(治理栈全线程/协程安全),6 路 gather 进化无新增风险;设计约束:Phase A-C 只读训练 state,**Phase D 是唯一写 state 的阶段**(probation/cooldown/rejected_buffer/changed_task_types 均在 Phase D 按字母序串行落账)
|
||||||
|
- 浪费上界:每题型判定瞬间 in-flight 的任务 ≤ 并发宽度;排队未启动的全部省下。相比"全发不早停"省 ~50-70% token,相比现行块早停多耗 ≤32 题次/题型
|
||||||
|
|
||||||
|
## 3. 改动 ③:Redis 复用保障
|
||||||
|
|
||||||
|
| 措施 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| 缓存键成分不动 | 消息内容(skills/prompts v1 正文、冻结题池、`build_batches(seed=epoch)` 确定性批次)与盐(run_id 派生自 `infer_adhoc`)零变化 → 重启后 rollout / 诊断 / 进化调用命中已写缓存 |
|
||||||
|
| gate 臂 run_id 变化的影响 | ②″ 将 `_b{i}_base` 改为 `_base`,今日旧 gate 调用(仅 Action Reasoning 部分块)的盐失配不复用——量小(~100 题次),可接受,如实记录 |
|
||||||
|
| `.env` `REDIS_CACHE_TTL` 86400→604800 | 仅一行;只影响未来写入,保障多天训练中断重启后早期调用仍在 |
|
||||||
|
| 一次性续期今日键 | 重启前运行一次(运维动作,不入库代码):`conda run -n Video-Tree-TRM python -c` 连 `.env` 的 REDIS_URL,`for k in scan_iter('llm_cache:*'): 0 < ttl(k) < 604800 and expire(k, 604800)` |
|
||||||
|
| 复用边界(诚实声明) | Agent 逐步追加 messages,某步 miss(如原调用 503 未入缓存 / TTL 过期)后该轨迹后续全 miss 并可能分叉——复用是"命中前缀零成本"的尽力而为,非全量保证。今日实测命中率 42% 佐证机制有效 |
|
||||||
|
|
||||||
|
## 4. 非功能四维
|
||||||
|
|
||||||
|
| 维度 | 保障 |
|
||||||
|
|---|---|
|
||||||
|
| 持久化 | predictions/traces 逐题落库(HarnessLog 单连接+锁,不变);BaselineCache 逐条原子写(不变);判定即写 gate_evidence/step_report |
|
||||||
|
| 幂等 | step 重跑前清 rollout run_id **及全部 gate 派生行**(§2.2 修复项,现行代码只清前者);BaselineCache 同键同值重写无害 |
|
||||||
|
| 断点续跑 | checkpoint 粒度仍为 step(gate 中途崩溃 → 重启重跑该 step,与现行一致);Redis/BaselineCache 命中使重跑近零成本 |
|
||||||
|
| 原子性 | 判定冻结为内存事件,落库单条原子;无跨文件半写窗口 |
|
||||||
|
|
||||||
|
## 5. 前序行为审计
|
||||||
|
|
||||||
|
| 现有行为 | 处置 |
|
||||||
|
|---|---|
|
||||||
|
| 题型串行 for(字母序滚动版本) | **替换**:题型并行,accepts 按字母序统一合并(内容等价,记账顺序确定) |
|
||||||
|
| 进化串行 | **替换**:题型间 gather 并行(rejected_buffer / budget per-type 独立,无共享) |
|
||||||
|
| 块序贯(8/块)+ 两臂串行 | **替换**:逐对连续序贯 + 全臂共享并发池 |
|
||||||
|
| base 臂缓存 miss 新鲜跑 / INFRA 不写缓存 | **保留** |
|
||||||
|
| INFRA 护栏(分母≥10 且 >10% 中止) | **等价迁移**(§2.3) |
|
||||||
|
| cooldown / skipped / 无改动跳过 + step_report | **保留** |
|
||||||
|
| gate_evidence / quadrant_pairs / candidate_correctness 增量合并 | **保留** |
|
||||||
|
| e-process 判定与四出口、题尽通道、probation | **不动** |
|
||||||
|
|
||||||
|
## 6. 测试
|
||||||
|
|
||||||
|
- 单测(调度器,注入假 run_inference 与可控延迟):
|
||||||
|
- **前缀有序性(核心)**:构造"阶梯尾部单元先完成"的乱序到达,断言 (W,L) 更新顺序严格等于阶梯序、前缀未齐时不判定;
|
||||||
|
- 同一组固定对错序列下,前缀逐对判定结果(action/W/L/E)与旧块序贯逐块判定一致(早停点可更早,判定方向一致);
|
||||||
|
- 过线后排队任务不再启动(计数断言),in-flight 结果不改变已冻结 (W,L);
|
||||||
|
- step 重跑幂等:预插旧 gate 行 → 重跑 step → 断言无重复行(C2 修复);
|
||||||
|
- fail-fast:两题型映射同一 target_file 时启动即报错;
|
||||||
|
- 题尽路径:单元耗尽 → 第四出口与现行一致;
|
||||||
|
- INFRA:单臂 INFRA 单元剔除不入配对,护栏计数阈值触发中止;
|
||||||
|
- 题型并行:两题型交错完成,互不污染彼此 (W,L) 与判定。
|
||||||
|
- 单测(进化并行):两题型 gather 进化,rejected_buffer 各自独立生效。
|
||||||
|
- 集成:小型真实 workspace 跑一个 step,断言 gate_evidence/step_report/quadrant_pairs 落库完整、accepts 正确推进版本。
|
||||||
|
- 回归:validate/gate_ladder/runner 现有测试全绿(块相关测试改写为连续语义)。
|
||||||
|
|
||||||
|
## 7. 已否决备选(含 v1 失败记录)
|
||||||
|
|
||||||
|
- **①(v1)预灌 BaselineCache**:❌ 两个致命伤。(a) 统计:冷启动阶梯按 seed 基线"错题优先"选题,预灌使 base 臂 = 选题依据的同一份旧样本;对选中的 seed 错题 base 被钉死为错、新鲜 cand 以概率 p 答对 → W 系统性膨胀,候选无改进也会假接受。现行"miss 后新鲜跑"的测量-选题独立性是刻意设计,不可省。(b) resume 污染:预灌用"当前"skill 内容算 hash,resume 时已进化题型会把 v1 旧预测灌到新内容 hash 下,base 臂错误命中(Codex Critical ×2,均已验证)。
|
||||||
|
- **到达序消费配对(②″ 初稿)**:❌ base 缓存命中/cand 必新鲜跑的延迟不对称 + 对错-延迟相关,使早到翻转对偏向 W 型,e-值虚高假接受;改为阶梯序前缀消费(§1)。
|
||||||
|
- **B. 小题型合并进化 default-strategy.md**:future work。合并 gate 池(9+14+17+23=63 单元)可达正式接受线且信号更密,但存在子群伤害风险(平均变好掩盖单型受损)且叠加变量;本轮跑 A(小题型走 provisional+probation 通道),其结果作为 B 的对照证据。
|
||||||
|
- **top-K 题型/step**:用户否决,保留全部有错题型进化。
|
||||||
|
- **砍 gate_n_max / 放宽 e 阈值**:动统计参数,收益已被 ②″ 覆盖。
|
||||||
|
- **块轮次锁步(②′)**:被更优的连续监控取代(锁步有轮末空转,连续无)。
|
||||||
|
|
||||||
|
## 8. 预期收益
|
||||||
|
|
||||||
|
| 项 | 现状 | ②″ 后 |
|
||||||
|
|---|---|---|
|
||||||
|
| step 内 gate(~6 题型) | 三重串行 ~9-18h | 共享 32 并发连续跑,**~1.5-2.5h**(≈总需题次×单题时长/32) |
|
||||||
|
| 进化阶段(~6 次 LLM) | 串行 ~30-60min | 并行 ~10min |
|
||||||
|
| 3 epochs(5 step/epoch + epoch 末 val/holdout) | 数天-一周 | **~1.5-2.5 天** |
|
||||||
|
|
||||||
|
改动面:`app/harness/validate.py`(块循环→连续调度器)、`app/harness/runner.py`(`_gate_batch_skills` 并行装配)、`config`(删 gate_block)、`.env` 一行、运维一次性续期。
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
# 训练前缺陷修复设计(video-split → Video-MME 900 自进化训练)
|
||||||
|
|
||||||
|
> 2026-07-16。两轮审查(第一轮接线预检 + 第二轮 8 维度多代理审查:**19 条确认 / 4 条存疑 / 12 条误报**)。本设计覆盖:19 条确认缺陷 + 4 条存疑中采纳 2 条 + 接线缺口 2 项 + 切分层优化 3 项,逐条映射见 §2.5 覆盖矩阵。
|
||||||
|
> **目标**:让 video-split 冻结切分驱动的 Video-MME 900 题自进化训练**一次跑出有效结果**(不崩、不静默失效、信号不被污染)。
|
||||||
|
|
||||||
|
## 1. 背景与问题总览
|
||||||
|
|
||||||
|
自进化训练对标 PyTorch 训练循环(出题=DataLoader / 推理=forward / 诊断=backward / 进化=optimizer.step)。审查发现三类根问题:
|
||||||
|
|
||||||
|
| 类别 | 代表 | 后果 |
|
||||||
|
|------|------|------|
|
||||||
|
| **进化引擎哑火** | 5 个 evolve/momentum 模板 TRM4→TRM5 迁移遗漏,`else ""` 静默兜底 | 3 epochs 跑完但 skills 零进化,best 恒 v1 |
|
||||||
|
| **诊断链路坍缩** | traces 表从不写入(TracePlugin 未迁移) | 诊断瀑布拿空轨迹,算法保真 #7 失效 |
|
||||||
|
| **崩溃与信号污染** | 微型题型断言崩、prediction 非标量击穿、缓存跨 epoch 重放、SSE 截断毒化等 | 训练中途崩溃或统计信号失真 |
|
||||||
|
|
||||||
|
叠加**接线缺口**:冻结 `pools.json` 无路径进入训练 workspace(`--fresh` 从 seed 重建不带切分),直接训练会静默丢弃标定成果、按 unit 重切。
|
||||||
|
|
||||||
|
## 2. 设计目标与非目标
|
||||||
|
|
||||||
|
**目标**:修复 §2.5 矩阵所列全部缺陷(19 确认 + 2 存疑采纳)+ 接线;切分层做三处优化(可训练性门槛、val_ratio、tier 感知分配)提升信号质量;产出零参可复现的训练入口。
|
||||||
|
|
||||||
|
**非目标**:不改 gate/EMA 核心数学(算法保真 #4/#5);不实现 epoch 层 per-skill best 混搭(见 §9 future work);不引入向后兼容(P4 显式优于隐式,直接改)。
|
||||||
|
|
||||||
|
## 2.5 缺陷覆盖矩阵(可追溯性)
|
||||||
|
|
||||||
|
原始编号取自第二轮工作流报告(`research-wiki/reviews/2026-07-16-preflight-train-review.md`)与第一轮接线预检。状态:C=确认 / P=存疑采纳 / W=接线(已知缺口)。
|
||||||
|
|
||||||
|
| 设计编号 | 原始位置 | 状态 | 修法节 |
|
||||||
|
|---------|---------|------|--------|
|
||||||
|
| P0-1 | runner.py:2272 evolve 模板缺失 | C | §4 |
|
||||||
|
| P0-6 | momentum.py:151 slow_momentum.md 缺失 | C | §4 |
|
||||||
|
| P0-2 | inference.py:462 traces 未写 | C | §6 |
|
||||||
|
| P0-3 | validate.py:682 + gate.py:99 + runner.py:2044 微型题型三连雷 | C | §6 |
|
||||||
|
| P0-4 | inference.py:423 prediction 非标量击穿 | C | §6 |
|
||||||
|
| P0-5 | runner.py:316 early_stop step 单位 | C | §6 |
|
||||||
|
| P1-1 | main.py:93 缓存跨 epoch 重放 + TTL | C | §8 |
|
||||||
|
| P1-2 | llm.py:116 SSE 截断毒化 | C | §8 |
|
||||||
|
| P1-3 | llm.py:182 断连不重试 | C | §8 |
|
||||||
|
| P1-4 | validate.py:304 基线臂 INFRA 污染 | C | §8 |
|
||||||
|
| P1-5 | diagnose.py:2194 + runner.py:1019 降级误入 defect | C | §6 |
|
||||||
|
| P1-6 | patch.py:343 冻结区跨度 | C | §7 |
|
||||||
|
| P1-7 | workspace.py:282 manifest 非原子写 | C | §8 |
|
||||||
|
| P2-1 | video_split_cli.py:557 冻结产物无覆盖保护 | C | §5.2 |
|
||||||
|
| P2-2 | log.py:77 基线元数据被改写 | C | §8 |
|
||||||
|
| P2-3 | diagnose.py:2081 + inference.py:461 重复行双计 | C+P | §6(step DELETE) |
|
||||||
|
| P2-4 | diagnose.py:2194(离线)uncertain 永久降级 | C | §8(--retry-uncertain) |
|
||||||
|
| P2-5 | runner.py:1444 dual_metric 口径歧义 | P | §8 |
|
||||||
|
| P2-6 | breaker.py:36 熔断半开烧穿 | P | §8 |
|
||||||
|
| 慢更新非幂等 | 第一轮已知 #5 | W | §6(gate_epoch_observed 立即落盘) |
|
||||||
|
| 接线-1 | 冻结 pools 无路径进 workspace(已知 #1) | W | §5.3 |
|
||||||
|
| 接线-2 | global 加载零一致性校验(已知 #2) | W | §5.3 |
|
||||||
|
|
||||||
|
**有意合并/不单列的项**(说明去向,避免"静默漏修"):
|
||||||
|
|
||||||
|
| 第一轮原始项 | 处置 |
|
||||||
|
|-------------|------|
|
||||||
|
| gate_ladder.py:93 strict=False 缺题静默当错题 | 由 §6 可训练性预检消除主场景(缺题类被剔除);残余以 fail-loud 补强(预检后 gate 建立时断言 baseline 覆盖全部 ladder 单元) |
|
||||||
|
| batching.py:200 缺 correctness 静默丢弃 | 同上,预检保证 diag 池题全在 correctness 内;保留 fail-loud(缺 correctness 即报错而非丢弃) |
|
||||||
|
| resume 硬依赖 checkpoint(已知 #8) | 本轮为 fresh 训练,seed 携带 pools 后无需 resume 建池;不改 resume 语义 |
|
||||||
|
| baseline_diagnosis 断点续跑粒度(存疑,非采纳) | 仅修 docstring 表述(§8 末),行为不改(Redis 缓存已缓解重烧) |
|
||||||
|
|
||||||
|
## 3. 架构:四工作包按依赖序
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph LR
|
||||||
|
WP1[WP1 资产迁移<br/>模板+fail-loud+死字段] --> WP3[WP3 训练循环与进化引擎]
|
||||||
|
WP2[WP2 切分与接线<br/>产出训练输入] --> RUN[训练启动]
|
||||||
|
WP3 --> RUN
|
||||||
|
WP4[WP4 韧性与持久化] --> RUN
|
||||||
|
WP1 -.集成测试依赖.-> WP3
|
||||||
|
```
|
||||||
|
|
||||||
|
WP1 零风险纯搬运先解锁引擎;WP2/WP4 与 WP3 无代码交集可并行;WP3 主体修复依赖 WP1 模板做集成测试。
|
||||||
|
|
||||||
|
## 4. WP1 —— 资产迁移(修 P0-1 / P0-6)
|
||||||
|
|
||||||
|
从 `/home/iomgaa/Projects/Video-Tree-TRM4/prompts/` 拷 5 文件到 TRM5 根 `prompts/`:`evolve_skill/system/tool/rank.md`、`slow_momentum.md`(进化引导 prompt = 引擎一部分,不参与版本化进化)。
|
||||||
|
|
||||||
|
- **迁移验收(强制)**:逐个核对模板输出契约与 TRM5 解析代码期望——`evolve_*` 的 `{suggestions, edits}` 结构 vs `_parse_llm_json`/`_apply_one`;`slow_momentum.md` vs `run_slow_momentum`。TRM4→TRM5 代码有漂移,裸拷贝不核对 = 新 bug。
|
||||||
|
- **加载器 fail-loud**:`_load_evolve_prompts`/`_load_diagnose_prompts` 的 `else ""` 改 `FileNotFoundError`(列缺失文件名)。
|
||||||
|
- **死字段清理**:删 `EvolvePrompts.consolidate_system`(consolidate_appendix 用内联 prompt)、`DiagnosePrompts.span_eval_user`(无消费点)及对应 runner 加载行。
|
||||||
|
|
||||||
|
## 5. WP2 —— 切分与接线
|
||||||
|
|
||||||
|
### 5.1 tier 感知切分(`split_selection.py` / `build_split.py`)
|
||||||
|
|
||||||
|
trainval 内部视频组分配从"correctness 分层随机"升级为 **tier 感知确定性贪心**(纯函数、固定 seed):
|
||||||
|
|
||||||
|
```
|
||||||
|
输入: trainval 视频组集合 G(每组: t2_count/t1_count/wrong_count/question_count,
|
||||||
|
tier 来自 baseline_diagnosis 按 fingerprint 读);val_ratio;val_wrong_min
|
||||||
|
目标: 最大化 diag 池 T2 捕获,同时满足 val 功效
|
||||||
|
约束: ①视频原子(组内题同进同出) ②val 题数 ≈ val_ratio×trainval题数 ③val 错题 ≥ val_wrong_min
|
||||||
|
|
||||||
|
阶段1 初分(单趟,O(n log n)):
|
||||||
|
按 (t2_count desc, wrong_count desc, video_id) 排序 G
|
||||||
|
依次装入 diag,累计题数达 (1-val_ratio)×总题数 即停 → 余下全部进 val
|
||||||
|
阶段2 功效修复(单调收敛,每组至多移动一次):
|
||||||
|
while val 错题 < val_wrong_min:
|
||||||
|
从 diag 中挑「wrong_count>0 且 t2_count 最小」的组移入 val # 严格减少 diag 错题冗余
|
||||||
|
该组标记 moved,后续不再参与挑选 # 单调变元:候选集严格缩小
|
||||||
|
if 无 moved=false 且 wrong_count>0 的组可挑 → raise InfeasibleSplitError
|
||||||
|
# 终止性:每轮候选集 |{未 moved 且 wrong>0}| 至少减 1,有限步内收敛或 fail loud
|
||||||
|
```
|
||||||
|
|
||||||
|
与既有 video-split 设计(`2026-07-15-results-driven-video-split-design.md §8`)的口径变更:**保留** correctness 分层的本质(阶段2 只在 wrong_count>0 的组间挪动,保证两池都有对/错题)、`eval_min_per_class`(由 §6 预检独立强制)、McNemar 功效(`val_wrong_min≈20` 硬约束不变);**替代**"随机分层"为"tier 优先确定性贪心"(新增 T2 捕获目标 + 固定 seed 可复现)。
|
||||||
|
|
||||||
|
配置:`val_ratio` 作用域 = **trainval 池(当前 300 题)内部**的 val 题数占比,`0.3 → 0.4`。数字推算基于当前冻结切分统计(trainval 错题 93 = val 29 + diag 64):val 错题 ≈ ⌈93×0.4⌉ = 38、diag 错题 ≈ ⌊93×0.6⌋ = 55~56(tier 感知使 T2 尽量留 diag,实际错题落点略有偏移,以重切后统计为准)。甜点判断:val 38 让整包终审更抗噪、diag 56 进化信号仍过剩;0.5 过冲(每类 3 epochs 仅 ~4 次进化机会,迭代受损)。`val_ratio`、`val_wrong_min` 作为**数值参数进科研 YAML + run 快照**(会被扫动/对比);tier 感知**策略**固定不加 on/off 开关(无扫动需求,YAGNI)。重切零 LLM 成本(诊断已按指纹落库)。
|
||||||
|
|
||||||
|
### 5.2 冻结产物覆盖保护(修 P2-1)
|
||||||
|
|
||||||
|
`build_split` 冻结前检查产物存在性:存在且内容指纹不同 → 报错并提示 `--force`(CLI 补上此缺失参数);`--force` 时旧产物重命名 `pools.json.bak.<旧指纹前8>` 再写。
|
||||||
|
|
||||||
|
### 5.3 seed 携带切分(接线核心)
|
||||||
|
|
||||||
|
| 项 | 设计 |
|
||||||
|
|----|------|
|
||||||
|
| `init_seed` | 新增可选 `pools_json`/`split_manifest`,提供则拷入 seed 目录 |
|
||||||
|
| `init_workspace_from_seed` | seed 目录有 `pools.json` 则连 manifest 拷入 workspace(拷 baseline.db 之后) |
|
||||||
|
| `build_or_load_pools`(global 补校验,修已知 #2) | 加载冻结 pools 强校验 `baseline_run_id` 匹配 seed;有 manifest 时校验 `sha256(pools.json)==manifest.pools_sha256` |
|
||||||
|
| 新 seed `adhoc-baseline` | `extract_run_db` 加 `dedupe_per_question`(每 qid 按 rowid 取首行,902→900 对齐 canonical)+ `init_seed(baseline_run_id='infer_adhoc', pools_json=…)` |
|
||||||
|
|
||||||
|
seed 最终形态:`seed.json / baseline.db(900) / pools.json / split_manifest.json / skills/v1 / prompts/v1`。
|
||||||
|
|
||||||
|
## 6. WP3 —— 训练循环与进化引擎
|
||||||
|
|
||||||
|
| # | 缺陷 | 修法 |
|
||||||
|
|---|------|------|
|
||||||
|
| P0-3 | 微型题型三连雷 | `train()` 加**可训练性预检**(纯函数):题型 `n_val<eval_min_per_class(2)` 或 `n_units<trainable_min_units(8)` → 从 diag/val + gate task_types 剔除并打印清单;断言/空阶梯保留作纵深防御;test 池不过滤 |
|
||||||
|
| P0-4 | prediction 非标量击穿 | 落库前 `_to_text_field` 归一化;`log.insert` 移入单题 try 块,绑定异常记 error 不击穿 gather |
|
||||||
|
| P0-5 | early_stop 单位错误 | step→**epoch** 计数(`epochs_since_best_improved`),config 注释对齐,checkpoint 字段同步改名 |
|
||||||
|
| P0-2 | traces 空 | 复用离线管线 `StepsJsonRunLog` 适配器包装训练 RunLog,`get_traces` 空时从 steps_json 转换,算法 #7 恢复 |
|
||||||
|
| holdout | 四向重复评估 | baseline 从 seed 基线预测推导(0 推理);final 真评 600;best_hard 版本备忘录(未评过且≠final 才评);best_mixed 引用赢家成绩标指针(0 推理)。**去重做在 harness 逻辑层**(配合 epoch 盐,同版本不重采样)。**resume 语义**:备忘录不单独持久化,而在 resume 时从 `holdout_eval` 表 hydrate——查已落库的 `(skills_v,prompts_v)→test_acc` 重建已评集合(该表本就是每次评估的落库处),checkpoint 无需新增字段、无旧 checkpoint 迁移问题;hydrate 后同版本不再重评 |
|
||||||
|
| P1-5 | 诊断降级误入 defect | `cause_category=None`(judge 基础设施异常)与 `degraded=True` 错题按 **lapse 方向分流**;step 降级占比>50% 报错中止 |
|
||||||
|
| P2-3/#5 | step 重跑双计 + 慢更新非幂等 | `_run_step` 开始先 `DELETE ... WHERE run_id=<本step>`;`_refresh_gate_ladder` 保存 gate_pools 后**立即**原子落盘 checkpoint(`gate_epoch_observed=True`) 消除双计窗口 |
|
||||||
|
|
||||||
|
`run_holdout_eval` 保持开启(去重版),逐 epoch test 曲线四线齐全,成本从 2400 降至 600~1200 次/epoch。
|
||||||
|
|
||||||
|
## 7. WP3 —— 进化引擎 patch 加固(修 P1-6,算法 #8 防御)
|
||||||
|
|
||||||
|
| 层 | 改动 |
|
||||||
|
|----|------|
|
||||||
|
| 跨度判定 | `_in_ranges` 从"只查起点"改为"整个 target 跨度 [pos,pos+len) 与保护区相交即拒";`_do_insert_after` 同 |
|
||||||
|
| 注入检查 | edit payload/target 含 `APPENDIX_START/END`、`MOMENTUM_START/END` 字面量 → 拒绝该 edit |
|
||||||
|
| 最后防线 | `validate_skill` 增 marker 完整性校验(成对、有序、各至多一对),违反整体 reject |
|
||||||
|
|
||||||
|
不改"保护跨度"语义方向,只把已声明的保护做完整。
|
||||||
|
|
||||||
|
## 8. WP4 —— 韧性与持久化
|
||||||
|
|
||||||
|
| # | 缺陷 | 修法 |
|
||||||
|
|---|------|------|
|
||||||
|
| P1-1 | 缓存跨 epoch 重放 | `chat()` 加 `cache_salt`;训练 rollout/val/test/**gate 候选臂**推理注入含 epoch 的盐(跨 epoch 真实重采样、同 epoch 续跑仍命中);judge/evolve/离线切分无盐。TTL 修:`REDIS_CACHE_TTL≤0` 启动即 ValueError(消灭"0=永不过期"隐式语义)。**同步改 `.env`/`.env.example` 给正整数**(当前值 0,不改则修复后启动即崩),训练场景建议 ≥ 单次训练时长 |
|
||||||
|
| P1-1a | epoch 盐 vs gate 配对保真(Codex Critical) | **gate 双臂配对语义由架构保证、不受 LLM 缓存盐影响**:基线臂走 app 层 `BaselineCache`(unit 键 `task_type+s_hash+prompts_version+unit_id`,固定快照,`validate.py:_resolve_baseline_block`),候选臂每块新采样(`_run_candidate_block`)。配对翻转比较的是「固定基线快照 vs 新候选采样」(算法保真 #6 现有语义),epoch 盐只作用于**候选臂/rollout 的 LLM 层缓存键**,不触碰基线臂、不改变配对语义。gate 序贯 e-process 消费阶梯上**不同 block(不同 unit)**累积 e-value,非对同 unit 反复测;候选臂盐用 block 级 `run_id`(已含 epoch+block+candidate 区分)保证每次真实重测独立采样。**实现约束**:基线臂 miss 时的新鲜推理**不注入 epoch 盐**(保持基线快照跨 epoch 稳定,与 BaselineCache 语义一致)|
|
||||||
|
| P1-2 | SSE 截断毒化 | `_consume_stream` 校验 `[DONE]`/finish_reason 才算成功;流耗尽未完成抛 `_SseAnomaly` 进重试梯,**绝不写缓存** |
|
||||||
|
| P1-3 | 断连不重试 | `_is_transient_error` 扩为 `httpx.TimeoutException`+`httpx.TransportError` 两族 |
|
||||||
|
| P2-6 | 熔断半开烧穿 | cooldown 到期只放行 1 探针(half-open 锁),成功才闭合 |
|
||||||
|
| P1-4 | 基线臂 INFRA 污染 | INFRA 单元不写 BaselineCache、不计 W/L;护栏检查移到写缓存**之前** |
|
||||||
|
| P1-7 | manifest 非原子写 | `update_manifest/record_run/update_best/_scaffold` 统一 tmp+`os.replace`(复用 checkpoint 模式)|
|
||||||
|
| P2-2 | 基线元数据被改写 | 只读查询基线的两处改走只读连接,不经 `HarnessLog.__init__` upsert |
|
||||||
|
| P2-5 | dual_metric 口径歧义 | 慢更新 R2 行 `version_kind` 改 `slow_candidate`,`final` 恢复唯一语义 |
|
||||||
|
| P2-4 | 离线 uncertain 不可重试 | `video_split_cli` 加 `--retry-uncertain`(done 集排除 uncertain 行);docstring "逐行落库"表述修正为"run 末批量落库" |
|
||||||
|
|
||||||
|
## 9. Rejected alternatives / Future work
|
||||||
|
|
||||||
|
| 选项 | 为何不做 |
|
||||||
|
|------|---------|
|
||||||
|
| epoch 层 per-skill best 混搭 | per-class val 样本太小(中类 5-10 题)终审频繁选错;prompts 随慢更新漂移致跨 epoch 成绩不可拼接。合理形态是训练**结束后**一次性 final assembly(每类取 gate 末次接受版本 + 最终 prompts,全 val 评一次比 best_hard),成本一次推理、不动循环 → **记 future work** |
|
||||||
|
| val_ratio 0.5 | 每类 3 epochs 仅 ~4 次进化机会,迭代受损;val 错题 38→46 边际收益趋零 |
|
||||||
|
| 确定性评估(temp=0) | 统计效率最高但基线用默认温度采,换口径要重烧全部基线,本轮不可行 |
|
||||||
|
| 会计层去重替代 epoch 盐 | 需动 gate/EMA 保真区且造不出新信息(每题永远只见一次抽样)|
|
||||||
|
|
||||||
|
## 10. 非功能四维(全设计汇总)
|
||||||
|
|
||||||
|
| 维度 | 保障 |
|
||||||
|
|------|------|
|
||||||
|
| **持久化** | 冻结产物/manifest/checkpoint 全 tmp+replace 原子写;holdout 备忘录入 checkpoint |
|
||||||
|
| **幂等** | step 重跑先 DELETE 后写;seed/冻结产物存在即拒覆盖(--force 显式);诊断按指纹 upsert 不变 |
|
||||||
|
| **断点续跑** | epoch 盐保同 epoch 命中;gate_epoch_observed 在 gate_pools 保存后立即落盘,消除慢更新双计窗口 |
|
||||||
|
| **原子性** | manifest 补齐原子写(对齐 checkpoint 先例);pools 冻结沿用现有原子写 |
|
||||||
|
|
||||||
|
## 11. 测试策略
|
||||||
|
|
||||||
|
| 层 | 覆盖 |
|
||||||
|
|----|------|
|
||||||
|
| 单元(新增) | tier 选择器(真实 baseline_diagnosis 二次构造:T2 捕获/val 错题≥20/视频原子);可训练性预检(真实 pools 驱动,5 微型类被剔);patch 三层校验;SSE 未完成流拒收;瞬时错误清单;缓存盐入键;extract_run_db 去重;seed 携带拷贝;holdout 备忘录 |
|
||||||
|
| 单元(改语义) | early_stop 按 epoch;加载器 fail-loud;prediction 归一化 |
|
||||||
|
| 集成 | fake-LLM 小池 train 冒烟(预检→模板→traces 适配→holdout 去重计数→step DELETE);5 模板契约测试 |
|
||||||
|
| 回归 | 全量 1473 保持绿;agent/LLM 测试产 MD 到 `tests/outputs/` |
|
||||||
|
|
||||||
|
## 12. 训练启动 runbook(修复完成后)
|
||||||
|
|
||||||
|
```
|
||||||
|
1. 四工作包完成 + make test 绿
|
||||||
|
2. 备份 workspaces/video-split/ → 改 val_ratio:0.4 → 重跑 build_video_split.sh(诊断命中缓存秒级)
|
||||||
|
→ 核对: val 错题≥20 / T2 入 diag 数 / 预检模拟的可训类清单
|
||||||
|
3. 建 seed: extract_run_db(infer_adhoc,dedupe) + init_seed('adhoc-baseline', pools_json=…)
|
||||||
|
4. 新增 config/train_videomme.yaml + scripts/train_videomme.sh(零参可复现):
|
||||||
|
epochs=3, early_stop_patience=2(epoch), run_holdout_eval=true(去重版),
|
||||||
|
trainable_min_units=8, 其余 gate/batch 沿用 default.yaml
|
||||||
|
5. tmux: CUDA_VISIBLE_DEVICES=0 bash scripts/train_videomme.sh
|
||||||
|
```
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# 遥测 SQLite 高并发写锁修复设计
|
||||||
|
|
||||||
|
> 2026-07-16。修复 `SQLiteTelemetryRecorder` 在 concurrency≥12 时 `database is locked`、遥测大量丢失的问题。诉求:训练(concurrency 24)时遥测**零丢失**——保住"详细 agent 日志用于回溯分析"这一核心诉求。
|
||||||
|
|
||||||
|
## 1. 根因(对照 HarnessLog 暴露)
|
||||||
|
|
||||||
|
同样高并发写 SQLite,`app/harness/log.py:HarnessLog` 从不锁死,`adapters/telemetry.py:SQLiteTelemetryRecorder` 频繁 locked。差异在**并发控制放在哪一层**:
|
||||||
|
|
||||||
|
| | HarnessLog(不锁死) | SQLiteTelemetryRecorder(锁死) |
|
||||||
|
|--|---------------------|-------------------------------|
|
||||||
|
| 连接 | 单长连接(构造建一次,`check_same_thread=False`,全程复用) | 每次 `_write` 新建连接 + close |
|
||||||
|
| 并发控制 | 进程内 `threading.Lock` 串行化写 | 无 Lock,靠 SQLite `busy_timeout=5000` 跨连接协调 |
|
||||||
|
| 结果 | 进程内始终只有一个连接写 → 零 SQLite 锁竞争 | 12-24 连接并发写同一 db → 高频撑爆 busy_timeout → locked |
|
||||||
|
|
||||||
|
**根因一句话**:telemetry 把并发控制交给 SQLite 的跨连接锁(busy_timeout,高频不可靠),HarnessLog 把它拉到进程内(threading.Lock,可靠串行)。telemetry 已有 WAL + busy_timeout + try/except 降级,但缺"单连接 + 进程内 Lock"这层,故高频下仍锁。
|
||||||
|
|
||||||
|
## 2. 为什么不直接用 HarnessLog(架构溯源)
|
||||||
|
|
||||||
|
telemetry **不能**复用 HarnessLog 实例,三条硬隔离:
|
||||||
|
|
||||||
|
| 维度 | telemetry | HarnessLog |
|
||||||
|
|------|-----------|------------|
|
||||||
|
| DB 文件 | `logs/telemetry.db`(全局、跨 run/workspace) | `workspaces/<ws>/harness.db`(per-workspace 训练数据) |
|
||||||
|
| 分层 | `adapters/`(实现 core `TelemetryRecorder` Protocol) | `app/harness/`(应用层具体类) |
|
||||||
|
| 职责/表 | 每次 LLM 调用 raw I/O 可观测性,`llm_calls` 表 | 训练数据(predictions/traces/gate),无 `llm_calls` |
|
||||||
|
|
||||||
|
依赖方向禁止 `adapters` 依赖 `app/harness` 具体类(Clean Architecture)。telemetry 当初独立实现是**架构正确**的。
|
||||||
|
|
||||||
|
## 3. 决策:对齐同模式,不抽共享基座
|
||||||
|
|
||||||
|
真正的次优点是"单连接+Lock+WAL"这套可靠模式被**重复实现**(HarnessLog 一份、telemetry 一份且写错)。理想是抽共享基座,但受阻:
|
||||||
|
|
||||||
|
- 共享基座放 `adapters/` → app(HarnessLog)不依赖 adapters,用不了。
|
||||||
|
- 放 `core/` → 违反"core 不含 SQLite 具体实现"。
|
||||||
|
- 要抽须新开双方都能依赖的基础设施模块 + 改动刚改过 register_run 的 HarnessLog(回归风险)+ 改动面大。
|
||||||
|
|
||||||
|
**决策(Rejected: 抽共享基座)**:telemetry **对齐** HarnessLog 已验证的连接管理模式,两处加交叉引用注释。接受"同一可靠模式在两处应用"(模式复用,非逻辑重复),换取改动局限、零风险、不动 HarnessLog。共享基座记 future work(若第三处再出现同款 SQLite 写需求,届时抽)。
|
||||||
|
|
||||||
|
## 4. 改动(局限 `adapters/telemetry.py` 一个类)
|
||||||
|
|
||||||
|
- **构造 `__init__`**:建一个长连接 `sqlite3.connect(db_path, check_same_thread=False)`(asyncio.to_thread 在线程池不同线程调用,共享连接需此 flag + Lock 保证串行);`PRAGMA journal_mode=WAL`;建表一次(去掉懒建表 `_ensure_table`/`_table_ready`);建 `threading.Lock`。
|
||||||
|
- **`_write`**:改为 `with self._lock: self._conn.execute(INSERT); self._conn.commit()`(不再新建/关闭连接)。
|
||||||
|
- **保留**:`INSERT OR IGNORE`(call_id 主键幂等)、`try/except sqlite3.Error → 降级 warning`(遥测失败绝不冒泡拖垮 LLM 调用)、`async record_llm_call` 经 `asyncio.to_thread` 卸载阻塞写。
|
||||||
|
- **注释**:`telemetry.py` 与 `log.py` 各加一行交叉引用,标注共用"单连接+Lock+WAL"并发写模式。
|
||||||
|
|
||||||
|
## 5. 前序版本行为审计
|
||||||
|
|
||||||
|
| 现有行为 | 处置 |
|
||||||
|
|---------|------|
|
||||||
|
| `record_llm_call` async Protocol 接口 | **保留**(GovernedLLMClient 依赖签名) |
|
||||||
|
| 每次写新建连接 + close | **替换**(锁竞争根源)→ 单长连接 |
|
||||||
|
| 无进程内 Lock | **新增** threading.Lock |
|
||||||
|
| `asyncio.to_thread` 卸载 | **保留**(Lock 线程内持有,串行化) |
|
||||||
|
| `INSERT OR IGNORE`(幂等) | **保留** |
|
||||||
|
| try/except 降级不冒泡 | **保留**(核心哲学,遥测失败不拖垮主流程) |
|
||||||
|
| 懒建表 | **替换**为构造时建一次(长连接下无需懒建) |
|
||||||
|
| WAL + busy_timeout | 保留 WAL;busy_timeout 可保留(长连接下已无跨连接竞争,作纵深防御) |
|
||||||
|
|
||||||
|
## 6. 非功能四维
|
||||||
|
|
||||||
|
| 维度 | 保障 |
|
||||||
|
|------|------|
|
||||||
|
| **持久化** | 每次 `commit` 同步落 WAL → 零丢失(满足硬约束) |
|
||||||
|
| **幂等** | `INSERT OR IGNORE` + call_id 主键,重复写安全 |
|
||||||
|
| **续跑** | 不适用(遥测无状态;进程退出 WAL 自动恢复已 commit 的) |
|
||||||
|
| **原子性** | 单条 insert+commit 原子,无半写 |
|
||||||
|
|
||||||
|
**零丢失保证的适用范围(Codex 审明确)**:
|
||||||
|
- **单进程、单 recorder 实例**:锁与连接是实例字段,串行化只在同一实例内成立。同进程多个 recorder 指向同一 db 会退回跨连接竞争——当前 `main.py` / `video_split_cli` 均单实例注入,不踩;本实现不支持多实例同库(YAGNI,若未来需要再引 class-level registry)。
|
||||||
|
- **唯一 call_id**:`INSERT OR IGNORE` 下重复 call_id 是**预期忽略**(幂等),不计作丢失。
|
||||||
|
|
||||||
|
**降级边界(Codex 审加固)**:`__init__` 的 mkdir / connect / PRAGMA / 建表统一纳入 `except (OSError, sqlite3.Error)` 降级(`self._conn=None`),任一失败都不冒泡拖垮初始化;`_write` 遇 `self._conn is None` 或 execute 抛错均降级 warning。守住"遥测失败绝不拖垮 LLM 调用"哲学。
|
||||||
|
|
||||||
|
**生命周期**:补幂等 `close()`(对齐 HarnessLog)供进程退出前可选调释放 fd;不调也不丢数据(WAL 已 commit)。telemetry 是长生命周期单例,无 context-manager 场景,故 close 为可选而非强制。
|
||||||
|
|
||||||
|
## 7. 测试
|
||||||
|
|
||||||
|
- **并发写不锁死**(核心):多线程/多协程并发调 `record_llm_call`(如 32 并发 × N 条),断言全部落库、零 `database is locked`、零丢失(行数 == 写入数)。这是复现 bug 的真实场景测试。
|
||||||
|
- **幂等**:同 call_id 重复写,只 1 行。
|
||||||
|
- **降级不冒泡**:DB 错误(如目录不可写)时 `record_llm_call` 不抛,只 warning。
|
||||||
|
- 现有 `test_telemetry.py` 回归全绿。
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:fix-diagnosis-tree-data-link
|
||||||
|
title: "修复诊断 tree_data 断链 bug(TRM4→TRM5 迁移 regression)"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 修复诊断 tree_data 断链 bug(TRM4→TRM5 迁移 regression)
|
||||||
|
|
||||||
|
> 诊断编排全程传 `tree_data={}`,导致 ground_truth 恒空、error_type 归因 100% 坍缩成 extraction_failure。本设计接通断掉的树加载环,补齐 P5 fail-loud,双注入点(离线诊断 + 训练循环)一并修复。
|
||||||
|
>
|
||||||
|
> **状态**:已含 Codex 独立审查修订(2 Critical + 8 Important 全部核验并采纳;展平实现从"复用 to_dict"改为"遍历 json",level 改由遍历深度赋值,补视频覆盖 fail-loud,重冻 test 成员变化诚实化)。
|
||||||
|
|
||||||
|
## 1. 根因(已坐实)
|
||||||
|
|
||||||
|
| 环节 | 位置 | 事实 |
|
||||||
|
|------|------|------|
|
||||||
|
| CLI 注入空树 | `app/harness/video_split_cli.py:335` | `tree_data={}`,注释谎称"由诊断管线内部按需加载" |
|
||||||
|
| 训练循环同病 | `app/harness/runner.py:2180` | `tree_data={}` + 同一句假注释 |
|
||||||
|
| 编排层无加载 | `core/evolution/diagnose.py:2068-2074` | `if tree_data and "nodes"...` → `{}` 为假 → else 分支 `tree_data_by_video={}`,**根本无加载逻辑** |
|
||||||
|
| ground_truth 恒空 | `diagnose.py:736,755` | `{}.get("nodes",{})` → node card 取空 → `""` |
|
||||||
|
| span 评估无参照 | `diagnose.py:509-515` | judge 看不到"应提取什么",`extraction_completeness` 系统性偏低,解析失败还默认 0.0 |
|
||||||
|
| 归因瀑布短路 | `diagnose.py:928` | `avg_completeness < 0.5` 恒真 → **100% extraction_failure** |
|
||||||
|
|
||||||
|
实测:82 个 T2 全部 extraction_failure,evolution_target 全 tool,48 格多样性退化成 11 格(error_type 维坍缩成单值)。`judge_missed_nodes` 的 `tree_content` 同样为空,search_failure 分支永不触发。
|
||||||
|
|
||||||
|
## 2. Regression 溯源:TRM4 无此 bug
|
||||||
|
|
||||||
|
| 层面 | TRM4(正常) | TRM5(坏了) |
|
||||||
|
|------|------|------|
|
||||||
|
| tree.json 结构 | 扁平 `{"nodes": {id: {node_id,level,time_range,parent_id,children_ids,card}}}` | 嵌套 `{"metadata", "roots": [{id,card,time_range,children}]}` |
|
||||||
|
| tree_cache 填充 | `diagnose.py:1677` `_load_json(tree.json)` 直接得 nodes | `tree_data={}`,填充逻辑被删 |
|
||||||
|
| 诊断拿 ground_truth | ✅ `_load_json(...)["nodes"]` 直接可用 | ❌ 恒空 |
|
||||||
|
|
||||||
|
**双重 regression**:(a) 建树模块重写把产物格式从扁平 nodes 演进成嵌套 roots(合理架构升级);(b) 诊断代码迁移时既未适配新格式、又把加载逻辑删成空 dict。因此 TRM5 需要一个 TRM4 不需要的**格式桥接展平器**(roots → nodes)。
|
||||||
|
|
||||||
|
## 3. 影响范围
|
||||||
|
|
||||||
|
| 维度 | 是否污染 | 原因 |
|
||||||
|
|------|---------|------|
|
||||||
|
| tier 分层 T0/T1/T2/uncertain | ✅ 干净 | `classify_defect_vs_lapse` **不吃 tree_data** |
|
||||||
|
| floor_k / test 代表性 | ✅ 干净 | 按 task_type 总数,与 error_type 无关 |
|
||||||
|
| error_type 归因 | ❌ 全坍缩 | ground_truth 恒空 |
|
||||||
|
| evolution_target 路由 | ❌ 全 tool | error_type 的下游派生 |
|
||||||
|
| 48 格多样性覆盖 | ⚠️ 退化 | error_type 维失效 → 实际仅按 task_type 单维 |
|
||||||
|
| `judge_missed_nodes` | ⚠️ 退化 | tree_content 空 |
|
||||||
|
|
||||||
|
结论:已冻结 `pools.json` 的 tier/floor/代表性**可信**,仅 error_type 维及其下游被污染。
|
||||||
|
|
||||||
|
## 4. 修复设计
|
||||||
|
|
||||||
|
### 4.1 单元拆分(单一职责)
|
||||||
|
|
||||||
|
| 单元 | 位置 | 职责 | 依赖 |
|
||||||
|
|------|------|------|------|
|
||||||
|
| 树展平器 | `app/harness/tree_nodes.py`(新) | `load_tree_nodes(store_dir, video_id) -> dict`:**递归遍历 tree.json 的 `roots`(嵌套 dict)**,按遍历深度赋 `level`(root=1/child=2/孙=3),抽取每节点 `id/card/time_range`,组装成扁平 `{"nodes": {node_id: {card, level, time_range}}}` | 只读文件 + json(**不走 TreeIndex 对象层**,见 §4.2 核验) |
|
||||||
|
| 离线注入 | `app/harness/video_split_cli.py`(`build_diagnosis_deps`/`run_pipeline`) | 由 `wrong_ids` → 涉及 video 去重加载成 `{video_id: {"nodes":...}}`,填充 `DiagnosisDeps.tree_data`(`baseline_diagnosis` 仅透传) | ← 展平器 |
|
||||||
|
| 训练注入 | `app/harness/runner.py`(`_run_diagnosis`) | 由 batch `question_ids` → 涉及 video 同法加载注入,删 `tree_data={}` 假注释 | ← 展平器 |
|
||||||
|
|
||||||
|
### 4.2 关键决策
|
||||||
|
|
||||||
|
| 边界 | 决策 | 理由 |
|
||||||
|
|------|------|------|
|
||||||
|
| 加载职责 | **app 层**(video_split_cli / runner),core 只消费 dict | Clean Architecture:core 不碰文件系统;`run_diagnosis` else 分支已支持 `{video_id:...}` 形态,core 零改动 |
|
||||||
|
| 展平实现 | **递归遍历 tree.json `roots` dict**(原始 json,card 已是 dict 直接取),不走对象层 | 核验:仅 `L1Node` 有 `to_dict`(`index.py:260`),L2/L3 是其内部闭包;且 to_dict 输出无 `level`、L3 无 `time_range`(用 `timestamp`)。遍历 json 更省且零改建树模块 |
|
||||||
|
| level 字段 | **由遍历深度直接赋值**(root=1/child=2/孙=3),不解析 node_id | node_id 累积式 `..._L1_..._L2_..._L3_`,正则首匹配会把 L2/L3 误判成 1(Codex I7) |
|
||||||
|
| 预加载范围 | 只加载被诊断题涉及的 video,按 video 缓存去重 | YAGNI;236 错题涉及 <200 视频,避免加载无关树 |
|
||||||
|
| fail-loud | tree.json 缺失 → `FileNotFoundError`(沿用 `factory.py:87` 先例);展平后 nodes 为空 → `ValueError`;**本次诊断涉及的每个 video 必须被 tree_data 覆盖**,`run_diagnosis` 的 `.get(vid, {})` 静默回退(`diagnose.py:2144-2145`)改为缺失即 raise | 补 P5:空树/漏加载本应报错,不再静默退化 |
|
||||||
|
| 诊断瀑布本身 | **不动**(algo §4.7 #7) | 修复只接通输入,不改归因逻辑 |
|
||||||
|
|
||||||
|
> **L3 无 `time_range` 的已知次要行为**:`node.to_dict` 语义下 L3 节点用 `timestamp`;`_load_tree_content` 取 `time_range` 时 L3 退化为默认 `[0,0]`,仅影响 `judge_missed_nodes` 文本里 L3 的时间显示,不影响 ground_truth(card)。展平器可选:从 L3 `timestamp` 合成 `[timestamp, timestamp]`。
|
||||||
|
|
||||||
|
## 5. 非功能性四维
|
||||||
|
|
||||||
|
| 维度 | 结论 |
|
||||||
|
|------|------|
|
||||||
|
| 持久化 | 展平器纯内存只读,不落盘;诊断结果仍逐题 upsert(不变) |
|
||||||
|
| 幂等性 | 展平确定性(同 tree.json → 同 nodes);诊断 upsert 幂等(不变) |
|
||||||
|
| 断点续跑 | `done_question_ids` 续跑机制不变;fingerprint 因代码改动而变 → 全量重跑一次 |
|
||||||
|
| 原子性 | 无新写操作;pools.json 重冻仍走既有原子写 |
|
||||||
|
|
||||||
|
## 6. 重跑与重冻衔接
|
||||||
|
|
||||||
|
代码改动 → git short SHA 变 → `diag_fingerprint` 变 → 全量重跑 236 题诊断。
|
||||||
|
|
||||||
|
- `classify_defect_vs_lapse` / evidence / bias / skill judge 输入不含 tree_data → **预期 Redis 缓存命中,tier 分层 T2/T1 应不变**。但这是"大概率"而非"必然":缓存键内容未逐字段核证,且存在非 tree_data 的不稳定源(如 C3 异常吞并 attribution,`diagnose.py:2186`)。**重跑后须实测校验 T2=82/T1=152 是否保持**,偏差需归因。
|
||||||
|
- `evaluate_span`(ground_truth 由空变有)+ `judge_missed_nodes`(tree_content 由空变有)缓存失效 → 重算(约半数 LLM 调用,~1h)。
|
||||||
|
- 诊断完成后重跑 Phase 2 重新冻结覆盖 `pools.json`:**error_type 恢复判别力会改变多样性阶段的选择顺序**(cells 按 `(task_type, error_type)` 排序,`split_selection.py:139/193`)→ trainval 组成变 → **test 补集成员随之变化,不保证稳定**。floor 与代表性 ε 约束仍保证 test **代表性合格**,但具体成员会变——用户已接受重冻覆盖,这是预期结果而非风险。
|
||||||
|
|
||||||
|
## 7. 测试策略
|
||||||
|
|
||||||
|
| 测试 | 类型 | 验证点 |
|
||||||
|
|------|------|--------|
|
||||||
|
| 展平器正确性 | unit | 真实 tree.json → `{"nodes":{...}}`,节点数=递归总数,node 含 card(dict)/level/time_range |
|
||||||
|
| 展平器 level 赋值 | unit | 遍历深度 → level 1/2/3,与节点嵌套层级一致(不依赖 node_id 解析) |
|
||||||
|
| 展平器 fail-loud | unit | tree.json 缺失 → FileNotFoundError;空树/空 nodes → ValueError |
|
||||||
|
| 视频覆盖校验 | unit | 诊断涉及的 video 未被 tree_data 覆盖 → raise(不静默回退) |
|
||||||
|
| ground_truth 接通 | integration | 对真实 T2 样本注入真实树,断言 `evaluate_span` 收到**非空 ground_truth**(充分条件);**观察** error_type 恢复多值(软验收——分布是否脱离单值取决于 judge,非硬保证) |
|
||||||
|
| core 依赖方向 | 结构检查 | `core/evolution/diagnose.py` 不 import `app.tree` |
|
||||||
|
|
||||||
|
## 8. 算法保真声明
|
||||||
|
|
||||||
|
本修复触及核心算法 **§4.7 #7 诊断瀑布**:仅接通其输入(tree_data),**不改** `attribute_error` 归因分支、severity 函数或 defect/lapse 判定逻辑。参考 TRM4 `core/harness/diagnose.py` 的 tree_cache 语义对照,确保 TRM5 诊断拿到与 TRM4 一致的 ground_truth。
|
||||||
|
|
||||||
|
## 9. 被拒方案
|
||||||
|
|
||||||
|
| 方案 | 拒因 |
|
||||||
|
|------|------|
|
||||||
|
| 复用 `TreeIndex.load_json` + `node.to_dict` 走对象层 | 核验否决:仅 `L1Node` 有 `to_dict`(`index.py:260`),L2/L3 为其内部闭包;输出无 `level`、L3 无 `time_range`,补了也要后处理 |
|
||||||
|
| 补 L2/L3Node.to_dict 再走对象层 | 改核心建树模块 `index.py`(推理/save_json/建树全链路依赖,algo §4.7 #1-3);为诊断读取需求反向改生产方接口,违反 YAGNI |
|
||||||
|
| 让 core 诊断直接吃 `TreeEnvironment` 对象 | 破坏依赖方向(core 依赖 app.tree);且需构建 env(要 frames_dir)过重 |
|
||||||
|
| 保留现有 pools.json 不重冻 | train 多样性仍基于污染的 error_type,违背修复初衷 |
|
||||||
|
| 只修离线诊断、不修 runner | 训练循环 backward 每轮复现坍缩,进化盲目只改 tool |
|
||||||
|
|
||||||
|
## 关联
|
||||||
|
- 影响指标:`metric:split-cell-coverage`(48 格覆盖)、`metric:split-signal-tier-distribution`
|
||||||
|
- 数据表:`schema:baseline-diagnosis`
|
||||||
|
- 上游设计:`design:results-driven-video-split`
|
||||||
|
- 参考实现:TRM4 `core/harness/diagnose.py:1677`(tree_cache 填充)
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:gate-speedup
|
||||||
|
title: "gate 验证提速:预灌 BaselineCache + 双臂并行"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# gate 验证提速:预灌 BaselineCache + 双臂并行
|
||||||
|
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:preflight-fixes
|
||||||
|
title: 训练前缺陷修复设计
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# 训练前缺陷修复设计
|
||||||
|
|
||||||
|
正文见 [`2026-07-16-preflight-fixes-design.md`](./2026-07-16-preflight-fixes-design.md)。
|
||||||
|
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:results-driven-video-split
|
||||||
|
title: "结果驱动的视频级 train/val/test 切分(替代自造题模块)"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 结果驱动的视频级 train/val/test 切分(替代自造题模块)
|
||||||
|
|
||||||
|
**全文设计**:[2026-07-15-results-driven-video-split-design.md](./2026-07-15-results-driven-video-split-design.md)
|
||||||
|
|
||||||
|
## 决策与理由
|
||||||
|
|
||||||
|
- **砍自造题**:14 天内造 Video-MME 同质量题风险过高;DataLoader 本就只加载+切分、不合成,故事更忠实。
|
||||||
|
- **方案 A(内部切分)**:300 视频按**视频原子**切 ~100 train+val / 200 test,零内容泄漏。
|
||||||
|
- **核心洞察(耦合)**:`train错题+test错题=240` 固定 → 富集 train 会注水 test;故 **test 神圣代表性冻结**,train 只靠 floor + 多样性富集(不偷 test 难题),ε 约束是耦合的算法化身。
|
||||||
|
- **signal_score**:`DiagnosisResult` 的确定性投影,T0 INFRA / T1 lapse(混无解,接受) / T2 defect(训练主体)。
|
||||||
|
- **多样性**:对 `(task_type × error_type)` 48 格的 submodular 最大覆盖(evolution_target 为 error_type 派生标注,非独立轴),非错题数。
|
||||||
|
- **选择算法**:贪心联合约束 max-coverage,floor 硬约束先满足、多样性目标后最大化、不可行 fail loud。
|
||||||
|
|
||||||
|
## 被替代
|
||||||
|
- 自造题 v3:[2026-07-15-question-gen-v3-construction-paradigm-design.md](./2026-07-15-question-gen-v3-construction-paradigm-design.md)
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:telemetry-concurrency-fix
|
||||||
|
title: "遥测 SQLite 高并发写锁修复"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# 遥测 SQLite 高并发写锁修复
|
||||||
|
|
||||||
+84
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
id: question-gen-paradigm-shift-construction-over-filtering
|
||||||
|
title: 出题范式转变 — 从"生成-打分-过滤"转向"构造优先 + 两正交独立信号"(六篇原文深读)
|
||||||
|
type: finding
|
||||||
|
created: 2026-07-15
|
||||||
|
status: active
|
||||||
|
supersedes-frame-of: 2026-07-15-question-gen-v2-grounded-contrastive-design.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# 出题范式转变:构造优先,而非生成-过滤
|
||||||
|
|
||||||
|
**触发**:v2 设计 + Codex 修法经两轮对抗审核后,判定撞上两堵"结构性墙"(2 VLM 凑不出独立 grounding 裁判;所有门合取减产不可行区)。用户指出这是"工程补丁/表面模仿",要求回原文挖核心可行机制。遂对 6 篇论文做原文深读(非二手摘要)。
|
||||||
|
|
||||||
|
## 一、最关键的发现:两堵"墙"是错误框架的产物
|
||||||
|
|
||||||
|
v2 设计(含 Phase A、Codex 修法)自始至终是**"自由生成候选池 → 用裁判打分 → 过滤挑好的"**。深读证明这是 **AFLite 亲口认栽的死路**:后置过滤的唯一信号(predictability)同时混入"真难题"和"歧义/烂题",一把尺量两样东西,拆不开——SNLI 过滤后**人类精度反降 10 分**(Table 3),证明筛出的"难题"塞满连人都做不对的歧义题。
|
||||||
|
|
||||||
|
两堵墙只在这个框架里存在:
|
||||||
|
- **墙 1(凑不出独立 grounding 裁判)** 只在"事后打分 grounding"时成立。构造范式里 grounding 由"锚定真实事实的最小编辑"保证,**根本不需要 grounding 裁判**。
|
||||||
|
- **墙 2(合取门减产不可行区)** 只在"生成垃圾再硬过滤"时成立。构造正确 by design,yield 天然高,不需要一摞过滤门相乘。
|
||||||
|
|
||||||
|
## 二、两种可行范式(按子模式可构造性分流)
|
||||||
|
|
||||||
|
| 范式 | 代表论文 | grounding 由什么保证 | wrongness/唯一正解由什么保证 | 适用子模式 |
|
||||||
|
|------|---------|---------------------|---------------------------|-----------|
|
||||||
|
| **A. 受控构造** | Vinoground / TempCompass / VITATECS | 对真实时序/结构事实做**最小编辑** / **冲突孪生对**,锚定真实事实 → 结构保证 | 单轴取值改变 / 孪生体互斥 → **结构保证** | 可锚定到结构化事件时序属性者(顺序/方向/次数/主宾/时序性位置) |
|
||||||
|
| **B. 生成-独立筛选** | GroundAttack | 独立模型对**真实帧的几何对齐度量**(把干扰项视觉对齐度顶到正解水平,抹掉 EOB) | **无保证——它的已知软肋**(§5 自认未验 wrongness);须外补独立核验 | 构造不了者(细粒度动作方式/纯感知) |
|
||||||
|
|
||||||
|
**判定证据(构造 vs 过滤)**:
|
||||||
|
- Vinoground:caption 侧受控构造(GPT-4 硬约束"exact same words, only permuted"),video 侧检索+人工核验。非生成-过滤。
|
||||||
|
- TempCompass:纯受控构造(倒放/拼接/换序造冲突孪生),无生成后过滤,质量靠人工逐条校订。
|
||||||
|
- VITATECS:受控生成 + precision-over-recall 三级严过滤兜底(NLI + 微调判别器"正反双序一致"+ 人工改标)。
|
||||||
|
- GroundAttack:过量生成 128 + CLIP 视觉对齐筛 4。**唯一的生成-筛选范式**。
|
||||||
|
|
||||||
|
## 三、三根之前完全没有的新支柱(原文证据)
|
||||||
|
|
||||||
|
### 支柱 1:冲突孪生对 + 去捷径坍缩度量(TempCompass)
|
||||||
|
"A 的干扰项 = B(孪生体)的正确答案"——静态帧完全相同、只翻转时序,**诱导 A 答对的捷径必让 B 答错**,单帧偏置+语言先验被结构性对冲归零。**有效性证明不是原始准确率,而是"加入冲突后掉回随机基线的幅度"(配对准确率/一致性)**——纯算法/结构验证,不需循环裁判。Table 5 铁证:Image LLM 41-49%→35-41%(逼近随机 30)。
|
||||||
|
|
||||||
|
### 支柱 2:难度 vs 歧义 = 两个永不合并的正交独立信号(AdVQA)
|
||||||
|
- 难度信号 = 活求解器答错(对抗前移出题当下,骗不过就重写,5-tries 循环)。
|
||||||
|
- 质量/歧义信号 = **独立于求解器的多裁判答案一致性**(10 标注者,<6 confident 且无共识→剔)。
|
||||||
|
- **核心不变量**:答案核验器**必须独立于被攻求解器**,用多裁判一致性证伪答案唯一性。把"作弊门"做成"求解器既当难度裁判又当答案裁判"= 退回 AFLite 单信号困境。
|
||||||
|
- §4.5 直接证据:自动对抗无独立核验会制造答案漂移/歧义(Textfooler 错率 1.4% vs 人类闭环 38.1%)。
|
||||||
|
|
||||||
|
### 支柱 3:grounding 靠构造锚定真实事实(Vinoground/VITATECS/GroundAttack 共识)
|
||||||
|
"grounded"不来自"用了某模型打分",而来自:(A) 构造时锚定真实时序事实的最小编辑,或 (B) 独立模型对真实视觉证据的对齐度量。**表面模仿"加个打分器"会漏掉:打分对象必须是"与真实视觉 V 的对齐度",且打分器必须独立且真的看帧;且打分器不顺带保证 wrongness(GroundAttack 软肋)**。
|
||||||
|
|
||||||
|
## 四、原文的关键警告(防止再次表面模仿)
|
||||||
|
|
||||||
|
| 论文 | 警告 | 对我们的意义 |
|
||||||
|
|------|------|------------|
|
||||||
|
| VITATECS Table 8 | 随机改 1 个名词→74.5-82.1% 可解;改词数 1→3→90.3% 可解;"cautions against purely lexical methods" | 反事实必须沿**正交时序轴**做等信息量、非蕴含的**最小编辑**;名词/属性替换必退化成静态捷径可解的伪难题 |
|
||||||
|
| VITATECS §3 | 纯静态感知(VISUAL/OCR/属性)**没有时序轴可翻转**,硬套单轴反事实会自败 | 不是所有 AR 子模式都适合构造范式;纯感知类需另想办法或剔除 |
|
||||||
|
| GroundAttack §5 | selector 只优化 grounding+多样性,**wrongness 从未独立验证**;CLIP 选出的"视觉最贴合"候选恰最可能是第二正解 | 生成-筛选范式必须**外挂独立 wrongness 核验**(对齐 AdVQA) |
|
||||||
|
| AFLite p8 | 过滤误删真易题 + 留下歧义题(人类精度降 10 分) | 后置作弊门只能兜底,不能当主力;质量必须构造/生成端保证 |
|
||||||
|
| TempCompass Table 5 | 沿用原始准确率会自欺 | 度量必须用配对/一致性/去捷径坍缩,而非单题准确率 |
|
||||||
|
|
||||||
|
## 五、对 6 个 AR 子模式的范式分流(初判,待 brainstorming 细化)
|
||||||
|
|
||||||
|
| 子模式 | 范式 | 构造手段 | 备注 |
|
||||||
|
|--------|------|---------|------|
|
||||||
|
| temporal_reasoning_failure | **A 构造** | 树事件序列换序 → 冲突孪生对 + 双向 AND | 最佳适配(Vinoground/TempCompass Event-Order) |
|
||||||
|
| cross_segment_entity_tracking | **A 构造** | 主宾互换 / first-last 翻转,保词袋 | Compositionality 最小编辑 |
|
||||||
|
| fine_grained_visual_action | **B 生成-筛选** | qwen 过量生成 + MiniMax 看帧筛视觉对齐 + 独立 wrongness 核验 | 构造不了动作方式;2-VLM 独立性问题局限在此一个子模式 |
|
||||||
|
| premature_evidence_anchoring | **A 构造(改造)** | 段位随机 + 跨段唯一性;本质是"时序性锚点"单轴 | 需确保锚定时序而非静态 |
|
||||||
|
| evidence_gap_confabulation | **待定** | 二元(诚实 vs 虚构因果链),非静态感知 | 可能需独立设计;不套单轴反事实 |
|
||||||
|
| semantic_rigidity | **待定/可能剔除** | 唯一同义改写 vs 字幕陷阱——偏静态语义 | VITATECS 警告静态类不适合;重新评估是否保留 |
|
||||||
|
|
||||||
|
## 六、范式转变对 v2 设计的意义
|
||||||
|
|
||||||
|
**现 v2 设计(2026-07-15-question-gen-v2-grounded-contrastive-design.md)的整个框架(生成-打分-过滤 + 事后 grounding 裁判 + 一摞合取门)被本 finding 取代。** 新框架:
|
||||||
|
|
||||||
|
1. **构造优先**:能锚定到树的结构化事实的子模式,走"最小编辑 + 冲突孪生对",grounding/wrongness/唯一性由构造保证,绕开两堵墙。
|
||||||
|
2. **生成-独立筛选仅用于构造不了的子模式**(fine_grained),且必须解耦 grounding(看帧对齐)与 wrongness(独立核验)两道关卡。
|
||||||
|
3. **难度/歧义两正交独立信号**:难度=活求解器 Agent 答错(对抗前移,复用现有 AgentLoop);歧义=独立于求解器的多裁判/构造式答案唯一性核验。
|
||||||
|
4. **度量革命**:验收用配对准确率/去捷径坍缩(text-only 盲答、单帧基线掉回随机),纯算法,非循环裁判。
|
||||||
|
5. **LLM 角色收缩**:从"自由发明干扰项"变为"对结构化树事实做受控最小编辑的表面实现 + 结构化属性抽取"。
|
||||||
|
|
||||||
|
## 相关
|
||||||
|
- 论文原文:`reference/papers-distractor-gen/`(6 篇深读证据页码见各专读报告)
|
||||||
|
- 被取代框架:[2026-07-15-question-gen-v2-grounded-contrastive-design.md]
|
||||||
|
- 前序审核:[2026-07-15-question-gen-v2-adversarial-audit.md]
|
||||||
|
- 前序诊断:[2026-07-15-question-gen-v2-diagnosis-and-strategy.md]
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
id: v3-frame-perception-spike-validation
|
||||||
|
title: v3 §9 帧感知抽取机制 — 小样本实证验证结果
|
||||||
|
type: finding
|
||||||
|
created: 2026-07-15
|
||||||
|
status: active
|
||||||
|
---
|
||||||
|
|
||||||
|
# v3 §9 帧感知抽取机制:小样本实证验证
|
||||||
|
|
||||||
|
**目的**:进 writing-plans 前用真实数据验证 §9"帧感知 grounded 事实抽取 + negative_at_target + 双 VLM 交叉"能否成立(对治 Codex 三审 C3 密度 / C5 双正解,避免 v2 盲飞)。
|
||||||
|
|
||||||
|
**方法**:5 个真实视频 × 3 个 L2 段,双 VLM(qwen3.6-plus + MiniMax-M3)对真实关键帧(5 帧/段)做结构化事实抽取 + 交叉核实 + 跨段 negative_at_target 检验,94 次 VLM 调用。脚本 `scratchpad/spike_frame_perception.py`(一次性)。**Claude 亲自 Read 帧图片当第三方裁判估 shared-error。**
|
||||||
|
|
||||||
|
## 一、量化结果
|
||||||
|
|
||||||
|
| 指标 | 结果 | 对照 §9.1 假设 |
|
||||||
|
|------|------|--------------|
|
||||||
|
| 可抽取率 | **15/15 = 100%** | 5 帧能感知结构化绑定 ✓ |
|
||||||
|
| 判别性占比 | **~83%**(跨段 negative_at_target 持续=17%,5/30) | 判别性事实是可用主体 ✓ |
|
||||||
|
| negative_at_target 有效性 | **有效**:被标"其它段也成立"的 yes **全是真持续事实**(魔方演示跨段真、舞台剧跨段真、performer 站台上跨段真),且 Claude 看帧确认 | C5 双正解可被抓 ✓ |
|
||||||
|
| 双 VLM 一致率(holds 判定) | **26/30 = 87%** | 交叉核实高一致 |
|
||||||
|
| 粗时序(顺序)可感知性 | **可以**:两 VLM 频繁从 5 帧感知有序变化("Frame0…Frame1…"/"transitions from…to…"),Claude 看帧确认"倒立→落地"序列真实 | C3 密度:粗时序 5 帧够 ✓ |
|
||||||
|
| 细粒度方式(manner) | **确认是难点**:大量 `not_determinable` + 两 VLM 各说各话 | C3/C1:fine manner 5 帧不够 ⚠ |
|
||||||
|
|
||||||
|
## 二、Claude 第三方看帧核查(shared-error)
|
||||||
|
|
||||||
|
亲自 Read GsYi 3 帧核查:
|
||||||
|
- seg0 frame0:确认男表演者桌上**倒立**、smartwater 舞台、旋转木马、观众——两 VLM 粗层感知**正确且一致,无幻觉**。
|
||||||
|
- seg0 frame2:确认已**站地面**(桌空)——m3 的"倒立→落地"时序 claim **真实非幻觉**;qwen "manner 不可定"是保守。两者非"一致地错",是"保守 vs 详细"。
|
||||||
|
- seg6 frame0:确认 performer **站平台、smartwater+旋转木马背景、女助手**——negative_at_target 判该事实在 seg6 成立**正确**,正是该剔的双正解。
|
||||||
|
|
||||||
|
**结论:抽样中未见双 VLM 一致地幻觉;粗层事实、粗时序、negative_at_target 判定均锚在真实帧、经人工确认正确。**
|
||||||
|
|
||||||
|
## 三、验证结论
|
||||||
|
|
||||||
|
**§9 帧感知抽取机制在真实数据上站得住。** 两个 Codex Critical 的处置:
|
||||||
|
- **C5 双正解 = 已缓解**:negative_at_target 实证有效(正确抓出跨段持续事实),剔除率 ~17% 可控,判别性主体 ~83% 可用。
|
||||||
|
- **C3 密度 = 部分确认**:粗时序/顺序 5 帧可感知(利好 temporal/cross_segment/premature);**细粒度 manner 5 帧确实不够**(只影响 fine_grained,已有密帧/fallback 设计)。
|
||||||
|
|
||||||
|
## 四、须折进设计的实证细化
|
||||||
|
|
||||||
|
1. **fact_sampler 必须过滤到动作丰富段**:5 视频里 MYxL(幻灯片)/y2kg(录屏) 非动作视频——抽取虽成功但对 AR 无意义。fact_sampler 加"动作性"前置筛。
|
||||||
|
2. **fine_grained 密帧从"可选兜底"升为"硬前置"**:manner 5 帧不够是实证事实 → fine_grained 必须密帧重采或走 B 路线,不是运行时才发现。
|
||||||
|
3. **manner 层不能靠双 VLM 交叉核**(一致性低)→ manner 类 fact 走更严核实(更多帧+人工小样本),对齐 §9.1 C1 难度分层。
|
||||||
|
4. **多主体绑定未被本 spike 压测**(样本多为单主体)——繁忙场景的唯一绑定(Codex C4)仍是开放风险,writing-plans 或实现早期补一次多主体段实测。
|
||||||
|
5. 抽取器**倾向用详细但准确的 VLM**(m3 风格)作主抽、保守 VLM(qwen)作交叉。
|
||||||
|
|
||||||
|
## 相关
|
||||||
|
- 设计: [2026-07-15-question-gen-v3-construction-paradigm-design.md](§9/§9.1)
|
||||||
|
- 脚本/原始数据: `scratchpad/spike_frame_perception.py`、`scratchpad/spike_results.json`、遥测 `logs/spike_vlm_telemetry.db`
|
||||||
@@ -190,6 +190,106 @@
|
|||||||
"id": "plan:adversarial-question-gen-phaseB",
|
"id": "plan:adversarial-question-gen-phaseB",
|
||||||
"label": "Adversarial Question-Gen Phase B",
|
"label": "Adversarial Question-Gen Phase B",
|
||||||
"type": "plan"
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:question-gen-v3-phase1-contract",
|
||||||
|
"label": "question-gen v3 Phase1 契约地基实现计划",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:results-driven-video-split",
|
||||||
|
"label": "结果驱动的视频级 train/val/test 切分(替代自造题模块)",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:results-driven-video-split-plan",
|
||||||
|
"label": "结果驱动的视频级切分实现计划",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "schema:baseline-diagnosis",
|
||||||
|
"label": "表结构: baseline_diagnosis(逐题诊断信号)",
|
||||||
|
"type": "schema"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "schema:split-manifest",
|
||||||
|
"label": "表结构: split_manifest(切分冻结溯源)",
|
||||||
|
"type": "schema"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "metric:split-signal-tier-distribution",
|
||||||
|
"label": "指标: 诊断信号 tier 分布(T0/T1/T2/uncertain 占比)",
|
||||||
|
"type": "metric"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "metric:split-cell-coverage",
|
||||||
|
"label": "指标: 48格(task_type×error_type)多样性覆盖率",
|
||||||
|
"type": "metric"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "metric:split-floor-and-representativeness",
|
||||||
|
"label": "指标: floor 达标率 + test 代表性偏差",
|
||||||
|
"type": "metric"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:fix-diagnosis-tree-data-link",
|
||||||
|
"label": "修复诊断 tree_data 断链 bug(TRM4→TRM5 迁移 regression)",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:fix-diagnosis-tree-data-link-plan",
|
||||||
|
"label": "实现计划: 修复诊断 tree_data 断链 bug",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:preflight-fixes",
|
||||||
|
"label": "训练前缺陷修复设计",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:preflight-wp1-asset-migration",
|
||||||
|
"label": "WP1 资产迁移",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:preflight-wp2-split-wiring",
|
||||||
|
"label": "WP2 切分与接线",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:preflight-wp3-train-loop",
|
||||||
|
"label": "WP3 训练循环与进化引擎",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:preflight-wp4-resilience",
|
||||||
|
"label": "WP4 韧性与持久化",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "review:preflight-final-review",
|
||||||
|
"label": "训练前修复分支终审",
|
||||||
|
"type": "review"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:telemetry-concurrency-fix",
|
||||||
|
"label": "遥测 SQLite 高并发写锁修复",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "design:gate-speedup",
|
||||||
|
"label": "gate 验证提速:预灌 BaselineCache + 双臂并行",
|
||||||
|
"type": "design"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "plan:gate-speedup",
|
||||||
|
"label": "连续并发 gate + Redis 复用实现计划",
|
||||||
|
"type": "plan"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "review:gate-speedup-final",
|
||||||
|
"label": "连续并发 gate 终审与交付",
|
||||||
|
"type": "review"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"links": [
|
"links": [
|
||||||
@@ -360,6 +460,125 @@
|
|||||||
"relation": "implements",
|
"relation": "implements",
|
||||||
"evidence": "Phase B 实现计划",
|
"evidence": "Phase B 实现计划",
|
||||||
"added": "2026-07-14T21:09:48.115521+00:00"
|
"added": "2026-07-14T21:09:48.115521+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "schema:v3-question-gen-logging",
|
||||||
|
"target": "design:question-gen-v3-construction-paradigm",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "v3 出题运行时数据 schema 实现设计 §12.1",
|
||||||
|
"added": "2026-07-15T09:18:41.713855+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:question-gen-v3-phase1-contract",
|
||||||
|
"target": "design:question-gen-v3-construction-paradigm",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "Phase1 契约地基",
|
||||||
|
"added": "2026-07-15T09:22:08.800078+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:results-driven-video-split-plan",
|
||||||
|
"target": "design:results-driven-video-split",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "实现结果驱动视频级切分设计",
|
||||||
|
"added": "2026-07-15T15:27:39.569247+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "schema:baseline-diagnosis",
|
||||||
|
"target": "design:results-driven-video-split",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "逐题诊断信号表",
|
||||||
|
"added": "2026-07-15T15:42:38.107624+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "schema:split-manifest",
|
||||||
|
"target": "design:results-driven-video-split",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "切分冻结溯源",
|
||||||
|
"added": "2026-07-15T15:42:38.147922+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "metric:split-signal-tier-distribution",
|
||||||
|
"target": "schema:baseline-diagnosis",
|
||||||
|
"relation": "measures",
|
||||||
|
"evidence": "tier 分布",
|
||||||
|
"added": "2026-07-15T15:42:38.187336+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "metric:split-cell-coverage",
|
||||||
|
"target": "schema:baseline-diagnosis",
|
||||||
|
"relation": "measures",
|
||||||
|
"evidence": "48格覆盖",
|
||||||
|
"added": "2026-07-15T15:42:38.228857+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "metric:split-floor-and-representativeness",
|
||||||
|
"target": "schema:split-manifest",
|
||||||
|
"relation": "measures",
|
||||||
|
"evidence": "floor+代表性",
|
||||||
|
"added": "2026-07-15T15:42:38.269455+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "design:fix-diagnosis-tree-data-link",
|
||||||
|
"target": "design:results-driven-video-split",
|
||||||
|
"relation": "informs",
|
||||||
|
"evidence": "修复诊断 error_type 污染,恢复 video-split 多样性维真实信号",
|
||||||
|
"added": "2026-07-16T01:31:36.665025+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "design:fix-diagnosis-tree-data-link",
|
||||||
|
"target": "schema:baseline-diagnosis",
|
||||||
|
"relation": "refines",
|
||||||
|
"evidence": "修正 error_type/evolution_target 列取值不再坍缩",
|
||||||
|
"added": "2026-07-16T01:31:37.792648+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:fix-diagnosis-tree-data-link-plan",
|
||||||
|
"target": "design:fix-diagnosis-tree-data-link",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "实现修复设计的 5 个 Task",
|
||||||
|
"added": "2026-07-16T02:01:19.350441+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:preflight-wp1-asset-migration",
|
||||||
|
"target": "design:preflight-fixes",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "四工作包实现训练前缺陷修复设计",
|
||||||
|
"added": "2026-07-16T08:32:42.737883+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:preflight-wp2-split-wiring",
|
||||||
|
"target": "design:preflight-fixes",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "四工作包实现训练前缺陷修复设计",
|
||||||
|
"added": "2026-07-16T08:32:42.822569+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:preflight-wp3-train-loop",
|
||||||
|
"target": "design:preflight-fixes",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "四工作包实现训练前缺陷修复设计",
|
||||||
|
"added": "2026-07-16T08:32:42.906881+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:preflight-wp4-resilience",
|
||||||
|
"target": "design:preflight-fixes",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "四工作包实现训练前缺陷修复设计",
|
||||||
|
"added": "2026-07-16T08:32:42.990273+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "review:preflight-final-review",
|
||||||
|
"target": "design:preflight-fixes",
|
||||||
|
"relation": "informs",
|
||||||
|
"evidence": "跨 task 终审确认设计 21 项缺陷全部落地 + 4 项集成发现",
|
||||||
|
"added": "2026-07-16T11:06:43.709183+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:gate-speedup",
|
||||||
|
"target": "design:gate-speedup",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "实现设计 v3 的 ②″+③",
|
||||||
|
"added": "2026-07-17T03:12:53.530801+00:00"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
+48
-6
@@ -1,8 +1,8 @@
|
|||||||
# Research Wiki 索引
|
# Research Wiki 索引
|
||||||
|
|
||||||
> 自动生成,更新时间:2026-07-14 21:09 UTC
|
> 自动生成,更新时间:2026-07-17 09:38 UTC
|
||||||
|
|
||||||
## design (29)
|
## design (40)
|
||||||
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
|
||||||
- [2026-07-07-app-harness-design](designs/2026-07-07-app-harness-design.md) `design:2026-07-07-app-harness-design`
|
- [2026-07-07-app-harness-design](designs/2026-07-07-app-harness-design.md) `design:2026-07-07-app-harness-design`
|
||||||
- [2026-07-07-core-evolution-design](designs/2026-07-07-core-evolution-design.md) `design:2026-07-07-core-evolution-design`
|
- [2026-07-07-core-evolution-design](designs/2026-07-07-core-evolution-design.md) `design:2026-07-07-core-evolution-design`
|
||||||
@@ -12,13 +12,20 @@
|
|||||||
- [2026-07-11-batch-tree-build-design](designs/2026-07-11-batch-tree-build-design.md) `design:2026-07-11-batch-tree-build-design`
|
- [2026-07-11-batch-tree-build-design](designs/2026-07-11-batch-tree-build-design.md) `design:2026-07-11-batch-tree-build-design`
|
||||||
- [2026-07-11-question-gen-v2-design](designs/2026-07-11-question-gen-v2-design.md) `design:2026-07-11-question-gen-v2-design`
|
- [2026-07-11-question-gen-v2-design](designs/2026-07-11-question-gen-v2-design.md) `design:2026-07-11-question-gen-v2-design`
|
||||||
- [2026-07-12-per-category-pool-strategy-design](designs/2026-07-12-per-category-pool-strategy-design.md) `design:2026-07-12-per-category-pool-strategy-design`
|
- [2026-07-12-per-category-pool-strategy-design](designs/2026-07-12-per-category-pool-strategy-design.md) `design:2026-07-12-per-category-pool-strategy-design`
|
||||||
|
- [2026-07-16-gate-speedup-design](designs/2026-07-16-gate-speedup-design.md) `design:2026-07-16-gate-speedup-design`
|
||||||
|
- [2026-07-16-preflight-fixes-design](designs/2026-07-16-preflight-fixes-design.md) `design:2026-07-16-preflight-fixes-design`
|
||||||
|
- [2026-07-16-telemetry-concurrency-fix-design](designs/2026-07-16-telemetry-concurrency-fix-design.md) `design:2026-07-16-telemetry-concurrency-fix-design`
|
||||||
- [Action Recognition 单题型首次训练实验设计](designs/2026-07-14-action-recognition-training-design.md) `design:2026-07-14-action-recognition-training-design`
|
- [Action Recognition 单题型首次训练实验设计](designs/2026-07-14-action-recognition-training-design.md) `design:2026-07-14-action-recognition-training-design`
|
||||||
|
- [gate 验证提速:预灌 BaselineCache + 双臂并行](designs/gate-speedup.md) `design:gate-speedup`
|
||||||
- [main.py 推理入口 + 初始 Prompt 集设计](designs/2026-07-09-main-inference-entry-design.md) `design:2026-07-09-main-inference-entry-design`
|
- [main.py 推理入口 + 初始 Prompt 集设计](designs/2026-07-09-main-inference-entry-design.md) `design:2026-07-09-main-inference-entry-design`
|
||||||
- [main.py 推理入口 + 初始 Prompt 集设计](designs/main-inference-entry.md) `design:main-inference-entry`
|
- [main.py 推理入口 + 初始 Prompt 集设计](designs/main-inference-entry.md) `design:main-inference-entry`
|
||||||
- [Per-Category Pool Strategy 设计](designs/per-category-pool-strategy.md) `design:per-category-pool-strategy`
|
- [Per-Category Pool Strategy 设计](designs/per-category-pool-strategy.md) `design:per-category-pool-strategy`
|
||||||
|
- [question-gen v2 — AR grounded-contrastive 重构(双模式生成 + 独立 VLM grounding + 生成端对抗)](designs/2026-07-15-question-gen-v2-grounded-contrastive-design.md) `design:2026-07-15-question-gen-v2-grounded-contrastive-design`
|
||||||
|
- [question-gen v3 — 构造优先范式重构(全孪生 + 四家族真独立 + 坍缩度量)](designs/2026-07-15-question-gen-v3-construction-paradigm-design.md) `design:2026-07-15-question-gen-v3-construction-paradigm-design`
|
||||||
- [Spec-1 Agent 执行环境修复(解析容错+步级重试+摘要附实体)](designs/agent-runtime-fixes.md) `design:agent-runtime-fixes`
|
- [Spec-1 Agent 执行环境修复(解析容错+步级重试+摘要附实体)](designs/agent-runtime-fixes.md) `design:agent-runtime-fixes`
|
||||||
- [Spec-2 建树批量并行入口](designs/batch-tree-build.md) `design:batch-tree-build`
|
- [Spec-2 建树批量并行入口](designs/batch-tree-build.md) `design:batch-tree-build`
|
||||||
- [Spec-3 出题管线 v2(失败机理靶向+逐题质量门)](designs/question-gen-v2.md) `design:question-gen-v2`
|
- [Spec-3 出题管线 v2(失败机理靶向+逐题质量门)](designs/question-gen-v2.md) `design:question-gen-v2`
|
||||||
|
- [修复诊断 tree_data 断链 bug(TRM4→TRM5 迁移 regression)](designs/fix-diagnosis-tree-data-link.md) `design:fix-diagnosis-tree-data-link`
|
||||||
- [出题模块迁移设计(question_gen)](designs/2026-07-07-question-gen-design.md) `design:2026-07-07-question-gen-design`
|
- [出题模块迁移设计(question_gen)](designs/2026-07-07-question-gen-design.md) `design:2026-07-07-question-gen-design`
|
||||||
- [出题管线 TaskTypeStrategy 拆分设计(Clean Architecture)](designs/2026-07-14-task-type-strategy-design.md) `design:2026-07-14-task-type-strategy-design`
|
- [出题管线 TaskTypeStrategy 拆分设计(Clean Architecture)](designs/2026-07-14-task-type-strategy-design.md) `design:2026-07-14-task-type-strategy-design`
|
||||||
- [出题质量提升 Phase A — Grounded 单题质量(候选池 + VLM 视觉打分 selector + 单维反事实)](designs/2026-07-14-grounded-question-gen-phaseA-design.md) `design:2026-07-14-grounded-question-gen-phaseA-design`
|
- [出题质量提升 Phase A — Grounded 单题质量(候选池 + VLM 视觉打分 selector + 单维反事实)](designs/2026-07-14-grounded-question-gen-phaseA-design.md) `design:2026-07-14-grounded-question-gen-phaseA-design`
|
||||||
@@ -27,22 +34,29 @@
|
|||||||
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/2026-07-07-tree-module-design.md) `design:2026-07-07-tree-module-design`
|
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/2026-07-07-tree-module-design.md) `design:2026-07-07-tree-module-design`
|
||||||
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/tree-module-vertical-slice.md) `design:tree-module-vertical-slice`
|
- [建树模块竖切设计:数据结构 + 建树 + 修复 + 迁移](designs/tree-module-vertical-slice.md) `design:tree-module-vertical-slice`
|
||||||
- [搜索 Agent 装配层设计(app/search/)](designs/2026-07-07-search-module-design.md) `design:2026-07-07-search-module-design`
|
- [搜索 Agent 装配层设计(app/search/)](designs/2026-07-07-search-module-design.md) `design:2026-07-07-search-module-design`
|
||||||
|
- [结果驱动的视频级 train/val/test 切分(替代自造题模块)](designs/2026-07-15-results-driven-video-split-design.md) `design:2026-07-15-results-driven-video-split-design`
|
||||||
|
- [结果驱动的视频级 train/val/test 切分(替代自造题模块)](designs/results-driven-video-split.md) `design:results-driven-video-split`
|
||||||
|
- [训练前缺陷修复设计](designs/preflight-fixes.md) `design:preflight-fixes`
|
||||||
- [训练池 Maintenance 正确题自动补入机制](designs/2026-07-14-maintenance-pool-design.md) `design:2026-07-14-maintenance-pool-design`
|
- [训练池 Maintenance 正确题自动补入机制](designs/2026-07-14-maintenance-pool-design.md) `design:2026-07-14-maintenance-pool-design`
|
||||||
- [训练池 Maintenance 正确题自动补入机制](designs/maintenance-pool.md) `design:maintenance-pool`
|
- [训练池 Maintenance 正确题自动补入机制](designs/maintenance-pool.md) `design:maintenance-pool`
|
||||||
- [论文主图:Self-Evolving Search Agent 推理训练闭环](designs/paper-main-figure.md) `design:paper-main-figure`
|
- [论文主图:Self-Evolving Search Agent 推理训练闭环](designs/paper-main-figure.md) `design:paper-main-figure`
|
||||||
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
- [赛题生成工具设计](designs/question-gen-synth.md) `design:question-gen-synth`
|
||||||
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
- [赛题生成工具设计(Question Generation Synthesis)](designs/2026-07-09-question-gen-synth-design.md) `design:2026-07-09-question-gen-synth-design`
|
||||||
|
- [遥测 SQLite 高并发写锁修复](designs/telemetry-concurrency-fix.md) `design:telemetry-concurrency-fix`
|
||||||
|
|
||||||
## finding (7)
|
## finding (10)
|
||||||
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
- [2026-07-11-benchmark-failure-taxonomy](findings/2026-07-11-benchmark-failure-taxonomy.md) `finding:2026-07-11-benchmark-failure-taxonomy`
|
||||||
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
- [2026-07-11-question-gen-calibration-analysis](findings/2026-07-11-question-gen-calibration-analysis.md) `finding:2026-07-11-question-gen-calibration-analysis`
|
||||||
- [2026-07-14-ar30-too-easy-analysis](findings/2026-07-14-ar30-too-easy-analysis.md) `finding:2026-07-14-ar30-too-easy-analysis`
|
- [2026-07-14-ar30-too-easy-analysis](findings/2026-07-14-ar30-too-easy-analysis.md) `finding:2026-07-14-ar30-too-easy-analysis`
|
||||||
- [2026-07-14-hard-distractor-literature](findings/2026-07-14-hard-distractor-literature.md) `finding:2026-07-14-hard-distractor-literature`
|
- [2026-07-14-hard-distractor-literature](findings/2026-07-14-hard-distractor-literature.md) `finding:2026-07-14-hard-distractor-literature`
|
||||||
- [2026-07-14-question-quality-gap-analysis](findings/2026-07-14-question-quality-gap-analysis.md) `finding:2026-07-14-question-quality-gap-analysis`
|
- [2026-07-14-question-quality-gap-analysis](findings/2026-07-14-question-quality-gap-analysis.md) `finding:2026-07-14-question-quality-gap-analysis`
|
||||||
|
- [2026-07-15-question-gen-v2-diagnosis-and-strategy](findings/2026-07-15-question-gen-v2-diagnosis-and-strategy.md) `finding:2026-07-15-question-gen-v2-diagnosis-and-strategy`
|
||||||
- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check`
|
- [Harness 评估: Spec-1 修复验证 (infer_spec1check)](findings/eval-spec1check.md) `finding:eval-spec1check`
|
||||||
- [Harness 评估: Spec-2 批量并行建树](findings/eval-spec2-batch-tree-build.md) `finding:eval-spec2-batch-tree-build`
|
- [Harness 评估: Spec-2 批量并行建树](findings/eval-spec2-batch-tree-build.md) `finding:eval-spec2-batch-tree-build`
|
||||||
|
- [v3 §9 帧感知抽取机制 — 小样本实证验证结果](findings/2026-07-15-v3-frame-perception-spike-validation.md) `finding:2026-07-15-v3-frame-perception-spike-validation`
|
||||||
|
- [出题范式转变 — 从"生成-打分-过滤"转向"构造优先 + 两正交独立信号"(六篇原文深读)](findings/2026-07-15-question-gen-paradigm-shift-construction-over-filtering.md) `finding:2026-07-15-question-gen-paradigm-shift-construction-over-filtering`
|
||||||
|
|
||||||
## plan (36)
|
## plan (50)
|
||||||
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
- [2026-07-06-core-agent-adapters-llm](plans/2026-07-06-core-agent-adapters-llm.md) `plan:2026-07-06-core-agent-adapters-llm`
|
||||||
- [2026-07-07-app-harness](plans/2026-07-07-app-harness.md) `plan:2026-07-07-app-harness`
|
- [2026-07-07-app-harness](plans/2026-07-07-app-harness.md) `plan:2026-07-07-app-harness`
|
||||||
- [2026-07-07-core-evolution](plans/2026-07-07-core-evolution.md) `plan:2026-07-07-core-evolution`
|
- [2026-07-07-core-evolution](plans/2026-07-07-core-evolution.md) `plan:2026-07-07-core-evolution`
|
||||||
@@ -60,6 +74,13 @@
|
|||||||
- [2026-07-14-grounded-question-gen-phaseA-plan](plans/2026-07-14-grounded-question-gen-phaseA-plan.md) `plan:2026-07-14-grounded-question-gen-phaseA-plan`
|
- [2026-07-14-grounded-question-gen-phaseA-plan](plans/2026-07-14-grounded-question-gen-phaseA-plan.md) `plan:2026-07-14-grounded-question-gen-phaseA-plan`
|
||||||
- [2026-07-14-maintenance-pool](plans/2026-07-14-maintenance-pool.md) `plan:2026-07-14-maintenance-pool`
|
- [2026-07-14-maintenance-pool](plans/2026-07-14-maintenance-pool.md) `plan:2026-07-14-maintenance-pool`
|
||||||
- [2026-07-14-task-type-strategy-framework](plans/2026-07-14-task-type-strategy-framework.md) `plan:2026-07-14-task-type-strategy-framework`
|
- [2026-07-14-task-type-strategy-framework](plans/2026-07-14-task-type-strategy-framework.md) `plan:2026-07-14-task-type-strategy-framework`
|
||||||
|
- [2026-07-15-question-gen-v3-phase1-contract](plans/2026-07-15-question-gen-v3-phase1-contract.md) `plan:2026-07-15-question-gen-v3-phase1-contract`
|
||||||
|
- [2026-07-15-results-driven-video-split](plans/2026-07-15-results-driven-video-split.md) `plan:2026-07-15-results-driven-video-split`
|
||||||
|
- [2026-07-16-gate-speedup](plans/2026-07-16-gate-speedup.md) `plan:2026-07-16-gate-speedup`
|
||||||
|
- [2026-07-16-preflight-wp1-asset-migration](plans/2026-07-16-preflight-wp1-asset-migration.md) `plan:2026-07-16-preflight-wp1-asset-migration`
|
||||||
|
- [2026-07-16-preflight-wp2-split-wiring](plans/2026-07-16-preflight-wp2-split-wiring.md) `plan:2026-07-16-preflight-wp2-split-wiring`
|
||||||
|
- [2026-07-16-preflight-wp3-train-loop](plans/2026-07-16-preflight-wp3-train-loop.md) `plan:2026-07-16-preflight-wp3-train-loop`
|
||||||
|
- [2026-07-16-preflight-wp4-resilience](plans/2026-07-16-preflight-wp4-resilience.md) `plan:2026-07-16-preflight-wp4-resilience`
|
||||||
- [Action Recognition 单题型首次训练实验计划](plans/action-recognition-training.md) `plan:action-recognition-training`
|
- [Action Recognition 单题型首次训练实验计划](plans/action-recognition-training.md) `plan:action-recognition-training`
|
||||||
- [ActionRecognitionStrategy 特化实现计划 (Plan B)](plans/action-recognition-strategy.md) `plan:action-recognition-strategy`
|
- [ActionRecognitionStrategy 特化实现计划 (Plan B)](plans/action-recognition-strategy.md) `plan:action-recognition-strategy`
|
||||||
- [Adversarial Question-Gen Phase B](plans/adversarial-question-gen-phaseB.md) `plan:adversarial-question-gen-phaseB`
|
- [Adversarial Question-Gen Phase B](plans/adversarial-question-gen-phaseB.md) `plan:adversarial-question-gen-phaseB`
|
||||||
@@ -74,19 +95,40 @@
|
|||||||
- [Spec-1 Agent 执行环境修复实现计划](plans/agent-runtime-fixes-plan.md) `plan:agent-runtime-fixes-plan`
|
- [Spec-1 Agent 执行环境修复实现计划](plans/agent-runtime-fixes-plan.md) `plan:agent-runtime-fixes-plan`
|
||||||
- [Spec-2 建树批量并行实现计划](plans/batch-tree-build-plan.md) `plan:batch-tree-build-plan`
|
- [Spec-2 建树批量并行实现计划](plans/batch-tree-build-plan.md) `plan:batch-tree-build-plan`
|
||||||
- [TaskTypeStrategy 框架实现计划 (Plan A)](plans/task-type-strategy-framework.md) `plan:task-type-strategy-framework`
|
- [TaskTypeStrategy 框架实现计划 (Plan A)](plans/task-type-strategy-framework.md) `plan:task-type-strategy-framework`
|
||||||
|
- [WP1 资产迁移](plans/preflight-wp1-asset-migration.md) `plan:preflight-wp1-asset-migration`
|
||||||
|
- [WP2 切分与接线](plans/preflight-wp2-split-wiring.md) `plan:preflight-wp2-split-wiring`
|
||||||
|
- [WP3 训练循环与进化引擎](plans/preflight-wp3-train-loop.md) `plan:preflight-wp3-train-loop`
|
||||||
|
- [WP4 韧性与持久化](plans/preflight-wp4-resilience.md) `plan:preflight-wp4-resilience`
|
||||||
- [出题管线 v2 实现计划](plans/2026-07-11-question-gen-v2.md) `plan:2026-07-11-question-gen-v2`
|
- [出题管线 v2 实现计划](plans/2026-07-11-question-gen-v2.md) `plan:2026-07-11-question-gen-v2`
|
||||||
|
- [实现计划: 修复诊断 tree_data 断链 bug](plans/fix-diagnosis-tree-data-link-plan.md) `plan:fix-diagnosis-tree-data-link-plan`
|
||||||
- [建树修复管线三项改造实现计划](plans/tree-repair-resilience.md) `plan:tree-repair-resilience`
|
- [建树修复管线三项改造实现计划](plans/tree-repair-resilience.md) `plan:tree-repair-resilience`
|
||||||
- [建树模块竖切实现计划](plans/tree-module-vertical-slice.md) `plan:tree-module-vertical-slice`
|
- [建树模块竖切实现计划](plans/tree-module-vertical-slice.md) `plan:tree-module-vertical-slice`
|
||||||
|
- [结果驱动的视频级切分实现计划](plans/results-driven-video-split-plan.md) `plan:results-driven-video-split-plan`
|
||||||
- [赛题生成工具实现计划](plans/question-gen-synth.md) `plan:question-gen-synth`
|
- [赛题生成工具实现计划](plans/question-gen-synth.md) `plan:question-gen-synth`
|
||||||
|
- [连续并发 gate + Redis 复用实现计划](plans/gate-speedup.md) `plan:gate-speedup`
|
||||||
- [项目基础设施初始化计划](plans/infrastructure-setup.md) `plan:infrastructure-setup`
|
- [项目基础设施初始化计划](plans/infrastructure-setup.md) `plan:infrastructure-setup`
|
||||||
|
|
||||||
## schema (3)
|
## review (6)
|
||||||
|
- [2026-07-16-preflight-final-review](reviews/2026-07-16-preflight-final-review.md) `review:2026-07-16-preflight-final-review`
|
||||||
|
- [2026-07-16-preflight-train-review](reviews/2026-07-16-preflight-train-review.md) `review:2026-07-16-preflight-train-review`
|
||||||
|
- [2026-07-17-gate-speedup-final-review](reviews/2026-07-17-gate-speedup-final-review.md) `review:2026-07-17-gate-speedup-final-review`
|
||||||
|
- [question-gen v2 设计对抗审核 — 六路独立核验(四层病灶闭合度 + 契约一致性)](reviews/2026-07-15-question-gen-v2-adversarial-audit.md) `review:2026-07-15-question-gen-v2-adversarial-audit`
|
||||||
|
- [训练前修复分支终审](reviews/preflight-final-review.md) `review:preflight-final-review`
|
||||||
|
- [连续并发 gate 终审与交付](reviews/gate-speedup-final.md) `review:gate-speedup-final`
|
||||||
|
|
||||||
|
## schema (6)
|
||||||
|
- [表结构 v3 出题日志/观测(unit_verdict / collapse_metrics / quarantine / facts / resume)](schemas/v3-question-gen-logging.md) `schema:v3-question-gen-logging`
|
||||||
- [表结构: adversarial_verdicts(Phase B agent 门判定)](schemas/adversarial-verdicts.md) `schema:adversarial-verdicts`
|
- [表结构: adversarial_verdicts(Phase B agent 门判定)](schemas/adversarial-verdicts.md) `schema:adversarial-verdicts`
|
||||||
|
- [表结构: baseline_diagnosis(逐题诊断信号)](schemas/baseline-diagnosis.md) `schema:baseline-diagnosis`
|
||||||
- [表结构: question_gen_items(逐题门判定)](schemas/question-gen-items.md) `schema:question-gen-items`
|
- [表结构: question_gen_items(逐题门判定)](schemas/question-gen-items.md) `schema:question-gen-items`
|
||||||
- [表结构: question_gen_runs(出题批次)](schemas/question-gen-runs.md) `schema:question-gen-runs`
|
- [表结构: question_gen_runs(出题批次)](schemas/question-gen-runs.md) `schema:question-gen-runs`
|
||||||
|
- [表结构: split_manifest(切分冻结溯源)](schemas/split-manifest.md) `schema:split-manifest`
|
||||||
|
|
||||||
## metric (4)
|
## metric (7)
|
||||||
- [出题四门总拦截率](metrics/qgen-gate-rejection-rate.md) `metric:qgen-gate-rejection-rate`
|
- [出题四门总拦截率](metrics/qgen-gate-rejection-rate.md) `metric:qgen-gate-rejection-rate`
|
||||||
|
- [指标: 48格(task_type×error_type)多样性覆盖率](metrics/split-cell-coverage.md) `metric:split-cell-coverage`
|
||||||
|
- [指标: floor 达标率 + test 代表性偏差](metrics/split-floor-and-representativeness.md) `metric:split-floor-and-representativeness`
|
||||||
|
- [指标: 诊断信号 tier 分布(T0/T1/T2/uncertain 占比)](metrics/split-signal-tier-distribution.md) `metric:split-signal-tier-distribution`
|
||||||
- [最终接受率](metrics/qgen-acceptance-rate.md) `metric:qgen-acceptance-rate`
|
- [最终接受率](metrics/qgen-acceptance-rate.md) `metric:qgen-acceptance-rate`
|
||||||
- [答案位置分布均匀性](metrics/qgen-answer-uniformity.md) `metric:qgen-answer-uniformity`
|
- [答案位置分布均匀性](metrics/qgen-answer-uniformity.md) `metric:qgen-answer-uniformity`
|
||||||
- [重出收敛率(3轮内过门)](metrics/qgen-regen-convergence.md) `metric:qgen-regen-convergence`
|
- [重出收敛率(3轮内过门)](metrics/qgen-regen-convergence.md) `metric:qgen-regen-convergence`
|
||||||
|
|||||||
@@ -92,3 +92,55 @@
|
|||||||
- [2026-07-14 21:09 UTC] 新增 plan: Adversarial Question-Gen Phase B (plan:adversarial-question-gen-phaseB)
|
- [2026-07-14 21:09 UTC] 新增 plan: Adversarial Question-Gen Phase B (plan:adversarial-question-gen-phaseB)
|
||||||
- [2026-07-14 21:09 UTC] 新增边: plan:adversarial-question-gen-phaseB --implements--> design:adversarial-question-gen-phaseB
|
- [2026-07-14 21:09 UTC] 新增边: plan:adversarial-question-gen-phaseB --implements--> design:adversarial-question-gen-phaseB
|
||||||
- [2026-07-14 21:09 UTC] 重建索引: 79 篇页面
|
- [2026-07-14 21:09 UTC] 重建索引: 79 篇页面
|
||||||
|
- [2026-07-15 09:18 UTC] 新增边: schema:v3-question-gen-logging --implements--> design:question-gen-v3-construction-paradigm
|
||||||
|
- [2026-07-15 09:18 UTC] 重建索引: 86 篇页面
|
||||||
|
- [2026-07-15 09:22 UTC] 新增 plan: question-gen v3 Phase1 契约地基实现计划 (plan:question-gen-v3-phase1-contract)
|
||||||
|
- [2026-07-15 09:22 UTC] 新增边: plan:question-gen-v3-phase1-contract --implements--> design:question-gen-v3-construction-paradigm
|
||||||
|
- [2026-07-15 09:22 UTC] 重建索引: 88 篇页面
|
||||||
|
- [2026-07-15 15:06 UTC] 新增 design: 结果驱动的视频级 train/val/test 切分(替代自造题模块) (design:results-driven-video-split)
|
||||||
|
- [2026-07-15 15:06 UTC] 重建索引: 89 篇页面
|
||||||
|
- [2026-07-15 15:27 UTC] 新增 plan: 结果驱动的视频级切分实现计划 (plan:results-driven-video-split-plan)
|
||||||
|
- [2026-07-15 15:27 UTC] 新增边: plan:results-driven-video-split-plan --implements--> design:results-driven-video-split
|
||||||
|
- [2026-07-15 15:27 UTC] 重建索引: 91 篇页面
|
||||||
|
- [2026-07-15 15:40 UTC] 新增 schema: 表结构: baseline_diagnosis(逐题诊断信号) (schema:baseline-diagnosis)
|
||||||
|
- [2026-07-15 15:40 UTC] 新增 schema: 表结构: split_manifest(切分冻结溯源) (schema:split-manifest)
|
||||||
|
- [2026-07-15 15:40 UTC] 新增 metric: 指标: 诊断信号 tier 分布(T0/T1/T2/uncertain 占比) (metric:split-signal-tier-distribution)
|
||||||
|
- [2026-07-15 15:40 UTC] 新增 metric: 指标: 48格(task_type×error_type)多样性覆盖率 (metric:split-cell-coverage)
|
||||||
|
- [2026-07-15 15:40 UTC] 新增 metric: 指标: floor 达标率 + test 代表性偏差 (metric:split-floor-and-representativeness)
|
||||||
|
- [2026-07-15 15:42 UTC] 新增边: schema:baseline-diagnosis --implements--> design:results-driven-video-split
|
||||||
|
- [2026-07-15 15:42 UTC] 新增边: schema:split-manifest --implements--> design:results-driven-video-split
|
||||||
|
- [2026-07-15 15:42 UTC] 新增边: metric:split-signal-tier-distribution --measures--> schema:baseline-diagnosis
|
||||||
|
- [2026-07-15 15:42 UTC] 新增边: metric:split-cell-coverage --measures--> schema:baseline-diagnosis
|
||||||
|
- [2026-07-15 15:42 UTC] 新增边: metric:split-floor-and-representativeness --measures--> schema:split-manifest
|
||||||
|
- [2026-07-15 15:42 UTC] 重建索引: 96 篇页面
|
||||||
|
- [2026-07-16 01:29 UTC] 新增 design: 修复诊断 tree_data 断链 bug(TRM4→TRM5 迁移 regression) (design:fix-diagnosis-tree-data-link)
|
||||||
|
- [2026-07-16 01:31 UTC] 重建索引: 97 篇页面
|
||||||
|
- [2026-07-16 01:31 UTC] 新增边: design:fix-diagnosis-tree-data-link --informs--> design:results-driven-video-split
|
||||||
|
- [2026-07-16 01:31 UTC] 新增边: design:fix-diagnosis-tree-data-link --refines--> schema:baseline-diagnosis
|
||||||
|
- [2026-07-16 01:31 UTC] 重建索引: 97 篇页面
|
||||||
|
- [2026-07-16 01:49 UTC] 新增 plan: 实现计划: 修复诊断 tree_data 断链 bug (plan:fix-diagnosis-tree-data-link-plan)
|
||||||
|
- [2026-07-16 02:01 UTC] 新增边: plan:fix-diagnosis-tree-data-link-plan --implements--> design:fix-diagnosis-tree-data-link
|
||||||
|
- [2026-07-16 02:01 UTC] 重建索引: 98 篇页面
|
||||||
|
- [2026-07-16 07:43 UTC] 新增 design: 训练前缺陷修复设计 (design:preflight-fixes)
|
||||||
|
- [2026-07-16 07:43 UTC] 重建索引: 101 篇页面
|
||||||
|
- [2026-07-16 08:32 UTC] 新增 plan: WP1 资产迁移 (plan:preflight-wp1-asset-migration)
|
||||||
|
- [2026-07-16 08:32 UTC] 新增边: plan:preflight-wp1-asset-migration --implements--> design:preflight-fixes
|
||||||
|
- [2026-07-16 08:32 UTC] 新增 plan: WP2 切分与接线 (plan:preflight-wp2-split-wiring)
|
||||||
|
- [2026-07-16 08:32 UTC] 新增边: plan:preflight-wp2-split-wiring --implements--> design:preflight-fixes
|
||||||
|
- [2026-07-16 08:32 UTC] 新增 plan: WP3 训练循环与进化引擎 (plan:preflight-wp3-train-loop)
|
||||||
|
- [2026-07-16 08:32 UTC] 新增边: plan:preflight-wp3-train-loop --implements--> design:preflight-fixes
|
||||||
|
- [2026-07-16 08:32 UTC] 新增 plan: WP4 韧性与持久化 (plan:preflight-wp4-resilience)
|
||||||
|
- [2026-07-16 08:32 UTC] 新增边: plan:preflight-wp4-resilience --implements--> design:preflight-fixes
|
||||||
|
- [2026-07-16 08:32 UTC] 重建索引: 109 篇页面
|
||||||
|
- [2026-07-16 11:06 UTC] 新增 review: 训练前修复分支终审 (review:preflight-final-review)
|
||||||
|
- [2026-07-16 11:06 UTC] 新增边: review:preflight-final-review --informs--> design:preflight-fixes
|
||||||
|
- [2026-07-16 11:06 UTC] 重建索引: 111 篇页面
|
||||||
|
- [2026-07-16 12:47 UTC] 新增 design: 遥测 SQLite 高并发写锁修复 (design:telemetry-concurrency-fix)
|
||||||
|
- [2026-07-16 12:47 UTC] 重建索引: 113 篇页面
|
||||||
|
- [2026-07-16 18:23 UTC] 新增 design: gate 验证提速:预灌 BaselineCache + 双臂并行 (design:gate-speedup)
|
||||||
|
- [2026-07-16 18:23 UTC] 重建索引: 115 篇页面
|
||||||
|
- [2026-07-17 03:12 UTC] 新增 plan: 连续并发 gate + Redis 复用实现计划 (plan:gate-speedup)
|
||||||
|
- [2026-07-17 03:12 UTC] 新增边: plan:gate-speedup --implements--> design:gate-speedup
|
||||||
|
- [2026-07-17 03:12 UTC] 重建索引: 117 篇页面
|
||||||
|
- [2026-07-17 09:38 UTC] 新增 review: 连续并发 gate 终审与交付 (review:gate-speedup-final)
|
||||||
|
- [2026-07-17 09:38 UTC] 重建索引: 119 篇页面
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
type: metric
|
||||||
|
node_id: metric:split-cell-coverage
|
||||||
|
title: "指标: 48格(task_type×error_type)多样性覆盖率"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 指标: 48格(task_type×error_type)多样性覆盖率
|
||||||
|
|
||||||
|
> 衡量 train+val 的 T2 defect 覆盖了多少种不同失败模式(多样性目标函数的验收)。
|
||||||
|
|
||||||
|
## 定义与判定
|
||||||
|
|
||||||
|
| 项 | 值 |
|
||||||
|
|----|----|
|
||||||
|
| 格子空间 | `task_type(12) × error_type(4)` = 48 格;但长尾类型无 defect 时对应格子天然空 |
|
||||||
|
| 覆盖数 | train+val 内 T2 题 `cell_of` 并集去重后的格子数 |
|
||||||
|
| 覆盖率 | 覆盖数 / 全数据集实际可达格子数(分母=全 240 非对中 T2 覆盖的格子集) |
|
||||||
|
| 基线 | **待首次诊断后建立**(实际可达格子数依赖 error_type 分布,运行前未知) |
|
||||||
|
| 判定 | 贪心 submodular 有 1−1/e 保证;验收要求 train+val 覆盖率接近全集可达格子(如 ≥90%),显著偏低提示 floor/ε 约束挤占了多样性 |
|
||||||
|
|
||||||
|
## 关联
|
||||||
|
- 度量表:`schema:baseline-diagnosis`、`schema:split-manifest`(coverage_report)
|
||||||
|
- 实现设计:`design:results-driven-video-split`
|
||||||
|
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
---
|
||||||
|
type: metric
|
||||||
|
node_id: metric:split-floor-and-representativeness
|
||||||
|
title: "指标: floor 达标率 + test 代表性偏差"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 指标: floor 达标率 + test 代表性偏差
|
||||||
|
|
||||||
|
> 衡量切分是否满足两条硬约束:train 每高信号类型 defect 下限 + test 相对全局的代表性。
|
||||||
|
|
||||||
|
## 定义与判定
|
||||||
|
|
||||||
|
| 项 | 定义 | 判定 |
|
||||||
|
|----|------|------|
|
||||||
|
| floor 达标率 | 各高信号类型 train+val 内 T2 数 ≥ `floor_k[type]` 的类型占比 | **必须 100%**;否则选择器 `InfeasibleSplitError` fail loud |
|
||||||
|
| test 类型比例偏差 | reportable 类型(总题 ≥ report_floor=27)在 test 的比例 vs 全局比例的最大偏差 | ≤ `epsilon`(拟定 0.1) |
|
||||||
|
| test 难度画像偏差 | test 的 0/1/2/3-对视频占比 vs 全局(125/115/55/5=41.7/38.3/18.3/1.7%)的最大偏差 | ≤ `epsilon` |
|
||||||
|
| val 功效 | val 内错题数 | ≥ `val_wrong_min`(拟定 ≥20,McNemar 功效);否则 `InsufficientValSignal` |
|
||||||
|
|
||||||
|
## 基线(全局分布,诊断无关,已知)
|
||||||
|
|
||||||
|
| 维度 | 全局值 |
|
||||||
|
|------|--------|
|
||||||
|
| 视频难度画像 | 全对125 / 1错115 / 2错55 / 3错5(共300) |
|
||||||
|
| reportable 类型(≥27题) | Object Reasoning 240、Action Reasoning 180、Info Synopsis 163、Temporal Reasoning 91、Action Recognition 63、Object Recognition 54、Counting 48、Attribute Perception 27 |
|
||||||
|
| 非 reportable | OCR 14、Spatial Reasoning 11、Temporal Perception 6、Spatial Perception 3(折进 overall) |
|
||||||
|
|
||||||
|
## 关联
|
||||||
|
- 度量表:`schema:split-manifest`(coverage_report)、`schema:baseline-diagnosis`
|
||||||
|
- 实现设计:`design:results-driven-video-split`
|
||||||
|
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
---
|
||||||
|
type: metric
|
||||||
|
node_id: metric:split-signal-tier-distribution
|
||||||
|
title: "指标: 诊断信号 tier 分布(T0/T1/T2/uncertain 占比)"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 指标: 诊断信号 tier 分布(T0/T1/T2/uncertain 占比)
|
||||||
|
|
||||||
|
> 衡量 240 非对题经诊断后各信号层占比,决定训练可用信号量。
|
||||||
|
|
||||||
|
## 定义与判定
|
||||||
|
|
||||||
|
| 项 | 值 |
|
||||||
|
|----|----|
|
||||||
|
| 数据源 | `baseline_diagnosis`(run_id=`infer_adhoc`,某 diag_fingerprint) |
|
||||||
|
| 计算 | 按 tier 分组计数 / 总非对题数 |
|
||||||
|
| 硬性锚点(诊断前已知) | 总题 900、正确 660、非对 **240**(236 可诊断错 + **4 INFRA→T0**) |
|
||||||
|
| T1/T2 分布 | **待首次诊断后建立**(defect vs lapse 由 judge 判定,运行前未知) |
|
||||||
|
| uncertain | judge 降级题占比,**待建立**;期望低(<5%),过高提示 judge 不稳 |
|
||||||
|
| 判定 | T2(可训练 defect)占比过低(如 <30% 可诊断错)→ 训练信号不足告警 → 触发设计 §14 方案 B 评估 |
|
||||||
|
|
||||||
|
## 关联
|
||||||
|
- 度量表:`schema:baseline-diagnosis`
|
||||||
|
- 实现设计:`design:results-driven-video-split`
|
||||||
|
|
||||||
@@ -0,0 +1,461 @@
|
|||||||
|
# question-gen v3 — Phase 1 契约地基 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 建立 `QuestionUnit(single|pair)` 契约并让它贯穿 harness 采样/分批/聚合/续跑/gate 全部拆-pair 入口,使后续阶段产出的 AR pair 题能被评测/训练正确消费,且 11 非 AR single 题行为字节级不变。
|
||||||
|
|
||||||
|
**Architecture:** 引入 `QuestionUnit` 判别联合(core 领域实体)+ `question_units.py` helper(build/flatten/validate/unit-correctness)。改造 pools/loader/batching/inference/runner/gate_ladder 使 pair **同池、同 batch 整锁、配对聚合(双向 AND)、同池切分**;correctness 分"逐题 predictions(溯源)/ unit correctness(进化消费)"两口径;pair 原子成对落盘;gate_ladder 加 schema_version 迁移;非 AR rng 独立 namespace。**在现有 `app/question_gen/` 内原地改,不建 _v3 目录。**
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11 / pytest / SQLite(run_store)/ asyncio。全程 `conda run -n Video-Tree-TRM`。
|
||||||
|
|
||||||
|
**依据**:设计 `research-wiki/designs/2026-07-15-question-gen-v3-construction-paradigm-design.md` §5/§8/§12;日志 schema `research-wiki/schemas/v3-question-gen-logging.md`。本阶段**不产出题目**,只建契约,交付物 = 全套 pair-契约回归测试绿 + 非 AR byte-identical 绿。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 文件结构映射
|
||||||
|
|
||||||
|
| 文件 | 职责 | 改动 |
|
||||||
|
|------|------|------|
|
||||||
|
| `core/types.py` | 领域实体 | 加 `QuestionUnit` + `GeneratedQuestion` 4 字段 |
|
||||||
|
| `app/harness/question_units.py` | unit helper(新建) | build/flatten/validate_units/unit_correctness |
|
||||||
|
| `app/harness/pools.py` | 三池切分 | build_pools/_sample_excluding/_split_one_category 以 unit 为原子 + _q_to_dict/_dict_to_q 序列化 pair 字段 |
|
||||||
|
| `app/question_gen/loader.py` | 采样/加载 | stratified_sample 按 unit + load_benchmark 读回 pair 字段 |
|
||||||
|
| `app/harness/batching.py` | 分批 | build_batches unit 整锁 + _select_mixed_by_task_type 按 unit correctness 分桶 + 非 AR 独立 rng |
|
||||||
|
| `app/harness/inference.py` | 聚合 | run_inference pair-level 双向 AND + unit 粒度 total/correct |
|
||||||
|
| `app/harness/runner.py` | 训练主环 | correctness 消费点改 unit 视图 + checkpoint 存 unit_id 序列 |
|
||||||
|
| `app/harness/gate_ladder.py` | 信息阶梯 | entry 迁 unit_id + schema_version 迁移 + BaselineCache unit 键 |
|
||||||
|
| `core/evolution/validate.py` | e-process 统计 | pair_block/compute_accuracy 按 unit |
|
||||||
|
| `app/harness/validate.py` | **gate 块实际执行**(Codex C-2)| baseline/candidate block、baseline_cache.get/put、n_used 按 unit |
|
||||||
|
| `app/question_gen/run_store.py` | 落库(唯一 run_store,非 app/harness/)| 加 facts/unit_verdict/collapse_metrics/quarantine/resume_state 表 |
|
||||||
|
| `app/question_gen/pair_atomic_writer.py` | pair 原子写 helper(新建)| pending buffer + os.replace + 孤儿剔除(wiring 挪 Phase 2)|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: QuestionUnit 领域实体 + GeneratedQuestion 字段扩展
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/types.py`
|
||||||
|
- Test: `tests/unit/test_question_unit.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_question_unit.py
|
||||||
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
def _q(qid, role="single", pair_id=None, flip_axis=None):
|
||||||
|
return GeneratedQuestion(
|
||||||
|
question_id=qid, video_id="v", task_type="Action Recognition",
|
||||||
|
question="?", options=("A. a", "B. b", "C. c", "D. d"), answer="A",
|
||||||
|
source_nodes=("n1",), difficulty="hard",
|
||||||
|
unit_id=pair_id or qid, pair_id=pair_id,
|
||||||
|
question_role=role, flip_axis=flip_axis,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_single_defaults_backward_compatible():
|
||||||
|
q = GeneratedQuestion(question_id="q", video_id="v", task_type="X",
|
||||||
|
question="?", options=("A. a","B. b","C. c","D. d"), answer="A",
|
||||||
|
source_nodes=("n1",), difficulty="hard")
|
||||||
|
assert q.question_role == "single"
|
||||||
|
assert q.pair_id is None and q.flip_axis is None and q.unit_id == "q"
|
||||||
|
|
||||||
|
def test_pair_unit_carries_two_questions():
|
||||||
|
p = _q("q_o", "pair_original", "pid", "before_after")
|
||||||
|
m = _q("q_m", "pair_mirror", "pid", "before_after")
|
||||||
|
u = QuestionUnit.from_pair(p, m)
|
||||||
|
assert u.kind == "pair" and u.size == 2 and u.unit_id == "pid"
|
||||||
|
assert {qq.question_id for qq in u.questions} == {"q_o", "q_m"}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 跑测试确认失败** — Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_question_unit.py -v` — Expected: FAIL(`QuestionUnit` 未定义 / 字段缺失)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# core/types.py —— GeneratedQuestion 增字段(默认值保非 AR 不变)
|
||||||
|
# 在 GeneratedQuestion dataclass 追加:
|
||||||
|
unit_id: str = "" # 默认在 __post_init__ 回填为 question_id
|
||||||
|
pair_id: str | None = None
|
||||||
|
question_role: str = "single" # single | pair_original | pair_mirror
|
||||||
|
flip_axis: str | None = None
|
||||||
|
# __post_init__ 中:if not self.unit_id: object.__setattr__(self, "unit_id", self.pair_id or self.question_id)
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class QuestionUnit:
|
||||||
|
kind: str # "single" | "pair"
|
||||||
|
unit_id: str
|
||||||
|
task_type: str
|
||||||
|
questions: tuple[GeneratedQuestion, ...]
|
||||||
|
unit_hash: str = "" # P/Q payload 合成 hash,断点续跑失效检测(T11 用)
|
||||||
|
# 注:设计 §8 的 collapse_metric 是验收产物,不进 Phase 1 契约实体,Phase 3/5 落 collapse_metrics 表
|
||||||
|
@property
|
||||||
|
def size(self) -> int: return len(self.questions)
|
||||||
|
@classmethod
|
||||||
|
def from_single(cls, q):
|
||||||
|
return cls("single", q.unit_id, q.task_type, (q,))
|
||||||
|
@classmethod
|
||||||
|
def from_pair(cls, original, mirror):
|
||||||
|
assert original.pair_id and original.pair_id == mirror.pair_id
|
||||||
|
assert original.video_id == mirror.video_id and original.task_type == mirror.task_type
|
||||||
|
assert original.flip_axis == mirror.flip_axis
|
||||||
|
return cls("pair", original.pair_id, original.task_type, (original, mirror))
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 跑测试确认通过** — Run: 同上 — Expected: PASS
|
||||||
|
- [ ] **Step 5: 提交** — `git add core/types.py tests/unit/test_question_unit.py && git commit`(用 commit skill)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: question_units.py helper(组装/展开/校验/unit correctness)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/question_units.py`
|
||||||
|
- Test: `tests/unit/test_question_units_helper.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_question_units_helper.py
|
||||||
|
from app.harness.question_units import build_units, flatten_units, validate_units, unit_correctness
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
def _q(qid, role="single", pid=None):
|
||||||
|
return GeneratedQuestion(question_id=qid, video_id="v", task_type="AR",
|
||||||
|
question="?", options=("A. a","B. b","C. c","D. d"), answer="A",
|
||||||
|
source_nodes=("n",), difficulty="hard", pair_id=pid, question_role=role,
|
||||||
|
unit_id=pid or qid, flip_axis="ax" if pid else None)
|
||||||
|
|
||||||
|
def test_build_units_groups_pair_and_keeps_single():
|
||||||
|
qs = [_q("s1"), _q("po","pair_original","p"), _q("pm","pair_mirror","p")]
|
||||||
|
units = build_units(qs)
|
||||||
|
kinds = sorted(u.kind for u in units)
|
||||||
|
assert kinds == ["pair", "single"]
|
||||||
|
|
||||||
|
def test_validate_units_rejects_orphan_pair():
|
||||||
|
import pytest
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
validate_units(build_units([_q("po","pair_original","p")])) # 只 1 条
|
||||||
|
|
||||||
|
def test_flatten_roundtrip():
|
||||||
|
qs = [_q("po","pair_original","p"), _q("pm","pair_mirror","p")]
|
||||||
|
assert {q.question_id for q in flatten_units(build_units(qs))} == {"po","pm"}
|
||||||
|
|
||||||
|
def test_unit_correctness_bidirectional_and():
|
||||||
|
qs = [_q("po","pair_original","p"), _q("pm","pair_mirror","p")]
|
||||||
|
u = build_units(qs)[0]
|
||||||
|
assert unit_correctness(u, {"po": True, "pm": True}) is True
|
||||||
|
assert unit_correctness(u, {"po": True, "pm": False}) is False # AND
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 跑确认失败** — `conda run -n Video-Tree-TRM pytest tests/unit/test_question_units_helper.py -v` — Expected: FAIL
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/harness/question_units.py
|
||||||
|
"""QuestionUnit 组装/展开/校验/单元正确性——pair 契约唯一入口。"""
|
||||||
|
from collections import defaultdict
|
||||||
|
from core.types import GeneratedQuestion, QuestionUnit
|
||||||
|
|
||||||
|
def build_units(questions: list[GeneratedQuestion]) -> list[QuestionUnit]:
|
||||||
|
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||||
|
singles: list[QuestionUnit] = []
|
||||||
|
for q in questions:
|
||||||
|
if q.pair_id:
|
||||||
|
by_pair[q.pair_id].append(q)
|
||||||
|
else:
|
||||||
|
singles.append(QuestionUnit.from_single(q))
|
||||||
|
pairs = []
|
||||||
|
for pid, qs in by_pair.items():
|
||||||
|
if len(qs) != 2:
|
||||||
|
raise ValueError(f"pair {pid} 数量={len(qs)}≠2(孤儿)")
|
||||||
|
o = next(q for q in qs if q.question_role == "pair_original")
|
||||||
|
m = next(q for q in qs if q.question_role == "pair_mirror")
|
||||||
|
pairs.append(QuestionUnit.from_pair(o, m))
|
||||||
|
return singles + pairs
|
||||||
|
|
||||||
|
def flatten_units(units: list[QuestionUnit]) -> list[GeneratedQuestion]:
|
||||||
|
return [q for u in units for q in u.questions]
|
||||||
|
|
||||||
|
def validate_units(units: list[QuestionUnit]) -> list[QuestionUnit]:
|
||||||
|
for u in units:
|
||||||
|
if u.kind == "pair" and u.size != 2:
|
||||||
|
raise ValueError(f"unit {u.unit_id} pair 不成对")
|
||||||
|
return units
|
||||||
|
|
||||||
|
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool]) -> bool:
|
||||||
|
"""AR pair = P AND Q;single = 单题。缺任一条 → KeyError(防静默)。"""
|
||||||
|
return all(per_q[q.question_id] for q in unit.questions)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 跑确认通过** — Expected: PASS
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: pools.py 三池切分以 unit 为原子(pair 同池)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py`(`build_pools` / `_sample_excluding` / `_split_one_category`)
|
||||||
|
- Test: `tests/unit/test_pools_pair_atomic.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**(pair 两题必同池,不被 progressive exclusion 劈开)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_pools_pair_atomic.py
|
||||||
|
from app.harness.pools import build_pools
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
def _pair(pid):
|
||||||
|
base = dict(video_id="v", task_type="AR", question="?",
|
||||||
|
options=("A. a","B. b","C. c","D. d"), answer="A",
|
||||||
|
source_nodes=("n",), difficulty="hard", pair_id=pid, unit_id=pid, flip_axis="ax")
|
||||||
|
return [GeneratedQuestion(question_id=f"{pid}_o", question_role="pair_original", **base),
|
||||||
|
GeneratedQuestion(question_id=f"{pid}_m", question_role="pair_mirror", **base)]
|
||||||
|
|
||||||
|
def test_pair_never_split_across_pools():
|
||||||
|
# 真实签名(app/harness/pools.py:51):build_pools(questions, correctness, diag_cfg, val_cfg, test_cfg, baseline_run_id)
|
||||||
|
qs = [q for pid in [f"p{i}" for i in range(12)] for q in _pair(pid)]
|
||||||
|
cfg = {"ratio": 0.34} # 用真实 PoolConfig/口径填三档;此处示意
|
||||||
|
pools = build_pools(qs, correctness={}, diag_cfg=cfg, val_cfg=cfg, test_cfg=cfg, baseline_run_id="b")
|
||||||
|
loc = {}
|
||||||
|
for name, pool in pools.items():
|
||||||
|
for q in pool:
|
||||||
|
loc.setdefault(q.pair_id, set()).add(name)
|
||||||
|
assert all(len(s) == 1 for s in loc.values()), "pair 被劈到多个池"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **实现者注**:先 Read `app/harness/pools.py:51` 确认 `diag_cfg/val_cfg/test_cfg` 的真实类型(PoolConfig dataclass 还是 dict),测试用真实构造。**红因必须是"pair 被拆"而非 TypeError**——签名不对会假红。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 跑确认失败**(现按 question_id 互斥会劈开)— Expected: FAIL(断言 pair 被拆,非 TypeError)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现** — `build_pools` 内先 `units = build_units(questions)`;三次 `_sample_excluding` 的互斥集合与采样对象都改 **unit_id**;`_split_one_category` 输入改 unit;**`build_incremental`(`pools.py:822-860`)的 categories qid 返回也改 unit 口径**;最后 `flatten_units` 展开。exclude 集合用 `unit.unit_id`。保留原 test→val→diag 顺序与比例(按 unit 计数)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 跑确认通过** — Expected: PASS
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: loader.stratified_sample 按 unit + load_benchmark 读回 pair 字段
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/question_gen/loader.py`(`stratified_sample` / `load_benchmark`)
|
||||||
|
- Test: `tests/unit/test_loader_unit_sampling.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:(a) pair 采样同进同出不拆;(b) 比例按 unit 计数;(c) **min_per_class 补足路径 `_backfill_per_class`(loader.py:154-172)不拆 pair**;(d) load_benchmark 读回 `pair_id`/`question_role`/`flip_axis`(`.get` 兼容旧 JSON)。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — `stratified_sample` 先 `build_units`,按 unit 分层/去重/补足/`rng.sample`(用 Task 5 的 `_rng_ns`)、`_backfill_per_class` candidates 按 unit 枚举,返回前 `flatten_units`;`load_benchmark` 反序列化补 `pair_id=d.get("pair_id")` 等 4 字段。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: batching unit 整锁 + 按 unit correctness 分桶 + 非 AR 独立 rng
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/batching.py`(`build_batches` / `_select_mixed_by_task_type` / `_distribute_large_classes`)
|
||||||
|
- Test: `tests/unit/test_batching_pair_lock.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:(a) 同 pair_id 两题落**同一 batch**(像小类整组不拆);(b) pair 按 **unit correctness(双向 AND)** 落 correct/error 桶,不因 P 对 Q 错被劈;(c) **非 AR byte-identical**:AR pair 折叠不改变非 AR 的 `rng.sample`/`shuffle` 抽样序列(黄金对照)。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — 分批输入改 unit 列表;FFD 容量按 `unit.size`;`_select_mixed_by_task_type` 用 `unit_correctness` 分桶、以 unit 为分发粒度;**AR 与非 AR 各用独立稳定派生的 rng**(**禁用 Python 内置 `hash()`——受 hash randomization 影响跨进程不可复现**):
|
||||||
|
|
||||||
|
```python
|
||||||
|
import hashlib
|
||||||
|
def _rng_ns(seed: int, ns: str):
|
||||||
|
import random
|
||||||
|
h = int.from_bytes(hashlib.sha256(f"{ns}:{seed}".encode()).digest()[:8], "big")
|
||||||
|
return random.Random(h)
|
||||||
|
# 非 AR 用 _rng_ns(seed, "nonAR"),AR pair 用 _rng_ns(seed, "AR"),二者 draw 流互不干扰
|
||||||
|
```
|
||||||
|
|
||||||
|
把 namespace 派生集中到此 helper(loader/pools/batching 共用),使 unit 折叠不干扰非 AR draw 流。flatten 前保证 pair 同 batch。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: inference pair-level 双向 AND 聚合
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/inference.py`(`run_inference` 聚合段)
|
||||||
|
- Test: `tests/unit/test_inference_pair_aggregate.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:逐题推理照常写 predictions;pair 按 pair_id 收齐两条合成 1 条 unit record,`pair 正确 = P对 AND Q对`;`InferenceResult.total/correct/per_task_type` 按 **unit 粒度**(total=single 数+pair 数);孤儿 pair(收不齐)→ 告警并剔除不计入 total(不静默)。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — 聚合前 `build_units`;per-question prediction 仍逐题落 predictions 表;unit 层用 `unit_correctness` 计 correct;孤儿按设计 §8(聚合入口)+ §12(原子性/读回校验)剔除+告警。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 7: correctness 三对象 API + runner 消费点改 unit 视图
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py`(rollout 回写 163-190 / accept 合并 1117-1141 / val 回写 1442-1454 / quadrant / momentum 1561 / probation 1385-1428)、`core/evolution/validate.py`(`pair_block:12` / `compute_accuracy:72`)、**`app/harness/validate.py`(C-2 关键遗漏:gate 块实际执行路径)**
|
||||||
|
- Test: `tests/unit/test_correctness_unit_view.py`、`tests/unit/test_gate_block_unit.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:(a) 进化引擎(gate/quadrant/momentum/probation/pair_block/compute_accuracy)消费 **unit correctness**(AR=双向 AND、非 AR=单题);逐题 predictions 仅溯源;混格下 e-process delta / 准确率分母按 unit 计、不被 P/Q 单题计分污染。(b) **`app/harness/validate.py::validate_skill_local` 的 gate 块按 unit 跑**——`baseline_cache.get/put` 键含 unit_id(`validate.py:282-301`)、块 qids 换 unit 折叠(`:553-565`)、`n_used` 按 unit 累加(非 `len(chunk)` 逐题)。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — 新增 `unit_correctness_view(units, per_q) -> dict[unit_id,bool]`;上述 6+ runner 调用点从 `correctness[qid]` 改消费 unit view;`core/evolution/validate.py::pair_block` 按 unit 折叠比对基线/候选臂、`compute_accuracy` 分母改 unit 数;**`app/harness/validate.py` 的 baseline/candidate block、evidence rows、`n_used`、`baseline_cache.get/put` 全部改 unit 口径,predictions 仍逐题溯源**(否则 gate 保真定义不进实际运行路径);quadrant/momentum 键改 unit_id。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**(commit message 标注核心算法保真#5 相关)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 8: gate_ladder 迁 unit_id + schema_version 迁移
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/gate_ladder.py`(`LadderEntry:43` / `build_cold_entries:57` / `ladder_for:135` / `update_probs:167` / `GatePools.save/load:181/200` / `BaselineCache:277`)
|
||||||
|
- 依赖: `app/harness/validate.py` 的 `baseline_cache.get/put(..., q.question_id)` 调用侧(T7 已改 unit 键;本 Task 保证 gate_ladder 侧 key schema 与之对齐)
|
||||||
|
- Test: `tests/unit/test_gate_ladder_unit_migration.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:(a) `LadderEntry` 按 unit_id;(b) 冷启动"错优先 2:1"unit 错=P 或 Q 任一错;(c) `update_probs` 观测先折叠成 unit 再匹配(防按 qid 匹配失效致 EMA 停摆);(d) `GatePools.save/load` 带 `schema_version`;存量无版本 json 加载→明确报错或走迁移(不静默混用);(e) `BaselineCache` 键含 unit_id。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — 见设计 §17 gate-unit 迁移 ADR 七项;`p_hat` 初值 Beta 先验按 unit 定义;probe_quota 按 unit 抽;反泄漏(run_id 含 `_gate_` 过滤)不变。写一次性迁移函数 `migrate_gate_pools_v1_to_unit()`。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**(标注核心算法保真#5,需逐行比对参考 `Video-Tree-TRM4/core/harness/gate_ladder.py`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 9: pools.json 冻结/解冻序列化 pair 字段
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py`(`_q_to_dict` / `_dict_to_q` / `save_pools.categories`)
|
||||||
|
- Test: `tests/unit/test_pools_serialization.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:pair 冻结进 pools.json 再 `load_pools`,`pair_id`/`question_role`/`flip_axis`/`unit_id` 不丢;`categories` 块以 unit 记 train/val;旧 JSON(无字段)`.get` 兼容不崩。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — `_q_to_dict` 写出 4 字段;`_dict_to_q` `.get(..., 默认)` 读回并回填 unit_id。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 10: checkpoint/resume + 非 AR byte-identical 黄金测试
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py`(`epoch_batches` / `_batch_from_ids`)
|
||||||
|
- Test: `tests/integration/test_checkpoint_pair.py`、`tests/unit/test_non_ar_byte_identical.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:(a) checkpoint 存 **unit_id 序列**,`_batch_from_ids` 恢复时展开完整 unit(断点续跑后 pair 不拆);(b) **纯非 AR 题库**过 pools/batching/inference 的抽样与分批结果与"引入 QuestionUnit 前"**逐字节一致**(黄金文件);(c) **`runner.py:1561` momentum 采样**(`random.Random(epoch).sample(candidates)`,Codex I-3):混格下 AR pair 折叠会改 candidates 长度/顺序→非 AR momentum 样本漂移。**要么** momentum candidates 用独立 rng namespace(Task 5 helper)+ 加混格 momentum golden,**要么**在计划显式记录"Phase 1 不保证 momentum 混格 byte-identical"作为设计偏差。二选一,不留隐患。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — checkpoint 序列化 unit_id;恢复用 `build_units` + 展开;非 AR 走 size=1 unit 且独立 rng namespace(Task 5)保证黄金一致。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 11: pair 原子成对落盘 helper(可复用纯件;wiring 挪 Phase 2)
|
||||||
|
|
||||||
|
**背景(Codex C-3)**:`app/question_gen/run_store.py` 是 SQLite 日志类,**无 on_accept/accepted-JSON 钩子**;真实"accepted 题库文件写入"在生成侧(旧 `pipeline_v2` 的 on_accept 回调 / `adversarial_filter.write_final_bank`,均 Phase 2 重建)。故 Phase 1 **只建可复用、可独立测试的 pair 原子写 helper**,实际接到新 `pipeline.on_accept` 的 wiring **在 Phase 2 做**(新 pipeline 落地时)。
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/question_gen/pair_atomic_writer.py`(纯 helper)
|
||||||
|
- Test: `tests/unit/test_pair_atomic_write.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 审计定位真实 accepted 写入点** — `grep -rn "on_accept\|write_final_bank\|accepted_questions" app/question_gen/ tools/` 记录当前 accepted 文件写入函数(供 Phase 2 wiring 参考),写入计划注释。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 写失败测试** — 针对 `pair_atomic_writer`:pending buffer 按 pair_id 收齐 original+mirror 才一次性 emit unit;`write_accepted(path, units)` 全量 tmp + `os.replace`;模拟"只落 P 未落 Q"→读回 `validate_units` 剔孤儿;single 恒直接成 unit;`unit_hash` 不一致→拒。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 跑确认失败**
|
||||||
|
|
||||||
|
- [ ] **Step 4: 实现** — `PairPendingBuffer.add(q)`(按 pair_id 收齐才 emit)+ `write_accepted(path, units)`(tmp+os.replace)+ `read_accepted(path)`(`validate_units` 剔孤儿)。纯函数,不依赖 pipeline。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 跑确认通过 + 提交**(计划注释标注:on_accept wiring 见 Phase 2)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 12: run_store v3 表(facts/unit_verdict/collapse_metrics/quarantine/resume_state)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/question_gen/run_store.py`(建表 + insert 方法)
|
||||||
|
- Test: `tests/unit/test_run_store_v3_tables.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**:`store.insert_fact/insert_unit_verdict/insert_collapse_metrics/quarantine/upsert_resume_state` 落库+读回;quarantine 内容指纹去重(同指纹 upsert 不重复);ts 由外部传入(禁进程内 now,保幂等/可复现)。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 实现** — 按 `research-wiki/schemas/v3-question-gen-logging.md` 5 张表建表 + 索引 + insert 方法;对齐 CLAUDE.md §4.8。**ts 边界(Codex I-4)明确**:v3 新表(facts/unit_verdict/collapse_metrics/quarantine/resume_state)**ts 一律外部传入、禁进程内 now**(保幂等可复现);v2 deprecated 表(question_gen_runs/items/adversarial_verdicts)**暂保留现有 now() 行为不动**(Phase 2/4 删)——不因本 Task 误改旧表。
|
||||||
|
- [ ] **Step 4: 跑确认通过**
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 13: Phase 1 集成回归(pair 契约全链路 + 混格评分)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Test: `tests/integration/test_v3_contract_e2e.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写集成测试**:构造混格题库(若干 AR pair + 若干非 AR single),跑 build_pools→build_batches→run_inference 全链路,断言:pair 全程不拆、同批、双向 AND 聚合正确、unit 粒度 total/correct 正确、孤儿被剔、非 AR byte-identical。
|
||||||
|
- [ ] **Step 2: 跑确认失败**
|
||||||
|
- [ ] **Step 3: 补齐**前序 Task 遗漏
|
||||||
|
- [ ] **Step 4: 跑全套** — `conda run -n Video-Tree-TRM pytest tests/ -q` 全绿 + 覆盖率≥80%
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 14: 旧代码处理与死代码清除(Phase 1 收尾)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py` / `batching.py` / `inference.py` / `runner.py` / `gate_ladder.py` / `core/evolution/validate.py`(Task 3-10 改过的文件)、`app/question_gen/run_store.py`
|
||||||
|
- Test: `tests/unit/test_no_dead_perquestion_paths.py`
|
||||||
|
|
||||||
|
**背景**:Task 3-10 按 CLAUDE.md §4.2"直接改原文件"做**原地替换**(非新增并行路径),本 Task 确保替换后不留孤儿死代码,并明确 v2 落库表的过渡去向。**不删 v2 生成模块**(那在 Phase 2,替代品落地后才删)。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 审计 unit 迁移后的孤儿函数** — 对 Task 3-10 改过的文件,用 `conda run -n Video-Tree-TRM ruff check --select F811,F401 app/harness/ core/evolution/` + `grep -rn "def _batch_from_ids\|def <被替换的逐题helper>" app/harness/` 逐一确认:被 unit 版替换掉的旧逐题函数/helper(如仅旧 `_batch_from_ids` 逐题重建、旧逐题 correctness helper)是否仍被引用。列出无引用者。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 写守卫测试**(防旧逐题路径复活/残留)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_no_dead_perquestion_paths.py
|
||||||
|
import inspect
|
||||||
|
from app.harness import batching, pools, inference
|
||||||
|
|
||||||
|
def test_no_parallel_perquestion_split_helpers():
|
||||||
|
"""契约迁 unit 后,不得残留会拆 pair 的旧逐题分批/切分/gate 块路径。"""
|
||||||
|
from app.harness import validate as hvalidate # gate 块真实路径
|
||||||
|
src = inspect.getsource(batching) + inspect.getsource(pools) + inspect.getsource(hvalidate)
|
||||||
|
# 旧逐题标志(按实际被替换的函数名调整):断言已被 unit 版取代、无并行残留
|
||||||
|
assert "correctness.get(qid)" not in src, "batching 仍有逐题分桶残留"
|
||||||
|
# gate 块残留探测(Codex:validate.py 是逐题 gate 核心残留点)
|
||||||
|
assert "baseline_cache.get(" not in src or "unit_id" in src, "validate baseline_cache 仍按 qid"
|
||||||
|
assert "n_used += len(chunk)" not in src, "validate n_used 仍逐题累加(应按 unit)"
|
||||||
|
assert src.count("build_units") >= 1 or "unit_id" in src, "pools/batching/validate 未走 unit 化"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: 删除孤儿死代码 + 标注 v2 表过渡** — 删掉 Step 1 确认无引用的旧逐题函数;`run_store.py` 里 v2 表(`question_gen_runs`/`question_gen_items`/`adversarial_verdicts`)**保留但加 deprecation 注释**(`# DEPRECATED(v3): v2 生成落库,Phase 2 生成替换后删;过渡期与 v3 表并存`)——不在 Phase 1 删(生成逻辑还没换)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 跑测试 + lint** — Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_no_dead_perquestion_paths.py -v && ruff check app/harness/ core/evolution/` — Expected: PASS + 无 F811/F401
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review 检查
|
||||||
|
|
||||||
|
- **Spec 覆盖**:设计 §8 的 ≥13 入口逐一对应——pools 三池(T3)/序列化(T9)、loader(T4)、batching 整锁+分桶+rng(T5,T10)、inference 聚合(T6)、correctness+validate(T7)、gate_ladder+迁移(T8)、checkpoint+BaselineCache(T10,T8)、load_benchmark(T4)、pair 原子写(T11)、run_store 表(T12)。§12 非功能性(原子/续跑/幂等)→ T10/T11/T12。日志 schema → T12。**全覆盖。**
|
||||||
|
- **类型一致**:`QuestionUnit`/`build_units`/`unit_correctness`/`flatten_units`/`validate_units` 在 T1/T2 定义,T3-T13 一致引用。
|
||||||
|
- **无占位**:各 Task 有真实测试+实现骨架+命令。机械 Task(T4/T9)复用 T2/T3 已给模式。
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
本阶段触及**核心算法#5 信息阶梯**(gate_ladder 迁 unit,Task 7/8):须逐行比对参考 `/home/iomgaa/Projects/Video-Tree-TRM4/core/harness/gate_ladder.py`(冷启动 2:1/gamma-EMA/Beta 先验/反泄漏),按 unit 重定义**不简化**,Task 8 已设"保真校验"检查点。其余 12 项不涉及。
|
||||||
|
|
||||||
|
## 后续阶段(各自成计划)+ 旧代码删除清单(显式,防隐式漂)
|
||||||
|
|
||||||
|
Phase 2 帧感知抽取+构造器 / Phase 3 六层验证栈 / Phase 4 对抗前移+产量 / Phase 5 验收面板+混格全链路——进入时各写独立 plan。
|
||||||
|
|
||||||
|
**旧代码删除必须落成对应阶段的显式任务(不得只"替换"而留死壳):**
|
||||||
|
|
||||||
|
| 废弃模块 | 处置 | 落哪阶段(显式删除任务)|
|
||||||
|
|---------|------|----------------------|
|
||||||
|
| `pipeline_v2.py` | 被 `pipeline`(帧感知主编排)原地替换 | **Phase 2** |
|
||||||
|
| `generator_v2.py` | 被 `constructor`+`twin_builder`(构造)取代 | **Phase 2** |
|
||||||
|
| `synthesizer.py` | v1 遗留生成,直接删 | **Phase 2** |
|
||||||
|
| `distractor_selector.py` | 打分 selector 废(GroundAttack 软肋),删 | **Phase 2** |
|
||||||
|
| `gates.py` | 四门(AFLite 单信号死路)废;`blind_answer` 思路迁 §7 坍缩度量后删 | **Phase 3**(坍缩度量落地后)|
|
||||||
|
| `adversarial_filter.py` | 拆解:孪生构造→Phase 2、活求解器探针/verdicts 续跑→Phase 4;机制迁完删壳 | **Phase 4** |
|
||||||
|
| `store/prompts/question_gen/ar_distractor_*`、`gate_*` | 打分/后置门 prompt 废;`ar_mirror_question`/`gate_blind_answer` 借鉴重写后删旧 | Phase 2/3 |
|
||||||
|
| `config/question_gen_ar30.yaml` 的 `candidate_pool_size`/`selector_delta_*` | selector 参数废 | Phase 2 |
|
||||||
|
| v2 run_store 表 `question_gen_runs/items/adversarial_verdicts` | Phase 1 加 deprecation 注释保留;生成替换后删 | Phase 2(items/runs)/ Phase 4(verdicts)|
|
||||||
|
|
||||||
|
> 每阶段计划的最后一个 Task **必须**是"旧代码删除 + 死代码清除 + lint 无 F811/F401",与本 Phase 的 Task 14 同构。
|
||||||
@@ -0,0 +1,694 @@
|
|||||||
|
# 结果驱动的视频级切分 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 用离线、结果驱动的视频级 train/val/test 切分(基于已有 `infer_adhoc` baseline + 240 错题诊断),产出冻结 `pools.json`,替代自造训练题。
|
||||||
|
|
||||||
|
**Architecture:** 复用现有 `run_diagnosis` 拿逐题 `error_attributions`;新增纯函数 `split_selection`(signal 分层 → 视频聚合 → 贪心联合约束选择);改 `pools.py` 切分原子 unit→video 并补原子写;全程离线可复现(fixed seed + baseline_run_id + diag_fingerprint 溯源)。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、pytest、SQLite(harness.db)、asyncio、loguru、pydantic-settings、YAML(科研配置)。
|
||||||
|
|
||||||
|
**设计源**:`research-wiki/designs/2026-07-15-results-driven-video-split-design.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键代码锚点(实现前必读)
|
||||||
|
|
||||||
|
| 用途 | 位置 |
|
||||||
|
|------|------|
|
||||||
|
| 诊断入口(复用) | `core/evolution/diagnose.py:2017` `run_diagnosis(...) -> DiagnosisResult` |
|
||||||
|
| 逐题归因产物 | `DiagnosisResult.error_attributions: list[ErrorAttribution]`(`core/evolution/types.py:257,149`) |
|
||||||
|
| INFRA 排除集 | `core/evolution/diagnose.py:56` `_INFRA_STOP_REASONS={"error","parse_error"}` |
|
||||||
|
| error_type 瀑布 | `core/evolution/diagnose.py:910-941`(extraction/search/reasoning/mixed) |
|
||||||
|
| RunLog 读端口 | `core/evolution/protocols.py:68`(`get_predictions`/`get_traces`) |
|
||||||
|
| 三池结构 | `app/harness/pools.py:33` `Pools`;`build_pools:53`;`_split_one_category:809`(签名行,822 是 docstring) |
|
||||||
|
| 切分入口/冻结 | `app/harness/pools.py:473` `build_or_load_pools`;`save_pools:289`(`path.write_text:341` 非原子);per_category 写(566) 非原子 |
|
||||||
|
| PoolConfig | `core/types.py:146`(seed/diag_size/val_size/test_size/train_ratio) |
|
||||||
|
| baseline 数据 | `workspaces/default/harness.db`,`predictions` run_id=`infer_adhoc`;`traces` 表**空**,轨迹在 `predictions.steps_json` |
|
||||||
|
|
||||||
|
### Canonical baseline 事实(Codex 计划审 C1/C2 核实,实现须严格对齐)
|
||||||
|
|
||||||
|
- **902 原始行 / 900 distinct question / 300 视频**;`743-1` 有 **3 行**(error/budget_exceeded/finished)→ **canonical 取行策略 = 每 question_id 取第一行**(`ORDER BY rowid`),全流程统一。
|
||||||
|
- **660 对 / 236 可诊断错题(非空 pred,非 INFRA)/ 4 空 pred(全 INFRA:2 error + 2 parse_error)** → 非对合计 240。
|
||||||
|
- **`only_incorrect=True` + INFRA 排除**后进诊断的是 **236 题**;4 个 INFRA 空 pred 直接落 **T0**(不进 judge)。
|
||||||
|
- **`steps_json` 步字段 = `{thought, tool_call:{tool, args}, tool_output}`**——工具名在 **`tool_call.tool`**(非 `name`;对齐 `core/evolution/diagnose.py:258` 的 `tool_call.get("tool")`)。899/902 行非空,空的走空返回。
|
||||||
|
- **防御断言按 distinct question(900)计,不按裸行(902)**。
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
本计划**不迁移/不改**核心算法(`ARCHITECTURE.md §6`)。唯一相邻项是第 5 项"信息阶梯冷启动 2:1"(`gate_ladder.py`):本计划只把 **pools 切分边界**改为 video,**pools 内部仍是 unit 粒度**,`gate_ladder` 的输入(unit + correctness)不变。**保真检查点(Task 9 Step 4)**:确认 video 切分后 `Pools.diagnosis/validation` 内仍是逐 unit 列表,`gate_ladder.build_cold_entries` 消费不变、冷启动 2:1 未受影响。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 0:schema 前置门(编码前必须完成)
|
||||||
|
|
||||||
|
### Task 0: 用 structured-logging 定稿并注册 schema
|
||||||
|
|
||||||
|
**Files:** 无代码;产出 `research-wiki/schemas/` 注册 + 本计划 §schema 表定稿。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 调用 structured-logging skill**,为下述两张表定稿列/类型/索引/基线指标并注册 Wiki。设计已给骨架,structured-logging 负责批准与注册:
|
||||||
|
|
||||||
|
`baseline_diagnosis`(逐题诊断信号,harness.db):
|
||||||
|
|
||||||
|
| 列 | 类型 | 说明 |
|
||||||
|
|----|------|------|
|
||||||
|
| question_id | TEXT | 题 ID |
|
||||||
|
| video_id | TEXT | 视频 ID |
|
||||||
|
| baseline_run_id | TEXT | 溯源 run(`infer_adhoc`) |
|
||||||
|
| diag_fingerprint | TEXT | `hash(诊断prompt版本+model+诊断代码版本)` |
|
||||||
|
| task_type | TEXT | 12 类之一 |
|
||||||
|
| error_type | TEXT **NULL** | extraction/search/reasoning/mixed;**T0/uncertain 行为 NULL**(Codex I4:INFRA/degraded 题无合法 error_type) |
|
||||||
|
| cause_category | TEXT NULL | defect/lapse/NULL |
|
||||||
|
| tier | TEXT | T0/T1/T2/uncertain |
|
||||||
|
| evolution_target | TEXT **NULL** | 由 error_type 派生(tool/skill/system);error_type 为 NULL 时亦 NULL |
|
||||||
|
| degraded | INTEGER | judge 降级 0/1 |
|
||||||
|
| infra | INTEGER | INFRA 排除 0/1 |
|
||||||
|
| session_id | TEXT | 遥测关联 |
|
||||||
|
| **PK** | | (question_id, baseline_run_id, diag_fingerprint) |
|
||||||
|
|
||||||
|
> **T0/uncertain 写规则**:INFRA 题 `infra=1, tier=T0, error_type/cause_category/evolution_target=NULL`;degraded/诊断失败题 `degraded=1, tier=uncertain, error_type` 可为 NULL。二者均不进 T2、不入 48 格覆盖。
|
||||||
|
|
||||||
|
`split_manifest`(冻结产物溯源,随 pools.json 同目录 JSON):
|
||||||
|
|
||||||
|
| 键 | 说明 |
|
||||||
|
|----|------|
|
||||||
|
| baseline_run_id / diag_fingerprint / seed | 复现三元组 |
|
||||||
|
| config | N_trainval/floor_K/ε/report_floor/val 尺寸 快照 |
|
||||||
|
| pools_sha256 | pools.json 内容 hash |
|
||||||
|
| coverage_report | 48 格覆盖数、floor 达标情况、test 代表性偏差 |
|
||||||
|
|
||||||
|
- [ ] **Step 2:** 确认 structured-logging 已在 `research-wiki/schemas/` 注册二表并给出基线指标(各 tier 占比、48 格覆盖率、floor 达标率)。**此门通过前不得进入 Phase 1。**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1:离线诊断管线
|
||||||
|
|
||||||
|
### Task 1: steps_json → trace 行适配器
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/steps_json_traces.py`
|
||||||
|
- Test: `tests/unit/test_steps_json_traces.py`
|
||||||
|
|
||||||
|
`run_diagnosis` 经 `get_traces` 取轨迹,但 `infer_adhoc` 的 `traces` 表空、轨迹在 `predictions.steps_json`。此适配器把 `{thought, tool_call, tool_output}` 步转成 `get_traces` 的行形 `{video_id, question_id, step, tool_name, tool_args, tool_output, thought}`。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_steps_json_traces.py
|
||||||
|
import json
|
||||||
|
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||||||
|
|
||||||
|
|
||||||
|
def test_parses_tool_call_into_name_and_args():
|
||||||
|
# 真实 infer_adhoc steps_json 形态:tool_call={"tool":..., "args":...}(非 "name")
|
||||||
|
steps = [
|
||||||
|
{"thought": "看根节点", "tool_call": {"tool": "view_node", "args": {"node_id": "v_L1_000"}}, "tool_output": "o0"},
|
||||||
|
{"thought": "搜索", "tool_call": {"tool": "search_similar", "args": {"query": "gadget"}}, "tool_output": "hit"},
|
||||||
|
]
|
||||||
|
rows = steps_json_to_trace_rows("vid1", "q1", json.dumps(steps))
|
||||||
|
assert [r["step"] for r in rows] == [0, 1]
|
||||||
|
assert rows[0]["tool_name"] == "view_node"
|
||||||
|
assert rows[0]["tool_args"] == {"node_id": "v_L1_000"}
|
||||||
|
assert rows[0]["video_id"] == "vid1" and rows[0]["question_id"] == "q1"
|
||||||
|
assert rows[1]["tool_output"] == "hit"
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_or_blank_steps_json_returns_empty():
|
||||||
|
assert steps_json_to_trace_rows("v", "q", "") == []
|
||||||
|
assert steps_json_to_trace_rows("v", "q", "[]") == []
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败** — `conda activate Video-Tree-TRM && pytest tests/unit/test_steps_json_traces.py -v`,预期 `ModuleNotFoundError`。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 最小实现**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/harness/steps_json_traces.py
|
||||||
|
"""把 predictions.steps_json 转成 RunLog.get_traces 的行形。
|
||||||
|
|
||||||
|
infer_adhoc 的 traces 表为空,轨迹存于 steps_json({thought, tool_call, tool_output})。
|
||||||
|
诊断管线经 get_traces 消费轨迹,故需此确定性转换适配。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def steps_json_to_trace_rows(video_id: str, question_id: str, steps_json: str) -> list[dict[str, Any]]:
|
||||||
|
"""将单题 steps_json 解析为 trace 行列表(step 从 0 递增)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
video_id: 视频 ID。
|
||||||
|
question_id: 题 ID。
|
||||||
|
steps_json: predictions.steps_json 原文(JSON 数组字符串)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
行字典列表,字段对齐 traces 表 schema;空/空数组返回 []。
|
||||||
|
"""
|
||||||
|
if not steps_json or not steps_json.strip():
|
||||||
|
return []
|
||||||
|
steps = json.loads(steps_json)
|
||||||
|
if not isinstance(steps, list):
|
||||||
|
raise ValueError(f"steps_json 非数组: {question_id}")
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for i, s in enumerate(steps):
|
||||||
|
call = s.get("tool_call") or {}
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"video_id": video_id,
|
||||||
|
"question_id": question_id,
|
||||||
|
"step": i,
|
||||||
|
# infer_adhoc 用 "tool";back-compat 兼容极少数 "name"
|
||||||
|
"tool_name": call.get("tool", call.get("name")),
|
||||||
|
"tool_args": call.get("args", {}),
|
||||||
|
"tool_output": s.get("tool_output"),
|
||||||
|
"thought": s.get("thought"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return rows
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_steps_json_traces.py -v`,预期 PASS。
|
||||||
|
- [ ] **Step 5: 提交** — `git add app/harness/steps_json_traces.py tests/unit/test_steps_json_traces.py && git commit -m "feat: add steps_json to trace-row adapter"`
|
||||||
|
|
||||||
|
### Task 2: RunLog 包装器(预测直取 + 轨迹从 steps_json)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/baseline_run_log.py`
|
||||||
|
- Test: `tests/unit/test_baseline_run_log.py`
|
||||||
|
|
||||||
|
包装现有 SQLite RunLog:`get_predictions` 透传;`get_traces` 当底层 traces 空时,从 predictions.steps_json 经 Task 1 生成。不写 SQL 于 app 之外的裸路径——复用底层适配器。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_baseline_run_log.py
|
||||||
|
import json
|
||||||
|
import pytest
|
||||||
|
from app.harness.baseline_run_log import StepsJsonRunLog
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeInner:
|
||||||
|
def __init__(self, preds, traces):
|
||||||
|
self._preds, self._traces = preds, traces
|
||||||
|
async def get_predictions(self, run_id, *, question_ids=None):
|
||||||
|
return [p for p in self._preds if not question_ids or p["question_id"] in question_ids]
|
||||||
|
async def get_traces(self, run_id, *, question_ids=None):
|
||||||
|
return list(self._traces)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_traces_falls_back_to_steps_json_when_table_empty():
|
||||||
|
steps = [{"thought": "t", "tool_call": {"tool": "view_node", "args": {}}, "tool_output": "o"}]
|
||||||
|
preds = [{"video_id": "v1", "question_id": "q1", "steps_json": json.dumps(steps)}]
|
||||||
|
log = StepsJsonRunLog(_FakeInner(preds, traces=[]))
|
||||||
|
rows = await log.get_traces("r", question_ids=["q1"])
|
||||||
|
assert rows[0]["tool_name"] == "view_node" and rows[0]["question_id"] == "q1"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_traces_prefers_nonempty_inner_table():
|
||||||
|
inner_traces = [{"video_id": "v1", "question_id": "q1", "step": 0, "tool_name": "x"}]
|
||||||
|
log = StepsJsonRunLog(_FakeInner([], inner_traces))
|
||||||
|
rows = await log.get_traces("r")
|
||||||
|
assert rows == inner_traces
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败** — `pytest tests/unit/test_baseline_run_log.py -v`,预期 `ModuleNotFoundError`。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 最小实现**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/harness/baseline_run_log.py
|
||||||
|
"""RunLog 包装器:traces 表空时从 predictions.steps_json 重建轨迹。
|
||||||
|
|
||||||
|
用于对 infer_adhoc 这类 traces 未落表、轨迹在 steps_json 的历史 run 跑离线诊断。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||||||
|
|
||||||
|
|
||||||
|
class StepsJsonRunLog:
|
||||||
|
"""委托内层 RunLog;get_traces 空表时回退 steps_json。"""
|
||||||
|
|
||||||
|
def __init__(self, inner: Any) -> None:
|
||||||
|
self._inner = inner
|
||||||
|
|
||||||
|
async def get_predictions(self, run_id: str, *, question_ids: list[str] | None = None) -> list[dict[str, Any]]:
|
||||||
|
return await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||||
|
|
||||||
|
async def get_traces(self, run_id: str, *, question_ids: list[str] | None = None) -> list[dict[str, Any]]:
|
||||||
|
inner_rows = await self._inner.get_traces(run_id, question_ids=question_ids)
|
||||||
|
if inner_rows:
|
||||||
|
return inner_rows
|
||||||
|
preds = await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for p in preds:
|
||||||
|
rows.extend(steps_json_to_trace_rows(p["video_id"], p["question_id"], p.get("steps_json") or ""))
|
||||||
|
return rows
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_baseline_run_log.py -v`,预期 PASS。
|
||||||
|
- [ ] **Step 5: 提交** — `git add app/harness/baseline_run_log.py tests/unit/test_baseline_run_log.py && git commit -m "feat: add steps_json-backed RunLog wrapper"`
|
||||||
|
|
||||||
|
### Task 3: baseline_diagnosis 持久化端口 + SQLite 适配
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/protocols.py`(新增 `DiagnosisSignalStore` Protocol)
|
||||||
|
- Create: `adapters/baseline_diagnosis_store.py`(SQLite 实现)
|
||||||
|
- Test: `tests/unit/test_baseline_diagnosis_store.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_baseline_diagnosis_store.py
|
||||||
|
import sqlite3
|
||||||
|
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore, DiagnosisSignalRow
|
||||||
|
|
||||||
|
|
||||||
|
def _store(tmp_path):
|
||||||
|
return SqliteDiagnosisSignalStore(str(tmp_path / "h.db"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_upsert_and_get_done_ids(tmp_path):
|
||||||
|
s = _store(tmp_path)
|
||||||
|
row = DiagnosisSignalRow(question_id="q1", video_id="v1", baseline_run_id="r", diag_fingerprint="fp",
|
||||||
|
task_type="Counting", error_type="search_failure", cause_category="defect",
|
||||||
|
tier="T2", evolution_target="skill", degraded=False, infra=False, session_id="s")
|
||||||
|
s.upsert(row)
|
||||||
|
assert s.done_question_ids("r", "fp") == {"q1"}
|
||||||
|
# upsert 同键覆盖,不重复
|
||||||
|
s.upsert(row)
|
||||||
|
assert s.done_question_ids("r", "fp") == {"q1"}
|
||||||
|
# 不同 fingerprint 隔离
|
||||||
|
assert s.done_question_ids("r", "other_fp") == set()
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_rows_roundtrip(tmp_path):
|
||||||
|
s = _store(tmp_path)
|
||||||
|
s.upsert(DiagnosisSignalRow("q2", "v2", "r", "fp", "OCR Problems", "extraction_failure",
|
||||||
|
"lapse", "T1", "tool", False, False, "s"))
|
||||||
|
rows = s.load("r", "fp")
|
||||||
|
assert len(rows) == 1 and rows[0].tier == "T1" and rows[0].evolution_target == "tool"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败** — `pytest tests/unit/test_baseline_diagnosis_store.py -v`,预期 `ModuleNotFoundError`。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现 Protocol + 适配**
|
||||||
|
|
||||||
|
在 `core/evolution/protocols.py` 追加:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class DiagnosisSignalStore(Protocol):
|
||||||
|
"""逐题诊断信号存储端口(隔离 SQLite,app/core 不写 SQL)。"""
|
||||||
|
|
||||||
|
def upsert(self, row: "DiagnosisSignalRow") -> None: ...
|
||||||
|
def done_question_ids(self, baseline_run_id: str, diag_fingerprint: str) -> set[str]: ...
|
||||||
|
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list["DiagnosisSignalRow"]: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
创建 `adapters/baseline_diagnosis_store.py`:`DiagnosisSignalRow` dataclass(字段对齐 Task 0 表);`SqliteDiagnosisSignalStore` 建表(PK `(question_id, baseline_run_id, diag_fingerprint)`)、`INSERT OR REPLACE` upsert、按 (run,fp) 查已完成集与全量。所有写在单事务提交(原子)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过** — `pytest tests/unit/test_baseline_diagnosis_store.py -v`,预期 PASS。
|
||||||
|
- [ ] **Step 5: 提交** — `git add core/evolution/protocols.py adapters/baseline_diagnosis_store.py tests/unit/test_baseline_diagnosis_store.py && git commit -m "feat: add baseline diagnosis signal store"`
|
||||||
|
|
||||||
|
### Task 4: 离线诊断编排
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/baseline_diagnosis.py`
|
||||||
|
- Test: `tests/integration/test_baseline_diagnosis.py`(LLM 类 → 产出 MD)
|
||||||
|
|
||||||
|
编排:算 `diag_fingerprint` → 查已完成集(续跑)→ 对剩余错题调 `run_diagnosis`(经 `StepsJsonRunLog`)→ 投影 `error_attributions` + INFRA + degraded 为 `DiagnosisSignalRow`(tier 由 Task 6 `score_signal`,此处先落 error_type/cause_category,tier 计算在 Task 6 引入后接入)→ upsert。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败集成测试**(mock LLM,真实 infer_adhoc 抽 3 题)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/integration/test_baseline_diagnosis.py
|
||||||
|
import pytest
|
||||||
|
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||||
|
from app.harness.baseline_diagnosis import run_baseline_diagnosis, DiagnosisDeps
|
||||||
|
|
||||||
|
|
||||||
|
def _deps(monkeypatch, calls):
|
||||||
|
async def fake_run_diagnosis(run_id, questions, tree_data, llm, run_log, skill_store, prompts,
|
||||||
|
*, concurrency, question_ids=None, task_types=None, only_incorrect=False):
|
||||||
|
calls.append(tuple(question_ids or []))
|
||||||
|
from core.evolution.types import DiagnosisResult, ErrorAttribution
|
||||||
|
return DiagnosisResult(run_id=run_id, error_attributions=[
|
||||||
|
ErrorAttribution("q1", "search_failure", None, "defect"),
|
||||||
|
ErrorAttribution("q2", "mixed", None, "lapse"),
|
||||||
|
], infra_question_ids=[], degraded_question_ids=[])
|
||||||
|
monkeypatch.setattr("app.harness.baseline_diagnosis.run_diagnosis", fake_run_diagnosis)
|
||||||
|
# 全部依赖显式 fake,无占位
|
||||||
|
return DiagnosisDeps(run_log=_FakeRunLog(), llm=_FakeLLM(), skill_store=_FakeSkillStore(),
|
||||||
|
prompts=_fake_prompts(), tree_data={}, concurrency=2)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_resume_skips_done(tmp_path, monkeypatch):
|
||||||
|
calls = []
|
||||||
|
deps = _deps(monkeypatch, calls)
|
||||||
|
store = SqliteDiagnosisSignalStore(str(tmp_path / "h.db"))
|
||||||
|
q_by_id = {"q1": _mk_q("q1"), "q2": _mk_q("q2")}
|
||||||
|
await run_baseline_diagnosis(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||||||
|
wrong_ids=["q1", "q2"], questions=q_by_id, store=store, deps=deps)
|
||||||
|
assert store.done_question_ids("infer_adhoc", "fp") == {"q1", "q2"}
|
||||||
|
await run_baseline_diagnosis(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||||||
|
wrong_ids=["q1", "q2"], questions=q_by_id, store=store, deps=deps)
|
||||||
|
assert calls[-1] == () # 第二次无剩余
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**,预期 `ModuleNotFoundError`。
|
||||||
|
- [ ] **Step 3: 实现** `run_baseline_diagnosis(baseline_run_id, diag_fingerprint, wrong_ids, questions, store, deps)`:`remaining = wrong_ids - store.done_question_ids(run,fp)`;空则返回;否则 `await run_diagnosis(baseline_run_id, [questions[i] for i in remaining], deps.tree_data, deps.llm, StepsJsonRunLog(deps.run_log), deps.skill_store, deps.prompts, concurrency=deps.concurrency, question_ids=list(remaining), only_incorrect=True)`;投影 `error_attributions`(tier 由 Task 6 `score_signal`,error_type/cause_category 落列)+ `infra_question_ids`(infra=1,tier=T0,error_type/evolution_target=NULL)+ `degraded_question_ids`(degraded=1,tier=uncertain)→ upsert。
|
||||||
|
> **错误处理实况(Codex I3)**:`run_diagnosis` 的 defect/lapse 判别处对 judge 异常是 `except Exception` 后 warning + 默认 lapse(`core/evolution/diagnose.py:2186-2192`),**非全传播**。本编排不谎称"全传播":网络/API 层失败经 GovernedLLMClient 重试栈后仍失败会向上抛;judge 语义歧义按现有保护性 lapse 处理并**计数上报**(`degraded_count`/默认 lapse 数写入 manifest)。产出 MD 到 `tests/outputs/`。
|
||||||
|
- [ ] **Step 4: 运行确认通过**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: add offline baseline diagnosis orchestration"`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2:signal 分层 + 视频聚合 + 贪心选择(纯函数)
|
||||||
|
|
||||||
|
### Task 5: evolution_target 派生 + cell 定义
|
||||||
|
|
||||||
|
**Files:** Create `app/harness/split_selection.py`(本 Task 起逐步充实);Test `tests/unit/test_split_selection.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_split_selection.py
|
||||||
|
from app.harness.split_selection import evolution_target_of, cell_of
|
||||||
|
|
||||||
|
|
||||||
|
def test_evolution_target_mapping():
|
||||||
|
assert evolution_target_of("extraction_failure") == "tool"
|
||||||
|
assert evolution_target_of("search_failure") == "skill"
|
||||||
|
assert evolution_target_of("reasoning_failure") == "skill"
|
||||||
|
assert evolution_target_of("mixed") == "system"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cell_is_task_type_x_error_type():
|
||||||
|
assert cell_of("Counting Problem", "search_failure") == ("Counting Problem", "search_failure")
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/harness/split_selection.py (片段)
|
||||||
|
_EVOLUTION_TARGET = {
|
||||||
|
"extraction_failure": "tool",
|
||||||
|
"search_failure": "skill",
|
||||||
|
"reasoning_failure": "skill",
|
||||||
|
"mixed": "system",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def evolution_target_of(error_type: str) -> str:
|
||||||
|
"""error_type → 进化目标(派生标注,非独立多样性轴)。"""
|
||||||
|
if error_type not in _EVOLUTION_TARGET:
|
||||||
|
raise ValueError(f"未知 error_type: {error_type}")
|
||||||
|
return _EVOLUTION_TARGET[error_type]
|
||||||
|
|
||||||
|
|
||||||
|
def cell_of(task_type: str, error_type: str) -> tuple[str, str]:
|
||||||
|
"""多样性主格子 = (task_type, error_type)。"""
|
||||||
|
return (task_type, error_type)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: add evolution_target derivation and cell"`
|
||||||
|
|
||||||
|
### Task 6: score_signal 分层
|
||||||
|
|
||||||
|
**Files:** Modify `app/harness/split_selection.py`;Test 同上文件追加
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_tiers():
|
||||||
|
from app.harness.split_selection import score_signal
|
||||||
|
assert score_signal(cause_category="defect", infra=False, degraded=False).tier == "T2"
|
||||||
|
assert score_signal(cause_category="lapse", infra=False, degraded=False).tier == "T1"
|
||||||
|
assert score_signal(cause_category="defect", infra=True, degraded=False).tier == "T0"
|
||||||
|
assert score_signal(cause_category=None, infra=False, degraded=True).tier == "uncertain"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** `score_signal(...) -> SignalLabel`(dataclass `tier: str`):优先级 INFRA→T0;degraded/诊断失败→uncertain;`cause_category=='defect'`→T2;`=='lapse'`→T1;其余→uncertain。**顺序固定,不用魔法权重**。
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: add signal tiering"`
|
||||||
|
|
||||||
|
### Task 7: 全视频记录构建(VideoRecord)+ 信号聚合
|
||||||
|
|
||||||
|
**Files:** Modify `app/harness/split_selection.py`;Test 追加
|
||||||
|
|
||||||
|
> **Codex 计划审 C4**:`select_split` 需**全 300 视频**的 type/difficulty/correctness 分布(含 125 个全对零信号视频)来算 ε 代表性与 test 补集,**不能只喂诊断行**。故先由全 900 predictions + questions 构建 `VideoRecord`,再叠加 T2 诊断信号。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**(真实数据二次构造:从 infer_adhoc 抽一个真实视频的 3 题结构)
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_build_video_records_covers_all_videos_with_difficulty_and_types():
|
||||||
|
from app.harness.split_selection import build_video_records
|
||||||
|
# 二次构造:真实 3 题/视频、真实 type、真实对错
|
||||||
|
preds = [
|
||||||
|
{"video_id": "v1", "question_id": "v1-1", "task_type": "Counting Problem", "correct": False},
|
||||||
|
{"video_id": "v1", "question_id": "v1-2", "task_type": "Action Reasoning", "correct": True},
|
||||||
|
{"video_id": "v1", "question_id": "v1-3", "task_type": "OCR Problems", "correct": True},
|
||||||
|
{"video_id": "v2", "question_id": "v2-1", "task_type": "Counting Problem", "correct": True},
|
||||||
|
{"video_id": "v2", "question_id": "v2-2", "task_type": "Counting Problem", "correct": True},
|
||||||
|
{"video_id": "v2", "question_id": "v2-3", "task_type": "Counting Problem", "correct": True},
|
||||||
|
]
|
||||||
|
signal_rows = [{"question_id": "v1-1", "task_type": "Counting Problem",
|
||||||
|
"error_type": "search_failure", "tier": "T2"}]
|
||||||
|
recs = build_video_records(preds, signal_rows)
|
||||||
|
assert {r.video_id for r in recs} == {"v1", "v2"} # 全视频(含零信号 v2)
|
||||||
|
v1 = next(r for r in recs if r.video_id == "v1")
|
||||||
|
v2 = next(r for r in recs if r.video_id == "v2")
|
||||||
|
assert v1.n_correct == 2 and v1.difficulty == 1 # 3题对2 → 难度画像桶=1错
|
||||||
|
assert v2.difficulty == 0 and v2.cells == set() # 零信号视频进 test 无 T2 格子
|
||||||
|
assert v1.cells == {("Counting Problem", "search_failure")}
|
||||||
|
assert v1.type_set == {"Counting Problem", "Action Reasoning", "OCR Problems"}
|
||||||
|
assert v1.wrong_by_type == {"Counting Problem": 1} # T2 计数供 floor
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** `build_video_records(preds, signal_rows) -> list[VideoRecord]`:按 video_id 聚合全 900 题 → `VideoRecord(video_id, type_set, n_correct, difficulty=3-n_correct 的错题数桶, cells:set(仅 T2 题 cell_of 并集去重), wrong_by_type:dict(各type T2 数))`。零信号视频 cells 空、仍在列表(供 test 与 ε)。
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: build all-video records with difficulty and signal overlay"`
|
||||||
|
|
||||||
|
### Task 8: 贪心联合约束选择器
|
||||||
|
|
||||||
|
**Files:** Modify `app/harness/split_selection.py`;Test 追加
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**(输入为 Task 7 的 `VideoRecord`;fixture 用真实 infer_adhoc 各 type 分布二次构造,非纯虚构——满足 §4.6)
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _real_shaped_video_records():
|
||||||
|
# 从 workspaces/default/harness.db infer_adhoc 抽真实(video,3题type,correct)二次构造
|
||||||
|
# 保留真实类型长尾(Counting/AR/OCR...)与 0/1/2/3-对难度画像分布
|
||||||
|
... # helper:读真实 DB → build_video_records;见 conftest 提供的 fixture
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_split_video_disjoint_and_floor_and_deterministic():
|
||||||
|
from app.harness.split_selection import select_split, SelectConfig, derive_reportable_types
|
||||||
|
videos = _real_shaped_video_records()
|
||||||
|
total_by_type = _count_questions_by_type(videos)
|
||||||
|
cfg = SelectConfig(n_trainval=100, floor_k={"Counting Problem": 3}, epsilon=0.1,
|
||||||
|
reportable_types=derive_reportable_types(total_by_type, report_floor=27),
|
||||||
|
seed=7)
|
||||||
|
a = select_split(videos, config=cfg)
|
||||||
|
b = select_split(videos, config=cfg)
|
||||||
|
assert set(a.trainval) & set(a.test) == set() # 互斥
|
||||||
|
assert set(a.trainval) | set(a.test) == {v.video_id for v in videos}
|
||||||
|
assert a.trainval == b.trainval # 同 seed 同解
|
||||||
|
assert sum(v.wrong_by_type.get("Counting Problem", 0) # floor 达标
|
||||||
|
for v in videos if v.video_id in a.trainval) >= 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_infeasible_floor_vs_epsilon_raises():
|
||||||
|
from app.harness.split_selection import select_split, SelectConfig, InfeasibleSplitError
|
||||||
|
import pytest
|
||||||
|
videos = _real_shaped_video_records()
|
||||||
|
with pytest.raises(InfeasibleSplitError): # 极小 ε + 高 floor → 死锁
|
||||||
|
select_split(videos, config=SelectConfig(n_trainval=2, floor_k={"OCR Problems": 50},
|
||||||
|
epsilon=0.001, reportable_types=set(), seed=1))
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** `select_split(videos: list[VideoRecord], config) -> SplitAssignment(trainval, test)`(全局统计从 `videos` 内部算,无需外传):按设计 §6 两阶段贪心——floor 阶段与多样性阶段**均带 ε 可行性检查**(移入 trainval 后 test=补集 仍满足 per-type 比例 + 0/1/2/3-对难度画像 ±ε);floor 死锁/欠额抛 `InfeasibleSplitError`。**确定性**:先按 `seed` 对候选做一次固定预洗牌,再按 `-边际增益` 稳定排序(seed 只控预洗牌打破等增益平局,非二次 key)。`_epsilon_ok(test_videos, ε)` 校验 reportable 类型比例与难度画像偏差 ≤ ε。同文件加 `derive_reportable_types(total_by_type, report_floor) -> set[str]`(总题数 ≥ report_floor 的类型,落地设计 §7)。
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: add greedy joint-constrained split selector"`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3:pools 视频原子 + 冻结
|
||||||
|
|
||||||
|
### Task 9: pools 切分原子 unit→video
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py`(`build_pools`/`_split_one_category`/`build_or_load_pools`)
|
||||||
|
- Test: `tests/unit/test_pools_video_atomic.py` + 回归 `tests/unit/test_run_store.py`(若涉及)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_pools_video_atomic.py
|
||||||
|
from app.harness.pools import split_by_video_assignment
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
|
||||||
|
def _q(qid, vid, tt="Counting Problem"):
|
||||||
|
return GeneratedQuestion(question_id=qid, video_id=vid, task_type=tt, question="", options=("A","B","C","D"), answer="A")
|
||||||
|
|
||||||
|
|
||||||
|
def test_video_never_split_across_pools():
|
||||||
|
qs = [_q("v1-1","v1"), _q("v1-2","v1"), _q("v1-3","v1"), _q("v2-1","v2")]
|
||||||
|
assignment = {"v1": "trainval", "v2": "test"}
|
||||||
|
pools = split_by_video_assignment(qs, assignment, correctness={q.question_id: True for q in qs},
|
||||||
|
val_ratio=0.0, seed=0)
|
||||||
|
test_vids = {q.video_id for q in pools.test}
|
||||||
|
train_vids = {q.video_id for q in pools.diagnosis + pools.validation}
|
||||||
|
assert test_vids & train_vids == set() # 视频不跨池
|
||||||
|
assert test_vids == {"v2"} and train_vids == {"v1"}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** 新增 `split_by_video_assignment(questions, assignment, correctness, val_ratio, seed) -> Pools`:先按 assignment 把题分到 trainval/test(video 原子);trainval 内用**视频组**做 correctness 分层切 train(diagnosis)/val——改 `_split_one_category` 采样原子为 video 组(同 video 全 unit 同进同出,复用现有 correctness 分层 + `eval_min_per_class`);test 直接为 test 视频全部题。保留 `baseline_val_accuracy`/`correctness` 计算。
|
||||||
|
- [ ] **Step 4: 通过确认** + **保真检查点**:断言 `pools.diagnosis` 内仍是逐 unit 列表(`gate_ladder` 输入不变)。运行 `pytest tests/unit/test_pools*.py tests/unit/test_gate.py -v`。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "refactor: add video-atomic pool split (algo #5 gate input preserved)"`
|
||||||
|
|
||||||
|
### Task 10: 原子冻结 + manifest
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py`(`save_pools` 与 per_category 写路径抽共用原子助手)
|
||||||
|
- Create: `app/harness/split_manifest.py`
|
||||||
|
- Test: `tests/unit/test_atomic_save_pools.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_atomic_save_pools.py
|
||||||
|
import json
|
||||||
|
from app.harness.pools import _atomic_write_json # 新共用助手
|
||||||
|
|
||||||
|
|
||||||
|
def test_atomic_write_replaces_and_no_tmp_left(tmp_path):
|
||||||
|
p = tmp_path / "pools.json"
|
||||||
|
_atomic_write_json(p, {"a": 1})
|
||||||
|
assert json.loads(p.read_text())["a"] == 1
|
||||||
|
assert list(tmp_path.glob("*.tmp")) == [] # 无残留 tmp
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** `_atomic_write_json(path, obj)`:写 `path.with_suffix(".tmp")` 后 `os.replace`;`save_pools`(341) 与 per_category 写(566) 均改调它。`split_manifest.py::write_manifest(...)` 写 baseline_run_id/diag_fingerprint/seed/config/pools_sha256/coverage_report。
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "fix: make pools.json freeze atomic + add split manifest"`
|
||||||
|
|
||||||
|
### Task 11: 端到端集成 + 防御断言
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/build_split.py`(顶层编排:诊断结果 → signal → 视频聚合 → select_split → split_by_video_assignment → 冻结 + manifest)
|
||||||
|
- Test: `tests/integration/test_build_split_e2e.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败集成测试**(用真实 infer_adhoc 900 预测 + Task 3 缓存诊断表;若无缓存则 mock 诊断行)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/integration/test_build_split_e2e.py
|
||||||
|
import pytest
|
||||||
|
from app.harness.build_split import build_split
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_end_to_end_freezes_valid_pools(tmp_path):
|
||||||
|
out = tmp_path / "pools.json"
|
||||||
|
result = await build_split(baseline_run_id="infer_adhoc", diag_fingerprint="fp",
|
||||||
|
harness_db="workspaces/default/harness.db",
|
||||||
|
signal_store=_cached_or_mock_store(), out_path=out, config=_calibrated_cfg())
|
||||||
|
pools = result.pools
|
||||||
|
# 防御断言①: 三池视频互斥
|
||||||
|
tv = {q.video_id for q in pools.diagnosis + pools.validation}
|
||||||
|
te = {q.video_id for q in pools.test}
|
||||||
|
assert tv & te == set()
|
||||||
|
# 防御断言②: 覆盖全 900 题
|
||||||
|
assert len(pools.diagnosis) + len(pools.validation) + len(pools.test) == 900
|
||||||
|
# manifest 存在且 pools_sha256 校验一致
|
||||||
|
assert result.manifest["pools_sha256"] == _sha256(out.read_text())
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** `build_split(...)`:从 signal_store.load(run,fp) 取逐题 tier/cell → `aggregate_video_signal` 逐视频 → `select_split` → `split_by_video_assignment` → `save_pools`(原子) + `write_manifest`。落地设计 §10 防御断言清单①②③④⑤⑥(三池互斥/900 覆盖/每视频 3 题/fingerprint 一致/manifest hash/ID 唯一),任一不满足 fail-fast。
|
||||||
|
- [ ] **Step 4: 通过确认** — `pytest tests/integration/test_build_split_e2e.py -v`。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: wire end-to-end results-driven split with defensive asserts"`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4:配置与标定
|
||||||
|
|
||||||
|
### Task 12: 科研 YAML 旋钮 + 标定程序
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/types.py`(`PoolConfig` 加 `n_trainval/floor_k/epsilon/report_floor/val_wrong_min`)
|
||||||
|
- Create: `config/video_split.yaml`
|
||||||
|
- Create: `scripts/build_video_split.sh`
|
||||||
|
- Test: `tests/unit/test_pool_config_video_split.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试** — ① `PoolConfig` 可从 `config/video_split.yaml` 载入新字段且缺失关键项 fail-fast(不兜底默认);② `diag_fingerprint(prompt_version, model, code_version)` 确定性、任一输入变则变;③ `val_wrong_min` 进入 `split_by_video_assignment` 可行性:val 错题数 < `val_wrong_min` 时抛错。
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_diag_fingerprint_deterministic_and_sensitive():
|
||||||
|
from app.harness.split_selection import diag_fingerprint
|
||||||
|
a = diag_fingerprint("p1", "deepseek-v4", "abc123")
|
||||||
|
assert a == diag_fingerprint("p1", "deepseek-v4", "abc123") # 确定性
|
||||||
|
assert a != diag_fingerprint("p2", "deepseek-v4", "abc123") # prompt 变则变
|
||||||
|
assert a != diag_fingerprint("p1", "kimi", "abc123") # model 变则变
|
||||||
|
|
||||||
|
|
||||||
|
def test_val_wrong_min_enforced(tmp_path):
|
||||||
|
from app.harness.pools import split_by_video_assignment, InsufficientValSignal
|
||||||
|
import pytest
|
||||||
|
with pytest.raises(InsufficientValSignal): # val 错题不足功效阈
|
||||||
|
split_by_video_assignment(_all_correct_qs(), {"v": "trainval"},
|
||||||
|
correctness=_all_true(), val_ratio=0.5, seed=0, val_wrong_min=5)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 失败确认**。
|
||||||
|
- [ ] **Step 3: 实现** 扩 `PoolConfig`(`n_trainval/floor_k/epsilon/report_floor/val_wrong_min`);`diag_fingerprint(...)= hashlib.sha256("|".join(...)).hexdigest()[:16]`(来源:诊断 prompt 文件 hash + `.env` 模型名 + `git rev-parse HEAD` 短 SHA,`--force` 用**新 fingerprint** 写、不覆盖旧记录);`split_by_video_assignment` 加 `val_wrong_min` 参数,val 切出后校验错题数 ≥ 阈值否则 `InsufficientValSignal`;写 `config/video_split.yaml`(占位阈值 + 注释"诊断后标定");`scripts/build_video_split.sh` 自包含零参复现(GPU 卡号除外)。**标定程序**:诊断跑完读 `baseline_diagnosis` 各 type T2 数 → 定 `floor_k`(如 min(可用defect,3))、`n_trainval`(~100)、`epsilon`(如 0.1)、`val_wrong_min`(McNemar 功效阈,如 ≥20)。
|
||||||
|
- [ ] **Step 4: 通过确认**。
|
||||||
|
- [ ] **Step 5: 提交** — `git commit -m "feat: add video-split config knobs and reproducible script"`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review(作者自查,已执行)
|
||||||
|
|
||||||
|
- **Spec 覆盖**:设计 §4 组件→Task 1-12 一一对应;§5 signal/多样性→Task 5-7;§6 贪心→Task 8;§7 长尾→Task 8/12(reportable_types+report_floor);§8 val→Task 9(val_ratio+min_per_class)+Task 12(val_wrong_min);§9 非功能→Task 3(逐题续跑/upsert)、Task 10(原子写);§10 错误→Task 4(P5 传播)、Task 8(InfeasibleSplitError)、Task 11(防御断言);§12 前序继承→Task 9(三池/pair/baseline_val_accuracy)+Task 10(per_category 原子写)。
|
||||||
|
- **占位扫描**:无 TBD/TODO;阈值项在 Task 12 明确"诊断后标定"并给标定程序,非占位。
|
||||||
|
- **类型一致**:`DiagnosisSignalRow`/`DiagnosisDeps`/`VideoRecord`/`SignalLabel`/`SelectConfig`/`SplitAssignment`/`InfeasibleSplitError`/`InsufficientValSignal`/`derive_reportable_types`/`diag_fingerprint`/`_atomic_write_json`/`split_by_video_assignment`/`build_video_records` 跨 Task 命名一致。
|
||||||
|
- **Codex 计划审修订已并入**:C1 字段名 `tool`(Task 1/2)、C2 canonical 事实(900/660/236错/4 INFRA-null,按 distinct question 断言)、C3 Task 4 去占位显式 fake、C4 全视频 `VideoRecord` 模型(Task 7);I1 val_wrong_min 接回切分(Task 12)、I2 `derive_reportable_types`(Task 8)、I3 诚实标注 run_diagnosis 错误处理、I4 schema NULL 规则(Task 0)、I5 真实数据二次构造 fixture、I6 `diag_fingerprint` 函数+测试;M1 锚点 809、M2 保真点 Task 9、M3 seed 只控预洗牌。
|
||||||
|
|
||||||
|
## 里程碑
|
||||||
|
|
||||||
|
| 里程碑 | 完成 Task |
|
||||||
|
|--------|----------|
|
||||||
|
| M1 诊断信号就绪(236 可诊断错题分层 + 4 INFRA-null 落 T0) | 0-4 |
|
||||||
|
| M2 选择器可产合法切分(纯函数全绿) | 5-8 |
|
||||||
|
| M3 冻结 pools.json + manifest 端到端 | 9-11 |
|
||||||
|
| M4 可复现脚本 + 标定 | 12 |
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
|
||||||
|
- 冻结 `pools.json`:三池视频互斥、覆盖 900 题、floor 达标、test 代表性偏差 ≤ ε;manifest hash 自洽。
|
||||||
|
- 全测试绿(unit + integration);LLM 类测试产出 MD。
|
||||||
|
- `bash scripts/build_video_split.sh` 零参复现(GPU 卡号除外)。
|
||||||
|
- gate_ladder 冷启动 2:1 输入不变(保真检查点通过)。
|
||||||
|
|
||||||
|
## 风险
|
||||||
|
|
||||||
|
| 风险 | 缓解 |
|
||||||
|
|------|------|
|
||||||
|
| steps_json 部分为空(3/902)或字段异常 | Task 1 空返回 [];诊断对应题落 uncertain,计数上报 |
|
||||||
|
| 100 训练视频信号不足 | 切设计 §14 方案 B(外部 benchmark 训练) |
|
||||||
|
| floor 与 ε 联合不可行 | Task 8 InfeasibleSplitError fail loud,Task 12 放松旋钮 |
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,253 @@
|
|||||||
|
# WP1 资产迁移 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 迁移 5 个遗漏的 TRM4 进化/动量模板到 TRM5,删除 2 个死字段,把加载器的静默空串兜底改为 fail-loud,解锁自进化引擎。
|
||||||
|
|
||||||
|
**Architecture:** 进化引导 prompt 是引擎的一部分(放项目根 `prompts/`,不参与版本化进化)。TRM4 五模板的输出 JSON 契约与 TRM5 解析代码已核实完全对齐,可直接拷贝。执行顺序:先迁移模板 → 删死字段(`consolidate_system`/`span_eval_user`,零消费)→ 加载器 fail-loud(顺序关键:先删死字段,fail-loud 才不会对不存在也不需要的模板报错)。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、pytest、frozen dataclass(`core/evolution/types.py`)。
|
||||||
|
|
||||||
|
**设计源**:`research-wiki/designs/2026-07-16-preflight-fixes-design.md §4`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键锚点(实现前必读)
|
||||||
|
|
||||||
|
| 用途 | 位置 |
|
||||||
|
|------|------|
|
||||||
|
| TRM4 源模板 | `/home/iomgaa/Projects/Video-Tree-TRM4/prompts/{evolve_skill,evolve_system,evolve_tool,evolve_rank,slow_momentum}.md` |
|
||||||
|
| evolve 加载器 | `app/harness/runner.py:2266-2280` `_load_evolve_prompts` |
|
||||||
|
| diagnose 加载器 | `app/harness/runner.py:2282-2299` `_load_diagnose_prompts` |
|
||||||
|
| 平行 diagnose 加载器 | `app/harness/video_split_cli.py:362-378` `_load_diagnose_prompts` |
|
||||||
|
| dataclass 定义 | `core/evolution/types.py:477-522`(`DiagnosePrompts` L494-501 / `EvolvePrompts` L518-522) |
|
||||||
|
| 死字段 `consolidate_system` 内联替代 | `core/evolution/evolve.py:997` `_CONSOLIDATE_SYSTEM`(消费点 L1036) |
|
||||||
|
| 死字段 `span_eval_user` 无消费 | diagnose 只用 `prompts.span_eval_system`(`core/evolution/diagnose.py:511`),user_prompt 内联构造 |
|
||||||
|
| 测试 fixture | `test_evolve.py:682` `consolidate_system="cons"`;`test_diagnose.py:753` `span_eval_user=""`;`test_evolution_types.py:352` `span_eval_user="p4"` / `:370` `consolidate_system="consolidate_tmpl"` |
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
本计划不迁移/不改核心算法(ARCHITECTURE §6),只搬运模板文件 + 清理死字段 + 加 fail-loud。模板内容是 evolve 引擎(算法 #8)的输入数据,非算法逻辑本身;迁移已核实输出契约(`suggestions`/`edits`/`edits_extract`/`edits_verify`/`selected_indices`/`slow_update_content`)与 TRM5 解析代码逐字对齐。**保真检查点(Task 1 Step 4)**:加载 evolve_tool.md 后确认其要求 LLM 返回 `edits_extract`+`edits_verify` 双键(对齐 `evolve.py:1433-1435`)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: 迁移 5 个 TRM4 模板
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `prompts/evolve_skill.md`、`prompts/evolve_system.md`、`prompts/evolve_tool.md`、`prompts/evolve_rank.md`、`prompts/slow_momentum.md`(从 TRM4 拷贝)
|
||||||
|
- Test: `tests/unit/test_evolve_prompts_present.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写契约冒烟测试(先失败)**
|
||||||
|
|
||||||
|
`tests/unit/test_evolve_prompts_present.py`:
|
||||||
|
```python
|
||||||
|
"""校验 5 个进化/动量模板存在且输出契约关键词与解析代码对齐。"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
_PROMPTS_DIR = Path("prompts")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"name, required_tokens",
|
||||||
|
[
|
||||||
|
("evolve_skill.md", ["suggestions", "edits"]),
|
||||||
|
("evolve_system.md", ["suggestions", "edits"]),
|
||||||
|
("evolve_tool.md", ["edits_extract", "edits_verify"]),
|
||||||
|
("evolve_rank.md", ["selected_indices"]),
|
||||||
|
("slow_momentum.md", ["slow_update_content"]),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_evolve_template_present_and_contract(name: str, required_tokens: list[str]) -> None:
|
||||||
|
path = _PROMPTS_DIR / name
|
||||||
|
assert path.exists(), f"缺模板: {path}"
|
||||||
|
text = path.read_text(encoding="utf-8")
|
||||||
|
assert text.strip(), f"模板为空: {path}"
|
||||||
|
for token in required_tokens:
|
||||||
|
assert token in text, f"{name} 缺输出契约关键词 {token!r}(与解析代码不对齐)"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败(模板尚未迁移)**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_evolve_prompts_present.py -v`
|
||||||
|
Expected: 5 参数化用例全 FAIL(`AssertionError: 缺模板: prompts/evolve_skill.md` 等;TRM5 当前无这 5 个模板)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 拷贝 5 个模板**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cp /home/iomgaa/Projects/Video-Tree-TRM4/prompts/evolve_skill.md prompts/evolve_skill.md
|
||||||
|
cp /home/iomgaa/Projects/Video-Tree-TRM4/prompts/evolve_system.md prompts/evolve_system.md
|
||||||
|
cp /home/iomgaa/Projects/Video-Tree-TRM4/prompts/evolve_tool.md prompts/evolve_tool.md
|
||||||
|
cp /home/iomgaa/Projects/Video-Tree-TRM4/prompts/evolve_rank.md prompts/evolve_rank.md
|
||||||
|
cp /home/iomgaa/Projects/Video-Tree-TRM4/prompts/slow_momentum.md prompts/slow_momentum.md
|
||||||
|
```
|
||||||
|
Expected: 5 文件存在于 `prompts/`。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_evolve_prompts_present.py -v`
|
||||||
|
Expected: 5 参数化用例全 PASS。若 evolve_tool.md 缺 `edits_extract`/`edits_verify` 则契约不符——停止并逐行比对 TRM4 源。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 保真检查点**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -c "print('edits_extract' in open('prompts/evolve_tool.md').read() and 'edits_verify' in open('prompts/evolve_tool.md').read())"`
|
||||||
|
Expected: `True`(对齐 `evolve.py:1433-1435` 的 `parsed["edits_extract"]`/`parsed["edits_verify"]`)。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add prompts/evolve_skill.md prompts/evolve_system.md prompts/evolve_tool.md prompts/evolve_rank.md prompts/slow_momentum.md tests/unit/test_evolve_prompts_present.py
|
||||||
|
git commit -m "feat: migrate 5 evolve/momentum templates from TRM4 (algo #8)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: 删除 2 个死字段(consolidate_system / span_eval_user)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/types.py:494-501,518-522`
|
||||||
|
- Modify: `app/harness/runner.py:2279,2294`
|
||||||
|
- Modify: `app/harness/video_split_cli.py:374`
|
||||||
|
- Modify: `tests/unit/test_evolve.py:682`、`tests/unit/test_diagnose.py:753`、`tests/unit/test_evolution_types.py:352,370`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 删 dataclass 字段与 docstring**
|
||||||
|
|
||||||
|
`core/evolution/types.py` — `DiagnosePrompts` 删 `span_eval_user`:
|
||||||
|
- docstring 删行 ` span_eval_user: span 评估用户提示模板。`(L487)
|
||||||
|
- 字段删行 ` span_eval_user: str`(L497)
|
||||||
|
|
||||||
|
`EvolvePrompts` 删 `consolidate_system`:
|
||||||
|
- docstring 删行 ` consolidate_system: appendix 压缩系统提示。`(L515)
|
||||||
|
- 字段删行 ` consolidate_system: str`(L522)
|
||||||
|
|
||||||
|
- [ ] **Step 2: 删加载器对死字段的 `_read` 行**
|
||||||
|
|
||||||
|
`app/harness/runner.py`:
|
||||||
|
- `_load_evolve_prompts` 删行 ` consolidate_system=_read("consolidate_system.md"),`(L2279)
|
||||||
|
- `_load_diagnose_prompts` 删行 ` span_eval_user=_read("span_eval_user.md"),`(L2294)
|
||||||
|
|
||||||
|
`app/harness/video_split_cli.py`:
|
||||||
|
- `_load_diagnose_prompts` 删行 ` span_eval_user=_read("span_eval_user.md"),`(L374)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 删测试 fixture 对死字段的赋值 + 更新 docstring**
|
||||||
|
|
||||||
|
- `tests/unit/test_evolve.py:684` 删行 ` consolidate_system="cons",`
|
||||||
|
- `tests/unit/test_diagnose.py:753` 删行 ` span_eval_user="",`
|
||||||
|
- `tests/unit/test_evolution_types.py:352` 删行 ` span_eval_user="p4",`
|
||||||
|
- `tests/unit/test_evolution_types.py:370` 删行 ` consolidate_system="consolidate_tmpl",`
|
||||||
|
- `tests/unit/test_evolution_types.py:347` 的文档字符串"DiagnosePrompts 8 个模板字段"改为"7 个";`:364` 的"EvolvePrompts 5 个模板字段"改为"4 个"(删字段后数量变化)。
|
||||||
|
|
||||||
|
注:`test_evolution_types.py` 若有断言逐字段比对或字段计数,同步移除对两个死字段的断言(读该测试确认,删净引用)。上述行号以当前代码为准,实现前 `grep -n consolidate_system\|span_eval_user tests/unit/test_evolution_types.py` 复核。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行相关测试确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_evolve.py tests/unit/test_diagnose.py tests/unit/test_evolution_types.py -q`
|
||||||
|
Expected: 全 PASS(无 `TypeError: unexpected keyword argument` / 无 `missing positional argument`)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 全库确认无残留引用**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -c "import subprocess; r=subprocess.run(['grep','-rn','consolidate_system\|span_eval_user','core/','app/','adapters/','tests/'],capture_output=True,text=True); print(r.stdout)"`
|
||||||
|
Expected: 空输出(`consolidate_appendix` 用内联 `_CONSOLIDATE_SYSTEM` 不算 `consolidate_system` 字段引用;若出现请确认非 dataclass 字段引用)。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/evolution/types.py app/harness/runner.py app/harness/video_split_cli.py tests/unit/test_evolve.py tests/unit/test_diagnose.py tests/unit/test_evolution_types.py
|
||||||
|
git commit -m "refactor: drop dead prompt fields consolidate_system/span_eval_user"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: 加载器 fail-loud(缺模板即报错)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:2270-2272,2286-2288`
|
||||||
|
- Modify: `app/harness/video_split_cli.py:365-367`
|
||||||
|
- Test: `tests/unit/test_evolve_prompts_present.py`(追加)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 追加 fail-loud 测试(直接调真实加载器)**
|
||||||
|
|
||||||
|
在 `tests/unit/test_evolve_prompts_present.py` 追加——直接驱动真实 loader(不复制 _read 逻辑),在无模板的空 cwd 下断言 `FileNotFoundError`:
|
||||||
|
```python
|
||||||
|
def test_video_split_loader_fail_loud_on_missing(tmp_path, monkeypatch):
|
||||||
|
"""video_split_cli 的真实 diagnose 加载器缺模板必须 FileNotFoundError。"""
|
||||||
|
monkeypatch.chdir(tmp_path) # 空目录,无 prompts/*.md
|
||||||
|
from app.harness.video_split_cli import _load_diagnose_prompts
|
||||||
|
|
||||||
|
with pytest.raises(FileNotFoundError, match="缺进化/诊断模板"):
|
||||||
|
_load_diagnose_prompts()
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_evolve_loader_fail_loud_on_missing(tmp_path, monkeypatch):
|
||||||
|
"""runner 的真实 evolve 加载器缺模板必须 FileNotFoundError。"""
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
from app.harness.runner import Runner
|
||||||
|
|
||||||
|
r = object.__new__(Runner) # 绕过 __init__,仅测无状态加载器方法
|
||||||
|
with pytest.raises(FileNotFoundError, match="缺进化/诊断模板"):
|
||||||
|
r._load_evolve_prompts()
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_diagnose_loader_fail_loud_on_missing(tmp_path, monkeypatch):
|
||||||
|
"""runner 的真实 diagnose 加载器缺模板必须 FileNotFoundError。"""
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
from app.harness.runner import Runner
|
||||||
|
|
||||||
|
r = object.__new__(Runner)
|
||||||
|
with pytest.raises(FileNotFoundError, match="缺进化/诊断模板"):
|
||||||
|
r._load_diagnose_prompts()
|
||||||
|
```
|
||||||
|
> 这三个测试直接调真实 loader(`_load_evolve_prompts`/`_load_diagnose_prompts` 无 self 状态依赖,`object.__new__` 可安全调用);修复前静默返回空串不抛,故 Step 2 前必 FAIL。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 三处 `_read` 闭包改 fail-loud**
|
||||||
|
|
||||||
|
`app/harness/runner.py` `_load_evolve_prompts`(L2270-2272)与 `_load_diagnose_prompts`(L2286-2288),以及 `app/harness/video_split_cli.py` `_load_diagnose_prompts`(L365-367),把:
|
||||||
|
```python
|
||||||
|
def _read(name: str) -> str:
|
||||||
|
p = Path("prompts") / name
|
||||||
|
return p.read_text(encoding="utf-8") if p.exists() else ""
|
||||||
|
```
|
||||||
|
改为:
|
||||||
|
```python
|
||||||
|
def _read(name: str) -> str:
|
||||||
|
p = Path("prompts") / name
|
||||||
|
if not p.exists():
|
||||||
|
raise FileNotFoundError(f"缺进化/诊断模板: {p}(请从 TRM4 迁移或检查 prompts/)")
|
||||||
|
return p.read_text(encoding="utf-8")
|
||||||
|
```
|
||||||
|
(runner.py 缩进 12 空格;video_split_cli.py 的 `_read` 缩进按其函数体,见 L365 为 8 空格——按各自现场缩进套用。)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 运行测试确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_evolve_prompts_present.py -v`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 4: runner 其他行为回归(非 fail-loud 验证)**
|
||||||
|
|
||||||
|
fail-loud 已由 Step 3 的三个真实 loader 测试验证;此步仅确认模板迁移 + loader 改动未破坏 runner 其他行为。
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -q`
|
||||||
|
Expected: 全 PASS(模板已迁移,真实加载走成功分支)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py app/harness/video_split_cli.py tests/unit/test_evolve_prompts_present.py
|
||||||
|
git commit -m "fix: fail-loud on missing evolve/diagnose templates (no silent empty)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review(作者自查,执行者复核)
|
||||||
|
|
||||||
|
- [ ] 5 模板均已迁移且契约测试覆盖关键字段。
|
||||||
|
- [ ] `consolidate_system`/`span_eval_user` 在 core/app/tests 全库无残留字段引用。
|
||||||
|
- [ ] 三处 loader(runner 两处 + video_split_cli 一处)均已 fail-loud。
|
||||||
|
- [ ] 执行顺序正确:Task 2(删死字段)先于 Task 3(fail-loud),避免对不需要的模板报错。
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
|
||||||
|
1. `pytest tests/unit/test_evolve_prompts_present.py tests/unit/test_evolve.py tests/unit/test_diagnose.py tests/unit/test_evolution_types.py tests/unit/test_harness_runner.py` 全绿。
|
||||||
|
2. `grep -rn 'consolidate_system\|span_eval_user' core/ app/ tests/` 无 dataclass 字段残留。
|
||||||
|
3. `prompts/` 下 5 个新模板存在且非空。
|
||||||
@@ -0,0 +1,627 @@
|
|||||||
|
# WP2 切分与接线 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 让冻结的 video-split 切分能正确进入训练 workspace(seed 携带 pools.json + global 一致性校验),并把切分质量三处优化(val_ratio 0.4、tier 感知 diag/val 分配、val 功效修复)与冻结产物覆盖保护落地。
|
||||||
|
|
||||||
|
**Architecture:** 切分产物由 `video_split_cli` 冻结到 `workspaces/video-split/`;本 WP 让 seed 携带该产物、训练 fresh 时拷入 workspace 并校验一致性。tier 感知在 `_split_trainval_by_video_group` 内实现——错题视频组按 T2(defect) 含量升序进 val(保留 T2 高的组在 diag),并把 `val_wrong_min` 前置到切分内做功效修复(不足则从 diag 换出低 T2 错题组补 val,耗尽 fail-loud)。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、pytest、SQLite、frozen dataclass、shutil、原子写(tmp+os.replace)。
|
||||||
|
|
||||||
|
**设计源**:`research-wiki/designs/2026-07-16-preflight-fixes-design.md §5`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键锚点(实现前必读)
|
||||||
|
|
||||||
|
| 用途 | 位置 |
|
||||||
|
|------|------|
|
||||||
|
| trainval→diag/val 切分 | `app/harness/pools.py:126-199` `split_by_video_assignment`;`:289-338` `_split_trainval_by_video_group`;`:118` `InsufficientValSignal` |
|
||||||
|
| global 加载(无校验) | `app/harness/pools.py:713-813` `build_or_load_pools`(L813 `return load_pools` 前无 global 校验);`:587-620` `load_pools` |
|
||||||
|
| 冻结编排 | `app/harness/build_split.py:104-219` `build_split`(signal_rows 含 tier L154;split_by_video_assignment 调用 L184-191;save_pools L192);`:46-71` `SplitBuildConfig`(无 val_wrong_min) |
|
||||||
|
| CLI 构造 | `app/harness/video_split_cli.py:559-577` `SplitBuildConfig(...)`;`:580` `check_mcnemar_power`;`:751-773` `build_arg_parser`(无 --force) |
|
||||||
|
| seed | `app/harness/store.py:184-232` `init_seed`(拷 skills/prompts/baseline.db,不拷 pools);`:269-307` `extract_run_db`(不去重) |
|
||||||
|
| workspace | `app/harness/workspace.py:156-200` `init_workspace_from_seed`(copy2 baseline.db→harness.db L197,不拷 pools) |
|
||||||
|
| manifest | `app/harness/split_manifest.py:19-59` `write_manifest`(pools_sha256 L54) |
|
||||||
|
| 配置 | `config/video_split.yaml`(val_ratio L15=0.3、val_wrong_min L14=20) |
|
||||||
|
| 测试 | `tests/unit/test_pools_video_atomic.py`、`test_split_selection.py`、`test_harness_pools.py`、`test_harness_store.py`、`test_harness_workspace.py` |
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
触及算法 #5(信息阶梯)的**上游输入**:本 WP 只改"哪些视频进 diag/val",pools 内仍是逐 unit 列表,`gate_ladder` 消费的 unit+correctness 结构不变。**保真检查点(Task 3 Step 6)**:确认 `_split_trainval_by_video_group` 返回后 diagnosis/validation 仍是逐题 `GeneratedQuestion` 列表、视频组原子性(同 video 全部题同池)不被 tier 排序破坏。不改算法 #6/#9。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: val_ratio 0.3 → 0.4
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `config/video_split.yaml:15`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 改配置**
|
||||||
|
|
||||||
|
`config/video_split.yaml` 第 15 行:
|
||||||
|
```yaml
|
||||||
|
val_ratio: 0.3 # validation 占 trainval 视频组总数的比例
|
||||||
|
```
|
||||||
|
改为:
|
||||||
|
```yaml
|
||||||
|
val_ratio: 0.4 # validation 占 trainval 视频组总数的比例(0.3→0.4 提升整包终审功效,WP2)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add config/video_split.yaml
|
||||||
|
git commit -m "chore: bump video_split val_ratio 0.3->0.4 for terminal-eval power"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: extract_run_db 每题去重(902→900 canonical)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/store.py:269-307`
|
||||||
|
- Test: `tests/unit/test_harness_store.py`(`TestExtractRunDb`)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_store.py` 的 `TestExtractRunDb` 类追加:
|
||||||
|
```python
|
||||||
|
def test_dedupe_per_question_keeps_first_row(self, tmp_path):
|
||||||
|
"""dedupe_per_question=True 时每 question_id 只保留 rowid 最小的首行。"""
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
src = tmp_path / "src.db"
|
||||||
|
conn = sqlite3.connect(src)
|
||||||
|
conn.execute(
|
||||||
|
"CREATE TABLE _runs (run_id TEXT PRIMARY KEY, started_at TEXT)"
|
||||||
|
)
|
||||||
|
conn.execute("INSERT INTO _runs VALUES ('r1', 't0')")
|
||||||
|
conn.execute(
|
||||||
|
"CREATE TABLE predictions (run_id TEXT, question_id TEXT, prediction TEXT)"
|
||||||
|
)
|
||||||
|
# 743-1 三行(模拟 error/budget/finished),首行 prediction=NULL
|
||||||
|
conn.executemany(
|
||||||
|
"INSERT INTO predictions VALUES (?,?,?)",
|
||||||
|
[
|
||||||
|
("r1", "743-1", None),
|
||||||
|
("r1", "743-1", None),
|
||||||
|
("r1", "743-1", "C"),
|
||||||
|
("r1", "q2", "A"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
dst = tmp_path / "dst.db"
|
||||||
|
from app.harness.store import extract_run_db
|
||||||
|
|
||||||
|
extract_run_db(src, dst, "r1", dedupe_per_question=True)
|
||||||
|
|
||||||
|
out = sqlite3.connect(dst)
|
||||||
|
rows = out.execute(
|
||||||
|
"SELECT question_id, prediction FROM predictions ORDER BY question_id"
|
||||||
|
).fetchall()
|
||||||
|
out.close()
|
||||||
|
assert rows == [("743-1", None), ("q2", "A")], f"未按 rowid 首行去重: {rows}"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_store.py::TestExtractRunDb::test_dedupe_per_question_keeps_first_row -v`
|
||||||
|
Expected: FAIL(`extract_run_db() got an unexpected keyword argument 'dedupe_per_question'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现去重**
|
||||||
|
|
||||||
|
`app/harness/store.py` `extract_run_db` 签名改为:
|
||||||
|
```python
|
||||||
|
def extract_run_db(
|
||||||
|
src_db: Path, dst_db: Path, run_id: str, *, dedupe_per_question: bool = False
|
||||||
|
) -> None:
|
||||||
|
```
|
||||||
|
docstring 补一句参数说明:
|
||||||
|
```
|
||||||
|
dedupe_per_question: True 时 predictions 表每 question_id 仅保留 rowid 最小
|
||||||
|
的首行(对齐 canonical「每 question_id 取第一行 ORDER BY rowid」口径,
|
||||||
|
902→900)。_runs 表不受影响。
|
||||||
|
```
|
||||||
|
把 predictions 分支的取行 SQL(L297-299)改为按 `dedupe_per_question` 分派:
|
||||||
|
```python
|
||||||
|
if table == "predictions" and dedupe_per_question:
|
||||||
|
rows = src.execute(
|
||||||
|
f"SELECT {col_sql} FROM {table} WHERE run_id=? "
|
||||||
|
"AND rowid IN (SELECT MIN(rowid) FROM predictions "
|
||||||
|
"WHERE run_id=? GROUP BY question_id)",
|
||||||
|
(run_id, run_id),
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
|
rows = src.execute(
|
||||||
|
f"SELECT {col_sql} FROM {table} WHERE run_id=?", (run_id,)
|
||||||
|
).fetchall()
|
||||||
|
```
|
||||||
|
> NULL question_id 说明:predictions 的 question_id 是题标识、语义上非空(canonical 900 题均有 id),`GROUP BY question_id` 的 NULL 折叠风险不适用。若源库异常出现 NULL question_id,`MIN(rowid) GROUP BY` 会把它们折叠成一行——本任务不为该异常兜底(预测数据契约保证非空),保持 fail-visible。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_store.py -q`
|
||||||
|
Expected: 全 PASS(默认 `dedupe_per_question=False` 保持既有行为,旧测试不受影响)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/store.py tests/unit/test_harness_store.py
|
||||||
|
git commit -m "feat: add dedupe_per_question to extract_run_db (canonical 902->900)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: tier 感知 + val 功效修复的 diag/val 分配
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py:126-199,289-338`
|
||||||
|
- Modify: `app/harness/build_split.py:46-71,181-192`
|
||||||
|
- Modify: `app/harness/video_split_cli.py:565-573`
|
||||||
|
- Test: `tests/unit/test_pools_video_atomic.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(tier 优先 + 功效修复)**
|
||||||
|
|
||||||
|
在 `tests/unit/test_pools_video_atomic.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_tier_aware_keeps_high_t2_in_diag():
|
||||||
|
"""错题视频组按 T2 含量升序进 val:T2 高的组保留在 diagnosis。"""
|
||||||
|
from app.harness.pools import split_by_video_assignment
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
def _q(qid, vid):
|
||||||
|
return GeneratedQuestion(
|
||||||
|
question_id=qid, video_id=vid, task_type="X", question="q",
|
||||||
|
options=["A", "B"], answer="A", source_nodes=[], difficulty="easy",
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4 个错题视频(每视频 1 题),T2 数分别 2/1/0/0
|
||||||
|
questions = [_q(f"{v}-1", v) for v in ("vA", "vB", "vC", "vD")]
|
||||||
|
assignment = {v: "trainval" for v in ("vA", "vB", "vC", "vD")}
|
||||||
|
correctness = {f"{v}-1": False for v in ("vA", "vB", "vC", "vD")}
|
||||||
|
wrong_tier = {"vA": 2, "vB": 1, "vC": 0, "vD": 0}
|
||||||
|
|
||||||
|
pools = split_by_video_assignment(
|
||||||
|
questions, assignment, correctness, val_ratio=0.5, seed=7,
|
||||||
|
wrong_tier_by_video=wrong_tier,
|
||||||
|
)
|
||||||
|
diag_vids = {q.video_id for q in pools.diagnosis}
|
||||||
|
# T2 最高的 vA 必留 diag;T2=0 的组优先进 val
|
||||||
|
assert "vA" in diag_vids
|
||||||
|
assert "vB" in diag_vids
|
||||||
|
|
||||||
|
|
||||||
|
def test_val_wrong_min_repair_pulls_from_diag():
|
||||||
|
"""val 错题不足 val_wrong_min 时从 diag 换入低 T2 错题组补足。"""
|
||||||
|
from app.harness.pools import split_by_video_assignment
|
||||||
|
from core.types import GeneratedQuestion
|
||||||
|
|
||||||
|
def _q(qid, vid, correct):
|
||||||
|
return GeneratedQuestion(
|
||||||
|
question_id=qid, video_id=vid, task_type="X", question="q",
|
||||||
|
options=["A", "B"], answer="A", source_nodes=[], difficulty="easy",
|
||||||
|
)
|
||||||
|
|
||||||
|
# 8 错题视频 + 2 正确视频;val_ratio 小使初分 val 错题不足,触发修复
|
||||||
|
vids_wrong = [f"w{i}" for i in range(8)]
|
||||||
|
vids_correct = ["c0", "c1"]
|
||||||
|
questions = [_q(f"{v}-1", v, False) for v in vids_wrong] + [
|
||||||
|
_q(f"{v}-1", v, True) for v in vids_correct
|
||||||
|
]
|
||||||
|
assignment = {v: "trainval" for v in vids_wrong + vids_correct}
|
||||||
|
correctness = {f"{v}-1": False for v in vids_wrong}
|
||||||
|
correctness.update({f"{v}-1": True for v in vids_correct})
|
||||||
|
wrong_tier = {v: i for i, v in enumerate(vids_wrong)} # 递增 T2
|
||||||
|
|
||||||
|
pools = split_by_video_assignment(
|
||||||
|
questions, assignment, correctness, val_ratio=0.1, seed=7,
|
||||||
|
wrong_tier_by_video=wrong_tier, val_wrong_min=4,
|
||||||
|
)
|
||||||
|
val_wrong = sum(1 for q in pools.validation if not correctness[q.question_id])
|
||||||
|
assert val_wrong >= 4, f"功效修复后 val 错题 {val_wrong} < 4"
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注:`GeneratedQuestion` 的真实字段以 `app/question_gen/types.py` 为准;若构造签名不符,读该文件对齐必填字段(勿臆造)。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_pools_video_atomic.py -k "tier_aware or val_wrong_min_repair" -v`
|
||||||
|
Expected: FAIL(`unexpected keyword argument 'wrong_tier_by_video'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 改 `_split_trainval_by_video_group` 加 tier 感知 + 功效修复**
|
||||||
|
|
||||||
|
`app/harness/pools.py` 函数签名改为:
|
||||||
|
```python
|
||||||
|
def _split_trainval_by_video_group(
|
||||||
|
trainval_qs: list[GeneratedQuestion],
|
||||||
|
correctness: dict[str, bool],
|
||||||
|
val_ratio: float,
|
||||||
|
rng: random.Random,
|
||||||
|
wrong_tier_by_video: dict[str, int] | None = None,
|
||||||
|
val_wrong_min: int = 0,
|
||||||
|
) -> tuple[list[GeneratedQuestion], list[GeneratedQuestion]]:
|
||||||
|
```
|
||||||
|
把分层块(L329-334 的 else 分支)改为 tier 感知:`wrong_vids` 按 T2 含量升序(T2 少的优先进 val),保留 T2 高的组在 diag;`wrong_tier_by_video=None` 时退化为原 shuffle:
|
||||||
|
```python
|
||||||
|
else:
|
||||||
|
val_correct = math.floor(n_correct * n_val / n_total)
|
||||||
|
val_wrong = n_val - val_correct
|
||||||
|
rng.shuffle(correct_vids)
|
||||||
|
if wrong_tier_by_video is None:
|
||||||
|
rng.shuffle(wrong_vids)
|
||||||
|
else:
|
||||||
|
# T2 少的错题组优先进 val(保留 T2 高的组在 diag),确定性排序
|
||||||
|
wrong_vids.sort(key=lambda v: (wrong_tier_by_video.get(v, 0), v))
|
||||||
|
val_vids = set(correct_vids[:val_correct] + wrong_vids[:val_wrong])
|
||||||
|
```
|
||||||
|
在 `val_vids` 确定后、返回前,加**功效修复**(从 diag 的错题组按 T2 升序补入 val 直到满足 val_wrong_min):
|
||||||
|
```python
|
||||||
|
if val_wrong_min > 0:
|
||||||
|
val_wrong_now = sum(
|
||||||
|
1 for v in val_vids for q in groups[v] if not correctness[q.question_id]
|
||||||
|
)
|
||||||
|
# diag 侧仍在的错题组,按 T2 升序(低价值优先移交 val)
|
||||||
|
diag_wrong_pool = sorted(
|
||||||
|
(v for v in wrong_vids if v not in val_vids),
|
||||||
|
key=lambda v: ((wrong_tier_by_video or {}).get(v, 0), v),
|
||||||
|
)
|
||||||
|
for v in diag_wrong_pool:
|
||||||
|
if val_wrong_now >= val_wrong_min:
|
||||||
|
break
|
||||||
|
val_vids.add(v)
|
||||||
|
val_wrong_now += sum(1 for q in groups[v] if not correctness[q.question_id])
|
||||||
|
if val_wrong_now < val_wrong_min:
|
||||||
|
raise InsufficientValSignal(
|
||||||
|
f"trainval 错题不足以让 val 达到 val_wrong_min={val_wrong_min}"
|
||||||
|
f"(修复后仅 {val_wrong_now}),请放大 val_ratio 或调整 trainval 归属。"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(`InsufficientValSignal` 已在 pools.py:118 定义,无需新增;需确认函数内可见 `math`/`defaultdict`,文件顶部已 import。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: `split_by_video_assignment` 透传新参数**
|
||||||
|
|
||||||
|
`app/harness/pools.py` `split_by_video_assignment` 签名加 `wrong_tier_by_video: dict[str, int] | None = None`(放在 `val_wrong_min` 之后),并把 `_split_trainval_by_video_group` 调用(L175-177)改为:
|
||||||
|
```python
|
||||||
|
diagnosis, validation = _split_trainval_by_video_group(
|
||||||
|
trainval_qs, correctness, val_ratio, random.Random(seed),
|
||||||
|
wrong_tier_by_video=wrong_tier_by_video,
|
||||||
|
val_wrong_min=val_wrong_min,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
删除原 L179-186 的独立 `val_wrong_min` 事后校验块(功效已在 `_split_trainval_by_video_group` 内保证,避免重复校验语义)。docstring 的 `val_wrong_min` 说明改为"切分时保证(不足则从 diag 换入低 T2 错题组补足,耗尽 fail-loud)"。
|
||||||
|
|
||||||
|
- [ ] **Step 5: build_split 计算并传入 tier + val_wrong_min**
|
||||||
|
|
||||||
|
`app/harness/build_split.py`:`SplitBuildConfig` 加字段 `val_wrong_min: int`(放 `split_seed` 之后,docstring 补"validation 池最少错题数,切分时保证功效")。build_split Phase 3(L182-191)改为:
|
||||||
|
```python
|
||||||
|
questions = load_benchmark(questions_dir)
|
||||||
|
correctness = {pred["question_id"]: pred["correct"] for pred in preds}
|
||||||
|
tier_by_q = {row["question_id"]: row["tier"] for row in signal_rows}
|
||||||
|
wrong_tier_by_video: dict[str, int] = defaultdict(int)
|
||||||
|
for pred in preds:
|
||||||
|
if not pred["correct"] and tier_by_q.get(pred["question_id"]) == "T2":
|
||||||
|
wrong_tier_by_video[pred["video_id"]] += 1
|
||||||
|
pools = split_by_video_assignment(
|
||||||
|
questions,
|
||||||
|
assignment,
|
||||||
|
correctness,
|
||||||
|
config.val_ratio,
|
||||||
|
config.split_seed,
|
||||||
|
baseline_run_id=baseline_run_id,
|
||||||
|
val_wrong_min=config.val_wrong_min,
|
||||||
|
wrong_tier_by_video=dict(wrong_tier_by_video),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
更新 build_split docstring 的"契约(Task 11...)"段:删除"有意保持 val_wrong_min-agnostic"表述,改为"val_wrong_min 前置到切分内保证功效;CLI 的 check_mcnemar_power 作冗余最终确认"。确认 `defaultdict` 已 import(`from collections import Counter, defaultdict`)。
|
||||||
|
|
||||||
|
- [ ] **Step 6: CLI 传 val_wrong_min + 保真检查**
|
||||||
|
|
||||||
|
`app/harness/video_split_cli.py` 的 `SplitBuildConfig(...)`(L565-573)加一行 `val_wrong_min=config.val_wrong_min,`。
|
||||||
|
保真检查点:确认 `pools.diagnosis`/`pools.validation` 仍是逐题 `GeneratedQuestion` 列表、同 video 全部题同池(`test_pools_video_atomic.py::test_video_group_atomic_in_trainval_split` 覆盖)。
|
||||||
|
|
||||||
|
- [ ] **Step 7: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_pools_video_atomic.py tests/unit/test_harness_pools.py tests/unit/test_split_selection.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 8: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/pools.py app/harness/build_split.py app/harness/video_split_cli.py tests/unit/test_pools_video_atomic.py
|
||||||
|
git commit -m "feat: tier-aware diag/val split with val-power repair (design 5.1)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: 冻结产物覆盖保护 + --force
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/build_split.py:104-192`
|
||||||
|
- Modify: `app/harness/video_split_cli.py:751-773`(build_arg_parser)+ run_pipeline 传参
|
||||||
|
- Test: `tests/unit/test_video_split_cli.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_video_split_cli.py` 追加(用最小 build_split 覆盖场景,或直接测保护函数):
|
||||||
|
```python
|
||||||
|
def test_build_split_refuses_overwrite_without_force(tmp_path):
|
||||||
|
"""已存在指纹不同的 pools.json 时,force=False 必须报错不覆盖。"""
|
||||||
|
from app.harness.build_split import _guard_frozen_products
|
||||||
|
|
||||||
|
out_path = tmp_path / "pools.json"
|
||||||
|
out_path.write_text('{"split_mode":"global"}', encoding="utf-8")
|
||||||
|
manifest_path = tmp_path / "split_manifest.json"
|
||||||
|
|
||||||
|
with pytest.raises(FileExistsError, match="已存在冻结产物"):
|
||||||
|
_guard_frozen_products(out_path, manifest_path, force=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_split_force_backs_up_old(tmp_path):
|
||||||
|
"""force=True 时旧产物被备份为 .bak.* 再允许覆盖。"""
|
||||||
|
from app.harness.build_split import _guard_frozen_products
|
||||||
|
|
||||||
|
out_path = tmp_path / "pools.json"
|
||||||
|
out_path.write_text('{"old":1}', encoding="utf-8")
|
||||||
|
manifest_path = tmp_path / "split_manifest.json"
|
||||||
|
manifest_path.write_text('{"pools_sha256":"deadbeef00000000"}', encoding="utf-8")
|
||||||
|
|
||||||
|
_guard_frozen_products(out_path, manifest_path, force=True)
|
||||||
|
baks = list(tmp_path.glob("pools.json.bak.*"))
|
||||||
|
assert len(baks) == 1, f"未备份旧产物: {list(tmp_path.iterdir())}"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_video_split_cli.py -k "refuses_overwrite or force_backs_up" -v`
|
||||||
|
Expected: FAIL(`cannot import name '_guard_frozen_products'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现覆盖保护函数**
|
||||||
|
|
||||||
|
`app/harness/build_split.py` 顶部 import 区确认有 `import shutil`(无则加)。新增函数(放 build_split 之前):
|
||||||
|
```python
|
||||||
|
def _guard_frozen_products(out_path: Path, manifest_path: Path, *, force: bool) -> None:
|
||||||
|
"""冻结前的覆盖保护:产物已存在时按 force 决定报错或备份。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
out_path: 目标 pools.json 路径。
|
||||||
|
manifest_path: 目标 split_manifest.json 路径。
|
||||||
|
force: False 时已存在即 FileExistsError;True 时把旧产物重命名为
|
||||||
|
.bak.<旧 pools_sha256 前 8 位或 timestamp-less 序号> 再放行。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
FileExistsError: force=False 且产物已存在(防静默覆盖冻结锚点)。
|
||||||
|
"""
|
||||||
|
if not out_path.exists() and not manifest_path.exists():
|
||||||
|
return
|
||||||
|
if not force:
|
||||||
|
raise FileExistsError(
|
||||||
|
f"已存在冻结产物 {out_path}(或其 manifest)。重跑切分会覆盖训练依赖的"
|
||||||
|
"冻结锚点——确认要替换请加 --force(旧产物将备份为 .bak.*)。"
|
||||||
|
)
|
||||||
|
# 备份后缀取旧 manifest 的 pools_sha256 前 8 位,无则用 'prev'
|
||||||
|
suffix = "prev"
|
||||||
|
if manifest_path.exists():
|
||||||
|
try:
|
||||||
|
old = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
suffix = str(old.get("pools_sha256", "prev"))[:8] or "prev"
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
suffix = "prev"
|
||||||
|
for p in (out_path, manifest_path):
|
||||||
|
if p.exists():
|
||||||
|
p.rename(p.with_name(f"{p.name}.bak.{suffix}"))
|
||||||
|
```
|
||||||
|
确认 build_split.py 已 import `json`(无则加 `import json`)。在 `build_split` 签名加参数 `force: bool = False`(放 `generated_at` 之后),并在 Phase 3 `save_pools` 之前(L192 前)调用 `_guard_frozen_products(out_path, manifest_path, force=force)`。
|
||||||
|
|
||||||
|
- [ ] **Step 4: CLI 暴露 --force 并透传**
|
||||||
|
|
||||||
|
`app/harness/video_split_cli.py` `build_arg_parser`(L751-773)追加:
|
||||||
|
```python
|
||||||
|
parser.add_argument(
|
||||||
|
"--force",
|
||||||
|
action="store_true",
|
||||||
|
help="覆盖已存在的冻结 pools.json/manifest(旧产物备份为 .bak.*)",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
`run_pipeline` 签名加 `force: bool = False` 参数,build_split 调用(L559-577)加 `force=force,`;`main()` 里把 `args.force` 透传给 `run_pipeline`。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行测试确认通过 + CLI 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_video_split_cli.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/build_split.py app/harness/video_split_cli.py tests/unit/test_video_split_cli.py
|
||||||
|
git commit -m "feat: guard frozen split products against silent overwrite (--force)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: seed 携带 pools.json + 训练拷入
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/store.py:184-232`(init_seed)
|
||||||
|
- Modify: `app/harness/workspace.py:156-200`(init_workspace_from_seed)
|
||||||
|
- Test: `tests/unit/test_harness_store.py`、`tests/unit/test_harness_workspace.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(seed 携带)**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_store.py::TestInitSeed` 追加:
|
||||||
|
```python
|
||||||
|
def test_init_seed_carries_pools(self, tmp_path):
|
||||||
|
"""提供 pools_json/split_manifest 时拷入 seed 目录。"""
|
||||||
|
from app.harness.store import init_seed
|
||||||
|
|
||||||
|
store = tmp_path / "store"
|
||||||
|
skills = tmp_path / "sk"; skills.mkdir(); (skills / "s.md").write_text("x")
|
||||||
|
prompts = tmp_path / "pr"; prompts.mkdir(); (prompts / "p.md").write_text("y")
|
||||||
|
db = tmp_path / "b.db"; db.write_text("db")
|
||||||
|
pools = tmp_path / "pools.json"; pools.write_text('{"split_mode":"global"}')
|
||||||
|
manifest = tmp_path / "split_manifest.json"; manifest.write_text('{"pools_sha256":"a"}')
|
||||||
|
|
||||||
|
seed_dir = init_seed(
|
||||||
|
store, "s1", skills, prompts, db, "infer_adhoc", None, "d",
|
||||||
|
pools_json=pools, split_manifest=manifest,
|
||||||
|
)
|
||||||
|
assert (seed_dir / "pools.json").exists()
|
||||||
|
assert (seed_dir / "split_manifest.json").exists()
|
||||||
|
```
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_workspace.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_init_workspace_from_seed_carries_pools(store_dir, workspace_dir):
|
||||||
|
"""seed 目录含 pools.json 时拷入 workspace。"""
|
||||||
|
import shutil
|
||||||
|
from app.harness.store import init_seed
|
||||||
|
from app.harness.workspace import init_workspace_from_seed
|
||||||
|
# 复用现有 fixture 构造 seed 的方式;此处补 pools.json 到 seed 后初始化 workspace
|
||||||
|
# (具体 fixture 依 test_harness_workspace.py 现有 helper,读文件对齐)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 该 workspace 测试需依 `test_harness_workspace.py` 现有 fixture(`store_dir`/`workspace_dir` 及既有 seed 构造 helper)填充;实现前读该文件 `test_init_workspace_from_seed`(L151)复用其 seed 搭建,再在 seed 目录写 `pools.json` 后断言 workspace 内出现 `pools.json`。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_store.py::TestInitSeed::test_init_seed_carries_pools -v`
|
||||||
|
Expected: FAIL(`unexpected keyword argument 'pools_json'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: init_seed 加可选携带参数**
|
||||||
|
|
||||||
|
`app/harness/store.py` `init_seed` 签名加:
|
||||||
|
```python
|
||||||
|
def init_seed(
|
||||||
|
store_dir: Path,
|
||||||
|
name: str,
|
||||||
|
skills_dir: Path,
|
||||||
|
prompts_dir: Path,
|
||||||
|
baseline_db: Path,
|
||||||
|
baseline_run_id: str,
|
||||||
|
parent: str | None,
|
||||||
|
description: str,
|
||||||
|
*,
|
||||||
|
pools_json: Path | None = None,
|
||||||
|
split_manifest: Path | None = None,
|
||||||
|
) -> Path:
|
||||||
|
```
|
||||||
|
在 `copy2(baseline_db, ...)`(L218)之后加:
|
||||||
|
```python
|
||||||
|
if pools_json is not None:
|
||||||
|
shutil.copy2(pools_json, seed_dir / "pools.json")
|
||||||
|
if split_manifest is not None:
|
||||||
|
shutil.copy2(split_manifest, seed_dir / "split_manifest.json")
|
||||||
|
```
|
||||||
|
docstring 补两参说明。
|
||||||
|
|
||||||
|
- [ ] **Step 4: init_workspace_from_seed 拷入 pools**
|
||||||
|
|
||||||
|
`app/harness/workspace.py` `init_workspace_from_seed` 在 `shutil.copy2(seed_dir / "baseline.db", workspace_dir / "harness.db")`(L197)之后加:
|
||||||
|
```python
|
||||||
|
seed_pools = seed_dir / "pools.json"
|
||||||
|
if seed_pools.exists():
|
||||||
|
shutil.copy2(seed_pools, workspace_dir / "pools.json")
|
||||||
|
seed_manifest = seed_dir / "split_manifest.json"
|
||||||
|
if seed_manifest.exists():
|
||||||
|
shutil.copy2(seed_manifest, workspace_dir / "split_manifest.json")
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_store.py tests/unit/test_harness_workspace.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/store.py app/harness/workspace.py tests/unit/test_harness_store.py tests/unit/test_harness_workspace.py
|
||||||
|
git commit -m "feat: seed carries frozen pools.json into training workspace"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: build_or_load_pools global 一致性校验
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/pools.py:746-813`
|
||||||
|
- Test: `tests/unit/test_harness_pools.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_pools.py::TestBuildOrLoadPoolsFrozen` 追加:
|
||||||
|
```python
|
||||||
|
def test_global_frozen_rejects_baseline_mismatch(self, tmp_path, ...):
|
||||||
|
"""global 冻结 pools 的 baseline_run_id 与 seed 不符时 fail-loud。"""
|
||||||
|
# 依现有 fixture 造 workspace + 冻结 pools.json(split_mode=global,
|
||||||
|
# baseline_run_id="other"),seed.json baseline_run_id="infer_adhoc"
|
||||||
|
# 调 build_or_load_pools 应 raise ValueError(match="baseline_run_id")
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依 `TestBuildOrLoadPoolsFrozen`(L246)现有 fixture 复用其 workspace/seed 搭建;实现前读该类对齐 RunConfig/strategy 构造,勿臆造。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_pools.py::TestBuildOrLoadPoolsFrozen -v`
|
||||||
|
Expected: 新用例 FAIL(当前 global 分支无校验,误加载不报错)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 加 global 一致性校验**
|
||||||
|
|
||||||
|
`app/harness/pools.py` `build_or_load_pools`,在 global 加载分支(`if pools_path.exists():` 块内、`per_category` 校验的 `else` 侧,即 L813 `return load_pools(pools_path)` 之前)加:
|
||||||
|
```python
|
||||||
|
else: # global:校验 baseline_run_id 与(若有)manifest 内容指纹
|
||||||
|
frozen_baseline = raw.get("baseline_run_id")
|
||||||
|
if frozen_baseline != baseline_run_id:
|
||||||
|
raise ValueError(
|
||||||
|
f"冻结 pools.json 的 baseline_run_id={frozen_baseline!r} 与 seed "
|
||||||
|
f"的 {baseline_run_id!r} 不一致,拒绝静默加载错配切分。"
|
||||||
|
)
|
||||||
|
manifest_path = config.workspace_dir / "split_manifest.json"
|
||||||
|
if manifest_path.exists():
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
actual_sha = hashlib.sha256(
|
||||||
|
pools_path.read_text(encoding="utf-8").encode("utf-8")
|
||||||
|
).hexdigest()
|
||||||
|
if manifest.get("pools_sha256") != actual_sha:
|
||||||
|
raise ValueError(
|
||||||
|
"pools.json 内容指纹与 split_manifest.pools_sha256 不符,"
|
||||||
|
"冻结产物疑被篡改,拒绝加载。"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(确认该 `else` 与 L751 `if frozen_split_mode == "per_category":` 配对;若现有结构非 if/else 而是 if 后直接 return,则把校验插在 `return load_pools(pools_path)` 前并用 `if frozen_split_mode != "per_category":` 守卫。实现前读 L746-813 对齐控制流。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_pools.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/pools.py tests/unit/test_harness_pools.py
|
||||||
|
git commit -m "fix: validate global frozen pools baseline_run_id + sha256 on load"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review(作者自查,执行者复核)
|
||||||
|
|
||||||
|
- [ ] val_ratio=0.4 已改;tier 感知 + 功效修复在同一函数、退化路径(`wrong_tier_by_video=None`)保持旧行为。
|
||||||
|
- [ ] val_wrong_min 从 CLI→SplitBuildConfig→build_split→split_by_video_assignment→_split_trainval_by_video_group 全链路贯通;旧的 pools.py 事后校验块已删(不重复)。
|
||||||
|
- [ ] extract_run_db 去重默认关闭,不破坏既有调用。
|
||||||
|
- [ ] seed 携带 + workspace 拷入 + global 一致性校验三者闭环:冻结产物有唯一路径进训练且被校验。
|
||||||
|
- [ ] 覆盖保护默认 force=False,离线 CLI 重跑需显式 --force。
|
||||||
|
|
||||||
|
## 核心算法保真校验结论
|
||||||
|
|
||||||
|
本计划触及算法 #5 的上游输入(哪些视频进 diag/val),**不改** gate_ladder 的 unit+correctness 消费结构;Task 3 Step 6 已设保真检查点确认逐 unit 列表与视频组原子性。不涉及算法 #4/#6/#8/#9 逻辑。
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
|
||||||
|
1. `pytest tests/unit/test_pools_video_atomic.py tests/unit/test_harness_pools.py tests/unit/test_harness_store.py tests/unit/test_harness_workspace.py tests/unit/test_split_selection.py tests/unit/test_video_split_cli.py` 全绿。
|
||||||
|
2. tier 感知:T2 高的错题视频组留 diag,T2 低的优先进 val。
|
||||||
|
3. seed 携带 pools.json → init_workspace_from_seed 拷入 → build_or_load_pools 校验 baseline_run_id + sha256。
|
||||||
|
4. 冻结产物 force=False 时拒绝覆盖。
|
||||||
@@ -0,0 +1,609 @@
|
|||||||
|
# WP3 训练循环与进化引擎 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
> **前置依赖:WP1(模板已迁移,进化引擎可运行)+ WP4(cache_salt 能力已就绪)必须先完成。**
|
||||||
|
|
||||||
|
**Goal:** 修复训练循环与进化引擎的 9 处缺陷,使诊断拿到真实轨迹、早停按 epoch 语义、微型题型不崩、崩溃可幂等续跑、进化 patch 不破坏冻结区、降级信号不驱动错误进化、跨 epoch 评估真实重采样。
|
||||||
|
|
||||||
|
**Architecture:** 诊断经 `StepsJsonRunLog` 从 steps_json 重建轨迹(算法 #7 恢复);早停计数单位 step→epoch;可训练性预检在 gate 建立前剔除微型题型;`_run_step` 幂等(先 DELETE 再写);patch 冻结区检查整个 target 跨度(算法 #8 加固);降级/未判定题按 lapse 保守分流;训练推理用 run_id 作 cache_salt(run_id 已含 epoch,天然跨 epoch 重采样)。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、asyncio、SQLite、pytest。
|
||||||
|
|
||||||
|
**设计源**:`research-wiki/designs/2026-07-16-preflight-fixes-design.md §6-7`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键锚点(实现前必读)
|
||||||
|
|
||||||
|
| 用途 | 位置 |
|
||||||
|
|------|------|
|
||||||
|
| 训练主流程 | `app/harness/runner.py:789-859` `train`;`_setup_train_run:865-879`(预检插入点 L874 前);`_run_step:1001-1026`(run_id L1011、诊断 L1019、无 DELETE) |
|
||||||
|
| 早停 | `app/harness/runner.py:291-316` `_should_early_stop`(L315 `+= steps_this_epoch`);`_TrainState:95-121`(`steps_since_best_improved` L118);`_maybe_promote_best:1587`(置 0) |
|
||||||
|
| 诊断调用 | `app/harness/runner.py:2163-2196` `_run_diagnosis`(`RunLogImpl` L2173 未包 StepsJsonRunLog);DiagnosisResult `degraded_count` 未被引用 |
|
||||||
|
| gate 刷新 | `app/harness/runner.py:1833-1879` `_refresh_gate_ladder`(save L1878 → set observed L1879);checkpoint 落盘晚在 train L837 |
|
||||||
|
| holdout | `app/harness/runner.py:1881-1921` `_holdout_four_way`;`_pick_mixed_best:1923-1963`;`_eval_version_on_pool:2148-2161` |
|
||||||
|
| 推理落库 | `app/harness/inference.py:363-447` `_run_single_question`(prediction L422 未归一、insert L446 try 外);`_to_text_field:147-162`;`run_inference:473` |
|
||||||
|
| Agent Loop | `core/agent/loop.py:103` `run`(session_id L110);`_call_llm` chat 调用 `:329`(`self._llm.chat(messages, session_id=session_id)`) |
|
||||||
|
| batching | `app/harness/batching.py:186-202` `_classify_unit`(L200 缺 correctness→None) |
|
||||||
|
| checkpoint | `app/harness/checkpoint.py:37-44` `_STRUCTURAL_KEYS`;`serialize_state:76-106`;`write_checkpoint:212-260`(原子写) |
|
||||||
|
| 诊断分流 | `core/evolution/diagnose.py:1485-1519` `_build_skill_case_packs`(lapse 分流 L1492);`_process_question:2139-2206`(except L2194 cause_category 留 None) |
|
||||||
|
| traces 适配 | `app/harness/baseline_run_log.py:13` `StepsJsonRunLog`;`steps_json_traces.py:13` |
|
||||||
|
| patch | `core/evolution/patch.py:285-287` `_in_ranges`;`_do_insert_after:309-326`(L320);`_do_replace_delete:329-347`(L343);markers L11-18;`validate_skill` in `evolve.py:296` |
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
触及算法 #7(诊断瀑布,Task 1 恢复轨迹)、#8(patch 引擎,Task 5 冻结区加固)、#10(Agent Loop,Task 9 透传 cache_salt)、#5/#12(信息阶梯/训练编排,Task 7 checkpoint 时序)。**均为恢复/加固/透传,不改算法逻辑**:Task 1 让诊断拿到本就该有的轨迹;Task 5 把"只查起点"补成"查整跨度"(保护方向不变);Task 9 只加透传参数;Task 7 只调 checkpoint 落盘时机。每个相关 Task 设保真检查点。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: traces 适配(诊断拿到真实轨迹,算法 #7)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:2163-2196`(_run_diagnosis)
|
||||||
|
- Test: `tests/unit/test_runner_diag_tree_inject.py` 或 `test_harness_runner.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_runner_diag_tree_inject.py` 追加(构造只写 steps_json 不写 traces 表的 run,断言诊断能拿到轨迹):
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_diagnosis_reads_traces_from_steps_json(...):
|
||||||
|
"""traces 表为空但 predictions.steps_json 有轨迹时,诊断仍拿到非空 traces。"""
|
||||||
|
# 依现有 runner 测试 fixture 造一个 run:predictions 有 steps_json,traces 表空;
|
||||||
|
# 调 _run_diagnosis 后断言 diagnose 收到的 traces 非空(可 patch run_diagnosis 捕获入参)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依 `test_runner_diag_tree_inject.py` 现有 fixture;实现前读对齐 runner 构造与 patch 点。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_runner_diag_tree_inject.py -k reads_traces_from_steps_json -v`
|
||||||
|
Expected: FAIL(当前 RunLogImpl 直读空 traces 表)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 包 StepsJsonRunLog**
|
||||||
|
|
||||||
|
`app/harness/runner.py` `_run_diagnosis`,把传给 `run_diagnosis` 的 `run_log`(当前 `RunLogImpl(...)`,L2173 附近)包一层:
|
||||||
|
```python
|
||||||
|
from app.harness.baseline_run_log import StepsJsonRunLog
|
||||||
|
from app.harness.log import RunLogImpl
|
||||||
|
|
||||||
|
run_log = StepsJsonRunLog(RunLogImpl(str(self._paths.db_path)))
|
||||||
|
```
|
||||||
|
(`StepsJsonRunLog.get_traces` 在底层 traces 空时从 predictions.steps_json 经 `steps_json_to_trace_rows` 重建;`get_predictions` 透传。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: 保真检查点 + 测试通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_runner_diag_tree_inject.py tests/unit/test_baseline_run_log.py -q`
|
||||||
|
Expected: 全 PASS。保真:确认诊断瀑布拿到的是逐 step `{tool_name,tool_args,tool_output,thought}` 行(对齐 TRM4 诊断输入)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py tests/unit/test_runner_diag_tree_inject.py
|
||||||
|
git commit -m "fix: wrap diagnosis run_log with StepsJsonRunLog (restore algo #7 traces)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: prediction 归一化 + 落库加固
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/inference.py:417-447`
|
||||||
|
- Test: `tests/unit/test_harness_inference.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_inference.py` 追加(LLM 提交非标量 answer 不崩 gather):
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_nonscalar_prediction_does_not_crash(...):
|
||||||
|
"""submit_answer 返回 {'answer': ['B']} 等非标量时归一化落库,不抛 sqlite 绑定异常。"""
|
||||||
|
# 依现有 inference 测试 fixture,让 AgentLoop 返回 result={'answer': ['B']};
|
||||||
|
# run_inference 应正常完成、predictions 行 prediction 为字符串(如 '["B"]'),不崩
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依 `test_harness_inference.py` 现有 fake loop/dispatch fixture;实现前读对齐。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_inference.py -k nonscalar_prediction -v`
|
||||||
|
Expected: FAIL(sqlite `InterfaceError: Error binding parameter`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 归一化 prediction + insert 加固**
|
||||||
|
|
||||||
|
`app/harness/inference.py`:新增归一化 helper(None 保留、str 原样、其余 `_to_text_field`):
|
||||||
|
```python
|
||||||
|
def _normalize_prediction(answer: object) -> str | None:
|
||||||
|
"""归一化 prediction:None 保留(INFRA 空预测语义),str 原样,其余 JSON 序列化。"""
|
||||||
|
if answer is None or isinstance(answer, str):
|
||||||
|
return answer
|
||||||
|
return _to_text_field(answer)
|
||||||
|
```
|
||||||
|
L422 `"prediction": result_dict.get("answer"),` 改为 `"prediction": _normalize_prediction(result_dict.get("answer")),`。
|
||||||
|
L446 的 `await asyncio.to_thread(log.insert, "predictions", record)` 包 try,绑定异常降级为最小 error 行不击穿 gather:
|
||||||
|
```python
|
||||||
|
try:
|
||||||
|
await asyncio.to_thread(log.insert, "predictions", record)
|
||||||
|
except (sqlite3.InterfaceError, sqlite3.ProgrammingError):
|
||||||
|
logger.exception("[{}] QA {} 落库绑定异常,降级为 error 行", qa.video_id, qa.question_id)
|
||||||
|
record["prediction"] = None
|
||||||
|
record["stop_reason"] = "error"
|
||||||
|
await asyncio.to_thread(
|
||||||
|
log.insert, "predictions",
|
||||||
|
{k: v for k, v in record.items() if isinstance(v, (str, int, float, type(None)))},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
确认 `import sqlite3` 在文件顶部(无则加)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 测试通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_inference.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/inference.py tests/unit/test_harness_inference.py
|
||||||
|
git commit -m "fix: normalize non-scalar prediction; harden predictions insert"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: early_stop 改 epoch 计数
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:291-316,118,1587,849-857`
|
||||||
|
- Modify: `app/harness/checkpoint.py`(若字段入 state)
|
||||||
|
- Test: `tests/unit/test_harness_runner.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_runner.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_early_stop_counts_epochs_not_steps(tmp_path):
|
||||||
|
"""patience=2 表示连续 2 个 epoch 无 best 刷新才停(不是步数)。"""
|
||||||
|
from app.harness.runner import _should_early_stop, _TrainState
|
||||||
|
# 造 state + workspace,best 停在 epoch 1;
|
||||||
|
# epoch 2 无刷新 → epochs_since_best_improved=1 → 不停;
|
||||||
|
# epoch 3 无刷新 → =2 → 停
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依现有 `_TrainState`/`read_best` fixture;实现前读对齐 workspace best 写入。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -k early_stop_counts_epochs -v`
|
||||||
|
Expected: FAIL(当前累加 steps_this_epoch)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 字段改名 + 计数改 epoch**
|
||||||
|
|
||||||
|
全局把 `steps_since_best_improved` 改名 `epochs_since_best_improved`(`grep -rn steps_since_best_improved app/`:`_TrainState:118`、`_should_early_stop:313,315`、`_maybe_promote_best:1587`,以及 checkpoint serialize/deserialize 若含此字段)。
|
||||||
|
`_should_early_stop`(L315)`state.steps_since_best_improved += steps_this_epoch` 改为 `state.epochs_since_best_improved += 1`;签名删除 `steps_this_epoch` 参数(改为 `_should_early_stop(workspace_dir, epoch, state, patience)`),train 调用点(L849-857)同步去掉 `len(batches)` 实参。docstring 改为"epoch 粒度"。
|
||||||
|
|
||||||
|
- [ ] **Step 4: checkpoint 兼容**
|
||||||
|
|
||||||
|
若 `epochs_since_best_improved` 入 checkpoint state(`grep -n steps_since_best_improved app/harness/checkpoint.py`),同步改名。本轮为 fresh 训练无旧 checkpoint,无迁移负担。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 测试通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py tests/unit/test_harness_checkpoint.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py app/harness/checkpoint.py tests/unit/test_harness_runner.py
|
||||||
|
git commit -m "fix: early_stop patience counts epochs not steps"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: 可训练性预检
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py`(`train` 入口过滤 + `_setup_train_run` 接收 filtered task_types)
|
||||||
|
- Modify: `app/harness/config.py`(RunConfig 加 `trainable_min_units` + 正整数校验)
|
||||||
|
- Modify: `app/harness/checkpoint.py:37`(`trainable_min_units` 入 `_STRUCTURAL_KEYS` 指纹)
|
||||||
|
- Test: `tests/unit/test_harness_runner.py`、`tests/unit/test_harness_checkpoint.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_untrainable_types_filtered_before_gate():
|
||||||
|
"""val<eval_min_per_class 或 非test单元<trainable_min_units 的题型从 diag/val/task_types 剔除。"""
|
||||||
|
from app.harness.runner import _filter_untrainable_types
|
||||||
|
# 构造 pools:题型 A(val=5, units=40)可训;B(val=0, units=2)不可训
|
||||||
|
# 调 _filter_untrainable_types(pools, task_types=[A,B], eval_min_per_class=2, trainable_min_units=8)
|
||||||
|
# 断言返回 pools 不含 B、task_types 不含 B
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -k untrainable_types_filtered -v`
|
||||||
|
Expected: FAIL(`_filter_untrainable_types` 不存在)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现纯函数预检**
|
||||||
|
|
||||||
|
`app/harness/runner.py` 新增模块级纯函数:
|
||||||
|
```python
|
||||||
|
def _filter_untrainable_types(
|
||||||
|
pools: Pools,
|
||||||
|
task_types: list[str] | None,
|
||||||
|
eval_min_per_class: int,
|
||||||
|
trainable_min_units: int,
|
||||||
|
) -> tuple[Pools, list[str] | None]:
|
||||||
|
"""剔除不可训练题型(val<eval_min_per_class 或 非test单元<trainable_min_units)。
|
||||||
|
|
||||||
|
非test单元数 = 该题型 diag+val 题数(single 题 unit==题;等于 gate 阶梯该类候选数)。
|
||||||
|
test 池不过滤(继续报告全题型准确率)。返回过滤后 (pools, task_types)。
|
||||||
|
"""
|
||||||
|
from collections import Counter
|
||||||
|
|
||||||
|
diag_by_type = Counter(q.task_type for q in pools.diagnosis)
|
||||||
|
val_by_type = Counter(q.task_type for q in pools.validation)
|
||||||
|
keep: set[str] = set()
|
||||||
|
dropped: list[tuple[str, str]] = []
|
||||||
|
for tt in set(diag_by_type) | set(val_by_type):
|
||||||
|
n_val = val_by_type.get(tt, 0)
|
||||||
|
n_units = diag_by_type.get(tt, 0) + n_val
|
||||||
|
if n_val < eval_min_per_class:
|
||||||
|
dropped.append((tt, f"val={n_val}<{eval_min_per_class}"))
|
||||||
|
elif n_units < trainable_min_units:
|
||||||
|
dropped.append((tt, f"units={n_units}<{trainable_min_units}"))
|
||||||
|
else:
|
||||||
|
keep.add(tt)
|
||||||
|
for tt, why in sorted(dropped):
|
||||||
|
logger.warning("可训练性预检剔除题型 {}({})", tt, why)
|
||||||
|
new_pools = replace(
|
||||||
|
pools,
|
||||||
|
diagnosis=[q for q in pools.diagnosis if q.task_type in keep],
|
||||||
|
validation=[q for q in pools.validation if q.task_type in keep],
|
||||||
|
)
|
||||||
|
new_types = [t for t in task_types if t in keep] if task_types is not None else sorted(keep)
|
||||||
|
return new_pools, new_types
|
||||||
|
```
|
||||||
|
(确认 `from dataclasses import replace` 已 import;`Pools` 是否 frozen dataclass 支持 `replace`——若非,按其构造方式重建。)
|
||||||
|
|
||||||
|
**过滤结果必须回传主循环(Codex Critical)**:`RunConfig` 是 `@dataclass(frozen=True)`,**不能** `self._config.task_types = ...`(会 FrozenInstanceError),且过滤后的 pools 必须被 `train()` 后续的 batch/step/slow-update/final-eval 全部使用。实现方式:
|
||||||
|
- 在 `train(pools)` **入口第一步**(`_setup_train_run` 调用之前)过滤:
|
||||||
|
```python
|
||||||
|
pools, filtered_task_types = _filter_untrainable_types(
|
||||||
|
pools, self._config.task_types,
|
||||||
|
self._config.eval_min_per_class, self._config.trainable_min_units,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- 把 filtered `pools` 传给 `_setup_train_run(pools)` 与 train() 后续所有消费点(build_batches / slow_update / final_eval 均用这个 filtered pools,不再触碰原始 pools)。
|
||||||
|
- 把 `filtered_task_types` 传给 gate 建立(`_setup_train_run`/`_init_gate_pools` 用它而非 `self._config.task_types`)——新增参数透传,不改 frozen config。
|
||||||
|
|
||||||
|
`app/harness/config.py`:`RunConfig` 加字段 `trainable_min_units: int`(无默认,显式配置;train yaml 提供);在配置校验函数(如 `validate_config`,config.py:277 附近)加 `trainable_min_units >= 1` 断言(<1 报错)。
|
||||||
|
`app/harness/checkpoint.py:37` `_STRUCTURAL_KEYS` 加入 `"trainable_min_units"`——该值改变会改变 pools 过滤结果与训练轨迹,必须纳入 checkpoint 结构指纹,resume 时变化即拒绝复用旧 checkpoint。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 测试通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py app/harness/config.py tests/unit/test_harness_runner.py
|
||||||
|
git commit -m "feat: pre-flight filter of untrainable task types before gate"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: patch 冻结区跨度 + 注入 + marker 校验(算法 #8)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/patch.py:285-347`
|
||||||
|
- Modify: `core/evolution/evolve.py:296`(validate_skill)
|
||||||
|
- Test: `tests/unit/test_patch.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_patch.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_replace_spanning_into_protected_is_skipped():
|
||||||
|
"""target 起点在正文、末端伸入冻结区的 replace 被跳过(不破坏 marker)。"""
|
||||||
|
from core.evolution.patch import apply_patch_with_report, APPENDIX_START, APPENDIX_END
|
||||||
|
|
||||||
|
body = "正文最后一段。"
|
||||||
|
appendix = f"{APPENDIX_START}\n## 执行提醒\n- 规则A\n{APPENDIX_END}"
|
||||||
|
content = body + "\n\n" + appendix
|
||||||
|
# target 从正文末尾跨入 APPENDIX_START
|
||||||
|
target = "正文最后一段。\n\n" + APPENDIX_START
|
||||||
|
edits = [{"op": "delete", "target": target, "content": ""}]
|
||||||
|
new_content, report = apply_patch_with_report(content, edits, protected_spans=[appendix])
|
||||||
|
assert APPENDIX_START in new_content and APPENDIX_END in new_content # marker 未被破坏
|
||||||
|
|
||||||
|
|
||||||
|
def test_edit_payload_with_marker_literal_rejected():
|
||||||
|
"""edit payload/target 含 marker 字面量 → 拒绝该 edit。"""
|
||||||
|
from core.evolution.patch import apply_patch_with_report, APPENDIX_START
|
||||||
|
edits = [{"op": "append", "target": "", "content": f"注入 {APPENDIX_START} 破坏"}]
|
||||||
|
_, report = apply_patch_with_report("正文", edits, protected_spans=[])
|
||||||
|
assert any("marker" in str(s).lower() or "reject" in str(s).lower() for s in report)
|
||||||
|
```
|
||||||
|
> marker 常量名以 `core/evolution/patch.py:11-18` 为准(`APPENDIX_START` 等),实现前读对齐 import 名。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_patch.py -k "spanning_into_protected or marker_literal" -v`
|
||||||
|
Expected: FAIL(当前只查起点 pos,跨入未拦;无注入检查)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 跨度检查 + 注入检查**
|
||||||
|
|
||||||
|
`core/evolution/patch.py` 新增跨度 helper:
|
||||||
|
```python
|
||||||
|
def _span_overlaps_ranges(pos: int, length: int, ranges: list[tuple[int, int]]) -> bool:
|
||||||
|
"""判断 [pos, pos+length) 是否与任一冻结区间相交(不止起点)。"""
|
||||||
|
end = pos + length
|
||||||
|
return any(start < end and pos < r_end for start, r_end in ranges)
|
||||||
|
```
|
||||||
|
`_do_insert_after` L320 `if _in_ranges(pos, ranges):` 改为 `if _span_overlaps_ranges(pos, len(target), ranges):`。
|
||||||
|
`_do_replace_delete` L343 `if _in_ranges(pos, ranges):` 改为 `if _span_overlaps_ranges(pos, len(target), ranges):`。
|
||||||
|
在 `apply_patch_with_report`(L387)应用每个 edit 前加注入检查:payload/target 含任一 marker 字面量(`APPENDIX_START/END`、`MOMENTUM_START/END`)→ 跳过该 edit 并记 `skipped_marker_injection`。
|
||||||
|
|
||||||
|
- [ ] **Step 4: validate_skill 加 marker 完整性校验**
|
||||||
|
|
||||||
|
`core/evolution/evolve.py:296` `validate_skill`:在现有 frontmatter/长度/代码块校验后加——统计 evolved 中 `APPENDIX_START/END`、`MOMENTUM_START/END` 出现次数,要求成对(START 数==END 数)、各至多一对、START 在 END 前;违反则返回校验失败(该候选整体 reject)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 保真检查点 + 测试通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_patch.py tests/unit/test_evolve.py -q`
|
||||||
|
Expected: 全 PASS。保真:确认"保护跨度"方向未变(仍是保护 appendix/momentum 不被误改),只是从"查起点"补成"查整跨度"+ 注入/完整性双防线。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/evolution/patch.py core/evolution/evolve.py tests/unit/test_patch.py
|
||||||
|
git commit -m "fix: patch checks full target span + marker injection/integrity (algo #8)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: 诊断降级分流 + 占比中止
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/diagnose.py:1485-1497`
|
||||||
|
- Modify: `app/harness/runner.py:1019`(诊断后 degraded 占比检查)
|
||||||
|
- Test: `tests/unit/test_diagnose.py`、`tests/unit/test_harness_runner.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(分流)**
|
||||||
|
|
||||||
|
在 `tests/unit/test_diagnose.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_none_cause_and_degraded_route_to_lapse():
|
||||||
|
"""cause_category=None(判别失败)与 degraded 题按 lapse 处置,不进 defect 正文路径。"""
|
||||||
|
from core.evolution.diagnose import _build_skill_case_packs
|
||||||
|
# 构造 metrics_group:一题 attr.cause_category=None(非 degraded)、一题 qm.degraded=True;
|
||||||
|
# 断言二者都不出现在 failure_cases(wrong_by_error),只可能进 lapse_notes
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依 `test_diagnose.py` 现有 QuestionMetrics/ErrorAttribution 构造(`:753` 附近);实现前读对齐字段。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_diagnose.py -k none_cause_and_degraded_route -v`
|
||||||
|
Expected: FAIL(当前 None → wrong_by_error 走 defect 正文)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 改分流逻辑**
|
||||||
|
|
||||||
|
`core/evolution/diagnose.py` `_build_skill_case_packs` 的分流循环(L1488-1497)改为"仅明确 defect 且非 degraded 才进正文路径":
|
||||||
|
```python
|
||||||
|
for qm in metrics_group:
|
||||||
|
if qm.correct:
|
||||||
|
continue
|
||||||
|
attr = attribution_map.get(qm.question_id)
|
||||||
|
is_defect = (
|
||||||
|
attr is not None
|
||||||
|
and attr.cause_category == "defect"
|
||||||
|
and not qm.degraded
|
||||||
|
)
|
||||||
|
if not is_defect:
|
||||||
|
# lapse / None(判别失败)/ degraded → 保守,不驱动正文进化
|
||||||
|
if attr is not None and attr.lapse_note and attr.lapse_note.strip():
|
||||||
|
lapse_notes.append(attr.lapse_note)
|
||||||
|
continue
|
||||||
|
wrong_by_error[attr.error_type].append(qm)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 写失败测试(占比中止)+ 实现**
|
||||||
|
|
||||||
|
`tests/unit/test_harness_runner.py` 追加:诊断结果 degraded_count/总题数 > 0.5 时 `_run_step` 后应 raise(疑似基础设施故障)。
|
||||||
|
`app/harness/runner.py` `_run_step`(L1019 拿到 `diagnosis` 后)加:
|
||||||
|
```python
|
||||||
|
n_wrong = sum(1 for q in batch if not state.correctness.get(q.question_id, True))
|
||||||
|
if n_wrong > 0 and diagnosis.degraded_count / n_wrong > 0.5:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"本 step 诊断降级占比 {diagnosis.degraded_count}/{n_wrong} > 50%,"
|
||||||
|
"疑似 judge 基础设施故障,中止训练(不以降级信号驱动进化)。"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(确认 `DiagnosisResult.degraded_count` 字段可用,定义在 `core/evolution/types.py:299`。)
|
||||||
|
|
||||||
|
- [ ] **Step 5: 测试通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_diagnose.py tests/unit/test_harness_runner.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/evolution/diagnose.py app/harness/runner.py tests/unit/test_diagnose.py tests/unit/test_harness_runner.py
|
||||||
|
git commit -m "fix: route None/degraded diagnoses to lapse; abort on high degrade rate"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 7: step 幂等(DELETE)+ gate_epoch_observed 立即落盘
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:1001-1026`(_run_step 开头 DELETE)
|
||||||
|
- Modify: `app/harness/runner.py:1378-1498,1833-1879`(checkpoint 提到 gate save 之后)
|
||||||
|
- Test: `tests/unit/test_harness_runner.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(幂等)**
|
||||||
|
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_step_deletes_stale_rows_before_rerun(...):
|
||||||
|
"""同 run_id 重跑前先清 predictions/traces,避免重复行双计。"""
|
||||||
|
# 预置该 step run_id 的旧 predictions 行;调 _run_step;断言旧行被清、只剩本次
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -k deletes_stale_rows -v`
|
||||||
|
Expected: FAIL(当前 append,无 DELETE)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: _run_step 开头 DELETE**
|
||||||
|
|
||||||
|
`app/harness/runner.py` `_run_step`,在 rollout(L1012)之前加(**用 `IF EXISTS` 避免 fresh workspace 首跑时 predictions/traces 表尚未由 `run_inference._ensure_tables` 创建导致 `OperationalError: no such table`**):
|
||||||
|
```python
|
||||||
|
with HarnessLog(str(self._paths.db_path), run_id, register_run=False) as log:
|
||||||
|
log.execute("DELETE FROM predictions WHERE run_id=?", (run_id,))
|
||||||
|
log.execute("DELETE FROM traces WHERE run_id=?", (run_id,))
|
||||||
|
```
|
||||||
|
> SQLite `DELETE FROM <t>` 对不存在的表会抛 `no such table`。两种消解方式择一(实现前读 log.py 确认):① rollout 由 `run_inference` 先建表——把 DELETE 移到**首次 rollout 之后、诊断之前**并只在 resume 重跑(step 已有旧行)时执行;② 或 DELETE 前先 `CREATE TABLE IF NOT EXISTS`(复用 inference 的 PREDICTIONS_SCHEMA/TRACES_SCHEMA),保证幂等无害。推荐 ②(无害且简单)。`register_run=False` 来自 WP4;若 HarnessLog 无 `execute` 便捷方法,用其现有连接接口。
|
||||||
|
|
||||||
|
- [ ] **Step 4: checkpoint 提到 gate save 之后(消除双计窗口)**
|
||||||
|
|
||||||
|
目标:`_refresh_gate_ladder` 内 `gate_pools.save`(L1878)+ `gate_epoch_observed=True`(L1879)之后,**立即** `write_checkpoint`(phase="epoch_done"),不等到 `train` L837。实现:把 `_slow_update_cycle` Phase 10(调 `_refresh_gate_ladder` L1496-1498)之后的 checkpoint 落盘从 `train`(L837)移入 `_slow_update_cycle` 末尾,或让 `_refresh_gate_ladder` 接收 checkpoint 所需上下文(epoch/progress/batches)并在 save 后落盘。实现前读 `write_checkpoint` 签名(checkpoint.py:212)与 `train` L833-848 对齐参数,确保 gate_pools.json 与 checkpoint 的 `gate_epoch_observed` 同一时刻一致。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 保真检查点 + 测试通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py tests/unit/test_harness_checkpoint.py -q`
|
||||||
|
Expected: 全 PASS。保真(算法 #5/#12):确认 γ-EMA 更新(`update_probs`)仍每 epoch 一次、checkpoint 落盘不改变慢更新十步序的语义顺序。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py tests/unit/test_harness_runner.py
|
||||||
|
git commit -m "fix: idempotent _run_step (DELETE stale) + checkpoint after gate save"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 8: holdout 四向去重
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:1881-1963`(_holdout_four_way / _pick_mixed_best)
|
||||||
|
- Test: `tests/unit/test_harness_runner.py`
|
||||||
|
|
||||||
|
> **说明**:目标是每 epoch 的四向 test 评估从"4×600 全跑"降为"仅 final 必跑 + best_hard 未评过才跑 + baseline 从基线预测推导 + best_mixed 引用赢家"。去重做在 harness 逻辑层(配合 WP4 epoch 盐,同版本不重采样)。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_holdout_dedup_skips_reevaluated_versions(...):
|
||||||
|
"""baseline 不跑推理(从基线预测推导);best_hard==final 时不重复评估。"""
|
||||||
|
# 统计 _eval_version_on_pool 被调次数:baseline=0,best_hard==final 时该向复用不重跑
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依现有 runner holdout fixture;实现前读 `_holdout_four_way`/`write_holdout_eval` 对齐。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -k holdout_dedup -v`
|
||||||
|
Expected: FAIL(当前四向各跑一次)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现去重(进程内备忘录,不改 holdout_eval schema)**
|
||||||
|
|
||||||
|
> **schema 约束(Codex Critical)**:当前 `holdout_eval` 表(observation.py:78)不存 skills_version/prompts_version/pointer,无法按版本反查做跨-run hydrate。本 task **不扩展该 schema**(避免结构性风险),改用 **train() 进程内备忘录** `dict[(skills_v,prompts_v), float]` 去重。代价:resume 后备忘录清空、已评版本会重评一次——resume 是异常路径、重评 600 题成本可接受,换取零 schema 变更风险。完整跨-run hydrate 记 future work。
|
||||||
|
|
||||||
|
`app/harness/runner.py` `_holdout_four_way`(在 `_TrainState` 加一个 `holdout_memo: dict[tuple[str,str], float] = field(default_factory=dict)` 字段):
|
||||||
|
- **baseline 向**:不调 `_eval_version_on_pool`,改从基线 predictions(`baseline_run_id`)读 test 题对错算 acc(test 题在 infer_adhoc 已全推理过);结果存 memo,epoch>1 直接复用(0 推理)。
|
||||||
|
- **final 向**:真评 600,算完存 `memo[(final_sv,final_pv)]`。
|
||||||
|
- **best_hard 向**:若 `(best_sv,best_pv)` 已在 `memo`(== final 或往轮已评)则引用,否则真评并存 memo。
|
||||||
|
- **best_mixed 向**:`_pick_mixed_best` 选出的赢家必是 best_hard 或 final 之一,其 test acc 已在 memo,直接引用写 holdout_eval,0 推理。
|
||||||
|
|
||||||
|
实现前完整读 `_holdout_four_way`(~1885)、`write_holdout_eval`、`_eval_version_on_pool` 对齐;`write_holdout_eval` 调用保持不变(仍逐向写观测行,只是 acc 来源改为 memo 复用/基线推导)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 测试通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_runner.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py tests/unit/test_harness_runner.py
|
||||||
|
git commit -m "perf: dedup holdout four-way eval (baseline derive, best_hard memo)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 9: 训练推理注入 epoch 盐(cache_salt=run_id,算法 #10 透传)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/agent/loop.py:103-141,271-329`
|
||||||
|
- Modify: `app/harness/inference.py:408-415,473`
|
||||||
|
- Test: `tests/unit/test_agent_loop`(或现有 loop 测试)、`tests/unit/test_harness_inference.py`
|
||||||
|
|
||||||
|
> **原理**:训练/val/test/holdout 推理的 run_id 已含 `_e{epoch}`(如 `{base}_e{epoch}_s{step}`、`{run_id}_holdout_{kind}_e{epoch}`),用 run_id 作 cache_salt 即天然跨 epoch 重采样、同 epoch 续跑仍命中。judge/evolve 不经此路径(默认 salt=None)。gate 基线臂走 BaselineCache 不受影响;候选臂 messages 含 skill 版本天然区分。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_agent_loop_forwards_cache_salt():
|
||||||
|
"""AgentLoop.run(cache_salt=...) 透传到 llm.chat。"""
|
||||||
|
from core.agent.loop import AgentLoop
|
||||||
|
# fake llm 记录 chat 收到的 cache_salt kwarg;loop.run(..., cache_salt='run:e2')
|
||||||
|
# 断言 fake_llm.chat 收到 cache_salt='run:e2'
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest -k agent_loop_forwards_cache_salt -v`
|
||||||
|
Expected: FAIL(`run()` 无 cache_salt 参数)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: AgentLoop 透传 cache_salt**
|
||||||
|
|
||||||
|
`core/agent/loop.py`:`run`(L103)、`_step`/`_call_llm`(L271,317)签名加 `cache_salt: str | None = None`(keyword,随 session_id 透传);L329 `self._llm.chat(messages, session_id=session_id)` 改为 `self._llm.chat(messages, session_id=session_id, cache_salt=cache_salt)`。
|
||||||
|
|
||||||
|
- [ ] **Step 4: inference 用 run_id 作 salt**
|
||||||
|
|
||||||
|
`app/harness/inference.py` `_run_single_question`:`loop.run(...)`(L409)加 `cache_salt=run_id`(run_id 从 run_inference 透传到每题;`_run_single_question` 已有 run_id 上下文——若无则从 run_inference 参数透传)。确认 run_inference→_run_single_question 的 run_id 传递链完整。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 保真检查点 + 测试通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_inference.py tests/integration/test_agent_governed_e2e.py -q`
|
||||||
|
Expected: 全 PASS。保真(算法 #10):确认只加透传参数,Thinking+JSON/json_repair/pluggy hook 逻辑不变。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/agent/loop.py app/harness/inference.py tests/unit/
|
||||||
|
git commit -m "feat: inject run_id as cache_salt for per-epoch resampling (algo #10)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review(作者自查,执行者复核)
|
||||||
|
|
||||||
|
- [ ] traces 适配后诊断拿到真实轨迹(算法 #7 恢复)。
|
||||||
|
- [ ] prediction 归一化 None 保留、非标量序列化;insert 绑定异常不击穿 gather。
|
||||||
|
- [ ] early_stop 字段全局改名一致、计数改 epoch。
|
||||||
|
- [ ] 预检剔除不可训练题型(test 池不动)。
|
||||||
|
- [ ] patch 查整跨度 + 注入 + marker 完整性三防线(算法 #8 保护方向不变)。
|
||||||
|
- [ ] None/degraded 按 lapse 保守分流;降级占比>50% 中止。
|
||||||
|
- [ ] _run_step 幂等;gate_pools 与 checkpoint 的 gate_epoch_observed 同刻一致。
|
||||||
|
- [ ] holdout 去重后 baseline 0 推理、best_hard 备忘录 resume 可 hydrate。
|
||||||
|
- [ ] cache_salt=run_id 贯穿 AgentLoop,run_id 含 epoch 保证跨 epoch 重采样。
|
||||||
|
|
||||||
|
## 核心算法保真校验结论
|
||||||
|
|
||||||
|
触及算法 #5/#7/#8/#10/#12,均为恢复(#7 轨迹)/加固(#8 跨度)/透传(#10 salt)/时序(#5/#12 checkpoint),各 Task 已设保真检查点,不改算法核心逻辑。Task 5/7/9 需在实现时对照 TRM4 参考确认无行为漂移。
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
|
||||||
|
1. `pytest tests/unit/test_harness_runner.py tests/unit/test_harness_inference.py tests/unit/test_diagnose.py tests/unit/test_patch.py tests/unit/test_harness_checkpoint.py tests/unit/test_baseline_run_log.py tests/unit/test_runner_diag_tree_inject.py` 全绿。
|
||||||
|
2. 诊断拿到非空轨迹;非标量 prediction 不崩;early_stop 按 epoch。
|
||||||
|
3. 微型题型被预检剔除;patch 不破坏 marker;降级题不驱动进化。
|
||||||
|
4. _run_step 幂等;cache_salt=run_id 贯穿。
|
||||||
@@ -0,0 +1,624 @@
|
|||||||
|
# WP4 韧性与持久化 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 修复 LLM 治理栈与持久化层的 10 处韧性缺陷,使训练信号不被缓存重放/截断响应/断连/INFRA 故障污染,且 workspace 元数据崩溃可恢复。
|
||||||
|
|
||||||
|
**Architecture:** 缓存加 salt 维度让跨 epoch 评估真实重采样(同 epoch 续跑仍命中);SSE 未收 `[DONE]` 视为截断进重试且不写缓存;扩展瞬时错误清单覆盖断连族;熔断半开只放一个探针;gate 基线臂 INFRA 故障不污染 BaselineCache(算法 #6 保真区,逐行比对);manifest 补原子写;只读查询不改基线元数据。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、httpx、Redis、asyncio、SQLite、pytest、pydantic-settings。
|
||||||
|
|
||||||
|
**设计源**:`research-wiki/designs/2026-07-16-preflight-fixes-design.md §8`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键锚点(实现前必读)
|
||||||
|
|
||||||
|
| 用途 | 位置 |
|
||||||
|
|------|------|
|
||||||
|
| chat 缓存/重试 | `adapters/llm.py:271` chat(get L299 / set L373 / 重试 L336);`_call_streaming:506`;`_consume_stream:533`(不校验 done);`_iter_sse_deltas:94`(done 标志 L116);`_is_transient_error:173-186`;`_SseAnomaly:40` |
|
||||||
|
| 缓存键 | `adapters/redis_cache.py:32` `_build_key`(无 salt);`get:50`;`set:73`(ttl 分支 L89-92) |
|
||||||
|
| Protocol | `core/protocols.py:22-28` `LLMProvider.chat` |
|
||||||
|
| TTL | `main.py:41` `redis_cache_ttl=86400`;`:93` `ttl_s = ... if >0 else None` |
|
||||||
|
| 熔断 | `adapters/breaker.py:24` `is_open`(L36-37 到期即放行,无探针锁);`record_failure:39`;`record_success:63` |
|
||||||
|
| gate INFRA | `app/harness/validate.py:257` `_resolve_baseline_block`(put L303-304);`_check_infra_guard:384`(累计检查 L593);`BaselineCache` in `gate_ladder.py:344`(put L384 原子 L401-403) |
|
||||||
|
| 持久化 | `app/harness/workspace.py` 4 处非原子 `manifest.json` write_text(WP2 后行号):`_scaffold:111`/`update_manifest:289`/`record_run:318`/`update_best:372`;原子范式 `checkpoint.py:257-260`。**以 `grep -n 'manifest.json").write_text' app/harness/workspace.py` 现场定位为准** |
|
||||||
|
| 基线元数据 | `app/harness/log.py:57` `HarnessLog.__init__`(upsert `_runs` L73-82);`RunLogImpl._read_table:247`(只读) |
|
||||||
|
| dual_metric | `app/harness/observation.py:119` `write_dual_metric`(version_kind final L138);runner 调用 `runner.py:1444-1448` |
|
||||||
|
| 配置 | `.env:65` `REDIS_CACHE_TTL=0`;`.env.example:49` `=86400` |
|
||||||
|
|
||||||
|
## 核心算法保真校验
|
||||||
|
|
||||||
|
触及算法 #6(块顺序验证:基线缓存/INFRA 护栏/配对翻转)——Task 6(gate 基线臂 INFRA 隔离)改 `_resolve_baseline_block` 与护栏时序,**必须逐行比对** TRM4 `/home/iomgaa/Projects/Video-Tree-TRM4/core/harness/validate.py` 的 INFRA 护栏语义,确保只改"INFRA unit 不写缓存/不计 W/L + 护栏前置",不动配对翻转与基线快照复用逻辑。其余 task 不触碰核心算法。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: 缓存 cache_salt 贯穿
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/protocols.py:22-28`
|
||||||
|
- Modify: `adapters/llm.py:271-373`
|
||||||
|
- Modify: `adapters/redis_cache.py:32-92`
|
||||||
|
- Modify: VLM 适配转发(`adapters/` 内实现 `LLMProvider`/转发 chat 的类,grep 定位)
|
||||||
|
- Test: `tests/unit/test_redis_cache.py`、`tests/unit/test_governed_llm.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(salt 入键)**
|
||||||
|
|
||||||
|
在 `tests/unit/test_redis_cache.py` 追加:
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_salt_changes_key(fake_redis):
|
||||||
|
from adapters.redis_cache import RedisResponseCache
|
||||||
|
|
||||||
|
cache = RedisResponseCache(redis=fake_redis, ttl_s=None)
|
||||||
|
k_none = cache._build_key("m", [{"role": "user", "content": "x"}], None)
|
||||||
|
k_e1 = cache._build_key("m", [{"role": "user", "content": "x"}], "run:e1")
|
||||||
|
k_e2 = cache._build_key("m", [{"role": "user", "content": "x"}], "run:e2")
|
||||||
|
assert k_none != k_e1 != k_e2 and k_e1 != k_e2
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_redis_cache.py::test_salt_changes_key -v`
|
||||||
|
Expected: FAIL(`_build_key() takes 3 positional arguments but 4 were given`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: redis_cache 加 salt 维度**
|
||||||
|
|
||||||
|
`adapters/redis_cache.py`:
|
||||||
|
- `_build_key(self, model, messages, cache_salt: str | None = None)`:**仅当 `cache_salt is not None` 才加入 `salt` 字段**(默认 None 时 payload 结构与现状一字节不差,旧缓存键不失效):
|
||||||
|
```python
|
||||||
|
key_obj: dict = {"model": model, "messages": messages}
|
||||||
|
if cache_salt is not None:
|
||||||
|
key_obj["salt"] = cache_salt
|
||||||
|
payload = json.dumps(key_obj, sort_keys=True, ensure_ascii=False)
|
||||||
|
```
|
||||||
|
- `get(self, model, messages, cache_salt: str | None = None)`:`key = self._build_key(model, messages, cache_salt)`
|
||||||
|
- `set(self, model, messages, response, cache_salt: str | None = None)`:同样传 `cache_salt`
|
||||||
|
|
||||||
|
- [ ] **Step 4: Protocol + chat 透传 salt**
|
||||||
|
|
||||||
|
`core/protocols.py` `LLMProvider.chat`(L22)签名加 `cache_salt: str | None = None`(keyword-only,放 `parent_call_id` 后)。
|
||||||
|
`adapters/llm.py` `chat`(L271)签名加同参;L299 改 `await self._cache.get(self._model, messages, cache_salt)`;L373 改 `await self._cache.set(self._model, messages, response, cache_salt)`。
|
||||||
|
`adapters/redis_cache.py` `get`(L50)/`set`(L73)签名各加 `cache_salt: str | None = None`,内部 `_build_key(model, messages, cache_salt)`。
|
||||||
|
**VLM 转发**:`core/protocols.py` `VLMProvider.chat_with_images`(L35)与 `adapters/vlm.py` `GovernedVLMClient.chat_with_images`(L32)签名加 keyword-only `cache_salt: str | None = None`,L53 转发 `self._llm.chat(..., cache_salt=cache_salt)`。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_redis_cache.py tests/unit/test_governed_llm.py -q`
|
||||||
|
Expected: 全 PASS(默认 `cache_salt=None` 保持旧键,既有缓存测试不受影响)。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/protocols.py adapters/llm.py adapters/redis_cache.py tests/unit/test_redis_cache.py
|
||||||
|
git commit -m "feat: add cache_salt dimension to LLM response cache"
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注:训练链路注入 epoch 盐(`f"{run_id}:e{epoch}"`)在 WP3 的推理调用点完成(本 WP 只提供 salt 能力)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: TTL 语义修正(≤0 报错)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `adapters/redis_cache.py`(新增 `_resolve_cache_ttl`)
|
||||||
|
- Modify: `main.py:93`、`app/harness/video_split_cli.py:257`(两处 TTL 入口共用)
|
||||||
|
- Modify: `.env:65`、`.env.example:49`
|
||||||
|
- Test: 新增 `tests/unit/test_infra_settings.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
新增 `tests/unit/test_infra_settings.py`:
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def test_redis_cache_ttl_zero_rejected():
|
||||||
|
"""REDIS_CACHE_TTL<=0 必须启动即报错,消灭'0=永不过期'隐式语义。"""
|
||||||
|
from adapters.redis_cache import _resolve_cache_ttl # Step 3 抽出的纯函数
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="REDIS_CACHE_TTL"):
|
||||||
|
_resolve_cache_ttl(0)
|
||||||
|
with pytest.raises(ValueError, match="REDIS_CACHE_TTL"):
|
||||||
|
_resolve_cache_ttl(-1)
|
||||||
|
assert _resolve_cache_ttl(86400) == 86400
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_infra_settings.py -v`
|
||||||
|
Expected: FAIL(`cannot import name '_resolve_cache_ttl'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 抽出 fail-loud 并替换两处 TTL 入口**
|
||||||
|
|
||||||
|
`adapters/redis_cache.py` 新增模块级纯函数(配置校验贴近 cache 实现、避免 app→main 依赖):
|
||||||
|
```python
|
||||||
|
def _resolve_cache_ttl(ttl: int) -> int:
|
||||||
|
"""校验 Redis 缓存 TTL:必须为正整数(消灭 0=永不过期 的隐式语义)。"""
|
||||||
|
if ttl <= 0:
|
||||||
|
raise ValueError(
|
||||||
|
f"REDIS_CACHE_TTL 必须为正整数秒,实际 {ttl}。"
|
||||||
|
"训练场景建议 >= 单次训练时长(如 86400)。"
|
||||||
|
)
|
||||||
|
return ttl
|
||||||
|
```
|
||||||
|
`main.py:93` 改为 `ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)`(import 上述函数,删 `if >0 else None`)。
|
||||||
|
`app/harness/video_split_cli.py:257`(`_build_redis_cache` 内)同样从 `ttl_s = ... if > 0 else None` 改为 `ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)`——消灭第二条 `0=永不过期` 入口。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 改 .env / .env.example**
|
||||||
|
|
||||||
|
`.env:65` `REDIS_CACHE_TTL=0` → `REDIS_CACHE_TTL=86400`。
|
||||||
|
`.env.example:49` 确认为正整数(当前 86400,OK);补注释 `# 正整数秒,禁止 0(0 会被拒绝启动)`。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行测试确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_infra_settings.py -v`
|
||||||
|
Expected: PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add main.py .env.example tests/unit/test_infra_settings.py
|
||||||
|
git commit -m "fix: reject REDIS_CACHE_TTL<=0 (kill implicit never-expire)"
|
||||||
|
```
|
||||||
|
> 注:`.env` 不提交(gitignore);改动需手动同步到运行环境,runbook 会提示。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: SSE 截断检测(未收 [DONE] 即重试,不写缓存)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `adapters/llm.py:533-578`(_consume_stream)
|
||||||
|
- Test: `tests/unit/test_governed_llm.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_governed_llm.py` 追加:
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_truncated_stream_without_done_raises(_build_client):
|
||||||
|
"""SSE 流耗尽但未收 [DONE] → _SseAnomaly(进重试,不当成功)。"""
|
||||||
|
from adapters.llm import _SseAnomaly, GovernedLLMClient
|
||||||
|
|
||||||
|
async def _lines():
|
||||||
|
yield 'data: {"choices":[{"delta":{"content":"半"}}]}'
|
||||||
|
# 无 data: [DONE] —— 模拟服务端截断
|
||||||
|
|
||||||
|
client = _build_client() # 依现有 fixture
|
||||||
|
with pytest.raises(_SseAnomaly):
|
||||||
|
await client._consume_stream(_lines())
|
||||||
|
```
|
||||||
|
> 依 `test_governed_llm.py` 现有 `_build_client`(L50)/`_FakeRedisCache` 构造;实现前读该文件对齐 client 构造签名。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_governed_llm.py -k truncated_stream -v`
|
||||||
|
Expected: FAIL(当前耗尽即正常返回,不抛)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: _consume_stream 校验 done**
|
||||||
|
|
||||||
|
`adapters/llm.py` `_consume_stream` 在 `return content, thinking, ttft_ms, max_inter_token_ms, usage`(L578)之前加:
|
||||||
|
```python
|
||||||
|
if not usage_sink.get("done"):
|
||||||
|
raise _SseAnomaly("truncated_no_done")
|
||||||
|
```
|
||||||
|
(`_iter_sse_deltas` 收到 `[DONE]` 时置 `usage_sink["done"]=True`,L116;未置说明流被截断。`_SseAnomaly` 已在 `_is_transient_error` L186 归为可重试,故自动进重试梯且不走成功路径、不写缓存。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_governed_llm.py tests/unit/test_streaming.py -q`
|
||||||
|
Expected: 全 PASS(正常流带 `[DONE]` 的既有测试仍通过;若既有 streaming 测试的假流未含 `[DONE]`,同步补 `data: [DONE]` 终帧)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add adapters/llm.py tests/unit/test_governed_llm.py
|
||||||
|
git commit -m "fix: treat SSE stream without [DONE] as truncated (retry, no cache)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: 瞬时错误清单扩展(断连族)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `adapters/llm.py:182-186`
|
||||||
|
- Test: `tests/unit/test_governed_llm.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_governed_llm.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_transient_covers_disconnect_family():
|
||||||
|
import httpx
|
||||||
|
from adapters.llm import _is_transient_error
|
||||||
|
|
||||||
|
assert _is_transient_error(httpx.RemoteProtocolError("peer reset"))
|
||||||
|
assert _is_transient_error(httpx.ReadError("read"))
|
||||||
|
assert _is_transient_error(httpx.ConnectTimeout("ct"))
|
||||||
|
assert _is_transient_error(httpx.PoolTimeout("pt"))
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_governed_llm.py -k disconnect_family -v`
|
||||||
|
Expected: FAIL(RemoteProtocolError/PoolTimeout 不在当前清单)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 扩展 _is_transient_error**
|
||||||
|
|
||||||
|
`adapters/llm.py` L182-183 的:
|
||||||
|
```python
|
||||||
|
if isinstance(exc, (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout)):
|
||||||
|
return True
|
||||||
|
```
|
||||||
|
改为(两族基类覆盖 RemoteProtocolError/ReadError/ConnectTimeout/PoolTimeout 等):
|
||||||
|
```python
|
||||||
|
if isinstance(exc, (httpx.TimeoutException, httpx.TransportError)):
|
||||||
|
return True
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_governed_llm.py -q`
|
||||||
|
Expected: 全 PASS(`httpx.HTTPStatusError` 非 TransportError 子类,401/403 致命分支不受影响)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add adapters/llm.py tests/unit/test_governed_llm.py
|
||||||
|
git commit -m "fix: cover httpx disconnect family in transient error set"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 5: 熔断半开单探针锁
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `adapters/breaker.py:18-70`
|
||||||
|
- Test: `tests/unit/test_breaker.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_breaker.py::TestCircuitBreaker` 追加:
|
||||||
|
```python
|
||||||
|
def test_half_open_admits_single_probe(self):
|
||||||
|
from adapters.breaker import CircuitBreaker
|
||||||
|
|
||||||
|
b = CircuitBreaker(fail_threshold=2, cooldown_s=10.0)
|
||||||
|
b.record_failure("p", now=0.0)
|
||||||
|
b.record_failure("p", now=0.0) # 开路至 t=10
|
||||||
|
assert b.is_open("p", now=5.0) is True # 冷却中
|
||||||
|
# 冷却到期:只放行第一个探针
|
||||||
|
assert b.is_open("p", now=11.0) is False # 探针 1 放行
|
||||||
|
assert b.is_open("p", now=11.0) is True # 探针 2 被挡(探针在途)
|
||||||
|
b.record_success("p") # 探针成功 → 闭合
|
||||||
|
assert b.is_open("p", now=12.0) is False
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_breaker.py -k half_open_admits_single -v`
|
||||||
|
Expected: FAIL(当前到期后所有调用都返回 False)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现半开单探针锁**
|
||||||
|
|
||||||
|
`adapters/breaker.py`:
|
||||||
|
- `__init__` 加 `self._half_open_inflight: dict[str, bool] = {}`
|
||||||
|
- `is_open` 改为:
|
||||||
|
```python
|
||||||
|
def is_open(self, source_name: str, now: float) -> bool:
|
||||||
|
until = self._open_until.get(source_name)
|
||||||
|
if until is None:
|
||||||
|
return False
|
||||||
|
if now < until:
|
||||||
|
return True # 冷却中,全挡
|
||||||
|
# 冷却到期:half-open,只放行一个探针
|
||||||
|
if self._half_open_inflight.get(source_name):
|
||||||
|
return True # 已有探针在途,继续挡
|
||||||
|
self._half_open_inflight[source_name] = True
|
||||||
|
return False
|
||||||
|
```
|
||||||
|
- `record_success` 加 `self._half_open_inflight.pop(source_name, None)`(探针成功 → 清在途 + 已有的清 fails/open_until)
|
||||||
|
- `record_failure`:探针失败会累计并可能重开路;末尾加 `self._half_open_inflight.pop(source_name, None)`(让下一轮 cooldown 后可再探)
|
||||||
|
- `force_open`:加 `self._half_open_inflight.pop(source_name, None)`
|
||||||
|
|
||||||
|
(is_open 在 asyncio 单线程内同步执行,"检查+标记探针"原子,无竞态。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_breaker.py tests/unit/test_governed_llm.py -q`
|
||||||
|
Expected: 全 PASS(既有 circuit_open 测试若假设"到期即多次放行"需同步更新为单探针语义)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add adapters/breaker.py tests/unit/test_breaker.py
|
||||||
|
git commit -m "fix: half-open circuit admits single probe (no thundering herd)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 6: gate 基线臂 INFRA 隔离(算法 #6 保真区)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/validate.py:257-311`(_resolve_baseline_block)+ 护栏时序
|
||||||
|
- Test: `tests/unit/test_harness_validate.py`
|
||||||
|
|
||||||
|
> **保真前置**:实现前完整读 `app/harness/validate.py:257-311` `_resolve_baseline_block`、`_candidate_correctness_from_db`、块循环(575-600),并逐行比对 TRM4 `core/harness/validate.py` 的 INFRA 护栏语义。目标仅为:① 基线臂 INFRA 故障(stop_reason∈{error,parse_error})的 unit **不写 BaselineCache**、**不计入 W/L 翻转**;② 护栏检查移到"写缓存之前"。不得改动配对翻转、基线快照复用、unit 折叠逻辑。
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_validate.py` 追加(用 fake run_inference 让某基线 unit 返回 stop_reason=error):
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_baseline_infra_error_not_cached(tmp_path):
|
||||||
|
"""基线臂 INFRA error 的 unit 不写入 BaselineCache(不永久污染)。"""
|
||||||
|
# 依现有 test_harness_validate.py 的 fake run_inference / BaselineCache fixture 构造;
|
||||||
|
# 让 miss unit u1 的推理返回 prediction=None, stop_reason='error';
|
||||||
|
# 调 _resolve_baseline_block 后断言 baseline_cache.get(..., u1) is None
|
||||||
|
...
|
||||||
|
```
|
||||||
|
> 依 `test_harness_validate.py` 现有 fixture(fake `run_inference`、`BaselineCache`、`HarnessLog`);实现前读该文件对齐构造,勿臆造签名。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_validate.py -k baseline_infra_error_not_cached -v`
|
||||||
|
Expected: FAIL(当前 INFRA unit 的对错被 put 进缓存)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现 INFRA 隔离(精确数据路径)**
|
||||||
|
|
||||||
|
**数据路径**(Codex 审指出:当前 `_load_run_rows`(validate.py:212)只查 `question_id,prediction,answer,steps_json`,拿不到 per-unit stop_reason,只有汇总 `stop_reason_counts` 无法定位哪个 unit 是 INFRA):
|
||||||
|
1. `_load_run_rows`(validate.py:194-212)的 SELECT 增加 `stop_reason` 列;`_candidate_correctness_from_db` 相应可返回每题 stop_reason(或新增 `_infra_question_ids_from_db(log, run_id, questions)` 返回 stop_reason∈{"error","parse_error"} 的 qid 集)。
|
||||||
|
2. `_resolve_baseline_block`:miss 跑完后,计算本块 INFRA unit 集(unit 内**任一题** stop_reason∈{"error","parse_error"})。对这些 unit:**不 `baseline_cache.put`**、**不计入 `b_units`**、并从返回给调用方的有效 unit 集中排除。护栏所需 `errors_inc` 在 put 之前累计并调 `_check_infra_guard`(护栏前置于缓存写入)。
|
||||||
|
3. **贯穿调用方**(validate.py:575-600):`_resolve_baseline_block` 返回"有效 unit 子集 `valid_unit_chunk`",后续 `_run_candidate_block`、`unit_correctness_view`、`pair_block`、`_build_evidence_rows` 全部用 `valid_unit_chunk`(而非原始 `unit_chunk`),确保配对 `unit_ids` 与基线侧一致、不含 INFRA unit。
|
||||||
|
4. 保真:非 INFRA unit 的配对翻转、基线快照复用、unit 折叠逻辑一字不改;仅"把 INFRA unit 从本块整体剔除"。按 Step 前"保真前置"逐行比对 TRM4。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 保真回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_validate.py tests/unit/test_gate_block_unit.py tests/unit/test_gates.py -q`
|
||||||
|
Expected: 全 PASS(配对翻转/e-process 既有测试不受影响 = 保真达成)。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/validate.py tests/unit/test_harness_validate.py
|
||||||
|
git commit -m "fix: isolate gate baseline-arm INFRA errors from BaselineCache (algo #6)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 7: manifest 原子写
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/workspace.py`(`_scaffold:111`/`update_manifest:282`/`record_run:311`/`update_best:365`)
|
||||||
|
- Test: `tests/unit/test_harness_workspace.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_workspace.py` 追加:
|
||||||
|
```python
|
||||||
|
def test_update_manifest_is_atomic(workspace_dir, monkeypatch):
|
||||||
|
"""写 manifest 途中崩溃不产生半截 JSON(原子写:tmp 存在即失败也不损原文件)。"""
|
||||||
|
import json
|
||||||
|
from app.harness import workspace as ws
|
||||||
|
|
||||||
|
# 先建合法 manifest
|
||||||
|
... # 依现有 fixture 初始化 workspace
|
||||||
|
original = (workspace_dir / "manifest.json").read_text()
|
||||||
|
|
||||||
|
# monkeypatch os.replace 抛异常,模拟替换阶段崩溃
|
||||||
|
def _boom(src, dst):
|
||||||
|
raise OSError("crash during replace")
|
||||||
|
monkeypatch.setattr(ws.os, "replace", _boom)
|
||||||
|
with pytest.raises(OSError):
|
||||||
|
ws.update_manifest(workspace_dir, skills="skills/v2")
|
||||||
|
# 原 manifest 未被破坏
|
||||||
|
assert (workspace_dir / "manifest.json").read_text() == original
|
||||||
|
assert json.loads((workspace_dir / "manifest.json").read_text())
|
||||||
|
```
|
||||||
|
> 依 `test_harness_workspace.py` 现有 workspace 初始化 fixture;实现前读对齐。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_workspace.py -k update_manifest_is_atomic -v`
|
||||||
|
Expected: FAIL(当前 write_text 非原子,崩溃留半截)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 加原子写 helper 并替换 4 处**
|
||||||
|
|
||||||
|
`app/harness/workspace.py` 顶部确认 `import os`。新增模块级 helper:
|
||||||
|
```python
|
||||||
|
def _atomic_write_json(path: Path, data: dict) -> None:
|
||||||
|
"""原子写 JSON:tmp + os.replace(对齐 checkpoint.py 范式,防半截损坏)。"""
|
||||||
|
tmp = path.with_name(path.name + ".tmp")
|
||||||
|
tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
os.replace(tmp, path)
|
||||||
|
```
|
||||||
|
把 `_scaffold_workspace`(~L111)、`update_manifest`(~L289)、`record_run`(~L318)、`update_best`(~L372)四处的 `(workspace_dir / "manifest.json").write_text(json.dumps(...))` 替换为 `_atomic_write_json(workspace_dir / "manifest.json", <data>)`(用 `grep -n 'manifest.json").write_text' app/harness/workspace.py` 定位全部 4 处,行号随 WP2 改动漂移)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_workspace.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/workspace.py tests/unit/test_harness_workspace.py
|
||||||
|
git commit -m "fix: atomic writes for manifest/record_run/update_best (tmp+replace)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 8: 只读查询不改基线元数据
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/log.py:57-83`(HarnessLog.__init__ 加 register_run 开关)
|
||||||
|
- Modify: 只读查询基线的调用点(`app/harness/runner.py:923` `_init_gate`、`app/harness/pools.py:765,818`)
|
||||||
|
- Test: `tests/unit/test_harness_log.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
在 `tests/unit/test_harness_log.py::TestHarnessLogUpsert` 追加:
|
||||||
|
```python
|
||||||
|
def test_register_run_false_skips_upsert(self, tmp_path):
|
||||||
|
"""register_run=False 时只读打开不改写已有 _runs 行(started_at/status 不变)。"""
|
||||||
|
db = str(tmp_path / "h.db")
|
||||||
|
from app.harness.log import HarnessLog
|
||||||
|
|
||||||
|
with HarnessLog(db, "r1") as log:
|
||||||
|
log # 初次注册
|
||||||
|
row0 = _read_run_row(db, "r1") # 依现有 helper 读 started_at/status
|
||||||
|
|
||||||
|
with HarnessLog(db, "r1", register_run=False) as log:
|
||||||
|
log.query("SELECT 1") # 只读
|
||||||
|
row1 = _read_run_row(db, "r1")
|
||||||
|
assert row1["started_at"] == row0["started_at"]
|
||||||
|
assert row1["status"] == row0["status"]
|
||||||
|
```
|
||||||
|
> 依 `test_harness_log.py` 现有读行 helper;实现前读 `TestHarnessLogUpsert`(L196)对齐。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_log.py -k register_run_false -v`
|
||||||
|
Expected: FAIL(`unexpected keyword argument 'register_run'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: HarnessLog 加 register_run 开关**
|
||||||
|
|
||||||
|
`app/harness/log.py` `__init__` 签名加 `register_run: bool = True`(keyword)。把 L73-82 的 `_runs` upsert 包进 `if register_run:`;`register_run=False` 时跳过 upsert(仅 `_init_fixed_tables` 建表 + 连接,供只读查询)。`__exit__`/`close` 的 status 更新同样在 `register_run` 为 True 时才执行(避免只读关闭把 status 改 completed)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 只读调用点传 register_run=False**
|
||||||
|
|
||||||
|
`app/harness/runner.py:923`(_init_gate 用 baseline_run_id 只读查 predictions)、`app/harness/pools.py:765,818`(build_or_load_pools 只读查 predictions)三处 `HarnessLog(...)` 调用加 `register_run=False`。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_log.py tests/unit/test_harness_pools.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 6: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/log.py app/harness/runner.py app/harness/pools.py tests/unit/test_harness_log.py
|
||||||
|
git commit -m "fix: read-only baseline queries skip _runs upsert (register_run flag)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 9: dual_metric version_kind 口径修正
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:1444-1448`(第二次 write_dual_metric)
|
||||||
|
- Test: `tests/unit/test_harness_observation.py` 或 `test_harness_runner.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
`read_dual_metric` 真实签名为 `(db_path, *, run_id)`(无 epoch 参数,observation.py:164),故 observation 层 write/read 对任意 version_kind 都已支持——真正要验证的是 **runner 慢更新第二次写出的是 `slow_candidate` 而非 `final`**。在 `tests/unit/test_harness_observation.py` 追加 observation 层可区分性基线测试:
|
||||||
|
```python
|
||||||
|
def test_dual_metric_kinds_distinguishable(tmp_path):
|
||||||
|
from app.harness.observation import write_dual_metric, read_dual_metric
|
||||||
|
|
||||||
|
db = str(tmp_path / "h.db")
|
||||||
|
write_dual_metric(db, run_id="r", epoch=1, version_kind="final",
|
||||||
|
skills_version="v1", prompts_version="v1", pool="val",
|
||||||
|
hard_acc=0.7, soft_score=None, mixed_score=None)
|
||||||
|
write_dual_metric(db, run_id="r", epoch=1, version_kind="slow_candidate",
|
||||||
|
skills_version="v2", prompts_version="v2", pool="val",
|
||||||
|
hard_acc=0.6, soft_score=None, mixed_score=None)
|
||||||
|
rows = read_dual_metric(db, run_id="r") # 真实签名无 epoch,返回该 run 全部行
|
||||||
|
kinds = {row["version_kind"] for row in rows if row["epoch"] == 1}
|
||||||
|
assert kinds == {"final", "slow_candidate"}
|
||||||
|
```
|
||||||
|
> 实现前读 `test_harness_observation.py` 与 `write_dual_metric`/`read_dual_metric`(observation.py:119/164)确认参数名(epoch/pool/hard_acc 等)与返回行字段。真正的 `slow_candidate` 语义由 Step 3 改 runner 调用点落地。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行基线(observation 层已支持任意 kind)**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_observation.py -k dual_metric_kinds -v`
|
||||||
|
Expected: PASS(observation 层本就接受任意 version_kind);本 task 的实质改动在 Step 3 的 runner 调用点。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 改 runner 慢更新第二次写为 slow_candidate**
|
||||||
|
|
||||||
|
`app/harness/runner.py` L1444-1448 的 `write_dual_metric(..., version_kind="final", ...)`(`_slow_update_cycle` Phase 8 的 R2 行)改为 `version_kind="slow_candidate"`,使被 revert 的慢更新候选不再占用 `final` 语义。确认 Phase 2 的第一次(L1399-1403)仍为 `final`(epoch 终值唯一)。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_harness_observation.py tests/unit/test_harness_runner.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py tests/unit/test_harness_observation.py
|
||||||
|
git commit -m "fix: slow-update R2 dual_metric uses slow_candidate kind (not final)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 10: 离线 --retry-uncertain + docstring 修正
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/protocols.py:130`(`DiagnosisSignalStore.done_question_ids` 加参数)
|
||||||
|
- Modify: `adapters/baseline_diagnosis_store.py:109`(SqliteDiagnosisSignalStore 实现 + SQL)
|
||||||
|
- Modify: `app/harness/baseline_diagnosis.py:96`(透传 + docstring)
|
||||||
|
- Modify: `app/harness/video_split_cli.py`(build_arg_parser + run_pipeline 透传)
|
||||||
|
- Test: `tests/unit/`(对 SqliteDiagnosisSignalStore.done_question_ids)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试**
|
||||||
|
|
||||||
|
`done_question_ids` 的真实定义在 Protocol `core/evolution/protocols.py:130` 与实现 `adapters/baseline_diagnosis_store.py:109`(不在 baseline_diagnosis.py)。对 SqliteDiagnosisSignalStore 追加测试:
|
||||||
|
```python
|
||||||
|
def test_done_question_ids_retry_uncertain_excludes(tmp_path):
|
||||||
|
# 构造 baseline_diagnosis 表含 tier=T2 行与 tier=uncertain 行(同 run+fingerprint);
|
||||||
|
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||||
|
store = SqliteDiagnosisSignalStore(str(tmp_path / "h.db"))
|
||||||
|
# ... upsert 一条 T2、一条 uncertain ...
|
||||||
|
assert "<uncertain_qid>" in store.done_question_ids("run", "fp") # 默认含
|
||||||
|
assert "<uncertain_qid>" not in store.done_question_ids("run", "fp", retry_uncertain=True) # 排除
|
||||||
|
assert "<t2_qid>" in store.done_question_ids("run", "fp", retry_uncertain=True) # T2 仍算完成
|
||||||
|
```
|
||||||
|
> 实现前读 `adapters/baseline_diagnosis_store.py` 的 upsert/表结构与 `done_question_ids` SQL 对齐构造。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest -k done_question_ids_retry_uncertain -v`
|
||||||
|
Expected: FAIL(`unexpected keyword argument 'retry_uncertain'`)。
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现 retry_uncertain(贯穿 Protocol→Adapter→编排→CLI)**
|
||||||
|
|
||||||
|
- `core/evolution/protocols.py:130` `DiagnosisSignalStore.done_question_ids` 签名加 `retry_uncertain: bool = False`(keyword)。
|
||||||
|
- `adapters/baseline_diagnosis_store.py:109` 实现同签名;SQL 在 `retry_uncertain=True` 时追加 `AND tier != 'uncertain'`(uncertain 题不算完成,会被重诊)。
|
||||||
|
- `app/harness/baseline_diagnosis.py:96` `done = store.done_question_ids(baseline_run_id, diag_fingerprint)` 透传 `retry_uncertain`;修正模块 docstring 把"崩溃最多丢正在写的一行/逐行落库"改为实际的"run 末批量落库(Phase 3),崩溃丢本次 run 未落库结果,靠 Redis 缓存缓解重烧"。
|
||||||
|
- `app/harness/video_split_cli.py`:`build_arg_parser` 加 `--retry-uncertain`(store_true);`run_pipeline` 透传到 `run_baseline_diagnosis`。
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行测试确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM python -m pytest tests/unit/test_video_split_cli.py -q`
|
||||||
|
Expected: 全 PASS。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 提交**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/video_split_cli.py app/harness/baseline_diagnosis.py tests/unit/
|
||||||
|
git commit -m "feat: --retry-uncertain re-diagnoses uncertain rows; fix docstring"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review(作者自查,执行者复核)
|
||||||
|
|
||||||
|
- [ ] cache_salt 四处(protocols/llm/redis_cache/VLM 转发)贯通,默认 None 保持旧键。
|
||||||
|
- [ ] TTL≤0 fail-loud + .env/.env.example 同步(.env 手动,runbook 提示)。
|
||||||
|
- [ ] SSE 未收 [DONE] → _SseAnomaly(已在瞬时清单 → 重试 + 不写缓存)。
|
||||||
|
- [ ] 瞬时清单用 TimeoutException+TransportError 两族基类覆盖断连族,不误纳 HTTPStatusError。
|
||||||
|
- [ ] 熔断半开单探针:is_open 检查+标记原子(asyncio 单线程)。
|
||||||
|
- [ ] Task 6 触及算法 #6,已设保真前置(逐行比对 TRM4,只改 INFRA 不写缓存 + 护栏前置)。
|
||||||
|
- [ ] manifest 4 处原子写;只读查询 register_run=False 不改基线元数据。
|
||||||
|
|
||||||
|
## 核心算法保真校验结论
|
||||||
|
|
||||||
|
Task 6 触及算法 #6(块顺序验证),已在该 Task 设"保真前置"要求逐行比对 TRM4 `validate.py`,仅改 INFRA 隔离与护栏时序,不动配对翻转/基线快照/unit 折叠。其余 Task 均为治理栈/持久化层,不涉及核心算法。
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
|
||||||
|
1. `pytest tests/unit/test_redis_cache.py tests/unit/test_governed_llm.py tests/unit/test_breaker.py tests/unit/test_streaming.py tests/unit/test_harness_validate.py tests/unit/test_harness_workspace.py tests/unit/test_harness_log.py tests/unit/test_harness_observation.py tests/unit/test_infra_settings.py` 全绿。
|
||||||
|
2. 缓存加 salt 后跨 epoch 键不同、同 epoch 键相同。
|
||||||
|
3. 熔断半开只放一个探针;SSE 截断进重试不写缓存。
|
||||||
|
4. manifest 原子写;基线 _runs 元数据只读查询不被改写。
|
||||||
@@ -0,0 +1,632 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:fix-diagnosis-tree-data-link-plan
|
||||||
|
title: "实现计划: 修复诊断 tree_data 断链 bug"
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 修复诊断 tree_data 断链 bug Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** 接通 TRM4→TRM5 迁移时断掉的诊断树加载环,让 `evaluate_span` 拿到真实 ground_truth、error_type 归因不再坍缩。
|
||||||
|
|
||||||
|
**Architecture:** app 层新增树展平器(递归遍历 tree.json 的嵌套 roots → 扁平 `{"nodes":{id:{card,level,time_range}}}`),在离线诊断(`video_split_cli`)与训练循环(`runner`)两个注入点按诊断涉及的 video 加载填充 `tree_data`;core 侧把 `run_diagnosis` 的静默回退改为缺失即 fail-loud。core 只消费 dict,不碰归因瀑布(算法保真 §4.7#7)。
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.11、pytest、loguru、asyncio;参考 TRM4 `core/harness/diagnose.py:1677` 的 tree_cache 语义。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 设计来源
|
||||||
|
`research-wiki/designs/fix-diagnosis-tree-data-link.md`(已含 Codex 审查修订)。
|
||||||
|
|
||||||
|
## 文件结构映射
|
||||||
|
|
||||||
|
| 文件 | 动作 | 责任 |
|
||||||
|
|------|------|------|
|
||||||
|
| `app/harness/tree_nodes.py` | 新建 | 树展平器:`load_tree_nodes` + `load_tree_data_for_videos` |
|
||||||
|
| `core/evolution/diagnose.py` | 改 `:2144-2145` | 视频未覆盖即 raise(fail-loud,唯一 core 改动) |
|
||||||
|
| `app/harness/video_split_cli.py` | 改 `build_diagnosis_deps`(`:262-337`) + `_execute_real`(`:590-600`) | 离线注入:wrong_ids→video 加载填充 tree_data |
|
||||||
|
| `app/harness/runner.py` | 改 `_run_diagnosis`(`:2163-2187`) | 训练注入:question_ids→video 加载注入 |
|
||||||
|
| `tests/unit/test_tree_nodes.py` | 新建 | 展平器单测 |
|
||||||
|
| `tests/integration/test_baseline_diagnosis.py` | 改 `:99` | 更新传 `tree_data={}` 的用例为真实/伪造树 |
|
||||||
|
| `tests/integration/test_diagnosis_tree_link.py` | 新建 | ground_truth 接通 + 依赖方向 |
|
||||||
|
|
||||||
|
## 关键代码事实(Codex 已核验)
|
||||||
|
- 仅 `L1Node` 有 `to_dict`(`app/tree/index.py:260`);L2/L3 为其内部闭包;输出无 `level`、L3 用 `timestamp` 无 `time_range` → **不走对象层,直接遍历 json**。
|
||||||
|
- node_id 累积式 `..._L1_000_L2_000_L3_000` → level **按遍历深度赋值**,不解析 node_id。
|
||||||
|
- `GeneratedQuestion.video_id: str`(`core/types.py:60`);`load_questions_by_id(dir) -> dict[str, GeneratedQuestion]`(`video_split_cli.py:443`)。
|
||||||
|
- `run_diagnosis` else 分支已支持 `{video_id: {...}}` 形态(`diagnose.py:2068-2074`)。
|
||||||
|
- `diagnose.py:2145` 的 `td = ...get(vid,{})` 在 `:2148` try 之前 → 在此处 raise **不会**被 `:2159` 的 `except ValueError`(judge 降级)吞。
|
||||||
|
- store 根 = `Path("store")`,tree.json 在 `store/videos/<vid>/tree.json`(与 `factory.py:85` 一致)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: 树展平器 `app/harness/tree_nodes.py`
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `app/harness/tree_nodes.py`
|
||||||
|
- Test: `tests/unit/test_tree_nodes.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(正确性 + level + fail-loud)**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_tree_nodes.py
|
||||||
|
"""树展平器单测:用真实 store/videos/0RxMZBLeqRI/tree.json 验证展平正确性与 fail-loud。"""
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.harness.tree_nodes import load_tree_data_for_videos, load_tree_nodes
|
||||||
|
|
||||||
|
_STORE = Path("store")
|
||||||
|
_VID = "0RxMZBLeqRI" # 真实样本,111 节点
|
||||||
|
|
||||||
|
|
||||||
|
def _recursive_count(tree_json: dict) -> int:
|
||||||
|
def walk(n: dict) -> int:
|
||||||
|
return 1 + sum(walk(c) for c in (n.get("children") or []))
|
||||||
|
return sum(walk(r) for r in tree_json["roots"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_tree_nodes_flattens_all_nodes():
|
||||||
|
result = load_tree_nodes(_STORE, _VID)
|
||||||
|
assert set(result.keys()) == {"nodes"}
|
||||||
|
nodes = result["nodes"]
|
||||||
|
raw = json.loads((_STORE / "videos" / _VID / "tree.json").read_text(encoding="utf-8"))
|
||||||
|
assert len(nodes) == _recursive_count(raw)
|
||||||
|
sample = next(iter(nodes.values()))
|
||||||
|
assert set(sample.keys()) == {"card", "level", "time_range"}
|
||||||
|
assert isinstance(sample["card"], dict)
|
||||||
|
|
||||||
|
|
||||||
|
def test_level_assigned_by_depth_not_node_id():
|
||||||
|
nodes = load_tree_nodes(_STORE, _VID)["nodes"]
|
||||||
|
l1_id = f"{_VID}_L1_000"
|
||||||
|
l3_id = f"{_VID}_L1_000_L2_000_L3_000"
|
||||||
|
assert nodes[l1_id]["level"] == 1
|
||||||
|
assert nodes[l3_id]["level"] == 3 # 若按 node_id 首个 _L\d_ 会误判成 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_tree_raises_file_not_found():
|
||||||
|
with pytest.raises(FileNotFoundError):
|
||||||
|
load_tree_nodes(_STORE, "__no_such_video__")
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_roots_raises_value_error(tmp_path):
|
||||||
|
vdir = tmp_path / "videos" / "vX"
|
||||||
|
vdir.mkdir(parents=True)
|
||||||
|
(vdir / "tree.json").write_text(json.dumps({"metadata": {}, "roots": []}), encoding="utf-8")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
load_tree_nodes(tmp_path, "vX")
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_for_videos_dedups():
|
||||||
|
data = load_tree_data_for_videos(_STORE, [_VID, _VID])
|
||||||
|
assert set(data.keys()) == {_VID}
|
||||||
|
assert data[_VID]["nodes"]
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_tree_nodes.py -q`
|
||||||
|
Expected: FAIL(`ModuleNotFoundError: No module named 'app.harness.tree_nodes'`)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 实现展平器**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# app/harness/tree_nodes.py
|
||||||
|
"""诊断侧树读取适配:把嵌套 tree.json 展平成诊断消费的扁平 nodes dict。
|
||||||
|
|
||||||
|
诊断编排(core/evolution/diagnose.py)期望 tree_data 形如
|
||||||
|
{"nodes": {node_id: {card, level, time_range}}},但 TRM5 建树产物
|
||||||
|
store/videos/<vid>/tree.json 是嵌套 {"metadata","roots":[...]}。本模块递归展平,
|
||||||
|
接通 TRM4→TRM5 迁移时断掉的 ground_truth 加载环。
|
||||||
|
|
||||||
|
不走 TreeIndex 对象层:仅 L1Node 有 to_dict(app/tree/index.py:260),L2/L3 为其内部闭包,
|
||||||
|
且 to_dict 输出无 level、L3 用 timestamp 无 time_range。直接遍历 json 更省且零改建树模块。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def load_tree_nodes(store_dir: Path, video_id: str) -> dict[str, Any]:
|
||||||
|
"""加载单视频 tree.json 并展平成扁平 nodes dict。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
store_dir: store 根目录(含 videos/<video_id>/tree.json)。
|
||||||
|
video_id: 视频标识。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{"nodes": {node_id: {"card": dict, "level": int, "time_range": list}}}。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
FileNotFoundError: tree.json 不存在(沿用 factory.py fail-loud 先例)。
|
||||||
|
ValueError: 树无有效 roots、节点缺 id、或展平后 nodes 为空。
|
||||||
|
|
||||||
|
关键实现:
|
||||||
|
level 由遍历深度赋值(root=1/child=2/孙=3),不解析 node_id——node_id 累积式
|
||||||
|
(..._L1_..._L2_..._L3_)用正则首匹配会把 L2/L3 误判成 1。
|
||||||
|
L3 无 time_range,用 timestamp 合成 [t, t]。
|
||||||
|
"""
|
||||||
|
tree_path = store_dir / "videos" / video_id / "tree.json"
|
||||||
|
if not tree_path.exists():
|
||||||
|
raise FileNotFoundError(f"树索引文件不存在: {tree_path}(诊断需真实树,P5 fail loud)")
|
||||||
|
tree = json.loads(tree_path.read_text(encoding="utf-8"))
|
||||||
|
roots = tree.get("roots")
|
||||||
|
if not isinstance(roots, list) or not roots:
|
||||||
|
raise ValueError(f"树无有效 roots: {tree_path}")
|
||||||
|
|
||||||
|
nodes: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def _walk(node: dict[str, Any], level: int) -> None:
|
||||||
|
node_id = node.get("id")
|
||||||
|
if not isinstance(node_id, str) or not node_id:
|
||||||
|
raise ValueError(f"节点缺 id: {tree_path}")
|
||||||
|
time_range = node.get("time_range")
|
||||||
|
if time_range is None:
|
||||||
|
ts = node.get("timestamp")
|
||||||
|
time_range = [ts, ts] if ts is not None else [0, 0]
|
||||||
|
nodes[node_id] = {
|
||||||
|
"card": node.get("card", {}),
|
||||||
|
"level": level,
|
||||||
|
"time_range": time_range,
|
||||||
|
}
|
||||||
|
for child in node.get("children", []) or []:
|
||||||
|
_walk(child, level + 1)
|
||||||
|
|
||||||
|
for root in roots:
|
||||||
|
_walk(root, 1)
|
||||||
|
|
||||||
|
if not nodes:
|
||||||
|
raise ValueError(f"展平后 nodes 为空: {tree_path}")
|
||||||
|
return {"nodes": nodes}
|
||||||
|
|
||||||
|
|
||||||
|
def load_tree_data_for_videos(store_dir: Path, video_ids: list[str]) -> dict[str, Any]:
|
||||||
|
"""按一组 video_id 去重加载展平树,供诊断按 video 注入。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
store_dir: store 根目录。
|
||||||
|
video_ids: 视频标识列表(可含重复,内部按首次出现顺序去重)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{video_id: {"nodes": {...}}}。
|
||||||
|
|
||||||
|
异常:
|
||||||
|
同 load_tree_nodes(任一视频树缺失/无效即 fail-loud)。
|
||||||
|
"""
|
||||||
|
return {vid: load_tree_nodes(store_dir, vid) for vid in dict.fromkeys(video_ids)}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_tree_nodes.py -q`
|
||||||
|
Expected: PASS(5 passed)
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/tree_nodes.py tests/unit/test_tree_nodes.py
|
||||||
|
git commit -m "feat: add tree.json flattener for diagnosis ground_truth"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: core 视频覆盖 fail-loud(`diagnose.py`)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `core/evolution/diagnose.py:2144-2145`
|
||||||
|
- Modify: `tests/integration/test_baseline_diagnosis.py:99`(更新受影响用例)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(缺树 video → raise,不降级)**
|
||||||
|
|
||||||
|
在 `tests/integration/test_baseline_diagnosis.py` 追加:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_diagnosis_raises_when_video_tree_missing():
|
||||||
|
"""诊断视频未被 tree_data 覆盖时 fail-loud(不静默回退、不走 judge 降级)。"""
|
||||||
|
from core.evolution.diagnose import run_diagnosis
|
||||||
|
|
||||||
|
q = _make_one_wrong_question(video_id="vMISS", question_id="vMISS-1")
|
||||||
|
with pytest.raises(ValueError, match="诊断视频树未覆盖"):
|
||||||
|
await run_diagnosis(
|
||||||
|
run_id="infer_adhoc",
|
||||||
|
questions=[q],
|
||||||
|
tree_data={"vOTHER": {"nodes": {}}}, # 故意不含 vMISS
|
||||||
|
llm=_FakeLLM(),
|
||||||
|
run_log=_fake_run_log_with_one_wrong(q),
|
||||||
|
skill_store=_FakeSkillStore(),
|
||||||
|
prompts=_fake_diagnose_prompts(),
|
||||||
|
concurrency=1,
|
||||||
|
question_ids=["vMISS-1"],
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
> helper(`_make_one_wrong_question` / `_fake_run_log_with_one_wrong` / `_FakeLLM` / `_FakeSkillStore` / `_fake_diagnose_prompts`):复用该测试文件已有的伪造装配;若无 `_make_one_wrong_question`,构造 `GeneratedQuestion(question_id="vMISS-1", video_id="vMISS", task_type="Object Reasoning", ...)` 并让 run_log 返回一条 `correct=0` 且带非空 `steps_json`(含一个 `view_node` 步)的 prediction。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/integration/test_baseline_diagnosis.py::test_run_diagnosis_raises_when_video_tree_missing -q`
|
||||||
|
Expected: FAIL(当前静默回退 `{}`,不抛异常)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 改 core 加 fail-loud**
|
||||||
|
|
||||||
|
`core/evolution/diagnose.py` 的 `_process_question`,把 `:2144-2145` 的:
|
||||||
|
|
||||||
|
```python
|
||||||
|
vid = prediction.get("video_id", "")
|
||||||
|
td = tree_data_by_video.get(vid, {})
|
||||||
|
```
|
||||||
|
|
||||||
|
改为(位置在 `:2148` 的 `try` 之前,故不被 `:2159` 的 `except ValueError` 吞):
|
||||||
|
|
||||||
|
```python
|
||||||
|
vid = prediction.get("video_id", "")
|
||||||
|
if vid not in tree_data_by_video:
|
||||||
|
# P5 fail-loud:诊断需真实树,调用方须为每个诊断视频加载 tree_data;
|
||||||
|
# 静默回退空树会让 ground_truth 恒空、error_type 归因坍缩(本次修复的根因)。
|
||||||
|
raise ValueError(
|
||||||
|
f"诊断视频树未覆盖: video_id={vid!r} 不在注入的 tree_data 中"
|
||||||
|
"(调用方须为每个诊断视频加载树,P5 fail loud)"
|
||||||
|
)
|
||||||
|
td = tree_data_by_video[vid]
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: 更新既有传 `tree_data={}` 的三处用例**
|
||||||
|
|
||||||
|
`tests/integration/test_baseline_diagnosis.py` 有**三处** `tree_data={}`(`:99` / `:224` / `:311`,形参分别在 `:66` / `:197` / `:283`)。逐处判断:
|
||||||
|
|
||||||
|
- 若该用例**真诊断题**(`wrong_ids` 非空、进 `run_diagnosis`)→ 改为覆盖其诊断 video 的伪造树:
|
||||||
|
```python
|
||||||
|
tree_data={"<该用例的 video_id>": {"nodes": {"<node_id>": {"card": {}, "level": 1, "time_range": [0, 0]}}}},
|
||||||
|
```
|
||||||
|
(`video_id`/`node_id` 填该用例 prediction 实际用的值。)
|
||||||
|
- 若该用例期望**"无题诊断"早返回**(`wrong_ids=[]`,不进 `run_diagnosis`)→ `tree_data={}` 可保留,并在该用例加一行注释说明豁免原因。
|
||||||
|
|
||||||
|
逐处核对:读每个用例构造的 prediction 是否 `correct=0` 且被诊断——是则改树,否则注释豁免。
|
||||||
|
|
||||||
|
- [ ] **Step 4b: 全仓兜底扫描其它 `tree_data={}` 调用点**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM grep -rn "tree_data={}\|tree_data = {}" tests/ app/`
|
||||||
|
对每个命中判断:进 `run_diagnosis` 且诊断非空题的必须提供覆盖树;dry-run `fake_deps`(`video_split_cli.py` 内,`wrong_ids=[]` 早返回)豁免。确保 core fail-loud 不误伤既有用例。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 运行确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/integration/test_baseline_diagnosis.py -q`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add core/evolution/diagnose.py tests/integration/test_baseline_diagnosis.py
|
||||||
|
git commit -m "fix: fail loud when diagnosis video tree not covered (algo #7 input)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: 离线注入(`video_split_cli.py`)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/video_split_cli.py`(新增 `_DEFAULT_STORE_DIR` + `--store-dir` CLI、`_resolve_paths` 返回 store_dir、改 `build_diagnosis_deps` 签名与 tree_data、`_execute_real` 传 store_dir/video_ids、删模块顶层假注释 `:19-21`)
|
||||||
|
- Modify: `tests/unit/test_video_split_cli.py:269-273` 与 `:290-292`(两处旧调用补 `store_dir`/`video_ids`,否则改签名后先抛 TypeError 而非期望的 SystemExit)
|
||||||
|
- Test: `tests/unit/test_video_split_cli_tree_inject.py`(新建)
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(build_diagnosis_deps 填充真实树)**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_video_split_cli_tree_inject.py
|
||||||
|
"""离线诊断注入:build_diagnosis_deps 按 video_ids 填充非空 tree_data。"""
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from app.harness.video_split_cli import build_diagnosis_deps
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_diagnosis_deps_loads_tree_for_videos():
|
||||||
|
with patch("app.harness.video_split_cli._DiagLLMSettings") as S, \
|
||||||
|
patch("adapters.llm.GovernedLLMClient"), \
|
||||||
|
patch("adapters.telemetry.SQLiteTelemetryRecorder"), \
|
||||||
|
patch("app.harness.video_split_cli._build_redis_cache", return_value=None):
|
||||||
|
s = S.return_value
|
||||||
|
s.search_llm_model = "deepseek-v4-pro"
|
||||||
|
s.search_llm_base_url = "http://x"
|
||||||
|
s.search_llm_api_key = "k"
|
||||||
|
s.llm_circuit_breaker_threshold = 32
|
||||||
|
s.llm_circuit_breaker_cooldown = 60
|
||||||
|
s.llm_timeout = s.llm_ttft_timeout = s.llm_inter_token_timeout = 60
|
||||||
|
s.llm_max_retries = 1
|
||||||
|
s.llm_retry_base_delay = s.llm_retry_max_delay = 1
|
||||||
|
deps = build_diagnosis_deps(
|
||||||
|
harness_db=Path("workspaces/default/harness.db"),
|
||||||
|
store_dir=Path("store"),
|
||||||
|
video_ids=["0RxMZBLeqRI"],
|
||||||
|
concurrency=1,
|
||||||
|
expected_model="deepseek-v4-pro",
|
||||||
|
)
|
||||||
|
assert "0RxMZBLeqRI" in deps.tree_data
|
||||||
|
assert deps.tree_data["0RxMZBLeqRI"]["nodes"]
|
||||||
|
```
|
||||||
|
|
||||||
|
> `GovernedLLMClient` / `SQLiteTelemetryRecorder` 在 `build_diagnosis_deps` 内是函数级 import,patch 其源模块(`adapters.llm` / `adapters.telemetry`)。测试目的仅验证 `deps.tree_data` 被真实树填充;若装配桩不足以走到 return,可进一步 patch `RunLogImpl`/`VersionedSkillStore`。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_video_split_cli_tree_inject.py -q`
|
||||||
|
Expected: FAIL(当前 `build_diagnosis_deps` 无 `store_dir`/`video_ids` 参数 → `TypeError`)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 加常量 + `--store-dir` + `_resolve_paths` + 删模块假注释 + 改签名 + 填 tree_data**
|
||||||
|
|
||||||
|
(a) `app/harness/video_split_cli.py` 路径常量区(`:65-67` 附近)新增:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_DEFAULT_STORE_DIR = Path("store") # tree.json 在 store/videos/<vid>/
|
||||||
|
```
|
||||||
|
|
||||||
|
(b) argparse(`:732` `--out-dir` 之后)新增:
|
||||||
|
|
||||||
|
```python
|
||||||
|
parser.add_argument("--store-dir", type=Path, default=None, dest="store_dir")
|
||||||
|
```
|
||||||
|
|
||||||
|
(c) `_resolve_paths`(`:582-587`)改为返回 4 元组(含 store_dir):
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _resolve_paths(args: argparse.Namespace) -> tuple[Path, Path, Path, Path]:
|
||||||
|
"""解析 harness_db / questions_dir / out_dir / store_dir(CLI 覆盖默认工程路径)。"""
|
||||||
|
harness_db = args.harness_db or _DEFAULT_HARNESS_DB
|
||||||
|
questions_dir = args.questions_dir or _DEFAULT_QUESTIONS_DIR
|
||||||
|
out_dir = args.out_dir or _DEFAULT_OUT_DIR
|
||||||
|
store_dir = args.store_dir or _DEFAULT_STORE_DIR
|
||||||
|
return harness_db, questions_dir, out_dir, store_dir
|
||||||
|
```
|
||||||
|
|
||||||
|
(d) 删模块顶层假注释:`app/harness/video_split_cli.py:19-21` 把 “+ tree_data={}(由诊断管线内部按需加载)” 改为 “+ tree_data 按 wrong_ids 涉及 video 预加载(store/videos/<vid>/tree.json 展平)”。
|
||||||
|
|
||||||
|
(e) `build_diagnosis_deps` 签名(`:262-264`)改为:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def build_diagnosis_deps(
|
||||||
|
*,
|
||||||
|
harness_db: Path,
|
||||||
|
store_dir: Path,
|
||||||
|
video_ids: list[str],
|
||||||
|
concurrency: int,
|
||||||
|
expected_model: str,
|
||||||
|
) -> DiagnosisDeps:
|
||||||
|
```
|
||||||
|
|
||||||
|
函数末尾 `return DiagnosisDeps(...)`(`:330-337`)改为按 video 加载(并删 docstring 里"tree_data={} 由诊断管线内部按需加载"假注释):
|
||||||
|
|
||||||
|
```python
|
||||||
|
from app.harness.tree_nodes import load_tree_data_for_videos
|
||||||
|
|
||||||
|
return DiagnosisDeps(
|
||||||
|
run_log=RunLogImpl(str(harness_db)),
|
||||||
|
llm=llm,
|
||||||
|
skill_store=VersionedSkillStore(_diagnosis_skills_dir()),
|
||||||
|
prompts=_load_diagnose_prompts(),
|
||||||
|
tree_data=load_tree_data_for_videos(store_dir, video_ids),
|
||||||
|
concurrency=concurrency,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: `_execute_real` 解包 store_dir 并算 video_ids 传入**
|
||||||
|
|
||||||
|
`app/harness/video_split_cli.py:592`(解包改 4 元组)+ `:595-600`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
harness_db, questions_dir, out_dir, store_dir = _resolve_paths(args)
|
||||||
|
if not harness_db.exists():
|
||||||
|
raise SystemExit(f"harness.db 不存在: {harness_db}(P5 fail loud)")
|
||||||
|
canonical_preds = load_canonical_predictions(harness_db, config.baseline_run_id)
|
||||||
|
wrong_ids = select_diagnosable_wrong_ids(canonical_preds)
|
||||||
|
questions = load_questions_by_id(questions_dir)
|
||||||
|
video_ids = [questions[qid].video_id for qid in wrong_ids]
|
||||||
|
deps = build_diagnosis_deps(
|
||||||
|
harness_db=harness_db,
|
||||||
|
store_dir=store_dir,
|
||||||
|
video_ids=video_ids,
|
||||||
|
concurrency=args.concurrency,
|
||||||
|
expected_model=config.model,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
> 全仓其它 `_resolve_paths(args)` 解包处(如 dry-run `_execute_dry` 路径若有)同步改 4 元组解包,避免 `ValueError: too many values to unpack`。先 `grep -n "_resolve_paths(args)" app/harness/video_split_cli.py` 逐处核对。
|
||||||
|
|
||||||
|
- [ ] **Step 5: 更新受签名影响的既有单测**
|
||||||
|
|
||||||
|
`tests/unit/test_video_split_cli.py:269-273` 与 `:290-292` 两处 `cli.build_diagnosis_deps(...)` 调用补必填参数(这俩测试验的是**凭证/模型漂移 fail-loud(SystemExit)**,该校验在 tree_data 加载之前,故 `video_ids` 传空即可):
|
||||||
|
|
||||||
|
```python
|
||||||
|
cli.build_diagnosis_deps(
|
||||||
|
harness_db=tmp_path / "h.db",
|
||||||
|
store_dir=tmp_path,
|
||||||
|
video_ids=[],
|
||||||
|
concurrency=2,
|
||||||
|
expected_model="deepseek-v4-pro",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
(两处调用都照此加 `store_dir=tmp_path, video_ids=[]`。)
|
||||||
|
|
||||||
|
- [ ] **Step 6: 运行确认通过 + 回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_video_split_cli_tree_inject.py tests/unit/test_video_split_cli.py tests/unit/test_generate_questions.py -q`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
- [ ] **Step 7: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/video_split_cli.py tests/unit/test_video_split_cli_tree_inject.py tests/unit/test_video_split_cli.py
|
||||||
|
git commit -m "fix: load real tree_data for offline diagnosis (video_split_cli)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: 训练注入(`runner._run_diagnosis`)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `app/harness/runner.py:2163-2187`
|
||||||
|
- Test: `tests/unit/test_runner_diag_tree_inject.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写失败测试(_run_diagnosis 注入非空树)**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/unit/test_runner_diag_tree_inject.py
|
||||||
|
"""训练循环诊断注入:_run_diagnosis 按 batch question_ids 加载真实树注入 run_diagnosis。"""
|
||||||
|
from unittest.mock import AsyncMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_diagnosis_injects_tree_data(runner_with_real_store):
|
||||||
|
"""question_ids 对应的 video 树被加载并作为 tree_data 传入 run_diagnosis。"""
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
async def _fake_run_diagnosis(**kwargs):
|
||||||
|
captured["tree_data"] = kwargs["tree_data"]
|
||||||
|
return _empty_diagnosis_result()
|
||||||
|
|
||||||
|
with patch("core.evolution.diagnose.run_diagnosis", new=AsyncMock(side_effect=_fake_run_diagnosis)):
|
||||||
|
await runner_with_real_store._run_diagnosis("infer_adhoc", question_ids=["604-2"])
|
||||||
|
|
||||||
|
assert "0RxMZBLeqRI" in captured["tree_data"] # 604-2 属于 0RxMZBLeqRI
|
||||||
|
assert captured["tree_data"]["0RxMZBLeqRI"]["nodes"]
|
||||||
|
```
|
||||||
|
|
||||||
|
> `runner_with_real_store` fixture:构造 `self._config.store_dir="store"`、`self._paths.questions_dir` 指向含 604-2 的真实 benchmark 的 runner(复用该测试目录已有 runner helper;若无,最小构造使 `load_benchmark` 能取到 604-2)。`_empty_diagnosis_result()`:返回 `DiagnosisResult` 空壳(error_attributions=[]、infra=[]、degraded=[])。`_run_diagnosis` 内 `run_diagnosis` 为函数级 import,patch 其源符号 `core.evolution.diagnose.run_diagnosis`。
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认失败**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_runner_diag_tree_inject.py -q`
|
||||||
|
Expected: FAIL(当前 `tree_data={}` → captured 不含 `0RxMZBLeqRI`)
|
||||||
|
|
||||||
|
- [ ] **Step 3: 改 `_run_diagnosis` 注入树**
|
||||||
|
|
||||||
|
`app/harness/runner.py:2172-2187`,在 `questions = load_benchmark(...)` 后、`run_diagnosis(...)` 调用处:
|
||||||
|
|
||||||
|
```python
|
||||||
|
questions = load_benchmark(self._paths.questions_dir)
|
||||||
|
run_log = RunLogImpl(str(self._paths.db_path))
|
||||||
|
skill_store = VersionedSkillStore(self._paths.skills_dir)
|
||||||
|
diagnose_prompts = self._load_diagnose_prompts()
|
||||||
|
|
||||||
|
from app.harness.tree_nodes import load_tree_data_for_videos
|
||||||
|
|
||||||
|
if question_ids is not None:
|
||||||
|
qid_set = set(question_ids)
|
||||||
|
video_ids = [q.video_id for q in questions if q.question_id in qid_set]
|
||||||
|
else:
|
||||||
|
video_ids = [q.video_id for q in questions]
|
||||||
|
tree_data = load_tree_data_for_videos(Path(self._config.store_dir), video_ids)
|
||||||
|
|
||||||
|
return await run_diagnosis(
|
||||||
|
run_id=run_id,
|
||||||
|
questions=questions,
|
||||||
|
tree_data=tree_data,
|
||||||
|
llm=self._llm,
|
||||||
|
run_log=run_log,
|
||||||
|
skill_store=skill_store,
|
||||||
|
prompts=diagnose_prompts,
|
||||||
|
concurrency=self._config.concurrency,
|
||||||
|
question_ids=question_ids,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
(删 `tree_data={}, # tree_data 由诊断管线内部按需加载` 假注释;确认文件顶部已 `from pathlib import Path`,否则补 import。)
|
||||||
|
|
||||||
|
- [ ] **Step 4: 运行确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/unit/test_runner_diag_tree_inject.py -q`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/harness/runner.py tests/unit/test_runner_diag_tree_inject.py
|
||||||
|
git commit -m "fix: load real tree_data for training-loop diagnosis (runner)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: 集成验证 ground_truth 接通 + 依赖方向
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `tests/integration/test_diagnosis_tree_link.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: 写 integration 测试(ground_truth 非空 + 依赖方向)**
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/integration/test_diagnosis_tree_link.py
|
||||||
|
"""集成验证:真实树注入后 evaluate_span 收到非空 ground_truth;core 不依赖 app。"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app.harness.tree_nodes import load_tree_nodes
|
||||||
|
from core.evolution.diagnose import _get_ground_truth_for_trace
|
||||||
|
|
||||||
|
|
||||||
|
def test_ground_truth_nonempty_with_real_tree():
|
||||||
|
"""对真实 T2 样本 604-2(video 0RxMZBLeqRI)的 view_node 调用,ground_truth 非空。"""
|
||||||
|
td = load_tree_nodes(Path("store"), "0RxMZBLeqRI")
|
||||||
|
node_id = "0RxMZBLeqRI_L1_000" # 该视频真实存在的节点
|
||||||
|
gt = _get_ground_truth_for_trace(td, "view_node", {"node_id": node_id})
|
||||||
|
assert gt and gt != "{}" # 拿到该节点 card 的 JSON,非空
|
||||||
|
|
||||||
|
|
||||||
|
def test_core_diagnose_does_not_import_app():
|
||||||
|
"""算法保真 + 依赖方向:core/evolution/diagnose.py 不 import app。"""
|
||||||
|
src = Path("core/evolution/diagnose.py").read_text(encoding="utf-8")
|
||||||
|
assert "import app." not in src
|
||||||
|
assert "from app." not in src
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: 运行确认通过**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/integration/test_diagnosis_tree_link.py -q`
|
||||||
|
Expected: PASS
|
||||||
|
|
||||||
|
- [ ] **Step 3: 全量回归**
|
||||||
|
|
||||||
|
Run: `conda run -n Video-Tree-TRM pytest tests/ -q`
|
||||||
|
Expected: PASS(无回归;诊断相关用例因 core fail-loud 需补树的已在 Task 2 Step 4 处理)
|
||||||
|
|
||||||
|
- [ ] **Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add tests/integration/test_diagnosis_tree_link.py
|
||||||
|
git commit -m "test: integration for diagnosis tree_data link + core dep direction"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 重跑与重冻衔接(代码计划外的运行步骤)
|
||||||
|
|
||||||
|
代码合入后 git short SHA 变 → `diag_fingerprint` 变 → 需全量重跑:
|
||||||
|
|
||||||
|
1. **重跑诊断 + 重冻切分**(一条命令走完两阶段):
|
||||||
|
`CUDA_VISIBLE_DEVICES=0 CONCURRENCY=12 bash scripts/build_video_split.sh`
|
||||||
|
2. **实测校验 tier**:重跑后查新 fingerprint 的 `baseline_diagnosis`,确认 `T2≈82 / T1≈152`(缓存命中预期);偏差需归因(设计 §6:C3 异常吞并等非 tree_data 不稳定源)。
|
||||||
|
3. **观察 error_type 恢复多值**:确认 `error_type` 不再 100% extraction_failure、`evolution_target` 不再全 tool(软验收,充分性依赖 judge)。
|
||||||
|
4. **接受 pools.json 成员变化**:test/train 具体成员随多样性维恢复而变,floor/代表性 ε 约束仍保证 test 代表性合格(设计 §6,用户已确认重冻覆盖)。
|
||||||
|
|
||||||
|
## 算法保真校验(§4.7)
|
||||||
|
|
||||||
|
| 算法 | 是否涉及 | 结论 |
|
||||||
|
|------|---------|------|
|
||||||
|
| #7 诊断瀑布 | 是(仅接通输入) | **不改** `attribute_error`(`diagnose.py:910-941`)、severity 函数、defect/lapse 判定;Task 2 仅在 `_process_question` 加 fail-loud 输入护栏。参考 TRM4 `core/harness/diagnose.py:1677` tree_cache 语义对齐 ground_truth。 |
|
||||||
|
| #12 训练循环编排 | 是(仅换 tree_data 来源) | `runner._run_diagnosis` 只把 `tree_data={}` 换成真实加载,不改三级嵌套/慢更新/断点续训。 |
|
||||||
|
| 其余 11 项 | 否 | 不涉及。 |
|
||||||
|
|
||||||
|
## 验收标准
|
||||||
|
- 5 个 Task 全绿;`pytest tests/` 无回归。
|
||||||
|
- `evaluate_span`/`_get_ground_truth_for_trace` 在真实树下拿到非空 ground_truth(Task 5)。
|
||||||
|
- `core/evolution/diagnose.py` 不 import app(依赖方向)。
|
||||||
|
- 两**生产注入点**(离线 `build_diagnosis_deps` + 训练 `_run_diagnosis`)均加载真实树。允许保留 `tree_data={}` 的**豁免场景**:dry-run `fake_deps`(`wrong_ids=[]` 早返回)、以及测试里走"无题诊断早返回"路径的用例(须带注释说明)。Step 4b 的 grep 兜底确保无遗漏的会真正进 `run_diagnosis` 的空树调用。
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:gate-speedup
|
||||||
|
title: "连续并发 gate + Redis 复用实现计划"
|
||||||
|
date: 2026-07-17
|
||||||
|
---
|
||||||
|
|
||||||
|
# 连续并发 gate + Redis 复用实现计划
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:preflight-wp1-asset-migration
|
||||||
|
title: "WP1 资产迁移"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# WP1 资产迁移
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:preflight-wp2-split-wiring
|
||||||
|
title: "WP2 切分与接线"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# WP2 切分与接线
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:preflight-wp3-train-loop
|
||||||
|
title: "WP3 训练循环与进化引擎"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# WP3 训练循环与进化引擎
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:preflight-wp4-resilience
|
||||||
|
title: "WP4 韧性与持久化"
|
||||||
|
date: 2026-07-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# WP4 韧性与持久化
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:results-driven-video-split-plan
|
||||||
|
title: 结果驱动的视频级切分实现计划
|
||||||
|
date: 2026-07-15
|
||||||
|
---
|
||||||
|
|
||||||
|
# 结果驱动的视频级切分实现计划
|
||||||
|
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
id: question-gen-v2-adversarial-audit
|
||||||
|
title: question-gen v2 设计对抗审核 — 六路独立核验(四层病灶闭合度 + 契约一致性)
|
||||||
|
type: review
|
||||||
|
created: 2026-07-15
|
||||||
|
status: blocking
|
||||||
|
target: research-wiki/designs/2026-07-15-question-gen-v2-grounded-contrastive-design.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# question-gen v2 设计对抗审核
|
||||||
|
|
||||||
|
**方法**:6 路独立审核官(5 Claude + 1 Codex),互不通气,各自站"找漏洞"立场对抗式核验。5 路死磕 finding 四层病灶各一层 + QuestionUnit 契约一致性;1 路 Codex 端到端质疑质量。
|
||||||
|
|
||||||
|
**总判定**:**设计不能保证出题质量,不应直接进 writing-plans。** 四层病灶无一层 CLOSED;②不可翻转侧与契约层各带 Critical 级结构性缺陷(设计返工,非实现补丁)。
|
||||||
|
|
||||||
|
## 四层闭合度矩阵
|
||||||
|
|
||||||
|
| 层 | 判定 | 最致命残留 | 来源 |
|
||||||
|
|----|------|-----------|------|
|
||||||
|
| ① grounding 循环自证 | PARTIALLY | 换独立 VLM 只断"同模型自评"一半;打分仍黑盒标量、无帧证据可溯源(I1 自认)。时序逆序干扰项穿透 grounding/NLI/快解/后置四门 | ①+Codex |
|
||||||
|
| ② 结构性捷径 | **不可翻转侧 OPEN/Critical** | 数学必然:单维反事实互不重叠→正解=逐维众数→逐槽投票 100% 命中,且满足全部硬约束。病灶②被从"偏置"升级成"数学保证" | ②+Codex |
|
||||||
|
| ③ 歧义控制 | PARTIALLY(实操近 OPEN) | 现网 gate 仍 Phase A 松绑版且 multi_true 判官 text-only 不看帧;镜像模式引入两条新歧义路径(翻转轴模糊、搬运干扰项未复检) | ③ |
|
||||||
|
| ④ 对抗信号位置 | PARTIALLY | 前移的是"便宜替身",完整 agent 仍后置且单裁判;回灌环无一致性校验+无收敛上界(踩 findings 漂移警告) | ④ |
|
||||||
|
| 契约一致性 | **BROKEN** | 设计只覆盖采样/分批/聚合 3 处,真实代码另有 5+ 处按单题拆 pair;镜像"即用即弃"架构与"生成期双入库"互斥=重写非改造 | 契约 |
|
||||||
|
|
||||||
|
## 跨审核官强收敛根因(高置信)
|
||||||
|
|
||||||
|
| # | 根因 | 指认者 | 硬伤 |
|
||||||
|
|---|------|-------|------|
|
||||||
|
| R1 | **MiniMax 单模型多角色=系统性单点** | ①③④+Codex | grounding打分+快解探针+歧义门三合一;仅歧义门加 NLI 第二路,grounding/快解裸奔→某类视频系统性偏差三门同向失效。缺 AdVQA 多裁判一致性门 |
|
||||||
|
| R2 | **不可翻转 4 子模式结构性未修** | ②+Codex | 逐维众数数学证明;覆盖 AR 题 60-70% 主体 |
|
||||||
|
| R3 | **契约拆 pair 远超设计覆盖的 3 处** | 契约+Codex | pools.py 三池切分把 P/Q 劈到不同池→AR pair 几乎全变孤儿被剔;correctness 逐题 dict 与 pair 双向 AND 口径无法共存→污染进化 |
|
||||||
|
| R4 | **验收指标自我确认 + 缺结构性/人工指标** | 全部+Codex | I4 测"模型是否用了捷径"非"捷径是否结构存在";MiniMax 相关指标全同源自证。缺帧错配安慰剂测试/逐维众数命中率(纯算法)/人工双正解率κ/槽位卡方/pair 关系可解率 |
|
||||||
|
| R5 | **回灌环+补生成 recall 泄漏** | ③④ | 回灌无一致性校验+无轮次上限→漂移;补生成"重试直到通过"+门非确定+无 quarantine→precision-over-recall 稀释回 recall |
|
||||||
|
|
||||||
|
## 逐层高severity 残留(矩阵未尽项)
|
||||||
|
|
||||||
|
### ① grounding
|
||||||
|
- V1 打分仍 `list[float]` 黑盒,observation 无帧证据/rationale/帧号(`distractor_selector.py` `_score_options`)——换 MiniMax 不改此数据结构。
|
||||||
|
- V2 粗档下区间选择退化为平局任意 top-3,near-miss 甜区失效;扩池无梯度救不了。
|
||||||
|
- V4 快解探针非对称谬误:弱解题器"解不出"被当"题够硬"绿灯。
|
||||||
|
- V5 时序逆序干扰项被 NLI 判非等价放过、被时序弱的 MiniMax 判非双正解放过。
|
||||||
|
|
||||||
|
### ② 结构性捷径
|
||||||
|
- C1(Critical)逐维众数:见 R2。修法=放弃 hub-and-spoke,改平衡区组/因子设计(每维每值出现次数相等→逐维众数无定义),加纯算法硬校验 `argmax(逐维众数)==正解→fail`。
|
||||||
|
- C2(高)镜像对 flip 反相关把双向 AND 从 1/16 打回 1/1(单个"事件时序"文本线索通杀 P+Q);bag-of-words 是"尽量"软约束未强制。
|
||||||
|
- C3/C4/C5(高/中高)evidence_gap 的"无法确定"恒定文本靶、semantic_rigidity 字幕匹配反向信号、premature 最终段文本靶——单维反事实改不掉这些语用/类别差异。
|
||||||
|
- C6(中)恒 A 正解槽位偏置无约束(Codex 亦独立指认 I1)。
|
||||||
|
- C7(中)NLI 的 embedding 回退是假绿灯,测不出语义泛化蕴含。
|
||||||
|
|
||||||
|
### ③ 歧义控制
|
||||||
|
- R1 视觉双真+文本独立(拿杯子/放盘子同框):NLI 放行,视觉侧唯一裁判 MiniMax 漏判即穿门。
|
||||||
|
- R2/R3(高,v2 新引入)镜像翻转轴模糊→P/Q 各自可辩护;搬运进 Q 的干扰项未在翻转题干下重新 grounding 复检。
|
||||||
|
- R6(实操高)现网 `gate_multi_true.md` 仍松绑版;`_gate_multi_true` text-only 不看帧,"视觉双真"结构上无输入通道。
|
||||||
|
|
||||||
|
### ④ 对抗信号位置
|
||||||
|
- R1 便宜探针能挡的题 ⊂ 完整 agent 能挡的题,差集=结构性/多步 shortcut,仍靠后置。
|
||||||
|
- R2 回灌环缺一致性校验+无 `max_regen_rounds` 上界。
|
||||||
|
- R4 后置难度判定单裁判,缺 AdVQA 多裁判一致性;"agent 恰好能解/不能解"的偶然性主导难度标签。
|
||||||
|
|
||||||
|
### 契约(BROKEN,设计未覆盖的 5+ 处)
|
||||||
|
|
||||||
|
| 处 | 文件/函数 | 拆 pair / 错口径 | 严重度 |
|
||||||
|
|----|----------|-----------------|-------|
|
||||||
|
| 三池 progressive exclusion | `pools.py::build_pools`/`_sample_excluding` | 拆 pair 到 test/val/diag→几乎全变孤儿 | Critical |
|
||||||
|
| per-category train/val 切分 | `pools.py::_split_one_category` | 拆 pair 到 train/val | Critical |
|
||||||
|
| batching 对/错桶分离(先于 FFD) | `batching.py::_select_mixed_by_task_type` | 按逐题 correctness 把 P/Q 分进 correct/error 桶并抽样丢弃 | Critical |
|
||||||
|
| gate 信息量阶梯 | `gate_ladder.py`+`runner._run_gate_validation` | 逐题跑 gate,双向 AND 不生效,逐题分污染 e-process/进化 | High |
|
||||||
|
| pools.json 冻结/解冻 | `pools.py::_q_to_dict`/`_dict_to_q` | 丢 pair 三字段→孤儿 | High |
|
||||||
|
| 作弊门逐题剔除 | `adversarial_filter.py::run_cheater_gate`/`write_final_bank` | 半剔成孤儿 | High |
|
||||||
|
| correctness dict 逐题键 | `runner` 6+ 处消费 | pair 双向 AND 无处安放,与逐题 predictions 口径不自洽 | High |
|
||||||
|
| 镜像即用即弃架构 | `adversarial_filter.py::generate_mirror_question` | `_mirror`后缀/临时 pair_id/不入库,与"生成期双入库"互斥=重写非改造 | High |
|
||||||
|
| 非 AR byte-identical | 混合池 `rng.sample` | unit 折叠改变 population 长度→非 AR 选择漂移,破坏铁律 | Medium-High |
|
||||||
|
|
||||||
|
## 必须返工的设计条目(按优先级)
|
||||||
|
|
||||||
|
1. **§4.1.B 推倒重来**(R2, Critical):hub-and-spoke → 平衡区组/因子设计 + 逐维众数硬校验。
|
||||||
|
2. **§4.2/§4.4 补第二独立视觉裁判 + 修快解探针非对称**(R1, Critical):grounding 打分带帧证据核验 + 第二 VLM 交叉(qwen/MiniMax 多模态看帧,非 CLIP 几何);快解探针要求 solver 先证胜任;歧义门补第二视觉裁判凑多裁判。
|
||||||
|
3. **§5 契约大幅扩写**(R3, Critical):采样上移 pools.py(三池/train-val 以 unit 为原子);新增 unit 粒度 correctness 视图与逐题 dict 分离;序列化补 pools.json;作弊门 unit 粒度剔除;gate_ladder pair 口径。
|
||||||
|
4. **§2/§4.5 如实标注镜像是重写**(R3):给持久 pair 落库 schema。
|
||||||
|
5. **§4.4 multi_true AR 变体吃帧**(③):VLM 看真实帧;先落地收紧 rubric;镜像对增设翻转轴清晰度门 + 干扰项翻转题干下重新 grounding。
|
||||||
|
6. **§4.3 回灌环加独立一致性校验+收敛上界;补生成加 quarantine**(R5)。
|
||||||
|
7. **§4.1 I4 验收指标重构**(R4):补纯算法结构性指标 + 人工/异构审计 + 帧错配安慰剂测试。
|
||||||
|
|
||||||
|
## 关键约束修正
|
||||||
|
|
||||||
|
- **grounding 不需要 CLIP**:排除的是专用图像嵌入模型(CLIP/SigLIP 太难装配);VLM 多模态(qwen/MiniMax 直接看帧+文本)可用。故"第二独立视觉裁判 / multi_true 吃帧 / grounding 帧证据核验"均可用 VLM 多模态落地,只是拿不到几何相似度。
|
||||||
|
|
||||||
|
## 反自证验收指标清单(R4 落地)
|
||||||
|
|
||||||
|
| 指标 | 靶向 | 手段(须非同源自证) |
|
||||||
|
|------|------|---------------------|
|
||||||
|
| 帧错配安慰剂测试 | ①虚假 grounding | 喂错误视频帧打分,分数须显著坍塌 |
|
||||||
|
| 逐维众数命中率 | ②C1 | 纯算法机械投票,阈值≈0.25 |
|
||||||
|
| 答案槽位卡方 + pair 槽位反置率 | ②C6 | 纯统计 |
|
||||||
|
| pair 关系可解率 | ②C2 | P+Q 一起喂盲答器+时序排序规则解法,须≈1/16 |
|
||||||
|
| 双正解率 + 裁判间 κ | ③ | 双 VLM 视觉裁判 + 求解器分歧探针三源投票,基线 33%→阈值≤5% |
|
||||||
|
| 难/烂混淆矩阵 | ④ | 人工/强裁判双标签,对照纯 Phase B 基线 |
|
||||||
|
| 难度判定翻转率 | ④R4 | N 次独立 agent 试答,测判定稳定性 |
|
||||||
|
| 前移拦截覆盖率 | ④R1 | 后置剔除题里"本应生成端拦下"占比 |
|
||||||
|
|
||||||
|
## 相关
|
||||||
|
- 设计: [2026-07-15-question-gen-v2-grounded-contrastive-design.md]
|
||||||
|
- 诊断: [2026-07-15-question-gen-v2-diagnosis-and-strategy.md]
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# 训练前修复分支 final whole-implementation review
|
||||||
|
|
||||||
|
> 2026-07-16。分支 `feat/preflight-train-fixes`,4 工作包 33 个实现 commit。SDD 每 task 三审 + 本次 Codex 跨 task 整分支终审(gpt-5.4 xhigh)。
|
||||||
|
> 覆盖矩阵:21 项确认缺陷 + 接线 2 项 + WP1 死字段清理**全部实现落地**(Codex 逐项核对 file:line 证据)。核心算法保真(#4/#5/#6/#7/#8/#9/#10/#12)、依赖方向(core 不依赖 app/adapters)、loguru、原子写、死字段无残留——全部 verified OK。
|
||||||
|
|
||||||
|
## 执行结果汇总
|
||||||
|
|
||||||
|
| WP | 内容 | commit 数 | Codex 三审结果 |
|
||||||
|
|----|------|----------|---------------|
|
||||||
|
| WP1 | 模板迁移 + 死字段 + fail-loud | 3 | 无任何问题(模板 cmp TRM4 逐字节一致) |
|
||||||
|
| WP2 | 切分与接线(tier/功效/seed/覆盖保护) | 7 | 无 Critical;2 Important(--force 备份健壮性)已修 |
|
||||||
|
| WP4 | 韧性与持久化(10 项) | 11 | 无 Critical;2 Important(全 INFRA/parse_error 护栏)已修 |
|
||||||
|
| WP3 | 训练循环与进化(9 项,5 算法保真区) | 12 | 无 Critical;2 Important(单元计数/全过滤 fail-fast)已修 |
|
||||||
|
|
||||||
|
全量 1523 tests passed,ruff 全绿。
|
||||||
|
|
||||||
|
## Final review 发现与处置
|
||||||
|
|
||||||
|
### C-1(必须,runbook)——.env 未同步导致 train 启动崩溃
|
||||||
|
`.env:65` 仍 `REDIS_CACHE_TTL=0`,WP4 Task 2 的 fail-loud 会在 `main.py` / `video_split_cli` 构建 Redis 缓存时抛 ValueError,训练进不到 `runner.train`。`.env.example` 已改 86400 但 `.env`(gitignore)需**手动**改。→ 训练前 runbook 第一步:`REDIS_CACHE_TTL` 改为正整数(如 86400)。
|
||||||
|
|
||||||
|
### I-3 / I-4(真实缺陷,✅ 已修)
|
||||||
|
| # | 缺陷 | 首跑是否触发 | 处置(commit) |
|
||||||
|
|---|------|:---:|------|
|
||||||
|
| I-4 | 显式 task_types 子集不过滤冻结全局 pools → 训练非请求题型 | 全 12 类首跑**不触发** | ✅ `ee69721`:`_filter_untrainable_types` 候选集先与 task_types 取交集,非请求题型剔除,fail-fast 保留 |
|
||||||
|
| I-3 | gate INFRA 护栏分子按 record、分母按 unit,AR pair 误触发 gate_guard_err | 全 single 首跑**不触发** | ✅ `b3ba11c`:INFRA 分子改按 unit 数;顺带修正 2 个既有护栏测试的 mock 不真实性(per-record 与 summary 不一致,真实推理不会发生),保真套件全绿 |
|
||||||
|
|
||||||
|
> M-2(`6911c83`):`INFRA_STOP_REASONS` 提为 core 公共常量、app import。**残留 future work**:`app/harness/video_split_cli.py` 仍有第三份独立副本(本次 scope 外),待后续 dedup。
|
||||||
|
|
||||||
|
### I-1 / I-2(技术论证:实际影响可控,记录不强修)
|
||||||
|
| # | Codex 关切 | 论证 | 首跑建议 |
|
||||||
|
|---|-----------|------|---------|
|
||||||
|
| I-1 | cache_salt 未贯穿工具内 LLM/VLM(observe_frame VLM、summarizer) | 工具内是**确定性子程序**——同帧→同 VLM 描述、同轨迹→同 summary,重放语义正确甚至期望;需跨 epoch 重采样的 agent **决策** LLM(主 loop chat)已正确注入 run_id 盐 | 改 .env 后可跑;若谨慎可首跑关 Redis 缓存 |
|
||||||
|
| I-2 | 基线臂 miss 新鲜推理注入 epoch salt,不符固定快照 | BaselineCache(`baseline_cache.json`)**跨 run 持久**:快照一旦建立即稳定,epoch salt 只作用于"首次建立快照的那一次采样"(本就要采一次),中断重跑命中持久缓存 | 影响限于首次采样,可接受 |
|
||||||
|
|
||||||
|
> 若后续要彻底贯彻 P1-1(把 cache_salt 显式化、基线臂传 None、工具内也隔离),记 future work——需把 Task 9 的"inference 内部用 run_id"重构为"调用方显式传 cache_salt",风险中等。
|
||||||
|
|
||||||
|
### Minor(future work / 顺手)
|
||||||
|
- **M-2**:INFRA stop-reason 集合在 core/app 各一份 → 已派 app import core 常量消除漂移。
|
||||||
|
- **M-1**:`_atomic_write_json` 在 pools.py/workspace.py 两份等价实现(行为一致,可抽共享 helper)。
|
||||||
|
- **M-3**:部分 analyses JSON(非 checkpoint/manifest/pools)仍直接 write_text(不破坏续跑主状态)。
|
||||||
|
|
||||||
|
## 训练前 runbook(结合本分支)
|
||||||
|
|
||||||
|
1. **改 `.env`:`REDIS_CACHE_TTL=0` → `86400`**(C-1,不改则启动崩)。
|
||||||
|
2. 备份 `workspaces/video-split/` 冻结产物。改 `config/video_split.yaml` 已含 `val_ratio: 0.4`(WP2),重跑 `build_video_split.sh`(诊断命中缓存秒级重切)→ 核对 val 错题≥20 / T2 入 diag / tier 感知生效。
|
||||||
|
3. 建 seed:`extract_run_db(infer_adhoc, dedupe_per_question=True)` + `init_seed('adhoc-baseline', pools_json=…, split_manifest=…)`(WP2 接线)。
|
||||||
|
4. 新建 `config/train_videomme.yaml` + `scripts/train_videomme.sh`:`epochs=3`、`early_stop_patience=2`(epoch 语义 WP3)、`run_holdout_eval=true`(去重版 WP3)、`trainable_min_units=8`(WP3 新字段,必填)、全 12 类(避免 I-4)、其余 gate/batch 沿用 default.yaml。
|
||||||
|
5. tmux 启动 `CUDA_VISIBLE_DEVICES=0 bash scripts/train_videomme.sh`。预检会自动剔除微型题型(OCR/Spatial/Temporal Perception 等 5 类),打印剔除清单。
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# 训练前多维度审查(video-split 冻结切分 → Video-MME 900 训练)
|
||||||
|
|
||||||
|
> 2026-07-16。8 维度并行审查 + 每条发现 2 名独立验证员对抗核实:**19 条确认 / 4 条存疑 / 12 条误报被反驳**。
|
||||||
|
> 场景锚定:global 冻结池 210/90/600、12 题型、baseline=infer_adhoc、concurrency=24、3 epochs、skill_update_mode=patch。
|
||||||
|
> 关键条目(P0-1/P0-2/微型题型)已由主会话人工复核确认。
|
||||||
|
|
||||||
|
## P0 —— 不修则训练无效或必崩
|
||||||
|
|
||||||
|
| # | 位置 | 缺陷 | 后果 |
|
||||||
|
|---|------|------|------|
|
||||||
|
| P0-1 | `runner.py:2272` + `prompts/` | 进化模板 `evolve_skill/system/tool/rank.md`、`consolidate_system.md` 全部缺失(TRM4 有),`_load_evolve_prompts` 用 `else ""` 静默兜底空串 | 全部进化 LLM 调用以空 system prompt 运行,edits 恒空 → 3 epochs 表面正常跑完但 **零进化**。已跑过的 train-ar30 / train-action-recognition 同样受影响,结果需回查 |
|
||||||
|
| P0-2 | `inference.py:462` | traces 表只建表、全仓库无写入(TRM4 TracePlugin 未迁移);训练轨迹只存 `predictions.steps_json` | 训练内 `run_diagnosis` 经 `get_traces` 拿空轨迹 → 诊断瀑布坍缩(算法保真 #7 失效)。离线管线已有 `StepsJsonRunLog` 适配器可复用 |
|
||||||
|
| P0-3 | `runner.py:2044` / `validate.py:682` / `gate.py:99` | 微型题型三连雷(根因同一):新冻结切分中 Temporal Perception、Spatial Perception、Spatial Reasoning 在 val 池 **0 题** → `_class_baseline_acc` AssertionError;非 test 单元仅 2 个 → 案例包排除后阶梯为空 raise ValueError;n_plan=1 时 e-process 结构性 reject_inertia | 训练中途必崩两处 + 微型类白烧进化成本。建议训练 task_types 排除微型类,或切分加 per-class val 下限 |
|
||||||
|
| P0-4 | `inference.py:423,446` | prediction 未归一化且落库 insert 在 try 块之外;LLM 提交 `{"answer": ["B"]}` 等非标量 → sqlite 绑定异常击穿整个 gather | 训练崩溃,且 Redis 缓存重放使 resume 后确定性复现 → 死循环 |
|
||||||
|
| P0-5 | `runner.py:316` vs `config.py:63` | early_stop_patience 实际按 **step** 计数(每 epoch 一次性累加 ~20),文档语义是"轮" | patience=4 时 epoch 1 只要没严格超过 baseline 就终止全部训练,交付 v1 基线 |
|
||||||
|
| P0-6 | `momentum.py:151` | `prompts/slow_momentum.md` 缺失,`use_slow_momentum=true` 下无条件 read_text | 修复 P0-1 后必现:首个 accept 的 epoch 末 FileNotFoundError,崩在慢更新中段(叠加慢更新非幂等 → resume 二次污染) |
|
||||||
|
|
||||||
|
## P1 —— 信号污染类,强烈建议训练前修
|
||||||
|
|
||||||
|
| # | 位置 | 缺陷 |
|
||||||
|
|---|------|------|
|
||||||
|
| P1-1 | `main.py:93` + `redis_cache.py:42` | REDIS_CACHE_TTL=0 → 永不过期;缓存键仅 hash(model+messages),不含采样参数。prompt 未被进化修改的题跨 epoch 逐字节重放首次采样,γ-EMA/e-process 把重放当独立证据 |
|
||||||
|
| P1-2 | `llm.py:116,575` | SSE 流截断(无 [DONE])当成功处理并写入永不过期缓存 → 半截答案永久毒化 |
|
||||||
|
| P1-3 | `llm.py:182` | httpx.RemoteProtocolError/ReadError/ConnectTimeout/PoolTimeout 不在瞬时错误清单,零重试直接落错题;8 次重试预算对最常见断连完全无效 |
|
||||||
|
| P1-4 | `validate.py:304,593` | 基线臂 INFRA 错误(prediction=None)折叠为"基线答错"永久写 BaselineCache(内容寻址无失效),INFRA 护栏在缓存写入之后才检查 |
|
||||||
|
| P1-5 | `diagnose.py:2194` + `runner.py:1019` | cause_category=None(judge 基础设施异常)与 degraded 题在训练链路中进 defect 正文进化路径,反转"判不准默认 lapse"的保护方向;runner 对 degraded_count 零检查 |
|
||||||
|
| P1-6 | `patch.py:343,320` | 冻结区判定只查 edit target 起点不查跨度:起点在正文、末端延伸进 appendix/momentum 区的 delete/replace 被放行 → marker 破坏 → epoch≥2 时 `appendix_region_bounds` ValueError 崩溃 |
|
||||||
|
| P1-7 | `workspace.py:282,311,365` | manifest.json 全部写路径为裸 write_text 非原子写(checkpoint.py 已有 tmp+replace 先例);训练高频重写,截断即 workspace 不可恢复 |
|
||||||
|
|
||||||
|
## P2 —— 操作规程可规避 / 影响半径小
|
||||||
|
|
||||||
|
| # | 位置 | 缺陷 | 规避 |
|
||||||
|
|---|------|------|------|
|
||||||
|
| P2-1 | `video_split_cli.py:557` | 冻结产物无覆盖保护(承诺的 --force 门不存在),commit 换 SHA 后重跑会静默替换 pools.json | 立即备份当前冻结产物并记录 sha256;训练前不再重跑切分脚本 |
|
||||||
|
| P2-2 | `log.py:77` | 只读查询也以 baseline_run_id 打开 HarnessLog → upsert 改写 infer_adhoc 的 _runs 溯源元数据 | 违反 log.py 自身 docstring 约定,宜改走 RunLogImpl |
|
||||||
|
| P2-3 | `diagnose.py:2081` / `inference.py:461` | predictions/traces 无主键 + step 中途崩溃 resume 同 run_id 重跑 → 重复行双计入诊断统计 | 避免中途 kill;后续补去重 |
|
||||||
|
| P2-4 | `diagnose.py:2194`(离线) | 孤立 C3 瞬时失败 → 题永久 uncertain 且断点续跑不重试(影响个位数题;C1/C2 失败会 fail-loud 全崩,批量污染不可达) | 检查 uncertain 占比(本轮 3/236) |
|
||||||
|
| P2-5 | `runner.py:1444` | R2 的 dual_metric "final" 行在 revert 判定前落库,回退版本留下污染记录(存疑级) | harness-eval 读数时注意 |
|
||||||
|
| P2-6 | `breaker.py:36` | 熔断半开无单探针语义,cooldown 到期 24 并发同时放行(存疑级) | 持续故障时人工暂停 |
|
||||||
|
| P2-7 | `runner.py:2286` | `prompts/span_eval_user.md` 缺失(同款空串兜底),但 diagnose.py 实际未消费该字段 | 迁移时顺手补 |
|
||||||
|
|
||||||
|
## 反驳的 12 条(误报,不需处理)
|
||||||
|
|
||||||
|
McNemar 护栏时序、diag_fingerprint 声明式指纹、T0 计入 val 错题、sub_pattern 序列化丢失、correctness 重复行、双题目权威、resume 结构键缺池参数、evolve_single_tool TypeError、momentum guidance 消毒、缓存反序列化无防护、CLI store_true 覆盖 YAML、load_config 静默过滤未知键 —— 均经双验证员核实为不可达或有上游防御。
|
||||||
|
|
||||||
|
## 结合前一轮预检的完整训练前 checklist
|
||||||
|
|
||||||
|
1. 接线:seed 携带 pools.json 机制(或 frozen_pools_path 配置)+ 新建 adhoc-baseline seed + 训练 yaml/sh(`questions: benchmarks/Video-MME`、run_holdout_eval 显式决策)。
|
||||||
|
2. P0-1~P0-6 全部修复;P1 按成本尽量修(P1-1 至少给训练 run 加缓存 salt 或 TTL)。
|
||||||
|
3. 备份当前冻结产物(P2-1)。
|
||||||
|
4. 训练启动前预检脚本:val per-class 覆盖、微型题型排除、evolve/diagnose 模板存在性 fail-loud。
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# 连续并发 gate 重构 · 终审与交付记录
|
||||||
|
|
||||||
|
> 2026-07-17。分支 feat/gate-speedup(18 commits)→ merge 172b7a8 入 feat/question-gen-v3。
|
||||||
|
> 设计 research-wiki/designs/2026-07-16-gate-speedup-design.md(v3);计划 research-wiki/plans/2026-07-16-gate-speedup.md。
|
||||||
|
|
||||||
|
## 交付摘要
|
||||||
|
|
||||||
|
| 项 | 结果 |
|
||||||
|
|---|---|
|
||||||
|
| 7 Task(SDD:实现+三层 Codex 审/任务) | 全部收口;全量 tests/ 1561 passed,覆盖率 83% |
|
||||||
|
| 核心算法 | #6 已批准语义修订(块序贯→阶梯序前缀逐对序贯);#4/#5 及 core/evolution 零改动(diff 为空) |
|
||||||
|
| 终审 | Codex 整分支五维审(1 Critical:gate_evidence 旧表迁移,已修 eb12006)+ Opus 独立残留/计划符合性审:VERDICT CLEAN |
|
||||||
|
|
||||||
|
## 审查抓出并修复的真缺陷(按发现轮次)
|
||||||
|
|
||||||
|
| 缺陷 | 严重度 | 修复 |
|
||||||
|
|---|---|---|
|
||||||
|
| 预灌 BaselineCache 回归均值偏差(设计期) | 设计 Critical | 方案废弃,改连续并发 gate |
|
||||||
|
| 到达序消费配对偏差(设计期) | 设计 Critical | 阶梯序前缀消费 |
|
||||||
|
| 尾部 INFRA 绕过题尽出口 | Critical | 剔除后重判(1e92928) |
|
||||||
|
| acquire 超宽自死锁 + 取消半持有泄漏 | Critical/Important | fail-fast + 回滚(232afd5/30c1cf1) |
|
||||||
|
| 编排器物化中途泄漏 + gather 首异常悬挂任务 | Critical×2 | 逐个登记 + cancel-drain(9e8a254) |
|
||||||
|
| step 重跑不清 gate 行(前序潜伏 bug) | Critical | _clear_step_rows(ea6bec5)+ LIKE 全转义(b3aba7c) |
|
||||||
|
| **共享 gate_log 下 predictions run_id 契约断裂(gate 静默全拒)** | Critical | inference record 显式 run_id(0b83993) |
|
||||||
|
| gate_evidence 旧表缺 ladder_rank 迁移 | Critical(跨版本) | 幂等 ALTER(eb12006) |
|
||||||
|
|
||||||
|
## 遗留与豁免(记录在案)
|
||||||
|
|
||||||
|
- runner.py 两处 HEAD 前既有 ruff format 债(~:1096/~:2296)与 `_filter_untrainable_types` C(17):存量,未触碰。
|
||||||
|
- 旧 checkpoint(含 gate_block 指纹键)resume 时静默兼容不告警:中低风险,本项目训练均 --fresh。
|
||||||
|
- skipped 记录不做 target_file disjoint 检查:无冲突路径,防御加固候选。
|
||||||
|
- 小题型合并进化 default-strategy.md(方案 B):future work,本轮 A 方案结果作对照。
|
||||||
|
|
||||||
|
## 重启
|
||||||
|
|
||||||
|
2026-07-17 05:37 训练在新代码上重启(tmux train_videomme,--fresh,workspace 全新)。Redis 复用:今日键已一次性续期 7 天,.env TTL=604800。
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user