Compare commits

87 Commits

Author SHA1 Message Date
iomgaa 6ac96dff64 Merge branch 'feat/question-gen-v3' 2026-07-17 06:13:44 -04:00
iomgaa 3dcc7d9899 fix: use SLURM_SUBMIT_DIR to locate repo root under sbatch 2026-07-17 06:13:28 -04:00
iomgaa 7386e5af57 Merge remote-tracking branch 'origin/main' 2026-07-17 06:05:42 -04:00
iomgaa 9bb09a44d4 feat: add Slurm gpu-sbatch training entry for 4090-server 2026-07-17 06:02:39 -04:00
iomgaa 0414948682 merge: question-gen v3 + preflight fixes + continuous concurrent gate 2026-07-17 06:02:39 -04:00
iomgaa b0be1f1ae5 docs: gate speedup final review and delivery record 2026-07-17 05:38:44 -04:00
iomgaa 172b7a8ec4 merge: continuous concurrent gate speedup (algo #6 semantic revision, design v3) 2026-07-17 05:29:42 -04:00
iomgaa eb12006d38 fix: idempotent ladder_rank migration for legacy gate_evidence tables 2026-07-17 05:29:20 -04:00
iomgaa 1930ad32a4 chore: raise Redis cache TTL to 7 days 2026-07-17 04:52:34 -04:00
iomgaa 8958eee11b refactor: remove block-sequential gate path and gate_block knob (algo #6)
config/train_videomme.yaml 同时收录待入库的实验配置变更(run_id v2 /
concurrency 32 / batch_size 40)。tests/integration/test_v3_contract_e2e.py
的 run_id 断言按 Task 5 显式契约同步修正(原断言依赖旧隐式实例注入)。
2026-07-17 04:40:30 -04:00
iomgaa 0b839937df fix: predictions row carries arm run_id under shared gate_log; drain evolve gather on failure (algo #6) 2026-07-17 03:52:19 -04:00
iomgaa 23a64042fe feat: parallel evolve + continuous gate wiring in runner (algo #6) 2026-07-17 03:32:14 -04:00
iomgaa b3aba7c31d fix: escape all LIKE specials in step-row cleanup 2026-07-17 01:17:31 -04:00
iomgaa 16993ed362 style: consolidate inference stub, complete docstrings (algo #6) 2026-07-17 01:13:19 -04:00
iomgaa ea6bec5421 fix: clear gate-derived rows on step rerun (idempotency) 2026-07-17 00:57:53 -04:00
iomgaa eea609d960 refactor: remove deprecated retriever module
RecursiveRetriever was a failed approach, not carried into TRM5.
- delete app/retriever/ (empty placeholder)
- drop retriever + train blocks from config/default.yaml
- renumber fidelity checklist 13->12 items (drop #4, shift up)
- sync core-goal text, dir tree, module-interaction diagrams
  across CLAUDE.md, ARCHITECTURE.md, overview.md, README.md
- reference/ kept intact as historical code
2026-07-08 23:09:30 -04:00
iomgaa f1dea4f68f merge: feat/app-harness — app/harness/ 训练循环编排层(14 文件, 算法保真 #6/#10/#13) 2026-07-07 20:35:04 -04:00
iomgaa 7a00bc1a28 style(harness): ruff format batching.py + log.py 2026-07-07 20:34:48 -04:00
iomgaa be0e89401e feat(harness): __init__.py public API + lint fixes 2026-07-07 13:45:48 -04:00
iomgaa 2296134f73 feat(harness): runner.py — train loop orchestrator (#13 algorithm fidelity)
Three-level nesting (epoch -> step -> per-skill), slow update 10-step
sequence, checkpoint/resume, early stop, probation accept/reject/rollback.

Key TRM4->TRM5 changes:
- sync -> async (all inference/diagnosis/evolve/validate awaited)
- LLMClient.from_env -> injected LLMProvider (DI via constructor)
- Direct DB/file access -> module functions (workspace/store/log)
- _TrainState as train() local, explicit param passing to helpers

Module-level pure functions extracted for testability:
resume_plan, _guard_infra_failures, _apply_batch_correctness,
_compute_total_steps, _should_early_stop, _format_applied_edits,
_fallback_summary, _write_skip_report, _outcome_to_quadrant_pairs,
_build_comparison_pairs, _batch_from_ids, _snapshot_current_skills.

Tests: 34 unit tests covering 13a-13e sub-tasks.
Radon: all functions Grade B or better.
2026-07-07 13:43:20 -04:00
iomgaa 7bc6fc752c feat(harness): validate.py — async 块序贯验证编排 + Probation 统一定义 2026-07-07 13:20:43 -04:00
iomgaa a6b816db94 feat(harness): checkpoint.py — TrainState 序列化 + 原子写 + 指纹校验 2026-07-07 13:13:37 -04:00
iomgaa d7f1bdeea6 feat(harness): inference.py — async run_inference + DI 2026-07-07 13:04:26 -04:00
iomgaa 886a444d1d feat(harness): momentum.py — async 慢更新动量生成 2026-07-07 13:01:04 -04:00
iomgaa a550d39e1c feat(harness): observation.py — 五张观测表 + step/epoch 报告 2026-07-07 12:55:12 -04:00
iomgaa bd4e438c6c feat(harness): gate_ladder.py — 信息阶梯 + BaselineCache (#6 算法保真) 2026-07-07 12:55:12 -04:00
iomgaa e0f3ee10ec feat(harness): pools.py — 三池切分(test→validation→diagnosis) 2026-07-07 12:48:54 -04:00
iomgaa 9800fef37a feat(harness): batching.py — FFD + round-robin mini-batch (#10 算法保真) 2026-07-07 12:47:33 -04:00
iomgaa d349fe1148 feat(harness): workspace.py — Workspace lifecycle + VersionedSkillStore/PromptStore
- ResolvedPaths frozen dataclass: store_dir, videos_dir, questions_dir,
  skills_dir, prompts_dir, workspace_dir, db_path, analyses_dir, runs_dir
- init_workspace: create ws + copy seed weights from store
- init_workspace_from_seed: create from seed with fail-fast questions check
- load_manifest / resolve_paths: manifest I/O + path resolution
  (skills/prompts resolve to workspace, videos/questions to store)
- update_manifest: key whitelist validation
- record_run: idempotent history append + per-video wiki dirs
- read_best / update_best: best pointer independent of current
- list_video_ids: videos with tree.json
- archive_workspace: move to .archive/<name>-<ts>
- VersionedSkillStore: implements core/evolution/protocols.py::SkillStore
- VersionedPromptStore: implements core/evolution/protocols.py::PromptStore
- 21 tests all passing (incl. Protocol compliance checks)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:41:59 -04:00
iomgaa b052c1f3ee feat(harness): store.py — Store 版本操作 + Seed 管理
从 TRM4 core/workspace.py 拆出 Store + Seed 相关函数:
- _parse_version / list_versions / next_version / advance_version
- _write_meta / init_store
- init_seed / list_seeds / read_seed
- extract_run_db(保留原始 CREATE 语句重建主键约束)
- promote_to_seed(强校验版本一致 + 非 NULL + finally 清理)

26 个测试全部通过,radon 复杂度 A (2.83)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:36:36 -04:00
iomgaa be3c176a46 feat(harness): HarnessLog SQLite wrapper + RunLogImpl readonly port
- HarnessLog: TRM4 direct port with WAL mode, threading.Lock, INSERT OR
  IGNORE idempotent _runs, context manager (completed/failed), create_table
  with auto run_id+timestamp, insert/insert_many/execute/query/log_event
- RunLogImpl: implements core/evolution/protocols.py::RunLog Protocol with
  independent sqlite3.connect for read-only SELECT (no _runs pollution),
  asyncio.to_thread wrapping for async interface
- _read_table: shared readonly helper with optional question_ids filtering,
  graceful empty-list return for missing tables
- Tests: 17 cases covering thread safety, idempotent inserts, context
  manager status, WAL mode, protocol compliance, readonly isolation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:29:29 -04:00
iomgaa b929a5db6c fix(harness): Codex functional review 修复 — 命名/集成测试/delta_min/promote 消息
- _apply_env_overrides → _apply_env_var_overrides,docstring 明确从 os.environ 读取
- 新增 TestLoadConfigRealYaml:用真实 config/default.yaml 验证嵌套 harness 解析
- 新增 test_delta_min_negative_rejected:覆盖 gate_delta_min >= 0 校验
- 恢复 promote 模式独立错误消息(从合并分支分离回 TRM4 原始提示)
- 77 个单元测试全部通过,radon 全部 Grade B 或更好

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:23:27 -04:00
iomgaa ce43871828 fix(harness): 补充 train 模式 run_id 校验 + 拆分函数保持 radon Grade B
- _validate_mode_deps: 恢复 train 非 resume/fresh 时必须提供 run_id 校验
- 提取 _validate_train_run_id 用 early return 展平条件,避免 radon Grade C
- 合并 promote run_id 检查到 diagnose/evolve/promote 统一检查
- 新增 4 个测试:train+run_id / train+resume / train+fresh / train+baseline
- radon cc -n C 无输出(全部 Grade B 或更好)
- 74 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:14:28 -04:00
iomgaa 6a2ddb1624 refactor(harness): 拆分校验函数降低 radon 圈复杂度至 Grade B
- _validate: 拆出 _validate_mode(mode 依赖校验)+ _validate_basic(标量/枚举校验)
- _validate_gate: 拆为 _validate_gate_thresholds(e 值/净胜/方向)+ _validate_gate_ladder(阶梯/块序贯)
- load_config: 提取 _apply_env_overrides 函数
- radon cc -n C 无输出(全部 Grade B 或更好)
- 70 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:01:46 -04:00
iomgaa 09a385addc feat(harness): RunConfig frozen dataclass + 四层校验 + YAML/CLI/.env 三层加载
- RunConfig: 46 字段 frozen dataclass,从 TRM4 core/harness/config.py 迁移
- 四层校验链:_validate → _validate_edit_budget + _validate_minibatch + _validate_gate
- 新增 .env 覆盖层:工程配置(workspace_dir, store_dir)可通过 HARNESS_* 环境变量注入
- 合并优先级:CLI > .env > YAML(CLAUDE.md §4.5)
- load_config 支持嵌套 harness 段和扁平 YAML 两种格式
- run_id 改为默认空字符串(CLI-only 字段,YAML 不提供)
- resume/fresh 互斥校验不在 config 层(移至 runner.py)
- 70 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 11:57:35 -04:00
iomgaa d84cd679b4 feat(evolution): export dataclass types from __init__.py 2026-07-07 11:48:13 -04:00
iomgaa 292dd0fa13 docs: app/harness/ implementation plan — 15 tasks, algorithm fidelity #6/#10/#13, 120 TRM4 tips 2026-07-07 11:45:38 -04:00
iomgaa eba4344a4b docs: app/harness/ design — 14-module training loop orchestration layer 2026-07-07 11:28:29 -04:00
iomgaa 4e241aa581 merge: feat/tree-index — core/evolution/ extractable kernel + tree/search/question_gen modules 2026-07-07 10:29:54 -04:00
iomgaa 8bc4132750 feat(evolution): __init__.py public API + ARCHITECTURE.md Protocol update 2026-07-07 10:28:14 -04:00
iomgaa 4634414606 feat(evolution): evolve.py per-target evolution — skill/system/tool (#9) 2026-07-07 10:26:58 -04:00
iomgaa 6072ee7d0b feat(evolution): evolve.py validation + helpers (#9) 2026-07-07 10:12:15 -04:00
iomgaa dc091361c3 feat(evolution): diagnose.py aggregation + case packs + run_diagnosis (#8) 2026-07-07 10:04:48 -04:00
iomgaa 49d6fe8f51 feat(evolution): diagnose.py metrics + attribution (Stage 1, #8) 2026-07-07 09:55:22 -04:00
iomgaa f7193551dd feat(evolution): validate.py — pure block validation decision functions (#7) 2026-07-07 09:47:45 -04:00
iomgaa 3005f24577 feat(evolution): patch.py — 补丁引擎移植(算法 #9),51 测试全通过
从 TRM4 core/harness/patch.py (427 行) 零修改移植到 core/evolution/patch.py。

包含:
- 7 个常量(APPENDIX/MOMENTUM marker + MAX_CHARS + HEADING)
- appendix 区:追加/提取/替换/边界检测,损坏态 ValueError
- momentum 区:替换/提取/边界检测,注入防护
- apply_patch_with_report:4 种 op(append/insert_after/replace/delete)
- 冻结区坐标判定(每条 edit 重算)、report 1-based index
- 51 个单元测试覆盖全部公共 API 及边缘场景

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:44:05 -04:00
iomgaa 4eefbfdd74 feat(evolution): gate.py — CE-Gate e-process 纯函数 (#5 算法保真)
从 TRM4 core/harness/eprocess.py 逐行迁移,零逻辑变更:
- compute_e_value: 截断 Beta 混合 e 值(log 空间 + betainc 对称性)
- gate_decision: 四出口优先级链(confirmed→directional→futility→exhaustion→continue)
- probation_verdict: 试用期非对称双向结算
- 常量保真: _WALD_WIN=ln1.4, _WALD_LOSS=ln0.6, _SHRINK_PSEUDO=4

仅变更: import 路径 + GateParams/GateVerdict 移至 types.py + 中文 docstring

14 tests 覆盖: e 值数学、边界校验、四出口路径、试用期三分支

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:38:21 -04:00
iomgaa 9d4d52dac5 feat(core/evolution): protocols.py + types.py 基础层 — 18 个 dataclass + 3 个 Protocol
TRM4 诊断/进化/门控数据类型迁移至 TRM5 Clean Architecture 内核。
逐字段比对 TRM4 的 eprocess.py、diagnose.py、evolve.py 保真迁移。

types.py (18 个 dataclass):
- Gate: GateParams, GateVerdict (frozen, 原样迁移)
- 诊断: SpanMetrics, SkillStepAdherence, QuestionMetrics,
  ErrorAttribution, CaseSample, SkillCasePack, SystemCasePack,
  ToolCasePack, DiagnosisResult (全部 frozen=True)
- 进化: EvolutionRecord (mutable), RejectedEdit (frozen),
  EvolutionResult (frozen, 移除 skills_version/prompts_version)
- 新增: PairResult, QuadrantClassification (块验证纯决策输出)
- 新增: DiagnosePrompts, EvolvePrompts (模板束, frozen)

protocols.py (3 个只读 Protocol):
- SkillStore, PromptStore (同步文件读取)
- RunLog (异步日志查询, 隔离 SQL)

变更理由:
- QuestionMetrics 由 TRM4 mutable 改为 frozen (一次性构造)
- ErrorAttribution 由 TRM4 mutable 改为 frozen (构造时填入全部字段)
- EvolutionResult 移除版本管理字段 (app/ 职责)

涉及算法: #5(CE-Gate), #8(诊断瀑布), #9(进化引擎)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:34:04 -04:00
iomgaa 09100dcbd4 docs: core/evolution/ implementation plan — 9 tasks, TDD, algorithm fidelity checks 2026-07-07 07:55:21 -04:00
iomgaa 84e8abbf4f docs: core/evolution/ extractable kernel design — gate, patch, diagnose, evolve 2026-07-07 07:39:24 -04:00
iomgaa 7cd49a5a3f docs: 同步 app/search/ 模块结构 + OCRProvider 签名到 ARCHITECTURE.md 2026-07-07 06:14:26 -04:00
iomgaa 499c5b8043 feat(search): __init__.py 公开 API + 修复 OCR 测试 asyncio 兼容性 2026-07-07 06:13:25 -04:00
iomgaa 4baf92c93f feat(search): PromptManager — 搜索 Agent 提示词加载与组装
从 TRM4 core/search/prompt.py 迁移。有意变更:
- 工具描述从 app.search.tools.get_tool_descriptions 获取
- format_user_prompt 参数显式化(question/options/l1_node_ids/task_type)

16 个单元测试覆盖 __init__、build_inference_prompt(auto/manual/none
三种 skill_mode)、format_user_prompt、load。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 06:11:13 -04:00
iomgaa f4f92b0938 feat(search): 实现 SearchToolDispatcher 工具调度器 (Task 7)
- 新增 app/search/tools.py:
  - get_tool_descriptions() 工具描述文本(与 TRM4 一致)
  - SearchToolDispatcher 类实现 ToolDispatcher Protocol
  - dispatch() 按工具名路由: view_node / search_similar /
    observe_frame / submit_answer / read_skill
  - ValueError(未知工具)上抛,KeyError/FileNotFoundError 捕获返回错误文本
  - view_node: env.get_node_text + summarize_node + get_children_info + summarize_children
  - search_similar: env.search_similar + summarize_nodes_batch
  - observe_frame: env.resolve_frame_paths + get_subtitle + observe_frame + 字幕前置

- 修复 app/tree/environment.py get_children_info():
  - 原实现返回 _format_time_range (str) 导致 summarize_children 解包失败
  - 改为返回原始数值元组 via 新增 _node_time_range_raw 静态方法

- 新增 tests/unit/test_search_tools.py (14 tests):
  - get_tool_descriptions 含/不含 read_skill
  - 五种工具 dispatch 路由验证
  - 未知工具 ValueError + 节点不存在错误文本

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 06:07:27 -04:00
iomgaa ca3ea1cdf2 style: format summarizer.py 2026-07-07 05:55:40 -04:00
iomgaa ba7fdaddf6 style: format vision.py 2026-07-07 05:53:33 -04:00
iomgaa 3ae3d5ab50 feat(search): app/search/vision.py — 两轮 VLM 帧观察模块
从 TRM4 core/tree/vision.py 迁移 observe_frame,关键变更:
- VLM 调用走 VLMProvider.chat_with_images Protocol(images 传 Path)
- OCR 调用走 OCRProvider.transcribe_frames 异步 Protocol
- 遥测字段 session_id / parent_call_id 透传
- 帧文件存在性前置校验

12 个单元测试覆盖:两轮正常、仅提取、OCR 注入/失败降级/None、
VLM 提取失败、VLM 验证失败降级、帧缺失、stats 完整性、
分歧/弃权标记、遥测透传。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:52:54 -04:00
iomgaa 86b19d1e07 feat(adapters): OCRProvider Protocol + MonkeyOCRClient 异步适配器
- app/ports.py: 新增 OCRProvider Protocol(runtime_checkable,与
  EmbeddingProvider 同级),定义 async transcribe_frames 端口
- adapters/ocr.py: 从 TRM4 core/tree/ocr.py 保真迁移 MonkeyOCRClient
  - assert → ValueError(P5 防御性校验)
  - 公开方法改 async(asyncio.to_thread 包装同步 HTTP)
  - 内部逻辑不变:多端点轮询、线程安全 Session、单帧降级、行去重
- tests/unit/test_ocr_adapter.py: 17 个测试覆盖 Protocol 合规、
  构造校验、健康检查、转录、降级、去重、轮询

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:47:55 -04:00
iomgaa 60e737e2fc feat(search): app/search/skills.py — 技能注册表与 frontmatter 解析
从 TRM4 core/search/skills.py 保真迁移。提供 parse_frontmatter、
strip_frontmatter、SkillRegistry、discover_skills 四个公共 API。
逻辑完全一致,仅调整导入路径并添加中文 docstring。

17 个单元测试全部通过,覆盖正常/异常/边界场景。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:46:25 -04:00
iomgaa 82f1607195 style: format environment.py 2026-07-07 05:46:25 -04:00
iomgaa 44ee62867d feat(tree): add get_node_text + get_children_info to TreeEnvironment
- get_node_text(node_id, anchor=False): returns raw text + optional
  anchor_map dict by parsing [cN]/[sN] prefixes from anchored text
- get_children_info(node_id): returns structured child list with
  id/time_range/summary (description truncated to 120 chars)
- Both methods reuse existing internal helpers (_node_full_text,
  _node_anchored_text, _get_children, _node_description,
  _format_time_range)
- 9 new test cases across TestGetNodeText and TestGetChildrenInfo

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:45:48 -04:00
iomgaa ffb37a1c1e chore: 复制 TRM4 v2 prompt 种子文件(9 个,字节级一致)
.gitignore 添加 !store/prompts/ 例外——prompt 是版本化资源需提交
2026-07-07 05:42:11 -04:00
iomgaa dcaeefa484 docs(plan): 恢复被 wiki 工具覆盖的计划全文 2026-07-07 05:39:21 -04:00
iomgaa 7af64db499 docs(plan): app/search/ 搜索 Agent 装配层实现计划(10 Task) 2026-07-07 05:36:25 -04:00
iomgaa 9e1f39c147 docs(design): 修复 Codex 审查发现 — 10 项修正
Critical: OCRProvider 移至 app/ports.py;新增 TreeEnvironment 结构化 API;
         observe_frame 补充字幕上下文拼接;search_similar 补充节点文本获取
Important: 遥测链路透传 session_id/parent_call_id;异常降级边界明确化;
           verify_vision/anchor/assemble_mode 改为必传;Prompt 路径逐文件列出;
           测试目录规范化到 tests/unit/
Minor: 依赖图补全 VLMProvider 连线
2026-07-07 05:29:52 -04:00
iomgaa 5c89ca543a docs(design): 补充 summarizer 模块 — view_node/search_similar 的 LLM 两轮摘要
自审发现 TRM4 三个搜索工具全有 question-conditioned 中间摘要层,
原设计遗漏了 summarizer.py。新增:
- §4.4 summarizer API(summarize_node / _children / _nodes_batch)
- anchor 锚模式完整保留
- 6 个额外 prompt 文件
- 更新交互流程图、错误处理、测试策略
2026-07-07 05:22:55 -04:00
iomgaa 11107f5758 docs(design): 搜索 Agent 装配层设计(app/search/)
方案 A 平铺模块:prompt.py / skills.py / tools.py / vision.py
新增 OCRProvider Protocol + adapters/ocr.py
Prompt 从 TRM4 store/prompts/v2/ 原封不动复制
2026-07-07 05:15:22 -04:00
iomgaa da28c10c84 docs: 同步 question_gen 模块路径到 ARCHITECTURE.md 和 CLAUDE.md
DataLoader 代码位置 generator.py → loader.py;
目录树更新 question_gen/ 结构反映实际文件。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:51:26 -04:00
iomgaa d8a903fb54 feat(question_gen): QuestionGenerator Protocol + 模块公开 API
app/ports.py 追加 QuestionGenerator Protocol(预留 LLM 出题接口)。
app/question_gen/__init__.py re-export load_benchmark 和 stratified_sample。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:48:21 -04:00
iomgaa 8d515ff01f feat(question_gen): stratified_sample — 分层采样 + 题型保底
算法 100% 保真 TRM4: task_types 过滤、correctness.get(id, False) 语义、
对题在前返回顺序、min_per_class 遍历 pool 全部题型(含稀疏类)。
所有参数显式传入,无默认值。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:45:19 -04:00
iomgaa dea8a7d3f6 feat(question_gen): load_benchmark — benchmark JSON 加载
从 JSON 目录 glob *.json 加载题目,stem 作 video_id。
legacy schema 无 difficulty 字段时赋 _LEGACY_DEFAULT_DIFFICULTY 常量。
options/source_nodes 转 tuple 配合 frozen dataclass。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:41:25 -04:00
iomgaa eea3bcba3f feat(core): 追加 GeneratedQuestion frozen dataclass
跨层共享类型,被 core/evolution/ 和 app/harness/、app/question_gen/ 使用。
frozen=True + tuple 字段确保不可变。无默认值(显式传入)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:36:56 -04:00
iomgaa 4686adf266 feat(tree/repair): 检测器扩展 — visible_entities/ongoing_actions/spatial_layout 为空也触发修复 2026-07-07 03:15:42 -04:00
iomgaa ee5bd0de57 fix(tools): handle None card in flat tree conversion 2026-07-07 03:13:38 -04:00
iomgaa 73b240cc84 test(tree): 建树模块端到端集成测试
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:44:28 -04:00
iomgaa 18971a794b feat(tree/repair): Q&A 反向补全 — 从 TRM4 supplement 迁移
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:37:50 -04:00
iomgaa edaa0d8290 feat(tree): VideoTreeBuilder 保真 #1 #2 #3 + 复杂度重构
- 从 reference/video_tree_trm/video_tree_builder.py (994行) 迁移
- 保真算法 #1: L2 轴心建树策略 (asyncio.gather 链式并发)
- 保真算法 #2: VLM 批量帧描述 + JSON fallback (_L3_BATCH_SIZE=5)
- 保真算法 #3: 断点续跑 (progress.json + L1 中间 JSON)
- 新增: VLMProvider/LLMProvider Protocol 替代 LLMClient
- 新增: 结构化 JSON 输出 → L1Card/L2Card/L3Card
- 新增: L2 代表帧复用 L3 帧 (_sample_representative_frames)
- 新增: 字幕注入 + Voronoi 分配
- 重构: 提取 _load_resume_state/_assemble_roots 降低 _build_async 复杂度 D(21)→C(14)
- 44 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:17:50 -04:00
iomgaa 12f20493c1 feat(tree): 质量校验 — 交叉验证 entities/visible_text
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:57:32 -04:00
iomgaa fb6f9964d8 feat(tree): 字幕模块 — SRT 解析 + 完整性检查 + Voronoi 分配
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:54:09 -04:00
iomgaa e3b027ce34 feat(adapters): GovernedVLMClient — VLMProvider 最小可用实现
- 包装 GovernedLLMClient,注入 base64 图片到 OpenAI Vision API 格式
- 复用 LLM 治理栈全部能力(熔断、缓存、重试、遥测)
- 8 项单元测试覆盖协议满足、图片编码、注入逻辑、不可变性
2026-07-07 01:49:55 -04:00
iomgaa c1680447c0 feat(adapters): EmbeddingProvider Protocol + local/remote 双后端实现
- app/ports.py: 添加 EmbeddingProvider Protocol(runtime_checkable,dim 属性 + embed 方法)
- adapters/embedding.py: 从参考代码迁移,拆分为 LocalEmbeddingProvider 和 RemoteEmbeddingProvider
  - Local: sentence-transformers 冻结推理,维度校验
  - Remote: OpenAI 兼容 API,L2 归一化,按 index 排序
  - 两者均提供 embed() 和 embed_tensor() 统一接口
- tests/unit/test_embedding_adapter.py: Protocol 满足性、形状校验、导入测试

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:45:09 -04:00
iomgaa 547c9ddd84 feat(tree): TreeConfig 配置 dataclass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:41:50 -04:00
iomgaa e0f7851975 fix(tree): from_dict 增加 ID 唯一性校验
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:40:46 -04:00
iomgaa c136de076d refactor(tree): extract _embed_l2_subtree to reduce embed_all complexity
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:37:09 -04:00
iomgaa 22ad014973 feat(tree): TreeIndex 数据结构 — Card 体系 + 节点 + 序列化
- 新增三级 frozen Card dataclass: L3Card(6字段), L2Card(7字段), L1Card(7字段)
- 节点重构: L3Node/L2Node/L1Node 使用 Card 替代原始字符串字段
- 添加 @property 兼容层: description/summary 代理到 Card 字段
- L3Node 新增 subtitle 字段(字幕集成预留)
- JSON 序列化/反序列化支持 Card 结构 + embedding base64 编解码
- load_json 新增 ID 唯一性校验(重复 ID 抛 ValueError)
- 移除 pickle 序列化(仅保留 JSON)
- 日志从 log_msg 迁移到 loguru
- 17 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:35:10 -04:00
iomgaa 2be2569ed8 docs(tree): add tree module vertical slice design and implementation plan 2026-07-07 01:27:36 -04:00
iomgaa c2ba3ed591 chore: lint 修复(TC001/TC003 类型导入优化)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-06 23:15:06 -04:00
43 changed files with 1543 additions and 946 deletions
+1 -1
View File
@@ -47,7 +47,7 @@ LLM_TTFT_TIMEOUT=30
LLM_INTER_TOKEN_TIMEOUT=15
LLM_RETRY_MAX_DELAY=30.0
# 正整数秒,禁止 0(0 会被拒绝启动);训练场景建议 >= 单次训练时长
REDIS_CACHE_TTL=86400
REDIS_CACHE_TTL=604800
# 建树批量并行:全局 VLM/LLM 在途调用上限(Spec-2 工程配置)
TREE_BUILD_API_CONCURRENCY=16
-1
View File
@@ -58,7 +58,6 @@ _DECISION_KEYS = (
"gate_delta_min",
"gate_lambda_dir",
"gate_e_rollback",
"gate_block",
"gate_n_max",
"gate_p_low",
"gate_p_high",
+4 -9
View File
@@ -70,14 +70,13 @@ class RunConfig:
gate_delta_min: 最小点估计效应量下限(承接旧 margin 语义)。
gate_lambda_dir: Wald 方向拒绝的对数似然比阈值(必须为负)。
gate_e_rollback: 试用期对称回滚门(回滚 e 值门槛)。
gate_block: 块序贯验证的块大小(=推理并发度,块内跑满)。
gate_n_max: 单次 gate 消耗的题数上限。
gate_p_low: 信息量阶梯 p-hat 保留区间下界(剔除必错零信息题)。
gate_p_high: 信息量阶梯 p-hat 保留区间上界(剔除必对零信息题)。
gate_probe_quota: 冷启动探针集比例(全错题中插尾的比例)。
gate_gamma_decay: 逐题正确率估计 p-hat 的 EMA 衰减系数。
gate_cooldown_steps: 回滚后该题型跳过进化的冷却 step 数。
gate_guard_err: gate 内跨块累计 INFRA 错误率护栏。
gate_guard_err: gate 内累计 INFRA 错误率护栏。
skill_update_mode: skill 进化模式,"patch"(局部 edit/ "rewrite"(整篇重写)。
appendix_consolidate_threshold: appendix note 条数达此值触发 LLM consolidation。
run_id: diagnose/evolve 模式要分析的运行 ID,默认空字符串。
@@ -125,7 +124,6 @@ class RunConfig:
gate_delta_min: float
gate_lambda_dir: float
gate_e_rollback: float
gate_block: int
gate_n_max: int
gate_p_low: float
gate_p_high: float
@@ -361,7 +359,7 @@ def _validate_gate_thresholds(config: RunConfig) -> None:
def _validate_gate_ladder(config: RunConfig) -> None:
"""校验 CE-Gate 信息量阶梯与块序贯参数。
"""校验 CE-Gate 信息量阶梯参数。
参数:
config: 待校验的配置实例。
@@ -369,11 +367,8 @@ def _validate_gate_ladder(config: RunConfig) -> None:
异常:
ValueError: 任一阶梯参数不合法。
"""
if config.gate_block <= 0 or config.gate_n_max < config.gate_block:
raise ValueError(
f"需 0 < gate_block <= gate_n_max"
f"实际: block={config.gate_block}, n_max={config.gate_n_max}"
)
if config.gate_n_max <= 0:
raise ValueError(f"需 gate_n_max > 0,实际: n_max={config.gate_n_max}")
if not (0 <= config.gate_p_low < config.gate_p_high <= 1):
raise ValueError(
f"需 0 <= gate_p_low < gate_p_high <= 1"
+4
View File
@@ -409,7 +409,11 @@ async def _run_single_question(
返回:
预测结果字典(含 video_id, question_id, prediction, answer 等)。
"""
# run_id 必须显式入 record:HarnessLog.insert 缺省用**实例** run_id 填充,
# 连续并发 gate 共享单一 gate_log(实例 run_id 为 step 级)时,各臂行必须
# 落自己的臂 run_id,否则 validate 回读 _load_run_rows(臂 run_id) 为空。
record: dict[str, Any] = {
"run_id": run_id,
"video_id": qa.video_id,
"question_id": qa.question_id,
"task_type": qa.task_type,
+9
View File
@@ -348,6 +348,9 @@ def write_gate_evidence(
question_id 字段承载 **unit_id**single=question_idpair=pair_id)——
逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
返回:
无。
关键实现:
逐行 insert(非 insert_many),保证每行独立事务。
"""
@@ -355,6 +358,12 @@ def write_gate_evidence(
with HarnessLog(db_path, run_id) as log:
log.create_table("gate_evidence", _GATE_EVIDENCE_COLS)
# 幂等迁移(对齐 question_gen/run_store 先例):块序贯时代的旧表只有
# block_idx 列,CREATE TABLE IF NOT EXISTS 不补列,直接插 ladder_rank
# 会 OperationalError——为旧 workspace 复用补列,新表恒为 no-op。
cols = {r["name"] for r in log.query("PRAGMA table_info(gate_evidence)")}
if "ladder_rank" not in cols:
log.execute("ALTER TABLE gate_evidence ADD COLUMN ladder_rank INTEGER")
for row in rows:
log.insert("gate_evidence", {"epoch": epoch, "step": step, **row})
+345 -122
View File
@@ -12,6 +12,7 @@
from __future__ import annotations
import asyncio
import json
import math
import random
@@ -34,6 +35,7 @@ from app.harness.checkpoint import (
)
from app.harness.config import RunConfig # noqa: TC001 — 运行时 _compute_total_steps 使用
from app.harness.gate_ladder import BaselineCache, GatePools, build_or_load_gate_pools
from app.harness.log import HarnessLog
from app.harness.observation import (
write_dual_metric,
write_epoch_report,
@@ -45,7 +47,13 @@ from app.harness.observation import (
)
from app.harness.question_units import build_units, unit_correctness_view
from app.harness.store import advance_version
from app.harness.validate import Probation, ValidationOutcome
from app.harness.validate import (
GateSpec,
Probation,
ValidationOutcome,
_ladder_units,
validate_skills_concurrent,
)
from app.harness.workspace import (
ResolvedPaths,
archive_workspace,
@@ -602,6 +610,95 @@ def _write_skip_report(
)
def _assert_disjoint_target_files(targets_by_type: dict[str, str]) -> None:
"""断言本 step 各题型进化目标文件互不相同(设计 v3 §1 fail-fast)。
题型并行进化 + 并行 gate 的前提是 skill 文件不相交;两题型 fallback 到
同一 default-strategy.md 时并行会互相覆盖候选与 accept,必须显式中止
而非静默串行(当前 12 题型均有专属文件,此断言防未来配置漂移)。
参数:
targets_by_type: {题型: 解析后 skill 文件名}。
返回:
无。
异常:
RuntimeError: 存在两个题型映射同一文件。
"""
seen: dict[str, str] = {}
for task_type, target in targets_by_type.items():
if target in seen:
raise RuntimeError(
f"题型 {seen[target]!r}{task_type!r} 映射同一 skill 文件 {target!r},"
"并行进化/gate 不支持共享目标文件(设计 v3 §1)"
)
seen[target] = task_type
def _escape_sql_like(text: str) -> str:
"""转义 SQL LIKE 模式中的全部特殊字符(`\\`、`%`、`_`)为字面匹配。
参数:
text: 待作为 LIKE 前缀字面使用的原始字符串。
返回:
可安全拼入 `LIKE ? ESCAPE '\\'` 模式的转义串。
关键实现细节:
反斜杠必须最先转义,否则会二次转义后续替换产生的转义符。
"""
return text.replace("\\", "\\\\").replace("%", r"\%").replace("_", r"\_")
def _clear_step_rows(db_path: str, *, baseline_run_id: str, epoch: int, step: int) -> None:
"""清空一个 step 的全部旧行(rollout + gate 派生),保证崩溃重跑幂等。
修复前序潜伏 bug:旧实现只清 rollout run_idgate 派生 run_id
`{step_run_id}_gate_%`)从不清理,重跑会累积重复 predictionsHarnessLog
无主键去重),_load_run_rows 的 dict 覆盖使结果依赖 SELECT 顺序。
gate_evidence / quadrant_pair 以 (run_id, epoch, step) 过滤删除;
表不存在(首个 step)时跳过。step_report 为按文件名覆盖写的 JSON,天然幂等。
参数:
db_path: harness.db 路径。
baseline_run_id: 基线 rungate_evidence/quadrant_pair 的 run_id 维度)。
epoch: 轮次(1-based)。
step: epoch 内 step 序号(0-based)。
返回:
无。
关键实现细节:
predictions/traces 的 gate 行按 LIKE 前缀删除,`\\`/`%`/`_` 三个 LIKE
特殊字符全部显式转义(ESCAPE)钉死字面匹配,避免 `..._s1` 误匹配
`..._s10` 类前缀陷阱,也防 run_id 含 `%`/`\\` 时通配误删他 run 行。
"""
from app.harness.inference import PREDICTIONS_SCHEMA, TRACES_SCHEMA
from app.harness.log import HarnessLog
step_run_id = f"{baseline_run_id}_e{epoch}_s{step}"
escaped = _escape_sql_like(step_run_id)
with HarnessLog(db_path, step_run_id, register_run=False) as log:
log.create_table("predictions", PREDICTIONS_SCHEMA)
log.create_table("traces", TRACES_SCHEMA)
for table in ("predictions", "traces"):
log.execute(f"DELETE FROM {table} WHERE run_id=?", (step_run_id,))
log.execute(
f"DELETE FROM {table} WHERE run_id LIKE ? ESCAPE '\\'",
(escaped + r"\_gate\_%",),
)
for table in ("gate_evidence", "quadrant_pair"):
exists = log.query(
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
)
if exists:
log.execute(
f"DELETE FROM {table} WHERE run_id=? AND epoch=? AND step=?",
(baseline_run_id, epoch, step),
)
# ---------------------------------------------------------------------------
# Runner 主类
# ---------------------------------------------------------------------------
@@ -1096,17 +1193,16 @@ class Runner:
"""单 steprollout → correctness 增量 → 诊断 → 累加 system/tool → 按类 gate。"""
run_id = f"{pools.baseline_run_id}_e{epoch}_s{step}"
from app.harness.inference import PREDICTIONS_SCHEMA, TRACES_SCHEMA
from app.harness.log import HarnessLog
# 幂等:重跑同一 step 前先清旧行,避免断点续跑重复累计双计。
# 先 CREATE TABLE IF NOT EXISTSfresh workspace 首跑时表尚未由 run_inference 建),
# register_run=False 避免只读清理污染 _runs 运行状态。
with HarnessLog(str(self._paths.db_path), run_id, register_run=False) as log:
log.create_table("predictions", PREDICTIONS_SCHEMA)
log.create_table("traces", TRACES_SCHEMA)
log.execute("DELETE FROM predictions WHERE run_id=?", (run_id,))
log.execute("DELETE FROM traces WHERE run_id=?", (run_id,))
# 幂等:重跑同一 step 前清 rollout + 全部 gate 派生旧行(修复潜伏 bug:
# 旧实现只清 rollout,gate 行崩溃重跑会累积重复)。
_clear_step_rows(
str(self._paths.db_path),
baseline_run_id=pools.baseline_run_id,
epoch=epoch,
step=step,
)
await self._rollout_batch(batch, run_id)
@@ -1137,7 +1233,7 @@ class Runner:
_guard_infra_failures(result, context="rollout")
# -----------------------------------------------------------------------
# _gate_batch_skillsper task_type gate
# _gate_batch_skills并行进化 + 连续并发 gate(四阶段)
# -----------------------------------------------------------------------
async def _gate_batch_skills(
@@ -1149,18 +1245,95 @@ class Runner:
pools: Pools,
state: _TrainState,
) -> None:
"""按 task_type 独立 evolve → 局部验证 → accept/reject。"""
from app.harness.workspace import VersionedSkillStore
from core.evolution import evolve_single_skill
"""按 task_type 并行 evolve → 连续并发 gate → 字母序统一落账。
四阶段(设计 v3 §2.1):Phase A 并行进化(cooldown/无改动照旧跳过);
Phase B 装配 GateSpec(阶梯出题 + 案例单元排除 + n_max 截断);
Phase C validate_skills_concurrent(共享题槽,统计按阶梯序前缀推进,
只读 state);Phase D 唯一写 state 阶段——按字母序 accept/reject 落账,
与原串行语义等价(题型 skill 文件不相交,合并顺序仅为确定性)。
参数:
epoch / step / total_steps: 训练坐标。
diagnosis: 本 step 诊断结果(skill_case_packs 按题型分组)。
pools: 冻结三池。
state: 训练状态(Phase D 唯一写入点)。
返回:
无。
"""
budget = edit_budget_at(
global_step=state.global_step,
total_steps=total_steps,
start=self._config.edit_budget_start,
end=self._config.edit_budget_end,
)
# ---- Phase A: 并行进化(冷却/无真实改动照旧写 skip 后出清) ----
records = await self._evolve_types_parallel(epoch, step, diagnosis, budget, pools, state)
if not records:
return
_assert_disjoint_target_files({t: r.target_file for t, r in records.items()})
# ---- Phase B: 装配 GateSpec(阶梯出题,收编原 _run_gate_validation 前半) ----
specs = self._assemble_gate_specs(epoch, step, diagnosis, records, pools, state)
# ---- Phase C: 连续并发 gate(只读 state) ----
with HarnessLog(str(self._paths.db_path), f"gate_e{epoch}_s{step}") as gate_log:
outcomes = await validate_skills_concurrent(
workspace_dir=self._config.workspace_dir,
base_skills_version=self._current_version("skills"),
specs=specs,
gate_params=GateParams(
e_confirm=self._config.gate_e_confirm,
e_provisional=self._config.gate_e_provisional,
w_net_min=self._config.gate_w_net_min,
delta_min=self._config.gate_delta_min,
lambda_dir=self._config.gate_lambda_dir,
e_rollback=self._config.gate_e_rollback,
),
gate_guard_err=self._config.gate_guard_err,
baseline_cache=state.baseline_cache,
prompts_version=self._current_version("prompts"),
run_inference=self._make_validate_run_inference_fn(gate_log),
log=gate_log,
concurrency=self._config.concurrency,
)
# ---- Phase D: 唯一写 state 阶段(字母序确定性落账) ----
self._settle_gate_outcomes(epoch, step, records, outcomes, budget, pools, state)
async def _evolve_types_parallel(
self,
epoch: int,
step: int,
diagnosis: DiagnosisResult,
budget: int,
pools: Pools,
state: _TrainState,
) -> dict[str, EvolutionRecord]:
"""Phase A:各题型进化 asyncio.gather 并行,冷却/无改动路径写 skip 出队。
cooldown 与"进化未产出真实改动"(rejected/skipped/内容未变)两类路径
与原串行实现语义一致:写 skip_report 后不进 gate。进化互相独立
(各题型 skill 文件不相交,VersionedSkillStore 只读基线版本),
gather 并行不改变单题型结果。
参数:
epoch / step: 训练坐标。
diagnosis: 本 step 诊断结果。
budget: 当步编辑预算。
pools: 冻结三池。
state: 训练状态(只读)。
返回:
{题型: EvolutionRecord},仅含产出真实改动、待 gate 的题型。
"""
from app.harness.workspace import VersionedSkillStore
from core.evolution import evolve_single_skill
active_types: list[str] = []
for task_type in sorted(diagnosis.skill_case_packs):
# 冷却 admission control
if state.gate_cooldown.get(task_type, 0) > 0:
_write_skip_report(
self._config.workspace_dir,
@@ -1175,22 +1348,44 @@ class Runner:
budget=budget,
)
continue
active_types.append(task_type)
if not active_types:
return {}
evolve_prompts = self._load_evolve_prompts()
skills_version = self._current_version("skills")
async def _evolve_one(task_type: str) -> EvolutionRecord:
pack = diagnosis.skill_case_packs[task_type]
skill_store = VersionedSkillStore(self._paths.skills_dir)
evolve_prompts = self._load_evolve_prompts()
record = await evolve_single_skill(
return await evolve_single_skill(
self._evolve_llm,
pack,
skill_store,
evolve_prompts,
self._current_version("skills"),
skills_version,
budget,
self._config.appendix_consolidate_threshold,
skill_update_mode=self._config.skill_update_mode,
rejected=state.rejected_buffer.get(task_type, []),
)
# 进化未产出真实改动
# 首异常先取消其余进化任务并排水再向上传播(与 validate_skills_concurrent
# 同款语义):避免失败后残留 in-flight LLM 任务与 pending task 警告。
tasks = [asyncio.ensure_future(_evolve_one(t)) for t in active_types]
try:
evolved = await asyncio.gather(*tasks)
except BaseException:
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
raise
records = dict(zip(active_types, evolved, strict=True))
# 无真实改动的题型照旧写 skipped 后出队
gated: dict[str, EvolutionRecord] = {}
for task_type in active_types:
record = records[task_type]
if record.status in ("rejected", "skipped") or (
record.evolved_content == record.original_content
):
@@ -1208,11 +1403,107 @@ class Runner:
rank_clip_triggered=bool(record.clip_info.get("triggered", False)),
)
continue
gated[task_type] = record
return gated
outcome = await self._run_gate_validation(
epoch, step, task_type, pack, record, pools, state
def _assemble_gate_specs(
self,
epoch: int,
step: int,
diagnosis: DiagnosisResult,
records: dict[str, EvolutionRecord],
pools: Pools,
state: _TrainState,
) -> list[GateSpec]:
"""Phase B:为每个待 gate 题型装配 GateSpec(阶梯出题 + 截断)。
案例包按 unit 排除:把每个 case 的 question_id 映射到其所属 unit_id,
命中单元整体排除,防止只排 AR pair 半个成员而给 gate 池灌半个 pair
(下游 _ladder_units 会 fail-fast)。base_skill_content 读 step 起点
版本(self._paths 在 Phase D accept 前不变),保证所有题型对同一
基线版本验证。核心算法保真 #5。
参数:
epoch / step: 训练坐标(拼 gate_run_prefix)。
diagnosis: 本 step 诊断结果(案例排除来源)。
records: Phase A 产出的待 gate 进化记录。
pools: 冻结三池(baseline_run_id)。
state: 训练状态(只读 gate_pools / gate_epoch_observed)。
返回:
与 records 键序一致的 GateSpec 列表。
异常:
RuntimeError: 阶梯引用了题库中不存在的 unit_id。
"""
specs: list[GateSpec] = []
for task_type, record in records.items():
pack = diagnosis.skill_case_packs[task_type]
exclude_units = {
self._gate_questions_by_id[c.question_id].unit_id
for c in pack.failure_cases + pack.success_cases
if c.question_id in self._gate_questions_by_id
}
ladder_unit_ids = state.gate_pools.ladder_for(
task_type,
exclude_units,
p_low=self._config.gate_p_low,
p_high=self._config.gate_p_high,
cold=not state.gate_epoch_observed,
)
# 观测落库
missing = [uid for uid in ladder_unit_ids if uid not in self._gate_units_by_id]
if missing:
raise RuntimeError(
f"gate 阶梯引用未知 unit: {missing[:5]}(gate_pools.json 与题库失配)"
)
ladder_items = [
q for uid in ladder_unit_ids for q in self._gate_units_by_id[uid].questions
]
slug = task_type.lower().replace(" ", "-")
specs.append(
GateSpec(
task_type=task_type,
target_file=record.target_file,
candidate_content=record.evolved_content,
base_skill_content=(self._paths.skills_dir / record.target_file).read_text(
encoding="utf-8"
),
units=tuple(_ladder_units(ladder_items)[: self._config.gate_n_max]),
gate_run_prefix=f"{pools.baseline_run_id}_e{epoch}_s{step}_gate_{slug}",
)
)
return specs
def _settle_gate_outcomes(
self,
epoch: int,
step: int,
records: dict[str, EvolutionRecord],
outcomes: dict[str, ValidationOutcome],
budget: int,
pools: Pools,
state: _TrainState,
) -> None:
"""Phase D:按字母序统一落账(观测落库 + accept/reject 写 state)。
本阶段是 _gate_batch_skills 唯一写 state 的阶段。字母序仅为确定性
(题型 skill 文件不相交,accept 串行叠加时 _accept_skill 基于最新
manifest 版本追加各自 target_file,互不覆盖),与原串行语义等价。
参数:
epoch / step: 训练坐标。
records: Phase A 产出的进化记录。
outcomes: Phase C 产出的 gate 判定。
budget: 当步编辑预算(step_report 落账)。
pools: 冻结三池。
state: 训练状态(唯一写入点)。
返回:
无。
"""
for task_type in sorted(outcomes):
record = records[task_type]
outcome = outcomes[task_type]
write_gate_evidence(
str(self._paths.db_path),
run_id=pools.baseline_run_id,
@@ -1251,88 +1542,6 @@ class Runner:
state.rejected_buffer, task_type, record, outcome, state.global_step
)
async def _run_gate_validation(
self,
epoch: int,
step: int,
task_type: str,
pack: Any,
record: EvolutionRecord,
pools: Pools,
state: _TrainState,
) -> ValidationOutcome:
"""CE-Gate 块序贯配对验证:阶梯出题 → 基线/候选逐块配对 → e-process 四出口。
参数:
epoch: 轮次。
step: epoch 内 step。
task_type: 待验证题型。
pack: SkillCasePack。
record: 进化产物。
pools: 冻结三池。
state: 训练状态。
返回:
ValidationOutcome。
"""
from app.harness.log import HarnessLog
from app.harness.validate import validate_skill_local
# 案例包按 unit 排除:把每个 case 的 question_id 映射到其所属 unit_id
# 命中单元整体排除,防止只排 AR pair 半个成员而给 gate 池灌半个 pair
# (下游 _ladder_units 会 fail-fast)。核心算法保真 #5。
exclude_units = {
self._gate_questions_by_id[c.question_id].unit_id
for c in pack.failure_cases + pack.success_cases
if c.question_id in self._gate_questions_by_id
}
ladder_unit_ids = state.gate_pools.ladder_for(
task_type,
exclude_units,
p_low=self._config.gate_p_low,
p_high=self._config.gate_p_high,
cold=not state.gate_epoch_observed,
)
missing = [uid for uid in ladder_unit_ids if uid not in self._gate_units_by_id]
if missing:
raise ValueError(
f"gate 阶梯[{task_type}] 含 benchmark 中不存在的单元: "
f"{missing[:5]}gate_pools.json 与题库失配)"
)
# 单元展开为逐题(unit 内成员顺序保持),下游 validate 再按阶梯序聚合回单元。
ladder_items = [q for uid in ladder_unit_ids for q in self._gate_units_by_id[uid].questions]
base_skill_content = (self._paths.skills_dir / record.target_file).read_text(
encoding="utf-8"
)
slug = task_type.lower().replace(" ", "-")
run_inference_fn = self._make_validate_run_inference_fn()
with HarnessLog(str(self._paths.db_path), f"gate_{slug}") as gate_log:
return await validate_skill_local(
workspace_dir=self._config.workspace_dir,
base_skills_version=self._current_version("skills"),
task_type=task_type,
target_file=record.target_file,
candidate_content=record.evolved_content,
base_skill_content=base_skill_content,
ladder_items=ladder_items,
gate_params=GateParams(
e_confirm=self._config.gate_e_confirm,
e_provisional=self._config.gate_e_provisional,
w_net_min=self._config.gate_w_net_min,
delta_min=self._config.gate_delta_min,
lambda_dir=self._config.gate_lambda_dir,
e_rollback=self._config.gate_e_rollback,
),
gate_block=self._config.gate_block,
gate_n_max=self._config.gate_n_max,
gate_guard_err=self._config.gate_guard_err,
baseline_cache=state.baseline_cache,
prompts_version=self._current_version("prompts"),
run_inference=run_inference_fn,
log=gate_log,
gate_run_prefix=(f"{pools.baseline_run_id}_e{epoch}_s{step}_gate_{slug}"),
)
# -----------------------------------------------------------------------
# accept / reject / probation
# -----------------------------------------------------------------------
@@ -2413,10 +2622,23 @@ class Runner:
return _noop_builder
def _make_validate_run_inference_fn(self):
"""构造 validate 用的 RunInferenceFn绑定共享依赖)。"""
def _make_validate_run_inference_fn(self, gate_log: HarnessLog):
"""构造 validate 用的 RunInferenceFn(绑定共享依赖与共享 HarnessLog)。
连续并发 gate 下本函数被逐单元高频并发调用:每次调用新建 HarnessLog
连接会重现多连接争 SQLite 写锁(遥测同款教训),故复用调用方传入的
单一 gate_log(单连接 + threading.Lock 串行化)。_record_run 按 run_id
去重,避免逐单元重复 upsert。
参数:
gate_log: 本 step gate 阶段共享的 HarnessLog 实例。
返回:
符合 RunInferenceFn 协议的异步推理函数。
"""
from app.harness.inference import run_inference
from app.harness.log import HarnessLog
recorded: set[str] = set()
async def _run(
questions: list[GeneratedQuestion],
@@ -2424,21 +2646,22 @@ class Runner:
run_id: str,
skills_dir: Path,
) -> InferenceResult:
self._record_run(run_id)
with HarnessLog(str(self._paths.db_path), run_id) as log:
return await run_inference(
questions=questions,
llm=self._llm,
tool_dispatch_fn=self._make_tool_dispatch_fn(skills_dir=skills_dir),
prompt_builder=self._make_prompt_builder(
skills_dir=skills_dir, prompts_dir=self._paths.prompts_dir
),
log=log,
run_id=run_id,
concurrency=self._config.concurrency,
max_steps=self._config.max_steps,
skill_mode=self._config.skill_mode,
)
if run_id not in recorded:
recorded.add(run_id)
self._record_run(run_id)
return await run_inference(
questions=questions,
llm=self._llm,
tool_dispatch_fn=self._make_tool_dispatch_fn(skills_dir=skills_dir),
prompt_builder=self._make_prompt_builder(
skills_dir=skills_dir, prompts_dir=self._paths.prompts_dir
),
log=gate_log,
run_id=run_id,
concurrency=self._config.concurrency,
max_steps=self._config.max_steps,
skill_mode=self._config.skill_mode,
)
return _run
+21 -423
View File
@@ -1,15 +1,15 @@
"""async 块序贯验证编排 — CE-Gate 局部验证的唯一独立子编排器。
"""async 连续并发 gate 验证编排 — CE-Gate 局部验证的唯一独立子编排器。
从 TRM4 core/harness/validate.py (626 行) 迁移,重大重构:
- 同步 → asyncrun_inference 注入为 async callable
- _classify_quadrants → core.evolution.classify_quadrants 纯函数
- 配对逻辑 → 复用 core.evolution.pair_block + 本地证据行组装
- _load_run_rows / _candidate_correctness_from_db → 共享 log.query()
- materialize_candidate_skill 保持同步(纯文件操作)
多题型全部 (单元, 臂) 任务共享题槽并发(validate_skills_concurrent),
统计推进不按到达序,而按预声明的阶梯序前缀消费(_advance_prefix):
base 臂缓存命中瞬间返回、cand 臂必新鲜跑,两臂延迟不对称,按到达序判定
会系统性偏向早到翻转;前缀消费把判定顺序钉回阶梯序,anytime-valid 无条件
成立(核心算法保真 #6,语义修订:块序贯 → 阶梯序前缀逐对序贯)。
基线与候选在同一阶梯前缀上逐配对,只数翻转(基线错→候选对 = W,
基线对→候选错 = L),每块结束调 gate_decision 做四出口判定
基线侧逐题对错走 BaselineCache 内容寻址缓存,miss 才新鲜跑。
基线与候选在同一阶梯前缀上逐单元配对,只数翻转(基线错→候选对 = W,
基线对→候选错 = L),每消费一个单元调一次 gate_decision 做四出口判定
过线即冻结、τ 之后的 in-flight 结果整体丢弃。基线侧单元级对错走
BaselineCache 内容寻址缓存,miss 才新鲜跑;INFRA 单元不写缓存、从配对剔除。
判定逻辑全部在 core/evolution/gate,本模块只负责推理编排与证据收集。
"""
@@ -26,7 +26,7 @@ from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from loguru import logger
from app.harness.gate_ladder import BaselineCache, skill_hash
from app.harness.question_units import build_units, flatten_units, unit_correctness_view
from app.harness.question_units import build_units, unit_correctness_view
from core.evolution import (
INFRA_STOP_REASONS,
GateParams,
@@ -68,7 +68,7 @@ class RunInferenceFn(Protocol):
调用方(runner)负责绑定 llm、tool_dispatch_fn、prompt_builder、
log、concurrency、max_steps、skill_mode 等共享依赖。
validate 侧只传 questions、run_id、skills_dir 三个逐变化的参数。
validate 侧只传 questions、run_id、skills_dir 三个逐任务变化的参数。
"""
async def __call__(
@@ -85,21 +85,6 @@ class RunInferenceFn(Protocol):
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class InferenceRunConfig:
"""一次推理运行的配置三元组,把"如何跑推理"内聚成一组。
字段:
concurrency: 推理并发度。
max_steps: 单题最大推理步数。
skill_mode: 推理 skill 模式("auto" / "manual" / "none")。
"""
concurrency: int
max_steps: int
skill_mode: str
@dataclass
class ValidationOutcome:
"""CE-Gate 局部验证结果:三态动作 + e-process 证据(单元口径)+ 逐题溯源对错。
@@ -260,23 +245,6 @@ def _infra_question_ids_from_db(
}
def _count_infra_units(units: list[QuestionUnit], infra_qids: set[str]) -> int:
"""统计含 INFRA record 的 unit 数(一个 unit 任一题 INFRA 即计 1)。
使护栏分子与分母(r.total,unit 粒度)同口径:AR pair 一 unit 含两 record
逐 record 计数会放大分子致 gate_guard_err 误触发,破坏 unit 粒度一致性
(核心算法保真 #5/#6)。
参数:
units: 当前块的单元列表(single 或 AR pair)。
infra_qids: 本 run 中 stop_reason 属 INFRA 故障族的 question_id 集合。
返回:
含至少一题 INFRA 的 unit 数。
"""
return sum(1 for u in units if any(q.question_id in infra_qids for q in u.questions))
def _candidate_correctness_from_db(
log: HarnessLog,
run_id: str,
@@ -296,164 +264,13 @@ def _candidate_correctness_from_db(
return {q.question_id: rows.get(q.question_id, {}).get("_correct", False) for q in chunk}
# ---------------------------------------------------------------------------
# 块级 async 函数
# ---------------------------------------------------------------------------
async def _resolve_baseline_block(
units: list[QuestionUnit],
task_type: str,
s_hash: str,
prompts_version: str,
baseline_cache: BaselineCache,
base_skills_dir: Path,
run_inference: RunInferenceFn,
log: HarnessLog,
run_id: str,
) -> tuple[dict[str, bool], list[QuestionUnit], int, int]:
"""基线侧处理一个块:缓存优先(unit 键),miss 的单元新鲜跑基线版本并回写缓存。
缓存以 unit_id 为键、存单元级对错(AR pair 双向 AND 折叠后一个布尔)。
miss 的单元展开为逐题送推理,读回逐题预测后经 unit_correctness_view 折叠成
单元级对错再写缓存(核心算法保真 #5)。逐题 predictions 仍逐题落库溯源。
INFRA 隔离(算法 #6):miss 单元内**任一题** stop_reason ∈ {error, parse_error}
即判定该单元为 INFRA 故障——**不写 BaselineCache**(否则瞬时故障永久污染基线
快照)、**不入 b_units**、并从返回的有效单元集中剔除,避免污染 W/L 翻转与配对。
命中缓存的单元恒为有效(此前已成功验证过)。
参数:
units: 当前块的单元列表(single 或 AR pair)。
task_type: 当前验证题型(缓存键成分)。
s_hash: 基线侧生效 skill 的内容哈希(缓存键成分)。
prompts_version: 当前 prompts 版本(缓存键成分)。
baseline_cache: 基线侧单元级对错缓存(键含 unit_id)。
base_skills_dir: 基线 skills 版本目录。
run_inference: 注入的 async 推理函数。
log: HarnessLog 共享实例(推理后读预测)。
run_id: 本块基线 run_id。
返回:
(b_units, valid_units, errors_inc, denom_inc):块内有效 unit_id -> 基线单元
对错、剔除 INFRA 后的有效单元列表、本块新增的 INFRA error 计数与推理题次
分母增量(全命中时为 0, 0)。
"""
miss_units = [
u
for u in units
if baseline_cache.get(task_type, s_hash, prompts_version, u.unit_id) is None
]
errors_inc = 0
denom_inc = 0
infra_qids: set[str] = set()
if miss_units:
miss_questions = flatten_units(miss_units)
r_b = await run_inference(miss_questions, run_id=run_id, skills_dir=base_skills_dir)
infra_qids = _infra_question_ids_from_db(log, r_b.run_id, miss_questions)
# 护栏分子与分母(r.total,unit 粒度)同口径:含 INFRA record 的 unit 计 1
# 避免 AR pair(一 unit 两 record)逐 record 计数放大分子致误触发;仍涵盖
# error + parse_error_infra_question_ids_from_db 口径),parse_error 风暴不被绕过。
errors_inc = _count_infra_units(miss_units, infra_qids)
denom_inc = r_b.total
fresh_per_q = _candidate_correctness_from_db(log, r_b.run_id, miss_questions)
fresh_units = unit_correctness_view(miss_units, fresh_per_q)
# 只回写非 INFRA 单元;INFRA 单元不入缓存(不永久污染基线快照)
for u in miss_units:
if any(q.question_id in infra_qids for q in u.questions):
continue
baseline_cache.put(task_type, s_hash, prompts_version, u.unit_id, fresh_units[u.unit_id])
valid_units = [
u for u in units if not any(q.question_id in infra_qids for q in u.questions)
]
b_units: dict[str, bool] = {}
for u in valid_units:
val = baseline_cache.get(task_type, s_hash, prompts_version, u.unit_id)
assert val is not None, f"基线缓存补齐后仍有 miss: unit={u.unit_id} run_id={run_id}"
b_units[u.unit_id] = val
return b_units, valid_units, errors_inc, denom_inc
async def _run_candidate_block(
units: list[QuestionUnit],
cand_dir: Path,
run_inference: RunInferenceFn,
log: HarnessLog,
run_id: str,
) -> tuple[dict[str, bool], int, int]:
"""候选侧处理一个块:单元展开为逐题全块新鲜跑候选版本并从 db 读逐题对错。
返回逐题对错映射(question_id -> bool),折叠为单元视图交由调用方完成,
逐题结果同时用于 candidate_correctness 溯源与二轨 correctness 合并。
参数:
units: 当前块的单元列表。
cand_dir: 已物化的候选 skills 目录。
run_inference: 注入的 async 推理函数。
log: HarnessLog 共享实例(推理后读预测)。
run_id: 本块候选 run_id。
返回:
(c_per_q, errors_inc, denom_inc):块内 question_id -> 候选对错。
"""
questions = flatten_units(units)
r_c = await run_inference(questions, run_id=run_id, skills_dir=cand_dir)
c_per_q = _candidate_correctness_from_db(log, r_c.run_id, questions)
infra_qids = _infra_question_ids_from_db(log, r_c.run_id, questions)
# 护栏分子与分母(r.total,unit 粒度)同口径:含 INFRA record 的 unit 计 1
# (见 _count_infra_units),涵盖 error + parse_error。
errors_inc = _count_infra_units(units, infra_qids)
return c_per_q, errors_inc, r_c.total
def _build_evidence_rows(
units: list[QuestionUnit],
b_units: dict[str, bool],
c_units: dict[str, bool],
task_type: str,
block_idx: int,
) -> list[dict]:
"""组装一个块的 gate_evidence 单元级证据行。
证据行按 unit 口径(question_id 字段存 unit_id、correct 存单元级对错),
与 e-process 判定同粒度;逐题预测明细仍在 predictions 表逐题溯源。
e_value 留 None 待块判定后回填,stop_reason 留空串待终态回填。
参数:
units: 当前块的单元列表。
b_units: 块内 unit_id -> 基线单元对错。
c_units: 块内 unit_id -> 候选单元对错。
task_type: 当前验证题型。
block_idx: 当前块序号。
返回:
单元级证据行列表。
"""
return [
{
"question_id": u.unit_id,
"task_type": task_type,
# 落库列已更名 ladder_rank(阶梯序号);旧块路径此处值仍为块号,
# 仅键名对齐 gate_evidence 表结构以保持落库兼容。
"ladder_rank": block_idx,
"baseline_correct": b_units[u.unit_id],
"candidate_correct": c_units[u.unit_id],
"e_value": None,
"stop_reason": "",
}
for u in units
]
# ---------------------------------------------------------------------------
# INFRA 护栏
# ---------------------------------------------------------------------------
def _check_infra_guard(errors: int, infra_denom: int, gate_guard_err: float) -> None:
"""跨块累计 INFRA 错误率护栏:分母 >=10 且超阈值时 raise。
"""累计 INFRA 错误率护栏:分母 >=10 且超阈值时 raise。
参数:
errors: 两侧累计 error 计数。
@@ -484,13 +301,13 @@ def _finalize_outcome(
evidence_rows: list[dict],
task_type: str,
) -> ValidationOutcome:
"""块循环终态判定组装为 ValidationOutcome。
"""将终态判定组装为 ValidationOutcome。
四象限/准确率/W/L 均按单元口径(base_obs/cand_obs 为 unit_id -> bool),
candidate_correctness 独立保留逐题溯源(供 runner 二轨合并进 state.correctness)。
参数:
verdict: 最后一块的 gate 判定结果。
verdict: 终态 gate 判定结果。
w: 累计 W(基线错→候选对单元翻转)。
l: 累计 L(基线对→候选错单元翻转)。
n_used: 已消费的阶梯单元数。
@@ -576,228 +393,6 @@ def _ladder_units(ladder_items: list[GeneratedQuestion]) -> list[QuestionUnit]:
return units
async def _run_local_validation(
workspace_dir: Path,
cand_dir: Path,
base_skills_version: str,
task_type: str,
base_skill_content: str,
units: list[QuestionUnit],
gate_params: GateParams,
gate_block: int,
gate_guard_err: float,
baseline_cache: BaselineCache,
prompts_version: str,
run_inference: RunInferenceFn,
log: HarnessLog,
gate_run_prefix: str,
) -> ValidationOutcome:
"""块序贯循环主体:逐块基线(缓存优先)/候选按单元配对推理,块间 e-process 判定。
按 gate_block 切**单元**前缀(AR pair 整锁在同一块,不跨块拆分),每块先补齐
基线侧缓存 miss(新鲜跑基线版本并按 unit_id 写 BaselineCache),再全块跑候选,
折叠成单元视图后配对累计 W/L 调 gate_decision;非 continue 即早停。单元尽时
最后一块的判定即终态(n_remaining=0 走 provisional/inertia 分支),无循环外补判。
参数:
workspace_dir: Workspace 根目录。
cand_dir: 已物化的候选 skills 目录。
base_skills_version: 基线 skills 版本名。
task_type: 当前验证题型。
base_skill_content: 基线侧生效 skill 全文(skill_hash 作缓存键成分)。
units: 已截断到 gate_n_max 的阶梯单元序(single 或 AR pair)。
gate_params: e-process 判据阈值组。
gate_block: 块大小(单位为**单元数**)。
gate_guard_err: 跨块累计 INFRA 错误率护栏(分母 >=10 才触发)。
baseline_cache: 基线侧单元级对错缓存(键含 unit_id)。
prompts_version: 当前 prompts 版本(缓存键成分)。
run_inference: 注入的 async 推理函数。
log: HarnessLog 共享实例。
gate_run_prefix: 块 run_id 前缀(含 "_gate_" 标记)。
返回:
ValidationOutcome。
关键实现:
INFRA 护栏跨块累计基线+候选两侧的 error 计数,分母(总推理题次,仍逐题计)
>=10 且错误率超 gate_guard_err 时直接 raise,避免坏批次污染判定。
"""
w = 0
l = 0 # noqa: E741
n_used = 0
n_excluded = 0 # 累计被 INFRA 隔离剔除的单元数(从阶梯分母扣除)
errors = 0
infra_denom = 0
evidence_rows: list[dict] = []
base_obs: dict[str, bool] = {}
cand_obs: dict[str, bool] = {}
candidate_per_q: dict[str, bool] = {}
s_hash = skill_hash(base_skill_content)
base_skills_dir = workspace_dir / "skills" / base_skills_version
unit_chunks = [units[i : i + gate_block] for i in range(0, len(units), gate_block)]
n_plan = len(units)
verdict: GateVerdict | None = None
for block_idx, unit_chunk in enumerate(unit_chunks):
# Phase 1: 基线侧(缓存优先,miss 新鲜跑,INFRA 单元剔除)
b_units, valid_chunk, err_b, den_b = await _resolve_baseline_block(
units=unit_chunk,
task_type=task_type,
s_hash=s_hash,
prompts_version=prompts_version,
baseline_cache=baseline_cache,
base_skills_dir=base_skills_dir,
run_inference=run_inference,
log=log,
run_id=f"{gate_run_prefix}_b{block_idx}_base",
)
# 本块全 INFRA:无有效单元可配对——候选无需空跑,仅把基线侧错误计入护栏后
# 累计剔除数进入下一块(护栏仍能在整轮 INFRA 错误率超阈值时熔断)。
n_excluded += len(unit_chunk) - len(valid_chunk)
if not valid_chunk:
errors += err_b
infra_denom += den_b
_check_infra_guard(errors, infra_denom, gate_guard_err)
continue
# 候选侧只跑基线侧判定有效(非 INFRA)的单元,保证配对 unit_ids 两侧一致
c_per_q, err_c, den_c = await _run_candidate_block(
units=valid_chunk,
cand_dir=cand_dir,
run_inference=run_inference,
log=log,
run_id=f"{gate_run_prefix}_b{block_idx}_cand",
)
# Phase 2: INFRA 护栏(跨块累计,分母 >=10 才触发)——写缓存前置于此已由
# _resolve_baseline_block 保证 INFRA 单元不落缓存,此处仅做整轮错误率熔断。
errors += err_b + err_c
infra_denom += den_b + den_c
_check_infra_guard(errors, infra_denom, gate_guard_err)
# Phase 3: 折叠成单元视图 + 配对 + 证据行 + 块间判定(均用有效单元)
c_units = unit_correctness_view(valid_chunk, c_per_q)
candidate_per_q.update(c_per_q)
unit_ids = [u.unit_id for u in valid_chunk]
pair_result = pair_block(b_units, c_units, unit_ids)
for uid, (b, c) in pair_result.observed.items():
base_obs[uid] = b
cand_obs[uid] = c
block_rows = _build_evidence_rows(valid_chunk, b_units, c_units, task_type, block_idx)
w += pair_result.w
l += pair_result.l # noqa: E741
n_used += len(valid_chunk)
# 阶梯剩余按扣除 INFRA 后的有效分母计:n_remaining = (n_plan - n_excluded) - n_used
verdict = gate_decision(w, l, n_used, (n_plan - n_excluded) - n_used, params=gate_params)
for row in block_rows:
row["e_value"] = verdict.e_value
evidence_rows.extend(block_rows)
if verdict.decision != "continue":
break
# verdict 仍为 None ⟺ 全部单元被 INFRA 排除(空 ladder 已在入口拒绝)。
# 明确失败,避免落到误导性的"空阶梯"断言而无法定位为 INFRA 原因。
if verdict is None:
raise RuntimeError("gate 阶梯所有 unit 被判为 INFRA 排除,无法验证(检查推理基础设施)")
# 最后一块判定即终态(n_remaining=0 → provisional/inertia
return _finalize_outcome(
verdict=verdict,
w=w,
l=l,
n_used=n_used,
n_plan=n_plan,
base_obs=base_obs,
cand_obs=cand_obs,
candidate_per_q=candidate_per_q,
evidence_rows=evidence_rows,
task_type=task_type,
)
async def validate_skill_local(
workspace_dir: Path,
base_skills_version: str,
task_type: str,
target_file: str,
candidate_content: str,
base_skill_content: str,
ladder_items: list[GeneratedQuestion],
gate_params: GateParams,
gate_block: int,
gate_n_max: int,
gate_guard_err: float,
baseline_cache: BaselineCache,
prompts_version: str,
run_inference: RunInferenceFn,
log: HarnessLog,
gate_run_prefix: str,
) -> ValidationOutcome:
"""块序贯配对验证:阶梯出题,基线/候选逐块配对,e-process 四出口早停。
参数:
workspace_dir: workspace 根目录。
base_skills_version: 基线 skills 版本名(候选物化复制源)。
task_type: 待验证题型。
target_file: fallback 解析后该题型的真实生效 skill 文件名
record.target_file,可能是共享 default-strategy.md);
候选物化写此文件,与 accept 路径同源。
candidate_content: 候选 skill 全文。
base_skill_content: 基线侧该题型解析后生效 skill 文件全文
skill_hash(base_skill_content) 作 BaselineCache 键成分)。
ladder_items: 阶梯序题目列表(已排除本 step 案例包题)。
gate_params: e-process 判据阈值组。
gate_block: 块大小(单位为**单元数**,AR pair 整锁不跨块拆)。
gate_n_max: 单 gate 单元数上限(阶梯截断到此数量个单元)。
gate_guard_err: 跨块累计 INFRA 错误率护栏(分母 >=10 才触发)。
baseline_cache: 基线侧单元级对错缓存(键含 unit_id)。
prompts_version: 当前 prompts 版本(缓存键成分)。
run_inference: 注入的 async 推理函数(RunInferenceFn 协议)。
log: HarnessLog 共享实例(供 DB 回读逐题对错)。
gate_run_prefix: gate 内推理 run_id 前缀,必须含 "_gate_"
(防泄露过滤靠它识别)。块 run_id = f"{prefix}_b{block_idx}_{arm}"
返回:
ValidationOutcome。单元级证据记入 outcome.evidence_rows 随结果返回,
gate_evidence 落库由调用方(runner)负责。
"""
if "_gate_" not in gate_run_prefix:
raise ValueError(f"gate_run_prefix 必须含 '_gate_'(防泄露过滤依赖): {gate_run_prefix!r}")
if not ladder_items:
raise ValueError(f"task_type={task_type} 阶梯为空,无法验证")
# 阶梯题序聚合为单元并按信息阶梯序截断到 gate_n_max 个单元(AR pair 整锁不拆)
units = _ladder_units(ladder_items)[:gate_n_max]
cand_dir = materialize_candidate_skill(
workspace_dir, base_skills_version, target_file, candidate_content
)
try:
return await _run_local_validation(
workspace_dir=workspace_dir,
cand_dir=cand_dir,
base_skills_version=base_skills_version,
task_type=task_type,
base_skill_content=base_skill_content,
units=units,
gate_params=gate_params,
gate_block=gate_block,
gate_guard_err=gate_guard_err,
baseline_cache=baseline_cache,
prompts_version=prompts_version,
run_inference=run_inference,
log=log,
gate_run_prefix=gate_run_prefix,
)
finally:
try:
shutil.rmtree(cand_dir)
except OSError as e:
logger.warning("候选临时目录清理失败 {}: {}", cand_dir, e)
# ---------------------------------------------------------------------------
# 连续并发 gate:数据结构 + 前缀消费(algo #6 语义修订:块序贯 → 阶梯序前缀逐对序贯)
# ---------------------------------------------------------------------------
@@ -1246,9 +841,12 @@ async def validate_skills_concurrent(
) -> dict[str, ValidationOutcome]:
"""连续并发 gate:多题型全部臂共享题槽并发,统计按阶梯序前缀有序推进。
发射顺序 = 题型 round-robin × 题型内阶梯序(base 先 cand 后);题型过线即
冻结,其排队任务启动时自查冻结标志撤销,in-flight 结果不计入(τ 之后样本,
合法丢弃)。全部题型判定后统一组装 ValidationOutcome。
关键实现细节:
发射顺序 = 题型 round-robin × 题型内阶梯序(base 先 cand 后);题型过线
即冻结,其排队任务启动时自查冻结标志撤销,in-flight 结果不计入(τ 之后
样本,合法丢弃);候选目录逐个物化即登记、统一 finally 清理(中途失败不
泄漏);任一任务异常先 cancel+排水其余任务再向上传播;全部题型判定后
统一经 _finalize_outcome 组装。
参数:
workspace_dir: workspace 根目录(候选物化用)。
-1
View File
@@ -42,7 +42,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
-1
View File
@@ -39,7 +39,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
-1
View File
@@ -42,7 +42,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
-1
View File
@@ -22,7 +22,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
-1
View File
@@ -23,7 +23,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
+6 -5
View File
@@ -10,8 +10,8 @@ harness:
workspace_dir: "workspaces/train-videomme"
store_dir: store
mode: train
run_id: train_videomme_v1
concurrency: 24
run_id: train_videomme_v2
concurrency: 32
max_steps: 40
skill_mode: auto
n_samples: 0
@@ -26,7 +26,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
@@ -51,8 +50,10 @@ harness:
# 可训练性预检(WP3):val 单元 < eval_min_per_class 或 非test单元 < trainable_min_units 的题型剔除
eval_min_per_class: 2
trainable_min_units: 8
# mini-batch
batch_size: 10
# mini-batch —— 对齐 TRM4 正式实验 batch=40sh --batch-size 40 覆盖 yaml 15 的最终生效值):
# 8 可训题型 × 每型约 5 题/step,保住题型级诊断信号;同时 steps/epoch 180/40≈5
# 进化/gate 验证轮数比 batch=10 少 4 倍。
batch_size: 40
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
+5
View File
@@ -285,6 +285,11 @@
"id": "plan:gate-speedup",
"label": "连续并发 gate + Redis 复用实现计划",
"type": "plan"
},
{
"id": "review:gate-speedup-final",
"label": "连续并发 gate 终审与交付",
"type": "review"
}
],
"links": [
+4 -2
View File
@@ -1,6 +1,6 @@
# Research Wiki 索引
> 自动生成,更新时间:2026-07-17 03:12 UTC
> 自动生成,更新时间:2026-07-17 09:38 UTC
## design (40)
- [2026-07-06-core-agent-adapters-llm-design](designs/2026-07-06-core-agent-adapters-llm-design.md) `design:2026-07-06-core-agent-adapters-llm-design`
@@ -108,11 +108,13 @@
- [连续并发 gate + Redis 复用实现计划](plans/gate-speedup.md) `plan:gate-speedup`
- [项目基础设施初始化计划](plans/infrastructure-setup.md) `plan:infrastructure-setup`
## review (4)
## review (6)
- [2026-07-16-preflight-final-review](reviews/2026-07-16-preflight-final-review.md) `review:2026-07-16-preflight-final-review`
- [2026-07-16-preflight-train-review](reviews/2026-07-16-preflight-train-review.md) `review:2026-07-16-preflight-train-review`
- [2026-07-17-gate-speedup-final-review](reviews/2026-07-17-gate-speedup-final-review.md) `review:2026-07-17-gate-speedup-final-review`
- [question-gen v2 设计对抗审核 — 六路独立核验(四层病灶闭合度 + 契约一致性)](reviews/2026-07-15-question-gen-v2-adversarial-audit.md) `review:2026-07-15-question-gen-v2-adversarial-audit`
- [训练前修复分支终审](reviews/preflight-final-review.md) `review:preflight-final-review`
- [连续并发 gate 终审与交付](reviews/gate-speedup-final.md) `review:gate-speedup-final`
## schema (6)
- [表结构 v3 出题日志/观测(unit_verdict / collapse_metrics / quarantine / facts / resume](schemas/v3-question-gen-logging.md) `schema:v3-question-gen-logging`
+2
View File
@@ -142,3 +142,5 @@
- [2026-07-17 03:12 UTC] 新增 plan: 连续并发 gate + Redis 复用实现计划 (plan:gate-speedup)
- [2026-07-17 03:12 UTC] 新增边: plan:gate-speedup --implements--> design:gate-speedup
- [2026-07-17 03:12 UTC] 重建索引: 117 篇页面
- [2026-07-17 09:38 UTC] 新增 review: 连续并发 gate 终审与交付 (review:gate-speedup-final)
- [2026-07-17 09:38 UTC] 重建索引: 119 篇页面
@@ -0,0 +1,36 @@
# 连续并发 gate 重构 · 终审与交付记录
> 2026-07-17。分支 feat/gate-speedup(18 commits)→ merge 172b7a8 入 feat/question-gen-v3。
> 设计 research-wiki/designs/2026-07-16-gate-speedup-design.md(v3);计划 research-wiki/plans/2026-07-16-gate-speedup.md。
## 交付摘要
| 项 | 结果 |
|---|---|
| 7 Task(SDD:实现+三层 Codex 审/任务) | 全部收口;全量 tests/ 1561 passed,覆盖率 83% |
| 核心算法 | #6 已批准语义修订(块序贯→阶梯序前缀逐对序贯);#4/#5 及 core/evolution 零改动(diff 为空) |
| 终审 | Codex 整分支五维审(1 Critical:gate_evidence 旧表迁移,已修 eb12006)+ Opus 独立残留/计划符合性审:VERDICT CLEAN |
## 审查抓出并修复的真缺陷(按发现轮次)
| 缺陷 | 严重度 | 修复 |
|---|---|---|
| 预灌 BaselineCache 回归均值偏差(设计期) | 设计 Critical | 方案废弃,改连续并发 gate |
| 到达序消费配对偏差(设计期) | 设计 Critical | 阶梯序前缀消费 |
| 尾部 INFRA 绕过题尽出口 | Critical | 剔除后重判(1e92928) |
| acquire 超宽自死锁 + 取消半持有泄漏 | Critical/Important | fail-fast + 回滚(232afd5/30c1cf1) |
| 编排器物化中途泄漏 + gather 首异常悬挂任务 | Critical×2 | 逐个登记 + cancel-drain(9e8a254) |
| step 重跑不清 gate 行(前序潜伏 bug) | Critical | _clear_step_rows(ea6bec5)+ LIKE 全转义(b3aba7c) |
| **共享 gate_log 下 predictions run_id 契约断裂(gate 静默全拒)** | Critical | inference record 显式 run_id(0b83993) |
| gate_evidence 旧表缺 ladder_rank 迁移 | Critical(跨版本) | 幂等 ALTER(eb12006) |
## 遗留与豁免(记录在案)
- runner.py 两处 HEAD 前既有 ruff format 债(~:1096/~:2296)与 `_filter_untrainable_types` C(17):存量,未触碰。
- 旧 checkpoint(含 gate_block 指纹键)resume 时静默兼容不告警:中低风险,本项目训练均 --fresh。
- skipped 记录不做 target_file disjoint 检查:无冲突路径,防御加固候选。
- 小题型合并进化 default-strategy.md(方案 B):future work,本轮 A 方案结果作对照。
## 重启
2026-07-17 05:37 训练在新代码上重启(tmux train_videomme,--fresh,workspace 全新)。Redis 复用:今日键已一次性续期 7 天,.env TTL=604800。
@@ -0,0 +1,9 @@
---
type: review
node_id: review:gate-speedup-final
title: "连续并发 gate 终审与交付"
date: 2026-07-17
---
# 连续并发 gate 终审与交付
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# ============================================================================
# Video-MME 900 自进化训练 —— 4090-server Slurm(gpu-sbatch)入口
# ----------------------------------------------------------------------------
# 提交(仓库根目录下,1 卡 200 小时):
# gpu-sbatch 1 scripts/train_videomme_slurm.sh --time 200:00:00
#
# 断点续跑(上一任务超时/被杀后,不删 workspace,从 checkpoint 恢复):
# RESUME=1 gpu-sbatch 1 scripts/train_videomme_slurm.sh --time 200:00:00
#
# 观察:
# tail -f slurm-<jobid>.out # Slurm 标准输出(与 logs/ 双写)
# tail -f logs/train_videomme.log
# gpu-status / gpu-cancel
#
# 前置(那台机器上须就绪):
# 1. 分支 = feat/question-gen-v3(main 是旧代码,没有本训练管线)
# 2. .env 存在且 REDIS_URL / LLM API 可从该机访问;REDIS_CACHE_TTL 为正整数
# 3. store/seeds/adhoc-baseline/、store/videos/、workspaces/video-split/pools.json 已同步
# 4. 全新重跑前:rm -rf workspaces/train-videomme logs/train_videomme.log
#
# 说明:CUDA_VISIBLE_DEVICES 由 Slurm cgroup 注入,本脚本与内层脚本均不覆盖;
# 日志三重保险即时输出(PYTHONUNBUFFERED + python -u + loguru stderr)。
# ============================================================================
set -euo pipefail
# sbatch 会把脚本拷贝到 Slurm spool 目录执行,$0 不指向仓库路径——须用
# SLURM_SUBMIT_DIR(提交命令时所在目录,即仓库根)定位;本地直跑时回退 dirname。
cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")/..}"
echo "== 工作目录: $(pwd) =="
[[ -f "config/train_videomme.yaml" ]] || {
echo "错误: 当前目录不是仓库根(找不到 config/train_videomme.yaml)。请在仓库根目录提交 gpu-sbatch。" >&2
exit 1
}
echo "== Slurm Job: ${SLURM_JOB_ID:-unknown} | CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-unset} =="
nvidia-smi || true
mkdir -p logs
if [[ "${RESUME:-0}" == "1" ]]; then
# 续跑:跳过 Phase 0(seed 已在)与 --fresh,直接从 workspace checkpoint 恢复
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTHONUNBUFFERED=1
set -a; source .env; set +a
PYTHON="$(conda run -n Video-Tree-TRM which python)"
"${PYTHON}" -u main.py --config config/train_videomme.yaml --resume \
2>&1 | tee -a logs/train_videomme.log
else
# 全新训练:复用自包含实验脚本(Phase 0 建/校验 seed + Phase 1 --fresh)
bash scripts/train_videomme.sh 2>&1 | tee logs/train_videomme.log
fi
echo "== 训练进程退出,Job ${SLURM_JOB_ID:-unknown} 结束 =="
+25
View File
@@ -0,0 +1,25 @@
你是一个视觉证据提取器。你服务于一个视频问答推理系统,该系统通过工具调用你来查看视频关键帧的画面内容。该系统掌握完整的视频上下文,而你只能看到当前这几帧。因此,你的职责是准确描述画面内容,推理和判断由该系统完成。
## 你会收到的输入
1. 1-4 张视频关键帧图片
2. 一个针对画面内容的视觉问题
## 工作原则
先陈述画面事实,后回答问题。你必须先逐帧列出画面中直接可见的原子事实——人物外观与着装、正在发生的动作、物体及其空间位置关系——然后才基于这些事实回答问题。[视觉回答] 中的每个断言都要标注它依据的帧号和事实编号,[画面事实] 中没有列出的内容不得出现在回答里。
画面内的文字(计分板、字幕、标牌、卡牌文本等)必须逐字转录,并为每处文字标注可读性:清晰可读、部分可读或模糊不可辨。标注"模糊不可辨"时禁止给出猜测的内容——承认看不清比编造一个流畅的答案更有价值。
不要凭部分外观特征(发色、胡须、体型)断定画面人物是某个具体的人。你只描述看到的特征,身份匹配由掌握完整视频上下文的推理系统完成。
不要做超出当前画面的推断。你看不到这几帧之前或之后发生了什么,因此不要推断事件的先后顺序、因果关系或累计次数。例如,你可以说"9 号球衣的球员正在射门",但不要说"这是他的第 3 次射门"——你无法从当前帧中得知这一点。
如果画面中没有回答问题所需的证据,输出 [证据不存在] 并具体说明缺少什么要素。
## 输出格式
[画面事实] <逐帧编号列出直接可见的原子事实,如"帧1-a: ……";画面内文字逐字转录并标注可读性>
[视觉回答] <基于画面事实回答问题中询问的每个要素,每个断言标注依据的帧号和事实编号,如(帧1-a)>
[证据不存在] <画面中未出现回答该问题所需的具体要素时,说明缺少什么;有充分证据时省略此段>
[其他信息] <画面中与问题无关但可能有用的视觉信息,没有则省略>
+29
View File
@@ -0,0 +1,29 @@
你是一个视觉证据核实器。你将收到一段关于图片的描述(由另一个模型生成),你的任务是对照原始图片,逐条核实该描述的准确性。
## 你会收到的输入
1. 与描述生成时相同的图片
2. 用户当时提出的问题
3. 另一个模型基于这些图片生成的描述
## 工作原则
首先检查描述是否回答了问题中的每个要素。然后逐条检查每一个事实性陈述:
- 问题中询问的每个要素是否都得到了回答?
- 描述提到的实体是否确实存在于画面中?
- 描述的动作是否确实正在发生?
- 描述引用的文字(计分板、字幕等)是否与画面中的文字一致?
- 描述是否包含了画面中不存在的信息?
- 描述的外观细节(颜色、发型、穿着)是否与画面一致?
如果描述中包含超出画面的推断(如因果关系、时序判断、累计计数),指出这些是推断而非画面事实。
## 输出格式
details=<逐条核实结果>; confidence=<0.0-1.0>
置信度含义:
- 1.0: 描述完全准确,每个细节都与画面一致
- 0.7-0.9: 主要内容准确,个别细节有出入或无法确认
- 0.4-0.6: 部分准确,但存在明显错误或过度推断
- 0.0-0.3: 描述与画面严重不符
+21
View File
@@ -0,0 +1,21 @@
你是一个视频搜索结果摘要器。你服务于一个视频问答推理系统,该系统通过语义搜索找到了一个可能相关的视频节点,需要你快速判断该节点与问题的相关性并提取关键信息。推理和最终判断由该系统完成。
## 你会收到的输入
1. 用户正在研究的问题
2. 一个语义搜索命中的视频节点的描述文本和字幕
## 工作原则
仅基于提供的内容回答,不使用外部知识。你看不到其他节点的内容,因此不要推断跨节点的事件顺序、因果关系或全局结论。
由于推理系统需要快速扫描多个搜索结果,请保持输出简洁(3-5 句关键信息)。优先报告能直接回答问题的事实,其次报告间接相关的背景信息。
字幕中的引用是重要证据来源,请保留关键原文片段。
如果内容与问题无关,明确说明"该节点未包含与问题直接相关的信息",并用一句话概括该节点的实际内容。
## 输出格式
[关键信息] <3-5 句与问题相关的关键事实,按相关性排列>
[原文] <1-3 句与问题最相关的字幕原文或描述原文,保留原始措辞,不改写不概括。无关则省略>
+24
View File
@@ -0,0 +1,24 @@
你是一个搜索结果摘要核实器。你将收到一段关于视频搜索结果的摘要(由另一个模型生成),以及该节点的原始描述和字幕。请核实摘要是否准确。
## 你会收到的输入
1. 用户正在研究的问题
2. 节点的原始描述文本和字幕
3. 另一个模型基于上述内容生成的摘要
## 检查要点
- 摘要提到的事实是否确实存在于原始内容中?
- 摘要是否包含了原始内容中不存在的推断?
- 摘要是否遗漏了原始内容中与问题高度相关的重要信息?
- [原文] 引用是否准确保留了原始措辞?
## 输出格式
details=<逐条核实结果>; confidence=<0.0-1.0>
置信度含义:
- 1.0: 摘要完全准确,无遗漏
- 0.7-0.9: 主要内容准确,个别细节有出入
- 0.4-0.6: 部分准确,但存在明显错误或过度推断
- 0.0-0.3: 摘要与原始内容严重不符
+102
View File
@@ -0,0 +1,102 @@
## 角色
你是一个视频树搜索 Agent,任务是在预构建的层次化视频树上导航,收集视频证据并回答四选一单选题(A/B/C/D)。你是一个谨慎的证据收集者,宁可多搜一步验证也不轻易下结论。
你最常犯的错误是找到第一条支持证据就急于提交答案,而没有为竞争选项做独立搜索。为了避免这一点,你应该在每次工具调用前通过 reflect 审视已有证据是否真的足以区分选项,在每次工具调用后通过 plan 评估下一步是否值得花费步数预算。对每个选项都应形成判断——"无直接证据"本身也是有效的判断。
## 能力边界
你通过工具浏览节点的文本摘要、字幕转写和结构化描述,但无法直接观看视频画面。如果需要确认画面中的视觉细节(人物外观、计分板数字、物体空间位置等),必须使用 observe_frame 工具。
需要注意的是,你获取的所有信息都是文本形式的二次表示,而非视频原始内容。文本摘要可能存在概括偏差或遗漏细节,字幕转写可能存在 OCR 识别错误。因此,对于决定最终答案的关键证据,应尽可能通过多个节点或多种信息源(摘要 + 字幕 + 视觉)进行交叉验证。
## 输出格式
你的 thinking(深度推理)可以自由分析,不受格式约束。你的 content 必须输出纯 JSON,包含三个顶层字段:
```json
{
"reflect": { ... },
"plan": { ... },
"action": {"tool": "工具名称", "args": { ... }}
}
```
其中 reflect 用于结构化反思(第一轮可省略),plan 用于结构化规划,action 指定本轮要调用的工具及其参数。reflect 和 plan 的具体字段由当前加载的搜索策略定义。action 的格式是固定的:tool 为工具名称字符串,args 为该工具的参数字典。
## 视频树结构
视频被组织为三层树,每层提供不同粒度的信息。你应该根据当前需要的信息精度选择在哪一层搜索。
### L1 — 场景(~5 分钟)
L1 是最粗粒度的层级,每个节点覆盖约 5 分钟的视频片段。适合快速建立全局认知,了解视频的整体结构、主题和时间线。
| 字段 | 内容 |
|------|------|
| scene_summary | 场景整体摘要 |
| main_setting | 主要场景设定 |
| key_entities | 关键实体列表 |
| main_actions | 主要动作 |
| topic_keywords | 主题关键词 |
| temporal_flow | 时间推进描述 |
| visible_text | 画面中可见的文字 |
| subtitle | 完整字幕(较长) |
### L2 — 事件(~30 秒)
L2 是中间粒度,每个节点覆盖约 30 秒的视频片段。适合缩小搜索范围后深入理解具体事件的因果关系和实体行为。
| 字段 | 内容 |
|------|------|
| event_description | 事件描述 |
| entities | 出现的实体 |
| actions | 发生的动作 |
| action_subjects | 动作主体 |
| spatial_relations | 空间关系变化 |
| state_changes | 状态变化 |
| visible_text | 画面中可见的文字 |
| subtitle | 字幕片段 |
### L3 — 关键帧(单帧)
L3 是最细粒度的层级,每个节点对应一张关键帧。适合获取精确证据、确认具体的视觉细节和时间戳。
| 字段 | 内容 |
|------|------|
| frame_summary | 帧内容描述 |
| visible_entities | 可见实体 |
| ongoing_actions | 正在发生的动作 |
| spatial_layout | 空间布局 |
| visual_attributes | 光照、主色调、机位 |
| visible_text | 画面中可见的文字 |
| subtitle | 字幕(短) |
### 信任层级
三个层级的信息有不同的信任度。L1 和 L2 的摘要是概括性的,适合用于导航和定位相关区域,但它们可能遗漏关键细节或存在概括偏差。L3 关键帧是最细粒度的信息来源——在给出最终答案前,你应该优先基于 L3 级证据做判断,而非仅凭 L1/L2 摘要下结论。当外部知识与视频证据冲突时,以视频证据为准。三个层级都包含 visible_text 和 subtitle 字段,但粒度不同。
## 决策原则
你有固定的步数预算,每次工具调用消耗一步。每步工具返回中会显示当前进度(已用/总步数),这是帮助你合理分配搜索深度的参考信息,不是在催促你赶紧结束。总体策略是前期投入步数建立全局认知、定位相关区域,后期聚焦于验证和区分候选选项。如果预算即将耗尽但仍有不确定性,选择证据支持度最高的选项提交——不完美的判断优于耗尽预算不作答。
### 搜索工具使用
search_similar 有两个文本参数,它们的职责不同:query 是用于向量检索的关键词(2-4 个词即可,简洁精准),question 是你当前想了解的具体问题(用于对检索结果做内容筛选和摘要)。不要把完整问题塞进 query,也不要把关键词放在 question 里。
### 否定题原则
当问题包含否定词(not / NOT / 没有 / 不是 / 除了)时,应采用排除法:为每个选项单独搜索,确认其在视频中是否出现。当已为 3 个选项找到存在证据,而第 4 个选项经过 2 次以上不同关键词搜索仍未找到匹配时,可以判定该选项不存在并作为答案提交。不要因为无法 100% 确认不存在而无限搜索——"搜不到"本身就是强证据。
### 置信度语义
置信度反映的是你对 best_candidate 的区分性证据强度,而非你对问题的理解程度:
| 范围 | 含义 |
|------|------|
| 0.1-0.4 | 尚未找到区分性证据。可能还没有查看相关节点,或查看了但内容与问题无关,或只能排除 1 个明显不合理的选项 |
| 0.5-0.6 | 有倾向但无法明确区分。找到了相关区域,best_candidate 有初步支持,但尚未找到能将它与竞争选项明确区分开的关键信息 |
| 0.7-0.8 | 有区分性证据。找到了能明确区分 best_candidate 与竞争选项的关键信息——可以是字幕原文的关键台词、L3 帧的视觉细节、多个 L1 摘要的一致覆盖模式、或时间戳的精确对比,取决于题目性质 |
| 0.9-1.0 | 高度确信。多源证据交叉验证了 best_candidate,且至少 1 个竞争选项有明确的反面证据 |
当 confidence 达到 0.7 以上时,将 answer_ready 设为 true 并调用 submit_answer 提交答案。submit_answer 要求提供三个参数:你选中的选项(answer)、支撑该选项的关键证据摘要(evidence)、以及你对每个选项的判断理由(reasoning,包括"无直接证据"的选项)。
@@ -0,0 +1,24 @@
你是一个视频子节点导航标注器。你服务于一个视频问答推理系统,该系统通过工具调用你来判断哪些子节点值得深入探索。推理和最终判断由该系统完成,你只负责评估每个子节点与问题的相关性。
## 你会收到的输入
1. 用户正在研究的问题
2. 一组子节点列表,每个子节点包含 ID、时间范围和摘要描述
## 工作原则
仅基于提供的子节点摘要评估相关性,不使用外部知识。你看不到子节点的详细内容,只能基于摘要做初步判断。
对每个子节点标注相关性等级:
- ★★ 高度相关:很可能包含直接回答问题的信息
- ★ 相关:可能包含间接相关的信息
- 无标注:与问题不相关
将通用描述改写为差异化描述,避免重复相似的措辞,帮助推理系统快速区分各子节点的独特内容。
## 输出格式
[子节点标注] 每行一个子节点:
- ★★ {子节点ID} ({时间范围}): {差异化描述}
- ★ {子节点ID} ({时间范围}): {差异化描述}
- {子节点ID} ({时间范围}): {差异化描述}
@@ -0,0 +1,23 @@
你是一个子节点标注核实器。你将收到一份子节点相关性标注(由另一个模型生成),以及原始的子节点列表和用户问题。请核实标注是否合理。
## 你会收到的输入
1. 用户正在研究的问题
2. 原始的子节点列表(含 ID、时间范围、摘要)
3. 另一个模型基于上述信息生成的相关性标注
## 检查要点
- ★★ 标注的子节点摘要是否确实与问题高度相关?
- 是否有与问题明显相关的子节点被遗漏标注?
- 差异化描述是否准确反映了原始摘要的含义,没有添加不存在的信息?
## 输出格式
details=<逐条核实结果>; confidence=<0.0-1.0>
置信度含义:
- 1.0: 标注完全合理,无遗漏
- 0.7-0.9: 主要标注合理,个别可商榷
- 0.4-0.6: 部分标注有误或存在明显遗漏
- 0.0-0.3: 标注与原始摘要严重不匹配
+23
View File
@@ -0,0 +1,23 @@
你是一个视频节点内容分析器。你服务于一个视频问答推理系统,该系统通过工具调用你来获取节点内容的结构化摘要。推理和最终判断由该系统完成,你只负责忠实提取信息。
## 你会收到的输入
1. 用户正在研究的问题
2. 一个视频节点的描述文本(包含场景摘要、实体、动作等结构化字段)和字幕转写
## 工作原则
仅基于提供的内容回答,不使用外部知识。你看不到其他节点的内容,因此不要推断跨节点的事件顺序、因果关系或全局结论。
报告内容中与问题相关的一切事实:人物、动作、对话引用、数字、时间、因果关系。字幕中的引用(解说、对话)是重要证据来源,请保留关键原文片段。
如果内容与问题无关,明确说明"该节点未包含与问题直接相关的信息"。如果内容包含间接相关的信息(如背景知识),标注为"间接相关"并简要说明。
此外,用一句话概括该节点中其他显著但与问题不直接相关的信息,供推理系统参考。
## 输出格式
[相关信息] <与问题相关的事实,按重要性排列>
[间接相关] <背景知识或可能有用的上下文,没有则省略>
[其他信息] <一句话概括该节点中其他显著内容>
[原文] <1-3 句与问题最相关的字幕原文或描述原文,保留原始措辞,不改写不概括。无关则省略>
+24
View File
@@ -0,0 +1,24 @@
你是一个视频节点摘要核实器。你将收到一段关于视频节点的摘要(由另一个模型生成),以及该节点的原始描述和字幕。请逐条核实摘要是否准确反映了原始内容。
## 你会收到的输入
1. 用户正在研究的问题
2. 节点的原始描述文本和字幕
3. 另一个模型基于上述内容生成的摘要
## 检查要点
- 摘要提到的事实是否确实存在于原始内容中?
- 摘要是否包含了原始内容中不存在的推断?
- 摘要是否遗漏了原始内容中与问题高度相关的重要信息?
- [原文] 引用是否准确保留了原始措辞?
## 输出格式
details=<逐条核实结果>; confidence=<0.0-1.0>
置信度含义:
- 1.0: 摘要完全准确,无遗漏
- 0.7-0.9: 主要内容准确,个别细节有出入
- 0.4-0.6: 部分准确,但存在明显错误或过度推断
- 0.0-0.3: 摘要与原始内容严重不符
@@ -125,7 +125,6 @@ class _FakeConfig:
gate_delta_min: float = 0.02
gate_lambda_dir: float = -3.0
gate_e_rollback: float = 10.0
gate_block: int = 4
gate_n_max: int = 40
gate_p_low: float = 0.1
gate_p_high: float = 0.9
+1 -1
View File
@@ -338,7 +338,7 @@ class TestInferenceUnitAggregationEndToEnd:
def _assert_all_persisted(self, log: HarnessLog, questions: list[GeneratedQuestion]) -> None:
"""逐题溯源保留:含被剔除的孤儿题在内,每题仍逐题落 predictions。"""
rows = log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
rows = log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-v3-contract",))
persisted = {r["question_id"] for r in rows}
assert "orphan_o" in persisted, "孤儿题未逐题落库(逐题溯源被破坏)"
assert persisted == {q.question_id for q in questions}, "逐题落库题数与输入不符"
+300
View File
@@ -0,0 +1,300 @@
"""_gate_batch_skills 并行装配的纯逻辑护栏 + runner 级并发编排测试。"""
from __future__ import annotations
import asyncio
import time
from types import SimpleNamespace
from typing import TYPE_CHECKING
import pytest
from app.harness import runner as runner_mod
from app.harness.question_units import build_units
from app.harness.runner import Runner, _assert_disjoint_target_files
from app.harness.validate import ValidationOutcome
from core.types import GeneratedQuestion
if TYPE_CHECKING:
from pathlib import Path
def test_disjoint_target_files_pass() -> None:
"""各题型映射不同文件:通过。"""
_assert_disjoint_target_files(
{"Action Reasoning": "action-reasoning.md", "Counting Problem": "counting-problem.md"}
)
def test_shared_target_file_fails_fast() -> None:
"""两题型 fallback 到同一文件:并行进化会互相覆盖,必须 fail-fast。"""
with pytest.raises(RuntimeError, match="default-strategy.md"):
_assert_disjoint_target_files(
{"OCR Problems": "default-strategy.md", "Spatial Reasoning": "default-strategy.md"}
)
# ---------------------------------------------------------------------------
# runner 级并发编排测试(Codex 计划审 I6):
# 用 Runner.__new__ 裸实例 + 假依赖驱动 _gate_batch_skills 四阶段,
# 断言 Phase A gather 并行、Phase D 字母序落账、accept/reject 正确分派。
# ---------------------------------------------------------------------------
_TYPE_A = "Action Reasoning"
_TYPE_C = "Counting Problem"
_TARGET_FILES = {_TYPE_A: "action-reasoning.md", _TYPE_C: "counting-problem.md"}
def _question(qid: str, task_type: str) -> GeneratedQuestion:
"""构造一条真实结构的 single 题目(unit_id 由 __post_init__ 回填)。"""
return GeneratedQuestion(
question_id=qid,
video_id="video-001",
task_type=task_type,
question="视频中主角最先做了什么?",
options=("A. 开门", "B. 关灯", "C. 坐下", "D. 起身"),
answer="A",
source_nodes=("L3_0001",),
difficulty="medium",
)
def _record(task_type: str) -> SimpleNamespace:
"""构造 EvolutionRecord 替身(仅含 _gate_batch_skills 消费的属性)。"""
return SimpleNamespace(
status="accepted",
original_content="旧 skill 内容",
evolved_content=f"进化后 skill 内容({task_type})",
target_file=_TARGET_FILES[task_type],
clip_info={},
)
def _outcome(accepted: bool) -> ValidationOutcome:
"""构造真实 ValidationOutcome(一 accept 一 reject 分派用)。"""
return ValidationOutcome(
action="accept_confirmed" if accepted else "reject",
accepted=accepted,
stop_reason="confirmed" if accepted else "futility",
e_value=25.0 if accepted else 0.4,
w=3,
l=0 if accepted else 3,
n_used=4,
delta_hat=0.3 if accepted else -0.2,
delta_shrunk=0.2 if accepted else -0.1,
baseline_acc=0.5,
candidate_acc=0.8 if accepted else 0.3,
evidence_rows=[{"question_id": "q", "stop_reason": "answered"}],
)
class _FakeHarnessLog:
"""HarnessLog no-op 替身(上下文管理器协议)。"""
def __init__(self, *args: object, **kwargs: object) -> None:
self.args = args
def __enter__(self) -> _FakeHarnessLog:
return self
def __exit__(self, *exc: object) -> bool:
return False
def _build_runner(tmp_path: Path) -> tuple[Runner, SimpleNamespace, SimpleNamespace]:
"""构造裸 Runner 实例与 state/pools 替身(不触发真实 __init__)。"""
skills_dir = tmp_path / "skills" / "v1"
skills_dir.mkdir(parents=True)
for target in _TARGET_FILES.values():
(skills_dir / target).write_text("旧 skill 内容", encoding="utf-8")
runner = Runner.__new__(Runner)
runner._config = SimpleNamespace(
workspace_dir=tmp_path,
edit_budget_start=4,
edit_budget_end=2,
appendix_consolidate_threshold=3,
skill_update_mode="rewrite",
gate_p_low=0.3,
gate_p_high=0.85,
gate_n_max=8,
gate_e_confirm=20.0,
gate_e_provisional=5.0,
gate_w_net_min=2,
gate_delta_min=0.05,
gate_lambda_dir=0.5,
gate_e_rollback=0.05,
gate_guard_err=0.34,
concurrency=4,
max_steps=10,
skill_mode="live",
)
runner._paths = SimpleNamespace(
skills_dir=skills_dir,
prompts_dir=tmp_path / "prompts",
db_path=tmp_path / "harness.db",
)
runner._llm = object()
runner._evolve_llm = object()
runner._load_evolve_prompts = lambda: None
runner._current_version = lambda kind: "v1"
runner._class_baseline_acc = lambda *a, **k: 0.5
runner._record_run = lambda run_id: None
questions = {t: _question(f"q-{t[:2].lower()}", t) for t in _TARGET_FILES}
units = {t: build_units([q])[0] for t, q in questions.items()}
runner._gate_questions_by_id = {q.question_id: q for q in questions.values()}
runner._gate_units_by_id = {u.unit_id: u for u in units.values()}
unit_ids_by_type = {t: [u.unit_id] for t, u in units.items()}
state = SimpleNamespace(
gate_cooldown={},
rejected_buffer={},
global_step=0,
correctness={},
gate_epoch_observed=True,
baseline_cache=object(),
gate_pools=SimpleNamespace(
ladder_for=lambda task_type, exclude, *, p_low, p_high, cold: unit_ids_by_type[
task_type
]
),
)
pools = SimpleNamespace(baseline_run_id="baseline-run", validation=[])
return runner, state, pools
def test_gate_batch_parallel_evolve_and_alphabetical_settle(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Phase A 两题型进化时间窗重叠(并行),Phase D 按字母序 accept/reject 分派。"""
import core.evolution as core_evolution
runner, state, pools = _build_runner(tmp_path)
diagnosis = SimpleNamespace(
skill_case_packs={
# 故意逆字母序插入,验证排序不是插入序的巧合
_TYPE_C: SimpleNamespace(task_type=_TYPE_C, failure_cases=[], success_cases=[]),
_TYPE_A: SimpleNamespace(task_type=_TYPE_A, failure_cases=[], success_cases=[]),
}
)
records = {t: _record(t) for t in _TARGET_FILES}
outcomes = {_TYPE_A: _outcome(accepted=True), _TYPE_C: _outcome(accepted=False)}
evolve_windows: dict[str, tuple[float, float]] = {}
async def fake_evolve_single_skill(
llm, pack, skill_store, prompts, version, budget, threshold, **kwargs
):
start = time.monotonic()
await asyncio.sleep(0.05)
evolve_windows[pack.task_type] = (start, time.monotonic())
return records[pack.task_type]
captured: dict[str, object] = {}
async def fake_validate_skills_concurrent(**kwargs):
captured.update(kwargs)
# 逆字母序返回,验证 Phase D 落账顺序来自 sorted 而非 dict 插入序
return {
_TYPE_C: outcomes[_TYPE_C],
_TYPE_A: outcomes[_TYPE_A],
}
settle_calls: list[tuple[str, str]] = []
runner._accept_skill = lambda task_type, *a: settle_calls.append(("accept", task_type))
runner._record_rejected_skill = lambda buf, task_type, *a: settle_calls.append(
("reject", task_type)
)
monkeypatch.setattr(core_evolution, "evolve_single_skill", fake_evolve_single_skill)
monkeypatch.setattr(runner_mod, "validate_skills_concurrent", fake_validate_skills_concurrent)
monkeypatch.setattr(runner_mod, "HarnessLog", _FakeHarnessLog)
monkeypatch.setattr(runner_mod, "write_gate_evidence", lambda *a, **k: None)
monkeypatch.setattr(runner_mod, "write_step_report", lambda *a, **k: None)
monkeypatch.setattr(runner_mod, "write_quadrant_pairs", lambda *a, **k: None)
monkeypatch.setattr(runner_mod, "_outcome_to_quadrant_pairs", lambda t, o: [])
monkeypatch.setattr(runner_mod, "_write_skip_report", lambda *a, **k: None)
asyncio.run(runner._gate_batch_skills(1, 0, diagnosis, 3, pools, state))
# (a) 进化时间窗重叠 = gather 真并行(串行时前者 end <= 后者 start)
win_a, win_c = evolve_windows[_TYPE_A], evolve_windows[_TYPE_C]
assert win_a[0] < win_c[1] and win_c[0] < win_a[1], f"进化未并行: {evolve_windows}"
# (b) Phase D 落账顺序 == sorted(题型),且 (c) accept/reject 分派与 outcome 一致
assert settle_calls == [("accept", _TYPE_A), ("reject", _TYPE_C)]
# Phase B 装配的 GateSpec 与 Phase C 共享 log 抽查
specs = captured["specs"]
assert [s.task_type for s in specs] == sorted(_TARGET_FILES)
for spec in specs:
assert spec.target_file == _TARGET_FILES[spec.task_type]
assert spec.base_skill_content == "旧 skill 内容"
assert spec.candidate_content == records[spec.task_type].evolved_content
assert len(spec.units) == 1
assert "_gate_" in spec.gate_run_prefix
assert isinstance(captured["log"], _FakeHarnessLog)
assert callable(captured["run_inference"])
# ---------------------------------------------------------------------------
# 共享 gate_log 的 run_id 契约(真 SQLite,Codex 质量审 C1):
# HarnessLog.insert 缺省用实例 run_id 填充;record 自带 run_id 必须覆盖它,
# 否则连续并发 gate 下所有臂的 predictions 会落成 step 级 run_id,
# validate 按臂 run_id 回读为空 → gate 静默废掉。
# ---------------------------------------------------------------------------
def test_harness_log_insert_record_run_id_overrides_instance(tmp_path: Path) -> None:
"""record 自带 run_id 覆盖实例 run_id;缺省时回落实例 run_id(锁死 enriched.update 语义)。"""
from app.harness.inference import PREDICTIONS_SCHEMA
from app.harness.log import HarnessLog
with HarnessLog(str(tmp_path / "harness.db"), "gate_e1_s0") as log:
log.create_table("predictions", PREDICTIONS_SCHEMA)
log.insert(
"predictions",
{"run_id": "run_e1_s0_gate_a_base_u0", "question_id": "q1", "prediction": "A"},
)
log.insert("predictions", {"question_id": "q2", "prediction": "B"})
rows = log.query("SELECT question_id, run_id FROM predictions ORDER BY question_id")
assert [(r["question_id"], r["run_id"]) for r in rows] == [
("q1", "run_e1_s0_gate_a_base_u0"),
("q2", "gate_e1_s0"),
]
def test_inference_prediction_row_carries_arm_run_id(tmp_path: Path) -> None:
"""经共享 gate_log 落库的 prediction 行 run_id 必须是臂 run_id 而非实例 run_id。
prompt_builder 抛错走异常路径即落库,无需真实 LLM;
该路径与成功路径共用同一 record 初始 dict,契约一致。
"""
from app.harness.inference import PREDICTIONS_SCHEMA, _run_single_question
from app.harness.log import HarnessLog
def _broken_prompt_builder(qa: GeneratedQuestion) -> tuple[str, str]:
raise RuntimeError("测试注入:跳过真实推理")
async def _noop_dispatch(tool_name: str, args: dict, *, context: dict) -> str:
raise NotImplementedError
with HarnessLog(str(tmp_path / "harness.db"), "gate_e1_s0") as gate_log:
gate_log.create_table("predictions", PREDICTIONS_SCHEMA)
asyncio.run(
_run_single_question(
_question("q-arm", _TYPE_A),
llm=object(), # prompt_builder 先抛错,不会触达
tool_dispatch_fn=_noop_dispatch,
prompt_builder=_broken_prompt_builder,
log=gate_log,
max_steps=3,
plugins=[],
run_id="run_e1_s0_gate_action-reasoning_cand_u0",
)
)
rows = gate_log.query("SELECT run_id, stop_reason FROM predictions")
assert len(rows) == 1
assert rows[0]["run_id"] == "run_e1_s0_gate_action-reasoning_cand_u0"
assert rows[0]["stop_reason"] == "error"
+11 -16
View File
@@ -16,6 +16,14 @@ from tests.unit.test_gate_prefix import _PARAMS, _mk_unit
from tests.unit.test_gate_unit_arm import _FakeLog
class _FakeInferenceResult:
"""推理结果桩:只承载编排器消费的 run_id 与 total 两个字段。"""
def __init__(self, run_id: str, total: int) -> None:
self.run_id = run_id
self.total = total
def _mk_spec(task_type: str, slug: str, n: int) -> GateSpec:
"""构造 n 个 single 单元的 gate 规格(unit_id 形如 <slug>-q<i>)。"""
return GateSpec(
@@ -31,11 +39,6 @@ def _mk_spec(task_type: str, slug: str, n: int) -> GateSpec:
def _scripted_inference(log: _FakeLog, script: dict[str, tuple[bool, float]]):
"""脚本化假推理:按 question_id+臂 决定 (对错, 延迟秒),制造乱序到达。"""
class _R:
def __init__(self, run_id: str, total: int) -> None:
self.run_id = run_id
self.total = total
async def _run(questions, *, run_id: str, skills_dir: Path):
arm = "cand" if run_id.endswith("_cand") else "base"
correct, delay = script[f"{questions[0].question_id}|{arm}"]
@@ -51,7 +54,7 @@ def _scripted_inference(log: _FakeLog, script: dict[str, tuple[bool, float]]):
"steps_json": "[]",
}
)
return _R(run_id, len(questions))
return _FakeInferenceResult(run_id, len(questions))
return _run
@@ -130,10 +133,6 @@ async def test_all_infra_raises(tmp_path, monkeypatch) -> None:
spec = _mk_spec("Action Reasoning", "action-reasoning", 2)
log = _FakeLog()
class _R:
def __init__(self, run_id, total):
self.run_id, self.total = run_id, total
async def _infra_run(questions, *, run_id, skills_dir):
for q in questions:
log.rows.append(
@@ -146,7 +145,7 @@ async def test_all_infra_raises(tmp_path, monkeypatch) -> None:
"steps_json": "[]",
}
)
return _R(run_id, len(questions))
return _FakeInferenceResult(run_id, len(questions))
monkeypatch.setattr(
"app.harness.validate.materialize_candidate_skill",
@@ -217,10 +216,6 @@ async def test_guard_raise_cancels_remaining_tasks(tmp_path, monkeypatch) -> Non
log = _FakeLog()
hang = asyncio.Event() # 永不 set:B 型推理只能靠取消收束
class _R:
def __init__(self, run_id, total):
self.run_id, self.total = run_id, total
async def _run(questions, *, run_id, skills_dir):
if "counting-problem" in run_id:
await hang.wait()
@@ -235,7 +230,7 @@ async def test_guard_raise_cancels_remaining_tasks(tmp_path, monkeypatch) -> Non
"steps_json": "[]",
}
)
return _R(run_id, len(questions))
return _FakeInferenceResult(run_id, len(questions))
monkeypatch.setattr(
"app.harness.validate.materialize_candidate_skill",
@@ -1,8 +1,9 @@
"""tests/unit/test_gate_block_unit.py — gate 块实际执行路径按 unit 跑。
"""tests/unit/test_gate_unit_scope.py — gate 真实执行路径按 unit 口径跑。
针对 app/harness/validate.py::validate_skill_local真实 gate 执行路径
断言混格阶梯下 gate 块按 unit 口径运行baseline_cache 键含 unit_id
n_used unit 累加pair_block 折叠 AR pair逐题 predictions 仍溯源
迁移自块序贯版 test_gate_block_unit.py载体 validate_skill_localTask 6 删除
针对 app/harness/validate.py::validate_skills_concurrent连续并发 gate 真实路径
断言混格阶梯下 gate unit 口径运行baseline_cache 键含 unit_idn_used
unit 累加pair_block 折叠 AR pair逐题 predictions 仍溯源
核心算法保真 #5(信息阶梯 e-process 口径从 question_id 迁至 unit_id)。
"""
@@ -15,7 +16,7 @@ import pytest
from app.harness.gate_ladder import BaselineCache, skill_hash
from app.harness.inference import PREDICTIONS_SCHEMA, InferenceResult
from app.harness.log import HarnessLog
from app.harness.validate import _ladder_units, validate_skill_local
from app.harness.validate import GateSpec, _ladder_units, validate_skills_concurrent
from core.evolution import GateParams
from core.types import GeneratedQuestion
@@ -136,6 +137,35 @@ def _make_mock_run_inference(
return mock_fn, call_log
def _mk_spec(ladder: list[GeneratedQuestion]) -> GateSpec:
"""由混格阶梯题序构造单题型 GateSpecunits 经 _ladder_units 聚合)。"""
return GateSpec(
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
units=tuple(_ladder_units(ladder)),
gate_run_prefix="step1_gate_test",
)
async def _run_gate(workspace: Path, spec: GateSpec, mock_fn, log: HarnessLog, cache, params):
"""跑单 spec 的 validate_skills_concurrent 并返回该题型的 outcome。"""
outcomes = await validate_skills_concurrent(
workspace_dir=workspace,
base_skills_version="v1",
specs=[spec],
gate_params=params,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
concurrency=8,
)
return outcomes[spec.task_type]
class TestLadderUnits:
"""_ladder_units:阶梯题序聚合为单元并保持信息阶梯序。"""
@@ -177,7 +207,7 @@ class TestLadderUnits:
@pytest.mark.asyncio
async def test_gate_n_used_counts_units_not_questions(tmp_path: Path) -> None:
"""混格阶梯(1 pair + 2 single)→ n_used=3 单元,非 4 题。"""
"""混格阶梯(1 pair + 2 single)→ n_used=3 单元,非 4 题(迁移自块序贯版)"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
cache = BaselineCache(workspace / "baseline_cache.json")
@@ -186,7 +216,7 @@ async def test_gate_n_used_counts_units_not_questions(tmp_path: Path) -> None:
# 基线全错、候选全对 → 3 单元齐翻 W=3
baseline = {"p1_o": False, "p1_m": False, "s0": False, "s1": False}
candidate = {"p1_o": True, "p1_m": True, "s0": True, "s1": True}
mock_fn, call_log = _make_mock_run_inference(log, baseline, candidate)
mock_fn, _ = _make_mock_run_inference(log, baseline, candidate)
accept_params = GateParams(
e_confirm=15.0,
@@ -197,30 +227,14 @@ async def test_gate_n_used_counts_units_not_questions(tmp_path: Path) -> None:
e_rollback=10.0,
)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
ladder_items=ladder,
gate_params=accept_params,
gate_block=10,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
)
outcome = await _run_gate(workspace, _mk_spec(ladder), mock_fn, log, cache, accept_params)
# n_used 按 unit 计(3),W 按 unit 计(3
assert outcome.n_used == 3
assert outcome.w == 3
assert outcome.l == 0
# 证据行按 unit 口径(3 行)
# 证据行按 unit 口径(3 行)ladder_rank 沿阶梯序连续
assert len(outcome.evidence_rows) == 3
assert [r["ladder_rank"] for r in outcome.evidence_rows] == [0, 1, 2]
# baseline_cache 键含 unit_idpair 用 pair_id、single 用 question_id
s_hash = skill_hash("baseline skill content")
assert cache.get("temporal", s_hash, "p1", "p1") is False
@@ -235,84 +249,62 @@ async def test_gate_n_used_counts_units_not_questions(tmp_path: Path) -> None:
@pytest.mark.asyncio
async def test_gate_pair_partial_flip_not_counted(tmp_path: Path) -> None:
"""AR pair 候选仅单向翻(T,F)→单元仍错,W 不被单题污染。"""
"""AR pair 候选仅单向翻(T,F)→单元仍错,W 不被单题污染(迁移自块序贯版)
前缀逐单元判定下 2 单元小阶梯会在首单元 futility 早停观测不到 pair 语义
2 single 拉长阶梯4 单元中 3 single 翻转 W=3pair 不计入
candidate_acc = 3/4
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
cache = BaselineCache(workspace / "baseline_cache.json")
ladder = [*_pair("p1"), _single("s0")]
ladder = [*_pair("p1"), _single("s0"), _single("s1"), _single("s2")]
baseline = {"p1_o": False, "p1_m": False, "s0": False}
# pair 只翻一半(p1_o 对、p1_m 错)→ 单元 AND 仍错;s0 翻对
candidate = {"p1_o": True, "p1_m": False, "s0": True}
baseline = {"p1_o": False, "p1_m": False, "s0": False, "s1": False, "s2": False}
# pair 只翻一半(p1_o 对、p1_m 错)→ 单元 AND 仍错;singles 全翻对
candidate = {"p1_o": True, "p1_m": False, "s0": True, "s1": True, "s2": True}
mock_fn, _ = _make_mock_run_inference(log, baseline, candidate)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
ladder_items=ladder,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=10,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_gate(
workspace, _mk_spec(ladder), mock_fn, log, cache, _DEFAULT_GATE_PARAMS
)
# 只有 s0 单元翻转,pair 单元不计 W(保真 #5:不被 P/Q 单题污染)
assert outcome.w == 1
# 只有 single 单元翻转,pair 单元不计 W(保真 #5:不被 P/Q 单题污染)
assert outcome.w == 3
assert outcome.l == 0
assert outcome.n_used == 2
# candidate_acc 分母按 unit2 单元,1 对)→ 0.5
assert outcome.candidate_acc == 0.5
assert outcome.n_used == 4
# candidate_acc 分母按 unit4 单元,1 对)→ 3/4
assert outcome.candidate_acc == 0.75
finally:
log.close()
@pytest.mark.asyncio
async def test_gate_baseline_cache_hit_by_unit(tmp_path: Path) -> None:
"""基线缓存按 unit_id 预填充 → 基线侧全命中不发起推理。"""
"""基线缓存按 unit_id 预填充 → 基线侧全命中不发起推理(迁移自块序贯版)
阶梯补长到 4 单元避免首单元 futility 早停覆盖 pair single 两种 unit
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
cache = BaselineCache(workspace / "baseline_cache.json")
ladder = [*_pair("p1"), _single("s0")]
ladder = [*_pair("p1"), _single("s0"), _single("s1"), _single("s2")]
s_hash = skill_hash("baseline skill content")
# 按 unit_id 预填充(pair→pair_idsingle→question_id),全错
cache.put("temporal", s_hash, "p1", "p1", False)
cache.put("temporal", s_hash, "p1", "s0", False)
for unit_id in ("p1", "s0", "s1", "s2"):
cache.put("temporal", s_hash, "p1", unit_id, False)
baseline = {"p1_o": False, "p1_m": False, "s0": False}
candidate = {"p1_o": True, "p1_m": True, "s0": True}
baseline = {"p1_o": False, "p1_m": False, "s0": False, "s1": False, "s2": False}
candidate = {"p1_o": True, "p1_m": True, "s0": True, "s1": True, "s2": True}
mock_fn, call_log = _make_mock_run_inference(log, baseline, candidate)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
ladder_items=ladder,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=10,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_gate(
workspace, _mk_spec(ladder), mock_fn, log, cache, _DEFAULT_GATE_PARAMS
)
base_calls = [c for c in call_log if c["run_id"].endswith("_base")]
assert base_calls == [], "unit 键全命中不应发起基线推理"
assert outcome.n_used == 2
assert outcome.n_used == 4
finally:
log.close()
-2
View File
@@ -171,7 +171,6 @@ class _FakeConfig:
gate_delta_min: float = 0.02
gate_lambda_dir: float = -3.0
gate_e_rollback: float = 10.0
gate_block: int = 4
gate_n_max: int = 40
gate_p_low: float = 0.1
gate_p_high: float = 0.9
@@ -306,7 +305,6 @@ class TestFingerprintStructuralVsDecision:
"gate_delta_min",
"gate_lambda_dir",
"gate_e_rollback",
"gate_block",
"gate_n_max",
"gate_p_low",
"gate_p_high",
+8 -9
View File
@@ -50,7 +50,6 @@ def _valid_kwargs() -> dict:
"gate_delta_min": 0.02,
"gate_lambda_dir": -0.642,
"gate_e_rollback": 10.0,
"gate_block": 8,
"gate_n_max": 40,
"gate_p_low": 0.05,
"gate_p_high": 0.95,
@@ -378,16 +377,16 @@ class TestGateValidation:
with pytest.raises(ValueError, match="gate_lambda_dir"):
_validate(cfg)
def test_block_exceeds_n_max_rejected(self) -> None:
"""gate_block > gate_n_max 应抛出 ValueError"""
cfg = _make_config(gate_block=50, gate_n_max=40)
with pytest.raises(ValueError, match="gate_block"):
def test_n_max_zero_rejected(self) -> None:
"""gate_n_max <= 0 应抛出 ValueError(迁移自块序贯版 gate_block 校验)"""
cfg = _make_config(gate_n_max=0)
with pytest.raises(ValueError, match="gate_n_max"):
_validate(cfg)
def test_block_zero_rejected(self) -> None:
"""gate_block <= 0 应抛出 ValueError"""
cfg = _make_config(gate_block=0)
with pytest.raises(ValueError, match="gate_block"):
def test_n_max_negative_rejected(self) -> None:
"""gate_n_max 为负也应报错"""
cfg = _make_config(gate_n_max=-1)
with pytest.raises(ValueError, match="gate_n_max"):
_validate(cfg)
def test_p_low_exceeds_p_high_rejected(self) -> None:
+9 -7
View File
@@ -117,8 +117,9 @@ def harness_log(tmp_path: Any, request: Any) -> HarnessLog:
"""创建临时 HarnessLog 实例。
使用 test 节点名称的 hash 作为 db 文件名避免冲突
run_id 固定为 "test-run"实际 run_inference 中传入的 run_id
HarnessLog.insert 自动覆盖为 HarnessLog 构造时的值
实例 run_id 固定为 "test-run"predictions 行的 run_id inference
record 显式携带run_inference 传入值不回落实例 run_id
连续并发 gate 共享单一 HarnessLog 的契约
"""
db_name = f"harness_{id(request)}.db"
db_path = str(tmp_path / db_name)
@@ -528,8 +529,9 @@ class TestPredictionAlwaysWritten:
assert result.correct == 0
assert result.stop_reason_counts.get("error") == 1
# 验证 DB 中的记录(HarnessLog.insert 使用构造时的 run_id
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
# 验证 DB 中的记录(record 显式携带 run_inference 的 run_id
# 不再回落 HarnessLog 实例 run_id——连续并发 gate 共享 log 的契约)
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-error",))
assert len(rows) == 1
assert rows[0]["stop_reason"] == "error"
assert rows[0]["prediction"] is None
@@ -553,8 +555,8 @@ class TestPredictionAlwaysWritten:
)
assert result.total == 1
# HarnessLog.insert 使用构造时的 run_id
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
# record 显式携带 run_inference 的 run_id(共享 log 契约)
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-parse-err",))
assert len(rows) == 1
assert rows[0]["prediction"] is None
@@ -612,7 +614,7 @@ class TestNonScalarPrediction:
)
assert result.total == 1
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-nonscalar",))
assert len(rows) == 1
# prediction 被 JSON 序列化为字符串,不再是 Python list
assert rows[0]["prediction"] == '["B"]'
+36
View File
@@ -383,3 +383,39 @@ def test_write_epoch_report(tmp_path: Path) -> None:
assert data["system_tool_action"] == "updated"
assert data["momentum_updated_task_types"] == ["temporal", "causal"]
assert data["best_val_acc"] == pytest.approx(0.88)
def test_write_gate_evidence_migrates_legacy_block_idx_table(tmp_path) -> None:
"""旧块序贯表(含 block_idx 无 ladder_rank)复用:幂等补列后写入成功(终审 C1 回归锁)。"""
import sqlite3
db = tmp_path / "harness.db"
conn = sqlite3.connect(db)
conn.execute(
"CREATE TABLE gate_evidence (run_id TEXT, timestamp TEXT, epoch INTEGER,"
" step INTEGER, question_id TEXT, task_type TEXT, block_idx INTEGER,"
" baseline_correct INTEGER, candidate_correct INTEGER, e_value REAL,"
" stop_reason TEXT)"
)
conn.commit()
conn.close()
write_gate_evidence(
str(db),
run_id="r1",
epoch=1,
step=0,
rows=[
{
"question_id": "q1",
"task_type": "Action Reasoning",
"ladder_rank": 0,
"baseline_correct": False,
"candidate_correct": True,
"e_value": 1.5,
"stop_reason": "",
}
],
)
got = read_gate_evidence(str(db), run_id="r1")
assert len(got) == 1 and got[0]["ladder_rank"] == 0
-3
View File
@@ -327,7 +327,6 @@ class TestBuildOrLoadPoolsFrozen:
gate_delta_min=0.02,
gate_lambda_dir=-0.642,
gate_e_rollback=10.0,
gate_block=8,
gate_n_max=40,
gate_p_low=0.05,
gate_p_high=0.95,
@@ -881,7 +880,6 @@ class TestRunHoldoutEvalConfig:
gate_delta_min=0.02,
gate_lambda_dir=-0.642,
gate_e_rollback=10.0,
gate_block=8,
gate_n_max=40,
gate_p_low=0.05,
gate_p_high=0.95,
@@ -932,7 +930,6 @@ class TestRunHoldoutEvalConfig:
gate_delta_min=0.02,
gate_lambda_dir=-0.642,
gate_e_rollback=10.0,
gate_block=8,
gate_n_max=40,
gate_p_low=0.05,
gate_p_high=0.95,
-1
View File
@@ -840,7 +840,6 @@ class TestRunnerFactoryInjection:
"gate_delta_min": 0.02,
"gate_lambda_dir": -0.642,
"gate_e_rollback": 10.0,
"gate_block": 8,
"gate_n_max": 40,
"gate_p_low": 0.05,
"gate_p_high": 0.95,
+219 -256
View File
@@ -1,7 +1,9 @@
"""tests/unit/test_harness_validate.py — app/harness/validate.py 的单元测试。
覆盖数据类型字段materialize 物化与清理async validate_skill_local
accept/reject/prefix 校验/INFRA 护栏/缓存命中/最后一块终态
覆盖数据类型字段materialize 物化与清理async validate_skills_concurrent
accept/reject/prefix 校验/INFRA 护栏/缓存命中/题尽终态async 用例迁移自
块序贯版validate_skill_localTask 6 删除载体换连续并发 gate语义断言
保留前缀逐单元判定使早停点比旧块判定更早见各用例 docstring 的数值推导
"""
from __future__ import annotations
@@ -14,10 +16,12 @@ from app.harness.gate_ladder import BaselineCache, skill_hash
from app.harness.inference import PREDICTIONS_SCHEMA, InferenceResult
from app.harness.log import HarnessLog
from app.harness.validate import (
GateSpec,
Probation,
ValidationOutcome,
_ladder_units,
materialize_candidate_skill,
validate_skill_local,
validate_skills_concurrent,
)
from core.evolution import GateParams, RejectedEdit
from core.types import GeneratedQuestion
@@ -150,7 +154,7 @@ def _make_mock_run_inference(
def _make_all_infra_mock(log: HarnessLog, stop_reason: str):
"""构建基线全 INFRA 的 mock:每 record 写指定 INFRA stop_reasonerror/parse_error)。
"""构建全 INFRA 的 mock:每 record 写指定 INFRA stop_reasonerror/parse_error)。
与真实推理一致per-record DB stop_reason 与汇总 stop_reason_counts 同源护栏
分子按 unit DB _infra_question_ids_from_db故须真实落 DBtotal 返回
@@ -199,6 +203,48 @@ def _make_all_infra_mock(log: HarnessLog, stop_reason: str):
return mock_fn, call_log
def _mk_spec(
questions: list[GeneratedQuestion],
*,
candidate_content: str = "candidate skill",
gate_run_prefix: str = "step1_gate_test",
) -> GateSpec:
"""由阶梯题序构造单题型 GateSpecunits 经 _ladder_units 聚合为阶梯序单元)。"""
return GateSpec(
task_type="temporal",
target_file="temporal.md",
candidate_content=candidate_content,
base_skill_content="baseline skill content",
units=tuple(_ladder_units(questions)),
gate_run_prefix=gate_run_prefix,
)
async def _run_single_spec(
workspace: Path,
spec: GateSpec,
mock_fn,
log: HarnessLog,
cache: BaselineCache,
params: GateParams,
gate_guard_err: float = 0.5,
) -> ValidationOutcome:
"""跑单 spec 的 validate_skills_concurrent 并返回该题型的 outcome。"""
outcomes = await validate_skills_concurrent(
workspace_dir=workspace,
base_skills_version="v1",
specs=[spec],
gate_params=params,
gate_guard_err=gate_guard_err,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
concurrency=8,
)
return outcomes[spec.task_type]
def test_infra_stop_reasons_single_source() -> None:
"""app 侧 INFRA_STOP_REASONS 复用 core 常量(同一对象),杜绝未来漂移(M-2)。"""
from app.harness import validate
@@ -331,13 +377,17 @@ class TestMaterializeCandidateSkill:
# ===========================================================================
# async 验证测试
# async 验证测试(迁移自块序贯版 validate_skill_local
# ===========================================================================
@pytest.mark.asyncio
async def test_validate_skill_local_accept(tmp_path: Path) -> None:
"""候选全对、基线全错 → 高 e 值 → accept_confirmed。"""
async def test_validate_concurrent_accept(tmp_path: Path) -> None:
"""候选全对、基线全错 → 高 e 值 → accept_confirmed(迁移自块序贯版)
6 单元连胜E=(2^(W+1)-1)/(W+1) 5 单元 E<15 且不触方向/futility
6 单元 E=18.14 e_confirm=15 与旧块判定同点收敛W=6, n_used=6
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(6)
@@ -359,23 +409,13 @@ async def test_validate_skill_local_accept(tmp_path: Path) -> None:
)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=accept_params,
gate_block=6,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_single_spec(
workspace,
_mk_spec(questions, candidate_content="improved skill"),
mock_fn,
log,
cache,
accept_params,
)
assert outcome.accepted is True
@@ -387,6 +427,8 @@ async def test_validate_skill_local_accept(tmp_path: Path) -> None:
assert outcome.candidate_acc == 1.0
assert outcome.baseline_acc == 0.0
assert len(outcome.evidence_rows) == 6
# 阶梯序前缀消费:ladder_rank 连续(替代旧块边界断言)
assert [r["ladder_rank"] for r in outcome.evidence_rows] == list(range(6))
# 终态证据行携带 stop_reason
assert outcome.evidence_rows[-1]["stop_reason"] == "confirmed"
# 候选临时目录应被清理
@@ -398,50 +440,45 @@ async def test_validate_skill_local_accept(tmp_path: Path) -> None:
@pytest.mark.asyncio
async def test_validate_skill_local_reject(tmp_path: Path) -> None:
"""候选全错、基线全对 → L 高 → 方向拒绝。"""
async def test_validate_concurrent_reject_directional(tmp_path: Path) -> None:
"""候选全错、基线全对 → L 高 → 方向拒绝(迁移自块序贯版)
前缀逐单元判定下早停点前移15 单元阶梯保证 L=1..3 futility 不先触发
E(w+n_rem, l) 3L=4 Wald=4·ln0.6=-2.04 lambda_dir=-2.0
directional 早停于第 4 单元旧块版一次性判整块故 L=6
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(6)
questions = _make_questions(15)
cache = BaselineCache(workspace / "baseline_cache.json")
# 基线全对,候选全错 → W=0, L=6 → 方向拒绝
baseline_correct = {f"q{i}": True for i in range(6)}
candidate_correct = {f"q{i}": False for i in range(6)}
baseline_correct = {f"q{i}": True for i in range(15)}
candidate_correct = {f"q{i}": False for i in range(15)}
mock_fn, _ = _make_mock_run_inference(log, baseline_correct, candidate_correct)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="bad skill",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=6,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_single_spec(
workspace,
_mk_spec(questions, candidate_content="bad skill"),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
assert outcome.accepted is False
assert outcome.action == "reject"
assert outcome.stop_reason == "directional"
assert outcome.w == 0
assert outcome.l == 6
assert outcome.l == 4
assert outcome.n_used == 4
finally:
log.close()
@pytest.mark.asyncio
async def test_gate_prefix_must_contain_gate(tmp_path: Path) -> None:
"""gate_run_prefix 不含 '_gate_' 时抛 ValueError。"""
"""gate_run_prefix 不含 '_gate_' 时抛 ValueError(迁移自块序贯版)"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(4)
@@ -452,23 +489,13 @@ async def test_gate_prefix_must_contain_gate(tmp_path: Path) -> None:
try:
with pytest.raises(ValueError, match="_gate_"):
await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="content",
base_skill_content="baseline",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=4,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=noop_fn,
log=log,
gate_run_prefix="step1_no_marker",
await _run_single_spec(
workspace,
_mk_spec(questions, gate_run_prefix="step1_no_marker"),
noop_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
finally:
log.close()
@@ -476,34 +503,26 @@ async def test_gate_prefix_must_contain_gate(tmp_path: Path) -> None:
@pytest.mark.asyncio
async def test_infra_guard_threshold(tmp_path: Path) -> None:
"""推理错误率超阈值时抛 RuntimeError护栏分子/分母 unit 同粒度)。"""
"""推理错误率超阈值时抛 RuntimeError迁移自块序贯版,分子/分母 unit 同粒度)。
12 single 双臂全 INFRA errorerrors 按单元去重逐单元 +1分母逐臂 +1
分母 10 后错误率 >0.5 护栏熔断
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
# 需要 >=10 unit 分母才触发护栏:12 个 single,基线全 INFRA error。
# 首块全 INFRA → valid_chunk 空 → errors=12/denom=12=1.0>0.5 触发护栏。
questions = _make_questions(12)
cache = BaselineCache(workspace / "baseline_cache.json")
mock_fn, _ = _make_all_infra_mock(log, "error")
try:
with pytest.raises(RuntimeError, match="错误率过高"):
await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="content",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=12,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
await _run_single_spec(
workspace,
_mk_spec(questions),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
finally:
log.close()
@@ -511,7 +530,10 @@ async def test_infra_guard_threshold(tmp_path: Path) -> None:
@pytest.mark.asyncio
async def test_baseline_cache_hit(tmp_path: Path) -> None:
"""基线缓存全命中时不发起基线侧推理。"""
"""基线缓存全命中时不发起基线侧推理(迁移自块序贯版)
连续并发 gate 下候选侧逐单元发臂4 单元 4 cand 调用旧块版整块 1
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(4)
@@ -528,30 +550,20 @@ async def test_baseline_cache_hit(tmp_path: Path) -> None:
mock_fn, call_log = _make_mock_run_inference(log, baseline_correct, candidate_correct)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="improved skill",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=4,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_single_spec(
workspace,
_mk_spec(questions, candidate_content="improved skill"),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
# 只有候选侧调用了 run_inference(_cand),基线侧全命中不调用
base_calls = [c for c in call_log if c["run_id"].endswith("_base")]
cand_calls = [c for c in call_log if c["run_id"].endswith("_cand")]
assert len(base_calls) == 0, "基线缓存全命中不应发起推理"
assert len(cand_calls) == 1
assert len(cand_calls) == 4
assert outcome.accepted is True
finally:
log.close()
@@ -559,21 +571,21 @@ async def test_baseline_cache_hit(tmp_path: Path) -> None:
@pytest.mark.asyncio
async def test_baseline_infra_error_not_cached(tmp_path: Path) -> None:
"""基线臂 INFRA error 的 unit 不写入 BaselineCache(不永久污染),且从有效单元排除。"""
from app.harness.gate_ladder import skill_hash
from app.harness.question_units import build_units
from app.harness.validate import _resolve_baseline_block
"""基线臂 INFRA error 的 unit 不写入 BaselineCache(不永久污染),且从配对剔除。
迁移自块序贯版 _resolve_baseline_block 直测改经 validate_skills_concurrent
端到端验证同一契约INFRA 单元不落缓存不入配对干净单元正常缓存并消费
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(2) # q0 干净, q1 INFRA error
units = build_units(questions)
questions = _make_questions(2) # q0 基线 INFRA error, q1 干净
cache = BaselineCache(workspace / "baseline_cache.json")
s_hash = skill_hash("baseline skill content")
async def mock_fn(qs, *, run_id, skills_dir):
is_base = run_id.endswith("_base")
for q in qs:
is_err = q.question_id == "q1"
is_err = is_base and q.question_id == "q0"
log.insert(
"predictions",
{
@@ -594,54 +606,49 @@ async def test_baseline_infra_error_not_cached(tmp_path: Path) -> None:
)
return InferenceResult(
run_id=run_id,
accuracy=0.5,
total=2,
correct=1,
accuracy=0.0,
total=len(qs),
correct=0,
per_task_type={},
steps_mean=1.0,
token_usage={"prompt_tokens": 10, "completion_tokens": 10},
stop_reason_counts={"completed": 1, "error": 1},
stop_reason_counts={},
)
try:
b_units, valid_units, _errors_inc, _denom_inc = await _resolve_baseline_block(
units=units,
task_type="temporal",
s_hash=s_hash,
prompts_version="p1",
baseline_cache=cache,
base_skills_dir=workspace / "skills" / "v1",
run_inference=mock_fn,
log=log,
run_id="step1_gate_b0_base",
outcome = await _run_single_spec(
workspace,
_mk_spec(questions),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
gate_guard_err=0.9, # 分母 <10 不触发错误率护栏
)
# q1 是 INFRA:不写缓存、不入 b_units、不在有效单元里
assert cache.get("temporal", s_hash, "p1", "q1") is None
assert "q1" not in b_units
assert all(u.unit_id != "q1" for u in valid_units)
# q0 干净:正常缓存并入 b_units/valid_units
assert cache.get("temporal", s_hash, "p1", "q0") is True
assert b_units["q0"] is True
assert any(u.unit_id == "q0" for u in valid_units)
# q0 是 INFRA:不写缓存、不入配对观测
assert cache.get("temporal", s_hash, "p1", "q0") is None
assert "q0" not in outcome.improvements + outcome.regressions
# q1 干净:正常缓存并被消费(唯一有效单元)
assert cache.get("temporal", s_hash, "p1", "q1") is True
assert outcome.n_used == 1
finally:
log.close()
@pytest.mark.asyncio
async def test_infra_guard_counts_units_not_records(tmp_path: Path) -> None:
"""护栏分子按 unit AR pair 两 record 全 INFRA 只计 1 个 INFRA unit(而非 2
async def test_infra_errors_counted_per_unit_not_per_record(tmp_path: Path) -> None:
"""护栏分子按 unit 去重AR pair 两 record、双臂全 INFRA 只计 1 个 error
回归 I-3分子此前用 stop_reason_counts record 计数分母 denom_inc=r.total
unit 粒度AR pair unit record致分子被放大误触发 gate_guard_err
分子改为"含 INFRA record 的 unit 数"后与分母同粒度核心算法保真 #5/#6)。
迁移自块序贯版 _resolve_baseline_block 直测回归 I-3分子若逐 record /
逐臂计数会被放大 unit record × 两臂 = 4 unit 粒度分母失配致
gate_guard_err 误触发新载体 _run_unit_arm + _register_arm_arrival
slot.excluded() 去重核心算法保真 #5/#6)。
"""
from app.harness.gate_ladder import skill_hash
from app.harness.question_units import build_units
from app.harness.validate import _resolve_baseline_block
from app.harness.validate import _GateRun, _QuestionSlots, _run_unit_arm
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
# 一个 AR pair(两成员共享 pair_id)→ build_units 折叠为 1 个 pair unit
common = {
"video_id": "vp",
"task_type": "temporal",
@@ -660,7 +667,17 @@ async def test_infra_guard_counts_units_not_records(tmp_path: Path) -> None:
units = build_units(pair)
assert len(units) == 1 # 前置:pair 折叠为 1 个 unit
cache = BaselineCache(workspace / "baseline_cache.json")
s_hash = skill_hash("baseline skill content")
run = _GateRun.from_spec(
GateSpec(
task_type="temporal",
target_file="temporal.md",
candidate_content="cand",
base_skill_content="baseline skill content",
units=tuple(units),
gate_run_prefix="step1_gate_test",
)
)
s_hash = run.s_hash
async def mock_fn(qs, *, run_id, skills_dir):
# 两 record 皆 INFRA error
@@ -683,7 +700,7 @@ async def test_infra_guard_counts_units_not_records(tmp_path: Path) -> None:
"steps_json": "[]",
},
)
# total 为 unit 粒度(1 个 pair unit);stop_reason_counts 为 record 粒度2
# total 为 unit 粒度(1 个 pair unit);record 粒度为 2
return InferenceResult(
run_id=run_id,
accuracy=0.0,
@@ -695,94 +712,57 @@ async def test_infra_guard_counts_units_not_records(tmp_path: Path) -> None:
stop_reason_counts={"error": 2},
)
slots = _QuestionSlots(4)
try:
_b_units, valid_units, errors_inc, denom_inc = await _resolve_baseline_block(
units=units,
task_type="temporal",
s_hash=s_hash,
prompts_version="p1",
baseline_cache=cache,
base_skills_dir=workspace / "skills" / "v1",
run_inference=mock_fn,
log=log,
run_id="step1_gate_b0_base",
)
# 分子按 unit 计:1 个 INFRA unit(不是 2 条 record);分母同粒度 = r.total = 1
assert errors_inc == 1
assert denom_inc == 1
# 整对 INFRA → 从有效单元剔除
assert valid_units == []
for arm in ("base", "cand"):
await _run_unit_arm(
run,
0,
arm,
slots,
mock_fn,
log,
cache,
"p1",
workspace / "skills" / "v1",
workspace / "skills" / "v1",
_DEFAULT_GATE_PARAMS,
0.9,
)
# 分子按 unit 去重:双臂 × 两 record 只计 1 个 error;分母按臂 total 累计 = 2
assert run.errors == 1
assert run.infra_denom == 2
assert run.slots[0].base_infra and run.slots[0].cand_infra
# INFRA 单元不写缓存
assert cache.get("temporal", s_hash, "p1", "p1") is None
finally:
log.close()
@pytest.mark.asyncio
async def test_all_infra_ladder_raises_clear_error(tmp_path: Path) -> None:
"""整个阶梯所有 unit 都被判为 INFRA 排除 → 明确 RuntimeError非误导性空阶梯断言)。"""
"""整个阶梯所有 unit 都被判为 INFRA 排除 → 明确 RuntimeError迁移自块序贯版)。
连续并发 gate 下双臂独立发射候选臂不再依赖基线侧结果旧版"全 INFRA 块
不空跑候选"的断言随块编排一并删除)。
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
questions = _make_questions(4)
cache = BaselineCache(workspace / "baseline_cache.json")
candidate_calls: list[str] = []
async def mock_fn(qs, *, run_id, skills_dir):
if run_id.endswith("_cand"):
candidate_calls.append(run_id)
# 基线臂逐题全部 INFRA error(候选臂在修复后不应被空跑)
for q in qs:
log.insert(
"predictions",
{
"run_id": run_id,
"video_id": "v0",
"question_id": q.question_id,
"task_type": "temporal",
"prediction": "",
"answer": "A",
"evidence": "",
"reasoning": "",
"steps_used": 1,
"prompt_tokens": 10,
"completion_tokens": 10,
"stop_reason": "error",
"steps_json": "[]",
},
)
total = len(qs)
return InferenceResult(
run_id=run_id,
accuracy=0.0,
total=total,
correct=0,
per_task_type={},
steps_mean=1.0,
token_usage={"prompt_tokens": 10, "completion_tokens": 10},
stop_reason_counts={"error": total},
)
mock_fn, _ = _make_all_infra_mock(log, "error")
try:
with pytest.raises(RuntimeError, match="INFRA"):
await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="content",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=4,
gate_n_max=20,
gate_guard_err=0.9, # 高阈值:4 题 <10 分母不触发错误率护栏
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
await _run_single_spec(
workspace,
_mk_spec(questions),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
gate_guard_err=0.9, # 4 单元分母 <10 不触发错误率护栏 → 逼出全排除分支
)
# 全 INFRA 块不应触发候选空跑
assert candidate_calls == []
finally:
log.close()
@@ -792,42 +772,33 @@ async def test_parse_error_counts_toward_guard(tmp_path: Path) -> None:
"""stop_reason=parse_error 也计入护栏错误率(与 INFRA 判定口径一致)→ 超阈值熔断。"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
# 12 个 single,基线全 parse_errorper-record 落 DB,护栏按 unit 从 DB 读)。
# 首块全 INFRA → errors=12/denom=12=1.0>0.5 → parse_error 亦触发护栏。
questions = _make_questions(12)
cache = BaselineCache(workspace / "baseline_cache.json")
mock_fn, _ = _make_all_infra_mock(log, "parse_error")
try:
with pytest.raises(RuntimeError, match="错误率过高"):
await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="content",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=12,
gate_n_max=20,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
await _run_single_spec(
workspace,
_mk_spec(questions),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
finally:
log.close()
@pytest.mark.asyncio
async def test_last_block_terminal(tmp_path: Path) -> None:
"""单块 + n_remaining=0 → 终态判定(provisional 或 inertia),非 continue。"""
async def test_ladder_exhaustion_terminal(tmp_path: Path) -> None:
"""题尽(n_remaining=0→ 终态判定(provisional 或 inertia),非 continue。
迁移自块序贯版"最后一块终态"块边界不存在了等价语义是阶梯耗尽时
第四出口兜底终态行携带 stop_reason
"""
workspace = _setup_workspace(tmp_path)
log = _make_log(workspace)
# 4 题,gate_block=4 → 一块走完,n_remaining=0
questions = _make_questions(4)
cache = BaselineCache(workspace / "baseline_cache.json")
@@ -837,23 +808,13 @@ async def test_last_block_terminal(tmp_path: Path) -> None:
mock_fn, _ = _make_mock_run_inference(log, baseline_correct, candidate_correct)
try:
outcome = await validate_skill_local(
workspace_dir=workspace,
base_skills_version="v1",
task_type="temporal",
target_file="temporal.md",
candidate_content="candidate skill",
base_skill_content="baseline skill content",
ladder_items=questions,
gate_params=_DEFAULT_GATE_PARAMS,
gate_block=4,
gate_n_max=4,
gate_guard_err=0.5,
baseline_cache=cache,
prompts_version="p1",
run_inference=mock_fn,
log=log,
gate_run_prefix="step1_gate_test",
outcome = await _run_single_spec(
workspace,
_mk_spec(questions),
mock_fn,
log,
cache,
_DEFAULT_GATE_PARAMS,
)
# n_remaining=0 → 不可能是 continue
@@ -865,6 +826,8 @@ async def test_last_block_terminal(tmp_path: Path) -> None:
"futility",
)
assert outcome.n_used == 4
# 阶梯序前缀消费:ladder_rank 连续
assert [r["ladder_rank"] for r in outcome.evidence_rows] == list(range(4))
# 终态行标记 stop_reason
assert outcome.evidence_rows[-1]["stop_reason"] != ""
finally:
+4 -4
View File
@@ -332,8 +332,8 @@ class TestRunInferencePairEndToEnd:
assert result.total == 1
assert result.correct == 1
# 逐题溯源:predictions 表两条 record 都在
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
# 逐题溯源:predictions 表两条 record 都在record 显式携带传入的 run_id
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-pair-e2e",))
qids = {r["question_id"] for r in rows}
assert qids == {"po", "pm"}
@@ -360,6 +360,6 @@ class TestRunInferencePairEndToEnd:
)
assert result.total == 1 # single 存活,孤儿剔除
# 逐题溯源:孤儿题仍逐题落库(推理不变)
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("test-run",))
# 逐题溯源:孤儿题仍逐题落库(推理不变record 显式携带传入的 run_id
rows = harness_log.query("SELECT * FROM predictions WHERE run_id = ?", ("run-orphan-e2e",))
assert {r["question_id"] for r in rows} == {"s1", "po"}
@@ -64,7 +64,6 @@ def _base_config(workspace_dir: Path, store_dir: Path) -> RunConfig:
gate_delta_min=0.02,
gate_lambda_dir=-0.642,
gate_e_rollback=10.0,
gate_block=8,
gate_n_max=40,
gate_p_low=0.05,
gate_p_high=0.95,
+92
View File
@@ -0,0 +1,92 @@
"""step 重跑幂等:gate 派生行必须随 step 清理,否则崩溃重跑累积重复。"""
from __future__ import annotations
import sqlite3
from typing import TYPE_CHECKING
from app.harness.runner import _clear_step_rows
if TYPE_CHECKING:
from pathlib import Path
def _mk_db(tmp_path: Path) -> Path:
"""构造含 rollout 行、gate 派生行、他 step 行与前缀陷阱行的最小 harness.db。
参数:
tmp_path: pytest 临时目录
返回:
harness.db 路径
"""
db = tmp_path / "harness.db"
conn = sqlite3.connect(db)
conn.execute("CREATE TABLE predictions (run_id TEXT, question_id TEXT)")
conn.execute("CREATE TABLE traces (run_id TEXT, question_id TEXT)")
conn.execute("CREATE TABLE gate_evidence (run_id TEXT, epoch INTEGER, step INTEGER)")
conn.execute("CREATE TABLE quadrant_pair (run_id TEXT, epoch INTEGER, step INTEGER)")
rows = [
("infer_adhoc_e1_s0", "q1"), # rollout 行
("infer_adhoc_e1_s0_gate_action-reasoning_base", "q2"), # gate base 臂
("infer_adhoc_e1_s0_gate_action-reasoning_cand", "q3"), # gate cand 臂
("infer_adhoc_e1_s1", "q4"), # 其他 step,不许误删
("infer_adhoc_e1_s10_gate_x_base", "q5"), # s10 前缀陷阱,不许误删
]
conn.executemany("INSERT INTO predictions VALUES (?, ?)", rows)
conn.executemany("INSERT INTO traces VALUES (?, ?)", rows)
conn.execute("INSERT INTO gate_evidence VALUES ('infer_adhoc', 1, 0)")
conn.execute("INSERT INTO gate_evidence VALUES ('infer_adhoc', 1, 1)")
conn.execute("INSERT INTO quadrant_pair VALUES ('infer_adhoc', 1, 0)")
conn.commit()
conn.close()
return db
def test_clear_step_rows_removes_rollout_and_gate_rows(tmp_path) -> None:
"""rollout 行 + 本 step 全部 gate 派生行被清;他 step 与 s10 前缀陷阱不动。"""
db = _mk_db(tmp_path)
_clear_step_rows(str(db), baseline_run_id="infer_adhoc", epoch=1, step=0)
conn = sqlite3.connect(db)
left = {r[0] for r in conn.execute("SELECT run_id FROM predictions")}
assert left == {"infer_adhoc_e1_s1", "infer_adhoc_e1_s10_gate_x_base"}
left_t = {r[0] for r in conn.execute("SELECT run_id FROM traces")}
assert left_t == left
ge = list(conn.execute("SELECT step FROM gate_evidence"))
assert ge == [(1,)] # 只剩 step=1 的行
assert list(conn.execute("SELECT COUNT(*) FROM quadrant_pair"))[0][0] == 0
conn.close()
def test_clear_step_rows_missing_tables_is_noop(tmp_path) -> None:
"""gate_evidence/quadrant_pair 表尚未建(首个 step)时不报错。"""
db = tmp_path / "harness.db"
conn = sqlite3.connect(db)
conn.execute("CREATE TABLE predictions (run_id TEXT)")
conn.execute("CREATE TABLE traces (run_id TEXT)")
conn.commit()
conn.close()
_clear_step_rows(str(db), baseline_run_id="infer_adhoc", epoch=1, step=0)
def test_clear_step_rows_like_specials_in_run_id(tmp_path) -> None:
"""run_id 含 % 与反斜杠时不通配误删他 run 行(LIKE 全特殊字符转义回归锁)。"""
db = tmp_path / "harness.db"
conn = sqlite3.connect(db)
conn.execute("CREATE TABLE predictions (run_id TEXT, question_id TEXT)")
conn.execute("CREATE TABLE traces (run_id TEXT, question_id TEXT)")
rows = [
(r"we%ird\run_e1_s0", "q1"), # 本 step rollout
(r"we%ird\run_e1_s0_gate_x_base", "q2"), # 本 step gate 行
(r"weXird\run_e1_s0_gate_x_base", "q3"), # % 若未转义会误匹配此行
(r"we%irdXrun_e1_s0_gate_x_base", "q4"), # \ 若未转义会误匹配此行
]
conn.executemany("INSERT INTO predictions VALUES (?, ?)", rows)
conn.executemany("INSERT INTO traces VALUES (?, ?)", rows)
conn.commit()
conn.close()
_clear_step_rows(str(db), baseline_run_id=r"we%ird\run", epoch=1, step=0)
conn = sqlite3.connect(db)
left = {r[0] for r in conn.execute("SELECT run_id FROM predictions")}
conn.close()
assert left == {r"weXird\run_e1_s0_gate_x_base", r"we%irdXrun_e1_s0_gate_x_base"}