Compare commits

...

399 Commits

Author SHA1 Message Date
iomgaa 3dcc7d9899 fix: use SLURM_SUBMIT_DIR to locate repo root under sbatch 2026-07-17 06:13:28 -04:00
iomgaa 9bb09a44d4 feat: add Slurm gpu-sbatch training entry for 4090-server 2026-07-17 06:02:39 -04:00
iomgaa b0be1f1ae5 docs: gate speedup final review and delivery record 2026-07-17 05:38:44 -04:00
iomgaa 172b7a8ec4 merge: continuous concurrent gate speedup (algo #6 semantic revision, design v3) 2026-07-17 05:29:42 -04:00
iomgaa eb12006d38 fix: idempotent ladder_rank migration for legacy gate_evidence tables 2026-07-17 05:29:20 -04:00
iomgaa 1930ad32a4 chore: raise Redis cache TTL to 7 days 2026-07-17 04:52:34 -04:00
iomgaa 8958eee11b refactor: remove block-sequential gate path and gate_block knob (algo #6)
config/train_videomme.yaml 同时收录待入库的实验配置变更(run_id v2 /
concurrency 32 / batch_size 40)。tests/integration/test_v3_contract_e2e.py
的 run_id 断言按 Task 5 显式契约同步修正(原断言依赖旧隐式实例注入)。
2026-07-17 04:40:30 -04:00
iomgaa 0b839937df fix: predictions row carries arm run_id under shared gate_log; drain evolve gather on failure (algo #6) 2026-07-17 03:52:19 -04:00
iomgaa 23a64042fe feat: parallel evolve + continuous gate wiring in runner (algo #6) 2026-07-17 03:32:14 -04:00
iomgaa b3aba7c31d fix: escape all LIKE specials in step-row cleanup 2026-07-17 01:17:31 -04:00
iomgaa 16993ed362 style: consolidate inference stub, complete docstrings (algo #6) 2026-07-17 01:13:19 -04:00
iomgaa ea6bec5421 fix: clear gate-derived rows on step rerun (idempotency) 2026-07-17 00:57:53 -04:00
iomgaa 9e8a254fbb fix: gate orchestrator materialize leak + cancel-drain on abort (algo #6)
Codex 质量审 2 Critical:
- C001: 候选物化移入 try、成功一个登记一个,第 N 个题型物化失败时
  finally 仍清理前 N-1 个已建目录,不泄漏
- C002: gather 首异常(护栏 raise)后显式取消其余任务并排水,确保
  finally 删除候选目录时无在飞任务访问、事件循环无 pending task 警告;
  护栏中止整轮语义不变
- 新增 2 测试:部分物化失败清理 / 护栏 raise 取消收束不悬挂(wait_for 5s)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 00:45:47 -04:00
iomgaa e8b66f85ab feat: continuous concurrent gate orchestrator (algo #6)
- validate_skills_concurrent: 多题型全部臂共享题槽并发编排,发射序 =
  题型 round-robin × 阶梯序(base 先 cand 后),终态统一组装 outcome,
  verdict None(全 INFRA)保留 RuntimeError 语义
- gate_evidence 列 block_idx → ladder_rank(阶梯序号,0-based);旧块路径
  _build_evidence_rows 仅键名同步(值仍为块号)保持落库兼容
- 新增 3 项编排测试:乱序到达前缀有序性/双题型隔离/全 INFRA raise

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 00:36:01 -04:00
iomgaa c61a6dac84 docs: soften slot fairness claim, complete docstrings (algo #6) 2026-07-17 00:28:25 -04:00
iomgaa 30c1cf10c0 fix: gate slot cancel-safety + post-inference freeze discard (algo #6)
Codex 质量审 4 项:推理后二次冻结检查(τ 后 in-flight 结果整体丢弃)、
acquire 取消回滚(半持有 permit 自动归还)、BoundedSemaphore 防静默扩容、
补取消恢复与冻结丢弃两个回归测试。
2026-07-17 00:25:18 -04:00
iomgaa 232afd525b feat: gate unit-arm tasks with question-slot gate (algo #6) 2026-07-17 00:13:51 -04:00
iomgaa 0a8e1ad18b docs: complete four-part docstrings for gate prefix section (algo #6) 2026-07-17 00:06:35 -04:00
iomgaa 2ab8088c81 test: strengthen frozen-noop full-state assertion (algo #6) 2026-07-17 00:03:36 -04:00
iomgaa 1e92928d4e fix: lock all-INFRA contract, idempotency tests, tuple units (algo #6) 2026-07-17 00:00:56 -04:00
iomgaa 21c360bc87 feat: gate prefix-ordered consumption core (algo #6)
CE-Gate 语义修订获批:块序贯 → 阶梯序前缀逐对序贯。新增 GateSpec/_UnitSlot/
_GateRun 数据结构与 _advance_prefix 纯逻辑(乱序到达下统计严格按预声明阶梯序
消费,INFRA 剔除后重判防 continue 悬置,过线即冻结)。旧块路径共存,Task 6 删。
2026-07-16 23:44:32 -04:00
iomgaa b02db74237 docs: apply Codex plan review fixes (tail-INFRA rejudge, slot deadlock guard) 2026-07-16 23:30:56 -04:00
iomgaa 85ccad3b72 docs: implementation plan for continuous concurrent gate 2026-07-16 23:12:53 -04:00
iomgaa 0e8069af27 docs: gate speedup v3 - ladder-prefix consumption fixes arrival-order bias (Codex round 2) 2026-07-16 22:46:01 -04:00
iomgaa 53a8c4be02 docs: redesign gate speedup v2 (continuous concurrent gate, drop preload) 2026-07-16 22:24:28 -04:00
iomgaa 269e0d294c docs: design for gate speedup (preload BaselineCache + parallel arms) 2026-07-16 14:23:47 -04:00
iomgaa cc01d5ed62 fix: address Codex review of telemetry fix (mkdir degrade, close, degrade tests) 2026-07-16 08:53:50 -04:00
iomgaa 065c8ae1b9 fix: telemetry single persistent connection + lock (kill concurrent write lock) 2026-07-16 08:47:23 -04:00
iomgaa 766592d855 docs: design for telemetry SQLite concurrency fix (align HarnessLog pattern) 2026-07-16 08:47:23 -04:00
iomgaa 933de6251e chore: add FORCE flag to split script; align train config pool sizes to 0.4 split 2026-07-16 07:46:31 -04:00
iomgaa 5fdcd9b5aa feat: add Video-MME 900 training entry (config + self-contained script) 2026-07-16 07:34:14 -04:00
iomgaa a1c4e051e7 Merge branch 'feat/preflight-train-fixes': preflight training fixes (4 WPs, 21+ defects) 2026-07-16 07:25:50 -04:00
iomgaa 6a9b1ea188 docs: record preflight final review + integration fixes (I-3/I-4/M-2) 2026-07-16 07:21:10 -04:00
iomgaa 6911c83d66 refactor: share INFRA stop-reason set from core.evolution 2026-07-16 07:19:31 -04:00
iomgaa b3ba11c7a5 fix: count gate INFRA guard numerator by unit not record 2026-07-16 07:16:33 -04:00
iomgaa ee69721ea3 fix: filter frozen pools by requested task_types subset 2026-07-16 07:13:31 -04:00
iomgaa e1f08dcd3b fix: count units not questions in trainability pre-flight; fail-fast when all types filtered 2026-07-16 06:45:16 -04:00
iomgaa ef244c52bd chore: add trainable_min_units to train_ar30 config 2026-07-16 06:35:10 -04:00
iomgaa ed59c6091a test: accept cache_salt kwarg in AgentLoop mock LLM stub 2026-07-16 06:34:16 -04:00
iomgaa 9be3bdb8eb feat: inject run_id as cache_salt for per-epoch resampling (algo #10) 2026-07-16 06:33:09 -04:00
iomgaa 58a0203522 perf: dedup holdout four-way eval (baseline derive, best_hard memo) 2026-07-16 06:31:00 -04:00
iomgaa 77fd35830c fix: idempotent _run_step (DELETE stale) + checkpoint after gate save 2026-07-16 06:27:38 -04:00
iomgaa efbdeb1647 fix: route None/degraded diagnoses to lapse; abort on high degrade rate 2026-07-16 06:24:11 -04:00
iomgaa 65126feada fix: patch checks full target span + marker injection/integrity (algo #8) 2026-07-16 06:20:29 -04:00
iomgaa caea9150b3 feat: pre-flight filter of untrainable task types before gate 2026-07-16 06:16:02 -04:00
iomgaa c44f6010eb fix: early_stop patience counts epochs not steps 2026-07-16 06:09:11 -04:00
iomgaa 25918a73ff fix: normalize non-scalar prediction; harden predictions insert 2026-07-16 06:06:58 -04:00
iomgaa 7d02cded99 fix: wrap diagnosis run_log with StepsJsonRunLog (restore algo #7 traces) 2026-07-16 06:05:55 -04:00
iomgaa e1cac644c8 docs: address Codex review of WP3 plan (pools passthrough, holdout memo, DELETE table) 2026-07-16 06:03:17 -04:00
iomgaa 03337af8f8 fix: gate INFRA isolation edge cases (all-INFRA fail-loud, parse_error in guard) 2026-07-16 05:54:02 -04:00
iomgaa b307f51340 feat: --retry-uncertain re-diagnoses uncertain rows; fix docstring 2026-07-16 05:41:19 -04:00
iomgaa c68de849da fix: slow-update R2 dual_metric uses slow_candidate kind (not final) 2026-07-16 05:38:05 -04:00
iomgaa 1468a53b7a fix: read-only baseline queries skip _runs upsert (register_run flag) 2026-07-16 05:36:47 -04:00
iomgaa 96884dd149 fix: atomic writes for manifest/record_run/update_best (tmp+replace) 2026-07-16 05:34:45 -04:00
iomgaa d1516bf56b fix: isolate gate baseline-arm INFRA errors from BaselineCache (algo #6) 2026-07-16 05:33:21 -04:00
iomgaa 87908b23eb fix: half-open circuit admits single probe (no thundering herd) 2026-07-16 05:27:46 -04:00
iomgaa 1cbfa97b8d fix: cover httpx disconnect family in transient error set 2026-07-16 05:26:52 -04:00
iomgaa 200053fbdc fix: treat SSE stream without [DONE] as truncated (retry, no cache) 2026-07-16 05:26:21 -04:00
iomgaa 7cc6aa5b23 fix: reject REDIS_CACHE_TTL<=0 (kill implicit never-expire) 2026-07-16 05:25:36 -04:00
iomgaa 564c92a043 feat: add cache_salt dimension to LLM response cache 2026-07-16 05:23:58 -04:00
iomgaa c5c11ea81b docs: address Codex review of WP4 plan (salt key compat, store/protocol, INFRA path) 2026-07-16 05:21:18 -04:00
iomgaa 04bda25335 docs: update WP4 plan workspace anchors after WP2 line drift 2026-07-16 05:14:12 -04:00
iomgaa 8f349c5c62 fix: make frozen product backup unique and atomic on --force 2026-07-16 05:12:08 -04:00
iomgaa a0c7e043e8 fix: validate global frozen pools baseline_run_id + sha256 on load 2026-07-16 05:05:15 -04:00
iomgaa 4d60a545f6 feat: seed carries frozen pools.json into training workspace 2026-07-16 05:02:16 -04:00
iomgaa a39846af6e feat: guard frozen split products against silent overwrite (--force) 2026-07-16 05:00:42 -04:00
iomgaa 5bb8319220 feat: tier-aware diag/val split with val-power repair (design 5.1) 2026-07-16 04:58:41 -04:00
iomgaa 6432ffef9a feat: add dedupe_per_question to extract_run_db (canonical 902->900) 2026-07-16 04:54:19 -04:00
iomgaa 06c575f0a4 chore: bump video_split val_ratio 0.3->0.4 for terminal-eval power 2026-07-16 04:53:14 -04:00
iomgaa 4fd8a6bfef docs: fix WP2 plan GeneratedQuestion import + NULL qid note (Codex review) 2026-07-16 04:52:31 -04:00
iomgaa 232a56e826 fix: fail-loud on missing evolve/diagnose templates (no silent empty) 2026-07-16 04:43:22 -04:00
iomgaa 1108235818 refactor: drop dead prompt fields consolidate_system/span_eval_user 2026-07-16 04:42:26 -04:00
iomgaa d3639a94b8 feat: migrate 5 evolve/momentum templates from TRM4 (algo #8) 2026-07-16 04:41:02 -04:00
iomgaa 6cdf05d56b docs: address Codex review of WP1 plan (real loader tests, red-green order) 2026-07-16 04:38:51 -04:00
iomgaa 39a9210e4d docs: add 4 work-package implementation plans for preflight fixes 2026-07-16 04:32:47 -04:00
iomgaa a4c429b247 chore: snapshot in-progress question-gen work before preflight fixes 2026-07-16 04:12:21 -04:00
iomgaa 11a5545f57 docs: address Codex design review (coverage matrix, salt/gate pairing, tier termination) 2026-07-16 03:52:50 -04:00
iomgaa d7e965639e docs: add preflight training-fixes design (23 bugs + wiring + split tuning) 2026-07-16 03:44:31 -04:00
iomgaa 57f590b57e test: integration for diagnosis tree_data link + core dep direction 2026-07-15 23:05:27 -04:00
iomgaa 224bb25bb5 fix: load real tree_data for training-loop diagnosis (runner)
算法保真 §4.7#12(训练循环编排):仅将 _run_diagnosis 内 tree_data={} 替换为
按 batch question_ids 涉及 video 加载真实树注入 run_diagnosis,未改动三级嵌套/
慢更新/断点续训/gate 等编排逻辑。覆盖 question_ids=None 全量诊断与
question_ids=[...] batch 两条路径。
2026-07-15 23:02:17 -04:00
iomgaa 8dedd082ff fix: self-contained tree-inject test + fail-loud on missing question (review) 2026-07-15 22:48:37 -04:00
iomgaa 83056688cf fix: load real tree_data for offline diagnosis (video_split_cli) 2026-07-15 22:38:23 -04:00
iomgaa f1b6865861 fix: include question_id in diagnosis tree-coverage fail-loud message 2026-07-15 22:33:23 -04:00
iomgaa c5fff7f8b3 fix: fail loud when diagnosis video tree not covered (algo #7 input) 2026-07-15 22:25:30 -04:00
iomgaa d6c595c4a4 fix: fail-loud on node without time info; anchor test store path 2026-07-15 22:16:39 -04:00
iomgaa 05294412df test: cover missing-id and non-list roots; drop unreachable empty-nodes guard 2026-07-15 22:11:36 -04:00
iomgaa e6adaad8b3 feat: add tree.json flattener for diagnosis ground_truth 2026-07-15 22:05:50 -04:00
iomgaa 748a969e0e docs: implementation plan for diagnosis tree_data fix 2026-07-15 22:01:20 -04:00
iomgaa 63dfee1b9b docs: design for diagnosis tree_data broken-link fix (TRM4 regression) 2026-07-15 21:44:42 -04:00
iomgaa 266a1ab4bc fix: use conda run --no-capture-output for unbuffered real-time logs 2026-07-15 14:02:13 -04:00
iomgaa 94728c5220 chore: set video-split diagnosis concurrency to 12 (overridable) 2026-07-15 13:57:59 -04:00
iomgaa 26ab536fea chore: migrate diagnosis prompts from TRM4 (rename to TRM5 loader filenames) 2026-07-15 13:53:16 -04:00
iomgaa 8fef7ced42 fix: address whole-impl review (INFRA T0 rows, reproducible manifest, evolution_target report, dead config, canonical DRY)
C-1: persist_infra_t0_rows 补 INFRA/空预测错题的 T0 信号行(不进诊断故须单独落库),run_pipeline 加 Phase 0,dry-run 用假数据走通。
C-2: CLI 加 --generated-at,真实运行默认盖真实 UTC now,可显式固定以字节级复现 manifest。
I-1: coverage_report 增 evolution_target_distribution(T2 信号按 tool/skill/system 计数)。
I-2: 删除 PoolConfig 死字段 n_trainval/floor_k/epsilon/report_floor/val_wrong_min(grep 确认无消费者,视频级切分用独立 VideoSplitConfig/SplitBuildConfig/SelectConfig)。
I-3: 抽共享 load_canonical_predictions(db_path, run_id),CLI 与 build_split 共用;消除 canonical 取行 + correct 判定重复。
M-1: build_split docstring 注明 val_wrong_min-agnostic 契约(McNemar 护栏由 CLI 冻结后执行,Task 11 契约)。
2026-07-15 13:39:14 -04:00
iomgaa 02b8145b7f fix: fail loud on diag model fingerprint drift vs .env SEARCH_LLM_MODEL
Codex CHANGES_REQUESTED 复审:
- Critical:diag_fingerprint 用 config.diag.model,但 Phase 1 诊断 LLM 从 .env
  SEARCH_LLM_MODEL 构建,两者不一致会让信号以错误模型指纹落库,破坏可复现/
  resume/隔离。build_diagnosis_deps 新增 expected_model 参数,Phase 1 执行前
  fail loud 校验 config.model == settings.search_llm_model(附两值)。
- Minor:config/video_split.yaml diag.model 注释由 JUDGE_LLM_MODEL 更正为
  SEARCH_LLM_MODEL,与实现对齐。
- 补两个单测:模型不一致 fail loud + 缺凭证 fail loud。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 13:20:51 -04:00
iomgaa 2844732126 refactor: self-contained two-phase video-split CLI entry
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 13:12:21 -04:00
iomgaa 6a21d80313 feat: add video-split config knobs and reproducible script 2026-07-15 12:54:25 -04:00
iomgaa 43d7346526 feat: wire end-to-end results-driven split with defensive asserts 2026-07-15 12:43:25 -04:00
iomgaa b497db97ba fix: guard floor phase against n_trainval budget + assert epsilon on output
Codex Task 8 审查修复:
- Important 1: _satisfy_floors 每步移入前检查预算,floor 需求超 n_trainval 时
  抛 InfeasibleSplitError(fail loud),保证 trainval 永不超额挤占 test;补预算超限测试。
- Important 2: 确定性测试末尾用 _epsilon_ok 断言产出 test 真满足 ε(两维偏差回归护栏),
  并断言 trainval <= n_trainval。
- Minor: fixture docstring 注明结构真实 / signal 二次构造,正式运行由真实诊断替换。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 12:38:33 -04:00
iomgaa c3187167c8 feat: add greedy joint-constrained split selector
两阶段贪心视频级切分:Floor 硬约束优先满足 + 多样性 submodular 覆盖最大化,
两阶段均带 ε 守护保 test 代表性;不可行 fail loud(InfeasibleSplitError),
多样性欠额记 loguru warning 不静默。derive_reportable_types 落地长尾报告门限。
确定性:Random(seed) 预洗牌打破等增益/等槽数平局,同 config 同 videos 同解。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 12:32:24 -04:00
iomgaa 84b0b30213 fix: dedup three-bucket projection + honest C3 error-handling note 2026-07-15 12:29:58 -04:00
iomgaa aa10485b9f fix: make pools.json freeze atomic + add split manifest 2026-07-15 12:28:48 -04:00
iomgaa fd907aab46 fix: extend correctness fail-fast to test-side pool questions (P5) 2026-07-15 12:22:11 -04:00
iomgaa 53989078a0 feat: add offline baseline diagnosis orchestration 2026-07-15 12:22:07 -04:00
iomgaa 3d8bd75372 feat: build all-video records with difficulty and signal overlay 2026-07-15 12:18:56 -04:00
iomgaa 20eea98cdd refactor: add video-atomic pool split (algo #5 gate input preserved) 2026-07-15 12:14:36 -04:00
iomgaa 9d19328cc9 feat: add signal tiering 2026-07-15 12:12:48 -04:00
iomgaa 75a2aac1c0 test: cover unknown error_type ValueError path 2026-07-15 12:10:24 -04:00
iomgaa 0a7ba724a9 feat: add evolution_target derivation and cell 2026-07-15 12:06:56 -04:00
iomgaa 3a8dd5c167 fix: add close/context-manager to diagnosis store; tighten import boundary 2026-07-15 12:06:29 -04:00
iomgaa 6b85fcbe1c feat: add baseline diagnosis signal store 2026-07-15 12:01:30 -04:00
iomgaa 01f2e7c7b9 feat: add steps_json-backed RunLog wrapper 2026-07-15 11:51:55 -04:00
iomgaa cf7f15d8bb feat: add steps_json to trace-row adapter 2026-07-15 11:47:00 -04:00
iomgaa 745b600b5d docs: register baseline_diagnosis and split_manifest schemas with baseline metrics 2026-07-15 11:42:38 -04:00
iomgaa cfa510a7e5 docs: add results-driven video-split implementation plan 2026-07-15 11:38:32 -04:00
iomgaa 312cee7be7 docs: add results-driven video-level split design 2026-07-15 11:17:45 -04:00
iomgaa f8ad8f0143 test: harden baseline_cache guard to per-call unit_id check 2026-07-15 10:13:33 -04:00
iomgaa a536a81b0d chore: apply ruff format to dead-path guard test 2026-07-15 10:07:19 -04:00
iomgaa 13234426b5 chore: guard perquestion revival + mark v2 tables deprecated 2026-07-15 10:06:13 -04:00
iomgaa e9349b55c0 test: add v3 pair-contract full-chain integration regression 2026-07-15 09:57:39 -04:00
iomgaa 58a84dc72b refactor: add CHECK constraints and drop redundant index in v3 tables 2026-07-15 09:47:00 -04:00
iomgaa 3cc8dc9105 refactor: assert warn sink, rename disk-pair sifter, trim WHAT docstrings
Capture loguru warning via project sink pattern and assert the dangling-orphan
warning is emitted; rename _sift_disk_pairs to _keep_complete_disk_pairs; drop
pure-WHAT docstrings on __init__/pending_orphans while keeping WHY notes.
2026-07-15 09:42:27 -04:00
iomgaa 5ecbac620c feat: add v3 question-gen observation tables to run_store 2026-07-15 09:33:40 -04:00
iomgaa 1ecb6ba25c fix: fail-loud on disk-corrupt pairs and preserve buffer on reject
read_accepted now distinguishes size==1 dangling orphans (warn+drop) from
structural corruption / role duplication (raise), and runs explicit binding
checks (video_id/task_type/flip_axis) that survive python -O. add() validates
before evicting the buffered partner so pending_orphans can recover it.
2026-07-15 09:31:42 -04:00
iomgaa bdcc93d7de feat: add pair atomic accepted-bank writer helper
PairPendingBuffer collects pair members by pair_id and emits units;
write_accepted does tmp+os.replace atomic write; read_accepted rebuilds
units and drops disk-dangling orphans. Pure helper; on_accept wiring is
Phase 2 (real write point: adversarial_filter.write_final_bank).
2026-07-15 09:22:41 -04:00
iomgaa 19911e18e0 test(harness): 黄金测试加非空护栏、补类型注解、端到端对照旧逻辑
回应 Codex 审查三项测试质量问题:
- C1: 每个 byte-identical 断言前加非空护栏(sum(len)>0 / len==samples>0),
  防空==空误通过。
- I1: TestBatchingByteIdentical._assert 补齐完整类型注解。
- I2: 端到端 checkpoint 用例增加 _reference_build_batches 对照,断言真实池划分
  →batching 与旧逐题逻辑逐字节一致(不只是 rebuilt==batches 自往返)。
2026-07-15 08:28:12 -04:00
iomgaa 6c6fb576ee feat(harness): checkpoint 存 unit_id 序列,断点续跑孪生对不拆
核心算法保真 #3(断点续跑):checkpoint 从逐题 question_id 改为存 unit_id
序列(孪生对折叠为单个 unit_id),恢复时 build_units + 按完整 unit 展开,
续跑后 pair 两成员同进同出、绝不被劈开。

- _batch_unit_ids/_batch_from_ids 对称折叠/展开,保序去重,纯非 AR 下
  unit_id==question_id、与旧逐题序列逐字节一致。
- momentum 采样抽取为 _sample_momentum_candidates 纯函数,docstring 显式
  记录 Phase 1 设计偏差:仅保证纯非 AR byte-identical,混格 momentum 不保证。
- 新增 test_checkpoint_pair(unit_id 落盘往返、pair 不拆)与
  test_non_ar_byte_identical(pools→batching→checkpoint→momentum 端到端黄金)。
2026-07-15 08:22:18 -04:00
iomgaa bd1f7a22a2 feat(harness): pools.json 序列化 pair 四字段防孤儿 single
_q_to_dict 写出 pair_id/question_role/flip_axis/unit_id,_dict_to_q 用 .get
兼容旧 workspace 的 pools.json 读回并回填(unit_id 缺省交 __post_init__)。
pools.json 是训练主回路读回题目处,此前漏写会让孪生对解冻后退化成孤儿
single、配对指标失真。categories 块沿用 per-qid 记录,Task 3 的 unit 原子
切分已保证两 pair 成员同池同 key,序列化不破坏该原子性。
2026-07-15 08:05:15 -04:00
iomgaa 273984674b feat(harness): 迁移 gate 信息量阶梯到 unit 粒度 + schema_version
核心算法保真#5(信息阶梯):gate_ladder.py 从逐题迁移到 unit 粒度,
只换键 question_id→unit_id,冷启动 2:1 错优先交错、gamma-EMA 公式、
Beta(1,1) 先验、反泄漏 _gate_ 过滤的公式/比例/顺序语义一字不改。

- LadderEntry 按 unit_id 键;AR pair 折叠为一个阶梯单元
- build_cold_entries 收单元列表,unit 错 = 任一成员错(双向 AND)折叠,
  2:1 交错 + probe 探针按 unit 抽,Beta 先验 p0 不变
- ladder_for 返回 unit_id 序、exclude 迁到 unit 口径(防半 pair 灌入
  触发下游 _ladder_units fail-fast)
- update_probs 先把逐题观测折叠成单元观测再按 unit_id 匹配更新,
  半观测单元跳过(防按 qid 匹配 pair 失效致 gamma-EMA 停摆)
- GatePools.save/load 加 schema_version=2;存量无版本/旧版本 json
  加载直接报错,拒绝静默混用 qid/unit 键
- BaselineCache 第四维键改名 unit_id(与 T7 validate 路径对齐)
- build_or_load_gate_pools 先折叠单元再排除 test(抽 helper 控复杂度 B)
- runner:_init_gate_pools 建 unit 索引;gate 验证 exclude/展开、
  _refresh_gate_ladder 折叠观测走 units_by_id

反泄漏 run_id 含 _gate_ 过滤不受影响(未改)。
测试:新增 test_gate_ladder_unit_migration.py(15 例覆盖 a-e),
既有 test_harness_gate_ladder.py 迁移到 unit API。全量 1363 passed。
2026-07-15 07:56:15 -04:00
iomgaa 7e97081779 test(harness): 补 _ladder_units 直测 + 澄清 gate 观测表 unit_id 口径注释
M1:quadrant_pair / gate_evidence 的 question_id 列注释与 write_* docstring
更正为承载 unit_id(single=question_id、pair=pair_id),提示逐题明细在
predictions 表溯源、按 pair_id join 真实 question 表会 join 不上。

M2:给 _ladder_units 补直接单测——纯非 AR 恒等(unit 序==原题序、
unit_id==question_id)、混格交错保持信息阶梯序(按单元最早出现下标重排、
pair 整锁)、且与 build_units 的 single-first 默认序显式区分(防阶梯序被污染)。
2026-07-15 07:40:36 -04:00
iomgaa 4b6d1d8a50 feat(harness): correctness 三口径 + gate 块按 unit 跑
进化引擎与 gate e-process 从 question_id 口径迁至 unit_id 口径,AR pair
双向 AND 折叠为单元、不被 P/Q 单题计分污染;逐题 predictions 仅作溯源。

- question_units: 新增 unit_correctness_view(units, per_q)->dict[unit_id,bool]
  作为逐题→单元折叠的唯一入口(复用 unit_correctness)。
- core/evolution/validate: pair_block/compute_accuracy 参数改 unit_ids、
  分母按单元数(键即 unit_id)。
- app/harness/validate(gate 实际执行路径):阶梯题序聚合为单元并保持信息
  阶梯序(_ladder_units),gate 块按单元切分(AR pair 整锁不跨块拆);
  baseline_cache 键含 unit_id、存单元级对错;候选逐题读回后折叠成单元视图;
  n_used/W/L/四象限/准确率均按单元计;证据行按 unit 口径,candidate_correctness
  独立保留逐题对错供 runner 二轨合并。
- runner: probation 结算按 unit 折叠计 W/L(_probation_unit_flips);quadrant
  四象限 id 承载 unit_id。

核心算法保真 #5(信息阶梯 e-process):本次仅迁移 correctness 口径,不改冷启动
2:1 / gamma-EMA / 反泄漏算法本身(gate_ladder 迁移见 Task 8)。
2026-07-15 07:31:03 -04:00
iomgaa dee6bf4896 test: assert specific missing question_id in T6 invariant error match 2026-07-15 07:09:54 -04:00
iomgaa 8a69a54078 refactor: fail-loud unit grading and stricter orphan pair drop
Address review: replace bare KeyError with a contextual ValueError invariant
check in _grade_unit (fail-loud, no catch/skip/fallback); tighten
_drop_orphan_pairs to require exactly one original + one mirror with no extra
illegal-role records (total==2); clarify InferenceResult docstring to unit-grained
semantics. Add tests for missing-prediction descriptive error and extra-illegal-role
pair drop.
2026-07-15 07:05:37 -04:00
iomgaa 61bfa0e633 style: apply ruff format to T6 inference aggregation 2026-07-15 06:58:51 -04:00
iomgaa 730caa7e9a feat: aggregate inference by question unit with pair AND
Reuse build_units/unit_correctness (pair contract single entry) in the
inference aggregation step: single questions count as one unit, AR pairs
collapse original+mirror into one unit scored by bidirectional AND. total/
correct/per_task_type are unit-grained; orphan pairs (missing one side) are
warned and dropped, not counted. Per-question predictions still land row by
row (traceability unchanged).
2026-07-15 06:57:48 -04:00
iomgaa 2429dad393 feat(batching): unit 粒度切分——pair 整锁 + 单元级分桶 + 非 AR 独立 rng
build_batches 改以 QuestionUnit 为原子调度单元:孪生对 2 题整锁进同一 batch、
按单元级正确性(双向 AND)落 correct/error 桶,不再因 P 对 Q 错被劈或被 FFD 拆箱。

- 非 AR(single)用 random.Random(seed) 复现旧逐题算法确切 draw 序列,AR(pair)
  用 _rng_ns(seed,"AR") SHA-256 派生独立流;二者 draw 流互不干扰,故 AR 折叠不改变
  非 AR 抽样/洗牌序列——纯非 AR 输入 build_batches 结果与引入 QuestionUnit 前逐字节一致。
- FFD 容量按 unit.size(pair 占 2),round-robin 遇碎片新开 bin 兜底而非报错。
- _select_mixed_by_task_type 分流各跑一次后合并,大类洗牌按 kind 拆分各用对应 rng。

新增黄金测试 test_batching_pair_lock.py 覆盖三条铁律(同 batch / 单元分桶 /
非 AR byte-identical + draw 流独立);既有 batching 测试全绿。
2026-07-15 06:46:28 -04:00
iomgaa c412698cff style(test): 修 loader unit-sampling 测试的 ruff 告警
TC003 将 Path 导入移入 TYPE_CHECKING 块(仅注解使用),
C408 将 base = dict(...) 改为字典字面量。行为不变。
2026-07-15 06:30:14 -04:00
iomgaa d6a3107e4e feat(question_gen): loader 按 unit 分层采样 + load_benchmark 读回 pair 字段
stratified_sample 先 build_units 聚合,以 QuestionUnit 为采样原子做
分层/去重/补足/rng.sample,返回前 flatten_units 展开为逐题列表;
size/correct_ratio/min_per_class 均按 unit 计数,单元正确性走成员 AND,
孪生对两题永不被劈开。纯 single 输入下 build_units 1:1 折叠、顺序不变,
rng 消耗与旧逐题实现字节级一致(新增回归测试守护)。

_backfill_per_class candidates 改按 unit 枚举去重;build_units/flatten_units
函数内延迟导入以规避 question_gen<->harness 循环依赖(沿用 adversarial_filter)。

load_benchmark 反序列化补 pair_id/question_role/flip_axis/unit_id 四字段,
用 .get 兼容旧 JSON(缺失退化为 single,unit_id 由 __post_init__ 回填)。

pools._sample_excluding 随之改为透传 flatten_units(candidates) 给已单元化的
stratified_sample(不再用 lone pair-original 代表),行为对 single-only 保持等价。
2026-07-15 06:27:41 -04:00
iomgaa ddb9a44f75 feat(pools): 三池切分以 unit 为原子,孪生对同池不被拆散
build_pools/_sample_excluding 与 PerCategoryPoolStrategy._split_one_category/
build_incremental 两条切分路径均改为以 QuestionUnit 为采样原子:progressive
exclusion 互斥集合与 train/val 分层划分都按 unit_id 计数(pair 计 1 个 unit),
命中单元整体展开,AR 孪生对两题永不落入不同池/split。

复用 app.harness.question_units 的 build_units/flatten_units,不重写分组逻辑。
single-only 输入下 unit 与 question 一一对应、rng 消耗量不变,采样与划分结果
与逐题口径完全一致;抽出 _unit_correct/_assert_correctness_complete 两个 helper
将 _split_one_category 复杂度压回基线以下。

新增 tests/unit/test_pools_pair_atomic.py 覆盖两条路径的 pair 原子性回归。
2026-07-15 06:09:14 -04:00
iomgaa 5ef5f2b8b7 refactor: extract _assemble_pair to lower build_units complexity to B
将 pair 分组的数量/角色校验与组装下沉到 _assemble_pair 私有 helper,
build_units 主体只做分组+派发,radon 复杂度从 C(11) 降到 A(4)。
fail-fast 语义、公共函数签名、稳定顺序均不变。
2026-07-15 05:54:08 -04:00
iomgaa bef46636fe feat: add question_units helper as pair contract entry point
build_units/flatten_units/validate_units/unit_correctness——pair 契约唯一入口。
build_units 对孤儿/超员/角色缺失重复 fail-fast raise ValueError(防 next 静默
StopIteration);unit_correctness 走 per_q[qid] KeyError 防静默兜底。
2026-07-15 05:52:30 -04:00
iomgaa 7ef9b99217 feat: add QuestionUnit entity and GeneratedQuestion pair fields
新增 QuestionUnit 契约实体(kind/unit_id/task_type/questions/unit_hash + size/from_single/from_pair),
作为 AR pair 孪生对贯穿评测/训练 harness 的最小调度单元。
GeneratedQuestion 追加 unit_id/pair_id/question_role/flip_axis 四字段,
__post_init__ 回填 unit_id,默认值保证现有非 AR single 题构造点行为不变。
2026-07-15 05:45:34 -04:00
iomgaa 0fe1c96393 docs: add question-gen v3 construction-paradigm design and phase1 plan
Complete v3 planning: construction-first paradigm (frame-perception grounded fact extraction + 4-family independent judges + 6-layer verification + QuestionUnit contract), adversarial audit, paradigm-shift finding, real-data spike validation, logging schema, and phase1 contract implementation plan.
2026-07-15 05:41:10 -04:00
iomgaa d9f7dee2df docs: revise question-gen v2 design per Codex review (pair-unit contract, AR-only gate, phasing) 2026-07-15 01:21:28 -04:00
iomgaa 7949f16906 docs: add question-gen v2 grounded-contrastive design 2026-07-15 01:10:00 -04:00
iomgaa fd96f906c2 docs: add question-gen v2 diagnosis and locked redesign direction 2026-07-15 00:50:29 -04:00
iomgaa 49b7be78ae fix: derive OUTPUT from config output_dir so CONFIG override counts correctly 2026-07-14 21:12:48 -04:00
iomgaa d84f9bf777 chore: harden generate_ar30.sh (drop no-op mock, add zero-gain early stop) 2026-07-14 21:08:00 -04:00
iomgaa b1c1bf7aac fix: isolate selector VLM exceptions at slot level (no batch crash) 2026-07-14 21:08:00 -04:00
iomgaa 441a0aa6c3 chore: format Phase B modules and register Phase B plan in wiki 2026-07-14 17:11:04 -04:00
iomgaa f36eb66c18 fix: tolerate backfill under-delivery, fix predict None-fill, drop dead session_id 2026-07-14 17:07:55 -04:00
iomgaa 1d222d9f18 feat: wire real agent runner, backfill assembly and adversarial-filter CLI 2026-07-14 16:58:32 -04:00
iomgaa 73d0bb9190 feat: add adversarial round loop with backfill iteration and difficulty report 2026-07-14 16:40:05 -04:00
iomgaa c1565a01c2 test: assert stage/verdict/pair_id and Q-refusal flip_skipped path 2026-07-14 16:33:09 -04:00
iomgaa 15aee0cfc1 feat: add flip gate reusing P prediction and mirror-Q agent run 2026-07-14 16:26:51 -04:00
iomgaa 8731e448fe fix: harden mirror material rebuild (empty/invalid frames, dedup, observability) 2026-07-14 16:16:38 -04:00
iomgaa 4af99b700b feat: add mirror question generation with canonical distinctness check 2026-07-14 16:08:13 -04:00
iomgaa d4e9852864 feat: add cheater gate with resume-safe survivor recovery 2026-07-14 16:00:14 -04:00
iomgaa 24ed7ca322 fix: harden canonical_answer_text and align decision-core signatures to spec 2026-07-14 15:55:36 -04:00
iomgaa c109f2257a feat: add pure decision core for adversarial filter (hash/fingerprint/canonical/flip) 2026-07-14 15:51:28 -04:00
iomgaa 334fbbc94d feat: add AdversarialFilterConfig for Phase B post-hoc filter layer 2026-07-14 15:47:21 -04:00
iomgaa ac115d96fb test: cover backfill params end-to-end (mutation isolation + seq_offset) 2026-07-14 15:44:43 -04:00
iomgaa 8b9e8aa19f feat: add optional backfill params to run_pipeline_v2 2026-07-14 15:39:57 -04:00
iomgaa d77cbc95eb feat: add adversarial_verdicts table with resume and terminal-verdict query
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 15:35:11 -04:00
iomgaa e41a2b0d08 test: guard non-AR flip isolation and AR flip-capable set 2026-07-14 15:30:18 -04:00
iomgaa f12dd7e559 feat: add supports_flip/flip_axis to SubPattern for Phase B flip gate 2026-07-14 15:27:21 -04:00
iomgaa 841112c6af docs: revise Phase B plan per Codex review (C1-C3,I1-I6,M1-M3) 2026-07-14 15:24:44 -04:00
iomgaa abc9c097d2 docs: add Phase B adversarial question-gen implementation plan 2026-07-14 15:05:36 -04:00
iomgaa 96e314c3a0 fix: mark item rejected on selector_error for consistent bookkeeping
selector_error 分支(捕获 ValueError/FileNotFoundError)此前只返回 reason,
未 mark_item_rejected,导致 Phase 3 已 record 的 pending attempt 行永远停在
pending;而 hard-fail 分支会标记 rejected。两条失败路径落库风格现统一为
mark_item_rejected(异常路径无 outcome/observation,故不写 selector_scores)。

补单测 test_apply_grounded_selector_marks_rejected_on_error 守卫该路径。
2026-07-14 14:49:41 -04:00
iomgaa 46ac848176 fix: converge non-numeric selector scores to ValueError 2026-07-14 14:47:21 -04:00
iomgaa eaab865d84 docs: register Phase A plan in research wiki 2026-07-14 14:41:53 -04:00
iomgaa e64c26e578 style: format distractor_selector 2026-07-14 14:41:21 -04:00
iomgaa 76f719018c feat: loosen multi_true gate to qualifier-scoped correctness 2026-07-14 14:33:22 -04:00
iomgaa b1f15ddb3a fix: keep cross_segment rule single-dimension, guard no-absent-events 2026-07-14 14:31:30 -04:00
iomgaa 58278c6de4 feat: enforce single-dimension counterfactual in AR distractor rules 2026-07-14 14:28:32 -04:00
iomgaa ea38e4717b refactor: extract grounded selector application into helper
将 Phase 3.5 的 grounded selector 逻辑抽成模块级 _apply_grounded_selector,
内部完成策略门控 / 异常捕获 / observation 落库 / hard-fail 拒绝,调用点仅剩
单一失败分支(selector_reason → 重出)。行为不变,测试全绿。

顺带移除 resample_video_interval 死子表达式(硬编码 1,(attempt-1)%1==0 恒真),
使 _process_one_slot 圈复杂度回落至 Task 6 前的 D(25)。_apply_grounded_selector
自身为 A(4),均满足质量门。
2026-07-14 14:22:30 -04:00
iomgaa b13eab0659 feat: wire grounded selector into AR slot processing
将 Task 5 的 grounded selector 织入 AR 出题路径(Phase 3.5,位于
record_item 与 postprocess 之间),仅在 strategy.uses_grounded_selector
为真时进入。observation 始终落库(含 hard-fail),硬失败走重出。
PipelineConfig 新增 candidate_pool_size/selector_delta_low/
selector_delta_high 三参,YAML 与 CLI seed override 同步。
2026-07-14 14:15:50 -04:00
iomgaa d0194f5840 fix: degrade distractor pool gracefully on malformed VLM response 2026-07-14 14:11:13 -04:00
iomgaa 8a54055d02 feat: add grounded distractor selector with visual scoring 2026-07-14 14:06:07 -04:00
iomgaa 207e834f30 feat: add selector_scores observation column to question_gen_items 2026-07-14 13:59:29 -04:00
iomgaa 3f984acc18 feat: add uses_grounded_selector strategy switch (AR only) 2026-07-14 13:53:17 -04:00
iomgaa 2608a3841f style: reformat sub_pattern round-trip tests 2026-07-14 13:51:51 -04:00
iomgaa e68e4b7d57 fix: restore sub_pattern when loading benchmark JSON 2026-07-14 13:51:05 -04:00
iomgaa 111c88488f style: format sub_pattern test file 2026-07-14 13:46:25 -04:00
iomgaa ae0a718f67 feat: thread and persist sub_pattern into accepted questions 2026-07-14 13:45:31 -04:00
iomgaa 25f2a845ff fix: run_gates must use current_tree after video resample 2026-07-14 13:40:46 -04:00
iomgaa 92714db370 docs: revise Phase A plan per Codex review (C1-C2,I1-I5,M1) 2026-07-14 13:38:35 -04:00
iomgaa 645585e784 docs: add Phase A grounded question-gen implementation plan 2026-07-14 13:28:43 -04:00
iomgaa ad0bfe3a51 docs: add grounded + adversarial question-gen design (Phase A/B) 2026-07-14 13:21:06 -04:00
iomgaa 84b52a0311 feat(pools): auto-supplement maintenance correct questions in PerCategoryPoolStrategy
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 10:41:46 -04:00
iomgaa 72befa2bd4 feat(pools): add batch_correct_ratio field to PoolConfig
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 10:34:28 -04:00
iomgaa 453cf62088 docs: add maintenance pool implementation plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 10:22:52 -04:00
iomgaa 73c5d1e380 docs: add high-correctness warning to maintenance pool design
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 10:13:57 -04:00
iomgaa 89cad891de docs: add maintenance pool auto-supplement design
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 10:11:26 -04:00
iomgaa d7d7ce5bdc feat(question_gen): register ActionRecognitionStrategy, replace temp VISUAL binding
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:51:50 -04:00
iomgaa d0a8019fe1 feat(question_gen): add ActionRecognitionStrategy with 6 SubPatterns
Self-contained strategy targeting 6 Agent failure modes in Action
Recognition: premature_evidence_anchoring, temporal_reasoning_failure,
semantic_rigidity, fine_grained_visual_action,
cross_segment_entity_tracking, evidence_gap_confabulation.

- L2 default sampling (upgrade from L3) with 3 patterns overriding to L1
- Weighted random SubPattern selection (0.20/0.20/0.15/0.15/0.15/0.15)
- Each SubPattern includes instruction, examples, distractor rules
- Satisfies TaskTypeStrategy Protocol without extending BaseTaskTypeStrategy
- 32 unit tests covering all properties, definitions, and selection behavior

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:49:03 -04:00
iomgaa eaa58604b3 feat(question_gen): add Action Recognition specialized prompt template
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:45:09 -04:00
iomgaa b976c789d2 docs: add Plan B — ActionRecognitionStrategy implementation plan
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:34:40 -04:00
iomgaa cd0805aad6 chore: fix N806 lint - rename _RESAMPLE_VIDEO_INTERVAL to lowercase
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:11:01 -04:00
iomgaa 7c7ad67d81 fix(pipeline): move extra_gates after standard gate pass check
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:08:52 -04:00
iomgaa 36c712defa feat(pipeline): replace QuestionFamilySpec with TaskTypeStrategy
- SlotAssignment: remove family field, strategy looked up at process time
- PipelineConfig: remove family_ratios field
- _assign_slots: remove family_ratios and rng params (pure deterministic)
- _process_one_slot: use get_strategy() for sampling, generation, gates
- Add sub_pattern support (level/constraint override, instruction injection)
- Add strategy.extra_gates() check after standard gates
- load_pipeline_config: stop reading family_ratios from YAML
- Update tools/generate_questions.py seed override and dry-run log
- Update all integration tests to match new API

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 06:05:23 -04:00
iomgaa b9616de21e fix(tests): update test_generator_v2 to use new generator signatures
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 05:55:08 -04:00
iomgaa afa77173e3 refactor(question_gen): adapt generator/gates/store signatures for strategy
- generator_v2: _load_prompt_template takes template_name str instead of
  QuestionFamilySpec; _build_v2_prompt takes prompt_template + strategy_name
  + sub_pattern_instruction; generate_one_v2 takes discrete params
  (prompt_template, strategy_name, skill_target, sub_pattern_instruction)
- gates: _gate_leak_test and run_gates take leak_probe_template str
  instead of QuestionFamilySpec
- run_store: add sub_pattern column to DDL + idempotent migration;
  record_item accepts optional sub_pattern param
- Remove QuestionFamilySpec imports from generator_v2 and gates modules
- Update test call sites accordingly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 05:50:47 -04:00
iomgaa c49d0ff12f fix(sampler): validate level param rejects invalid values
Add ValueError guard at the top of sample_material_v2 for level not in
{1, 2, 3}, preventing silent fallthrough to L1 sampling. Add unit test
for the new validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 05:43:28 -04:00
iomgaa e2325b6535 refactor(sampler): replace family_spec param with level+constraint
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 05:38:14 -04:00
iomgaa b6b6a48503 feat(question_gen): add TaskTypeStrategy Protocol and BaseTaskTypeStrategy
- TaskTypeStrategy Protocol: pipeline 的唯一接口,定义 task_type、
  sampling_level、sampling_constraint、prompt_template 等属性
- SubPattern frozen dataclass: 出题子模式,靶向特定失败机制
- BaseTaskTypeStrategy: 封装现有 QuestionFamilySpec 行为的默认策略,
  所有属性委托给绑定的 family
- _TASK_TYPE_TO_FAMILY: 消歧绑定表,12 个题型确定性绑定到 1 个 family
- register_strategy/get_strategy: 注册表 API,未注册题型自动创建
  BaseTaskTypeStrategy
- 13 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-14 05:24:29 -04:00
iomgaa 832838350a docs: add TaskTypeStrategy design + Plan A framework plan 2026-07-14 05:15:28 -04:00
iomgaa 9ee37a8534 fix(loader): use video_id from JSON data instead of filename
load_benchmark used the JSON filename stem as video_id, which broke
v2-360 questions (all 180 questions got video_id='accepted_questions').
Now uses qa['video_id'] when present, falls back to filename for
Video-MME format compatibility.
2026-07-14 01:02:55 -04:00
iomgaa dec7346da3 feat(harness): add Action Recognition training experiment
- PerCategoryPoolStrategy: filter test pool by task_types
- RunConfig: add run_holdout_eval toggle (default true)
- load_config: fix YAML task_types list-to-tuple conversion
- Runner: conditionally skip _holdout_four_way when disabled
- CLI: add --no-run-holdout-eval flag
- New config/train_action_recognition.yaml (3 epochs, per_category)
- New scripts/train_action_recognition.sh (baseline + seed + train)
2026-07-14 00:58:54 -04:00
iomgaa 37d4519905 chore: lint and format per-category pool strategy implementation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:58:10 -04:00
iomgaa 671db2f88c test(integration): add PerCategoryPoolStrategy end-to-end test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:56:42 -04:00
iomgaa 5563d0d675 feat(cli): wire train mode with PoolStrategy selection and task_types in RunConfig
- Add --pool-split-mode, --train-ratio, --test-questions CLI arguments
- Include task_types in cli_overrides (convert list to tuple for RunConfig)
- Wire train mode branch: select strategy based on pool_split_mode,
  resolve workspace paths, build pools, and call runner.train()
- Infer mode now reads task_types from config instead of raw args

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:56:18 -04:00
iomgaa c66a00c924 feat(harness): refactor build_or_load_pools to accept PoolStrategy + per_category freeze format
- save_pools: extended with split_mode and config params; per_category
  mode writes categories metadata (seed, train_ratio, test_source) for
  incremental append and consistency validation
- load_pools: compatible with both old format (no split_mode) and new
  format; extra metadata fields ignored during load
- build_or_load_pools: signature changed to (config, strategy, db_path);
  baseline_run_id read from seed.json (not config.run_id); per_category
  mode does consistency check on reload and supports incremental category
  append via strategy.build_incremental
- Added _to_pool_config, _read_baseline_run_id,
  _validate_per_category_consistency helpers
- Tests: TestPerCategorySaveLoad with 5 test cases covering roundtrip,
  missing config error, global split_mode field, legacy format compat,
  multi-type categories

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:53:19 -04:00
iomgaa e5b07ac974 feat(harness): add task_types, pool_split_mode, train_ratio, test_questions to RunConfig
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:46:28 -04:00
iomgaa 73ae1f7143 fix(harness): change _runs INSERT OR IGNORE to ON CONFLICT DO UPDATE for incremental infer
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:45:18 -04:00
iomgaa 8b48005a17 config: add pool_split_mode, train_ratio, test_questions to default.yaml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:43:43 -04:00
iomgaa 21c6a53aed feat(harness): add PerCategoryPoolStrategy with correctness-stratified 2:1 split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:41:48 -04:00
iomgaa ec4cbbdd44 style(core): move Path import to TYPE_CHECKING block in types.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 22:36:00 -04:00
iomgaa cd5c9c01fb feat(app): add PoolStrategy Protocol to application ports 2026-07-12 22:34:47 -04:00
iomgaa 0d0f275134 feat(core): add PoolConfig dataclass for pool strategy configuration 2026-07-12 22:33:49 -04:00
iomgaa 4fb7a61f8b fix(question_gen): resolve pipeline integration issues from final review
1. Apply postprocess shuffle result (pp.options, pp.answer) to final
   GeneratedQuestion output instead of using original candidate values.

2. Record dedup rejection in store via new mark_item_rejected() method,
   preventing items from staying as 'accepted' after dedup rejects them.

3. Add .flatten() to embed_fn outputs in _is_duplicate and embed_pool
   append to handle 2D (1,D) arrays from embedding implementations.

4. Validate exactly 4 options in _validate_parsed_fields (was >= 2),
   matching the A-D answer constraint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 00:14:12 -04:00
iomgaa eecb86e27a feat(question_gen): add generate-v2 CLI subcommand and experiment script
- Add generate-v2 subparser with --config, --store-dir, --db-path,
  --seed, and --dry-run arguments to tools/generate_questions.py
- Implement _run_generate_v2 async handler: config loading, video
  discovery, DI client construction, TreeIndex loading, pipeline
  invocation, and result persistence
- Add scripts/generate_questions_v2.sh following build_trees.sh
  conventions (source .env, conda run python path, MODE=mock support)
- Update app/question_gen/__init__.py to export full v2 public API:
  run_pipeline_v2, PipelineConfig, PipelineResult, QuestionFamilySpec,
  ALL_FAMILIES, CandidateQuestion, generate_one_v2, GateReport, run_gates
- Add QuestionGenStore.load_progress() for pipeline resumption
- Add integration tests for CLI help and dry-run behavior
- Update test_question_gen_api to match expanded __all__

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-12 00:04:42 -04:00
iomgaa f46e87258c refactor(question_gen): extract helpers to reduce pipeline_v2 CC below grade C
Extract _get_git_sha, _filter_pending_slots, and _run_heavy_sampling
from run_pipeline_v2. Reduces cyclomatic complexity from C(15) to B(6).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:57:35 -04:00
iomgaa 206c553143 feat(question_gen): add v2 pipeline with retry loop and heavy check
- PipelineConfig: YAML-driven configuration with family_ratios, retry,
  concurrency, dedup threshold, and heavy sampling rate
- _assign_slots: deterministic round-robin slot assignment across videos
  with per-family weighted random selection
- _process_one_slot: full retry loop (generate → postprocess → gates →
  dedup) with reject-reason feedback to VLM on retry
- _heavy_check_one: blind LLM agent trial-answer for difficulty_steps
- run_pipeline_v2: orchestration with semaphore-bounded concurrency,
  progress/resume support, and store integration
- is_duplicate: cosine similarity dedup against embedding pool

Tests: 11 integration tests covering slot assignment, retry behavior,
max-retries exhaustion, full pipeline flow, progress resume, heavy
sampling, and store record completeness.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:55:15 -04:00
iomgaa 6d6eb8e3a3 refactor(question_gen): extract _validate_parsed_fields to reduce CC
Split field validation logic out of _parse_v2_response into a dedicated
_validate_parsed_fields helper. This brings _parse_v2_response from CC=11
(grade C) down to CC=3 (grade A). The extracted validator is CC=9 (grade B).
No grade-C functions remain.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:43:35 -04:00
iomgaa cf51d2de9d feat(question_gen): add v2 generator with per-family prompt templates
Implement generator_v2.py with:
- CandidateQuestion dataclass (canonical location)
- _load_prompt_template: loads per-family .md from store/prompts/
- _build_v2_prompt: constructs system+user messages with material context
- _parse_v2_response: JSON extraction, json_repair, field validation
- generate_one_v2: async VLM call orchestration with reject_reason support

Add 5 family-specific prompt templates:
- retrieval.md: factual recall from visible content
- reasoning.md: multi-hop inference across segments
- enumeration.md: counting/listing entities and actions
- visual.md: visual details requiring frame observation
- spatial.md: spatial relationships between objects/people

Tests: 11 unit tests covering prompt build, parse, and e2e generation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:41:36 -04:00
iomgaa 9053233f99 fix(question_gen): gates reject_reason returns raw reason; short-circuit skips LLM
- GateReport.reject_reason now returns gate.reason directly (no [name] prefix)
- verbatim short-circuit sets other 3 gates to SKIP without calling LLM
- test_high_verbatim_shortcircuits asserts zero LLM calls and SKIP verdicts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:33:57 -04:00
iomgaa 9f739e831d refactor(question_gen): reduce cyclomatic complexity in sampler_v2
Extract shared _resolve_subtree helper to eliminate repeated tri-level
node resolution. Break _validate_sampling_constraints into focused
single-purpose helpers:
- _count_l3_descendants
- _has_frames
- _count_subtitles
- _resolve_subtree / _find_l3_parent

Extract _subtitles_from_l2_list and _frames_from_l2_list to simplify
collection functions.

Complexity improvements:
- _validate_sampling_constraints: D(23) -> B(8)
- _collect_subtitle_sentences: C(16) -> A(3)
- _collect_frame_paths: C(13) -> A(2)

All functions now grade B or better per radon cc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:33:48 -04:00
iomgaa f74711cd11 feat(question_gen): add v2 material sampler with family constraints
Implement sample_material_v2 module that samples tree nodes with
QuestionFamilySpec-aware constraint validation, providing richer
MaterialContext output (subtitles, cross-L2 context, frame paths).

Key components:
- AnchorContext/MaterialContext frozen dataclasses
- _validate_sampling_constraints: multi-level constraint checking
- _collect_subtitle_sentences: subtree subtitle extraction
- _collect_cross_l2_context: peer L2 event descriptions
- sample_material_v2: main entry with retry-on-constraint-violation

Tests: 11 unit tests covering normal sampling, used-node exclusion,
constraint violation retries, cross-L2 population, and subtitle
collection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:30:56 -04:00
iomgaa 271d1682c9 feat(question_gen): add lightweight 4-gate quality check
Implement 4 concurrent LLM-based quality gates for generated questions:
- key_verify: validates answer evidence in source material
- blind_answer: rejects questions answerable without video context
- multi_true: detects ambiguous multi-correct options
- leak_test: per-family shortcut detection (5 probe templates)

Includes run_gates orchestrator with verbatim_ratio short-circuit,
JSON response parsing with fallback, and 9 unit tests (all passing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:30:13 -04:00
iomgaa 9627ac9cf9 fix(question_gen): raise ValueError on UPDATE of missing rows
record_run_end, update_gates, and update_difficulty now check
cursor.rowcount after UPDATE+commit and raise ValueError if 0 rows
were affected. Prevents silent telemetry loss.

Adds three negative-path tests:
- test_record_run_end_missing_run_raises
- test_update_gates_missing_item_raises
- test_update_difficulty_missing_item_raises

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:23:48 -04:00
iomgaa 7abe92eb1c fix(question_gen): check_verbatim covers question_text + add missing blacklist patterns
- check_verbatim now computes n-gram overlap for BOTH question_text and
  correct_option vs source texts, returning max(question_ratio, option_ratio).
  Extracted _ngram_overlap_ratio helper for reuse.

- Added 4 missing blacklist patterns: 'this segment', 'this frame',
  'the current frame', 'frame summary'.

- Added 5 new test cases covering the above changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:23:37 -04:00
iomgaa 9525726133 fix(question_gen): rename finished_at to ended_at per schema spec
Aligns DDL column name with research-wiki/schemas/question-gen-runs.md
which specifies 'ended_at' (not 'finished_at').

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:18:52 -04:00
iomgaa 74686dde68 refactor(question_gen): reduce check_forbidden_material complexity to A(3)
Extract _match_any helper and declarative _FORBIDDEN_MATERIAL_RULES table
to replace repetitive per-category for-loops. Reduces cyclomatic complexity
from C(11) to A(3).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:16:34 -04:00
iomgaa 6a5424a618 feat(question_gen): add SQLite run store for generation telemetry
Implements QuestionGenStore with:
- Idempotent schema initialization (question_gen_runs + question_gen_items)
- Run lifecycle: record_run_start / record_run_end / get_run_stats
- Per-item recording: record_item / update_gates / update_difficulty
- GateReportLike Protocol for duck-type gate report compatibility
- WAL mode + foreign keys + check_same_thread=False

DDL aligns with research-wiki/schemas/question-gen-{runs,items}.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:15:53 -04:00
iomgaa c83d771923 feat(question_gen): add deterministic postprocess layer
Add app/question_gen/postprocess.py with zero-LLM deterministic
post-processing for generated questions:

- shuffle_options: deterministic option permutation with answer remapping
- check_referent_blacklist: detect self-referential language (this clip, etc.)
- check_verbatim: word-level n-gram overlap ratio measurement
- has_time_anchor: timestamp and temporal phrase detection
- check_forbidden_material: T1/T7 source material validation
- run_postprocess: orchestration returning PostprocessResult

Tests: 31 unit tests covering all functions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:14:43 -04:00
iomgaa 75e6d8c550 feat(question_gen): add 5 question family specs with sampling constraints
Define QuestionFamilySpec, LeakTestProfile, SamplingConstraint dataclasses
and instantiate 5 families (RETRIEVAL/REASONING/ENUMERATION/VISUAL/SPATIAL)
targeting failure mechanisms M1-M5. Implement get_family_for_slot with
legal-type filtering + weighted random selection.

13 unit tests cover: full task-type coverage, skill_target uniqueness,
deterministic seeding, invalid input errors, and chi-square distribution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:14:22 -04:00
iomgaa 811ffa648b feat(types): extend GeneratedQuestion with skill_target & difficulty_steps
- Add skill_target (str | None) and difficulty_steps (int | None) fields
  to GeneratedQuestion dataclass with field(default=None)
- Update loader.py to pass new fields from JSON (backward-compatible)
- Update pools.py _q_to_dict/_dict_to_q for serialization compat
- Add question_gen_v2 config section to default.yaml
- Add comprehensive test coverage (7 tests)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-11 23:05:38 -04:00
iomgaa 043d4aa46f docs(plans): add Spec-3 question-gen-v2 implementation plan
9 tasks: type extension → postprocess → families → sampler_v2 →
generator_v2 → gates → run_store → pipeline_v2 → CLI integration.
Includes structured-logging schemas/metrics and Codex review revisions.
2026-07-11 22:57:47 -04:00
iomgaa 8c9adfd3fa docs(wiki): record Spec-2 harness-eval finding (N/A — pure tooling addition) 2026-07-11 12:13:22 -04:00
iomgaa a51b3c4238 fix(tools): validate concurrency params to prevent zero-semaphore deadlock
Codex 最终审查发现 video_concurrency=0 或 TREE_BUILD_API_CONCURRENCY=0
会创建永不放行的 Semaphore 导致整批挂死。追加 >= 1 校验,fail-fast 报错。
2026-07-11 12:11:48 -04:00
iomgaa 0290005129 fix(scripts): align build_trees.sh with project shell conventions
Codex 审查发现两处偏离既有脚本惯例:
1. conda activate 改为 PYTHON=$(conda run ...) + ${PYTHON} 模式
2. 追加 source .env 确保 TREE_BUILD_API_CONCURRENCY 等工程配置可用
2026-07-11 12:07:14 -04:00
iomgaa 928ec8c37b feat(scripts): add batch tree build entry with smoke-test guide 2026-07-11 12:04:34 -04:00
iomgaa c5bf7e1a61 fix(tools): widen error isolation to cover full single-video pipeline
Codex 功能质量审查发现 _build_one 的 try/except 仅包裹 build_async,
构造器、SRT 解析、save_json、save_progress 抛异常会逃逸到 gather 中止整批。
将 try 边界扩大到覆盖完整单视频流程(构建→建树→保存→进度记录)。
2026-07-11 12:02:21 -04:00
iomgaa 56fe138a46 feat(tools): batch tree build orchestration with shared API semaphore 2026-07-11 11:56:36 -04:00
iomgaa 978ddef91b feat(tools): add build_trees skeleton with discovery helpers 2026-07-11 11:41:22 -04:00
iomgaa e9073bfdc2 feat(tree): expose build_async and accept injected API semaphore
Core algorithms #1/#2/#3 unchanged: only entry wrapping and semaphore
source switch (injected vs self-built); build logic untouched.
- rename _build_async to public build_async (body unchanged)
- __init__ accepts keyword-only api_semaphore for cross-video sharing
- default path (no injection) is verbatim-equivalent to previous code
2026-07-11 11:14:45 -04:00
iomgaa 25f5537974 docs(plans): add Spec-2 batch tree build implementation plan 2026-07-11 11:11:02 -04:00
iomgaa 1d2385291a docs(wiki): record Spec-1 harness-eval finding (all metrics pass) 2026-07-11 09:33:53 -04:00
iomgaa 40b156cf69 docs(wiki): record final Codex PASS for recipe redesign round 2026-07-11 09:08:27 -04:00
iomgaa ebbd112d50 fix(tree): move Iterable import into type-checking block 2026-07-11 09:05:04 -04:00
iomgaa a046862ca1 docs(wiki): record build-tree recipe redesign and diagnosis stage tags 2026-07-11 09:04:57 -04:00
iomgaa f0284199b8 refactor(tree): narrow _clean_join parameter type to Iterable 2026-07-11 09:04:25 -04:00
iomgaa 1c21e215e2 test(tree): cover L1 entity fields and block ordering in view_node 2026-07-11 09:02:03 -04:00
iomgaa a2c6a97621 refactor(tree): extract _clean_join to keep entity-fields complexity at B
质量门要求新增代码复杂度不超过 B 级:将 node_entity_fields 中重复的
"清洗-去重-拼接"逻辑抽为模块级纯函数 _clean_join,主体收敛为两次
调用。radon:node_entity_fields C(13) → B(6),_clean_join A(5)。

算法 #11(树环境语义搜索)数据访问层,行为不变(既有 4 测试未改全绿)。
2026-07-11 08:56:17 -04:00
iomgaa 4e0e05210d feat(search): append raw entity fields after view_node summary
view_node 按题两轮摘要(summarize_node)会吞掉 entities/visible_text
字段信号,Agent 站在证据节点上仍漏读实体(benchmark 错题 M1,案例
786-2、872-3、750-1)。dispatcher 侧在摘要后确定性追加 [实体]/[画面
文字] 原文区块,LLM 无法吞掉。

- TreeEnvironment.node_entity_fields:按层级取 card 实体字段原文,
  去空白、去重、分号拼接;空字段省键;未知节点抛 KeyError
- _handle_view_node Phase 2.5:摘要后、子节点概览前追加实体区块
- 附带 ruff format 修正 test_tree_environment.py 两处既有格式

算法 #11(树环境语义搜索)数据访问层扩展,不改搜索算法本身。
2026-07-11 08:53:56 -04:00
iomgaa 291a8108e1 test(agent): tighten type annotations in step-retry tests 2026-07-11 08:48:04 -04:00
iomgaa badfcce4cb fix(agent): validate non-empty retry delays; pin exhaustion semantics
核心算法 #10(Agent Loop):Codex 质量审查跟进,仅加固防御与
可观测,不改变循环语义。
- __init__ 校验 step_retry_delays 非空,空序列直接抛 ValueError
  (P5 fail-fast,避免首次可重试异常时 IndexError 掩盖原始 LLM
  异常、违背方法契约)
- 补耗尽语义窄测试(耗尽后原样抛出最后一次原始异常)与空序列
  构造校验测试
- run() 最终失败日志补异常类型名,便于回溯归因
2026-07-11 08:44:47 -04:00
iomgaa e3184c11f9 feat(agent): step-level retry for transient LLM errors (20s/40s backoff)
核心算法 #10(Agent Loop):仅加固异常路径的韧性兜底,不改变
解析协议、hook 时序与步数语义。benchmark 错题 796-3 显示一次
SSL BAD_RECORD_MAC 穿透 GovernedLLMClient 重试栈后废掉 13 步
已积累上下文;本次在 run() Phase 1 增加步级重试(默认 2 次,
20s/40s 退避),可重试异常限定 (TimeoutError, OSError),非可
重试异常照旧 fail-fast 整题终止,行为与现状一致(P5 显式异常)。
为满足 radon C 级复杂度约束,重试循环抽取为私有方法
_call_llm_with_step_retry,行为不变。
2026-07-11 08:37:35 -04:00
iomgaa cf529f2c8f test(agent): add docstring to empty-content boundary test 2026-07-11 08:33:36 -04:00
iomgaa 439dc29b3b fix(agent): reject argless action in normalization; add boundary test
核心算法 #10(Agent Loop):修复 Codex 质量审查 Critical——
_normalize_action 仅在除 tool 外至少存在一个平铺参数键时才收拢,
{"tool": "x"} 无参结构不再被静默升级为空 args 合法结构,照旧
返回 None 走 retry 追问路径。补边界测试 + 测试辅助方法类型注解
与中文 docstring。
2026-07-11 08:28:59 -04:00
iomgaa 6034d4172d test(question_gen): update __all__ assertion to include synthesizer exports 2026-07-11 08:21:06 -04:00
iomgaa a31b1fbf37 test(question_gen): drop tests for removed calibrate pairing validator 2026-07-11 08:19:45 -04:00
iomgaa 8d84d5e236 fix(agent): normalize fenced and flat-args LLM outputs in parser
核心算法 #10(Agent Loop):仅加固 _parse_response 解析路径——
剥除 ```json 围栏 + 收拢 action 平铺参数(deepseek 稳定输出变体,
案例 637-3/615-3 三连拒 0 步阵亡)。Thinking+JSON 协议、json_repair
兜底链、hook 时序与步数语义均未改动。
2026-07-11 08:16:28 -04:00
iomgaa 39c6352781 docs(plans): add Spec-1 implementation plan with Codex review fixes 2026-07-11 08:12:07 -04:00
iomgaa ef402c46a2 docs(designs): apply Codex review fixes to Spec-1/2/3 2026-07-11 07:54:48 -04:00
iomgaa a5666b4f16 docs(wiki): record tree-figure style unification and review fixes 2026-07-11 07:52:22 -04:00
iomgaa a79c2ec753 docs(wiki): add three-round analysis findings and Spec-1/2/3 designs 2026-07-11 07:45:51 -04:00
iomgaa 658e62054e feat(scripts): add generated-questions infer and calibrate entry 2026-07-11 07:45:42 -04:00
iomgaa da70eb6e23 refactor(question_gen): slim calibrate to compare two existing runs 2026-07-11 07:45:42 -04:00
iomgaa 307c64c388 fix(harness): infer mode honors CLI --questions over manifest 2026-07-11 07:45:33 -04:00
iomgaa 735471df3c fix(question_gen): Information Synopsis 去重阈值放宽至 0.90
该题型天然高相似(benchmark 均值 0.66,中位数 0.67),
通用阈值 0.75/0.85 导致几乎所有新题被误判重复。
按题型覆盖为 0.90,只拦截真正的复制品。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-10 08:55:47 -04:00
iomgaa 25c8d5ec42 fix(question_gen): 每次重试换视频避免同视频反复去重失败
Information Synopsis 163 道 benchmark 题,同一视频反复出题
必然相似。改为每次重试随机选不同视频。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-10 08:15:03 -04:00
iomgaa d0148e78ad docs(wiki): record main-figure final review verdict and caption notes 2026-07-10 01:40:04 -04:00
iomgaa 6a6aae081b docs(wiki): record main-figure final layout and panel decisions 2026-07-10 01:38:25 -04:00
iomgaa 69a4d17c0b config: LLM 总超时 120s→300s(5 分钟)
VLM 题目生成(图片+出题+4选项)输出长,120s 不够。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-10 01:11:40 -04:00
iomgaa 5b51f4bd0c fix(synthesizer): generate_one 捕获所有异常避免 VLM 超时穿透崩溃
except (ValueError, KeyError) → except Exception,
覆盖 StreamLivenessTimeout 等网络/超时异常。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-10 01:08:03 -04:00
iomgaa 8d11513e54 fix: 禁止日志缓存,确保所有日志立刻输出
- scripts/*.sh: PYTHONUNBUFFERED=1
- tools/generate_questions.py: loguru file sink enqueue=False
- main.py: loguru file sink enqueue=False

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 23:15:02 -04:00
iomgaa c191be16e2 fix(question_gen): L3 frame_path 拼接视频目录前缀
tree.json 中 frame_path 是相对路径(frames/xxx.jpg),
generate_questions 需要拼上 store/videos/{video_id}/ 前缀供 VLM 读取。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 23:10:18 -04:00
iomgaa 09b2773375 config(scripts): generate_questions 默认并发 8→16
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 23:07:57 -04:00
iomgaa c84de49dc4 feat(main): loguru 文件 sink 持久化运行日志
logs/run_YYYYMMDD_HHmmss.log,500MB 轮转,保留 30 天,
enqueue=True 异步写入不阻塞推理。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 22:56:34 -04:00
iomgaa 75016c4892 feat(scripts): generate_questions.sh 题目生成实验入口
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 22:53:28 -04:00
iomgaa 4ebdf0b0ba fix(main): InfraSettings 默认值同步 retry_base=20 retry_max=120
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 22:16:22 -04:00
iomgaa 1f83da9eb7 fix: embedding 改 cuda + OCR 超时 300s + 重试退避 20s 起 + jitter 防雷群
- config/default.yaml: embed.device cpu → cuda
- adapters/ocr.py: _TIMEOUT_S 15 → 300(5 分钟)
- adapters/llm.py: 指数退避加 ±50% jitter 避免 24 路同时重试
- .env: LLM_RETRY_BASE_DELAY 2→20s, LLM_RETRY_MAX_DELAY 30→120s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 22:14:57 -04:00
iomgaa 38a855b33e fix(scripts): 加 HF_HUB_OFFLINE 跳过在线检查
embedding 模型已有本地缓存,网络不通时避免连接 huggingface.co 失败。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 13:56:25 -04:00
iomgaa d6ae0d85cd fix: Redis 缓存构造修正 + TTL=0 永不过期支持
- main.py: 先创建 aioredis 客户端再传入 RedisResponseCache
- redis_cache.py: ttl_s=None 时不设过期时间
- .env: REDIS_CACHE_TTL=0(永不过期)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 13:52:33 -04:00
iomgaa 953fb7a456 feat(scripts): infer_video_mme.sh 推理实验入口
零参数全量 900 题、N_SAMPLES smoke test、TASK_TYPES 筛选、版本覆盖。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 13:40:31 -04:00
iomgaa cabe5be038 fix(main): assemble_mode 从 'default' 修正为 'ids_expand'
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 13:21:05 -04:00
iomgaa f8f8b5b12d feat: main.py Composition Root(仅 infer 模式)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:27:13 -04:00
iomgaa ba400417e2 config: concurrency=24, max_steps=40, breaker_threshold=48
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:27:05 -04:00
iomgaa a7ca6d15ed feat(harness): InferenceDepsRouter per-video 路由器
按 video_id 懒加载 InferenceDeps 并缓存,路由 dispatch/prompt_builder:
- create_dispatch: 按 session_id 路由到对应视频的工具调度
- create_prompt_builder: 自动注册 qid→vid 映射并路由 prompt 构建
- 三元组 (video_id, skills_dir, prompts_dir) 缓存键

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:20:32 -04:00
iomgaa f21bf345a6 feat(runner): 注入 tool_dispatch_factory/prompt_builder_factory + fail-fast
Runner.__init__ 新增 2 个可选参数:
- tool_dispatch_factory: 工具调度工厂
- prompt_builder_factory: prompt 构建工厂

infer/eval/train 模式缺少工厂时 fail-fast 抛 ValueError。
_make_tool_dispatch_fn/_make_prompt_builder 优先使用注入工厂。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:20:25 -04:00
iomgaa 924160c779 feat(ports): 新增 ToolDispatchFactory/PromptBuilderFactory Protocol
4 个新 Protocol 类型:
- ToolDispatchFn: 工具调度函数签名
- ToolDispatchFactory: per-version 工具调度工厂
- PromptBuilderFn: Prompt 构建函数签名
- PromptBuilderFactory: per-version prompt 构建工厂

含 runtime_checkable isinstance 测试。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:20:16 -04:00
iomgaa e7be42570d feat(store): skills/v1 初始集 — TRM4 精简 + TRM5 card 字段注入
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:12:47 -04:00
iomgaa bb6dad7005 refactor(store): prompts 版本化目录重组 + skills/v1 骨架 + system.md L1 去 subtitle
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:07:19 -04:00
iomgaa 16140cafcc fix(tree): from_dict L2 subtitle 旧格式兼容 fallback
旧格式 tree.json 无 card-level L2 subtitle 时,从已构造的 L3
children 聚合 subtitle,避免运行时加载静默丢失 L2 字幕。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 12:04:36 -04:00
iomgaa d3be9b1322 refactor(tree): subtitle 迁入 L3Card/L2Card + 建树管线修正
- L3Card/L2Card 新增 subtitle: str 字段(L1Card 不加)
- L3Node 移除 subtitle 字段(数据迁入 Card)
- assign_subtitles_voronoi 改写 Card.subtitle + L2 聚合
- _collect_card_strings 增加 skip_fields 排除 subtitle
- _node_full_text/_node_anchored_text 保持 字幕:/[sN] 语义
- get_subtitle 读 Card.subtitle(L2/L3)
- verify.py/synthesizer.py: l3.subtitle → l3.card.subtitle
- 迁移脚本 tools/migrate_subtitle_to_card.py(幂等,300 棵树已迁移)
- 9→6 个测试文件适配(3 个无需改动)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 11:57:41 -04:00
iomgaa c72b55508a fix(plan): Task 0 Codex 审查修复 — 锚标保持/双写消除/幂等迁移/遗漏文件
- [C4] 加入 verify.py/synthesizer.py 到变更清单
- [I3a] L2 subtitle 统一由 Voronoi+聚合处理,移除 _build_l2 双写
- [I3b] File Structure 修正为'保留 Phase 7'
- [I5] 迁移脚本幂等性修复
- [I6] 保持 [cN]/[sN] 双锚语义,_collect_card_strings 跳过 subtitle

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 11:43:33 -04:00
iomgaa 160fb3bc7c docs(plan): main.py 推理入口实现计划 + Codex 审查修复
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 10:43:15 -04:00
iomgaa 457944bbfa fix(design): 修复 Codex 审查发现的 5 项问题
- C1: 13 个 skill(非 14)
- C2: factory 从 noop fallback 改为 fail-fast 校验
- I1: 新增 InfraSettings(BaseSettings) 工程配置模型
- I2: 显式 supersede 搜索模块设计的 prompt 路径约定
- I3: Protocol 返回类型精确化
- I4: dispatch 对缺失 session_id 显式 raise

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 08:47:23 -04:00
iomgaa 5d0caa06e1 docs(design): main.py 推理入口 + 初始 Prompt 集设计
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 08:41:22 -04:00
iomgaa 09d337658b merge: feat/repair-concurrent — 赛题生成工具 + repair 管线完善
新增模块:
- app/question_gen/synthesizer.py — 题型映射、节点采样、prompt 构造、去重
- app/harness/factory.py — 推理依赖组装(可复用)
- tools/generate_questions.py — generate + calibrate CLI

修复:
- _VIDEO_MME_TASK_TYPE_COUNT 11→12
- repair/supplement 防御性修复
- repair/regenerator VLM 重生成器
2026-07-09 07:54:21 -04:00
iomgaa 45403b23b4 feat(repair): regenerator + supplement 防御修复 + 迁移脚本
- 新增 app/tree/repair/regenerator.py(VLM 重生成 + 级联修复)
- supplement.py: deduplicate_field str() 防御 + inject_value strip
- patch.py: ruff format 格式化
- repair_trees.sh: conda source 激活修复
- 新增 migrate_from_trm4.sh 迁移工具
- enhance/__init__.py → repair/__init__.py 重命名

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 07:53:59 -04:00
iomgaa ace0738546 docs(wiki): record main-figure composition rework and review outcomes 2026-07-09 07:50:24 -04:00
iomgaa f57ee45dc0 fix: Codex 全量审查修正
Critical:
- C1: assemble_mode 'plain' → 'ids'(合法枚举值)
- C2: question_id 加入 task_type slug 避免跨题型冲突

Important/Minor:
- generate_one 移除未用的 embed_fn/similarity_threshold 参数
- config.py 注释 11→12 同步
- 测试 question_id 断言更新

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 07:40:27 -04:00
iomgaa fad8147d71 refactor(question_gen): __init__.py 追加 synthesizer re-export
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:49:05 -04:00
iomgaa 93c9be8bfa feat(tools): generate_questions.py calibrate 子命令
- Fisher exact test + effect size 组合判定(PASS/WARN/FAIL)
- 按 video_id 分组推理,避免跨视频树错用
- baseline 支持从 DB 读取或自动跑推理
- 对比表输出 + 退出码控制

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:47:52 -04:00
iomgaa 11f3c90200 feat(tools): generate_questions.py generate 子命令
- VLM 出题 + embedding 去重 + 断点续跑 + 并发控制
- 单线程汇总点保证去重原子性
- 18 个单元测试覆盖 progress/exemplar/pool rebuild/JSON append

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:42:50 -04:00
iomgaa 6e46d184b8 feat(harness): add factory.py — InferenceDeps dataclass + build_inference_deps
组装一次推理所需的全套依赖的工厂函数:
- TreeIndex 加载(FileNotFoundError if missing)
- TreeEnvironment 构建
- SkillRegistry 按需发现
- SearchToolDispatcher 装配
- PromptManager + prompt_builder 闭包

测试覆盖:正常路径、缺失树文件、skills 注入、frozen 不可变性。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:36:07 -04:00
iomgaa 5aa7cc48c5 feat(question_gen): is_duplicate + generate_one — 去重判定与单题生成编排
- is_duplicate: 余弦相似度去重,空池短路
- generate_one: 异步重试循环,不含去重(由调用方汇总点原子执行)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:32:39 -04:00
iomgaa 90f17e330e feat(question_gen): build_generation_prompt + parse_vlm_response
- prompt 组装:system(角色+题型+约束+few-shot) + user(card+字幕+干扰项)
- VLM 响应解析:JSON 直接 + markdown code block 回退,四选一 schema 校验

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:28:25 -04:00
iomgaa 40b04f886e fix(question_gen): sample_anchor Codex 审查修正
- C1/C2: Temporal Reasoning ≥3 L2 + Object Reasoning ≥2 L2 下限检查
- I1: L2 题型子帧不足时 ValueError
- I2: L3 过滤无 frame_path 的节点
- I3/I4: distractor_texts 扩展到整棵树范围
- I5-I7: 测试补强 Information Synopsis/Temporal Reasoning/Object Reasoning
- M1: Spatial Reasoning 测试断言 spatial_layout

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:24:30 -04:00
iomgaa a597a9f901 feat(question_gen): sample_anchor — 按题型层级采样锚节点
含 6 种层级分支:L3 单帧、L2 多帧、Temporal Perception 特例、
L1 全量/采样 L2、L1-L2 混合。时间排序 + used_node_ids 排除。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:18:25 -04:00
iomgaa 9eb9b86954 feat(question_gen): AnchorContext + 12 题型-层级映射常量
- AnchorContext frozen dataclass: 锚节点生成上下文(node_id, card_text, frame_paths, subtitle, distractor_texts)
- TaskTypeSpec frozen dataclass: 题型生成规格(level, needs_frames, frame_count, context_fields)
- TASK_TYPE_LEVEL_MAP: 12 种 Video-MME 题型 → 树层级 + 生成规格映射
- 11 项单元测试覆盖:映射完整性、值类型、层级合法性、frozen 不变性

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:10:58 -04:00
iomgaa eb15ab315e fix(config): _VIDEO_MME_TASK_TYPE_COUNT 11→12,Video-MME 实际有 12 种题型
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:08:05 -04:00
iomgaa 0b48b889e0 docs(wiki): 赛题生成工具设计 + 实现计划
design: synthesizer + factory + CLI 三模块架构
plan: 9 个 Task(前置修复 + synthesizer 4 步 + factory + CLI generate/calibrate + re-export)
calibrate: Fisher exact test 组合判定替代固定阈值

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 05:05:57 -04:00
iomgaa 8c383354c5 fix(detector): 移除 ongoing_actions/visible_entities 空字段检测 + 新增 repair sh
抽检确认这两个字段为空是合法内容状态(静物/黑帧/模糊帧),
VLM 重修也修不好,保留会导致断点续跑死循环。
L3 空字段检测缩减为 frame_summary + spatial_layout。
2026-07-09 00:54:27 -04:00
iomgaa 86573735aa docs(wiki): apply Codex review fixes to main-figure design 2026-07-09 00:46:08 -04:00
iomgaa 8cb1158a2d docs(wiki): paper main-figure design — self-evolving loop layout 2026-07-09 00:39:45 -04:00
iomgaa d6bcf41336 fix(repair): Codex 审查修正 — progress 完成判据加严 + load_progress 防御增强
1. 保存 progress 前重新 detect_issues,只有 empty_field 清零才记 finished
   (修复 L2/L1 空字段被检测但未修复仍写 finished 的问题)
2. load_progress 增加 AttributeError 捕获(防 JSON 非 dict 形态崩溃)
2026-07-09 00:30:23 -04:00
iomgaa 4612123ec4 style: ruff format telemetry + repair_trees, remove unused SRTEntry import 2026-07-09 00:25:36 -04:00
iomgaa 847def4a03 feat(repair): asyncio.Semaphore 并发 + 断点续跑 + CLI 参数
--concurrency 默认 16,--reaggregate-all 强制全量重聚合。
Semaphore 限视频并发数,视频内四步串行。progress 文件
asyncio.Lock + os.replace 原子写入。熔断阈值 max(.env, concurrency*2)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 00:23:40 -04:00
iomgaa afe80a8b32 feat(repair): 断点续跑 progress 文件管理
load_progress / save_progress(asyncio.Lock + os.replace 原子写入)
/ should_skip_video。支持并发安全的读改写和 --reaggregate-all 兜底。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 00:23:04 -04:00
iomgaa 8182cb86b1 feat(detector): 扩展空字段检测到 L2 event_description / L1 scene_summary
断点续跑判据需要 L2/L1 层的 empty_field 检测。零 LLM 成本。
2026-07-09 00:12:55 -04:00
iomgaa f733c13dd1 fix(llm): call_id 移入重试循环,每 attempt 独立
消除重试时遥测主键冲突的根因。每次 attempt 独立记录,
parent_call_id 不受影响(循环外固定),更利于事后诊断重试轨迹。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 00:12:38 -04:00
iomgaa 5a91f392f0 fix(telemetry): INSERT OR IGNORE + WAL + try/except 三层防御加固
根治遥测写入主键冲突(UNIQUE constraint)和并发写锁(database is locked)
导致的异常冒泡,遥测侧信道错误不再污染 LLM 重试链。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-09 00:11:54 -04:00
iomgaa dbc9d38cd7 plan(tree/repair): 三项改造实现计划(遥测加固+断点续跑+并发)
6 个 Task: telemetry 防御加固 → call_id 根因修复 → detector L2/L1 扩展
→ progress 管理 → 并发编排+CLI → lint+全量测试
2026-07-09 00:08:23 -04:00
iomgaa 8a49ef18e6 design(tree/repair): 建树修复管线熔断根因修复+断点续跑+并发改造设计
三事: (1)遥测防御性加固(INSERT OR IGNORE+WAL+busy_timeout+try/except不冒泡)
根治call_id重试复用导致的主键冲突污染重试与熔断器; (2)判据双轨
(detect重扫empty_field+progress文件)视频级断点续跑,首次用detect
自动接历史无需手补状态; (3)asyncio.Semaphore并发默认16,熔断阈值
max(5,concurrency*2),--concurrency走CLI避免与tree.concurrency串台
2026-07-08 23:52:48 -04:00
iomgaa 53286ac3ab refactor: remove deprecated retriever module
RecursiveRetriever was a failed approach, not carried into TRM5.
- delete app/retriever/ (empty placeholder)
- drop retriever + train blocks from config/default.yaml
- renumber fidelity checklist 13->12 items (drop #4, shift up)
- sync core-goal text, dir tree, module-interaction diagrams
  across CLAUDE.md, ARCHITECTURE.md, overview.md, README.md
- reference/ kept intact as historical code
2026-07-08 23:09:30 -04:00
iomgaa 91281cb995 merge: feat/app-harness — app/harness/ 训练循环编排层(14 文件, 算法保真 #6/#10/#13) 2026-07-07 20:35:04 -04:00
iomgaa 9a42bbcf85 style(harness): ruff format batching.py + log.py 2026-07-07 20:34:48 -04:00
iomgaa a668fc173c feat(harness): __init__.py public API + lint fixes 2026-07-07 13:45:48 -04:00
iomgaa d6e74f2734 feat(harness): runner.py — train loop orchestrator (#13 algorithm fidelity)
Three-level nesting (epoch -> step -> per-skill), slow update 10-step
sequence, checkpoint/resume, early stop, probation accept/reject/rollback.

Key TRM4->TRM5 changes:
- sync -> async (all inference/diagnosis/evolve/validate awaited)
- LLMClient.from_env -> injected LLMProvider (DI via constructor)
- Direct DB/file access -> module functions (workspace/store/log)
- _TrainState as train() local, explicit param passing to helpers

Module-level pure functions extracted for testability:
resume_plan, _guard_infra_failures, _apply_batch_correctness,
_compute_total_steps, _should_early_stop, _format_applied_edits,
_fallback_summary, _write_skip_report, _outcome_to_quadrant_pairs,
_build_comparison_pairs, _batch_from_ids, _snapshot_current_skills.

Tests: 34 unit tests covering 13a-13e sub-tasks.
Radon: all functions Grade B or better.
2026-07-07 13:43:20 -04:00
iomgaa 6baddcc17d feat(harness): validate.py — async 块序贯验证编排 + Probation 统一定义 2026-07-07 13:20:43 -04:00
iomgaa 8c9f7d2d37 feat(harness): checkpoint.py — TrainState 序列化 + 原子写 + 指纹校验 2026-07-07 13:13:37 -04:00
iomgaa 6b56fc8f98 feat(harness): inference.py — async run_inference + DI 2026-07-07 13:04:26 -04:00
iomgaa 49e272719e feat(harness): momentum.py — async 慢更新动量生成 2026-07-07 13:01:04 -04:00
iomgaa 767ac55787 feat(harness): observation.py — 五张观测表 + step/epoch 报告 2026-07-07 12:55:12 -04:00
iomgaa d7d8633dae feat(harness): gate_ladder.py — 信息阶梯 + BaselineCache (#6 算法保真) 2026-07-07 12:55:12 -04:00
iomgaa 48b423ef35 feat(harness): pools.py — 三池切分(test→validation→diagnosis) 2026-07-07 12:48:54 -04:00
iomgaa ad8fc176d0 feat(harness): batching.py — FFD + round-robin mini-batch (#10 算法保真) 2026-07-07 12:47:33 -04:00
iomgaa 461abf0329 feat(harness): workspace.py — Workspace lifecycle + VersionedSkillStore/PromptStore
- ResolvedPaths frozen dataclass: store_dir, videos_dir, questions_dir,
  skills_dir, prompts_dir, workspace_dir, db_path, analyses_dir, runs_dir
- init_workspace: create ws + copy seed weights from store
- init_workspace_from_seed: create from seed with fail-fast questions check
- load_manifest / resolve_paths: manifest I/O + path resolution
  (skills/prompts resolve to workspace, videos/questions to store)
- update_manifest: key whitelist validation
- record_run: idempotent history append + per-video wiki dirs
- read_best / update_best: best pointer independent of current
- list_video_ids: videos with tree.json
- archive_workspace: move to .archive/<name>-<ts>
- VersionedSkillStore: implements core/evolution/protocols.py::SkillStore
- VersionedPromptStore: implements core/evolution/protocols.py::PromptStore
- 21 tests all passing (incl. Protocol compliance checks)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:41:59 -04:00
iomgaa 6858da80c7 feat(harness): store.py — Store 版本操作 + Seed 管理
从 TRM4 core/workspace.py 拆出 Store + Seed 相关函数:
- _parse_version / list_versions / next_version / advance_version
- _write_meta / init_store
- init_seed / list_seeds / read_seed
- extract_run_db(保留原始 CREATE 语句重建主键约束)
- promote_to_seed(强校验版本一致 + 非 NULL + finally 清理)

26 个测试全部通过,radon 复杂度 A (2.83)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:36:36 -04:00
iomgaa fd76720dcf feat(harness): HarnessLog SQLite wrapper + RunLogImpl readonly port
- HarnessLog: TRM4 direct port with WAL mode, threading.Lock, INSERT OR
  IGNORE idempotent _runs, context manager (completed/failed), create_table
  with auto run_id+timestamp, insert/insert_many/execute/query/log_event
- RunLogImpl: implements core/evolution/protocols.py::RunLog Protocol with
  independent sqlite3.connect for read-only SELECT (no _runs pollution),
  asyncio.to_thread wrapping for async interface
- _read_table: shared readonly helper with optional question_ids filtering,
  graceful empty-list return for missing tables
- Tests: 17 cases covering thread safety, idempotent inserts, context
  manager status, WAL mode, protocol compliance, readonly isolation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:29:29 -04:00
iomgaa b47f634e61 fix(harness): Codex functional review 修复 — 命名/集成测试/delta_min/promote 消息
- _apply_env_overrides → _apply_env_var_overrides,docstring 明确从 os.environ 读取
- 新增 TestLoadConfigRealYaml:用真实 config/default.yaml 验证嵌套 harness 解析
- 新增 test_delta_min_negative_rejected:覆盖 gate_delta_min >= 0 校验
- 恢复 promote 模式独立错误消息(从合并分支分离回 TRM4 原始提示)
- 77 个单元测试全部通过,radon 全部 Grade B 或更好

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:23:27 -04:00
iomgaa 535bd320b0 fix(harness): 补充 train 模式 run_id 校验 + 拆分函数保持 radon Grade B
- _validate_mode_deps: 恢复 train 非 resume/fresh 时必须提供 run_id 校验
- 提取 _validate_train_run_id 用 early return 展平条件,避免 radon Grade C
- 合并 promote run_id 检查到 diagnose/evolve/promote 统一检查
- 新增 4 个测试:train+run_id / train+resume / train+fresh / train+baseline
- radon cc -n C 无输出(全部 Grade B 或更好)
- 74 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:14:28 -04:00
iomgaa 5e57cf19b6 refactor(harness): 拆分校验函数降低 radon 圈复杂度至 Grade B
- _validate: 拆出 _validate_mode(mode 依赖校验)+ _validate_basic(标量/枚举校验)
- _validate_gate: 拆为 _validate_gate_thresholds(e 值/净胜/方向)+ _validate_gate_ladder(阶梯/块序贯)
- load_config: 提取 _apply_env_overrides 函数
- radon cc -n C 无输出(全部 Grade B 或更好)
- 70 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:01:46 -04:00
iomgaa ca02b5a91d feat(harness): RunConfig frozen dataclass + 四层校验 + YAML/CLI/.env 三层加载
- RunConfig: 46 字段 frozen dataclass,从 TRM4 core/harness/config.py 迁移
- 四层校验链:_validate → _validate_edit_budget + _validate_minibatch + _validate_gate
- 新增 .env 覆盖层:工程配置(workspace_dir, store_dir)可通过 HARNESS_* 环境变量注入
- 合并优先级:CLI > .env > YAML(CLAUDE.md §4.5)
- load_config 支持嵌套 harness 段和扁平 YAML 两种格式
- run_id 改为默认空字符串(CLI-only 字段,YAML 不提供)
- resume/fresh 互斥校验不在 config 层(移至 runner.py)
- 70 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 11:57:35 -04:00
iomgaa 63ad68f737 feat(evolution): export dataclass types from __init__.py 2026-07-07 11:48:13 -04:00
iomgaa 2db612e0bb docs: app/harness/ implementation plan — 15 tasks, algorithm fidelity #6/#10/#13, 120 TRM4 tips 2026-07-07 11:45:38 -04:00
iomgaa af8081e9ed docs: app/harness/ design — 14-module training loop orchestration layer 2026-07-07 11:28:29 -04:00
iomgaa 283ea27b51 merge: feat/tree-index — core/evolution/ extractable kernel + tree/search/question_gen modules 2026-07-07 10:29:54 -04:00
iomgaa 9ab6a89298 feat(evolution): __init__.py public API + ARCHITECTURE.md Protocol update 2026-07-07 10:28:14 -04:00
iomgaa 5c42e0309b feat(evolution): evolve.py per-target evolution — skill/system/tool (#9) 2026-07-07 10:26:58 -04:00
iomgaa 2c5b4c819c feat(evolution): evolve.py validation + helpers (#9) 2026-07-07 10:12:15 -04:00
iomgaa c6a81f8504 feat(evolution): diagnose.py aggregation + case packs + run_diagnosis (#8) 2026-07-07 10:04:48 -04:00
iomgaa fdbf9222ab feat(evolution): diagnose.py metrics + attribution (Stage 1, #8) 2026-07-07 09:55:22 -04:00
iomgaa ebd83810aa feat(evolution): validate.py — pure block validation decision functions (#7) 2026-07-07 09:47:45 -04:00
iomgaa 7f8ed56428 feat(evolution): patch.py — 补丁引擎移植(算法 #9),51 测试全通过
从 TRM4 core/harness/patch.py (427 行) 零修改移植到 core/evolution/patch.py。

包含:
- 7 个常量(APPENDIX/MOMENTUM marker + MAX_CHARS + HEADING)
- appendix 区:追加/提取/替换/边界检测,损坏态 ValueError
- momentum 区:替换/提取/边界检测,注入防护
- apply_patch_with_report:4 种 op(append/insert_after/replace/delete)
- 冻结区坐标判定(每条 edit 重算)、report 1-based index
- 51 个单元测试覆盖全部公共 API 及边缘场景

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:44:05 -04:00
iomgaa 57793fb0f8 feat(evolution): gate.py — CE-Gate e-process 纯函数 (#5 算法保真)
从 TRM4 core/harness/eprocess.py 逐行迁移,零逻辑变更:
- compute_e_value: 截断 Beta 混合 e 值(log 空间 + betainc 对称性)
- gate_decision: 四出口优先级链(confirmed→directional→futility→exhaustion→continue)
- probation_verdict: 试用期非对称双向结算
- 常量保真: _WALD_WIN=ln1.4, _WALD_LOSS=ln0.6, _SHRINK_PSEUDO=4

仅变更: import 路径 + GateParams/GateVerdict 移至 types.py + 中文 docstring

14 tests 覆盖: e 值数学、边界校验、四出口路径、试用期三分支

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:38:21 -04:00
iomgaa 47c2de71a2 feat(core/evolution): protocols.py + types.py 基础层 — 18 个 dataclass + 3 个 Protocol
TRM4 诊断/进化/门控数据类型迁移至 TRM5 Clean Architecture 内核。
逐字段比对 TRM4 的 eprocess.py、diagnose.py、evolve.py 保真迁移。

types.py (18 个 dataclass):
- Gate: GateParams, GateVerdict (frozen, 原样迁移)
- 诊断: SpanMetrics, SkillStepAdherence, QuestionMetrics,
  ErrorAttribution, CaseSample, SkillCasePack, SystemCasePack,
  ToolCasePack, DiagnosisResult (全部 frozen=True)
- 进化: EvolutionRecord (mutable), RejectedEdit (frozen),
  EvolutionResult (frozen, 移除 skills_version/prompts_version)
- 新增: PairResult, QuadrantClassification (块验证纯决策输出)
- 新增: DiagnosePrompts, EvolvePrompts (模板束, frozen)

protocols.py (3 个只读 Protocol):
- SkillStore, PromptStore (同步文件读取)
- RunLog (异步日志查询, 隔离 SQL)

变更理由:
- QuestionMetrics 由 TRM4 mutable 改为 frozen (一次性构造)
- ErrorAttribution 由 TRM4 mutable 改为 frozen (构造时填入全部字段)
- EvolutionResult 移除版本管理字段 (app/ 职责)

涉及算法: #5(CE-Gate), #8(诊断瀑布), #9(进化引擎)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 09:34:04 -04:00
iomgaa 8bbfbc9b8a docs: core/evolution/ implementation plan — 9 tasks, TDD, algorithm fidelity checks 2026-07-07 07:55:21 -04:00
iomgaa 2761ca3e82 docs: core/evolution/ extractable kernel design — gate, patch, diagnose, evolve 2026-07-07 07:39:24 -04:00
iomgaa dc19e9e007 docs: 同步 app/search/ 模块结构 + OCRProvider 签名到 ARCHITECTURE.md 2026-07-07 06:14:26 -04:00
iomgaa c42813992f feat(search): __init__.py 公开 API + 修复 OCR 测试 asyncio 兼容性 2026-07-07 06:13:25 -04:00
iomgaa 502681694b feat(search): PromptManager — 搜索 Agent 提示词加载与组装
从 TRM4 core/search/prompt.py 迁移。有意变更:
- 工具描述从 app.search.tools.get_tool_descriptions 获取
- format_user_prompt 参数显式化(question/options/l1_node_ids/task_type)

16 个单元测试覆盖 __init__、build_inference_prompt(auto/manual/none
三种 skill_mode)、format_user_prompt、load。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 06:11:13 -04:00
iomgaa 96e63a3e19 feat(search): 实现 SearchToolDispatcher 工具调度器 (Task 7)
- 新增 app/search/tools.py:
  - get_tool_descriptions() 工具描述文本(与 TRM4 一致)
  - SearchToolDispatcher 类实现 ToolDispatcher Protocol
  - dispatch() 按工具名路由: view_node / search_similar /
    observe_frame / submit_answer / read_skill
  - ValueError(未知工具)上抛,KeyError/FileNotFoundError 捕获返回错误文本
  - view_node: env.get_node_text + summarize_node + get_children_info + summarize_children
  - search_similar: env.search_similar + summarize_nodes_batch
  - observe_frame: env.resolve_frame_paths + get_subtitle + observe_frame + 字幕前置

- 修复 app/tree/environment.py get_children_info():
  - 原实现返回 _format_time_range (str) 导致 summarize_children 解包失败
  - 改为返回原始数值元组 via 新增 _node_time_range_raw 静态方法

- 新增 tests/unit/test_search_tools.py (14 tests):
  - get_tool_descriptions 含/不含 read_skill
  - 五种工具 dispatch 路由验证
  - 未知工具 ValueError + 节点不存在错误文本

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 06:07:27 -04:00
iomgaa 3fe077a249 style: format summarizer.py 2026-07-07 05:55:40 -04:00
iomgaa b8706adb33 style: format vision.py 2026-07-07 05:53:33 -04:00
iomgaa d83e720c4b feat(search): app/search/vision.py — 两轮 VLM 帧观察模块
从 TRM4 core/tree/vision.py 迁移 observe_frame,关键变更:
- VLM 调用走 VLMProvider.chat_with_images Protocol(images 传 Path)
- OCR 调用走 OCRProvider.transcribe_frames 异步 Protocol
- 遥测字段 session_id / parent_call_id 透传
- 帧文件存在性前置校验

12 个单元测试覆盖:两轮正常、仅提取、OCR 注入/失败降级/None、
VLM 提取失败、VLM 验证失败降级、帧缺失、stats 完整性、
分歧/弃权标记、遥测透传。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:52:54 -04:00
iomgaa 224c75ba9e feat(adapters): OCRProvider Protocol + MonkeyOCRClient 异步适配器
- app/ports.py: 新增 OCRProvider Protocol(runtime_checkable,与
  EmbeddingProvider 同级),定义 async transcribe_frames 端口
- adapters/ocr.py: 从 TRM4 core/tree/ocr.py 保真迁移 MonkeyOCRClient
  - assert → ValueError(P5 防御性校验)
  - 公开方法改 async(asyncio.to_thread 包装同步 HTTP)
  - 内部逻辑不变:多端点轮询、线程安全 Session、单帧降级、行去重
- tests/unit/test_ocr_adapter.py: 17 个测试覆盖 Protocol 合规、
  构造校验、健康检查、转录、降级、去重、轮询

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:47:55 -04:00
iomgaa 83e6475f90 feat(search): app/search/skills.py — 技能注册表与 frontmatter 解析
从 TRM4 core/search/skills.py 保真迁移。提供 parse_frontmatter、
strip_frontmatter、SkillRegistry、discover_skills 四个公共 API。
逻辑完全一致,仅调整导入路径并添加中文 docstring。

17 个单元测试全部通过,覆盖正常/异常/边界场景。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:46:25 -04:00
iomgaa f0a49b08e5 style: format environment.py 2026-07-07 05:46:25 -04:00
iomgaa 2bc5dc74c2 feat(tree): add get_node_text + get_children_info to TreeEnvironment
- get_node_text(node_id, anchor=False): returns raw text + optional
  anchor_map dict by parsing [cN]/[sN] prefixes from anchored text
- get_children_info(node_id): returns structured child list with
  id/time_range/summary (description truncated to 120 chars)
- Both methods reuse existing internal helpers (_node_full_text,
  _node_anchored_text, _get_children, _node_description,
  _format_time_range)
- 9 new test cases across TestGetNodeText and TestGetChildrenInfo

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 05:45:48 -04:00
iomgaa 112876de2e chore: 复制 TRM4 v2 prompt 种子文件(9 个,字节级一致)
.gitignore 添加 !store/prompts/ 例外——prompt 是版本化资源需提交
2026-07-07 05:42:11 -04:00
iomgaa e36f7c63f3 docs(plan): 恢复被 wiki 工具覆盖的计划全文 2026-07-07 05:39:21 -04:00
iomgaa e20bfac48d docs(plan): app/search/ 搜索 Agent 装配层实现计划(10 Task) 2026-07-07 05:36:25 -04:00
iomgaa 961e85111b docs(design): 修复 Codex 审查发现 — 10 项修正
Critical: OCRProvider 移至 app/ports.py;新增 TreeEnvironment 结构化 API;
         observe_frame 补充字幕上下文拼接;search_similar 补充节点文本获取
Important: 遥测链路透传 session_id/parent_call_id;异常降级边界明确化;
           verify_vision/anchor/assemble_mode 改为必传;Prompt 路径逐文件列出;
           测试目录规范化到 tests/unit/
Minor: 依赖图补全 VLMProvider 连线
2026-07-07 05:29:52 -04:00
iomgaa bc18fc8635 docs(design): 补充 summarizer 模块 — view_node/search_similar 的 LLM 两轮摘要
自审发现 TRM4 三个搜索工具全有 question-conditioned 中间摘要层,
原设计遗漏了 summarizer.py。新增:
- §4.4 summarizer API(summarize_node / _children / _nodes_batch)
- anchor 锚模式完整保留
- 6 个额外 prompt 文件
- 更新交互流程图、错误处理、测试策略
2026-07-07 05:22:55 -04:00
iomgaa 5ab5adec51 docs(design): 搜索 Agent 装配层设计(app/search/)
方案 A 平铺模块:prompt.py / skills.py / tools.py / vision.py
新增 OCRProvider Protocol + adapters/ocr.py
Prompt 从 TRM4 store/prompts/v2/ 原封不动复制
2026-07-07 05:15:22 -04:00
iomgaa 253f36797b docs: 同步 question_gen 模块路径到 ARCHITECTURE.md 和 CLAUDE.md
DataLoader 代码位置 generator.py → loader.py;
目录树更新 question_gen/ 结构反映实际文件。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:51:26 -04:00
iomgaa f94c352d66 feat(question_gen): QuestionGenerator Protocol + 模块公开 API
app/ports.py 追加 QuestionGenerator Protocol(预留 LLM 出题接口)。
app/question_gen/__init__.py re-export load_benchmark 和 stratified_sample。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:48:21 -04:00
iomgaa c4d42eeca0 feat(question_gen): stratified_sample — 分层采样 + 题型保底
算法 100% 保真 TRM4: task_types 过滤、correctness.get(id, False) 语义、
对题在前返回顺序、min_per_class 遍历 pool 全部题型(含稀疏类)。
所有参数显式传入,无默认值。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:45:19 -04:00
iomgaa 8f5fbf8d2d feat(question_gen): load_benchmark — benchmark JSON 加载
从 JSON 目录 glob *.json 加载题目,stem 作 video_id。
legacy schema 无 difficulty 字段时赋 _LEGACY_DEFAULT_DIFFICULTY 常量。
options/source_nodes 转 tuple 配合 frozen dataclass。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:41:25 -04:00
iomgaa 9bdfd28c63 feat(core): 追加 GeneratedQuestion frozen dataclass
跨层共享类型,被 core/evolution/ 和 app/harness/、app/question_gen/ 使用。
frozen=True + tuple 字段确保不可变。无默认值(显式传入)。

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 04:36:56 -04:00
iomgaa 15fad65b1b feat(tree/repair): 检测器扩展 — visible_entities/ongoing_actions/spatial_layout 为空也触发修复 2026-07-07 03:15:42 -04:00
iomgaa faf6f1bd63 fix(tools): handle None card in flat tree conversion 2026-07-07 03:13:38 -04:00
iomgaa c61ceca538 test(tree): 建树模块端到端集成测试
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:44:28 -04:00
iomgaa 8cfdc841da feat(tree/repair): Q&A 反向补全 — 从 TRM4 supplement 迁移
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:37:50 -04:00
iomgaa 79761852df feat(tree): VideoTreeBuilder 保真 #1 #2 #3 + 复杂度重构
- 从 reference/video_tree_trm/video_tree_builder.py (994行) 迁移
- 保真算法 #1: L2 轴心建树策略 (asyncio.gather 链式并发)
- 保真算法 #2: VLM 批量帧描述 + JSON fallback (_L3_BATCH_SIZE=5)
- 保真算法 #3: 断点续跑 (progress.json + L1 中间 JSON)
- 新增: VLMProvider/LLMProvider Protocol 替代 LLMClient
- 新增: 结构化 JSON 输出 → L1Card/L2Card/L3Card
- 新增: L2 代表帧复用 L3 帧 (_sample_representative_frames)
- 新增: 字幕注入 + Voronoi 分配
- 重构: 提取 _load_resume_state/_assemble_roots 降低 _build_async 复杂度 D(21)→C(14)
- 44 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 02:17:50 -04:00
iomgaa af2aa751b9 feat(tree): 质量校验 — 交叉验证 entities/visible_text
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:57:32 -04:00
iomgaa e87a744df9 feat(tree): 字幕模块 — SRT 解析 + 完整性检查 + Voronoi 分配
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:54:09 -04:00
iomgaa 3e348f5b21 feat(adapters): GovernedVLMClient — VLMProvider 最小可用实现
- 包装 GovernedLLMClient,注入 base64 图片到 OpenAI Vision API 格式
- 复用 LLM 治理栈全部能力(熔断、缓存、重试、遥测)
- 8 项单元测试覆盖协议满足、图片编码、注入逻辑、不可变性
2026-07-07 01:49:55 -04:00
iomgaa 6fd9522bf3 feat(adapters): EmbeddingProvider Protocol + local/remote 双后端实现
- app/ports.py: 添加 EmbeddingProvider Protocol(runtime_checkable,dim 属性 + embed 方法)
- adapters/embedding.py: 从参考代码迁移,拆分为 LocalEmbeddingProvider 和 RemoteEmbeddingProvider
  - Local: sentence-transformers 冻结推理,维度校验
  - Remote: OpenAI 兼容 API,L2 归一化,按 index 排序
  - 两者均提供 embed() 和 embed_tensor() 统一接口
- tests/unit/test_embedding_adapter.py: Protocol 满足性、形状校验、导入测试

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:45:09 -04:00
iomgaa 5ec8dd2f82 feat(tree): TreeConfig 配置 dataclass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:41:50 -04:00
iomgaa 1b2442b9c5 fix(tree): from_dict 增加 ID 唯一性校验
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:40:46 -04:00
iomgaa c4bc5e7cc6 refactor(tree): extract _embed_l2_subtree to reduce embed_all complexity
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:37:09 -04:00
iomgaa fd58a3f02b feat(tree): TreeIndex 数据结构 — Card 体系 + 节点 + 序列化
- 新增三级 frozen Card dataclass: L3Card(6字段), L2Card(7字段), L1Card(7字段)
- 节点重构: L3Node/L2Node/L1Node 使用 Card 替代原始字符串字段
- 添加 @property 兼容层: description/summary 代理到 Card 字段
- L3Node 新增 subtitle 字段(字幕集成预留)
- JSON 序列化/反序列化支持 Card 结构 + embedding base64 编解码
- load_json 新增 ID 唯一性校验(重复 ID 抛 ValueError)
- 移除 pickle 序列化(仅保留 JSON)
- 日志从 log_msg 迁移到 loguru
- 17 个单元测试全部通过

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 01:35:10 -04:00
iomgaa d95b2a3142 docs(tree): add tree module vertical slice design and implementation plan 2026-07-07 01:27:36 -04:00
iomgaa adf14633e5 chore: lint 修复(TC001/TC003 类型导入优化)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-06 23:15:06 -04:00
407 changed files with 101506 additions and 237 deletions
+9 -1
View File
@@ -22,6 +22,7 @@ Every project goes through this process. A todo list, a single-function utility,
You MUST create a task for each of these items and complete them in order:
1. **Explore project context** — check files, docs, recent commits
1.5. **Prior-version audit (mandatory for rewrites/refactors)** — if the task replaces or rewrites an existing module, list every behavior of the old version (including persistence, crash recovery, idempotency, resume) and confirm each is kept, replaced, or deliberately dropped. Undocumented implicit drops = bugs.
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
@@ -92,7 +93,12 @@ digraph brainstorming {
- Once you believe you understand what you're building, present the design
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
- Ask after each section whether it looks right so far
- Cover: architecture, components, data flow, error handling, testing
- Cover: architecture, components, data flow, error handling, testing, **non-functional requirements** (see below)
- **Non-functional requirements (mandatory section):** Every design MUST explicitly address these four dimensions — even if the answer is "not applicable":
- **Persistence strategy:** When does data hit disk? How much is lost on crash? Overwrite or append?
- **Idempotency:** Is the same operation safe to repeat? Does it produce the same result?
- **Resume/checkpoint:** Can the process recover from interruption? How is progress persisted?
- **Atomicity:** Are writes atomic? Can a partial write corrupt data?
- Be ready to go back and clarify if something doesn't make sense
**Design for isolation and clarity:**
@@ -124,6 +130,8 @@ After writing the spec document, look at it with fresh eyes:
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
5. **Non-functional coverage:** Does the design explicitly address persistence, idempotency, resume, and atomicity? If any dimension is missing, add it now — even if the answer is "not applicable."
6. **Prior-version regression check (rewrites only):** If this replaces an existing module, confirm every behavior from the prior-version audit (step 1.5) is accounted for in the design. Any gap = a spec bug.
Fix any issues inline. No need to re-review — just fix and move on.
+15 -11
View File
@@ -4,28 +4,28 @@ NO_PROXY=dashscope.aliyuncs.com,api.deepseek.com
# ── 搜索 Agent LLM ──
SEARCH_LLM_MODEL=deepseek-v4-pro
SEARCH_LLM_BASE_URL=https://api.deepseek.com/v1
SEARCH_LLM_API_KEY=sk-xxx
SEARCH_LLM_BASE_URL=https://newapi.iomgaa.online/v1
SEARCH_LLM_API_KEY=sk-your-api-key-here
# ── 评估 Judge LLM ──
JUDGE_LLM_MODEL=deepseek-v4-pro
JUDGE_LLM_BASE_URL=https://api.deepseek.com/v1
JUDGE_LLM_API_KEY=sk-xxx
JUDGE_LLM_BASE_URL=https://newapi.iomgaa.online/v1
JUDGE_LLM_API_KEY=sk-your-api-key-here
# ── 视觉模型(Qwen VL)──
VL_LLM_MODEL=qwen3.6-plus
VL_LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
VL_LLM_API_KEY=sk-xxx
VL_LLM_BASE_URL=https://newapi.iomgaa.online/v1
VL_LLM_API_KEY=sk-your-api-key-here
# ── 进化 LLM(Prompt 改写)──
EVOLVE_LLM_MODEL=deepseek-v4-pro
EVOLVE_LLM_BASE_URL=https://api.deepseek.com/v1
EVOLVE_LLM_API_KEY=sk-xxx
EVOLVE_LLM_BASE_URL=https://newapi.iomgaa.online/v1
EVOLVE_LLM_API_KEY=sk-your-api-key-here
# ── ASR 字幕生成(Groq Whisper)──
ASR_MODEL=whisper-large-v3
ASR_BASE_URL=https://api.groq.com/openai/v1
ASR_API_KEY=gsk-xxx
ASR_API_KEY=gsk_your-groq-api-key-here
# ── MonkeyOCR ──
MONKEY_OCR_URLS=http://10.77.0.20:7866,http://10.77.0.20:7867
@@ -41,9 +41,13 @@ REDIS_URL=redis://localhost:6379/0
LLM_TIMEOUT=120
LLM_MAX_RETRIES=3
LLM_RETRY_BASE_DELAY=2.0
LLM_CIRCUIT_BREAKER_THRESHOLD=5
LLM_CIRCUIT_BREAKER_THRESHOLD=48 # 实际阈值 = max(此值, concurrency*2)
LLM_CIRCUIT_BREAKER_COOLDOWN=60
LLM_TTFT_TIMEOUT=30
LLM_INTER_TOKEN_TIMEOUT=15
LLM_RETRY_MAX_DELAY=30.0
REDIS_CACHE_TTL=86400
# 正整数秒,禁止 0(0 会被拒绝启动);训练场景建议 >= 单次训练时长
REDIS_CACHE_TTL=604800
# 建树批量并行:全局 VLM/LLM 在途调用上限(Spec-2 工程配置)
TREE_BUILD_API_CONCURRENCY=16
+2
View File
@@ -181,6 +181,8 @@ pencil/
# 数据与实验产物(不提交)
store/
!store/prompts/
!store/skills/
workspaces/
results/
+29 -17
View File
@@ -6,7 +6,7 @@
> 2. 你的所有思考过程和回复必须使用 **简体中文**。
## 1. 项目元数据 (Metadata)
- **核心目标**: 在层次化视频树上构建可自我进化的搜索 Agent + 可训练的递归检索器(RecursiveRetriever,通过 Harness Engineering(工具、技能、记忆、中间件)的持续改进实现长视频理解;服务于科研产出。详见 `research-wiki/ARCHITECTURE.md``README.md`
- **核心目标**: 在层次化视频树上构建可自我进化的搜索 Agent,通过 Harness Engineering(工具、技能、记忆、中间件)的持续改进实现长视频理解;服务于科研产出。详见 `research-wiki/ARCHITECTURE.md``README.md`
- **项目类型**: 科研工程混合体 + 生产级(非 MVP)
- **目标会议**: AAAI 20262026年6月25日)
- **后端架构**: Python 3.11Clean Architecture 四层分层,详见 `research-wiki/ARCHITECTURE.md §2`
@@ -19,7 +19,7 @@
| PyTorch 概念 | 本项目对应 | 代码位置 |
|-------------|-----------|----------|
| `DataLoader` | 出题 question_gen | `app/question_gen/generator.py` |
| `DataLoader` | 出题 question_gen | `app/question_gen/loader.py` |
| `model.forward()` | 推理 inference | `app/harness/inference.py` + `core/agent/loop.py` |
| `loss.backward()` | 诊断 diagnose | `core/evolution/diagnose.py` |
| `optimizer.step()` | 进化 evolve | `core/evolution/evolve.py` |
@@ -96,9 +96,10 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
### Phase 1: 规划与设计 (Planning)
1. **需求探索**: 涉及创建新功能、新组件、修改行为时,**必须**先调用 `brainstorming` skill 进行需求探索与设计。无论用户的指令多么具体、改动多么简单,都不得跳过此步骤(除非用户显式说"跳过 brainstorming")。
2. **查阅规格 & 讨论**: 仔细阅读 `research-wiki/`(单一事实源)下对应的文档,了解项目最新情况。对于不理解的地方请与人类进行多轮讨论,确保理解人类的设计意图。
3. **日志方案设计**: 功能会产生运行时数据时,**必须**调用 `structured-logging` skill 设计日志方案
4. **撰写计划**: 正式编码前**必须**调用 `writing-plans` skill 撰写实现计划
5. **审核门控(差异化)**:
3. **前序版本对照(重写/重构时强制)**: 当任务涉及重写或重构已有模块时,**必须**列出前序版本的所有行为(包括持久化策略、崩溃恢复、幂等性、断点续跑等非功能性行为),逐一确认新版本是保留、替代、还是删除。未经确认的隐式删除 = bug
4. **日志方案设计**: 功能会产生运行时数据时**必须**调用 `structured-logging` skill 设计日志方案
5. **撰写计划**: 正式编码前,**必须**调用 `writing-plans` skill 撰写实现计划。
6. **审核门控(差异化)**:
- **designClaude 自审 → Codex 审 → 人类审**(保留人类门,批准后方可进入计划阶段)。
- **planClaude 自审 → Codex 审 → 直接执行**(无 plan 人类门);plan 经 Claude 自审 + Codex 审通过后直接进入 Phase 2 执行。
@@ -159,6 +160,19 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
- **功能修改**:
- **必须** 不考虑向后兼容,直接修改原文件。代码简洁性优先。
### 4.2.1 设计文档非功能性需求覆盖(强制)
> **教训来源**: v2 出题管线重写时未继承 v1 的逐题追加持久化策略,导致多次 run 的题目丢失。
设计文档**必须**显式覆盖以下非功能性维度(即使答案是"不适用"也要写明):
| 维度 | 必答问题 |
|------|---------|
| **持久化策略** | 数据何时落盘?崩溃时最多丢多少?是覆盖写还是追加? |
| **幂等性** | 同一操作重复执行是否安全?结果是否一致? |
| **断点续跑** | 中断后重启能否从断点恢复?进度如何持久化? |
| **原子性** | 写操作是否原子?部分写入是否会损坏数据? |
### 4.3 Git 工作流规范
- **Feature Branch**: 所有开发工作在 feature 分支上进行,**严禁**直接在 main/master 上修改。
- **增量提交**: 频繁提交,每个提交有明确的语义。
@@ -221,23 +235,22 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
### 4.7 核心算法保真
迁移时逐一比对参考代码,不可简化。完整清单见 `research-wiki/ARCHITECTURE.md §6`(建树 4 项 + 训练 9 项 = 13 项)。
迁移时逐一比对参考代码,不可简化。完整清单见 `research-wiki/ARCHITECTURE.md §6`(建树 4 项 + 训练 8 项 = 12 项)。
| # | 算法 | 核心逻辑 |
|---|------|---------|
| 1 | L2 轴心建树策略 | L2 先行→L3 向下→L1 向上,asyncio 链式并发 |
| 2 | VLM 批量帧描述 + JSON fallback | `_L3_BATCH_SIZE=5` 批量调用,解析失败逐帧 fallback |
| 3 | 断点续跑机制 | `progress.json` + L1 中间 JSON,按段恢复 |
| 4 | RecursiveRetriever | Cross-Attention 选择器 + ACT halt + z 状态累积 |
| 5 | CE-Gate e-process | 截断 Beta 混合、四出口门控 |
| 6 | 信息阶梯 | 冷启动 2:1、gamma-EMA、反泄漏 |
| 7 | 块顺序验证 | 基线缓存、INFRA 护栏、配对翻转 |
| 8 | 诊断瀑布 | 错误归因级联、缺陷 vs 失误、D1-D5 |
| 9 | 进化 patch 引擎 | 保护跨度、rank-and-clip、附录/动量 |
| 10 | Mini-batch 构建 | FFD + round-robin + 正确率混合 |
| 11 | Agent Loop | Thinking+JSON、json_repair、pluggy hook |
| 12 | 树环境语义搜索 | 分块 embedding、祖先去重、锚定验证 |
| 13 | 训练循环编排 | 三级嵌套、慢更新10步、断点续训 |
| 4 | CE-Gate e-process | 截断 Beta 混合、四出口门控 |
| 5 | 信息阶梯 | 冷启动 2:1、gamma-EMA、反泄漏 |
| 6 | 块顺序验证 | 基线缓存、INFRA 护栏、配对翻转 |
| 7 | 诊断瀑布 | 错误归因级联、缺陷 vs 失误、D1-D5 |
| 8 | 进化 patch 引擎 | 保护跨度、rank-and-clip、附录/动量 |
| 9 | Mini-batch 构建 | FFD + round-robin + 正确率混合 |
| 10 | Agent Loop | Thinking+JSON、json_repair、pluggy hook |
| 11 | 树环境语义搜索 | 分块 embedding、祖先去重、锚定验证 |
| 12 | 训练循环编排 | 三级嵌套、慢更新10步、断点续训 |
> **任何 PR 涉及上述算法的修改,必须在 commit message 中标注对应序号并说明变更理由。**
@@ -301,7 +314,6 @@ project_root/
│ ├── harness/ # 训练 harnessrunner, inference, batching
│ ├── question_gen/ # 新题构建
│ ├── search/ # 搜索 Agent 装配(prompt, skills
│ ├── retriever/ # 可训练检索器(RecursiveRetriever
│ └── ports.py # 应用层端口
├── adapters/ # 外部实现层(LLM/VLM/embedding/cache/遥测)
+1 -1
View File
@@ -1,6 +1,6 @@
# Video-Tree-TRM5
> 在层次化视频树上构建可自我进化的搜索 Agent 与可训练递归检索器,实现长视频理解。目标会议:EMNLP 2026。
> 在层次化视频树上构建可自我进化的搜索 Agent,实现长视频理解。目标会议:EMNLP 2026。
## 系统概览
+183
View File
@@ -0,0 +1,183 @@
"""SqliteDiagnosisSignalStorebaseline 逐题诊断信号的 SQLite 持久化适配器。
实现 core/evolution/protocols.py::DiagnosisSignalStore 端口。信号行以
(question_id, baseline_run_id, diag_fingerprint) 为主键,INSERT OR REPLACE
保证逐题幂等 upsert;bool 字段以 0/1 存储,可空字段以 NULL 存储。表建在
harness.db,供离线诊断编排写入、视频级切分选择器读取。
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
from typing import TYPE_CHECKING
from core.evolution.types import DiagnosisSignalRow
if TYPE_CHECKING:
from types import TracebackType
# 表列顺序即 DiagnosisSignalRow 字段顺序(question_id..session_id),
# upsert 写入与 load 还原共用,避免手写列名两处漂移。
_COLUMNS: tuple[str, ...] = (
"question_id",
"video_id",
"baseline_run_id",
"diag_fingerprint",
"task_type",
"error_type",
"cause_category",
"tier",
"evolution_target",
"degraded",
"infra",
"session_id",
)
class SqliteDiagnosisSignalStore:
"""逐题诊断信号的 SQLite 存储实现。
构造时按需建表(CREATE TABLE IF NOT EXISTS),主键
(question_id, baseline_run_id, diag_fingerprint) 保证同键覆盖。
每次写操作单事务 commit,保证原子落盘与断点续跑。
参数:
db_path: SQLite 数据库文件路径(通常为 harness.db)。
"""
def __init__(self, db_path: str) -> None:
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
self._conn = sqlite3.connect(db_path)
self._conn.row_factory = sqlite3.Row
self._init_table()
def _init_table(self) -> None:
"""创建 baseline_diagnosis 表(若不存在)。"""
self._conn.execute(
"""
CREATE TABLE IF NOT EXISTS baseline_diagnosis (
question_id TEXT NOT NULL,
video_id TEXT NOT NULL,
baseline_run_id TEXT NOT NULL,
diag_fingerprint TEXT NOT NULL,
task_type TEXT NOT NULL,
error_type TEXT,
cause_category TEXT,
tier TEXT NOT NULL,
evolution_target TEXT,
degraded INTEGER NOT NULL,
infra INTEGER NOT NULL,
session_id TEXT,
PRIMARY KEY (question_id, baseline_run_id, diag_fingerprint)
)
"""
)
self._conn.commit()
def upsert(self, row: DiagnosisSignalRow) -> None:
"""写入或覆盖单题诊断信号(按主键幂等,单事务)。
参数:
row: 待持久化的诊断信号行。bool 字段转 0/1None 存 NULL。
关键实现:
用 INSERT OR REPLACE 按主键覆盖,避免重复行;commit 保证原子。
"""
placeholders = ", ".join("?" for _ in _COLUMNS)
col_names = ", ".join(_COLUMNS)
values = (
row.question_id,
row.video_id,
row.baseline_run_id,
row.diag_fingerprint,
row.task_type,
row.error_type,
row.cause_category,
row.tier,
row.evolution_target,
int(row.degraded),
int(row.infra),
row.session_id,
)
self._conn.execute(
f"INSERT OR REPLACE INTO baseline_diagnosis ({col_names}) VALUES ({placeholders})",
values,
)
self._conn.commit()
def done_question_ids(
self,
baseline_run_id: str,
diag_fingerprint: str,
*,
retry_uncertain: bool = False,
) -> set[str]:
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
参数:
baseline_run_id: baseline run 标识。
diag_fingerprint: 诊断口径指纹。
retry_uncertain: True 时追加 `AND tier != 'uncertain'`,把 uncertain
(信号不可信降级)题排除出已完成集,令其被重新诊断;默认 False。
返回:
已落盘信号的 question_id 集合;无匹配时为空集,供断点续跑跳过。
"""
sql = (
"SELECT DISTINCT question_id FROM baseline_diagnosis"
" WHERE baseline_run_id = ? AND diag_fingerprint = ?"
)
if retry_uncertain:
sql += " AND tier != 'uncertain'"
cursor = self._conn.execute(sql, (baseline_run_id, diag_fingerprint))
return {r["question_id"] for r in cursor.fetchall()}
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
"""加载指定 run 与诊断指纹下的全部诊断信号行。
参数:
baseline_run_id: baseline run 标识。
diag_fingerprint: 诊断口径指纹。
返回:
还原后的 DiagnosisSignalRow 列表(0/1→boolNULL→None)。
"""
col_names = ", ".join(_COLUMNS)
cursor = self._conn.execute(
f"SELECT {col_names} FROM baseline_diagnosis"
" WHERE baseline_run_id = ? AND diag_fingerprint = ?",
(baseline_run_id, diag_fingerprint),
)
return [
DiagnosisSignalRow(
question_id=r["question_id"],
video_id=r["video_id"],
baseline_run_id=r["baseline_run_id"],
diag_fingerprint=r["diag_fingerprint"],
task_type=r["task_type"],
error_type=r["error_type"],
cause_category=r["cause_category"],
tier=r["tier"],
evolution_target=r["evolution_target"],
degraded=bool(r["degraded"]),
infra=bool(r["infra"]),
session_id=r["session_id"],
)
for r in cursor.fetchall()
]
def close(self) -> None:
"""关闭底层 SQLite 连接,释放文件描述符与锁。"""
self._conn.close()
def __enter__(self) -> SqliteDiagnosisSignalStore:
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None:
self.close()
+18 -3
View File
@@ -20,21 +20,32 @@ class CircuitBreaker:
self._cooldown_s = cooldown_s
self._fails: dict[str, int] = {}
self._open_until: dict[str, float] = {}
self._half_open_inflight: dict[str, bool] = {}
def is_open(self, source_name: str, now: float) -> bool:
"""判断指定源是否处于开路状态。
冷却截止时刻之前为开路;到期返回 False(放行一个试探,即半开)。
冷却截止时刻之前为开路;到期进入半开,**只放行一个探针**(其余仍被挡),
避免冷却到期瞬间惊群重连再次压垮上游。"检查+标记探针"在 asyncio 单线程内
同步执行,天然原子无竞态。
Args:
source_name: 被熔断的源标识。
now: 当前时刻(秒级时间戳),由调用方注入。
Returns:
True 表示开路(拒绝请求),False 表示关闭或半开放行
True 表示开路(拒绝请求),False 表示关闭或半开放行探针
"""
until = self._open_until.get(source_name)
return until is not None and now < until
if until is None:
return False
if now < until:
return True # 冷却中,全挡
# 冷却到期:half-open,只放行一个探针
if self._half_open_inflight.get(source_name):
return True # 已有探针在途,继续挡
self._half_open_inflight[source_name] = True
return False
def record_failure(self, source_name: str, now: float) -> None:
"""记录一次失败;累计达阈值则开路至 now + cooldown。
@@ -47,6 +58,8 @@ class CircuitBreaker:
self._fails[source_name] = count
if count >= self._fail_threshold:
self._open_until[source_name] = now + self._cooldown_s
# 探针失败清在途标记,使下一轮 cooldown 到期后可再放行探针
self._half_open_inflight.pop(source_name, None)
def force_open(self, source_name: str, now: float) -> None:
"""强制开路(用于 401/403 等不可恢复错误),一次即熔断。
@@ -59,6 +72,7 @@ class CircuitBreaker:
"""
self._fails[source_name] = self._fail_threshold
self._open_until[source_name] = now + self._cooldown_s
self._half_open_inflight.pop(source_name, None)
def record_success(self, source_name: str) -> None:
"""记录一次成功;清零失败计数与开路状态(关闭熔断器)。
@@ -68,3 +82,4 @@ class CircuitBreaker:
"""
self._fails.pop(source_name, None)
self._open_until.pop(source_name, None)
self._half_open_inflight.pop(source_name, None)
+184
View File
@@ -0,0 +1,184 @@
"""嵌入适配器 —— local/remote 双后端实现。
封装文本嵌入器,支持本地 sentence-transformers 和远程 OpenAI 兼容 API 两种后端。
提供统一的 ``embed()`` / ``embed_tensor()`` 接口,冻结不训练。
两个类均满足 ``app.ports.EmbeddingProvider`` Protocol。
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import numpy as np
import torch
from loguru import logger
if TYPE_CHECKING:
from numpy import ndarray
from torch import Tensor
class LocalEmbeddingProvider:
"""本地 sentence-transformers 嵌入器(冻结)。
使用 HuggingFace sentence-transformers 加载模型进行本地推理,
所有参数冻结,仅用于嵌入提取。
属性:
dim: 嵌入维度 D。
"""
def __init__(self, model_name: str, embed_dim: int, device: str = "cpu") -> None:
"""初始化本地嵌入模型。
参数:
model_name: HuggingFace 模型名称(如 'BAAI/bge-base-zh-v1.5')。
embed_dim: 期望的嵌入维度。
device: 推理设备('cpu' / 'cuda' 等)。
异常:
AssertionError: 模型实际维度与 embed_dim 不一致。
"""
from sentence_transformers import SentenceTransformer
self._dim = embed_dim
self._model = SentenceTransformer(model_name, device=device)
self._model.eval()
# 冻结所有参数
for param in self._model.parameters():
param.requires_grad = False
actual_dim = self._model.get_sentence_embedding_dimension()
assert actual_dim == self._dim, (
f"模型实际维度 ({actual_dim}) 与配置 embed_dim ({self._dim}) 不一致"
)
logger.info("本地嵌入模型初始化完成", model=model_name, device=device)
# ------------------------------------------------------------------
# 公共接口
# ------------------------------------------------------------------
@property
def dim(self) -> int:
"""嵌入维度 D。"""
return self._dim
def embed(self, texts: str | list[str]) -> ndarray:
"""文本 → 嵌入向量(L2 归一化)。
参数:
texts: 单条文本或文本列表。
返回:
[N, D] ndarray,每行 L2 范数为 1.0。单条文本时 N=1。
"""
if isinstance(texts, str):
texts = [texts]
with torch.no_grad():
embeddings = self._model.encode(
texts,
normalize_embeddings=True,
convert_to_numpy=True,
)
# sentence-transformers encode 返回 ndarray [N, D]
if embeddings.ndim == 1:
embeddings = embeddings.reshape(1, -1)
return embeddings
def embed_tensor(self, texts: str | list[str]) -> Tensor:
"""文本 → 嵌入 Tensor(L2 归一化)。
参数:
texts: 单条文本或文本列表。
返回:
[N, D] torch.Tensorfloat32)。
"""
arr = self.embed(texts)
return torch.from_numpy(arr).float()
class RemoteEmbeddingProvider:
"""远程 OpenAI 兼容 API 嵌入器。
通过 OpenAI 兼容 API(如 GPUStack)调用远程嵌入模型。
属性:
dim: 嵌入维度 D。
"""
def __init__(self, model_name: str, embed_dim: int, api_key: str, api_url: str) -> None:
"""初始化远程嵌入客户端。
参数:
model_name: 远程模型名称。
embed_dim: 期望的嵌入维度。
api_key: API 密钥。
api_url: API 基础 URL。
异常:
ValueError: api_key 或 api_url 为空。
"""
if not api_key:
raise ValueError("远程模式必须提供 api_key")
if not api_url:
raise ValueError("远程模式必须提供 api_url")
from openai import OpenAI
self._dim = embed_dim
self._model_name = model_name
self._client = OpenAI(base_url=api_url, api_key=api_key)
logger.info("远程嵌入客户端初始化完成", model=model_name, api_url=api_url)
# ------------------------------------------------------------------
# 公共接口
# ------------------------------------------------------------------
@property
def dim(self) -> int:
"""嵌入维度 D。"""
return self._dim
def embed(self, texts: str | list[str]) -> ndarray:
"""文本 → 嵌入向量(L2 归一化)。
参数:
texts: 单条文本或文本列表。
返回:
[N, D] ndarray,每行 L2 范数为 1.0。单条文本时 N=1。
"""
if isinstance(texts, str):
texts = [texts]
response = self._client.embeddings.create(
model=self._model_name,
input=texts,
)
# 按 index 排序,确保顺序一致
sorted_data = sorted(response.data, key=lambda x: x.index)
embeddings = np.array([item.embedding for item in sorted_data], dtype=np.float32)
# L2 归一化
norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
norms = np.maximum(norms, 1e-12) # 避免除零
embeddings = embeddings / norms
return embeddings
def embed_tensor(self, texts: str | list[str]) -> Tensor:
"""文本 → 嵌入 Tensor(L2 归一化)。
参数:
texts: 单条文本或文本列表。
返回:
[N, D] torch.Tensorfloat32)。
"""
arr = self.embed(texts)
return torch.from_numpy(arr).float()
+28 -13
View File
@@ -179,7 +179,10 @@ def _is_transient_error(exc: Exception) -> bool:
返回:
True 表示可重试,False 表示不可重试。
"""
if isinstance(exc, (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout)):
# 两族基类覆盖断连族:TimeoutExceptionConnectTimeout/ReadTimeout/WriteTimeout/PoolTimeout
# 与 TransportErrorConnectError/ReadError/RemoteProtocolError 等)。
# 注意 HTTPStatusError 非 TransportError 子类,401/403 致命分支不受影响。
if isinstance(exc, (httpx.TimeoutException, httpx.TransportError)):
return True
if isinstance(exc, httpx.HTTPStatusError):
return exc.response.status_code in _TRANSIENT_STATUS_CODES
@@ -274,6 +277,7 @@ class GovernedLLMClient:
*,
session_id: str | None = None,
parent_call_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse:
"""发起 LLM 调用,经四层治理栈:熔断 → 缓存 → 重试+流式 → 遥测。
@@ -281,6 +285,7 @@ class GovernedLLMClient:
messages: OpenAI 格式消息列表。
session_id: 会话 ID(传递到遥测)。
parent_call_id: 父调用 ID(传递到遥测)。
cache_salt: 可选缓存盐,透传到 Redis 缓存键(如跨 epoch 重采样)。
返回:
LLMResponse 统一响应。
@@ -295,12 +300,14 @@ class GovernedLLMClient:
if self._breaker.is_open(self._provider, time.monotonic()):
raise CircuitOpenError(f"熔断器已开启,拒绝调用 provider={self._provider}")
# ② call_id 生成
call_id = str(uuid4())
# ③ 缓存查询(cache 为 None 时跳过)
cached = await self._cache.get(self._model, messages) if self._cache is not None else None
# ② 缓存查询(cache 为 None 时跳过)— call_id 在缓存路径独立生成
cached = (
await self._cache.get(self._model, messages, cache_salt)
if self._cache is not None
else None
)
if cached is not None:
cache_call_id = str(uuid4())
response = LLMResponse(
content=cached.content,
thinking=cached.thinking,
@@ -312,10 +319,10 @@ class GovernedLLMClient:
ttft_ms=None,
max_inter_token_ms=None,
cache_hit=True,
call_id=call_id,
call_id=cache_call_id,
)
await self._telemetry.record_llm_call(
call_id=call_id,
call_id=cache_call_id,
parent_call_id=parent_call_id,
session_id=session_id,
model_name=self._model,
@@ -333,9 +340,10 @@ class GovernedLLMClient:
)
return response
# 重试循环 + 流式消费
# 重试循环 + 流式消费(每次 attempt 独立 call_id
last_exc: Exception | None = None
for attempt in range(self._max_retries):
call_id = str(uuid4())
attempt_start = time.monotonic()
try:
content, thinking_text, ttft_ms, max_itoken_ms, usage = await self._call_streaming(
@@ -371,7 +379,7 @@ class GovernedLLMClient:
# ④ 写缓存(cache 为 None 时跳过)
if self._cache is not None:
await self._cache.set(self._model, messages, response)
await self._cache.set(self._model, messages, response, cache_salt)
# ⑤ 遥测
await self._telemetry.record_llm_call(
@@ -450,12 +458,15 @@ class GovernedLLMClient:
},
)
if attempt < self._max_retries - 1:
delay = min(
import random
base = min(
self._retry_base_delay_s * (2**attempt),
self._retry_max_delay_s,
)
if delay > 0:
await asyncio.sleep(delay)
jitter = base * random.uniform(0.5, 1.5)
if jitter > 0:
await asyncio.sleep(jitter)
continue
# 非瞬时、非致命 → 记遥测后直接抛出
@@ -570,6 +581,10 @@ class GovernedLLMClient:
else:
thinking_parts.append(text)
# 流耗尽但未收 [DONE] → 服务端截断,视为可重试的 SSE 异常(不写缓存/不当成功)
if not usage_sink.get("done"):
raise _SseAnomaly("truncated_no_done")
content = "".join(content_parts)
thinking = "".join(thinking_parts)
usage = usage_sink.get("usage", {})
+128
View File
@@ -0,0 +1,128 @@
"""MonkeyOCR HTTP 客户端 — 帧文字转录的异构硬证据源。
服务由用户在 LAN 部署(双端点轮询);请求必须绕过代理(trust_env=False)。
实现 OCRProvider Protocolapp/ports.py)。
"""
from __future__ import annotations
import asyncio
import itertools
import threading
from pathlib import Path # noqa: TC003 — 运行时需要(方法签名 + open())
import requests
from loguru import logger
_TIMEOUT_S = 300
class MonkeyOCRClient:
"""MonkeyOCR 服务客户端:多端点轮询、单帧失败降级为跳过。
关键实现细节:实例可被多线程共享——端点轮询加锁、Session 线程局部
(A/B 评测会以 4 线程并发调用同一实例)。
参数:
urls: 服务端点列表(如 ["http://10.77.0.20:7866", ...]),非空。
异常:
ValueError: urls 为空时抛出。
"""
def __init__(self, urls: list[str]) -> None:
if not urls:
raise ValueError("MonkeyOCR 端点列表不能为空")
self._urls = [u.rstrip("/") for u in urls]
self._rr = itertools.cycle(self._urls)
self._rr_lock = threading.Lock()
self._local = threading.local()
def _get_session(self) -> requests.Session:
"""返回当前线程专属的 Session(惰性创建并复用,trust_env=False 绕代理)。"""
session = getattr(self._local, "session", None)
if session is None:
session = requests.Session()
session.trust_env = False # LAN 直连,绕过代理
self._local.session = session
return session
def _check_health_sync(self) -> None:
"""同步预检所有端点,任一不可达即抛错(供 asyncio.to_thread 调用)。
异常:
RuntimeError: 端点不可达或 /health 非 2xx。
"""
for url in self._urls:
try:
resp = self._get_session().get(f"{url}/health", timeout=5)
except requests.RequestException as e:
raise RuntimeError(f"MonkeyOCR 端点不可达: {url}: {e}") from e
if not resp.ok:
raise RuntimeError(f"MonkeyOCR 健康检查失败: {url}: {resp.status_code}")
async def check_health(self) -> None:
"""异步预检所有端点,任一不可达即抛错(A/B qtr_ocr 臂启动门)。
异常:
RuntimeError: 端点不可达或 /health 非 2xx。
"""
await asyncio.to_thread(self._check_health_sync)
def _transcribe_frames_sync(self, frame_paths: list[Path]) -> str:
"""同步逐帧转录并拼接(供 asyncio.to_thread 调用)。
参数:
frame_paths: 帧文件路径列表。
返回:
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本;无任何有效结果时空串。
"""
parts: list[str] = []
for i, path in enumerate(frame_paths, 1):
lines = self._transcribe_one(path)
if lines:
parts.append(f"{i}: " + " | ".join(lines))
return "\n".join(parts)
async def transcribe_frames(self, frame_paths: list[Path]) -> str:
"""异步逐帧转录并拼接为注入文本;单帧失败跳过,全失败返回空串。
参数:
frame_paths: 帧文件路径列表。
返回:
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本;无任何有效结果时空串。
"""
return await asyncio.to_thread(self._transcribe_frames_sync, frame_paths)
def _transcribe_one(self, path: Path) -> list[str]:
"""单帧转录:空结果/单字符行过滤 + 帧内行级去重。
参数:
path: 帧文件路径。
返回:
过滤去重后的文本行列表;请求失败或无有效行时空列表。
"""
with self._rr_lock:
url = next(self._rr)
try:
with open(path, "rb") as f:
resp = self._get_session().post(
f"{url}/ocr/text", files={"file": f}, timeout=_TIMEOUT_S
)
resp.raise_for_status()
content = resp.json().get("content", "")
except (requests.RequestException, ValueError) as e:
logger.warning("MonkeyOCR 单帧转录失败,跳过 {}: {}", path.name, e)
return []
seen: set[str] = set()
lines: list[str] = []
for ln in content.splitlines():
ln = ln.strip()
if len(ln) <= 1 or ln in seen:
continue
seen.add(ln)
lines.append(ln)
return lines
+47 -14
View File
@@ -12,6 +12,26 @@ from loguru import logger
from core.types import LLMResponse
def _resolve_cache_ttl(ttl: int) -> int:
"""校验 Redis 缓存 TTL:必须为正整数(消灭 0=永不过期 的隐式语义)。
Args:
ttl: 待校验的 TTL 秒数。
Returns:
校验通过的正整数 TTL。
Raises:
ValueError: ttl <= 0。
"""
if ttl <= 0:
raise ValueError(
f"REDIS_CACHE_TTL 必须为正整数秒,实际 {ttl}"
"训练场景建议 >= 单次训练时长(如 86400)。"
)
return ttl
class RedisResponseCache:
"""基于 Redis 的 LLM 响应缓存。
@@ -22,45 +42,55 @@ class RedisResponseCache:
Args:
redis: 异步 Redis 客户端实例(duck-typed,需支持 get/set 方法)。
ttl_s: 缓存过期时间(秒)。
ttl_s: 缓存过期时间(秒)。None 表示永不过期。
"""
def __init__(self, redis: Any, ttl_s: int) -> None:
def __init__(self, redis: Any, ttl_s: int | None) -> None:
self._redis = redis
self._ttl_s = ttl_s
def _build_key(self, model: str, messages: list[dict[str, str]]) -> str:
def _build_key(
self,
model: str,
messages: list[dict[str, str]],
cache_salt: str | None = None,
) -> str:
"""构造 content-addressed 缓存键。
Args:
model: 模型名称。
messages: 消息列表。
cache_salt: 可选缓存盐(如跨 epoch 强制重采样)。仅当非 None 时才加入
键 payload,保证默认 None 时键结构与旧缓存一字节不差、旧键不失效。
Returns:
sha256 哈希字符串作为 Redis 键。
"""
payload = json.dumps(
{"model": model, "messages": messages},
sort_keys=True,
ensure_ascii=False,
)
key_obj: dict[str, Any] = {"model": model, "messages": messages}
if cache_salt is not None:
key_obj["salt"] = cache_salt
payload = json.dumps(key_obj, sort_keys=True, ensure_ascii=False)
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
return f"llm_cache:{digest}"
async def get(
self, model: str, messages: list[dict[str, str]]
self,
model: str,
messages: list[dict[str, str]],
cache_salt: str | None = None,
) -> LLMResponse | None:
"""从缓存读取 LLM 响应。
Args:
model: 模型名称。
messages: 消息列表。
cache_salt: 可选缓存盐,透传到键构造。
Returns:
缓存命中时返回 LLMResponse,未命中或 Redis 异常时返回 None。
"""
try:
key = self._build_key(model, messages)
key = self._build_key(model, messages, cache_salt)
raw = await self._redis.get(key)
except Exception:
logger.warning("Redis 缓存读取失败,降级为未命中")
@@ -77,6 +107,7 @@ class RedisResponseCache:
model: str,
messages: list[dict[str, str]],
response: LLMResponse,
cache_salt: str | None = None,
) -> None:
"""将 LLM 响应写入缓存。
@@ -84,12 +115,14 @@ class RedisResponseCache:
model: 模型名称。
messages: 消息列表。
response: 待缓存的 LLMResponse。
cache_salt: 可选缓存盐,透传到键构造。
"""
try:
key = self._build_key(model, messages)
value = json.dumps(
dataclasses.asdict(response), ensure_ascii=False
)
key = self._build_key(model, messages, cache_salt)
value = json.dumps(dataclasses.asdict(response), ensure_ascii=False)
if self._ttl_s:
await self._redis.set(key, value, ex=self._ttl_s)
else:
await self._redis.set(key, value)
except Exception:
logger.warning("Redis 缓存写入失败,跳过缓存")
+60 -17
View File
@@ -1,14 +1,22 @@
"""SQLite 遥测记录器 — TelemetryRecorder Protocol 的生产实现。
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,
确保事件循环不被阻塞。表在首次写入时懒初始化。
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,确保事件循环不被阻塞。
构造时建单持久连接 + 建表(对齐 app/harness/log.py:HarnessLog 的并发写模式),
写入经进程内 threading.Lock 串行化,消除多连接并发写的 database is locked。
零丢失保证范围 = 单进程、单 recorder 实例(当前 main.py / video_split_cli 均单实例
注入)。同进程多个 recorder 指向同一 db 会退回跨连接竞争——本实现不支持该场景。
"""
from __future__ import annotations
import asyncio
import sqlite3
import threading
from pathlib import Path
from loguru import logger
class SQLiteTelemetryRecorder:
"""基于 SQLite 的 LLM 调用遥测记录器。
@@ -41,7 +49,7 @@ class SQLiteTelemetryRecorder:
"""
_INSERT_SQL = """
INSERT INTO llm_calls (
INSERT OR IGNORE INTO llm_calls (
call_id, parent_call_id, session_id,
model_name, provider, messages, response, thinking,
prompt_tokens, completion_tokens, latency_ms,
@@ -51,16 +59,43 @@ class SQLiteTelemetryRecorder:
"""
def __init__(self, db_path: Path) -> None:
self._db_path = db_path
self._table_ready = False
"""建单持久连接 + 进程内 Lock(对齐 app/harness/log.py:HarnessLog 并发写模式)。
def _ensure_table(self, conn: sqlite3.Connection) -> None:
"""懒初始化:首次写入时创建 llm_calls 表。"""
if self._table_ready:
return
把并发控制拉到进程内(threading.Lock 串行化写),消除"每次新连接并发写同一
db、靠 SQLite busy_timeout 跨连接协调"在高频下撑爆 timeout → database is locked
的根因。check_same_thread=Falserecord_llm_call 经 asyncio.to_thread 在线程池
不同线程调用,共享连接跨线程访问需此 flag,串行性由 self._lock 保证。
遥测哲学(P5):连接初始化失败降级不冒泡(self._conn=None,写入直接丢弃 warning),
绝不因遥测故障拖垮 LLM 调用 / 训练。
"""
self._db_path = db_path
self._lock = threading.Lock()
self._conn: sqlite3.Connection | None = None
# mkdir / connect / PRAGMA / 建表统一纳入降级边界:任一失败(OSError 含
# PermissionError、sqlite3.Error)都降级为 self._conn=None,绝不冒泡拖垮初始化。
try:
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(db_path), check_same_thread=False)
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=5000")
conn.execute(self._CREATE_TABLE_SQL)
conn.commit()
self._table_ready = True
self._conn = conn
except (OSError, sqlite3.Error) as exc:
logger.warning("遥测连接初始化失败(已降级,后续写入丢弃): {}", exc)
self._conn = None
def close(self) -> None:
"""幂等关闭持久连接(对齐 HarnessLog;进程退出前可选调以释放 fd)。
不调也不丢数据——每次 _write 已 commit 落 WAL,进程退出 OS 回收 fd、
WAL 已提交内容下次打开自动 checkpoint 恢复。
"""
with self._lock:
if self._conn is not None:
self._conn.close()
self._conn = None
def _write(
self,
@@ -81,11 +116,19 @@ class SQLiteTelemetryRecorder:
cache_hit: bool,
error: str | None,
) -> None:
"""同步写入一条 LLM 调用记录到 SQLite。"""
conn = sqlite3.connect(str(self._db_path))
"""同步写入一条 LLM 调用记录(单持久连接 + Lock 串行化,对齐 HarnessLog)。
三层防御:
1. INSERT OR IGNORE — call_id 主键冲突静默忽略(幂等)
2. 进程内 threading.Lock 串行化写 — 消除并发锁竞争(非依赖 SQLite busy_timeout
3. try/except sqlite3.Error — DB 错误降级不冒泡,遥测失败绝不拖垮 LLM 调用
"""
if self._conn is None:
logger.warning("遥测连接不可用(已降级),丢弃 call_id={}", call_id)
return
try:
self._ensure_table(conn)
conn.execute(
with self._lock:
self._conn.execute(
self._INSERT_SQL,
(
call_id,
@@ -105,9 +148,9 @@ class SQLiteTelemetryRecorder:
error,
),
)
conn.commit()
finally:
conn.close()
self._conn.commit()
except sqlite3.Error as exc:
logger.warning("遥测写入失败(已降级),call_id={}: {}", call_id, exc)
async def record_llm_call(
self,
+131
View File
@@ -0,0 +1,131 @@
"""GovernedVLMClient -- VLMProvider 最小可用实现。
将图片编码为 base64,构造 OpenAI Vision API 格式的 messages
委托给已有的 GovernedLLMClient 发送。复用 LLM 治理栈的全部能力
(熔断、缓存、重试、遥测)。
"""
from __future__ import annotations
import base64
import mimetypes
from pathlib import Path
from typing import TYPE_CHECKING, Any
from loguru import logger
if TYPE_CHECKING:
from adapters.llm import GovernedLLMClient
from core.types import LLMResponse
class GovernedVLMClient:
"""VLMProvider 实现——包装 GovernedLLMClient,注入 base64 图片。
参数:
governed_llm: 已初始化的 GovernedLLMClient 实例。
"""
def __init__(self, governed_llm: GovernedLLMClient) -> None:
self._llm = governed_llm
async def chat_with_images(
self,
messages: list[dict[str, Any]],
images: list[str | Path],
*,
session_id: str | None = None,
parent_call_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse:
"""图文调用:将图片编码为 base64 嵌入 messages,委托给 LLM 客户端。
参数:
messages: 对话消息列表。最后一条 user message 的 content 会被扩展为
包含图片的多模态格式。
images: 图片文件路径列表。
session_id: 会话 ID(遥测用)。
parent_call_id: 父调用 ID(遥测用)。
cache_salt: 可选缓存盐,透传到底层 LLM 缓存键。
返回:
LLMResponse。
"""
vision_messages = self._inject_images(messages, images)
return await self._llm.chat(
vision_messages,
session_id=session_id,
parent_call_id=parent_call_id,
cache_salt=cache_salt,
)
@staticmethod
def _encode_image(image_path: str | Path) -> str:
"""将图片文件编码为 base64 data URL。
参数:
image_path: 图片文件路径。
返回:
data:image/<type>;base64,<data> 格式的字符串。
"""
path = Path(image_path)
mime_type = mimetypes.guess_type(str(path))[0] or "image/jpeg"
with open(path, "rb") as f:
b64 = base64.b64encode(f.read()).decode("utf-8")
return f"data:{mime_type};base64,{b64}"
@staticmethod
def _inject_images(
messages: list[dict[str, Any]],
images: list[str | Path],
) -> list[dict[str, Any]]:
"""将图片注入最后一条 user message,构造 OpenAI Vision API 格式。
参数:
messages: 原始消息列表。
images: 图片路径列表。
返回:
新消息列表(不修改原列表)。
"""
if not images:
return messages
result = [m.copy() for m in messages]
# 找到最后一条 user message
last_user_idx = -1
for i in range(len(result) - 1, -1, -1):
if result[i].get("role") == "user":
last_user_idx = i
break
if last_user_idx == -1:
logger.warning("messages 中无 user 角色消息,图片未注入")
return result
user_msg = result[last_user_idx]
original_content = user_msg.get("content", "")
# 构造多模态 content
content_parts: list[dict[str, Any]] = []
# 图片在前
for img_path in images:
data_url = GovernedVLMClient._encode_image(img_path)
content_parts.append(
{
"type": "image_url",
"image_url": {"url": data_url},
}
)
# 文本在后
if isinstance(original_content, str) and original_content:
content_parts.append({"type": "text", "text": original_content})
elif isinstance(original_content, list):
content_parts.extend(original_content)
result[last_user_idx] = {**user_msg, "content": content_parts}
return result
+36
View File
@@ -0,0 +1,36 @@
"""app/harness/ — 训练循环编排层。
组合 core/evolution/(决策内核)+ core/agent/AgentLoop+ adapters/LLM/VLM/telemetry),
实现自进化闭环的训练循环三级嵌套、块序贯验证、快慢双速进化、checkpoint/resume。
"""
from app.harness.config import RunConfig, load_config
from app.harness.inference import InferenceResult, run_inference
from app.harness.log import HarnessLog, RunLogImpl
from app.harness.pools import Pools, build_or_load_pools, build_pools, load_pools, save_pools
from app.harness.runner import Runner
from app.harness.workspace import (
ResolvedPaths,
VersionedPromptStore,
VersionedSkillStore,
resolve_paths,
)
__all__ = [
"HarnessLog",
"InferenceResult",
"Pools",
"ResolvedPaths",
"RunConfig",
"RunLogImpl",
"Runner",
"VersionedPromptStore",
"VersionedSkillStore",
"build_or_load_pools",
"build_pools",
"load_config",
"load_pools",
"resolve_paths",
"run_inference",
"save_pools",
]
+266
View File
@@ -0,0 +1,266 @@
"""离线诊断编排:把 baseline run 的错题诊断投影为逐题信号行并断点续跑落库。
"结果驱动视频级切分"离线管线的诊断步。给定一批可诊断错题:
1. 算 remaining(跳过 store 已完成题)实现续跑幂等;
2. 对剩余错题调 core.evolution.diagnose.run_diagnosis(经 StepsJsonRunLog
包装内层 RunLog,兼容 traces 未落表的历史 run);
3. 把 error_attributions / infra / degraded 三类产物确定性投影为
DiagnosisSignalRowtier 由 split_selection.score_signal 判定);
4. run 末(Phase 3)逐行 store.upsert 落库——诊断在 Phase 2 全部跑完后才落库,
故崩溃丢本次 run 未落库的全部结果(不是"仅一行");靠 GovernedLLMClient 的
Redis 缓存缓解重跑时的 LLM 重烧,下次调用命中缓存直接续。
错误处理诚实标注(不谎称全传播):
- run_diagnosis 的 C1/C2 阶段(指标计算、错误归因)网络/API 失败经
GovernedLLMClient 重试栈后仍失败会向上抛出,本编排不捕获、不掩盖,
直接冒泡给调用方。
- 但 C3 阶段(defect/lapse judge)的调用整体包在 `except Exception` 内
core/evolution/diagnose.py:2186),故 C3 judge 的**全部异常(含网络/API
失败)都被吞并→warning→默认归为 lapse**,不会向上抛;judge 语义歧义同样
按此保护性 fallback 处理。本编排原样接受该判定,不二次兜底、也不谎称
C3 阶段网络失败会传播。
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from loguru import logger
from app.harness.baseline_run_log import StepsJsonRunLog
from app.harness.split_selection import evolution_target_of, score_signal
from core.evolution.diagnose import run_diagnosis
from core.evolution.types import DiagnosisSignalRow
if TYPE_CHECKING:
from core.evolution.protocols import DiagnosisSignalStore
from core.evolution.types import DiagnosisResult
from core.types import GeneratedQuestion
@dataclass(frozen=True)
class DiagnosisDeps:
"""离线诊断编排的依赖束(一次编排的全部外部端口 + 运行参数)。
frozen 保证一次编排内依赖不可变;LLM/RunLog/SkillStore/prompts 走 Protocol
注入,便于测试替换成假实现。
属性:
run_log: 内层 RunLog 实现(提供 get_predictions/get_traces),
编排内部再用 StepsJsonRunLog 包装以兼容 traces 未落表的 run。
llm: LLM 调用端口(治理后的 GovernedLLMClient)。
skill_store: 技能文件读取端口。
prompts: 诊断模板束(DiagnosePrompts)。
tree_data: 树结构字典(多视频 {video_id: tree} 或单棵树),透传给 run_diagnosis。
concurrency: 诊断并发上限。
"""
run_log: Any
llm: Any
skill_store: Any
prompts: Any
tree_data: dict[str, Any]
concurrency: int
async def run_baseline_diagnosis(
*,
baseline_run_id: str,
diag_fingerprint: str,
wrong_ids: list[str],
questions: dict[str, GeneratedQuestion],
store: DiagnosisSignalStore,
deps: DiagnosisDeps,
retry_uncertain: bool = False,
) -> None:
"""对 baseline run 的错题跑离线诊断并把信号落库(断点续跑幂等)。
参数:
baseline_run_id: baseline run 标识(如 "infer_adhoc"),信号行主键之一。
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号,主键之一。
wrong_ids: 本次待诊断的可诊断错题 question_id 列表(保序)。
questions: question_id → GeneratedQuestion 映射,需覆盖 wrong_ids 全部题
及 run_diagnosis 返回的所有 infra/degraded 题(用于取 video_id/task_type)。
store: 诊断信号存储端口,upsert 落盘并提供 done_question_ids 续跑查询。
deps: 外部依赖束(见 DiagnosisDeps)。
retry_uncertain: True 时把已落 tier='uncertain'(信号不可信降级)的题也纳入
remaining 重新诊断,透传给 store.done_question_ids;默认 False。
返回:
None。副作用为把逐题 DiagnosisSignalRow 写入 store。
关键实现:
- remaining = wrong_ids 去除 store 已完成题;空则直接 return(续跑幂等,
重复调用零副作用)。
- run_diagnosis 只诊断 remaining,避免重复 LLM 调用浪费。
- 三类产物投影互斥落库:error_attributionsdefect/lapse)、infra_question_ids
T0)、degraded_question_idsuncertain)。
"""
# Phase 1: 算 remaining(续跑幂等)
done = store.done_question_ids(
baseline_run_id, diag_fingerprint, retry_uncertain=retry_uncertain
)
remaining = [qid for qid in wrong_ids if qid not in done]
if not remaining:
logger.info(
"离线诊断续跑:baseline={} fingerprint={} 无剩余错题(已完成 {} 题),跳过。",
baseline_run_id,
diag_fingerprint,
len(done),
)
return
logger.info(
"离线诊断开始:baseline={} fingerprint={} 剩余 {}/{} 题待诊断。",
baseline_run_id,
diag_fingerprint,
len(remaining),
len(wrong_ids),
)
# Phase 2: 对剩余错题跑诊断(StepsJsonRunLog 兼容 traces 未落表的历史 run)
result = await run_diagnosis(
baseline_run_id,
[questions[qid] for qid in remaining],
deps.tree_data,
deps.llm,
StepsJsonRunLog(deps.run_log),
deps.skill_store,
deps.prompts,
concurrency=deps.concurrency,
question_ids=list(remaining),
only_incorrect=True,
)
# Phase 3: 投影落库
counts = _project_and_persist(
result=result,
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
questions=questions,
store=store,
)
logger.info(
"离线诊断落库完成:baseline={} fingerprint={} "
"T2={} T1={} T0(infra)={} uncertain(degraded)={}{} 行。",
baseline_run_id,
diag_fingerprint,
counts["T2"],
counts["T1"],
counts["T0"],
counts["uncertain"],
sum(counts.values()),
)
def _project_and_persist(
*,
result: DiagnosisResult,
baseline_run_id: str,
diag_fingerprint: str,
questions: dict[str, GeneratedQuestion],
store: DiagnosisSignalStore,
) -> dict[str, int]:
"""把 DiagnosisResult 三类产物投影为信号行并逐行 upsert,返回各 tier 计数。
参数:
result: run_diagnosis 的返回,含 error_attributions/infra/degraded 三类产物。
baseline_run_id: 信号行主键之一。
diag_fingerprint: 信号行主键之一。
questions: question_id → GeneratedQuestion,用于取 video_id/task_type。
store: 诊断信号存储端口。
返回:
{tier: 行数} 计数字典(T2/T1/T0/uncertain),供上层日志与 manifest。
关键实现:
本函数在 run 末(Phase 3)逐行 upsert(单行单事务);诊断已在 Phase 2 全部
跑完,故本阶段中途崩溃丢本次 run 未落库的余下行。三桶**非互斥**:
同一 degraded 错题可能同时出现在 error_attributionsjudge 解析失败仍建
attribution)里,故按 **degraded > infra > attribution** 优先级去重——先落
degraded/infra,再在 attribution 循环跳过已落题,保证**每题恰写一行、
counts 恰计一次**(否则同 PK 覆盖会导致 counts 双计且分层错乱)。
"""
counts = {"T2": 0, "T1": 0, "T0": 0, "uncertain": 0}
# 优先级去重:degraded > infra > attribution。先记录高优先集合,
# attribution 循环遇到已落题即跳过,确保每题唯一落库。
persisted: set[str] = set()
# degraded_question_ids(最高优先):judge 解析失败降级 → uncertain,信号不可信排除出 T2
for qid in result.degraded_question_ids:
q = questions[qid]
store.upsert(
DiagnosisSignalRow(
question_id=qid,
video_id=q.video_id,
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
task_type=q.task_type,
error_type=None,
cause_category=None,
tier="uncertain",
evolution_target=None,
degraded=True,
infra=False,
session_id=None,
)
)
persisted.add(qid)
counts["uncertain"] += 1
# infra_question_ids:基础设施失败护栏排除 → T0,不参与训练主体
# (防御性跳过已落 degraded 题,虽 infra 通常已在诊断前过滤不重叠)
for qid in result.infra_question_ids:
if qid in persisted:
continue
q = questions[qid]
store.upsert(
DiagnosisSignalRow(
question_id=qid,
video_id=q.video_id,
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
task_type=q.task_type,
error_type=None,
cause_category=None,
tier="T0",
evolution_target=None,
degraded=False,
infra=True,
session_id=None,
)
)
persisted.add(qid)
counts["T0"] += 1
# error_attributions(最低优先):defect→T2 / lapse→T1 / 其它→uncertain(由 score_signal 判定)
# 跳过已作为 degraded/infra 落库的题,避免同 PK 覆盖与 counts 双计。
for ea in result.error_attributions:
if ea.question_id in persisted:
continue
q = questions[ea.question_id]
tier = score_signal(cause_category=ea.cause_category, infra=False, degraded=False).tier
# error_type 是 ErrorAttribution 必填字段(永远已知),确定性派生进化目标。
evolution_target = evolution_target_of(ea.error_type)
store.upsert(
DiagnosisSignalRow(
question_id=ea.question_id,
video_id=q.video_id,
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
task_type=q.task_type,
error_type=ea.error_type,
cause_category=ea.cause_category,
tier=tier,
evolution_target=evolution_target,
degraded=False,
infra=False,
session_id=None,
)
)
persisted.add(ea.question_id)
counts[tier] = counts.get(tier, 0) + 1
return counts
+71
View File
@@ -0,0 +1,71 @@
"""RunLog 包装器:traces 表空时从 predictions.steps_json 重建轨迹。
用于对 infer_adhoc 这类 traces 未落表、轨迹在 steps_json 的历史 run 跑离线诊断。
"""
from __future__ import annotations
from typing import Any
from app.harness.steps_json_traces import steps_json_to_trace_rows
class StepsJsonRunLog:
"""委托内层 RunLogget_traces 空表时回退 steps_json。
实现 core/evolution/protocols.py 的 RunLog Protocolduck-typing)。
对 traces 已落表的正常 run 完全透传;仅当底层 traces 为空时,
才从 predictions.steps_json 经 steps_json_to_trace_rows 重建轨迹行。
"""
def __init__(self, inner: Any) -> None:
"""构造包装器。
参数:
inner: 内层 RunLog 实现(如 app/harness/log.py::RunLogImpl),
需提供 get_predictions / get_traces 两个 async 方法。
"""
self._inner = inner
async def get_predictions(
self, run_id: str, *, question_ids: list[str] | None = None
) -> list[dict[str, Any]]:
"""透传内层预测查询。
参数:
run_id: 运行标识。
question_ids: 可选的题目 ID 过滤列表。
返回:
内层返回的预测记录字典列表,原样透传。
"""
return await self._inner.get_predictions(run_id, question_ids=question_ids)
async def get_traces(
self, run_id: str, *, question_ids: list[str] | None = None
) -> list[dict[str, Any]]:
"""查询轨迹;底层 traces 表空时从 steps_json 回退重建。
参数:
run_id: 运行标识。
question_ids: 可选的题目 ID 过滤列表。
返回:
轨迹行字典列表。
关键实现细节:
- 内层 traces 非空 → 原样返回,不触发回退(正常 run 路径)。
- 内层 traces 为空 → 拉取同一过滤条件下的 predictions
逐题经 steps_json_to_trace_rows 展开为轨迹行并拼接。
- steps_json 缺失时以空串传入,由下游确定性返回 []。
"""
inner_rows = await self._inner.get_traces(run_id, question_ids=question_ids)
if inner_rows:
return inner_rows
preds = await self._inner.get_predictions(run_id, question_ids=question_ids)
rows: list[dict[str, Any]] = []
for p in preds:
rows.extend(
steps_json_to_trace_rows(p["video_id"], p["question_id"], p.get("steps_json") or "")
)
return rows
+351
View File
@@ -0,0 +1,351 @@
"""混合 mini-batch 切分:以 QuestionUnit 为最小调度粒度,大类打散、小类整锁。
供 runner 每 step 处理一个 batch。孪生对(AR pair)作为 2 题单元整锁不拆、按单元级
正确性分桶;非 AR single 单元的抽样/洗牌 draw 流与"引入 QuestionUnit 前"的旧逐题算法
逐字节一致(AR 折叠不干扰非 AR draw 流)。
"""
from __future__ import annotations
import hashlib
import math
import random
from typing import TYPE_CHECKING
from app.harness.question_units import build_units, flatten_units, unit_correctness
if TYPE_CHECKING:
from core.types import GeneratedQuestion, QuestionUnit
def _rng_ns(seed: int, ns: str) -> random.Random:
"""由 (seed, 命名空间) 稳定派生独立随机数发生器。
用 SHA-256 派生而非 Python 内置 ``hash()``——后者受 hash randomization 影响,
跨进程不可复现。不同命名空间的 draw 流互不干扰,使 AR 单元折叠不扰动非 AR 抽样。
参数:
seed: 实验随机种子。
ns: 命名空间标签(如 "AR")。
返回:
以 SHA-256(f"{ns}:{seed}") 前 8 字节为种子的 ``random.Random``。
"""
digest = hashlib.sha256(f"{ns}:{seed}".encode()).digest()
return random.Random(int.from_bytes(digest[:8], "big"))
def build_batches(
items: list[GeneratedQuestion],
correctness: dict[str, bool],
batch_size: int,
min_class_per_batch: int,
seed: int,
correct_ratio: float = 0.0,
) -> tuple[list[list[GeneratedQuestion]], int]:
"""把诊断池里的题目切成多个混合 mini-batch(以 QuestionUnit 为原子调度单元)。
single 题为 1 题单元,AR pair 孪生对为 2 题单元;同一 pair 的两题整锁进同一 batch,
按单元级正确性(双向 AND)分桶。当 ``correct_ratio > 0`` 时,按题型为每组错误单元配比
一定数量的正确单元("动量"机制);``correct_ratio <= 0`` 时退化为纯错误单元模式。
参数:
items: 候选题目全集(可混含 single 与孪生对成员)。
correctness: question_id -> 基线是否答对。
batch_size: 单个 batch 的题目数上限(> 0pair 占 2)。
min_class_per_batch: 小类判定阈值——单元题目总数 ≤ 此值的题型整组锁进单一
batch> 0)。
seed: 随机种子,保证相同输入产出完全一致的切分。
correct_ratio: 正确题占比(0.0 ~ 1.0)。0.0 = 纯错误单元;0.5 = 错:正 = 1:1。
返回:
(非空 mini-batch 列表, selected_count);无错误单元时返回 ([], 0)。
selected_count 是所有 batch 中题目(展开后)总数。
异常:
ValueError: batch_size 或 min_class_per_batch < 1, 或
min_class_per_batch >= batch_size(破坏小类整组装箱不超容的前提)。
关键实现细节:
非 ARsingle)与 ARpair)各用独立稳定派生的 rng:非 AR 用 ``random.Random(seed)``
(复现旧逐题算法的确切 draw 序列,保证纯非 AR 输入逐字节一致),AR 用
``_rng_ns(seed, "AR")``;二者 draw 流互不干扰,故加入/移除 pair 不改变非 AR 的
抽样/洗牌序列。抽样在合并前按流分别进行(``_select_mixed_by_task_type`` 各跑一次),
大类洗牌按单元 kind 拆分后各用对应流。装箱顺序「先小类后大类」:小类整组
first-fit-decreasing(容量按单元 ``size`` 计,pair 占 2)装入首个容得下的 batch,
装不下新开 bin;大类洗牌后 round-robin 分发,遇碎片(size-2 单元放不进任一现存
batch 的剩余容量)新开 bin 兜底而非报错。最终每个 batch 展开回题目列表。
题型按名称排序处理以保证跨运行确定性。
"""
_validate_params(batch_size, min_class_per_batch)
# 非 AR 复现旧版 random.Random(seed) 的确切序列以满足黄金 byte-identity
# AR 走独立命名空间派生流,二者互不干扰。
rng_nonar = random.Random(seed)
rng_ar = _rng_ns(seed, "AR")
grouped = _group_units_by_task_type(items, correctness, correct_ratio, rng_nonar, rng_ar)
total = sum(_group_load(g) for g in grouped.values())
if total == 0:
return [], 0
nb = max(1, math.ceil(total / batch_size))
batches: list[list[QuestionUnit]] = [[] for _ in range(nb)]
small, large = _split_by_size(grouped, min_class_per_batch)
for group in _small_groups_decreasing(small):
_pack_small_class(batches, group, batch_size)
_distribute_large_classes(batches, large, batch_size, rng_nonar, rng_ar)
result = [flatten_units(b) for b in batches if b]
selected_count = sum(len(b) for b in result)
return result, selected_count
def _validate_params(batch_size: int, min_class_per_batch: int) -> None:
"""校验切分参数,非法值直接报错而非用默认值掩盖。
除各自 >= 1 外,强制 min_class_per_batch < batch_size:小类组题目总数 ≤
min_class_per_batch,唯有此前提成立才能保证小类整组放入单一 batch 而不超容;否则
_pack_small_class 新开的 bin 会装入超 batch_size 的整组,静默违反容量合约。此约束
与 config._validate_minibatch 一致,是 build_batches 对自身前提的防御性自校验(P5)。
"""
if batch_size < 1:
raise ValueError(f"batch_size 必须 >= 1, 实为 {batch_size}")
if min_class_per_batch < 1:
raise ValueError(f"min_class_per_batch 必须 >= 1, 实为 {min_class_per_batch}")
if min_class_per_batch >= batch_size:
raise ValueError(
f"min_class_per_batch 必须严格 < batch_size, 否则无法保证小类整组放入单一 "
f"batch 不超容; 实为 min_class_per_batch={min_class_per_batch}, "
f"batch_size={batch_size}"
)
def _group_units_by_task_type(
items: list[GeneratedQuestion],
correctness: dict[str, bool],
correct_ratio: float,
rng_nonar: random.Random,
rng_ar: random.Random,
) -> dict[str, list[QuestionUnit]]:
"""把题目聚合为单元并按题型分组:非 AR 与 AR 各走独立 draw 流后合并。
参数:
items: 候选题目全集。
correctness: question_id -> 基线是否答对。
correct_ratio: 正确题占比。
rng_nonar: 非 ARsingle 单元)抽样用 rng。
rng_ar: ARpair 单元)抽样用 rng。
返回:
task_type -> 混合后的单元列表(single 单元在前、pair 单元在后)。
"""
units = build_units(items)
singles = [u for u in units if u.kind == "single"]
pairs = [u for u in units if u.kind == "pair"]
grouped_nonar = _select_mixed_by_task_type(singles, correctness, correct_ratio, rng_nonar)
grouped_ar = _select_mixed_by_task_type(pairs, correctness, correct_ratio, rng_ar)
return _merge_grouped(grouped_nonar, grouped_ar)
def _group_load(group: list[QuestionUnit]) -> int:
"""一组单元展开后的题目总数(single 计 1,pair 计 2),即占用的 batch 容量。"""
return sum(u.size for u in group)
def _batch_load(batch: list[QuestionUnit]) -> int:
"""一个 batch 内单元展开后的题目总数,用于容量判断。"""
return sum(u.size for u in batch)
def _merge_grouped(
grouped_nonar: dict[str, list[QuestionUnit]],
grouped_ar: dict[str, list[QuestionUnit]],
) -> dict[str, list[QuestionUnit]]:
"""按 task_type 合并非 AR 与 AR 两条流的分组(single 在前、pair 在后)。
参数:
grouped_nonar: 非 ARsingle 单元)分组。
grouped_ar: ARpair 单元)分组。
返回:
task_type -> 合并后的单元列表;每类 single 单元在前、pair 单元在后,顺序稳定。
"""
merged: dict[str, list[QuestionUnit]] = {}
for task_type in sorted({*grouped_nonar, *grouped_ar}):
merged[task_type] = grouped_nonar.get(task_type, []) + grouped_ar.get(task_type, [])
return merged
def _split_by_size(
grouped: dict[str, list[QuestionUnit]],
min_class_per_batch: int,
) -> tuple[dict[str, list[QuestionUnit]], dict[str, list[QuestionUnit]]]:
"""按题目总数(单元展开)把题型分为小类(≤ 阈值)与大类(> 阈值)两组。"""
small = {t: g for t, g in grouped.items() if _group_load(g) <= min_class_per_batch}
large = {t: g for t, g in grouped.items() if _group_load(g) > min_class_per_batch}
return small, large
def _classify_unit(unit: QuestionUnit, correctness: dict[str, bool]) -> str | None:
"""判定单元落入哪个桶:error / correct / None(未知,跳过)。
参数:
unit: 目标单元。
correctness: question_id -> 是否答对(缺键视为未知)。
返回:
"error"(单元级正确性为 False)、"correct"(双向 AND 为 True);单元内任一题
未知(correctness 缺该键)返回 None,与旧逐题算法把未知题排除在错/对两桶之外
的语义一致。
关键实现:
先探测是否有未知题(get 返回 None ⟺ 键缺失,因 correctness 值恒为 bool),
全部已知后交由 unit_correctness 计双向 AND(此时 KeyError 不可达)。
"""
if any(correctness.get(q.question_id) is None for q in unit.questions):
return None
return "correct" if unit_correctness(unit, correctness) else "error"
def _select_mixed_by_task_type(
units: list[QuestionUnit],
correctness: dict[str, bool],
correct_ratio: float,
rng: random.Random,
) -> dict[str, list[QuestionUnit]]:
"""按题型分组,为每组错误单元按比例采样正确单元混入(单元粒度)。
只对有错误单元的题型做混合——无错误单元的题型不进 batch,即使有正确单元。
``correct_ratio <= 0`` 时退化为纯错误单元模式。本函数只处理单一 draw 流(全 single
或全 pair),使非 AR 与 AR 的抽样互不干扰。
参数:
units: 同一流的候选单元(全 single 或全 pair)。
correctness: question_id -> 基线是否答对。
correct_ratio: 正确题占比(0.0 ~ 1.0)。
rng: 本流专用随机数发生器,用于采样正确单元。
返回:
task_type -> 该题型的混合单元列表(错误单元全部 + 按比例采样的正确单元)。
关键实现:
n_correct 按错误单元「题目总数」而非单元数计,与旧逐题语义对齐(纯 single 时
单元数 == 题目数,采样序列逐字节一致)。
"""
errors_by_type: dict[str, list[QuestionUnit]] = {}
correct_by_type: dict[str, list[QuestionUnit]] = {}
for unit in units:
bucket = _classify_unit(unit, correctness)
if bucket == "error":
errors_by_type.setdefault(unit.task_type, []).append(unit)
elif bucket == "correct":
correct_by_type.setdefault(unit.task_type, []).append(unit)
if correct_ratio <= 0:
return errors_by_type
grouped: dict[str, list[QuestionUnit]] = {}
for task_type in sorted(errors_by_type):
errs = errors_by_type[task_type]
n_err = _group_load(errs)
n_correct = round(n_err * correct_ratio / (1 - correct_ratio))
available = correct_by_type.get(task_type, [])
sampled = (
list(available) if len(available) <= n_correct else rng.sample(available, n_correct)
)
grouped[task_type] = errs + sampled
return grouped
def _small_groups_decreasing(
small: dict[str, list[QuestionUnit]],
) -> list[list[QuestionUnit]]:
"""按组题目总数降序、同大小按 task_type 升序排出小类组(first-fit-decreasing 顺序)。
参数:
small: task_type -> 小类单元列表。
返回:
排好序的小类组列表;降序处理可降低碎片,确定性 tie-break 保证跨运行一致。
"""
return [small[t] for t in sorted(small, key=lambda t: (-_group_load(small[t]), t))]
def _pack_small_class(
batches: list[list[QuestionUnit]],
group: list[QuestionUnit],
batch_size: int,
) -> None:
"""用 first-fit 把一个小类整组放入首个容得下的 batch,装不下则新开 bin(就地修改)。
因小类组题目总数 ≤ min_class_per_batch < batch_size,新开的空 batch 必能容纳整组,
故此函数永不抛 ValueError,且整组(含内部 pair 单元)不拆。
参数:
batches: 当前各 batch(就地追加,必要时 append 新空 batch)。
group: 待锁定的小类单元组(整组不拆)。
batch_size: 单 batch 题目容量上限。
"""
load = _group_load(group)
for b in batches:
if _batch_load(b) + load <= batch_size:
b.extend(group)
return
batches.append(list(group))
def _distribute_large_classes(
batches: list[list[QuestionUnit]],
large: dict[str, list[QuestionUnit]],
batch_size: int,
rng_nonar: random.Random,
rng_ar: random.Random,
) -> None:
"""将各大类单元洗牌后 round-robin 分发到所有现存 batch(就地修改)。
参数:
batches: 当前各 batch(含小类装箱可能新开的 bin,就地追加)。
large: task_type -> 大类单元列表。
batch_size: 单 batch 题目容量上限。
rng_nonar: 非 ARsingle 单元)洗牌用 rng。
rng_ar: ARpair 单元)洗牌用 rng。
关键实现细节:
每组按单元 kind 拆成 single 子列与 pair 子列,分别用 rng_nonar / rng_ar 洗牌后
拼接(single 在前),使非 AR 洗牌 draw 流不受 pair 存在与否影响(纯 single 时
single 子列即整组,复现旧版单一 rng.shuffle 的序列)。全局指针在所有大类单元间
持续轮转,遇满箱跳过、遇碎片新开 bin。题型按名称排序以保证分发顺序确定。
"""
pointer = 0
for task_type in sorted(large):
group = large[task_type]
singles = [u for u in group if u.kind == "single"]
pairs = [u for u in group if u.kind == "pair"]
rng_nonar.shuffle(singles)
rng_ar.shuffle(pairs)
for unit in singles + pairs:
pointer = _place_round_robin(batches, unit, pointer, batch_size)
def _place_round_robin(
batches: list[list[QuestionUnit]],
unit: QuestionUnit,
pointer: int,
batch_size: int,
) -> int:
"""从 pointer 起找第一个容量够放 unit 的 batch 放入,返回下一次起始指针。
参数:
batches: 当前各 batch(就地追加)。
unit: 待放置的单元(占用 unit.size 个容量)。
pointer: 本次轮转起始 batch 下标。
batch_size: 单 batch 题目容量上限。
返回:
下一次轮转的起始指针(已前移一位)。
关键实现:
单个单元容量 ≤ batch_size 是前提(pair 占 2,而 batch_size > min_class ≥ 1 ⇒
batch_size ≥ 2),故此处断言防御。扫描一轮所有现存 batch 都放不下(size-2 单元
遇满地碎片)时新开 bin 兜底而非报错——聚合容量足够但单箱剩余不足是合法碎片场景。
纯 single(size 1)永不触发新开分支,故与旧逐题 round-robin 逐字节一致。
"""
assert unit.size <= batch_size, f"单元 size={unit.size} 超过 batch_size={batch_size}"
nb = len(batches)
for offset in range(nb):
idx = (pointer + offset) % nb
if _batch_load(batches[idx]) + unit.size <= batch_size:
batches[idx].append(unit)
return (idx + 1) % nb
batches.append([unit])
return len(batches) % len(batches)
+634
View File
@@ -0,0 +1,634 @@
"""结果驱动视频级切分的顶层编排:诊断信号 → 冻结 pools.json + manifest。
把已实现的组件串成 capstone 管线:从 harness.db 读 canonical 基线预测、从
DiagnosisSignalStore 读逐题诊断信号,构建全视频画像、贪心联合约束选择 trainval /
test,再以视频组为原子切出诊断 / 验证池,原子冻结 pools.json 并写溯源 manifest。
全程带六条防御断言(P5,任一不满足即 fail-fast,绝不静默兜底)。
只有基线推理与诊断是上游产物;本模块纯 code-controlled,不发起任何 LLM 调用,
读预测走只读连接,不改动 harness.db。
"""
from __future__ import annotations
import hashlib
import json
import sqlite3
from collections import Counter, defaultdict
from dataclasses import asdict, dataclass, field
from typing import TYPE_CHECKING
from loguru import logger
from app.harness.pools import save_pools, split_by_video_assignment
from app.harness.split_manifest import write_manifest
from app.harness.split_selection import (
SelectConfig,
build_video_records,
derive_reportable_types,
select_split,
)
from app.question_gen.loader import load_benchmark
if TYPE_CHECKING:
from pathlib import Path
from app.harness.pools import Pools
from app.harness.split_selection import SplitAssignment, VideoRecord
from core.evolution.protocols import DiagnosisSignalStore
from core.evolution.types import DiagnosisSignalRow
from core.types import GeneratedQuestion
# 多样性主格子 = 12 题型 × 4 错误类别 = 48 格,覆盖报告以此为分母。
_DIVERSITY_GRID_TOTAL = 48
_QUESTIONS_PER_VIDEO = 3
@dataclass(frozen=True)
class SplitBuildConfig:
"""结果驱动切分的旋钮快照(科研配置,随实验扫动)。
承载贪心选择器与视频组题级切分的全部可扫参数;asdict 后直接写入 manifest 的
config 快照,保证复现时可比对。floor_k 为不可哈希容器,标 hash=False 排除出
自动 __hash__,避免 frozen dataclass 被哈希时报错(本类不作字典键,仅承载配置)。
字段:
n_trainval: trainval 目标视频数(多样性阶段填充上限)。
floor_k: 各高信号 task_type 的 T2 defect 下限(select_split 硬约束)。
epsilon: test 相对全局的最大允许分布偏差(题型 / 难度两维)。
report_floor: per-type 报告门限,题数 ≥ 此值的 task_type 才入 ε 约束。
select_seed: 贪心选择器预洗牌种子(打破等增益平局)。
val_ratio: validation 占 trainval 视频组总数的比例。
split_seed: 视频组题级切分的洗牌种子。
val_wrong_min: validation 池最少错题数,切分时保证功效(不足则从 diag 换入
低 T2 错题组补足,耗尽 fail-loud)。
"""
n_trainval: int
floor_k: dict[str, int] = field(hash=False)
epsilon: float
report_floor: int
select_seed: int
val_ratio: float
split_seed: int
val_wrong_min: int
@dataclass(frozen=True)
class SplitBuildResult:
"""build_split 的返回结果(冻结产物 + 溯源)。
字段:
pools: 冻结的三池(diagnosis / validation / test)。
manifest: 写入 split_manifest.json 的溯源字典(含 pools_sha256)。
assignment: video_id -> "trainval" | "test" 归属字典。
"""
pools: Pools
manifest: dict
assignment: dict[str, str]
def __getitem__(self, key: str) -> object:
"""兼容字典式访问(result["pools"] / ["manifest"] / ["assignment"])。
参数:
key: 字段名,取值 pools / manifest / assignment。
返回:
对应字段值。
异常:
KeyError: key 非上述三者之一。
"""
if key not in {"pools", "manifest", "assignment"}:
raise KeyError(f"未知字段: {key}")
return getattr(self, key)
def _unique_backup_path(path: Path, suffix: str) -> Path:
"""求 path 的唯一 .bak.<suffix> 备份路径,已存在则追加递增序号避免覆盖。
首选 ``<name>.bak.<suffix>``;若已存在,退化为 ``<name>.bak.<suffix>.2``、
``.3`` … 直到找到不存在的名字。保证连续 forced freeze(同 suffix 或都缺
manifest 用 'prev')不会静默覆盖此前保留的备份。
参数:
path: 待备份的原文件路径。
suffix: 备份后缀(旧 pools_sha256 前 8 位或 'prev')。
返回:
目录内唯一、尚不存在的备份路径。
"""
candidate = path.with_name(f"{path.name}.bak.{suffix}")
counter = 2
while candidate.exists():
candidate = path.with_name(f"{path.name}.bak.{suffix}.{counter}")
counter += 1
return candidate
def _guard_frozen_products(out_path: Path, manifest_path: Path, *, force: bool) -> None:
"""冻结前的覆盖保护:产物已存在时按 force 决定报错或备份。
参数:
out_path: 目标 pools.json 路径。
manifest_path: 目标 split_manifest.json 路径。
force: False 时已存在即 FileExistsErrorTrue 时把旧产物重命名为唯一的
.bak.<旧 pools_sha256 前 8 位或 'prev'>(同名已存在则追加递增序号)再放行。
异常:
FileExistsError: force=False 且产物已存在(防静默覆盖冻结锚点)。
OSError: 备份 rename 失败;已备份的文件先 rollback 回原名再抛出,保证
要么两文件都备份、要么都不动(原子性,不留半备份的不一致目录)。
"""
if not out_path.exists() and not manifest_path.exists():
return
if not force:
raise FileExistsError(
f"已存在冻结产物 {out_path}(或其 manifest)。重跑切分会覆盖训练依赖的"
"冻结锚点——确认要替换请加 --force(旧产物将备份为 .bak.*)。"
)
# 备份后缀取旧 manifest 的 pools_sha256 前 8 位,无则用 'prev'
suffix = "prev"
if manifest_path.exists():
try:
old = json.loads(manifest_path.read_text(encoding="utf-8"))
suffix = str(old.get("pools_sha256", "prev"))[:8] or "prev"
except (json.JSONDecodeError, OSError):
suffix = "prev"
# 先为每个存在的文件求唯一备份路径(互不冲突),再逐个 rename;
# 中途失败则把已备份的 rollback 回原名,保证原子性。
to_backup = [p for p in (out_path, manifest_path) if p.exists()]
done: list[tuple[Path, Path]] = [] # (备份路径, 原路径),供 rollback
try:
for p in to_backup:
dst = _unique_backup_path(p, suffix)
p.rename(dst)
done.append((dst, p))
except OSError:
for backup_path, original in reversed(done):
backup_path.rename(original)
raise
def build_split(
*,
db_path: Path,
baseline_run_id: str,
signal_store: DiagnosisSignalStore,
diag_fingerprint: str,
questions_dir: Path,
config: SplitBuildConfig,
out_path: Path,
manifest_path: Path,
generated_at: str,
force: bool = False,
) -> SplitBuildResult:
"""顶层编排结果驱动视频级切分,冻结 pools.json + manifest 并跑防御断言。
步骤:读 canonical 基线预测 → 读诊断信号 → 构建全视频画像 → 贪心选择 trainval /
test → 加载题库并以视频归属切三池 → 原子冻结 pools.json → 写溯源 manifest →
六条防御断言 fail-fast 校验。
契约(Task 11):val_wrong_min 前置到切分内保证功效——build_split 计算
wrong_tier_by_video 并连同 config.val_wrong_min 传入 split_by_video_assignment
切分时若 val 错题不足即从 diag 换入低 T2 错题组补足(耗尽 fail-loud)。CLI 的
check_mcnemar_power 作切分冻结后的冗余最终确认。
参数:
db_path: harness.db 路径(只读读取 predictions,不改动)。
baseline_run_id: 基线 run 标识(如 "infer_adhoc")。
signal_store: 逐题诊断信号存储端口,读 (run, fingerprint) 下全部信号行。
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号。
questions_dir: benchmark 题库目录,加载 GeneratedQuestion。
config: 切分旋钮快照。
out_path: 冻结 pools.json 目标路径(原子写)。
manifest_path: 溯源 manifest 目标路径(原子写)。
generated_at: 生成时间戳(ISO 字符串),由调用方传入以保证可复现。
force: 覆盖保护开关。False(默认)时若 out_path/manifest_path 已存在即
FileExistsError(防静默覆盖训练依赖的冻结锚点);True 时先把旧产物备份为
.bak.* 再放行覆盖。
返回:
SplitBuildResult,含 pools / manifest / assignment,支持字典式访问。
异常:
AssertionError: 六条防御断言任一不满足(fail-fast,不静默)。
ValueError: 上游依赖校验失败(如 correctness 缺题、assignment 非法)。
"""
# Phase 1: canonical 基线预测 + 诊断信号。
preds = load_canonical_predictions(db_path, baseline_run_id)
signal_rows_raw = signal_store.load(baseline_run_id, diag_fingerprint)
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint)
signal_rows = [
{
"question_id": row.question_id,
"task_type": row.task_type,
"error_type": row.error_type,
"tier": row.tier,
}
for row in signal_rows_raw
]
# Phase 2: 全视频画像 + 贪心联合约束选择。
videos = build_video_records(preds, signal_rows)
total_by_type = Counter(pred["task_type"] for pred in preds)
reportable_types = derive_reportable_types(dict(total_by_type), config.report_floor)
assignment_obj = select_split(
videos,
config=SelectConfig(
n_trainval=config.n_trainval,
floor_k=config.floor_k,
epsilon=config.epsilon,
reportable_types=reportable_types,
seed=config.select_seed,
),
)
assignment = _assignment_to_dict(assignment_obj)
logger.info(
"视频级切分完成: trainval={} test={} (总 {} 视频)",
len(assignment_obj.trainval),
len(assignment_obj.test),
len(videos),
)
# Phase 3: 加载题库 + 视频归属切三池 + 原子冻结。
questions = load_benchmark(questions_dir)
correctness = {pred["question_id"]: pred["correct"] for pred in preds}
tier_by_q = {row["question_id"]: row["tier"] for row in signal_rows}
wrong_tier_by_video: dict[str, int] = defaultdict(int)
for pred in preds:
if not pred["correct"] and tier_by_q.get(pred["question_id"]) == "T2":
wrong_tier_by_video[pred["video_id"]] += 1
pools = split_by_video_assignment(
questions,
assignment,
correctness,
config.val_ratio,
config.split_seed,
baseline_run_id=baseline_run_id,
val_wrong_min=config.val_wrong_min,
wrong_tier_by_video=dict(wrong_tier_by_video),
)
_guard_frozen_products(out_path, manifest_path, force=force)
save_pools(pools, out_path)
# Phase 4: 溯源 manifestpools_sha256 锚定冻结内容)。
coverage_report = _build_coverage_report(
videos, assignment_obj, signal_rows_raw, reportable_types, config
)
manifest = write_manifest(
manifest_path,
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
seed=config.split_seed,
config=asdict(config),
pools_json_text=out_path.read_text(encoding="utf-8"),
coverage_report=coverage_report,
generated_at=generated_at,
)
# Phase 5: 防御断言 fail-fast。
_assert_split_invariants(
pools=pools,
expected_question_ids={q.question_id for q in questions},
signal_rows_raw=signal_rows_raw,
diag_fingerprint=diag_fingerprint,
out_path=out_path,
manifest=manifest,
)
return SplitBuildResult(pools=pools, manifest=manifest, assignment=assignment)
def _normalize_choice(choice: str | None) -> str:
"""选项归一:strip → 大写 → 取首字母,None 归一为空串。
参数:
choice: 原始选项文本(预测或答案),可为 None。
返回:
归一后的单字母(无内容时为空串)。
"""
return (choice or "").strip().upper()[:1]
def load_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
"""从 harness.db 只读取指定 run 每题首行(ORDER BY rowid)为 canonical 预测。
共享口径 helperCLI(可诊断错题筛选 + INFRA T0 补录)与 build_split(切分)
共用同一"每 qid 取 rowid 最小首行 + 归一化 correct 判定"口径,消除两处重复实现。
同一 question_id 可能有多行(重跑 / 补测),canonical 口径取 rowid 最小的首行,
保证 distinct question 计数与对错判定确定。correct = 预测与答案各自归一
(strip → 大写 → 取首字母)后逐字符相等。
口径边界:旧 build_or_load_pools 的 legacy 池构建路径(app/harness/pools.py)是
另一条独立既有链路,不共用本 helper,两者刻意不统一(本次不动 legacy 路径)。
参数:
db_path: harness.db 路径(URI mode=ro 只读打开,绝不改动基线 db)。
baseline_run_id: 基线 run 标识。
返回:
canonical 预测行列表,每行含 question_id / video_id / task_type /
prediction / answer / stop_reason / correctbool)。按 rowid 升序去重,
每 qid 保留首行。
异常:
ValueError: 该 run 无任何预测行(fail-fast,不返回空切分)。
实现细节:
按 rowid 升序遍历,首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
"""
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn.row_factory = sqlite3.Row
try:
rows = conn.execute(
"SELECT question_id, video_id, task_type, prediction, answer, stop_reason "
"FROM predictions WHERE run_id = ? ORDER BY rowid",
(baseline_run_id,),
).fetchall()
finally:
conn.close()
canonical: dict[str, dict] = {}
for row in rows:
qid = row["question_id"]
if qid in canonical:
continue
canonical[qid] = {
"question_id": qid,
"video_id": row["video_id"],
"task_type": row["task_type"],
"prediction": row["prediction"],
"answer": row["answer"],
"stop_reason": row["stop_reason"],
"correct": _normalize_choice(row["prediction"]) == _normalize_choice(row["answer"]),
}
if not canonical:
raise ValueError(f"run_id={baseline_run_id} 无任何预测行,无法切分")
return list(canonical.values())
def _assignment_to_dict(assignment_obj: SplitAssignment) -> dict[str, str]:
"""把 SplitAssignment 展平为 video_id -> "trainval" | "test" 归属字典。
参数:
assignment_obj: 贪心选择器产出的切分归属。
返回:
全部视频的归属字典(trainval 与 test 并集,键互斥)。
"""
assignment = dict.fromkeys(assignment_obj.trainval, "trainval")
for vid in assignment_obj.test:
assignment[vid] = "test"
return assignment
def _assert_fingerprint_consistent(
signal_rows_raw: list[DiagnosisSignalRow],
diag_fingerprint: str,
) -> None:
"""防御④:全部诊断信号行的 diag_fingerprint 必须与传入指纹一致。
参数:
signal_rows_raw: store 读回的诊断信号行。
diag_fingerprint: 期望的诊断口径指纹。
异常:
AssertionError: 存在指纹不一致的信号行(store 未正确按指纹过滤)。
"""
mismatched = [
row.question_id for row in signal_rows_raw if row.diag_fingerprint != diag_fingerprint
]
if mismatched:
raise AssertionError(
f"诊断信号指纹不一致 {len(mismatched)} 行,期望 {diag_fingerprint}: {mismatched[:5]}"
)
def _fraction_by_type(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
"""各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
参数:
records: 视频记录子集。
keys: 需计算占比的 task_type 键集。
返回:
{task_type: 占比}records 为空时全部记 0.0。
"""
total = len(records)
if total == 0:
return dict.fromkeys(keys, 0.0)
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
def _fraction_by_difficulty(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
"""各难度桶(错题数)在给定视频集中的占比。
参数:
records: 视频记录子集。
buckets: 难度桶键集。
返回:
{难度桶: 占比};records 为空时全部记 0.0。
"""
total = len(records)
if total == 0:
return dict.fromkeys(buckets, 0.0)
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
def _max_dev(global_dist: dict, subset_dist: dict, keys: set) -> float:
"""逐键取全局与子集分布的最大绝对偏差(键集为空约定 0.0)。"""
if not keys:
return 0.0
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
def _build_coverage_report(
videos: list[VideoRecord],
assignment_obj: SplitAssignment,
signal_rows_raw: list[DiagnosisSignalRow],
reportable_types: set[str],
config: SplitBuildConfig,
) -> dict:
"""组装 manifest 覆盖报告:48 格覆盖 / floor 达标 / test 代表性偏差 / tier 占比。
参数:
videos: 全视频画像记录。
assignment_obj: 切分归属(trainval / test)。
signal_rows_raw: 诊断信号行(统计 tier 占比)。
reportable_types: 参与 ε 代表性校验的题型集。
config: 切分旋钮(floor_k / epsilon)。
返回:
覆盖报告字典,含 cells_covered / grid_total / floor_satisfied /
test_representativeness_deviation / tier_distribution /
evolution_target_distribution。
实现细节:
evolution_target_distribution 只统计 T2 信号(可训练缺陷),按
tool / skill / system 计数,报告"哪层参数组拿到梯度"T0/T1/uncertain 行
evolution_target 恒为 None,不入该分布。
"""
by_id = {v.video_id: v for v in videos}
trainval = [by_id[vid] for vid in assignment_obj.trainval]
test = [by_id[vid] for vid in assignment_obj.test]
covered_cells: set[tuple[str, str]] = set()
trainval_wrong: Counter[str] = Counter()
for video in trainval:
covered_cells |= set(video.cells)
trainval_wrong.update(video.wrong_by_type)
floor_satisfied = {
task_type: trainval_wrong.get(task_type, 0) >= floor
for task_type, floor in config.floor_k.items()
}
type_dev = _max_dev(
_fraction_by_type(videos, reportable_types),
_fraction_by_type(test, reportable_types),
reportable_types,
)
diff_buckets = {r.difficulty for r in videos}
diff_dev = _max_dev(
_fraction_by_difficulty(videos, diff_buckets),
_fraction_by_difficulty(test, diff_buckets),
diff_buckets,
)
tier_distribution, evolution_target_distribution = _signal_distributions(signal_rows_raw)
return {
"cells_covered": len(covered_cells),
"grid_total": _DIVERSITY_GRID_TOTAL,
"floor_satisfied": floor_satisfied,
"test_representativeness_deviation": {
"type_max": type_dev,
"difficulty_max": diff_dev,
"epsilon": config.epsilon,
},
"tier_distribution": tier_distribution,
"evolution_target_distribution": evolution_target_distribution,
}
def _signal_distributions(
signal_rows_raw: list[DiagnosisSignalRow],
) -> tuple[dict[str, float], dict[str, int]]:
"""由诊断信号行算 tier 占比分布与 T2 进化目标计数分布。
参数:
signal_rows_raw: 诊断信号行。
返回:
(tier_distribution, evolution_target_distribution) 二元组:
- tier_distribution: {tier: 占比},无信号时为空 dict;
- evolution_target_distribution: 仅统计 T2(可训练缺陷)信号,按
tool / skill / system 计数,报告哪层参数组拿到梯度;T0/T1/uncertain 行
evolution_target 恒为 None,不入该分布。
"""
tier_counts = Counter(row.tier for row in signal_rows_raw)
total_signals = sum(tier_counts.values())
tier_distribution = (
{tier: count / total_signals for tier, count in tier_counts.items()}
if total_signals
else {}
)
evolution_target_distribution = dict(
Counter(
row.evolution_target
for row in signal_rows_raw
if row.tier == "T2" and row.evolution_target is not None
)
)
return tier_distribution, evolution_target_distribution
def _assert_split_invariants(
*,
pools: Pools,
expected_question_ids: set[str],
signal_rows_raw: list[DiagnosisSignalRow],
diag_fingerprint: str,
out_path: Path,
manifest: dict,
) -> None:
"""六条防御断言 fail-fast:任一不满足即 AssertionErrorP5,不静默不兜底)。
① 三池视频集两两不相交;② 三池覆盖全部题(按 distinct question);
③ 每 video 恰 3 题;④ 诊断信号指纹一致;⑤ manifest pools_sha256 == sha256(冻结内容)
⑥ question_id 全局唯一(无重复行)。
参数:
pools: 冻结的三池。
expected_question_ids: 加载题库的 question_id 全集(覆盖基准)。
signal_rows_raw: 诊断信号行(指纹校验)。
diag_fingerprint: 期望诊断指纹。
out_path: 冻结 pools.json 路径。
manifest: 已写入的 manifest 字典。
异常:
AssertionError: 任一防御断言不满足。
"""
all_questions = pools.diagnosis + pools.validation + pools.test
_assert_pools_video_disjoint(pools) # ①
_assert_question_ids_unique(all_questions) # ⑥
_assert_question_coverage(all_questions, expected_question_ids) # ②
_assert_three_questions_per_video(all_questions) # ③
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint) # ④
_assert_pools_sha256(out_path, manifest) # ⑤
def _assert_pools_video_disjoint(pools: Pools) -> None:
"""防御①:diagnosis / validation / test 三池视频集两两不相交。"""
diag_v = {q.video_id for q in pools.diagnosis}
val_v = {q.video_id for q in pools.validation}
test_v = {q.video_id for q in pools.test}
if diag_v & val_v or diag_v & test_v or val_v & test_v:
raise AssertionError(
f"三池视频集非互斥: diag∩val={diag_v & val_v}, "
f"diag∩test={diag_v & test_v}, val∩test={val_v & test_v}"
)
def _assert_question_ids_unique(all_questions: list[GeneratedQuestion]) -> None:
"""防御⑥:三池合并后 question_id 全局唯一(无重复行)。"""
qids = [q.question_id for q in all_questions]
if len(qids) != len(set(qids)):
duplicates = [qid for qid, count in Counter(qids).items() if count > 1]
raise AssertionError(f"question_id 重复 {len(duplicates)} 个: {duplicates[:5]}")
def _assert_question_coverage(
all_questions: list[GeneratedQuestion],
expected_question_ids: set[str],
) -> None:
"""防御②:三池覆盖题库全部题(按 distinct question,缺题 / 多题均 fail-fast)。"""
actual = {q.question_id for q in all_questions}
if actual != expected_question_ids:
missing = expected_question_ids - actual
extra = actual - expected_question_ids
raise AssertionError(
f"三池题目覆盖不完整: 缺 {len(missing)}{len(extra)} (缺样例 {sorted(missing)[:5]})"
)
def _assert_three_questions_per_video(all_questions: list[GeneratedQuestion]) -> None:
"""防御③:每 video 恰 3 题(视频组原子切分不应劈裂视频的题)。"""
per_video = Counter(q.video_id for q in all_questions)
bad_videos = {vid: n for vid, n in per_video.items() if n != _QUESTIONS_PER_VIDEO}
if bad_videos:
raise AssertionError(
f"存在 video 题数 != {_QUESTIONS_PER_VIDEO}: {dict(list(bad_videos.items())[:5])}"
)
def _assert_pools_sha256(out_path: Path, manifest: dict) -> None:
"""防御⑤:manifest 的 pools_sha256 == sha256(冻结 pools.json 内容)。"""
actual_sha = hashlib.sha256(out_path.read_text(encoding="utf-8").encode("utf-8")).hexdigest()
if actual_sha != manifest["pools_sha256"]:
raise AssertionError(
f"pools_sha256 不一致: manifest={manifest['pools_sha256']} 实际={actual_sha}"
)
+275
View File
@@ -0,0 +1,275 @@
"""step 级续训 checkpoint_TrainState 可持久化字段的序列化 / 反序列化。
_TrainState 的累加包均为扁平纯数据 dataclass,经 dataclasses.asdict 序列化为
纯 JSON dict;反序列化时用 Cls(**d) 还原,其中 SystemCasePack 含嵌套 CaseSample
列表、Probation 含嵌套 RejectedEdit 列表,需逐个重建。
不持久化的字段:gate_pools / baseline_cache(各自文件级自持久化,resume 时按
指纹重载)、best_*(从 manifest best 指针读)、global_step(存 progress 块,
由 train 单独赋值)。gate_epoch_observed 持久化:warm p-hat 在 gate_pools.json
幸存,观测开关须随行,否则 resume 后阶梯排序回退冷启动序。
"""
from __future__ import annotations
import json
import os
from dataclasses import asdict
from typing import TYPE_CHECKING, Any
from app.harness.validate import Probation
from core.evolution.types import (
CaseSample,
RejectedEdit,
SystemCasePack,
ToolCasePack,
)
if TYPE_CHECKING:
from pathlib import Path
CHECKPOINT_SCHEMA_VERSION = 1
# ---------------------------------------------------------------------------
# 结构性 / 决策性指纹键
# ---------------------------------------------------------------------------
_STRUCTURAL_KEYS = (
"batch_size",
"min_class_per_batch",
"epochs",
"diag_size",
"val_size",
"batch_correct_ratio",
"trainable_min_units",
)
_DECISION_KEYS = (
"edit_budget_start",
"edit_budget_end",
"early_stop_patience",
"use_slow_momentum",
"skill_update_mode",
"appendix_consolidate_threshold",
"momentum_samples",
"gate_e_confirm",
"gate_e_provisional",
"gate_w_net_min",
"gate_delta_min",
"gate_lambda_dir",
"gate_e_rollback",
"gate_n_max",
"gate_p_low",
"gate_p_high",
"gate_probe_quota",
"gate_gamma_decay",
"gate_cooldown_steps",
"gate_guard_err",
)
# ---------------------------------------------------------------------------
# 序列化 / 反序列化
# ---------------------------------------------------------------------------
def serialize_state(state: Any) -> dict[str, Any]:
"""把 _TrainState 的可持久化字段转为纯 JSON dict。
参数:
state: _TrainState 实例(duck-typed,仅需含可持久化字段)。
返回:
纯 JSON 可序列化的 dict,不含 gate_pools / baseline_cache /
best_* / global_step。
关键实现细节:
- changed_task_types_this_epoch 是 setJSON 无 set,故 sorted 成有序列表。
- dataclass 均经 asdict 递归转 dict(含 SystemCasePack 嵌套 CaseSample、
Probation 嵌套 RejectedEdit)。
"""
return {
"correctness": state.correctness,
"eval_prev_acc": state.eval_prev_acc,
"eval_prev_run_id": state.eval_prev_run_id,
"baseline_skills_version": state.baseline_skills_version,
"baseline_prompts_version": state.baseline_prompts_version,
"epochs_since_best_improved": state.epochs_since_best_improved,
"epoch_start_skills": state.epoch_start_skills,
"changed_task_types_this_epoch": sorted(state.changed_task_types_this_epoch),
"rejected_buffer": {k: [asdict(x) for x in v] for k, v in state.rejected_buffer.items()},
"system_packs": [asdict(x) for x in state.system_packs],
"tool_packs": [asdict(x) for x in state.tool_packs],
"probations": {t: asdict(p) for t, p in state.probations.items()},
"gate_cooldown": state.gate_cooldown,
"gate_epoch_observed": state.gate_epoch_observed,
}
def _restore_system_pack(d: dict[str, Any]) -> SystemCasePack:
"""还原 SystemCasePack,含嵌套 CaseSample 列表。
参数:
d: asdict(SystemCasePack) 产出的 dict。
返回:
复活的 SystemCasePackfailure_cases / success_cases 重建为 CaseSample 实例。
"""
return SystemCasePack(
stats=d["stats"],
failure_cases=[CaseSample(**c) for c in d["failure_cases"]],
success_cases=[CaseSample(**c) for c in d["success_cases"]],
)
def deserialize_state_fields(d: dict[str, Any]) -> dict[str, Any]:
"""把序列化 dict 还原为可填入 _TrainState 的字段字典(dataclass 复活)。
参数:
d: serialize_state 产出并经 JSON 往返的 dict。
返回:
字段名 -> 值的 dict,可直接铺到 _TrainState;其中各 dataclass 已复活、
changed_task_types_this_epoch 还原为 set。
关键实现细节:
- RejectedEdit / ToolCasePack 字段均为标量/dict/list[dict]Cls(**d) 直接构造。
- SystemCasePack 含嵌套 CaseSample,交由 _restore_system_pack 重建。
- Probation 含嵌套 RejectedEdit 列表(pending_edits),先重建内层再构造外层。
- 直接取 d[...] 不用 .get 兜底:serialize 后的 checkpoint 必带全部键,
缺键即 checkpoint 损坏,应硬失败(P5 不掩盖)。
"""
return {
"correctness": d["correctness"],
"eval_prev_acc": d["eval_prev_acc"],
"eval_prev_run_id": d["eval_prev_run_id"],
"baseline_skills_version": d["baseline_skills_version"],
"baseline_prompts_version": d["baseline_prompts_version"],
"epochs_since_best_improved": d["epochs_since_best_improved"],
"epoch_start_skills": d["epoch_start_skills"],
"changed_task_types_this_epoch": set(d["changed_task_types_this_epoch"]),
"rejected_buffer": {
k: [RejectedEdit(**x) for x in v] for k, v in d["rejected_buffer"].items()
},
"system_packs": [_restore_system_pack(x) for x in d["system_packs"]],
"tool_packs": [ToolCasePack(**x) for x in d["tool_packs"]],
"probations": {
t: Probation(
**{
**d_p,
"pending_edits": [RejectedEdit(**x) for x in d_p["pending_edits"]],
}
)
for t, d_p in d["probations"].items()
},
"gate_cooldown": d["gate_cooldown"],
"gate_epoch_observed": d["gate_epoch_observed"],
}
# ---------------------------------------------------------------------------
# 配置指纹
# ---------------------------------------------------------------------------
def compute_fingerprint(config: Any) -> dict[str, Any]:
"""采集影响训练轨迹的配置项(结构性 + 决策性)。
参数:
config: 训练配置对象(duck-typed,需含 _STRUCTURAL_KEYS + _DECISION_KEYS 属性)。
返回:
指纹 dict,键为配置项名,值为对应配置值。
"""
return {k: getattr(config, k) for k in _STRUCTURAL_KEYS + _DECISION_KEYS}
def check_fingerprint(saved: dict[str, Any], config: Any) -> tuple[list[str], list[str]]:
"""比对保存的指纹与当前配置。返回 (结构性不一致项, 决策性不一致项)。
参数:
saved: checkpoint 中保存的 config_fingerprint。
config: 当前训练配置对象。
返回:
(structural, decision) 两个不一致项名列表。
关键实现细节:
结构性不一致(batch_size/min_class_per_batch/epochs/diag_size/val_size/
batch_correct_ratio)→ 调用方应拒绝 resume;决策性不一致 → 仅告警放行。
"""
cur = compute_fingerprint(config)
structural = [k for k in _STRUCTURAL_KEYS if saved.get(k) != cur[k]]
decision = [k for k in _DECISION_KEYS if saved.get(k) != cur[k]]
return structural, decision
# ---------------------------------------------------------------------------
# 读写 checkpoint
# ---------------------------------------------------------------------------
def write_checkpoint(
workspace_dir: Path,
*,
state: Any,
epoch: int,
step_completed: int,
phase: str,
global_step: int,
total_steps: int,
version_snapshot: dict[str, str],
epoch_batches: list[list[str]],
config: Any,
) -> None:
"""原子写 checkpoint.json.tmp 再 os.replace)。
参数:
workspace_dir: workspace 目录,checkpoint.json 写入其下。
state: _TrainState 实例,交由 serialize_state 序列化。
epoch: 当前 epoch 序号。
step_completed: 本 epoch 内已完成的 step 数。
phase: 续训阶段标识(如 "in_epoch")。
global_step: 全局 step 序号。
total_steps: 全局总 step 数。
version_snapshot: skills/prompts 版本快照。
epoch_batches: 本 epoch 的 batch 划分(unit_id 列表的列表,孪生对折叠为
单个 unit_id;纯非 AR 下 unit_id==question_id)。
config: 训练配置对象,用于计算 config_fingerprint。
关键实现细节:
先写 checkpoint.json.tmp 再 os.replace,保证 checkpoint 不被写一半的中断破坏。
"""
payload = {
"schema_version": CHECKPOINT_SCHEMA_VERSION,
"progress": {
"epoch": epoch,
"step_completed": step_completed,
"phase": phase,
"global_step": global_step,
"total_steps": total_steps,
},
"version_snapshot": version_snapshot,
"epoch_batches": epoch_batches,
"config_fingerprint": compute_fingerprint(config),
"state": serialize_state(state),
}
path = workspace_dir / "checkpoint.json"
tmp = path.with_name("checkpoint.json.tmp")
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2))
os.replace(tmp, path)
def load_checkpoint(workspace_dir: Path) -> dict[str, Any] | None:
"""读 checkpoint.json,不存在返回 None。
参数:
workspace_dir: workspace 目录。
返回:
checkpoint payload dictcheckpoint.json 不存在时返回 None。
"""
path = workspace_dir / "checkpoint.json"
if not path.exists():
return None
return json.loads(path.read_text())
+450
View File
@@ -0,0 +1,450 @@
"""运行配置:RunConfig frozen dataclass 与 YAML + CLI + .env 三层加载。
三层合并优先级CLI > .env > YAML遵循 CLAUDE.md §4.5 配置管理规范
- YAML科研实验配置会在实验中反复扫动的参数存放于 config/
- .env工程配置少变路径如 workspace_dirstore_dir通过环境变量注入
- CLI单次临时覆盖
"""
from __future__ import annotations
import dataclasses
import os
from dataclasses import dataclass
from pathlib import Path
import yaml
_VALID_MODES = {"infer", "train", "diagnose", "evolve", "eval", "promote"}
_VALID_SKILL_MODES = {"auto", "manual", "none"}
_VALID_SKILL_UPDATE_MODES = {"patch", "rewrite"}
_VALID_POOL_SPLIT_MODES = {"global", "per_category"}
_PATH_FIELDS = {"workspace_dir", "store_dir"}
# Video-MME 的任务类型数量:验证池每类至少保底 eval_min_per_class 题,共 12 类。
_VIDEO_MME_TASK_TYPE_COUNT = 12
# .env 工程配置字段映射(环境变量名 → RunConfig 字段名)。
# 仅路径类工程配置走 .env,科研实验参数走 YAML。
_ENV_FIELD_MAP: dict[str, str] = {
"HARNESS_WORKSPACE_DIR": "workspace_dir",
"HARNESS_STORE_DIR": "store_dir",
}
@dataclass(frozen=True)
class RunConfig:
"""实验运行配置,所有参数的唯一归口。
frozen=True 确保配置在创建后不可变防止运行中被意外修改
三层合并优先级CLI > .env > YAML
字段:
workspace_dir: Workspace 根目录
store_dir: Store 根目录
mode: 运行模式"infer" / "train" / "diagnose" / "evolve" / "eval" / "promote"
concurrency: 并行 worker
max_steps: AgentLoop 单题最大步数
skill_mode: Skill 加载模式"auto" / "manual" / "none"
n_samples: 题目截取数0 表示全量
questions: 题目在 questions/ 下的相对路径
skills_version: Skills 版本号
prompts_version: Prompts 版本号
epochs: 训练轮数
diag_size: 诊断池题目数
diag_correct_ratio: 诊断池中正确题目占比
val_size: 验证池题目数
val_correct_ratio: 验证池中正确题目占比
edit_budget_start: 编辑预算前期上限
edit_budget_end: 编辑预算后期下限
batch_size: mini-batch 单批题目数
min_class_per_batch: 单批中每个任务类型至少保留的题目数< batch_size
eval_min_per_class: 验证池中每个任务类型至少保底的题目数
trainable_min_units: 可训练性预检每题型 diag+val 单元数下限低于则剔除该题型
early_stop_patience: 全局 best 连续未提升的容忍轮数达到即早停
test_size: held-out 测试池题目数
use_slow_momentum: 是否启用快慢双速进化中的慢速 momentum 更新
gate_e_confirm: CE-Gate CONFIRMED 接受的 e 值门槛1/alphaVille 界假阳率 alpha
gate_e_provisional: 题尽暂定接受门 + futility 提前止损的代数界
gate_w_net_min: 题尽暂定接受要求的最小净胜数win - loss
gate_delta_min: 最小点估计效应量下限承接旧 margin 语义
gate_lambda_dir: Wald 方向拒绝的对数似然比阈值必须为负
gate_e_rollback: 试用期对称回滚门回滚 e 值门槛
gate_n_max: 单次 gate 消耗的题数上限
gate_p_low: 信息量阶梯 p-hat 保留区间下界剔除必错零信息题
gate_p_high: 信息量阶梯 p-hat 保留区间上界剔除必对零信息题
gate_probe_quota: 冷启动探针集比例全错题中插尾的比例
gate_gamma_decay: 逐题正确率估计 p-hat EMA 衰减系数
gate_cooldown_steps: 回滚后该题型跳过进化的冷却 step
gate_guard_err: gate 内累计 INFRA 错误率护栏
skill_update_mode: skill 进化模式"patch"局部 edit/ "rewrite"整篇重写
appendix_consolidate_threshold: appendix note 条数达此值触发 LLM consolidation
run_id: diagnose/evolve 模式要分析的运行 ID默认空字符串
batch_correct_ratio: 单批中正确题目占比范围 [0, 1)
momentum_samples: 慢速 momentum 更新时从诊断池采样的题目数必须 >= 1
seed: fresh 训练的种子名对应 seed.json默认 "initial"
version: eval/promote 模式指定的 store 版本号 "v3"
resume: train 模式是否从已有 checkpoint 续训
fresh: train 模式是否从种子全新开始
task_types: 限定参与的任务类型子集None 表示全部
pool_split_mode: 池划分策略"global"全局统一划分/ "per_category"按类别独立划分
train_ratio: 训练集占比范围 (0, 1)
test_questions: 测试题目集路径相对路径
run_holdout_eval: 是否在 epoch 末执行 held-out 四向评估默认 True
"""
# ── 必填字段(无默认值,来自 YAML 或 CLI) ──
workspace_dir: Path
store_dir: Path
mode: str
concurrency: int
max_steps: int
skill_mode: str
n_samples: int
questions: str
skills_version: str
prompts_version: str
epochs: int
diag_size: int
diag_correct_ratio: float
val_size: int
val_correct_ratio: float
edit_budget_start: int
edit_budget_end: int
batch_size: int
min_class_per_batch: int
eval_min_per_class: int
trainable_min_units: int
early_stop_patience: int
test_size: int
use_slow_momentum: bool
gate_e_confirm: float
gate_e_provisional: float
gate_w_net_min: int
gate_delta_min: float
gate_lambda_dir: float
gate_e_rollback: float
gate_n_max: int
gate_p_low: float
gate_p_high: float
gate_probe_quota: float
gate_gamma_decay: float
gate_cooldown_steps: int
gate_guard_err: float
skill_update_mode: str
appendix_consolidate_threshold: int
# ── 有默认值的字段(通常由 CLI 传入或可选) ──
run_id: str = ""
batch_correct_ratio: float = 0.5
momentum_samples: int = 20
seed: str = "initial"
version: str = ""
resume: bool = False
fresh: bool = False
task_types: tuple[str, ...] | None = None
pool_split_mode: str = "global"
train_ratio: float = 0.667
test_questions: str = "benchmarks/Video-MME"
run_holdout_eval: bool = True
def _validate(config: RunConfig) -> None:
"""校验 RunConfig 全部字段约束。
六层校验链mode 基础标量 编辑预算 mini-batch gate 阈值 gate 阶梯
参数:
config: 待校验的配置实例
异常:
ValueError: 任一字段值不合法
"""
_validate_mode(config)
_validate_mode_deps(config)
_validate_basic(config)
_validate_edit_budget(config)
_validate_minibatch(config)
_validate_gate(config)
def _validate_mode(config: RunConfig) -> None:
"""校验运行模式枚举合法性。
参数:
config: 待校验的配置实例
异常:
ValueError: mode 值不在合法集合中
"""
if config.mode not in _VALID_MODES:
raise ValueError(f"mode 必须为 {_VALID_MODES} 之一,实际: {config.mode!r}")
def _validate_mode_deps(config: RunConfig) -> None:
"""校验各运行模式的依赖字段(run_id、version)。
参数:
config: 待校验的配置实例
异常:
ValueError: 模式依赖字段缺失
"""
if config.mode in ("diagnose", "evolve") and not config.run_id:
raise ValueError(f"mode 为 {config.mode!r} 时必须提供 run_id。")
if config.mode in ("eval", "promote") and not config.version:
raise ValueError(f"mode 为 {config.mode!r} 时必须提供 --version。")
if config.mode == "promote" and not config.run_id:
raise ValueError("promote 必须提供 --run-id(指定 canonical eval run)。")
_validate_train_run_id(config)
def _validate_train_run_id(config: RunConfig) -> None:
"""校验 train 模式非 resume/fresh 时必须提供 run_id。
参数:
config: 待校验的配置实例
异常:
ValueError: train 模式既非 resume 也非 fresh 且缺少 run_id
"""
if config.mode != "train":
return
if config.resume or config.fresh:
return
if not config.run_id:
raise ValueError("train 非 resume/fresh 时必须提供 run_id(旧式基线 run)。")
def _validate_basic(config: RunConfig) -> None:
"""校验基础标量字段:枚举合法性与正整数约束。
参数:
config: 待校验的配置实例
异常:
ValueError: 任一基础字段值不合法
"""
if config.skill_mode not in _VALID_SKILL_MODES:
raise ValueError(
f"skill_mode 必须为 {_VALID_SKILL_MODES} 之一,实际: {config.skill_mode!r}"
)
if config.concurrency <= 0:
raise ValueError(f"concurrency 必须 > 0,实际: {config.concurrency}")
if config.max_steps <= 0:
raise ValueError(f"max_steps 必须 > 0,实际: {config.max_steps}")
if config.n_samples < 0:
raise ValueError(f"n_samples 必须 >= 0,实际: {config.n_samples}")
if config.epochs <= 0:
raise ValueError(f"epochs 必须 > 0,实际: {config.epochs}")
if config.skill_update_mode not in _VALID_SKILL_UPDATE_MODES:
raise ValueError(
f"skill_update_mode 必须为 {_VALID_SKILL_UPDATE_MODES} 之一,"
f"实际: {config.skill_update_mode!r}"
)
if config.appendix_consolidate_threshold < 1:
raise ValueError(
f"appendix_consolidate_threshold 必须 >= 1"
f"实际: {config.appendix_consolidate_threshold}"
)
if config.pool_split_mode not in _VALID_POOL_SPLIT_MODES:
raise ValueError(
f"pool_split_mode 必须为 {_VALID_POOL_SPLIT_MODES} 之一,"
f"实际: {config.pool_split_mode!r}"
)
if not (0 < config.train_ratio < 1):
raise ValueError(f"train_ratio 必须在 (0, 1) 内,实际: {config.train_ratio}")
def _validate_edit_budget(config: RunConfig) -> None:
"""校验编辑预算退火的前期/后期上限约束。
参数:
config: 待校验的配置实例
异常:
ValueError: edit_budget_start < edit_budget_end end <= 0
"""
if config.edit_budget_start < config.edit_budget_end:
raise ValueError(
f"edit_budget_start({config.edit_budget_start}) 必须 >= "
f"edit_budget_end({config.edit_budget_end})"
)
if config.edit_budget_end <= 0:
raise ValueError(f"edit_budget_end 必须 > 0,实际: {config.edit_budget_end}")
def _validate_minibatch(config: RunConfig) -> None:
"""校验 mini-batch 自进化闭环参数约束。
参数:
config: 待校验的 RunConfig 配置对象
异常:
ValueError: 任一约束被违反
关键实现细节:
pool_split_mode != "per_category" val_size 必须 >= eval_min_per_class *
_VIDEO_MME_TASK_TYPE_COUNT保证验证池能为 Video-MME 的全部 12 个任务类型
各保底 eval_min_per_class per_category 模式下跳过此硬编码 12 类保底检查
"""
if config.batch_size <= 0:
raise ValueError(f"batch_size 必须 > 0,实际: {config.batch_size}")
if not (1 <= config.min_class_per_batch < config.batch_size):
raise ValueError(
f"min_class_per_batch 必须满足 1 <= 值 < batch_size"
f"({config.batch_size}),实际: {config.min_class_per_batch}"
)
if config.eval_min_per_class < 1:
raise ValueError(f"eval_min_per_class 必须 >= 1,实际: {config.eval_min_per_class}")
if config.trainable_min_units < 1:
raise ValueError(f"trainable_min_units 必须 >= 1,实际: {config.trainable_min_units}")
if config.pool_split_mode != "per_category":
floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT
if config.val_size < floor:
raise ValueError(
f"val_size 必须 >= eval_min_per_class * {_VIDEO_MME_TASK_TYPE_COUNT}"
f"(={floor})Video-MME 共 {_VIDEO_MME_TASK_TYPE_COUNT} 个任务类型,"
f"每类需 eval_min_per_class 题保底,故验证池下限为 {floor}"
f"实际: {config.val_size}"
)
if config.early_stop_patience <= 0:
raise ValueError(f"early_stop_patience 必须 > 0,实际: {config.early_stop_patience}")
if config.test_size <= 0:
raise ValueError(f"test_size 必须 > 0,实际: {config.test_size}")
if not (0 <= config.batch_correct_ratio < 1):
raise ValueError(
f"batch_correct_ratio 必须满足 0 <= 值 < 1,实际: {config.batch_correct_ratio}"
)
if config.momentum_samples < 1:
raise ValueError(f"momentum_samples 必须 >= 1,实际: {config.momentum_samples}")
def _validate_gate(config: RunConfig) -> None:
"""校验 CE-Gate 全部参数:判据阈值 + 信息量阶梯。
参数:
config: 待校验的配置实例
异常:
ValueError: 任一 gate 参数不合法
"""
_validate_gate_thresholds(config)
_validate_gate_ladder(config)
def _validate_gate_thresholds(config: RunConfig) -> None:
"""校验 CE-Gate 判据阈值参数(e 值、净胜数、效应量、方向拒绝)。
参数:
config: 待校验的配置实例
异常:
ValueError: 任一阈值参数不合法
"""
if config.gate_e_confirm <= 1:
raise ValueError(f"gate_e_confirm 必须 > 1,实际: {config.gate_e_confirm}")
if not (1 < config.gate_e_provisional <= config.gate_e_confirm):
raise ValueError(
f"gate_e_provisional 必须在 (1, gate_e_confirm] 内,实际: {config.gate_e_provisional}"
)
if config.gate_e_rollback <= 1:
raise ValueError(f"gate_e_rollback 必须 > 1,实际: {config.gate_e_rollback}")
if config.gate_w_net_min < 1:
raise ValueError(f"gate_w_net_min 必须 >= 1,实际: {config.gate_w_net_min}")
if config.gate_delta_min < 0:
raise ValueError(f"gate_delta_min 必须 >= 0,实际: {config.gate_delta_min}")
if config.gate_lambda_dir >= 0:
raise ValueError(f"gate_lambda_dir 必须 < 0,实际: {config.gate_lambda_dir}")
def _validate_gate_ladder(config: RunConfig) -> None:
"""校验 CE-Gate 信息量阶梯参数。
参数:
config: 待校验的配置实例
异常:
ValueError: 任一阶梯参数不合法
"""
if config.gate_n_max <= 0:
raise ValueError(f"需 gate_n_max > 0,实际: n_max={config.gate_n_max}")
if not (0 <= config.gate_p_low < config.gate_p_high <= 1):
raise ValueError(
f"需 0 <= gate_p_low < gate_p_high <= 1"
f"实际: [{config.gate_p_low}, {config.gate_p_high}]"
)
if not (0 <= config.gate_probe_quota <= 1):
raise ValueError(f"gate_probe_quota 须在 [0,1],实际: {config.gate_probe_quota}")
if not (0 < config.gate_gamma_decay < 1):
raise ValueError(f"gate_gamma_decay 须在 (0,1),实际: {config.gate_gamma_decay}")
if config.gate_cooldown_steps < 1:
raise ValueError(f"gate_cooldown_steps 必须 >= 1,实际: {config.gate_cooldown_steps}")
if not (0 < config.gate_guard_err < 1):
raise ValueError(f"gate_guard_err 须在 (0,1),实际: {config.gate_guard_err}")
def _apply_env_var_overrides(data: dict) -> None:
"""从环境变量覆盖路径字段(原地修改)。
.env 文件由入口脚本 load_dotenv 加载到环境变量本函数仅从 os.environ 读取
仅覆盖 _ENV_FIELD_MAP 中声明的工程配置字段workspace_dirstore_dir
参数:
data: 待覆盖的配置字典
"""
for env_key, field_name in _ENV_FIELD_MAP.items():
env_val = os.environ.get(env_key)
if env_val is not None:
data[field_name] = env_val
def load_config(
yaml_path: Path,
cli_overrides: dict[str, object] | None = None,
) -> RunConfig:
"""从 YAML 加载配置,叠加 .env 和 CLI 覆盖层后构造 RunConfig。
三层合并优先级CLI > .env > YAML
参数:
yaml_path: YAML 配置文件路径需包含 ``harness``
cli_overrides: CLI 参数字典值为 None 表示未传入不覆盖
返回:
构造并校验后的 RunConfig 实例
异常:
FileNotFoundError: YAML 文件不存在
ValueError: 校验失败
"""
# Phase 1: 加载 YAML 基础层
with open(yaml_path, encoding="utf-8") as f:
raw: dict = yaml.safe_load(f)
# 支持嵌套 harness 段和扁平 YAML 两种格式
yaml_data: dict = raw.get("harness", raw)
# Phase 2: .env 覆盖层(仅工程配置字段)
_apply_env_var_overrides(yaml_data)
# Phase 3: CLI 覆盖层(最高优先级)
valid_fields = {f.name for f in dataclasses.fields(RunConfig)}
if cli_overrides:
for key, value in cli_overrides.items():
if value is not None and key in valid_fields:
yaml_data[key] = value
# Phase 4: 类型转换 — 路径字段转 Path
for field_name in _PATH_FIELDS:
if field_name in yaml_data:
yaml_data[field_name] = Path(yaml_data[field_name])
# Phase 4b: 类型转换 — task_types list → tuple
if "task_types" in yaml_data and yaml_data["task_types"] is not None:
yaml_data["task_types"] = tuple(yaml_data["task_types"])
# Phase 5: 构造并校验
config = RunConfig(**{k: v for k, v in yaml_data.items() if k in valid_fields})
_validate(config)
return config
+183
View File
@@ -0,0 +1,183 @@
"""按 video_id 懒加载 InferenceDeps 并路由 dispatch/prompt_builder。
每个视频的 TreeIndexTreeEnvironmentSkillRegistry 等重量级对象
只在首次访问时构建并缓存后续同视频的请求直接复用
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from loguru import logger
from app.harness.factory import InferenceDeps, build_inference_deps
if TYPE_CHECKING:
from pathlib import Path
from app.ports import EmbeddingProvider, OCRProvider
from core.protocols import LLMProvider, VLMProvider
from core.types import GeneratedQuestion
class InferenceDepsRouter:
"""按 video_id 懒加载 InferenceDeps 并路由工具调度和 prompt 构建。
职责:
1. 维护 question_id video_id 的映射表 prompt_builder 自动注册
2. (video_id, skills_dir, prompts_dir) 三元组缓存 InferenceDeps
3. 提供 create_dispatch / create_prompt_builder 工厂方法
返回的闭包符合 ToolDispatchFn / PromptBuilderFn Protocol
参数:
store_dir: store 根目录
embed_provider: 嵌入端口实例
llm: LLM 端口实例
vlm: VLM 端口实例
ocr: OCR 端口实例None 不启用
default_prompts_dir: 默认 prompt 文件目录
default_skills_dir: 默认 skill 文件目录None 则不加载 skill
skill_mode: skill 模式"auto"/"manual"/"none"
verify_vision: observe_frame 是否执行验证轮
anchor: view_node 是否启用行号锚模式
assemble_mode: 锚模式装配形态
"""
def __init__(
self,
*,
store_dir: Path,
embed_provider: EmbeddingProvider,
llm: LLMProvider,
vlm: VLMProvider,
ocr: OCRProvider | None,
default_prompts_dir: Path,
default_skills_dir: Path | None,
skill_mode: str,
verify_vision: bool,
anchor: bool,
assemble_mode: str,
) -> None:
self._store_dir = store_dir
self._embed = embed_provider
self._llm = llm
self._vlm = vlm
self._ocr = ocr
self._default_prompts_dir = default_prompts_dir
self._default_skills_dir = default_skills_dir
self._skill_mode = skill_mode
self._verify_vision = verify_vision
self._anchor = anchor
self._assemble_mode = assemble_mode
self._deps_cache: dict[tuple[str, str, str], InferenceDeps] = {}
self._qid_to_vid: dict[str, str] = {}
def create_dispatch(self, *, skills_dir: Path | None = None) -> Any:
"""创建工具调度闭包,按 session_id 路由到对应视频的 InferenceDeps。
参数:
skills_dir: skill 文件目录覆盖None 使用默认值
返回:
符合 ToolDispatchFn 签名的 async 闭包
"""
effective_skills = skills_dir or self._default_skills_dir
async def _dispatch(
tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
) -> str:
"""按 session_id 查找视频 → 获取缓存 deps → 委托执行。"""
session_id = context.get("session_id")
if not session_id or session_id not in self._qid_to_vid:
raise KeyError(
f"未注册的 session_id={session_id!r},已注册 {len(self._qid_to_vid)} 条映射"
)
video_id = self._qid_to_vid[session_id]
deps = self._ensure_deps(video_id, effective_skills, self._default_prompts_dir)
return await deps.tool_dispatch_fn(tool_name, args, context=context)
return _dispatch
def create_prompt_builder(
self,
*,
skills_dir: Path | None = None,
prompts_dir: Path | None = None,
) -> Any:
"""创建 prompt 构建闭包,自动注册 qid→vid 映射并路由到对应视频的 deps。
参数:
skills_dir: skill 文件目录覆盖None 使用默认值
prompts_dir: prompt 文件目录覆盖None 使用默认值
返回:
符合 PromptBuilderFn 签名的闭包
"""
effective_skills = skills_dir or self._default_skills_dir
effective_prompts = prompts_dir or self._default_prompts_dir
def _builder(qa: GeneratedQuestion) -> tuple[str, str]:
"""注册 qid→vid 映射 → 获取缓存 deps → 委托构建 prompt。"""
self._qid_to_vid[qa.question_id] = qa.video_id
deps = self._ensure_deps(qa.video_id, effective_skills, effective_prompts)
return deps.prompt_builder(qa)
return _builder
def _ensure_deps(
self,
video_id: str,
skills_dir: Path | None,
prompts_dir: Path,
) -> InferenceDeps:
"""按 (video_id, skills_dir, prompts_dir) 三元组缓存 InferenceDeps。
参数:
video_id: 视频标识
skills_dir: skill 文件目录
prompts_dir: prompt 文件目录
返回:
缓存命中或新建的 InferenceDeps 实例
"""
key = (video_id, str(skills_dir), str(prompts_dir))
if key not in self._deps_cache:
self._deps_cache[key] = self._build_deps(video_id, skills_dir, prompts_dir)
logger.debug("InferenceDeps 已缓存: video_id={}", video_id)
return self._deps_cache[key]
def _build_deps(
self,
video_id: str,
skills_dir: Path | None,
prompts_dir: Path,
) -> InferenceDeps:
"""调用 build_inference_deps 构建 InferenceDeps 实例。
参数:
video_id: 视频标识
skills_dir: skill 文件目录
prompts_dir: prompt 文件目录
返回:
新建的 InferenceDeps 实例
"""
return build_inference_deps(
store_dir=self._store_dir,
video_id=video_id,
prompts_dir=prompts_dir,
skills_dir=skills_dir,
skill_mode=self._skill_mode,
embed_provider=self._embed,
llm=self._llm,
vlm=self._vlm,
ocr=self._ocr,
verify_vision=self._verify_vision,
anchor=self._anchor,
assemble_mode=self._assemble_mode,
)
def clear_cache(self) -> None:
"""清空 deps 缓存和 qid→vid 映射表。"""
self._deps_cache.clear()
self._qid_to_vid.clear()
+155
View File
@@ -0,0 +1,155 @@
"""推理依赖工厂 — 组装一次推理所需的全套依赖。
TreeIndex 加载TreeEnvironment 构建SkillRegistry 发现
SearchToolDispatcher 装配PromptManager 初始化等步骤封装为
单一工厂函数 ``build_inference_deps``返回不可变的 ``InferenceDeps``
调用方runner / inference只需传入配置参数无需了解内部装配逻辑
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from loguru import logger
from app.search.prompt import PromptManager
from app.search.skills import discover_skills
from app.search.tools import SearchToolDispatcher
from app.tree.environment import TreeEnvironment
from app.tree.index import TreeIndex
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
from app.ports import EmbeddingProvider, OCRProvider
from core.protocols import LLMProvider, VLMProvider
from core.types import GeneratedQuestion
@dataclass(frozen=True)
class InferenceDeps:
"""跑一次推理所需的全套依赖(不含 HarnessLog,其生命周期由调用方管理)。
属性:
llm: LLM 端口实例
tool_dispatch_fn: SearchToolDispatcher.dispatch 的绑定方法
prompt_builder: (GeneratedQuestion) -> (system_prompt, user_prompt)
"""
llm: LLMProvider
tool_dispatch_fn: Callable[..., Any]
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]]
def build_inference_deps(
*,
store_dir: Path,
video_id: str,
prompts_dir: Path,
skills_dir: Path | None,
skill_mode: str,
embed_provider: EmbeddingProvider,
llm: LLMProvider,
vlm: VLMProvider,
ocr: OCRProvider | None,
verify_vision: bool,
anchor: bool,
assemble_mode: str,
) -> InferenceDeps:
"""组装一次推理所需的全套依赖。
参数:
store_dir: store 根目录包含 videos/{video_id}/tree.json
video_id: 视频标识
prompts_dir: prompt 文件目录
skills_dir: skill 文件目录None 则不加载 skill
skill_mode: skill 模式"auto"/"manual"/"none"
embed_provider: 嵌入端口实例
llm: LLM 端口实例
vlm: VLM 端口实例
ocr: OCR 端口实例None 不启用
verify_vision: observe_frame 是否执行验证轮
anchor: view_node 是否启用行号锚模式
assemble_mode: 锚模式装配形态
返回:
InferenceDeps 实例
异常:
FileNotFoundError: tree.json 不存在
"""
# Phase 1: 加载 TreeIndex
tree_path = store_dir / "videos" / video_id / "tree.json"
if not tree_path.exists():
raise FileNotFoundError(f"树索引文件不存在: {tree_path}")
tree_index = TreeIndex.load_json(str(tree_path))
logger.info("已加载 TreeIndex: video_id={}, L1 节点数={}", video_id, len(tree_index.roots))
# Phase 2: 构建 TreeEnvironment
frames_dir = store_dir / "videos" / video_id / "frames"
env = TreeEnvironment(index=tree_index, frames_dir=frames_dir)
# Phase 3: 构建 SkillRegistry
skills = None
always_skills_text = ""
task_skill_map: dict[str, str] = {}
catalog_text = ""
if skills_dir is not None:
always_skills_text, task_skill_map, catalog_text, skills = discover_skills(skills_dir)
logger.info(
"已发现 skills: always={} 字符, task_map={}",
len(always_skills_text),
len(task_skill_map),
)
# Phase 4: 构建 SearchToolDispatcher
dispatcher = SearchToolDispatcher(
env,
tool_llm=llm,
vlm=vlm,
ocr=ocr,
prompts_dir=prompts_dir,
skills=skills,
embed_fn=embed_provider.embed,
verify_vision=verify_vision,
anchor=anchor,
assemble_mode=assemble_mode,
)
# Phase 5: 构建 PromptManager + _prompt_builder 闭包
pm = PromptManager(prompts_dir)
l1_ids = [root.id for root in tree_index.roots]
def _prompt_builder(qa: GeneratedQuestion) -> tuple[str, str]:
"""为单条题目生成 (system_prompt, user_prompt)。
参数:
qa: 生成的题目实例
返回:
(system_prompt, user_prompt) 二元组
"""
system = pm.build_inference_prompt(
skill_mode,
qa.task_type,
always_skills_text,
task_skill_map,
catalog_text,
)
user = pm.format_user_prompt(
qa.question,
list(qa.options),
l1_ids,
qa.task_type,
)
return system, user
logger.info("InferenceDeps 组装完成: video_id={}, skill_mode={}", video_id, skill_mode)
return InferenceDeps(
llm=llm,
tool_dispatch_fn=dispatcher.dispatch,
prompt_builder=_prompt_builder,
)
+404
View File
@@ -0,0 +1,404 @@
"""CE-Gate 信息量阶梯与基线缓存(unit 粒度,核心算法保真 #5)。
阶梯每题型一条gate 的出题顺序表键为 **unit_id**single unit_id
等于 question_idAR pair 折叠为一个单元unit_id 等于共享 pair_id冷启动
FRESH用种子基线的**单元级**对错两档粗排 unit 高优先 2:1 交错 + 全错
unit probe_quota 探针插尾epoch >=1 用非 gate run 观测**折叠成单元观测**后做
gamma-EMA 更新 p_hat按信息量 p_hat(1-p_hat) 降序 p_hat 不在 [p_low, p_high]
单元错 = 该单元任一成员错AR pair 双向 AND防泄露铁律gate rollout 永不
回流 p_hat调用方以 run_id "_gate_" 过滤观测源本迁移不改此过滤
持久化门控gate_pools.json schema_version当前 = 2unit 旧版无
schema_versionv1qid 加载时**直接报错**拒绝静默混用 qid/unit
BaselineCache基线侧单元级对错缓存 = (task_type, skill_hash,
prompts_version, unit_id) 内容寻址无显式失效JSON 持久化到 workspace
resume 后合法复用已冻结阶梯上的新鲜 draw
"""
from __future__ import annotations
import hashlib
import json
import os
import random
from dataclasses import dataclass
from typing import TYPE_CHECKING
from loguru import logger
from app.harness.question_units import build_units, unit_correctness
if TYPE_CHECKING:
from pathlib import Path
from core.types import GeneratedQuestion, QuestionUnit
# gate_pools.json 结构版本。v1(隐式、无此字段)为逐题 qid 键的存量格式;
# v2 起改为 unit_id 键。load 时严格校验,不匹配即报错(不静默迁移/混用)。
SCHEMA_VERSION = 2
def skill_hash(content: str) -> str:
"""对 skill 正文取 sha1 摘要,作缓存键的内容维度。
参数:
content: skill 文件全文基线侧为解析后生效文件的正文
返回:
sha1 十六进制摘要
"""
return hashlib.sha1(content.encode("utf-8")).hexdigest()
@dataclass
class LadderEntry:
"""阶梯单元:题目单元与其估计答对率。
字段:
unit_id: 单元唯一标识single 等于 question_idAR pair 等于共享 pair_id
p_hat: 估计答对率冷启动为 Beta(1,1) 平滑的单次观测后验均值
=1/3=2/3此后经 gamma-EMA 更新
"""
unit_id: str
p_hat: float
def build_cold_entries(
units: list[QuestionUnit],
correctness: dict[str, bool],
probe_quota: float,
seed: int,
) -> list[LadderEntry]:
"""冷启动排序(unit 粒度):错 unit 高优先 2:1 交错 + 全错 unit 探针插尾。
参数:
units: 该题型的全部候选单元已排除 test AR pair 已折叠成单元
correctness: question_id -> 种子基线是否答对900 题全量逐题对错
单元级对错由 unit_correctness(strict=False) 折叠任一成员错 单元错
probe_quota: 从错 unit 中随机抽出插到梯尾的探针比例"解锁新能力"盲区
seed: 洗牌种子保证确定性重建
返回:
排序后的 LadderEntry 列表=unit_idp_hat Beta(1,1) 平滑=1/3
=2/3 warm 阶段 gamma-EMA / 信息量排序自然衔接
关键实现细节:
与逐题版**同公式同比例同顺序**仅把调度粒度从题换成单元 unit
unit 各自固定种子洗牌 -> probe_quota 从错 unit 抽探针 -> 剩余按
错错对 2:1 交错一方耗尽后顺排另一方-> 探针追加尾部
"""
rng = random.Random(seed)
wrong = [u for u in units if not unit_correctness(u, correctness, strict=False)]
right = [u for u in units if unit_correctness(u, correctness, strict=False)]
rng.shuffle(wrong)
rng.shuffle(right)
n_probe = int(len(wrong) * probe_quota)
probes, wrong_main = wrong[:n_probe], wrong[n_probe:]
interleaved: list[QuestionUnit] = []
wi, ri = 0, 0
while wi < len(wrong_main) or ri < len(right):
for _ in range(2):
if wi < len(wrong_main):
interleaved.append(wrong_main[wi])
wi += 1
if ri < len(right):
interleaved.append(right[ri])
ri += 1
interleaved.extend(probes)
def _p0(u: QuestionUnit) -> float:
return 2 / 3 if unit_correctness(u, correctness, strict=False) else 1 / 3
return [LadderEntry(u.unit_id, _p0(u)) for u in interleaved]
def order_ladder(entries: list[LadderEntry], p_low: float, p_high: float) -> list[LadderEntry]:
"""warm 排序:剔 p_hat 不在 [p_low, p_high] 的零信息题,按信息量 p_hat(1-p_hat) 降序。
参数:
entries: 待排序的阶梯单元
p_low / p_high: p_hat 保留区间
返回:
过滤并排序后的新列表稳定排序同信息量保持原相对序
"""
kept = [e for e in entries if p_low <= e.p_hat <= p_high]
return sorted(kept, key=lambda e: e.p_hat * (1 - e.p_hat), reverse=True)
@dataclass
class GatePools:
"""全部题型的阶梯容器,含构建种子与数据指纹(确定性重建凭据)。
字段:
entries: task_type -> 冷启动序 LadderEntry 列表warm 排序在取用时做
保持存储序稳定避免每次更新重写全表顺序
seed: 冷启动洗牌种子
fingerprint: 构建输入指纹基线 run_id + 题集 hash resume 校验用
"""
entries: dict[str, list[LadderEntry]]
seed: int
fingerprint: str
def ladder_for(
self,
task_type: str,
exclude_units: set[str],
p_low: float,
p_high: float,
cold: bool,
) -> list[str]:
"""取该题型的 gate 出题序(unit_id 列表),排除本 step 进化案例包所在单元。
参数:
task_type: 目标题型
exclude_units: step 案例包failure/success cases所在单元的
unit_id防止在"刚学的那道题"上自测 **unit** 排除命中单元
整体剔除避免只排 AR pair 半个成员而向 gate 池灌入半个 pair
p_low / p_high: warm 阶段的 p_hat 保留区间
cold: True 表示尚无 epoch 级观测epoch 1用冷启动存储序
False order_ladder 信息量排序
返回:
排除后的有序 unit_id 列表
异常:
ValueError: 该题型无阶梯冷启动构建缺失或该题型阶梯为空
"""
if task_type not in self.entries:
raise ValueError(f"task_type={task_type} 无阶梯,冷启动构建缺失该题型")
pool = self.entries[task_type]
if not pool:
raise ValueError(f"task_type={task_type} 阶梯为空,无可出题目")
ordered = pool if cold else order_ladder(pool, p_low, p_high)
return [e.unit_id for e in ordered if e.unit_id not in exclude_units]
def update_probs(
self,
per_q_observations: dict[str, bool],
units_by_id: dict[str, QuestionUnit],
gamma: float,
) -> None:
"""gamma-EMA 更新 p_hat:先把逐题观测折叠成单元观测,再按 unit_id 匹配更新。
p_hat <- gamma * p_hat + (1-gamma) * unit_obs只更新"整个单元都被观测到"
的单元单元观测 = 成员逐题对错的 AND任一成员错 单元错折叠是必需的
AR pair unit_id pair_id若直接按 unit_id 去逐题观测里匹配将永不命中
导致 gamma-EMA 停摆核心算法保真 #5)。
参数:
per_q_observations: question_id -> epoch gate run 的最新逐题对错
调用方必须已按 run_id 过滤掉 gate rollout防泄露铁律
units_by_id: unit_id -> QuestionUnit用于把逐题观测折叠成单元观测
gamma: EMA 衰减系数
关键实现细节:
单元只有在其**全部**成员都出现在 per_q_observations 时才更新半观测
AR pair 只见一半跳过避免用不完整证据污染 p_hat
"""
for entries in self.entries.values():
for e in entries:
unit = units_by_id.get(e.unit_id)
if unit is None:
continue
if not all(q.question_id in per_q_observations for q in unit.questions):
continue
unit_correct = all(per_q_observations[q.question_id] for q in unit.questions)
obs = 1.0 if unit_correct else 0.0
e.p_hat = gamma * e.p_hat + (1 - gamma) * obs
def save(self, path: Path) -> None:
"""原子写 gate_pools.json.tmp 再 replace),落 schema_version + unit_id 键。
参数:
path: 目标 JSON 路径
"""
payload = {
"schema_version": SCHEMA_VERSION,
"seed": self.seed,
"fingerprint": self.fingerprint,
"entries": {
t: [{"unit_id": e.unit_id, "p_hat": e.p_hat} for e in es]
for t, es in self.entries.items()
},
}
tmp = path.with_suffix(".json.tmp")
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(tmp, path)
@classmethod
def load(cls, path: Path) -> GatePools:
"""从 gate_pools.json 恢复;schema_version 不匹配直接报错(不静默混用)。
参数:
path: gate_pools.json 路径
返回:
复活的 GatePools
异常:
RuntimeError: schema_version存量 v1qid 或版本不等于
SCHEMA_VERSION拒绝把 qid 键当 unit 键静默复用 FRESH 重建
"""
d = json.loads(path.read_text(encoding="utf-8"))
version = d.get("schema_version")
if version != SCHEMA_VERSION:
raise RuntimeError(
f"gate_pools.json schema_version={version!r} 与当前 {SCHEMA_VERSION} 不符"
f"(存量 qid 键池不可当 unit 键复用),请删除后 FRESH 重建: {path}"
)
return cls(
entries={
t: [LadderEntry(x["unit_id"], x["p_hat"]) for x in es]
for t, es in d["entries"].items()
},
seed=d["seed"],
fingerprint=d["fingerprint"],
)
def build_or_load_gate_pools(
workspace_dir: Path,
questions: list[GeneratedQuestion],
test_qids: set[str],
baseline_correctness: dict[str, bool],
task_types: list[str],
probe_quota: float,
seed: int,
baseline_run_id: str,
) -> GatePools:
"""gate 阶梯获取入口:gate_pools.json 存在且指纹一致则加载,否则冷启动构建。
参数:
workspace_dir: workspace 根目录gate_pools.json 落其下
questions: benchmark 全量题900
test_qids: held-out test 池题目 id阶梯题源必须排除
baseline_correctness: 种子基线 900 题全量对错从基线 run db
task_types: 参与进化的题型列表
probe_quota: 冷启动探针比例
seed: 冷启动洗牌种子
baseline_run_id: 指纹成分
返回:
GatePools
关键实现细节:
指纹 = sha1(baseline_run_id| qid|seed|probe_quota|task_types|test_qids)
指纹不一致题集/基线/参数变了直接报错FRESH 语义下不该发生
防御性拒绝而非静默重建
"""
joined = ",".join(sorted(q.question_id for q in questions))
fp_src = (
f"{baseline_run_id}|{joined}|{seed}|{probe_quota}"
f"|{','.join(sorted(task_types))}|{','.join(sorted(test_qids))}"
)
fingerprint = hashlib.sha1(fp_src.encode()).hexdigest()
path = workspace_dir / "gate_pools.json"
if path.exists():
pools = GatePools.load(path)
if pools.fingerprint != fingerprint:
raise RuntimeError(
f"gate_pools.json 指纹不一致(题集或基线变更),拒绝静默重建: {path}"
)
return pools
entries: dict[str, list[LadderEntry]] = {}
for t in task_types:
units = _task_units_excluding_test(questions, t, test_qids)
if not units:
raise ValueError(f"task_type={t} 无非 test 单元,无法建阶梯")
entries[t] = build_cold_entries(units, baseline_correctness, probe_quota, seed)
logger.info("gate 阶梯[{}]: {} 单元(冷启动)", t, len(entries[t]))
pools = GatePools(entries=entries, seed=seed, fingerprint=fingerprint)
pools.save(path)
return pools
def _task_units_excluding_test(
questions: list[GeneratedQuestion], task_type: str, test_qids: set[str]
) -> list[QuestionUnit]:
"""取某题型的非 test 候选单元:先按 unit 折叠,再整体排除含 test 成员的单元。
先折叠后排除保证 AR pair 不被拆半否则半个 pair 交给下游会触发 build_units
孤儿 fail-fastsingle 单元等价于逐题排除核心算法保真 #5)。
参数:
questions: benchmark 全量题
task_type: 目标题型
test_qids: held-out test 池题目 id
返回:
该题型下不含任何 test 成员的候选单元列表
"""
pool = [q for q in questions if q.task_type == task_type]
return [
u for u in build_units(pool) if all(q.question_id not in test_qids for q in u.questions)
]
class BaselineCache:
"""基线侧单元级对错缓存(内容寻址,JSON 持久化)。
= (task_type, skill_hash, prompts_version, unit_id)任何影响该题型
有效 skill 的变化含共享 default-strategy.md 被他类 accept 改写
都使 skill_hash 变化缓存自然 missprompts 版本变化同理unit_id 维度
使 single 题以自身 question_idAR pair 以共享 pair_id 寻址缓存单元级
对错pair 双向 AND 折叠后一个布尔
"""
def __init__(self, path: Path) -> None:
"""加载或初始化缓存文件。
参数:
path: 缓存 JSON 路径workspace/baseline_cache.json
"""
self._path = path
self._store: dict[str, bool] = {}
if path.exists():
self._store = json.loads(path.read_text(encoding="utf-8"))
@staticmethod
def _key(task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> str:
"""拼缓存键(四维内容寻址,第四维为 unit_id)。"""
return f"{task_type}|{s_hash}|{prompts_version}|{unit_id}"
def get(self, task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> bool | None:
"""读缓存;未命中返回 None。
参数:
task_type: 题型
s_hash: 基线侧生效 skill 文件的内容哈希
prompts_version: 当前 prompts 版本
unit_id: 单元 idsingle=question_idAR pair=pair_id
返回:
缓存的单元级对错未命中 None
"""
return self._store.get(self._key(task_type, s_hash, prompts_version, unit_id))
def put(
self, task_type: str, s_hash: str, prompts_version: str, unit_id: str, correct: bool
) -> None:
"""写缓存并落盘(原子写,gate 频度低、全量重写成本可忽略)。
参数:
task_type / s_hash / prompts_version / unit_id: 缓存键四维
correct: 基线侧该单元对错AR pair 双向 AND 折叠后一个布尔
关键实现细节:
先盘后存新条目先原子落盘tmp + os.replace成功后才更新
内存磁盘写失败时内存与磁盘一致均无新条目无分裂窗口
"""
updated = {
**self._store,
self._key(task_type, s_hash, prompts_version, unit_id): correct,
}
tmp = self._path.with_suffix(".json.tmp")
tmp.write_text(json.dumps(updated, ensure_ascii=False), encoding="utf-8")
os.replace(tmp, self._path)
self._store = updated
+590
View File
@@ -0,0 +1,590 @@
"""async 推理编排 — 训练循环的 forward()。
TRM4 core/harness/inference.py (~560 ) 迁移重大重构:
- 同步 ThreadPoolExecutor asyncio.Semaphore + asyncio.gather
- LLMClient.from_env() 每题构造 llm: LLMProvider 注入共享
- SentenceTransformer/OCR 内部构造 调用方通过 tool_dispatch_fn 注入
- run_id 必传空串 ValueError
- _aggregate_results 从内存 results 聚合 DB 回读
- record_run 由调用方Runner负责
- prompt 构建由调用方注入 prompt_builder
"""
from __future__ import annotations
import asyncio
import json
import sqlite3
from collections import defaultdict
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from loguru import logger
from app.harness.question_units import build_units, unit_correctness
from core.agent.loop import AgentLoop
if TYPE_CHECKING:
from collections.abc import Callable
from app.harness.log import HarnessLog
from core.agent.types import LoopResult
from core.protocols import LLMProvider
from core.types import GeneratedQuestion, QuestionUnit
@dataclass(frozen=True)
class InferenceResult:
"""推理聚合结果(正确率按 unit 粒度)。
属性:
run_id: 运行标识
accuracy: unit 级正确率correct / total
total: unit 总数single + pair 孤儿 pair 已剔除不计入
correct: 正确 unit single 单题正确pair original/mirror 双向 AND
per_task_type: 按题型分组的 unit 级指标 {task_type: {accuracy, total, correct}}
steps_mean: 平均步数record 粒度逐题溯源
token_usage: token 总用量 {prompt_tokens, completion_tokens}record 粒度
stop_reason_counts: 终止原因计数 {reason: count}record 粒度
"""
run_id: str
accuracy: float
total: int
correct: int
per_task_type: dict[str, dict]
steps_mean: float
token_usage: dict[str, int]
stop_reason_counts: dict[str, int]
# ---------------------------------------------------------------------------
# 表 Schema 定义(5 张表,保留 TRM4 全部 schema
# ---------------------------------------------------------------------------
PREDICTIONS_SCHEMA: dict[str, str] = {
"video_id": "TEXT",
"question_id": "TEXT",
"task_type": "TEXT",
"prediction": "TEXT",
"answer": "TEXT",
"evidence": "TEXT",
"reasoning": "TEXT",
"steps_used": "INTEGER",
"prompt_tokens": "INTEGER",
"completion_tokens": "INTEGER",
"stop_reason": "TEXT",
"steps_json": "JSON",
}
TRACES_SCHEMA: dict[str, str] = {
"video_id": "TEXT",
"question_id": "TEXT",
"step": "INTEGER",
"tool_name": "TEXT",
"tool_args": "JSON",
"tool_output": "TEXT",
"thought": "TEXT",
}
VALIDATION_FLAGS_SCHEMA: dict[str, str] = {
"video_id": "TEXT",
"question_id": "TEXT",
"has_l3_visit": "INTEGER",
"l1_count": "INTEGER",
"l2_count": "INTEGER",
"l3_count": "INTEGER",
}
ANCHOR_CHECK_SCHEMA: dict[str, str] = {
"video_id": "TEXT",
"question_id": "TEXT",
"step": "INTEGER",
"n_assertions": "INTEGER",
"n_anchored": "INTEGER",
"n_illegal": "INTEGER",
"n_expanded": "INTEGER",
"n_trunc": "INTEGER",
"output_chars": "INTEGER",
}
OF_HEALTH_SCHEMA: dict[str, str] = {
"video_id": "TEXT",
"question_id": "TEXT",
"step": "INTEGER",
"ocr_injected": "INTEGER",
"ocr_chars": "INTEGER",
"ocr_failed": "INTEGER",
"discrepancy": "INTEGER",
"abstain": "INTEGER",
}
# ---------------------------------------------------------------------------
# 内部工具
# ---------------------------------------------------------------------------
class _DispatcherAdapter:
"""将裸 async callable 包装为 ToolDispatcher Protocol 实例。
AgentLoop 要求 ToolDispatcher dispatch 方法 run_inference
接收的 tool_dispatch_fn 是裸 async callable此适配器桥接两者
参数:
fn: async def (tool_name, args, *, context) -> str
"""
def __init__(self, fn: Callable[..., Any]) -> None:
self._fn = fn
async def dispatch(
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
) -> str:
"""转发工具调用给被包装的 callable。"""
return await self._fn(tool_name, args, context=context)
def _to_text_field(value: Any) -> str:
"""把 prediction 的 evidence/reasoning 归一为可入库的文本。
LLM 有时把这些字段返回成 list dict而非字符串sqlite 无法绑定
非标量类型直接入库会抛 ProgrammingError 致该题丢失预测行进而触发
rollout 完整性护栏中止整轮凡非 str 一律 JSON 序列化为文本
参数:
value: evidence/reasoning 原始值可能是 str/list/dict
返回:
可直接入库的字符串
"""
if isinstance(value, str):
return value
return json.dumps(value, ensure_ascii=False)
def _normalize_prediction(answer: object) -> str | None:
"""归一化 prediction 落库值。
LLM 提交的 answer 有时是 list/dict {'answer': ['B']}sqlite 无法绑定
非标量类型直接入库会抛 ProgrammingError 击穿整轮 gatherNone 保留INFRA
预测语义供正确率判定天然计错str 原样其余 JSON 序列化为文本
参数:
answer: LoopResult.result 中的 answer 原始值可能是 None/str/list/dict
返回:
None保留空预测语义或可直接入库的字符串
"""
if answer is None or isinstance(answer, str):
return answer
return _to_text_field(answer)
def _zero_result(run_id: str) -> InferenceResult:
"""空记录时的零值 InferenceResult。
参数:
run_id: 运行标识
返回:
全零的 InferenceResult
"""
return InferenceResult(
run_id=run_id,
accuracy=0.0,
total=0,
correct=0,
per_task_type={},
steps_mean=0.0,
token_usage={"prompt_tokens": 0, "completion_tokens": 0},
stop_reason_counts={},
)
def _group_by_task_type(graded: list[tuple[QuestionUnit, bool]]) -> dict[str, dict[str, Any]]:
"""按 task_type 分组聚合 unit 级正确率指标。
pair 单元整体计 1 unit归入其 task_typesingle 单元计 1 unit
参数:
graded: (单元, 该单元是否整体正确) 元组列表
返回:
{task_type: {accuracy, total, correct}} 映射unit 粒度
"""
task_groups: dict[str, list[bool]] = defaultdict(list)
for unit, is_correct in graded:
task_groups[unit.task_type].append(is_correct)
per_task_type: dict[str, dict[str, Any]] = {}
for task_type, verdicts in task_groups.items():
t_total = len(verdicts)
t_correct = sum(verdicts)
per_task_type[task_type] = {
"accuracy": t_correct / t_total,
"total": t_total,
"correct": t_correct,
}
return per_task_type
def _is_valid_pair(group: list[GeneratedQuestion]) -> bool:
"""判定同一 pair_id 分组是否为合法孪生对(恰好 1 original + 1 mirror,无多余)。
要求分组总数恰为 2 且角色齐备唯一有额外非法 role 记录total>2或角色
缺失/重复均视为非法交由调用方剔除防非法记录混入 build_units
参数:
group: 归属同一 pair_id 的题目列表
返回:
合法孪生对为 True否则 False
"""
if len(group) != 2:
return False
originals = sum(1 for q in group if q.question_role == "pair_original")
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
return originals == 1 and mirrors == 1
def _drop_orphan_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
"""剔除收不齐 2 条 / 角色非法的孤儿 pair,告警不静默。
每条题目均会各答一次并逐题落库能否合成 pair 单元仅取决于 questions
是否同时含该 pair_id original + mirror且无多余非法记录非法者告警并
整对剔除使后续 build_units 只面对合法孪生对不触发 fail-fast孤儿 unit
不计入 total对齐设计 §8 聚合入口的"告警 + 剔除"
参数:
questions: 待聚合的题目列表可混含 single 与孪生对成员
返回:
可安全交给 build_units 的题目列表single 全保留pair 仅保留合法成对者
"""
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
singles: list[GeneratedQuestion] = []
for q in questions:
if q.pair_id:
by_pair[q.pair_id].append(q)
else:
singles.append(q)
kept_pairs: list[GeneratedQuestion] = []
for pair_id, group in by_pair.items():
if _is_valid_pair(group):
kept_pairs.extend(group)
else:
logger.warning(
"孤儿 pair {}:非法配对(total={}),剔除该 unit 不计入 total",
pair_id,
len(group),
)
return singles + kept_pairs
def _per_question_correctness(records: list[dict[str, Any]]) -> dict[str, bool]:
"""由逐题 record 构造 question_id → 该题作答是否正确 的映射。
prediction None作答异常时与 answer 不相等 False天然计错
参数:
records: _run_single_question 返回的逐题 record 列表
返回:
{question_id: prediction == answer} 映射 unit_correctness 取值
"""
return {r["question_id"]: r["prediction"] == r["answer"] for r in records}
def _grade_unit(unit: QuestionUnit, per_q: dict[str, bool]) -> bool:
"""判定单元整体正确性,缺 prediction 时 fail-loud(带上下文)。
_drop_orphan_pairs 已剔除孤儿/非法配对正常情况下 unit 内每题都应有对应
record若仍缺失说明聚合不变量被破坏 records questions 不同源此处
显式抛带上下文的 ValueErrorfail-loud catch/不跳过/不兜底而非放任
unit_correctness 抛裸 KeyError 丢失定位信息
参数:
unit: 待判定单元
per_q: question_id 该题是否作答正确 的映射
返回:
单元整体是否正确single 即单题正确pair 走双向 AND
异常:
ValueError: unit 内某 question_id 不在 per_q 聚合不变量被破坏
"""
missing = [q.question_id for q in unit.questions if q.question_id not in per_q]
if missing:
raise ValueError(
f"unit {unit.unit_id} 的 question {missing} 缺 prediction"
"_drop_orphan_pairs 后不应发生,聚合不变量被破坏)"
)
return unit_correctness(unit, per_q)
def _aggregate_results(
records: list[dict[str, Any]],
questions: list[GeneratedQuestion],
run_id: str,
) -> InferenceResult:
"""从内存 records + 题目列表按 unit 粒度聚合推理指标。
逐题 record 保留逐题溯源token/steps/stop_reason 诊断仍按 record 汇总
正确率则按 unit 粒度计single 1AR pair build_units 收齐 original +
mirror 后走 unit_correctness 的双向 AND 判定整对计 1 unit孤儿 pair
_drop_orphan_pairs 中告警 + 剔除不计入 total
参数:
records: _run_single_question 返回的逐题 record 列表
questions: records 对应的题目列表提供 pair_id/question_role 元数据
run_id: 当前运行标识
返回:
InferenceResult 冻结实例total/correct/per_task_type unit 粒度
"""
if not records:
return _zero_result(run_id)
per_q = _per_question_correctness(records)
units = build_units(_drop_orphan_pairs(questions))
graded = [(unit, _grade_unit(unit, per_q)) for unit in units]
total = len(graded)
correct = sum(1 for _, is_correct in graded if is_correct)
stop_counts: dict[str, int] = defaultdict(int)
for r in records:
stop_counts[r["stop_reason"]] += 1
n_records = len(records)
return InferenceResult(
run_id=run_id,
accuracy=correct / total if total else 0.0,
total=total,
correct=correct,
per_task_type=_group_by_task_type(graded),
steps_mean=sum(r["steps_used"] for r in records) / n_records,
token_usage={
"prompt_tokens": sum(r["prompt_tokens"] for r in records),
"completion_tokens": sum(r["completion_tokens"] for r in records),
},
stop_reason_counts=dict(stop_counts),
)
# ---------------------------------------------------------------------------
# 单题推理
# ---------------------------------------------------------------------------
async def _run_single_question(
qa: GeneratedQuestion,
*,
llm: LLMProvider,
tool_dispatch_fn: Callable[..., Any],
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]],
log: HarnessLog,
max_steps: int,
plugins: list[object],
run_id: str,
) -> dict[str, Any]:
"""执行单道题目的 Agent 推理。
悲观默认值record 初始 stop_reason="error"成功后覆盖
prediction 必落库log.insert try/except 之后无论成败
参数:
qa: 待推理的题目
llm: LLMProvider 共享实例
tool_dispatch_fn: async 工具调度函数 (tool_name, args, *, context) -> str
prompt_builder: (GeneratedQuestion) -> (system_prompt, user_prompt)
log: HarnessLog 实例线程安全
max_steps: AgentLoop 最大步数
plugins: pluggy 插件列表
run_id: 运行标识用作 cache_saltrun_id _e{epoch} 天然跨 epoch 重采样
epoch 续跑命中缓存算法 #10 透传)。
返回:
预测结果字典 video_id, question_id, prediction, answer
"""
# run_id 必须显式入 record:HarnessLog.insert 缺省用**实例** run_id 填充,
# 连续并发 gate 共享单一 gate_log(实例 run_id 为 step 级)时,各臂行必须
# 落自己的臂 run_id,否则 validate 回读 _load_run_rows(臂 run_id) 为空。
record: dict[str, Any] = {
"run_id": run_id,
"video_id": qa.video_id,
"question_id": qa.question_id,
"task_type": qa.task_type,
"prediction": None,
"answer": qa.answer,
"evidence": "",
"reasoning": "",
"steps_used": 0,
"prompt_tokens": 0,
"completion_tokens": 0,
"stop_reason": "error", # 悲观默认
"steps_json": "[]",
}
try:
system_prompt, user_prompt = prompt_builder(qa)
dispatcher = _DispatcherAdapter(tool_dispatch_fn)
loop = AgentLoop(llm, max_steps=max_steps)
loop_result: LoopResult = await loop.run(
system_prompt,
user_prompt,
dispatcher,
plugins=plugins,
session_id=qa.question_id,
cache_salt=run_id,
)
result_dict = loop_result.result if isinstance(loop_result.result, dict) else {}
evidence = _to_text_field(result_dict.get("evidence", ""))
reasoning = _to_text_field(result_dict.get("reasoning", ""))
record.update(
{
"prediction": _normalize_prediction(result_dict.get("answer")),
"evidence": evidence,
"reasoning": reasoning,
"steps_used": loop_result.steps_used,
"prompt_tokens": loop_result.token_usage["prompt_tokens"],
"completion_tokens": loop_result.token_usage["completion_tokens"],
"stop_reason": loop_result.stop_reason,
"steps_json": json.dumps(
[
{
"thought": s.thought,
"tool_call": s.tool_call,
"tool_output": s.tool_output,
}
for s in loop_result.steps
],
ensure_ascii=False,
),
}
)
except Exception:
logger.exception("[{}] QA {} 执行异常", qa.video_id, qa.question_id)
# prediction 必落库(try 外,无论成败);绑定异常降级为最小 error 行,不击穿 gather
try:
await asyncio.to_thread(log.insert, "predictions", record)
except (sqlite3.InterfaceError, sqlite3.ProgrammingError):
logger.exception("[{}] QA {} 落库绑定异常,降级为 error 行", qa.video_id, qa.question_id)
record["prediction"] = None
record["stop_reason"] = "error"
await asyncio.to_thread(
log.insert,
"predictions",
{k: v for k, v in record.items() if isinstance(v, (str, int, float, type(None)))},
)
return record
# ---------------------------------------------------------------------------
# 建表
# ---------------------------------------------------------------------------
def _ensure_tables(log: HarnessLog) -> None:
"""创建推理所需的 5 张表。
参数:
log: HarnessLog 实例
"""
log.create_table("predictions", PREDICTIONS_SCHEMA)
log.create_table("traces", TRACES_SCHEMA)
log.create_table("validation_flags", VALIDATION_FLAGS_SCHEMA)
log.create_table("anchor_check", ANCHOR_CHECK_SCHEMA)
log.create_table("observe_frame_health", OF_HEALTH_SCHEMA)
# ---------------------------------------------------------------------------
# 公共入口
# ---------------------------------------------------------------------------
async def run_inference(
questions: list[GeneratedQuestion],
*,
llm: LLMProvider,
tool_dispatch_fn: Callable[..., Any],
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]],
log: HarnessLog,
run_id: str,
concurrency: int,
max_steps: int,
skill_mode: str,
plugins_factory: Callable[[str, str], list[object]] | None = None,
) -> InferenceResult:
"""在视频树上执行 Agent 推理,对应训练循环的 forward()。
参数:
questions: 待推理的题目列表
llm: LLMProvider 共享实例依赖注入
tool_dispatch_fn: async 工具调度函数 (tool_name, args, *, context) -> str
prompt_builder: prompt 构建函数 (GeneratedQuestion) -> (system_prompt, user_prompt)
log: HarnessLog 实例由调用方管理生命周期
run_id: 运行标识必传空串 ValueError
concurrency: 最大并发数asyncio.Semaphore 控制
max_steps: AgentLoop 单题最大步数
skill_mode: "auto" / "manual" / "none"传递给调用方的 prompt/plugin 构建逻辑
plugins_factory: 可选的插件工厂 (video_id, question_id) -> plugins 列表
返回:
InferenceResult accuracyper_task_type 等聚合指标
异常:
ValueError: run_id 为空串或纯空白
"""
if not run_id or not run_id.strip():
raise ValueError("run_id 不得为空串或纯空白")
_ensure_tables(log)
if not questions:
logger.info("题目列表为空,返回零值 InferenceResult")
return _aggregate_results([], [], run_id)
sem = asyncio.Semaphore(concurrency)
total_count = len(questions)
async def _bounded(index: int, qa: GeneratedQuestion) -> dict[str, Any]:
"""信号量限流的单题推理包装。"""
async with sem:
plugins = (
plugins_factory(qa.video_id, qa.question_id) if plugins_factory is not None else []
)
result = await _run_single_question(
qa,
llm=llm,
tool_dispatch_fn=tool_dispatch_fn,
prompt_builder=prompt_builder,
log=log,
max_steps=max_steps,
plugins=plugins,
run_id=run_id,
)
logger.info(
"[{}/{}] {} QA {} 完成 (stop={})",
index + 1,
total_count,
qa.video_id,
qa.question_id,
result["stop_reason"],
)
return result
results = await asyncio.gather(*[_bounded(i, qa) for i, qa in enumerate(questions)])
inference_result = _aggregate_results(list(results), questions, run_id)
logger.info(
"推理完成: accuracy={:.2%} ({}/{})",
inference_result.accuracy,
inference_result.correct,
inference_result.total,
)
return inference_result
+354
View File
@@ -0,0 +1,354 @@
"""HarnessLogSQLite 薄包装 + RunLogImpl 只读查询端口。
HarnessLog 提供统一的结构化日志接口 TRM4 直搬保留全部线程安全与幂等语义
run_id 重复创建时通过 ON CONFLICT DO UPDATE 更新 started_at/config/status
RunLogImpl 实现 core/evolution/protocols.py::RunLog Protocol用独立连接做只读 SELECT
不经 HarnessLog 生命周期不触发 _runs upsert避免污染运行状态
"""
from __future__ import annotations
import asyncio
import json
import sqlite3
import subprocess
import threading
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
def _get_git_sha() -> str | None:
"""获取当前 git commit SHA。"""
try:
result = subprocess.run(
["git", "rev-parse", "HEAD"],
capture_output=True,
text=True,
check=True,
)
return result.stdout.strip()
except (subprocess.CalledProcessError, FileNotFoundError):
return None
def _now_iso() -> str:
"""返回当前 UTC 时间的 ISO 格式字符串。"""
return datetime.now(UTC).isoformat()
class HarnessLog:
"""SQLite 薄包装,为科研项目提供统一的结构化日志接口。
关键设计
- WAL 模式 + threading.Lock 保证共享连接下并发安全
- ON CONFLICT DO UPDATE 保证幂等 run_id 多次创建不报错并更新 started_at/config/status
- query 也持锁共享连接check_same_thread=False下并发 SELECT + INSERT
在同一连接上 execute 会损坏游标状态故读也须串行化
- context manager 语义正常退出 completed异常退出 failed
参数:
db_path: SQLite 数据库文件路径
run_id: 本次运行的唯一标识
git_sha: 代码版本默认自动获取
config_snapshot: 本次运行的配置快照
register_run: 是否注册运行upsert _runs + 退出时同步 status默认 True
只读查询已有 run如基线预测回读时传 False避免把该 run
started_at/config/status 改写把基线元数据污染成本次进程的运行状态
"""
def __init__(
self,
db_path: str,
run_id: str,
git_sha: str | None = None,
config_snapshot: dict[str, Any] | None = None,
*,
register_run: bool = True,
) -> None:
self._run_id = run_id
self._register_run = register_run
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
# 单持久连接 + 进程内 Lock 串行化写:把并发控制拉到进程内,消除多连接争
# SQLite 写锁。同款模式复用于 adapters/telemetry.py:SQLiteTelemetryRecorder。
self._conn = sqlite3.connect(db_path, check_same_thread=False)
self._lock = threading.Lock()
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL")
self._init_fixed_tables()
if register_run:
resolved_sha = git_sha or _get_git_sha()
config_json = (
json.dumps(config_snapshot, ensure_ascii=False) if config_snapshot else None
)
self._conn.execute(
"INSERT INTO _runs"
" (run_id, git_sha, started_at, config, status)"
" VALUES (?, ?, ?, ?, ?)"
" ON CONFLICT(run_id) DO UPDATE SET"
" started_at=excluded.started_at,"
" config=excluded.config,"
" status=excluded.status",
(run_id, resolved_sha, _now_iso(), config_json, "running"),
)
self._conn.commit()
def _init_fixed_tables(self) -> None:
"""创建 _runs 和 _events 固定表。"""
self._conn.execute("""
CREATE TABLE IF NOT EXISTS _runs (
run_id TEXT PRIMARY KEY,
git_sha TEXT,
started_at TEXT,
finished_at TEXT,
config JSON,
status TEXT DEFAULT 'running',
skills_version TEXT,
prompts_version TEXT,
questions_ref TEXT
)
""")
self._conn.execute("""
CREATE TABLE IF NOT EXISTS _events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id TEXT,
timestamp TEXT,
event_type TEXT,
payload JSON
)
""")
self._conn.commit()
def create_table(
self,
name: str,
columns: dict[str, str],
primary_key: str | None = None,
) -> None:
"""创建自定义表,自动追加 run_id 和 timestamp 列。
参数:
name: 表名
columns: 列定义 {"epoch": "INTEGER", "loss": "REAL"}
primary_key: 主键列名
"""
all_columns = {"run_id": "TEXT", "timestamp": "TEXT"}
all_columns.update(columns)
col_defs = []
for col_name, col_type in all_columns.items():
pk_suffix = " PRIMARY KEY" if col_name == primary_key else ""
col_defs.append(f"{col_name} {col_type}{pk_suffix}")
sql = f"CREATE TABLE IF NOT EXISTS {name} ({', '.join(col_defs)})"
self._conn.execute(sql)
self._conn.commit()
def insert(self, table: str, record: dict[str, Any], mode: str = "append") -> None:
"""插入一条记录,自动填充 run_id 和 timestamp。
参数:
table: 目标表名
record: 要插入的数据
mode: "append" "upsert"
"""
enriched = {"run_id": self._run_id, "timestamp": _now_iso()}
enriched.update(record)
cols = list(enriched.keys())
placeholders = ", ".join(["?"] * len(cols))
col_names = ", ".join(cols)
values = [enriched[c] for c in cols]
if mode == "upsert":
sql = f"INSERT OR REPLACE INTO {table} ({col_names}) VALUES ({placeholders})"
else:
sql = f"INSERT INTO {table} ({col_names}) VALUES ({placeholders})"
with self._lock:
self._conn.execute(sql, values)
self._conn.commit()
def insert_many(self, table: str, records: list[dict[str, Any]], mode: str = "append") -> None:
"""批量插入多条记录。
参数:
table: 目标表名
records: 要插入的数据列表
mode: "append" "upsert"
"""
for record in records:
self.insert(table, record, mode=mode)
def execute(self, sql: str, params: tuple[Any, ...] = ()) -> None:
"""执行原生 SQL 写操作。
参数:
sql: SQL 语句
params: 参数元组
"""
with self._lock:
self._conn.execute(sql, params)
self._conn.commit()
def query(self, sql: str, params: tuple[Any, ...] = ()) -> list[dict[str, Any]]:
"""执行原生 SQL 查询,返回 list[dict]。
与所有写方法同持 self._lock共享连接check_same_thread=False
并发 SELECT INSERT 在同一连接上 execute 会损坏游标状态故读也须串行化
参数:
sql: SQL 查询语句
params: 参数元组
返回:
查询结果列表每行为一个字典
"""
with self._lock:
cursor = self._conn.execute(sql, params)
columns = [desc[0] for desc in cursor.description]
return [dict(zip(columns, row, strict=True)) for row in cursor.fetchall()]
def log_event(self, event_type: str, payload: dict[str, Any]) -> None:
"""向 _events 表写入一条事件。
参数:
event_type: 事件类型标识
payload: 事件数据
"""
with self._lock:
self._conn.execute(
"INSERT INTO _events (run_id, timestamp, event_type, payload) VALUES (?, ?, ?, ?)",
(
self._run_id,
_now_iso(),
event_type,
json.dumps(payload, ensure_ascii=False),
),
)
self._conn.commit()
def close(self, status: str = "completed") -> None:
"""更新运行状态并关闭连接。
参数:
status: 最终状态"completed" "failed"
关键实现:
register_run=False只读打开时跳过 status 更新仅关闭连接
避免只读回读把已有 run finished_at/status 改写
"""
with self._lock:
if self._register_run:
self._conn.execute(
"UPDATE _runs SET finished_at = ?, status = ? WHERE run_id = ?",
(_now_iso(), status, self._run_id),
)
self._conn.commit()
self._conn.close()
def __enter__(self) -> HarnessLog:
return self
def __exit__(
self,
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: Any,
) -> None:
status = "failed" if exc_type is not None else "completed"
self.close(status=status)
# ---------------------------------------------------------------------------
# RunLogImpl — core/evolution/protocols.py::RunLog 的只读实现
# ---------------------------------------------------------------------------
def _read_table(
db_path: str,
table: str,
run_id: str,
*,
question_ids: list[str] | None = None,
) -> list[dict[str, Any]]:
"""纯读某表指定 run 的行——不经 HarnessLog 生命周期,避免回读污染 _runs 运行状态。
HarnessLog.__enter__/__exit__ 会对 run_id INSERT OR IGNORE 并在退出时标 completed
回读指标绝不应改运行状态故走独立只读连接 SELECT
参数:
db_path: SQLite 路径
table: 表名内部固定常量非外部输入无注入风险
run_id: 过滤的 run ID
question_ids: 可选的 question_id 过滤列表
返回:
dict 列表表尚未建没写过视为无数据返 []
"""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
exists = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if exists is None:
return []
if question_ids is not None:
placeholders = ", ".join(["?"] * len(question_ids))
sql = f"SELECT * FROM {table} WHERE run_id = ? AND question_id IN ({placeholders})"
rows = conn.execute(sql, (run_id, *question_ids)).fetchall()
else:
rows = conn.execute(f"SELECT * FROM {table} WHERE run_id = ?", (run_id,)).fetchall()
return [dict(r) for r in rows]
finally:
conn.close()
class RunLogImpl:
"""RunLog Protocol 的只读实现。
用独立 sqlite3.connect SELECT不经 HarnessLog 生命周期不触发 _runs INSERT
asyncio.to_thread 包装同步 SQL 查询避免引入 aiosqlite 新依赖
参数:
db_path: SQLite 数据库文件路径
"""
def __init__(self, db_path: str) -> None:
self._db_path = db_path
async def get_predictions(
self,
run_id: str,
*,
question_ids: list[str] | None = None,
) -> list[dict[str, Any]]:
"""查询指定 run 的预测记录。
参数:
run_id: 运行标识
question_ids: 可选的题目 ID 过滤列表
返回:
预测记录字典列表
"""
return await asyncio.to_thread(
_read_table, self._db_path, "predictions", run_id, question_ids=question_ids
)
async def get_traces(
self,
run_id: str,
*,
question_ids: list[str] | None = None,
) -> list[dict[str, Any]]:
"""查询指定 run 的推理轨迹。
参数:
run_id: 运行标识
question_ids: 可选的题目 ID 过滤列表
返回:
轨迹记录字典列表
"""
return await asyncio.to_thread(
_read_table, self._db_path, "traces", run_id, question_ids=question_ids
)
+178
View File
@@ -0,0 +1,178 @@
"""慢更新动量生成 — epoch 末为单个 skill 产出新的动量指导。
对标 SkillOpt slow_update 机制拿上一 epoch 末与当前 epoch 末两版 skill
在固定样本上各跑一遍得到纵向对比comparison_pairs反思上一轮动量指导是否奏效
本轮正文改动是改善还是漂移据此重写动量指导新指导经 patch 引擎的 replace_momentum
写回 skill momentum 受保护区作为下一轮进化的方向锚
TRM4 core/harness/momentum.py156 迁移 + async
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from loguru import logger
from core.evolution.diagnose import extract_json_from_response
if TYPE_CHECKING:
from pathlib import Path
from core.protocols import LLMProvider
# =========================================================================
# 四类纵向对比类别名(单一真源)
# =========================================================================
IMPROVED = "improved" # 错→对
REGRESSED = "regressed" # 对→错
PERSISTENT_FAIL = "persistent_fail" # 错→错
STABLE_SUCCESS = "stable_success" # 对→对
# 类别名 → 展示标题,列表顺序即展示顺序。
# 回退(REGRESSED)刻意排在改善(IMPROVED)之前——它是最该警惕的伤害信号。
_CATEGORY_LABELS: tuple[tuple[str, str], ...] = (
(REGRESSED, "从对变错(回退,最高优先级)"),
(PERSISTENT_FAIL, "始终答错(持续失败)"),
(IMPROVED, "从错变对(改善)"),
(STABLE_SUCCESS, "始终答对(稳定成功)"),
)
# =========================================================================
# 辅助函数
# =========================================================================
def _categorize_pair(pair: dict[str, Any]) -> str:
"""按两版正误派生纵向对比类别。
用键值的真值bool(...)表示该题在两版上各自的正误 correct_prev/
correct_curr 键时直接抛 KeyError 向上传播这是上游数据损坏不是裁判语义
歧义静默当 False 会伪造 persistent_fail 证据污染动量指导故不掩盖
参数:
pair: 单个纵向对比对须含 correct_prev/correct_curr 两键
返回:
四个类别命名常量之一IMPROVED/REGRESSED/
PERSISTENT_FAIL/STABLE_SUCCESS
异常:
KeyError: correct_prev correct_curr 键时
"""
correct_prev = bool(pair["correct_prev"])
correct_curr = bool(pair["correct_curr"])
if not correct_prev and correct_curr:
return IMPROVED
if correct_prev and not correct_curr:
return REGRESSED
if not correct_prev and not correct_curr:
return PERSISTENT_FAIL
return STABLE_SUCCESS
def _format_comparison_pairs(comparison_pairs: list[dict[str, Any]]) -> str:
"""将纵向对比对格式化为裁判可读文本,按 _CATEGORY_LABELS 分组与排序。
参数:
comparison_pairs: 每个 dict question/prev_prediction/curr_prediction/
correct_prev/correct_curr 字段描述一道固定样本上两版的成对结果
返回:
可读的纵向对比文本空列表返回占位说明
异常:
KeyError: 任一 pair correct_prev/correct_curr 键时不掩盖的理由见
_categorize_pair docstring
"""
if not comparison_pairs:
return "(本轮无可用纵向对比样本)"
grouped: dict[str, list[dict[str, Any]]] = {key: [] for key, _ in _CATEGORY_LABELS}
for pair in comparison_pairs:
grouped[_categorize_pair(pair)].append(pair)
lines: list[str] = [f"固定样本总数:{len(comparison_pairs)}"]
for key, label in _CATEGORY_LABELS:
entries = grouped[key]
lines.append(f"\n### {label}{len(entries)} 题)")
if not entries:
lines.append("(无)")
continue
for pair in entries:
lines.append(
f"- 题目:{pair.get('question', '')}\n"
f" 上版预测:{pair.get('prev_prediction', '')} | "
f"当前版预测:{pair.get('curr_prediction', '')}"
)
return "\n".join(lines)
# =========================================================================
# 入口
# =========================================================================
async def run_slow_momentum(
llm: LLMProvider,
diagnose_prompts_dir: Path,
skill_content: str,
prev_skill: str,
prev_guidance: str,
comparison_pairs: list[dict[str, Any]],
) -> str:
"""为单个 skill 生成新的慢更新动量指导。
参数:
llm: LLM 端口async chat
diagnose_prompts_dir: 诊断 prompt 目录 prompts/slow_momentum.md 在此
skill_content: 当前版 skill 正文
prev_skill: 上一版 skill 正文
prev_guidance: 上一轮写下的动量指导
comparison_pairs: 固定样本上两版 rollout 的成对结果 question/
prev_prediction/curr_prediction/correct_prev/correct_curr
返回:
新的动量指导文本解析失败时保留 prev_guidance
关键实现细节:
- _format_comparison_pairs 刻意置于 try 块之外prompt 构造阶段它对每个
pair correct_prev/correct_curr缺键抛 KeyError 直接向上传播不被下方
针对裁判语义歧义的 except ValueError 吞掉
- 解析失败保留上轮指导extract_json_from_response ValueError
slow_update_content 字段或该字段非 str均视为语义解析失败返回
prev_guidance判不准时保守保留上轮指导对标 diagnose 的保护性 fallback
- P5 边界仅捕 ValueError 这一语义歧义llm.chat 的基础设施失败
网络/API 异常刻意不捕向上传播绝不用默认值掩盖
"""
system_prompt = (diagnose_prompts_dir / "slow_momentum.md").read_text(encoding="utf-8")
# _format_comparison_pairs 刻意置于下方 try 块之外(prompt 构造阶段):它对每个
# pair 取 correct_prev/correct_curr,缺键抛 KeyError 直接向上传播,不被下方针对
# 裁判语义歧义的 except ValueError 吞掉。异常类型选 KeyError(非 ValueError),
# 即便位置疏忽落入 try 也不会被误吞。
user_prompt = (
f"## 上一版 skill 正文\n{prev_skill}\n\n"
f"## 当前版 skill 正文\n{skill_content}\n\n"
f"## 上一轮的动量指导\n{prev_guidance}\n\n"
f"## 固定样本纵向对比(上版 vs 当前版)\n"
f"{_format_comparison_pairs(comparison_pairs)}"
)
response = await llm.chat(
[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
)
raw = response.content
try:
parsed = extract_json_from_response(raw)
new_guidance = parsed.get("slow_update_content")
if not isinstance(new_guidance, str):
raise ValueError("slow_update_content 字段缺失或非字符串")
except ValueError:
logger.warning("慢更新动量解析失败,保留上轮动量指导")
return prev_guidance
return new_guidance
+479
View File
@@ -0,0 +1,479 @@
"""五张观测表的落库写入与回读 + step/epoch 报告文件输出。
合并 TRM4 metric_log.py五表 loop_report.py报告
五张表均经 structured-logging 定义DDL 与之逐列一致
dual_metric_eval / shadow_gate / holdout_eval / quadrant_pair / gate_evidence
公共契约 P5soft/mixed Noneinvalid span / 诊断失败时存 NULL**绝不存 0**
SQLite dict None 值写入即 NULL分析时按 NULL 跳过每个写函数内幂等建表
``HarnessLog.create_table`` CREATE TABLE IF NOT EXISTSrun_id/timestamp 列由
HarnessLog 自动补
报告函数输出 JSON workspace analyses/ 目录供人工审查诊断 prompt 与进化 prompt
"""
from __future__ import annotations
import json
import sqlite3
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from pathlib import Path
def _read_table(db_path: str, table: str, run_id: str) -> list[dict[str, Any]]:
"""纯读某表指定 run 的全部行——不经 HarnessLog 生命周期,避免回读污染 _runs 运行状态。
HarnessLog.__enter__/__exit__ 会对 run_id INSERT OR IGNORE 并在退出时标 completed
回读指标绝不应改运行状态 read_* 一律走本只读连接 SELECT
参数:
db_path: SQLite 路径
table: 表名内部固定常量非外部输入无注入风险
run_id: 过滤的 run ID
返回:
dict 列表表尚未建没写过视为无数据返 []
"""
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
exists = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
).fetchone()
if exists is None:
return []
rows = conn.execute(f"SELECT * FROM {table} WHERE run_id=?", (run_id,)).fetchall()
return [dict(r) for r in rows]
finally:
conn.close()
# ---------------------------------------------------------------------------
# 列定义严格对齐 research-wiki/schemas/*.mdrun_id/timestamp 由 create_table 自动补)
# ---------------------------------------------------------------------------
_DUAL_COLS: dict[str, str] = {
"epoch": "INTEGER",
"version_kind": "TEXT",
"skills_version": "TEXT",
"prompts_version": "TEXT",
"pool": "TEXT",
"hard_acc": "REAL",
"soft_score": "REAL",
"mixed_score": "REAL",
}
_SHADOW_COLS: dict[str, str] = {
"epoch": "INTEGER",
"candidate_version": "TEXT",
"hard_acc": "REAL",
"soft_score": "REAL",
"mixed_score": "REAL",
"is_mixed_best": "INTEGER",
}
_HOLDOUT_COLS: dict[str, str] = {
"epoch": "INTEGER",
"version_kind": "TEXT",
"hard_acc": "REAL",
"soft_score": "REAL",
"mixed_score": "REAL",
"per_task_type_json": "TEXT",
}
_QUADRANT_COLS: dict[str, str] = {
"epoch": "INTEGER",
"step": "INTEGER",
# question_id 列承载 unit_idsingle=question_idpair=pair_id);
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
"question_id": "TEXT",
"task_type": "TEXT",
"prev_correct": "INTEGER",
"curr_correct": "INTEGER",
"category": "TEXT",
}
_GATE_EVIDENCE_COLS: dict[str, str] = {
"epoch": "INTEGER",
"step": "INTEGER",
"task_type": "TEXT",
# question_id 列承载 unit_idsingle=question_idpair=pair_id);
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
"question_id": "TEXT",
"ladder_rank": "INTEGER",
"baseline_correct": "INTEGER",
"candidate_correct": "INTEGER",
"e_value": "REAL",
"stop_reason": "TEXT",
}
# ---------------------------------------------------------------------------
# dual_metric_eval
# ---------------------------------------------------------------------------
def write_dual_metric(
db_path: str,
*,
run_id: str,
epoch: int,
version_kind: str,
skills_version: str,
prompts_version: str,
pool: str,
hard_acc: float,
soft_score: float | None,
mixed_score: float | None,
) -> None:
"""落 dual_metric_eval 一行:epoch 末关键版本的 hard+soft+mixed 双轨度量。
参数:
db_path: SQLite 路径
run_id: 训练 run ID
epoch: 轮次1-based
version_kind: baseline / best_hard / best_mixed / final / slow_candidate
slow_candidate = 慢更新 R2 可能被 revert 的候选不占 epoch 终值 final 口径
skills_version / prompts_version: 评估的资源版本
pool: val / test
hard_acc: hard 准确率
soft_score: soft 连续分invalid None -> NULL
mixed_score: 0.5*hard+0.5*softsoft 缺失传 None -> NULL
"""
from app.harness.log import HarnessLog
with HarnessLog(db_path, run_id) as log:
log.create_table("dual_metric_eval", _DUAL_COLS)
log.insert(
"dual_metric_eval",
{
"epoch": epoch,
"version_kind": version_kind,
"skills_version": skills_version,
"prompts_version": prompts_version,
"pool": pool,
"hard_acc": hard_acc,
"soft_score": soft_score,
"mixed_score": mixed_score,
},
)
def read_dual_metric(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
"""回读指定 run 的 dual_metric_eval 全部行(纯读,不污染运行状态)。"""
return _read_table(db_path, "dual_metric_eval", run_id)
# ---------------------------------------------------------------------------
# shadow_gate
# ---------------------------------------------------------------------------
def write_shadow_gate(
db_path: str,
*,
run_id: str,
epoch: int,
candidate_version: str,
hard_acc: float,
soft_score: float | None,
mixed_score: float | None,
is_mixed_best: bool,
) -> None:
"""落 shadow_gate 一行:mixed 影子 best 候选的 hard/soft/mixed 及是否 argmax 选中。
参数:
db_path: SQLite 路径
run_id: 训练 run ID
epoch: 轮次1-based
candidate_version: 候选版本标识 skills/vX+prompts/vY
hard_acc: hard 准确率
soft_score: soft 连续分invalid None -> NULL该版本不进 argmax
mixed_score: 0.5*hard+0.5*softsoft 缺失传 None -> NULL
is_mixed_best: 是否本 epoch mixed argmax 选中 1/0
"""
from app.harness.log import HarnessLog
with HarnessLog(db_path, run_id) as log:
log.create_table("shadow_gate", _SHADOW_COLS)
log.insert(
"shadow_gate",
{
"epoch": epoch,
"candidate_version": candidate_version,
"hard_acc": hard_acc,
"soft_score": soft_score,
"mixed_score": mixed_score,
"is_mixed_best": int(is_mixed_best),
},
)
def read_shadow_gate(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
"""回读指定 run 的 shadow_gate 全部行(纯读,不污染运行状态)。"""
return _read_table(db_path, "shadow_gate", run_id)
# ---------------------------------------------------------------------------
# holdout_eval
# ---------------------------------------------------------------------------
def write_holdout_eval(
db_path: str,
*,
run_id: str,
epoch: int,
version_kind: str,
hard_acc: float,
soft_score: float | None,
mixed_score: float | None,
per_task_type_json: str,
) -> None:
"""落 holdout_eval 一行:四向 held-out 在 test 池的 hard+soft+mixed 及按题型细分。
参数:
db_path: SQLite 路径
run_id: 训练 run ID
epoch: 轮次1-based
version_kind: baseline / best_hard / best_mixed / final
hard_acc: hard 准确率
soft_score: soft 连续分invalid None -> NULL
mixed_score: 0.5*hard+0.5*softsoft 缺失传 None -> NULL
per_task_type_json: task_type {accuracy,total,correct} JSON
"""
from app.harness.log import HarnessLog
with HarnessLog(db_path, run_id) as log:
log.create_table("holdout_eval", _HOLDOUT_COLS)
log.insert(
"holdout_eval",
{
"epoch": epoch,
"version_kind": version_kind,
"hard_acc": hard_acc,
"soft_score": soft_score,
"mixed_score": mixed_score,
"per_task_type_json": per_task_type_json,
},
)
def read_holdout_eval(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
"""回读指定 run 的 holdout_eval 全部行(纯读,不污染运行状态)。"""
return _read_table(db_path, "holdout_eval", run_id)
# ---------------------------------------------------------------------------
# quadrant_pair
# ---------------------------------------------------------------------------
def write_quadrant_pairs(
db_path: str,
*,
run_id: str,
epoch: int,
step: int,
pairs: list[dict[str, Any]],
) -> None:
"""落 quadrant_pair 多行:fast gate 后按 **unit** 四象限(prev/curr 翻转 + category)落库。
参数:
db_path: SQLite 路径
run_id: 训练 run ID
epoch: 轮次1-based
step: epoch step 序号0-based
pairs: 每条含 question_id/task_type/prev_correct/curr_correct/category
question_id 字段承载 **unit_id**single=question_idpair=pair_id
gate e-process 同粒度逐题明细在 predictions 表溯源 pair_id
join 真实 question 表会 join 不上prev_correct/curr_correct bool
写库前转 0/1
关键实现:
insert_many 批量落库pairs 为空时只建表不插入fast gate 无翻转的极端情况
"""
records = [
{
"epoch": epoch,
"step": step,
"question_id": pair["question_id"],
"task_type": pair["task_type"],
"prev_correct": int(pair["prev_correct"]),
"curr_correct": int(pair["curr_correct"]),
"category": pair["category"],
}
for pair in pairs
]
from app.harness.log import HarnessLog
with HarnessLog(db_path, run_id) as log:
log.create_table("quadrant_pair", _QUADRANT_COLS)
if records:
log.insert_many("quadrant_pair", records)
def read_quadrant_pairs(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
"""回读指定 run 的 quadrant_pair 全部行(纯读,不污染运行状态)。"""
return _read_table(db_path, "quadrant_pair", run_id)
# ---------------------------------------------------------------------------
# gate_evidence
# ---------------------------------------------------------------------------
def write_gate_evidence(
db_path: str,
*,
run_id: str,
epoch: int,
step: int,
rows: list[dict[str, Any]],
) -> None:
"""落 gate_evidence 单元行:CE-Gate 每次决策的可回放审计记录(unit 口径)。
参数:
db_path: SQLite 路径
run_id: 训练 run ID
epoch: gate 所属的轮次1-based
step: epoch step 序号0-based
rows: **单元** 一行 question_id/task_type/ladder_rank阶梯序号
0-based/baseline_correct/
candidate_correct/e_value该单元判定后的累计 e /
stop_reason仅最后一单元携带最终 stop_reason其余空串
question_id 字段承载 **unit_id**single=question_idpair=pair_id
逐题明细在 predictions 表溯源 pair_id join 真实 question 表会 join 不上
返回:
关键实现:
逐行 insert insert_many保证每行独立事务
"""
from app.harness.log import HarnessLog
with HarnessLog(db_path, run_id) as log:
log.create_table("gate_evidence", _GATE_EVIDENCE_COLS)
# 幂等迁移(对齐 question_gen/run_store 先例):块序贯时代的旧表只有
# block_idx 列,CREATE TABLE IF NOT EXISTS 不补列,直接插 ladder_rank
# 会 OperationalError——为旧 workspace 复用补列,新表恒为 no-op。
cols = {r["name"] for r in log.query("PRAGMA table_info(gate_evidence)")}
if "ladder_rank" not in cols:
log.execute("ALTER TABLE gate_evidence ADD COLUMN ladder_rank INTEGER")
for row in rows:
log.insert("gate_evidence", {"epoch": epoch, "step": step, **row})
def read_gate_evidence(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
"""回读指定 run 的 gate_evidence 全部行(纯读,不污染运行状态)。"""
return _read_table(db_path, "gate_evidence", run_id)
# ---------------------------------------------------------------------------
# 报告函数(从 TRM4 loop_report.py 迁移)
# ---------------------------------------------------------------------------
def write_step_report(
workspace_dir: Path,
epoch: int,
step: int,
global_step: int,
task_type: str,
gate_action: str,
candidate_acc: float,
class_baseline_acc: float,
edit_budget: int,
rank_clip_triggered: bool,
gate_w: int | None,
gate_l: int | None,
gate_e_value: float | None,
gate_n_used: int | None,
gate_stop_reason: str | None,
) -> Path:
"""写单个 (step, task_type) 快路径 gate 的最小观测记录 JSON。
文件名按 (epoch, step, task_type) 命名slug task_type 规范化小写空格转 '-'得到
参数:
workspace_dir: 实验工作区目录
epoch: 当前轮次1-based
step: epoch step 序号0-based
global_step: 全局步计数驱动 edit_budget 退火
task_type: 本条 gate 的任务类型
gate_action: 闸门动作accept_confirmed / accept_provisional / reject /
skipped / cooldown
candidate_acc: 候选在 gate 已观测题上的准确率观测口径
class_baseline_acc: 基线在 gate 已观测题上的准确率观测口径
edit_budget: step global_step 退火得到的 per-target 编辑预算上限
rank_clip_triggered: skill 进化是否触发了 rank 裁剪
gate_w: e-process 累计 W基线错->候选对翻转数skipped/cooldown 路径传 None
gate_l: e-process 累计 L基线对->候选错翻转数skipped/cooldown 路径传 None
gate_e_value: 停时的 e skipped/cooldown 路径传 None
gate_n_used: gate 实际消费的阶梯题数skipped/cooldown 路径传 None
gate_stop_reason: e-process 停止原因skipped/cooldown 路径传 None
返回:
写入的 step_report 文件路径
"""
report = {
"epoch": epoch,
"step": step,
"global_step": global_step,
"task_type": task_type,
"gate_action": gate_action,
"candidate_acc": candidate_acc,
"class_baseline_acc": class_baseline_acc,
"edit_budget": edit_budget,
"rank_clip_triggered": rank_clip_triggered,
"gate_w": gate_w,
"gate_l": gate_l,
"gate_e_value": gate_e_value,
"gate_n_used": gate_n_used,
"gate_stop_reason": gate_stop_reason,
}
slug = task_type.lower().replace(" ", "-")
out_dir = workspace_dir / "analyses"
out_dir.mkdir(parents=True, exist_ok=True)
path = out_dir / f"step_report_e{epoch}_s{step}_{slug}.json"
path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
return path
def write_epoch_report(
workspace_dir: Path,
epoch: int,
system_tool_action: str,
momentum_updated_task_types: list[str],
best_val_acc: float,
) -> Path:
"""写 epoch 末慢更新汇总 JSON。
慢更新无单一 ValidationOutcome故本函数只落慢更新可观测的最小集
system/tool gate 动作 epoch 写过 momentum 的题型慢更新后的全局 best
参数:
workspace_dir: 实验工作区目录
epoch: 当前轮次1-based
system_tool_action: 慢更新 system/tool 动作updated / reverted / none
momentum_updated_task_types: epoch 写过 momentum 的题型列表
best_val_acc: 慢更新后 best argmax的全局 best 验证准确率
返回:
写入的 epoch_report 文件路径
"""
report = {
"epoch": epoch,
"system_tool_action": system_tool_action,
"momentum_updated_task_types": momentum_updated_task_types,
"best_val_acc": best_val_acc,
}
out_dir = workspace_dir / "analyses"
out_dir.mkdir(parents=True, exist_ok=True)
path = out_dir / f"epoch_report_{epoch}.json"
path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
return path
+1206
View File
File diff suppressed because it is too large Load Diff
+146
View File
@@ -0,0 +1,146 @@
"""QuestionUnit 组装/展开/校验/单元正确性——pair 契约的唯一入口。
pool 构建批处理推理评测Task 3+均通过本模块聚合/展开孪生对
保证 AR pair "两题作为整体调度"契约只在一处实现fail-fast 暴露非法配对
"""
from collections import defaultdict
from core.types import GeneratedQuestion, QuestionUnit
def _assemble_pair(pair_id: str, group: list[GeneratedQuestion]) -> QuestionUnit:
"""校验单个 pair 分组的数量/角色并组装为 pair 单元(fail-fast)。
参数:
pair_id: 该分组共享的孪生对标识
group: 归属同一 pair_id 的题目列表
返回:
kind="pair" QuestionUnit
关键实现:
- 分组必须恰好 2 否则视为孤儿/超员raise ValueError
- 显式检查 pair_original / pair_mirror 角色齐备且唯一缺失或重复
直接 raise ValueError next(...) 静默 StopIteration
- 合法孪生对交由 QuestionUnit.from_pair video_id/task_type/flip_axis
一致性断言
"""
if len(group) != 2:
raise ValueError(f"pair {pair_id} 数量={len(group)}≠2(孤儿或超员)")
originals = [q for q in group if q.question_role == "pair_original"]
mirrors = [q for q in group if q.question_role == "pair_mirror"]
if len(originals) != 1 or len(mirrors) != 1:
raise ValueError(
f"pair {pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)}"
"需各恰好 1 条"
)
return QuestionUnit.from_pair(originals[0], mirrors[0])
def build_units(questions: list[GeneratedQuestion]) -> list[QuestionUnit]:
"""将扁平题目列表聚合为单元列表:single 单封、pair 按 pair_id 成对聚合。
参数:
questions: 待聚合的题目列表可混含 single 与孪生对成员
返回:
单元列表 single pair顺序稳定single 保留输入顺序
pair 按首次出现的 pair_id 顺序
关键实现:
pair_id 为空 single 单元非空 归入对应 pair pair 桶的
数量/角色校验与组装下沉到 _assemble_pairfail-fast本体只做分组
与派发
"""
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
singles: list[QuestionUnit] = []
for q in questions:
if q.pair_id:
by_pair[q.pair_id].append(q)
else:
singles.append(QuestionUnit.from_single(q))
pairs = [_assemble_pair(pid, qs) for pid, qs in by_pair.items()]
return singles + pairs
def flatten_units(units: list[QuestionUnit]) -> list[GeneratedQuestion]:
"""将单元列表无损展开回扁平题目列表。
参数:
units: 单元列表
返回:
展开后的题目列表保持单元顺序及单元内题目顺序
"""
return [q for u in units for q in u.questions]
def validate_units(units: list[QuestionUnit]) -> list[QuestionUnit]:
"""校验单元列表结构合法性,通过则原样返回(便于链式调用)。
参数:
units: 待校验单元列表
返回:
校验通过的原单元列表
关键实现:
pair 单元必须恰好含 2 否则 raise ValueErrorsingle 单元无需额外
校验构造时即为 1 用于消费方在使用前做一道防御闸门
"""
for u in units:
if u.kind == "pair" and u.size != 2:
raise ValueError(f"unit {u.unit_id} pair 不成对(size={u.size}")
return units
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool], *, strict: bool = True) -> bool:
"""计算单元级正确性:AR pair 走双向 AND,single 即单题正确性。
参数:
unit: 目标单元
per_q: 题目 question_id 该题是否作答正确的映射
strict: 缺键策略True默认时以 per_q[q.question_id] 取值缺任一题
触发 KeyError防静默兜底强制上游先补齐全部单题结果False 时以
per_q.get(q.question_id, False) 取值缺键计 False宽松口径供池
构建 / gate 冷启动 / 采样等"缺基线对错即视为未答对"的调用点复用
返回:
单元内所有题目均正确时为 True否则 False
关键实现:
pool 构建poolsgate 冷启动gate_ladder分层采样loader三处
原各自持有的 loose _unit_correct 副本统一收敛到本函数 strict=False 分支
消除重复逻辑与 missing-key 策略分叉
"""
if strict:
return all(per_q[q.question_id] for q in unit.questions)
return all(per_q.get(q.question_id, False) for q in unit.questions)
def unit_correctness_view(
units: list[QuestionUnit], per_q: dict[str, bool], *, strict: bool = True
) -> dict[str, bool]:
"""把逐题对错折叠成单元级视图:unit_id → 单元是否整体正确。
进化引擎gate e-process / quadrant / probation / pair_block / compute_accuracy
统一消费此单元视图保证 AR pair 双向 AND AR single 单题混格池中
孪生对折叠为一个单元不被 P/Q 单题计分污染核心算法保真 #5)。
参数:
units: 目标单元列表single pair
per_q: 题目 question_id 该题是否作答正确唯一逐题溯源来源
strict: 缺键策略透传给 unit_correctnessTrue默认缺任一题 raise
KeyErrorFalse 缺键计 False宽松口径
返回:
unit_id 单元级正确性single unit_id 等于其 question_id
pair unit_id 等于共享 pair_id
关键实现:
逐单元复用 unit_correctnessstrict 透传默认 strict 禁静默兜底
强制上游先补齐全部单题结果
"""
return {u.unit_id: unit_correctness(u, per_q, strict=strict) for u in units}
File diff suppressed because it is too large Load Diff
+59
View File
@@ -0,0 +1,59 @@
"""结果驱动视频级切分的冻结溯源 manifest。
冻结的 pools.json 是切分产物manifest 记录产出这份切分的关键输入
baseline_run_id诊断指纹随机种子配置 pools.json 的内容指纹
pools_sha256供后续 build_split 写溯源以及复现校验时比对
"""
from __future__ import annotations
import hashlib
from typing import TYPE_CHECKING
from app.harness.pools import _atomic_write_json
if TYPE_CHECKING:
from pathlib import Path
def write_manifest(
path: Path,
*,
baseline_run_id: str,
diag_fingerprint: str,
seed: int,
config: dict,
pools_json_text: str,
coverage_report: dict,
generated_at: str,
) -> dict:
"""写切分冻结溯源 manifest(原子写),返回写入的 dict。
pools_sha256 = sha256(pools_json_text)供复现时校验冻结的 pools.json 内容
是否与本次切分一致generated_at 由调用方传入库内不用 datetime.now
以保证相同输入产出相同 manifest可复现
参数:
path: manifest 目标 JSON 文件路径
baseline_run_id: 产出本次切分所依据的基线 run 标识
diag_fingerprint: 诊断结果指纹决定 train/val 归属的输入
seed: 切分使用的随机种子
config: 切分相关配置快照 train_ratio
pools_json_text: 冻结的 pools.json 完整文本用于计算内容指纹
coverage_report: 各类别 train/val 覆盖统计报告
generated_at: 生成时间戳ISO 字符串由调用方传入
返回:
写入 manifest dict与落盘内容一致
"""
manifest = {
"baseline_run_id": baseline_run_id,
"diag_fingerprint": diag_fingerprint,
"seed": seed,
"config": config,
"pools_sha256": hashlib.sha256(pools_json_text.encode("utf-8")).hexdigest(),
"coverage_report": coverage_report,
"generated_at": generated_at,
}
_atomic_write_json(path, manifest)
return manifest
+551
View File
@@ -0,0 +1,551 @@
"""视频级切分选择:signal 分层、视频聚合、贪心联合约束选择(纯函数)。
结果驱动切分管线的核心把诊断信号投影为多样性格子供贪心选择器最大化覆盖
本模块起步定义 evolution_target 派生与多样性格子后续追加 score_signal /
build_video_records / select_split
"""
from __future__ import annotations
import hashlib
import random
from dataclasses import dataclass, field
from loguru import logger
def diag_fingerprint(prompt_version: str, model: str, code_version: str) -> str:
"""由 (诊断 prompt 版本, 模型名, 代码 git 短 SHA) 合成诊断口径指纹。
诊断信号以 (question_id, baseline_run_id, diag_fingerprint) 为主键持久化
指纹隔离不同诊断配置的信号 prompt 版本 / 换模型 / 换代码实现都会得到
新指纹从而 `--force` 用新指纹重跑诊断时**不覆盖旧记录**旧指纹行仍在
保证不同口径的诊断结果可并存可回溯可比对
参数:
prompt_version: 诊断 prompt 的版本标识 prompts/diagnose_*.md 的版本
model: 执行诊断的模型名 "deepseek-v4"
code_version: 诊断代码的版本约定为 git SHA
返回:
16 位十六进制指纹sha256 截断对三分量任一变化敏感对相同三元组确定
实现细节:
三分量用 "|" 分隔后 sha256取前 16 分隔符防止 ("ab","c") ("a","bc")
碰撞成同一指纹纯函数相同输入永远同输出可安全用于主键
"""
return hashlib.sha256("|".join([prompt_version, model, code_version]).encode()).hexdigest()[:16]
_EVOLUTION_TARGET = {
"extraction_failure": "tool",
"search_failure": "skill",
"reasoning_failure": "skill",
"mixed": "system",
}
def evolution_target_of(error_type: str) -> str:
"""由 error_type 确定性派生进化目标(tool/skill/system)。
这是报告用的派生标注非独立多样性轴多样性主格子=task_type×error_type
参数:
error_type: 诊断瀑布归因的错误类别extraction/search/reasoning/mixed_failure
返回:
进化目标字符串 tool / skill / system
异常:
ValueError: error_type 不在已知集合内不静默兜底
"""
if error_type not in _EVOLUTION_TARGET:
raise ValueError(f"未知 error_type: {error_type}")
return _EVOLUTION_TARGET[error_type]
def cell_of(task_type: str, error_type: str) -> tuple[str, str]:
"""构造多样性主格子 = (task_type, error_type)。
参数:
task_type: 题型12 类之一
error_type: 错误类别4 类之一
返回:
(task_type, error_type) 二元组作为覆盖计数的格子键
"""
return (task_type, error_type)
@dataclass(frozen=True)
class SignalLabel:
"""诊断信号分层标签(DiagnosisResult 的确定性投影)。
字段:
tier: 信号层级取值 T0 / T1 / T2 / uncertain判据见 score_signal
"""
tier: str
def score_signal(*, cause_category: str | None, infra: bool, degraded: bool) -> SignalLabel:
"""把诊断产物投影为信号分层 tier(不发明新分类,是确定性投影)。
分层优先级顺序固定用早返回表达不用魔法权重
先判 INFRA再判 degraded然后 defect / lapse最后兜底 uncertain
各层判据来源:
T0 infra=True即诊断 INFRA 排除stop_reason {error, parse_error}
基础设施失败先于一切判定排除出可训练主体
uncertain degraded=Truejudge 解析失败 cause_category 落不到
defect/lapse 如为 None信号不可信排除出 T2
T2 cause_category == "defect"可训练核心进多样性覆盖与训练主体
T1 cause_category == "lapse"低信号含无解题接受但不作训练主体
参数:
cause_category: 诊断的缺陷归因"defect" / "lapse" / None
infra: 是否被 INFRA 护栏排除基础设施失败
degraded: judge 是否解析失败导致诊断降级
返回:
SignalLabel tier 字段为上述四层之一
实现细节:
关键字参数强制传入防止 infra / degraded 两个 bool 位置混淆
"""
if infra:
return SignalLabel(tier="T0")
if degraded:
return SignalLabel(tier="uncertain")
if cause_category == "defect":
return SignalLabel(tier="T2")
if cause_category == "lapse":
return SignalLabel(tier="T1")
return SignalLabel(tier="uncertain")
@dataclass(frozen=True)
class VideoRecord:
"""全视频画像单元(贪心选择器 Task 8 的输入单元)。
覆盖全部视频含全对零诊断信号的视频既承载 test 代表性所需的难度/题型画像
也叠加 T2 可训练缺陷的多样性格子供选择器算覆盖与补集
字段:
video_id: 视频唯一标识
type_set: 该视频所有题的 task_type 集合去重画像用
n_correct: 该视频答对题数
difficulty: 难度画像桶 = 错题数 = 题数 - n_correct
cells: tier=="T2" 信号行投影的 (task_type, error_type) 主格子并集去重
wrong_by_type: task_type T2 计数供选择器 floor 约束普通 dict
实现细节:
frozen 生成的 __hash__ 会遍历各字段wrong_by_type 为不可哈希 dict
故显式标注 hash=False 将其排除出哈希避免 VideoRecord set/dict 键时报错
仍保留其参与相等性比较
"""
video_id: str
type_set: frozenset[str]
n_correct: int
difficulty: int
cells: frozenset[tuple[str, str]]
wrong_by_type: dict[str, int] = field(hash=False)
def build_video_records(preds: list[dict], signal_rows: list[dict]) -> list[VideoRecord]:
"""由全量 predictions 与诊断信号行构建全视频 VideoRecord 列表。
先按 video_id 聚合全部 predictions覆盖全对零信号视频再叠加仅 tier=="T2"
的诊断信号为多样性格子与 wrong_by_type 计数 T2 信号行T0/T1/uncertain
不计入格子与计数
参数:
preds: 全量预测行每行含 video_id / question_id / task_type / correct
每视频含其全部题不限于错题correct 为布尔答对标记
signal_rows: 诊断信号行每行含 question_id / task_type / error_type / tier
诊断只覆盖错题子集正确题无对应信号行属正常不视为错误
返回:
全部视频的 VideoRecord 列表按视频在 preds 中首次出现顺序排列
无任何 T2 信号的视频其 cells 为空 frozensetwrong_by_type 为空 dict
实现细节:
signal_rows question_id 若不在 preds 中则忽略诊断可能滞后于当前预测集
非数据损坏 fail-fast缺失必需键则按 KeyError 直接暴露不静默兜底
异常:
KeyError: preds signal_rows 行缺少必需键校验前置防脏数据静默通过
"""
# Phase 1: 按 video_id 聚合 preds(保持首次出现顺序)。
signal_by_qid = {row["question_id"]: row for row in signal_rows}
aggregates: dict[str, dict] = {}
for pred in preds:
video_id = pred["video_id"]
bucket = aggregates.setdefault(
video_id, {"types": set(), "question_ids": [], "n_correct": 0}
)
bucket["types"].add(pred["task_type"])
bucket["question_ids"].append(pred["question_id"])
if pred["correct"]:
bucket["n_correct"] += 1
# Phase 2: 逐视频叠加 T2 信号为格子与 wrong_by_type。
records: list[VideoRecord] = []
for video_id, bucket in aggregates.items():
cells: set[tuple[str, str]] = set()
wrong_by_type: dict[str, int] = {}
for question_id in bucket["question_ids"]:
row = signal_by_qid.get(question_id)
if row is None or row["tier"] != "T2":
continue
task_type = row["task_type"]
cells.add(cell_of(task_type, row["error_type"]))
wrong_by_type[task_type] = wrong_by_type.get(task_type, 0) + 1
n_questions = len(bucket["question_ids"])
records.append(
VideoRecord(
video_id=video_id,
type_set=frozenset(bucket["types"]),
n_correct=bucket["n_correct"],
difficulty=n_questions - bucket["n_correct"],
cells=frozenset(cells),
wrong_by_type=wrong_by_type,
)
)
return records
@dataclass(frozen=True)
class SelectConfig:
"""贪心联合约束选择器的实验配置(科研配置,随实验扫动)。
字段:
n_trainval: trainval 目标视频数多样性阶段的填充上限
floor_k: 各高信号 task_type T2 defect 数下限硬约束floor 阶段满足
epsilon: test 相对全局的最大允许分布偏差题型占比 / 难度画像两维逐桶
reportable_types: 参与 ε 题型代表性校验的 task_type 长尾类型不入约束
seed: 预洗牌随机种子仅用于打破等增益平局保证同 config videos 同解
实现细节:
floor_k / reportable_types 为不可哈希容器 hash=False 排除出自动 __hash__
避免 frozen dataclass 被哈希时报错本类不作为字典键仅承载配置
"""
n_trainval: int
floor_k: dict[str, int] = field(hash=False)
epsilon: float
reportable_types: frozenset[str] | set[str] = field(hash=False)
seed: int
@dataclass(frozen=True)
class SplitAssignment:
"""视频级切分归属结果(交给 split_by_video_assignment 做题级切分)。
字段:
trainval: 进入 trainval video_id 元组按选择顺序确定性
test: 补集视频的 video_id 元组 videos 原始顺序
"""
trainval: tuple[str, ...]
test: tuple[str, ...]
class InfeasibleSplitError(Exception):
"""floor 硬约束与 ε 守护死锁、无法在不破坏 test 代表性下满足 floor 时抛出。
fail loudP5不静默兜底不随机塞题直接暴露不可行并报告未达标类型
"""
def derive_reportable_types(total_by_type: dict[str, int], report_floor: int) -> set[str]:
"""派生可 per-type 报告的 task_type 集(长尾处理:题数 ≥ report_floor 才报告)。
参数:
total_by_type: task_type 的总题数或代理承载数
report_floor: 报告门限低于此的类型并入长尾不单独报告也不入 ε 约束
返回:
总题数 report_floor task_type 集合
"""
return {task_type for task_type, total in total_by_type.items() if total >= report_floor}
def _type_membership_fraction(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
"""计算各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
参数:
records: 视频记录子集非空调用方保证
keys: 需计算占比的 task_type 键集
返回:
{task_type: 占比}占比 [0, 1]
"""
total = len(records)
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
def _difficulty_fraction(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
"""计算各难度桶在给定视频集中的占比(难度 = 错题数)。
参数:
records: 视频记录子集非空调用方保证
buckets: 需计算占比的难度桶键集
返回:
{难度桶: 占比}占比 [0, 1]
"""
total = len(records)
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
def _max_deviation(global_dist: dict, subset_dist: dict, keys: set) -> float:
"""逐键取全局与子集分布的最大绝对偏差(键集为空时约定为 0.0)。
参数:
global_dist: 全局分布 占比
subset_dist: 子集分布 占比
keys: 参与比较的键集
返回:
逐键 |global - subset| 的最大值keys 为空返回 0.0
"""
if not keys:
return 0.0
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
def _epsilon_ok(
test_video_records: list[VideoRecord],
videos_all: list[VideoRecord],
config: SelectConfig,
) -> bool:
"""校验 test 子集相对全局在题型占比与难度画像两维的偏差是否均 ≤ epsilon。
test 越简单则 headline 越虚高 test 必须保持代表性 reportable 题型逐难度桶
比较 test 与全局占比任一维超 epsilon 即判不合格
参数:
test_video_records: 候选 test 子集trainval 补集
videos_all: 全部视频全局分布基准
config: 选择配置提供 epsilon reportable_types
返回:
两维最大偏差均 epsilon Truetest 为空视为不合格返回 False
"""
if not test_video_records:
return False
type_keys = set(config.reportable_types)
global_type = _type_membership_fraction(videos_all, type_keys)
subset_type = _type_membership_fraction(test_video_records, type_keys)
if _max_deviation(global_type, subset_type, type_keys) > config.epsilon:
return False
diff_keys = {r.difficulty for r in videos_all}
global_diff = _difficulty_fraction(videos_all, diff_keys)
subset_diff = _difficulty_fraction(test_video_records, diff_keys)
return _max_deviation(global_diff, subset_diff, diff_keys) <= config.epsilon
def _current_wrong_counts(selected: list[VideoRecord]) -> dict[str, int]:
"""聚合已选 trainval 视频的 T2 defect 计数(供 floor 达标判定)。
参数:
selected: 当前已进入 trainval 的视频记录
返回:
{task_type: T2 defect 累计数}
"""
counts: dict[str, int] = {}
for video in selected:
for task_type, wrong in video.wrong_by_type.items():
counts[task_type] = counts.get(task_type, 0) + wrong
return counts
def _unmet_floors(selected: list[VideoRecord], floor_k: dict[str, int]) -> dict[str, int]:
"""计算尚未达标的 floor 类型及其缺口(已达标类型不返回)。
参数:
selected: 当前已进入 trainval 的视频记录
floor_k: 各高信号 task_type defect 下限
返回:
{task_type: 缺口数}仅含 current < floor 的类型全达标返回空 dict
"""
counts = _current_wrong_counts(selected)
return {
task_type: floor - counts.get(task_type, 0)
for task_type, floor in floor_k.items()
if counts.get(task_type, 0) < floor
}
def _floor_fill_count(video: VideoRecord, deficits: dict[str, int]) -> int:
"""计算某视频能填补的 floor 缺口槽数(逐类型取 min(defect, 缺口) 求和)。
参数:
video: 候选视频记录
deficits: 各未达标类型的缺口
返回:
该视频实际可填的槽数总和0 表示对当前缺口无贡献
"""
return sum(
min(video.wrong_by_type.get(task_type, 0), deficit)
for task_type, deficit in deficits.items()
)
def _marginal_gain(video: VideoRecord, current_cells: set[tuple[str, str]]) -> int:
"""计算把某视频移入 trainval 的边际覆盖增益(新开的 T2 格子数)。
参数:
video: 候选视频记录
current_cells: 当前 trainval T2 格子并集
返回:
video.cells 相对 current_cells 的新增格子数去重
"""
return len(video.cells - current_cells)
def _prospective_test(pool: list[VideoRecord], candidate: VideoRecord) -> list[VideoRecord]:
"""构造"把候选移入 trainval 后"的 test 子集 = 当前剩余池去掉候选。
参数:
pool: 当前尚未进入 trainval 的视频即当前 test 补集
candidate: 拟移入 trainval 的候选视频
返回:
pool 去掉 candidate 后的视频列表
"""
return [r for r in pool if r.video_id != candidate.video_id]
def _satisfy_floors(
selected: list[VideoRecord],
pool: list[VideoRecord],
videos_all: list[VideoRecord],
config: SelectConfig,
) -> None:
"""Floor 阶段:硬约束优先,逐步移入能填 floor 槽且不破 ε 的视频(就地改 selected/pool)。
每轮取未达标类型的缺口候选 = 能填 1 移入后 test 仍满足 ε 的视频候选为空即
死锁抛 InfeasibleSplitError否则选填槽最多者等槽数按预洗牌顺序取首个确定性
n_trainval 是硬预算floor 需求超出预算尚有缺口却已达 n_trainval也判不可行 fail loud
保证返回的 trainval 永不超过 n_trainval不因硬约束悄悄超额挤占 test
参数:
selected: 当前 trainval就地追加
pool: 当前剩余池 = test 补集就地移除
videos_all: 全部视频ε 全局基准
config: 选择配置
异常:
InfeasibleSplitError: 存在未达标类型但无候选可在不破 ε 下填补
floor 需求超过 n_trainval 预算
"""
while True:
deficits = _unmet_floors(selected, config.floor_k)
if not deficits:
return
if len(selected) >= config.n_trainval:
raise InfeasibleSplitError(
f"floor 需求超过 n_trainval={config.n_trainval} 预算,剩余缺口: {dict(deficits)}"
)
candidates = [
video
for video in pool
if _floor_fill_count(video, deficits) > 0
and _epsilon_ok(_prospective_test(pool, video), videos_all, config)
]
if not candidates:
raise InfeasibleSplitError(
f"floor 无法在 ε≤{config.epsilon} 下满足,未达标类型缺口: {dict(deficits)}"
)
pick = max(candidates, key=lambda video: _floor_fill_count(video, deficits))
selected.append(pick)
pool.remove(pick)
def _maximize_diversity(
selected: list[VideoRecord],
pool: list[VideoRecord],
videos_all: list[VideoRecord],
config: SelectConfig,
) -> None:
"""多样性阶段:submodular 贪心,按边际覆盖增益降序填至 n_trainval(就地改 selected/pool)。
每轮对剩余视频算新开格子数 -增益稳定排序等增益按预洗牌顺序取第一个移入后 test
仍满足 ε 的视频若无任一视频可加而不破 ε则停并记 warning欠额不静默不报错
参数:
selected: 当前 trainval就地追加
pool: 当前剩余池 = test 补集就地移除
videos_all: 全部视频ε 全局基准
config: 选择配置
"""
while len(selected) < config.n_trainval:
if not pool:
logger.warning(
"多样性阶段剩余池耗尽,trainval 欠额: {}/{}", len(selected), config.n_trainval
)
return
current_cells = set().union(*(v.cells for v in selected)) if selected else set()
ranked = sorted(pool, key=lambda video: -_marginal_gain(video, current_cells))
pick = next(
(
video
for video in ranked
if _epsilon_ok(_prospective_test(pool, video), videos_all, config)
),
None,
)
if pick is None:
logger.warning(
"多样性阶段 ε 守护阻断全部候选,trainval 欠额: {}/{}",
len(selected),
config.n_trainval,
)
return
selected.append(pick)
pool.remove(pick)
def select_split(videos: list[VideoRecord], *, config: SelectConfig) -> SplitAssignment:
"""贪心联合约束视频级切分:floor 硬约束先满足、多样性覆盖后最大化、ε 守护 test 代表性。
核心洞察全数据集错题总数固定越把信号塞 trainvaltest 越简单headline 越虚高
test 必须保持代表性ε 约束trainval 只靠 floor + 多样性覆盖富集不从 test 偷难题
两阶段贪心均带 ε 守护 Floor 阶段满足各高信号类型 defect 下限不可行 fail loud
再多样性阶段按边际覆盖增益填至 n_trainval欠额记 warningtest = trainval 补集
确定性入场用 random.Random(seed) 对视频列表做一次预洗牌此后 max / 稳定排序仅取首个
seed 只打破等增益 / 等槽数平局 config videos 同结果
参数:
videos: 全部视频记录Task 7 build_video_records 产物
config: 选择配置关键字传入 n_trainval / floor_k / epsilon / reportable_types / seed
返回:
SplitAssignmenttrainval 按选择顺序test videos 原始顺序
异常:
InfeasibleSplitError: videos 为空floor ε 死锁无法满足
floor 需求超过 n_trainval 预算
"""
if not videos:
raise InfeasibleSplitError("videos 为空,无法执行切分")
rng = random.Random(config.seed)
pool = list(videos)
rng.shuffle(pool)
selected: list[VideoRecord] = []
_satisfy_floors(selected, pool, videos, config)
_maximize_diversity(selected, pool, videos, config)
trainval_ids = {video.video_id for video in selected}
trainval = tuple(video.video_id for video in selected)
test = tuple(video.video_id for video in videos if video.video_id not in trainval_ids)
return SplitAssignment(trainval=trainval, test=test)
+51
View File
@@ -0,0 +1,51 @@
"""把 predictions.steps_json 转成 RunLog.get_traces 的行形。
infer_adhoc traces 表为空轨迹存于 steps_json{thought, tool_call, tool_output}
诊断管线经 get_traces 消费轨迹故需此确定性转换适配
"""
from __future__ import annotations
import json
from typing import Any
def steps_json_to_trace_rows(
video_id: str, question_id: str, steps_json: str
) -> list[dict[str, Any]]:
"""将单题 steps_json 解析为 trace 行列表(step 从 0 递增)。
参数:
video_id: 视频 ID
question_id: ID
steps_json: predictions.steps_json 原文JSON 数组字符串
返回:
行字典列表字段对齐 traces schema/空数组返回 []
关键实现细节:
- 工具名读 tool_call.toolinfer_adhoc 真实字段对极少数历史
数据的 name back-compat 回退
- steps_json JSON 数组时直接报错不做兜底掩盖
"""
if not steps_json or not steps_json.strip():
return []
steps = json.loads(steps_json)
if not isinstance(steps, list):
raise ValueError(f"steps_json 非数组: {question_id}")
rows: list[dict[str, Any]] = []
for i, s in enumerate(steps):
call = s.get("tool_call") or {}
rows.append(
{
"video_id": video_id,
"question_id": question_id,
"step": i,
# infer_adhoc 用 "tool"back-compat 兼容极少数 "name"
"tool_name": call.get("tool", call.get("name")),
"tool_args": call.get("args", {}),
"tool_output": s.get("tool_output"),
"thought": s.get("thought"),
}
)
return rows
+402
View File
@@ -0,0 +1,402 @@
"""Store 版本操作 + Seed 管理。
Store 存储版本化资源视频题目SkillPrompt
通过版本号v1, v2, ...管理资源的演化历史
Seed 是可复现的实验起点包含权重快照 + baseline 数据库
"""
from __future__ import annotations
import json
import re
import shutil
import sqlite3
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from pathlib import Path
def _now_iso() -> str:
"""返回当前 UTC 时间的 ISO 格式字符串。"""
return datetime.now(UTC).isoformat()
def _parse_version(name: str) -> int:
"""解析版本目录名 ``v\\d+`` 为整数。
参数:
name: 版本目录名 ``"v1"````"v10"``
返回:
版本号整数
异常:
ValueError: 版本目录名格式不合法不匹配 ``v\\d+``
"""
match = re.match(r"v(\d+)$", name)
if not match:
raise ValueError(f"无效版本号: {name}")
return int(match.group(1))
def list_versions(store_dir: Path, resource_type: str) -> list[str]:
"""列出 Store 中某类资源的所有版本号,按数字值排序。
按数字排序保证 v10 排在 v2 后面而非字典序 v10 < v2
参数:
store_dir: Store 根目录
resource_type: 资源类型路径 ``"skills"````"questions/generated"``
返回:
排序后的版本号列表 ``["v1", "v2", "v10"]``
"""
resource_dir = store_dir / resource_type
if not resource_dir.is_dir():
return []
versions = []
for entry in resource_dir.iterdir():
if entry.is_dir() and re.match(r"v\d+$", entry.name):
versions.append(entry.name)
return sorted(versions, key=_parse_version)
def next_version(store_dir: Path, resource_type: str) -> str:
"""返回某类资源的下一个可用版本号。
参数:
store_dir: Store 根目录
resource_type: 资源类型路径
返回:
下一个版本号字符串 ``"v3"``
"""
versions = list_versions(store_dir, resource_type)
if not versions:
return "v1"
latest = _parse_version(versions[-1])
return f"v{latest + 1}"
def _write_meta(target_dir: Path, version: str, source: str, **extra: str | None) -> None:
"""写入版本元数据文件 ``meta.json``。
参数:
target_dir: 版本目录
version: 版本号
source: 来源标识``"manual"`` / ``"evolution"`` / ``"auto-gen"``
**extra: 额外字段parent, trigger_run, trigger_workspace, description
"""
meta = {
"version": version,
"created_at": _now_iso(),
"parent": extra.get("parent"),
"source": source,
"trigger_run": extra.get("trigger_run"),
"trigger_workspace": extra.get("trigger_workspace"),
"description": extra.get("description", ""),
}
(target_dir / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2))
def advance_version(
store_dir: Path,
resource_type: str,
source_dir: Path,
meta: dict,
) -> str:
"""将 source_dir 的内容写入 Store 的下一个版本目录,写入 meta.json。
参数:
store_dir: Store 根目录
resource_type: 资源类型路径 ``"skills"````"questions/generated"``
source_dir: 包含新版本资源文件的源目录
meta: 元数据字典至少包含 ``source`` 字段
返回:
新版本号字符串 ``"v2"``
"""
version = next_version(store_dir, resource_type)
target = store_dir / resource_type / version
shutil.copytree(source_dir, target)
_write_meta(
target,
version,
meta.get("source", "manual"),
parent=meta.get("parent"),
trigger_run=meta.get("trigger_run"),
trigger_workspace=meta.get("trigger_workspace"),
description=meta.get("description", ""),
)
logger.info("Store 版本推进: {}/{}", resource_type, version)
return version
def init_store(
store_dir: Path,
videos_source: Path,
skills_dir: Path,
prompts_dir: Path,
) -> None:
"""初始化 Store:拷贝视频数据,创建 skills/v1、prompts/v1 和 questions 目录。
参数:
store_dir: Store 目标路径不得已存在
videos_source: 视频数据源目录
skills_dir: 初始 Skill 文件目录
prompts_dir: 初始 Prompt 文件目录
异常:
FileExistsError: Store 目录已存在
"""
if store_dir.exists():
raise FileExistsError(f"Store 已存在: {store_dir}")
store_dir.mkdir(parents=True)
shutil.copytree(videos_source, store_dir / "videos")
(store_dir / "questions" / "benchmarks").mkdir(parents=True)
(store_dir / "questions" / "generated").mkdir(parents=True)
shutil.copytree(skills_dir, store_dir / "skills" / "v1")
_write_meta(
store_dir / "skills" / "v1",
"v1",
"manual",
description="手工创建的初始版本",
)
shutil.copytree(prompts_dir, store_dir / "prompts" / "v1")
_write_meta(
store_dir / "prompts" / "v1",
"v1",
"manual",
description="手工创建的初始版本",
)
logger.info("Store 初始化完成: {}", store_dir)
# ---------------------------------------------------------------------------
# 种子库(Seed)函数
# ---------------------------------------------------------------------------
def init_seed(
store_dir: Path,
name: str,
skills_dir: Path,
prompts_dir: Path,
baseline_db: Path,
baseline_run_id: str,
parent: str | None,
description: str,
*,
pools_json: Path | None = None,
split_manifest: Path | None = None,
) -> Path:
"""在 store/seeds/<name> 写一个种子:权重 + baseline.db + seed.json。
参数:
store_dir: Store 根目录
name: 种子名 ``'initial'````'from-evolve-v20'``
skills_dir: 该版本 Skill 权重源目录
prompts_dir: 该版本 Prompt 权重源目录
baseline_db: 该版本全量记录 db _runs + predictions
baseline_run_id: 全量记录的 run_idfresh 时注入 build_pools
parent: 来源initial None
description: 人类可读说明
pools_json: 可选冻结切分 pools.json 源路径提供时拷入 seed 目录
fresh 训练时携带冻结切分进 workspace init_workspace_from_seed
split_manifest: 可选冻结切分 split_manifest.json 源路径提供时拷入 seed
目录供加载时校验 pools.json 内容指纹pools_sha256
返回:
种子目录路径
异常:
FileExistsError: 同名种子已存在不覆盖
"""
seed_dir = store_dir / "seeds" / name
if seed_dir.exists():
raise FileExistsError(f"种子已存在,不覆盖: {seed_dir}")
seed_dir.mkdir(parents=True)
shutil.copytree(skills_dir, seed_dir / "skills")
shutil.copytree(prompts_dir, seed_dir / "prompts")
shutil.copy2(baseline_db, seed_dir / "baseline.db")
if pools_json is not None:
shutil.copy2(pools_json, seed_dir / "pools.json")
if split_manifest is not None:
shutil.copy2(split_manifest, seed_dir / "split_manifest.json")
(seed_dir / "seed.json").write_text(
json.dumps(
{
"baseline_run_id": baseline_run_id,
"parent": parent,
"created_at": _now_iso(),
"description": description,
},
ensure_ascii=False,
indent=2,
)
)
logger.info("种子创建完成: {}", seed_dir)
return seed_dir
def list_seeds(store_dir: Path) -> list[str]:
"""列出 store/seeds 下所有种子名(按名排序)。
参数:
store_dir: Store 根目录
返回:
种子名列表仅含 seed.json 存在的目录按名排序
"""
seeds_root = store_dir / "seeds"
if not seeds_root.is_dir():
return []
return sorted(e.name for e in seeds_root.iterdir() if (e / "seed.json").exists())
def read_seed(store_dir: Path, name: str) -> dict:
"""读取种子 seed.json;不存在则报错。
参数:
store_dir: Store 根目录
name: 种子名
返回:
seed.json 解析后的字典
异常:
FileNotFoundError: 该种子不存在
"""
seed_json = store_dir / "seeds" / name / "seed.json"
if not seed_json.exists():
raise FileNotFoundError(f"种子不存在: {name}{seed_json}")
return json.loads(seed_json.read_text())
def extract_run_db(
src_db: Path, dst_db: Path, run_id: str, *, dedupe_per_question: bool = False
) -> None:
"""从 src_db 抽出某 run_id 的 _runs + predictions 行,写一个最小 db(种子 baseline.db)。
用源表的**原始 CREATE 语句**重建目标表保留主键/列类型/约束
``_runs.run_id TEXT PRIMARY KEY`` HarnessLog ``INSERT OR IGNORE`` 去重的依据
seed db 丢主键则续训/fresh-bootstrap 的去重失效
参数:
src_db: harness.db
dst_db: 目标 db不得已存在
run_id: 要抽取的 run
dedupe_per_question: True predictions 表每 question_id 仅保留 rowid 最小
的首行对齐 canonical question_id 取第一行 ORDER BY rowid口径
902900_runs 表不受影响
异常:
RuntimeError: 源中无该表或无该 run 的行
"""
src = sqlite3.connect(src_db)
dst = sqlite3.connect(dst_db)
try:
for table in ("_runs", "predictions"):
create_sql = src.execute(
"SELECT sql FROM sqlite_master WHERE type='table' AND name=?",
(table,),
).fetchone()
if create_sql is None or create_sql[0] is None:
raise RuntimeError(f"源 db 无表 {table}")
dst.execute(create_sql[0])
cols = [r[1] for r in src.execute(f"PRAGMA table_info({table})")]
col_sql = ", ".join(cols)
if table == "predictions" and dedupe_per_question:
rows = src.execute(
f"SELECT {col_sql} FROM {table} WHERE run_id=? "
"AND rowid IN (SELECT MIN(rowid) FROM predictions "
"WHERE run_id=? GROUP BY question_id)",
(run_id, run_id),
).fetchall()
else:
rows = src.execute(
f"SELECT {col_sql} FROM {table} WHERE run_id=?", (run_id,)
).fetchall()
if not rows:
raise RuntimeError(f"{table} 中无 run_id={run_id} 的行")
ph = ", ".join("?" * len(cols))
dst.executemany(f"INSERT INTO {table} ({col_sql}) VALUES ({ph})", rows)
dst.commit()
finally:
dst.close()
src.close()
def promote_to_seed(
workspace_dir: Path,
store_dir: Path,
version: str,
eval_run_id: str,
name: str,
description: str,
) -> Path:
"""把 workspace 的指定版本 + 配套 prompts + 指定 eval run 全量记录固化成新种子。
强校验 eval_run_id 对应的 _runs 行中 skills_version 必须与 version 一致
skills_version/prompts_version 均不得为 NULL
参数:
workspace_dir: 来源 workspace
store_dir: Store 根目录
version: skills 版本号
eval_run_id: canonical eval run _runs 行提供配套 prompts 版本与全量记录
name: 新种子名冲突报错不覆盖
description: 说明
返回:
新种子目录
异常:
ValueError: eval_run_id 不存在或其 skills_version version 不符或版本为 NULL
FileExistsError: 同名种子已存在 init_seed 抛出
"""
con = sqlite3.connect(workspace_dir / "harness.db")
con.row_factory = sqlite3.Row
try:
row = con.execute(
"SELECT skills_version, prompts_version FROM _runs WHERE run_id=?",
(eval_run_id,),
).fetchone()
finally:
con.close()
if row is None:
raise ValueError(f"eval run 不存在: {eval_run_id}")
skills_v, prompts_v = row["skills_version"], row["prompts_version"]
# 强校验——eval run 的版本必须与 --version 一致,且不得为 NULL
if skills_v is None or prompts_v is None:
raise ValueError(f"eval run {eval_run_id} 的 _runs 版本对为 NULL(未回填?),无法 promote")
if skills_v != version:
raise ValueError(f"eval run {eval_run_id} 的版本 {skills_v} 与 --version {version} 不符")
tmp_db = workspace_dir / "_promote_tmp.db"
if tmp_db.exists():
tmp_db.unlink()
extract_run_db(workspace_dir / "harness.db", tmp_db, eval_run_id)
try:
seed_dir = init_seed(
store_dir,
name,
workspace_dir / "skills" / skills_v,
workspace_dir / "prompts" / prompts_v,
tmp_db,
baseline_run_id=eval_run_id,
parent=f"{workspace_dir.name}:{version}",
description=description,
)
finally:
tmp_db.unlink()
logger.info("Promote 完成: {} -> {}", workspace_dir.name, seed_dir)
return seed_dir
+89
View File
@@ -0,0 +1,89 @@
"""诊断侧树读取适配:把嵌套 tree.json 展平成诊断消费的扁平 nodes dict。
诊断编排core/evolution/diagnose.py期望 tree_data 形如
{"nodes": {node_id: {card, level, time_range}}} TRM5 建树产物
store/videos/<vid>/tree.json 是嵌套 {"metadata","roots":[...]}本模块递归展平
接通 TRM4TRM5 迁移时断掉的 ground_truth 加载环
不走 TreeIndex 对象层 L1Node to_dictapp/tree/index.py:260L2/L3 为其内部闭包
to_dict 输出无 levelL3 timestamp time_range直接遍历 json 更省且零改建树模块
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from pathlib import Path
def load_tree_nodes(store_dir: Path, video_id: str) -> dict[str, Any]:
"""加载单视频 tree.json 并展平成扁平 nodes dict。
参数:
store_dir: store 根目录 videos/<video_id>/tree.json
video_id: 视频标识
返回:
{"nodes": {node_id: {"card": dict, "level": int, "time_range": list}}}
异常:
FileNotFoundError: tree.json 不存在沿用 factory.py fail-loud 先例
ValueError: roots list 或为空节点缺 id或节点既无 time_range 又无 timestamp
关键实现:
level 由遍历深度赋值root=1/child=2/=3不解析 node_idnode_id 累积式
..._L1_..._L2_..._L3_用正则首匹配会把 L2/L3 误判成 1
L3 time_range timestamp 合成 [t, t]
"""
tree_path = store_dir / "videos" / video_id / "tree.json"
if not tree_path.exists():
raise FileNotFoundError(f"树索引文件不存在: {tree_path}(诊断需真实树,P5 fail loud")
tree = json.loads(tree_path.read_text(encoding="utf-8"))
roots = tree.get("roots")
if not isinstance(roots, list) or not roots:
raise ValueError(f"树无有效 roots: {tree_path}")
nodes: dict[str, Any] = {}
def _walk(node: dict[str, Any], level: int) -> None:
node_id = node.get("id")
if not isinstance(node_id, str) or not node_id:
raise ValueError(f"节点缺 id: {tree_path}")
time_range = node.get("time_range")
if time_range is None:
ts = node.get("timestamp")
if ts is None:
raise ValueError(
f"节点既无 time_range 又无 timestamp(树损坏): {node_id} in {tree_path}"
)
time_range = [ts, ts]
nodes[node_id] = {
"card": node.get("card", {}),
"level": level,
"time_range": time_range,
}
for child in node.get("children", []) or []:
_walk(child, level + 1)
for root in roots:
_walk(root, 1)
return {"nodes": nodes}
def load_tree_data_for_videos(store_dir: Path, video_ids: list[str]) -> dict[str, Any]:
"""按一组 video_id 去重加载展平树,供诊断按 video 注入。
参数:
store_dir: store 根目录
video_ids: 视频标识列表可含重复内部按首次出现顺序去重
返回:
{video_id: {"nodes": {...}}}
异常:
load_tree_nodes任一视频树缺失/无效即 fail-loud
"""
return {vid: load_tree_nodes(store_dir, vid) for vid in dict.fromkeys(video_ids)}
+933
View File
@@ -0,0 +1,933 @@
"""async 连续并发 gate 验证编排 — CE-Gate 局部验证的唯一独立子编排器。
多题型全部 (单元, ) 任务共享题槽并发validate_skills_concurrent
统计推进不按到达序而按预声明的阶梯序前缀消费_advance_prefix
base 臂缓存命中瞬间返回cand 臂必新鲜跑两臂延迟不对称按到达序判定
会系统性偏向早到翻转前缀消费把判定顺序钉回阶梯序anytime-valid 无条件
成立核心算法保真 #6,语义修订:块序贯 → 阶梯序前缀逐对序贯)。
基线与候选在同一阶梯前缀上逐单元配对只数翻转基线错候选对 = W
基线对候选错 = L每消费一个单元调一次 gate_decision 做四出口判定
过线即冻结τ 之后的 in-flight 结果整体丢弃基线侧单元级对错走
BaselineCache 内容寻址缓存miss 才新鲜跑INFRA 单元不写缓存从配对剔除
判定逻辑全部在 core/evolution/gate本模块只负责推理编排与证据收集
"""
from __future__ import annotations
import asyncio
import json
import shutil
import tempfile
from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from loguru import logger
from app.harness.gate_ladder import BaselineCache, skill_hash
from app.harness.question_units import build_units, unit_correctness_view
from core.evolution import (
INFRA_STOP_REASONS,
GateParams,
GateVerdict,
RejectedEdit,
classify_quadrants,
gate_decision,
pair_block,
)
# INFRA_STOP_REASONS 复用 core.evolution.diagnose 的单一定义(M-2):INFRA 故障
# stop_reason(推理侧基础设施错误,非模型答错)在诊断与 gate 两处必须同口径,
# 避免各自维护副本致未来漂移。
if TYPE_CHECKING:
from app.harness.inference import InferenceResult
from app.harness.log import HarnessLog
from core.types import GeneratedQuestion, QuestionUnit
# gate_decision 的 decision → ValidationOutcome.stop_reason 映射
_STOP_REASON_BY_DECISION: dict[str, str] = {
"accept_confirmed": "confirmed",
"reject_directional": "directional",
"reject_futility": "futility",
"accept_provisional": "provisional",
"reject_inertia": "inertia",
}
# ---------------------------------------------------------------------------
# 注入协议
# ---------------------------------------------------------------------------
@runtime_checkable
class RunInferenceFn(Protocol):
"""注入的推理函数协议。
调用方runner负责绑定 llmtool_dispatch_fnprompt_builder
logconcurrencymax_stepsskill_mode 等共享依赖
validate 侧只传 questionsrun_idskills_dir 三个逐任务变化的参数
"""
async def __call__(
self,
questions: list[GeneratedQuestion],
*,
run_id: str,
skills_dir: Path,
) -> InferenceResult: ...
# ---------------------------------------------------------------------------
# 数据类型
# ---------------------------------------------------------------------------
@dataclass
class ValidationOutcome:
"""CE-Gate 局部验证结果:三态动作 + e-process 证据(单元口径)+ 逐题溯源对错。
correctness 二轨语义W/L准确率四象限均按 **unit 口径** 统计
AR pair 双向 AND 折叠为一个单元不被 P/Q 单题计分污染
candidate_correctness 独立保留 **逐题** 对错只含已观测题早停后是阶梯前缀
子集accept 时由 runner question_id 粒度增量合并进 state.correctness
"""
action: str # accept_confirmed | accept_provisional | reject
accepted: bool
stop_reason: str # confirmed | directional | futility | provisional | inertia
e_value: float
w: int
l: int # noqa: E741
n_used: int
delta_hat: float
delta_shrunk: float
baseline_acc: float # 已观测单元上的基线准确率(unit 口径)
candidate_acc: float # 已观测单元上的候选准确率(unit 口径)
improvements: list[str] = field(default_factory=list)
regressions: list[str] = field(default_factory=list)
persistent_fails: list[str] = field(default_factory=list)
stable_successes: list[str] = field(default_factory=list)
candidate_correctness: dict[str, bool] = field(default_factory=dict)
evidence_rows: list[dict] = field(default_factory=list) # gate_evidence 逐题行,runner 落库
@dataclass
class Probation:
"""一个题型的在途试用账本(每题型至多一个)。
字段:
task_type: 题型
anchor_skills_version: 锚版本名最近一个 CONFIRMED skills 版本
回滚时恢复该版本中本题型 skill 文件的内容
target_file: 该题型解析后的 skill 文件名
correctness_snapshot: 开账时该题型 val 题的对错快照回滚时恢复
opened_step: 开账时的 global_step观测用
pending_edits: 试用链上全部候选 edit 的黑名单素材回滚时整链入黑名单
"""
task_type: str
anchor_skills_version: str
target_file: str
correctness_snapshot: dict[str, bool]
opened_step: int
pending_edits: list[RejectedEdit] = field(default_factory=list)
# ---------------------------------------------------------------------------
# 同步辅助函数
# ---------------------------------------------------------------------------
def materialize_candidate_skill(
workspace_dir: Path,
base_skills_version: str,
target_file: str,
content: str,
) -> Path:
"""将候选 skill 正文物化为 workspace 专用临时目录下唯一命名的候选 skills 目录。
复制基线 skills 目录到 .cand_tmp/ 下的唯一命名临时目录然后覆写 target_file
构建失败时尽力清理已建临时目录再重抛原始异常
参数:
workspace_dir: Workspace 根目录基线 skills workspace_dir/skills/<base>
复制临时候选落 workspace_dir/.cand_tmp/
base_skills_version: 基线 skills 版本名
target_file: 被替换的 skill 文件名
content: 候选 skill 文件全文
返回:
新建的临时候选目录绝对路径
契约:
构建失败OSError时尽力清理已建临时目录再重抛原始异常
清理本身失败记 warning
"""
cand_tmp_root = workspace_dir / ".cand_tmp"
cand_tmp_root.mkdir(parents=True, exist_ok=True)
cand_dir = Path(tempfile.mkdtemp(prefix=f"{base_skills_version}_cand_", dir=cand_tmp_root))
try:
base_dir = workspace_dir / "skills" / base_skills_version
shutil.copytree(base_dir, cand_dir, dirs_exist_ok=True)
(cand_dir / target_file).write_text(content, encoding="utf-8")
except OSError:
try:
shutil.rmtree(cand_dir)
except OSError as cleanup_err:
logger.warning("候选物化失败后清理临时目录也失败 {}: {}", cand_dir, cleanup_err)
raise
return cand_dir
def _load_run_rows(
log: HarnessLog,
run_id: str,
) -> dict[str, dict[str, Any]]:
"""读取单个 run 的逐题预测行并规范化轨迹字段。
predictions 表读取指定 run 的题目级记录补充 _correct
与规范化后的 steps 字段保持同步log.query仅在推理完成后调用
参数:
log: HarnessLog 共享实例 query 方法做只读 SELECT
run_id: 待读取的预测 run_id
返回:
question_id 为键的行字典每行至少包含 predictionanswer
_correctsteps 等字段
"""
rows = log.query(
"SELECT question_id, prediction, answer, stop_reason, steps_json "
"FROM predictions WHERE run_id=?",
(run_id,),
)
normalized: dict[str, dict[str, Any]] = {}
for row in rows:
raw_steps = row.get("steps_json")
parsed_steps: Any = raw_steps
if isinstance(raw_steps, str):
try:
parsed_steps = json.loads(raw_steps)
except json.JSONDecodeError:
parsed_steps = []
steps = parsed_steps if isinstance(parsed_steps, list) else []
normalized[row["question_id"]] = {
**row,
"_correct": row.get("prediction") == row.get("answer"),
"steps": steps,
}
return normalized
def _infra_question_ids_from_db(
log: HarnessLog,
run_id: str,
chunk: list[GeneratedQuestion],
) -> set[str]:
"""从 db 读取一个 run 中 stop_reason 属 INFRA 故障族的 question_id 集合。
参数:
log: HarnessLog 共享实例
run_id: 推理 run_id
chunk: 题目列表
返回:
stop_reason {"error", "parse_error"} question_id 集合
"""
rows = _load_run_rows(log, run_id)
return {
q.question_id
for q in chunk
if rows.get(q.question_id, {}).get("stop_reason") in INFRA_STOP_REASONS
}
def _candidate_correctness_from_db(
log: HarnessLog,
run_id: str,
chunk: list[GeneratedQuestion],
) -> dict[str, bool]:
"""从 db 读取候选/基线 run 在指定题目上的逐题对错。
参数:
log: HarnessLog 共享实例
run_id: 推理 run_id
chunk: 题目列表
返回:
question_id -> 是否答对的映射缺行的题目记为 False
"""
rows = _load_run_rows(log, run_id)
return {q.question_id: rows.get(q.question_id, {}).get("_correct", False) for q in chunk}
# ---------------------------------------------------------------------------
# INFRA 护栏
# ---------------------------------------------------------------------------
def _check_infra_guard(errors: int, infra_denom: int, gate_guard_err: float) -> None:
"""累计 INFRA 错误率护栏:分母 >=10 且超阈值时 raise。
参数:
errors: 两侧累计 error 计数
infra_denom: 两侧累计推理题次分母
gate_guard_err: 错误率阈值
异常:
RuntimeError: 错误率超阈值
"""
if infra_denom >= 10 and errors / infra_denom > gate_guard_err:
raise RuntimeError(f"gate 推理累计错误率过高 {errors / infra_denom:.0%},中止本轮")
# ---------------------------------------------------------------------------
# 终态组装
# ---------------------------------------------------------------------------
def _finalize_outcome(
verdict: GateVerdict,
w: int,
l: int, # noqa: E741
n_used: int,
n_plan: int,
base_obs: dict[str, bool],
cand_obs: dict[str, bool],
candidate_per_q: dict[str, bool],
evidence_rows: list[dict],
task_type: str,
) -> ValidationOutcome:
"""将终态判定组装为 ValidationOutcome。
四象限/准确率/W/L 均按单元口径base_obs/cand_obs unit_id -> bool
candidate_correctness 独立保留逐题溯源 runner 二轨合并进 state.correctness
参数:
verdict: 终态 gate 判定结果
w: 累计 W基线错候选对单元翻转
l: 累计 L基线对候选错单元翻转
n_used: 已消费的阶梯单元数
n_plan: 阶梯总单元数
base_obs: 累计基线已观测单元对错unit_id -> bool
cand_obs: 累计候选已观测单元对错unit_id -> bool
candidate_per_q: 累计候选逐题对错question_id -> bool溯源用
evidence_rows: 单元级证据行
task_type: 验证题型日志用
返回:
ValidationOutcome
"""
action = {
"accept_confirmed": "accept_confirmed",
"accept_provisional": "accept_provisional",
}.get(verdict.decision, "reject")
stop_reason = _STOP_REASON_BY_DECISION[verdict.decision]
# 只有终态单元的证据行才携带 stop_reason
evidence_rows[-1]["stop_reason"] = stop_reason
quadrants = classify_quadrants({uid: (base_obs[uid], cand_obs[uid]) for uid in base_obs})
baseline_acc = sum(base_obs.values()) / len(base_obs)
candidate_acc = sum(cand_obs.values()) / len(cand_obs)
accepted = action != "reject"
logger.info(
"gate 局部验证[{}]: 基线{:.1%} → 候选{:.1%} (W={} L={} E={:.2f} n={}/{} 单元) {}",
task_type,
baseline_acc,
candidate_acc,
w,
l,
verdict.e_value,
n_used,
n_plan,
"接受" if accepted else "回滚",
)
return ValidationOutcome(
action=action,
accepted=accepted,
stop_reason=stop_reason,
e_value=verdict.e_value,
w=w,
l=l,
n_used=n_used,
delta_hat=verdict.delta_hat,
delta_shrunk=verdict.delta_shrunk,
baseline_acc=baseline_acc,
candidate_acc=candidate_acc,
improvements=quadrants.improvements,
regressions=quadrants.regressions,
persistent_fails=quadrants.persistent_fails,
stable_successes=quadrants.stable_successes,
candidate_correctness=candidate_per_q,
evidence_rows=evidence_rows,
)
# ---------------------------------------------------------------------------
# 主编排
# ---------------------------------------------------------------------------
def _ladder_units(ladder_items: list[GeneratedQuestion]) -> list[QuestionUnit]:
"""把阶梯题序聚合为单元并保持信息阶梯顺序(按单元最早出现位置排序)。
build_units 会把 single pair 分组重排single pair 破坏"难题优先"
的阶梯序此处按单元内题目在 ladder 中的最早下标重排恢复原阶梯优先级
保证 AR pair 折叠不改变 e-process 的出题顺序核心算法保真 #5)。非 AR 全 single
时排序为恒等unit_id 等于 question_id位置即原序与迁移前逐题行为一致
参数:
ladder_items: 阶梯出题序可混含 single AR pair 成员
返回:
按阶梯序排列的单元列表
"""
units = build_units(ladder_items)
position = {q.question_id: i for i, q in enumerate(ladder_items)}
units.sort(key=lambda u: min(position[q.question_id] for q in u.questions))
return units
# ---------------------------------------------------------------------------
# 连续并发 gate:数据结构 + 前缀消费(algo #6 语义修订:块序贯 → 阶梯序前缀逐对序贯)
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class GateSpec:
"""单题型 gate 验证规格(runner 装配阶段产物,调度器输入)。
字段:
task_type: 题型
target_file: 解析后生效 skill 文件名(候选物化写此文件)
candidate_content: 候选 skill 全文
base_skill_content: 基线侧生效 skill 全文(skill_hash 作缓存键)
units: 阶梯序单元元组(已排除案例单元截断 gate_n_max);元组,装配后
不可变, spec.units run.slots 漂移
gate_run_prefix: run_id 前缀,必须含 "_gate_"(防泄露过滤依赖)
"""
task_type: str
target_file: str
candidate_content: str
base_skill_content: str
units: tuple[QuestionUnit, ...]
gate_run_prefix: str
@dataclass
class _UnitSlot:
"""单个阶梯单元的双臂到达状态。
base 为单元级对错(AR pair 已折叠);cand_per_q 为逐题对错(折叠交给消费时,
以复用 unit_correctness_view 并保留逐题溯源)INFRA 标志与结果互斥
"""
unit: QuestionUnit
base: bool | None = None
cand_per_q: dict[str, bool] | None = None
base_infra: bool = False
cand_infra: bool = False
def resolved(self) -> bool:
"""双臂均已出结果(含 INFRA 判定)。
返回:
base (结果或 INFRA) cand (结果或 INFRA)都已到达时为 True
"""
base_done = self.base is not None or self.base_infra
cand_done = self.cand_per_q is not None or self.cand_infra
return base_done and cand_done
def excluded(self) -> bool:
"""任一臂 INFRA 即整单元剔除(不入配对)。
返回:
base_infra cand_infra 任一为 True 时为 True
"""
return self.base_infra or self.cand_infra
@dataclass
class _GateRun:
"""单题型 gate 的运行时状态(计数器 + 前缀指针 + 证据)。"""
spec: GateSpec
slots: list[_UnitSlot]
s_hash: str
prefix_ptr: int = 0
w: int = 0
l: int = 0 # noqa: E741
n_used: int = 0
n_excluded: int = 0
errors: int = 0
infra_denom: int = 0
frozen: bool = False
verdict: GateVerdict | None = None
base_obs: dict[str, bool] = field(default_factory=dict)
cand_obs: dict[str, bool] = field(default_factory=dict)
candidate_per_q: dict[str, bool] = field(default_factory=dict)
evidence_rows: list[dict] = field(default_factory=list)
@classmethod
def from_spec(cls, spec: GateSpec) -> _GateRun:
"""由规格构造初始状态(slots 与阶梯序一一对应)。
参数:
spec: 单题型 gate 规格(units 已阶梯序)
返回:
计数器归零slots 逐单元初始化s_hash 已计算的 _GateRun
"""
return cls(
spec=spec,
slots=[_UnitSlot(unit=u) for u in spec.units],
s_hash=skill_hash(spec.base_skill_content),
)
def _advance_prefix(run: _GateRun, params: GateParams) -> None:
"""沿阶梯序消费"已配齐前缀",逐单元更新 (W,L) 并判定,过线即冻结。
统计合法性关键(设计 v3 §1 / Codex C1):严禁按到达序消费base 臂缓存命中
瞬间返回cand 臂必新鲜跑,两臂延迟不对称, cand 延迟与对错相关,早到翻转
对系统性偏向 W e-值虚高假接受前缀消费把判定顺序钉回预声明阶梯序,
anytime-valid 无条件成立;INFRA 单元视为"已解决(剔除)"不阻塞前缀
契约:全部单元被剔除时 verdict 保持 Nonefrozen 保持 False,由调度编排层
(Task 3 validate_skills_concurrent)检测 verdict None raise
RuntimeError;本函数不负责该终态
参数:
run: 单题型 gate 运行时状态(原地更新计数器/指针/证据)
params: e-process 判据阈值组
返回:
(所有效果原地写入 run;可重复调用,已消费前缀不重复消费)
"""
while not run.frozen and run.prefix_ptr < len(run.slots):
slot = run.slots[run.prefix_ptr]
if not slot.resolved():
return
rank = run.prefix_ptr
run.prefix_ptr += 1
if slot.excluded():
run.n_excluded += 1
# 剔除使 n_remaining 缩小,必须重判(Codex plan 审 C1):否则尾部全 INFRA
# 时 verdict 停留在 "continue",绕过题尽第四出口且 _finalize_outcome
# 查 stop_reason 映射 KeyError。n_used==0(纯前导 INFRA)时无证据可判,跳过。
if run.n_used > 0:
n_remaining = (len(run.slots) - run.n_excluded) - run.n_used
run.verdict = gate_decision(run.w, run.l, run.n_used, n_remaining, params=params)
if run.verdict.decision != "continue":
run.frozen = True
continue
uid = slot.unit.unit_id
assert slot.base is not None and slot.cand_per_q is not None, (
f"slot 未配齐即被消费: unit={slot.unit.unit_id}"
)
c_units = unit_correctness_view([slot.unit], slot.cand_per_q)
pair_result = pair_block({uid: slot.base}, c_units, [uid])
run.candidate_per_q.update(slot.cand_per_q)
for u, (b, c) in pair_result.observed.items():
run.base_obs[u] = b
run.cand_obs[u] = c
run.w += pair_result.w
run.l += pair_result.l
run.n_used += 1
n_remaining = (len(run.slots) - run.n_excluded) - run.n_used
run.verdict = gate_decision(run.w, run.l, run.n_used, n_remaining, params=params)
run.evidence_rows.append(
{
"question_id": uid,
"task_type": run.spec.task_type,
"ladder_rank": rank,
"baseline_correct": slot.base,
"candidate_correct": c_units[uid],
"e_value": run.verdict.e_value,
"stop_reason": "",
}
)
if run.verdict.decision != "continue":
run.frozen = True
class _QuestionSlots:
"""按题数计数的共享并发闸:峰值在飞请求恒 ≤ width(设计 v3 §2.4)。
多槽获取(AR pair 一单元两题)经内部锁串行化,防多任务半持有交错死锁
本类只承诺"并发上限 + 多槽获取原子性";公平性由调用方按题型 round-robin
顺序创建任务实现(实践中 asyncio 等待队列近似先来先服务,但那不是本类契约)
"""
def __init__(self, width: int) -> None:
"""初始化题槽闸。
参数:
width: 并发宽度( gate 同时在飞的题数上限),必须为正
返回:
关键实现细节:
_width acquire 做超宽 fail-fast;BoundedSemaphore 使多还立即
ValueError 而非静默扩容;_acquire_lock 串行化多槽获取防交错死锁
"""
assert width > 0, f"并发宽度必须为正: {width}"
self._width = width
# BoundedSemaphore:多还立即 ValueError 而非静默扩容(Codex 质量审 3)
self._sem = asyncio.BoundedSemaphore(width)
self._acquire_lock = asyncio.Lock()
async def acquire(self, n: int) -> None:
"""原子获取 n 个题槽。
fail-fast:n > 宽度时任务持锁等待永不满足的槽位 自死锁
(AR pair 单元 2 + width=1 的病态配置,Codex plan C2),直接报错
取消安全:半持有自动回滚逐槽获取途中被取消(或任何 BaseException)
,已拿到的 permit 全部归还再重抛,容量不泄漏(Codex 质量审 2)
参数:
n: 申请的题槽数(单元内题目数,single=1 / AR pair=2)
返回:
(成功返回即持有 n 个槽,须与 release(n) 配对)
异常:
ValueError: n 超过并发宽度(否则自死锁)
"""
if n > self._width:
raise ValueError(f"单次申请题槽 {n} 超过并发宽度 {self._width},将自死锁")
async with self._acquire_lock:
got = 0
try:
for _ in range(n):
await self._sem.acquire()
got += 1
except BaseException:
for _ in range(got):
self._sem.release()
raise
def release(self, n: int) -> None:
"""归还 n 个题槽。
参数:
n: acquire 对应的题槽数
返回:
关键实现细节:
底层为 BoundedSemaphore多还(release 数超过 acquire)立即
ValueError 暴露调用方配对错误,属防御性设计
"""
for _ in range(n):
self._sem.release()
async def _run_unit_arm(
run: _GateRun,
slot_idx: int,
arm: str,
slots: _QuestionSlots,
run_inference: RunInferenceFn,
log: HarnessLog,
baseline_cache: BaselineCache,
prompts_version: str,
base_skills_dir: Path,
cand_dir: Path,
gate_params: GateParams,
gate_guard_err: float,
) -> None:
"""执行一个 (单元, 臂) 任务:缓存/推理 → 到达登记 → 前缀消费推进。
冻结检查三次:启动时(排队任务撤销点)获得题槽后(获槽期间被冻结)
推理返回后(τ 之后的 in-flight 结果不计入,整体丢弃)
base 臂缓存命中不占题槽(零推理);INFRA 单元不写缓存(不永久污染基线快照)
护栏在每次臂完成时检查(等价迁移自跨块累计,设计 v3 §2.3),超阈值 raise
中止整轮(与现行行为一致)
参数:
run: 该题型的 gate 运行时状态
slot_idx: 单元在阶梯中的下标
arm: "base" "cand"
slots: gate 共享题槽闸
run_inference: 注入推理函数
log: HarnessLog 共享实例(推理后读预测)
baseline_cache / prompts_version: 基线缓存及键成分
base_skills_dir / cand_dir: 两臂各自的 skills 目录
gate_params: e-process 判据(前缀消费用)
gate_guard_err: INFRA 错误率护栏阈值
返回:
(结果写入 run.slots[slot_idx] 并触发 _advance_prefix)
异常:
RuntimeError: 累计 INFRA 错误率超护栏阈值( _check_infra_guard)
"""
assert arm in ("base", "cand")
if run.frozen:
return
slot = run.slots[slot_idx]
spec = run.spec
if arm == "base":
cached = baseline_cache.get(spec.task_type, run.s_hash, prompts_version, slot.unit.unit_id)
if cached is not None:
slot.base = cached
_advance_prefix(run, gate_params)
return
questions = list(slot.unit.questions)
await slots.acquire(len(questions))
try:
if run.frozen:
return
run_id = f"{spec.gate_run_prefix}_{arm}"
skills_dir = base_skills_dir if arm == "base" else cand_dir
r = await run_inference(questions, run_id=run_id, skills_dir=skills_dir)
# 推理 await 期间该题型可能已被其他任务判定冻结:设计语义是
# "τ(冻结时刻)之后的 in-flight 结果不计入"——整体丢弃,不写
# slot/infra_denom/errors,滞后 INFRA 也不得触发护栏 raise 掀翻
# 整轮 gather(Codex 质量审 1)。
if run.frozen:
return
_register_arm_arrival(
run=run,
slot=slot,
arm=arm,
questions=questions,
inference_run_id=r.run_id,
inference_total=r.total,
log=log,
baseline_cache=baseline_cache,
prompts_version=prompts_version,
)
_check_infra_guard(run.errors, run.infra_denom, gate_guard_err)
finally:
slots.release(len(questions))
_advance_prefix(run, gate_params)
def _register_arm_arrival(
run: _GateRun,
slot: _UnitSlot,
arm: str,
questions: list[GeneratedQuestion],
inference_run_id: str,
inference_total: int,
log: HarnessLog,
baseline_cache: BaselineCache,
prompts_version: str,
) -> None:
"""把一次臂推理结果登记进 slot 与 run 计数器(INFRA 判定 + 对错折叠 + 回写缓存)。
INFRA 臂只标记不写缓存(不永久污染基线快照);正常 base 臂折叠为单元级对错并
回写 BaselineCache,正常 cand 臂保留逐题对错(折叠交给前缀消费,保留逐题溯源)
参数:
run: 该题型的 gate 运行时状态(errors / infra_denom 原地累加)
slot: 本单元的双臂到达状态(结果或 INFRA 标志原地写入)
arm: "base" "cand"
questions: 本单元展开后的题目列表
inference_run_id: 本次推理的 run_id(DB 回读键)
inference_total: 本次推理的题次数(护栏分母增量)
log: HarnessLog 共享实例(推理后读预测)
baseline_cache / prompts_version: 基线缓存及键成分
返回:
(所有效果原地写入 run slot)
关键实现细节:
errors 按单元级去重(Codex plan I3):同一单元双臂都 INFRA 只计 1
error,与设计 §2.3"分子=INFRA 单元数(任一臂)"及旧块实现口径一致
(旧实现 cand 不跑 base-INFRA 单元,天然无双计)
"""
spec = run.spec
infra_qids = _infra_question_ids_from_db(log, inference_run_id, questions)
run.infra_denom += inference_total
if infra_qids:
if not slot.excluded():
run.errors += 1
if arm == "base":
slot.base_infra = True
else:
slot.cand_infra = True
return
per_q = _candidate_correctness_from_db(log, inference_run_id, questions)
if arm == "base":
folded = unit_correctness_view([slot.unit], per_q)
slot.base = folded[slot.unit.unit_id]
baseline_cache.put(
spec.task_type, run.s_hash, prompts_version, slot.unit.unit_id, slot.base
)
else:
slot.cand_per_q = per_q
def _validate_gate_specs(specs: list[GateSpec]) -> None:
"""校验各题型 gate 规格,不合法直接报错(不兜底)。
参数:
specs: 各题型 gate 规格
异常:
ValueError: 阶梯为空, gate_run_prefix "_gate_"(防泄露过滤依赖
该标记识别 gate run)
"""
for spec in specs:
if "_gate_" not in spec.gate_run_prefix:
raise ValueError(f"gate_run_prefix 必须含 '_gate_': {spec.gate_run_prefix!r}")
if not spec.units:
raise ValueError(f"task_type={spec.task_type} 阶梯为空,无法验证")
def _cleanup_candidate_dirs(cand_dirs: dict[str, Path]) -> None:
"""尽力清理全部候选临时目录,单个失败只记 warning 不中断其余清理。
参数:
cand_dirs: task_type -> 候选临时目录路径
返回:
"""
for d in cand_dirs.values():
try:
shutil.rmtree(d)
except OSError as e:
logger.warning("候选临时目录清理失败 {}: {}", d, e)
def _build_launch_order(runs: list[_GateRun]) -> list[tuple[_GateRun, int, str]]:
"""构建 (run, rank, arm) 发射队列:题型 round-robin × 题型内阶梯序。
交错顺序 = rank 0 各题型 rank 1 各题型 ...;同一 (题型, rank)
base cand round-robin 让各题型的阶梯头部同批起跑,配合前缀消费
使统计推进不因某题型阶梯过长而饿死其他题型
参数:
runs: 各题型 gate 运行时状态(slots 已按阶梯序初始化)
返回:
(run, rank, arm) 三元组列表,即任务创建顺序
"""
order: list[tuple[_GateRun, int, str]] = []
max_rank = max((len(r.slots) for r in runs), default=0)
for rank in range(max_rank):
for r in runs:
if rank < len(r.slots):
for arm in ("base", "cand"):
order.append((r, rank, arm))
return order
async def validate_skills_concurrent(
workspace_dir: Path,
base_skills_version: str,
specs: list[GateSpec],
gate_params: GateParams,
gate_guard_err: float,
baseline_cache: BaselineCache,
prompts_version: str,
run_inference: RunInferenceFn,
log: HarnessLog,
concurrency: int,
) -> dict[str, ValidationOutcome]:
"""连续并发 gate:多题型全部臂共享题槽并发,统计按阶梯序前缀有序推进。
关键实现细节:
发射顺序 = 题型 round-robin × 题型内阶梯序(base cand );题型过线
即冻结,其排队任务启动时自查冻结标志撤销,in-flight 结果不计入(τ 之后
样本,合法丢弃);候选目录逐个物化即登记统一 finally 清理(中途失败不
泄漏);任一任务异常先 cancel+排水其余任务再向上传播;全部题型判定后
统一经 _finalize_outcome 组装
参数:
workspace_dir: workspace 根目录(候选物化用)
base_skills_version: 基线 skills 版本名
specs: 各题型 gate 规格(units 已阶梯序 + 截断 n_max)
gate_params: e-process 判据阈值组
gate_guard_err: INFRA 错误率护栏阈值
baseline_cache: 基线侧单元级对错缓存
prompts_version: 当前 prompts 版本(缓存键成分)
run_inference: 注入推理函数(调用方须绑定共享 HarnessLog)
log: HarnessLog 共享实例(推理后读预测, run_inference 同库)
concurrency: 题槽宽度(峰值在飞题数上限)
返回:
{task_type: ValidationOutcome}
异常:
RuntimeError: INFRA 护栏超阈值,或某题型全部单元被 INFRA 排除
ValueError: spec 校验失败(空阶梯 / run_prefix "_gate_")
"""
_validate_gate_specs(specs)
base_skills_dir = workspace_dir / "skills" / base_skills_version
runs = [_GateRun.from_spec(s) for s in specs]
cand_dirs: dict[str, Path] = {}
slots_gate = _QuestionSlots(concurrency)
try:
# 成功一个登记一个:第 N 个题型物化抛 OSError 时,已登记的前 N-1 个
# 目录仍由 finally 统一清理,不泄漏(Codex 质量审 C001)。
for r in runs:
cand_dirs[r.spec.task_type] = materialize_candidate_skill(
workspace_dir, base_skills_version, r.spec.target_file, r.spec.candidate_content
)
tasks = [
asyncio.ensure_future(
_run_unit_arm(
r,
rank,
arm,
slots_gate,
run_inference,
log,
baseline_cache,
prompts_version,
base_skills_dir,
cand_dirs[r.spec.task_type],
gate_params,
gate_guard_err,
)
)
for r, rank, arm in _build_launch_order(runs)
]
# 护栏 raise 中止整轮的语义不变(Codex 质量审 C002):首异常先取消其余
# 任务并排水(return_exceptions 吞取消回报),确保外层 finally 删除候选
# 目录时已无在飞任务访问该目录、事件循环收尾无 pending task 警告;
# _run_unit_arm 的题槽获取自带取消回滚,cancel 安全。
try:
await asyncio.gather(*tasks)
except BaseException:
for t in tasks:
t.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
raise
finally:
_cleanup_candidate_dirs(cand_dirs)
outcomes: dict[str, ValidationOutcome] = {}
for r in runs:
if r.verdict is None:
raise RuntimeError(
f"gate[{r.spec.task_type}] 全部 unit 被判为 INFRA 排除,无法验证(检查推理基础设施)"
)
outcomes[r.spec.task_type] = _finalize_outcome(
verdict=r.verdict,
w=r.w,
l=r.l,
n_used=r.n_used,
n_plan=len(r.slots),
base_obs=r.base_obs,
cand_obs=r.cand_obs,
candidate_per_q=r.candidate_per_q,
evidence_rows=r.evidence_rows,
task_type=r.spec.task_type,
)
return outcomes
+824
View File
@@ -0,0 +1,824 @@
"""结果驱动视频级切分的自包含两阶段 CLI 入口。
把整条离线管线的编排从 shell 搬进 Python一次调用内联串起
Phase 1 离线诊断run_baseline_diagnosisLLM 重活断点续跑幂等
Phase 2 冻结切分build_split code-controlled产出 pools.json + manifest
McNemar 功效护栏validation 池错题数达阈校验
复现锚点约定C-2
- pools.json 的内容+ seed + diag_fingerprint是切分的**复现锚点**相同输入
产出字节级相同的 pools.json pools_sha256
- manifest generated_at **溯源元数据**非复现锚点真实运行默认盖真实 UTC
now记录本次切分何时产出但可用 `--generated-at <ISO>` 显式固定以对
manifest 做字节级复现比对write_manifest 库内不调 datetime.now时间戳一律由
CLI 传入
设计要点
- 诊断口径指纹 = (诊断 prompt 版本, 模型名, git SHA) 三分量合成隔离不同
诊断配置的信号 prompt / 模型 / 代码实现即换指纹旧信号不被覆盖
- 真实依赖组装参考 app/harness/runner.py::_run_diagnosisGovernedLLMClient
(search llm, thinking=True) + RunLogImpl(harness.db) + VersionedSkillStore +
DiagnosePrompts(项目根 prompts/) + tree_data wrong_ids 涉及 video 预加载
store/videos/<vid>/tree.json 展平
- .env / config 关键项一律 fail loudP5绝不静默兜底
- `--dry-run` 用假 deps 跑通两阶段 wiring 不真调 LLM打印将执行的步骤 + 指纹
用于校验装配正确性对齐 CLAUDE.md §2.5 smoke test
编排函数run_pipeline通过依赖注入接收 DiagnosisDeps / signal_store / wrong_ids /
questions / canonical_preds便于单测用假实现替换不触真实 LLM harness.db
其中 canonical_preds Phase 0 INFRA / 空预测错题的 T0 信号这些题不进诊断
"""
from __future__ import annotations
import argparse
import asyncio
import datetime
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Any
import yaml
from loguru import logger
from app.harness.baseline_diagnosis import DiagnosisDeps, run_baseline_diagnosis
from app.harness.build_split import (
SplitBuildConfig,
SplitBuildResult,
build_split,
load_canonical_predictions,
)
from app.harness.split_selection import diag_fingerprint
from app.question_gen.loader import load_benchmark
from core.evolution.types import DiagnosisSignalRow
if TYPE_CHECKING:
from app.harness.pools import Pools
from core.evolution.protocols import DiagnosisSignalStore
from core.types import GeneratedQuestion
# 与 core.evolution.diagnose._INFRA_STOP_REASONS 对齐:执行/解析层失败排除出可诊断错题。
_INFRA_STOP_REASONS: frozenset[str] = frozenset({"error", "parse_error"})
# 工程路径默认值(少变;可经 CLI 单次覆盖)。诊断信号表建在 harness.db。
_DEFAULT_HARNESS_DB = Path("workspaces/default/harness.db")
_DEFAULT_QUESTIONS_DIR = Path("store/questions/benchmarks/Video-MME")
_DEFAULT_OUT_DIR = Path("workspaces/video-split")
_DEFAULT_STORE_DIR = Path("store") # tree.json 在 store/videos/<vid>/
# ---------------------------------------------------------------------------
# 配置解析(fail loud
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class VideoSplitConfig:
"""结果驱动视频级切分的科研旋钮快照(从 config/video_split.yaml 解析)。
字段:
baseline_run_id: 基线 run 标识错题诊断与切分依据
n_trainval: trainval 目标视频数多样性阶段填充上限
epsilon: test 相对全局最大允许分布偏差题型 / 难度两维
report_floor: per-type 报告门限题数 此值的 task_type 才入 ε 约束
val_wrong_min: validation 池最少错题数McNemar 功效阈0=不检查
val_ratio: validation trainval 视频组总数的比例
seed: 贪心选择器预洗牌 + 视频组题级切分种子
floor_k: 各高信号 task_type T2 defect 下限硬约束
prompt_version: 诊断 prompt 版本标识指纹分量
model: 执行诊断的模型名指纹分量
"""
baseline_run_id: str
n_trainval: int
epsilon: float
report_floor: int
val_wrong_min: int
val_ratio: float
seed: int
floor_k: dict[str, int]
prompt_version: str
model: str
def _require(section: dict[str, Any], keys: tuple[str, ...], where: str) -> None:
"""校验 section 含全部必填键,缺任一即 fail loud(P5,不静默兜底)。
参数:
section: 待校验的配置子字典
keys: 必填键元组
where: 出错信息中标注的段名 "video_split"
异常:
SystemExit: 存在缺失键
"""
missing = [k for k in keys if k not in section]
if missing:
raise SystemExit(f"config {where} 段缺关键项 {missing},无法运行(P5 fail loud")
def parse_config(raw: dict[str, Any]) -> VideoSplitConfig:
"""把 yaml 原始字典解析为 VideoSplitConfig,缺关键项 fail loud。
参数:
raw: yaml.safe_load 的顶层字典需含 video_split / diag 两段
返回:
VideoSplitConfig 冻结快照
异常:
SystemExit: video_split / diag 段或段内关键项
"""
if "video_split" not in raw or "diag" not in raw:
raise SystemExit("config 缺 video_split / diag 段,无法运行(P5 fail loud")
vs = raw["video_split"]
dg = raw["diag"]
_require(
vs,
(
"baseline_run_id",
"n_trainval",
"epsilon",
"report_floor",
"val_wrong_min",
"val_ratio",
"seed",
"floor_k",
),
"video_split",
)
_require(dg, ("prompt_version", "model"), "diag")
return VideoSplitConfig(
baseline_run_id=vs["baseline_run_id"],
n_trainval=vs["n_trainval"],
epsilon=vs["epsilon"],
report_floor=vs["report_floor"],
val_wrong_min=vs["val_wrong_min"],
val_ratio=vs["val_ratio"],
seed=vs["seed"],
floor_k=dict(vs["floor_k"]),
prompt_version=dg["prompt_version"],
model=dg["model"],
)
def load_config(config_path: Path) -> VideoSplitConfig:
"""读取并解析 video_split yaml 配置文件(缺文件 / 关键项 fail loud)。
参数:
config_path: yaml 配置路径
返回:
VideoSplitConfig
异常:
SystemExit: 文件不存在或缺关键项
"""
if not config_path.exists():
raise SystemExit(f"config 文件不存在: {config_path}P5 fail loud")
raw = yaml.safe_load(config_path.read_text(encoding="utf-8"))
return parse_config(raw)
def git_short_sha() -> str:
"""取当前 git 短 SHA 作为诊断口径指纹的代码分量(诊断代码变则指纹变)。
返回:
git rev-parse --short HEAD 输出去空白
异常:
SystemExit: git 仓库或 git 不可用fail loud指纹不可缺分量
"""
try:
out = subprocess.run(
["git", "rev-parse", "--short", "HEAD"],
capture_output=True,
text=True,
check=True,
)
except (subprocess.CalledProcessError, FileNotFoundError) as exc:
raise SystemExit(f"无法获取 git 短 SHA 作为诊断代码版本: {exc}P5 fail loud") from exc
sha = out.stdout.strip()
if not sha:
raise SystemExit("git rev-parse --short HEAD 返回空,诊断指纹缺代码分量(P5 fail loud")
return sha
# ---------------------------------------------------------------------------
# 真实依赖组装(参考 runner.py::_run_diagnosis
# ---------------------------------------------------------------------------
class _DiagLLMSettings:
"""诊断 LLM 的工程配置(从 .env 读取 search llm 凭证 + 韧性旋钮)。
仅承载诊断所需字段搜索 LLM = 诊断 judge不复用 main.InfraSettings 以免
构造整套适配器embed / vlm的重活缺关键凭证 fail loud
"""
def __init__(self) -> None:
from pydantic_settings import BaseSettings, SettingsConfigDict
class _Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
search_llm_model: str = ""
search_llm_base_url: str = ""
search_llm_api_key: str = ""
redis_url: str = ""
redis_cache_ttl: int = 86400
llm_timeout: float = 300.0
llm_max_retries: int = 3
llm_retry_base_delay: float = 20.0
llm_retry_max_delay: float = 120.0
llm_circuit_breaker_threshold: int = 48
llm_circuit_breaker_cooldown: float = 60.0
llm_ttft_timeout: float = 30.0
llm_inter_token_timeout: float = 15.0
self._s = _Settings()
def __getattr__(self, name: str) -> Any:
return getattr(self._s, name)
def _build_redis_cache(settings: Any) -> Any | None:
"""按 .env redis_url 构建响应缓存(不可用则降级 None,与 main 一致)。"""
if not settings.redis_url:
return None
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
try:
import redis.asyncio as aioredis
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
return RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
except Exception:
logger.warning("Redis 缓存不可用,诊断降级为无缓存模式")
return None
def build_diagnosis_deps(
*,
harness_db: Path,
store_dir: Path,
video_ids: list[str],
concurrency: int,
expected_model: str,
) -> DiagnosisDeps:
"""组装 Phase 1 诊断的真实依赖束(GovernedLLMClient + RunLogImpl + prompts)。
runner.py::_run_diagnosis 对齐search LLMthinking=True作诊断 judge
RunLogImpl 只读读取 harness.db predictions/tracesVersionedSkillStore 读技能
DiagnosePrompts 从项目根 prompts/ 加载tree_data video_ids
store/videos/<vid>/tree.json 展平预加载诊断需真实树缺失即 fail-loud
参数:
harness_db: harness.db 路径诊断读预测 + 信号落库同库
store_dir: store 根目录 videos/<vid>/tree.json
video_ids: wrong_ids 涉及的 video 标识列表可含重复内部去重加载树
concurrency: 诊断并发上限
expected_model: config.diag.model诊断口径指纹的模型分量必须与 .env
SEARCH_LLM_MODEL 一致指纹里的 model 与实际诊断所用 model 不一致会让
信号以错误模型指纹落库破坏可复现 / resume / 口径隔离故此处 fail loud
返回:
DiagnosisDeps 冻结依赖束
异常:
SystemExit: .env search LLM 凭证model / base_url / api_key 任一为空
config.diag.model .env SEARCH_LLM_MODEL 不一致指纹漂移防护
FileNotFoundError / ValueError: video_ids 中任一 video tree.json 缺失或
无效load_tree_data_for_videos fail-loud诊断需真实树
"""
from adapters.breaker import CircuitBreaker
from adapters.llm import GovernedLLMClient
from adapters.telemetry import SQLiteTelemetryRecorder
from app.harness.log import RunLogImpl
from app.harness.workspace import VersionedSkillStore
settings = _DiagLLMSettings()
if not (
settings.search_llm_model and settings.search_llm_base_url and settings.search_llm_api_key
):
raise SystemExit(
"诊断 LLM 凭证缺失:.env 需配置 SEARCH_LLM_MODEL / SEARCH_LLM_BASE_URL / "
"SEARCH_LLM_API_KEYP5 fail loud,不静默兜底)"
)
if expected_model != settings.search_llm_model:
raise SystemExit(
"诊断模型指纹漂移:config.diag.model="
f"{expected_model!r} 与 .env SEARCH_LLM_MODEL={settings.search_llm_model!r} "
"不一致;指纹里的 model 必须等于实际诊断所用 modelP5 fail loud"
"请对齐 config/video_split.yaml diag.model 与 .env SEARCH_LLM_MODEL"
)
telemetry_db = Path("logs/telemetry.db")
telemetry_db.parent.mkdir(parents=True, exist_ok=True)
telemetry = SQLiteTelemetryRecorder(telemetry_db)
breaker = CircuitBreaker(
fail_threshold=max(settings.llm_circuit_breaker_threshold, 1),
cooldown_s=settings.llm_circuit_breaker_cooldown,
)
llm = GovernedLLMClient(
model=settings.search_llm_model,
base_url=settings.search_llm_base_url,
api_key=settings.search_llm_api_key,
provider=settings.search_llm_model.split("-")[0],
thinking=True,
breaker=breaker,
cache=_build_redis_cache(settings),
telemetry=telemetry,
timeout_s=settings.llm_timeout,
ttft_timeout_s=settings.llm_ttft_timeout,
inter_token_timeout_s=settings.llm_inter_token_timeout,
max_retries=settings.llm_max_retries,
retry_base_delay_s=settings.llm_retry_base_delay,
retry_max_delay_s=settings.llm_retry_max_delay,
)
from app.harness.tree_nodes import load_tree_data_for_videos
return DiagnosisDeps(
run_log=RunLogImpl(str(harness_db)),
llm=llm,
skill_store=VersionedSkillStore(_diagnosis_skills_dir()),
prompts=_load_diagnose_prompts(),
tree_data=load_tree_data_for_videos(store_dir, video_ids),
concurrency=concurrency,
)
def _diagnosis_skills_dir() -> Path:
"""诊断用技能目录:种子 store 的当前技能版本(诊断读技能遵从判定)。
诊断只读技能内容判断"是否遵从技能" store 种子 v1 即可与基线 run 一致
"""
return Path("store/skills/v1")
def _load_diagnose_prompts() -> Any:
"""加载诊断模板束(从项目根 prompts/ 读取;与 runner._load_diagnose_prompts 一致)。"""
from core.evolution.types import DiagnosePrompts
def _read(name: str) -> str:
p = Path("prompts") / name
if not p.exists():
raise FileNotFoundError(f"缺进化/诊断模板: {p}(请从 TRM4 迁移或检查 prompts/")
return p.read_text(encoding="utf-8")
return DiagnosePrompts(
defect_vs_lapse=_read("defect_vs_lapse.md"),
reasoning_sub=_read("reasoning_sub.md"),
span_eval_system=_read("span_eval_system.md"),
missed_nodes=_read("missed_nodes.md"),
skill_adherence=_read("skill_adherence.md"),
confirmation_bias=_read("confirmation_bias.md"),
evidence_sufficiency=_read("evidence_sufficiency.md"),
)
def select_diagnosable_wrong_ids(preds: list[dict]) -> list[str]:
"""从 canonical 预测筛出可诊断错题 question_id(保序)。
可诊断错题判据预测非空 stop_reason INFRAerror / parse_error
归一后预测 != 答案INFRA / 空预测错题不进 wrong_ids它们改由
persist_infra_t0_rows 直接落 T0run_diagnosis 内部也会二次排除同类题
参数:
preds: load_canonical_predictions 产出的 canonical 预测行已按 qid 去重
返回:
可诊断错题 question_id 列表 preds 顺序
"""
wrong_ids: list[str] = []
for pred in preds:
prediction = (pred["prediction"] or "").strip()
if not prediction or pred["stop_reason"] in _INFRA_STOP_REASONS:
continue
if not pred["correct"]:
wrong_ids.append(pred["question_id"])
return wrong_ids
def persist_infra_t0_rows(
store: DiagnosisSignalStore,
preds: list[dict],
baseline_run_id: str,
diag_fingerprint: str,
) -> int:
"""把非正确且 INFRA / 空预测的错题以 T0 信号行 upsert 落库(幂等)。
这些题stop_reason {error, parse_error} 或预测为空从不进入 run_diagnosis
筛选时被前置排除故其 T0 信号必须在此单独补齐否则 signal store 缺这些行
tier 分布 / manifest 不完整计划要求 4 INFRA 空预测错题 T0
投影口径与 baseline_diagnosis INFRA 投影一致infra=Truetier="T0"
error_type / cause_category / evolution_target Nonedegraded=False
video_id / task_type canonical 预测取store.upsert 按主键
(question_id, baseline_run_id, diag_fingerprint) 幂等重复调用零副作用
参数:
store: 诊断信号存储端口与诊断落库同一 store
preds: load_canonical_predictions 产出的 canonical 预测行
baseline_run_id: 基线 run 标识信号行主键之一
diag_fingerprint: 诊断口径指纹信号行主键之一
返回:
落库的 T0 行数供日志
"""
count = 0
for pred in preds:
prediction = (pred["prediction"] or "").strip()
is_infra_or_empty = pred["stop_reason"] in _INFRA_STOP_REASONS or not prediction
if pred["correct"] or not is_infra_or_empty:
continue
store.upsert(
DiagnosisSignalRow(
question_id=pred["question_id"],
video_id=pred["video_id"],
baseline_run_id=baseline_run_id,
diag_fingerprint=diag_fingerprint,
task_type=pred["task_type"],
error_type=None,
cause_category=None,
tier="T0",
evolution_target=None,
degraded=False,
infra=True,
session_id=None,
)
)
count += 1
return count
def load_questions_by_id(questions_dir: Path) -> dict[str, GeneratedQuestion]:
"""加载 benchmark 全部题并建 question_id → GeneratedQuestion 映射。
覆盖 wrong_ids run_diagnosis 返回的全部 infra/degraded video_id/task_type
参数:
questions_dir: benchmark 题库目录
返回:
question_id GeneratedQuestion 映射
"""
return {q.question_id: q for q in load_benchmark(questions_dir)}
# ---------------------------------------------------------------------------
# McNemar 功效护栏
# ---------------------------------------------------------------------------
def check_mcnemar_power(pools: Pools, val_wrong_min: int) -> int:
"""校验 validation 池错题数达 McNemar 功效阈,不足即 fail loud。
val_wrong_min 已前置到 build_split 内的切分保证功效不足即从 diag 换入低 T2
错题组补足耗尽 fail-loud本函数作切分冻结后的冗余最终确认val 错题数 <
验证信号不足以支撑可靠比较
参数:
pools: 冻结三池 validation correctness
val_wrong_min: 最少错题数阈0 = 不检查
返回:
validation 池实际错题数供日志
异常:
SystemExit: val_wrong_min > 0 val 错题数 < P5 fail loud不静默放行
"""
val_wrong = sum(1 for q in pools.validation if not pools.correctness[q.question_id])
if val_wrong_min > 0 and val_wrong < val_wrong_min:
raise SystemExit(
f"validation 池错题数 {val_wrong} < val_wrong_min={val_wrong_min}"
"验证信号不足以支撑可靠比较(McNemar 检验功效不够)。"
"请放大 val_ratio / 调整旋钮后重跑,勿静默放行。"
)
return val_wrong
# ---------------------------------------------------------------------------
# 两阶段编排(依赖注入,便于单测)
# ---------------------------------------------------------------------------
async def run_pipeline(
*,
config: VideoSplitConfig,
fingerprint: str,
diagnosis_deps: DiagnosisDeps,
signal_store: DiagnosisSignalStore,
wrong_ids: list[str],
questions: dict[str, GeneratedQuestion],
canonical_preds: list[dict],
harness_db: Path,
questions_dir: Path,
out_dir: Path,
generated_at: str,
force: bool = False,
retry_uncertain: bool = False,
) -> SplitBuildResult:
"""内联三阶段:Phase 0 INFRA T0 补录 → Phase 1 诊断 → Phase 2 冻结切分 → McNemar 护栏。
参数:
config: 科研旋钮快照
fingerprint: 诊断口径指纹已合成作诊断信号主键之一
diagnosis_deps: Phase 1 诊断依赖束真实或假实现
signal_store: 诊断信号存储端口Phase 0/1 Phase 2
wrong_ids: 待诊断的可诊断错题 question_id 列表
questions: question_id GeneratedQuestion 映射
canonical_preds: canonical 预测行Phase 0 从中筛 INFRA / 空预测错题补 T0
harness_db: harness.db 路径Phase 2 canonical 预测
questions_dir: benchmark 题库目录Phase 2 加载题库切池
out_dir: 冻结产物目录pools.json + split_manifest.json
generated_at: 生成时间戳ISO 字符串由调用方传入见模块 C-2 复现锚点约定
force: 覆盖已存在冻结产物开关透传给 build_splitFalse 时已存在即报错
retry_uncertain: 透传给 run_baseline_diagnosis令已落 uncertain 题被重新诊断
返回:
SplitBuildResult冻结三池 + manifest + assignment
"""
# Phase 0: INFRA / 空预测错题补 T0(这些题不进诊断,须单独落库保证 tier 分布/manifest 完整)。
n_t0 = persist_infra_t0_rows(signal_store, canonical_preds, config.baseline_run_id, fingerprint)
logger.info("Phase 0 INFRA T0 补录:落库 {} 行(INFRA / 空预测错题不进诊断)", n_t0)
# Phase 1: 离线诊断(断点续跑幂等:done_question_ids 已完成题跳过)。
logger.info(
"Phase 1 离线诊断:baseline={} 待诊断错题 {}", config.baseline_run_id, len(wrong_ids)
)
await run_baseline_diagnosis(
baseline_run_id=config.baseline_run_id,
diag_fingerprint=fingerprint,
wrong_ids=wrong_ids,
questions=questions,
store=signal_store,
deps=diagnosis_deps,
retry_uncertain=retry_uncertain,
)
# Phase 2: 冻结切分(读诊断信号 → 贪心选择 → 视频组原子切三池 → 冻结 + 六条断言)。
out_dir.mkdir(parents=True, exist_ok=True)
logger.info("Phase 2 冻结切分:out={}", out_dir)
result = build_split(
db_path=harness_db,
baseline_run_id=config.baseline_run_id,
signal_store=signal_store,
diag_fingerprint=fingerprint,
questions_dir=questions_dir,
config=SplitBuildConfig(
n_trainval=config.n_trainval,
floor_k=config.floor_k,
epsilon=config.epsilon,
report_floor=config.report_floor,
select_seed=config.seed,
val_ratio=config.val_ratio,
split_seed=config.seed,
val_wrong_min=config.val_wrong_min,
),
out_path=out_dir / "pools.json",
manifest_path=out_dir / "split_manifest.json",
generated_at=generated_at,
force=force,
)
# McNemar 功效护栏(build_split 契约外的 capstone 层校验)。
val_wrong = check_mcnemar_power(result.pools, config.val_wrong_min)
logger.info(
"切分冻结完成:pools={} manifest={} val错题={}/{}(阈)",
out_dir / "pools.json",
out_dir / "split_manifest.json",
val_wrong,
config.val_wrong_min,
)
return result
# ---------------------------------------------------------------------------
# 真实执行 / dry-run 入口
# ---------------------------------------------------------------------------
def _resolve_paths(args: argparse.Namespace) -> tuple[Path, Path, Path, Path]:
"""解析 harness_db / questions_dir / out_dir / store_dirCLI 覆盖默认工程路径)。"""
harness_db = args.harness_db or _DEFAULT_HARNESS_DB
questions_dir = args.questions_dir or _DEFAULT_QUESTIONS_DIR
out_dir = args.out_dir or _DEFAULT_OUT_DIR
store_dir = args.store_dir or _DEFAULT_STORE_DIR
return harness_db, questions_dir, out_dir, store_dir
def _execute_real(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
"""真实执行两阶段管线:组装真实 deps、读错题、跑诊断 + 冻结切分。"""
harness_db, questions_dir, out_dir, store_dir = _resolve_paths(args)
if not harness_db.exists():
raise SystemExit(f"harness.db 不存在: {harness_db}P5 fail loud")
canonical_preds = load_canonical_predictions(harness_db, config.baseline_run_id)
wrong_ids = select_diagnosable_wrong_ids(canonical_preds)
questions = load_questions_by_id(questions_dir)
video_ids: list[str] = []
for qid in wrong_ids:
q = questions.get(qid)
if q is None:
raise SystemExit(
f"wrong_id {qid!r} 不在 questions_dir 题库中"
"baseline predictions 与题库不匹配,P5 fail loud"
)
video_ids.append(q.video_id)
deps = build_diagnosis_deps(
harness_db=harness_db,
store_dir=store_dir,
video_ids=video_ids,
concurrency=args.concurrency,
expected_model=config.model,
)
# generated_at:默认盖真实 UTC now(溯源用),--generated-at 可显式固定以复现(C-2)。
generated_at = args.generated_at or datetime.datetime.now(datetime.UTC).isoformat()
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
store = SqliteDiagnosisSignalStore(str(harness_db))
try:
asyncio.run(
run_pipeline(
config=config,
fingerprint=fingerprint,
diagnosis_deps=deps,
signal_store=store,
wrong_ids=wrong_ids,
questions=questions,
canonical_preds=canonical_preds,
harness_db=harness_db,
questions_dir=questions_dir,
out_dir=out_dir,
generated_at=generated_at,
force=args.force,
retry_uncertain=args.retry_uncertain,
)
)
finally:
store.close()
class _DryRunLLM:
"""dry-run 假 LLM:被真实调用即报错,保证不真调 LLM。"""
async def complete(self, *args: Any, **kwargs: Any) -> Any:
raise AssertionError("dry-run 不应真调 LLM.complete")
class _DryRunLog:
"""dry-run 假 RunLogpredictions/traces 均返回空,诊断不真正执行。"""
async def get_predictions(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
return []
async def get_traces(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
return []
def _execute_dry_run(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
"""dry-run:用假 deps 跑通 Phase 1 wiring(空错题 → 诊断早返回),打印步骤 + 指纹。
Phase 2 build_split 需真实诊断信号方能冻结dry-run 不真实冻结仅打印其计划
Phase 1 用空 wrong_ids run_baseline_diagnosis 早返回路径验证装配可调用而不触 LLM
"""
harness_db, questions_dir, out_dir, _store_dir = _resolve_paths(args)
logger.info("=== dry-run:校验两阶段装配(不真调 LLM / 不冻结产物)===")
logger.info(
"诊断口径指纹 diag_fingerprint={} (prompt={} model={})",
fingerprint,
config.prompt_version,
config.model,
)
logger.info(
"解析路径:harness_db={} questions_dir={} out_dir={}", harness_db, questions_dir, out_dir
)
logger.info(
"旋钮:n_trainval={} epsilon={} report_floor={} val_ratio={} seed={} "
"val_wrong_min={} floor_k={}",
config.n_trainval,
config.epsilon,
config.report_floor,
config.val_ratio,
config.seed,
config.val_wrong_min,
config.floor_k,
)
fake_deps = DiagnosisDeps(
run_log=_DryRunLog(),
llm=_DryRunLLM(),
skill_store=object(),
prompts=object(),
tree_data={},
concurrency=args.concurrency,
)
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
dry_db = out_dir / "_dry_run_signals.db"
dry_db.parent.mkdir(parents=True, exist_ok=True)
store = SqliteDiagnosisSignalStore(str(dry_db))
try:
# Phase 0 装配:用一条假 INFRA 空预测走通 persist_infra_t0_rows(不触 LLM)。
fake_infra_preds = [
{
"question_id": "_dry_infra",
"video_id": "_dry_v",
"task_type": "Counting Problem",
"prediction": "",
"answer": "A",
"stop_reason": "error",
"correct": False,
}
]
n_t0 = persist_infra_t0_rows(store, fake_infra_preds, config.baseline_run_id, fingerprint)
logger.info("Phase 0 装配 OKpersist_infra_t0_rows 落 {} 行 INFRA T0(假数据)", n_t0)
logger.info("Phase 1 装配 OKrun_baseline_diagnosis 以空错题走早返回路径(不触 LLM)")
asyncio.run(
run_baseline_diagnosis(
baseline_run_id=config.baseline_run_id,
diag_fingerprint=fingerprint,
wrong_ids=[],
questions={},
store=store,
deps=fake_deps,
)
)
finally:
store.close()
dry_db.unlink(missing_ok=True)
logger.info(
"Phase 2 装配 OK:真实执行将调 build_split 冻结 pools.json + manifestdry-run 跳过)"
)
logger.info("=== dry-run 通过:两阶段装配可调用,指纹已算出 ===")
def build_arg_parser() -> argparse.ArgumentParser:
"""构建 CLI 参数解析器。"""
parser = argparse.ArgumentParser(description="结果驱动视频级切分两阶段 CLI(诊断 → 冻结切分)")
parser.add_argument("--config", type=Path, default=Path("config/video_split.yaml"))
parser.add_argument("--dry-run", action="store_true", dest="dry_run")
parser.add_argument("--gpu", type=str, default=None, help="可选:设置 CUDA_VISIBLE_DEVICES")
parser.add_argument("--concurrency", type=int, default=8, help="诊断并发上限")
parser.add_argument("--harness-db", type=Path, default=None, dest="harness_db")
parser.add_argument("--questions-dir", type=Path, default=None, dest="questions_dir")
parser.add_argument("--out-dir", type=Path, default=None, dest="out_dir")
parser.add_argument("--store-dir", type=Path, default=None, dest="store_dir")
parser.add_argument(
"--generated-at",
type=str,
default=None,
dest="generated_at",
help=(
"manifest generated_at 时间戳(ISO 字符串);默认盖真实 UTC now(溯源元数据)。"
"复现锚点是 pools.json 内容 + seed + fingerprintgenerated_at 可显式传入以"
"对 manifest 做字节级复现比对。"
),
)
parser.add_argument(
"--force",
action="store_true",
help="覆盖已存在的冻结 pools.json/manifest(旧产物备份为 .bak.*",
)
parser.add_argument(
"--retry-uncertain",
action="store_true",
dest="retry_uncertain",
help="把已落 tier='uncertain'(信号不可信降级)的题重新诊断,而非当作已完成跳过",
)
return parser
def main(argv: list[str] | None = None) -> None:
"""CLI 入口:解析参数 → 载配置 → 算指纹 → dry-run 或真实两阶段执行。
参数:
argv: 可选参数列表默认 sys.argv[1:]便于测试注入
"""
from dotenv import load_dotenv
load_dotenv()
args = build_arg_parser().parse_args(argv)
if args.gpu is not None:
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu
logger.info("CUDA_VISIBLE_DEVICES={}", args.gpu)
config = load_config(args.config)
fingerprint = diag_fingerprint(config.prompt_version, config.model, git_short_sha())
if args.dry_run:
_execute_dry_run(config, fingerprint, args)
return
_execute_real(config, fingerprint, args)
if __name__ == "__main__":
main()
+498
View File
@@ -0,0 +1,498 @@
"""Workspace 生命周期管理 + manifest 读写 + Protocol 实现。
Workspace 是一次实验的独立工作区通过 manifest.json 引用 Store 中的
特定版本资源并记录实验过程Skills/Prompts 权重拷入 workspace 本地
训练产物只进 workspace 不污染 Store
VersionedSkillStore / VersionedPromptStore 实现 core/evolution/protocols.py
中定义的只读端口 core/ 层以 Protocol 方式读取技能和提示词
"""
from __future__ import annotations
import json
import os
import shutil
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from loguru import logger
from app.harness.store import read_seed
if TYPE_CHECKING:
from pathlib import Path
@dataclass(frozen=True)
class ResolvedPaths:
"""manifest 解析后的绝对路径集合。
属性:
store_dir: Store 根目录绝对路径
videos_dir: 视频数据目录
questions_dir: 当前引用的题目目录
skills_dir: 当前引用的 Skill 版本目录workspace
prompts_dir: 当前引用的 Prompt 版本目录workspace
workspace_dir: Workspace 根目录
db_path: harness.db 路径
analyses_dir: 分析报告目录
runs_dir: 运行临时状态目录
"""
store_dir: Path
videos_dir: Path
questions_dir: Path
skills_dir: Path
prompts_dir: Path
workspace_dir: Path
db_path: Path
analyses_dir: Path
runs_dir: Path
# ---------------------------------------------------------------------------
# 内部工具
# ---------------------------------------------------------------------------
_MANIFEST_CURRENT_KEYS = {"videos", "questions", "skills", "prompts"}
def _now_iso() -> str:
"""返回当前 UTC 时间的 ISO 格式字符串。"""
return datetime.now(UTC).isoformat()
def _atomic_write_json(path: Path, data: dict) -> None:
"""原子写 JSONtmp + os.replace(对齐 checkpoint.py 范式,防半截损坏)。
先写同目录临时文件 os.replace 原子替换目标替换阶段崩溃不会留下半截
JSON原文件保持完好
参数:
path: 目标 JSON 文件路径
data: 待序列化的字典
"""
tmp = path.with_name(path.name + ".tmp")
tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(tmp, path)
# ---------------------------------------------------------------------------
# Workspace 核心函数
# ---------------------------------------------------------------------------
def _scaffold_workspace(
workspace_dir: Path,
store_dir: Path,
questions: str,
skills_version: str,
prompts_version: str,
) -> None:
"""写 manifest + 建 analyses/runs 目录(不拷权重;权重由调用方按来源拷入)。
参数:
workspace_dir: 目标 workspace由调用方保证不存在
store_dir: Store 根目录
questions: 题目相对路径 ``'benchmarks/Video-MME'``
skills_version: manifest.current.skills 初始版本号
prompts_version: manifest.current.prompts 初始版本号
关键实现:
不依赖任何外部资源源store 中的 skills/prompts 是否存在不在此校验
因此可被 init_workspace 与种子初始化复用store 引用以相对路径写入 manifest
"""
workspace_dir.mkdir(parents=True)
(workspace_dir / "analyses").mkdir()
(workspace_dir / "runs").mkdir()
store_abs = store_dir.resolve()
store_rel = os.path.relpath(store_abs, workspace_dir.resolve())
manifest = {
"name": workspace_dir.name,
"created_at": _now_iso(),
"store": store_rel,
"current": {
"videos": "videos",
"questions": f"questions/{questions}",
"skills": f"skills/{skills_version}",
"prompts": f"prompts/{prompts_version}",
},
"history": [],
}
_atomic_write_json(workspace_dir / "manifest.json", manifest)
def init_workspace(
workspace_dir: Path,
store_dir: Path,
questions: str,
skills_version: str,
prompts_version: str,
) -> None:
"""创建 Workspace 目录并写入初始 manifest.json,拷贝种子权重。
参数:
workspace_dir: Workspace 目标路径不得已存在
store_dir: Store 根目录
questions: 题目在 questions/ 下的相对路径 ``"benchmarks/Video-MME"``
skills_version: Skills 版本号 ``"v1"``
prompts_version: Prompts 版本号 ``"v1"``
异常:
FileExistsError: Workspace 目录已存在
FileNotFoundError: 引用的资源在 Store 中不存在
"""
if workspace_dir.exists():
raise FileExistsError(f"Workspace 已存在: {workspace_dir}")
store_abs = store_dir.resolve()
refs = {
"skills": f"skills/{skills_version}",
"prompts": f"prompts/{prompts_version}",
"questions": f"questions/{questions}",
}
for label, rel in refs.items():
full = store_abs / rel
if not full.is_dir():
raise FileNotFoundError(f"Store 中不存在 {label}: {full}")
_scaffold_workspace(workspace_dir, store_dir, questions, skills_version, prompts_version)
# 拷种子权重进 workspacev2+ 训练产物只进 workspace,不污染 store
shutil.copytree(store_abs / refs["skills"], workspace_dir / refs["skills"])
shutil.copytree(store_abs / refs["prompts"], workspace_dir / refs["prompts"])
logger.info("Workspace 初始化完成: {}", workspace_dir)
def init_workspace_from_seed(
workspace_dir: Path,
store_dir: Path,
seed_name: str,
questions: str,
) -> str:
"""从种子全新建 workspace:拷权重 -> v1、baseline.db -> harness.db、读 baseline_run_id。
参数:
workspace_dir: 目标 workspace不得已存在
store_dir: Store 根目录
seed_name: 种子名store/seeds
questions: 题目相对路径 ``'benchmarks/Video-MME'``
返回:
baseline_run_id build_pools 使用
异常:
FileExistsError: workspace 已存在
FileNotFoundError: 种子不存在 read_seed 抛出 questions ref 目录不存在
关键实现:
破坏性/创建操作前先校验 questions ref 存在fresh 路径在 runner 侧已先
归档旧 ws若到 build_pools 才发现 questions 缺失则旧 ws 已被毁
故在此尽早报错fail-fast让新 ws 在创建前失败
"""
if workspace_dir.exists():
raise FileExistsError(f"Workspace 已存在: {workspace_dir}")
# 校验种子存在 + 取 baseline_run_id
meta = read_seed(store_dir, seed_name)
# fail-fast:校验 questions ref 存在
questions_ref = store_dir / "questions" / questions
if not questions_ref.is_dir():
raise FileNotFoundError(f"questions ref 目录不存在: {questions_ref}")
seed_dir = store_dir / "seeds" / seed_name
_scaffold_workspace(workspace_dir, store_dir, questions, "v1", "v1")
shutil.copytree(seed_dir / "skills", workspace_dir / "skills" / "v1")
shutil.copytree(seed_dir / "prompts", workspace_dir / "prompts" / "v1")
shutil.copy2(seed_dir / "baseline.db", workspace_dir / "harness.db")
seed_pools = seed_dir / "pools.json"
if seed_pools.exists():
shutil.copy2(seed_pools, workspace_dir / "pools.json")
seed_manifest = seed_dir / "split_manifest.json"
if seed_manifest.exists():
shutil.copy2(seed_manifest, workspace_dir / "split_manifest.json")
logger.info("Workspace 从种子 '{}' 初始化完成: {}", seed_name, workspace_dir)
return meta["baseline_run_id"]
def load_manifest(workspace_dir: Path) -> dict:
"""读取并返回 workspace 的 manifest.json。
参数:
workspace_dir: Workspace 根目录
返回:
manifest 字典
异常:
FileNotFoundError: manifest.json 不存在
"""
manifest_path = workspace_dir / "manifest.json"
if not manifest_path.exists():
raise FileNotFoundError(f"manifest.json 不存在: {manifest_path}")
return json.loads(manifest_path.read_text())
def resolve_paths(workspace_dir: Path) -> ResolvedPaths:
"""读取 manifest,解析 current 中所有资源的绝对路径。
skills_dir/prompts_dir 解析到 workspace store
videos_dir/questions_dir 解析到 store
参数:
workspace_dir: Workspace 根目录
返回:
ResolvedPaths 实例包含所有资源的绝对路径
"""
manifest = load_manifest(workspace_dir)
ws_abs = workspace_dir.resolve()
store_abs = (ws_abs / manifest["store"]).resolve()
current = manifest["current"]
return ResolvedPaths(
store_dir=store_abs,
videos_dir=store_abs / current["videos"],
questions_dir=store_abs / current["questions"],
skills_dir=ws_abs / current["skills"],
prompts_dir=ws_abs / current["prompts"],
workspace_dir=ws_abs,
db_path=ws_abs / "harness.db",
analyses_dir=ws_abs / "analyses",
runs_dir=ws_abs / "runs",
)
def list_video_ids(workspace_dir: Path) -> list[str]:
"""列出 workspace 引用的所有视频 ID(含 tree.json 的子目录名)。
参数:
workspace_dir: Workspace 根目录
返回:
排序后的视频 ID 列表
"""
paths = resolve_paths(workspace_dir)
video_ids = []
for entry in paths.videos_dir.iterdir():
if entry.is_dir() and (entry / "tree.json").exists():
video_ids.append(entry.name)
return sorted(video_ids)
def update_manifest(workspace_dir: Path, **version_updates: str) -> None:
"""更新 manifest 的 current 字段。
参数:
workspace_dir: Workspace 根目录
**version_updates: 要更新的字段及其新值 ``skills="skills/v2"``
异常:
KeyError: 更新的字段不在 current 允许的 key 白名单中
"""
invalid = set(version_updates) - _MANIFEST_CURRENT_KEYS
if invalid:
raise KeyError(f"无效的 manifest current 字段: {invalid}")
manifest = load_manifest(workspace_dir)
manifest["current"].update(version_updates)
_atomic_write_json(workspace_dir / "manifest.json", manifest)
def record_run(workspace_dir: Path, run_id: str) -> Path:
"""将 current 版本快照追加到 manifest history,创建 run 目录和 per-video wiki 目录。
幂等 run_id 不重复追加 history长跑中断后重启 / held-out 复用 run_id
参数:
workspace_dir: Workspace 根目录
run_id: 本次运行的唯一标识 ``"run_001"``
返回:
创建的 run 目录路径
"""
manifest = load_manifest(workspace_dir)
current = manifest["current"]
# 幂等:同 run_id 不重复追加 history
if not any(h["run_id"] == run_id for h in manifest["history"]):
manifest["history"].append(
{
"run_id": run_id,
"started_at": _now_iso(),
"skills": current["skills"],
"prompts": current["prompts"],
"questions": current["questions"],
}
)
_atomic_write_json(workspace_dir / "manifest.json", manifest)
run_dir = workspace_dir / "runs" / run_id
# exist_ok:同 run_id 重跑时 run 目录已存在不应崩溃
run_dir.mkdir(parents=True, exist_ok=True)
for video_id in list_video_ids(workspace_dir):
(run_dir / video_id / "wiki").mkdir(parents=True, exist_ok=True)
logger.debug("Run 已记录: {}", run_id)
return run_dir
def read_best(workspace_dir: Path) -> dict | None:
"""读取 manifest 的 best 指针,未设置时返回 None。
参数:
workspace_dir: Workspace 根目录
返回:
best 字典skills/prompts/val_acc/run_id/epoch未设置时 None
"""
return load_manifest(workspace_dir).get("best")
def update_best(
workspace_dir: Path,
skills: str,
prompts: str,
val_acc: float,
run_id: str,
epoch: int,
) -> None:
"""写入 manifest 的 best 指针(历史最优版本快照,与 current 平级)。
best 独立于 current更新 best 不影响 current
参数:
workspace_dir: Workspace 根目录
skills: 最优 skills 版本完整 ref ``'skills/v2'``
prompts: 最优 prompts 版本完整 ref ``'prompts/v2'``
val_acc: 该版本验证集准确率
run_id: 该版本验证 run_id
epoch: 达成该最优的轮次
"""
manifest = load_manifest(workspace_dir)
manifest["best"] = {
"skills": skills,
"prompts": prompts,
"val_acc": val_acc,
"run_id": run_id,
"epoch": epoch,
}
_atomic_write_json(workspace_dir / "manifest.json", manifest)
logger.info("Best 已更新: val_acc={}, run={}, epoch={}", val_acc, run_id, epoch)
def archive_workspace(workspace_dir: Path) -> Path:
"""把 workspace 整体移动到同级 .archive/<name>-<ts>,返回归档路径。
参数:
workspace_dir: 要归档的 Workspace 根目录
返回:
归档后的目标路径
异常:
FileNotFoundError: workspace 不存在
"""
if not workspace_dir.exists():
raise FileNotFoundError(f"workspace 不存在: {workspace_dir}")
archive_root = workspace_dir.parent / ".archive"
archive_root.mkdir(exist_ok=True)
ts = datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
target = archive_root / f"{workspace_dir.name}-{ts}"
shutil.move(str(workspace_dir), str(target))
logger.info("Workspace 已归档: {} -> {}", workspace_dir, target)
return target
# ---------------------------------------------------------------------------
# Protocol 实现:VersionedSkillStore / VersionedPromptStore
# ---------------------------------------------------------------------------
class VersionedSkillStore:
"""版本化技能读取端口实现。
满足 ``core/evolution/protocols.py::SkillStore`` Protocol
从指定的 skills 版本目录读取 ``.md`` 文件
参数:
skills_dir: skills 版本目录绝对路径 ``workspace/skills/v1``
"""
def __init__(self, skills_dir: Path) -> None:
if not skills_dir.is_dir():
raise FileNotFoundError(f"Skills 目录不存在: {skills_dir}")
self._dir = skills_dir
def read_skill(self, filename: str) -> str:
"""读取指定 skill 文件的全文内容。
参数:
filename: skill 文件名 ``'temporal-reasoning.md'``
返回:
文件全文内容
异常:
FileNotFoundError: 文件不存在
"""
path = self._dir / filename
if not path.exists():
raise FileNotFoundError(f"Skill 文件不存在: {path}")
return path.read_text()
def list_skill_files(self) -> list[str]:
"""列出当前版本所有 skill 文件名。
返回:
文件名列表排序
"""
return sorted(entry.name for entry in self._dir.iterdir() if entry.is_file())
class VersionedPromptStore:
"""版本化提示词读取端口实现。
满足 ``core/evolution/protocols.py::PromptStore`` Protocol
从指定的 prompts 版本目录读取 ``.md`` 文件
参数:
prompts_dir: prompts 版本目录绝对路径 ``workspace/prompts/v1``
"""
def __init__(self, prompts_dir: Path) -> None:
if not prompts_dir.is_dir():
raise FileNotFoundError(f"Prompts 目录不存在: {prompts_dir}")
self._dir = prompts_dir
def read_prompt(self, filename: str) -> str:
"""读取指定 prompt 文件的全文内容。
参数:
filename: prompt 文件名 ``'system.md'``
返回:
文件全文内容
异常:
FileNotFoundError: 文件不存在
"""
path = self._dir / filename
if not path.exists():
raise FileNotFoundError(f"Prompt 文件不存在: {path}")
return path.read_text()
def list_prompt_files(self) -> list[str]:
"""列出当前版本所有 prompt 文件名。
返回:
文件名列表排序
"""
return sorted(entry.name for entry in self._dir.iterdir() if entry.is_file())
+172
View File
@@ -1 +1,173 @@
"""应用层 Protocol 端口定义。"""
from __future__ import annotations
from pathlib import Path # noqa: TC003 — runtime_checkable Protocol 需运行时可见
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
if TYPE_CHECKING:
import numpy as np
from app.harness.pools import Pools
from app.tree.index import TreeIndex
from core.types import GeneratedQuestion, PoolConfig
@runtime_checkable
class EmbeddingProvider(Protocol):
"""文本嵌入端口。
属性:
dim: 嵌入维度 D
"""
@property
def dim(self) -> int: ...
def embed(self, texts: str | list[str]) -> np.ndarray:
"""文本 → 嵌入向量(L2 归一化)。
参数:
texts: 单条文本或文本列表
返回:
[N, D] ndarray每行 L2 范数为 1.0
"""
...
@runtime_checkable
class QuestionGenerator(Protocol):
"""LLM 驱动的题目生成端口(预留接口)。
参数:
video_id: 视频标识
task_type: 题型
tree: 视频树索引提供锚节点上下文
exemplars: 风格示例题目列表
返回:
生成的单条题目
"""
async def generate(
self,
video_id: str,
task_type: str,
tree: TreeIndex,
*,
exemplars: list[GeneratedQuestion],
) -> GeneratedQuestion: ...
@runtime_checkable
class OCRProvider(Protocol):
"""帧文字转录端口。
实现方负责将帧图像发送给 OCR 服务并返回拼接后的文本
单帧失败应降级跳过不得抛出异常中断整体流程
参数:
frame_paths: 帧文件路径列表
返回:
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本无有效结果时空串
"""
async def transcribe_frames(self, frame_paths: list[Path]) -> str: ...
@runtime_checkable
class ToolDispatchFn(Protocol):
"""工具调度函数签名。
参数:
tool_name: 工具名称
args: 工具参数字典
context: 上下文字典包含 session_id
返回:
工具执行结果文本
"""
async def __call__(
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
) -> str: ...
@runtime_checkable
class ToolDispatchFactory(Protocol):
"""per-version 工具调度工厂。
通过注入 skills_dir 生产对应版本的 ToolDispatchFn
参数:
skills_dir: skill 文件目录None 则不加载 skill
返回:
ToolDispatchFn 实例
"""
def __call__(self, *, skills_dir: Path | None = None) -> ToolDispatchFn: ...
@runtime_checkable
class PromptBuilderFn(Protocol):
"""Prompt 构建函数签名。
参数:
qa: 待构建 prompt 的题目
返回:
(system_prompt, user_prompt) 二元组
"""
def __call__(self, qa: GeneratedQuestion) -> tuple[str, str]: ...
@runtime_checkable
class PromptBuilderFactory(Protocol):
"""per-version prompt 构建工厂。
通过注入 skills_dir prompts_dir 生产对应版本的 PromptBuilderFn
参数:
skills_dir: skill 文件目录None 则不加载 skill
prompts_dir: prompt 文件目录None 则使用默认目录
返回:
PromptBuilderFn 实例
"""
def __call__(
self,
*,
skills_dir: Path | None = None,
prompts_dir: Path | None = None,
) -> PromptBuilderFn: ...
@runtime_checkable
class PoolStrategy(Protocol):
"""池构建策略端口。
应用层端口 core 因为返回类型 Pools 定义在 app/harness/pools.py
两个具体策略GlobalPoolStrategy / PerCategoryPoolStrategy实现此接口
"""
def build(
self,
questions: list[GeneratedQuestion],
correctness: dict[str, bool],
config: PoolConfig,
*,
db_path: Path | None = None,
) -> Pools: ...
def build_incremental(
self,
new_task_types: list[str],
questions: list[GeneratedQuestion],
correctness: dict[str, bool],
config: PoolConfig,
) -> dict[str, dict[str, list[str]]]: ...
+33
View File
@@ -0,0 +1,33 @@
"""出题模块 — benchmark 加载、分层采样、赛题合成与 v2 出题管线。"""
from app.question_gen.families import ALL_FAMILIES, QuestionFamilySpec
from app.question_gen.gates import GateReport, run_gates
from app.question_gen.generator_v2 import CandidateQuestion, generate_one_v2
from app.question_gen.loader import load_benchmark, stratified_sample
from app.question_gen.pipeline_v2 import PipelineConfig, PipelineResult, run_pipeline_v2
from app.question_gen.synthesizer import (
TASK_TYPE_LEVEL_MAP,
AnchorContext,
generate_one,
sample_anchor,
)
__all__ = [
# v1 接口
"load_benchmark",
"stratified_sample",
"TASK_TYPE_LEVEL_MAP",
"AnchorContext",
"generate_one",
"sample_anchor",
# v2 接口
"run_pipeline_v2",
"PipelineConfig",
"PipelineResult",
"QuestionFamilySpec",
"ALL_FAMILIES",
"CandidateQuestion",
"generate_one_v2",
"GateReport",
"run_gates",
]
+59
View File
@@ -0,0 +1,59 @@
"""Phase B 对抗过滤层配置 — filter 层配置(非 strategy 属性)。
设计: research-wiki/designs/2026-07-14-adversarial-question-gen-phaseB-design.md §8
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
import yaml
if TYPE_CHECKING:
from pathlib import Path
@dataclass(frozen=True)
class AdversarialFilterConfig:
"""后置对抗过滤配置。
属性:
filter_task_types: 被过滤的题型仅这些走 agent 默认仅 AR
adversarial_max_rounds: 补生成迭代上限
adversarial_agent_max_steps: agent 试答步数上限
difficulty_warn_threshold: 批次 agent 正确率告警阈值
"""
filter_task_types: tuple[str, ...] = ("Action Recognition",)
adversarial_max_rounds: int = 5
adversarial_agent_max_steps: int = 40
difficulty_warn_threshold: float = 0.85
def load_adversarial_config(config_path: Path) -> AdversarialFilterConfig:
"""从 YAML 的 adversarial_filter 区段加载配置,缺段/缺键用默认值。
参数:
config_path: YAML 配置文件路径
返回:
AdversarialFilterConfig 实例
"""
with open(config_path, encoding="utf-8") as f:
raw = yaml.safe_load(f) or {}
section = raw.get("adversarial_filter", {}) or {}
default = AdversarialFilterConfig()
types = section.get("filter_task_types")
return AdversarialFilterConfig(
filter_task_types=tuple(types) if types else default.filter_task_types,
adversarial_max_rounds=int(
section.get("adversarial_max_rounds", default.adversarial_max_rounds)
),
adversarial_agent_max_steps=int(
section.get("adversarial_agent_max_steps", default.adversarial_agent_max_steps)
),
difficulty_warn_threshold=float(
section.get("difficulty_warn_threshold", default.difficulty_warn_threshold)
),
)
File diff suppressed because it is too large Load Diff
+291
View File
@@ -0,0 +1,291 @@
"""Grounded 干扰项 selector — 候选池 + VLM 视觉打分 + 区间选择(仅 AR 路径)。
把干扰项从"VLM 主观写得像"下沉到机制层VLM 生成 N 个候选干扰项再对
候选 + 正解逐一打"视觉可信度" [正解分-δ_high, 正解分-δ_low] 区间
3 grounded near-miss从机制上消灭 Easy-Options Bias
设计: research-wiki/designs/2026-07-14-grounded-question-gen-phaseA-design.md §3
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
from json_repair import repair_json
from loguru import logger
if TYPE_CHECKING:
from app.question_gen.sampler_v2 import MaterialContext
from core.protocols import VLMProvider
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
@dataclass(frozen=True)
class SelectorConfig:
"""selector 科研参数。
属性:
candidate_pool_size: 首轮候选干扰项数 N
delta_low: 干扰项视觉分与正解的最小差上界太近=真歧义
delta_high: 干扰项视觉分与正解的最大差下界太低=负空间
max_delta_relax: δ_high 放宽次数上限退火
delta_relax_step: 每次放宽 δ_high 的增量
"""
candidate_pool_size: int
delta_low: float
delta_high: float
max_delta_relax: int = 2
delta_relax_step: float = 0.1
@dataclass(frozen=True)
class SelectorOutcome:
"""selector 产出。observation 始终存在(含 hard-fail),供 run_store 落库。
属性:
observation: 打分观测 dictcorrect_score/chosen/pool_size/anneal_rounds/hard_fail
options: 重组四选项A=正解hard-fail 时为 None
answer: 正解字母 "A"hard-fail 时为 None
"""
observation: dict[str, object]
options: tuple[str, ...] | None = None
answer: str | None = None
@property
def hard_fail(self) -> bool:
"""是否硬失败(凑不齐 3 个 grounded 干扰项)。"""
return self.options is None
def _select_in_interval(
correct_score: float,
candidates: list[str],
candidate_scores: list[float],
delta_low: float,
delta_high: float,
) -> list[str] | None:
"""从候选中选 3 个视觉分落 [correct-δ_high, correct-δ_low] 区间的干扰项。
落区间者按分数降序取前 3分数越高越接近正解=越难不足 3 个返回 None
参数:
correct_score: 正解视觉可信度分
candidates: 候选干扰项文本列表
candidate_scores: candidates 对齐的视觉分列表
delta_low: 最小差上界 = correct - delta_low
delta_high: 最大差下界 = correct - delta_high
返回:
选中的 3 个候选文本降序 None不足 3
"""
upper = correct_score - delta_low
lower = correct_score - delta_high
eligible = [
(c, s) for c, s in zip(candidates, candidate_scores, strict=True) if lower <= s <= upper
]
if len(eligible) < 3:
return None
eligible.sort(key=lambda cs: cs[1], reverse=True)
return [c for c, _ in eligible[:3]]
def _load_prompt(name: str) -> str:
path = _PROMPTS_DIR / name
if not path.exists():
msg = f"Prompt 模板不存在: {path}"
raise FileNotFoundError(msg)
return path.read_text(encoding="utf-8")
def _material_context_block(question: str, correct_text: str, material: MaterialContext) -> str:
parts = [f"## Question\n{question}", f"## Correct Answer\n{correct_text}"]
if material.subtitle_sentences:
parts.append("## Subtitles")
parts.extend(f" - {s}" for s in material.subtitle_sentences)
if getattr(material, "cross_l2_texts", None):
parts.append("## Cross-Segment Context")
parts.extend(f" - {t}" for t in material.cross_l2_texts)
return "\n".join(parts)
def _parse_json_object(raw: str) -> dict:
content = raw.strip()
if "```" in content:
for part in content.split("```"):
stripped = part.strip()
if stripped.startswith("json"):
stripped = stripped[4:].strip()
if stripped.startswith("{"):
content = stripped
break
data = json.loads(repair_json(content, return_objects=False))
if not isinstance(data, dict):
msg = f"selector 响应顶层非 JSON 对象: {type(data).__name__}"
raise ValueError(msg)
return data
async def _generate_pool(
vlm: VLMProvider,
question: str,
correct_text: str,
material: MaterialContext,
n: int,
*,
session_id: str,
) -> list[str]:
"""VLM 生成 n 个候选干扰项文本。"""
system = _load_prompt("ar_distractor_pool.md")
user = (
_material_context_block(question, correct_text, material)
+ f"\n## N\nGenerate exactly {n} distractors."
)
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
resp = await vlm.chat_with_images(messages, list(material.frame_paths), session_id=session_id)
# graceful 降级:解析爆炸时返回空池,让退火/hard_fail 接管(区别于契约违反的打分校验)
try:
data = _parse_json_object(resp.content)
except (ValueError, json.JSONDecodeError) as exc:
logger.warning("grounded selector 候选池响应解析失败,降级为空池: {}", exc)
return []
raw = data.get("distractors", [])
if not isinstance(raw, list):
return []
# 防御:去空、去重、剔除与正解字面相同者
seen: set[str] = set()
out: list[str] = []
for item in raw:
text = str(item).strip()
if not text or text == correct_text.strip() or text in seen:
continue
seen.add(text)
out.append(text)
return out
async def _score_options(
vlm: VLMProvider,
question: str,
options: list[str],
material: MaterialContext,
*,
session_id: str,
) -> list[float]:
"""VLM 对 options(首个为正解)逐一打视觉可信度分 [0,1],返回对齐分数列表。"""
system = _load_prompt("ar_distractor_score.md")
numbered = "\n".join(f"{i}. {opt}" for i, opt in enumerate(options, 1))
user = f"## Question\n{question}\n\n## Candidates\n{numbered}"
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
resp = await vlm.chat_with_images(messages, list(material.frame_paths), session_id=session_id)
data = _parse_json_object(resp.content)
scores_raw = data.get("scores", [])
if not isinstance(scores_raw, list) or len(scores_raw) != len(options):
msg = f"打分数量({len(scores_raw) if isinstance(scores_raw, list) else 'NA'}) != 选项数({len(options)})"
raise ValueError(msg)
try:
return [max(0.0, min(1.0, float(s))) for s in scores_raw]
except (TypeError, ValueError) as e:
msg = f"打分含非数值元素: {e}"
raise ValueError(msg) from e
async def build_grounded_options(
vlm: VLMProvider,
question: str,
correct_text: str,
material: MaterialContext,
config: SelectorConfig,
*,
session_id: str,
) -> SelectorOutcome:
"""生成候选池 → 视觉打分 → 区间选 3 干扰项 → 重组四选项。
退火凑不齐 3 个时按序 最多追加 N 个新候选去重后实际增量可能更少再打分 逐步放宽
δ_high纯重选不再调 VLM 仍不足则 hard_fail调用方走重出
参数:
vlm: VLM 端口
question: 题干
correct_text: 正解文本无字母前缀
material: 采样素材提供 frame_paths / subtitles
config: selector 科研参数
session_id: 遥测会话 ID
返回:
SelectorOutcome成功时 options=A 正解+3 grounded 干扰项hard_fail
options=None observation 始终存在供落库
"""
candidates = await _generate_pool(
vlm, question, correct_text, material, config.candidate_pool_size, session_id=session_id
)
# options[0] 恒为正解
scored = await _score_options(
vlm, question, [correct_text, *candidates], material, session_id=session_id
)
correct_score, cand_scores = scored[0], scored[1:]
anneal_rounds = 0
chosen = _select_in_interval(
correct_score, candidates, cand_scores, config.delta_low, config.delta_high
)
# 退火 1: 最多追加 N 个新候选(去重后实际增量可能更少),仅对新增候选打分,正解分保持首轮值
if chosen is None:
anneal_rounds += 1
more = await _generate_pool(
vlm, question, correct_text, material, config.candidate_pool_size, session_id=session_id
)
more = [m for m in more if m not in candidates]
if more:
more_scores = await _score_options(
vlm, question, [correct_text, *more], material, session_id=session_id
)
candidates = candidates + more
cand_scores = cand_scores + more_scores[1:]
chosen = _select_in_interval(
correct_score, candidates, cand_scores, config.delta_low, config.delta_high
)
# 退火 2: 放宽 δ_high(下界下移,纳入更低分候选),δ_low 不动
relax = 0
delta_high = config.delta_high
while chosen is None and relax < config.max_delta_relax:
relax += 1
anneal_rounds += 1
delta_high = delta_high + config.delta_relax_step
chosen = _select_in_interval(
correct_score, candidates, cand_scores, config.delta_low, delta_high
)
hard_fail = chosen is None
observation: dict[str, object] = {
"correct_score": correct_score,
"chosen": [cand_scores[candidates.index(c)] for c in (chosen or [])],
"pool_size": len(candidates),
"anneal_rounds": anneal_rounds,
"delta_high_final": delta_high,
"hard_fail": hard_fail,
}
if hard_fail:
logger.warning(
"grounded selector 硬失败: correct={:.3f}, pool={}, anneal={}",
correct_score,
len(candidates),
anneal_rounds,
)
# observation 仍返回,供 pipeline 落 selector_scores(设计 §3.3 退化观测)
return SelectorOutcome(observation=observation)
options = (
f"A. {correct_text}",
f"B. {chosen[0]}",
f"C. {chosen[1]}",
f"D. {chosen[2]}",
)
return SelectorOutcome(observation=observation, options=options, answer="A")
+252
View File
@@ -0,0 +1,252 @@
"""题族规格声明 — 定义 5 大问题家族及其采样、泄漏检测、提示模板约束。
每个 QuestionFamilySpec 对应一种失败机制skill_target M1M5
get_family_for_slot 在出题时按权重分配
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import random
@dataclass(frozen=True)
class LeakTestProfile:
"""泄漏测试配置 — 定义快捷答题捷径类型与通过阈值。
Attributes:
shortcut_type: 捷径类型标识
probe_template: store/prompts/question_gen/ 下的探测模板文件名
pass_threshold: 通过阈值01低于此值视为存在泄漏
"""
shortcut_type: str
probe_template: str
pass_threshold: float
@dataclass(frozen=True)
class SamplingConstraint:
"""采样约束 — 对树节点的最低要求。
Attributes:
min_subtitles: 最少字幕段数
min_l3_nodes: 最少 L3 节点数
require_frames: 是否要求帧图像可用
cross_l2_span: 是否要求跨 L2 段采样
"""
min_subtitles: int
min_l3_nodes: int
require_frames: bool
cross_l2_span: bool
@dataclass(frozen=True)
class QuestionFamilySpec:
"""问题家族规格 — 一个家族的完整声明。
Attributes:
name: 家族标识名 "RETRIEVAL"
skill_target: 目标失败机制编号M1M5
sampling: 采样约束
legal_task_types: 该家族合法的任务类型集合frozenset
leak_profile: 泄漏测试配置
prompt_template: 出题 prompt 模板文件名
"""
name: str
skill_target: str
sampling: SamplingConstraint
legal_task_types: frozenset[str]
leak_profile: LeakTestProfile
prompt_template: str
# ---------------------------------------------------------------------------
# 5 大家族实例
# ---------------------------------------------------------------------------
RETRIEVAL_FAMILY = QuestionFamilySpec(
name="RETRIEVAL",
skill_target="M1",
sampling=SamplingConstraint(
min_subtitles=2,
min_l3_nodes=3,
require_frames=False,
cross_l2_span=False,
),
legal_task_types=frozenset(
[
"Object Recognition",
"Object Reasoning",
"Action Recognition",
"Attribute Perception",
"OCR Problems",
]
),
leak_profile=LeakTestProfile(
shortcut_type="temporal_proximity",
probe_template="gate_leak_retrieval.md",
pass_threshold=0.6,
),
prompt_template="retrieval.md",
)
REASONING_FAMILY = QuestionFamilySpec(
name="REASONING",
skill_target="M2",
sampling=SamplingConstraint(
min_subtitles=3,
min_l3_nodes=4,
require_frames=False,
cross_l2_span=True,
),
legal_task_types=frozenset(
[
"Action Reasoning",
"Object Reasoning",
"Information Synopsis",
]
),
leak_profile=LeakTestProfile(
shortcut_type="frequency",
probe_template="gate_leak_reasoning.md",
pass_threshold=0.5,
),
prompt_template="reasoning.md",
)
ENUMERATION_FAMILY = QuestionFamilySpec(
name="ENUMERATION",
skill_target="M3",
sampling=SamplingConstraint(
min_subtitles=2,
min_l3_nodes=5,
require_frames=False,
cross_l2_span=False,
),
legal_task_types=frozenset(
[
"Counting Problem",
"Temporal Reasoning",
"Temporal Perception",
"Information Synopsis",
]
),
leak_profile=LeakTestProfile(
shortcut_type="option_length",
probe_template="gate_leak_enumeration.md",
pass_threshold=0.6,
),
prompt_template="enumeration.md",
)
VISUAL_FAMILY = QuestionFamilySpec(
name="VISUAL",
skill_target="M4",
sampling=SamplingConstraint(
min_subtitles=0,
min_l3_nodes=3,
require_frames=True,
cross_l2_span=False,
),
legal_task_types=frozenset(
[
"Attribute Perception",
"Counting Problem",
"OCR Problems",
"Action Recognition",
]
),
leak_profile=LeakTestProfile(
shortcut_type="visual_salience",
probe_template="gate_leak_visual.md",
pass_threshold=0.5,
),
prompt_template="visual.md",
)
SPATIAL_FAMILY = QuestionFamilySpec(
name="SPATIAL",
skill_target="M5",
sampling=SamplingConstraint(
min_subtitles=0,
min_l3_nodes=3,
require_frames=True,
cross_l2_span=False,
),
legal_task_types=frozenset(
[
"Spatial Perception",
"Spatial Reasoning",
]
),
leak_profile=LeakTestProfile(
shortcut_type="spatial_default",
probe_template="gate_leak_spatial.md",
pass_threshold=0.5,
),
prompt_template="spatial.md",
)
ALL_FAMILIES: tuple[QuestionFamilySpec, ...] = (
RETRIEVAL_FAMILY,
REASONING_FAMILY,
ENUMERATION_FAMILY,
VISUAL_FAMILY,
SPATIAL_FAMILY,
)
# 按 name 索引,用于 get_family_for_slot 快速查找
_FAMILY_BY_NAME: dict[str, QuestionFamilySpec] = {f.name: f for f in ALL_FAMILIES}
def get_family_for_slot(
task_type: str,
family_ratios: dict[str, float],
rng: random.Random,
) -> QuestionFamilySpec:
"""根据任务类型和家族权重比例,随机选择一个合法的问题家族。
Args:
task_type: 任务类型字符串必须是 12 种合法类型之一
family_ratios: 家族名称到权重的映射 {"RETRIEVAL": 0.30, ...}
rng: 随机数生成器实例确保可复现
Returns:
被选中的 QuestionFamilySpec
Raises:
ValueError: task_type 不在任何家族的 legal_task_types
ValueError: 给定 task_type 下没有合法家族所有合法族权重为 0 或不在 ratios
"""
# 检查 task_type 是否被任一家族接受
all_legal_types: set[str] = set()
for family in ALL_FAMILIES:
all_legal_types.update(family.legal_task_types)
if task_type not in all_legal_types:
msg = f"task_type '{task_type}' 不在任何家族的合法类型中"
raise ValueError(msg)
# 过滤出接受该 task_type 且在 ratios 中有正权重的家族
candidates: list[QuestionFamilySpec] = []
weights: list[float] = []
for family_name, weight in family_ratios.items():
family = _FAMILY_BY_NAME.get(family_name)
if family is None:
continue
if task_type in family.legal_task_types and weight > 0:
candidates.append(family)
weights.append(weight)
if not candidates:
msg = f"task_type '{task_type}' 下没有合法家族可选(检查 family_ratios"
raise ValueError(msg)
# 归一化权重 + 加权随机选择
chosen = rng.choices(candidates, weights=weights, k=1)[0]
return chosen
+511
View File
@@ -0,0 +1,511 @@
"""轻量四门质量检查 — 对生成题目执行 4 个独立的 LLM 质量门控。
四门
1. key_verify: 验证答案在来源素材中有证据支撑
2. blind_answer: 无上下文时 LLM 能否答对若答对 题目泄漏
3. multi_true: 检测是否有多个选项可被视为正确
4. leak_test: 按策略特定模板探测答题捷径
设计要点
- run_gates 先做 verbatim_ratio 前置短路> 0.5 直接 FAIL key_verify
- 四门并发执行asyncio.gather
- 每门加载 store/prompts/question_gen/ 下对应模板构造 messages LLM
- LLM 返回 JSON {"verdict": "pass"|"fail", "reason": "..."}解析失败视为 FAIL
- session_id 必须透传至每次 LLM 调用遥测关联
"""
from __future__ import annotations
import asyncio
import json
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from app.question_gen.postprocess import PostprocessResult
from app.tree.index import TreeIndex
from core.protocols import LLMProvider, VLMProvider
# ---------------------------------------------------------------------------
# 常量
# ---------------------------------------------------------------------------
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
_VERBATIM_THRESHOLD = 0.5
# ---------------------------------------------------------------------------
# CandidateQuestionTask 5 尚未实现,本地定义,后续迁移至 generator_v2.py
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class CandidateQuestion:
"""候选题目 — 出题管线生成、待门控审核的题目数据。
属性:
question_id: 题目唯一标识
video_id: 所属视频标识
task_type: 题型 "Action Reasoning"
skill_target: 目标失败机制编号M1-M5
question: 题目文本
options: 选项元组 ("A. ...", "B. ...", "C. ...", "D. ...")
answer: 正确答案字母 "A"
source_nodes: 来源节点 ID 元组
difficulty: 难度等级
subtitle_sentences: 字幕句子元组
frame_paths: 帧图片路径元组
"""
question_id: str
video_id: str
task_type: str
skill_target: str
question: str
options: tuple[str, ...]
answer: str
source_nodes: tuple[str, ...]
difficulty: str
subtitle_sentences: tuple[str, ...] = field(default_factory=tuple)
frame_paths: tuple[str, ...] = field(default_factory=tuple)
# ---------------------------------------------------------------------------
# 门控结果类型
# ---------------------------------------------------------------------------
class GateVerdict(Enum):
"""门控判决枚举。"""
PASS = "pass"
FAIL = "fail"
SKIP = "skip"
@dataclass(frozen=True)
class GateResult:
"""单门判决结果。
属性:
verdict: 门控判决PASS / FAIL / SKIP
reason: 判决原因说明
raw_response: LLM 原始返回内容
"""
verdict: GateVerdict
reason: str
raw_response: str
@dataclass(frozen=True)
class GateReport:
"""四门判决汇总报告。
属性:
key_verify: 关键验证门结果
blind_answer: 盲答门结果
multi_true: 多正确门结果
leak_test: 泄漏测试门结果
"""
key_verify: GateResult
blind_answer: GateResult
multi_true: GateResult
leak_test: GateResult
@property
def passed(self) -> bool:
"""全门 PASS 或 SKIP 时视为通过。"""
for gate in (self.key_verify, self.blind_answer, self.multi_true, self.leak_test):
if gate.verdict == GateVerdict.FAIL:
return False
return True
@property
def reject_reason(self) -> str | None:
"""首个 FAIL 门的 reason,全部通过返回 None。"""
for gate in (self.key_verify, self.blind_answer, self.multi_true, self.leak_test):
if gate.verdict == GateVerdict.FAIL:
return gate.reason
return None
# ---------------------------------------------------------------------------
# 内部工具函数
# ---------------------------------------------------------------------------
def _load_prompt_template(filename: str) -> str:
"""加载 store/prompts/question_gen/ 下的模板文件。
参数:
filename: 模板文件名 "gate_key_verify.md"
返回:
模板内容字符串
异常:
FileNotFoundError: 模板文件不存在
"""
path = _PROMPTS_DIR / filename
if not path.exists():
msg = f"门控模板文件不存在: {path}"
raise FileNotFoundError(msg)
return path.read_text(encoding="utf-8")
def _resolve_source_text(candidate: CandidateQuestion, tree: TreeIndex) -> str:
"""从树中解析候选题的来源节点文本。
遍历树的所有层级匹配 source_nodes 中的 ID提取对应描述文本
参数:
candidate: 候选题目
tree: 视频树索引
返回:
拼接的来源节点描述文本
"""
texts: list[str] = []
target_ids = set(candidate.source_nodes)
for l1 in tree.roots:
if l1.id in target_ids:
texts.append(f"[L1 {l1.id}] {l1.card.scene_summary}")
for l2 in l1.children:
if l2.id in target_ids:
texts.append(f"[L2 {l2.id}] {l2.card.event_description}")
for l3 in l2.children:
if l3.id in target_ids:
texts.append(f"[L3 {l3.id}] {l3.card.frame_summary}")
if not texts:
logger.warning(
"未找到来源节点: candidate={}, source_nodes={}",
candidate.question_id,
candidate.source_nodes,
)
return "(no source material found)"
return "\n".join(texts)
def _format_options(options: tuple[str, ...]) -> str:
"""将选项元组格式化为可读字符串。"""
return "\n".join(options)
def _parse_gate_response(raw_content: str) -> tuple[GateVerdict, str]:
"""解析 LLM 返回的门控 JSON 响应。
期望格式: {"verdict": "pass"|"fail", "reason": "..."}
解析失败时返回 FAIL + parse_error
参数:
raw_content: LLM 返回的原始文本
返回:
(verdict, reason) 元组
"""
try:
# 尝试从文本中提取 JSON(可能被 markdown 代码块包裹)
content = raw_content.strip()
if "```" in content:
# 提取代码块中的内容
parts = content.split("```")
for part in parts:
stripped = part.strip()
if stripped.startswith("json"):
stripped = stripped[4:].strip()
if stripped.startswith("{"):
content = stripped
break
data = json.loads(content)
verdict_str = data.get("verdict", "").lower()
reason = data.get("reason", "")
if verdict_str == "pass":
return GateVerdict.PASS, reason
elif verdict_str == "fail":
return GateVerdict.FAIL, reason
else:
return GateVerdict.FAIL, f"parse_error: invalid verdict '{verdict_str}'"
except (json.JSONDecodeError, AttributeError, TypeError) as e:
return GateVerdict.FAIL, f"parse_error: {e}"
# ---------------------------------------------------------------------------
# 四门实现
# ---------------------------------------------------------------------------
def _resolve_source_frames(candidate: CandidateQuestion, tree: TreeIndex) -> list[str]:
"""从树中收集候选题来源节点关联的帧路径。
参数:
candidate: 候选题目
tree: 视频树索引
返回:
去重后的帧路径列表最多 10 避免 VLM 输入过长
"""
frames: list[str] = []
target_ids = set(candidate.source_nodes)
for l1 in tree.roots:
for l2 in l1.children:
if l2.id in target_ids:
for l3 in l2.children:
if l3.frame_path:
frames.append(l3.frame_path)
for l3 in l2.children:
if l3.id in target_ids and l3.frame_path:
frames.append(l3.frame_path)
# 也使用候选题自带的帧路径
frames.extend(candidate.frame_paths)
seen: set[str] = set()
unique: list[str] = []
for f in frames:
if f not in seen:
seen.add(f)
unique.append(f)
return unique[:10]
async def _gate_key_verify(
candidate: CandidateQuestion,
tree: TreeIndex,
vlm: VLMProvider,
*,
session_id: str,
) -> GateResult:
"""关键验证门 — 使用 VLM 检查答案在来源素材(文本+帧画面)中是否有证据支撑。
参数:
candidate: 候选题目
tree: 视频树索引
vlm: VLM 图文调用端口同时看文本和帧画面
session_id: 会话 ID遥测关联
返回:
GateResult 实例
"""
source_text = _resolve_source_text(candidate, tree)
frames = _resolve_source_frames(candidate, tree)
template = _load_prompt_template("gate_key_verify.md")
prompt = template.format(
source_text=source_text,
question=candidate.question,
options=_format_options(candidate.options),
answer=candidate.answer,
)
if frames:
response = await vlm.chat_with_images(
[{"role": "user", "content": prompt}],
images=frames,
session_id=session_id,
)
else:
# 无帧时降级为纯文本(不应常见)
logger.warning("key_verify 无可用帧,降级纯文本: {}", candidate.question_id)
response = await vlm.chat_with_images(
[{"role": "user", "content": prompt}],
images=[],
session_id=session_id,
)
verdict, reason = _parse_gate_response(response.content)
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
async def _gate_blind_answer(
candidate: CandidateQuestion,
llm: LLMProvider,
*,
session_id: str,
) -> GateResult:
"""盲答门 — 无上下文时 LLM 能否答对(答对 → 题目泄漏)。
参数:
candidate: 候选题目
llm: LLM 调用端口
session_id: 会话 ID遥测关联
返回:
GateResult 实例
"""
template = _load_prompt_template("gate_blind_answer.md")
prompt = template.format(
question=candidate.question,
options=_format_options(candidate.options),
)
response = await llm.chat(
[{"role": "user", "content": prompt}],
session_id=session_id,
)
verdict, reason = _parse_gate_response(response.content)
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
async def _gate_multi_true(
candidate: CandidateQuestion,
tree: TreeIndex,
llm: LLMProvider,
*,
session_id: str,
) -> GateResult:
"""多正确门 — 检测是否有多个选项可被视为正确。
参数:
candidate: 候选题目
tree: 视频树索引
llm: LLM 调用端口
session_id: 会话 ID遥测关联
返回:
GateResult 实例
"""
source_text = _resolve_source_text(candidate, tree)
template = _load_prompt_template("gate_multi_true.md")
prompt = template.format(
source_text=source_text,
question=candidate.question,
options=_format_options(candidate.options),
)
response = await llm.chat(
[{"role": "user", "content": prompt}],
session_id=session_id,
)
verdict, reason = _parse_gate_response(response.content)
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
async def _gate_leak_test(
candidate: CandidateQuestion,
leak_probe_template: str,
llm: LLMProvider,
*,
session_id: str,
) -> GateResult:
"""泄漏测试门 — 按策略特定模板探测答题捷径。
参数:
candidate: 候选题目
leak_probe_template: 泄漏探测模板文件名store/prompts/question_gen/
llm: LLM 调用端口
session_id: 会话 ID遥测关联
返回:
GateResult 实例
"""
template = _load_prompt_template(leak_probe_template)
prompt = template.format(
question=candidate.question,
options=_format_options(candidate.options),
answer=candidate.answer,
)
response = await llm.chat(
[{"role": "user", "content": prompt}],
session_id=session_id,
)
verdict, reason = _parse_gate_response(response.content)
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
# ---------------------------------------------------------------------------
# 编排入口
# ---------------------------------------------------------------------------
async def run_gates(
candidate: CandidateQuestion,
tree: TreeIndex,
llm: LLMProvider,
leak_probe_template: str,
postprocess: PostprocessResult,
*,
vlm: VLMProvider | None = None,
session_id: str,
) -> GateReport:
"""编排四门并发执行,返回汇总报告。
前置规则verbatim_ratio > 0.5 时直接短路 key_verify FAIL不调 LLM
参数:
candidate: 候选题目
tree: 视频树索引
llm: LLM 调用端口
leak_probe_template: 泄漏探测模板文件名store/prompts/question_gen/
postprocess: 后处理结果 verbatim_ratio
vlm: VLM 图文调用端口key_verify 使用None 时降级为 LLM
session_id: 会话 ID遥测关联
返回:
GateReport 四门汇总
"""
# Phase 1: verbatim 前置短路 — 不调 LLM,直接返回
if postprocess.verbatim_ratio > _VERBATIM_THRESHOLD:
logger.info(
"verbatim_ratio={:.3f} > {:.1f}key_verify 短路 FAIL,其余三门 SKIP",
postprocess.verbatim_ratio,
_VERBATIM_THRESHOLD,
)
skip_result = GateResult(
verdict=GateVerdict.SKIP,
reason="skipped due to verbatim short-circuit",
raw_response="",
)
return GateReport(
key_verify=GateResult(
verdict=GateVerdict.FAIL,
reason=f"verbatim_ratio={postprocess.verbatim_ratio:.3f} exceeds threshold {_VERBATIM_THRESHOLD}",
raw_response="",
),
blind_answer=skip_result,
multi_true=skip_result,
leak_test=skip_result,
)
# Phase 2: 四门并发执行(key_verify 使用 VLM 看帧+文本)
key_verify_provider = vlm if vlm is not None else llm
key_result, blind_result, multi_result, leak_result = await asyncio.gather(
_gate_key_verify(candidate, tree, key_verify_provider, session_id=session_id),
_gate_blind_answer(candidate, llm, session_id=session_id),
_gate_multi_true(candidate, tree, llm, session_id=session_id),
_gate_leak_test(candidate, leak_probe_template, llm, session_id=session_id),
)
report = GateReport(
key_verify=key_result,
blind_answer=blind_result,
multi_true=multi_result,
leak_test=leak_result,
)
if report.passed:
logger.debug("四门全部通过: question_id={}", candidate.question_id)
else:
logger.info(
"门控拒绝: question_id={}, reason={}",
candidate.question_id,
report.reject_reason,
)
return report
+461
View File
@@ -0,0 +1,461 @@
"""v2 生成器 — 基于策略特化 prompt 模板的单题 VLM 出题模块。
使用 VLMProvider 接口调用视觉语言模型结合 per-strategy prompt 模板
MaterialContext 素材上下文生成一道四选一候选题
典型调用路径::
candidate = await generate_one_v2(
vlm=vlm_client,
tree=tree_index,
material=material_ctx,
task_type="Action Reasoning",
seq=1,
video_id="vid_001",
prompt_template="retrieval.md",
strategy_name="RETRIEVAL",
skill_target="M1",
session_id="sess_001",
)
"""
from __future__ import annotations
import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING
from json_repair import repair_json
from loguru import logger
if TYPE_CHECKING:
from app.question_gen.sampler_v2 import MaterialContext
from app.tree.index import TreeIndex
from core.protocols import VLMProvider
# ---------------------------------------------------------------------------
# 常量
# ---------------------------------------------------------------------------
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
_VALID_ANSWERS = frozenset({"A", "B", "C", "D"})
_VALID_DIFFICULTIES = frozenset({"easy", "medium", "hard"})
# ---------------------------------------------------------------------------
# CandidateQuestion(规范定义位置)
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class CandidateQuestion:
"""候选题目 — 出题管线生成、待门控审核的题目数据。
属性:
question_id: 题目唯一标识格式 "{video_id}_{task_type}_{seq:04d}"
video_id: 所属视频标识
task_type: 题型 "Action Reasoning"
skill_target: 目标失败机制编号M1-M5
question: 题目文本
options: 选项元组 ("A. ...", "B. ...", "C. ...", "D. ...")
answer: 正确答案字母 "A"
source_nodes: 来源节点 ID 元组
difficulty: 难度等级easy/medium/hard
subtitle_sentences: 验证材料 字幕句子元组
frame_paths: 验证材料 帧图片路径元组
"""
question_id: str
video_id: str
task_type: str
skill_target: str
question: str
options: tuple[str, ...]
answer: str
source_nodes: tuple[str, ...]
difficulty: str
subtitle_sentences: tuple[str, ...] = field(default_factory=tuple)
frame_paths: tuple[str, ...] = field(default_factory=tuple)
# ---------------------------------------------------------------------------
# Prompt 模板加载
# ---------------------------------------------------------------------------
def _load_prompt_template(template_name: str) -> str:
"""加载 prompt 模板文件。
参数:
template_name: store/prompts/question_gen/ 下的模板文件名
返回:
模板内容字符串
异常:
FileNotFoundError: 模板文件不存在
"""
path = _PROMPTS_DIR / template_name
if not path.exists():
msg = f"Prompt 模板文件不存在: {path}"
raise FileNotFoundError(msg)
return path.read_text(encoding="utf-8")
# ---------------------------------------------------------------------------
# Prompt 构建
# ---------------------------------------------------------------------------
def _build_v2_prompt(
prompt_template: str,
strategy_name: str,
material: MaterialContext,
task_type: str,
seq: int,
*,
reject_reason: str | None = None,
sub_pattern_instruction: str | None = None,
) -> tuple[list[dict[str, str]], list[str]]:
"""构建 VLM 出题调用的 messages 和帧路径列表。
参数:
prompt_template: prompt 模板文件名store/prompts/question_gen/
strategy_name: 策略名称 "RETRIEVAL"
material: 采样素材上下文
task_type: 任务类型字符串
seq: 当前序号
reject_reason: 上一次被门控拒绝的原因用于引导 VLM 避免相同错误
sub_pattern_instruction: 子模式特殊指令如有
返回:
二元组:
- messages: 适配 VLMProvider message 列表system + user
- frame_paths: 需发送给 VLM 的帧路径列表
"""
# Phase 1: 加载策略模板作为 system prompt
template_content = _load_prompt_template(prompt_template)
system_message = template_content
# Phase 2: 构建 user prompt — 聚合素材信息
user_parts: list[str] = []
user_parts.append(f"## Task Type: {task_type}")
user_parts.append(f"## Question Family: {strategy_name}")
user_parts.append(f"## Sequence: #{seq}")
# 字幕素材
if material.subtitle_sentences:
user_parts.append("\n## Subtitle Content:")
for i, sent in enumerate(material.subtitle_sentences, 1):
user_parts.append(f" {i}. {sent}")
# 跨 L2 上下文
if material.cross_l2_texts:
user_parts.append("\n## Cross-Segment Context:")
for text in material.cross_l2_texts:
user_parts.append(f" - {text}")
# 帧路径提示(VLM 会接收实际图像,此处仅作文本参考)
if material.frame_paths:
user_parts.append(f"\n## Visual Frames: {len(material.frame_paths)} frames attached.")
# 拒绝原因注入
if reject_reason is not None:
user_parts.append(
f"\n## IMPORTANT - Previous Attempt Rejected:\n"
f"Your previous question was rejected for the following reason:\n"
f'"{reject_reason}"\n'
f"Please generate a NEW question that avoids this issue."
)
# 子模式特殊指令注入
if sub_pattern_instruction is not None:
user_parts.append(f"\n## Special Focus:\n{sub_pattern_instruction}")
# 输出格式指令
user_parts.append(
"\n## Output Format:\n"
"Respond with ONLY a JSON object in this exact format:\n"
"```json\n"
"{\n"
' "question": "Your question text here",\n'
' "options": ["A. ...", "B. ...", "C. ...", "D. ..."],\n'
' "answer": "A",\n'
' "difficulty": "easy|medium|hard"\n'
"}\n"
"```"
)
user_content = "\n".join(user_parts)
messages = [
{"role": "system", "content": system_message},
{"role": "user", "content": user_content},
]
# Phase 3: 帧路径
frame_paths = list(material.frame_paths)
return messages, frame_paths
# ---------------------------------------------------------------------------
# 响应解析
# ---------------------------------------------------------------------------
def _extract_json_from_text(raw: str) -> str:
"""从可能被 markdown 代码块包裹的文本中提取 JSON 部分。
参数:
raw: VLM 原始返回文本
返回:
清理后的 JSON 字符串
"""
content = raw.strip()
if "```" in content:
parts = content.split("```")
for part in parts:
stripped = part.strip()
if stripped.startswith("json"):
stripped = stripped[4:].strip()
if stripped.startswith("{"):
return stripped
return content
@dataclass(frozen=True)
class _ValidatedFields:
"""字段校验通过后的中间结构。"""
question: str
options: tuple[str, ...]
answer: str
difficulty: str
def _validate_parsed_fields(data: dict) -> _ValidatedFields:
"""校验 VLM 响应 JSON 的必填字段并规范化。
参数:
data: 已解析的 JSON 字典
返回:
_ValidatedFields 实例字段已规范化
异常:
ValueError: 缺少必填字段或字段值非法
"""
missing = [f for f in ("question", "options", "answer", "difficulty") if f not in data]
if missing:
msg = f"VLM 响应缺少必填字段: {', '.join(missing)}"
raise ValueError(msg)
question_text = str(data["question"])
options_raw = data["options"]
answer = str(data["answer"]).strip().upper()
difficulty = str(data["difficulty"]).strip().lower()
# 校验 optionsanswer 约束为 A-D,因此必须恰好 4 个选项)
if not isinstance(options_raw, list) or len(options_raw) != 4:
msg = f"options 字段必须恰好包含 4 个选项,实际数量: {len(options_raw) if isinstance(options_raw, list) else type(options_raw).__name__}"
raise ValueError(msg)
# 校验 answer
if answer not in _VALID_ANSWERS:
msg = f"answer 字段值 '{answer}' 非法,必须为 A/B/C/D 之一"
raise ValueError(msg)
# 校验 difficulty(宽容处理:非法值回退为 medium)
if difficulty not in _VALID_DIFFICULTIES:
logger.warning(
"difficulty '{}' 不在预设范围 {},回退为 'medium'",
difficulty,
_VALID_DIFFICULTIES,
)
difficulty = "medium"
return _ValidatedFields(
question=question_text,
options=tuple(str(o) for o in options_raw),
answer=answer,
difficulty=difficulty,
)
def _parse_v2_response(
raw: str,
video_id: str,
task_type: str,
skill_target: str,
seq: int,
source_nodes: tuple[str, ...],
) -> CandidateQuestion:
"""解析 VLM 返回的 JSON 响应,构造 CandidateQuestion。
流程
1. 提取 JSON处理 markdown 包裹
2. json_repair 修复常见格式错误
3. 校验必填字段委托 _validate_parsed_fields
4. 构造 CandidateQuestion 实例
参数:
raw: VLM 原始返回文本
video_id: 视频标识
task_type: 任务类型
skill_target: 目标技能编号
seq: 当前序号
source_nodes: 来源节点 ID 元组
返回:
CandidateQuestion 实例
异常:
ValueError: JSON 解析失败或缺少必填字段或字段值非法
"""
# Phase 1: 提取 + 修复 JSON
json_text = _extract_json_from_text(raw)
repaired = repair_json(json_text, return_objects=False)
# Phase 2: 解析为字典
try:
data = json.loads(repaired)
except json.JSONDecodeError as e:
msg = f"VLM 响应 JSON 解析失败: {e}. 原始文本: {raw[:200]}"
raise ValueError(msg) from e
if not isinstance(data, dict):
msg = f"VLM 响应顶层不是 JSON 对象: type={type(data).__name__}"
raise ValueError(msg)
# Phase 3: 字段校验
fields = _validate_parsed_fields(data)
# Phase 4: 构造 CandidateQuestion
question_id = f"{video_id}_{task_type}_{seq:04d}"
return CandidateQuestion(
question_id=question_id,
video_id=video_id,
task_type=task_type,
skill_target=skill_target,
question=fields.question,
options=fields.options,
answer=fields.answer,
source_nodes=source_nodes,
difficulty=fields.difficulty,
)
# ---------------------------------------------------------------------------
# 主入口
# ---------------------------------------------------------------------------
async def generate_one_v2(
vlm: VLMProvider,
tree: TreeIndex,
material: MaterialContext,
task_type: str,
seq: int,
*,
video_id: str,
prompt_template: str,
strategy_name: str,
skill_target: str,
reject_reason: str | None = None,
sub_pattern_instruction: str | None = None,
session_id: str,
) -> CandidateQuestion:
"""调用 VLM 生成一道候选题目。
流程
1. 构建 per-strategy prompt + 帧路径
2. 调用 VLMProvider.chat_with_images
3. 解析响应为 CandidateQuestion
4. 附加素材验证信息subtitle_sentencesframe_paths
参数:
vlm: VLM 调用端口
tree: 视频树索引当前未直接使用预留后续扩展
material: 采样素材上下文
task_type: 任务类型字符串
seq: 当前序号
video_id: 视频标识
prompt_template: prompt 模板文件名
strategy_name: 策略名称 "RETRIEVAL"
skill_target: 目标失败机制编号M1-M5
reject_reason: 上一次被门控拒绝的原因
sub_pattern_instruction: 子模式特殊指令如有
session_id: 会话 ID遥测关联
返回:
CandidateQuestion 实例包含验证材料
异常:
ValueError: VLM 响应解析失败
FileNotFoundError: prompt 模板不存在
"""
# Phase 1: 构建 prompt
messages, frame_paths = _build_v2_prompt(
prompt_template=prompt_template,
strategy_name=strategy_name,
material=material,
task_type=task_type,
seq=seq,
reject_reason=reject_reason,
sub_pattern_instruction=sub_pattern_instruction,
)
# Phase 2: 调用 VLM
logger.debug(
"generate_one_v2: strategy={}, task_type={}, seq={}, frames={}",
strategy_name,
task_type,
seq,
len(frame_paths),
)
response = await vlm.chat_with_images(
messages,
frame_paths,
session_id=session_id,
)
# Phase 3: 解析响应
candidate = _parse_v2_response(
raw=response.content,
video_id=video_id,
task_type=task_type,
skill_target=skill_target,
seq=seq,
source_nodes=material.source_nodes,
)
# Phase 4: 附加验证材料(构造新实例,因 frozen=True
candidate = CandidateQuestion(
question_id=candidate.question_id,
video_id=candidate.video_id,
task_type=candidate.task_type,
skill_target=candidate.skill_target,
question=candidate.question,
options=candidate.options,
answer=candidate.answer,
source_nodes=candidate.source_nodes,
difficulty=candidate.difficulty,
subtitle_sentences=tuple(material.subtitle_sentences),
frame_paths=tuple(material.frame_paths),
)
logger.debug(
"generate_one_v2 完成: question_id={}, difficulty={}",
candidate.question_id,
candidate.difficulty,
)
return candidate
+205
View File
@@ -0,0 +1,205 @@
"""题目加载与分层采样。
benchmark JSON 目录加载题目提供按对错比例的分层采样
对应训练循环中的 DataLoader 角色
"""
from __future__ import annotations
import json
import random
from typing import TYPE_CHECKING
from core.types import GeneratedQuestion
if TYPE_CHECKING:
from pathlib import Path
from core.types import QuestionUnit
_LEGACY_DEFAULT_DIFFICULTY = "medium"
def load_benchmark(questions_dir: Path) -> list[GeneratedQuestion]:
"""从 benchmark JSON 目录加载题目列表。
video_id 优先使用题目 JSON 中的 ``video_id`` 字段若缺失则回退到
文件名不含扩展名Video-MME benchmark 按视频拆文件文件名即
video_idv2 生成题把多视频题目合并在单个 JSON 每条记录自带
``video_id``两种格式均兼容
pair 契约字段``pair_id`` / ``question_role`` / ``flip_axis`` / ``unit_id``
``.get`` 读取 benchmark 无这些键时退化为 single``question_role``
默认 "single"``unit_id`` 留空由 __post_init__ 回填为 question_id
保证历史题库可无缝加载
参数:
questions_dir: 包含 *.json 文件的目录路径
返回:
按文件名排序加载的题目列表
"""
results: list[GeneratedQuestion] = []
for path in sorted(questions_dir.glob("*.json")):
fallback_video_id = path.stem
with open(path, encoding="utf-8") as f:
qa_list: list[dict] = json.load(f)
for qa in qa_list:
results.append(
GeneratedQuestion(
question_id=qa["question_id"],
video_id=qa.get("video_id", fallback_video_id),
task_type=qa["task_type"],
question=qa["question"],
options=tuple(qa["options"]),
answer=qa["answer"],
source_nodes=tuple(qa.get("source_nodes", ())),
difficulty=qa.get("difficulty", _LEGACY_DEFAULT_DIFFICULTY),
family=qa.get("family"),
skill_target=qa.get("skill_target"),
difficulty_steps=qa.get("difficulty_steps"),
sub_pattern=qa.get("sub_pattern"),
# pair 契约字段:旧 benchmark 无这些键时按 single 默认兜底,
# unit_id 留空交由 GeneratedQuestion.__post_init__ 回填。
pair_id=qa.get("pair_id"),
question_role=qa.get("question_role", "single"),
flip_axis=qa.get("flip_axis"),
unit_id=qa.get("unit_id", ""),
)
)
return results
def stratified_sample(
questions: list[GeneratedQuestion],
correctness: dict[str, bool],
size: int,
correct_ratio: float | None,
task_types: list[str] | None,
seed: int,
min_per_class: int | None,
) -> list[GeneratedQuestion]:
"""按题型过滤后采样 size 个单元,可选按对错比例分层并按题型保底。
参数:
questions: 候选题目全集single 与孪生对成员可混含
correctness: question_id -> 基线是否答对单元级正确性取成员 AND
size: 采样单元总量single 1pair 1
correct_ratio: 采样中"基线答对"单元的占比None 表示自然分布
task_types: 限定题型None 表示不限
seed: 随机种子保证可复现
min_per_class: 每个题型补足到的单元下限None 表示不补足
返回:
采样后的题目列表pair 单元展开为原始的两道题
异常:
ValueError: 自然分布时单元池不足 size或分层时某层单元不足
关键实现:
**QuestionUnit 为采样原子**single 1pair 1size /
correct_ratio / min_per_class 均按 unit 计数孪生对两题永不被劈开
采样完成后 flatten_units 展开回逐题列表 single 输入时 build_units
与题目一一对应顺序不变rng 消耗与旧逐题实现完全一致字节级回归
build_units / flatten_units 采用函数内延迟导入loader question_gen
question_units harness harness 包初始化会反向 import question_gen
模块级导入将触发循环依赖沿用 adversarial_filter 的既有做法
"""
from app.harness.question_units import build_units, flatten_units
rng = random.Random(seed)
units = build_units(questions)
pool = [u for u in units if task_types is None or u.task_type in task_types]
if correct_ratio is None:
if len(pool) < size:
raise ValueError(f"自然分布采样不足: 需 {size} 个单元, 实有 {len(pool)}")
sampled = rng.sample(pool, size)
else:
sampled = _ratio_stratified_sample(pool, correctness, size, correct_ratio, rng)
if min_per_class is not None:
sampled = _backfill_per_class(sampled, pool, min_per_class, rng)
return flatten_units(sampled)
def _ratio_stratified_sample(
pool: list[QuestionUnit],
correctness: dict[str, bool],
size: int,
correct_ratio: float,
rng: random.Random,
) -> list[QuestionUnit]:
"""按对错比例分层采样:对单元占 correct_ratio,其余为错单元。
参数:
pool: 题型过滤后的候选单元
correctness: question_id -> 基线是否答对
size: 采样单元总量
correct_ratio: 对单元占比
rng: 随机数发生器
返回:
采样后的单元列表对单元在前错单元在后
异常:
ValueError: 对单元或错单元层不足
关键实现:
unit_correctness 采用函数内延迟导入loader question_gen
question_units harness模块级导入将触发循环依赖沿用 build_units /
flatten_units 的既有做法 strict=False 保持"缺基线对错即视为未答对"
loose 语义不变
"""
from app.harness.question_units import unit_correctness
correct = [u for u in pool if unit_correctness(u, correctness, strict=False)]
wrong = [u for u in pool if not unit_correctness(u, correctness, strict=False)]
n_correct = round(size * correct_ratio)
n_wrong = size - n_correct
if len(correct) < n_correct or len(wrong) < n_wrong:
raise ValueError(
f"分层不足: 需对{n_correct}/错{n_wrong}, 实有对{len(correct)}/错{len(wrong)}"
)
return rng.sample(correct, n_correct) + rng.sample(wrong, n_wrong)
def _backfill_per_class(
sampled: list[QuestionUnit],
pool: list[QuestionUnit],
min_per_class: int,
rng: random.Random,
) -> list[QuestionUnit]:
"""对候选池中出现的每个题型,将采样单元补足到 min_per_class 个。
遍历对象是候选池 pool 里出现的全部题型非仅 sampled 命中的
保证任意稀疏题型都能拿到足额样本补足以 unit 为原子孪生对整进整出
参数:
sampled: 主采样结果单元不修改返回新列表
pool: 候选单元全集补足来源 + 题型枚举来源
min_per_class: 每个题型的单元下限
rng: 随机数发生器
返回:
补足后的单元列表
"""
selected_ids = {u.unit_id for u in sampled}
result = list(sampled)
counts: dict[str, int] = {}
for u in sampled:
counts[u.task_type] = counts.get(u.task_type, 0) + 1
ordered_task_types: dict[str, None] = {}
for u in pool:
ordered_task_types.setdefault(u.task_type, None)
for task_type in ordered_task_types:
deficit = min_per_class - counts.get(task_type, 0)
if deficit <= 0:
continue
candidates = [u for u in pool if u.task_type == task_type and u.unit_id not in selected_ids]
take = rng.sample(candidates, min(deficit, len(candidates)))
for u in take:
selected_ids.add(u.unit_id)
result.append(u)
return result
+196
View File
@@ -0,0 +1,196 @@
"""accepted 题库的 pair 原子成对落盘 helper(纯件,不依赖 pipeline)。
三件可复用纯件供 Phase 2 pipeline on_accept 回调复用``PairPendingBuffer``
pair_id 收齐才 emit 单元``write_accepted``tmp + os.replace 原子写
``read_accepted``聚合成单元并剔除磁盘悬挂孤儿
wiring 归属本模块**只是纯件**不接任何生成侧回调真正的 on_accept wiring
** Phase 2**当前项目真实 accepted 写入点是 ``adversarial_filter.write_final_bank``
Phase 2 待重建的旧代码本模块沿用其 tmp + os.replace 模式但不 import/不改动它
unit_hash 校验``QuestionUnit.unit_hash`` 目前恒为 ""填充是 Phase 2 的事
``GeneratedQuestion`` 不携带 unit_hash 字段故把"unit_hash 不一致→拒"落地为**
pair_id 两成员的绑定一致性校验**video_id / task_type / flip_axis不一致即
fail-fast raise语义等价只有 payload 绑定一致的孪生对才允许聚合
"""
from __future__ import annotations
import json
import os
from collections import defaultdict
from typing import TYPE_CHECKING
from loguru import logger
from core.types import GeneratedQuestion, QuestionUnit
if TYPE_CHECKING:
from pathlib import Path
def _check_pair_binding(first: GeneratedQuestion, second: GeneratedQuestion) -> None:
"""校验同 pair_id 两成员的绑定一致性,不一致 fail-fastunit_hash 语义代偿)。
参数 first/second 为先后到达的孪生对成员video_id / task_type / flip_axis 任一
不一致即 raise ValueError绑定不一致的两条题目不构成同一 payload 的孪生对
拒绝聚合而非静默兜底
"""
mismatches: list[str] = []
if first.video_id != second.video_id:
mismatches.append(f"video_id: {first.video_id} != {second.video_id}")
if first.task_type != second.task_type:
mismatches.append(f"task_type: {first.task_type} != {second.task_type}")
if first.flip_axis != second.flip_axis:
mismatches.append(f"flip_axis: {first.flip_axis} != {second.flip_axis}")
if mismatches:
raise ValueError(
f"pair {first.pair_id} 两成员绑定不一致(" + "".join(mismatches) + "),拒绝聚合"
)
class PairPendingBuffer:
"""按 pair_id 收齐孪生对才 emit 单元的有状态缓冲器。
喂题接口 ``add`` 逐条消费题目single 立即 emit ``QuestionUnit.from_single``
pair 成员先缓存等同一 pair_id original+mirror 都到齐才 emit 一个 pair 单元
复用 ``QuestionUnit.from_pair`` fail-fast 校验批次末尾用
``pending_orphans`` 检测"只落 P 未落 Q"的悬挂项
"""
def __init__(self) -> None:
"""pending 以 pair_id 索引首个到达的孪生对成员,等伙伴到齐再 emit。"""
self._pending: dict[str, GeneratedQuestion] = {}
def add(self, q: GeneratedQuestion) -> QuestionUnit | None:
"""喂入一条题目 q,返回本次凑齐的单元或 None(pair 尚未配齐)。
single 立即返回 kind="single" 单元pair 首个成员缓存并返回 None第二个
成员到齐后返回 kind="pair" 单元 pair_id 两成员绑定不一致或角色非法
两个 original raise ValueError
关键实现:配齐后**先校验成功组装出 unit 才从 pending 删除**
``_check_pair_binding`` / ``_order_pair`` raise首成员仍留在 pending调用方
``pending_orphans`` 可取回被拒的悬挂成员绑定校验用显式 ValueError不依赖
会被 ``-O`` 剥除的 assert
"""
if not q.pair_id:
return QuestionUnit.from_single(q)
partner = self._pending.get(q.pair_id)
if partner is None:
self._pending[q.pair_id] = q
return None
_check_pair_binding(partner, q)
original, mirror = _order_pair(partner, q)
unit = QuestionUnit.from_pair(original, mirror)
del self._pending[q.pair_id]
return unit
def pending_orphans(self) -> list[GeneratedQuestion]:
"""返回仍未配齐的悬挂成员,供调用方在批次末尾检测"只落 P 未落 Q""""
return list(self._pending.values())
def _order_pair(
a: GeneratedQuestion, b: GeneratedQuestion
) -> tuple[GeneratedQuestion, GeneratedQuestion]:
"""按 question_role 把两成员 a/b 定序为 (original, mirror)。
两成员不构成恰好 1 original + 1 mirror角色缺失或重复 raise ValueError
next(...) 静默 StopIteration
"""
originals = [q for q in (a, b) if q.question_role == "pair_original"]
mirrors = [q for q in (a, b) if q.question_role == "pair_mirror"]
if len(originals) != 1 or len(mirrors) != 1:
raise ValueError(
f"pair {a.pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)}"
"需各恰好 1 条"
)
return originals[0], mirrors[0]
def _validate_disk_pair(pair_id: str, group: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
"""三态判定磁盘上同 pair_id 分组,区分悬挂孤儿(drop)与结构损坏(raise)。
- 恰好 1 original + 1 mirror合法孪生对额外做 ``_check_pair_binding`` 显式绑定
校验video_id/task_type/flip_axis-O 下仍生效返回其两题
- size==1只落 P 未落 Q业务上合法的悬挂孤儿warn + drop返回 []
- 其余size>2 超员 size==2 角色重复/缺角色数据损坏/外部篡改 P5
fail-loudraise ValueError pair_id 与成员构成绝不静默吞
"""
originals = sum(1 for q in group if q.question_role == "pair_original")
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
if len(group) == 2 and originals == 1 and mirrors == 1:
_check_pair_binding(group[0], group[1])
return group
if len(group) == 1:
logger.warning("磁盘悬挂孤儿 pair {}:仅 1 成员(缺伙伴),warn+drop 该 unit", pair_id)
return []
raise ValueError(
f"磁盘 pair {pair_id} 结构损坏:成员数={len(group)}"
f"original={originals} mirror={mirrors}),需恰好 1 original + 1 mirror"
)
def _keep_complete_disk_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
"""筛选磁盘读回的题目:single 全保留、pair 按 ``_validate_disk_pair`` 三态处理。
悬挂孤儿 warn+drop结构损坏/绑定不一致 raise ValueError合法成对保留后交给
build_unitssingle 全保留
"""
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
singles: list[GeneratedQuestion] = []
for q in questions:
if q.pair_id:
by_pair[q.pair_id].append(q)
else:
singles.append(q)
kept_pairs = [q for pid, grp in by_pair.items() for q in _validate_disk_pair(pid, grp)]
return singles + kept_pairs
def write_accepted(path: Path, units: list[QuestionUnit]) -> None:
"""把 units 全量原子写到 pathtmp + os.replace),孪生对两题相邻落盘。
parent 不存在则自动创建任一 pair 单元结构非法size2落盘前 fail-fast raise
关键实现:沿用 ``write_final_bank`` 的原子写模式先写同目录 ``.tmp``
``os.replace`` 覆盖保证读到的 JSON 恒完整序列化复用 T9 pools.py
``_q_to_dict``唯一 GeneratedQuestiondict schema pair 四字段函数内
import 规避 app.question_genapp.harness 循环依赖
"""
from app.harness.pools import _q_to_dict
from app.harness.question_units import flatten_units, validate_units
validate_units(units)
records = [_q_to_dict(q) for q in flatten_units(units)]
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8")
os.replace(str(tmp), str(path))
logger.info("accepted 题库全量原子写: {} 单元 / {} 题 → {}", len(units), len(records), path)
def read_accepted(path: Path) -> list[QuestionUnit]:
"""读回 path 的 accepted JSON → 聚合为单元列表。
磁盘是外部输入 P5 全量校验后再用``_keep_complete_disk_pairs`` pair 分组三态处理
"只落 P 未落 Q"的悬挂孤儿 warn+drop不进结果 raise结构损坏超员/
角色重复或绑定不一致video_id/task_type/flip_axis显式 raise ValueError
依赖 build_units 内会被 ``-O`` 剥除的 assertsingle 全保留
关键实现:反序列化复用 T9 pools.py ``_dict_to_q``pair 四字段 .get 兼容
函数内 import 规避循环依赖sift ``build_units`` 聚合``validate_units`` 二次
防御闸门
"""
from app.harness.pools import _dict_to_q
from app.harness.question_units import build_units, validate_units
raw = json.loads(path.read_text(encoding="utf-8"))
questions = [_dict_to_q(d) for d in raw]
kept = _keep_complete_disk_pairs(questions)
return validate_units(build_units(kept))
File diff suppressed because it is too large Load Diff
+381
View File
@@ -0,0 +1,381 @@
"""确定性后处理层 — 零 LLM 的选项洗牌、指代黑名单、逐字重复率、时间锚点、素材禁区检测。
所有函数均为纯函数给定输入必定产出相同输出
用于 pipeline 出题后门控前的确定性质量检查
"""
from __future__ import annotations
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
import random
# ---------------------------------------------------------------------------
# 数据结构
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class PostprocessResult:
"""后处理结果汇总。
属性:
options: 洗牌后的选项元组
answer: 重映射后的答案字母
referent_violations: 指代黑名单违规描述列表
verbatim_ratio: 正确选项与来源素材的逐字重复率 [0.0, 1.0]
has_time_anchor: 题目是否包含时间锚点
"""
options: tuple[str, ...]
answer: str
referent_violations: list[str]
verbatim_ratio: float
has_time_anchor: bool
# ---------------------------------------------------------------------------
# 指代黑名单(预编译正则)
# ---------------------------------------------------------------------------
_BLACKLIST_PATTERNS: list[tuple[re.Pattern[str], str]] = [
(re.compile(r"this\s+clip", re.IGNORECASE), "this clip"),
(re.compile(r"the\s+video", re.IGNORECASE), "the video"),
(re.compile(r"this\s+video", re.IGNORECASE), "this video"),
(re.compile(r"this\s+scene", re.IGNORECASE), "this scene"),
(re.compile(r"the\s+clip", re.IGNORECASE), "the clip"),
(re.compile(r"this\s+footage", re.IGNORECASE), "this footage"),
(re.compile(r"the\s+footage", re.IGNORECASE), "the footage"),
(re.compile(r"this\s+segment", re.IGNORECASE), "this segment"),
(re.compile(r"this\s+frame", re.IGNORECASE), "this frame"),
(re.compile(r"the\s+current\s+frame", re.IGNORECASE), "the current frame"),
(re.compile(r"frame\s+summary", re.IGNORECASE), "frame summary"),
(re.compile(r"上面的片段", re.IGNORECASE), "上面的片段"),
(re.compile(r"这段视频", re.IGNORECASE), "这段视频"),
(re.compile(r"该视频", re.IGNORECASE), "该视频"),
(re.compile(r"这个片段", re.IGNORECASE), "这个片段"),
(re.compile(r"视频中", re.IGNORECASE), "视频中"),
]
# ---------------------------------------------------------------------------
# 时间锚点正则与短语
# ---------------------------------------------------------------------------
_TIMESTAMP_PATTERN = re.compile(r"\d{1,2}:\d{2}")
_TIME_ANCHOR_PHRASES: list[re.Pattern[str]] = [
re.compile(r"at\s+the\s+beginning", re.IGNORECASE),
re.compile(r"at\s+the\s+end", re.IGNORECASE),
re.compile(r"in\s+the\s+beginning", re.IGNORECASE),
re.compile(r"at\s+the\s+start", re.IGNORECASE),
re.compile(r"开头", re.IGNORECASE),
re.compile(r"结尾", re.IGNORECASE),
re.compile(r"末尾", re.IGNORECASE),
re.compile(r"片头", re.IGNORECASE),
re.compile(r"片尾", re.IGNORECASE),
]
# ---------------------------------------------------------------------------
# T1 / T7 素材禁区正则
# ---------------------------------------------------------------------------
# T1: 瞬时动作
_T1_INSTANT_ACTION_PATTERNS: list[re.Pattern[str]] = [
re.compile(r"瞬间|一闪而过|转瞬即逝|一瞬间|刹那", re.IGNORECASE),
re.compile(r"flash|instant|split\s*second|blink", re.IGNORECASE),
]
# T1: 记分牌时序
_T1_SCOREBOARD_PATTERNS: list[re.Pattern[str]] = [
re.compile(r"记分牌|比分.*变|比分.*更新|得分.*变化", re.IGNORECASE),
re.compile(r"scoreboard|score\s*(changed|updated|went)", re.IGNORECASE),
]
# T1: 无对白因果
_T1_NO_DIALOGUE_CAUSAL_PATTERNS: list[re.Pattern[str]] = [
re.compile(r"无对白.*因果|因果.*无法.*判断", re.IGNORECASE),
re.compile(r"no\s+dialogue.*caus|cannot.*determin.*caus", re.IGNORECASE),
]
# T7: 选项重复
_T7_OPTION_REPETITION_PATTERNS: list[re.Pattern[str]] = [
re.compile(r"重复选项|选项.*相同|选项.*重复|完全相同", re.IGNORECASE),
re.compile(r"duplicate\s+option|identical\s+option|same\s+option", re.IGNORECASE),
]
# T7: 计数边界口径含糊
_T7_COUNTING_AMBIGUITY_PATTERNS: list[re.Pattern[str]] = [
re.compile(r"大约.*数量|数量不确定|难以.*计数|不确定.*几|约.*个", re.IGNORECASE),
re.compile(r"approximate.*count|uncertain.*number|hard\s+to\s+count", re.IGNORECASE),
]
# ---------------------------------------------------------------------------
# 公开函数
# ---------------------------------------------------------------------------
def shuffle_options(
options: tuple[str, ...],
answer: str,
rng: random.Random,
) -> tuple[tuple[str, ...], str]:
"""确定性洗牌选项并重映射答案字母。
通过 index permutation 打乱选项顺序为每个选项重新分配 A/B/C/D 前缀
并将答案字母映射到正确选项的新位置
参数:
options: 原始选项元组格式为 ("A. text1", "B. text2", ...)
answer: 原始答案字母"A"/"B"/"C"/"D"
rng: 可控随机数生成器保证确定性
返回:
(新选项元组, 新答案字母) 选项文本不变仅前缀和顺序改变
"""
# Phase 1: 提取纯文本(去掉 "X. " 前缀)
texts = [opt[3:] for opt in options]
# Phase 2: 确定原正确选项的文本
correct_idx = ord(answer) - ord("A")
correct_text = texts[correct_idx]
# Phase 3: 生成随机排列
indices = list(range(len(texts)))
rng.shuffle(indices)
# Phase 4: 按排列重组,分配新前缀
new_options: list[str] = []
new_answer = ""
for new_pos, old_idx in enumerate(indices):
letter = chr(ord("A") + new_pos)
new_options.append(f"{letter}. {texts[old_idx]}")
if texts[old_idx] == correct_text:
new_answer = letter
return tuple(new_options), new_answer
def check_referent_blacklist(question_text: str) -> list[str]:
"""检测题目文本中的自指代语言。
benchmark 题目中自指代 "this clip""这段视频"会泄露视频上下文
使题目脱离视频后无法独立理解
参数:
question_text: 题目文本
返回:
违规描述列表空列表表示通过
"""
violations: list[str] = []
for pattern, label in _BLACKLIST_PATTERNS:
if pattern.search(question_text):
violations.append(f"检测到自指代: '{label}'")
return violations
def _ngram_overlap_ratio(
text_words: list[str],
source_ngrams: set[tuple[str, ...]],
window: int,
) -> float:
"""计算单段文本与来源 n-gram 集合的重叠率。
参数:
text_words: 待比较文本的分词列表已小写化
source_ngrams: 来源素材的 n-gram 集合
window: n-gram 窗口大小
返回:
重叠率 [0.0, 1.0]词数不足窗口大小时返回 0.0
"""
if len(text_words) < window:
return 0.0
text_ngrams: set[tuple[str, ...]] = {
tuple(text_words[i : i + window]) for i in range(len(text_words) - window + 1)
}
if not text_ngrams:
return 0.0
overlap = text_ngrams & source_ngrams
return len(overlap) / len(text_ngrams)
def check_verbatim(
question_text: str,
correct_option: str,
source_texts: list[str],
window: int = 6,
) -> float:
"""计算题目文本和正确选项与来源素材的最大逐字重复率。
使用滑动窗口 n-gram 集合交集方法分别从题目文本和选项文本提取 n-gram
各自与来源文本的 n-gram 集合求交集返回两者中较大的重叠比例
参数:
question_text: 题目文本
correct_option: 正确选项文本 "X. " 前缀
source_texts: 来源素材文本列表
window: n-gram 窗口大小
返回:
最大重复率 [0.0, 1.0]0.0 表示无重叠1.0 表示完全复制
"""
# Phase 1: 构造来源文本的 n-gram 集合
source_ngrams: set[tuple[str, ...]] = set()
for source in source_texts:
words = source.lower().split()
for i in range(len(words) - window + 1):
source_ngrams.add(tuple(words[i : i + window]))
if not source_ngrams:
return 0.0
# Phase 2: 计算题目文本的重叠率
question_words = question_text.lower().split()
question_ratio = _ngram_overlap_ratio(question_words, source_ngrams, window)
# Phase 3: 提取选项纯文本(去掉可能的 "X. " 前缀)
option_text = correct_option
if len(option_text) >= 3 and option_text[1] == "." and option_text[2] == " ":
option_text = option_text[3:]
option_words = option_text.lower().split()
option_ratio = _ngram_overlap_ratio(option_words, source_ngrams, window)
# Phase 4: 返回两者中较大的重叠率
return max(question_ratio, option_ratio)
def has_time_anchor(question_text: str) -> bool:
"""检测题目中是否包含时间锚点。
时间锚点包括数字时间戳 "01:30"和时间短语 "at the beginning""开头"
参数:
question_text: 题目文本
返回:
True 表示包含时间锚点
"""
# Phase 1: 检查数字时间戳
if _TIMESTAMP_PATTERN.search(question_text):
return True
# Phase 2: 检查时间短语
return any(pattern.search(question_text) for pattern in _TIME_ANCHOR_PHRASES)
def _match_any(text: str, patterns: list[re.Pattern[str]]) -> bool:
"""判断文本是否匹配任一正则模式。
参数:
text: 待检测文本
patterns: 预编译正则列表
返回:
True 表示至少命中一条模式
"""
return any(p.search(text) for p in patterns)
# 素材禁区规则表:(模式列表, 违规描述)
_FORBIDDEN_MATERIAL_RULES: list[tuple[list[re.Pattern[str]], str]] = [
(_T1_INSTANT_ACTION_PATTERNS, "T1 违规: 素材包含瞬时动作描述,不适合出题"),
(_T1_SCOREBOARD_PATTERNS, "T1 违规: 素材包含记分牌时序信息,不适合出题"),
(_T1_NO_DIALOGUE_CAUSAL_PATTERNS, "T1 违规: 素材缺乏对白因果线索,不适合出题"),
(_T7_OPTION_REPETITION_PATTERNS, "T7 违规: 素材暗示可能产生重复选项"),
(_T7_COUNTING_AMBIGUITY_PATTERNS, "T7 违规: 素材包含计数边界含糊描述"),
]
def check_forbidden_material(source_nodes_text: str, task_type: str) -> list[str]:
"""出题禁区:检测 T1 类素材和 T7 噪声模式。
T1 类素材不适合出题的内容:
- 瞬时动作画面一闪而过无法稳定观察
- 记分牌时序依赖数字变化的时序信息
- 无对白因果缺乏语言线索的因果推理
T7 噪声模式选项质量问题:
- 选项重复多个选项表述相同
- 计数边界口径含糊数量描述不确定
参数:
source_nodes_text: 来源节点的拼接文本
task_type: 题型名称用于上下文感知检测
返回:
违规描述列表空列表表示通过
"""
return [
message
for patterns, message in _FORBIDDEN_MATERIAL_RULES
if _match_any(source_nodes_text, patterns)
]
def run_postprocess(
question_text: str,
options: tuple[str, ...],
answer: str,
source_texts: list[str],
rng: random.Random,
) -> PostprocessResult:
"""编排全部后处理检查,返回汇总结果。
执行流程:
1. 选项洗牌确定性
2. 指代黑名单检测
3. 逐字重复率计算基于洗牌后的正确选项
4. 时间锚点检测
参数:
question_text: 题目文本
options: 原始选项元组"A. text", "B. text", ...
answer: 原始答案字母
source_texts: 来源素材文本列表
rng: 可控随机数生成器
返回:
PostprocessResult 汇总实例
"""
# Phase 1: 选项洗牌
shuffled_options, new_answer = shuffle_options(options, answer, rng)
# Phase 2: 指代黑名单
referent_violations = check_referent_blacklist(question_text)
# Phase 3: 逐字重复率(用洗牌后正确选项的文本)
correct_idx = ord(new_answer) - ord("A")
correct_option_text = shuffled_options[correct_idx]
verbatim_ratio = check_verbatim(question_text, correct_option_text, source_texts)
# Phase 4: 时间锚点
time_anchor = has_time_anchor(question_text)
logger.debug(
"后处理完成: referent_violations={}, verbatim_ratio={:.3f}, has_time_anchor={}",
len(referent_violations),
verbatim_ratio,
time_anchor,
)
return PostprocessResult(
options=shuffled_options,
answer=new_answer,
referent_violations=referent_violations,
verbatim_ratio=verbatim_ratio,
has_time_anchor=time_anchor,
)
File diff suppressed because it is too large Load Diff
+597
View File
@@ -0,0 +1,597 @@
"""v2 素材采样器 — 基于采样约束的树节点采样与上下文收集。
v1 synthesizer 的基础上引入 SamplingConstraint 约束验证
为每次出题提供更丰富的素材上下文字幕 L2 上下文帧路径
典型调用路径::
material = sample_material_v2(
tree=tree_index,
task_type="Action Reasoning",
used_node_ids=already_used,
rng=rng,
level=2,
constraint=my_constraint,
)
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
import random
from app.question_gen.families import SamplingConstraint
from app.tree.index import L1Node, L2Node, TreeIndex
# ---------------------------------------------------------------------------
# 题型 → 采样层级映射
# ---------------------------------------------------------------------------
_TASK_TYPE_TO_LEVEL: dict[str, int] = {
# Level 3(细粒度帧级)
"Action Recognition": 3,
"Object Recognition": 3,
"Attribute Perception": 3,
"OCR Problems": 3,
# Level 2(片段/事件级)
"Action Reasoning": 2,
"Object Reasoning": 2,
"Information Synopsis": 2,
"Counting Problem": 2,
# Level 1(段落/场景级)
"Temporal Reasoning": 1,
"Temporal Perception": 1,
"Spatial Reasoning": 1,
"Spatial Perception": 1,
}
# ---------------------------------------------------------------------------
# 数据类型
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class AnchorContext:
"""采样锚点上下文。
属性:
node_id: 锚节点 ID
level: 锚节点所在层级1/2/3
l2_id: 锚节点所属的 L2 节点 ID若自身为 L2 则等于 node_id
若为 L1 则取其第一个 L2 子节点 ID
"""
node_id: str
level: int
l2_id: str
@dataclass(frozen=True)
class MaterialContext:
"""采样素材上下文 — 出题所需的全部素材打包。
属性:
anchor: 采样锚点信息
source_nodes: 参与采样的节点 ID 元组
subtitle_sentences: 锚节点子树中收集的字幕句列表
frame_paths: 锚节点子树中可用的帧路径列表
cross_l2_texts: L2 段的上下文文本列表 cross_l2_span 时填充
"""
anchor: AnchorContext
source_nodes: tuple[str, ...]
subtitle_sentences: list[str]
frame_paths: list[str]
cross_l2_texts: list[str]
# ---------------------------------------------------------------------------
# 内部索引辅助
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class _ResolvedSubtree:
"""节点解析结果 — 将任意层级 node_id 统一解析为 L2 节点列表 + 父 L1。
属性:
l2_nodes: 与目标节点关联的 L2 节点列表
parent_l1: L1 节点用于跨 L2 判断
"""
l2_nodes: list[L2Node]
parent_l1: L1Node | None
def _find_l2_node(tree: TreeIndex, l2_id: str) -> tuple[L2Node, L1Node] | None:
"""按 ID 定位 L2 节点及其父 L1。
参数:
tree: 三层树索引
l2_id: L2 节点 ID
返回:
(L2Node, 父L1Node) 元组未找到返回 None
"""
for l1 in tree.roots:
for l2 in l1.children:
if l2.id == l2_id:
return (l2, l1)
return None
def _find_l1_node(tree: TreeIndex, l1_id: str) -> L1Node | None:
"""按 ID 定位 L1 节点。
参数:
tree: 三层树索引
l1_id: L1 节点 ID
返回:
L1Node未找到返回 None
"""
for l1 in tree.roots:
if l1.id == l1_id:
return l1
return None
def _find_l3_parent(tree: TreeIndex, l3_id: str) -> tuple[L2Node, L1Node] | None:
"""按 L3 ID 定位其所属的 L2 和 L1 节点。
参数:
tree: 三层树索引
l3_id: L3 节点 ID
返回:
(L2Node, L1Node) 元组未找到返回 None
"""
for l1 in tree.roots:
for l2 in l1.children:
for l3 in l2.children:
if l3.id == l3_id:
return (l2, l1)
return None
def _resolve_subtree(tree: TreeIndex, node_id: str) -> _ResolvedSubtree | None:
"""将任意层级节点 ID 解析为关联的 L2 节点列表和父 L1。
参数:
tree: 三层树索引
node_id: 任意层级的节点 ID
返回:
_ResolvedSubtree 实例节点不存在时返回 None
"""
# 尝试作为 L2
result = _find_l2_node(tree, node_id)
if result is not None:
l2_node, parent_l1 = result
return _ResolvedSubtree(l2_nodes=[l2_node], parent_l1=parent_l1)
# 尝试作为 L1
l1_node = _find_l1_node(tree, node_id)
if l1_node is not None:
return _ResolvedSubtree(l2_nodes=list(l1_node.children), parent_l1=l1_node)
# 尝试作为 L3
l3_result = _find_l3_parent(tree, node_id)
if l3_result is not None:
l2_node, parent_l1 = l3_result
return _ResolvedSubtree(l2_nodes=[l2_node], parent_l1=parent_l1)
return None
# ---------------------------------------------------------------------------
# 约束检查辅助(单一职责)
# ---------------------------------------------------------------------------
def _count_l3_descendants(l2_nodes: list[L2Node]) -> int:
"""统计 L2 节点列表下的 L3 总数。
参数:
l2_nodes: L2 节点列表
返回:
L3 节点总数
"""
return sum(len(l2.children) for l2 in l2_nodes)
def _has_frames(l2_nodes: list[L2Node]) -> bool:
"""检查 L2 节点列表的子树中是否有可用帧。
参数:
l2_nodes: L2 节点列表
返回:
True 表示至少有一个 L3 frame_path
"""
return any(l3.frame_path for l2 in l2_nodes for l3 in l2.children)
def _count_subtitles(l2_nodes: list[L2Node]) -> int:
"""统计 L2 节点列表中全部字幕数(L2 + L3)。
参数:
l2_nodes: L2 节点列表
返回:
非空字幕总数
"""
count = 0
for l2 in l2_nodes:
if l2.card.subtitle:
count += 1
count += sum(1 for l3 in l2.children if l3.card.subtitle)
return count
# ---------------------------------------------------------------------------
# 公开辅助函数
# ---------------------------------------------------------------------------
def _validate_sampling_constraints(
tree: TreeIndex, node_id: str, constraint: SamplingConstraint
) -> bool:
"""校验指定节点是否满足采样约束。
根据节点层级自动判断检查范围
- L2 节点检查其子 L3 的帧/字幕数量
- L1 节点检查其下全部 L2/L3 的帧/字幕总数
- L3 节点检查其所属 L2 的子树
参数:
tree: 三层树索引
node_id: 待检查节点 ID
constraint: 采样约束条件
返回:
True 表示满足所有约束False 表示至少一项不满足
"""
resolved = _resolve_subtree(tree, node_id)
if resolved is None:
return False
if _count_l3_descendants(resolved.l2_nodes) < constraint.min_l3_nodes:
return False
if constraint.require_frames and not _has_frames(resolved.l2_nodes):
return False
if _count_subtitles(resolved.l2_nodes) < constraint.min_subtitles:
return False
return not (
constraint.cross_l2_span
and (resolved.parent_l1 is None or len(resolved.parent_l1.children) < 2)
)
def _subtitles_from_l2_list(l2_nodes: list[L2Node]) -> list[str]:
"""从 L2 节点列表收集全部非空字幕。
参数:
l2_nodes: L2 节点列表
返回:
非空字幕字符串列表
"""
sentences: list[str] = []
for l2 in l2_nodes:
if l2.card.subtitle:
sentences.append(l2.card.subtitle)
for l3 in l2.children:
if l3.card.subtitle:
sentences.append(l3.card.subtitle)
return sentences
def _collect_subtitle_sentences(tree: TreeIndex, node_ids: tuple[str, ...]) -> list[str]:
"""从指定节点集合中收集字幕句。
遍历每个 node_id 对应的子树提取非空字幕
L2 节点提取自身 + L3 字幕 L1 提取下属全部
参数:
tree: 三层树索引
node_ids: 待收集字幕的节点 ID 元组
返回:
非空字幕句列表
"""
sentences: list[str] = []
for nid in node_ids:
resolved = _resolve_subtree(tree, nid)
if resolved is not None:
sentences.extend(_subtitles_from_l2_list(resolved.l2_nodes))
return sentences
def _collect_cross_l2_context(tree: TreeIndex, anchor_l2_id: str, max_peers: int = 3) -> list[str]:
"""收集锚 L2 的同级 L2 节点描述文本(跨 L2 上下文)。
找到锚 L2 所属的 L1 父节点取该父节点下除锚 L2 之外的其他 L2 描述
参数:
tree: 三层树索引
anchor_l2_id: L2 节点 ID
max_peers: 最多返回的同级 L2 描述数量
返回:
同级 L2 event_description 列表最多 max_peers
"""
result = _find_l2_node(tree, anchor_l2_id)
if result is None:
return []
_, parent_l1 = result
peers: list[str] = []
for l2 in parent_l1.children:
if l2.id != anchor_l2_id:
peers.append(l2.card.event_description)
if len(peers) >= max_peers:
break
return peers
# ---------------------------------------------------------------------------
# 层级采样策略
# ---------------------------------------------------------------------------
def _sample_l3_node(
tree: TreeIndex,
used_node_ids: set[str],
rng: random.Random,
) -> tuple[str, str] | None:
"""随机采样一个未使用的 L3 节点,返回 (l3_id, 所属l2_id)。
参数:
tree: 三层树索引
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
(l3_id, l2_id) 元组无候选返回 None
"""
candidates: list[tuple[str, str]] = []
for l1 in tree.roots:
for l2 in l1.children:
for l3 in l2.children:
if l3.id not in used_node_ids:
candidates.append((l3.id, l2.id))
if not candidates:
return None
return rng.choice(candidates)
def _sample_l2_node(
tree: TreeIndex,
used_node_ids: set[str],
rng: random.Random,
) -> tuple[str, str] | None:
"""随机采样一个未使用的 L2 节点,返回 (l2_id, l2_id)。
参数:
tree: 三层树索引
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
(l2_id, l2_id) 元组无候选返回 None
"""
candidates: list[str] = []
for l1 in tree.roots:
for l2 in l1.children:
if l2.id not in used_node_ids:
candidates.append(l2.id)
if not candidates:
return None
chosen = rng.choice(candidates)
return (chosen, chosen)
def _sample_l1_node(
tree: TreeIndex,
used_node_ids: set[str],
rng: random.Random,
) -> tuple[str, str] | None:
"""随机采样一个未使用的 L1 节点,返回 (l1_id, 首个子l2_id)。
参数:
tree: 三层树索引
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
(l1_id, first_l2_id) 元组无候选返回 None
"""
candidates: list[tuple[str, str]] = []
for l1 in tree.roots:
if l1.id not in used_node_ids and l1.children:
candidates.append((l1.id, l1.children[0].id))
if not candidates:
return None
return rng.choice(candidates)
def _frames_from_l2_list(l2_nodes: list[L2Node]) -> list[str]:
"""从 L2 节点列表收集全部可用帧路径。
参数:
l2_nodes: L2 节点列表
返回:
帧路径字符串列表
"""
return [l3.frame_path for l2 in l2_nodes for l3 in l2.children if l3.frame_path]
def _collect_frame_paths(tree: TreeIndex, node_id: str) -> list[str]:
"""收集节点子树下的所有可用帧路径。
参数:
tree: 三层树索引
node_id: 目标节点 ID
返回:
帧路径列表
"""
resolved = _resolve_subtree(tree, node_id)
if resolved is None:
return []
return _frames_from_l2_list(resolved.l2_nodes)
def _collect_source_nodes(tree: TreeIndex, node_id: str) -> tuple[str, ...]:
"""收集节点子树涉及的全部节点 ID(包含自身)。
参数:
tree: 三层树索引
node_id: 目标节点 ID
返回:
相关节点 ID 元组
"""
ids: list[str] = [node_id]
# L2 节点:加入子 L3
result = _find_l2_node(tree, node_id)
if result is not None:
l2_node, _ = result
for l3 in l2_node.children:
ids.append(l3.id)
return tuple(ids)
# L1 节点:加入子 L2 + L3
l1_node = _find_l1_node(tree, node_id)
if l1_node is not None:
for l2 in l1_node.children:
ids.append(l2.id)
for l3 in l2.children:
ids.append(l3.id)
return tuple(ids)
# L3 节点:仅自身
return tuple(ids)
# ---------------------------------------------------------------------------
# 主入口
# ---------------------------------------------------------------------------
def sample_material_v2(
tree: TreeIndex,
task_type: str,
used_node_ids: set[str],
rng: random.Random,
*,
level: int,
constraint: SamplingConstraint,
max_attempts: int = 10,
) -> MaterialContext:
"""基于采样约束从视频树中采样素材上下文。
采样流程
1. 按指定 level 确定采样层级
2. 随机选取候选节点排除 used_node_ids
3. 验证 SamplingConstraint 约束
4. 约束不满足则重试最多 max_attempts
5. 收集字幕帧路径 L2 上下文
参数:
tree: 三层树索引
task_type: 任务类型字符串
used_node_ids: 本轮已用节点 ID 集合
rng: 可控随机数生成器
level: 采样层级1/2/3
constraint: 采样约束条件
max_attempts: 最大尝试次数
返回:
MaterialContext 实例
异常:
ValueError: level 不在 {1, 2, 3}
RuntimeError: 耗尽 max_attempts 次尝试仍无法满足约束
"""
if level not in (1, 2, 3):
msg = f"level 必须为 1、2 或 3,收到: {level}"
raise ValueError(msg)
for attempt in range(max_attempts):
# Phase 1: 按层级采样候选节点
if level == 3:
sampled = _sample_l3_node(tree, used_node_ids, rng)
elif level == 2:
sampled = _sample_l2_node(tree, used_node_ids, rng)
else:
sampled = _sample_l1_node(tree, used_node_ids, rng)
if sampled is None:
logger.debug(
"sample_material_v2 尝试 {}/{}: 无可用候选节点 (level={})",
attempt + 1,
max_attempts,
level,
)
continue
node_id, l2_id = sampled
# Phase 2: 验证约束
if not _validate_sampling_constraints(tree, node_id, constraint):
logger.debug(
"sample_material_v2 尝试 {}/{}: 约束违反 (node={})",
attempt + 1,
max_attempts,
node_id,
)
continue
# Phase 3: 构造 AnchorContext
anchor = AnchorContext(node_id=node_id, level=level, l2_id=l2_id)
# Phase 4: 收集素材
source_nodes = _collect_source_nodes(tree, node_id)
subtitle_sentences = _collect_subtitle_sentences(tree, (node_id,))
frame_paths = _collect_frame_paths(tree, node_id)
# Phase 5: 跨 L2 上下文(仅 cross_l2_span 时收集)
cross_l2_texts: list[str] = []
if constraint.cross_l2_span:
cross_l2_texts = _collect_cross_l2_context(tree, l2_id)
logger.debug(
"sample_material_v2 成功: node={}, level={}, attempt={}/{}",
node_id,
level,
attempt + 1,
max_attempts,
)
return MaterialContext(
anchor=anchor,
source_nodes=source_nodes,
subtitle_sentences=subtitle_sentences,
frame_paths=frame_paths,
cross_l2_texts=cross_l2_texts,
)
raise RuntimeError(
f"sample_material_v2: 耗尽 max_attempts={max_attempts} 次尝试,"
f"无法为 task_type='{task_type}' (level={level}) 满足采样约束"
)
+262
View File
@@ -0,0 +1,262 @@
"""题型出题策略 — Clean Architecture 的 Strategy 层。
将出题管线从 5 个粗粒度 QuestionFamilySpec 替换为 12 个题型级别的
TaskTypeStrategyBaseTaskTypeStrategy 封装现有 family 行为
特化策略 ActionRecognitionStrategy自包含
典型用法::
strategy = get_strategy("Action Reasoning")
level = strategy.sampling_level
constraint = strategy.sampling_constraint
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from app.question_gen.families import (
ENUMERATION_FAMILY,
REASONING_FAMILY,
RETRIEVAL_FAMILY,
SPATIAL_FAMILY,
VISUAL_FAMILY,
QuestionFamilySpec,
SamplingConstraint,
)
from app.question_gen.sampler_v2 import _TASK_TYPE_TO_LEVEL
if TYPE_CHECKING:
import random
# ---------------------------------------------------------------------------
# SubPattern
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class SubPattern:
"""出题子模式 — 靶向特定失败机制。
属性:
name: 子模式标识名
weight: 选择权重按比例分配
sampling_level_override: 覆盖策略默认采样层级None = 不覆盖
constraint_override: 覆盖策略默认采样约束None = 不覆盖
instruction: 核心出题指令1-3 句话
positive_examples: VME 原题 few-shot 示范
negative_examples: 反面示例
distractor_rules: 干扰项构造规则
supports_flip: 是否支持配对翻转门Phase B 默认 False
flip_axis: 翻转轴"before/after" | "first/last"None 表示不翻转
"""
name: str
weight: float
sampling_level_override: int | None
constraint_override: SamplingConstraint | None
instruction: str
positive_examples: list[dict] = field(default_factory=list)
negative_examples: list[dict] = field(default_factory=list)
distractor_rules: str = ""
supports_flip: bool = False
flip_axis: str | None = None
# ---------------------------------------------------------------------------
# Protocol
# ---------------------------------------------------------------------------
@runtime_checkable
class TaskTypeStrategy(Protocol):
"""题型出题策略 — pipeline 的唯一接口。"""
@property
def task_type(self) -> str: ...
@property
def sampling_level(self) -> int: ...
@property
def sampling_constraint(self) -> SamplingConstraint: ...
@property
def prompt_template(self) -> str: ...
@property
def strategy_name(self) -> str: ...
@property
def skill_target(self) -> str: ...
@property
def leak_probe_template(self) -> str: ...
@property
def uses_grounded_selector(self) -> bool: ...
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None: ...
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict: ...
def extra_gates(self, candidate: Any) -> list: ...
# ---------------------------------------------------------------------------
# BaseTaskTypeStrategy
# ---------------------------------------------------------------------------
# 消歧绑定表:多归属题型确定性绑定到一个 family
_TASK_TYPE_TO_FAMILY: dict[str, QuestionFamilySpec] = {
"Action Recognition": VISUAL_FAMILY, # fallback — 注册表中已被 ActionRecognitionStrategy 替换
"Object Recognition": RETRIEVAL_FAMILY,
"Object Reasoning": REASONING_FAMILY,
"Action Reasoning": REASONING_FAMILY,
"Attribute Perception": VISUAL_FAMILY,
"OCR Problems": VISUAL_FAMILY,
"Counting Problem": ENUMERATION_FAMILY,
"Information Synopsis": REASONING_FAMILY,
"Temporal Reasoning": ENUMERATION_FAMILY,
"Temporal Perception": ENUMERATION_FAMILY,
"Spatial Reasoning": SPATIAL_FAMILY,
"Spatial Perception": SPATIAL_FAMILY,
}
class BaseTaskTypeStrategy:
"""封装现有 QuestionFamilySpec 行为的默认策略。
所有属性委托给绑定的 family确保未特化题型的行为不变
参数:
task_type: 题型名
family: 绑定的 QuestionFamilySpec
"""
def __init__(self, task_type: str, family: QuestionFamilySpec) -> None:
self._task_type = task_type
self._family = family
@property
def task_type(self) -> str:
"""返回题型名。"""
return self._task_type
@property
def sampling_level(self) -> int:
"""返回采样层级(从 _TASK_TYPE_TO_LEVEL 查询)。"""
return _TASK_TYPE_TO_LEVEL[self._task_type]
@property
def sampling_constraint(self) -> SamplingConstraint:
"""返回采样约束(委托给绑定的 family)。"""
return self._family.sampling
@property
def prompt_template(self) -> str:
"""返回 prompt 模板文件名(委托给绑定的 family)。"""
return self._family.prompt_template
@property
def strategy_name(self) -> str:
"""返回策略名(即 family.name)。"""
return self._family.name
@property
def skill_target(self) -> str:
"""返回目标失败机制编号(委托给绑定的 family)。"""
return self._family.skill_target
@property
def leak_probe_template(self) -> str:
"""返回泄漏探测模板文件名(委托给绑定的 family.leak_profile)。"""
return self._family.leak_profile.probe_template
@property
def uses_grounded_selector(self) -> bool:
"""默认不启用 grounded selector11 类题型走原路径)。"""
return False
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None:
"""BaseTaskTypeStrategy 无子模式。"""
return None
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict:
"""返回基础 prompt 上下文(family_name + prompt_template)。"""
return {
"family_name": self._family.name,
"prompt_template": self._family.prompt_template,
}
def extra_gates(self, candidate: Any) -> list:
"""BaseTaskTypeStrategy 无额外 gate。"""
return []
# ---------------------------------------------------------------------------
# 注册表
# ---------------------------------------------------------------------------
_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {}
def register_strategy(strategy: TaskTypeStrategy) -> None:
"""注册一个题型策略。同一 task_type 重复注册会覆盖。
参数:
strategy: 实现 TaskTypeStrategy 接口的策略实例
"""
_STRATEGY_REGISTRY[strategy.task_type] = strategy
_BUILTIN_REGISTERED = False
def get_strategy(task_type: str) -> TaskTypeStrategy:
"""获取题型策略。未注册的自动创建 BaseTaskTypeStrategy。
首次调用时延迟注册内建特化策略避免循环导入
参数:
task_type: 题型名
返回:
TaskTypeStrategy 实例
异常:
KeyError: task_type 不在消歧绑定表和注册表中
"""
global _BUILTIN_REGISTERED # noqa: PLW0603
if not _BUILTIN_REGISTERED:
_BUILTIN_REGISTERED = True
_register_builtin_strategies()
if task_type in _STRATEGY_REGISTRY:
return _STRATEGY_REGISTRY[task_type]
return _build_default_strategy(task_type)
def _build_default_strategy(task_type: str) -> BaseTaskTypeStrategy:
"""根据消歧绑定表自动构造 BaseTaskTypeStrategy。
参数:
task_type: 题型名
返回:
BaseTaskTypeStrategy 实例
异常:
KeyError: task_type 不在消歧绑定表中
"""
family = _TASK_TYPE_TO_FAMILY[task_type]
return BaseTaskTypeStrategy(task_type=task_type, family=family)
def _register_builtin_strategies() -> None:
"""注册内建的特化策略。由 get_strategy 首次调用时延迟执行。"""
from app.question_gen.strategy_action_recognition import ActionRecognitionStrategy
register_strategy(ActionRecognitionStrategy())
@@ -0,0 +1,347 @@
"""Action Recognition 特化出题策略。
靶向 Agent 在动作识别类题目上的 6 种典型失败机制通过加权随机
SubPattern 选择为 VLM 出题提供聚焦指令
BaseTaskTypeStrategy 完全自包含不依赖 QuestionFamilySpec
采样从 L3 提升至 L2跨段推理需要更广视野3 SubPattern
进一步覆盖至 L1 以测试全局追踪能力
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from app.question_gen.families import SamplingConstraint
from app.question_gen.strategy import SubPattern
if TYPE_CHECKING:
import random
# ---------------------------------------------------------------------------
# 6 个 SubPattern 定义
# ---------------------------------------------------------------------------
_PREMATURE_EVIDENCE_ANCHORING = SubPattern(
name="premature_evidence_anchoring",
weight=0.20,
sampling_level_override=1,
constraint_override=None,
instruction=(
"设计一道需要考察视频中完整证据链的动作识别题。"
"正确答案的关键证据出现在视频后半段或跨多个片段,"
"但视频前段包含一个看似合理的局部匹配——"
"Agent 若仅凭首条匹配停止搜索就会出错。"
),
positive_examples=[
{
"question": "视频中厨师最终采用了哪种烹饪方式?",
"answer": "B. 蒸",
"why": "厨师先演示了炒(前段),但最终菜品使用蒸制(后段),"
"锚定首段证据的 Agent 会错选''",
},
],
negative_examples=[
{
"question": "视频开头厨师在做什么?",
"why": "答案仅需前段信息,不会触发过早锚定失败。",
},
],
distractor_rules=(
"将视频前段真实出现的局部匹配动作设为强干扰项——它真实发生,"
"仅在【时点】这一单一维度上与正解不同(前段 vs 最终结论段)。"
"其余干扰项亦须是视频中真实发生的动作,严禁缺席事件。"
),
)
_TEMPORAL_REASONING_FAILURE = SubPattern(
name="temporal_reasoning_failure",
weight=0.20,
sampling_level_override=1,
constraint_override=None,
instruction=(
"设计一道要求正确排序或定位第 N 次出现的动作识别题。"
"题目需要 Agent 追踪事件的时间先后顺序,"
"或准确识别某动作在视频中第几次出现。"
"打乱时序或错误计数即会答错。"
),
positive_examples=[
{
"question": "运动员第三次尝试起跳前做了什么准备动作?",
"answer": "C. 深蹲热身",
"why": "需要准确定位'第三次'起跳而非其他次,"
"时序推理失败的 Agent 会混淆不同次尝试的准备动作。",
},
],
negative_examples=[
{
"question": "运动员在视频中做了什么?",
"why": "不涉及时序排序或计数,Agent 无需追踪顺序。",
},
],
distractor_rules=(
"干扰项必须是视频中真实发生的事件,仅在【事件时序/顺序】这一单一维度上与正解不同——"
"即同一组真实事件的错误排列或错误的第 N 次定位。"
"严禁使用视频中未出现的缺席事件作为干扰项。"
),
supports_flip=True,
flip_axis="before/after",
)
_SEMANTIC_RIGIDITY = SubPattern(
name="semantic_rigidity",
weight=0.15,
sampling_level_override=None,
constraint_override=None,
instruction=(
"设计一道动作识别题,正确选项使用与视频原始描述不同的同义表达。"
"Agent 需理解语义等价而非依赖字面匹配——"
"例如视频字幕说'奔跑',正确选项写作'快速移动'"
),
positive_examples=[
{
"question": "工人对墙面进行了什么操作?",
"answer": "A. 涂覆保护层",
"why": "视频中字幕描述为'刷漆',正确答案改写为'涂覆保护层'"
"依赖字面匹配的 Agent 会因找不到完全一致的表述而错选。",
},
],
negative_examples=[
{
"question": "工人在刷漆吗?",
"why": "选项直接复用视频原文,不考察语义理解。",
},
],
distractor_rules=(
"干扰项须基于视频真实内容,仅在【表述/语义】这一单一维度上做文章:"
"保留一个复用视频原始字幕字面、但在题干限定下语义为假的选项作为陷阱,"
"其余选项描述真实动作的不同同义表述。严禁凭空编造缺席动作。"
),
)
_FINE_GRAINED_VISUAL_ACTION = SubPattern(
name="fine_grained_visual_action",
weight=0.15,
sampling_level_override=None,
constraint_override=None,
instruction=(
"设计一道需要区分细粒度动作方式的识别题。"
"题目聚焦于 HOW(怎么做)而非 WHAT(做什么),"
"如区分'搅拌''翻炒'''''等视觉上相似但方式不同的动作。"
),
positive_examples=[
{
"question": "维修人员是如何拆卸螺丝的?",
"answer": "D. 用扳手逆时针旋转",
"why": "需要区分拆卸的具体方式(扳手 vs 螺丝刀、顺时针 vs 逆时针),"
"粗粒度识别只能判断'在拆螺丝',无法区分方式。",
},
],
negative_examples=[
{
"question": "维修人员在做什么?",
"why": "只需粗粒度动作识别('拆螺丝'),不考察具体方式。",
},
],
distractor_rules=(
"四个选项须是【同一大类动作】的不同执行方式,全部为视频中真实可见的做法,"
"仅在【执行方式】这一单一维度上不同(如顺/逆时针、扳手/螺丝刀)。"
"严禁使用明显不相关或视频中未出现的动作作为干扰项。"
),
)
_CROSS_SEGMENT_ENTITY_TRACKING = SubPattern(
name="cross_segment_entity_tracking",
weight=0.15,
sampling_level_override=1,
constraint_override=None,
instruction=(
"设计一道需要跨视频段落追踪同一实体动作的识别题。"
"目标实体在不同片段中外观、称呼或上下文发生变化,"
"Agent 需要将多段信息合并才能正确回答关于该实体的动作问题。"
),
positive_examples=[
{
"question": "穿红色外套的人在视频中总共完成了哪些动作?",
"answer": "B. 先讲解、后示范、最后总结",
"why": "该人物在前段穿外套讲解,中段脱外套示范,后段重新穿上总结,"
"无法跨段追踪的 Agent 会遗漏某段动作。",
},
{
"question": "主持人在节目不同环节中分别做了什么?",
"answer": "A. 开场介绍、采访嘉宾、总结点评",
"why": "主持人在不同场景切换中持续出现,需要跨段聚合。",
},
],
negative_examples=[
{
"question": "视频第一个片段中的人在做什么?",
"why": "仅需单段信息,不考察跨段追踪。",
},
],
distractor_rules=(
"干扰项须是视频中【另一真实实体】在相应片段真实做过的动作,"
"仅在【动作主体】这一单一维度上与正解不同。"
"严禁编造任何实体未做过的缺席动作。"
),
supports_flip=True,
flip_axis="first/last",
)
_EVIDENCE_GAP_CONFABULATION = SubPattern(
name="evidence_gap_confabulation",
weight=0.15,
sampling_level_override=None,
constraint_override=None,
instruction=(
"设计一道动作识别题,视频中存在证据空缺(如遮挡、跳切、画外音)。"
"正确答案承认信息不足或基于间接证据推断,"
"而非凭空编造因果链。Agent 若虚构缺失证据即会出错。"
),
positive_examples=[
{
"question": "画面切走后,演讲者下一步做了什么?",
"answer": "C. 无法从视频中直接确定",
"why": "画面跳切导致该动作无直接视觉证据,"
"倾向于虚构的 Agent 会编造一个看似合理的动作。",
},
],
negative_examples=[
{
"question": "画面中演讲者正在做什么?",
"why": "动作在画面中可见,不存在证据空缺。",
},
],
distractor_rules=(
"正解仅陈述视频中可观测的事实或诚实承认证据不足;"
"干扰项在【因果完整性】这一单一维度上越界——补上一段视频未展示的因果链,"
"但其前提元素仍取自视频真实内容(诱导 Agent 顺势编造),而非完全凭空的缺席事件。"
),
)
AR_SUB_PATTERNS: tuple[SubPattern, ...] = (
_PREMATURE_EVIDENCE_ANCHORING,
_TEMPORAL_REASONING_FAILURE,
_SEMANTIC_RIGIDITY,
_FINE_GRAINED_VISUAL_ACTION,
_CROSS_SEGMENT_ENTITY_TRACKING,
_EVIDENCE_GAP_CONFABULATION,
)
# 预计算:名称列表 + 权重列表(避免每次 select 重新构建)
_AR_PATTERN_NAMES: list[str] = [sp.name for sp in AR_SUB_PATTERNS]
_AR_PATTERN_WEIGHTS: list[float] = [sp.weight for sp in AR_SUB_PATTERNS]
_AR_PATTERN_BY_NAME: dict[str, SubPattern] = {sp.name: sp for sp in AR_SUB_PATTERNS}
# ---------------------------------------------------------------------------
# ActionRecognitionStrategy
# ---------------------------------------------------------------------------
_SAMPLING_CONSTRAINT = SamplingConstraint(
min_subtitles=3,
min_l3_nodes=5,
require_frames=True,
cross_l2_span=True,
)
class ActionRecognitionStrategy:
"""Action Recognition 特化出题策略。
自包含实现不依赖 BaseTaskTypeStrategy QuestionFamilySpec
靶向 6 种典型失败机制通过加权随机 SubPattern 引导 VLM 出题
属性:
task_type: 题型名 "Action Recognition"
strategy_name: 策略标识 "ACTION_RECOGNITION"
skill_target: 目标失败机制 "M1_AR"
sampling_level: 采样层级 2L2 L3 升级以获得跨段视野
sampling_constraint: 采样约束min_subtitles=3, min_l3_nodes=5, require_frames, cross_l2_span
prompt_template: 出题模板 "action_recognition.md"
leak_probe_template: 泄漏探测模板 "gate_leak_retrieval.md"
"""
@property
def task_type(self) -> str:
"""返回题型名。"""
return "Action Recognition"
@property
def strategy_name(self) -> str:
"""返回策略标识名。"""
return "ACTION_RECOGNITION"
@property
def skill_target(self) -> str:
"""返回目标失败机制编号。"""
return "M1_AR"
@property
def sampling_level(self) -> int:
"""返回采样层级(L2)。"""
return 2
@property
def sampling_constraint(self) -> SamplingConstraint:
"""返回采样约束。"""
return _SAMPLING_CONSTRAINT
@property
def prompt_template(self) -> str:
"""返回出题 prompt 模板文件名。"""
return "action_recognition.md"
@property
def leak_probe_template(self) -> str:
"""返回泄漏探测模板文件名。"""
return "gate_leak_retrieval.md"
@property
def uses_grounded_selector(self) -> bool:
"""AR 启用候选池 + VLM 视觉打分 selector。"""
return True
def select_sub_pattern(self, rng: random.Random) -> SubPattern:
"""按权重随机选择一个 SubPattern。
参数:
rng: 随机数生成器确保可复现
返回:
选中的 SubPattern 实例永不为 None
"""
chosen_name = rng.choices(
_AR_PATTERN_NAMES,
weights=_AR_PATTERN_WEIGHTS,
k=1,
)[0]
return _AR_PATTERN_BY_NAME[chosen_name]
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict:
"""构建 prompt 上下文字典。
参数:
material: 采样素材当前未使用留给管线扩展
sub_pattern: 选中的子模式
返回:
包含 family_name, prompt_template, sub_pattern 的字典
"""
return {
"family_name": self.strategy_name,
"prompt_template": self.prompt_template,
"sub_pattern": sub_pattern.name if sub_pattern is not None else None,
}
def extra_gates(self, candidate: Any) -> list:
"""返回额外门控列表(当前为空)。
参数:
candidate: 候选题目
返回:
空列表
"""
return []
+756
View File
@@ -0,0 +1,756 @@
"""赛题合成核心逻辑 — 节点采样、prompt 构造、VLM 响应解析、去重。
纯函数为主异步编排仅 generate_one
通过 DI 接收 VLMProvider / EmbeddingProvider import adapters/
"""
from __future__ import annotations
import contextlib
import json
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING
import numpy as np
from loguru import logger
if TYPE_CHECKING:
import random
from collections.abc import Callable
from app.tree.index import L2Node, L3Node, TreeIndex
from core.protocols import VLMProvider
from core.types import GeneratedQuestion
@dataclass(frozen=True)
class AnchorContext:
"""锚节点上下文——生成单道题所需的全部素材。
属性:
node_id: 锚节点 ID
card_text: 锚节点 card 序列化文本
frame_paths: 帧图片路径列表
subtitle: 对应字幕可空
distractor_texts: 同视频其他节点摘要 VLM 生成干扰项
"""
node_id: str
card_text: str
frame_paths: list[str]
subtitle: str
distractor_texts: list[str]
@dataclass(frozen=True)
class TaskTypeSpec:
"""题型的生成规格。
属性:
level: 锚定层级"L3" / "L2" / "L1" / "L1-L2"
needs_frames: 是否必须提供帧图
frame_count: 帧数范围描述 "1", "2-3", "0-1"
context_fields: 需要提取的 card 字段元组
"""
level: str
needs_frames: bool
frame_count: str
context_fields: tuple[str, ...]
# ---------------------------------------------------------------------------
# 12 种 Video-MME 题型 → 树层级 + 生成规格映射
# ---------------------------------------------------------------------------
TASK_TYPE_LEVEL_MAP: dict[str, TaskTypeSpec] = {
# --- L3 单帧题型 ---
"Object Recognition": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
"Attribute Perception": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
"OCR Problems": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
"Spatial Reasoning": TaskTypeSpec("L3", True, "1", ("frame_summary", "spatial_layout")),
"Spatial Perception": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
# --- L2 多帧 / 事件级题型 ---
"Action Recognition": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
"Action Reasoning": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
"Counting Problem": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
"Temporal Perception": TaskTypeSpec("L2", False, "0-1", ("event_description", "time_range")),
# --- L1 / 跨层级题型 ---
"Temporal Reasoning": TaskTypeSpec("L1", True, "per-L2", ("scene_summary",)),
"Information Synopsis": TaskTypeSpec("L1", True, "per-L2", ("scene_summary",)),
"Object Reasoning": TaskTypeSpec("L1-L2", True, "per-L2", ("event_description",)),
}
# ---------------------------------------------------------------------------
# 内部辅助函数
# ---------------------------------------------------------------------------
def _serialize_l3_card(l3: L3Node, context_fields: tuple[str, ...]) -> str:
"""将 L3 节点 card 按 context_fields 序列化为可读文本。
参数:
l3: L3 节点
context_fields: 需提取的字段名元组
返回:
多行 "field: value" 格式的文本
"""
parts: list[str] = []
for fld in context_fields:
val = getattr(l3.card, fld, None)
if val is not None:
parts.append(f"{fld}: {val}")
return "\n".join(parts)
def _l2_time_range_str(l2: L2Node) -> str:
"""将 L2 的 time_range 格式化为可读字符串。
参数:
l2: L2 节点
返回:
"time_range: (start, end)" 格式 "time_range: unknown"
"""
if l2.time_range is not None:
return f"time_range: ({l2.time_range[0]:.2f}, {l2.time_range[1]:.2f})"
return "time_range: unknown"
def _representative_frame(l2: L2Node) -> str | None:
"""取 L2 的代表帧路径——第一个有 frame_path 的 L3 子节点。
参数:
l2: L2 节点
返回:
帧路径字符串 None无可用帧时
"""
for l3 in l2.children:
if l3.frame_path:
return l3.frame_path
return None
def _sort_l2_by_time(l2_nodes: list[L2Node]) -> list[L2Node]:
"""按 time_range 升序排列 L2 节点(None 排末尾)。
参数:
l2_nodes: 待排序的 L2 节点列表
返回:
排序后的新列表不修改原列表
"""
return sorted(
l2_nodes,
key=lambda n: n.time_range[0] if n.time_range is not None else float("inf"),
)
# ---------------------------------------------------------------------------
# 各层级采样策略
# ---------------------------------------------------------------------------
def _sample_l3(
tree: TreeIndex,
task_type: str,
spec: TaskTypeSpec,
used_node_ids: set[str],
rng: random.Random,
) -> AnchorContext:
"""L3 层级锚节点采样。
收集全部 L3 节点排除已用节点随机选取一个
参数:
tree: 三层树索引
task_type: 题型名称
spec: 题型规格
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
AnchorContext 实例
异常:
ValueError: 候选 L3 节点不足
"""
# Phase 1: 收集所有 L3 候选(必须有 frame_path
candidates: list[tuple[L3Node, L2Node]] = []
for root in tree.roots:
for l2 in root.children:
for l3 in l2.children:
if l3.id not in used_node_ids and l3.frame_path:
candidates.append((l3, l2))
if not candidates:
raise ValueError(f"锚节点不足: {task_type} 无可用 L3 节点(需具备 frame_path")
# Phase 2: 随机选取
chosen_l3, parent_l2 = rng.choice(candidates)
# Phase 3: 构造上下文(frame_path 已在候选过滤中保证非 None)
card_text = _serialize_l3_card(chosen_l3, spec.context_fields)
frame_paths = [chosen_l3.frame_path] # type: ignore[list-item]
subtitle = chosen_l3.card.subtitle or ""
# Phase 4: 干扰项——整棵树中其他 L3 的 frame_summary
distractor_texts = [
l3.card.frame_summary
for root in tree.roots
for l2 in root.children
for l3 in l2.children
if l3.id != chosen_l3.id
]
return AnchorContext(
node_id=chosen_l3.id,
card_text=card_text,
frame_paths=frame_paths,
subtitle=subtitle,
distractor_texts=distractor_texts,
)
def _sample_l2(
tree: TreeIndex,
task_type: str,
spec: TaskTypeSpec,
used_node_ids: set[str],
rng: random.Random,
) -> AnchorContext:
"""L2 层级锚节点采样(含 Temporal Perception 特殊处理)。
普通 L2 题型随机选 1 L2 2-3 个子 L3
Temporal Perception0-1 card_text 必含 time_range
参数:
tree: 三层树索引
task_type: 题型名称
spec: 题型规格
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
AnchorContext 实例
异常:
ValueError: 候选 L2 节点不足
"""
# Phase 1: 收集所有 L2 候选
all_l2: list[L2Node] = []
for root in tree.roots:
for l2 in root.children:
if l2.id not in used_node_ids:
all_l2.append(l2)
if not all_l2:
raise ValueError(f"锚节点不足: {task_type} 无可用 L2 节点")
# Phase 2: 随机选取
chosen_l2 = rng.choice(all_l2)
is_temporal_perception = task_type == "Temporal Perception"
# Phase 3: 帧路径
if is_temporal_perception:
# 0-1 帧:有子节点则取 1 帧,否则 0 帧
frame_paths: list[str] = []
if chosen_l2.children:
first_frame = chosen_l2.children[0].frame_path
if first_frame:
frame_paths = [first_frame]
else:
# 普通 L2:随机采样 2-3 个 L3 帧
children_with_frames = [l3 for l3 in chosen_l2.children if l3.frame_path]
if len(children_with_frames) < 2:
raise ValueError(
f"锚节点不足: {task_type} 需要 >=2 个子帧,"
f"{chosen_l2.id} 仅有 {len(children_with_frames)} 个可用帧"
)
n_frames = min(rng.randint(2, 3), len(children_with_frames))
sampled = rng.sample(children_with_frames, n_frames)
frame_paths = [l3.frame_path for l3 in sampled if l3.frame_path]
# Phase 4: card_text
card_text = f"event_description: {chosen_l2.card.event_description}"
if is_temporal_perception:
card_text += f"\n{_l2_time_range_str(chosen_l2)}"
# Phase 5: 字幕(优先使用 L2 自身字幕,否则取首个子 L3 字幕)
subtitle = chosen_l2.card.subtitle or ""
if not subtitle and chosen_l2.children and chosen_l2.children[0].card.subtitle:
subtitle = chosen_l2.children[0].card.subtitle
# Phase 6: 干扰项——整棵树中其他 L2 的 event_description
distractor_texts = [
l2.card.event_description
for root in tree.roots
for l2 in root.children
if l2.id != chosen_l2.id
]
return AnchorContext(
node_id=chosen_l2.id,
card_text=card_text,
frame_paths=frame_paths,
subtitle=subtitle,
distractor_texts=distractor_texts,
)
def _sample_l1(
tree: TreeIndex,
task_type: str,
spec: TaskTypeSpec,
used_node_ids: set[str],
rng: random.Random,
) -> AnchorContext:
"""L1 层级锚节点采样(Temporal Reasoning / Information Synopsis)。
Information Synopsis使用目标 L1 下全部 L2 子节点
Temporal Reasoning严格要求 >=3 L2 子节点不足则抛 ValueError
L2 time_range 升序排列每个 L2 取一帧代表
参数:
tree: 三层树索引
task_type: 题型名称
spec: 题型规格
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
AnchorContext 实例
异常:
ValueError: 候选 L1 节点不足 Temporal Reasoning L2 子节点 <3
"""
# Phase 1: 收集可用 L1
candidates = [r for r in tree.roots if r.id not in used_node_ids]
if not candidates:
raise ValueError(f"锚节点不足: {task_type} 无可用 L1 节点")
# Phase 2: 随机选取
chosen_l1 = rng.choice(candidates)
# Phase 3: 选定 L2 子集
if task_type == "Information Synopsis":
# 必须使用全部 L2
selected_l2 = list(chosen_l1.children)
else:
# Temporal Reasoning:严格要求 >=3 个 L2
if len(chosen_l1.children) < 3:
raise ValueError(
f"锚节点不足: {task_type} 需要 >=3 个 L2 子节点,"
f"{chosen_l1.id} 仅有 {len(chosen_l1.children)}"
)
if len(chosen_l1.children) == 3:
selected_l2 = list(chosen_l1.children)
else:
selected_l2 = rng.sample(chosen_l1.children, rng.randint(3, len(chosen_l1.children)))
# Phase 4: 按 time_range 升序排列
selected_l2 = _sort_l2_by_time(selected_l2)
# Phase 5: card_text(场景摘要)
card_text = f"scene_summary: {chosen_l1.card.scene_summary}"
# Phase 6: 帧路径——每个 L2 取一帧代表
frame_paths: list[str] = []
for l2 in selected_l2:
rep = _representative_frame(l2)
if rep:
frame_paths.append(rep)
# Phase 7: 字幕(L1 无字幕)
subtitle = ""
# Phase 8: 干扰项——其他 L1 的 scene_summary
distractor_texts = [r.card.scene_summary for r in tree.roots if r.id != chosen_l1.id]
return AnchorContext(
node_id=chosen_l1.id,
card_text=card_text,
frame_paths=frame_paths,
subtitle=subtitle,
distractor_texts=distractor_texts,
)
def _sample_l1_l2(
tree: TreeIndex,
task_type: str,
spec: TaskTypeSpec,
used_node_ids: set[str],
rng: random.Random,
) -> AnchorContext:
"""L1-L2 跨层级锚节点采样(Object Reasoning)。
从全部 L2 中随机选 2-3 time_range 排序
card_text 为各 L2 event_description 拼接
参数:
tree: 三层树索引
task_type: 题型名称
spec: 题型规格
used_node_ids: 已用节点 ID 集合
rng: 随机数生成器
返回:
AnchorContext 实例
异常:
ValueError: 候选 L2 节点不足
"""
# Phase 1: 收集全部 L2
all_l2: list[L2Node] = []
for root in tree.roots:
for l2 in root.children:
if l2.id not in used_node_ids:
all_l2.append(l2)
if len(all_l2) < 2:
raise ValueError(
f"锚节点不足: {task_type} 需要 >=2 个 L2 节点,但仅有 {len(all_l2)} 个可用"
)
# Phase 2: 随机选 2-3 个
n_pick = min(rng.randint(2, 3), len(all_l2))
selected = rng.sample(all_l2, n_pick)
# Phase 3: 按 time_range 升序排列
selected = _sort_l2_by_time(selected)
# Phase 4: card_text = 各 L2 event_description 拼接
card_text = "\n".join(f"event_description: {l2.card.event_description}" for l2 in selected)
# Phase 5: 帧路径——每个 L2 取一帧代表
frame_paths: list[str] = []
for l2 in selected:
rep = _representative_frame(l2)
if rep:
frame_paths.append(rep)
# Phase 6: 字幕
subtitle = ""
# Phase 7: 干扰项——未被选中的 L2 的 event_description
selected_ids = {l2.id for l2 in selected}
distractor_texts = [l2.card.event_description for l2 in all_l2 if l2.id not in selected_ids]
# 使用第一个被选中节点的 ID 作为锚节点 ID
anchor_id = selected[0].id
return AnchorContext(
node_id=anchor_id,
card_text=card_text,
frame_paths=frame_paths,
subtitle=subtitle,
distractor_texts=distractor_texts,
)
# ---------------------------------------------------------------------------
# 公开接口
# ---------------------------------------------------------------------------
def sample_anchor(
tree: TreeIndex,
task_type: str,
used_node_ids: set[str],
rng: random.Random,
) -> AnchorContext:
"""根据题型从视频树中采样锚节点及上下文素材。
依据 TASK_TYPE_LEVEL_MAP 中的层级规格分发到对应的层级采样策略
每种层级有不同的帧选取card 序列化和干扰项收集逻辑
参数:
tree: 三层树索引
task_type: 12 Video-MME 题型之一
used_node_ids: 本轮已用节点 ID 集合避免重复采样
rng: 可控随机数生成器保证可复现
返回:
AnchorContext 实例包含锚节点 IDcard 文本帧路径字幕和干扰项
异常:
KeyError: task_type 不在 TASK_TYPE_LEVEL_MAP
ValueError: 候选节点不足全部被 used_node_ids 排除
"""
spec = TASK_TYPE_LEVEL_MAP[task_type]
if spec.level == "L3":
return _sample_l3(tree, task_type, spec, used_node_ids, rng)
elif spec.level == "L2":
return _sample_l2(tree, task_type, spec, used_node_ids, rng)
elif spec.level == "L1":
return _sample_l1(tree, task_type, spec, used_node_ids, rng)
elif spec.level == "L1-L2":
return _sample_l1_l2(tree, task_type, spec, used_node_ids, rng)
else:
raise ValueError(f"未知层级: {spec.level}")
# ---------------------------------------------------------------------------
# Prompt 构造与 VLM 响应解析
# ---------------------------------------------------------------------------
_VALID_ANSWERS = frozenset({"A", "B", "C", "D"})
def build_generation_prompt(
task_type: str,
anchor: AnchorContext,
exemplars: list[GeneratedQuestion],
) -> tuple[list[dict[str, str]], list[str]]:
"""组装 VLM 出题 prompt。
构造 OpenAI 格式的 messages 列表和帧图片路径列表
VLMProvider.chat_with_images 直接消费
参数:
task_type: 题型名称 "Object Recognition"
anchor: 锚节点上下文card_text, subtitle, distractor_texts, frame_paths
exemplars: 少样本示例列表可为空
返回:
(messages, image_paths) messages OpenAI 格式消息列表
image_paths 为帧图片路径列表直接喂给 VLMProvider.chat_with_images
"""
# Phase 1: 构造 system message
system_parts: list[str] = [
"你是一个视频理解题目生成器。",
f"题型: {task_type}",
"约束:",
"- 题目必须基于提供的节点内容",
"- 干扰选项应来自其他节点的信息",
"- 生成风格应与示例保持一致",
'- 以 JSON 格式返回: {"question": "...", "options": ["A. ...", "B. ...", "C. ...", "D. ..."], "answer": "A/B/C/D"}',
]
# Phase 2: 加入 few-shot 示例
if exemplars:
system_parts.append("\n示例:")
for i, ex in enumerate(exemplars, 1):
system_parts.append(f" 示例 {i}:")
system_parts.append(f" question: {ex.question}")
system_parts.append(f" options: {list(ex.options)}")
system_parts.append(f" answer: {ex.answer}")
system_content = "\n".join(system_parts)
# Phase 3: 构造 user message
user_parts: list[str] = [f"节点内容:\n{anchor.card_text}"]
if anchor.subtitle:
user_parts.append(f"\n字幕:\n{anchor.subtitle}")
if anchor.distractor_texts:
user_parts.append("\n干扰项来源节点摘要:")
for dt in anchor.distractor_texts:
user_parts.append(f"- {dt}")
user_content = "\n".join(user_parts)
messages = [
{"role": "system", "content": system_content},
{"role": "user", "content": user_content},
]
return messages, list(anchor.frame_paths)
def parse_vlm_response(
raw: str,
video_id: str,
task_type: str,
seq: int,
) -> dict:
"""解析 VLM 返回的 JSON → 部分字段字典。
尝试直接解析 JSON若失败 markdown 代码块中提取后重试
校验必需字段选项数量和答案合法性
参数:
raw: VLM 原始返回文本
video_id: 所属视频标识
task_type: 题型名称用于错误消息
seq: 序列号用于生成 question_id
返回:
{"question_id": "gen-{video_id}-{seq:03d}", "question": ..., "options": [...], "answer": ...}
调用方generate_one补齐 source_nodes/difficulty 后构造 GeneratedQuestion
异常:
ValueError: JSON 解析失败缺必需字段options 4 answer 不在 A-D
"""
# Phase 1: 尝试直接解析 JSON
data = None
with contextlib.suppress(json.JSONDecodeError):
data = json.loads(raw)
# Phase 2: 从 markdown 代码块提取 JSON
if data is None:
match = re.search(r"```(?:json)?\s*\n?(.*?)\n?\s*```", raw, re.DOTALL)
if match:
with contextlib.suppress(json.JSONDecodeError):
data = json.loads(match.group(1))
if data is None:
raise ValueError(f"VLM 返回无法解析为 JSON: {raw[:200]}")
# Phase 3: 校验必需字段
required = ("question", "options", "answer")
missing = [f for f in required if f not in data]
if missing:
raise ValueError(f"VLM 返回缺少必需字段 {missing}: {raw[:200]}")
# Phase 4: options 必须恰好 4 项
options = data["options"]
if not isinstance(options, list) or len(options) != 4:
raise ValueError(
f"options 必须恰好 4 项,实际 {len(options) if isinstance(options, list) else type(options).__name__}: {raw[:200]}"
)
# Phase 5: answer 必须是 A-D
answer = data["answer"]
if answer not in _VALID_ANSWERS:
raise ValueError(f"answer 必须是 A/B/C/D 之一,实际 '{answer}': {raw[:200]}")
return {
"question_id": f"gen-{video_id}-{task_type.lower().replace(' ', '_')}-{seq:03d}",
"question": data["question"],
"options": list(options),
"answer": answer,
}
# ---------------------------------------------------------------------------
# Embedding 去重
# ---------------------------------------------------------------------------
def is_duplicate(
question_text: str,
pool_embeddings: np.ndarray,
embed_fn: Callable[[str | list[str]], np.ndarray],
threshold: float,
) -> bool:
"""embedding 去重判定。
参数:
question_text: 待检查的题目文本
pool_embeddings: 已有题目的 embedding 矩阵 [N, D]L2 归一化
embed_fn: 文本嵌入函数返回 [N, D] ndarrayL2 归一化
threshold: 余弦相似度阈值
返回:
True 表示与池中某题重复空池永远返回 False
"""
if pool_embeddings.shape[0] == 0:
return False
query = embed_fn(question_text) # [1, D]
query = query.squeeze(0) # [D]
similarities = pool_embeddings @ query # [N]
return bool(np.max(similarities) >= threshold)
# ---------------------------------------------------------------------------
# 单题生成
# ---------------------------------------------------------------------------
async def generate_one(
vlm: VLMProvider,
tree: TreeIndex,
video_id: str,
task_type: str,
seq: int,
*,
exemplars: list[GeneratedQuestion],
used_node_ids: set[str],
max_retries: int,
rng: random.Random,
session_id: str,
) -> GeneratedQuestion | None:
"""生成单道候选题(不含去重——去重在调用方汇总点原子执行)。
循环最多 max_retries 次尝试生成每次尝试
1. 采样锚节点
2. 构造 prompt
3. 调用 VLM
4. 解析响应
5. 构造 GeneratedQuestion
返回 None 表示耗尽重试
参数:
vlm: VLM 调用端口
tree: 三层树索引
video_id: 所属视频标识
task_type: 12 Video-MME 题型之一
seq: 序列号用于生成 question_id
exemplars: 少样本示例列表
used_node_ids: 已用节点 ID 集合
max_retries: 最大重试次数
rng: 可控随机数生成器
session_id: 会话 ID传递给 VLM 遥测
返回:
GeneratedQuestion 实例 None耗尽重试
"""
from core.types import GeneratedQuestion as _GeneratedQuestion
for attempt in range(max_retries):
try:
# Phase 1: 采样锚节点
anchor = sample_anchor(tree, task_type, used_node_ids, rng)
# Phase 2: 构造 prompt
messages, images = build_generation_prompt(task_type, anchor, exemplars)
# Phase 3: 调用 VLM
response = await vlm.chat_with_images(
messages,
images,
session_id=session_id,
)
# Phase 4: 解析响应
parsed = parse_vlm_response(response.content, video_id, task_type, seq)
# Phase 5: 构造 GeneratedQuestion
return _GeneratedQuestion(
question_id=parsed["question_id"],
video_id=video_id,
task_type=task_type,
question=parsed["question"],
options=tuple(parsed["options"]),
answer=parsed["answer"],
source_nodes=(anchor.node_id,),
difficulty="medium",
)
except Exception as exc:
logger.warning(
"generate_one 尝试 {}/{} 失败 ({}): {}",
attempt + 1,
max_retries,
task_type,
exc,
)
continue
logger.warning(
"generate_one 耗尽 {} 次重试 (video={}, task_type={})",
max_retries,
video_id,
task_type,
)
return None
+13
View File
@@ -0,0 +1,13 @@
"""搜索 Agent 装配层 — prompt 管理、skill 注册、工具分发、LLM 摘要、视觉观察。"""
from app.search.prompt import PromptManager
from app.search.skills import SkillRegistry, discover_skills
from app.search.tools import SearchToolDispatcher, get_tool_descriptions
__all__ = [
"PromptManager",
"SkillRegistry",
"SearchToolDispatcher",
"discover_skills",
"get_tool_descriptions",
]
+124
View File
@@ -0,0 +1,124 @@
"""搜索 Agent 提示词管理模块。
提供 PromptManager 统一管理循环级 prompt 的加载与组装
工具级 promptextract/verify不在管理范围内
TRM4 ``core/search/prompt.py`` 的差异:
- 工具描述从 ``app.search.tools.get_tool_descriptions`` 获取路径变更;
- ``format_user_prompt`` 参数显式化question/options/l1_node_ids/task_type
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from app.search.tools import get_tool_descriptions
if TYPE_CHECKING:
from pathlib import Path
class PromptManager:
"""管理循环级 prompt 的加载与组装。
构造时缓存 system.md 作为 inference 基础模板
后续步骤diagnose/evolve/question_gen通过 load() 按文件名读取
参数:
prompts_dir: prompt 文件目录的绝对路径
"""
def __init__(self, prompts_dir: Path) -> None:
self._prompts_dir = prompts_dir
system_path = prompts_dir / "system.md"
if not system_path.exists():
raise FileNotFoundError(f"system.md 不存在: {system_path}")
self._system_base = system_path.read_text(encoding="utf-8")
def build_inference_prompt(
self,
skill_mode: str,
task_type: str,
always_skills_text: str,
task_skill_map: dict[str, str],
catalog_text: str,
) -> str:
"""组装 inference 步骤的完整 system prompt。
参数:
skill_mode: "auto" / "manual" / "none"
task_type: 当前 QA 的题型
always_skills_text: always skill 正文已拼接
task_skill_map: {task_type: skill_body} 映射
catalog_text: manual 模式的 skill 目录文本
返回:
拼装后的完整 system prompt
"""
include_read_skill = skill_mode == "manual"
parts = [
self._system_base,
f"\n\n---\n\n{get_tool_descriptions(include_read_skill=include_read_skill)}",
]
if always_skills_text:
parts.append(f"\n\n---\n\n# 通用搜索策略\n\n{always_skills_text}")
if skill_mode == "auto":
skill_text = task_skill_map.get(task_type) or task_skill_map.get("_default")
if skill_text:
parts.append(f"\n\n---\n\n# 当前题型搜索策略\n\n{skill_text}")
elif skill_mode == "manual":
if catalog_text:
parts.append(
"\n\n---\n\n# 可用搜索策略\n\n"
"以下技能扩展了你的导航能力。当问题匹配某技能的适用题型时,"
"用 read_skill 工具加载该技能,然后按其指引操作。\n\n"
f"{catalog_text}"
)
return "".join(parts)
def format_user_prompt(
self,
question: str,
options: list[str],
l1_node_ids: list[str],
task_type: str | None = None,
) -> str:
"""格式化 inference 步骤的用户提示词。
参数:
question: 问题文本
options: 选项列表 ["A. 历史", "B. 科学"]
l1_node_ids: L1 根节点 ID 列表 ["L1_000", "L1_001"]
task_type: 可选题型标签 None 时插入题型行oracle 实验用
返回:
格式化后的用户提示词
"""
options_text = "\n".join(options)
roots_text = ", ".join(l1_node_ids)
task_type_line = f"**题型**: {task_type}\n" if task_type else ""
return (
f"请回答以下关于这个视频的多选题:\n\n"
f"{task_type_line}"
f"**问题**: {question}\n"
f"**选项**:\n{options_text}\n\n"
f"**视频树 L1 根节点**: {roots_text}\n"
f"请从以上 L1 节点开始导航,收集证据后回答。"
)
def load(self, name: str) -> str:
"""按文件名加载 prompt 内容。
参数:
name: prompt 文件名 "diagnose_span.md"
返回:
文件内容字符串
异常:
FileNotFoundError: 文件不存在
"""
path = self._prompts_dir / name
if not path.exists():
raise FileNotFoundError(f"prompt 文件不存在: {path}")
return path.read_text(encoding="utf-8")
+195
View File
@@ -0,0 +1,195 @@
"""技能注册表与 Markdown frontmatter 解析工具。
提供 Skill 文件的 frontmatter 解析正文提取注册表管理和目录扫描功能
供搜索 Agent 装配层使用
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from pathlib import Path
_FRONTMATTER_FIELDS = {"name", "description", "always", "task_type"}
def _extract_frontmatter_lines(text: str) -> tuple[list[str], int] | None:
"""提取 frontmatter 行与正文起始偏移。
参数:
text: 原始 Markdown 文本
返回:
(frontmatter 行列表, 正文起始字节偏移) 二元组
若不存在完整 frontmatter 则返回 None
"""
lines = text.splitlines(keepends=True)
if not lines or lines[0].strip() != "---":
return None
offset = len(lines[0])
frontmatter_lines: list[str] = []
for line in lines[1:]:
if line.strip() == "---":
return frontmatter_lines, offset + len(line)
frontmatter_lines.append(line)
offset += len(line)
logger.debug("frontmatter 缺少结束分隔符,按普通正文处理")
return None
def strip_frontmatter(text: str) -> str:
"""移除 Markdown 文本开头的 frontmatter,并返回正文。
参数:
text: 原始 Markdown 文本
返回:
去除 frontmatter 后的正文 frontmatter 不完整或不存在则返回原文
"""
extracted = _extract_frontmatter_lines(text)
if extracted is None:
return text
_, body_start = extracted
return text[body_start:]
def parse_frontmatter(text: str) -> dict[str, str]:
"""解析 Markdown frontmatter 中的目标字段。
仅识别 ``name````description````always````task_type`` 四个字段
其余字段会被忽略引号包裹的值会自动去除引号
参数:
text: 原始 Markdown 文本
返回:
仅包含目标字段的字符串字典
若不存在完整 frontmatter则返回空字典
"""
extracted = _extract_frontmatter_lines(text)
if extracted is None:
return {}
frontmatter_lines, _ = extracted
parsed: dict[str, str] = {}
for raw_line in frontmatter_lines:
line = raw_line.strip()
if not line or ":" not in line:
continue
key, _, raw_value = line.partition(":")
normalized_key = key.strip()
if normalized_key not in _FRONTMATTER_FIELDS:
continue
value = raw_value.strip()
if len(value) >= 2 and (
(value.startswith('"') and value.endswith('"'))
or (value.startswith("'") and value.endswith("'"))
):
value = value[1:-1]
parsed[normalized_key] = value
return parsed
class SkillRegistry:
"""管理技能名称到文件路径映射并读取技能正文。
通过 ``set_paths`` 注入名称路径映射后
可用 ``read`` 按名读取技能 Markdown 正文自动去除 frontmatter
"""
def __init__(self) -> None:
self._paths: dict[str, Path] = {}
def set_paths(self, mapping: dict[str, Path]) -> None:
"""注入技能名称到文件路径的映射。
参数:
mapping: 技能名到 Markdown 文件路径的映射
"""
self._paths = dict(mapping)
logger.debug("SkillRegistry 已载入 {} 个技能路径", len(self._paths))
def read(self, name: str) -> str:
"""读取指定技能文件,并返回去除 frontmatter 后的正文。
参数:
name: 技能名称
返回:
技能 Markdown 正文
异常:
KeyError: 技能名称未注册时抛出
"""
try:
path = self._paths[name]
except KeyError:
logger.error("技能未注册: {}", name)
raise
logger.debug("读取技能文件: name={}, path={}", name, path)
return strip_frontmatter(path.read_text(encoding="utf-8"))
def discover_skills(
skills_dir: Path,
) -> tuple[str, dict[str, str], str, SkillRegistry]:
"""扫描 skills 目录,按 frontmatter 分类返回。
遍历 ``*.md`` 文件根据 frontmatter ``always`` / ``task_type`` 字段分类
- ``always=true`` skill 拼入 ``always_skills_text``
- ``task_type`` skill 加入 ``task_skill_map``
- always skill 生成 ``catalog_text`` 并注册到 registry
参数:
skills_dir: Skill 文件目录
返回:
``(always_skills_text, task_skill_map, catalog_text, registry)`` 四元组
"""
if not skills_dir.exists():
return "", {}, "", SkillRegistry()
always_parts: list[str] = []
task_skill_map: dict[str, str] = {}
catalog_lines: list[str] = []
registry_paths: dict[str, Path] = {}
for path in sorted(skills_dir.glob("*.md")):
raw = path.read_text(encoding="utf-8")
meta = parse_frontmatter(raw)
if "name" not in meta:
logger.warning("跳过无 name 的 skill 文件: {}", path)
continue
body = strip_frontmatter(raw)
name = meta["name"]
desc = meta.get("description", "")
task_type = meta.get("task_type", "")
is_always = str(meta.get("always", "false")).lower() == "true"
if is_always:
always_parts.append(body)
else:
if task_type:
task_skill_map[task_type] = body
catalog_lines.append(f"- **{name}**: {desc}")
registry_paths[name] = path
always_text = "\n\n---\n\n".join(always_parts)
catalog_text = "\n".join(catalog_lines)
registry = SkillRegistry()
registry.set_paths(registry_paths)
return always_text, task_skill_map, catalog_text, registry
+487
View File
@@ -0,0 +1,487 @@
"""节点内容摘要模块 — 两轮 LLM 调用生成 question-conditioned 摘要。
提取轮带防幻觉 system prompt提取与问题相关的信息
验证轮带核实 system prompt逐条核实并给置信度
TRM4 core/tree/summarizer.py 保真迁移
同步 async_call_llm await llm.chat()ThreadPoolExecutor asyncio.gather
"""
from __future__ import annotations
import asyncio
import re
from typing import TYPE_CHECKING, Any
from loguru import logger
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
from core.protocols import LLMProvider
# ── 正则常量 ──────────────────────────────────────────────────────────
# 行号引注组:括号包裹的 s/c 行号列表,如 (s1) / (c2,s5) / (c70-c73,s196-s200)
# (兼容全角括号与逗号;单元允许范围语法 s3-s5 / s3-560-span 实测模型常用)
_ANCHOR_GROUP = re.compile(
r"[(]\s*([sc]\d+(?:-[sc]?\d+)?(?:\s*[,]\s*[sc]\d+(?:-[sc]?\d+)?)*)\s*[)]"
)
_ANCHOR_RANGE = re.compile(r"([sc])(\d+)-([sc]?)(\d+)")
_RELEVANT_SECTION = re.compile(r"\[相关信息\](.*?)(?=\n\[|\Z)", re.DOTALL)
# 无相关信息声明句:60-span 实测全为"该节点未包含与问题直接相关的信息"类变体
_NO_INFO_STATEMENT = re.compile(r"未包含.*相关.*信息")
# 范围展开条数上限:防 (s1-s9999) 这类爆炸展开
_RANGE_MAX_IDS = 50
# 双封顶参数:上轮 A/B 证明无上限引用膨胀至 8.4 条/span 挤占提取预算(hall +51%
_EXPAND_MAX_ITEMS = 5
_EXPAND_MAX_CHARS = 800
_EXPAND_LINE_CAP = 200
# ── Prompt 加载 ──────────────────────────────────────────────────────
def _load_prompt(prompts_dir: Path, filename: str) -> str:
"""从 prompts 目录加载 system prompt 文件。
参数:
prompts_dir: prompt 文件所在目录
filename: prompt 文件名
返回:
文件内容字符串
"""
return (prompts_dir / filename).read_text(encoding="utf-8")
# ── Anchor 工具函数 ──────────────────────────────────────────────────
def _expand_anchor_ids(group_text: str) -> list[str]:
"""把引注组文本展开为逐 id 列表(支持范围语法)。
参数:
group_text: _ANCHOR_GROUP 捕获的组内文本 "s3-s5, c1"
返回:
id 列表合法范围同前缀起点<=终点展开条数<=50展开为
id"s3-s5"/"s3-5" -> s3,s4,s5非法范围跨前缀如 c3-s5
起点>终点展开条数超限防爆炸保留原 token后续查表必然失配
整段按 1 个非法锚计罚剔除
"""
ids: list[str] = []
for token in re.split(r"[,]\s*", group_text):
token = token.strip()
m = _ANCHOR_RANGE.fullmatch(token)
if m is None:
ids.append(token)
continue
prefix, start = m.group(1), int(m.group(2))
end_prefix, end = m.group(3), int(m.group(4))
legal_range = (
(not end_prefix or end_prefix == prefix)
and start <= end
and end - start + 1 <= _RANGE_MAX_IDS
)
if not legal_range:
ids.append(token)
continue
ids.extend(f"{prefix}{i}" for i in range(start, end + 1))
return ids
def check_anchors(summary: str, anchor_map: dict[str, str]) -> tuple[str, dict[str, int]]:
"""校验行号引注:非法行号删锚不删断言。
参数:
summary: 提取轮输出含行号引注
anchor_map: {: 原文行} 查表
返回:
(清理后文本, {"n_assertions", "n_anchored", "n_illegal"})
关键实现细节:
清洗全文统计限段非法锚无论出现在哪一段都删除并计入 n_illegal
避免未校验段落的编造锚流入装配展开断言统计
n_assertions/n_anchored仅数 [相关信息] 段内非空内容行
引注组先经 _expand_anchor_ids 把范围语法展开为逐 id 再逐 id 校验
合法子集重写为逐 id 列表如 (s3,s4,s5)组内全非法则整组删除
组外文本一律不动删锚不删断言分母口径匹配"未包含...相关...
信息"词面的声明句不计入 n_assertions——它们天然无锚,计入会虚压
遵从率
"""
stats: dict[str, int] = {"n_assertions": 0, "n_anchored": 0, "n_illegal": 0}
def _clean_group(gm: re.Match) -> str:
ids = _expand_anchor_ids(gm.group(1))
legal = [i for i in ids if i in anchor_map]
stats["n_illegal"] += len(ids) - len(legal)
return f"({','.join(legal)})" if legal else ""
cleaned = _ANCHOR_GROUP.sub(_clean_group, summary)
m = _RELEVANT_SECTION.search(cleaned)
if m is None:
return cleaned, stats
for line in m.group(1).splitlines():
line = line.strip().lstrip("-•*").strip()
if not line:
continue
if _NO_INFO_STATEMENT.search(line):
continue
stats["n_assertions"] += 1
if _ANCHOR_GROUP.search(line):
stats["n_anchored"] += 1
return cleaned, stats
def _cited_anchor_ids(summary: str, anchor_map: dict[str, str]) -> list[str]:
"""按引注首次出现顺序收集合法锚 id(去重)。
参数:
summary: 含行号引注的文本
anchor_map: {: 原文行} 查表
返回:
去重后的合法锚 id 列表保持首次出现顺序
关键实现细节:
assemble_anchored_output 提取以满足圈复杂度门槛范围语法经
_expand_anchor_ids 展开后逐 id 收集只收合法锚非法锚已由
check_anchors 清除此处过滤是防御性双保险
"""
ordered: list[str] = []
for gm in _ANCHOR_GROUP.finditer(summary):
for aid in _expand_anchor_ids(gm.group(1)):
if aid in anchor_map and aid not in ordered:
ordered.append(aid)
return ordered
def assemble_anchored_output(
summary: str, anchor_map: dict[str, str], mode: str
) -> tuple[str, dict[str, int]]:
"""按装配形态生成最终输出:展开引文并施加双封顶。
参数:
summary: check_anchors 清理后的文本
anchor_map: {: 原文行}
mode: "ids"裸行号| "ids_expand"行号+展开| "expand_only"展开剥行号
返回:
(最终文本, {"n_expanded", "n_trunc"})
关键实现细节:
展开按引注首次出现顺序取前 5 总额帽按 [引文] 条目完整长度
含前缀与引号记账<=800 字符单行原文超 200 字符先截断
n_expanded/n_trunc 仅计实际输出的条目expand_only 先对正文剥除
全部引注 token再拼接 [引文] judge 探针判定 id token 被计罚
时的回退形态引文行不经过剥离原文行中的括号文本得以保留
"""
assert mode in ("ids", "ids_expand", "expand_only"), f"未知装配形态: {mode}"
stats: dict[str, int] = {"n_expanded": 0, "n_trunc": 0}
if mode != "ids":
ordered = _cited_anchor_ids(summary, anchor_map)
expansions: list[str] = []
total = 0
for aid in ordered[:_EXPAND_MAX_ITEMS]:
line = anchor_map[aid]
truncated = len(line) > _EXPAND_LINE_CAP
if truncated:
line = line[:_EXPAND_LINE_CAP] + ""
entry = f'{aid}: "{line}"'
if total + len(entry) > _EXPAND_MAX_CHARS:
break
total += len(entry)
expansions.append(entry)
stats["n_expanded"] += 1
if truncated:
stats["n_trunc"] += 1
if mode == "expand_only":
summary = _ANCHOR_GROUP.sub("", summary)
if expansions:
summary = summary + "\n[引文]\n" + "\n".join(expansions)
return summary, stats
# ── LLM 调用辅助 ─────────────────────────────────────────────────────
async def _call_llm(
llm: LLMProvider,
system_prompt: str,
user_text: str,
*,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> str:
"""调用 LLM 并返回响应文本。
参数:
llm: LLMProvider 端口实例
system_prompt: 系统提示词
user_text: 用户消息文本
session_id: 会话 ID透传遥测
parent_call_id: 父调用 ID透传遥测
返回:
模型回答文本
"""
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_text},
]
response = await llm.chat(messages, session_id=session_id, parent_call_id=parent_call_id)
return response.content
# ── 摘要函数 ─────────────────────────────────────────────────────────
async def summarize_node(
llm: LLMProvider,
raw_text: str,
question: str,
prompts_dir: Path,
*,
anchor_map: dict[str, str] | None,
assemble_mode: str,
stats_sink: Callable[[dict[str, Any]], None] | None = None,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> str:
"""对单个节点做 question-conditioned 两轮摘要(可选行号锚模式)。
参数:
llm: LLMProvider 端口实例
raw_text: 节点文本锚模式下为带 [c1]/[s1] 行号的素材
question: Agent 当前关注的具体问题
prompts_dir: prompt 文件目录
anchor_map: {: 原文行}None 表示 v1 行为无校验无装配无统计
assemble_mode: 装配形态"ids"/"ids_expand"/"expand_only"
anchor_map None 时忽略
stats_sink: 统计回调None 不收集统计严禁写入输出文本
session_id: 会话 ID透传遥测
parent_call_id: 父调用 ID透传遥测
返回:
"[内容摘要] {结果}\\n[核实] {验证结果}" 或错误信息
关键实现细节:
锚模式流程提取 -> check_anchors 清洗 -> 核实轮见清洗后未装配文本
-> assemble_anchored_output 装配 -> sink 上报sink dict 完整键名
n_assertions/n_anchored/n_illegalcheck_anchors
n_expanded/n_trunc装配output_chars最终输出字符数
pre_assembly清洗后未装配文本快照anchor_map原样透传
"""
extract_input = f"问题: {question}\n\n以下是视频片段的描述和字幕:\n{raw_text}"
try:
raw_summary = await _call_llm(
llm,
_load_prompt(prompts_dir, "view_node_extract.md"),
extract_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
except Exception as e:
return f"[摘要错误] {e}"
anchor_stats: dict[str, int] = {}
if anchor_map is not None:
raw_summary, anchor_stats = check_anchors(raw_summary, anchor_map)
pre_assembly = raw_summary
verify_input = (
f"问题: {question}\n\n"
f"原始内容:\n{raw_text}\n\n"
f"以下是另一个模型基于上述内容生成的摘要,请核实:\n{raw_summary}"
)
try:
verify_result = await _call_llm(
llm,
_load_prompt(prompts_dir, "view_node_verify.md"),
verify_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
except Exception as e:
logger.warning("验证轮调用失败,跳过: {}", e)
verify_result = "跳过(调用失败)"
if anchor_map is not None:
raw_summary, asm_stats = assemble_anchored_output(raw_summary, anchor_map, assemble_mode)
anchor_stats.update(asm_stats)
result = f"[内容摘要] {raw_summary}\n[核实] {verify_result}"
if anchor_map is not None and stats_sink is not None:
stats_sink(
{
**anchor_stats,
"output_chars": len(result),
"pre_assembly": pre_assembly,
"anchor_map": anchor_map,
}
)
return result
async def summarize_children(
llm: LLMProvider,
children_info: list[dict[str, Any]],
question: str,
prompts_dir: Path,
*,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> str:
"""对子节点列表做 question-conditioned 相关性标注(两轮)。
参数:
llm: LLMProvider 端口实例
children_info: 子节点信息列表每项含 id, time_range, summary
question: Agent 当前关注的具体问题
prompts_dir: prompt 文件目录
session_id: 会话 ID透传遥测
parent_call_id: 父调用 ID透传遥测
返回:
带相关性标注的子节点概览文本失败时降级返回原始列表
"""
lines = []
for child in children_info:
t_start, t_end = child["time_range"]
lines.append(f"- {child['id']} ({t_start:.0f}-{t_end:.0f}s): {child['summary']}")
children_text = "\n".join(lines)
extract_input = f"问题: {question}\n\n{children_text}"
try:
raw_ranking = await _call_llm(
llm,
_load_prompt(prompts_dir, "view_node_children_extract.md"),
extract_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
except Exception as e:
logger.warning("子节点标注失败,回退原始列表: {}", e)
return children_text
verify_input = (
f"问题: {question}\n\n"
f"原始子节点列表:\n{children_text}\n\n"
f"以下是另一个模型基于上述信息生成的相关性标注,请核实:\n{raw_ranking}"
)
try:
verify_result = await _call_llm(
llm,
_load_prompt(prompts_dir, "view_node_children_verify.md"),
verify_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
return f"{raw_ranking}\n[核实] {verify_result}"
except Exception as e:
logger.warning("子节点标注验证轮失败,跳过: {}", e)
return raw_ranking
async def _summarize_search_result(
llm: LLMProvider,
raw_text: str,
question: str,
prompts_dir: Path,
*,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> str:
"""对搜索结果做两轮摘要(search_similar 专用)。
参数:
llm: LLMProvider 端口实例
raw_text: 节点原始文本
question: Agent 当前关注的具体问题
prompts_dir: prompt 文件目录
session_id: 会话 ID透传遥测
parent_call_id: 父调用 ID透传遥测
返回:
"[内容摘要] {提取结果}\\n[核实] {验证结果}" 或错误信息
"""
extract_input = f"问题: {question}\n\n以下是语义搜索命中的视频节点描述和字幕:\n{raw_text}"
try:
raw_summary = await _call_llm(
llm,
_load_prompt(prompts_dir, "search_similar_extract.md"),
extract_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
except Exception as e:
return f"[摘要错误] {e}"
verify_input = (
f"问题: {question}\n\n"
f"原始内容:\n{raw_text}\n\n"
f"以下是另一个模型基于上述内容生成的摘要,请核实:\n{raw_summary}"
)
try:
verify_result = await _call_llm(
llm,
_load_prompt(prompts_dir, "search_similar_verify.md"),
verify_input,
session_id=session_id,
parent_call_id=parent_call_id,
)
return f"[内容摘要] {raw_summary}\n[核实] {verify_result}"
except Exception as e:
logger.warning("搜索结果验证轮失败,跳过: {}", e)
return f"[内容摘要] {raw_summary}\n[核实] 跳过(调用失败)"
async def summarize_nodes_batch(
llm: LLMProvider,
items: list[tuple[str, str, str]],
question: str,
prompts_dir: Path,
*,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> list[tuple[str, str]]:
"""并发对多个搜索结果做两轮摘要。
参数:
llm: LLMProvider 端口实例
items: [(node_id, raw_text, extra_info), ...] 列表
question: Agent 当前关注的具体问题
prompts_dir: prompt 文件目录
session_id: 会话 ID透传遥测
parent_call_id: 父调用 ID透传遥测
返回:
[(node_id, summary_text), ...] 列表顺序与输入一致
"""
if not items:
return []
async def _worker(idx: int, node_id: str, raw_text: str) -> tuple[int, str, str]:
"""单个节点的摘要工作协程。"""
summary = await _summarize_search_result(
llm,
raw_text,
question,
prompts_dir,
session_id=session_id,
parent_call_id=parent_call_id,
)
return idx, node_id, summary
tasks = [_worker(i, nid, text) for i, (nid, text, _) in enumerate(items)]
results_raw = await asyncio.gather(*tasks)
results: dict[int, tuple[str, str]] = {}
for idx, node_id, summary in results_raw:
results[idx] = (node_id, summary)
return [results[i] for i in range(len(items))]
+325
View File
@@ -0,0 +1,325 @@
"""搜索 Agent 工具调度器 — 工具描述与 dispatch 分发。
实现 ``core/agent/protocols.ToolDispatcher`` Protocol
连接 TreeEnvironment数据summarizerLLM 摘要
visionVLM 观察 skills策略加载
TRM4 ``core/tree/tools.py`` 的差异:
- 自由函数 ``dispatch()`` ``SearchToolDispatcher`` 依赖注入;
- 同步 全异步;
- view_node / search_similar 内部拆分为 env 数据读取 + summarizer LLM 摘要
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from app.search.summarizer import summarize_children, summarize_node, summarize_nodes_batch
from app.search.vision import observe_frame
from app.tree.environment import _LEVEL_LABEL, TreeEnvironment, _node_level
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
import numpy as np
from app.ports import OCRProvider
from app.search.skills import SkillRegistry
from core.protocols import LLMProvider, VLMProvider
# ── 工具描述文本(与 TRM4 core/tree/tools.py 完全一致) ─────────────────
_BASE_DESCRIPTIONS = """\
## 可用工具
action 中指定 tool args 来调用工具
### view_node
查看节点信息获取与问题相关的内容摘要和子节点概览
- args: {"node_id": "节点 ID", "question": "当前关注的具体问题"}
### search_similar
语义检索最相关的节点返回与问题相关的内容摘要
- args: {"query": "搜索关键词(2-4 词)", "question": "当前关注的具体问题", "k": 返回数量可选默认 5}
### observe_frame
调用视觉模型查看关键帧图像回答针对性的视觉问题
- args: {"node_ids": ["L3 节点 ID 列表(1-4 个),或单个 L2 节点 ID"], "question": "针对帧内容的具体视觉问题"}
### submit_answer
提交最终答案
- args: {"answer": "选项字母 A/B/C/D", "evidence": "关键证据摘要", "reasoning": "每个选项的判断理由"}"""
_SKILL_DESCRIPTION = """
### read_skill
加载指定题型技能的详细搜索策略
- args: {"name": "技能名称"}"""
def get_tool_descriptions(include_read_skill: bool = False) -> str:
"""返回工具描述文本,用于写入 system prompt。
参数:
include_read_skill: 是否包含 read_skill 工具manual 模式用
返回:
Markdown 格式的工具描述文本
"""
text = _BASE_DESCRIPTIONS
if include_read_skill:
text += _SKILL_DESCRIPTION
return text
# ── SearchToolDispatcher ──────────────────────────────────────────────
class SearchToolDispatcher:
"""搜索 Agent 工具调度器,实现 ToolDispatcher Protocol。
按工具名路由到对应私有处理方法未知工具抛 ValueError
AgentLoop 捕获后不计步数节点不存在等运行时错误
捕获后返回错误文本
参数:
env: 视频树运行时环境纯数据访问
tool_llm: 摘要用 LLM 端口
vlm: 视觉模型端口
ocr: 帧文字转录端口None 不启用
prompts_dir: prompt 文件目录
skills: 技能注册表None 不启用 read_skill
embed_fn: 文本嵌入函数search_similar
verify_vision: observe_frame 是否执行验证轮
anchor: view_node 是否启用行号锚模式
assemble_mode: 锚模式装配形态"ids"/"ids_expand"/"expand_only"
stats_sink: 统计回调None 不收集
"""
def __init__(
self,
env: TreeEnvironment,
tool_llm: LLMProvider,
vlm: VLMProvider,
ocr: OCRProvider | None,
prompts_dir: Path,
skills: SkillRegistry | None,
*,
embed_fn: Callable[[str | list[str]], np.ndarray],
verify_vision: bool,
anchor: bool,
assemble_mode: str,
stats_sink: Callable[[dict[str, Any]], None] | None = None,
) -> None:
self._env = env
self._tool_llm = tool_llm
self._vlm = vlm
self._ocr = ocr
self._prompts_dir = prompts_dir
self._skills = skills
self._embed_fn = embed_fn
self._verify_vision = verify_vision
self._anchor = anchor
self._assemble_mode = assemble_mode
self._stats_sink = stats_sink
# ── ToolDispatcher Protocol 实现 ──────────────────────────────────
async def dispatch(
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
) -> str:
"""按工具名分发到对应处理方法。
参数:
tool_name: 工具名称
args: 工具参数字典
context: 调用上下文 session_idparent_call_id 等遥测字段
返回:
工具执行结果文本
异常:
ValueError: 未知工具名上抛给 AgentLoop不计步数
"""
try:
if tool_name == "view_node":
return await self._handle_view_node(args, context)
if tool_name == "search_similar":
return await self._handle_search_similar(args, context)
if tool_name == "observe_frame":
return await self._handle_observe_frame(args, context)
if tool_name == "submit_answer":
return f"[ok] 答案已提交: {args['answer']}"
if tool_name == "read_skill":
return self._handle_read_skill(args)
except (KeyError, FileNotFoundError) as e:
return f"工具执行错误: {e}"
raise ValueError(f"未知工具: {tool_name}")
# ── 私有处理方法 ──────────────────────────────────────────────────
async def _handle_view_node(self, args: dict[str, Any], context: dict[str, Any]) -> str:
"""view_node:节点摘要 + 子节点概览。
参数:
args: {"node_id": str, "question": str}
context: 遥测上下文
返回:
"[节点] {id} | {level} | {time}\\n\\n{summary}\\n\\n[子节点概览] ..."
"""
node_id: str = args["node_id"]
question: str = args["question"]
session_id = context.get("session_id")
parent_call_id = context.get("parent_call_id")
# Phase 1: 节点元数据(头部格式化)
node = self._env._id_to_node[node_id]
level = _node_level(node)
level_label = _LEVEL_LABEL[level]
time_str = TreeEnvironment._format_time_range(node)
# Phase 2: 节点内容摘要
raw_text, anchor_map = self._env.get_node_text(node_id, anchor=self._anchor)
summary = await summarize_node(
self._tool_llm,
raw_text,
question,
self._prompts_dir,
anchor_map=anchor_map,
assemble_mode=self._assemble_mode,
stats_sink=self._stats_sink,
session_id=session_id,
parent_call_id=parent_call_id,
)
parts: list[str] = [
f"[节点] {node_id} | {level_label} | {time_str}",
"",
summary,
]
# Phase 2.5: 确定性追加实体/画面文字原文(防按题摘要吞噬,Spec-1 B)
for label, text in self._env.node_entity_fields(node_id).items():
parts.append(f"[{label}] {text}")
# Phase 3: 子节点概览
children_info = self._env.get_children_info(node_id)
if children_info:
children_text = await summarize_children(
self._tool_llm,
children_info,
question,
self._prompts_dir,
session_id=session_id,
parent_call_id=parent_call_id,
)
parts.append(f"\n[子节点概览] {len(children_info)} 个子节点\n{children_text}")
return "\n".join(parts)
async def _handle_search_similar(self, args: dict[str, Any], context: dict[str, Any]) -> str:
"""search_similar:语义检索 + 批量摘要。
参数:
args: {"query": str, "question": str, "k": int (可选)}
context: 遥测上下文
返回:
"[搜索结果] 查询 \\"{query}\\" → N 个相关节点\\n\\n1. ..."
"""
query: str = args["query"]
question: str = args["question"]
top_k: int = args.get("k", 5)
session_id = context.get("session_id")
parent_call_id = context.get("parent_call_id")
# Phase 1: 语义检索
results = self._env.search_similar(query, top_k=top_k, embed_fn=self._embed_fn)
if not results:
return f'[搜索结果] 查询 "{query}" → 0 个相关节点'
# Phase 2: 构建摘要输入
items: list[tuple[str, str, str]] = []
for nid, score in results:
node = self._env._id_to_node[nid]
raw_text, _ = self._env.get_node_text(nid)
level = _node_level(node)
time_str = TreeEnvironment._format_time_range(node)
extra = f"{level} score={score:.4f} [{time_str}]"
items.append((nid, raw_text, extra))
# Phase 3: 并发批量摘要
summaries = await summarize_nodes_batch(
self._tool_llm,
items,
question,
self._prompts_dir,
session_id=session_id,
parent_call_id=parent_call_id,
)
# Phase 4: 格式化输出
lines: list[str] = []
for i, (nid, summary_text) in enumerate(summaries):
_, _, extra = items[i]
lines.append(f"{i + 1}. {nid} | {extra}\n {summary_text}")
header = f'[搜索结果] 查询 "{query}"{len(results)} 个相关节点'
return header + "\n\n" + "\n\n".join(lines)
async def _handle_observe_frame(self, args: dict[str, Any], context: dict[str, Any]) -> str:
"""observe_frameVLM 帧观察 + 字幕前置。
参数:
args: {"node_ids": list[str], "question": str}
context: 遥测上下文
返回:
"[字幕上下文] ...\\n[视觉观察] ..." "[视觉观察] ..."
"""
node_ids: list[str] = args["node_ids"]
question: str = args.get("question", "")
session_id = context.get("session_id")
parent_call_id = context.get("parent_call_id")
if not question.strip():
return "工具执行错误: question 不能为空"
# Phase 1: 解析帧路径和字幕
frame_paths = self._env.resolve_frame_paths(node_ids)
subtitle = self._env.get_subtitle(node_ids[0])
# Phase 2: VLM 调用
result = await observe_frame(
self._vlm,
frame_paths,
question,
self._prompts_dir,
ocr=self._ocr,
verify=self._verify_vision,
stats_sink=self._stats_sink,
session_id=session_id,
parent_call_id=parent_call_id,
)
# Phase 3: 字幕前置拼接
if subtitle:
return f"[字幕上下文] {subtitle}\n{result}"
return result
def _handle_read_skill(self, args: dict[str, Any]) -> str:
"""read_skill:加载指定技能的搜索策略正文。
参数:
args: {"name": str}
返回:
技能正文或错误提示
"""
if self._skills is None:
return "错误: skills 未启用"
return self._skills.read(args["name"])
+157
View File
@@ -0,0 +1,157 @@
"""视觉模型调用模块 -- 两轮 VLM 调用查看关键帧图像。
提取轮带防幻觉 system prompt提取原始视觉证据
验证轮把初稿全文喂回逐条核实并给置信度
TRM4 ``core/tree/vision.py`` 迁移关键变更
- VLM 调用走 ``VLMProvider.chat_with_images`` Protocolimages Path 列表
- OCR 调用走 ``OCRProvider.transcribe_frames`` 异步 Protocol
- 遥测字段session_id / parent_call_id透传给 VLM 调用
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
from app.ports import OCRProvider
from core.protocols import VLMProvider
_OCR_PREFIX = (
"以下是 OCR 工具对这些帧的文字转录,仅供参考;与你实际看到的不一致时,报告双读数并标注分歧:\n"
)
def _load_prompt(prompts_dir: Path, filename: str) -> str:
"""从 prompts 目录加载 system prompt 文件。
参数:
prompts_dir: prompt 文件所在目录
filename: prompt 文件名
返回:
文件内容字符串
"""
return (prompts_dir / filename).read_text(encoding="utf-8")
async def observe_frame(
vlm: VLMProvider,
frame_paths: list[Path],
question: str,
prompts_dir: Path,
*,
ocr: OCRProvider | None,
verify: bool,
stats_sink: Callable[[dict[str, int]], None] | None = None,
session_id: str | None = None,
parent_call_id: str | None = None,
) -> str:
"""调用 VLM 查看帧图像:可选 OCR 事前并置 + 提取轮 + 可选验证轮。
参数:
vlm: VLM 图文调用端口
frame_paths: 帧文件路径列表
question: 针对帧内容的视觉问题
prompts_dir: prompt 文件目录
ocr: 帧文字转录端口None=不注入返回空串视为无结果不注入
verify: 是否执行验证轮False 时仅提取轮输出无 [验证]
stats_sink: 统计回调None 不收集统计严禁写入输出文本
session_id: 遥测会话 ID透传给 VLM 调用
parent_call_id: 遥测父调用 ID透传给 VLM 调用
返回:
verify=True ``"[视觉观察] {证据}\\n[验证] {核实结果}"``
verify=False ``"[视觉观察] {证据}"``或错误信息
关键实现细节:
OCR 文本作为额外文本并置于问题之前事前并置OCR 误读不进
工具输出故零 judge 口径风险OCR 异常降级为不注入并计
ocr_failedocr 是外部注入依赖任何异常都不得中断工具主流程
故此处 except Exception 是刻意的降级边界sink
ocr_injected / ocr_chars / ocr_failed / discrepancy输出含"分歧"词面/
abstain [证据不存在]
"""
stats: dict[str, int] = {
"ocr_injected": 0,
"ocr_chars": 0,
"ocr_failed": 0,
"discrepancy": 0,
"abstain": 0,
}
def _emit(output: str) -> str:
"""计算语义标记并回调 stats_sink。"""
stats["abstain"] = int("[证据不存在]" in output)
stats["discrepancy"] = int("分歧" in output)
if stats_sink is not None:
stats_sink(stats)
return output
# -- 帧文件存在性校验 --
for p in frame_paths:
if not p.exists():
return _emit(f"[VL错误] 帧文件不存在: {p}")
# -- OCR 转录(可选) --
ocr_text = ""
if ocr is not None:
try:
ocr_text = await ocr.transcribe_frames(frame_paths)
except Exception as e: # noqa: BLE001 — 刻意的降级边界
logger.warning("OCR 转录失败,降级不注入: {}", e)
stats["ocr_failed"] = 1
# -- 拼装提取轮 user 消息 --
user_parts: list[str] = []
if ocr_text:
stats["ocr_injected"] = 1
stats["ocr_chars"] = len(ocr_text)
user_parts.append(_OCR_PREFIX + ocr_text)
user_parts.append(question)
user_text = "\n".join(user_parts)
extract_messages = [
{"role": "system", "content": _load_prompt(prompts_dir, "observe_frame_extract.md")},
{"role": "user", "content": user_text},
]
# -- 提取轮 --
try:
extract_response = await vlm.chat_with_images(
extract_messages,
images=frame_paths,
session_id=session_id,
parent_call_id=parent_call_id,
)
raw_evidence = extract_response.content
except Exception as e: # noqa: BLE001
return _emit(f"[VL错误] {e}")
if not verify:
return _emit(f"[视觉观察] {raw_evidence}")
# -- 验证轮 --
verify_text = (
f"问题: {question}\n\n以下是另一个模型基于这些图片生成的描述,请核实:\n{raw_evidence}"
)
verify_messages = [
{"role": "system", "content": _load_prompt(prompts_dir, "observe_frame_verify.md")},
{"role": "user", "content": verify_text},
]
try:
verify_response = await vlm.chat_with_images(
verify_messages,
images=frame_paths,
session_id=session_id,
parent_call_id=parent_call_id,
)
return _emit(f"[视觉观察] {raw_evidence}\n[验证] {verify_response.content}")
except Exception as e: # noqa: BLE001
logger.warning("验证轮调用失败,跳过: {}", e)
return _emit(f"[视觉观察] {raw_evidence}\n[验证] 跳过(调用失败)")
+42
View File
@@ -0,0 +1,42 @@
"""建树模块配置。"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class TreeConfig:
"""建树配置参数,字段对齐 config/default.yaml 的 tree: 段。
参数:
l1_segment_duration: L1 段时长
l2_clip_duration: L2 clip 时长
l3_fps: L3 帧提取频率/
l2_representative_frames: L2 VLM 描述用的代表帧数
cache_dir: 树索引缓存目录
concurrency: asyncio Semaphore 上限
subtitle_inject: 建树时是否注入 SRT 字幕
srt_window_sec: 字幕匹配时间窗口前后各 N
"""
l1_segment_duration: float = 600.0
l2_clip_duration: float = 60.0
l3_fps: float = 0.5
l2_representative_frames: int = 6
cache_dir: str = "cache/trees"
concurrency: int = 16
subtitle_inject: bool = True
srt_window_sec: float = 5.0
@classmethod
def from_dict(cls, d: dict) -> TreeConfig:
"""从 YAML 解析后的 dict 构造,忽略未知字段。
参数:
d: 配置字典
返回:
TreeConfig 实例
"""
return cls(**{k: v for k, v in d.items() if k in cls.__dataclass_fields__})
View File
+616
View File
@@ -0,0 +1,616 @@
"""TreeEnvironment:单棵视频树的运行时环境。
提供节点查询字幕获取帧路径解析和语义检索能力
纯数据访问层不涉及 LLM 调用LLM 摘要逻辑属于 app/search/
算法 #12 变更:分块 embedding → 单节点 embedding。
祖先去重 + 锚定验证逻辑保留自 TRM4
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import TYPE_CHECKING, Any
import numpy as np
from loguru import logger
from app.tree.index import L1Node, L2Node, L3Node, TreeIndex
if TYPE_CHECKING:
from collections.abc import Callable, Iterable
# 节点联合类型(内部使用)
AnyNode = L1Node | L2Node | L3Node
# 各层级节点对应的主描述字段名
_LEVEL_LABEL = {
"L1": "场景层",
"L2": "事件层",
"L3": "关键帧层",
}
def _node_level(node: AnyNode) -> str:
"""判断节点层级标签。
参数:
node: 树节点实例
返回:
"L1" / "L2" / "L3"
"""
if isinstance(node, L1Node):
return "L1"
if isinstance(node, L2Node):
return "L2"
return "L3"
def _node_description(node: AnyNode) -> str:
"""提取节点的主描述文本。
参数:
node: 树节点实例
返回:
描述文本字符串
"""
if isinstance(node, L1Node):
return node.card.scene_summary
if isinstance(node, L2Node):
return node.card.event_description
return node.card.frame_summary
def _collect_card_strings(
node: AnyNode,
skip_fields: frozenset[str] = frozenset(),
) -> list[str]:
"""从节点 card 中递归收集所有非空字符串字段。
参数:
node: 树节点实例
skip_fields: 需要跳过的 dataclass 字段名集合 subtitle
因为它需要单独添加"字幕:"标签和 [sN] 锚标
返回:
字符串列表每个非空字段值一项含内嵌换行的按行拆分
"""
result: list[str] = []
_collect_from_obj(node.card, result, skip_fields=skip_fields)
return result
# subtitle 字段在 _node_full_text / _node_anchored_text 中单独处理
_SUBTITLE_SKIP: frozenset[str] = frozenset({"subtitle"})
# 各层级 card 的实体字段名(B 修复:dispatcher 追加原文用)
_ENTITY_FIELDS_BY_LEVEL: dict[str, tuple[str, ...]] = {
"L1": ("key_entities",),
"L2": ("entities",),
"L3": ("visible_entities",),
}
def _clean_join(values: Iterable[object] | None) -> str:
"""过滤非空字符串、去重(保序)、分号拼接;无有效值返回空串。
参数:
values: 待清洗的值集合通常为 list[str]容忍 None / 混入非字符串
返回:
"a; b; c" 形式的拼接串无有效值时为空串
"""
cleaned = [v.strip() for v in (values or []) if isinstance(v, str) and v.strip()]
return "; ".join(dict.fromkeys(cleaned))
def _collect_from_obj(
obj: object,
out: list[str],
*,
skip_fields: frozenset[str] = frozenset(),
) -> None:
"""递归收集任意嵌套结构中的非空字符串。
参数:
obj: dict / list / str / 其他
out: 收集结果列表原地修改
skip_fields: 需要跳过的 dataclass 字段名集合
"""
if isinstance(obj, str):
stripped = obj.strip()
if stripped:
out.append(stripped)
elif isinstance(obj, dict):
for v in obj.values():
_collect_from_obj(v, out, skip_fields=skip_fields)
elif isinstance(obj, (list, tuple)):
for item in obj:
_collect_from_obj(item, out, skip_fields=skip_fields)
elif hasattr(obj, "__dataclass_fields__"):
# frozen dataclassCard 类型)
for field_name in obj.__dataclass_fields__:
if field_name in skip_fields:
continue
_collect_from_obj(getattr(obj, field_name), out, skip_fields=skip_fields)
class TreeEnvironment:
"""单棵视频树的运行时环境,提供节点查询和语义检索。
纯数据访问层不涉及 LLM 调用
参数:
index: 已加载的 TreeIndex 实例
frames_dir: 帧文件目录路径可选未提供时使用节点自带的 frame_path
"""
def __init__(
self,
index: TreeIndex,
frames_dir: Path | None = None,
) -> None:
self._index = index
self._frames_dir = frames_dir
# O(1) 查找表:node_id → 节点实例
self._id_to_node: dict[str, AnyNode] = {}
# 父节点映射:node_id → parent_id(根节点为 None
self._id_to_parent: dict[str, str | None] = {}
self._build_lookup_tables()
logger.debug(
"TreeEnvironment 初始化完成,节点数={}",
len(self._id_to_node),
)
# ------------------------------------------------------------------
# 初始化辅助
# ------------------------------------------------------------------
def _build_lookup_tables(self) -> None:
"""遍历 TreeIndex 构建 _id_to_node 和 _id_to_parent 映射表。"""
for l1 in self._index.roots:
self._id_to_node[l1.id] = l1
self._id_to_parent[l1.id] = None
for l2 in l1.children:
self._id_to_node[l2.id] = l2
self._id_to_parent[l2.id] = l1.id
for l3 in l2.children:
self._id_to_node[l3.id] = l3
self._id_to_parent[l3.id] = l2.id
# ------------------------------------------------------------------
# 公开方法
# ------------------------------------------------------------------
def view_node(self, node_id: str, *, anchor: bool = False) -> str:
"""返回节点卡片内容 + 子节点概览。
参数:
node_id: 节点 ID
anchor: 为卡片字段添加行锚标 [c1] [s1] 供引用验证
返回:
格式化文本
异常:
KeyError: 节点不存在
"""
node = self._id_to_node.get(node_id)
if node is None:
raise KeyError(f"节点不存在: {node_id}")
level = _node_level(node)
level_label = _LEVEL_LABEL[level]
# 时间范围
time_range_str = self._format_time_range(node)
# 节点内容
content = self._node_anchored_text(node) if anchor else self._node_full_text(node)
parts = [
f"[节点] {node_id} | {level_label} | {time_range_str}",
"",
content,
]
# 子节点概览
children = self._get_children(node)
if children:
parts.append("")
parts.append(f"[子节点概览] {len(children)} 个子节点")
for child in children:
child_desc = _node_description(child)
child_time = self._format_time_range(child)
# 截断描述到 120 字符
if len(child_desc) > 120:
child_desc = child_desc[:120] + "..."
parts.append(f" - {child.id} | {child_time} | {child_desc}")
return "\n".join(parts)
def node_entity_fields(self, node_id: str) -> dict[str, str]:
"""返回节点 card 的实体/画面文字字段原文。
dispatcher 在按题摘要后确定性追加防止 LLM 摘要吞掉
entities/visible_text 信号benchmark 错题 M1 恶化因素
参数:
node_id: 节点 ID
返回:
{"实体": "...", "画面文字": "..."}空字段不含对应键
异常:
KeyError: 节点不存在
"""
node = self._id_to_node.get(node_id)
if node is None:
raise KeyError(f"节点不存在: {node_id}")
level = _node_level(node)
entity_values: list[str] = []
for field_name in _ENTITY_FIELDS_BY_LEVEL[level]:
entity_values.extend(getattr(node.card, field_name) or [])
out: dict[str, str] = {}
entities_text = _clean_join(entity_values)
if entities_text:
out["实体"] = entities_text
visible_text = _clean_join(getattr(node.card, "visible_text", None))
if visible_text:
out["画面文字"] = visible_text
return out
def search_similar(
self,
query: str,
top_k: int = 5,
*,
embed_fn: Callable[[str | list[str]], np.ndarray] | None = None,
) -> list[tuple[str, float]]:
"""语义搜索 + 祖先去重。
算法 #12 变更:单节点 embedding(非分块),祖先去重 + 锚定验证保留。
参数:
query: 搜索文本
top_k: 返回数量
embed_fn: 嵌入函数未提供时使用 TreeIndex 已有 embedding
返回:
[(node_id, score), ...] 按相似度降序
异常:
ValueError: 节点未 embed 且未提供 embed_fn
"""
if embed_fn is None:
raise ValueError(
"embed_fn 为必需参数:搜索 query 需要 embed_fn 来编码。请传入 embed_fn 参数。"
)
# 收集所有节点的 embedding(优先使用 TreeIndex 已有 embedding
node_ids: list[str] = []
embeddings: list[np.ndarray] = []
if self._index.is_embedded:
# 使用已有 embedding
for nid, node in self._id_to_node.items():
if node.embedding is not None:
node_ids.append(nid)
embeddings.append(node.embedding)
else:
# 使用 embed_fn 为所有节点生成 embedding
all_ids = list(self._id_to_node.keys())
all_texts = [_node_description(self._id_to_node[nid]) for nid in all_ids]
all_embs = embed_fn(all_texts) # [N, D]
for i, nid in enumerate(all_ids):
node_ids.append(nid)
embeddings.append(all_embs[i])
if not embeddings:
return []
node_embeddings = np.stack(embeddings, axis=0) # [N, D]
# 归一化(确保余弦相似度正确)
norms = np.linalg.norm(node_embeddings, axis=1, keepdims=True)
norms = np.where(norms == 0, 1.0, norms)
node_embeddings = node_embeddings / norms
# 编码 query
query_emb = embed_fn(query) # [1, D]
if query_emb.ndim == 1:
query_emb = query_emb.reshape(1, -1)
# 归一化 query
q_norm = np.linalg.norm(query_emb)
if q_norm > 0:
query_emb = query_emb / q_norm
# 余弦相似度
scores = (node_embeddings @ query_emb.T).squeeze() # [N]
if scores.ndim == 0:
scores = scores.reshape(1)
# 按分数排序
scored_pairs = sorted(
zip(node_ids, scores.tolist(), strict=True),
key=lambda x: x[1],
reverse=True,
)
# 祖先去重:如果更细粒度的子节点已入选,跳过其祖先
deduped: list[tuple[str, float]] = []
seen_prefixes: set[str] = set()
for nid, score in scored_pairs:
is_ancestor_of_seen = any(s.startswith(nid + "_") for s in seen_prefixes)
if is_ancestor_of_seen:
continue
deduped.append((nid, score))
seen_prefixes.add(nid)
if len(deduped) >= top_k:
break
return deduped
def get_node_text(
self,
node_id: str,
*,
anchor: bool = False,
) -> tuple[str, dict[str, str] | None]:
"""返回节点原始文本及可选的锚映射表。
SearchToolDispatcher 使用将原始文本和锚映射传给
summarizer.summarize_node()实现引用验证
参数:
node_id: 节点 ID
anchor: True返回带 [cN]/[sN] 锚标的文本并构建 anchor_map
返回:
(text, anchor_map) 元组anchor=False anchor_map None
anchor=True anchor_map {"c1": "行文本", "s1": "字幕行", ...}
异常:
KeyError: 节点不存在
"""
node = self._id_to_node.get(node_id)
if node is None:
raise KeyError(f"节点不存在: {node_id}")
if not anchor:
return self._node_full_text(node), None
anchored_text = self._node_anchored_text(node)
# 解析锚标行 "[c1] xxx" / "[s2] yyy" 构建映射
anchor_map: dict[str, str] = {}
anchor_pattern = re.compile(r"^\[([cs]\d+)\]\s(.+)$")
for line in anchored_text.splitlines():
m = anchor_pattern.match(line)
if m:
anchor_map[m.group(1)] = m.group(2)
return anchored_text, anchor_map
def get_children_info(self, node_id: str) -> list[dict[str, Any]]:
"""返回节点的直接子节点结构化信息。
SearchToolDispatcher 使用将子节点列表传给
summarizer.summarize_children()用于层级摘要
参数:
node_id: 节点 ID
返回:
子节点信息列表每项包含 {"id", "time_range", "summary"}
time_range (start, end) 数值元组L3 节点退化为 (ts, ts)
L3 叶子节点返回空列表
异常:
KeyError: 节点不存在
"""
node = self._id_to_node.get(node_id)
if node is None:
raise KeyError(f"节点不存在: {node_id}")
children = self._get_children(node)
result: list[dict[str, Any]] = []
for child in children:
desc = _node_description(child)
if len(desc) > 120:
desc = desc[:120] + "..."
result.append(
{
"id": child.id,
"time_range": self._node_time_range_raw(child),
"summary": desc,
}
)
return result
def get_subtitle(self, node_id: str) -> str:
"""返回节点字幕文本。
L2/L3 节点从 card.subtitle 读取L1 节点不含字幕
参数:
node_id: 节点 ID
返回:
字幕文本无字幕L1 节点或节点不存在时返回空字符串
"""
node = self._id_to_node.get(node_id)
if node is None:
return ""
if isinstance(node, (L2Node, L3Node)):
return node.card.subtitle or ""
return ""
def resolve_frame_paths(self, node_ids: list[str]) -> list[Path]:
"""node_id → 帧文件路径。支持 L3(直接映射)和 L2(展开为 L3 children)。
参数:
node_ids: 节点 ID 列表
返回:
帧文件 Path 列表
异常:
KeyError: 节点不存在
"""
if not node_ids:
return []
paths: list[Path] = []
for nid in node_ids:
node = self._id_to_node.get(nid)
if node is None:
raise KeyError(f"节点不存在: {nid}")
if isinstance(node, L3Node):
paths.append(self._l3_frame_path(node))
elif isinstance(node, L2Node):
# 展开为所有 L3 子节点
for l3 in node.children:
paths.append(self._l3_frame_path(l3))
else:
# L1 节点:展开为所有 L2 下的 L3
assert isinstance(node, L1Node)
for l2 in node.children:
for l3 in l2.children:
paths.append(self._l3_frame_path(l3))
return paths
# ------------------------------------------------------------------
# 内部辅助方法
# ------------------------------------------------------------------
def _l3_frame_path(self, node: L3Node) -> Path:
"""将 L3 节点映射到帧文件路径。
参数:
node: L3 节点
返回:
帧文件 Path
"""
if self._frames_dir is not None:
# 从 node.id 中提取后缀(去掉 video_id 前缀)
# ID 格式: {video_id}_{L1_xxx_L2_xxx_L3_xxx}
# frame_path 格式: frames/{L1_xxx_L2_xxx_L3_xxx}.jpg
if node.frame_path:
return self._frames_dir / Path(node.frame_path).name
# fallback: 从 ID 推断
parts = node.id.split("_", 1)
suffix = parts[1] if len(parts) > 1 else node.id
return self._frames_dir / f"{suffix}.jpg"
# 无 frames_dir 时使用节点自带路径
if node.frame_path:
return Path(node.frame_path)
raise ValueError(f"L3 节点无 frame_path 且未提供 frames_dir: {node.id}")
def _node_full_text(self, node: AnyNode) -> str:
"""获取节点完整文本(card 所有字段 + subtitle)。
subtitle card.subtitle 读取 L2/L3 节点附加"字幕:"标签
参数:
node: 树节点
返回:
拼接后的全文本
"""
card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP)
text = "\n".join(card_strings)
if isinstance(node, (L2Node, L3Node)) and node.card.subtitle:
text += f"\n字幕: {node.card.subtitle}"
return text
def _node_anchored_text(self, node: AnyNode) -> str:
"""获取带行号锚的节点文本。
card 字符串逐行编 [c1]..[cN]字幕逐行编 [s1]..[sM]
字幕从 card.subtitle 读取 L2/L3 节点产生 [sN] 锚标
参数:
node: 树节点
返回:
带锚文本
"""
card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP)
# 拆分内嵌换行,确保一锚一行
card_lines: list[str] = []
for s in card_strings:
card_lines.extend(ln for ln in s.splitlines() if ln.strip())
sub_lines: list[str] = []
if isinstance(node, (L2Node, L3Node)) and node.card.subtitle:
sub_lines = [ln for ln in node.card.subtitle.splitlines() if ln.strip()]
anchored: list[str] = []
for i, line in enumerate(card_lines, 1):
anchored.append(f"[c{i}] {line}")
for i, line in enumerate(sub_lines, 1):
anchored.append(f"[s{i}] {line}")
return "\n".join(anchored)
@staticmethod
def _format_time_range(node: AnyNode) -> str:
"""格式化节点的时间范围。
参数:
node: 树节点
返回:
"start-end s" 格式字符串 timestamp "N/A"
"""
if isinstance(node, (L1Node, L2Node)) and node.time_range:
return f"{node.time_range[0]:.1f}-{node.time_range[1]:.1f}s"
if isinstance(node, L3Node) and node.timestamp is not None:
return f"{node.timestamp:.1f}s"
return "N/A"
@staticmethod
def _node_time_range_raw(node: AnyNode) -> tuple[float, float]:
"""提取节点时间范围的原始数值元组。
L1/L2 返回 time_range 元组L3 退化为 (timestamp, timestamp)
全部为 None 时兜底 (0.0, 0.0)
参数:
node: 树节点
返回:
(start, end) 秒级数值元组
"""
if isinstance(node, (L1Node, L2Node)) and node.time_range:
return node.time_range
if isinstance(node, L3Node) and node.timestamp is not None:
return (node.timestamp, node.timestamp)
return (0.0, 0.0)
@staticmethod
def _get_children(node: AnyNode) -> list[AnyNode]:
"""获取节点的直接子节点列表。
参数:
node: 树节点
返回:
子节点列表L3 节点返回空列表
"""
if isinstance(node, L1Node):
return list(node.children)
if isinstance(node, L2Node):
return list(node.children)
return []
+774
View File
@@ -0,0 +1,774 @@
"""三层树索引核心数据结构。
定义 Video-Tree-TRM 的三层树状索引结构是所有后续模块
builderharnesssearch的基础依赖
数据结构层次::
TreeIndex
List[L1Node] 全局叙事节点
List[L2Node] 片段级语义节点
List[L3Node] /细节级节点
与参考项目 (TRM4) 的关键区别:
- Card 体系每层节点的描述信息封装为 frozen dataclassL1Card/L2Card/L3Card
字段来自 VLM 结构化输出保证不可变
- 序列化方式仅保留 JSON移除 pickle
- 统一嵌入空间所有 embedding 均来自 text_embed()无跨模态问题
"""
from __future__ import annotations
import base64
import dataclasses
import json
from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, Any
import numpy as np
from loguru import logger
if TYPE_CHECKING:
from collections.abc import Callable
# ---------------------------------------------------------------------------
# Embedding 序列化辅助函数
# ---------------------------------------------------------------------------
def _embed_to_str(arr: np.ndarray | None) -> str | None:
"""float32 ndarray -> base64 字符串(用于 JSON 序列化)。
参数:
arr: float32 数组形状任意
返回:
base64 编码字符串 None输入为 None
"""
if arr is None:
return None
return base64.b64encode(arr.astype(np.float32).tobytes()).decode()
def _embed_from_str(s: str | None) -> np.ndarray | None:
"""base64 字符串 -> float32 ndarray(用于 JSON 反序列化)。
参数:
s: base64 编码字符串
返回:
float32 数组 None输入为 None/空时
"""
if s is None or s == "":
return None
return np.frombuffer(base64.b64decode(s), dtype=np.float32)
# ---------------------------------------------------------------------------
# Card 数据结构(frozen,来自 VLM 结构化输出)
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class L3Card:
"""L3 帧级语义卡片(不可变)。
封装 VLM 对单帧的结构化描述输出
属性:
frame_summary: 帧内容摘要
visible_entities: 可见实体列表
ongoing_actions: 正在进行的动作列表
visible_text: 画面中可见的文字列表
spatial_layout: 空间布局描述
visual_attributes: 视觉属性字典如光照色调等
subtitle: 字幕文本Voronoi 分配后填充默认空
"""
frame_summary: str
visible_entities: list[str]
ongoing_actions: list[str]
visible_text: list[str]
spatial_layout: str
visual_attributes: dict[str, Any]
subtitle: str = ""
@dataclass(frozen=True)
class L2Card:
"""L2 事件级语义卡片(不可变)。
封装 VLM 对一个事件片段的结构化描述输出
属性:
event_description: 事件描述
entities: 参与实体列表
actions: 动作列表
action_subjects: 动作主体列表
visible_text: 片段中可见的文字列表
spatial_relations: 空间关系描述
state_changes: 状态变化描述可选
subtitle: L3 字幕聚合文本Voronoi 分配后填充默认空
"""
event_description: str
entities: list[str]
actions: list[str]
action_subjects: list[str]
visible_text: list[str]
spatial_relations: str
state_changes: str | None
subtitle: str = ""
@dataclass(frozen=True)
class L1Card:
"""L1 场景级语义卡片(不可变)。
封装 VLM 对一个完整场景的结构化描述输出
属性:
scene_summary: 场景摘要
main_setting: 主要场景设定"室内""户外"
key_entities: 关键实体列表
main_actions: 主要动作列表
topic_keywords: 主题关键词列表
visible_text: 场景中可见的文字列表
temporal_flow: 时间流描述
"""
scene_summary: str
main_setting: str
key_entities: list[str]
main_actions: list[str]
topic_keywords: list[str]
visible_text: list[str]
temporal_flow: str
# ---------------------------------------------------------------------------
# 元数据
# ---------------------------------------------------------------------------
@dataclass
class IndexMeta:
"""树索引元数据。
属性:
source_path: 原始数据路径视频文件或文本文件
modality: 数据模态"text" "video"
embed_model: 嵌入模型名称建树时为 Noneembed_all 后填充
embed_dim: 嵌入向量维度建树时为 Noneembed_all 后填充
created_at: 创建时间ISO 格式字符串
"""
source_path: str
modality: str
embed_model: str | None = None
embed_dim: int | None = None
created_at: str = field(default_factory=lambda: datetime.now().isoformat())
# ---------------------------------------------------------------------------
# 节点数据结构
# ---------------------------------------------------------------------------
@dataclass
class L3Node:
"""L3 帧/细节级节点(叶子层)。
代表最细粒度的语义单元对应一个具体的帧描述
属性:
id: 节点唯一标识
card: 帧级语义卡片VLM 结构化输出
embedding: 文本嵌入向量形状 [D]float32
timestamp: 对应的时间戳可选
frame_path: 关联的帧图像路径可选仅视频模态
"""
id: str
card: L3Card
embedding: np.ndarray | None = None
timestamp: float | None = None
frame_path: str | None = None
@property
def description(self) -> str:
"""帧描述文本(取自 card.frame_summary)。"""
return self.card.frame_summary
@dataclass
class L2Node:
"""L2 片段级语义节点(中间层)。
连接 L1 宏观叙事与 L3 细节描述
属性:
id: 节点唯一标识
card: 事件级语义卡片VLM 结构化输出
embedding: 文本嵌入向量形状 [D]float32
time_range: 时间范围 (start, end)可选
children: 所属的 L3 子节点列表
"""
id: str
card: L2Card
embedding: np.ndarray | None = None
time_range: tuple[float, float] | None = None
children: list[L3Node] = field(default_factory=list)
@property
def description(self) -> str:
"""事件描述文本(取自 card.event_description)。"""
return self.card.event_description
@dataclass
class L1Node:
"""L1 全局叙事节点(根层)。
代表最粗粒度的语义单元包含宏观场景摘要
属性:
id: 节点唯一标识
card: 场景级语义卡片VLM 结构化输出
embedding: 文本嵌入向量形状 [D]float32
time_range: 时间范围 (start, end)可选
children: 所属的 L2 子节点列表
"""
id: str
card: L1Card
embedding: np.ndarray | None = None
time_range: tuple[float, float] | None = None
children: list[L2Node] = field(default_factory=list)
@property
def summary(self) -> str:
"""场景摘要文本(取自 card.scene_summary)。"""
return self.card.scene_summary
# ------------------------------------------------------------------
# JSON 辅助方法(单个 L1 段的轻量序列化)
# ------------------------------------------------------------------
def to_dict(self, include_embedding: bool = False) -> dict[str, Any]:
"""将当前 L1 节点(及其全部 L2/L3 子树)序列化为纯 dict。
参数:
include_embedding: True embedding 向量序列化为 base64 字符串
返回:
包含 id/card/time_range/children 的字典可选包含 embedding
"""
def l3_to_dict(n: L3Node) -> dict[str, Any]:
d: dict[str, Any] = {
"id": n.id,
"card": {
"frame_summary": n.card.frame_summary,
"visible_entities": n.card.visible_entities,
"ongoing_actions": n.card.ongoing_actions,
"visible_text": n.card.visible_text,
"spatial_layout": n.card.spatial_layout,
"visual_attributes": n.card.visual_attributes,
"subtitle": n.card.subtitle,
},
"timestamp": n.timestamp,
"frame_path": n.frame_path,
}
if include_embedding:
d["embedding"] = _embed_to_str(n.embedding)
return d
def l2_to_dict(n: L2Node) -> dict[str, Any]:
d: dict[str, Any] = {
"id": n.id,
"card": {
"event_description": n.card.event_description,
"entities": n.card.entities,
"actions": n.card.actions,
"action_subjects": n.card.action_subjects,
"visible_text": n.card.visible_text,
"spatial_relations": n.card.spatial_relations,
"state_changes": n.card.state_changes,
"subtitle": n.card.subtitle,
},
"time_range": list(n.time_range) if n.time_range else None,
"children": [l3_to_dict(c) for c in n.children],
}
if include_embedding:
d["embedding"] = _embed_to_str(n.embedding)
return d
d: dict[str, Any] = {
"id": self.id,
"card": {
"scene_summary": self.card.scene_summary,
"main_setting": self.card.main_setting,
"key_entities": self.card.key_entities,
"main_actions": self.card.main_actions,
"topic_keywords": self.card.topic_keywords,
"visible_text": self.card.visible_text,
"temporal_flow": self.card.temporal_flow,
},
"time_range": list(self.time_range) if self.time_range else None,
"children": [l2_to_dict(c) for c in self.children],
}
if include_embedding:
d["embedding"] = _embed_to_str(self.embedding)
return d
@staticmethod
def from_dict(d: dict[str, Any]) -> L1Node:
"""从 dict 反序列化单个 L1 节点(支持 embedding 恢复)。
参数:
d: to_dict() 输出的字典可包含 embedding 字段
返回:
L1Node 实例embedding 自动从 base64 恢复若无则为 None
"""
l2_nodes: list[L2Node] = []
for l2d in d.get("children", []):
l3_nodes: list[L3Node] = []
for l3d in l2d.get("children", []):
# 向后兼容:旧格式 subtitle 在节点级,新格式在 card 内
l3_subtitle = l3d["card"].get("subtitle", "") or l3d.get("subtitle", "") or ""
l3_card = L3Card(
frame_summary=l3d["card"]["frame_summary"],
visible_entities=l3d["card"]["visible_entities"],
ongoing_actions=l3d["card"]["ongoing_actions"],
visible_text=l3d["card"]["visible_text"],
spatial_layout=l3d["card"]["spatial_layout"],
visual_attributes=l3d["card"]["visual_attributes"],
subtitle=l3_subtitle,
)
l3_nodes.append(
L3Node(
id=l3d["id"],
card=l3_card,
embedding=_embed_from_str(l3d.get("embedding")),
timestamp=l3d.get("timestamp"),
frame_path=l3d.get("frame_path"),
)
)
l2_card = L2Card(
event_description=l2d["card"]["event_description"],
entities=l2d["card"]["entities"],
actions=l2d["card"]["actions"],
action_subjects=l2d["card"]["action_subjects"],
visible_text=l2d["card"]["visible_text"],
spatial_relations=l2d["card"]["spatial_relations"],
state_changes=l2d["card"]["state_changes"],
subtitle=l2d["card"].get("subtitle", ""),
)
# 旧格式兼容:L2 无 card subtitle 时从 L3 children 聚合
if not l2_card.subtitle:
l3_subs = [n.card.subtitle for n in l3_nodes if n.card.subtitle]
if l3_subs:
l2_card = dataclasses.replace(l2_card, subtitle="\n".join(l3_subs))
tr2 = l2d.get("time_range")
l2_nodes.append(
L2Node(
id=l2d["id"],
card=l2_card,
embedding=_embed_from_str(l2d.get("embedding")),
time_range=tuple(tr2) if tr2 else None,
children=l3_nodes,
)
)
l1_card = L1Card(
scene_summary=d["card"]["scene_summary"],
main_setting=d["card"]["main_setting"],
key_entities=d["card"]["key_entities"],
main_actions=d["card"]["main_actions"],
topic_keywords=d["card"]["topic_keywords"],
visible_text=d["card"]["visible_text"],
temporal_flow=d["card"]["temporal_flow"],
)
tr1 = d.get("time_range")
return L1Node(
id=d["id"],
card=l1_card,
embedding=_embed_from_str(d.get("embedding")),
time_range=tuple(tr1) if tr1 else None,
children=l2_nodes,
)
# ---------------------------------------------------------------------------
# 树索引容器
# ---------------------------------------------------------------------------
@dataclass
class TreeIndex:
"""三层树索引容器。
组织和管理三层节点结构提供嵌入矩阵提取节点访问
以及 JSON 序列化/反序列化接口
典型工作流::
# 1. 构建索引
index = TreeIndex(metadata=meta, roots=[l1_node_1, l1_node_2])
# 2. 批量 embed(首次检索前)
index.embed_all(embed_fn, "model-name", 768)
# 3. 提取嵌入矩阵(用于检索)
M_L1 = index.l1_embeddings()
M_L2 = index.l2_embeddings_of(l1_idx=0)
M_L3 = index.l3_embeddings_of(0, 1)
# 4. 序列化
index.save_json("cache/my_index.json")
loaded = TreeIndex.load_json("cache/my_index.json")
属性:
metadata: 索引元数据
roots: L1 节点列表
"""
metadata: IndexMeta
roots: list[L1Node] = field(default_factory=list)
# ------------------------------------------------------------------ #
# 嵌入状态检查
# ------------------------------------------------------------------ #
@property
def is_embedded(self) -> bool:
"""检查所有节点是否已填充嵌入向量。
返回:
True 表示所有 L1/L2/L3 节点的 embedding 均非 None
False 表示尚未 embed
"""
for l1 in self.roots:
if l1.embedding is None:
return False
for l2 in l1.children:
if l2.embedding is None:
return False
for l3 in l2.children:
if l3.embedding is None:
return False
return True
# ------------------------------------------------------------------ #
# 批量嵌入
# ------------------------------------------------------------------ #
def embed_all(
self,
embed_fn: Callable[[str | list[str]], np.ndarray],
model_name: str,
embed_dim: int,
) -> None:
"""对所有节点批量执行 embedding,更新 metadata。
建树阶段不调用此方法embedding=None
首次检索前由 Pipeline 调用结果缓存在节点上
参数:
embed_fn: EmbeddingModel.embed 方法接受 str List[str]
返回 [N, D] ndarray
model_name: 嵌入模型名称写入 metadata
embed_dim: 嵌入维度写入 metadata
实现细节:
- L3 节点按 L2 分组批量 embed一次调用减少 API 开销
- L1/L2 各单独 embed数量少不值得合并
- 仅对 embedding None 的节点执行支持增量更新
"""
assert len(self.roots) > 0, "embed_all: 树为空,无节点可 embed"
for l1 in self.roots:
if l1.embedding is None:
l1.embedding = embed_fn(l1.summary)[0].astype(np.float32)
for l2 in l1.children:
self._embed_l2_subtree(l2, embed_fn)
self.metadata.embed_model = model_name
self.metadata.embed_dim = embed_dim
logger.info(
"embed_all 完成",
model=model_name,
embed_dim=embed_dim,
)
def _embed_l2_subtree(
self,
l2: L2Node,
embed_fn: Callable[[str | list[str]], np.ndarray],
) -> None:
"""对单个 L2 节点及其 L3 子节点执行 embedding(仅处理 embedding 为 None 的节点)。
参数:
l2: embed L2 节点
embed_fn: EmbeddingModel.embed 方法接受 str List[str]
返回 [N, D] ndarray
"""
if l2.embedding is None:
l2.embedding = embed_fn(l2.description)[0].astype(np.float32)
# L3 批量 embed
need_embed = [l3 for l3 in l2.children if l3.embedding is None]
if need_embed:
texts = [l3.description for l3 in need_embed]
embs = embed_fn(texts).astype(np.float32) # [N, D]
for l3, emb in zip(need_embed, embs, strict=True):
l3.embedding = emb
# ------------------------------------------------------------------ #
# 嵌入矩阵提取
# ------------------------------------------------------------------ #
def l1_embeddings(self) -> np.ndarray:
"""返回所有 L1 节点的嵌入矩阵。
返回:
形状 [N1, D] float32 矩阵空树返回 [0, D]
异常:
AssertionError: 节点 embedding 尚未计算请先调用 embed_all
"""
assert self.is_embedded, "L1 embedding 尚未计算,请先调用 tree.embed_all()"
if not self.roots:
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
return np.stack([r.embedding for r in self.roots], axis=0).astype(np.float32)
def l2_embeddings_of(self, l1_idx: int) -> np.ndarray:
"""返回指定 L1 节点下所有 L2 子节点的嵌入矩阵。
参数:
l1_idx: L1 节点索引
返回:
形状 [N2, D] float32 矩阵
异常:
IndexError: l1_idx 越界
AssertionError: embedding 尚未计算
"""
assert self.is_embedded, "L2 embedding 尚未计算,请先调用 tree.embed_all()"
if not (0 <= l1_idx < len(self.roots)):
raise IndexError(f"l1_idx={l1_idx} 越界,L1 节点数={len(self.roots)}")
children = self.roots[l1_idx].children
if not children:
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
return np.stack([c.embedding for c in children], axis=0).astype(np.float32)
def l3_embeddings_of(self, l1_idx: int, l2_idx: int) -> np.ndarray:
"""返回指定 L2 节点下所有 L3 子节点的嵌入矩阵。
参数:
l1_idx: L1 节点索引
l2_idx: L2 节点索引相对于 L1
返回:
形状 [N3, D] float32 矩阵
异常:
IndexError: 索引越界
AssertionError: embedding 尚未计算
"""
assert self.is_embedded, "L3 embedding 尚未计算,请先调用 tree.embed_all()"
if not (0 <= l1_idx < len(self.roots)):
raise IndexError(f"l1_idx={l1_idx} 越界,L1 节点数={len(self.roots)}")
l2_children = self.roots[l1_idx].children
if not (0 <= l2_idx < len(l2_children)):
raise IndexError(f"l2_idx={l2_idx} 越界,L2 节点数={len(l2_children)}")
l3_children = l2_children[l2_idx].children
if not l3_children:
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
return np.stack([c.embedding for c in l3_children], axis=0).astype(np.float32)
# ------------------------------------------------------------------ #
# 节点访问
# ------------------------------------------------------------------ #
def get_node(self, l1: int, l2: int, l3: int) -> L3Node:
"""按三级路径索引获取 L3 节点。
参数:
l1: L1 节点索引
l2: L2 节点索引
l3: L3 节点索引
返回:
目标 L3Node
异常:
IndexError: 任意层级索引越界
"""
if l1 < 0 or l1 >= len(self.roots):
raise IndexError(f"l1={l1} 越界,L1 节点数={len(self.roots)}")
l2_children = self.roots[l1].children
if l2 < 0 or l2 >= len(l2_children):
raise IndexError(f"l2={l2} 越界,L2 节点数={len(l2_children)}")
l3_children = l2_children[l2].children
if l3 < 0 or l3 >= len(l3_children):
raise IndexError(f"l3={l3} 越界,L3 节点数={len(l3_children)}")
return l3_children[l3]
# ------------------------------------------------------------------ #
# JSON 序列化
# ------------------------------------------------------------------ #
def to_dict(self, include_embedding: bool = False) -> dict[str, Any]:
"""将树索引序列化为纯 Python dict。
参数:
include_embedding: True将所有节点的 embedding 向量序列化为 base64
返回:
可直接 json.dump 的字典结构为 {metadata, roots[...]}
"""
metadata_dict: dict[str, Any] = {
"source_path": self.metadata.source_path,
"modality": self.metadata.modality,
"created_at": self.metadata.created_at,
}
if include_embedding:
metadata_dict["embed_model"] = self.metadata.embed_model
metadata_dict["embed_dim"] = self.metadata.embed_dim
return {
"metadata": metadata_dict,
"roots": [r.to_dict(include_embedding=include_embedding) for r in self.roots],
}
@classmethod
def from_dict(cls, d: dict[str, Any]) -> TreeIndex:
"""从 dict 反序列化为 TreeIndex(支持 embedding 恢复)。
参数:
d: to_dict() 的输出或等价结构可包含 embedding 字段
返回:
TreeIndex 实例
异常:
ValueError: 存在重复的节点 ID
"""
meta = IndexMeta(
source_path=d["metadata"]["source_path"],
modality=d["metadata"]["modality"],
embed_model=d["metadata"].get("embed_model"),
embed_dim=d["metadata"].get("embed_dim"),
created_at=d["metadata"].get("created_at", datetime.now().isoformat()),
)
roots: list[L1Node] = []
for r in d["roots"]:
roots.append(L1Node.from_dict(r))
obj = cls(metadata=meta, roots=roots)
obj._validate_id_uniqueness()
return obj
def _validate_id_uniqueness(self) -> None:
"""校验树中所有节点 ID 的唯一性。
异常:
ValueError: 存在重复的节点 ID
"""
seen: set[str] = set()
for l1 in self.roots:
if l1.id in seen:
raise ValueError(f"重复的节点 ID: {l1.id}")
seen.add(l1.id)
for l2 in l1.children:
if l2.id in seen:
raise ValueError(f"重复的节点 ID: {l2.id}")
seen.add(l2.id)
for l3 in l2.children:
if l3.id in seen:
raise ValueError(f"重复的节点 ID: {l3.id}")
seen.add(l3.id)
def save_json(self, path: str, include_embedding: bool = False) -> None:
"""将树索引以 JSON 格式保存到磁盘。
参数:
path: 保存文件路径推荐 .json 后缀
include_embedding: True将所有节点的 embedding 向量保存到 JSON
"""
with open(path, "w", encoding="utf-8") as f:
json.dump(
self.to_dict(include_embedding=include_embedding),
f,
ensure_ascii=False,
indent=2,
)
logger.info(
"树索引(JSON)已保存至 {}",
path,
n_l1=len(self.roots),
include_embedding=include_embedding,
)
@classmethod
def load_json(cls, path: str) -> TreeIndex:
"""从 JSON 文件加载树索引(自动检测并恢复 embedding)。
参数:
path: JSON 文件路径
返回:
TreeIndex 实例 JSON 中包含 embedding 字段自动反序列化填充
否则 embedding=None向后兼容旧格式
异常:
FileNotFoundError: 文件不存在
ValueError: 存在重复的节点 ID
"""
with open(path, encoding="utf-8") as f:
d = json.load(f)
obj = cls.from_dict(d)
obj._validate_id_uniqueness()
logger.info(
"树索引(JSON)已从 {} 加载",
path,
n_l1=len(obj.roots),
is_embedded=obj.is_embedded,
)
return obj
# ---------------------------------------------------------------------------
# 单 L1 段的轻量序列化(用于断点续跑)
# ---------------------------------------------------------------------------
def save_l1_json(path: str, l1_node: L1Node) -> None:
"""将单个 L1 节点(及其子树)以 JSON 形式保存到磁盘。
参数:
path: 目标文件路径
l1_node: 待序列化的 L1 节点
"""
with open(path, "w", encoding="utf-8") as f:
json.dump(l1_node.to_dict(), f, ensure_ascii=False, indent=2)
logger.info("L1 中间结果已保存", path=path, l1_id=l1_node.id)
def load_l1_json(path: str) -> L1Node:
"""从 JSON 文件加载单个 L1 节点(embedding=None)。
参数:
path: JSON 文件路径
返回:
L1Node 实例
"""
with open(path, encoding="utf-8") as f:
data = json.load(f)
node = L1Node.from_dict(data)
logger.info("L1 中间结果已加载", path=path, l1_id=node.id)
return node
+174
View File
@@ -0,0 +1,174 @@
"""树修复检测器:扫描 TreeIndex 识别缺失/低质量节点。"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from pathlib import Path
from app.tree.index import TreeIndex
# 相邻 L2 片段之间允许的最大时间间隙(秒)
_MAX_TIME_GAP_S = 1.0
@dataclass(frozen=True)
class NodeIssue:
"""检测到的节点问题。
参数:
node_id: 问题节点 ID
level: 节点层级1/2/3
issue_type: 问题类型
details: 详细描述
"""
node_id: str
level: int
issue_type: str # "empty_field" | "missing_frame" | "no_children" | "time_gap"
details: str
def detect_issues(
index: TreeIndex,
frames_dir: Path | None = None,
) -> list[NodeIssue]:
"""扫描树,返回所有问题节点列表。
检查项:
- L3: card 必填字段为空frame_summary / spatial_layout
- : visible_entities / ongoing_actions 为空是合法状态静物/黑帧不纳入检测
- L3: frame_path 对应文件不存在需提供 frames_dir
- L2: event_description 为空
- L2/L1: children 列表为空
- L2: 相邻 clips 时间范围不连续gap > 1
- L1: scene_summary 为空
参数:
index: 待检测的 TreeIndex
frames_dir: 帧文件根目录可选提供时检查帧文件存在性
返回:
问题列表 level 降序L3 L2 L1排列
"""
issues: list[NodeIssue] = []
for l1 in index.roots:
# L1: scene_summary 不为空
if not l1.card.scene_summary:
issues.append(
NodeIssue(
node_id=l1.id,
level=1,
issue_type="empty_field",
details="L1 节点字段为空: scene_summary",
)
)
# L1: children 不为空
if not l1.children:
issues.append(
NodeIssue(
node_id=l1.id,
level=1,
issue_type="no_children",
details="L1 节点无 L2 子节点",
)
)
continue
# L2: 相邻 clips 时间间隙检查
_check_time_gaps(l1.children, issues)
for l2 in l1.children:
# L2: event_description 不为空
if not l2.card.event_description:
issues.append(
NodeIssue(
node_id=l2.id,
level=2,
issue_type="empty_field",
details="L2 节点字段为空: event_description",
)
)
# L2: children 不为空
if not l2.children:
issues.append(
NodeIssue(
node_id=l2.id,
level=2,
issue_type="no_children",
details="L2 节点无 L3 子节点",
)
)
continue
for l3 in l2.children:
# L3: 核心必填字段不为空(visible_entities/ongoing_actions 为空是合法状态)
empty_fields: list[str] = []
if not l3.card.frame_summary:
empty_fields.append("frame_summary")
if not l3.card.spatial_layout:
empty_fields.append("spatial_layout")
if empty_fields:
issues.append(
NodeIssue(
node_id=l3.id,
level=3,
issue_type="empty_field",
details=f"L3 节点字段为空: {', '.join(empty_fields)}",
)
)
# L3: frame_path 文件存在性
if (
frames_dir is not None
and l3.frame_path is not None
and not (frames_dir / l3.frame_path).exists()
):
issues.append(
NodeIssue(
node_id=l3.id,
level=3,
issue_type="missing_frame",
details=f"帧文件不存在: {l3.frame_path}",
)
)
# 按 level 降序排列(L3=3 → L2=2 → L1=1
issues.sort(key=lambda i: -i.level)
logger.info("树缺陷检测完成,发现 {} 个问题", len(issues))
return issues
def _check_time_gaps(
l2_nodes: list,
issues: list[NodeIssue],
) -> None:
"""检查同一 L1 下相邻 L2 节点之间的时间间隙。
参数:
l2_nodes: 同一 L1 节点下的 L2 子节点列表
issues: 问题列表原地追加
"""
for i in range(len(l2_nodes) - 1):
curr = l2_nodes[i]
nxt = l2_nodes[i + 1]
if curr.time_range is None or nxt.time_range is None:
continue
gap = nxt.time_range[0] - curr.time_range[1]
if gap > _MAX_TIME_GAP_S:
issues.append(
NodeIssue(
node_id=nxt.id,
level=2,
issue_type="time_gap",
details=f"与前一片段间隙 {gap:.1f}s(阈值 {_MAX_TIME_GAP_S}s",
)
)
+513
View File
@@ -0,0 +1,513 @@
"""树修复重生成器:VLM 重新描述问题节点 + 底向上级联。
底向上修复流程:
1. 收集需修复的 L3 节点 VLM 重新描述帧
2. 收集受影响的 L2 LLM L3 children 聚合
3. 收集受影响的 L1 LLM L2 children 聚合
仅处理 issue_type == "empty_field" level == 3 的问题节点
帧文件不存在时跳过该节点不中断整体修复流程
"""
from __future__ import annotations
import json
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from loguru import logger
from app.tree.index import (
L1Card,
L1Node,
L2Card,
L2Node,
L3Card,
L3Node,
TreeIndex,
)
from app.tree.subtitle import extract_subtitle_for_range
if TYPE_CHECKING:
from pathlib import Path
from app.tree.repair.detector import NodeIssue
from app.tree.subtitle import SRTEntry
from core.protocols import LLMProvider, VLMProvider
# ---------------------------------------------------------------------------
# Prompt 常量(与 VideoTreeBuilder 保持一致风格)
# ---------------------------------------------------------------------------
_L3_REPAIR_PROMPT = (
'该片段的整体内容: "{l2_description}"\n'
"用一到两句话描述这帧画面的具体内容。"
"重点关注: 动作、物体变化、文字信息、人物表情。\n"
"{subtitle_block}"
"返回 JSON 对象,包含以下字段:\n"
"- frame_summary: 画面描述\n"
"- visible_entities: 可见实体列表\n"
"- ongoing_actions: 动作列表\n"
"- visible_text: 可见文字列表\n"
"- spatial_layout: 空间布局\n"
'- visual_attributes: {{"lighting": "...", "dominant_colors": [...], "camera_angle": "..."}}\n'
"只返回 JSON 对象,不要其他内容。"
)
_L2_REGEN_PROMPT = (
"以下是一个视频片段中各帧的描述:\n{l3_texts}\n"
"用1-2句话描述该片段的核心内容。\n"
"返回 JSON 对象,包含以下字段:\n"
"- event_description: 1-2句片段描述\n"
"- entities: 可见实体列表\n"
"- actions: 动作列表\n"
"- action_subjects: 动作主体列表\n"
"- visible_text: 画面中可见文字列表\n"
"- spatial_relations: 空间关系描述\n"
"- state_changes: 状态变化描述(无则 null)\n"
"只返回 JSON 对象,不要其他内容。"
)
_L1_REGEN_PROMPT = (
"以下是一个视频段落中各片段的描述:\n{l2_texts}\n"
"用2-3句话总结该段落的整体内容,涵盖所有片段的主题。\n"
"返回 JSON 对象,包含以下字段:\n"
"- scene_summary: 2-3句段落摘要\n"
"- main_setting: 主要场景\n"
"- key_entities: 关键实体列表\n"
"- main_actions: 主要动作列表\n"
"- topic_keywords: 主题关键词列表\n"
"- visible_text: 出现的文字列表\n"
"- temporal_flow: 时间流向描述\n"
"只返回 JSON 对象,不要其他内容。"
)
# ---------------------------------------------------------------------------
# 统计数据类
# ---------------------------------------------------------------------------
@dataclass
class RepairStats:
"""修复统计信息。
属性:
l3_repaired: 修复的 L3 节点数
l2_regenerated: 重生成的 L2 节点数
l1_regenerated: 重生成的 L1 节点数
"""
l3_repaired: int = 0
l2_regenerated: int = 0
l1_regenerated: int = 0
# ---------------------------------------------------------------------------
# JSON 解析辅助(复用 VideoTreeBuilder 的解析逻辑)
# ---------------------------------------------------------------------------
def _extract_json(raw: str) -> Any:
"""从 VLM/LLM 原始输出中提取 JSON(处理 markdown 代码块包裹)。
参数:
raw: 原始返回字符串
返回:
解析后的 Python 对象dict/list解析失败返回 None
"""
raw = raw.strip()
# Phase 1: 尝试提取 markdown 代码块中的 JSON
code_match = re.search(
r"```(?:json)?\s*([\[{].*?[\]}])\s*```",
raw,
re.DOTALL,
)
if code_match:
raw = code_match.group(1)
# Phase 2: 直接解析
try:
return json.loads(raw)
except json.JSONDecodeError:
pass
# Phase 3: 尝试提取裸 JSON 对象/数组
json_match = re.search(r"[\[{].*[\]}]", raw, re.DOTALL)
if json_match:
try:
return json.loads(json_match.group())
except json.JSONDecodeError:
pass
return None
def _parse_l3_card(raw: str) -> L3Card | None:
"""解析 VLM 输出为 L3Card。解析失败返回 None。
参数:
raw: VLM 原始返回字符串
返回:
L3Card 实例或 None解析失败时
"""
data = _extract_json(raw)
if isinstance(data, dict):
try:
return L3Card(
frame_summary=str(data["frame_summary"]),
visible_entities=list(data["visible_entities"]),
ongoing_actions=list(data["ongoing_actions"]),
visible_text=list(data["visible_text"]),
spatial_layout=str(data["spatial_layout"]),
visual_attributes=dict(data["visual_attributes"]),
)
except (KeyError, TypeError, ValueError):
pass
return None
def _parse_l2_card(raw: str) -> L2Card | None:
"""解析 LLM 输出为 L2Card。解析失败返回 None。
参数:
raw: LLM 原始返回字符串
返回:
L2Card 实例或 None解析失败时
"""
data = _extract_json(raw)
if isinstance(data, dict):
try:
state_changes = data.get("state_changes")
if state_changes is not None:
state_changes = str(state_changes)
return L2Card(
event_description=str(data["event_description"]),
entities=list(data["entities"]),
actions=list(data["actions"]),
action_subjects=list(data["action_subjects"]),
visible_text=list(data["visible_text"]),
spatial_relations=str(data["spatial_relations"]),
state_changes=state_changes,
)
except (KeyError, TypeError, ValueError):
pass
return None
def _parse_l1_card(raw: str) -> L1Card | None:
"""解析 LLM 输出为 L1Card。解析失败返回 None。
参数:
raw: LLM 原始返回字符串
返回:
L1Card 实例或 None解析失败时
"""
data = _extract_json(raw)
if isinstance(data, dict):
try:
return L1Card(
scene_summary=str(data["scene_summary"]),
main_setting=str(data["main_setting"]),
key_entities=list(data["key_entities"]),
main_actions=list(data["main_actions"]),
topic_keywords=list(data["topic_keywords"]),
visible_text=list(data["visible_text"]),
temporal_flow=str(data["temporal_flow"]),
)
except (KeyError, TypeError, ValueError):
pass
return None
# ---------------------------------------------------------------------------
# 节点查找辅助
# ---------------------------------------------------------------------------
def _build_node_lookup(
index: TreeIndex,
) -> tuple[
dict[str, L3Node],
dict[str, L2Node],
dict[str, L1Node],
dict[str, L2Node],
dict[str, L1Node],
]:
"""构建节点 ID 到节点的查找表 + 子节点到父节点的映射。
参数:
index: 树索引
返回:
(l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1)
- l3_by_id: L3 节点 ID L3Node
- l2_by_id: L2 节点 ID L2Node
- l1_by_id: L1 节点 ID L1Node
- l3_parent_l2: L3 节点 ID 其父 L2Node
- l2_parent_l1: L2 节点 ID 其父 L1Node
"""
l3_by_id: dict[str, L3Node] = {}
l2_by_id: dict[str, L2Node] = {}
l1_by_id: dict[str, L1Node] = {}
l3_parent_l2: dict[str, L2Node] = {}
l2_parent_l1: dict[str, L1Node] = {}
for l1 in index.roots:
l1_by_id[l1.id] = l1
for l2 in l1.children:
l2_by_id[l2.id] = l2
l2_parent_l1[l2.id] = l1
for l3 in l2.children:
l3_by_id[l3.id] = l3
l3_parent_l2[l3.id] = l2
return l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1
# ---------------------------------------------------------------------------
# 字幕辅助
# ---------------------------------------------------------------------------
def _build_subtitle_block(
srt_entries: list[SRTEntry] | None,
timestamp: float | None,
) -> str:
"""构建字幕注入文本块。
参数:
srt_entries: SRT 字幕条目列表
timestamp: 帧时间戳
返回:
字幕文本块字符串无匹配时返回空字符串
"""
if not srt_entries or timestamp is None:
return ""
window = 2.0
start = max(0.0, timestamp - window)
end = timestamp + window
text = extract_subtitle_for_range(srt_entries, (start, end))
if not text:
return ""
return f"字幕信息:\n{text}\n"
# ---------------------------------------------------------------------------
# 主修复函数
# ---------------------------------------------------------------------------
async def repair_tree(
index: TreeIndex,
issues: list[NodeIssue],
vlm: VLMProvider,
llm: LLMProvider,
frames_dir: Path,
srt_entries: list[SRTEntry] | None = None,
) -> RepairStats:
"""修复有问题的节点,底向上级联。
流程:
1. 收集需修复的 L3 节点 VLM 重新描述帧
2. 收集受影响的 L2 LLM L3 children 聚合
3. 收集受影响的 L1 LLM L2 children 聚合
参数:
index: 待修复的 TreeIndex原地修改
issues: detect_issues() 返回的问题列表
vlm: VLM 调用端口
llm: LLM 调用端口
frames_dir: 帧文件根目录
srt_entries: 字幕条目列表可选
返回:
RepairStats 统计
"""
stats = RepairStats()
if not issues:
logger.info("无修复任务,跳过")
return stats
# 构建查找表
l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1 = _build_node_lookup(index)
# Step 1: 修复 L3 节点(仅处理 empty_field + level 3
l3_issues = [
issue for issue in issues if issue.issue_type == "empty_field" and issue.level == 3
]
affected_l2_ids: set[str] = set()
for issue in l3_issues:
l3_node = l3_by_id.get(issue.node_id)
if l3_node is None:
logger.warning(
"L3 节点 ID 未在树中找到,跳过",
node_id=issue.node_id,
)
continue
# 查找帧文件
if l3_node.frame_path is None:
logger.warning(
"L3 节点无 frame_path,跳过",
node_id=issue.node_id,
)
continue
frame_file = frames_dir / l3_node.frame_path
if not frame_file.exists():
logger.warning(
"L3 帧文件不存在,跳过修复",
node_id=issue.node_id,
frame_path=str(frame_file),
)
continue
# 获取 L2 父节点描述作为上下文
parent_l2 = l3_parent_l2.get(issue.node_id)
l2_description = parent_l2.card.event_description if parent_l2 else ""
# 构建字幕块
subtitle_block = _build_subtitle_block(srt_entries, l3_node.timestamp)
# VLM 重新描述帧
prompt = _L3_REPAIR_PROMPT.format(
l2_description=l2_description,
subtitle_block=subtitle_block,
)
messages = [{"role": "user", "content": prompt}]
try:
response = await vlm.chat_with_images(messages, [str(frame_file)])
except Exception as exc:
logger.warning(
"L3 修复 VLM 调用失败,跳过: {}",
exc,
node_id=issue.node_id,
)
continue
new_card = _parse_l3_card(response.content)
if new_card is None:
logger.warning(
"L3 修复 VLM 输出解析失败,跳过",
node_id=issue.node_id,
raw_preview=response.content[:200],
)
continue
# 原地替换 cardL3Node.card 不是 frozen dataclass 的限制字段)
l3_node.card = new_card
stats.l3_repaired += 1
# 标记受影响的 L2 父节点
if parent_l2 is not None:
affected_l2_ids.add(parent_l2.id)
logger.debug(
"L3 节点修复完成",
node_id=issue.node_id,
frame_summary=new_card.frame_summary[:50],
)
# Step 2: 重生成受影响的 L2 节点
affected_l1_ids: set[str] = set()
for l2_id in affected_l2_ids:
l2_node = l2_by_id.get(l2_id)
if l2_node is None:
continue
# 从 L3 children 聚合描述
l3_texts = "\n".join(f"- {l3.card.frame_summary}" for l3 in l2_node.children)
prompt = _L2_REGEN_PROMPT.format(l3_texts=l3_texts)
messages = [{"role": "user", "content": prompt}]
try:
response = await llm.chat(messages)
except Exception as exc:
logger.warning(
"L2 重生成 LLM 调用失败,跳过: {}",
exc,
l2_id=l2_id,
)
continue
new_card = _parse_l2_card(response.content)
if new_card is None:
logger.warning(
"L2 重生成 LLM 输出解析失败,跳过",
l2_id=l2_id,
raw_preview=response.content[:200],
)
continue
l2_node.card = new_card
stats.l2_regenerated += 1
# 标记受影响的 L1 父节点
parent_l1 = l2_parent_l1.get(l2_id)
if parent_l1 is not None:
affected_l1_ids.add(parent_l1.id)
logger.debug(
"L2 节点重生成完成",
l2_id=l2_id,
event_description=new_card.event_description[:50],
)
# Step 3: 重生成受影响的 L1 节点
for l1_id in affected_l1_ids:
l1_node = l1_by_id.get(l1_id)
if l1_node is None:
continue
# 从 L2 children 聚合描述
l2_texts = "\n".join(f"- {l2.card.event_description}" for l2 in l1_node.children)
prompt = _L1_REGEN_PROMPT.format(l2_texts=l2_texts)
messages = [{"role": "user", "content": prompt}]
try:
response = await llm.chat(messages)
except Exception as exc:
logger.warning(
"L1 重生成 LLM 调用失败,跳过: {}",
exc,
l1_id=l1_id,
)
continue
new_card = _parse_l1_card(response.content)
if new_card is None:
logger.warning(
"L1 重生成 LLM 输出解析失败,跳过",
l1_id=l1_id,
raw_preview=response.content[:200],
)
continue
l1_node.card = new_card
stats.l1_regenerated += 1
logger.debug(
"L1 节点重生成完成",
l1_id=l1_id,
scene_summary=new_card.scene_summary[:50],
)
logger.info(
"树修复完成",
l3_repaired=stats.l3_repaired,
l2_regenerated=stats.l2_regenerated,
l1_regenerated=stats.l1_regenerated,
)
return stats
+527
View File
@@ -0,0 +1,527 @@
"""Q&A 反向补全:基于问题答案分析,将树中缺失的事实注入节点。
通过 LLM 分析正确答案需要哪些关键事实再检查树中是否已有
对缺失事实执行注入仅注入客观事实人名地点得分物体名称
不注入情感因果推理时间推理等主观或高阶信息
TRM4 的关键差异:
- 树结构从扁平 dict 变为 TreeIndexL1Node L2Node L3Node
- Card frozen dataclass注入时使用 dataclasses.replace() 创建新实例
- LLMProvider 为异步接口返回 LLMResponse.content 获取文本
"""
from __future__ import annotations
import json
from dataclasses import dataclass, replace
from typing import TYPE_CHECKING, Any
from loguru import logger
if TYPE_CHECKING:
from app.tree.index import L1Node, L2Node, L3Node, TreeIndex
from core.protocols import LLMProvider
# ---------------------------------------------------------------------------
# 允许注入的类别白名单
# ---------------------------------------------------------------------------
_ALLOWED_CATEGORIES = frozenset(
{
"person_name",
"location",
"score_number",
"object_name",
}
)
# ---------------------------------------------------------------------------
# 类别 → 默认注入字段映射(L2 Card 字段名)
# ---------------------------------------------------------------------------
_CATEGORY_DEFAULT_FIELD: dict[str, str] = {
"person_name": "entities",
"location": "entities",
"score_number": "entities",
"object_name": "entities",
}
# ---------------------------------------------------------------------------
# 统计
# ---------------------------------------------------------------------------
@dataclass
class SupplementStats:
"""反向补全统计信息。
属性:
questions_analyzed: 分析的问题数量
facts_injected: 成功注入的事实数量
facts_skipped: 跳过的事实数量类别不在白名单中
"""
questions_analyzed: int = 0
facts_injected: int = 0
facts_skipped: int = 0
# ---------------------------------------------------------------------------
# 去重
# ---------------------------------------------------------------------------
def deduplicate_field(values: list[str]) -> list[str]:
"""大小写归一化去重,保留首次出现的原始形式。
参数:
values: 待去重字符串列表
返回:
去重后的列表保留各值首次出现时的大小写
空字符串和纯空白字符串会被跳过
"""
seen: set[str] = set()
result: list[str] = []
for v in values:
s = str(v).strip()
key = s.lower()
if key and key not in seen:
seen.add(key)
result.append(s)
return result
# ---------------------------------------------------------------------------
# 节点查找
# ---------------------------------------------------------------------------
def _find_node_by_id(
index: TreeIndex,
node_id: str,
) -> tuple[L1Node | L2Node | L3Node | None, int]:
"""在 TreeIndex 中按 ID 查找节点,返回节点和所属层级。
参数:
index: 树索引
node_id: 目标节点 ID
返回:
(node, level) 元组找不到时返回 (None, -1)
level: 1=L1, 2=L2, 3=L3
"""
for l1 in index.roots:
if l1.id == node_id:
return l1, 1
for l2 in l1.children:
if l2.id == node_id:
return l2, 2
for l3 in l2.children:
if l3.id == node_id:
return l3, 3
return None, -1
# ---------------------------------------------------------------------------
# 单值注入(适配 frozen Card
# ---------------------------------------------------------------------------
def _inject_into_l2(l2: L2Node, field: str, value: str) -> bool:
"""向 L2 节点的 Card 指定字段注入一个值。
使用 dataclasses.replace() 创建新的 frozen L2Card
仅支持 list[str] 类型字段entities / actions / action_subjects / visible_text
str 类型字段event_description / spatial_relations / state_changes
参数:
l2: L2 节点card 会被替换为新实例
field: 目标字段名
value: 要注入的值
返回:
True 表示实际注入了新内容False 表示已存在跳过
"""
card = l2.card
current = getattr(card, field, None)
if current is None:
# 字段不存在于 Card schema,跳过
logger.debug("L2Card 无字段 {},跳过注入", field)
return False
if isinstance(current, list):
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
if value.strip().lower() in lower_set:
return False
new_list = deduplicate_field([*current, value])
l2.card = replace(card, **{field: new_list})
return True
if isinstance(current, str):
if value.strip().lower() in current.lower():
return False
new_val = current + "; " + value if current else value
l2.card = replace(card, **{field: new_val})
return True
return False
def _inject_into_l3(l3: L3Node, field: str, value: str) -> bool:
"""向 L3 节点的 Card 指定字段注入一个值。
使用 dataclasses.replace() 创建新的 frozen L3Card
参数:
l3: L3 节点card 会被替换为新实例
field: 目标字段名
value: 要注入的值
返回:
True 表示实际注入了新内容False 表示已存在跳过
"""
card = l3.card
current = getattr(card, field, None)
if current is None:
logger.debug("L3Card 无字段 {},跳过注入", field)
return False
if isinstance(current, list):
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
if value.strip().lower() in lower_set:
return False
new_list = deduplicate_field([*current, value])
l3.card = replace(card, **{field: new_list})
return True
if isinstance(current, str):
if value.strip().lower() in current.lower():
return False
new_val = current + "; " + value if current else value
l3.card = replace(card, **{field: new_val})
return True
return False
def _inject_into_l1(l1: L1Node, field: str, value: str) -> bool:
"""向 L1 节点的 Card 指定字段注入一个值。
使用 dataclasses.replace() 创建新的 frozen L1Card
参数:
l1: L1 节点card 会被替换为新实例
field: 目标字段名
value: 要注入的值
返回:
True 表示实际注入了新内容False 表示已存在跳过
"""
card = l1.card
current = getattr(card, field, None)
if current is None:
logger.debug("L1Card 无字段 {},跳过注入", field)
return False
if isinstance(current, list):
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
if value.strip().lower() in lower_set:
return False
new_list = deduplicate_field([*current, value])
l1.card = replace(card, **{field: new_list})
return True
if isinstance(current, str):
if value.strip().lower() in current.lower():
return False
new_val = current + "; " + value if current else value
l1.card = replace(card, **{field: new_val})
return True
return False
# ---------------------------------------------------------------------------
# 批量注入
# ---------------------------------------------------------------------------
def apply_injections(index: TreeIndex, injections: list[dict[str, Any]]) -> SupplementStats:
"""执行一组注入指令,将事实写入树节点 Card。
每条指令格式::
{
"category": "person_name" | "location" | "score_number" | "object_name",
"inject_value": "...",
"targets": [{"node_id": "...", "field": "..."}, ...]
}
向后兼容: 若无 targets读取 target_node_id + target_field 构造单目标
参数:
index: TreeIndex 实例节点 Card 会被替换为新实例
injections: 注入指令列表
返回:
注入统计信息
"""
stats = SupplementStats()
for instr in injections:
category = instr.get("category", "")
if category not in _ALLOWED_CATEGORIES:
logger.debug("拒绝非法类别: {}", category)
stats.facts_skipped += 1
continue
inject_value = str(instr.get("inject_value", "")).strip()
if not inject_value:
stats.facts_skipped += 1
continue
# 解析目标列表(兼容新旧格式)
targets = instr.get("targets")
if not targets:
node_id = instr.get("target_node_id", "")
field = instr.get("target_field", "")
if node_id and field:
targets = [{"node_id": node_id, "field": field}]
else:
stats.facts_skipped += 1
continue
for target in targets:
node_id = target.get("node_id", "")
field = target.get("field", "")
node, level = _find_node_by_id(index, node_id)
if node is None:
logger.debug("跳过不存在的节点: {}", node_id)
stats.facts_skipped += 1
continue
injected = False
if level == 1:
injected = _inject_into_l1(node, field, inject_value) # type: ignore[arg-type]
elif level == 2:
injected = _inject_into_l2(node, field, inject_value) # type: ignore[arg-type]
elif level == 3:
injected = _inject_into_l3(node, field, inject_value) # type: ignore[arg-type]
if injected:
stats.facts_injected += 1
else:
stats.facts_skipped += 1
return stats
# ---------------------------------------------------------------------------
# LLM Prompt
# ---------------------------------------------------------------------------
_SUPPLEMENT_SYSTEM_PROMPT = """\
你是一个视频内容分析专家你的任务是分析回答某个问题需要哪些关键事实
并判断这些事实是否已存在于视频树的摘要中
## 输出规则
1. 只输出**客观事实**包括以下四类:
- person_name: 人物姓名
- location: 地点名称
- score_number: 比分数字
- object_name: 关键物体名称
2. **不要**输出以下类型:
- 情感态度心情
- 因果推理"因为…所以…"
- 时间顺序推理"先…后…"
- 主观评价
3. 对于 person_name 类别输出 targets 数组包含两个写入点:
- L2 节点的 entities 字段
- L3 节点的 visible_entities 字段
其他类别只写入最相关的单个节点的 entities 字段
4. 每条 missing fact 必须包含 inject_value要注入的值 targets 数组
## 输出格式 (严格 JSON)
```json
{
"needed_facts": [
{"category": "person_name", "value": "..."}
],
"found_in_tree": [
{"category": "person_name", "value": "...", "found_at": "node_id"}
],
"missing_facts": [
{
"category": "person_name",
"inject_value": "...",
"targets": [
{"node_id": "...", "field": "entities"},
{"node_id": "...", "field": "visible_entities"}
]
}
]
}
```
只输出 JSON不要输出其他内容
"""
def _build_user_prompt(
question: dict[str, Any],
index: TreeIndex,
srt_text: str,
) -> str:
"""构建 supplement 分析的 user prompt。
包含: 问题 + 选项 + 正确答案 + L2 摘要 + SRT 字幕截断至 3000 字符
参数:
question: 包含 question/options/answer 的字典
index: TreeIndex 实例
srt_text: SRT 字幕文本
返回:
拼装后的 user prompt 字符串
"""
# 问题部分
q_text = question.get("question", "")
options = question.get("options", [])
answer = question.get("answer", "")
options_str = "\n".join(f" {chr(65 + i)}. {opt}" for i, opt in enumerate(options))
# 树 L2 摘要(从 TreeIndex 结构中提取)
l2_summaries: list[str] = []
for l1 in index.roots:
for l2 in l1.children:
description = l2.card.event_description
entities_str = ", ".join(l2.card.entities) if l2.card.entities else ""
time_str = ""
if l2.time_range:
time_str = f"{l2.time_range[0]:.1f}-{l2.time_range[1]:.1f}s: "
l2_summaries.append(
f"[{l2.id}] {time_str}{description}"
+ (f" | entities: {entities_str}" if entities_str else "")
)
l2_block = "\n".join(l2_summaries) if l2_summaries else "(无 L2 摘要)"
# SRT 截断
srt_truncated = srt_text[:3000] if srt_text else "(无字幕)"
return (
f"## 问题\n{q_text}\n\n"
f"## 选项\n{options_str}\n\n"
f"## 正确答案\n{answer}\n\n"
f"## 视频树 L2 摘要\n{l2_block}\n\n"
f"## 字幕 (前 3000 字符)\n{srt_truncated}"
)
# ---------------------------------------------------------------------------
# LLM 调用
# ---------------------------------------------------------------------------
async def analyze_question(
llm: LLMProvider,
question: dict[str, Any],
index: TreeIndex,
srt_text: str,
) -> list[dict[str, Any]]:
"""调用 LLM 分析单个问题,返回需要注入的事实列表。
参数:
llm: LLMProvider 实例异步接口
question: 问题字典 question/options/answer
index: TreeIndex 实例
srt_text: SRT 字幕文本
返回:
missing_facts 列表每项含 category / inject_value / targets
解析失败时返回空列表
"""
user_prompt = _build_user_prompt(question, index, srt_text)
messages = [
{"role": "system", "content": _SUPPLEMENT_SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
]
response = await llm.chat(messages)
raw = response.content
# 提取 JSON(兼容 markdown 代码块包裹)
text = raw.strip()
if text.startswith("```"):
lines = text.split("\n")
lines = [ln for ln in lines if not ln.strip().startswith("```")]
text = "\n".join(lines)
try:
parsed = json.loads(text)
except json.JSONDecodeError:
logger.warning("supplement LLM 返回非法 JSON,跳过。原始内容: {}", raw[:200])
return []
missing = parsed.get("missing_facts", [])
if not isinstance(missing, list):
logger.warning("missing_facts 不是列表,跳过")
return []
return missing
# ---------------------------------------------------------------------------
# 主入口
# ---------------------------------------------------------------------------
async def supplement_tree(
index: TreeIndex,
questions: list[dict[str, Any]],
llm: LLMProvider,
srt_text: str = "",
) -> SupplementStats:
"""对树索引执行 Q&A 反向补全:遍历问题,分析缺失事实,注入节点。
参数:
index: TreeIndex 实例节点 Card 会被就地替换
questions: 问题列表每项含 question/options/answer
llm: LLMProvider 实例异步接口
srt_text: SRT 字幕文本可选默认空字符串
返回:
补全统计信息
"""
all_injections: list[dict[str, Any]] = []
for i, question in enumerate(questions):
logger.debug(
"supplement: 分析问题 {}/{}",
i + 1,
len(questions),
)
missing = await analyze_question(llm, question, index, srt_text)
all_injections.extend(missing)
stats = apply_injections(index, all_injections)
stats.questions_analyzed = len(questions)
logger.info(
"supplement_tree 完成: questions={} injections={} injected={} skipped={}",
len(questions),
len(all_injections),
stats.facts_injected,
stats.facts_skipped,
)
return stats
+320
View File
@@ -0,0 +1,320 @@
"""字幕模块:SRT 解析、完整性检查、时间范围提取、Voronoi 分配。
提供四个核心函数
- parse_srt: 解析 SRT 文件为结构化条目列表
- check_subtitle_completeness: 检查字幕覆盖率与完整性
- extract_subtitle_for_range: 提取指定时间范围内的字幕文本
- assign_subtitles_voronoi: 使用 Voronoi 中点策略将字幕分配给 L3 节点
迁移来源:
- TRM4 core/tree/enhance/merge.py (parse_srt)
- TRM3 tools/generate_subtitles.py (Voronoi 逻辑)
"""
from __future__ import annotations
import dataclasses
import re
from dataclasses import dataclass
from typing import TYPE_CHECKING
from loguru import logger
if TYPE_CHECKING:
from app.tree.index import TreeIndex
# ---------------------------------------------------------------------------
# 正则表达式
# ---------------------------------------------------------------------------
_HTML_TAG_RE = re.compile(r"<[^>]+>")
_MUSIC_ONLY_RE = re.compile(r"^[\s♪♫]*$")
_TIMECODE_RE = re.compile(r"(\d+):(\d+):(\d+)[,.](\d+)\s*-->\s*(\d+):(\d+):(\d+)[,.](\d+)")
# ---------------------------------------------------------------------------
# 数据类型
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class SRTEntry:
"""单条 SRT 字幕条目。
属性:
start: 开始时间
end: 结束时间
text: 字幕文本已清洗 HTML 标签
"""
start: float
end: float
text: str
@dataclass(frozen=True)
class SubtitleReport:
"""字幕完整性检查报告。
属性:
total_entries: 字幕条目总数
coverage_ratio: SRT 覆盖时长 / 视频总时长
max_gap_sec: 最大连续无字幕间隔
usable: 覆盖率是否达到最低要求
"""
total_entries: int
coverage_ratio: float
max_gap_sec: float
usable: bool
# ---------------------------------------------------------------------------
# 内部辅助
# ---------------------------------------------------------------------------
def _ts_to_seconds(h: str, m: str, s: str, ms: str) -> float:
"""SRT 时间戳组件 (HH:MM:SS,mmm) 转秒数。
参数:
h: 小时
m: 分钟
s:
ms: 毫秒
返回:
浮点秒数
"""
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000
# ---------------------------------------------------------------------------
# 公共 API
# ---------------------------------------------------------------------------
def parse_srt(srt_path: str) -> list[SRTEntry]:
"""解析 SRT 字幕文件,返回结构化条目列表。
- 剥离 HTML 标签 <i><b>
- 跳过纯音乐符号行仅含空白和
- 多行字幕合并为单行空格连接
- 跳过格式异常的块容错处理
参数:
srt_path: SRT 文件的绝对路径
返回:
按时间顺序排列的 SRTEntry 列表空文件或无有效条目返回空列表
迁移来源:
TRM4 core/tree/enhance/merge.py parse_srt
TRM3 tools/generate_subtitles.py parse_srt
"""
with open(srt_path, encoding="utf-8") as f:
content = f.read()
if not content.strip():
return []
entries: list[SRTEntry] = []
blocks = re.split(r"\n\s*\n", content.strip())
for block in blocks:
lines = block.strip().split("\n")
if len(lines) < 2:
continue
# 在块内搜索时间码行(可能是第 1 行或第 2 行)
ts_match = None
ts_line_idx = -1
for i, line in enumerate(lines):
ts_match = _TIMECODE_RE.search(line)
if ts_match:
ts_line_idx = i
break
if not ts_match:
continue
groups = [int(x) for x in ts_match.groups()]
start = _ts_to_seconds(str(groups[0]), str(groups[1]), str(groups[2]), str(groups[3]))
end = _ts_to_seconds(str(groups[4]), str(groups[5]), str(groups[6]), str(groups[7]))
# 时间码行之后的所有行为字幕文本
text_lines = lines[ts_line_idx + 1 :]
raw_text = " ".join(text_lines)
clean_text = _HTML_TAG_RE.sub("", raw_text).strip()
# 跳过空文本和纯音乐符号行
if not clean_text or _MUSIC_ONLY_RE.match(clean_text):
continue
entries.append(SRTEntry(start=start, end=end, text=clean_text))
logger.debug("SRT 解析完成: {} 条有效条目, 文件={}", len(entries), srt_path)
return entries
def check_subtitle_completeness(
entries: list[SRTEntry],
duration: float,
min_coverage: float = 0.3,
) -> SubtitleReport:
"""检查字幕完整性:覆盖率、最大间隔、可用性判定。
参数:
entries: 已排序的 SRTEntry 列表
duration: 视频总时长必须 > 0
min_coverage: 最低可用覆盖率阈值0~1
返回:
SubtitleReport 包含覆盖率最大间隔和可用性判定
"""
assert duration > 0, f"视频时长必须 > 0,实际={duration}"
if not entries:
return SubtitleReport(
total_entries=0,
coverage_ratio=0.0,
max_gap_sec=duration,
usable=False,
)
# 按开始时间排序
sorted_entries = sorted(entries, key=lambda e: e.start)
# 计算覆盖时长(合并重叠区间)
merged_intervals: list[tuple[float, float]] = []
for entry in sorted_entries:
if merged_intervals and entry.start <= merged_intervals[-1][1]:
# 与上一区间重叠,扩展
merged_intervals[-1] = (
merged_intervals[-1][0],
max(merged_intervals[-1][1], entry.end),
)
else:
merged_intervals.append((entry.start, entry.end))
covered = sum(end - start for start, end in merged_intervals)
coverage_ratio = min(covered / duration, 1.0)
# 计算最大间隔(包括视频开头到第一条字幕、最后一条到视频结尾)
max_gap = merged_intervals[0][0] # 视频开头到第一条字幕
for i in range(1, len(merged_intervals)):
gap = merged_intervals[i][0] - merged_intervals[i - 1][1]
max_gap = max(max_gap, gap)
# 最后一条字幕到视频结尾
max_gap = max(max_gap, duration - merged_intervals[-1][1])
return SubtitleReport(
total_entries=len(entries),
coverage_ratio=coverage_ratio,
max_gap_sec=max_gap,
usable=coverage_ratio >= min_coverage,
)
def extract_subtitle_for_range(
entries: list[SRTEntry],
time_range: tuple[float, float],
) -> str:
"""提取与指定时间范围重叠的字幕文本。
重叠判定entry.start < range_end entry.end > range_start
参数:
entries: SRTEntry 列表
time_range: (start, end) 时间范围
返回:
匹配的字幕文本多条用换行符连接无匹配返回空字符串
"""
range_start, range_end = time_range
matched = [
entry.text for entry in entries if entry.start < range_end and entry.end > range_start
]
return "\n".join(matched)
def assign_subtitles_voronoi(
index: TreeIndex,
entries: list[SRTEntry],
) -> None:
"""使用 Voronoi 中点策略将字幕分配给 L3 节点。
对每个 L2 节点内的 L3 子节点 timestamp 排序后计算 Voronoi 有效范围
- 相邻 L3 节点之间取中点作为边界
- 首个 L3 的左边界扩展到 L2 time_range 起点
- 末个 L3 的右边界扩展到 L2 time_range 终点
然后用 extract_subtitle_for_range 提取每个 L3 有效范围内的字幕文本
参数:
index: 树索引包含 L1L2L3 嵌套结构
entries: 已解析的 SRTEntry 列表
副作用:
通过 dataclasses.replace 替换 L3Node.card L2Node.card
将字幕写入 card.subtitle 字段
迁移来源:
TRM3 tools/generate_subtitles.py compute_effective_ranges + assign_subtitles
"""
for l1 in index.roots:
for l2 in l1.children:
if not l2.children:
continue
# 按 timestamp 排序 L3 子节点(保留原列表引用以便赋值)
siblings = sorted(
l2.children,
key=lambda n: n.timestamp if n.timestamp is not None else 0.0,
)
# L2 的时间范围作为边界
l2_start = l2.time_range[0] if l2.time_range else 0.0
l2_end = l2.time_range[1] if l2.time_range else 0.0
for idx, l3 in enumerate(siblings):
ts = l3.timestamp if l3.timestamp is not None else 0.0
# 计算 Voronoi 有效范围
if idx == 0:
left = l2_start
else:
prev_ts = (
siblings[idx - 1].timestamp
if siblings[idx - 1].timestamp is not None
else 0.0
)
left = (prev_ts + ts) / 2.0
if idx == len(siblings) - 1:
right = l2_end
else:
next_ts = (
siblings[idx + 1].timestamp
if siblings[idx + 1].timestamp is not None
else 0.0
)
right = (ts + next_ts) / 2.0
subtitle_text = extract_subtitle_for_range(entries, (left, right))
l3.card = dataclasses.replace(
l3.card,
subtitle=subtitle_text or "",
)
# L2 字幕聚合:拼接所有 L3 子节点的字幕
l3_subtitles = [l3.card.subtitle for l3 in l2.children if l3.card.subtitle]
l2.card = dataclasses.replace(
l2.card,
subtitle="\n".join(l3_subtitles),
)
logger.debug(
"Voronoi 字幕分配完成: {} 个 L1 节点, {} 条字幕条目",
len(index.roots),
len(entries),
)
+292
View File
@@ -0,0 +1,292 @@
"""质量校验模块:交叉验证树节点 Card 字段与子节点证据。
验证策略:
- L2 entities: 仅保留在子 L3 文本语料中模糊匹配到的实体
- L2 visible_text: 仅保留在子 L3 visible_text 中出现的条目
- L1 visible_text: 仅保留在后代 L2/L3 visible_text 中出现的条目
- L1 key_entities: 仅保留在后代 L2/L3 文本语料中模糊匹配到的实体
Card frozen dataclass无法原地修改移除幻觉字段时
创建新 Card 实例并赋值给 node.cardNode frozen
"""
from __future__ import annotations
import string
from dataclasses import dataclass
from loguru import logger
from app.tree.index import (
L1Card,
L1Node,
L2Card,
L2Node,
TreeIndex,
)
# ---------------------------------------------------------------------------
# 校验统计
# ---------------------------------------------------------------------------
@dataclass
class VerifyStats:
"""校验统计信息。"""
l2_entities_kept: int = 0
l2_entities_removed: int = 0
l2_visible_text_kept: int = 0
l2_visible_text_removed: int = 0
l1_visible_text_kept: int = 0
l1_visible_text_removed: int = 0
l1_key_entities_kept: int = 0
l1_key_entities_removed: int = 0
# ---------------------------------------------------------------------------
# 文本归一化 & 模糊匹配
# ---------------------------------------------------------------------------
def _normalize(text: str) -> str:
"""归一化文本:小写 + 去除标点。
参数:
text: 原始文本
返回:
归一化后的纯小写无标点字符串
"""
return text.lower().translate(str.maketrans("", "", string.punctuation))
def fuzzy_match(entity: str | None, corpus: str | None) -> bool:
"""模糊子串匹配:归一化后判断 entity 是否为 corpus 的子串。
参数:
entity: 待匹配的实体文本None 视为不匹配
corpus: 证据语料文本None 视为空
返回:
True 表示匹配成功
"""
if not entity or not corpus:
return False
return _normalize(str(entity)) in _normalize(str(corpus))
# ---------------------------------------------------------------------------
# 语料收集
# ---------------------------------------------------------------------------
def _collect_l3_text(l2_node: L2Node) -> str:
"""收集 L2 节点所有子 L3 的文本语料。
从每个 L3 子节点的 card 和顶层字段中提取:
frame_summaryvisible_textsubtitle
参数:
l2_node: L2 节点
返回:
拼接后的文本语料用换行分隔
"""
parts: list[str] = []
for l3 in l2_node.children:
parts.append(l3.card.frame_summary)
parts.extend(l3.card.visible_text)
if l3.card.subtitle:
parts.append(l3.card.subtitle)
return "\n".join(parts)
def _collect_descendant_visible_text(l1_node: L1Node) -> str:
"""收集 L1 节点所有后代(L2/L3)的 visible_text。
参数:
l1_node: L1 节点
返回:
所有后代 visible_text 拼接后的文本用换行分隔
"""
parts: list[str] = []
for l2 in l1_node.children:
parts.extend(l2.card.visible_text)
for l3 in l2.children:
parts.extend(l3.card.visible_text)
return "\n".join(parts)
def _collect_descendant_text_corpus(l1_node: L1Node) -> str:
"""收集 L1 节点所有后代(L2/L3)的完整文本语料。
用于 L1 key_entities 的交叉验证范围包括
L2/L3 的所有文本字段frame_summaryvisible_textsubtitle
参数:
l1_node: L1 节点
返回:
所有后代文本语料拼接后的文本用换行分隔
"""
parts: list[str] = []
for l2 in l1_node.children:
parts.append(l2.card.event_description)
parts.extend(l2.card.entities)
parts.extend(l2.card.visible_text)
for l3 in l2.children:
parts.append(l3.card.frame_summary)
parts.extend(l3.card.visible_text)
if l3.card.subtitle:
parts.append(l3.card.subtitle)
return "\n".join(parts)
# ---------------------------------------------------------------------------
# 主校验函数
# ---------------------------------------------------------------------------
def verify_tree(index: TreeIndex) -> VerifyStats:
"""交叉验证视频树的 Card 字段与子节点证据,原地替换不合格的 Card。
Cards frozen dataclass移除幻觉字段时创建新 Card 实例
并赋值给 node.card
参数:
index: 树索引会被原地修改
返回:
VerifyStats 校验统计
"""
stats = VerifyStats()
for l1 in index.roots:
# Phase 1: L2 字段验证
for l2 in l1.children:
_verify_l2(l2, stats)
# Phase 2: L1 字段验证
_verify_l1(l1, stats)
logger.info(
"verify_tree: source={} "
"l2_ent_kept={} l2_ent_rm={} "
"l2_vt_kept={} l2_vt_rm={} "
"l1_vt_kept={} l1_vt_rm={} "
"l1_ke_kept={} l1_ke_rm={}",
index.metadata.source_path,
stats.l2_entities_kept,
stats.l2_entities_removed,
stats.l2_visible_text_kept,
stats.l2_visible_text_removed,
stats.l1_visible_text_kept,
stats.l1_visible_text_removed,
stats.l1_key_entities_kept,
stats.l1_key_entities_removed,
)
return stats
def _verify_l2(l2: L2Node, stats: VerifyStats) -> None:
"""校验单个 L2 节点的 entities 和 visible_text。
参数:
l2: L2 节点card 可能被替换
stats: 统计对象原地累加
"""
corpus = _collect_l3_text(l2)
old_card = l2.card
# entities: 模糊匹配过滤
kept_entities = [e for e in old_card.entities if fuzzy_match(e, corpus)]
stats.l2_entities_kept += len(kept_entities)
stats.l2_entities_removed += len(old_card.entities) - len(kept_entities)
# visible_text: 子 L3 visible_text 中必须存在
l3_visible = _collect_l3_visible_text_set(l2)
kept_vt = [vt for vt in old_card.visible_text if _text_in_set(vt, l3_visible)]
stats.l2_visible_text_kept += len(kept_vt)
stats.l2_visible_text_removed += len(old_card.visible_text) - len(kept_vt)
# 创建新 Card 替换(frozen dataclass
l2.card = L2Card(
event_description=old_card.event_description,
entities=kept_entities,
actions=old_card.actions,
action_subjects=old_card.action_subjects,
visible_text=kept_vt,
spatial_relations=old_card.spatial_relations,
state_changes=old_card.state_changes,
subtitle=old_card.subtitle,
)
def _verify_l1(l1: L1Node, stats: VerifyStats) -> None:
"""校验单个 L1 节点的 visible_text 和 key_entities。
参数:
l1: L1 节点card 可能被替换
stats: 统计对象原地累加
"""
old_card = l1.card
# visible_text: 必须出现在后代 L2/L3 visible_text 中
descendant_vt = _collect_descendant_visible_text(l1)
kept_vt = [vt for vt in old_card.visible_text if fuzzy_match(vt, descendant_vt)]
stats.l1_visible_text_kept += len(kept_vt)
stats.l1_visible_text_removed += len(old_card.visible_text) - len(kept_vt)
# key_entities: 交叉验证后代文本语料
descendant_corpus = _collect_descendant_text_corpus(l1)
kept_ke = [ke for ke in old_card.key_entities if fuzzy_match(ke, descendant_corpus)]
stats.l1_key_entities_kept += len(kept_ke)
stats.l1_key_entities_removed += len(old_card.key_entities) - len(kept_ke)
# 创建新 Card 替换(frozen dataclass
l1.card = L1Card(
scene_summary=old_card.scene_summary,
main_setting=old_card.main_setting,
key_entities=kept_ke,
main_actions=old_card.main_actions,
topic_keywords=old_card.topic_keywords,
visible_text=kept_vt,
temporal_flow=old_card.temporal_flow,
)
# ---------------------------------------------------------------------------
# 辅助函数
# ---------------------------------------------------------------------------
def _collect_l3_visible_text_set(l2: L2Node) -> set[str]:
"""收集 L2 下所有 L3 子节点的 visible_text 归一化集合。
参数:
l2: L2 节点
返回:
归一化后的 visible_text 集合
"""
result: set[str] = set()
for l3 in l2.children:
for vt in l3.card.visible_text:
result.add(_normalize(vt))
return result
def _text_in_set(text: str, normalized_set: set[str]) -> bool:
"""检查文本归一化后是否存在于集合中。
参数:
text: 待检查文本
normalized_set: 归一化后的文本集合
返回:
True 表示匹配成功
"""
return _normalize(text) in normalized_set
File diff suppressed because it is too large Load Diff
+30 -35
View File
@@ -20,46 +20,15 @@ embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cpu"
# ── 可训练检索器 ──
retriever:
embed_dim: 768
num_heads: 4
L_layers: 2
L_cycles: 4
max_rounds: 5
ffn_expansion: 2.0
checkpoint: null
k_l1: 1
k_l2: 1
k_l3: 1
max_paths: 5
# ── 检索器训练 ──
train:
lr: 1.0e-4
weight_decay: 1.0e-5
batch_size: 1
max_epochs_phase1: 30
max_epochs_phase2: 20
nav_loss_weight: 1.0
act_loss_weight: 0.1
margin_loss_weight: 0.5
act_lambda_step: 0.1
act_gamma: 0.9
eval_interval: 5
save_dir: "checkpoints"
dataset: "videomme"
dataset_path: "data/videomme/splits/train.jsonl"
device: "cuda"
# ── Harness 自进化循环 ──
harness:
workspace_dir: "workspaces/default"
store_dir: store
mode: infer
concurrency: 12
max_steps: 15
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME"
@@ -73,7 +42,6 @@ harness:
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_block: 8
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
@@ -98,5 +66,32 @@ harness:
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
trainable_min_units: 8
early_stop_patience: 8
use_slow_momentum: true
# 池构建策略
pool_split_mode: global # global | per_category
train_ratio: 0.667 # per_category 模式下 train/(train+val) 比例
test_questions: "benchmarks/Video-MME" # test 池的题目来源
# ── 出题管线 v2 ──
question_gen_v2:
family_ratios:
retrieval: 0.30
reasoning: 0.25
enumeration: 0.20
visual: 0.15
spatial: 0.10
gate:
blind_answer_model: "gpt-4.1-mini"
leak_test_model: "gpt-4.1-mini"
key_verify_model: "gpt-4.1-mini"
multi_true_model: "gpt-4.1-mini"
dedup_threshold: 0.85
retry_limit: 3
heavy_sample_rate: 0.15
heavy_agent_model: "gpt-4.1-mini"
output_dir: "store/questions/generated-v2"
per_type: 20 # 12 类 x 20 = 240 题(设计 §3 硬约束)
concurrency: 4
seed: 42
+80
View File
@@ -0,0 +1,80 @@
# 补生 Video-MME 6 类缺失题型(180 题)
# 原 360 题中有 6 类错误类型已归档,此配置只生成缺失的 6 类。
# ── 建树模块 ──(与 question_gen_360.yaml 一致)
tree:
max_paragraphs_per_l2: 5
l1_segment_duration: 600.0
l2_clip_duration: 60.0
l3_fps: 0.5
l2_representative_frames: 6
cache_dir: "cache/trees"
concurrency: 16
subtitle_inject: true
srt_window_sec: 5.0
# ── Embedding ──
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
# ── Harness ──(占位,出题不使用)
harness:
workspace_dir: "workspaces/default"
store_dir: store
mode: infer
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME"
skills_version: v1
prompts_version: v1
epochs: 1
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
diag_size: 200
diag_correct_ratio: 0.5
val_size: 30
val_correct_ratio: 0.5
test_size: 60
batch_size: 15
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
early_stop_patience: 8
use_slow_momentum: true
# ── 出题管线 v2 ──
question_gen_v2:
family_ratios:
retrieval: 0.30
reasoning: 0.25
enumeration: 0.20
visual: 0.15
spatial: 0.10
dedup_threshold: 0.85
retry_limit: 10
heavy_sample_rate: 0.15
output_dir: "store/questions/generated-v2-180补"
per_type: 30 # 6 类 x 30 = 180 题
concurrency: 24
seed: 43 # 不同于原始 seed=42,避免生成相同题目
+86
View File
@@ -0,0 +1,86 @@
# config/default.yaml
# 科研实验配置默认值来源(会在实验中反复扫动/对比的参数)。
# 工程配置(少变、敏感)由 .env / pydantic-settings 管理,不在此文件。
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
# ── 建树模块 ──
tree:
max_paragraphs_per_l2: 5
l1_segment_duration: 600.0 # L1 段时长(秒)
l2_clip_duration: 60.0 # L2 clip 时长(秒)
l3_fps: 0.5 # L3 帧提取频率(帧/秒)
l2_representative_frames: 6 # L2 VLM 描述用的代表帧数
cache_dir: "cache/trees"
concurrency: 16 # asyncio Semaphore 上限
subtitle_inject: true # 建树时是否注入 SRT 字幕
srt_window_sec: 5.0 # 字幕匹配时间窗口(前后各 N 秒)
# ── Embedding ──
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
# ── Harness 自进化循环 ──
harness:
workspace_dir: "workspaces/default"
store_dir: store
mode: infer
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME"
skills_version: v1
prompts_version: v1
epochs: 1
# CE-Gate 参数
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 数据池
diag_size: 200
diag_correct_ratio: 0.5
val_size: 30
val_correct_ratio: 0.5
test_size: 60
# mini-batch
batch_size: 15
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
early_stop_patience: 8
use_slow_momentum: true
# ── 出题管线 v2 ──
question_gen_v2:
family_ratios:
retrieval: 0.30
reasoning: 0.25
enumeration: 0.20
visual: 0.15
spatial: 0.10
dedup_threshold: 0.85
retry_limit: 10
heavy_sample_rate: 0.15
output_dir: "store/questions/generated-v2-360"
per_type: 30 # 12 类 x 30 = 360 题
concurrency: 24
seed: 42
+21
View File
@@ -0,0 +1,21 @@
# AR 专项出题配置:使用 ActionRecognitionStrategy + 6 SubPattern 靶向生成
# 目标:生成 30 道 Action Recognition 题
question_gen_v2:
per_type: 30 # 只跑 AR 一类,30 题
retry_limit: 15 # AR 约束更严,给更多重试机会
heavy_sample_rate: 0.0 # 不需要重量抽检
dedup_threshold: 0.85
concurrency: 8 # AR 需要帧,适度并发
seed: 2024
output_dir: "store/questions/generated-ar30"
candidate_pool_size: 24 # grounded selector 首轮候选干扰项数 N
selector_delta_low: 0.05 # 干扰项视觉分与正解的最小差(区间上界)
selector_delta_high: 0.35 # 干扰项视觉分与正解的最大差(区间下界)
# Phase B 后置对抗过滤层配置(filter 层,非 strategy 属性)
adversarial_filter:
filter_task_types: [Action Recognition] # 仅这些题型走 agent 门(路径隔离开关)
adversarial_max_rounds: 5 # 补生成迭代上限
adversarial_agent_max_steps: 40 # agent 试答步数上限
difficulty_warn_threshold: 0.85 # 批次 agent 正确率告警阈值
+63
View File
@@ -0,0 +1,63 @@
# config/train_action_recognition.yaml
# Action Recognition 单题型首次训练实验
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
harness:
workspace_dir: "workspaces/train-action-recognition"
store_dir: store
mode: train
run_id: train_ar_v1
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "generated-v2-360"
skills_version: v1
prompts_version: v1
epochs: 3
# CE-Gate 参数(沿用 default.yaml
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 池配置 — per_category 单题型
pool_split_mode: per_category
task_types:
- "Action Recognition"
train_ratio: 0.667
test_questions: "benchmarks/Video-MME"
run_holdout_eval: false
# mini-batch
batch_size: 10
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
trainable_min_units: 8
early_stop_patience: 4
test_size: 63
diag_size: 20
diag_correct_ratio: 0.5
val_size: 10
val_correct_ratio: 0.5
use_slow_momentum: true
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
+64
View File
@@ -0,0 +1,64 @@
# config/train_ar30.yaml
# Action Recognition 训练 — 基于 SubPattern 靶向生成的 30 题
# 对比基线: v2-360 的 AR 题(100% 单帧,训练无效)
# 本次: AR30 题(6 种失败子模式靶向,跨段时序)
harness:
workspace_dir: "workspaces/train-ar30"
store_dir: store
mode: train
run_id: train_ar30_v1
concurrency: 24
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "generated-ar30"
skills_version: v1
prompts_version: v1
epochs: 3
# CE-Gate 参数(沿用 default.yaml
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 池配置 — per_category 单题型
pool_split_mode: per_category
task_types:
- "Action Recognition"
train_ratio: 0.667
test_questions: "benchmarks/Video-MME"
run_holdout_eval: false
# mini-batch
batch_size: 10
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
eval_min_per_class: 2
trainable_min_units: 8
early_stop_patience: 4
test_size: 63
diag_size: 20
diag_correct_ratio: 0.5
val_size: 10
val_correct_ratio: 0.5
use_slow_momentum: true
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
+69
View File
@@ -0,0 +1,69 @@
# config/train_videomme.yaml
# Video-MME 900 题自进化训练 —— 消费 video-split 冻结切分(global 三池)
#
# 数据来源: workspaces/video-split/pools.jsontier 感知 diag/val + val 功效修复)
# 经 adhoc-baseline seed 携带进 workspaceWP2 接线)。
# 训练前置: .env REDIS_CACHE_TTL 须为正整数(WP4 fail-loud);见
# research-wiki/reviews/2026-07-16-preflight-final-review.md runbook。
harness:
workspace_dir: "workspaces/train-videomme"
store_dir: store
mode: train
run_id: train_videomme_v2
concurrency: 32
max_steps: 40
skill_mode: auto
n_samples: 0
questions: "benchmarks/Video-MME" # gate 指纹依赖加载全 900 题
skills_version: v1
prompts_version: v1
epochs: 3
# CE-Gate 参数(沿用 default.yaml
gate_e_confirm: 20.0
gate_e_provisional: 3.0
gate_w_net_min: 2
gate_delta_min: 0.02
gate_lambda_dir: -0.642
gate_e_rollback: 10.0
gate_n_max: 40
gate_p_low: 0.05
gate_p_high: 0.95
gate_probe_quota: 0.2
gate_gamma_decay: 0.9
gate_cooldown_steps: 2
gate_guard_err: 0.10
# 进化参数
edit_budget_start: 5
edit_budget_end: 2
skill_update_mode: patch
appendix_consolidate_threshold: 6
# 池配置 —— global 冻结切分(diag/val/test 尺寸由 pools.json 冻结,以下采样旋钮加载时忽略)
# 实际冻结切分(0.4+tierfingerprint d456ef4f: diag=180 val=120 test=600
pool_split_mode: global
diag_size: 180
diag_correct_ratio: 0.5
val_size: 120
val_correct_ratio: 0.5
test_size: 600
test_questions: "benchmarks/Video-MME"
# 可训练性预检(WP3):val 单元 < eval_min_per_class 或 非test单元 < trainable_min_units 的题型剔除
eval_min_per_class: 2
trainable_min_units: 8
# mini-batch —— 对齐 TRM4 正式实验 batch=40sh --batch-size 40 覆盖 yaml 15 的最终生效值):
# 8 可训题型 × 每型约 5 题/step,保住题型级诊断信号;同时 steps/epoch 180/40≈5
# 进化/gate 验证轮数比 batch=10 少 4 倍。
batch_size: 40
min_class_per_batch: 2
batch_correct_ratio: 0.5
momentum_samples: 20
early_stop_patience: 2 # epoch 粒度(WP3):连续 2 epoch 无 best 刷新即停
use_slow_momentum: true
run_holdout_eval: true # 逐 epoch test 四向评估(WP3 去重版:baseline 推导 + 版本备忘录)
# 全 12 题型(不指定 task_types 子集,避免 I-4 语义偏差;微型类由预检自动剔除)
embed:
backend: "local"
model_name: "BAAI/bge-base-zh-v1.5"
embed_dim: 768
device: "cuda"
+33
View File
@@ -0,0 +1,33 @@
# config/video_split.yaml
# 结果驱动视频级切分 —— 科研旋钮快照(会在实验中被反复扫动 / 对比的参数)。
# 工程配置(harness.db 路径、LLM 凭证、超时)由 .env / pydantic-settings 管理,不在此文件。
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
#
# ⚠️ 下列阈值为「占位值」,必须在离线诊断跑完后用真实 T2 分布标定。
# 标定程序见 scripts/build_video_split.sh 顶部注释(诊断产物 → 逐旋钮定值)。
video_split:
baseline_run_id: infer_adhoc # 基线 run 标识(错题诊断与切分依据),对应 workspaces/default/harness.db
n_trainval: 100 # trainval 目标视频数(多样性阶段填充上限;标定取 ~100)
epsilon: 0.1 # test 相对全局最大允许分布偏差(题型占比 / 难度画像两维,逐桶)
report_floor: 27 # per-type 报告门限:题数 ≥ 此值的 task_type 才入 ε 代表性约束
val_wrong_min: 20 # validation 池最少错题数(McNemar 检验功效阈 ≈ 20,低于则信号不足)
val_ratio: 0.4 # validation 占 trainval 视频组总数的比例(0.3→0.4 提升整包终审功效,WP2)
seed: 7 # 贪心选择器预洗牌 + 视频组题级切分种子(打破等增益 / 等槽平局)
floor_k: # 各 task_type 的 T2 defect 下限(硬约束)—— 均衡覆盖全 11 类,标定于 1cb1c203 真实 T2 分布
Object Reasoning: 5 # T2=25
Information Synopsis: 5 # T2=13
Action Reasoning: 5 # T2=11
Counting Problem: 5 # T2=10
Temporal Reasoning: 2 # T2=5
Object Recognition: 2 # T2=5
Action Recognition: 2 # T2=5
Attribute Perception: 1 # T2=3
Temporal Perception: 1 # T2=2
OCR Problems: 1 # T2=2
Spatial Perception: 1 # T2=1
diag: # 诊断口径指纹三分量(隔离不同诊断配置的信号,参与主键)
prompt_version: diagnose_v1 # 诊断 prompt 版本标识(换 prompt 即换指纹,旧记录不被覆盖)
model: deepseek-v4-pro # 执行诊断的模型名(必须与 .env SEARCH_LLM_MODEL 一致,CLI 会 fail loud 校验)
# code_version 由 build_video_split.sh 注入 git 短 SHA,不写死在此(随代码变动)
+108 -9
View File
@@ -17,7 +17,9 @@ TRM4 → TRM5 有意变更(非简化):
from __future__ import annotations
import asyncio
import json
import re
from typing import TYPE_CHECKING, Any
import pluggy
@@ -31,6 +33,9 @@ if TYPE_CHECKING:
from core.protocols import LLMProvider
from core.types import LLMResponse
# deepseek 等模型稳定输出变体:```json 围栏包裹 JSON 体
_CODE_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*\n?|\n?\s*```\s*$")
async def _call_hook(hook: Any, **kwargs: Any) -> list[Any]:
"""调用 pluggy hook 并 await 异步返回值。
@@ -67,6 +72,13 @@ class AgentLoop:
llm: LLMProvider 实例Protocol 类型化提供 async chat 方法
max_steps: 最大有效步数每次成功工具调用计一步
max_retries: JSON 解析连续失败的最大容忍次数
step_retries: LLM 瞬时异常的步级重试次数不含首次调用
step_retry_delays: 步级重试的退避秒数序列超出部分取末值
不得为空空序列构造时抛 ValueErrorfail-fast
retryable_exceptions: 可重试异常元组默认 (TimeoutError, OSError)
ssl.SSLError/ConnectionError 均为 OSError 子类覆盖穿透
GovernedLLMClient 内部重试栈的瞬时异常openai API 类异常由
治理层负责core 不依赖 SDK
"""
def __init__(
@@ -74,10 +86,19 @@ class AgentLoop:
llm: LLMProvider,
max_steps: int,
max_retries: int = 3,
*,
step_retries: int = 2,
step_retry_delays: tuple[float, ...] = (20.0, 40.0),
retryable_exceptions: tuple[type[BaseException], ...] = (TimeoutError, OSError),
) -> None:
if not step_retry_delays:
raise ValueError("step_retry_delays 不得为空")
self._llm = llm
self._max_steps = max_steps
self._max_retries = max_retries
self._step_retries = step_retries
self._step_retry_delays = step_retry_delays
self._retryable_exceptions = retryable_exceptions
async def run(
self,
@@ -87,6 +108,7 @@ class AgentLoop:
plugins: list[object] | None = None,
*,
session_id: str | None = None,
cache_salt: str | None = None,
) -> LoopResult:
"""执行 Thinking+JSON 推理循环。
@@ -96,6 +118,7 @@ class AgentLoop:
tool_dispatcher: 工具调度器ToolDispatcher Protocol 实例
plugins: pluggy 插件列表
session_id: 会话 ID透传给 LLMProvider
cache_salt: 缓存盐透传给 LLMProvider如训练用 run_id epoch 重采样
返回:
LoopResult 实例包含推理步骤token 用量终止原因
@@ -114,11 +137,13 @@ class AgentLoop:
while step_count < self._max_steps:
await _call_hook(pm.hook.before_step, iteration=iteration, messages=messages)
# Phase 1: LLM 调用
# Phase 1: LLM 调用(步级重试:防穿透 GovernedLLMClient 的瞬时异常)
try:
response = await self._call_llm(messages, token_usage, session_id=session_id)
response = await self._call_llm_with_step_retry(
messages, token_usage, session_id=session_id, cache_salt=cache_salt
)
except Exception as e:
logger.error("LLM API 调用失败: {}", e)
logger.error("LLM API 调用失败{}: {}", type(e).__name__, e)
result = LoopResult(
steps=steps,
steps_used=step_count,
@@ -240,14 +265,21 @@ class AgentLoop:
pm.register(plugin)
return pm
async def _call_llm(
async def _call_llm_with_step_retry(
self,
messages: list[dict[str, Any]],
token_usage: dict[str, int],
*,
session_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse:
"""调用 LLMProvider 并累加 token 使用量
"""带步级重试的 LLM 调用,兜底穿透治理层重试栈的瞬时异常
可重试异常self._retryable_exceptions self._step_retry_delays
退避后重发重试预算self._step_retries耗尽后抛出最后一次异常
非可重试异常直接穿透fail-fastasyncio.CancelledError 继承
BaseException天然不被捕获取消信号照常传播失败尝试的 error
遥测由 GovernedLLMClient 内部负责此处仅 loguru 记录
参数:
messages: 消息历史
@@ -256,8 +288,52 @@ class AgentLoop:
返回:
LLMResponse 实例
异常:
重试耗尽或非可重试时原样抛出底层异常
"""
response = await self._llm.chat(messages, session_id=session_id)
step_attempt = 0
while True:
try:
return await self._call_llm(
messages, token_usage, session_id=session_id, cache_salt=cache_salt
)
except self._retryable_exceptions as e:
step_attempt += 1
if step_attempt > self._step_retries:
raise
delay = self._step_retry_delays[
min(step_attempt - 1, len(self._step_retry_delays) - 1)
]
logger.warning(
"LLM 瞬时异常,步级重试 {}/{}{}s 后重发): {}",
step_attempt,
self._step_retries,
delay,
e,
)
await asyncio.sleep(delay)
async def _call_llm(
self,
messages: list[dict[str, Any]],
token_usage: dict[str, int],
*,
session_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse:
"""调用 LLMProvider 并累加 token 使用量。
参数:
messages: 消息历史
token_usage: 可变字典就地累加
session_id: 会话 ID透传给 LLMProvider
cache_salt: 缓存盐透传给 LLMProvider epoch 重采样
返回:
LLMResponse 实例
"""
response = await self._llm.chat(messages, session_id=session_id, cache_salt=cache_salt)
token_usage["prompt_tokens"] += response.prompt_tokens
token_usage["completion_tokens"] += response.completion_tokens
return response
@@ -267,7 +343,8 @@ class AgentLoop:
) -> tuple[str, dict, dict, str, dict, str] | None:
"""从 LLMResponse 中提取结构化决策数据。
解析流程: content repair_json json.loads 校验 action/tool/args
解析流程: content 剥除 ```json 围栏 repair_json json.loads
收拢 action 平铺参数 校验 action/tool/args
参数:
response: LLMResponse 实例
@@ -282,7 +359,7 @@ class AgentLoop:
if not content.strip():
return None
repaired = repair_json(content)
repaired = repair_json(_CODE_FENCE_RE.sub("", content).strip())
try:
data = json.loads(repaired)
except (json.JSONDecodeError, ValueError):
@@ -291,7 +368,7 @@ class AgentLoop:
if not isinstance(data, dict) or "action" not in data:
return None
action = data["action"]
action = self._normalize_action(data["action"])
if not isinstance(action, dict) or "tool" not in action or "args" not in action:
return None
@@ -299,6 +376,28 @@ class AgentLoop:
plan = data.get("plan", {})
return thought, reflect, plan, content, action, response.call_id
@staticmethod
def _normalize_action(action: Any) -> Any:
"""收拢 deepseek 变体的 action 平铺参数。
deepseek 等模型稳定输出变体: 工具参数平铺在 action args
嵌套确定性收拢为标准 {"tool": ..., "args": {...}} 结构
仅当除 tool 外至少存在一个平铺参数键时才收拢无参结构
{"tool": "x"}原样返回交由调用方校验拒绝避免把缺参
错误静默升级为空 args 合法结构标准嵌套与非法结构同样原样返回
参数:
action: LLM 输出解析出的 action 字段任意类型
返回:
归一化后的 action仅带平铺参数的变体被改写其余原样返回
"""
if isinstance(action, dict) and "tool" in action and "args" not in action:
flat_args = {k: v for k, v in action.items() if k != "tool"}
if flat_args:
return {"tool": action["tool"], "args": flat_args}
return action
async def _execute_tool(
self,
dispatcher: ToolDispatcher,
+6 -10
View File
@@ -1,10 +1,12 @@
"""Agent 专属 Protocol 端口。"""
from __future__ import annotations
from typing import Any, Protocol, runtime_checkable
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
import pluggy
if TYPE_CHECKING:
from core.agent.types import LoopResult, Step
hookspec = pluggy.HookspecMarker("agent_loop")
@@ -27,19 +29,13 @@ class AgentLoopSpec:
"""
@hookspec
async def before_step(
self, iteration: int, messages: list[dict[str, Any]]
) -> None: ...
async def before_step(self, iteration: int, messages: list[dict[str, Any]]) -> None: ...
@hookspec
async def after_tool(
self, iteration: int, step: Step
) -> str | None: ...
async def after_tool(self, iteration: int, step: Step) -> str | None: ...
@hookspec
async def after_step(
self, iteration: int, messages: list[dict[str, Any]]
) -> None: ...
async def after_step(self, iteration: int, messages: list[dict[str, Any]]) -> None: ...
@hookspec
async def on_finish(self, result: LoopResult) -> None: ...
+1
View File
@@ -1,4 +1,5 @@
"""AgentLoop 数据类型。"""
from __future__ import annotations
from dataclasses import dataclass, field
+84
View File
@@ -0,0 +1,84 @@
"""core/evolution/ — 自进化循环决策内核。
诊断进化门控补丁的纯决策逻辑
只依赖 Protocol 接口和标准库可搬到无 adapters 的环境用假实现原样运行
"""
from core.evolution.diagnose import INFRA_STOP_REASONS, run_diagnosis
from core.evolution.evolve import (
edit_budget_at,
evolve_single_skill,
evolve_single_tool,
evolve_system_prompt,
resolve_skill_file,
)
from core.evolution.gate import compute_e_value, gate_decision, probation_verdict
from core.evolution.patch import (
append_to_appendix,
apply_patch_with_report,
extract_appendix_notes,
momentum_inner,
replace_appendix_notes,
replace_momentum,
)
from core.evolution.types import (
CaseSample,
DiagnosePrompts,
DiagnosisResult,
ErrorAttribution,
EvolutionRecord,
EvolutionResult,
EvolvePrompts,
GateParams,
GateVerdict,
PairResult,
QuadrantClassification,
QuestionMetrics,
RejectedEdit,
SkillCasePack,
SkillStepAdherence,
SpanMetrics,
SystemCasePack,
ToolCasePack,
)
from core.evolution.validate import classify_quadrants, compute_accuracy, pair_block
__all__ = [
"INFRA_STOP_REASONS",
"CaseSample",
"DiagnosePrompts",
"DiagnosisResult",
"ErrorAttribution",
"EvolutionRecord",
"EvolutionResult",
"EvolvePrompts",
"GateParams",
"GateVerdict",
"PairResult",
"QuadrantClassification",
"QuestionMetrics",
"RejectedEdit",
"SkillCasePack",
"SkillStepAdherence",
"SpanMetrics",
"SystemCasePack",
"ToolCasePack",
"append_to_appendix",
"apply_patch_with_report",
"classify_quadrants",
"compute_accuracy",
"compute_e_value",
"edit_budget_at",
"evolve_single_skill",
"evolve_single_tool",
"evolve_system_prompt",
"extract_appendix_notes",
"gate_decision",
"momentum_inner",
"pair_block",
"probation_verdict",
"replace_appendix_notes",
"replace_momentum",
"resolve_skill_file",
"run_diagnosis",
]
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+129
View File
@@ -0,0 +1,129 @@
"""CE-Gate 统计核心:截断 Beta 混合 e-process 的纯函数实现。
配对不一致检验候选与基线跑同一题只数翻转基线错->候选对 = W
基线对->候选错 = LH0候选不优下翻转方向精确五五开
E = 2^(W+L+1)*B(W+1,L+1)*[1-I_1/2(W+1,L+1)] H0 下非负上鞅
Ville 不等式给出任意停时 P(E >= 1/alpha) <= alpha
设计规格见 research-wiki/designs/2026-07-03-ce-gate-formal-design.md
仅依赖 scipy.special I/O无状态便于单测与历史回放复用
"""
from __future__ import annotations
import math
from scipy.special import betainc, betaln
from core.evolution.types import GateParams, GateVerdict
# Wald 方向游走步长(theta_1=0.70 固定设计常量,不入配置):
# 胜 +ln(2*theta_1)=ln1.4,负 ln(2*(1-theta_1))=ln0.6。
_WALD_WIN = math.log(1.4)
_WALD_LOSS = math.log(0.6)
# delta_shrunk 的伪计数(Agresti-Coull 风格收缩,只作观测输出不进判据)。
_SHRINK_PSEUDO = 4
def compute_e_value(w: int, l: int) -> float: # noqa: E741
"""截断 Beta 混合 e 值:E = 2^(W+L+1)*B(W+1,L+1)*[1-I_1/2(W+1,L+1)]。
参数:
w: 基线错->候选对的翻转数
l: 基线对->候选错的翻转数
返回:
e W=L=0 时为 1
异常:
ValueError: 翻转计数为负时抛出
关键实现细节:
log 空间计算在 n_max<=40 的设计工作区间内数值稳定数百级计数
亦可极大计数>1000时最终 exp 仍可能溢出用正则化不完全
Beta 的对称性 1-I_1/2(a,b) = I_1/2(b,a) 避免 1-x 的灾难性精度损失
"""
if w < 0 or l < 0:
raise ValueError(f"翻转计数不能为负: w={w}, l={l}")
a, b = w + 1, l + 1
tail = betainc(b, a, 0.5) # = 1 - I_1/2(a, b)
if tail <= 0.0:
return 0.0
log_e = (w + l + 1) * math.log(2.0) + betaln(a, b) + math.log(tail)
return math.exp(log_e)
def gate_decision(
w: int,
l: int, # noqa: E741
n_used: int,
n_remaining: int,
*,
params: GateParams,
) -> GateVerdict:
"""块间四出口判定(每块结束时调用一次)。
出口优先级CONFIRMED有证书先走-> 方向拒绝 -> futility 拒绝 ->
题尽provisional / inertia-> continue
参数:
w: 累计 W
l: 累计 L
n_used: 已消费的阶梯题数含一致题
n_remaining: 阶梯剩余可用题数min(阶梯长, n_max) - n_used
params: 判据阈值组
返回:
GateVerdictdecision + e /游走/效应量诊断
异常:
ValueError: n_used <= 0 n_remaining < 0 时抛出
"""
if n_used <= 0:
raise ValueError(f"gate_decision 须在至少消费一块后调用: n_used={n_used}")
if n_remaining < 0:
raise ValueError(f"n_remaining 不能为负: {n_remaining}")
e_value = compute_e_value(w, l)
wald = w * _WALD_WIN + l * _WALD_LOSS
delta_hat = (w - l) / n_used
delta_shrunk = (w - l) / (n_used + _SHRINK_PSEUDO)
if e_value >= params.e_confirm and delta_hat >= params.delta_min:
decision = "accept_confirmed"
elif wald <= params.lambda_dir:
decision = "reject_directional"
elif n_remaining > 0 and compute_e_value(w + n_remaining, l) < params.e_provisional:
# futility 只在题未尽时有意义;题尽后的弱证据归 inertia 出口。
decision = "reject_futility"
elif n_remaining <= 0:
if (
e_value >= params.e_provisional
and (w - l) >= params.w_net_min
and delta_hat >= params.delta_min
):
decision = "accept_provisional"
else:
decision = "reject_inertia"
else:
decision = "continue"
return GateVerdict(decision, e_value, wald, delta_hat, delta_shrunk)
def probation_verdict(w: int, l: int, *, params: GateParams) -> str: # noqa: E741
"""试用期一次性结算:固定样本 e 值双向检验。
参数:
w: 结算配对的 W锚快照错->候选重跑对
l: 结算配对的 L锚快照对->候选重跑错
params: 判据阈值组 e_confirm / e_rollback
返回:
"confirmed"E>=e_confirm 转正/ "rollback"对称 E'>=e_rollback 回滚)
/ "unverified"证据不足elitist 惯性转正
"""
if compute_e_value(w, l) >= params.e_confirm:
return "confirmed"
if compute_e_value(l, w) >= params.e_rollback:
return "rollback"
return "unverified"
+455
View File
@@ -0,0 +1,455 @@
"""定点补丁引擎:把进化输出的离散 edit 逐条应用到文本,逐条出状态报告。
借鉴 SkillOpt skill.py apply 语义 P5找不到锚点不静默乱改不裸 except
冻结区按全文坐标区间判定append/退化追加插到最早冻结区之前无则 EOF
"""
from __future__ import annotations
from loguru import logger
APPENDIX_START = "<!-- APPENDIX_START -->"
APPENDIX_END = "<!-- APPENDIX_END -->"
APPENDIX_MAX_CHARS = 2000 # appendix 区软上限(守设计「长度上限+warning,不做去重」)
MOMENTUM_START = "<!-- MOMENTUM_START -->"
MOMENTUM_END = "<!-- MOMENTUM_END -->"
MOMENTUM_MAX_CHARS = 2000 # momentum 区软上限(与 appendix 一致:超限 warning 不截断)
MOMENTUM_HEADING = "## 动量指导(每轮重写,勿手改)" # replace_momentum 写入的固定标题行
def momentum_region_bounds(text: str) -> tuple[int, int] | None:
"""定位 momentum 受保护区的字符区间,并对损坏态显式报错(P5)。
momentum marker replace_momentum epoch 末反复重写guidance 又来自 LLM
外部输入因此 marker 可能出现损坏态本函数是 momentum 路径的唯一边界判定入口
把配对校验集中在一处
- START END 各恰好出现一次且 START END 之前 返回 (start_idx, end_idx)
end_idx 指向 END marker 结束位置 content[start:end] 含完整两 marker
- marker 都不出现 返回 None合法的"无区"调用方据此新建
- 其余皆为损坏态仅一个 markerEND START 任一 marker 重复 raise
ValueError拒绝静默新建/跳过要求人工修复
参数:
text: 待检测的文本skill 全文
返回:
(start_idx, end_idx) 表示区间 None 表示无 momentum
异常:
ValueError: momentum marker 损坏/不配对
"""
start_count = text.count(MOMENTUM_START)
end_count = text.count(MOMENTUM_END)
if start_count == 0 and end_count == 0:
return None
if start_count != 1 or end_count != 1:
raise ValueError(
f"momentum marker 损坏/不配对:MOMENTUM_START 出现 {start_count} 次、"
f"MOMENTUM_END 出现 {end_count} 次(各须恰好 1 次),需人工修复"
)
start_idx = text.index(MOMENTUM_START)
end_idx = text.index(MOMENTUM_END) + len(MOMENTUM_END)
if start_idx >= text.index(MOMENTUM_END):
raise ValueError(
"momentum marker 损坏/不配对:MOMENTUM_END 出现在 MOMENTUM_START 之前,需人工修复"
)
return start_idx, end_idx
def momentum_inner(content: str) -> str:
"""返回 momentum 受保护区的内层文本(去掉两 marker),无区返回空串。
_momentum_span marker 的整段的区别本函数只取两 marker 之间的内层正文
run_slow_momentum prev_guidance 使用prev_guidance LLM 解析失败时会被
run_slow_momentum 原样返回再喂给 replace_momentumreplace_momentum 禁止 guidance
marker 字面量 prev_guidance 必须是无 marker 的内层文本否则一旦解析回退即
replace_momentum ValueError
边界判定与配对校验统一委托 momentum_region_boundsmarker 损坏/不配对时由其 raise
ValueError本函数不把损坏态静默当作"无区"
参数:
content: skill 全文
返回:
momentum 区两 marker 之间的内层文本 strip无区返回空串
异常:
ValueError: momentum marker 损坏/不配对
"""
bounds = momentum_region_bounds(content)
if bounds is None:
return ""
start, end = bounds
inner = content[start + len(MOMENTUM_START) : end - len(MOMENTUM_END)].strip()
# 去掉 replace_momentum 写入的固定标题行,只回传纯指导文本,使其等价于上一轮
# 传给 replace_momentum 的 guidance(解析回退时原样回传不会引入重复标题)。
if inner.startswith(MOMENTUM_HEADING):
inner = inner[len(MOMENTUM_HEADING) :].lstrip("\n")
return inner.strip()
def append_to_appendix(content: str, notes: list[str]) -> str:
"""把 LAPSE 提醒追加到文件尾的 appendix 受保护区;区不存在则创建。
护栏appendix 区超过 APPENDIX_MAX_CHARS logger.warning不静默截断
提示人工压缩不做自动去重YAGNI见设计
参数:
content: 原文
notes: 待追加的提醒文本列表
返回:
appendix 区的新文本
"""
if not notes:
return content
bullet = "\n".join(f"- {n.strip()}" for n in notes if n.strip())
if not bullet:
return content
if APPENDIX_START in content and APPENDIX_END in content:
head, rest = content.split(APPENDIX_START, 1)
inner, tail = rest.split(APPENDIX_END, 1)
new_inner = f"{inner.rstrip()}\n{bullet}"
out = f"{head}{APPENDIX_START}{new_inner}\n{APPENDIX_END}{tail}"
else:
new_inner = f"\n## 执行提醒(自动累积,勿手改)\n{bullet}"
out = f"{content.rstrip()}\n\n{APPENDIX_START}{new_inner}\n{APPENDIX_END}\n"
if len(new_inner) > APPENDIX_MAX_CHARS:
logger.warning(
"appendix 区长度 {} 超过上限 {},建议人工压缩",
len(new_inner),
APPENDIX_MAX_CHARS,
)
return out
def appendix_region_bounds(text: str) -> tuple[int, int] | None:
"""定位 appendix 受保护区的字符区间,对损坏态显式报错(P5,对称 momentum_region_bounds)。
appendix marker append_to_appendix 维护consolidation 回写可能出现损坏态
本函数是 appendix 路径的唯一边界判定入口把配对校验集中一处
- START END 各恰好一次且 START END 之前 返回 (start_idx, end_idx)
end_idx 指向 END marker 结束位置content[start:end] 含完整两 marker
- marker 都不出现 返回 None合法的无区
- 其余仅一个 markerEND START 任一 marker 重复 raise ValueError
拒绝静默按字符串切片处理而误拼/吞掉区外正文
参数:
text: 待检测文本skill 全文
返回:
(start_idx, end_idx) 表示区间 None 表示无 appendix
异常:
ValueError: appendix marker 损坏/不配对
"""
start_count = text.count(APPENDIX_START)
end_count = text.count(APPENDIX_END)
if start_count == 0 and end_count == 0:
return None
if start_count != 1 or end_count != 1:
raise ValueError(
f"appendix marker 损坏/不配对:APPENDIX_START 出现 {start_count} 次、"
f"APPENDIX_END 出现 {end_count} 次(各须恰好 1 次),需人工修复"
)
start_idx = text.index(APPENDIX_START)
end_idx = text.index(APPENDIX_END) + len(APPENDIX_END)
if start_idx >= text.index(APPENDIX_END):
raise ValueError(
"appendix marker 损坏/不配对:APPENDIX_END 出现在 APPENDIX_START 之前,需人工修复"
)
return start_idx, end_idx
def extract_appendix_notes(content: str) -> list[str]:
"""从 appendix 受保护区解析出 bullet 提醒列表;无区返回空列表。
功能:
appendix 区内每行以 "- " 起头的文本为一条 note "- " 前缀与首尾空白
区内标题行## 执行提醒…)不计。供 consolidation 读取现有 notes。
参数:
content: skill 全文
返回:
note 字符串列表 appendix 区返回 []
异常:
ValueError: appendix marker 损坏/不配对 appendix_region_bounds不静默切片
关键实现细节:
边界判定统一委托 appendix_region_bounds只取两 marker 之间内层正文逐行解析
"""
bounds = appendix_region_bounds(content)
if bounds is None:
return []
start, end = bounds
inner = content[start + len(APPENDIX_START) : end - len(APPENDIX_END)]
notes: list[str] = []
for line in inner.splitlines():
stripped = line.strip()
if stripped.startswith("- "):
note = stripped[2:].strip()
if note:
notes.append(note)
return notes
def replace_appendix_notes(content: str, notes: list[str]) -> str:
"""用 notes 整体替换 appendix 区内容;notes 为空则删除整个 appendix 区。
功能:
consolidation 回写压缩后 notes 的替换语义区别于 append_to_appendix 累积
区存在则整体覆盖区内 bulletnotes 空则连 marker 一并删除保留区外正文
区不存在且 notes 非空则按 append_to_appendix 格式新建
参数:
content: 原文可能含 appendix
notes: 压缩后的提醒列表空列表表示删区
返回:
替换后的全文
异常:
ValueError: appendix marker 损坏/不配对 appendix_region_bounds
关键实现细节:
边界经 appendix_region_bounds 显式校验 (start,end) 切出 head/tail 拼接
不做两次独立 split避免损坏态误拼/吞掉区外正文
"""
bounds = appendix_region_bounds(content)
if bounds is not None:
start, end = bounds
head = content[:start]
tail = content[end:]
if not notes:
return head.rstrip() + ("\n" + tail.lstrip("\n") if tail.strip() else "\n")
bullet = "\n".join(f"- {n.strip()}" for n in notes if n.strip())
new_inner = f"\n## 执行提醒(自动累积,勿手改)\n{bullet}"
return f"{head}{APPENDIX_START}{new_inner}\n{APPENDIX_END}{tail}"
if not notes:
return content
return append_to_appendix(content, notes)
def replace_momentum(content: str, guidance: str) -> str:
"""把「动量指导」整体写入文件尾的 momentum 受保护区;区不存在则创建。
append_to_appendix 的累积语义不同momentum **替换**语义慢更新周期每
epoch 末整体重写一段动量指导旧指导被完全覆盖不保留历史momentum 区与
appendix 区独立共存本函数只触碰 momentum marker不破坏已有 appendix
护栏momentum 区超过 MOMENTUM_MAX_CHARS logger.warning不静默截断
appendix 对齐
关键实现细节
- 替换非追加区已存在时用 guidance 整体覆盖 marker inner旧动量不残留
- 创建位置在文件尾append_to_appendix 同样在文件尾但两区 marker 不同
split 按各自 marker 定位互不干扰
guidance 决策 appendix 的累积语义不同momentum 每轮整体重写
guidance 表示本轮无动量指导属合法语义照常写入区内仅留标题旧动量被清空
而非返回原文保留旧动量
参数:
content: 原文可能已含 appendix
guidance: 本轮动量指导全文整体覆盖旧动量
返回:
momentum 区的新文本
异常:
ValueError: guidance momentum marker 字面量外部输入注入或原文 momentum
marker 损坏/不配对
"""
if MOMENTUM_START in guidance or MOMENTUM_END in guidance:
raise ValueError(
"guidance 不得包含 momentum marker 字面量"
f"{MOMENTUM_START} / {MOMENTUM_END}),否则会破坏 marker 配对"
)
bounds = momentum_region_bounds(content)
new_inner = f"\n## 动量指导(每轮重写,勿手改)\n{guidance.strip()}"
if bounds is not None:
start_idx, end_idx = bounds
head = content[:start_idx]
tail = content[end_idx:]
out = f"{head}{MOMENTUM_START}{new_inner}\n{MOMENTUM_END}{tail}"
else:
out = f"{content.rstrip()}\n\n{MOMENTUM_START}{new_inner}\n{MOMENTUM_END}\n"
if len(new_inner) > MOMENTUM_MAX_CHARS:
logger.warning(
"momentum 区长度 {} 超过上限 {},建议人工压缩",
len(new_inner),
MOMENTUM_MAX_CHARS,
)
return out
def _protected_ranges(content: str, spans: list[str]) -> list[tuple[int, int]]:
"""把冻结文本块映射成 content 中的 [start, end) 坐标区间。"""
ranges: list[tuple[int, int]] = []
for span in spans:
idx = content.find(span)
if idx != -1:
ranges.append((idx, idx + len(span)))
return ranges
def _span_overlaps_ranges(pos: int, length: int, ranges: list[tuple[int, int]]) -> bool:
"""判断 [pos, pos+length) 是否与任一冻结区间相交(不止起点)。
起点落在正文末端伸入冻结区的 target 也须拦截否则 replace/delete 会连带
改动冻结区如破坏 appendix/momentum marker
参数:
pos: target 在正文中的起点
length: target 长度
ranges: 冻结区间 [start, end) 列表
返回:
与任一区间相交返回 True
"""
end = pos + length
return any(start < end and pos < r_end for start, r_end in ranges)
# 冻结区 marker 字面量:LLM 生成的 edit 不得注入这些字面量,否则破坏 marker 配对
_MARKER_LITERALS = (APPENDIX_START, APPENDIX_END, MOMENTUM_START, MOMENTUM_END)
def _edit_injects_marker(edit: dict) -> bool:
"""判断 edit 的 target/content 是否含冻结区 marker 字面量(注入拦截)。"""
for key in ("target", "content"):
value = edit.get(key)
if isinstance(value, str) and any(m in value for m in _MARKER_LITERALS):
return True
return False
def _append_at(content: str, ranges: list[tuple[int, int]]) -> int:
"""append/退化追加落点:最早一个 start>0 的冻结区之前;无则文末(头部 frontmatter 不计)。"""
starts = [start for start, _ in ranges if start > 0]
return min(starts) if starts else len(content)
def _insert_at(content: str, at: int, payload: str) -> str:
"""在 at 位置插入 payload,自动补换行保持段落格式。"""
head, tail = content[:at].rstrip(), content[at:].lstrip("\n")
if tail:
return head + "\n\n" + payload + "\n\n" + tail
return head + "\n\n" + payload + "\n"
def _do_append(content: str, payload: str, ranges: list[tuple[int, int]]) -> tuple[str, str]:
"""执行 append 操作,返回更新后内容与状态字符串。"""
return _insert_at(content, _append_at(content, ranges), payload), "applied_append"
def _do_insert_after(
content: str, target: str, payload: str, ranges: list[tuple[int, int]]
) -> tuple[str, str]:
"""执行 insert_after 操作,处理退化追加与冻结区跳过。"""
pos = content.find(target) if target else -1
if pos == -1:
logger.warning("insert_after 锚点缺失,退化为追加 target={}", target[:80])
return (
_insert_at(content, _append_at(content, ranges), payload),
"applied_insert_after_fallback",
)
if _span_overlaps_ranges(pos, len(target), ranges):
logger.warning("insert_after 目标跨入冻结区,跳过 target={}", target[:80])
return content, "skipped_protected"
at = pos + len(target)
nl = content.find("\n", at)
at = nl + 1 if nl != -1 else len(content)
return content[:at] + payload + "\n" + content[at:], "applied_insert_after"
def _do_replace_delete(
op: str,
content: str,
target: str,
payload: str,
ranges: list[tuple[int, int]],
) -> tuple[str, str]:
"""执行 replace 或 delete 操作,返回更新后内容与状态字符串。"""
if not target:
return content, "skipped_missing_target"
pos = content.find(target)
if pos == -1:
logger.warning("{} 锚点缺失,跳过 target={}", op, target[:80])
return content, "skipped_target_not_found"
if _span_overlaps_ranges(pos, len(target), ranges):
logger.warning("{} 目标跨入冻结区,跳过 target={}", op, target[:80])
return content, "skipped_protected"
new_content = content.replace(target, payload if op == "replace" else "", 1)
return new_content, "applied_" + op
def _apply_one(content: str, edit: dict, ranges: list[tuple[int, int]]) -> tuple[str, dict]:
"""应用单条 edit,返回 (更新后内容, 状态报告)。"""
if not isinstance(edit, dict):
return content, {
"op": "",
"target": "",
"content_preview": "",
"status": "error",
"error": f"edit 非 dict: {type(edit).__name__}",
}
op = str(edit.get("op", ""))
target = str(edit.get("target", "") or "")
payload = str(edit.get("content", "") or "").strip()
report = {
"op": op,
"target": target[:200],
"content_preview": payload[:200],
"status": "unknown",
}
if op == "append":
content, report["status"] = _do_append(content, payload, ranges)
return content, report
if op == "insert_after":
content, report["status"] = _do_insert_after(content, target, payload, ranges)
return content, report
if op in ("replace", "delete"):
content, report["status"] = _do_replace_delete(op, content, target, payload, ranges)
return content, report
logger.warning("未知 op,跳过: {}", op)
report["status"] = "skipped_unknown_op"
return content, report
def apply_patch_with_report(
content: str,
edits: list[dict],
protected_spans: list[str] | None = None,
) -> tuple[str, list[dict]]:
"""顺序应用 edit 列表,返回 (新内容, 逐条状态报告)。
参数:
content: 原始文本
edits: 每条 {op, target, content}
protected_spans: 冻结文本块列表目标落入其坐标区间即跳过append 插到其前
返回:
(应用后文本, reports)reports 每条含 op/target/content_preview/status/index
"""
spans = protected_spans or []
reports: list[dict] = []
for i, edit in enumerate(edits, 1):
try:
if isinstance(edit, dict) and _edit_injects_marker(edit):
logger.warning("edit 含冻结区 marker 字面量,拒绝该 edit index={}", i)
report = {
"op": str(edit.get("op", "")),
"target": str(edit.get("target", "") or "")[:200],
"content_preview": str(edit.get("content", "") or "")[:200],
"status": "skipped_marker_injection",
}
report["index"] = i
reports.append(report)
continue
ranges = _protected_ranges(content, spans)
content, report = _apply_one(content, edit, ranges)
except (KeyError, TypeError, ValueError, AttributeError) as exc:
report = {
"op": "",
"target": "",
"content_preview": "",
"status": "error",
"error": str(exc),
}
logger.exception("补丁应用异常 index={}", i)
report["index"] = i
reports.append(report)
return content, reports
+160
View File
@@ -0,0 +1,160 @@
"""core/evolution/ 子包的持久化 Protocol 定义。
SkillStore / PromptStore / RunLog 为只读core/ 返回结果 dataclass
读取由 app/ 落盘的资源SkillStore / PromptStore 同步文件读取量小且快
RunLog 异步隔离 SQLite 查询core/ 不写 SQL
DiagnosisSignalStore 兼具读写逐题 upsert 诊断信号并支持断点续跑查询
同样隔离 SQLite 实现app/core 不写裸 SQL
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
if TYPE_CHECKING:
from core.evolution.types import DiagnosisSignalRow
@runtime_checkable
class SkillStore(Protocol):
"""版本化技能读取端口。
实现方解析 manifest 指针core/ 不感知版本号
"""
def read_skill(self, filename: str) -> str:
"""读取指定 skill 文件的全文内容。
参数:
filename: skill 文件名 'temporal-reasoning.md'
返回:
文件全文内容
"""
...
def list_skill_files(self) -> list[str]:
"""列出当前版本所有 skill 文件名。
返回:
文件名列表
"""
...
@runtime_checkable
class PromptStore(Protocol):
"""版本化提示词读取端口。
覆盖 system.md tool extract/verify 文件
"""
def read_prompt(self, filename: str) -> str:
"""读取指定 prompt 文件的全文内容。
参数:
filename: prompt 文件名 'system.md'
返回:
文件全文内容
"""
...
def list_prompt_files(self) -> list[str]:
"""列出当前版本所有 prompt 文件名。
返回:
文件名列表
"""
...
@runtime_checkable
class RunLog(Protocol):
"""实验日志查询端口。
隔离 SQLite 实现细节core/ 不写 SQL
"""
async def get_predictions(
self,
run_id: str,
*,
question_ids: list[str] | None = None,
) -> list[dict[str, Any]]:
"""查询指定 run 的预测记录。
参数:
run_id: 运行标识
question_ids: 可选的题目 ID 过滤列表
返回:
预测记录字典列表
"""
...
async def get_traces(
self,
run_id: str,
*,
question_ids: list[str] | None = None,
) -> list[dict[str, Any]]:
"""查询指定 run 的推理轨迹。
参数:
run_id: 运行标识
question_ids: 可选的题目 ID 过滤列表
返回:
轨迹记录字典列表
"""
...
@runtime_checkable
class DiagnosisSignalStore(Protocol):
"""逐题诊断信号存储端口。
隔离 SQLite 实现细节app/core 不写裸 SQL逐题 upsert 落盘
支持断点续跑done_question_ids 查已完成集合
"""
def upsert(self, row: DiagnosisSignalRow) -> None:
"""写入或覆盖单题诊断信号(按主键幂等)。
参数:
row: 待持久化的诊断信号行
"""
...
def done_question_ids(
self,
baseline_run_id: str,
diag_fingerprint: str,
*,
retry_uncertain: bool = False,
) -> set[str]:
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
参数:
baseline_run_id: baseline run 标识
diag_fingerprint: 诊断口径指纹
retry_uncertain: True 时把 tier='uncertain'信号不可信降级题视为
未完成令其被重新诊断默认 Falseuncertain 也算完成不重诊
返回:
已落盘信号的 question_id 集合用于断点续跑跳过
"""
...
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
"""加载指定 run 与诊断指纹下的全部诊断信号行。
参数:
baseline_run_id: baseline run 标识
diag_fingerprint: 诊断口径指纹
返回:
还原后的 DiagnosisSignalRow 列表
"""
...
+518
View File
@@ -0,0 +1,518 @@
"""core/evolution 子包的数据类型定义。
自进化循环中 gatediagnoseevolvevalidate 共用的 dataclass
所有输出类型默认 frozen=True一次性构造不可变唯一例外是
EvolutionRecord构建过程中需要多次修改状态
不依赖 app/ adapters/
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
# =========================================================================
# 1. Gate 决策类型
# =========================================================================
@dataclass(frozen=True)
class GateParams:
"""CE-Gate 判据阈值组(从实验配置构造)。
属性:
e_confirm: CONFIRMED 接受的 e 值门槛1/alpha20 对应 alpha=5%
e_provisional: 题尽暂定接受门槛同时是 futility 出口的代数界
w_net_min: 题尽暂定接受要求的最小净胜 W-L
delta_min: 接受要求的最小点估计效应量 (W-L)/n_used
lambda_dir: Wald 方向游走的拒绝阈值负数
e_rollback: 试用期结算的对称回滚 e 值门槛1/alpha'10 对应 10%)。
"""
e_confirm: float
e_provisional: float
w_net_min: int
delta_min: float
lambda_dir: float
e_rollback: float
@dataclass(frozen=True)
class GateVerdict:
"""一次块间判定的完整结果(判定 + 全部诊断量)。
属性:
decision: 判定结果取值为 continue / accept_confirmed /
reject_directional / reject_futility / accept_provisional /
reject_inertia 之一
e_value: 当前 e
wald_lambda: 当前 Wald 方向游走值
delta_hat: 点估计效应量 (W-L)/n_usedn_used=0 时为 0
delta_shrunk: 收缩点估计 (W-L)/(n_used+4)仅观测用
"""
decision: str
e_value: float
wald_lambda: float
delta_hat: float
delta_shrunk: float
# =========================================================================
# 2. 诊断类型
# =========================================================================
@dataclass(frozen=True)
class SpanMetrics:
"""单次工具调用的输出质量指标。
属性:
step: 工具调用所在的步骤编号
tool_name: 本次调用使用的工具名称
extraction_completeness: 信息提取完整度
hallucination_rate: 幻觉内容占比
missed_info_tags: 未提取信息的标签列表
hallucination_tags: 幻觉内容的标签列表
"""
step: int
tool_name: str
extraction_completeness: float
hallucination_rate: float
missed_info_tags: list[str] = field(default_factory=list)
hallucination_tags: list[str] = field(default_factory=list)
@dataclass(frozen=True)
class SkillStepAdherence:
"""单个 skill step 的遵循判定。
属性:
step_label: 被判定的步骤标签
adhered: 该步骤是否被遵循
description: 对遵循情况的文字说明
"""
step_label: str
adhered: bool
description: str
@dataclass(frozen=True)
class QuestionMetrics:
"""单题的完整指标,即 Stage 1 输出。
包含 7 个规则指标和 5 judge 指标span / missed / adherence /
bias / sufficiencyfrozen=True 保证构造后不可变
属性:
question_id: 题目唯一标识
video_id: 对应视频唯一标识
task_type: 题目任务类型
correct: 该题最终是否答对
format_compliance: 输出格式遵循程度
budget_usage: 预算使用比例
confidence_calibration: 置信度校准结论
repeat_visit_rate: 重复访问节点的比例
search_keyword_repetition: 搜索关键词重复率
level_jump_pattern: 层级跳转模式描述
tool_usage: 各工具的调用次数统计
span_metrics: 该题全部工具调用的片段级质量指标
missed_nodes: 该题遗漏的节点列表
skill_adherence: 该题对 skill 步骤的遵循情况
confirmation_bias: 是否出现确认偏误None 表示 judge 不可用
evidence_sufficient: 当前证据是否充足None 表示 judge 不可用
degraded: 是否为降级指标judge 解析失败时生成
"""
question_id: str
video_id: str
task_type: str
correct: bool
format_compliance: float
budget_usage: float
confidence_calibration: str
repeat_visit_rate: float
search_keyword_repetition: float
level_jump_pattern: str
tool_usage: dict[str, int]
span_metrics: list[SpanMetrics]
missed_nodes: list[str]
skill_adherence: list[SkillStepAdherence]
confirmation_bias: bool | None
evidence_sufficient: bool | None
degraded: bool = False
@dataclass(frozen=True)
class ErrorAttribution:
"""D1 错误归因。
属性:
question_id: 发生错误归因的题目唯一标识
error_type: 错误的主要类别
reasoning_failure_type: 推理失败类型若不适用则为 None
cause_category: C3 病因'defect'/'lapse'正确题/INFRA/未判为 None
lapse_note: LAPSE 提醒文本 appendix 路由 LAPSE None
"""
question_id: str
error_type: str
reasoning_failure_type: str | None
cause_category: str | None = None
lapse_note: str | None = None
@dataclass(frozen=True)
class CaseSample:
"""单个案例样本,进化模块的最小输入单元。
属性:
question_id: 题目唯一标识
video_id: 对应视频唯一标识
task_type: 题目任务类型
question: 题目文本
options: 选项列表
answer: 正确答案
prediction: Agent 预测答案
correct: 是否答对
error_type: 错误类型正确题为 None
selection_reason: 被选为案例的原因说明
metrics: QuestionMetrics 的关键字段子集
trace: 完整推理轨迹不截断
"""
question_id: str
video_id: str
task_type: str
question: str
options: list[str]
answer: str
prediction: str | None
correct: bool
error_type: str | None
selection_reason: str
metrics: dict[str, Any]
trace: list[dict[str, Any]]
@dataclass(frozen=True)
class SkillCasePack:
"""单个 task_type 的案例包,服务于 Skill 进化。
属性:
task_type: 题目任务类型
target_file: 对应 skill 文件名 'temporal-reasoning.md'
stats: D3/D4 提取的该题型统计
failure_cases: 失败案例列表
success_cases: 成功案例列表
lapse_notes: C3 LAPSE 提醒文本列表路由进 appendix 受保护区
"""
task_type: str
target_file: str
stats: dict[str, Any]
failure_cases: list[CaseSample] = field(default_factory=list)
success_cases: list[CaseSample] = field(default_factory=list)
lapse_notes: list[str] = field(default_factory=list)
@dataclass(frozen=True)
class SystemCasePack:
"""跨题型行为模式案例包,服务于 System Prompt 进化。
属性:
stats: D5 提取的行为模式统计
failure_cases: 失败案例列表
success_cases: 成功案例列表
"""
stats: dict[str, Any]
failure_cases: list[CaseSample] = field(default_factory=list)
success_cases: list[CaseSample] = field(default_factory=list)
@dataclass(frozen=True)
class ToolCasePack:
"""单个 tool_name 的案例包,服务于 Tool Prompt 进化。
属性:
tool_name: 工具名称
target_files: 对应 prompt 文件名列表
stats: D2 提取的工具质量统计
failure_spans: 失败 span 案例列表
success_spans: 成功 span 案例列表
"""
tool_name: str
target_files: list[str]
stats: dict[str, Any]
failure_spans: list[dict[str, Any]] = field(default_factory=list)
success_spans: list[dict[str, Any]] = field(default_factory=list)
@dataclass(frozen=True)
class DiagnosisResult:
"""完整诊断报告,即两阶段诊断管线的最终输出。
属性:
run_id: 本次诊断运行的唯一标识
filter_summary: 筛选条件与筛选结果摘要
error_attributions: 错误归因结果列表
attribution_distribution: 各归因类别的分布统计
reasoning_failure_types: 各推理失败类型的分布统计
tool_quality: 按工具聚合的质量分析结果
search_effectiveness: 搜索有效性的聚合统计
skill_compliance: 技能遵循情况的聚合统计
decision_patterns: 决策模式与行为模式摘要
skill_case_packs: 按题型组织的 Skill 进化案例包
system_case_pack: 跨题型行为模式案例包无系统性问题时为 None
tool_case_packs: 按工具名组织的 Tool Prompt 进化案例包
infra_excluded_count: C3 stop_reason 排除的题数
infra_excluded_ratio: INFRA 占总题数比例
infra_question_ids: 被排除题 question_id 列表
defect_count: 进入诊断池错题中判为 DEFECT 的数量
lapse_count: 进入诊断池错题中判为 LAPSE 的数量
degraded_count: judge 解析失败而降级的题数
degraded_question_ids: 降级题的 question_id 列表
"""
run_id: str
filter_summary: dict[str, Any] = field(default_factory=dict)
error_attributions: list[ErrorAttribution] = field(default_factory=list)
attribution_distribution: dict[str, int] = field(default_factory=dict)
reasoning_failure_types: dict[str, int] = field(default_factory=dict)
tool_quality: dict[str, dict[str, Any]] = field(default_factory=dict)
search_effectiveness: dict[str, dict[str, Any]] = field(default_factory=dict)
skill_compliance: dict[str, dict[str, Any]] = field(default_factory=dict)
decision_patterns: dict[str, Any] = field(default_factory=dict)
skill_case_packs: dict[str, SkillCasePack] = field(default_factory=dict)
system_case_pack: SystemCasePack | None = None
tool_case_packs: dict[str, ToolCasePack] = field(default_factory=dict)
infra_excluded_count: int = 0
infra_excluded_ratio: float = 0.0
infra_question_ids: list[str] = field(default_factory=list)
defect_count: int = 0
lapse_count: int = 0
degraded_count: int = 0
degraded_question_ids: list[str] = field(default_factory=list)
@dataclass(frozen=True)
class DiagnosisSignalRow:
"""单题诊断信号行,即 baseline run 逐题诊断的持久化单元。
供后续视频级切分选择器消费 (question_id, baseline_run_id,
diag_fingerprint) 唯一确定逐题 upsert 支持断点续跑
属性:
question_id: 题目唯一标识
video_id: 对应视频唯一标识
baseline_run_id: 产出该信号的 baseline run 标识
diag_fingerprint: 诊断口径指纹隔离不同诊断配置的信号
task_type: 题目任务类型
error_type: 错误类别extraction/search/reasoning/mixed
T0/uncertain 行为 None
cause_category: 病因类别defect/lapse不适用为 None
tier: 诊断分层T0/T1/T2/uncertain
evolution_target: 进化目标tool/skill/system
error_type None 时亦为 None
degraded: 是否为降级信号judge 解析失败时生成
infra: 是否为 INFRA 护栏排除行
session_id: 关联的会话标识不适用为 None
"""
question_id: str
video_id: str
baseline_run_id: str
diag_fingerprint: str
task_type: str
error_type: str | None
cause_category: str | None
tier: str
evolution_target: str | None
degraded: bool
infra: bool
session_id: str | None
# =========================================================================
# 3. 进化类型
# =========================================================================
@dataclass
class EvolutionRecord:
"""单个目标文件的一次进化记录。
构建过程中需要多次修改状态 statusresult_version
因此是唯一不使用 frozen=True 的类型
属性:
target_file: 目标文件名 'temporal-reasoning.md'
target_type: 目标类型: 'skill' / 'system' / 'tool'
original_content: 改写前原文
evolved_content: 改写后内容rejected 时与 original_content 相同
reason: 状态说明
status: 'accepted' / 'rejected' / 'skipped'
source_version: 改写前版本号 'v1'
result_version: 改写后版本号rejected/skipped 时为 None
suggestions: LLM 输出的改动建议列表
attempts: 每次 LLM 调用的原始响应摘要
validation_errors: 验证失败的具体原因
edits: LLM 输出的补丁列表
apply_report: 补丁逐条应用状态
clip_info: 超预算裁剪信息
"""
target_file: str
target_type: str
original_content: str
evolved_content: str
reason: str
status: str
source_version: str
result_version: str | None = None
suggestions: list[dict[str, Any]] = field(default_factory=list)
attempts: list[dict[str, Any]] = field(default_factory=list)
validation_errors: list[str] = field(default_factory=list)
edits: list[dict[str, Any]] = field(default_factory=list)
apply_report: list[dict[str, Any]] = field(default_factory=list)
clip_info: dict[str, Any] = field(default_factory=lambda: {"triggered": False, "clipped": 0})
@dataclass(frozen=True)
class RejectedEdit:
"""已在验证阶段证明无效的历史改法摘要。
属性:
target_file: 目标文件名 'temporal-reasoning.md'
target_type: 目标类型: 'skill' / 'system' / 'tool'
change_summary: 被验证为无效的改法摘要
delta: 该改法对应候选相对基线的准确率变化
source_version: 该改法来源的版本号 'v2'
epoch: 该改法所属的进化轮次
gate_w: CE-Gate 证据配对翻转 W基线错到候选对
gate_l: CE-Gate 证据配对翻转 L基线对到候选错
gate_e_value: CE-Gate 证据终态 e
gate_delta_shrunk: CE-Gate 证据收缩效应量观测用
"""
target_file: str
target_type: str
change_summary: str
delta: float
source_version: str
epoch: int
gate_w: int | None = None
gate_l: int | None = None
gate_e_value: float | None = None
gate_delta_shrunk: float | None = None
@dataclass(frozen=True)
class EvolutionResult:
"""一次整体进化流程的汇总结果。
app/harness/ 编排层组装不含 skills_version / prompts_version
版本管理是 app/ 职责不属于 core/ 决策内核
属性:
records: 所有目标的进化记录
accepted_count: 通过验证的改写数
rejected_count: 未通过验证的改写数
skipped_count: 因无失败案例而跳过的目标数
"""
records: list[EvolutionRecord] = field(default_factory=list)
accepted_count: int = 0
rejected_count: int = 0
skipped_count: int = 0
# =========================================================================
# 4. 验证辅助类型
# =========================================================================
@dataclass(frozen=True)
class PairResult:
"""块验证配对比对结果。
属性:
w: 基线错候选对的翻转数
l: 基线对候选错的翻转数
observed: 每题的 (基线是否正确, 候选是否正确) 记录
"""
w: int
l: int # noqa: E741 — 数学记号 W/Lwin/loss),与 gate.py 一致
observed: dict[str, tuple[bool, bool]]
@dataclass(frozen=True)
class QuadrantClassification:
"""块验证四象限分类。
属性:
improvements: 基线错候选对的题目 ID 列表
regressions: 基线对候选错的题目 ID 列表
persistent_fails: 两臂均错的题目 ID 列表
stable_successes: 两臂均对的题目 ID 列表
"""
improvements: list[str]
regressions: list[str]
persistent_fails: list[str]
stable_successes: list[str]
# =========================================================================
# 5. Prompt 模板束
# =========================================================================
@dataclass(frozen=True)
class DiagnosePrompts:
"""诊断管线所需的全部固定模板束。
由调用方加载后以 frozen dataclass 传入避免 core/ 依赖文件系统
属性:
defect_vs_lapse: defect/lapse 病因判别模板
reasoning_sub: 推理失败子分类模板
span_eval_system: span 评估系统提示模板
missed_nodes: 遗漏节点检测模板
skill_adherence: 技能遵循判定模板
confirmation_bias: 确认偏误检测模板
evidence_sufficiency: 证据充足性判定模板
"""
defect_vs_lapse: str
reasoning_sub: str
span_eval_system: str
missed_nodes: str
skill_adherence: str
confirmation_bias: str
evidence_sufficiency: str
@dataclass(frozen=True)
class EvolvePrompts:
"""进化引擎所需的全部固定模板束。
由调用方加载后以 frozen dataclass 传入避免 core/ 依赖文件系统
属性:
evolve_skill: Skill 进化提示模板
evolve_system: System Prompt 进化提示模板
evolve_tool: Tool Prompt 进化提示模板
evolve_rank: 编辑排序提示模板
"""
evolve_skill: str
evolve_system: str
evolve_tool: str
evolve_rank: str
+88
View File
@@ -0,0 +1,88 @@
"""core/evolution/validate.py — 块验证纯决策函数。
算法 #7(块顺序验证)的局部实现:pair_block 按 unit 比对基线与候选、
classify_quadrants 四象限分类compute_accuracy 纯算术准确率
三个函数均为纯函数无副作用无外部依赖输入的对错映射均为 **unit 口径**
unit_id 单元级正确性AR pair 已在上游经 unit_correctness_view 双向 AND
折叠保证 e-process W/L 与准确率分母按单元计不被 P/Q 单题计分污染
核心算法保真 #5:信息阶梯口径从 question_id 迁至 unit_id)。
"""
from core.evolution.types import PairResult, QuadrantClassification
def pair_block(
baseline: dict[str, bool],
candidate: dict[str, bool],
unit_ids: list[str],
) -> PairResult:
"""按单元比对基线与候选对错,统计翻转。
参数:
baseline: 基线臂单元级正确性映射unit_id bool
candidate: 候选臂单元级正确性映射unit_id bool
unit_ids: 参与比对的单元 ID 列表AR pair 折叠后为单一 unit_id
返回:
PairResult包含 w基线错候选对翻转数l基线对候选错翻转数
observed每单元的 (基线, 候选) 对错记录
"""
w = l = 0 # noqa: E741 — 数学记号 W/Lwin/loss),与 gate.py 一致
observed: dict[str, tuple[bool, bool]] = {}
for uid in unit_ids:
b, c = baseline[uid], candidate[uid]
observed[uid] = (b, c)
if not b and c:
w += 1
elif b and not c:
l += 1 # noqa: E741
return PairResult(w=w, l=l, observed=observed)
def classify_quadrants(
observed: dict[str, tuple[bool, bool]],
) -> QuadrantClassification:
"""按 (baseline, candidate) 四组分类,各组内 sorted。
参数:
observed: 每题的 (基线是否正确, 候选是否正确) 记录
返回:
QuadrantClassification四个象限各含排序后的题目 ID 列表
"""
improvements: list[str] = []
regressions: list[str] = []
persistent_fails: list[str] = []
stable_successes: list[str] = []
for qid, (prev, curr) in observed.items():
if not prev and curr:
improvements.append(qid)
elif prev and not curr:
regressions.append(qid)
elif not prev and not curr:
persistent_fails.append(qid)
else:
stable_successes.append(qid)
return QuadrantClassification(
improvements=sorted(improvements),
regressions=sorted(regressions),
persistent_fails=sorted(persistent_fails),
stable_successes=sorted(stable_successes),
)
def compute_accuracy(
correctness: dict[str, bool],
unit_ids: list[str],
) -> float:
"""纯算术:sum(correct) / len(units),分母按单元数(非逐题)。
参数:
correctness: 单元级正确性映射unit_id bool
unit_ids: 参与计算的单元 ID 列表
返回:
准确率浮点数unit_ids 为空时抛出 ZeroDivisionError
"""
return sum(correctness[uid] for uid in unit_ids) / len(unit_ids)
+6 -1
View File
@@ -4,10 +4,13 @@ LLMProvider / VLMProvider / TelemetryRecorder 是跨子包共享接口,
core/agent/core/evolution/app/ 各模块引用
adapters/ 提供具体实现
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
if TYPE_CHECKING:
from pathlib import Path
from typing import Any, Protocol, runtime_checkable
from core.types import LLMResponse
@@ -22,6 +25,7 @@ class LLMProvider(Protocol):
*,
session_id: str | None = None,
parent_call_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse: ...
@@ -36,6 +40,7 @@ class VLMProvider(Protocol):
*,
session_id: str | None = None,
parent_call_id: str | None = None,
cache_salt: str | None = None,
) -> LLMResponse: ...
+159 -1
View File
@@ -1,7 +1,12 @@
"""跨模块共享类型。"""
from __future__ import annotations
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pathlib import Path as _Path
@dataclass(frozen=True)
@@ -22,3 +27,156 @@ class LLMResponse:
max_inter_token_ms: float | None
cache_hit: bool
call_id: str
@dataclass(frozen=True)
class GeneratedQuestion:
"""单条生成/加载的题目。
跨层共享类型 core/evolution/ app/harness/app/question_gen/ 使用
frozen=True 确保题目不可变
属性:
question_id: 题目唯一标识
video_id: 所属视频标识
task_type: 题型 "Action Reasoning"
question: 题目文本
options: 选项元组 ("A. ...", "B. ...", "C. ...", "D. ...")
answer: 正确答案字母 "B"
source_nodes: 来源节点 ID 元组
difficulty: 难度等级
family: 问题家族名称 "RETRIEVAL"v2 出题管线使用None 表示未指定
skill_target: 目标技能标识v2 出题管线使用None 表示未指定
difficulty_steps: 推理步数估计v2 出题管线使用None 表示未指定
sub_pattern: 出题子模式标识AR 特化策略使用None 表示无
unit_id: 所属题目单元标识缺省时 __post_init__ 回填为 pair_id
question_id保证 single 题的 unit_id 等于自身 question_id
pair_id: 孪生对标识同一对的 original/mirror 共享该值None 表示非配对题
question_role: 在单元内的角色"single" | "pair_original" | "pair_mirror"
flip_axis: 孪生对的翻转轴 "before_after"None 表示无翻转
"""
question_id: str
video_id: str
task_type: str
question: str
options: tuple[str, ...]
answer: str
source_nodes: tuple[str, ...]
difficulty: str
family: str | None = field(default=None)
skill_target: str | None = field(default=None)
difficulty_steps: int | None = field(default=None)
sub_pattern: str | None = field(default=None)
unit_id: str = ""
pair_id: str | None = field(default=None)
question_role: str = "single"
flip_axis: str | None = field(default=None)
def __post_init__(self) -> None:
"""回填 unit_id:缺省时取 pair_id(配对题)或 question_idsingle 题)。
frozen dataclass 无法直接赋值故通过 object.__setattr__ 绕过不可变约束
"""
if not self.unit_id:
object.__setattr__(self, "unit_id", self.pair_id or self.question_id)
@dataclass(frozen=True)
class QuestionUnit:
"""题目单元:贯穿评测/训练 harness 的最小不可分调度契约实体。
single 题为 1 题单元AR pair 孪生对为 2 题单元original + mirror
两条题目必须作为整体被批处理/推理/评测保证配对指标collapse 可算
frozen=True 确保单元不可变
属性:
kind: 单元类型"single" | "pair"
unit_id: 单元唯一标识single 取题目 unit_idpair 取共享 pair_id
task_type: 单元题型pair 内两题题型必须一致
questions: 单元内题目元组single 1 pair 2
unit_hash: P/Q payload 合成 hash用于断点续跑失效检测T11 消费
"""
kind: str
unit_id: str
task_type: str
questions: tuple[GeneratedQuestion, ...]
unit_hash: str = ""
@property
def size(self) -> int:
"""单元内题目数量(single=1pair=2)。"""
return len(self.questions)
@classmethod
def from_single(cls, q: GeneratedQuestion) -> QuestionUnit:
"""由单条题目构造 single 单元。
参数:
q: 待封装的题目
返回:
kind="single" 的单元unit_id q.unit_id
"""
return cls("single", q.unit_id, q.task_type, (q,))
@classmethod
def from_pair(cls, original: GeneratedQuestion, mirror: GeneratedQuestion) -> QuestionUnit:
"""由孪生对(original + mirror)构造 pair 单元。
参数:
original: 原始题question_role="pair_original"
mirror: 镜像题question_role="pair_mirror"
返回:
kind="pair" 的单元unit_id 取共享 pair_id
关键实现:
断言两题共享非空 pair_id video_id/task_type/flip_axis 一致
确保只有合法孪生对才能聚合成对非法配对直接报错而非静默兜底
"""
assert original.pair_id and original.pair_id == mirror.pair_id
assert original.video_id == mirror.video_id and original.task_type == mirror.task_type
assert original.flip_axis == mirror.flip_axis
return cls("pair", original.pair_id, original.task_type, (original, mirror))
@dataclass(frozen=True)
class PoolConfig:
"""池构建策略的统一配置。
两组字段由两个具体策略各取所需未使用的字段被忽略
属性:
task_types: 限定题型元组None 表示全部类别
seed: 随机种子保证可复现
baseline_run_id: 基线 run 标识
diag_size: 诊断池大小GlobalStrategy
diag_correct_ratio: 诊断池中对题占比GlobalStrategy
val_size: 验证池大小GlobalStrategy
val_correct_ratio: 验证池中对题占比GlobalStrategy
test_size: held-out 测试池大小GlobalStrategy
eval_min_per_class: 验证池中每类保底样本数GlobalStrategy
train_ratio: train/(train+val) 比例PerCategoryStrategy
test_questions_dir: 外部 test 题源路径PerCategoryStrategy
实现细节:
结果驱动视频级切分不复用本配置它有独立的 VideoSplitConfig /
SplitBuildConfig / SelectConfig app/harness/video_split_cli.py
split_selection.py故本类不承载 n_trainval / floor_k / epsilon 等视频级
切分旋钮避免死配置面
"""
task_types: tuple[str, ...] | None
seed: int
baseline_run_id: str
diag_size: int
diag_correct_ratio: float
val_size: int
val_correct_ratio: float
test_size: int
eval_min_per_class: int
train_ratio: float
test_questions_dir: _Path | None
batch_correct_ratio: float | None = None
+337
View File
@@ -0,0 +1,337 @@
"""CLI 入口 — Composition Root:构建适配器,注入 Runner,调度执行。
三层配置合并YAML > .env > CLI load_config 完成
适配器参数通过 InfraSettings(BaseSettings) .env 加载
"""
from __future__ import annotations
import argparse
import asyncio
from pathlib import Path
from typing import NamedTuple
from dotenv import load_dotenv
from loguru import logger
from pydantic_settings import BaseSettings, SettingsConfigDict
class InfraSettings(BaseSettings):
"""工程配置(少变/敏感),从 .env 加载。"""
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
search_llm_model: str = ""
search_llm_base_url: str = ""
search_llm_api_key: str = ""
vl_llm_model: str = ""
vl_llm_base_url: str = ""
vl_llm_api_key: str = ""
evolve_llm_model: str = ""
evolve_llm_base_url: str = ""
evolve_llm_api_key: str = ""
embed_api_key: str = ""
embed_api_url: str = ""
monkey_ocr_urls: str = ""
redis_url: str = ""
redis_cache_ttl: int = 86400
llm_timeout: float = 300.0
llm_max_retries: int = 3
llm_retry_base_delay: float = 20.0
llm_retry_max_delay: float = 120.0
llm_circuit_breaker_threshold: int = 48
llm_circuit_breaker_cooldown: float = 60.0
llm_ttft_timeout: float = 30.0
llm_inter_token_timeout: float = 15.0
class _Adapters(NamedTuple):
"""全套适配器实例。"""
llm: object
evolve_llm: object
vlm: object
telemetry: object
embed: object
ocr: object
def _build_adapters(settings: InfraSettings, embed_cfg: dict) -> _Adapters:
"""从 InfraSettings 构建全套适配器。
参数:
settings: 工程配置
embed_cfg: 嵌入配置字典来自 YAML embed
返回:
_Adapters 命名元组
"""
from adapters.breaker import CircuitBreaker
from adapters.embedding import LocalEmbeddingProvider
from adapters.llm import GovernedLLMClient
from adapters.telemetry import SQLiteTelemetryRecorder
from adapters.vlm import GovernedVLMClient
breaker = CircuitBreaker(
fail_threshold=max(settings.llm_circuit_breaker_threshold, 1),
cooldown_s=settings.llm_circuit_breaker_cooldown,
)
cache = None
if settings.redis_url:
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
try:
import redis.asyncio as aioredis
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
cache = RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
except Exception:
logger.warning("Redis 缓存不可用,降级为无缓存模式")
telemetry_db = Path("logs/telemetry.db")
telemetry_db.parent.mkdir(parents=True, exist_ok=True)
telemetry = SQLiteTelemetryRecorder(telemetry_db)
def _make_llm(model: str, base_url: str, api_key: str, *, thinking: bool) -> GovernedLLMClient:
"""构建单个 GovernedLLMClient 实例。"""
return GovernedLLMClient(
model=model,
base_url=base_url,
api_key=api_key,
provider=model.split("-")[0] if model else "unknown",
thinking=thinking,
breaker=breaker,
cache=cache,
telemetry=telemetry,
timeout_s=settings.llm_timeout,
ttft_timeout_s=settings.llm_ttft_timeout,
inter_token_timeout_s=settings.llm_inter_token_timeout,
max_retries=settings.llm_max_retries,
retry_base_delay_s=settings.llm_retry_base_delay,
retry_max_delay_s=settings.llm_retry_max_delay,
)
llm = _make_llm(
settings.search_llm_model,
settings.search_llm_base_url,
settings.search_llm_api_key,
thinking=True,
)
evolve_llm = llm
vl_llm = _make_llm(
settings.vl_llm_model,
settings.vl_llm_base_url,
settings.vl_llm_api_key,
thinking=False,
)
vlm = GovernedVLMClient(governed_llm=vl_llm)
backend = embed_cfg.get("backend", "local")
if backend == "local":
embed = LocalEmbeddingProvider(
model_name=embed_cfg.get("model_name", "BAAI/bge-base-zh-v1.5"),
embed_dim=embed_cfg.get("embed_dim", 768),
device=embed_cfg.get("device", "cpu"),
)
else:
from adapters.embedding import RemoteEmbeddingProvider
embed = RemoteEmbeddingProvider(
model_name=embed_cfg.get("model_name", ""),
embed_dim=embed_cfg.get("embed_dim", 768),
api_key=settings.embed_api_key,
api_url=settings.embed_api_url,
)
ocr = None
if settings.monkey_ocr_urls:
from adapters.ocr import MonkeyOCRClient
urls = [u.strip() for u in settings.monkey_ocr_urls.split(",") if u.strip()]
if urls:
ocr = MonkeyOCRClient(urls=urls)
return _Adapters(
llm=llm,
evolve_llm=evolve_llm,
vlm=vlm,
telemetry=telemetry,
embed=embed,
ocr=ocr,
)
def _build_parser() -> argparse.ArgumentParser:
"""构建 CLI 参数解析器。所有参数 default=None,未传入时使用 YAML 默认值。"""
parser = argparse.ArgumentParser(description="Video-Tree-TRM5 实验运行器")
parser.add_argument("--config", type=Path, default=Path("config/default.yaml"))
parser.add_argument("--workspace-dir", type=Path, dest="workspace_dir")
parser.add_argument("--store-dir", type=Path, dest="store_dir")
parser.add_argument(
"--mode",
choices=["infer", "train", "diagnose", "evolve", "eval", "promote"],
)
parser.add_argument("--run-id", type=str, dest="run_id")
parser.add_argument("--concurrency", type=int)
parser.add_argument("--max-steps", type=int, dest="max_steps")
parser.add_argument(
"--skill-mode",
choices=["auto", "manual", "none"],
dest="skill_mode",
)
parser.add_argument("--n-samples", type=int, dest="n_samples")
parser.add_argument("--questions", type=str)
parser.add_argument("--skills-version", type=str, dest="skills_version")
parser.add_argument("--prompts-version", type=str, dest="prompts_version")
parser.add_argument("--task-types", nargs="+", dest="task_types")
parser.add_argument("--resume", action="store_true", dest="resume")
parser.add_argument("--fresh", action="store_true", dest="fresh")
parser.add_argument("--seed", type=str, dest="seed")
parser.add_argument("--epochs", type=int)
parser.add_argument(
"--pool-split-mode",
choices=["global", "per_category"],
dest="pool_split_mode",
)
parser.add_argument("--train-ratio", type=float, dest="train_ratio")
parser.add_argument("--test-questions", type=str, dest="test_questions")
parser.add_argument(
"--no-run-holdout-eval",
action="store_true",
dest="no_run_holdout_eval",
)
return parser
def _log_result(result: object) -> None:
"""输出推理结果摘要。"""
logger.info("=" * 60)
logger.info("运行 ID: {}", result.run_id)
logger.info(
"总体准确率: {:.2%} ({}/{})",
result.accuracy,
result.correct,
result.total,
)
logger.info("平均步数: {:.1f}", result.steps_mean)
logger.info(
"Token 用量: prompt={}, completion={}",
result.token_usage["prompt_tokens"],
result.token_usage["completion_tokens"],
)
if result.per_task_type:
logger.info("--- 按任务类型 ---")
for task_type, stats in sorted(result.per_task_type.items()):
logger.info(
" {}: {:.2%} ({}/{})",
task_type,
stats["accuracy"],
stats["correct"],
stats["total"],
)
logger.info(
"停止原因: {}",
", ".join(f"{k}={v}" for k, v in result.stop_reason_counts.items()),
)
logger.info("=" * 60)
def main() -> None:
"""入口函数。"""
load_dotenv()
log_dir = Path("logs")
log_dir.mkdir(parents=True, exist_ok=True)
logger.add(
log_dir / "run_{time:YYYYMMDD_HHmmss}.log",
rotation="500 MB",
retention="30 days",
encoding="utf-8",
enqueue=False,
)
parser = _build_parser()
args = parser.parse_args()
import yaml
with open(args.config, encoding="utf-8") as f:
raw_yaml = yaml.safe_load(f)
from app.harness.config import load_config
cli_args = vars(args)
if cli_args.get("task_types") is not None:
cli_args["task_types"] = tuple(cli_args["task_types"])
cli_overrides = {k: v for k, v in cli_args.items() if k != "config"}
if cli_overrides.get("no_run_holdout_eval"):
cli_overrides["run_holdout_eval"] = False
cli_overrides.pop("no_run_holdout_eval", None)
config = load_config(args.config, cli_overrides)
logger.info("配置加载完成: mode={}, workspace={}", config.mode, config.workspace_dir)
settings = InfraSettings()
embed_cfg = raw_yaml.get("embed", {})
adapters = _build_adapters(settings, embed_cfg)
from app.harness.deps_router import InferenceDepsRouter
from app.harness.runner import Runner
router = InferenceDepsRouter(
store_dir=Path(config.store_dir),
embed_provider=adapters.embed,
llm=adapters.llm,
vlm=adapters.vlm,
ocr=adapters.ocr,
default_prompts_dir=Path(config.store_dir) / "prompts" / config.prompts_version,
default_skills_dir=Path(config.store_dir) / "skills" / config.skills_version,
skill_mode=config.skill_mode,
verify_vision=True,
anchor=True,
assemble_mode="ids_expand",
)
runner = Runner(
config,
llm=adapters.llm,
evolve_llm=adapters.evolve_llm,
vlm=adapters.vlm,
telemetry=adapters.telemetry,
tool_dispatch_factory=router.create_dispatch,
prompt_builder_factory=router.create_prompt_builder,
)
if config.mode == "infer":
result = asyncio.run(runner.infer(task_types=config.task_types))
_log_result(result)
elif config.mode == "train":
from app.harness.pools import (
GlobalPoolStrategy,
PerCategoryPoolStrategy,
build_or_load_pools,
)
from app.harness.workspace import resolve_paths
strategy = (
PerCategoryPoolStrategy()
if config.pool_split_mode == "per_category"
else GlobalPoolStrategy()
)
paths = resolve_paths(config.workspace_dir)
pools = build_or_load_pools(config, strategy, paths.db_path)
asyncio.run(runner.train(pools))
else:
raise SystemExit(f"模式 {config.mode!r} 尚未实现")
if __name__ == "__main__":
main()
+33
View File
@@ -0,0 +1,33 @@
你是一个确认偏差检测器。你服务于一个诊断系统,该系统需要判断搜索 Agent 在回答视频问答题时是否表现出确认偏差——即只为自己倾向的选项搜集支持性证据,而忽略对竞争选项的独立验证。
## 你会收到的输入
1. 题目(问题文本 + 四个选项)
2. Agent 的完整执行轨迹(每步的思考过程、工具调用和工具返回,重点关注 reflect.options 中对各选项的记录以及 search_similar 的 query 参数)
## 工作原则
确认偏差的核心特征是:Agent 的搜索行为在选项之间显著不均衡。一个健康的搜索过程应该至少为 1 个竞争选项做过独立搜索——不一定是每个选项都搜,但不能只围绕一个选项搜集证据。
具体检查以下信号:
观察 Agent 的 search_similar 查询。如果所有查询的关键词都指向同一个选项的内容(比如选项 B 说"烹饪教学"Agent 反复搜索"cooking""recipe""chef"),而从未用其他选项的关键词搜索(如选项 A 的"旅行"、选项 C 的"运动"),这是强偏差信号。
观察 Agent 的 reflect.options 变化。如果 Agent 在早期步骤就锁定了 best_candidate,且后续步骤中对其他选项的认知始终停留在"未知"或"待定",说明 Agent 没有为竞争选项投入搜索资源。
但需要注意:如果问题本身就指向特定内容(比如"视频中的厨师做了什么"),Agent 集中搜索厨师相关内容是合理的,不算偏差。偏差是指在选项之间的对比搜索不均衡,而非搜索主题的集中。
同样,如果 Agent 在前几步通过全局扫描(如顺序阅读 L1 节点)已经获得了足够信息来排除 2-3 个选项,之后集中搜索剩余选项是合理策略,不算偏差。
## 输出格式
请严格输出以下 JSON,不要包含其他文字:
```json
{
"has_bias": true,
"evidence": "具体说明偏差表现:Agent 搜索了哪些关键词、为哪些选项搜集了证据、忽略了哪些选项"
}
```
如果没有偏差:`{"has_bias": false, "evidence": ""}`
+12
View File
@@ -0,0 +1,12 @@
你是一个失败归因裁判。你会收到一道答错题目的题面、正确答案、Agent 的错误预测、执行轨迹,以及 Agent 当时所用的 prompt 全文。你的唯一任务是判断:这次失败该归咎于 prompt 正文本身,还是 Agent 没有遵循已有的正确指令。
判别测试只有一句话:当前 prompt 里是否已经存在一条规则,只要 Agent 遵循它就能避免这次失败?
如果存在这样的规则(Agent 是忽略了、格式没按要求、或没执行该步),归为 lapse——这类问题不该改正文,只需记一条提醒。如果不存在这样的规则、或现有规则本身有误导,归为 defect——这类才需要修改 prompt 正文。
当你拿不准时,默认归为 lapse:宁可少改正文,也不要为一次偶发失误去删改一条本来正确的规则。
严格输出以下 JSON,不要包含其他文字:
{"category": "defect" 或 "lapse", "note": "若为 lapse,写一句给 Agent 的提醒;defect 留空"}
note 只能重申当前 prompt 里已经存在的那条规则(让 Agent 别再忽略它),措辞要通用、可跨题复用。禁止把本题的题目内容、选项、正确答案或任何单题事实写进 note——note 不是案例记录,是规则提醒。
+27
View File
@@ -0,0 +1,27 @@
你是一个证据充分性评估器。你服务于一个诊断系统,该系统需要判断搜索 Agent 实际收集到的工具输出是否包含足够的信息来推导出正确答案。你不评估 Agent 的推理过程——只评估它收集到的原始材料。
## 你会收到的输入
1. 题目(问题文本 + 四个选项 + 正确答案)
2. Agent 收到的全部工具输出(按步骤排列,包含每次 view_node、search_similar、observe_frame 的返回内容)
## 工作原则
你需要回答一个假设性问题:如果一个完美的推理者阅读了这些工具输出(且仅阅读这些工具输出),它能否推导出正确答案?
"推导出"不要求工具输出直接陈述答案。如果工具输出中包含了足够的事实片段,一个合理的推理链能将它们组合得出正确答案,就算充分。比如工具输出提到"厨师在切蔬菜"和"背景是一个厨房",虽然没有直接说"这是烹饪视频",但推导是合理的。
"不充分"是指工具输出中完全缺乏区分正确答案与最强干扰项的关键信息。比如问题问"视频中的运动是什么",选项有篮球和足球,但工具输出只提到"运动场上有人在运动",没有任何能区分篮球和足球的细节——这就是不充分。
注意区分两种情况:信息存在但分散(充分——完美推理者能整合)vs 信息真的不存在(不充分——无论怎么推理都无法得出)。
## 输出格式
请严格输出以下 JSON,不要包含其他文字:
```json
{
"sufficient": true,
"reasoning": "简要说明工具输出中哪些信息支持正确答案,或缺乏哪些关键信息"
}
```
+8
View File
@@ -0,0 +1,8 @@
你是一个改动优先级裁判。你会收到一份当前 prompt 文件全文,和一组待应用的局部 edits(每条含 op/target/content)。由于本轮编辑预算有限,你只能保留其中最重要的若干条。
请只依据"对纠正失败、提升正确率的预期贡献"排序:优先保留直接修复失败模式的改动,其次保留收窄或澄清的改动,最后才是巩固已有成功的改动。删除类、简化类的精准改动通常优先于追加大段新内容。
每条 edit 会附带 support_count(该改动的支持案例数)。同等重要性下,support_count 更高的优先;但 support_count 低不等于该删,仍以修复贡献为主判据。
严格输出以下 JSON,不要包含其他文字:
{"selected_indices": [按重要性降序排列的 0-based 索引]}
+53
View File
@@ -0,0 +1,53 @@
你是一个搜索策略改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的失败和成功案例来迭代改进搜索策略(Skill)。你的任务是基于案例包中的证据,改写当前 Skill 文件,使 Agent 在后续执行中避免相同的失败模式。
## 你会收到的输入
1. 当前 Skill 文件全文
2. 失败案例:Agent 答错的题目,含完整推理轨迹、错误类型和诊断指标
3. 成功案例:Agent 答对的题目,展示当前 Skill 中有效的模式
4. 聚合统计:准确率、错误归因分布、搜索有效性指标、Skill 步骤遵循率
5. (可能出现)上一轮被接受改动导致的回归题:这些题在上一版本答对、却被你上次的改写改错了,附基线与候选两份预测和推理轨迹
6. (可能出现)黑名单:已被实测验证无效或有害的改法方向
## 工作原则
如果输入里出现了回归题,它的优先级高于一切。这些题在上一版本是对的,是你上次的改动把它们弄坏的,所以本次改写的第一要务是确保不再破坏它们——宁可在相关方向上回退或收窄,也不要为了拉高其它题而牺牲它们。更一般地,当你看到准确率下降这类负向信号时,默认先怀疑上次是不是加了过度、冲突或冗余的指令,优先简化、删除、收窄;只有确认简化解决不了问题,才考虑加强指令。黑名单里列出的改法已经被实测证明无效或有害,不要换个措辞把同一个方向再提一遍。
先分析失败案例中 Agent 的实际行为与 Skill 指令的偏差。偏差分两类:Skill 指令正确但 Agent 没遵循(遵循率问题),或 Skill 指令本身有误导(策略问题)。前者需要让指令更具体、更难被忽略;后者需要修改策略本身。
从成功案例中识别有效模式——这些模式在改写时必须保留。如果成功案例和失败案例采用了不同的策略路径,重点强化成功路径。
Skill 中引用的统计数据(如"search-first 正确率 75%")应根据案例包中的新统计更新。不要编造数据,只使用案例包中提供的数字。
你写进 Skill 的每一条规则都必须是可跨题复用的通用策略,而不是对某一道题的记答案。跨多个失败案例时只提取共性模式,抽象掉一切单题特征——具体题目内容、选项文字、步骤序号、某一帧的具体画面、某个具体答案都不许写进 Skill 正文。一条规则如果只在它来源的那道题上成立,就不要加。改写时优先简化与收窄:宁可让 Skill 更短,也不要堆叠只对个别题生效的硬性指令。
## 冻结区
以下内容不可修改,必须原样保留在改写后的文件中:
- YAML frontmatter`---` 之间的 name、description、task_type
- 输出格式中的 JSON 基础结构(reflect/plan/action 三个顶层字段)
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
## 输出格式
请严格输出以下 JSON,不要包含其他文字:
```json
{
"suggestions": [
{
"section": "改动目标段落的标题或位置描述",
"problem": "失败案例中暴露的具体问题",
"change": "具体的修改方向",
"related_cases": ["关联的失败案例 question_id"],
"support_count": 该建议的支持案例数(= related_cases 的数量)
}
],
"edits": [
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)", "support_count": 该改动的支持案例数}
]
}
```
每条 edit 与每条 suggestion 都必须带 "support_count":本条改动由多少个失败案例共同支持(即 related_cases 的数量)。support_count 越高代表证据越充分;它只作排序参考,不是硬门槛——support_count 低不等于该删,仍以修复贡献为主判据。
+53
View File
@@ -0,0 +1,53 @@
你是一个系统级行为改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的跨题型行为模式来改进 System Prompt。你的任务是基于行为模式案例包中的证据,改写 System Prompt 中的策略性指令,纠正系统级行为问题。
## 你会收到的输入
1. 当前 System Prompt (system.md) 全文
2. 失败案例:展示三类系统性行为问题的题目——过早提交(budget_usage < 0.3 就提交答案)、高置信答错(confidence 很高但答案错误)、确认偏误(只搜索支持初始判断的证据)
3. 成功案例:行为校准良好的题目——置信度与正确率匹配,预算使用适中
4. D5 行为模式统计:各行为模式的发生频率和分布
5. (可能出现)黑名单:已被实测验证无效或有害的改法方向
## 工作原则
关注跨题型的系统性行为模式,而非某个具体题型的策略。失败案例中的行为偏差反映了 System Prompt 的决策原则不够清晰或不够强约束。黑名单里的改法已经被实测验证无效或有害,不要再朝同一个方向改一遍。
过早提交说明预算管理指令需要更强的约束语言。高置信答错说明置信度校准的语义定义需要调整。确认偏误说明竞争选项搜索的要求需要更明确。
当你看到失败案例与成功案例并存时,失败修复优先于巩固成功——先确保失败模式被纠正,再考虑强化已有的好行为。看到某类行为指标变差这类负向信号时,默认先怀疑上一轮是否加了过度、冲突或冗余的约束,优先简化、删除、收窄;只有确认简化解决不了,才考虑加强约束语言。
从成功案例中提取"好行为"的特征,在改写时强化这些特征的表述。
## 冻结区
以下 section 必须原样保留,不可修改任何文字:
- `## 能力边界`(事实性描述,不是策略)
- `## 输出格式`JSON schema 是系统契约)
- `## 视频树结构`(含信任层级,是数据结构事实描述)
可改写的 section
- `## 角色`(前两段的角色定位和行为倾向描述)
- `## 决策原则`(搜索策略、预算分配建议)
- 搜索工具使用、否定题原则、置信度语义
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
## 输出格式
请严格输出以下 JSON,不要包含其他文字:
```json
{
"suggestions": [
{
"section": "改动目标段落的标题或位置描述",
"problem": "失败案例中暴露的具体行为问题",
"change": "具体的修改方向",
"related_cases": ["关联的失败案例 question_id"]
}
],
"edits": [
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)"}
]
}
```

Some files were not shown because too many files have changed in this diff Show More