Compare commits
399 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3dcc7d9899 | |||
| 9bb09a44d4 | |||
| b0be1f1ae5 | |||
| 172b7a8ec4 | |||
| eb12006d38 | |||
| 1930ad32a4 | |||
| 8958eee11b | |||
| 0b839937df | |||
| 23a64042fe | |||
| b3aba7c31d | |||
| 16993ed362 | |||
| ea6bec5421 | |||
| 9e8a254fbb | |||
| e8b66f85ab | |||
| c61a6dac84 | |||
| 30c1cf10c0 | |||
| 232afd525b | |||
| 0a8e1ad18b | |||
| 2ab8088c81 | |||
| 1e92928d4e | |||
| 21c360bc87 | |||
| b02db74237 | |||
| 85ccad3b72 | |||
| 0e8069af27 | |||
| 53a8c4be02 | |||
| 269e0d294c | |||
| cc01d5ed62 | |||
| 065c8ae1b9 | |||
| 766592d855 | |||
| 933de6251e | |||
| 5fdcd9b5aa | |||
| a1c4e051e7 | |||
| 6a9b1ea188 | |||
| 6911c83d66 | |||
| b3ba11c7a5 | |||
| ee69721ea3 | |||
| e1f08dcd3b | |||
| ef244c52bd | |||
| ed59c6091a | |||
| 9be3bdb8eb | |||
| 58a0203522 | |||
| 77fd35830c | |||
| efbdeb1647 | |||
| 65126feada | |||
| caea9150b3 | |||
| c44f6010eb | |||
| 25918a73ff | |||
| 7d02cded99 | |||
| e1cac644c8 | |||
| 03337af8f8 | |||
| b307f51340 | |||
| c68de849da | |||
| 1468a53b7a | |||
| 96884dd149 | |||
| d1516bf56b | |||
| 87908b23eb | |||
| 1cbfa97b8d | |||
| 200053fbdc | |||
| 7cc6aa5b23 | |||
| 564c92a043 | |||
| c5c11ea81b | |||
| 04bda25335 | |||
| 8f349c5c62 | |||
| a0c7e043e8 | |||
| 4d60a545f6 | |||
| a39846af6e | |||
| 5bb8319220 | |||
| 6432ffef9a | |||
| 06c575f0a4 | |||
| 4fd8a6bfef | |||
| 232a56e826 | |||
| 1108235818 | |||
| d3639a94b8 | |||
| 6cdf05d56b | |||
| 39a9210e4d | |||
| a4c429b247 | |||
| 11a5545f57 | |||
| d7e965639e | |||
| 57f590b57e | |||
| 224bb25bb5 | |||
| 8dedd082ff | |||
| 83056688cf | |||
| f1b6865861 | |||
| c5fff7f8b3 | |||
| d6c595c4a4 | |||
| 05294412df | |||
| e6adaad8b3 | |||
| 748a969e0e | |||
| 63dfee1b9b | |||
| 266a1ab4bc | |||
| 94728c5220 | |||
| 26ab536fea | |||
| 8fef7ced42 | |||
| 02b8145b7f | |||
| 2844732126 | |||
| 6a21d80313 | |||
| 43d7346526 | |||
| b497db97ba | |||
| c3187167c8 | |||
| 84b0b30213 | |||
| aa10485b9f | |||
| fd907aab46 | |||
| 53989078a0 | |||
| 3d8bd75372 | |||
| 20eea98cdd | |||
| 9d19328cc9 | |||
| 75a2aac1c0 | |||
| 0a7ba724a9 | |||
| 3a8dd5c167 | |||
| 6b85fcbe1c | |||
| 01f2e7c7b9 | |||
| cf7f15d8bb | |||
| 745b600b5d | |||
| cfa510a7e5 | |||
| 312cee7be7 | |||
| f8ad8f0143 | |||
| a536a81b0d | |||
| 13234426b5 | |||
| e9349b55c0 | |||
| 58a84dc72b | |||
| 3cc8dc9105 | |||
| 5ecbac620c | |||
| 1ecb6ba25c | |||
| bdcc93d7de | |||
| 19911e18e0 | |||
| 6c6fb576ee | |||
| bd1f7a22a2 | |||
| 273984674b | |||
| 7e97081779 | |||
| 4b6d1d8a50 | |||
| dee6bf4896 | |||
| 8a69a54078 | |||
| 61bfa0e633 | |||
| 730caa7e9a | |||
| 2429dad393 | |||
| c412698cff | |||
| d6a3107e4e | |||
| ddb9a44f75 | |||
| 5ef5f2b8b7 | |||
| bef46636fe | |||
| 7ef9b99217 | |||
| 0fe1c96393 | |||
| d9f7dee2df | |||
| 7949f16906 | |||
| fd96f906c2 | |||
| 49b7be78ae | |||
| d84f9bf777 | |||
| b1c1bf7aac | |||
| 441a0aa6c3 | |||
| f36eb66c18 | |||
| 1d222d9f18 | |||
| 73d0bb9190 | |||
| c1565a01c2 | |||
| 15aee0cfc1 | |||
| 8731e448fe | |||
| 4af99b700b | |||
| d4e9852864 | |||
| 24ed7ca322 | |||
| c109f2257a | |||
| 334fbbc94d | |||
| ac115d96fb | |||
| 8b9e8aa19f | |||
| d77cbc95eb | |||
| e41a2b0d08 | |||
| f12dd7e559 | |||
| 841112c6af | |||
| abc9c097d2 | |||
| 96e314c3a0 | |||
| 46ac848176 | |||
| eaab865d84 | |||
| e64c26e578 | |||
| 76f719018c | |||
| b1f15ddb3a | |||
| 58278c6de4 | |||
| ea38e4717b | |||
| b13eab0659 | |||
| d0194f5840 | |||
| 8a54055d02 | |||
| 207e834f30 | |||
| 3f984acc18 | |||
| 2608a3841f | |||
| e68e4b7d57 | |||
| 111c88488f | |||
| ae0a718f67 | |||
| 25f2a845ff | |||
| 92714db370 | |||
| 645585e784 | |||
| ad0bfe3a51 | |||
| 84b52a0311 | |||
| 72befa2bd4 | |||
| 453cf62088 | |||
| 73c5d1e380 | |||
| 89cad891de | |||
| d7d7ce5bdc | |||
| d0a8019fe1 | |||
| eaa58604b3 | |||
| b976c789d2 | |||
| cd0805aad6 | |||
| 7c7ad67d81 | |||
| 36c712defa | |||
| b9616de21e | |||
| afa77173e3 | |||
| c49d0ff12f | |||
| e2325b6535 | |||
| b6b6a48503 | |||
| 832838350a | |||
| 9ee37a8534 | |||
| dec7346da3 | |||
| 37d4519905 | |||
| 671db2f88c | |||
| 5563d0d675 | |||
| c66a00c924 | |||
| e5b07ac974 | |||
| 73ae1f7143 | |||
| 8b48005a17 | |||
| 21c6a53aed | |||
| ec4cbbdd44 | |||
| cd5c9c01fb | |||
| 0d0f275134 | |||
| 4fb7a61f8b | |||
| eecb86e27a | |||
| f46e87258c | |||
| 206c553143 | |||
| 6d6eb8e3a3 | |||
| cf51d2de9d | |||
| 9053233f99 | |||
| 9f739e831d | |||
| f74711cd11 | |||
| 271d1682c9 | |||
| 9627ac9cf9 | |||
| 7abe92eb1c | |||
| 9525726133 | |||
| 74686dde68 | |||
| 6a5424a618 | |||
| c83d771923 | |||
| 75e6d8c550 | |||
| 811ffa648b | |||
| 043d4aa46f | |||
| 8c9adfd3fa | |||
| a51b3c4238 | |||
| 0290005129 | |||
| 928ec8c37b | |||
| c5bf7e1a61 | |||
| 56fe138a46 | |||
| 978ddef91b | |||
| e9073bfdc2 | |||
| 25f5537974 | |||
| 1d2385291a | |||
| 40b156cf69 | |||
| ebbd112d50 | |||
| a046862ca1 | |||
| f0284199b8 | |||
| 1c21e215e2 | |||
| a2c6a97621 | |||
| 4e0e05210d | |||
| 291a8108e1 | |||
| badfcce4cb | |||
| e3184c11f9 | |||
| cf529f2c8f | |||
| 439dc29b3b | |||
| 6034d4172d | |||
| a31b1fbf37 | |||
| 8d84d5e236 | |||
| 39c6352781 | |||
| ef402c46a2 | |||
| a5666b4f16 | |||
| a79c2ec753 | |||
| 658e62054e | |||
| da70eb6e23 | |||
| 307c64c388 | |||
| 735471df3c | |||
| 25c8d5ec42 | |||
| d0148e78ad | |||
| 6a6aae081b | |||
| 69a4d17c0b | |||
| 5b51f4bd0c | |||
| 8d11513e54 | |||
| c191be16e2 | |||
| 09b2773375 | |||
| c84de49dc4 | |||
| 75016c4892 | |||
| 4ebdf0b0ba | |||
| 1f83da9eb7 | |||
| 38a855b33e | |||
| d6ae0d85cd | |||
| 953fb7a456 | |||
| cabe5be038 | |||
| f8f8b5b12d | |||
| ba400417e2 | |||
| a7ca6d15ed | |||
| f21bf345a6 | |||
| 924160c779 | |||
| e7be42570d | |||
| bb6dad7005 | |||
| 16140cafcc | |||
| d3be9b1322 | |||
| c72b55508a | |||
| 160fb3bc7c | |||
| 457944bbfa | |||
| 5d0caa06e1 | |||
| 09d337658b | |||
| 45403b23b4 | |||
| ace0738546 | |||
| f57ee45dc0 | |||
| fad8147d71 | |||
| 93c9be8bfa | |||
| 11f3c90200 | |||
| 6e46d184b8 | |||
| 5aa7cc48c5 | |||
| 90f17e330e | |||
| 40b04f886e | |||
| a597a9f901 | |||
| 9eb9b86954 | |||
| eb15ab315e | |||
| 0b48b889e0 | |||
| 8c383354c5 | |||
| 86573735aa | |||
| 8cb1158a2d | |||
| d6bcf41336 | |||
| 4612123ec4 | |||
| 847def4a03 | |||
| afe80a8b32 | |||
| 8182cb86b1 | |||
| f733c13dd1 | |||
| 5a91f392f0 | |||
| dbc9d38cd7 | |||
| 8a49ef18e6 | |||
| 53286ac3ab | |||
| 91281cb995 | |||
| 9a42bbcf85 | |||
| a668fc173c | |||
| d6e74f2734 | |||
| 6baddcc17d | |||
| 8c9f7d2d37 | |||
| 6b56fc8f98 | |||
| 49e272719e | |||
| 767ac55787 | |||
| d7d8633dae | |||
| 48b423ef35 | |||
| ad8fc176d0 | |||
| 461abf0329 | |||
| 6858da80c7 | |||
| fd76720dcf | |||
| b47f634e61 | |||
| 535bd320b0 | |||
| 5e57cf19b6 | |||
| ca02b5a91d | |||
| 63ad68f737 | |||
| 2db612e0bb | |||
| af8081e9ed | |||
| 283ea27b51 | |||
| 9ab6a89298 | |||
| 5c42e0309b | |||
| 2c5b4c819c | |||
| c6a81f8504 | |||
| fdbf9222ab | |||
| ebd83810aa | |||
| 7f8ed56428 | |||
| 57793fb0f8 | |||
| 47c2de71a2 | |||
| 8bbfbc9b8a | |||
| 2761ca3e82 | |||
| dc19e9e007 | |||
| c42813992f | |||
| 502681694b | |||
| 96e63a3e19 | |||
| 3fe077a249 | |||
| b8706adb33 | |||
| d83e720c4b | |||
| 224c75ba9e | |||
| 83e6475f90 | |||
| f0a49b08e5 | |||
| 2bc5dc74c2 | |||
| 112876de2e | |||
| e36f7c63f3 | |||
| e20bfac48d | |||
| 961e85111b | |||
| bc18fc8635 | |||
| 5ab5adec51 | |||
| 253f36797b | |||
| f94c352d66 | |||
| c4d42eeca0 | |||
| 8f5fbf8d2d | |||
| 9bdfd28c63 | |||
| 15fad65b1b | |||
| faf6f1bd63 | |||
| c61ceca538 | |||
| 8cfdc841da | |||
| 79761852df | |||
| af2aa751b9 | |||
| e87a744df9 | |||
| 3e348f5b21 | |||
| 6fd9522bf3 | |||
| 5ec8dd2f82 | |||
| 1b2442b9c5 | |||
| c4bc5e7cc6 | |||
| fd58a3f02b | |||
| d95b2a3142 | |||
| adf14633e5 |
@@ -22,6 +22,7 @@ Every project goes through this process. A todo list, a single-function utility,
|
||||
You MUST create a task for each of these items and complete them in order:
|
||||
|
||||
1. **Explore project context** — check files, docs, recent commits
|
||||
1.5. **Prior-version audit (mandatory for rewrites/refactors)** — if the task replaces or rewrites an existing module, list every behavior of the old version (including persistence, crash recovery, idempotency, resume) and confirm each is kept, replaced, or deliberately dropped. Undocumented implicit drops = bugs.
|
||||
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
|
||||
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
|
||||
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
|
||||
@@ -92,7 +93,12 @@ digraph brainstorming {
|
||||
- Once you believe you understand what you're building, present the design
|
||||
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
|
||||
- Ask after each section whether it looks right so far
|
||||
- Cover: architecture, components, data flow, error handling, testing
|
||||
- Cover: architecture, components, data flow, error handling, testing, **non-functional requirements** (see below)
|
||||
- **Non-functional requirements (mandatory section):** Every design MUST explicitly address these four dimensions — even if the answer is "not applicable":
|
||||
- **Persistence strategy:** When does data hit disk? How much is lost on crash? Overwrite or append?
|
||||
- **Idempotency:** Is the same operation safe to repeat? Does it produce the same result?
|
||||
- **Resume/checkpoint:** Can the process recover from interruption? How is progress persisted?
|
||||
- **Atomicity:** Are writes atomic? Can a partial write corrupt data?
|
||||
- Be ready to go back and clarify if something doesn't make sense
|
||||
|
||||
**Design for isolation and clarity:**
|
||||
@@ -124,6 +130,8 @@ After writing the spec document, look at it with fresh eyes:
|
||||
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
|
||||
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
|
||||
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
|
||||
5. **Non-functional coverage:** Does the design explicitly address persistence, idempotency, resume, and atomicity? If any dimension is missing, add it now — even if the answer is "not applicable."
|
||||
6. **Prior-version regression check (rewrites only):** If this replaces an existing module, confirm every behavior from the prior-version audit (step 1.5) is accounted for in the design. Any gap = a spec bug.
|
||||
|
||||
Fix any issues inline. No need to re-review — just fix and move on.
|
||||
|
||||
|
||||
+15
-11
@@ -4,28 +4,28 @@ NO_PROXY=dashscope.aliyuncs.com,api.deepseek.com
|
||||
|
||||
# ── 搜索 Agent LLM ──
|
||||
SEARCH_LLM_MODEL=deepseek-v4-pro
|
||||
SEARCH_LLM_BASE_URL=https://api.deepseek.com/v1
|
||||
SEARCH_LLM_API_KEY=sk-xxx
|
||||
SEARCH_LLM_BASE_URL=https://newapi.iomgaa.online/v1
|
||||
SEARCH_LLM_API_KEY=sk-your-api-key-here
|
||||
|
||||
# ── 评估 Judge LLM ──
|
||||
JUDGE_LLM_MODEL=deepseek-v4-pro
|
||||
JUDGE_LLM_BASE_URL=https://api.deepseek.com/v1
|
||||
JUDGE_LLM_API_KEY=sk-xxx
|
||||
JUDGE_LLM_BASE_URL=https://newapi.iomgaa.online/v1
|
||||
JUDGE_LLM_API_KEY=sk-your-api-key-here
|
||||
|
||||
# ── 视觉模型(Qwen VL)──
|
||||
VL_LLM_MODEL=qwen3.6-plus
|
||||
VL_LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
||||
VL_LLM_API_KEY=sk-xxx
|
||||
VL_LLM_BASE_URL=https://newapi.iomgaa.online/v1
|
||||
VL_LLM_API_KEY=sk-your-api-key-here
|
||||
|
||||
# ── 进化 LLM(Prompt 改写)──
|
||||
EVOLVE_LLM_MODEL=deepseek-v4-pro
|
||||
EVOLVE_LLM_BASE_URL=https://api.deepseek.com/v1
|
||||
EVOLVE_LLM_API_KEY=sk-xxx
|
||||
EVOLVE_LLM_BASE_URL=https://newapi.iomgaa.online/v1
|
||||
EVOLVE_LLM_API_KEY=sk-your-api-key-here
|
||||
|
||||
# ── ASR 字幕生成(Groq Whisper)──
|
||||
ASR_MODEL=whisper-large-v3
|
||||
ASR_BASE_URL=https://api.groq.com/openai/v1
|
||||
ASR_API_KEY=gsk-xxx
|
||||
ASR_API_KEY=gsk_your-groq-api-key-here
|
||||
|
||||
# ── MonkeyOCR ──
|
||||
MONKEY_OCR_URLS=http://10.77.0.20:7866,http://10.77.0.20:7867
|
||||
@@ -41,9 +41,13 @@ REDIS_URL=redis://localhost:6379/0
|
||||
LLM_TIMEOUT=120
|
||||
LLM_MAX_RETRIES=3
|
||||
LLM_RETRY_BASE_DELAY=2.0
|
||||
LLM_CIRCUIT_BREAKER_THRESHOLD=5
|
||||
LLM_CIRCUIT_BREAKER_THRESHOLD=48 # 实际阈值 = max(此值, concurrency*2)
|
||||
LLM_CIRCUIT_BREAKER_COOLDOWN=60
|
||||
LLM_TTFT_TIMEOUT=30
|
||||
LLM_INTER_TOKEN_TIMEOUT=15
|
||||
LLM_RETRY_MAX_DELAY=30.0
|
||||
REDIS_CACHE_TTL=86400
|
||||
# 正整数秒,禁止 0(0 会被拒绝启动);训练场景建议 >= 单次训练时长
|
||||
REDIS_CACHE_TTL=604800
|
||||
|
||||
# 建树批量并行:全局 VLM/LLM 在途调用上限(Spec-2 工程配置)
|
||||
TREE_BUILD_API_CONCURRENCY=16
|
||||
|
||||
@@ -181,6 +181,8 @@ pencil/
|
||||
|
||||
# 数据与实验产物(不提交)
|
||||
store/
|
||||
!store/prompts/
|
||||
!store/skills/
|
||||
workspaces/
|
||||
results/
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
> 2. 你的所有思考过程和回复必须使用 **简体中文**。
|
||||
|
||||
## 1. 项目元数据 (Metadata)
|
||||
- **核心目标**: 在层次化视频树上构建可自我进化的搜索 Agent + 可训练的递归检索器(RecursiveRetriever),通过 Harness Engineering(工具、技能、记忆、中间件)的持续改进实现长视频理解;服务于科研产出。详见 `research-wiki/ARCHITECTURE.md`、`README.md`。
|
||||
- **核心目标**: 在层次化视频树上构建可自我进化的搜索 Agent,通过 Harness Engineering(工具、技能、记忆、中间件)的持续改进实现长视频理解;服务于科研产出。详见 `research-wiki/ARCHITECTURE.md`、`README.md`。
|
||||
- **项目类型**: 科研工程混合体 + 生产级(非 MVP)
|
||||
- **目标会议**: AAAI 2026(2026年6月25日)
|
||||
- **后端架构**: Python 3.11(Clean Architecture 四层分层,详见 `research-wiki/ARCHITECTURE.md §2`)
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
| PyTorch 概念 | 本项目对应 | 代码位置 |
|
||||
|-------------|-----------|----------|
|
||||
| `DataLoader` | 出题 question_gen | `app/question_gen/generator.py` |
|
||||
| `DataLoader` | 出题 question_gen | `app/question_gen/loader.py` |
|
||||
| `model.forward()` | 推理 inference | `app/harness/inference.py` + `core/agent/loop.py` |
|
||||
| `loss.backward()` | 诊断 diagnose | `core/evolution/diagnose.py` |
|
||||
| `optimizer.step()` | 进化 evolve | `core/evolution/evolve.py` |
|
||||
@@ -96,9 +96,10 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
|
||||
### Phase 1: 规划与设计 (Planning)
|
||||
1. **需求探索**: 涉及创建新功能、新组件、修改行为时,**必须**先调用 `brainstorming` skill 进行需求探索与设计。无论用户的指令多么具体、改动多么简单,都不得跳过此步骤(除非用户显式说"跳过 brainstorming")。
|
||||
2. **查阅规格 & 讨论**: 仔细阅读 `research-wiki/`(单一事实源)下对应的文档,了解项目最新情况。对于不理解的地方请与人类进行多轮讨论,确保理解人类的设计意图。
|
||||
3. **日志方案设计**: 功能会产生运行时数据时,**必须**调用 `structured-logging` skill 设计日志方案。
|
||||
4. **撰写计划**: 正式编码前,**必须**调用 `writing-plans` skill 撰写实现计划。
|
||||
5. **审核门控(差异化)**:
|
||||
3. **前序版本对照(重写/重构时强制)**: 当任务涉及重写或重构已有模块时,**必须**列出前序版本的所有行为(包括持久化策略、崩溃恢复、幂等性、断点续跑等非功能性行为),逐一确认新版本是保留、替代、还是删除。未经确认的隐式删除 = bug。
|
||||
4. **日志方案设计**: 功能会产生运行时数据时,**必须**调用 `structured-logging` skill 设计日志方案。
|
||||
5. **撰写计划**: 正式编码前,**必须**调用 `writing-plans` skill 撰写实现计划。
|
||||
6. **审核门控(差异化)**:
|
||||
- **design:Claude 自审 → Codex 审 → 人类审**(保留人类门,批准后方可进入计划阶段)。
|
||||
- **plan:Claude 自审 → Codex 审 → 直接执行**(无 plan 人类门);plan 经 Claude 自审 + Codex 审通过后直接进入 Phase 2 执行。
|
||||
|
||||
@@ -159,6 +160,19 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
|
||||
- **功能修改**:
|
||||
- **必须** 不考虑向后兼容,直接修改原文件。代码简洁性优先。
|
||||
|
||||
### 4.2.1 设计文档非功能性需求覆盖(强制)
|
||||
|
||||
> **教训来源**: v2 出题管线重写时未继承 v1 的逐题追加持久化策略,导致多次 run 的题目丢失。
|
||||
|
||||
设计文档**必须**显式覆盖以下非功能性维度(即使答案是"不适用"也要写明):
|
||||
|
||||
| 维度 | 必答问题 |
|
||||
|------|---------|
|
||||
| **持久化策略** | 数据何时落盘?崩溃时最多丢多少?是覆盖写还是追加? |
|
||||
| **幂等性** | 同一操作重复执行是否安全?结果是否一致? |
|
||||
| **断点续跑** | 中断后重启能否从断点恢复?进度如何持久化? |
|
||||
| **原子性** | 写操作是否原子?部分写入是否会损坏数据? |
|
||||
|
||||
### 4.3 Git 工作流规范
|
||||
- **Feature Branch**: 所有开发工作在 feature 分支上进行,**严禁**直接在 main/master 上修改。
|
||||
- **增量提交**: 频繁提交,每个提交有明确的语义。
|
||||
@@ -221,23 +235,22 @@ MODE=mock N_SAMPLES=10 bash scripts/<experiment>.sh # smoke test
|
||||
|
||||
### 4.7 核心算法保真
|
||||
|
||||
迁移时逐一比对参考代码,不可简化。完整清单见 `research-wiki/ARCHITECTURE.md §6`(建树 4 项 + 训练 9 项 = 13 项)。
|
||||
迁移时逐一比对参考代码,不可简化。完整清单见 `research-wiki/ARCHITECTURE.md §6`(建树 4 项 + 训练 8 项 = 12 项)。
|
||||
|
||||
| # | 算法 | 核心逻辑 |
|
||||
|---|------|---------|
|
||||
| 1 | L2 轴心建树策略 | L2 先行→L3 向下→L1 向上,asyncio 链式并发 |
|
||||
| 2 | VLM 批量帧描述 + JSON fallback | `_L3_BATCH_SIZE=5` 批量调用,解析失败逐帧 fallback |
|
||||
| 3 | 断点续跑机制 | `progress.json` + L1 中间 JSON,按段恢复 |
|
||||
| 4 | RecursiveRetriever | Cross-Attention 选择器 + ACT halt + z 状态累积 |
|
||||
| 5 | CE-Gate e-process | 截断 Beta 混合、四出口门控 |
|
||||
| 6 | 信息阶梯 | 冷启动 2:1、gamma-EMA、反泄漏 |
|
||||
| 7 | 块顺序验证 | 基线缓存、INFRA 护栏、配对翻转 |
|
||||
| 8 | 诊断瀑布 | 错误归因级联、缺陷 vs 失误、D1-D5 |
|
||||
| 9 | 进化 patch 引擎 | 保护跨度、rank-and-clip、附录/动量 |
|
||||
| 10 | Mini-batch 构建 | FFD + round-robin + 正确率混合 |
|
||||
| 11 | Agent Loop | Thinking+JSON、json_repair、pluggy hook |
|
||||
| 12 | 树环境语义搜索 | 分块 embedding、祖先去重、锚定验证 |
|
||||
| 13 | 训练循环编排 | 三级嵌套、慢更新10步、断点续训 |
|
||||
| 4 | CE-Gate e-process | 截断 Beta 混合、四出口门控 |
|
||||
| 5 | 信息阶梯 | 冷启动 2:1、gamma-EMA、反泄漏 |
|
||||
| 6 | 块顺序验证 | 基线缓存、INFRA 护栏、配对翻转 |
|
||||
| 7 | 诊断瀑布 | 错误归因级联、缺陷 vs 失误、D1-D5 |
|
||||
| 8 | 进化 patch 引擎 | 保护跨度、rank-and-clip、附录/动量 |
|
||||
| 9 | Mini-batch 构建 | FFD + round-robin + 正确率混合 |
|
||||
| 10 | Agent Loop | Thinking+JSON、json_repair、pluggy hook |
|
||||
| 11 | 树环境语义搜索 | 分块 embedding、祖先去重、锚定验证 |
|
||||
| 12 | 训练循环编排 | 三级嵌套、慢更新10步、断点续训 |
|
||||
|
||||
> **任何 PR 涉及上述算法的修改,必须在 commit message 中标注对应序号并说明变更理由。**
|
||||
|
||||
@@ -301,7 +314,6 @@ project_root/
|
||||
│ ├── harness/ # 训练 harness(runner, inference, batching)
|
||||
│ ├── question_gen/ # 新题构建
|
||||
│ ├── search/ # 搜索 Agent 装配(prompt, skills)
|
||||
│ ├── retriever/ # 可训练检索器(RecursiveRetriever)
|
||||
│ └── ports.py # 应用层端口
|
||||
│
|
||||
├── adapters/ # 外部实现层(LLM/VLM/embedding/cache/遥测)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Video-Tree-TRM5
|
||||
|
||||
> 在层次化视频树上构建可自我进化的搜索 Agent 与可训练递归检索器,实现长视频理解。目标会议:EMNLP 2026。
|
||||
> 在层次化视频树上构建可自我进化的搜索 Agent,实现长视频理解。目标会议:EMNLP 2026。
|
||||
|
||||
## 系统概览
|
||||
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
"""SqliteDiagnosisSignalStore:baseline 逐题诊断信号的 SQLite 持久化适配器。
|
||||
|
||||
实现 core/evolution/protocols.py::DiagnosisSignalStore 端口。信号行以
|
||||
(question_id, baseline_run_id, diag_fingerprint) 为主键,INSERT OR REPLACE
|
||||
保证逐题幂等 upsert;bool 字段以 0/1 存储,可空字段以 NULL 存储。表建在
|
||||
harness.db,供离线诊断编排写入、视频级切分选择器读取。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from core.evolution.types import DiagnosisSignalRow
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from types import TracebackType
|
||||
|
||||
# 表列顺序即 DiagnosisSignalRow 字段顺序(question_id..session_id),
|
||||
# upsert 写入与 load 还原共用,避免手写列名两处漂移。
|
||||
_COLUMNS: tuple[str, ...] = (
|
||||
"question_id",
|
||||
"video_id",
|
||||
"baseline_run_id",
|
||||
"diag_fingerprint",
|
||||
"task_type",
|
||||
"error_type",
|
||||
"cause_category",
|
||||
"tier",
|
||||
"evolution_target",
|
||||
"degraded",
|
||||
"infra",
|
||||
"session_id",
|
||||
)
|
||||
|
||||
|
||||
class SqliteDiagnosisSignalStore:
|
||||
"""逐题诊断信号的 SQLite 存储实现。
|
||||
|
||||
构造时按需建表(CREATE TABLE IF NOT EXISTS),主键
|
||||
(question_id, baseline_run_id, diag_fingerprint) 保证同键覆盖。
|
||||
每次写操作单事务 commit,保证原子落盘与断点续跑。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 数据库文件路径(通常为 harness.db)。
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: str) -> None:
|
||||
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
self._conn = sqlite3.connect(db_path)
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
self._init_table()
|
||||
|
||||
def _init_table(self) -> None:
|
||||
"""创建 baseline_diagnosis 表(若不存在)。"""
|
||||
self._conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS baseline_diagnosis (
|
||||
question_id TEXT NOT NULL,
|
||||
video_id TEXT NOT NULL,
|
||||
baseline_run_id TEXT NOT NULL,
|
||||
diag_fingerprint TEXT NOT NULL,
|
||||
task_type TEXT NOT NULL,
|
||||
error_type TEXT,
|
||||
cause_category TEXT,
|
||||
tier TEXT NOT NULL,
|
||||
evolution_target TEXT,
|
||||
degraded INTEGER NOT NULL,
|
||||
infra INTEGER NOT NULL,
|
||||
session_id TEXT,
|
||||
PRIMARY KEY (question_id, baseline_run_id, diag_fingerprint)
|
||||
)
|
||||
"""
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def upsert(self, row: DiagnosisSignalRow) -> None:
|
||||
"""写入或覆盖单题诊断信号(按主键幂等,单事务)。
|
||||
|
||||
参数:
|
||||
row: 待持久化的诊断信号行。bool 字段转 0/1,None 存 NULL。
|
||||
|
||||
关键实现:
|
||||
用 INSERT OR REPLACE 按主键覆盖,避免重复行;commit 保证原子。
|
||||
"""
|
||||
placeholders = ", ".join("?" for _ in _COLUMNS)
|
||||
col_names = ", ".join(_COLUMNS)
|
||||
values = (
|
||||
row.question_id,
|
||||
row.video_id,
|
||||
row.baseline_run_id,
|
||||
row.diag_fingerprint,
|
||||
row.task_type,
|
||||
row.error_type,
|
||||
row.cause_category,
|
||||
row.tier,
|
||||
row.evolution_target,
|
||||
int(row.degraded),
|
||||
int(row.infra),
|
||||
row.session_id,
|
||||
)
|
||||
self._conn.execute(
|
||||
f"INSERT OR REPLACE INTO baseline_diagnosis ({col_names}) VALUES ({placeholders})",
|
||||
values,
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def done_question_ids(
|
||||
self,
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
*,
|
||||
retry_uncertain: bool = False,
|
||||
) -> set[str]:
|
||||
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
|
||||
|
||||
参数:
|
||||
baseline_run_id: baseline run 标识。
|
||||
diag_fingerprint: 诊断口径指纹。
|
||||
retry_uncertain: True 时追加 `AND tier != 'uncertain'`,把 uncertain
|
||||
(信号不可信降级)题排除出已完成集,令其被重新诊断;默认 False。
|
||||
|
||||
返回:
|
||||
已落盘信号的 question_id 集合;无匹配时为空集,供断点续跑跳过。
|
||||
"""
|
||||
sql = (
|
||||
"SELECT DISTINCT question_id FROM baseline_diagnosis"
|
||||
" WHERE baseline_run_id = ? AND diag_fingerprint = ?"
|
||||
)
|
||||
if retry_uncertain:
|
||||
sql += " AND tier != 'uncertain'"
|
||||
cursor = self._conn.execute(sql, (baseline_run_id, diag_fingerprint))
|
||||
return {r["question_id"] for r in cursor.fetchall()}
|
||||
|
||||
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
|
||||
"""加载指定 run 与诊断指纹下的全部诊断信号行。
|
||||
|
||||
参数:
|
||||
baseline_run_id: baseline run 标识。
|
||||
diag_fingerprint: 诊断口径指纹。
|
||||
|
||||
返回:
|
||||
还原后的 DiagnosisSignalRow 列表(0/1→bool,NULL→None)。
|
||||
"""
|
||||
col_names = ", ".join(_COLUMNS)
|
||||
cursor = self._conn.execute(
|
||||
f"SELECT {col_names} FROM baseline_diagnosis"
|
||||
" WHERE baseline_run_id = ? AND diag_fingerprint = ?",
|
||||
(baseline_run_id, diag_fingerprint),
|
||||
)
|
||||
return [
|
||||
DiagnosisSignalRow(
|
||||
question_id=r["question_id"],
|
||||
video_id=r["video_id"],
|
||||
baseline_run_id=r["baseline_run_id"],
|
||||
diag_fingerprint=r["diag_fingerprint"],
|
||||
task_type=r["task_type"],
|
||||
error_type=r["error_type"],
|
||||
cause_category=r["cause_category"],
|
||||
tier=r["tier"],
|
||||
evolution_target=r["evolution_target"],
|
||||
degraded=bool(r["degraded"]),
|
||||
infra=bool(r["infra"]),
|
||||
session_id=r["session_id"],
|
||||
)
|
||||
for r in cursor.fetchall()
|
||||
]
|
||||
|
||||
def close(self) -> None:
|
||||
"""关闭底层 SQLite 连接,释放文件描述符与锁。"""
|
||||
self._conn.close()
|
||||
|
||||
def __enter__(self) -> SqliteDiagnosisSignalStore:
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> None:
|
||||
self.close()
|
||||
+18
-3
@@ -20,21 +20,32 @@ class CircuitBreaker:
|
||||
self._cooldown_s = cooldown_s
|
||||
self._fails: dict[str, int] = {}
|
||||
self._open_until: dict[str, float] = {}
|
||||
self._half_open_inflight: dict[str, bool] = {}
|
||||
|
||||
def is_open(self, source_name: str, now: float) -> bool:
|
||||
"""判断指定源是否处于开路状态。
|
||||
|
||||
冷却截止时刻之前为开路;到期返回 False(放行一个试探,即半开)。
|
||||
冷却截止时刻之前为开路;到期进入半开,**只放行一个探针**(其余仍被挡),
|
||||
避免冷却到期瞬间惊群重连再次压垮上游。"检查+标记探针"在 asyncio 单线程内
|
||||
同步执行,天然原子无竞态。
|
||||
|
||||
Args:
|
||||
source_name: 被熔断的源标识。
|
||||
now: 当前时刻(秒级时间戳),由调用方注入。
|
||||
|
||||
Returns:
|
||||
True 表示开路(拒绝请求),False 表示关闭或半开(放行)。
|
||||
True 表示开路(拒绝请求),False 表示关闭或半开放行探针。
|
||||
"""
|
||||
until = self._open_until.get(source_name)
|
||||
return until is not None and now < until
|
||||
if until is None:
|
||||
return False
|
||||
if now < until:
|
||||
return True # 冷却中,全挡
|
||||
# 冷却到期:half-open,只放行一个探针
|
||||
if self._half_open_inflight.get(source_name):
|
||||
return True # 已有探针在途,继续挡
|
||||
self._half_open_inflight[source_name] = True
|
||||
return False
|
||||
|
||||
def record_failure(self, source_name: str, now: float) -> None:
|
||||
"""记录一次失败;累计达阈值则开路至 now + cooldown。
|
||||
@@ -47,6 +58,8 @@ class CircuitBreaker:
|
||||
self._fails[source_name] = count
|
||||
if count >= self._fail_threshold:
|
||||
self._open_until[source_name] = now + self._cooldown_s
|
||||
# 探针失败清在途标记,使下一轮 cooldown 到期后可再放行探针
|
||||
self._half_open_inflight.pop(source_name, None)
|
||||
|
||||
def force_open(self, source_name: str, now: float) -> None:
|
||||
"""强制开路(用于 401/403 等不可恢复错误),一次即熔断。
|
||||
@@ -59,6 +72,7 @@ class CircuitBreaker:
|
||||
"""
|
||||
self._fails[source_name] = self._fail_threshold
|
||||
self._open_until[source_name] = now + self._cooldown_s
|
||||
self._half_open_inflight.pop(source_name, None)
|
||||
|
||||
def record_success(self, source_name: str) -> None:
|
||||
"""记录一次成功;清零失败计数与开路状态(关闭熔断器)。
|
||||
@@ -68,3 +82,4 @@ class CircuitBreaker:
|
||||
"""
|
||||
self._fails.pop(source_name, None)
|
||||
self._open_until.pop(source_name, None)
|
||||
self._half_open_inflight.pop(source_name, None)
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
"""嵌入适配器 —— local/remote 双后端实现。
|
||||
|
||||
封装文本嵌入器,支持本地 sentence-transformers 和远程 OpenAI 兼容 API 两种后端。
|
||||
提供统一的 ``embed()`` / ``embed_tensor()`` 接口,冻结不训练。
|
||||
两个类均满足 ``app.ports.EmbeddingProvider`` Protocol。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from numpy import ndarray
|
||||
from torch import Tensor
|
||||
|
||||
|
||||
class LocalEmbeddingProvider:
|
||||
"""本地 sentence-transformers 嵌入器(冻结)。
|
||||
|
||||
使用 HuggingFace sentence-transformers 加载模型进行本地推理,
|
||||
所有参数冻结,仅用于嵌入提取。
|
||||
|
||||
属性:
|
||||
dim: 嵌入维度 D。
|
||||
"""
|
||||
|
||||
def __init__(self, model_name: str, embed_dim: int, device: str = "cpu") -> None:
|
||||
"""初始化本地嵌入模型。
|
||||
|
||||
参数:
|
||||
model_name: HuggingFace 模型名称(如 'BAAI/bge-base-zh-v1.5')。
|
||||
embed_dim: 期望的嵌入维度。
|
||||
device: 推理设备('cpu' / 'cuda' 等)。
|
||||
|
||||
异常:
|
||||
AssertionError: 模型实际维度与 embed_dim 不一致。
|
||||
"""
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
self._dim = embed_dim
|
||||
|
||||
self._model = SentenceTransformer(model_name, device=device)
|
||||
self._model.eval()
|
||||
# 冻结所有参数
|
||||
for param in self._model.parameters():
|
||||
param.requires_grad = False
|
||||
|
||||
actual_dim = self._model.get_sentence_embedding_dimension()
|
||||
assert actual_dim == self._dim, (
|
||||
f"模型实际维度 ({actual_dim}) 与配置 embed_dim ({self._dim}) 不一致"
|
||||
)
|
||||
|
||||
logger.info("本地嵌入模型初始化完成", model=model_name, device=device)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 公共接口
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def dim(self) -> int:
|
||||
"""嵌入维度 D。"""
|
||||
return self._dim
|
||||
|
||||
def embed(self, texts: str | list[str]) -> ndarray:
|
||||
"""文本 → 嵌入向量(L2 归一化)。
|
||||
|
||||
参数:
|
||||
texts: 单条文本或文本列表。
|
||||
|
||||
返回:
|
||||
[N, D] ndarray,每行 L2 范数为 1.0。单条文本时 N=1。
|
||||
"""
|
||||
if isinstance(texts, str):
|
||||
texts = [texts]
|
||||
|
||||
with torch.no_grad():
|
||||
embeddings = self._model.encode(
|
||||
texts,
|
||||
normalize_embeddings=True,
|
||||
convert_to_numpy=True,
|
||||
)
|
||||
# sentence-transformers encode 返回 ndarray [N, D]
|
||||
if embeddings.ndim == 1:
|
||||
embeddings = embeddings.reshape(1, -1)
|
||||
return embeddings
|
||||
|
||||
def embed_tensor(self, texts: str | list[str]) -> Tensor:
|
||||
"""文本 → 嵌入 Tensor(L2 归一化)。
|
||||
|
||||
参数:
|
||||
texts: 单条文本或文本列表。
|
||||
|
||||
返回:
|
||||
[N, D] torch.Tensor(float32)。
|
||||
"""
|
||||
arr = self.embed(texts)
|
||||
return torch.from_numpy(arr).float()
|
||||
|
||||
|
||||
class RemoteEmbeddingProvider:
|
||||
"""远程 OpenAI 兼容 API 嵌入器。
|
||||
|
||||
通过 OpenAI 兼容 API(如 GPUStack)调用远程嵌入模型。
|
||||
|
||||
属性:
|
||||
dim: 嵌入维度 D。
|
||||
"""
|
||||
|
||||
def __init__(self, model_name: str, embed_dim: int, api_key: str, api_url: str) -> None:
|
||||
"""初始化远程嵌入客户端。
|
||||
|
||||
参数:
|
||||
model_name: 远程模型名称。
|
||||
embed_dim: 期望的嵌入维度。
|
||||
api_key: API 密钥。
|
||||
api_url: API 基础 URL。
|
||||
|
||||
异常:
|
||||
ValueError: api_key 或 api_url 为空。
|
||||
"""
|
||||
if not api_key:
|
||||
raise ValueError("远程模式必须提供 api_key")
|
||||
if not api_url:
|
||||
raise ValueError("远程模式必须提供 api_url")
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
self._dim = embed_dim
|
||||
self._model_name = model_name
|
||||
self._client = OpenAI(base_url=api_url, api_key=api_key)
|
||||
|
||||
logger.info("远程嵌入客户端初始化完成", model=model_name, api_url=api_url)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 公共接口
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def dim(self) -> int:
|
||||
"""嵌入维度 D。"""
|
||||
return self._dim
|
||||
|
||||
def embed(self, texts: str | list[str]) -> ndarray:
|
||||
"""文本 → 嵌入向量(L2 归一化)。
|
||||
|
||||
参数:
|
||||
texts: 单条文本或文本列表。
|
||||
|
||||
返回:
|
||||
[N, D] ndarray,每行 L2 范数为 1.0。单条文本时 N=1。
|
||||
"""
|
||||
if isinstance(texts, str):
|
||||
texts = [texts]
|
||||
|
||||
response = self._client.embeddings.create(
|
||||
model=self._model_name,
|
||||
input=texts,
|
||||
)
|
||||
# 按 index 排序,确保顺序一致
|
||||
sorted_data = sorted(response.data, key=lambda x: x.index)
|
||||
embeddings = np.array([item.embedding for item in sorted_data], dtype=np.float32)
|
||||
|
||||
# L2 归一化
|
||||
norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
|
||||
norms = np.maximum(norms, 1e-12) # 避免除零
|
||||
embeddings = embeddings / norms
|
||||
|
||||
return embeddings
|
||||
|
||||
def embed_tensor(self, texts: str | list[str]) -> Tensor:
|
||||
"""文本 → 嵌入 Tensor(L2 归一化)。
|
||||
|
||||
参数:
|
||||
texts: 单条文本或文本列表。
|
||||
|
||||
返回:
|
||||
[N, D] torch.Tensor(float32)。
|
||||
"""
|
||||
arr = self.embed(texts)
|
||||
return torch.from_numpy(arr).float()
|
||||
+28
-13
@@ -179,7 +179,10 @@ def _is_transient_error(exc: Exception) -> bool:
|
||||
返回:
|
||||
True 表示可重试,False 表示不可重试。
|
||||
"""
|
||||
if isinstance(exc, (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout)):
|
||||
# 两族基类覆盖断连族:TimeoutException(ConnectTimeout/ReadTimeout/WriteTimeout/PoolTimeout)
|
||||
# 与 TransportError(ConnectError/ReadError/RemoteProtocolError 等)。
|
||||
# 注意 HTTPStatusError 非 TransportError 子类,401/403 致命分支不受影响。
|
||||
if isinstance(exc, (httpx.TimeoutException, httpx.TransportError)):
|
||||
return True
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
return exc.response.status_code in _TRANSIENT_STATUS_CODES
|
||||
@@ -274,6 +277,7 @@ class GovernedLLMClient:
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse:
|
||||
"""发起 LLM 调用,经四层治理栈:熔断 → 缓存 → 重试+流式 → 遥测。
|
||||
|
||||
@@ -281,6 +285,7 @@ class GovernedLLMClient:
|
||||
messages: OpenAI 格式消息列表。
|
||||
session_id: 会话 ID(传递到遥测)。
|
||||
parent_call_id: 父调用 ID(传递到遥测)。
|
||||
cache_salt: 可选缓存盐,透传到 Redis 缓存键(如跨 epoch 重采样)。
|
||||
|
||||
返回:
|
||||
LLMResponse 统一响应。
|
||||
@@ -295,12 +300,14 @@ class GovernedLLMClient:
|
||||
if self._breaker.is_open(self._provider, time.monotonic()):
|
||||
raise CircuitOpenError(f"熔断器已开启,拒绝调用 provider={self._provider}")
|
||||
|
||||
# ② call_id 生成
|
||||
call_id = str(uuid4())
|
||||
|
||||
# ③ 缓存查询(cache 为 None 时跳过)
|
||||
cached = await self._cache.get(self._model, messages) if self._cache is not None else None
|
||||
# ② 缓存查询(cache 为 None 时跳过)— call_id 在缓存路径独立生成
|
||||
cached = (
|
||||
await self._cache.get(self._model, messages, cache_salt)
|
||||
if self._cache is not None
|
||||
else None
|
||||
)
|
||||
if cached is not None:
|
||||
cache_call_id = str(uuid4())
|
||||
response = LLMResponse(
|
||||
content=cached.content,
|
||||
thinking=cached.thinking,
|
||||
@@ -312,10 +319,10 @@ class GovernedLLMClient:
|
||||
ttft_ms=None,
|
||||
max_inter_token_ms=None,
|
||||
cache_hit=True,
|
||||
call_id=call_id,
|
||||
call_id=cache_call_id,
|
||||
)
|
||||
await self._telemetry.record_llm_call(
|
||||
call_id=call_id,
|
||||
call_id=cache_call_id,
|
||||
parent_call_id=parent_call_id,
|
||||
session_id=session_id,
|
||||
model_name=self._model,
|
||||
@@ -333,9 +340,10 @@ class GovernedLLMClient:
|
||||
)
|
||||
return response
|
||||
|
||||
# ④ 重试循环 + 流式消费
|
||||
# ③ 重试循环 + 流式消费(每次 attempt 独立 call_id)
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(self._max_retries):
|
||||
call_id = str(uuid4())
|
||||
attempt_start = time.monotonic()
|
||||
try:
|
||||
content, thinking_text, ttft_ms, max_itoken_ms, usage = await self._call_streaming(
|
||||
@@ -371,7 +379,7 @@ class GovernedLLMClient:
|
||||
|
||||
# ④ 写缓存(cache 为 None 时跳过)
|
||||
if self._cache is not None:
|
||||
await self._cache.set(self._model, messages, response)
|
||||
await self._cache.set(self._model, messages, response, cache_salt)
|
||||
|
||||
# ⑤ 遥测
|
||||
await self._telemetry.record_llm_call(
|
||||
@@ -450,12 +458,15 @@ class GovernedLLMClient:
|
||||
},
|
||||
)
|
||||
if attempt < self._max_retries - 1:
|
||||
delay = min(
|
||||
import random
|
||||
|
||||
base = min(
|
||||
self._retry_base_delay_s * (2**attempt),
|
||||
self._retry_max_delay_s,
|
||||
)
|
||||
if delay > 0:
|
||||
await asyncio.sleep(delay)
|
||||
jitter = base * random.uniform(0.5, 1.5)
|
||||
if jitter > 0:
|
||||
await asyncio.sleep(jitter)
|
||||
continue
|
||||
|
||||
# 非瞬时、非致命 → 记遥测后直接抛出
|
||||
@@ -570,6 +581,10 @@ class GovernedLLMClient:
|
||||
else:
|
||||
thinking_parts.append(text)
|
||||
|
||||
# 流耗尽但未收 [DONE] → 服务端截断,视为可重试的 SSE 异常(不写缓存/不当成功)
|
||||
if not usage_sink.get("done"):
|
||||
raise _SseAnomaly("truncated_no_done")
|
||||
|
||||
content = "".join(content_parts)
|
||||
thinking = "".join(thinking_parts)
|
||||
usage = usage_sink.get("usage", {})
|
||||
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
"""MonkeyOCR HTTP 客户端 — 帧文字转录的异构硬证据源。
|
||||
|
||||
服务由用户在 LAN 部署(双端点轮询);请求必须绕过代理(trust_env=False)。
|
||||
实现 OCRProvider Protocol(app/ports.py)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import itertools
|
||||
import threading
|
||||
from pathlib import Path # noqa: TC003 — 运行时需要(方法签名 + open())
|
||||
|
||||
import requests
|
||||
from loguru import logger
|
||||
|
||||
_TIMEOUT_S = 300
|
||||
|
||||
|
||||
class MonkeyOCRClient:
|
||||
"""MonkeyOCR 服务客户端:多端点轮询、单帧失败降级为跳过。
|
||||
|
||||
关键实现细节:实例可被多线程共享——端点轮询加锁、Session 线程局部
|
||||
(A/B 评测会以 4 线程并发调用同一实例)。
|
||||
|
||||
参数:
|
||||
urls: 服务端点列表(如 ["http://10.77.0.20:7866", ...]),非空。
|
||||
|
||||
异常:
|
||||
ValueError: urls 为空时抛出。
|
||||
"""
|
||||
|
||||
def __init__(self, urls: list[str]) -> None:
|
||||
if not urls:
|
||||
raise ValueError("MonkeyOCR 端点列表不能为空")
|
||||
self._urls = [u.rstrip("/") for u in urls]
|
||||
self._rr = itertools.cycle(self._urls)
|
||||
self._rr_lock = threading.Lock()
|
||||
self._local = threading.local()
|
||||
|
||||
def _get_session(self) -> requests.Session:
|
||||
"""返回当前线程专属的 Session(惰性创建并复用,trust_env=False 绕代理)。"""
|
||||
session = getattr(self._local, "session", None)
|
||||
if session is None:
|
||||
session = requests.Session()
|
||||
session.trust_env = False # LAN 直连,绕过代理
|
||||
self._local.session = session
|
||||
return session
|
||||
|
||||
def _check_health_sync(self) -> None:
|
||||
"""同步预检所有端点,任一不可达即抛错(供 asyncio.to_thread 调用)。
|
||||
|
||||
异常:
|
||||
RuntimeError: 端点不可达或 /health 非 2xx。
|
||||
"""
|
||||
for url in self._urls:
|
||||
try:
|
||||
resp = self._get_session().get(f"{url}/health", timeout=5)
|
||||
except requests.RequestException as e:
|
||||
raise RuntimeError(f"MonkeyOCR 端点不可达: {url}: {e}") from e
|
||||
if not resp.ok:
|
||||
raise RuntimeError(f"MonkeyOCR 健康检查失败: {url}: {resp.status_code}")
|
||||
|
||||
async def check_health(self) -> None:
|
||||
"""异步预检所有端点,任一不可达即抛错(A/B qtr_ocr 臂启动门)。
|
||||
|
||||
异常:
|
||||
RuntimeError: 端点不可达或 /health 非 2xx。
|
||||
"""
|
||||
await asyncio.to_thread(self._check_health_sync)
|
||||
|
||||
def _transcribe_frames_sync(self, frame_paths: list[Path]) -> str:
|
||||
"""同步逐帧转录并拼接(供 asyncio.to_thread 调用)。
|
||||
|
||||
参数:
|
||||
frame_paths: 帧文件路径列表。
|
||||
|
||||
返回:
|
||||
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本;无任何有效结果时空串。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for i, path in enumerate(frame_paths, 1):
|
||||
lines = self._transcribe_one(path)
|
||||
if lines:
|
||||
parts.append(f"帧{i}: " + " | ".join(lines))
|
||||
return "\n".join(parts)
|
||||
|
||||
async def transcribe_frames(self, frame_paths: list[Path]) -> str:
|
||||
"""异步逐帧转录并拼接为注入文本;单帧失败跳过,全失败返回空串。
|
||||
|
||||
参数:
|
||||
frame_paths: 帧文件路径列表。
|
||||
|
||||
返回:
|
||||
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本;无任何有效结果时空串。
|
||||
"""
|
||||
return await asyncio.to_thread(self._transcribe_frames_sync, frame_paths)
|
||||
|
||||
def _transcribe_one(self, path: Path) -> list[str]:
|
||||
"""单帧转录:空结果/单字符行过滤 + 帧内行级去重。
|
||||
|
||||
参数:
|
||||
path: 帧文件路径。
|
||||
|
||||
返回:
|
||||
过滤去重后的文本行列表;请求失败或无有效行时空列表。
|
||||
"""
|
||||
with self._rr_lock:
|
||||
url = next(self._rr)
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
resp = self._get_session().post(
|
||||
f"{url}/ocr/text", files={"file": f}, timeout=_TIMEOUT_S
|
||||
)
|
||||
resp.raise_for_status()
|
||||
content = resp.json().get("content", "")
|
||||
except (requests.RequestException, ValueError) as e:
|
||||
logger.warning("MonkeyOCR 单帧转录失败,跳过 {}: {}", path.name, e)
|
||||
return []
|
||||
seen: set[str] = set()
|
||||
lines: list[str] = []
|
||||
for ln in content.splitlines():
|
||||
ln = ln.strip()
|
||||
if len(ln) <= 1 or ln in seen:
|
||||
continue
|
||||
seen.add(ln)
|
||||
lines.append(ln)
|
||||
return lines
|
||||
+47
-14
@@ -12,6 +12,26 @@ from loguru import logger
|
||||
from core.types import LLMResponse
|
||||
|
||||
|
||||
def _resolve_cache_ttl(ttl: int) -> int:
|
||||
"""校验 Redis 缓存 TTL:必须为正整数(消灭 0=永不过期 的隐式语义)。
|
||||
|
||||
Args:
|
||||
ttl: 待校验的 TTL 秒数。
|
||||
|
||||
Returns:
|
||||
校验通过的正整数 TTL。
|
||||
|
||||
Raises:
|
||||
ValueError: ttl <= 0。
|
||||
"""
|
||||
if ttl <= 0:
|
||||
raise ValueError(
|
||||
f"REDIS_CACHE_TTL 必须为正整数秒,实际 {ttl}。"
|
||||
"训练场景建议 >= 单次训练时长(如 86400)。"
|
||||
)
|
||||
return ttl
|
||||
|
||||
|
||||
class RedisResponseCache:
|
||||
"""基于 Redis 的 LLM 响应缓存。
|
||||
|
||||
@@ -22,45 +42,55 @@ class RedisResponseCache:
|
||||
|
||||
Args:
|
||||
redis: 异步 Redis 客户端实例(duck-typed,需支持 get/set 方法)。
|
||||
ttl_s: 缓存过期时间(秒)。
|
||||
ttl_s: 缓存过期时间(秒)。None 表示永不过期。
|
||||
"""
|
||||
|
||||
def __init__(self, redis: Any, ttl_s: int) -> None:
|
||||
def __init__(self, redis: Any, ttl_s: int | None) -> None:
|
||||
self._redis = redis
|
||||
self._ttl_s = ttl_s
|
||||
|
||||
def _build_key(self, model: str, messages: list[dict[str, str]]) -> str:
|
||||
def _build_key(
|
||||
self,
|
||||
model: str,
|
||||
messages: list[dict[str, str]],
|
||||
cache_salt: str | None = None,
|
||||
) -> str:
|
||||
"""构造 content-addressed 缓存键。
|
||||
|
||||
Args:
|
||||
model: 模型名称。
|
||||
messages: 消息列表。
|
||||
cache_salt: 可选缓存盐(如跨 epoch 强制重采样)。仅当非 None 时才加入
|
||||
键 payload,保证默认 None 时键结构与旧缓存一字节不差、旧键不失效。
|
||||
|
||||
Returns:
|
||||
sha256 哈希字符串作为 Redis 键。
|
||||
"""
|
||||
payload = json.dumps(
|
||||
{"model": model, "messages": messages},
|
||||
sort_keys=True,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
key_obj: dict[str, Any] = {"model": model, "messages": messages}
|
||||
if cache_salt is not None:
|
||||
key_obj["salt"] = cache_salt
|
||||
payload = json.dumps(key_obj, sort_keys=True, ensure_ascii=False)
|
||||
digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
return f"llm_cache:{digest}"
|
||||
|
||||
async def get(
|
||||
self, model: str, messages: list[dict[str, str]]
|
||||
self,
|
||||
model: str,
|
||||
messages: list[dict[str, str]],
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse | None:
|
||||
"""从缓存读取 LLM 响应。
|
||||
|
||||
Args:
|
||||
model: 模型名称。
|
||||
messages: 消息列表。
|
||||
cache_salt: 可选缓存盐,透传到键构造。
|
||||
|
||||
Returns:
|
||||
缓存命中时返回 LLMResponse,未命中或 Redis 异常时返回 None。
|
||||
"""
|
||||
try:
|
||||
key = self._build_key(model, messages)
|
||||
key = self._build_key(model, messages, cache_salt)
|
||||
raw = await self._redis.get(key)
|
||||
except Exception:
|
||||
logger.warning("Redis 缓存读取失败,降级为未命中")
|
||||
@@ -77,6 +107,7 @@ class RedisResponseCache:
|
||||
model: str,
|
||||
messages: list[dict[str, str]],
|
||||
response: LLMResponse,
|
||||
cache_salt: str | None = None,
|
||||
) -> None:
|
||||
"""将 LLM 响应写入缓存。
|
||||
|
||||
@@ -84,12 +115,14 @@ class RedisResponseCache:
|
||||
model: 模型名称。
|
||||
messages: 消息列表。
|
||||
response: 待缓存的 LLMResponse。
|
||||
cache_salt: 可选缓存盐,透传到键构造。
|
||||
"""
|
||||
try:
|
||||
key = self._build_key(model, messages)
|
||||
value = json.dumps(
|
||||
dataclasses.asdict(response), ensure_ascii=False
|
||||
)
|
||||
key = self._build_key(model, messages, cache_salt)
|
||||
value = json.dumps(dataclasses.asdict(response), ensure_ascii=False)
|
||||
if self._ttl_s:
|
||||
await self._redis.set(key, value, ex=self._ttl_s)
|
||||
else:
|
||||
await self._redis.set(key, value)
|
||||
except Exception:
|
||||
logger.warning("Redis 缓存写入失败,跳过缓存")
|
||||
|
||||
+60
-17
@@ -1,14 +1,22 @@
|
||||
"""SQLite 遥测记录器 — TelemetryRecorder Protocol 的生产实现。
|
||||
|
||||
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,
|
||||
确保事件循环不被阻塞。表在首次写入时懒初始化。
|
||||
通过 asyncio.to_thread 将 SQLite 同步写入桥接到异步接口,确保事件循环不被阻塞。
|
||||
构造时建单持久连接 + 建表(对齐 app/harness/log.py:HarnessLog 的并发写模式),
|
||||
写入经进程内 threading.Lock 串行化,消除多连接并发写的 database is locked。
|
||||
|
||||
零丢失保证范围 = 单进程、单 recorder 实例(当前 main.py / video_split_cli 均单实例
|
||||
注入)。同进程多个 recorder 指向同一 db 会退回跨连接竞争——本实现不支持该场景。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class SQLiteTelemetryRecorder:
|
||||
"""基于 SQLite 的 LLM 调用遥测记录器。
|
||||
@@ -41,7 +49,7 @@ class SQLiteTelemetryRecorder:
|
||||
"""
|
||||
|
||||
_INSERT_SQL = """
|
||||
INSERT INTO llm_calls (
|
||||
INSERT OR IGNORE INTO llm_calls (
|
||||
call_id, parent_call_id, session_id,
|
||||
model_name, provider, messages, response, thinking,
|
||||
prompt_tokens, completion_tokens, latency_ms,
|
||||
@@ -51,16 +59,43 @@ class SQLiteTelemetryRecorder:
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path) -> None:
|
||||
self._db_path = db_path
|
||||
self._table_ready = False
|
||||
"""建单持久连接 + 进程内 Lock(对齐 app/harness/log.py:HarnessLog 并发写模式)。
|
||||
|
||||
def _ensure_table(self, conn: sqlite3.Connection) -> None:
|
||||
"""懒初始化:首次写入时创建 llm_calls 表。"""
|
||||
if self._table_ready:
|
||||
return
|
||||
把并发控制拉到进程内(threading.Lock 串行化写),消除"每次新连接并发写同一
|
||||
db、靠 SQLite busy_timeout 跨连接协调"在高频下撑爆 timeout → database is locked
|
||||
的根因。check_same_thread=False:record_llm_call 经 asyncio.to_thread 在线程池
|
||||
不同线程调用,共享连接跨线程访问需此 flag,串行性由 self._lock 保证。
|
||||
|
||||
遥测哲学(P5):连接初始化失败降级不冒泡(self._conn=None,写入直接丢弃 warning),
|
||||
绝不因遥测故障拖垮 LLM 调用 / 训练。
|
||||
"""
|
||||
self._db_path = db_path
|
||||
self._lock = threading.Lock()
|
||||
self._conn: sqlite3.Connection | None = None
|
||||
# mkdir / connect / PRAGMA / 建表统一纳入降级边界:任一失败(OSError 含
|
||||
# PermissionError、sqlite3.Error)都降级为 self._conn=None,绝不冒泡拖垮初始化。
|
||||
try:
|
||||
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(str(db_path), check_same_thread=False)
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA busy_timeout=5000")
|
||||
conn.execute(self._CREATE_TABLE_SQL)
|
||||
conn.commit()
|
||||
self._table_ready = True
|
||||
self._conn = conn
|
||||
except (OSError, sqlite3.Error) as exc:
|
||||
logger.warning("遥测连接初始化失败(已降级,后续写入丢弃): {}", exc)
|
||||
self._conn = None
|
||||
|
||||
def close(self) -> None:
|
||||
"""幂等关闭持久连接(对齐 HarnessLog;进程退出前可选调以释放 fd)。
|
||||
|
||||
不调也不丢数据——每次 _write 已 commit 落 WAL,进程退出 OS 回收 fd、
|
||||
WAL 已提交内容下次打开自动 checkpoint 恢复。
|
||||
"""
|
||||
with self._lock:
|
||||
if self._conn is not None:
|
||||
self._conn.close()
|
||||
self._conn = None
|
||||
|
||||
def _write(
|
||||
self,
|
||||
@@ -81,11 +116,19 @@ class SQLiteTelemetryRecorder:
|
||||
cache_hit: bool,
|
||||
error: str | None,
|
||||
) -> None:
|
||||
"""同步写入一条 LLM 调用记录到 SQLite。"""
|
||||
conn = sqlite3.connect(str(self._db_path))
|
||||
"""同步写入一条 LLM 调用记录(单持久连接 + Lock 串行化,对齐 HarnessLog)。
|
||||
|
||||
三层防御:
|
||||
1. INSERT OR IGNORE — call_id 主键冲突静默忽略(幂等)
|
||||
2. 进程内 threading.Lock 串行化写 — 消除并发锁竞争(非依赖 SQLite busy_timeout)
|
||||
3. try/except sqlite3.Error — DB 错误降级不冒泡,遥测失败绝不拖垮 LLM 调用
|
||||
"""
|
||||
if self._conn is None:
|
||||
logger.warning("遥测连接不可用(已降级),丢弃 call_id={}", call_id)
|
||||
return
|
||||
try:
|
||||
self._ensure_table(conn)
|
||||
conn.execute(
|
||||
with self._lock:
|
||||
self._conn.execute(
|
||||
self._INSERT_SQL,
|
||||
(
|
||||
call_id,
|
||||
@@ -105,9 +148,9 @@ class SQLiteTelemetryRecorder:
|
||||
error,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
self._conn.commit()
|
||||
except sqlite3.Error as exc:
|
||||
logger.warning("遥测写入失败(已降级),call_id={}: {}", call_id, exc)
|
||||
|
||||
async def record_llm_call(
|
||||
self,
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
"""GovernedVLMClient -- VLMProvider 最小可用实现。
|
||||
|
||||
将图片编码为 base64,构造 OpenAI Vision API 格式的 messages,
|
||||
委托给已有的 GovernedLLMClient 发送。复用 LLM 治理栈的全部能力
|
||||
(熔断、缓存、重试、遥测)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import mimetypes
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from adapters.llm import GovernedLLMClient
|
||||
from core.types import LLMResponse
|
||||
|
||||
|
||||
class GovernedVLMClient:
|
||||
"""VLMProvider 实现——包装 GovernedLLMClient,注入 base64 图片。
|
||||
|
||||
参数:
|
||||
governed_llm: 已初始化的 GovernedLLMClient 实例。
|
||||
"""
|
||||
|
||||
def __init__(self, governed_llm: GovernedLLMClient) -> None:
|
||||
self._llm = governed_llm
|
||||
|
||||
async def chat_with_images(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
images: list[str | Path],
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse:
|
||||
"""图文调用:将图片编码为 base64 嵌入 messages,委托给 LLM 客户端。
|
||||
|
||||
参数:
|
||||
messages: 对话消息列表。最后一条 user message 的 content 会被扩展为
|
||||
包含图片的多模态格式。
|
||||
images: 图片文件路径列表。
|
||||
session_id: 会话 ID(遥测用)。
|
||||
parent_call_id: 父调用 ID(遥测用)。
|
||||
cache_salt: 可选缓存盐,透传到底层 LLM 缓存键。
|
||||
|
||||
返回:
|
||||
LLMResponse。
|
||||
"""
|
||||
vision_messages = self._inject_images(messages, images)
|
||||
return await self._llm.chat(
|
||||
vision_messages,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
cache_salt=cache_salt,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _encode_image(image_path: str | Path) -> str:
|
||||
"""将图片文件编码为 base64 data URL。
|
||||
|
||||
参数:
|
||||
image_path: 图片文件路径。
|
||||
|
||||
返回:
|
||||
data:image/<type>;base64,<data> 格式的字符串。
|
||||
"""
|
||||
path = Path(image_path)
|
||||
mime_type = mimetypes.guess_type(str(path))[0] or "image/jpeg"
|
||||
with open(path, "rb") as f:
|
||||
b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
return f"data:{mime_type};base64,{b64}"
|
||||
|
||||
@staticmethod
|
||||
def _inject_images(
|
||||
messages: list[dict[str, Any]],
|
||||
images: list[str | Path],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""将图片注入最后一条 user message,构造 OpenAI Vision API 格式。
|
||||
|
||||
参数:
|
||||
messages: 原始消息列表。
|
||||
images: 图片路径列表。
|
||||
|
||||
返回:
|
||||
新消息列表(不修改原列表)。
|
||||
"""
|
||||
if not images:
|
||||
return messages
|
||||
|
||||
result = [m.copy() for m in messages]
|
||||
|
||||
# 找到最后一条 user message
|
||||
last_user_idx = -1
|
||||
for i in range(len(result) - 1, -1, -1):
|
||||
if result[i].get("role") == "user":
|
||||
last_user_idx = i
|
||||
break
|
||||
|
||||
if last_user_idx == -1:
|
||||
logger.warning("messages 中无 user 角色消息,图片未注入")
|
||||
return result
|
||||
|
||||
user_msg = result[last_user_idx]
|
||||
original_content = user_msg.get("content", "")
|
||||
|
||||
# 构造多模态 content
|
||||
content_parts: list[dict[str, Any]] = []
|
||||
|
||||
# 图片在前
|
||||
for img_path in images:
|
||||
data_url = GovernedVLMClient._encode_image(img_path)
|
||||
content_parts.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": data_url},
|
||||
}
|
||||
)
|
||||
|
||||
# 文本在后
|
||||
if isinstance(original_content, str) and original_content:
|
||||
content_parts.append({"type": "text", "text": original_content})
|
||||
elif isinstance(original_content, list):
|
||||
content_parts.extend(original_content)
|
||||
|
||||
result[last_user_idx] = {**user_msg, "content": content_parts}
|
||||
return result
|
||||
@@ -0,0 +1,36 @@
|
||||
"""app/harness/ — 训练循环编排层。
|
||||
|
||||
组合 core/evolution/(决策内核)+ core/agent/(AgentLoop)+ adapters/(LLM/VLM/telemetry),
|
||||
实现自进化闭环的训练循环三级嵌套、块序贯验证、快慢双速进化、checkpoint/resume。
|
||||
"""
|
||||
|
||||
from app.harness.config import RunConfig, load_config
|
||||
from app.harness.inference import InferenceResult, run_inference
|
||||
from app.harness.log import HarnessLog, RunLogImpl
|
||||
from app.harness.pools import Pools, build_or_load_pools, build_pools, load_pools, save_pools
|
||||
from app.harness.runner import Runner
|
||||
from app.harness.workspace import (
|
||||
ResolvedPaths,
|
||||
VersionedPromptStore,
|
||||
VersionedSkillStore,
|
||||
resolve_paths,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"HarnessLog",
|
||||
"InferenceResult",
|
||||
"Pools",
|
||||
"ResolvedPaths",
|
||||
"RunConfig",
|
||||
"RunLogImpl",
|
||||
"Runner",
|
||||
"VersionedPromptStore",
|
||||
"VersionedSkillStore",
|
||||
"build_or_load_pools",
|
||||
"build_pools",
|
||||
"load_config",
|
||||
"load_pools",
|
||||
"resolve_paths",
|
||||
"run_inference",
|
||||
"save_pools",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
"""离线诊断编排:把 baseline run 的错题诊断投影为逐题信号行并断点续跑落库。
|
||||
|
||||
"结果驱动视频级切分"离线管线的诊断步。给定一批可诊断错题:
|
||||
1. 算 remaining(跳过 store 已完成题)实现续跑幂等;
|
||||
2. 对剩余错题调 core.evolution.diagnose.run_diagnosis(经 StepsJsonRunLog
|
||||
包装内层 RunLog,兼容 traces 未落表的历史 run);
|
||||
3. 把 error_attributions / infra / degraded 三类产物确定性投影为
|
||||
DiagnosisSignalRow(tier 由 split_selection.score_signal 判定);
|
||||
4. run 末(Phase 3)逐行 store.upsert 落库——诊断在 Phase 2 全部跑完后才落库,
|
||||
故崩溃丢本次 run 未落库的全部结果(不是"仅一行");靠 GovernedLLMClient 的
|
||||
Redis 缓存缓解重跑时的 LLM 重烧,下次调用命中缓存直接续。
|
||||
|
||||
错误处理诚实标注(不谎称全传播):
|
||||
- run_diagnosis 的 C1/C2 阶段(指标计算、错误归因)网络/API 失败经
|
||||
GovernedLLMClient 重试栈后仍失败会向上抛出,本编排不捕获、不掩盖,
|
||||
直接冒泡给调用方。
|
||||
- 但 C3 阶段(defect/lapse judge)的调用整体包在 `except Exception` 内
|
||||
(core/evolution/diagnose.py:2186),故 C3 judge 的**全部异常(含网络/API
|
||||
失败)都被吞并→warning→默认归为 lapse**,不会向上抛;judge 语义歧义同样
|
||||
按此保护性 fallback 处理。本编排原样接受该判定,不二次兜底、也不谎称
|
||||
C3 阶段网络失败会传播。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.baseline_run_log import StepsJsonRunLog
|
||||
from app.harness.split_selection import evolution_target_of, score_signal
|
||||
from core.evolution.diagnose import run_diagnosis
|
||||
from core.evolution.types import DiagnosisSignalRow
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from core.evolution.protocols import DiagnosisSignalStore
|
||||
from core.evolution.types import DiagnosisResult
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiagnosisDeps:
|
||||
"""离线诊断编排的依赖束(一次编排的全部外部端口 + 运行参数)。
|
||||
|
||||
frozen 保证一次编排内依赖不可变;LLM/RunLog/SkillStore/prompts 走 Protocol
|
||||
注入,便于测试替换成假实现。
|
||||
|
||||
属性:
|
||||
run_log: 内层 RunLog 实现(提供 get_predictions/get_traces),
|
||||
编排内部再用 StepsJsonRunLog 包装以兼容 traces 未落表的 run。
|
||||
llm: LLM 调用端口(治理后的 GovernedLLMClient)。
|
||||
skill_store: 技能文件读取端口。
|
||||
prompts: 诊断模板束(DiagnosePrompts)。
|
||||
tree_data: 树结构字典(多视频 {video_id: tree} 或单棵树),透传给 run_diagnosis。
|
||||
concurrency: 诊断并发上限。
|
||||
"""
|
||||
|
||||
run_log: Any
|
||||
llm: Any
|
||||
skill_store: Any
|
||||
prompts: Any
|
||||
tree_data: dict[str, Any]
|
||||
concurrency: int
|
||||
|
||||
|
||||
async def run_baseline_diagnosis(
|
||||
*,
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
wrong_ids: list[str],
|
||||
questions: dict[str, GeneratedQuestion],
|
||||
store: DiagnosisSignalStore,
|
||||
deps: DiagnosisDeps,
|
||||
retry_uncertain: bool = False,
|
||||
) -> None:
|
||||
"""对 baseline run 的错题跑离线诊断并把信号落库(断点续跑幂等)。
|
||||
|
||||
参数:
|
||||
baseline_run_id: baseline run 标识(如 "infer_adhoc"),信号行主键之一。
|
||||
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号,主键之一。
|
||||
wrong_ids: 本次待诊断的可诊断错题 question_id 列表(保序)。
|
||||
questions: question_id → GeneratedQuestion 映射,需覆盖 wrong_ids 全部题
|
||||
及 run_diagnosis 返回的所有 infra/degraded 题(用于取 video_id/task_type)。
|
||||
store: 诊断信号存储端口,upsert 落盘并提供 done_question_ids 续跑查询。
|
||||
deps: 外部依赖束(见 DiagnosisDeps)。
|
||||
retry_uncertain: True 时把已落 tier='uncertain'(信号不可信降级)的题也纳入
|
||||
remaining 重新诊断,透传给 store.done_question_ids;默认 False。
|
||||
|
||||
返回:
|
||||
None。副作用为把逐题 DiagnosisSignalRow 写入 store。
|
||||
|
||||
关键实现:
|
||||
- remaining = wrong_ids 去除 store 已完成题;空则直接 return(续跑幂等,
|
||||
重复调用零副作用)。
|
||||
- run_diagnosis 只诊断 remaining,避免重复 LLM 调用浪费。
|
||||
- 三类产物投影互斥落库:error_attributions(defect/lapse)、infra_question_ids
|
||||
(T0)、degraded_question_ids(uncertain)。
|
||||
"""
|
||||
# Phase 1: 算 remaining(续跑幂等)
|
||||
done = store.done_question_ids(
|
||||
baseline_run_id, diag_fingerprint, retry_uncertain=retry_uncertain
|
||||
)
|
||||
remaining = [qid for qid in wrong_ids if qid not in done]
|
||||
if not remaining:
|
||||
logger.info(
|
||||
"离线诊断续跑:baseline={} fingerprint={} 无剩余错题(已完成 {} 题),跳过。",
|
||||
baseline_run_id,
|
||||
diag_fingerprint,
|
||||
len(done),
|
||||
)
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"离线诊断开始:baseline={} fingerprint={} 剩余 {}/{} 题待诊断。",
|
||||
baseline_run_id,
|
||||
diag_fingerprint,
|
||||
len(remaining),
|
||||
len(wrong_ids),
|
||||
)
|
||||
|
||||
# Phase 2: 对剩余错题跑诊断(StepsJsonRunLog 兼容 traces 未落表的历史 run)
|
||||
result = await run_diagnosis(
|
||||
baseline_run_id,
|
||||
[questions[qid] for qid in remaining],
|
||||
deps.tree_data,
|
||||
deps.llm,
|
||||
StepsJsonRunLog(deps.run_log),
|
||||
deps.skill_store,
|
||||
deps.prompts,
|
||||
concurrency=deps.concurrency,
|
||||
question_ids=list(remaining),
|
||||
only_incorrect=True,
|
||||
)
|
||||
|
||||
# Phase 3: 投影落库
|
||||
counts = _project_and_persist(
|
||||
result=result,
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
questions=questions,
|
||||
store=store,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"离线诊断落库完成:baseline={} fingerprint={} "
|
||||
"T2={} T1={} T0(infra)={} uncertain(degraded)={} 共 {} 行。",
|
||||
baseline_run_id,
|
||||
diag_fingerprint,
|
||||
counts["T2"],
|
||||
counts["T1"],
|
||||
counts["T0"],
|
||||
counts["uncertain"],
|
||||
sum(counts.values()),
|
||||
)
|
||||
|
||||
|
||||
def _project_and_persist(
|
||||
*,
|
||||
result: DiagnosisResult,
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
questions: dict[str, GeneratedQuestion],
|
||||
store: DiagnosisSignalStore,
|
||||
) -> dict[str, int]:
|
||||
"""把 DiagnosisResult 三类产物投影为信号行并逐行 upsert,返回各 tier 计数。
|
||||
|
||||
参数:
|
||||
result: run_diagnosis 的返回,含 error_attributions/infra/degraded 三类产物。
|
||||
baseline_run_id: 信号行主键之一。
|
||||
diag_fingerprint: 信号行主键之一。
|
||||
questions: question_id → GeneratedQuestion,用于取 video_id/task_type。
|
||||
store: 诊断信号存储端口。
|
||||
|
||||
返回:
|
||||
{tier: 行数} 计数字典(T2/T1/T0/uncertain),供上层日志与 manifest。
|
||||
|
||||
关键实现:
|
||||
本函数在 run 末(Phase 3)逐行 upsert(单行单事务);诊断已在 Phase 2 全部
|
||||
跑完,故本阶段中途崩溃丢本次 run 未落库的余下行。三桶**非互斥**:
|
||||
同一 degraded 错题可能同时出现在 error_attributions(judge 解析失败仍建
|
||||
attribution)里,故按 **degraded > infra > attribution** 优先级去重——先落
|
||||
degraded/infra,再在 attribution 循环跳过已落题,保证**每题恰写一行、
|
||||
counts 恰计一次**(否则同 PK 覆盖会导致 counts 双计且分层错乱)。
|
||||
"""
|
||||
counts = {"T2": 0, "T1": 0, "T0": 0, "uncertain": 0}
|
||||
# 优先级去重:degraded > infra > attribution。先记录高优先集合,
|
||||
# attribution 循环遇到已落题即跳过,确保每题唯一落库。
|
||||
persisted: set[str] = set()
|
||||
|
||||
# degraded_question_ids(最高优先):judge 解析失败降级 → uncertain,信号不可信排除出 T2
|
||||
for qid in result.degraded_question_ids:
|
||||
q = questions[qid]
|
||||
store.upsert(
|
||||
DiagnosisSignalRow(
|
||||
question_id=qid,
|
||||
video_id=q.video_id,
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
task_type=q.task_type,
|
||||
error_type=None,
|
||||
cause_category=None,
|
||||
tier="uncertain",
|
||||
evolution_target=None,
|
||||
degraded=True,
|
||||
infra=False,
|
||||
session_id=None,
|
||||
)
|
||||
)
|
||||
persisted.add(qid)
|
||||
counts["uncertain"] += 1
|
||||
|
||||
# infra_question_ids:基础设施失败护栏排除 → T0,不参与训练主体
|
||||
# (防御性跳过已落 degraded 题,虽 infra 通常已在诊断前过滤不重叠)
|
||||
for qid in result.infra_question_ids:
|
||||
if qid in persisted:
|
||||
continue
|
||||
q = questions[qid]
|
||||
store.upsert(
|
||||
DiagnosisSignalRow(
|
||||
question_id=qid,
|
||||
video_id=q.video_id,
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
task_type=q.task_type,
|
||||
error_type=None,
|
||||
cause_category=None,
|
||||
tier="T0",
|
||||
evolution_target=None,
|
||||
degraded=False,
|
||||
infra=True,
|
||||
session_id=None,
|
||||
)
|
||||
)
|
||||
persisted.add(qid)
|
||||
counts["T0"] += 1
|
||||
|
||||
# error_attributions(最低优先):defect→T2 / lapse→T1 / 其它→uncertain(由 score_signal 判定)
|
||||
# 跳过已作为 degraded/infra 落库的题,避免同 PK 覆盖与 counts 双计。
|
||||
for ea in result.error_attributions:
|
||||
if ea.question_id in persisted:
|
||||
continue
|
||||
q = questions[ea.question_id]
|
||||
tier = score_signal(cause_category=ea.cause_category, infra=False, degraded=False).tier
|
||||
# error_type 是 ErrorAttribution 必填字段(永远已知),确定性派生进化目标。
|
||||
evolution_target = evolution_target_of(ea.error_type)
|
||||
store.upsert(
|
||||
DiagnosisSignalRow(
|
||||
question_id=ea.question_id,
|
||||
video_id=q.video_id,
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
task_type=q.task_type,
|
||||
error_type=ea.error_type,
|
||||
cause_category=ea.cause_category,
|
||||
tier=tier,
|
||||
evolution_target=evolution_target,
|
||||
degraded=False,
|
||||
infra=False,
|
||||
session_id=None,
|
||||
)
|
||||
)
|
||||
persisted.add(ea.question_id)
|
||||
counts[tier] = counts.get(tier, 0) + 1
|
||||
|
||||
return counts
|
||||
@@ -0,0 +1,71 @@
|
||||
"""RunLog 包装器:traces 表空时从 predictions.steps_json 重建轨迹。
|
||||
|
||||
用于对 infer_adhoc 这类 traces 未落表、轨迹在 steps_json 的历史 run 跑离线诊断。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.harness.steps_json_traces import steps_json_to_trace_rows
|
||||
|
||||
|
||||
class StepsJsonRunLog:
|
||||
"""委托内层 RunLog;get_traces 空表时回退 steps_json。
|
||||
|
||||
实现 core/evolution/protocols.py 的 RunLog Protocol(duck-typing)。
|
||||
对 traces 已落表的正常 run 完全透传;仅当底层 traces 为空时,
|
||||
才从 predictions.steps_json 经 steps_json_to_trace_rows 重建轨迹行。
|
||||
"""
|
||||
|
||||
def __init__(self, inner: Any) -> None:
|
||||
"""构造包装器。
|
||||
|
||||
参数:
|
||||
inner: 内层 RunLog 实现(如 app/harness/log.py::RunLogImpl),
|
||||
需提供 get_predictions / get_traces 两个 async 方法。
|
||||
"""
|
||||
self._inner = inner
|
||||
|
||||
async def get_predictions(
|
||||
self, run_id: str, *, question_ids: list[str] | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""透传内层预测查询。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
内层返回的预测记录字典列表,原样透传。
|
||||
"""
|
||||
return await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||
|
||||
async def get_traces(
|
||||
self, run_id: str, *, question_ids: list[str] | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""查询轨迹;底层 traces 表空时从 steps_json 回退重建。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
轨迹行字典列表。
|
||||
|
||||
关键实现细节:
|
||||
- 内层 traces 非空 → 原样返回,不触发回退(正常 run 路径)。
|
||||
- 内层 traces 为空 → 拉取同一过滤条件下的 predictions,
|
||||
逐题经 steps_json_to_trace_rows 展开为轨迹行并拼接。
|
||||
- steps_json 缺失时以空串传入,由下游确定性返回 []。
|
||||
"""
|
||||
inner_rows = await self._inner.get_traces(run_id, question_ids=question_ids)
|
||||
if inner_rows:
|
||||
return inner_rows
|
||||
preds = await self._inner.get_predictions(run_id, question_ids=question_ids)
|
||||
rows: list[dict[str, Any]] = []
|
||||
for p in preds:
|
||||
rows.extend(
|
||||
steps_json_to_trace_rows(p["video_id"], p["question_id"], p.get("steps_json") or "")
|
||||
)
|
||||
return rows
|
||||
@@ -0,0 +1,351 @@
|
||||
"""混合 mini-batch 切分:以 QuestionUnit 为最小调度粒度,大类打散、小类整锁。
|
||||
|
||||
供 runner 每 step 处理一个 batch。孪生对(AR pair)作为 2 题单元整锁不拆、按单元级
|
||||
正确性分桶;非 AR single 单元的抽样/洗牌 draw 流与"引入 QuestionUnit 前"的旧逐题算法
|
||||
逐字节一致(AR 折叠不干扰非 AR draw 流)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import math
|
||||
import random
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from app.harness.question_units import build_units, flatten_units, unit_correctness
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
|
||||
def _rng_ns(seed: int, ns: str) -> random.Random:
|
||||
"""由 (seed, 命名空间) 稳定派生独立随机数发生器。
|
||||
|
||||
用 SHA-256 派生而非 Python 内置 ``hash()``——后者受 hash randomization 影响,
|
||||
跨进程不可复现。不同命名空间的 draw 流互不干扰,使 AR 单元折叠不扰动非 AR 抽样。
|
||||
|
||||
参数:
|
||||
seed: 实验随机种子。
|
||||
ns: 命名空间标签(如 "AR")。
|
||||
返回:
|
||||
以 SHA-256(f"{ns}:{seed}") 前 8 字节为种子的 ``random.Random``。
|
||||
"""
|
||||
digest = hashlib.sha256(f"{ns}:{seed}".encode()).digest()
|
||||
return random.Random(int.from_bytes(digest[:8], "big"))
|
||||
|
||||
|
||||
def build_batches(
|
||||
items: list[GeneratedQuestion],
|
||||
correctness: dict[str, bool],
|
||||
batch_size: int,
|
||||
min_class_per_batch: int,
|
||||
seed: int,
|
||||
correct_ratio: float = 0.0,
|
||||
) -> tuple[list[list[GeneratedQuestion]], int]:
|
||||
"""把诊断池里的题目切成多个混合 mini-batch(以 QuestionUnit 为原子调度单元)。
|
||||
|
||||
single 题为 1 题单元,AR pair 孪生对为 2 题单元;同一 pair 的两题整锁进同一 batch,
|
||||
按单元级正确性(双向 AND)分桶。当 ``correct_ratio > 0`` 时,按题型为每组错误单元配比
|
||||
一定数量的正确单元("动量"机制);``correct_ratio <= 0`` 时退化为纯错误单元模式。
|
||||
|
||||
参数:
|
||||
items: 候选题目全集(可混含 single 与孪生对成员)。
|
||||
correctness: question_id -> 基线是否答对。
|
||||
batch_size: 单个 batch 的题目数上限(> 0,pair 占 2)。
|
||||
min_class_per_batch: 小类判定阈值——单元题目总数 ≤ 此值的题型整组锁进单一
|
||||
batch(> 0)。
|
||||
seed: 随机种子,保证相同输入产出完全一致的切分。
|
||||
correct_ratio: 正确题占比(0.0 ~ 1.0)。0.0 = 纯错误单元;0.5 = 错:正 = 1:1。
|
||||
返回:
|
||||
(非空 mini-batch 列表, selected_count);无错误单元时返回 ([], 0)。
|
||||
selected_count 是所有 batch 中题目(展开后)总数。
|
||||
异常:
|
||||
ValueError: batch_size 或 min_class_per_batch < 1, 或
|
||||
min_class_per_batch >= batch_size(破坏小类整组装箱不超容的前提)。
|
||||
关键实现细节:
|
||||
非 AR(single)与 AR(pair)各用独立稳定派生的 rng:非 AR 用 ``random.Random(seed)``
|
||||
(复现旧逐题算法的确切 draw 序列,保证纯非 AR 输入逐字节一致),AR 用
|
||||
``_rng_ns(seed, "AR")``;二者 draw 流互不干扰,故加入/移除 pair 不改变非 AR 的
|
||||
抽样/洗牌序列。抽样在合并前按流分别进行(``_select_mixed_by_task_type`` 各跑一次),
|
||||
大类洗牌按单元 kind 拆分后各用对应流。装箱顺序「先小类后大类」:小类整组
|
||||
first-fit-decreasing(容量按单元 ``size`` 计,pair 占 2)装入首个容得下的 batch,
|
||||
装不下新开 bin;大类洗牌后 round-robin 分发,遇碎片(size-2 单元放不进任一现存
|
||||
batch 的剩余容量)新开 bin 兜底而非报错。最终每个 batch 展开回题目列表。
|
||||
题型按名称排序处理以保证跨运行确定性。
|
||||
"""
|
||||
_validate_params(batch_size, min_class_per_batch)
|
||||
|
||||
# 非 AR 复现旧版 random.Random(seed) 的确切序列以满足黄金 byte-identity;
|
||||
# AR 走独立命名空间派生流,二者互不干扰。
|
||||
rng_nonar = random.Random(seed)
|
||||
rng_ar = _rng_ns(seed, "AR")
|
||||
|
||||
grouped = _group_units_by_task_type(items, correctness, correct_ratio, rng_nonar, rng_ar)
|
||||
|
||||
total = sum(_group_load(g) for g in grouped.values())
|
||||
if total == 0:
|
||||
return [], 0
|
||||
|
||||
nb = max(1, math.ceil(total / batch_size))
|
||||
batches: list[list[QuestionUnit]] = [[] for _ in range(nb)]
|
||||
|
||||
small, large = _split_by_size(grouped, min_class_per_batch)
|
||||
for group in _small_groups_decreasing(small):
|
||||
_pack_small_class(batches, group, batch_size)
|
||||
_distribute_large_classes(batches, large, batch_size, rng_nonar, rng_ar)
|
||||
|
||||
result = [flatten_units(b) for b in batches if b]
|
||||
selected_count = sum(len(b) for b in result)
|
||||
return result, selected_count
|
||||
|
||||
|
||||
def _validate_params(batch_size: int, min_class_per_batch: int) -> None:
|
||||
"""校验切分参数,非法值直接报错而非用默认值掩盖。
|
||||
|
||||
除各自 >= 1 外,强制 min_class_per_batch < batch_size:小类组题目总数 ≤
|
||||
min_class_per_batch,唯有此前提成立才能保证小类整组放入单一 batch 而不超容;否则
|
||||
_pack_small_class 新开的 bin 会装入超 batch_size 的整组,静默违反容量合约。此约束
|
||||
与 config._validate_minibatch 一致,是 build_batches 对自身前提的防御性自校验(P5)。
|
||||
"""
|
||||
if batch_size < 1:
|
||||
raise ValueError(f"batch_size 必须 >= 1, 实为 {batch_size}")
|
||||
if min_class_per_batch < 1:
|
||||
raise ValueError(f"min_class_per_batch 必须 >= 1, 实为 {min_class_per_batch}")
|
||||
if min_class_per_batch >= batch_size:
|
||||
raise ValueError(
|
||||
f"min_class_per_batch 必须严格 < batch_size, 否则无法保证小类整组放入单一 "
|
||||
f"batch 不超容; 实为 min_class_per_batch={min_class_per_batch}, "
|
||||
f"batch_size={batch_size}"
|
||||
)
|
||||
|
||||
|
||||
def _group_units_by_task_type(
|
||||
items: list[GeneratedQuestion],
|
||||
correctness: dict[str, bool],
|
||||
correct_ratio: float,
|
||||
rng_nonar: random.Random,
|
||||
rng_ar: random.Random,
|
||||
) -> dict[str, list[QuestionUnit]]:
|
||||
"""把题目聚合为单元并按题型分组:非 AR 与 AR 各走独立 draw 流后合并。
|
||||
|
||||
参数:
|
||||
items: 候选题目全集。
|
||||
correctness: question_id -> 基线是否答对。
|
||||
correct_ratio: 正确题占比。
|
||||
rng_nonar: 非 AR(single 单元)抽样用 rng。
|
||||
rng_ar: AR(pair 单元)抽样用 rng。
|
||||
返回:
|
||||
task_type -> 混合后的单元列表(single 单元在前、pair 单元在后)。
|
||||
"""
|
||||
units = build_units(items)
|
||||
singles = [u for u in units if u.kind == "single"]
|
||||
pairs = [u for u in units if u.kind == "pair"]
|
||||
grouped_nonar = _select_mixed_by_task_type(singles, correctness, correct_ratio, rng_nonar)
|
||||
grouped_ar = _select_mixed_by_task_type(pairs, correctness, correct_ratio, rng_ar)
|
||||
return _merge_grouped(grouped_nonar, grouped_ar)
|
||||
|
||||
|
||||
def _group_load(group: list[QuestionUnit]) -> int:
|
||||
"""一组单元展开后的题目总数(single 计 1,pair 计 2),即占用的 batch 容量。"""
|
||||
return sum(u.size for u in group)
|
||||
|
||||
|
||||
def _batch_load(batch: list[QuestionUnit]) -> int:
|
||||
"""一个 batch 内单元展开后的题目总数,用于容量判断。"""
|
||||
return sum(u.size for u in batch)
|
||||
|
||||
|
||||
def _merge_grouped(
|
||||
grouped_nonar: dict[str, list[QuestionUnit]],
|
||||
grouped_ar: dict[str, list[QuestionUnit]],
|
||||
) -> dict[str, list[QuestionUnit]]:
|
||||
"""按 task_type 合并非 AR 与 AR 两条流的分组(single 在前、pair 在后)。
|
||||
|
||||
参数:
|
||||
grouped_nonar: 非 AR(single 单元)分组。
|
||||
grouped_ar: AR(pair 单元)分组。
|
||||
返回:
|
||||
task_type -> 合并后的单元列表;每类 single 单元在前、pair 单元在后,顺序稳定。
|
||||
"""
|
||||
merged: dict[str, list[QuestionUnit]] = {}
|
||||
for task_type in sorted({*grouped_nonar, *grouped_ar}):
|
||||
merged[task_type] = grouped_nonar.get(task_type, []) + grouped_ar.get(task_type, [])
|
||||
return merged
|
||||
|
||||
|
||||
def _split_by_size(
|
||||
grouped: dict[str, list[QuestionUnit]],
|
||||
min_class_per_batch: int,
|
||||
) -> tuple[dict[str, list[QuestionUnit]], dict[str, list[QuestionUnit]]]:
|
||||
"""按题目总数(单元展开)把题型分为小类(≤ 阈值)与大类(> 阈值)两组。"""
|
||||
small = {t: g for t, g in grouped.items() if _group_load(g) <= min_class_per_batch}
|
||||
large = {t: g for t, g in grouped.items() if _group_load(g) > min_class_per_batch}
|
||||
return small, large
|
||||
|
||||
|
||||
def _classify_unit(unit: QuestionUnit, correctness: dict[str, bool]) -> str | None:
|
||||
"""判定单元落入哪个桶:error / correct / None(未知,跳过)。
|
||||
|
||||
参数:
|
||||
unit: 目标单元。
|
||||
correctness: question_id -> 是否答对(缺键视为未知)。
|
||||
返回:
|
||||
"error"(单元级正确性为 False)、"correct"(双向 AND 为 True);单元内任一题
|
||||
未知(correctness 缺该键)返回 None,与旧逐题算法把未知题排除在错/对两桶之外
|
||||
的语义一致。
|
||||
关键实现:
|
||||
先探测是否有未知题(get 返回 None ⟺ 键缺失,因 correctness 值恒为 bool),
|
||||
全部已知后交由 unit_correctness 计双向 AND(此时 KeyError 不可达)。
|
||||
"""
|
||||
if any(correctness.get(q.question_id) is None for q in unit.questions):
|
||||
return None
|
||||
return "correct" if unit_correctness(unit, correctness) else "error"
|
||||
|
||||
|
||||
def _select_mixed_by_task_type(
|
||||
units: list[QuestionUnit],
|
||||
correctness: dict[str, bool],
|
||||
correct_ratio: float,
|
||||
rng: random.Random,
|
||||
) -> dict[str, list[QuestionUnit]]:
|
||||
"""按题型分组,为每组错误单元按比例采样正确单元混入(单元粒度)。
|
||||
|
||||
只对有错误单元的题型做混合——无错误单元的题型不进 batch,即使有正确单元。
|
||||
``correct_ratio <= 0`` 时退化为纯错误单元模式。本函数只处理单一 draw 流(全 single
|
||||
或全 pair),使非 AR 与 AR 的抽样互不干扰。
|
||||
|
||||
参数:
|
||||
units: 同一流的候选单元(全 single 或全 pair)。
|
||||
correctness: question_id -> 基线是否答对。
|
||||
correct_ratio: 正确题占比(0.0 ~ 1.0)。
|
||||
rng: 本流专用随机数发生器,用于采样正确单元。
|
||||
返回:
|
||||
task_type -> 该题型的混合单元列表(错误单元全部 + 按比例采样的正确单元)。
|
||||
关键实现:
|
||||
n_correct 按错误单元「题目总数」而非单元数计,与旧逐题语义对齐(纯 single 时
|
||||
单元数 == 题目数,采样序列逐字节一致)。
|
||||
"""
|
||||
errors_by_type: dict[str, list[QuestionUnit]] = {}
|
||||
correct_by_type: dict[str, list[QuestionUnit]] = {}
|
||||
for unit in units:
|
||||
bucket = _classify_unit(unit, correctness)
|
||||
if bucket == "error":
|
||||
errors_by_type.setdefault(unit.task_type, []).append(unit)
|
||||
elif bucket == "correct":
|
||||
correct_by_type.setdefault(unit.task_type, []).append(unit)
|
||||
|
||||
if correct_ratio <= 0:
|
||||
return errors_by_type
|
||||
|
||||
grouped: dict[str, list[QuestionUnit]] = {}
|
||||
for task_type in sorted(errors_by_type):
|
||||
errs = errors_by_type[task_type]
|
||||
n_err = _group_load(errs)
|
||||
n_correct = round(n_err * correct_ratio / (1 - correct_ratio))
|
||||
available = correct_by_type.get(task_type, [])
|
||||
sampled = (
|
||||
list(available) if len(available) <= n_correct else rng.sample(available, n_correct)
|
||||
)
|
||||
grouped[task_type] = errs + sampled
|
||||
|
||||
return grouped
|
||||
|
||||
|
||||
def _small_groups_decreasing(
|
||||
small: dict[str, list[QuestionUnit]],
|
||||
) -> list[list[QuestionUnit]]:
|
||||
"""按组题目总数降序、同大小按 task_type 升序排出小类组(first-fit-decreasing 顺序)。
|
||||
|
||||
参数:
|
||||
small: task_type -> 小类单元列表。
|
||||
返回:
|
||||
排好序的小类组列表;降序处理可降低碎片,确定性 tie-break 保证跨运行一致。
|
||||
"""
|
||||
return [small[t] for t in sorted(small, key=lambda t: (-_group_load(small[t]), t))]
|
||||
|
||||
|
||||
def _pack_small_class(
|
||||
batches: list[list[QuestionUnit]],
|
||||
group: list[QuestionUnit],
|
||||
batch_size: int,
|
||||
) -> None:
|
||||
"""用 first-fit 把一个小类整组放入首个容得下的 batch,装不下则新开 bin(就地修改)。
|
||||
|
||||
因小类组题目总数 ≤ min_class_per_batch < batch_size,新开的空 batch 必能容纳整组,
|
||||
故此函数永不抛 ValueError,且整组(含内部 pair 单元)不拆。
|
||||
|
||||
参数:
|
||||
batches: 当前各 batch(就地追加,必要时 append 新空 batch)。
|
||||
group: 待锁定的小类单元组(整组不拆)。
|
||||
batch_size: 单 batch 题目容量上限。
|
||||
"""
|
||||
load = _group_load(group)
|
||||
for b in batches:
|
||||
if _batch_load(b) + load <= batch_size:
|
||||
b.extend(group)
|
||||
return
|
||||
batches.append(list(group))
|
||||
|
||||
|
||||
def _distribute_large_classes(
|
||||
batches: list[list[QuestionUnit]],
|
||||
large: dict[str, list[QuestionUnit]],
|
||||
batch_size: int,
|
||||
rng_nonar: random.Random,
|
||||
rng_ar: random.Random,
|
||||
) -> None:
|
||||
"""将各大类单元洗牌后 round-robin 分发到所有现存 batch(就地修改)。
|
||||
|
||||
参数:
|
||||
batches: 当前各 batch(含小类装箱可能新开的 bin,就地追加)。
|
||||
large: task_type -> 大类单元列表。
|
||||
batch_size: 单 batch 题目容量上限。
|
||||
rng_nonar: 非 AR(single 单元)洗牌用 rng。
|
||||
rng_ar: AR(pair 单元)洗牌用 rng。
|
||||
关键实现细节:
|
||||
每组按单元 kind 拆成 single 子列与 pair 子列,分别用 rng_nonar / rng_ar 洗牌后
|
||||
拼接(single 在前),使非 AR 洗牌 draw 流不受 pair 存在与否影响(纯 single 时
|
||||
single 子列即整组,复现旧版单一 rng.shuffle 的序列)。全局指针在所有大类单元间
|
||||
持续轮转,遇满箱跳过、遇碎片新开 bin。题型按名称排序以保证分发顺序确定。
|
||||
"""
|
||||
pointer = 0
|
||||
for task_type in sorted(large):
|
||||
group = large[task_type]
|
||||
singles = [u for u in group if u.kind == "single"]
|
||||
pairs = [u for u in group if u.kind == "pair"]
|
||||
rng_nonar.shuffle(singles)
|
||||
rng_ar.shuffle(pairs)
|
||||
for unit in singles + pairs:
|
||||
pointer = _place_round_robin(batches, unit, pointer, batch_size)
|
||||
|
||||
|
||||
def _place_round_robin(
|
||||
batches: list[list[QuestionUnit]],
|
||||
unit: QuestionUnit,
|
||||
pointer: int,
|
||||
batch_size: int,
|
||||
) -> int:
|
||||
"""从 pointer 起找第一个容量够放 unit 的 batch 放入,返回下一次起始指针。
|
||||
|
||||
参数:
|
||||
batches: 当前各 batch(就地追加)。
|
||||
unit: 待放置的单元(占用 unit.size 个容量)。
|
||||
pointer: 本次轮转起始 batch 下标。
|
||||
batch_size: 单 batch 题目容量上限。
|
||||
返回:
|
||||
下一次轮转的起始指针(已前移一位)。
|
||||
关键实现:
|
||||
单个单元容量 ≤ batch_size 是前提(pair 占 2,而 batch_size > min_class ≥ 1 ⇒
|
||||
batch_size ≥ 2),故此处断言防御。扫描一轮所有现存 batch 都放不下(size-2 单元
|
||||
遇满地碎片)时新开 bin 兜底而非报错——聚合容量足够但单箱剩余不足是合法碎片场景。
|
||||
纯 single(size 1)永不触发新开分支,故与旧逐题 round-robin 逐字节一致。
|
||||
"""
|
||||
assert unit.size <= batch_size, f"单元 size={unit.size} 超过 batch_size={batch_size}"
|
||||
nb = len(batches)
|
||||
for offset in range(nb):
|
||||
idx = (pointer + offset) % nb
|
||||
if _batch_load(batches[idx]) + unit.size <= batch_size:
|
||||
batches[idx].append(unit)
|
||||
return (idx + 1) % nb
|
||||
batches.append([unit])
|
||||
return len(batches) % len(batches)
|
||||
@@ -0,0 +1,634 @@
|
||||
"""结果驱动视频级切分的顶层编排:诊断信号 → 冻结 pools.json + manifest。
|
||||
|
||||
把已实现的组件串成 capstone 管线:从 harness.db 读 canonical 基线预测、从
|
||||
DiagnosisSignalStore 读逐题诊断信号,构建全视频画像、贪心联合约束选择 trainval /
|
||||
test,再以视频组为原子切出诊断 / 验证池,原子冻结 pools.json 并写溯源 manifest。
|
||||
全程带六条防御断言(P5,任一不满足即 fail-fast,绝不静默兜底)。
|
||||
|
||||
只有基线推理与诊断是上游产物;本模块纯 code-controlled,不发起任何 LLM 调用,
|
||||
读预测走只读连接,不改动 harness.db。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import Counter, defaultdict
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.pools import save_pools, split_by_video_assignment
|
||||
from app.harness.split_manifest import write_manifest
|
||||
from app.harness.split_selection import (
|
||||
SelectConfig,
|
||||
build_video_records,
|
||||
derive_reportable_types,
|
||||
select_split,
|
||||
)
|
||||
from app.question_gen.loader import load_benchmark
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from app.harness.pools import Pools
|
||||
from app.harness.split_selection import SplitAssignment, VideoRecord
|
||||
from core.evolution.protocols import DiagnosisSignalStore
|
||||
from core.evolution.types import DiagnosisSignalRow
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
# 多样性主格子 = 12 题型 × 4 错误类别 = 48 格,覆盖报告以此为分母。
|
||||
_DIVERSITY_GRID_TOTAL = 48
|
||||
_QUESTIONS_PER_VIDEO = 3
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SplitBuildConfig:
|
||||
"""结果驱动切分的旋钮快照(科研配置,随实验扫动)。
|
||||
|
||||
承载贪心选择器与视频组题级切分的全部可扫参数;asdict 后直接写入 manifest 的
|
||||
config 快照,保证复现时可比对。floor_k 为不可哈希容器,标 hash=False 排除出
|
||||
自动 __hash__,避免 frozen dataclass 被哈希时报错(本类不作字典键,仅承载配置)。
|
||||
|
||||
字段:
|
||||
n_trainval: trainval 目标视频数(多样性阶段填充上限)。
|
||||
floor_k: 各高信号 task_type 的 T2 defect 下限(select_split 硬约束)。
|
||||
epsilon: test 相对全局的最大允许分布偏差(题型 / 难度两维)。
|
||||
report_floor: per-type 报告门限,题数 ≥ 此值的 task_type 才入 ε 约束。
|
||||
select_seed: 贪心选择器预洗牌种子(打破等增益平局)。
|
||||
val_ratio: validation 占 trainval 视频组总数的比例。
|
||||
split_seed: 视频组题级切分的洗牌种子。
|
||||
val_wrong_min: validation 池最少错题数,切分时保证功效(不足则从 diag 换入
|
||||
低 T2 错题组补足,耗尽 fail-loud)。
|
||||
"""
|
||||
|
||||
n_trainval: int
|
||||
floor_k: dict[str, int] = field(hash=False)
|
||||
epsilon: float
|
||||
report_floor: int
|
||||
select_seed: int
|
||||
val_ratio: float
|
||||
split_seed: int
|
||||
val_wrong_min: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SplitBuildResult:
|
||||
"""build_split 的返回结果(冻结产物 + 溯源)。
|
||||
|
||||
字段:
|
||||
pools: 冻结的三池(diagnosis / validation / test)。
|
||||
manifest: 写入 split_manifest.json 的溯源字典(含 pools_sha256)。
|
||||
assignment: video_id -> "trainval" | "test" 归属字典。
|
||||
"""
|
||||
|
||||
pools: Pools
|
||||
manifest: dict
|
||||
assignment: dict[str, str]
|
||||
|
||||
def __getitem__(self, key: str) -> object:
|
||||
"""兼容字典式访问(result["pools"] / ["manifest"] / ["assignment"])。
|
||||
|
||||
参数:
|
||||
key: 字段名,取值 pools / manifest / assignment。
|
||||
|
||||
返回:
|
||||
对应字段值。
|
||||
|
||||
异常:
|
||||
KeyError: key 非上述三者之一。
|
||||
"""
|
||||
if key not in {"pools", "manifest", "assignment"}:
|
||||
raise KeyError(f"未知字段: {key}")
|
||||
return getattr(self, key)
|
||||
|
||||
|
||||
def _unique_backup_path(path: Path, suffix: str) -> Path:
|
||||
"""求 path 的唯一 .bak.<suffix> 备份路径,已存在则追加递增序号避免覆盖。
|
||||
|
||||
首选 ``<name>.bak.<suffix>``;若已存在,退化为 ``<name>.bak.<suffix>.2``、
|
||||
``.3`` … 直到找到不存在的名字。保证连续 forced freeze(同 suffix 或都缺
|
||||
manifest 用 'prev')不会静默覆盖此前保留的备份。
|
||||
|
||||
参数:
|
||||
path: 待备份的原文件路径。
|
||||
suffix: 备份后缀(旧 pools_sha256 前 8 位或 'prev')。
|
||||
|
||||
返回:
|
||||
目录内唯一、尚不存在的备份路径。
|
||||
"""
|
||||
candidate = path.with_name(f"{path.name}.bak.{suffix}")
|
||||
counter = 2
|
||||
while candidate.exists():
|
||||
candidate = path.with_name(f"{path.name}.bak.{suffix}.{counter}")
|
||||
counter += 1
|
||||
return candidate
|
||||
|
||||
|
||||
def _guard_frozen_products(out_path: Path, manifest_path: Path, *, force: bool) -> None:
|
||||
"""冻结前的覆盖保护:产物已存在时按 force 决定报错或备份。
|
||||
|
||||
参数:
|
||||
out_path: 目标 pools.json 路径。
|
||||
manifest_path: 目标 split_manifest.json 路径。
|
||||
force: False 时已存在即 FileExistsError;True 时把旧产物重命名为唯一的
|
||||
.bak.<旧 pools_sha256 前 8 位或 'prev'>(同名已存在则追加递增序号)再放行。
|
||||
|
||||
异常:
|
||||
FileExistsError: force=False 且产物已存在(防静默覆盖冻结锚点)。
|
||||
OSError: 备份 rename 失败;已备份的文件先 rollback 回原名再抛出,保证
|
||||
要么两文件都备份、要么都不动(原子性,不留半备份的不一致目录)。
|
||||
"""
|
||||
if not out_path.exists() and not manifest_path.exists():
|
||||
return
|
||||
if not force:
|
||||
raise FileExistsError(
|
||||
f"已存在冻结产物 {out_path}(或其 manifest)。重跑切分会覆盖训练依赖的"
|
||||
"冻结锚点——确认要替换请加 --force(旧产物将备份为 .bak.*)。"
|
||||
)
|
||||
# 备份后缀取旧 manifest 的 pools_sha256 前 8 位,无则用 'prev'
|
||||
suffix = "prev"
|
||||
if manifest_path.exists():
|
||||
try:
|
||||
old = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
suffix = str(old.get("pools_sha256", "prev"))[:8] or "prev"
|
||||
except (json.JSONDecodeError, OSError):
|
||||
suffix = "prev"
|
||||
# 先为每个存在的文件求唯一备份路径(互不冲突),再逐个 rename;
|
||||
# 中途失败则把已备份的 rollback 回原名,保证原子性。
|
||||
to_backup = [p for p in (out_path, manifest_path) if p.exists()]
|
||||
done: list[tuple[Path, Path]] = [] # (备份路径, 原路径),供 rollback
|
||||
try:
|
||||
for p in to_backup:
|
||||
dst = _unique_backup_path(p, suffix)
|
||||
p.rename(dst)
|
||||
done.append((dst, p))
|
||||
except OSError:
|
||||
for backup_path, original in reversed(done):
|
||||
backup_path.rename(original)
|
||||
raise
|
||||
|
||||
|
||||
def build_split(
|
||||
*,
|
||||
db_path: Path,
|
||||
baseline_run_id: str,
|
||||
signal_store: DiagnosisSignalStore,
|
||||
diag_fingerprint: str,
|
||||
questions_dir: Path,
|
||||
config: SplitBuildConfig,
|
||||
out_path: Path,
|
||||
manifest_path: Path,
|
||||
generated_at: str,
|
||||
force: bool = False,
|
||||
) -> SplitBuildResult:
|
||||
"""顶层编排结果驱动视频级切分,冻结 pools.json + manifest 并跑防御断言。
|
||||
|
||||
步骤:读 canonical 基线预测 → 读诊断信号 → 构建全视频画像 → 贪心选择 trainval /
|
||||
test → 加载题库并以视频归属切三池 → 原子冻结 pools.json → 写溯源 manifest →
|
||||
六条防御断言 fail-fast 校验。
|
||||
|
||||
契约(Task 11):val_wrong_min 前置到切分内保证功效——build_split 计算
|
||||
wrong_tier_by_video 并连同 config.val_wrong_min 传入 split_by_video_assignment,
|
||||
切分时若 val 错题不足即从 diag 换入低 T2 错题组补足(耗尽 fail-loud)。CLI 的
|
||||
check_mcnemar_power 作切分冻结后的冗余最终确认。
|
||||
|
||||
参数:
|
||||
db_path: harness.db 路径(只读读取 predictions,不改动)。
|
||||
baseline_run_id: 基线 run 标识(如 "infer_adhoc")。
|
||||
signal_store: 逐题诊断信号存储端口,读 (run, fingerprint) 下全部信号行。
|
||||
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号。
|
||||
questions_dir: benchmark 题库目录,加载 GeneratedQuestion。
|
||||
config: 切分旋钮快照。
|
||||
out_path: 冻结 pools.json 目标路径(原子写)。
|
||||
manifest_path: 溯源 manifest 目标路径(原子写)。
|
||||
generated_at: 生成时间戳(ISO 字符串),由调用方传入以保证可复现。
|
||||
force: 覆盖保护开关。False(默认)时若 out_path/manifest_path 已存在即
|
||||
FileExistsError(防静默覆盖训练依赖的冻结锚点);True 时先把旧产物备份为
|
||||
.bak.* 再放行覆盖。
|
||||
|
||||
返回:
|
||||
SplitBuildResult,含 pools / manifest / assignment,支持字典式访问。
|
||||
|
||||
异常:
|
||||
AssertionError: 六条防御断言任一不满足(fail-fast,不静默)。
|
||||
ValueError: 上游依赖校验失败(如 correctness 缺题、assignment 非法)。
|
||||
"""
|
||||
# Phase 1: canonical 基线预测 + 诊断信号。
|
||||
preds = load_canonical_predictions(db_path, baseline_run_id)
|
||||
signal_rows_raw = signal_store.load(baseline_run_id, diag_fingerprint)
|
||||
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint)
|
||||
signal_rows = [
|
||||
{
|
||||
"question_id": row.question_id,
|
||||
"task_type": row.task_type,
|
||||
"error_type": row.error_type,
|
||||
"tier": row.tier,
|
||||
}
|
||||
for row in signal_rows_raw
|
||||
]
|
||||
|
||||
# Phase 2: 全视频画像 + 贪心联合约束选择。
|
||||
videos = build_video_records(preds, signal_rows)
|
||||
total_by_type = Counter(pred["task_type"] for pred in preds)
|
||||
reportable_types = derive_reportable_types(dict(total_by_type), config.report_floor)
|
||||
assignment_obj = select_split(
|
||||
videos,
|
||||
config=SelectConfig(
|
||||
n_trainval=config.n_trainval,
|
||||
floor_k=config.floor_k,
|
||||
epsilon=config.epsilon,
|
||||
reportable_types=reportable_types,
|
||||
seed=config.select_seed,
|
||||
),
|
||||
)
|
||||
assignment = _assignment_to_dict(assignment_obj)
|
||||
logger.info(
|
||||
"视频级切分完成: trainval={} test={} (总 {} 视频)",
|
||||
len(assignment_obj.trainval),
|
||||
len(assignment_obj.test),
|
||||
len(videos),
|
||||
)
|
||||
|
||||
# Phase 3: 加载题库 + 视频归属切三池 + 原子冻结。
|
||||
questions = load_benchmark(questions_dir)
|
||||
correctness = {pred["question_id"]: pred["correct"] for pred in preds}
|
||||
tier_by_q = {row["question_id"]: row["tier"] for row in signal_rows}
|
||||
wrong_tier_by_video: dict[str, int] = defaultdict(int)
|
||||
for pred in preds:
|
||||
if not pred["correct"] and tier_by_q.get(pred["question_id"]) == "T2":
|
||||
wrong_tier_by_video[pred["video_id"]] += 1
|
||||
pools = split_by_video_assignment(
|
||||
questions,
|
||||
assignment,
|
||||
correctness,
|
||||
config.val_ratio,
|
||||
config.split_seed,
|
||||
baseline_run_id=baseline_run_id,
|
||||
val_wrong_min=config.val_wrong_min,
|
||||
wrong_tier_by_video=dict(wrong_tier_by_video),
|
||||
)
|
||||
_guard_frozen_products(out_path, manifest_path, force=force)
|
||||
save_pools(pools, out_path)
|
||||
|
||||
# Phase 4: 溯源 manifest(pools_sha256 锚定冻结内容)。
|
||||
coverage_report = _build_coverage_report(
|
||||
videos, assignment_obj, signal_rows_raw, reportable_types, config
|
||||
)
|
||||
manifest = write_manifest(
|
||||
manifest_path,
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
seed=config.split_seed,
|
||||
config=asdict(config),
|
||||
pools_json_text=out_path.read_text(encoding="utf-8"),
|
||||
coverage_report=coverage_report,
|
||||
generated_at=generated_at,
|
||||
)
|
||||
|
||||
# Phase 5: 防御断言 fail-fast。
|
||||
_assert_split_invariants(
|
||||
pools=pools,
|
||||
expected_question_ids={q.question_id for q in questions},
|
||||
signal_rows_raw=signal_rows_raw,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
out_path=out_path,
|
||||
manifest=manifest,
|
||||
)
|
||||
|
||||
return SplitBuildResult(pools=pools, manifest=manifest, assignment=assignment)
|
||||
|
||||
|
||||
def _normalize_choice(choice: str | None) -> str:
|
||||
"""选项归一:strip → 大写 → 取首字母,None 归一为空串。
|
||||
|
||||
参数:
|
||||
choice: 原始选项文本(预测或答案),可为 None。
|
||||
|
||||
返回:
|
||||
归一后的单字母(无内容时为空串)。
|
||||
"""
|
||||
return (choice or "").strip().upper()[:1]
|
||||
|
||||
|
||||
def load_canonical_predictions(db_path: Path, baseline_run_id: str) -> list[dict]:
|
||||
"""从 harness.db 只读取指定 run 每题首行(ORDER BY rowid)为 canonical 预测。
|
||||
|
||||
共享口径 helper:CLI(可诊断错题筛选 + INFRA T0 补录)与 build_split(切分)
|
||||
共用同一"每 qid 取 rowid 最小首行 + 归一化 correct 判定"口径,消除两处重复实现。
|
||||
同一 question_id 可能有多行(重跑 / 补测),canonical 口径取 rowid 最小的首行,
|
||||
保证 distinct question 计数与对错判定确定。correct = 预测与答案各自归一
|
||||
(strip → 大写 → 取首字母)后逐字符相等。
|
||||
|
||||
口径边界:旧 build_or_load_pools 的 legacy 池构建路径(app/harness/pools.py)是
|
||||
另一条独立既有链路,不共用本 helper,两者刻意不统一(本次不动 legacy 路径)。
|
||||
|
||||
参数:
|
||||
db_path: harness.db 路径(URI mode=ro 只读打开,绝不改动基线 db)。
|
||||
baseline_run_id: 基线 run 标识。
|
||||
|
||||
返回:
|
||||
canonical 预测行列表,每行含 question_id / video_id / task_type /
|
||||
prediction / answer / stop_reason / correct(bool)。按 rowid 升序去重,
|
||||
每 qid 保留首行。
|
||||
|
||||
异常:
|
||||
ValueError: 该 run 无任何预测行(fail-fast,不返回空切分)。
|
||||
|
||||
实现细节:
|
||||
按 rowid 升序遍历,首次见到的 question_id 即 canonical 行,后续同 qid 行跳过。
|
||||
"""
|
||||
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT question_id, video_id, task_type, prediction, answer, stop_reason "
|
||||
"FROM predictions WHERE run_id = ? ORDER BY rowid",
|
||||
(baseline_run_id,),
|
||||
).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
canonical: dict[str, dict] = {}
|
||||
for row in rows:
|
||||
qid = row["question_id"]
|
||||
if qid in canonical:
|
||||
continue
|
||||
canonical[qid] = {
|
||||
"question_id": qid,
|
||||
"video_id": row["video_id"],
|
||||
"task_type": row["task_type"],
|
||||
"prediction": row["prediction"],
|
||||
"answer": row["answer"],
|
||||
"stop_reason": row["stop_reason"],
|
||||
"correct": _normalize_choice(row["prediction"]) == _normalize_choice(row["answer"]),
|
||||
}
|
||||
if not canonical:
|
||||
raise ValueError(f"run_id={baseline_run_id} 无任何预测行,无法切分")
|
||||
return list(canonical.values())
|
||||
|
||||
|
||||
def _assignment_to_dict(assignment_obj: SplitAssignment) -> dict[str, str]:
|
||||
"""把 SplitAssignment 展平为 video_id -> "trainval" | "test" 归属字典。
|
||||
|
||||
参数:
|
||||
assignment_obj: 贪心选择器产出的切分归属。
|
||||
|
||||
返回:
|
||||
全部视频的归属字典(trainval 与 test 并集,键互斥)。
|
||||
"""
|
||||
assignment = dict.fromkeys(assignment_obj.trainval, "trainval")
|
||||
for vid in assignment_obj.test:
|
||||
assignment[vid] = "test"
|
||||
return assignment
|
||||
|
||||
|
||||
def _assert_fingerprint_consistent(
|
||||
signal_rows_raw: list[DiagnosisSignalRow],
|
||||
diag_fingerprint: str,
|
||||
) -> None:
|
||||
"""防御④:全部诊断信号行的 diag_fingerprint 必须与传入指纹一致。
|
||||
|
||||
参数:
|
||||
signal_rows_raw: store 读回的诊断信号行。
|
||||
diag_fingerprint: 期望的诊断口径指纹。
|
||||
|
||||
异常:
|
||||
AssertionError: 存在指纹不一致的信号行(store 未正确按指纹过滤)。
|
||||
"""
|
||||
mismatched = [
|
||||
row.question_id for row in signal_rows_raw if row.diag_fingerprint != diag_fingerprint
|
||||
]
|
||||
if mismatched:
|
||||
raise AssertionError(
|
||||
f"诊断信号指纹不一致 {len(mismatched)} 行,期望 {diag_fingerprint}: {mismatched[:5]}"
|
||||
)
|
||||
|
||||
|
||||
def _fraction_by_type(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
|
||||
"""各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
|
||||
|
||||
参数:
|
||||
records: 视频记录子集。
|
||||
keys: 需计算占比的 task_type 键集。
|
||||
|
||||
返回:
|
||||
{task_type: 占比};records 为空时全部记 0.0。
|
||||
"""
|
||||
total = len(records)
|
||||
if total == 0:
|
||||
return dict.fromkeys(keys, 0.0)
|
||||
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
|
||||
|
||||
|
||||
def _fraction_by_difficulty(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
|
||||
"""各难度桶(错题数)在给定视频集中的占比。
|
||||
|
||||
参数:
|
||||
records: 视频记录子集。
|
||||
buckets: 难度桶键集。
|
||||
|
||||
返回:
|
||||
{难度桶: 占比};records 为空时全部记 0.0。
|
||||
"""
|
||||
total = len(records)
|
||||
if total == 0:
|
||||
return dict.fromkeys(buckets, 0.0)
|
||||
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
|
||||
|
||||
|
||||
def _max_dev(global_dist: dict, subset_dist: dict, keys: set) -> float:
|
||||
"""逐键取全局与子集分布的最大绝对偏差(键集为空约定 0.0)。"""
|
||||
if not keys:
|
||||
return 0.0
|
||||
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
|
||||
|
||||
|
||||
def _build_coverage_report(
|
||||
videos: list[VideoRecord],
|
||||
assignment_obj: SplitAssignment,
|
||||
signal_rows_raw: list[DiagnosisSignalRow],
|
||||
reportable_types: set[str],
|
||||
config: SplitBuildConfig,
|
||||
) -> dict:
|
||||
"""组装 manifest 覆盖报告:48 格覆盖 / floor 达标 / test 代表性偏差 / tier 占比。
|
||||
|
||||
参数:
|
||||
videos: 全视频画像记录。
|
||||
assignment_obj: 切分归属(trainval / test)。
|
||||
signal_rows_raw: 诊断信号行(统计 tier 占比)。
|
||||
reportable_types: 参与 ε 代表性校验的题型集。
|
||||
config: 切分旋钮(floor_k / epsilon)。
|
||||
|
||||
返回:
|
||||
覆盖报告字典,含 cells_covered / grid_total / floor_satisfied /
|
||||
test_representativeness_deviation / tier_distribution /
|
||||
evolution_target_distribution。
|
||||
|
||||
实现细节:
|
||||
evolution_target_distribution 只统计 T2 信号(可训练缺陷),按
|
||||
tool / skill / system 计数,报告"哪层参数组拿到梯度";T0/T1/uncertain 行
|
||||
evolution_target 恒为 None,不入该分布。
|
||||
"""
|
||||
by_id = {v.video_id: v for v in videos}
|
||||
trainval = [by_id[vid] for vid in assignment_obj.trainval]
|
||||
test = [by_id[vid] for vid in assignment_obj.test]
|
||||
|
||||
covered_cells: set[tuple[str, str]] = set()
|
||||
trainval_wrong: Counter[str] = Counter()
|
||||
for video in trainval:
|
||||
covered_cells |= set(video.cells)
|
||||
trainval_wrong.update(video.wrong_by_type)
|
||||
floor_satisfied = {
|
||||
task_type: trainval_wrong.get(task_type, 0) >= floor
|
||||
for task_type, floor in config.floor_k.items()
|
||||
}
|
||||
|
||||
type_dev = _max_dev(
|
||||
_fraction_by_type(videos, reportable_types),
|
||||
_fraction_by_type(test, reportable_types),
|
||||
reportable_types,
|
||||
)
|
||||
diff_buckets = {r.difficulty for r in videos}
|
||||
diff_dev = _max_dev(
|
||||
_fraction_by_difficulty(videos, diff_buckets),
|
||||
_fraction_by_difficulty(test, diff_buckets),
|
||||
diff_buckets,
|
||||
)
|
||||
|
||||
tier_distribution, evolution_target_distribution = _signal_distributions(signal_rows_raw)
|
||||
|
||||
return {
|
||||
"cells_covered": len(covered_cells),
|
||||
"grid_total": _DIVERSITY_GRID_TOTAL,
|
||||
"floor_satisfied": floor_satisfied,
|
||||
"test_representativeness_deviation": {
|
||||
"type_max": type_dev,
|
||||
"difficulty_max": diff_dev,
|
||||
"epsilon": config.epsilon,
|
||||
},
|
||||
"tier_distribution": tier_distribution,
|
||||
"evolution_target_distribution": evolution_target_distribution,
|
||||
}
|
||||
|
||||
|
||||
def _signal_distributions(
|
||||
signal_rows_raw: list[DiagnosisSignalRow],
|
||||
) -> tuple[dict[str, float], dict[str, int]]:
|
||||
"""由诊断信号行算 tier 占比分布与 T2 进化目标计数分布。
|
||||
|
||||
参数:
|
||||
signal_rows_raw: 诊断信号行。
|
||||
|
||||
返回:
|
||||
(tier_distribution, evolution_target_distribution) 二元组:
|
||||
- tier_distribution: {tier: 占比},无信号时为空 dict;
|
||||
- evolution_target_distribution: 仅统计 T2(可训练缺陷)信号,按
|
||||
tool / skill / system 计数,报告哪层参数组拿到梯度;T0/T1/uncertain 行
|
||||
evolution_target 恒为 None,不入该分布。
|
||||
"""
|
||||
tier_counts = Counter(row.tier for row in signal_rows_raw)
|
||||
total_signals = sum(tier_counts.values())
|
||||
tier_distribution = (
|
||||
{tier: count / total_signals for tier, count in tier_counts.items()}
|
||||
if total_signals
|
||||
else {}
|
||||
)
|
||||
evolution_target_distribution = dict(
|
||||
Counter(
|
||||
row.evolution_target
|
||||
for row in signal_rows_raw
|
||||
if row.tier == "T2" and row.evolution_target is not None
|
||||
)
|
||||
)
|
||||
return tier_distribution, evolution_target_distribution
|
||||
|
||||
|
||||
def _assert_split_invariants(
|
||||
*,
|
||||
pools: Pools,
|
||||
expected_question_ids: set[str],
|
||||
signal_rows_raw: list[DiagnosisSignalRow],
|
||||
diag_fingerprint: str,
|
||||
out_path: Path,
|
||||
manifest: dict,
|
||||
) -> None:
|
||||
"""六条防御断言 fail-fast:任一不满足即 AssertionError(P5,不静默不兜底)。
|
||||
|
||||
① 三池视频集两两不相交;② 三池覆盖全部题(按 distinct question);
|
||||
③ 每 video 恰 3 题;④ 诊断信号指纹一致;⑤ manifest pools_sha256 == sha256(冻结内容);
|
||||
⑥ question_id 全局唯一(无重复行)。
|
||||
|
||||
参数:
|
||||
pools: 冻结的三池。
|
||||
expected_question_ids: 加载题库的 question_id 全集(覆盖基准)。
|
||||
signal_rows_raw: 诊断信号行(指纹校验)。
|
||||
diag_fingerprint: 期望诊断指纹。
|
||||
out_path: 冻结 pools.json 路径。
|
||||
manifest: 已写入的 manifest 字典。
|
||||
|
||||
异常:
|
||||
AssertionError: 任一防御断言不满足。
|
||||
"""
|
||||
all_questions = pools.diagnosis + pools.validation + pools.test
|
||||
_assert_pools_video_disjoint(pools) # ①
|
||||
_assert_question_ids_unique(all_questions) # ⑥
|
||||
_assert_question_coverage(all_questions, expected_question_ids) # ②
|
||||
_assert_three_questions_per_video(all_questions) # ③
|
||||
_assert_fingerprint_consistent(signal_rows_raw, diag_fingerprint) # ④
|
||||
_assert_pools_sha256(out_path, manifest) # ⑤
|
||||
|
||||
|
||||
def _assert_pools_video_disjoint(pools: Pools) -> None:
|
||||
"""防御①:diagnosis / validation / test 三池视频集两两不相交。"""
|
||||
diag_v = {q.video_id for q in pools.diagnosis}
|
||||
val_v = {q.video_id for q in pools.validation}
|
||||
test_v = {q.video_id for q in pools.test}
|
||||
if diag_v & val_v or diag_v & test_v or val_v & test_v:
|
||||
raise AssertionError(
|
||||
f"三池视频集非互斥: diag∩val={diag_v & val_v}, "
|
||||
f"diag∩test={diag_v & test_v}, val∩test={val_v & test_v}"
|
||||
)
|
||||
|
||||
|
||||
def _assert_question_ids_unique(all_questions: list[GeneratedQuestion]) -> None:
|
||||
"""防御⑥:三池合并后 question_id 全局唯一(无重复行)。"""
|
||||
qids = [q.question_id for q in all_questions]
|
||||
if len(qids) != len(set(qids)):
|
||||
duplicates = [qid for qid, count in Counter(qids).items() if count > 1]
|
||||
raise AssertionError(f"question_id 重复 {len(duplicates)} 个: {duplicates[:5]}")
|
||||
|
||||
|
||||
def _assert_question_coverage(
|
||||
all_questions: list[GeneratedQuestion],
|
||||
expected_question_ids: set[str],
|
||||
) -> None:
|
||||
"""防御②:三池覆盖题库全部题(按 distinct question,缺题 / 多题均 fail-fast)。"""
|
||||
actual = {q.question_id for q in all_questions}
|
||||
if actual != expected_question_ids:
|
||||
missing = expected_question_ids - actual
|
||||
extra = actual - expected_question_ids
|
||||
raise AssertionError(
|
||||
f"三池题目覆盖不完整: 缺 {len(missing)} 多 {len(extra)} (缺样例 {sorted(missing)[:5]})"
|
||||
)
|
||||
|
||||
|
||||
def _assert_three_questions_per_video(all_questions: list[GeneratedQuestion]) -> None:
|
||||
"""防御③:每 video 恰 3 题(视频组原子切分不应劈裂视频的题)。"""
|
||||
per_video = Counter(q.video_id for q in all_questions)
|
||||
bad_videos = {vid: n for vid, n in per_video.items() if n != _QUESTIONS_PER_VIDEO}
|
||||
if bad_videos:
|
||||
raise AssertionError(
|
||||
f"存在 video 题数 != {_QUESTIONS_PER_VIDEO}: {dict(list(bad_videos.items())[:5])}"
|
||||
)
|
||||
|
||||
|
||||
def _assert_pools_sha256(out_path: Path, manifest: dict) -> None:
|
||||
"""防御⑤:manifest 的 pools_sha256 == sha256(冻结 pools.json 内容)。"""
|
||||
actual_sha = hashlib.sha256(out_path.read_text(encoding="utf-8").encode("utf-8")).hexdigest()
|
||||
if actual_sha != manifest["pools_sha256"]:
|
||||
raise AssertionError(
|
||||
f"pools_sha256 不一致: manifest={manifest['pools_sha256']} 实际={actual_sha}"
|
||||
)
|
||||
@@ -0,0 +1,275 @@
|
||||
"""step 级续训 checkpoint:_TrainState 可持久化字段的序列化 / 反序列化。
|
||||
|
||||
_TrainState 的累加包均为扁平纯数据 dataclass,经 dataclasses.asdict 序列化为
|
||||
纯 JSON dict;反序列化时用 Cls(**d) 还原,其中 SystemCasePack 含嵌套 CaseSample
|
||||
列表、Probation 含嵌套 RejectedEdit 列表,需逐个重建。
|
||||
|
||||
不持久化的字段:gate_pools / baseline_cache(各自文件级自持久化,resume 时按
|
||||
指纹重载)、best_*(从 manifest best 指针读)、global_step(存 progress 块,
|
||||
由 train 单独赋值)。gate_epoch_observed 持久化:warm p-hat 在 gate_pools.json
|
||||
幸存,观测开关须随行,否则 resume 后阶梯排序回退冷启动序。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import asdict
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.harness.validate import Probation
|
||||
from core.evolution.types import (
|
||||
CaseSample,
|
||||
RejectedEdit,
|
||||
SystemCasePack,
|
||||
ToolCasePack,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
CHECKPOINT_SCHEMA_VERSION = 1
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 结构性 / 决策性指纹键
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_STRUCTURAL_KEYS = (
|
||||
"batch_size",
|
||||
"min_class_per_batch",
|
||||
"epochs",
|
||||
"diag_size",
|
||||
"val_size",
|
||||
"batch_correct_ratio",
|
||||
"trainable_min_units",
|
||||
)
|
||||
|
||||
_DECISION_KEYS = (
|
||||
"edit_budget_start",
|
||||
"edit_budget_end",
|
||||
"early_stop_patience",
|
||||
"use_slow_momentum",
|
||||
"skill_update_mode",
|
||||
"appendix_consolidate_threshold",
|
||||
"momentum_samples",
|
||||
"gate_e_confirm",
|
||||
"gate_e_provisional",
|
||||
"gate_w_net_min",
|
||||
"gate_delta_min",
|
||||
"gate_lambda_dir",
|
||||
"gate_e_rollback",
|
||||
"gate_n_max",
|
||||
"gate_p_low",
|
||||
"gate_p_high",
|
||||
"gate_probe_quota",
|
||||
"gate_gamma_decay",
|
||||
"gate_cooldown_steps",
|
||||
"gate_guard_err",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 序列化 / 反序列化
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def serialize_state(state: Any) -> dict[str, Any]:
|
||||
"""把 _TrainState 的可持久化字段转为纯 JSON dict。
|
||||
|
||||
参数:
|
||||
state: _TrainState 实例(duck-typed,仅需含可持久化字段)。
|
||||
|
||||
返回:
|
||||
纯 JSON 可序列化的 dict,不含 gate_pools / baseline_cache /
|
||||
best_* / global_step。
|
||||
|
||||
关键实现细节:
|
||||
- changed_task_types_this_epoch 是 set,JSON 无 set,故 sorted 成有序列表。
|
||||
- dataclass 均经 asdict 递归转 dict(含 SystemCasePack 嵌套 CaseSample、
|
||||
Probation 嵌套 RejectedEdit)。
|
||||
"""
|
||||
return {
|
||||
"correctness": state.correctness,
|
||||
"eval_prev_acc": state.eval_prev_acc,
|
||||
"eval_prev_run_id": state.eval_prev_run_id,
|
||||
"baseline_skills_version": state.baseline_skills_version,
|
||||
"baseline_prompts_version": state.baseline_prompts_version,
|
||||
"epochs_since_best_improved": state.epochs_since_best_improved,
|
||||
"epoch_start_skills": state.epoch_start_skills,
|
||||
"changed_task_types_this_epoch": sorted(state.changed_task_types_this_epoch),
|
||||
"rejected_buffer": {k: [asdict(x) for x in v] for k, v in state.rejected_buffer.items()},
|
||||
"system_packs": [asdict(x) for x in state.system_packs],
|
||||
"tool_packs": [asdict(x) for x in state.tool_packs],
|
||||
"probations": {t: asdict(p) for t, p in state.probations.items()},
|
||||
"gate_cooldown": state.gate_cooldown,
|
||||
"gate_epoch_observed": state.gate_epoch_observed,
|
||||
}
|
||||
|
||||
|
||||
def _restore_system_pack(d: dict[str, Any]) -> SystemCasePack:
|
||||
"""还原 SystemCasePack,含嵌套 CaseSample 列表。
|
||||
|
||||
参数:
|
||||
d: asdict(SystemCasePack) 产出的 dict。
|
||||
|
||||
返回:
|
||||
复活的 SystemCasePack;failure_cases / success_cases 重建为 CaseSample 实例。
|
||||
"""
|
||||
return SystemCasePack(
|
||||
stats=d["stats"],
|
||||
failure_cases=[CaseSample(**c) for c in d["failure_cases"]],
|
||||
success_cases=[CaseSample(**c) for c in d["success_cases"]],
|
||||
)
|
||||
|
||||
|
||||
def deserialize_state_fields(d: dict[str, Any]) -> dict[str, Any]:
|
||||
"""把序列化 dict 还原为可填入 _TrainState 的字段字典(dataclass 复活)。
|
||||
|
||||
参数:
|
||||
d: serialize_state 产出并经 JSON 往返的 dict。
|
||||
|
||||
返回:
|
||||
字段名 -> 值的 dict,可直接铺到 _TrainState;其中各 dataclass 已复活、
|
||||
changed_task_types_this_epoch 还原为 set。
|
||||
|
||||
关键实现细节:
|
||||
- RejectedEdit / ToolCasePack 字段均为标量/dict/list[dict],Cls(**d) 直接构造。
|
||||
- SystemCasePack 含嵌套 CaseSample,交由 _restore_system_pack 重建。
|
||||
- Probation 含嵌套 RejectedEdit 列表(pending_edits),先重建内层再构造外层。
|
||||
- 直接取 d[...] 不用 .get 兜底:serialize 后的 checkpoint 必带全部键,
|
||||
缺键即 checkpoint 损坏,应硬失败(P5 不掩盖)。
|
||||
"""
|
||||
return {
|
||||
"correctness": d["correctness"],
|
||||
"eval_prev_acc": d["eval_prev_acc"],
|
||||
"eval_prev_run_id": d["eval_prev_run_id"],
|
||||
"baseline_skills_version": d["baseline_skills_version"],
|
||||
"baseline_prompts_version": d["baseline_prompts_version"],
|
||||
"epochs_since_best_improved": d["epochs_since_best_improved"],
|
||||
"epoch_start_skills": d["epoch_start_skills"],
|
||||
"changed_task_types_this_epoch": set(d["changed_task_types_this_epoch"]),
|
||||
"rejected_buffer": {
|
||||
k: [RejectedEdit(**x) for x in v] for k, v in d["rejected_buffer"].items()
|
||||
},
|
||||
"system_packs": [_restore_system_pack(x) for x in d["system_packs"]],
|
||||
"tool_packs": [ToolCasePack(**x) for x in d["tool_packs"]],
|
||||
"probations": {
|
||||
t: Probation(
|
||||
**{
|
||||
**d_p,
|
||||
"pending_edits": [RejectedEdit(**x) for x in d_p["pending_edits"]],
|
||||
}
|
||||
)
|
||||
for t, d_p in d["probations"].items()
|
||||
},
|
||||
"gate_cooldown": d["gate_cooldown"],
|
||||
"gate_epoch_observed": d["gate_epoch_observed"],
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 配置指纹
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def compute_fingerprint(config: Any) -> dict[str, Any]:
|
||||
"""采集影响训练轨迹的配置项(结构性 + 决策性)。
|
||||
|
||||
参数:
|
||||
config: 训练配置对象(duck-typed,需含 _STRUCTURAL_KEYS + _DECISION_KEYS 属性)。
|
||||
|
||||
返回:
|
||||
指纹 dict,键为配置项名,值为对应配置值。
|
||||
"""
|
||||
return {k: getattr(config, k) for k in _STRUCTURAL_KEYS + _DECISION_KEYS}
|
||||
|
||||
|
||||
def check_fingerprint(saved: dict[str, Any], config: Any) -> tuple[list[str], list[str]]:
|
||||
"""比对保存的指纹与当前配置。返回 (结构性不一致项, 决策性不一致项)。
|
||||
|
||||
参数:
|
||||
saved: checkpoint 中保存的 config_fingerprint。
|
||||
config: 当前训练配置对象。
|
||||
|
||||
返回:
|
||||
(structural, decision) 两个不一致项名列表。
|
||||
|
||||
关键实现细节:
|
||||
结构性不一致(batch_size/min_class_per_batch/epochs/diag_size/val_size/
|
||||
batch_correct_ratio)→ 调用方应拒绝 resume;决策性不一致 → 仅告警放行。
|
||||
"""
|
||||
cur = compute_fingerprint(config)
|
||||
structural = [k for k in _STRUCTURAL_KEYS if saved.get(k) != cur[k]]
|
||||
decision = [k for k in _DECISION_KEYS if saved.get(k) != cur[k]]
|
||||
return structural, decision
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 读写 checkpoint
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_checkpoint(
|
||||
workspace_dir: Path,
|
||||
*,
|
||||
state: Any,
|
||||
epoch: int,
|
||||
step_completed: int,
|
||||
phase: str,
|
||||
global_step: int,
|
||||
total_steps: int,
|
||||
version_snapshot: dict[str, str],
|
||||
epoch_batches: list[list[str]],
|
||||
config: Any,
|
||||
) -> None:
|
||||
"""原子写 checkpoint.json(.tmp 再 os.replace)。
|
||||
|
||||
参数:
|
||||
workspace_dir: workspace 目录,checkpoint.json 写入其下。
|
||||
state: _TrainState 实例,交由 serialize_state 序列化。
|
||||
epoch: 当前 epoch 序号。
|
||||
step_completed: 本 epoch 内已完成的 step 数。
|
||||
phase: 续训阶段标识(如 "in_epoch")。
|
||||
global_step: 全局 step 序号。
|
||||
total_steps: 全局总 step 数。
|
||||
version_snapshot: skills/prompts 版本快照。
|
||||
epoch_batches: 本 epoch 的 batch 划分(unit_id 列表的列表,孪生对折叠为
|
||||
单个 unit_id;纯非 AR 下 unit_id==question_id)。
|
||||
config: 训练配置对象,用于计算 config_fingerprint。
|
||||
|
||||
关键实现细节:
|
||||
先写 checkpoint.json.tmp 再 os.replace,保证 checkpoint 不被写一半的中断破坏。
|
||||
"""
|
||||
payload = {
|
||||
"schema_version": CHECKPOINT_SCHEMA_VERSION,
|
||||
"progress": {
|
||||
"epoch": epoch,
|
||||
"step_completed": step_completed,
|
||||
"phase": phase,
|
||||
"global_step": global_step,
|
||||
"total_steps": total_steps,
|
||||
},
|
||||
"version_snapshot": version_snapshot,
|
||||
"epoch_batches": epoch_batches,
|
||||
"config_fingerprint": compute_fingerprint(config),
|
||||
"state": serialize_state(state),
|
||||
}
|
||||
path = workspace_dir / "checkpoint.json"
|
||||
tmp = path.with_name("checkpoint.json.tmp")
|
||||
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def load_checkpoint(workspace_dir: Path) -> dict[str, Any] | None:
|
||||
"""读 checkpoint.json,不存在返回 None。
|
||||
|
||||
参数:
|
||||
workspace_dir: workspace 目录。
|
||||
|
||||
返回:
|
||||
checkpoint payload dict;checkpoint.json 不存在时返回 None。
|
||||
"""
|
||||
path = workspace_dir / "checkpoint.json"
|
||||
if not path.exists():
|
||||
return None
|
||||
return json.loads(path.read_text())
|
||||
@@ -0,0 +1,450 @@
|
||||
"""运行配置:RunConfig frozen dataclass 与 YAML + CLI + .env 三层加载。
|
||||
|
||||
三层合并优先级:CLI > .env > YAML(遵循 CLAUDE.md §4.5 配置管理规范)。
|
||||
- YAML:科研实验配置(会在实验中反复扫动的参数),存放于 config/ 下。
|
||||
- .env:工程配置(少变路径如 workspace_dir、store_dir),通过环境变量注入。
|
||||
- CLI:单次临时覆盖。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
_VALID_MODES = {"infer", "train", "diagnose", "evolve", "eval", "promote"}
|
||||
_VALID_SKILL_MODES = {"auto", "manual", "none"}
|
||||
_VALID_SKILL_UPDATE_MODES = {"patch", "rewrite"}
|
||||
_VALID_POOL_SPLIT_MODES = {"global", "per_category"}
|
||||
_PATH_FIELDS = {"workspace_dir", "store_dir"}
|
||||
|
||||
# Video-MME 的任务类型数量:验证池每类至少保底 eval_min_per_class 题,共 12 类。
|
||||
_VIDEO_MME_TASK_TYPE_COUNT = 12
|
||||
|
||||
# .env 工程配置字段映射(环境变量名 → RunConfig 字段名)。
|
||||
# 仅路径类工程配置走 .env,科研实验参数走 YAML。
|
||||
_ENV_FIELD_MAP: dict[str, str] = {
|
||||
"HARNESS_WORKSPACE_DIR": "workspace_dir",
|
||||
"HARNESS_STORE_DIR": "store_dir",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RunConfig:
|
||||
"""实验运行配置,所有参数的唯一归口。
|
||||
|
||||
frozen=True 确保配置在创建后不可变,防止运行中被意外修改。
|
||||
三层合并优先级:CLI > .env > YAML。
|
||||
|
||||
字段:
|
||||
workspace_dir: Workspace 根目录。
|
||||
store_dir: Store 根目录。
|
||||
mode: 运行模式,"infer" / "train" / "diagnose" / "evolve" / "eval" / "promote"。
|
||||
concurrency: 并行 worker 数。
|
||||
max_steps: AgentLoop 单题最大步数。
|
||||
skill_mode: Skill 加载模式,"auto" / "manual" / "none"。
|
||||
n_samples: 题目截取数,0 表示全量。
|
||||
questions: 题目在 questions/ 下的相对路径。
|
||||
skills_version: Skills 版本号。
|
||||
prompts_version: Prompts 版本号。
|
||||
epochs: 训练轮数。
|
||||
diag_size: 诊断池题目数。
|
||||
diag_correct_ratio: 诊断池中正确题目占比。
|
||||
val_size: 验证池题目数。
|
||||
val_correct_ratio: 验证池中正确题目占比。
|
||||
edit_budget_start: 编辑预算前期上限。
|
||||
edit_budget_end: 编辑预算后期下限。
|
||||
batch_size: mini-batch 单批题目数。
|
||||
min_class_per_batch: 单批中每个任务类型至少保留的题目数(< batch_size)。
|
||||
eval_min_per_class: 验证池中每个任务类型至少保底的题目数。
|
||||
trainable_min_units: 可训练性预检:每题型 diag+val 单元数下限,低于则剔除该题型。
|
||||
early_stop_patience: 全局 best 连续未提升的容忍轮数,达到即早停。
|
||||
test_size: held-out 测试池题目数。
|
||||
use_slow_momentum: 是否启用快慢双速进化中的慢速 momentum 更新。
|
||||
gate_e_confirm: CE-Gate CONFIRMED 接受的 e 值门槛(1/alpha,Ville 界假阳率 alpha)。
|
||||
gate_e_provisional: 题尽暂定接受门 + futility 提前止损的代数界。
|
||||
gate_w_net_min: 题尽暂定接受要求的最小净胜数(win - loss)。
|
||||
gate_delta_min: 最小点估计效应量下限(承接旧 margin 语义)。
|
||||
gate_lambda_dir: Wald 方向拒绝的对数似然比阈值(必须为负)。
|
||||
gate_e_rollback: 试用期对称回滚门(回滚 e 值门槛)。
|
||||
gate_n_max: 单次 gate 消耗的题数上限。
|
||||
gate_p_low: 信息量阶梯 p-hat 保留区间下界(剔除必错零信息题)。
|
||||
gate_p_high: 信息量阶梯 p-hat 保留区间上界(剔除必对零信息题)。
|
||||
gate_probe_quota: 冷启动探针集比例(全错题中插尾的比例)。
|
||||
gate_gamma_decay: 逐题正确率估计 p-hat 的 EMA 衰减系数。
|
||||
gate_cooldown_steps: 回滚后该题型跳过进化的冷却 step 数。
|
||||
gate_guard_err: gate 内累计 INFRA 错误率护栏。
|
||||
skill_update_mode: skill 进化模式,"patch"(局部 edit)/ "rewrite"(整篇重写)。
|
||||
appendix_consolidate_threshold: appendix note 条数达此值触发 LLM consolidation。
|
||||
run_id: diagnose/evolve 模式要分析的运行 ID,默认空字符串。
|
||||
batch_correct_ratio: 单批中正确题目占比,范围 [0, 1)。
|
||||
momentum_samples: 慢速 momentum 更新时从诊断池采样的题目数,必须 >= 1。
|
||||
seed: fresh 训练的种子名(对应 seed.json),默认 "initial"。
|
||||
version: eval/promote 模式指定的 store 版本号(如 "v3")。
|
||||
resume: train 模式是否从已有 checkpoint 续训。
|
||||
fresh: train 模式是否从种子全新开始。
|
||||
task_types: 限定参与的任务类型子集,None 表示全部。
|
||||
pool_split_mode: 池划分策略,"global"(全局统一划分)/ "per_category"(按类别独立划分)。
|
||||
train_ratio: 训练集占比,范围 (0, 1)。
|
||||
test_questions: 测试题目集路径(相对路径)。
|
||||
run_holdout_eval: 是否在 epoch 末执行 held-out 四向评估,默认 True。
|
||||
"""
|
||||
|
||||
# ── 必填字段(无默认值,来自 YAML 或 CLI) ──
|
||||
workspace_dir: Path
|
||||
store_dir: Path
|
||||
mode: str
|
||||
concurrency: int
|
||||
max_steps: int
|
||||
skill_mode: str
|
||||
n_samples: int
|
||||
questions: str
|
||||
skills_version: str
|
||||
prompts_version: str
|
||||
epochs: int
|
||||
diag_size: int
|
||||
diag_correct_ratio: float
|
||||
val_size: int
|
||||
val_correct_ratio: float
|
||||
edit_budget_start: int
|
||||
edit_budget_end: int
|
||||
batch_size: int
|
||||
min_class_per_batch: int
|
||||
eval_min_per_class: int
|
||||
trainable_min_units: int
|
||||
early_stop_patience: int
|
||||
test_size: int
|
||||
use_slow_momentum: bool
|
||||
gate_e_confirm: float
|
||||
gate_e_provisional: float
|
||||
gate_w_net_min: int
|
||||
gate_delta_min: float
|
||||
gate_lambda_dir: float
|
||||
gate_e_rollback: float
|
||||
gate_n_max: int
|
||||
gate_p_low: float
|
||||
gate_p_high: float
|
||||
gate_probe_quota: float
|
||||
gate_gamma_decay: float
|
||||
gate_cooldown_steps: int
|
||||
gate_guard_err: float
|
||||
skill_update_mode: str
|
||||
appendix_consolidate_threshold: int
|
||||
|
||||
# ── 有默认值的字段(通常由 CLI 传入或可选) ──
|
||||
run_id: str = ""
|
||||
batch_correct_ratio: float = 0.5
|
||||
momentum_samples: int = 20
|
||||
seed: str = "initial"
|
||||
version: str = ""
|
||||
resume: bool = False
|
||||
fresh: bool = False
|
||||
task_types: tuple[str, ...] | None = None
|
||||
pool_split_mode: str = "global"
|
||||
train_ratio: float = 0.667
|
||||
test_questions: str = "benchmarks/Video-MME"
|
||||
run_holdout_eval: bool = True
|
||||
|
||||
|
||||
def _validate(config: RunConfig) -> None:
|
||||
"""校验 RunConfig 全部字段约束。
|
||||
|
||||
六层校验链:mode → 基础标量 → 编辑预算 → mini-batch → gate 阈值 → gate 阶梯。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 任一字段值不合法。
|
||||
"""
|
||||
_validate_mode(config)
|
||||
_validate_mode_deps(config)
|
||||
_validate_basic(config)
|
||||
_validate_edit_budget(config)
|
||||
_validate_minibatch(config)
|
||||
_validate_gate(config)
|
||||
|
||||
|
||||
def _validate_mode(config: RunConfig) -> None:
|
||||
"""校验运行模式枚举合法性。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: mode 值不在合法集合中。
|
||||
"""
|
||||
if config.mode not in _VALID_MODES:
|
||||
raise ValueError(f"mode 必须为 {_VALID_MODES} 之一,实际: {config.mode!r}")
|
||||
|
||||
|
||||
def _validate_mode_deps(config: RunConfig) -> None:
|
||||
"""校验各运行模式的依赖字段(run_id、version)。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 模式依赖字段缺失。
|
||||
"""
|
||||
if config.mode in ("diagnose", "evolve") and not config.run_id:
|
||||
raise ValueError(f"mode 为 {config.mode!r} 时必须提供 run_id。")
|
||||
if config.mode in ("eval", "promote") and not config.version:
|
||||
raise ValueError(f"mode 为 {config.mode!r} 时必须提供 --version。")
|
||||
if config.mode == "promote" and not config.run_id:
|
||||
raise ValueError("promote 必须提供 --run-id(指定 canonical eval run)。")
|
||||
_validate_train_run_id(config)
|
||||
|
||||
|
||||
def _validate_train_run_id(config: RunConfig) -> None:
|
||||
"""校验 train 模式非 resume/fresh 时必须提供 run_id。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: train 模式既非 resume 也非 fresh 且缺少 run_id。
|
||||
"""
|
||||
if config.mode != "train":
|
||||
return
|
||||
if config.resume or config.fresh:
|
||||
return
|
||||
if not config.run_id:
|
||||
raise ValueError("train 非 resume/fresh 时必须提供 run_id(旧式基线 run)。")
|
||||
|
||||
|
||||
def _validate_basic(config: RunConfig) -> None:
|
||||
"""校验基础标量字段:枚举合法性与正整数约束。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 任一基础字段值不合法。
|
||||
"""
|
||||
if config.skill_mode not in _VALID_SKILL_MODES:
|
||||
raise ValueError(
|
||||
f"skill_mode 必须为 {_VALID_SKILL_MODES} 之一,实际: {config.skill_mode!r}"
|
||||
)
|
||||
if config.concurrency <= 0:
|
||||
raise ValueError(f"concurrency 必须 > 0,实际: {config.concurrency}")
|
||||
if config.max_steps <= 0:
|
||||
raise ValueError(f"max_steps 必须 > 0,实际: {config.max_steps}")
|
||||
if config.n_samples < 0:
|
||||
raise ValueError(f"n_samples 必须 >= 0,实际: {config.n_samples}")
|
||||
if config.epochs <= 0:
|
||||
raise ValueError(f"epochs 必须 > 0,实际: {config.epochs}")
|
||||
if config.skill_update_mode not in _VALID_SKILL_UPDATE_MODES:
|
||||
raise ValueError(
|
||||
f"skill_update_mode 必须为 {_VALID_SKILL_UPDATE_MODES} 之一,"
|
||||
f"实际: {config.skill_update_mode!r}"
|
||||
)
|
||||
if config.appendix_consolidate_threshold < 1:
|
||||
raise ValueError(
|
||||
f"appendix_consolidate_threshold 必须 >= 1,"
|
||||
f"实际: {config.appendix_consolidate_threshold}"
|
||||
)
|
||||
if config.pool_split_mode not in _VALID_POOL_SPLIT_MODES:
|
||||
raise ValueError(
|
||||
f"pool_split_mode 必须为 {_VALID_POOL_SPLIT_MODES} 之一,"
|
||||
f"实际: {config.pool_split_mode!r}"
|
||||
)
|
||||
if not (0 < config.train_ratio < 1):
|
||||
raise ValueError(f"train_ratio 必须在 (0, 1) 内,实际: {config.train_ratio}")
|
||||
|
||||
|
||||
def _validate_edit_budget(config: RunConfig) -> None:
|
||||
"""校验编辑预算退火的前期/后期上限约束。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: edit_budget_start < edit_budget_end,或 end <= 0。
|
||||
"""
|
||||
if config.edit_budget_start < config.edit_budget_end:
|
||||
raise ValueError(
|
||||
f"edit_budget_start({config.edit_budget_start}) 必须 >= "
|
||||
f"edit_budget_end({config.edit_budget_end})"
|
||||
)
|
||||
if config.edit_budget_end <= 0:
|
||||
raise ValueError(f"edit_budget_end 必须 > 0,实际: {config.edit_budget_end}")
|
||||
|
||||
|
||||
def _validate_minibatch(config: RunConfig) -> None:
|
||||
"""校验 mini-batch 自进化闭环参数约束。
|
||||
|
||||
参数:
|
||||
config: 待校验的 RunConfig 配置对象。
|
||||
|
||||
异常:
|
||||
ValueError: 任一约束被违反。
|
||||
|
||||
关键实现细节:
|
||||
pool_split_mode != "per_category" 时,val_size 必须 >= eval_min_per_class *
|
||||
_VIDEO_MME_TASK_TYPE_COUNT,保证验证池能为 Video-MME 的全部 12 个任务类型
|
||||
各保底 eval_min_per_class 题。per_category 模式下跳过此硬编码 12 类保底检查。
|
||||
"""
|
||||
if config.batch_size <= 0:
|
||||
raise ValueError(f"batch_size 必须 > 0,实际: {config.batch_size}")
|
||||
if not (1 <= config.min_class_per_batch < config.batch_size):
|
||||
raise ValueError(
|
||||
f"min_class_per_batch 必须满足 1 <= 值 < batch_size"
|
||||
f"({config.batch_size}),实际: {config.min_class_per_batch}"
|
||||
)
|
||||
if config.eval_min_per_class < 1:
|
||||
raise ValueError(f"eval_min_per_class 必须 >= 1,实际: {config.eval_min_per_class}")
|
||||
if config.trainable_min_units < 1:
|
||||
raise ValueError(f"trainable_min_units 必须 >= 1,实际: {config.trainable_min_units}")
|
||||
if config.pool_split_mode != "per_category":
|
||||
floor = config.eval_min_per_class * _VIDEO_MME_TASK_TYPE_COUNT
|
||||
if config.val_size < floor:
|
||||
raise ValueError(
|
||||
f"val_size 必须 >= eval_min_per_class * {_VIDEO_MME_TASK_TYPE_COUNT}"
|
||||
f"(={floor}):Video-MME 共 {_VIDEO_MME_TASK_TYPE_COUNT} 个任务类型,"
|
||||
f"每类需 eval_min_per_class 题保底,故验证池下限为 {floor},"
|
||||
f"实际: {config.val_size}"
|
||||
)
|
||||
if config.early_stop_patience <= 0:
|
||||
raise ValueError(f"early_stop_patience 必须 > 0,实际: {config.early_stop_patience}")
|
||||
if config.test_size <= 0:
|
||||
raise ValueError(f"test_size 必须 > 0,实际: {config.test_size}")
|
||||
if not (0 <= config.batch_correct_ratio < 1):
|
||||
raise ValueError(
|
||||
f"batch_correct_ratio 必须满足 0 <= 值 < 1,实际: {config.batch_correct_ratio}"
|
||||
)
|
||||
if config.momentum_samples < 1:
|
||||
raise ValueError(f"momentum_samples 必须 >= 1,实际: {config.momentum_samples}")
|
||||
|
||||
|
||||
def _validate_gate(config: RunConfig) -> None:
|
||||
"""校验 CE-Gate 全部参数:判据阈值 + 信息量阶梯。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 任一 gate 参数不合法。
|
||||
"""
|
||||
_validate_gate_thresholds(config)
|
||||
_validate_gate_ladder(config)
|
||||
|
||||
|
||||
def _validate_gate_thresholds(config: RunConfig) -> None:
|
||||
"""校验 CE-Gate 判据阈值参数(e 值、净胜数、效应量、方向拒绝)。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 任一阈值参数不合法。
|
||||
"""
|
||||
if config.gate_e_confirm <= 1:
|
||||
raise ValueError(f"gate_e_confirm 必须 > 1,实际: {config.gate_e_confirm}")
|
||||
if not (1 < config.gate_e_provisional <= config.gate_e_confirm):
|
||||
raise ValueError(
|
||||
f"gate_e_provisional 必须在 (1, gate_e_confirm] 内,实际: {config.gate_e_provisional}"
|
||||
)
|
||||
if config.gate_e_rollback <= 1:
|
||||
raise ValueError(f"gate_e_rollback 必须 > 1,实际: {config.gate_e_rollback}")
|
||||
if config.gate_w_net_min < 1:
|
||||
raise ValueError(f"gate_w_net_min 必须 >= 1,实际: {config.gate_w_net_min}")
|
||||
if config.gate_delta_min < 0:
|
||||
raise ValueError(f"gate_delta_min 必须 >= 0,实际: {config.gate_delta_min}")
|
||||
if config.gate_lambda_dir >= 0:
|
||||
raise ValueError(f"gate_lambda_dir 必须 < 0,实际: {config.gate_lambda_dir}")
|
||||
|
||||
|
||||
def _validate_gate_ladder(config: RunConfig) -> None:
|
||||
"""校验 CE-Gate 信息量阶梯参数。
|
||||
|
||||
参数:
|
||||
config: 待校验的配置实例。
|
||||
|
||||
异常:
|
||||
ValueError: 任一阶梯参数不合法。
|
||||
"""
|
||||
if config.gate_n_max <= 0:
|
||||
raise ValueError(f"需 gate_n_max > 0,实际: n_max={config.gate_n_max}")
|
||||
if not (0 <= config.gate_p_low < config.gate_p_high <= 1):
|
||||
raise ValueError(
|
||||
f"需 0 <= gate_p_low < gate_p_high <= 1,"
|
||||
f"实际: [{config.gate_p_low}, {config.gate_p_high}]"
|
||||
)
|
||||
if not (0 <= config.gate_probe_quota <= 1):
|
||||
raise ValueError(f"gate_probe_quota 须在 [0,1],实际: {config.gate_probe_quota}")
|
||||
if not (0 < config.gate_gamma_decay < 1):
|
||||
raise ValueError(f"gate_gamma_decay 须在 (0,1),实际: {config.gate_gamma_decay}")
|
||||
if config.gate_cooldown_steps < 1:
|
||||
raise ValueError(f"gate_cooldown_steps 必须 >= 1,实际: {config.gate_cooldown_steps}")
|
||||
if not (0 < config.gate_guard_err < 1):
|
||||
raise ValueError(f"gate_guard_err 须在 (0,1),实际: {config.gate_guard_err}")
|
||||
|
||||
|
||||
def _apply_env_var_overrides(data: dict) -> None:
|
||||
"""从环境变量覆盖路径字段(原地修改)。
|
||||
|
||||
.env 文件由入口脚本 load_dotenv 加载到环境变量,本函数仅从 os.environ 读取。
|
||||
仅覆盖 _ENV_FIELD_MAP 中声明的工程配置字段(workspace_dir、store_dir)。
|
||||
|
||||
参数:
|
||||
data: 待覆盖的配置字典。
|
||||
"""
|
||||
for env_key, field_name in _ENV_FIELD_MAP.items():
|
||||
env_val = os.environ.get(env_key)
|
||||
if env_val is not None:
|
||||
data[field_name] = env_val
|
||||
|
||||
|
||||
def load_config(
|
||||
yaml_path: Path,
|
||||
cli_overrides: dict[str, object] | None = None,
|
||||
) -> RunConfig:
|
||||
"""从 YAML 加载配置,叠加 .env 和 CLI 覆盖层后构造 RunConfig。
|
||||
|
||||
三层合并优先级:CLI > .env > YAML。
|
||||
|
||||
参数:
|
||||
yaml_path: YAML 配置文件路径,需包含 ``harness`` 段。
|
||||
cli_overrides: CLI 参数字典,值为 None 表示未传入(不覆盖)。
|
||||
|
||||
返回:
|
||||
构造并校验后的 RunConfig 实例。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: YAML 文件不存在。
|
||||
ValueError: 校验失败。
|
||||
"""
|
||||
# Phase 1: 加载 YAML 基础层
|
||||
with open(yaml_path, encoding="utf-8") as f:
|
||||
raw: dict = yaml.safe_load(f)
|
||||
|
||||
# 支持嵌套 harness 段和扁平 YAML 两种格式
|
||||
yaml_data: dict = raw.get("harness", raw)
|
||||
|
||||
# Phase 2: .env 覆盖层(仅工程配置字段)
|
||||
_apply_env_var_overrides(yaml_data)
|
||||
|
||||
# Phase 3: CLI 覆盖层(最高优先级)
|
||||
valid_fields = {f.name for f in dataclasses.fields(RunConfig)}
|
||||
if cli_overrides:
|
||||
for key, value in cli_overrides.items():
|
||||
if value is not None and key in valid_fields:
|
||||
yaml_data[key] = value
|
||||
|
||||
# Phase 4: 类型转换 — 路径字段转 Path
|
||||
for field_name in _PATH_FIELDS:
|
||||
if field_name in yaml_data:
|
||||
yaml_data[field_name] = Path(yaml_data[field_name])
|
||||
|
||||
# Phase 4b: 类型转换 — task_types list → tuple
|
||||
if "task_types" in yaml_data and yaml_data["task_types"] is not None:
|
||||
yaml_data["task_types"] = tuple(yaml_data["task_types"])
|
||||
|
||||
# Phase 5: 构造并校验
|
||||
config = RunConfig(**{k: v for k, v in yaml_data.items() if k in valid_fields})
|
||||
_validate(config)
|
||||
return config
|
||||
@@ -0,0 +1,183 @@
|
||||
"""按 video_id 懒加载 InferenceDeps 并路由 dispatch/prompt_builder。
|
||||
|
||||
每个视频的 TreeIndex、TreeEnvironment、SkillRegistry 等重量级对象
|
||||
只在首次访问时构建并缓存,后续同视频的请求直接复用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.factory import InferenceDeps, build_inference_deps
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from app.ports import EmbeddingProvider, OCRProvider
|
||||
from core.protocols import LLMProvider, VLMProvider
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
|
||||
class InferenceDepsRouter:
|
||||
"""按 video_id 懒加载 InferenceDeps 并路由工具调度和 prompt 构建。
|
||||
|
||||
职责:
|
||||
1. 维护 question_id → video_id 的映射表(由 prompt_builder 自动注册)。
|
||||
2. 按 (video_id, skills_dir, prompts_dir) 三元组缓存 InferenceDeps。
|
||||
3. 提供 create_dispatch / create_prompt_builder 工厂方法,
|
||||
返回的闭包符合 ToolDispatchFn / PromptBuilderFn Protocol。
|
||||
|
||||
参数:
|
||||
store_dir: store 根目录。
|
||||
embed_provider: 嵌入端口实例。
|
||||
llm: LLM 端口实例。
|
||||
vlm: VLM 端口实例。
|
||||
ocr: OCR 端口实例(None 不启用)。
|
||||
default_prompts_dir: 默认 prompt 文件目录。
|
||||
default_skills_dir: 默认 skill 文件目录(None 则不加载 skill)。
|
||||
skill_mode: skill 模式("auto"/"manual"/"none")。
|
||||
verify_vision: observe_frame 是否执行验证轮。
|
||||
anchor: view_node 是否启用行号锚模式。
|
||||
assemble_mode: 锚模式装配形态。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
store_dir: Path,
|
||||
embed_provider: EmbeddingProvider,
|
||||
llm: LLMProvider,
|
||||
vlm: VLMProvider,
|
||||
ocr: OCRProvider | None,
|
||||
default_prompts_dir: Path,
|
||||
default_skills_dir: Path | None,
|
||||
skill_mode: str,
|
||||
verify_vision: bool,
|
||||
anchor: bool,
|
||||
assemble_mode: str,
|
||||
) -> None:
|
||||
self._store_dir = store_dir
|
||||
self._embed = embed_provider
|
||||
self._llm = llm
|
||||
self._vlm = vlm
|
||||
self._ocr = ocr
|
||||
self._default_prompts_dir = default_prompts_dir
|
||||
self._default_skills_dir = default_skills_dir
|
||||
self._skill_mode = skill_mode
|
||||
self._verify_vision = verify_vision
|
||||
self._anchor = anchor
|
||||
self._assemble_mode = assemble_mode
|
||||
self._deps_cache: dict[tuple[str, str, str], InferenceDeps] = {}
|
||||
self._qid_to_vid: dict[str, str] = {}
|
||||
|
||||
def create_dispatch(self, *, skills_dir: Path | None = None) -> Any:
|
||||
"""创建工具调度闭包,按 session_id 路由到对应视频的 InferenceDeps。
|
||||
|
||||
参数:
|
||||
skills_dir: skill 文件目录覆盖(None 使用默认值)。
|
||||
|
||||
返回:
|
||||
符合 ToolDispatchFn 签名的 async 闭包。
|
||||
"""
|
||||
effective_skills = skills_dir or self._default_skills_dir
|
||||
|
||||
async def _dispatch(
|
||||
tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
|
||||
) -> str:
|
||||
"""按 session_id 查找视频 → 获取缓存 deps → 委托执行。"""
|
||||
session_id = context.get("session_id")
|
||||
if not session_id or session_id not in self._qid_to_vid:
|
||||
raise KeyError(
|
||||
f"未注册的 session_id={session_id!r},已注册 {len(self._qid_to_vid)} 条映射"
|
||||
)
|
||||
video_id = self._qid_to_vid[session_id]
|
||||
deps = self._ensure_deps(video_id, effective_skills, self._default_prompts_dir)
|
||||
return await deps.tool_dispatch_fn(tool_name, args, context=context)
|
||||
|
||||
return _dispatch
|
||||
|
||||
def create_prompt_builder(
|
||||
self,
|
||||
*,
|
||||
skills_dir: Path | None = None,
|
||||
prompts_dir: Path | None = None,
|
||||
) -> Any:
|
||||
"""创建 prompt 构建闭包,自动注册 qid→vid 映射并路由到对应视频的 deps。
|
||||
|
||||
参数:
|
||||
skills_dir: skill 文件目录覆盖(None 使用默认值)。
|
||||
prompts_dir: prompt 文件目录覆盖(None 使用默认值)。
|
||||
|
||||
返回:
|
||||
符合 PromptBuilderFn 签名的闭包。
|
||||
"""
|
||||
effective_skills = skills_dir or self._default_skills_dir
|
||||
effective_prompts = prompts_dir or self._default_prompts_dir
|
||||
|
||||
def _builder(qa: GeneratedQuestion) -> tuple[str, str]:
|
||||
"""注册 qid→vid 映射 → 获取缓存 deps → 委托构建 prompt。"""
|
||||
self._qid_to_vid[qa.question_id] = qa.video_id
|
||||
deps = self._ensure_deps(qa.video_id, effective_skills, effective_prompts)
|
||||
return deps.prompt_builder(qa)
|
||||
|
||||
return _builder
|
||||
|
||||
def _ensure_deps(
|
||||
self,
|
||||
video_id: str,
|
||||
skills_dir: Path | None,
|
||||
prompts_dir: Path,
|
||||
) -> InferenceDeps:
|
||||
"""按 (video_id, skills_dir, prompts_dir) 三元组缓存 InferenceDeps。
|
||||
|
||||
参数:
|
||||
video_id: 视频标识。
|
||||
skills_dir: skill 文件目录。
|
||||
prompts_dir: prompt 文件目录。
|
||||
|
||||
返回:
|
||||
缓存命中或新建的 InferenceDeps 实例。
|
||||
"""
|
||||
key = (video_id, str(skills_dir), str(prompts_dir))
|
||||
if key not in self._deps_cache:
|
||||
self._deps_cache[key] = self._build_deps(video_id, skills_dir, prompts_dir)
|
||||
logger.debug("InferenceDeps 已缓存: video_id={}", video_id)
|
||||
return self._deps_cache[key]
|
||||
|
||||
def _build_deps(
|
||||
self,
|
||||
video_id: str,
|
||||
skills_dir: Path | None,
|
||||
prompts_dir: Path,
|
||||
) -> InferenceDeps:
|
||||
"""调用 build_inference_deps 构建 InferenceDeps 实例。
|
||||
|
||||
参数:
|
||||
video_id: 视频标识。
|
||||
skills_dir: skill 文件目录。
|
||||
prompts_dir: prompt 文件目录。
|
||||
|
||||
返回:
|
||||
新建的 InferenceDeps 实例。
|
||||
"""
|
||||
return build_inference_deps(
|
||||
store_dir=self._store_dir,
|
||||
video_id=video_id,
|
||||
prompts_dir=prompts_dir,
|
||||
skills_dir=skills_dir,
|
||||
skill_mode=self._skill_mode,
|
||||
embed_provider=self._embed,
|
||||
llm=self._llm,
|
||||
vlm=self._vlm,
|
||||
ocr=self._ocr,
|
||||
verify_vision=self._verify_vision,
|
||||
anchor=self._anchor,
|
||||
assemble_mode=self._assemble_mode,
|
||||
)
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""清空 deps 缓存和 qid→vid 映射表。"""
|
||||
self._deps_cache.clear()
|
||||
self._qid_to_vid.clear()
|
||||
@@ -0,0 +1,155 @@
|
||||
"""推理依赖工厂 — 组装一次推理所需的全套依赖。
|
||||
|
||||
将 TreeIndex 加载、TreeEnvironment 构建、SkillRegistry 发现、
|
||||
SearchToolDispatcher 装配、PromptManager 初始化等步骤封装为
|
||||
单一工厂函数 ``build_inference_deps``,返回不可变的 ``InferenceDeps``。
|
||||
|
||||
调用方(runner / inference)只需传入配置参数,无需了解内部装配逻辑。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.search.prompt import PromptManager
|
||||
from app.search.skills import discover_skills
|
||||
from app.search.tools import SearchToolDispatcher
|
||||
from app.tree.environment import TreeEnvironment
|
||||
from app.tree.index import TreeIndex
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from app.ports import EmbeddingProvider, OCRProvider
|
||||
from core.protocols import LLMProvider, VLMProvider
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InferenceDeps:
|
||||
"""跑一次推理所需的全套依赖(不含 HarnessLog,其生命周期由调用方管理)。
|
||||
|
||||
属性:
|
||||
llm: LLM 端口实例。
|
||||
tool_dispatch_fn: SearchToolDispatcher.dispatch 的绑定方法。
|
||||
prompt_builder: (GeneratedQuestion) -> (system_prompt, user_prompt)。
|
||||
"""
|
||||
|
||||
llm: LLMProvider
|
||||
tool_dispatch_fn: Callable[..., Any]
|
||||
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]]
|
||||
|
||||
|
||||
def build_inference_deps(
|
||||
*,
|
||||
store_dir: Path,
|
||||
video_id: str,
|
||||
prompts_dir: Path,
|
||||
skills_dir: Path | None,
|
||||
skill_mode: str,
|
||||
embed_provider: EmbeddingProvider,
|
||||
llm: LLMProvider,
|
||||
vlm: VLMProvider,
|
||||
ocr: OCRProvider | None,
|
||||
verify_vision: bool,
|
||||
anchor: bool,
|
||||
assemble_mode: str,
|
||||
) -> InferenceDeps:
|
||||
"""组装一次推理所需的全套依赖。
|
||||
|
||||
参数:
|
||||
store_dir: store 根目录(包含 videos/{video_id}/tree.json)。
|
||||
video_id: 视频标识。
|
||||
prompts_dir: prompt 文件目录。
|
||||
skills_dir: skill 文件目录(None 则不加载 skill)。
|
||||
skill_mode: skill 模式("auto"/"manual"/"none")。
|
||||
embed_provider: 嵌入端口实例。
|
||||
llm: LLM 端口实例。
|
||||
vlm: VLM 端口实例。
|
||||
ocr: OCR 端口实例(None 不启用)。
|
||||
verify_vision: observe_frame 是否执行验证轮。
|
||||
anchor: view_node 是否启用行号锚模式。
|
||||
assemble_mode: 锚模式装配形态。
|
||||
|
||||
返回:
|
||||
InferenceDeps 实例。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: tree.json 不存在。
|
||||
"""
|
||||
# Phase 1: 加载 TreeIndex
|
||||
tree_path = store_dir / "videos" / video_id / "tree.json"
|
||||
if not tree_path.exists():
|
||||
raise FileNotFoundError(f"树索引文件不存在: {tree_path}")
|
||||
tree_index = TreeIndex.load_json(str(tree_path))
|
||||
logger.info("已加载 TreeIndex: video_id={}, L1 节点数={}", video_id, len(tree_index.roots))
|
||||
|
||||
# Phase 2: 构建 TreeEnvironment
|
||||
frames_dir = store_dir / "videos" / video_id / "frames"
|
||||
env = TreeEnvironment(index=tree_index, frames_dir=frames_dir)
|
||||
|
||||
# Phase 3: 构建 SkillRegistry
|
||||
skills = None
|
||||
always_skills_text = ""
|
||||
task_skill_map: dict[str, str] = {}
|
||||
catalog_text = ""
|
||||
if skills_dir is not None:
|
||||
always_skills_text, task_skill_map, catalog_text, skills = discover_skills(skills_dir)
|
||||
logger.info(
|
||||
"已发现 skills: always={} 字符, task_map={} 项",
|
||||
len(always_skills_text),
|
||||
len(task_skill_map),
|
||||
)
|
||||
|
||||
# Phase 4: 构建 SearchToolDispatcher
|
||||
dispatcher = SearchToolDispatcher(
|
||||
env,
|
||||
tool_llm=llm,
|
||||
vlm=vlm,
|
||||
ocr=ocr,
|
||||
prompts_dir=prompts_dir,
|
||||
skills=skills,
|
||||
embed_fn=embed_provider.embed,
|
||||
verify_vision=verify_vision,
|
||||
anchor=anchor,
|
||||
assemble_mode=assemble_mode,
|
||||
)
|
||||
|
||||
# Phase 5: 构建 PromptManager + _prompt_builder 闭包
|
||||
pm = PromptManager(prompts_dir)
|
||||
l1_ids = [root.id for root in tree_index.roots]
|
||||
|
||||
def _prompt_builder(qa: GeneratedQuestion) -> tuple[str, str]:
|
||||
"""为单条题目生成 (system_prompt, user_prompt)。
|
||||
|
||||
参数:
|
||||
qa: 生成的题目实例。
|
||||
|
||||
返回:
|
||||
(system_prompt, user_prompt) 二元组。
|
||||
"""
|
||||
system = pm.build_inference_prompt(
|
||||
skill_mode,
|
||||
qa.task_type,
|
||||
always_skills_text,
|
||||
task_skill_map,
|
||||
catalog_text,
|
||||
)
|
||||
user = pm.format_user_prompt(
|
||||
qa.question,
|
||||
list(qa.options),
|
||||
l1_ids,
|
||||
qa.task_type,
|
||||
)
|
||||
return system, user
|
||||
|
||||
logger.info("InferenceDeps 组装完成: video_id={}, skill_mode={}", video_id, skill_mode)
|
||||
return InferenceDeps(
|
||||
llm=llm,
|
||||
tool_dispatch_fn=dispatcher.dispatch,
|
||||
prompt_builder=_prompt_builder,
|
||||
)
|
||||
@@ -0,0 +1,404 @@
|
||||
"""CE-Gate 信息量阶梯与基线缓存(unit 粒度,核心算法保真 #5)。
|
||||
|
||||
阶梯(每题型一条):gate 的出题顺序表,键为 **unit_id**(single 题 unit_id
|
||||
等于 question_id,AR pair 折叠为一个单元、unit_id 等于共享 pair_id)。冷启动
|
||||
(FRESH)用种子基线的**单元级**对错两档粗排(错 unit 高优先 2:1 交错 + 全错
|
||||
unit 的 probe_quota 探针插尾);epoch >=1 用非 gate run 观测**折叠成单元观测**后做
|
||||
gamma-EMA 更新 p_hat,按信息量 p_hat(1-p_hat) 降序、剔 p_hat 不在 [p_low, p_high]。
|
||||
单元错 = 该单元任一成员错(AR pair 双向 AND)。防泄露铁律:gate 内 rollout 永不
|
||||
回流 p_hat(调用方以 run_id 含 "_gate_" 过滤观测源),本迁移不改此过滤。
|
||||
|
||||
持久化门控:gate_pools.json 带 schema_version(当前 = 2,unit 键)。旧版无
|
||||
schema_version(v1、qid 键)加载时**直接报错**,拒绝静默混用 qid/unit 键。
|
||||
|
||||
BaselineCache:基线侧单元级对错缓存,键 = (task_type, skill_hash,
|
||||
prompts_version, unit_id) 内容寻址、无显式失效。JSON 持久化到 workspace,
|
||||
供 resume 后合法复用已冻结阶梯上的新鲜 draw。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.question_units import build_units, unit_correctness
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
# gate_pools.json 结构版本。v1(隐式、无此字段)为逐题 qid 键的存量格式;
|
||||
# v2 起改为 unit_id 键。load 时严格校验,不匹配即报错(不静默迁移/混用)。
|
||||
SCHEMA_VERSION = 2
|
||||
|
||||
|
||||
def skill_hash(content: str) -> str:
|
||||
"""对 skill 正文取 sha1 摘要,作缓存键的内容维度。
|
||||
|
||||
参数:
|
||||
content: skill 文件全文(基线侧为解析后生效文件的正文)。
|
||||
|
||||
返回:
|
||||
sha1 十六进制摘要。
|
||||
"""
|
||||
return hashlib.sha1(content.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@dataclass
|
||||
class LadderEntry:
|
||||
"""阶梯单元:题目单元与其估计答对率。
|
||||
|
||||
字段:
|
||||
unit_id: 单元唯一标识(single 等于 question_id,AR pair 等于共享 pair_id)。
|
||||
p_hat: 估计答对率。冷启动为 Beta(1,1) 平滑的单次观测后验均值
|
||||
(错=1/3、对=2/3),此后经 gamma-EMA 更新。
|
||||
"""
|
||||
|
||||
unit_id: str
|
||||
p_hat: float
|
||||
|
||||
|
||||
def build_cold_entries(
|
||||
units: list[QuestionUnit],
|
||||
correctness: dict[str, bool],
|
||||
probe_quota: float,
|
||||
seed: int,
|
||||
) -> list[LadderEntry]:
|
||||
"""冷启动排序(unit 粒度):错 unit 高优先 2:1 交错 + 全错 unit 探针插尾。
|
||||
|
||||
参数:
|
||||
units: 该题型的全部候选单元(已排除 test 池;AR pair 已折叠成单元)。
|
||||
correctness: question_id -> 种子基线是否答对(900 题全量逐题对错)。
|
||||
单元级对错由 unit_correctness(strict=False) 折叠(任一成员错 → 单元错)。
|
||||
probe_quota: 从错 unit 中随机抽出插到梯尾的探针比例(防"解锁新能力"盲区)。
|
||||
seed: 洗牌种子,保证确定性重建。
|
||||
|
||||
返回:
|
||||
排序后的 LadderEntry 列表(键=unit_id;p_hat 用 Beta(1,1) 平滑:错=1/3、
|
||||
对=2/3,与 warm 阶段 gamma-EMA / 信息量排序自然衔接)。
|
||||
|
||||
关键实现细节:
|
||||
与逐题版**同公式、同比例、同顺序**,仅把调度粒度从题换成单元:错 unit、
|
||||
对 unit 各自固定种子洗牌 -> 按 probe_quota 从错 unit 抽探针 -> 剩余按
|
||||
错错对 2:1 交错(一方耗尽后顺排另一方)-> 探针追加尾部。
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
wrong = [u for u in units if not unit_correctness(u, correctness, strict=False)]
|
||||
right = [u for u in units if unit_correctness(u, correctness, strict=False)]
|
||||
rng.shuffle(wrong)
|
||||
rng.shuffle(right)
|
||||
|
||||
n_probe = int(len(wrong) * probe_quota)
|
||||
probes, wrong_main = wrong[:n_probe], wrong[n_probe:]
|
||||
|
||||
interleaved: list[QuestionUnit] = []
|
||||
wi, ri = 0, 0
|
||||
while wi < len(wrong_main) or ri < len(right):
|
||||
for _ in range(2):
|
||||
if wi < len(wrong_main):
|
||||
interleaved.append(wrong_main[wi])
|
||||
wi += 1
|
||||
if ri < len(right):
|
||||
interleaved.append(right[ri])
|
||||
ri += 1
|
||||
interleaved.extend(probes)
|
||||
|
||||
def _p0(u: QuestionUnit) -> float:
|
||||
return 2 / 3 if unit_correctness(u, correctness, strict=False) else 1 / 3
|
||||
|
||||
return [LadderEntry(u.unit_id, _p0(u)) for u in interleaved]
|
||||
|
||||
|
||||
def order_ladder(entries: list[LadderEntry], p_low: float, p_high: float) -> list[LadderEntry]:
|
||||
"""warm 排序:剔 p_hat 不在 [p_low, p_high] 的零信息题,按信息量 p_hat(1-p_hat) 降序。
|
||||
|
||||
参数:
|
||||
entries: 待排序的阶梯单元。
|
||||
p_low / p_high: p_hat 保留区间。
|
||||
|
||||
返回:
|
||||
过滤并排序后的新列表(稳定排序,同信息量保持原相对序)。
|
||||
"""
|
||||
kept = [e for e in entries if p_low <= e.p_hat <= p_high]
|
||||
return sorted(kept, key=lambda e: e.p_hat * (1 - e.p_hat), reverse=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GatePools:
|
||||
"""全部题型的阶梯容器,含构建种子与数据指纹(确定性重建凭据)。
|
||||
|
||||
字段:
|
||||
entries: task_type -> 冷启动序 LadderEntry 列表(warm 排序在取用时做,
|
||||
保持存储序稳定、避免每次更新重写全表顺序)。
|
||||
seed: 冷启动洗牌种子。
|
||||
fingerprint: 构建输入指纹(基线 run_id + 题集 hash 等),resume 校验用。
|
||||
"""
|
||||
|
||||
entries: dict[str, list[LadderEntry]]
|
||||
seed: int
|
||||
fingerprint: str
|
||||
|
||||
def ladder_for(
|
||||
self,
|
||||
task_type: str,
|
||||
exclude_units: set[str],
|
||||
p_low: float,
|
||||
p_high: float,
|
||||
cold: bool,
|
||||
) -> list[str]:
|
||||
"""取该题型的 gate 出题序(unit_id 列表),排除本 step 进化案例包所在单元。
|
||||
|
||||
参数:
|
||||
task_type: 目标题型。
|
||||
exclude_units: 本 step 案例包(failure/success cases)所在单元的
|
||||
unit_id,防止在"刚学的那道题"上自测。按 **unit** 排除:命中单元
|
||||
整体剔除,避免只排 AR pair 半个成员而向 gate 池灌入半个 pair。
|
||||
p_low / p_high: warm 阶段的 p_hat 保留区间。
|
||||
cold: True 表示尚无 epoch 级观测(epoch 1),用冷启动存储序;
|
||||
False 走 order_ladder 信息量排序。
|
||||
|
||||
返回:
|
||||
排除后的有序 unit_id 列表。
|
||||
|
||||
异常:
|
||||
ValueError: 该题型无阶梯(冷启动构建缺失),或该题型阶梯为空。
|
||||
"""
|
||||
if task_type not in self.entries:
|
||||
raise ValueError(f"task_type={task_type} 无阶梯,冷启动构建缺失该题型")
|
||||
pool = self.entries[task_type]
|
||||
if not pool:
|
||||
raise ValueError(f"task_type={task_type} 阶梯为空,无可出题目")
|
||||
ordered = pool if cold else order_ladder(pool, p_low, p_high)
|
||||
return [e.unit_id for e in ordered if e.unit_id not in exclude_units]
|
||||
|
||||
def update_probs(
|
||||
self,
|
||||
per_q_observations: dict[str, bool],
|
||||
units_by_id: dict[str, QuestionUnit],
|
||||
gamma: float,
|
||||
) -> None:
|
||||
"""gamma-EMA 更新 p_hat:先把逐题观测折叠成单元观测,再按 unit_id 匹配更新。
|
||||
|
||||
p_hat <- gamma * p_hat + (1-gamma) * unit_obs。只更新"整个单元都被观测到"
|
||||
的单元;单元观测 = 成员逐题对错的 AND(任一成员错 → 单元错)。折叠是必需的:
|
||||
AR pair 的 unit_id 是 pair_id,若直接按 unit_id 去逐题观测里匹配将永不命中、
|
||||
导致 gamma-EMA 停摆(核心算法保真 #5)。
|
||||
|
||||
参数:
|
||||
per_q_observations: question_id -> 本 epoch 非 gate run 的最新逐题对错。
|
||||
调用方必须已按 run_id 过滤掉 gate 内 rollout(防泄露铁律)。
|
||||
units_by_id: unit_id -> QuestionUnit,用于把逐题观测折叠成单元观测。
|
||||
gamma: EMA 衰减系数。
|
||||
|
||||
关键实现细节:
|
||||
单元只有在其**全部**成员都出现在 per_q_observations 时才更新;半观测
|
||||
(AR pair 只见一半)跳过,避免用不完整证据污染 p_hat。
|
||||
"""
|
||||
for entries in self.entries.values():
|
||||
for e in entries:
|
||||
unit = units_by_id.get(e.unit_id)
|
||||
if unit is None:
|
||||
continue
|
||||
if not all(q.question_id in per_q_observations for q in unit.questions):
|
||||
continue
|
||||
unit_correct = all(per_q_observations[q.question_id] for q in unit.questions)
|
||||
obs = 1.0 if unit_correct else 0.0
|
||||
e.p_hat = gamma * e.p_hat + (1 - gamma) * obs
|
||||
|
||||
def save(self, path: Path) -> None:
|
||||
"""原子写 gate_pools.json(.tmp 再 replace),落 schema_version + unit_id 键。
|
||||
|
||||
参数:
|
||||
path: 目标 JSON 路径。
|
||||
"""
|
||||
payload = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"seed": self.seed,
|
||||
"fingerprint": self.fingerprint,
|
||||
"entries": {
|
||||
t: [{"unit_id": e.unit_id, "p_hat": e.p_hat} for e in es]
|
||||
for t, es in self.entries.items()
|
||||
},
|
||||
}
|
||||
tmp = path.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
os.replace(tmp, path)
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: Path) -> GatePools:
|
||||
"""从 gate_pools.json 恢复;schema_version 不匹配直接报错(不静默混用)。
|
||||
|
||||
参数:
|
||||
path: gate_pools.json 路径。
|
||||
|
||||
返回:
|
||||
复活的 GatePools。
|
||||
|
||||
异常:
|
||||
RuntimeError: 缺 schema_version(存量 v1、qid 键)或版本不等于
|
||||
SCHEMA_VERSION——拒绝把 qid 键当 unit 键静默复用,须 FRESH 重建。
|
||||
"""
|
||||
d = json.loads(path.read_text(encoding="utf-8"))
|
||||
version = d.get("schema_version")
|
||||
if version != SCHEMA_VERSION:
|
||||
raise RuntimeError(
|
||||
f"gate_pools.json schema_version={version!r} 与当前 {SCHEMA_VERSION} 不符"
|
||||
f"(存量 qid 键池不可当 unit 键复用),请删除后 FRESH 重建: {path}"
|
||||
)
|
||||
return cls(
|
||||
entries={
|
||||
t: [LadderEntry(x["unit_id"], x["p_hat"]) for x in es]
|
||||
for t, es in d["entries"].items()
|
||||
},
|
||||
seed=d["seed"],
|
||||
fingerprint=d["fingerprint"],
|
||||
)
|
||||
|
||||
|
||||
def build_or_load_gate_pools(
|
||||
workspace_dir: Path,
|
||||
questions: list[GeneratedQuestion],
|
||||
test_qids: set[str],
|
||||
baseline_correctness: dict[str, bool],
|
||||
task_types: list[str],
|
||||
probe_quota: float,
|
||||
seed: int,
|
||||
baseline_run_id: str,
|
||||
) -> GatePools:
|
||||
"""gate 阶梯获取入口:gate_pools.json 存在且指纹一致则加载,否则冷启动构建。
|
||||
|
||||
参数:
|
||||
workspace_dir: workspace 根目录(gate_pools.json 落其下)。
|
||||
questions: benchmark 全量题(900 题)。
|
||||
test_qids: held-out test 池题目 id(阶梯题源必须排除)。
|
||||
baseline_correctness: 种子基线 900 题全量对错(从基线 run 的 db 读)。
|
||||
task_types: 参与进化的题型列表。
|
||||
probe_quota: 冷启动探针比例。
|
||||
seed: 冷启动洗牌种子。
|
||||
baseline_run_id: 指纹成分。
|
||||
|
||||
返回:
|
||||
GatePools。
|
||||
|
||||
关键实现细节:
|
||||
指纹 = sha1(baseline_run_id|全 qid|seed|probe_quota|task_types|test_qids)。
|
||||
指纹不一致(题集/基线/参数变了)直接报错——FRESH 语义下不该发生,
|
||||
防御性拒绝而非静默重建。
|
||||
"""
|
||||
joined = ",".join(sorted(q.question_id for q in questions))
|
||||
fp_src = (
|
||||
f"{baseline_run_id}|{joined}|{seed}|{probe_quota}"
|
||||
f"|{','.join(sorted(task_types))}|{','.join(sorted(test_qids))}"
|
||||
)
|
||||
fingerprint = hashlib.sha1(fp_src.encode()).hexdigest()
|
||||
path = workspace_dir / "gate_pools.json"
|
||||
if path.exists():
|
||||
pools = GatePools.load(path)
|
||||
if pools.fingerprint != fingerprint:
|
||||
raise RuntimeError(
|
||||
f"gate_pools.json 指纹不一致(题集或基线变更),拒绝静默重建: {path}"
|
||||
)
|
||||
return pools
|
||||
|
||||
entries: dict[str, list[LadderEntry]] = {}
|
||||
for t in task_types:
|
||||
units = _task_units_excluding_test(questions, t, test_qids)
|
||||
if not units:
|
||||
raise ValueError(f"task_type={t} 无非 test 单元,无法建阶梯")
|
||||
entries[t] = build_cold_entries(units, baseline_correctness, probe_quota, seed)
|
||||
logger.info("gate 阶梯[{}]: {} 单元(冷启动)", t, len(entries[t]))
|
||||
pools = GatePools(entries=entries, seed=seed, fingerprint=fingerprint)
|
||||
pools.save(path)
|
||||
return pools
|
||||
|
||||
|
||||
def _task_units_excluding_test(
|
||||
questions: list[GeneratedQuestion], task_type: str, test_qids: set[str]
|
||||
) -> list[QuestionUnit]:
|
||||
"""取某题型的非 test 候选单元:先按 unit 折叠,再整体排除含 test 成员的单元。
|
||||
|
||||
先折叠后排除保证 AR pair 不被拆半(否则半个 pair 交给下游会触发 build_units 的
|
||||
孤儿 fail-fast);single 单元等价于逐题排除(核心算法保真 #5)。
|
||||
|
||||
参数:
|
||||
questions: benchmark 全量题。
|
||||
task_type: 目标题型。
|
||||
test_qids: held-out test 池题目 id。
|
||||
|
||||
返回:
|
||||
该题型下不含任何 test 成员的候选单元列表。
|
||||
"""
|
||||
pool = [q for q in questions if q.task_type == task_type]
|
||||
return [
|
||||
u for u in build_units(pool) if all(q.question_id not in test_qids for q in u.questions)
|
||||
]
|
||||
|
||||
|
||||
class BaselineCache:
|
||||
"""基线侧单元级对错缓存(内容寻址,JSON 持久化)。
|
||||
|
||||
键 = (task_type, skill_hash, prompts_version, unit_id):任何影响该题型
|
||||
有效 skill 的变化(含共享 default-strategy.md 被他类 accept 改写)
|
||||
都使 skill_hash 变化、缓存自然 miss;prompts 版本变化同理。unit_id 维度
|
||||
使 single 题以自身 question_id、AR pair 以共享 pair_id 寻址,缓存单元级
|
||||
对错(pair 双向 AND 折叠后一个布尔)。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
"""加载或初始化缓存文件。
|
||||
|
||||
参数:
|
||||
path: 缓存 JSON 路径(workspace/baseline_cache.json)。
|
||||
"""
|
||||
self._path = path
|
||||
self._store: dict[str, bool] = {}
|
||||
if path.exists():
|
||||
self._store = json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
@staticmethod
|
||||
def _key(task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> str:
|
||||
"""拼缓存键(四维内容寻址,第四维为 unit_id)。"""
|
||||
return f"{task_type}|{s_hash}|{prompts_version}|{unit_id}"
|
||||
|
||||
def get(self, task_type: str, s_hash: str, prompts_version: str, unit_id: str) -> bool | None:
|
||||
"""读缓存;未命中返回 None。
|
||||
|
||||
参数:
|
||||
task_type: 题型。
|
||||
s_hash: 基线侧生效 skill 文件的内容哈希。
|
||||
prompts_version: 当前 prompts 版本。
|
||||
unit_id: 单元 id(single=question_id,AR pair=pair_id)。
|
||||
|
||||
返回:
|
||||
缓存的单元级对错;未命中 None。
|
||||
"""
|
||||
return self._store.get(self._key(task_type, s_hash, prompts_version, unit_id))
|
||||
|
||||
def put(
|
||||
self, task_type: str, s_hash: str, prompts_version: str, unit_id: str, correct: bool
|
||||
) -> None:
|
||||
"""写缓存并落盘(原子写,gate 频度低、全量重写成本可忽略)。
|
||||
|
||||
参数:
|
||||
task_type / s_hash / prompts_version / unit_id: 缓存键四维。
|
||||
correct: 基线侧该单元对错(AR pair 双向 AND 折叠后一个布尔)。
|
||||
|
||||
关键实现细节:
|
||||
先盘后存:新条目先原子落盘(tmp 写 + os.replace)成功后才更新
|
||||
内存,磁盘写失败时内存与磁盘一致(均无新条目),无分裂窗口。
|
||||
"""
|
||||
updated = {
|
||||
**self._store,
|
||||
self._key(task_type, s_hash, prompts_version, unit_id): correct,
|
||||
}
|
||||
tmp = self._path.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(updated, ensure_ascii=False), encoding="utf-8")
|
||||
os.replace(tmp, self._path)
|
||||
self._store = updated
|
||||
@@ -0,0 +1,590 @@
|
||||
"""async 推理编排 — 训练循环的 forward()。
|
||||
|
||||
从 TRM4 core/harness/inference.py (~560 行) 迁移,重大重构:
|
||||
- 同步 ThreadPoolExecutor → asyncio.Semaphore + asyncio.gather
|
||||
- LLMClient.from_env() 每题构造 → llm: LLMProvider 注入共享
|
||||
- SentenceTransformer/OCR 内部构造 → 调用方通过 tool_dispatch_fn 注入
|
||||
- run_id 必传,空串 → ValueError
|
||||
- _aggregate_results 从内存 results 聚合(非 DB 回读)
|
||||
- record_run 由调用方(Runner)负责
|
||||
- prompt 构建由调用方注入 prompt_builder
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.question_units import build_units, unit_correctness
|
||||
from core.agent.loop import AgentLoop
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from app.harness.log import HarnessLog
|
||||
from core.agent.types import LoopResult
|
||||
from core.protocols import LLMProvider
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InferenceResult:
|
||||
"""推理聚合结果(正确率按 unit 粒度)。
|
||||
|
||||
属性:
|
||||
run_id: 运行标识。
|
||||
accuracy: unit 级正确率(correct / total)。
|
||||
total: unit 总数(single 数 + pair 数,孤儿 pair 已剔除不计入)。
|
||||
correct: 正确 unit 数(single 单题正确;pair 走 original/mirror 双向 AND)。
|
||||
per_task_type: 按题型分组的 unit 级指标 {task_type: {accuracy, total, correct}}。
|
||||
steps_mean: 平均步数(record 粒度,逐题溯源)。
|
||||
token_usage: token 总用量 {prompt_tokens, completion_tokens}(record 粒度)。
|
||||
stop_reason_counts: 终止原因计数 {reason: count}(record 粒度)。
|
||||
"""
|
||||
|
||||
run_id: str
|
||||
accuracy: float
|
||||
total: int
|
||||
correct: int
|
||||
per_task_type: dict[str, dict]
|
||||
steps_mean: float
|
||||
token_usage: dict[str, int]
|
||||
stop_reason_counts: dict[str, int]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 表 Schema 定义(5 张表,保留 TRM4 全部 schema)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
PREDICTIONS_SCHEMA: dict[str, str] = {
|
||||
"video_id": "TEXT",
|
||||
"question_id": "TEXT",
|
||||
"task_type": "TEXT",
|
||||
"prediction": "TEXT",
|
||||
"answer": "TEXT",
|
||||
"evidence": "TEXT",
|
||||
"reasoning": "TEXT",
|
||||
"steps_used": "INTEGER",
|
||||
"prompt_tokens": "INTEGER",
|
||||
"completion_tokens": "INTEGER",
|
||||
"stop_reason": "TEXT",
|
||||
"steps_json": "JSON",
|
||||
}
|
||||
|
||||
TRACES_SCHEMA: dict[str, str] = {
|
||||
"video_id": "TEXT",
|
||||
"question_id": "TEXT",
|
||||
"step": "INTEGER",
|
||||
"tool_name": "TEXT",
|
||||
"tool_args": "JSON",
|
||||
"tool_output": "TEXT",
|
||||
"thought": "TEXT",
|
||||
}
|
||||
|
||||
VALIDATION_FLAGS_SCHEMA: dict[str, str] = {
|
||||
"video_id": "TEXT",
|
||||
"question_id": "TEXT",
|
||||
"has_l3_visit": "INTEGER",
|
||||
"l1_count": "INTEGER",
|
||||
"l2_count": "INTEGER",
|
||||
"l3_count": "INTEGER",
|
||||
}
|
||||
|
||||
ANCHOR_CHECK_SCHEMA: dict[str, str] = {
|
||||
"video_id": "TEXT",
|
||||
"question_id": "TEXT",
|
||||
"step": "INTEGER",
|
||||
"n_assertions": "INTEGER",
|
||||
"n_anchored": "INTEGER",
|
||||
"n_illegal": "INTEGER",
|
||||
"n_expanded": "INTEGER",
|
||||
"n_trunc": "INTEGER",
|
||||
"output_chars": "INTEGER",
|
||||
}
|
||||
|
||||
OF_HEALTH_SCHEMA: dict[str, str] = {
|
||||
"video_id": "TEXT",
|
||||
"question_id": "TEXT",
|
||||
"step": "INTEGER",
|
||||
"ocr_injected": "INTEGER",
|
||||
"ocr_chars": "INTEGER",
|
||||
"ocr_failed": "INTEGER",
|
||||
"discrepancy": "INTEGER",
|
||||
"abstain": "INTEGER",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部工具
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _DispatcherAdapter:
|
||||
"""将裸 async callable 包装为 ToolDispatcher Protocol 实例。
|
||||
|
||||
AgentLoop 要求 ToolDispatcher(有 dispatch 方法),而 run_inference
|
||||
接收的 tool_dispatch_fn 是裸 async callable。此适配器桥接两者。
|
||||
|
||||
参数:
|
||||
fn: async def (tool_name, args, *, context) -> str。
|
||||
"""
|
||||
|
||||
def __init__(self, fn: Callable[..., Any]) -> None:
|
||||
self._fn = fn
|
||||
|
||||
async def dispatch(
|
||||
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
|
||||
) -> str:
|
||||
"""转发工具调用给被包装的 callable。"""
|
||||
return await self._fn(tool_name, args, context=context)
|
||||
|
||||
|
||||
def _to_text_field(value: Any) -> str:
|
||||
"""把 prediction 的 evidence/reasoning 归一为可入库的文本。
|
||||
|
||||
LLM 有时把这些字段返回成 list 或 dict(而非字符串)。sqlite 无法绑定
|
||||
非标量类型,直接入库会抛 ProgrammingError 致该题丢失预测行、进而触发
|
||||
rollout 完整性护栏中止整轮。凡非 str 一律 JSON 序列化为文本。
|
||||
|
||||
参数:
|
||||
value: evidence/reasoning 原始值(可能是 str/list/dict)。
|
||||
|
||||
返回:
|
||||
可直接入库的字符串。
|
||||
"""
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
|
||||
|
||||
def _normalize_prediction(answer: object) -> str | None:
|
||||
"""归一化 prediction 落库值。
|
||||
|
||||
LLM 提交的 answer 有时是 list/dict(如 {'answer': ['B']}),sqlite 无法绑定
|
||||
非标量类型直接入库会抛 ProgrammingError 击穿整轮 gather。None 保留(INFRA 空
|
||||
预测语义,供正确率判定天然计错);str 原样;其余 JSON 序列化为文本。
|
||||
|
||||
参数:
|
||||
answer: LoopResult.result 中的 answer 原始值(可能是 None/str/list/dict)。
|
||||
|
||||
返回:
|
||||
None(保留空预测语义)或可直接入库的字符串。
|
||||
"""
|
||||
if answer is None or isinstance(answer, str):
|
||||
return answer
|
||||
return _to_text_field(answer)
|
||||
|
||||
|
||||
def _zero_result(run_id: str) -> InferenceResult:
|
||||
"""空记录时的零值 InferenceResult。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
|
||||
返回:
|
||||
全零的 InferenceResult。
|
||||
"""
|
||||
return InferenceResult(
|
||||
run_id=run_id,
|
||||
accuracy=0.0,
|
||||
total=0,
|
||||
correct=0,
|
||||
per_task_type={},
|
||||
steps_mean=0.0,
|
||||
token_usage={"prompt_tokens": 0, "completion_tokens": 0},
|
||||
stop_reason_counts={},
|
||||
)
|
||||
|
||||
|
||||
def _group_by_task_type(graded: list[tuple[QuestionUnit, bool]]) -> dict[str, dict[str, Any]]:
|
||||
"""按 task_type 分组聚合 unit 级正确率指标。
|
||||
|
||||
pair 单元整体计 1 个 unit,归入其 task_type;single 单元计 1 个 unit。
|
||||
|
||||
参数:
|
||||
graded: (单元, 该单元是否整体正确) 元组列表。
|
||||
|
||||
返回:
|
||||
{task_type: {accuracy, total, correct}} 映射(unit 粒度)。
|
||||
"""
|
||||
task_groups: dict[str, list[bool]] = defaultdict(list)
|
||||
for unit, is_correct in graded:
|
||||
task_groups[unit.task_type].append(is_correct)
|
||||
|
||||
per_task_type: dict[str, dict[str, Any]] = {}
|
||||
for task_type, verdicts in task_groups.items():
|
||||
t_total = len(verdicts)
|
||||
t_correct = sum(verdicts)
|
||||
per_task_type[task_type] = {
|
||||
"accuracy": t_correct / t_total,
|
||||
"total": t_total,
|
||||
"correct": t_correct,
|
||||
}
|
||||
return per_task_type
|
||||
|
||||
|
||||
def _is_valid_pair(group: list[GeneratedQuestion]) -> bool:
|
||||
"""判定同一 pair_id 分组是否为合法孪生对(恰好 1 original + 1 mirror,无多余)。
|
||||
|
||||
要求分组总数恰为 2 且角色齐备唯一;有额外非法 role 记录(total>2)或角色
|
||||
缺失/重复均视为非法,交由调用方剔除,防非法记录混入 build_units。
|
||||
|
||||
参数:
|
||||
group: 归属同一 pair_id 的题目列表。
|
||||
|
||||
返回:
|
||||
合法孪生对为 True,否则 False。
|
||||
"""
|
||||
if len(group) != 2:
|
||||
return False
|
||||
originals = sum(1 for q in group if q.question_role == "pair_original")
|
||||
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
|
||||
return originals == 1 and mirrors == 1
|
||||
|
||||
|
||||
def _drop_orphan_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||
"""剔除收不齐 2 条 / 角色非法的孤儿 pair,告警不静默。
|
||||
|
||||
每条题目均会各答一次并逐题落库;能否合成 pair 单元仅取决于 questions
|
||||
是否同时含该 pair_id 的 original + mirror(且无多余非法记录)。非法者告警并
|
||||
整对剔除,使后续 build_units 只面对合法孪生对(不触发 fail-fast),孤儿 unit
|
||||
不计入 total(对齐设计 §8 聚合入口的"告警 + 剔除")。
|
||||
|
||||
参数:
|
||||
questions: 待聚合的题目列表(可混含 single 与孪生对成员)。
|
||||
|
||||
返回:
|
||||
可安全交给 build_units 的题目列表(single 全保留,pair 仅保留合法成对者)。
|
||||
"""
|
||||
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||
singles: list[GeneratedQuestion] = []
|
||||
for q in questions:
|
||||
if q.pair_id:
|
||||
by_pair[q.pair_id].append(q)
|
||||
else:
|
||||
singles.append(q)
|
||||
|
||||
kept_pairs: list[GeneratedQuestion] = []
|
||||
for pair_id, group in by_pair.items():
|
||||
if _is_valid_pair(group):
|
||||
kept_pairs.extend(group)
|
||||
else:
|
||||
logger.warning(
|
||||
"孤儿 pair {}:非法配对(total={}),剔除该 unit 不计入 total",
|
||||
pair_id,
|
||||
len(group),
|
||||
)
|
||||
return singles + kept_pairs
|
||||
|
||||
|
||||
def _per_question_correctness(records: list[dict[str, Any]]) -> dict[str, bool]:
|
||||
"""由逐题 record 构造 question_id → 该题作答是否正确 的映射。
|
||||
|
||||
prediction 为 None(作答异常)时与 answer 不相等 → False,天然计错。
|
||||
|
||||
参数:
|
||||
records: _run_single_question 返回的逐题 record 列表。
|
||||
|
||||
返回:
|
||||
{question_id: prediction == answer} 映射,供 unit_correctness 取值。
|
||||
"""
|
||||
return {r["question_id"]: r["prediction"] == r["answer"] for r in records}
|
||||
|
||||
|
||||
def _grade_unit(unit: QuestionUnit, per_q: dict[str, bool]) -> bool:
|
||||
"""判定单元整体正确性,缺 prediction 时 fail-loud(带上下文)。
|
||||
|
||||
_drop_orphan_pairs 已剔除孤儿/非法配对,正常情况下 unit 内每题都应有对应
|
||||
record;若仍缺失说明聚合不变量被破坏(如 records 与 questions 不同源)。此处
|
||||
显式抛带上下文的 ValueError(fail-loud,不 catch/不跳过/不兜底),而非放任
|
||||
unit_correctness 抛裸 KeyError 丢失定位信息。
|
||||
|
||||
参数:
|
||||
unit: 待判定单元。
|
||||
per_q: question_id → 该题是否作答正确 的映射。
|
||||
|
||||
返回:
|
||||
单元整体是否正确(single 即单题正确;pair 走双向 AND)。
|
||||
|
||||
异常:
|
||||
ValueError: unit 内某 question_id 不在 per_q 中(聚合不变量被破坏)。
|
||||
"""
|
||||
missing = [q.question_id for q in unit.questions if q.question_id not in per_q]
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"unit {unit.unit_id} 的 question {missing} 缺 prediction"
|
||||
"(_drop_orphan_pairs 后不应发生,聚合不变量被破坏)"
|
||||
)
|
||||
return unit_correctness(unit, per_q)
|
||||
|
||||
|
||||
def _aggregate_results(
|
||||
records: list[dict[str, Any]],
|
||||
questions: list[GeneratedQuestion],
|
||||
run_id: str,
|
||||
) -> InferenceResult:
|
||||
"""从内存 records + 题目列表按 unit 粒度聚合推理指标。
|
||||
|
||||
逐题 record 保留逐题溯源(token/steps/stop_reason 诊断仍按 record 汇总);
|
||||
正确率则按 unit 粒度计:single 计 1,AR pair 经 build_units 收齐 original +
|
||||
mirror 后走 unit_correctness 的双向 AND 判定,整对计 1 个 unit。孤儿 pair
|
||||
在 _drop_orphan_pairs 中告警 + 剔除,不计入 total。
|
||||
|
||||
参数:
|
||||
records: _run_single_question 返回的逐题 record 列表。
|
||||
questions: 与 records 对应的题目列表(提供 pair_id/question_role 元数据)。
|
||||
run_id: 当前运行标识。
|
||||
|
||||
返回:
|
||||
InferenceResult 冻结实例(total/correct/per_task_type 为 unit 粒度)。
|
||||
"""
|
||||
if not records:
|
||||
return _zero_result(run_id)
|
||||
|
||||
per_q = _per_question_correctness(records)
|
||||
units = build_units(_drop_orphan_pairs(questions))
|
||||
graded = [(unit, _grade_unit(unit, per_q)) for unit in units]
|
||||
|
||||
total = len(graded)
|
||||
correct = sum(1 for _, is_correct in graded if is_correct)
|
||||
|
||||
stop_counts: dict[str, int] = defaultdict(int)
|
||||
for r in records:
|
||||
stop_counts[r["stop_reason"]] += 1
|
||||
|
||||
n_records = len(records)
|
||||
return InferenceResult(
|
||||
run_id=run_id,
|
||||
accuracy=correct / total if total else 0.0,
|
||||
total=total,
|
||||
correct=correct,
|
||||
per_task_type=_group_by_task_type(graded),
|
||||
steps_mean=sum(r["steps_used"] for r in records) / n_records,
|
||||
token_usage={
|
||||
"prompt_tokens": sum(r["prompt_tokens"] for r in records),
|
||||
"completion_tokens": sum(r["completion_tokens"] for r in records),
|
||||
},
|
||||
stop_reason_counts=dict(stop_counts),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 单题推理
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _run_single_question(
|
||||
qa: GeneratedQuestion,
|
||||
*,
|
||||
llm: LLMProvider,
|
||||
tool_dispatch_fn: Callable[..., Any],
|
||||
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]],
|
||||
log: HarnessLog,
|
||||
max_steps: int,
|
||||
plugins: list[object],
|
||||
run_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""执行单道题目的 Agent 推理。
|
||||
|
||||
悲观默认值:record 初始 stop_reason="error",成功后覆盖。
|
||||
prediction 必落库:log.insert 在 try/except 之后(无论成败)。
|
||||
|
||||
参数:
|
||||
qa: 待推理的题目。
|
||||
llm: LLMProvider 共享实例。
|
||||
tool_dispatch_fn: async 工具调度函数 (tool_name, args, *, context) -> str。
|
||||
prompt_builder: (GeneratedQuestion) -> (system_prompt, user_prompt)。
|
||||
log: HarnessLog 实例(线程安全)。
|
||||
max_steps: AgentLoop 最大步数。
|
||||
plugins: pluggy 插件列表。
|
||||
run_id: 运行标识,用作 cache_salt——run_id 含 _e{epoch} 天然跨 epoch 重采样、
|
||||
同 epoch 续跑命中缓存(算法 #10 透传)。
|
||||
|
||||
返回:
|
||||
预测结果字典(含 video_id, question_id, prediction, answer 等)。
|
||||
"""
|
||||
# run_id 必须显式入 record:HarnessLog.insert 缺省用**实例** run_id 填充,
|
||||
# 连续并发 gate 共享单一 gate_log(实例 run_id 为 step 级)时,各臂行必须
|
||||
# 落自己的臂 run_id,否则 validate 回读 _load_run_rows(臂 run_id) 为空。
|
||||
record: dict[str, Any] = {
|
||||
"run_id": run_id,
|
||||
"video_id": qa.video_id,
|
||||
"question_id": qa.question_id,
|
||||
"task_type": qa.task_type,
|
||||
"prediction": None,
|
||||
"answer": qa.answer,
|
||||
"evidence": "",
|
||||
"reasoning": "",
|
||||
"steps_used": 0,
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"stop_reason": "error", # 悲观默认
|
||||
"steps_json": "[]",
|
||||
}
|
||||
|
||||
try:
|
||||
system_prompt, user_prompt = prompt_builder(qa)
|
||||
dispatcher = _DispatcherAdapter(tool_dispatch_fn)
|
||||
loop = AgentLoop(llm, max_steps=max_steps)
|
||||
loop_result: LoopResult = await loop.run(
|
||||
system_prompt,
|
||||
user_prompt,
|
||||
dispatcher,
|
||||
plugins=plugins,
|
||||
session_id=qa.question_id,
|
||||
cache_salt=run_id,
|
||||
)
|
||||
|
||||
result_dict = loop_result.result if isinstance(loop_result.result, dict) else {}
|
||||
evidence = _to_text_field(result_dict.get("evidence", ""))
|
||||
reasoning = _to_text_field(result_dict.get("reasoning", ""))
|
||||
record.update(
|
||||
{
|
||||
"prediction": _normalize_prediction(result_dict.get("answer")),
|
||||
"evidence": evidence,
|
||||
"reasoning": reasoning,
|
||||
"steps_used": loop_result.steps_used,
|
||||
"prompt_tokens": loop_result.token_usage["prompt_tokens"],
|
||||
"completion_tokens": loop_result.token_usage["completion_tokens"],
|
||||
"stop_reason": loop_result.stop_reason,
|
||||
"steps_json": json.dumps(
|
||||
[
|
||||
{
|
||||
"thought": s.thought,
|
||||
"tool_call": s.tool_call,
|
||||
"tool_output": s.tool_output,
|
||||
}
|
||||
for s in loop_result.steps
|
||||
],
|
||||
ensure_ascii=False,
|
||||
),
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("[{}] QA {} 执行异常", qa.video_id, qa.question_id)
|
||||
|
||||
# prediction 必落库(try 外,无论成败);绑定异常降级为最小 error 行,不击穿 gather
|
||||
try:
|
||||
await asyncio.to_thread(log.insert, "predictions", record)
|
||||
except (sqlite3.InterfaceError, sqlite3.ProgrammingError):
|
||||
logger.exception("[{}] QA {} 落库绑定异常,降级为 error 行", qa.video_id, qa.question_id)
|
||||
record["prediction"] = None
|
||||
record["stop_reason"] = "error"
|
||||
await asyncio.to_thread(
|
||||
log.insert,
|
||||
"predictions",
|
||||
{k: v for k, v in record.items() if isinstance(v, (str, int, float, type(None)))},
|
||||
)
|
||||
return record
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 建表
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ensure_tables(log: HarnessLog) -> None:
|
||||
"""创建推理所需的 5 张表。
|
||||
|
||||
参数:
|
||||
log: HarnessLog 实例。
|
||||
"""
|
||||
log.create_table("predictions", PREDICTIONS_SCHEMA)
|
||||
log.create_table("traces", TRACES_SCHEMA)
|
||||
log.create_table("validation_flags", VALIDATION_FLAGS_SCHEMA)
|
||||
log.create_table("anchor_check", ANCHOR_CHECK_SCHEMA)
|
||||
log.create_table("observe_frame_health", OF_HEALTH_SCHEMA)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 公共入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def run_inference(
|
||||
questions: list[GeneratedQuestion],
|
||||
*,
|
||||
llm: LLMProvider,
|
||||
tool_dispatch_fn: Callable[..., Any],
|
||||
prompt_builder: Callable[[GeneratedQuestion], tuple[str, str]],
|
||||
log: HarnessLog,
|
||||
run_id: str,
|
||||
concurrency: int,
|
||||
max_steps: int,
|
||||
skill_mode: str,
|
||||
plugins_factory: Callable[[str, str], list[object]] | None = None,
|
||||
) -> InferenceResult:
|
||||
"""在视频树上执行 Agent 推理,对应训练循环的 forward()。
|
||||
|
||||
参数:
|
||||
questions: 待推理的题目列表。
|
||||
llm: LLMProvider 共享实例(依赖注入)。
|
||||
tool_dispatch_fn: async 工具调度函数 (tool_name, args, *, context) -> str。
|
||||
prompt_builder: prompt 构建函数 (GeneratedQuestion) -> (system_prompt, user_prompt)。
|
||||
log: HarnessLog 实例(由调用方管理生命周期)。
|
||||
run_id: 运行标识(必传,空串 → ValueError)。
|
||||
concurrency: 最大并发数(asyncio.Semaphore 控制)。
|
||||
max_steps: AgentLoop 单题最大步数。
|
||||
skill_mode: "auto" / "manual" / "none"(传递给调用方的 prompt/plugin 构建逻辑)。
|
||||
plugins_factory: 可选的插件工厂 (video_id, question_id) -> plugins 列表。
|
||||
|
||||
返回:
|
||||
InferenceResult(含 accuracy、per_task_type 等聚合指标)。
|
||||
|
||||
异常:
|
||||
ValueError: run_id 为空串或纯空白。
|
||||
"""
|
||||
if not run_id or not run_id.strip():
|
||||
raise ValueError("run_id 不得为空串或纯空白")
|
||||
|
||||
_ensure_tables(log)
|
||||
|
||||
if not questions:
|
||||
logger.info("题目列表为空,返回零值 InferenceResult")
|
||||
return _aggregate_results([], [], run_id)
|
||||
|
||||
sem = asyncio.Semaphore(concurrency)
|
||||
total_count = len(questions)
|
||||
|
||||
async def _bounded(index: int, qa: GeneratedQuestion) -> dict[str, Any]:
|
||||
"""信号量限流的单题推理包装。"""
|
||||
async with sem:
|
||||
plugins = (
|
||||
plugins_factory(qa.video_id, qa.question_id) if plugins_factory is not None else []
|
||||
)
|
||||
result = await _run_single_question(
|
||||
qa,
|
||||
llm=llm,
|
||||
tool_dispatch_fn=tool_dispatch_fn,
|
||||
prompt_builder=prompt_builder,
|
||||
log=log,
|
||||
max_steps=max_steps,
|
||||
plugins=plugins,
|
||||
run_id=run_id,
|
||||
)
|
||||
logger.info(
|
||||
"[{}/{}] {} QA {} 完成 (stop={})",
|
||||
index + 1,
|
||||
total_count,
|
||||
qa.video_id,
|
||||
qa.question_id,
|
||||
result["stop_reason"],
|
||||
)
|
||||
return result
|
||||
|
||||
results = await asyncio.gather(*[_bounded(i, qa) for i, qa in enumerate(questions)])
|
||||
|
||||
inference_result = _aggregate_results(list(results), questions, run_id)
|
||||
logger.info(
|
||||
"推理完成: accuracy={:.2%} ({}/{})",
|
||||
inference_result.accuracy,
|
||||
inference_result.correct,
|
||||
inference_result.total,
|
||||
)
|
||||
return inference_result
|
||||
@@ -0,0 +1,354 @@
|
||||
"""HarnessLog:SQLite 薄包装 + RunLogImpl 只读查询端口。
|
||||
|
||||
HarnessLog 提供统一的结构化日志接口,从 TRM4 直搬,保留全部线程安全与幂等语义。
|
||||
同 run_id 重复创建时通过 ON CONFLICT DO UPDATE 更新 started_at/config/status。
|
||||
RunLogImpl 实现 core/evolution/protocols.py::RunLog Protocol,用独立连接做只读 SELECT,
|
||||
不经 HarnessLog 生命周期(不触发 _runs upsert),避免污染运行状态。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import threading
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _get_git_sha() -> str | None:
|
||||
"""获取当前 git commit SHA。"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
return None
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 格式字符串。"""
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
class HarnessLog:
|
||||
"""SQLite 薄包装,为科研项目提供统一的结构化日志接口。
|
||||
|
||||
关键设计:
|
||||
- WAL 模式 + threading.Lock 保证共享连接下并发安全。
|
||||
- ON CONFLICT DO UPDATE 保证幂等(同 run_id 多次创建不报错)并更新 started_at/config/status。
|
||||
- query 也持锁:共享连接(check_same_thread=False)下并发 SELECT + INSERT
|
||||
在同一连接上 execute 会损坏游标状态,故读也须串行化。
|
||||
- context manager 语义:正常退出 completed,异常退出 failed。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 数据库文件路径。
|
||||
run_id: 本次运行的唯一标识。
|
||||
git_sha: 代码版本,默认自动获取。
|
||||
config_snapshot: 本次运行的配置快照。
|
||||
register_run: 是否注册运行(upsert _runs + 退出时同步 status)。默认 True;
|
||||
只读查询已有 run(如基线预测回读)时传 False,避免把该 run 的
|
||||
started_at/config/status 改写、把基线元数据污染成本次进程的运行状态。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db_path: str,
|
||||
run_id: str,
|
||||
git_sha: str | None = None,
|
||||
config_snapshot: dict[str, Any] | None = None,
|
||||
*,
|
||||
register_run: bool = True,
|
||||
) -> None:
|
||||
self._run_id = run_id
|
||||
self._register_run = register_run
|
||||
Path(db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
# 单持久连接 + 进程内 Lock 串行化写:把并发控制拉到进程内,消除多连接争
|
||||
# SQLite 写锁。同款模式复用于 adapters/telemetry.py:SQLiteTelemetryRecorder。
|
||||
self._conn = sqlite3.connect(db_path, check_same_thread=False)
|
||||
self._lock = threading.Lock()
|
||||
self._conn.row_factory = sqlite3.Row
|
||||
self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
self._init_fixed_tables()
|
||||
if register_run:
|
||||
resolved_sha = git_sha or _get_git_sha()
|
||||
config_json = (
|
||||
json.dumps(config_snapshot, ensure_ascii=False) if config_snapshot else None
|
||||
)
|
||||
self._conn.execute(
|
||||
"INSERT INTO _runs"
|
||||
" (run_id, git_sha, started_at, config, status)"
|
||||
" VALUES (?, ?, ?, ?, ?)"
|
||||
" ON CONFLICT(run_id) DO UPDATE SET"
|
||||
" started_at=excluded.started_at,"
|
||||
" config=excluded.config,"
|
||||
" status=excluded.status",
|
||||
(run_id, resolved_sha, _now_iso(), config_json, "running"),
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def _init_fixed_tables(self) -> None:
|
||||
"""创建 _runs 和 _events 固定表。"""
|
||||
self._conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS _runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
git_sha TEXT,
|
||||
started_at TEXT,
|
||||
finished_at TEXT,
|
||||
config JSON,
|
||||
status TEXT DEFAULT 'running',
|
||||
skills_version TEXT,
|
||||
prompts_version TEXT,
|
||||
questions_ref TEXT
|
||||
)
|
||||
""")
|
||||
self._conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS _events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
run_id TEXT,
|
||||
timestamp TEXT,
|
||||
event_type TEXT,
|
||||
payload JSON
|
||||
)
|
||||
""")
|
||||
self._conn.commit()
|
||||
|
||||
def create_table(
|
||||
self,
|
||||
name: str,
|
||||
columns: dict[str, str],
|
||||
primary_key: str | None = None,
|
||||
) -> None:
|
||||
"""创建自定义表,自动追加 run_id 和 timestamp 列。
|
||||
|
||||
参数:
|
||||
name: 表名。
|
||||
columns: 列定义,如 {"epoch": "INTEGER", "loss": "REAL"}。
|
||||
primary_key: 主键列名。
|
||||
"""
|
||||
all_columns = {"run_id": "TEXT", "timestamp": "TEXT"}
|
||||
all_columns.update(columns)
|
||||
col_defs = []
|
||||
for col_name, col_type in all_columns.items():
|
||||
pk_suffix = " PRIMARY KEY" if col_name == primary_key else ""
|
||||
col_defs.append(f"{col_name} {col_type}{pk_suffix}")
|
||||
sql = f"CREATE TABLE IF NOT EXISTS {name} ({', '.join(col_defs)})"
|
||||
self._conn.execute(sql)
|
||||
self._conn.commit()
|
||||
|
||||
def insert(self, table: str, record: dict[str, Any], mode: str = "append") -> None:
|
||||
"""插入一条记录,自动填充 run_id 和 timestamp。
|
||||
|
||||
参数:
|
||||
table: 目标表名。
|
||||
record: 要插入的数据。
|
||||
mode: "append" 或 "upsert"。
|
||||
"""
|
||||
enriched = {"run_id": self._run_id, "timestamp": _now_iso()}
|
||||
enriched.update(record)
|
||||
cols = list(enriched.keys())
|
||||
placeholders = ", ".join(["?"] * len(cols))
|
||||
col_names = ", ".join(cols)
|
||||
values = [enriched[c] for c in cols]
|
||||
if mode == "upsert":
|
||||
sql = f"INSERT OR REPLACE INTO {table} ({col_names}) VALUES ({placeholders})"
|
||||
else:
|
||||
sql = f"INSERT INTO {table} ({col_names}) VALUES ({placeholders})"
|
||||
with self._lock:
|
||||
self._conn.execute(sql, values)
|
||||
self._conn.commit()
|
||||
|
||||
def insert_many(self, table: str, records: list[dict[str, Any]], mode: str = "append") -> None:
|
||||
"""批量插入多条记录。
|
||||
|
||||
参数:
|
||||
table: 目标表名。
|
||||
records: 要插入的数据列表。
|
||||
mode: "append" 或 "upsert"。
|
||||
"""
|
||||
for record in records:
|
||||
self.insert(table, record, mode=mode)
|
||||
|
||||
def execute(self, sql: str, params: tuple[Any, ...] = ()) -> None:
|
||||
"""执行原生 SQL 写操作。
|
||||
|
||||
参数:
|
||||
sql: SQL 语句。
|
||||
params: 参数元组。
|
||||
"""
|
||||
with self._lock:
|
||||
self._conn.execute(sql, params)
|
||||
self._conn.commit()
|
||||
|
||||
def query(self, sql: str, params: tuple[Any, ...] = ()) -> list[dict[str, Any]]:
|
||||
"""执行原生 SQL 查询,返回 list[dict]。
|
||||
|
||||
与所有写方法同持 self._lock:共享连接(check_same_thread=False)下,
|
||||
并发 SELECT 与 INSERT 在同一连接上 execute 会损坏游标状态,故读也须串行化。
|
||||
|
||||
参数:
|
||||
sql: SQL 查询语句。
|
||||
params: 参数元组。
|
||||
|
||||
返回:
|
||||
查询结果列表,每行为一个字典。
|
||||
"""
|
||||
with self._lock:
|
||||
cursor = self._conn.execute(sql, params)
|
||||
columns = [desc[0] for desc in cursor.description]
|
||||
return [dict(zip(columns, row, strict=True)) for row in cursor.fetchall()]
|
||||
|
||||
def log_event(self, event_type: str, payload: dict[str, Any]) -> None:
|
||||
"""向 _events 表写入一条事件。
|
||||
|
||||
参数:
|
||||
event_type: 事件类型标识。
|
||||
payload: 事件数据。
|
||||
"""
|
||||
with self._lock:
|
||||
self._conn.execute(
|
||||
"INSERT INTO _events (run_id, timestamp, event_type, payload) VALUES (?, ?, ?, ?)",
|
||||
(
|
||||
self._run_id,
|
||||
_now_iso(),
|
||||
event_type,
|
||||
json.dumps(payload, ensure_ascii=False),
|
||||
),
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def close(self, status: str = "completed") -> None:
|
||||
"""更新运行状态并关闭连接。
|
||||
|
||||
参数:
|
||||
status: 最终状态,"completed" 或 "failed"。
|
||||
|
||||
关键实现:
|
||||
register_run=False(只读打开)时跳过 status 更新,仅关闭连接,
|
||||
避免只读回读把已有 run 的 finished_at/status 改写。
|
||||
"""
|
||||
with self._lock:
|
||||
if self._register_run:
|
||||
self._conn.execute(
|
||||
"UPDATE _runs SET finished_at = ?, status = ? WHERE run_id = ?",
|
||||
(_now_iso(), status, self._run_id),
|
||||
)
|
||||
self._conn.commit()
|
||||
self._conn.close()
|
||||
|
||||
def __enter__(self) -> HarnessLog:
|
||||
return self
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: Any,
|
||||
) -> None:
|
||||
status = "failed" if exc_type is not None else "completed"
|
||||
self.close(status=status)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RunLogImpl — core/evolution/protocols.py::RunLog 的只读实现
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _read_table(
|
||||
db_path: str,
|
||||
table: str,
|
||||
run_id: str,
|
||||
*,
|
||||
question_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""纯读某表指定 run 的行——不经 HarnessLog 生命周期,避免回读污染 _runs 运行状态。
|
||||
|
||||
HarnessLog.__enter__/__exit__ 会对 run_id 做 INSERT OR IGNORE 并在退出时标 completed;
|
||||
回读指标绝不应改运行状态,故走独立只读连接(仅 SELECT)。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
table: 表名(内部固定常量,非外部输入,无注入风险)。
|
||||
run_id: 过滤的 run ID。
|
||||
question_ids: 可选的 question_id 过滤列表。
|
||||
|
||||
返回:
|
||||
行 dict 列表;表尚未建(没写过)视为无数据返 []。
|
||||
"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
exists = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
|
||||
).fetchone()
|
||||
if exists is None:
|
||||
return []
|
||||
|
||||
if question_ids is not None:
|
||||
placeholders = ", ".join(["?"] * len(question_ids))
|
||||
sql = f"SELECT * FROM {table} WHERE run_id = ? AND question_id IN ({placeholders})"
|
||||
rows = conn.execute(sql, (run_id, *question_ids)).fetchall()
|
||||
else:
|
||||
rows = conn.execute(f"SELECT * FROM {table} WHERE run_id = ?", (run_id,)).fetchall()
|
||||
|
||||
return [dict(r) for r in rows]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
class RunLogImpl:
|
||||
"""RunLog Protocol 的只读实现。
|
||||
|
||||
用独立 sqlite3.connect 做 SELECT,不经 HarnessLog 生命周期(不触发 _runs INSERT),
|
||||
asyncio.to_thread 包装同步 SQL 查询,避免引入 aiosqlite 新依赖。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 数据库文件路径。
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: str) -> None:
|
||||
self._db_path = db_path
|
||||
|
||||
async def get_predictions(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
question_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""查询指定 run 的预测记录。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
预测记录字典列表。
|
||||
"""
|
||||
return await asyncio.to_thread(
|
||||
_read_table, self._db_path, "predictions", run_id, question_ids=question_ids
|
||||
)
|
||||
|
||||
async def get_traces(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
question_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""查询指定 run 的推理轨迹。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
轨迹记录字典列表。
|
||||
"""
|
||||
return await asyncio.to_thread(
|
||||
_read_table, self._db_path, "traces", run_id, question_ids=question_ids
|
||||
)
|
||||
@@ -0,0 +1,178 @@
|
||||
"""慢更新动量生成 — epoch 末为单个 skill 产出新的动量指导。
|
||||
|
||||
对标 SkillOpt 的 slow_update 机制:拿上一 epoch 末与当前 epoch 末两版 skill,
|
||||
在固定样本上各跑一遍得到纵向对比(comparison_pairs),反思上一轮动量指导是否奏效、
|
||||
本轮正文改动是改善还是漂移,据此重写动量指导。新指导经 patch 引擎的 replace_momentum
|
||||
写回 skill 的 momentum 受保护区,作为下一轮进化的方向锚。
|
||||
|
||||
从 TRM4 core/harness/momentum.py(156 行)迁移 + async 化。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from core.evolution.diagnose import extract_json_from_response
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from core.protocols import LLMProvider
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 四类纵向对比类别名(单一真源)
|
||||
# =========================================================================
|
||||
|
||||
IMPROVED = "improved" # 错→对
|
||||
REGRESSED = "regressed" # 对→错
|
||||
PERSISTENT_FAIL = "persistent_fail" # 错→错
|
||||
STABLE_SUCCESS = "stable_success" # 对→对
|
||||
|
||||
# 类别名 → 展示标题,列表顺序即展示顺序。
|
||||
# 回退(REGRESSED)刻意排在改善(IMPROVED)之前——它是最该警惕的伤害信号。
|
||||
_CATEGORY_LABELS: tuple[tuple[str, str], ...] = (
|
||||
(REGRESSED, "从对变错(回退,最高优先级)"),
|
||||
(PERSISTENT_FAIL, "始终答错(持续失败)"),
|
||||
(IMPROVED, "从错变对(改善)"),
|
||||
(STABLE_SUCCESS, "始终答对(稳定成功)"),
|
||||
)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 辅助函数
|
||||
# =========================================================================
|
||||
|
||||
|
||||
def _categorize_pair(pair: dict[str, Any]) -> str:
|
||||
"""按两版正误派生纵向对比类别。
|
||||
|
||||
用键值的真值(bool(...))表示该题在两版上各自的正误:缺 correct_prev/
|
||||
correct_curr 键时直接抛 KeyError 向上传播——这是上游数据损坏(不是裁判语义
|
||||
歧义),静默当 False 会伪造 persistent_fail 证据、污染动量指导,故不掩盖。
|
||||
|
||||
参数:
|
||||
pair: 单个纵向对比对,须含 correct_prev/correct_curr 两键。
|
||||
|
||||
返回:
|
||||
四个类别命名常量之一:IMPROVED(错→对)/REGRESSED(对→错)/
|
||||
PERSISTENT_FAIL(错→错)/STABLE_SUCCESS(对→对)。
|
||||
|
||||
异常:
|
||||
KeyError: 缺 correct_prev 或 correct_curr 键时。
|
||||
"""
|
||||
correct_prev = bool(pair["correct_prev"])
|
||||
correct_curr = bool(pair["correct_curr"])
|
||||
if not correct_prev and correct_curr:
|
||||
return IMPROVED
|
||||
if correct_prev and not correct_curr:
|
||||
return REGRESSED
|
||||
if not correct_prev and not correct_curr:
|
||||
return PERSISTENT_FAIL
|
||||
return STABLE_SUCCESS
|
||||
|
||||
|
||||
def _format_comparison_pairs(comparison_pairs: list[dict[str, Any]]) -> str:
|
||||
"""将纵向对比对格式化为裁判可读文本,按 _CATEGORY_LABELS 分组与排序。
|
||||
|
||||
参数:
|
||||
comparison_pairs: 每个 dict 含 question/prev_prediction/curr_prediction/
|
||||
correct_prev/correct_curr 字段,描述一道固定样本上两版的成对结果。
|
||||
|
||||
返回:
|
||||
可读的纵向对比文本;空列表返回占位说明。
|
||||
|
||||
异常:
|
||||
KeyError: 任一 pair 缺 correct_prev/correct_curr 键时;不掩盖的理由见
|
||||
_categorize_pair docstring。
|
||||
"""
|
||||
if not comparison_pairs:
|
||||
return "(本轮无可用纵向对比样本)"
|
||||
|
||||
grouped: dict[str, list[dict[str, Any]]] = {key: [] for key, _ in _CATEGORY_LABELS}
|
||||
for pair in comparison_pairs:
|
||||
grouped[_categorize_pair(pair)].append(pair)
|
||||
|
||||
lines: list[str] = [f"固定样本总数:{len(comparison_pairs)}"]
|
||||
for key, label in _CATEGORY_LABELS:
|
||||
entries = grouped[key]
|
||||
lines.append(f"\n### {label}({len(entries)} 题)")
|
||||
if not entries:
|
||||
lines.append("(无)")
|
||||
continue
|
||||
for pair in entries:
|
||||
lines.append(
|
||||
f"- 题目:{pair.get('question', '')}\n"
|
||||
f" 上版预测:{pair.get('prev_prediction', '')} | "
|
||||
f"当前版预测:{pair.get('curr_prediction', '')}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 入口
|
||||
# =========================================================================
|
||||
|
||||
|
||||
async def run_slow_momentum(
|
||||
llm: LLMProvider,
|
||||
diagnose_prompts_dir: Path,
|
||||
skill_content: str,
|
||||
prev_skill: str,
|
||||
prev_guidance: str,
|
||||
comparison_pairs: list[dict[str, Any]],
|
||||
) -> str:
|
||||
"""为单个 skill 生成新的慢更新动量指导。
|
||||
|
||||
参数:
|
||||
llm: LLM 端口(async chat)。
|
||||
diagnose_prompts_dir: 诊断 prompt 目录(根 prompts/,slow_momentum.md 在此)。
|
||||
skill_content: 当前版 skill 正文。
|
||||
prev_skill: 上一版 skill 正文。
|
||||
prev_guidance: 上一轮写下的动量指导。
|
||||
comparison_pairs: 固定样本上两版 rollout 的成对结果(含 question/
|
||||
prev_prediction/curr_prediction/correct_prev/correct_curr)。
|
||||
|
||||
返回:
|
||||
新的动量指导文本;解析失败时保留 prev_guidance。
|
||||
|
||||
关键实现细节:
|
||||
- _format_comparison_pairs 刻意置于 try 块之外(prompt 构造阶段):它对每个
|
||||
pair 取 correct_prev/correct_curr,缺键抛 KeyError 直接向上传播,不被下方
|
||||
针对裁判语义歧义的 except ValueError 吞掉。
|
||||
- 解析失败保留上轮指导:extract_json_from_response 抛 ValueError、缺
|
||||
slow_update_content 字段、或该字段非 str,均视为语义解析失败,返回
|
||||
prev_guidance(判不准时保守保留上轮指导,对标 diagnose 的保护性 fallback)。
|
||||
- P5 边界:仅捕 ValueError 这一语义歧义;llm.chat 的基础设施失败
|
||||
(网络/API 异常)刻意不捕,向上传播,绝不用默认值掩盖。
|
||||
"""
|
||||
system_prompt = (diagnose_prompts_dir / "slow_momentum.md").read_text(encoding="utf-8")
|
||||
# _format_comparison_pairs 刻意置于下方 try 块之外(prompt 构造阶段):它对每个
|
||||
# pair 取 correct_prev/correct_curr,缺键抛 KeyError 直接向上传播,不被下方针对
|
||||
# 裁判语义歧义的 except ValueError 吞掉。异常类型选 KeyError(非 ValueError),
|
||||
# 即便位置疏忽落入 try 也不会被误吞。
|
||||
user_prompt = (
|
||||
f"## 上一版 skill 正文\n{prev_skill}\n\n"
|
||||
f"## 当前版 skill 正文\n{skill_content}\n\n"
|
||||
f"## 上一轮的动量指导\n{prev_guidance}\n\n"
|
||||
f"## 固定样本纵向对比(上版 vs 当前版)\n"
|
||||
f"{_format_comparison_pairs(comparison_pairs)}"
|
||||
)
|
||||
response = await llm.chat(
|
||||
[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_prompt},
|
||||
]
|
||||
)
|
||||
raw = response.content
|
||||
try:
|
||||
parsed = extract_json_from_response(raw)
|
||||
new_guidance = parsed.get("slow_update_content")
|
||||
if not isinstance(new_guidance, str):
|
||||
raise ValueError("slow_update_content 字段缺失或非字符串")
|
||||
except ValueError:
|
||||
logger.warning("慢更新动量解析失败,保留上轮动量指导")
|
||||
return prev_guidance
|
||||
return new_guidance
|
||||
@@ -0,0 +1,479 @@
|
||||
"""五张观测表的落库写入与回读 + step/epoch 报告文件输出。
|
||||
|
||||
合并 TRM4 的 metric_log.py(五表)和 loop_report.py(报告)。
|
||||
|
||||
五张表均经 structured-logging 定义,DDL 与之逐列一致:
|
||||
dual_metric_eval / shadow_gate / holdout_eval / quadrant_pair / gate_evidence。
|
||||
|
||||
公共契约(守 P5):soft/mixed 为 None(invalid,无 span / 诊断失败)时存 NULL,**绝不存 0**——
|
||||
SQLite 对 dict 中 None 值写入即 NULL,分析时按 NULL 跳过。每个写函数内幂等建表
|
||||
(``HarnessLog.create_table`` 用 CREATE TABLE IF NOT EXISTS),run_id/timestamp 列由
|
||||
HarnessLog 自动补。
|
||||
|
||||
报告函数输出 JSON 到 workspace 的 analyses/ 目录,供人工审查诊断 prompt 与进化 prompt。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _read_table(db_path: str, table: str, run_id: str) -> list[dict[str, Any]]:
|
||||
"""纯读某表指定 run 的全部行——不经 HarnessLog 生命周期,避免回读污染 _runs 运行状态。
|
||||
|
||||
HarnessLog.__enter__/__exit__ 会对 run_id 做 INSERT OR IGNORE 并在退出时标 completed;
|
||||
回读指标绝不应改运行状态,故 read_* 一律走本只读连接(仅 SELECT)。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
table: 表名(内部固定常量,非外部输入,无注入风险)。
|
||||
run_id: 过滤的 run ID。
|
||||
|
||||
返回:
|
||||
行 dict 列表;表尚未建(没写过)视为无数据返 []。
|
||||
"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
exists = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
|
||||
).fetchone()
|
||||
if exists is None:
|
||||
return []
|
||||
rows = conn.execute(f"SELECT * FROM {table} WHERE run_id=?", (run_id,)).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 列定义严格对齐 research-wiki/schemas/*.md(run_id/timestamp 由 create_table 自动补)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_DUAL_COLS: dict[str, str] = {
|
||||
"epoch": "INTEGER",
|
||||
"version_kind": "TEXT",
|
||||
"skills_version": "TEXT",
|
||||
"prompts_version": "TEXT",
|
||||
"pool": "TEXT",
|
||||
"hard_acc": "REAL",
|
||||
"soft_score": "REAL",
|
||||
"mixed_score": "REAL",
|
||||
}
|
||||
|
||||
_SHADOW_COLS: dict[str, str] = {
|
||||
"epoch": "INTEGER",
|
||||
"candidate_version": "TEXT",
|
||||
"hard_acc": "REAL",
|
||||
"soft_score": "REAL",
|
||||
"mixed_score": "REAL",
|
||||
"is_mixed_best": "INTEGER",
|
||||
}
|
||||
|
||||
_HOLDOUT_COLS: dict[str, str] = {
|
||||
"epoch": "INTEGER",
|
||||
"version_kind": "TEXT",
|
||||
"hard_acc": "REAL",
|
||||
"soft_score": "REAL",
|
||||
"mixed_score": "REAL",
|
||||
"per_task_type_json": "TEXT",
|
||||
}
|
||||
|
||||
_QUADRANT_COLS: dict[str, str] = {
|
||||
"epoch": "INTEGER",
|
||||
"step": "INTEGER",
|
||||
# question_id 列承载 unit_id(single=question_id,pair=pair_id);
|
||||
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||
"question_id": "TEXT",
|
||||
"task_type": "TEXT",
|
||||
"prev_correct": "INTEGER",
|
||||
"curr_correct": "INTEGER",
|
||||
"category": "TEXT",
|
||||
}
|
||||
|
||||
_GATE_EVIDENCE_COLS: dict[str, str] = {
|
||||
"epoch": "INTEGER",
|
||||
"step": "INTEGER",
|
||||
"task_type": "TEXT",
|
||||
# question_id 列承载 unit_id(single=question_id,pair=pair_id);
|
||||
# 逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||
"question_id": "TEXT",
|
||||
"ladder_rank": "INTEGER",
|
||||
"baseline_correct": "INTEGER",
|
||||
"candidate_correct": "INTEGER",
|
||||
"e_value": "REAL",
|
||||
"stop_reason": "TEXT",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# dual_metric_eval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_dual_metric(
|
||||
db_path: str,
|
||||
*,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
version_kind: str,
|
||||
skills_version: str,
|
||||
prompts_version: str,
|
||||
pool: str,
|
||||
hard_acc: float,
|
||||
soft_score: float | None,
|
||||
mixed_score: float | None,
|
||||
) -> None:
|
||||
"""落 dual_metric_eval 一行:epoch 末关键版本的 hard+soft+mixed 双轨度量。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
run_id: 训练 run ID。
|
||||
epoch: 轮次(1-based)。
|
||||
version_kind: baseline / best_hard / best_mixed / final / slow_candidate
|
||||
(slow_candidate = 慢更新 R2 可能被 revert 的候选,不占 epoch 终值 final 口径)。
|
||||
skills_version / prompts_version: 评估的资源版本。
|
||||
pool: val / test。
|
||||
hard_acc: hard 准确率。
|
||||
soft_score: soft 连续分;invalid 传 None -> 存 NULL。
|
||||
mixed_score: 0.5*hard+0.5*soft;soft 缺失传 None -> 存 NULL。
|
||||
"""
|
||||
from app.harness.log import HarnessLog
|
||||
|
||||
with HarnessLog(db_path, run_id) as log:
|
||||
log.create_table("dual_metric_eval", _DUAL_COLS)
|
||||
log.insert(
|
||||
"dual_metric_eval",
|
||||
{
|
||||
"epoch": epoch,
|
||||
"version_kind": version_kind,
|
||||
"skills_version": skills_version,
|
||||
"prompts_version": prompts_version,
|
||||
"pool": pool,
|
||||
"hard_acc": hard_acc,
|
||||
"soft_score": soft_score,
|
||||
"mixed_score": mixed_score,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def read_dual_metric(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
|
||||
"""回读指定 run 的 dual_metric_eval 全部行(纯读,不污染运行状态)。"""
|
||||
return _read_table(db_path, "dual_metric_eval", run_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# shadow_gate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_shadow_gate(
|
||||
db_path: str,
|
||||
*,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
candidate_version: str,
|
||||
hard_acc: float,
|
||||
soft_score: float | None,
|
||||
mixed_score: float | None,
|
||||
is_mixed_best: bool,
|
||||
) -> None:
|
||||
"""落 shadow_gate 一行:mixed 影子 best 候选的 hard/soft/mixed 及是否 argmax 选中。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
run_id: 训练 run ID。
|
||||
epoch: 轮次(1-based)。
|
||||
candidate_version: 候选版本标识(如 skills/vX+prompts/vY)。
|
||||
hard_acc: hard 准确率。
|
||||
soft_score: soft 连续分;invalid 传 None -> 存 NULL(该版本不进 argmax)。
|
||||
mixed_score: 0.5*hard+0.5*soft;soft 缺失传 None -> 存 NULL。
|
||||
is_mixed_best: 是否本 epoch mixed argmax 选中(存 1/0)。
|
||||
"""
|
||||
from app.harness.log import HarnessLog
|
||||
|
||||
with HarnessLog(db_path, run_id) as log:
|
||||
log.create_table("shadow_gate", _SHADOW_COLS)
|
||||
log.insert(
|
||||
"shadow_gate",
|
||||
{
|
||||
"epoch": epoch,
|
||||
"candidate_version": candidate_version,
|
||||
"hard_acc": hard_acc,
|
||||
"soft_score": soft_score,
|
||||
"mixed_score": mixed_score,
|
||||
"is_mixed_best": int(is_mixed_best),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def read_shadow_gate(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
|
||||
"""回读指定 run 的 shadow_gate 全部行(纯读,不污染运行状态)。"""
|
||||
return _read_table(db_path, "shadow_gate", run_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# holdout_eval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_holdout_eval(
|
||||
db_path: str,
|
||||
*,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
version_kind: str,
|
||||
hard_acc: float,
|
||||
soft_score: float | None,
|
||||
mixed_score: float | None,
|
||||
per_task_type_json: str,
|
||||
) -> None:
|
||||
"""落 holdout_eval 一行:四向 held-out 在 test 池的 hard+soft+mixed 及按题型细分。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
run_id: 训练 run ID。
|
||||
epoch: 轮次(1-based)。
|
||||
version_kind: baseline / best_hard / best_mixed / final。
|
||||
hard_acc: hard 准确率。
|
||||
soft_score: soft 连续分;invalid 传 None -> 存 NULL。
|
||||
mixed_score: 0.5*hard+0.5*soft;soft 缺失传 None -> 存 NULL。
|
||||
per_task_type_json: 按 task_type 的 {accuracy,total,correct} JSON 串。
|
||||
"""
|
||||
from app.harness.log import HarnessLog
|
||||
|
||||
with HarnessLog(db_path, run_id) as log:
|
||||
log.create_table("holdout_eval", _HOLDOUT_COLS)
|
||||
log.insert(
|
||||
"holdout_eval",
|
||||
{
|
||||
"epoch": epoch,
|
||||
"version_kind": version_kind,
|
||||
"hard_acc": hard_acc,
|
||||
"soft_score": soft_score,
|
||||
"mixed_score": mixed_score,
|
||||
"per_task_type_json": per_task_type_json,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def read_holdout_eval(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
|
||||
"""回读指定 run 的 holdout_eval 全部行(纯读,不污染运行状态)。"""
|
||||
return _read_table(db_path, "holdout_eval", run_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# quadrant_pair
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_quadrant_pairs(
|
||||
db_path: str,
|
||||
*,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
step: int,
|
||||
pairs: list[dict[str, Any]],
|
||||
) -> None:
|
||||
"""落 quadrant_pair 多行:fast gate 后按 **unit** 四象限(prev/curr 翻转 + category)落库。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
run_id: 训练 run ID。
|
||||
epoch: 轮次(1-based)。
|
||||
step: epoch 内 step 序号(0-based)。
|
||||
pairs: 每条含 question_id/task_type/prev_correct/curr_correct/category;
|
||||
question_id 字段承载 **unit_id**(single=question_id,pair=pair_id,
|
||||
与 gate e-process 同粒度)——逐题明细在 predictions 表溯源,按 pair_id
|
||||
join 真实 question 表会 join 不上;prev_correct/curr_correct 为 bool,
|
||||
写库前转 0/1。
|
||||
|
||||
关键实现:
|
||||
用 insert_many 批量落库;pairs 为空时只建表不插入(fast gate 无翻转的极端情况)。
|
||||
"""
|
||||
records = [
|
||||
{
|
||||
"epoch": epoch,
|
||||
"step": step,
|
||||
"question_id": pair["question_id"],
|
||||
"task_type": pair["task_type"],
|
||||
"prev_correct": int(pair["prev_correct"]),
|
||||
"curr_correct": int(pair["curr_correct"]),
|
||||
"category": pair["category"],
|
||||
}
|
||||
for pair in pairs
|
||||
]
|
||||
from app.harness.log import HarnessLog
|
||||
|
||||
with HarnessLog(db_path, run_id) as log:
|
||||
log.create_table("quadrant_pair", _QUADRANT_COLS)
|
||||
if records:
|
||||
log.insert_many("quadrant_pair", records)
|
||||
|
||||
|
||||
def read_quadrant_pairs(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
|
||||
"""回读指定 run 的 quadrant_pair 全部行(纯读,不污染运行状态)。"""
|
||||
return _read_table(db_path, "quadrant_pair", run_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# gate_evidence
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_gate_evidence(
|
||||
db_path: str,
|
||||
*,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
step: int,
|
||||
rows: list[dict[str, Any]],
|
||||
) -> None:
|
||||
"""落 gate_evidence 单元行:CE-Gate 每次决策的可回放审计记录(unit 口径)。
|
||||
|
||||
参数:
|
||||
db_path: SQLite 路径。
|
||||
run_id: 训练 run ID。
|
||||
epoch: 该 gate 所属的轮次(1-based)。
|
||||
step: epoch 内 step 序号(0-based)。
|
||||
rows: 每 **单元** 一行,含 question_id/task_type/ladder_rank(阶梯序号,
|
||||
0-based)/baseline_correct/
|
||||
candidate_correct/e_value(该单元判定后的累计 e 值)/
|
||||
stop_reason(仅最后一单元携带最终 stop_reason,其余空串)。
|
||||
question_id 字段承载 **unit_id**(single=question_id,pair=pair_id)——
|
||||
逐题明细在 predictions 表溯源,按 pair_id join 真实 question 表会 join 不上。
|
||||
|
||||
返回:
|
||||
无。
|
||||
|
||||
关键实现:
|
||||
逐行 insert(非 insert_many),保证每行独立事务。
|
||||
"""
|
||||
from app.harness.log import HarnessLog
|
||||
|
||||
with HarnessLog(db_path, run_id) as log:
|
||||
log.create_table("gate_evidence", _GATE_EVIDENCE_COLS)
|
||||
# 幂等迁移(对齐 question_gen/run_store 先例):块序贯时代的旧表只有
|
||||
# block_idx 列,CREATE TABLE IF NOT EXISTS 不补列,直接插 ladder_rank
|
||||
# 会 OperationalError——为旧 workspace 复用补列,新表恒为 no-op。
|
||||
cols = {r["name"] for r in log.query("PRAGMA table_info(gate_evidence)")}
|
||||
if "ladder_rank" not in cols:
|
||||
log.execute("ALTER TABLE gate_evidence ADD COLUMN ladder_rank INTEGER")
|
||||
for row in rows:
|
||||
log.insert("gate_evidence", {"epoch": epoch, "step": step, **row})
|
||||
|
||||
|
||||
def read_gate_evidence(db_path: str, *, run_id: str) -> list[dict[str, Any]]:
|
||||
"""回读指定 run 的 gate_evidence 全部行(纯读,不污染运行状态)。"""
|
||||
return _read_table(db_path, "gate_evidence", run_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 报告函数(从 TRM4 loop_report.py 迁移)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def write_step_report(
|
||||
workspace_dir: Path,
|
||||
epoch: int,
|
||||
step: int,
|
||||
global_step: int,
|
||||
task_type: str,
|
||||
gate_action: str,
|
||||
candidate_acc: float,
|
||||
class_baseline_acc: float,
|
||||
edit_budget: int,
|
||||
rank_clip_triggered: bool,
|
||||
gate_w: int | None,
|
||||
gate_l: int | None,
|
||||
gate_e_value: float | None,
|
||||
gate_n_used: int | None,
|
||||
gate_stop_reason: str | None,
|
||||
) -> Path:
|
||||
"""写单个 (step, task_type) 快路径 gate 的最小观测记录 JSON。
|
||||
|
||||
文件名按 (epoch, step, task_type) 命名,slug 由 task_type 规范化(小写、空格转 '-')得到。
|
||||
|
||||
参数:
|
||||
workspace_dir: 实验工作区目录。
|
||||
epoch: 当前轮次(1-based)。
|
||||
step: epoch 内 step 序号(0-based)。
|
||||
global_step: 全局步计数(驱动 edit_budget 退火)。
|
||||
task_type: 本条 gate 的任务类型。
|
||||
gate_action: 闸门动作(accept_confirmed / accept_provisional / reject /
|
||||
skipped / cooldown)。
|
||||
candidate_acc: 候选在 gate 已观测题上的准确率(观测口径)。
|
||||
class_baseline_acc: 基线在 gate 已观测题上的准确率(观测口径)。
|
||||
edit_budget: 该 step 按 global_step 退火得到的 per-target 编辑预算上限。
|
||||
rank_clip_triggered: 该 skill 进化是否触发了 rank 裁剪。
|
||||
gate_w: e-process 累计 W(基线错->候选对翻转数);skipped/cooldown 路径传 None。
|
||||
gate_l: e-process 累计 L(基线对->候选错翻转数);skipped/cooldown 路径传 None。
|
||||
gate_e_value: 停时的 e 值;skipped/cooldown 路径传 None。
|
||||
gate_n_used: gate 实际消费的阶梯题数;skipped/cooldown 路径传 None。
|
||||
gate_stop_reason: e-process 停止原因;skipped/cooldown 路径传 None。
|
||||
|
||||
返回:
|
||||
写入的 step_report 文件路径。
|
||||
"""
|
||||
report = {
|
||||
"epoch": epoch,
|
||||
"step": step,
|
||||
"global_step": global_step,
|
||||
"task_type": task_type,
|
||||
"gate_action": gate_action,
|
||||
"candidate_acc": candidate_acc,
|
||||
"class_baseline_acc": class_baseline_acc,
|
||||
"edit_budget": edit_budget,
|
||||
"rank_clip_triggered": rank_clip_triggered,
|
||||
"gate_w": gate_w,
|
||||
"gate_l": gate_l,
|
||||
"gate_e_value": gate_e_value,
|
||||
"gate_n_used": gate_n_used,
|
||||
"gate_stop_reason": gate_stop_reason,
|
||||
}
|
||||
slug = task_type.lower().replace(" ", "-")
|
||||
out_dir = workspace_dir / "analyses"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / f"step_report_e{epoch}_s{step}_{slug}.json"
|
||||
path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def write_epoch_report(
|
||||
workspace_dir: Path,
|
||||
epoch: int,
|
||||
system_tool_action: str,
|
||||
momentum_updated_task_types: list[str],
|
||||
best_val_acc: float,
|
||||
) -> Path:
|
||||
"""写 epoch 末慢更新汇总 JSON。
|
||||
|
||||
慢更新无单一 ValidationOutcome,故本函数只落慢更新可观测的最小集:
|
||||
system/tool gate 动作、本 epoch 写过 momentum 的题型、慢更新后的全局 best。
|
||||
|
||||
参数:
|
||||
workspace_dir: 实验工作区目录。
|
||||
epoch: 当前轮次(1-based)。
|
||||
system_tool_action: 慢更新 system/tool 动作(updated / reverted / none)。
|
||||
momentum_updated_task_types: 本 epoch 写过 momentum 的题型列表。
|
||||
best_val_acc: 慢更新后(含 best argmax)的全局 best 验证准确率。
|
||||
|
||||
返回:
|
||||
写入的 epoch_report 文件路径。
|
||||
"""
|
||||
report = {
|
||||
"epoch": epoch,
|
||||
"system_tool_action": system_tool_action,
|
||||
"momentum_updated_task_types": momentum_updated_task_types,
|
||||
"best_val_acc": best_val_acc,
|
||||
}
|
||||
out_dir = workspace_dir / "analyses"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / f"epoch_report_{epoch}.json"
|
||||
path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return path
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,146 @@
|
||||
"""QuestionUnit 组装/展开/校验/单元正确性——pair 契约的唯一入口。
|
||||
|
||||
pool 构建、批处理、推理、评测(Task 3+)均通过本模块聚合/展开孪生对,
|
||||
保证 AR pair 的"两题作为整体调度"契约只在一处实现、fail-fast 暴露非法配对。
|
||||
"""
|
||||
|
||||
from collections import defaultdict
|
||||
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
|
||||
def _assemble_pair(pair_id: str, group: list[GeneratedQuestion]) -> QuestionUnit:
|
||||
"""校验单个 pair 分组的数量/角色并组装为 pair 单元(fail-fast)。
|
||||
|
||||
参数:
|
||||
pair_id: 该分组共享的孪生对标识。
|
||||
group: 归属同一 pair_id 的题目列表。
|
||||
|
||||
返回:
|
||||
kind="pair" 的 QuestionUnit。
|
||||
|
||||
关键实现:
|
||||
- 分组必须恰好 2 条,否则视为孤儿/超员,raise ValueError。
|
||||
- 显式检查 pair_original / pair_mirror 角色齐备且唯一,缺失或重复
|
||||
直接 raise ValueError(防 next(...) 静默 StopIteration)。
|
||||
- 合法孪生对交由 QuestionUnit.from_pair 做 video_id/task_type/flip_axis
|
||||
一致性断言。
|
||||
"""
|
||||
if len(group) != 2:
|
||||
raise ValueError(f"pair {pair_id} 数量={len(group)}≠2(孤儿或超员)")
|
||||
originals = [q for q in group if q.question_role == "pair_original"]
|
||||
mirrors = [q for q in group if q.question_role == "pair_mirror"]
|
||||
if len(originals) != 1 or len(mirrors) != 1:
|
||||
raise ValueError(
|
||||
f"pair {pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)},"
|
||||
"需各恰好 1 条"
|
||||
)
|
||||
return QuestionUnit.from_pair(originals[0], mirrors[0])
|
||||
|
||||
|
||||
def build_units(questions: list[GeneratedQuestion]) -> list[QuestionUnit]:
|
||||
"""将扁平题目列表聚合为单元列表:single 单封、pair 按 pair_id 成对聚合。
|
||||
|
||||
参数:
|
||||
questions: 待聚合的题目列表,可混含 single 与孪生对成员。
|
||||
|
||||
返回:
|
||||
单元列表,先 single 后 pair,顺序稳定(single 保留输入顺序,
|
||||
pair 按首次出现的 pair_id 顺序)。
|
||||
|
||||
关键实现:
|
||||
pair_id 为空 → single 单元;非空 → 归入对应 pair 桶。各 pair 桶的
|
||||
数量/角色校验与组装下沉到 _assemble_pair(fail-fast),本体只做分组
|
||||
与派发。
|
||||
"""
|
||||
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||
singles: list[QuestionUnit] = []
|
||||
for q in questions:
|
||||
if q.pair_id:
|
||||
by_pair[q.pair_id].append(q)
|
||||
else:
|
||||
singles.append(QuestionUnit.from_single(q))
|
||||
|
||||
pairs = [_assemble_pair(pid, qs) for pid, qs in by_pair.items()]
|
||||
return singles + pairs
|
||||
|
||||
|
||||
def flatten_units(units: list[QuestionUnit]) -> list[GeneratedQuestion]:
|
||||
"""将单元列表无损展开回扁平题目列表。
|
||||
|
||||
参数:
|
||||
units: 单元列表。
|
||||
|
||||
返回:
|
||||
展开后的题目列表,保持单元顺序及单元内题目顺序。
|
||||
"""
|
||||
return [q for u in units for q in u.questions]
|
||||
|
||||
|
||||
def validate_units(units: list[QuestionUnit]) -> list[QuestionUnit]:
|
||||
"""校验单元列表结构合法性,通过则原样返回(便于链式调用)。
|
||||
|
||||
参数:
|
||||
units: 待校验单元列表。
|
||||
|
||||
返回:
|
||||
校验通过的原单元列表。
|
||||
|
||||
关键实现:
|
||||
pair 单元必须恰好含 2 题,否则 raise ValueError;single 单元无需额外
|
||||
校验(构造时即为 1 题)。用于消费方在使用前做一道防御闸门。
|
||||
"""
|
||||
for u in units:
|
||||
if u.kind == "pair" and u.size != 2:
|
||||
raise ValueError(f"unit {u.unit_id} pair 不成对(size={u.size})")
|
||||
return units
|
||||
|
||||
|
||||
def unit_correctness(unit: QuestionUnit, per_q: dict[str, bool], *, strict: bool = True) -> bool:
|
||||
"""计算单元级正确性:AR pair 走双向 AND,single 即单题正确性。
|
||||
|
||||
参数:
|
||||
unit: 目标单元。
|
||||
per_q: 题目 question_id → 该题是否作答正确的映射。
|
||||
strict: 缺键策略。True(默认)时以 per_q[q.question_id] 取值,缺任一题
|
||||
触发 KeyError(防静默兜底,强制上游先补齐全部单题结果);False 时以
|
||||
per_q.get(q.question_id, False) 取值,缺键计 False(宽松口径,供池
|
||||
构建 / gate 冷启动 / 采样等"缺基线对错即视为未答对"的调用点复用)。
|
||||
|
||||
返回:
|
||||
单元内所有题目均正确时为 True,否则 False。
|
||||
|
||||
关键实现:
|
||||
pool 构建(pools)、gate 冷启动(gate_ladder)、分层采样(loader)三处
|
||||
原各自持有的 loose 版 _unit_correct 副本统一收敛到本函数 strict=False 分支,
|
||||
消除重复逻辑与 missing-key 策略分叉。
|
||||
"""
|
||||
if strict:
|
||||
return all(per_q[q.question_id] for q in unit.questions)
|
||||
return all(per_q.get(q.question_id, False) for q in unit.questions)
|
||||
|
||||
|
||||
def unit_correctness_view(
|
||||
units: list[QuestionUnit], per_q: dict[str, bool], *, strict: bool = True
|
||||
) -> dict[str, bool]:
|
||||
"""把逐题对错折叠成单元级视图:unit_id → 单元是否整体正确。
|
||||
|
||||
进化引擎(gate e-process / quadrant / probation / pair_block / compute_accuracy)
|
||||
统一消费此单元视图,保证 AR pair 双向 AND、非 AR single 单题,混格池中
|
||||
孪生对折叠为一个单元、不被 P/Q 单题计分污染(核心算法保真 #5)。
|
||||
|
||||
参数:
|
||||
units: 目标单元列表(single 或 pair)。
|
||||
per_q: 题目 question_id → 该题是否作答正确(唯一逐题溯源来源)。
|
||||
strict: 缺键策略,透传给 unit_correctness。True(默认)缺任一题 raise
|
||||
KeyError;False 缺键计 False(宽松口径)。
|
||||
|
||||
返回:
|
||||
unit_id → 单元级正确性。single 的 unit_id 等于其 question_id,
|
||||
pair 的 unit_id 等于共享 pair_id。
|
||||
|
||||
关键实现:
|
||||
逐单元复用 unit_correctness(strict 透传),默认 strict 禁静默兜底、
|
||||
强制上游先补齐全部单题结果。
|
||||
"""
|
||||
return {u.unit_id: unit_correctness(u, per_q, strict=strict) for u in units}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,59 @@
|
||||
"""结果驱动视频级切分的冻结溯源 manifest。
|
||||
|
||||
冻结的 pools.json 是切分产物;manifest 记录产出这份切分的关键输入
|
||||
(baseline_run_id、诊断指纹、随机种子、配置)与 pools.json 的内容指纹
|
||||
(pools_sha256),供后续 build_split 写溯源、以及复现校验时比对。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from app.harness.pools import _atomic_write_json
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def write_manifest(
|
||||
path: Path,
|
||||
*,
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
seed: int,
|
||||
config: dict,
|
||||
pools_json_text: str,
|
||||
coverage_report: dict,
|
||||
generated_at: str,
|
||||
) -> dict:
|
||||
"""写切分冻结溯源 manifest(原子写),返回写入的 dict。
|
||||
|
||||
pools_sha256 = sha256(pools_json_text),供复现时校验冻结的 pools.json 内容
|
||||
是否与本次切分一致。generated_at 由调用方传入(库内不用 datetime.now),
|
||||
以保证相同输入产出相同 manifest,可复现。
|
||||
|
||||
参数:
|
||||
path: manifest 目标 JSON 文件路径。
|
||||
baseline_run_id: 产出本次切分所依据的基线 run 标识。
|
||||
diag_fingerprint: 诊断结果指纹(决定 train/val 归属的输入)。
|
||||
seed: 切分使用的随机种子。
|
||||
config: 切分相关配置快照(如 train_ratio 等)。
|
||||
pools_json_text: 冻结的 pools.json 完整文本,用于计算内容指纹。
|
||||
coverage_report: 各类别 train/val 覆盖统计报告。
|
||||
generated_at: 生成时间戳(ISO 字符串),由调用方传入。
|
||||
|
||||
返回:
|
||||
写入 manifest 的 dict(与落盘内容一致)。
|
||||
"""
|
||||
manifest = {
|
||||
"baseline_run_id": baseline_run_id,
|
||||
"diag_fingerprint": diag_fingerprint,
|
||||
"seed": seed,
|
||||
"config": config,
|
||||
"pools_sha256": hashlib.sha256(pools_json_text.encode("utf-8")).hexdigest(),
|
||||
"coverage_report": coverage_report,
|
||||
"generated_at": generated_at,
|
||||
}
|
||||
_atomic_write_json(path, manifest)
|
||||
return manifest
|
||||
@@ -0,0 +1,551 @@
|
||||
"""视频级切分选择:signal 分层、视频聚合、贪心联合约束选择(纯函数)。
|
||||
|
||||
结果驱动切分管线的核心:把诊断信号投影为多样性格子,供贪心选择器最大化覆盖。
|
||||
本模块起步定义 evolution_target 派生与多样性格子;后续追加 score_signal /
|
||||
build_video_records / select_split。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import random
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def diag_fingerprint(prompt_version: str, model: str, code_version: str) -> str:
|
||||
"""由 (诊断 prompt 版本, 模型名, 代码 git 短 SHA) 合成诊断口径指纹。
|
||||
|
||||
诊断信号以 (question_id, baseline_run_id, diag_fingerprint) 为主键持久化,
|
||||
指纹隔离不同诊断配置的信号——换 prompt 版本 / 换模型 / 换代码实现都会得到
|
||||
新指纹,从而 `--force` 用新指纹重跑诊断时**不覆盖旧记录**(旧指纹行仍在),
|
||||
保证不同口径的诊断结果可并存、可回溯、可比对。
|
||||
|
||||
参数:
|
||||
prompt_version: 诊断 prompt 的版本标识(如 prompts/diagnose_*.md 的版本)。
|
||||
model: 执行诊断的模型名(如 "deepseek-v4")。
|
||||
code_version: 诊断代码的版本(约定为 git 短 SHA)。
|
||||
|
||||
返回:
|
||||
16 位十六进制指纹(sha256 截断),对三分量任一变化敏感、对相同三元组确定。
|
||||
|
||||
实现细节:
|
||||
三分量用 "|" 分隔后 sha256,取前 16 位;分隔符防止 ("ab","c") 与 ("a","bc")
|
||||
碰撞成同一指纹。纯函数,相同输入永远同输出,可安全用于主键。
|
||||
"""
|
||||
return hashlib.sha256("|".join([prompt_version, model, code_version]).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
_EVOLUTION_TARGET = {
|
||||
"extraction_failure": "tool",
|
||||
"search_failure": "skill",
|
||||
"reasoning_failure": "skill",
|
||||
"mixed": "system",
|
||||
}
|
||||
|
||||
|
||||
def evolution_target_of(error_type: str) -> str:
|
||||
"""由 error_type 确定性派生进化目标(tool/skill/system)。
|
||||
|
||||
这是报告用的派生标注,非独立多样性轴(多样性主格子=task_type×error_type)。
|
||||
|
||||
参数:
|
||||
error_type: 诊断瀑布归因的错误类别(extraction/search/reasoning/mixed_failure)。
|
||||
|
||||
返回:
|
||||
进化目标字符串 tool / skill / system。
|
||||
|
||||
异常:
|
||||
ValueError: error_type 不在已知集合内(不静默兜底)。
|
||||
"""
|
||||
if error_type not in _EVOLUTION_TARGET:
|
||||
raise ValueError(f"未知 error_type: {error_type}")
|
||||
return _EVOLUTION_TARGET[error_type]
|
||||
|
||||
|
||||
def cell_of(task_type: str, error_type: str) -> tuple[str, str]:
|
||||
"""构造多样性主格子 = (task_type, error_type)。
|
||||
|
||||
参数:
|
||||
task_type: 题型(12 类之一)。
|
||||
error_type: 错误类别(4 类之一)。
|
||||
|
||||
返回:
|
||||
(task_type, error_type) 二元组,作为覆盖计数的格子键。
|
||||
"""
|
||||
return (task_type, error_type)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SignalLabel:
|
||||
"""诊断信号分层标签(DiagnosisResult 的确定性投影)。
|
||||
|
||||
字段:
|
||||
tier: 信号层级,取值 T0 / T1 / T2 / uncertain(判据见 score_signal)。
|
||||
"""
|
||||
|
||||
tier: str
|
||||
|
||||
|
||||
def score_signal(*, cause_category: str | None, infra: bool, degraded: bool) -> SignalLabel:
|
||||
"""把诊断产物投影为信号分层 tier(不发明新分类,是确定性投影)。
|
||||
|
||||
分层优先级顺序固定(用早返回表达,不用魔法权重):
|
||||
先判 INFRA,再判 degraded,然后 defect / lapse,最后兜底 uncertain。
|
||||
|
||||
各层判据来源:
|
||||
T0 — infra=True,即诊断 INFRA 排除(stop_reason ∈ {error, parse_error}),
|
||||
基础设施失败先于一切判定,排除出可训练主体。
|
||||
uncertain — degraded=True(judge 解析失败)或 cause_category 落不到
|
||||
defect/lapse 上(如为 None),信号不可信,排除出 T2。
|
||||
T2 — cause_category == "defect",可训练核心,进多样性覆盖与训练主体。
|
||||
T1 — cause_category == "lapse",低信号(含无解题),接受但不作训练主体。
|
||||
|
||||
参数:
|
||||
cause_category: 诊断的缺陷归因("defect" / "lapse" / None)。
|
||||
infra: 是否被 INFRA 护栏排除(基础设施失败)。
|
||||
degraded: judge 是否解析失败导致诊断降级。
|
||||
|
||||
返回:
|
||||
SignalLabel,其 tier 字段为上述四层之一。
|
||||
|
||||
实现细节:
|
||||
关键字参数强制传入,防止 infra / degraded 两个 bool 位置混淆。
|
||||
"""
|
||||
if infra:
|
||||
return SignalLabel(tier="T0")
|
||||
if degraded:
|
||||
return SignalLabel(tier="uncertain")
|
||||
if cause_category == "defect":
|
||||
return SignalLabel(tier="T2")
|
||||
if cause_category == "lapse":
|
||||
return SignalLabel(tier="T1")
|
||||
return SignalLabel(tier="uncertain")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VideoRecord:
|
||||
"""全视频画像单元(贪心选择器 Task 8 的输入单元)。
|
||||
|
||||
覆盖全部视频(含全对、零诊断信号的视频),既承载 test 代表性所需的难度/题型画像,
|
||||
也叠加 T2 可训练缺陷的多样性格子,供选择器算覆盖与补集。
|
||||
|
||||
字段:
|
||||
video_id: 视频唯一标识。
|
||||
type_set: 该视频所有题的 task_type 集合(去重,画像用)。
|
||||
n_correct: 该视频答对题数。
|
||||
difficulty: 难度画像桶 = 错题数 = 题数 - n_correct。
|
||||
cells: 仅 tier=="T2" 信号行投影的 (task_type, error_type) 主格子并集(去重)。
|
||||
wrong_by_type: 各 task_type 的 T2 计数,供选择器 floor 约束(普通 dict)。
|
||||
|
||||
实现细节:
|
||||
frozen 生成的 __hash__ 会遍历各字段;wrong_by_type 为不可哈希 dict,
|
||||
故显式标注 hash=False 将其排除出哈希,避免 VideoRecord 入 set/dict 键时报错,
|
||||
仍保留其参与相等性比较。
|
||||
"""
|
||||
|
||||
video_id: str
|
||||
type_set: frozenset[str]
|
||||
n_correct: int
|
||||
difficulty: int
|
||||
cells: frozenset[tuple[str, str]]
|
||||
wrong_by_type: dict[str, int] = field(hash=False)
|
||||
|
||||
|
||||
def build_video_records(preds: list[dict], signal_rows: list[dict]) -> list[VideoRecord]:
|
||||
"""由全量 predictions 与诊断信号行构建全视频 VideoRecord 列表。
|
||||
|
||||
先按 video_id 聚合全部 predictions(覆盖全对、零信号视频),再叠加仅 tier=="T2"
|
||||
的诊断信号为多样性格子与 wrong_by_type 计数。非 T2 信号行(T0/T1/uncertain)
|
||||
不计入格子与计数。
|
||||
|
||||
参数:
|
||||
preds: 全量预测行,每行含 video_id / question_id / task_type / correct。
|
||||
每视频含其全部题(不限于错题),correct 为布尔答对标记。
|
||||
signal_rows: 诊断信号行,每行含 question_id / task_type / error_type / tier。
|
||||
诊断只覆盖错题子集,正确题无对应信号行属正常,不视为错误。
|
||||
|
||||
返回:
|
||||
全部视频的 VideoRecord 列表,按视频在 preds 中首次出现顺序排列。
|
||||
无任何 T2 信号的视频其 cells 为空 frozenset、wrong_by_type 为空 dict。
|
||||
|
||||
实现细节:
|
||||
signal_rows 的 question_id 若不在 preds 中则忽略(诊断可能滞后于当前预测集,
|
||||
非数据损坏),不 fail-fast;缺失必需键则按 KeyError 直接暴露(不静默兜底)。
|
||||
|
||||
异常:
|
||||
KeyError: preds 或 signal_rows 行缺少必需键(校验前置,防脏数据静默通过)。
|
||||
"""
|
||||
# Phase 1: 按 video_id 聚合 preds(保持首次出现顺序)。
|
||||
signal_by_qid = {row["question_id"]: row for row in signal_rows}
|
||||
aggregates: dict[str, dict] = {}
|
||||
for pred in preds:
|
||||
video_id = pred["video_id"]
|
||||
bucket = aggregates.setdefault(
|
||||
video_id, {"types": set(), "question_ids": [], "n_correct": 0}
|
||||
)
|
||||
bucket["types"].add(pred["task_type"])
|
||||
bucket["question_ids"].append(pred["question_id"])
|
||||
if pred["correct"]:
|
||||
bucket["n_correct"] += 1
|
||||
|
||||
# Phase 2: 逐视频叠加 T2 信号为格子与 wrong_by_type。
|
||||
records: list[VideoRecord] = []
|
||||
for video_id, bucket in aggregates.items():
|
||||
cells: set[tuple[str, str]] = set()
|
||||
wrong_by_type: dict[str, int] = {}
|
||||
for question_id in bucket["question_ids"]:
|
||||
row = signal_by_qid.get(question_id)
|
||||
if row is None or row["tier"] != "T2":
|
||||
continue
|
||||
task_type = row["task_type"]
|
||||
cells.add(cell_of(task_type, row["error_type"]))
|
||||
wrong_by_type[task_type] = wrong_by_type.get(task_type, 0) + 1
|
||||
n_questions = len(bucket["question_ids"])
|
||||
records.append(
|
||||
VideoRecord(
|
||||
video_id=video_id,
|
||||
type_set=frozenset(bucket["types"]),
|
||||
n_correct=bucket["n_correct"],
|
||||
difficulty=n_questions - bucket["n_correct"],
|
||||
cells=frozenset(cells),
|
||||
wrong_by_type=wrong_by_type,
|
||||
)
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SelectConfig:
|
||||
"""贪心联合约束选择器的实验配置(科研配置,随实验扫动)。
|
||||
|
||||
字段:
|
||||
n_trainval: trainval 目标视频数(多样性阶段的填充上限)。
|
||||
floor_k: 各高信号 task_type 的 T2 defect 数下限(硬约束,floor 阶段满足)。
|
||||
epsilon: test 相对全局的最大允许分布偏差(题型占比 / 难度画像两维,逐桶)。
|
||||
reportable_types: 参与 ε 题型代表性校验的 task_type 集(长尾类型不入约束)。
|
||||
seed: 预洗牌随机种子,仅用于打破等增益平局,保证同 config 同 videos 同解。
|
||||
|
||||
实现细节:
|
||||
floor_k / reportable_types 为不可哈希容器,标 hash=False 排除出自动 __hash__,
|
||||
避免 frozen dataclass 被哈希时报错(本类不作为字典键,仅承载配置)。
|
||||
"""
|
||||
|
||||
n_trainval: int
|
||||
floor_k: dict[str, int] = field(hash=False)
|
||||
epsilon: float
|
||||
reportable_types: frozenset[str] | set[str] = field(hash=False)
|
||||
seed: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SplitAssignment:
|
||||
"""视频级切分归属结果(交给 split_by_video_assignment 做题级切分)。
|
||||
|
||||
字段:
|
||||
trainval: 进入 trainval 的 video_id 元组(按选择顺序,确定性)。
|
||||
test: 补集视频的 video_id 元组(按 videos 原始顺序)。
|
||||
"""
|
||||
|
||||
trainval: tuple[str, ...]
|
||||
test: tuple[str, ...]
|
||||
|
||||
|
||||
class InfeasibleSplitError(Exception):
|
||||
"""floor 硬约束与 ε 守护死锁、无法在不破坏 test 代表性下满足 floor 时抛出。
|
||||
|
||||
fail loud(P5):不静默兜底、不随机塞题,直接暴露不可行并报告未达标类型。
|
||||
"""
|
||||
|
||||
|
||||
def derive_reportable_types(total_by_type: dict[str, int], report_floor: int) -> set[str]:
|
||||
"""派生可 per-type 报告的 task_type 集(长尾处理:题数 ≥ report_floor 才报告)。
|
||||
|
||||
参数:
|
||||
total_by_type: 各 task_type 的总题数(或代理承载数)。
|
||||
report_floor: 报告门限,低于此的类型并入长尾、不单独报告也不入 ε 约束。
|
||||
|
||||
返回:
|
||||
总题数 ≥ report_floor 的 task_type 集合。
|
||||
"""
|
||||
return {task_type for task_type, total in total_by_type.items() if total >= report_floor}
|
||||
|
||||
|
||||
def _type_membership_fraction(records: list[VideoRecord], keys: set[str]) -> dict[str, float]:
|
||||
"""计算各 task_type 在给定视频集中的承载占比(含该题型的视频数 / 总视频数)。
|
||||
|
||||
参数:
|
||||
records: 视频记录子集(非空,调用方保证)。
|
||||
keys: 需计算占比的 task_type 键集。
|
||||
|
||||
返回:
|
||||
{task_type: 占比},占比 ∈ [0, 1]。
|
||||
"""
|
||||
total = len(records)
|
||||
return {key: sum(1 for r in records if key in r.type_set) / total for key in keys}
|
||||
|
||||
|
||||
def _difficulty_fraction(records: list[VideoRecord], buckets: set[int]) -> dict[int, float]:
|
||||
"""计算各难度桶在给定视频集中的占比(难度 = 错题数)。
|
||||
|
||||
参数:
|
||||
records: 视频记录子集(非空,调用方保证)。
|
||||
buckets: 需计算占比的难度桶键集。
|
||||
|
||||
返回:
|
||||
{难度桶: 占比},占比 ∈ [0, 1]。
|
||||
"""
|
||||
total = len(records)
|
||||
return {bucket: sum(1 for r in records if r.difficulty == bucket) / total for bucket in buckets}
|
||||
|
||||
|
||||
def _max_deviation(global_dist: dict, subset_dist: dict, keys: set) -> float:
|
||||
"""逐键取全局与子集分布的最大绝对偏差(键集为空时约定为 0.0)。
|
||||
|
||||
参数:
|
||||
global_dist: 全局分布(键 → 占比)。
|
||||
subset_dist: 子集分布(键 → 占比)。
|
||||
keys: 参与比较的键集。
|
||||
|
||||
返回:
|
||||
逐键 |global - subset| 的最大值;keys 为空返回 0.0。
|
||||
"""
|
||||
if not keys:
|
||||
return 0.0
|
||||
return max(abs(global_dist.get(k, 0.0) - subset_dist.get(k, 0.0)) for k in keys)
|
||||
|
||||
|
||||
def _epsilon_ok(
|
||||
test_video_records: list[VideoRecord],
|
||||
videos_all: list[VideoRecord],
|
||||
config: SelectConfig,
|
||||
) -> bool:
|
||||
"""校验 test 子集相对全局在题型占比与难度画像两维的偏差是否均 ≤ epsilon。
|
||||
|
||||
test 越简单则 headline 越虚高,故 test 必须保持代表性:逐 reportable 题型、逐难度桶
|
||||
比较 test 与全局占比,任一维超 epsilon 即判不合格。
|
||||
|
||||
参数:
|
||||
test_video_records: 候选 test 子集(trainval 补集)。
|
||||
videos_all: 全部视频(全局分布基准)。
|
||||
config: 选择配置,提供 epsilon 与 reportable_types。
|
||||
|
||||
返回:
|
||||
两维最大偏差均 ≤ epsilon 为 True;test 为空视为不合格返回 False。
|
||||
"""
|
||||
if not test_video_records:
|
||||
return False
|
||||
type_keys = set(config.reportable_types)
|
||||
global_type = _type_membership_fraction(videos_all, type_keys)
|
||||
subset_type = _type_membership_fraction(test_video_records, type_keys)
|
||||
if _max_deviation(global_type, subset_type, type_keys) > config.epsilon:
|
||||
return False
|
||||
diff_keys = {r.difficulty for r in videos_all}
|
||||
global_diff = _difficulty_fraction(videos_all, diff_keys)
|
||||
subset_diff = _difficulty_fraction(test_video_records, diff_keys)
|
||||
return _max_deviation(global_diff, subset_diff, diff_keys) <= config.epsilon
|
||||
|
||||
|
||||
def _current_wrong_counts(selected: list[VideoRecord]) -> dict[str, int]:
|
||||
"""聚合已选 trainval 视频的 T2 defect 计数(供 floor 达标判定)。
|
||||
|
||||
参数:
|
||||
selected: 当前已进入 trainval 的视频记录。
|
||||
|
||||
返回:
|
||||
{task_type: T2 defect 累计数}。
|
||||
"""
|
||||
counts: dict[str, int] = {}
|
||||
for video in selected:
|
||||
for task_type, wrong in video.wrong_by_type.items():
|
||||
counts[task_type] = counts.get(task_type, 0) + wrong
|
||||
return counts
|
||||
|
||||
|
||||
def _unmet_floors(selected: list[VideoRecord], floor_k: dict[str, int]) -> dict[str, int]:
|
||||
"""计算尚未达标的 floor 类型及其缺口(已达标类型不返回)。
|
||||
|
||||
参数:
|
||||
selected: 当前已进入 trainval 的视频记录。
|
||||
floor_k: 各高信号 task_type 的 defect 下限。
|
||||
|
||||
返回:
|
||||
{task_type: 缺口数},仅含 current < floor 的类型;全达标返回空 dict。
|
||||
"""
|
||||
counts = _current_wrong_counts(selected)
|
||||
return {
|
||||
task_type: floor - counts.get(task_type, 0)
|
||||
for task_type, floor in floor_k.items()
|
||||
if counts.get(task_type, 0) < floor
|
||||
}
|
||||
|
||||
|
||||
def _floor_fill_count(video: VideoRecord, deficits: dict[str, int]) -> int:
|
||||
"""计算某视频能填补的 floor 缺口槽数(逐类型取 min(defect, 缺口) 求和)。
|
||||
|
||||
参数:
|
||||
video: 候选视频记录。
|
||||
deficits: 各未达标类型的缺口。
|
||||
|
||||
返回:
|
||||
该视频实际可填的槽数总和(0 表示对当前缺口无贡献)。
|
||||
"""
|
||||
return sum(
|
||||
min(video.wrong_by_type.get(task_type, 0), deficit)
|
||||
for task_type, deficit in deficits.items()
|
||||
)
|
||||
|
||||
|
||||
def _marginal_gain(video: VideoRecord, current_cells: set[tuple[str, str]]) -> int:
|
||||
"""计算把某视频移入 trainval 的边际覆盖增益(新开的 T2 格子数)。
|
||||
|
||||
参数:
|
||||
video: 候选视频记录。
|
||||
current_cells: 当前 trainval 的 T2 格子并集。
|
||||
|
||||
返回:
|
||||
video.cells 相对 current_cells 的新增格子数(去重)。
|
||||
"""
|
||||
return len(video.cells - current_cells)
|
||||
|
||||
|
||||
def _prospective_test(pool: list[VideoRecord], candidate: VideoRecord) -> list[VideoRecord]:
|
||||
"""构造"把候选移入 trainval 后"的 test 子集 = 当前剩余池去掉候选。
|
||||
|
||||
参数:
|
||||
pool: 当前尚未进入 trainval 的视频(即当前 test 补集)。
|
||||
candidate: 拟移入 trainval 的候选视频。
|
||||
|
||||
返回:
|
||||
pool 去掉 candidate 后的视频列表。
|
||||
"""
|
||||
return [r for r in pool if r.video_id != candidate.video_id]
|
||||
|
||||
|
||||
def _satisfy_floors(
|
||||
selected: list[VideoRecord],
|
||||
pool: list[VideoRecord],
|
||||
videos_all: list[VideoRecord],
|
||||
config: SelectConfig,
|
||||
) -> None:
|
||||
"""Floor 阶段:硬约束优先,逐步移入能填 floor 槽且不破 ε 的视频(就地改 selected/pool)。
|
||||
|
||||
每轮取未达标类型的缺口,候选 = 能填 ≥1 槽 且 移入后 test 仍满足 ε 的视频;候选为空即
|
||||
死锁抛 InfeasibleSplitError;否则选填槽最多者(等槽数按预洗牌顺序取首个,确定性)。
|
||||
|
||||
n_trainval 是硬预算:floor 需求超出预算(尚有缺口却已达 n_trainval)也判不可行 fail loud,
|
||||
保证返回的 trainval 永不超过 n_trainval(不因硬约束悄悄超额、挤占 test)。
|
||||
|
||||
参数:
|
||||
selected: 当前 trainval(就地追加)。
|
||||
pool: 当前剩余池 = test 补集(就地移除)。
|
||||
videos_all: 全部视频(ε 全局基准)。
|
||||
config: 选择配置。
|
||||
|
||||
异常:
|
||||
InfeasibleSplitError: 存在未达标类型但无候选可在不破 ε 下填补,
|
||||
或 floor 需求超过 n_trainval 预算。
|
||||
"""
|
||||
while True:
|
||||
deficits = _unmet_floors(selected, config.floor_k)
|
||||
if not deficits:
|
||||
return
|
||||
if len(selected) >= config.n_trainval:
|
||||
raise InfeasibleSplitError(
|
||||
f"floor 需求超过 n_trainval={config.n_trainval} 预算,剩余缺口: {dict(deficits)}"
|
||||
)
|
||||
candidates = [
|
||||
video
|
||||
for video in pool
|
||||
if _floor_fill_count(video, deficits) > 0
|
||||
and _epsilon_ok(_prospective_test(pool, video), videos_all, config)
|
||||
]
|
||||
if not candidates:
|
||||
raise InfeasibleSplitError(
|
||||
f"floor 无法在 ε≤{config.epsilon} 下满足,未达标类型缺口: {dict(deficits)}"
|
||||
)
|
||||
pick = max(candidates, key=lambda video: _floor_fill_count(video, deficits))
|
||||
selected.append(pick)
|
||||
pool.remove(pick)
|
||||
|
||||
|
||||
def _maximize_diversity(
|
||||
selected: list[VideoRecord],
|
||||
pool: list[VideoRecord],
|
||||
videos_all: list[VideoRecord],
|
||||
config: SelectConfig,
|
||||
) -> None:
|
||||
"""多样性阶段:submodular 贪心,按边际覆盖增益降序填至 n_trainval(就地改 selected/pool)。
|
||||
|
||||
每轮对剩余视频算新开格子数,按 -增益稳定排序(等增益按预洗牌顺序),取第一个移入后 test
|
||||
仍满足 ε 的视频;若无任一视频可加而不破 ε,则停并记 warning(欠额,不静默不报错)。
|
||||
|
||||
参数:
|
||||
selected: 当前 trainval(就地追加)。
|
||||
pool: 当前剩余池 = test 补集(就地移除)。
|
||||
videos_all: 全部视频(ε 全局基准)。
|
||||
config: 选择配置。
|
||||
"""
|
||||
while len(selected) < config.n_trainval:
|
||||
if not pool:
|
||||
logger.warning(
|
||||
"多样性阶段剩余池耗尽,trainval 欠额: {}/{}", len(selected), config.n_trainval
|
||||
)
|
||||
return
|
||||
current_cells = set().union(*(v.cells for v in selected)) if selected else set()
|
||||
ranked = sorted(pool, key=lambda video: -_marginal_gain(video, current_cells))
|
||||
pick = next(
|
||||
(
|
||||
video
|
||||
for video in ranked
|
||||
if _epsilon_ok(_prospective_test(pool, video), videos_all, config)
|
||||
),
|
||||
None,
|
||||
)
|
||||
if pick is None:
|
||||
logger.warning(
|
||||
"多样性阶段 ε 守护阻断全部候选,trainval 欠额: {}/{}",
|
||||
len(selected),
|
||||
config.n_trainval,
|
||||
)
|
||||
return
|
||||
selected.append(pick)
|
||||
pool.remove(pick)
|
||||
|
||||
|
||||
def select_split(videos: list[VideoRecord], *, config: SelectConfig) -> SplitAssignment:
|
||||
"""贪心联合约束视频级切分:floor 硬约束先满足、多样性覆盖后最大化、ε 守护 test 代表性。
|
||||
|
||||
核心洞察:全数据集错题总数固定,越把信号塞 trainval、test 越简单、headline 越虚高,
|
||||
故 test 必须保持代表性(ε 约束),trainval 只靠 floor + 多样性覆盖富集,不从 test 偷难题。
|
||||
|
||||
两阶段贪心(均带 ε 守护):先 Floor 阶段满足各高信号类型 defect 下限(不可行 fail loud),
|
||||
再多样性阶段按边际覆盖增益填至 n_trainval(欠额记 warning)。test = trainval 补集。
|
||||
|
||||
确定性:入场用 random.Random(seed) 对视频列表做一次预洗牌,此后 max / 稳定排序仅取首个,
|
||||
seed 只打破等增益 / 等槽数平局;同 config 同 videos → 同结果。
|
||||
|
||||
参数:
|
||||
videos: 全部视频记录(Task 7 build_video_records 产物)。
|
||||
config: 选择配置(关键字传入,含 n_trainval / floor_k / epsilon / reportable_types / seed)。
|
||||
|
||||
返回:
|
||||
SplitAssignment,trainval 按选择顺序、test 按 videos 原始顺序。
|
||||
|
||||
异常:
|
||||
InfeasibleSplitError: videos 为空、floor 与 ε 死锁无法满足、
|
||||
或 floor 需求超过 n_trainval 预算。
|
||||
"""
|
||||
if not videos:
|
||||
raise InfeasibleSplitError("videos 为空,无法执行切分")
|
||||
rng = random.Random(config.seed)
|
||||
pool = list(videos)
|
||||
rng.shuffle(pool)
|
||||
selected: list[VideoRecord] = []
|
||||
_satisfy_floors(selected, pool, videos, config)
|
||||
_maximize_diversity(selected, pool, videos, config)
|
||||
trainval_ids = {video.video_id for video in selected}
|
||||
trainval = tuple(video.video_id for video in selected)
|
||||
test = tuple(video.video_id for video in videos if video.video_id not in trainval_ids)
|
||||
return SplitAssignment(trainval=trainval, test=test)
|
||||
@@ -0,0 +1,51 @@
|
||||
"""把 predictions.steps_json 转成 RunLog.get_traces 的行形。
|
||||
|
||||
infer_adhoc 的 traces 表为空,轨迹存于 steps_json({thought, tool_call, tool_output})。
|
||||
诊断管线经 get_traces 消费轨迹,故需此确定性转换适配。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
def steps_json_to_trace_rows(
|
||||
video_id: str, question_id: str, steps_json: str
|
||||
) -> list[dict[str, Any]]:
|
||||
"""将单题 steps_json 解析为 trace 行列表(step 从 0 递增)。
|
||||
|
||||
参数:
|
||||
video_id: 视频 ID。
|
||||
question_id: 题 ID。
|
||||
steps_json: predictions.steps_json 原文(JSON 数组字符串)。
|
||||
|
||||
返回:
|
||||
行字典列表,字段对齐 traces 表 schema;空/空数组返回 []。
|
||||
|
||||
关键实现细节:
|
||||
- 工具名读 tool_call.tool(infer_adhoc 真实字段),对极少数历史
|
||||
数据的 name 做 back-compat 回退。
|
||||
- steps_json 非 JSON 数组时直接报错,不做兜底掩盖。
|
||||
"""
|
||||
if not steps_json or not steps_json.strip():
|
||||
return []
|
||||
steps = json.loads(steps_json)
|
||||
if not isinstance(steps, list):
|
||||
raise ValueError(f"steps_json 非数组: {question_id}")
|
||||
rows: list[dict[str, Any]] = []
|
||||
for i, s in enumerate(steps):
|
||||
call = s.get("tool_call") or {}
|
||||
rows.append(
|
||||
{
|
||||
"video_id": video_id,
|
||||
"question_id": question_id,
|
||||
"step": i,
|
||||
# infer_adhoc 用 "tool";back-compat 兼容极少数 "name"
|
||||
"tool_name": call.get("tool", call.get("name")),
|
||||
"tool_args": call.get("args", {}),
|
||||
"tool_output": s.get("tool_output"),
|
||||
"thought": s.get("thought"),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
@@ -0,0 +1,402 @@
|
||||
"""Store 版本操作 + Seed 管理。
|
||||
|
||||
Store 存储版本化资源(视频、题目、Skill、Prompt),
|
||||
通过版本号(v1, v2, ...)管理资源的演化历史。
|
||||
Seed 是可复现的实验起点,包含权重快照 + baseline 数据库。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 格式字符串。"""
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
def _parse_version(name: str) -> int:
|
||||
"""解析版本目录名 ``v\\d+`` 为整数。
|
||||
|
||||
参数:
|
||||
name: 版本目录名,如 ``"v1"``、``"v10"``。
|
||||
|
||||
返回:
|
||||
版本号整数。
|
||||
|
||||
异常:
|
||||
ValueError: 版本目录名格式不合法(不匹配 ``v\\d+``)。
|
||||
"""
|
||||
match = re.match(r"v(\d+)$", name)
|
||||
if not match:
|
||||
raise ValueError(f"无效版本号: {name}")
|
||||
return int(match.group(1))
|
||||
|
||||
|
||||
def list_versions(store_dir: Path, resource_type: str) -> list[str]:
|
||||
"""列出 Store 中某类资源的所有版本号,按数字值排序。
|
||||
|
||||
按数字排序保证 v10 排在 v2 后面(而非字典序 v10 < v2)。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
resource_type: 资源类型路径,如 ``"skills"``、``"questions/generated"``。
|
||||
|
||||
返回:
|
||||
排序后的版本号列表,如 ``["v1", "v2", "v10"]``。
|
||||
"""
|
||||
resource_dir = store_dir / resource_type
|
||||
if not resource_dir.is_dir():
|
||||
return []
|
||||
versions = []
|
||||
for entry in resource_dir.iterdir():
|
||||
if entry.is_dir() and re.match(r"v\d+$", entry.name):
|
||||
versions.append(entry.name)
|
||||
return sorted(versions, key=_parse_version)
|
||||
|
||||
|
||||
def next_version(store_dir: Path, resource_type: str) -> str:
|
||||
"""返回某类资源的下一个可用版本号。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
resource_type: 资源类型路径。
|
||||
|
||||
返回:
|
||||
下一个版本号字符串,如 ``"v3"``。
|
||||
"""
|
||||
versions = list_versions(store_dir, resource_type)
|
||||
if not versions:
|
||||
return "v1"
|
||||
latest = _parse_version(versions[-1])
|
||||
return f"v{latest + 1}"
|
||||
|
||||
|
||||
def _write_meta(target_dir: Path, version: str, source: str, **extra: str | None) -> None:
|
||||
"""写入版本元数据文件 ``meta.json``。
|
||||
|
||||
参数:
|
||||
target_dir: 版本目录。
|
||||
version: 版本号。
|
||||
source: 来源标识(``"manual"`` / ``"evolution"`` / ``"auto-gen"``)。
|
||||
**extra: 额外字段(parent, trigger_run, trigger_workspace, description)。
|
||||
"""
|
||||
meta = {
|
||||
"version": version,
|
||||
"created_at": _now_iso(),
|
||||
"parent": extra.get("parent"),
|
||||
"source": source,
|
||||
"trigger_run": extra.get("trigger_run"),
|
||||
"trigger_workspace": extra.get("trigger_workspace"),
|
||||
"description": extra.get("description", ""),
|
||||
}
|
||||
(target_dir / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def advance_version(
|
||||
store_dir: Path,
|
||||
resource_type: str,
|
||||
source_dir: Path,
|
||||
meta: dict,
|
||||
) -> str:
|
||||
"""将 source_dir 的内容写入 Store 的下一个版本目录,写入 meta.json。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
resource_type: 资源类型路径,如 ``"skills"``、``"questions/generated"``。
|
||||
source_dir: 包含新版本资源文件的源目录。
|
||||
meta: 元数据字典,至少包含 ``source`` 字段。
|
||||
|
||||
返回:
|
||||
新版本号字符串,如 ``"v2"``。
|
||||
"""
|
||||
version = next_version(store_dir, resource_type)
|
||||
target = store_dir / resource_type / version
|
||||
shutil.copytree(source_dir, target)
|
||||
_write_meta(
|
||||
target,
|
||||
version,
|
||||
meta.get("source", "manual"),
|
||||
parent=meta.get("parent"),
|
||||
trigger_run=meta.get("trigger_run"),
|
||||
trigger_workspace=meta.get("trigger_workspace"),
|
||||
description=meta.get("description", ""),
|
||||
)
|
||||
logger.info("Store 版本推进: {}/{}", resource_type, version)
|
||||
return version
|
||||
|
||||
|
||||
def init_store(
|
||||
store_dir: Path,
|
||||
videos_source: Path,
|
||||
skills_dir: Path,
|
||||
prompts_dir: Path,
|
||||
) -> None:
|
||||
"""初始化 Store:拷贝视频数据,创建 skills/v1、prompts/v1 和 questions 目录。
|
||||
|
||||
参数:
|
||||
store_dir: Store 目标路径(不得已存在)。
|
||||
videos_source: 视频数据源目录。
|
||||
skills_dir: 初始 Skill 文件目录。
|
||||
prompts_dir: 初始 Prompt 文件目录。
|
||||
|
||||
异常:
|
||||
FileExistsError: Store 目录已存在。
|
||||
"""
|
||||
if store_dir.exists():
|
||||
raise FileExistsError(f"Store 已存在: {store_dir}")
|
||||
store_dir.mkdir(parents=True)
|
||||
shutil.copytree(videos_source, store_dir / "videos")
|
||||
(store_dir / "questions" / "benchmarks").mkdir(parents=True)
|
||||
(store_dir / "questions" / "generated").mkdir(parents=True)
|
||||
shutil.copytree(skills_dir, store_dir / "skills" / "v1")
|
||||
_write_meta(
|
||||
store_dir / "skills" / "v1",
|
||||
"v1",
|
||||
"manual",
|
||||
description="手工创建的初始版本",
|
||||
)
|
||||
shutil.copytree(prompts_dir, store_dir / "prompts" / "v1")
|
||||
_write_meta(
|
||||
store_dir / "prompts" / "v1",
|
||||
"v1",
|
||||
"manual",
|
||||
description="手工创建的初始版本",
|
||||
)
|
||||
logger.info("Store 初始化完成: {}", store_dir)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 种子库(Seed)函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def init_seed(
|
||||
store_dir: Path,
|
||||
name: str,
|
||||
skills_dir: Path,
|
||||
prompts_dir: Path,
|
||||
baseline_db: Path,
|
||||
baseline_run_id: str,
|
||||
parent: str | None,
|
||||
description: str,
|
||||
*,
|
||||
pools_json: Path | None = None,
|
||||
split_manifest: Path | None = None,
|
||||
) -> Path:
|
||||
"""在 store/seeds/<name> 写一个种子:权重 + baseline.db + seed.json。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
name: 种子名(如 ``'initial'``、``'from-evolve-v20'``)。
|
||||
skills_dir: 该版本 Skill 权重源目录。
|
||||
prompts_dir: 该版本 Prompt 权重源目录。
|
||||
baseline_db: 该版本全量记录 db(含 _runs + predictions 行)。
|
||||
baseline_run_id: 全量记录的 run_id,fresh 时注入 build_pools。
|
||||
parent: 来源(initial 为 None)。
|
||||
description: 人类可读说明。
|
||||
pools_json: 可选,冻结切分 pools.json 源路径;提供时拷入 seed 目录,
|
||||
供 fresh 训练时携带冻结切分进 workspace(见 init_workspace_from_seed)。
|
||||
split_manifest: 可选,冻结切分 split_manifest.json 源路径;提供时拷入 seed
|
||||
目录,供加载时校验 pools.json 内容指纹(pools_sha256)。
|
||||
|
||||
返回:
|
||||
种子目录路径。
|
||||
|
||||
异常:
|
||||
FileExistsError: 同名种子已存在(不覆盖)。
|
||||
"""
|
||||
seed_dir = store_dir / "seeds" / name
|
||||
if seed_dir.exists():
|
||||
raise FileExistsError(f"种子已存在,不覆盖: {seed_dir}")
|
||||
seed_dir.mkdir(parents=True)
|
||||
shutil.copytree(skills_dir, seed_dir / "skills")
|
||||
shutil.copytree(prompts_dir, seed_dir / "prompts")
|
||||
shutil.copy2(baseline_db, seed_dir / "baseline.db")
|
||||
if pools_json is not None:
|
||||
shutil.copy2(pools_json, seed_dir / "pools.json")
|
||||
if split_manifest is not None:
|
||||
shutil.copy2(split_manifest, seed_dir / "split_manifest.json")
|
||||
(seed_dir / "seed.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"baseline_run_id": baseline_run_id,
|
||||
"parent": parent,
|
||||
"created_at": _now_iso(),
|
||||
"description": description,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
logger.info("种子创建完成: {}", seed_dir)
|
||||
return seed_dir
|
||||
|
||||
|
||||
def list_seeds(store_dir: Path) -> list[str]:
|
||||
"""列出 store/seeds 下所有种子名(按名排序)。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
|
||||
返回:
|
||||
种子名列表(仅含 seed.json 存在的目录),按名排序。
|
||||
"""
|
||||
seeds_root = store_dir / "seeds"
|
||||
if not seeds_root.is_dir():
|
||||
return []
|
||||
return sorted(e.name for e in seeds_root.iterdir() if (e / "seed.json").exists())
|
||||
|
||||
|
||||
def read_seed(store_dir: Path, name: str) -> dict:
|
||||
"""读取种子 seed.json;不存在则报错。
|
||||
|
||||
参数:
|
||||
store_dir: Store 根目录。
|
||||
name: 种子名。
|
||||
|
||||
返回:
|
||||
seed.json 解析后的字典。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 该种子不存在。
|
||||
"""
|
||||
seed_json = store_dir / "seeds" / name / "seed.json"
|
||||
if not seed_json.exists():
|
||||
raise FileNotFoundError(f"种子不存在: {name}({seed_json})")
|
||||
return json.loads(seed_json.read_text())
|
||||
|
||||
|
||||
def extract_run_db(
|
||||
src_db: Path, dst_db: Path, run_id: str, *, dedupe_per_question: bool = False
|
||||
) -> None:
|
||||
"""从 src_db 抽出某 run_id 的 _runs + predictions 行,写一个最小 db(种子 baseline.db)。
|
||||
|
||||
用源表的**原始 CREATE 语句**重建目标表,保留主键/列类型/约束——
|
||||
``_runs.run_id TEXT PRIMARY KEY`` 是 HarnessLog ``INSERT OR IGNORE`` 去重的依据,
|
||||
若 seed db 丢主键则续训/fresh-bootstrap 的去重失效。
|
||||
|
||||
参数:
|
||||
src_db: 源 harness.db。
|
||||
dst_db: 目标 db(不得已存在)。
|
||||
run_id: 要抽取的 run。
|
||||
dedupe_per_question: True 时 predictions 表每 question_id 仅保留 rowid 最小
|
||||
的首行(对齐 canonical「每 question_id 取第一行 ORDER BY rowid」口径,
|
||||
902→900)。_runs 表不受影响。
|
||||
|
||||
异常:
|
||||
RuntimeError: 源中无该表或无该 run 的行。
|
||||
"""
|
||||
src = sqlite3.connect(src_db)
|
||||
dst = sqlite3.connect(dst_db)
|
||||
try:
|
||||
for table in ("_runs", "predictions"):
|
||||
create_sql = src.execute(
|
||||
"SELECT sql FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table,),
|
||||
).fetchone()
|
||||
if create_sql is None or create_sql[0] is None:
|
||||
raise RuntimeError(f"源 db 无表 {table}")
|
||||
dst.execute(create_sql[0])
|
||||
cols = [r[1] for r in src.execute(f"PRAGMA table_info({table})")]
|
||||
col_sql = ", ".join(cols)
|
||||
if table == "predictions" and dedupe_per_question:
|
||||
rows = src.execute(
|
||||
f"SELECT {col_sql} FROM {table} WHERE run_id=? "
|
||||
"AND rowid IN (SELECT MIN(rowid) FROM predictions "
|
||||
"WHERE run_id=? GROUP BY question_id)",
|
||||
(run_id, run_id),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = src.execute(
|
||||
f"SELECT {col_sql} FROM {table} WHERE run_id=?", (run_id,)
|
||||
).fetchall()
|
||||
if not rows:
|
||||
raise RuntimeError(f"{table} 中无 run_id={run_id} 的行")
|
||||
ph = ", ".join("?" * len(cols))
|
||||
dst.executemany(f"INSERT INTO {table} ({col_sql}) VALUES ({ph})", rows)
|
||||
dst.commit()
|
||||
finally:
|
||||
dst.close()
|
||||
src.close()
|
||||
|
||||
|
||||
def promote_to_seed(
|
||||
workspace_dir: Path,
|
||||
store_dir: Path,
|
||||
version: str,
|
||||
eval_run_id: str,
|
||||
name: str,
|
||||
description: str,
|
||||
) -> Path:
|
||||
"""把 workspace 的指定版本 + 配套 prompts + 指定 eval run 全量记录固化成新种子。
|
||||
|
||||
强校验 eval_run_id 对应的 _runs 行中 skills_version 必须与 version 一致,
|
||||
且 skills_version/prompts_version 均不得为 NULL。
|
||||
|
||||
参数:
|
||||
workspace_dir: 来源 workspace。
|
||||
store_dir: Store 根目录。
|
||||
version: skills 版本号。
|
||||
eval_run_id: canonical eval run(其 _runs 行提供配套 prompts 版本与全量记录)。
|
||||
name: 新种子名(冲突报错不覆盖)。
|
||||
description: 说明。
|
||||
|
||||
返回:
|
||||
新种子目录。
|
||||
|
||||
异常:
|
||||
ValueError: eval_run_id 不存在,或其 skills_version 与 version 不符,或版本为 NULL。
|
||||
FileExistsError: 同名种子已存在(由 init_seed 抛出)。
|
||||
"""
|
||||
con = sqlite3.connect(workspace_dir / "harness.db")
|
||||
con.row_factory = sqlite3.Row
|
||||
try:
|
||||
row = con.execute(
|
||||
"SELECT skills_version, prompts_version FROM _runs WHERE run_id=?",
|
||||
(eval_run_id,),
|
||||
).fetchone()
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
if row is None:
|
||||
raise ValueError(f"eval run 不存在: {eval_run_id}")
|
||||
|
||||
skills_v, prompts_v = row["skills_version"], row["prompts_version"]
|
||||
|
||||
# 强校验——eval run 的版本必须与 --version 一致,且不得为 NULL
|
||||
if skills_v is None or prompts_v is None:
|
||||
raise ValueError(f"eval run {eval_run_id} 的 _runs 版本对为 NULL(未回填?),无法 promote")
|
||||
if skills_v != version:
|
||||
raise ValueError(f"eval run {eval_run_id} 的版本 {skills_v} 与 --version {version} 不符")
|
||||
|
||||
tmp_db = workspace_dir / "_promote_tmp.db"
|
||||
if tmp_db.exists():
|
||||
tmp_db.unlink()
|
||||
extract_run_db(workspace_dir / "harness.db", tmp_db, eval_run_id)
|
||||
try:
|
||||
seed_dir = init_seed(
|
||||
store_dir,
|
||||
name,
|
||||
workspace_dir / "skills" / skills_v,
|
||||
workspace_dir / "prompts" / prompts_v,
|
||||
tmp_db,
|
||||
baseline_run_id=eval_run_id,
|
||||
parent=f"{workspace_dir.name}:{version}",
|
||||
description=description,
|
||||
)
|
||||
finally:
|
||||
tmp_db.unlink()
|
||||
|
||||
logger.info("Promote 完成: {} -> {}", workspace_dir.name, seed_dir)
|
||||
return seed_dir
|
||||
@@ -0,0 +1,89 @@
|
||||
"""诊断侧树读取适配:把嵌套 tree.json 展平成诊断消费的扁平 nodes dict。
|
||||
|
||||
诊断编排(core/evolution/diagnose.py)期望 tree_data 形如
|
||||
{"nodes": {node_id: {card, level, time_range}}},但 TRM5 建树产物
|
||||
store/videos/<vid>/tree.json 是嵌套 {"metadata","roots":[...]}。本模块递归展平,
|
||||
接通 TRM4→TRM5 迁移时断掉的 ground_truth 加载环。
|
||||
|
||||
不走 TreeIndex 对象层:仅 L1Node 有 to_dict(app/tree/index.py:260),L2/L3 为其内部闭包,
|
||||
且 to_dict 输出无 level、L3 用 timestamp 无 time_range。直接遍历 json 更省且零改建树模块。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_tree_nodes(store_dir: Path, video_id: str) -> dict[str, Any]:
|
||||
"""加载单视频 tree.json 并展平成扁平 nodes dict。
|
||||
|
||||
参数:
|
||||
store_dir: store 根目录(含 videos/<video_id>/tree.json)。
|
||||
video_id: 视频标识。
|
||||
|
||||
返回:
|
||||
{"nodes": {node_id: {"card": dict, "level": int, "time_range": list}}}。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: tree.json 不存在(沿用 factory.py fail-loud 先例)。
|
||||
ValueError: roots 非 list 或为空、节点缺 id、或节点既无 time_range 又无 timestamp。
|
||||
|
||||
关键实现:
|
||||
level 由遍历深度赋值(root=1/child=2/孙=3),不解析 node_id——node_id 累积式
|
||||
(..._L1_..._L2_..._L3_)用正则首匹配会把 L2/L3 误判成 1。
|
||||
L3 无 time_range,用 timestamp 合成 [t, t]。
|
||||
"""
|
||||
tree_path = store_dir / "videos" / video_id / "tree.json"
|
||||
if not tree_path.exists():
|
||||
raise FileNotFoundError(f"树索引文件不存在: {tree_path}(诊断需真实树,P5 fail loud)")
|
||||
tree = json.loads(tree_path.read_text(encoding="utf-8"))
|
||||
roots = tree.get("roots")
|
||||
if not isinstance(roots, list) or not roots:
|
||||
raise ValueError(f"树无有效 roots: {tree_path}")
|
||||
|
||||
nodes: dict[str, Any] = {}
|
||||
|
||||
def _walk(node: dict[str, Any], level: int) -> None:
|
||||
node_id = node.get("id")
|
||||
if not isinstance(node_id, str) or not node_id:
|
||||
raise ValueError(f"节点缺 id: {tree_path}")
|
||||
time_range = node.get("time_range")
|
||||
if time_range is None:
|
||||
ts = node.get("timestamp")
|
||||
if ts is None:
|
||||
raise ValueError(
|
||||
f"节点既无 time_range 又无 timestamp(树损坏): {node_id} in {tree_path}"
|
||||
)
|
||||
time_range = [ts, ts]
|
||||
nodes[node_id] = {
|
||||
"card": node.get("card", {}),
|
||||
"level": level,
|
||||
"time_range": time_range,
|
||||
}
|
||||
for child in node.get("children", []) or []:
|
||||
_walk(child, level + 1)
|
||||
|
||||
for root in roots:
|
||||
_walk(root, 1)
|
||||
|
||||
return {"nodes": nodes}
|
||||
|
||||
|
||||
def load_tree_data_for_videos(store_dir: Path, video_ids: list[str]) -> dict[str, Any]:
|
||||
"""按一组 video_id 去重加载展平树,供诊断按 video 注入。
|
||||
|
||||
参数:
|
||||
store_dir: store 根目录。
|
||||
video_ids: 视频标识列表(可含重复,内部按首次出现顺序去重)。
|
||||
|
||||
返回:
|
||||
{video_id: {"nodes": {...}}}。
|
||||
|
||||
异常:
|
||||
同 load_tree_nodes(任一视频树缺失/无效即 fail-loud)。
|
||||
"""
|
||||
return {vid: load_tree_nodes(store_dir, vid) for vid in dict.fromkeys(video_ids)}
|
||||
@@ -0,0 +1,933 @@
|
||||
"""async 连续并发 gate 验证编排 — CE-Gate 局部验证的唯一独立子编排器。
|
||||
|
||||
多题型全部 (单元, 臂) 任务共享题槽并发(validate_skills_concurrent),
|
||||
统计推进不按到达序,而按预声明的阶梯序前缀消费(_advance_prefix):
|
||||
base 臂缓存命中瞬间返回、cand 臂必新鲜跑,两臂延迟不对称,按到达序判定
|
||||
会系统性偏向早到翻转;前缀消费把判定顺序钉回阶梯序,anytime-valid 无条件
|
||||
成立(核心算法保真 #6,语义修订:块序贯 → 阶梯序前缀逐对序贯)。
|
||||
|
||||
基线与候选在同一阶梯前缀上逐单元配对,只数翻转(基线错→候选对 = W,
|
||||
基线对→候选错 = L),每消费一个单元调一次 gate_decision 做四出口判定,
|
||||
过线即冻结、τ 之后的 in-flight 结果整体丢弃。基线侧单元级对错走
|
||||
BaselineCache 内容寻址缓存,miss 才新鲜跑;INFRA 单元不写缓存、从配对剔除。
|
||||
判定逻辑全部在 core/evolution/gate,本模块只负责推理编排与证据收集。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import shutil
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.gate_ladder import BaselineCache, skill_hash
|
||||
from app.harness.question_units import build_units, unit_correctness_view
|
||||
from core.evolution import (
|
||||
INFRA_STOP_REASONS,
|
||||
GateParams,
|
||||
GateVerdict,
|
||||
RejectedEdit,
|
||||
classify_quadrants,
|
||||
gate_decision,
|
||||
pair_block,
|
||||
)
|
||||
|
||||
# INFRA_STOP_REASONS 复用 core.evolution.diagnose 的单一定义(M-2):INFRA 故障
|
||||
# stop_reason(推理侧基础设施错误,非模型答错)在诊断与 gate 两处必须同口径,
|
||||
# 避免各自维护副本致未来漂移。
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.harness.inference import InferenceResult
|
||||
from app.harness.log import HarnessLog
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
|
||||
# gate_decision 的 decision → ValidationOutcome.stop_reason 映射
|
||||
_STOP_REASON_BY_DECISION: dict[str, str] = {
|
||||
"accept_confirmed": "confirmed",
|
||||
"reject_directional": "directional",
|
||||
"reject_futility": "futility",
|
||||
"accept_provisional": "provisional",
|
||||
"reject_inertia": "inertia",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 注入协议
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class RunInferenceFn(Protocol):
|
||||
"""注入的推理函数协议。
|
||||
|
||||
调用方(runner)负责绑定 llm、tool_dispatch_fn、prompt_builder、
|
||||
log、concurrency、max_steps、skill_mode 等共享依赖。
|
||||
validate 侧只传 questions、run_id、skills_dir 三个逐任务变化的参数。
|
||||
"""
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
questions: list[GeneratedQuestion],
|
||||
*,
|
||||
run_id: str,
|
||||
skills_dir: Path,
|
||||
) -> InferenceResult: ...
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 数据类型
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationOutcome:
|
||||
"""CE-Gate 局部验证结果:三态动作 + e-process 证据(单元口径)+ 逐题溯源对错。
|
||||
|
||||
correctness 二轨语义:W/L、准确率、四象限均按 **unit 口径** 统计
|
||||
(AR pair 双向 AND 折叠为一个单元,不被 P/Q 单题计分污染);
|
||||
candidate_correctness 独立保留 **逐题** 对错(只含已观测题,早停后是阶梯前缀
|
||||
子集),accept 时由 runner 按 question_id 粒度增量合并进 state.correctness。
|
||||
"""
|
||||
|
||||
action: str # accept_confirmed | accept_provisional | reject
|
||||
accepted: bool
|
||||
stop_reason: str # confirmed | directional | futility | provisional | inertia
|
||||
e_value: float
|
||||
w: int
|
||||
l: int # noqa: E741
|
||||
n_used: int
|
||||
delta_hat: float
|
||||
delta_shrunk: float
|
||||
baseline_acc: float # 已观测单元上的基线准确率(unit 口径)
|
||||
candidate_acc: float # 已观测单元上的候选准确率(unit 口径)
|
||||
improvements: list[str] = field(default_factory=list)
|
||||
regressions: list[str] = field(default_factory=list)
|
||||
persistent_fails: list[str] = field(default_factory=list)
|
||||
stable_successes: list[str] = field(default_factory=list)
|
||||
candidate_correctness: dict[str, bool] = field(default_factory=dict)
|
||||
evidence_rows: list[dict] = field(default_factory=list) # gate_evidence 逐题行,runner 落库
|
||||
|
||||
|
||||
@dataclass
|
||||
class Probation:
|
||||
"""一个题型的在途试用账本(每题型至多一个)。
|
||||
|
||||
字段:
|
||||
task_type: 题型。
|
||||
anchor_skills_version: 锚版本名(最近一个 CONFIRMED 的 skills 版本)——
|
||||
回滚时恢复该版本中本题型 skill 文件的内容。
|
||||
target_file: 该题型解析后的 skill 文件名。
|
||||
correctness_snapshot: 开账时该题型 val 题的对错快照(回滚时恢复)。
|
||||
opened_step: 开账时的 global_step(观测用)。
|
||||
pending_edits: 试用链上全部候选 edit 的黑名单素材(回滚时整链入黑名单)。
|
||||
"""
|
||||
|
||||
task_type: str
|
||||
anchor_skills_version: str
|
||||
target_file: str
|
||||
correctness_snapshot: dict[str, bool]
|
||||
opened_step: int
|
||||
pending_edits: list[RejectedEdit] = field(default_factory=list)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 同步辅助函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def materialize_candidate_skill(
|
||||
workspace_dir: Path,
|
||||
base_skills_version: str,
|
||||
target_file: str,
|
||||
content: str,
|
||||
) -> Path:
|
||||
"""将候选 skill 正文物化为 workspace 专用临时目录下唯一命名的候选 skills 目录。
|
||||
|
||||
复制基线 skills 目录到 .cand_tmp/ 下的唯一命名临时目录,然后覆写 target_file。
|
||||
构建失败时尽力清理已建临时目录再重抛原始异常。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。基线 skills 从 workspace_dir/skills/<base>
|
||||
复制,临时候选落 workspace_dir/.cand_tmp/。
|
||||
base_skills_version: 基线 skills 版本名。
|
||||
target_file: 被替换的 skill 文件名。
|
||||
content: 候选 skill 文件全文。
|
||||
|
||||
返回:
|
||||
新建的临时候选目录绝对路径。
|
||||
|
||||
契约:
|
||||
构建失败(OSError)时尽力清理已建临时目录再重抛原始异常;
|
||||
清理本身失败记 warning。
|
||||
"""
|
||||
cand_tmp_root = workspace_dir / ".cand_tmp"
|
||||
cand_tmp_root.mkdir(parents=True, exist_ok=True)
|
||||
cand_dir = Path(tempfile.mkdtemp(prefix=f"{base_skills_version}_cand_", dir=cand_tmp_root))
|
||||
try:
|
||||
base_dir = workspace_dir / "skills" / base_skills_version
|
||||
shutil.copytree(base_dir, cand_dir, dirs_exist_ok=True)
|
||||
(cand_dir / target_file).write_text(content, encoding="utf-8")
|
||||
except OSError:
|
||||
try:
|
||||
shutil.rmtree(cand_dir)
|
||||
except OSError as cleanup_err:
|
||||
logger.warning("候选物化失败后清理临时目录也失败 {}: {}", cand_dir, cleanup_err)
|
||||
raise
|
||||
return cand_dir
|
||||
|
||||
|
||||
def _load_run_rows(
|
||||
log: HarnessLog,
|
||||
run_id: str,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""读取单个 run 的逐题预测行并规范化轨迹字段。
|
||||
|
||||
从 predictions 表读取指定 run 的题目级记录,补充 _correct
|
||||
与规范化后的 steps 字段。保持同步(log.query)——仅在推理完成后调用。
|
||||
|
||||
参数:
|
||||
log: HarnessLog 共享实例(用 query 方法做只读 SELECT)。
|
||||
run_id: 待读取的预测 run_id。
|
||||
|
||||
返回:
|
||||
以 question_id 为键的行字典。每行至少包含 prediction、answer、
|
||||
_correct、steps 等字段。
|
||||
"""
|
||||
rows = log.query(
|
||||
"SELECT question_id, prediction, answer, stop_reason, steps_json "
|
||||
"FROM predictions WHERE run_id=?",
|
||||
(run_id,),
|
||||
)
|
||||
normalized: dict[str, dict[str, Any]] = {}
|
||||
for row in rows:
|
||||
raw_steps = row.get("steps_json")
|
||||
parsed_steps: Any = raw_steps
|
||||
if isinstance(raw_steps, str):
|
||||
try:
|
||||
parsed_steps = json.loads(raw_steps)
|
||||
except json.JSONDecodeError:
|
||||
parsed_steps = []
|
||||
steps = parsed_steps if isinstance(parsed_steps, list) else []
|
||||
normalized[row["question_id"]] = {
|
||||
**row,
|
||||
"_correct": row.get("prediction") == row.get("answer"),
|
||||
"steps": steps,
|
||||
}
|
||||
return normalized
|
||||
|
||||
|
||||
def _infra_question_ids_from_db(
|
||||
log: HarnessLog,
|
||||
run_id: str,
|
||||
chunk: list[GeneratedQuestion],
|
||||
) -> set[str]:
|
||||
"""从 db 读取一个 run 中 stop_reason 属 INFRA 故障族的 question_id 集合。
|
||||
|
||||
参数:
|
||||
log: HarnessLog 共享实例。
|
||||
run_id: 推理 run_id。
|
||||
chunk: 题目列表。
|
||||
|
||||
返回:
|
||||
stop_reason ∈ {"error", "parse_error"} 的 question_id 集合。
|
||||
"""
|
||||
rows = _load_run_rows(log, run_id)
|
||||
return {
|
||||
q.question_id
|
||||
for q in chunk
|
||||
if rows.get(q.question_id, {}).get("stop_reason") in INFRA_STOP_REASONS
|
||||
}
|
||||
|
||||
|
||||
def _candidate_correctness_from_db(
|
||||
log: HarnessLog,
|
||||
run_id: str,
|
||||
chunk: list[GeneratedQuestion],
|
||||
) -> dict[str, bool]:
|
||||
"""从 db 读取候选/基线 run 在指定题目上的逐题对错。
|
||||
|
||||
参数:
|
||||
log: HarnessLog 共享实例。
|
||||
run_id: 推理 run_id。
|
||||
chunk: 题目列表。
|
||||
|
||||
返回:
|
||||
question_id -> 是否答对的映射。缺行的题目记为 False。
|
||||
"""
|
||||
rows = _load_run_rows(log, run_id)
|
||||
return {q.question_id: rows.get(q.question_id, {}).get("_correct", False) for q in chunk}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# INFRA 护栏
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_infra_guard(errors: int, infra_denom: int, gate_guard_err: float) -> None:
|
||||
"""累计 INFRA 错误率护栏:分母 >=10 且超阈值时 raise。
|
||||
|
||||
参数:
|
||||
errors: 两侧累计 error 计数。
|
||||
infra_denom: 两侧累计推理题次分母。
|
||||
gate_guard_err: 错误率阈值。
|
||||
|
||||
异常:
|
||||
RuntimeError: 错误率超阈值。
|
||||
"""
|
||||
if infra_denom >= 10 and errors / infra_denom > gate_guard_err:
|
||||
raise RuntimeError(f"gate 推理累计错误率过高 {errors / infra_denom:.0%},中止本轮")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 终态组装
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _finalize_outcome(
|
||||
verdict: GateVerdict,
|
||||
w: int,
|
||||
l: int, # noqa: E741
|
||||
n_used: int,
|
||||
n_plan: int,
|
||||
base_obs: dict[str, bool],
|
||||
cand_obs: dict[str, bool],
|
||||
candidate_per_q: dict[str, bool],
|
||||
evidence_rows: list[dict],
|
||||
task_type: str,
|
||||
) -> ValidationOutcome:
|
||||
"""将终态判定组装为 ValidationOutcome。
|
||||
|
||||
四象限/准确率/W/L 均按单元口径(base_obs/cand_obs 为 unit_id -> bool),
|
||||
candidate_correctness 独立保留逐题溯源(供 runner 二轨合并进 state.correctness)。
|
||||
|
||||
参数:
|
||||
verdict: 终态 gate 判定结果。
|
||||
w: 累计 W(基线错→候选对单元翻转)。
|
||||
l: 累计 L(基线对→候选错单元翻转)。
|
||||
n_used: 已消费的阶梯单元数。
|
||||
n_plan: 阶梯总单元数。
|
||||
base_obs: 累计基线已观测单元对错(unit_id -> bool)。
|
||||
cand_obs: 累计候选已观测单元对错(unit_id -> bool)。
|
||||
candidate_per_q: 累计候选逐题对错(question_id -> bool,溯源用)。
|
||||
evidence_rows: 单元级证据行。
|
||||
task_type: 验证题型(日志用)。
|
||||
|
||||
返回:
|
||||
ValidationOutcome。
|
||||
"""
|
||||
action = {
|
||||
"accept_confirmed": "accept_confirmed",
|
||||
"accept_provisional": "accept_provisional",
|
||||
}.get(verdict.decision, "reject")
|
||||
stop_reason = _STOP_REASON_BY_DECISION[verdict.decision]
|
||||
# 只有终态单元的证据行才携带 stop_reason
|
||||
evidence_rows[-1]["stop_reason"] = stop_reason
|
||||
|
||||
quadrants = classify_quadrants({uid: (base_obs[uid], cand_obs[uid]) for uid in base_obs})
|
||||
baseline_acc = sum(base_obs.values()) / len(base_obs)
|
||||
candidate_acc = sum(cand_obs.values()) / len(cand_obs)
|
||||
accepted = action != "reject"
|
||||
|
||||
logger.info(
|
||||
"gate 局部验证[{}]: 基线{:.1%} → 候选{:.1%} (W={} L={} E={:.2f} n={}/{} 单元) {}",
|
||||
task_type,
|
||||
baseline_acc,
|
||||
candidate_acc,
|
||||
w,
|
||||
l,
|
||||
verdict.e_value,
|
||||
n_used,
|
||||
n_plan,
|
||||
"接受" if accepted else "回滚",
|
||||
)
|
||||
|
||||
return ValidationOutcome(
|
||||
action=action,
|
||||
accepted=accepted,
|
||||
stop_reason=stop_reason,
|
||||
e_value=verdict.e_value,
|
||||
w=w,
|
||||
l=l,
|
||||
n_used=n_used,
|
||||
delta_hat=verdict.delta_hat,
|
||||
delta_shrunk=verdict.delta_shrunk,
|
||||
baseline_acc=baseline_acc,
|
||||
candidate_acc=candidate_acc,
|
||||
improvements=quadrants.improvements,
|
||||
regressions=quadrants.regressions,
|
||||
persistent_fails=quadrants.persistent_fails,
|
||||
stable_successes=quadrants.stable_successes,
|
||||
candidate_correctness=candidate_per_q,
|
||||
evidence_rows=evidence_rows,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主编排
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ladder_units(ladder_items: list[GeneratedQuestion]) -> list[QuestionUnit]:
|
||||
"""把阶梯题序聚合为单元并保持信息阶梯顺序(按单元最早出现位置排序)。
|
||||
|
||||
build_units 会把 single 与 pair 分组重排(single 先、pair 后),破坏"难题优先"
|
||||
的阶梯序;此处按单元内题目在 ladder 中的最早下标重排,恢复原阶梯优先级,
|
||||
保证 AR pair 折叠不改变 e-process 的出题顺序(核心算法保真 #5)。非 AR 全 single
|
||||
时排序为恒等(unit_id 等于 question_id、位置即原序),与迁移前逐题行为一致。
|
||||
|
||||
参数:
|
||||
ladder_items: 阶梯出题序(可混含 single 与 AR pair 成员)。
|
||||
|
||||
返回:
|
||||
按阶梯序排列的单元列表。
|
||||
"""
|
||||
units = build_units(ladder_items)
|
||||
position = {q.question_id: i for i, q in enumerate(ladder_items)}
|
||||
units.sort(key=lambda u: min(position[q.question_id] for q in u.questions))
|
||||
return units
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 连续并发 gate:数据结构 + 前缀消费(algo #6 语义修订:块序贯 → 阶梯序前缀逐对序贯)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateSpec:
|
||||
"""单题型 gate 验证规格(runner 装配阶段产物,调度器输入)。
|
||||
|
||||
字段:
|
||||
task_type: 题型。
|
||||
target_file: 解析后生效 skill 文件名(候选物化写此文件)。
|
||||
candidate_content: 候选 skill 全文。
|
||||
base_skill_content: 基线侧生效 skill 全文(skill_hash 作缓存键)。
|
||||
units: 阶梯序单元元组(已排除案例单元、截断 gate_n_max);元组,装配后
|
||||
不可变,防 spec.units 与 run.slots 漂移。
|
||||
gate_run_prefix: run_id 前缀,必须含 "_gate_"(防泄露过滤依赖)。
|
||||
"""
|
||||
|
||||
task_type: str
|
||||
target_file: str
|
||||
candidate_content: str
|
||||
base_skill_content: str
|
||||
units: tuple[QuestionUnit, ...]
|
||||
gate_run_prefix: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class _UnitSlot:
|
||||
"""单个阶梯单元的双臂到达状态。
|
||||
|
||||
base 为单元级对错(AR pair 已折叠);cand_per_q 为逐题对错(折叠交给消费时,
|
||||
以复用 unit_correctness_view 并保留逐题溯源)。INFRA 标志与结果互斥。
|
||||
"""
|
||||
|
||||
unit: QuestionUnit
|
||||
base: bool | None = None
|
||||
cand_per_q: dict[str, bool] | None = None
|
||||
base_infra: bool = False
|
||||
cand_infra: bool = False
|
||||
|
||||
def resolved(self) -> bool:
|
||||
"""双臂均已出结果(含 INFRA 判定)。
|
||||
|
||||
返回:
|
||||
base 臂(结果或 INFRA)与 cand 臂(结果或 INFRA)都已到达时为 True。
|
||||
"""
|
||||
base_done = self.base is not None or self.base_infra
|
||||
cand_done = self.cand_per_q is not None or self.cand_infra
|
||||
return base_done and cand_done
|
||||
|
||||
def excluded(self) -> bool:
|
||||
"""任一臂 INFRA 即整单元剔除(不入配对)。
|
||||
|
||||
返回:
|
||||
base_infra 或 cand_infra 任一为 True 时为 True。
|
||||
"""
|
||||
return self.base_infra or self.cand_infra
|
||||
|
||||
|
||||
@dataclass
|
||||
class _GateRun:
|
||||
"""单题型 gate 的运行时状态(计数器 + 前缀指针 + 证据)。"""
|
||||
|
||||
spec: GateSpec
|
||||
slots: list[_UnitSlot]
|
||||
s_hash: str
|
||||
prefix_ptr: int = 0
|
||||
w: int = 0
|
||||
l: int = 0 # noqa: E741
|
||||
n_used: int = 0
|
||||
n_excluded: int = 0
|
||||
errors: int = 0
|
||||
infra_denom: int = 0
|
||||
frozen: bool = False
|
||||
verdict: GateVerdict | None = None
|
||||
base_obs: dict[str, bool] = field(default_factory=dict)
|
||||
cand_obs: dict[str, bool] = field(default_factory=dict)
|
||||
candidate_per_q: dict[str, bool] = field(default_factory=dict)
|
||||
evidence_rows: list[dict] = field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_spec(cls, spec: GateSpec) -> _GateRun:
|
||||
"""由规格构造初始状态(slots 与阶梯序一一对应)。
|
||||
|
||||
参数:
|
||||
spec: 单题型 gate 规格(units 已阶梯序)。
|
||||
|
||||
返回:
|
||||
计数器归零、slots 逐单元初始化、s_hash 已计算的 _GateRun。
|
||||
"""
|
||||
return cls(
|
||||
spec=spec,
|
||||
slots=[_UnitSlot(unit=u) for u in spec.units],
|
||||
s_hash=skill_hash(spec.base_skill_content),
|
||||
)
|
||||
|
||||
|
||||
def _advance_prefix(run: _GateRun, params: GateParams) -> None:
|
||||
"""沿阶梯序消费"已配齐前缀",逐单元更新 (W,L) 并判定,过线即冻结。
|
||||
|
||||
统计合法性关键(设计 v3 §1 / Codex C1):严禁按到达序消费——base 臂缓存命中
|
||||
瞬间返回、cand 臂必新鲜跑,两臂延迟不对称,若 cand 延迟与对错相关,早到翻转
|
||||
对系统性偏向 W 型 → e-值虚高假接受。前缀消费把判定顺序钉回预声明阶梯序,
|
||||
anytime-valid 无条件成立;INFRA 单元视为"已解决(剔除)"不阻塞前缀。
|
||||
|
||||
契约:全部单元被剔除时 verdict 保持 None、frozen 保持 False,由调度编排层
|
||||
(Task 3 的 validate_skills_concurrent)检测 verdict None 并 raise
|
||||
RuntimeError;本函数不负责该终态。
|
||||
|
||||
参数:
|
||||
run: 单题型 gate 运行时状态(原地更新计数器/指针/证据)。
|
||||
params: e-process 判据阈值组。
|
||||
|
||||
返回:
|
||||
无(所有效果原地写入 run;可重复调用,已消费前缀不重复消费)。
|
||||
"""
|
||||
while not run.frozen and run.prefix_ptr < len(run.slots):
|
||||
slot = run.slots[run.prefix_ptr]
|
||||
if not slot.resolved():
|
||||
return
|
||||
rank = run.prefix_ptr
|
||||
run.prefix_ptr += 1
|
||||
if slot.excluded():
|
||||
run.n_excluded += 1
|
||||
# 剔除使 n_remaining 缩小,必须重判(Codex plan 审 C1):否则尾部全 INFRA
|
||||
# 时 verdict 停留在 "continue",绕过题尽第四出口且 _finalize_outcome
|
||||
# 查 stop_reason 映射 KeyError。n_used==0(纯前导 INFRA)时无证据可判,跳过。
|
||||
if run.n_used > 0:
|
||||
n_remaining = (len(run.slots) - run.n_excluded) - run.n_used
|
||||
run.verdict = gate_decision(run.w, run.l, run.n_used, n_remaining, params=params)
|
||||
if run.verdict.decision != "continue":
|
||||
run.frozen = True
|
||||
continue
|
||||
uid = slot.unit.unit_id
|
||||
assert slot.base is not None and slot.cand_per_q is not None, (
|
||||
f"slot 未配齐即被消费: unit={slot.unit.unit_id}"
|
||||
)
|
||||
c_units = unit_correctness_view([slot.unit], slot.cand_per_q)
|
||||
pair_result = pair_block({uid: slot.base}, c_units, [uid])
|
||||
run.candidate_per_q.update(slot.cand_per_q)
|
||||
for u, (b, c) in pair_result.observed.items():
|
||||
run.base_obs[u] = b
|
||||
run.cand_obs[u] = c
|
||||
run.w += pair_result.w
|
||||
run.l += pair_result.l
|
||||
run.n_used += 1
|
||||
n_remaining = (len(run.slots) - run.n_excluded) - run.n_used
|
||||
run.verdict = gate_decision(run.w, run.l, run.n_used, n_remaining, params=params)
|
||||
run.evidence_rows.append(
|
||||
{
|
||||
"question_id": uid,
|
||||
"task_type": run.spec.task_type,
|
||||
"ladder_rank": rank,
|
||||
"baseline_correct": slot.base,
|
||||
"candidate_correct": c_units[uid],
|
||||
"e_value": run.verdict.e_value,
|
||||
"stop_reason": "",
|
||||
}
|
||||
)
|
||||
if run.verdict.decision != "continue":
|
||||
run.frozen = True
|
||||
|
||||
|
||||
class _QuestionSlots:
|
||||
"""按题数计数的共享并发闸:峰值在飞请求恒 ≤ width(设计 v3 §2.4)。
|
||||
|
||||
多槽获取(AR pair 一单元两题)经内部锁串行化,防多任务半持有交错死锁。
|
||||
本类只承诺"并发上限 + 多槽获取原子性";公平性由调用方按题型 round-robin
|
||||
顺序创建任务实现(实践中 asyncio 等待队列近似先来先服务,但那不是本类契约)。
|
||||
"""
|
||||
|
||||
def __init__(self, width: int) -> None:
|
||||
"""初始化题槽闸。
|
||||
|
||||
参数:
|
||||
width: 并发宽度(全 gate 同时在飞的题数上限),必须为正。
|
||||
|
||||
返回:
|
||||
无。
|
||||
|
||||
关键实现细节:
|
||||
_width 供 acquire 做超宽 fail-fast;BoundedSemaphore 使多还立即
|
||||
ValueError 而非静默扩容;_acquire_lock 串行化多槽获取防交错死锁。
|
||||
"""
|
||||
assert width > 0, f"并发宽度必须为正: {width}"
|
||||
self._width = width
|
||||
# BoundedSemaphore:多还立即 ValueError 而非静默扩容(Codex 质量审 3)
|
||||
self._sem = asyncio.BoundedSemaphore(width)
|
||||
self._acquire_lock = asyncio.Lock()
|
||||
|
||||
async def acquire(self, n: int) -> None:
|
||||
"""原子获取 n 个题槽。
|
||||
|
||||
fail-fast:n > 宽度时任务持锁等待永不满足的槽位 → 自死锁
|
||||
(AR pair 单元 2 题 + width=1 的病态配置,Codex plan 审 C2),直接报错。
|
||||
取消安全:半持有自动回滚——逐槽获取途中被取消(或任何 BaseException)
|
||||
时,已拿到的 permit 全部归还再重抛,容量不泄漏(Codex 质量审 2)。
|
||||
|
||||
参数:
|
||||
n: 申请的题槽数(单元内题目数,single=1 / AR pair=2)。
|
||||
|
||||
返回:
|
||||
无(成功返回即持有 n 个槽,须与 release(n) 配对)。
|
||||
|
||||
异常:
|
||||
ValueError: n 超过并发宽度(否则自死锁)。
|
||||
"""
|
||||
if n > self._width:
|
||||
raise ValueError(f"单次申请题槽 {n} 超过并发宽度 {self._width},将自死锁")
|
||||
async with self._acquire_lock:
|
||||
got = 0
|
||||
try:
|
||||
for _ in range(n):
|
||||
await self._sem.acquire()
|
||||
got += 1
|
||||
except BaseException:
|
||||
for _ in range(got):
|
||||
self._sem.release()
|
||||
raise
|
||||
|
||||
def release(self, n: int) -> None:
|
||||
"""归还 n 个题槽。
|
||||
|
||||
参数:
|
||||
n: 与 acquire 对应的题槽数。
|
||||
|
||||
返回:
|
||||
无。
|
||||
|
||||
关键实现细节:
|
||||
底层为 BoundedSemaphore——多还(release 数超过 acquire)立即
|
||||
ValueError 暴露调用方配对错误,属防御性设计。
|
||||
"""
|
||||
for _ in range(n):
|
||||
self._sem.release()
|
||||
|
||||
|
||||
async def _run_unit_arm(
|
||||
run: _GateRun,
|
||||
slot_idx: int,
|
||||
arm: str,
|
||||
slots: _QuestionSlots,
|
||||
run_inference: RunInferenceFn,
|
||||
log: HarnessLog,
|
||||
baseline_cache: BaselineCache,
|
||||
prompts_version: str,
|
||||
base_skills_dir: Path,
|
||||
cand_dir: Path,
|
||||
gate_params: GateParams,
|
||||
gate_guard_err: float,
|
||||
) -> None:
|
||||
"""执行一个 (单元, 臂) 任务:缓存/推理 → 到达登记 → 前缀消费推进。
|
||||
|
||||
冻结检查三次:启动时(排队任务撤销点)、获得题槽后(获槽期间被冻结)、
|
||||
推理返回后(τ 之后的 in-flight 结果不计入,整体丢弃)。
|
||||
base 臂缓存命中不占题槽(零推理);INFRA 单元不写缓存(不永久污染基线快照)。
|
||||
护栏在每次臂完成时检查(等价迁移自跨块累计,设计 v3 §2.3),超阈值 raise
|
||||
中止整轮(与现行行为一致)。
|
||||
|
||||
参数:
|
||||
run: 该题型的 gate 运行时状态。
|
||||
slot_idx: 单元在阶梯中的下标。
|
||||
arm: "base" 或 "cand"。
|
||||
slots: 全 gate 共享题槽闸。
|
||||
run_inference: 注入推理函数。
|
||||
log: HarnessLog 共享实例(推理后读预测)。
|
||||
baseline_cache / prompts_version: 基线缓存及键成分。
|
||||
base_skills_dir / cand_dir: 两臂各自的 skills 目录。
|
||||
gate_params: e-process 判据(前缀消费用)。
|
||||
gate_guard_err: INFRA 错误率护栏阈值。
|
||||
|
||||
返回:
|
||||
无(结果写入 run.slots[slot_idx] 并触发 _advance_prefix)。
|
||||
|
||||
异常:
|
||||
RuntimeError: 累计 INFRA 错误率超护栏阈值(经 _check_infra_guard)。
|
||||
"""
|
||||
assert arm in ("base", "cand")
|
||||
if run.frozen:
|
||||
return
|
||||
slot = run.slots[slot_idx]
|
||||
spec = run.spec
|
||||
|
||||
if arm == "base":
|
||||
cached = baseline_cache.get(spec.task_type, run.s_hash, prompts_version, slot.unit.unit_id)
|
||||
if cached is not None:
|
||||
slot.base = cached
|
||||
_advance_prefix(run, gate_params)
|
||||
return
|
||||
|
||||
questions = list(slot.unit.questions)
|
||||
await slots.acquire(len(questions))
|
||||
try:
|
||||
if run.frozen:
|
||||
return
|
||||
run_id = f"{spec.gate_run_prefix}_{arm}"
|
||||
skills_dir = base_skills_dir if arm == "base" else cand_dir
|
||||
r = await run_inference(questions, run_id=run_id, skills_dir=skills_dir)
|
||||
# 推理 await 期间该题型可能已被其他任务判定冻结:设计语义是
|
||||
# "τ(冻结时刻)之后的 in-flight 结果不计入"——整体丢弃,不写
|
||||
# slot/infra_denom/errors,滞后 INFRA 也不得触发护栏 raise 掀翻
|
||||
# 整轮 gather(Codex 质量审 1)。
|
||||
if run.frozen:
|
||||
return
|
||||
_register_arm_arrival(
|
||||
run=run,
|
||||
slot=slot,
|
||||
arm=arm,
|
||||
questions=questions,
|
||||
inference_run_id=r.run_id,
|
||||
inference_total=r.total,
|
||||
log=log,
|
||||
baseline_cache=baseline_cache,
|
||||
prompts_version=prompts_version,
|
||||
)
|
||||
_check_infra_guard(run.errors, run.infra_denom, gate_guard_err)
|
||||
finally:
|
||||
slots.release(len(questions))
|
||||
_advance_prefix(run, gate_params)
|
||||
|
||||
|
||||
def _register_arm_arrival(
|
||||
run: _GateRun,
|
||||
slot: _UnitSlot,
|
||||
arm: str,
|
||||
questions: list[GeneratedQuestion],
|
||||
inference_run_id: str,
|
||||
inference_total: int,
|
||||
log: HarnessLog,
|
||||
baseline_cache: BaselineCache,
|
||||
prompts_version: str,
|
||||
) -> None:
|
||||
"""把一次臂推理结果登记进 slot 与 run 计数器(INFRA 判定 + 对错折叠 + 回写缓存)。
|
||||
|
||||
INFRA 臂只标记不写缓存(不永久污染基线快照);正常 base 臂折叠为单元级对错并
|
||||
回写 BaselineCache,正常 cand 臂保留逐题对错(折叠交给前缀消费,保留逐题溯源)。
|
||||
|
||||
参数:
|
||||
run: 该题型的 gate 运行时状态(errors / infra_denom 原地累加)。
|
||||
slot: 本单元的双臂到达状态(结果或 INFRA 标志原地写入)。
|
||||
arm: "base" 或 "cand"。
|
||||
questions: 本单元展开后的题目列表。
|
||||
inference_run_id: 本次推理的 run_id(DB 回读键)。
|
||||
inference_total: 本次推理的题次数(护栏分母增量)。
|
||||
log: HarnessLog 共享实例(推理后读预测)。
|
||||
baseline_cache / prompts_version: 基线缓存及键成分。
|
||||
|
||||
返回:
|
||||
无(所有效果原地写入 run 与 slot)。
|
||||
|
||||
关键实现细节:
|
||||
errors 按单元级去重(Codex plan 审 I3):同一单元双臂都 INFRA 只计 1 个
|
||||
error,与设计 §2.3"分子=INFRA 单元数(任一臂)"及旧块实现口径一致
|
||||
(旧实现 cand 不跑 base-INFRA 单元,天然无双计)。
|
||||
"""
|
||||
spec = run.spec
|
||||
infra_qids = _infra_question_ids_from_db(log, inference_run_id, questions)
|
||||
run.infra_denom += inference_total
|
||||
if infra_qids:
|
||||
if not slot.excluded():
|
||||
run.errors += 1
|
||||
if arm == "base":
|
||||
slot.base_infra = True
|
||||
else:
|
||||
slot.cand_infra = True
|
||||
return
|
||||
per_q = _candidate_correctness_from_db(log, inference_run_id, questions)
|
||||
if arm == "base":
|
||||
folded = unit_correctness_view([slot.unit], per_q)
|
||||
slot.base = folded[slot.unit.unit_id]
|
||||
baseline_cache.put(
|
||||
spec.task_type, run.s_hash, prompts_version, slot.unit.unit_id, slot.base
|
||||
)
|
||||
else:
|
||||
slot.cand_per_q = per_q
|
||||
|
||||
|
||||
def _validate_gate_specs(specs: list[GateSpec]) -> None:
|
||||
"""校验各题型 gate 规格,不合法直接报错(不兜底)。
|
||||
|
||||
参数:
|
||||
specs: 各题型 gate 规格。
|
||||
|
||||
异常:
|
||||
ValueError: 阶梯为空,或 gate_run_prefix 缺 "_gate_"(防泄露过滤依赖
|
||||
该标记识别 gate run)。
|
||||
"""
|
||||
for spec in specs:
|
||||
if "_gate_" not in spec.gate_run_prefix:
|
||||
raise ValueError(f"gate_run_prefix 必须含 '_gate_': {spec.gate_run_prefix!r}")
|
||||
if not spec.units:
|
||||
raise ValueError(f"task_type={spec.task_type} 阶梯为空,无法验证")
|
||||
|
||||
|
||||
def _cleanup_candidate_dirs(cand_dirs: dict[str, Path]) -> None:
|
||||
"""尽力清理全部候选临时目录,单个失败只记 warning 不中断其余清理。
|
||||
|
||||
参数:
|
||||
cand_dirs: task_type -> 候选临时目录路径。
|
||||
|
||||
返回:
|
||||
无。
|
||||
"""
|
||||
for d in cand_dirs.values():
|
||||
try:
|
||||
shutil.rmtree(d)
|
||||
except OSError as e:
|
||||
logger.warning("候选临时目录清理失败 {}: {}", d, e)
|
||||
|
||||
|
||||
def _build_launch_order(runs: list[_GateRun]) -> list[tuple[_GateRun, int, str]]:
|
||||
"""构建 (run, rank, arm) 发射队列:题型 round-robin × 题型内阶梯序。
|
||||
|
||||
交错顺序 = rank 0 各题型 → rank 1 各题型 → ...;同一 (题型, rank) 内
|
||||
base 先 cand 后。round-robin 让各题型的阶梯头部同批起跑,配合前缀消费
|
||||
使统计推进不因某题型阶梯过长而饿死其他题型。
|
||||
|
||||
参数:
|
||||
runs: 各题型 gate 运行时状态(slots 已按阶梯序初始化)。
|
||||
|
||||
返回:
|
||||
(run, rank, arm) 三元组列表,即任务创建顺序。
|
||||
"""
|
||||
order: list[tuple[_GateRun, int, str]] = []
|
||||
max_rank = max((len(r.slots) for r in runs), default=0)
|
||||
for rank in range(max_rank):
|
||||
for r in runs:
|
||||
if rank < len(r.slots):
|
||||
for arm in ("base", "cand"):
|
||||
order.append((r, rank, arm))
|
||||
return order
|
||||
|
||||
|
||||
async def validate_skills_concurrent(
|
||||
workspace_dir: Path,
|
||||
base_skills_version: str,
|
||||
specs: list[GateSpec],
|
||||
gate_params: GateParams,
|
||||
gate_guard_err: float,
|
||||
baseline_cache: BaselineCache,
|
||||
prompts_version: str,
|
||||
run_inference: RunInferenceFn,
|
||||
log: HarnessLog,
|
||||
concurrency: int,
|
||||
) -> dict[str, ValidationOutcome]:
|
||||
"""连续并发 gate:多题型全部臂共享题槽并发,统计按阶梯序前缀有序推进。
|
||||
|
||||
关键实现细节:
|
||||
发射顺序 = 题型 round-robin × 题型内阶梯序(base 先 cand 后);题型过线
|
||||
即冻结,其排队任务启动时自查冻结标志撤销,in-flight 结果不计入(τ 之后
|
||||
样本,合法丢弃);候选目录逐个物化即登记、统一 finally 清理(中途失败不
|
||||
泄漏);任一任务异常先 cancel+排水其余任务再向上传播;全部题型判定后
|
||||
统一经 _finalize_outcome 组装。
|
||||
|
||||
参数:
|
||||
workspace_dir: workspace 根目录(候选物化用)。
|
||||
base_skills_version: 基线 skills 版本名。
|
||||
specs: 各题型 gate 规格(units 已阶梯序 + 截断 n_max)。
|
||||
gate_params: e-process 判据阈值组。
|
||||
gate_guard_err: INFRA 错误率护栏阈值。
|
||||
baseline_cache: 基线侧单元级对错缓存。
|
||||
prompts_version: 当前 prompts 版本(缓存键成分)。
|
||||
run_inference: 注入推理函数(调用方须绑定共享 HarnessLog)。
|
||||
log: HarnessLog 共享实例(推理后读预测,与 run_inference 同库)。
|
||||
concurrency: 题槽宽度(峰值在飞题数上限)。
|
||||
|
||||
返回:
|
||||
{task_type: ValidationOutcome}。
|
||||
|
||||
异常:
|
||||
RuntimeError: INFRA 护栏超阈值,或某题型全部单元被 INFRA 排除。
|
||||
ValueError: spec 校验失败(空阶梯 / run_prefix 缺 "_gate_")。
|
||||
"""
|
||||
_validate_gate_specs(specs)
|
||||
base_skills_dir = workspace_dir / "skills" / base_skills_version
|
||||
runs = [_GateRun.from_spec(s) for s in specs]
|
||||
cand_dirs: dict[str, Path] = {}
|
||||
slots_gate = _QuestionSlots(concurrency)
|
||||
try:
|
||||
# 成功一个登记一个:第 N 个题型物化抛 OSError 时,已登记的前 N-1 个
|
||||
# 目录仍由 finally 统一清理,不泄漏(Codex 质量审 C001)。
|
||||
for r in runs:
|
||||
cand_dirs[r.spec.task_type] = materialize_candidate_skill(
|
||||
workspace_dir, base_skills_version, r.spec.target_file, r.spec.candidate_content
|
||||
)
|
||||
tasks = [
|
||||
asyncio.ensure_future(
|
||||
_run_unit_arm(
|
||||
r,
|
||||
rank,
|
||||
arm,
|
||||
slots_gate,
|
||||
run_inference,
|
||||
log,
|
||||
baseline_cache,
|
||||
prompts_version,
|
||||
base_skills_dir,
|
||||
cand_dirs[r.spec.task_type],
|
||||
gate_params,
|
||||
gate_guard_err,
|
||||
)
|
||||
)
|
||||
for r, rank, arm in _build_launch_order(runs)
|
||||
]
|
||||
# 护栏 raise 中止整轮的语义不变(Codex 质量审 C002):首异常先取消其余
|
||||
# 任务并排水(return_exceptions 吞取消回报),确保外层 finally 删除候选
|
||||
# 目录时已无在飞任务访问该目录、事件循环收尾无 pending task 警告;
|
||||
# _run_unit_arm 的题槽获取自带取消回滚,cancel 安全。
|
||||
try:
|
||||
await asyncio.gather(*tasks)
|
||||
except BaseException:
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
raise
|
||||
finally:
|
||||
_cleanup_candidate_dirs(cand_dirs)
|
||||
|
||||
outcomes: dict[str, ValidationOutcome] = {}
|
||||
for r in runs:
|
||||
if r.verdict is None:
|
||||
raise RuntimeError(
|
||||
f"gate[{r.spec.task_type}] 全部 unit 被判为 INFRA 排除,无法验证(检查推理基础设施)"
|
||||
)
|
||||
outcomes[r.spec.task_type] = _finalize_outcome(
|
||||
verdict=r.verdict,
|
||||
w=r.w,
|
||||
l=r.l,
|
||||
n_used=r.n_used,
|
||||
n_plan=len(r.slots),
|
||||
base_obs=r.base_obs,
|
||||
cand_obs=r.cand_obs,
|
||||
candidate_per_q=r.candidate_per_q,
|
||||
evidence_rows=r.evidence_rows,
|
||||
task_type=r.spec.task_type,
|
||||
)
|
||||
return outcomes
|
||||
@@ -0,0 +1,824 @@
|
||||
"""结果驱动视频级切分的自包含两阶段 CLI 入口。
|
||||
|
||||
把整条离线管线的编排从 shell 搬进 Python:一次调用内联串起
|
||||
Phase 1 离线诊断(run_baseline_diagnosis,LLM 重活,断点续跑幂等)→
|
||||
Phase 2 冻结切分(build_split,纯 code-controlled,产出 pools.json + manifest)→
|
||||
McNemar 功效护栏(validation 池错题数达阈校验)。
|
||||
|
||||
复现锚点约定(C-2):
|
||||
- pools.json 的内容(+ seed + diag_fingerprint)是切分的**复现锚点**——相同输入
|
||||
产出字节级相同的 pools.json 与 pools_sha256。
|
||||
- manifest 的 generated_at 是**溯源元数据**,非复现锚点:真实运行默认盖真实 UTC
|
||||
now(记录本次切分何时产出),但可用 `--generated-at <ISO>` 显式固定,以对
|
||||
manifest 做字节级复现比对。write_manifest 库内不调 datetime.now,时间戳一律由
|
||||
本 CLI 传入。
|
||||
|
||||
设计要点:
|
||||
- 诊断口径指纹 = (诊断 prompt 版本, 模型名, git 短 SHA) 三分量合成,隔离不同
|
||||
诊断配置的信号;换 prompt / 模型 / 代码实现即换指纹,旧信号不被覆盖。
|
||||
- 真实依赖组装参考 app/harness/runner.py::_run_diagnosis:GovernedLLMClient
|
||||
(search llm, thinking=True) + RunLogImpl(harness.db) + VersionedSkillStore +
|
||||
DiagnosePrompts(项目根 prompts/) + tree_data 按 wrong_ids 涉及 video 预加载
|
||||
(store/videos/<vid>/tree.json 展平)。
|
||||
- 缺 .env / config 关键项一律 fail loud(P5),绝不静默兜底。
|
||||
- `--dry-run` 用假 deps 跑通两阶段 wiring 不真调 LLM,打印将执行的步骤 + 指纹,
|
||||
用于校验装配正确性(对齐 CLAUDE.md §2.5 smoke test)。
|
||||
|
||||
编排函数(run_pipeline)通过依赖注入接收 DiagnosisDeps / signal_store / wrong_ids /
|
||||
questions / canonical_preds,便于单测用假实现替换、不触真实 LLM 与 harness.db。
|
||||
其中 canonical_preds 供 Phase 0 补 INFRA / 空预测错题的 T0 信号(这些题不进诊断)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import datetime
|
||||
import os
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import yaml
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.baseline_diagnosis import DiagnosisDeps, run_baseline_diagnosis
|
||||
from app.harness.build_split import (
|
||||
SplitBuildConfig,
|
||||
SplitBuildResult,
|
||||
build_split,
|
||||
load_canonical_predictions,
|
||||
)
|
||||
from app.harness.split_selection import diag_fingerprint
|
||||
from app.question_gen.loader import load_benchmark
|
||||
from core.evolution.types import DiagnosisSignalRow
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.harness.pools import Pools
|
||||
from core.evolution.protocols import DiagnosisSignalStore
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
# 与 core.evolution.diagnose._INFRA_STOP_REASONS 对齐:执行/解析层失败排除出可诊断错题。
|
||||
_INFRA_STOP_REASONS: frozenset[str] = frozenset({"error", "parse_error"})
|
||||
|
||||
# 工程路径默认值(少变;可经 CLI 单次覆盖)。诊断信号表建在 harness.db。
|
||||
_DEFAULT_HARNESS_DB = Path("workspaces/default/harness.db")
|
||||
_DEFAULT_QUESTIONS_DIR = Path("store/questions/benchmarks/Video-MME")
|
||||
_DEFAULT_OUT_DIR = Path("workspaces/video-split")
|
||||
_DEFAULT_STORE_DIR = Path("store") # tree.json 在 store/videos/<vid>/
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 配置解析(fail loud)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VideoSplitConfig:
|
||||
"""结果驱动视频级切分的科研旋钮快照(从 config/video_split.yaml 解析)。
|
||||
|
||||
字段:
|
||||
baseline_run_id: 基线 run 标识(错题诊断与切分依据)。
|
||||
n_trainval: trainval 目标视频数(多样性阶段填充上限)。
|
||||
epsilon: test 相对全局最大允许分布偏差(题型 / 难度两维)。
|
||||
report_floor: per-type 报告门限,题数 ≥ 此值的 task_type 才入 ε 约束。
|
||||
val_wrong_min: validation 池最少错题数(McNemar 功效阈;0=不检查)。
|
||||
val_ratio: validation 占 trainval 视频组总数的比例。
|
||||
seed: 贪心选择器预洗牌 + 视频组题级切分种子。
|
||||
floor_k: 各高信号 task_type 的 T2 defect 下限(硬约束)。
|
||||
prompt_version: 诊断 prompt 版本标识(指纹分量)。
|
||||
model: 执行诊断的模型名(指纹分量)。
|
||||
"""
|
||||
|
||||
baseline_run_id: str
|
||||
n_trainval: int
|
||||
epsilon: float
|
||||
report_floor: int
|
||||
val_wrong_min: int
|
||||
val_ratio: float
|
||||
seed: int
|
||||
floor_k: dict[str, int]
|
||||
prompt_version: str
|
||||
model: str
|
||||
|
||||
|
||||
def _require(section: dict[str, Any], keys: tuple[str, ...], where: str) -> None:
|
||||
"""校验 section 含全部必填键,缺任一即 fail loud(P5,不静默兜底)。
|
||||
|
||||
参数:
|
||||
section: 待校验的配置子字典。
|
||||
keys: 必填键元组。
|
||||
where: 出错信息中标注的段名(如 "video_split")。
|
||||
|
||||
异常:
|
||||
SystemExit: 存在缺失键。
|
||||
"""
|
||||
missing = [k for k in keys if k not in section]
|
||||
if missing:
|
||||
raise SystemExit(f"config {where} 段缺关键项 {missing},无法运行(P5 fail loud)")
|
||||
|
||||
|
||||
def parse_config(raw: dict[str, Any]) -> VideoSplitConfig:
|
||||
"""把 yaml 原始字典解析为 VideoSplitConfig,缺关键项 fail loud。
|
||||
|
||||
参数:
|
||||
raw: yaml.safe_load 的顶层字典,需含 video_split / diag 两段。
|
||||
|
||||
返回:
|
||||
VideoSplitConfig 冻结快照。
|
||||
|
||||
异常:
|
||||
SystemExit: 缺 video_split / diag 段或段内关键项。
|
||||
"""
|
||||
if "video_split" not in raw or "diag" not in raw:
|
||||
raise SystemExit("config 缺 video_split / diag 段,无法运行(P5 fail loud)")
|
||||
vs = raw["video_split"]
|
||||
dg = raw["diag"]
|
||||
_require(
|
||||
vs,
|
||||
(
|
||||
"baseline_run_id",
|
||||
"n_trainval",
|
||||
"epsilon",
|
||||
"report_floor",
|
||||
"val_wrong_min",
|
||||
"val_ratio",
|
||||
"seed",
|
||||
"floor_k",
|
||||
),
|
||||
"video_split",
|
||||
)
|
||||
_require(dg, ("prompt_version", "model"), "diag")
|
||||
return VideoSplitConfig(
|
||||
baseline_run_id=vs["baseline_run_id"],
|
||||
n_trainval=vs["n_trainval"],
|
||||
epsilon=vs["epsilon"],
|
||||
report_floor=vs["report_floor"],
|
||||
val_wrong_min=vs["val_wrong_min"],
|
||||
val_ratio=vs["val_ratio"],
|
||||
seed=vs["seed"],
|
||||
floor_k=dict(vs["floor_k"]),
|
||||
prompt_version=dg["prompt_version"],
|
||||
model=dg["model"],
|
||||
)
|
||||
|
||||
|
||||
def load_config(config_path: Path) -> VideoSplitConfig:
|
||||
"""读取并解析 video_split yaml 配置文件(缺文件 / 关键项 fail loud)。
|
||||
|
||||
参数:
|
||||
config_path: yaml 配置路径。
|
||||
|
||||
返回:
|
||||
VideoSplitConfig。
|
||||
|
||||
异常:
|
||||
SystemExit: 文件不存在或缺关键项。
|
||||
"""
|
||||
if not config_path.exists():
|
||||
raise SystemExit(f"config 文件不存在: {config_path}(P5 fail loud)")
|
||||
raw = yaml.safe_load(config_path.read_text(encoding="utf-8"))
|
||||
return parse_config(raw)
|
||||
|
||||
|
||||
def git_short_sha() -> str:
|
||||
"""取当前 git 短 SHA 作为诊断口径指纹的代码分量(诊断代码变则指纹变)。
|
||||
|
||||
返回:
|
||||
git rev-parse --short HEAD 输出(去空白)。
|
||||
|
||||
异常:
|
||||
SystemExit: 非 git 仓库或 git 不可用(fail loud,指纹不可缺分量)。
|
||||
"""
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["git", "rev-parse", "--short", "HEAD"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError) as exc:
|
||||
raise SystemExit(f"无法获取 git 短 SHA 作为诊断代码版本: {exc}(P5 fail loud)") from exc
|
||||
sha = out.stdout.strip()
|
||||
if not sha:
|
||||
raise SystemExit("git rev-parse --short HEAD 返回空,诊断指纹缺代码分量(P5 fail loud)")
|
||||
return sha
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 真实依赖组装(参考 runner.py::_run_diagnosis)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _DiagLLMSettings:
|
||||
"""诊断 LLM 的工程配置(从 .env 读取 search llm 凭证 + 韧性旋钮)。
|
||||
|
||||
仅承载诊断所需字段(搜索 LLM = 诊断 judge),不复用 main.InfraSettings 以免
|
||||
构造整套适配器(embed / vlm)的重活;缺关键凭证 fail loud。
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
class _Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
search_llm_model: str = ""
|
||||
search_llm_base_url: str = ""
|
||||
search_llm_api_key: str = ""
|
||||
redis_url: str = ""
|
||||
redis_cache_ttl: int = 86400
|
||||
llm_timeout: float = 300.0
|
||||
llm_max_retries: int = 3
|
||||
llm_retry_base_delay: float = 20.0
|
||||
llm_retry_max_delay: float = 120.0
|
||||
llm_circuit_breaker_threshold: int = 48
|
||||
llm_circuit_breaker_cooldown: float = 60.0
|
||||
llm_ttft_timeout: float = 30.0
|
||||
llm_inter_token_timeout: float = 15.0
|
||||
|
||||
self._s = _Settings()
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
return getattr(self._s, name)
|
||||
|
||||
|
||||
def _build_redis_cache(settings: Any) -> Any | None:
|
||||
"""按 .env redis_url 构建响应缓存(不可用则降级 None,与 main 一致)。"""
|
||||
if not settings.redis_url:
|
||||
return None
|
||||
|
||||
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
|
||||
|
||||
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
|
||||
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
|
||||
try:
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
|
||||
return RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
|
||||
except Exception:
|
||||
logger.warning("Redis 缓存不可用,诊断降级为无缓存模式")
|
||||
return None
|
||||
|
||||
|
||||
def build_diagnosis_deps(
|
||||
*,
|
||||
harness_db: Path,
|
||||
store_dir: Path,
|
||||
video_ids: list[str],
|
||||
concurrency: int,
|
||||
expected_model: str,
|
||||
) -> DiagnosisDeps:
|
||||
"""组装 Phase 1 诊断的真实依赖束(GovernedLLMClient + RunLogImpl + prompts)。
|
||||
|
||||
与 runner.py::_run_diagnosis 对齐:search LLM(thinking=True)作诊断 judge,
|
||||
RunLogImpl 只读读取 harness.db 的 predictions/traces,VersionedSkillStore 读技能,
|
||||
DiagnosePrompts 从项目根 prompts/ 加载,tree_data 按 video_ids 从
|
||||
store/videos/<vid>/tree.json 展平预加载(诊断需真实树,缺失即 fail-loud)。
|
||||
|
||||
参数:
|
||||
harness_db: harness.db 路径(诊断读预测 + 信号落库同库)。
|
||||
store_dir: store 根目录(含 videos/<vid>/tree.json)。
|
||||
video_ids: wrong_ids 涉及的 video 标识列表(可含重复,内部去重加载树)。
|
||||
concurrency: 诊断并发上限。
|
||||
expected_model: config.diag.model(诊断口径指纹的模型分量)。必须与 .env
|
||||
SEARCH_LLM_MODEL 一致——指纹里的 model 与实际诊断所用 model 不一致会让
|
||||
信号以错误模型指纹落库,破坏可复现 / resume / 口径隔离,故此处 fail loud。
|
||||
|
||||
返回:
|
||||
DiagnosisDeps 冻结依赖束。
|
||||
|
||||
异常:
|
||||
SystemExit: .env 缺 search LLM 凭证(model / base_url / api_key 任一为空),
|
||||
或 config.diag.model 与 .env SEARCH_LLM_MODEL 不一致(指纹漂移防护)。
|
||||
FileNotFoundError / ValueError: video_ids 中任一 video 的 tree.json 缺失或
|
||||
无效(load_tree_data_for_videos fail-loud,诊断需真实树)。
|
||||
"""
|
||||
from adapters.breaker import CircuitBreaker
|
||||
from adapters.llm import GovernedLLMClient
|
||||
from adapters.telemetry import SQLiteTelemetryRecorder
|
||||
from app.harness.log import RunLogImpl
|
||||
from app.harness.workspace import VersionedSkillStore
|
||||
|
||||
settings = _DiagLLMSettings()
|
||||
if not (
|
||||
settings.search_llm_model and settings.search_llm_base_url and settings.search_llm_api_key
|
||||
):
|
||||
raise SystemExit(
|
||||
"诊断 LLM 凭证缺失:.env 需配置 SEARCH_LLM_MODEL / SEARCH_LLM_BASE_URL / "
|
||||
"SEARCH_LLM_API_KEY(P5 fail loud,不静默兜底)"
|
||||
)
|
||||
if expected_model != settings.search_llm_model:
|
||||
raise SystemExit(
|
||||
"诊断模型指纹漂移:config.diag.model="
|
||||
f"{expected_model!r} 与 .env SEARCH_LLM_MODEL={settings.search_llm_model!r} "
|
||||
"不一致;指纹里的 model 必须等于实际诊断所用 model(P5 fail loud,"
|
||||
"请对齐 config/video_split.yaml diag.model 与 .env SEARCH_LLM_MODEL)"
|
||||
)
|
||||
|
||||
telemetry_db = Path("logs/telemetry.db")
|
||||
telemetry_db.parent.mkdir(parents=True, exist_ok=True)
|
||||
telemetry = SQLiteTelemetryRecorder(telemetry_db)
|
||||
breaker = CircuitBreaker(
|
||||
fail_threshold=max(settings.llm_circuit_breaker_threshold, 1),
|
||||
cooldown_s=settings.llm_circuit_breaker_cooldown,
|
||||
)
|
||||
llm = GovernedLLMClient(
|
||||
model=settings.search_llm_model,
|
||||
base_url=settings.search_llm_base_url,
|
||||
api_key=settings.search_llm_api_key,
|
||||
provider=settings.search_llm_model.split("-")[0],
|
||||
thinking=True,
|
||||
breaker=breaker,
|
||||
cache=_build_redis_cache(settings),
|
||||
telemetry=telemetry,
|
||||
timeout_s=settings.llm_timeout,
|
||||
ttft_timeout_s=settings.llm_ttft_timeout,
|
||||
inter_token_timeout_s=settings.llm_inter_token_timeout,
|
||||
max_retries=settings.llm_max_retries,
|
||||
retry_base_delay_s=settings.llm_retry_base_delay,
|
||||
retry_max_delay_s=settings.llm_retry_max_delay,
|
||||
)
|
||||
from app.harness.tree_nodes import load_tree_data_for_videos
|
||||
|
||||
return DiagnosisDeps(
|
||||
run_log=RunLogImpl(str(harness_db)),
|
||||
llm=llm,
|
||||
skill_store=VersionedSkillStore(_diagnosis_skills_dir()),
|
||||
prompts=_load_diagnose_prompts(),
|
||||
tree_data=load_tree_data_for_videos(store_dir, video_ids),
|
||||
concurrency=concurrency,
|
||||
)
|
||||
|
||||
|
||||
def _diagnosis_skills_dir() -> Path:
|
||||
"""诊断用技能目录:种子 store 的当前技能版本(诊断读技能遵从判定)。
|
||||
|
||||
诊断只读技能内容判断"是否遵从技能",用 store 种子 v1 即可(与基线 run 一致)。
|
||||
"""
|
||||
return Path("store/skills/v1")
|
||||
|
||||
|
||||
def _load_diagnose_prompts() -> Any:
|
||||
"""加载诊断模板束(从项目根 prompts/ 读取;与 runner._load_diagnose_prompts 一致)。"""
|
||||
from core.evolution.types import DiagnosePrompts
|
||||
|
||||
def _read(name: str) -> str:
|
||||
p = Path("prompts") / name
|
||||
if not p.exists():
|
||||
raise FileNotFoundError(f"缺进化/诊断模板: {p}(请从 TRM4 迁移或检查 prompts/)")
|
||||
return p.read_text(encoding="utf-8")
|
||||
|
||||
return DiagnosePrompts(
|
||||
defect_vs_lapse=_read("defect_vs_lapse.md"),
|
||||
reasoning_sub=_read("reasoning_sub.md"),
|
||||
span_eval_system=_read("span_eval_system.md"),
|
||||
missed_nodes=_read("missed_nodes.md"),
|
||||
skill_adherence=_read("skill_adherence.md"),
|
||||
confirmation_bias=_read("confirmation_bias.md"),
|
||||
evidence_sufficiency=_read("evidence_sufficiency.md"),
|
||||
)
|
||||
|
||||
|
||||
def select_diagnosable_wrong_ids(preds: list[dict]) -> list[str]:
|
||||
"""从 canonical 预测筛出可诊断错题 question_id(保序)。
|
||||
|
||||
可诊断错题判据:预测非空 且 stop_reason 非 INFRA(error / parse_error)且
|
||||
归一后预测 != 答案。INFRA / 空预测错题不进 wrong_ids——它们改由
|
||||
persist_infra_t0_rows 直接落 T0(run_diagnosis 内部也会二次排除同类题)。
|
||||
|
||||
参数:
|
||||
preds: load_canonical_predictions 产出的 canonical 预测行(已按 qid 去重)。
|
||||
|
||||
返回:
|
||||
可诊断错题 question_id 列表(保 preds 顺序)。
|
||||
"""
|
||||
wrong_ids: list[str] = []
|
||||
for pred in preds:
|
||||
prediction = (pred["prediction"] or "").strip()
|
||||
if not prediction or pred["stop_reason"] in _INFRA_STOP_REASONS:
|
||||
continue
|
||||
if not pred["correct"]:
|
||||
wrong_ids.append(pred["question_id"])
|
||||
return wrong_ids
|
||||
|
||||
|
||||
def persist_infra_t0_rows(
|
||||
store: DiagnosisSignalStore,
|
||||
preds: list[dict],
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
) -> int:
|
||||
"""把非正确且 INFRA / 空预测的错题以 T0 信号行 upsert 落库(幂等)。
|
||||
|
||||
这些题(stop_reason ∈ {error, parse_error} 或预测为空)从不进入 run_diagnosis
|
||||
(筛选时被前置排除),故其 T0 信号必须在此单独补齐——否则 signal store 缺这些行,
|
||||
tier 分布 / manifest 不完整(计划要求 4 个 INFRA 空预测错题 → T0)。
|
||||
|
||||
投影口径与 baseline_diagnosis 的 INFRA 投影一致:infra=True、tier="T0"、
|
||||
error_type / cause_category / evolution_target 均 None、degraded=False;
|
||||
video_id / task_type 从 canonical 预测取。store.upsert 按主键
|
||||
(question_id, baseline_run_id, diag_fingerprint) 幂等,重复调用零副作用。
|
||||
|
||||
参数:
|
||||
store: 诊断信号存储端口(与诊断落库同一 store)。
|
||||
preds: load_canonical_predictions 产出的 canonical 预测行。
|
||||
baseline_run_id: 基线 run 标识(信号行主键之一)。
|
||||
diag_fingerprint: 诊断口径指纹(信号行主键之一)。
|
||||
|
||||
返回:
|
||||
落库的 T0 行数(供日志)。
|
||||
"""
|
||||
count = 0
|
||||
for pred in preds:
|
||||
prediction = (pred["prediction"] or "").strip()
|
||||
is_infra_or_empty = pred["stop_reason"] in _INFRA_STOP_REASONS or not prediction
|
||||
if pred["correct"] or not is_infra_or_empty:
|
||||
continue
|
||||
store.upsert(
|
||||
DiagnosisSignalRow(
|
||||
question_id=pred["question_id"],
|
||||
video_id=pred["video_id"],
|
||||
baseline_run_id=baseline_run_id,
|
||||
diag_fingerprint=diag_fingerprint,
|
||||
task_type=pred["task_type"],
|
||||
error_type=None,
|
||||
cause_category=None,
|
||||
tier="T0",
|
||||
evolution_target=None,
|
||||
degraded=False,
|
||||
infra=True,
|
||||
session_id=None,
|
||||
)
|
||||
)
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def load_questions_by_id(questions_dir: Path) -> dict[str, GeneratedQuestion]:
|
||||
"""加载 benchmark 全部题并建 question_id → GeneratedQuestion 映射。
|
||||
|
||||
覆盖 wrong_ids 与 run_diagnosis 返回的全部 infra/degraded 题(取 video_id/task_type)。
|
||||
|
||||
参数:
|
||||
questions_dir: benchmark 题库目录。
|
||||
|
||||
返回:
|
||||
question_id → GeneratedQuestion 映射。
|
||||
"""
|
||||
return {q.question_id: q for q in load_benchmark(questions_dir)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# McNemar 功效护栏
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def check_mcnemar_power(pools: Pools, val_wrong_min: int) -> int:
|
||||
"""校验 validation 池错题数达 McNemar 功效阈,不足即 fail loud。
|
||||
|
||||
val_wrong_min 已前置到 build_split 内的切分保证功效(不足即从 diag 换入低 T2
|
||||
错题组补足,耗尽 fail-loud);本函数作切分冻结后的冗余最终确认:val 错题数 < 阈
|
||||
→ 验证信号不足以支撑可靠比较。
|
||||
|
||||
参数:
|
||||
pools: 冻结三池(含 validation 与 correctness)。
|
||||
val_wrong_min: 最少错题数阈(0 = 不检查)。
|
||||
|
||||
返回:
|
||||
validation 池实际错题数(供日志)。
|
||||
|
||||
异常:
|
||||
SystemExit: val_wrong_min > 0 且 val 错题数 < 阈(P5 fail loud,不静默放行)。
|
||||
"""
|
||||
val_wrong = sum(1 for q in pools.validation if not pools.correctness[q.question_id])
|
||||
if val_wrong_min > 0 and val_wrong < val_wrong_min:
|
||||
raise SystemExit(
|
||||
f"validation 池错题数 {val_wrong} < val_wrong_min={val_wrong_min},"
|
||||
"验证信号不足以支撑可靠比较(McNemar 检验功效不够)。"
|
||||
"请放大 val_ratio / 调整旋钮后重跑,勿静默放行。"
|
||||
)
|
||||
return val_wrong
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 两阶段编排(依赖注入,便于单测)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def run_pipeline(
|
||||
*,
|
||||
config: VideoSplitConfig,
|
||||
fingerprint: str,
|
||||
diagnosis_deps: DiagnosisDeps,
|
||||
signal_store: DiagnosisSignalStore,
|
||||
wrong_ids: list[str],
|
||||
questions: dict[str, GeneratedQuestion],
|
||||
canonical_preds: list[dict],
|
||||
harness_db: Path,
|
||||
questions_dir: Path,
|
||||
out_dir: Path,
|
||||
generated_at: str,
|
||||
force: bool = False,
|
||||
retry_uncertain: bool = False,
|
||||
) -> SplitBuildResult:
|
||||
"""内联三阶段:Phase 0 INFRA T0 补录 → Phase 1 诊断 → Phase 2 冻结切分 → McNemar 护栏。
|
||||
|
||||
参数:
|
||||
config: 科研旋钮快照。
|
||||
fingerprint: 诊断口径指纹(已合成,作诊断信号主键之一)。
|
||||
diagnosis_deps: Phase 1 诊断依赖束(真实或假实现)。
|
||||
signal_store: 诊断信号存储端口(Phase 0/1 写、Phase 2 读)。
|
||||
wrong_ids: 待诊断的可诊断错题 question_id 列表。
|
||||
questions: question_id → GeneratedQuestion 映射。
|
||||
canonical_preds: canonical 预测行(Phase 0 从中筛 INFRA / 空预测错题补 T0)。
|
||||
harness_db: harness.db 路径(Phase 2 读 canonical 预测)。
|
||||
questions_dir: benchmark 题库目录(Phase 2 加载题库切池)。
|
||||
out_dir: 冻结产物目录(pools.json + split_manifest.json)。
|
||||
generated_at: 生成时间戳(ISO 字符串,由调用方传入;见模块 C-2 复现锚点约定)。
|
||||
force: 覆盖已存在冻结产物开关,透传给 build_split(False 时已存在即报错)。
|
||||
retry_uncertain: 透传给 run_baseline_diagnosis,令已落 uncertain 题被重新诊断。
|
||||
|
||||
返回:
|
||||
SplitBuildResult(冻结三池 + manifest + assignment)。
|
||||
"""
|
||||
# Phase 0: INFRA / 空预测错题补 T0(这些题不进诊断,须单独落库保证 tier 分布/manifest 完整)。
|
||||
n_t0 = persist_infra_t0_rows(signal_store, canonical_preds, config.baseline_run_id, fingerprint)
|
||||
logger.info("Phase 0 INFRA T0 补录:落库 {} 行(INFRA / 空预测错题不进诊断)", n_t0)
|
||||
|
||||
# Phase 1: 离线诊断(断点续跑幂等:done_question_ids 已完成题跳过)。
|
||||
logger.info(
|
||||
"Phase 1 离线诊断:baseline={} 待诊断错题 {} 题", config.baseline_run_id, len(wrong_ids)
|
||||
)
|
||||
await run_baseline_diagnosis(
|
||||
baseline_run_id=config.baseline_run_id,
|
||||
diag_fingerprint=fingerprint,
|
||||
wrong_ids=wrong_ids,
|
||||
questions=questions,
|
||||
store=signal_store,
|
||||
deps=diagnosis_deps,
|
||||
retry_uncertain=retry_uncertain,
|
||||
)
|
||||
|
||||
# Phase 2: 冻结切分(读诊断信号 → 贪心选择 → 视频组原子切三池 → 冻结 + 六条断言)。
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info("Phase 2 冻结切分:out={}", out_dir)
|
||||
result = build_split(
|
||||
db_path=harness_db,
|
||||
baseline_run_id=config.baseline_run_id,
|
||||
signal_store=signal_store,
|
||||
diag_fingerprint=fingerprint,
|
||||
questions_dir=questions_dir,
|
||||
config=SplitBuildConfig(
|
||||
n_trainval=config.n_trainval,
|
||||
floor_k=config.floor_k,
|
||||
epsilon=config.epsilon,
|
||||
report_floor=config.report_floor,
|
||||
select_seed=config.seed,
|
||||
val_ratio=config.val_ratio,
|
||||
split_seed=config.seed,
|
||||
val_wrong_min=config.val_wrong_min,
|
||||
),
|
||||
out_path=out_dir / "pools.json",
|
||||
manifest_path=out_dir / "split_manifest.json",
|
||||
generated_at=generated_at,
|
||||
force=force,
|
||||
)
|
||||
|
||||
# McNemar 功效护栏(build_split 契约外的 capstone 层校验)。
|
||||
val_wrong = check_mcnemar_power(result.pools, config.val_wrong_min)
|
||||
logger.info(
|
||||
"切分冻结完成:pools={} manifest={} val错题={}/{}(阈)",
|
||||
out_dir / "pools.json",
|
||||
out_dir / "split_manifest.json",
|
||||
val_wrong,
|
||||
config.val_wrong_min,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 真实执行 / dry-run 入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_paths(args: argparse.Namespace) -> tuple[Path, Path, Path, Path]:
|
||||
"""解析 harness_db / questions_dir / out_dir / store_dir(CLI 覆盖默认工程路径)。"""
|
||||
harness_db = args.harness_db or _DEFAULT_HARNESS_DB
|
||||
questions_dir = args.questions_dir or _DEFAULT_QUESTIONS_DIR
|
||||
out_dir = args.out_dir or _DEFAULT_OUT_DIR
|
||||
store_dir = args.store_dir or _DEFAULT_STORE_DIR
|
||||
return harness_db, questions_dir, out_dir, store_dir
|
||||
|
||||
|
||||
def _execute_real(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
|
||||
"""真实执行两阶段管线:组装真实 deps、读错题、跑诊断 + 冻结切分。"""
|
||||
harness_db, questions_dir, out_dir, store_dir = _resolve_paths(args)
|
||||
if not harness_db.exists():
|
||||
raise SystemExit(f"harness.db 不存在: {harness_db}(P5 fail loud)")
|
||||
canonical_preds = load_canonical_predictions(harness_db, config.baseline_run_id)
|
||||
wrong_ids = select_diagnosable_wrong_ids(canonical_preds)
|
||||
questions = load_questions_by_id(questions_dir)
|
||||
video_ids: list[str] = []
|
||||
for qid in wrong_ids:
|
||||
q = questions.get(qid)
|
||||
if q is None:
|
||||
raise SystemExit(
|
||||
f"wrong_id {qid!r} 不在 questions_dir 题库中"
|
||||
"(baseline predictions 与题库不匹配,P5 fail loud)"
|
||||
)
|
||||
video_ids.append(q.video_id)
|
||||
deps = build_diagnosis_deps(
|
||||
harness_db=harness_db,
|
||||
store_dir=store_dir,
|
||||
video_ids=video_ids,
|
||||
concurrency=args.concurrency,
|
||||
expected_model=config.model,
|
||||
)
|
||||
|
||||
# generated_at:默认盖真实 UTC now(溯源用),--generated-at 可显式固定以复现(C-2)。
|
||||
generated_at = args.generated_at or datetime.datetime.now(datetime.UTC).isoformat()
|
||||
|
||||
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||
|
||||
store = SqliteDiagnosisSignalStore(str(harness_db))
|
||||
try:
|
||||
asyncio.run(
|
||||
run_pipeline(
|
||||
config=config,
|
||||
fingerprint=fingerprint,
|
||||
diagnosis_deps=deps,
|
||||
signal_store=store,
|
||||
wrong_ids=wrong_ids,
|
||||
questions=questions,
|
||||
canonical_preds=canonical_preds,
|
||||
harness_db=harness_db,
|
||||
questions_dir=questions_dir,
|
||||
out_dir=out_dir,
|
||||
generated_at=generated_at,
|
||||
force=args.force,
|
||||
retry_uncertain=args.retry_uncertain,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
class _DryRunLLM:
|
||||
"""dry-run 假 LLM:被真实调用即报错,保证不真调 LLM。"""
|
||||
|
||||
async def complete(self, *args: Any, **kwargs: Any) -> Any:
|
||||
raise AssertionError("dry-run 不应真调 LLM.complete")
|
||||
|
||||
|
||||
class _DryRunLog:
|
||||
"""dry-run 假 RunLog:predictions/traces 均返回空,诊断不真正执行。"""
|
||||
|
||||
async def get_predictions(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
|
||||
return []
|
||||
|
||||
async def get_traces(self, run_id: str, *, question_ids: list[str] | None = None) -> list:
|
||||
return []
|
||||
|
||||
|
||||
def _execute_dry_run(config: VideoSplitConfig, fingerprint: str, args: argparse.Namespace) -> None:
|
||||
"""dry-run:用假 deps 跑通 Phase 1 wiring(空错题 → 诊断早返回),打印步骤 + 指纹。
|
||||
|
||||
Phase 2 build_split 需真实诊断信号方能冻结,dry-run 不真实冻结,仅打印其计划;
|
||||
Phase 1 用空 wrong_ids 走 run_baseline_diagnosis 早返回路径,验证装配可调用而不触 LLM。
|
||||
"""
|
||||
harness_db, questions_dir, out_dir, _store_dir = _resolve_paths(args)
|
||||
logger.info("=== dry-run:校验两阶段装配(不真调 LLM / 不冻结产物)===")
|
||||
logger.info(
|
||||
"诊断口径指纹 diag_fingerprint={} (prompt={} model={})",
|
||||
fingerprint,
|
||||
config.prompt_version,
|
||||
config.model,
|
||||
)
|
||||
logger.info(
|
||||
"解析路径:harness_db={} questions_dir={} out_dir={}", harness_db, questions_dir, out_dir
|
||||
)
|
||||
logger.info(
|
||||
"旋钮:n_trainval={} epsilon={} report_floor={} val_ratio={} seed={} "
|
||||
"val_wrong_min={} floor_k={}",
|
||||
config.n_trainval,
|
||||
config.epsilon,
|
||||
config.report_floor,
|
||||
config.val_ratio,
|
||||
config.seed,
|
||||
config.val_wrong_min,
|
||||
config.floor_k,
|
||||
)
|
||||
|
||||
fake_deps = DiagnosisDeps(
|
||||
run_log=_DryRunLog(),
|
||||
llm=_DryRunLLM(),
|
||||
skill_store=object(),
|
||||
prompts=object(),
|
||||
tree_data={},
|
||||
concurrency=args.concurrency,
|
||||
)
|
||||
|
||||
from adapters.baseline_diagnosis_store import SqliteDiagnosisSignalStore
|
||||
|
||||
dry_db = out_dir / "_dry_run_signals.db"
|
||||
dry_db.parent.mkdir(parents=True, exist_ok=True)
|
||||
store = SqliteDiagnosisSignalStore(str(dry_db))
|
||||
try:
|
||||
# Phase 0 装配:用一条假 INFRA 空预测走通 persist_infra_t0_rows(不触 LLM)。
|
||||
fake_infra_preds = [
|
||||
{
|
||||
"question_id": "_dry_infra",
|
||||
"video_id": "_dry_v",
|
||||
"task_type": "Counting Problem",
|
||||
"prediction": "",
|
||||
"answer": "A",
|
||||
"stop_reason": "error",
|
||||
"correct": False,
|
||||
}
|
||||
]
|
||||
n_t0 = persist_infra_t0_rows(store, fake_infra_preds, config.baseline_run_id, fingerprint)
|
||||
logger.info("Phase 0 装配 OK:persist_infra_t0_rows 落 {} 行 INFRA T0(假数据)", n_t0)
|
||||
logger.info("Phase 1 装配 OK:run_baseline_diagnosis 以空错题走早返回路径(不触 LLM)")
|
||||
asyncio.run(
|
||||
run_baseline_diagnosis(
|
||||
baseline_run_id=config.baseline_run_id,
|
||||
diag_fingerprint=fingerprint,
|
||||
wrong_ids=[],
|
||||
questions={},
|
||||
store=store,
|
||||
deps=fake_deps,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
store.close()
|
||||
dry_db.unlink(missing_ok=True)
|
||||
logger.info(
|
||||
"Phase 2 装配 OK:真实执行将调 build_split 冻结 pools.json + manifest(dry-run 跳过)"
|
||||
)
|
||||
logger.info("=== dry-run 通过:两阶段装配可调用,指纹已算出 ===")
|
||||
|
||||
|
||||
def build_arg_parser() -> argparse.ArgumentParser:
|
||||
"""构建 CLI 参数解析器。"""
|
||||
parser = argparse.ArgumentParser(description="结果驱动视频级切分两阶段 CLI(诊断 → 冻结切分)")
|
||||
parser.add_argument("--config", type=Path, default=Path("config/video_split.yaml"))
|
||||
parser.add_argument("--dry-run", action="store_true", dest="dry_run")
|
||||
parser.add_argument("--gpu", type=str, default=None, help="可选:设置 CUDA_VISIBLE_DEVICES")
|
||||
parser.add_argument("--concurrency", type=int, default=8, help="诊断并发上限")
|
||||
parser.add_argument("--harness-db", type=Path, default=None, dest="harness_db")
|
||||
parser.add_argument("--questions-dir", type=Path, default=None, dest="questions_dir")
|
||||
parser.add_argument("--out-dir", type=Path, default=None, dest="out_dir")
|
||||
parser.add_argument("--store-dir", type=Path, default=None, dest="store_dir")
|
||||
parser.add_argument(
|
||||
"--generated-at",
|
||||
type=str,
|
||||
default=None,
|
||||
dest="generated_at",
|
||||
help=(
|
||||
"manifest generated_at 时间戳(ISO 字符串);默认盖真实 UTC now(溯源元数据)。"
|
||||
"复现锚点是 pools.json 内容 + seed + fingerprint;generated_at 可显式传入以"
|
||||
"对 manifest 做字节级复现比对。"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="覆盖已存在的冻结 pools.json/manifest(旧产物备份为 .bak.*)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--retry-uncertain",
|
||||
action="store_true",
|
||||
dest="retry_uncertain",
|
||||
help="把已落 tier='uncertain'(信号不可信降级)的题重新诊断,而非当作已完成跳过",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> None:
|
||||
"""CLI 入口:解析参数 → 载配置 → 算指纹 → dry-run 或真实两阶段执行。
|
||||
|
||||
参数:
|
||||
argv: 可选参数列表(默认 sys.argv[1:]),便于测试注入。
|
||||
"""
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
args = build_arg_parser().parse_args(argv)
|
||||
if args.gpu is not None:
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu
|
||||
logger.info("CUDA_VISIBLE_DEVICES={}", args.gpu)
|
||||
|
||||
config = load_config(args.config)
|
||||
fingerprint = diag_fingerprint(config.prompt_version, config.model, git_short_sha())
|
||||
|
||||
if args.dry_run:
|
||||
_execute_dry_run(config, fingerprint, args)
|
||||
return
|
||||
_execute_real(config, fingerprint, args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,498 @@
|
||||
"""Workspace 生命周期管理 + manifest 读写 + Protocol 实现。
|
||||
|
||||
Workspace 是一次实验的独立工作区,通过 manifest.json 引用 Store 中的
|
||||
特定版本资源并记录实验过程。Skills/Prompts 权重拷入 workspace 本地,
|
||||
训练产物只进 workspace 不污染 Store。
|
||||
|
||||
VersionedSkillStore / VersionedPromptStore 实现 core/evolution/protocols.py
|
||||
中定义的只读端口,供 core/ 层以 Protocol 方式读取技能和提示词。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.harness.store import read_seed
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedPaths:
|
||||
"""manifest 解析后的绝对路径集合。
|
||||
|
||||
属性:
|
||||
store_dir: Store 根目录绝对路径。
|
||||
videos_dir: 视频数据目录。
|
||||
questions_dir: 当前引用的题目目录。
|
||||
skills_dir: 当前引用的 Skill 版本目录(workspace 内)。
|
||||
prompts_dir: 当前引用的 Prompt 版本目录(workspace 内)。
|
||||
workspace_dir: Workspace 根目录。
|
||||
db_path: harness.db 路径。
|
||||
analyses_dir: 分析报告目录。
|
||||
runs_dir: 运行临时状态目录。
|
||||
"""
|
||||
|
||||
store_dir: Path
|
||||
videos_dir: Path
|
||||
questions_dir: Path
|
||||
skills_dir: Path
|
||||
prompts_dir: Path
|
||||
workspace_dir: Path
|
||||
db_path: Path
|
||||
analyses_dir: Path
|
||||
runs_dir: Path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部工具
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MANIFEST_CURRENT_KEYS = {"videos", "questions", "skills", "prompts"}
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
"""返回当前 UTC 时间的 ISO 格式字符串。"""
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
def _atomic_write_json(path: Path, data: dict) -> None:
|
||||
"""原子写 JSON:tmp + os.replace(对齐 checkpoint.py 范式,防半截损坏)。
|
||||
|
||||
先写同目录临时文件,再 os.replace 原子替换目标;替换阶段崩溃不会留下半截
|
||||
JSON,原文件保持完好。
|
||||
|
||||
参数:
|
||||
path: 目标 JSON 文件路径。
|
||||
data: 待序列化的字典。
|
||||
"""
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workspace 核心函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _scaffold_workspace(
|
||||
workspace_dir: Path,
|
||||
store_dir: Path,
|
||||
questions: str,
|
||||
skills_version: str,
|
||||
prompts_version: str,
|
||||
) -> None:
|
||||
"""写 manifest + 建 analyses/runs 目录(不拷权重;权重由调用方按来源拷入)。
|
||||
|
||||
参数:
|
||||
workspace_dir: 目标 workspace(由调用方保证不存在)。
|
||||
store_dir: Store 根目录。
|
||||
questions: 题目相对路径,如 ``'benchmarks/Video-MME'``。
|
||||
skills_version: manifest.current.skills 初始版本号。
|
||||
prompts_version: manifest.current.prompts 初始版本号。
|
||||
|
||||
关键实现:
|
||||
不依赖任何外部资源源(store 中的 skills/prompts 是否存在不在此校验),
|
||||
因此可被 init_workspace 与种子初始化复用;store 引用以相对路径写入 manifest。
|
||||
"""
|
||||
workspace_dir.mkdir(parents=True)
|
||||
(workspace_dir / "analyses").mkdir()
|
||||
(workspace_dir / "runs").mkdir()
|
||||
|
||||
store_abs = store_dir.resolve()
|
||||
store_rel = os.path.relpath(store_abs, workspace_dir.resolve())
|
||||
|
||||
manifest = {
|
||||
"name": workspace_dir.name,
|
||||
"created_at": _now_iso(),
|
||||
"store": store_rel,
|
||||
"current": {
|
||||
"videos": "videos",
|
||||
"questions": f"questions/{questions}",
|
||||
"skills": f"skills/{skills_version}",
|
||||
"prompts": f"prompts/{prompts_version}",
|
||||
},
|
||||
"history": [],
|
||||
}
|
||||
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||
|
||||
|
||||
def init_workspace(
|
||||
workspace_dir: Path,
|
||||
store_dir: Path,
|
||||
questions: str,
|
||||
skills_version: str,
|
||||
prompts_version: str,
|
||||
) -> None:
|
||||
"""创建 Workspace 目录并写入初始 manifest.json,拷贝种子权重。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 目标路径(不得已存在)。
|
||||
store_dir: Store 根目录。
|
||||
questions: 题目在 questions/ 下的相对路径,如 ``"benchmarks/Video-MME"``。
|
||||
skills_version: Skills 版本号,如 ``"v1"``。
|
||||
prompts_version: Prompts 版本号,如 ``"v1"``。
|
||||
|
||||
异常:
|
||||
FileExistsError: Workspace 目录已存在。
|
||||
FileNotFoundError: 引用的资源在 Store 中不存在。
|
||||
"""
|
||||
if workspace_dir.exists():
|
||||
raise FileExistsError(f"Workspace 已存在: {workspace_dir}")
|
||||
|
||||
store_abs = store_dir.resolve()
|
||||
refs = {
|
||||
"skills": f"skills/{skills_version}",
|
||||
"prompts": f"prompts/{prompts_version}",
|
||||
"questions": f"questions/{questions}",
|
||||
}
|
||||
for label, rel in refs.items():
|
||||
full = store_abs / rel
|
||||
if not full.is_dir():
|
||||
raise FileNotFoundError(f"Store 中不存在 {label}: {full}")
|
||||
|
||||
_scaffold_workspace(workspace_dir, store_dir, questions, skills_version, prompts_version)
|
||||
|
||||
# 拷种子权重进 workspace:v2+ 训练产物只进 workspace,不污染 store
|
||||
shutil.copytree(store_abs / refs["skills"], workspace_dir / refs["skills"])
|
||||
shutil.copytree(store_abs / refs["prompts"], workspace_dir / refs["prompts"])
|
||||
logger.info("Workspace 初始化完成: {}", workspace_dir)
|
||||
|
||||
|
||||
def init_workspace_from_seed(
|
||||
workspace_dir: Path,
|
||||
store_dir: Path,
|
||||
seed_name: str,
|
||||
questions: str,
|
||||
) -> str:
|
||||
"""从种子全新建 workspace:拷权重 -> v1、baseline.db -> harness.db、读 baseline_run_id。
|
||||
|
||||
参数:
|
||||
workspace_dir: 目标 workspace(不得已存在)。
|
||||
store_dir: Store 根目录。
|
||||
seed_name: 种子名(store/seeds 下)。
|
||||
questions: 题目相对路径,如 ``'benchmarks/Video-MME'``。
|
||||
|
||||
返回:
|
||||
baseline_run_id(供 build_pools 使用)。
|
||||
|
||||
异常:
|
||||
FileExistsError: workspace 已存在。
|
||||
FileNotFoundError: 种子不存在(由 read_seed 抛出),或 questions ref 目录不存在。
|
||||
|
||||
关键实现:
|
||||
破坏性/创建操作前先校验 questions ref 存在:fresh 路径在 runner 侧已先
|
||||
归档旧 ws,若到 build_pools 才发现 questions 缺失则旧 ws 已被毁;
|
||||
故在此尽早报错(fail-fast),让新 ws 在创建前失败。
|
||||
"""
|
||||
if workspace_dir.exists():
|
||||
raise FileExistsError(f"Workspace 已存在: {workspace_dir}")
|
||||
|
||||
# 校验种子存在 + 取 baseline_run_id
|
||||
meta = read_seed(store_dir, seed_name)
|
||||
|
||||
# fail-fast:校验 questions ref 存在
|
||||
questions_ref = store_dir / "questions" / questions
|
||||
if not questions_ref.is_dir():
|
||||
raise FileNotFoundError(f"questions ref 目录不存在: {questions_ref}")
|
||||
|
||||
seed_dir = store_dir / "seeds" / seed_name
|
||||
_scaffold_workspace(workspace_dir, store_dir, questions, "v1", "v1")
|
||||
shutil.copytree(seed_dir / "skills", workspace_dir / "skills" / "v1")
|
||||
shutil.copytree(seed_dir / "prompts", workspace_dir / "prompts" / "v1")
|
||||
shutil.copy2(seed_dir / "baseline.db", workspace_dir / "harness.db")
|
||||
|
||||
seed_pools = seed_dir / "pools.json"
|
||||
if seed_pools.exists():
|
||||
shutil.copy2(seed_pools, workspace_dir / "pools.json")
|
||||
seed_manifest = seed_dir / "split_manifest.json"
|
||||
if seed_manifest.exists():
|
||||
shutil.copy2(seed_manifest, workspace_dir / "split_manifest.json")
|
||||
|
||||
logger.info("Workspace 从种子 '{}' 初始化完成: {}", seed_name, workspace_dir)
|
||||
return meta["baseline_run_id"]
|
||||
|
||||
|
||||
def load_manifest(workspace_dir: Path) -> dict:
|
||||
"""读取并返回 workspace 的 manifest.json。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
|
||||
返回:
|
||||
manifest 字典。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: manifest.json 不存在。
|
||||
"""
|
||||
manifest_path = workspace_dir / "manifest.json"
|
||||
if not manifest_path.exists():
|
||||
raise FileNotFoundError(f"manifest.json 不存在: {manifest_path}")
|
||||
return json.loads(manifest_path.read_text())
|
||||
|
||||
|
||||
def resolve_paths(workspace_dir: Path) -> ResolvedPaths:
|
||||
"""读取 manifest,解析 current 中所有资源的绝对路径。
|
||||
|
||||
skills_dir/prompts_dir 解析到 workspace(非 store),
|
||||
videos_dir/questions_dir 解析到 store。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
|
||||
返回:
|
||||
ResolvedPaths 实例,包含所有资源的绝对路径。
|
||||
"""
|
||||
manifest = load_manifest(workspace_dir)
|
||||
ws_abs = workspace_dir.resolve()
|
||||
store_abs = (ws_abs / manifest["store"]).resolve()
|
||||
current = manifest["current"]
|
||||
return ResolvedPaths(
|
||||
store_dir=store_abs,
|
||||
videos_dir=store_abs / current["videos"],
|
||||
questions_dir=store_abs / current["questions"],
|
||||
skills_dir=ws_abs / current["skills"],
|
||||
prompts_dir=ws_abs / current["prompts"],
|
||||
workspace_dir=ws_abs,
|
||||
db_path=ws_abs / "harness.db",
|
||||
analyses_dir=ws_abs / "analyses",
|
||||
runs_dir=ws_abs / "runs",
|
||||
)
|
||||
|
||||
|
||||
def list_video_ids(workspace_dir: Path) -> list[str]:
|
||||
"""列出 workspace 引用的所有视频 ID(含 tree.json 的子目录名)。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
|
||||
返回:
|
||||
排序后的视频 ID 列表。
|
||||
"""
|
||||
paths = resolve_paths(workspace_dir)
|
||||
video_ids = []
|
||||
for entry in paths.videos_dir.iterdir():
|
||||
if entry.is_dir() and (entry / "tree.json").exists():
|
||||
video_ids.append(entry.name)
|
||||
return sorted(video_ids)
|
||||
|
||||
|
||||
def update_manifest(workspace_dir: Path, **version_updates: str) -> None:
|
||||
"""更新 manifest 的 current 字段。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
**version_updates: 要更新的字段及其新值,如 ``skills="skills/v2"``。
|
||||
|
||||
异常:
|
||||
KeyError: 更新的字段不在 current 允许的 key 白名单中。
|
||||
"""
|
||||
invalid = set(version_updates) - _MANIFEST_CURRENT_KEYS
|
||||
if invalid:
|
||||
raise KeyError(f"无效的 manifest current 字段: {invalid}")
|
||||
manifest = load_manifest(workspace_dir)
|
||||
manifest["current"].update(version_updates)
|
||||
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||
|
||||
|
||||
def record_run(workspace_dir: Path, run_id: str) -> Path:
|
||||
"""将 current 版本快照追加到 manifest history,创建 run 目录和 per-video wiki 目录。
|
||||
|
||||
幂等:同 run_id 不重复追加 history(长跑中断后重启 / held-out 复用 run_id 时)。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
run_id: 本次运行的唯一标识,如 ``"run_001"``。
|
||||
|
||||
返回:
|
||||
创建的 run 目录路径。
|
||||
"""
|
||||
manifest = load_manifest(workspace_dir)
|
||||
current = manifest["current"]
|
||||
|
||||
# 幂等:同 run_id 不重复追加 history
|
||||
if not any(h["run_id"] == run_id for h in manifest["history"]):
|
||||
manifest["history"].append(
|
||||
{
|
||||
"run_id": run_id,
|
||||
"started_at": _now_iso(),
|
||||
"skills": current["skills"],
|
||||
"prompts": current["prompts"],
|
||||
"questions": current["questions"],
|
||||
}
|
||||
)
|
||||
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||
|
||||
run_dir = workspace_dir / "runs" / run_id
|
||||
# exist_ok:同 run_id 重跑时 run 目录已存在不应崩溃
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
for video_id in list_video_ids(workspace_dir):
|
||||
(run_dir / video_id / "wiki").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.debug("Run 已记录: {}", run_id)
|
||||
return run_dir
|
||||
|
||||
|
||||
def read_best(workspace_dir: Path) -> dict | None:
|
||||
"""读取 manifest 的 best 指针,未设置时返回 None。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
|
||||
返回:
|
||||
best 字典(skills/prompts/val_acc/run_id/epoch),未设置时 None。
|
||||
"""
|
||||
return load_manifest(workspace_dir).get("best")
|
||||
|
||||
|
||||
def update_best(
|
||||
workspace_dir: Path,
|
||||
skills: str,
|
||||
prompts: str,
|
||||
val_acc: float,
|
||||
run_id: str,
|
||||
epoch: int,
|
||||
) -> None:
|
||||
"""写入 manifest 的 best 指针(历史最优版本快照,与 current 平级)。
|
||||
|
||||
best 独立于 current——更新 best 不影响 current。
|
||||
|
||||
参数:
|
||||
workspace_dir: Workspace 根目录。
|
||||
skills: 最优 skills 版本完整 ref,如 ``'skills/v2'``。
|
||||
prompts: 最优 prompts 版本完整 ref,如 ``'prompts/v2'``。
|
||||
val_acc: 该版本验证集准确率。
|
||||
run_id: 该版本验证 run_id。
|
||||
epoch: 达成该最优的轮次。
|
||||
"""
|
||||
manifest = load_manifest(workspace_dir)
|
||||
manifest["best"] = {
|
||||
"skills": skills,
|
||||
"prompts": prompts,
|
||||
"val_acc": val_acc,
|
||||
"run_id": run_id,
|
||||
"epoch": epoch,
|
||||
}
|
||||
_atomic_write_json(workspace_dir / "manifest.json", manifest)
|
||||
logger.info("Best 已更新: val_acc={}, run={}, epoch={}", val_acc, run_id, epoch)
|
||||
|
||||
|
||||
def archive_workspace(workspace_dir: Path) -> Path:
|
||||
"""把 workspace 整体移动到同级 .archive/<name>-<ts>,返回归档路径。
|
||||
|
||||
参数:
|
||||
workspace_dir: 要归档的 Workspace 根目录。
|
||||
|
||||
返回:
|
||||
归档后的目标路径。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: workspace 不存在。
|
||||
"""
|
||||
if not workspace_dir.exists():
|
||||
raise FileNotFoundError(f"workspace 不存在: {workspace_dir}")
|
||||
|
||||
archive_root = workspace_dir.parent / ".archive"
|
||||
archive_root.mkdir(exist_ok=True)
|
||||
ts = datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
|
||||
target = archive_root / f"{workspace_dir.name}-{ts}"
|
||||
shutil.move(str(workspace_dir), str(target))
|
||||
|
||||
logger.info("Workspace 已归档: {} -> {}", workspace_dir, target)
|
||||
return target
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Protocol 实现:VersionedSkillStore / VersionedPromptStore
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class VersionedSkillStore:
|
||||
"""版本化技能读取端口实现。
|
||||
|
||||
满足 ``core/evolution/protocols.py::SkillStore`` Protocol。
|
||||
从指定的 skills 版本目录读取 ``.md`` 文件。
|
||||
|
||||
参数:
|
||||
skills_dir: skills 版本目录绝对路径(如 ``workspace/skills/v1``)。
|
||||
"""
|
||||
|
||||
def __init__(self, skills_dir: Path) -> None:
|
||||
if not skills_dir.is_dir():
|
||||
raise FileNotFoundError(f"Skills 目录不存在: {skills_dir}")
|
||||
self._dir = skills_dir
|
||||
|
||||
def read_skill(self, filename: str) -> str:
|
||||
"""读取指定 skill 文件的全文内容。
|
||||
|
||||
参数:
|
||||
filename: skill 文件名,如 ``'temporal-reasoning.md'``。
|
||||
|
||||
返回:
|
||||
文件全文内容。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 文件不存在。
|
||||
"""
|
||||
path = self._dir / filename
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Skill 文件不存在: {path}")
|
||||
return path.read_text()
|
||||
|
||||
def list_skill_files(self) -> list[str]:
|
||||
"""列出当前版本所有 skill 文件名。
|
||||
|
||||
返回:
|
||||
文件名列表(排序)。
|
||||
"""
|
||||
return sorted(entry.name for entry in self._dir.iterdir() if entry.is_file())
|
||||
|
||||
|
||||
class VersionedPromptStore:
|
||||
"""版本化提示词读取端口实现。
|
||||
|
||||
满足 ``core/evolution/protocols.py::PromptStore`` Protocol。
|
||||
从指定的 prompts 版本目录读取 ``.md`` 文件。
|
||||
|
||||
参数:
|
||||
prompts_dir: prompts 版本目录绝对路径(如 ``workspace/prompts/v1``)。
|
||||
"""
|
||||
|
||||
def __init__(self, prompts_dir: Path) -> None:
|
||||
if not prompts_dir.is_dir():
|
||||
raise FileNotFoundError(f"Prompts 目录不存在: {prompts_dir}")
|
||||
self._dir = prompts_dir
|
||||
|
||||
def read_prompt(self, filename: str) -> str:
|
||||
"""读取指定 prompt 文件的全文内容。
|
||||
|
||||
参数:
|
||||
filename: prompt 文件名,如 ``'system.md'``。
|
||||
|
||||
返回:
|
||||
文件全文内容。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 文件不存在。
|
||||
"""
|
||||
path = self._dir / filename
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Prompt 文件不存在: {path}")
|
||||
return path.read_text()
|
||||
|
||||
def list_prompt_files(self) -> list[str]:
|
||||
"""列出当前版本所有 prompt 文件名。
|
||||
|
||||
返回:
|
||||
文件名列表(排序)。
|
||||
"""
|
||||
return sorted(entry.name for entry in self._dir.iterdir() if entry.is_file())
|
||||
+172
@@ -1 +1,173 @@
|
||||
"""应用层 Protocol 端口定义。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path # noqa: TC003 — runtime_checkable Protocol 需运行时可见
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy as np
|
||||
|
||||
from app.harness.pools import Pools
|
||||
from app.tree.index import TreeIndex
|
||||
from core.types import GeneratedQuestion, PoolConfig
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class EmbeddingProvider(Protocol):
|
||||
"""文本嵌入端口。
|
||||
|
||||
属性:
|
||||
dim: 嵌入维度 D。
|
||||
"""
|
||||
|
||||
@property
|
||||
def dim(self) -> int: ...
|
||||
|
||||
def embed(self, texts: str | list[str]) -> np.ndarray:
|
||||
"""文本 → 嵌入向量(L2 归一化)。
|
||||
|
||||
参数:
|
||||
texts: 单条文本或文本列表。
|
||||
|
||||
返回:
|
||||
[N, D] ndarray,每行 L2 范数为 1.0。
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class QuestionGenerator(Protocol):
|
||||
"""LLM 驱动的题目生成端口(预留接口)。
|
||||
|
||||
参数:
|
||||
video_id: 视频标识。
|
||||
task_type: 题型。
|
||||
tree: 视频树索引,提供锚节点上下文。
|
||||
exemplars: 风格示例题目列表。
|
||||
|
||||
返回:
|
||||
生成的单条题目。
|
||||
"""
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
video_id: str,
|
||||
task_type: str,
|
||||
tree: TreeIndex,
|
||||
*,
|
||||
exemplars: list[GeneratedQuestion],
|
||||
) -> GeneratedQuestion: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class OCRProvider(Protocol):
|
||||
"""帧文字转录端口。
|
||||
|
||||
实现方负责将帧图像发送给 OCR 服务并返回拼接后的文本。
|
||||
单帧失败应降级跳过,不得抛出异常中断整体流程。
|
||||
|
||||
参数:
|
||||
frame_paths: 帧文件路径列表。
|
||||
|
||||
返回:
|
||||
"帧1: <行1> | <行2>\\n帧2: ..." 格式文本;无有效结果时空串。
|
||||
"""
|
||||
|
||||
async def transcribe_frames(self, frame_paths: list[Path]) -> str: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ToolDispatchFn(Protocol):
|
||||
"""工具调度函数签名。
|
||||
|
||||
参数:
|
||||
tool_name: 工具名称。
|
||||
args: 工具参数字典。
|
||||
context: 上下文字典(包含 session_id)。
|
||||
|
||||
返回:
|
||||
工具执行结果文本。
|
||||
"""
|
||||
|
||||
async def __call__(
|
||||
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
|
||||
) -> str: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class ToolDispatchFactory(Protocol):
|
||||
"""per-version 工具调度工厂。
|
||||
|
||||
通过注入 skills_dir 生产对应版本的 ToolDispatchFn。
|
||||
|
||||
参数:
|
||||
skills_dir: skill 文件目录(None 则不加载 skill)。
|
||||
|
||||
返回:
|
||||
ToolDispatchFn 实例。
|
||||
"""
|
||||
|
||||
def __call__(self, *, skills_dir: Path | None = None) -> ToolDispatchFn: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PromptBuilderFn(Protocol):
|
||||
"""Prompt 构建函数签名。
|
||||
|
||||
参数:
|
||||
qa: 待构建 prompt 的题目。
|
||||
|
||||
返回:
|
||||
(system_prompt, user_prompt) 二元组。
|
||||
"""
|
||||
|
||||
def __call__(self, qa: GeneratedQuestion) -> tuple[str, str]: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PromptBuilderFactory(Protocol):
|
||||
"""per-version prompt 构建工厂。
|
||||
|
||||
通过注入 skills_dir 和 prompts_dir 生产对应版本的 PromptBuilderFn。
|
||||
|
||||
参数:
|
||||
skills_dir: skill 文件目录(None 则不加载 skill)。
|
||||
prompts_dir: prompt 文件目录(None 则使用默认目录)。
|
||||
|
||||
返回:
|
||||
PromptBuilderFn 实例。
|
||||
"""
|
||||
|
||||
def __call__(
|
||||
self,
|
||||
*,
|
||||
skills_dir: Path | None = None,
|
||||
prompts_dir: Path | None = None,
|
||||
) -> PromptBuilderFn: ...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PoolStrategy(Protocol):
|
||||
"""池构建策略端口。
|
||||
|
||||
应用层端口(非 core 层),因为返回类型 Pools 定义在 app/harness/pools.py。
|
||||
两个具体策略(GlobalPoolStrategy / PerCategoryPoolStrategy)实现此接口。
|
||||
"""
|
||||
|
||||
def build(
|
||||
self,
|
||||
questions: list[GeneratedQuestion],
|
||||
correctness: dict[str, bool],
|
||||
config: PoolConfig,
|
||||
*,
|
||||
db_path: Path | None = None,
|
||||
) -> Pools: ...
|
||||
|
||||
def build_incremental(
|
||||
self,
|
||||
new_task_types: list[str],
|
||||
questions: list[GeneratedQuestion],
|
||||
correctness: dict[str, bool],
|
||||
config: PoolConfig,
|
||||
) -> dict[str, dict[str, list[str]]]: ...
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
"""出题模块 — benchmark 加载、分层采样、赛题合成与 v2 出题管线。"""
|
||||
|
||||
from app.question_gen.families import ALL_FAMILIES, QuestionFamilySpec
|
||||
from app.question_gen.gates import GateReport, run_gates
|
||||
from app.question_gen.generator_v2 import CandidateQuestion, generate_one_v2
|
||||
from app.question_gen.loader import load_benchmark, stratified_sample
|
||||
from app.question_gen.pipeline_v2 import PipelineConfig, PipelineResult, run_pipeline_v2
|
||||
from app.question_gen.synthesizer import (
|
||||
TASK_TYPE_LEVEL_MAP,
|
||||
AnchorContext,
|
||||
generate_one,
|
||||
sample_anchor,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# v1 接口
|
||||
"load_benchmark",
|
||||
"stratified_sample",
|
||||
"TASK_TYPE_LEVEL_MAP",
|
||||
"AnchorContext",
|
||||
"generate_one",
|
||||
"sample_anchor",
|
||||
# v2 接口
|
||||
"run_pipeline_v2",
|
||||
"PipelineConfig",
|
||||
"PipelineResult",
|
||||
"QuestionFamilySpec",
|
||||
"ALL_FAMILIES",
|
||||
"CandidateQuestion",
|
||||
"generate_one_v2",
|
||||
"GateReport",
|
||||
"run_gates",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Phase B 对抗过滤层配置 — filter 层配置(非 strategy 属性)。
|
||||
|
||||
设计: research-wiki/designs/2026-07-14-adversarial-question-gen-phaseB-design.md §8
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import yaml
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdversarialFilterConfig:
|
||||
"""后置对抗过滤配置。
|
||||
|
||||
属性:
|
||||
filter_task_types: 被过滤的题型(仅这些走 agent 门),默认仅 AR。
|
||||
adversarial_max_rounds: 补生成迭代上限。
|
||||
adversarial_agent_max_steps: agent 试答步数上限。
|
||||
difficulty_warn_threshold: 批次 agent 正确率告警阈值。
|
||||
"""
|
||||
|
||||
filter_task_types: tuple[str, ...] = ("Action Recognition",)
|
||||
adversarial_max_rounds: int = 5
|
||||
adversarial_agent_max_steps: int = 40
|
||||
difficulty_warn_threshold: float = 0.85
|
||||
|
||||
|
||||
def load_adversarial_config(config_path: Path) -> AdversarialFilterConfig:
|
||||
"""从 YAML 的 adversarial_filter 区段加载配置,缺段/缺键用默认值。
|
||||
|
||||
参数:
|
||||
config_path: YAML 配置文件路径。
|
||||
|
||||
返回:
|
||||
AdversarialFilterConfig 实例。
|
||||
"""
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
raw = yaml.safe_load(f) or {}
|
||||
section = raw.get("adversarial_filter", {}) or {}
|
||||
default = AdversarialFilterConfig()
|
||||
types = section.get("filter_task_types")
|
||||
return AdversarialFilterConfig(
|
||||
filter_task_types=tuple(types) if types else default.filter_task_types,
|
||||
adversarial_max_rounds=int(
|
||||
section.get("adversarial_max_rounds", default.adversarial_max_rounds)
|
||||
),
|
||||
adversarial_agent_max_steps=int(
|
||||
section.get("adversarial_agent_max_steps", default.adversarial_agent_max_steps)
|
||||
),
|
||||
difficulty_warn_threshold=float(
|
||||
section.get("difficulty_warn_threshold", default.difficulty_warn_threshold)
|
||||
),
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,291 @@
|
||||
"""Grounded 干扰项 selector — 候选池 + VLM 视觉打分 + 区间选择(仅 AR 路径)。
|
||||
|
||||
把干扰项从"VLM 主观写得像"下沉到机制层:VLM 生成 N 个候选干扰项,再对
|
||||
候选 + 正解逐一打"视觉可信度"分,按 [正解分-δ_high, 正解分-δ_low] 区间
|
||||
选 3 个 grounded near-miss,从机制上消灭 Easy-Options Bias。
|
||||
|
||||
设计: research-wiki/designs/2026-07-14-grounded-question-gen-phaseA-design.md §3
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from json_repair import repair_json
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.question_gen.sampler_v2 import MaterialContext
|
||||
from core.protocols import VLMProvider
|
||||
|
||||
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SelectorConfig:
|
||||
"""selector 科研参数。
|
||||
|
||||
属性:
|
||||
candidate_pool_size: 首轮候选干扰项数 N。
|
||||
delta_low: 干扰项视觉分与正解的最小差(上界,太近=真歧义)。
|
||||
delta_high: 干扰项视觉分与正解的最大差(下界,太低=负空间)。
|
||||
max_delta_relax: δ_high 放宽次数上限(退火)。
|
||||
delta_relax_step: 每次放宽 δ_high 的增量。
|
||||
"""
|
||||
|
||||
candidate_pool_size: int
|
||||
delta_low: float
|
||||
delta_high: float
|
||||
max_delta_relax: int = 2
|
||||
delta_relax_step: float = 0.1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SelectorOutcome:
|
||||
"""selector 产出。observation 始终存在(含 hard-fail),供 run_store 落库。
|
||||
|
||||
属性:
|
||||
observation: 打分观测 dict(correct_score/chosen/pool_size/anneal_rounds/hard_fail)。
|
||||
options: 重组四选项(A=正解),hard-fail 时为 None。
|
||||
answer: 正解字母(恒 "A"),hard-fail 时为 None。
|
||||
"""
|
||||
|
||||
observation: dict[str, object]
|
||||
options: tuple[str, ...] | None = None
|
||||
answer: str | None = None
|
||||
|
||||
@property
|
||||
def hard_fail(self) -> bool:
|
||||
"""是否硬失败(凑不齐 3 个 grounded 干扰项)。"""
|
||||
return self.options is None
|
||||
|
||||
|
||||
def _select_in_interval(
|
||||
correct_score: float,
|
||||
candidates: list[str],
|
||||
candidate_scores: list[float],
|
||||
delta_low: float,
|
||||
delta_high: float,
|
||||
) -> list[str] | None:
|
||||
"""从候选中选 3 个视觉分落 [correct-δ_high, correct-δ_low] 区间的干扰项。
|
||||
|
||||
落区间者按分数降序取前 3(分数越高越接近正解=越难)。不足 3 个返回 None。
|
||||
|
||||
参数:
|
||||
correct_score: 正解视觉可信度分。
|
||||
candidates: 候选干扰项文本列表。
|
||||
candidate_scores: 与 candidates 对齐的视觉分列表。
|
||||
delta_low: 最小差(上界 = correct - delta_low)。
|
||||
delta_high: 最大差(下界 = correct - delta_high)。
|
||||
|
||||
返回:
|
||||
选中的 3 个候选文本(降序)或 None(不足 3 个)。
|
||||
"""
|
||||
upper = correct_score - delta_low
|
||||
lower = correct_score - delta_high
|
||||
eligible = [
|
||||
(c, s) for c, s in zip(candidates, candidate_scores, strict=True) if lower <= s <= upper
|
||||
]
|
||||
if len(eligible) < 3:
|
||||
return None
|
||||
eligible.sort(key=lambda cs: cs[1], reverse=True)
|
||||
return [c for c, _ in eligible[:3]]
|
||||
|
||||
|
||||
def _load_prompt(name: str) -> str:
|
||||
path = _PROMPTS_DIR / name
|
||||
if not path.exists():
|
||||
msg = f"Prompt 模板不存在: {path}"
|
||||
raise FileNotFoundError(msg)
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _material_context_block(question: str, correct_text: str, material: MaterialContext) -> str:
|
||||
parts = [f"## Question\n{question}", f"## Correct Answer\n{correct_text}"]
|
||||
if material.subtitle_sentences:
|
||||
parts.append("## Subtitles")
|
||||
parts.extend(f" - {s}" for s in material.subtitle_sentences)
|
||||
if getattr(material, "cross_l2_texts", None):
|
||||
parts.append("## Cross-Segment Context")
|
||||
parts.extend(f" - {t}" for t in material.cross_l2_texts)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _parse_json_object(raw: str) -> dict:
|
||||
content = raw.strip()
|
||||
if "```" in content:
|
||||
for part in content.split("```"):
|
||||
stripped = part.strip()
|
||||
if stripped.startswith("json"):
|
||||
stripped = stripped[4:].strip()
|
||||
if stripped.startswith("{"):
|
||||
content = stripped
|
||||
break
|
||||
data = json.loads(repair_json(content, return_objects=False))
|
||||
if not isinstance(data, dict):
|
||||
msg = f"selector 响应顶层非 JSON 对象: {type(data).__name__}"
|
||||
raise ValueError(msg)
|
||||
return data
|
||||
|
||||
|
||||
async def _generate_pool(
|
||||
vlm: VLMProvider,
|
||||
question: str,
|
||||
correct_text: str,
|
||||
material: MaterialContext,
|
||||
n: int,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> list[str]:
|
||||
"""VLM 生成 n 个候选干扰项文本。"""
|
||||
system = _load_prompt("ar_distractor_pool.md")
|
||||
user = (
|
||||
_material_context_block(question, correct_text, material)
|
||||
+ f"\n## N\nGenerate exactly {n} distractors."
|
||||
)
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
resp = await vlm.chat_with_images(messages, list(material.frame_paths), session_id=session_id)
|
||||
# graceful 降级:解析爆炸时返回空池,让退火/hard_fail 接管(区别于契约违反的打分校验)
|
||||
try:
|
||||
data = _parse_json_object(resp.content)
|
||||
except (ValueError, json.JSONDecodeError) as exc:
|
||||
logger.warning("grounded selector 候选池响应解析失败,降级为空池: {}", exc)
|
||||
return []
|
||||
raw = data.get("distractors", [])
|
||||
if not isinstance(raw, list):
|
||||
return []
|
||||
# 防御:去空、去重、剔除与正解字面相同者
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for item in raw:
|
||||
text = str(item).strip()
|
||||
if not text or text == correct_text.strip() or text in seen:
|
||||
continue
|
||||
seen.add(text)
|
||||
out.append(text)
|
||||
return out
|
||||
|
||||
|
||||
async def _score_options(
|
||||
vlm: VLMProvider,
|
||||
question: str,
|
||||
options: list[str],
|
||||
material: MaterialContext,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> list[float]:
|
||||
"""VLM 对 options(首个为正解)逐一打视觉可信度分 [0,1],返回对齐分数列表。"""
|
||||
system = _load_prompt("ar_distractor_score.md")
|
||||
numbered = "\n".join(f"{i}. {opt}" for i, opt in enumerate(options, 1))
|
||||
user = f"## Question\n{question}\n\n## Candidates\n{numbered}"
|
||||
messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
|
||||
resp = await vlm.chat_with_images(messages, list(material.frame_paths), session_id=session_id)
|
||||
data = _parse_json_object(resp.content)
|
||||
scores_raw = data.get("scores", [])
|
||||
if not isinstance(scores_raw, list) or len(scores_raw) != len(options):
|
||||
msg = f"打分数量({len(scores_raw) if isinstance(scores_raw, list) else 'NA'}) != 选项数({len(options)})"
|
||||
raise ValueError(msg)
|
||||
try:
|
||||
return [max(0.0, min(1.0, float(s))) for s in scores_raw]
|
||||
except (TypeError, ValueError) as e:
|
||||
msg = f"打分含非数值元素: {e}"
|
||||
raise ValueError(msg) from e
|
||||
|
||||
|
||||
async def build_grounded_options(
|
||||
vlm: VLMProvider,
|
||||
question: str,
|
||||
correct_text: str,
|
||||
material: MaterialContext,
|
||||
config: SelectorConfig,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> SelectorOutcome:
|
||||
"""生成候选池 → 视觉打分 → 区间选 3 干扰项 → 重组四选项。
|
||||
|
||||
退火(凑不齐 3 个时按序):① 最多追加 N 个新候选(去重后实际增量可能更少)再打分;② 逐步放宽
|
||||
δ_high(纯重选,不再调 VLM);③ 仍不足则 hard_fail(调用方走重出)。
|
||||
|
||||
参数:
|
||||
vlm: VLM 端口。
|
||||
question: 题干。
|
||||
correct_text: 正解文本(无字母前缀)。
|
||||
material: 采样素材(提供 frame_paths / subtitles)。
|
||||
config: selector 科研参数。
|
||||
session_id: 遥测会话 ID。
|
||||
|
||||
返回:
|
||||
SelectorOutcome。成功时 options=A 正解+3 grounded 干扰项;hard_fail
|
||||
时 options=None,但 observation 始终存在供落库。
|
||||
"""
|
||||
candidates = await _generate_pool(
|
||||
vlm, question, correct_text, material, config.candidate_pool_size, session_id=session_id
|
||||
)
|
||||
# options[0] 恒为正解
|
||||
scored = await _score_options(
|
||||
vlm, question, [correct_text, *candidates], material, session_id=session_id
|
||||
)
|
||||
correct_score, cand_scores = scored[0], scored[1:]
|
||||
anneal_rounds = 0
|
||||
|
||||
chosen = _select_in_interval(
|
||||
correct_score, candidates, cand_scores, config.delta_low, config.delta_high
|
||||
)
|
||||
|
||||
# 退火 1: 最多追加 N 个新候选(去重后实际增量可能更少),仅对新增候选打分,正解分保持首轮值
|
||||
if chosen is None:
|
||||
anneal_rounds += 1
|
||||
more = await _generate_pool(
|
||||
vlm, question, correct_text, material, config.candidate_pool_size, session_id=session_id
|
||||
)
|
||||
more = [m for m in more if m not in candidates]
|
||||
if more:
|
||||
more_scores = await _score_options(
|
||||
vlm, question, [correct_text, *more], material, session_id=session_id
|
||||
)
|
||||
candidates = candidates + more
|
||||
cand_scores = cand_scores + more_scores[1:]
|
||||
chosen = _select_in_interval(
|
||||
correct_score, candidates, cand_scores, config.delta_low, config.delta_high
|
||||
)
|
||||
|
||||
# 退火 2: 放宽 δ_high(下界下移,纳入更低分候选),δ_low 不动
|
||||
relax = 0
|
||||
delta_high = config.delta_high
|
||||
while chosen is None and relax < config.max_delta_relax:
|
||||
relax += 1
|
||||
anneal_rounds += 1
|
||||
delta_high = delta_high + config.delta_relax_step
|
||||
chosen = _select_in_interval(
|
||||
correct_score, candidates, cand_scores, config.delta_low, delta_high
|
||||
)
|
||||
|
||||
hard_fail = chosen is None
|
||||
observation: dict[str, object] = {
|
||||
"correct_score": correct_score,
|
||||
"chosen": [cand_scores[candidates.index(c)] for c in (chosen or [])],
|
||||
"pool_size": len(candidates),
|
||||
"anneal_rounds": anneal_rounds,
|
||||
"delta_high_final": delta_high,
|
||||
"hard_fail": hard_fail,
|
||||
}
|
||||
if hard_fail:
|
||||
logger.warning(
|
||||
"grounded selector 硬失败: correct={:.3f}, pool={}, anneal={}",
|
||||
correct_score,
|
||||
len(candidates),
|
||||
anneal_rounds,
|
||||
)
|
||||
# observation 仍返回,供 pipeline 落 selector_scores(设计 §3.3 退化观测)
|
||||
return SelectorOutcome(observation=observation)
|
||||
|
||||
options = (
|
||||
f"A. {correct_text}",
|
||||
f"B. {chosen[0]}",
|
||||
f"C. {chosen[1]}",
|
||||
f"D. {chosen[2]}",
|
||||
)
|
||||
return SelectorOutcome(observation=observation, options=options, answer="A")
|
||||
@@ -0,0 +1,252 @@
|
||||
"""题族规格声明 — 定义 5 大问题家族及其采样、泄漏检测、提示模板约束。
|
||||
|
||||
每个 QuestionFamilySpec 对应一种失败机制(skill_target M1–M5),
|
||||
由 get_family_for_slot 在出题时按权重分配。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LeakTestProfile:
|
||||
"""泄漏测试配置 — 定义快捷答题捷径类型与通过阈值。
|
||||
|
||||
Attributes:
|
||||
shortcut_type: 捷径类型标识。
|
||||
probe_template: store/prompts/question_gen/ 下的探测模板文件名。
|
||||
pass_threshold: 通过阈值(0–1),低于此值视为存在泄漏。
|
||||
"""
|
||||
|
||||
shortcut_type: str
|
||||
probe_template: str
|
||||
pass_threshold: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SamplingConstraint:
|
||||
"""采样约束 — 对树节点的最低要求。
|
||||
|
||||
Attributes:
|
||||
min_subtitles: 最少字幕段数。
|
||||
min_l3_nodes: 最少 L3 节点数。
|
||||
require_frames: 是否要求帧图像可用。
|
||||
cross_l2_span: 是否要求跨 L2 段采样。
|
||||
"""
|
||||
|
||||
min_subtitles: int
|
||||
min_l3_nodes: int
|
||||
require_frames: bool
|
||||
cross_l2_span: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QuestionFamilySpec:
|
||||
"""问题家族规格 — 一个家族的完整声明。
|
||||
|
||||
Attributes:
|
||||
name: 家族标识名(如 "RETRIEVAL")。
|
||||
skill_target: 目标失败机制编号(M1–M5)。
|
||||
sampling: 采样约束。
|
||||
legal_task_types: 该家族合法的任务类型集合(frozenset)。
|
||||
leak_profile: 泄漏测试配置。
|
||||
prompt_template: 出题 prompt 模板文件名。
|
||||
"""
|
||||
|
||||
name: str
|
||||
skill_target: str
|
||||
sampling: SamplingConstraint
|
||||
legal_task_types: frozenset[str]
|
||||
leak_profile: LeakTestProfile
|
||||
prompt_template: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5 大家族实例
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
RETRIEVAL_FAMILY = QuestionFamilySpec(
|
||||
name="RETRIEVAL",
|
||||
skill_target="M1",
|
||||
sampling=SamplingConstraint(
|
||||
min_subtitles=2,
|
||||
min_l3_nodes=3,
|
||||
require_frames=False,
|
||||
cross_l2_span=False,
|
||||
),
|
||||
legal_task_types=frozenset(
|
||||
[
|
||||
"Object Recognition",
|
||||
"Object Reasoning",
|
||||
"Action Recognition",
|
||||
"Attribute Perception",
|
||||
"OCR Problems",
|
||||
]
|
||||
),
|
||||
leak_profile=LeakTestProfile(
|
||||
shortcut_type="temporal_proximity",
|
||||
probe_template="gate_leak_retrieval.md",
|
||||
pass_threshold=0.6,
|
||||
),
|
||||
prompt_template="retrieval.md",
|
||||
)
|
||||
|
||||
REASONING_FAMILY = QuestionFamilySpec(
|
||||
name="REASONING",
|
||||
skill_target="M2",
|
||||
sampling=SamplingConstraint(
|
||||
min_subtitles=3,
|
||||
min_l3_nodes=4,
|
||||
require_frames=False,
|
||||
cross_l2_span=True,
|
||||
),
|
||||
legal_task_types=frozenset(
|
||||
[
|
||||
"Action Reasoning",
|
||||
"Object Reasoning",
|
||||
"Information Synopsis",
|
||||
]
|
||||
),
|
||||
leak_profile=LeakTestProfile(
|
||||
shortcut_type="frequency",
|
||||
probe_template="gate_leak_reasoning.md",
|
||||
pass_threshold=0.5,
|
||||
),
|
||||
prompt_template="reasoning.md",
|
||||
)
|
||||
|
||||
ENUMERATION_FAMILY = QuestionFamilySpec(
|
||||
name="ENUMERATION",
|
||||
skill_target="M3",
|
||||
sampling=SamplingConstraint(
|
||||
min_subtitles=2,
|
||||
min_l3_nodes=5,
|
||||
require_frames=False,
|
||||
cross_l2_span=False,
|
||||
),
|
||||
legal_task_types=frozenset(
|
||||
[
|
||||
"Counting Problem",
|
||||
"Temporal Reasoning",
|
||||
"Temporal Perception",
|
||||
"Information Synopsis",
|
||||
]
|
||||
),
|
||||
leak_profile=LeakTestProfile(
|
||||
shortcut_type="option_length",
|
||||
probe_template="gate_leak_enumeration.md",
|
||||
pass_threshold=0.6,
|
||||
),
|
||||
prompt_template="enumeration.md",
|
||||
)
|
||||
|
||||
VISUAL_FAMILY = QuestionFamilySpec(
|
||||
name="VISUAL",
|
||||
skill_target="M4",
|
||||
sampling=SamplingConstraint(
|
||||
min_subtitles=0,
|
||||
min_l3_nodes=3,
|
||||
require_frames=True,
|
||||
cross_l2_span=False,
|
||||
),
|
||||
legal_task_types=frozenset(
|
||||
[
|
||||
"Attribute Perception",
|
||||
"Counting Problem",
|
||||
"OCR Problems",
|
||||
"Action Recognition",
|
||||
]
|
||||
),
|
||||
leak_profile=LeakTestProfile(
|
||||
shortcut_type="visual_salience",
|
||||
probe_template="gate_leak_visual.md",
|
||||
pass_threshold=0.5,
|
||||
),
|
||||
prompt_template="visual.md",
|
||||
)
|
||||
|
||||
SPATIAL_FAMILY = QuestionFamilySpec(
|
||||
name="SPATIAL",
|
||||
skill_target="M5",
|
||||
sampling=SamplingConstraint(
|
||||
min_subtitles=0,
|
||||
min_l3_nodes=3,
|
||||
require_frames=True,
|
||||
cross_l2_span=False,
|
||||
),
|
||||
legal_task_types=frozenset(
|
||||
[
|
||||
"Spatial Perception",
|
||||
"Spatial Reasoning",
|
||||
]
|
||||
),
|
||||
leak_profile=LeakTestProfile(
|
||||
shortcut_type="spatial_default",
|
||||
probe_template="gate_leak_spatial.md",
|
||||
pass_threshold=0.5,
|
||||
),
|
||||
prompt_template="spatial.md",
|
||||
)
|
||||
|
||||
ALL_FAMILIES: tuple[QuestionFamilySpec, ...] = (
|
||||
RETRIEVAL_FAMILY,
|
||||
REASONING_FAMILY,
|
||||
ENUMERATION_FAMILY,
|
||||
VISUAL_FAMILY,
|
||||
SPATIAL_FAMILY,
|
||||
)
|
||||
|
||||
# 按 name 索引,用于 get_family_for_slot 快速查找
|
||||
_FAMILY_BY_NAME: dict[str, QuestionFamilySpec] = {f.name: f for f in ALL_FAMILIES}
|
||||
|
||||
|
||||
def get_family_for_slot(
|
||||
task_type: str,
|
||||
family_ratios: dict[str, float],
|
||||
rng: random.Random,
|
||||
) -> QuestionFamilySpec:
|
||||
"""根据任务类型和家族权重比例,随机选择一个合法的问题家族。
|
||||
|
||||
Args:
|
||||
task_type: 任务类型字符串(必须是 12 种合法类型之一)。
|
||||
family_ratios: 家族名称到权重的映射(如 {"RETRIEVAL": 0.30, ...})。
|
||||
rng: 随机数生成器实例(确保可复现)。
|
||||
|
||||
Returns:
|
||||
被选中的 QuestionFamilySpec。
|
||||
|
||||
Raises:
|
||||
ValueError: task_type 不在任何家族的 legal_task_types 中。
|
||||
ValueError: 给定 task_type 下没有合法家族(所有合法族权重为 0 或不在 ratios 中)。
|
||||
"""
|
||||
# 检查 task_type 是否被任一家族接受
|
||||
all_legal_types: set[str] = set()
|
||||
for family in ALL_FAMILIES:
|
||||
all_legal_types.update(family.legal_task_types)
|
||||
if task_type not in all_legal_types:
|
||||
msg = f"task_type '{task_type}' 不在任何家族的合法类型中"
|
||||
raise ValueError(msg)
|
||||
|
||||
# 过滤出接受该 task_type 且在 ratios 中有正权重的家族
|
||||
candidates: list[QuestionFamilySpec] = []
|
||||
weights: list[float] = []
|
||||
for family_name, weight in family_ratios.items():
|
||||
family = _FAMILY_BY_NAME.get(family_name)
|
||||
if family is None:
|
||||
continue
|
||||
if task_type in family.legal_task_types and weight > 0:
|
||||
candidates.append(family)
|
||||
weights.append(weight)
|
||||
|
||||
if not candidates:
|
||||
msg = f"task_type '{task_type}' 下没有合法家族可选(检查 family_ratios)"
|
||||
raise ValueError(msg)
|
||||
|
||||
# 归一化权重 + 加权随机选择
|
||||
chosen = rng.choices(candidates, weights=weights, k=1)[0]
|
||||
return chosen
|
||||
@@ -0,0 +1,511 @@
|
||||
"""轻量四门质量检查 — 对生成题目执行 4 个独立的 LLM 质量门控。
|
||||
|
||||
四门:
|
||||
1. key_verify: 验证答案在来源素材中有证据支撑。
|
||||
2. blind_answer: 无上下文时 LLM 能否答对(若答对 → 题目泄漏)。
|
||||
3. multi_true: 检测是否有多个选项可被视为正确。
|
||||
4. leak_test: 按策略特定模板探测答题捷径。
|
||||
|
||||
设计要点:
|
||||
- run_gates 先做 verbatim_ratio 前置短路(> 0.5 直接 FAIL key_verify)。
|
||||
- 四门并发执行(asyncio.gather)。
|
||||
- 每门加载 store/prompts/question_gen/ 下对应模板,构造 messages,调 LLM。
|
||||
- LLM 返回 JSON {"verdict": "pass"|"fail", "reason": "..."},解析失败视为 FAIL。
|
||||
- session_id 必须透传至每次 LLM 调用(遥测关联)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.question_gen.postprocess import PostprocessResult
|
||||
from app.tree.index import TreeIndex
|
||||
from core.protocols import LLMProvider, VLMProvider
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 常量
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
|
||||
|
||||
_VERBATIM_THRESHOLD = 0.5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CandidateQuestion(Task 5 尚未实现,本地定义,后续迁移至 generator_v2.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateQuestion:
|
||||
"""候选题目 — 出题管线生成、待门控审核的题目数据。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识。
|
||||
video_id: 所属视频标识。
|
||||
task_type: 题型(如 "Action Reasoning")。
|
||||
skill_target: 目标失败机制编号(M1-M5)。
|
||||
question: 题目文本。
|
||||
options: 选项元组(如 ("A. ...", "B. ...", "C. ...", "D. ..."))。
|
||||
answer: 正确答案字母(如 "A")。
|
||||
source_nodes: 来源节点 ID 元组。
|
||||
difficulty: 难度等级。
|
||||
subtitle_sentences: 字幕句子元组。
|
||||
frame_paths: 帧图片路径元组。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
task_type: str
|
||||
skill_target: str
|
||||
question: str
|
||||
options: tuple[str, ...]
|
||||
answer: str
|
||||
source_nodes: tuple[str, ...]
|
||||
difficulty: str
|
||||
subtitle_sentences: tuple[str, ...] = field(default_factory=tuple)
|
||||
frame_paths: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 门控结果类型
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class GateVerdict(Enum):
|
||||
"""门控判决枚举。"""
|
||||
|
||||
PASS = "pass"
|
||||
FAIL = "fail"
|
||||
SKIP = "skip"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateResult:
|
||||
"""单门判决结果。
|
||||
|
||||
属性:
|
||||
verdict: 门控判决(PASS / FAIL / SKIP)。
|
||||
reason: 判决原因说明。
|
||||
raw_response: LLM 原始返回内容。
|
||||
"""
|
||||
|
||||
verdict: GateVerdict
|
||||
reason: str
|
||||
raw_response: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateReport:
|
||||
"""四门判决汇总报告。
|
||||
|
||||
属性:
|
||||
key_verify: 关键验证门结果。
|
||||
blind_answer: 盲答门结果。
|
||||
multi_true: 多正确门结果。
|
||||
leak_test: 泄漏测试门结果。
|
||||
"""
|
||||
|
||||
key_verify: GateResult
|
||||
blind_answer: GateResult
|
||||
multi_true: GateResult
|
||||
leak_test: GateResult
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
"""全门 PASS 或 SKIP 时视为通过。"""
|
||||
for gate in (self.key_verify, self.blind_answer, self.multi_true, self.leak_test):
|
||||
if gate.verdict == GateVerdict.FAIL:
|
||||
return False
|
||||
return True
|
||||
|
||||
@property
|
||||
def reject_reason(self) -> str | None:
|
||||
"""首个 FAIL 门的 reason,全部通过返回 None。"""
|
||||
for gate in (self.key_verify, self.blind_answer, self.multi_true, self.leak_test):
|
||||
if gate.verdict == GateVerdict.FAIL:
|
||||
return gate.reason
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部工具函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _load_prompt_template(filename: str) -> str:
|
||||
"""加载 store/prompts/question_gen/ 下的模板文件。
|
||||
|
||||
参数:
|
||||
filename: 模板文件名(如 "gate_key_verify.md")。
|
||||
|
||||
返回:
|
||||
模板内容字符串。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 模板文件不存在。
|
||||
"""
|
||||
path = _PROMPTS_DIR / filename
|
||||
if not path.exists():
|
||||
msg = f"门控模板文件不存在: {path}"
|
||||
raise FileNotFoundError(msg)
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _resolve_source_text(candidate: CandidateQuestion, tree: TreeIndex) -> str:
|
||||
"""从树中解析候选题的来源节点文本。
|
||||
|
||||
遍历树的所有层级,匹配 source_nodes 中的 ID,提取对应描述文本。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
tree: 视频树索引。
|
||||
|
||||
返回:
|
||||
拼接的来源节点描述文本。
|
||||
"""
|
||||
texts: list[str] = []
|
||||
target_ids = set(candidate.source_nodes)
|
||||
|
||||
for l1 in tree.roots:
|
||||
if l1.id in target_ids:
|
||||
texts.append(f"[L1 {l1.id}] {l1.card.scene_summary}")
|
||||
for l2 in l1.children:
|
||||
if l2.id in target_ids:
|
||||
texts.append(f"[L2 {l2.id}] {l2.card.event_description}")
|
||||
for l3 in l2.children:
|
||||
if l3.id in target_ids:
|
||||
texts.append(f"[L3 {l3.id}] {l3.card.frame_summary}")
|
||||
|
||||
if not texts:
|
||||
logger.warning(
|
||||
"未找到来源节点: candidate={}, source_nodes={}",
|
||||
candidate.question_id,
|
||||
candidate.source_nodes,
|
||||
)
|
||||
return "(no source material found)"
|
||||
|
||||
return "\n".join(texts)
|
||||
|
||||
|
||||
def _format_options(options: tuple[str, ...]) -> str:
|
||||
"""将选项元组格式化为可读字符串。"""
|
||||
return "\n".join(options)
|
||||
|
||||
|
||||
def _parse_gate_response(raw_content: str) -> tuple[GateVerdict, str]:
|
||||
"""解析 LLM 返回的门控 JSON 响应。
|
||||
|
||||
期望格式: {"verdict": "pass"|"fail", "reason": "..."}
|
||||
解析失败时返回 FAIL + parse_error。
|
||||
|
||||
参数:
|
||||
raw_content: LLM 返回的原始文本。
|
||||
|
||||
返回:
|
||||
(verdict, reason) 元组。
|
||||
"""
|
||||
try:
|
||||
# 尝试从文本中提取 JSON(可能被 markdown 代码块包裹)
|
||||
content = raw_content.strip()
|
||||
if "```" in content:
|
||||
# 提取代码块中的内容
|
||||
parts = content.split("```")
|
||||
for part in parts:
|
||||
stripped = part.strip()
|
||||
if stripped.startswith("json"):
|
||||
stripped = stripped[4:].strip()
|
||||
if stripped.startswith("{"):
|
||||
content = stripped
|
||||
break
|
||||
|
||||
data = json.loads(content)
|
||||
verdict_str = data.get("verdict", "").lower()
|
||||
reason = data.get("reason", "")
|
||||
|
||||
if verdict_str == "pass":
|
||||
return GateVerdict.PASS, reason
|
||||
elif verdict_str == "fail":
|
||||
return GateVerdict.FAIL, reason
|
||||
else:
|
||||
return GateVerdict.FAIL, f"parse_error: invalid verdict '{verdict_str}'"
|
||||
|
||||
except (json.JSONDecodeError, AttributeError, TypeError) as e:
|
||||
return GateVerdict.FAIL, f"parse_error: {e}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 四门实现
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_source_frames(candidate: CandidateQuestion, tree: TreeIndex) -> list[str]:
|
||||
"""从树中收集候选题来源节点关联的帧路径。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
tree: 视频树索引。
|
||||
|
||||
返回:
|
||||
去重后的帧路径列表(最多 10 张,避免 VLM 输入过长)。
|
||||
"""
|
||||
frames: list[str] = []
|
||||
target_ids = set(candidate.source_nodes)
|
||||
|
||||
for l1 in tree.roots:
|
||||
for l2 in l1.children:
|
||||
if l2.id in target_ids:
|
||||
for l3 in l2.children:
|
||||
if l3.frame_path:
|
||||
frames.append(l3.frame_path)
|
||||
for l3 in l2.children:
|
||||
if l3.id in target_ids and l3.frame_path:
|
||||
frames.append(l3.frame_path)
|
||||
|
||||
# 也使用候选题自带的帧路径
|
||||
frames.extend(candidate.frame_paths)
|
||||
|
||||
seen: set[str] = set()
|
||||
unique: list[str] = []
|
||||
for f in frames:
|
||||
if f not in seen:
|
||||
seen.add(f)
|
||||
unique.append(f)
|
||||
return unique[:10]
|
||||
|
||||
|
||||
async def _gate_key_verify(
|
||||
candidate: CandidateQuestion,
|
||||
tree: TreeIndex,
|
||||
vlm: VLMProvider,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> GateResult:
|
||||
"""关键验证门 — 使用 VLM 检查答案在来源素材(文本+帧画面)中是否有证据支撑。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
tree: 视频树索引。
|
||||
vlm: VLM 图文调用端口(同时看文本和帧画面)。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
GateResult 实例。
|
||||
"""
|
||||
source_text = _resolve_source_text(candidate, tree)
|
||||
frames = _resolve_source_frames(candidate, tree)
|
||||
template = _load_prompt_template("gate_key_verify.md")
|
||||
prompt = template.format(
|
||||
source_text=source_text,
|
||||
question=candidate.question,
|
||||
options=_format_options(candidate.options),
|
||||
answer=candidate.answer,
|
||||
)
|
||||
|
||||
if frames:
|
||||
response = await vlm.chat_with_images(
|
||||
[{"role": "user", "content": prompt}],
|
||||
images=frames,
|
||||
session_id=session_id,
|
||||
)
|
||||
else:
|
||||
# 无帧时降级为纯文本(不应常见)
|
||||
logger.warning("key_verify 无可用帧,降级纯文本: {}", candidate.question_id)
|
||||
response = await vlm.chat_with_images(
|
||||
[{"role": "user", "content": prompt}],
|
||||
images=[],
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
verdict, reason = _parse_gate_response(response.content)
|
||||
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
|
||||
|
||||
|
||||
async def _gate_blind_answer(
|
||||
candidate: CandidateQuestion,
|
||||
llm: LLMProvider,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> GateResult:
|
||||
"""盲答门 — 无上下文时 LLM 能否答对(答对 → 题目泄漏)。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
llm: LLM 调用端口。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
GateResult 实例。
|
||||
"""
|
||||
template = _load_prompt_template("gate_blind_answer.md")
|
||||
prompt = template.format(
|
||||
question=candidate.question,
|
||||
options=_format_options(candidate.options),
|
||||
)
|
||||
|
||||
response = await llm.chat(
|
||||
[{"role": "user", "content": prompt}],
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
verdict, reason = _parse_gate_response(response.content)
|
||||
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
|
||||
|
||||
|
||||
async def _gate_multi_true(
|
||||
candidate: CandidateQuestion,
|
||||
tree: TreeIndex,
|
||||
llm: LLMProvider,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> GateResult:
|
||||
"""多正确门 — 检测是否有多个选项可被视为正确。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
tree: 视频树索引。
|
||||
llm: LLM 调用端口。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
GateResult 实例。
|
||||
"""
|
||||
source_text = _resolve_source_text(candidate, tree)
|
||||
template = _load_prompt_template("gate_multi_true.md")
|
||||
prompt = template.format(
|
||||
source_text=source_text,
|
||||
question=candidate.question,
|
||||
options=_format_options(candidate.options),
|
||||
)
|
||||
|
||||
response = await llm.chat(
|
||||
[{"role": "user", "content": prompt}],
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
verdict, reason = _parse_gate_response(response.content)
|
||||
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
|
||||
|
||||
|
||||
async def _gate_leak_test(
|
||||
candidate: CandidateQuestion,
|
||||
leak_probe_template: str,
|
||||
llm: LLMProvider,
|
||||
*,
|
||||
session_id: str,
|
||||
) -> GateResult:
|
||||
"""泄漏测试门 — 按策略特定模板探测答题捷径。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
leak_probe_template: 泄漏探测模板文件名(store/prompts/question_gen/ 下)。
|
||||
llm: LLM 调用端口。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
GateResult 实例。
|
||||
"""
|
||||
template = _load_prompt_template(leak_probe_template)
|
||||
prompt = template.format(
|
||||
question=candidate.question,
|
||||
options=_format_options(candidate.options),
|
||||
answer=candidate.answer,
|
||||
)
|
||||
|
||||
response = await llm.chat(
|
||||
[{"role": "user", "content": prompt}],
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
verdict, reason = _parse_gate_response(response.content)
|
||||
return GateResult(verdict=verdict, reason=reason, raw_response=response.content)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 编排入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def run_gates(
|
||||
candidate: CandidateQuestion,
|
||||
tree: TreeIndex,
|
||||
llm: LLMProvider,
|
||||
leak_probe_template: str,
|
||||
postprocess: PostprocessResult,
|
||||
*,
|
||||
vlm: VLMProvider | None = None,
|
||||
session_id: str,
|
||||
) -> GateReport:
|
||||
"""编排四门并发执行,返回汇总报告。
|
||||
|
||||
前置规则:verbatim_ratio > 0.5 时直接短路 key_verify 为 FAIL(不调 LLM)。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
tree: 视频树索引。
|
||||
llm: LLM 调用端口。
|
||||
leak_probe_template: 泄漏探测模板文件名(store/prompts/question_gen/ 下)。
|
||||
postprocess: 后处理结果(含 verbatim_ratio)。
|
||||
vlm: VLM 图文调用端口(key_verify 使用,None 时降级为 LLM)。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
GateReport 四门汇总。
|
||||
"""
|
||||
# Phase 1: verbatim 前置短路 — 不调 LLM,直接返回
|
||||
if postprocess.verbatim_ratio > _VERBATIM_THRESHOLD:
|
||||
logger.info(
|
||||
"verbatim_ratio={:.3f} > {:.1f},key_verify 短路 FAIL,其余三门 SKIP",
|
||||
postprocess.verbatim_ratio,
|
||||
_VERBATIM_THRESHOLD,
|
||||
)
|
||||
skip_result = GateResult(
|
||||
verdict=GateVerdict.SKIP,
|
||||
reason="skipped due to verbatim short-circuit",
|
||||
raw_response="",
|
||||
)
|
||||
return GateReport(
|
||||
key_verify=GateResult(
|
||||
verdict=GateVerdict.FAIL,
|
||||
reason=f"verbatim_ratio={postprocess.verbatim_ratio:.3f} exceeds threshold {_VERBATIM_THRESHOLD}",
|
||||
raw_response="",
|
||||
),
|
||||
blind_answer=skip_result,
|
||||
multi_true=skip_result,
|
||||
leak_test=skip_result,
|
||||
)
|
||||
|
||||
# Phase 2: 四门并发执行(key_verify 使用 VLM 看帧+文本)
|
||||
key_verify_provider = vlm if vlm is not None else llm
|
||||
key_result, blind_result, multi_result, leak_result = await asyncio.gather(
|
||||
_gate_key_verify(candidate, tree, key_verify_provider, session_id=session_id),
|
||||
_gate_blind_answer(candidate, llm, session_id=session_id),
|
||||
_gate_multi_true(candidate, tree, llm, session_id=session_id),
|
||||
_gate_leak_test(candidate, leak_probe_template, llm, session_id=session_id),
|
||||
)
|
||||
|
||||
report = GateReport(
|
||||
key_verify=key_result,
|
||||
blind_answer=blind_result,
|
||||
multi_true=multi_result,
|
||||
leak_test=leak_result,
|
||||
)
|
||||
|
||||
if report.passed:
|
||||
logger.debug("四门全部通过: question_id={}", candidate.question_id)
|
||||
else:
|
||||
logger.info(
|
||||
"门控拒绝: question_id={}, reason={}",
|
||||
candidate.question_id,
|
||||
report.reject_reason,
|
||||
)
|
||||
|
||||
return report
|
||||
@@ -0,0 +1,461 @@
|
||||
"""v2 生成器 — 基于策略特化 prompt 模板的单题 VLM 出题模块。
|
||||
|
||||
使用 VLMProvider 接口调用视觉语言模型,结合 per-strategy prompt 模板
|
||||
和 MaterialContext 素材上下文,生成一道四选一候选题。
|
||||
|
||||
典型调用路径::
|
||||
|
||||
candidate = await generate_one_v2(
|
||||
vlm=vlm_client,
|
||||
tree=tree_index,
|
||||
material=material_ctx,
|
||||
task_type="Action Reasoning",
|
||||
seq=1,
|
||||
video_id="vid_001",
|
||||
prompt_template="retrieval.md",
|
||||
strategy_name="RETRIEVAL",
|
||||
skill_target="M1",
|
||||
session_id="sess_001",
|
||||
)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from json_repair import repair_json
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.question_gen.sampler_v2 import MaterialContext
|
||||
from app.tree.index import TreeIndex
|
||||
from core.protocols import VLMProvider
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 常量
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PROMPTS_DIR = Path(__file__).resolve().parent.parent.parent / "store" / "prompts" / "question_gen"
|
||||
|
||||
_VALID_ANSWERS = frozenset({"A", "B", "C", "D"})
|
||||
|
||||
_VALID_DIFFICULTIES = frozenset({"easy", "medium", "hard"})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CandidateQuestion(规范定义位置)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateQuestion:
|
||||
"""候选题目 — 出题管线生成、待门控审核的题目数据。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识(格式 "{video_id}_{task_type}_{seq:04d}")。
|
||||
video_id: 所属视频标识。
|
||||
task_type: 题型(如 "Action Reasoning")。
|
||||
skill_target: 目标失败机制编号(M1-M5)。
|
||||
question: 题目文本。
|
||||
options: 选项元组(如 ("A. ...", "B. ...", "C. ...", "D. ..."))。
|
||||
answer: 正确答案字母(如 "A")。
|
||||
source_nodes: 来源节点 ID 元组。
|
||||
difficulty: 难度等级(easy/medium/hard)。
|
||||
subtitle_sentences: 验证材料 — 字幕句子元组。
|
||||
frame_paths: 验证材料 — 帧图片路径元组。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
task_type: str
|
||||
skill_target: str
|
||||
question: str
|
||||
options: tuple[str, ...]
|
||||
answer: str
|
||||
source_nodes: tuple[str, ...]
|
||||
difficulty: str
|
||||
subtitle_sentences: tuple[str, ...] = field(default_factory=tuple)
|
||||
frame_paths: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt 模板加载
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _load_prompt_template(template_name: str) -> str:
|
||||
"""加载 prompt 模板文件。
|
||||
|
||||
参数:
|
||||
template_name: store/prompts/question_gen/ 下的模板文件名。
|
||||
|
||||
返回:
|
||||
模板内容字符串。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 模板文件不存在。
|
||||
"""
|
||||
path = _PROMPTS_DIR / template_name
|
||||
if not path.exists():
|
||||
msg = f"Prompt 模板文件不存在: {path}"
|
||||
raise FileNotFoundError(msg)
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt 构建
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_v2_prompt(
|
||||
prompt_template: str,
|
||||
strategy_name: str,
|
||||
material: MaterialContext,
|
||||
task_type: str,
|
||||
seq: int,
|
||||
*,
|
||||
reject_reason: str | None = None,
|
||||
sub_pattern_instruction: str | None = None,
|
||||
) -> tuple[list[dict[str, str]], list[str]]:
|
||||
"""构建 VLM 出题调用的 messages 和帧路径列表。
|
||||
|
||||
参数:
|
||||
prompt_template: prompt 模板文件名(store/prompts/question_gen/ 下)。
|
||||
strategy_name: 策略名称(如 "RETRIEVAL")。
|
||||
material: 采样素材上下文。
|
||||
task_type: 任务类型字符串。
|
||||
seq: 当前序号。
|
||||
reject_reason: 上一次被门控拒绝的原因(用于引导 VLM 避免相同错误)。
|
||||
sub_pattern_instruction: 子模式特殊指令(如有)。
|
||||
|
||||
返回:
|
||||
二元组:
|
||||
- messages: 适配 VLMProvider 的 message 列表(system + user)。
|
||||
- frame_paths: 需发送给 VLM 的帧路径列表。
|
||||
"""
|
||||
# Phase 1: 加载策略模板作为 system prompt
|
||||
template_content = _load_prompt_template(prompt_template)
|
||||
system_message = template_content
|
||||
|
||||
# Phase 2: 构建 user prompt — 聚合素材信息
|
||||
user_parts: list[str] = []
|
||||
|
||||
user_parts.append(f"## Task Type: {task_type}")
|
||||
user_parts.append(f"## Question Family: {strategy_name}")
|
||||
user_parts.append(f"## Sequence: #{seq}")
|
||||
|
||||
# 字幕素材
|
||||
if material.subtitle_sentences:
|
||||
user_parts.append("\n## Subtitle Content:")
|
||||
for i, sent in enumerate(material.subtitle_sentences, 1):
|
||||
user_parts.append(f" {i}. {sent}")
|
||||
|
||||
# 跨 L2 上下文
|
||||
if material.cross_l2_texts:
|
||||
user_parts.append("\n## Cross-Segment Context:")
|
||||
for text in material.cross_l2_texts:
|
||||
user_parts.append(f" - {text}")
|
||||
|
||||
# 帧路径提示(VLM 会接收实际图像,此处仅作文本参考)
|
||||
if material.frame_paths:
|
||||
user_parts.append(f"\n## Visual Frames: {len(material.frame_paths)} frames attached.")
|
||||
|
||||
# 拒绝原因注入
|
||||
if reject_reason is not None:
|
||||
user_parts.append(
|
||||
f"\n## IMPORTANT - Previous Attempt Rejected:\n"
|
||||
f"Your previous question was rejected for the following reason:\n"
|
||||
f'"{reject_reason}"\n'
|
||||
f"Please generate a NEW question that avoids this issue."
|
||||
)
|
||||
|
||||
# 子模式特殊指令注入
|
||||
if sub_pattern_instruction is not None:
|
||||
user_parts.append(f"\n## Special Focus:\n{sub_pattern_instruction}")
|
||||
|
||||
# 输出格式指令
|
||||
user_parts.append(
|
||||
"\n## Output Format:\n"
|
||||
"Respond with ONLY a JSON object in this exact format:\n"
|
||||
"```json\n"
|
||||
"{\n"
|
||||
' "question": "Your question text here",\n'
|
||||
' "options": ["A. ...", "B. ...", "C. ...", "D. ..."],\n'
|
||||
' "answer": "A",\n'
|
||||
' "difficulty": "easy|medium|hard"\n'
|
||||
"}\n"
|
||||
"```"
|
||||
)
|
||||
|
||||
user_content = "\n".join(user_parts)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system_message},
|
||||
{"role": "user", "content": user_content},
|
||||
]
|
||||
|
||||
# Phase 3: 帧路径
|
||||
frame_paths = list(material.frame_paths)
|
||||
|
||||
return messages, frame_paths
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 响应解析
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _extract_json_from_text(raw: str) -> str:
|
||||
"""从可能被 markdown 代码块包裹的文本中提取 JSON 部分。
|
||||
|
||||
参数:
|
||||
raw: VLM 原始返回文本。
|
||||
|
||||
返回:
|
||||
清理后的 JSON 字符串。
|
||||
"""
|
||||
content = raw.strip()
|
||||
if "```" in content:
|
||||
parts = content.split("```")
|
||||
for part in parts:
|
||||
stripped = part.strip()
|
||||
if stripped.startswith("json"):
|
||||
stripped = stripped[4:].strip()
|
||||
if stripped.startswith("{"):
|
||||
return stripped
|
||||
return content
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _ValidatedFields:
|
||||
"""字段校验通过后的中间结构。"""
|
||||
|
||||
question: str
|
||||
options: tuple[str, ...]
|
||||
answer: str
|
||||
difficulty: str
|
||||
|
||||
|
||||
def _validate_parsed_fields(data: dict) -> _ValidatedFields:
|
||||
"""校验 VLM 响应 JSON 的必填字段并规范化。
|
||||
|
||||
参数:
|
||||
data: 已解析的 JSON 字典。
|
||||
|
||||
返回:
|
||||
_ValidatedFields 实例(字段已规范化)。
|
||||
|
||||
异常:
|
||||
ValueError: 缺少必填字段或字段值非法。
|
||||
"""
|
||||
missing = [f for f in ("question", "options", "answer", "difficulty") if f not in data]
|
||||
if missing:
|
||||
msg = f"VLM 响应缺少必填字段: {', '.join(missing)}"
|
||||
raise ValueError(msg)
|
||||
|
||||
question_text = str(data["question"])
|
||||
options_raw = data["options"]
|
||||
answer = str(data["answer"]).strip().upper()
|
||||
difficulty = str(data["difficulty"]).strip().lower()
|
||||
|
||||
# 校验 options(answer 约束为 A-D,因此必须恰好 4 个选项)
|
||||
if not isinstance(options_raw, list) or len(options_raw) != 4:
|
||||
msg = f"options 字段必须恰好包含 4 个选项,实际数量: {len(options_raw) if isinstance(options_raw, list) else type(options_raw).__name__}"
|
||||
raise ValueError(msg)
|
||||
|
||||
# 校验 answer
|
||||
if answer not in _VALID_ANSWERS:
|
||||
msg = f"answer 字段值 '{answer}' 非法,必须为 A/B/C/D 之一"
|
||||
raise ValueError(msg)
|
||||
|
||||
# 校验 difficulty(宽容处理:非法值回退为 medium)
|
||||
if difficulty not in _VALID_DIFFICULTIES:
|
||||
logger.warning(
|
||||
"difficulty '{}' 不在预设范围 {},回退为 'medium'",
|
||||
difficulty,
|
||||
_VALID_DIFFICULTIES,
|
||||
)
|
||||
difficulty = "medium"
|
||||
|
||||
return _ValidatedFields(
|
||||
question=question_text,
|
||||
options=tuple(str(o) for o in options_raw),
|
||||
answer=answer,
|
||||
difficulty=difficulty,
|
||||
)
|
||||
|
||||
|
||||
def _parse_v2_response(
|
||||
raw: str,
|
||||
video_id: str,
|
||||
task_type: str,
|
||||
skill_target: str,
|
||||
seq: int,
|
||||
source_nodes: tuple[str, ...],
|
||||
) -> CandidateQuestion:
|
||||
"""解析 VLM 返回的 JSON 响应,构造 CandidateQuestion。
|
||||
|
||||
流程:
|
||||
1. 提取 JSON(处理 markdown 包裹)。
|
||||
2. json_repair 修复常见格式错误。
|
||||
3. 校验必填字段(委托 _validate_parsed_fields)。
|
||||
4. 构造 CandidateQuestion 实例。
|
||||
|
||||
参数:
|
||||
raw: VLM 原始返回文本。
|
||||
video_id: 视频标识。
|
||||
task_type: 任务类型。
|
||||
skill_target: 目标技能编号。
|
||||
seq: 当前序号。
|
||||
source_nodes: 来源节点 ID 元组。
|
||||
|
||||
返回:
|
||||
CandidateQuestion 实例。
|
||||
|
||||
异常:
|
||||
ValueError: JSON 解析失败或缺少必填字段或字段值非法。
|
||||
"""
|
||||
# Phase 1: 提取 + 修复 JSON
|
||||
json_text = _extract_json_from_text(raw)
|
||||
repaired = repair_json(json_text, return_objects=False)
|
||||
|
||||
# Phase 2: 解析为字典
|
||||
try:
|
||||
data = json.loads(repaired)
|
||||
except json.JSONDecodeError as e:
|
||||
msg = f"VLM 响应 JSON 解析失败: {e}. 原始文本: {raw[:200]}"
|
||||
raise ValueError(msg) from e
|
||||
|
||||
if not isinstance(data, dict):
|
||||
msg = f"VLM 响应顶层不是 JSON 对象: type={type(data).__name__}"
|
||||
raise ValueError(msg)
|
||||
|
||||
# Phase 3: 字段校验
|
||||
fields = _validate_parsed_fields(data)
|
||||
|
||||
# Phase 4: 构造 CandidateQuestion
|
||||
question_id = f"{video_id}_{task_type}_{seq:04d}"
|
||||
|
||||
return CandidateQuestion(
|
||||
question_id=question_id,
|
||||
video_id=video_id,
|
||||
task_type=task_type,
|
||||
skill_target=skill_target,
|
||||
question=fields.question,
|
||||
options=fields.options,
|
||||
answer=fields.answer,
|
||||
source_nodes=source_nodes,
|
||||
difficulty=fields.difficulty,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def generate_one_v2(
|
||||
vlm: VLMProvider,
|
||||
tree: TreeIndex,
|
||||
material: MaterialContext,
|
||||
task_type: str,
|
||||
seq: int,
|
||||
*,
|
||||
video_id: str,
|
||||
prompt_template: str,
|
||||
strategy_name: str,
|
||||
skill_target: str,
|
||||
reject_reason: str | None = None,
|
||||
sub_pattern_instruction: str | None = None,
|
||||
session_id: str,
|
||||
) -> CandidateQuestion:
|
||||
"""调用 VLM 生成一道候选题目。
|
||||
|
||||
流程:
|
||||
1. 构建 per-strategy prompt + 帧路径。
|
||||
2. 调用 VLMProvider.chat_with_images。
|
||||
3. 解析响应为 CandidateQuestion。
|
||||
4. 附加素材验证信息(subtitle_sentences、frame_paths)。
|
||||
|
||||
参数:
|
||||
vlm: VLM 调用端口。
|
||||
tree: 视频树索引(当前未直接使用,预留后续扩展)。
|
||||
material: 采样素材上下文。
|
||||
task_type: 任务类型字符串。
|
||||
seq: 当前序号。
|
||||
video_id: 视频标识。
|
||||
prompt_template: prompt 模板文件名。
|
||||
strategy_name: 策略名称(如 "RETRIEVAL")。
|
||||
skill_target: 目标失败机制编号(M1-M5)。
|
||||
reject_reason: 上一次被门控拒绝的原因。
|
||||
sub_pattern_instruction: 子模式特殊指令(如有)。
|
||||
session_id: 会话 ID(遥测关联)。
|
||||
|
||||
返回:
|
||||
CandidateQuestion 实例(包含验证材料)。
|
||||
|
||||
异常:
|
||||
ValueError: VLM 响应解析失败。
|
||||
FileNotFoundError: prompt 模板不存在。
|
||||
"""
|
||||
# Phase 1: 构建 prompt
|
||||
messages, frame_paths = _build_v2_prompt(
|
||||
prompt_template=prompt_template,
|
||||
strategy_name=strategy_name,
|
||||
material=material,
|
||||
task_type=task_type,
|
||||
seq=seq,
|
||||
reject_reason=reject_reason,
|
||||
sub_pattern_instruction=sub_pattern_instruction,
|
||||
)
|
||||
|
||||
# Phase 2: 调用 VLM
|
||||
logger.debug(
|
||||
"generate_one_v2: strategy={}, task_type={}, seq={}, frames={}",
|
||||
strategy_name,
|
||||
task_type,
|
||||
seq,
|
||||
len(frame_paths),
|
||||
)
|
||||
|
||||
response = await vlm.chat_with_images(
|
||||
messages,
|
||||
frame_paths,
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
# Phase 3: 解析响应
|
||||
candidate = _parse_v2_response(
|
||||
raw=response.content,
|
||||
video_id=video_id,
|
||||
task_type=task_type,
|
||||
skill_target=skill_target,
|
||||
seq=seq,
|
||||
source_nodes=material.source_nodes,
|
||||
)
|
||||
|
||||
# Phase 4: 附加验证材料(构造新实例,因 frozen=True)
|
||||
candidate = CandidateQuestion(
|
||||
question_id=candidate.question_id,
|
||||
video_id=candidate.video_id,
|
||||
task_type=candidate.task_type,
|
||||
skill_target=candidate.skill_target,
|
||||
question=candidate.question,
|
||||
options=candidate.options,
|
||||
answer=candidate.answer,
|
||||
source_nodes=candidate.source_nodes,
|
||||
difficulty=candidate.difficulty,
|
||||
subtitle_sentences=tuple(material.subtitle_sentences),
|
||||
frame_paths=tuple(material.frame_paths),
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
"generate_one_v2 完成: question_id={}, difficulty={}",
|
||||
candidate.question_id,
|
||||
candidate.difficulty,
|
||||
)
|
||||
|
||||
return candidate
|
||||
@@ -0,0 +1,205 @@
|
||||
"""题目加载与分层采样。
|
||||
|
||||
从 benchmark JSON 目录加载题目,提供按对错比例的分层采样。
|
||||
对应训练循环中的 DataLoader 角色。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from core.types import QuestionUnit
|
||||
|
||||
_LEGACY_DEFAULT_DIFFICULTY = "medium"
|
||||
|
||||
|
||||
def load_benchmark(questions_dir: Path) -> list[GeneratedQuestion]:
|
||||
"""从 benchmark JSON 目录加载题目列表。
|
||||
|
||||
video_id 优先使用题目 JSON 中的 ``video_id`` 字段;若缺失则回退到
|
||||
文件名(不含扩展名)。Video-MME benchmark 按视频拆文件(文件名即
|
||||
video_id),v2 生成题把多视频题目合并在单个 JSON 中(每条记录自带
|
||||
``video_id``),两种格式均兼容。
|
||||
|
||||
pair 契约字段(``pair_id`` / ``question_role`` / ``flip_axis`` / ``unit_id``)
|
||||
用 ``.get`` 读取:旧 benchmark 无这些键时退化为 single(``question_role``
|
||||
默认 "single",``unit_id`` 留空由 __post_init__ 回填为 question_id),
|
||||
保证历史题库可无缝加载。
|
||||
|
||||
参数:
|
||||
questions_dir: 包含 *.json 文件的目录路径。
|
||||
|
||||
返回:
|
||||
按文件名排序加载的题目列表。
|
||||
"""
|
||||
results: list[GeneratedQuestion] = []
|
||||
for path in sorted(questions_dir.glob("*.json")):
|
||||
fallback_video_id = path.stem
|
||||
with open(path, encoding="utf-8") as f:
|
||||
qa_list: list[dict] = json.load(f)
|
||||
for qa in qa_list:
|
||||
results.append(
|
||||
GeneratedQuestion(
|
||||
question_id=qa["question_id"],
|
||||
video_id=qa.get("video_id", fallback_video_id),
|
||||
task_type=qa["task_type"],
|
||||
question=qa["question"],
|
||||
options=tuple(qa["options"]),
|
||||
answer=qa["answer"],
|
||||
source_nodes=tuple(qa.get("source_nodes", ())),
|
||||
difficulty=qa.get("difficulty", _LEGACY_DEFAULT_DIFFICULTY),
|
||||
family=qa.get("family"),
|
||||
skill_target=qa.get("skill_target"),
|
||||
difficulty_steps=qa.get("difficulty_steps"),
|
||||
sub_pattern=qa.get("sub_pattern"),
|
||||
# pair 契约字段:旧 benchmark 无这些键时按 single 默认兜底,
|
||||
# unit_id 留空交由 GeneratedQuestion.__post_init__ 回填。
|
||||
pair_id=qa.get("pair_id"),
|
||||
question_role=qa.get("question_role", "single"),
|
||||
flip_axis=qa.get("flip_axis"),
|
||||
unit_id=qa.get("unit_id", ""),
|
||||
)
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
def stratified_sample(
|
||||
questions: list[GeneratedQuestion],
|
||||
correctness: dict[str, bool],
|
||||
size: int,
|
||||
correct_ratio: float | None,
|
||||
task_types: list[str] | None,
|
||||
seed: int,
|
||||
min_per_class: int | None,
|
||||
) -> list[GeneratedQuestion]:
|
||||
"""按题型过滤后采样 size 个单元,可选按对错比例分层并按题型保底。
|
||||
|
||||
参数:
|
||||
questions: 候选题目全集(single 与孪生对成员可混含)。
|
||||
correctness: question_id -> 基线是否答对(单元级正确性取成员 AND)。
|
||||
size: 采样单元总量(single 计 1、pair 计 1)。
|
||||
correct_ratio: 采样中"基线答对"单元的占比;None 表示自然分布。
|
||||
task_types: 限定题型;None 表示不限。
|
||||
seed: 随机种子,保证可复现。
|
||||
min_per_class: 每个题型补足到的单元下限;None 表示不补足。
|
||||
|
||||
返回:
|
||||
采样后的题目列表(pair 单元展开为原始的两道题)。
|
||||
|
||||
异常:
|
||||
ValueError: 自然分布时单元池不足 size,或分层时某层单元不足。
|
||||
|
||||
关键实现:
|
||||
以 **QuestionUnit 为采样原子**(single 计 1、pair 计 1),size /
|
||||
correct_ratio / min_per_class 均按 unit 计数,孪生对两题永不被劈开。
|
||||
采样完成后 flatten_units 展开回逐题列表。纯 single 输入时 build_units
|
||||
与题目一一对应、顺序不变,rng 消耗与旧逐题实现完全一致(字节级回归)。
|
||||
|
||||
build_units / flatten_units 采用函数内延迟导入:loader 属 question_gen,
|
||||
question_units 属 harness,而 harness 包初始化会反向 import question_gen,
|
||||
模块级导入将触发循环依赖(沿用 adversarial_filter 的既有做法)。
|
||||
"""
|
||||
from app.harness.question_units import build_units, flatten_units
|
||||
|
||||
rng = random.Random(seed)
|
||||
units = build_units(questions)
|
||||
pool = [u for u in units if task_types is None or u.task_type in task_types]
|
||||
|
||||
if correct_ratio is None:
|
||||
if len(pool) < size:
|
||||
raise ValueError(f"自然分布采样不足: 需 {size} 个单元, 实有 {len(pool)} 个")
|
||||
sampled = rng.sample(pool, size)
|
||||
else:
|
||||
sampled = _ratio_stratified_sample(pool, correctness, size, correct_ratio, rng)
|
||||
|
||||
if min_per_class is not None:
|
||||
sampled = _backfill_per_class(sampled, pool, min_per_class, rng)
|
||||
return flatten_units(sampled)
|
||||
|
||||
|
||||
def _ratio_stratified_sample(
|
||||
pool: list[QuestionUnit],
|
||||
correctness: dict[str, bool],
|
||||
size: int,
|
||||
correct_ratio: float,
|
||||
rng: random.Random,
|
||||
) -> list[QuestionUnit]:
|
||||
"""按对错比例分层采样:对单元占 correct_ratio,其余为错单元。
|
||||
|
||||
参数:
|
||||
pool: 题型过滤后的候选单元。
|
||||
correctness: question_id -> 基线是否答对。
|
||||
size: 采样单元总量。
|
||||
correct_ratio: 对单元占比。
|
||||
rng: 随机数发生器。
|
||||
|
||||
返回:
|
||||
采样后的单元列表(对单元在前、错单元在后)。
|
||||
|
||||
异常:
|
||||
ValueError: 对单元或错单元层不足。
|
||||
|
||||
关键实现:
|
||||
unit_correctness 采用函数内延迟导入:loader 属 question_gen,
|
||||
question_units 属 harness,模块级导入将触发循环依赖(沿用 build_units /
|
||||
flatten_units 的既有做法)。以 strict=False 保持"缺基线对错即视为未答对"的
|
||||
原 loose 语义不变。
|
||||
"""
|
||||
from app.harness.question_units import unit_correctness
|
||||
|
||||
correct = [u for u in pool if unit_correctness(u, correctness, strict=False)]
|
||||
wrong = [u for u in pool if not unit_correctness(u, correctness, strict=False)]
|
||||
n_correct = round(size * correct_ratio)
|
||||
n_wrong = size - n_correct
|
||||
if len(correct) < n_correct or len(wrong) < n_wrong:
|
||||
raise ValueError(
|
||||
f"分层不足: 需对{n_correct}/错{n_wrong}, 实有对{len(correct)}/错{len(wrong)}"
|
||||
)
|
||||
return rng.sample(correct, n_correct) + rng.sample(wrong, n_wrong)
|
||||
|
||||
|
||||
def _backfill_per_class(
|
||||
sampled: list[QuestionUnit],
|
||||
pool: list[QuestionUnit],
|
||||
min_per_class: int,
|
||||
rng: random.Random,
|
||||
) -> list[QuestionUnit]:
|
||||
"""对候选池中出现的每个题型,将采样单元补足到 min_per_class 个。
|
||||
|
||||
遍历对象是候选池 pool 里出现的全部题型(非仅 sampled 命中的),
|
||||
保证任意稀疏题型都能拿到足额样本。补足以 unit 为原子,孪生对整进整出。
|
||||
|
||||
参数:
|
||||
sampled: 主采样结果单元(不修改,返回新列表)。
|
||||
pool: 候选单元全集(补足来源 + 题型枚举来源)。
|
||||
min_per_class: 每个题型的单元下限。
|
||||
rng: 随机数发生器。
|
||||
|
||||
返回:
|
||||
补足后的单元列表。
|
||||
"""
|
||||
selected_ids = {u.unit_id for u in sampled}
|
||||
result = list(sampled)
|
||||
counts: dict[str, int] = {}
|
||||
for u in sampled:
|
||||
counts[u.task_type] = counts.get(u.task_type, 0) + 1
|
||||
ordered_task_types: dict[str, None] = {}
|
||||
for u in pool:
|
||||
ordered_task_types.setdefault(u.task_type, None)
|
||||
for task_type in ordered_task_types:
|
||||
deficit = min_per_class - counts.get(task_type, 0)
|
||||
if deficit <= 0:
|
||||
continue
|
||||
candidates = [u for u in pool if u.task_type == task_type and u.unit_id not in selected_ids]
|
||||
take = rng.sample(candidates, min(deficit, len(candidates)))
|
||||
for u in take:
|
||||
selected_ids.add(u.unit_id)
|
||||
result.append(u)
|
||||
return result
|
||||
@@ -0,0 +1,196 @@
|
||||
"""accepted 题库的 pair 原子成对落盘 helper(纯件,不依赖 pipeline)。
|
||||
|
||||
三件可复用纯件供 Phase 2 新 pipeline 的 on_accept 回调复用:``PairPendingBuffer``
|
||||
(按 pair_id 收齐才 emit 单元)、``write_accepted``(tmp + os.replace 原子写)、
|
||||
``read_accepted``(聚合成单元并剔除磁盘悬挂孤儿)。
|
||||
|
||||
wiring 归属:本模块**只是纯件**,不接任何生成侧回调;真正的 on_accept wiring
|
||||
**见 Phase 2**。当前项目真实 accepted 写入点是 ``adversarial_filter.write_final_bank``
|
||||
(Phase 2 待重建的旧代码),本模块沿用其 tmp + os.replace 模式但不 import/不改动它。
|
||||
|
||||
unit_hash 校验:``QuestionUnit.unit_hash`` 目前恒为 ""(填充是 Phase 2 的事),且
|
||||
``GeneratedQuestion`` 不携带 unit_hash 字段。故把"unit_hash 不一致→拒"落地为**同
|
||||
pair_id 两成员的绑定一致性校验**(video_id / task_type / flip_axis),不一致即
|
||||
fail-fast raise,语义等价——只有 payload 绑定一致的孪生对才允许聚合。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from collections import defaultdict
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from core.types import GeneratedQuestion, QuestionUnit
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _check_pair_binding(first: GeneratedQuestion, second: GeneratedQuestion) -> None:
|
||||
"""校验同 pair_id 两成员的绑定一致性,不一致 fail-fast(unit_hash 语义代偿)。
|
||||
|
||||
参数 first/second 为先后到达的孪生对成员。video_id / task_type / flip_axis 任一
|
||||
不一致即 raise ValueError——绑定不一致的两条题目不构成同一 payload 的孪生对,
|
||||
拒绝聚合而非静默兜底。
|
||||
"""
|
||||
mismatches: list[str] = []
|
||||
if first.video_id != second.video_id:
|
||||
mismatches.append(f"video_id: {first.video_id} != {second.video_id}")
|
||||
if first.task_type != second.task_type:
|
||||
mismatches.append(f"task_type: {first.task_type} != {second.task_type}")
|
||||
if first.flip_axis != second.flip_axis:
|
||||
mismatches.append(f"flip_axis: {first.flip_axis} != {second.flip_axis}")
|
||||
if mismatches:
|
||||
raise ValueError(
|
||||
f"pair {first.pair_id} 两成员绑定不一致(" + ";".join(mismatches) + "),拒绝聚合"
|
||||
)
|
||||
|
||||
|
||||
class PairPendingBuffer:
|
||||
"""按 pair_id 收齐孪生对才 emit 单元的有状态缓冲器。
|
||||
|
||||
喂题接口 ``add`` 逐条消费题目:single 立即 emit ``QuestionUnit.from_single``;
|
||||
pair 成员先缓存,等同一 pair_id 的 original+mirror 都到齐才 emit 一个 pair 单元
|
||||
(复用 ``QuestionUnit.from_pair`` 走 fail-fast 校验)。批次末尾用
|
||||
``pending_orphans`` 检测"只落 P 未落 Q"的悬挂项。
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""pending 以 pair_id 索引首个到达的孪生对成员,等伙伴到齐再 emit。"""
|
||||
self._pending: dict[str, GeneratedQuestion] = {}
|
||||
|
||||
def add(self, q: GeneratedQuestion) -> QuestionUnit | None:
|
||||
"""喂入一条题目 q,返回本次凑齐的单元或 None(pair 尚未配齐)。
|
||||
|
||||
single 立即返回 kind="single" 单元;pair 首个成员缓存并返回 None,第二个
|
||||
成员到齐后返回 kind="pair" 单元。同 pair_id 两成员绑定不一致或角色非法(如
|
||||
两个 original)即 raise ValueError。
|
||||
|
||||
关键实现:配齐后**先校验、成功组装出 unit 才从 pending 删除**——若
|
||||
``_check_pair_binding`` / ``_order_pair`` raise,首成员仍留在 pending,调用方
|
||||
``pending_orphans`` 可取回被拒的悬挂成员。绑定校验用显式 ValueError(不依赖
|
||||
会被 ``-O`` 剥除的 assert)。
|
||||
"""
|
||||
if not q.pair_id:
|
||||
return QuestionUnit.from_single(q)
|
||||
|
||||
partner = self._pending.get(q.pair_id)
|
||||
if partner is None:
|
||||
self._pending[q.pair_id] = q
|
||||
return None
|
||||
|
||||
_check_pair_binding(partner, q)
|
||||
original, mirror = _order_pair(partner, q)
|
||||
unit = QuestionUnit.from_pair(original, mirror)
|
||||
del self._pending[q.pair_id]
|
||||
return unit
|
||||
|
||||
def pending_orphans(self) -> list[GeneratedQuestion]:
|
||||
"""返回仍未配齐的悬挂成员,供调用方在批次末尾检测"只落 P 未落 Q"。"""
|
||||
return list(self._pending.values())
|
||||
|
||||
|
||||
def _order_pair(
|
||||
a: GeneratedQuestion, b: GeneratedQuestion
|
||||
) -> tuple[GeneratedQuestion, GeneratedQuestion]:
|
||||
"""按 question_role 把两成员 a/b 定序为 (original, mirror)。
|
||||
|
||||
两成员不构成恰好 1 original + 1 mirror(角色缺失或重复)即 raise ValueError,
|
||||
防 next(...) 静默 StopIteration。
|
||||
"""
|
||||
originals = [q for q in (a, b) if q.question_role == "pair_original"]
|
||||
mirrors = [q for q in (a, b) if q.question_role == "pair_mirror"]
|
||||
if len(originals) != 1 or len(mirrors) != 1:
|
||||
raise ValueError(
|
||||
f"pair {a.pair_id} 角色非法:original={len(originals)} mirror={len(mirrors)},"
|
||||
"需各恰好 1 条"
|
||||
)
|
||||
return originals[0], mirrors[0]
|
||||
|
||||
|
||||
def _validate_disk_pair(pair_id: str, group: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||
"""三态判定磁盘上同 pair_id 分组,区分悬挂孤儿(drop)与结构损坏(raise)。
|
||||
|
||||
- 恰好 1 original + 1 mirror:合法孪生对,额外做 ``_check_pair_binding`` 显式绑定
|
||||
校验(video_id/task_type/flip_axis,-O 下仍生效),返回其两题。
|
||||
- size==1(只落 P 未落 Q):业务上合法的悬挂孤儿,warn + drop,返回 []。
|
||||
- 其余(size>2 超员、或 size==2 角色重复/缺角色):数据损坏/外部篡改,按 P5
|
||||
fail-loud,raise ValueError(含 pair_id 与成员构成),绝不静默吞。
|
||||
"""
|
||||
originals = sum(1 for q in group if q.question_role == "pair_original")
|
||||
mirrors = sum(1 for q in group if q.question_role == "pair_mirror")
|
||||
if len(group) == 2 and originals == 1 and mirrors == 1:
|
||||
_check_pair_binding(group[0], group[1])
|
||||
return group
|
||||
if len(group) == 1:
|
||||
logger.warning("磁盘悬挂孤儿 pair {}:仅 1 成员(缺伙伴),warn+drop 该 unit", pair_id)
|
||||
return []
|
||||
raise ValueError(
|
||||
f"磁盘 pair {pair_id} 结构损坏:成员数={len(group)}"
|
||||
f"(original={originals} mirror={mirrors}),需恰好 1 original + 1 mirror"
|
||||
)
|
||||
|
||||
|
||||
def _keep_complete_disk_pairs(questions: list[GeneratedQuestion]) -> list[GeneratedQuestion]:
|
||||
"""筛选磁盘读回的题目:single 全保留、pair 按 ``_validate_disk_pair`` 三态处理。
|
||||
|
||||
悬挂孤儿 warn+drop、结构损坏/绑定不一致 raise ValueError、合法成对保留后交给
|
||||
build_units(single 全保留)。
|
||||
"""
|
||||
by_pair: dict[str, list[GeneratedQuestion]] = defaultdict(list)
|
||||
singles: list[GeneratedQuestion] = []
|
||||
for q in questions:
|
||||
if q.pair_id:
|
||||
by_pair[q.pair_id].append(q)
|
||||
else:
|
||||
singles.append(q)
|
||||
|
||||
kept_pairs = [q for pid, grp in by_pair.items() for q in _validate_disk_pair(pid, grp)]
|
||||
return singles + kept_pairs
|
||||
|
||||
|
||||
def write_accepted(path: Path, units: list[QuestionUnit]) -> None:
|
||||
"""把 units 全量原子写到 path(tmp + os.replace),孪生对两题相邻落盘。
|
||||
|
||||
parent 不存在则自动创建;任一 pair 单元结构非法(size≠2)落盘前 fail-fast raise。
|
||||
|
||||
关键实现:沿用 ``write_final_bank`` 的原子写模式(先写同目录 ``.tmp`` 再
|
||||
``os.replace`` 覆盖,保证读到的 JSON 恒完整)。序列化复用 T9 pools.py 的
|
||||
``_q_to_dict``(唯一 GeneratedQuestion↔dict schema,含 pair 四字段),函数内
|
||||
import 规避 app.question_gen↔app.harness 循环依赖。
|
||||
"""
|
||||
from app.harness.pools import _q_to_dict
|
||||
from app.harness.question_units import flatten_units, validate_units
|
||||
|
||||
validate_units(units)
|
||||
records = [_q_to_dict(q) for q in flatten_units(units)]
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
os.replace(str(tmp), str(path))
|
||||
logger.info("accepted 题库全量原子写: {} 单元 / {} 题 → {}", len(units), len(records), path)
|
||||
|
||||
|
||||
def read_accepted(path: Path) -> list[QuestionUnit]:
|
||||
"""读回 path 的 accepted JSON → 聚合为单元列表。
|
||||
|
||||
磁盘是外部输入,按 P5 全量校验后再用:``_keep_complete_disk_pairs`` 对 pair 分组三态处理
|
||||
——"只落 P 未落 Q"的悬挂孤儿 warn+drop(不进结果、不 raise);结构损坏(超员/
|
||||
角色重复)或绑定不一致(video_id/task_type/flip_axis)显式 raise ValueError(不
|
||||
依赖 build_units 内会被 ``-O`` 剥除的 assert)。single 全保留。
|
||||
|
||||
关键实现:反序列化复用 T9 pools.py 的 ``_dict_to_q``(pair 四字段 .get 兼容),
|
||||
函数内 import 规避循环依赖。sift 后 ``build_units`` 聚合、``validate_units`` 二次
|
||||
防御闸门。
|
||||
"""
|
||||
from app.harness.pools import _dict_to_q
|
||||
from app.harness.question_units import build_units, validate_units
|
||||
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
questions = [_dict_to_q(d) for d in raw]
|
||||
kept = _keep_complete_disk_pairs(questions)
|
||||
return validate_units(build_units(kept))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,381 @@
|
||||
"""确定性后处理层 — 零 LLM 的选项洗牌、指代黑名单、逐字重复率、时间锚点、素材禁区检测。
|
||||
|
||||
所有函数均为纯函数(给定输入必定产出相同输出),
|
||||
用于 pipeline 出题后、门控前的确定性质量检查。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 数据结构
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PostprocessResult:
|
||||
"""后处理结果汇总。
|
||||
|
||||
属性:
|
||||
options: 洗牌后的选项元组。
|
||||
answer: 重映射后的答案字母。
|
||||
referent_violations: 指代黑名单违规描述列表。
|
||||
verbatim_ratio: 正确选项与来源素材的逐字重复率 [0.0, 1.0]。
|
||||
has_time_anchor: 题目是否包含时间锚点。
|
||||
"""
|
||||
|
||||
options: tuple[str, ...]
|
||||
answer: str
|
||||
referent_violations: list[str]
|
||||
verbatim_ratio: float
|
||||
has_time_anchor: bool
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 指代黑名单(预编译正则)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_BLACKLIST_PATTERNS: list[tuple[re.Pattern[str], str]] = [
|
||||
(re.compile(r"this\s+clip", re.IGNORECASE), "this clip"),
|
||||
(re.compile(r"the\s+video", re.IGNORECASE), "the video"),
|
||||
(re.compile(r"this\s+video", re.IGNORECASE), "this video"),
|
||||
(re.compile(r"this\s+scene", re.IGNORECASE), "this scene"),
|
||||
(re.compile(r"the\s+clip", re.IGNORECASE), "the clip"),
|
||||
(re.compile(r"this\s+footage", re.IGNORECASE), "this footage"),
|
||||
(re.compile(r"the\s+footage", re.IGNORECASE), "the footage"),
|
||||
(re.compile(r"this\s+segment", re.IGNORECASE), "this segment"),
|
||||
(re.compile(r"this\s+frame", re.IGNORECASE), "this frame"),
|
||||
(re.compile(r"the\s+current\s+frame", re.IGNORECASE), "the current frame"),
|
||||
(re.compile(r"frame\s+summary", re.IGNORECASE), "frame summary"),
|
||||
(re.compile(r"上面的片段", re.IGNORECASE), "上面的片段"),
|
||||
(re.compile(r"这段视频", re.IGNORECASE), "这段视频"),
|
||||
(re.compile(r"该视频", re.IGNORECASE), "该视频"),
|
||||
(re.compile(r"这个片段", re.IGNORECASE), "这个片段"),
|
||||
(re.compile(r"视频中", re.IGNORECASE), "视频中"),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 时间锚点正则与短语
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_TIMESTAMP_PATTERN = re.compile(r"\d{1,2}:\d{2}")
|
||||
|
||||
_TIME_ANCHOR_PHRASES: list[re.Pattern[str]] = [
|
||||
re.compile(r"at\s+the\s+beginning", re.IGNORECASE),
|
||||
re.compile(r"at\s+the\s+end", re.IGNORECASE),
|
||||
re.compile(r"in\s+the\s+beginning", re.IGNORECASE),
|
||||
re.compile(r"at\s+the\s+start", re.IGNORECASE),
|
||||
re.compile(r"开头", re.IGNORECASE),
|
||||
re.compile(r"结尾", re.IGNORECASE),
|
||||
re.compile(r"末尾", re.IGNORECASE),
|
||||
re.compile(r"片头", re.IGNORECASE),
|
||||
re.compile(r"片尾", re.IGNORECASE),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# T1 / T7 素材禁区正则
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# T1: 瞬时动作
|
||||
_T1_INSTANT_ACTION_PATTERNS: list[re.Pattern[str]] = [
|
||||
re.compile(r"瞬间|一闪而过|转瞬即逝|一瞬间|刹那", re.IGNORECASE),
|
||||
re.compile(r"flash|instant|split\s*second|blink", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# T1: 记分牌时序
|
||||
_T1_SCOREBOARD_PATTERNS: list[re.Pattern[str]] = [
|
||||
re.compile(r"记分牌|比分.*变|比分.*更新|得分.*变化", re.IGNORECASE),
|
||||
re.compile(r"scoreboard|score\s*(changed|updated|went)", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# T1: 无对白因果
|
||||
_T1_NO_DIALOGUE_CAUSAL_PATTERNS: list[re.Pattern[str]] = [
|
||||
re.compile(r"无对白.*因果|因果.*无法.*判断", re.IGNORECASE),
|
||||
re.compile(r"no\s+dialogue.*caus|cannot.*determin.*caus", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# T7: 选项重复
|
||||
_T7_OPTION_REPETITION_PATTERNS: list[re.Pattern[str]] = [
|
||||
re.compile(r"重复选项|选项.*相同|选项.*重复|完全相同", re.IGNORECASE),
|
||||
re.compile(r"duplicate\s+option|identical\s+option|same\s+option", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# T7: 计数边界口径含糊
|
||||
_T7_COUNTING_AMBIGUITY_PATTERNS: list[re.Pattern[str]] = [
|
||||
re.compile(r"大约.*数量|数量不确定|难以.*计数|不确定.*几|约.*个", re.IGNORECASE),
|
||||
re.compile(r"approximate.*count|uncertain.*number|hard\s+to\s+count", re.IGNORECASE),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 公开函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def shuffle_options(
|
||||
options: tuple[str, ...],
|
||||
answer: str,
|
||||
rng: random.Random,
|
||||
) -> tuple[tuple[str, ...], str]:
|
||||
"""确定性洗牌选项并重映射答案字母。
|
||||
|
||||
通过 index permutation 打乱选项顺序,为每个选项重新分配 A/B/C/D 前缀,
|
||||
并将答案字母映射到正确选项的新位置。
|
||||
|
||||
参数:
|
||||
options: 原始选项元组,格式为 ("A. text1", "B. text2", ...)。
|
||||
answer: 原始答案字母("A"/"B"/"C"/"D")。
|
||||
rng: 可控随机数生成器(保证确定性)。
|
||||
|
||||
返回:
|
||||
(新选项元组, 新答案字母) — 选项文本不变,仅前缀和顺序改变。
|
||||
"""
|
||||
# Phase 1: 提取纯文本(去掉 "X. " 前缀)
|
||||
texts = [opt[3:] for opt in options]
|
||||
|
||||
# Phase 2: 确定原正确选项的文本
|
||||
correct_idx = ord(answer) - ord("A")
|
||||
correct_text = texts[correct_idx]
|
||||
|
||||
# Phase 3: 生成随机排列
|
||||
indices = list(range(len(texts)))
|
||||
rng.shuffle(indices)
|
||||
|
||||
# Phase 4: 按排列重组,分配新前缀
|
||||
new_options: list[str] = []
|
||||
new_answer = ""
|
||||
for new_pos, old_idx in enumerate(indices):
|
||||
letter = chr(ord("A") + new_pos)
|
||||
new_options.append(f"{letter}. {texts[old_idx]}")
|
||||
if texts[old_idx] == correct_text:
|
||||
new_answer = letter
|
||||
|
||||
return tuple(new_options), new_answer
|
||||
|
||||
|
||||
def check_referent_blacklist(question_text: str) -> list[str]:
|
||||
"""检测题目文本中的自指代语言。
|
||||
|
||||
在 benchmark 题目中,自指代(如 "this clip"、"这段视频")会泄露视频上下文,
|
||||
使题目脱离视频后无法独立理解。
|
||||
|
||||
参数:
|
||||
question_text: 题目文本。
|
||||
|
||||
返回:
|
||||
违规描述列表,空列表表示通过。
|
||||
"""
|
||||
violations: list[str] = []
|
||||
for pattern, label in _BLACKLIST_PATTERNS:
|
||||
if pattern.search(question_text):
|
||||
violations.append(f"检测到自指代: '{label}'")
|
||||
return violations
|
||||
|
||||
|
||||
def _ngram_overlap_ratio(
|
||||
text_words: list[str],
|
||||
source_ngrams: set[tuple[str, ...]],
|
||||
window: int,
|
||||
) -> float:
|
||||
"""计算单段文本与来源 n-gram 集合的重叠率。
|
||||
|
||||
参数:
|
||||
text_words: 待比较文本的分词列表(已小写化)。
|
||||
source_ngrams: 来源素材的 n-gram 集合。
|
||||
window: n-gram 窗口大小。
|
||||
|
||||
返回:
|
||||
重叠率 [0.0, 1.0],词数不足窗口大小时返回 0.0。
|
||||
"""
|
||||
if len(text_words) < window:
|
||||
return 0.0
|
||||
|
||||
text_ngrams: set[tuple[str, ...]] = {
|
||||
tuple(text_words[i : i + window]) for i in range(len(text_words) - window + 1)
|
||||
}
|
||||
|
||||
if not text_ngrams:
|
||||
return 0.0
|
||||
|
||||
overlap = text_ngrams & source_ngrams
|
||||
return len(overlap) / len(text_ngrams)
|
||||
|
||||
|
||||
def check_verbatim(
|
||||
question_text: str,
|
||||
correct_option: str,
|
||||
source_texts: list[str],
|
||||
window: int = 6,
|
||||
) -> float:
|
||||
"""计算题目文本和正确选项与来源素材的最大逐字重复率。
|
||||
|
||||
使用滑动窗口 n-gram 集合交集方法:分别从题目文本和选项文本提取 n-gram,
|
||||
各自与来源文本的 n-gram 集合求交集,返回两者中较大的重叠比例。
|
||||
|
||||
参数:
|
||||
question_text: 题目文本。
|
||||
correct_option: 正确选项文本(含 "X. " 前缀)。
|
||||
source_texts: 来源素材文本列表。
|
||||
window: n-gram 窗口大小。
|
||||
|
||||
返回:
|
||||
最大重复率 [0.0, 1.0]。0.0 表示无重叠,1.0 表示完全复制。
|
||||
"""
|
||||
# Phase 1: 构造来源文本的 n-gram 集合
|
||||
source_ngrams: set[tuple[str, ...]] = set()
|
||||
for source in source_texts:
|
||||
words = source.lower().split()
|
||||
for i in range(len(words) - window + 1):
|
||||
source_ngrams.add(tuple(words[i : i + window]))
|
||||
|
||||
if not source_ngrams:
|
||||
return 0.0
|
||||
|
||||
# Phase 2: 计算题目文本的重叠率
|
||||
question_words = question_text.lower().split()
|
||||
question_ratio = _ngram_overlap_ratio(question_words, source_ngrams, window)
|
||||
|
||||
# Phase 3: 提取选项纯文本(去掉可能的 "X. " 前缀)
|
||||
option_text = correct_option
|
||||
if len(option_text) >= 3 and option_text[1] == "." and option_text[2] == " ":
|
||||
option_text = option_text[3:]
|
||||
|
||||
option_words = option_text.lower().split()
|
||||
option_ratio = _ngram_overlap_ratio(option_words, source_ngrams, window)
|
||||
|
||||
# Phase 4: 返回两者中较大的重叠率
|
||||
return max(question_ratio, option_ratio)
|
||||
|
||||
|
||||
def has_time_anchor(question_text: str) -> bool:
|
||||
"""检测题目中是否包含时间锚点。
|
||||
|
||||
时间锚点包括:数字时间戳(如 "01:30")和时间短语(如 "at the beginning"、"开头")。
|
||||
|
||||
参数:
|
||||
question_text: 题目文本。
|
||||
|
||||
返回:
|
||||
True 表示包含时间锚点。
|
||||
"""
|
||||
# Phase 1: 检查数字时间戳
|
||||
if _TIMESTAMP_PATTERN.search(question_text):
|
||||
return True
|
||||
|
||||
# Phase 2: 检查时间短语
|
||||
return any(pattern.search(question_text) for pattern in _TIME_ANCHOR_PHRASES)
|
||||
|
||||
|
||||
def _match_any(text: str, patterns: list[re.Pattern[str]]) -> bool:
|
||||
"""判断文本是否匹配任一正则模式。
|
||||
|
||||
参数:
|
||||
text: 待检测文本。
|
||||
patterns: 预编译正则列表。
|
||||
|
||||
返回:
|
||||
True 表示至少命中一条模式。
|
||||
"""
|
||||
return any(p.search(text) for p in patterns)
|
||||
|
||||
|
||||
# 素材禁区规则表:(模式列表, 违规描述)
|
||||
_FORBIDDEN_MATERIAL_RULES: list[tuple[list[re.Pattern[str]], str]] = [
|
||||
(_T1_INSTANT_ACTION_PATTERNS, "T1 违规: 素材包含瞬时动作描述,不适合出题"),
|
||||
(_T1_SCOREBOARD_PATTERNS, "T1 违规: 素材包含记分牌时序信息,不适合出题"),
|
||||
(_T1_NO_DIALOGUE_CAUSAL_PATTERNS, "T1 违规: 素材缺乏对白因果线索,不适合出题"),
|
||||
(_T7_OPTION_REPETITION_PATTERNS, "T7 违规: 素材暗示可能产生重复选项"),
|
||||
(_T7_COUNTING_AMBIGUITY_PATTERNS, "T7 违规: 素材包含计数边界含糊描述"),
|
||||
]
|
||||
|
||||
|
||||
def check_forbidden_material(source_nodes_text: str, task_type: str) -> list[str]:
|
||||
"""出题禁区:检测 T1 类素材和 T7 噪声模式。
|
||||
|
||||
T1 类素材(不适合出题的内容):
|
||||
- 瞬时动作:画面一闪而过,无法稳定观察
|
||||
- 记分牌时序:依赖数字变化的时序信息
|
||||
- 无对白因果:缺乏语言线索的因果推理
|
||||
|
||||
T7 噪声模式(选项质量问题):
|
||||
- 选项重复:多个选项表述相同
|
||||
- 计数边界口径含糊:数量描述不确定
|
||||
|
||||
参数:
|
||||
source_nodes_text: 来源节点的拼接文本。
|
||||
task_type: 题型名称(用于上下文感知检测)。
|
||||
|
||||
返回:
|
||||
违规描述列表,空列表表示通过。
|
||||
"""
|
||||
return [
|
||||
message
|
||||
for patterns, message in _FORBIDDEN_MATERIAL_RULES
|
||||
if _match_any(source_nodes_text, patterns)
|
||||
]
|
||||
|
||||
|
||||
def run_postprocess(
|
||||
question_text: str,
|
||||
options: tuple[str, ...],
|
||||
answer: str,
|
||||
source_texts: list[str],
|
||||
rng: random.Random,
|
||||
) -> PostprocessResult:
|
||||
"""编排全部后处理检查,返回汇总结果。
|
||||
|
||||
执行流程:
|
||||
1. 选项洗牌(确定性)
|
||||
2. 指代黑名单检测
|
||||
3. 逐字重复率计算(基于洗牌后的正确选项)
|
||||
4. 时间锚点检测
|
||||
|
||||
参数:
|
||||
question_text: 题目文本。
|
||||
options: 原始选项元组("A. text", "B. text", ...)。
|
||||
answer: 原始答案字母。
|
||||
source_texts: 来源素材文本列表。
|
||||
rng: 可控随机数生成器。
|
||||
|
||||
返回:
|
||||
PostprocessResult 汇总实例。
|
||||
"""
|
||||
# Phase 1: 选项洗牌
|
||||
shuffled_options, new_answer = shuffle_options(options, answer, rng)
|
||||
|
||||
# Phase 2: 指代黑名单
|
||||
referent_violations = check_referent_blacklist(question_text)
|
||||
|
||||
# Phase 3: 逐字重复率(用洗牌后正确选项的文本)
|
||||
correct_idx = ord(new_answer) - ord("A")
|
||||
correct_option_text = shuffled_options[correct_idx]
|
||||
verbatim_ratio = check_verbatim(question_text, correct_option_text, source_texts)
|
||||
|
||||
# Phase 4: 时间锚点
|
||||
time_anchor = has_time_anchor(question_text)
|
||||
|
||||
logger.debug(
|
||||
"后处理完成: referent_violations={}, verbatim_ratio={:.3f}, has_time_anchor={}",
|
||||
len(referent_violations),
|
||||
verbatim_ratio,
|
||||
time_anchor,
|
||||
)
|
||||
|
||||
return PostprocessResult(
|
||||
options=shuffled_options,
|
||||
answer=new_answer,
|
||||
referent_violations=referent_violations,
|
||||
verbatim_ratio=verbatim_ratio,
|
||||
has_time_anchor=time_anchor,
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,597 @@
|
||||
"""v2 素材采样器 — 基于采样约束的树节点采样与上下文收集。
|
||||
|
||||
在 v1 synthesizer 的基础上引入 SamplingConstraint 约束验证,
|
||||
为每次出题提供更丰富的素材上下文(字幕、跨 L2 上下文、帧路径)。
|
||||
|
||||
典型调用路径::
|
||||
|
||||
material = sample_material_v2(
|
||||
tree=tree_index,
|
||||
task_type="Action Reasoning",
|
||||
used_node_ids=already_used,
|
||||
rng=rng,
|
||||
level=2,
|
||||
constraint=my_constraint,
|
||||
)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
|
||||
from app.question_gen.families import SamplingConstraint
|
||||
from app.tree.index import L1Node, L2Node, TreeIndex
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 题型 → 采样层级映射
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_TASK_TYPE_TO_LEVEL: dict[str, int] = {
|
||||
# Level 3(细粒度帧级)
|
||||
"Action Recognition": 3,
|
||||
"Object Recognition": 3,
|
||||
"Attribute Perception": 3,
|
||||
"OCR Problems": 3,
|
||||
# Level 2(片段/事件级)
|
||||
"Action Reasoning": 2,
|
||||
"Object Reasoning": 2,
|
||||
"Information Synopsis": 2,
|
||||
"Counting Problem": 2,
|
||||
# Level 1(段落/场景级)
|
||||
"Temporal Reasoning": 1,
|
||||
"Temporal Perception": 1,
|
||||
"Spatial Reasoning": 1,
|
||||
"Spatial Perception": 1,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 数据类型
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AnchorContext:
|
||||
"""采样锚点上下文。
|
||||
|
||||
属性:
|
||||
node_id: 锚节点 ID。
|
||||
level: 锚节点所在层级(1/2/3)。
|
||||
l2_id: 锚节点所属的 L2 节点 ID(若自身为 L2 则等于 node_id;
|
||||
若为 L1 则取其第一个 L2 子节点 ID)。
|
||||
"""
|
||||
|
||||
node_id: str
|
||||
level: int
|
||||
l2_id: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MaterialContext:
|
||||
"""采样素材上下文 — 出题所需的全部素材打包。
|
||||
|
||||
属性:
|
||||
anchor: 采样锚点信息。
|
||||
source_nodes: 参与采样的节点 ID 元组。
|
||||
subtitle_sentences: 锚节点子树中收集的字幕句列表。
|
||||
frame_paths: 锚节点子树中可用的帧路径列表。
|
||||
cross_l2_texts: 跨 L2 段的上下文文本列表(仅 cross_l2_span 时填充)。
|
||||
"""
|
||||
|
||||
anchor: AnchorContext
|
||||
source_nodes: tuple[str, ...]
|
||||
subtitle_sentences: list[str]
|
||||
frame_paths: list[str]
|
||||
cross_l2_texts: list[str]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部索引辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _ResolvedSubtree:
|
||||
"""节点解析结果 — 将任意层级 node_id 统一解析为 L2 节点列表 + 父 L1。
|
||||
|
||||
属性:
|
||||
l2_nodes: 与目标节点关联的 L2 节点列表。
|
||||
parent_l1: 父 L1 节点(用于跨 L2 判断)。
|
||||
"""
|
||||
|
||||
l2_nodes: list[L2Node]
|
||||
parent_l1: L1Node | None
|
||||
|
||||
|
||||
def _find_l2_node(tree: TreeIndex, l2_id: str) -> tuple[L2Node, L1Node] | None:
|
||||
"""按 ID 定位 L2 节点及其父 L1。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
l2_id: L2 节点 ID。
|
||||
|
||||
返回:
|
||||
(L2Node, 父L1Node) 元组;未找到返回 None。
|
||||
"""
|
||||
for l1 in tree.roots:
|
||||
for l2 in l1.children:
|
||||
if l2.id == l2_id:
|
||||
return (l2, l1)
|
||||
return None
|
||||
|
||||
|
||||
def _find_l1_node(tree: TreeIndex, l1_id: str) -> L1Node | None:
|
||||
"""按 ID 定位 L1 节点。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
l1_id: L1 节点 ID。
|
||||
|
||||
返回:
|
||||
L1Node;未找到返回 None。
|
||||
"""
|
||||
for l1 in tree.roots:
|
||||
if l1.id == l1_id:
|
||||
return l1
|
||||
return None
|
||||
|
||||
|
||||
def _find_l3_parent(tree: TreeIndex, l3_id: str) -> tuple[L2Node, L1Node] | None:
|
||||
"""按 L3 ID 定位其所属的 L2 和 L1 节点。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
l3_id: L3 节点 ID。
|
||||
|
||||
返回:
|
||||
(L2Node, L1Node) 元组;未找到返回 None。
|
||||
"""
|
||||
for l1 in tree.roots:
|
||||
for l2 in l1.children:
|
||||
for l3 in l2.children:
|
||||
if l3.id == l3_id:
|
||||
return (l2, l1)
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_subtree(tree: TreeIndex, node_id: str) -> _ResolvedSubtree | None:
|
||||
"""将任意层级节点 ID 解析为关联的 L2 节点列表和父 L1。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
node_id: 任意层级的节点 ID。
|
||||
|
||||
返回:
|
||||
_ResolvedSubtree 实例;节点不存在时返回 None。
|
||||
"""
|
||||
# 尝试作为 L2
|
||||
result = _find_l2_node(tree, node_id)
|
||||
if result is not None:
|
||||
l2_node, parent_l1 = result
|
||||
return _ResolvedSubtree(l2_nodes=[l2_node], parent_l1=parent_l1)
|
||||
|
||||
# 尝试作为 L1
|
||||
l1_node = _find_l1_node(tree, node_id)
|
||||
if l1_node is not None:
|
||||
return _ResolvedSubtree(l2_nodes=list(l1_node.children), parent_l1=l1_node)
|
||||
|
||||
# 尝试作为 L3
|
||||
l3_result = _find_l3_parent(tree, node_id)
|
||||
if l3_result is not None:
|
||||
l2_node, parent_l1 = l3_result
|
||||
return _ResolvedSubtree(l2_nodes=[l2_node], parent_l1=parent_l1)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 约束检查辅助(单一职责)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _count_l3_descendants(l2_nodes: list[L2Node]) -> int:
|
||||
"""统计 L2 节点列表下的 L3 总数。
|
||||
|
||||
参数:
|
||||
l2_nodes: L2 节点列表。
|
||||
|
||||
返回:
|
||||
L3 节点总数。
|
||||
"""
|
||||
return sum(len(l2.children) for l2 in l2_nodes)
|
||||
|
||||
|
||||
def _has_frames(l2_nodes: list[L2Node]) -> bool:
|
||||
"""检查 L2 节点列表的子树中是否有可用帧。
|
||||
|
||||
参数:
|
||||
l2_nodes: L2 节点列表。
|
||||
|
||||
返回:
|
||||
True 表示至少有一个 L3 有 frame_path。
|
||||
"""
|
||||
return any(l3.frame_path for l2 in l2_nodes for l3 in l2.children)
|
||||
|
||||
|
||||
def _count_subtitles(l2_nodes: list[L2Node]) -> int:
|
||||
"""统计 L2 节点列表中全部字幕数(L2 + L3)。
|
||||
|
||||
参数:
|
||||
l2_nodes: L2 节点列表。
|
||||
|
||||
返回:
|
||||
非空字幕总数。
|
||||
"""
|
||||
count = 0
|
||||
for l2 in l2_nodes:
|
||||
if l2.card.subtitle:
|
||||
count += 1
|
||||
count += sum(1 for l3 in l2.children if l3.card.subtitle)
|
||||
return count
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 公开辅助函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _validate_sampling_constraints(
|
||||
tree: TreeIndex, node_id: str, constraint: SamplingConstraint
|
||||
) -> bool:
|
||||
"""校验指定节点是否满足采样约束。
|
||||
|
||||
根据节点层级自动判断检查范围:
|
||||
- L2 节点:检查其子 L3 的帧/字幕数量。
|
||||
- L1 节点:检查其下全部 L2/L3 的帧/字幕总数。
|
||||
- L3 节点:检查其所属 L2 的子树。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
node_id: 待检查节点 ID。
|
||||
constraint: 采样约束条件。
|
||||
|
||||
返回:
|
||||
True 表示满足所有约束,False 表示至少一项不满足。
|
||||
"""
|
||||
resolved = _resolve_subtree(tree, node_id)
|
||||
if resolved is None:
|
||||
return False
|
||||
|
||||
if _count_l3_descendants(resolved.l2_nodes) < constraint.min_l3_nodes:
|
||||
return False
|
||||
if constraint.require_frames and not _has_frames(resolved.l2_nodes):
|
||||
return False
|
||||
if _count_subtitles(resolved.l2_nodes) < constraint.min_subtitles:
|
||||
return False
|
||||
return not (
|
||||
constraint.cross_l2_span
|
||||
and (resolved.parent_l1 is None or len(resolved.parent_l1.children) < 2)
|
||||
)
|
||||
|
||||
|
||||
def _subtitles_from_l2_list(l2_nodes: list[L2Node]) -> list[str]:
|
||||
"""从 L2 节点列表收集全部非空字幕。
|
||||
|
||||
参数:
|
||||
l2_nodes: L2 节点列表。
|
||||
|
||||
返回:
|
||||
非空字幕字符串列表。
|
||||
"""
|
||||
sentences: list[str] = []
|
||||
for l2 in l2_nodes:
|
||||
if l2.card.subtitle:
|
||||
sentences.append(l2.card.subtitle)
|
||||
for l3 in l2.children:
|
||||
if l3.card.subtitle:
|
||||
sentences.append(l3.card.subtitle)
|
||||
return sentences
|
||||
|
||||
|
||||
def _collect_subtitle_sentences(tree: TreeIndex, node_ids: tuple[str, ...]) -> list[str]:
|
||||
"""从指定节点集合中收集字幕句。
|
||||
|
||||
遍历每个 node_id 对应的子树,提取非空字幕。
|
||||
对 L2 节点提取自身 + 子 L3 字幕;对 L1 提取下属全部。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
node_ids: 待收集字幕的节点 ID 元组。
|
||||
|
||||
返回:
|
||||
非空字幕句列表。
|
||||
"""
|
||||
sentences: list[str] = []
|
||||
for nid in node_ids:
|
||||
resolved = _resolve_subtree(tree, nid)
|
||||
if resolved is not None:
|
||||
sentences.extend(_subtitles_from_l2_list(resolved.l2_nodes))
|
||||
return sentences
|
||||
|
||||
|
||||
def _collect_cross_l2_context(tree: TreeIndex, anchor_l2_id: str, max_peers: int = 3) -> list[str]:
|
||||
"""收集锚 L2 的同级 L2 节点描述文本(跨 L2 上下文)。
|
||||
|
||||
找到锚 L2 所属的 L1 父节点,取该父节点下除锚 L2 之外的其他 L2 描述。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
anchor_l2_id: 锚 L2 节点 ID。
|
||||
max_peers: 最多返回的同级 L2 描述数量。
|
||||
|
||||
返回:
|
||||
同级 L2 的 event_description 列表(最多 max_peers 条)。
|
||||
"""
|
||||
result = _find_l2_node(tree, anchor_l2_id)
|
||||
if result is None:
|
||||
return []
|
||||
|
||||
_, parent_l1 = result
|
||||
peers: list[str] = []
|
||||
for l2 in parent_l1.children:
|
||||
if l2.id != anchor_l2_id:
|
||||
peers.append(l2.card.event_description)
|
||||
if len(peers) >= max_peers:
|
||||
break
|
||||
|
||||
return peers
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 层级采样策略
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sample_l3_node(
|
||||
tree: TreeIndex,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> tuple[str, str] | None:
|
||||
"""随机采样一个未使用的 L3 节点,返回 (l3_id, 所属l2_id)。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
(l3_id, l2_id) 元组;无候选返回 None。
|
||||
"""
|
||||
candidates: list[tuple[str, str]] = []
|
||||
for l1 in tree.roots:
|
||||
for l2 in l1.children:
|
||||
for l3 in l2.children:
|
||||
if l3.id not in used_node_ids:
|
||||
candidates.append((l3.id, l2.id))
|
||||
|
||||
if not candidates:
|
||||
return None
|
||||
return rng.choice(candidates)
|
||||
|
||||
|
||||
def _sample_l2_node(
|
||||
tree: TreeIndex,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> tuple[str, str] | None:
|
||||
"""随机采样一个未使用的 L2 节点,返回 (l2_id, l2_id)。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
(l2_id, l2_id) 元组;无候选返回 None。
|
||||
"""
|
||||
candidates: list[str] = []
|
||||
for l1 in tree.roots:
|
||||
for l2 in l1.children:
|
||||
if l2.id not in used_node_ids:
|
||||
candidates.append(l2.id)
|
||||
|
||||
if not candidates:
|
||||
return None
|
||||
chosen = rng.choice(candidates)
|
||||
return (chosen, chosen)
|
||||
|
||||
|
||||
def _sample_l1_node(
|
||||
tree: TreeIndex,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> tuple[str, str] | None:
|
||||
"""随机采样一个未使用的 L1 节点,返回 (l1_id, 首个子l2_id)。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
(l1_id, first_l2_id) 元组;无候选返回 None。
|
||||
"""
|
||||
candidates: list[tuple[str, str]] = []
|
||||
for l1 in tree.roots:
|
||||
if l1.id not in used_node_ids and l1.children:
|
||||
candidates.append((l1.id, l1.children[0].id))
|
||||
|
||||
if not candidates:
|
||||
return None
|
||||
return rng.choice(candidates)
|
||||
|
||||
|
||||
def _frames_from_l2_list(l2_nodes: list[L2Node]) -> list[str]:
|
||||
"""从 L2 节点列表收集全部可用帧路径。
|
||||
|
||||
参数:
|
||||
l2_nodes: L2 节点列表。
|
||||
|
||||
返回:
|
||||
帧路径字符串列表。
|
||||
"""
|
||||
return [l3.frame_path for l2 in l2_nodes for l3 in l2.children if l3.frame_path]
|
||||
|
||||
|
||||
def _collect_frame_paths(tree: TreeIndex, node_id: str) -> list[str]:
|
||||
"""收集节点子树下的所有可用帧路径。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
node_id: 目标节点 ID。
|
||||
|
||||
返回:
|
||||
帧路径列表。
|
||||
"""
|
||||
resolved = _resolve_subtree(tree, node_id)
|
||||
if resolved is None:
|
||||
return []
|
||||
return _frames_from_l2_list(resolved.l2_nodes)
|
||||
|
||||
|
||||
def _collect_source_nodes(tree: TreeIndex, node_id: str) -> tuple[str, ...]:
|
||||
"""收集节点子树涉及的全部节点 ID(包含自身)。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
node_id: 目标节点 ID。
|
||||
|
||||
返回:
|
||||
相关节点 ID 元组。
|
||||
"""
|
||||
ids: list[str] = [node_id]
|
||||
|
||||
# L2 节点:加入子 L3
|
||||
result = _find_l2_node(tree, node_id)
|
||||
if result is not None:
|
||||
l2_node, _ = result
|
||||
for l3 in l2_node.children:
|
||||
ids.append(l3.id)
|
||||
return tuple(ids)
|
||||
|
||||
# L1 节点:加入子 L2 + L3
|
||||
l1_node = _find_l1_node(tree, node_id)
|
||||
if l1_node is not None:
|
||||
for l2 in l1_node.children:
|
||||
ids.append(l2.id)
|
||||
for l3 in l2.children:
|
||||
ids.append(l3.id)
|
||||
return tuple(ids)
|
||||
|
||||
# L3 节点:仅自身
|
||||
return tuple(ids)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sample_material_v2(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
*,
|
||||
level: int,
|
||||
constraint: SamplingConstraint,
|
||||
max_attempts: int = 10,
|
||||
) -> MaterialContext:
|
||||
"""基于采样约束从视频树中采样素材上下文。
|
||||
|
||||
采样流程:
|
||||
1. 按指定 level 确定采样层级
|
||||
2. 随机选取候选节点(排除 used_node_ids)
|
||||
3. 验证 SamplingConstraint 约束
|
||||
4. 约束不满足则重试(最多 max_attempts 次)
|
||||
5. 收集字幕、帧路径、跨 L2 上下文
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 任务类型字符串。
|
||||
used_node_ids: 本轮已用节点 ID 集合。
|
||||
rng: 可控随机数生成器。
|
||||
level: 采样层级(1/2/3)。
|
||||
constraint: 采样约束条件。
|
||||
max_attempts: 最大尝试次数。
|
||||
|
||||
返回:
|
||||
MaterialContext 实例。
|
||||
|
||||
异常:
|
||||
ValueError: level 不在 {1, 2, 3} 中。
|
||||
RuntimeError: 耗尽 max_attempts 次尝试仍无法满足约束。
|
||||
"""
|
||||
if level not in (1, 2, 3):
|
||||
msg = f"level 必须为 1、2 或 3,收到: {level}"
|
||||
raise ValueError(msg)
|
||||
|
||||
for attempt in range(max_attempts):
|
||||
# Phase 1: 按层级采样候选节点
|
||||
if level == 3:
|
||||
sampled = _sample_l3_node(tree, used_node_ids, rng)
|
||||
elif level == 2:
|
||||
sampled = _sample_l2_node(tree, used_node_ids, rng)
|
||||
else:
|
||||
sampled = _sample_l1_node(tree, used_node_ids, rng)
|
||||
|
||||
if sampled is None:
|
||||
logger.debug(
|
||||
"sample_material_v2 尝试 {}/{}: 无可用候选节点 (level={})",
|
||||
attempt + 1,
|
||||
max_attempts,
|
||||
level,
|
||||
)
|
||||
continue
|
||||
|
||||
node_id, l2_id = sampled
|
||||
|
||||
# Phase 2: 验证约束
|
||||
if not _validate_sampling_constraints(tree, node_id, constraint):
|
||||
logger.debug(
|
||||
"sample_material_v2 尝试 {}/{}: 约束违反 (node={})",
|
||||
attempt + 1,
|
||||
max_attempts,
|
||||
node_id,
|
||||
)
|
||||
continue
|
||||
|
||||
# Phase 3: 构造 AnchorContext
|
||||
anchor = AnchorContext(node_id=node_id, level=level, l2_id=l2_id)
|
||||
|
||||
# Phase 4: 收集素材
|
||||
source_nodes = _collect_source_nodes(tree, node_id)
|
||||
subtitle_sentences = _collect_subtitle_sentences(tree, (node_id,))
|
||||
frame_paths = _collect_frame_paths(tree, node_id)
|
||||
|
||||
# Phase 5: 跨 L2 上下文(仅 cross_l2_span 时收集)
|
||||
cross_l2_texts: list[str] = []
|
||||
if constraint.cross_l2_span:
|
||||
cross_l2_texts = _collect_cross_l2_context(tree, l2_id)
|
||||
|
||||
logger.debug(
|
||||
"sample_material_v2 成功: node={}, level={}, attempt={}/{}",
|
||||
node_id,
|
||||
level,
|
||||
attempt + 1,
|
||||
max_attempts,
|
||||
)
|
||||
|
||||
return MaterialContext(
|
||||
anchor=anchor,
|
||||
source_nodes=source_nodes,
|
||||
subtitle_sentences=subtitle_sentences,
|
||||
frame_paths=frame_paths,
|
||||
cross_l2_texts=cross_l2_texts,
|
||||
)
|
||||
|
||||
raise RuntimeError(
|
||||
f"sample_material_v2: 耗尽 max_attempts={max_attempts} 次尝试,"
|
||||
f"无法为 task_type='{task_type}' (level={level}) 满足采样约束"
|
||||
)
|
||||
@@ -0,0 +1,262 @@
|
||||
"""题型出题策略 — Clean Architecture 的 Strategy 层。
|
||||
|
||||
将出题管线从 5 个粗粒度 QuestionFamilySpec 替换为 12 个题型级别的
|
||||
TaskTypeStrategy。BaseTaskTypeStrategy 封装现有 family 行为,
|
||||
特化策略(如 ActionRecognitionStrategy)自包含。
|
||||
|
||||
典型用法::
|
||||
|
||||
strategy = get_strategy("Action Reasoning")
|
||||
level = strategy.sampling_level
|
||||
constraint = strategy.sampling_constraint
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
from app.question_gen.families import (
|
||||
ENUMERATION_FAMILY,
|
||||
REASONING_FAMILY,
|
||||
RETRIEVAL_FAMILY,
|
||||
SPATIAL_FAMILY,
|
||||
VISUAL_FAMILY,
|
||||
QuestionFamilySpec,
|
||||
SamplingConstraint,
|
||||
)
|
||||
from app.question_gen.sampler_v2 import _TASK_TYPE_TO_LEVEL
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SubPattern
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SubPattern:
|
||||
"""出题子模式 — 靶向特定失败机制。
|
||||
|
||||
属性:
|
||||
name: 子模式标识名。
|
||||
weight: 选择权重(按比例分配)。
|
||||
sampling_level_override: 覆盖策略默认采样层级(None = 不覆盖)。
|
||||
constraint_override: 覆盖策略默认采样约束(None = 不覆盖)。
|
||||
instruction: 核心出题指令(1-3 句话)。
|
||||
positive_examples: VME 原题 few-shot 示范。
|
||||
negative_examples: 反面示例。
|
||||
distractor_rules: 干扰项构造规则。
|
||||
supports_flip: 是否支持配对翻转门(Phase B 用,默认 False)。
|
||||
flip_axis: 翻转轴("before/after" | "first/last"),None 表示不翻转。
|
||||
"""
|
||||
|
||||
name: str
|
||||
weight: float
|
||||
sampling_level_override: int | None
|
||||
constraint_override: SamplingConstraint | None
|
||||
instruction: str
|
||||
positive_examples: list[dict] = field(default_factory=list)
|
||||
negative_examples: list[dict] = field(default_factory=list)
|
||||
distractor_rules: str = ""
|
||||
supports_flip: bool = False
|
||||
flip_axis: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Protocol
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class TaskTypeStrategy(Protocol):
|
||||
"""题型出题策略 — pipeline 的唯一接口。"""
|
||||
|
||||
@property
|
||||
def task_type(self) -> str: ...
|
||||
|
||||
@property
|
||||
def sampling_level(self) -> int: ...
|
||||
|
||||
@property
|
||||
def sampling_constraint(self) -> SamplingConstraint: ...
|
||||
|
||||
@property
|
||||
def prompt_template(self) -> str: ...
|
||||
|
||||
@property
|
||||
def strategy_name(self) -> str: ...
|
||||
|
||||
@property
|
||||
def skill_target(self) -> str: ...
|
||||
|
||||
@property
|
||||
def leak_probe_template(self) -> str: ...
|
||||
|
||||
@property
|
||||
def uses_grounded_selector(self) -> bool: ...
|
||||
|
||||
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None: ...
|
||||
|
||||
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict: ...
|
||||
|
||||
def extra_gates(self, candidate: Any) -> list: ...
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BaseTaskTypeStrategy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# 消歧绑定表:多归属题型确定性绑定到一个 family
|
||||
_TASK_TYPE_TO_FAMILY: dict[str, QuestionFamilySpec] = {
|
||||
"Action Recognition": VISUAL_FAMILY, # fallback — 注册表中已被 ActionRecognitionStrategy 替换
|
||||
"Object Recognition": RETRIEVAL_FAMILY,
|
||||
"Object Reasoning": REASONING_FAMILY,
|
||||
"Action Reasoning": REASONING_FAMILY,
|
||||
"Attribute Perception": VISUAL_FAMILY,
|
||||
"OCR Problems": VISUAL_FAMILY,
|
||||
"Counting Problem": ENUMERATION_FAMILY,
|
||||
"Information Synopsis": REASONING_FAMILY,
|
||||
"Temporal Reasoning": ENUMERATION_FAMILY,
|
||||
"Temporal Perception": ENUMERATION_FAMILY,
|
||||
"Spatial Reasoning": SPATIAL_FAMILY,
|
||||
"Spatial Perception": SPATIAL_FAMILY,
|
||||
}
|
||||
|
||||
|
||||
class BaseTaskTypeStrategy:
|
||||
"""封装现有 QuestionFamilySpec 行为的默认策略。
|
||||
|
||||
所有属性委托给绑定的 family,确保未特化题型的行为不变。
|
||||
|
||||
参数:
|
||||
task_type: 题型名。
|
||||
family: 绑定的 QuestionFamilySpec。
|
||||
"""
|
||||
|
||||
def __init__(self, task_type: str, family: QuestionFamilySpec) -> None:
|
||||
self._task_type = task_type
|
||||
self._family = family
|
||||
|
||||
@property
|
||||
def task_type(self) -> str:
|
||||
"""返回题型名。"""
|
||||
return self._task_type
|
||||
|
||||
@property
|
||||
def sampling_level(self) -> int:
|
||||
"""返回采样层级(从 _TASK_TYPE_TO_LEVEL 查询)。"""
|
||||
return _TASK_TYPE_TO_LEVEL[self._task_type]
|
||||
|
||||
@property
|
||||
def sampling_constraint(self) -> SamplingConstraint:
|
||||
"""返回采样约束(委托给绑定的 family)。"""
|
||||
return self._family.sampling
|
||||
|
||||
@property
|
||||
def prompt_template(self) -> str:
|
||||
"""返回 prompt 模板文件名(委托给绑定的 family)。"""
|
||||
return self._family.prompt_template
|
||||
|
||||
@property
|
||||
def strategy_name(self) -> str:
|
||||
"""返回策略名(即 family.name)。"""
|
||||
return self._family.name
|
||||
|
||||
@property
|
||||
def skill_target(self) -> str:
|
||||
"""返回目标失败机制编号(委托给绑定的 family)。"""
|
||||
return self._family.skill_target
|
||||
|
||||
@property
|
||||
def leak_probe_template(self) -> str:
|
||||
"""返回泄漏探测模板文件名(委托给绑定的 family.leak_profile)。"""
|
||||
return self._family.leak_profile.probe_template
|
||||
|
||||
@property
|
||||
def uses_grounded_selector(self) -> bool:
|
||||
"""默认不启用 grounded selector(11 类题型走原路径)。"""
|
||||
return False
|
||||
|
||||
def select_sub_pattern(self, rng: random.Random) -> SubPattern | None:
|
||||
"""BaseTaskTypeStrategy 无子模式。"""
|
||||
return None
|
||||
|
||||
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict:
|
||||
"""返回基础 prompt 上下文(family_name + prompt_template)。"""
|
||||
return {
|
||||
"family_name": self._family.name,
|
||||
"prompt_template": self._family.prompt_template,
|
||||
}
|
||||
|
||||
def extra_gates(self, candidate: Any) -> list:
|
||||
"""BaseTaskTypeStrategy 无额外 gate。"""
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 注册表
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_STRATEGY_REGISTRY: dict[str, TaskTypeStrategy] = {}
|
||||
|
||||
|
||||
def register_strategy(strategy: TaskTypeStrategy) -> None:
|
||||
"""注册一个题型策略。同一 task_type 重复注册会覆盖。
|
||||
|
||||
参数:
|
||||
strategy: 实现 TaskTypeStrategy 接口的策略实例。
|
||||
"""
|
||||
_STRATEGY_REGISTRY[strategy.task_type] = strategy
|
||||
|
||||
|
||||
_BUILTIN_REGISTERED = False
|
||||
|
||||
|
||||
def get_strategy(task_type: str) -> TaskTypeStrategy:
|
||||
"""获取题型策略。未注册的自动创建 BaseTaskTypeStrategy。
|
||||
|
||||
首次调用时延迟注册内建特化策略(避免循环导入)。
|
||||
|
||||
参数:
|
||||
task_type: 题型名。
|
||||
|
||||
返回:
|
||||
TaskTypeStrategy 实例。
|
||||
|
||||
异常:
|
||||
KeyError: task_type 不在消歧绑定表和注册表中。
|
||||
"""
|
||||
global _BUILTIN_REGISTERED # noqa: PLW0603
|
||||
if not _BUILTIN_REGISTERED:
|
||||
_BUILTIN_REGISTERED = True
|
||||
_register_builtin_strategies()
|
||||
|
||||
if task_type in _STRATEGY_REGISTRY:
|
||||
return _STRATEGY_REGISTRY[task_type]
|
||||
return _build_default_strategy(task_type)
|
||||
|
||||
|
||||
def _build_default_strategy(task_type: str) -> BaseTaskTypeStrategy:
|
||||
"""根据消歧绑定表自动构造 BaseTaskTypeStrategy。
|
||||
|
||||
参数:
|
||||
task_type: 题型名。
|
||||
|
||||
返回:
|
||||
BaseTaskTypeStrategy 实例。
|
||||
|
||||
异常:
|
||||
KeyError: task_type 不在消歧绑定表中。
|
||||
"""
|
||||
family = _TASK_TYPE_TO_FAMILY[task_type]
|
||||
return BaseTaskTypeStrategy(task_type=task_type, family=family)
|
||||
|
||||
|
||||
def _register_builtin_strategies() -> None:
|
||||
"""注册内建的特化策略。由 get_strategy 首次调用时延迟执行。"""
|
||||
from app.question_gen.strategy_action_recognition import ActionRecognitionStrategy
|
||||
|
||||
register_strategy(ActionRecognitionStrategy())
|
||||
@@ -0,0 +1,347 @@
|
||||
"""Action Recognition 特化出题策略。
|
||||
|
||||
靶向 Agent 在动作识别类题目上的 6 种典型失败机制,通过加权随机
|
||||
SubPattern 选择为 VLM 出题提供聚焦指令。
|
||||
|
||||
与 BaseTaskTypeStrategy 完全自包含,不依赖 QuestionFamilySpec。
|
||||
采样从 L3 提升至 L2(跨段推理需要更广视野),3 个 SubPattern
|
||||
进一步覆盖至 L1 以测试全局追踪能力。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.question_gen.families import SamplingConstraint
|
||||
from app.question_gen.strategy import SubPattern
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6 个 SubPattern 定义
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_PREMATURE_EVIDENCE_ANCHORING = SubPattern(
|
||||
name="premature_evidence_anchoring",
|
||||
weight=0.20,
|
||||
sampling_level_override=1,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道需要考察视频中完整证据链的动作识别题。"
|
||||
"正确答案的关键证据出现在视频后半段或跨多个片段,"
|
||||
"但视频前段包含一个看似合理的局部匹配——"
|
||||
"Agent 若仅凭首条匹配停止搜索就会出错。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "视频中厨师最终采用了哪种烹饪方式?",
|
||||
"answer": "B. 蒸",
|
||||
"why": "厨师先演示了炒(前段),但最终菜品使用蒸制(后段),"
|
||||
"锚定首段证据的 Agent 会错选'炒'。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "视频开头厨师在做什么?",
|
||||
"why": "答案仅需前段信息,不会触发过早锚定失败。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"将视频前段真实出现的局部匹配动作设为强干扰项——它真实发生,"
|
||||
"仅在【时点】这一单一维度上与正解不同(前段 vs 最终结论段)。"
|
||||
"其余干扰项亦须是视频中真实发生的动作,严禁缺席事件。"
|
||||
),
|
||||
)
|
||||
|
||||
_TEMPORAL_REASONING_FAILURE = SubPattern(
|
||||
name="temporal_reasoning_failure",
|
||||
weight=0.20,
|
||||
sampling_level_override=1,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道要求正确排序或定位第 N 次出现的动作识别题。"
|
||||
"题目需要 Agent 追踪事件的时间先后顺序,"
|
||||
"或准确识别某动作在视频中第几次出现。"
|
||||
"打乱时序或错误计数即会答错。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "运动员第三次尝试起跳前做了什么准备动作?",
|
||||
"answer": "C. 深蹲热身",
|
||||
"why": "需要准确定位'第三次'起跳而非其他次,"
|
||||
"时序推理失败的 Agent 会混淆不同次尝试的准备动作。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "运动员在视频中做了什么?",
|
||||
"why": "不涉及时序排序或计数,Agent 无需追踪顺序。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"干扰项必须是视频中真实发生的事件,仅在【事件时序/顺序】这一单一维度上与正解不同——"
|
||||
"即同一组真实事件的错误排列或错误的第 N 次定位。"
|
||||
"严禁使用视频中未出现的缺席事件作为干扰项。"
|
||||
),
|
||||
supports_flip=True,
|
||||
flip_axis="before/after",
|
||||
)
|
||||
|
||||
_SEMANTIC_RIGIDITY = SubPattern(
|
||||
name="semantic_rigidity",
|
||||
weight=0.15,
|
||||
sampling_level_override=None,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道动作识别题,正确选项使用与视频原始描述不同的同义表达。"
|
||||
"Agent 需理解语义等价而非依赖字面匹配——"
|
||||
"例如视频字幕说'奔跑',正确选项写作'快速移动'。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "工人对墙面进行了什么操作?",
|
||||
"answer": "A. 涂覆保护层",
|
||||
"why": "视频中字幕描述为'刷漆',正确答案改写为'涂覆保护层',"
|
||||
"依赖字面匹配的 Agent 会因找不到完全一致的表述而错选。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "工人在刷漆吗?",
|
||||
"why": "选项直接复用视频原文,不考察语义理解。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"干扰项须基于视频真实内容,仅在【表述/语义】这一单一维度上做文章:"
|
||||
"保留一个复用视频原始字幕字面、但在题干限定下语义为假的选项作为陷阱,"
|
||||
"其余选项描述真实动作的不同同义表述。严禁凭空编造缺席动作。"
|
||||
),
|
||||
)
|
||||
|
||||
_FINE_GRAINED_VISUAL_ACTION = SubPattern(
|
||||
name="fine_grained_visual_action",
|
||||
weight=0.15,
|
||||
sampling_level_override=None,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道需要区分细粒度动作方式的识别题。"
|
||||
"题目聚焦于 HOW(怎么做)而非 WHAT(做什么),"
|
||||
"如区分'搅拌'与'翻炒'、'拧'与'拉'等视觉上相似但方式不同的动作。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "维修人员是如何拆卸螺丝的?",
|
||||
"answer": "D. 用扳手逆时针旋转",
|
||||
"why": "需要区分拆卸的具体方式(扳手 vs 螺丝刀、顺时针 vs 逆时针),"
|
||||
"粗粒度识别只能判断'在拆螺丝',无法区分方式。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "维修人员在做什么?",
|
||||
"why": "只需粗粒度动作识别('拆螺丝'),不考察具体方式。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"四个选项须是【同一大类动作】的不同执行方式,全部为视频中真实可见的做法,"
|
||||
"仅在【执行方式】这一单一维度上不同(如顺/逆时针、扳手/螺丝刀)。"
|
||||
"严禁使用明显不相关或视频中未出现的动作作为干扰项。"
|
||||
),
|
||||
)
|
||||
|
||||
_CROSS_SEGMENT_ENTITY_TRACKING = SubPattern(
|
||||
name="cross_segment_entity_tracking",
|
||||
weight=0.15,
|
||||
sampling_level_override=1,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道需要跨视频段落追踪同一实体动作的识别题。"
|
||||
"目标实体在不同片段中外观、称呼或上下文发生变化,"
|
||||
"Agent 需要将多段信息合并才能正确回答关于该实体的动作问题。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "穿红色外套的人在视频中总共完成了哪些动作?",
|
||||
"answer": "B. 先讲解、后示范、最后总结",
|
||||
"why": "该人物在前段穿外套讲解,中段脱外套示范,后段重新穿上总结,"
|
||||
"无法跨段追踪的 Agent 会遗漏某段动作。",
|
||||
},
|
||||
{
|
||||
"question": "主持人在节目不同环节中分别做了什么?",
|
||||
"answer": "A. 开场介绍、采访嘉宾、总结点评",
|
||||
"why": "主持人在不同场景切换中持续出现,需要跨段聚合。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "视频第一个片段中的人在做什么?",
|
||||
"why": "仅需单段信息,不考察跨段追踪。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"干扰项须是视频中【另一真实实体】在相应片段真实做过的动作,"
|
||||
"仅在【动作主体】这一单一维度上与正解不同。"
|
||||
"严禁编造任何实体未做过的缺席动作。"
|
||||
),
|
||||
supports_flip=True,
|
||||
flip_axis="first/last",
|
||||
)
|
||||
|
||||
_EVIDENCE_GAP_CONFABULATION = SubPattern(
|
||||
name="evidence_gap_confabulation",
|
||||
weight=0.15,
|
||||
sampling_level_override=None,
|
||||
constraint_override=None,
|
||||
instruction=(
|
||||
"设计一道动作识别题,视频中存在证据空缺(如遮挡、跳切、画外音)。"
|
||||
"正确答案承认信息不足或基于间接证据推断,"
|
||||
"而非凭空编造因果链。Agent 若虚构缺失证据即会出错。"
|
||||
),
|
||||
positive_examples=[
|
||||
{
|
||||
"question": "画面切走后,演讲者下一步做了什么?",
|
||||
"answer": "C. 无法从视频中直接确定",
|
||||
"why": "画面跳切导致该动作无直接视觉证据,"
|
||||
"倾向于虚构的 Agent 会编造一个看似合理的动作。",
|
||||
},
|
||||
],
|
||||
negative_examples=[
|
||||
{
|
||||
"question": "画面中演讲者正在做什么?",
|
||||
"why": "动作在画面中可见,不存在证据空缺。",
|
||||
},
|
||||
],
|
||||
distractor_rules=(
|
||||
"正解仅陈述视频中可观测的事实或诚实承认证据不足;"
|
||||
"干扰项在【因果完整性】这一单一维度上越界——补上一段视频未展示的因果链,"
|
||||
"但其前提元素仍取自视频真实内容(诱导 Agent 顺势编造),而非完全凭空的缺席事件。"
|
||||
),
|
||||
)
|
||||
|
||||
AR_SUB_PATTERNS: tuple[SubPattern, ...] = (
|
||||
_PREMATURE_EVIDENCE_ANCHORING,
|
||||
_TEMPORAL_REASONING_FAILURE,
|
||||
_SEMANTIC_RIGIDITY,
|
||||
_FINE_GRAINED_VISUAL_ACTION,
|
||||
_CROSS_SEGMENT_ENTITY_TRACKING,
|
||||
_EVIDENCE_GAP_CONFABULATION,
|
||||
)
|
||||
|
||||
# 预计算:名称列表 + 权重列表(避免每次 select 重新构建)
|
||||
_AR_PATTERN_NAMES: list[str] = [sp.name for sp in AR_SUB_PATTERNS]
|
||||
_AR_PATTERN_WEIGHTS: list[float] = [sp.weight for sp in AR_SUB_PATTERNS]
|
||||
_AR_PATTERN_BY_NAME: dict[str, SubPattern] = {sp.name: sp for sp in AR_SUB_PATTERNS}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ActionRecognitionStrategy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SAMPLING_CONSTRAINT = SamplingConstraint(
|
||||
min_subtitles=3,
|
||||
min_l3_nodes=5,
|
||||
require_frames=True,
|
||||
cross_l2_span=True,
|
||||
)
|
||||
|
||||
|
||||
class ActionRecognitionStrategy:
|
||||
"""Action Recognition 特化出题策略。
|
||||
|
||||
自包含实现,不依赖 BaseTaskTypeStrategy 或 QuestionFamilySpec。
|
||||
靶向 6 种典型失败机制,通过加权随机 SubPattern 引导 VLM 出题。
|
||||
|
||||
属性:
|
||||
task_type: 题型名 — "Action Recognition"。
|
||||
strategy_name: 策略标识 — "ACTION_RECOGNITION"。
|
||||
skill_target: 目标失败机制 — "M1_AR"。
|
||||
sampling_level: 采样层级 — 2(L2,从 L3 升级以获得跨段视野)。
|
||||
sampling_constraint: 采样约束(min_subtitles=3, min_l3_nodes=5, require_frames, cross_l2_span)。
|
||||
prompt_template: 出题模板 — "action_recognition.md"。
|
||||
leak_probe_template: 泄漏探测模板 — "gate_leak_retrieval.md"。
|
||||
"""
|
||||
|
||||
@property
|
||||
def task_type(self) -> str:
|
||||
"""返回题型名。"""
|
||||
return "Action Recognition"
|
||||
|
||||
@property
|
||||
def strategy_name(self) -> str:
|
||||
"""返回策略标识名。"""
|
||||
return "ACTION_RECOGNITION"
|
||||
|
||||
@property
|
||||
def skill_target(self) -> str:
|
||||
"""返回目标失败机制编号。"""
|
||||
return "M1_AR"
|
||||
|
||||
@property
|
||||
def sampling_level(self) -> int:
|
||||
"""返回采样层级(L2)。"""
|
||||
return 2
|
||||
|
||||
@property
|
||||
def sampling_constraint(self) -> SamplingConstraint:
|
||||
"""返回采样约束。"""
|
||||
return _SAMPLING_CONSTRAINT
|
||||
|
||||
@property
|
||||
def prompt_template(self) -> str:
|
||||
"""返回出题 prompt 模板文件名。"""
|
||||
return "action_recognition.md"
|
||||
|
||||
@property
|
||||
def leak_probe_template(self) -> str:
|
||||
"""返回泄漏探测模板文件名。"""
|
||||
return "gate_leak_retrieval.md"
|
||||
|
||||
@property
|
||||
def uses_grounded_selector(self) -> bool:
|
||||
"""AR 启用候选池 + VLM 视觉打分 selector。"""
|
||||
return True
|
||||
|
||||
def select_sub_pattern(self, rng: random.Random) -> SubPattern:
|
||||
"""按权重随机选择一个 SubPattern。
|
||||
|
||||
参数:
|
||||
rng: 随机数生成器(确保可复现)。
|
||||
|
||||
返回:
|
||||
选中的 SubPattern 实例(永不为 None)。
|
||||
"""
|
||||
chosen_name = rng.choices(
|
||||
_AR_PATTERN_NAMES,
|
||||
weights=_AR_PATTERN_WEIGHTS,
|
||||
k=1,
|
||||
)[0]
|
||||
return _AR_PATTERN_BY_NAME[chosen_name]
|
||||
|
||||
def build_prompt_context(self, material: Any, sub_pattern: SubPattern | None) -> dict:
|
||||
"""构建 prompt 上下文字典。
|
||||
|
||||
参数:
|
||||
material: 采样素材(当前未使用,留给管线扩展)。
|
||||
sub_pattern: 选中的子模式。
|
||||
|
||||
返回:
|
||||
包含 family_name, prompt_template, sub_pattern 的字典。
|
||||
"""
|
||||
return {
|
||||
"family_name": self.strategy_name,
|
||||
"prompt_template": self.prompt_template,
|
||||
"sub_pattern": sub_pattern.name if sub_pattern is not None else None,
|
||||
}
|
||||
|
||||
def extra_gates(self, candidate: Any) -> list:
|
||||
"""返回额外门控列表(当前为空)。
|
||||
|
||||
参数:
|
||||
candidate: 候选题目。
|
||||
|
||||
返回:
|
||||
空列表。
|
||||
"""
|
||||
return []
|
||||
@@ -0,0 +1,756 @@
|
||||
"""赛题合成核心逻辑 — 节点采样、prompt 构造、VLM 响应解析、去重。
|
||||
|
||||
纯函数为主,异步编排仅 generate_one。
|
||||
通过 DI 接收 VLMProvider / EmbeddingProvider,不 import adapters/。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import random
|
||||
from collections.abc import Callable
|
||||
|
||||
from app.tree.index import L2Node, L3Node, TreeIndex
|
||||
from core.protocols import VLMProvider
|
||||
from core.types import GeneratedQuestion
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AnchorContext:
|
||||
"""锚节点上下文——生成单道题所需的全部素材。
|
||||
|
||||
属性:
|
||||
node_id: 锚节点 ID。
|
||||
card_text: 锚节点 card 序列化文本。
|
||||
frame_paths: 帧图片路径列表。
|
||||
subtitle: 对应字幕(可空)。
|
||||
distractor_texts: 同视频其他节点摘要(供 VLM 生成干扰项)。
|
||||
"""
|
||||
|
||||
node_id: str
|
||||
card_text: str
|
||||
frame_paths: list[str]
|
||||
subtitle: str
|
||||
distractor_texts: list[str]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TaskTypeSpec:
|
||||
"""题型的生成规格。
|
||||
|
||||
属性:
|
||||
level: 锚定层级("L3" / "L2" / "L1" / "L1-L2")。
|
||||
needs_frames: 是否必须提供帧图。
|
||||
frame_count: 帧数范围描述(如 "1", "2-3", "0-1")。
|
||||
context_fields: 需要提取的 card 字段元组。
|
||||
"""
|
||||
|
||||
level: str
|
||||
needs_frames: bool
|
||||
frame_count: str
|
||||
context_fields: tuple[str, ...]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 12 种 Video-MME 题型 → 树层级 + 生成规格映射
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TASK_TYPE_LEVEL_MAP: dict[str, TaskTypeSpec] = {
|
||||
# --- L3 单帧题型 ---
|
||||
"Object Recognition": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
|
||||
"Attribute Perception": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
|
||||
"OCR Problems": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
|
||||
"Spatial Reasoning": TaskTypeSpec("L3", True, "1", ("frame_summary", "spatial_layout")),
|
||||
"Spatial Perception": TaskTypeSpec("L3", True, "1", ("frame_summary",)),
|
||||
# --- L2 多帧 / 事件级题型 ---
|
||||
"Action Recognition": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
|
||||
"Action Reasoning": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
|
||||
"Counting Problem": TaskTypeSpec("L2", True, "2-3", ("event_description",)),
|
||||
"Temporal Perception": TaskTypeSpec("L2", False, "0-1", ("event_description", "time_range")),
|
||||
# --- L1 / 跨层级题型 ---
|
||||
"Temporal Reasoning": TaskTypeSpec("L1", True, "per-L2", ("scene_summary",)),
|
||||
"Information Synopsis": TaskTypeSpec("L1", True, "per-L2", ("scene_summary",)),
|
||||
"Object Reasoning": TaskTypeSpec("L1-L2", True, "per-L2", ("event_description",)),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部辅助函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _serialize_l3_card(l3: L3Node, context_fields: tuple[str, ...]) -> str:
|
||||
"""将 L3 节点 card 按 context_fields 序列化为可读文本。
|
||||
|
||||
参数:
|
||||
l3: L3 节点。
|
||||
context_fields: 需提取的字段名元组。
|
||||
|
||||
返回:
|
||||
多行 "field: value" 格式的文本。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for fld in context_fields:
|
||||
val = getattr(l3.card, fld, None)
|
||||
if val is not None:
|
||||
parts.append(f"{fld}: {val}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _l2_time_range_str(l2: L2Node) -> str:
|
||||
"""将 L2 的 time_range 格式化为可读字符串。
|
||||
|
||||
参数:
|
||||
l2: L2 节点。
|
||||
|
||||
返回:
|
||||
"time_range: (start, end)" 格式,或 "time_range: unknown"。
|
||||
"""
|
||||
if l2.time_range is not None:
|
||||
return f"time_range: ({l2.time_range[0]:.2f}, {l2.time_range[1]:.2f})"
|
||||
return "time_range: unknown"
|
||||
|
||||
|
||||
def _representative_frame(l2: L2Node) -> str | None:
|
||||
"""取 L2 的代表帧路径——第一个有 frame_path 的 L3 子节点。
|
||||
|
||||
参数:
|
||||
l2: L2 节点。
|
||||
|
||||
返回:
|
||||
帧路径字符串,或 None(无可用帧时)。
|
||||
"""
|
||||
for l3 in l2.children:
|
||||
if l3.frame_path:
|
||||
return l3.frame_path
|
||||
return None
|
||||
|
||||
|
||||
def _sort_l2_by_time(l2_nodes: list[L2Node]) -> list[L2Node]:
|
||||
"""按 time_range 升序排列 L2 节点(None 排末尾)。
|
||||
|
||||
参数:
|
||||
l2_nodes: 待排序的 L2 节点列表。
|
||||
|
||||
返回:
|
||||
排序后的新列表(不修改原列表)。
|
||||
"""
|
||||
return sorted(
|
||||
l2_nodes,
|
||||
key=lambda n: n.time_range[0] if n.time_range is not None else float("inf"),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 各层级采样策略
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sample_l3(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
spec: TaskTypeSpec,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> AnchorContext:
|
||||
"""L3 层级锚节点采样。
|
||||
|
||||
收集全部 L3 节点,排除已用节点,随机选取一个。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 题型名称。
|
||||
spec: 题型规格。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
AnchorContext 实例。
|
||||
|
||||
异常:
|
||||
ValueError: 候选 L3 节点不足。
|
||||
"""
|
||||
# Phase 1: 收集所有 L3 候选(必须有 frame_path)
|
||||
candidates: list[tuple[L3Node, L2Node]] = []
|
||||
for root in tree.roots:
|
||||
for l2 in root.children:
|
||||
for l3 in l2.children:
|
||||
if l3.id not in used_node_ids and l3.frame_path:
|
||||
candidates.append((l3, l2))
|
||||
|
||||
if not candidates:
|
||||
raise ValueError(f"锚节点不足: {task_type} 无可用 L3 节点(需具备 frame_path)")
|
||||
|
||||
# Phase 2: 随机选取
|
||||
chosen_l3, parent_l2 = rng.choice(candidates)
|
||||
|
||||
# Phase 3: 构造上下文(frame_path 已在候选过滤中保证非 None)
|
||||
card_text = _serialize_l3_card(chosen_l3, spec.context_fields)
|
||||
frame_paths = [chosen_l3.frame_path] # type: ignore[list-item]
|
||||
subtitle = chosen_l3.card.subtitle or ""
|
||||
|
||||
# Phase 4: 干扰项——整棵树中其他 L3 的 frame_summary
|
||||
distractor_texts = [
|
||||
l3.card.frame_summary
|
||||
for root in tree.roots
|
||||
for l2 in root.children
|
||||
for l3 in l2.children
|
||||
if l3.id != chosen_l3.id
|
||||
]
|
||||
|
||||
return AnchorContext(
|
||||
node_id=chosen_l3.id,
|
||||
card_text=card_text,
|
||||
frame_paths=frame_paths,
|
||||
subtitle=subtitle,
|
||||
distractor_texts=distractor_texts,
|
||||
)
|
||||
|
||||
|
||||
def _sample_l2(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
spec: TaskTypeSpec,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> AnchorContext:
|
||||
"""L2 层级锚节点采样(含 Temporal Perception 特殊处理)。
|
||||
|
||||
普通 L2 题型:随机选 1 个 L2,取 2-3 个子 L3 帧。
|
||||
Temporal Perception:0-1 帧,card_text 必含 time_range。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 题型名称。
|
||||
spec: 题型规格。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
AnchorContext 实例。
|
||||
|
||||
异常:
|
||||
ValueError: 候选 L2 节点不足。
|
||||
"""
|
||||
# Phase 1: 收集所有 L2 候选
|
||||
all_l2: list[L2Node] = []
|
||||
for root in tree.roots:
|
||||
for l2 in root.children:
|
||||
if l2.id not in used_node_ids:
|
||||
all_l2.append(l2)
|
||||
|
||||
if not all_l2:
|
||||
raise ValueError(f"锚节点不足: {task_type} 无可用 L2 节点")
|
||||
|
||||
# Phase 2: 随机选取
|
||||
chosen_l2 = rng.choice(all_l2)
|
||||
|
||||
is_temporal_perception = task_type == "Temporal Perception"
|
||||
|
||||
# Phase 3: 帧路径
|
||||
if is_temporal_perception:
|
||||
# 0-1 帧:有子节点则取 1 帧,否则 0 帧
|
||||
frame_paths: list[str] = []
|
||||
if chosen_l2.children:
|
||||
first_frame = chosen_l2.children[0].frame_path
|
||||
if first_frame:
|
||||
frame_paths = [first_frame]
|
||||
else:
|
||||
# 普通 L2:随机采样 2-3 个 L3 帧
|
||||
children_with_frames = [l3 for l3 in chosen_l2.children if l3.frame_path]
|
||||
if len(children_with_frames) < 2:
|
||||
raise ValueError(
|
||||
f"锚节点不足: {task_type} 需要 >=2 个子帧,"
|
||||
f"但 {chosen_l2.id} 仅有 {len(children_with_frames)} 个可用帧"
|
||||
)
|
||||
n_frames = min(rng.randint(2, 3), len(children_with_frames))
|
||||
sampled = rng.sample(children_with_frames, n_frames)
|
||||
frame_paths = [l3.frame_path for l3 in sampled if l3.frame_path]
|
||||
|
||||
# Phase 4: card_text
|
||||
card_text = f"event_description: {chosen_l2.card.event_description}"
|
||||
if is_temporal_perception:
|
||||
card_text += f"\n{_l2_time_range_str(chosen_l2)}"
|
||||
|
||||
# Phase 5: 字幕(优先使用 L2 自身字幕,否则取首个子 L3 字幕)
|
||||
subtitle = chosen_l2.card.subtitle or ""
|
||||
if not subtitle and chosen_l2.children and chosen_l2.children[0].card.subtitle:
|
||||
subtitle = chosen_l2.children[0].card.subtitle
|
||||
|
||||
# Phase 6: 干扰项——整棵树中其他 L2 的 event_description
|
||||
distractor_texts = [
|
||||
l2.card.event_description
|
||||
for root in tree.roots
|
||||
for l2 in root.children
|
||||
if l2.id != chosen_l2.id
|
||||
]
|
||||
|
||||
return AnchorContext(
|
||||
node_id=chosen_l2.id,
|
||||
card_text=card_text,
|
||||
frame_paths=frame_paths,
|
||||
subtitle=subtitle,
|
||||
distractor_texts=distractor_texts,
|
||||
)
|
||||
|
||||
|
||||
def _sample_l1(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
spec: TaskTypeSpec,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> AnchorContext:
|
||||
"""L1 层级锚节点采样(Temporal Reasoning / Information Synopsis)。
|
||||
|
||||
Information Synopsis:使用目标 L1 下全部 L2 子节点。
|
||||
Temporal Reasoning:严格要求 >=3 个 L2 子节点,不足则抛 ValueError。
|
||||
L2 按 time_range 升序排列,每个 L2 取一帧代表。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 题型名称。
|
||||
spec: 题型规格。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
AnchorContext 实例。
|
||||
|
||||
异常:
|
||||
ValueError: 候选 L1 节点不足,或 Temporal Reasoning 的 L2 子节点 <3。
|
||||
"""
|
||||
# Phase 1: 收集可用 L1
|
||||
candidates = [r for r in tree.roots if r.id not in used_node_ids]
|
||||
if not candidates:
|
||||
raise ValueError(f"锚节点不足: {task_type} 无可用 L1 节点")
|
||||
|
||||
# Phase 2: 随机选取
|
||||
chosen_l1 = rng.choice(candidates)
|
||||
|
||||
# Phase 3: 选定 L2 子集
|
||||
if task_type == "Information Synopsis":
|
||||
# 必须使用全部 L2
|
||||
selected_l2 = list(chosen_l1.children)
|
||||
else:
|
||||
# Temporal Reasoning:严格要求 >=3 个 L2
|
||||
if len(chosen_l1.children) < 3:
|
||||
raise ValueError(
|
||||
f"锚节点不足: {task_type} 需要 >=3 个 L2 子节点,"
|
||||
f"但 {chosen_l1.id} 仅有 {len(chosen_l1.children)} 个"
|
||||
)
|
||||
if len(chosen_l1.children) == 3:
|
||||
selected_l2 = list(chosen_l1.children)
|
||||
else:
|
||||
selected_l2 = rng.sample(chosen_l1.children, rng.randint(3, len(chosen_l1.children)))
|
||||
|
||||
# Phase 4: 按 time_range 升序排列
|
||||
selected_l2 = _sort_l2_by_time(selected_l2)
|
||||
|
||||
# Phase 5: card_text(场景摘要)
|
||||
card_text = f"scene_summary: {chosen_l1.card.scene_summary}"
|
||||
|
||||
# Phase 6: 帧路径——每个 L2 取一帧代表
|
||||
frame_paths: list[str] = []
|
||||
for l2 in selected_l2:
|
||||
rep = _representative_frame(l2)
|
||||
if rep:
|
||||
frame_paths.append(rep)
|
||||
|
||||
# Phase 7: 字幕(L1 无字幕)
|
||||
subtitle = ""
|
||||
|
||||
# Phase 8: 干扰项——其他 L1 的 scene_summary
|
||||
distractor_texts = [r.card.scene_summary for r in tree.roots if r.id != chosen_l1.id]
|
||||
|
||||
return AnchorContext(
|
||||
node_id=chosen_l1.id,
|
||||
card_text=card_text,
|
||||
frame_paths=frame_paths,
|
||||
subtitle=subtitle,
|
||||
distractor_texts=distractor_texts,
|
||||
)
|
||||
|
||||
|
||||
def _sample_l1_l2(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
spec: TaskTypeSpec,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> AnchorContext:
|
||||
"""L1-L2 跨层级锚节点采样(Object Reasoning)。
|
||||
|
||||
从全部 L2 中随机选 2-3 个,按 time_range 排序,
|
||||
card_text 为各 L2 的 event_description 拼接。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 题型名称。
|
||||
spec: 题型规格。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
rng: 随机数生成器。
|
||||
|
||||
返回:
|
||||
AnchorContext 实例。
|
||||
|
||||
异常:
|
||||
ValueError: 候选 L2 节点不足。
|
||||
"""
|
||||
# Phase 1: 收集全部 L2
|
||||
all_l2: list[L2Node] = []
|
||||
for root in tree.roots:
|
||||
for l2 in root.children:
|
||||
if l2.id not in used_node_ids:
|
||||
all_l2.append(l2)
|
||||
|
||||
if len(all_l2) < 2:
|
||||
raise ValueError(
|
||||
f"锚节点不足: {task_type} 需要 >=2 个 L2 节点,但仅有 {len(all_l2)} 个可用"
|
||||
)
|
||||
|
||||
# Phase 2: 随机选 2-3 个
|
||||
n_pick = min(rng.randint(2, 3), len(all_l2))
|
||||
selected = rng.sample(all_l2, n_pick)
|
||||
|
||||
# Phase 3: 按 time_range 升序排列
|
||||
selected = _sort_l2_by_time(selected)
|
||||
|
||||
# Phase 4: card_text = 各 L2 event_description 拼接
|
||||
card_text = "\n".join(f"event_description: {l2.card.event_description}" for l2 in selected)
|
||||
|
||||
# Phase 5: 帧路径——每个 L2 取一帧代表
|
||||
frame_paths: list[str] = []
|
||||
for l2 in selected:
|
||||
rep = _representative_frame(l2)
|
||||
if rep:
|
||||
frame_paths.append(rep)
|
||||
|
||||
# Phase 6: 字幕
|
||||
subtitle = ""
|
||||
|
||||
# Phase 7: 干扰项——未被选中的 L2 的 event_description
|
||||
selected_ids = {l2.id for l2 in selected}
|
||||
distractor_texts = [l2.card.event_description for l2 in all_l2 if l2.id not in selected_ids]
|
||||
|
||||
# 使用第一个被选中节点的 ID 作为锚节点 ID
|
||||
anchor_id = selected[0].id
|
||||
|
||||
return AnchorContext(
|
||||
node_id=anchor_id,
|
||||
card_text=card_text,
|
||||
frame_paths=frame_paths,
|
||||
subtitle=subtitle,
|
||||
distractor_texts=distractor_texts,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 公开接口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sample_anchor(
|
||||
tree: TreeIndex,
|
||||
task_type: str,
|
||||
used_node_ids: set[str],
|
||||
rng: random.Random,
|
||||
) -> AnchorContext:
|
||||
"""根据题型从视频树中采样锚节点及上下文素材。
|
||||
|
||||
依据 TASK_TYPE_LEVEL_MAP 中的层级规格,分发到对应的层级采样策略。
|
||||
每种层级有不同的帧选取、card 序列化和干扰项收集逻辑。
|
||||
|
||||
参数:
|
||||
tree: 三层树索引。
|
||||
task_type: 12 种 Video-MME 题型之一。
|
||||
used_node_ids: 本轮已用节点 ID 集合(避免重复采样)。
|
||||
rng: 可控随机数生成器(保证可复现)。
|
||||
|
||||
返回:
|
||||
AnchorContext 实例,包含锚节点 ID、card 文本、帧路径、字幕和干扰项。
|
||||
|
||||
异常:
|
||||
KeyError: task_type 不在 TASK_TYPE_LEVEL_MAP 中。
|
||||
ValueError: 候选节点不足(全部被 used_node_ids 排除)。
|
||||
"""
|
||||
spec = TASK_TYPE_LEVEL_MAP[task_type]
|
||||
|
||||
if spec.level == "L3":
|
||||
return _sample_l3(tree, task_type, spec, used_node_ids, rng)
|
||||
elif spec.level == "L2":
|
||||
return _sample_l2(tree, task_type, spec, used_node_ids, rng)
|
||||
elif spec.level == "L1":
|
||||
return _sample_l1(tree, task_type, spec, used_node_ids, rng)
|
||||
elif spec.level == "L1-L2":
|
||||
return _sample_l1_l2(tree, task_type, spec, used_node_ids, rng)
|
||||
else:
|
||||
raise ValueError(f"未知层级: {spec.level}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt 构造与 VLM 响应解析
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_VALID_ANSWERS = frozenset({"A", "B", "C", "D"})
|
||||
|
||||
|
||||
def build_generation_prompt(
|
||||
task_type: str,
|
||||
anchor: AnchorContext,
|
||||
exemplars: list[GeneratedQuestion],
|
||||
) -> tuple[list[dict[str, str]], list[str]]:
|
||||
"""组装 VLM 出题 prompt。
|
||||
|
||||
构造 OpenAI 格式的 messages 列表和帧图片路径列表,
|
||||
供 VLMProvider.chat_with_images 直接消费。
|
||||
|
||||
参数:
|
||||
task_type: 题型名称(如 "Object Recognition")。
|
||||
anchor: 锚节点上下文(card_text, subtitle, distractor_texts, frame_paths)。
|
||||
exemplars: 少样本示例列表(可为空)。
|
||||
|
||||
返回:
|
||||
(messages, image_paths) — messages 为 OpenAI 格式消息列表,
|
||||
image_paths 为帧图片路径列表,直接喂给 VLMProvider.chat_with_images。
|
||||
"""
|
||||
# Phase 1: 构造 system message
|
||||
system_parts: list[str] = [
|
||||
"你是一个视频理解题目生成器。",
|
||||
f"题型: {task_type}",
|
||||
"约束:",
|
||||
"- 题目必须基于提供的节点内容",
|
||||
"- 干扰选项应来自其他节点的信息",
|
||||
"- 生成风格应与示例保持一致",
|
||||
'- 以 JSON 格式返回: {"question": "...", "options": ["A. ...", "B. ...", "C. ...", "D. ..."], "answer": "A/B/C/D"}',
|
||||
]
|
||||
|
||||
# Phase 2: 加入 few-shot 示例
|
||||
if exemplars:
|
||||
system_parts.append("\n示例:")
|
||||
for i, ex in enumerate(exemplars, 1):
|
||||
system_parts.append(f" 示例 {i}:")
|
||||
system_parts.append(f" question: {ex.question}")
|
||||
system_parts.append(f" options: {list(ex.options)}")
|
||||
system_parts.append(f" answer: {ex.answer}")
|
||||
|
||||
system_content = "\n".join(system_parts)
|
||||
|
||||
# Phase 3: 构造 user message
|
||||
user_parts: list[str] = [f"节点内容:\n{anchor.card_text}"]
|
||||
|
||||
if anchor.subtitle:
|
||||
user_parts.append(f"\n字幕:\n{anchor.subtitle}")
|
||||
|
||||
if anchor.distractor_texts:
|
||||
user_parts.append("\n干扰项来源节点摘要:")
|
||||
for dt in anchor.distractor_texts:
|
||||
user_parts.append(f"- {dt}")
|
||||
|
||||
user_content = "\n".join(user_parts)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system_content},
|
||||
{"role": "user", "content": user_content},
|
||||
]
|
||||
|
||||
return messages, list(anchor.frame_paths)
|
||||
|
||||
|
||||
def parse_vlm_response(
|
||||
raw: str,
|
||||
video_id: str,
|
||||
task_type: str,
|
||||
seq: int,
|
||||
) -> dict:
|
||||
"""解析 VLM 返回的 JSON → 部分字段字典。
|
||||
|
||||
尝试直接解析 JSON;若失败,从 markdown 代码块中提取后重试。
|
||||
校验必需字段、选项数量和答案合法性。
|
||||
|
||||
参数:
|
||||
raw: VLM 原始返回文本。
|
||||
video_id: 所属视频标识。
|
||||
task_type: 题型名称(用于错误消息)。
|
||||
seq: 序列号,用于生成 question_id。
|
||||
|
||||
返回:
|
||||
{"question_id": "gen-{video_id}-{seq:03d}", "question": ..., "options": [...], "answer": ...}
|
||||
调用方(generate_one)补齐 source_nodes/difficulty 后构造 GeneratedQuestion。
|
||||
|
||||
异常:
|
||||
ValueError: JSON 解析失败、缺必需字段、options 非 4 项、answer 不在 A-D。
|
||||
"""
|
||||
# Phase 1: 尝试直接解析 JSON
|
||||
data = None
|
||||
with contextlib.suppress(json.JSONDecodeError):
|
||||
data = json.loads(raw)
|
||||
|
||||
# Phase 2: 从 markdown 代码块提取 JSON
|
||||
if data is None:
|
||||
match = re.search(r"```(?:json)?\s*\n?(.*?)\n?\s*```", raw, re.DOTALL)
|
||||
if match:
|
||||
with contextlib.suppress(json.JSONDecodeError):
|
||||
data = json.loads(match.group(1))
|
||||
|
||||
if data is None:
|
||||
raise ValueError(f"VLM 返回无法解析为 JSON: {raw[:200]}")
|
||||
|
||||
# Phase 3: 校验必需字段
|
||||
required = ("question", "options", "answer")
|
||||
missing = [f for f in required if f not in data]
|
||||
if missing:
|
||||
raise ValueError(f"VLM 返回缺少必需字段 {missing}: {raw[:200]}")
|
||||
|
||||
# Phase 4: options 必须恰好 4 项
|
||||
options = data["options"]
|
||||
if not isinstance(options, list) or len(options) != 4:
|
||||
raise ValueError(
|
||||
f"options 必须恰好 4 项,实际 {len(options) if isinstance(options, list) else type(options).__name__}: {raw[:200]}"
|
||||
)
|
||||
|
||||
# Phase 5: answer 必须是 A-D
|
||||
answer = data["answer"]
|
||||
if answer not in _VALID_ANSWERS:
|
||||
raise ValueError(f"answer 必须是 A/B/C/D 之一,实际 '{answer}': {raw[:200]}")
|
||||
|
||||
return {
|
||||
"question_id": f"gen-{video_id}-{task_type.lower().replace(' ', '_')}-{seq:03d}",
|
||||
"question": data["question"],
|
||||
"options": list(options),
|
||||
"answer": answer,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Embedding 去重
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def is_duplicate(
|
||||
question_text: str,
|
||||
pool_embeddings: np.ndarray,
|
||||
embed_fn: Callable[[str | list[str]], np.ndarray],
|
||||
threshold: float,
|
||||
) -> bool:
|
||||
"""embedding 去重判定。
|
||||
|
||||
参数:
|
||||
question_text: 待检查的题目文本。
|
||||
pool_embeddings: 已有题目的 embedding 矩阵 [N, D](L2 归一化)。
|
||||
embed_fn: 文本嵌入函数,返回 [N, D] ndarray(L2 归一化)。
|
||||
threshold: 余弦相似度阈值。
|
||||
|
||||
返回:
|
||||
True 表示与池中某题重复。空池永远返回 False。
|
||||
"""
|
||||
if pool_embeddings.shape[0] == 0:
|
||||
return False
|
||||
|
||||
query = embed_fn(question_text) # [1, D]
|
||||
query = query.squeeze(0) # [D]
|
||||
similarities = pool_embeddings @ query # [N]
|
||||
return bool(np.max(similarities) >= threshold)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 单题生成
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def generate_one(
|
||||
vlm: VLMProvider,
|
||||
tree: TreeIndex,
|
||||
video_id: str,
|
||||
task_type: str,
|
||||
seq: int,
|
||||
*,
|
||||
exemplars: list[GeneratedQuestion],
|
||||
used_node_ids: set[str],
|
||||
max_retries: int,
|
||||
rng: random.Random,
|
||||
session_id: str,
|
||||
) -> GeneratedQuestion | None:
|
||||
"""生成单道候选题(不含去重——去重在调用方汇总点原子执行)。
|
||||
|
||||
循环最多 max_retries 次尝试生成。每次尝试:
|
||||
1. 采样锚节点
|
||||
2. 构造 prompt
|
||||
3. 调用 VLM
|
||||
4. 解析响应
|
||||
5. 构造 GeneratedQuestion
|
||||
|
||||
返回 None 表示耗尽重试。
|
||||
|
||||
参数:
|
||||
vlm: VLM 调用端口。
|
||||
tree: 三层树索引。
|
||||
video_id: 所属视频标识。
|
||||
task_type: 12 种 Video-MME 题型之一。
|
||||
seq: 序列号,用于生成 question_id。
|
||||
exemplars: 少样本示例列表。
|
||||
used_node_ids: 已用节点 ID 集合。
|
||||
max_retries: 最大重试次数。
|
||||
rng: 可控随机数生成器。
|
||||
session_id: 会话 ID(传递给 VLM 遥测)。
|
||||
|
||||
返回:
|
||||
GeneratedQuestion 实例,或 None(耗尽重试)。
|
||||
"""
|
||||
from core.types import GeneratedQuestion as _GeneratedQuestion
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
# Phase 1: 采样锚节点
|
||||
anchor = sample_anchor(tree, task_type, used_node_ids, rng)
|
||||
|
||||
# Phase 2: 构造 prompt
|
||||
messages, images = build_generation_prompt(task_type, anchor, exemplars)
|
||||
|
||||
# Phase 3: 调用 VLM
|
||||
response = await vlm.chat_with_images(
|
||||
messages,
|
||||
images,
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
# Phase 4: 解析响应
|
||||
parsed = parse_vlm_response(response.content, video_id, task_type, seq)
|
||||
|
||||
# Phase 5: 构造 GeneratedQuestion
|
||||
return _GeneratedQuestion(
|
||||
question_id=parsed["question_id"],
|
||||
video_id=video_id,
|
||||
task_type=task_type,
|
||||
question=parsed["question"],
|
||||
options=tuple(parsed["options"]),
|
||||
answer=parsed["answer"],
|
||||
source_nodes=(anchor.node_id,),
|
||||
difficulty="medium",
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"generate_one 尝试 {}/{} 失败 ({}): {}",
|
||||
attempt + 1,
|
||||
max_retries,
|
||||
task_type,
|
||||
exc,
|
||||
)
|
||||
continue
|
||||
|
||||
logger.warning(
|
||||
"generate_one 耗尽 {} 次重试 (video={}, task_type={})",
|
||||
max_retries,
|
||||
video_id,
|
||||
task_type,
|
||||
)
|
||||
return None
|
||||
@@ -0,0 +1,13 @@
|
||||
"""搜索 Agent 装配层 — prompt 管理、skill 注册、工具分发、LLM 摘要、视觉观察。"""
|
||||
|
||||
from app.search.prompt import PromptManager
|
||||
from app.search.skills import SkillRegistry, discover_skills
|
||||
from app.search.tools import SearchToolDispatcher, get_tool_descriptions
|
||||
|
||||
__all__ = [
|
||||
"PromptManager",
|
||||
"SkillRegistry",
|
||||
"SearchToolDispatcher",
|
||||
"discover_skills",
|
||||
"get_tool_descriptions",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""搜索 Agent 提示词管理模块。
|
||||
|
||||
提供 PromptManager 类,统一管理循环级 prompt 的加载与组装。
|
||||
工具级 prompt(extract/verify)不在管理范围内。
|
||||
|
||||
与 TRM4 ``core/search/prompt.py`` 的差异:
|
||||
- 工具描述从 ``app.search.tools.get_tool_descriptions`` 获取(路径变更);
|
||||
- ``format_user_prompt`` 参数显式化(question/options/l1_node_ids/task_type)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from app.search.tools import get_tool_descriptions
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class PromptManager:
|
||||
"""管理循环级 prompt 的加载与组装。
|
||||
|
||||
构造时缓存 system.md 作为 inference 基础模板。
|
||||
后续步骤(diagnose/evolve/question_gen)通过 load() 按文件名读取。
|
||||
|
||||
参数:
|
||||
prompts_dir: prompt 文件目录的绝对路径。
|
||||
"""
|
||||
|
||||
def __init__(self, prompts_dir: Path) -> None:
|
||||
self._prompts_dir = prompts_dir
|
||||
system_path = prompts_dir / "system.md"
|
||||
if not system_path.exists():
|
||||
raise FileNotFoundError(f"system.md 不存在: {system_path}")
|
||||
self._system_base = system_path.read_text(encoding="utf-8")
|
||||
|
||||
def build_inference_prompt(
|
||||
self,
|
||||
skill_mode: str,
|
||||
task_type: str,
|
||||
always_skills_text: str,
|
||||
task_skill_map: dict[str, str],
|
||||
catalog_text: str,
|
||||
) -> str:
|
||||
"""组装 inference 步骤的完整 system prompt。
|
||||
|
||||
参数:
|
||||
skill_mode: "auto" / "manual" / "none"。
|
||||
task_type: 当前 QA 的题型。
|
||||
always_skills_text: always 层 skill 正文(已拼接)。
|
||||
task_skill_map: {task_type: skill_body} 映射。
|
||||
catalog_text: manual 模式的 skill 目录文本。
|
||||
|
||||
返回:
|
||||
拼装后的完整 system prompt。
|
||||
"""
|
||||
include_read_skill = skill_mode == "manual"
|
||||
parts = [
|
||||
self._system_base,
|
||||
f"\n\n---\n\n{get_tool_descriptions(include_read_skill=include_read_skill)}",
|
||||
]
|
||||
if always_skills_text:
|
||||
parts.append(f"\n\n---\n\n# 通用搜索策略\n\n{always_skills_text}")
|
||||
if skill_mode == "auto":
|
||||
skill_text = task_skill_map.get(task_type) or task_skill_map.get("_default")
|
||||
if skill_text:
|
||||
parts.append(f"\n\n---\n\n# 当前题型搜索策略\n\n{skill_text}")
|
||||
elif skill_mode == "manual":
|
||||
if catalog_text:
|
||||
parts.append(
|
||||
"\n\n---\n\n# 可用搜索策略\n\n"
|
||||
"以下技能扩展了你的导航能力。当问题匹配某技能的适用题型时,"
|
||||
"用 read_skill 工具加载该技能,然后按其指引操作。\n\n"
|
||||
f"{catalog_text}"
|
||||
)
|
||||
return "".join(parts)
|
||||
|
||||
def format_user_prompt(
|
||||
self,
|
||||
question: str,
|
||||
options: list[str],
|
||||
l1_node_ids: list[str],
|
||||
task_type: str | None = None,
|
||||
) -> str:
|
||||
"""格式化 inference 步骤的用户提示词。
|
||||
|
||||
参数:
|
||||
question: 问题文本。
|
||||
options: 选项列表(如 ["A. 历史", "B. 科学"])。
|
||||
l1_node_ids: L1 根节点 ID 列表(如 ["L1_000", "L1_001"])。
|
||||
task_type: 可选题型标签,非 None 时插入题型行(oracle 实验用)。
|
||||
|
||||
返回:
|
||||
格式化后的用户提示词。
|
||||
"""
|
||||
options_text = "\n".join(options)
|
||||
roots_text = ", ".join(l1_node_ids)
|
||||
task_type_line = f"**题型**: {task_type}\n" if task_type else ""
|
||||
return (
|
||||
f"请回答以下关于这个视频的多选题:\n\n"
|
||||
f"{task_type_line}"
|
||||
f"**问题**: {question}\n"
|
||||
f"**选项**:\n{options_text}\n\n"
|
||||
f"**视频树 L1 根节点**: {roots_text}\n"
|
||||
f"请从以上 L1 节点开始导航,收集证据后回答。"
|
||||
)
|
||||
|
||||
def load(self, name: str) -> str:
|
||||
"""按文件名加载 prompt 内容。
|
||||
|
||||
参数:
|
||||
name: prompt 文件名(如 "diagnose_span.md")。
|
||||
|
||||
返回:
|
||||
文件内容字符串。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 文件不存在。
|
||||
"""
|
||||
path = self._prompts_dir / name
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"prompt 文件不存在: {path}")
|
||||
return path.read_text(encoding="utf-8")
|
||||
@@ -0,0 +1,195 @@
|
||||
"""技能注册表与 Markdown frontmatter 解析工具。
|
||||
|
||||
提供 Skill 文件的 frontmatter 解析、正文提取、注册表管理和目录扫描功能,
|
||||
供搜索 Agent 装配层使用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
_FRONTMATTER_FIELDS = {"name", "description", "always", "task_type"}
|
||||
|
||||
|
||||
def _extract_frontmatter_lines(text: str) -> tuple[list[str], int] | None:
|
||||
"""提取 frontmatter 行与正文起始偏移。
|
||||
|
||||
参数:
|
||||
text: 原始 Markdown 文本。
|
||||
|
||||
返回:
|
||||
(frontmatter 行列表, 正文起始字节偏移) 二元组;
|
||||
若不存在完整 frontmatter 则返回 None。
|
||||
"""
|
||||
lines = text.splitlines(keepends=True)
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return None
|
||||
|
||||
offset = len(lines[0])
|
||||
frontmatter_lines: list[str] = []
|
||||
for line in lines[1:]:
|
||||
if line.strip() == "---":
|
||||
return frontmatter_lines, offset + len(line)
|
||||
frontmatter_lines.append(line)
|
||||
offset += len(line)
|
||||
|
||||
logger.debug("frontmatter 缺少结束分隔符,按普通正文处理")
|
||||
return None
|
||||
|
||||
|
||||
def strip_frontmatter(text: str) -> str:
|
||||
"""移除 Markdown 文本开头的 frontmatter,并返回正文。
|
||||
|
||||
参数:
|
||||
text: 原始 Markdown 文本。
|
||||
|
||||
返回:
|
||||
去除 frontmatter 后的正文;若 frontmatter 不完整或不存在,则返回原文。
|
||||
"""
|
||||
extracted = _extract_frontmatter_lines(text)
|
||||
if extracted is None:
|
||||
return text
|
||||
|
||||
_, body_start = extracted
|
||||
return text[body_start:]
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> dict[str, str]:
|
||||
"""解析 Markdown frontmatter 中的目标字段。
|
||||
|
||||
仅识别 ``name``、``description``、``always``、``task_type`` 四个字段,
|
||||
其余字段会被忽略。引号包裹的值会自动去除引号。
|
||||
|
||||
参数:
|
||||
text: 原始 Markdown 文本。
|
||||
|
||||
返回:
|
||||
仅包含目标字段的字符串字典。
|
||||
若不存在完整 frontmatter,则返回空字典。
|
||||
"""
|
||||
extracted = _extract_frontmatter_lines(text)
|
||||
if extracted is None:
|
||||
return {}
|
||||
|
||||
frontmatter_lines, _ = extracted
|
||||
parsed: dict[str, str] = {}
|
||||
for raw_line in frontmatter_lines:
|
||||
line = raw_line.strip()
|
||||
if not line or ":" not in line:
|
||||
continue
|
||||
|
||||
key, _, raw_value = line.partition(":")
|
||||
normalized_key = key.strip()
|
||||
if normalized_key not in _FRONTMATTER_FIELDS:
|
||||
continue
|
||||
|
||||
value = raw_value.strip()
|
||||
if len(value) >= 2 and (
|
||||
(value.startswith('"') and value.endswith('"'))
|
||||
or (value.startswith("'") and value.endswith("'"))
|
||||
):
|
||||
value = value[1:-1]
|
||||
parsed[normalized_key] = value
|
||||
|
||||
return parsed
|
||||
|
||||
|
||||
class SkillRegistry:
|
||||
"""管理技能名称到文件路径映射并读取技能正文。
|
||||
|
||||
通过 ``set_paths`` 注入名称→路径映射后,
|
||||
可用 ``read`` 按名读取技能 Markdown 正文(自动去除 frontmatter)。
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._paths: dict[str, Path] = {}
|
||||
|
||||
def set_paths(self, mapping: dict[str, Path]) -> None:
|
||||
"""注入技能名称到文件路径的映射。
|
||||
|
||||
参数:
|
||||
mapping: 技能名到 Markdown 文件路径的映射。
|
||||
"""
|
||||
self._paths = dict(mapping)
|
||||
logger.debug("SkillRegistry 已载入 {} 个技能路径", len(self._paths))
|
||||
|
||||
def read(self, name: str) -> str:
|
||||
"""读取指定技能文件,并返回去除 frontmatter 后的正文。
|
||||
|
||||
参数:
|
||||
name: 技能名称。
|
||||
|
||||
返回:
|
||||
技能 Markdown 正文。
|
||||
|
||||
异常:
|
||||
KeyError: 技能名称未注册时抛出。
|
||||
"""
|
||||
try:
|
||||
path = self._paths[name]
|
||||
except KeyError:
|
||||
logger.error("技能未注册: {}", name)
|
||||
raise
|
||||
|
||||
logger.debug("读取技能文件: name={}, path={}", name, path)
|
||||
return strip_frontmatter(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def discover_skills(
|
||||
skills_dir: Path,
|
||||
) -> tuple[str, dict[str, str], str, SkillRegistry]:
|
||||
"""扫描 skills 目录,按 frontmatter 分类返回。
|
||||
|
||||
遍历 ``*.md`` 文件,根据 frontmatter 的 ``always`` / ``task_type`` 字段分类:
|
||||
|
||||
- ``always=true`` 的 skill 拼入 ``always_skills_text``
|
||||
- 有 ``task_type`` 的 skill 加入 ``task_skill_map``
|
||||
- 非 always 的 skill 生成 ``catalog_text`` 并注册到 registry
|
||||
|
||||
参数:
|
||||
skills_dir: Skill 文件目录。
|
||||
|
||||
返回:
|
||||
``(always_skills_text, task_skill_map, catalog_text, registry)`` 四元组。
|
||||
"""
|
||||
if not skills_dir.exists():
|
||||
return "", {}, "", SkillRegistry()
|
||||
|
||||
always_parts: list[str] = []
|
||||
task_skill_map: dict[str, str] = {}
|
||||
catalog_lines: list[str] = []
|
||||
registry_paths: dict[str, Path] = {}
|
||||
|
||||
for path in sorted(skills_dir.glob("*.md")):
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
meta = parse_frontmatter(raw)
|
||||
if "name" not in meta:
|
||||
logger.warning("跳过无 name 的 skill 文件: {}", path)
|
||||
continue
|
||||
|
||||
body = strip_frontmatter(raw)
|
||||
name = meta["name"]
|
||||
desc = meta.get("description", "")
|
||||
task_type = meta.get("task_type", "")
|
||||
is_always = str(meta.get("always", "false")).lower() == "true"
|
||||
|
||||
if is_always:
|
||||
always_parts.append(body)
|
||||
else:
|
||||
if task_type:
|
||||
task_skill_map[task_type] = body
|
||||
catalog_lines.append(f"- **{name}**: {desc}")
|
||||
registry_paths[name] = path
|
||||
|
||||
always_text = "\n\n---\n\n".join(always_parts)
|
||||
catalog_text = "\n".join(catalog_lines)
|
||||
|
||||
registry = SkillRegistry()
|
||||
registry.set_paths(registry_paths)
|
||||
|
||||
return always_text, task_skill_map, catalog_text, registry
|
||||
@@ -0,0 +1,487 @@
|
||||
"""节点内容摘要模块 — 两轮 LLM 调用生成 question-conditioned 摘要。
|
||||
|
||||
提取轮:带防幻觉 system prompt,提取与问题相关的信息。
|
||||
验证轮:带核实 system prompt,逐条核实并给置信度。
|
||||
与 TRM4 core/tree/summarizer.py 保真迁移:
|
||||
同步 → async、_call_llm → await llm.chat()、ThreadPoolExecutor → asyncio.gather。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from core.protocols import LLMProvider
|
||||
|
||||
# ── 正则常量 ──────────────────────────────────────────────────────────
|
||||
|
||||
# 行号引注组:括号包裹的 s/c 行号列表,如 (s1) / (c2,s5) / (c70-c73,s196-s200)
|
||||
# (兼容全角括号与逗号;单元允许范围语法 s3-s5 / s3-5,60-span 实测模型常用)
|
||||
_ANCHOR_GROUP = re.compile(
|
||||
r"[((]\s*([sc]\d+(?:-[sc]?\d+)?(?:\s*[,,]\s*[sc]\d+(?:-[sc]?\d+)?)*)\s*[))]"
|
||||
)
|
||||
_ANCHOR_RANGE = re.compile(r"([sc])(\d+)-([sc]?)(\d+)")
|
||||
_RELEVANT_SECTION = re.compile(r"\[相关信息\](.*?)(?=\n\[|\Z)", re.DOTALL)
|
||||
# 无相关信息声明句:60-span 实测全为"该节点未包含与问题直接相关的信息"类变体
|
||||
_NO_INFO_STATEMENT = re.compile(r"未包含.*相关.*信息")
|
||||
|
||||
# 范围展开条数上限:防 (s1-s9999) 这类爆炸展开
|
||||
_RANGE_MAX_IDS = 50
|
||||
|
||||
# 双封顶参数:上轮 A/B 证明无上限引用膨胀至 8.4 条/span 挤占提取预算(hall +51%)
|
||||
_EXPAND_MAX_ITEMS = 5
|
||||
_EXPAND_MAX_CHARS = 800
|
||||
_EXPAND_LINE_CAP = 200
|
||||
|
||||
|
||||
# ── Prompt 加载 ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _load_prompt(prompts_dir: Path, filename: str) -> str:
|
||||
"""从 prompts 目录加载 system prompt 文件。
|
||||
|
||||
参数:
|
||||
prompts_dir: prompt 文件所在目录。
|
||||
filename: prompt 文件名。
|
||||
|
||||
返回:
|
||||
文件内容字符串。
|
||||
"""
|
||||
return (prompts_dir / filename).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
# ── Anchor 工具函数 ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _expand_anchor_ids(group_text: str) -> list[str]:
|
||||
"""把引注组文本展开为逐 id 列表(支持范围语法)。
|
||||
|
||||
参数:
|
||||
group_text: _ANCHOR_GROUP 捕获的组内文本,如 "s3-s5, c1"。
|
||||
|
||||
返回:
|
||||
逐 id 列表。合法范围(同前缀、起点<=终点、展开条数<=50)展开为
|
||||
逐 id("s3-s5"/"s3-5" -> s3,s4,s5);非法范围(跨前缀如 c3-s5、
|
||||
起点>终点、展开条数超限防爆炸)保留原 token——后续查表必然失配,
|
||||
整段按 1 个非法锚计罚剔除。
|
||||
"""
|
||||
ids: list[str] = []
|
||||
for token in re.split(r"[,,]\s*", group_text):
|
||||
token = token.strip()
|
||||
m = _ANCHOR_RANGE.fullmatch(token)
|
||||
if m is None:
|
||||
ids.append(token)
|
||||
continue
|
||||
prefix, start = m.group(1), int(m.group(2))
|
||||
end_prefix, end = m.group(3), int(m.group(4))
|
||||
legal_range = (
|
||||
(not end_prefix or end_prefix == prefix)
|
||||
and start <= end
|
||||
and end - start + 1 <= _RANGE_MAX_IDS
|
||||
)
|
||||
if not legal_range:
|
||||
ids.append(token)
|
||||
continue
|
||||
ids.extend(f"{prefix}{i}" for i in range(start, end + 1))
|
||||
return ids
|
||||
|
||||
|
||||
def check_anchors(summary: str, anchor_map: dict[str, str]) -> tuple[str, dict[str, int]]:
|
||||
"""校验行号引注:非法行号删锚不删断言。
|
||||
|
||||
参数:
|
||||
summary: 提取轮输出(含行号引注)。
|
||||
anchor_map: {锚: 原文行} 查表。
|
||||
|
||||
返回:
|
||||
(清理后文本, {"n_assertions", "n_anchored", "n_illegal"})。
|
||||
|
||||
关键实现细节:
|
||||
清洗全文、统计限段:非法锚无论出现在哪一段都删除并计入 n_illegal
|
||||
(避免未校验段落的编造锚流入装配展开);断言统计
|
||||
(n_assertions/n_anchored)仅数 [相关信息] 段内非空内容行。
|
||||
引注组先经 _expand_anchor_ids 把范围语法展开为逐 id 再逐 id 校验
|
||||
(合法子集重写为逐 id 列表如 (s3,s4,s5)),组内全非法则整组删除;
|
||||
组外文本一律不动(删锚不删断言)。分母口径:匹配"未包含...相关...
|
||||
信息"词面的声明句不计入 n_assertions——它们天然无锚,计入会虚压
|
||||
遵从率。
|
||||
"""
|
||||
stats: dict[str, int] = {"n_assertions": 0, "n_anchored": 0, "n_illegal": 0}
|
||||
|
||||
def _clean_group(gm: re.Match) -> str:
|
||||
ids = _expand_anchor_ids(gm.group(1))
|
||||
legal = [i for i in ids if i in anchor_map]
|
||||
stats["n_illegal"] += len(ids) - len(legal)
|
||||
return f"({','.join(legal)})" if legal else ""
|
||||
|
||||
cleaned = _ANCHOR_GROUP.sub(_clean_group, summary)
|
||||
m = _RELEVANT_SECTION.search(cleaned)
|
||||
if m is None:
|
||||
return cleaned, stats
|
||||
for line in m.group(1).splitlines():
|
||||
line = line.strip().lstrip("-•*").strip()
|
||||
if not line:
|
||||
continue
|
||||
if _NO_INFO_STATEMENT.search(line):
|
||||
continue
|
||||
stats["n_assertions"] += 1
|
||||
if _ANCHOR_GROUP.search(line):
|
||||
stats["n_anchored"] += 1
|
||||
return cleaned, stats
|
||||
|
||||
|
||||
def _cited_anchor_ids(summary: str, anchor_map: dict[str, str]) -> list[str]:
|
||||
"""按引注首次出现顺序收集合法锚 id(去重)。
|
||||
|
||||
参数:
|
||||
summary: 含行号引注的文本。
|
||||
anchor_map: {锚: 原文行} 查表。
|
||||
|
||||
返回:
|
||||
去重后的合法锚 id 列表(保持首次出现顺序)。
|
||||
|
||||
关键实现细节:
|
||||
从 assemble_anchored_output 提取以满足圈复杂度门槛;范围语法经
|
||||
_expand_anchor_ids 展开后逐 id 收集;只收合法锚(非法锚已由
|
||||
check_anchors 清除,此处过滤是防御性双保险)。
|
||||
"""
|
||||
ordered: list[str] = []
|
||||
for gm in _ANCHOR_GROUP.finditer(summary):
|
||||
for aid in _expand_anchor_ids(gm.group(1)):
|
||||
if aid in anchor_map and aid not in ordered:
|
||||
ordered.append(aid)
|
||||
return ordered
|
||||
|
||||
|
||||
def assemble_anchored_output(
|
||||
summary: str, anchor_map: dict[str, str], mode: str
|
||||
) -> tuple[str, dict[str, int]]:
|
||||
"""按装配形态生成最终输出:展开引文并施加双封顶。
|
||||
|
||||
参数:
|
||||
summary: check_anchors 清理后的文本。
|
||||
anchor_map: {锚: 原文行}。
|
||||
mode: "ids"(裸行号)| "ids_expand"(行号+展开)| "expand_only"(展开剥行号)。
|
||||
|
||||
返回:
|
||||
(最终文本, {"n_expanded", "n_trunc"})。
|
||||
|
||||
关键实现细节:
|
||||
展开按引注首次出现顺序取前 5 条;总额帽按 [引文] 条目完整长度
|
||||
(含前缀与引号)记账,<=800 字符;单行原文超 200 字符先截断。
|
||||
n_expanded/n_trunc 仅计实际输出的条目。expand_only 先对正文剥除
|
||||
全部引注 token、再拼接 [引文] 段(judge 探针判定 id token 被计罚
|
||||
时的回退形态)——引文行不经过剥离,原文行中的括号文本得以保留。
|
||||
"""
|
||||
assert mode in ("ids", "ids_expand", "expand_only"), f"未知装配形态: {mode}"
|
||||
stats: dict[str, int] = {"n_expanded": 0, "n_trunc": 0}
|
||||
if mode != "ids":
|
||||
ordered = _cited_anchor_ids(summary, anchor_map)
|
||||
expansions: list[str] = []
|
||||
total = 0
|
||||
for aid in ordered[:_EXPAND_MAX_ITEMS]:
|
||||
line = anchor_map[aid]
|
||||
truncated = len(line) > _EXPAND_LINE_CAP
|
||||
if truncated:
|
||||
line = line[:_EXPAND_LINE_CAP] + "…"
|
||||
entry = f' ▸ {aid}: "{line}"'
|
||||
if total + len(entry) > _EXPAND_MAX_CHARS:
|
||||
break
|
||||
total += len(entry)
|
||||
expansions.append(entry)
|
||||
stats["n_expanded"] += 1
|
||||
if truncated:
|
||||
stats["n_trunc"] += 1
|
||||
if mode == "expand_only":
|
||||
summary = _ANCHOR_GROUP.sub("", summary)
|
||||
if expansions:
|
||||
summary = summary + "\n[引文]\n" + "\n".join(expansions)
|
||||
return summary, stats
|
||||
|
||||
|
||||
# ── LLM 调用辅助 ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _call_llm(
|
||||
llm: LLMProvider,
|
||||
system_prompt: str,
|
||||
user_text: str,
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> str:
|
||||
"""调用 LLM 并返回响应文本。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 端口实例。
|
||||
system_prompt: 系统提示词。
|
||||
user_text: 用户消息文本。
|
||||
session_id: 会话 ID(透传遥测)。
|
||||
parent_call_id: 父调用 ID(透传遥测)。
|
||||
|
||||
返回:
|
||||
模型回答文本。
|
||||
"""
|
||||
messages = [
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_text},
|
||||
]
|
||||
response = await llm.chat(messages, session_id=session_id, parent_call_id=parent_call_id)
|
||||
return response.content
|
||||
|
||||
|
||||
# ── 摘要函数 ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def summarize_node(
|
||||
llm: LLMProvider,
|
||||
raw_text: str,
|
||||
question: str,
|
||||
prompts_dir: Path,
|
||||
*,
|
||||
anchor_map: dict[str, str] | None,
|
||||
assemble_mode: str,
|
||||
stats_sink: Callable[[dict[str, Any]], None] | None = None,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> str:
|
||||
"""对单个节点做 question-conditioned 两轮摘要(可选行号锚模式)。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 端口实例。
|
||||
raw_text: 节点文本(锚模式下为带 [c1]/[s1] 行号的素材)。
|
||||
question: Agent 当前关注的具体问题。
|
||||
prompts_dir: prompt 文件目录。
|
||||
anchor_map: {锚: 原文行};None 表示 v1 行为(无校验无装配无统计)。
|
||||
assemble_mode: 装配形态("ids"/"ids_expand"/"expand_only"),
|
||||
anchor_map 为 None 时忽略。
|
||||
stats_sink: 统计回调(None 不收集);统计严禁写入输出文本。
|
||||
session_id: 会话 ID(透传遥测)。
|
||||
parent_call_id: 父调用 ID(透传遥测)。
|
||||
|
||||
返回:
|
||||
"[内容摘要] {结果}\\n[核实] {验证结果}" 或错误信息。
|
||||
|
||||
关键实现细节:
|
||||
锚模式流程:提取 -> check_anchors 清洗 -> 核实轮(见清洗后未装配文本)
|
||||
-> assemble_anchored_output 装配 -> sink 上报。sink dict 完整键名:
|
||||
n_assertions/n_anchored/n_illegal(check_anchors)、
|
||||
n_expanded/n_trunc(装配)、output_chars(最终输出字符数)、
|
||||
pre_assembly(清洗后未装配文本快照)、anchor_map(原样透传)。
|
||||
"""
|
||||
extract_input = f"问题: {question}\n\n以下是视频片段的描述和字幕:\n{raw_text}"
|
||||
try:
|
||||
raw_summary = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "view_node_extract.md"),
|
||||
extract_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
except Exception as e:
|
||||
return f"[摘要错误] {e}"
|
||||
|
||||
anchor_stats: dict[str, int] = {}
|
||||
if anchor_map is not None:
|
||||
raw_summary, anchor_stats = check_anchors(raw_summary, anchor_map)
|
||||
pre_assembly = raw_summary
|
||||
|
||||
verify_input = (
|
||||
f"问题: {question}\n\n"
|
||||
f"原始内容:\n{raw_text}\n\n"
|
||||
f"以下是另一个模型基于上述内容生成的摘要,请核实:\n{raw_summary}"
|
||||
)
|
||||
try:
|
||||
verify_result = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "view_node_verify.md"),
|
||||
verify_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("验证轮调用失败,跳过: {}", e)
|
||||
verify_result = "跳过(调用失败)"
|
||||
|
||||
if anchor_map is not None:
|
||||
raw_summary, asm_stats = assemble_anchored_output(raw_summary, anchor_map, assemble_mode)
|
||||
anchor_stats.update(asm_stats)
|
||||
|
||||
result = f"[内容摘要] {raw_summary}\n[核实] {verify_result}"
|
||||
if anchor_map is not None and stats_sink is not None:
|
||||
stats_sink(
|
||||
{
|
||||
**anchor_stats,
|
||||
"output_chars": len(result),
|
||||
"pre_assembly": pre_assembly,
|
||||
"anchor_map": anchor_map,
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def summarize_children(
|
||||
llm: LLMProvider,
|
||||
children_info: list[dict[str, Any]],
|
||||
question: str,
|
||||
prompts_dir: Path,
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> str:
|
||||
"""对子节点列表做 question-conditioned 相关性标注(两轮)。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 端口实例。
|
||||
children_info: 子节点信息列表,每项含 id, time_range, summary。
|
||||
question: Agent 当前关注的具体问题。
|
||||
prompts_dir: prompt 文件目录。
|
||||
session_id: 会话 ID(透传遥测)。
|
||||
parent_call_id: 父调用 ID(透传遥测)。
|
||||
|
||||
返回:
|
||||
带相关性标注的子节点概览文本。失败时降级返回原始列表。
|
||||
"""
|
||||
lines = []
|
||||
for child in children_info:
|
||||
t_start, t_end = child["time_range"]
|
||||
lines.append(f"- {child['id']} ({t_start:.0f}-{t_end:.0f}s): {child['summary']}")
|
||||
children_text = "\n".join(lines)
|
||||
|
||||
extract_input = f"问题: {question}\n\n{children_text}"
|
||||
try:
|
||||
raw_ranking = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "view_node_children_extract.md"),
|
||||
extract_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("子节点标注失败,回退原始列表: {}", e)
|
||||
return children_text
|
||||
|
||||
verify_input = (
|
||||
f"问题: {question}\n\n"
|
||||
f"原始子节点列表:\n{children_text}\n\n"
|
||||
f"以下是另一个模型基于上述信息生成的相关性标注,请核实:\n{raw_ranking}"
|
||||
)
|
||||
try:
|
||||
verify_result = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "view_node_children_verify.md"),
|
||||
verify_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
return f"{raw_ranking}\n[核实] {verify_result}"
|
||||
except Exception as e:
|
||||
logger.warning("子节点标注验证轮失败,跳过: {}", e)
|
||||
return raw_ranking
|
||||
|
||||
|
||||
async def _summarize_search_result(
|
||||
llm: LLMProvider,
|
||||
raw_text: str,
|
||||
question: str,
|
||||
prompts_dir: Path,
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> str:
|
||||
"""对搜索结果做两轮摘要(search_similar 专用)。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 端口实例。
|
||||
raw_text: 节点原始文本。
|
||||
question: Agent 当前关注的具体问题。
|
||||
prompts_dir: prompt 文件目录。
|
||||
session_id: 会话 ID(透传遥测)。
|
||||
parent_call_id: 父调用 ID(透传遥测)。
|
||||
|
||||
返回:
|
||||
"[内容摘要] {提取结果}\\n[核实] {验证结果}" 或错误信息。
|
||||
"""
|
||||
extract_input = f"问题: {question}\n\n以下是语义搜索命中的视频节点描述和字幕:\n{raw_text}"
|
||||
try:
|
||||
raw_summary = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "search_similar_extract.md"),
|
||||
extract_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
except Exception as e:
|
||||
return f"[摘要错误] {e}"
|
||||
|
||||
verify_input = (
|
||||
f"问题: {question}\n\n"
|
||||
f"原始内容:\n{raw_text}\n\n"
|
||||
f"以下是另一个模型基于上述内容生成的摘要,请核实:\n{raw_summary}"
|
||||
)
|
||||
try:
|
||||
verify_result = await _call_llm(
|
||||
llm,
|
||||
_load_prompt(prompts_dir, "search_similar_verify.md"),
|
||||
verify_input,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
return f"[内容摘要] {raw_summary}\n[核实] {verify_result}"
|
||||
except Exception as e:
|
||||
logger.warning("搜索结果验证轮失败,跳过: {}", e)
|
||||
return f"[内容摘要] {raw_summary}\n[核实] 跳过(调用失败)"
|
||||
|
||||
|
||||
async def summarize_nodes_batch(
|
||||
llm: LLMProvider,
|
||||
items: list[tuple[str, str, str]],
|
||||
question: str,
|
||||
prompts_dir: Path,
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> list[tuple[str, str]]:
|
||||
"""并发对多个搜索结果做两轮摘要。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 端口实例。
|
||||
items: [(node_id, raw_text, extra_info), ...] 列表。
|
||||
question: Agent 当前关注的具体问题。
|
||||
prompts_dir: prompt 文件目录。
|
||||
session_id: 会话 ID(透传遥测)。
|
||||
parent_call_id: 父调用 ID(透传遥测)。
|
||||
|
||||
返回:
|
||||
[(node_id, summary_text), ...] 列表,顺序与输入一致。
|
||||
"""
|
||||
if not items:
|
||||
return []
|
||||
|
||||
async def _worker(idx: int, node_id: str, raw_text: str) -> tuple[int, str, str]:
|
||||
"""单个节点的摘要工作协程。"""
|
||||
summary = await _summarize_search_result(
|
||||
llm,
|
||||
raw_text,
|
||||
question,
|
||||
prompts_dir,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
return idx, node_id, summary
|
||||
|
||||
tasks = [_worker(i, nid, text) for i, (nid, text, _) in enumerate(items)]
|
||||
results_raw = await asyncio.gather(*tasks)
|
||||
|
||||
results: dict[int, tuple[str, str]] = {}
|
||||
for idx, node_id, summary in results_raw:
|
||||
results[idx] = (node_id, summary)
|
||||
|
||||
return [results[i] for i in range(len(items))]
|
||||
@@ -0,0 +1,325 @@
|
||||
"""搜索 Agent 工具调度器 — 工具描述与 dispatch 分发。
|
||||
|
||||
实现 ``core/agent/protocols.ToolDispatcher`` Protocol。
|
||||
连接 TreeEnvironment(数据)、summarizer(LLM 摘要)、
|
||||
vision(VLM 观察)和 skills(策略加载)。
|
||||
|
||||
与 TRM4 ``core/tree/tools.py`` 的差异:
|
||||
- 自由函数 ``dispatch()`` → ``SearchToolDispatcher`` 类(依赖注入);
|
||||
- 同步 → 全异步;
|
||||
- view_node / search_similar 内部拆分为 env 数据读取 + summarizer LLM 摘要。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.search.summarizer import summarize_children, summarize_node, summarize_nodes_batch
|
||||
from app.search.vision import observe_frame
|
||||
from app.tree.environment import _LEVEL_LABEL, TreeEnvironment, _node_level
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from app.ports import OCRProvider
|
||||
from app.search.skills import SkillRegistry
|
||||
from core.protocols import LLMProvider, VLMProvider
|
||||
|
||||
# ── 工具描述文本(与 TRM4 core/tree/tools.py 完全一致) ─────────────────
|
||||
|
||||
_BASE_DESCRIPTIONS = """\
|
||||
## 可用工具
|
||||
|
||||
在 action 中指定 tool 和 args 来调用工具。
|
||||
|
||||
### view_node
|
||||
查看节点信息,获取与问题相关的内容摘要和子节点概览。
|
||||
- args: {"node_id": "节点 ID", "question": "当前关注的具体问题"}
|
||||
|
||||
### search_similar
|
||||
语义检索最相关的节点,返回与问题相关的内容摘要。
|
||||
- args: {"query": "搜索关键词(2-4 词)", "question": "当前关注的具体问题", "k": 返回数量(可选,默认 5)}
|
||||
|
||||
### observe_frame
|
||||
调用视觉模型查看关键帧图像,回答针对性的视觉问题。
|
||||
- args: {"node_ids": ["L3 节点 ID 列表(1-4 个),或单个 L2 节点 ID"], "question": "针对帧内容的具体视觉问题"}
|
||||
|
||||
### submit_answer
|
||||
提交最终答案。
|
||||
- args: {"answer": "选项字母 A/B/C/D", "evidence": "关键证据摘要", "reasoning": "每个选项的判断理由"}"""
|
||||
|
||||
_SKILL_DESCRIPTION = """
|
||||
|
||||
### read_skill
|
||||
加载指定题型技能的详细搜索策略。
|
||||
- args: {"name": "技能名称"}"""
|
||||
|
||||
|
||||
def get_tool_descriptions(include_read_skill: bool = False) -> str:
|
||||
"""返回工具描述文本,用于写入 system prompt。
|
||||
|
||||
参数:
|
||||
include_read_skill: 是否包含 read_skill 工具(manual 模式用)。
|
||||
|
||||
返回:
|
||||
Markdown 格式的工具描述文本。
|
||||
"""
|
||||
text = _BASE_DESCRIPTIONS
|
||||
if include_read_skill:
|
||||
text += _SKILL_DESCRIPTION
|
||||
return text
|
||||
|
||||
|
||||
# ── SearchToolDispatcher ──────────────────────────────────────────────
|
||||
|
||||
|
||||
class SearchToolDispatcher:
|
||||
"""搜索 Agent 工具调度器,实现 ToolDispatcher Protocol。
|
||||
|
||||
按工具名路由到对应私有处理方法。未知工具抛 ValueError
|
||||
(AgentLoop 捕获后不计步数);节点不存在等运行时错误
|
||||
捕获后返回错误文本。
|
||||
|
||||
参数:
|
||||
env: 视频树运行时环境(纯数据访问)。
|
||||
tool_llm: 摘要用 LLM 端口。
|
||||
vlm: 视觉模型端口。
|
||||
ocr: 帧文字转录端口(None 不启用)。
|
||||
prompts_dir: prompt 文件目录。
|
||||
skills: 技能注册表(None 不启用 read_skill)。
|
||||
embed_fn: 文本嵌入函数(search_similar 用)。
|
||||
verify_vision: observe_frame 是否执行验证轮。
|
||||
anchor: view_node 是否启用行号锚模式。
|
||||
assemble_mode: 锚模式装配形态("ids"/"ids_expand"/"expand_only")。
|
||||
stats_sink: 统计回调(None 不收集)。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
env: TreeEnvironment,
|
||||
tool_llm: LLMProvider,
|
||||
vlm: VLMProvider,
|
||||
ocr: OCRProvider | None,
|
||||
prompts_dir: Path,
|
||||
skills: SkillRegistry | None,
|
||||
*,
|
||||
embed_fn: Callable[[str | list[str]], np.ndarray],
|
||||
verify_vision: bool,
|
||||
anchor: bool,
|
||||
assemble_mode: str,
|
||||
stats_sink: Callable[[dict[str, Any]], None] | None = None,
|
||||
) -> None:
|
||||
self._env = env
|
||||
self._tool_llm = tool_llm
|
||||
self._vlm = vlm
|
||||
self._ocr = ocr
|
||||
self._prompts_dir = prompts_dir
|
||||
self._skills = skills
|
||||
self._embed_fn = embed_fn
|
||||
self._verify_vision = verify_vision
|
||||
self._anchor = anchor
|
||||
self._assemble_mode = assemble_mode
|
||||
self._stats_sink = stats_sink
|
||||
|
||||
# ── ToolDispatcher Protocol 实现 ──────────────────────────────────
|
||||
|
||||
async def dispatch(
|
||||
self, tool_name: str, args: dict[str, Any], *, context: dict[str, Any]
|
||||
) -> str:
|
||||
"""按工具名分发到对应处理方法。
|
||||
|
||||
参数:
|
||||
tool_name: 工具名称。
|
||||
args: 工具参数字典。
|
||||
context: 调用上下文(含 session_id、parent_call_id 等遥测字段)。
|
||||
|
||||
返回:
|
||||
工具执行结果文本。
|
||||
|
||||
异常:
|
||||
ValueError: 未知工具名——上抛给 AgentLoop,不计步数。
|
||||
"""
|
||||
try:
|
||||
if tool_name == "view_node":
|
||||
return await self._handle_view_node(args, context)
|
||||
if tool_name == "search_similar":
|
||||
return await self._handle_search_similar(args, context)
|
||||
if tool_name == "observe_frame":
|
||||
return await self._handle_observe_frame(args, context)
|
||||
if tool_name == "submit_answer":
|
||||
return f"[ok] 答案已提交: {args['answer']}"
|
||||
if tool_name == "read_skill":
|
||||
return self._handle_read_skill(args)
|
||||
except (KeyError, FileNotFoundError) as e:
|
||||
return f"工具执行错误: {e}"
|
||||
|
||||
raise ValueError(f"未知工具: {tool_name}")
|
||||
|
||||
# ── 私有处理方法 ──────────────────────────────────────────────────
|
||||
|
||||
async def _handle_view_node(self, args: dict[str, Any], context: dict[str, Any]) -> str:
|
||||
"""view_node:节点摘要 + 子节点概览。
|
||||
|
||||
参数:
|
||||
args: {"node_id": str, "question": str}。
|
||||
context: 遥测上下文。
|
||||
|
||||
返回:
|
||||
"[节点] {id} | {level} | {time}\\n\\n{summary}\\n\\n[子节点概览] ..."
|
||||
"""
|
||||
node_id: str = args["node_id"]
|
||||
question: str = args["question"]
|
||||
session_id = context.get("session_id")
|
||||
parent_call_id = context.get("parent_call_id")
|
||||
|
||||
# Phase 1: 节点元数据(头部格式化)
|
||||
node = self._env._id_to_node[node_id]
|
||||
level = _node_level(node)
|
||||
level_label = _LEVEL_LABEL[level]
|
||||
time_str = TreeEnvironment._format_time_range(node)
|
||||
|
||||
# Phase 2: 节点内容摘要
|
||||
raw_text, anchor_map = self._env.get_node_text(node_id, anchor=self._anchor)
|
||||
summary = await summarize_node(
|
||||
self._tool_llm,
|
||||
raw_text,
|
||||
question,
|
||||
self._prompts_dir,
|
||||
anchor_map=anchor_map,
|
||||
assemble_mode=self._assemble_mode,
|
||||
stats_sink=self._stats_sink,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
|
||||
parts: list[str] = [
|
||||
f"[节点] {node_id} | {level_label} | {time_str}",
|
||||
"",
|
||||
summary,
|
||||
]
|
||||
|
||||
# Phase 2.5: 确定性追加实体/画面文字原文(防按题摘要吞噬,Spec-1 B)
|
||||
for label, text in self._env.node_entity_fields(node_id).items():
|
||||
parts.append(f"[{label}] {text}")
|
||||
|
||||
# Phase 3: 子节点概览
|
||||
children_info = self._env.get_children_info(node_id)
|
||||
if children_info:
|
||||
children_text = await summarize_children(
|
||||
self._tool_llm,
|
||||
children_info,
|
||||
question,
|
||||
self._prompts_dir,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
parts.append(f"\n[子节点概览] {len(children_info)} 个子节点\n{children_text}")
|
||||
|
||||
return "\n".join(parts)
|
||||
|
||||
async def _handle_search_similar(self, args: dict[str, Any], context: dict[str, Any]) -> str:
|
||||
"""search_similar:语义检索 + 批量摘要。
|
||||
|
||||
参数:
|
||||
args: {"query": str, "question": str, "k": int (可选)}。
|
||||
context: 遥测上下文。
|
||||
|
||||
返回:
|
||||
"[搜索结果] 查询 \\"{query}\\" → N 个相关节点\\n\\n1. ..."
|
||||
"""
|
||||
query: str = args["query"]
|
||||
question: str = args["question"]
|
||||
top_k: int = args.get("k", 5)
|
||||
session_id = context.get("session_id")
|
||||
parent_call_id = context.get("parent_call_id")
|
||||
|
||||
# Phase 1: 语义检索
|
||||
results = self._env.search_similar(query, top_k=top_k, embed_fn=self._embed_fn)
|
||||
|
||||
if not results:
|
||||
return f'[搜索结果] 查询 "{query}" → 0 个相关节点'
|
||||
|
||||
# Phase 2: 构建摘要输入
|
||||
items: list[tuple[str, str, str]] = []
|
||||
for nid, score in results:
|
||||
node = self._env._id_to_node[nid]
|
||||
raw_text, _ = self._env.get_node_text(nid)
|
||||
level = _node_level(node)
|
||||
time_str = TreeEnvironment._format_time_range(node)
|
||||
extra = f"{level} score={score:.4f} [{time_str}]"
|
||||
items.append((nid, raw_text, extra))
|
||||
|
||||
# Phase 3: 并发批量摘要
|
||||
summaries = await summarize_nodes_batch(
|
||||
self._tool_llm,
|
||||
items,
|
||||
question,
|
||||
self._prompts_dir,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
|
||||
# Phase 4: 格式化输出
|
||||
lines: list[str] = []
|
||||
for i, (nid, summary_text) in enumerate(summaries):
|
||||
_, _, extra = items[i]
|
||||
lines.append(f"{i + 1}. {nid} | {extra}\n {summary_text}")
|
||||
|
||||
header = f'[搜索结果] 查询 "{query}" → {len(results)} 个相关节点'
|
||||
return header + "\n\n" + "\n\n".join(lines)
|
||||
|
||||
async def _handle_observe_frame(self, args: dict[str, Any], context: dict[str, Any]) -> str:
|
||||
"""observe_frame:VLM 帧观察 + 字幕前置。
|
||||
|
||||
参数:
|
||||
args: {"node_ids": list[str], "question": str}。
|
||||
context: 遥测上下文。
|
||||
|
||||
返回:
|
||||
"[字幕上下文] ...\\n[视觉观察] ..." 或 "[视觉观察] ..."
|
||||
"""
|
||||
node_ids: list[str] = args["node_ids"]
|
||||
question: str = args.get("question", "")
|
||||
session_id = context.get("session_id")
|
||||
parent_call_id = context.get("parent_call_id")
|
||||
|
||||
if not question.strip():
|
||||
return "工具执行错误: question 不能为空"
|
||||
|
||||
# Phase 1: 解析帧路径和字幕
|
||||
frame_paths = self._env.resolve_frame_paths(node_ids)
|
||||
subtitle = self._env.get_subtitle(node_ids[0])
|
||||
|
||||
# Phase 2: VLM 调用
|
||||
result = await observe_frame(
|
||||
self._vlm,
|
||||
frame_paths,
|
||||
question,
|
||||
self._prompts_dir,
|
||||
ocr=self._ocr,
|
||||
verify=self._verify_vision,
|
||||
stats_sink=self._stats_sink,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
|
||||
# Phase 3: 字幕前置拼接
|
||||
if subtitle:
|
||||
return f"[字幕上下文] {subtitle}\n{result}"
|
||||
return result
|
||||
|
||||
def _handle_read_skill(self, args: dict[str, Any]) -> str:
|
||||
"""read_skill:加载指定技能的搜索策略正文。
|
||||
|
||||
参数:
|
||||
args: {"name": str}。
|
||||
|
||||
返回:
|
||||
技能正文或错误提示。
|
||||
"""
|
||||
if self._skills is None:
|
||||
return "错误: skills 未启用"
|
||||
return self._skills.read(args["name"])
|
||||
@@ -0,0 +1,157 @@
|
||||
"""视觉模型调用模块 -- 两轮 VLM 调用查看关键帧图像。
|
||||
|
||||
提取轮:带防幻觉 system prompt,提取原始视觉证据。
|
||||
验证轮:把初稿全文喂回,逐条核实并给置信度。
|
||||
|
||||
从 TRM4 ``core/tree/vision.py`` 迁移,关键变更:
|
||||
- VLM 调用走 ``VLMProvider.chat_with_images`` Protocol,images 传 Path 列表;
|
||||
- OCR 调用走 ``OCRProvider.transcribe_frames`` 异步 Protocol;
|
||||
- 遥测字段(session_id / parent_call_id)透传给 VLM 调用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from app.ports import OCRProvider
|
||||
from core.protocols import VLMProvider
|
||||
|
||||
_OCR_PREFIX = (
|
||||
"以下是 OCR 工具对这些帧的文字转录,仅供参考;与你实际看到的不一致时,报告双读数并标注分歧:\n"
|
||||
)
|
||||
|
||||
|
||||
def _load_prompt(prompts_dir: Path, filename: str) -> str:
|
||||
"""从 prompts 目录加载 system prompt 文件。
|
||||
|
||||
参数:
|
||||
prompts_dir: prompt 文件所在目录。
|
||||
filename: prompt 文件名。
|
||||
|
||||
返回:
|
||||
文件内容字符串。
|
||||
"""
|
||||
return (prompts_dir / filename).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
async def observe_frame(
|
||||
vlm: VLMProvider,
|
||||
frame_paths: list[Path],
|
||||
question: str,
|
||||
prompts_dir: Path,
|
||||
*,
|
||||
ocr: OCRProvider | None,
|
||||
verify: bool,
|
||||
stats_sink: Callable[[dict[str, int]], None] | None = None,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
) -> str:
|
||||
"""调用 VLM 查看帧图像:可选 OCR 事前并置 + 提取轮 + 可选验证轮。
|
||||
|
||||
参数:
|
||||
vlm: VLM 图文调用端口。
|
||||
frame_paths: 帧文件路径列表。
|
||||
question: 针对帧内容的视觉问题。
|
||||
prompts_dir: prompt 文件目录。
|
||||
ocr: 帧文字转录端口(None=不注入;返回空串视为无结果不注入)。
|
||||
verify: 是否执行验证轮(False 时仅提取轮,输出无 [验证] 段)。
|
||||
stats_sink: 统计回调(None 不收集);统计严禁写入输出文本。
|
||||
session_id: 遥测会话 ID,透传给 VLM 调用。
|
||||
parent_call_id: 遥测父调用 ID,透传给 VLM 调用。
|
||||
|
||||
返回:
|
||||
verify=True 为 ``"[视觉观察] {证据}\\n[验证] {核实结果}"``,
|
||||
verify=False 为 ``"[视觉观察] {证据}"``,或错误信息。
|
||||
|
||||
关键实现细节:
|
||||
OCR 文本作为额外文本并置于问题之前(事前并置——OCR 误读不进
|
||||
工具输出故零 judge 口径风险);OCR 异常降级为不注入并计
|
||||
ocr_failed(ocr 是外部注入依赖,任何异常都不得中断工具主流程,
|
||||
故此处 except Exception 是刻意的降级边界)。sink 键:
|
||||
ocr_injected / ocr_chars / ocr_failed / discrepancy(输出含"分歧"词面)/
|
||||
abstain(含 [证据不存在])。
|
||||
"""
|
||||
stats: dict[str, int] = {
|
||||
"ocr_injected": 0,
|
||||
"ocr_chars": 0,
|
||||
"ocr_failed": 0,
|
||||
"discrepancy": 0,
|
||||
"abstain": 0,
|
||||
}
|
||||
|
||||
def _emit(output: str) -> str:
|
||||
"""计算语义标记并回调 stats_sink。"""
|
||||
stats["abstain"] = int("[证据不存在]" in output)
|
||||
stats["discrepancy"] = int("分歧" in output)
|
||||
if stats_sink is not None:
|
||||
stats_sink(stats)
|
||||
return output
|
||||
|
||||
# -- 帧文件存在性校验 --
|
||||
for p in frame_paths:
|
||||
if not p.exists():
|
||||
return _emit(f"[VL错误] 帧文件不存在: {p}")
|
||||
|
||||
# -- OCR 转录(可选) --
|
||||
ocr_text = ""
|
||||
if ocr is not None:
|
||||
try:
|
||||
ocr_text = await ocr.transcribe_frames(frame_paths)
|
||||
except Exception as e: # noqa: BLE001 — 刻意的降级边界
|
||||
logger.warning("OCR 转录失败,降级不注入: {}", e)
|
||||
stats["ocr_failed"] = 1
|
||||
|
||||
# -- 拼装提取轮 user 消息 --
|
||||
user_parts: list[str] = []
|
||||
if ocr_text:
|
||||
stats["ocr_injected"] = 1
|
||||
stats["ocr_chars"] = len(ocr_text)
|
||||
user_parts.append(_OCR_PREFIX + ocr_text)
|
||||
user_parts.append(question)
|
||||
user_text = "\n".join(user_parts)
|
||||
|
||||
extract_messages = [
|
||||
{"role": "system", "content": _load_prompt(prompts_dir, "observe_frame_extract.md")},
|
||||
{"role": "user", "content": user_text},
|
||||
]
|
||||
|
||||
# -- 提取轮 --
|
||||
try:
|
||||
extract_response = await vlm.chat_with_images(
|
||||
extract_messages,
|
||||
images=frame_paths,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
raw_evidence = extract_response.content
|
||||
except Exception as e: # noqa: BLE001
|
||||
return _emit(f"[VL错误] {e}")
|
||||
|
||||
if not verify:
|
||||
return _emit(f"[视觉观察] {raw_evidence}")
|
||||
|
||||
# -- 验证轮 --
|
||||
verify_text = (
|
||||
f"问题: {question}\n\n以下是另一个模型基于这些图片生成的描述,请核实:\n{raw_evidence}"
|
||||
)
|
||||
verify_messages = [
|
||||
{"role": "system", "content": _load_prompt(prompts_dir, "observe_frame_verify.md")},
|
||||
{"role": "user", "content": verify_text},
|
||||
]
|
||||
try:
|
||||
verify_response = await vlm.chat_with_images(
|
||||
verify_messages,
|
||||
images=frame_paths,
|
||||
session_id=session_id,
|
||||
parent_call_id=parent_call_id,
|
||||
)
|
||||
return _emit(f"[视觉观察] {raw_evidence}\n[验证] {verify_response.content}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("验证轮调用失败,跳过: {}", e)
|
||||
return _emit(f"[视觉观察] {raw_evidence}\n[验证] 跳过(调用失败)")
|
||||
@@ -0,0 +1,42 @@
|
||||
"""建树模块配置。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TreeConfig:
|
||||
"""建树配置参数,字段对齐 config/default.yaml 的 tree: 段。
|
||||
|
||||
参数:
|
||||
l1_segment_duration: L1 段时长(秒)。
|
||||
l2_clip_duration: L2 clip 时长(秒)。
|
||||
l3_fps: L3 帧提取频率(帧/秒)。
|
||||
l2_representative_frames: L2 VLM 描述用的代表帧数。
|
||||
cache_dir: 树索引缓存目录。
|
||||
concurrency: asyncio Semaphore 上限。
|
||||
subtitle_inject: 建树时是否注入 SRT 字幕。
|
||||
srt_window_sec: 字幕匹配时间窗口(前后各 N 秒)。
|
||||
"""
|
||||
|
||||
l1_segment_duration: float = 600.0
|
||||
l2_clip_duration: float = 60.0
|
||||
l3_fps: float = 0.5
|
||||
l2_representative_frames: int = 6
|
||||
cache_dir: str = "cache/trees"
|
||||
concurrency: int = 16
|
||||
subtitle_inject: bool = True
|
||||
srt_window_sec: float = 5.0
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> TreeConfig:
|
||||
"""从 YAML 解析后的 dict 构造,忽略未知字段。
|
||||
|
||||
参数:
|
||||
d: 配置字典。
|
||||
|
||||
返回:
|
||||
TreeConfig 实例。
|
||||
"""
|
||||
return cls(**{k: v for k, v in d.items() if k in cls.__dataclass_fields__})
|
||||
@@ -0,0 +1,616 @@
|
||||
"""TreeEnvironment:单棵视频树的运行时环境。
|
||||
|
||||
提供节点查询、字幕获取、帧路径解析和语义检索能力。
|
||||
纯数据访问层——不涉及 LLM 调用,LLM 摘要逻辑属于 app/search/。
|
||||
|
||||
算法 #12 变更:分块 embedding → 单节点 embedding。
|
||||
祖先去重 + 锚定验证逻辑保留自 TRM4。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
from app.tree.index import L1Node, L2Node, L3Node, TreeIndex
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
# 节点联合类型(内部使用)
|
||||
AnyNode = L1Node | L2Node | L3Node
|
||||
|
||||
# 各层级节点对应的主描述字段名
|
||||
_LEVEL_LABEL = {
|
||||
"L1": "场景层",
|
||||
"L2": "事件层",
|
||||
"L3": "关键帧层",
|
||||
}
|
||||
|
||||
|
||||
def _node_level(node: AnyNode) -> str:
|
||||
"""判断节点层级标签。
|
||||
|
||||
参数:
|
||||
node: 树节点实例。
|
||||
|
||||
返回:
|
||||
"L1" / "L2" / "L3"。
|
||||
"""
|
||||
if isinstance(node, L1Node):
|
||||
return "L1"
|
||||
if isinstance(node, L2Node):
|
||||
return "L2"
|
||||
return "L3"
|
||||
|
||||
|
||||
def _node_description(node: AnyNode) -> str:
|
||||
"""提取节点的主描述文本。
|
||||
|
||||
参数:
|
||||
node: 树节点实例。
|
||||
|
||||
返回:
|
||||
描述文本字符串。
|
||||
"""
|
||||
if isinstance(node, L1Node):
|
||||
return node.card.scene_summary
|
||||
if isinstance(node, L2Node):
|
||||
return node.card.event_description
|
||||
return node.card.frame_summary
|
||||
|
||||
|
||||
def _collect_card_strings(
|
||||
node: AnyNode,
|
||||
skip_fields: frozenset[str] = frozenset(),
|
||||
) -> list[str]:
|
||||
"""从节点 card 中递归收集所有非空字符串字段。
|
||||
|
||||
参数:
|
||||
node: 树节点实例。
|
||||
skip_fields: 需要跳过的 dataclass 字段名集合(如 subtitle,
|
||||
因为它需要单独添加"字幕:"标签和 [sN] 锚标)。
|
||||
|
||||
返回:
|
||||
字符串列表(每个非空字段值一项,含内嵌换行的按行拆分)。
|
||||
"""
|
||||
result: list[str] = []
|
||||
_collect_from_obj(node.card, result, skip_fields=skip_fields)
|
||||
return result
|
||||
|
||||
|
||||
# subtitle 字段在 _node_full_text / _node_anchored_text 中单独处理
|
||||
_SUBTITLE_SKIP: frozenset[str] = frozenset({"subtitle"})
|
||||
|
||||
# 各层级 card 的实体字段名(B 修复:dispatcher 追加原文用)
|
||||
_ENTITY_FIELDS_BY_LEVEL: dict[str, tuple[str, ...]] = {
|
||||
"L1": ("key_entities",),
|
||||
"L2": ("entities",),
|
||||
"L3": ("visible_entities",),
|
||||
}
|
||||
|
||||
|
||||
def _clean_join(values: Iterable[object] | None) -> str:
|
||||
"""过滤非空字符串、去重(保序)、分号拼接;无有效值返回空串。
|
||||
|
||||
参数:
|
||||
values: 待清洗的值集合(通常为 list[str],容忍 None / 混入非字符串)。
|
||||
|
||||
返回:
|
||||
"a; b; c" 形式的拼接串,无有效值时为空串。
|
||||
"""
|
||||
cleaned = [v.strip() for v in (values or []) if isinstance(v, str) and v.strip()]
|
||||
return "; ".join(dict.fromkeys(cleaned))
|
||||
|
||||
|
||||
def _collect_from_obj(
|
||||
obj: object,
|
||||
out: list[str],
|
||||
*,
|
||||
skip_fields: frozenset[str] = frozenset(),
|
||||
) -> None:
|
||||
"""递归收集任意嵌套结构中的非空字符串。
|
||||
|
||||
参数:
|
||||
obj: dict / list / str / 其他。
|
||||
out: 收集结果列表(原地修改)。
|
||||
skip_fields: 需要跳过的 dataclass 字段名集合。
|
||||
"""
|
||||
if isinstance(obj, str):
|
||||
stripped = obj.strip()
|
||||
if stripped:
|
||||
out.append(stripped)
|
||||
elif isinstance(obj, dict):
|
||||
for v in obj.values():
|
||||
_collect_from_obj(v, out, skip_fields=skip_fields)
|
||||
elif isinstance(obj, (list, tuple)):
|
||||
for item in obj:
|
||||
_collect_from_obj(item, out, skip_fields=skip_fields)
|
||||
elif hasattr(obj, "__dataclass_fields__"):
|
||||
# frozen dataclass(Card 类型)
|
||||
for field_name in obj.__dataclass_fields__:
|
||||
if field_name in skip_fields:
|
||||
continue
|
||||
_collect_from_obj(getattr(obj, field_name), out, skip_fields=skip_fields)
|
||||
|
||||
|
||||
class TreeEnvironment:
|
||||
"""单棵视频树的运行时环境,提供节点查询和语义检索。
|
||||
|
||||
纯数据访问层,不涉及 LLM 调用。
|
||||
|
||||
参数:
|
||||
index: 已加载的 TreeIndex 实例。
|
||||
frames_dir: 帧文件目录路径(可选;未提供时使用节点自带的 frame_path)。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
index: TreeIndex,
|
||||
frames_dir: Path | None = None,
|
||||
) -> None:
|
||||
self._index = index
|
||||
self._frames_dir = frames_dir
|
||||
|
||||
# O(1) 查找表:node_id → 节点实例
|
||||
self._id_to_node: dict[str, AnyNode] = {}
|
||||
# 父节点映射:node_id → parent_id(根节点为 None)
|
||||
self._id_to_parent: dict[str, str | None] = {}
|
||||
|
||||
self._build_lookup_tables()
|
||||
logger.debug(
|
||||
"TreeEnvironment 初始化完成,节点数={}",
|
||||
len(self._id_to_node),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 初始化辅助
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_lookup_tables(self) -> None:
|
||||
"""遍历 TreeIndex 构建 _id_to_node 和 _id_to_parent 映射表。"""
|
||||
for l1 in self._index.roots:
|
||||
self._id_to_node[l1.id] = l1
|
||||
self._id_to_parent[l1.id] = None
|
||||
for l2 in l1.children:
|
||||
self._id_to_node[l2.id] = l2
|
||||
self._id_to_parent[l2.id] = l1.id
|
||||
for l3 in l2.children:
|
||||
self._id_to_node[l3.id] = l3
|
||||
self._id_to_parent[l3.id] = l2.id
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 公开方法
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def view_node(self, node_id: str, *, anchor: bool = False) -> str:
|
||||
"""返回节点卡片内容 + 子节点概览。
|
||||
|
||||
参数:
|
||||
node_id: 节点 ID。
|
||||
anchor: 为卡片字段添加行锚标 [c1] [s1] 供引用验证。
|
||||
|
||||
返回:
|
||||
格式化文本。
|
||||
|
||||
异常:
|
||||
KeyError: 节点不存在。
|
||||
"""
|
||||
node = self._id_to_node.get(node_id)
|
||||
if node is None:
|
||||
raise KeyError(f"节点不存在: {node_id}")
|
||||
|
||||
level = _node_level(node)
|
||||
level_label = _LEVEL_LABEL[level]
|
||||
|
||||
# 时间范围
|
||||
time_range_str = self._format_time_range(node)
|
||||
|
||||
# 节点内容
|
||||
content = self._node_anchored_text(node) if anchor else self._node_full_text(node)
|
||||
|
||||
parts = [
|
||||
f"[节点] {node_id} | {level_label} | {time_range_str}",
|
||||
"",
|
||||
content,
|
||||
]
|
||||
|
||||
# 子节点概览
|
||||
children = self._get_children(node)
|
||||
if children:
|
||||
parts.append("")
|
||||
parts.append(f"[子节点概览] {len(children)} 个子节点")
|
||||
for child in children:
|
||||
child_desc = _node_description(child)
|
||||
child_time = self._format_time_range(child)
|
||||
# 截断描述到 120 字符
|
||||
if len(child_desc) > 120:
|
||||
child_desc = child_desc[:120] + "..."
|
||||
parts.append(f" - {child.id} | {child_time} | {child_desc}")
|
||||
|
||||
return "\n".join(parts)
|
||||
|
||||
def node_entity_fields(self, node_id: str) -> dict[str, str]:
|
||||
"""返回节点 card 的实体/画面文字字段原文。
|
||||
|
||||
供 dispatcher 在按题摘要后确定性追加,防止 LLM 摘要吞掉
|
||||
entities/visible_text 信号(benchmark 错题 M1 恶化因素)。
|
||||
|
||||
参数:
|
||||
node_id: 节点 ID。
|
||||
|
||||
返回:
|
||||
{"实体": "...", "画面文字": "..."},空字段不含对应键。
|
||||
|
||||
异常:
|
||||
KeyError: 节点不存在。
|
||||
"""
|
||||
node = self._id_to_node.get(node_id)
|
||||
if node is None:
|
||||
raise KeyError(f"节点不存在: {node_id}")
|
||||
level = _node_level(node)
|
||||
|
||||
entity_values: list[str] = []
|
||||
for field_name in _ENTITY_FIELDS_BY_LEVEL[level]:
|
||||
entity_values.extend(getattr(node.card, field_name) or [])
|
||||
|
||||
out: dict[str, str] = {}
|
||||
entities_text = _clean_join(entity_values)
|
||||
if entities_text:
|
||||
out["实体"] = entities_text
|
||||
visible_text = _clean_join(getattr(node.card, "visible_text", None))
|
||||
if visible_text:
|
||||
out["画面文字"] = visible_text
|
||||
return out
|
||||
|
||||
def search_similar(
|
||||
self,
|
||||
query: str,
|
||||
top_k: int = 5,
|
||||
*,
|
||||
embed_fn: Callable[[str | list[str]], np.ndarray] | None = None,
|
||||
) -> list[tuple[str, float]]:
|
||||
"""语义搜索 + 祖先去重。
|
||||
|
||||
算法 #12 变更:单节点 embedding(非分块),祖先去重 + 锚定验证保留。
|
||||
|
||||
参数:
|
||||
query: 搜索文本。
|
||||
top_k: 返回数量。
|
||||
embed_fn: 嵌入函数(未提供时使用 TreeIndex 已有 embedding)。
|
||||
|
||||
返回:
|
||||
[(node_id, score), ...] 按相似度降序。
|
||||
|
||||
异常:
|
||||
ValueError: 节点未 embed 且未提供 embed_fn。
|
||||
"""
|
||||
if embed_fn is None:
|
||||
raise ValueError(
|
||||
"embed_fn 为必需参数:搜索 query 需要 embed_fn 来编码。请传入 embed_fn 参数。"
|
||||
)
|
||||
|
||||
# 收集所有节点的 embedding(优先使用 TreeIndex 已有 embedding)
|
||||
node_ids: list[str] = []
|
||||
embeddings: list[np.ndarray] = []
|
||||
|
||||
if self._index.is_embedded:
|
||||
# 使用已有 embedding
|
||||
for nid, node in self._id_to_node.items():
|
||||
if node.embedding is not None:
|
||||
node_ids.append(nid)
|
||||
embeddings.append(node.embedding)
|
||||
else:
|
||||
# 使用 embed_fn 为所有节点生成 embedding
|
||||
all_ids = list(self._id_to_node.keys())
|
||||
all_texts = [_node_description(self._id_to_node[nid]) for nid in all_ids]
|
||||
all_embs = embed_fn(all_texts) # [N, D]
|
||||
for i, nid in enumerate(all_ids):
|
||||
node_ids.append(nid)
|
||||
embeddings.append(all_embs[i])
|
||||
|
||||
if not embeddings:
|
||||
return []
|
||||
|
||||
node_embeddings = np.stack(embeddings, axis=0) # [N, D]
|
||||
# 归一化(确保余弦相似度正确)
|
||||
norms = np.linalg.norm(node_embeddings, axis=1, keepdims=True)
|
||||
norms = np.where(norms == 0, 1.0, norms)
|
||||
node_embeddings = node_embeddings / norms
|
||||
|
||||
# 编码 query
|
||||
query_emb = embed_fn(query) # [1, D]
|
||||
|
||||
if query_emb.ndim == 1:
|
||||
query_emb = query_emb.reshape(1, -1)
|
||||
# 归一化 query
|
||||
q_norm = np.linalg.norm(query_emb)
|
||||
if q_norm > 0:
|
||||
query_emb = query_emb / q_norm
|
||||
|
||||
# 余弦相似度
|
||||
scores = (node_embeddings @ query_emb.T).squeeze() # [N]
|
||||
if scores.ndim == 0:
|
||||
scores = scores.reshape(1)
|
||||
|
||||
# 按分数排序
|
||||
scored_pairs = sorted(
|
||||
zip(node_ids, scores.tolist(), strict=True),
|
||||
key=lambda x: x[1],
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
# 祖先去重:如果更细粒度的子节点已入选,跳过其祖先
|
||||
deduped: list[tuple[str, float]] = []
|
||||
seen_prefixes: set[str] = set()
|
||||
for nid, score in scored_pairs:
|
||||
is_ancestor_of_seen = any(s.startswith(nid + "_") for s in seen_prefixes)
|
||||
if is_ancestor_of_seen:
|
||||
continue
|
||||
deduped.append((nid, score))
|
||||
seen_prefixes.add(nid)
|
||||
if len(deduped) >= top_k:
|
||||
break
|
||||
|
||||
return deduped
|
||||
|
||||
def get_node_text(
|
||||
self,
|
||||
node_id: str,
|
||||
*,
|
||||
anchor: bool = False,
|
||||
) -> tuple[str, dict[str, str] | None]:
|
||||
"""返回节点原始文本及可选的锚映射表。
|
||||
|
||||
供 SearchToolDispatcher 使用:将原始文本和锚映射传给
|
||||
summarizer.summarize_node(),实现引用验证。
|
||||
|
||||
参数:
|
||||
node_id: 节点 ID。
|
||||
anchor: 若 True,返回带 [cN]/[sN] 锚标的文本并构建 anchor_map。
|
||||
|
||||
返回:
|
||||
(text, anchor_map) 元组。anchor=False 时 anchor_map 为 None;
|
||||
anchor=True 时 anchor_map 为 {"c1": "行文本", "s1": "字幕行", ...}。
|
||||
|
||||
异常:
|
||||
KeyError: 节点不存在。
|
||||
"""
|
||||
node = self._id_to_node.get(node_id)
|
||||
if node is None:
|
||||
raise KeyError(f"节点不存在: {node_id}")
|
||||
|
||||
if not anchor:
|
||||
return self._node_full_text(node), None
|
||||
|
||||
anchored_text = self._node_anchored_text(node)
|
||||
# 解析锚标行 "[c1] xxx" / "[s2] yyy" 构建映射
|
||||
anchor_map: dict[str, str] = {}
|
||||
anchor_pattern = re.compile(r"^\[([cs]\d+)\]\s(.+)$")
|
||||
for line in anchored_text.splitlines():
|
||||
m = anchor_pattern.match(line)
|
||||
if m:
|
||||
anchor_map[m.group(1)] = m.group(2)
|
||||
|
||||
return anchored_text, anchor_map
|
||||
|
||||
def get_children_info(self, node_id: str) -> list[dict[str, Any]]:
|
||||
"""返回节点的直接子节点结构化信息。
|
||||
|
||||
供 SearchToolDispatcher 使用:将子节点列表传给
|
||||
summarizer.summarize_children(),用于层级摘要。
|
||||
|
||||
参数:
|
||||
node_id: 节点 ID。
|
||||
|
||||
返回:
|
||||
子节点信息列表,每项包含 {"id", "time_range", "summary"}。
|
||||
time_range 为 (start, end) 数值元组(L3 节点退化为 (ts, ts))。
|
||||
L3 叶子节点返回空列表。
|
||||
|
||||
异常:
|
||||
KeyError: 节点不存在。
|
||||
"""
|
||||
node = self._id_to_node.get(node_id)
|
||||
if node is None:
|
||||
raise KeyError(f"节点不存在: {node_id}")
|
||||
|
||||
children = self._get_children(node)
|
||||
result: list[dict[str, Any]] = []
|
||||
for child in children:
|
||||
desc = _node_description(child)
|
||||
if len(desc) > 120:
|
||||
desc = desc[:120] + "..."
|
||||
result.append(
|
||||
{
|
||||
"id": child.id,
|
||||
"time_range": self._node_time_range_raw(child),
|
||||
"summary": desc,
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
def get_subtitle(self, node_id: str) -> str:
|
||||
"""返回节点字幕文本。
|
||||
|
||||
L2/L3 节点从 card.subtitle 读取,L1 节点不含字幕。
|
||||
|
||||
参数:
|
||||
node_id: 节点 ID。
|
||||
|
||||
返回:
|
||||
字幕文本;无字幕、L1 节点或节点不存在时返回空字符串。
|
||||
"""
|
||||
node = self._id_to_node.get(node_id)
|
||||
if node is None:
|
||||
return ""
|
||||
if isinstance(node, (L2Node, L3Node)):
|
||||
return node.card.subtitle or ""
|
||||
return ""
|
||||
|
||||
def resolve_frame_paths(self, node_ids: list[str]) -> list[Path]:
|
||||
"""node_id → 帧文件路径。支持 L3(直接映射)和 L2(展开为 L3 children)。
|
||||
|
||||
参数:
|
||||
node_ids: 节点 ID 列表。
|
||||
|
||||
返回:
|
||||
帧文件 Path 列表。
|
||||
|
||||
异常:
|
||||
KeyError: 节点不存在。
|
||||
"""
|
||||
if not node_ids:
|
||||
return []
|
||||
|
||||
paths: list[Path] = []
|
||||
for nid in node_ids:
|
||||
node = self._id_to_node.get(nid)
|
||||
if node is None:
|
||||
raise KeyError(f"节点不存在: {nid}")
|
||||
|
||||
if isinstance(node, L3Node):
|
||||
paths.append(self._l3_frame_path(node))
|
||||
elif isinstance(node, L2Node):
|
||||
# 展开为所有 L3 子节点
|
||||
for l3 in node.children:
|
||||
paths.append(self._l3_frame_path(l3))
|
||||
else:
|
||||
# L1 节点:展开为所有 L2 下的 L3
|
||||
assert isinstance(node, L1Node)
|
||||
for l2 in node.children:
|
||||
for l3 in l2.children:
|
||||
paths.append(self._l3_frame_path(l3))
|
||||
|
||||
return paths
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 内部辅助方法
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _l3_frame_path(self, node: L3Node) -> Path:
|
||||
"""将 L3 节点映射到帧文件路径。
|
||||
|
||||
参数:
|
||||
node: L3 节点。
|
||||
|
||||
返回:
|
||||
帧文件 Path。
|
||||
"""
|
||||
if self._frames_dir is not None:
|
||||
# 从 node.id 中提取后缀(去掉 video_id 前缀)
|
||||
# ID 格式: {video_id}_{L1_xxx_L2_xxx_L3_xxx}
|
||||
# frame_path 格式: frames/{L1_xxx_L2_xxx_L3_xxx}.jpg
|
||||
if node.frame_path:
|
||||
return self._frames_dir / Path(node.frame_path).name
|
||||
# fallback: 从 ID 推断
|
||||
parts = node.id.split("_", 1)
|
||||
suffix = parts[1] if len(parts) > 1 else node.id
|
||||
return self._frames_dir / f"{suffix}.jpg"
|
||||
|
||||
# 无 frames_dir 时使用节点自带路径
|
||||
if node.frame_path:
|
||||
return Path(node.frame_path)
|
||||
raise ValueError(f"L3 节点无 frame_path 且未提供 frames_dir: {node.id}")
|
||||
|
||||
def _node_full_text(self, node: AnyNode) -> str:
|
||||
"""获取节点完整文本(card 所有字段 + subtitle)。
|
||||
|
||||
subtitle 从 card.subtitle 读取,仅 L2/L3 节点附加"字幕:"标签。
|
||||
|
||||
参数:
|
||||
node: 树节点。
|
||||
|
||||
返回:
|
||||
拼接后的全文本。
|
||||
"""
|
||||
card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP)
|
||||
text = "\n".join(card_strings)
|
||||
if isinstance(node, (L2Node, L3Node)) and node.card.subtitle:
|
||||
text += f"\n字幕: {node.card.subtitle}"
|
||||
return text
|
||||
|
||||
def _node_anchored_text(self, node: AnyNode) -> str:
|
||||
"""获取带行号锚的节点文本。
|
||||
|
||||
card 字符串逐行编 [c1]..[cN],字幕逐行编 [s1]..[sM]。
|
||||
字幕从 card.subtitle 读取,仅 L2/L3 节点产生 [sN] 锚标。
|
||||
|
||||
参数:
|
||||
node: 树节点。
|
||||
|
||||
返回:
|
||||
带锚文本。
|
||||
"""
|
||||
card_strings = _collect_card_strings(node, skip_fields=_SUBTITLE_SKIP)
|
||||
# 拆分内嵌换行,确保一锚一行
|
||||
card_lines: list[str] = []
|
||||
for s in card_strings:
|
||||
card_lines.extend(ln for ln in s.splitlines() if ln.strip())
|
||||
|
||||
sub_lines: list[str] = []
|
||||
if isinstance(node, (L2Node, L3Node)) and node.card.subtitle:
|
||||
sub_lines = [ln for ln in node.card.subtitle.splitlines() if ln.strip()]
|
||||
|
||||
anchored: list[str] = []
|
||||
for i, line in enumerate(card_lines, 1):
|
||||
anchored.append(f"[c{i}] {line}")
|
||||
for i, line in enumerate(sub_lines, 1):
|
||||
anchored.append(f"[s{i}] {line}")
|
||||
|
||||
return "\n".join(anchored)
|
||||
|
||||
@staticmethod
|
||||
def _format_time_range(node: AnyNode) -> str:
|
||||
"""格式化节点的时间范围。
|
||||
|
||||
参数:
|
||||
node: 树节点。
|
||||
|
||||
返回:
|
||||
"start-end s" 格式字符串,或 timestamp,或 "N/A"。
|
||||
"""
|
||||
if isinstance(node, (L1Node, L2Node)) and node.time_range:
|
||||
return f"{node.time_range[0]:.1f}-{node.time_range[1]:.1f}s"
|
||||
if isinstance(node, L3Node) and node.timestamp is not None:
|
||||
return f"{node.timestamp:.1f}s"
|
||||
return "N/A"
|
||||
|
||||
@staticmethod
|
||||
def _node_time_range_raw(node: AnyNode) -> tuple[float, float]:
|
||||
"""提取节点时间范围的原始数值元组。
|
||||
|
||||
L1/L2 返回 time_range 元组;L3 退化为 (timestamp, timestamp);
|
||||
全部为 None 时兜底 (0.0, 0.0)。
|
||||
|
||||
参数:
|
||||
node: 树节点。
|
||||
|
||||
返回:
|
||||
(start, end) 秒级数值元组。
|
||||
"""
|
||||
if isinstance(node, (L1Node, L2Node)) and node.time_range:
|
||||
return node.time_range
|
||||
if isinstance(node, L3Node) and node.timestamp is not None:
|
||||
return (node.timestamp, node.timestamp)
|
||||
return (0.0, 0.0)
|
||||
|
||||
@staticmethod
|
||||
def _get_children(node: AnyNode) -> list[AnyNode]:
|
||||
"""获取节点的直接子节点列表。
|
||||
|
||||
参数:
|
||||
node: 树节点。
|
||||
|
||||
返回:
|
||||
子节点列表(L3 节点返回空列表)。
|
||||
"""
|
||||
if isinstance(node, L1Node):
|
||||
return list(node.children)
|
||||
if isinstance(node, L2Node):
|
||||
return list(node.children)
|
||||
return []
|
||||
@@ -0,0 +1,774 @@
|
||||
"""三层树索引核心数据结构。
|
||||
|
||||
定义 Video-Tree-TRM 的三层树状索引结构,是所有后续模块
|
||||
(builder、harness、search)的基础依赖。
|
||||
|
||||
数据结构层次::
|
||||
|
||||
TreeIndex
|
||||
└─ List[L1Node] 全局叙事节点
|
||||
└─ List[L2Node] 片段级语义节点
|
||||
└─ List[L3Node] 帧/细节级节点
|
||||
|
||||
与参考项目 (TRM4) 的关键区别:
|
||||
- Card 体系:每层节点的描述信息封装为 frozen dataclass(L1Card/L2Card/L3Card),
|
||||
字段来自 VLM 结构化输出,保证不可变。
|
||||
- 序列化方式:仅保留 JSON(移除 pickle)。
|
||||
- 统一嵌入空间:所有 embedding 均来自 text_embed(),无跨模态问题。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import dataclasses
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import numpy as np
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Embedding 序列化辅助函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _embed_to_str(arr: np.ndarray | None) -> str | None:
|
||||
"""float32 ndarray -> base64 字符串(用于 JSON 序列化)。
|
||||
|
||||
参数:
|
||||
arr: float32 数组,形状任意。
|
||||
|
||||
返回:
|
||||
base64 编码字符串,或 None(输入为 None 时)。
|
||||
"""
|
||||
if arr is None:
|
||||
return None
|
||||
return base64.b64encode(arr.astype(np.float32).tobytes()).decode()
|
||||
|
||||
|
||||
def _embed_from_str(s: str | None) -> np.ndarray | None:
|
||||
"""base64 字符串 -> float32 ndarray(用于 JSON 反序列化)。
|
||||
|
||||
参数:
|
||||
s: base64 编码字符串。
|
||||
|
||||
返回:
|
||||
float32 数组,或 None(输入为 None/空时)。
|
||||
"""
|
||||
if s is None or s == "":
|
||||
return None
|
||||
return np.frombuffer(base64.b64decode(s), dtype=np.float32)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Card 数据结构(frozen,来自 VLM 结构化输出)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class L3Card:
|
||||
"""L3 帧级语义卡片(不可变)。
|
||||
|
||||
封装 VLM 对单帧的结构化描述输出。
|
||||
|
||||
属性:
|
||||
frame_summary: 帧内容摘要。
|
||||
visible_entities: 可见实体列表。
|
||||
ongoing_actions: 正在进行的动作列表。
|
||||
visible_text: 画面中可见的文字列表。
|
||||
spatial_layout: 空间布局描述。
|
||||
visual_attributes: 视觉属性字典(如光照、色调等)。
|
||||
subtitle: 字幕文本(Voronoi 分配后填充,默认空)。
|
||||
"""
|
||||
|
||||
frame_summary: str
|
||||
visible_entities: list[str]
|
||||
ongoing_actions: list[str]
|
||||
visible_text: list[str]
|
||||
spatial_layout: str
|
||||
visual_attributes: dict[str, Any]
|
||||
subtitle: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class L2Card:
|
||||
"""L2 事件级语义卡片(不可变)。
|
||||
|
||||
封装 VLM 对一个事件片段的结构化描述输出。
|
||||
|
||||
属性:
|
||||
event_description: 事件描述。
|
||||
entities: 参与实体列表。
|
||||
actions: 动作列表。
|
||||
action_subjects: 动作主体列表。
|
||||
visible_text: 片段中可见的文字列表。
|
||||
spatial_relations: 空间关系描述。
|
||||
state_changes: 状态变化描述(可选)。
|
||||
subtitle: 子 L3 字幕聚合文本(Voronoi 分配后填充,默认空)。
|
||||
"""
|
||||
|
||||
event_description: str
|
||||
entities: list[str]
|
||||
actions: list[str]
|
||||
action_subjects: list[str]
|
||||
visible_text: list[str]
|
||||
spatial_relations: str
|
||||
state_changes: str | None
|
||||
subtitle: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class L1Card:
|
||||
"""L1 场景级语义卡片(不可变)。
|
||||
|
||||
封装 VLM 对一个完整场景的结构化描述输出。
|
||||
|
||||
属性:
|
||||
scene_summary: 场景摘要。
|
||||
main_setting: 主要场景设定(如"室内"、"户外"等)。
|
||||
key_entities: 关键实体列表。
|
||||
main_actions: 主要动作列表。
|
||||
topic_keywords: 主题关键词列表。
|
||||
visible_text: 场景中可见的文字列表。
|
||||
temporal_flow: 时间流描述。
|
||||
"""
|
||||
|
||||
scene_summary: str
|
||||
main_setting: str
|
||||
key_entities: list[str]
|
||||
main_actions: list[str]
|
||||
topic_keywords: list[str]
|
||||
visible_text: list[str]
|
||||
temporal_flow: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 元数据
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class IndexMeta:
|
||||
"""树索引元数据。
|
||||
|
||||
属性:
|
||||
source_path: 原始数据路径(视频文件或文本文件)。
|
||||
modality: 数据模态,"text" 或 "video"。
|
||||
embed_model: 嵌入模型名称(建树时为 None,embed_all 后填充)。
|
||||
embed_dim: 嵌入向量维度(建树时为 None,embed_all 后填充)。
|
||||
created_at: 创建时间(ISO 格式字符串)。
|
||||
"""
|
||||
|
||||
source_path: str
|
||||
modality: str
|
||||
embed_model: str | None = None
|
||||
embed_dim: int | None = None
|
||||
created_at: str = field(default_factory=lambda: datetime.now().isoformat())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 节点数据结构
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class L3Node:
|
||||
"""L3 帧/细节级节点(叶子层)。
|
||||
|
||||
代表最细粒度的语义单元,对应一个具体的帧描述。
|
||||
|
||||
属性:
|
||||
id: 节点唯一标识。
|
||||
card: 帧级语义卡片(VLM 结构化输出)。
|
||||
embedding: 文本嵌入向量,形状 [D],float32。
|
||||
timestamp: 对应的时间戳(秒,可选)。
|
||||
frame_path: 关联的帧图像路径(可选,仅视频模态)。
|
||||
"""
|
||||
|
||||
id: str
|
||||
card: L3Card
|
||||
embedding: np.ndarray | None = None
|
||||
timestamp: float | None = None
|
||||
frame_path: str | None = None
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
"""帧描述文本(取自 card.frame_summary)。"""
|
||||
return self.card.frame_summary
|
||||
|
||||
|
||||
@dataclass
|
||||
class L2Node:
|
||||
"""L2 片段级语义节点(中间层)。
|
||||
|
||||
连接 L1 宏观叙事与 L3 细节描述。
|
||||
|
||||
属性:
|
||||
id: 节点唯一标识。
|
||||
card: 事件级语义卡片(VLM 结构化输出)。
|
||||
embedding: 文本嵌入向量,形状 [D],float32。
|
||||
time_range: 时间范围 (start, end)(秒,可选)。
|
||||
children: 所属的 L3 子节点列表。
|
||||
"""
|
||||
|
||||
id: str
|
||||
card: L2Card
|
||||
embedding: np.ndarray | None = None
|
||||
time_range: tuple[float, float] | None = None
|
||||
children: list[L3Node] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
"""事件描述文本(取自 card.event_description)。"""
|
||||
return self.card.event_description
|
||||
|
||||
|
||||
@dataclass
|
||||
class L1Node:
|
||||
"""L1 全局叙事节点(根层)。
|
||||
|
||||
代表最粗粒度的语义单元,包含宏观场景摘要。
|
||||
|
||||
属性:
|
||||
id: 节点唯一标识。
|
||||
card: 场景级语义卡片(VLM 结构化输出)。
|
||||
embedding: 文本嵌入向量,形状 [D],float32。
|
||||
time_range: 时间范围 (start, end)(秒,可选)。
|
||||
children: 所属的 L2 子节点列表。
|
||||
"""
|
||||
|
||||
id: str
|
||||
card: L1Card
|
||||
embedding: np.ndarray | None = None
|
||||
time_range: tuple[float, float] | None = None
|
||||
children: list[L2Node] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def summary(self) -> str:
|
||||
"""场景摘要文本(取自 card.scene_summary)。"""
|
||||
return self.card.scene_summary
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# JSON 辅助方法(单个 L1 段的轻量序列化)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def to_dict(self, include_embedding: bool = False) -> dict[str, Any]:
|
||||
"""将当前 L1 节点(及其全部 L2/L3 子树)序列化为纯 dict。
|
||||
|
||||
参数:
|
||||
include_embedding: 若 True,将 embedding 向量序列化为 base64 字符串。
|
||||
|
||||
返回:
|
||||
包含 id/card/time_range/children 的字典,可选包含 embedding。
|
||||
"""
|
||||
|
||||
def l3_to_dict(n: L3Node) -> dict[str, Any]:
|
||||
d: dict[str, Any] = {
|
||||
"id": n.id,
|
||||
"card": {
|
||||
"frame_summary": n.card.frame_summary,
|
||||
"visible_entities": n.card.visible_entities,
|
||||
"ongoing_actions": n.card.ongoing_actions,
|
||||
"visible_text": n.card.visible_text,
|
||||
"spatial_layout": n.card.spatial_layout,
|
||||
"visual_attributes": n.card.visual_attributes,
|
||||
"subtitle": n.card.subtitle,
|
||||
},
|
||||
"timestamp": n.timestamp,
|
||||
"frame_path": n.frame_path,
|
||||
}
|
||||
if include_embedding:
|
||||
d["embedding"] = _embed_to_str(n.embedding)
|
||||
return d
|
||||
|
||||
def l2_to_dict(n: L2Node) -> dict[str, Any]:
|
||||
d: dict[str, Any] = {
|
||||
"id": n.id,
|
||||
"card": {
|
||||
"event_description": n.card.event_description,
|
||||
"entities": n.card.entities,
|
||||
"actions": n.card.actions,
|
||||
"action_subjects": n.card.action_subjects,
|
||||
"visible_text": n.card.visible_text,
|
||||
"spatial_relations": n.card.spatial_relations,
|
||||
"state_changes": n.card.state_changes,
|
||||
"subtitle": n.card.subtitle,
|
||||
},
|
||||
"time_range": list(n.time_range) if n.time_range else None,
|
||||
"children": [l3_to_dict(c) for c in n.children],
|
||||
}
|
||||
if include_embedding:
|
||||
d["embedding"] = _embed_to_str(n.embedding)
|
||||
return d
|
||||
|
||||
d: dict[str, Any] = {
|
||||
"id": self.id,
|
||||
"card": {
|
||||
"scene_summary": self.card.scene_summary,
|
||||
"main_setting": self.card.main_setting,
|
||||
"key_entities": self.card.key_entities,
|
||||
"main_actions": self.card.main_actions,
|
||||
"topic_keywords": self.card.topic_keywords,
|
||||
"visible_text": self.card.visible_text,
|
||||
"temporal_flow": self.card.temporal_flow,
|
||||
},
|
||||
"time_range": list(self.time_range) if self.time_range else None,
|
||||
"children": [l2_to_dict(c) for c in self.children],
|
||||
}
|
||||
if include_embedding:
|
||||
d["embedding"] = _embed_to_str(self.embedding)
|
||||
return d
|
||||
|
||||
@staticmethod
|
||||
def from_dict(d: dict[str, Any]) -> L1Node:
|
||||
"""从 dict 反序列化单个 L1 节点(支持 embedding 恢复)。
|
||||
|
||||
参数:
|
||||
d: to_dict() 输出的字典,可包含 embedding 字段。
|
||||
|
||||
返回:
|
||||
L1Node 实例(embedding 自动从 base64 恢复,若无则为 None)。
|
||||
"""
|
||||
l2_nodes: list[L2Node] = []
|
||||
for l2d in d.get("children", []):
|
||||
l3_nodes: list[L3Node] = []
|
||||
for l3d in l2d.get("children", []):
|
||||
# 向后兼容:旧格式 subtitle 在节点级,新格式在 card 内
|
||||
l3_subtitle = l3d["card"].get("subtitle", "") or l3d.get("subtitle", "") or ""
|
||||
l3_card = L3Card(
|
||||
frame_summary=l3d["card"]["frame_summary"],
|
||||
visible_entities=l3d["card"]["visible_entities"],
|
||||
ongoing_actions=l3d["card"]["ongoing_actions"],
|
||||
visible_text=l3d["card"]["visible_text"],
|
||||
spatial_layout=l3d["card"]["spatial_layout"],
|
||||
visual_attributes=l3d["card"]["visual_attributes"],
|
||||
subtitle=l3_subtitle,
|
||||
)
|
||||
l3_nodes.append(
|
||||
L3Node(
|
||||
id=l3d["id"],
|
||||
card=l3_card,
|
||||
embedding=_embed_from_str(l3d.get("embedding")),
|
||||
timestamp=l3d.get("timestamp"),
|
||||
frame_path=l3d.get("frame_path"),
|
||||
)
|
||||
)
|
||||
l2_card = L2Card(
|
||||
event_description=l2d["card"]["event_description"],
|
||||
entities=l2d["card"]["entities"],
|
||||
actions=l2d["card"]["actions"],
|
||||
action_subjects=l2d["card"]["action_subjects"],
|
||||
visible_text=l2d["card"]["visible_text"],
|
||||
spatial_relations=l2d["card"]["spatial_relations"],
|
||||
state_changes=l2d["card"]["state_changes"],
|
||||
subtitle=l2d["card"].get("subtitle", ""),
|
||||
)
|
||||
# 旧格式兼容:L2 无 card subtitle 时从 L3 children 聚合
|
||||
if not l2_card.subtitle:
|
||||
l3_subs = [n.card.subtitle for n in l3_nodes if n.card.subtitle]
|
||||
if l3_subs:
|
||||
l2_card = dataclasses.replace(l2_card, subtitle="\n".join(l3_subs))
|
||||
tr2 = l2d.get("time_range")
|
||||
l2_nodes.append(
|
||||
L2Node(
|
||||
id=l2d["id"],
|
||||
card=l2_card,
|
||||
embedding=_embed_from_str(l2d.get("embedding")),
|
||||
time_range=tuple(tr2) if tr2 else None,
|
||||
children=l3_nodes,
|
||||
)
|
||||
)
|
||||
l1_card = L1Card(
|
||||
scene_summary=d["card"]["scene_summary"],
|
||||
main_setting=d["card"]["main_setting"],
|
||||
key_entities=d["card"]["key_entities"],
|
||||
main_actions=d["card"]["main_actions"],
|
||||
topic_keywords=d["card"]["topic_keywords"],
|
||||
visible_text=d["card"]["visible_text"],
|
||||
temporal_flow=d["card"]["temporal_flow"],
|
||||
)
|
||||
tr1 = d.get("time_range")
|
||||
return L1Node(
|
||||
id=d["id"],
|
||||
card=l1_card,
|
||||
embedding=_embed_from_str(d.get("embedding")),
|
||||
time_range=tuple(tr1) if tr1 else None,
|
||||
children=l2_nodes,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 树索引容器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class TreeIndex:
|
||||
"""三层树索引容器。
|
||||
|
||||
组织和管理三层节点结构,提供嵌入矩阵提取、节点访问、
|
||||
以及 JSON 序列化/反序列化接口。
|
||||
|
||||
典型工作流::
|
||||
|
||||
# 1. 构建索引
|
||||
index = TreeIndex(metadata=meta, roots=[l1_node_1, l1_node_2])
|
||||
|
||||
# 2. 批量 embed(首次检索前)
|
||||
index.embed_all(embed_fn, "model-name", 768)
|
||||
|
||||
# 3. 提取嵌入矩阵(用于检索)
|
||||
M_L1 = index.l1_embeddings()
|
||||
M_L2 = index.l2_embeddings_of(l1_idx=0)
|
||||
M_L3 = index.l3_embeddings_of(0, 1)
|
||||
|
||||
# 4. 序列化
|
||||
index.save_json("cache/my_index.json")
|
||||
loaded = TreeIndex.load_json("cache/my_index.json")
|
||||
|
||||
属性:
|
||||
metadata: 索引元数据。
|
||||
roots: L1 节点列表。
|
||||
"""
|
||||
|
||||
metadata: IndexMeta
|
||||
roots: list[L1Node] = field(default_factory=list)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 嵌入状态检查
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
@property
|
||||
def is_embedded(self) -> bool:
|
||||
"""检查所有节点是否已填充嵌入向量。
|
||||
|
||||
返回:
|
||||
True 表示所有 L1/L2/L3 节点的 embedding 均非 None;
|
||||
False 表示尚未 embed。
|
||||
"""
|
||||
for l1 in self.roots:
|
||||
if l1.embedding is None:
|
||||
return False
|
||||
for l2 in l1.children:
|
||||
if l2.embedding is None:
|
||||
return False
|
||||
for l3 in l2.children:
|
||||
if l3.embedding is None:
|
||||
return False
|
||||
return True
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 批量嵌入
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def embed_all(
|
||||
self,
|
||||
embed_fn: Callable[[str | list[str]], np.ndarray],
|
||||
model_name: str,
|
||||
embed_dim: int,
|
||||
) -> None:
|
||||
"""对所有节点批量执行 embedding,更新 metadata。
|
||||
|
||||
建树阶段不调用此方法(embedding=None)。
|
||||
首次检索前由 Pipeline 调用,结果缓存在节点上。
|
||||
|
||||
参数:
|
||||
embed_fn: EmbeddingModel.embed 方法,接受 str 或 List[str],
|
||||
返回 [N, D] ndarray。
|
||||
model_name: 嵌入模型名称,写入 metadata。
|
||||
embed_dim: 嵌入维度,写入 metadata。
|
||||
|
||||
实现细节:
|
||||
- L3 节点按 L2 分组批量 embed(一次调用),减少 API 开销。
|
||||
- L1/L2 各单独 embed(数量少,不值得合并)。
|
||||
- 仅对 embedding 为 None 的节点执行(支持增量更新)。
|
||||
"""
|
||||
assert len(self.roots) > 0, "embed_all: 树为空,无节点可 embed"
|
||||
for l1 in self.roots:
|
||||
if l1.embedding is None:
|
||||
l1.embedding = embed_fn(l1.summary)[0].astype(np.float32)
|
||||
for l2 in l1.children:
|
||||
self._embed_l2_subtree(l2, embed_fn)
|
||||
self.metadata.embed_model = model_name
|
||||
self.metadata.embed_dim = embed_dim
|
||||
logger.info(
|
||||
"embed_all 完成",
|
||||
model=model_name,
|
||||
embed_dim=embed_dim,
|
||||
)
|
||||
|
||||
def _embed_l2_subtree(
|
||||
self,
|
||||
l2: L2Node,
|
||||
embed_fn: Callable[[str | list[str]], np.ndarray],
|
||||
) -> None:
|
||||
"""对单个 L2 节点及其 L3 子节点执行 embedding(仅处理 embedding 为 None 的节点)。
|
||||
|
||||
参数:
|
||||
l2: 待 embed 的 L2 节点。
|
||||
embed_fn: EmbeddingModel.embed 方法,接受 str 或 List[str],
|
||||
返回 [N, D] ndarray。
|
||||
"""
|
||||
if l2.embedding is None:
|
||||
l2.embedding = embed_fn(l2.description)[0].astype(np.float32)
|
||||
# L3 批量 embed
|
||||
need_embed = [l3 for l3 in l2.children if l3.embedding is None]
|
||||
if need_embed:
|
||||
texts = [l3.description for l3 in need_embed]
|
||||
embs = embed_fn(texts).astype(np.float32) # [N, D]
|
||||
for l3, emb in zip(need_embed, embs, strict=True):
|
||||
l3.embedding = emb
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 嵌入矩阵提取
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def l1_embeddings(self) -> np.ndarray:
|
||||
"""返回所有 L1 节点的嵌入矩阵。
|
||||
|
||||
返回:
|
||||
形状 [N1, D] 的 float32 矩阵。空树返回 [0, D]。
|
||||
|
||||
异常:
|
||||
AssertionError: 节点 embedding 尚未计算(请先调用 embed_all)。
|
||||
"""
|
||||
assert self.is_embedded, "L1 embedding 尚未计算,请先调用 tree.embed_all()"
|
||||
if not self.roots:
|
||||
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
|
||||
return np.stack([r.embedding for r in self.roots], axis=0).astype(np.float32)
|
||||
|
||||
def l2_embeddings_of(self, l1_idx: int) -> np.ndarray:
|
||||
"""返回指定 L1 节点下所有 L2 子节点的嵌入矩阵。
|
||||
|
||||
参数:
|
||||
l1_idx: L1 节点索引。
|
||||
|
||||
返回:
|
||||
形状 [N2, D] 的 float32 矩阵。
|
||||
|
||||
异常:
|
||||
IndexError: l1_idx 越界。
|
||||
AssertionError: embedding 尚未计算。
|
||||
"""
|
||||
assert self.is_embedded, "L2 embedding 尚未计算,请先调用 tree.embed_all()"
|
||||
if not (0 <= l1_idx < len(self.roots)):
|
||||
raise IndexError(f"l1_idx={l1_idx} 越界,L1 节点数={len(self.roots)}")
|
||||
children = self.roots[l1_idx].children
|
||||
if not children:
|
||||
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
|
||||
return np.stack([c.embedding for c in children], axis=0).astype(np.float32)
|
||||
|
||||
def l3_embeddings_of(self, l1_idx: int, l2_idx: int) -> np.ndarray:
|
||||
"""返回指定 L2 节点下所有 L3 子节点的嵌入矩阵。
|
||||
|
||||
参数:
|
||||
l1_idx: L1 节点索引。
|
||||
l2_idx: L2 节点索引(相对于 L1)。
|
||||
|
||||
返回:
|
||||
形状 [N3, D] 的 float32 矩阵。
|
||||
|
||||
异常:
|
||||
IndexError: 索引越界。
|
||||
AssertionError: embedding 尚未计算。
|
||||
"""
|
||||
assert self.is_embedded, "L3 embedding 尚未计算,请先调用 tree.embed_all()"
|
||||
if not (0 <= l1_idx < len(self.roots)):
|
||||
raise IndexError(f"l1_idx={l1_idx} 越界,L1 节点数={len(self.roots)}")
|
||||
l2_children = self.roots[l1_idx].children
|
||||
if not (0 <= l2_idx < len(l2_children)):
|
||||
raise IndexError(f"l2_idx={l2_idx} 越界,L2 节点数={len(l2_children)}")
|
||||
l3_children = l2_children[l2_idx].children
|
||||
if not l3_children:
|
||||
return np.zeros((0, self.metadata.embed_dim), dtype=np.float32)
|
||||
return np.stack([c.embedding for c in l3_children], axis=0).astype(np.float32)
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 节点访问
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def get_node(self, l1: int, l2: int, l3: int) -> L3Node:
|
||||
"""按三级路径索引获取 L3 节点。
|
||||
|
||||
参数:
|
||||
l1: L1 节点索引。
|
||||
l2: L2 节点索引。
|
||||
l3: L3 节点索引。
|
||||
|
||||
返回:
|
||||
目标 L3Node。
|
||||
|
||||
异常:
|
||||
IndexError: 任意层级索引越界。
|
||||
"""
|
||||
if l1 < 0 or l1 >= len(self.roots):
|
||||
raise IndexError(f"l1={l1} 越界,L1 节点数={len(self.roots)}")
|
||||
l2_children = self.roots[l1].children
|
||||
if l2 < 0 or l2 >= len(l2_children):
|
||||
raise IndexError(f"l2={l2} 越界,L2 节点数={len(l2_children)}")
|
||||
l3_children = l2_children[l2].children
|
||||
if l3 < 0 or l3 >= len(l3_children):
|
||||
raise IndexError(f"l3={l3} 越界,L3 节点数={len(l3_children)}")
|
||||
return l3_children[l3]
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# JSON 序列化
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def to_dict(self, include_embedding: bool = False) -> dict[str, Any]:
|
||||
"""将树索引序列化为纯 Python dict。
|
||||
|
||||
参数:
|
||||
include_embedding: 若 True,将所有节点的 embedding 向量序列化为 base64。
|
||||
|
||||
返回:
|
||||
可直接 json.dump 的字典,结构为 {metadata, roots[...]}。
|
||||
"""
|
||||
metadata_dict: dict[str, Any] = {
|
||||
"source_path": self.metadata.source_path,
|
||||
"modality": self.metadata.modality,
|
||||
"created_at": self.metadata.created_at,
|
||||
}
|
||||
if include_embedding:
|
||||
metadata_dict["embed_model"] = self.metadata.embed_model
|
||||
metadata_dict["embed_dim"] = self.metadata.embed_dim
|
||||
|
||||
return {
|
||||
"metadata": metadata_dict,
|
||||
"roots": [r.to_dict(include_embedding=include_embedding) for r in self.roots],
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict[str, Any]) -> TreeIndex:
|
||||
"""从 dict 反序列化为 TreeIndex(支持 embedding 恢复)。
|
||||
|
||||
参数:
|
||||
d: to_dict() 的输出或等价结构,可包含 embedding 字段。
|
||||
|
||||
返回:
|
||||
TreeIndex 实例。
|
||||
|
||||
异常:
|
||||
ValueError: 存在重复的节点 ID。
|
||||
"""
|
||||
meta = IndexMeta(
|
||||
source_path=d["metadata"]["source_path"],
|
||||
modality=d["metadata"]["modality"],
|
||||
embed_model=d["metadata"].get("embed_model"),
|
||||
embed_dim=d["metadata"].get("embed_dim"),
|
||||
created_at=d["metadata"].get("created_at", datetime.now().isoformat()),
|
||||
)
|
||||
|
||||
roots: list[L1Node] = []
|
||||
for r in d["roots"]:
|
||||
roots.append(L1Node.from_dict(r))
|
||||
|
||||
obj = cls(metadata=meta, roots=roots)
|
||||
obj._validate_id_uniqueness()
|
||||
return obj
|
||||
|
||||
def _validate_id_uniqueness(self) -> None:
|
||||
"""校验树中所有节点 ID 的唯一性。
|
||||
|
||||
异常:
|
||||
ValueError: 存在重复的节点 ID。
|
||||
"""
|
||||
seen: set[str] = set()
|
||||
for l1 in self.roots:
|
||||
if l1.id in seen:
|
||||
raise ValueError(f"重复的节点 ID: {l1.id}")
|
||||
seen.add(l1.id)
|
||||
for l2 in l1.children:
|
||||
if l2.id in seen:
|
||||
raise ValueError(f"重复的节点 ID: {l2.id}")
|
||||
seen.add(l2.id)
|
||||
for l3 in l2.children:
|
||||
if l3.id in seen:
|
||||
raise ValueError(f"重复的节点 ID: {l3.id}")
|
||||
seen.add(l3.id)
|
||||
|
||||
def save_json(self, path: str, include_embedding: bool = False) -> None:
|
||||
"""将树索引以 JSON 格式保存到磁盘。
|
||||
|
||||
参数:
|
||||
path: 保存文件路径(推荐 .json 后缀)。
|
||||
include_embedding: 若 True,将所有节点的 embedding 向量保存到 JSON。
|
||||
"""
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(
|
||||
self.to_dict(include_embedding=include_embedding),
|
||||
f,
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
logger.info(
|
||||
"树索引(JSON)已保存至 {}",
|
||||
path,
|
||||
n_l1=len(self.roots),
|
||||
include_embedding=include_embedding,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def load_json(cls, path: str) -> TreeIndex:
|
||||
"""从 JSON 文件加载树索引(自动检测并恢复 embedding)。
|
||||
|
||||
参数:
|
||||
path: JSON 文件路径。
|
||||
|
||||
返回:
|
||||
TreeIndex 实例。若 JSON 中包含 embedding 字段,自动反序列化填充;
|
||||
否则 embedding=None(向后兼容旧格式)。
|
||||
|
||||
异常:
|
||||
FileNotFoundError: 文件不存在。
|
||||
ValueError: 存在重复的节点 ID。
|
||||
"""
|
||||
with open(path, encoding="utf-8") as f:
|
||||
d = json.load(f)
|
||||
obj = cls.from_dict(d)
|
||||
obj._validate_id_uniqueness()
|
||||
logger.info(
|
||||
"树索引(JSON)已从 {} 加载",
|
||||
path,
|
||||
n_l1=len(obj.roots),
|
||||
is_embedded=obj.is_embedded,
|
||||
)
|
||||
return obj
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 单 L1 段的轻量序列化(用于断点续跑)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def save_l1_json(path: str, l1_node: L1Node) -> None:
|
||||
"""将单个 L1 节点(及其子树)以 JSON 形式保存到磁盘。
|
||||
|
||||
参数:
|
||||
path: 目标文件路径。
|
||||
l1_node: 待序列化的 L1 节点。
|
||||
"""
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(l1_node.to_dict(), f, ensure_ascii=False, indent=2)
|
||||
logger.info("L1 中间结果已保存", path=path, l1_id=l1_node.id)
|
||||
|
||||
|
||||
def load_l1_json(path: str) -> L1Node:
|
||||
"""从 JSON 文件加载单个 L1 节点(embedding=None)。
|
||||
|
||||
参数:
|
||||
path: JSON 文件路径。
|
||||
|
||||
返回:
|
||||
L1Node 实例。
|
||||
"""
|
||||
with open(path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
node = L1Node.from_dict(data)
|
||||
logger.info("L1 中间结果已加载", path=path, l1_id=node.id)
|
||||
return node
|
||||
@@ -0,0 +1,174 @@
|
||||
"""树修复检测器:扫描 TreeIndex 识别缺失/低质量节点。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from app.tree.index import TreeIndex
|
||||
|
||||
# 相邻 L2 片段之间允许的最大时间间隙(秒)
|
||||
_MAX_TIME_GAP_S = 1.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NodeIssue:
|
||||
"""检测到的节点问题。
|
||||
|
||||
参数:
|
||||
node_id: 问题节点 ID。
|
||||
level: 节点层级(1/2/3)。
|
||||
issue_type: 问题类型。
|
||||
details: 详细描述。
|
||||
"""
|
||||
|
||||
node_id: str
|
||||
level: int
|
||||
issue_type: str # "empty_field" | "missing_frame" | "no_children" | "time_gap"
|
||||
details: str
|
||||
|
||||
|
||||
def detect_issues(
|
||||
index: TreeIndex,
|
||||
frames_dir: Path | None = None,
|
||||
) -> list[NodeIssue]:
|
||||
"""扫描树,返回所有问题节点列表。
|
||||
|
||||
检查项:
|
||||
- L3: card 必填字段为空(frame_summary / spatial_layout)
|
||||
- 注: visible_entities / ongoing_actions 为空是合法状态(静物/黑帧),不纳入检测
|
||||
- L3: frame_path 对应文件不存在(需提供 frames_dir)
|
||||
- L2: event_description 为空
|
||||
- L2/L1: children 列表为空
|
||||
- L2: 相邻 clips 时间范围不连续(gap > 1秒)
|
||||
- L1: scene_summary 为空
|
||||
|
||||
参数:
|
||||
index: 待检测的 TreeIndex。
|
||||
frames_dir: 帧文件根目录(可选,提供时检查帧文件存在性)。
|
||||
|
||||
返回:
|
||||
问题列表,按 level 降序(L3 → L2 → L1)排列。
|
||||
"""
|
||||
issues: list[NodeIssue] = []
|
||||
|
||||
for l1 in index.roots:
|
||||
# L1: scene_summary 不为空
|
||||
if not l1.card.scene_summary:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l1.id,
|
||||
level=1,
|
||||
issue_type="empty_field",
|
||||
details="L1 节点字段为空: scene_summary",
|
||||
)
|
||||
)
|
||||
|
||||
# L1: children 不为空
|
||||
if not l1.children:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l1.id,
|
||||
level=1,
|
||||
issue_type="no_children",
|
||||
details="L1 节点无 L2 子节点",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# L2: 相邻 clips 时间间隙检查
|
||||
_check_time_gaps(l1.children, issues)
|
||||
|
||||
for l2 in l1.children:
|
||||
# L2: event_description 不为空
|
||||
if not l2.card.event_description:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l2.id,
|
||||
level=2,
|
||||
issue_type="empty_field",
|
||||
details="L2 节点字段为空: event_description",
|
||||
)
|
||||
)
|
||||
|
||||
# L2: children 不为空
|
||||
if not l2.children:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l2.id,
|
||||
level=2,
|
||||
issue_type="no_children",
|
||||
details="L2 节点无 L3 子节点",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
for l3 in l2.children:
|
||||
# L3: 核心必填字段不为空(visible_entities/ongoing_actions 为空是合法状态)
|
||||
empty_fields: list[str] = []
|
||||
if not l3.card.frame_summary:
|
||||
empty_fields.append("frame_summary")
|
||||
if not l3.card.spatial_layout:
|
||||
empty_fields.append("spatial_layout")
|
||||
if empty_fields:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l3.id,
|
||||
level=3,
|
||||
issue_type="empty_field",
|
||||
details=f"L3 节点字段为空: {', '.join(empty_fields)}",
|
||||
)
|
||||
)
|
||||
|
||||
# L3: frame_path 文件存在性
|
||||
if (
|
||||
frames_dir is not None
|
||||
and l3.frame_path is not None
|
||||
and not (frames_dir / l3.frame_path).exists()
|
||||
):
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=l3.id,
|
||||
level=3,
|
||||
issue_type="missing_frame",
|
||||
details=f"帧文件不存在: {l3.frame_path}",
|
||||
)
|
||||
)
|
||||
|
||||
# 按 level 降序排列(L3=3 → L2=2 → L1=1)
|
||||
issues.sort(key=lambda i: -i.level)
|
||||
|
||||
logger.info("树缺陷检测完成,发现 {} 个问题", len(issues))
|
||||
return issues
|
||||
|
||||
|
||||
def _check_time_gaps(
|
||||
l2_nodes: list,
|
||||
issues: list[NodeIssue],
|
||||
) -> None:
|
||||
"""检查同一 L1 下相邻 L2 节点之间的时间间隙。
|
||||
|
||||
参数:
|
||||
l2_nodes: 同一 L1 节点下的 L2 子节点列表。
|
||||
issues: 问题列表(原地追加)。
|
||||
"""
|
||||
for i in range(len(l2_nodes) - 1):
|
||||
curr = l2_nodes[i]
|
||||
nxt = l2_nodes[i + 1]
|
||||
if curr.time_range is None or nxt.time_range is None:
|
||||
continue
|
||||
gap = nxt.time_range[0] - curr.time_range[1]
|
||||
if gap > _MAX_TIME_GAP_S:
|
||||
issues.append(
|
||||
NodeIssue(
|
||||
node_id=nxt.id,
|
||||
level=2,
|
||||
issue_type="time_gap",
|
||||
details=f"与前一片段间隙 {gap:.1f}s(阈值 {_MAX_TIME_GAP_S}s)",
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,513 @@
|
||||
"""树修复重生成器:VLM 重新描述问题节点 + 底向上级联。
|
||||
|
||||
底向上修复流程:
|
||||
1. 收集需修复的 L3 节点 → VLM 重新描述帧
|
||||
2. 收集受影响的 L2 → LLM 从 L3 children 聚合
|
||||
3. 收集受影响的 L1 → LLM 从 L2 children 聚合
|
||||
|
||||
仅处理 issue_type == "empty_field" 且 level == 3 的问题节点。
|
||||
帧文件不存在时跳过该节点(不中断整体修复流程)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.tree.index import (
|
||||
L1Card,
|
||||
L1Node,
|
||||
L2Card,
|
||||
L2Node,
|
||||
L3Card,
|
||||
L3Node,
|
||||
TreeIndex,
|
||||
)
|
||||
from app.tree.subtitle import extract_subtitle_for_range
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from app.tree.repair.detector import NodeIssue
|
||||
from app.tree.subtitle import SRTEntry
|
||||
from core.protocols import LLMProvider, VLMProvider
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt 常量(与 VideoTreeBuilder 保持一致风格)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_L3_REPAIR_PROMPT = (
|
||||
'该片段的整体内容: "{l2_description}"\n'
|
||||
"用一到两句话描述这帧画面的具体内容。"
|
||||
"重点关注: 动作、物体变化、文字信息、人物表情。\n"
|
||||
"{subtitle_block}"
|
||||
"返回 JSON 对象,包含以下字段:\n"
|
||||
"- frame_summary: 画面描述\n"
|
||||
"- visible_entities: 可见实体列表\n"
|
||||
"- ongoing_actions: 动作列表\n"
|
||||
"- visible_text: 可见文字列表\n"
|
||||
"- spatial_layout: 空间布局\n"
|
||||
'- visual_attributes: {{"lighting": "...", "dominant_colors": [...], "camera_angle": "..."}}\n'
|
||||
"只返回 JSON 对象,不要其他内容。"
|
||||
)
|
||||
|
||||
_L2_REGEN_PROMPT = (
|
||||
"以下是一个视频片段中各帧的描述:\n{l3_texts}\n"
|
||||
"用1-2句话描述该片段的核心内容。\n"
|
||||
"返回 JSON 对象,包含以下字段:\n"
|
||||
"- event_description: 1-2句片段描述\n"
|
||||
"- entities: 可见实体列表\n"
|
||||
"- actions: 动作列表\n"
|
||||
"- action_subjects: 动作主体列表\n"
|
||||
"- visible_text: 画面中可见文字列表\n"
|
||||
"- spatial_relations: 空间关系描述\n"
|
||||
"- state_changes: 状态变化描述(无则 null)\n"
|
||||
"只返回 JSON 对象,不要其他内容。"
|
||||
)
|
||||
|
||||
_L1_REGEN_PROMPT = (
|
||||
"以下是一个视频段落中各片段的描述:\n{l2_texts}\n"
|
||||
"用2-3句话总结该段落的整体内容,涵盖所有片段的主题。\n"
|
||||
"返回 JSON 对象,包含以下字段:\n"
|
||||
"- scene_summary: 2-3句段落摘要\n"
|
||||
"- main_setting: 主要场景\n"
|
||||
"- key_entities: 关键实体列表\n"
|
||||
"- main_actions: 主要动作列表\n"
|
||||
"- topic_keywords: 主题关键词列表\n"
|
||||
"- visible_text: 出现的文字列表\n"
|
||||
"- temporal_flow: 时间流向描述\n"
|
||||
"只返回 JSON 对象,不要其他内容。"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 统计数据类
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class RepairStats:
|
||||
"""修复统计信息。
|
||||
|
||||
属性:
|
||||
l3_repaired: 修复的 L3 节点数。
|
||||
l2_regenerated: 重生成的 L2 节点数。
|
||||
l1_regenerated: 重生成的 L1 节点数。
|
||||
"""
|
||||
|
||||
l3_repaired: int = 0
|
||||
l2_regenerated: int = 0
|
||||
l1_regenerated: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON 解析辅助(复用 VideoTreeBuilder 的解析逻辑)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _extract_json(raw: str) -> Any:
|
||||
"""从 VLM/LLM 原始输出中提取 JSON(处理 markdown 代码块包裹)。
|
||||
|
||||
参数:
|
||||
raw: 原始返回字符串。
|
||||
|
||||
返回:
|
||||
解析后的 Python 对象(dict/list),解析失败返回 None。
|
||||
"""
|
||||
raw = raw.strip()
|
||||
# Phase 1: 尝试提取 markdown 代码块中的 JSON
|
||||
code_match = re.search(
|
||||
r"```(?:json)?\s*([\[{].*?[\]}])\s*```",
|
||||
raw,
|
||||
re.DOTALL,
|
||||
)
|
||||
if code_match:
|
||||
raw = code_match.group(1)
|
||||
|
||||
# Phase 2: 直接解析
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Phase 3: 尝试提取裸 JSON 对象/数组
|
||||
json_match = re.search(r"[\[{].*[\]}]", raw, re.DOTALL)
|
||||
if json_match:
|
||||
try:
|
||||
return json.loads(json_match.group())
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _parse_l3_card(raw: str) -> L3Card | None:
|
||||
"""解析 VLM 输出为 L3Card。解析失败返回 None。
|
||||
|
||||
参数:
|
||||
raw: VLM 原始返回字符串。
|
||||
|
||||
返回:
|
||||
L3Card 实例或 None(解析失败时)。
|
||||
"""
|
||||
data = _extract_json(raw)
|
||||
if isinstance(data, dict):
|
||||
try:
|
||||
return L3Card(
|
||||
frame_summary=str(data["frame_summary"]),
|
||||
visible_entities=list(data["visible_entities"]),
|
||||
ongoing_actions=list(data["ongoing_actions"]),
|
||||
visible_text=list(data["visible_text"]),
|
||||
spatial_layout=str(data["spatial_layout"]),
|
||||
visual_attributes=dict(data["visual_attributes"]),
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _parse_l2_card(raw: str) -> L2Card | None:
|
||||
"""解析 LLM 输出为 L2Card。解析失败返回 None。
|
||||
|
||||
参数:
|
||||
raw: LLM 原始返回字符串。
|
||||
|
||||
返回:
|
||||
L2Card 实例或 None(解析失败时)。
|
||||
"""
|
||||
data = _extract_json(raw)
|
||||
if isinstance(data, dict):
|
||||
try:
|
||||
state_changes = data.get("state_changes")
|
||||
if state_changes is not None:
|
||||
state_changes = str(state_changes)
|
||||
return L2Card(
|
||||
event_description=str(data["event_description"]),
|
||||
entities=list(data["entities"]),
|
||||
actions=list(data["actions"]),
|
||||
action_subjects=list(data["action_subjects"]),
|
||||
visible_text=list(data["visible_text"]),
|
||||
spatial_relations=str(data["spatial_relations"]),
|
||||
state_changes=state_changes,
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _parse_l1_card(raw: str) -> L1Card | None:
|
||||
"""解析 LLM 输出为 L1Card。解析失败返回 None。
|
||||
|
||||
参数:
|
||||
raw: LLM 原始返回字符串。
|
||||
|
||||
返回:
|
||||
L1Card 实例或 None(解析失败时)。
|
||||
"""
|
||||
data = _extract_json(raw)
|
||||
if isinstance(data, dict):
|
||||
try:
|
||||
return L1Card(
|
||||
scene_summary=str(data["scene_summary"]),
|
||||
main_setting=str(data["main_setting"]),
|
||||
key_entities=list(data["key_entities"]),
|
||||
main_actions=list(data["main_actions"]),
|
||||
topic_keywords=list(data["topic_keywords"]),
|
||||
visible_text=list(data["visible_text"]),
|
||||
temporal_flow=str(data["temporal_flow"]),
|
||||
)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 节点查找辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_node_lookup(
|
||||
index: TreeIndex,
|
||||
) -> tuple[
|
||||
dict[str, L3Node],
|
||||
dict[str, L2Node],
|
||||
dict[str, L1Node],
|
||||
dict[str, L2Node],
|
||||
dict[str, L1Node],
|
||||
]:
|
||||
"""构建节点 ID 到节点的查找表 + 子节点到父节点的映射。
|
||||
|
||||
参数:
|
||||
index: 树索引。
|
||||
|
||||
返回:
|
||||
(l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1)
|
||||
- l3_by_id: L3 节点 ID → L3Node
|
||||
- l2_by_id: L2 节点 ID → L2Node
|
||||
- l1_by_id: L1 节点 ID → L1Node
|
||||
- l3_parent_l2: L3 节点 ID → 其父 L2Node
|
||||
- l2_parent_l1: L2 节点 ID → 其父 L1Node
|
||||
"""
|
||||
l3_by_id: dict[str, L3Node] = {}
|
||||
l2_by_id: dict[str, L2Node] = {}
|
||||
l1_by_id: dict[str, L1Node] = {}
|
||||
l3_parent_l2: dict[str, L2Node] = {}
|
||||
l2_parent_l1: dict[str, L1Node] = {}
|
||||
|
||||
for l1 in index.roots:
|
||||
l1_by_id[l1.id] = l1
|
||||
for l2 in l1.children:
|
||||
l2_by_id[l2.id] = l2
|
||||
l2_parent_l1[l2.id] = l1
|
||||
for l3 in l2.children:
|
||||
l3_by_id[l3.id] = l3
|
||||
l3_parent_l2[l3.id] = l2
|
||||
|
||||
return l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 字幕辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_subtitle_block(
|
||||
srt_entries: list[SRTEntry] | None,
|
||||
timestamp: float | None,
|
||||
) -> str:
|
||||
"""构建字幕注入文本块。
|
||||
|
||||
参数:
|
||||
srt_entries: SRT 字幕条目列表。
|
||||
timestamp: 帧时间戳(秒)。
|
||||
|
||||
返回:
|
||||
字幕文本块字符串(无匹配时返回空字符串)。
|
||||
"""
|
||||
if not srt_entries or timestamp is None:
|
||||
return ""
|
||||
window = 2.0
|
||||
start = max(0.0, timestamp - window)
|
||||
end = timestamp + window
|
||||
text = extract_subtitle_for_range(srt_entries, (start, end))
|
||||
if not text:
|
||||
return ""
|
||||
return f"字幕信息:\n{text}\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主修复函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def repair_tree(
|
||||
index: TreeIndex,
|
||||
issues: list[NodeIssue],
|
||||
vlm: VLMProvider,
|
||||
llm: LLMProvider,
|
||||
frames_dir: Path,
|
||||
srt_entries: list[SRTEntry] | None = None,
|
||||
) -> RepairStats:
|
||||
"""修复有问题的节点,底向上级联。
|
||||
|
||||
流程:
|
||||
1. 收集需修复的 L3 节点 → VLM 重新描述帧
|
||||
2. 收集受影响的 L2 → LLM 从 L3 children 聚合
|
||||
3. 收集受影响的 L1 → LLM 从 L2 children 聚合
|
||||
|
||||
参数:
|
||||
index: 待修复的 TreeIndex(原地修改)。
|
||||
issues: detect_issues() 返回的问题列表。
|
||||
vlm: VLM 调用端口。
|
||||
llm: LLM 调用端口。
|
||||
frames_dir: 帧文件根目录。
|
||||
srt_entries: 字幕条目列表(可选)。
|
||||
|
||||
返回:
|
||||
RepairStats 统计。
|
||||
"""
|
||||
stats = RepairStats()
|
||||
|
||||
if not issues:
|
||||
logger.info("无修复任务,跳过")
|
||||
return stats
|
||||
|
||||
# 构建查找表
|
||||
l3_by_id, l2_by_id, l1_by_id, l3_parent_l2, l2_parent_l1 = _build_node_lookup(index)
|
||||
|
||||
# Step 1: 修复 L3 节点(仅处理 empty_field + level 3)
|
||||
l3_issues = [
|
||||
issue for issue in issues if issue.issue_type == "empty_field" and issue.level == 3
|
||||
]
|
||||
|
||||
affected_l2_ids: set[str] = set()
|
||||
|
||||
for issue in l3_issues:
|
||||
l3_node = l3_by_id.get(issue.node_id)
|
||||
if l3_node is None:
|
||||
logger.warning(
|
||||
"L3 节点 ID 未在树中找到,跳过",
|
||||
node_id=issue.node_id,
|
||||
)
|
||||
continue
|
||||
|
||||
# 查找帧文件
|
||||
if l3_node.frame_path is None:
|
||||
logger.warning(
|
||||
"L3 节点无 frame_path,跳过",
|
||||
node_id=issue.node_id,
|
||||
)
|
||||
continue
|
||||
|
||||
frame_file = frames_dir / l3_node.frame_path
|
||||
if not frame_file.exists():
|
||||
logger.warning(
|
||||
"L3 帧文件不存在,跳过修复",
|
||||
node_id=issue.node_id,
|
||||
frame_path=str(frame_file),
|
||||
)
|
||||
continue
|
||||
|
||||
# 获取 L2 父节点描述作为上下文
|
||||
parent_l2 = l3_parent_l2.get(issue.node_id)
|
||||
l2_description = parent_l2.card.event_description if parent_l2 else ""
|
||||
|
||||
# 构建字幕块
|
||||
subtitle_block = _build_subtitle_block(srt_entries, l3_node.timestamp)
|
||||
|
||||
# VLM 重新描述帧
|
||||
prompt = _L3_REPAIR_PROMPT.format(
|
||||
l2_description=l2_description,
|
||||
subtitle_block=subtitle_block,
|
||||
)
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
|
||||
try:
|
||||
response = await vlm.chat_with_images(messages, [str(frame_file)])
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"L3 修复 VLM 调用失败,跳过: {}",
|
||||
exc,
|
||||
node_id=issue.node_id,
|
||||
)
|
||||
continue
|
||||
|
||||
new_card = _parse_l3_card(response.content)
|
||||
if new_card is None:
|
||||
logger.warning(
|
||||
"L3 修复 VLM 输出解析失败,跳过",
|
||||
node_id=issue.node_id,
|
||||
raw_preview=response.content[:200],
|
||||
)
|
||||
continue
|
||||
|
||||
# 原地替换 card(L3Node.card 不是 frozen dataclass 的限制字段)
|
||||
l3_node.card = new_card
|
||||
stats.l3_repaired += 1
|
||||
|
||||
# 标记受影响的 L2 父节点
|
||||
if parent_l2 is not None:
|
||||
affected_l2_ids.add(parent_l2.id)
|
||||
|
||||
logger.debug(
|
||||
"L3 节点修复完成",
|
||||
node_id=issue.node_id,
|
||||
frame_summary=new_card.frame_summary[:50],
|
||||
)
|
||||
|
||||
# Step 2: 重生成受影响的 L2 节点
|
||||
affected_l1_ids: set[str] = set()
|
||||
|
||||
for l2_id in affected_l2_ids:
|
||||
l2_node = l2_by_id.get(l2_id)
|
||||
if l2_node is None:
|
||||
continue
|
||||
|
||||
# 从 L3 children 聚合描述
|
||||
l3_texts = "\n".join(f"- {l3.card.frame_summary}" for l3 in l2_node.children)
|
||||
prompt = _L2_REGEN_PROMPT.format(l3_texts=l3_texts)
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
|
||||
try:
|
||||
response = await llm.chat(messages)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"L2 重生成 LLM 调用失败,跳过: {}",
|
||||
exc,
|
||||
l2_id=l2_id,
|
||||
)
|
||||
continue
|
||||
|
||||
new_card = _parse_l2_card(response.content)
|
||||
if new_card is None:
|
||||
logger.warning(
|
||||
"L2 重生成 LLM 输出解析失败,跳过",
|
||||
l2_id=l2_id,
|
||||
raw_preview=response.content[:200],
|
||||
)
|
||||
continue
|
||||
|
||||
l2_node.card = new_card
|
||||
stats.l2_regenerated += 1
|
||||
|
||||
# 标记受影响的 L1 父节点
|
||||
parent_l1 = l2_parent_l1.get(l2_id)
|
||||
if parent_l1 is not None:
|
||||
affected_l1_ids.add(parent_l1.id)
|
||||
|
||||
logger.debug(
|
||||
"L2 节点重生成完成",
|
||||
l2_id=l2_id,
|
||||
event_description=new_card.event_description[:50],
|
||||
)
|
||||
|
||||
# Step 3: 重生成受影响的 L1 节点
|
||||
for l1_id in affected_l1_ids:
|
||||
l1_node = l1_by_id.get(l1_id)
|
||||
if l1_node is None:
|
||||
continue
|
||||
|
||||
# 从 L2 children 聚合描述
|
||||
l2_texts = "\n".join(f"- {l2.card.event_description}" for l2 in l1_node.children)
|
||||
prompt = _L1_REGEN_PROMPT.format(l2_texts=l2_texts)
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
|
||||
try:
|
||||
response = await llm.chat(messages)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"L1 重生成 LLM 调用失败,跳过: {}",
|
||||
exc,
|
||||
l1_id=l1_id,
|
||||
)
|
||||
continue
|
||||
|
||||
new_card = _parse_l1_card(response.content)
|
||||
if new_card is None:
|
||||
logger.warning(
|
||||
"L1 重生成 LLM 输出解析失败,跳过",
|
||||
l1_id=l1_id,
|
||||
raw_preview=response.content[:200],
|
||||
)
|
||||
continue
|
||||
|
||||
l1_node.card = new_card
|
||||
stats.l1_regenerated += 1
|
||||
|
||||
logger.debug(
|
||||
"L1 节点重生成完成",
|
||||
l1_id=l1_id,
|
||||
scene_summary=new_card.scene_summary[:50],
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"树修复完成",
|
||||
l3_repaired=stats.l3_repaired,
|
||||
l2_regenerated=stats.l2_regenerated,
|
||||
l1_regenerated=stats.l1_regenerated,
|
||||
)
|
||||
return stats
|
||||
@@ -0,0 +1,527 @@
|
||||
"""Q&A 反向补全:基于问题答案分析,将树中缺失的事实注入节点。
|
||||
|
||||
通过 LLM 分析正确答案需要哪些关键事实,再检查树中是否已有,
|
||||
对缺失事实执行注入。仅注入客观事实(人名、地点、得分、物体名称),
|
||||
不注入情感、因果推理、时间推理等主观或高阶信息。
|
||||
|
||||
与 TRM4 的关键差异:
|
||||
- 树结构从扁平 dict 变为 TreeIndex(L1Node → L2Node → L3Node)。
|
||||
- Card 为 frozen dataclass,注入时使用 dataclasses.replace() 创建新实例。
|
||||
- LLMProvider 为异步接口,返回 LLMResponse(.content 获取文本)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, replace
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.tree.index import L1Node, L2Node, L3Node, TreeIndex
|
||||
from core.protocols import LLMProvider
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 允许注入的类别白名单
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ALLOWED_CATEGORIES = frozenset(
|
||||
{
|
||||
"person_name",
|
||||
"location",
|
||||
"score_number",
|
||||
"object_name",
|
||||
}
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 类别 → 默认注入字段映射(L2 Card 字段名)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_CATEGORY_DEFAULT_FIELD: dict[str, str] = {
|
||||
"person_name": "entities",
|
||||
"location": "entities",
|
||||
"score_number": "entities",
|
||||
"object_name": "entities",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 统计
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class SupplementStats:
|
||||
"""反向补全统计信息。
|
||||
|
||||
属性:
|
||||
questions_analyzed: 分析的问题数量。
|
||||
facts_injected: 成功注入的事实数量。
|
||||
facts_skipped: 跳过的事实数量(类别不在白名单中)。
|
||||
"""
|
||||
|
||||
questions_analyzed: int = 0
|
||||
facts_injected: int = 0
|
||||
facts_skipped: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 去重
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def deduplicate_field(values: list[str]) -> list[str]:
|
||||
"""大小写归一化去重,保留首次出现的原始形式。
|
||||
|
||||
参数:
|
||||
values: 待去重字符串列表。
|
||||
|
||||
返回:
|
||||
去重后的列表,保留各值首次出现时的大小写。
|
||||
空字符串和纯空白字符串会被跳过。
|
||||
"""
|
||||
seen: set[str] = set()
|
||||
result: list[str] = []
|
||||
for v in values:
|
||||
s = str(v).strip()
|
||||
key = s.lower()
|
||||
if key and key not in seen:
|
||||
seen.add(key)
|
||||
result.append(s)
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 节点查找
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _find_node_by_id(
|
||||
index: TreeIndex,
|
||||
node_id: str,
|
||||
) -> tuple[L1Node | L2Node | L3Node | None, int]:
|
||||
"""在 TreeIndex 中按 ID 查找节点,返回节点和所属层级。
|
||||
|
||||
参数:
|
||||
index: 树索引。
|
||||
node_id: 目标节点 ID。
|
||||
|
||||
返回:
|
||||
(node, level) 元组。找不到时返回 (None, -1)。
|
||||
level: 1=L1, 2=L2, 3=L3。
|
||||
"""
|
||||
for l1 in index.roots:
|
||||
if l1.id == node_id:
|
||||
return l1, 1
|
||||
for l2 in l1.children:
|
||||
if l2.id == node_id:
|
||||
return l2, 2
|
||||
for l3 in l2.children:
|
||||
if l3.id == node_id:
|
||||
return l3, 3
|
||||
return None, -1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 单值注入(适配 frozen Card)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _inject_into_l2(l2: L2Node, field: str, value: str) -> bool:
|
||||
"""向 L2 节点的 Card 指定字段注入一个值。
|
||||
|
||||
使用 dataclasses.replace() 创建新的 frozen L2Card。
|
||||
仅支持 list[str] 类型字段(entities / actions / action_subjects / visible_text)
|
||||
和 str 类型字段(event_description / spatial_relations / state_changes)。
|
||||
|
||||
参数:
|
||||
l2: L2 节点(card 会被替换为新实例)。
|
||||
field: 目标字段名。
|
||||
value: 要注入的值。
|
||||
|
||||
返回:
|
||||
True 表示实际注入了新内容,False 表示已存在(跳过)。
|
||||
"""
|
||||
card = l2.card
|
||||
current = getattr(card, field, None)
|
||||
|
||||
if current is None:
|
||||
# 字段不存在于 Card schema,跳过
|
||||
logger.debug("L2Card 无字段 {},跳过注入", field)
|
||||
return False
|
||||
|
||||
if isinstance(current, list):
|
||||
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
|
||||
if value.strip().lower() in lower_set:
|
||||
return False
|
||||
new_list = deduplicate_field([*current, value])
|
||||
l2.card = replace(card, **{field: new_list})
|
||||
return True
|
||||
|
||||
if isinstance(current, str):
|
||||
if value.strip().lower() in current.lower():
|
||||
return False
|
||||
new_val = current + "; " + value if current else value
|
||||
l2.card = replace(card, **{field: new_val})
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _inject_into_l3(l3: L3Node, field: str, value: str) -> bool:
|
||||
"""向 L3 节点的 Card 指定字段注入一个值。
|
||||
|
||||
使用 dataclasses.replace() 创建新的 frozen L3Card。
|
||||
|
||||
参数:
|
||||
l3: L3 节点(card 会被替换为新实例)。
|
||||
field: 目标字段名。
|
||||
value: 要注入的值。
|
||||
|
||||
返回:
|
||||
True 表示实际注入了新内容,False 表示已存在(跳过)。
|
||||
"""
|
||||
card = l3.card
|
||||
current = getattr(card, field, None)
|
||||
|
||||
if current is None:
|
||||
logger.debug("L3Card 无字段 {},跳过注入", field)
|
||||
return False
|
||||
|
||||
if isinstance(current, list):
|
||||
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
|
||||
if value.strip().lower() in lower_set:
|
||||
return False
|
||||
new_list = deduplicate_field([*current, value])
|
||||
l3.card = replace(card, **{field: new_list})
|
||||
return True
|
||||
|
||||
if isinstance(current, str):
|
||||
if value.strip().lower() in current.lower():
|
||||
return False
|
||||
new_val = current + "; " + value if current else value
|
||||
l3.card = replace(card, **{field: new_val})
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _inject_into_l1(l1: L1Node, field: str, value: str) -> bool:
|
||||
"""向 L1 节点的 Card 指定字段注入一个值。
|
||||
|
||||
使用 dataclasses.replace() 创建新的 frozen L1Card。
|
||||
|
||||
参数:
|
||||
l1: L1 节点(card 会被替换为新实例)。
|
||||
field: 目标字段名。
|
||||
value: 要注入的值。
|
||||
|
||||
返回:
|
||||
True 表示实际注入了新内容,False 表示已存在(跳过)。
|
||||
"""
|
||||
card = l1.card
|
||||
current = getattr(card, field, None)
|
||||
|
||||
if current is None:
|
||||
logger.debug("L1Card 无字段 {},跳过注入", field)
|
||||
return False
|
||||
|
||||
if isinstance(current, list):
|
||||
lower_set = {v.strip().lower() for v in current if isinstance(v, str)}
|
||||
if value.strip().lower() in lower_set:
|
||||
return False
|
||||
new_list = deduplicate_field([*current, value])
|
||||
l1.card = replace(card, **{field: new_list})
|
||||
return True
|
||||
|
||||
if isinstance(current, str):
|
||||
if value.strip().lower() in current.lower():
|
||||
return False
|
||||
new_val = current + "; " + value if current else value
|
||||
l1.card = replace(card, **{field: new_val})
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 批量注入
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def apply_injections(index: TreeIndex, injections: list[dict[str, Any]]) -> SupplementStats:
|
||||
"""执行一组注入指令,将事实写入树节点 Card。
|
||||
|
||||
每条指令格式::
|
||||
|
||||
{
|
||||
"category": "person_name" | "location" | "score_number" | "object_name",
|
||||
"inject_value": "...",
|
||||
"targets": [{"node_id": "...", "field": "..."}, ...]
|
||||
}
|
||||
|
||||
向后兼容: 若无 targets,读取 target_node_id + target_field 构造单目标。
|
||||
|
||||
参数:
|
||||
index: TreeIndex 实例(节点 Card 会被替换为新实例)。
|
||||
injections: 注入指令列表。
|
||||
|
||||
返回:
|
||||
注入统计信息。
|
||||
"""
|
||||
stats = SupplementStats()
|
||||
|
||||
for instr in injections:
|
||||
category = instr.get("category", "")
|
||||
if category not in _ALLOWED_CATEGORIES:
|
||||
logger.debug("拒绝非法类别: {}", category)
|
||||
stats.facts_skipped += 1
|
||||
continue
|
||||
|
||||
inject_value = str(instr.get("inject_value", "")).strip()
|
||||
if not inject_value:
|
||||
stats.facts_skipped += 1
|
||||
continue
|
||||
|
||||
# 解析目标列表(兼容新旧格式)
|
||||
targets = instr.get("targets")
|
||||
if not targets:
|
||||
node_id = instr.get("target_node_id", "")
|
||||
field = instr.get("target_field", "")
|
||||
if node_id and field:
|
||||
targets = [{"node_id": node_id, "field": field}]
|
||||
else:
|
||||
stats.facts_skipped += 1
|
||||
continue
|
||||
|
||||
for target in targets:
|
||||
node_id = target.get("node_id", "")
|
||||
field = target.get("field", "")
|
||||
node, level = _find_node_by_id(index, node_id)
|
||||
|
||||
if node is None:
|
||||
logger.debug("跳过不存在的节点: {}", node_id)
|
||||
stats.facts_skipped += 1
|
||||
continue
|
||||
|
||||
injected = False
|
||||
if level == 1:
|
||||
injected = _inject_into_l1(node, field, inject_value) # type: ignore[arg-type]
|
||||
elif level == 2:
|
||||
injected = _inject_into_l2(node, field, inject_value) # type: ignore[arg-type]
|
||||
elif level == 3:
|
||||
injected = _inject_into_l3(node, field, inject_value) # type: ignore[arg-type]
|
||||
|
||||
if injected:
|
||||
stats.facts_injected += 1
|
||||
else:
|
||||
stats.facts_skipped += 1
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM Prompt
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SUPPLEMENT_SYSTEM_PROMPT = """\
|
||||
你是一个视频内容分析专家。你的任务是分析回答某个问题需要哪些关键事实,
|
||||
并判断这些事实是否已存在于视频树的摘要中。
|
||||
|
||||
## 输出规则
|
||||
|
||||
1. 只输出**客观事实**,包括以下四类:
|
||||
- person_name: 人物姓名
|
||||
- location: 地点名称
|
||||
- score_number: 比分、数字
|
||||
- object_name: 关键物体名称
|
||||
|
||||
2. **不要**输出以下类型:
|
||||
- 情感、态度、心情
|
||||
- 因果推理("因为…所以…")
|
||||
- 时间顺序推理("先…后…")
|
||||
- 主观评价
|
||||
|
||||
3. 对于 person_name 类别,输出 targets 数组包含两个写入点:
|
||||
- L2 节点的 entities 字段
|
||||
- L3 节点的 visible_entities 字段
|
||||
其他类别只写入最相关的单个节点的 entities 字段。
|
||||
|
||||
4. 每条 missing fact 必须包含 inject_value(要注入的值)和 targets 数组。
|
||||
|
||||
## 输出格式 (严格 JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"needed_facts": [
|
||||
{"category": "person_name", "value": "..."}
|
||||
],
|
||||
"found_in_tree": [
|
||||
{"category": "person_name", "value": "...", "found_at": "node_id"}
|
||||
],
|
||||
"missing_facts": [
|
||||
{
|
||||
"category": "person_name",
|
||||
"inject_value": "...",
|
||||
"targets": [
|
||||
{"node_id": "...", "field": "entities"},
|
||||
{"node_id": "...", "field": "visible_entities"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
只输出 JSON,不要输出其他内容。
|
||||
"""
|
||||
|
||||
|
||||
def _build_user_prompt(
|
||||
question: dict[str, Any],
|
||||
index: TreeIndex,
|
||||
srt_text: str,
|
||||
) -> str:
|
||||
"""构建 supplement 分析的 user prompt。
|
||||
|
||||
包含: 问题 + 选项 + 正确答案 + 树 L2 摘要 + SRT 字幕(截断至 3000 字符)。
|
||||
|
||||
参数:
|
||||
question: 包含 question/options/answer 的字典。
|
||||
index: TreeIndex 实例。
|
||||
srt_text: SRT 字幕文本。
|
||||
|
||||
返回:
|
||||
拼装后的 user prompt 字符串。
|
||||
"""
|
||||
# 问题部分
|
||||
q_text = question.get("question", "")
|
||||
options = question.get("options", [])
|
||||
answer = question.get("answer", "")
|
||||
options_str = "\n".join(f" {chr(65 + i)}. {opt}" for i, opt in enumerate(options))
|
||||
|
||||
# 树 L2 摘要(从 TreeIndex 结构中提取)
|
||||
l2_summaries: list[str] = []
|
||||
for l1 in index.roots:
|
||||
for l2 in l1.children:
|
||||
description = l2.card.event_description
|
||||
entities_str = ", ".join(l2.card.entities) if l2.card.entities else ""
|
||||
time_str = ""
|
||||
if l2.time_range:
|
||||
time_str = f"{l2.time_range[0]:.1f}-{l2.time_range[1]:.1f}s: "
|
||||
l2_summaries.append(
|
||||
f"[{l2.id}] {time_str}{description}"
|
||||
+ (f" | entities: {entities_str}" if entities_str else "")
|
||||
)
|
||||
|
||||
l2_block = "\n".join(l2_summaries) if l2_summaries else "(无 L2 摘要)"
|
||||
|
||||
# SRT 截断
|
||||
srt_truncated = srt_text[:3000] if srt_text else "(无字幕)"
|
||||
|
||||
return (
|
||||
f"## 问题\n{q_text}\n\n"
|
||||
f"## 选项\n{options_str}\n\n"
|
||||
f"## 正确答案\n{answer}\n\n"
|
||||
f"## 视频树 L2 摘要\n{l2_block}\n\n"
|
||||
f"## 字幕 (前 3000 字符)\n{srt_truncated}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM 调用
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def analyze_question(
|
||||
llm: LLMProvider,
|
||||
question: dict[str, Any],
|
||||
index: TreeIndex,
|
||||
srt_text: str,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""调用 LLM 分析单个问题,返回需要注入的事实列表。
|
||||
|
||||
参数:
|
||||
llm: LLMProvider 实例(异步接口)。
|
||||
question: 问题字典(含 question/options/answer)。
|
||||
index: TreeIndex 实例。
|
||||
srt_text: SRT 字幕文本。
|
||||
|
||||
返回:
|
||||
missing_facts 列表,每项含 category / inject_value / targets。
|
||||
解析失败时返回空列表。
|
||||
"""
|
||||
user_prompt = _build_user_prompt(question, index, srt_text)
|
||||
messages = [
|
||||
{"role": "system", "content": _SUPPLEMENT_SYSTEM_PROMPT},
|
||||
{"role": "user", "content": user_prompt},
|
||||
]
|
||||
|
||||
response = await llm.chat(messages)
|
||||
raw = response.content
|
||||
|
||||
# 提取 JSON(兼容 markdown 代码块包裹)
|
||||
text = raw.strip()
|
||||
if text.startswith("```"):
|
||||
lines = text.split("\n")
|
||||
lines = [ln for ln in lines if not ln.strip().startswith("```")]
|
||||
text = "\n".join(lines)
|
||||
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("supplement LLM 返回非法 JSON,跳过。原始内容: {}", raw[:200])
|
||||
return []
|
||||
|
||||
missing = parsed.get("missing_facts", [])
|
||||
if not isinstance(missing, list):
|
||||
logger.warning("missing_facts 不是列表,跳过")
|
||||
return []
|
||||
|
||||
return missing
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主入口
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def supplement_tree(
|
||||
index: TreeIndex,
|
||||
questions: list[dict[str, Any]],
|
||||
llm: LLMProvider,
|
||||
srt_text: str = "",
|
||||
) -> SupplementStats:
|
||||
"""对树索引执行 Q&A 反向补全:遍历问题,分析缺失事实,注入节点。
|
||||
|
||||
参数:
|
||||
index: TreeIndex 实例(节点 Card 会被就地替换)。
|
||||
questions: 问题列表,每项含 question/options/answer。
|
||||
llm: LLMProvider 实例(异步接口)。
|
||||
srt_text: SRT 字幕文本(可选,默认空字符串)。
|
||||
|
||||
返回:
|
||||
补全统计信息。
|
||||
"""
|
||||
all_injections: list[dict[str, Any]] = []
|
||||
|
||||
for i, question in enumerate(questions):
|
||||
logger.debug(
|
||||
"supplement: 分析问题 {}/{}",
|
||||
i + 1,
|
||||
len(questions),
|
||||
)
|
||||
missing = await analyze_question(llm, question, index, srt_text)
|
||||
all_injections.extend(missing)
|
||||
|
||||
stats = apply_injections(index, all_injections)
|
||||
stats.questions_analyzed = len(questions)
|
||||
|
||||
logger.info(
|
||||
"supplement_tree 完成: questions={} injections={} injected={} skipped={}",
|
||||
len(questions),
|
||||
len(all_injections),
|
||||
stats.facts_injected,
|
||||
stats.facts_skipped,
|
||||
)
|
||||
return stats
|
||||
@@ -0,0 +1,320 @@
|
||||
"""字幕模块:SRT 解析、完整性检查、时间范围提取、Voronoi 分配。
|
||||
|
||||
提供四个核心函数:
|
||||
- parse_srt: 解析 SRT 文件为结构化条目列表
|
||||
- check_subtitle_completeness: 检查字幕覆盖率与完整性
|
||||
- extract_subtitle_for_range: 提取指定时间范围内的字幕文本
|
||||
- assign_subtitles_voronoi: 使用 Voronoi 中点策略将字幕分配给 L3 节点
|
||||
|
||||
迁移来源:
|
||||
- TRM4 core/tree/enhance/merge.py (parse_srt)
|
||||
- TRM3 tools/generate_subtitles.py (Voronoi 逻辑)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.tree.index import TreeIndex
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 正则表达式
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_HTML_TAG_RE = re.compile(r"<[^>]+>")
|
||||
_MUSIC_ONLY_RE = re.compile(r"^[\s♪♫]*$")
|
||||
_TIMECODE_RE = re.compile(r"(\d+):(\d+):(\d+)[,.](\d+)\s*-->\s*(\d+):(\d+):(\d+)[,.](\d+)")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 数据类型
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SRTEntry:
|
||||
"""单条 SRT 字幕条目。
|
||||
|
||||
属性:
|
||||
start: 开始时间(秒)。
|
||||
end: 结束时间(秒)。
|
||||
text: 字幕文本(已清洗 HTML 标签)。
|
||||
"""
|
||||
|
||||
start: float
|
||||
end: float
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SubtitleReport:
|
||||
"""字幕完整性检查报告。
|
||||
|
||||
属性:
|
||||
total_entries: 字幕条目总数。
|
||||
coverage_ratio: SRT 覆盖时长 / 视频总时长。
|
||||
max_gap_sec: 最大连续无字幕间隔(秒)。
|
||||
usable: 覆盖率是否达到最低要求。
|
||||
"""
|
||||
|
||||
total_entries: int
|
||||
coverage_ratio: float
|
||||
max_gap_sec: float
|
||||
usable: bool
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内部辅助
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ts_to_seconds(h: str, m: str, s: str, ms: str) -> float:
|
||||
"""SRT 时间戳组件 (HH:MM:SS,mmm) 转秒数。
|
||||
|
||||
参数:
|
||||
h: 小时。
|
||||
m: 分钟。
|
||||
s: 秒。
|
||||
ms: 毫秒。
|
||||
|
||||
返回:
|
||||
浮点秒数。
|
||||
"""
|
||||
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms) / 1000
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 公共 API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_srt(srt_path: str) -> list[SRTEntry]:
|
||||
"""解析 SRT 字幕文件,返回结构化条目列表。
|
||||
|
||||
- 剥离 HTML 标签(如 <i>、<b>)
|
||||
- 跳过纯音乐符号行(仅含空白和 ♪♫)
|
||||
- 多行字幕合并为单行(空格连接)
|
||||
- 跳过格式异常的块(容错处理)
|
||||
|
||||
参数:
|
||||
srt_path: SRT 文件的绝对路径。
|
||||
|
||||
返回:
|
||||
按时间顺序排列的 SRTEntry 列表;空文件或无有效条目返回空列表。
|
||||
|
||||
迁移来源:
|
||||
TRM4 core/tree/enhance/merge.py parse_srt
|
||||
TRM3 tools/generate_subtitles.py parse_srt
|
||||
"""
|
||||
with open(srt_path, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
if not content.strip():
|
||||
return []
|
||||
|
||||
entries: list[SRTEntry] = []
|
||||
blocks = re.split(r"\n\s*\n", content.strip())
|
||||
|
||||
for block in blocks:
|
||||
lines = block.strip().split("\n")
|
||||
if len(lines) < 2:
|
||||
continue
|
||||
|
||||
# 在块内搜索时间码行(可能是第 1 行或第 2 行)
|
||||
ts_match = None
|
||||
ts_line_idx = -1
|
||||
for i, line in enumerate(lines):
|
||||
ts_match = _TIMECODE_RE.search(line)
|
||||
if ts_match:
|
||||
ts_line_idx = i
|
||||
break
|
||||
|
||||
if not ts_match:
|
||||
continue
|
||||
|
||||
groups = [int(x) for x in ts_match.groups()]
|
||||
start = _ts_to_seconds(str(groups[0]), str(groups[1]), str(groups[2]), str(groups[3]))
|
||||
end = _ts_to_seconds(str(groups[4]), str(groups[5]), str(groups[6]), str(groups[7]))
|
||||
|
||||
# 时间码行之后的所有行为字幕文本
|
||||
text_lines = lines[ts_line_idx + 1 :]
|
||||
raw_text = " ".join(text_lines)
|
||||
clean_text = _HTML_TAG_RE.sub("", raw_text).strip()
|
||||
|
||||
# 跳过空文本和纯音乐符号行
|
||||
if not clean_text or _MUSIC_ONLY_RE.match(clean_text):
|
||||
continue
|
||||
|
||||
entries.append(SRTEntry(start=start, end=end, text=clean_text))
|
||||
|
||||
logger.debug("SRT 解析完成: {} 条有效条目, 文件={}", len(entries), srt_path)
|
||||
return entries
|
||||
|
||||
|
||||
def check_subtitle_completeness(
|
||||
entries: list[SRTEntry],
|
||||
duration: float,
|
||||
min_coverage: float = 0.3,
|
||||
) -> SubtitleReport:
|
||||
"""检查字幕完整性:覆盖率、最大间隔、可用性判定。
|
||||
|
||||
参数:
|
||||
entries: 已排序的 SRTEntry 列表。
|
||||
duration: 视频总时长(秒),必须 > 0。
|
||||
min_coverage: 最低可用覆盖率阈值(0~1)。
|
||||
|
||||
返回:
|
||||
SubtitleReport 包含覆盖率、最大间隔和可用性判定。
|
||||
"""
|
||||
assert duration > 0, f"视频时长必须 > 0,实际={duration}"
|
||||
|
||||
if not entries:
|
||||
return SubtitleReport(
|
||||
total_entries=0,
|
||||
coverage_ratio=0.0,
|
||||
max_gap_sec=duration,
|
||||
usable=False,
|
||||
)
|
||||
|
||||
# 按开始时间排序
|
||||
sorted_entries = sorted(entries, key=lambda e: e.start)
|
||||
|
||||
# 计算覆盖时长(合并重叠区间)
|
||||
merged_intervals: list[tuple[float, float]] = []
|
||||
for entry in sorted_entries:
|
||||
if merged_intervals and entry.start <= merged_intervals[-1][1]:
|
||||
# 与上一区间重叠,扩展
|
||||
merged_intervals[-1] = (
|
||||
merged_intervals[-1][0],
|
||||
max(merged_intervals[-1][1], entry.end),
|
||||
)
|
||||
else:
|
||||
merged_intervals.append((entry.start, entry.end))
|
||||
|
||||
covered = sum(end - start for start, end in merged_intervals)
|
||||
coverage_ratio = min(covered / duration, 1.0)
|
||||
|
||||
# 计算最大间隔(包括视频开头到第一条字幕、最后一条到视频结尾)
|
||||
max_gap = merged_intervals[0][0] # 视频开头到第一条字幕
|
||||
for i in range(1, len(merged_intervals)):
|
||||
gap = merged_intervals[i][0] - merged_intervals[i - 1][1]
|
||||
max_gap = max(max_gap, gap)
|
||||
# 最后一条字幕到视频结尾
|
||||
max_gap = max(max_gap, duration - merged_intervals[-1][1])
|
||||
|
||||
return SubtitleReport(
|
||||
total_entries=len(entries),
|
||||
coverage_ratio=coverage_ratio,
|
||||
max_gap_sec=max_gap,
|
||||
usable=coverage_ratio >= min_coverage,
|
||||
)
|
||||
|
||||
|
||||
def extract_subtitle_for_range(
|
||||
entries: list[SRTEntry],
|
||||
time_range: tuple[float, float],
|
||||
) -> str:
|
||||
"""提取与指定时间范围重叠的字幕文本。
|
||||
|
||||
重叠判定:entry.start < range_end 且 entry.end > range_start。
|
||||
|
||||
参数:
|
||||
entries: SRTEntry 列表。
|
||||
time_range: (start, end) 时间范围(秒)。
|
||||
|
||||
返回:
|
||||
匹配的字幕文本,多条用换行符连接;无匹配返回空字符串。
|
||||
"""
|
||||
range_start, range_end = time_range
|
||||
matched = [
|
||||
entry.text for entry in entries if entry.start < range_end and entry.end > range_start
|
||||
]
|
||||
return "\n".join(matched)
|
||||
|
||||
|
||||
def assign_subtitles_voronoi(
|
||||
index: TreeIndex,
|
||||
entries: list[SRTEntry],
|
||||
) -> None:
|
||||
"""使用 Voronoi 中点策略将字幕分配给 L3 节点。
|
||||
|
||||
对每个 L2 节点内的 L3 子节点,按 timestamp 排序后计算 Voronoi 有效范围:
|
||||
- 相邻 L3 节点之间取中点作为边界
|
||||
- 首个 L3 的左边界扩展到 L2 的 time_range 起点
|
||||
- 末个 L3 的右边界扩展到 L2 的 time_range 终点
|
||||
|
||||
然后用 extract_subtitle_for_range 提取每个 L3 有效范围内的字幕文本。
|
||||
|
||||
参数:
|
||||
index: 树索引,包含 L1→L2→L3 嵌套结构。
|
||||
entries: 已解析的 SRTEntry 列表。
|
||||
|
||||
副作用:
|
||||
通过 dataclasses.replace 替换 L3Node.card 和 L2Node.card,
|
||||
将字幕写入 card.subtitle 字段。
|
||||
|
||||
迁移来源:
|
||||
TRM3 tools/generate_subtitles.py compute_effective_ranges + assign_subtitles
|
||||
"""
|
||||
for l1 in index.roots:
|
||||
for l2 in l1.children:
|
||||
if not l2.children:
|
||||
continue
|
||||
|
||||
# 按 timestamp 排序 L3 子节点(保留原列表引用以便赋值)
|
||||
siblings = sorted(
|
||||
l2.children,
|
||||
key=lambda n: n.timestamp if n.timestamp is not None else 0.0,
|
||||
)
|
||||
|
||||
# L2 的时间范围作为边界
|
||||
l2_start = l2.time_range[0] if l2.time_range else 0.0
|
||||
l2_end = l2.time_range[1] if l2.time_range else 0.0
|
||||
|
||||
for idx, l3 in enumerate(siblings):
|
||||
ts = l3.timestamp if l3.timestamp is not None else 0.0
|
||||
|
||||
# 计算 Voronoi 有效范围
|
||||
if idx == 0:
|
||||
left = l2_start
|
||||
else:
|
||||
prev_ts = (
|
||||
siblings[idx - 1].timestamp
|
||||
if siblings[idx - 1].timestamp is not None
|
||||
else 0.0
|
||||
)
|
||||
left = (prev_ts + ts) / 2.0
|
||||
|
||||
if idx == len(siblings) - 1:
|
||||
right = l2_end
|
||||
else:
|
||||
next_ts = (
|
||||
siblings[idx + 1].timestamp
|
||||
if siblings[idx + 1].timestamp is not None
|
||||
else 0.0
|
||||
)
|
||||
right = (ts + next_ts) / 2.0
|
||||
|
||||
subtitle_text = extract_subtitle_for_range(entries, (left, right))
|
||||
l3.card = dataclasses.replace(
|
||||
l3.card,
|
||||
subtitle=subtitle_text or "",
|
||||
)
|
||||
|
||||
# L2 字幕聚合:拼接所有 L3 子节点的字幕
|
||||
l3_subtitles = [l3.card.subtitle for l3 in l2.children if l3.card.subtitle]
|
||||
l2.card = dataclasses.replace(
|
||||
l2.card,
|
||||
subtitle="\n".join(l3_subtitles),
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
"Voronoi 字幕分配完成: {} 个 L1 节点, {} 条字幕条目",
|
||||
len(index.roots),
|
||||
len(entries),
|
||||
)
|
||||
@@ -0,0 +1,292 @@
|
||||
"""质量校验模块:交叉验证树节点 Card 字段与子节点证据。
|
||||
|
||||
验证策略:
|
||||
- L2 entities: 仅保留在子 L3 文本语料中模糊匹配到的实体。
|
||||
- L2 visible_text: 仅保留在子 L3 visible_text 中出现的条目。
|
||||
- L1 visible_text: 仅保留在后代 L2/L3 visible_text 中出现的条目。
|
||||
- L1 key_entities: 仅保留在后代 L2/L3 文本语料中模糊匹配到的实体。
|
||||
|
||||
Card 为 frozen dataclass,无法原地修改——移除幻觉字段时
|
||||
创建新 Card 实例并赋值给 node.card(Node 非 frozen)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import string
|
||||
from dataclasses import dataclass
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from app.tree.index import (
|
||||
L1Card,
|
||||
L1Node,
|
||||
L2Card,
|
||||
L2Node,
|
||||
TreeIndex,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 校验统计
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class VerifyStats:
|
||||
"""校验统计信息。"""
|
||||
|
||||
l2_entities_kept: int = 0
|
||||
l2_entities_removed: int = 0
|
||||
l2_visible_text_kept: int = 0
|
||||
l2_visible_text_removed: int = 0
|
||||
l1_visible_text_kept: int = 0
|
||||
l1_visible_text_removed: int = 0
|
||||
l1_key_entities_kept: int = 0
|
||||
l1_key_entities_removed: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 文本归一化 & 模糊匹配
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _normalize(text: str) -> str:
|
||||
"""归一化文本:小写 + 去除标点。
|
||||
|
||||
参数:
|
||||
text: 原始文本。
|
||||
|
||||
返回:
|
||||
归一化后的纯小写无标点字符串。
|
||||
"""
|
||||
return text.lower().translate(str.maketrans("", "", string.punctuation))
|
||||
|
||||
|
||||
def fuzzy_match(entity: str | None, corpus: str | None) -> bool:
|
||||
"""模糊子串匹配:归一化后判断 entity 是否为 corpus 的子串。
|
||||
|
||||
参数:
|
||||
entity: 待匹配的实体文本(None 视为不匹配)。
|
||||
corpus: 证据语料文本(None 视为空)。
|
||||
|
||||
返回:
|
||||
True 表示匹配成功。
|
||||
"""
|
||||
if not entity or not corpus:
|
||||
return False
|
||||
return _normalize(str(entity)) in _normalize(str(corpus))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 语料收集
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _collect_l3_text(l2_node: L2Node) -> str:
|
||||
"""收集 L2 节点所有子 L3 的文本语料。
|
||||
|
||||
从每个 L3 子节点的 card 和顶层字段中提取:
|
||||
frame_summary、visible_text、subtitle。
|
||||
|
||||
参数:
|
||||
l2_node: L2 节点。
|
||||
|
||||
返回:
|
||||
拼接后的文本语料(用换行分隔)。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for l3 in l2_node.children:
|
||||
parts.append(l3.card.frame_summary)
|
||||
parts.extend(l3.card.visible_text)
|
||||
if l3.card.subtitle:
|
||||
parts.append(l3.card.subtitle)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _collect_descendant_visible_text(l1_node: L1Node) -> str:
|
||||
"""收集 L1 节点所有后代(L2/L3)的 visible_text。
|
||||
|
||||
参数:
|
||||
l1_node: L1 节点。
|
||||
|
||||
返回:
|
||||
所有后代 visible_text 拼接后的文本(用换行分隔)。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for l2 in l1_node.children:
|
||||
parts.extend(l2.card.visible_text)
|
||||
for l3 in l2.children:
|
||||
parts.extend(l3.card.visible_text)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _collect_descendant_text_corpus(l1_node: L1Node) -> str:
|
||||
"""收集 L1 节点所有后代(L2/L3)的完整文本语料。
|
||||
|
||||
用于 L1 key_entities 的交叉验证,范围包括
|
||||
L2/L3 的所有文本字段(frame_summary、visible_text、subtitle 等)。
|
||||
|
||||
参数:
|
||||
l1_node: L1 节点。
|
||||
|
||||
返回:
|
||||
所有后代文本语料拼接后的文本(用换行分隔)。
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for l2 in l1_node.children:
|
||||
parts.append(l2.card.event_description)
|
||||
parts.extend(l2.card.entities)
|
||||
parts.extend(l2.card.visible_text)
|
||||
for l3 in l2.children:
|
||||
parts.append(l3.card.frame_summary)
|
||||
parts.extend(l3.card.visible_text)
|
||||
if l3.card.subtitle:
|
||||
parts.append(l3.card.subtitle)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 主校验函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def verify_tree(index: TreeIndex) -> VerifyStats:
|
||||
"""交叉验证视频树的 Card 字段与子节点证据,原地替换不合格的 Card。
|
||||
|
||||
Cards 为 frozen dataclass,移除幻觉字段时创建新 Card 实例
|
||||
并赋值给 node.card。
|
||||
|
||||
参数:
|
||||
index: 树索引(会被原地修改)。
|
||||
|
||||
返回:
|
||||
VerifyStats 校验统计。
|
||||
"""
|
||||
stats = VerifyStats()
|
||||
|
||||
for l1 in index.roots:
|
||||
# Phase 1: L2 字段验证
|
||||
for l2 in l1.children:
|
||||
_verify_l2(l2, stats)
|
||||
|
||||
# Phase 2: L1 字段验证
|
||||
_verify_l1(l1, stats)
|
||||
|
||||
logger.info(
|
||||
"verify_tree: source={} "
|
||||
"l2_ent_kept={} l2_ent_rm={} "
|
||||
"l2_vt_kept={} l2_vt_rm={} "
|
||||
"l1_vt_kept={} l1_vt_rm={} "
|
||||
"l1_ke_kept={} l1_ke_rm={}",
|
||||
index.metadata.source_path,
|
||||
stats.l2_entities_kept,
|
||||
stats.l2_entities_removed,
|
||||
stats.l2_visible_text_kept,
|
||||
stats.l2_visible_text_removed,
|
||||
stats.l1_visible_text_kept,
|
||||
stats.l1_visible_text_removed,
|
||||
stats.l1_key_entities_kept,
|
||||
stats.l1_key_entities_removed,
|
||||
)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def _verify_l2(l2: L2Node, stats: VerifyStats) -> None:
|
||||
"""校验单个 L2 节点的 entities 和 visible_text。
|
||||
|
||||
参数:
|
||||
l2: L2 节点(card 可能被替换)。
|
||||
stats: 统计对象(原地累加)。
|
||||
"""
|
||||
corpus = _collect_l3_text(l2)
|
||||
old_card = l2.card
|
||||
|
||||
# entities: 模糊匹配过滤
|
||||
kept_entities = [e for e in old_card.entities if fuzzy_match(e, corpus)]
|
||||
stats.l2_entities_kept += len(kept_entities)
|
||||
stats.l2_entities_removed += len(old_card.entities) - len(kept_entities)
|
||||
|
||||
# visible_text: 子 L3 visible_text 中必须存在
|
||||
l3_visible = _collect_l3_visible_text_set(l2)
|
||||
kept_vt = [vt for vt in old_card.visible_text if _text_in_set(vt, l3_visible)]
|
||||
stats.l2_visible_text_kept += len(kept_vt)
|
||||
stats.l2_visible_text_removed += len(old_card.visible_text) - len(kept_vt)
|
||||
|
||||
# 创建新 Card 替换(frozen dataclass)
|
||||
l2.card = L2Card(
|
||||
event_description=old_card.event_description,
|
||||
entities=kept_entities,
|
||||
actions=old_card.actions,
|
||||
action_subjects=old_card.action_subjects,
|
||||
visible_text=kept_vt,
|
||||
spatial_relations=old_card.spatial_relations,
|
||||
state_changes=old_card.state_changes,
|
||||
subtitle=old_card.subtitle,
|
||||
)
|
||||
|
||||
|
||||
def _verify_l1(l1: L1Node, stats: VerifyStats) -> None:
|
||||
"""校验单个 L1 节点的 visible_text 和 key_entities。
|
||||
|
||||
参数:
|
||||
l1: L1 节点(card 可能被替换)。
|
||||
stats: 统计对象(原地累加)。
|
||||
"""
|
||||
old_card = l1.card
|
||||
|
||||
# visible_text: 必须出现在后代 L2/L3 visible_text 中
|
||||
descendant_vt = _collect_descendant_visible_text(l1)
|
||||
kept_vt = [vt for vt in old_card.visible_text if fuzzy_match(vt, descendant_vt)]
|
||||
stats.l1_visible_text_kept += len(kept_vt)
|
||||
stats.l1_visible_text_removed += len(old_card.visible_text) - len(kept_vt)
|
||||
|
||||
# key_entities: 交叉验证后代文本语料
|
||||
descendant_corpus = _collect_descendant_text_corpus(l1)
|
||||
kept_ke = [ke for ke in old_card.key_entities if fuzzy_match(ke, descendant_corpus)]
|
||||
stats.l1_key_entities_kept += len(kept_ke)
|
||||
stats.l1_key_entities_removed += len(old_card.key_entities) - len(kept_ke)
|
||||
|
||||
# 创建新 Card 替换(frozen dataclass)
|
||||
l1.card = L1Card(
|
||||
scene_summary=old_card.scene_summary,
|
||||
main_setting=old_card.main_setting,
|
||||
key_entities=kept_ke,
|
||||
main_actions=old_card.main_actions,
|
||||
topic_keywords=old_card.topic_keywords,
|
||||
visible_text=kept_vt,
|
||||
temporal_flow=old_card.temporal_flow,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 辅助函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _collect_l3_visible_text_set(l2: L2Node) -> set[str]:
|
||||
"""收集 L2 下所有 L3 子节点的 visible_text 归一化集合。
|
||||
|
||||
参数:
|
||||
l2: L2 节点。
|
||||
|
||||
返回:
|
||||
归一化后的 visible_text 集合。
|
||||
"""
|
||||
result: set[str] = set()
|
||||
for l3 in l2.children:
|
||||
for vt in l3.card.visible_text:
|
||||
result.add(_normalize(vt))
|
||||
return result
|
||||
|
||||
|
||||
def _text_in_set(text: str, normalized_set: set[str]) -> bool:
|
||||
"""检查文本归一化后是否存在于集合中。
|
||||
|
||||
参数:
|
||||
text: 待检查文本。
|
||||
normalized_set: 归一化后的文本集合。
|
||||
|
||||
返回:
|
||||
True 表示匹配成功。
|
||||
"""
|
||||
return _normalize(text) in normalized_set
|
||||
File diff suppressed because it is too large
Load Diff
+30
-35
@@ -20,46 +20,15 @@ embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cpu"
|
||||
|
||||
# ── 可训练检索器 ──
|
||||
retriever:
|
||||
embed_dim: 768
|
||||
num_heads: 4
|
||||
L_layers: 2
|
||||
L_cycles: 4
|
||||
max_rounds: 5
|
||||
ffn_expansion: 2.0
|
||||
checkpoint: null
|
||||
k_l1: 1
|
||||
k_l2: 1
|
||||
k_l3: 1
|
||||
max_paths: 5
|
||||
|
||||
# ── 检索器训练 ──
|
||||
train:
|
||||
lr: 1.0e-4
|
||||
weight_decay: 1.0e-5
|
||||
batch_size: 1
|
||||
max_epochs_phase1: 30
|
||||
max_epochs_phase2: 20
|
||||
nav_loss_weight: 1.0
|
||||
act_loss_weight: 0.1
|
||||
margin_loss_weight: 0.5
|
||||
act_lambda_step: 0.1
|
||||
act_gamma: 0.9
|
||||
eval_interval: 5
|
||||
save_dir: "checkpoints"
|
||||
dataset: "videomme"
|
||||
dataset_path: "data/videomme/splits/train.jsonl"
|
||||
device: "cuda"
|
||||
|
||||
# ── Harness 自进化循环 ──
|
||||
harness:
|
||||
workspace_dir: "workspaces/default"
|
||||
store_dir: store
|
||||
mode: infer
|
||||
concurrency: 12
|
||||
max_steps: 15
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "benchmarks/Video-MME"
|
||||
@@ -73,7 +42,6 @@ harness:
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_block: 8
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
@@ -98,5 +66,32 @@ harness:
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
trainable_min_units: 8
|
||||
early_stop_patience: 8
|
||||
use_slow_momentum: true
|
||||
# 池构建策略
|
||||
pool_split_mode: global # global | per_category
|
||||
train_ratio: 0.667 # per_category 模式下 train/(train+val) 比例
|
||||
test_questions: "benchmarks/Video-MME" # test 池的题目来源
|
||||
|
||||
# ── 出题管线 v2 ──
|
||||
question_gen_v2:
|
||||
family_ratios:
|
||||
retrieval: 0.30
|
||||
reasoning: 0.25
|
||||
enumeration: 0.20
|
||||
visual: 0.15
|
||||
spatial: 0.10
|
||||
gate:
|
||||
blind_answer_model: "gpt-4.1-mini"
|
||||
leak_test_model: "gpt-4.1-mini"
|
||||
key_verify_model: "gpt-4.1-mini"
|
||||
multi_true_model: "gpt-4.1-mini"
|
||||
dedup_threshold: 0.85
|
||||
retry_limit: 3
|
||||
heavy_sample_rate: 0.15
|
||||
heavy_agent_model: "gpt-4.1-mini"
|
||||
output_dir: "store/questions/generated-v2"
|
||||
per_type: 20 # 12 类 x 20 = 240 题(设计 §3 硬约束)
|
||||
concurrency: 4
|
||||
seed: 42
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# 补生 Video-MME 6 类缺失题型(180 题)
|
||||
# 原 360 题中有 6 类错误类型已归档,此配置只生成缺失的 6 类。
|
||||
|
||||
# ── 建树模块 ──(与 question_gen_360.yaml 一致)
|
||||
tree:
|
||||
max_paragraphs_per_l2: 5
|
||||
l1_segment_duration: 600.0
|
||||
l2_clip_duration: 60.0
|
||||
l3_fps: 0.5
|
||||
l2_representative_frames: 6
|
||||
cache_dir: "cache/trees"
|
||||
concurrency: 16
|
||||
subtitle_inject: true
|
||||
srt_window_sec: 5.0
|
||||
|
||||
# ── Embedding ──
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
|
||||
# ── Harness ──(占位,出题不使用)
|
||||
harness:
|
||||
workspace_dir: "workspaces/default"
|
||||
store_dir: store
|
||||
mode: infer
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "benchmarks/Video-MME"
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 1
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
diag_size: 200
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 30
|
||||
val_correct_ratio: 0.5
|
||||
test_size: 60
|
||||
batch_size: 15
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
early_stop_patience: 8
|
||||
use_slow_momentum: true
|
||||
|
||||
# ── 出题管线 v2 ──
|
||||
question_gen_v2:
|
||||
family_ratios:
|
||||
retrieval: 0.30
|
||||
reasoning: 0.25
|
||||
enumeration: 0.20
|
||||
visual: 0.15
|
||||
spatial: 0.10
|
||||
dedup_threshold: 0.85
|
||||
retry_limit: 10
|
||||
heavy_sample_rate: 0.15
|
||||
output_dir: "store/questions/generated-v2-180补"
|
||||
per_type: 30 # 6 类 x 30 = 180 题
|
||||
concurrency: 24
|
||||
seed: 43 # 不同于原始 seed=42,避免生成相同题目
|
||||
@@ -0,0 +1,86 @@
|
||||
# config/default.yaml
|
||||
# 科研实验配置默认值来源(会在实验中反复扫动/对比的参数)。
|
||||
# 工程配置(少变、敏感)由 .env / pydantic-settings 管理,不在此文件。
|
||||
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
|
||||
|
||||
# ── 建树模块 ──
|
||||
tree:
|
||||
max_paragraphs_per_l2: 5
|
||||
l1_segment_duration: 600.0 # L1 段时长(秒)
|
||||
l2_clip_duration: 60.0 # L2 clip 时长(秒)
|
||||
l3_fps: 0.5 # L3 帧提取频率(帧/秒)
|
||||
l2_representative_frames: 6 # L2 VLM 描述用的代表帧数
|
||||
cache_dir: "cache/trees"
|
||||
concurrency: 16 # asyncio Semaphore 上限
|
||||
subtitle_inject: true # 建树时是否注入 SRT 字幕
|
||||
srt_window_sec: 5.0 # 字幕匹配时间窗口(前后各 N 秒)
|
||||
|
||||
# ── Embedding ──
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
|
||||
# ── Harness 自进化循环 ──
|
||||
harness:
|
||||
workspace_dir: "workspaces/default"
|
||||
store_dir: store
|
||||
mode: infer
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "benchmarks/Video-MME"
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 1
|
||||
# CE-Gate 参数
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
# 进化参数
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
# 数据池
|
||||
diag_size: 200
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 30
|
||||
val_correct_ratio: 0.5
|
||||
test_size: 60
|
||||
# mini-batch
|
||||
batch_size: 15
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
early_stop_patience: 8
|
||||
use_slow_momentum: true
|
||||
|
||||
# ── 出题管线 v2 ──
|
||||
question_gen_v2:
|
||||
family_ratios:
|
||||
retrieval: 0.30
|
||||
reasoning: 0.25
|
||||
enumeration: 0.20
|
||||
visual: 0.15
|
||||
spatial: 0.10
|
||||
dedup_threshold: 0.85
|
||||
retry_limit: 10
|
||||
heavy_sample_rate: 0.15
|
||||
output_dir: "store/questions/generated-v2-360"
|
||||
per_type: 30 # 12 类 x 30 = 360 题
|
||||
concurrency: 24
|
||||
seed: 42
|
||||
@@ -0,0 +1,21 @@
|
||||
# AR 专项出题配置:使用 ActionRecognitionStrategy + 6 SubPattern 靶向生成
|
||||
# 目标:生成 30 道 Action Recognition 题
|
||||
|
||||
question_gen_v2:
|
||||
per_type: 30 # 只跑 AR 一类,30 题
|
||||
retry_limit: 15 # AR 约束更严,给更多重试机会
|
||||
heavy_sample_rate: 0.0 # 不需要重量抽检
|
||||
dedup_threshold: 0.85
|
||||
concurrency: 8 # AR 需要帧,适度并发
|
||||
seed: 2024
|
||||
output_dir: "store/questions/generated-ar30"
|
||||
candidate_pool_size: 24 # grounded selector 首轮候选干扰项数 N
|
||||
selector_delta_low: 0.05 # 干扰项视觉分与正解的最小差(区间上界)
|
||||
selector_delta_high: 0.35 # 干扰项视觉分与正解的最大差(区间下界)
|
||||
|
||||
# Phase B 后置对抗过滤层配置(filter 层,非 strategy 属性)
|
||||
adversarial_filter:
|
||||
filter_task_types: [Action Recognition] # 仅这些题型走 agent 门(路径隔离开关)
|
||||
adversarial_max_rounds: 5 # 补生成迭代上限
|
||||
adversarial_agent_max_steps: 40 # agent 试答步数上限
|
||||
difficulty_warn_threshold: 0.85 # 批次 agent 正确率告警阈值
|
||||
@@ -0,0 +1,63 @@
|
||||
# config/train_action_recognition.yaml
|
||||
# Action Recognition 单题型首次训练实验
|
||||
# 设计文档: research-wiki/designs/2026-07-14-action-recognition-training-design.md
|
||||
|
||||
harness:
|
||||
workspace_dir: "workspaces/train-action-recognition"
|
||||
store_dir: store
|
||||
mode: train
|
||||
run_id: train_ar_v1
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "generated-v2-360"
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 3
|
||||
# CE-Gate 参数(沿用 default.yaml)
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
# 进化参数
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
# 池配置 — per_category 单题型
|
||||
pool_split_mode: per_category
|
||||
task_types:
|
||||
- "Action Recognition"
|
||||
train_ratio: 0.667
|
||||
test_questions: "benchmarks/Video-MME"
|
||||
run_holdout_eval: false
|
||||
# mini-batch
|
||||
batch_size: 10
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
trainable_min_units: 8
|
||||
early_stop_patience: 4
|
||||
test_size: 63
|
||||
diag_size: 20
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 10
|
||||
val_correct_ratio: 0.5
|
||||
use_slow_momentum: true
|
||||
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
@@ -0,0 +1,64 @@
|
||||
# config/train_ar30.yaml
|
||||
# Action Recognition 训练 — 基于 SubPattern 靶向生成的 30 题
|
||||
# 对比基线: v2-360 的 AR 题(100% 单帧,训练无效)
|
||||
# 本次: AR30 题(6 种失败子模式靶向,跨段时序)
|
||||
|
||||
harness:
|
||||
workspace_dir: "workspaces/train-ar30"
|
||||
store_dir: store
|
||||
mode: train
|
||||
run_id: train_ar30_v1
|
||||
concurrency: 24
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "generated-ar30"
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 3
|
||||
# CE-Gate 参数(沿用 default.yaml)
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
# 进化参数
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
# 池配置 — per_category 单题型
|
||||
pool_split_mode: per_category
|
||||
task_types:
|
||||
- "Action Recognition"
|
||||
train_ratio: 0.667
|
||||
test_questions: "benchmarks/Video-MME"
|
||||
run_holdout_eval: false
|
||||
# mini-batch
|
||||
batch_size: 10
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
eval_min_per_class: 2
|
||||
trainable_min_units: 8
|
||||
early_stop_patience: 4
|
||||
test_size: 63
|
||||
diag_size: 20
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 10
|
||||
val_correct_ratio: 0.5
|
||||
use_slow_momentum: true
|
||||
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
@@ -0,0 +1,69 @@
|
||||
# config/train_videomme.yaml
|
||||
# Video-MME 900 题自进化训练 —— 消费 video-split 冻结切分(global 三池)
|
||||
#
|
||||
# 数据来源: workspaces/video-split/pools.json(tier 感知 diag/val + val 功效修复)
|
||||
# 经 adhoc-baseline seed 携带进 workspace(WP2 接线)。
|
||||
# 训练前置: .env REDIS_CACHE_TTL 须为正整数(WP4 fail-loud);见
|
||||
# research-wiki/reviews/2026-07-16-preflight-final-review.md runbook。
|
||||
|
||||
harness:
|
||||
workspace_dir: "workspaces/train-videomme"
|
||||
store_dir: store
|
||||
mode: train
|
||||
run_id: train_videomme_v2
|
||||
concurrency: 32
|
||||
max_steps: 40
|
||||
skill_mode: auto
|
||||
n_samples: 0
|
||||
questions: "benchmarks/Video-MME" # gate 指纹依赖加载全 900 题
|
||||
skills_version: v1
|
||||
prompts_version: v1
|
||||
epochs: 3
|
||||
# CE-Gate 参数(沿用 default.yaml)
|
||||
gate_e_confirm: 20.0
|
||||
gate_e_provisional: 3.0
|
||||
gate_w_net_min: 2
|
||||
gate_delta_min: 0.02
|
||||
gate_lambda_dir: -0.642
|
||||
gate_e_rollback: 10.0
|
||||
gate_n_max: 40
|
||||
gate_p_low: 0.05
|
||||
gate_p_high: 0.95
|
||||
gate_probe_quota: 0.2
|
||||
gate_gamma_decay: 0.9
|
||||
gate_cooldown_steps: 2
|
||||
gate_guard_err: 0.10
|
||||
# 进化参数
|
||||
edit_budget_start: 5
|
||||
edit_budget_end: 2
|
||||
skill_update_mode: patch
|
||||
appendix_consolidate_threshold: 6
|
||||
# 池配置 —— global 冻结切分(diag/val/test 尺寸由 pools.json 冻结,以下采样旋钮加载时忽略)
|
||||
# 实际冻结切分(0.4+tier,fingerprint d456ef4f): diag=180 val=120 test=600
|
||||
pool_split_mode: global
|
||||
diag_size: 180
|
||||
diag_correct_ratio: 0.5
|
||||
val_size: 120
|
||||
val_correct_ratio: 0.5
|
||||
test_size: 600
|
||||
test_questions: "benchmarks/Video-MME"
|
||||
# 可训练性预检(WP3):val 单元 < eval_min_per_class 或 非test单元 < trainable_min_units 的题型剔除
|
||||
eval_min_per_class: 2
|
||||
trainable_min_units: 8
|
||||
# mini-batch —— 对齐 TRM4 正式实验 batch=40(sh --batch-size 40 覆盖 yaml 15 的最终生效值):
|
||||
# 8 可训题型 × 每型约 5 题/step,保住题型级诊断信号;同时 steps/epoch 180/40≈5,
|
||||
# 进化/gate 验证轮数比 batch=10 少 4 倍。
|
||||
batch_size: 40
|
||||
min_class_per_batch: 2
|
||||
batch_correct_ratio: 0.5
|
||||
momentum_samples: 20
|
||||
early_stop_patience: 2 # epoch 粒度(WP3):连续 2 epoch 无 best 刷新即停
|
||||
use_slow_momentum: true
|
||||
run_holdout_eval: true # 逐 epoch test 四向评估(WP3 去重版:baseline 推导 + 版本备忘录)
|
||||
# 全 12 题型(不指定 task_types 子集,避免 I-4 语义偏差;微型类由预检自动剔除)
|
||||
|
||||
embed:
|
||||
backend: "local"
|
||||
model_name: "BAAI/bge-base-zh-v1.5"
|
||||
embed_dim: 768
|
||||
device: "cuda"
|
||||
@@ -0,0 +1,33 @@
|
||||
# config/video_split.yaml
|
||||
# 结果驱动视频级切分 —— 科研旋钮快照(会在实验中被反复扫动 / 对比的参数)。
|
||||
# 工程配置(harness.db 路径、LLM 凭证、超时)由 .env / pydantic-settings 管理,不在此文件。
|
||||
# 优先级: CLI args > 此文件。CLI 仅用于单次临时覆盖。
|
||||
#
|
||||
# ⚠️ 下列阈值为「占位值」,必须在离线诊断跑完后用真实 T2 分布标定。
|
||||
# 标定程序见 scripts/build_video_split.sh 顶部注释(诊断产物 → 逐旋钮定值)。
|
||||
|
||||
video_split:
|
||||
baseline_run_id: infer_adhoc # 基线 run 标识(错题诊断与切分依据),对应 workspaces/default/harness.db
|
||||
n_trainval: 100 # trainval 目标视频数(多样性阶段填充上限;标定取 ~100)
|
||||
epsilon: 0.1 # test 相对全局最大允许分布偏差(题型占比 / 难度画像两维,逐桶)
|
||||
report_floor: 27 # per-type 报告门限:题数 ≥ 此值的 task_type 才入 ε 代表性约束
|
||||
val_wrong_min: 20 # validation 池最少错题数(McNemar 检验功效阈 ≈ 20,低于则信号不足)
|
||||
val_ratio: 0.4 # validation 占 trainval 视频组总数的比例(0.3→0.4 提升整包终审功效,WP2)
|
||||
seed: 7 # 贪心选择器预洗牌 + 视频组题级切分种子(打破等增益 / 等槽平局)
|
||||
floor_k: # 各 task_type 的 T2 defect 下限(硬约束)—— 均衡覆盖全 11 类,标定于 1cb1c203 真实 T2 分布
|
||||
Object Reasoning: 5 # T2=25
|
||||
Information Synopsis: 5 # T2=13
|
||||
Action Reasoning: 5 # T2=11
|
||||
Counting Problem: 5 # T2=10
|
||||
Temporal Reasoning: 2 # T2=5
|
||||
Object Recognition: 2 # T2=5
|
||||
Action Recognition: 2 # T2=5
|
||||
Attribute Perception: 1 # T2=3
|
||||
Temporal Perception: 1 # T2=2
|
||||
OCR Problems: 1 # T2=2
|
||||
Spatial Perception: 1 # T2=1
|
||||
|
||||
diag: # 诊断口径指纹三分量(隔离不同诊断配置的信号,参与主键)
|
||||
prompt_version: diagnose_v1 # 诊断 prompt 版本标识(换 prompt 即换指纹,旧记录不被覆盖)
|
||||
model: deepseek-v4-pro # 执行诊断的模型名(必须与 .env SEARCH_LLM_MODEL 一致,CLI 会 fail loud 校验)
|
||||
# code_version 由 build_video_split.sh 注入 git 短 SHA,不写死在此(随代码变动)
|
||||
+108
-9
@@ -17,7 +17,9 @@ TRM4 → TRM5 有意变更(非简化):
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pluggy
|
||||
@@ -31,6 +33,9 @@ if TYPE_CHECKING:
|
||||
from core.protocols import LLMProvider
|
||||
from core.types import LLMResponse
|
||||
|
||||
# deepseek 等模型稳定输出变体:```json 围栏包裹 JSON 体
|
||||
_CODE_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*\n?|\n?\s*```\s*$")
|
||||
|
||||
|
||||
async def _call_hook(hook: Any, **kwargs: Any) -> list[Any]:
|
||||
"""调用 pluggy hook 并 await 异步返回值。
|
||||
@@ -67,6 +72,13 @@ class AgentLoop:
|
||||
llm: LLMProvider 实例(Protocol 类型化,提供 async chat 方法)。
|
||||
max_steps: 最大有效步数(每次成功工具调用计一步)。
|
||||
max_retries: JSON 解析连续失败的最大容忍次数。
|
||||
step_retries: LLM 瞬时异常的步级重试次数(不含首次调用)。
|
||||
step_retry_delays: 步级重试的退避秒数序列,超出部分取末值;
|
||||
不得为空(空序列构造时抛 ValueError,fail-fast)。
|
||||
retryable_exceptions: 可重试异常元组,默认 (TimeoutError, OSError)——
|
||||
ssl.SSLError/ConnectionError 均为 OSError 子类,覆盖穿透
|
||||
GovernedLLMClient 内部重试栈的瞬时异常;openai API 类异常由
|
||||
治理层负责,core 不依赖 SDK。
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -74,10 +86,19 @@ class AgentLoop:
|
||||
llm: LLMProvider,
|
||||
max_steps: int,
|
||||
max_retries: int = 3,
|
||||
*,
|
||||
step_retries: int = 2,
|
||||
step_retry_delays: tuple[float, ...] = (20.0, 40.0),
|
||||
retryable_exceptions: tuple[type[BaseException], ...] = (TimeoutError, OSError),
|
||||
) -> None:
|
||||
if not step_retry_delays:
|
||||
raise ValueError("step_retry_delays 不得为空")
|
||||
self._llm = llm
|
||||
self._max_steps = max_steps
|
||||
self._max_retries = max_retries
|
||||
self._step_retries = step_retries
|
||||
self._step_retry_delays = step_retry_delays
|
||||
self._retryable_exceptions = retryable_exceptions
|
||||
|
||||
async def run(
|
||||
self,
|
||||
@@ -87,6 +108,7 @@ class AgentLoop:
|
||||
plugins: list[object] | None = None,
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LoopResult:
|
||||
"""执行 Thinking+JSON 推理循环。
|
||||
|
||||
@@ -96,6 +118,7 @@ class AgentLoop:
|
||||
tool_dispatcher: 工具调度器,ToolDispatcher Protocol 实例。
|
||||
plugins: pluggy 插件列表。
|
||||
session_id: 会话 ID,透传给 LLMProvider。
|
||||
cache_salt: 缓存盐,透传给 LLMProvider(如训练用 run_id 跨 epoch 重采样)。
|
||||
|
||||
返回:
|
||||
LoopResult 实例,包含推理步骤、token 用量、终止原因。
|
||||
@@ -114,11 +137,13 @@ class AgentLoop:
|
||||
while step_count < self._max_steps:
|
||||
await _call_hook(pm.hook.before_step, iteration=iteration, messages=messages)
|
||||
|
||||
# Phase 1: LLM 调用
|
||||
# Phase 1: LLM 调用(步级重试:防穿透 GovernedLLMClient 的瞬时异常)
|
||||
try:
|
||||
response = await self._call_llm(messages, token_usage, session_id=session_id)
|
||||
response = await self._call_llm_with_step_retry(
|
||||
messages, token_usage, session_id=session_id, cache_salt=cache_salt
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("LLM API 调用失败: {}", e)
|
||||
logger.error("LLM API 调用失败({}): {}", type(e).__name__, e)
|
||||
result = LoopResult(
|
||||
steps=steps,
|
||||
steps_used=step_count,
|
||||
@@ -240,14 +265,21 @@ class AgentLoop:
|
||||
pm.register(plugin)
|
||||
return pm
|
||||
|
||||
async def _call_llm(
|
||||
async def _call_llm_with_step_retry(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
token_usage: dict[str, int],
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse:
|
||||
"""调用 LLMProvider 并累加 token 使用量。
|
||||
"""带步级重试的 LLM 调用,兜底穿透治理层重试栈的瞬时异常。
|
||||
|
||||
可重试异常(self._retryable_exceptions)按 self._step_retry_delays
|
||||
退避后重发,重试预算(self._step_retries)耗尽后抛出最后一次异常;
|
||||
非可重试异常直接穿透(fail-fast)。asyncio.CancelledError 继承
|
||||
BaseException,天然不被捕获——取消信号照常传播。失败尝试的 error
|
||||
遥测由 GovernedLLMClient 内部负责,此处仅 loguru 记录。
|
||||
|
||||
参数:
|
||||
messages: 消息历史。
|
||||
@@ -256,8 +288,52 @@ class AgentLoop:
|
||||
|
||||
返回:
|
||||
LLMResponse 实例。
|
||||
|
||||
异常:
|
||||
重试耗尽或非可重试时,原样抛出底层异常。
|
||||
"""
|
||||
response = await self._llm.chat(messages, session_id=session_id)
|
||||
step_attempt = 0
|
||||
while True:
|
||||
try:
|
||||
return await self._call_llm(
|
||||
messages, token_usage, session_id=session_id, cache_salt=cache_salt
|
||||
)
|
||||
except self._retryable_exceptions as e:
|
||||
step_attempt += 1
|
||||
if step_attempt > self._step_retries:
|
||||
raise
|
||||
delay = self._step_retry_delays[
|
||||
min(step_attempt - 1, len(self._step_retry_delays) - 1)
|
||||
]
|
||||
logger.warning(
|
||||
"LLM 瞬时异常,步级重试 {}/{}({}s 后重发): {}",
|
||||
step_attempt,
|
||||
self._step_retries,
|
||||
delay,
|
||||
e,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
async def _call_llm(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
token_usage: dict[str, int],
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse:
|
||||
"""调用 LLMProvider 并累加 token 使用量。
|
||||
|
||||
参数:
|
||||
messages: 消息历史。
|
||||
token_usage: 可变字典,就地累加。
|
||||
session_id: 会话 ID,透传给 LLMProvider。
|
||||
cache_salt: 缓存盐,透传给 LLMProvider(跨 epoch 重采样)。
|
||||
|
||||
返回:
|
||||
LLMResponse 实例。
|
||||
"""
|
||||
response = await self._llm.chat(messages, session_id=session_id, cache_salt=cache_salt)
|
||||
token_usage["prompt_tokens"] += response.prompt_tokens
|
||||
token_usage["completion_tokens"] += response.completion_tokens
|
||||
return response
|
||||
@@ -267,7 +343,8 @@ class AgentLoop:
|
||||
) -> tuple[str, dict, dict, str, dict, str] | None:
|
||||
"""从 LLMResponse 中提取结构化决策数据。
|
||||
|
||||
解析流程: content → repair_json → json.loads → 校验 action/tool/args。
|
||||
解析流程: content → 剥除 ```json 围栏 → repair_json → json.loads
|
||||
→ 收拢 action 平铺参数 → 校验 action/tool/args。
|
||||
|
||||
参数:
|
||||
response: LLMResponse 实例。
|
||||
@@ -282,7 +359,7 @@ class AgentLoop:
|
||||
if not content.strip():
|
||||
return None
|
||||
|
||||
repaired = repair_json(content)
|
||||
repaired = repair_json(_CODE_FENCE_RE.sub("", content).strip())
|
||||
try:
|
||||
data = json.loads(repaired)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
@@ -291,7 +368,7 @@ class AgentLoop:
|
||||
if not isinstance(data, dict) or "action" not in data:
|
||||
return None
|
||||
|
||||
action = data["action"]
|
||||
action = self._normalize_action(data["action"])
|
||||
if not isinstance(action, dict) or "tool" not in action or "args" not in action:
|
||||
return None
|
||||
|
||||
@@ -299,6 +376,28 @@ class AgentLoop:
|
||||
plan = data.get("plan", {})
|
||||
return thought, reflect, plan, content, action, response.call_id
|
||||
|
||||
@staticmethod
|
||||
def _normalize_action(action: Any) -> Any:
|
||||
"""收拢 deepseek 变体的 action 平铺参数。
|
||||
|
||||
deepseek 等模型稳定输出变体: 工具参数平铺在 action 下(缺 args
|
||||
嵌套),确定性收拢为标准 {"tool": ..., "args": {...}} 结构。
|
||||
仅当除 tool 外至少存在一个平铺参数键时才收拢;无参结构
|
||||
(如 {"tool": "x"})原样返回交由调用方校验拒绝,避免把缺参
|
||||
错误静默升级为空 args 合法结构。标准嵌套与非法结构同样原样返回。
|
||||
|
||||
参数:
|
||||
action: 从 LLM 输出解析出的 action 字段(任意类型)。
|
||||
|
||||
返回:
|
||||
归一化后的 action(仅带平铺参数的变体被改写,其余原样返回)。
|
||||
"""
|
||||
if isinstance(action, dict) and "tool" in action and "args" not in action:
|
||||
flat_args = {k: v for k, v in action.items() if k != "tool"}
|
||||
if flat_args:
|
||||
return {"tool": action["tool"], "args": flat_args}
|
||||
return action
|
||||
|
||||
async def _execute_tool(
|
||||
self,
|
||||
dispatcher: ToolDispatcher,
|
||||
|
||||
+7
-11
@@ -1,11 +1,13 @@
|
||||
"""Agent 专属 Protocol 端口。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
import pluggy
|
||||
|
||||
from core.agent.types import LoopResult, Step
|
||||
if TYPE_CHECKING:
|
||||
from core.agent.types import LoopResult, Step
|
||||
|
||||
hookspec = pluggy.HookspecMarker("agent_loop")
|
||||
hookimpl = pluggy.HookimplMarker("agent_loop")
|
||||
@@ -27,19 +29,13 @@ class AgentLoopSpec:
|
||||
"""
|
||||
|
||||
@hookspec
|
||||
async def before_step(
|
||||
self, iteration: int, messages: list[dict[str, Any]]
|
||||
) -> None: ...
|
||||
async def before_step(self, iteration: int, messages: list[dict[str, Any]]) -> None: ...
|
||||
|
||||
@hookspec
|
||||
async def after_tool(
|
||||
self, iteration: int, step: Step
|
||||
) -> str | None: ...
|
||||
async def after_tool(self, iteration: int, step: Step) -> str | None: ...
|
||||
|
||||
@hookspec
|
||||
async def after_step(
|
||||
self, iteration: int, messages: list[dict[str, Any]]
|
||||
) -> None: ...
|
||||
async def after_step(self, iteration: int, messages: list[dict[str, Any]]) -> None: ...
|
||||
|
||||
@hookspec
|
||||
async def on_finish(self, result: LoopResult) -> None: ...
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
"""AgentLoop 数据类型。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
"""core/evolution/ — 自进化循环决策内核。
|
||||
|
||||
诊断、进化、门控、补丁的纯决策逻辑。
|
||||
只依赖 Protocol 接口和标准库,可搬到无 adapters 的环境用假实现原样运行。
|
||||
"""
|
||||
|
||||
from core.evolution.diagnose import INFRA_STOP_REASONS, run_diagnosis
|
||||
from core.evolution.evolve import (
|
||||
edit_budget_at,
|
||||
evolve_single_skill,
|
||||
evolve_single_tool,
|
||||
evolve_system_prompt,
|
||||
resolve_skill_file,
|
||||
)
|
||||
from core.evolution.gate import compute_e_value, gate_decision, probation_verdict
|
||||
from core.evolution.patch import (
|
||||
append_to_appendix,
|
||||
apply_patch_with_report,
|
||||
extract_appendix_notes,
|
||||
momentum_inner,
|
||||
replace_appendix_notes,
|
||||
replace_momentum,
|
||||
)
|
||||
from core.evolution.types import (
|
||||
CaseSample,
|
||||
DiagnosePrompts,
|
||||
DiagnosisResult,
|
||||
ErrorAttribution,
|
||||
EvolutionRecord,
|
||||
EvolutionResult,
|
||||
EvolvePrompts,
|
||||
GateParams,
|
||||
GateVerdict,
|
||||
PairResult,
|
||||
QuadrantClassification,
|
||||
QuestionMetrics,
|
||||
RejectedEdit,
|
||||
SkillCasePack,
|
||||
SkillStepAdherence,
|
||||
SpanMetrics,
|
||||
SystemCasePack,
|
||||
ToolCasePack,
|
||||
)
|
||||
from core.evolution.validate import classify_quadrants, compute_accuracy, pair_block
|
||||
|
||||
__all__ = [
|
||||
"INFRA_STOP_REASONS",
|
||||
"CaseSample",
|
||||
"DiagnosePrompts",
|
||||
"DiagnosisResult",
|
||||
"ErrorAttribution",
|
||||
"EvolutionRecord",
|
||||
"EvolutionResult",
|
||||
"EvolvePrompts",
|
||||
"GateParams",
|
||||
"GateVerdict",
|
||||
"PairResult",
|
||||
"QuadrantClassification",
|
||||
"QuestionMetrics",
|
||||
"RejectedEdit",
|
||||
"SkillCasePack",
|
||||
"SkillStepAdherence",
|
||||
"SpanMetrics",
|
||||
"SystemCasePack",
|
||||
"ToolCasePack",
|
||||
"append_to_appendix",
|
||||
"apply_patch_with_report",
|
||||
"classify_quadrants",
|
||||
"compute_accuracy",
|
||||
"compute_e_value",
|
||||
"edit_budget_at",
|
||||
"evolve_single_skill",
|
||||
"evolve_single_tool",
|
||||
"evolve_system_prompt",
|
||||
"extract_appendix_notes",
|
||||
"gate_decision",
|
||||
"momentum_inner",
|
||||
"pair_block",
|
||||
"probation_verdict",
|
||||
"replace_appendix_notes",
|
||||
"replace_momentum",
|
||||
"resolve_skill_file",
|
||||
"run_diagnosis",
|
||||
]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,129 @@
|
||||
"""CE-Gate 统计核心:截断 Beta 混合 e-process 的纯函数实现。
|
||||
|
||||
配对不一致检验:候选与基线跑同一题,只数翻转(基线错->候选对 = W;
|
||||
基线对->候选错 = L)。H0(候选不优)下翻转方向精确五五开,
|
||||
E = 2^(W+L+1)*B(W+1,L+1)*[1-I_1/2(W+1,L+1)] 为 H0 下非负上鞅,
|
||||
Ville 不等式给出任意停时 P(E >= 1/alpha) <= alpha。
|
||||
|
||||
设计规格见 research-wiki/designs/2026-07-03-ce-gate-formal-design.md。
|
||||
仅依赖 scipy.special,无 I/O、无状态,便于单测与历史回放复用。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from scipy.special import betainc, betaln
|
||||
|
||||
from core.evolution.types import GateParams, GateVerdict
|
||||
|
||||
# Wald 方向游走步长(theta_1=0.70 固定设计常量,不入配置):
|
||||
# 胜 +ln(2*theta_1)=ln1.4,负 ln(2*(1-theta_1))=ln0.6。
|
||||
_WALD_WIN = math.log(1.4)
|
||||
_WALD_LOSS = math.log(0.6)
|
||||
|
||||
# delta_shrunk 的伪计数(Agresti-Coull 风格收缩,只作观测输出不进判据)。
|
||||
_SHRINK_PSEUDO = 4
|
||||
|
||||
|
||||
def compute_e_value(w: int, l: int) -> float: # noqa: E741
|
||||
"""截断 Beta 混合 e 值:E = 2^(W+L+1)*B(W+1,L+1)*[1-I_1/2(W+1,L+1)]。
|
||||
|
||||
参数:
|
||||
w: 基线错->候选对的翻转数。
|
||||
l: 基线对->候选错的翻转数。
|
||||
|
||||
返回:
|
||||
e 值(W=L=0 时为 1)。
|
||||
|
||||
异常:
|
||||
ValueError: 翻转计数为负时抛出。
|
||||
|
||||
关键实现细节:
|
||||
log 空间计算在 n_max<=40 的设计工作区间内数值稳定(数百级计数
|
||||
亦可);极大计数(>1000)时最终 exp 仍可能溢出。用正则化不完全
|
||||
Beta 的对称性 1-I_1/2(a,b) = I_1/2(b,a) 避免 1-x 的灾难性精度损失。
|
||||
"""
|
||||
if w < 0 or l < 0:
|
||||
raise ValueError(f"翻转计数不能为负: w={w}, l={l}")
|
||||
a, b = w + 1, l + 1
|
||||
tail = betainc(b, a, 0.5) # = 1 - I_1/2(a, b)
|
||||
if tail <= 0.0:
|
||||
return 0.0
|
||||
log_e = (w + l + 1) * math.log(2.0) + betaln(a, b) + math.log(tail)
|
||||
return math.exp(log_e)
|
||||
|
||||
|
||||
def gate_decision(
|
||||
w: int,
|
||||
l: int, # noqa: E741
|
||||
n_used: int,
|
||||
n_remaining: int,
|
||||
*,
|
||||
params: GateParams,
|
||||
) -> GateVerdict:
|
||||
"""块间四出口判定(每块结束时调用一次)。
|
||||
|
||||
出口优先级:CONFIRMED(有证书先走)-> 方向拒绝 -> futility 拒绝 ->
|
||||
题尽(provisional / inertia)-> continue。
|
||||
|
||||
参数:
|
||||
w: 累计 W。
|
||||
l: 累计 L。
|
||||
n_used: 已消费的阶梯题数(含一致题)。
|
||||
n_remaining: 阶梯剩余可用题数(min(阶梯长, n_max) - n_used)。
|
||||
params: 判据阈值组。
|
||||
|
||||
返回:
|
||||
GateVerdict(decision + e 值/游走/效应量诊断)。
|
||||
|
||||
异常:
|
||||
ValueError: n_used <= 0 或 n_remaining < 0 时抛出。
|
||||
"""
|
||||
if n_used <= 0:
|
||||
raise ValueError(f"gate_decision 须在至少消费一块后调用: n_used={n_used}")
|
||||
if n_remaining < 0:
|
||||
raise ValueError(f"n_remaining 不能为负: {n_remaining}")
|
||||
e_value = compute_e_value(w, l)
|
||||
wald = w * _WALD_WIN + l * _WALD_LOSS
|
||||
delta_hat = (w - l) / n_used
|
||||
delta_shrunk = (w - l) / (n_used + _SHRINK_PSEUDO)
|
||||
|
||||
if e_value >= params.e_confirm and delta_hat >= params.delta_min:
|
||||
decision = "accept_confirmed"
|
||||
elif wald <= params.lambda_dir:
|
||||
decision = "reject_directional"
|
||||
elif n_remaining > 0 and compute_e_value(w + n_remaining, l) < params.e_provisional:
|
||||
# futility 只在题未尽时有意义;题尽后的弱证据归 inertia 出口。
|
||||
decision = "reject_futility"
|
||||
elif n_remaining <= 0:
|
||||
if (
|
||||
e_value >= params.e_provisional
|
||||
and (w - l) >= params.w_net_min
|
||||
and delta_hat >= params.delta_min
|
||||
):
|
||||
decision = "accept_provisional"
|
||||
else:
|
||||
decision = "reject_inertia"
|
||||
else:
|
||||
decision = "continue"
|
||||
return GateVerdict(decision, e_value, wald, delta_hat, delta_shrunk)
|
||||
|
||||
|
||||
def probation_verdict(w: int, l: int, *, params: GateParams) -> str: # noqa: E741
|
||||
"""试用期一次性结算:固定样本 e 值双向检验。
|
||||
|
||||
参数:
|
||||
w: 结算配对的 W(锚快照错->候选重跑对)。
|
||||
l: 结算配对的 L(锚快照对->候选重跑错)。
|
||||
params: 判据阈值组(用 e_confirm / e_rollback)。
|
||||
|
||||
返回:
|
||||
"confirmed"(E>=e_confirm 转正)/ "rollback"(对称 E'>=e_rollback 回滚)
|
||||
/ "unverified"(证据不足,elitist 惯性转正)。
|
||||
"""
|
||||
if compute_e_value(w, l) >= params.e_confirm:
|
||||
return "confirmed"
|
||||
if compute_e_value(l, w) >= params.e_rollback:
|
||||
return "rollback"
|
||||
return "unverified"
|
||||
@@ -0,0 +1,455 @@
|
||||
"""定点补丁引擎:把进化输出的离散 edit 逐条应用到文本,逐条出状态报告。
|
||||
|
||||
借鉴 SkillOpt skill.py 的 apply 语义;守 P5:找不到锚点不静默乱改、不裸 except。
|
||||
冻结区按全文坐标区间判定;append/退化追加插到最早冻结区之前(无则 EOF)。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from loguru import logger
|
||||
|
||||
APPENDIX_START = "<!-- APPENDIX_START -->"
|
||||
APPENDIX_END = "<!-- APPENDIX_END -->"
|
||||
APPENDIX_MAX_CHARS = 2000 # appendix 区软上限(守设计「长度上限+warning,不做去重」)
|
||||
|
||||
MOMENTUM_START = "<!-- MOMENTUM_START -->"
|
||||
MOMENTUM_END = "<!-- MOMENTUM_END -->"
|
||||
MOMENTUM_MAX_CHARS = 2000 # momentum 区软上限(与 appendix 一致:超限 warning 不截断)
|
||||
MOMENTUM_HEADING = "## 动量指导(每轮重写,勿手改)" # replace_momentum 写入的固定标题行
|
||||
|
||||
|
||||
def momentum_region_bounds(text: str) -> tuple[int, int] | None:
|
||||
"""定位 momentum 受保护区的字符区间,并对损坏态显式报错(P5)。
|
||||
|
||||
momentum marker 由 replace_momentum 在 epoch 末反复重写,guidance 又来自 LLM
|
||||
外部输入,因此 marker 可能出现损坏态。本函数是 momentum 路径的唯一边界判定入口,
|
||||
把配对校验集中在一处:
|
||||
|
||||
- START 与 END 各恰好出现一次且 START 在 END 之前 → 返回 (start_idx, end_idx),
|
||||
end_idx 指向 END marker 结束位置(即 content[start:end] 含完整两 marker)。
|
||||
- 两 marker 都不出现 → 返回 None(合法的"无区"态,调用方据此新建)。
|
||||
- 其余皆为损坏态(仅一个 marker、END 在 START 前、任一 marker 重复)→ raise
|
||||
ValueError,拒绝静默新建/跳过,要求人工修复。
|
||||
|
||||
参数:
|
||||
text: 待检测的文本(skill 全文)。
|
||||
返回:
|
||||
(start_idx, end_idx) 表示区间,或 None 表示无 momentum 区。
|
||||
异常:
|
||||
ValueError: momentum marker 损坏/不配对。
|
||||
"""
|
||||
start_count = text.count(MOMENTUM_START)
|
||||
end_count = text.count(MOMENTUM_END)
|
||||
if start_count == 0 and end_count == 0:
|
||||
return None
|
||||
if start_count != 1 or end_count != 1:
|
||||
raise ValueError(
|
||||
f"momentum marker 损坏/不配对:MOMENTUM_START 出现 {start_count} 次、"
|
||||
f"MOMENTUM_END 出现 {end_count} 次(各须恰好 1 次),需人工修复"
|
||||
)
|
||||
start_idx = text.index(MOMENTUM_START)
|
||||
end_idx = text.index(MOMENTUM_END) + len(MOMENTUM_END)
|
||||
if start_idx >= text.index(MOMENTUM_END):
|
||||
raise ValueError(
|
||||
"momentum marker 损坏/不配对:MOMENTUM_END 出现在 MOMENTUM_START 之前,需人工修复"
|
||||
)
|
||||
return start_idx, end_idx
|
||||
|
||||
|
||||
def momentum_inner(content: str) -> str:
|
||||
"""返回 momentum 受保护区的内层文本(去掉两 marker),无区返回空串。
|
||||
|
||||
与 _momentum_span(含 marker 的整段)的区别:本函数只取两 marker 之间的内层正文,
|
||||
供 run_slow_momentum 的 prev_guidance 使用。prev_guidance 在 LLM 解析失败时会被
|
||||
run_slow_momentum 原样返回、再喂给 replace_momentum;replace_momentum 禁止 guidance
|
||||
含 marker 字面量,故 prev_guidance 必须是无 marker 的内层文本,否则一旦解析回退即
|
||||
在 replace_momentum 抛 ValueError。
|
||||
|
||||
边界判定与配对校验统一委托 momentum_region_bounds:marker 损坏/不配对时由其 raise
|
||||
ValueError,本函数不把损坏态静默当作"无区"。
|
||||
|
||||
参数:
|
||||
content: skill 全文。
|
||||
返回:
|
||||
momentum 区两 marker 之间的内层文本(已 strip);无区返回空串。
|
||||
异常:
|
||||
ValueError: momentum marker 损坏/不配对。
|
||||
"""
|
||||
bounds = momentum_region_bounds(content)
|
||||
if bounds is None:
|
||||
return ""
|
||||
start, end = bounds
|
||||
inner = content[start + len(MOMENTUM_START) : end - len(MOMENTUM_END)].strip()
|
||||
# 去掉 replace_momentum 写入的固定标题行,只回传纯指导文本,使其等价于上一轮
|
||||
# 传给 replace_momentum 的 guidance(解析回退时原样回传不会引入重复标题)。
|
||||
if inner.startswith(MOMENTUM_HEADING):
|
||||
inner = inner[len(MOMENTUM_HEADING) :].lstrip("\n")
|
||||
return inner.strip()
|
||||
|
||||
|
||||
def append_to_appendix(content: str, notes: list[str]) -> str:
|
||||
"""把 LAPSE 提醒追加到文件尾的 appendix 受保护区;区不存在则创建。
|
||||
|
||||
护栏:appendix 区超过 APPENDIX_MAX_CHARS 时 logger.warning(不静默截断,
|
||||
提示人工压缩;不做自动去重——YAGNI,见设计)。
|
||||
|
||||
参数:
|
||||
content: 原文。
|
||||
notes: 待追加的提醒文本列表。
|
||||
返回:
|
||||
含 appendix 区的新文本。
|
||||
"""
|
||||
if not notes:
|
||||
return content
|
||||
bullet = "\n".join(f"- {n.strip()}" for n in notes if n.strip())
|
||||
if not bullet:
|
||||
return content
|
||||
if APPENDIX_START in content and APPENDIX_END in content:
|
||||
head, rest = content.split(APPENDIX_START, 1)
|
||||
inner, tail = rest.split(APPENDIX_END, 1)
|
||||
new_inner = f"{inner.rstrip()}\n{bullet}"
|
||||
out = f"{head}{APPENDIX_START}{new_inner}\n{APPENDIX_END}{tail}"
|
||||
else:
|
||||
new_inner = f"\n## 执行提醒(自动累积,勿手改)\n{bullet}"
|
||||
out = f"{content.rstrip()}\n\n{APPENDIX_START}{new_inner}\n{APPENDIX_END}\n"
|
||||
if len(new_inner) > APPENDIX_MAX_CHARS:
|
||||
logger.warning(
|
||||
"appendix 区长度 {} 超过上限 {},建议人工压缩",
|
||||
len(new_inner),
|
||||
APPENDIX_MAX_CHARS,
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def appendix_region_bounds(text: str) -> tuple[int, int] | None:
|
||||
"""定位 appendix 受保护区的字符区间,对损坏态显式报错(P5,对称 momentum_region_bounds)。
|
||||
|
||||
appendix marker 由 append_to_appendix 维护、consolidation 回写,可能出现损坏态。
|
||||
本函数是 appendix 路径的唯一边界判定入口,把配对校验集中一处:
|
||||
|
||||
- START 与 END 各恰好一次且 START 在 END 之前 → 返回 (start_idx, end_idx),
|
||||
end_idx 指向 END marker 结束位置(content[start:end] 含完整两 marker)。
|
||||
- 两 marker 都不出现 → 返回 None(合法的「无区」态)。
|
||||
- 其余(仅一个 marker、END 在 START 前、任一 marker 重复)→ raise ValueError,
|
||||
拒绝静默按字符串切片处理而误拼/吞掉区外正文。
|
||||
|
||||
参数:
|
||||
text: 待检测文本(skill 全文)。
|
||||
返回:
|
||||
(start_idx, end_idx) 表示区间,或 None 表示无 appendix 区。
|
||||
异常:
|
||||
ValueError: appendix marker 损坏/不配对。
|
||||
"""
|
||||
start_count = text.count(APPENDIX_START)
|
||||
end_count = text.count(APPENDIX_END)
|
||||
if start_count == 0 and end_count == 0:
|
||||
return None
|
||||
if start_count != 1 or end_count != 1:
|
||||
raise ValueError(
|
||||
f"appendix marker 损坏/不配对:APPENDIX_START 出现 {start_count} 次、"
|
||||
f"APPENDIX_END 出现 {end_count} 次(各须恰好 1 次),需人工修复"
|
||||
)
|
||||
start_idx = text.index(APPENDIX_START)
|
||||
end_idx = text.index(APPENDIX_END) + len(APPENDIX_END)
|
||||
if start_idx >= text.index(APPENDIX_END):
|
||||
raise ValueError(
|
||||
"appendix marker 损坏/不配对:APPENDIX_END 出现在 APPENDIX_START 之前,需人工修复"
|
||||
)
|
||||
return start_idx, end_idx
|
||||
|
||||
|
||||
def extract_appendix_notes(content: str) -> list[str]:
|
||||
"""从 appendix 受保护区解析出 bullet 提醒列表;无区返回空列表。
|
||||
|
||||
功能:
|
||||
取 appendix 区内每行以 "- " 起头的文本为一条 note(去 "- " 前缀与首尾空白),
|
||||
区内标题行(## 执行提醒…)不计。供 consolidation 读取现有 notes。
|
||||
参数:
|
||||
content: skill 全文。
|
||||
返回:
|
||||
note 字符串列表;无 appendix 区返回 []。
|
||||
异常:
|
||||
ValueError: appendix marker 损坏/不配对(经 appendix_region_bounds,不静默切片)。
|
||||
关键实现细节:
|
||||
边界判定统一委托 appendix_region_bounds,只取两 marker 之间内层正文逐行解析。
|
||||
"""
|
||||
bounds = appendix_region_bounds(content)
|
||||
if bounds is None:
|
||||
return []
|
||||
start, end = bounds
|
||||
inner = content[start + len(APPENDIX_START) : end - len(APPENDIX_END)]
|
||||
notes: list[str] = []
|
||||
for line in inner.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("- "):
|
||||
note = stripped[2:].strip()
|
||||
if note:
|
||||
notes.append(note)
|
||||
return notes
|
||||
|
||||
|
||||
def replace_appendix_notes(content: str, notes: list[str]) -> str:
|
||||
"""用 notes 整体替换 appendix 区内容;notes 为空则删除整个 appendix 区。
|
||||
|
||||
功能:
|
||||
consolidation 回写压缩后 notes 的替换语义(区别于 append_to_appendix 累积):
|
||||
区存在则整体覆盖区内 bullet;notes 空则连 marker 一并删除、保留区外正文;
|
||||
区不存在且 notes 非空则按 append_to_appendix 格式新建。
|
||||
参数:
|
||||
content: 原文(可能含 appendix 区)。
|
||||
notes: 压缩后的提醒列表;空列表表示删区。
|
||||
返回:
|
||||
替换后的全文。
|
||||
异常:
|
||||
ValueError: appendix marker 损坏/不配对(经 appendix_region_bounds)。
|
||||
关键实现细节:
|
||||
边界经 appendix_region_bounds 显式校验,按 (start,end) 切出 head/tail 拼接,
|
||||
不做两次独立 split(避免损坏态误拼/吞掉区外正文)。
|
||||
"""
|
||||
bounds = appendix_region_bounds(content)
|
||||
if bounds is not None:
|
||||
start, end = bounds
|
||||
head = content[:start]
|
||||
tail = content[end:]
|
||||
if not notes:
|
||||
return head.rstrip() + ("\n" + tail.lstrip("\n") if tail.strip() else "\n")
|
||||
bullet = "\n".join(f"- {n.strip()}" for n in notes if n.strip())
|
||||
new_inner = f"\n## 执行提醒(自动累积,勿手改)\n{bullet}"
|
||||
return f"{head}{APPENDIX_START}{new_inner}\n{APPENDIX_END}{tail}"
|
||||
if not notes:
|
||||
return content
|
||||
return append_to_appendix(content, notes)
|
||||
|
||||
|
||||
def replace_momentum(content: str, guidance: str) -> str:
|
||||
"""把「动量指导」整体写入文件尾的 momentum 受保护区;区不存在则创建。
|
||||
|
||||
与 append_to_appendix 的累积语义不同,momentum 是**替换**语义:慢更新周期每
|
||||
epoch 末整体重写一段动量指导,旧指导被完全覆盖(不保留历史)。momentum 区与
|
||||
appendix 区独立共存——本函数只触碰 momentum marker,不破坏已有 appendix 区。
|
||||
|
||||
护栏:momentum 区超过 MOMENTUM_MAX_CHARS 时 logger.warning(不静默截断,与
|
||||
appendix 对齐)。
|
||||
|
||||
关键实现细节:
|
||||
- 替换非追加:区已存在时用 guidance 整体覆盖 marker 内 inner,旧动量不残留。
|
||||
- 创建位置在文件尾(append_to_appendix 同样在文件尾,但两区 marker 不同,
|
||||
split 按各自 marker 定位,互不干扰)。
|
||||
|
||||
空 guidance 决策:与 appendix 的累积语义不同,momentum 是「每轮整体重写」,空
|
||||
guidance 表示「本轮无动量指导」,属合法语义——照常写入(区内仅留标题,旧动量被清空),
|
||||
而非返回原文保留旧动量。
|
||||
|
||||
参数:
|
||||
content: 原文(可能已含 appendix 区)。
|
||||
guidance: 本轮动量指导全文(整体覆盖旧动量)。
|
||||
返回:
|
||||
含 momentum 区的新文本。
|
||||
异常:
|
||||
ValueError: guidance 含 momentum marker 字面量(外部输入注入),或原文 momentum
|
||||
marker 损坏/不配对。
|
||||
"""
|
||||
if MOMENTUM_START in guidance or MOMENTUM_END in guidance:
|
||||
raise ValueError(
|
||||
"guidance 不得包含 momentum marker 字面量"
|
||||
f"({MOMENTUM_START} / {MOMENTUM_END}),否则会破坏 marker 配对"
|
||||
)
|
||||
bounds = momentum_region_bounds(content)
|
||||
new_inner = f"\n## 动量指导(每轮重写,勿手改)\n{guidance.strip()}"
|
||||
if bounds is not None:
|
||||
start_idx, end_idx = bounds
|
||||
head = content[:start_idx]
|
||||
tail = content[end_idx:]
|
||||
out = f"{head}{MOMENTUM_START}{new_inner}\n{MOMENTUM_END}{tail}"
|
||||
else:
|
||||
out = f"{content.rstrip()}\n\n{MOMENTUM_START}{new_inner}\n{MOMENTUM_END}\n"
|
||||
if len(new_inner) > MOMENTUM_MAX_CHARS:
|
||||
logger.warning(
|
||||
"momentum 区长度 {} 超过上限 {},建议人工压缩",
|
||||
len(new_inner),
|
||||
MOMENTUM_MAX_CHARS,
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _protected_ranges(content: str, spans: list[str]) -> list[tuple[int, int]]:
|
||||
"""把冻结文本块映射成 content 中的 [start, end) 坐标区间。"""
|
||||
ranges: list[tuple[int, int]] = []
|
||||
for span in spans:
|
||||
idx = content.find(span)
|
||||
if idx != -1:
|
||||
ranges.append((idx, idx + len(span)))
|
||||
return ranges
|
||||
|
||||
|
||||
def _span_overlaps_ranges(pos: int, length: int, ranges: list[tuple[int, int]]) -> bool:
|
||||
"""判断 [pos, pos+length) 是否与任一冻结区间相交(不止起点)。
|
||||
|
||||
起点落在正文、末端伸入冻结区的 target 也须拦截,否则 replace/delete 会连带
|
||||
改动冻结区(如破坏 appendix/momentum marker)。
|
||||
|
||||
参数:
|
||||
pos: target 在正文中的起点。
|
||||
length: target 长度。
|
||||
ranges: 冻结区间 [start, end) 列表。
|
||||
返回:
|
||||
与任一区间相交返回 True。
|
||||
"""
|
||||
end = pos + length
|
||||
return any(start < end and pos < r_end for start, r_end in ranges)
|
||||
|
||||
|
||||
# 冻结区 marker 字面量:LLM 生成的 edit 不得注入这些字面量,否则破坏 marker 配对
|
||||
_MARKER_LITERALS = (APPENDIX_START, APPENDIX_END, MOMENTUM_START, MOMENTUM_END)
|
||||
|
||||
|
||||
def _edit_injects_marker(edit: dict) -> bool:
|
||||
"""判断 edit 的 target/content 是否含冻结区 marker 字面量(注入拦截)。"""
|
||||
for key in ("target", "content"):
|
||||
value = edit.get(key)
|
||||
if isinstance(value, str) and any(m in value for m in _MARKER_LITERALS):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _append_at(content: str, ranges: list[tuple[int, int]]) -> int:
|
||||
"""append/退化追加落点:最早一个 start>0 的冻结区之前;无则文末(头部 frontmatter 不计)。"""
|
||||
starts = [start for start, _ in ranges if start > 0]
|
||||
return min(starts) if starts else len(content)
|
||||
|
||||
|
||||
def _insert_at(content: str, at: int, payload: str) -> str:
|
||||
"""在 at 位置插入 payload,自动补换行保持段落格式。"""
|
||||
head, tail = content[:at].rstrip(), content[at:].lstrip("\n")
|
||||
if tail:
|
||||
return head + "\n\n" + payload + "\n\n" + tail
|
||||
return head + "\n\n" + payload + "\n"
|
||||
|
||||
|
||||
def _do_append(content: str, payload: str, ranges: list[tuple[int, int]]) -> tuple[str, str]:
|
||||
"""执行 append 操作,返回更新后内容与状态字符串。"""
|
||||
return _insert_at(content, _append_at(content, ranges), payload), "applied_append"
|
||||
|
||||
|
||||
def _do_insert_after(
|
||||
content: str, target: str, payload: str, ranges: list[tuple[int, int]]
|
||||
) -> tuple[str, str]:
|
||||
"""执行 insert_after 操作,处理退化追加与冻结区跳过。"""
|
||||
pos = content.find(target) if target else -1
|
||||
if pos == -1:
|
||||
logger.warning("insert_after 锚点缺失,退化为追加 target={}", target[:80])
|
||||
return (
|
||||
_insert_at(content, _append_at(content, ranges), payload),
|
||||
"applied_insert_after_fallback",
|
||||
)
|
||||
if _span_overlaps_ranges(pos, len(target), ranges):
|
||||
logger.warning("insert_after 目标跨入冻结区,跳过 target={}", target[:80])
|
||||
return content, "skipped_protected"
|
||||
at = pos + len(target)
|
||||
nl = content.find("\n", at)
|
||||
at = nl + 1 if nl != -1 else len(content)
|
||||
return content[:at] + payload + "\n" + content[at:], "applied_insert_after"
|
||||
|
||||
|
||||
def _do_replace_delete(
|
||||
op: str,
|
||||
content: str,
|
||||
target: str,
|
||||
payload: str,
|
||||
ranges: list[tuple[int, int]],
|
||||
) -> tuple[str, str]:
|
||||
"""执行 replace 或 delete 操作,返回更新后内容与状态字符串。"""
|
||||
if not target:
|
||||
return content, "skipped_missing_target"
|
||||
pos = content.find(target)
|
||||
if pos == -1:
|
||||
logger.warning("{} 锚点缺失,跳过 target={}", op, target[:80])
|
||||
return content, "skipped_target_not_found"
|
||||
if _span_overlaps_ranges(pos, len(target), ranges):
|
||||
logger.warning("{} 目标跨入冻结区,跳过 target={}", op, target[:80])
|
||||
return content, "skipped_protected"
|
||||
new_content = content.replace(target, payload if op == "replace" else "", 1)
|
||||
return new_content, "applied_" + op
|
||||
|
||||
|
||||
def _apply_one(content: str, edit: dict, ranges: list[tuple[int, int]]) -> tuple[str, dict]:
|
||||
"""应用单条 edit,返回 (更新后内容, 状态报告)。"""
|
||||
if not isinstance(edit, dict):
|
||||
return content, {
|
||||
"op": "",
|
||||
"target": "",
|
||||
"content_preview": "",
|
||||
"status": "error",
|
||||
"error": f"edit 非 dict: {type(edit).__name__}",
|
||||
}
|
||||
op = str(edit.get("op", ""))
|
||||
target = str(edit.get("target", "") or "")
|
||||
payload = str(edit.get("content", "") or "").strip()
|
||||
report = {
|
||||
"op": op,
|
||||
"target": target[:200],
|
||||
"content_preview": payload[:200],
|
||||
"status": "unknown",
|
||||
}
|
||||
|
||||
if op == "append":
|
||||
content, report["status"] = _do_append(content, payload, ranges)
|
||||
return content, report
|
||||
|
||||
if op == "insert_after":
|
||||
content, report["status"] = _do_insert_after(content, target, payload, ranges)
|
||||
return content, report
|
||||
|
||||
if op in ("replace", "delete"):
|
||||
content, report["status"] = _do_replace_delete(op, content, target, payload, ranges)
|
||||
return content, report
|
||||
|
||||
logger.warning("未知 op,跳过: {}", op)
|
||||
report["status"] = "skipped_unknown_op"
|
||||
return content, report
|
||||
|
||||
|
||||
def apply_patch_with_report(
|
||||
content: str,
|
||||
edits: list[dict],
|
||||
protected_spans: list[str] | None = None,
|
||||
) -> tuple[str, list[dict]]:
|
||||
"""顺序应用 edit 列表,返回 (新内容, 逐条状态报告)。
|
||||
|
||||
参数:
|
||||
content: 原始文本。
|
||||
edits: 每条 {op, target, content}。
|
||||
protected_spans: 冻结文本块列表;目标落入其坐标区间即跳过,append 插到其前。
|
||||
|
||||
返回:
|
||||
(应用后文本, reports);reports 每条含 op/target/content_preview/status/index。
|
||||
"""
|
||||
spans = protected_spans or []
|
||||
reports: list[dict] = []
|
||||
for i, edit in enumerate(edits, 1):
|
||||
try:
|
||||
if isinstance(edit, dict) and _edit_injects_marker(edit):
|
||||
logger.warning("edit 含冻结区 marker 字面量,拒绝该 edit index={}", i)
|
||||
report = {
|
||||
"op": str(edit.get("op", "")),
|
||||
"target": str(edit.get("target", "") or "")[:200],
|
||||
"content_preview": str(edit.get("content", "") or "")[:200],
|
||||
"status": "skipped_marker_injection",
|
||||
}
|
||||
report["index"] = i
|
||||
reports.append(report)
|
||||
continue
|
||||
ranges = _protected_ranges(content, spans)
|
||||
content, report = _apply_one(content, edit, ranges)
|
||||
except (KeyError, TypeError, ValueError, AttributeError) as exc:
|
||||
report = {
|
||||
"op": "",
|
||||
"target": "",
|
||||
"content_preview": "",
|
||||
"status": "error",
|
||||
"error": str(exc),
|
||||
}
|
||||
logger.exception("补丁应用异常 index={}", i)
|
||||
report["index"] = i
|
||||
reports.append(report)
|
||||
return content, reports
|
||||
@@ -0,0 +1,160 @@
|
||||
"""core/evolution/ 子包的持久化 Protocol 定义。
|
||||
|
||||
SkillStore / PromptStore / RunLog 为只读——core/ 返回结果 dataclass,
|
||||
读取由 app/ 落盘的资源。SkillStore / PromptStore 同步(文件读取量小且快),
|
||||
RunLog 异步(隔离 SQLite 查询,core/ 不写 SQL)。
|
||||
DiagnosisSignalStore 兼具读写:逐题 upsert 诊断信号并支持断点续跑查询,
|
||||
同样隔离 SQLite 实现,app/core 不写裸 SQL。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from core.evolution.types import DiagnosisSignalRow
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class SkillStore(Protocol):
|
||||
"""版本化技能读取端口。
|
||||
|
||||
实现方解析 manifest 指针,core/ 不感知版本号。
|
||||
"""
|
||||
|
||||
def read_skill(self, filename: str) -> str:
|
||||
"""读取指定 skill 文件的全文内容。
|
||||
|
||||
参数:
|
||||
filename: skill 文件名,如 'temporal-reasoning.md'。
|
||||
|
||||
返回:
|
||||
文件全文内容。
|
||||
"""
|
||||
...
|
||||
|
||||
def list_skill_files(self) -> list[str]:
|
||||
"""列出当前版本所有 skill 文件名。
|
||||
|
||||
返回:
|
||||
文件名列表。
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class PromptStore(Protocol):
|
||||
"""版本化提示词读取端口。
|
||||
|
||||
覆盖 system.md 和 tool extract/verify 文件。
|
||||
"""
|
||||
|
||||
def read_prompt(self, filename: str) -> str:
|
||||
"""读取指定 prompt 文件的全文内容。
|
||||
|
||||
参数:
|
||||
filename: prompt 文件名,如 'system.md'。
|
||||
|
||||
返回:
|
||||
文件全文内容。
|
||||
"""
|
||||
...
|
||||
|
||||
def list_prompt_files(self) -> list[str]:
|
||||
"""列出当前版本所有 prompt 文件名。
|
||||
|
||||
返回:
|
||||
文件名列表。
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class RunLog(Protocol):
|
||||
"""实验日志查询端口。
|
||||
|
||||
隔离 SQLite 实现细节,core/ 不写 SQL。
|
||||
"""
|
||||
|
||||
async def get_predictions(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
question_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""查询指定 run 的预测记录。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
预测记录字典列表。
|
||||
"""
|
||||
...
|
||||
|
||||
async def get_traces(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
question_ids: list[str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""查询指定 run 的推理轨迹。
|
||||
|
||||
参数:
|
||||
run_id: 运行标识。
|
||||
question_ids: 可选的题目 ID 过滤列表。
|
||||
|
||||
返回:
|
||||
轨迹记录字典列表。
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class DiagnosisSignalStore(Protocol):
|
||||
"""逐题诊断信号存储端口。
|
||||
|
||||
隔离 SQLite 实现细节,app/core 不写裸 SQL。逐题 upsert 落盘、
|
||||
支持断点续跑(done_question_ids 查已完成集合)。
|
||||
"""
|
||||
|
||||
def upsert(self, row: DiagnosisSignalRow) -> None:
|
||||
"""写入或覆盖单题诊断信号(按主键幂等)。
|
||||
|
||||
参数:
|
||||
row: 待持久化的诊断信号行。
|
||||
"""
|
||||
...
|
||||
|
||||
def done_question_ids(
|
||||
self,
|
||||
baseline_run_id: str,
|
||||
diag_fingerprint: str,
|
||||
*,
|
||||
retry_uncertain: bool = False,
|
||||
) -> set[str]:
|
||||
"""查询指定 run 与诊断指纹下已完成的 question_id 集合。
|
||||
|
||||
参数:
|
||||
baseline_run_id: baseline run 标识。
|
||||
diag_fingerprint: 诊断口径指纹。
|
||||
retry_uncertain: True 时把 tier='uncertain'(信号不可信降级)题视为
|
||||
未完成,令其被重新诊断;默认 False(uncertain 也算完成,不重诊)。
|
||||
|
||||
返回:
|
||||
已落盘信号的 question_id 集合,用于断点续跑跳过。
|
||||
"""
|
||||
...
|
||||
|
||||
def load(self, baseline_run_id: str, diag_fingerprint: str) -> list[DiagnosisSignalRow]:
|
||||
"""加载指定 run 与诊断指纹下的全部诊断信号行。
|
||||
|
||||
参数:
|
||||
baseline_run_id: baseline run 标识。
|
||||
diag_fingerprint: 诊断口径指纹。
|
||||
|
||||
返回:
|
||||
还原后的 DiagnosisSignalRow 列表。
|
||||
"""
|
||||
...
|
||||
@@ -0,0 +1,518 @@
|
||||
"""core/evolution 子包的数据类型定义。
|
||||
|
||||
自进化循环中 gate、diagnose、evolve、validate 共用的 dataclass。
|
||||
所有输出类型默认 frozen=True(一次性构造、不可变),唯一例外是
|
||||
EvolutionRecord(构建过程中需要多次修改状态)。
|
||||
|
||||
不依赖 app/ 或 adapters/。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
# =========================================================================
|
||||
# 1. Gate 决策类型
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateParams:
|
||||
"""CE-Gate 判据阈值组(从实验配置构造)。
|
||||
|
||||
属性:
|
||||
e_confirm: CONFIRMED 接受的 e 值门槛(1/alpha,20 对应 alpha=5%)。
|
||||
e_provisional: 题尽暂定接受门槛,同时是 futility 出口的代数界。
|
||||
w_net_min: 题尽暂定接受要求的最小净胜 W-L。
|
||||
delta_min: 接受要求的最小点估计效应量 (W-L)/n_used。
|
||||
lambda_dir: Wald 方向游走的拒绝阈值(负数)。
|
||||
e_rollback: 试用期结算的对称回滚 e 值门槛(1/alpha',10 对应 10%)。
|
||||
"""
|
||||
|
||||
e_confirm: float
|
||||
e_provisional: float
|
||||
w_net_min: int
|
||||
delta_min: float
|
||||
lambda_dir: float
|
||||
e_rollback: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateVerdict:
|
||||
"""一次块间判定的完整结果(判定 + 全部诊断量)。
|
||||
|
||||
属性:
|
||||
decision: 判定结果,取值为 continue / accept_confirmed /
|
||||
reject_directional / reject_futility / accept_provisional /
|
||||
reject_inertia 之一。
|
||||
e_value: 当前 e 值。
|
||||
wald_lambda: 当前 Wald 方向游走值。
|
||||
delta_hat: 点估计效应量 (W-L)/n_used;n_used=0 时为 0。
|
||||
delta_shrunk: 收缩点估计 (W-L)/(n_used+4),仅观测用。
|
||||
"""
|
||||
|
||||
decision: str
|
||||
e_value: float
|
||||
wald_lambda: float
|
||||
delta_hat: float
|
||||
delta_shrunk: float
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 2. 诊断类型
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpanMetrics:
|
||||
"""单次工具调用的输出质量指标。
|
||||
|
||||
属性:
|
||||
step: 工具调用所在的步骤编号。
|
||||
tool_name: 本次调用使用的工具名称。
|
||||
extraction_completeness: 信息提取完整度。
|
||||
hallucination_rate: 幻觉内容占比。
|
||||
missed_info_tags: 未提取信息的标签列表。
|
||||
hallucination_tags: 幻觉内容的标签列表。
|
||||
"""
|
||||
|
||||
step: int
|
||||
tool_name: str
|
||||
extraction_completeness: float
|
||||
hallucination_rate: float
|
||||
missed_info_tags: list[str] = field(default_factory=list)
|
||||
hallucination_tags: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkillStepAdherence:
|
||||
"""单个 skill step 的遵循判定。
|
||||
|
||||
属性:
|
||||
step_label: 被判定的步骤标签。
|
||||
adhered: 该步骤是否被遵循。
|
||||
description: 对遵循情况的文字说明。
|
||||
"""
|
||||
|
||||
step_label: str
|
||||
adhered: bool
|
||||
description: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QuestionMetrics:
|
||||
"""单题的完整指标,即 Stage 1 输出。
|
||||
|
||||
包含 7 个规则指标和 5 类 judge 指标(span / missed / adherence /
|
||||
bias / sufficiency)。frozen=True 保证构造后不可变。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识。
|
||||
video_id: 对应视频唯一标识。
|
||||
task_type: 题目任务类型。
|
||||
correct: 该题最终是否答对。
|
||||
format_compliance: 输出格式遵循程度。
|
||||
budget_usage: 预算使用比例。
|
||||
confidence_calibration: 置信度校准结论。
|
||||
repeat_visit_rate: 重复访问节点的比例。
|
||||
search_keyword_repetition: 搜索关键词重复率。
|
||||
level_jump_pattern: 层级跳转模式描述。
|
||||
tool_usage: 各工具的调用次数统计。
|
||||
span_metrics: 该题全部工具调用的片段级质量指标。
|
||||
missed_nodes: 该题遗漏的节点列表。
|
||||
skill_adherence: 该题对 skill 步骤的遵循情况。
|
||||
confirmation_bias: 是否出现确认偏误。None 表示 judge 不可用。
|
||||
evidence_sufficient: 当前证据是否充足。None 表示 judge 不可用。
|
||||
degraded: 是否为降级指标(judge 解析失败时生成)。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
task_type: str
|
||||
correct: bool
|
||||
format_compliance: float
|
||||
budget_usage: float
|
||||
confidence_calibration: str
|
||||
repeat_visit_rate: float
|
||||
search_keyword_repetition: float
|
||||
level_jump_pattern: str
|
||||
tool_usage: dict[str, int]
|
||||
span_metrics: list[SpanMetrics]
|
||||
missed_nodes: list[str]
|
||||
skill_adherence: list[SkillStepAdherence]
|
||||
confirmation_bias: bool | None
|
||||
evidence_sufficient: bool | None
|
||||
degraded: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ErrorAttribution:
|
||||
"""D1 错误归因。
|
||||
|
||||
属性:
|
||||
question_id: 发生错误归因的题目唯一标识。
|
||||
error_type: 错误的主要类别。
|
||||
reasoning_failure_type: 推理失败类型;若不适用则为 None。
|
||||
cause_category: C3 病因:'defect'/'lapse';正确题/INFRA/未判为 None。
|
||||
lapse_note: LAPSE 提醒文本(供 appendix 路由);非 LAPSE 为 None。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
error_type: str
|
||||
reasoning_failure_type: str | None
|
||||
cause_category: str | None = None
|
||||
lapse_note: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaseSample:
|
||||
"""单个案例样本,进化模块的最小输入单元。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识。
|
||||
video_id: 对应视频唯一标识。
|
||||
task_type: 题目任务类型。
|
||||
question: 题目文本。
|
||||
options: 选项列表。
|
||||
answer: 正确答案。
|
||||
prediction: Agent 预测答案。
|
||||
correct: 是否答对。
|
||||
error_type: 错误类型;正确题为 None。
|
||||
selection_reason: 被选为案例的原因说明。
|
||||
metrics: QuestionMetrics 的关键字段子集。
|
||||
trace: 完整推理轨迹,不截断。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
task_type: str
|
||||
question: str
|
||||
options: list[str]
|
||||
answer: str
|
||||
prediction: str | None
|
||||
correct: bool
|
||||
error_type: str | None
|
||||
selection_reason: str
|
||||
metrics: dict[str, Any]
|
||||
trace: list[dict[str, Any]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkillCasePack:
|
||||
"""单个 task_type 的案例包,服务于 Skill 进化。
|
||||
|
||||
属性:
|
||||
task_type: 题目任务类型。
|
||||
target_file: 对应 skill 文件名,如 'temporal-reasoning.md'。
|
||||
stats: 从 D3/D4 提取的该题型统计。
|
||||
failure_cases: 失败案例列表。
|
||||
success_cases: 成功案例列表。
|
||||
lapse_notes: C3 LAPSE 提醒文本列表(路由进 appendix 受保护区)。
|
||||
"""
|
||||
|
||||
task_type: str
|
||||
target_file: str
|
||||
stats: dict[str, Any]
|
||||
failure_cases: list[CaseSample] = field(default_factory=list)
|
||||
success_cases: list[CaseSample] = field(default_factory=list)
|
||||
lapse_notes: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SystemCasePack:
|
||||
"""跨题型行为模式案例包,服务于 System Prompt 进化。
|
||||
|
||||
属性:
|
||||
stats: 从 D5 提取的行为模式统计。
|
||||
failure_cases: 失败案例列表。
|
||||
success_cases: 成功案例列表。
|
||||
"""
|
||||
|
||||
stats: dict[str, Any]
|
||||
failure_cases: list[CaseSample] = field(default_factory=list)
|
||||
success_cases: list[CaseSample] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ToolCasePack:
|
||||
"""单个 tool_name 的案例包,服务于 Tool Prompt 进化。
|
||||
|
||||
属性:
|
||||
tool_name: 工具名称。
|
||||
target_files: 对应 prompt 文件名列表。
|
||||
stats: 从 D2 提取的工具质量统计。
|
||||
failure_spans: 失败 span 案例列表。
|
||||
success_spans: 成功 span 案例列表。
|
||||
"""
|
||||
|
||||
tool_name: str
|
||||
target_files: list[str]
|
||||
stats: dict[str, Any]
|
||||
failure_spans: list[dict[str, Any]] = field(default_factory=list)
|
||||
success_spans: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiagnosisResult:
|
||||
"""完整诊断报告,即两阶段诊断管线的最终输出。
|
||||
|
||||
属性:
|
||||
run_id: 本次诊断运行的唯一标识。
|
||||
filter_summary: 筛选条件与筛选结果摘要。
|
||||
error_attributions: 错误归因结果列表。
|
||||
attribution_distribution: 各归因类别的分布统计。
|
||||
reasoning_failure_types: 各推理失败类型的分布统计。
|
||||
tool_quality: 按工具聚合的质量分析结果。
|
||||
search_effectiveness: 搜索有效性的聚合统计。
|
||||
skill_compliance: 技能遵循情况的聚合统计。
|
||||
decision_patterns: 决策模式与行为模式摘要。
|
||||
skill_case_packs: 按题型组织的 Skill 进化案例包。
|
||||
system_case_pack: 跨题型行为模式案例包;无系统性问题时为 None。
|
||||
tool_case_packs: 按工具名组织的 Tool Prompt 进化案例包。
|
||||
infra_excluded_count: C3:被 stop_reason 排除的题数。
|
||||
infra_excluded_ratio: INFRA 占总题数比例。
|
||||
infra_question_ids: 被排除题 question_id 列表。
|
||||
defect_count: 进入诊断池错题中判为 DEFECT 的数量。
|
||||
lapse_count: 进入诊断池错题中判为 LAPSE 的数量。
|
||||
degraded_count: judge 解析失败而降级的题数。
|
||||
degraded_question_ids: 降级题的 question_id 列表。
|
||||
"""
|
||||
|
||||
run_id: str
|
||||
filter_summary: dict[str, Any] = field(default_factory=dict)
|
||||
error_attributions: list[ErrorAttribution] = field(default_factory=list)
|
||||
attribution_distribution: dict[str, int] = field(default_factory=dict)
|
||||
reasoning_failure_types: dict[str, int] = field(default_factory=dict)
|
||||
tool_quality: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
search_effectiveness: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
skill_compliance: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
decision_patterns: dict[str, Any] = field(default_factory=dict)
|
||||
skill_case_packs: dict[str, SkillCasePack] = field(default_factory=dict)
|
||||
system_case_pack: SystemCasePack | None = None
|
||||
tool_case_packs: dict[str, ToolCasePack] = field(default_factory=dict)
|
||||
infra_excluded_count: int = 0
|
||||
infra_excluded_ratio: float = 0.0
|
||||
infra_question_ids: list[str] = field(default_factory=list)
|
||||
defect_count: int = 0
|
||||
lapse_count: int = 0
|
||||
degraded_count: int = 0
|
||||
degraded_question_ids: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiagnosisSignalRow:
|
||||
"""单题诊断信号行,即 baseline run 逐题诊断的持久化单元。
|
||||
|
||||
供后续视频级切分选择器消费;由 (question_id, baseline_run_id,
|
||||
diag_fingerprint) 唯一确定,逐题 upsert 支持断点续跑。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识。
|
||||
video_id: 对应视频唯一标识。
|
||||
baseline_run_id: 产出该信号的 baseline run 标识。
|
||||
diag_fingerprint: 诊断口径指纹,隔离不同诊断配置的信号。
|
||||
task_type: 题目任务类型。
|
||||
error_type: 错误类别(extraction/search/reasoning/mixed);
|
||||
T0/uncertain 行为 None。
|
||||
cause_category: 病因类别(defect/lapse);不适用为 None。
|
||||
tier: 诊断分层(T0/T1/T2/uncertain)。
|
||||
evolution_target: 进化目标(tool/skill/system);
|
||||
error_type 为 None 时亦为 None。
|
||||
degraded: 是否为降级信号(judge 解析失败时生成)。
|
||||
infra: 是否为 INFRA 护栏排除行。
|
||||
session_id: 关联的会话标识;不适用为 None。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
baseline_run_id: str
|
||||
diag_fingerprint: str
|
||||
task_type: str
|
||||
error_type: str | None
|
||||
cause_category: str | None
|
||||
tier: str
|
||||
evolution_target: str | None
|
||||
degraded: bool
|
||||
infra: bool
|
||||
session_id: str | None
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 3. 进化类型
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvolutionRecord:
|
||||
"""单个目标文件的一次进化记录。
|
||||
|
||||
构建过程中需要多次修改状态(如 status、result_version),
|
||||
因此是唯一不使用 frozen=True 的类型。
|
||||
|
||||
属性:
|
||||
target_file: 目标文件名,如 'temporal-reasoning.md'。
|
||||
target_type: 目标类型: 'skill' / 'system' / 'tool'。
|
||||
original_content: 改写前原文。
|
||||
evolved_content: 改写后内容;rejected 时与 original_content 相同。
|
||||
reason: 状态说明。
|
||||
status: 'accepted' / 'rejected' / 'skipped'。
|
||||
source_version: 改写前版本号,如 'v1'。
|
||||
result_version: 改写后版本号;rejected/skipped 时为 None。
|
||||
suggestions: LLM 输出的改动建议列表。
|
||||
attempts: 每次 LLM 调用的原始响应摘要。
|
||||
validation_errors: 验证失败的具体原因。
|
||||
edits: LLM 输出的补丁列表。
|
||||
apply_report: 补丁逐条应用状态。
|
||||
clip_info: 超预算裁剪信息。
|
||||
"""
|
||||
|
||||
target_file: str
|
||||
target_type: str
|
||||
original_content: str
|
||||
evolved_content: str
|
||||
reason: str
|
||||
status: str
|
||||
source_version: str
|
||||
result_version: str | None = None
|
||||
suggestions: list[dict[str, Any]] = field(default_factory=list)
|
||||
attempts: list[dict[str, Any]] = field(default_factory=list)
|
||||
validation_errors: list[str] = field(default_factory=list)
|
||||
edits: list[dict[str, Any]] = field(default_factory=list)
|
||||
apply_report: list[dict[str, Any]] = field(default_factory=list)
|
||||
clip_info: dict[str, Any] = field(default_factory=lambda: {"triggered": False, "clipped": 0})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RejectedEdit:
|
||||
"""已在验证阶段证明无效的历史改法摘要。
|
||||
|
||||
属性:
|
||||
target_file: 目标文件名,如 'temporal-reasoning.md'。
|
||||
target_type: 目标类型: 'skill' / 'system' / 'tool'。
|
||||
change_summary: 被验证为无效的改法摘要。
|
||||
delta: 该改法对应候选相对基线的准确率变化。
|
||||
source_version: 该改法来源的版本号,如 'v2'。
|
||||
epoch: 该改法所属的进化轮次。
|
||||
gate_w: CE-Gate 证据:配对翻转 W(基线错到候选对)。
|
||||
gate_l: CE-Gate 证据:配对翻转 L(基线对到候选错)。
|
||||
gate_e_value: CE-Gate 证据:终态 e 值。
|
||||
gate_delta_shrunk: CE-Gate 证据:收缩效应量(观测用)。
|
||||
"""
|
||||
|
||||
target_file: str
|
||||
target_type: str
|
||||
change_summary: str
|
||||
delta: float
|
||||
source_version: str
|
||||
epoch: int
|
||||
gate_w: int | None = None
|
||||
gate_l: int | None = None
|
||||
gate_e_value: float | None = None
|
||||
gate_delta_shrunk: float | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvolutionResult:
|
||||
"""一次整体进化流程的汇总结果。
|
||||
|
||||
由 app/harness/ 编排层组装。不含 skills_version / prompts_version
|
||||
(版本管理是 app/ 职责,不属于 core/ 决策内核)。
|
||||
|
||||
属性:
|
||||
records: 所有目标的进化记录。
|
||||
accepted_count: 通过验证的改写数。
|
||||
rejected_count: 未通过验证的改写数。
|
||||
skipped_count: 因无失败案例而跳过的目标数。
|
||||
"""
|
||||
|
||||
records: list[EvolutionRecord] = field(default_factory=list)
|
||||
accepted_count: int = 0
|
||||
rejected_count: int = 0
|
||||
skipped_count: int = 0
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 4. 验证辅助类型
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PairResult:
|
||||
"""块验证配对比对结果。
|
||||
|
||||
属性:
|
||||
w: 基线错、候选对的翻转数。
|
||||
l: 基线对、候选错的翻转数。
|
||||
observed: 每题的 (基线是否正确, 候选是否正确) 记录。
|
||||
"""
|
||||
|
||||
w: int
|
||||
l: int # noqa: E741 — 数学记号 W/L(win/loss),与 gate.py 一致
|
||||
observed: dict[str, tuple[bool, bool]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QuadrantClassification:
|
||||
"""块验证四象限分类。
|
||||
|
||||
属性:
|
||||
improvements: 基线错、候选对的题目 ID 列表。
|
||||
regressions: 基线对、候选错的题目 ID 列表。
|
||||
persistent_fails: 两臂均错的题目 ID 列表。
|
||||
stable_successes: 两臂均对的题目 ID 列表。
|
||||
"""
|
||||
|
||||
improvements: list[str]
|
||||
regressions: list[str]
|
||||
persistent_fails: list[str]
|
||||
stable_successes: list[str]
|
||||
|
||||
|
||||
# =========================================================================
|
||||
# 5. Prompt 模板束
|
||||
# =========================================================================
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiagnosePrompts:
|
||||
"""诊断管线所需的全部固定模板束。
|
||||
|
||||
由调用方加载后以 frozen dataclass 传入,避免 core/ 依赖文件系统。
|
||||
|
||||
属性:
|
||||
defect_vs_lapse: defect/lapse 病因判别模板。
|
||||
reasoning_sub: 推理失败子分类模板。
|
||||
span_eval_system: span 评估系统提示模板。
|
||||
missed_nodes: 遗漏节点检测模板。
|
||||
skill_adherence: 技能遵循判定模板。
|
||||
confirmation_bias: 确认偏误检测模板。
|
||||
evidence_sufficiency: 证据充足性判定模板。
|
||||
"""
|
||||
|
||||
defect_vs_lapse: str
|
||||
reasoning_sub: str
|
||||
span_eval_system: str
|
||||
missed_nodes: str
|
||||
skill_adherence: str
|
||||
confirmation_bias: str
|
||||
evidence_sufficiency: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvolvePrompts:
|
||||
"""进化引擎所需的全部固定模板束。
|
||||
|
||||
由调用方加载后以 frozen dataclass 传入,避免 core/ 依赖文件系统。
|
||||
|
||||
属性:
|
||||
evolve_skill: Skill 进化提示模板。
|
||||
evolve_system: System Prompt 进化提示模板。
|
||||
evolve_tool: Tool Prompt 进化提示模板。
|
||||
evolve_rank: 编辑排序提示模板。
|
||||
"""
|
||||
|
||||
evolve_skill: str
|
||||
evolve_system: str
|
||||
evolve_tool: str
|
||||
evolve_rank: str
|
||||
@@ -0,0 +1,88 @@
|
||||
"""core/evolution/validate.py — 块验证纯决策函数。
|
||||
|
||||
算法 #7(块顺序验证)的局部实现:pair_block 按 unit 比对基线与候选、
|
||||
classify_quadrants 四象限分类、compute_accuracy 纯算术准确率。
|
||||
|
||||
三个函数均为纯函数,无副作用、无外部依赖。输入的对错映射均为 **unit 口径**
|
||||
(unit_id → 单元级正确性,AR pair 已在上游经 unit_correctness_view 双向 AND
|
||||
折叠),保证 e-process W/L 与准确率分母按单元计、不被 P/Q 单题计分污染
|
||||
(核心算法保真 #5:信息阶梯口径从 question_id 迁至 unit_id)。
|
||||
"""
|
||||
|
||||
from core.evolution.types import PairResult, QuadrantClassification
|
||||
|
||||
|
||||
def pair_block(
|
||||
baseline: dict[str, bool],
|
||||
candidate: dict[str, bool],
|
||||
unit_ids: list[str],
|
||||
) -> PairResult:
|
||||
"""按单元比对基线与候选对错,统计翻转。
|
||||
|
||||
参数:
|
||||
baseline: 基线臂单元级正确性映射(unit_id → bool)。
|
||||
candidate: 候选臂单元级正确性映射(unit_id → bool)。
|
||||
unit_ids: 参与比对的单元 ID 列表(AR pair 折叠后为单一 unit_id)。
|
||||
|
||||
返回:
|
||||
PairResult,包含 w(基线错→候选对翻转数)、l(基线对→候选错翻转数)
|
||||
和 observed(每单元的 (基线, 候选) 对错记录)。
|
||||
"""
|
||||
w = l = 0 # noqa: E741 — 数学记号 W/L(win/loss),与 gate.py 一致
|
||||
observed: dict[str, tuple[bool, bool]] = {}
|
||||
for uid in unit_ids:
|
||||
b, c = baseline[uid], candidate[uid]
|
||||
observed[uid] = (b, c)
|
||||
if not b and c:
|
||||
w += 1
|
||||
elif b and not c:
|
||||
l += 1 # noqa: E741
|
||||
return PairResult(w=w, l=l, observed=observed)
|
||||
|
||||
|
||||
def classify_quadrants(
|
||||
observed: dict[str, tuple[bool, bool]],
|
||||
) -> QuadrantClassification:
|
||||
"""按 (baseline, candidate) 四组分类,各组内 sorted。
|
||||
|
||||
参数:
|
||||
observed: 每题的 (基线是否正确, 候选是否正确) 记录。
|
||||
|
||||
返回:
|
||||
QuadrantClassification,四个象限各含排序后的题目 ID 列表。
|
||||
"""
|
||||
improvements: list[str] = []
|
||||
regressions: list[str] = []
|
||||
persistent_fails: list[str] = []
|
||||
stable_successes: list[str] = []
|
||||
for qid, (prev, curr) in observed.items():
|
||||
if not prev and curr:
|
||||
improvements.append(qid)
|
||||
elif prev and not curr:
|
||||
regressions.append(qid)
|
||||
elif not prev and not curr:
|
||||
persistent_fails.append(qid)
|
||||
else:
|
||||
stable_successes.append(qid)
|
||||
return QuadrantClassification(
|
||||
improvements=sorted(improvements),
|
||||
regressions=sorted(regressions),
|
||||
persistent_fails=sorted(persistent_fails),
|
||||
stable_successes=sorted(stable_successes),
|
||||
)
|
||||
|
||||
|
||||
def compute_accuracy(
|
||||
correctness: dict[str, bool],
|
||||
unit_ids: list[str],
|
||||
) -> float:
|
||||
"""纯算术:sum(correct) / len(units),分母按单元数(非逐题)。
|
||||
|
||||
参数:
|
||||
correctness: 单元级正确性映射(unit_id → bool)。
|
||||
unit_ids: 参与计算的单元 ID 列表。
|
||||
|
||||
返回:
|
||||
准确率浮点数。unit_ids 为空时抛出 ZeroDivisionError。
|
||||
"""
|
||||
return sum(correctness[uid] for uid in unit_ids) / len(unit_ids)
|
||||
+8
-3
@@ -4,12 +4,15 @@ LLMProvider / VLMProvider / TelemetryRecorder 是跨子包共享接口,
|
||||
被 core/agent/、core/evolution/、app/ 各模块引用。
|
||||
adapters/ 提供具体实现。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
|
||||
|
||||
from core.types import LLMResponse
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from core.types import LLMResponse
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
@@ -22,6 +25,7 @@ class LLMProvider(Protocol):
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse: ...
|
||||
|
||||
|
||||
@@ -36,6 +40,7 @@ class VLMProvider(Protocol):
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
parent_call_id: str | None = None,
|
||||
cache_salt: str | None = None,
|
||||
) -> LLMResponse: ...
|
||||
|
||||
|
||||
|
||||
+159
-1
@@ -1,7 +1,12 @@
|
||||
"""跨模块共享类型。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path as _Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -22,3 +27,156 @@ class LLMResponse:
|
||||
max_inter_token_ms: float | None
|
||||
cache_hit: bool
|
||||
call_id: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GeneratedQuestion:
|
||||
"""单条生成/加载的题目。
|
||||
|
||||
跨层共享类型,被 core/evolution/ 和 app/harness/、app/question_gen/ 使用。
|
||||
frozen=True 确保题目不可变。
|
||||
|
||||
属性:
|
||||
question_id: 题目唯一标识。
|
||||
video_id: 所属视频标识。
|
||||
task_type: 题型(如 "Action Reasoning")。
|
||||
question: 题目文本。
|
||||
options: 选项元组(如 ("A. ...", "B. ...", "C. ...", "D. ..."))。
|
||||
answer: 正确答案字母(如 "B")。
|
||||
source_nodes: 来源节点 ID 元组。
|
||||
difficulty: 难度等级。
|
||||
family: 问题家族名称(如 "RETRIEVAL",v2 出题管线使用,None 表示未指定)。
|
||||
skill_target: 目标技能标识(v2 出题管线使用,None 表示未指定)。
|
||||
difficulty_steps: 推理步数估计(v2 出题管线使用,None 表示未指定)。
|
||||
sub_pattern: 出题子模式标识(AR 特化策略使用,None 表示无)。
|
||||
unit_id: 所属题目单元标识;缺省时 __post_init__ 回填为 pair_id 或
|
||||
question_id,保证 single 题的 unit_id 等于自身 question_id。
|
||||
pair_id: 孪生对标识;同一对的 original/mirror 共享该值,None 表示非配对题。
|
||||
question_role: 在单元内的角色("single" | "pair_original" | "pair_mirror")。
|
||||
flip_axis: 孪生对的翻转轴(如 "before_after"),None 表示无翻转。
|
||||
"""
|
||||
|
||||
question_id: str
|
||||
video_id: str
|
||||
task_type: str
|
||||
question: str
|
||||
options: tuple[str, ...]
|
||||
answer: str
|
||||
source_nodes: tuple[str, ...]
|
||||
difficulty: str
|
||||
family: str | None = field(default=None)
|
||||
skill_target: str | None = field(default=None)
|
||||
difficulty_steps: int | None = field(default=None)
|
||||
sub_pattern: str | None = field(default=None)
|
||||
unit_id: str = ""
|
||||
pair_id: str | None = field(default=None)
|
||||
question_role: str = "single"
|
||||
flip_axis: str | None = field(default=None)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""回填 unit_id:缺省时取 pair_id(配对题)或 question_id(single 题)。
|
||||
|
||||
frozen dataclass 无法直接赋值,故通过 object.__setattr__ 绕过不可变约束。
|
||||
"""
|
||||
if not self.unit_id:
|
||||
object.__setattr__(self, "unit_id", self.pair_id or self.question_id)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QuestionUnit:
|
||||
"""题目单元:贯穿评测/训练 harness 的最小不可分调度契约实体。
|
||||
|
||||
single 题为 1 题单元,AR pair 孪生对为 2 题单元(original + mirror),
|
||||
两条题目必须作为整体被批处理/推理/评测,保证配对指标(collapse 等)可算。
|
||||
frozen=True 确保单元不可变。
|
||||
|
||||
属性:
|
||||
kind: 单元类型("single" | "pair")。
|
||||
unit_id: 单元唯一标识;single 取题目 unit_id,pair 取共享 pair_id。
|
||||
task_type: 单元题型;pair 内两题题型必须一致。
|
||||
questions: 单元内题目元组(single 为 1 条,pair 为 2 条)。
|
||||
unit_hash: P/Q payload 合成 hash,用于断点续跑失效检测(T11 消费)。
|
||||
"""
|
||||
|
||||
kind: str
|
||||
unit_id: str
|
||||
task_type: str
|
||||
questions: tuple[GeneratedQuestion, ...]
|
||||
unit_hash: str = ""
|
||||
|
||||
@property
|
||||
def size(self) -> int:
|
||||
"""单元内题目数量(single=1,pair=2)。"""
|
||||
return len(self.questions)
|
||||
|
||||
@classmethod
|
||||
def from_single(cls, q: GeneratedQuestion) -> QuestionUnit:
|
||||
"""由单条题目构造 single 单元。
|
||||
|
||||
参数:
|
||||
q: 待封装的题目。
|
||||
|
||||
返回:
|
||||
kind="single" 的单元,unit_id 取 q.unit_id。
|
||||
"""
|
||||
return cls("single", q.unit_id, q.task_type, (q,))
|
||||
|
||||
@classmethod
|
||||
def from_pair(cls, original: GeneratedQuestion, mirror: GeneratedQuestion) -> QuestionUnit:
|
||||
"""由孪生对(original + mirror)构造 pair 单元。
|
||||
|
||||
参数:
|
||||
original: 原始题(question_role="pair_original")。
|
||||
mirror: 镜像题(question_role="pair_mirror")。
|
||||
|
||||
返回:
|
||||
kind="pair" 的单元,unit_id 取共享 pair_id。
|
||||
|
||||
关键实现:
|
||||
断言两题共享非空 pair_id、且 video_id/task_type/flip_axis 一致,
|
||||
确保只有合法孪生对才能聚合成对,非法配对直接报错而非静默兜底。
|
||||
"""
|
||||
assert original.pair_id and original.pair_id == mirror.pair_id
|
||||
assert original.video_id == mirror.video_id and original.task_type == mirror.task_type
|
||||
assert original.flip_axis == mirror.flip_axis
|
||||
return cls("pair", original.pair_id, original.task_type, (original, mirror))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PoolConfig:
|
||||
"""池构建策略的统一配置。
|
||||
|
||||
两组字段由两个具体策略各取所需,未使用的字段被忽略。
|
||||
|
||||
属性:
|
||||
task_types: 限定题型元组;None 表示全部类别。
|
||||
seed: 随机种子,保证可复现。
|
||||
baseline_run_id: 基线 run 标识。
|
||||
diag_size: 诊断池大小(GlobalStrategy 用)。
|
||||
diag_correct_ratio: 诊断池中对题占比(GlobalStrategy 用)。
|
||||
val_size: 验证池大小(GlobalStrategy 用)。
|
||||
val_correct_ratio: 验证池中对题占比(GlobalStrategy 用)。
|
||||
test_size: held-out 测试池大小(GlobalStrategy 用)。
|
||||
eval_min_per_class: 验证池中每类保底样本数(GlobalStrategy 用)。
|
||||
train_ratio: train/(train+val) 比例(PerCategoryStrategy 用)。
|
||||
test_questions_dir: 外部 test 题源路径(PerCategoryStrategy 用)。
|
||||
|
||||
实现细节:
|
||||
结果驱动视频级切分不复用本配置——它有独立的 VideoSplitConfig /
|
||||
SplitBuildConfig / SelectConfig(见 app/harness/video_split_cli.py 与
|
||||
split_selection.py),故本类不承载 n_trainval / floor_k / epsilon 等视频级
|
||||
切分旋钮,避免死配置面。
|
||||
"""
|
||||
|
||||
task_types: tuple[str, ...] | None
|
||||
seed: int
|
||||
baseline_run_id: str
|
||||
diag_size: int
|
||||
diag_correct_ratio: float
|
||||
val_size: int
|
||||
val_correct_ratio: float
|
||||
test_size: int
|
||||
eval_min_per_class: int
|
||||
train_ratio: float
|
||||
test_questions_dir: _Path | None
|
||||
batch_correct_ratio: float | None = None
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
"""CLI 入口 — Composition Root:构建适配器,注入 Runner,调度执行。
|
||||
|
||||
三层配置合并(YAML > .env > CLI)由 load_config 完成。
|
||||
适配器参数通过 InfraSettings(BaseSettings) 从 .env 加载。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class InfraSettings(BaseSettings):
|
||||
"""工程配置(少变/敏感),从 .env 加载。"""
|
||||
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
search_llm_model: str = ""
|
||||
search_llm_base_url: str = ""
|
||||
search_llm_api_key: str = ""
|
||||
|
||||
vl_llm_model: str = ""
|
||||
vl_llm_base_url: str = ""
|
||||
vl_llm_api_key: str = ""
|
||||
|
||||
evolve_llm_model: str = ""
|
||||
evolve_llm_base_url: str = ""
|
||||
evolve_llm_api_key: str = ""
|
||||
|
||||
embed_api_key: str = ""
|
||||
embed_api_url: str = ""
|
||||
|
||||
monkey_ocr_urls: str = ""
|
||||
redis_url: str = ""
|
||||
redis_cache_ttl: int = 86400
|
||||
|
||||
llm_timeout: float = 300.0
|
||||
llm_max_retries: int = 3
|
||||
llm_retry_base_delay: float = 20.0
|
||||
llm_retry_max_delay: float = 120.0
|
||||
llm_circuit_breaker_threshold: int = 48
|
||||
llm_circuit_breaker_cooldown: float = 60.0
|
||||
llm_ttft_timeout: float = 30.0
|
||||
llm_inter_token_timeout: float = 15.0
|
||||
|
||||
|
||||
class _Adapters(NamedTuple):
|
||||
"""全套适配器实例。"""
|
||||
|
||||
llm: object
|
||||
evolve_llm: object
|
||||
vlm: object
|
||||
telemetry: object
|
||||
embed: object
|
||||
ocr: object
|
||||
|
||||
|
||||
def _build_adapters(settings: InfraSettings, embed_cfg: dict) -> _Adapters:
|
||||
"""从 InfraSettings 构建全套适配器。
|
||||
|
||||
参数:
|
||||
settings: 工程配置。
|
||||
embed_cfg: 嵌入配置字典(来自 YAML embed 段)。
|
||||
|
||||
返回:
|
||||
_Adapters 命名元组。
|
||||
"""
|
||||
from adapters.breaker import CircuitBreaker
|
||||
from adapters.embedding import LocalEmbeddingProvider
|
||||
from adapters.llm import GovernedLLMClient
|
||||
from adapters.telemetry import SQLiteTelemetryRecorder
|
||||
from adapters.vlm import GovernedVLMClient
|
||||
|
||||
breaker = CircuitBreaker(
|
||||
fail_threshold=max(settings.llm_circuit_breaker_threshold, 1),
|
||||
cooldown_s=settings.llm_circuit_breaker_cooldown,
|
||||
)
|
||||
|
||||
cache = None
|
||||
if settings.redis_url:
|
||||
from adapters.redis_cache import RedisResponseCache, _resolve_cache_ttl
|
||||
|
||||
# 配置校验 fail-loud(不属于 Redis 连接故障,不得被下方降级 except 吞掉)
|
||||
ttl_s = _resolve_cache_ttl(settings.redis_cache_ttl)
|
||||
try:
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
redis_client = aioredis.from_url(settings.redis_url, decode_responses=True)
|
||||
cache = RedisResponseCache(redis=redis_client, ttl_s=ttl_s)
|
||||
except Exception:
|
||||
logger.warning("Redis 缓存不可用,降级为无缓存模式")
|
||||
|
||||
telemetry_db = Path("logs/telemetry.db")
|
||||
telemetry_db.parent.mkdir(parents=True, exist_ok=True)
|
||||
telemetry = SQLiteTelemetryRecorder(telemetry_db)
|
||||
|
||||
def _make_llm(model: str, base_url: str, api_key: str, *, thinking: bool) -> GovernedLLMClient:
|
||||
"""构建单个 GovernedLLMClient 实例。"""
|
||||
return GovernedLLMClient(
|
||||
model=model,
|
||||
base_url=base_url,
|
||||
api_key=api_key,
|
||||
provider=model.split("-")[0] if model else "unknown",
|
||||
thinking=thinking,
|
||||
breaker=breaker,
|
||||
cache=cache,
|
||||
telemetry=telemetry,
|
||||
timeout_s=settings.llm_timeout,
|
||||
ttft_timeout_s=settings.llm_ttft_timeout,
|
||||
inter_token_timeout_s=settings.llm_inter_token_timeout,
|
||||
max_retries=settings.llm_max_retries,
|
||||
retry_base_delay_s=settings.llm_retry_base_delay,
|
||||
retry_max_delay_s=settings.llm_retry_max_delay,
|
||||
)
|
||||
|
||||
llm = _make_llm(
|
||||
settings.search_llm_model,
|
||||
settings.search_llm_base_url,
|
||||
settings.search_llm_api_key,
|
||||
thinking=True,
|
||||
)
|
||||
evolve_llm = llm
|
||||
|
||||
vl_llm = _make_llm(
|
||||
settings.vl_llm_model,
|
||||
settings.vl_llm_base_url,
|
||||
settings.vl_llm_api_key,
|
||||
thinking=False,
|
||||
)
|
||||
vlm = GovernedVLMClient(governed_llm=vl_llm)
|
||||
|
||||
backend = embed_cfg.get("backend", "local")
|
||||
if backend == "local":
|
||||
embed = LocalEmbeddingProvider(
|
||||
model_name=embed_cfg.get("model_name", "BAAI/bge-base-zh-v1.5"),
|
||||
embed_dim=embed_cfg.get("embed_dim", 768),
|
||||
device=embed_cfg.get("device", "cpu"),
|
||||
)
|
||||
else:
|
||||
from adapters.embedding import RemoteEmbeddingProvider
|
||||
|
||||
embed = RemoteEmbeddingProvider(
|
||||
model_name=embed_cfg.get("model_name", ""),
|
||||
embed_dim=embed_cfg.get("embed_dim", 768),
|
||||
api_key=settings.embed_api_key,
|
||||
api_url=settings.embed_api_url,
|
||||
)
|
||||
|
||||
ocr = None
|
||||
if settings.monkey_ocr_urls:
|
||||
from adapters.ocr import MonkeyOCRClient
|
||||
|
||||
urls = [u.strip() for u in settings.monkey_ocr_urls.split(",") if u.strip()]
|
||||
if urls:
|
||||
ocr = MonkeyOCRClient(urls=urls)
|
||||
|
||||
return _Adapters(
|
||||
llm=llm,
|
||||
evolve_llm=evolve_llm,
|
||||
vlm=vlm,
|
||||
telemetry=telemetry,
|
||||
embed=embed,
|
||||
ocr=ocr,
|
||||
)
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
"""构建 CLI 参数解析器。所有参数 default=None,未传入时使用 YAML 默认值。"""
|
||||
parser = argparse.ArgumentParser(description="Video-Tree-TRM5 实验运行器")
|
||||
parser.add_argument("--config", type=Path, default=Path("config/default.yaml"))
|
||||
parser.add_argument("--workspace-dir", type=Path, dest="workspace_dir")
|
||||
parser.add_argument("--store-dir", type=Path, dest="store_dir")
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["infer", "train", "diagnose", "evolve", "eval", "promote"],
|
||||
)
|
||||
parser.add_argument("--run-id", type=str, dest="run_id")
|
||||
parser.add_argument("--concurrency", type=int)
|
||||
parser.add_argument("--max-steps", type=int, dest="max_steps")
|
||||
parser.add_argument(
|
||||
"--skill-mode",
|
||||
choices=["auto", "manual", "none"],
|
||||
dest="skill_mode",
|
||||
)
|
||||
parser.add_argument("--n-samples", type=int, dest="n_samples")
|
||||
parser.add_argument("--questions", type=str)
|
||||
parser.add_argument("--skills-version", type=str, dest="skills_version")
|
||||
parser.add_argument("--prompts-version", type=str, dest="prompts_version")
|
||||
parser.add_argument("--task-types", nargs="+", dest="task_types")
|
||||
parser.add_argument("--resume", action="store_true", dest="resume")
|
||||
parser.add_argument("--fresh", action="store_true", dest="fresh")
|
||||
parser.add_argument("--seed", type=str, dest="seed")
|
||||
parser.add_argument("--epochs", type=int)
|
||||
parser.add_argument(
|
||||
"--pool-split-mode",
|
||||
choices=["global", "per_category"],
|
||||
dest="pool_split_mode",
|
||||
)
|
||||
parser.add_argument("--train-ratio", type=float, dest="train_ratio")
|
||||
parser.add_argument("--test-questions", type=str, dest="test_questions")
|
||||
parser.add_argument(
|
||||
"--no-run-holdout-eval",
|
||||
action="store_true",
|
||||
dest="no_run_holdout_eval",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _log_result(result: object) -> None:
|
||||
"""输出推理结果摘要。"""
|
||||
logger.info("=" * 60)
|
||||
logger.info("运行 ID: {}", result.run_id)
|
||||
logger.info(
|
||||
"总体准确率: {:.2%} ({}/{})",
|
||||
result.accuracy,
|
||||
result.correct,
|
||||
result.total,
|
||||
)
|
||||
logger.info("平均步数: {:.1f}", result.steps_mean)
|
||||
logger.info(
|
||||
"Token 用量: prompt={}, completion={}",
|
||||
result.token_usage["prompt_tokens"],
|
||||
result.token_usage["completion_tokens"],
|
||||
)
|
||||
if result.per_task_type:
|
||||
logger.info("--- 按任务类型 ---")
|
||||
for task_type, stats in sorted(result.per_task_type.items()):
|
||||
logger.info(
|
||||
" {}: {:.2%} ({}/{})",
|
||||
task_type,
|
||||
stats["accuracy"],
|
||||
stats["correct"],
|
||||
stats["total"],
|
||||
)
|
||||
logger.info(
|
||||
"停止原因: {}",
|
||||
", ".join(f"{k}={v}" for k, v in result.stop_reason_counts.items()),
|
||||
)
|
||||
logger.info("=" * 60)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""入口函数。"""
|
||||
load_dotenv()
|
||||
|
||||
log_dir = Path("logs")
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.add(
|
||||
log_dir / "run_{time:YYYYMMDD_HHmmss}.log",
|
||||
rotation="500 MB",
|
||||
retention="30 days",
|
||||
encoding="utf-8",
|
||||
enqueue=False,
|
||||
)
|
||||
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
import yaml
|
||||
|
||||
with open(args.config, encoding="utf-8") as f:
|
||||
raw_yaml = yaml.safe_load(f)
|
||||
|
||||
from app.harness.config import load_config
|
||||
|
||||
cli_args = vars(args)
|
||||
if cli_args.get("task_types") is not None:
|
||||
cli_args["task_types"] = tuple(cli_args["task_types"])
|
||||
cli_overrides = {k: v for k, v in cli_args.items() if k != "config"}
|
||||
if cli_overrides.get("no_run_holdout_eval"):
|
||||
cli_overrides["run_holdout_eval"] = False
|
||||
cli_overrides.pop("no_run_holdout_eval", None)
|
||||
config = load_config(args.config, cli_overrides)
|
||||
logger.info("配置加载完成: mode={}, workspace={}", config.mode, config.workspace_dir)
|
||||
|
||||
settings = InfraSettings()
|
||||
embed_cfg = raw_yaml.get("embed", {})
|
||||
adapters = _build_adapters(settings, embed_cfg)
|
||||
|
||||
from app.harness.deps_router import InferenceDepsRouter
|
||||
from app.harness.runner import Runner
|
||||
|
||||
router = InferenceDepsRouter(
|
||||
store_dir=Path(config.store_dir),
|
||||
embed_provider=adapters.embed,
|
||||
llm=adapters.llm,
|
||||
vlm=adapters.vlm,
|
||||
ocr=adapters.ocr,
|
||||
default_prompts_dir=Path(config.store_dir) / "prompts" / config.prompts_version,
|
||||
default_skills_dir=Path(config.store_dir) / "skills" / config.skills_version,
|
||||
skill_mode=config.skill_mode,
|
||||
verify_vision=True,
|
||||
anchor=True,
|
||||
assemble_mode="ids_expand",
|
||||
)
|
||||
|
||||
runner = Runner(
|
||||
config,
|
||||
llm=adapters.llm,
|
||||
evolve_llm=adapters.evolve_llm,
|
||||
vlm=adapters.vlm,
|
||||
telemetry=adapters.telemetry,
|
||||
tool_dispatch_factory=router.create_dispatch,
|
||||
prompt_builder_factory=router.create_prompt_builder,
|
||||
)
|
||||
|
||||
if config.mode == "infer":
|
||||
result = asyncio.run(runner.infer(task_types=config.task_types))
|
||||
_log_result(result)
|
||||
elif config.mode == "train":
|
||||
from app.harness.pools import (
|
||||
GlobalPoolStrategy,
|
||||
PerCategoryPoolStrategy,
|
||||
build_or_load_pools,
|
||||
)
|
||||
from app.harness.workspace import resolve_paths
|
||||
|
||||
strategy = (
|
||||
PerCategoryPoolStrategy()
|
||||
if config.pool_split_mode == "per_category"
|
||||
else GlobalPoolStrategy()
|
||||
)
|
||||
paths = resolve_paths(config.workspace_dir)
|
||||
pools = build_or_load_pools(config, strategy, paths.db_path)
|
||||
asyncio.run(runner.train(pools))
|
||||
else:
|
||||
raise SystemExit(f"模式 {config.mode!r} 尚未实现")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,33 @@
|
||||
你是一个确认偏差检测器。你服务于一个诊断系统,该系统需要判断搜索 Agent 在回答视频问答题时是否表现出确认偏差——即只为自己倾向的选项搜集支持性证据,而忽略对竞争选项的独立验证。
|
||||
|
||||
## 你会收到的输入
|
||||
|
||||
1. 题目(问题文本 + 四个选项)
|
||||
2. Agent 的完整执行轨迹(每步的思考过程、工具调用和工具返回,重点关注 reflect.options 中对各选项的记录以及 search_similar 的 query 参数)
|
||||
|
||||
## 工作原则
|
||||
|
||||
确认偏差的核心特征是:Agent 的搜索行为在选项之间显著不均衡。一个健康的搜索过程应该至少为 1 个竞争选项做过独立搜索——不一定是每个选项都搜,但不能只围绕一个选项搜集证据。
|
||||
|
||||
具体检查以下信号:
|
||||
|
||||
观察 Agent 的 search_similar 查询。如果所有查询的关键词都指向同一个选项的内容(比如选项 B 说"烹饪教学",Agent 反复搜索"cooking""recipe""chef"),而从未用其他选项的关键词搜索(如选项 A 的"旅行"、选项 C 的"运动"),这是强偏差信号。
|
||||
|
||||
观察 Agent 的 reflect.options 变化。如果 Agent 在早期步骤就锁定了 best_candidate,且后续步骤中对其他选项的认知始终停留在"未知"或"待定",说明 Agent 没有为竞争选项投入搜索资源。
|
||||
|
||||
但需要注意:如果问题本身就指向特定内容(比如"视频中的厨师做了什么"),Agent 集中搜索厨师相关内容是合理的,不算偏差。偏差是指在选项之间的对比搜索不均衡,而非搜索主题的集中。
|
||||
|
||||
同样,如果 Agent 在前几步通过全局扫描(如顺序阅读 L1 节点)已经获得了足够信息来排除 2-3 个选项,之后集中搜索剩余选项是合理策略,不算偏差。
|
||||
|
||||
## 输出格式
|
||||
|
||||
请严格输出以下 JSON,不要包含其他文字:
|
||||
|
||||
```json
|
||||
{
|
||||
"has_bias": true,
|
||||
"evidence": "具体说明偏差表现:Agent 搜索了哪些关键词、为哪些选项搜集了证据、忽略了哪些选项"
|
||||
}
|
||||
```
|
||||
|
||||
如果没有偏差:`{"has_bias": false, "evidence": ""}`
|
||||
@@ -0,0 +1,12 @@
|
||||
你是一个失败归因裁判。你会收到一道答错题目的题面、正确答案、Agent 的错误预测、执行轨迹,以及 Agent 当时所用的 prompt 全文。你的唯一任务是判断:这次失败该归咎于 prompt 正文本身,还是 Agent 没有遵循已有的正确指令。
|
||||
|
||||
判别测试只有一句话:当前 prompt 里是否已经存在一条规则,只要 Agent 遵循它就能避免这次失败?
|
||||
|
||||
如果存在这样的规则(Agent 是忽略了、格式没按要求、或没执行该步),归为 lapse——这类问题不该改正文,只需记一条提醒。如果不存在这样的规则、或现有规则本身有误导,归为 defect——这类才需要修改 prompt 正文。
|
||||
|
||||
当你拿不准时,默认归为 lapse:宁可少改正文,也不要为一次偶发失误去删改一条本来正确的规则。
|
||||
|
||||
严格输出以下 JSON,不要包含其他文字:
|
||||
{"category": "defect" 或 "lapse", "note": "若为 lapse,写一句给 Agent 的提醒;defect 留空"}
|
||||
|
||||
note 只能重申当前 prompt 里已经存在的那条规则(让 Agent 别再忽略它),措辞要通用、可跨题复用。禁止把本题的题目内容、选项、正确答案或任何单题事实写进 note——note 不是案例记录,是规则提醒。
|
||||
@@ -0,0 +1,27 @@
|
||||
你是一个证据充分性评估器。你服务于一个诊断系统,该系统需要判断搜索 Agent 实际收集到的工具输出是否包含足够的信息来推导出正确答案。你不评估 Agent 的推理过程——只评估它收集到的原始材料。
|
||||
|
||||
## 你会收到的输入
|
||||
|
||||
1. 题目(问题文本 + 四个选项 + 正确答案)
|
||||
2. Agent 收到的全部工具输出(按步骤排列,包含每次 view_node、search_similar、observe_frame 的返回内容)
|
||||
|
||||
## 工作原则
|
||||
|
||||
你需要回答一个假设性问题:如果一个完美的推理者阅读了这些工具输出(且仅阅读这些工具输出),它能否推导出正确答案?
|
||||
|
||||
"推导出"不要求工具输出直接陈述答案。如果工具输出中包含了足够的事实片段,一个合理的推理链能将它们组合得出正确答案,就算充分。比如工具输出提到"厨师在切蔬菜"和"背景是一个厨房",虽然没有直接说"这是烹饪视频",但推导是合理的。
|
||||
|
||||
"不充分"是指工具输出中完全缺乏区分正确答案与最强干扰项的关键信息。比如问题问"视频中的运动是什么",选项有篮球和足球,但工具输出只提到"运动场上有人在运动",没有任何能区分篮球和足球的细节——这就是不充分。
|
||||
|
||||
注意区分两种情况:信息存在但分散(充分——完美推理者能整合)vs 信息真的不存在(不充分——无论怎么推理都无法得出)。
|
||||
|
||||
## 输出格式
|
||||
|
||||
请严格输出以下 JSON,不要包含其他文字:
|
||||
|
||||
```json
|
||||
{
|
||||
"sufficient": true,
|
||||
"reasoning": "简要说明工具输出中哪些信息支持正确答案,或缺乏哪些关键信息"
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,8 @@
|
||||
你是一个改动优先级裁判。你会收到一份当前 prompt 文件全文,和一组待应用的局部 edits(每条含 op/target/content)。由于本轮编辑预算有限,你只能保留其中最重要的若干条。
|
||||
|
||||
请只依据"对纠正失败、提升正确率的预期贡献"排序:优先保留直接修复失败模式的改动,其次保留收窄或澄清的改动,最后才是巩固已有成功的改动。删除类、简化类的精准改动通常优先于追加大段新内容。
|
||||
|
||||
每条 edit 会附带 support_count(该改动的支持案例数)。同等重要性下,support_count 更高的优先;但 support_count 低不等于该删,仍以修复贡献为主判据。
|
||||
|
||||
严格输出以下 JSON,不要包含其他文字:
|
||||
{"selected_indices": [按重要性降序排列的 0-based 索引]}
|
||||
@@ -0,0 +1,53 @@
|
||||
你是一个搜索策略改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的失败和成功案例来迭代改进搜索策略(Skill)。你的任务是基于案例包中的证据,改写当前 Skill 文件,使 Agent 在后续执行中避免相同的失败模式。
|
||||
|
||||
## 你会收到的输入
|
||||
|
||||
1. 当前 Skill 文件全文
|
||||
2. 失败案例:Agent 答错的题目,含完整推理轨迹、错误类型和诊断指标
|
||||
3. 成功案例:Agent 答对的题目,展示当前 Skill 中有效的模式
|
||||
4. 聚合统计:准确率、错误归因分布、搜索有效性指标、Skill 步骤遵循率
|
||||
5. (可能出现)上一轮被接受改动导致的回归题:这些题在上一版本答对、却被你上次的改写改错了,附基线与候选两份预测和推理轨迹
|
||||
6. (可能出现)黑名单:已被实测验证无效或有害的改法方向
|
||||
|
||||
## 工作原则
|
||||
|
||||
如果输入里出现了回归题,它的优先级高于一切。这些题在上一版本是对的,是你上次的改动把它们弄坏的,所以本次改写的第一要务是确保不再破坏它们——宁可在相关方向上回退或收窄,也不要为了拉高其它题而牺牲它们。更一般地,当你看到准确率下降这类负向信号时,默认先怀疑上次是不是加了过度、冲突或冗余的指令,优先简化、删除、收窄;只有确认简化解决不了问题,才考虑加强指令。黑名单里列出的改法已经被实测证明无效或有害,不要换个措辞把同一个方向再提一遍。
|
||||
|
||||
先分析失败案例中 Agent 的实际行为与 Skill 指令的偏差。偏差分两类:Skill 指令正确但 Agent 没遵循(遵循率问题),或 Skill 指令本身有误导(策略问题)。前者需要让指令更具体、更难被忽略;后者需要修改策略本身。
|
||||
|
||||
从成功案例中识别有效模式——这些模式在改写时必须保留。如果成功案例和失败案例采用了不同的策略路径,重点强化成功路径。
|
||||
|
||||
Skill 中引用的统计数据(如"search-first 正确率 75%")应根据案例包中的新统计更新。不要编造数据,只使用案例包中提供的数字。
|
||||
|
||||
你写进 Skill 的每一条规则都必须是可跨题复用的通用策略,而不是对某一道题的记答案。跨多个失败案例时只提取共性模式,抽象掉一切单题特征——具体题目内容、选项文字、步骤序号、某一帧的具体画面、某个具体答案都不许写进 Skill 正文。一条规则如果只在它来源的那道题上成立,就不要加。改写时优先简化与收窄:宁可让 Skill 更短,也不要堆叠只对个别题生效的硬性指令。
|
||||
|
||||
## 冻结区
|
||||
|
||||
以下内容不可修改,必须原样保留在改写后的文件中:
|
||||
- YAML frontmatter(`---` 之间的 name、description、task_type)
|
||||
- 输出格式中的 JSON 基础结构(reflect/plan/action 三个顶层字段)
|
||||
|
||||
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
|
||||
|
||||
## 输出格式
|
||||
|
||||
请严格输出以下 JSON,不要包含其他文字:
|
||||
|
||||
```json
|
||||
{
|
||||
"suggestions": [
|
||||
{
|
||||
"section": "改动目标段落的标题或位置描述",
|
||||
"problem": "失败案例中暴露的具体问题",
|
||||
"change": "具体的修改方向",
|
||||
"related_cases": ["关联的失败案例 question_id"],
|
||||
"support_count": 该建议的支持案例数(= related_cases 的数量)
|
||||
}
|
||||
],
|
||||
"edits": [
|
||||
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)", "support_count": 该改动的支持案例数}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
每条 edit 与每条 suggestion 都必须带 "support_count":本条改动由多少个失败案例共同支持(即 related_cases 的数量)。support_count 越高代表证据越充分;它只作排序参考,不是硬门槛——support_count 低不等于该删,仍以修复贡献为主判据。
|
||||
@@ -0,0 +1,53 @@
|
||||
你是一个系统级行为改进专家。你服务于一个自进化视频搜索系统,该系统通过分析 Agent 的跨题型行为模式来改进 System Prompt。你的任务是基于行为模式案例包中的证据,改写 System Prompt 中的策略性指令,纠正系统级行为问题。
|
||||
|
||||
## 你会收到的输入
|
||||
|
||||
1. 当前 System Prompt (system.md) 全文
|
||||
2. 失败案例:展示三类系统性行为问题的题目——过早提交(budget_usage < 0.3 就提交答案)、高置信答错(confidence 很高但答案错误)、确认偏误(只搜索支持初始判断的证据)
|
||||
3. 成功案例:行为校准良好的题目——置信度与正确率匹配,预算使用适中
|
||||
4. D5 行为模式统计:各行为模式的发生频率和分布
|
||||
5. (可能出现)黑名单:已被实测验证无效或有害的改法方向
|
||||
|
||||
## 工作原则
|
||||
|
||||
关注跨题型的系统性行为模式,而非某个具体题型的策略。失败案例中的行为偏差反映了 System Prompt 的决策原则不够清晰或不够强约束。黑名单里的改法已经被实测验证无效或有害,不要再朝同一个方向改一遍。
|
||||
|
||||
过早提交说明预算管理指令需要更强的约束语言。高置信答错说明置信度校准的语义定义需要调整。确认偏误说明竞争选项搜索的要求需要更明确。
|
||||
|
||||
当你看到失败案例与成功案例并存时,失败修复优先于巩固成功——先确保失败模式被纠正,再考虑强化已有的好行为。看到某类行为指标变差这类负向信号时,默认先怀疑上一轮是否加了过度、冲突或冗余的约束,优先简化、删除、收窄;只有确认简化解决不了,才考虑加强约束语言。
|
||||
|
||||
从成功案例中提取"好行为"的特征,在改写时强化这些特征的表述。
|
||||
|
||||
## 冻结区
|
||||
|
||||
以下 section 必须原样保留,不可修改任何文字:
|
||||
- `## 能力边界`(事实性描述,不是策略)
|
||||
- `## 输出格式`(JSON schema 是系统契约)
|
||||
- `## 视频树结构`(含信任层级,是数据结构事实描述)
|
||||
|
||||
可改写的 section:
|
||||
- `## 角色`(前两段的角色定位和行为倾向描述)
|
||||
- `## 决策原则`(搜索策略、预算分配建议)
|
||||
- 搜索工具使用、否定题原则、置信度语义
|
||||
|
||||
这次不要返回整份改写后的文件,而是只返回一组局部 edits。`append` 用来在文件末尾追加一个新 section,`insert_after` 用来把内容紧跟着插到某个锚点段落之后,`replace` 用来用新内容整体替换 target 对应的原文,`delete` 则直接删除 target 对应的原文并让 content 留空。target 必须是从当前文件里逐字复制出来的原文,而且要长到足以唯一定位;只要有任何一个字不完全匹配,这条改动就会被跳过。改动应尽量小而局部,优先做精确补丁,不要动辄重写整段整节;另外,冻结区里的文字绝不能作为 target。
|
||||
|
||||
## 输出格式
|
||||
|
||||
请严格输出以下 JSON,不要包含其他文字:
|
||||
|
||||
```json
|
||||
{
|
||||
"suggestions": [
|
||||
{
|
||||
"section": "改动目标段落的标题或位置描述",
|
||||
"problem": "失败案例中暴露的具体行为问题",
|
||||
"change": "具体的修改方向",
|
||||
"related_cases": ["关联的失败案例 question_id"]
|
||||
}
|
||||
],
|
||||
"edits": [
|
||||
{"op": "append|insert_after|replace|delete", "target": "锚点原文(append 留空)", "content": "新内容(delete 留空)"}
|
||||
]
|
||||
}
|
||||
```
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user