Compare commits
20 Commits
v1.3.10
...
dce063ebcd
| Author | SHA1 | Date | |
|---|---|---|---|
| dce063ebcd | |||
| 6850ed0d54 | |||
| a64c680f64 | |||
| cb1e08f1f3 | |||
| bb621cad4d | |||
| d80feafb93 | |||
| 6691e52195 | |||
| c4dd403e0f | |||
| 42b7f0defc | |||
| 80bb8aba71 | |||
| 61cace71c4 | |||
| 0357d5afb3 | |||
| 3db7d9979c | |||
| c13c4fd28f | |||
| 398b16e136 | |||
| b31df842a8 | |||
| 944b3f3798 | |||
| 1f08194b96 | |||
| fb67b68440 | |||
| 4117eae501 |
@@ -193,6 +193,15 @@ PGW_TELEMETRY_BACKEND=none # sqlite | postgres | none(必填)
|
|||||||
# OCR__MONKEY__2__TIMEOUT_S=300
|
# OCR__MONKEY__2__TIMEOUT_S=300
|
||||||
# OCR__MONKEY__2__TRUST_ENV=false
|
# OCR__MONKEY__2__TRUST_ENV=false
|
||||||
# OCR__RETRY__MAX_ATTEMPTS=3 # per-scope 韧性键与 LLM scope 同一套
|
# OCR__RETRY__MAX_ATTEMPTS=3 # per-scope 韧性键与 LLM scope 同一套
|
||||||
|
# 整份文档解析(parse_document,1.3.11):建议用**独立 scope** 指向同一服务,自带超时/重试/期限:
|
||||||
|
# OCR_DOC__MONKEY__1__BASE_URL=http://10.77.0.20:7866
|
||||||
|
# OCR_DOC__MONKEY__1__API_KEY=none
|
||||||
|
# OCR_DOC__MONKEY__1__MODEL=monkey-ocr
|
||||||
|
# OCR_DOC__MONKEY__1__TIMEOUT_S=600 # ≥ 最大页数 × 4s × 余量(实测 1.5–4 s/页,共用 GPU 争用再乘 3–5 倍);不是总耗时上界
|
||||||
|
# OCR_DOC__MONKEY__1__MAX_CONCURRENCY=2 # 服务为 sync-only 后端时取 1(每实例 /parse 全局串行)
|
||||||
|
# OCR_DOC__MONKEY__1__TRUST_ENV=false
|
||||||
|
# OCR_DOC__RETRY__MAX_ATTEMPTS=2 # 超时/GET 失败的重试会重新 POST 整份文档重算,压低
|
||||||
|
# OCR_DOC__CALL_DEADLINE_S=900 # 单份硬上限;须 ≤ PGW_LEASE_TTL_S(缺省 1500)
|
||||||
|
|
||||||
# ══ SOAK_OCR scope(P7 OCR 压测;tools/soak/run_soak.py --scenario P7 --scope SOAK_OCR)══
|
# ══ SOAK_OCR scope(P7 OCR 压测;tools/soak/run_soak.py --scenario P7 --scope SOAK_OCR)══
|
||||||
# 双真实实例 + 黑洞(连接超时)+ 坏端口(连接拒绝)故障池;
|
# 双真实实例 + 黑洞(连接超时)+ 坏端口(连接拒绝)故障池;
|
||||||
|
|||||||
@@ -1,5 +1,39 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 1.3.11(2026-10-22)
|
||||||
|
|
||||||
|
#34: OCR 新增整份文档解析 `OcrClient.parse_document`——CHSAnalyzer3 要把几百篇医学论文 PDF 整份转成带页码的正文与表格,而库只会返回版面框,且 PDF 被写死的 `image.jpg` 标签当成图片上传。两个症状同根: 库把 OCR 输入建模成「一张图 + 写死的标签」、输出建模成「只有框」,没有「文档」这个概念。设计 `research-wiki/designs/2026-10-21-issue34-ocr-document-design.md`。
|
||||||
|
|
||||||
|
### 新增
|
||||||
|
|
||||||
|
- **`OcrClient.parse_document(document, *, session_id, parent_call_id, tenant_id, meta, call_deadline_s) -> OcrDocumentResult`**: PDF 或图片整份交给 MonkeyOCR,返回逐块 `OcrDocumentElement`(继承 `OcrLayoutElement`,仅多一个 `text`:`text`/`title` 纯文本、`interline_equation` 裸 LaTeX、`table` HTML、`image` 空串)、`page_sizes` 与整篇 `markdown`。治理链路(限流/熔断/重试/换源/遥测/调用方维度/期限)复用 `parse_layout` 的同一循环。
|
||||||
|
- 公共类型 `OcrDocumentElement` / `OcrDocumentResult`(包根导出);端口 `OcrDocumentPort`;transport 协议 `OcrDocumentTransport`(**独立于 `OcrTransport`**,不动旧 `runtime_checkable` Protocol 的成员,既有实现与测试替身不受影响)。
|
||||||
|
- `CallOperation` / 遥测 `operation` 列新增取值 `parse_document`(无 DDL、无新列);占位 `<ocr:document image_bytes=N>`,response 摘要 `<document elements=N pages=M md_chars=K>` **永不写入正文**。
|
||||||
|
- 内容只取自 `_middle.json` 与配对的 `.md`,**有意不读 `_content_list.json`**(它是展示格式,星号、反引号、波浪线、美元符会被转义,实测 134 个文字块中 23 个含 `\$`)。
|
||||||
|
|
||||||
|
### 修复
|
||||||
|
|
||||||
|
- **`recognize_text(PDF)` 的静默错误结果**: 此前一切上传都标 `image.jpg`,而 MonkeyOCR 按扩展名分派。PDF 走图片分支时 `/ocr/text` 返回 `success:true` 与幻觉文本(实测 `"The image contains a single character: 1"`),下游完全无感。上传标签现在按**内容**判定(图片魔数优先,再看前 1KB 内的 `%PDF-`),三个方法共用一处;非 PDF 一律沿用 `image.jpg`(字节级不变)。
|
||||||
|
|
||||||
|
### 行为变更
|
||||||
|
|
||||||
|
- **`parse_layout(PDF)` 的上传分支改变**(图片数据集 → PDF 数据集)。差分实测: 文本型与无文字层的位图型 2 页 PDF,`_middle.json` / `_content_list.json` / `.md` **逐字相同**;**旋转页、带批注页未测,不构成保证**。该路径无已知消费者(医院线只传图片)。
|
||||||
|
- **`parse_document` 对坏输入的处置**(仅新路径;共享分类不变): 无法识别的字节本地拒绝、不发请求(`RequestRejectedError`,不带 `status_code`);服务把一切失败包成 `500 "Parsing failed: …"`,其中已知的确定性坏输入(`Failed to open stream` / `Cannot open empty stream` / `cannot save with zero pages` / `is no PDF`)归 `RequestRejectedError`(`status_code=500`)——**不重试、熔断记成功不计窗口**。其余 500 仍是瞬时错误。标记集合可经 `MonkeyOcrTransport(deterministic_failure_markers=...)` 覆盖。
|
||||||
|
- 结果包解析的外部数据异常边界统一落 `ResultInvalidError`(含加密成员、未知压缩方法、巨大整数、深嵌套 JSON),不漏出原生异常——漏出会让客户端既不写终态遥测行、又不归还半开探针。
|
||||||
|
- 文档路径流式下载并限制整包 256 MiB;`_middle.json` ≤ 64 MiB、`.md` ≤ 16 MiB;解析在线程里执行,取消与 `call_deadline_s` **基本不被占住**(`json.loads` 持有 GIL:实测 54 MiB 的 middle 解析约 1 秒、事件循环最大停顿约 0.3 秒;取消后线程自然收尾、占用默认 executor,正常大小是毫秒级)。`parse_layout` 的口径(bbox 整数化退化校验、`page_idx` 回退、无体积上限)**保持原样**。
|
||||||
|
|
||||||
|
- **共用的第一段协议新增 `download_url` 校验**: 服务响应里的畸形 `download_url`(控制字符、坏方括号、越界端口)此前会让 `httpx.InvalidURL` 等原生异常漏出 `OcrClient`(不写终态遥测行、半开探针不归还),现归瞬时错误、不发第二段请求。`parse_layout` 共用这一处,属顺带修复;**不限制主机**(基线接受绝对地址)。
|
||||||
|
- **嗅探副作用**: 前 1KB 内含 `%PDF-` 字样的非 PDF 文本在 `recognize_text` / `parse_layout` 上的上传标签由 `image.jpg` 变为 `doc.pdf`;BMP 魔数现要求 DIB 头大小字段合法(不再单凭 `BM` 两字节)。
|
||||||
|
- 文档路径读取错误响应体只读前 64 KiB(摘要只需要几 KiB),不再把巨大的错误体整个读进内存。
|
||||||
|
|
||||||
|
### 已知限制与运维须知(详见 README「1.3.11 OCR 整份文档解析」)
|
||||||
|
|
||||||
|
- **服务端丢弃脚注、页眉页脚、页码**,表格脚注里的显著性/缩写说明不会出现在结果里。
|
||||||
|
- **`timeout_s` 不是总耗时上界**(逐次网络操作的超时);长文档的硬上限靠 `call_deadline_s`,并应 `≤ lease_ttl_s`。库内无基于延迟的熔断,长文档不会被判慢。
|
||||||
|
- 若 MonkeyOCR 部署为 sync-only 后端(出厂默认 `lmdeploy`),`/parse` 每实例全局串行,长文档会队头阻塞同实例上医院线的请求;请向运维确认 backend。
|
||||||
|
- 多行/多 span 文本块、`inline_equation` span、嵌套 caption/footnote 子块三条路径无真实样本,仅由构造 fixture 测试。
|
||||||
|
- 后续: #35(共享分类对确定性坏输入 500 仍归瞬时)、#36(`recognize_text` 不校验服务响应的 `success`)、#37(`parse_layout` 结果包解析仍漏出原生异常、`download_url` 不限制主机)。
|
||||||
|
|
||||||
## 1.3.10(2026-10-15)
|
## 1.3.10(2026-10-15)
|
||||||
|
|
||||||
#30/#31 同根修复: 1.3.9 以 `[DONE]` 存在、正文非空、字段白名单判定成功,上游的正常结局被当瞬时错误重试到耗尽(magpie/glm 不发 `[DONE]` 时单次调用挂 1216s;只有思考的 length 截断每次写满 max_tokens 后重抽)。本版改以上游 `finish_reason` 与传输完整性为准,流式/非流式/缓存读写共用一份判定。设计 `research-wiki/designs/2026-10-14-issue30-31-completion-verdict-design.md`。
|
#30/#31 同根修复: 1.3.9 以 `[DONE]` 存在、正文非空、字段白名单判定成功,上游的正常结局被当瞬时错误重试到耗尽(magpie/glm 不发 `[DONE]` 时单次调用挂 1216s;只有思考的 length 截断每次写满 max_tokens 后重抽)。本版改以上游 `finish_reason` 与传输完整性为准,流式/非流式/缓存读写共用一份判定。设计 `research-wiki/designs/2026-10-14-issue30-31-completion-verdict-design.md`。
|
||||||
|
|||||||
@@ -26,12 +26,12 @@
|
|||||||
| 推理可观测性 | "这次到底推理没推理"由多信号裁定(推理正文压倒 usage 明细),三态落在 `LLMResponse.thinking_observation`:`observed` / `absent` / `unknown`——**`unknown` 是"本次判不出",不是"没推理"**;本次实发档位与实测观测矛盾时按 `(源, 模型, 生效档位)` 各告警一次(能力表过期、开启未生效、注入了却观测不到;同一模型的 low 与 max 是两个独立的矛盾,不共用节流键);裁定结果随遥测落库 |
|
| 推理可观测性 | "这次到底推理没推理"由多信号裁定(推理正文压倒 usage 明细),三态落在 `LLMResponse.thinking_observation`:`observed` / `absent` / `unknown`——**`unknown` 是"本次判不出",不是"没推理"**;本次实发档位与实测观测矛盾时按 `(源, 模型, 生效档位)` 各告警一次(能力表过期、开启未生效、注入了却观测不到;同一模型的 low 与 max 是两个独立的矛盾,不共用节流键);裁定结果随遥测落库 |
|
||||||
| 推理档位 | 推理是**八档**(`none`/`auto`/`minimal`/`low`/`medium`/`high`/`xhigh`/`max`)而非开关:源级 `REASONING_EFFORT` + 请求级 `chat(reasoning_effort=...)`,`ENABLE_THINKING` 保留为语法糖;库带 24 条能力表(逐条 evidence 自报实测/文档推定),档位打空**默认报错并给出该模型最省的可用档与该配的键**,要静默映射需显式配 `EFFORT_FALLBACK=nearest`;实发档随 `LLMResponse.applied_effort` 与遥测落库 |
|
| 推理档位 | 推理是**八档**(`none`/`auto`/`minimal`/`low`/`medium`/`high`/`xhigh`/`max`)而非开关:源级 `REASONING_EFFORT` + 请求级 `chat(reasoning_effort=...)`,`ENABLE_THINKING` 保留为语法糖;库带 24 条能力表(逐条 evidence 自报实测/文档推定),档位打空**默认报错并给出该模型最省的可用档与该配的键**,要静默映射需显式配 `EFFORT_FALLBACK=nearest`;实发档随 `LLMResponse.applied_effort` 与遥测落库 |
|
||||||
| 遥测与成本 | 每次调用(含缓存命中与失败)必录 39 字段;三类行(`event_kind` = `attempt` / `cache_hit` / `terminal_failure`)加逐源诊断列(`http_status_code` / `error_type` / `cause_type` / `error_body`);SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 |
|
| 遥测与成本 | 每次调用(含缓存命中与失败)必录 39 字段;三类行(`event_kind` = `attempt` / `cache_hit` / `terminal_failure`)加逐源诊断列(`http_status_code` / `error_type` / `cause_type` / `error_body`);SQLite / Postgres 后端(表已存在时**不需要** schema 建表权限,最小权限账号可直接用);按价格表折算成本落库(注意 `LLMResponse.cost` 本身恒为 `None`,成本只进遥测);多模态内容摘要落库不存原图 |
|
||||||
| 逻辑调用统计 | 治理单位是**一次逻辑调用**而非一次尝试:四种响应(chat / embedding / OCR 两种)带 `call_stats`(`logical_call_id` / `attempts` / `total_latency_ms`,及生成时间 `generation_ms`——**1.3.8 起为净口径**:赢家那次 transport 调用墙钟 − Σ超阈值中段沉默,仍含 TTFT 与尾部,排除准入排队/退避/触发前等待;1.3.7 的墙钟值由 `generation_ms + stall_ms` 还原——与沉默归因 `stall_ms` / `stall_count`(成对报告,None = 无沉默观测)、对冲计数 `hedges` / `hedge_won`),重试、换源、结构化重问、embedding 分批共享同一逻辑 ID;每次**领域失败**另落一条 `terminal_failure` 行,失败调用数从此是一条 `WHERE event_kind = 'terminal_failure'`,详见[1.3.5 逻辑调用统计与失败诊断](#135-逻辑调用统计与失败诊断)与[1.3.8 生成时间口径与中段沉默对冲](#138-生成时间口径与中段沉默对冲) |
|
| 逻辑调用统计 | 治理单位是**一次逻辑调用**而非一次尝试:五种响应(chat / embedding / OCR 三种)带 `call_stats`(`logical_call_id` / `attempts` / `total_latency_ms`,及生成时间 `generation_ms`——**1.3.8 起为净口径**:赢家那次 transport 调用墙钟 − Σ超阈值中段沉默,仍含 TTFT 与尾部,排除准入排队/退避/触发前等待;1.3.7 的墙钟值由 `generation_ms + stall_ms` 还原——与沉默归因 `stall_ms` / `stall_count`(成对报告,None = 无沉默观测)、对冲计数 `hedges` / `hedge_won`),重试、换源、结构化重问、embedding 分批共享同一逻辑 ID;每次**领域失败**另落一条 `terminal_failure` 行,失败调用数从此是一条 `WHERE event_kind = 'terminal_failure'`,详见[1.3.5 逻辑调用统计与失败诊断](#135-逻辑调用统计与失败诊断)与[1.3.8 生成时间口径与中段沉默对冲](#138-生成时间口径与中段沉默对冲) |
|
||||||
| 遥测的资源与降级 | Postgres 池**闲时占 0 条连接**、忙时上限可配(`PGW_TELEMETRY_PG_POOL_MAX`,缺省 4),每次写入有硬预算(`PGW_TELEMETRY_PG_WRITE_TIMEOUT_S`,缺省 5s);后端不可用是**可恢复的降级**(冷却 60s 后自动重试,DBA 建完表/放开权限即自愈),永久失能只留给 DSN 本身写错;降级状态可编程查询——`client.telemetry_status` 给出 `degraded`/`fatal`/`reason`/`dropped_rows` 等只读快照,不必再靠人工对账。**对账要同时看 `degraded` 与 `dropped_rows`**: 池饱和超预算丢的行走行级丢弃,`degraded` 保持 `False`(后端没挂,是本进程并发超了),只按 `degraded` 告警会看不见这一类丢行——而它恰是 `pool_max` 配小了的唯一信号 |
|
| 遥测的资源与降级 | Postgres 池**闲时占 0 条连接**、忙时上限可配(`PGW_TELEMETRY_PG_POOL_MAX`,缺省 4),每次写入有硬预算(`PGW_TELEMETRY_PG_WRITE_TIMEOUT_S`,缺省 5s);后端不可用是**可恢复的降级**(冷却 60s 后自动重试,DBA 建完表/放开权限即自愈),永久失能只留给 DSN 本身写错;降级状态可编程查询——`client.telemetry_status` 给出 `degraded`/`fatal`/`reason`/`dropped_rows` 等只读快照,不必再靠人工对账。**对账要同时看 `degraded` 与 `dropped_rows`**: 池饱和超预算丢的行走行级丢弃,`degraded` 保持 `False`(后端没挂,是本进程并发超了),只按 `degraded` 告警会看不见这一类丢行——而它恰是 `pool_max` 配小了的唯一信号 |
|
||||||
| 调用方维度 | 每次调用可带 `tenant_id`(遥测表的真实列,可挂 RLS、可建复合索引)与 `meta`(≤16 个自定义 KV);四个公共方法全覆盖,校验超限即报错;**库只交付列,不启用 RLS、不建索引** |
|
| 调用方维度 | 每次调用可带 `tenant_id`(遥测表的真实列,可挂 RLS、可建复合索引)与 `meta`(≤16 个自定义 KV);五个公共方法全覆盖,校验超限即报错;**库只交付列,不启用 RLS、不建索引** |
|
||||||
| 遥测表治理 | `llm_calls` 是**下游的表**:PG 侧缺省**不再自动 `ALTER` 补列**(`PGW_TELEMETRY_SCHEMA_MODE` 三态,不设则 sqlite→auto、postgres→manual),manual 档点名缺列并按现有列裁剪写入;`telemetry_schema_sql(backend)` 自取可粘进迁移文件的建表/补列 SQL;`PGW_TELEMETRY_TEXT_CAP` 限正文长度(**不设 = 存全文**);保留期与访问控制走[生产部署 DDL 模板](#生产部署-ddl-模板postgresql)加 `tools/telemetry_retention.py` |
|
| 遥测表治理 | `llm_calls` 是**下游的表**:PG 侧缺省**不再自动 `ALTER` 补列**(`PGW_TELEMETRY_SCHEMA_MODE` 三态,不设则 sqlite→auto、postgres→manual),manual 档点名缺列并按现有列裁剪写入;`telemetry_schema_sql(backend)` 自取可粘进迁移文件的建表/补列 SQL;`PGW_TELEMETRY_TEXT_CAP` 限正文长度(**不设 = 存全文**);保留期与访问控制走[生产部署 DDL 模板](#生产部署-ddl-模板postgresql)加 `tools/telemetry_retention.py` |
|
||||||
| 结构化输出 | json_repair 修复 / 原生 schema 双策略 + 校验失败有界带反馈重问 |
|
| 结构化输出 | json_repair 修复 / 原生 schema 双策略 + 校验失败有界带反馈重问 |
|
||||||
| OCR | MonkeyOCR 双端点(文本转录 + 版面解析),bbox 数值防御下沉,逐源健康预检 `check_health()` |
|
| OCR | MonkeyOCR 双端点(文本转录 + 版面解析)+ 整份文档解析 `parse_document`(1.3.11:PDF 按内容标注,逐块类型/页码/bbox/文字 + 整篇 Markdown,详见[1.3.11 OCR 整份文档解析](#1311-ocr-整份文档解析)),bbox 数值防御下沉,逐源健康预检 `check_health()` |
|
||||||
| Embedding | 分批、维度校验、与 chat 同一治理栈 |
|
| Embedding | 分批、维度校验、与 chat 同一治理栈 |
|
||||||
|
|
||||||
**降级方向是铁律**:缓存/遥测后端掉线 → 降级而不冒泡(业务调用照常返回);限流/熔断后端掉线 → 报错而非放行(防击穿上游)。遥测的降级**不是静默的**——进入/恢复各一条日志、期间按行数与时间节流复述,并随时可经 `client.telemetry_status` 读到。`asyncio.CancelledError` 全链路穿透,in-flight 资源在 finally 释放;**资源所有权的纪律是「谁建的谁关」**——`aclose()` 只关自己 `from_env()`/`from_settings()` 建出来的组件,注入进来的 transport / recorder / limiter / breaker / cache 一律不碰(由注入方自己关)。
|
**降级方向是铁律**:缓存/遥测后端掉线 → 降级而不冒泡(业务调用照常返回);限流/熔断后端掉线 → 报错而非放行(防击穿上游)。遥测的降级**不是静默的**——进入/恢复各一条日志、期间按行数与时间节流复述,并随时可经 `client.telemetry_status` 读到。`asyncio.CancelledError` 全链路穿透,in-flight 资源在 finally 释放;**资源所有权的纪律是「谁建的谁关」**——`aclose()` 只关自己 `from_env()`/`from_settings()` 建出来的组件,注入进来的 transport / recorder / limiter / breaker / cache 一律不碰(由注入方自己关)。
|
||||||
@@ -147,7 +147,7 @@ stats.total_latency_ms # 含缓存 IO、退避、准入等待、重
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install --extra-index-url https://gitea.iomgaa.online/api/packages/iomgaa/pypi/simple/ \
|
pip install --extra-index-url https://gitea.iomgaa.online/api/packages/iomgaa/pypi/simple/ \
|
||||||
"polygateway[redis,postgres,structured]>=1.3.10,<2"
|
"polygateway[redis,postgres,structured]>=1.3.11,<2"
|
||||||
```
|
```
|
||||||
|
|
||||||
核心仅依赖 `httpx` + `pydantic`;按需选 extras:
|
核心仅依赖 `httpx` + `pydantic`;按需选 extras:
|
||||||
@@ -208,6 +208,7 @@ from polygateway.ocr import OcrClient
|
|||||||
ocr = OcrClient.from_env("OCR") # OCR__MONKEY__1__* 多源
|
ocr = OcrClient.from_env("OCR") # OCR__MONKEY__1__* 多源
|
||||||
text = await ocr.recognize_text(image_bytes) # 文本转录
|
text = await ocr.recognize_text(image_bytes) # 文本转录
|
||||||
layout = await ocr.parse_layout(image_bytes) # 版面解析(带 bbox 的元素列表)
|
layout = await ocr.parse_layout(image_bytes) # 版面解析(带 bbox 的元素列表)
|
||||||
|
doc = await ocr.parse_document(pdf_bytes) # 整份文档(PDF 或图片):逐块文字 + 整篇 Markdown
|
||||||
health = await ocr.check_health() # 逐源预检 {"monkey_1": True, ...}
|
health = await ocr.check_health() # 逐源预检 {"monkey_1": True, ...}
|
||||||
|
|
||||||
embed = EmbeddingClient.from_env("EMBED") # EMBED__*__* + EMBED__BATCH_SIZE
|
embed = EmbeddingClient.from_env("EMBED") # EMBED__*__* + EMBED__BATCH_SIZE
|
||||||
@@ -242,7 +243,7 @@ resp = await client.chat(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
**1.2.1 起**,四个公共方法(`chat` / `embed` / `recognize_text` / `parse_layout`)都接受这两个关键字参数,都可省略,既有调用点无需改动。校验在入口收口、**超限报 `ValueError` 而非静默丢弃**:`tenant_id` ≤128 字符、非空串、不含首尾空白(空白**拒绝而非 strip**——`" t1"` 与 `"t1"` 在 RLS 等值比较下是两个租户);`meta` 最多 16 个键,键须匹配 `[a-z0-9_.]{1,64}`(`pg_` 前缀保留给库),值仅限 `str` / `int` / `float` / `bool`,字符串值 ≤256 字符、`float` 须有限(`nan` / `inf` 不是合法 JSON,JSONB 会拒收)。两者**都不进缓存 key**——缓存隔离由 `cache_namespace` 负责。
|
**1.2.1 起**,公共方法(`chat` / `embed` / `recognize_text` / `parse_layout`,1.3.11 起含 `parse_document`)都接受这两个关键字参数,都可省略,既有调用点无需改动。校验在入口收口、**超限报 `ValueError` 而非静默丢弃**:`tenant_id` ≤128 字符、非空串、不含首尾空白(空白**拒绝而非 strip**——`" t1"` 与 `"t1"` 在 RLS 等值比较下是两个租户);`meta` 最多 16 个键,键须匹配 `[a-z0-9_.]{1,64}`(`pg_` 前缀保留给库),值仅限 `str` / `int` / `float` / `bool`,字符串值 ≤256 字符、`float` 须有限(`nan` / `inf` 不是合法 JSON,JSONB 会拒收)。两者**都不进缓存 key**——缓存隔离由 `cache_namespace` 负责。
|
||||||
|
|
||||||
存储上 `tenant_id` 两端都是 `TEXT NOT NULL DEFAULT ''`,`meta` 在 Postgres 是 `JSONB`、在 SQLite 是 `TEXT`;老表要补上这两列(补列是否由库自动执行取决于 `PGW_TELEMETRY_SCHEMA_MODE`,见[遥测表 schema 与升级纪律](#遥测表-schema-与升级纪律)),**补列后老行读出是空串而非 NULL**(NULL 在任何 RLS policy 下都对所有人不可见,空串则可用一条 SQL 审出还有多少行待归属)。
|
存储上 `tenant_id` 两端都是 `TEXT NOT NULL DEFAULT ''`,`meta` 在 Postgres 是 `JSONB`、在 SQLite 是 `TEXT`;老表要补上这两列(补列是否由库自动执行取决于 `PGW_TELEMETRY_SCHEMA_MODE`,见[遥测表 schema 与升级纪律](#遥测表-schema-与升级纪律)),**补列后老行读出是空串而非 NULL**(NULL 在任何 RLS policy 下都对所有人不可见,空串则可用一条 SQL 审出还有多少行待归属)。
|
||||||
|
|
||||||
@@ -715,6 +716,72 @@ except TimeoutError:
|
|||||||
遥测 `llm_calls` 追加可空列 `stream_end`;`TelemetryRecorder.record_llm_call` 新增必填
|
遥测 `llm_calls` 追加可空列 `stream_end`;`TelemetryRecorder.record_llm_call` 新增必填
|
||||||
keyword `stream_end`(**自建 recorder 须跟进,否则装配期被形状闸拒绝**)。
|
keyword `stream_end`(**自建 recorder 须跟进,否则装配期被形状闸拒绝**)。
|
||||||
|
|
||||||
|
## 1.3.11 OCR 整份文档解析
|
||||||
|
|
||||||
|
#34: `OcrClient.parse_document(bytes)` 把 PDF(或图片)整份交给 MonkeyOCR,返回逐块的
|
||||||
|
类型 / 页码 / bbox / 文字与整篇 Markdown。治理链路(限流、熔断、重试、遥测、
|
||||||
|
`tenant_id` / `meta`、`call_deadline_s`)与 `parse_layout` 完全一致;`parse_layout` /
|
||||||
|
`recognize_text` 的签名与返回类型不变。
|
||||||
|
|
||||||
|
```python
|
||||||
|
doc = await ocr.parse_document(pdf_bytes) # 上传类型由库按内容判定,无需也无法声明
|
||||||
|
for el in doc.elements: # 服务阅读序;页内按页序
|
||||||
|
el.page_index, el.type, el.bbox, el.text
|
||||||
|
doc.page_sizes[el.page_index] # 与 bbox 同一原生坐标系,页码零基
|
||||||
|
doc.markdown # 服务生成的整篇 Markdown
|
||||||
|
```
|
||||||
|
|
||||||
|
| 字段 | 含义 |
|
||||||
|
| --- | --- |
|
||||||
|
| `elements[].text` | `text` / `title`: 纯文本(行内公式保留 `$…$`);`interline_equation`: 裸 LaTeX;`table`: HTML 表体;`image`: 空串 |
|
||||||
|
| `elements[].type` | 开放字符串(服务给什么就是什么)。嵌套的 caption / footnote 子块若出现,作为独立元素紧随父块输出 |
|
||||||
|
| `markdown` | 服务生成的整篇原文,**含悬空的 `` 图片引用**——库不外传图片 |
|
||||||
|
|
||||||
|
### 行为变更(请先读)
|
||||||
|
|
||||||
|
| 变更 | 说明 |
|
||||||
|
| --- | --- |
|
||||||
|
| PDF 上传标签 | MonkeyOCR 按**上传文件名的扩展名**分派。此前所有上传都标 `image.jpg`;现在**前 1KB 内含 `%PDF-` 且起始不是已知图片魔数**的字节改标 `doc.pdf`(容忍垃圾前缀;副作用是前 1KB 里恰好出现 `%PDF-` 字样的非 PDF 文本也会被改标),**其余一律沿用 `image.jpg`**。三个方法共用这一个判定 |
|
||||||
|
| `recognize_text(PDF)` | **修复静默错误**: PDF 标成 `image.jpg` 时服务返回 `success:true` 与幻觉文本(实测 `"The image contains a single character: 1"`);现在返回整份全文(无页码,请改用 `parse_document`) |
|
||||||
|
| `parse_layout(PDF)` | 上传分支由图片数据集改为 PDF 数据集。已测的两种 2 页 PDF(文本型、无文字层的位图型)结果与此前**逐字相同**;**旋转页、带批注页未测**,不构成保证。医院线只传图片,不受影响 |
|
||||||
|
| 坏输入(仅 `parse_document`) | 无法识别的字节(HTML 页面、乱码、空串)本地拒绝、不发请求;服务对截断/伪造 PDF 返回的已知确定性 500(`Failed to open stream` 等)归 `RequestRejectedError`(`status_code=500`),**不重试、不计熔断**。其余 500 仍是瞬时错误 |
|
||||||
|
|
||||||
|
### 部署配方(文档线)
|
||||||
|
|
||||||
|
长文档与医院线的图片请求共用同一个 MonkeyOCR 服务时,建议文档线用**独立 scope**,自带超时 / 重试 / 熔断:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
OCR_DOC__MONKEY__1__BASE_URL=http://10.77.0.20:7866
|
||||||
|
OCR_DOC__MONKEY__1__API_KEY=none
|
||||||
|
OCR_DOC__MONKEY__1__MODEL=monkey-ocr
|
||||||
|
OCR_DOC__MONKEY__1__TIMEOUT_S=600 # ≥ 最大页数 × 4s × 余量(实测 1.5–4 s/页;共用 GPU 的争用再乘 3–5 倍)
|
||||||
|
OCR_DOC__MONKEY__1__MAX_CONCURRENCY=2 # 服务为同步串行时取 1,见下
|
||||||
|
OCR_DOC__MONKEY__1__TRUST_ENV=false
|
||||||
|
OCR_DOC__RETRY__MAX_ATTEMPTS=2
|
||||||
|
OCR_DOC__CALL_DEADLINE_S=900 # 单份文档的硬上限;须 ≤ PGW_LEASE_TTL_S(缺省 1500)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 运维须知
|
||||||
|
|
||||||
|
| # | 事项 |
|
||||||
|
| --- | --- |
|
||||||
|
| 1 | **`timeout_s` 不是总耗时上界**: 它是逐次网络操作的超时。POST 阶段服务处理期间无字节,读超时约等于处理时长上限;GET 下载只要持续有分块到达,总时长可超过任意多个 `timeout_s`。**硬上限只来自 `call_deadline_s`**。permit 租约与半开探针租约到期后会被回收 / 接管,所以文档线应让 `call_deadline_s ≤ lease_ttl_s` |
|
||||||
|
| 2 | 库内**没有任何基于延迟的熔断 / 慢判定**,长文档不会因为"慢"被判故障;唯一把慢变成失败的是 httpx 超时(瞬时错误,计熔断) |
|
||||||
|
| 3 | **重试放大**: POST 成功(服务已算完)而 GET 失败时,重试会重新 POST 整份文档重算;超时重试同理,且服务端是否因断连取消计算无证据——请把 `RETRY__MAX_ATTEMPTS` 压低 |
|
||||||
|
| 4 | **无页数预检**: 库不依赖 PDF 解析库,发送前不知页数。数百页的文档会占满超时被重投,页数过滤是调用方的职责 |
|
||||||
|
| 5 | **服务端磁盘**: 每次调用在服务 `/app/tmp` 与 `/static` 留下产物(几百篇约几百 MB 到 GB 级),库不管理,请告知服务运维 |
|
||||||
|
| 6 | **多 worker 共享并发上限**需要 `PGW_LIMITER_BACKEND=redis` 且各进程同 scope;默认 memory 后端的 `MAX_CONCURRENCY` 是进程内的 |
|
||||||
|
| 7 | **服务端并发模型**: 若 MonkeyOCR 部署为 sync-only 后端(出厂默认 `lmdeploy`),`/parse` 在每个实例上**全局串行**——"并发 2"不提速,`timeout_s` 须覆盖"排队 + 处理",且一份 15 页文档(按实测 1.5–4 s/页估算,约 22–60 秒)的队头阻塞会拖慢同实例上医院线的请求。请向运维确认 backend(服务启动日志有一行 `async-capable` / `sync-only`),sync-only 时文档线每实例并发 1、错峰运行 |
|
||||||
|
| 8 | 源 `timeout_s` 与 permit 租约的装配守卫只比较 1×(`timeout_s ≤ lease_ttl_s`);跨 scope 的守卫改造另行立项,不在本版 |
|
||||||
|
|
||||||
|
### 已知限制
|
||||||
|
|
||||||
|
- **服务端丢弃脚注、页眉页脚、页码**(版面模型把它们标为丢弃类,span 内容为空,未被 OCR):论文表格脚注里的显著性与缩写说明不会出现在 `elements` 或 `markdown` 里,库无从补偿。
|
||||||
|
- 多行 / 多 span 的文本块、`inline_equation` span、嵌套 caption / footnote 子块这三条路径**没有真实样本**(实测样本全是 1 行 1 span,默认版面模型不产生嵌套子块),只由构造 fixture 测试。
|
||||||
|
- **内存**: 文档路径整包上限 256 MiB(流式累计),合并分块会让瞬时峰值接近 2 倍上限;**字节上限不等于内存上限**——JSON 解析后的对象体积可达原文数十倍(实测 11 MiB 的空数组 JSON 解析后约 270 MiB),异常大的包请按此预留内存。真实样本约 72 KB/页,正常大小是毫秒级;`_middle.json` 的 64 MiB 上限与整包 256 MiB 上限都只界定字节数,不界定成员条数。
|
||||||
|
- `parse_document` 仅接受 PDF 与魔数表内的图片(JPEG / PNG / TIFF / BMP / GIF / WebP);表外的可解码图片格式请走 `parse_layout`。
|
||||||
|
- `parse_layout` 的校验口径保持原样(bbox 整数化退化拒绝、`page_idx` 非法回退枚举下标、无体积上限);`parse_document` 更严的有零页拒绝与 `page_idx` 必须等于页序,更宽的是 bbox 只要求有限且有序。
|
||||||
|
|
||||||
## 许可证
|
## 许可证
|
||||||
|
|
||||||
PolyGateway 使用 [Apache License 2.0](LICENSE)。内部移植代码的原创 owner 已明确授权;
|
PolyGateway 使用 [Apache License 2.0](LICENSE)。内部移植代码的原创 owner 已明确授权;
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "polygateway"
|
name = "polygateway"
|
||||||
version = "1.3.10"
|
version = "1.3.11"
|
||||||
description = "PolyGateway:实验室统一的大语言模型(LLM/VLM/OCR)调度与中转库——多源、限流、重试、熔断、缓存、遥测"
|
description = "PolyGateway:实验室统一的大语言模型(LLM/VLM/OCR)调度与中转库——多源、限流、重试、熔断、缓存、遥测"
|
||||||
# registry 包页面的正文只认这一项:缺了页面就是一片空白(1.1.2 的教训,twine 会警告
|
# registry 包页面的正文只认这一项:缺了页面就是一片空白(1.1.2 的教训,twine 会警告
|
||||||
# long_description missing 但不阻塞上传)。README 在打包时被固化进产物,发布后再改无效。
|
# long_description missing 但不阻塞上传)。README 在打包时被固化进产物,发布后再改无效。
|
||||||
|
|||||||
@@ -210,6 +210,8 @@ HTTP API → arq 队列 → worker 协程 脚本 → asyncio.gather 协
|
|||||||
|
|
||||||
**背景(调研结论)**: Video-Tree 用 `/ocr/text` 把帧文字作为"硬证据"并置进 VLM 提示词,**裸调**(无重试/限流/熔断,单帧失败跳过,手写双端点轮询)——迁移后免费升级为全治理。CHSAnalyzer 用 `/parse` 定位表格 bbox,已全治理。GovDoc 无 OCR 且是明确架构否决(旧系统被"OCR 挂死"拖垮,教训反哺 §6.4 取消语义)。多后端预留有直接证据:CHSAnalyzer provider 白名单已含 `glm`(延后),设计文档明确三种后端图片输入形态各异(VLM=base64、Monkey=multipart、GLM=URL)——因此端口收 `bytes`,编码差异封装在 invoker 内部。
|
**背景(调研结论)**: Video-Tree 用 `/ocr/text` 把帧文字作为"硬证据"并置进 VLM 提示词,**裸调**(无重试/限流/熔断,单帧失败跳过,手写双端点轮询)——迁移后免费升级为全治理。CHSAnalyzer 用 `/parse` 定位表格 bbox,已全治理。GovDoc 无 OCR 且是明确架构否决(旧系统被"OCR 挂死"拖垮,教训反哺 §6.4 取消语义)。多后端预留有直接证据:CHSAnalyzer provider 白名单已含 `glm`(延后),设计文档明确三种后端图片输入形态各异(VLM=base64、Monkey=multipart、GLM=URL)——因此端口收 `bytes`,编码差异封装在 invoker 内部。
|
||||||
|
|
||||||
|
**1.3.11 增补(#34,整份文档解析)**: 端口族增第三成员 `OcrDocumentPort.parse_document(bytes)`(PDF 或图片整份 → 逐块 `OcrDocumentElement` + `page_sizes` + 整篇 `markdown`)。根因与取舍见 `designs/2026-10-21-issue34-ocr-document-design.md`: M3 的"服务按内容处理"只在 35 张图片样本上验证过,而 MonkeyOCR **按上传文件名扩展名分派**(PDF 标 `image.jpg` 时 `/ocr/text` 返回 `success:true` 的幻觉文本);库把输入建模成"一张图 + 写死的标签"、输出建模成"只有框",没有"文档"这个概念。决策要点: ① 新 Protocol `OcrDocumentTransport` **独立于** `OcrTransport`(旧 `runtime_checkable` Protocol 不加成员);② 上传类型由库按内容判定(图片魔数优先 → 前 1KB 内 `%PDF-`),不提供调用方声明参数;③ 内容只取自 `_middle.json` + 配对 `.md`,不读 `_content_list.json`(展示格式,转义 `* ` ~ $`);④ 确定性坏输入的 500 只在新路径窄化为"请求被拒",共享分类不变(#35)。
|
||||||
|
|
||||||
**影响**: 协议细节与后处理分界见 §7.10;`TableLocator` 这类业务化端口留在业务侧,由库端口结果组装。
|
**影响**: 协议细节与后处理分界见 §7.10;`TableLocator` 这类业务化端口留在业务侧,由库端口结果组装。
|
||||||
|
|
||||||
### D10 音频只留端口
|
### D10 音频只留端口
|
||||||
@@ -424,13 +426,13 @@ flowchart TB
|
|||||||
|
|
||||||
### 5.2 其他类型与 `chat()` 公共签名
|
### 5.2 其他类型与 `chat()` 公共签名
|
||||||
|
|
||||||
`ChatRequest`(model/messages/结构化输出参数/per-call 覆盖项)、`Usage`(tokens + elapsed,OCR 无计费填 0)、`OcrTextResult`(text + 溯源五件 source_name/usage/latency_ms/call_id/raw)、`OcrLayoutResult`(elements 含 bbox/type/page_index + page_sizes + 同套溯源五件;元素自 `_middle.json` para_blocks 全量提取,M3 设计 §1.2 取证)。全部 frozen dataclass。空结果语义:合法"无内容"用空值/None 表达,调用失败必须走异常——二者严格区分。
|
`ChatRequest`(model/messages/结构化输出参数/per-call 覆盖项)、`Usage`(tokens + elapsed,OCR 无计费填 0)、`OcrTextResult`(text + 溯源五件 source_name/usage/latency_ms/call_id/raw)、`OcrLayoutResult`(elements 含 bbox/type/page_index + page_sizes + 同套溯源五件;元素自 `_middle.json` para_blocks 全量提取,M3 设计 §1.2 取证)、`OcrDocumentResult`(1.3.11,#34: `OcrDocumentElement`=版面元素+`text`,加 page_sizes、整篇 `markdown` 与同套溯源五件;**独立定义不继承** `OcrLayoutResult`——父类 `call_stats` 带默认值,子类无默认字段会在导入期 `TypeError`)。全部 frozen dataclass。空结果语义:合法"无内容"用空值/None 表达,调用失败必须走异常——二者严格区分。
|
||||||
|
|
||||||
**`chat()` 公共签名定稿(2026-07-20,GovDoc 迁移缺口 G1/G2)**: `chat(messages, *, session_id=None, parent_call_id=None, cache_salt=None, cache_namespace=None, structured=None, stream=True)`。要点: ① `session_id`/`parent_call_id` 与三项目现有 `LLMProvider.chat` Protocol 逐字兼容——这是"调用点零改动"承诺的前提;② **per-call `cache_namespace`**: GovDoc 是单 client 服务多租户、tenant 每请求变化,装配级 namespace 只是默认值,per-call 传入时覆盖并进入缓存 key(§7.5);③ `cache_salt` per-call 可传(Video-Tree 跨 epoch 重采样);④ `structured` 三档语义(D14),类型定稿 `type[BaseModel] | Literal["json"] | None`(M1 设计): 不传 = 原始文本,`"json"` = 仅修复,pydantic 模型 = 完整阶梯(修复+形态校验+有界带反馈重问)。
|
**`chat()` 公共签名定稿(2026-07-20,GovDoc 迁移缺口 G1/G2)**: `chat(messages, *, session_id=None, parent_call_id=None, cache_salt=None, cache_namespace=None, structured=None, stream=True)`。要点: ① `session_id`/`parent_call_id` 与三项目现有 `LLMProvider.chat` Protocol 逐字兼容——这是"调用点零改动"承诺的前提;② **per-call `cache_namespace`**: GovDoc 是单 client 服务多租户、tenant 每请求变化,装配级 namespace 只是默认值,per-call 传入时覆盖并进入缓存 key(§7.5);③ `cache_salt` per-call 可传(Video-Tree 跨 epoch 重采样);④ `structured` 三档语义(D14),类型定稿 `type[BaseModel] | Literal["json"] | None`(M1 设计): 不传 = 原始文本,`"json"` = 仅修复,pydantic 模型 = 完整阶梯(修复+形态校验+有界带反馈重问)。
|
||||||
|
|
||||||
**`overlay` 追加(2026-07-31,issue #4)**: 签名末尾增 `overlay: Mapping[str, Any] | None = None`,承载采样参数(`temperature`/`seed`/`max_tokens` 等)。带默认值的 keyword-only 参数不改变既有调用点,"签名冻结"承诺不破。要点: ① 优先级 **结构化注入 > 调用级 overlay > 源级 `extra_body`**,由 `StructuredMW` 的 `{**request.overlay, **strategy_overlay}` 与 transport `_build_payload` 的 update 顺序天然给出,无新机制;② 保护键 `{model, messages, stream, stream_options}` 与不可 JSON 序列化的值在**进洋葱之前**报 `ValueError`(前者被覆盖会击穿成本换算/缓存口径/流式看门狗/usage 帧,后者会在 `CacheMW` 的降级 try 之外抛裸 `TypeError` 且一行遥测都没有);③ 同时填 `ChatRequest.sampling` 快照字段——`overlay` 在洋葱不同深度取值不同(内层含 `response_format`),缓存 key 与遥测需要一个跨层恒定的读取点,否则同一列在不同行口径分叉。
|
**`overlay` 追加(2026-07-31,issue #4)**: 签名末尾增 `overlay: Mapping[str, Any] | None = None`,承载采样参数(`temperature`/`seed`/`max_tokens` 等)。带默认值的 keyword-only 参数不改变既有调用点,"签名冻结"承诺不破。要点: ① 优先级 **结构化注入 > 调用级 overlay > 源级 `extra_body`**,由 `StructuredMW` 的 `{**request.overlay, **strategy_overlay}` 与 transport `_build_payload` 的 update 顺序天然给出,无新机制;② 保护键 `{model, messages, stream, stream_options}` 与不可 JSON 序列化的值在**进洋葱之前**报 `ValueError`(前者被覆盖会击穿成本换算/缓存口径/流式看门狗/usage 帧,后者会在 `CacheMW` 的降级 try 之外抛裸 `TypeError` 且一行遥测都没有);③ 同时填 `ChatRequest.sampling` 快照字段——`overlay` 在洋葱不同深度取值不同(内层含 `response_format`),缓存 key 与遥测需要一个跨层恒定的读取点,否则同一列在不同行口径分叉。
|
||||||
|
|
||||||
**调用方维度追加(2026-08-17,issue #11)**: 四个公共方法(`chat` / `embed` / `recognize_text` / `parse_layout`)签名末尾各增 `tenant_id: str | None = None` 与 `meta: Mapping[str, Any] | None = None`。同 `overlay` 的形态——带默认值的 keyword-only,既有调用点零改动,"签名冻结"承诺不破。要点:
|
**调用方维度追加(2026-08-17,issue #11)**: 五个公共方法(`chat` / `embed` / `recognize_text` / `parse_layout` / `parse_document`)签名末尾各增 `tenant_id: str | None = None` 与 `meta: Mapping[str, Any] | None = None`。同 `overlay` 的形态——带默认值的 keyword-only,既有调用点零改动,"签名冻结"承诺不破。要点:
|
||||||
|
|
||||||
① **校验在公共入口抛 `ValueError`,不静默丢弃**(与 `overlay` 保护键同一先例:构造期错误,发生在洋葱之外,不入四分类)。规则:`tenant_id` ≤128 字符、不含首尾空白(**拒绝而非 strip**——`" t1"` 与 `"t1"` 在 RLS 的等值比较下是两个租户,替调用方改写等于把行藏进另一个租户且不报错)、不得空串(空串是"未归属"哨兵);`meta` ≤16 键,键匹配 `[a-z0-9_.]{1,64}` 且 `pg_` 前缀保留给库,值仅限 `str`/`int`/`float`/`bool`,字符串值 ≤256 字符,**非有限 float 必须挡在入口**(`json.dumps` 会把它写成裸 `NaN`/`Infinity` 字面量——不是合法 JSON,PG 的 JSONB 拒收;放行则写入失败被遥测的降级 try 吞成 warning,即调用方的输入错误转成静默丢遥测)。emitter 侧 `allow_nan=False` 是第二道闸,它保的是 **SQLite**:那边 `meta` 是 TEXT 列不做 JSON 校验,没有这道闸会把非法 JSON 静默存进去,而它抛出的异常同样被降级 try 接住 → 丢一行而非报错。
|
① **校验在公共入口抛 `ValueError`,不静默丢弃**(与 `overlay` 保护键同一先例:构造期错误,发生在洋葱之外,不入四分类)。规则:`tenant_id` ≤128 字符、不含首尾空白(**拒绝而非 strip**——`" t1"` 与 `"t1"` 在 RLS 的等值比较下是两个租户,替调用方改写等于把行藏进另一个租户且不报错)、不得空串(空串是"未归属"哨兵);`meta` ≤16 键,键匹配 `[a-z0-9_.]{1,64}` 且 `pg_` 前缀保留给库,值仅限 `str`/`int`/`float`/`bool`,字符串值 ≤256 字符,**非有限 float 必须挡在入口**(`json.dumps` 会把它写成裸 `NaN`/`Infinity` 字面量——不是合法 JSON,PG 的 JSONB 拒收;放行则写入失败被遥测的降级 try 吞成 warning,即调用方的输入错误转成静默丢遥测)。emitter 侧 `allow_nan=False` 是第二道闸,它保的是 **SQLite**:那边 `meta` 是 TEXT 列不做 JSON 校验,没有这道闸会把非法 JSON 静默存进去,而它抛出的异常同样被降级 try 接住 → 丢一行而非报错。
|
||||||
|
|
||||||
@@ -594,7 +596,7 @@ flowchart TB
|
|||||||
| 列 | 语义 |
|
| 列 | 语义 |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `scope` | 池名,`TelemetryEmitter` 构造期注入(三个 client 各一行),三类行都带;**不拿 `source_name` 顶替**(终态失败可能根本没选出源) |
|
| `scope` | 池名,`TelemetryEmitter` 构造期注入(三个 client 各一行),三类行都带;**不拿 `source_name` 顶替**(终态失败可能根本没选出源) |
|
||||||
| `operation` | `chat` / `embed` / `recognize_text` / `parse_layout`,**由调用点给定**。它与 `PolyGatewayError.operation` 是两个语义: 后者是异常自报的出错环节(可为 `download_result` 这类子步骤),链路上任何位置不得读它来填本列 |
|
| `operation` | `chat` / `embed` / `recognize_text` / `parse_layout` / `parse_document`,**由调用点给定**。它与 `PolyGatewayError.operation` 是两个语义: 后者是异常自报的出错环节(可为 `download_result` 这类子步骤),链路上任何位置不得读它来填本列 |
|
||||||
| `logical_call_id` | 一次逻辑调用一个 ID;上下文缺席(库内现场构造的请求)落 NULL,**不造 ID** |
|
| `logical_call_id` | 一次逻辑调用一个 ID;上下文缺席(库内现场构造的请求)落 NULL,**不造 ID** |
|
||||||
| `event_kind` | 三态: `attempt` / `cache_hit` / `terminal_failure`,三类行的唯一机械判据 |
|
| `event_kind` | 三态: `attempt` / `cache_hit` / `terminal_failure`,三类行的唯一机械判据 |
|
||||||
| `http_status_code` / `cause_type` / `error_body` | 只在失败的 `attempt` 行上非空(成功行不统一填 200: 那会让"有状态码"不再等价于"失败了");终态行三列恒 NULL(**C1 红线**) |
|
| `http_status_code` / `cause_type` / `error_body` | 只在失败的 `attempt` 行上非空(成功行不统一填 200: 那会让"有状态码"不再等价于"失败了");终态行三列恒 NULL(**C1 红线**) |
|
||||||
@@ -668,6 +670,7 @@ flowchart TB
|
|||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `OcrTextPort.recognize_text(image: bytes)` | `POST /ocr/text` | multipart 上传 → JSON `{content}` | `OcrTextResult`(多行纯文本) |
|
| `OcrTextPort.recognize_text(image: bytes)` | `POST /ocr/text` | multipart 上传 → JSON `{content}` | `OcrTextResult`(多行纯文本) |
|
||||||
| `OcrLayoutPort.parse_layout(image: bytes)` | `POST /parse` | multipart → JSON(download_url) → GET ZIP → 解包 `*_middle.json` | `OcrLayoutResult`(elements 含 bbox + page_size) |
|
| `OcrLayoutPort.parse_layout(image: bytes)` | `POST /parse` | multipart → JSON(download_url) → GET ZIP → 解包 `*_middle.json` | `OcrLayoutResult`(elements 含 bbox + page_size) |
|
||||||
|
| `OcrDocumentPort.parse_document(document: bytes)`(1.3.11,#34) | `POST /parse`(同上两段协议) | 上传类型按内容判定;流式 GET(整包 ≤256 MiB)→ 线程内解析 `*_middle.json` + 配对 `.md` | `OcrDocumentResult`(逐块 text + bbox + 页码,整篇 markdown) |
|
||||||
|
|
||||||
设计要点:输入统一 `bytes`(路径读取/多帧批量拼接留业务侧);bbox 返回 OCR 原生页面坐标,几何映射(裁剪偏移/归一化/marker 推算)留业务侧;`None`/空表达"合法无内容",异常表达"调用失败";ZIP 内容不可解析抛 `ResultInvalidError`(坏图≠坏服务);OCR 复用同一套治理算法件、循环形态同 EmbeddingClient 先例(见 D9 修订;无 token 计费,Usage 填 0,latency_ms 照记);多后端经 provider 注册表扩展(GLM 已在 CHSAnalyzer 白名单,输入形态为 URL,届时封装在其 invoker 内部,端口签名不变)。数值防御(bbox 有限性/顺序/退化校验)随协议解析下沉进库。OCR transport 支持 per-source `trust_env` 开关(LAN 直连绕过本地代理,Video-Tree `ocr.py:46` 教训;2026-07-20 缺口 R9);客户端暴露 `check_health() -> dict[str, bool]` 逐源健康预检(M3 设计 §3.3 细化:bool 无法承载逐源结果,业务 `all()` 即得启动门布尔;Video-Tree A/B 评测消费;缺口 R10 已闭)。协议事实(2026-07-21 实测): `/parse` 的 download_url 为**相对路径**(`/static/*.zip`),transport 相对 base_url 解析并兼容绝对 URL。
|
设计要点:输入统一 `bytes`(路径读取/多帧批量拼接留业务侧);bbox 返回 OCR 原生页面坐标,几何映射(裁剪偏移/归一化/marker 推算)留业务侧;`None`/空表达"合法无内容",异常表达"调用失败";ZIP 内容不可解析抛 `ResultInvalidError`(坏图≠坏服务);OCR 复用同一套治理算法件、循环形态同 EmbeddingClient 先例(见 D9 修订;无 token 计费,Usage 填 0,latency_ms 照记);多后端经 provider 注册表扩展(GLM 已在 CHSAnalyzer 白名单,输入形态为 URL,届时封装在其 invoker 内部,端口签名不变)。数值防御(bbox 有限性/顺序/退化校验)随协议解析下沉进库。OCR transport 支持 per-source `trust_env` 开关(LAN 直连绕过本地代理,Video-Tree `ocr.py:46` 教训;2026-07-20 缺口 R9);客户端暴露 `check_health() -> dict[str, bool]` 逐源健康预检(M3 设计 §3.3 细化:bool 无法承载逐源结果,业务 `all()` 即得启动门布尔;Video-Tree A/B 评测消费;缺口 R10 已闭)。协议事实(2026-07-21 实测): `/parse` 的 download_url 为**相对路径**(`/static/*.zip`),transport 相对 base_url 解析并兼容绝对 URL。
|
||||||
|
|
||||||
|
|||||||
@@ -72,6 +72,8 @@
|
|||||||
|
|
||||||
**已交付增补(2026-09-14)**: 流式中段沉默计量与对冲(issue #27/#28,1.3.8)——`generation_ms` 口径重定义为**净生成时间**(墙钟 − Σ超阈值中段沉默,仍含 TTFT 与尾部;1.3.7 墙钟口径由 `generation_ms + stall_ms` 还原,披露见 CHANGELOG 未发布段),`CallStats` 增 `stall_ms`/`stall_count` 归因字段,遥测表加同两列(INSERT 字段 36→38);`{SCOPE}__HEDGE__STALL_AFTER_S` 中段沉默对冲(独立于 `HEDGE__AFTER_S` 生效、默认关闭;无业界公开先例,超业界现状能力),`{SCOPE}__STREAM__STALL_THRESHOLD_MS` 计量阈值(默认 1000,三层可调)。设计 designs/2026-09-11-streaming-stall-design.md(人类已批准 + 独立审查补遗 B1-B4/N1-N9),实施 plans/2026-09-11-streaming-stall.md,验证 findings/2026-09-11-streaming-stall-validation.md。后续储备:自适应分位阈值(防反馈环,现不做)、流式透传下的 commit 点重审、embedding/OCR 对冲(现不做)。
|
**已交付增补(2026-09-14)**: 流式中段沉默计量与对冲(issue #27/#28,1.3.8)——`generation_ms` 口径重定义为**净生成时间**(墙钟 − Σ超阈值中段沉默,仍含 TTFT 与尾部;1.3.7 墙钟口径由 `generation_ms + stall_ms` 还原,披露见 CHANGELOG 未发布段),`CallStats` 增 `stall_ms`/`stall_count` 归因字段,遥测表加同两列(INSERT 字段 36→38);`{SCOPE}__HEDGE__STALL_AFTER_S` 中段沉默对冲(独立于 `HEDGE__AFTER_S` 生效、默认关闭;无业界公开先例,超业界现状能力),`{SCOPE}__STREAM__STALL_THRESHOLD_MS` 计量阈值(默认 1000,三层可调)。设计 designs/2026-09-11-streaming-stall-design.md(人类已批准 + 独立审查补遗 B1-B4/N1-N9),实施 plans/2026-09-11-streaming-stall.md,验证 findings/2026-09-11-streaming-stall-validation.md。后续储备:自适应分位阈值(防反馈环,现不做)、流式透传下的 commit 点重审、embedding/OCR 对冲(现不做)。
|
||||||
|
|
||||||
|
**已交付增补(2026-10-22,1.3.11)**: OCR 整份文档解析(issue #34,1.3.11)——`OcrClient.parse_document` + `OcrDocumentPort`/`OcrDocumentTransport`,PDF 上传按内容标注(同时修复 `recognize_text(PDF)` 的静默幻觉文本),结果只取自 `_middle.json` + 配对 `.md`;新路径对确定性坏输入归"请求被拒"。后续: #35(共享分类)、#36(`recognize_text` 不校验 `success`)、#37(`parse_layout` 原生异常漏出与 `download_url` 主机)。设计/计划见 `designs/2026-10-21-issue34-ocr-document-design.md`、`plans/2026-10-21-issue34-ocr-document.md`。
|
||||||
|
|
||||||
**已交付增补(2026-10-15,1.3.10 已发布)**: 统一完成判定(issue #30/#31,1.3.10)——以上游 `finish_reason` 与传输完整性判定流完成与结局,流式/非流式/缓存读写共用内核纯函数;缺 `[DONE]` 但 finish+实测 usage 完整即成功(两模式),文本 length/过滤无正文改 `ResultInvalidError`,保真模式 null/未知字段无损合并;`LLMResponse` 增 `finish_reason`/`stream_end`,遥测加 `stream_end` 列。设计 designs/2026-10-14-issue30-31-completion-verdict-design.md(人类授权 Claude 与独立 subagent 两轮审查后直接执行),实施 plans/2026-10-14-issue30-31-completion-verdict.md。后续储备:文本 n>1 混流修复、文本"只有思考+stop"证据收集。
|
**已交付增补(2026-10-15,1.3.10 已发布)**: 统一完成判定(issue #30/#31,1.3.10)——以上游 `finish_reason` 与传输完整性判定流完成与结局,流式/非流式/缓存读写共用内核纯函数;缺 `[DONE]` 但 finish+实测 usage 完整即成功(两模式),文本 length/过滤无正文改 `ResultInvalidError`,保真模式 null/未知字段无损合并;`LLMResponse` 增 `finish_reason`/`stream_end`,遥测加 `stream_end` 列。设计 designs/2026-10-14-issue30-31-completion-verdict-design.md(人类授权 Claude 与独立 subagent 两轮审查后直接执行),实施 plans/2026-10-14-issue30-31-completion-verdict.md。后续储备:文本 n>1 混流修复、文本"只有思考+stop"证据收集。
|
||||||
|
|
||||||
## 7. 开放决策依赖
|
## 7. 开放决策依赖
|
||||||
|
|||||||
@@ -85,7 +85,7 @@ class OcrTransport(Protocol): # 协议细节;不含治理
|
|||||||
| 关切 | 决策 | 蓝本出处 |
|
| 关切 | 决策 | 蓝本出处 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| HTTP 客户端 | per-source `httpx.AsyncClient`(惰性建,keyed by source.name),`trust_env=source.trust_env`、`timeout=source.timeout_s` | **R9 销账**: SourceConfig.trust_env + `TRUST_ENV` 配置键 M1 已落地,openai_compat.py:203 已尊重,本 transport 同款;VT LAN 直连绕代理场景配 `OCR__MONKEY__N__TRUST_ENV=false` |
|
| HTTP 客户端 | per-source `httpx.AsyncClient`(惰性建,keyed by source.name),`trust_env=source.trust_env`、`timeout=source.timeout_s` | **R9 销账**: SourceConfig.trust_env + `TRUST_ENV` 配置键 M1 已落地,openai_compat.py:203 已尊重,本 transport 同款;VT LAN 直连绕代理场景配 `OCR__MONKEY__N__TRUST_ENV=false` |
|
||||||
| multipart 上传 | `files={"file": ("image.jpg", image, "image/jpeg")}` 固定名(CHS invokers.py:496 原样;服务按内容处理,35 样本实测通过,核验输出存档 `data/soak/m3_xcheck_results.txt`) | CHS |
|
| multipart 上传 | `files={"file": ("image.jpg", image, "image/jpeg")}` 固定名(CHS invokers.py:496 原样;服务按内容处理,35 样本实测通过,核验输出存档 `data/soak/m3_xcheck_results.txt`)。**勘误(2026-10-22,#34)**: "服务按内容处理"只在图片样本上验证过,**不适用于 PDF**——服务按文件名扩展名分派(`.pdf`/`.jpg`/`.jpeg`/`.png`),PDF 标 `image.jpg` 时 `/ocr/text` 返回 `success:true` 的幻觉文本;上传标签现按内容判定,见 `2026-10-21-issue34-ocr-document-design.md` §1 E3/E4/E10 | CHS |
|
||||||
| `/ocr/text` 解析 | 2xx + JSON + `content` 必须为 str,缺失/非 str → `ResultInvalidError`;空串合法 | VT(`.get("content","")` 的静默兜底**有意替换**为显式校验——P5 严禁默认值掩盖错误) |
|
| `/ocr/text` 解析 | 2xx + JSON + `content` 必须为 str,缺失/非 str → `ResultInvalidError`;空串合法 | VT(`.get("content","")` 的静默兜底**有意替换**为显式校验——P5 严禁默认值掩盖错误) |
|
||||||
| `/parse` 两段协议 | POST `/parse` → 校验 `success is True` + `download_url` 非空 str → GET(相对路径 urljoin base_url,绝对 URL 原样)→ ZIP | CHS invokers.py:489-530 逐段保真;相对路径显式化(取证 §1.2) |
|
| `/parse` 两段协议 | POST `/parse` → 校验 `success is True` + `download_url` 非空 str → GET(相对路径 urljoin base_url,绝对 URL 原样)→ ZIP | CHS invokers.py:489-530 逐段保真;相对路径显式化(取证 §1.2) |
|
||||||
| `_middle.json` 校验 | ZIP 内首个 `*_middle.json`;`pdf_info` 必须 list;每页: dict、`page_size` 为 2 正有限数、`para_blocks` 缺省按空列表、每块 type 为 str、bbox 为 4 有限数且 x2>x1、y2>y1 且**整数化后不退化**(库返回 float 原生 bbox,此校验专为 CHS shim 的 `int()` 裁剪路径兜底——亚像素宽的退化框裁剪即空图,实现中须注释此由来防止被当死代码删除) | CHS `_finite_number`/`_parse_table_result`(invokers.py:427-479)数值防御**全量下沉**,提取面从"首表"泛化为"全元素"(取证 §1.2 证明无损) |
|
| `_middle.json` 校验 | ZIP 内首个 `*_middle.json`;`pdf_info` 必须 list;每页: dict、`page_size` 为 2 正有限数、`para_blocks` 缺省按空列表、每块 type 为 str、bbox 为 4 有限数且 x2>x1、y2>y1 且**整数化后不退化**(库返回 float 原生 bbox,此校验专为 CHS shim 的 `int()` 裁剪路径兜底——亚像素宽的退化框裁剪即空图,实现中须注释此由来防止被当死代码删除) | CHS `_finite_number`/`_parse_table_result`(invokers.py:427-479)数值防御**全量下沉**,提取面从"首表"泛化为"全元素"(取证 §1.2 证明无损) |
|
||||||
|
|||||||
@@ -0,0 +1,288 @@
|
|||||||
|
---
|
||||||
|
type: design
|
||||||
|
node_id: design:2026-10-21-issue34-ocr-document-design
|
||||||
|
title: OCR 整份文档解析设计(#34)
|
||||||
|
date: 2026-10-21
|
||||||
|
---
|
||||||
|
|
||||||
|
# #34 OCR 整份文档解析设计
|
||||||
|
|
||||||
|
> 状态: **已获人类批准(2026-10-21,H1/H5/H6 按推荐批准;H2 授权另开两条 issue;H3 待下游确认,不阻塞;H4 发布逐步等指令)**。已完成 Claude 自审与一轮独立 subagent 论证(§12);Codex 一轮审查已完成并逐条处置(§12 审查记录)。
|
||||||
|
> 基线: `25a90eb`(1.3.10);目标版本 **1.3.11**;分支 `fix/issue34-ocr-document`。
|
||||||
|
> 需求: [#34](https://gitea.iomgaa.online/iomgaa/PolyGateway/issues/34)(CHSAnalyzer3 把几百篇医学论文 PDF 整份转成带页码正文与表格)。
|
||||||
|
> 修订对象: [M3 OCR 设计](2026-07-21-m3-ocr-design.md) §4 "multipart 固定名 image.jpg,服务按内容处理"(该论断只在图片样本上验证过,见 §1 E3/E4)、ARCHITECTURE §7.10。
|
||||||
|
|
||||||
|
## 1. 问题与根因
|
||||||
|
|
||||||
|
issue 把症状拆成两条;**两条同根**: 库把 OCR 输入建模成"一张图 + 写死的标签",把输出建模成"只有框",**没有"文档"这个概念**——输入类型既不声明也不检测,输出不含内容。M3 设计 §4 的"服务按内容处理"是从 35 张图片样本外推到 PDF 的未验证假设。
|
||||||
|
|
||||||
|
以下事实 2026-10-21 在真实 MonkeyOCR 服务实测取得(样本为第三方 AI 生成论文,仅本地存证,**不入库**):
|
||||||
|
|
||||||
|
| # | 事实 | 设计影响 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| E1 | 服务按**扩展名**分派,仅 `.jpg/.jpeg/.png/.pdf`(大小写不敏感);其余(`.webp`/无扩展名)→ HTTP 500 `{"detail":"Parsing failed: 400: Unsupported file type…"}` | 上传标签必须落在这四个之内;服务把客户端错误包成 500 |
|
||||||
|
| E2 | 内容与扩展名不符(jpg 字节标 `doc.pdf`)→ HTTP 500 `Parsing failed: is no PDF` | **调用方声明类型会新增一个失败入口**,故类型由库按内容判定 |
|
||||||
|
| E3 | `/parse`: 同一 PDF 以 `image.jpg` 与 `doc.pdf` 上传,`_middle.json`/`_content_list.json`/`.md` **逐字相同**(只有 message 与包内文件名前缀不同)——文本型 2 页 PDF 与**无文字层的 2 页位图 PDF**(扫描件模拟,`pdftotext` 字符数 0)两种均验证; **旋转页、带批注页未测**;PNG/TIFF/BMP 字节标 `image.jpg` 亦成功(服务按内容解码);前缀 200 字节垃圾的 PDF 以 `doc.pdf` 上传仍被解析 | 问题一在 `/parse` 上**目前未观察到结果差异**,但不是保证(E10);非 PDF 沿用 `image.jpg` 才是零回归;前缀垃圾的 PDF 服务可解析,嗅探须容忍 |
|
||||||
|
| E4 | `/ocr/text`: PDF 字节标 `image.jpg` → HTTP 200 `success:true`,content=`"The image contains a single character: 1"`(**静默错误结果**);标 `doc.pdf` → 整份全文(无页码) | 问题一的真实危害在 `/ocr/text`,issue 未提及;须一并修 |
|
||||||
|
| E5 | 截断 PDF / 伪 `%PDF-` 头 / 乱码图 / 空文件 → 全部 **HTTP 500** `{"detail":"Parsing failed: <原因>"}` | 当前 ≥500 一律 `TransientError`,见 §6 |
|
||||||
|
| E6 | 结果包含 `*.md`、`*_middle.json`、`*_content_list.json`、三份 `*.pdf`;**含图/表的文档另有 `images/*.jpg`**(4 页含 4 图 ≈598KB;无图的 2 页文档没有该目录) | 包体随插图增长;只读取需要的成员,不解出图片 |
|
||||||
|
| E7 | `_content_list.json` 与 `_middle.json` 逐块 1:1(三样本 29/29、89/89、30/30),但 content_list **无 bbox**、type 词汇不同(`title→text+text_level`、`interline_equation→equation`),且**文本里 `$` 被转义成 `\$`**(134 个 text/title 块中 23 个) | 不用 content_list,见 §3 Q3 |
|
||||||
|
| E8 | middle 的 text/title 块恰 1 行 1 span,`content` 与 content_list 反转义后 134/134 一致;table 的 span 带 `html`;interline_equation 的 span 内容为裸 LaTeX;image 嵌套 `image_body` 仅含 `image_path`;嵌套 `*_caption/*_footnote` 子块**从未出现** | 内容可只从 middle 取得;caption/footnote 路径无真实样本 |
|
||||||
|
| E9 | 速度约 **1.5–4 s/页**(2026-10-21 实测:2 页 3.5s/4.0s[位图型]/5.1s/8s[issue 原文],4 页 6.2–6.4s;样本极少、无争用,不是 SLA。验证者指出初稿的「约 1.5 s/页」偏乐观,已订正)。库内**无任何基于延迟的熔断/慢判定**(`grep latency` 在 breaker/admission/sources 无命中) | 长文档不会被"判慢",真正的风险是 §7 的超时被算成失败 |
|
||||||
|
| E10 | 服务源码(本机 `~/Code/MonkeyOCR`,`a4d853f`,版本 1.1.0 与样本 `_version_name` 一致)核对: ① 分派只看上传文件名后缀(白名单 `api/main.py:435-441`),`pdf` 走 `PymuDocDataset`、其余走 `ImageDataset`(`api/main.py:164-170`;后者先 `convert_to_pdf()`,`magic_pdf/data/dataset.py:231-235`)——**PDF 标成 `image.jpg` 走的是图片数据集分支**,E3 的"逐字相同"只在那份 2 页文本型 PDF 上验证,扫描件/旋转页/带批注页无证据; ② `/parse` 把一切异常(含 400)改写成 `500 "Parsing failed: …"`(`:501-503`),连 `Model not initialized` 也变成同前缀 | 支持嗅探(让 PDF 走它本该走的分支);**前缀 `Parsing failed:` 不能单独作为"坏输入"判据**,只有具体标记白名单可行 |
|
||||||
|
| E11 | `content_list` 是**展示格式**: 文本经 `markdown_utils` 转义 `* \` ~ $` 四个字符(`ocr_mkcontent.py:148`);医学论文里 `*`(显著性)与 `~` 比 `$` 更常见,污染面大于 E7 的 23/134 | 强化 Q3: 绝不读 content_list 取文字 |
|
||||||
|
| E12 | **服务端丢弃**脚注/页眉/页脚/页码/印章(默认版面模型 PP-DocLayout_plus-L 把这些标签映射到 `Abandon` → `discarded_blocks`,span `content` 全空,即**未被 OCR**);`figure_title` 映射为普通 `text` 块,故默认部署**不产生**嵌套 `*_caption/*_footnote` 子块(换 `doclayout_yolo` 才可能);`level` 恒为 1;`need_drop` 在该流水线硬编码为 False | 表格脚注(显著性/缩写说明)在服务端就已丢失,库找不回——写成"服务限制"告知下游;`level` 不加字段;嵌套子块仅作最小防御 |
|
||||||
|
| E13 | `/ocr/text` 对**无法解码的输入**返回 `success:true` + 幻觉文本: 乱码字节(`GARBAGE-NOT-AN-IMAGE`)得到与 E4 完全相同的 `"The image contains a single character: 1"`——即 E4 的本质是"服务把 PDF 当作无法解码的图"。另据源码,该端点内部异常会返回 **HTTP 200 + `success:false` + `content:""`**(`api/main.py:622-629`),而 `recognize_text` 只校验 `content` 为 str、不看 `success`(`monkey_ocr.py:240-247`)。后者**未经线上触发,仅源码推断** | 标签修正只能消除"PDF 被当图"这一类; 对无法解码输入的静默错误是服务行为,库侧要治需另行设计(§12 H2) |
|
||||||
|
| E14 | 服务并发模型: `/parse` 推理在 sync-only 后端下持有**全局 `asyncio.Lock`**(`api/main.py:183-189`);`supports_async` 取决于 chat 后端是否有 `async_batch_inference`,出厂默认 `lmdeploy` 为 sync-only(`model_configs.yaml`),**线上实际 backend 未知**(启动日志有 `async-capable`/`sync-only` 一行) | 若 sync-only: 每实例 `/parse` 全局串行,我们的长文档会**队头阻塞医院线**; `timeout_s` 须覆盖"排队+处理" |
|
||||||
|
|
||||||
|
## 2. 现状代码面
|
||||||
|
|
||||||
|
| 位置 | 现状 |
|
||||||
|
| --- | --- |
|
||||||
|
| `transports/monkey_ocr.py:207-210` | `_files()` 固定 `("image.jpg", …, "image/jpeg")`,`/ocr/text` 与 `/parse` 共用 |
|
||||||
|
| `monkey_ocr.py:136-144,147-166` | `_parse_block` 只取 type/bbox;`_parse_pages` 对非法 `page_idx` 回退枚举下标;bbox 含 CHS shim 专用的整数化退化校验 |
|
||||||
|
| `monkey_ocr.py:79-100` | `_classify_status`: ≥500/429→Transient、401/403→SourceDead、其余→RequestRejected |
|
||||||
|
| `types.py:1187-1250` | `OcrLayoutElement(type,bbox,page_index)`、`OcrLayoutResult`、`OcrLayoutTransportResult` |
|
||||||
|
| `ocr.py:625-628` | `_summarize` 用 `hasattr(result,"text")` 区分文本/版面——**对新结果类型是陷阱** |
|
||||||
|
| `ports.py:105-121` | `OcrTransport` 是 `@runtime_checkable`,加成员会让所有现有实现(测试替身、下游 fake)在 `isinstance` 上不合格 |
|
||||||
|
|
||||||
|
## 3. 备选方案对比
|
||||||
|
|
||||||
|
### Q1 公共面形态
|
||||||
|
|
||||||
|
| 方案 | 做法 | 权衡 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **A(推荐)** | 新增 `OcrClient.parse_document` + `OcrDocumentPort`;新类型 `OcrDocumentElement(OcrLayoutElement)`(仅增 `text`)与 `OcrDocumentResult`;`parse_layout`/`recognize_text` 签名与返回类型不动 | 医院线零风险;文档线的内容解析故障不波及版面路径 |
|
||||||
|
| B | 给 `OcrLayoutElement` 追加 `text` 字段,`parse_layout` 同时返回内容 | 违反 issue "parse_layout 不动";`text: str\|None` 在一条路径上恒为 None;医院线耦合到新增的包成员解析,任一故障波及它 |
|
||||||
|
| C | 只加 `filename/media_type` 参数并把 ZIP 原样交给调用方 | 把校验、数值防御、阅读序整理全推给下游,违背库的价值;每个下游各写一份解析 |
|
||||||
|
|
||||||
|
### Q2 上传类型如何确定
|
||||||
|
|
||||||
|
| 方案 | 做法 | 权衡 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **A(推荐)** | transport 按内容嗅探,判定顺序固定: ① 起始为已知图片魔数(JPEG/PNG/TIFF/BMP/GIF/WebP)→ 图片;② 否则前 1024 字节内含 `%PDF-` → PDF(Adobe 阅读器的头部容忍惯例,E3 实测服务亦可解析);③ 否则未知。PDF → `doc.pdf`/`application/pdf`,**图片与未知一律沿用 `image.jpg`/`image/jpeg`**。单点 `_upload_part()`,三个方法共用 | 图片魔数先判,含 XMP 元数据的图片不会因内嵌 `%PDF-` 字样被误判;非 PDF 路径字节级不变;顺带修 E4 |
|
||||||
|
| B | 调用方传 `media_type`/`filename`(**issue #34 原文提议**) | 声明与内容不符 → E2 的 500;若放在旧两方法上,仍走共享分类 → Transient → 重试并计熔断: 笔误变熔断事件。库本就能从内容判定,声明是冗余信息; 唯一得益场景(前缀垃圾 PDF)已被 A 的 ② 覆盖。**这是对 issue 提议的有意替换,列入 H5 由人类批准** |
|
||||||
|
| C | 嗅探全部图片格式并精确标注(png→`image.png`) | 零收益;webp/gif 等无对应扩展名反而要拒绝,把现在能成功的路径(E3)变成失败 |
|
||||||
|
|
||||||
|
三个方法对同一输入的结局(Codex 审查要求显式化):
|
||||||
|
|
||||||
|
| 输入 | `recognize_text` | `parse_layout` | `parse_document` |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 正常 PDF | `doc.pdf`,整份全文(E4 修复) | `doc.pdf`(上传分支变更,见 §8) | `doc.pdf`,完整文档结果 |
|
||||||
|
| 前缀垃圾的 PDF | 同上 | 同上 | 同上 |
|
||||||
|
| 魔数表内的图片(JPEG/PNG/TIFF/BMP/GIF/WebP) | `image.jpg`(不变) | `image.jpg`(不变) | `image.jpg`,1 页结果 |
|
||||||
|
| 魔数表之外的可解码图片 | `image.jpg`(不变,服务按内容解码) | `image.jpg`(不变) | **本地 `RequestRejectedError`**(请改走 `parse_layout`;表内格式的扩充另议) |
|
||||||
|
| 无法识别的字节(HTML 页面、乱码、空) | `image.jpg`(不变,服务给幻觉文本 E13) | `image.jpg`(不变,服务 500) | **本地 `RequestRejectedError`,不发 HTTP** |
|
||||||
|
|
||||||
|
### Q3 文字内容从哪里来
|
||||||
|
|
||||||
|
| 方案 | 做法 | 权衡 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **A(推荐)** | **只读 `_middle.json`**(已是 bbox/type/页码/阅读序的来源),从 `lines[].spans[]` 取 `content`/`html`;`.md` 原文单独取 | 单一来源,无对位、无类型映射、无版本漂移故障面;不受 `\$` 转义污染 |
|
||||||
|
| B | `_content_list.json` 取文字 + middle 取 bbox,按页内顺序对位并严格校验(本设计初稿方案) | 1:1 是上游构造性成立但**无契约**;漂移时整份 `ResultInvalid`,几百篇批处理会"一批全挂";文本含 `\$` 需反转义,且 title 的 level 等字段样本中无信息量(21/21 均为 1) |
|
||||||
|
| C | 只用 content_list、不要 bbox | 不满足 issue 的"每块含 bbox" |
|
||||||
|
|
||||||
|
### Q4 确定性坏输入(E5)的分类
|
||||||
|
|
||||||
|
| 方案 | 做法 | 权衡 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **A(推荐)** | **只在 `parse_document` 路径**: ① 本地魔数预检(既非 PDF 也非已知图片魔数 → 本地 `RequestRejectedError`,零 HTTP); ② 500 + `Parsing failed:` 且含已知确定性标记 → `RequestRejectedError`(保留 `status_code=500`)。共享 `_classify_status` 默认行为字节级不变 | 既有 `parse_layout`/`recognize_text` 的行为零变更;未知 500 仍 Transient(安全方向) |
|
||||||
|
| B | 直接改共享 `_classify_status`,所有 OCR 方法受益 | 改变医院线对坏图的重试/熔断行为,属公共行为变更,应单独走设计 → **另开 issue**(待人类决定) |
|
||||||
|
| C | 不处理 | 批处理后果见 §6 |
|
||||||
|
|
||||||
|
### Q5 长文档超时
|
||||||
|
|
||||||
|
| 方案 | 做法 | 权衡 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **A(推荐)** | 不加逐调用 HTTP 超时;文档线用**独立 scope**(如 `OCR_DOC`)指向同一服务 URL,自带 `TIMEOUT_S`/重试/熔断阈值;文档 + 测试钉死"无延迟判慢" | 与现有租约不变式一致 |
|
||||||
|
| B | `parse_document(timeout_s=…)` 逐调用超时 | `RateLimiter.try_acquire(source_key, est_tokens)`(`ports.py:137`)无 TTL 参数,租约长度是构造期常量,逐调用放大超时会让请求活过租约 → 并发配额被突破 |
|
||||||
|
|
||||||
|
## 4. 公共 API(冻结面,人类门审这里)
|
||||||
|
|
||||||
|
### 4.1 类型(`types.py`,全部 frozen dataclass)
|
||||||
|
|
||||||
|
| 类型 | 字段 | 说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `OcrDocumentElement(OcrLayoutElement)` | 继承 `type/bbox/page_index`;新增 `text: str` | `text` 随 type: `text`/`title` → 纯文本(行内公式保留 `$…$`);`interline_equation` → 裸 LaTeX;`table` → HTML 表体;`image` → `""`;嵌套的非 body 子块(`*_caption`/`*_footnote`)作为**独立元素**输出,type 取子块类型 |
|
||||||
|
| `OcrDocumentResult` | `elements: list[OcrDocumentElement]`、`page_sizes: list[tuple[float,float]]`、`markdown: str` + 溯源五件 `source_name/usage/latency_ms/call_id/raw` + `call_stats=None` | **独立定义,不继承** `OcrLayoutResult`: 父类末尾 `call_stats` 带默认值,子类无默认字段 `markdown` 会在导入期 `TypeError` |
|
||||||
|
| `OcrDocumentTransportResult` | `elements`、`page_sizes`、`markdown`、`raw` | transport → client 内部产物 |
|
||||||
|
|
||||||
|
约束与理由:
|
||||||
|
|
||||||
|
| 点 | 决定 |
|
||||||
|
| --- | --- |
|
||||||
|
| 为何继承 | 文档元素"是"版面元素 + 内容;按 bbox 裁剪的下游函数可同时吃两种。代价: dataclass `__eq__` 要求同类,`OcrDocumentElement != OcrLayoutElement`;`OcrLayoutElement` docstring 加警示"追加带默认值字段前须评估子类" |
|
||||||
|
| 为何不加 `level`/`captions`/`footnotes` | 样本中 21 个 title 全是 level 1(无信息量);caption/footnote 在全部样本里恒为空——为从未见过的路径加公共字段不可验证。嵌套子块独立成元素的方案零新增字段,且服务将来开始关联 caption 时不丢信息 |
|
||||||
|
| `markdown` | 整篇产物,属 result 不属元素。**含悬空图片引用** ``(库不外传图片),docstring 注明 |
|
||||||
|
| 阅读序 | 与服务 `para_blocks` 顺序一致,页内按页序;嵌套子块紧随其父块 |
|
||||||
|
|
||||||
|
### 4.2 端口(`ports.py`)
|
||||||
|
|
||||||
|
| 端口 | 签名 | 说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `OcrDocumentPort`(新,`runtime_checkable`) | `async def parse_document(self, document: bytes) -> OcrDocumentResult` | 对应 `/parse` 整份文档 |
|
||||||
|
| `OcrDocumentTransport`(新,`runtime_checkable`) | `async def parse_document(self, *, document: bytes, source: SourceConfig, call_id: str) -> OcrDocumentTransportResult` | **不往 `OcrTransport` 加成员**(会破坏全部现有实现的 `isinstance`);`MonkeyOcrTransport` 同时实现两者 |
|
||||||
|
|
||||||
|
`OcrClient.parse_document(document, *, session_id=None, parent_call_id=None, tenant_id=None, meta=None, call_deadline_s=None)`: 签名与 `parse_layout` 对称。入口校验先于 `_CallContext`(同 `_call` 既有顺序): `bytes` 非空、`validate_caller_dimensions`、`ensure_call_deadline`、**transport 须满足 `OcrDocumentTransport`,否则 `TypeError`**(装配错误当场报)。
|
||||||
|
|
||||||
|
### 4.3 其它公共面
|
||||||
|
|
||||||
|
| 项 | 变更 |
|
||||||
|
| --- | --- |
|
||||||
|
| `CallOperation` / `CALL_OPERATIONS`(`types.py:279,285`) | 新增 `"parse_document"`。`CALL_OPERATIONS` 目前**无任何读者**,新增测试钉死 `get_args(CallOperation) == CALL_OPERATIONS` |
|
||||||
|
| 包根导出 | `OcrDocumentElement`、`OcrDocumentResult` |
|
||||||
|
| 遥测 | 同一张 `llm_calls`,`operation="parse_document"`;占位仍为 `<ocr:document image_bytes=N>`(**不改字面**: 改名会破坏既有分组与 `test_telemetry.py` 对格式的锁定);response 摘要 `<document elements=N pages=M md_chars=K>`,**永不写入正文**;无新列、无 DDL 变更 |
|
||||||
|
| 配置 | **无新键**。文档线独立 scope 是部署方式,不是新配置(`.env.example` 加示例段) |
|
||||||
|
|
||||||
|
## 5. Transport 行为(`transports/monkey_ocr.py`)
|
||||||
|
|
||||||
|
| 关切 | 决策 |
|
||||||
|
| --- | --- |
|
||||||
|
| 上传标签 | `_files()` 改为 `_upload_part(data)`,按 §3 Q2 的固定顺序嗅探: PDF → `("doc.pdf", data, "application/pdf")`,其余 → 原样 `("image.jpg", data, "image/jpeg")`。三个方法共用。包内文件名前缀由 `image_` 变 `doc_`,解包靠后缀 `endswith("_middle.json")`(`monkey_ocr.py:179`),不受影响 |
|
||||||
|
| 本地预检(仅 document) | 嗅探结果为"未知"(HTML 付费墙页面、乱码、空串)→ `RequestRejectedError`(**不带 status_code**,服务未响应),不发 HTTP。已知图片魔数: JPEG `FFD8FF`、PNG、TIFF(`II*\0`/`MM\0*`)、BMP(`BM`)、GIF、WebP(`RIFF…WEBP`);魔数表之外的可解码图片格式在 document 路径会被拒,需要时由调用方走 `parse_layout` |
|
||||||
|
| 两段协议 | 与 `parse_layout` 同: POST `/parse` → 校验 `success is True` + `download_url` → GET → ZIP。document 路径的 GET 改**流式读取并累计字节**,整包上限 256 MiB(模块常量),超限 → `ResultInvalidError`;HTTP 状态错误须先 `aread()` 再 `raise_for_status()`,保持"响应体摘要随所有分支"的 issue #10 口径 |
|
||||||
|
| 解析卸载 | ZIP 解包 + JSON 解析 + 结构校验是 bytes→结果的**纯函数**,在 `asyncio.to_thread` 中执行: 同步解析大对象会占住事件循环,使取消与 `call_deadline_s`(基于 `asyncio.timeout`,`deadline.py:69-79`)无法及时生效。取消后线程自然收尾、结果丢弃。`parse_layout` 不改 |
|
||||||
|
| 外部数据异常边界 | 纯函数内**统一**把外部字节引起的已知异常转 `ResultInvalidError`: 函数级 `ValueError`(含 `JSONDecodeError`/`UnicodeDecodeError`)、`TypeError`、`KeyError`、`OverflowError`(巨大整数转 float,实测 `float(10**400)` 抛它)、`RecursionError`(深嵌套 JSON)、`zipfile.BadZipFile`/`LargeZipFile`;**仅包在 `archive.open/read` 外**的窄 `try`: `RuntimeError`(标准库对加密成员抛它)、`NotImplementedError`(未知压缩方法)、`zlib.error`、`EOFError`。取消(`CancelledError`)不在其内,原样穿透。理由: `ocr.py:303-321` 的 `_call` 只接 `PolyGatewayError`,漏出的原生异常既不写终态遥测行,又会让持有的半开探针不被归还 |
|
||||||
|
| ZIP 成员与体积 | 读 `*_middle.json`(取第一个,同 layout);`.md` 按 `middle 名去掉 "_middle.json" + ".md"` 精确配对(不取"第一个 .md");缺失 → `ResultInvalidError`。读取前检查 `ZipInfo.file_size`: middle ≤ 64 MiB、md ≤ 16 MiB,超限 → `ResultInvalidError`。**不用 `extractall`**,不解出 `images/`;成员数由整包上限间接约束。仅 document 路径;`parse_layout` 不改(医院线零变更) |
|
||||||
|
| 文字提取 | 块内 `lines[].spans[]`: 取 `content`;`inline_equation` 类型 span 包 `$…$`(上游惯例,**样本中从未出现**);行内 span 按 `""` 拼、行间按 `"\n"` 拼。**样本全是 1 行 1 span**,多行/多 span 路径无真实样本,仅构造 fixture 测试,文档标"未经实测" |
|
||||||
|
| 其它页内容 | `discarded_blocks` 忽略(E12: 服务端已丢弃脚注/页眉页脚/页码,span 内容全空,库无物可取);`need_drop` 不读取、不失败(该流水线硬编码为 False) |
|
||||||
|
| 嵌套子块 | 默认部署不产生(E12);仅作最小防御: 非 `*_body` 的嵌套子块若有文字,作为独立元素紧随父块输出,**不静默丢弃**。该路径无真实样本,仅构造 fixture 测试 |
|
||||||
|
| `raw` | `{"success": True, "download_url": …}`,与 `parse_layout` 同形 |
|
||||||
|
|
||||||
|
### 5.1 结构契约(缺失 / 空值 / 类型错误 / 未知类型逐项裁决)
|
||||||
|
|
||||||
|
架构要求严格区分合法"无内容"与调用失败(`ARCHITECTURE.md:427,672`)。下表是实现与测试的唯一依据。
|
||||||
|
|
||||||
|
| 情形 | 处理 |
|
||||||
|
| --- | --- |
|
||||||
|
| `pdf_info` 缺失或非 list | `ResultInvalidError` |
|
||||||
|
| `pdf_info == []`(零页) | `ResultInvalidError`: 成功解析的文档至少 1 页(与 layout 路径"elements 空合法"不同,差异写入 docstring) |
|
||||||
|
| 页非 dict / `page_size` 非 2 个正有限数 | `ResultInvalidError`(复用 `_parse_page_size`) |
|
||||||
|
| `page_idx` 非 int、为 bool、为负、或 ≠ 页在列表中的位置 | `ResultInvalidError`(layout 路径的"回退枚举"会使 `page_sizes` 按 `page_index` 索引的契约静默错位) |
|
||||||
|
| `para_blocks` 缺失 | 合法空页(空白页 / 无可识别块);非 list → `ResultInvalidError` |
|
||||||
|
| 块非 dict、`type` 缺失/非 str/空白 | `ResultInvalidError` |
|
||||||
|
| `bbox` | 须为长度 4 的有限数值且 `x1 ≤ x2`、`y1 ≤ y2`(允许零宽/零高,**不做**整数化退化校验——那条专为 CHS shim 的 `int()` 裁剪);反序/非有限 → `ResultInvalidError`。坐标系与 `page_sizes` 同为服务原生坐标,页码从 0 起;越界框不检查、不裁剪(图内语义归调用方,`ARCHITECTURE.md:250`) |
|
||||||
|
| 含 `lines` 的块(text/title/interline_equation/未知类型) | `lines` 须 list,每行 dict 且 `spans` 为 list,每 span 为 dict 且 `content` 为 str;违反 → `ResultInvalidError`;`lines == []` → `text=""` |
|
||||||
|
| table | 嵌套 `table_body` 的 span 取 `html`,无则取 `latex`(上游有该分支);二者皆无 → `ResultInvalidError`(表格无内容不静默为空串) |
|
||||||
|
| image | 嵌套 `image_body` → `text=""` |
|
||||||
|
| 未知类型且既无 `lines` 也无 `blocks` | `text=""`,`type` 原样保留(开放字符串) |
|
||||||
|
| `.md` | 成员缺失 → `ResultInvalidError`;按 `utf-8-sig` 解码(吞 BOM),失败 → `ResultInvalidError`;空串合法 |
|
||||||
|
|
||||||
|
## 6. 错误分类与批处理后果
|
||||||
|
|
||||||
|
| 失败 | 分类 | 治理结果 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 上传前: 非 bytes/空 | `TypeError`/`ValueError` | 统计边界外(同 `_call` 既有) |
|
||||||
|
| 嗅探为"未知"(§5) | `RequestRejectedError`(无 status) | 不重试、不换源;仅归还探针,**不计熔断** |
|
||||||
|
| 500 + `Parsing failed:` + 已知标记 | `RequestRejectedError`(`status_code=500`) | 不重试;熔断**记成功**且不计窗口(`_gate_on_terminal`: `exc.status_code is not None`);不喂选源器 |
|
||||||
|
| 429 | `TransientError`(`status_code=429`) | 照扣重试预算、喂选源器失败; **熔断两通道均不计**(`memory/breaker.py:235-244`: `rate_limited` 是背压不是故障) |
|
||||||
|
| 其余 5xx、超时、网络错误 | `TransientError` | 重试 + 计熔断(现状) |
|
||||||
|
| `CallDeadlineExceeded`(整体期限到期) / 外部取消 | 专用异常 / `CancelledError` | **不是**源故障: 不计熔断; 在飞探针归还; 终态遥测行照写(`ocr.py:303-321`)。期限不会被混入 `TransientError` |
|
||||||
|
| 401/403 | `SourceDeadError` | 现状 |
|
||||||
|
| 其余 4xx、`success!=true`(附 200) | `RequestRejectedError` | 现状 |
|
||||||
|
| ZIP 坏 / 加密或未知压缩成员 / middle 或 md 缺失 / 超限 / 结构异常(§5.1)/ `page_idx` 错位 / 零页 | `ResultInvalidError` | 不重试;熔断记成功不计窗口(`_gate_on_terminal`: `ocr.py:493-499`) |
|
||||||
|
|
||||||
|
已知标记(模块常量,小写子串匹配,取 E5 实测): `failed to open stream`、`cannot open empty stream`、`cannot save with zero pages`、`is no pdf`。**不含** `unsupported file type`: 嗅探后它只可能由服务契约变化或我方标签 bug 引起,应响亮失败并计熔断。`MonkeyOcrTransport` 构造参数可覆盖该元组,不新增 `SourceConfig` 字段或环境变量键。
|
||||||
|
|
||||||
|
**为什么 Q4 不能不做**(读代码确认的机制链): ① 熔断按 `source_name` 计数(内存 `memory/breaker.py:56-61`;Redis 键 `pgw:gate:{scope}:{source}`,`backends/redis/breaker.py:308-311`);② 有效阈值 = `max(配置值, 源并发×2)`(`config.py:503-508`);③ 连续失败通道仅在**窗口无充分健康证据**时开路——"充分健康证据"= 窗口样本 ≥ `min_calls`(默认 10)**且**失败率 < `fail_rate`(默认 0.6)(`memory/breaker.py:84-90`、开路判据 `:252`),故样本不足 **或** 失败率 ≥ 0.6 时连败即开路。每份文档约 15 秒、并发 2,窗口凑不够 10 个样本,抑制不生效;④ 一份坏 PDF 重试 `max_attempts` 次即可在低并发配置下开路,冷却内同 scope 的好文档被 `fail_fast` 拒绝。影响范围是**同 scope**: 默认 memory 后端是进程内,只有 Redis 熔断后端加相同 scope 才跨进程(是否波及医院线取决于部署)。⑤ 下游语义: `migrations/chsanalyzer.md:72` 记录 CHSAnalyzer 把库侧"服务不可用"一族按 arq 延期重投且不耗业务失败预算——**若 CHSAnalyzer3 沿用,永久损坏的 PDF 会被无限延期。此点未向下游证实,需人类确认。**
|
||||||
|
|
||||||
|
白名单的脆弱性(诚实声明): 标记是 PyMuPDF 的界面文案,随服务版本漂移; 服务把所有异常都包成 `Parsing failed:`,若把磁盘满等真故障也包成同样文案会被误判为坏输入。**误判代价是"该文档失败而非延期"(下游重跑失败集合即可),反向代价是熔断误开**——取前者。
|
||||||
|
|
||||||
|
## 7. 长文档与资源(issue 要求 4)
|
||||||
|
|
||||||
|
| 结论 | 依据 |
|
||||||
|
| --- | --- |
|
||||||
|
| 默认值与熔断**不会把长文档判为慢** | 无延迟型判定(§1 E9);选源器只数成败(`sources.py:61-64`);`StallClock` 只计非生产性等待(`ocr.py:343-344`) |
|
||||||
|
| 把"慢"变成失败的路径有两条,性质不同 | ① httpx 超时: `monkey_ocr.py:63-68` → `TransientError`(`__cause__` 为 `TimeoutException`)→ 重试 + 计熔断; ② `call_deadline_s` 到期 → `CallDeadlineExceeded`,**不计熔断**(§6) |
|
||||||
|
| **`timeout_s` 不是总耗时上界**(Codex 审查纠正) | `httpx.Timeout(source.timeout_s)` 是**逐次网络操作**的超时(`httpcore` 每次读单独计时): POST 读超时≈服务处理时长上限(服务处理期间无字节),但 GET 下载只要持续有分块到达,总时长可超过任意多个 `timeout_s`。故初稿"最坏约 2×`timeout_s`"不成立,**不能据此推出租约安全** |
|
||||||
|
| 租约与探针租约的真实关系 | `config.py:338-345` 只守卫 `timeout_s ≤ lease_ttl_s`(默认 1500);限流租约到期即被移除(`memory/limiter.py:98-103`),探针租约到期可被接管(`memory/breaker.py:161-163`,派生 TTL 见 `config.py:509-518`)。**硬保证只能来自 `call_deadline_s`**: 文档线应配 `call_deadline_s ≤ lease_ttl_s`,使任何一次尝试都不会活过租约; 守卫改造(跨 scope 配置语义)**不在本次**,另列 follow-up |
|
||||||
|
| 配置口径(写入 `.env.example` 与 README) | 文档线独立 scope + `MAX_CONCURRENCY=2`(满足"并发不超过 2";sync-only 见下行)+ `RETRY__MAX_ATTEMPTS=2` + `timeout_s ≥ 最大页数×4s×余量`(实测 1.5–4 s/页;共用 GPU 争用再乘 3–5 倍)+ **`call_deadline_s ≤ lease_ttl_s` 给单份封顶**。有效熔断阈值 `max(配置, 并发×2)=4` > 单份坏文档的重试次数 |
|
||||||
|
| 重试放大 | POST 成功(服务已算完)而 GET 失败时,重试**重新 POST 整份文档重算**; 超时重试同理,且服务端是否因断连取消计算无证据——假定不取消 |
|
||||||
|
| 无页数预检 | 库不依赖 PDF 库,发送前不知页数; 300 页文档会占满超时被重投。页数过滤是下游职责 |
|
||||||
|
| 内存 | document 路径 GET 改流式并累计字节,**整包上限 256 MiB**(§5);只解必要成员,成员再受声明大小上限约束。**诚实标注(验证者 M-4)**: 合并分块 `b"".join` 与随后的 `BytesIO` 会让瞬时峰值接近 2 倍上限,README 已写明预留。httpx 非流式 `get` 会先 `aread()` 整包(`httpx/_client.py:1635-1637`),这是改流式的原因; `parse_layout` 不改 |
|
||||||
|
| 服务端磁盘 | 每次调用在服务 `/app/tmp` 与 `/static` 留产物,几百篇约几百 MB 到 GB 级——**告知服务运维**(库不管理) |
|
||||||
|
| 并发上限的进程边界 | `MAX_CONCURRENCY` 默认进程内; 多 worker 共享上限需 `PGW_LIMITER_BACKEND=redis` 且同 scope |
|
||||||
|
| **服务端串行与队头阻塞(E14)** | 若线上为 sync-only: ① 每实例 `/parse` 全局串行,"并发 2"不提速,第二份文档在服务端排队 → `timeout_s` 须覆盖"排队+处理"≈ 并发数×单份耗时×余量; ② 一份 15 页文档(按 1.5–4 s/页估算约 22–60 秒)的队头阻塞可能把医院线(`table_locator`)请求推到它们自己的 `timeout_s`,触发**它们**的 Transient 与熔断——"我们的长文档误伤他们"比"他们的判定误伤我们"更现实; ③ 7867 实例与 embedding 服务同卡(部署记录),重负载互相影响,不在库治理范围。**须向运维确认 backend**(启动日志 `model_manager.py:37-38`)。指引: sync-only 时文档线每实例并发 1、错峰运行、页数过滤在调用方 |
|
||||||
|
| 服务限制(告知下游) | 脚注/页眉页脚/页码在服务端即被丢弃(E12): 论文表格脚注的显著性与缩写说明不会出现在结果里; `markdown`/`elements` 均无此内容。库不能也不应补偿 |
|
||||||
|
|
||||||
|
## 8. 旧版行为审计
|
||||||
|
|
||||||
|
| 行为 | 裁决 |
|
||||||
|
| --- | --- |
|
||||||
|
| `_files()` 固定 `image.jpg` | **替换**: 嗅探为 PDF → `doc.pdf`; 其余**保留**(字节级不变) |
|
||||||
|
| `parse_layout` 收到 PDF | **有意改变上传分支**(`ImageDataset` → `PymuDocDataset`,E10)。差分证据: 文本型与位图型 2 页 PDF 结果逐字相同(E3); **旋转页、带批注页未测,不构成保证**。该路径无已知消费者(医院线只传图片,CHSAnalyzer3 走新方法),故列 H6 由人类批准,而不是声称"零风险" |
|
||||||
|
| `recognize_text` 收到 PDF | **修复**: 静默错误结果(E4)变为整份全文; 文档注明"不承诺 PDF 语义,请用 `parse_document`" |
|
||||||
|
| `parse_layout` 的 bbox 整数化退化校验 / `page_idx` 回退 / 无 ZIP 体积上限 | **保留不动**(医院线零变更); document 路径各自更严/更宽,差异写入 docstring |
|
||||||
|
| `_classify_status` 共享分类 | **保留**(默认参数不变) |
|
||||||
|
| `OcrTransport` 协议 | **保留**,不加成员 |
|
||||||
|
| `check_health`、重试/选源/限流/熔断/遥测编排 | **保留**: `parse_document` 复用同一 `_call/_run/_attempt` 循环,只扩 `_OcrKind` 与 `_invoke` 分派 |
|
||||||
|
|
||||||
|
## 9. 非功能维度
|
||||||
|
|
||||||
|
| 维度 | 回答 |
|
||||||
|
| --- | --- |
|
||||||
|
| 并发与取消 | 无新共享可变状态; `CancelledError` 路径与 `parse_layout` 一致(退避 sleep/限流等待/两段 HTTP 均可取消,permit 在 `finally` settle+release,探针取消归还); **新增**的 ZIP 解析在 `to_thread` 中,取消时立即返回(线程自然收尾、结果丢弃),不会像同步解析那样占住事件循环使取消/期限失效。`call_deadline_s` 到期会取消在飞请求,服务端是否继续计算未知 |
|
||||||
|
| 降级方向 | 准入阶段(选源/限流/熔断)后端故障 → 报错不放行(铁律,复用既有 `SourceAdmission`); 已发出请求之后的记账写回(`_record_quietly`,`ocr.py:511-517`)与遥测写失败 → warning 降级。**不能**把"后端故障一律上抛"理解为所有阶段 |
|
||||||
|
| 幂等与重复 | 只读幂等,重放安全; 代价见 §7"重试放大" |
|
||||||
|
| 持久化与原子性 | 库内无持久化; ZIP 解包在内存(`BytesIO`),不落盘 |
|
||||||
|
| 零业务假设 | 无医学/论文词汇进入库代码与 fixtures; `type` 仍为开放字符串 |
|
||||||
|
|
||||||
|
## 10. 测试策略
|
||||||
|
|
||||||
|
fixtures 一律**合成**(`tests/unit/test_monkey_ocr.py` 的 `_zip_bytes`/`_page` 风格上扩一个 document builder),**不提交任何真实 ZIP**(第三方内容许可不明; 医疗数据不入库,同 M3 设计 §11)。
|
||||||
|
|
||||||
|
| 层 | 内容(每条先红后绿) |
|
||||||
|
| --- | --- |
|
||||||
|
| unit/transport | 嗅探: 纯 PDF、前缀垃圾的 PDF、起始为图片魔数但内含 `%PDF-` 字样的图片(须判图片)→ 各自标签,非 PDF 逐字 `image.jpg`/`image/jpeg`,**三个方法各钉一次**(`test_multipart_shape` 现有断言保持成立); 文档解析: title/text(含 `$X$`)/interline_equation/带 nested `table_body` 的 table/带 `image_body` 的 image 的 text 取值; 垃圾 `content_list.json` 与 `images/x.jpg` 并存时输出与其无关; nested caption/footnote 成独立元素(标未经实测); `inline_equation` 包 `$`; 多行/多 span 拼接; `markdown` 精确配对与缺失; 体积上限; §5.1 契约表**逐行一条用例**(零页、`page_idx` 为 bool/负/错位、`para_blocks` 缺失=空页、非 list、块无 type、反序/非有限 bbox、零宽 bbox 合法、`lines` 各层类型错误、table 无 html/latex、未知类型、`.md` 缺失/非法编码/BOM/空串); `discarded_blocks` 不进结果 |
|
||||||
|
| unit/资源与异常边界 | 加密成员、未知压缩方法、`10**400` 整数、深嵌套 JSON、CRC 损坏 → 全部 `ResultInvalidError`(**不漏出原生异常**),并断言经 `OcrClient` 后终态遥测行已写、半开探针已归还; 整包流式累计超限(大图片包 + 小 middle/md)→ ResultInvalid; 声明大小超限成员 → ResultInvalid; 解析期间取消 → 立即 `CancelledError` 穿透(用阻塞式解析桩验证事件循环未被占住) |
|
||||||
|
| unit/分类 | 魔数预检(HTML 页面、空串、未知魔数); 四个已知标记 → `RequestRejectedError(status_code=500)`; 未知 500 仍 Transient; `unsupported file type` 仍 Transient; 默认 `_classify_status` 行为不变(现有 `test_http_status` 保持) |
|
||||||
|
| unit/client | 桩 transport 走完整治理循环; `isinstance` 缺 `OcrDocumentTransport` → `TypeError`; **无延迟判慢**: 在成功尝试内推进远大于 `timeout_s` 的时钟,断言成功、熔断 CLOSED、`fails==0`(**含"熔断已有失败计数"与半开探针两种前置状态**,不只测初始 CLOSED); 对照: 超时 Transient → 计失败 → 重试; **持续慢速下载**(每个分块都在 read 超时内、总时长超过 `call_deadline_s`)→ `CallDeadlineExceeded` 而非 Transient,且不计熔断; 429 → 照扣重试预算但熔断窗口不变; ResultInvalid/本地预检不计熔断窗口; 取消穿透且探针归还; 遥测行 `operation="parse_document"`、占位字面量、摘要不含正文; `_summarize` 三类结果各一条(新结果必须显式区分,防止静默套用 layout 摘要而漏记 `md_chars`); 终态行; 调用方维度(`tenant_id`/`meta`)与 `reasoning_applies=False` |
|
||||||
|
| unit/契约 | `get_args(CallOperation) == CALL_OPERATIONS`; `test_ports.py` 新增 Protocol 的 `runtime_checkable` 用例,并断言 `OcrTransport` 成员集合**未变** |
|
||||||
|
| integration(`slow`,opt-in,串行) | 手写最小合法 PDF(无第三方内容)打真实服务: `parse_document` 返回 ≥1 页、元素均有 `text` 类型; 同一 PDF 的 `recognize_text` 不再返回 E4 的幻觉文本; 损坏 PDF → `RequestRejectedError` 且熔断仍 CLOSED。成败取决于外部服务当下状态,按 CLAUDE.md §4.6 标 `slow`,不进日常套件 |
|
||||||
|
|
||||||
|
## 11. 落地时需同步的位置(grep 全引用面所得)
|
||||||
|
|
||||||
|
| 类别 | 位置 |
|
||||||
|
| --- | --- |
|
||||||
|
| 代码 | `types.py`(Literal/常量/新类型)、`ports.py`、`ocr.py`(`_OcrKind`/`_AttemptOutcome`/`_invoke`/`_emit`/`_summarize` 签名)、`transports/monkey_ocr.py`、`__init__.py`(导出 + `__version__`)、`pyproject.toml` |
|
||||||
|
| 校验工具 | `tools/check_wiki_alignment.py:37-46`(`_SIGNATURE_TARGETS` 加 `OcrClient.parse_document`) |
|
||||||
|
| wiki 注册 | `research-wiki/schemas/llm-calls.md`(operation 取值)、`metrics/ocr-call-success.md`、`metrics/call-telemetry-coverage.md`(无推理入口列表) |
|
||||||
|
| 文档 | `README.md`(方法数、能力表、安装版本约束)、`CHANGELOG.md`、`.env.example`(文档线 scope 示例)、`ARCHITECTURE.md` §5.1/§7.10/D9 注记、M3 设计 §4 勘误(指向本设计 E3/E4)、`migrations/chsanalyzer.md`(CHSAnalyzer3 新消费路径) |
|
||||||
|
| 测试 | `test_telemetry.py` 的 operation 参数化、`test_ports.py`、`test_monkey_ocr.py`、`test_ocr_client.py`、`tests/integration/test_monkey_ocr_live.py` |
|
||||||
|
| 无需迁移 | `llm_calls.operation` 列为 `TEXT` 无 CHECK(`telemetry/schema.py`),新值不需 DDL; 外部按 operation 分组的看板会自然多一组 |
|
||||||
|
|
||||||
|
CLAUDE.md §4.4.1 发布流程按其清单执行(README 在构建前改完; 1.3.11 与历史把新增公共方法放 patch 的惯例一致)。
|
||||||
|
|
||||||
|
## 12. 评审与待人类拍板
|
||||||
|
|
||||||
|
**已做**: Claude 自审; 一轮独立 subagent(全新上下文、只读)对 Q1-Q5 与遥测/引用面逐项论证,并核对本机 MonkeyOCR 源码(E10-E14,关键行已由我对源码与线上各复核一次)。**采纳其修正**: 弃初稿"content_list+middle 对位"(Q3,以 `\$` 污染 23/134 为据)、砍 `level/captions/footnotes`(Q1)、`OcrDocumentTransport` 另起 Protocol 而非扩 `OcrTransport`(§4.2)、document 路径的 `page_idx` 严格与 bbox 宽松(§5)、确定性 500 只做新路径窄化(Q4)、ZIP 体积上限、`_summarize` 改法。**更正我方事实**: 初稿称"结果包无 `images/`",实为含图文档才有(E6)。**Codex 独立审**: 一轮已完成,初审结论"否——先修正再审批";逐条核验后修订,见文末"审查记录"。
|
||||||
|
|
||||||
|
**需要人类决定**:
|
||||||
|
|
||||||
|
| # | 决策 | 推荐 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| H1 | §4 公共面(新方法 + 新 Protocol + 继承式元素、`text` 唯一新增字段、嵌套子块独立成元素) | 批准 |
|
||||||
|
| H2 | 既有端点的分类/校验问题: 本设计只在新路径窄化; 以下两项属**医院线既有行为变更**,另开 issue 供评估: (a) 共享 `_classify_status` 对确定性坏输入 500 仍归 Transient; (b) `recognize_text` 不校验 `success`(服务内部异常时返回 200+`success:false`+空文本,被当"合法无文字"且熔断记成功,E13,源码推断未线上复现) | 同意,并授权我建这两个 issue |
|
||||||
|
| H5 | **需求替换**: issue #34 原文提议 `parse_document(data, *, filename 或 media_type)` 由调用方声明类型; 本设计改为库按内容嗅探、**不提供声明参数**(理由见 §3 Q2: 冗余信息、笔误会变熔断事件、前缀垃圾 PDF 已被嗅探覆盖) | 批准替换; 若你更想要显式参数,代价是新增公共参数 + 与嗅探结果不符时的拒绝规则 |
|
||||||
|
| H6 | **有意改变既有方法的上传分支**: `parse_layout(PDF)` 与 `recognize_text(PDF)` 的 PDF 将走 `PymuDocDataset`/`doc.pdf`(前者结果在已测样本上逐字相同,旋转/批注页未测;后者是修复 E4) | 批准 |
|
||||||
|
| H3 | CHSAnalyzer3 是否把"服务不可用"一族按延期重投处理(§6 ⑤)——决定 `RequestRejectedError` 路径是否必要地"关键" | 请确认; 不影响实现,影响下游使用指引 |
|
||||||
|
| H4 | 实现完成后的发布(合并 main、tag、构建、上传 registry、建 Release、关闭 #34)**逐步等你明确指令**,不随实现自动执行 | 同意 |
|
||||||
|
|
||||||
|
## 审查记录
|
||||||
|
|
||||||
|
Codex 一轮独立只读审查(上游无法读取 Gitea issue,需求判断基于设计转述)。结论: **否,先修正**。逐条核验与处置:
|
||||||
|
|
||||||
|
| # | Codex 意见 | 核验 | 处置 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 1 阻断 | "2×`timeout_s`"不是总耗时上界,租约安全结论失效 | **属实**: httpcore 逐次读计时,GET 持续分块可超任意多个 `timeout_s`; 租约/探针租约到期语义见 `memory/limiter.py:98-103`、`memory/breaker.py:161-163` | §7 删除\"2×即安全\",改为硬保证只来自 `call_deadline_s ≤ lease_ttl_s`; 测试加慢速持续下载 |
|
||||||
|
| 2 高 | 调用方声明类型被自动嗅探替换,理由与 Q4 自洽性不足; 前缀垃圾 PDF 描述矛盾; \"1024 字节\"依据未核 | **部分属实**: 替换确属需求变更,须显式批准; 前缀垃圾 PDF 服务**实测可解析**,故改嗅探为\"图片魔数优先 → 前 1KB 内 `%PDF-`\",该场景不再被拒; 1024 字节来自 Adobe 阅读器的头部容忍惯例,不是 ISO 32000 正文,已改述为兼容性描述 | §3 Q2 重写 + 三方法结局表; 新增 H5 |
|
||||||
|
| 3 高 | `parse_layout(PDF)`\"零风险/不变\"与 E10 矛盾 | **属实**: 分支确实变了。补做位图型 PDF 差分(逐字相同); 旋转/批注页仍未测 | §8 改述\"有意改变\",新增 H6 |
|
||||||
|
| 4 高 | 成员大小限制不限整包内存; 同步解析使取消/期限失效 | **属实**(`httpx/_client.py:1635-1637` 非流式先 `aread()`; `deadline.py` 基于 `asyncio.timeout` 无法打断同步代码) | §5: document 路径流式 + 整包 256 MiB 上限 + `to_thread` 卸载; 测试加解析期间取消 |
|
||||||
|
| 5 中 | ZIP/结构异常边界缺失,加密成员/未知压缩/巨大整数会漏出原生异常 | **属实**: 实测 `float(10**400)` 抛 `OverflowError`,不在既有异常元组内; 漏出后 `_call` 不写终态行、探针不归还 | §5 明确异常转换边界,§10 加专项测试 |
|
||||||
|
| 6 中 | 逐块文字/空结果/未知类型/编码契约不完整 | **属实** | 新增 §5.1 结构契约表,§10 逐行一条用例 |
|
||||||
|
| 7 中 | bbox 放宽与\"继承使裁剪函数可通吃\"自相矛盾 | **属实** | 改为 `x1≤x2` 且 `y1≤y2`(允许零宽,去掉整数化退化),坐标系/页码约定写明 |
|
||||||
|
| 8 中 | 429 记账写错; 窗口条件写反; 遗漏 `call_deadline_s` | **全部属实**(`memory/breaker.py:84-90,235-244`): \"失败率<0.6\"是健康证据而非\"证据不足\" | §6/§7 订正,增 429 与期限两行 |
|
||||||
|
| 9 低 | `_summarize` 风险描述不准 | **属实**: 实际是漏记 `md_chars` | 措辞订正 |
|
||||||
|
| 引用核对 | `api/main.py:435-441` 行号偏移(分派在 `:164-170`); `config.py:503-507` 应为 `:508`; Redis 键缺 `pgw:` 前缀 | **属实** | 已订正 |
|
||||||
|
|
||||||
|
未采纳: 无(九条均成立,部分处置方式有调整)。**订正后未再送 Codex 复审**——修订量大,建议人类审阅时一并关注 §3 Q2、§5、§5.1、§7。
|
||||||
@@ -525,6 +525,20 @@
|
|||||||
"relation": "implements",
|
"relation": "implements",
|
||||||
"evidence": "计划头部与设计 §9 批准状态(2026-09-10 H1-H8 全数获批)",
|
"evidence": "计划头部与设计 §9 批准状态(2026-09-10 H1-H8 全数获批)",
|
||||||
"added": "2026-09-10T16:48:37.849653+00:00"
|
"added": "2026-09-10T16:48:37.849653+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "plan:2026-10-21-issue34-ocr-document",
|
||||||
|
"target": "design:2026-10-21-issue34-ocr-document-design",
|
||||||
|
"relation": "implements",
|
||||||
|
"evidence": "实现 #34 已批准设计 H1/H5/H6",
|
||||||
|
"added": "2026-10-22T02:45:49.844612+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "review:2026-10-22-issue34-branch-review",
|
||||||
|
"target": "plan:2026-10-21-issue34-ocr-document",
|
||||||
|
"relation": "informs",
|
||||||
|
"evidence": "合并前验收与审查结论",
|
||||||
|
"added": "2026-10-22T06:15:33.014451+00:00"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
+9
-10
@@ -1,9 +1,8 @@
|
|||||||
# Research Wiki 索引
|
# Research Wiki 索引
|
||||||
|
|
||||||
> 自动生成,更新时间:2026-10-02 02:23 UTC
|
> 自动生成,更新时间:2026-10-22 06:15 UTC
|
||||||
|
|
||||||
## design (47)
|
|
||||||
|
|
||||||
|
## design (49)
|
||||||
- [1.3.4 推理意图与测试证据设计](designs/2026-09-09-134-thinking-contracts-design.md) `design:2026-09-09-134-thinking-contracts-design`
|
- [1.3.4 推理意图与测试证据设计](designs/2026-09-09-134-thinking-contracts-design.md) `design:2026-09-09-134-thinking-contracts-design`
|
||||||
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
|
- [2026-07-20-m1-core-design](designs/2026-07-20-m1-core-design.md) `design:2026-07-20-m1-core-design`
|
||||||
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
|
- [2026-07-20-m2-distributed-design](designs/2026-07-20-m2-distributed-design.md) `design:2026-07-20-m2-distributed-design`
|
||||||
@@ -41,6 +40,7 @@
|
|||||||
- [M2.5 治理韧性: 半死源隔离与健康感知调度](designs/m25-resilience.md) `design:m25-resilience`
|
- [M2.5 治理韧性: 半死源隔离与健康感知调度](designs/m25-resilience.md) `design:m25-resilience`
|
||||||
- [M3 OCR 端口族设计](designs/m3-ocr.md) `design:m3-ocr`
|
- [M3 OCR 端口族设计](designs/m3-ocr.md) `design:m3-ocr`
|
||||||
- [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration`
|
- [M4 迁移验证设计(GovDoc→CHS,发 v1.0)](designs/m4-migration.md) `design:m4-migration`
|
||||||
|
- [OCR 整份文档解析设计(#34)](designs/2026-10-21-issue34-ocr-document-design.md) `design:2026-10-21-issue34-ocr-document-design`
|
||||||
- [OpenAI消息保真设计](designs/2026-10-01-issue29-openai-fidelity-design.md) `design:2026-10-01-issue29-openai-fidelity-design`
|
- [OpenAI消息保真设计](designs/2026-10-01-issue29-openai-fidelity-design.md) `design:2026-10-01-issue29-openai-fidelity-design`
|
||||||
- [stall 判定改为非生产性等待口径](designs/issue8-stall-budget.md) `design:issue8-stall-budget`
|
- [stall 判定改为非生产性等待口径](designs/issue8-stall-budget.md) `design:issue8-stall-budget`
|
||||||
- [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields`
|
- [响应可观测字段扩展(Issue #3)](designs/response-observability-fields.md) `design:response-observability-fields`
|
||||||
@@ -49,11 +49,11 @@
|
|||||||
- [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design`
|
- [推理开关能力建模与 reasoning_tokens 采集(issue #5 + #6)](designs/2026-08-02-thinking-capability-design.md) `design:2026-08-02-thinking-capability-design`
|
||||||
- [推理档位一等化设计(issue #20 及其一般形式)](designs/reasoning-effort.md) `design:reasoning-effort`
|
- [推理档位一等化设计(issue #20 及其一般形式)](designs/reasoning-effort.md) `design:reasoning-effort`
|
||||||
- [治理后端故障归位为 scope 级不可用(Issue #7)](designs/governance-backend-error.md) `design:governance-backend-error`
|
- [治理后端故障归位为 scope 级不可用(Issue #7)](designs/governance-backend-error.md) `design:governance-backend-error`
|
||||||
|
- [统一完成判定设计(#30/#31)](designs/2026-10-14-issue30-31-completion-verdict-design.md) `design:2026-10-14-issue30-31-completion-verdict-design`
|
||||||
- [调用方自定义维度设计(issue #11)](designs/issue11-caller-dimensions.md) `design:issue11-caller-dimensions`
|
- [调用方自定义维度设计(issue #11)](designs/issue11-caller-dimensions.md) `design:issue11-caller-dimensions`
|
||||||
- [采样参数透传设计(issue #4)](designs/sampling-params.md) `design:sampling-params`
|
- [采样参数透传设计(issue #4)](designs/sampling-params.md) `design:sampling-params`
|
||||||
|
|
||||||
## finding (22)
|
## finding (22)
|
||||||
|
|
||||||
- [1.3.4 推理契约验证与发布准备](findings/2026-09-09-134-thinking-contracts-validation.md) `finding:2026-09-09-134-thinking-contracts-validation`
|
- [1.3.4 推理契约验证与发布准备](findings/2026-09-09-134-thinking-contracts-validation.md) `finding:2026-09-09-134-thinking-contracts-validation`
|
||||||
- [1.3.5 T2/T3/T4 验收证据:36 列遥测、失败终态、PG 存储兼容与变异矩阵](findings/2026-09-09-135-call-observability-validation.md) `finding:2026-09-09-135-call-observability-validation`
|
- [1.3.5 T2/T3/T4 验收证据:36 列遥测、失败终态、PG 存储兼容与变异矩阵](findings/2026-09-09-135-call-observability-validation.md) `finding:2026-09-09-135-call-observability-validation`
|
||||||
- [1.3.5 发布完成与外部验收:合并后门、registry 产物与页面亲查](findings/2026-09-09-135-release-completion.md) `finding:2026-09-09-135-release-completion`
|
- [1.3.5 发布完成与外部验收:合并后门、registry 产物与页面亲查](findings/2026-09-09-135-release-completion.md) `finding:2026-09-09-135-release-completion`
|
||||||
@@ -77,8 +77,7 @@
|
|||||||
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
|
- [P7 OCR soak 验收: 99.73% 与 13 不变量全 PASS](findings/p7-ocr-soak.md) `finding:p7-ocr-soak`
|
||||||
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
|
- [推理开关与 reasoning_tokens: 供应商实测与业界做法](findings/2026-08-02-thinking-switch-and-reasoning-tokens.md) `finding:2026-08-02-thinking-switch-and-reasoning-tokens`
|
||||||
|
|
||||||
## plan (42)
|
## plan (44)
|
||||||
|
|
||||||
- [1.3.4 推理契约实施计划](plans/2026-09-09-134-thinking-contracts.md) `plan:2026-09-09-134-thinking-contracts`
|
- [1.3.4 推理契约实施计划](plans/2026-09-09-134-thinking-contracts.md) `plan:2026-09-09-134-thinking-contracts`
|
||||||
- [1.3.5 逻辑调用统计与结构化失败诊断实施计划](plans/2026-09-09-135-call-observability.md) `plan:2026-09-09-135-call-observability`
|
- [1.3.5 逻辑调用统计与结构化失败诊断实施计划](plans/2026-09-09-135-call-observability.md) `plan:2026-09-09-135-call-observability`
|
||||||
- [1.3.6 可选调用期限与取消结算修复实施计划](plans/2026-09-10-136-call-deadline.md) `plan:2026-09-10-136-call-deadline`
|
- [1.3.6 可选调用期限与取消结算修复实施计划](plans/2026-09-10-136-call-deadline.md) `plan:2026-09-10-136-call-deadline`
|
||||||
@@ -108,6 +107,7 @@
|
|||||||
- [M2.5 治理韧性实现计划](plans/m25-resilience.md) `plan:m25-resilience`
|
- [M2.5 治理韧性实现计划](plans/m25-resilience.md) `plan:m25-resilience`
|
||||||
- [M3 OCR 实现计划](plans/m3-ocr.md) `plan:m3-ocr`
|
- [M3 OCR 实现计划](plans/m3-ocr.md) `plan:m3-ocr`
|
||||||
- [M4 迁移实现计划(T0-T14)](plans/m4-migration.md) `plan:m4-migration`
|
- [M4 迁移实现计划(T0-T14)](plans/m4-migration.md) `plan:m4-migration`
|
||||||
|
- [OCR 整份文档解析实现计划(#34)](plans/2026-10-21-issue34-ocr-document.md) `plan:2026-10-21-issue34-ocr-document`
|
||||||
- [OpenAI消息保真实现计划](plans/2026-10-01-issue29-openai-fidelity.md) `plan:2026-10-01-issue29-openai-fidelity`
|
- [OpenAI消息保真实现计划](plans/2026-10-01-issue29-openai-fidelity.md) `plan:2026-10-01-issue29-openai-fidelity`
|
||||||
- [plan-issue14-admission-wait-policy](plans/plan-issue14-admission-wait-policy.md) `plan:plan-issue14-admission-wait-policy`
|
- [plan-issue14-admission-wait-policy](plans/plan-issue14-admission-wait-policy.md) `plan:plan-issue14-admission-wait-policy`
|
||||||
- [响应可观测字段扩展实现计划](plans/response-observability-fields.md) `plan:response-observability-fields`
|
- [响应可观测字段扩展实现计划](plans/response-observability-fields.md) `plan:response-observability-fields`
|
||||||
@@ -119,11 +119,12 @@
|
|||||||
- [实现计划: 遥测连接池的资源语义与生命周期(issue #15)](plans/plan-issue15-telemetry-pool-lifecycle.md) `plan:plan-issue15-telemetry-pool-lifecycle`
|
- [实现计划: 遥测连接池的资源语义与生命周期(issue #15)](plans/plan-issue15-telemetry-pool-lifecycle.md) `plan:plan-issue15-telemetry-pool-lifecycle`
|
||||||
- [推理可观测性一等化实现计划(issue #16 + #17,发 1.3.1)](plans/2026-08-25-thinking-observability-plan.md) `plan:2026-08-25-thinking-observability-plan`
|
- [推理可观测性一等化实现计划(issue #16 + #17,发 1.3.1)](plans/2026-08-25-thinking-observability-plan.md) `plan:2026-08-25-thinking-observability-plan`
|
||||||
- [推理开关能力建模与 reasoning_tokens 采集实施计划(issue #5 + #6)](plans/2026-08-02-thinking-capability.md) `plan:2026-08-02-thinking-capability`
|
- [推理开关能力建模与 reasoning_tokens 采集实施计划(issue #5 + #6)](plans/2026-08-02-thinking-capability.md) `plan:2026-08-02-thinking-capability`
|
||||||
|
- [统一完成判定实现计划(#30/#31)](plans/2026-10-14-issue30-31-completion-verdict.md) `plan:2026-10-14-issue30-31-completion-verdict`
|
||||||
- [调用方自定义维度实现计划(issue #11)](plans/issue11-caller-dimensions.md) `plan:issue11-caller-dimensions`
|
- [调用方自定义维度实现计划(issue #11)](plans/issue11-caller-dimensions.md) `plan:issue11-caller-dimensions`
|
||||||
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
|
- [采样参数透传实现计划(issue #4)](plans/sampling-params-plan.md) `plan:sampling-params-plan`
|
||||||
|
|
||||||
## review (6)
|
## review (7)
|
||||||
|
- [#34 OCR 整份文档解析 合并前验收与审查报告](reviews/2026-10-22-issue34-branch-review.md) `review:2026-10-22-issue34-branch-review`
|
||||||
- [2026-09-11-138-branch-review](reviews/2026-09-11-138-branch-review.md) `review:2026-09-11-138-branch-review`
|
- [2026-09-11-138-branch-review](reviews/2026-09-11-138-branch-review.md) `review:2026-09-11-138-branch-review`
|
||||||
- [2026-09-11-138-verifier](reviews/2026-09-11-138-verifier.md) `review:2026-09-11-138-verifier`
|
- [2026-09-11-138-verifier](reviews/2026-09-11-138-verifier.md) `review:2026-09-11-138-verifier`
|
||||||
- [2026-09-11-streaming-stall-design-review](reviews/2026-09-11-streaming-stall-design-review.md) `review:2026-09-11-streaming-stall-design-review`
|
- [2026-09-11-streaming-stall-design-review](reviews/2026-09-11-streaming-stall-design-review.md) `review:2026-09-11-streaming-stall-design-review`
|
||||||
@@ -132,10 +133,8 @@
|
|||||||
- [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review`
|
- [整分支审查: issue #14 熔断等待档](reviews/issue14-branch-review.md) `review:issue14-branch-review`
|
||||||
|
|
||||||
## schema (1)
|
## schema (1)
|
||||||
|
|
||||||
- [表结构: llm_calls(遥测 36 字段)](schemas/llm-calls.md) `schema:llm-calls`
|
- [表结构: llm_calls(遥测 36 字段)](schemas/llm-calls.md) `schema:llm-calls`
|
||||||
|
|
||||||
## metric (2)
|
## metric (2)
|
||||||
|
|
||||||
- [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success`
|
- [OCR 治理调用成功率与错误分类分布](metrics/ocr-call-success.md) `metric:ocr-call-success`
|
||||||
- [每次调用必录覆盖率(含缓存命中/失败/取消)](metrics/call-telemetry-coverage.md) `metric:call-telemetry-coverage`
|
- [每次调用必录覆盖率(含缓存命中/失败/取消)](metrics/call-telemetry-coverage.md) `metric:call-telemetry-coverage`
|
||||||
|
|||||||
@@ -171,3 +171,10 @@
|
|||||||
- [2026-10-01 18:27 UTC] 新增边: review:2026-10-01-issue29-cache-review --informs--> design:2026-10-01-issue29-openai-fidelity-design
|
- [2026-10-01 18:27 UTC] 新增边: review:2026-10-01-issue29-cache-review --informs--> design:2026-10-01-issue29-openai-fidelity-design
|
||||||
- [2026-10-01 18:27 UTC] 重建索引: 120 篇页面
|
- [2026-10-01 18:27 UTC] 重建索引: 120 篇页面
|
||||||
- [2026-10-02 02:23 UTC] 重建索引: 120 篇页面
|
- [2026-10-02 02:23 UTC] 重建索引: 120 篇页面
|
||||||
|
- [2026-10-22 02:35 UTC] 重建索引: 123 篇页面
|
||||||
|
- [2026-10-22 02:45 UTC] 新增边: plan:2026-10-21-issue34-ocr-document --implements--> design:2026-10-21-issue34-ocr-document-design
|
||||||
|
- [2026-10-22 02:45 UTC] 重建索引: 124 篇页面
|
||||||
|
- [2026-10-22 04:02 UTC] 重建索引: 124 篇页面
|
||||||
|
- [2026-10-22 04:23 UTC] 重建索引: 124 篇页面
|
||||||
|
- [2026-10-22 06:15 UTC] 新增边: review:2026-10-22-issue34-branch-review --informs--> plan:2026-10-21-issue34-ocr-document
|
||||||
|
- [2026-10-22 06:15 UTC] 重建索引: 125 篇页面
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ date: 2026-07-20
|
|||||||
|
|
||||||
| 指标 | 确定性阈值/证据 | 实际 live 基线 |
|
| 指标 | 确定性阈值/证据 | 实际 live 基线 |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 三个无推理入口成败行档位 NULL | embed、recognize_text、parse_layout 真实 client→emitter→临时 SQLite,非空失败/成功计数与 NULL 全部成立,契约要求100% | 待首次实际运行,不填伪百分比 |
|
| 三个无推理入口成败行档位 NULL | embed、recognize_text、parse_layout、parse_document 真实 client→emitter→临时 SQLite,非空失败/成功计数与 NULL 全部成立,契约要求100% | 待首次实际运行,不填伪百分比 |
|
||||||
| chat 阳性 | 糖失败 auto、显式意图失败、nearest 成功实际档均精确匹配,要求100%;防恒 NULL 假绿 | 待首次实际运行 |
|
| chat 阳性 | 糖失败 auto、显式意图失败、nearest 成功实际档均精确匹配,要求100%;防恒 NULL 假绿 | 待首次实际运行 |
|
||||||
| 四种行来源 | 真实成功=applied;失败=effective;cache_hit/scope终态=本次请求级 | 排除缓存/失败后才可做实际档分析 |
|
| 四种行来源 | 真实成功=applied;失败=effective;cache_hit/scope终态=本次请求级 | 排除缓存/失败后才可做实际档分析 |
|
||||||
| live 覆盖 | 逐轮 PASS/FAIL/UNCOVERED、计划轮数与缺轮分别统计;必需单元不因 pytest exit 0 自动放行 | 未执行;UNKNOWN/缺轮/skip 不能记PASS |
|
| live 覆盖 | 逐轮 PASS/FAIL/UNCOVERED、计划轮数与缺轮分别统计;必需单元不因 pytest exit 0 自动放行 | 未执行;UNKNOWN/缺轮/skip 不能记PASS |
|
||||||
|
|||||||
@@ -7,11 +7,11 @@ date: 2026-07-21
|
|||||||
|
|
||||||
# OCR 治理调用成功率与错误分类分布
|
# OCR 治理调用成功率与错误分类分布
|
||||||
|
|
||||||
- **数据源**: `llm_calls` 表(复用,无新表;M3 设计 §7)。OCR 行特征: `provider="monkey"`、tokens=0、cost=NULL、ttft/inter=NULL、cache_hit=0;`messages` 为 `<ocr:text|layout image_bytes=N>` 占位(图像 bytes 绝不入库),`response` 为 text 截断 200 字 / `<elements n=X pages=Y>` 摘要。
|
- **数据源**: `llm_calls` 表(复用,无新表;M3 设计 §7)。OCR 行特征: `provider="monkey"`、tokens=0、cost=NULL、ttft/inter=NULL、cache_hit=0;`messages` 为 `<ocr:text|layout|document image_bytes=N>` 占位(图像 bytes 绝不入库),`response` 为 text 截断 200 字 / `<elements n=X pages=Y>` 摘要。
|
||||||
- **埋点位置**(唯一路径,禁止旁路): `src/polygateway/ocr.py` OcrClient 治理循环内经 `TelemetryEmitter.emit_attempt`(单 helper 铁律),每次尝试(成功/失败/取消)必录;写失败 warning 降级不冒泡。
|
- **埋点位置**(唯一路径,禁止旁路): `src/polygateway/ocr.py` OcrClient 治理循环内经 `TelemetryEmitter.emit_attempt`(单 helper 铁律),每次尝试(成功/失败/取消)必录;写失败 warning 降级不冒泡。
|
||||||
- **指标口径**:
|
- **指标口径**:
|
||||||
- 调用成功率 = `error IS NULL` 行数 / 总行数(按 provider="monkey" 过滤,窗口按 run/日);
|
- 调用成功率 = `error IS NULL` 行数 / 总行数(按 provider="monkey" 过滤,窗口按 run/日);
|
||||||
- 错误分类分布 = `error` 非空行按异常类名前缀(Transient/SourceDead/RequestRejected/ResultInvalid/CircuitOpen/AllSourcesExhausted)归组计数;
|
- 错误分类分布 = `error` 非空行按异常类名前缀(Transient/SourceDead/RequestRejected/ResultInvalid/CircuitOpen/AllSourcesExhausted)归组计数;
|
||||||
- 端点维度经 `messages` 占位前缀(`ocr:text` vs `ocr:layout`)区分;
|
- 端点维度经 `messages` 占位前缀(`ocr:text` / `ocr:layout` / `ocr:document`)区分,或直接用 `operation` 列(`recognize_text` / `parse_layout` / `parse_document`);文档行的 `response` 摘要为 `<document elements=N pages=M md_chars=K>`,**不含正文**;
|
||||||
- 延迟 = `latency_ms` 分位数(text 与 layout 分列——layout 含两段 HTTP,显著更慢)。
|
- 延迟 = `latency_ms` 分位数(text、layout、document 分列——后两者含两段 HTTP,显著更慢;document 随页数增长,约 1.5–4 s/页[2026-10-21 实测:2 页 3.5–8 秒、4 页 6.2–6.4 秒;样本极少、无争用,非 SLA],不要与 layout 混算分位数)。
|
||||||
- **评估基线**: 待 P7 OCR soak 首跑后建立(健康双真源池预期成功率 ≥98%,故障池口径阈值在 P7 计划冻结);单调用延迟基线以 integration 实测为初值(2026-07-21 手测: /ocr/text 约 3-5s,/parse 约 5-15s)。
|
- **评估基线**: 待 P7 OCR soak 首跑后建立(健康双真源池预期成功率 ≥98%,故障池口径阈值在 P7 计划冻结);单调用延迟基线以 integration 实测为初值(2026-07-21 手测: /ocr/text 约 3-5s,/parse 约 5-15s)。
|
||||||
|
|||||||
@@ -206,3 +206,15 @@ stack = ExtractionProviderStack(
|
|||||||
## 1.3.10 影响核对(2026-10-14,#30/#31)
|
## 1.3.10 影响核对(2026-10-14,#30/#31)
|
||||||
|
|
||||||
CHS 已把 `ResultInvalidError` 作终态(坏结果≠坏服务,熔断记成功),1.3.10 文本 length/过滤且无正文落该分类与既有语义同向,worker `_TERMINAL` 分支无需改;缺 `[DONE]` 而完整的流从此直接成功。
|
CHS 已把 `ResultInvalidError` 作终态(坏结果≠坏服务,熔断记成功),1.3.10 文本 length/过滤且无正文落该分类与既有语义同向,worker `_TERMINAL` 分支无需改;缺 `[DONE]` 而完整的流从此直接成功。
|
||||||
|
|
||||||
|
## 1.3.11 CHSAnalyzer3 的整份文档消费路径(2026-10-22,#34)
|
||||||
|
|
||||||
|
CHSAnalyzer3 要把几百篇医学论文 PDF(每篇约 8–15 页,并发 ≤2,与医院线共用 MonkeyOCR)整份转成带页码的正文与表格。库侧提供 `OcrClient.parse_document`(设计 `designs/2026-10-21-issue34-ocr-document-design.md`):
|
||||||
|
|
||||||
|
| 事项 | 结论 |
|
||||||
|
| --- | --- |
|
||||||
|
| 入口 | `await ocr.parse_document(pdf_bytes, tenant_id=..., meta=...)`;结果 `elements[]`(`type`/`page_index`/`bbox`/`text`)、`page_sizes`、`markdown`。`parse_layout` 医院线路径**不动** |
|
||||||
|
| 表格 | `type=="table"` 的元素 `text` 是 HTML 表体;**表格脚注、页眉页脚、页码在服务端即被丢弃**(显著性/缩写说明会缺失),库无从补偿——抽取逻辑不要假设能拿到 |
|
||||||
|
| 部署 | 文档线用独立 scope(如 `OCR_DOC`),`MAX_CONCURRENCY=2`、`RETRY__MAX_ATTEMPTS=2`、`CALL_DEADLINE_S ≤ PGW_LEASE_TTL_S`、`TIMEOUT_S` 按最大页数估算;配方见 README「1.3.11 OCR 整份文档解析」。若服务为 sync-only 后端,每实例并发 1 并错峰,避免队头阻塞医院线 |
|
||||||
|
| 坏文件 | 无法识别的字节本地拒绝;截断/伪造 PDF 的已知确定性 500 归 `RequestRejectedError`(`status_code=500`),不重试、不计熔断。**页数过滤、可读性预检归调用方** |
|
||||||
|
| ⚠️ 待确认(H3) | 本文件 §9 记录 CHSAnalyzer 把库侧「服务不可用」一族(`GatewayUnavailableError`:`CircuitOpenError`/`AllSourcesExhausted`)翻译成 `ProviderUnavailableError` → arq 延期重投且不耗业务失败预算。**CHSAnalyzer3 是否沿用该语义未向下游证实**: 若沿用,一份永久损坏的 PDF 若落入 `AllSourcesExhausted(retry_exhausted)`(例如共享分类对未收录的确定性 500 仍归瞬时,见 #35)会被无限延期——请在 CHSAnalyzer3 侧为「同一文档连续 N 次延期」设上限 |
|
||||||
|
|||||||
@@ -0,0 +1,254 @@
|
|||||||
|
---
|
||||||
|
type: plan
|
||||||
|
node_id: plan:2026-10-21-issue34-ocr-document
|
||||||
|
title: OCR 整份文档解析实现计划(#34)
|
||||||
|
date: 2026-10-21
|
||||||
|
---
|
||||||
|
|
||||||
|
# #34 OCR 整份文档解析实现计划
|
||||||
|
|
||||||
|
> 实现已获批准的设计 [2026-10-21-issue34-ocr-document-design](../designs/2026-10-21-issue34-ocr-document-design.md)(下称「设计」,章节号引自它)。
|
||||||
|
> 分支 `fix/issue34-ocr-document`;基线 `25a90eb`(1.3.10);目标 **1.3.11**。**发布(合并/tag/上传 registry/建 Release/关 #34)不在本计划内,等人类逐步指令(设计 H4)。**
|
||||||
|
> 保真校验: 本计划不涉及参考实现迁移,保真校验不适用。
|
||||||
|
|
||||||
|
**目标**: 让 OCR 客户端支持整份 PDF 文档解析,返回带页码/bbox/类型/文字的逐块结果与整篇 Markdown,并修正上传类型标签与确定性坏输入的分类。
|
||||||
|
|
||||||
|
**方案概述**: 新增 `OcrClient.parse_document` 与独立 Protocol `OcrDocumentTransport`;transport 按内容嗅探 PDF 并设置上传标签;结果只从 `_middle.json` + 配对 `.md` 解析(不读 `_content_list.json`);新路径对一组已知的确定性 500 改归「请求被拒」。`parse_layout`/`recognize_text` 的签名与返回类型不动。
|
||||||
|
|
||||||
|
**技术**: Python 3.12、httpx(流式 GET)、`asyncio.to_thread`、标准库 `zipfile`/`json`;**零新依赖**。
|
||||||
|
|
||||||
|
## 0. 全局约束(每个任务都适用)
|
||||||
|
|
||||||
|
| 约束 | 内容 |
|
||||||
|
| --- | --- |
|
||||||
|
| 先红后绿 | 每个任务先写测试并**亲眼看到因目标行为缺失而失败**,红灯输出存 `scratchpad/<任务>-red.log`,再实现到绿。测试结果门要求的证据,最终报告逐条引用 |
|
||||||
|
| 提交门 | `.claude/scripts/hooks/pre-commit-guard.sh` 在提交时跑**全套** `pytest tests/` 并拒绝「本次改动触及的函数圈复杂度 ≥ C」(radon ≥ 11)。故所有新函数须拆小(见 T3 的分解),提交必须调用 `commit` skill、**不带 AI 签名** |
|
||||||
|
| 命令 | 一律 `conda run -n PolyGateway ...`;长命令输出重定向到文件并**直接取退出码**,不接管道(CLAUDE.md §3 执行模式) |
|
||||||
|
| 复杂度 | 新函数的复杂度在实现前只能标「预计」,**每次提交前对改动模块跑 `radon cc <文件> -n C`**,有输出即拆分;不得提前宣称已通过 |
|
||||||
|
| 层次 | `import-linter` 契约:`ports/types/errors` 最内层;`transports/` 只实现端口、不依赖 `middleware/`;每个任务结束跑 `make lint` |
|
||||||
|
| 中文 | 模块/类/方法中文 docstring;注释解释「为什么」;日志用 loguru,禁 `print` |
|
||||||
|
| 不顺手清理 | 只动本计划列出的文件与函数;`parse_layout` 的行为(bbox 整数化退化校验、`page_idx` 回退、无体积上限)**保持原样** |
|
||||||
|
| 基线 | 动手前 OCR 相关单测 257 passed(`test_monkey_ocr/test_ocr_client/test_ports/test_types/test_package`) |
|
||||||
|
|
||||||
|
## 1. 文件结构
|
||||||
|
|
||||||
|
| 文件 | 动作 | 职责 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `src/polygateway/types.py` | 改 | `CallOperation`+`CALL_OPERATIONS` 增 `"parse_document"`;新增 `OcrDocumentElement`/`OcrDocumentResult`/`OcrDocumentTransportResult`;`OcrLayoutElement` docstring 加继承警示 |
|
||||||
|
| `src/polygateway/ports.py` | 改 | 新增 `OcrDocumentPort`、`OcrDocumentTransport`(**不动** `OcrTransport` 成员) |
|
||||||
|
| `src/polygateway/transports/_monkey_document.py` | **新** | 结果包解析纯函数(无 HTTP、无治理):`parse_document_package`;承接并**原样迁入** `_finite_number`/`_parse_page_size`(避免与 `monkey_ocr.py` 循环导入;`monkey_ocr.py` 反向导入它们) |
|
||||||
|
| `src/polygateway/transports/monkey_ocr.py` | 改 | `_sniff_media`/`_upload_part`/`_files`;`_classify_status`+`_translate_http_errors` 增 `rejected_markers`;`MonkeyOcrTransport.parse_document`、流式下载、第一段协议抽共用 |
|
||||||
|
| `src/polygateway/ocr.py` | 改 | `OcrClient.parse_document`;`_OcrKind` 增 `"document"`;私有方法形参 `image`→`payload`;`_invoke`/`_summarize` 分派 |
|
||||||
|
| `src/polygateway/__init__.py` | 改 | 导出 `OcrDocumentElement`/`OcrDocumentResult`(版本号放 T9) |
|
||||||
|
| `tools/check_wiki_alignment.py` | 改 | `_SIGNATURE_TARGETS` 加 `OcrClient.parse_document` |
|
||||||
|
| `research-wiki/schemas/llm-calls.md`、`metrics/ocr-call-success.md`、`metrics/call-telemetry-coverage.md` | 改 | operation 取值与摘要格式 |
|
||||||
|
| `tests/unit/{test_types,test_ports,test_package,test_monkey_ocr,test_ocr_client,test_telemetry}.py` | 改 | 见各任务 |
|
||||||
|
| `research-wiki/{index.md,log.md,graph/edges.json}` | 改(自动生成) | T7 的 `make wiki` 产物,随该任务提交 |
|
||||||
|
| `tests/unit/test_monkey_document.py` | **新** | 结果包解析的契约用例(设计 §5.1 逐行一条) |
|
||||||
|
| `tests/integration/test_monkey_ocr_document_live.py` | **新** | `slow` 活体用例(T8)。**不改**既有 `test_monkey_ocr_live.py`: 它带模块级 `skipif`(依赖图片语料)且用例未标 `slow`,默认套件会收集它(实测 6 项),混入新用例会改变提交门的行为 |
|
||||||
|
| `CHANGELOG.md`、`README.md`、`.env.example`、`pyproject.toml`、`research-wiki/{ARCHITECTURE.md,designs/2026-07-21-m3-ocr-design.md,migrations/chsanalyzer.md}` | 改 | T9 |
|
||||||
|
|
||||||
|
## 2. 跨任务关键接口(执行时以此为准)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# types.py
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentElement(OcrLayoutElement):
|
||||||
|
text: str # 随 type: text/title→纯文本; interline_equation→裸 LaTeX; table→HTML; image→""
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentTransportResult:
|
||||||
|
elements: list[OcrDocumentElement]
|
||||||
|
page_sizes: list[tuple[float, float]]
|
||||||
|
markdown: str
|
||||||
|
raw: dict[str, Any]
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentResult: # 独立定义,不继承 OcrLayoutResult(父类 call_stats 带默认值)
|
||||||
|
elements: list[OcrDocumentElement]
|
||||||
|
page_sizes: list[tuple[float, float]]
|
||||||
|
markdown: str
|
||||||
|
source_name: str
|
||||||
|
usage: Usage
|
||||||
|
latency_ms: int
|
||||||
|
call_id: str
|
||||||
|
raw: dict[str, Any]
|
||||||
|
call_stats: CallStats | None = None
|
||||||
|
|
||||||
|
# ports.py
|
||||||
|
@runtime_checkable
|
||||||
|
class OcrDocumentPort(Protocol):
|
||||||
|
async def parse_document(self, document: bytes) -> OcrDocumentResult: ...
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class OcrDocumentTransport(Protocol):
|
||||||
|
async def parse_document(
|
||||||
|
self, *, document: bytes, source: SourceConfig, call_id: str
|
||||||
|
) -> OcrDocumentTransportResult: ...
|
||||||
|
|
||||||
|
# transports/_monkey_document.py (全部无 I/O,可在线程中运行)
|
||||||
|
MAX_MIDDLE_BYTES = 64 * 1024 * 1024
|
||||||
|
MAX_MARKDOWN_BYTES = 16 * 1024 * 1024
|
||||||
|
def parse_document_package(
|
||||||
|
zip_bytes: bytes,
|
||||||
|
) -> tuple[list[OcrDocumentElement], list[tuple[float, float]], str]: ... # 失败一律 ResultInvalidError
|
||||||
|
|
||||||
|
# transports/monkey_ocr.py
|
||||||
|
_MediaKind = Literal["pdf", "image", "unknown"]
|
||||||
|
def _sniff_media(data: bytes) -> _MediaKind: ...
|
||||||
|
def _upload_part(data: bytes) -> tuple[str, bytes, str]: ...
|
||||||
|
_MAX_PACKAGE_BYTES = 256 * 1024 * 1024
|
||||||
|
_DEFAULT_REJECTED_MARKERS = ( # 小写子串;取设计 §6 实测
|
||||||
|
"failed to open stream", "cannot open empty stream",
|
||||||
|
"cannot save with zero pages", "is no pdf",
|
||||||
|
)
|
||||||
|
def _classify_status(exc, source_name, operation, rejected_markers: tuple[str, ...] = ()): ...
|
||||||
|
|
||||||
|
# ocr.py
|
||||||
|
async def parse_document(
|
||||||
|
self, document: bytes, *, session_id: str | None = None, parent_call_id: str | None = None,
|
||||||
|
tenant_id: str | None = None, meta: Mapping[str, Any] | None = None,
|
||||||
|
call_deadline_s: float | None = None,
|
||||||
|
) -> OcrDocumentResult: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. 任务
|
||||||
|
|
||||||
|
### T1 内核类型与端口 — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `types.py`、`ports.py`、`__init__.py`(仅导出)、`tests/unit/test_types.py`、`tests/unit/test_ports.py`、`tests/unit/test_package.py`(其显式导出名单加两个新名)。
|
||||||
|
- **行为**: 按 §2 增加三个数据类与两个 Protocol;`CallOperation`/`CALL_OPERATIONS` 增 `"parse_document"`;`OcrLayoutElement` docstring 加「追加带默认值字段前须评估子类 `OcrDocumentElement`(无默认字段会在导入期 `TypeError`)」;`OcrDocumentResult` 不继承 `OcrLayoutResult`,docstring 写明原因;`OcrDocumentResult.markdown` 的 docstring 写明「含悬空 `` 引用——库不外传图片」,`OcrDocumentElement` docstring 写明 `text` 随 type 的含义(设计 §4.1)。
|
||||||
|
- **测试(先红)**:
|
||||||
|
- `test_document_element_is_layout_element`: `isinstance(el, OcrLayoutElement)` 为真,且 `OcrDocumentElement(type="t",bbox=(0,0,1,1),page_index=0,text="x") != OcrLayoutElement(type="t",bbox=(0,0,1,1),page_index=0)`(dataclass `__eq__` 要求同类,钉死该已知语义)。
|
||||||
|
- `test_document_types_frozen`: 三个新类型赋值抛 `FrozenInstanceError`。
|
||||||
|
- `test_call_operations_matches_literal`: `get_args(CallOperation) == CALL_OPERATIONS`(该常量此前无任何读者)。
|
||||||
|
- `test_ocr_document_ports_runtime_checkable`: 合格/缺方法的桩各一;`test_ocr_transport_members_unchanged`: `OcrTransport` 的成员集合恰为 `{recognize_text, parse_layout, check_health}`(防止有人把新方法塞进旧 Protocol);`test_ports.py:190-196` 的参数化补 `(OcrDocumentTransport, "parse_document")` 断言无 reasoning 档。
|
||||||
|
- `test_package.py` 已有导出一致性用例须覆盖两个新导出名。
|
||||||
|
- **验证**: `conda run -n PolyGateway pytest tests/unit/test_types.py tests/unit/test_ports.py tests/unit/test_package.py -q --no-cov > out.log 2>&1; echo $?` → `0`;`make lint` 通过。
|
||||||
|
- **提交**: `feat: add OCR document types and ports`。
|
||||||
|
|
||||||
|
### T2 上传类型嗅探 — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `transports/monkey_ocr.py`、`tests/unit/test_monkey_ocr.py`。
|
||||||
|
- **行为**(设计 §3 Q2、§5): `_sniff_media` 判定顺序固定——① 起始为图片魔数(JPEG `\xff\xd8\xff`、PNG `\x89PNG\r\n\x1a\n`、TIFF `II*\x00`/`MM\x00*`、BMP `BM`、GIF `GIF87a`/`GIF89a`、WebP `RIFF????WEBP`)→ `"image"`;② 否则前 1024 字节内含 `%PDF-` → `"pdf"`;③ 否则 `"unknown"`。`_upload_part`: pdf → `("doc.pdf", data, "application/pdf")`,**其余(含 unknown)原样** `("image.jpg", data, "image/jpeg")`。`MonkeyOcrTransport._files` 改为 `{"file": _upload_part(data)}`,`recognize_text`/`parse_layout` 无需改动调用点。
|
||||||
|
- **测试(先红)**: 参数化 `_sniff_media`(纯 PDF、前缀 200 字节垃圾的 PDF、JPEG/PNG/TIFF 两种字节序/BMP/GIF/WebP、**起始为 JPEG 魔数但 1KB 内含 `%PDF-` 字样的图片须判 image**、HTML、乱码、空串);`test_upload_label_per_method`: `recognize_text`/`parse_layout` 对 PDF 字节发出 `filename="doc.pdf"` 与 `Content-Type: application/pdf`,对非 PDF 逐字 `filename="image.jpg"`(现有 `test_multipart_shape` 保持通过)。
|
||||||
|
- **验证**: `pytest tests/unit/test_monkey_ocr.py -q --no-cov` 全绿。
|
||||||
|
- **提交**: `fix: label PDF uploads by content instead of a fixed image name`。
|
||||||
|
|
||||||
|
### T3 结果包解析纯函数 — [ ] 一次提交(本计划最大任务)
|
||||||
|
|
||||||
|
- **文件**: `transports/_monkey_document.py`(新)、`transports/monkey_ocr.py`(仅把 `_finite_number`/`_parse_page_size` 改为从新模块导入,行为不变;新模块**不得**反向导入 `monkey_ocr`)、`tests/unit/test_monkey_document.py`(新)。
|
||||||
|
- **行为**: 严格按设计 §5/§5.1。**异常边界(Codex 审查纠正)**: 广义数据异常转换放在 **`parse_document_package` 最外层**,必须同时覆盖「读取」与「全部结构解析」——`except (ValueError, TypeError, KeyError, OverflowError, RecursionError, zipfile.BadZipFile, zipfile.LargeZipFile)` → `ResultInvalidError("OCR 文档结果包内容无效: …")`(`CancelledError` 不在其内);`_read_members` 内**仅包在 `archive.open/read` 外**的窄 `try` 另接 `RuntimeError`(加密成员)/`NotImplementedError`(未知压缩)/`zlib.error`/`EOFError`。已是 `ResultInvalidError` 的原样透传(先 `except ResultInvalidError: raise`)。
|
||||||
|
- **函数分解(预计均 < C,以提交前 radon 实测为准)**:
|
||||||
|
- `parse_document_package`: 仅编排 `_read_members` → `_parse_pages`,并持有外层异常边界。
|
||||||
|
- `_read_members(zip_bytes) -> tuple[object, str]`: 开 ZIP、取首个 `*_middle.json`、`.md` 按 `去掉 "_middle.json" + ".md"` 精确配对、读取前检查 `ZipInfo.file_size`(middle ≤ `MAX_MIDDLE_BYTES`、md ≤ `MAX_MARKDOWN_BYTES`)、`json.loads`、`utf-8-sig` 解码。
|
||||||
|
- `_parse_pages(payload)`: `pdf_info` 须 list 且**非空**;逐页 `_page_elements`。
|
||||||
|
- `_page_elements(page, position)`: 页须 dict;`_parse_page_size`;`_page_index(page, position)`(独立小函数: 须 int、非 bool、非负且等于 position,缺失同样报错);`para_blocks` 缺失=空页、非 list 报错。
|
||||||
|
- `_block_elements(block, page_index)`: 块须 dict、`type` 须 str 且 `strip()` 非空;`_loose_bbox`;父元素 + 非 `*_body` 嵌套子块各成独立元素(紧随父块、保持顺序)。
|
||||||
|
- 文字提取拆四个小函数: `_lines_text(lines)` → `_line_text(line)` → `_span_text(span)`(`content` 须 str;`inline_equation` 包 `$…$`);`_body_text(block)` 只做分派,表格走 `_table_text(sub_blocks)`(找 `table_body` 子块 → 其 span 取 `html`,无则 `latex`,皆无报错),image 走 `""`。
|
||||||
|
- **测试(先红,`test_monkey_document.py`,合成 builder,不含任何真实文本)**: 设计 §5.1 契约表**逐行一条,参数化列明各分支**:
|
||||||
|
- 页级: 零页;`pdf_info` 缺失/非 list;页非 dict;`page_size` 缺失/非 2 元/含 bool/非正/非有限/巨大整数;`page_idx` 缺失/非 int/bool/负/错位;`para_blocks` 缺失=空页、非 list。
|
||||||
|
- 块级: 块非 dict;`type` 缺失/非 str/空串/**纯空白**;bbox 长度≠4/含非数值/含 bool/非有限/反序(x 与 y 各一)/**零宽与零高均合法**/**越界页面框不被裁剪**;`lines` 非 list/行非 dict/`spans` 非 list/span 非 dict/`content` 非 str/`lines: []`→`""`;table 取 html、取 latex、皆无、无 `table_body` 子块;image(用真实结构的 `image_body`+`image_path`)→`""`;未知类型无 lines 无 blocks→`""` 且 type 保留;嵌套 caption/footnote 成独立元素且**紧随父块**(标「未经实测」);`inline_equation` 包 `$`;多行多 span 拼接。
|
||||||
|
- 包级: `.md` 缺失/非 UTF-8/带 BOM/空串;**多个 `_middle.json` 取首个**;**干扰 `.md`(名称不配对)不被取用**;`discarded_blocks` 不进结果;`need_drop=true` 的页**不失败、不影响结果**;ZIP 里放垃圾 `*_content_list.json` 与 `images/x.jpg`,输出与其无关。
|
||||||
|
- 资源与异常边界: 加密成员、未知压缩方法、CRC 损坏、声明大小超限(middle 与 md 各一)、**`10**400` 整数出现在页结构里(而非仅 JSON 层)**、深嵌套 JSON、span 内嵌套异常类型——全部 `ResultInvalidError`,**不漏出原生异常**。
|
||||||
|
- 回归: 既有 `_parse_middle_json` 对相同输入行为不变(bbox 整数化退化仍拒绝、`page_idx` 非法仍回退枚举)。
|
||||||
|
- docstring 验收: `parse_document_package` docstring 写明与 layout 路径的三处差异(零页拒绝、`page_idx` 严格、bbox 宽松)。
|
||||||
|
- **验证**: `pytest tests/unit/test_monkey_document.py tests/unit/test_monkey_ocr.py -q --no-cov` 全绿;`radon cc src/polygateway/transports/_monkey_document.py src/polygateway/transports/monkey_ocr.py -n C` 无输出。
|
||||||
|
- **提交**: `feat: parse document result packages from middle json only`。
|
||||||
|
|
||||||
|
### T4 确定性 500 窄化(仅新路径可启用) — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `transports/monkey_ocr.py`、`tests/unit/test_monkey_ocr.py`。
|
||||||
|
- **行为**: `_classify_status(exc, source_name, operation, rejected_markers=())`: 当 `status == 500` 且 `rejected_markers` 非空且响应体(小写)同时含 `parsing failed:` 与任一标记 → 返回 `RequestRejectedError`(**保留 `status_code=500`**,其余 ctx 与摘要同既有);否则走原分类。判定拆成独立小函数 `_is_deterministic_failure(body, markers)`。`_translate_http_errors(source_name, operation, *, rejected_markers=())` 透传。`MonkeyOcrTransport.__init__` 增关键字参数 `deterministic_failure_markers: tuple[str, ...] = _DEFAULT_REJECTED_MARKERS`,存为 `self._rejected_markers`。**默认调用(不传 markers)行为字节级不变。**
|
||||||
|
- **测试(先红)**: 四个标记各一(用设计 §1 E5 的真实响应体形态 `{"detail":"Parsing failed: …"}`)→ `RequestRejectedError` 且 `status_code==500`、摘要保留;同标记但**不带** `Parsing failed:` 前缀 → 仍 Transient;`Parsing failed: 500: Model not initialized` → Transient;`Unsupported file type` → Transient;空 markers → 与现状一致(现有 `TestErrorTranslation` 全部通过);构造参数可覆盖标记集合;`429`/`503`/`401` 在传入 markers 时分类不变。
|
||||||
|
- **红灯预期**: 分类断言失败(返回 `TransientError` 而非 `RequestRejectedError`),不是导入/fixture 错误。
|
||||||
|
- **验证**: `pytest tests/unit/test_monkey_ocr.py -q --no-cov` 全绿。
|
||||||
|
- **提交**: `feat: let document calls treat deterministic parse failures as rejected input`。
|
||||||
|
|
||||||
|
### T5 transport.parse_document — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `transports/monkey_ocr.py`、`tests/unit/test_monkey_ocr.py`。
|
||||||
|
- **行为**: 把 `parse_layout` 的「POST `/parse` → 校验 `success is True` → 取 `download_url`」抽为共用私有方法(`parse_layout` 行为与错误消息**逐字不变**,传空 markers);新增 `parse_document(*, document, source, call_id)`,并把下载拆成独立私有方法 `_download_package(client, download_url, source) -> bytes`:
|
||||||
|
1. `_sniff_media(document) == "unknown"` → `RequestRejectedError`(无 `status_code`,`operation="parse"`),**不发 HTTP**。
|
||||||
|
2. POST(传 `self._rejected_markers`)。
|
||||||
|
3. `_download_package`: `client.stream("GET")`;**先处理错误状态**——`resp.is_error` 时 `await resp.aread()` 再 `raise_for_status()`(保持响应体摘要随所有分支、保持 401/429/5xx 分类);**成功状态之后**才检查 `Content-Length` 超 `_MAX_PACKAGE_BYTES` 即拒,并对 `aiter_bytes()` 逐块累计,超限 → `ResultInvalidError`(不得让结果包资源检查改变错误分类)。
|
||||||
|
4. `await asyncio.to_thread(parse_document_package, content)`。
|
||||||
|
5. 返回 `OcrDocumentTransportResult(raw={"success": True, "download_url": …})`。
|
||||||
|
- **测试(先红)**:
|
||||||
|
- 全链路(相对与绝对 `download_url`)、`markdown` 与元素取值;`parse_layout` 既有 `TestParseLayout` 全部通过。
|
||||||
|
- **新入口上传标签矩阵**: PDF/前缀垃圾 PDF → `doc.pdf`;图片 → `image.jpg`;未知字节 → 本地 `RequestRejectedError` 且 `MockTransport` 收到 **0 次请求**。
|
||||||
|
- POST 返回四个标记之一 → `RequestRejectedError(500)`,未知 500 → Transient;`success!=true` → `RequestRejectedError(status_code=200)`;GET 的 401/429/500 响应体摘要保留且分类不变;**错误状态 + 超限 `Content-Length` 仍按 HTTP 分类**(不得变成 `ResultInvalidError`);成功状态下 `Content-Length` 超限与多块累计超限 → `ResultInvalidError`。
|
||||||
|
- **解析期间取消不占事件循环**(写法固定): `monkeypatch` 把 transport 引用的 `parse_document_package` 换成同步桩,桩内 `threading.Event.wait(timeout=5)` 阻塞,进入时用 `loop.call_soon_threadsafe(entered.set)` 通知(**不得**在线程里直接用 `asyncio.Event`);测试 `await entered.wait()`,确认另一个协程仍能运行,然后 `task.cancel()`,在释放线程事件**之前**断言 `CancelledError`;`finally` 释放并等待线程收尾。
|
||||||
|
- **验证**: `pytest tests/unit/test_monkey_ocr.py tests/unit/test_monkey_document.py -q --no-cov` 全绿;radon 同 T3。
|
||||||
|
- **提交**: `feat: add parse_document to the MonkeyOCR transport`。
|
||||||
|
|
||||||
|
### T6 OcrClient.parse_document — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `ocr.py`、`tests/unit/test_ocr_client.py`、`tests/unit/test_telemetry.py`。
|
||||||
|
- **行为**: 按 §2 签名新增公共方法(与 `parse_layout` 对称): 先 `validate_caller_dimensions(origin="parse_document(tenant_id=..., meta=...)")`,再 `_document_transport()` 校验(`isinstance(self._transport, OcrDocumentTransport)`,否则 `TypeError`,**早于 `_CallContext`**)。其余改动:
|
||||||
|
- `_OcrKind` 增 `"document"`;私有方法形参 `image`→`payload`。**旧入口错误文案不变**: `_call` 的校验文案按 kind 取名(`text`/`layout` 仍是「image 必须是 bytes(路径读取/批量拼帧留业务侧,D9)」「image 不能为空」,`document` 为「document 必须是 bytes」「document 不能为空」)。
|
||||||
|
- **四处结果联合类型同步**: `_AttemptOutcome.result`(`ocr.py:91`)、`_invoke` 返回(`:486`)、`_emit` 的 `result`(`:582`)、`_summarize` 入参(`:625`)都加 `OcrDocumentTransportResult`;用于 `isinstance` 的 `OcrDocumentTransport` 与三种 TransportResult 从 `TYPE_CHECKING` 提到运行时导入。
|
||||||
|
- `_invoke` 的 document 分支调 `_document_transport().parse_document(document=payload, ...)`;`_summarize` 改 `isinstance` 三分支(document→`<document elements=N pages=M md_chars=K>`,text→截断正文,layout→原摘要);占位字面量不变 `<ocr:document image_bytes=N>`(`_request_for` 的 f-string 单点)。
|
||||||
|
- 遥测调用点 `_emit` 照旧传 `reasoning_applies=False`(单一 helper 铁律,不复制参数列表)。
|
||||||
|
- **测试(先红)**:
|
||||||
|
- `test_ocr_client.py`: `ScriptedOcrTransport` 增 `parse_document` 与脚本动作 `"document"`(并把现有「非 text 一律 layout」改为三分支);成功路径字段(元素/`markdown`/`usage(0,0)`/`call_stats`/gate 记成功);输入校验(`TypeError`/`ValueError` 与旧文案不变的回归);transport 缺 `parse_document` → `TypeError` 且**无遥测行、无统计**;`call_deadline_s` 非法值/继承值与 `parse_layout` 同行为;换源/退避/`ResultInvalid` 不计窗口(`gate.successes==[("m1", False)]`)、本地拒绝不记成功、`status_code=500` 的 `RequestRejectedError` 记成功不计窗口且不喂选源器;超时 Transient → 计失败 → 重试(文档路径的对照);429 → 照扣重试预算、熔断窗口不变;取消穿透且在飞探针归还。
|
||||||
|
- **无延迟判慢**: 扩展 `ClockAdvancingOcrTransport` 的 `"document"` 动作,source `timeout_s` 设小值,在成功尝试内把 `FakeClock` 推进远超它;断言成功、gate 为 CLOSED、`fails==0`。分别构造两种前置状态: 「CLOSED 但已有失败计数」与「OPEN 冷却后的半开探针」(门与 client 共用同一注入时钟,显式配置 `probe_ttl_s`/租约 TTL,避免把租约接管混进断言)。
|
||||||
|
- **持续慢速 → `CallDeadlineExceeded`**(写法固定,真实事件循环时间): 真实 `MonkeyOcrTransport` + `httpx.MockTransport`,POST 返回合法 `download_url`,GET 返回自定义 `httpx.AsyncByteStream`,每产出一个小块前 `await asyncio.sleep(0.01)`;`call_deadline_s=0.1`;经真实 `OcrClient` 调用,断言 `CallDeadlineExceeded`(不是 `TransientError`)、流已关闭、permit 释放、探针归还、熔断窗口不变,并用计数证明确实已进入多块下载。**局限写进测试 docstring**: MockTransport 不执行 httpcore 的逐次读超时,此用例证明整体期限而非 read timeout。
|
||||||
|
- **真实解析异常贯穿客户端**: 真实 `MonkeyOcrTransport` + MockTransport 返回含 `10**400` 整数 / 加密成员的 ZIP,经真实 `OcrClient`,断言 `ResultInvalidError`、**终态遥测行已写**、半开探针已归还、熔断窗口不变(不能只用桩直接抛预制 `ResultInvalidError`)。
|
||||||
|
- `test_telemetry.py`(其 `_StubOcrTransport` 同步增 `parse_document`): `operation` 参数化加 `parse_document`;文档行 `operation=="parse_document"`、messages 占位、response 摘要含 `md_chars` 且**不含正文**、`tenant_id`/`meta` 落库、记录行 **`reasoning_effort is None`**(`reasoning_applies=False` 是传给 emitter 的参数,落库字段是 `reasoning_effort`)、终态失败行类名前缀;`_summarize` 三类各一条;`text_cap` 对文档占位同样生效。
|
||||||
|
- **验证**: `pytest tests/unit/test_ocr_client.py tests/unit/test_telemetry.py tests/unit/test_ports.py -q --no-cov` 全绿;`make lint`;`radon cc src/polygateway/ocr.py -n C` 无新增输出。
|
||||||
|
- **提交**: `feat: add OcrClient.parse_document with full governance`。
|
||||||
|
|
||||||
|
### T7 注册面与对齐工具 — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `tools/check_wiki_alignment.py`、`research-wiki/schemas/llm-calls.md`(operation 取值,第 35 行附近)、`research-wiki/metrics/ocr-call-success.md`、`research-wiki/metrics/call-telemetry-coverage.md`、`make wiki` 重建的 `research-wiki/{index.md,log.md,graph/edges.json}`。
|
||||||
|
- **行为**: `_SIGNATURE_TARGETS` 加 `("OcrClient.parse_document", OcrClient.parse_document)`;三份 wiki 页补 `parse_document`。
|
||||||
|
- **红绿证据**: 红 = 改动前执行 `conda run -n PolyGateway python -c "import runpy; ns = runpy.run_path('tools/check_wiki_alignment.py', run_name='x'); assert any(n == 'OcrClient.parse_document' for n, _ in ns['_SIGNATURE_TARGETS'])"` 以 `AssertionError` 退出非零;绿 = 改动后同命令退出 0。`make wiki-check` 若依赖外部 Gitea Wiki 仓库而不可达,如实记录「不可达,未执行」,**不编造通过**。
|
||||||
|
- **提交**: `docs: register parse_document in telemetry schema and alignment targets`。
|
||||||
|
|
||||||
|
### T8 活体集成用例(`slow`,opt-in) — [ ] 一次提交
|
||||||
|
|
||||||
|
- **文件**: `tests/integration/test_monkey_ocr_document_live.py`(**新文件**,模块级 `pytestmark = pytest.mark.slow`,**不**带图片语料的 `skipif`)。
|
||||||
|
- **行为**: 测试内手写最小合法 PDF 构造函数 `_minimal_pdf(text)`(5 个对象、**计算** xref 偏移、大号 Helvetica 文字,无第三方内容、无新依赖)。统一放在 `class TestLiveDocument`,三条用例,串行、单份小文档(服务与医院线共用,不压测):
|
||||||
|
1. `test_parse_document_returns_pages_and_text`: PDF 文字取 `POLYGATEWAY SYNTHETIC SAMPLE`;断言 ≥1 页、`page_sizes[0]` 均为正、`"SYNTHETIC"` 出现在 `markdown.upper()` 与某个元素 `text.upper()` 中(**正向断言**,空串不能通过)。
|
||||||
|
2. `test_recognize_text_on_pdf_is_not_the_hallucination`: 同一 PDF 的 `recognize_text`: `text.strip()` 非空且 `"single character"` 不在其中,并含 `"SYNTHETIC"`。
|
||||||
|
3. `test_corrupt_documents_do_not_open_the_breaker`: 以阈值 2 的环境,**连续两份**截断但保留 `%PDF-` 头的 PDF → 均 `RequestRejectedError`;随后有效 PDF 仍成功(若坏文档被误计失败,两次即达阈值,第三次会被熔断拒绝)。
|
||||||
|
- **红绿证据**: 本任务是**活体验收**,不存在「先红」;诚实记录为「无行为红灯,证据 = 单元层 T2/T4/T5 的红绿 + 本任务的活体输出」。手动命令选中全部三条: `pytest tests/integration/test_monkey_ocr_document_live.py -m slow -q --no-cov`,输出存 `tests/outputs/issue34/`(不提交);服务不可达时如实记录,**不 skip 掩盖**。默认套件下 `pytest tests/integration/test_monkey_ocr_document_live.py -q --no-cov` 应显示 3 deselected。
|
||||||
|
- **提交**: `test: add live PDF document parsing checks`。
|
||||||
|
|
||||||
|
### T9 文档与版本 — [ ] 一次提交(bump 不得裸发,CLAUDE.md §4.4.1)
|
||||||
|
|
||||||
|
- **性质**: 文档/版本任务,无行为红灯;证据 = `test_package` 的版本一致性用例 + 对版本串与方法名的 `grep` 输出。
|
||||||
|
- **文件与内容**:
|
||||||
|
- `pyproject.toml` 与 `src/polygateway/__init__.py` 的版本 → `1.3.11`。
|
||||||
|
- `CHANGELOG.md`: 新增「未发布」段(发布时定版): 新增公共面(`parse_document`/`OcrDocumentElement`/`OcrDocumentResult`/`OcrDocumentPort`/`OcrDocumentTransport`/`CallOperation "parse_document"`);**行为变更**: PDF 上传改标 `doc.pdf`(`recognize_text` 修复静默错误、`parse_layout(PDF)` 上传分支变更并写明已测[文本型、位图型]与未测[旋转页、批注页]范围);新路径确定性坏输入归「请求被拒」;**已知限制**: 服务端丢弃脚注/页眉页脚/页码、`markdown` 含悬空 `images/` 引用、多行/多 span/`inline_equation`/嵌套 caption 路径未经实测、`parse_document` 不接受魔数表之外的图片格式;关联 #34,后续 #35/#36。
|
||||||
|
- `README.md`: OCR 小节加 `parse_document` 用法与能力表一行、方法数、文档线配置示例;**设计 §7 的运维要点逐条落字**: ① 文档线独立 scope、`MAX_CONCURRENCY=2`、`RETRY__MAX_ATTEMPTS=2`、`TIMEOUT_S ≥ 最大页数×1.5s×余量(争用取 5 倍以上)`、`OCR_DOC__CALL_DEADLINE_S ≤ PGW_LEASE_TTL_S`(`timeout_s` 不是总耗时上界);② 重试放大(GET 失败会重新 POST 整份文档重算);③ 无页数预检,过滤归调用方;④ 服务端磁盘产物累积,需告知服务运维;⑤ 多 worker 共享并发上限需 `PGW_LIMITER_BACKEND=redis` 且同 scope;⑥ sync-only 后端的排队与对医院线的队头阻塞、须向运维确认 backend(启动日志 `async-capable`/`sync-only`);⑦ 服务丢弃脚注等;⑧ 租约守卫只比 1× `timeout_s` 的缺口与「守卫改造另行立项」。**安装命令的版本约束逐项核对**(CLAUDE.md §4.4.1 第 1 步)。
|
||||||
|
- `.env.example`: OCR 段加文档线独立 scope 示例(`OCR_DOC__MONKEY__1__*`、`OCR_DOC__CALL_DEADLINE_S` 等,键名以 `config.py` 为准)。
|
||||||
|
- `research-wiki/ARCHITECTURE.md`: §5.1 类型清单、§7.10 端口表加 `OcrDocumentPort`、D9 注记、**:597 的 operation 枚举加 `parse_document`、:433 的「四个公共方法」补新增入口**;`designs/2026-07-21-m3-ocr-design.md` §4 加勘误(「服务按内容处理」只验证图片,指向本设计 E3/E4/E10);`migrations/chsanalyzer.md` 加 CHSAnalyzer3 新消费路径与 H3 待确认项。
|
||||||
|
- `research-wiki/ROADMAP.md` 若有 OCR 状态行则同步。
|
||||||
|
- **提交**: `docs: document parse_document and bump version to 1.3.11`。
|
||||||
|
|
||||||
|
### T10 验证与审查(合并前硬门)— [ ] 验收任务,无独立红灯;审查修复另提
|
||||||
|
|
||||||
|
1. `make ci`(只读验证: `ruff format --check` + `ruff check` + `lint-imports` + 全套件)——输出重定向文件取退出码,**不接管道**。
|
||||||
|
2. 覆盖率: 新增/改动模块行覆盖率 ≥ 80%,未覆盖行逐一说明;对改动的三个模块运行 radon,**以实测为准**报告。
|
||||||
|
3. 派**全新上下文 verifier subagent**(`verification-before-completion`): 逐条对照设计 §4-§10 与本计划验收标准,只读核对测试证据(红灯日志存在且失败原因正确);任何声明须对应本会话工具输出。
|
||||||
|
4. `requesting-code-review` 整分支审查(含 Codex `--fresh --wait`);逐条核验后修订,不盲从。
|
||||||
|
5. 活体: T8 手动跑一次的输出;并用本会话在 `/tmp/claude-1000/-home-iomgaa-Projects-PolyGateway/a9dd153e-445b-42f7-9808-4ac914c3f176/scratchpad/ocr/` 里**本地存证**的样本(`x/`、`r/`、`t/`、`scanA/`,第三方内容,**不入库**)对 `parse_document_package` 做离线冒烟: 先检查目录存在,核对 `elements` 数量 = middle 的 `para_blocks` 总数(29/89/30/29)、`text` 与 `content_list` 反转义后一致、`markdown` 等于 `.md` 原文;**样本不存在则在报告中列为未验证项**,不得声称已验证。
|
||||||
|
6. 最终报告 ≤300 行,写入 `research-wiki/reviews/`;向人类汇报**未验证项**(多行/多 span、嵌套 caption、旋转/批注页差分、线上 backend、CHSAnalyzer3 延期语义、真实 httpcore read-timeout 行为)。
|
||||||
|
7. **停在此处,等待人类的发布指令**(合并 main、打 tag、构建、上传、验证已发布、建 Release、关闭 #34 逐步执行,CLAUDE.md §4.4.1)。
|
||||||
|
|
||||||
|
## 4. 自审清单
|
||||||
|
|
||||||
|
| 检查 | 结论 |
|
||||||
|
| --- | --- |
|
||||||
|
| 设计每条决策有对应任务 | Q1→T1/T6;Q2→T2/T5;Q3→T3;Q4→T4/T5;Q5 与 §7 运维要点→T6(测试)/T9(逐条落字);§5.1 契约→T3(参数化逐分支);§6 分类表→T4-T6;§8 旧行为→T2/T5/T3 回归断言;H5/H6 变更说明→T9 CHANGELOG |
|
||||||
|
| 跨任务签名一致 | `parse_document_package` 返回三元组(T3)被 T5 `to_thread` 消费;`rejected_markers`(T4)被 T5 透传;`_document_transport()`(T6)是唯一 isinstance 点;四处结果联合类型(T6)已逐处列出 |
|
||||||
|
| 无占位语 | 无 TBD/TODO;关键接口已在 §2 给出实际签名;时间/并发类测试的写法已固定 |
|
||||||
|
| 复杂度 | 只能「预计」;T3 已拆到 `_line_text/_span_text/_table_text/_page_index`;每次提交前 radon 实测 |
|
||||||
|
| 证据完整性 | 每个行为任务写明红灯预期;T7 有可执行红绿命令;T8-T10 如实声明无行为红灯 |
|
||||||
|
| 范围 | 不含 #35/#36 的改动;不含发布动作;不改 `parse_layout` 的校验口径与旧入口文案;不动既有 `test_monkey_ocr_live.py` |
|
||||||
|
|
||||||
|
## 5. 计划审查记录
|
||||||
|
|
||||||
|
Codex 一轮只读审查,结论「修订后进入执行」。逐条核验并全部采纳(第 1 条为我方计划自身的缺陷): ① T3 异常边界放错位置 → 移到最外层(已验证 `_parse_pages` 在 `_read_members` 之后执行);② T8 与既有集成测试文件冲突 → 改新文件(已验证默认收集 6 项旧用例);③ 三类时间/取消测试写法 → 已固定;④ T5 顺序 → 先错误状态后体积;⑤ T6 遥测字段名与联合类型 → 订正(已验证落库字段是 `reasoning_effort`);⑥ 覆盖缺口 → 补入 T3/T5/T9;⑦ 红绿证据 → 逐任务声明;⑧ T8 断言强度与 T10 样本前置 → 订正;⑨ 复杂度表述 → 改「预计」;⑩ 文件清单/文案兼容 → 订正。
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
---
|
||||||
|
type: review
|
||||||
|
node_id: review:2026-10-22-issue34-branch-review
|
||||||
|
title: "#34 OCR 整份文档解析 合并前验收与审查报告"
|
||||||
|
date: 2026-10-22
|
||||||
|
severity: approved
|
||||||
|
---
|
||||||
|
|
||||||
|
# #34 OCR 整份文档解析 合并前验收与审查报告
|
||||||
|
|
||||||
|
- 分支 `fix/issue34-ocr-document`,基线 `25a90eb`(1.3.10),Head `cb1e08f`(17 个提交,目标 1.3.11;本报告随后追加一个文档提交)。
|
||||||
|
- 设计 `designs/2026-10-21-issue34-ocr-document-design.md`(已获人类批准:H1/H5/H6 批准,H2 授权另开 #35/#36,H3 待下游确认,H4 发布逐步等指令);计划 `plans/2026-10-21-issue34-ocr-document.md`。
|
||||||
|
- **结论**: 实现与验证完成,Critical 0、Important 已全部修复。**发布(合并/tag/上传/Release/关 #34)未执行,等人类指令。**
|
||||||
|
|
||||||
|
## 1. 需求覆盖(issue #34 → 实现)
|
||||||
|
|
||||||
|
| issue 期望 | 实现 | 证据 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 上传按类型设 filename/Content-Type | 库按**内容**判定(H5 有意替换「调用方声明」): 图片魔数优先 → 前 1KB 内 `%PDF-` → `doc.pdf`,其余沿用 `image.jpg`;三个方法共用 | `test_monkey_ocr.py::TestSniffMedia`、`TestUploadLabelPerMethod`、`TestParseDocument::test_upload_label_matrix_for_the_document_entry`;活体 `recognize_text(PDF)` 返回真实文本而非幻觉串 |
|
||||||
|
| 逐块 类型/页码/bbox/文字,服务阅读序 | `OcrDocumentElement`(继承 `OcrLayoutElement` + `text`);只读 `_middle.json`;表格为 HTML | `test_monkey_document.py`(§5.1 契约逐行);4 份真实包离线冒烟 29/89/30/29 块全部与 middle/content_list(反转义)一致 |
|
||||||
|
| 整篇 Markdown | `OcrDocumentResult.markdown`,按名精确配对 `.md` | 同上;真实包 `markdown == .md` 原文 |
|
||||||
|
| 长文档超时与熔断确认 | 库内无延迟型判定(grep 已证);`timeout_s` 非总耗时上界,硬上限 = `call_deadline_s ≤ lease_ttl_s`;文档线独立 scope 配方 | `TestNoLatencyJudgement`(含已有失败计数/半开探针两种前置状态)、持续慢速下载 → `CallDeadlineExceeded`、README「运维须知」 |
|
||||||
|
| `parse_layout` 不动 | 签名/返回类型/错误文案不变;仅上传分支(H6)与共用第一段协议新增 `download_url` 校验 | 独立审查对 5501 个输入与 2486 个场景做基线差分,仅 `%PDF-` 字样载荷的上传标签有差异(已登记) |
|
||||||
|
|
||||||
|
## 2. 本会话证据(命令 → 结果)
|
||||||
|
|
||||||
|
| 声明 | 命令 | 结果 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 全套通过 | `pytest tests/ -q --cov=src/polygateway`(退出码取自文件,无管道) | 退出 0;**2239 passed, 23 skipped, 113 deselected(slow)**;总覆盖 96% |
|
||||||
|
| 格式/lint/依赖契约 | `make ci` | 退出 0;108 files formatted、ruff All checks passed、`Contracts: 1 kept, 0 broken` |
|
||||||
|
| 复杂度 | `radon cc <ocr.py, _monkey_document.py, monkey_ocr.py> -n C -s` | 无 C 及以上(仅 `openai_compat.py::complete` 为未触及的既有 C) |
|
||||||
|
| 活体 | `pytest tests/integration/test_monkey_ocr_document_live.py -m slow` | 3 passed(8.85s);默认套件 3 deselected |
|
||||||
|
| 真实多页 PDF 端到端 | `OcrClient.from_env("OCR_DOC")` 调 `rich.pdf`(4 页)/`tbl.pdf`(2 页含表) | 89/30 个元素,与直连存证一致;表格 HTML 取出;各 1 次尝试 |
|
||||||
|
| 红→绿 | 每任务 `scratchpad/T*-red.log` → `T*-green.log` | T1–T6 红灯原因均为目标行为缺失(导入失败或 `AttributeError: no attribute 'parse_document'`) |
|
||||||
|
| 测试非空转 | 变异检查(还原/改坏实现 → 测试转红 → 恢复) | 本会话十余类变异全部被抓到: 异常边界放错位置(47 个失败)、`status==500`/前缀判据、同步解析、体积检查先于错误状态、`_summarize`、早期装配校验、整包上限、`aread`/有界读取、`reasoning_applies`、探针归还、`download_url` 校验、`isinstance` 分派;独立审查另做 86 个变异、0 存活 |
|
||||||
|
|
||||||
|
## 3. 评审轮次与处置
|
||||||
|
|
||||||
|
| 轮次 | 审查者 | 结论 | 处置 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 设计论证 | 独立 Plan subagent(含本机 MonkeyOCR 源码核对) | 弃「content_list 对位」改 middle-only 等 | 全部采纳(设计 §12) |
|
||||||
|
| 设计审查 | Codex(只读) | 否→修订后进入;9 条全部成立 | 全部修订(设计「审查记录」) |
|
||||||
|
| 计划审查 | Codex(只读) | 修订后进入执行;10 条全部成立,第 1 条为我方计划缺陷 | 全部修订(计划 §5) |
|
||||||
|
| 合并前独立验证 | 全新上下文 verifier | 无 Critical;Important 4 条 | I-1 异常漏出、I-2 整包上限无测试、I-3 业务词汇、I-4 三处空转测试——**全部修复**并用变异证明 |
|
||||||
|
| **整分支 Codex 审查** | Codex | **未完成**: 沙箱报 usage limit,只读了设计/计划/部分代码即终止,无任何分级结论 | **以全新上下文的独立 Claude 对抗式审查替代**(见下行);Codex 额度恢复后可补跑,不影响本报告其余结论 |
|
||||||
|
| 对抗式代码审查 | 全新上下文 Claude(自写模糊/变异脚本,假服务压测) | 建议合并;Critical 0,Important 3 | 见 §4 |
|
||||||
|
|
||||||
|
## 4. 对抗式审查发现与处置
|
||||||
|
|
||||||
|
| # | 发现 | 核验 | 处置 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| I-1 | `_summarize` 由 `hasattr` 改 `isinstance`,鸭子类型 transport 结果以 `AttributeError` 漏出 | 本机复现(`SimpleNamespace(text=...)`) | 改回**结构分派**(有 `markdown` → 文档;有 `text` → 文本;其余版面),加回归测试 |
|
||||||
|
| I-2 | 文档路径的错误体 `aread()` 无上限(600 MiB 的 500 使 RSS 涨到 1.8 GiB) | 属实(基线非流式 `get` 同样,但新代码声称有界) | 错误体只读前 64 KiB 并复制成已缓冲响应;加测试证明只消费上限那么多 |
|
||||||
|
| I-3 | 畸形 `download_url`(控制字符/坏方括号/越界端口)让 `httpx.InvalidURL` 漏出;探针不归还 | 本机复现;`parse_layout` 基线同样 | 在共用的 `_post_parse` 一处校验,归瞬时错误、不发第二段请求;`parse_layout` 顺带受益(CHANGELOG 登记);**不限制主机**(基线接受绝对地址) |
|
||||||
|
| M-1 | 嗅探副作用: 前 1KB 含 `%PDF-` 字样的非 PDF 文本被改标;`BM` 两字节魔数太弱 | 属实 | BMP 加 DIB 头大小校验;CHANGELOG/README 登记副作用 |
|
||||||
|
| M-2 | 字节上限≠内存上限(JSON 解析后可达 23 倍;成员条数无上限) | 属实,仅服务异常时成立 | README 改为诚实口径;不实现 EOCD 预读(真实样本约 72 KB/页) |
|
||||||
|
| M-3 | `to_thread` 不是「立即生效」: GIL 下事件循环仍有停顿,取消后线程自然收尾 | 属实 | 文档改「基本不被占住」并给实测数 |
|
||||||
|
| M-4 | `parse_layout` 的 ZIP 路径仍会漏原生异常(基线即有) | 属实,按设计保持原样 | **后续项**(见 §6) |
|
||||||
|
| M-5/6/7 | 设计文档结局表与实现不一致;类 docstring 过时;测试注释含业务词汇 | 属实 | 全部修正 |
|
||||||
|
| M-8 | `page_idx` 必须等于页序,服务若跳页则整份失败 | 设计取舍 | 保留;列入未验证项 |
|
||||||
|
|
||||||
|
## 5. 没能验证的部分(如实登记)
|
||||||
|
|
||||||
|
- **线上服务**: 旋转页/带批注页/加密 PDF 的 `doc.pdf` vs `image.jpg` 差分;带密码 PDF 的服务端报错文案(若不在标记集合,走瞬时重试并计熔断);线上 backend 是否 sync-only(队头阻塞);真实 httpcore 逐次读超时行为(MockTransport 不执行它);`need_drop` 恒为 False 的线上证实。
|
||||||
|
- **样本局限**: 多行/多 span 文本块、`inline_equation` span、嵌套 caption/footnote 三条路径无真实样本,仅由构造 fixture 测试(已在 CHANGELOG/README 标注「未经实测」)。
|
||||||
|
- **未执行**: `make wiki-check`(需外部 Gitea Wiki 克隆路径,站点自 2026-08-02 下线);Redis 限流/熔断后端与 Postgres 遥测对新入口的组合(无对应环境新增用例,既有套件中的后端用例通过);整分支 Codex 审查(额度)。
|
||||||
|
- **下游**: H3(CHSAnalyzer3 是否把「服务不可用」一族按延期重投处理)未向下游证实。
|
||||||
|
|
||||||
|
## 6. 后续项与待人类指令
|
||||||
|
|
||||||
|
| 项 | 状态 |
|
||||||
|
| --- | --- |
|
||||||
|
| #35 共享 `_classify_status` 对确定性坏输入 500 仍归瞬时 | 已开 |
|
||||||
|
| #36 `recognize_text` 不校验 `success` | 已开 |
|
||||||
|
| `parse_layout` ZIP 路径的原生异常漏出(OverflowError/NotImplementedError/RuntimeError)与 `download_url` 主机限制 | 人类 2026-10-22 批准新开,已开 #37 |
|
||||||
|
| 发布: 合并 main → push → tag `v1.3.11` → 构建 → 上传 registry → 验证已发布 → 建 Release/挂仓库 → 关闭 #34 | 人类 2026-10-22 下令发布,按 CLAUDE.md §4.4.1 逐步执行;CHANGELOG 定版日期 2026-10-22 |
|
||||||
@@ -32,7 +32,7 @@ date: 2026-07-20
|
|||||||
| thinking_observation | TEXT | 本次推理是否真的发生的三态裁定(2026-08-25,issue #16/#17);`observed` / `absent` / `unknown`。见下方口径 |
|
| thinking_observation | TEXT | 本次推理是否真的发生的三态裁定(2026-08-25,issue #16/#17);`observed` / `absent` / `unknown`。见下方口径 |
|
||||||
| reasoning_effort | TEXT | 按四种行来源记录的推理意图/实际编码档位(2026-09-09 澄清,issue #20/#26);八档 `Effort` 字面量之一,NULL = 调用方未表态(与 `none`「明确要求不推理」不可混同)。见下方口径 |
|
| reasoning_effort | TEXT | 按四种行来源记录的推理意图/实际编码档位(2026-09-09 澄清,issue #20/#26);八档 `Effort` 字面量之一,NULL = 调用方未表态(与 `none`「明确要求不推理」不可混同)。见下方口径 |
|
||||||
| scope | TEXT | 池名(2026-09-09,issue #19/#23);Emitter 构造期注入,三类行都带,**不拿 `source_name` 顶替** |
|
| scope | TEXT | 池名(2026-09-09,issue #19/#23);Emitter 构造期注入,三类行都带,**不拿 `source_name` 顶替** |
|
||||||
| operation | TEXT | `chat` / `embed` / `recognize_text` / `parse_layout`,由调用点给定;与 `PolyGatewayError.operation` 是两个语义,链路上不得读后者填本列 |
|
| operation | TEXT | `chat` / `embed` / `recognize_text` / `parse_layout` / `parse_document`(2026-10-22,#34),由调用点给定;与 `PolyGatewayError.operation` 是两个语义,链路上不得读后者填本列 |
|
||||||
| logical_call_id | TEXT | 一次逻辑调用一个 ID(重试/换源/重问/分批共享);上下文缺席落 NULL,**不造 ID** |
|
| logical_call_id | TEXT | 一次逻辑调用一个 ID(重试/换源/重问/分批共享);上下文缺席落 NULL,**不造 ID** |
|
||||||
| event_kind | TEXT | 三态 `attempt` / `cache_hit` / `terminal_failure`——三类行的唯一机械判据。见下方口径 |
|
| event_kind | TEXT | 三态 `attempt` / `cache_hit` / `terminal_failure`——三类行的唯一机械判据。见下方口径 |
|
||||||
| http_status_code | INTEGER | 失败 attempt 行的实收状态码(中转改写过就记改写后的,**不猜回原值**);成功行与终态行恒 NULL,且 200 也可能是失败行(MonkeyOCR `success != true`) |
|
| http_status_code | INTEGER | 失败 attempt 行的实收状态码(中转改写过就记改写后的,**不猜回原值**);成功行与终态行恒 NULL,且 200 也可能是失败行(MonkeyOCR `success != true`) |
|
||||||
|
|||||||
@@ -44,6 +44,8 @@ from polygateway.types import (
|
|||||||
Effort,
|
Effort,
|
||||||
EmbeddingResponse,
|
EmbeddingResponse,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
|
OcrDocumentElement,
|
||||||
|
OcrDocumentResult,
|
||||||
OcrLayoutElement,
|
OcrLayoutElement,
|
||||||
OcrLayoutResult,
|
OcrLayoutResult,
|
||||||
OcrTextResult,
|
OcrTextResult,
|
||||||
@@ -55,7 +57,7 @@ from polygateway.types import (
|
|||||||
ThinkingObservation,
|
ThinkingObservation,
|
||||||
)
|
)
|
||||||
|
|
||||||
__version__ = "1.3.10"
|
__version__ = "1.3.11"
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"DEFAULT_PROFILES",
|
"DEFAULT_PROFILES",
|
||||||
@@ -78,6 +80,8 @@ __all__ = [
|
|||||||
"OpenAIChatCompletion",
|
"OpenAIChatCompletion",
|
||||||
"ModelPrice",
|
"ModelPrice",
|
||||||
"OcrClient",
|
"OcrClient",
|
||||||
|
"OcrDocumentElement",
|
||||||
|
"OcrDocumentResult",
|
||||||
"OcrLayoutElement",
|
"OcrLayoutElement",
|
||||||
"OcrLayoutResult",
|
"OcrLayoutResult",
|
||||||
"OcrSettings",
|
"OcrSettings",
|
||||||
|
|||||||
+123
-36
@@ -39,13 +39,17 @@ from polygateway.middleware.breaker import BreakerGate
|
|||||||
from polygateway.middleware.ratelimit import QuotaGate
|
from polygateway.middleware.ratelimit import QuotaGate
|
||||||
from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
|
from polygateway.middleware.retry import StallClock, _failure_reason, backoff_delay
|
||||||
from polygateway.middleware.telemetry import TelemetryEmitter, emit_terminal_once
|
from polygateway.middleware.telemetry import TelemetryEmitter, emit_terminal_once
|
||||||
from polygateway.ports import OutcomeAwareSelector
|
from polygateway.ports import OcrDocumentTransport, OutcomeAwareSelector
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
CallStats,
|
CallStats,
|
||||||
ChatRequest,
|
ChatRequest,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
|
OcrDocumentResult,
|
||||||
|
OcrDocumentTransportResult,
|
||||||
OcrLayoutResult,
|
OcrLayoutResult,
|
||||||
|
OcrLayoutTransportResult,
|
||||||
OcrTextResult,
|
OcrTextResult,
|
||||||
|
OcrTextTransportResult,
|
||||||
TelemetryStatus,
|
TelemetryStatus,
|
||||||
Usage,
|
Usage,
|
||||||
_CallContext,
|
_CallContext,
|
||||||
@@ -69,15 +73,18 @@ if TYPE_CHECKING:
|
|||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
BackpressurePolicy,
|
BackpressurePolicy,
|
||||||
CallOperation,
|
CallOperation,
|
||||||
OcrLayoutTransportResult,
|
|
||||||
OcrTextTransportResult,
|
|
||||||
RetryPolicy,
|
RetryPolicy,
|
||||||
SourceConfig,
|
SourceConfig,
|
||||||
)
|
)
|
||||||
|
|
||||||
_RESPONSE_TEXT_CAP = 200 # 遥测行 text 截断长度(与 embedding 口径一致)
|
_RESPONSE_TEXT_CAP = 200 # 遥测行 text 截断长度(与 embedding 口径一致)
|
||||||
|
|
||||||
_OcrKind = Literal["text", "layout"]
|
_OcrKind = Literal["text", "layout", "document"]
|
||||||
|
|
||||||
|
# 输入校验文案里的名字。旧两个入口(text/layout)的报错文本保持不变,文档入口自称 document
|
||||||
|
_PAYLOAD_NAME: dict[_OcrKind, str] = {"text": "image", "layout": "image", "document": "document"}
|
||||||
|
|
||||||
|
_TransportResult = OcrTextTransportResult | OcrLayoutTransportResult | OcrDocumentTransportResult
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -88,14 +95,14 @@ class _FailedAttempt:
|
|||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class _AttemptOutcome:
|
class _AttemptOutcome:
|
||||||
result: OcrTextTransportResult | OcrLayoutTransportResult
|
result: _TransportResult
|
||||||
source: SourceConfig
|
source: SourceConfig
|
||||||
call_id: str
|
call_id: str
|
||||||
latency_ms: int
|
latency_ms: int
|
||||||
|
|
||||||
|
|
||||||
class OcrClient:
|
class OcrClient:
|
||||||
"""治理化 OCR 入口: 同时实现 OcrTextPort 与 OcrLayoutPort(D9 端口族)。
|
"""治理化 OCR 入口: 同时实现 OcrTextPort、OcrLayoutPort 与 OcrDocumentPort(D9 端口族;文档入口见 #34)。
|
||||||
|
|
||||||
两端点打同一服务实例池,共享同一 scope 的限流/熔断/选源状态;
|
两端点打同一服务实例池,共享同一 scope 的限流/熔断/选源状态;
|
||||||
与其他 scope 共享后端实例即共享全局闸(显式传入,禁止隐式全局)。
|
与其他 scope 共享后端实例即共享全局闸(显式传入,禁止隐式全局)。
|
||||||
@@ -254,6 +261,53 @@ class OcrClient:
|
|||||||
call_stats=call_stats,
|
call_stats=call_stats,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def parse_document(
|
||||||
|
self,
|
||||||
|
document: bytes,
|
||||||
|
*,
|
||||||
|
session_id: str | None = None,
|
||||||
|
parent_call_id: str | None = None,
|
||||||
|
tenant_id: str | None = None,
|
||||||
|
meta: Mapping[str, Any] | None = None,
|
||||||
|
call_deadline_s: float | None = None,
|
||||||
|
) -> OcrDocumentResult:
|
||||||
|
"""一次治理整份文档解析(/parse → ZIP);PDF 或图片,逐块带类型/页码/bbox/文字。
|
||||||
|
|
||||||
|
上传类型由 transport 按内容判定,调用方无需也无法声明。结果的 `markdown` 是服务
|
||||||
|
生成的整篇原文,含悬空的图片引用;服务端会丢弃脚注、页眉页脚与页码。
|
||||||
|
|
||||||
|
`tenant_id` 与 `meta` 是调用方自定义维度,只进遥测(issue #11)。
|
||||||
|
`call_deadline_s` 同 `recognize_text`(issue #22): `None` = 继承装配值。长文档的
|
||||||
|
硬上限靠它——源的 `timeout_s` 是逐次网络操作的超时,不是总耗时上界。
|
||||||
|
"""
|
||||||
|
# 校验早于链路,理由同 recognize_text;transport 能力校验也在统计边界之外
|
||||||
|
dimension_tenant_id, dimensions = validate_caller_dimensions(
|
||||||
|
tenant_id, meta, origin="parse_document(tenant_id=..., meta=...)"
|
||||||
|
)
|
||||||
|
self._document_transport()
|
||||||
|
outcome, call_stats = await self._call(
|
||||||
|
"document",
|
||||||
|
"parse_document",
|
||||||
|
document,
|
||||||
|
session_id,
|
||||||
|
parent_call_id,
|
||||||
|
dimension_tenant_id,
|
||||||
|
dimensions,
|
||||||
|
call_deadline_s,
|
||||||
|
)
|
||||||
|
result = outcome.result
|
||||||
|
return OcrDocumentResult(
|
||||||
|
elements=result.elements,
|
||||||
|
page_sizes=result.page_sizes,
|
||||||
|
markdown=result.markdown,
|
||||||
|
source_name=outcome.source.name,
|
||||||
|
usage=Usage(0, 0),
|
||||||
|
latency_ms=outcome.latency_ms,
|
||||||
|
call_id=outcome.call_id,
|
||||||
|
raw=result.raw,
|
||||||
|
call_stats=call_stats,
|
||||||
|
)
|
||||||
|
|
||||||
async def check_health(self) -> dict[str, bool]:
|
async def check_health(self) -> dict[str, bool]:
|
||||||
"""逐源并发健康预检(R10);探测失败=False 不上抛,取消穿透。"""
|
"""逐源并发健康预检(R10);探测失败=False 不上抛,取消穿透。"""
|
||||||
names = [s.name for s in self._sources]
|
names = [s.name for s in self._sources]
|
||||||
@@ -272,37 +326,38 @@ class OcrClient:
|
|||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
operation: CallOperation,
|
operation: CallOperation,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
session_id: str | None,
|
session_id: str | None,
|
||||||
parent_call_id: str | None,
|
parent_call_id: str | None,
|
||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
meta: dict[str, Any],
|
meta: dict[str, Any],
|
||||||
call_deadline_s: float | None = None,
|
call_deadline_s: float | None = None,
|
||||||
) -> tuple[_AttemptOutcome, CallStats]:
|
) -> tuple[_AttemptOutcome, CallStats]:
|
||||||
if not isinstance(image, bytes):
|
name = _PAYLOAD_NAME[kind]
|
||||||
raise TypeError("image 必须是 bytes(路径读取/批量拼帧留业务侧,D9)")
|
if not isinstance(payload, bytes):
|
||||||
if not image:
|
raise TypeError(f"{name} 必须是 bytes(路径读取/批量拼帧留业务侧,D9)")
|
||||||
raise ValueError("image 不能为空")
|
if not payload:
|
||||||
# 期限校验与 `image` 校验同列(仍在 `_CallContext` 之前、创建 awaitable 之前)
|
raise ValueError(f"{name} 不能为空")
|
||||||
|
# 期限校验与 `payload` 校验同列(仍在 `_CallContext` 之前、创建 awaitable 之前)
|
||||||
deadline = (
|
deadline = (
|
||||||
self._call_deadline_s
|
self._call_deadline_s
|
||||||
if call_deadline_s is None
|
if call_deadline_s is None
|
||||||
else ensure_call_deadline(call_deadline_s, f"{operation}(call_deadline_s=...)")
|
else ensure_call_deadline(call_deadline_s, f"{operation}(call_deadline_s=...)")
|
||||||
)
|
)
|
||||||
# M1 例外: `image` 校验在 `_call` 内而非公开方法,故上下文在该校验
|
# M1 例外: `payload` 校验在 `_call` 内而非公开方法,故上下文在该校验
|
||||||
# **通过之后**创建——这样设计 §3 的"校验在统计边界外"对 OCR 才成立
|
# **通过之后**创建——这样设计 §3 的"校验在统计边界外"对 OCR 才成立
|
||||||
context = _CallContext(now=self._now)
|
context = _CallContext(now=self._now)
|
||||||
try:
|
try:
|
||||||
return await with_call_deadline(
|
return await with_call_deadline(
|
||||||
self._run(
|
self._run(
|
||||||
kind, operation, image, session_id, parent_call_id, tenant_id, meta, context
|
kind, operation, payload, session_id, parent_call_id, tenant_id, meta, context
|
||||||
),
|
),
|
||||||
deadline_s=deadline,
|
deadline_s=deadline,
|
||||||
scope=self._scope,
|
scope=self._scope,
|
||||||
)
|
)
|
||||||
except PolyGatewayError as exc:
|
except PolyGatewayError as exc:
|
||||||
await self._emit_terminal(
|
await self._emit_terminal(
|
||||||
kind, operation, image, session_id, parent_call_id, tenant_id, meta, context, exc
|
kind, operation, payload, session_id, parent_call_id, tenant_id, meta, context, exc
|
||||||
)
|
)
|
||||||
raise
|
raise
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
@@ -310,7 +365,7 @@ class OcrClient:
|
|||||||
await self._emit_terminal(
|
await self._emit_terminal(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
session_id,
|
session_id,
|
||||||
parent_call_id,
|
parent_call_id,
|
||||||
tenant_id,
|
tenant_id,
|
||||||
@@ -324,7 +379,7 @@ class OcrClient:
|
|||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
operation: CallOperation,
|
operation: CallOperation,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
session_id: str | None,
|
session_id: str | None,
|
||||||
parent_call_id: str | None,
|
parent_call_id: str | None,
|
||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
@@ -351,7 +406,7 @@ class OcrClient:
|
|||||||
outcome = await self._attempt(
|
outcome = await self._attempt(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
*picked,
|
*picked,
|
||||||
reasons,
|
reasons,
|
||||||
session_id,
|
session_id,
|
||||||
@@ -377,7 +432,7 @@ class OcrClient:
|
|||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
operation: CallOperation,
|
operation: CallOperation,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
source: SourceConfig,
|
source: SourceConfig,
|
||||||
permit: Permit,
|
permit: Permit,
|
||||||
entry: GateDecision,
|
entry: GateDecision,
|
||||||
@@ -398,7 +453,7 @@ class OcrClient:
|
|||||||
# 裸生成时间(1.3.7 H8): 计时只包 transport 调用本身(同 RetryMW 口径);
|
# 裸生成时间(1.3.7 H8): 计时只包 transport 调用本身(同 RetryMW 口径);
|
||||||
# layout 的 POST + ZIP GET 在同一次 `_invoke` 内,属同一段生成耗时
|
# layout 的 POST + ZIP GET 在同一次 `_invoke` 内,属同一段生成耗时
|
||||||
gen_started = self._now()
|
gen_started = self._now()
|
||||||
result = await self._invoke(kind, image, source, call_id)
|
result = await self._invoke(kind, payload, source, call_id)
|
||||||
gen_ms = int((self._now() - gen_started) * 1000)
|
gen_ms = int((self._now() - gen_started) * 1000)
|
||||||
await self._record_quietly(self._breaker.record_success(entry))
|
await self._record_quietly(self._breaker.record_success(entry))
|
||||||
await self._record_quietly(self._quota.mark_progress())
|
await self._record_quietly(self._quota.mark_progress())
|
||||||
@@ -408,7 +463,7 @@ class OcrClient:
|
|||||||
await self._emit(
|
await self._emit(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
source,
|
source,
|
||||||
call_id,
|
call_id,
|
||||||
started,
|
started,
|
||||||
@@ -425,7 +480,7 @@ class OcrClient:
|
|||||||
await self._emit(
|
await self._emit(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
source,
|
source,
|
||||||
call_id,
|
call_id,
|
||||||
started,
|
started,
|
||||||
@@ -443,7 +498,7 @@ class OcrClient:
|
|||||||
await self._emit(
|
await self._emit(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
source,
|
source,
|
||||||
call_id,
|
call_id,
|
||||||
started,
|
started,
|
||||||
@@ -464,7 +519,7 @@ class OcrClient:
|
|||||||
await self._emit(
|
await self._emit(
|
||||||
kind,
|
kind,
|
||||||
operation,
|
operation,
|
||||||
image,
|
payload,
|
||||||
source,
|
source,
|
||||||
call_id,
|
call_id,
|
||||||
started,
|
started,
|
||||||
@@ -482,11 +537,31 @@ class OcrClient:
|
|||||||
await settle_and_release(permit, 0)
|
await settle_and_release(permit, 0)
|
||||||
|
|
||||||
async def _invoke(
|
async def _invoke(
|
||||||
self, kind: _OcrKind, image: bytes, source: SourceConfig, call_id: str
|
self, kind: _OcrKind, payload: bytes, source: SourceConfig, call_id: str
|
||||||
) -> OcrTextTransportResult | OcrLayoutTransportResult:
|
) -> _TransportResult:
|
||||||
if kind == "text":
|
if kind == "text":
|
||||||
return await self._transport.recognize_text(image=image, source=source, call_id=call_id)
|
return await self._transport.recognize_text(
|
||||||
return await self._transport.parse_layout(image=image, source=source, call_id=call_id)
|
image=payload, source=source, call_id=call_id
|
||||||
|
)
|
||||||
|
if kind == "document":
|
||||||
|
return await self._document_transport().parse_document(
|
||||||
|
document=payload, source=source, call_id=call_id
|
||||||
|
)
|
||||||
|
return await self._transport.parse_layout(image=payload, source=source, call_id=call_id)
|
||||||
|
|
||||||
|
def _document_transport(self) -> OcrDocumentTransport:
|
||||||
|
"""文档入口所需的 transport 能力校验(唯一的 isinstance 点)。
|
||||||
|
|
||||||
|
`OcrDocumentTransport` 独立于 `OcrTransport`: 往旧 Protocol 加成员会让所有现有
|
||||||
|
实现(测试替身、下游自带 fake)一夜之间不合格。代价是这里必须在运行时校验——
|
||||||
|
公共入口在进入统计边界前调用一次,让装配错误当场报。
|
||||||
|
"""
|
||||||
|
if not isinstance(self._transport, OcrDocumentTransport):
|
||||||
|
raise TypeError(
|
||||||
|
"parse_document 需要实现 OcrDocumentTransport.parse_document 的 transport: "
|
||||||
|
f"{type(self._transport).__name__} 不满足"
|
||||||
|
)
|
||||||
|
return self._transport
|
||||||
|
|
||||||
# —— 辅助(与 embedding 同口径)——
|
# —— 辅助(与 embedding 同口径)——
|
||||||
|
|
||||||
@@ -519,7 +594,7 @@ class OcrClient:
|
|||||||
def _request_for(
|
def _request_for(
|
||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
session_id: str | None,
|
session_id: str | None,
|
||||||
parent_call_id: str | None,
|
parent_call_id: str | None,
|
||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
@@ -533,7 +608,7 @@ class OcrClient:
|
|||||||
维度恒为空、`logical_call_id` 恒为 NULL。
|
维度恒为空、`logical_call_id` 恒为 NULL。
|
||||||
"""
|
"""
|
||||||
return ChatRequest(
|
return ChatRequest(
|
||||||
messages=[{"role": "user", "content": f"<ocr:{kind} image_bytes={len(image)}>"}],
|
messages=[{"role": "user", "content": f"<ocr:{kind} image_bytes={len(payload)}>"}],
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
parent_call_id=parent_call_id,
|
parent_call_id=parent_call_id,
|
||||||
tenant_id=tenant_id,
|
tenant_id=tenant_id,
|
||||||
@@ -545,7 +620,7 @@ class OcrClient:
|
|||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
operation: CallOperation,
|
operation: CallOperation,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
session_id: str | None,
|
session_id: str | None,
|
||||||
parent_call_id: str | None,
|
parent_call_id: str | None,
|
||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
@@ -557,7 +632,7 @@ class OcrClient:
|
|||||||
await emit_terminal_once(
|
await emit_terminal_once(
|
||||||
self._emitter,
|
self._emitter,
|
||||||
request=self._request_for(
|
request=self._request_for(
|
||||||
kind, image, session_id, parent_call_id, tenant_id, meta, context
|
kind, payload, session_id, parent_call_id, tenant_id, meta, context
|
||||||
),
|
),
|
||||||
context=context,
|
context=context,
|
||||||
error=error,
|
error=error,
|
||||||
@@ -570,7 +645,7 @@ class OcrClient:
|
|||||||
self,
|
self,
|
||||||
kind: _OcrKind,
|
kind: _OcrKind,
|
||||||
operation: CallOperation,
|
operation: CallOperation,
|
||||||
image: bytes,
|
payload: bytes,
|
||||||
source: SourceConfig,
|
source: SourceConfig,
|
||||||
call_id: str,
|
call_id: str,
|
||||||
started: float,
|
started: float,
|
||||||
@@ -579,14 +654,14 @@ class OcrClient:
|
|||||||
tenant_id: str | None,
|
tenant_id: str | None,
|
||||||
meta: dict[str, Any],
|
meta: dict[str, Any],
|
||||||
context: _CallContext,
|
context: _CallContext,
|
||||||
result: OcrTextTransportResult | OcrLayoutTransportResult | None = None,
|
result: _TransportResult | None = None,
|
||||||
error: PolyGatewayError | str | None = None,
|
error: PolyGatewayError | str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""逐尝试遥测(单一 Emitter): messages 占位摘要,图像 bytes 绝不入库。"""
|
"""逐尝试遥测(单一 Emitter): messages 占位摘要,图像 bytes 绝不入库。"""
|
||||||
if self._emitter is None:
|
if self._emitter is None:
|
||||||
return
|
return
|
||||||
request = self._request_for(
|
request = self._request_for(
|
||||||
kind, image, session_id, parent_call_id, tenant_id, meta, context
|
kind, payload, session_id, parent_call_id, tenant_id, meta, context
|
||||||
)
|
)
|
||||||
latency_ms = int((self._now() - started) * 1000)
|
latency_ms = int((self._now() - started) * 1000)
|
||||||
response = None
|
response = None
|
||||||
@@ -622,7 +697,19 @@ class OcrClient:
|
|||||||
)
|
)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _summarize(result: OcrTextTransportResult | OcrLayoutTransportResult) -> str:
|
def _summarize(result: _TransportResult) -> str:
|
||||||
|
"""遥测行 response 摘要。**永不写入文档正文**:文档只记规模(元素数/页数/md 字数)。
|
||||||
|
|
||||||
|
**按结构分派而不是 isinstance**: 旧实现就是 `hasattr(result, "text")`,下游测试替身
|
||||||
|
可能返回非库类型的鸭子对象,换成 isinstance 会让它们在开启遥测后以原生
|
||||||
|
AttributeError 漏出。顺序要紧:文档结果有 `markdown`,先于只看 `text` 的判定,否则
|
||||||
|
会被静默套成 layout 摘要而漏记 `md_chars`。
|
||||||
|
"""
|
||||||
|
if hasattr(result, "markdown"):
|
||||||
|
return (
|
||||||
|
f"<document elements={len(result.elements)} pages={len(result.page_sizes)} "
|
||||||
|
f"md_chars={len(result.markdown)}>"
|
||||||
|
)
|
||||||
if hasattr(result, "text"):
|
if hasattr(result, "text"):
|
||||||
return result.text[:_RESPONSE_TEXT_CAP]
|
return result.text[:_RESPONSE_TEXT_CAP]
|
||||||
return f"<elements n={len(result.elements)} pages={len(result.page_sizes)}>"
|
return f"<elements n={len(result.elements)} pages={len(result.page_sizes)}>"
|
||||||
|
|||||||
@@ -16,6 +16,8 @@ from .types import (
|
|||||||
Effort,
|
Effort,
|
||||||
EmbeddingTransportResult,
|
EmbeddingTransportResult,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
|
OcrDocumentResult,
|
||||||
|
OcrDocumentTransportResult,
|
||||||
OcrLayoutResult,
|
OcrLayoutResult,
|
||||||
OcrLayoutTransportResult,
|
OcrLayoutTransportResult,
|
||||||
OcrTextResult,
|
OcrTextResult,
|
||||||
@@ -102,6 +104,26 @@ class OcrLayoutPort(Protocol):
|
|||||||
async def parse_layout(self, image: bytes) -> OcrLayoutResult: ...
|
async def parse_layout(self, image: bytes) -> OcrLayoutResult: ...
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class OcrDocumentPort(Protocol):
|
||||||
|
"""OCR 整份文档解析端口(#34;对应 MonkeyOCR POST /parse → GET ZIP,PDF 或图片)。"""
|
||||||
|
|
||||||
|
async def parse_document(self, document: bytes) -> OcrDocumentResult: ...
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class OcrDocumentTransport(Protocol):
|
||||||
|
"""整份文档解析的协议细节(#34 设计 §4.2);不含任何治理。
|
||||||
|
|
||||||
|
**独立于 `OcrTransport`**: 往那个 `runtime_checkable` Protocol 加成员,会让所有
|
||||||
|
现有实现(测试替身、下游自带 fake)在 `isinstance` 上一夜之间不合格。
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def parse_document(
|
||||||
|
self, *, document: bytes, source: SourceConfig, call_id: str
|
||||||
|
) -> OcrDocumentTransportResult: ...
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
@runtime_checkable
|
||||||
class OcrTransport(Protocol):
|
class OcrTransport(Protocol):
|
||||||
"""一次原始 OCR 调用的协议细节(M3 设计 §3.2);不含任何治理。
|
"""一次原始 OCR 调用的协议细节(M3 设计 §3.2);不含任何治理。
|
||||||
|
|||||||
@@ -0,0 +1,328 @@
|
|||||||
|
"""MonkeyOCR 结果包解析: bytes → 文档元素的纯函数(#34 设计 §5/§5.1)。
|
||||||
|
|
||||||
|
本模块**无 HTTP、无治理、无全局状态**,所以能在 `asyncio.to_thread` 里运行——
|
||||||
|
同步解析大对象会占住事件循环,使取消与 `call_deadline_s`(基于 `asyncio.timeout`)
|
||||||
|
无法及时生效。
|
||||||
|
|
||||||
|
内容只取自 `_middle.json`(bbox/类型/页码/阅读序的权威来源,spans 里直接带
|
||||||
|
`content`/`html`)与配对的 `.md`。**有意不读 `_content_list.json`**: 它是展示格式,
|
||||||
|
文本里 `* \\` ~ $` 会被转义成 `\\$` 之类(实测 134 个 text/title 块中 23 个含 `\\$`,
|
||||||
|
取它会污染 LaTeX),且无 bbox、类型词汇与 middle 不同,逐块对位还会引入版本漂移
|
||||||
|
时整份失败的新故障面。
|
||||||
|
|
||||||
|
与 layout 路径(`monkey_ocr._parse_middle_json`,既有调用方在用,保持原样)的三处差异:
|
||||||
|
零页拒绝(成功解析的文档至少 1 页)、`page_idx` 必须等于页序(layout 路径回退枚举
|
||||||
|
下标,会让 page_sizes 按 page_index 索引的契约静默错位)、bbox 只要求有限且有序
|
||||||
|
(整数化退化校验专为 CHS shim 的 `int()` 裁剪,文档消费者不裁剪,一个退化框不该
|
||||||
|
让数百个块的整份文档失败)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import lzma
|
||||||
|
import math
|
||||||
|
import zipfile
|
||||||
|
import zlib
|
||||||
|
from io import BytesIO
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from polygateway.errors import ResultInvalidError
|
||||||
|
from polygateway.types import OcrDocumentElement
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
MAX_MIDDLE_BYTES = 64 * 1024 * 1024
|
||||||
|
"""`_middle.json` 声明大小上限。样本约 72KB/页,64MiB 足够数百页;超限即拒,
|
||||||
|
防止一个畸形/恶意包把解析线程的内存吃满。"""
|
||||||
|
|
||||||
|
MAX_MARKDOWN_BYTES = 16 * 1024 * 1024
|
||||||
|
"""`.md` 声明大小上限(样本约 3KB/页)。"""
|
||||||
|
|
||||||
|
_MIDDLE_SUFFIX = "_middle.json"
|
||||||
|
|
||||||
|
# ZIP 层「外部数据不可读」的标准库约定用法。只在 ZIP I/O 区域(构造 + 读取)内捕获,
|
||||||
|
# 不放到外层边界——宽泛捕获 RuntimeError/OSError 会掩盖编程错误:
|
||||||
|
# 加密成员 → RuntimeError;未知压缩方法/版本字段损坏 → NotImplementedError;
|
||||||
|
# DEFLATE 流损坏 → zlib.error / EOFError;BZIP2 流损坏 → OSError;LZMA 流损坏 → LZMAError
|
||||||
|
_ZIP_IO_ERRORS = (RuntimeError, NotImplementedError, zlib.error, EOFError, OSError, lzma.LZMAError)
|
||||||
|
|
||||||
|
|
||||||
|
def finite_number(value: object) -> float:
|
||||||
|
"""JSON 数值 → 有限浮点;布尔与非有限数拒绝(CHS invokers.py:427 逐字)。"""
|
||||||
|
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||||
|
raise ValueError("字段不是数值")
|
||||||
|
number = float(value)
|
||||||
|
if not math.isfinite(number):
|
||||||
|
raise ValueError("字段不是有限数值")
|
||||||
|
return number
|
||||||
|
|
||||||
|
|
||||||
|
def parse_page_size(page: dict) -> tuple[float, float]:
|
||||||
|
"""`page_size` → (宽, 高);须为 2 个正有限数(CHS invokers.py 数值防御,原样迁入)。"""
|
||||||
|
page_size = page.get("page_size")
|
||||||
|
if not isinstance(page_size, list) or len(page_size) != 2:
|
||||||
|
raise ValueError("page_size 无效")
|
||||||
|
width, height = (finite_number(item) for item in page_size)
|
||||||
|
if width <= 0 or height <= 0:
|
||||||
|
raise ValueError("page_size 必须为正数")
|
||||||
|
return (width, height)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_document_package(
|
||||||
|
zip_bytes: bytes,
|
||||||
|
) -> tuple[list[OcrDocumentElement], list[tuple[float, float]], str]:
|
||||||
|
"""ZIP → (elements, page_sizes, markdown);一切形态异常归 `ResultInvalidError`。
|
||||||
|
|
||||||
|
与 layout 路径的三处差异(理由见模块 docstring): ① 零页拒绝(成功解析的文档至少 1 页);
|
||||||
|
② `page_idx` 必须等于页序,不回退枚举下标;③ bbox 只要求有限且有序,不做整数化退化校验。
|
||||||
|
|
||||||
|
外层异常边界**同时盖住「读取」与「全部结构解析」**: `finite_number` 里
|
||||||
|
`float(10**400)` 会抛 `OverflowError`,它发生在读取之后。放错位置的后果是原生
|
||||||
|
异常漏出——`OcrClient._call` 只接 `PolyGatewayError`,于是既不写终态遥测行,又让
|
||||||
|
持有的半开探针不被归还。`CancelledError` 是 `BaseException`,天然不在其内。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
payload, markdown = _read_members(zip_bytes)
|
||||||
|
elements, page_sizes = _parse_pages(payload)
|
||||||
|
except ResultInvalidError:
|
||||||
|
raise
|
||||||
|
except (
|
||||||
|
ValueError, # 含 JSONDecodeError / UnicodeDecodeError
|
||||||
|
TypeError,
|
||||||
|
KeyError,
|
||||||
|
OverflowError,
|
||||||
|
RecursionError,
|
||||||
|
zipfile.BadZipFile,
|
||||||
|
zipfile.LargeZipFile,
|
||||||
|
) as exc:
|
||||||
|
raise ResultInvalidError(f"OCR 文档结果包内容无效: {exc}") from exc
|
||||||
|
return elements, page_sizes, markdown
|
||||||
|
|
||||||
|
|
||||||
|
def _read_members(zip_bytes: bytes) -> tuple[object, str]:
|
||||||
|
"""取首个 `*_middle.json` 与按名配对的 `.md`,返回 (middle 解码对象, md 文本)。
|
||||||
|
|
||||||
|
`ZipFile(...)` 的构造与成员读取在**同一个**窄 `try` 里: 独立验证发现中央目录版本字段
|
||||||
|
损坏会在构造阶段抛 `NotImplementedError`,而旧写法只包了成员读取。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(BytesIO(zip_bytes)) as archive:
|
||||||
|
middle_raw, markdown_raw = _read_pair(archive)
|
||||||
|
except _ZIP_IO_ERRORS as exc:
|
||||||
|
raise ResultInvalidError(f"OCR 文档结果包无法读取: {exc}") from exc
|
||||||
|
return json.loads(middle_raw), markdown_raw.decode("utf-8-sig")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_pair(archive: zipfile.ZipFile) -> tuple[bytes, bytes]:
|
||||||
|
"""定位并读取 middle 与配对的 md;缺失或超限抛 `ResultInvalidError`。"""
|
||||||
|
names = archive.namelist()
|
||||||
|
middle_names = [n for n in names if n.endswith(_MIDDLE_SUFFIX)]
|
||||||
|
if not middle_names:
|
||||||
|
raise ResultInvalidError("OCR 文档结果包缺少 _middle.json")
|
||||||
|
middle_name = middle_names[0]
|
||||||
|
# 精确配对而非"取第一个 .md": 包里可能有多个 .md,取错就是把别的文档当本文档
|
||||||
|
markdown_name = middle_name[: -len(_MIDDLE_SUFFIX)] + ".md"
|
||||||
|
if markdown_name not in names:
|
||||||
|
raise ResultInvalidError(f"OCR 文档结果包缺少与 {middle_name} 配对的 {markdown_name}")
|
||||||
|
return (
|
||||||
|
_read_member(archive, middle_name, MAX_MIDDLE_BYTES),
|
||||||
|
_read_member(archive, markdown_name, MAX_MARKDOWN_BYTES),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_member(archive: zipfile.ZipFile, name: str, cap: int) -> bytes:
|
||||||
|
"""读一个成员:先查**声明大小**再解压。
|
||||||
|
|
||||||
|
`zipfile` 读取不会超过声明大小(超出即 CRC 失败),所以声明大小检查足以界定内存。
|
||||||
|
解压异常由调用方 `_read_members` 的窄 `try` 统一转换。
|
||||||
|
"""
|
||||||
|
info = archive.getinfo(name)
|
||||||
|
if info.file_size > cap:
|
||||||
|
raise ResultInvalidError(
|
||||||
|
f"OCR 文档结果包成员 {name} 声明大小 {info.file_size} 超过上限 {cap}"
|
||||||
|
)
|
||||||
|
with archive.open(info) as stream:
|
||||||
|
return stream.read()
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_pages(
|
||||||
|
payload: object,
|
||||||
|
) -> tuple[list[OcrDocumentElement], list[tuple[float, float]]]:
|
||||||
|
"""`pdf_info` → 全部页的元素与页尺寸;零页拒绝。"""
|
||||||
|
if not isinstance(payload, dict) or not isinstance(payload.get("pdf_info"), list):
|
||||||
|
raise ValueError("_middle.json 缺少 pdf_info 列表")
|
||||||
|
pages = payload["pdf_info"]
|
||||||
|
if not pages:
|
||||||
|
raise ResultInvalidError("OCR 文档结果包 pdf_info 为空: 成功解析的文档至少应有 1 页")
|
||||||
|
elements: list[OcrDocumentElement] = []
|
||||||
|
page_sizes: list[tuple[float, float]] = []
|
||||||
|
for position, page in enumerate(pages):
|
||||||
|
page_elements, page_size = _parse_page(page, position)
|
||||||
|
elements.extend(page_elements)
|
||||||
|
page_sizes.append(page_size)
|
||||||
|
return elements, page_sizes
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_page(
|
||||||
|
page: object, position: int
|
||||||
|
) -> tuple[list[OcrDocumentElement], tuple[float, float]]:
|
||||||
|
"""单页 → (该页元素, 页尺寸);`para_blocks` 缺失按空页处理。"""
|
||||||
|
if not isinstance(page, dict):
|
||||||
|
raise ValueError("页面结构无效")
|
||||||
|
page_size = parse_page_size(page)
|
||||||
|
page_index = _page_index(page, position)
|
||||||
|
blocks = page.get("para_blocks", []) # 缺失 = 空页(空白页/无可识别块)是合法"无内容"
|
||||||
|
if not isinstance(blocks, list):
|
||||||
|
raise ValueError("para_blocks 无效")
|
||||||
|
return [e for block in blocks for e in _block_elements(block, page_index)], page_size
|
||||||
|
|
||||||
|
|
||||||
|
def _page_index(page: dict, position: int) -> int:
|
||||||
|
"""`page_idx` 必须是等于页序的整数(非 bool)。
|
||||||
|
|
||||||
|
不沿用 layout 路径的"非法则回退枚举下标": 回退会掩盖服务端跳页/乱序,而
|
||||||
|
`page_sizes` 是按 page_index 索引的,静默错位比响亮失败更糟。
|
||||||
|
"""
|
||||||
|
raw = page.get("page_idx")
|
||||||
|
if isinstance(raw, bool) or not isinstance(raw, int) or raw != position:
|
||||||
|
raise ValueError(f"page_idx 无效或与页序不符: {raw!r} != {position}")
|
||||||
|
return raw
|
||||||
|
|
||||||
|
|
||||||
|
def _block_type(block: object) -> str:
|
||||||
|
"""校验块是 dict 且 `type` 为非空白字符串,返回 type。"""
|
||||||
|
if not isinstance(block, dict):
|
||||||
|
raise ValueError("块结构无效")
|
||||||
|
block_type = block.get("type")
|
||||||
|
if not isinstance(block_type, str) or not block_type.strip():
|
||||||
|
raise ValueError("块缺少 type")
|
||||||
|
return block_type
|
||||||
|
|
||||||
|
|
||||||
|
def _loose_bbox(raw: object) -> tuple[float, float, float, float]:
|
||||||
|
"""长度 4 的有限数值且有序(允许零宽/零高);差异理由见模块 docstring。"""
|
||||||
|
if not isinstance(raw, list) or len(raw) != 4:
|
||||||
|
raise ValueError("bbox 无效")
|
||||||
|
x1, y1, x2, y2 = (finite_number(item) for item in raw)
|
||||||
|
if x2 < x1 or y2 < y1:
|
||||||
|
raise ValueError("bbox 边界顺序无效")
|
||||||
|
return (x1, y1, x2, y2)
|
||||||
|
|
||||||
|
|
||||||
|
def _sub_blocks(block: dict) -> list[dict]:
|
||||||
|
"""嵌套子块列表(缺失为空);子块须是 dict 且 type 合法。"""
|
||||||
|
subs = block.get("blocks", [])
|
||||||
|
if not isinstance(subs, list):
|
||||||
|
raise ValueError("blocks 无效")
|
||||||
|
for sub in subs:
|
||||||
|
_block_type(sub) # 顺带校验子块是 dict 且 type 合法
|
||||||
|
return subs
|
||||||
|
|
||||||
|
|
||||||
|
def _is_body(sub: dict) -> bool:
|
||||||
|
"""子块是否正文部分(type 以 `_body` 结尾,如 table_body / image_body)。"""
|
||||||
|
return sub["type"].endswith("_body")
|
||||||
|
|
||||||
|
|
||||||
|
def _block_elements(block: object, page_index: int) -> list[OcrDocumentElement]:
|
||||||
|
"""父块 → 元素,外加其非 `*_body` 的嵌套子块(紧随父块、保持服务顺序)。"""
|
||||||
|
block_type = _block_type(block)
|
||||||
|
assert isinstance(block, dict) # _block_type 已校验;仅为类型收窄(内部不变量)
|
||||||
|
parent = OcrDocumentElement(
|
||||||
|
type=block_type,
|
||||||
|
bbox=_loose_bbox(block.get("bbox")),
|
||||||
|
page_index=page_index,
|
||||||
|
text=_block_text(block_type, block),
|
||||||
|
)
|
||||||
|
return [parent, *_extra_elements(block, page_index)]
|
||||||
|
|
||||||
|
|
||||||
|
def _extra_elements(block: dict, page_index: int) -> list[OcrDocumentElement]:
|
||||||
|
"""caption/footnote 一类子块:默认部署不产生,出现时有文字就独立成元素,不静默丢弃。"""
|
||||||
|
extras: list[OcrDocumentElement] = []
|
||||||
|
for sub in _sub_blocks(block):
|
||||||
|
if _is_body(sub):
|
||||||
|
continue
|
||||||
|
text = _lines_text(sub.get("lines", []))
|
||||||
|
if text:
|
||||||
|
extras.append(
|
||||||
|
OcrDocumentElement(
|
||||||
|
type=sub["type"],
|
||||||
|
bbox=_loose_bbox(sub.get("bbox")),
|
||||||
|
page_index=page_index,
|
||||||
|
text=text,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return extras
|
||||||
|
|
||||||
|
|
||||||
|
_TEXT_BEARING_TYPES = frozenset({"text", "title", "interline_equation"})
|
||||||
|
"""必带 `lines` 的块类型。缺 `lines` 不能当作"空文本"——与 table 缺内容就报错同口径,
|
||||||
|
否则下游分不清"服务识别出空块"和"服务给了坏结构"。其余未知类型无 lines 仍是合法空文本。"""
|
||||||
|
|
||||||
|
|
||||||
|
def _block_text(block_type: str, block: dict) -> str:
|
||||||
|
"""块 → 文字;随 type 分派(table 取 html/latex,image 为空串,其余取 lines)。"""
|
||||||
|
if block_type == "table":
|
||||||
|
return _table_text(_sub_blocks(block))
|
||||||
|
if block_type == "image":
|
||||||
|
return "" # 图内文字服务不识别;span 只有 image_path
|
||||||
|
if block_type in _TEXT_BEARING_TYPES and "lines" not in block:
|
||||||
|
raise ValueError(f"{block_type} 块缺少 lines")
|
||||||
|
if "lines" in block:
|
||||||
|
return _lines_text(block["lines"])
|
||||||
|
# 未知容器类型: 其 `*_body` 子块的文字并入父元素(非 body 子块已由 _extra_elements 输出)
|
||||||
|
return "\n".join(
|
||||||
|
_lines_text(sub.get("lines", [])) for sub in _sub_blocks(block) if _is_body(sub)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _table_text(subs: list[dict]) -> str:
|
||||||
|
"""table 块 → 其 `table_body` 子块的 HTML(或 latex);没有表体即畸形。"""
|
||||||
|
bodies = [sub for sub in subs if sub["type"] == "table_body"]
|
||||||
|
if not bodies:
|
||||||
|
raise ValueError("table 块缺少 table_body 子块")
|
||||||
|
return "\n".join(_lines_text(body.get("lines"), _table_span_text) for body in bodies)
|
||||||
|
|
||||||
|
|
||||||
|
def _table_span_text(span: dict) -> str:
|
||||||
|
"""表体 span 取 html;上游另有 latex 分支。皆无则响亮失败——表格"无内容"与
|
||||||
|
合法空串在结果里无法区分,静默补空会让下游把缺失当成空表。"""
|
||||||
|
for key in ("html", "latex"):
|
||||||
|
value = span.get(key)
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value
|
||||||
|
raise ValueError("table span 缺少 html/latex")
|
||||||
|
|
||||||
|
|
||||||
|
def _lines_text(lines: object, span_text: Callable[[dict], str] | None = None) -> str:
|
||||||
|
"""行内 span 按 "" 拼、行间按 "\\n" 拼。
|
||||||
|
|
||||||
|
样本全是 1 行 1 span;多行/多 span 路径无真实样本,拼接规则由构造 fixture 钉住,
|
||||||
|
文档注明"未经实测"。
|
||||||
|
"""
|
||||||
|
if not isinstance(lines, list):
|
||||||
|
raise ValueError("lines 无效")
|
||||||
|
return "\n".join(_line_text(line, span_text or _span_text) for line in lines)
|
||||||
|
|
||||||
|
|
||||||
|
def _line_text(line: object, span_text: Callable[[dict], str]) -> str:
|
||||||
|
"""一行 → 文字:该行全部 span 的文字直接相接。"""
|
||||||
|
if not isinstance(line, dict) or not isinstance(line.get("spans"), list):
|
||||||
|
raise ValueError("行结构无效")
|
||||||
|
parts = []
|
||||||
|
for span in line["spans"]:
|
||||||
|
if not isinstance(span, dict):
|
||||||
|
raise ValueError("span 结构无效")
|
||||||
|
parts.append(span_text(span))
|
||||||
|
return "".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _span_text(span: dict) -> str:
|
||||||
|
"""文本 span 取 `content`(必须是 str);`inline_equation` 包 `$…$`(上游惯例)。"""
|
||||||
|
content = span.get("content")
|
||||||
|
if not isinstance(content, str):
|
||||||
|
raise ValueError("span 缺少字符串 content")
|
||||||
|
return f"${content}$" if span.get("type") == "inline_equation" else content
|
||||||
@@ -10,11 +10,10 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import math
|
|
||||||
import zipfile
|
import zipfile
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from io import BytesIO
|
from io import BytesIO
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any, Literal
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
@@ -29,7 +28,13 @@ from polygateway.transports._http_errors import (
|
|||||||
response_body,
|
response_body,
|
||||||
summarize_body,
|
summarize_body,
|
||||||
)
|
)
|
||||||
|
from polygateway.transports._monkey_document import (
|
||||||
|
finite_number,
|
||||||
|
parse_document_package,
|
||||||
|
parse_page_size,
|
||||||
|
)
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
|
OcrDocumentTransportResult,
|
||||||
OcrLayoutElement,
|
OcrLayoutElement,
|
||||||
OcrLayoutTransportResult,
|
OcrLayoutTransportResult,
|
||||||
OcrTextTransportResult,
|
OcrTextTransportResult,
|
||||||
@@ -42,6 +47,88 @@ if TYPE_CHECKING:
|
|||||||
|
|
||||||
_HEALTH_TIMEOUT_S = 5.0 # 探测常量(同 jitter 系数先例,不进配置)
|
_HEALTH_TIMEOUT_S = 5.0 # 探测常量(同 jitter 系数先例,不进配置)
|
||||||
|
|
||||||
|
_MediaKind = Literal["pdf", "image", "unknown"]
|
||||||
|
|
||||||
|
# 已知图片魔数。起始命中即判图片,且**优先于** PDF 判定: 含 XMP 元数据的图片
|
||||||
|
# 可能在前 1KB 内带 `%PDF-` 字样,不能因此被当成 PDF。
|
||||||
|
_IMAGE_MAGICS: tuple[bytes, ...] = (
|
||||||
|
b"\xff\xd8\xff", # JPEG
|
||||||
|
b"\x89PNG\r\n\x1a\n", # PNG
|
||||||
|
b"II*\x00", # TIFF 小端
|
||||||
|
b"MM\x00*", # TIFF 大端
|
||||||
|
b"GIF87a",
|
||||||
|
b"GIF89a",
|
||||||
|
)
|
||||||
|
_PDF_HEADER = b"%PDF-"
|
||||||
|
# BMP 的 "BM" 只有两字节,任何以 BM 开头的文本都会撞上。再要求偏移 14 处的 DIB 头大小字段合法
|
||||||
|
_BMP_DIB_HEADER_SIZES = frozenset({12, 40, 52, 56, 64, 108, 124})
|
||||||
|
|
||||||
|
# 文档路径把这些「确定性坏输入」的 500 改归 RequestRejected(#34 设计 §6)。小写子串匹配,
|
||||||
|
# 取 2026-10-21 实测(截断 PDF / 伪 PDF 头 / 乱码 / 空文件 / 扩展名与内容不符)。
|
||||||
|
# **不含** `unsupported file type`: 嗅探后它只可能由服务契约变化或我方标签 bug 引起,
|
||||||
|
# 应响亮失败并计熔断。标记是 PyMuPDF 的界面文案,随服务版本漂移,故可经构造参数覆盖。
|
||||||
|
_DEFAULT_REJECTED_MARKERS: tuple[str, ...] = (
|
||||||
|
"failed to open stream",
|
||||||
|
"cannot open empty stream",
|
||||||
|
"cannot save with zero pages",
|
||||||
|
"is no pdf",
|
||||||
|
)
|
||||||
|
# 服务把一切异常(含它自己抛的 400、`Model not initialized` 这类真故障)都包成同一前缀,
|
||||||
|
# 所以前缀只是必要条件,必须同时命中具体标记
|
||||||
|
_PARSING_FAILED_PREFIX = "parsing failed:"
|
||||||
|
|
||||||
|
_ERROR_BODY_READ_CAP = 64 * 1024
|
||||||
|
"""文档路径读取错误响应体的字节上限。摘要只取几 KiB(`summarize_body`),服务或中间人回一个
|
||||||
|
巨大的错误体不该被整个读进内存(独立审查实测 600 MiB 的 500 让 RSS 涨到 1.8 GiB)。"""
|
||||||
|
|
||||||
|
_MAX_PACKAGE_BYTES = 256 * 1024 * 1024
|
||||||
|
"""整包(含 `images/*.jpg`)体积上限,仅文档路径。4 页含 4 图约 598KB,数百页含图的文档
|
||||||
|
也远低于此;超限即拒,防止畸形/恶意响应把内存吃满。httpx 非流式 `get` 会先整包 `aread()`,
|
||||||
|
这正是文档路径改用流式下载并逐块累计的原因。"""
|
||||||
|
# Adobe 阅读器容忍头部前有垃圾字节(惯例而非 ISO 32000 正文);服务端实测可解析
|
||||||
|
# 前缀 200 字节垃圾的 PDF(#34 设计 E3)
|
||||||
|
_PDF_HEADER_WINDOW = 1024
|
||||||
|
|
||||||
|
|
||||||
|
def _is_webp(data: bytes) -> bool:
|
||||||
|
"""RIFF 容器且类型码为 WEBP。"""
|
||||||
|
return data[:4] == b"RIFF" and data[8:12] == b"WEBP"
|
||||||
|
|
||||||
|
|
||||||
|
def _is_bmp(data: bytes) -> bool:
|
||||||
|
"""`BM` 开头且 DIB 头大小字段合法(单靠两字节魔数会误判以 BM 开头的文本)。"""
|
||||||
|
return (
|
||||||
|
data.startswith(b"BM")
|
||||||
|
and len(data) >= 18
|
||||||
|
and int.from_bytes(data[14:18], "little") in _BMP_DIB_HEADER_SIZES
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _sniff_media(data: bytes) -> _MediaKind:
|
||||||
|
"""按内容判定上传类型: 图片魔数 → 前 1KB 内 `%PDF-` → 未知。
|
||||||
|
|
||||||
|
服务端按**文件名扩展名**分派(`.pdf` 走 PDF 数据集,其余走图片数据集),而标签
|
||||||
|
若与内容不符会得到 HTTP 500(`is no PDF`)或——在 `/ocr/text` 上——`success:true`
|
||||||
|
的幻觉文本(#34 设计 E2/E4)。所以标签只能由库按内容定,不开放给调用方声明。
|
||||||
|
"""
|
||||||
|
if data.startswith(_IMAGE_MAGICS) or _is_webp(data) or _is_bmp(data):
|
||||||
|
return "image"
|
||||||
|
if _PDF_HEADER in data[:_PDF_HEADER_WINDOW]:
|
||||||
|
return "pdf"
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def _upload_part(data: bytes) -> tuple[str, bytes, str]:
|
||||||
|
"""multipart 的 (filename, content, content_type)。
|
||||||
|
|
||||||
|
只有 PDF 改标 `doc.pdf`;**其余(含未知)一律沿用历史的 `image.jpg`**——服务对
|
||||||
|
PNG/TIFF/BMP 字节标 `image.jpg` 照样按内容解码(2026-10-21 实测),任何"更精确"
|
||||||
|
的改标都只会把现在能成功的路径变成失败(webp/gif 没有对应的服务端扩展名)。
|
||||||
|
"""
|
||||||
|
if _sniff_media(data) == "pdf":
|
||||||
|
return ("doc.pdf", data, "application/pdf")
|
||||||
|
return ("image.jpg", data, "image/jpeg")
|
||||||
|
|
||||||
|
|
||||||
def _default_client_factory(source: SourceConfig) -> httpx.AsyncClient:
|
def _default_client_factory(source: SourceConfig) -> httpx.AsyncClient:
|
||||||
return httpx.AsyncClient(
|
return httpx.AsyncClient(
|
||||||
@@ -52,11 +139,14 @@ def _default_client_factory(source: SourceConfig) -> httpx.AsyncClient:
|
|||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def _translate_http_errors(source_name: str, operation: str) -> Iterator[None]:
|
def _translate_http_errors(
|
||||||
|
source_name: str, operation: str, *, rejected_markers: tuple[str, ...] = ()
|
||||||
|
) -> Iterator[None]:
|
||||||
"""httpx 异常 → 错误四分类(CHS `_translate_http_errors` 语义主体保真)。
|
"""httpx 异常 → 错误四分类(CHS `_translate_http_errors` 语义主体保真)。
|
||||||
|
|
||||||
429 细分(insufficient_quota/Retry-After)有意放弃——MonkeyOCR 无鉴权
|
429 细分(insufficient_quota/Retry-After)有意放弃——MonkeyOCR 无鉴权
|
||||||
无计费(设计 §10.2);`from exc` 保留 __cause__ 供 RetryMW 归类 timeout。
|
无计费(设计 §10.2);`from exc` 保留 __cause__ 供 RetryMW 归类 timeout。
|
||||||
|
`rejected_markers` 非空时启用确定性坏输入窄化(仅文档路径传入,见 `_classify_status`)。
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
@@ -67,7 +157,7 @@ def _translate_http_errors(source_name: str, operation: str) -> Iterator[None]:
|
|||||||
operation=operation,
|
operation=operation,
|
||||||
) from exc
|
) from exc
|
||||||
except httpx.HTTPStatusError as exc:
|
except httpx.HTTPStatusError as exc:
|
||||||
raise _classify_status(exc, source_name, operation) from exc
|
raise _classify_status(exc, source_name, operation, rejected_markers) from exc
|
||||||
except httpx.HTTPError as exc:
|
except httpx.HTTPError as exc:
|
||||||
raise TransientError(
|
raise TransientError(
|
||||||
f"{source_name} OCR {operation} 网络错误: {exc}",
|
f"{source_name} OCR {operation} 网络错误: {exc}",
|
||||||
@@ -76,16 +166,31 @@ def _translate_http_errors(source_name: str, operation: str) -> Iterator[None]:
|
|||||||
) from exc
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _is_deterministic_failure(body: str, markers: tuple[str, ...]) -> bool:
|
||||||
|
"""响应体同时含 `Parsing failed:` 前缀与任一已知确定性标记。"""
|
||||||
|
text = body.lower()
|
||||||
|
return _PARSING_FAILED_PREFIX in text and any(marker in text for marker in markers)
|
||||||
|
|
||||||
|
|
||||||
def _classify_status(
|
def _classify_status(
|
||||||
exc: httpx.HTTPStatusError, source_name: str, operation: str
|
exc: httpx.HTTPStatusError,
|
||||||
|
source_name: str,
|
||||||
|
operation: str,
|
||||||
|
rejected_markers: tuple[str, ...] = (),
|
||||||
) -> TransientError | SourceDeadError | RequestRejectedError:
|
) -> TransientError | SourceDeadError | RequestRejectedError:
|
||||||
"""HTTP 状态码 → 错误四分类,**全部分支**携带响应体摘要(issue #10)。
|
"""HTTP 状态码 → 错误四分类,**全部分支**携带响应体摘要(issue #10)。
|
||||||
|
|
||||||
分类映射本身零变更;摘要口径与 chat 侧共用同一实现,不得在此另起一份——
|
分类映射本身零变更;摘要口径与 chat 侧共用同一实现,不得在此另起一份——
|
||||||
"只有一个分支用了响应体"正是 issue #10 的成因。
|
"只有一个分支用了响应体"正是 issue #10 的成因。
|
||||||
|
|
||||||
|
唯一的可选窄化: `rejected_markers` 非空且为 500 且命中确定性坏输入标记时,归
|
||||||
|
`RequestRejectedError`(**保留 `status_code=500`**: 服务有响应,熔断按「响应即健康」
|
||||||
|
记成功且不计窗口)。默认空元组 = 既有行为字节级不变——既有调用方对坏图的处置属公共行为,
|
||||||
|
推广须另行设计(issue #35)。
|
||||||
"""
|
"""
|
||||||
status = exc.response.status_code
|
status = exc.response.status_code
|
||||||
summary = summarize_body(response_body(exc.response))
|
body = response_body(exc.response)
|
||||||
|
summary = summarize_body(body)
|
||||||
ctx: dict[str, Any] = {
|
ctx: dict[str, Any] = {
|
||||||
"source_name": source_name,
|
"source_name": source_name,
|
||||||
"status_code": status,
|
"status_code": status,
|
||||||
@@ -93,6 +198,8 @@ def _classify_status(
|
|||||||
"body_text": summary,
|
"body_text": summary,
|
||||||
}
|
}
|
||||||
message = compose_message(f"{source_name} OCR {operation} HTTP {status}", summary)
|
message = compose_message(f"{source_name} OCR {operation} HTTP {status}", summary)
|
||||||
|
if status == 500 and rejected_markers and _is_deterministic_failure(body, rejected_markers):
|
||||||
|
return RequestRejectedError(message, **ctx)
|
||||||
if status >= 500 or status == 429:
|
if status >= 500 or status == 429:
|
||||||
return TransientError(message, **ctx)
|
return TransientError(message, **ctx)
|
||||||
if status in (401, 403):
|
if status in (401, 403):
|
||||||
@@ -100,30 +207,10 @@ def _classify_status(
|
|||||||
return RequestRejectedError(message, **ctx)
|
return RequestRejectedError(message, **ctx)
|
||||||
|
|
||||||
|
|
||||||
def _finite_number(value: object) -> float:
|
|
||||||
"""JSON 数值 → 有限浮点;布尔与非有限数拒绝(CHS invokers.py:427 逐字)。"""
|
|
||||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
|
||||||
raise ValueError("字段不是数值")
|
|
||||||
number = float(value)
|
|
||||||
if not math.isfinite(number):
|
|
||||||
raise ValueError("字段不是有限数值")
|
|
||||||
return number
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_page_size(page: dict) -> tuple[float, float]:
|
|
||||||
page_size = page.get("page_size")
|
|
||||||
if not isinstance(page_size, list) or len(page_size) != 2:
|
|
||||||
raise ValueError("page_size 无效")
|
|
||||||
width, height = (_finite_number(item) for item in page_size)
|
|
||||||
if width <= 0 or height <= 0:
|
|
||||||
raise ValueError("page_size 必须为正数")
|
|
||||||
return (width, height)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_bbox(bbox: object) -> tuple[float, float, float, float]:
|
def _parse_bbox(bbox: object) -> tuple[float, float, float, float]:
|
||||||
if not isinstance(bbox, list) or len(bbox) != 4:
|
if not isinstance(bbox, list) or len(bbox) != 4:
|
||||||
raise ValueError("bbox 无效")
|
raise ValueError("bbox 无效")
|
||||||
x1, y1, x2, y2 = (_finite_number(item) for item in bbox)
|
x1, y1, x2, y2 = (finite_number(item) for item in bbox)
|
||||||
if x2 <= x1 or y2 <= y1:
|
if x2 <= x1 or y2 <= y1:
|
||||||
raise ValueError("bbox 边界顺序无效")
|
raise ValueError("bbox 边界顺序无效")
|
||||||
# 库返回 float 原生 bbox;整数化退化校验专为 CHS shim 的 int() 裁剪路径
|
# 库返回 float 原生 bbox;整数化退化校验专为 CHS shim 的 int() 裁剪路径
|
||||||
@@ -152,9 +239,9 @@ def _parse_pages(payload: object) -> tuple[list[OcrLayoutElement], list[tuple[fl
|
|||||||
for fallback_index, page in enumerate(payload["pdf_info"]):
|
for fallback_index, page in enumerate(payload["pdf_info"]):
|
||||||
if not isinstance(page, dict):
|
if not isinstance(page, dict):
|
||||||
raise ValueError("页面结构无效")
|
raise ValueError("页面结构无效")
|
||||||
page_sizes.append(_parse_page_size(page))
|
page_sizes.append(parse_page_size(page))
|
||||||
raw_index = page.get("page_idx", fallback_index)
|
raw_index = page.get("page_idx", fallback_index)
|
||||||
# bool 是 int 子类、负页号无意义——与 _finite_number 拒 bool 的口径一致
|
# bool 是 int 子类、负页号无意义——与 finite_number 拒 bool 的口径一致
|
||||||
valid_idx = (
|
valid_idx = (
|
||||||
isinstance(raw_index, int) and not isinstance(raw_index, bool) and raw_index >= 0
|
isinstance(raw_index, int) and not isinstance(raw_index, bool) and raw_index >= 0
|
||||||
)
|
)
|
||||||
@@ -186,6 +273,64 @@ def _parse_middle_json(
|
|||||||
raise ResultInvalidError(f"OCR 结果包内容无效: {exc}") from exc
|
raise ResultInvalidError(f"OCR 结果包内容无效: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _check_download_url(download_url: str, source: SourceConfig) -> None:
|
||||||
|
"""`download_url` 来自服务端响应,属外部输入:畸形值先于第二段请求拒绝。
|
||||||
|
|
||||||
|
控制字符、坏方括号会让 httpx 抛 `InvalidURL`(不属 `HTTPError` 体系),越界端口会在连接阶段
|
||||||
|
抛 ExceptionGroup;二者都会从 `_translate_http_errors` 下漏出四分类,`OcrClient._call` 只接
|
||||||
|
`PolyGatewayError`,于是不写终态遥测行、持有的半开探针也不归还。服务返回畸形 URL 是协议故障,
|
||||||
|
与"响应缺少 download_url"同归瞬时错误。**不限制主机**(基线接受绝对地址,测试钉住)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
url = httpx.URL(download_url)
|
||||||
|
except httpx.InvalidURL as exc:
|
||||||
|
raise TransientError(
|
||||||
|
f"{source.name} /parse 响应的 download_url 非法: {exc}",
|
||||||
|
source_name=source.name,
|
||||||
|
operation="parse",
|
||||||
|
) from exc
|
||||||
|
if url.port is not None and not 0 < url.port <= 65535:
|
||||||
|
raise TransientError(
|
||||||
|
f"{source.name} /parse 响应的 download_url 端口越界: {url.port}",
|
||||||
|
source_name=source.name,
|
||||||
|
operation="parse",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _package_too_large(source: SourceConfig, size: int) -> ResultInvalidError:
|
||||||
|
return ResultInvalidError(
|
||||||
|
f"{source.name} OCR 结果包体积超过上限 {_MAX_PACKAGE_BYTES}(已读 {size})",
|
||||||
|
source_name=source.name,
|
||||||
|
operation="download_result",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _bounded_error_response(resp: httpx.Response) -> httpx.Response:
|
||||||
|
"""读错误响应体的前 `_ERROR_BODY_READ_CAP` 字节,复制成已缓冲的响应。"""
|
||||||
|
data = bytearray()
|
||||||
|
async for chunk in resp.aiter_bytes():
|
||||||
|
data += chunk
|
||||||
|
if len(data) >= _ERROR_BODY_READ_CAP:
|
||||||
|
break
|
||||||
|
headers = {k: v for k, v in resp.headers.items() if k.lower() == "content-type"}
|
||||||
|
return httpx.Response(
|
||||||
|
resp.status_code,
|
||||||
|
headers=headers,
|
||||||
|
content=bytes(data[:_ERROR_BODY_READ_CAP]),
|
||||||
|
request=resp.request,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_declared_length(resp: httpx.Response, source: SourceConfig) -> None:
|
||||||
|
"""声明长度超限就不必读了;缺失/非法的头部不作判据(靠累计字节兜底)。"""
|
||||||
|
try:
|
||||||
|
declared = int(resp.headers.get("content-length", ""))
|
||||||
|
except ValueError:
|
||||||
|
return
|
||||||
|
if declared > _MAX_PACKAGE_BYTES:
|
||||||
|
raise _package_too_large(source, declared)
|
||||||
|
|
||||||
|
|
||||||
class MonkeyOcrTransport:
|
class MonkeyOcrTransport:
|
||||||
"""MonkeyOCR transport: 每源一个预配 httpx client,懒创建,aclose 统一释放。"""
|
"""MonkeyOCR transport: 每源一个预配 httpx client,懒创建,aclose 统一释放。"""
|
||||||
|
|
||||||
@@ -193,8 +338,11 @@ class MonkeyOcrTransport:
|
|||||||
self,
|
self,
|
||||||
*,
|
*,
|
||||||
client_factory: Callable[[SourceConfig], httpx.AsyncClient] | None = None,
|
client_factory: Callable[[SourceConfig], httpx.AsyncClient] | None = None,
|
||||||
|
deterministic_failure_markers: tuple[str, ...] = _DEFAULT_REJECTED_MARKERS,
|
||||||
) -> None:
|
) -> None:
|
||||||
self._client_factory = client_factory or _default_client_factory
|
self._client_factory = client_factory or _default_client_factory
|
||||||
|
# 仅文档路径使用(见 `_classify_status`);recognize_text / parse_layout 不传
|
||||||
|
self._rejected_markers = deterministic_failure_markers
|
||||||
self._clients: dict[str, httpx.AsyncClient] = {}
|
self._clients: dict[str, httpx.AsyncClient] = {}
|
||||||
|
|
||||||
def _client_for(self, source: SourceConfig) -> httpx.AsyncClient:
|
def _client_for(self, source: SourceConfig) -> httpx.AsyncClient:
|
||||||
@@ -205,9 +353,10 @@ class MonkeyOcrTransport:
|
|||||||
return client
|
return client
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _files(image: bytes) -> dict[str, tuple[str, bytes, str]]:
|
def _files(data: bytes) -> dict[str, tuple[str, bytes, str]]:
|
||||||
# 固定文件名/类型(CHS invokers.py:496 原样;服务按内容处理,35 样本实测)
|
# 标签按内容判定(#34): 非 PDF 沿用 CHS invokers.py:496 的 image.jpg,
|
||||||
return {"file": ("image.jpg", image, "image/jpeg")}
|
# PDF 改标 doc.pdf——服务按扩展名分派,固定名对 PDF 是错的
|
||||||
|
return {"file": _upload_part(data)}
|
||||||
|
|
||||||
def _response_json(self, resp: httpx.Response, source: SourceConfig, operation: str) -> dict:
|
def _response_json(self, resp: httpx.Response, source: SourceConfig, operation: str) -> dict:
|
||||||
"""响应体 JSON 化;坏 JSON/非 dict → Transient(设计 §4: 服务侧瞬时异常)。"""
|
"""响应体 JSON 化;坏 JSON/非 dict → Transient(设计 §4: 服务侧瞬时异常)。"""
|
||||||
@@ -249,32 +398,48 @@ class MonkeyOcrTransport:
|
|||||||
text=content, raw={k: v for k, v in data.items() if k != "content"}
|
text=content, raw={k: v for k, v in data.items() if k != "content"}
|
||||||
)
|
)
|
||||||
|
|
||||||
async def parse_layout(
|
async def _post_parse(
|
||||||
self, *, image: bytes, source: SourceConfig, call_id: str
|
self,
|
||||||
) -> OcrLayoutTransportResult:
|
client: httpx.AsyncClient,
|
||||||
"""两段协议: POST /parse → GET download_url → ZIP → 全元素提取。"""
|
data: bytes,
|
||||||
del call_id
|
source: SourceConfig,
|
||||||
client = self._client_for(source)
|
rejected_markers: tuple[str, ...],
|
||||||
with _translate_http_errors(source.name, "parse"):
|
) -> str:
|
||||||
resp = await client.post("/parse", files=self._files(image))
|
"""两段协议的第一段: POST /parse → 校验 `success` → 返回 download_url。
|
||||||
|
|
||||||
|
layout 与 document 两条路径共用,保证"success 为假 → 附 status_code=200"等
|
||||||
|
设计 §10.2 的有意修复只有一份。
|
||||||
|
"""
|
||||||
|
with _translate_http_errors(source.name, "parse", rejected_markers=rejected_markers):
|
||||||
|
resp = await client.post("/parse", files=self._files(data))
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
data = self._response_json(resp, source, "parse")
|
body = self._response_json(resp, source, "parse")
|
||||||
if data.get("success") is not True:
|
if body.get("success") is not True:
|
||||||
# 设计 §10.2 有意修复: 200 响应确证服务活着,附 status_code
|
# 设计 §10.2 有意修复: 200 响应确证服务活着,附 status_code
|
||||||
# 让治理侧按"响应即健康"记熔断成功(CHS 原样会误走本地拒绝分支)
|
# 让治理侧按"响应即健康"记熔断成功(CHS 原样会误走本地拒绝分支)
|
||||||
raise RequestRejectedError(
|
raise RequestRejectedError(
|
||||||
f"{source.name} OCR 解析失败: {data.get('message')}",
|
f"{source.name} OCR 解析失败: {body.get('message')}",
|
||||||
source_name=source.name,
|
source_name=source.name,
|
||||||
status_code=resp.status_code,
|
status_code=resp.status_code,
|
||||||
operation="parse",
|
operation="parse",
|
||||||
)
|
)
|
||||||
download_url = data.get("download_url")
|
download_url = body.get("download_url")
|
||||||
if not isinstance(download_url, str) or not download_url.strip():
|
if not isinstance(download_url, str) or not download_url.strip():
|
||||||
raise TransientError(
|
raise TransientError(
|
||||||
f"{source.name} /parse 响应缺少 download_url",
|
f"{source.name} /parse 响应缺少 download_url",
|
||||||
source_name=source.name,
|
source_name=source.name,
|
||||||
operation="parse",
|
operation="parse",
|
||||||
)
|
)
|
||||||
|
_check_download_url(download_url, source)
|
||||||
|
return download_url
|
||||||
|
|
||||||
|
async def parse_layout(
|
||||||
|
self, *, image: bytes, source: SourceConfig, call_id: str
|
||||||
|
) -> OcrLayoutTransportResult:
|
||||||
|
"""两段协议: POST /parse → GET download_url → ZIP → 全元素提取。"""
|
||||||
|
del call_id
|
||||||
|
client = self._client_for(source)
|
||||||
|
download_url = await self._post_parse(client, image, source, ())
|
||||||
with _translate_http_errors(source.name, "download_result"):
|
with _translate_http_errors(source.name, "download_result"):
|
||||||
# 实测 download_url 为相对路径(/static/*.zip),相对 base_url 解析;
|
# 实测 download_url 为相对路径(/static/*.zip),相对 base_url 解析;
|
||||||
# httpx 对绝对 URL 会忽略 base_url,两种形态天然兼容
|
# httpx 对绝对 URL 会忽略 base_url,两种形态天然兼容
|
||||||
@@ -287,6 +452,59 @@ class MonkeyOcrTransport:
|
|||||||
raw={"success": True, "download_url": download_url},
|
raw={"success": True, "download_url": download_url},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def parse_document(
|
||||||
|
self, *, document: bytes, source: SourceConfig, call_id: str
|
||||||
|
) -> OcrDocumentTransportResult:
|
||||||
|
"""整份文档: 本地预检 → POST /parse → 流式 GET → 线程内解析(#34 设计 §5)。"""
|
||||||
|
del call_id
|
||||||
|
if _sniff_media(document) == "unknown":
|
||||||
|
# 服务从未被打扰: HTML 付费墙页面、乱码、空串在批处理语料里最常见。不带
|
||||||
|
# status_code(服务没响应),治理侧只归还探针、不计熔断成败
|
||||||
|
raise RequestRejectedError(
|
||||||
|
f"{source.name} OCR 无法识别文档格式(既非 PDF 也非已知图片),拒绝发送",
|
||||||
|
source_name=source.name,
|
||||||
|
operation="parse",
|
||||||
|
)
|
||||||
|
client = self._client_for(source)
|
||||||
|
download_url = await self._post_parse(client, document, source, self._rejected_markers)
|
||||||
|
package = await self._download_package(client, download_url, source)
|
||||||
|
# 纯函数、可能处理大对象: 放线程里,取消与 call_deadline_s 才能及时生效
|
||||||
|
elements, page_sizes, markdown = await asyncio.to_thread(parse_document_package, package)
|
||||||
|
return OcrDocumentTransportResult(
|
||||||
|
elements=elements,
|
||||||
|
page_sizes=page_sizes,
|
||||||
|
markdown=markdown,
|
||||||
|
raw={"success": True, "download_url": download_url},
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _download_package(
|
||||||
|
self, client: httpx.AsyncClient, download_url: str, source: SourceConfig
|
||||||
|
) -> bytes:
|
||||||
|
"""流式下载结果包并限制体积。
|
||||||
|
|
||||||
|
**先处理错误状态,再检查体积**: 结果包的资源检查不得改变 HTTP 错误分类
|
||||||
|
(一个声明超限的 401 仍是 SourceDead,响应体摘要也要保留——issue #10)。
|
||||||
|
"""
|
||||||
|
chunks: list[bytes] = []
|
||||||
|
total = 0
|
||||||
|
with _translate_http_errors(source.name, "download_result"):
|
||||||
|
async with client.stream("GET", download_url) as resp:
|
||||||
|
if not resp.is_success:
|
||||||
|
# 条件是「非 2xx」而不是 `is_error`:1xx/3xx 同样需要保留摘要。错误体只读
|
||||||
|
# 有界的前缀并复制成已缓冲的响应(摘要口径只认已缓冲的响应体)
|
||||||
|
raise httpx.HTTPStatusError(
|
||||||
|
f"HTTP {resp.status_code}",
|
||||||
|
request=resp.request,
|
||||||
|
response=await _bounded_error_response(resp),
|
||||||
|
)
|
||||||
|
_check_declared_length(resp, source)
|
||||||
|
async for chunk in resp.aiter_bytes():
|
||||||
|
total += len(chunk)
|
||||||
|
if total > _MAX_PACKAGE_BYTES:
|
||||||
|
raise _package_too_large(source, total)
|
||||||
|
chunks.append(chunk)
|
||||||
|
return b"".join(chunks)
|
||||||
|
|
||||||
async def check_health(self, *, source: SourceConfig) -> bool:
|
async def check_health(self, *, source: SourceConfig) -> bool:
|
||||||
"""GET /health 探测: 2xx → True,失败 → False,取消穿透(R10)。"""
|
"""GET /health 探测: 2xx → True,失败 → False,取消穿透(R10)。"""
|
||||||
client = self._client_for(source)
|
client = self._client_for(source)
|
||||||
|
|||||||
@@ -276,13 +276,19 @@ class ThinkingObservation(StrEnum):
|
|||||||
UNKNOWN = "unknown"
|
UNKNOWN = "unknown"
|
||||||
|
|
||||||
|
|
||||||
CallOperation = Literal["chat", "embed", "recognize_text", "parse_layout"]
|
CallOperation = Literal["chat", "embed", "recognize_text", "parse_layout", "parse_document"]
|
||||||
"""遥测 `operation` 列的值域: **公开方法**四值,由调用点给定。
|
"""遥测 `operation` 列的值域: **公开方法**五值,由调用点给定。
|
||||||
|
|
||||||
与 `PolyGatewayError.operation`(HTTP 子操作,如 `download_result`)是**两个语义**,
|
与 `PolyGatewayError.operation`(HTTP 子操作,如 `download_result`)是**两个语义**,
|
||||||
不做自动转换;链路上任何位置都不得读 `exc.operation` 来填本列(设计 §5 I1/I2)。"""
|
不做自动转换;链路上任何位置都不得读 `exc.operation` 来填本列(设计 §5 I1/I2)。"""
|
||||||
|
|
||||||
CALL_OPERATIONS: tuple[CallOperation, ...] = ("chat", "embed", "recognize_text", "parse_layout")
|
CALL_OPERATIONS: tuple[CallOperation, ...] = (
|
||||||
|
"chat",
|
||||||
|
"embed",
|
||||||
|
"recognize_text",
|
||||||
|
"parse_layout",
|
||||||
|
"parse_document",
|
||||||
|
)
|
||||||
|
|
||||||
EventKind = Literal["attempt", "cache_hit", "terminal_failure"]
|
EventKind = Literal["attempt", "cache_hit", "terminal_failure"]
|
||||||
"""一行遥测描述的事件形态;旧行 NULL,不回填。
|
"""一行遥测描述的事件形态;旧行 NULL,不回填。
|
||||||
@@ -1189,6 +1195,10 @@ class OcrLayoutElement:
|
|||||||
|
|
||||||
type 为开放字符串(实测 table/image/text,不枚举锁死——零业务假设);
|
type 为开放字符串(实测 table/image/text,不枚举锁死——零业务假设);
|
||||||
bbox 为 OCR 原生页面坐标 (x1, y1, x2, y2),几何映射留业务侧(D9)。
|
bbox 为 OCR 原生页面坐标 (x1, y1, x2, y2),几何映射留业务侧(D9)。
|
||||||
|
|
||||||
|
注意: `OcrDocumentElement` 继承本类并追加**无默认值**的 `text`。dataclass 要求
|
||||||
|
无默认字段不得排在带默认字段之后,故日后若给本类追加带默认值的字段,必须先评估
|
||||||
|
子类——否则 `OcrDocumentElement` 会在导入期抛 `TypeError`(#34 设计 §4.1)。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
type: str
|
type: str
|
||||||
@@ -1196,6 +1206,22 @@ class OcrLayoutElement:
|
|||||||
page_index: int
|
page_index: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentElement(OcrLayoutElement):
|
||||||
|
"""文档解析的逐块结果(#34 设计 §4.1): 版面元素 + 该块的文字内容。
|
||||||
|
|
||||||
|
`text` 的含义随 `type`,由 `parse_document` 的解析规则固定:
|
||||||
|
`text`/`title` 为纯文本(行内公式保留 `$…$`);`interline_equation` 为裸 LaTeX
|
||||||
|
(无 `$$` 定界);`table` 为 HTML 表体;`image` 为空串。服务将来若给出带 caption
|
||||||
|
/footnote 的嵌套子块,它们作为独立元素紧随父块输出,`type` 取子块类型。
|
||||||
|
|
||||||
|
继承而非组合: 文档元素"是"版面元素加内容,按 bbox 裁剪的下游函数可同时处理两者。
|
||||||
|
代价是 dataclass 的 `__eq__` 要求同类,`OcrDocumentElement != OcrLayoutElement`。
|
||||||
|
"""
|
||||||
|
|
||||||
|
text: str
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class OcrTextResult:
|
class OcrTextResult:
|
||||||
"""一次治理 OCR 文本转录的统一响应(/ocr/text;M3 设计 §3.1)。
|
"""一次治理 OCR 文本转录的统一响应(/ocr/text;M3 设计 §3.1)。
|
||||||
@@ -1232,6 +1258,31 @@ class OcrLayoutResult:
|
|||||||
"""本次逻辑调用的统计快照(1.3.5);`None` = 未知。"""
|
"""本次逻辑调用的统计快照(1.3.5);`None` = 未知。"""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentResult:
|
||||||
|
"""一次治理整份文档解析的统一响应(/parse → ZIP;#34 设计 §4.1)。
|
||||||
|
|
||||||
|
**独立定义,不继承 `OcrLayoutResult`**: 父类末尾的 `call_stats` 带默认值,
|
||||||
|
子类追加无默认字段 `markdown` 会在导入期 `TypeError`。
|
||||||
|
|
||||||
|
elements 按服务阅读序排列,页内按页序;page_sizes 按 page_index 索引(零基)。
|
||||||
|
`markdown` 是服务生成的整篇 Markdown 原文,**含悬空的 ``
|
||||||
|
图片引用**——库不外传图片资源。服务端会丢弃脚注、页眉页脚与页码,这些内容
|
||||||
|
既不在 elements 也不在 markdown 里,库无从补偿。
|
||||||
|
"""
|
||||||
|
|
||||||
|
elements: list[OcrDocumentElement]
|
||||||
|
page_sizes: list[tuple[float, float]]
|
||||||
|
markdown: str
|
||||||
|
source_name: str
|
||||||
|
usage: Usage
|
||||||
|
latency_ms: int
|
||||||
|
call_id: str
|
||||||
|
raw: dict[str, Any]
|
||||||
|
call_stats: CallStats | None = None
|
||||||
|
"""本次逻辑调用的统计快照(1.3.5);`None` = 未知。"""
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class OcrTextTransportResult:
|
class OcrTextTransportResult:
|
||||||
"""transport 单次 /ocr/text 调用产物;治理字段由 OcrClient 补齐。"""
|
"""transport 单次 /ocr/text 调用产物;治理字段由 OcrClient 补齐。"""
|
||||||
@@ -1249,6 +1300,16 @@ class OcrLayoutTransportResult:
|
|||||||
raw: dict[str, Any]
|
raw: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OcrDocumentTransportResult:
|
||||||
|
"""transport 单次整份文档两段调用的产物;治理字段由 OcrClient 补齐。"""
|
||||||
|
|
||||||
|
elements: list[OcrDocumentElement]
|
||||||
|
page_sizes: list[tuple[float, float]]
|
||||||
|
markdown: str
|
||||||
|
raw: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class EmbeddingTransportResult:
|
class EmbeddingTransportResult:
|
||||||
"""一次原始 embedding 调用的解析结果(M2 设计 §7.2;transport → client)。"""
|
"""一次原始 embedding 调用的解析结果(M2 设计 §7.2;transport → client)。"""
|
||||||
|
|||||||
@@ -0,0 +1,135 @@
|
|||||||
|
"""MonkeyOCR 整份文档活体验收(#34 计划 T8)。
|
||||||
|
|
||||||
|
**成败取决于外部服务当下状态**,按 CLAUDE.md §4.6 一律标 `slow`: 默认套件不收集,发布清单
|
||||||
|
第 4 步按 diff 交集选跑(`pytest tests/integration/test_monkey_ocr_document_live.py -m slow`)。
|
||||||
|
服务与其它业务线共用——所以每条用例只发**单份极小文档**、串行、不压测。
|
||||||
|
|
||||||
|
文档字节由本文件手写构造(最小合法 PDF,无第三方内容、无新依赖)。
|
||||||
|
不依赖 `data/soak` 图片语料,故**不**并入带语料 `skipif` 的 `test_monkey_ocr_live.py`。
|
||||||
|
服务不可达即 FAIL——验收必打真实,不 skip。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from polygateway.errors import RequestRejectedError
|
||||||
|
from polygateway.ocr import OcrClient
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.slow
|
||||||
|
|
||||||
|
_PRIMARY = "http://10.77.0.20:7866"
|
||||||
|
_MARKER = "POLYGATEWAY SYNTHETIC SAMPLE"
|
||||||
|
_OUTPUTS = Path("tests/outputs/monkey_ocr_document")
|
||||||
|
|
||||||
|
|
||||||
|
def _minimal_pdf(text: str) -> bytes:
|
||||||
|
"""5 个对象的最小合法单页 PDF;xref 偏移按实际字节计算。`text` 不得含括号。"""
|
||||||
|
stream = f"BT /F1 28 Tf 40 700 Td ({text}) Tj ET".encode()
|
||||||
|
objects = [
|
||||||
|
b"<< /Type /Catalog /Pages 2 0 R >>",
|
||||||
|
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
|
||||||
|
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R "
|
||||||
|
b"/Resources << /Font << /F1 5 0 R >> >> >>",
|
||||||
|
b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream",
|
||||||
|
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
|
||||||
|
]
|
||||||
|
out = bytearray(b"%PDF-1.4\n")
|
||||||
|
offsets = []
|
||||||
|
for number, body in enumerate(objects, start=1):
|
||||||
|
offsets.append(len(out))
|
||||||
|
out += b"%d 0 obj\n" % number + body + b"\nendobj\n"
|
||||||
|
xref_at = len(out)
|
||||||
|
out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objects) + 1)
|
||||||
|
for offset in offsets:
|
||||||
|
out += b"%010d 00000 n \n" % offset
|
||||||
|
out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (
|
||||||
|
len(objects) + 1,
|
||||||
|
xref_at,
|
||||||
|
)
|
||||||
|
return bytes(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _env(**extra: str) -> dict[str, str]:
|
||||||
|
env = {
|
||||||
|
"LLM_MAX_RETRIES": "3",
|
||||||
|
"LLM_RETRY_BASE_DELAY": "1.0",
|
||||||
|
"LLM_RETRY_MAX_DELAY": "5.0",
|
||||||
|
# 阈值 2: 坏文档若被误记成源故障,两份就足以开路——下面第三条用例据此判别
|
||||||
|
"LLM_CIRCUIT_BREAKER_THRESHOLD": "2",
|
||||||
|
"LLM_CIRCUIT_BREAKER_COOLDOWN": "30",
|
||||||
|
"PGW_CACHE_BACKEND": "none",
|
||||||
|
"PGW_TELEMETRY_BACKEND": "none",
|
||||||
|
"OCR__MONKEY__1__BASE_URL": _PRIMARY,
|
||||||
|
"OCR__MONKEY__1__API_KEY": "none",
|
||||||
|
"OCR__MONKEY__1__MODEL": "monkey-ocr",
|
||||||
|
"OCR__MONKEY__1__TIMEOUT_S": "300",
|
||||||
|
"OCR__MONKEY__1__MAX_CONCURRENCY": "1", # 与其它业务线共用,串行
|
||||||
|
# 服务在 LAN,开发机的系统代理会对内网地址回 403
|
||||||
|
"OCR__MONKEY__1__TRUST_ENV": "false",
|
||||||
|
}
|
||||||
|
env.update(extra)
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
def _record(name: str, lines: list[str]) -> None:
|
||||||
|
"""结构化 Markdown 证据(CLAUDE.md §4.6);tests/outputs 不提交。"""
|
||||||
|
_OUTPUTS.mkdir(parents=True, exist_ok=True)
|
||||||
|
stamp = time.strftime("%Y%m%d_%H%M%S")
|
||||||
|
(_OUTPUTS / f"{name}_{stamp}.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
class TestLiveDocument:
|
||||||
|
async def test_parse_document_returns_pages_and_text(self):
|
||||||
|
async with OcrClient.from_env("OCR", env=_env()) as client:
|
||||||
|
result = await client.parse_document(_minimal_pdf(_MARKER))
|
||||||
|
assert len(result.page_sizes) >= 1 and all(w > 0 and h > 0 for w, h in result.page_sizes)
|
||||||
|
assert result.elements, "合成 PDF 应至少检出一个块"
|
||||||
|
# 正向断言: 空串不能通过。OCR 对大号印刷体应稳定识别出标记词
|
||||||
|
assert "SYNTHETIC" in result.markdown.upper()
|
||||||
|
assert any("SYNTHETIC" in e.text.upper() for e in result.elements)
|
||||||
|
assert all(e.page_index == 0 for e in result.elements)
|
||||||
|
_record(
|
||||||
|
"parse_document",
|
||||||
|
[
|
||||||
|
"# parse_document 活体",
|
||||||
|
f"- source: {result.source_name} latency_ms: {result.latency_ms}",
|
||||||
|
f"- pages: {len(result.page_sizes)} elements: {len(result.elements)}",
|
||||||
|
f"- types: {sorted({e.type for e in result.elements})}",
|
||||||
|
f"- markdown: {result.markdown[:200]!r}",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
async def test_recognize_text_on_pdf_is_not_the_hallucination(self):
|
||||||
|
"""E4: PDF 以 image.jpg 上传时 `/ocr/text` 返回 success 的幻觉文本。标签修正后应返回全文。"""
|
||||||
|
async with OcrClient.from_env("OCR", env=_env()) as client:
|
||||||
|
result = await client.recognize_text(_minimal_pdf(_MARKER))
|
||||||
|
assert result.text.strip()
|
||||||
|
assert "single character" not in result.text.lower()
|
||||||
|
assert "SYNTHETIC" in result.text.upper()
|
||||||
|
_record(
|
||||||
|
"recognize_text_pdf", ["# recognize_text(PDF) 活体", f"- text: {result.text[:200]!r}"]
|
||||||
|
)
|
||||||
|
|
||||||
|
async def test_corrupt_documents_do_not_open_the_breaker(self):
|
||||||
|
"""确定性坏输入归「请求被拒」,熔断记成功不计窗口。
|
||||||
|
|
||||||
|
阈值 2: 若坏文档被误记成源故障(旧行为: 500 → Transient → 重试并计熔断),
|
||||||
|
连续两份就会开路,第三次的有效文档会被 `fail_fast` 拒绝。
|
||||||
|
"""
|
||||||
|
corrupt = [b"%PDF-1.4\nnot really a pdf\n", b"%PDF-1.4\n" + b"garbage" * 50]
|
||||||
|
async with OcrClient.from_env("OCR", env=_env()) as client:
|
||||||
|
for bad in corrupt:
|
||||||
|
with pytest.raises(RequestRejectedError) as ei:
|
||||||
|
await client.parse_document(bad)
|
||||||
|
assert ei.value.status_code == 500 # 服务有响应: 按「响应即健康」记成功
|
||||||
|
result = await client.parse_document(_minimal_pdf(_MARKER))
|
||||||
|
assert result.elements
|
||||||
|
_record(
|
||||||
|
"corrupt_documents",
|
||||||
|
[
|
||||||
|
"# 坏文档不开熔断 活体",
|
||||||
|
f"- 两份坏文档均 RequestRejected(500); 随后有效文档成功, elements={len(result.elements)}",
|
||||||
|
],
|
||||||
|
)
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
"""OCR 文档结果包的合成 fixtures(#34)。
|
||||||
|
|
||||||
|
全部**合成**: 保留 2026-10-21 实测的结构骨架(para_blocks 块类型、table/image 的嵌套
|
||||||
|
`blocks`、span 字段),一切识别文本替换为 `SYNTH-*` 占位——第三方文档内容许可不明、
|
||||||
|
敏感数据不入库(零业务假设 + P5),真实 ZIP 不提交。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
_BBOX = (10.0, 20.0, 110.0, 40.0)
|
||||||
|
|
||||||
|
|
||||||
|
def _span(content="SYNTH-TEXT", type_="text"):
|
||||||
|
return {"type": type_, "content": content, "bbox": list(_BBOX), "score": 1.0}
|
||||||
|
|
||||||
|
|
||||||
|
def _text_block(content="SYNTH-TEXT", *, type_="text", bbox=_BBOX):
|
||||||
|
return {
|
||||||
|
"type": type_,
|
||||||
|
"bbox": list(bbox),
|
||||||
|
"index": 0,
|
||||||
|
"lines": [{"bbox": list(bbox), "spans": [_span(content)]}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _body(type_, spans, bbox=_BBOX):
|
||||||
|
return {"type": type_, "bbox": list(bbox), "lines": [{"bbox": list(bbox), "spans": spans}]}
|
||||||
|
|
||||||
|
|
||||||
|
def _table_block(html="<table><tr><td>SYNTH-CELL</td></tr></table>", extra_subs=()):
|
||||||
|
body = _body("table_body", [{"type": "table", "html": html, "image_path": "synth.jpg"}])
|
||||||
|
return {"type": "table", "bbox": list(_BBOX), "index": 1, "blocks": [*extra_subs, body]}
|
||||||
|
|
||||||
|
|
||||||
|
def _image_block():
|
||||||
|
body = _body("image_body", [{"type": "image", "image_path": "synth.jpg"}])
|
||||||
|
return {"type": "image", "bbox": list(_BBOX), "index": 2, "blocks": [body]}
|
||||||
|
|
||||||
|
|
||||||
|
def _page(blocks=None, *, page_idx=0, page_size=(612.0, 792.0), **extra):
|
||||||
|
page = {"page_idx": page_idx, "page_size": list(page_size), "discarded_blocks": []}
|
||||||
|
if blocks is not None:
|
||||||
|
page["para_blocks"] = blocks
|
||||||
|
page.update(extra)
|
||||||
|
return page
|
||||||
|
|
||||||
|
|
||||||
|
def _package(
|
||||||
|
pages=None,
|
||||||
|
*,
|
||||||
|
md="# synth\n",
|
||||||
|
middle=None,
|
||||||
|
middle_name="doc_middle.json",
|
||||||
|
md_name=None,
|
||||||
|
extra=(),
|
||||||
|
):
|
||||||
|
"""组装结果包。`middle` 给原始 bytes 时原样写入;`md_name` 默认由 middle 名配对。"""
|
||||||
|
if middle is None:
|
||||||
|
middle = json.dumps({"pdf_info": pages if pages is not None else [_page([])]}).encode()
|
||||||
|
if md_name is None:
|
||||||
|
md_name = middle_name[: -len("_middle.json")] + ".md"
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(buf, "w") as zf:
|
||||||
|
zf.writestr(middle_name, middle)
|
||||||
|
if md is not None:
|
||||||
|
zf.writestr(md_name, md if isinstance(md, bytes) else md.encode())
|
||||||
|
for name, data in extra:
|
||||||
|
zf.writestr(name, data)
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def _encrypted_package() -> bytes:
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(buf, "w") as zf:
|
||||||
|
info = zipfile.ZipInfo("doc_middle.json")
|
||||||
|
info.flag_bits |= 0x1 # 加密位:读取时 zipfile 要求密码
|
||||||
|
zf.writestr(info, b'{"pdf_info": []}')
|
||||||
|
zf.writestr("doc.md", "x")
|
||||||
|
return buf.getvalue()
|
||||||
@@ -0,0 +1,590 @@
|
|||||||
|
"""MonkeyOCR 文档结果包解析测试(#34 设计 §5/§5.1)。
|
||||||
|
|
||||||
|
fixtures 全部**合成**: 保留 2026-10-21 实测的结构骨架(para_blocks 的块类型、
|
||||||
|
table/image 的嵌套 `blocks`、span 字段),一切识别文本替换为 `SYNTH-*` 占位——
|
||||||
|
第三方文档内容许可不明、敏感数据不入库(零业务假设 + P5),真实 ZIP 不提交。
|
||||||
|
|
||||||
|
契约表逐行一条,参数化列明各分支。解析函数是 bytes → 结果的纯函数,失败一律
|
||||||
|
`ResultInvalidError`,**不得漏出原生异常**(漏出会让 `OcrClient._call` 既不写终态遥测
|
||||||
|
行也不归还半开探针)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import random
|
||||||
|
import struct
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from polygateway.errors import ResultInvalidError
|
||||||
|
from polygateway.transports import _monkey_document
|
||||||
|
from polygateway.transports._monkey_document import parse_document_package
|
||||||
|
from polygateway.types import OcrDocumentElement
|
||||||
|
from tests.unit._document_fixtures import (
|
||||||
|
_BBOX,
|
||||||
|
_body,
|
||||||
|
_encrypted_package,
|
||||||
|
_image_block,
|
||||||
|
_package,
|
||||||
|
_page,
|
||||||
|
_span,
|
||||||
|
_table_block,
|
||||||
|
_text_block,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse(pages=None, **kwargs):
|
||||||
|
return parse_document_package(_package(pages, **kwargs))
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_rejected_on_purpose(exc: ResultInvalidError) -> None:
|
||||||
|
"""拒绝必须出自显式校验,不能是 Python 自己绊倒后被外层边界吸收。
|
||||||
|
|
||||||
|
外层边界会把 TypeError/KeyError 也归为 ResultInvalidError,所以只断言异常类型,
|
||||||
|
「显式校验」与「碰巧在别处失败」无法区分(变异检查证实过: 删掉校验后测试仍绿)。
|
||||||
|
原因链里出现 TypeError/KeyError 即说明有校验缺口。
|
||||||
|
"""
|
||||||
|
assert not isinstance(exc.__cause__, TypeError | KeyError), repr(exc.__cause__)
|
||||||
|
|
||||||
|
|
||||||
|
def _invalid(pages=None, **kwargs):
|
||||||
|
with pytest.raises(ResultInvalidError) as ei:
|
||||||
|
_parse(pages, **kwargs)
|
||||||
|
_assert_rejected_on_purpose(ei.value)
|
||||||
|
|
||||||
|
|
||||||
|
class TestHappyPath:
|
||||||
|
def test_every_block_type_yields_text_per_contract(self):
|
||||||
|
pages = [
|
||||||
|
_page(
|
||||||
|
[
|
||||||
|
_text_block("SYNTH-TITLE", type_="title"),
|
||||||
|
_text_block("SYNTH-BODY"),
|
||||||
|
_text_block("x^{2}", type_="interline_equation"),
|
||||||
|
_table_block("<table>SYNTH</table>"),
|
||||||
|
_image_block(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
]
|
||||||
|
elements, page_sizes, markdown = _parse(pages)
|
||||||
|
assert [(e.type, e.text) for e in elements] == [
|
||||||
|
("title", "SYNTH-TITLE"),
|
||||||
|
("text", "SYNTH-BODY"),
|
||||||
|
("interline_equation", "x^{2}"),
|
||||||
|
("table", "<table>SYNTH</table>"),
|
||||||
|
("image", ""),
|
||||||
|
]
|
||||||
|
assert all(isinstance(e, OcrDocumentElement) for e in elements)
|
||||||
|
assert all(e.bbox == _BBOX and e.page_index == 0 for e in elements)
|
||||||
|
assert page_sizes == [(612.0, 792.0)]
|
||||||
|
assert markdown == "# synth\n"
|
||||||
|
|
||||||
|
def test_reading_order_and_page_index_across_pages(self):
|
||||||
|
pages = [
|
||||||
|
_page([_text_block("A")], page_idx=0),
|
||||||
|
_page([_text_block("B"), _text_block("C")], page_idx=1, page_size=(595.0, 842.0)),
|
||||||
|
]
|
||||||
|
elements, page_sizes, _ = _parse(pages)
|
||||||
|
assert [(e.page_index, e.text) for e in elements] == [(0, "A"), (1, "B"), (1, "C")]
|
||||||
|
assert page_sizes == [(612.0, 792.0), (595.0, 842.0)]
|
||||||
|
|
||||||
|
def test_content_list_and_images_are_ignored(self):
|
||||||
|
"""内容只来自 middle: content_list 把 `$` 转义成 `\\$`(展示格式),取它会污染 LaTeX。"""
|
||||||
|
garbage = json.dumps([{"type": "text", "text": "GARBAGE \\$"}]).encode()
|
||||||
|
elements, _, _ = parse_document_package(
|
||||||
|
_package(
|
||||||
|
[_page([_text_block("SYNTH $x$")])],
|
||||||
|
extra=[("doc_content_list.json", garbage), ("images/x.jpg", b"\xff\xd8")],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert [e.text for e in elements] == ["SYNTH $x$"]
|
||||||
|
|
||||||
|
def test_discarded_blocks_are_not_in_the_result(self):
|
||||||
|
pages = [_page([_text_block("KEEP")], discarded_blocks=[_text_block("DROPPED")])]
|
||||||
|
elements, _, _ = _parse(pages)
|
||||||
|
assert [e.text for e in elements] == ["KEEP"]
|
||||||
|
|
||||||
|
def test_need_drop_pages_are_neither_read_nor_rejected(self):
|
||||||
|
pages = [_page([_text_block("KEEP")], need_drop=True, drop_reason=["synthetic"])]
|
||||||
|
elements, _, _ = _parse(pages)
|
||||||
|
assert [e.text for e in elements] == ["KEEP"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestTextAssembly:
|
||||||
|
def test_multiple_lines_and_spans_are_joined(self):
|
||||||
|
"""样本全是 1 行 1 span;多行/多 span 路径未经实测,只由构造 fixture 钉住拼接规则。"""
|
||||||
|
block = {
|
||||||
|
"type": "text",
|
||||||
|
"bbox": list(_BBOX),
|
||||||
|
"lines": [
|
||||||
|
{"spans": [_span("A"), _span("B")]},
|
||||||
|
{"spans": [_span("C")]},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert elements[0].text == "AB\nC"
|
||||||
|
|
||||||
|
def test_inline_equation_spans_are_wrapped_in_dollars(self):
|
||||||
|
"""上游惯例;样本中从未出现该 span 类型,未经实测。"""
|
||||||
|
block = {
|
||||||
|
"type": "text",
|
||||||
|
"bbox": list(_BBOX),
|
||||||
|
"lines": [{"spans": [_span("see "), _span("a+b", "inline_equation"), _span(".")]}],
|
||||||
|
}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert elements[0].text == "see $a+b$."
|
||||||
|
|
||||||
|
def test_empty_lines_list_is_empty_text(self):
|
||||||
|
block = {"type": "text", "bbox": list(_BBOX), "lines": []}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert elements[0].text == ""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"mutate",
|
||||||
|
[
|
||||||
|
pytest.param(lambda b: b.update(lines="not-a-list"), id="lines-not-list"),
|
||||||
|
pytest.param(lambda b: b.update(lines=["not-a-dict"]), id="line-not-dict"),
|
||||||
|
pytest.param(lambda b: b["lines"][0].update(spans="x"), id="spans-not-list"),
|
||||||
|
pytest.param(lambda b: b["lines"][0].update(spans=["x"]), id="span-not-dict"),
|
||||||
|
pytest.param(lambda b: b["lines"][0]["spans"][0].pop("content"), id="content-missing"),
|
||||||
|
pytest.param(lambda b: b["lines"][0]["spans"][0].update(content=7), id="content-int"),
|
||||||
|
pytest.param(
|
||||||
|
lambda b: b["lines"][0]["spans"][0].update(content=None), id="content-null"
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_malformed_text_structure_is_rejected_not_silently_empty(self, mutate):
|
||||||
|
block = _text_block()
|
||||||
|
mutate(block)
|
||||||
|
_invalid([_page([block])])
|
||||||
|
|
||||||
|
|
||||||
|
class TestTableAndImage:
|
||||||
|
def test_table_falls_back_to_latex_when_no_html(self):
|
||||||
|
body = _body("table_body", [{"type": "table", "latex": "\\begin{tabular}SYNTH"}])
|
||||||
|
block = {"type": "table", "bbox": list(_BBOX), "blocks": [body]}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert elements[0].text == "\\begin{tabular}SYNTH"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"block",
|
||||||
|
[
|
||||||
|
pytest.param(
|
||||||
|
{
|
||||||
|
"type": "table",
|
||||||
|
"bbox": list(_BBOX),
|
||||||
|
"blocks": [_body("table_body", [{"type": "table"}])],
|
||||||
|
},
|
||||||
|
id="span-has-neither-html-nor-latex",
|
||||||
|
),
|
||||||
|
pytest.param(
|
||||||
|
{"type": "table", "bbox": list(_BBOX), "blocks": []}, id="no-body-sub-block"
|
||||||
|
),
|
||||||
|
pytest.param({"type": "table", "bbox": list(_BBOX)}, id="no-blocks-key"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_table_without_content_is_rejected_not_silently_empty(self, block):
|
||||||
|
"""表格无内容与"合法空串"无法在结果里区分,故响亮失败。"""
|
||||||
|
_invalid([_page([block])])
|
||||||
|
|
||||||
|
def test_image_text_is_empty_even_with_path_only_span(self):
|
||||||
|
elements, _, _ = _parse([_page([_image_block()])])
|
||||||
|
assert (elements[0].type, elements[0].text) == ("image", "")
|
||||||
|
|
||||||
|
def test_non_body_sub_blocks_become_independent_elements_after_their_parent(self):
|
||||||
|
"""默认部署不产生 caption/footnote 子块(未经实测);出现时不得静默丢弃。"""
|
||||||
|
caption = _body("table_caption", [_span("SYNTH-CAPTION")], bbox=(1.0, 2.0, 3.0, 4.0))
|
||||||
|
footnote = _body("table_footnote", [_span("SYNTH-NOTE")], bbox=(5.0, 6.0, 7.0, 8.0))
|
||||||
|
block = _table_block(extra_subs=[caption])
|
||||||
|
block["blocks"].append(footnote)
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert [(e.type, e.text) for e in elements] == [
|
||||||
|
("table", "<table><tr><td>SYNTH-CELL</td></tr></table>"),
|
||||||
|
("table_caption", "SYNTH-CAPTION"),
|
||||||
|
("table_footnote", "SYNTH-NOTE"),
|
||||||
|
]
|
||||||
|
assert elements[1].bbox == (1.0, 2.0, 3.0, 4.0)
|
||||||
|
assert all(e.page_index == 0 for e in elements)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("blocks", "reason"),
|
||||||
|
[
|
||||||
|
pytest.param("not-a-list", "blocks 无效", id="blocks-not-list"),
|
||||||
|
pytest.param(["not-a-dict"], "块结构无效", id="sub-block-not-dict"),
|
||||||
|
pytest.param([{"bbox": list(_BBOX)}], "块缺少 type", id="sub-block-without-type"),
|
||||||
|
pytest.param([{"type": " ", "bbox": list(_BBOX)}], "块缺少 type", id="blank-type"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_malformed_nested_blocks_are_rejected_with_their_own_reason(self, blocks, reason):
|
||||||
|
"""外层边界会把 TypeError 等也归为 ResultInvalid,所以必须断言**原因**,否则"显式校验"
|
||||||
|
与"碰巧在别处失败"无法区分(变异检查发现仅断言类型时删掉校验测试仍绿)。"""
|
||||||
|
block = {"type": "table", "bbox": list(_BBOX), "blocks": blocks}
|
||||||
|
with pytest.raises(ResultInvalidError, match=reason):
|
||||||
|
_parse([_page([block])])
|
||||||
|
|
||||||
|
def test_empty_non_body_sub_block_is_not_emitted(self):
|
||||||
|
caption = _body("table_caption", [_span("")])
|
||||||
|
elements, _, _ = _parse([_page([_table_block(extra_subs=[caption])])])
|
||||||
|
assert [e.type for e in elements] == ["table"]
|
||||||
|
|
||||||
|
def test_unknown_type_without_lines_or_blocks_keeps_type_and_empty_text(self):
|
||||||
|
block = {"type": "synthetic_widget", "bbox": list(_BBOX)}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert (elements[0].type, elements[0].text) == ("synthetic_widget", "")
|
||||||
|
|
||||||
|
def test_unknown_container_keeps_its_body_text(self):
|
||||||
|
"""未知容器类型的 `*_body` 子块文字并入父元素,不被静默丢弃。"""
|
||||||
|
block = {
|
||||||
|
"type": "synthetic_code",
|
||||||
|
"bbox": list(_BBOX),
|
||||||
|
"blocks": [_body("synthetic_code_body", [_span("SYNTH-CODE")])],
|
||||||
|
}
|
||||||
|
elements, _, _ = _parse([_page([block])])
|
||||||
|
assert (elements[0].type, elements[0].text) == ("synthetic_code", "SYNTH-CODE")
|
||||||
|
|
||||||
|
|
||||||
|
class TestPageStructure:
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"payload",
|
||||||
|
[
|
||||||
|
pytest.param({}, id="pdf_info-missing"),
|
||||||
|
pytest.param({"pdf_info": "x"}, id="pdf_info-not-list"),
|
||||||
|
pytest.param({"pdf_info": []}, id="zero-pages"),
|
||||||
|
pytest.param([], id="payload-not-dict"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_document_level_shape(self, payload):
|
||||||
|
"""零页被拒: 成功解析的文档至少 1 页(与 layout 路径"elements 空合法"不同)。"""
|
||||||
|
_invalid(middle=json.dumps(payload).encode())
|
||||||
|
|
||||||
|
def test_page_not_a_dict(self):
|
||||||
|
_invalid(["not-a-dict"])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"page_size",
|
||||||
|
[
|
||||||
|
pytest.param(None, id="missing"),
|
||||||
|
pytest.param([612.0], id="one-element"),
|
||||||
|
pytest.param([612.0, 792.0, 1.0], id="three-elements"),
|
||||||
|
pytest.param([0, 792.0], id="zero-width"),
|
||||||
|
pytest.param([612.0, -1], id="negative-height"),
|
||||||
|
pytest.param([True, 792.0], id="bool"),
|
||||||
|
pytest.param(["612", 792.0], id="string"),
|
||||||
|
pytest.param([10**400, 792.0], id="huge-int"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_page_size_invalid(self, page_size):
|
||||||
|
page = _page([])
|
||||||
|
if page_size is None:
|
||||||
|
del page["page_size"]
|
||||||
|
else:
|
||||||
|
page["page_size"] = page_size
|
||||||
|
_invalid([page])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"page_idx",
|
||||||
|
[
|
||||||
|
pytest.param("missing", id="missing"),
|
||||||
|
pytest.param("0", id="string"),
|
||||||
|
pytest.param(0.0, id="float"),
|
||||||
|
pytest.param(True, id="bool-true-off-position"),
|
||||||
|
pytest.param(-1, id="negative"),
|
||||||
|
pytest.param(3, id="out-of-position"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_page_idx_must_equal_position(self, page_idx):
|
||||||
|
"""layout 路径对非法 `page_idx` 回退枚举下标,会让 page_sizes 按 page_index 的索引契约静默错位。"""
|
||||||
|
page = _page([])
|
||||||
|
if page_idx == "missing":
|
||||||
|
del page["page_idx"]
|
||||||
|
else:
|
||||||
|
page["page_idx"] = page_idx
|
||||||
|
_invalid([page])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"pages",
|
||||||
|
[
|
||||||
|
pytest.param([_page([], page_idx=False)], id="false-equals-0-on-first-page"),
|
||||||
|
pytest.param(
|
||||||
|
[_page([], page_idx=0), _page([], page_idx=True)], id="true-equals-1-on-second-page"
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_bool_page_idx_is_rejected_even_when_it_equals_the_position(self, pages):
|
||||||
|
"""`False == 0`、`True == 1`: 没有显式的 bool 守卫时它们恰好等于页序而被放行。"""
|
||||||
|
with pytest.raises(ResultInvalidError, match="page_idx"):
|
||||||
|
_parse(pages)
|
||||||
|
|
||||||
|
def test_second_page_must_not_claim_the_first_index(self):
|
||||||
|
_invalid([_page([], page_idx=0), _page([], page_idx=0)])
|
||||||
|
|
||||||
|
def test_para_blocks_missing_is_a_legal_empty_page(self):
|
||||||
|
elements, page_sizes, _ = _parse([_page()])
|
||||||
|
assert elements == [] and page_sizes == [(612.0, 792.0)]
|
||||||
|
|
||||||
|
def test_para_blocks_not_a_list_is_rejected(self):
|
||||||
|
_invalid([_page("x")])
|
||||||
|
|
||||||
|
|
||||||
|
class TestBlockStructure:
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"block",
|
||||||
|
[
|
||||||
|
pytest.param("not-a-dict", id="block-not-dict"),
|
||||||
|
pytest.param({"bbox": list(_BBOX)}, id="type-missing"),
|
||||||
|
pytest.param({"type": 7, "bbox": list(_BBOX)}, id="type-not-str"),
|
||||||
|
pytest.param({"type": "", "bbox": list(_BBOX)}, id="type-empty"),
|
||||||
|
pytest.param({"type": " ", "bbox": list(_BBOX)}, id="type-whitespace"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_block_shape(self, block):
|
||||||
|
_invalid([_page([block])])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"bbox",
|
||||||
|
[
|
||||||
|
pytest.param(None, id="missing"),
|
||||||
|
pytest.param("1,2,3,4", id="string"),
|
||||||
|
pytest.param([1, 2, 3], id="length-3"),
|
||||||
|
pytest.param([1, 2, 3, 4, 5], id="length-5"),
|
||||||
|
pytest.param([1, 2, "3", 4], id="string-element"),
|
||||||
|
pytest.param([1, 2, True, 4], id="bool-element"),
|
||||||
|
pytest.param([1, 2, float("nan"), 4], id="nan"),
|
||||||
|
pytest.param([1, 2, float("inf"), 4], id="inf"),
|
||||||
|
pytest.param([10, 2, 5, 4], id="x-reversed"),
|
||||||
|
pytest.param([1, 20, 5, 4], id="y-reversed"),
|
||||||
|
pytest.param([0, 0, 10**400, 1], id="huge-int"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_bbox_invalid(self, bbox):
|
||||||
|
block = _text_block()
|
||||||
|
if bbox is None:
|
||||||
|
del block["bbox"]
|
||||||
|
else:
|
||||||
|
block["bbox"] = bbox
|
||||||
|
# NaN/Infinity 不是合法 JSON,但 Python 的 json 模块默认接受并往返——真实服务可能发出
|
||||||
|
_invalid([_page([block])])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bbox", [(5, 5, 5, 20), (5, 5, 50, 5), (5, 5, 5, 5)])
|
||||||
|
def test_zero_width_or_height_boxes_are_legal(self, bbox):
|
||||||
|
"""整数化退化校验专为 CHS shim 的 int() 裁剪;文档消费者不裁剪,一个退化框不该让整份失败。"""
|
||||||
|
elements, _, _ = _parse([_page([_text_block(bbox=bbox)])])
|
||||||
|
assert elements[0].bbox == tuple(float(v) for v in bbox)
|
||||||
|
|
||||||
|
def test_out_of_page_boxes_are_returned_as_is(self):
|
||||||
|
"""越界与否是图内语义,归调用方(ARCHITECTURE §5.1 边界);库不裁剪。"""
|
||||||
|
elements, _, _ = _parse([_page([_text_block(bbox=(0, 0, 9999, 9999))])])
|
||||||
|
assert elements[0].bbox == (0.0, 0.0, 9999.0, 9999.0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestMembers:
|
||||||
|
def test_missing_middle_json(self):
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(buf, "w") as zf:
|
||||||
|
zf.writestr("doc.md", "x")
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(buf.getvalue())
|
||||||
|
|
||||||
|
def test_missing_paired_markdown(self):
|
||||||
|
_invalid(md=None)
|
||||||
|
|
||||||
|
def test_markdown_is_paired_by_name_not_first_match(self):
|
||||||
|
elements, _, markdown = parse_document_package(
|
||||||
|
_package(
|
||||||
|
[_page([_text_block("A")])],
|
||||||
|
md="PAIRED",
|
||||||
|
extra=[("aaa_other.md", "DISTRACTOR")],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert markdown == "PAIRED"
|
||||||
|
|
||||||
|
def test_unpaired_markdown_alone_is_not_accepted(self):
|
||||||
|
_invalid(md=None, extra=[("other.md", "DISTRACTOR")])
|
||||||
|
|
||||||
|
def test_first_middle_json_wins_and_pairs_with_its_own_markdown(self):
|
||||||
|
pages_a = json.dumps({"pdf_info": [_page([_text_block("A")])]}).encode()
|
||||||
|
pages_b = json.dumps({"pdf_info": [_page([_text_block("B")])]}).encode()
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(buf, "w") as zf:
|
||||||
|
zf.writestr("a_middle.json", pages_a)
|
||||||
|
zf.writestr("b_middle.json", pages_b)
|
||||||
|
zf.writestr("a.md", "MD-A")
|
||||||
|
zf.writestr("b.md", "MD-B")
|
||||||
|
elements, _, markdown = parse_document_package(buf.getvalue())
|
||||||
|
assert [e.text for e in elements] == ["A"] and markdown == "MD-A"
|
||||||
|
|
||||||
|
def test_markdown_bom_is_stripped(self):
|
||||||
|
_, _, markdown = _parse(md=b"\xef\xbb\xbf# synth")
|
||||||
|
assert markdown == "# synth"
|
||||||
|
|
||||||
|
def test_empty_markdown_is_legal(self):
|
||||||
|
_, _, markdown = _parse(md="")
|
||||||
|
assert markdown == ""
|
||||||
|
|
||||||
|
def test_non_utf8_markdown_is_rejected(self):
|
||||||
|
_invalid(md=b"\xff\xfe\x00bad")
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"middle",
|
||||||
|
[
|
||||||
|
pytest.param(b"not json at all", id="not-json"),
|
||||||
|
pytest.param(b"", id="empty"),
|
||||||
|
pytest.param(b"\xff\xfe", id="not-utf8"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_middle_json_unparseable(self, middle):
|
||||||
|
_invalid(middle=middle)
|
||||||
|
|
||||||
|
def test_not_a_zip(self):
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(b"definitely not a zip")
|
||||||
|
|
||||||
|
|
||||||
|
def _patch_first_member_compression(zip_bytes: bytes, method: int) -> bytes:
|
||||||
|
"""把第一个成员的压缩方法号改成 `method`(本地头偏移 8,中央目录偏移 10)。"""
|
||||||
|
data = bytearray(zip_bytes)
|
||||||
|
local = data.index(b"PK\x03\x04")
|
||||||
|
central = data.index(b"PK\x01\x02")
|
||||||
|
struct.pack_into("<H", data, local + 8, method)
|
||||||
|
struct.pack_into("<H", data, central + 10, method)
|
||||||
|
return bytes(data)
|
||||||
|
|
||||||
|
|
||||||
|
class TestExternalDataExceptionBoundary:
|
||||||
|
"""外部字节引起的一切已知异常都归 `ResultInvalidError`,绝不漏出原生异常。
|
||||||
|
|
||||||
|
漏出的后果(`OcrClient._call` 只接 PolyGatewayError): 不写终态遥测行,持有的半开探针
|
||||||
|
也不归还。边界必须同时盖住「读取」与「全部结构解析」两段。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_encrypted_member(self):
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(_encrypted_package())
|
||||||
|
|
||||||
|
def test_unknown_compression_method(self):
|
||||||
|
patched = _patch_first_member_compression(_package([_page([])]), 99)
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(patched)
|
||||||
|
|
||||||
|
def test_crc_corruption(self):
|
||||||
|
corrupted = _package([_page([])]).replace(b'"pdf_info"', b'"pdf_infq"', 1)
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(corrupted)
|
||||||
|
|
||||||
|
def test_huge_integer_inside_page_structure_is_not_an_overflow_error(self):
|
||||||
|
"""`float(10**400)` 抛 OverflowError——发生在结构解析段,不在读取段。"""
|
||||||
|
block = _text_block()
|
||||||
|
block["bbox"] = [0, 0, 10**400, 1]
|
||||||
|
_invalid([_page([block])])
|
||||||
|
|
||||||
|
def test_deeply_nested_json_is_not_a_recursion_error(self):
|
||||||
|
deep = b'{"pdf_info": ' + b"[" * 100_000 + b"]" * 100_000 + b"}"
|
||||||
|
_invalid(middle=deep)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("cap_name", "member"),
|
||||||
|
[("MAX_MIDDLE_BYTES", "doc_middle.json"), ("MAX_MARKDOWN_BYTES", "doc.md")],
|
||||||
|
)
|
||||||
|
def test_declared_size_over_cap_names_the_offending_member(self, cap_name, member, monkeypatch):
|
||||||
|
"""断言消息里的成员名: 两个上限互换时,只断言异常类型的旧用例仍会通过。"""
|
||||||
|
monkeypatch.setattr(_monkey_document, cap_name, 8)
|
||||||
|
with pytest.raises(ResultInvalidError, match=member):
|
||||||
|
_parse([_page([_text_block("A")])], md="# a markdown longer than eight bytes")
|
||||||
|
|
||||||
|
def test_existing_result_invalid_messages_pass_through_unchanged(self):
|
||||||
|
with pytest.raises(ResultInvalidError, match="至少应有 1 页"):
|
||||||
|
_parse([])
|
||||||
|
|
||||||
|
|
||||||
|
class TestLayoutPathUntouched:
|
||||||
|
"""文档路径更严/更宽的口径不得外溢到既有调用方在用的 layout 解析。"""
|
||||||
|
|
||||||
|
def test_layout_parser_still_rejects_degenerate_int_boxes(self):
|
||||||
|
from polygateway.transports.monkey_ocr import _parse_middle_json
|
||||||
|
|
||||||
|
block = {"type": "text", "bbox": [5, 5, 5.4, 20], "lines": []}
|
||||||
|
zip_bytes = _package([_page([block])])
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
_parse_middle_json(zip_bytes)
|
||||||
|
|
||||||
|
def test_layout_parser_keeps_its_lenient_page_idx_handling(self):
|
||||||
|
"""layout 路径: 合法的非负整数 `page_idx` 原样接受(哪怕与页序不符),非法值才回退
|
||||||
|
枚举下标——这正是文档路径改为严格的原因,但既有调用方的行为不得被连带改动。"""
|
||||||
|
from polygateway.transports.monkey_ocr import _parse_middle_json
|
||||||
|
|
||||||
|
accepted, _ = _parse_middle_json(_package([_page([_text_block()], page_idx=7)]))
|
||||||
|
assert accepted[0].page_index == 7
|
||||||
|
fallback, _ = _parse_middle_json(_package([_page([_text_block()], page_idx="x")]))
|
||||||
|
assert fallback[0].page_index == 0
|
||||||
|
|
||||||
|
|
||||||
|
class TestZipLayerNeverLeaksNativeExceptions:
|
||||||
|
"""独立验证发现: BZIP2/LZMA 成员数据损坏会漏出 `OSError`/`LZMAError`,中央目录版本字段损坏会在
|
||||||
|
`ZipFile` 构造阶段漏出 `NotImplementedError`——它们都不在外层边界里。漏出的后果与设计 §5 一致:
|
||||||
|
`OcrClient._call` 只接 `PolyGatewayError`,既不写终态遥测行,也不归还半开探针。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _zip(method: int) -> bytes:
|
||||||
|
payload = json.dumps({"pdf_info": [_page([_text_block("SYNTH")])]}).encode() * 20
|
||||||
|
buf = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(buf, "w", compression=method) as zf:
|
||||||
|
zf.writestr("doc_middle.json", payload)
|
||||||
|
zf.writestr("doc.md", "# synth " * 50)
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"method",
|
||||||
|
[
|
||||||
|
pytest.param(zipfile.ZIP_DEFLATED, id="deflated"),
|
||||||
|
pytest.param(zipfile.ZIP_BZIP2, id="bzip2"),
|
||||||
|
pytest.param(zipfile.ZIP_LZMA, id="lzma"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_single_bit_flips_never_escape_as_native_exceptions(self, method):
|
||||||
|
"""确定性字节翻转(固定种子): 每个结果只能是成功或 `ResultInvalidError`。"""
|
||||||
|
base = self._zip(method)
|
||||||
|
rng = random.Random(34)
|
||||||
|
for _ in range(600):
|
||||||
|
data = bytearray(base)
|
||||||
|
position = rng.randrange(len(data))
|
||||||
|
data[position] ^= 1 << rng.randrange(8)
|
||||||
|
try:
|
||||||
|
parse_document_package(bytes(data))
|
||||||
|
except ResultInvalidError:
|
||||||
|
continue
|
||||||
|
except Exception as exc: # noqa: BLE001 — 测试要抓的正是任何漏出的原生异常
|
||||||
|
pytest.fail(
|
||||||
|
f"字节 {position} 翻转后漏出 {type(exc).__module__}.{type(exc).__name__}: {exc}"
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", [zipfile.ZIP_BZIP2, zipfile.ZIP_LZMA])
|
||||||
|
def test_corrupt_non_deflate_member_data(self, method):
|
||||||
|
"""压缩流内部损坏(而非 CRC 失败): 标准库抛 OSError / LZMAError。"""
|
||||||
|
data = bytearray(self._zip(method))
|
||||||
|
local = data.index(b"PK\x03\x04")
|
||||||
|
name_len, extra_len = struct.unpack_from("<HH", data, local + 26)
|
||||||
|
body = local + 30 + name_len + extra_len
|
||||||
|
for offset in range(body, body + 12):
|
||||||
|
data[offset] ^= 0xFF
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(bytes(data))
|
||||||
|
|
||||||
|
def test_zipfile_construction_stage_is_inside_the_boundary(self):
|
||||||
|
"""中央目录的 extract_version 损坏: `ZipFile(...)` 构造阶段抛 NotImplementedError。"""
|
||||||
|
data = bytearray(self._zip(zipfile.ZIP_STORED))
|
||||||
|
central = data.index(b"PK\x01\x02")
|
||||||
|
data[central + 6] = 99 # extract_version = 9.9 → "zip file version 9.9"
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
parse_document_package(bytes(data))
|
||||||
|
|
||||||
|
|
||||||
|
class TestTextBearingBlocksRequireLines:
|
||||||
|
"""text/title/interline_equation 缺 `lines` 不是「空文本」而是畸形: 与 table 缺内容就报错同口径,
|
||||||
|
否则下游分不清「服务识别出空块」和「服务给了坏结构」。"""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("block_type", ["text", "title", "interline_equation"])
|
||||||
|
def test_missing_lines_is_rejected_with_its_reason(self, block_type):
|
||||||
|
block = {"type": block_type, "bbox": list(_BBOX)}
|
||||||
|
with pytest.raises(ResultInvalidError, match="缺少 lines"):
|
||||||
|
_parse([_page([block])])
|
||||||
@@ -8,6 +8,7 @@ P5 医疗数据不入库)。
|
|||||||
import asyncio
|
import asyncio
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
|
import threading
|
||||||
import zipfile
|
import zipfile
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
@@ -19,12 +20,20 @@ from polygateway.errors import (
|
|||||||
SourceDeadError,
|
SourceDeadError,
|
||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
|
from polygateway.transports import monkey_ocr
|
||||||
from polygateway.transports.monkey_ocr import (
|
from polygateway.transports.monkey_ocr import (
|
||||||
|
_DEFAULT_REJECTED_MARKERS,
|
||||||
MonkeyOcrTransport,
|
MonkeyOcrTransport,
|
||||||
_classify_status,
|
_classify_status,
|
||||||
_parse_middle_json,
|
_parse_middle_json,
|
||||||
|
_sniff_media,
|
||||||
|
_translate_http_errors,
|
||||||
|
_upload_part,
|
||||||
)
|
)
|
||||||
from polygateway.types import SourceConfig
|
from polygateway.types import OcrDocumentElement, OcrDocumentTransportResult, SourceConfig
|
||||||
|
from tests.unit._document_fixtures import _package as _doc_package
|
||||||
|
from tests.unit._document_fixtures import _page as _doc_page
|
||||||
|
from tests.unit._document_fixtures import _table_block, _text_block
|
||||||
|
|
||||||
|
|
||||||
def _source(**overrides):
|
def _source(**overrides):
|
||||||
@@ -167,6 +176,96 @@ class TestRecognizeText:
|
|||||||
assert b'filename="image.jpg"' in seen["body"] # CHS invokers.py:496 固定名
|
assert b'filename="image.jpg"' in seen["body"] # CHS invokers.py:496 固定名
|
||||||
|
|
||||||
|
|
||||||
|
_PDF_BODY = b"%PDF-1.4\n%synthetic body"
|
||||||
|
# BMP: "BM" + 文件大小(4) + 保留(4) + 像素偏移(4) + DIB 头大小(4,常见 12/40/52/56/64/108/124)
|
||||||
|
_BMP_HEADER = b"BM" + b"\x00" * 12 + (40).to_bytes(4, "little") + b"\x00" * 20
|
||||||
|
_JUNK = b"x" * 200
|
||||||
|
|
||||||
|
|
||||||
|
class TestSniffMedia:
|
||||||
|
"""上传类型按内容判定(#34 设计 §3 Q2): 图片魔数优先 → 前 1KB 内 `%PDF-` → 未知。"""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("data", "expected"),
|
||||||
|
[
|
||||||
|
pytest.param(_PDF_BODY, "pdf", id="pdf"),
|
||||||
|
pytest.param(_JUNK + _PDF_BODY, "pdf", id="pdf-with-junk-prefix"),
|
||||||
|
pytest.param(b"\xff\xd8\xff\xe0" + b"0" * 20, "image", id="jpeg"),
|
||||||
|
pytest.param(b"\x89PNG\r\n\x1a\n" + b"0" * 20, "image", id="png"),
|
||||||
|
pytest.param(b"II*\x00" + b"0" * 20, "image", id="tiff-le"),
|
||||||
|
pytest.param(b"MM\x00*" + b"0" * 20, "image", id="tiff-be"),
|
||||||
|
pytest.param(_BMP_HEADER, "image", id="bmp"),
|
||||||
|
pytest.param(
|
||||||
|
b"BMW is a car maker, not a bitmap", "unknown", id="text-starting-with-BM"
|
||||||
|
),
|
||||||
|
pytest.param(b"BM" + b"\x00" * 40, "unknown", id="BM-with-bad-dib-header"),
|
||||||
|
pytest.param(b"GIF89a" + b"0" * 20, "image", id="gif89a"),
|
||||||
|
pytest.param(b"GIF87a" + b"0" * 20, "image", id="gif87a"),
|
||||||
|
pytest.param(b"RIFF\x00\x00\x00\x00WEBPVP8 ", "image", id="webp"),
|
||||||
|
# 图片魔数优先: 含 XMP 元数据的图片可能在 1KB 内带 `%PDF-` 字样
|
||||||
|
pytest.param(
|
||||||
|
b"\xff\xd8\xff\xe0" + b"<xmp>%PDF-</xmp>", "image", id="jpeg-mentions-pdf"
|
||||||
|
),
|
||||||
|
pytest.param(b"<html><body>paywall</body></html>", "unknown", id="html"),
|
||||||
|
pytest.param(b"GARBAGE-NOT-AN-IMAGE", "unknown", id="garbage"),
|
||||||
|
pytest.param(b"", "unknown", id="empty"),
|
||||||
|
# 1KB 窗口之外的 `%PDF-` 不算
|
||||||
|
pytest.param(b"x" * 1100 + b"%PDF-", "unknown", id="header-beyond-window"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_sniff(self, data, expected):
|
||||||
|
assert _sniff_media(data) == expected
|
||||||
|
|
||||||
|
def test_upload_part_labels_pdf_only(self):
|
||||||
|
assert _upload_part(_PDF_BODY) == ("doc.pdf", _PDF_BODY, "application/pdf")
|
||||||
|
assert _upload_part(_JUNK + _PDF_BODY)[0] == "doc.pdf"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"data", [b"\xff\xd8\xff\xe0abc", b"\x89PNG\r\n\x1a\nabc", b"junk", b""]
|
||||||
|
)
|
||||||
|
def test_everything_else_keeps_the_legacy_label(self, data):
|
||||||
|
"""非 PDF 路径字节级不变: 服务对图片按内容解码(2026-10-21 实测 PNG/TIFF/BMP 标 image.jpg 成功)。"""
|
||||||
|
assert _upload_part(data) == ("image.jpg", data, "image/jpeg")
|
||||||
|
|
||||||
|
|
||||||
|
class TestUploadLabelPerMethod:
|
||||||
|
"""三个调用点共用同一个标签点: 漏掉任何一个,E4 的静默错误结果就在那里复活。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _capturing_transport(seen):
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
if request.url.path == "/ocr/text":
|
||||||
|
seen["body"] = request.read()
|
||||||
|
return httpx.Response(200, json=_text_body())
|
||||||
|
if request.url.path == "/parse":
|
||||||
|
seen["body"] = request.read()
|
||||||
|
return httpx.Response(200, json=_parse_body())
|
||||||
|
return httpx.Response(200, content=_zip_bytes())
|
||||||
|
|
||||||
|
return _transport_for(handler)
|
||||||
|
|
||||||
|
async def test_recognize_text_labels_pdf_as_pdf(self):
|
||||||
|
seen = {}
|
||||||
|
t = self._capturing_transport(seen)
|
||||||
|
await t.recognize_text(image=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert b'filename="doc.pdf"' in seen["body"]
|
||||||
|
assert b"Content-Type: application/pdf" in seen["body"]
|
||||||
|
|
||||||
|
async def test_parse_layout_labels_pdf_as_pdf(self):
|
||||||
|
seen = {}
|
||||||
|
t = self._capturing_transport(seen)
|
||||||
|
await t.parse_layout(image=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert b'filename="doc.pdf"' in seen["body"]
|
||||||
|
assert b"Content-Type: application/pdf" in seen["body"]
|
||||||
|
|
||||||
|
async def test_parse_layout_keeps_image_label_for_images(self):
|
||||||
|
seen = {}
|
||||||
|
t = self._capturing_transport(seen)
|
||||||
|
await t.parse_layout(image=b"\xff\xd8\xff\xe0abc", source=_source(), call_id="c1")
|
||||||
|
assert b'filename="image.jpg"' in seen["body"]
|
||||||
|
assert b"Content-Type: image/jpeg" in seen["body"]
|
||||||
|
|
||||||
|
|
||||||
class TestParseLayout:
|
class TestParseLayout:
|
||||||
async def test_full_chain_relative_url(self):
|
async def test_full_chain_relative_url(self):
|
||||||
t = _transport_for(
|
t = _transport_for(
|
||||||
@@ -369,6 +468,424 @@ class TestErrorTranslation:
|
|||||||
assert isinstance(ei.value.__cause__, httpx.TimeoutException)
|
assert isinstance(ei.value.__cause__, httpx.TimeoutException)
|
||||||
|
|
||||||
|
|
||||||
|
def _http_error(status, body):
|
||||||
|
request = httpx.Request("POST", "http://ocr.example/parse")
|
||||||
|
return httpx.HTTPStatusError(
|
||||||
|
str(status),
|
||||||
|
request=request,
|
||||||
|
response=httpx.Response(status, content=body.encode(), request=request),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parsing_failed(reason):
|
||||||
|
# 真实形态(2026-10-21 实测): 服务把一切异常包成 500 + {"detail": "Parsing failed: …"}
|
||||||
|
return json.dumps({"detail": f"Parsing failed: {reason}"})
|
||||||
|
|
||||||
|
|
||||||
|
class TestRejectedMarkers:
|
||||||
|
"""#34 设计 §6: 确定性坏输入的 500 在文档路径归「请求被拒」,其余分类一概不动。
|
||||||
|
|
||||||
|
共享的 `_classify_status` 默认行为字节级不变——既有调用方对坏图的重试/熔断行为属公共行为,
|
||||||
|
推广须另行设计(issue #35)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("marker", _DEFAULT_REJECTED_MARKERS)
|
||||||
|
def test_known_deterministic_failures_become_rejected_with_status_500(self, marker):
|
||||||
|
exc = _http_error(500, _parsing_failed(marker.capitalize()))
|
||||||
|
err = _classify_status(exc, "monkey_1", "parse", _DEFAULT_REJECTED_MARKERS)
|
||||||
|
assert isinstance(err, RequestRejectedError)
|
||||||
|
assert err.status_code == 500 # 服务有响应: 熔断按「响应即健康」记成功
|
||||||
|
assert marker.capitalize() in err.body_text # 响应体摘要仍随该分支
|
||||||
|
|
||||||
|
def test_marker_matching_is_case_insensitive(self):
|
||||||
|
exc = _http_error(500, '{"detail": "PARSING FAILED: FAILED TO OPEN STREAM"}')
|
||||||
|
err = _classify_status(exc, "monkey_1", "parse", _DEFAULT_REJECTED_MARKERS)
|
||||||
|
assert isinstance(err, RequestRejectedError)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"body",
|
||||||
|
[
|
||||||
|
pytest.param('{"detail": "Failed to open stream"}', id="no-parsing-failed-prefix"),
|
||||||
|
pytest.param(
|
||||||
|
_parsing_failed("500: Model not initialized"), id="real-fault-same-prefix"
|
||||||
|
),
|
||||||
|
pytest.param(
|
||||||
|
_parsing_failed("400: Unsupported file type: .webp"), id="unsupported-type"
|
||||||
|
),
|
||||||
|
pytest.param(_parsing_failed("CUDA out of memory"), id="oom"),
|
||||||
|
pytest.param("", id="empty-body"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_everything_else_stays_transient(self, body):
|
||||||
|
"""`Parsing failed:` 前缀不能单独当判据: 服务把真故障也包成同一前缀。
|
||||||
|
`Unsupported file type` 有意不在白名单——嗅探后它只可能是契约变化或我方标签 bug。"""
|
||||||
|
err = _classify_status(
|
||||||
|
_http_error(500, body), "monkey_1", "parse", _DEFAULT_REJECTED_MARKERS
|
||||||
|
)
|
||||||
|
assert isinstance(err, TransientError)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("body", [_parsing_failed(m) for m in _DEFAULT_REJECTED_MARKERS])
|
||||||
|
def test_default_call_without_markers_is_unchanged(self, body):
|
||||||
|
err = _classify_status(_http_error(500, body), "monkey_1", "parse")
|
||||||
|
assert isinstance(err, TransientError)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("status", "exc_type"),
|
||||||
|
[
|
||||||
|
(429, TransientError),
|
||||||
|
(502, TransientError),
|
||||||
|
(503, TransientError),
|
||||||
|
(401, SourceDeadError),
|
||||||
|
(403, SourceDeadError),
|
||||||
|
(404, RequestRejectedError),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_other_statuses_are_never_narrowed(self, status, exc_type):
|
||||||
|
body = _parsing_failed(_DEFAULT_REJECTED_MARKERS[0])
|
||||||
|
err = _classify_status(_http_error(status, body), "m", "parse", _DEFAULT_REJECTED_MARKERS)
|
||||||
|
assert type(err) is exc_type
|
||||||
|
|
||||||
|
def test_translate_http_errors_passes_markers_through(self):
|
||||||
|
with (
|
||||||
|
pytest.raises(RequestRejectedError),
|
||||||
|
_translate_http_errors("monkey_1", "parse", rejected_markers=_DEFAULT_REJECTED_MARKERS),
|
||||||
|
):
|
||||||
|
raise _http_error(500, _parsing_failed(_DEFAULT_REJECTED_MARKERS[0]))
|
||||||
|
# 默认不传: 行为不变
|
||||||
|
with pytest.raises(TransientError), _translate_http_errors("monkey_1", "parse"):
|
||||||
|
raise _http_error(500, _parsing_failed(_DEFAULT_REJECTED_MARKERS[0]))
|
||||||
|
|
||||||
|
def test_markers_are_configurable_per_transport(self):
|
||||||
|
assert MonkeyOcrTransport()._rejected_markers == _DEFAULT_REJECTED_MARKERS
|
||||||
|
custom = ("synthetic marker",)
|
||||||
|
assert MonkeyOcrTransport(deterministic_failure_markers=custom)._rejected_markers == custom
|
||||||
|
|
||||||
|
async def test_layout_and_text_paths_do_not_apply_the_markers(self):
|
||||||
|
"""传输实例持有标记,但只有文档路径使用;既有的两个方法行为不变。"""
|
||||||
|
body = _parsing_failed(_DEFAULT_REJECTED_MARKERS[0]).encode()
|
||||||
|
t = _transport_for(
|
||||||
|
_routes(
|
||||||
|
text_resp=httpx.Response(500, content=body),
|
||||||
|
parse_resp=httpx.Response(500, content=body),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
with pytest.raises(TransientError):
|
||||||
|
await t.recognize_text(image=b"jpg", source=_source(), call_id="c1")
|
||||||
|
with pytest.raises(TransientError):
|
||||||
|
await t.parse_layout(image=b"jpg", source=_source(), call_id="c1")
|
||||||
|
|
||||||
|
|
||||||
|
class TestMalformedDownloadUrl:
|
||||||
|
"""`download_url` 来自服务端响应,属外部输入。畸形值(控制字符、坏方括号、越界端口)会让 httpx
|
||||||
|
抛 `InvalidURL`(不属 `HTTPError` 体系)或在连接阶段抛 ExceptionGroup,从 `_translate_http_errors`
|
||||||
|
下漏出四分类——`OcrClient._call` 只接 `PolyGatewayError`,于是不写终态遥测行、半开探针不归还。
|
||||||
|
(独立审查发现;`parse_layout` 的基线同样如此,共用的 `_post_parse` 一处修。)"""
|
||||||
|
|
||||||
|
_BAD_URLS = [
|
||||||
|
pytest.param("http://[::1", id="broken-bracket"),
|
||||||
|
pytest.param("/static/a\r\nb.zip", id="crlf"),
|
||||||
|
pytest.param("/static/a\x00.zip", id="nul"),
|
||||||
|
pytest.param("/static/a\tb.zip", id="tab"),
|
||||||
|
pytest.param("http://127.0.0.1:99999/x", id="port-out-of-range"),
|
||||||
|
]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _transport_with(download_url, seen):
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
seen.append((request.method, request.url.path))
|
||||||
|
if request.url.path == "/parse":
|
||||||
|
return httpx.Response(200, json=_parse_body(download_url))
|
||||||
|
return httpx.Response(200, content=b"never fetched")
|
||||||
|
|
||||||
|
return _transport_for(handler)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("download_url", _BAD_URLS)
|
||||||
|
@pytest.mark.parametrize("method", ["parse_layout", "parse_document"])
|
||||||
|
async def test_rejected_as_transient_without_fetching(self, download_url, method):
|
||||||
|
seen = []
|
||||||
|
t = self._transport_with(download_url, seen)
|
||||||
|
kwargs = {"image": b"jpg"} if method == "parse_layout" else {"document": _PDF_BODY}
|
||||||
|
with pytest.raises(TransientError, match="download_url"):
|
||||||
|
await getattr(t, method)(**kwargs, source=_source(), call_id="c1")
|
||||||
|
assert seen == [("POST", "/parse")] # 从未发出第二段请求
|
||||||
|
|
||||||
|
async def test_well_formed_urls_are_untouched(self):
|
||||||
|
for url in ("/static/ok.zip", "http://10.77.0.20:7866/static/abs.zip"):
|
||||||
|
seen = []
|
||||||
|
t = self._transport_with(url, seen)
|
||||||
|
with pytest.raises(ResultInvalidError): # 响应体不是 ZIP: 说明确实走到了第二段
|
||||||
|
await t.parse_layout(image=b"jpg", source=_source(), call_id="c1")
|
||||||
|
assert [m for m, _ in seen] == ["POST", "GET"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestParseDocument:
|
||||||
|
"""#34 设计 §5: 本地预检 → POST /parse → 流式 GET(体积上限)→ 线程内解析。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _doc_zip():
|
||||||
|
return _doc_package(
|
||||||
|
[_doc_page([_text_block("SYNTH-A"), _table_block("<table>SYNTH</table>")])]
|
||||||
|
)
|
||||||
|
|
||||||
|
def _transport(self, seen=None, *, zip_resp=None, parse_resp=None):
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
if seen is not None:
|
||||||
|
seen.append((request.method, request.url.path, request.read()))
|
||||||
|
if request.url.path == "/parse":
|
||||||
|
return parse_resp or httpx.Response(200, json=_parse_body())
|
||||||
|
if request.url.path.startswith("/static/"):
|
||||||
|
return zip_resp or httpx.Response(200, content=self._doc_zip())
|
||||||
|
return httpx.Response(500)
|
||||||
|
|
||||||
|
return _transport_for(handler)
|
||||||
|
|
||||||
|
async def test_full_chain_relative_download_url(self):
|
||||||
|
t = self._transport()
|
||||||
|
result = await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert isinstance(result, OcrDocumentTransportResult)
|
||||||
|
assert [(e.type, e.text) for e in result.elements] == [
|
||||||
|
("text", "SYNTH-A"),
|
||||||
|
("table", "<table>SYNTH</table>"),
|
||||||
|
]
|
||||||
|
assert all(isinstance(e, OcrDocumentElement) for e in result.elements)
|
||||||
|
assert result.page_sizes == [(612.0, 792.0)]
|
||||||
|
assert result.markdown == "# synth\n"
|
||||||
|
assert result.raw == {"success": True, "download_url": "/static/sample_parsed_1.zip"}
|
||||||
|
|
||||||
|
async def test_absolute_download_url(self):
|
||||||
|
url = "http://10.77.0.20:7866/static/abs.zip"
|
||||||
|
t = self._transport(parse_resp=httpx.Response(200, json=_parse_body(url)))
|
||||||
|
result = await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert result.raw["download_url"] == url
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("document", "filename", "content_type"),
|
||||||
|
[
|
||||||
|
pytest.param(_PDF_BODY, b"doc.pdf", b"application/pdf", id="pdf"),
|
||||||
|
pytest.param(_JUNK + _PDF_BODY, b"doc.pdf", b"application/pdf", id="pdf-junk-prefix"),
|
||||||
|
pytest.param(b"\xff\xd8\xff\xe0abc", b"image.jpg", b"image/jpeg", id="jpeg"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
async def test_upload_label_matrix_for_the_document_entry(
|
||||||
|
self, document, filename, content_type
|
||||||
|
):
|
||||||
|
seen = []
|
||||||
|
await self._transport(seen).parse_document(
|
||||||
|
document=document, source=_source(), call_id="c1"
|
||||||
|
)
|
||||||
|
post_body = next(body for method, path, body in seen if path == "/parse")
|
||||||
|
assert b'filename="' + filename + b'"' in post_body
|
||||||
|
assert b"Content-Type: " + content_type in post_body
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"document",
|
||||||
|
[b"<html>paywall</html>", b"GARBAGE-NOT-AN-IMAGE", b""],
|
||||||
|
ids=["html", "garbage", "empty"],
|
||||||
|
)
|
||||||
|
async def test_unrecognizable_bytes_are_rejected_locally_without_any_request(self, document):
|
||||||
|
seen = []
|
||||||
|
with pytest.raises(RequestRejectedError) as ei:
|
||||||
|
await self._transport(seen).parse_document(
|
||||||
|
document=document, source=_source(), call_id="c1"
|
||||||
|
)
|
||||||
|
assert seen == [] # 服务从未被打扰
|
||||||
|
assert ei.value.status_code is None # 服务未响应: 熔断侧只归还探针,不计成败
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("marker", _DEFAULT_REJECTED_MARKERS)
|
||||||
|
async def test_known_deterministic_post_failures_are_rejected_with_status_500(self, marker):
|
||||||
|
body = _parsing_failed(marker).encode()
|
||||||
|
t = self._transport(parse_resp=httpx.Response(500, content=body))
|
||||||
|
with pytest.raises(RequestRejectedError) as ei:
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert ei.value.status_code == 500 and ei.value.body_text
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"reason",
|
||||||
|
["CUDA out of memory", "400: Unsupported file type: .webp", "500: Model not initialized"],
|
||||||
|
)
|
||||||
|
async def test_unknown_post_500_stays_transient(self, reason):
|
||||||
|
t = self._transport(
|
||||||
|
parse_resp=httpx.Response(500, content=_parsing_failed(reason).encode())
|
||||||
|
)
|
||||||
|
with pytest.raises(TransientError):
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
|
||||||
|
async def test_success_false_is_rejected_with_status_200(self):
|
||||||
|
t = self._transport(
|
||||||
|
parse_resp=httpx.Response(200, json={"success": False, "message": "no"})
|
||||||
|
)
|
||||||
|
with pytest.raises(RequestRejectedError) as ei:
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert ei.value.status_code == 200
|
||||||
|
|
||||||
|
async def test_missing_download_url_and_non_json_are_transient(self):
|
||||||
|
body = {"success": True, "message": "ok"}
|
||||||
|
for resp in (httpx.Response(200, json=body), httpx.Response(200, content=b"not json")):
|
||||||
|
t = self._transport(parse_resp=resp)
|
||||||
|
with pytest.raises(TransientError):
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("status", "exc_type"),
|
||||||
|
[
|
||||||
|
(401, SourceDeadError),
|
||||||
|
(429, TransientError),
|
||||||
|
(500, TransientError),
|
||||||
|
(404, RequestRejectedError),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
async def test_download_errors_keep_their_class_and_body(self, status, exc_type):
|
||||||
|
"""GET 阶段不适用确定性窄化,且响应体摘要随每个分支(issue #10)。"""
|
||||||
|
body = _parsing_failed(_DEFAULT_REJECTED_MARKERS[0])
|
||||||
|
t = self._transport(zip_resp=httpx.Response(status, content=body.encode()))
|
||||||
|
with pytest.raises(exc_type) as ei:
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert ei.value.status_code == status and ei.value.body_text == body
|
||||||
|
|
||||||
|
async def test_huge_error_body_is_read_only_up_to_a_bound(self):
|
||||||
|
"""摘要只需要几 KiB;服务(或中间人)回一个巨大的错误体不该被整个读进内存。
|
||||||
|
|
||||||
|
(独立审查实测:600 MiB 的 500 响应让进程 RSS 涨到 1.8 GiB——新代码声称"流式 + 体积上限",
|
||||||
|
错误分支却是例外。)
|
||||||
|
"""
|
||||||
|
consumed = []
|
||||||
|
|
||||||
|
async def endless_error_body():
|
||||||
|
for _ in range(100_000):
|
||||||
|
consumed.append(1)
|
||||||
|
yield b"e" * 1024
|
||||||
|
|
||||||
|
t = self._transport(zip_resp=httpx.Response(500, content=endless_error_body()))
|
||||||
|
with pytest.raises(TransientError) as ei:
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert ei.value.status_code == 500
|
||||||
|
assert len(consumed) <= monkey_ocr._ERROR_BODY_READ_CAP // 1024 + 2 # 只读了上限那么多
|
||||||
|
assert ei.value.body_text.startswith("e" * 100) # 摘要仍在
|
||||||
|
|
||||||
|
async def test_error_status_with_oversized_content_length_is_still_classified_by_http(
|
||||||
|
self, monkeypatch
|
||||||
|
):
|
||||||
|
"""结果包资源检查不得改变错误分类: 超限的 Content-Length 出现在 401 上,仍是 SourceDead。"""
|
||||||
|
monkeypatch.setattr(monkey_ocr, "_MAX_PACKAGE_BYTES", 100)
|
||||||
|
resp = httpx.Response(401, content=b"denied", headers={"content-length": "101"})
|
||||||
|
with pytest.raises(SourceDeadError):
|
||||||
|
await self._transport(zip_resp=resp).parse_document(
|
||||||
|
document=_PDF_BODY, source=_source(), call_id="c1"
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _oversized_valid_package():
|
||||||
|
"""**合法**的 ZIP 但整体超过上限: 去掉上限检查时它能被正常解析,用例才会转红。
|
||||||
|
|
||||||
|
(独立验证发现: 旧用例用非 ZIP 垃圾字节,去掉上限后同样抛 ResultInvalidError——
|
||||||
|
走的是 BadZipFile 路径,上限从未被测到。)
|
||||||
|
"""
|
||||||
|
return _doc_package(
|
||||||
|
[_doc_page([_text_block("SYNTH-A")])], extra=[("images/big.jpg", b"x" * 5000)]
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _assert_cap_rejects_before_parsing(self, monkeypatch, resp):
|
||||||
|
monkeypatch.setattr(monkey_ocr, "_MAX_PACKAGE_BYTES", 2000)
|
||||||
|
parsed = []
|
||||||
|
|
||||||
|
def spy(package):
|
||||||
|
parsed.append(len(package))
|
||||||
|
return [], [(1.0, 1.0)], ""
|
||||||
|
|
||||||
|
monkeypatch.setattr(monkey_ocr, "parse_document_package", spy)
|
||||||
|
with pytest.raises(ResultInvalidError, match="体积超过上限"):
|
||||||
|
await self._transport(zip_resp=resp).parse_document(
|
||||||
|
document=_PDF_BODY, source=_source(), call_id="c1"
|
||||||
|
)
|
||||||
|
assert parsed == [] # 解析函数根本没被调用
|
||||||
|
|
||||||
|
async def test_oversized_valid_package_is_rejected_by_declared_content_length(
|
||||||
|
self, monkeypatch
|
||||||
|
):
|
||||||
|
resp = httpx.Response(200, content=self._oversized_valid_package()) # 自动带 Content-Length
|
||||||
|
await self._assert_cap_rejects_before_parsing(monkeypatch, resp)
|
||||||
|
|
||||||
|
async def test_oversized_valid_package_is_rejected_by_cumulative_count(self, monkeypatch):
|
||||||
|
"""没有 Content-Length(分块传输)时靠累计字节兜底。"""
|
||||||
|
package = self._oversized_valid_package()
|
||||||
|
|
||||||
|
async def chunks():
|
||||||
|
for offset in range(0, len(package), 700):
|
||||||
|
yield package[offset : offset + 700]
|
||||||
|
|
||||||
|
await self._assert_cap_rejects_before_parsing(
|
||||||
|
monkeypatch, httpx.Response(200, content=chunks())
|
||||||
|
)
|
||||||
|
|
||||||
|
async def test_package_under_the_cap_still_parses(self, monkeypatch):
|
||||||
|
"""对照: 上限只拦超限的,不误伤正常包。"""
|
||||||
|
monkeypatch.setattr(monkey_ocr, "_MAX_PACKAGE_BYTES", 10_000)
|
||||||
|
result = await self._transport(
|
||||||
|
zip_resp=httpx.Response(200, content=self._oversized_valid_package())
|
||||||
|
).parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert [e.text for e in result.elements] == ["SYNTH-A"]
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("status", "exc_type"),
|
||||||
|
[
|
||||||
|
(401, SourceDeadError),
|
||||||
|
(404, RequestRejectedError),
|
||||||
|
(500, TransientError),
|
||||||
|
(302, RequestRejectedError), # 3xx 不是 `is_error`,但 raise_for_status 会抛
|
||||||
|
],
|
||||||
|
)
|
||||||
|
async def test_streamed_error_body_is_read_before_classification(self, status, exc_type):
|
||||||
|
"""真正的流式响应(异步迭代器 body)不会被预缓冲: 不先 `aread()`,摘要读到的是空串。
|
||||||
|
|
||||||
|
(独立验证发现: `httpx.Response(content=bytes)` 已预缓冲,删掉 `aread()` 后旧用例仍绿;
|
||||||
|
且旧条件 `is_error` 漏掉 3xx。)
|
||||||
|
"""
|
||||||
|
body = '{"detail": "synthetic upstream reason"}'
|
||||||
|
|
||||||
|
async def streamed():
|
||||||
|
yield body.encode()
|
||||||
|
|
||||||
|
t = self._transport(zip_resp=httpx.Response(status, content=streamed()))
|
||||||
|
with pytest.raises(exc_type) as ei:
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
assert ei.value.status_code == status
|
||||||
|
assert ei.value.body_text == body
|
||||||
|
|
||||||
|
async def test_bad_package_is_result_invalid(self):
|
||||||
|
t = self._transport(zip_resp=httpx.Response(200, content=b"not a zip"))
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
await t.parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
|
||||||
|
async def test_cancellation_during_parse_does_not_block_the_event_loop(self, monkeypatch):
|
||||||
|
"""解析在线程里跑: 同步解析大对象会占住事件循环,取消与 call_deadline_s 都会失效。
|
||||||
|
|
||||||
|
线程内用 threading.Event 阻塞,并用 call_soon_threadsafe 通知(线程里不能直接用
|
||||||
|
asyncio.Event)。若实现退回同步解析,桩会把循环占满 1 秒后正常返回,
|
||||||
|
`task.cancel()` 就取不到 CancelledError——用例转红。
|
||||||
|
"""
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
entered = asyncio.Event()
|
||||||
|
release = threading.Event()
|
||||||
|
|
||||||
|
def blocking_parse(zip_bytes):
|
||||||
|
loop.call_soon_threadsafe(entered.set)
|
||||||
|
release.wait(timeout=1)
|
||||||
|
return [], [(1.0, 1.0)], ""
|
||||||
|
|
||||||
|
monkeypatch.setattr(monkey_ocr, "parse_document_package", blocking_parse)
|
||||||
|
task = asyncio.create_task(
|
||||||
|
self._transport().parse_document(document=_PDF_BODY, source=_source(), call_id="c1")
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(entered.wait(), timeout=2)
|
||||||
|
await asyncio.sleep(0.01) # 线程还卡着,循环必须仍能调度别的协程
|
||||||
|
task.cancel()
|
||||||
|
with pytest.raises(asyncio.CancelledError):
|
||||||
|
await task
|
||||||
|
finally:
|
||||||
|
release.set() # 让线程收尾,不泄漏
|
||||||
|
|
||||||
|
|
||||||
class TestCheckHealth:
|
class TestCheckHealth:
|
||||||
async def test_healthy(self):
|
async def test_healthy(self):
|
||||||
t = _transport_for(_routes(health_resp=httpx.Response(200, json={"status": "healthy"})))
|
t = _transport_for(_routes(health_resp=httpx.Response(200, json={"status": "healthy"})))
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ retry_exhausted/circuit_open/stalled 三组断言即设计 §6 ③ 的 G1 契约
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import httpx
|
||||||
import pytest
|
import pytest
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
@@ -21,20 +23,34 @@ from polygateway.errors import (
|
|||||||
TransientError,
|
TransientError,
|
||||||
)
|
)
|
||||||
from polygateway.ocr import OcrClient
|
from polygateway.ocr import OcrClient
|
||||||
|
from polygateway.transports.monkey_ocr import MonkeyOcrTransport
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
BackpressurePolicy,
|
BackpressurePolicy,
|
||||||
BreakerConfig,
|
BreakerConfig,
|
||||||
GlobalLimits,
|
GlobalLimits,
|
||||||
|
OcrDocumentElement,
|
||||||
|
OcrDocumentTransportResult,
|
||||||
OcrLayoutElement,
|
OcrLayoutElement,
|
||||||
OcrLayoutTransportResult,
|
OcrLayoutTransportResult,
|
||||||
OcrTextTransportResult,
|
OcrTextTransportResult,
|
||||||
RetryPolicy,
|
RetryPolicy,
|
||||||
SourceConfig,
|
SourceConfig,
|
||||||
)
|
)
|
||||||
|
from tests.unit import _document_fixtures as fx
|
||||||
|
|
||||||
_BREAKER = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0)
|
_BREAKER = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0)
|
||||||
_NO_GLOBAL = GlobalLimits(max_concurrency=0, rpm=0, tpm=0)
|
_NO_GLOBAL = GlobalLimits(max_concurrency=0, rpm=0, tpm=0)
|
||||||
_TEXT_OK = OcrTextTransportResult(text="LINE-1", raw={"task_type": "text"})
|
_TEXT_OK = OcrTextTransportResult(text="LINE-1", raw={"task_type": "text"})
|
||||||
|
_DOC_OK = OcrDocumentTransportResult(
|
||||||
|
elements=[
|
||||||
|
OcrDocumentElement(
|
||||||
|
type="text", bbox=(1.0, 2.0, 3.0, 4.0), page_index=0, text="SYNTH-DOC-BODY"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
page_sizes=[(612.0, 792.0)],
|
||||||
|
markdown="# SYNTH-MD",
|
||||||
|
raw={"success": True},
|
||||||
|
)
|
||||||
_LAYOUT_OK = OcrLayoutTransportResult(
|
_LAYOUT_OK = OcrLayoutTransportResult(
|
||||||
elements=[OcrLayoutElement(type="table", bbox=(41.0, 48.0, 218.0, 282.0), page_index=0)],
|
elements=[OcrLayoutElement(type="table", bbox=(41.0, 48.0, 218.0, 282.0), page_index=0)],
|
||||||
page_sizes=[(759.0, 540.0)],
|
page_sizes=[(759.0, 540.0)],
|
||||||
@@ -72,7 +88,11 @@ class ScriptedOcrTransport:
|
|||||||
if action == "hang":
|
if action == "hang":
|
||||||
self.entered.set()
|
self.entered.set()
|
||||||
await asyncio.Event().wait()
|
await asyncio.Event().wait()
|
||||||
return _TEXT_OK if action == "text" else _LAYOUT_OK
|
if action == "text":
|
||||||
|
return _TEXT_OK
|
||||||
|
if action == "document":
|
||||||
|
return _DOC_OK
|
||||||
|
return _LAYOUT_OK
|
||||||
|
|
||||||
async def recognize_text(self, *, image, source, call_id):
|
async def recognize_text(self, *, image, source, call_id):
|
||||||
return await self._next("text", source, call_id)
|
return await self._next("text", source, call_id)
|
||||||
@@ -80,6 +100,9 @@ class ScriptedOcrTransport:
|
|||||||
async def parse_layout(self, *, image, source, call_id):
|
async def parse_layout(self, *, image, source, call_id):
|
||||||
return await self._next("layout", source, call_id)
|
return await self._next("layout", source, call_id)
|
||||||
|
|
||||||
|
async def parse_document(self, *, document, source, call_id):
|
||||||
|
return await self._next("document", source, call_id)
|
||||||
|
|
||||||
async def check_health(self, *, source):
|
async def check_health(self, *, source):
|
||||||
raise NotImplementedError
|
raise NotImplementedError
|
||||||
|
|
||||||
@@ -548,7 +571,7 @@ class TestOcrCallerDimensions:
|
|||||||
assert [r["tenant_id"] for r in recorder.rows] == ["t1", "t1"]
|
assert [r["tenant_id"] for r in recorder.rows] == ["t1", "t1"]
|
||||||
assert [r["meta"] for r in recorder.rows] == ['{"batch": "b-42"}'] * 2
|
assert [r["meta"] for r in recorder.rows] == ['{"batch": "b-42"}'] * 2
|
||||||
|
|
||||||
@pytest.mark.parametrize("method", ["recognize_text", "parse_layout"])
|
@pytest.mark.parametrize("method", ["recognize_text", "parse_layout", "parse_document"])
|
||||||
async def test_invalid_meta_rejected_before_any_telemetry(self, method):
|
async def test_invalid_meta_rejected_before_any_telemetry(self, method):
|
||||||
"""校验必须早于遥测: 链路内的失败都被降级成 warning,放下游等于没有校验。"""
|
"""校验必须早于遥测: 链路内的失败都被降级成 warning,放下游等于没有校验。"""
|
||||||
recorder = _MemoryRecorder()
|
recorder = _MemoryRecorder()
|
||||||
@@ -558,6 +581,14 @@ class TestOcrCallerDimensions:
|
|||||||
assert recorder.rows == []
|
assert recorder.rows == []
|
||||||
assert client._transport.calls == [] # 连调用都没发出
|
assert client._transport.calls == [] # 连调用都没发出
|
||||||
|
|
||||||
|
async def test_parse_document_row_carries_dimensions(self):
|
||||||
|
"""第三个入口: 只测前两个会漏掉它的透传缺口。"""
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
client, _, _ = _client([_src()], ["document"], telemetry=recorder)
|
||||||
|
await client.parse_document(b"%PDF-1.4", tenant_id="t3", meta={"batch": "b-44"})
|
||||||
|
assert recorder.rows[0]["tenant_id"] == "t3"
|
||||||
|
assert recorder.rows[0]["meta"] == '{"batch": "b-44"}'
|
||||||
|
|
||||||
async def test_defaults_land_as_sentinels(self):
|
async def test_defaults_land_as_sentinels(self):
|
||||||
recorder = _MemoryRecorder()
|
recorder = _MemoryRecorder()
|
||||||
client, _, _ = _client([_src()], ["text"], telemetry=recorder)
|
client, _, _ = _client([_src()], ["text"], telemetry=recorder)
|
||||||
@@ -566,6 +597,408 @@ class TestOcrCallerDimensions:
|
|||||||
assert recorder.rows[0]["meta"] == "{}"
|
assert recorder.rows[0]["meta"] == "{}"
|
||||||
|
|
||||||
|
|
||||||
|
class _LegacyOcrTransport:
|
||||||
|
"""只实现旧 `OcrTransport` 三方法的下游 fake: 不满足 `OcrDocumentTransport`。"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.calls = []
|
||||||
|
|
||||||
|
async def recognize_text(self, *, image, source, call_id):
|
||||||
|
self.calls.append("text")
|
||||||
|
return _TEXT_OK
|
||||||
|
|
||||||
|
async def parse_layout(self, *, image, source, call_id):
|
||||||
|
self.calls.append("layout")
|
||||||
|
return _LAYOUT_OK
|
||||||
|
|
||||||
|
async def check_health(self, *, source):
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
class TestParseDocument:
|
||||||
|
"""#34 设计 §4.2/§6: 与 parse_layout 对称的整份文档入口,治理循环原样复用。"""
|
||||||
|
|
||||||
|
async def test_result_fields(self):
|
||||||
|
client, _, gate = _client([_src()], ["document"])
|
||||||
|
r = await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert [e.text for e in r.elements] == ["SYNTH-DOC-BODY"]
|
||||||
|
assert r.markdown == "# SYNTH-MD" and r.page_sizes == [(612.0, 792.0)]
|
||||||
|
assert r.source_name == "m1"
|
||||||
|
assert r.usage.prompt_tokens == 0 and r.usage.completion_tokens == 0
|
||||||
|
assert r.latency_ms >= 0 and r.call_id and r.raw == {"success": True}
|
||||||
|
assert r.call_stats is not None and r.call_stats.attempts == 1
|
||||||
|
assert gate.successes == [("m1", True)]
|
||||||
|
assert client._selector.outcomes == [("m1", True)]
|
||||||
|
|
||||||
|
async def test_input_validation_names_the_document(self):
|
||||||
|
client, _, _ = _client([_src()], [])
|
||||||
|
with pytest.raises(TypeError, match="document 必须是 bytes"):
|
||||||
|
await client.parse_document("not-bytes")
|
||||||
|
with pytest.raises(ValueError, match="document 不能为空"):
|
||||||
|
await client.parse_document(b"")
|
||||||
|
|
||||||
|
async def test_legacy_entrances_keep_their_error_text(self):
|
||||||
|
"""私有形参改名为 payload 不得改变旧入口的报错文案(含 D9 的业务边界提示)。"""
|
||||||
|
client, _, _ = _client([_src()], [])
|
||||||
|
with pytest.raises(TypeError, match=r"image 必须是 bytes\(路径读取/批量拼帧留业务侧,D9\)"):
|
||||||
|
await client.recognize_text("not-bytes")
|
||||||
|
with pytest.raises(ValueError, match="image 不能为空"):
|
||||||
|
await client.parse_layout(b"")
|
||||||
|
|
||||||
|
async def test_transport_without_document_support_is_an_assembly_error(self):
|
||||||
|
"""装配错误当场报,且早于统计边界: 没有遥测行、没有任何 transport 调用。"""
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
legacy = _LegacyOcrTransport()
|
||||||
|
client, limiter, _ = _client([_src(rpm=100)], [], transport=legacy, telemetry=recorder)
|
||||||
|
with pytest.raises(TypeError, match="OcrDocumentTransport"):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert recorder.rows == [] and legacy.calls == []
|
||||||
|
# 早于准入: 从未向限流后端申请过许可(否则 rpm 已被占用一次)
|
||||||
|
assert (await limiter.source_stats("m1")).rpm_used == 0
|
||||||
|
|
||||||
|
async def test_invalid_call_deadline_is_rejected_like_the_other_entrances(self):
|
||||||
|
client, _, _ = _client([_src()], ["document"])
|
||||||
|
with pytest.raises(ValueError, match="call_deadline_s"):
|
||||||
|
await client.parse_document(b"%PDF-1.4", call_deadline_s=-1)
|
||||||
|
|
||||||
|
async def test_transient_retries_then_succeeds(self):
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src()],
|
||||||
|
[TransientError("boom", status_code=500), "document"],
|
||||||
|
sleep=lambda _d: asyncio.sleep(0),
|
||||||
|
)
|
||||||
|
r = await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert r.markdown == "# SYNTH-MD"
|
||||||
|
assert client._selector.outcomes == [("m1", False), ("m1", True)]
|
||||||
|
|
||||||
|
async def test_source_dead_switches_to_the_second_source_immediately(self):
|
||||||
|
s1, s2 = _src(name="m1"), _src(name="m2", base_url="http://10.77.0.20:7867")
|
||||||
|
client, _, _ = _client(
|
||||||
|
[s1, s2], [SourceDeadError("401", status_code=401), "document"], sleep=_no_sleep
|
||||||
|
)
|
||||||
|
r = await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert r.source_name == "m2"
|
||||||
|
assert client._selector.outcomes == [("m1", False), ("m2", True)]
|
||||||
|
|
||||||
|
async def test_result_invalid_counts_no_attempt_and_no_feed(self):
|
||||||
|
client, _, gate = _client([_src()], [ResultInvalidError("bad package")])
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert gate.successes == [("m1", False)]
|
||||||
|
assert client._selector.outcomes == []
|
||||||
|
|
||||||
|
async def test_deterministic_rejection_with_status_500_counts_as_a_live_service(self):
|
||||||
|
"""服务有响应(500 + 确定性坏输入标记)= 服务活着: 记成功不入窗口,不喂选源器。"""
|
||||||
|
client, _, gate = _client(
|
||||||
|
[_src()], [RequestRejectedError("Parsing failed: is no PDF", status_code=500)]
|
||||||
|
)
|
||||||
|
with pytest.raises(RequestRejectedError):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert gate.successes == [("m1", False)]
|
||||||
|
assert client._selector.outcomes == []
|
||||||
|
|
||||||
|
async def test_local_rejection_does_not_touch_the_breaker(self):
|
||||||
|
client, _, gate = _client([_src()], [RequestRejectedError("unrecognizable")])
|
||||||
|
with pytest.raises(RequestRejectedError):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert gate.successes == []
|
||||||
|
|
||||||
|
async def test_rate_limited_responses_spend_retry_budget_but_not_breaker_failures(self):
|
||||||
|
"""429 是背压不是故障(熔断两通道均不计): 三次 429 耗尽重试后,熔断仍是 CLOSED。"""
|
||||||
|
client, _, gate = _client(
|
||||||
|
[_src()],
|
||||||
|
[TransientError("busy", status_code=429)] * 3,
|
||||||
|
sleep=lambda _d: asyncio.sleep(0),
|
||||||
|
)
|
||||||
|
with pytest.raises(AllSourcesExhausted):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
entry = await gate._inner.try_enter("m1", "check")
|
||||||
|
assert entry.allowed and not entry.is_probe
|
||||||
|
|
||||||
|
async def test_read_timeouts_are_counted_as_source_failures(self):
|
||||||
|
"""对照: 文档路径的 httpx 超时仍是瞬时故障、计熔断(`timeout_s` 不是免责额度)。"""
|
||||||
|
|
||||||
|
def timeout_exc():
|
||||||
|
exc = TransientError("timed out", operation="parse")
|
||||||
|
exc.__cause__ = httpx.ReadTimeout("read timed out")
|
||||||
|
return exc
|
||||||
|
|
||||||
|
client, _, gate = _client(
|
||||||
|
[_src()], [timeout_exc() for _ in range(3)], sleep=lambda _d: asyncio.sleep(0)
|
||||||
|
)
|
||||||
|
with pytest.raises(AllSourcesExhausted) as ei:
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert ei.value.per_source_reasons == {"m1": "timeout"}
|
||||||
|
entry = await gate._inner.try_enter("m1", "check")
|
||||||
|
assert not entry.allowed # 阈值 3 的熔断已开路
|
||||||
|
|
||||||
|
async def test_cancel_releases_permit_and_probe(self):
|
||||||
|
clock = FakeClock()
|
||||||
|
inner = InMemoryGate(config=_BREAKER, now=clock)
|
||||||
|
gate = RecordingGate(inner)
|
||||||
|
transport = ScriptedOcrTransport(["hang"])
|
||||||
|
client, limiter, _ = _client(
|
||||||
|
[_src(max_concurrency=1)], [], now=clock, breaker=gate, transport=transport
|
||||||
|
)
|
||||||
|
entry = await inner.try_enter("m1", "setup")
|
||||||
|
await inner.record_failure(entry, "source_dead", True)
|
||||||
|
clock.advance(61.0) # 越过冷却 → 半开探针
|
||||||
|
task = asyncio.create_task(client.parse_document(b"%PDF-1.4"))
|
||||||
|
await transport.entered.wait()
|
||||||
|
task.cancel()
|
||||||
|
with pytest.raises(asyncio.CancelledError):
|
||||||
|
await task
|
||||||
|
assert gate.probe_releases >= 1
|
||||||
|
assert (await limiter.source_stats("m1")).inflight == 0
|
||||||
|
|
||||||
|
|
||||||
|
async def _no_sleep(_delay):
|
||||||
|
await asyncio.sleep(0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestNoLatencyJudgement:
|
||||||
|
"""长文档不会被判慢(设计 §7): 库内没有任何基于延迟的熔断/选源判定。
|
||||||
|
|
||||||
|
时钟只在 transport 内前进(`ClockAdvancingOcrTransport`),门与 client 共用同一注入钟。
|
||||||
|
probe_ttl_s 取极大值,避免把探针租约接管混进"慢"的断言。
|
||||||
|
"""
|
||||||
|
|
||||||
|
_CFG = BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=1e12)
|
||||||
|
|
||||||
|
async def test_slow_success_leaves_breaker_closed_with_no_failures(self):
|
||||||
|
clock = FakeClock()
|
||||||
|
gate = RecordingGate(InMemoryGate(config=self._CFG, now=clock))
|
||||||
|
transport = ClockAdvancingOcrTransport([(100_000.0, "document")], clock)
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src(timeout_s=120.0)], [], now=clock, breaker=gate, transport=transport
|
||||||
|
)
|
||||||
|
r = await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert r.call_stats.attempts == 1 and r.call_stats.generation_ms >= 100_000_000
|
||||||
|
assert gate._inner._gates["m1"].fails == 0
|
||||||
|
assert gate.successes == [("m1", True)]
|
||||||
|
|
||||||
|
async def test_slow_success_clears_pre_existing_failure_count(self):
|
||||||
|
"""前置状态不是初始 CLOSED: 已有 2 次失败(阈值 3),一次很慢的成功也只会清零。"""
|
||||||
|
clock = FakeClock()
|
||||||
|
inner = InMemoryGate(config=self._CFG, now=clock)
|
||||||
|
for _ in range(2):
|
||||||
|
entry = await inner.try_enter("m1", "setup")
|
||||||
|
await inner.record_failure(entry, "network_error", False)
|
||||||
|
assert inner._gates["m1"].fails == 2
|
||||||
|
transport = ClockAdvancingOcrTransport([(100_000.0, "document")], clock)
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src()], [], now=clock, breaker=RecordingGate(inner), transport=transport
|
||||||
|
)
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert inner._gates["m1"].fails == 0
|
||||||
|
|
||||||
|
async def test_slow_success_on_the_half_open_probe_closes_the_breaker(self):
|
||||||
|
clock = FakeClock()
|
||||||
|
inner = InMemoryGate(config=self._CFG, now=clock)
|
||||||
|
entry = await inner.try_enter("m1", "setup")
|
||||||
|
await inner.record_failure(entry, "source_dead", True) # OPEN
|
||||||
|
clock.advance(61.0) # → HALF_OPEN
|
||||||
|
transport = ClockAdvancingOcrTransport([(100_000.0, "document")], clock)
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src()], [], now=clock, breaker=RecordingGate(inner), transport=transport
|
||||||
|
)
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
after = await inner.try_enter("m1", "check")
|
||||||
|
assert after.allowed and not after.is_probe # 探针成功 → CLOSED,而不是被慢耗尽
|
||||||
|
|
||||||
|
|
||||||
|
class TestDocumentTelemetry:
|
||||||
|
async def test_success_row(self):
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
client, _, _ = _client([_src()], ["document"], telemetry=recorder)
|
||||||
|
await client.parse_document(b"%PDF-RAW-BYTES")
|
||||||
|
(row,) = recorder.rows
|
||||||
|
assert row["operation"] == "parse_document"
|
||||||
|
assert "<ocr:document image_bytes=14>" in row["messages"]
|
||||||
|
assert "RAW-BYTES" not in row["messages"] # 文档字节绝不入库
|
||||||
|
assert row["response"] == "<document elements=1 pages=1 md_chars=10>"
|
||||||
|
assert "SYNTH-MD" not in row["response"] and "SYNTH-DOC-BODY" not in row["response"]
|
||||||
|
assert row["reasoning_effort"] is None # `reasoning_applies=False` 的落库形态
|
||||||
|
assert row["error"] is None and row["prompt_tokens"] == 0
|
||||||
|
|
||||||
|
async def test_summary_distinguishes_the_three_result_kinds(self):
|
||||||
|
"""新结果类型必须在 `_summarize` 里显式区分,否则会被静默套成 layout 摘要而漏记 md_chars。"""
|
||||||
|
assert OcrClient._summarize(_DOC_OK) == "<document elements=1 pages=1 md_chars=10>"
|
||||||
|
assert OcrClient._summarize(_LAYOUT_OK) == "<elements n=1 pages=1>"
|
||||||
|
assert OcrClient._summarize(_TEXT_OK) == "LINE-1"
|
||||||
|
|
||||||
|
def test_summary_keeps_structural_dispatch_for_duck_typed_results(self):
|
||||||
|
"""旧实现按结构分派(`hasattr(result, "text")`)。下游测试替身可能返回非库类型的鸭子对象,
|
||||||
|
改成 isinstance 会让它们在开启遥测后以原生 AttributeError 漏出——无登记的回归。"""
|
||||||
|
assert OcrClient._summarize(SimpleNamespace(text="hello", raw={})) == "hello"
|
||||||
|
assert (
|
||||||
|
OcrClient._summarize(SimpleNamespace(elements=[1, 2], page_sizes=[(1.0, 1.0)]))
|
||||||
|
== "<elements n=2 pages=1>"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
OcrClient._summarize(
|
||||||
|
SimpleNamespace(elements=[1], page_sizes=[(1.0, 1.0)], markdown="abc")
|
||||||
|
)
|
||||||
|
== "<document elements=1 pages=1 md_chars=3>"
|
||||||
|
)
|
||||||
|
|
||||||
|
async def test_failed_attempt_row_carries_the_class_prefix(self):
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
client, _, _ = _client([_src()], [ResultInvalidError("bad package")], telemetry=recorder)
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
assert any(r["error"].startswith("ResultInvalidError:") for r in recorder.rows)
|
||||||
|
assert {r["operation"] for r in recorder.rows} == {"parse_document"}
|
||||||
|
|
||||||
|
|
||||||
|
def _real_transport(handler):
|
||||||
|
return MonkeyOcrTransport(
|
||||||
|
client_factory=lambda src: httpx.AsyncClient(
|
||||||
|
base_url=src.base_url, transport=httpx.MockTransport(handler)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _doc_handler(package: bytes | None = None, *, download=None):
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
if request.url.path == "/parse":
|
||||||
|
return httpx.Response(
|
||||||
|
200,
|
||||||
|
json={"success": True, "message": "ok", "download_url": "/static/doc.zip"},
|
||||||
|
)
|
||||||
|
if download is not None:
|
||||||
|
return download()
|
||||||
|
return httpx.Response(200, content=package)
|
||||||
|
|
||||||
|
return handler
|
||||||
|
|
||||||
|
|
||||||
|
class TestDocumentThroughTheRealTransport:
|
||||||
|
"""真实 `MonkeyOcrTransport` + `MockTransport` + 真实 `OcrClient`: 证明异常/期限贯穿整条链。
|
||||||
|
|
||||||
|
MockTransport 不执行 httpcore 的逐次读超时,所以本组只证明**治理期限与异常分类**,
|
||||||
|
不证明 read timeout 的真实行为(后者留给活体用例)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def test_happy_path_end_to_end(self):
|
||||||
|
package = fx._package([fx._page([fx._text_block("SYNTH-E2E")])])
|
||||||
|
client, _, _ = _client([_src()], [], transport=_real_transport(_doc_handler(package)))
|
||||||
|
r = await client.parse_document(b"%PDF-1.4 synthetic")
|
||||||
|
assert [e.text for e in r.elements] == ["SYNTH-E2E"]
|
||||||
|
assert r.markdown == "# synth\n"
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("package_kind", ["huge-int", "encrypted"])
|
||||||
|
async def test_parser_exceptions_become_result_invalid_with_full_bookkeeping(
|
||||||
|
self, package_kind
|
||||||
|
):
|
||||||
|
"""真实解析器抛出的异常(OverflowError / RuntimeError)经真实客户端: 终态遥测行已写、
|
||||||
|
半开探针已归还、熔断窗口不变——不能只用桩直接抛预制的 `ResultInvalidError`。"""
|
||||||
|
if package_kind == "huge-int":
|
||||||
|
block = fx._text_block()
|
||||||
|
block["bbox"] = [0, 0, 10**400, 1]
|
||||||
|
package = fx._package([fx._page([block])])
|
||||||
|
else:
|
||||||
|
package = fx._encrypted_package()
|
||||||
|
clock = FakeClock()
|
||||||
|
inner = InMemoryGate(
|
||||||
|
config=BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0), now=clock
|
||||||
|
)
|
||||||
|
entry = await inner.try_enter("m1", "setup")
|
||||||
|
await inner.record_failure(entry, "source_dead", True)
|
||||||
|
clock.advance(61.0) # 半开: 本次调用持有探针
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
client, _, gate = _client(
|
||||||
|
[_src()],
|
||||||
|
[],
|
||||||
|
now=clock,
|
||||||
|
breaker=RecordingGate(inner),
|
||||||
|
telemetry=recorder,
|
||||||
|
transport=_real_transport(_doc_handler(package)),
|
||||||
|
)
|
||||||
|
with pytest.raises(ResultInvalidError):
|
||||||
|
await client.parse_document(b"%PDF-1.4 synthetic")
|
||||||
|
assert any(r["error"].startswith("ResultInvalidError:") for r in recorder.rows)
|
||||||
|
after = await inner.try_enter("m1", "check")
|
||||||
|
assert after.allowed and not after.is_probe # 探针已按"坏结果≠坏服务"结算,没有悬挂
|
||||||
|
assert inner._gates["m1"].fails == 0
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _slow_download_handler():
|
||||||
|
async def slow_body():
|
||||||
|
for _ in range(200):
|
||||||
|
await asyncio.sleep(0.01)
|
||||||
|
yield b"x" * 8
|
||||||
|
|
||||||
|
return _doc_handler(download=lambda: httpx.Response(200, content=slow_body()))
|
||||||
|
|
||||||
|
async def test_per_call_deadline_overrides_the_assembly_value(self):
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src(timeout_s=120.0)], [], transport=_real_transport(self._slow_download_handler())
|
||||||
|
) # 装配值未设期限
|
||||||
|
with pytest.raises(CallDeadlineExceeded):
|
||||||
|
await client.parse_document(b"%PDF-1.4 synthetic", call_deadline_s=0.1)
|
||||||
|
|
||||||
|
async def test_deadline_while_holding_the_half_open_probe_returns_it_and_writes_a_terminal_row(
|
||||||
|
self,
|
||||||
|
):
|
||||||
|
"""期限到期取消在飞请求时: 探针必须归还,且终态遥测行必须写出(遥测必录)。"""
|
||||||
|
clock = FakeClock()
|
||||||
|
inner = InMemoryGate(
|
||||||
|
config=BreakerConfig(fail_threshold=3, cooldown_s=60.0, probe_ttl_s=120.0), now=clock
|
||||||
|
)
|
||||||
|
entry = await inner.try_enter("m1", "setup")
|
||||||
|
await inner.record_failure(entry, "source_dead", True)
|
||||||
|
clock.advance(61.0) # 半开: 本次调用持有探针
|
||||||
|
gate = RecordingGate(inner)
|
||||||
|
recorder = _MemoryRecorder()
|
||||||
|
client, _, _ = _client(
|
||||||
|
[_src(timeout_s=120.0)],
|
||||||
|
[],
|
||||||
|
now=clock,
|
||||||
|
breaker=gate,
|
||||||
|
telemetry=recorder,
|
||||||
|
transport=_real_transport(self._slow_download_handler()),
|
||||||
|
call_deadline_s=0.1,
|
||||||
|
)
|
||||||
|
with pytest.raises(CallDeadlineExceeded):
|
||||||
|
await client.parse_document(b"%PDF-1.4 synthetic")
|
||||||
|
assert gate.probe_releases >= 1
|
||||||
|
again = await inner.try_enter("m1", "check")
|
||||||
|
assert again.allowed and again.is_probe # 探针没有悬挂,下一位调用方可以再探
|
||||||
|
terminals = [r for r in recorder.rows if r["event_kind"] == "terminal_failure"]
|
||||||
|
assert len(terminals) == 1 and terminals[0]["operation"] == "parse_document"
|
||||||
|
|
||||||
|
async def test_sustained_slow_download_hits_the_call_deadline_not_a_transient_error(self):
|
||||||
|
"""每个分块都在 read 超时内、总时长超过 call_deadline_s: 是期限,不是瞬时故障。"""
|
||||||
|
closed = []
|
||||||
|
produced = []
|
||||||
|
|
||||||
|
async def slow_body():
|
||||||
|
try:
|
||||||
|
for _ in range(200):
|
||||||
|
await asyncio.sleep(0.01)
|
||||||
|
produced.append(1)
|
||||||
|
yield b"x" * 8
|
||||||
|
finally:
|
||||||
|
closed.append(True)
|
||||||
|
|
||||||
|
handler = _doc_handler(download=lambda: httpx.Response(200, content=slow_body()))
|
||||||
|
client, limiter, gate = _client(
|
||||||
|
[_src(max_concurrency=1, timeout_s=120.0)],
|
||||||
|
[],
|
||||||
|
transport=_real_transport(handler),
|
||||||
|
call_deadline_s=0.1,
|
||||||
|
)
|
||||||
|
with pytest.raises(CallDeadlineExceeded):
|
||||||
|
await client.parse_document(b"%PDF-1.4 synthetic")
|
||||||
|
assert len(produced) >= 2 # 确实进入了多块下载,而不是一开始就超时
|
||||||
|
assert closed == [True] # 流已关闭
|
||||||
|
assert (await limiter.source_stats("m1")).inflight == 0 # permit 释放
|
||||||
|
assert gate.successes == [] # 期限不是服务故障也不是成功
|
||||||
|
entry = await gate._inner.try_enter("m1", "check")
|
||||||
|
assert entry.allowed and not entry.is_probe # 熔断窗口不变
|
||||||
|
|
||||||
|
|
||||||
class TestAssembly:
|
class TestAssembly:
|
||||||
_ENV = {
|
_ENV = {
|
||||||
"OCR__MONKEY__1__BASE_URL": "http://10.77.0.20:7866",
|
"OCR__MONKEY__1__BASE_URL": "http://10.77.0.20:7866",
|
||||||
@@ -600,10 +1033,14 @@ class TestAssembly:
|
|||||||
|
|
||||||
|
|
||||||
class TestReasonlessTelemetryContract:
|
class TestReasonlessTelemetryContract:
|
||||||
"""text/layout 两个入口分别验证错误行不受源级推理配置污染。"""
|
"""text/layout/document 三个入口分别验证错误行不受源级推理配置污染。
|
||||||
|
|
||||||
|
独立验证发现文档入口原先不在矩阵里: 让 OCR 一律按 `reasoning_applies=True` 记账,
|
||||||
|
15 个旧用例转红,但没有一个是 `parse_document` 的。"""
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
"method,action", [("recognize_text", "text"), ("parse_layout", "layout")]
|
"method,action",
|
||||||
|
[("recognize_text", "text"), ("parse_layout", "layout"), ("parse_document", "document")],
|
||||||
)
|
)
|
||||||
@pytest.mark.parametrize("config", [{"enable_thinking": True}, {"reasoning_effort": "high"}])
|
@pytest.mark.parametrize("config", [{"enable_thinking": True}, {"reasoning_effort": "high"}])
|
||||||
@pytest.mark.parametrize("backend", ["memory", "sqlite"])
|
@pytest.mark.parametrize("backend", ["memory", "sqlite"])
|
||||||
@@ -636,7 +1073,7 @@ class TestReasonlessTelemetryContract:
|
|||||||
if backend == "sqlite":
|
if backend == "sqlite":
|
||||||
recorder.close()
|
recorder.close()
|
||||||
|
|
||||||
@pytest.mark.parametrize("method", ["recognize_text", "parse_layout"])
|
@pytest.mark.parametrize("method", ["recognize_text", "parse_layout", "parse_document"])
|
||||||
@pytest.mark.parametrize("exhausted", [False, True])
|
@pytest.mark.parametrize("exhausted", [False, True])
|
||||||
async def test_failed_attempts_still_have_null_effort(self, method, exhausted):
|
async def test_failed_attempts_still_have_null_effort(self, method, exhausted):
|
||||||
script = (
|
script = (
|
||||||
|
|||||||
@@ -22,6 +22,8 @@ def test_ocr_public_surface_exported():
|
|||||||
"OcrTextResult",
|
"OcrTextResult",
|
||||||
"OcrLayoutResult",
|
"OcrLayoutResult",
|
||||||
"OcrLayoutElement",
|
"OcrLayoutElement",
|
||||||
|
"OcrDocumentResult",
|
||||||
|
"OcrDocumentElement",
|
||||||
):
|
):
|
||||||
assert hasattr(polygateway, name), name
|
assert hasattr(polygateway, name), name
|
||||||
assert name in polygateway.__all__, name
|
assert name in polygateway.__all__, name
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from polygateway.ports import (
|
|||||||
GateState,
|
GateState,
|
||||||
GateUpdate,
|
GateUpdate,
|
||||||
Middleware,
|
Middleware,
|
||||||
|
OcrDocumentTransport,
|
||||||
OcrTransport,
|
OcrTransport,
|
||||||
Permit,
|
Permit,
|
||||||
ProviderGate,
|
ProviderGate,
|
||||||
@@ -193,6 +194,7 @@ class TestReasoningTierIsOnlyOnTheChatPort:
|
|||||||
(EmbeddingTransport, "embed"),
|
(EmbeddingTransport, "embed"),
|
||||||
(OcrTransport, "recognize_text"),
|
(OcrTransport, "recognize_text"),
|
||||||
(OcrTransport, "parse_layout"),
|
(OcrTransport, "parse_layout"),
|
||||||
|
(OcrDocumentTransport, "parse_document"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_other_transports_have_no_reasoning_tier(self, protocol, method):
|
def test_other_transports_have_no_reasoning_tier(self, protocol, method):
|
||||||
@@ -372,6 +374,34 @@ class TestOcrPorts:
|
|||||||
assert isinstance(GoodClient(), OcrLayoutPort)
|
assert isinstance(GoodClient(), OcrLayoutPort)
|
||||||
assert isinstance(GoodTransport(), OcrTransport)
|
assert isinstance(GoodTransport(), OcrTransport)
|
||||||
|
|
||||||
|
def test_document_ports_are_runtime_checkable(self):
|
||||||
|
from polygateway.ports import OcrDocumentPort, OcrDocumentTransport
|
||||||
|
|
||||||
|
class GoodClient:
|
||||||
|
async def parse_document(self, document): ...
|
||||||
|
|
||||||
|
class GoodTransport:
|
||||||
|
async def parse_document(self, *, document, source, call_id): ...
|
||||||
|
|
||||||
|
class Empty: ...
|
||||||
|
|
||||||
|
assert isinstance(GoodClient(), OcrDocumentPort)
|
||||||
|
assert isinstance(GoodTransport(), OcrDocumentTransport)
|
||||||
|
assert not isinstance(Empty(), OcrDocumentPort)
|
||||||
|
assert not isinstance(Empty(), OcrDocumentTransport)
|
||||||
|
|
||||||
|
def test_ocr_transport_members_are_unchanged(self):
|
||||||
|
"""新能力走独立 Protocol: 往旧 runtime_checkable Protocol 加成员,会让所有
|
||||||
|
现有实现(测试替身、下游自带 fake)在 isinstance 上一夜之间不合格。"""
|
||||||
|
from polygateway.ports import OcrTransport
|
||||||
|
|
||||||
|
members = {
|
||||||
|
name
|
||||||
|
for name in vars(OcrTransport)
|
||||||
|
if not name.startswith("_") and callable(getattr(OcrTransport, name))
|
||||||
|
}
|
||||||
|
assert members == {"recognize_text", "parse_layout", "check_health"}
|
||||||
|
|
||||||
def test_missing_method_rejected(self):
|
def test_missing_method_rejected(self):
|
||||||
from polygateway.ports import OcrTransport
|
from polygateway.ports import OcrTransport
|
||||||
|
|
||||||
|
|||||||
@@ -37,6 +37,8 @@ from polygateway.types import (
|
|||||||
EmbeddingTransportResult,
|
EmbeddingTransportResult,
|
||||||
GlobalLimits,
|
GlobalLimits,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
|
OcrDocumentElement,
|
||||||
|
OcrDocumentTransportResult,
|
||||||
OcrTextTransportResult,
|
OcrTextTransportResult,
|
||||||
RetryPolicy,
|
RetryPolicy,
|
||||||
SourceConfig,
|
SourceConfig,
|
||||||
@@ -2422,6 +2424,18 @@ class _StubOcrTransport:
|
|||||||
async def parse_layout(self, *, image, source, call_id):
|
async def parse_layout(self, *, image, source, call_id):
|
||||||
raise NotImplementedError
|
raise NotImplementedError
|
||||||
|
|
||||||
|
async def parse_document(self, *, document, source, call_id):
|
||||||
|
return OcrDocumentTransportResult(
|
||||||
|
elements=[
|
||||||
|
OcrDocumentElement(
|
||||||
|
type="text", bbox=(0.0, 0.0, 1.0, 1.0), page_index=0, text="SYNTH-BODY"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
page_sizes=[(612.0, 792.0)],
|
||||||
|
markdown="# SYNTH-MD",
|
||||||
|
raw={"success": True},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _governance(scope, sources):
|
def _governance(scope, sources):
|
||||||
"""embed/OCR 两条链路共用的最小治理装配(真实内存后端,不 mock)。"""
|
"""embed/OCR 两条链路共用的最小治理装配(真实内存后端,不 mock)。"""
|
||||||
@@ -2478,6 +2492,20 @@ class TestTextCapCoversEmbedAndOcrChains:
|
|||||||
assert json.loads(row["messages"])[0]["content"] == "<ocr:tex…(略 16 字)"
|
assert json.loads(row["messages"])[0]["content"] == "<ocr:tex…(略 16 字)"
|
||||||
assert row["response"] == "识别结果识别结果…(略 32 字)" # 先经 OCR 自有的 200 字上限
|
assert row["response"] == "识别结果识别结果…(略 32 字)" # 先经 OCR 自有的 200 字上限
|
||||||
|
|
||||||
|
async def test_ocr_document_rows_are_capped(self):
|
||||||
|
"""第三个 OCR 入口的占位串同样受 text_cap 约束(单一遥测出口)。"""
|
||||||
|
rec = _MemoryRecorder()
|
||||||
|
client = OcrClient(
|
||||||
|
**_governance("ocr", [_source(name="m1", model="monkey-ocr")]),
|
||||||
|
transport=_StubOcrTransport(),
|
||||||
|
telemetry=rec,
|
||||||
|
text_cap=8,
|
||||||
|
)
|
||||||
|
await client.parse_document(b"%PDF-1.4")
|
||||||
|
row = rec.rows[0]
|
||||||
|
assert json.loads(row["messages"])[0]["content"] == "<ocr:doc…(略 20 字)"
|
||||||
|
assert row["operation"] == "parse_document"
|
||||||
|
|
||||||
|
|
||||||
# —— 遥测降级状态(issue #15 C 组): 共用 tracker + 只读快照 + 节流日志 ——
|
# —— 遥测降级状态(issue #15 C 组): 共用 tracker + 只读快照 + 节流日志 ——
|
||||||
|
|
||||||
@@ -3436,7 +3464,9 @@ class TestRowLevelObservability:
|
|||||||
assert error.count("v" * 200) == 3
|
assert error.count("v" * 200) == 3
|
||||||
assert len(error) < 1200
|
assert len(error) < 1200
|
||||||
|
|
||||||
@pytest.mark.parametrize("operation", ["chat", "embed", "recognize_text", "parse_layout"])
|
@pytest.mark.parametrize(
|
||||||
|
"operation", ["chat", "embed", "recognize_text", "parse_layout", "parse_document"]
|
||||||
|
)
|
||||||
async def test_operation_is_given_by_the_call_site_not_the_exception(self, operation):
|
async def test_operation_is_given_by_the_call_site_not_the_exception(self, operation):
|
||||||
"""新列 `operation` 与 `exc.operation` 是两个语义(设计 §5 I1/I2)。"""
|
"""新列 `operation` 与 `exc.operation` 是两个语义(设计 §5 I1/I2)。"""
|
||||||
exc = TransientError("boom", status_code=500, operation="download_result")
|
exc = TransientError("boom", status_code=500, operation="download_result")
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import inspect
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from polygateway.types import (
|
from polygateway.types import (
|
||||||
|
CALL_OPERATIONS,
|
||||||
USAGE_SOURCES,
|
USAGE_SOURCES,
|
||||||
BackpressurePolicy,
|
BackpressurePolicy,
|
||||||
BreakerConfig,
|
BreakerConfig,
|
||||||
@@ -13,6 +14,10 @@ from polygateway.types import (
|
|||||||
Effort,
|
Effort,
|
||||||
GlobalLimits,
|
GlobalLimits,
|
||||||
LLMResponse,
|
LLMResponse,
|
||||||
|
OcrDocumentElement,
|
||||||
|
OcrDocumentResult,
|
||||||
|
OcrDocumentTransportResult,
|
||||||
|
OcrLayoutElement,
|
||||||
RetryPolicy,
|
RetryPolicy,
|
||||||
SourceConfig,
|
SourceConfig,
|
||||||
ThinkingObservation,
|
ThinkingObservation,
|
||||||
@@ -849,3 +854,61 @@ class _FakeMonotonic:
|
|||||||
|
|
||||||
def advance(self, seconds: float) -> None:
|
def advance(self, seconds: float) -> None:
|
||||||
self.t += seconds
|
self.t += seconds
|
||||||
|
|
||||||
|
|
||||||
|
class TestOcrDocumentTypes:
|
||||||
|
"""#34 设计 §4.1: 文档元素是版面元素 + 内容;结果独立定义。"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _element(**overrides):
|
||||||
|
base = {"type": "text", "bbox": (0.0, 0.0, 1.0, 1.0), "page_index": 0, "text": "x"}
|
||||||
|
base.update(overrides)
|
||||||
|
return OcrDocumentElement(**base)
|
||||||
|
|
||||||
|
def test_document_element_is_a_layout_element(self):
|
||||||
|
"""继承的价值: 按 bbox 裁剪的下游函数可以同时吃两种元素。"""
|
||||||
|
assert isinstance(self._element(), OcrLayoutElement)
|
||||||
|
|
||||||
|
def test_document_element_never_equals_a_layout_element(self):
|
||||||
|
"""dataclass 的 `__eq__` 要求同类——已知语义,钉死以免下游拿版面元素去比对文档结果。"""
|
||||||
|
layout = OcrLayoutElement(type="text", bbox=(0.0, 0.0, 1.0, 1.0), page_index=0)
|
||||||
|
assert self._element() != layout
|
||||||
|
|
||||||
|
def test_new_types_are_frozen(self):
|
||||||
|
element = self._element()
|
||||||
|
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||||
|
element.text = "y" # type: ignore[misc]
|
||||||
|
transport = OcrDocumentTransportResult(elements=[], page_sizes=[], markdown="", raw={})
|
||||||
|
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||||
|
transport.markdown = "y" # type: ignore[misc]
|
||||||
|
|
||||||
|
def test_result_is_not_a_subclass_of_layout_result(self):
|
||||||
|
"""父类末尾 `call_stats` 带默认值,子类无默认字段 `markdown` 会在导入期 TypeError。"""
|
||||||
|
from polygateway.types import OcrLayoutResult
|
||||||
|
|
||||||
|
assert not issubclass(OcrDocumentResult, OcrLayoutResult)
|
||||||
|
|
||||||
|
def test_result_carries_provenance_five_plus_markdown(self):
|
||||||
|
fields = [f.name for f in dataclasses.fields(OcrDocumentResult)]
|
||||||
|
assert fields == [
|
||||||
|
"elements",
|
||||||
|
"page_sizes",
|
||||||
|
"markdown",
|
||||||
|
"source_name",
|
||||||
|
"usage",
|
||||||
|
"latency_ms",
|
||||||
|
"call_id",
|
||||||
|
"raw",
|
||||||
|
"call_stats",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class TestCallOperations:
|
||||||
|
def test_call_operations_matches_the_literal(self):
|
||||||
|
"""`CALL_OPERATIONS` 此前没有任何读者,与 `CallOperation` 漂移也无人发现。"""
|
||||||
|
from typing import get_args
|
||||||
|
|
||||||
|
from polygateway.types import CallOperation
|
||||||
|
|
||||||
|
assert get_args(CallOperation) == CALL_OPERATIONS
|
||||||
|
assert "parse_document" in CALL_OPERATIONS
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ _SIGNATURE_TARGETS = [
|
|||||||
("EmbeddingClient.embed", EmbeddingClient.embed),
|
("EmbeddingClient.embed", EmbeddingClient.embed),
|
||||||
("OcrClient.from_env", OcrClient.from_env),
|
("OcrClient.from_env", OcrClient.from_env),
|
||||||
("OcrClient.recognize_text", OcrClient.recognize_text),
|
("OcrClient.recognize_text", OcrClient.recognize_text),
|
||||||
|
("OcrClient.parse_document", OcrClient.parse_document),
|
||||||
("gather_bounded", polygateway.gather_bounded),
|
("gather_bounded", polygateway.gather_bounded),
|
||||||
("register_provider", register_provider),
|
("register_provider", register_provider),
|
||||||
]
|
]
|
||||||
|
|||||||
Reference in New Issue
Block a user