实现 ClinDB 第一批清洗组件
This commit is contained in:
@@ -6,10 +6,9 @@
|
|||||||
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
||||||
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
||||||
|
|
||||||
仓库当前已从纯文档治理进入第一版核心实现阶段：已经提供可安装的 Python 内存处理包和测试，用于验证
|
仓库当前已从纯文档治理进入第一版核心和 ClinDB 第一批组件实现阶段：已经提供可安装的 Python 内存处理包、
|
||||||
组件组合、精确修改和审计协议，并已有一个严格整行匹配的论文清洗组件。仓库另有一个只供本地评审的实验脚本，
|
8 个论文清洗组件和本地实验入口，能够保存指定论文的成功输出、审计和 diff。仓库仍不提供面向任意数据集的完整规则集、
|
||||||
可以保存指定论文的成功输出、审计和 diff。仓库仍不提供完整规则集、公共命令行工具、通用文件适配器或生产接口，
|
公共命令行工具、通用文件适配器或生产接口，因此目前还不是拿来即可完成任意 Markdown 清洗的成品工具。
|
||||||
因此目前还不是拿来即可完成整篇文档清洗的成品工具。
|
|
||||||
|
|
||||||
## 当前阶段
|
## 当前阶段
|
||||||
|
|
||||||
@@ -17,14 +16,15 @@ profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
|||||||
|
|
||||||
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
||||||
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
||||||
- 当前唯一正式组件是 `paper.arxiv_submission_stamp`，只删除严格整行匹配的 arXiv 提交边栏戳；
|
- 已实现 ClinDB 第一批 8 个正式组件，覆盖 Word 批注、手稿行号、arXiv 戳、重复页眉、批准映射断词、
|
||||||
|
HTML 表格实体与布局、参考文献空行；
|
||||||
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计和 unified diff，并保持输入不变；
|
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计和 unified diff，并保持输入不变；
|
||||||
- 该组件已对 5 份论文 Markdown 完成保存型实验，只修改 sim 和 springer 各一处，合法参考文献保持不变；
|
- 第一批流水线已对 5 份论文 Markdown 完成保存型实验，5/5 成功，共记录 155 条修改，第二次运行零修改；
|
||||||
- 当前基础检查为 Ruff、mypy 和 113 项 pytest 测试，实际命令见本文“当前可用检查”。
|
- 当前基础检查为 Ruff、mypy 和 204 项 pytest 测试，实际命令见本文“当前可用检查”。
|
||||||
|
|
||||||
项目还没有完整清洗规则集、Markdown/HTML parser、profile 格式、通用文件输入接口、公共 CLI、通用批处理或生产接口。
|
项目还没有面向任意数据集的完整清洗规则集、Markdown/HTML 通用 parser、profile 格式、通用文件输入接口、公共 CLI、
|
||||||
当前实验脚本只固定运行已批准的 5 份论文和一个组件；它只能证明第一条严格规则已经闭环，不能据此认为论文、GovDoc、
|
通用批处理或生产接口。当前 first-batch 实验脚本只固定运行已批准的 5 份论文和 8 个组件；它只能证明当前 8 类确定规则
|
||||||
表格或图片已经具备完整清洗能力。
|
已经闭环，不能据此认为论文中的缺失内容、乱码、复杂表格、图片或 GovDoc 已具备完整清洗能力。
|
||||||
|
|
||||||
## 服务对象与复用目标
|
## 服务对象与复用目标
|
||||||
|
|
||||||
@@ -77,21 +77,39 @@ mdpolish/
|
|||||||
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
||||||
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
||||||
│ ├── artifact_store.py # 私有产物目录和原子发布
|
│ ├── artifact_store.py # 私有产物目录和原子发布
|
||||||
|
│ ├── _html_table.py # 严格 HTML 表格词法范围
|
||||||
|
│ ├── _text_ranges.py # 精确物理行与换行范围
|
||||||
│ ├── py.typed # 类型信息声明
|
│ ├── py.typed # 类型信息声明
|
||||||
│ └── components/
|
│ └── components/
|
||||||
│ ├── __init__.py
|
│ ├── __init__.py
|
||||||
│ └── arxiv_submission_stamp.py
|
│ ├── arxiv_submission_stamp.py
|
||||||
|
│ ├── html_table_double_escape.py
|
||||||
|
│ ├── html_table_layout.py
|
||||||
|
│ ├── manuscript_line_number.py
|
||||||
|
│ ├── page_break_word_join.py
|
||||||
|
│ ├── reference_spacing.py
|
||||||
|
│ ├── repeated_running_header.py
|
||||||
|
│ └── word_review_comment.py
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ └── run_clindb_arxiv_experiment.py # 固定 5 份论文的仓库内实验入口
|
│ ├── run_clindb_arxiv_experiment.py # 只含 arXiv 组件的历史实验入口
|
||||||
|
│ └── run_clindb_first_batch_experiment.py # ClinDB 第一批 8 组件实验入口
|
||||||
├── tests/
|
├── tests/
|
||||||
│ ├── test_arxiv_submission_stamp.py
|
│ ├── test_arxiv_submission_stamp.py
|
||||||
│ ├── test_artifact_store.py
|
│ ├── test_artifact_store.py
|
||||||
|
│ ├── test_clindb_first_batch_pipeline.py
|
||||||
│ ├── test_component.py
|
│ ├── test_component.py
|
||||||
│ ├── test_edits.py
|
│ ├── test_edits.py
|
||||||
│ ├── test_experiment.py
|
│ ├── test_experiment.py
|
||||||
|
│ ├── test_html_table_double_escape.py
|
||||||
|
│ ├── test_html_table_layout.py
|
||||||
|
│ ├── test_manuscript_line_number.py
|
||||||
│ ├── test_models.py
|
│ ├── test_models.py
|
||||||
|
│ ├── test_page_break_word_join.py
|
||||||
│ ├── test_pipeline.py
|
│ ├── test_pipeline.py
|
||||||
│ └── test_reporting.py
|
│ ├── test_reference_spacing.py
|
||||||
|
│ ├── test_repeated_running_header.py
|
||||||
|
│ ├── test_reporting.py
|
||||||
|
│ └── test_word_review_comment.py
|
||||||
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
||||||
│ └── md/
|
│ └── md/
|
||||||
│ ├── dmp.md
|
│ ├── dmp.md
|
||||||
@@ -119,12 +137,11 @@ mdpolish/
|
|||||||
3. `research-wiki/README.md`，确认文档应放在哪里；
|
3. `research-wiki/README.md`，确认文档应放在哪里；
|
||||||
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
||||||
|
|
||||||
第一版核心的当前机制见 `research-wiki/explanation/first-executable-core.md`，首个真实组件见
|
第一版核心机制见 `research-wiki/explanation/first-executable-core.md`，ClinDB 第一批组件见
|
||||||
`research-wiki/explanation/arxiv-submission-stamp.md`，本地实验产物机制见
|
`research-wiki/explanation/clindb-first-batch-components.md`，本地实验产物机制见
|
||||||
`research-wiki/explanation/local-experiment-artifacts.md`，实际运行步骤见
|
`research-wiki/explanation/local-experiment-artifacts.md`，实际运行步骤见
|
||||||
`research-wiki/guides/run-local-clindb-arxiv-experiment.md`。下一项候选是 ClinDB 范围中的 HTML 实体双重转义，
|
`research-wiki/guides/run-local-clindb-first-batch-experiment.md`。解析器、CLI、文件适配器、profile 格式、图片资产打包和
|
||||||
但必须先用新 design 确定只在哪些 HTML 范围替换、如何避开字面示例以及实体替换边界。解析器、CLI、文件适配器、
|
独立检查能力仍需分别设计；当前本地实验适配器不能被推导成这些公共接口已经获批。
|
||||||
profile 格式和独立检查能力仍需分别设计；当前本地实验适配器不能被推导成这些公共接口已经获批。
|
|
||||||
|
|
||||||
## 当前可用检查
|
## 当前可用检查
|
||||||
|
|
||||||
@@ -135,7 +152,7 @@ python -m venv .venv
|
|||||||
|
|
||||||
# 第一版核心的基础验收
|
# 第一版核心的基础验收
|
||||||
.venv/bin/ruff check .
|
.venv/bin/ruff check .
|
||||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
.venv/bin/pytest
|
.venv/bin/pytest
|
||||||
|
|
||||||
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
||||||
@@ -148,6 +165,6 @@ find research-wiki -maxdepth 2 -type f | sort
|
|||||||
git status --short
|
git status --short
|
||||||
```
|
```
|
||||||
|
|
||||||
上述安装和三项基础验收已于 2026-08-22 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 19 个源码、
|
上述安装和三项基础验收已于 2026-08-22 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 37 个源码、
|
||||||
测试和实验脚本文件无问题，pytest 共 113 项测试通过。`requires-python` 仍以 `pyproject.toml` 声明的 Python 3.11 及以上为准；
|
测试和实验脚本文件无问题，pytest 共 204 项测试通过。`requires-python` 仍以 `pyproject.toml` 声明的 Python 3.11 及以上为准；
|
||||||
本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
||||||
|
|||||||
@@ -0,0 +1,531 @@
|
|||||||
|
# 0006：ClinDB 第一批清洗组件与组合顺序
|
||||||
|
|
||||||
|
## 状态
|
||||||
|
|
||||||
|
已批准并冻结（2026-08-22）。
|
||||||
|
|
||||||
|
本设计已经用户明确批准，授权按第 13 节实施。后续语义变化必须新增 design，不回写本文。
|
||||||
|
|
||||||
|
本文沿用 `0003` 的内存核心与组件契约、`0004` 已批准的 arXiv 组件，以及 `0005` 的本地实验产物契约。
|
||||||
|
它不替代这三份设计，也不修改核心数据模型、执行器、流水线或产物格式。
|
||||||
|
|
||||||
|
## 1. 为什么现在需要这一设计
|
||||||
|
|
||||||
|
当前 5 份 ClinDB 论文已经能够通过本地实验层运行，但正式组件只有
|
||||||
|
`paper.arxiv_submission_stamp`。现有 reference 列出了第一批 8 类自动清洗目标，却还没有逐项确定：
|
||||||
|
|
||||||
|
- 每个问题由哪个组件负责；
|
||||||
|
- 哪些前置条件同时满足时才允许修改；
|
||||||
|
- 一处修改究竟替换哪些字符；
|
||||||
|
- 组件之间是否存在先后依赖；
|
||||||
|
- 当前 5 份输入的真实命中数是多少；
|
||||||
|
- 哪些看似合理的通用化会把正文一起改掉。
|
||||||
|
|
||||||
|
只按问题名称直接写正则会产生实际误改。例如，JAMA 文档中 Abstract 之前的 59 个行首数字是作者单位编号，不能当作
|
||||||
|
手稿行号删除；Springer 正文里还有编号方法列表，不能由参考文献空行规则全局处理；HTML 表格中的双重实体也不能直接
|
||||||
|
替换为原始 `<`，否则可能改变 HTML 解析边界。
|
||||||
|
|
||||||
|
因此本轮先冻结第一批组件的严格语义和组合顺序，再实施代码。
|
||||||
|
|
||||||
|
## 2. 只读调研结论与待修订事实
|
||||||
|
|
||||||
|
本轮重新按物理行和原始字符检查了 `data/md/` 中的 5 份本地论文副本。下表是本设计拟采用的验收事实；在本文批准前，
|
||||||
|
它们只是待评审结论，不表示组件已经实现。
|
||||||
|
|
||||||
|
| 范围 | 只读结果 | 对现有 reference 的影响 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Word 审阅批注 | JAMA 有 2 条，批注文字都与 `Commented [...]` 位于同一物理行 | 第一版不猜测多行批注正文，只删除严格命中的单行批注及其后一个空行 |
|
||||||
|
| 手稿行号 | JAMA 共出现 134 个候选数字前缀；Abstract 之前 59 个是作者单位编号，Abstract 之后 75 个才是手稿行号，其中 6 个位于标题内 | `128 + 6` 的旧计数会误伤作者单位，应改为总计 75 |
|
||||||
|
| arXiv 提交戳 | sim、springer 各 1 条 | 沿用 `0004`，不改变组件语义 |
|
||||||
|
| 重复跑动页眉 | dmp 同一标题出现 2 次，其中 1 次切断正文，另 1 次落在参考文献中 | 页眉必须先于参考文献分隔执行 |
|
||||||
|
| 跨页断词 | 当前可确认 6 处，包括 3 处去连字符、2 处保留词内连字符和 1 处带错误句点的精确修复 | 不能只用“行尾连字符 + 小写字母”，也没有证据支持引入英文词表 |
|
||||||
|
| HTML 双重实体 | dmp 23 处、ejhf 8 处，共 31 处，全部在严格 HTML 表格单元格文本中 | “解除一层”应得到 `<`、`>`、`&`，而不是直接生成 HTML 源码中的 `<`、`>`、`&` |
|
||||||
|
| 单行 HTML 表格 | 共 9 张；每张都至少有一个非 `1` 的 `colspan`，没有可验证的简单无合并表格样本 | 第一批只保留 HTML 并按行展开，不批准未经真实样本验证的 GFM 转换 |
|
||||||
|
| 参考文献空行 | dmp 需要补 13 处，springer 需要补 15 处，共 28 处 | 旧的条目范围计数不准确；规则必须限定在 References 章节内 |
|
||||||
|
|
||||||
|
dmp 的一处输入实际是前一行以 `threshold.` 结束、下一非空行以 `olds` 开始。它并不是普通的
|
||||||
|
`thresh-` + `olds`，而是上游结果同时多了句点并重复了 `old`。将它恢复为 `thresholds` 属于一条基于当前上下文批准的
|
||||||
|
精确映射，不能推广成通用断词算法。
|
||||||
|
|
||||||
|
上述差异在设计批准后才同步修订
|
||||||
|
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。不能一边保留旧计数，
|
||||||
|
一边让测试按新计数通过。
|
||||||
|
|
||||||
|
## 3. 目标与非目标
|
||||||
|
|
||||||
|
### 3.1 目标
|
||||||
|
|
||||||
|
- 为第一批 8 类问题提供 7 个新组件，并复用现有 arXiv 组件；
|
||||||
|
- 每个组件只提出唯一、可立即应用的精确修改；
|
||||||
|
- 将论文场景的组合留在实验脚本，核心和组件不依赖 DOI、文件名或 `data/md/`；
|
||||||
|
- 明确组件顺序和少量真实依赖，避免靠多轮执行修正顺序错误；
|
||||||
|
- 保持标准库运行时、确定性、原子修改、完整审计和幂等性；
|
||||||
|
- 用合成测试覆盖正向、反向、异常边界和整条组合的稳定性；
|
||||||
|
- 设计批准并实现后，使用 `0005` 的现有产物结构保存 5 份论文的清洗结果、JSON 审计和 diff。
|
||||||
|
|
||||||
|
### 3.2 非目标
|
||||||
|
|
||||||
|
- 不恢复截断、缺表、缺句、乱码、OCR 表头或其他缺失内容；
|
||||||
|
- 不判断修订残留中哪个词正确，不改写写作、拼写或医学语义；
|
||||||
|
- 不增加 `detect_only`、建议、Inspector、人工审核或问题报告；
|
||||||
|
- 不引入 Markdown parser、HTML parser 第三方依赖、AST 或全文重新渲染；
|
||||||
|
- 不建立 Profile 文件格式、组件自动发现、公共 CLI 或稳定公共批处理接口；
|
||||||
|
- 不把无合并 HTML 表格转成 GFM；当前 5 份输入没有这种真实样本，表头推断也没有唯一答案；
|
||||||
|
- 不做一般段落重排、全局空行格式化或全局 HTML 实体解码；
|
||||||
|
- 不读取、复制、改写或打包图片；清洗产物中的图片相对路径仍可能因目录变化而无法直接显示；
|
||||||
|
- 不修改或覆盖 `data/md/` 输入，不对仓库外 GovDoc 数据保存实验产物；
|
||||||
|
- 不宣称输出已与 PDF 原文完全一致，也不宣称 ClinDB 中所有内容问题已经解决。
|
||||||
|
|
||||||
|
本文中的“ClinDB 第一批全清洗”只表示：本设计批准的 8 个自动组件全部运行并稳定，5 份输入中已确认的对应噪声全部
|
||||||
|
按契约处理。它不表示文档没有本设计明确排除的问题。
|
||||||
|
|
||||||
|
## 4. 采用的组件划分
|
||||||
|
|
||||||
|
第一批流水线固定包含以下组件：
|
||||||
|
|
||||||
|
| 顺序 | 组件标识 | 版本 | 职责 |
|
||||||
|
| ---: | --- | --- | --- |
|
||||||
|
| 1 | `paper.word_review_comment` | `1.0.0` | 删除严格单行 Word 审阅批注 |
|
||||||
|
| 2 | `paper.manuscript_line_number` | `1.0.0` | 删除通过整段证据确认的手稿行号前缀 |
|
||||||
|
| 3 | `paper.arxiv_submission_stamp` | `1.0.0` | 删除严格整行匹配的 arXiv 提交戳 |
|
||||||
|
| 4 | `paper.repeated_running_header` | `1.0.0` | 删除重复跑动页眉，并在证据充分处接回被切断段落 |
|
||||||
|
| 5 | `paper.page_break_word_join` | `1.0.0` | 按显式批准映射修复跨页断词 |
|
||||||
|
| 6 | `markdown.html_table_double_escape` | `1.0.0` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||||
|
| 7 | `markdown.html_table_layout` | `1.0.0` | 将严格单行 HTML 表格展开为一行一个 `<tr>` |
|
||||||
|
| 8 | `paper.reference_spacing` | `1.0.0` | 只在 References 章节内统一相邻编号条目的空行 |
|
||||||
|
|
||||||
|
不采用一个 `clindb.clean_all` 大组件。拆分后每项修改都有独立身份、版本、适用边界和命中计数；改变表格规则不会迫使
|
||||||
|
行号规则一起升级，其他项目也可以只组合自己需要的组件。
|
||||||
|
|
||||||
|
组件不互相导入、不直接调用另一个组件，也不知道自己位于 ClinDB 流水线。顺序由实验入口显式组装并由
|
||||||
|
`manifest.json` 记录。
|
||||||
|
|
||||||
|
## 5. 共同扫描基础
|
||||||
|
|
||||||
|
多个组件需要精确物理行范围，两个表格组件需要完全相同的保守 HTML 范围。批准后允许新增两个不公开的内部辅助模块：
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/mdpolish/
|
||||||
|
├── _text_ranges.py
|
||||||
|
└── _html_table.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.1 `_text_ranges.py`
|
||||||
|
|
||||||
|
它只负责返回原字符串中的物理行内容范围、行尾范围和空行关系，支持 `\n`、`\r\n`、单独 `\r` 和无末尾换行。
|
||||||
|
它不解释 Markdown 结构，也不规范化换行。
|
||||||
|
|
||||||
|
现有 arXiv 组件中的私有物理行扫描可以机械迁移到该模块。迁移不得改变
|
||||||
|
`paper.arxiv_submission_stamp` 的标识、版本、参数、识别范围或测试结果。
|
||||||
|
|
||||||
|
### 5.2 `_html_table.py`
|
||||||
|
|
||||||
|
它是针对当前转换器输出子集的严格词法扫描器，不是通用 HTML parser。只有同时满足以下条件的完整片段才返回结构范围：
|
||||||
|
|
||||||
|
- 具有完整、正确嵌套的小写 `<table>`、`<tr>`、`<td>` / `<th>` 起止标签；
|
||||||
|
- `<tr>` 是 `<table>` 的直接子元素，单元格是 `<tr>` 的直接子元素；
|
||||||
|
- 标签属性可以存在并原样保留，但引号和标签边界必须完整；
|
||||||
|
- 结构标签之间只能有空白，单元格文本中不能出现其他原始 HTML 标签；
|
||||||
|
- 不含嵌套表格、注释、声明、`script`、`style`、`pre` 或无法闭合的标签。
|
||||||
|
|
||||||
|
不满足条件的相似片段直接忽略，组件不尝试修复它，也不降级为全局正则。扫描器返回原字符串下标，不构造 DOM，
|
||||||
|
不解码实体，不重新输出 HTML。
|
||||||
|
|
||||||
|
第一批组件继续不识别 Markdown 围栏、引用块或其他块级上下文。严格目标文本即使出现在这些结构中仍可能命中；
|
||||||
|
当前 5 份证据没有这种反例，本设计不借共享扫描器重新加入围栏保护。
|
||||||
|
|
||||||
|
## 6. 各组件的精确契约
|
||||||
|
|
||||||
|
### 6.1 `paper.word_review_comment`
|
||||||
|
|
||||||
|
目标物理行必须完整满足：
|
||||||
|
|
||||||
|
```regex
|
||||||
|
^Commented \[[A-Za-z0-9]+\]: .*\S$
|
||||||
|
```
|
||||||
|
|
||||||
|
并且其后至少紧跟一个空白内容为空的物理行。每个命中产生一个删除编辑，范围包含：
|
||||||
|
|
||||||
|
1. 批注行的全部文字和自身行尾；
|
||||||
|
2. 紧随其后的第一个空行及其行尾。
|
||||||
|
|
||||||
|
组件不删除更多空行，不把后续普通行猜成批注正文，也不处理 `Commented` 出现在句中、行首带空格、缺少批注 ID、
|
||||||
|
缺少冒号或没有正文的相似文本。末尾没有后续空行时保持原样。
|
||||||
|
|
||||||
|
固定修改理由为“删除严格单行 Word 审阅批注及其后一个空行”。当前 JAMA 两条批注各产生一条 `Change`。
|
||||||
|
|
||||||
|
### 6.2 `paper.manuscript_line_number`
|
||||||
|
|
||||||
|
该组件不全局删除行首数字。它先按以下步骤确认一个完整手稿行号序列：
|
||||||
|
|
||||||
|
1. 文档中必须恰好有一个严格物理行 `## Abstract`；
|
||||||
|
2. 只扫描该行之后的内容，Abstract 之前的所有数字前缀一律排除；
|
||||||
|
3. 候选前缀只有两种：普通行的 `^[1-9][0-9]{0,2} `，或 ATX 标题中的
|
||||||
|
`^#{1,6} [1-9][0-9]{0,2} `；前缀后必须还有非空正文；
|
||||||
|
4. 按出现顺序取得的所有候选数字必须严格递增，允许跳号，不允许相等或回退；
|
||||||
|
5. 整段至少有 20 个候选，并至少有 2 个标题候选。
|
||||||
|
|
||||||
|
任一条件不满足时，整个组件返回零候选，不做部分删除，也不从某个看似合理的位置继续猜测。当前阈值用于证明这是
|
||||||
|
整篇带行号的手稿，而不是恰好出现几个数字开头的段落；未来出现短手稿时必须用新证据重新评审，不能暗中降低阈值。
|
||||||
|
|
||||||
|
确认序列后，每个前缀各产生一条精确修改：普通行只删除数字和其后的一个 ASCII 空格；标题只删除标题标记之后的
|
||||||
|
数字和一个空格，保留原有 `#` 级别和标题空格。正文中的数字、列表 `1.` / `1)`、四位年份、Abstract 之前的作者单位
|
||||||
|
编号均不修改。
|
||||||
|
|
||||||
|
固定修改理由为“删除已确认手稿序列中的行号前缀”。当前 JAMA 应产生 75 条 `Change`，其中普通行 69 条、标题 6 条。
|
||||||
|
|
||||||
|
### 6.3 `paper.arxiv_submission_stamp`
|
||||||
|
|
||||||
|
完全复用 `0004` 已批准的 `ArxivSubmissionStampComponent`，不修改代码语义和版本。第一批组合只把它放入新的流水线，
|
||||||
|
当前 sim 和 springer 各产生 1 条 `Change`。
|
||||||
|
|
||||||
|
### 6.4 `paper.repeated_running_header`
|
||||||
|
|
||||||
|
候选页眉首先必须是无首尾空格的完整 ATX 标题行：
|
||||||
|
|
||||||
|
```regex
|
||||||
|
^#{1,6} \S(?:.*\S)?$
|
||||||
|
```
|
||||||
|
|
||||||
|
相同的完整标题行必须原样出现至少 2 次。一个重复组只有在所有出现位置都能安全删除，并且至少有一个位置满足“正文被
|
||||||
|
切断”证据时，才整体成立。
|
||||||
|
|
||||||
|
“正文被切断”要求该标题前后各恰好有一个空行，前一非空物理行去掉行尾空白后不以中英文句号、问号、叹号、冒号或
|
||||||
|
分号结束，后一非空物理行以 ASCII 小写字母开头。该位置从前一正文行末到后一正文行首的整个间隔替换为一个 ASCII
|
||||||
|
空格，从而删除空行和页眉并接回同一句。
|
||||||
|
|
||||||
|
同组的其他位置只在标题后至少有一个空行时删除标题行及其后第一个空行，保留标题之前已有的分隔。若某个出现位置不满足
|
||||||
|
上述任一安全形态，则整组不修改。组件不使用标题关键词，不删除只出现一次的标题，也不把一般重复章节标题视为页眉。
|
||||||
|
|
||||||
|
固定修改理由为“删除经重复和断句证据确认的跑动页眉”。当前 dmp 的同一标题产生 2 条 `Change`。
|
||||||
|
|
||||||
|
### 6.5 `paper.page_break_word_join`
|
||||||
|
|
||||||
|
该组件不使用词典、拼写检查、概率模型或“行尾有连字符就合并”的启发式规则。调用方必须显式传入有序映射，每项包含：
|
||||||
|
|
||||||
|
```text
|
||||||
|
left_fragment
|
||||||
|
right_fragment
|
||||||
|
replacement
|
||||||
|
```
|
||||||
|
|
||||||
|
组件初始化时把映射规范化为确定顺序，并拒绝空字段、重复的左右片段组合或会产生无效修改的映射。实际参数完整写入
|
||||||
|
`ComponentInfo.parameters`，因此相同组件版本使用了哪些项目映射可以从运行清单复核。
|
||||||
|
|
||||||
|
一条映射只有在以下条件全部满足时才命中：
|
||||||
|
|
||||||
|
- `left_fragment` 是前一非空物理行的精确末尾；
|
||||||
|
- 右片段位于紧邻的下一物理行，或中间恰好隔一个空行；存在空行时，相关行尾使用同一种形式；
|
||||||
|
- `right_fragment` 是下一非空物理行的精确开头；
|
||||||
|
- 左片段前和右片段后都满足 ASCII 字母词边界，不能只是更长单词的一部分；
|
||||||
|
- 同一位置没有被另一条映射命中。
|
||||||
|
|
||||||
|
编辑范围从左片段起点到右片段终点，包含两个片段和中间一个或两个行尾，整体替换为显式 `replacement`。前后其余正文
|
||||||
|
一字不改。两个及以上空行通常表示段落边界，即使片段文字相同也不处理。
|
||||||
|
|
||||||
|
ClinDB 第一批参数固定为：
|
||||||
|
|
||||||
|
| `left_fragment` | `right_fragment` | `replacement` | 说明 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `medi-` | `cal` | `medical` | 去除分页连字符 |
|
||||||
|
| `possi-` | `bly` | `possibly` | 去除分页连字符 |
|
||||||
|
| `cre-` | `ated` | `created` | 去除分页连字符 |
|
||||||
|
| `SOFA-` | `based` | `SOFA-based` | 保留复合词连字符，只删除分页分隔 |
|
||||||
|
| `life-` | `threatening` | `life-threatening` | 保留复合词连字符，只删除分页分隔 |
|
||||||
|
| `threshold.` | `olds` | `thresholds` | 当前 dmp 的精确证据修复，不推广 |
|
||||||
|
|
||||||
|
每处固定理由为“按已批准映射修复跨页断词：`<left_fragment>` + `<right_fragment>` → `<replacement>`”；当前应产生
|
||||||
|
6 条 `Change`。增加或改变映射会改变实验参数和预期结果，必须先更新项目设计或已批准的项目配置，不能从外部词典自动扩张。
|
||||||
|
|
||||||
|
### 6.6 `markdown.html_table_double_escape`
|
||||||
|
|
||||||
|
该组件只处理 `_html_table.py` 接受的完整表格，并只扫描 `<td>` / `<th>` 的文本范围。第一版映射固定为：
|
||||||
|
|
||||||
|
| 修改前源码 | 修改后源码 | 浏览器最终显示意图 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `&lt;` | `<` | `<` |
|
||||||
|
| `&gt;` | `>` | `>` |
|
||||||
|
| `&amp;` | `&` | `&` |
|
||||||
|
|
||||||
|
这是对 HTML 源码解除一层实体转义，不是把最终显示字符直接写进 HTML。每个精确实体 token 产生一个编辑；已经是单层的
|
||||||
|
`<`、`>`、`&` 不再命中，因此再次运行稳定。
|
||||||
|
|
||||||
|
标签名、属性值、表格外文本、普通 Markdown、URL 和不完整实体均不修改。组件不调用 `html.unescape()`，因为它会扩大到
|
||||||
|
没有逐项批准的命名实体和数字实体，也无法保留 HTML 标签边界。
|
||||||
|
|
||||||
|
固定修改理由为“在严格 HTML 表格单元格文本中解除一层实体转义”。当前应产生 31 条 `Change`。
|
||||||
|
|
||||||
|
### 6.7 `markdown.html_table_layout`
|
||||||
|
|
||||||
|
该组件只处理 `_html_table.py` 接受、并且整个 `<table>...</table>` 片段不含任何 `\r` 或 `\n` 的单行表格。
|
||||||
|
它不计算行列数，不解释 `rowspan` / `colspan`，不推断表头，也不改变任何单元格内容、属性或标签书写。
|
||||||
|
|
||||||
|
输出形态固定为：
|
||||||
|
|
||||||
|
```html
|
||||||
|
<table>
|
||||||
|
<tr>...</tr>
|
||||||
|
<tr>...</tr>
|
||||||
|
</table>
|
||||||
|
```
|
||||||
|
|
||||||
|
实际 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和 `</table>` 都从输入原样复用；组件只规范化这些外层片段之间的
|
||||||
|
布局空白。文档只有一种行尾时沿用该行尾；文档没有任何行尾时使用 `\n`；文档混用多种行尾时保持表格原样。
|
||||||
|
|
||||||
|
每张表产生一个覆盖完整表格片段的替换编辑。已经是多行的表格不命中，因而第二次运行不再改动。当前 9 张表全部含
|
||||||
|
`colspan`，均保留 HTML，只展开为一行一个 `<tr>`，应产生 9 条 `Change`。
|
||||||
|
|
||||||
|
固定修改理由为“展开严格单行 HTML 表格的行布局”。未来是否把简单表格转为 GFM，必须等真实无合并样本出现后单独决定
|
||||||
|
表头、转义、换行和 HTML/GFM 等价性。
|
||||||
|
|
||||||
|
### 6.8 `paper.reference_spacing`
|
||||||
|
|
||||||
|
该组件先定位严格 ATX 标题，其标题文字按 ASCII 大小写折叠后必须恰好是 `references`。章节范围从该标题之后开始，
|
||||||
|
到下一个级别相同或更高的 ATX 标题之前结束；没有后续标题时到文档末尾。
|
||||||
|
|
||||||
|
章节中的编号条目起始行必须完整满足 `^([1-9][0-9]*)\. \S`。只有同时满足以下条件才处理整个章节：
|
||||||
|
|
||||||
|
- 至少有 2 个编号条目；
|
||||||
|
- 第一个编号为 1；
|
||||||
|
- 后续编号严格逐个加 1，没有缺号、重复或回退；
|
||||||
|
- 两个相邻条目之间的最后一条正文行与下一条目起始行之间只有同一种行尾和零个或多个空行。
|
||||||
|
|
||||||
|
组件把每个相邻条目边界规范为恰好一个空行，即两个相同的行尾。已经恰好一个空行的边界不产生修改；没有空行时插入
|
||||||
|
一个行尾，多余空行时收敛为一个。章节外编号列表、非 References 标题、编号不连续的章节和混合行尾边界保持原样。
|
||||||
|
|
||||||
|
固定修改理由为“统一 References 章节中相邻编号条目之间的一个空行”。该组件必须在跑动页眉之后执行，否则 dmp 位于
|
||||||
|
第 18、19 条之间的页眉会破坏章节连续性。当前 dmp 应产生 13 条、springer 应产生 15 条，共 28 条 `Change`。
|
||||||
|
|
||||||
|
## 7. 顺序为什么固定
|
||||||
|
|
||||||
|
第一批只执行一轮，不能依赖“第二轮自然修好”。固定顺序的直接理由是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Word 批注 ─┐
|
||||||
|
手稿行号 ──┼── 先去除论文编辑层噪声
|
||||||
|
arXiv 戳 ──┘
|
||||||
|
|
||||||
|
重复页眉 ─────► 参考文献分隔
|
||||||
|
|
||||||
|
跨页断词 ─────► 独立正文精确修复
|
||||||
|
|
||||||
|
HTML 双重实体 ─► HTML 表格布局
|
||||||
|
```
|
||||||
|
|
||||||
|
- 跑动页眉先删除，dmp 的参考文献编号才能形成完整连续序列；
|
||||||
|
- 实体先改、布局后改，使每个实体 `Change` 保留较小的局部范围，随后表格整体布局基于最新快照；
|
||||||
|
- Word 批注先删除，行号组件看到的是不含审阅插入物的手稿，但行号判定本身仍不得依赖批注一定存在；
|
||||||
|
- 其他顺序目前没有内容依赖，仍固定下来以保证审计、哈希和 diff 可复现。
|
||||||
|
|
||||||
|
脚本和组合测试必须断言第 4 节的完整顺序，不能按文件名排序组件，也不能针对不同文档临时增删组件。
|
||||||
|
|
||||||
|
## 8. 代码与依赖边界
|
||||||
|
|
||||||
|
批准后允许新增：
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/mdpolish/
|
||||||
|
├── _html_table.py
|
||||||
|
├── _text_ranges.py
|
||||||
|
└── components/
|
||||||
|
├── html_table_double_escape.py
|
||||||
|
├── html_table_layout.py
|
||||||
|
├── manuscript_line_number.py
|
||||||
|
├── page_break_word_join.py
|
||||||
|
├── reference_spacing.py
|
||||||
|
├── repeated_running_header.py
|
||||||
|
└── word_review_comment.py
|
||||||
|
scripts/
|
||||||
|
└── run_clindb_first_batch_experiment.py
|
||||||
|
tests/
|
||||||
|
├── test_clindb_first_batch_pipeline.py
|
||||||
|
├── test_html_table_double_escape.py
|
||||||
|
├── test_html_table_layout.py
|
||||||
|
├── test_manuscript_line_number.py
|
||||||
|
├── test_page_break_word_join.py
|
||||||
|
├── test_reference_spacing.py
|
||||||
|
├── test_repeated_running_header.py
|
||||||
|
└── test_word_review_comment.py
|
||||||
|
```
|
||||||
|
|
||||||
|
允许同步修改：
|
||||||
|
|
||||||
|
- `src/mdpolish/components/__init__.py`，导出 7 个新组件；
|
||||||
|
- `src/mdpolish/components/arxiv_submission_stamp.py`，只把物理行扫描机械迁移到 `_text_ranges.py`；
|
||||||
|
- 现有 arXiv 测试，验证迁移没有改变行为；
|
||||||
|
- README、对应 explanation 和经实际验证的 guide，使当前阶段、目录和检查命令与实现一致；
|
||||||
|
- ClinDB reference，修订第 2 节列出的计数、实体语义、表格边界和 `0005` 已实现的输出事实。
|
||||||
|
|
||||||
|
依赖方向固定为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
components ──► models / component
|
||||||
|
│
|
||||||
|
├───────► _text_ranges
|
||||||
|
└───────► _html_table
|
||||||
|
|
||||||
|
experiment script ──► components + Pipeline + experiment layer
|
||||||
|
```
|
||||||
|
|
||||||
|
内部辅助模块不导入具体组件、流水线、文件适配层或 reporter。7 个新组件不从顶层 `mdpolish.__init__` 导出，不承诺
|
||||||
|
稳定公共 API。现有核心签名、JSON `schema_version` 和产物目录结构均不改变。
|
||||||
|
|
||||||
|
`run_clindb_arxiv_experiment.py` 保留为首个组件实验的历史入口，不改写成新语义。新的 first-batch 脚本显式建立第 4 节
|
||||||
|
的 8 组件流水线，仍只接受 `--run-id`，并复用 `0005` 已批准的 5 份文档清单和
|
||||||
|
`artifacts/<run_date>/runs/<run_id>/` 保存逻辑。它不是公共 CLI，也不是 Profile 实现。
|
||||||
|
|
||||||
|
第一版继续只使用 Python 标准库，不修改最低 Python 版本，不新增运行依赖。
|
||||||
|
|
||||||
|
## 9. 合成测试要求
|
||||||
|
|
||||||
|
每个组件至少覆盖空输入、无命中、单命中、多命中、三种行尾、无末尾行尾、参数或结构反例、修改理由、确定顺序和
|
||||||
|
再次运行零修改。额外必须覆盖：
|
||||||
|
|
||||||
|
### 9.1 Word 批注
|
||||||
|
|
||||||
|
- 两条由空行隔开的单行批注都删除，并正确接回周围普通行；
|
||||||
|
- 行内 `Commented`、不合法 ID、缺正文、没有后续空行和假想多行正文保持原样；
|
||||||
|
- 不多删第二个及后续空行。
|
||||||
|
|
||||||
|
### 9.2 手稿行号
|
||||||
|
|
||||||
|
- Abstract 前的连续作者单位编号全部保留；
|
||||||
|
- 满足阈值和标题证据的递增序列同时清理普通行与标题；
|
||||||
|
- 候选少于 20、标题候选少于 2、重复、回退、多个 Abstract 或没有 Abstract 时整体不修改；
|
||||||
|
- `1.`、`1)`、四位年份、正文中间的 `35 pediatric experts`、`10 sites`、`4 continents` 保留。
|
||||||
|
|
||||||
|
### 9.3 重复页眉
|
||||||
|
|
||||||
|
- 一个桥接位置替换为单个空格，另一个独立位置删除后只保留一个原有段落分隔；
|
||||||
|
- 只出现一次、重复但没有断句证据、下一行大写或某次出现无法安全删除时保持原样；
|
||||||
|
- 页眉文字不硬编码，候选顺序按原文位置稳定。
|
||||||
|
|
||||||
|
### 9.4 跨页断词
|
||||||
|
|
||||||
|
- 6 条批准映射分别验证，包含删除连字符、保留连字符和 `threshold.` / `olds` 精确修复；
|
||||||
|
- 未配置词、不是一整个词边界、两个及以上空行、混合行尾、大小写不同和相似长词保持原样；
|
||||||
|
- 相邻物理行和中间恰好一个空行两种已批准边界都能正确合并；
|
||||||
|
- 映射输入顺序不同仍得到相同参数和结果，重复或冲突映射明确报契约错误。
|
||||||
|
|
||||||
|
### 9.5 HTML 实体与布局
|
||||||
|
|
||||||
|
- 实体只解除一层，第二次不再改变；属性、表外文本、单层实体和未批准实体保留；
|
||||||
|
- 9 张表所代表的 `colspan` 形态只改布局，不改属性、单元格文本和标签顺序；
|
||||||
|
- 已经多行的表格不改；混合行尾文档不改布局；
|
||||||
|
- 未闭合、嵌套、含额外标签或结构不合法的表格整体忽略；
|
||||||
|
- 两个 HTML 组件组合后再次运行稳定。
|
||||||
|
|
||||||
|
### 9.6 参考文献与完整组合
|
||||||
|
|
||||||
|
- References 章节内缺失或过多空行都收敛为一个；章节外编号方法列表保留；
|
||||||
|
- 大小写不同的准确 References 标题可识别，近似标题、缺号和混合行尾保持原样；
|
||||||
|
- 合成 dmp 形态证明页眉先删除后，18、19 条之间和后续连续条目能正确处理；
|
||||||
|
- 完整 8 组件流水线第一次成功，最终复查没有残留候选；对成功输出再次运行为 `success` 且零 `Change`；
|
||||||
|
- 每条 `Change` 的组件、版本、位置、理由、`before`、`after` 和前后哈希可以按 `0003` / `0005` 重放。
|
||||||
|
|
||||||
|
测试 fixture 只能使用小型虚构文本，不能复制真实论文段落或完整表格到 Git。
|
||||||
|
|
||||||
|
## 10. 本地 5 份论文验收
|
||||||
|
|
||||||
|
所有合成测试通过后，才允许对 `data/md/` 的 5 份本地副本执行新的保存型实验。预期计数如下：
|
||||||
|
|
||||||
|
| 组件 | 预期 `Change` 数 |
|
||||||
|
| --- | ---: |
|
||||||
|
| `paper.word_review_comment` | 2 |
|
||||||
|
| `paper.manuscript_line_number` | 75 |
|
||||||
|
| `paper.arxiv_submission_stamp` | 2 |
|
||||||
|
| `paper.repeated_running_header` | 2 |
|
||||||
|
| `paper.page_break_word_join` | 6 |
|
||||||
|
| `markdown.html_table_double_escape` | 31 |
|
||||||
|
| `markdown.html_table_layout` | 9 |
|
||||||
|
| `paper.reference_spacing` | 28 |
|
||||||
|
| **合计** | **155** |
|
||||||
|
|
||||||
|
验收必须确认：
|
||||||
|
|
||||||
|
1. `manifest.json` 中 5 份文档全部为 `success`，组件身份、版本、参数和顺序与本文一致；
|
||||||
|
2. 合计恰好 155 条 `Change`，各组件计数与上表一致；
|
||||||
|
3. 每个 `cleaned.md` 的哈希与 `result.json` 一致，最终复查没有残留候选；
|
||||||
|
4. 对 5 份成功输出再运行同一流水线，全部 `success` 且合计零修改；
|
||||||
|
5. JAMA 的作者单位编号保留，只清理 Abstract 后确认的 75 个手稿前缀；
|
||||||
|
6. Springer 的合法 arXiv 参考文献和正文编号方法列表保持原样；
|
||||||
|
7. 9 张表的标签、属性、单元格内容和顺序不变，只发生已批准的实体和行布局修改；
|
||||||
|
8. dmp 的重复页眉、跨页修复和参考文献空行在最终 diff 中可分别追溯；
|
||||||
|
9. 逐份人工查看 `changes.diff`，确认没有上表之外的正文改写；
|
||||||
|
10. 运行前后 5 份 `data/md/*.md` 输入字节哈希完全不变；
|
||||||
|
11. 产物只出现在 Git 忽略的 `artifacts/<run_date>/runs/<run_id>/`，`git status` 不列出产物；
|
||||||
|
12. 清洗产物不复制图片。图片是否可显示不属于本轮成功条件，README 或 guide 必须明确这一点。
|
||||||
|
|
||||||
|
只要实际计数、结构或正文 diff 与预期不一致，就停止收尾并回到本设计核对。不得通过放宽规则、删除断言或把额外修改
|
||||||
|
改名为“格式整理”来让实验通过。
|
||||||
|
|
||||||
|
基础检查继续以根目录 README 为唯一命令权威。实现后至少实际运行 Ruff、mypy、完整 pytest、Agent 镜像 diff、
|
||||||
|
Git diff 检查和上述本地实验；未执行的检查不得报告为通过。
|
||||||
|
|
||||||
|
## 11. 方案比较与否决项
|
||||||
|
|
||||||
|
### 11.1 一个组件处理全部 ClinDB 问题
|
||||||
|
|
||||||
|
文件少，但任何规则变化都会改变同一组件语义，审计无法按问题归因，也不能被其他项目选择性复用。不采用。
|
||||||
|
|
||||||
|
### 11.2 用全局正则清理数字、实体和空行
|
||||||
|
|
||||||
|
会分别误伤作者单位、普通 Markdown 实体和正文编号列表。已有 5 份输入就存在真实反例。不采用。
|
||||||
|
|
||||||
|
### 11.3 第一批引入 Markdown/HTML parser 和 AST 重写
|
||||||
|
|
||||||
|
成熟 parser 能扩大结构识别范围，但会引入方言、渲染和非目标内容重写问题。当前 9 张表只需要保留原字节的严格词法
|
||||||
|
范围，标准库小扫描器足够验证第一批。不采用。
|
||||||
|
|
||||||
|
### 11.4 用英文词典自动决定跨页合词
|
||||||
|
|
||||||
|
词典无法决定 `SOFA-based` 是否保留连字符，也无法解释 `threshold.` + `olds` 的上游错误；领域词、缩写和专名还会造成
|
||||||
|
漏判。第一批采用可审计的显式映射，不采用词典。
|
||||||
|
|
||||||
|
### 11.5 把无合并表格自动转为 GFM
|
||||||
|
|
||||||
|
当前没有真实样本，且没有 `<th>` 时必须猜测表头；转义、换行和合并语义也未验证。第一批保留 HTML，不采用。
|
||||||
|
|
||||||
|
### 11.6 顺便打包图片或建立项目 Profile
|
||||||
|
|
||||||
|
图片需要输入资产定位、路径改写、复制冲突和产物目录契约；Profile 需要身份、配置格式和兼容规则。两者与文本组件并非
|
||||||
|
同一问题，不能借本设计默认授权。不采用。
|
||||||
|
|
||||||
|
## 12. 风险与后续变化成本
|
||||||
|
|
||||||
|
- **严格规则会漏掉未来变体：** 这是保真优先的有意选择。新格式先补证据、测试和版本，不在原规则中静默放宽。
|
||||||
|
- **行号阈值针对长手稿：** 短文档可能不清理。组件与组合分离后，可以新增另一个有独立证据的组件或升级本组件，
|
||||||
|
不需要修改核心。
|
||||||
|
- **跑动页眉仍是启发式分类：** 重复加断句证据降低误删，但不能证明适用于所有论文。它只进入 ClinDB 第一批组合，
|
||||||
|
不成为全局默认规则。
|
||||||
|
- **显式断词映射需要维护：** 新文档会出现新词，但映射作为参数记录，不必修改执行器或其他组件；代价是每次扩张前要评审。
|
||||||
|
- **严格 HTML 子集会忽略复杂表格：** 失败关闭能够避免重写未知结构。以后若需要完整 HTML parser，只替换内部扫描实现或
|
||||||
|
升级两个表格组件，不改变 `DocumentSnapshot`、`TextEdit` 和产物契约。
|
||||||
|
- **整体表格布局编辑的 diff 较大：** 每张表只有一条可回放 `Change`，但人读 diff 时会看到整行展开。实体组件先执行，
|
||||||
|
`result.json` 仍能分别追踪局部实体修改和后续布局修改。
|
||||||
|
- **没有 Profile 对象：** 当前组合只在一个实验脚本中，适合第一批验证。未来新增第二个稳定项目组合时再设计 Profile，
|
||||||
|
组件本身无需迁移。
|
||||||
|
- **图片仍不可随产物查看：** 这是 `0005` 的既有边界。若人工评审必须在清洗目录直接显示图片，应新增图片资产产物设计，
|
||||||
|
不改变本轮文本组件。
|
||||||
|
|
||||||
|
这套拆分保持“业务识别组件 → 精确编辑核心 → 文件实验层”的单向依赖。后续调整某条识别规则通常只影响一个组件及其测试；
|
||||||
|
调整项目组合只影响实验入口或未来 Profile；调整产物日期和目录只影响 `0005` 的 artifact store。核心数据契约无需跟着变化。
|
||||||
|
|
||||||
|
## 13. 批准后的实施边界
|
||||||
|
|
||||||
|
如果用户明确批准本文，只授权：
|
||||||
|
|
||||||
|
1. 新增第 8 节列出的 7 个组件、2 个内部辅助模块、合成测试和 first-batch 实验脚本；
|
||||||
|
2. 对现有 arXiv 组件做不改变语义和版本的物理行辅助函数迁移；
|
||||||
|
3. 按第 4、6、7 节实现固定组件身份、严格识别、精确修改和组合顺序；
|
||||||
|
4. 按第 9 节运行合成检查；
|
||||||
|
5. 合成检查通过后，按 `0005` 对 `data/md/` 的 5 份副本保存一次本地 first-batch 实验；
|
||||||
|
6. 根据真实实现和实际验证更新 README、ClinDB reference、对应 explanation 和 guide。
|
||||||
|
|
||||||
|
批准仍不授权：
|
||||||
|
|
||||||
|
- 修改、覆盖、移动或删除任何输入和真实数据；
|
||||||
|
- 保存 GovDoc 或其他仓库外材料的清洗产物；
|
||||||
|
- 把 `artifacts/`、`data/` 或真实文本 fixture 加入 Git；
|
||||||
|
- 实现内容恢复、Inspector、建议、Profile 格式、公共 CLI、通用 parser、图片复制或路径改写；
|
||||||
|
- 提交、推送、创建 PR、发布或修改其他仓库。
|
||||||
@@ -75,9 +75,9 @@ arXiv:<新版数字编号和版本> [<ASCII 分类>] <日> <英文月份缩写>
|
|||||||
|
|
||||||
## 6. 剩余边界
|
## 6. 剩余边界
|
||||||
|
|
||||||
这个组件只证明第一条严格删除规则能够在公共核心上闭环，不表示论文已经清洗完成。HTML 实体、Word 批注、手稿
|
这个组件最初证明了第一条严格删除规则能够在公共核心上闭环。此后 ClinDB 第一批另外 7 个组件已经按 `0006` 实现，
|
||||||
行号、断词、表格和参考文献间距仍未实现。当前只有固定输入和固定组件的本地实验输出；通用文件接口、profile、
|
当前完整组合见 [`clindb-first-batch-components.md`](clindb-first-batch-components.md)。这仍不表示论文内容问题全部解决；
|
||||||
公共 CLI 和通用批处理仍不存在。
|
通用文件接口、profile、公共 CLI、图片资产和通用批处理仍不存在。
|
||||||
|
|
||||||
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
||||||
提高命中数量直接修改正则表达式。
|
提高命中数量直接修改正则表达式。
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# ClinDB 第一批组件如何在不猜正文的前提下完成清洗
|
||||||
|
|
||||||
|
## 1. 它解决的实际问题
|
||||||
|
|
||||||
|
5 份 ClinDB 论文 Markdown 同时包含编辑痕迹、转换噪声和排版噪声。它们看起来都像“删掉几行或整理一下格式”，
|
||||||
|
实际误删边界不同：行首数字可能是手稿行号，也可能是作者单位；`arXiv:` 可能是边栏戳，也可能是合法参考文献；
|
||||||
|
编号列表可能属于 References，也可能是正文方法步骤。
|
||||||
|
|
||||||
|
当前实现没有建立一个能随意改全文的“大清洗器”，而是把第一批确定问题拆成 8 个组件。每个组件只识别一种证据，
|
||||||
|
返回快照绑定的精确 `TextEdit`，由公共流水线统一验证、应用和记录。
|
||||||
|
|
||||||
|
清洗语义来自已批准的
|
||||||
|
[`0006-clindb-first-batch-cleaning-components.md`](../design/0006-clindb-first-batch-cleaning-components.md)，
|
||||||
|
输入范围和稳定计数见
|
||||||
|
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。
|
||||||
|
|
||||||
|
## 2. 当前组件和顺序
|
||||||
|
|
||||||
|
```text
|
||||||
|
Word 批注 ─┐
|
||||||
|
手稿行号 ──┼──► arXiv 戳 ──► 重复页眉 ──► 映射断词
|
||||||
|
│ │
|
||||||
|
│ └────────────► 参考文献空行
|
||||||
|
│
|
||||||
|
└────► HTML 双重实体 ──► HTML 表格布局
|
||||||
|
```
|
||||||
|
|
||||||
|
实验脚本固定按以下顺序组装：
|
||||||
|
|
||||||
|
| 顺序 | 组件 | 当前作用 |
|
||||||
|
| ---: | --- | --- |
|
||||||
|
| 1 | `paper.word_review_comment` | 删除完整单行 Word 批注及其后一个空行 |
|
||||||
|
| 2 | `paper.manuscript_line_number` | 删除由长递增序列确认的手稿行号 |
|
||||||
|
| 3 | `paper.arxiv_submission_stamp` | 删除严格整行提交戳 |
|
||||||
|
| 4 | `paper.repeated_running_header` | 删除重复页眉，并接回有明确续句证据的段落 |
|
||||||
|
| 5 | `paper.page_break_word_join` | 只应用本次运行参数中记录的词片段映射 |
|
||||||
|
| 6 | `markdown.html_table_double_escape` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||||
|
| 7 | `markdown.html_table_layout` | 保留 HTML 内容，把单行表格展开成一行一个 `<tr>` |
|
||||||
|
| 8 | `paper.reference_spacing` | 只在完整连续的 References 章节中统一条目空行 |
|
||||||
|
|
||||||
|
顺序不是为了让结果“看起来整齐”。dmp 的一个重复页眉正好位于第 18、19 条参考文献之间；如果不先删除页眉，
|
||||||
|
参考文献组件就不能确认这是完整连续序列。HTML 实体先修改小范围 token，表格布局再基于新快照替换整张表，
|
||||||
|
两类修改仍能在审计中分别追踪。
|
||||||
|
|
||||||
|
## 3. 为什么行号不能使用全局正则
|
||||||
|
|
||||||
|
JAMA 文档中一共有 134 个看似 `数字 + 空格` 的行首前缀。前 59 个位于 Abstract 之前，是作者单位编号；真正的手稿
|
||||||
|
行号只有 Abstract 之后的 75 个。
|
||||||
|
|
||||||
|
当前组件要求：
|
||||||
|
|
||||||
|
- 文档中恰好有一个严格 `## Abstract`；
|
||||||
|
- 只看它之后的候选；
|
||||||
|
- 候选数字全部严格递增；
|
||||||
|
- 至少有 20 个候选，并至少有 2 个数字标题。
|
||||||
|
|
||||||
|
任一条件失败就整篇不改。这样会漏掉较短的带行号手稿，但不会为了提高命中率删除作者单位或零散数字段落。
|
||||||
|
|
||||||
|
## 4. 为什么断词使用显式映射
|
||||||
|
|
||||||
|
“行尾连字符加下一行小写字母”无法决定连字符应删还是保留：`possi-` + `bly` 应成为 `possibly`，而
|
||||||
|
`SOFA-` + `based` 应保留为 `SOFA-based`。dmp 还存在 `threshold.` + `olds`，它同时包含多余句点和重复片段，
|
||||||
|
普通词典也无法解释。
|
||||||
|
|
||||||
|
因此组件的实际参数记录三项：左片段、右片段和结果词。只有相邻物理行或中间恰好一个空行、词边界完整且映射唯一时
|
||||||
|
才修改。增加新词不是自动学习行为，需要先批准新的项目映射；组件算法版本不变时，运行清单仍能通过参数区分实际语义。
|
||||||
|
|
||||||
|
## 5. 两个表格组件怎样共享范围
|
||||||
|
|
||||||
|
`_html_table.py` 只识别当前转换器输出的严格子集：`<tr>` 直接位于 `<table>` 下，`<td>` / `<th>` 直接位于
|
||||||
|
`<tr>` 下，标签完整闭合，单元格中没有嵌套标签。它返回原字符串下标，不生成 DOM，也不重新渲染全文。
|
||||||
|
|
||||||
|
实体组件只处理单元格文本中的三个精确 token：
|
||||||
|
|
||||||
|
```text
|
||||||
|
&lt; → <
|
||||||
|
&gt; → >
|
||||||
|
&amp; → &
|
||||||
|
```
|
||||||
|
|
||||||
|
结果仍是合法 HTML 源码中的单层实体。标签、属性、表格外文本和其他实体不受影响。
|
||||||
|
|
||||||
|
布局组件只处理整个片段没有换行的严格表格。它原样复用 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和结束标签，
|
||||||
|
只增加外层换行与两个空格缩进。当前 9 张真实表格都有 `colspan`，所以全部保留 HTML；实现没有猜测表头，也没有转 GFM。
|
||||||
|
|
||||||
|
遇到未闭合、嵌套、额外结构标签或混合换行时，扫描失败关闭，保持原文。
|
||||||
|
|
||||||
|
## 6. 共享代码为什么仍然很小
|
||||||
|
|
||||||
|
- `_text_ranges.py` 只提供 Python 字符下标下的物理行、行尾和空行关系；
|
||||||
|
- `_html_table.py` 只提供严格 HTML 表格、行和单元格范围；
|
||||||
|
- 业务组件依赖这两个私有模块，但辅助模块不依赖组件、流水线或文件层；
|
||||||
|
- 文件实验层只接收已经组装的 `Pipeline`，不知道任何识别规则。
|
||||||
|
|
||||||
|
因此以后放宽某个业务规则通常只改一个组件及其测试；替换 HTML 识别方式不会改变 `DocumentSnapshot`、
|
||||||
|
`ProposedChange`、`Change` 或 JSON 产物；未来引入 Profile 时，也只接管当前脚本里的组件组装。
|
||||||
|
|
||||||
|
## 7. 当前验证结果和边界
|
||||||
|
|
||||||
|
2026-08-22 在 Python 3.13.11 环境完成：
|
||||||
|
|
||||||
|
- Ruff 通过；
|
||||||
|
- mypy 检查 37 个文件无问题；
|
||||||
|
- pytest 204 项通过；
|
||||||
|
- 5 份本地论文全部 `success`，合计 155 条 `Change`；
|
||||||
|
- 对 5 份成功输出再次运行，全部 `success` 且零修改；
|
||||||
|
- 输入运行前后哈希不变；
|
||||||
|
- JAMA Abstract 前内容不变，Springer 两条合法 arXiv 参考文献保留；
|
||||||
|
- 图片引用文字不变，但实验产物没有复制图片资产。
|
||||||
|
|
||||||
|
保存型实验位于本机 Git 忽略的
|
||||||
|
`artifacts/2026-08-22/runs/clindb-first-batch-v1/`。运行和复核方法见
|
||||||
|
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。
|
||||||
|
|
||||||
|
这次成功只证明批准的 8 类规则在当前 5 份输入上闭环。截断、缺表、乱码、OCR 语义错误、图片资产、修订词选择和一般
|
||||||
|
段落重排仍不在自动清洗范围内。
|
||||||
@@ -29,7 +29,8 @@ experiment.py
|
|||||||
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
||||||
|
|
||||||
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
||||||
ClinDB 的 5 份论文和 arXiv 组件组合只存在于仓库内实验脚本，通用模块没有硬编码论文名。
|
ClinDB 的 5 份论文、历史 arXiv 单组件组合和当前 first-batch 组合只存在于两个仓库内实验脚本，通用模块没有硬编码
|
||||||
|
论文名或业务组件。
|
||||||
|
|
||||||
## 3. 输入怎样保持原样
|
## 3. 输入怎样保持原样
|
||||||
|
|
||||||
@@ -154,3 +155,21 @@ artifact store 先在同一日期的 `runs/` 下建立本次专用临时目录
|
|||||||
|
|
||||||
实际运行方法见
|
实际运行方法见
|
||||||
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
||||||
|
|
||||||
|
同日又使用 `design/0006` 的 8 组件流水线完成 ClinDB 第一批保存型实验：
|
||||||
|
|
||||||
|
| 项目 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 运行 ID | `clindb-first-batch-v1` |
|
||||||
|
| 输出位置 | `artifacts/2026-08-22/runs/clindb-first-batch-v1/` |
|
||||||
|
| 文档状态 | 5/5 `success` |
|
||||||
|
| 实际修改 | dmp 47、ejhf 9、jama 79、sim 3、springer 17，共 155 条 |
|
||||||
|
| 第二次运行 | 5/5 `success`，合计 0 条修改 |
|
||||||
|
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||||
|
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||||
|
| 内容反例 | JAMA Abstract 前内容不变；Springer 两条合法 arXiv 引用保留；图片引用文字不变 |
|
||||||
|
| 权限 | 运行目录 `0700`，产物文件 `0600` |
|
||||||
|
|
||||||
|
当前完整运行方法见
|
||||||
|
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。实验层的文件、
|
||||||
|
JSON、diff 和权限契约没有因组件增多而改变。
|
||||||
|
|||||||
@@ -19,7 +19,7 @@
|
|||||||
```bash
|
```bash
|
||||||
.venv/bin/python --version
|
.venv/bin/python --version
|
||||||
.venv/bin/ruff check .
|
.venv/bin/ruff check .
|
||||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
.venv/bin/pytest
|
.venv/bin/pytest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
# 运行本地 ClinDB 第一批完整清洗实验
|
||||||
|
|
||||||
|
## 1. 适用范围
|
||||||
|
|
||||||
|
本指南只运行仓库内已经批准的 first-batch 实验脚本：
|
||||||
|
|
||||||
|
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||||
|
- 流水线：`design/0006` 固定的 8 个组件和顺序；
|
||||||
|
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||||
|
- 输入只读，不覆盖原文件；
|
||||||
|
- 不读取或复制图片，不处理 `/home/lihaoze/gov_test_data`。
|
||||||
|
|
||||||
|
本指南于 2026-08-22 在 Python 3.13.11 环境实际验证。
|
||||||
|
|
||||||
|
## 2. 前置检查
|
||||||
|
|
||||||
|
在仓库根目录执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python --version
|
||||||
|
.venv/bin/ruff check .
|
||||||
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
|
.venv/bin/pytest
|
||||||
|
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||||
|
```
|
||||||
|
|
||||||
|
当前已验证结果是 Ruff 通过、mypy 37 个文件无问题、pytest 204 项通过。再确认 5 份输入存在：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
必须看到 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`。不要把真实论文复制进测试 fixture。
|
||||||
|
|
||||||
|
## 3. 运行实验
|
||||||
|
|
||||||
|
人工选择一个当天未使用的安全运行 ID：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/run_clindb_first_batch_experiment.py \
|
||||||
|
--run-id clindb-first-batch-review
|
||||||
|
```
|
||||||
|
|
||||||
|
成功时终端只显示运行身份和汇总，不打印原文：
|
||||||
|
|
||||||
|
```text
|
||||||
|
run_id=clindb-first-batch-review
|
||||||
|
status=success
|
||||||
|
documents=5
|
||||||
|
changes=155
|
||||||
|
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-first-batch-review
|
||||||
|
```
|
||||||
|
|
||||||
|
同一天同名目录已存在时脚本会拒绝覆盖。需要重跑时使用新 ID，不要删除旧目录来绕过检查。
|
||||||
|
|
||||||
|
## 4. 先看哪些结果
|
||||||
|
|
||||||
|
先打开运行目录的 `manifest.json`，确认：
|
||||||
|
|
||||||
|
- `run.status` 是 `success`；
|
||||||
|
- `summary.document_count` 和 `summary.success_count` 都是 5；
|
||||||
|
- `summary.failed_count`、`summary.unstable_count` 都是 0；
|
||||||
|
- `summary.change_count` 是 155；
|
||||||
|
- `pipeline.components` 的顺序与 `design/0006` 一致。
|
||||||
|
|
||||||
|
然后查看每份文档目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
documents/<document_id>/
|
||||||
|
├── result.json
|
||||||
|
├── cleaned.md
|
||||||
|
└── changes.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
- `changes.diff` 用于人工查看输入到最终输出的总变化；
|
||||||
|
- `result.json` 用于按组件、理由、位置和哈希追踪每条修改；
|
||||||
|
- `cleaned.md` 是成功输出全文。
|
||||||
|
|
||||||
|
当前 5 份输入的预期计数是：
|
||||||
|
|
||||||
|
| 文档 | `Change` 数 |
|
||||||
|
| --- | ---: |
|
||||||
|
| dmp | 47 |
|
||||||
|
| ejhf | 9 |
|
||||||
|
| jama | 79 |
|
||||||
|
| sim | 3 |
|
||||||
|
| springer | 17 |
|
||||||
|
| **合计** | **155** |
|
||||||
|
|
||||||
|
按组件应为：Word 批注 2、手稿行号 75、arXiv 戳 2、重复页眉 2、映射断词 6、HTML 实体 31、
|
||||||
|
HTML 表格布局 9、参考文献空行 28。
|
||||||
|
|
||||||
|
## 5. 人工复核重点
|
||||||
|
|
||||||
|
除了逐份查看 diff，至少确认：
|
||||||
|
|
||||||
|
- JAMA 的 Abstract 前作者和单位编号仍在，只删除 Abstract 后的 75 个手稿行号；
|
||||||
|
- Springer 两条 `arXiv preprint arXiv:` 合法参考文献仍在；
|
||||||
|
- Springer 正文中的编号方法列表没有被参考文献规则整理；
|
||||||
|
- dmp 的重复页眉删除后，正文句子接回，参考文献第 18、19 条之间仍有一个空行；
|
||||||
|
- 9 张表仍是 HTML，属性和单元格内容未被布局组件改写；
|
||||||
|
- 双重实体变成单层 `<`、`>` 或 `&`，没有直接生成标签边界；
|
||||||
|
- 图片引用文字保持不变。
|
||||||
|
|
||||||
|
清洗目录没有复制图片资产，所以直接打开 `cleaned.md` 时图片仍可能无法显示。这不表示图片引用被清洗组件删除；
|
||||||
|
资产打包和路径改写需要单独设计。
|
||||||
|
|
||||||
|
## 6. 验证幂等和输入不变
|
||||||
|
|
||||||
|
流水线会在每份文档结束时做最终稳定性复查。需要额外复核整个保存结果时，可以把 `cleaned.md` 作为内存输入再次运行
|
||||||
|
同一 `build_pipeline()`；5 份都应为 `success` 且合计零 `Change`。
|
||||||
|
|
||||||
|
实验层已经在发布前后复读输入并比较字节哈希。需要人工记录运行前后的摘要时，可在运行前后分别执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sha256sum data/md/*.md
|
||||||
|
```
|
||||||
|
|
||||||
|
两次输出必须逐项一致。每个 `cleaned.md` 的 SHA-256 还必须等于对应 `result.json.current_sha256`。
|
||||||
|
|
||||||
|
## 7. 常见失败
|
||||||
|
|
||||||
|
### 状态不是 `success`
|
||||||
|
|
||||||
|
查看对应 `result.json` 的 `errors` 或 `residual_proposals`。`failed` / `unstable` 文档不会有正式 `cleaned.md`，
|
||||||
|
不能把其他文档的部分成功当成整批成功。
|
||||||
|
|
||||||
|
### 修改数不是 155
|
||||||
|
|
||||||
|
先按组件和文档分组定位差异。输入变化、组件参数变化或识别边界变化都必须回到 design/reference 核对；不要放宽断言、
|
||||||
|
补跑第二轮或手工改产物。
|
||||||
|
|
||||||
|
### 图片不显示
|
||||||
|
|
||||||
|
当前运行只保存 Markdown、审计和 diff，不复制图片。不要为了显示图片而修改输入路径或把真实资产强制加入 Git。
|
||||||
|
|
||||||
|
### 私有目录无法被 Snap 工具读取
|
||||||
|
|
||||||
|
运行目录权限是 `0700`，文件是 `0600`。使用普通编辑器或当前虚拟环境中的 Python 读取，不要放宽权限。
|
||||||
|
|
||||||
|
## 8. 数据边界
|
||||||
|
|
||||||
|
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||||
|
其他仓库、云存储或外部系统。
|
||||||
|
|
||||||
|
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。当前不自动删除；到期后如需清理，必须先确认具体运行目录。
|
||||||
@@ -11,8 +11,8 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
||||||
是它的首批输入，未来会继续扩充同源转换产物。
|
是它的首批输入，未来会继续扩充同源转换产物。
|
||||||
|
|
||||||
本仓库（mdpolish）为该项目的数据提供清洗能力；ClinDB-ReviewBench 通过 profile
|
本仓库（mdpolish）为该项目的数据提供清洗能力。当前由仓库内 first-batch 实验脚本显式组合论文规则，
|
||||||
表达论文场景的规则组合，不把论文专属规则写进通用核心。
|
尚未建立 Profile 对象或配置格式；论文专属规则仍不写进通用核心。
|
||||||
|
|
||||||
## 2. 第一版清洗目标
|
## 2. 第一版清洗目标
|
||||||
|
|
||||||
@@ -21,14 +21,14 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
|
|
||||||
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 1 | HTML 实体双重转义（D2） | `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，还原一层；幂等 | dmp L27/L33 共 23 处、ejhf L143 共 8 处，全部在 `<table>` 行内 |
|
| 1 | HTML 实体双重转义（D2） | 只在严格 HTML 表格单元格文本中把 `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，解除源码的一层转义；幂等 | dmp 23 处、ejhf 8 处，共 31 处 |
|
||||||
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
||||||
| 3 | Word 审阅批注（B2） | 以 `Commented [xx]:` 开头的行及其批注正文整块删除 | jama L155–157 共 2 处 |
|
| 3 | Word 审阅批注（B2） | 删除严格单行 `Commented [xx]: ...` 及其后第一个空行；不猜测多行批注正文 | jama 2 处 |
|
||||||
| 4 | 手稿行号（B1） | 仅剥离**单调递增序列**的行首 `^\d{1,3} ` 与标题内 `^#{1,6} \d{1,3} `；序列中断即停止提出后续修改，防止误伤正文数字（如 "35 pediatric experts"） | jama 128 行正文 + 6 个标题（L127/149/151/165/193/195） |
|
| 4 | 手稿行号（B1） | 只处理唯一 `## Abstract` 后至少 20 个、严格递增且含至少 2 个标题证据的完整序列；剥离普通行和标题中的数字前缀 | jama 75 处，其中普通行 69、标题 6；Abstract 前 59 个作者单位编号必须保留 |
|
||||||
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
||||||
| 6 | 单行 HTML 表格展开（D1） | 无 rowspan/colspan 的表转多行 GFM；含合并属性的表保留 HTML 但按 `<tr>` 换行缩进；内容一字不改 | 全部 9 个表：dmp 7、ejhf 1、springer 1 |
|
| 6 | 单行 HTML 表格展开（D1） | 严格完整的单行表格保留 HTML、属性和单元格内容，只按 `<tr>` 换行缩进；当前不转 GFM | 全部 9 个表：dmp 7、ejhf 1、springer 1；每张都有非 `1` 的 `colspan` |
|
||||||
| 7 | 跨页断词（E2） | 行尾连字符 + 下一行首小写字母 → 合并；仅在拼出的词能通过英文词表校验时执行，否则保留原样且不提出修改 | dmp L125/127（thresh-olds）、sim L87/89（possi-bly）、L217/219（cre-ated） |
|
| 7 | 跨页断词（E2） | 只按项目批准的左右片段和结果词映射合并相邻行或只隔一个空行的片段；不使用英文词表猜测 | dmp 2 处、jama 2 处、sim 2 处，共 6 处 |
|
||||||
| 8 | 参考文献分隔统一（G3） | `^\d+\. ` 条目之间统一一个空行 | dmp refs 19–33、springer refs 9–19（连续堆叠段） |
|
| 8 | 参考文献分隔统一（G3） | 只在准确 References 章节内，对从 1 开始连续递增的编号条目统一一个空行 | dmp 13 处、springer 15 处，共 28 处 |
|
||||||
|
|
||||||
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
||||||
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
||||||
@@ -54,20 +54,24 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
|
|
||||||
## 4. 输入输出边界
|
## 4. 输入输出边界
|
||||||
|
|
||||||
- 项目输入位于 `data/<转换结果目录>/markdowns/*.md`，原文件只读；
|
- 当前本地输入是 `data/md/` 中按论文缩写命名的 5 份 Markdown 副本，原文件只读；
|
||||||
- 当前核心只接收内存 Markdown 字符串并返回内存结果；文件输出位置和保存流程尚未设计，不回写、不覆盖原文件；
|
- 核心仍只接收内存 Markdown 字符串；本地实验层按 `design/0005` 将成功输出、JSON 审计和 diff 保存到
|
||||||
- 当前批次不读取图片资产；未来 Inspector 和路径改写分别设计；
|
`artifacts/<YYYY-MM-DD>/runs/<run_id>/`，不回写、不覆盖输入；
|
||||||
- 每处实际修改必须在内存结果中记录组件、理由、原文、改后内容和批次哈希，保证可追踪。
|
- 当前批次不读取或复制图片资产；未来 Inspector、资产打包和路径改写分别设计；
|
||||||
|
- 每处实际修改记录组件、理由、原文、改后内容和批次哈希，完整落盘字段以代码和 `design/0005` 为准。
|
||||||
|
|
||||||
## 5. 验收口径（第一版）
|
## 5. 验收口径（第一版）
|
||||||
|
|
||||||
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，处理数与审计报告的实测数字一致；
|
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，共产生 155 条 `Change`；
|
||||||
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
||||||
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
||||||
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
||||||
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
||||||
等数字开头/含数字短语不受影响。
|
等数字开头/含数字短语不受影响。
|
||||||
|
|
||||||
|
2026-08-22 的保存型验收中，5 份文档全部为 `success`，各组件计数为：批注 2、行号 75、arXiv 2、页眉 2、
|
||||||
|
断词 6、HTML 实体 31、表格布局 9、参考文献空行 28。第二次运行 5 份合计 0 条修改，输入运行前后不变。
|
||||||
|
|
||||||
## 6. 与审计报告的编号对应
|
## 6. 与审计报告的编号对应
|
||||||
|
|
||||||
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
"""Run the approved first-batch ClinDB pipeline over five local paper copies."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from mdpolish.components import (
|
||||||
|
ArxivSubmissionStampComponent,
|
||||||
|
HtmlTableDoubleEscapeComponent,
|
||||||
|
HtmlTableLayoutComponent,
|
||||||
|
ManuscriptLineNumberComponent,
|
||||||
|
PageBreakWordJoinComponent,
|
||||||
|
ReferenceSpacingComponent,
|
||||||
|
RepeatedRunningHeaderComponent,
|
||||||
|
WordReviewCommentComponent,
|
||||||
|
)
|
||||||
|
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||||
|
from mdpolish.models import RunStatus
|
||||||
|
from mdpolish.pipeline import Pipeline
|
||||||
|
|
||||||
|
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||||
|
CLINDB_WORD_JOIN_MAPPINGS = (
|
||||||
|
("medi-", "cal", "medical"),
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("cre-", "ated", "created"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("life-", "threatening", "life-threatening"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build_pipeline() -> Pipeline:
|
||||||
|
"""Build the frozen ClinDB first-batch component order."""
|
||||||
|
return Pipeline(
|
||||||
|
[
|
||||||
|
WordReviewCommentComponent(),
|
||||||
|
ManuscriptLineNumberComponent(),
|
||||||
|
ArxivSubmissionStampComponent(),
|
||||||
|
RepeatedRunningHeaderComponent(),
|
||||||
|
PageBreakWordJoinComponent(CLINDB_WORD_JOIN_MAPPINGS),
|
||||||
|
HtmlTableDoubleEscapeComponent(),
|
||||||
|
HtmlTableLayoutComponent(),
|
||||||
|
ReferenceSpacingComponent(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = _parse_args()
|
||||||
|
repository_root = Path(__file__).resolve().parents[1]
|
||||||
|
documents = tuple(
|
||||||
|
InputDocument(
|
||||||
|
document_id=document_id,
|
||||||
|
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||||
|
source_label=f"data/md/{document_id}.md",
|
||||||
|
)
|
||||||
|
for document_id in _DOCUMENT_IDS
|
||||||
|
)
|
||||||
|
started_at = datetime.now().astimezone()
|
||||||
|
try:
|
||||||
|
result = run_experiment(
|
||||||
|
pipeline=build_pipeline(),
|
||||||
|
documents=documents,
|
||||||
|
run_id=args.run_id,
|
||||||
|
artifacts_root=repository_root / "artifacts",
|
||||||
|
started_at=started_at,
|
||||||
|
tool=collect_tool_metadata(repository_root),
|
||||||
|
)
|
||||||
|
except Exception as error:
|
||||||
|
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print(f"run_id={args.run_id}")
|
||||||
|
print(f"status={result.status.value}")
|
||||||
|
print(f"documents={result.document_count}")
|
||||||
|
print(f"changes={result.change_count}")
|
||||||
|
print(f"artifacts={result.run_directory}")
|
||||||
|
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
"""Fail-closed lexical ranges for the approved raw HTML table subset."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlCell:
|
||||||
|
"""The text-content range of one strict table cell."""
|
||||||
|
|
||||||
|
content_start: int
|
||||||
|
content_end: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlRow:
|
||||||
|
"""The full source range of one strict table row."""
|
||||||
|
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
cells: tuple[HtmlCell, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlTable:
|
||||||
|
"""Exact source ranges needed by table cleaning components."""
|
||||||
|
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
opening_end: int
|
||||||
|
closing_start: int
|
||||||
|
rows: tuple[HtmlRow, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _Tag:
|
||||||
|
name: str
|
||||||
|
closing: bool
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_tag(markdown: str, start: int) -> _Tag | None:
|
||||||
|
if start >= len(markdown) or markdown[start] != "<":
|
||||||
|
return None
|
||||||
|
|
||||||
|
position = start + 1
|
||||||
|
closing = position < len(markdown) and markdown[position] == "/"
|
||||||
|
if closing:
|
||||||
|
position += 1
|
||||||
|
|
||||||
|
name_start = position
|
||||||
|
while position < len(markdown) and markdown[position].isascii() and markdown[position].isalnum():
|
||||||
|
position += 1
|
||||||
|
if position == name_start:
|
||||||
|
return None
|
||||||
|
name = markdown[name_start:position]
|
||||||
|
if position >= len(markdown) or (not markdown[position].isspace() and markdown[position] != ">"):
|
||||||
|
return None
|
||||||
|
|
||||||
|
body_start = position
|
||||||
|
quote: str | None = None
|
||||||
|
while position < len(markdown):
|
||||||
|
character = markdown[position]
|
||||||
|
if quote is not None:
|
||||||
|
if character == quote:
|
||||||
|
quote = None
|
||||||
|
elif character in {'"', "'"}:
|
||||||
|
quote = character
|
||||||
|
elif character == "<":
|
||||||
|
return None
|
||||||
|
elif character == ">":
|
||||||
|
body = markdown[body_start:position]
|
||||||
|
if closing and body.strip():
|
||||||
|
return None
|
||||||
|
if not closing and body.rstrip().endswith("/"):
|
||||||
|
return None
|
||||||
|
return _Tag(name=name, closing=closing, start=start, end=position + 1)
|
||||||
|
position += 1
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _skip_whitespace(markdown: str, position: int, limit: int) -> int:
|
||||||
|
while position < limit and markdown[position].isspace():
|
||||||
|
position += 1
|
||||||
|
return position
|
||||||
|
|
||||||
|
|
||||||
|
def _balanced_table_end(markdown: str, opening: _Tag) -> int | None:
|
||||||
|
depth = 1
|
||||||
|
position = opening.end
|
||||||
|
while position < len(markdown):
|
||||||
|
tag_start = markdown.find("<", position)
|
||||||
|
if tag_start < 0:
|
||||||
|
return None
|
||||||
|
tag = _parse_tag(markdown, tag_start)
|
||||||
|
if tag is None:
|
||||||
|
position = tag_start + 1
|
||||||
|
continue
|
||||||
|
if tag.name == "table":
|
||||||
|
depth += -1 if tag.closing else 1
|
||||||
|
if depth == 0:
|
||||||
|
return tag.end
|
||||||
|
position = tag.end
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _strict_table_at(markdown: str, start: int, balanced_end: int) -> HtmlTable | None:
|
||||||
|
opening = _parse_tag(markdown, start)
|
||||||
|
if opening is None or opening.closing or opening.name != "table":
|
||||||
|
return None
|
||||||
|
|
||||||
|
rows: list[HtmlRow] = []
|
||||||
|
position = _skip_whitespace(markdown, opening.end, balanced_end)
|
||||||
|
closing_start = -1
|
||||||
|
while position < balanced_end:
|
||||||
|
tag = _parse_tag(markdown, position)
|
||||||
|
if tag is None:
|
||||||
|
return None
|
||||||
|
if tag.closing and tag.name == "table":
|
||||||
|
closing_start = tag.start
|
||||||
|
position = tag.end
|
||||||
|
break
|
||||||
|
if tag.closing or tag.name != "tr":
|
||||||
|
return None
|
||||||
|
|
||||||
|
row_start = tag.start
|
||||||
|
position = _skip_whitespace(markdown, tag.end, balanced_end)
|
||||||
|
cells: list[HtmlCell] = []
|
||||||
|
row_end = -1
|
||||||
|
while position < balanced_end:
|
||||||
|
child = _parse_tag(markdown, position)
|
||||||
|
if child is None:
|
||||||
|
return None
|
||||||
|
if child.closing and child.name == "tr":
|
||||||
|
row_end = child.end
|
||||||
|
position = child.end
|
||||||
|
break
|
||||||
|
if child.closing or child.name not in {"td", "th"}:
|
||||||
|
return None
|
||||||
|
|
||||||
|
content_start = child.end
|
||||||
|
close_start = markdown.find("<", content_start, balanced_end)
|
||||||
|
if close_start < 0:
|
||||||
|
return None
|
||||||
|
close = _parse_tag(markdown, close_start)
|
||||||
|
if close is None or not close.closing or close.name != child.name:
|
||||||
|
return None
|
||||||
|
cells.append(HtmlCell(content_start=content_start, content_end=close_start))
|
||||||
|
position = _skip_whitespace(markdown, close.end, balanced_end)
|
||||||
|
|
||||||
|
if not cells or row_end < 0:
|
||||||
|
return None
|
||||||
|
rows.append(HtmlRow(start=row_start, end=row_end, cells=tuple(cells)))
|
||||||
|
position = _skip_whitespace(markdown, position, balanced_end)
|
||||||
|
|
||||||
|
if not rows or closing_start < 0 or position != balanced_end:
|
||||||
|
return None
|
||||||
|
return HtmlTable(
|
||||||
|
start=start,
|
||||||
|
end=balanced_end,
|
||||||
|
opening_end=opening.end,
|
||||||
|
closing_start=closing_start,
|
||||||
|
rows=tuple(rows),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def strict_html_tables(markdown: str) -> tuple[HtmlTable, ...]:
|
||||||
|
"""Return non-overlapping complete tables from the approved strict subset."""
|
||||||
|
tables: list[HtmlTable] = []
|
||||||
|
position = 0
|
||||||
|
while position < len(markdown):
|
||||||
|
start = markdown.find("<table", position)
|
||||||
|
if start < 0:
|
||||||
|
break
|
||||||
|
opening = _parse_tag(markdown, start)
|
||||||
|
if opening is None or opening.closing or opening.name != "table":
|
||||||
|
position = start + 1
|
||||||
|
continue
|
||||||
|
balanced_end = _balanced_table_end(markdown, opening)
|
||||||
|
if balanced_end is None:
|
||||||
|
position = opening.end
|
||||||
|
continue
|
||||||
|
table = _strict_table_at(markdown, start, balanced_end)
|
||||||
|
if table is not None:
|
||||||
|
tables.append(table)
|
||||||
|
position = balanced_end
|
||||||
|
return tuple(tables)
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""Exact physical-line ranges shared by text cleaning components."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PhysicalLine:
|
||||||
|
"""A content range and its optional line ending in one source string."""
|
||||||
|
|
||||||
|
content_start: int
|
||||||
|
content_end: int
|
||||||
|
full_end: int
|
||||||
|
|
||||||
|
def content(self, markdown: str) -> str:
|
||||||
|
return markdown[self.content_start : self.content_end]
|
||||||
|
|
||||||
|
def line_ending(self, markdown: str) -> str:
|
||||||
|
return markdown[self.content_end : self.full_end]
|
||||||
|
|
||||||
|
def is_blank(self, markdown: str) -> bool:
|
||||||
|
return self.content_start == self.content_end
|
||||||
|
|
||||||
|
|
||||||
|
def iter_physical_lines(markdown: str) -> Iterator[PhysicalLine]:
|
||||||
|
"""Yield exact physical lines without normalizing CR or LF endings."""
|
||||||
|
line_start = 0
|
||||||
|
position = 0
|
||||||
|
while position < len(markdown):
|
||||||
|
character = markdown[position]
|
||||||
|
if character == "\n":
|
||||||
|
yield PhysicalLine(line_start, position, position + 1)
|
||||||
|
position += 1
|
||||||
|
line_start = position
|
||||||
|
elif character == "\r":
|
||||||
|
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
|
||||||
|
yield PhysicalLine(line_start, position, line_end)
|
||||||
|
position = line_end
|
||||||
|
line_start = position
|
||||||
|
else:
|
||||||
|
position += 1
|
||||||
|
|
||||||
|
if line_start < len(markdown):
|
||||||
|
yield PhysicalLine(line_start, len(markdown), len(markdown))
|
||||||
|
|
||||||
|
|
||||||
|
def physical_lines(markdown: str) -> tuple[PhysicalLine, ...]:
|
||||||
|
"""Return all physical-line ranges as an immutable sequence."""
|
||||||
|
return tuple(iter_physical_lines(markdown))
|
||||||
|
|
||||||
|
|
||||||
|
def line_ending_styles(markdown: str) -> frozenset[str]:
|
||||||
|
"""Return the distinct non-empty physical line endings in a string."""
|
||||||
|
return frozenset(
|
||||||
|
line.line_ending(markdown)
|
||||||
|
for line in iter_physical_lines(markdown)
|
||||||
|
if line.full_end > line.content_end
|
||||||
|
)
|
||||||
@@ -1,5 +1,22 @@
|
|||||||
"""Approved business cleaning components."""
|
"""Approved business cleaning components."""
|
||||||
|
|
||||||
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
||||||
|
from mdpolish.components.html_table_double_escape import HtmlTableDoubleEscapeComponent
|
||||||
|
from mdpolish.components.html_table_layout import HtmlTableLayoutComponent
|
||||||
|
from mdpolish.components.manuscript_line_number import ManuscriptLineNumberComponent
|
||||||
|
from mdpolish.components.page_break_word_join import PageBreakWordJoinComponent, WordJoinMapping
|
||||||
|
from mdpolish.components.reference_spacing import ReferenceSpacingComponent
|
||||||
|
from mdpolish.components.repeated_running_header import RepeatedRunningHeaderComponent
|
||||||
|
from mdpolish.components.word_review_comment import WordReviewCommentComponent
|
||||||
|
|
||||||
__all__ = ["ArxivSubmissionStampComponent"]
|
__all__ = [
|
||||||
|
"ArxivSubmissionStampComponent",
|
||||||
|
"HtmlTableDoubleEscapeComponent",
|
||||||
|
"HtmlTableLayoutComponent",
|
||||||
|
"ManuscriptLineNumberComponent",
|
||||||
|
"PageBreakWordJoinComponent",
|
||||||
|
"ReferenceSpacingComponent",
|
||||||
|
"RepeatedRunningHeaderComponent",
|
||||||
|
"WordJoinMapping",
|
||||||
|
"WordReviewCommentComponent",
|
||||||
|
]
|
||||||
|
|||||||
@@ -3,9 +3,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from collections.abc import Iterator, Mapping
|
from collections.abc import Mapping
|
||||||
from types import MappingProxyType
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import iter_physical_lines
|
||||||
from mdpolish.component import Component
|
from mdpolish.component import Component
|
||||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
@@ -18,28 +19,6 @@ _EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
|||||||
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||||
|
|
||||||
|
|
||||||
def _physical_line_ranges(markdown: str) -> Iterator[tuple[int, int, int]]:
|
|
||||||
"""Yield content start, content end, and full line end for CR/LF line endings."""
|
|
||||||
line_start = 0
|
|
||||||
position = 0
|
|
||||||
while position < len(markdown):
|
|
||||||
character = markdown[position]
|
|
||||||
if character == "\n":
|
|
||||||
yield line_start, position, position + 1
|
|
||||||
position += 1
|
|
||||||
line_start = position
|
|
||||||
elif character == "\r":
|
|
||||||
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
|
|
||||||
yield line_start, position, line_end
|
|
||||||
position = line_end
|
|
||||||
line_start = position
|
|
||||||
else:
|
|
||||||
position += 1
|
|
||||||
|
|
||||||
if line_start < len(markdown):
|
|
||||||
yield line_start, len(markdown), len(markdown)
|
|
||||||
|
|
||||||
|
|
||||||
class ArxivSubmissionStampComponent(Component):
|
class ArxivSubmissionStampComponent(Component):
|
||||||
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
||||||
|
|
||||||
@@ -61,13 +40,13 @@ class ArxivSubmissionStampComponent(Component):
|
|||||||
|
|
||||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
proposals: list[ProposedChange] = []
|
proposals: list[ProposedChange] = []
|
||||||
for line_start, content_end, line_end in _physical_line_ranges(snapshot.markdown):
|
for line in iter_physical_lines(snapshot.markdown):
|
||||||
if _STAMP_PATTERN.fullmatch(snapshot.markdown[line_start:content_end]) is None:
|
if _STAMP_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||||
continue
|
continue
|
||||||
expected_text = snapshot.markdown[line_start:line_end]
|
expected_text = snapshot.markdown[line.content_start : line.full_end]
|
||||||
edit = TextEdit(
|
edit = TextEdit(
|
||||||
snapshot_sha256=snapshot.sha256,
|
snapshot_sha256=snapshot.sha256,
|
||||||
span=TextSpan(line_start, line_end),
|
span=TextSpan(line.content_start, line.full_end),
|
||||||
expected_text=expected_text,
|
expected_text=expected_text,
|
||||||
replacement="",
|
replacement="",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
"""Remove one approved escape layer from strict HTML table cell text."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._html_table import strict_html_tables
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_ENTITY_PATTERN = re.compile(r"&(?:lt|gt|amp);")
|
||||||
|
_REPLACEMENTS = {
|
||||||
|
"&lt;": "<",
|
||||||
|
"&gt;": ">",
|
||||||
|
"&amp;": "&",
|
||||||
|
}
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "在严格 HTML 表格单元格文本中解除一层实体转义"
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlTableDoubleEscapeComponent(Component):
|
||||||
|
"""Unescape only approved double entities inside strict cell text."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "markdown.html_table_double_escape"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;，"
|
||||||
|
"排除属性、表格外文本和其他实体。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for table in strict_html_tables(snapshot.markdown):
|
||||||
|
for row in table.rows:
|
||||||
|
for cell in row.cells:
|
||||||
|
for match in _ENTITY_PATTERN.finditer(snapshot.markdown, cell.content_start, cell.content_end):
|
||||||
|
edits.append((match.start(), match.end(), _REPLACEMENTS[match.group()]))
|
||||||
|
edits.sort(key=lambda item: item[0])
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Expand strict one-line HTML tables to one row per physical line."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._html_table import strict_html_tables
|
||||||
|
from mdpolish._text_ranges import line_ending_styles
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "展开严格单行 HTML 表格的行布局"
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlTableLayoutComponent(Component):
|
||||||
|
"""Preserve strict HTML table bytes while placing each row on its own line."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "markdown.html_table_layout"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理严格完整且当前为单物理行的 HTML 表格，只规范 table/tr 外层布局；"
|
||||||
|
"排除混合行尾、复杂 HTML 和已是多行的表格。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
styles = line_ending_styles(snapshot.markdown)
|
||||||
|
if len(styles) > 1:
|
||||||
|
return ()
|
||||||
|
newline = next(iter(styles), "\n")
|
||||||
|
|
||||||
|
proposals: list[ProposedChange] = []
|
||||||
|
for table in strict_html_tables(snapshot.markdown):
|
||||||
|
expected = snapshot.markdown[table.start : table.end]
|
||||||
|
if "\r" in expected or "\n" in expected:
|
||||||
|
continue
|
||||||
|
opening = snapshot.markdown[table.start : table.opening_end]
|
||||||
|
rows = [snapshot.markdown[row.start : row.end] for row in table.rows]
|
||||||
|
closing = snapshot.markdown[table.closing_start : table.end]
|
||||||
|
replacement = opening + newline + newline.join(f" {row}" for row in rows) + newline + closing
|
||||||
|
proposals.append(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(table.start, table.end),
|
||||||
|
expected_text=expected,
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(proposals)
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Remove line-number prefixes from an evidence-backed manuscript sequence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_PLAIN_PATTERN = re.compile(r"^([1-9][0-9]{0,2}) (?=\S)")
|
||||||
|
_HEADING_PATTERN = re.compile(r"^(#{1,6}) ([1-9][0-9]{0,2}) (?=\S)")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除已确认手稿序列中的行号前缀"
|
||||||
|
_MINIMUM_CANDIDATES = 20
|
||||||
|
_MINIMUM_HEADING_CANDIDATES = 2
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate(line: PhysicalLine, markdown: str) -> tuple[int, int, int, bool] | None:
|
||||||
|
content = line.content(markdown)
|
||||||
|
heading_match = _HEADING_PATTERN.match(content)
|
||||||
|
if heading_match is not None:
|
||||||
|
number_start = line.content_start + len(heading_match.group(1)) + 1
|
||||||
|
return int(heading_match.group(2)), number_start, line.content_start + heading_match.end(), True
|
||||||
|
|
||||||
|
plain_match = _PLAIN_PATTERN.match(content)
|
||||||
|
if plain_match is None:
|
||||||
|
return None
|
||||||
|
return int(plain_match.group(1)), line.content_start, line.content_start + plain_match.end(), False
|
||||||
|
|
||||||
|
|
||||||
|
class ManuscriptLineNumberComponent(Component):
|
||||||
|
"""Strip prefixes only from one long, monotonic post-Abstract sequence."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.manuscript_line_number"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理唯一 ## Abstract 后至少 20 个严格递增且含至少 2 个标题证据的手稿行号序列，"
|
||||||
|
"排除作者单位、列表和短数字片段。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
abstract_indexes = [
|
||||||
|
index for index, line in enumerate(lines) if line.content(snapshot.markdown) == "## Abstract"
|
||||||
|
]
|
||||||
|
if len(abstract_indexes) != 1:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
candidates = [
|
||||||
|
candidate
|
||||||
|
for line in lines[abstract_indexes[0] + 1 :]
|
||||||
|
if (candidate := _candidate(line, snapshot.markdown)) is not None
|
||||||
|
]
|
||||||
|
if len(candidates) < _MINIMUM_CANDIDATES:
|
||||||
|
return ()
|
||||||
|
if sum(is_heading for _, _, _, is_heading in candidates) < _MINIMUM_HEADING_CANDIDATES:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
numbers = [number for number, _, _, _ in candidates]
|
||||||
|
if any(previous >= current for previous, current in pairwise(numbers)):
|
||||||
|
return ()
|
||||||
|
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement="",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for _, start, end, _ in candidates
|
||||||
|
)
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
"""Join approved word fragments split across adjacent physical lines."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import Counter
|
||||||
|
from collections.abc import Iterable, Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import physical_lines
|
||||||
|
from mdpolish.component import Component, ComponentContractError
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
WordJoinMapping = tuple[str, str, str]
|
||||||
|
|
||||||
|
|
||||||
|
def _is_ascii_letter(character: str) -> bool:
|
||||||
|
return character in "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||||
|
|
||||||
|
|
||||||
|
def _validated_mappings(mappings: Iterable[WordJoinMapping]) -> tuple[WordJoinMapping, ...]:
|
||||||
|
normalized: list[WordJoinMapping] = []
|
||||||
|
seen_pairs: set[tuple[str, str]] = set()
|
||||||
|
for mapping in mappings:
|
||||||
|
if not isinstance(mapping, tuple) or len(mapping) != 3 or any(not isinstance(value, str) for value in mapping):
|
||||||
|
raise ComponentContractError("word join mappings must be three-string tuples")
|
||||||
|
left, right, replacement = mapping
|
||||||
|
if not left or not right or not replacement:
|
||||||
|
raise ComponentContractError("word join mapping fields must be non-empty")
|
||||||
|
if any(character.isspace() for value in mapping for character in value):
|
||||||
|
raise ComponentContractError("word join mapping fields cannot contain whitespace")
|
||||||
|
pair = (left, right)
|
||||||
|
if pair in seen_pairs:
|
||||||
|
raise ComponentContractError("word join mappings cannot repeat a fragment pair")
|
||||||
|
seen_pairs.add(pair)
|
||||||
|
normalized.append(mapping)
|
||||||
|
return tuple(sorted(normalized))
|
||||||
|
|
||||||
|
|
||||||
|
class PageBreakWordJoinComponent(Component):
|
||||||
|
"""Apply only caller-approved exact fragment-to-word mappings."""
|
||||||
|
|
||||||
|
def __init__(self, mappings: Iterable[WordJoinMapping]) -> None:
|
||||||
|
self._mappings = _validated_mappings(mappings)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.page_break_word_join"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
records = tuple(
|
||||||
|
MappingProxyType(
|
||||||
|
{
|
||||||
|
"left_fragment": left,
|
||||||
|
"right_fragment": right,
|
||||||
|
"replacement": replacement,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
for left, right, replacement in self._mappings
|
||||||
|
)
|
||||||
|
return MappingProxyType({"mappings": records})
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理显式批准的左右词片段，片段必须位于相邻物理行或只隔一个空行并满足 ASCII 词边界；"
|
||||||
|
"排除未配置词和段落级猜测。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
matches: list[tuple[int, int, str, str, str]] = []
|
||||||
|
for index, left_line in enumerate(lines[:-1]):
|
||||||
|
left_text = left_line.content(snapshot.markdown)
|
||||||
|
left_ending = left_line.line_ending(snapshot.markdown)
|
||||||
|
if not left_ending:
|
||||||
|
continue
|
||||||
|
for left, right, replacement in self._mappings:
|
||||||
|
if not left_text.endswith(left):
|
||||||
|
continue
|
||||||
|
left_start = left_line.content_end - len(left)
|
||||||
|
if left_start > left_line.content_start and _is_ascii_letter(snapshot.markdown[left_start - 1]):
|
||||||
|
continue
|
||||||
|
|
||||||
|
right_index = index + 1
|
||||||
|
if lines[right_index].is_blank(snapshot.markdown):
|
||||||
|
blank = lines[right_index]
|
||||||
|
if blank.line_ending(snapshot.markdown) != left_ending or right_index + 1 >= len(lines):
|
||||||
|
continue
|
||||||
|
right_index += 1
|
||||||
|
if lines[right_index].is_blank(snapshot.markdown):
|
||||||
|
continue
|
||||||
|
right_line = lines[right_index]
|
||||||
|
right_text = right_line.content(snapshot.markdown)
|
||||||
|
if not right_text.startswith(right):
|
||||||
|
continue
|
||||||
|
if len(right_text) > len(right) and _is_ascii_letter(right_text[len(right)]):
|
||||||
|
continue
|
||||||
|
end = right_line.content_start + len(right)
|
||||||
|
matches.append((left_start, end, replacement, left, right))
|
||||||
|
|
||||||
|
matches.sort(key=lambda item: (item[0], item[1], item[2]))
|
||||||
|
span_counts = Counter(match[:2] for match in matches)
|
||||||
|
unique_matches = [match for match in matches if span_counts[match[:2]] == 1]
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(unique_matches)):
|
||||||
|
return ()
|
||||||
|
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=f"按已批准映射修复跨页断词：{left} + {right} → {replacement}",
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement, left, right in unique_matches
|
||||||
|
)
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
"""Normalize blank lines between numbered entries in References sections."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_ATX_HEADING_PATTERN = re.compile(r"^(#{1,6}) (\S(?:.*\S)?)$")
|
||||||
|
_ENTRY_PATTERN = re.compile(r"^([1-9][0-9]*)\. \S")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "统一 References 章节中相邻编号条目之间的一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
def _heading(line: PhysicalLine, markdown: str) -> tuple[int, str] | None:
|
||||||
|
match = _ATX_HEADING_PATTERN.fullmatch(line.content(markdown))
|
||||||
|
if match is None:
|
||||||
|
return None
|
||||||
|
return len(match.group(1)), match.group(2)
|
||||||
|
|
||||||
|
|
||||||
|
class ReferenceSpacingComponent(Component):
|
||||||
|
"""Normalize only complete, consecutive numbered reference sections."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.reference_spacing"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理准确 References ATX 章节内从 1 开始逐项递增的编号条目，统一为一个空行；"
|
||||||
|
"排除章节外列表、缺号和混合行尾边界。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for heading_index, line in enumerate(lines):
|
||||||
|
heading = _heading(line, snapshot.markdown)
|
||||||
|
if heading is None or heading[1].lower() != "references":
|
||||||
|
continue
|
||||||
|
heading_level = heading[0]
|
||||||
|
section_end = len(lines)
|
||||||
|
for index in range(heading_index + 1, len(lines)):
|
||||||
|
following_heading = _heading(lines[index], snapshot.markdown)
|
||||||
|
if following_heading is not None and following_heading[0] <= heading_level:
|
||||||
|
section_end = index
|
||||||
|
break
|
||||||
|
|
||||||
|
entries: list[tuple[int, int]] = []
|
||||||
|
section_is_safe = True
|
||||||
|
for index in range(heading_index + 1, section_end):
|
||||||
|
content = lines[index].content(snapshot.markdown)
|
||||||
|
match = _ENTRY_PATTERN.match(content)
|
||||||
|
if match is not None:
|
||||||
|
entries.append((index, int(match.group(1))))
|
||||||
|
elif _heading(lines[index], snapshot.markdown) is not None:
|
||||||
|
section_is_safe = False
|
||||||
|
if not section_is_safe or len(entries) < 2:
|
||||||
|
continue
|
||||||
|
if [number for _, number in entries] != list(range(1, len(entries) + 1)):
|
||||||
|
continue
|
||||||
|
|
||||||
|
section_edits: list[tuple[int, int, str]] = []
|
||||||
|
for previous, following in pairwise(entries):
|
||||||
|
previous_index = previous[0]
|
||||||
|
next_index = following[0]
|
||||||
|
last_nonblank_index = next(
|
||||||
|
index
|
||||||
|
for index in range(next_index - 1, previous_index - 1, -1)
|
||||||
|
if not lines[index].is_blank(snapshot.markdown)
|
||||||
|
)
|
||||||
|
between = lines[last_nonblank_index + 1 : next_index]
|
||||||
|
if any(not blank.is_blank(snapshot.markdown) for blank in between):
|
||||||
|
section_is_safe = False
|
||||||
|
break
|
||||||
|
line_ending = lines[last_nonblank_index].line_ending(snapshot.markdown)
|
||||||
|
if not line_ending or any(blank.line_ending(snapshot.markdown) != line_ending for blank in between):
|
||||||
|
section_is_safe = False
|
||||||
|
break
|
||||||
|
start = lines[last_nonblank_index].content_end
|
||||||
|
end = lines[next_index].content_start
|
||||||
|
replacement = line_ending * 2
|
||||||
|
if snapshot.markdown[start:end] != replacement:
|
||||||
|
section_edits.append((start, end, replacement))
|
||||||
|
if section_is_safe:
|
||||||
|
edits.extend(section_edits)
|
||||||
|
|
||||||
|
edits.sort(key=lambda item: item[0])
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||||
|
return ()
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""Remove repeated ATX running headers with paragraph-interruption evidence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import defaultdict
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_HEADING_PATTERN = re.compile(r"^#{1,6} \S(?:.*\S)?$")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除经重复和断句证据确认的跑动页眉"
|
||||||
|
_TERMINAL_PUNCTUATION = frozenset(".?!:;。！？：；")
|
||||||
|
|
||||||
|
|
||||||
|
def _bridge_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||||
|
if index < 2 or index + 2 >= len(lines):
|
||||||
|
return None
|
||||||
|
before_blank = lines[index - 1]
|
||||||
|
after_blank = lines[index + 1]
|
||||||
|
previous = lines[index - 2]
|
||||||
|
following = lines[index + 2]
|
||||||
|
if not before_blank.is_blank(markdown) or not after_blank.is_blank(markdown):
|
||||||
|
return None
|
||||||
|
if previous.is_blank(markdown) or following.is_blank(markdown):
|
||||||
|
return None
|
||||||
|
|
||||||
|
previous_text = previous.content(markdown).rstrip()
|
||||||
|
following_text = following.content(markdown)
|
||||||
|
if not previous_text or previous_text[-1] in _TERMINAL_PUNCTUATION:
|
||||||
|
return None
|
||||||
|
if not following_text or following_text[0] not in "abcdefghijklmnopqrstuvwxyz":
|
||||||
|
return None
|
||||||
|
return previous.content_end, following.content_start
|
||||||
|
|
||||||
|
|
||||||
|
def _deletion_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||||
|
if index + 1 >= len(lines) or not lines[index + 1].is_blank(markdown):
|
||||||
|
return None
|
||||||
|
return lines[index].content_start, lines[index + 1].full_end
|
||||||
|
|
||||||
|
|
||||||
|
class RepeatedRunningHeaderComponent(Component):
|
||||||
|
"""Remove a repeated heading only when one occurrence splits a sentence."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.repeated_running_header"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理原样重复至少两次、且至少一次由小写续句证据确认切断正文的 ATX 跑动页眉；"
|
||||||
|
"任一出现位置不安全时整组保留。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
groups: defaultdict[str, list[int]] = defaultdict(list)
|
||||||
|
for index, line in enumerate(lines):
|
||||||
|
content = line.content(snapshot.markdown)
|
||||||
|
if _HEADING_PATTERN.fullmatch(content) is not None:
|
||||||
|
groups[content].append(index)
|
||||||
|
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for indexes in groups.values():
|
||||||
|
if len(indexes) < 2:
|
||||||
|
continue
|
||||||
|
group_edits: list[tuple[int, int, str]] = []
|
||||||
|
bridge_found = False
|
||||||
|
for index in indexes:
|
||||||
|
bridge = _bridge_span(lines, index, snapshot.markdown)
|
||||||
|
if bridge is not None:
|
||||||
|
group_edits.append((*bridge, " "))
|
||||||
|
bridge_found = True
|
||||||
|
continue
|
||||||
|
deletion = _deletion_span(lines, index, snapshot.markdown)
|
||||||
|
if deletion is None:
|
||||||
|
group_edits = []
|
||||||
|
break
|
||||||
|
group_edits.append((*deletion, ""))
|
||||||
|
if bridge_found and group_edits:
|
||||||
|
edits.extend(group_edits)
|
||||||
|
|
||||||
|
edits.sort(key=lambda item: (item[0], item[1]))
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||||
|
return ()
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""Remove strict single-line Word review comments."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_COMMENT_PATTERN = re.compile(r"^Commented \[[A-Za-z0-9]+\]: .*\S$")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除严格单行 Word 审阅批注及其后一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
class WordReviewCommentComponent(Component):
|
||||||
|
"""Delete approved one-line Word comments followed by a blank line."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.word_review_comment"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return "处理独占物理行且后接空行的 Word Commented 审阅批注，排除多行正文和所有相似文本。"
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
proposals: list[ProposedChange] = []
|
||||||
|
for index, line in enumerate(lines[:-1]):
|
||||||
|
following = lines[index + 1]
|
||||||
|
if _COMMENT_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||||
|
continue
|
||||||
|
if not following.is_blank(snapshot.markdown):
|
||||||
|
continue
|
||||||
|
start = line.content_start
|
||||||
|
end = following.full_end
|
||||||
|
proposals.append(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement="",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(proposals)
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import (
|
||||||
|
ArxivSubmissionStampComponent,
|
||||||
|
HtmlTableDoubleEscapeComponent,
|
||||||
|
HtmlTableLayoutComponent,
|
||||||
|
ManuscriptLineNumberComponent,
|
||||||
|
PageBreakWordJoinComponent,
|
||||||
|
ReferenceSpacingComponent,
|
||||||
|
RepeatedRunningHeaderComponent,
|
||||||
|
WordReviewCommentComponent,
|
||||||
|
)
|
||||||
|
|
||||||
|
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||||
|
HEADER = "## Repeated Paper Header"
|
||||||
|
MAPPINGS = (
|
||||||
|
("medi-", "cal", "medical"),
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("cre-", "ated", "created"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("life-", "threatening", "life-threatening"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_pipeline() -> Pipeline:
|
||||||
|
return Pipeline(
|
||||||
|
[
|
||||||
|
WordReviewCommentComponent(),
|
||||||
|
ManuscriptLineNumberComponent(),
|
||||||
|
ArxivSubmissionStampComponent(),
|
||||||
|
RepeatedRunningHeaderComponent(),
|
||||||
|
PageBreakWordJoinComponent(MAPPINGS),
|
||||||
|
HtmlTableDoubleEscapeComponent(),
|
||||||
|
HtmlTableLayoutComponent(),
|
||||||
|
ReferenceSpacingComponent(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _numbered_manuscript() -> list[str]:
|
||||||
|
return [
|
||||||
|
f"## {number} Section {number}" if number in {5, 15} else f"{number} body {number}"
|
||||||
|
for number in range(1, 21)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _combined_markdown() -> str:
|
||||||
|
return "\n".join(
|
||||||
|
(
|
||||||
|
"1 Affiliation",
|
||||||
|
"## Abstract",
|
||||||
|
*_numbered_manuscript(),
|
||||||
|
"Commented [A1]: remove this",
|
||||||
|
"",
|
||||||
|
STAMP,
|
||||||
|
"Sentence continues in",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"the next line.",
|
||||||
|
"A word is possi-",
|
||||||
|
"",
|
||||||
|
"bly split.",
|
||||||
|
"<table><tr><td>&lt;5</td></tr><tr><td>B</td></tr></table>",
|
||||||
|
"## References",
|
||||||
|
"",
|
||||||
|
"1. First",
|
||||||
|
"",
|
||||||
|
"2. Second",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"3. Third",
|
||||||
|
"4. Fourth",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builds_frozen_component_order_and_parameters() -> None:
|
||||||
|
script_path = Path(__file__).parents[1] / "scripts" / "run_clindb_first_batch_experiment.py"
|
||||||
|
module = ast.parse(script_path.read_text(encoding="utf-8"))
|
||||||
|
build_function = next(
|
||||||
|
node for node in module.body if isinstance(node, ast.FunctionDef) and node.name == "build_pipeline"
|
||||||
|
)
|
||||||
|
component_names = [
|
||||||
|
call.func.id
|
||||||
|
for node in ast.walk(build_function)
|
||||||
|
if isinstance(node, ast.List)
|
||||||
|
for call in node.elts
|
||||||
|
if isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
|
||||||
|
]
|
||||||
|
mapping_assignment = next(
|
||||||
|
node
|
||||||
|
for node in module.body
|
||||||
|
if isinstance(node, ast.Assign)
|
||||||
|
and any(isinstance(target, ast.Name) and target.id == "CLINDB_WORD_JOIN_MAPPINGS" for target in node.targets)
|
||||||
|
)
|
||||||
|
assert component_names == [
|
||||||
|
"WordReviewCommentComponent",
|
||||||
|
"ManuscriptLineNumberComponent",
|
||||||
|
"ArxivSubmissionStampComponent",
|
||||||
|
"RepeatedRunningHeaderComponent",
|
||||||
|
"PageBreakWordJoinComponent",
|
||||||
|
"HtmlTableDoubleEscapeComponent",
|
||||||
|
"HtmlTableLayoutComponent",
|
||||||
|
"ReferenceSpacingComponent",
|
||||||
|
]
|
||||||
|
assert ast.literal_eval(mapping_assignment.value) == MAPPINGS
|
||||||
|
|
||||||
|
pipeline = _build_pipeline()
|
||||||
|
result = pipeline.transform("")
|
||||||
|
|
||||||
|
assert [component.component_id for component in result.components] == [
|
||||||
|
"paper.word_review_comment",
|
||||||
|
"paper.manuscript_line_number",
|
||||||
|
"paper.arxiv_submission_stamp",
|
||||||
|
"paper.repeated_running_header",
|
||||||
|
"paper.page_break_word_join",
|
||||||
|
"markdown.html_table_double_escape",
|
||||||
|
"markdown.html_table_layout",
|
||||||
|
"paper.reference_spacing",
|
||||||
|
]
|
||||||
|
assert len(MAPPINGS) == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_pipeline_is_audited_stable_and_idempotent() -> None:
|
||||||
|
pipeline = _build_pipeline()
|
||||||
|
first = pipeline.transform(_combined_markdown())
|
||||||
|
|
||||||
|
assert first.status is RunStatus.SUCCESS
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert first.residual_proposals == ()
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
for change in first.changes:
|
||||||
|
counts[change.component_id] = counts.get(change.component_id, 0) + 1
|
||||||
|
assert counts == {
|
||||||
|
"paper.word_review_comment": 1,
|
||||||
|
"paper.manuscript_line_number": 20,
|
||||||
|
"paper.arxiv_submission_stamp": 1,
|
||||||
|
"paper.repeated_running_header": 2,
|
||||||
|
"paper.page_break_word_join": 1,
|
||||||
|
"markdown.html_table_double_escape": 1,
|
||||||
|
"markdown.html_table_layout": 1,
|
||||||
|
"paper.reference_spacing": 1,
|
||||||
|
}
|
||||||
|
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.output_markdown == first.output_markdown
|
||||||
|
assert second.changes == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_business_components_are_not_exported_from_core_namespace() -> None:
|
||||||
|
import mdpolish
|
||||||
|
|
||||||
|
assert not hasattr(mdpolish, "WordReviewCommentComponent")
|
||||||
|
assert not hasattr(mdpolish, "HtmlTableLayoutComponent")
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import HtmlTableDoubleEscapeComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([HtmlTableDoubleEscapeComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unescapes_one_layer_only_in_strict_cell_text() -> None:
|
||||||
|
markdown = (
|
||||||
|
'<table data-note="&lt;"><tr><td>&lt;5</td><td>&gt;2 &amp; x <</td></tr></table>'
|
||||||
|
"\noutside &lt;"
|
||||||
|
)
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == (
|
||||||
|
'<table data-note="&lt;"><tr><td><5</td><td>>2 & x <</td></tr></table>'
|
||||||
|
"\noutside &lt;"
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_multiple_tables_and_cells_report_source_order() -> None:
|
||||||
|
markdown = "<table><tr><td>&gt;</td></tr></table> x <table><tr><th>&lt;</th></tr></table>"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown == "<table><tr><td>></td></tr></table> x <table><tr><th><</th></tr></table>"
|
||||||
|
assert [change.span.start for change in result.changes] == sorted(change.span.start for change in result.changes)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"outside &lt;",
|
||||||
|
"<table><tr><td>&lt;</tr></table>",
|
||||||
|
"<table><tr><td><em>&lt;</em></td></tr></table>",
|
||||||
|
"<table><tr><td><table><tr><td>&lt;</td></tr></table></td></tr></table>",
|
||||||
|
"<table><tbody><tr><td>&lt;</td></tr></tbody></table>",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_non_strict_or_outside_content_is_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_fenced_table_is_not_protected() -> None:
|
||||||
|
markdown = "```html\n<table><tr><td>&lt;</td></tr></table>\n```"
|
||||||
|
assert transform(markdown).output_markdown == "```html\n<table><tr><td><</td></tr></table>\n```"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([HtmlTableDoubleEscapeComponent()])
|
||||||
|
first = pipeline.transform("<table><tr><td>&lt;</td></tr></table>")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import HtmlTableLayoutComponent
|
||||||
|
|
||||||
|
TABLE = '<table class="x"><tr><td colspan="2">A</td></tr><tr><td>B</td><td>C</td></tr></table>'
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([HtmlTableLayoutComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_expands_rows_without_changing_tags_attributes_or_cells() -> None:
|
||||||
|
result = transform(f"before\n{TABLE}\nafter")
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == (
|
||||||
|
"before\n<table class=\"x\">\n"
|
||||||
|
" <tr><td colspan=\"2\">A</td></tr>\n"
|
||||||
|
" <tr><td>B</td><td>C</td></tr>\n"
|
||||||
|
"</table>\nafter"
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_uses_the_documents_single_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = f"before{line_ending}{TABLE}{line_ending}after"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert f"<table class=\"x\">{line_ending} <tr>" in result.output_markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_table_only_document_uses_lf() -> None:
|
||||||
|
assert transform(TABLE).output_markdown == (
|
||||||
|
'<table class="x">\n'
|
||||||
|
' <tr><td colspan="2">A</td></tr>\n'
|
||||||
|
" <tr><td>B</td><td>C</td></tr>\n"
|
||||||
|
"</table>"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"before\n<table><tr><td>A</td></tr>\r\nafter",
|
||||||
|
"<table>\n <tr><td>A</td></tr>\n</table>",
|
||||||
|
"<table><tr><td>A</tr></table>",
|
||||||
|
"<table><tbody><tr><td>A</td></tr></tbody></table>",
|
||||||
|
"<table><tr><td><em>A</em></td></tr></table>",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_mixed_multiline_or_non_strict_tables_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([HtmlTableLayoutComponent()])
|
||||||
|
first = pipeline.transform(TABLE)
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import ManuscriptLineNumberComponent
|
||||||
|
|
||||||
|
|
||||||
|
def _document(
|
||||||
|
*,
|
||||||
|
count: int = 20,
|
||||||
|
heading_numbers: frozenset[int] = frozenset({5, 15}),
|
||||||
|
numbers: tuple[int, ...] | None = None,
|
||||||
|
line_ending: str = "\n",
|
||||||
|
) -> str:
|
||||||
|
values = numbers if numbers is not None else tuple(range(1, count + 1))
|
||||||
|
body = [
|
||||||
|
f"## {number} Section {number}" if number in heading_numbers else f"{number} body {number}"
|
||||||
|
for number in values
|
||||||
|
]
|
||||||
|
return line_ending.join(("1 Affiliation", "2 Institute", "## Abstract", *body))
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([ManuscriptLineNumberComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_removes_long_monotonic_sequence_but_preserves_pre_abstract_affiliations() -> None:
|
||||||
|
result = transform(_document())
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.startswith("1 Affiliation\n2 Institute\n## Abstract\nbody 1")
|
||||||
|
assert "## Section 5" in result.output_markdown
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_all_supported_line_endings(line_ending: str) -> None:
|
||||||
|
result = transform(_document(line_ending=line_ending))
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.count(line_ending) == _document(line_ending=line_ending).count(line_ending)
|
||||||
|
|
||||||
|
|
||||||
|
def test_allows_skipped_numbers_when_sequence_is_strictly_increasing() -> None:
|
||||||
|
numbers = tuple(range(10, 30))
|
||||||
|
result = transform(_document(numbers=numbers, heading_numbers=frozenset({14, 24})))
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
_document(count=19, heading_numbers=frozenset({5, 15})),
|
||||||
|
_document(heading_numbers=frozenset({5})),
|
||||||
|
_document(numbers=(*tuple(range(1, 20)), 10), heading_numbers=frozenset({5, 15})),
|
||||||
|
_document().replace("## Abstract", "## ABSTRACT"),
|
||||||
|
_document() + "\n## Abstract",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_incomplete_or_ambiguous_evidence_preserves_the_document(markdown: str) -> None:
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == markdown
|
||||||
|
assert result.changes == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_lists_years_and_numbers_inside_body_are_not_candidates() -> None:
|
||||||
|
markdown = _document() + "\n1. list\n1) list\n2024 report\nThe panel included 35 experts"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.endswith("1. list\n1) list\n2024 report\nThe panel included 35 experts")
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([ManuscriptLineNumberComponent()])
|
||||||
|
first = pipeline.transform(_document())
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.changes == ()
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.component import ComponentContractError
|
||||||
|
from mdpolish.components import PageBreakWordJoinComponent
|
||||||
|
|
||||||
|
MAPPINGS = (
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([PageBreakWordJoinComponent(MAPPINGS)]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("markdown", "expected"),
|
||||||
|
[
|
||||||
|
("except possi-\nbly through care", "except possibly through care"),
|
||||||
|
("except possi-\n\nbly through care", "except possibly through care"),
|
||||||
|
("use SOFA-\nbased criteria", "use SOFA-based criteria"),
|
||||||
|
("at threshold.\n\nolds of eight", "at thresholds of eight"),
|
||||||
|
("except possi-\r\n\r\nbly now", "except possibly now"),
|
||||||
|
("except possi-\r\rbly now", "except possibly now"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_applies_exact_mapping_across_approved_line_shapes(markdown: str, expected: str) -> None:
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == expected
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"except unknown-\nword here",
|
||||||
|
"except possi-\n\n\nbly here",
|
||||||
|
"except possi-\r\n\nbly here",
|
||||||
|
"except POSSI-\nbly here",
|
||||||
|
"except possi-\nblymore here",
|
||||||
|
"except xSOFA-\nbased here",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_unknown_or_unsafe_boundaries_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_parameters_and_results_are_independent_of_mapping_order() -> None:
|
||||||
|
first = PageBreakWordJoinComponent(MAPPINGS)
|
||||||
|
second = PageBreakWordJoinComponent(reversed(MAPPINGS))
|
||||||
|
markdown = "possibly becomes possi-\nbly"
|
||||||
|
|
||||||
|
first_result = Pipeline([first]).transform(markdown)
|
||||||
|
second_result = Pipeline([second]).transform(markdown)
|
||||||
|
|
||||||
|
assert first_result.components == second_result.components
|
||||||
|
assert first_result.output_markdown == second_result.output_markdown
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"mappings",
|
||||||
|
[
|
||||||
|
(("", "right", "word"),),
|
||||||
|
(("left", "right", "two words"),),
|
||||||
|
(("left", "right", "word"), ("left", "right", "other")),
|
||||||
|
(("left", "right"),),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_invalid_mappings_raise_contract_error(mappings: object) -> None:
|
||||||
|
with pytest.raises(ComponentContractError):
|
||||||
|
PageBreakWordJoinComponent(mappings) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([PageBreakWordJoinComponent(MAPPINGS)])
|
||||||
|
first = pipeline.transform("except possi-\n\nbly here")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import ReferenceSpacingComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([ReferenceSpacingComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalizes_missing_and_extra_blank_lines_in_references_only() -> None:
|
||||||
|
markdown = "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n2. Second\n\n\n3. Third"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n\n2. Second\n\n3. Third"
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("heading", ["# References", "## REFERENCES", "### references", "#### ReFeReNcEs"])
|
||||||
|
def test_accepts_exact_references_heading_with_ascii_case_folding(heading: str) -> None:
|
||||||
|
markdown = f"{heading}\n\n1. First\n2. Second"
|
||||||
|
assert transform(markdown).output_markdown == f"{heading}\n\n1. First\n\n2. Second"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(("## References", "", "1. First", "2. Second"))
|
||||||
|
expected = line_ending.join(("## References", "", "1. First", "", "2. Second"))
|
||||||
|
assert transform(markdown).output_markdown == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"## Reference\n\n1. First\n2. Second",
|
||||||
|
"## References\n\n1. First\n3. Third",
|
||||||
|
"## References\n\n2. Second\n3. Third",
|
||||||
|
"## References\n\n1. First\n### Subsection\n2. Second",
|
||||||
|
"## References\r\n\r\n1. First\r\n\n2. Second",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_ambiguous_or_mixed_sections_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_same_or_higher_heading_ends_section() -> None:
|
||||||
|
markdown = "## References\n\n1. First\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "## References\n\n1. First\n\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||||
|
|
||||||
|
|
||||||
|
def test_multiline_reference_uses_its_last_text_line_as_boundary() -> None:
|
||||||
|
markdown = "## References\n\n1. First line\ncontinuation\n2. Second"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "## References\n\n1. First line\ncontinuation\n\n2. Second"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([ReferenceSpacingComponent()])
|
||||||
|
first = pipeline.transform("## References\n\n1. First\n2. Second")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import RepeatedRunningHeaderComponent
|
||||||
|
|
||||||
|
HEADER = "## Repeated Paper Header"
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([RepeatedRunningHeaderComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_bridges_interrupted_sentence_and_deletes_other_occurrence(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(
|
||||||
|
(
|
||||||
|
"Sentence continues in",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"the next line.",
|
||||||
|
"",
|
||||||
|
"18. Reference",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"19. Reference",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == line_ending.join(
|
||||||
|
("Sentence continues in the next line.", "", "18. Reference", "", "19. Reference")
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
f"before\n\n{HEADER}\n\nafter",
|
||||||
|
f"Sentence ends.\n\n{HEADER}\n\nAfter\n\nAnother sentence.\n\n{HEADER}\n\nOther",
|
||||||
|
f"Sentence continues\n\n{HEADER}\n\nAfter\n\nText ends.\n\n{HEADER}\n\nOther",
|
||||||
|
f"Sentence continues\n\n{HEADER}\n\nafter\n{HEADER}\nnot blank",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_missing_or_unsafe_group_evidence_preserves_document(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_header_matching_is_exact_and_not_keyword_based() -> None:
|
||||||
|
markdown = "continues\n\n## Any Header\n\nfrom here\n\n18. Ref\n\n## Any Header\n\n19. Ref"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "continues from here\n\n18. Ref\n\n19. Ref"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([RepeatedRunningHeaderComponent()])
|
||||||
|
first = pipeline.transform(f"continues\n\n{HEADER}\n\nfrom here\n\ntext\n\n{HEADER}\n\nend")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import WordReviewCommentComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([WordReviewCommentComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_deletes_single_line_comment_and_one_following_blank() -> None:
|
||||||
|
markdown = "before\nCommented [AB1]: review this\n\nafter"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == "before\nafter"
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
assert result.changes[0].reason == "删除严格单行 Word 审阅批注及其后一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
def test_adjacent_comment_blocks_are_deleted_without_overlap() -> None:
|
||||||
|
markdown = "before\nCommented [A1]: first\n\nCommented [B2R1]: second\n\nafter"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown == "before\nafter"
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
assert result.changes[0].span.end == result.changes[1].span.start
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(("before", "Commented [A1]: note", "", "after"))
|
||||||
|
assert transform(markdown).output_markdown == line_ending.join(("before", "after"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"comment",
|
||||||
|
[
|
||||||
|
"prefix Commented [A1]: note",
|
||||||
|
" Commented [A1]: note",
|
||||||
|
"Commented []: note",
|
||||||
|
"Commented [A-1]: note",
|
||||||
|
"Commented [A1]:",
|
||||||
|
"Commented [A1]: ",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_similar_lines_are_preserved(comment: str) -> None:
|
||||||
|
markdown = f"before\n{comment}\n\nafter"
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_requires_a_following_blank_line_and_does_not_delete_extra_blanks() -> None:
|
||||||
|
without_blank = "Commented [A1]: note\nafter"
|
||||||
|
with_two_blanks = "before\nCommented [A1]: note\n\n\nafter"
|
||||||
|
|
||||||
|
assert transform(without_blank).output_markdown == without_blank
|
||||||
|
assert transform(with_two_blanks).output_markdown == "before\n\nafter"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([WordReviewCommentComponent()])
|
||||||
|
first = pipeline.transform("Commented [A1]: note\n\nafter")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.changes == ()
|
||||||
Reference in New Issue
Block a user