Merge pull request #3 from Bepr4/feature/clindb-cleaning-components
Feature/clindb cleaning components
This commit is contained in:
Bepr4
@@ -9,6 +9,8 @@ __pycache__/
|
|||||||
dist/
|
dist/
|
||||||
build/
|
build/
|
||||||
*.egg-info/
|
*.egg-info/
|
||||||
|
reviewer/node_modules/
|
||||||
|
reviewer/coverage/
|
||||||
|
|
||||||
# Real documents and private datasets
|
# Real documents and private datasets
|
||||||
data/
|
data/
|
||||||
@@ -22,6 +24,7 @@ uploads/
|
|||||||
artifacts/
|
artifacts/
|
||||||
outputs/
|
outputs/
|
||||||
experiments/
|
experiments/
|
||||||
|
research-wiki/scratch/
|
||||||
*.cleaned.md
|
*.cleaned.md
|
||||||
report.json
|
report.json
|
||||||
|
|
||||||
|
|||||||
@@ -6,10 +6,10 @@
|
|||||||
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
||||||
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
||||||
|
|
||||||
仓库当前已从纯文档治理进入第一版核心实现阶段：已经提供可安装的 Python 内存处理包和测试，用于验证
|
仓库当前已从纯文档治理进入第一版核心和 ClinDB 第一批组件实现阶段：已经提供可安装的 Python 内存处理包、
|
||||||
组件组合、精确修改和审计协议，并已有一个严格整行匹配的论文清洗组件。仓库另有一个只供本地评审的实验脚本，
|
8 个论文清洗组件、本地实验入口，以及同仓但与清洗运行解耦的只读评审前端。实验可以保存指定论文的成功输出、
|
||||||
可以保存指定论文的成功输出、审计和 diff。仓库仍不提供完整规则集、公共命令行工具、通用文件适配器或生产接口，
|
审计和 diff；评审前端可以同时查看清洗前后全文，并逐组件复放修改。仓库仍不提供面向任意数据集的完整规则集、
|
||||||
因此目前还不是拿来即可完成整篇文档清洗的成品工具。
|
公共命令行工具、通用文件适配器或生产接口，因此目前还不是拿来即可完成任意 Markdown 清洗的成品工具。
|
||||||
|
|
||||||
## 当前阶段
|
## 当前阶段
|
||||||
|
|
||||||
@@ -17,14 +17,20 @@ profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
|||||||
|
|
||||||
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
||||||
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
||||||
- 当前唯一正式组件是 `paper.arxiv_submission_stamp`，只删除严格整行匹配的 arXiv 提交边栏戳；
|
- 已实现 ClinDB 第一批 8 个正式组件，覆盖 Word 批注、手稿行号、arXiv 戳、重复页眉、批准映射断词、
|
||||||
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计和 unified diff，并保持输入不变；
|
HTML 表格实体与布局、参考文献空行；
|
||||||
- 该组件已对 5 份论文 Markdown 完成保存型实验，只修改 sim 和 springer 各一处，合法参考文献保持不变；
|
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计、unified diff 和评审定位文件，
|
||||||
- 当前基础检查为 Ruff、mypy 和 113 项 pytest 测试，实际命令见本文“当前可用检查”。
|
并保持输入不变；
|
||||||
|
- 已有独立的 `reviewer/` 本地只读前端，服务只消费一次已发布的产物和定位文件，并与报告层共用纯 Python 快照重放逻辑，
|
||||||
|
不导入组件、不调用流水线，也不重新清洗；
|
||||||
|
- 第一批流水线已对 5 份论文 Markdown 完成保存型实验，5/5 成功，共记录 155 条修改，第二次运行零修改；
|
||||||
|
- 当前基础检查为 Ruff、mypy、229 项 pytest 测试，以及前端 ESLint、TypeScript、9 项 Vitest 测试和生产构建，
|
||||||
|
实际命令见本文“当前可用检查”。
|
||||||
|
|
||||||
项目还没有完整清洗规则集、Markdown/HTML parser、profile 格式、通用文件输入接口、公共 CLI、通用批处理或生产接口。
|
项目还没有面向任意数据集的完整清洗规则集、Markdown/HTML 通用 parser、profile 格式、通用文件输入接口、公共 CLI、
|
||||||
当前实验脚本只固定运行已批准的 5 份论文和一个组件；它只能证明第一条严格规则已经闭环，不能据此认为论文、GovDoc、
|
通用批处理或生产接口。评审前端也不是公共 Web 服务：它只绑定本机回环地址，只读展示 Markdown 源文和审计，不提供
|
||||||
表格或图片已经具备完整清洗能力。
|
渲染预览、在线编辑或重新清洗。当前 first-batch 实验脚本只固定运行已批准的 5 份论文和 8 个组件；它只能证明当前
|
||||||
|
8 类确定规则已经闭环，不能据此认为论文中的缺失内容、乱码、复杂表格、图片或 GovDoc 已具备完整清洗能力。
|
||||||
|
|
||||||
## 服务对象与复用目标
|
## 服务对象与复用目标
|
||||||
|
|
||||||
@@ -46,8 +52,9 @@ GovDoc 目录、具体客户名称或某一转换器的固定输出路径。
|
|||||||
|
|
||||||
两组数据都不是可提交的自动测试 fixture。`data/` 已被 Git 忽略，清洗实验不得覆盖这些输入。
|
两组数据都不是可提交的自动测试 fixture。`data/` 已被 Git 忽略，清洗实验不得覆盖这些输入。
|
||||||
|
|
||||||
本地清洗实验产物位于 `artifacts/<YYYY-MM-DD>/runs/<run_id>/`。产物可能包含完整原文，同样受 Git 忽略，
|
本地清洗实验产物位于 `artifacts/<YYYY-MM-DD>/runs/<run_id>/`。产物可能包含完整原文；其中评审定位文件还包含
|
||||||
默认保留 30 个日历日，不得提交或复制到外部系统。当前只批准为上述 5 份论文副本保存产物，未批准保存 GovDoc 输出。
|
输入源的绝对路径，因此整个目录均受 Git 忽略，默认保留 30 个日历日，不得提交或复制到外部系统。当前只批准为上述
|
||||||
|
5 份论文副本保存产物，未批准保存 GovDoc 输出。
|
||||||
|
|
||||||
## 面向复用的设计原则
|
## 面向复用的设计原则
|
||||||
|
|
||||||
@@ -70,6 +77,7 @@ mdpolish/
|
|||||||
├── src/
|
├── src/
|
||||||
│ └── mdpolish/
|
│ └── mdpolish/
|
||||||
│ ├── __init__.py # 第一版核心公共导出
|
│ ├── __init__.py # 第一版核心公共导出
|
||||||
|
│ ├── _artifact_replay.py # 报告与评审器共用的纯快照重放
|
||||||
│ ├── component.py # 组件基类和元数据契约
|
│ ├── component.py # 组件基类和元数据契约
|
||||||
│ ├── edits.py # 文本编辑验证与原子应用
|
│ ├── edits.py # 文本编辑验证与原子应用
|
||||||
│ ├── experiment.py # 本地实验输入预检与批量编排
|
│ ├── experiment.py # 本地实验输入预检与批量编排
|
||||||
@@ -77,21 +85,50 @@ mdpolish/
|
|||||||
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
||||||
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
||||||
│ ├── artifact_store.py # 私有产物目录和原子发布
|
│ ├── artifact_store.py # 私有产物目录和原子发布
|
||||||
|
│ ├── _html_table.py # 严格 HTML 表格词法范围
|
||||||
|
│ ├── _text_ranges.py # 精确物理行与换行范围
|
||||||
│ ├── py.typed # 类型信息声明
|
│ ├── py.typed # 类型信息声明
|
||||||
│ └── components/
|
│ └── components/
|
||||||
│ ├── __init__.py
|
│ ├── __init__.py
|
||||||
│ └── arxiv_submission_stamp.py
|
│ ├── arxiv_submission_stamp.py
|
||||||
|
│ ├── html_table_double_escape.py
|
||||||
|
│ ├── html_table_layout.py
|
||||||
|
│ ├── manuscript_line_number.py
|
||||||
|
│ ├── page_break_word_join.py
|
||||||
|
│ ├── reference_spacing.py
|
||||||
|
│ ├── repeated_running_header.py
|
||||||
|
│ └── word_review_comment.py
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ └── run_clindb_arxiv_experiment.py # 固定 5 份论文的仓库内实验入口
|
│ ├── run_clindb_arxiv_experiment.py # 只含 arXiv 组件的历史实验入口
|
||||||
|
│ └── run_clindb_first_batch_experiment.py # ClinDB 第一批 8 组件实验入口
|
||||||
|
├── reviewer/ # 与组件和流水线解耦的本地只读评审器
|
||||||
|
│ ├── .nvmrc # 前端开发使用 Node.js 24
|
||||||
|
│ ├── package.json # React 依赖、检查和构建命令
|
||||||
|
│ ├── server/ # Python 产物适配、快照重放接口和本地 HTTP 服务
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── client/ # React 全文对比、组件时间线和审计界面
|
||||||
|
│ │ └── shared/ # 浏览器使用的内部 API 类型
|
||||||
|
│ └── tests/ # 前端响应、界面和源码安全测试
|
||||||
├── tests/
|
├── tests/
|
||||||
│ ├── test_arxiv_submission_stamp.py
|
│ ├── test_arxiv_submission_stamp.py
|
||||||
│ ├── test_artifact_store.py
|
│ ├── test_artifact_store.py
|
||||||
|
│ ├── test_clindb_first_batch_pipeline.py
|
||||||
│ ├── test_component.py
|
│ ├── test_component.py
|
||||||
│ ├── test_edits.py
|
│ ├── test_edits.py
|
||||||
│ ├── test_experiment.py
|
│ ├── test_experiment.py
|
||||||
|
│ ├── test_html_table_double_escape.py
|
||||||
|
│ ├── test_html_table_layout.py
|
||||||
|
│ ├── test_manuscript_line_number.py
|
||||||
│ ├── test_models.py
|
│ ├── test_models.py
|
||||||
|
│ ├── test_page_break_word_join.py
|
||||||
│ ├── test_pipeline.py
|
│ ├── test_pipeline.py
|
||||||
│ └── test_reporting.py
|
│ ├── test_reference_spacing.py
|
||||||
|
│ ├── test_repeated_running_header.py
|
||||||
|
│ ├── test_reporting.py
|
||||||
|
│ ├── test_word_review_comment.py
|
||||||
|
│ ├── test_artifact_replay.py
|
||||||
|
│ ├── test_reviewer_artifacts.py
|
||||||
|
│ └── test_reviewer_server.py
|
||||||
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
||||||
│ └── md/
|
│ └── md/
|
||||||
│ ├── dmp.md
|
│ ├── dmp.md
|
||||||
@@ -101,6 +138,9 @@ mdpolish/
|
|||||||
│ └── springer.md
|
│ └── springer.md
|
||||||
├── artifacts/ # 本地敏感实验产物；Git 忽略
|
├── artifacts/ # 本地敏感实验产物；Git 忽略
|
||||||
│ └── <YYYY-MM-DD>/runs/<run_id>/
|
│ └── <YYYY-MM-DD>/runs/<run_id>/
|
||||||
|
│ ├── manifest.json
|
||||||
|
│ ├── review-locator.json # 输入源定位和哈希；仅供本地评审
|
||||||
|
│ └── documents/
|
||||||
└── research-wiki/
|
└── research-wiki/
|
||||||
├── README.md # Wiki 分类与维护规则
|
├── README.md # Wiki 分类与维护规则
|
||||||
├── design/ # 批准前的选择；批准后冻结
|
├── design/ # 批准前的选择；批准后冻结
|
||||||
@@ -119,12 +159,13 @@ mdpolish/
|
|||||||
3. `research-wiki/README.md`，确认文档应放在哪里；
|
3. `research-wiki/README.md`，确认文档应放在哪里；
|
||||||
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
||||||
|
|
||||||
第一版核心的当前机制见 `research-wiki/explanation/first-executable-core.md`，首个真实组件见
|
第一版核心机制见 `research-wiki/explanation/first-executable-core.md`，ClinDB 第一批组件见
|
||||||
`research-wiki/explanation/arxiv-submission-stamp.md`，本地实验产物机制见
|
`research-wiki/explanation/clindb-first-batch-components.md`，本地实验产物机制见
|
||||||
`research-wiki/explanation/local-experiment-artifacts.md`，实际运行步骤见
|
`research-wiki/explanation/local-experiment-artifacts.md`，本地评审器机制见
|
||||||
`research-wiki/guides/run-local-clindb-arxiv-experiment.md`。下一项候选是 ClinDB 范围中的 HTML 实体双重转义，
|
`research-wiki/explanation/local-markdown-reviewer.md`。实验和评审器的实际操作分别见
|
||||||
但必须先用新 design 确定只在哪些 HTML 范围替换、如何避开字面示例以及实体替换边界。解析器、CLI、文件适配器、
|
`research-wiki/guides/run-local-clindb-first-batch-experiment.md` 和 `research-wiki/guides/review-local-cleaning-run.md`。
|
||||||
profile 格式和独立检查能力仍需分别设计；当前本地实验适配器不能被推导成这些公共接口已经获批。
|
解析器、CLI、文件适配器、profile 格式、图片资产打包和独立检查能力仍需分别设计；当前本地实验适配器和评审器不能
|
||||||
|
被推导成这些公共接口已经获批。
|
||||||
|
|
||||||
## 当前可用检查
|
## 当前可用检查
|
||||||
|
|
||||||
@@ -135,9 +176,16 @@ python -m venv .venv
|
|||||||
|
|
||||||
# 第一版核心的基础验收
|
# 第一版核心的基础验收
|
||||||
.venv/bin/ruff check .
|
.venv/bin/ruff check .
|
||||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
.venv/bin/pytest
|
.venv/bin/pytest
|
||||||
|
|
||||||
|
# 本地评审器要求 Node.js 24 LTS；安装依赖后依次执行静态检查、测试和生产构建
|
||||||
|
cd reviewer
|
||||||
|
nvm use
|
||||||
|
npm ci
|
||||||
|
npm run check
|
||||||
|
cd ..
|
||||||
|
|
||||||
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
||||||
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||||
|
|
||||||
@@ -148,6 +196,8 @@ find research-wiki -maxdepth 2 -type f | sort
|
|||||||
git status --short
|
git status --short
|
||||||
```
|
```
|
||||||
|
|
||||||
上述安装和三项基础验收已于 2026-08-22 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 19 个源码、
|
上述 Python 验收已于 2026-08-24 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 43 个源码、测试和实验脚本文件
|
||||||
测试和实验脚本文件无问题，pytest 共 113 项测试通过。`requires-python` 仍以 `pyproject.toml` 声明的 Python 3.11 及以上为准；
|
无问题，pytest 共 229 项测试通过。评审器验收也在当前用户 nvm 的 Node.js 24.19.0 环境实际运行：ESLint 和 TypeScript
|
||||||
本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
通过，Vitest 共 9 项测试通过，生产构建成功；并以一批真实的 5 文档、8 组件、155 条修改产物验证了接口读取和逐阶段哈希。
|
||||||
|
当前环境没有可用的图形浏览器，因此页面视觉布局尚未进行真实浏览器人工验收。`requires-python` 仍以
|
||||||
|
`pyproject.toml` 声明的 Python 3.11 及以上为准；本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
||||||
|
|||||||
@@ -0,0 +1,531 @@
|
|||||||
|
# 0006：ClinDB 第一批清洗组件与组合顺序
|
||||||
|
|
||||||
|
## 状态
|
||||||
|
|
||||||
|
已批准并冻结（2026-08-22）。
|
||||||
|
|
||||||
|
本设计已经用户明确批准，授权按第 13 节实施。后续语义变化必须新增 design，不回写本文。
|
||||||
|
|
||||||
|
本文沿用 `0003` 的内存核心与组件契约、`0004` 已批准的 arXiv 组件，以及 `0005` 的本地实验产物契约。
|
||||||
|
它不替代这三份设计，也不修改核心数据模型、执行器、流水线或产物格式。
|
||||||
|
|
||||||
|
## 1. 为什么现在需要这一设计
|
||||||
|
|
||||||
|
当前 5 份 ClinDB 论文已经能够通过本地实验层运行，但正式组件只有
|
||||||
|
`paper.arxiv_submission_stamp`。现有 reference 列出了第一批 8 类自动清洗目标，却还没有逐项确定：
|
||||||
|
|
||||||
|
- 每个问题由哪个组件负责；
|
||||||
|
- 哪些前置条件同时满足时才允许修改；
|
||||||
|
- 一处修改究竟替换哪些字符；
|
||||||
|
- 组件之间是否存在先后依赖；
|
||||||
|
- 当前 5 份输入的真实命中数是多少；
|
||||||
|
- 哪些看似合理的通用化会把正文一起改掉。
|
||||||
|
|
||||||
|
只按问题名称直接写正则会产生实际误改。例如，JAMA 文档中 Abstract 之前的 59 个行首数字是作者单位编号，不能当作
|
||||||
|
手稿行号删除；Springer 正文里还有编号方法列表，不能由参考文献空行规则全局处理；HTML 表格中的双重实体也不能直接
|
||||||
|
替换为原始 `<`，否则可能改变 HTML 解析边界。
|
||||||
|
|
||||||
|
因此本轮先冻结第一批组件的严格语义和组合顺序，再实施代码。
|
||||||
|
|
||||||
|
## 2. 只读调研结论与待修订事实
|
||||||
|
|
||||||
|
本轮重新按物理行和原始字符检查了 `data/md/` 中的 5 份本地论文副本。下表是本设计拟采用的验收事实；在本文批准前，
|
||||||
|
它们只是待评审结论，不表示组件已经实现。
|
||||||
|
|
||||||
|
| 范围 | 只读结果 | 对现有 reference 的影响 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Word 审阅批注 | JAMA 有 2 条，批注文字都与 `Commented [...]` 位于同一物理行 | 第一版不猜测多行批注正文，只删除严格命中的单行批注及其后一个空行 |
|
||||||
|
| 手稿行号 | JAMA 共出现 134 个候选数字前缀；Abstract 之前 59 个是作者单位编号，Abstract 之后 75 个才是手稿行号，其中 6 个位于标题内 | `128 + 6` 的旧计数会误伤作者单位，应改为总计 75 |
|
||||||
|
| arXiv 提交戳 | sim、springer 各 1 条 | 沿用 `0004`，不改变组件语义 |
|
||||||
|
| 重复跑动页眉 | dmp 同一标题出现 2 次，其中 1 次切断正文，另 1 次落在参考文献中 | 页眉必须先于参考文献分隔执行 |
|
||||||
|
| 跨页断词 | 当前可确认 6 处，包括 3 处去连字符、2 处保留词内连字符和 1 处带错误句点的精确修复 | 不能只用“行尾连字符 + 小写字母”，也没有证据支持引入英文词表 |
|
||||||
|
| HTML 双重实体 | dmp 23 处、ejhf 8 处，共 31 处，全部在严格 HTML 表格单元格文本中 | “解除一层”应得到 `<`、`>`、`&`，而不是直接生成 HTML 源码中的 `<`、`>`、`&` |
|
||||||
|
| 单行 HTML 表格 | 共 9 张；每张都至少有一个非 `1` 的 `colspan`，没有可验证的简单无合并表格样本 | 第一批只保留 HTML 并按行展开，不批准未经真实样本验证的 GFM 转换 |
|
||||||
|
| 参考文献空行 | dmp 需要补 13 处，springer 需要补 15 处，共 28 处 | 旧的条目范围计数不准确；规则必须限定在 References 章节内 |
|
||||||
|
|
||||||
|
dmp 的一处输入实际是前一行以 `threshold.` 结束、下一非空行以 `olds` 开始。它并不是普通的
|
||||||
|
`thresh-` + `olds`，而是上游结果同时多了句点并重复了 `old`。将它恢复为 `thresholds` 属于一条基于当前上下文批准的
|
||||||
|
精确映射，不能推广成通用断词算法。
|
||||||
|
|
||||||
|
上述差异在设计批准后才同步修订
|
||||||
|
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。不能一边保留旧计数，
|
||||||
|
一边让测试按新计数通过。
|
||||||
|
|
||||||
|
## 3. 目标与非目标
|
||||||
|
|
||||||
|
### 3.1 目标
|
||||||
|
|
||||||
|
- 为第一批 8 类问题提供 7 个新组件，并复用现有 arXiv 组件；
|
||||||
|
- 每个组件只提出唯一、可立即应用的精确修改；
|
||||||
|
- 将论文场景的组合留在实验脚本，核心和组件不依赖 DOI、文件名或 `data/md/`；
|
||||||
|
- 明确组件顺序和少量真实依赖，避免靠多轮执行修正顺序错误；
|
||||||
|
- 保持标准库运行时、确定性、原子修改、完整审计和幂等性；
|
||||||
|
- 用合成测试覆盖正向、反向、异常边界和整条组合的稳定性；
|
||||||
|
- 设计批准并实现后，使用 `0005` 的现有产物结构保存 5 份论文的清洗结果、JSON 审计和 diff。
|
||||||
|
|
||||||
|
### 3.2 非目标
|
||||||
|
|
||||||
|
- 不恢复截断、缺表、缺句、乱码、OCR 表头或其他缺失内容；
|
||||||
|
- 不判断修订残留中哪个词正确，不改写写作、拼写或医学语义；
|
||||||
|
- 不增加 `detect_only`、建议、Inspector、人工审核或问题报告；
|
||||||
|
- 不引入 Markdown parser、HTML parser 第三方依赖、AST 或全文重新渲染；
|
||||||
|
- 不建立 Profile 文件格式、组件自动发现、公共 CLI 或稳定公共批处理接口；
|
||||||
|
- 不把无合并 HTML 表格转成 GFM；当前 5 份输入没有这种真实样本，表头推断也没有唯一答案；
|
||||||
|
- 不做一般段落重排、全局空行格式化或全局 HTML 实体解码；
|
||||||
|
- 不读取、复制、改写或打包图片；清洗产物中的图片相对路径仍可能因目录变化而无法直接显示；
|
||||||
|
- 不修改或覆盖 `data/md/` 输入，不对仓库外 GovDoc 数据保存实验产物；
|
||||||
|
- 不宣称输出已与 PDF 原文完全一致，也不宣称 ClinDB 中所有内容问题已经解决。
|
||||||
|
|
||||||
|
本文中的“ClinDB 第一批全清洗”只表示：本设计批准的 8 个自动组件全部运行并稳定，5 份输入中已确认的对应噪声全部
|
||||||
|
按契约处理。它不表示文档没有本设计明确排除的问题。
|
||||||
|
|
||||||
|
## 4. 采用的组件划分
|
||||||
|
|
||||||
|
第一批流水线固定包含以下组件：
|
||||||
|
|
||||||
|
| 顺序 | 组件标识 | 版本 | 职责 |
|
||||||
|
| ---: | --- | --- | --- |
|
||||||
|
| 1 | `paper.word_review_comment` | `1.0.0` | 删除严格单行 Word 审阅批注 |
|
||||||
|
| 2 | `paper.manuscript_line_number` | `1.0.0` | 删除通过整段证据确认的手稿行号前缀 |
|
||||||
|
| 3 | `paper.arxiv_submission_stamp` | `1.0.0` | 删除严格整行匹配的 arXiv 提交戳 |
|
||||||
|
| 4 | `paper.repeated_running_header` | `1.0.0` | 删除重复跑动页眉，并在证据充分处接回被切断段落 |
|
||||||
|
| 5 | `paper.page_break_word_join` | `1.0.0` | 按显式批准映射修复跨页断词 |
|
||||||
|
| 6 | `markdown.html_table_double_escape` | `1.0.0` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||||
|
| 7 | `markdown.html_table_layout` | `1.0.0` | 将严格单行 HTML 表格展开为一行一个 `<tr>` |
|
||||||
|
| 8 | `paper.reference_spacing` | `1.0.0` | 只在 References 章节内统一相邻编号条目的空行 |
|
||||||
|
|
||||||
|
不采用一个 `clindb.clean_all` 大组件。拆分后每项修改都有独立身份、版本、适用边界和命中计数；改变表格规则不会迫使
|
||||||
|
行号规则一起升级，其他项目也可以只组合自己需要的组件。
|
||||||
|
|
||||||
|
组件不互相导入、不直接调用另一个组件，也不知道自己位于 ClinDB 流水线。顺序由实验入口显式组装并由
|
||||||
|
`manifest.json` 记录。
|
||||||
|
|
||||||
|
## 5. 共同扫描基础
|
||||||
|
|
||||||
|
多个组件需要精确物理行范围，两个表格组件需要完全相同的保守 HTML 范围。批准后允许新增两个不公开的内部辅助模块：
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/mdpolish/
|
||||||
|
├── _text_ranges.py
|
||||||
|
└── _html_table.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.1 `_text_ranges.py`
|
||||||
|
|
||||||
|
它只负责返回原字符串中的物理行内容范围、行尾范围和空行关系，支持 `\n`、`\r\n`、单独 `\r` 和无末尾换行。
|
||||||
|
它不解释 Markdown 结构，也不规范化换行。
|
||||||
|
|
||||||
|
现有 arXiv 组件中的私有物理行扫描可以机械迁移到该模块。迁移不得改变
|
||||||
|
`paper.arxiv_submission_stamp` 的标识、版本、参数、识别范围或测试结果。
|
||||||
|
|
||||||
|
### 5.2 `_html_table.py`
|
||||||
|
|
||||||
|
它是针对当前转换器输出子集的严格词法扫描器，不是通用 HTML parser。只有同时满足以下条件的完整片段才返回结构范围：
|
||||||
|
|
||||||
|
- 具有完整、正确嵌套的小写 `<table>`、`<tr>`、`<td>` / `<th>` 起止标签；
|
||||||
|
- `<tr>` 是 `<table>` 的直接子元素，单元格是 `<tr>` 的直接子元素；
|
||||||
|
- 标签属性可以存在并原样保留，但引号和标签边界必须完整；
|
||||||
|
- 结构标签之间只能有空白，单元格文本中不能出现其他原始 HTML 标签；
|
||||||
|
- 不含嵌套表格、注释、声明、`script`、`style`、`pre` 或无法闭合的标签。
|
||||||
|
|
||||||
|
不满足条件的相似片段直接忽略，组件不尝试修复它，也不降级为全局正则。扫描器返回原字符串下标，不构造 DOM，
|
||||||
|
不解码实体，不重新输出 HTML。
|
||||||
|
|
||||||
|
第一批组件继续不识别 Markdown 围栏、引用块或其他块级上下文。严格目标文本即使出现在这些结构中仍可能命中；
|
||||||
|
当前 5 份证据没有这种反例，本设计不借共享扫描器重新加入围栏保护。
|
||||||
|
|
||||||
|
## 6. 各组件的精确契约
|
||||||
|
|
||||||
|
### 6.1 `paper.word_review_comment`
|
||||||
|
|
||||||
|
目标物理行必须完整满足：
|
||||||
|
|
||||||
|
```regex
|
||||||
|
^Commented \[[A-Za-z0-9]+\]: .*\S$
|
||||||
|
```
|
||||||
|
|
||||||
|
并且其后至少紧跟一个空白内容为空的物理行。每个命中产生一个删除编辑，范围包含：
|
||||||
|
|
||||||
|
1. 批注行的全部文字和自身行尾；
|
||||||
|
2. 紧随其后的第一个空行及其行尾。
|
||||||
|
|
||||||
|
组件不删除更多空行，不把后续普通行猜成批注正文，也不处理 `Commented` 出现在句中、行首带空格、缺少批注 ID、
|
||||||
|
缺少冒号或没有正文的相似文本。末尾没有后续空行时保持原样。
|
||||||
|
|
||||||
|
固定修改理由为“删除严格单行 Word 审阅批注及其后一个空行”。当前 JAMA 两条批注各产生一条 `Change`。
|
||||||
|
|
||||||
|
### 6.2 `paper.manuscript_line_number`
|
||||||
|
|
||||||
|
该组件不全局删除行首数字。它先按以下步骤确认一个完整手稿行号序列：
|
||||||
|
|
||||||
|
1. 文档中必须恰好有一个严格物理行 `## Abstract`；
|
||||||
|
2. 只扫描该行之后的内容，Abstract 之前的所有数字前缀一律排除；
|
||||||
|
3. 候选前缀只有两种：普通行的 `^[1-9][0-9]{0,2} `，或 ATX 标题中的
|
||||||
|
`^#{1,6} [1-9][0-9]{0,2} `；前缀后必须还有非空正文；
|
||||||
|
4. 按出现顺序取得的所有候选数字必须严格递增，允许跳号，不允许相等或回退；
|
||||||
|
5. 整段至少有 20 个候选，并至少有 2 个标题候选。
|
||||||
|
|
||||||
|
任一条件不满足时，整个组件返回零候选，不做部分删除，也不从某个看似合理的位置继续猜测。当前阈值用于证明这是
|
||||||
|
整篇带行号的手稿，而不是恰好出现几个数字开头的段落；未来出现短手稿时必须用新证据重新评审，不能暗中降低阈值。
|
||||||
|
|
||||||
|
确认序列后，每个前缀各产生一条精确修改：普通行只删除数字和其后的一个 ASCII 空格；标题只删除标题标记之后的
|
||||||
|
数字和一个空格，保留原有 `#` 级别和标题空格。正文中的数字、列表 `1.` / `1)`、四位年份、Abstract 之前的作者单位
|
||||||
|
编号均不修改。
|
||||||
|
|
||||||
|
固定修改理由为“删除已确认手稿序列中的行号前缀”。当前 JAMA 应产生 75 条 `Change`，其中普通行 69 条、标题 6 条。
|
||||||
|
|
||||||
|
### 6.3 `paper.arxiv_submission_stamp`
|
||||||
|
|
||||||
|
完全复用 `0004` 已批准的 `ArxivSubmissionStampComponent`，不修改代码语义和版本。第一批组合只把它放入新的流水线，
|
||||||
|
当前 sim 和 springer 各产生 1 条 `Change`。
|
||||||
|
|
||||||
|
### 6.4 `paper.repeated_running_header`
|
||||||
|
|
||||||
|
候选页眉首先必须是无首尾空格的完整 ATX 标题行：
|
||||||
|
|
||||||
|
```regex
|
||||||
|
^#{1,6} \S(?:.*\S)?$
|
||||||
|
```
|
||||||
|
|
||||||
|
相同的完整标题行必须原样出现至少 2 次。一个重复组只有在所有出现位置都能安全删除，并且至少有一个位置满足“正文被
|
||||||
|
切断”证据时，才整体成立。
|
||||||
|
|
||||||
|
“正文被切断”要求该标题前后各恰好有一个空行，前一非空物理行去掉行尾空白后不以中英文句号、问号、叹号、冒号或
|
||||||
|
分号结束，后一非空物理行以 ASCII 小写字母开头。该位置从前一正文行末到后一正文行首的整个间隔替换为一个 ASCII
|
||||||
|
空格，从而删除空行和页眉并接回同一句。
|
||||||
|
|
||||||
|
同组的其他位置只在标题后至少有一个空行时删除标题行及其后第一个空行，保留标题之前已有的分隔。若某个出现位置不满足
|
||||||
|
上述任一安全形态，则整组不修改。组件不使用标题关键词，不删除只出现一次的标题，也不把一般重复章节标题视为页眉。
|
||||||
|
|
||||||
|
固定修改理由为“删除经重复和断句证据确认的跑动页眉”。当前 dmp 的同一标题产生 2 条 `Change`。
|
||||||
|
|
||||||
|
### 6.5 `paper.page_break_word_join`
|
||||||
|
|
||||||
|
该组件不使用词典、拼写检查、概率模型或“行尾有连字符就合并”的启发式规则。调用方必须显式传入有序映射，每项包含：
|
||||||
|
|
||||||
|
```text
|
||||||
|
left_fragment
|
||||||
|
right_fragment
|
||||||
|
replacement
|
||||||
|
```
|
||||||
|
|
||||||
|
组件初始化时把映射规范化为确定顺序，并拒绝空字段、重复的左右片段组合或会产生无效修改的映射。实际参数完整写入
|
||||||
|
`ComponentInfo.parameters`，因此相同组件版本使用了哪些项目映射可以从运行清单复核。
|
||||||
|
|
||||||
|
一条映射只有在以下条件全部满足时才命中：
|
||||||
|
|
||||||
|
- `left_fragment` 是前一非空物理行的精确末尾；
|
||||||
|
- 右片段位于紧邻的下一物理行，或中间恰好隔一个空行；存在空行时，相关行尾使用同一种形式；
|
||||||
|
- `right_fragment` 是下一非空物理行的精确开头；
|
||||||
|
- 左片段前和右片段后都满足 ASCII 字母词边界，不能只是更长单词的一部分；
|
||||||
|
- 同一位置没有被另一条映射命中。
|
||||||
|
|
||||||
|
编辑范围从左片段起点到右片段终点，包含两个片段和中间一个或两个行尾，整体替换为显式 `replacement`。前后其余正文
|
||||||
|
一字不改。两个及以上空行通常表示段落边界，即使片段文字相同也不处理。
|
||||||
|
|
||||||
|
ClinDB 第一批参数固定为：
|
||||||
|
|
||||||
|
| `left_fragment` | `right_fragment` | `replacement` | 说明 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `medi-` | `cal` | `medical` | 去除分页连字符 |
|
||||||
|
| `possi-` | `bly` | `possibly` | 去除分页连字符 |
|
||||||
|
| `cre-` | `ated` | `created` | 去除分页连字符 |
|
||||||
|
| `SOFA-` | `based` | `SOFA-based` | 保留复合词连字符，只删除分页分隔 |
|
||||||
|
| `life-` | `threatening` | `life-threatening` | 保留复合词连字符，只删除分页分隔 |
|
||||||
|
| `threshold.` | `olds` | `thresholds` | 当前 dmp 的精确证据修复，不推广 |
|
||||||
|
|
||||||
|
每处固定理由为“按已批准映射修复跨页断词：`<left_fragment>` + `<right_fragment>` → `<replacement>`”；当前应产生
|
||||||
|
6 条 `Change`。增加或改变映射会改变实验参数和预期结果，必须先更新项目设计或已批准的项目配置，不能从外部词典自动扩张。
|
||||||
|
|
||||||
|
### 6.6 `markdown.html_table_double_escape`
|
||||||
|
|
||||||
|
该组件只处理 `_html_table.py` 接受的完整表格，并只扫描 `<td>` / `<th>` 的文本范围。第一版映射固定为：
|
||||||
|
|
||||||
|
| 修改前源码 | 修改后源码 | 浏览器最终显示意图 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `&lt;` | `<` | `<` |
|
||||||
|
| `&gt;` | `>` | `>` |
|
||||||
|
| `&amp;` | `&` | `&` |
|
||||||
|
|
||||||
|
这是对 HTML 源码解除一层实体转义，不是把最终显示字符直接写进 HTML。每个精确实体 token 产生一个编辑；已经是单层的
|
||||||
|
`<`、`>`、`&` 不再命中，因此再次运行稳定。
|
||||||
|
|
||||||
|
标签名、属性值、表格外文本、普通 Markdown、URL 和不完整实体均不修改。组件不调用 `html.unescape()`，因为它会扩大到
|
||||||
|
没有逐项批准的命名实体和数字实体，也无法保留 HTML 标签边界。
|
||||||
|
|
||||||
|
固定修改理由为“在严格 HTML 表格单元格文本中解除一层实体转义”。当前应产生 31 条 `Change`。
|
||||||
|
|
||||||
|
### 6.7 `markdown.html_table_layout`
|
||||||
|
|
||||||
|
该组件只处理 `_html_table.py` 接受、并且整个 `<table>...</table>` 片段不含任何 `\r` 或 `\n` 的单行表格。
|
||||||
|
它不计算行列数，不解释 `rowspan` / `colspan`，不推断表头，也不改变任何单元格内容、属性或标签书写。
|
||||||
|
|
||||||
|
输出形态固定为：
|
||||||
|
|
||||||
|
```html
|
||||||
|
<table>
|
||||||
|
<tr>...</tr>
|
||||||
|
<tr>...</tr>
|
||||||
|
</table>
|
||||||
|
```
|
||||||
|
|
||||||
|
实际 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和 `</table>` 都从输入原样复用；组件只规范化这些外层片段之间的
|
||||||
|
布局空白。文档只有一种行尾时沿用该行尾；文档没有任何行尾时使用 `\n`；文档混用多种行尾时保持表格原样。
|
||||||
|
|
||||||
|
每张表产生一个覆盖完整表格片段的替换编辑。已经是多行的表格不命中，因而第二次运行不再改动。当前 9 张表全部含
|
||||||
|
`colspan`，均保留 HTML，只展开为一行一个 `<tr>`，应产生 9 条 `Change`。
|
||||||
|
|
||||||
|
固定修改理由为“展开严格单行 HTML 表格的行布局”。未来是否把简单表格转为 GFM，必须等真实无合并样本出现后单独决定
|
||||||
|
表头、转义、换行和 HTML/GFM 等价性。
|
||||||
|
|
||||||
|
### 6.8 `paper.reference_spacing`
|
||||||
|
|
||||||
|
该组件先定位严格 ATX 标题，其标题文字按 ASCII 大小写折叠后必须恰好是 `references`。章节范围从该标题之后开始，
|
||||||
|
到下一个级别相同或更高的 ATX 标题之前结束；没有后续标题时到文档末尾。
|
||||||
|
|
||||||
|
章节中的编号条目起始行必须完整满足 `^([1-9][0-9]*)\. \S`。只有同时满足以下条件才处理整个章节：
|
||||||
|
|
||||||
|
- 至少有 2 个编号条目；
|
||||||
|
- 第一个编号为 1；
|
||||||
|
- 后续编号严格逐个加 1，没有缺号、重复或回退；
|
||||||
|
- 两个相邻条目之间的最后一条正文行与下一条目起始行之间只有同一种行尾和零个或多个空行。
|
||||||
|
|
||||||
|
组件把每个相邻条目边界规范为恰好一个空行，即两个相同的行尾。已经恰好一个空行的边界不产生修改；没有空行时插入
|
||||||
|
一个行尾，多余空行时收敛为一个。章节外编号列表、非 References 标题、编号不连续的章节和混合行尾边界保持原样。
|
||||||
|
|
||||||
|
固定修改理由为“统一 References 章节中相邻编号条目之间的一个空行”。该组件必须在跑动页眉之后执行，否则 dmp 位于
|
||||||
|
第 18、19 条之间的页眉会破坏章节连续性。当前 dmp 应产生 13 条、springer 应产生 15 条，共 28 条 `Change`。
|
||||||
|
|
||||||
|
## 7. 顺序为什么固定
|
||||||
|
|
||||||
|
第一批只执行一轮，不能依赖“第二轮自然修好”。固定顺序的直接理由是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Word 批注 ─┐
|
||||||
|
手稿行号 ──┼── 先去除论文编辑层噪声
|
||||||
|
arXiv 戳 ──┘
|
||||||
|
|
||||||
|
重复页眉 ─────► 参考文献分隔
|
||||||
|
|
||||||
|
跨页断词 ─────► 独立正文精确修复
|
||||||
|
|
||||||
|
HTML 双重实体 ─► HTML 表格布局
|
||||||
|
```
|
||||||
|
|
||||||
|
- 跑动页眉先删除，dmp 的参考文献编号才能形成完整连续序列；
|
||||||
|
- 实体先改、布局后改，使每个实体 `Change` 保留较小的局部范围，随后表格整体布局基于最新快照；
|
||||||
|
- Word 批注先删除，行号组件看到的是不含审阅插入物的手稿，但行号判定本身仍不得依赖批注一定存在；
|
||||||
|
- 其他顺序目前没有内容依赖，仍固定下来以保证审计、哈希和 diff 可复现。
|
||||||
|
|
||||||
|
脚本和组合测试必须断言第 4 节的完整顺序，不能按文件名排序组件，也不能针对不同文档临时增删组件。
|
||||||
|
|
||||||
|
## 8. 代码与依赖边界
|
||||||
|
|
||||||
|
批准后允许新增：
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/mdpolish/
|
||||||
|
├── _html_table.py
|
||||||
|
├── _text_ranges.py
|
||||||
|
└── components/
|
||||||
|
├── html_table_double_escape.py
|
||||||
|
├── html_table_layout.py
|
||||||
|
├── manuscript_line_number.py
|
||||||
|
├── page_break_word_join.py
|
||||||
|
├── reference_spacing.py
|
||||||
|
├── repeated_running_header.py
|
||||||
|
└── word_review_comment.py
|
||||||
|
scripts/
|
||||||
|
└── run_clindb_first_batch_experiment.py
|
||||||
|
tests/
|
||||||
|
├── test_clindb_first_batch_pipeline.py
|
||||||
|
├── test_html_table_double_escape.py
|
||||||
|
├── test_html_table_layout.py
|
||||||
|
├── test_manuscript_line_number.py
|
||||||
|
├── test_page_break_word_join.py
|
||||||
|
├── test_reference_spacing.py
|
||||||
|
├── test_repeated_running_header.py
|
||||||
|
└── test_word_review_comment.py
|
||||||
|
```
|
||||||
|
|
||||||
|
允许同步修改：
|
||||||
|
|
||||||
|
- `src/mdpolish/components/__init__.py`，导出 7 个新组件；
|
||||||
|
- `src/mdpolish/components/arxiv_submission_stamp.py`，只把物理行扫描机械迁移到 `_text_ranges.py`；
|
||||||
|
- 现有 arXiv 测试，验证迁移没有改变行为；
|
||||||
|
- README、对应 explanation 和经实际验证的 guide，使当前阶段、目录和检查命令与实现一致；
|
||||||
|
- ClinDB reference，修订第 2 节列出的计数、实体语义、表格边界和 `0005` 已实现的输出事实。
|
||||||
|
|
||||||
|
依赖方向固定为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
components ──► models / component
|
||||||
|
│
|
||||||
|
├───────► _text_ranges
|
||||||
|
└───────► _html_table
|
||||||
|
|
||||||
|
experiment script ──► components + Pipeline + experiment layer
|
||||||
|
```
|
||||||
|
|
||||||
|
内部辅助模块不导入具体组件、流水线、文件适配层或 reporter。7 个新组件不从顶层 `mdpolish.__init__` 导出，不承诺
|
||||||
|
稳定公共 API。现有核心签名、JSON `schema_version` 和产物目录结构均不改变。
|
||||||
|
|
||||||
|
`run_clindb_arxiv_experiment.py` 保留为首个组件实验的历史入口，不改写成新语义。新的 first-batch 脚本显式建立第 4 节
|
||||||
|
的 8 组件流水线，仍只接受 `--run-id`，并复用 `0005` 已批准的 5 份文档清单和
|
||||||
|
`artifacts/<run_date>/runs/<run_id>/` 保存逻辑。它不是公共 CLI，也不是 Profile 实现。
|
||||||
|
|
||||||
|
第一版继续只使用 Python 标准库，不修改最低 Python 版本，不新增运行依赖。
|
||||||
|
|
||||||
|
## 9. 合成测试要求
|
||||||
|
|
||||||
|
每个组件至少覆盖空输入、无命中、单命中、多命中、三种行尾、无末尾行尾、参数或结构反例、修改理由、确定顺序和
|
||||||
|
再次运行零修改。额外必须覆盖：
|
||||||
|
|
||||||
|
### 9.1 Word 批注
|
||||||
|
|
||||||
|
- 两条由空行隔开的单行批注都删除，并正确接回周围普通行；
|
||||||
|
- 行内 `Commented`、不合法 ID、缺正文、没有后续空行和假想多行正文保持原样；
|
||||||
|
- 不多删第二个及后续空行。
|
||||||
|
|
||||||
|
### 9.2 手稿行号
|
||||||
|
|
||||||
|
- Abstract 前的连续作者单位编号全部保留；
|
||||||
|
- 满足阈值和标题证据的递增序列同时清理普通行与标题；
|
||||||
|
- 候选少于 20、标题候选少于 2、重复、回退、多个 Abstract 或没有 Abstract 时整体不修改；
|
||||||
|
- `1.`、`1)`、四位年份、正文中间的 `35 pediatric experts`、`10 sites`、`4 continents` 保留。
|
||||||
|
|
||||||
|
### 9.3 重复页眉
|
||||||
|
|
||||||
|
- 一个桥接位置替换为单个空格，另一个独立位置删除后只保留一个原有段落分隔；
|
||||||
|
- 只出现一次、重复但没有断句证据、下一行大写或某次出现无法安全删除时保持原样；
|
||||||
|
- 页眉文字不硬编码，候选顺序按原文位置稳定。
|
||||||
|
|
||||||
|
### 9.4 跨页断词
|
||||||
|
|
||||||
|
- 6 条批准映射分别验证，包含删除连字符、保留连字符和 `threshold.` / `olds` 精确修复；
|
||||||
|
- 未配置词、不是一整个词边界、两个及以上空行、混合行尾、大小写不同和相似长词保持原样；
|
||||||
|
- 相邻物理行和中间恰好一个空行两种已批准边界都能正确合并；
|
||||||
|
- 映射输入顺序不同仍得到相同参数和结果，重复或冲突映射明确报契约错误。
|
||||||
|
|
||||||
|
### 9.5 HTML 实体与布局
|
||||||
|
|
||||||
|
- 实体只解除一层，第二次不再改变；属性、表外文本、单层实体和未批准实体保留；
|
||||||
|
- 9 张表所代表的 `colspan` 形态只改布局，不改属性、单元格文本和标签顺序；
|
||||||
|
- 已经多行的表格不改；混合行尾文档不改布局；
|
||||||
|
- 未闭合、嵌套、含额外标签或结构不合法的表格整体忽略；
|
||||||
|
- 两个 HTML 组件组合后再次运行稳定。
|
||||||
|
|
||||||
|
### 9.6 参考文献与完整组合
|
||||||
|
|
||||||
|
- References 章节内缺失或过多空行都收敛为一个；章节外编号方法列表保留；
|
||||||
|
- 大小写不同的准确 References 标题可识别，近似标题、缺号和混合行尾保持原样；
|
||||||
|
- 合成 dmp 形态证明页眉先删除后，18、19 条之间和后续连续条目能正确处理；
|
||||||
|
- 完整 8 组件流水线第一次成功，最终复查没有残留候选；对成功输出再次运行为 `success` 且零 `Change`；
|
||||||
|
- 每条 `Change` 的组件、版本、位置、理由、`before`、`after` 和前后哈希可以按 `0003` / `0005` 重放。
|
||||||
|
|
||||||
|
测试 fixture 只能使用小型虚构文本，不能复制真实论文段落或完整表格到 Git。
|
||||||
|
|
||||||
|
## 10. 本地 5 份论文验收
|
||||||
|
|
||||||
|
所有合成测试通过后，才允许对 `data/md/` 的 5 份本地副本执行新的保存型实验。预期计数如下：
|
||||||
|
|
||||||
|
| 组件 | 预期 `Change` 数 |
|
||||||
|
| --- | ---: |
|
||||||
|
| `paper.word_review_comment` | 2 |
|
||||||
|
| `paper.manuscript_line_number` | 75 |
|
||||||
|
| `paper.arxiv_submission_stamp` | 2 |
|
||||||
|
| `paper.repeated_running_header` | 2 |
|
||||||
|
| `paper.page_break_word_join` | 6 |
|
||||||
|
| `markdown.html_table_double_escape` | 31 |
|
||||||
|
| `markdown.html_table_layout` | 9 |
|
||||||
|
| `paper.reference_spacing` | 28 |
|
||||||
|
| **合计** | **155** |
|
||||||
|
|
||||||
|
验收必须确认：
|
||||||
|
|
||||||
|
1. `manifest.json` 中 5 份文档全部为 `success`，组件身份、版本、参数和顺序与本文一致；
|
||||||
|
2. 合计恰好 155 条 `Change`，各组件计数与上表一致；
|
||||||
|
3. 每个 `cleaned.md` 的哈希与 `result.json` 一致，最终复查没有残留候选；
|
||||||
|
4. 对 5 份成功输出再运行同一流水线，全部 `success` 且合计零修改；
|
||||||
|
5. JAMA 的作者单位编号保留，只清理 Abstract 后确认的 75 个手稿前缀；
|
||||||
|
6. Springer 的合法 arXiv 参考文献和正文编号方法列表保持原样；
|
||||||
|
7. 9 张表的标签、属性、单元格内容和顺序不变，只发生已批准的实体和行布局修改；
|
||||||
|
8. dmp 的重复页眉、跨页修复和参考文献空行在最终 diff 中可分别追溯；
|
||||||
|
9. 逐份人工查看 `changes.diff`，确认没有上表之外的正文改写；
|
||||||
|
10. 运行前后 5 份 `data/md/*.md` 输入字节哈希完全不变；
|
||||||
|
11. 产物只出现在 Git 忽略的 `artifacts/<run_date>/runs/<run_id>/`，`git status` 不列出产物；
|
||||||
|
12. 清洗产物不复制图片。图片是否可显示不属于本轮成功条件，README 或 guide 必须明确这一点。
|
||||||
|
|
||||||
|
只要实际计数、结构或正文 diff 与预期不一致，就停止收尾并回到本设计核对。不得通过放宽规则、删除断言或把额外修改
|
||||||
|
改名为“格式整理”来让实验通过。
|
||||||
|
|
||||||
|
基础检查继续以根目录 README 为唯一命令权威。实现后至少实际运行 Ruff、mypy、完整 pytest、Agent 镜像 diff、
|
||||||
|
Git diff 检查和上述本地实验；未执行的检查不得报告为通过。
|
||||||
|
|
||||||
|
## 11. 方案比较与否决项
|
||||||
|
|
||||||
|
### 11.1 一个组件处理全部 ClinDB 问题
|
||||||
|
|
||||||
|
文件少，但任何规则变化都会改变同一组件语义，审计无法按问题归因，也不能被其他项目选择性复用。不采用。
|
||||||
|
|
||||||
|
### 11.2 用全局正则清理数字、实体和空行
|
||||||
|
|
||||||
|
会分别误伤作者单位、普通 Markdown 实体和正文编号列表。已有 5 份输入就存在真实反例。不采用。
|
||||||
|
|
||||||
|
### 11.3 第一批引入 Markdown/HTML parser 和 AST 重写
|
||||||
|
|
||||||
|
成熟 parser 能扩大结构识别范围，但会引入方言、渲染和非目标内容重写问题。当前 9 张表只需要保留原字节的严格词法
|
||||||
|
范围，标准库小扫描器足够验证第一批。不采用。
|
||||||
|
|
||||||
|
### 11.4 用英文词典自动决定跨页合词
|
||||||
|
|
||||||
|
词典无法决定 `SOFA-based` 是否保留连字符，也无法解释 `threshold.` + `olds` 的上游错误；领域词、缩写和专名还会造成
|
||||||
|
漏判。第一批采用可审计的显式映射，不采用词典。
|
||||||
|
|
||||||
|
### 11.5 把无合并表格自动转为 GFM
|
||||||
|
|
||||||
|
当前没有真实样本，且没有 `<th>` 时必须猜测表头；转义、换行和合并语义也未验证。第一批保留 HTML，不采用。
|
||||||
|
|
||||||
|
### 11.6 顺便打包图片或建立项目 Profile
|
||||||
|
|
||||||
|
图片需要输入资产定位、路径改写、复制冲突和产物目录契约；Profile 需要身份、配置格式和兼容规则。两者与文本组件并非
|
||||||
|
同一问题，不能借本设计默认授权。不采用。
|
||||||
|
|
||||||
|
## 12. 风险与后续变化成本
|
||||||
|
|
||||||
|
- **严格规则会漏掉未来变体：** 这是保真优先的有意选择。新格式先补证据、测试和版本，不在原规则中静默放宽。
|
||||||
|
- **行号阈值针对长手稿：** 短文档可能不清理。组件与组合分离后，可以新增另一个有独立证据的组件或升级本组件，
|
||||||
|
不需要修改核心。
|
||||||
|
- **跑动页眉仍是启发式分类：** 重复加断句证据降低误删，但不能证明适用于所有论文。它只进入 ClinDB 第一批组合，
|
||||||
|
不成为全局默认规则。
|
||||||
|
- **显式断词映射需要维护：** 新文档会出现新词，但映射作为参数记录，不必修改执行器或其他组件；代价是每次扩张前要评审。
|
||||||
|
- **严格 HTML 子集会忽略复杂表格：** 失败关闭能够避免重写未知结构。以后若需要完整 HTML parser，只替换内部扫描实现或
|
||||||
|
升级两个表格组件，不改变 `DocumentSnapshot`、`TextEdit` 和产物契约。
|
||||||
|
- **整体表格布局编辑的 diff 较大：** 每张表只有一条可回放 `Change`，但人读 diff 时会看到整行展开。实体组件先执行，
|
||||||
|
`result.json` 仍能分别追踪局部实体修改和后续布局修改。
|
||||||
|
- **没有 Profile 对象：** 当前组合只在一个实验脚本中，适合第一批验证。未来新增第二个稳定项目组合时再设计 Profile，
|
||||||
|
组件本身无需迁移。
|
||||||
|
- **图片仍不可随产物查看：** 这是 `0005` 的既有边界。若人工评审必须在清洗目录直接显示图片，应新增图片资产产物设计，
|
||||||
|
不改变本轮文本组件。
|
||||||
|
|
||||||
|
这套拆分保持“业务识别组件 → 精确编辑核心 → 文件实验层”的单向依赖。后续调整某条识别规则通常只影响一个组件及其测试；
|
||||||
|
调整项目组合只影响实验入口或未来 Profile；调整产物日期和目录只影响 `0005` 的 artifact store。核心数据契约无需跟着变化。
|
||||||
|
|
||||||
|
## 13. 批准后的实施边界
|
||||||
|
|
||||||
|
如果用户明确批准本文，只授权：
|
||||||
|
|
||||||
|
1. 新增第 8 节列出的 7 个组件、2 个内部辅助模块、合成测试和 first-batch 实验脚本；
|
||||||
|
2. 对现有 arXiv 组件做不改变语义和版本的物理行辅助函数迁移；
|
||||||
|
3. 按第 4、6、7 节实现固定组件身份、严格识别、精确修改和组合顺序；
|
||||||
|
4. 按第 9 节运行合成检查；
|
||||||
|
5. 合成检查通过后，按 `0005` 对 `data/md/` 的 5 份副本保存一次本地 first-batch 实验；
|
||||||
|
6. 根据真实实现和实际验证更新 README、ClinDB reference、对应 explanation 和 guide。
|
||||||
|
|
||||||
|
批准仍不授权：
|
||||||
|
|
||||||
|
- 修改、覆盖、移动或删除任何输入和真实数据；
|
||||||
|
- 保存 GovDoc 或其他仓库外材料的清洗产物；
|
||||||
|
- 把 `artifacts/`、`data/` 或真实文本 fixture 加入 Git；
|
||||||
|
- 实现内容恢复、Inspector、建议、Profile 格式、公共 CLI、通用 parser、图片复制或路径改写；
|
||||||
|
- 提交、推送、创建 PR、发布或修改其他仓库。
|
||||||
@@ -0,0 +1,480 @@
|
|||||||
|
# 0007：同仓库本地 Markdown 清洗评审器
|
||||||
|
|
||||||
|
## 状态
|
||||||
|
|
||||||
|
已批准并冻结（2026-08-23）。
|
||||||
|
|
||||||
|
本设计已经用户明确批准，授权按第 16 节实施。后续契约或范围变化必须新增 design，不回写本文。
|
||||||
|
|
||||||
|
`supersedes: 0005`（范围有限）：本文只替代 `0005` 中“不建设 Web 界面”和运行目录只有既有文件的选择，
|
||||||
|
允许在本地实验层增加机器定位文件，并在同一仓库建立独立的只读评审器。`0005` 已确定的输入只读、成功输出边界、
|
||||||
|
JSON 审计权威、原子发布、私有权限、30 日保留和真实数据限制继续有效。
|
||||||
|
|
||||||
|
## 1. 问题与可观察现象
|
||||||
|
|
||||||
|
当前本地实验已经为每份成功文档保存 `cleaned.md`、`result.json` 和 `changes.diff`。这些文件可以证明最终文本和逐项
|
||||||
|
修改，但人工评审仍需要分别打开原文、清洗结果和 JSON，难以快速回答三个问题：
|
||||||
|
|
||||||
|
1. 原文和最终清洗结果在整篇文档中有什么差异；
|
||||||
|
2. 每个组件实际修改了什么、修改了多少处；
|
||||||
|
3. 多个组件顺序执行时，某个组件看到的输入和它产生的输出分别是什么。
|
||||||
|
|
||||||
|
现有运行层在预检时已经取得每份输入的绝对解析路径，在发布前也知道最终运行目录，但只把用于展示的
|
||||||
|
`source_label` 和内容哈希写入清单。运行结束后，评审工具不能只靠运行目录找到完整原文；`changes.diff` 只有差异上下文，
|
||||||
|
不能代替完整输入。
|
||||||
|
|
||||||
|
`result.json` 中的 `Change.span` 又绑定各组件执行前的中间快照。它使用 Python 字符串下标，不能被浏览器当作 JavaScript
|
||||||
|
字符串下标直接使用。前端如果自行猜测坐标或只把所有变化涂在最终文本上，可能把组件归属显示错。
|
||||||
|
|
||||||
|
用户已经确定前端与 Python 核心放在同一个仓库，减少跨仓开发、评审和版本协调成本。这里的解耦目标因此不是物理分仓，
|
||||||
|
而是保持依赖方向和数据契约清楚，使清洗核心与界面可以分别修改和验证。
|
||||||
|
|
||||||
|
## 2. 目标与非目标
|
||||||
|
|
||||||
|
### 2.1 目标
|
||||||
|
|
||||||
|
- 在本仓库增加一个只服务本机的 Markdown 清洗评审器；
|
||||||
|
- 同时展示一份成功文档的完整原文和完整清洗结果；
|
||||||
|
- 按流水线顺序列出组件身份、版本和实际修改数量；
|
||||||
|
- 选择组件后，准确展示该组件执行前后的 Markdown，而不是把中间坐标错误套到最终文本；
|
||||||
|
- 点击修改时显示修改理由、位置、`before` 和 `after`；
|
||||||
|
- 运行时记录最终运行目录和每份输入的本机路径，不复制或改写原文；
|
||||||
|
- 读取前验证原文、产物和修改链哈希，验证失败时拒绝近似展示；
|
||||||
|
- 让浏览器界面只依赖评审器内部的版本化 API，不依赖 Python 类、源码路径或 `TransformResult`；
|
||||||
|
- 让报告生成和评审服务共用同一套 Python 快照重放逻辑，不在 TypeScript 中复制审计规则；
|
||||||
|
- 前端代码、依赖和检查保存在独立子项目中，不进入 `mdpolish` Python 分发包。
|
||||||
|
|
||||||
|
### 2.2 非目标
|
||||||
|
|
||||||
|
- 不编辑 Markdown，不批准、拒绝或调整某条修改；
|
||||||
|
- 不从页面重新运行清洗，不改变组件、参数或顺序；
|
||||||
|
- 不原地覆盖输入，也不把页面状态写回运行目录；
|
||||||
|
- 不建设远程服务、多人协作、账户、数据库、上传、分享或长期归档；
|
||||||
|
- 不提供安装后的稳定公共 CLI、公共 HTTP API 或生产部署接口；
|
||||||
|
- 第一版不渲染 Markdown、原始 HTML、图片或外部资源，只展示忠实的 Markdown 源文本；
|
||||||
|
- 不为 `failed` 或 `unstable` 文档构造、保存或展示一份看似正式的完整输出；
|
||||||
|
- 不改变清洗语义、组件版本、规则顺序、统计口径或核心数据模型；
|
||||||
|
- 不增加 GovDoc 或其他仓库外真实数据的产物保存权限；
|
||||||
|
- 不兼容任意历史或未来产物格式，第一版只读取本文明确批准的版本。
|
||||||
|
|
||||||
|
源码视图是第一版的有意边界。当前规则包含换行、HTML 源码和精确字符修改，渲染后的页面可能隐藏这些变化；同时允许
|
||||||
|
原始 HTML 和远程图片进入页面会扩大安全与数据泄露风险。以后确实需要渲染预览时，应单独决定 Markdown 方言、HTML
|
||||||
|
净化、图片寻址和网络策略。
|
||||||
|
|
||||||
|
## 3. 不改变的现有事实
|
||||||
|
|
||||||
|
本文继续沿用以下权威：
|
||||||
|
|
||||||
|
- `manifest.json` 仍是运行身份、环境、流水线、文档索引和汇总的权威；
|
||||||
|
- `result.json` 仍是文档状态、修改、错误和残留候选的权威；
|
||||||
|
- `cleaned.md` 仍是 `success` 文档最终文本的权威；
|
||||||
|
- `changes.diff` 仍只是原文到最终成功输出的人工评审视图；
|
||||||
|
- `failed` 和 `unstable` 文档没有 `cleaned.md`，评审器不得把重放得到的部分文本命名或展示为成功结果；
|
||||||
|
- 所有产物继续位于 Git 忽略的 `artifacts/`，目录权限为 `0700`、文件权限为 `0600`，默认保留 30 个日历日；
|
||||||
|
- 组件和内存核心继续不知道文件路径、运行目录、HTTP 或前端。
|
||||||
|
|
||||||
|
本文不修改 `manifest.json` 和 `result.json` 的 `schema_version: 1`。本机路径进入独立定位文件，避免给已存在的审计字段
|
||||||
|
增加未版本化含义。
|
||||||
|
|
||||||
|
## 4. 方案比较
|
||||||
|
|
||||||
|
### 4.1 前端直接导入或调用 `mdpolish`
|
||||||
|
|
||||||
|
这种方式可以少写一层适配，但界面会依赖 Python 包布局、类和调用方式，浏览器也不能直接执行 Python。核心升级容易迫使
|
||||||
|
前端同步修改。不采用。
|
||||||
|
|
||||||
|
### 4.2 纯静态页面要求用户每次选择原文和所有产物
|
||||||
|
|
||||||
|
不需要本地服务，但浏览器对本机路径有权限限制，不同浏览器的目录选择能力也不同。每次手工配对多份文档容易选错，且无法
|
||||||
|
自然复用运行时已经验证过的路径。不采用为默认流程。
|
||||||
|
|
||||||
|
### 4.3 打包为桌面应用
|
||||||
|
|
||||||
|
桌面壳可以直接访问文件，但第一版会额外引入安装包、自动更新、签名和多平台问题，超过本地实验评审需要。不采用。
|
||||||
|
|
||||||
|
### 4.4 独立前端加 Node.js 本地只读服务
|
||||||
|
|
||||||
|
运行层只增加本机路径定位文件；评审器用独立适配器读取产物，校验后通过同源本地 API 提供给界面。前端不接触任意文件
|
||||||
|
路径，也不理解 Python 对象。这能让评审器脱离 Python 独立运行，但必须在 TypeScript 中重新实现 Python 已有的组件分批、
|
||||||
|
编辑排序、逐项原文验证和逐批哈希验证，还要持续处理 Python 码点与 JavaScript UTF-16 坐标的差异。
|
||||||
|
|
||||||
|
哈希校验能阻止错误重放被静默展示，却不能消除两套实现的维护成本。第一版没有“把运行目录交给一台不含 Python 和
|
||||||
|
`mdpolish` 的机器独立评审”的目标，因此不采用。
|
||||||
|
|
||||||
|
### 4.5 独立前端加 Python 标准库本地只读服务
|
||||||
|
|
||||||
|
Python 服务读取已经发布的文件契约，并通过一个从 `reporting.py` 提取的纯重放模块复核组件快照。报告生成和评审服务共用
|
||||||
|
同一套应用顺序与哈希验证；服务再把完整阶段文本和派生的编辑器坐标通过同源本地 API 提供给前端。React/TypeScript
|
||||||
|
只负责交互和展示，不解释 artifact，也不应用 Python span。
|
||||||
|
|
||||||
|
采用此方案。它保留浏览器与核心对象之间的 API 边界，同时把风险最高的可信重放留在唯一的 Python 实现中。代价是启动
|
||||||
|
评审器必须具有本仓库支持的 Python 环境和匹配版本的 `mdpolish`；这是第一版本地实验流程可以接受的约束。
|
||||||
|
|
||||||
|
## 5. 总体结构与依赖方向
|
||||||
|
|
||||||
|
```text
|
||||||
|
显式 Markdown 输入
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
mdpolish 本地实验层
|
||||||
|
│
|
||||||
|
├── 既有 manifest / result / cleaned / diff
|
||||||
|
└── 新增 review-locator.json
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Python 产物适配与重放服务
|
||||||
|
│ │
|
||||||
|
│ ├── 验证路径、哈希和状态
|
||||||
|
│ └── 用共享 Python 逻辑重放组件快照
|
||||||
|
▼
|
||||||
|
本地只读 API
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
浏览器评审界面
|
||||||
|
```
|
||||||
|
|
||||||
|
依赖规则固定为：
|
||||||
|
|
||||||
|
- `src/mdpolish/` 不导入 `reviewer/`；
|
||||||
|
- `reporting.py` 和评审服务只共同依赖一个不读写文件的 Python 重放模块；
|
||||||
|
- `reviewer/server/` 可以导入该重放模块，但不导入或调用组件、`Pipeline`、实验入口，也不重新运行清洗；
|
||||||
|
- Python 产物适配器只读取已经发布的文件契约，不把内部模型当作 artifact 格式；
|
||||||
|
- 前端组件只读取评审器 API，不读取磁盘，不解析 `manifest.json` 或 `result.json`；
|
||||||
|
- Python 服务负责产物版本差异、组件快照重放和编辑器坐标派生，页面不维护第二套产物解释逻辑；
|
||||||
|
- 评审器不成为 `mdpolish` wheel 的一部分。
|
||||||
|
|
||||||
|
评审服务依赖共享重放模块是本文唯一批准的源码级连接。它用于消除两套可信重放实现，不允许扩展为从页面调用清洗核心。
|
||||||
|
除此之外，同仓库只用于共享开发流程、提交历史和契约测试。
|
||||||
|
|
||||||
|
## 6. 仓库结构与技术选择
|
||||||
|
|
||||||
|
批准后允许新增以下结构：
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/mdpolish/
|
||||||
|
└── _artifact_replay.py # 无文件 I/O 的共享快照重放逻辑
|
||||||
|
reviewer/
|
||||||
|
├── .nvmrc
|
||||||
|
├── __init__.py
|
||||||
|
├── server/ # Python 标准库 HTTP 服务和产物版本适配器
|
||||||
|
├── package.json
|
||||||
|
├── package-lock.json
|
||||||
|
├── tsconfig.json
|
||||||
|
├── vite.config.ts
|
||||||
|
├── src/
|
||||||
|
│ ├── client/ # React 浏览器界面，只依赖本地 API
|
||||||
|
│ └── shared/ # 前端使用的 API 类型和运行时校验
|
||||||
|
└── tests/ # 前端合成数据和界面测试
|
||||||
|
tests/
|
||||||
|
└── test_reviewer_*.py # Python 产物适配、重放和服务测试
|
||||||
|
```
|
||||||
|
|
||||||
|
第一版采用：
|
||||||
|
|
||||||
|
- 仓库现有 Python 环境运行本地只读服务；
|
||||||
|
- Python 标准库实现 HTTP、文件读取和 JSON 解析，不增加 Python 运行依赖；
|
||||||
|
- `_artifact_replay.py` 同时供 `reporting.py` 和评审服务调用，保持唯一的可信重放实现；
|
||||||
|
- Node.js 24 LTS 只用于前端开发、测试和构建，不负责读取或重放 artifact；
|
||||||
|
- TypeScript 表达前端 API 数据和界面类型；
|
||||||
|
- React 构建交互界面；
|
||||||
|
- Vite 提供开发与构建入口；
|
||||||
|
- CodeMirror 6 Merge View 提供只读双栏文本比较；
|
||||||
|
- Vitest 和 React Testing Library 覆盖前端 API 数据校验与主要界面状态。
|
||||||
|
|
||||||
|
选择 React 与 Vite 是为了在一个独立目录内保留成熟的模块、类型和开发服务器，而不把 JavaScript 构建配置混入
|
||||||
|
Python 包。[React 官方文档](https://react.dev/learn/build-a-react-app-from-scratch)把 Vite 列为从零建立客户端应用可用的
|
||||||
|
构建工具；[Vite 官方文档](https://vite.dev/guide/features)说明它原生处理 TypeScript，但类型检查需要作为独立检查执行。
|
||||||
|
[CodeMirror 的 Merge View](https://codemirror.net/docs/ref/#merge.MergeView)能直接比较两个文本并标记插入与删除，避免本项目
|
||||||
|
自行实现文本 diff 编辑器。
|
||||||
|
|
||||||
|
精确前端依赖版本只在 `reviewer/package.json` 和锁文件中维护，本文不复制版本清单。运行时不得从 CDN 下载脚本、字体、
|
||||||
|
样式或其他资源。根据 [Node.js 官方版本状态](https://nodejs.org/en/about/previous-releases)，当前机器上的 Node.js 20 已结束
|
||||||
|
官方支持，不能作为前端实现验收环境。
|
||||||
|
|
||||||
|
批准本文同时授权实施者在当前用户已有的 nvm 中执行 `nvm install 24` 和 `nvm use 24`，在 `reviewer/.nvmrc` 固定主版本
|
||||||
|
`24`，并在 `package.json` 的 `engines.node` 中限制为 Node.js 24。该授权不包括使用系统包管理器安装 Node.js、替换
|
||||||
|
`/usr/bin/node` 或修改其他用户的环境；如果当前用户的 nvm 不可用，应停止并另行确认。
|
||||||
|
|
||||||
|
## 7. 本机运行定位文件
|
||||||
|
|
||||||
|
每次新实验在运行目录根部增加：
|
||||||
|
|
||||||
|
```text
|
||||||
|
artifacts/<run_date>/runs/<run_id>/review-locator.json
|
||||||
|
```
|
||||||
|
|
||||||
|
第一版结构固定为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
schema_version
|
||||||
|
run
|
||||||
|
run_id
|
||||||
|
run_directory
|
||||||
|
manifest_path
|
||||||
|
documents[]
|
||||||
|
document_id
|
||||||
|
source_path
|
||||||
|
input_sha256
|
||||||
|
```
|
||||||
|
|
||||||
|
字段语义如下：
|
||||||
|
|
||||||
|
- `schema_version` 固定为整数 `1`；
|
||||||
|
- `run_id` 必须与 `manifest.json` 和目录身份一致；
|
||||||
|
- `run_directory` 是运行发布时最终目录的绝对解析路径；
|
||||||
|
- `manifest_path` 固定为相对路径 `manifest.json`；
|
||||||
|
- `source_path` 是预检实际读取的普通文件的绝对解析路径，不保存调用方未解析的写法；
|
||||||
|
- `input_sha256` 必须与对应 manifest、result 和预检字节一致；
|
||||||
|
- 文档顺序必须与 manifest 一致。
|
||||||
|
|
||||||
|
定位文件使用与现有 JSON 相同的 UTF-8、无 BOM、两空格缩进、保留 Unicode 和末尾换行规则，权限为 `0600`。它与其他
|
||||||
|
文件一起写入临时运行目录、完成校验后原子发布；任一字段、写入或回读校验失败时不得发布最终运行目录。
|
||||||
|
|
||||||
|
职责边界固定为：`experiment.py` 根据已预检的文档身份、解析后的源路径、输入哈希和 artifact store 计算的最终目标目录
|
||||||
|
生成定位文件内容；`artifact_store.py` 不猜测或生成 `source_path` 等字段，只负责校验它与目标目录、manifest、result 的
|
||||||
|
结构一致性，以及权限、写入、回读和原子发布。最终路径布局仍只由 artifact store 决定，不能在实验层复制日期目录规则。
|
||||||
|
|
||||||
|
`review-locator.json` 只是本机寻址信息，不替代 manifest 或 result 的运行事实。绝对路径可能包含用户名和本机目录结构，
|
||||||
|
因此它属于本地敏感产物，不得提交、推送、上传、复制到 Wiki 或显示在普通终端摘要中。
|
||||||
|
|
||||||
|
评审器启动时由用户明确传入当前运行目录。若目录后来被移动，启动参数中的实际目录是读取产物的依据；定位文件中的
|
||||||
|
`run_directory` 只用于提示位置已经变化，不能让服务跳转读取另一个运行目录。`source_path` 失效或哈希不符时，该文档原文
|
||||||
|
标记为不可用，不能退化为按文件名搜索或继续展示不匹配内容。
|
||||||
|
|
||||||
|
已有运行目录没有定位文件，仍然是合法的历史实验产物。第一版评审器可以展示其清单和审计，但不承诺自动找到完整原文，
|
||||||
|
也不向历史目录补写定位文件。要进行完整双栏评审，应产生一次新的、具有不同运行 ID 的实验。
|
||||||
|
|
||||||
|
## 8. 本地服务入口与边界
|
||||||
|
|
||||||
|
评审器提供仓库内入口，概念调用方式为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
python -m reviewer.server --run-dir <run_directory>
|
||||||
|
```
|
||||||
|
|
||||||
|
生产构建得到的静态页面由 Python 服务与 API 一起提供。开发时 Vite 可以通过同源代理连接同一个 Python API，但 Node 进程不读取
|
||||||
|
artifact。精确的开发、构建和启动命令在实现并验证后只进入 README 的当前检查入口和对应 guide，不在多份文档维护不同
|
||||||
|
写法。入口不安装到系统，也不承诺长期参数兼容。
|
||||||
|
|
||||||
|
本地服务必须：
|
||||||
|
|
||||||
|
- 只绑定 `127.0.0.1`，默认使用操作系统分配的空闲端口；
|
||||||
|
- 只服务启动参数指定的一次运行，不扫描整个 `artifacts/`；
|
||||||
|
- 只接受允许的 `GET` 和 `HEAD`，其他方法返回明确错误；
|
||||||
|
- 不设置跨域许可，只接受本服务自身页面的同源请求；
|
||||||
|
- 校验 `Host`，拒绝非本机目标和路径穿越；
|
||||||
|
- 不提供任意文件路径读取接口；
|
||||||
|
- 不提供写、删、移动、清理、重新运行或 shell 执行接口；
|
||||||
|
- 对页面和 API 设置禁止缓存、内容类型保护和限制脚本来源的安全响应头；
|
||||||
|
- 退出时不修改运行目录、原文或浏览器外的任何状态。
|
||||||
|
|
||||||
|
服务读取的所有 artifact 相对路径都必须解析在启动运行目录内部。`source_path` 是唯一允许指向运行目录外的文件路径，且只在
|
||||||
|
定位文件、manifest 和 result 的文档身份与哈希全部一致后读取。绝对源路径不返回给浏览器，页面只显示 `source_label`。
|
||||||
|
|
||||||
|
## 9. 评审器内部 API
|
||||||
|
|
||||||
|
浏览器只使用同源 `/api/v1/`。第一版至少提供三个只读资源：
|
||||||
|
|
||||||
|
### 9.1 运行摘要
|
||||||
|
|
||||||
|
返回运行身份、状态、组件顺序、文档索引和汇总计数，并为每份文档说明原文和成功输出是否可用。它不返回绝对路径、Git
|
||||||
|
仓库路径或整篇 Markdown。
|
||||||
|
|
||||||
|
### 9.2 文档比较
|
||||||
|
|
||||||
|
对 `success` 文档返回：
|
||||||
|
|
||||||
|
- 完整且通过哈希验证的原始 Markdown；
|
||||||
|
- 完整且通过哈希验证的 `cleaned.md`；
|
||||||
|
- 文档状态、前后哈希和总修改数量；
|
||||||
|
- 按组件位置分组的修改数量；
|
||||||
|
- 每条修改的组件、版本、理由、派生行列、`before` 和 `after`；
|
||||||
|
- 由服务端根据对应组件前快照派生的 UTF-16 `editor_range`，只用于 CodeMirror 定位，不作为修改权威。
|
||||||
|
|
||||||
|
对 `failed` 和 `unstable` 文档只返回状态、已有审计、错误和残留候选，不返回或重建一份完整部分输出。原文可以作为只读
|
||||||
|
诊断背景返回，但页面必须明确该文档没有正式成功结果。
|
||||||
|
|
||||||
|
### 9.3 组件阶段比较
|
||||||
|
|
||||||
|
只对 `success` 文档提供。调用方按 `component_position` 请求一个组件，服务返回该组件执行前和执行后的完整 Markdown、
|
||||||
|
组件元数据和属于该组件的实际修改。零修改组件也返回相同的前后文本和零计数，使流水线顺序完整可见。
|
||||||
|
|
||||||
|
这是评审器内部契约，不是面向其他项目的公共 API。服务端和前端同属 `reviewer/`，字段改变仍需同步类型、运行时校验和
|
||||||
|
测试；不得让页面回退为直接读取磁盘 JSON。
|
||||||
|
|
||||||
|
## 10. 中间快照重放与编辑器坐标
|
||||||
|
|
||||||
|
组件阶段视图必须从已验证原文按实际 `Change` 重放，不保存新的中间 Markdown 文件。`manifest.pipeline.components[]`
|
||||||
|
是完整组件顺序的权威，`result.json.changes[]` 只记录实际发生的修改。对 `success` 文档，服务按以下规则处理：
|
||||||
|
|
||||||
|
1. 从 manifest 的第一个组件开始按位置遍历，先验证 `changes[]` 中的组件位置不倒退、同一位置连续出现，并且组件身份和
|
||||||
|
版本与 manifest 对应项一致；
|
||||||
|
2. 收集当前位置的全部 `Change`。没有 Change 时，该组件仍形成一个合法的零修改阶段：前后文本都等于当前重放文本，
|
||||||
|
修改数为零，不要求或伪造不存在的批次哈希；
|
||||||
|
3. 有 Change 时，要求该位置所有记录具有相同的 `before_sha256` 和 `after_sha256`，并验证当前文本哈希等于
|
||||||
|
`before_sha256`；
|
||||||
|
4. 逐项验证 Python 码点范围合法，且当前范围文字等于 `before`；
|
||||||
|
5. 按 `(span.start, span.end, proposal_index, edit_index)` 降序应用该组件的编辑。范围已经由核心冲突契约保证互不冲突，
|
||||||
|
完整排序键用于保持与现有 Python 执行器一致和结果确定；artifact 的记录顺序不是应用顺序；
|
||||||
|
6. 验证批次结果哈希等于 `after_sha256`，再把该阶段的前后文本和修改交给 API；
|
||||||
|
7. 全部 manifest 组件结束后，验证结果字节和哈希分别等于 `cleaned.md` 与 `current_sha256`。
|
||||||
|
|
||||||
|
因此，对 `success` 文档，“manifest 中存在、该位置没有 Change”明确表示组件运行过但没有修改；manifest 中没有该位置才是
|
||||||
|
组件缺失。`failed` 和 `unstable` 不能仅凭 manifest 推断全部组件已经运行，服务不为它们建立完整组件阶段链。
|
||||||
|
|
||||||
|
共享重放模块直接使用 Python 码点范围，不进行跨语言应用。为了让 CodeMirror 跳转，Python 服务在已经验证的组件前快照上
|
||||||
|
另外计算零起始、半开区间的 UTF-16 code unit `editor_range`。该范围只是 API 派生视图；前端不得用它重新应用修改，
|
||||||
|
`result.json` 的 Python span 仍是审计权威。
|
||||||
|
|
||||||
|
严格 UTF-8 输入不会包含无法重新编码的孤立代理项。仍需用包含中文、补充平面字符、组合字符、BOM、CRLF 和无末尾换行的
|
||||||
|
合成测试证明服务返回的 UTF-16 范围能准确定位对应 `before`。
|
||||||
|
|
||||||
|
任一步失败都把该文档标记为“产物无法可信重放”，不生成组件阶段数据，不用文本搜索、diff 猜测或跳过错误继续展示。
|
||||||
|
`residual_proposals` 只显示为最终复查证据，绝不应用。
|
||||||
|
|
||||||
|
## 11. 第一版界面行为
|
||||||
|
|
||||||
|
第一版页面分为三个区域：
|
||||||
|
|
||||||
|
1. 运行与文档列表：显示整体状态、文档状态、前后哈希和修改数量；
|
||||||
|
2. 主双栏：只读展示完整原文与最终成功 Markdown，支持差异标记、行号和联动滚动；
|
||||||
|
3. 组件侧栏：按实际流水线顺序显示组件标识、版本和修改数，选择后把主双栏切换为该组件执行前后视图。
|
||||||
|
|
||||||
|
修改详情显示理由、修改前位置、`before` 和 `after`；点击后跳转到对应组件阶段的差异位置。同一个候选修改中的多条编辑应
|
||||||
|
继续用 `proposal_ref` 关联，不能把一项多位置动作错误显示为互不相关的业务问题。
|
||||||
|
|
||||||
|
页面必须清楚区分：
|
||||||
|
|
||||||
|
- 总修改数量是实际 `Change` 条数，不是 unified diff hunk 数、问题数或修改字符数；
|
||||||
|
- `success` 只表示选中组件运行稳定，不表示文档没有其他问题；
|
||||||
|
- `failed` / `unstable` 没有正式清洗结果；
|
||||||
|
- 零修改组件确实运行过，与组件缺失不是同一状态；
|
||||||
|
- 原文路径失效、哈希变化和产物损坏属于不同错误。
|
||||||
|
|
||||||
|
第一版以桌面浏览器评审为主，但键盘应能切换文档、组件和修改，状态不能只依赖颜色表达。页面不提供编辑控件、文件拖放、
|
||||||
|
远程链接预览或任何会修改外部状态的按钮。
|
||||||
|
|
||||||
|
## 12. 版本兼容策略
|
||||||
|
|
||||||
|
第一版支持：
|
||||||
|
|
||||||
|
- `manifest.json` schema `1`；
|
||||||
|
- `result.json` schema `1`；
|
||||||
|
- `review-locator.json` schema `1`；
|
||||||
|
- 评审器内部 API `/api/v1/`。
|
||||||
|
|
||||||
|
遇到未知 schema 时必须说明不支持的文件和版本并拒绝读取，不能忽略版本继续猜测。以后 mdpolish 产物升级时，在
|
||||||
|
`reviewer/server/` 增加明确的 Python 版本适配器；前端仍只使用统一内部 API。只有无法保持原含义时才升级 API 版本。
|
||||||
|
|
||||||
|
Python 核心和前端因此可以在同一仓库分别演进，但“任意核心变化都无需调整评审器”不是目标。真正保证的是：变化集中在
|
||||||
|
文件契约适配器，并由兼容性测试暴露，不让内部类或目录变化直接扩散到页面。
|
||||||
|
|
||||||
|
## 13. 隐私与数据安全
|
||||||
|
|
||||||
|
- 定位文件、原文、成功 Markdown、diff 和审计继续按本地敏感数据处理；
|
||||||
|
- 页面和本地 API 不包含遥测、错误上报、CDN、远程字体或自动更新请求；
|
||||||
|
- 浏览器运行时只允许连接同源本地服务；
|
||||||
|
- Markdown 只作为文本交给只读编辑器，不注入 `innerHTML`；
|
||||||
|
- 服务不输出原文、diff、绝对源路径或 `before` / `after` 到终端日志；
|
||||||
|
- 关闭页面或服务不删除缓存以外的任何文件，也不改变 30 日保留规则；
|
||||||
|
- 定位文件不得解除 Git 忽略，不得进入测试 fixture；自动测试只使用虚构 Markdown 和临时目录。
|
||||||
|
|
||||||
|
本设计没有因为增加页面而扩大真实数据授权。当前仍只允许对 `data/md/` 中现有 5 份论文副本保存实验产物；
|
||||||
|
`/home/lihaoze/gov_test_data` 及其他外部真实材料继续只读且不得生成本项目产物。
|
||||||
|
|
||||||
|
## 14. 测试与验收
|
||||||
|
|
||||||
|
### 14.1 Python 实验层与 artifact store
|
||||||
|
|
||||||
|
至少覆盖：
|
||||||
|
|
||||||
|
- 定位文件记录实际最终运行目录、解析后的源路径、文档顺序和输入哈希；
|
||||||
|
- 实验层生成定位内容，artifact store 不自行推断源路径；
|
||||||
|
- 定位文件、manifest 和 result 的身份或哈希不一致时拒绝发布；
|
||||||
|
- 定位文件使用批准的 JSON 编码和 `0600` 权限；
|
||||||
|
- 任一定位文件写入、回读或校验失败时不发布最终运行目录；
|
||||||
|
- 输入路径和文件内容在运行前后不变；
|
||||||
|
- `failed` / `unstable` 仍不产生 `cleaned.md`；
|
||||||
|
- 既有 manifest 和 result schema 不被静默改变。
|
||||||
|
|
||||||
|
### 14.2 Python 重放、产物适配器与服务
|
||||||
|
|
||||||
|
合成测试至少覆盖：
|
||||||
|
|
||||||
|
- 合法 `success`、`failed` 和 `unstable` 产物；
|
||||||
|
- 缺失、未知版本、非法 JSON、BOM、错误编码和路径穿越；
|
||||||
|
- 源文件缺失、不是普通文件、哈希改变或身份不匹配；
|
||||||
|
- cleaned 哈希不符、修改原文不符和中间快照链断裂；
|
||||||
|
- 报告生成和评审服务对同一合成修改链得到完全相同的中间与最终文本；
|
||||||
|
- 精确按 `(span.start, span.end, proposal_index, edit_index)` 降序应用，不把 artifact 记录顺序当作应用顺序；
|
||||||
|
- 中文、补充平面字符、组合字符、CRLF、空文档和无末尾换行的快照重放及 UTF-16 编辑器范围；
|
||||||
|
- 多组件、首个/中间/末尾零修改组件、空流水线、一个候选多编辑和后续组件基于新快照修改；
|
||||||
|
- manifest 中缺失组件与合法零修改组件能被区分，倒序、非连续重复或身份不匹配的组件批次被拒绝；
|
||||||
|
- 未知文档和组件位置返回明确错误；
|
||||||
|
- 非 GET/HEAD 方法、非本机 Host、跨域和任意路径读取被拒绝；
|
||||||
|
- API 不泄露绝对源路径，并包含禁止缓存和内容类型保护头。
|
||||||
|
|
||||||
|
### 14.3 前端
|
||||||
|
|
||||||
|
至少覆盖：
|
||||||
|
|
||||||
|
- 运行和文档状态列表；
|
||||||
|
- 原文与成功结果双栏；
|
||||||
|
- 组件顺序、版本、零修改和修改数量；
|
||||||
|
- 组件视图切换、修改详情和跳转；
|
||||||
|
- `failed` / `unstable` 不显示正式结果；
|
||||||
|
- 路径失效、哈希不符、产物损坏和未知 schema 的清楚错误；
|
||||||
|
- 不把 Markdown 当 HTML 执行；
|
||||||
|
- 键盘操作和不依赖颜色的状态表达。
|
||||||
|
|
||||||
|
评审器检查至少包括 Python 的 Ruff、mypy 和 pytest，以及前端 TypeScript 类型检查、lint、单元/组件测试和生产构建。
|
||||||
|
精确命令和依赖版本在实现后进入各自唯一权威；根目录 README 只记录当前真实可用的总体验收入口。
|
||||||
|
|
||||||
|
### 14.4 本地 5 份论文验收
|
||||||
|
|
||||||
|
实现和合成测试通过后，允许使用新的运行 ID 对 `data/md/` 中 5 份本地论文副本再次运行已批准的第一批流水线，并验证：
|
||||||
|
|
||||||
|
1. 定位文件列出 5 份输入，路径和哈希正确，输入字节不变；
|
||||||
|
2. 页面列出 8 个组件及其实际顺序和版本；
|
||||||
|
3. 5 份文档都能同时打开原文和成功输出；
|
||||||
|
4. 页面总修改数与 manifest、result 一致；
|
||||||
|
5. 每个组件阶段都能重放到正确前后哈希，零修改组件显示为零而不是缺失；
|
||||||
|
6. JAMA Abstract 前内容、Springer 合法 arXiv 引用和图片引用文字等既有反例仍保持不变；
|
||||||
|
7. 浏览器和终端不发生外部网络请求，不输出真实文本或绝对源路径；
|
||||||
|
8. 新运行仍只位于 Git 忽略的 `artifacts/`，权限和保留日期符合 `0005`。
|
||||||
|
|
||||||
|
这一验收只证明当前 5 份论文和既有 8 个组件能够被本地评审，不表示通用数据集、GovDoc 或生产部署已经支持。
|
||||||
|
|
||||||
|
## 15. 风险与代价
|
||||||
|
|
||||||
|
- **评审服务依赖本仓库 Python 环境：** 它换来唯一的可信重放实现，但不能在只有静态产物和 Node.js 的机器上独立启动；
|
||||||
|
- **共享 Python 模块仍是源码耦合点：** 依赖只限纯重放函数，并用报告与评审一致性测试控制，不能扩展到组件或流水线；
|
||||||
|
- **绝对路径会泄露本机结构并可能失效：** 路径只进入私有定位文件，移动后明确报错，不把路径当可移植身份；
|
||||||
|
- **新增 Node.js 工具链：** Node.js 只用于前端开发和构建，但仓库仍需要第二套受支持环境和锁文件；
|
||||||
|
- **编辑器坐标仍有跨语言差异：** Python 服务派生并测试 UTF-16 范围；坐标错误只能影响跳转，不能改变已经在 Python 中完成并
|
||||||
|
验证哈希的阶段文本；
|
||||||
|
- **完整文本会占用浏览器内存：** 第一版面向当前本地实验，不宣称支持任意极端长度；出现真实瓶颈后再设计流式读取或虚拟化；
|
||||||
|
- **源码视图不能展示最终排版：** 它优先保证修改证据忠实；渲染预览另行处理安全、方言和资源边界；
|
||||||
|
- **本地 HTTP 仍有攻击面：** 仅回环监听、同源、Host 校验、无写接口和严格路径白名单共同缩小范围；
|
||||||
|
- **旧运行无法自动双栏：** 不回写历史或猜测路径，代价是完整查看需要重新产生带定位文件的新运行。
|
||||||
|
|
||||||
|
## 16. 批准后的实施边界
|
||||||
|
|
||||||
|
批准本文只授权：
|
||||||
|
|
||||||
|
1. 由现有实验层生成 `review-locator.json` 内容，由 artifact store 校验、写入、回读并随运行目录原子发布；
|
||||||
|
2. 从 `reporting.py` 提取第 10 节所需的纯 Python 重放逻辑，并由报告生成和评审服务共同调用；
|
||||||
|
3. 创建第 6 节列出的 Python 服务、前端子项目、锁文件、合成测试和构建配置；
|
||||||
|
4. 实现第 8 至 11 节的本地只读 API、产物适配、双栏源码比较和组件阶段视图；
|
||||||
|
5. 在当前用户已有的 nvm 中安装和使用 Node.js 24，并新增 `reviewer/.nvmrc` 和 `package.json` 的版本限制；
|
||||||
|
6. 在 `.gitignore` 中忽略评审器构建、依赖和覆盖率产物；
|
||||||
|
7. 更新根目录 README 当前能力与实际检查入口，并新增对应 explanation 和经验证 guide；
|
||||||
|
8. 在合成测试通过后，按第 14.4 节对本地 5 份论文副本产生一次新的私有运行并完成只读页面验收。
|
||||||
|
|
||||||
|
批准不授权：
|
||||||
|
|
||||||
|
- 改变清洗规则、组件顺序、核心模型或既有 JSON schema；
|
||||||
|
- 修改、覆盖、移动或复制任何输入 Markdown；
|
||||||
|
- 为 GovDoc 或其他仓库外数据生成产物；
|
||||||
|
- 实现 Markdown/HTML 渲染、图片访问、编辑、审核、回写、远程服务、认证或数据库；
|
||||||
|
- 修改系统级 Node.js 安装，提交、推送、创建 PR 或发布。
|
||||||
@@ -75,9 +75,9 @@ arXiv:<新版数字编号和版本> [<ASCII 分类>] <日> <英文月份缩写>
|
|||||||
|
|
||||||
## 6. 剩余边界
|
## 6. 剩余边界
|
||||||
|
|
||||||
这个组件只证明第一条严格删除规则能够在公共核心上闭环，不表示论文已经清洗完成。HTML 实体、Word 批注、手稿
|
这个组件最初证明了第一条严格删除规则能够在公共核心上闭环。此后 ClinDB 第一批另外 7 个组件已经按 `0006` 实现，
|
||||||
行号、断词、表格和参考文献间距仍未实现。当前只有固定输入和固定组件的本地实验输出；通用文件接口、profile、
|
当前完整组合见 [`clindb-first-batch-components.md`](clindb-first-batch-components.md)。这仍不表示论文内容问题全部解决；
|
||||||
公共 CLI 和通用批处理仍不存在。
|
通用文件接口、profile、公共 CLI、图片资产和通用批处理仍不存在。
|
||||||
|
|
||||||
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
||||||
提高命中数量直接修改正则表达式。
|
提高命中数量直接修改正则表达式。
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# ClinDB 第一批组件如何在不猜正文的前提下完成清洗
|
||||||
|
|
||||||
|
## 1. 它解决的实际问题
|
||||||
|
|
||||||
|
5 份 ClinDB 论文 Markdown 同时包含编辑痕迹、转换噪声和排版噪声。它们看起来都像“删掉几行或整理一下格式”，
|
||||||
|
实际误删边界不同：行首数字可能是手稿行号，也可能是作者单位；`arXiv:` 可能是边栏戳，也可能是合法参考文献；
|
||||||
|
编号列表可能属于 References，也可能是正文方法步骤。
|
||||||
|
|
||||||
|
当前实现没有建立一个能随意改全文的“大清洗器”，而是把第一批确定问题拆成 8 个组件。每个组件只识别一种证据，
|
||||||
|
返回快照绑定的精确 `TextEdit`，由公共流水线统一验证、应用和记录。
|
||||||
|
|
||||||
|
清洗语义来自已批准的
|
||||||
|
[`0006-clindb-first-batch-cleaning-components.md`](../design/0006-clindb-first-batch-cleaning-components.md)，
|
||||||
|
输入范围和稳定计数见
|
||||||
|
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。
|
||||||
|
|
||||||
|
## 2. 当前组件和顺序
|
||||||
|
|
||||||
|
```text
|
||||||
|
Word 批注 ─┐
|
||||||
|
手稿行号 ──┼──► arXiv 戳 ──► 重复页眉 ──► 映射断词
|
||||||
|
│ │
|
||||||
|
│ └────────────► 参考文献空行
|
||||||
|
│
|
||||||
|
└────► HTML 双重实体 ──► HTML 表格布局
|
||||||
|
```
|
||||||
|
|
||||||
|
实验脚本固定按以下顺序组装：
|
||||||
|
|
||||||
|
| 顺序 | 组件 | 当前作用 |
|
||||||
|
| ---: | --- | --- |
|
||||||
|
| 1 | `paper.word_review_comment` | 删除完整单行 Word 批注及其后一个空行 |
|
||||||
|
| 2 | `paper.manuscript_line_number` | 删除由长递增序列确认的手稿行号 |
|
||||||
|
| 3 | `paper.arxiv_submission_stamp` | 删除严格整行提交戳 |
|
||||||
|
| 4 | `paper.repeated_running_header` | 删除重复页眉，并接回有明确续句证据的段落 |
|
||||||
|
| 5 | `paper.page_break_word_join` | 只应用本次运行参数中记录的词片段映射 |
|
||||||
|
| 6 | `markdown.html_table_double_escape` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||||
|
| 7 | `markdown.html_table_layout` | 保留 HTML 内容，把单行表格展开成一行一个 `<tr>` |
|
||||||
|
| 8 | `paper.reference_spacing` | 只在完整连续的 References 章节中统一条目空行 |
|
||||||
|
|
||||||
|
顺序不是为了让结果“看起来整齐”。dmp 的一个重复页眉正好位于第 18、19 条参考文献之间；如果不先删除页眉，
|
||||||
|
参考文献组件就不能确认这是完整连续序列。HTML 实体先修改小范围 token，表格布局再基于新快照替换整张表，
|
||||||
|
两类修改仍能在审计中分别追踪。
|
||||||
|
|
||||||
|
## 3. 为什么行号不能使用全局正则
|
||||||
|
|
||||||
|
JAMA 文档中一共有 134 个看似 `数字 + 空格` 的行首前缀。前 59 个位于 Abstract 之前，是作者单位编号；真正的手稿
|
||||||
|
行号只有 Abstract 之后的 75 个。
|
||||||
|
|
||||||
|
当前组件要求：
|
||||||
|
|
||||||
|
- 文档中恰好有一个严格 `## Abstract`；
|
||||||
|
- 只看它之后的候选；
|
||||||
|
- 候选数字全部严格递增；
|
||||||
|
- 至少有 20 个候选，并至少有 2 个数字标题。
|
||||||
|
|
||||||
|
任一条件失败就整篇不改。这样会漏掉较短的带行号手稿，但不会为了提高命中率删除作者单位或零散数字段落。
|
||||||
|
|
||||||
|
## 4. 为什么断词使用显式映射
|
||||||
|
|
||||||
|
“行尾连字符加下一行小写字母”无法决定连字符应删还是保留：`possi-` + `bly` 应成为 `possibly`，而
|
||||||
|
`SOFA-` + `based` 应保留为 `SOFA-based`。dmp 还存在 `threshold.` + `olds`，它同时包含多余句点和重复片段，
|
||||||
|
普通词典也无法解释。
|
||||||
|
|
||||||
|
因此组件的实际参数记录三项：左片段、右片段和结果词。只有相邻物理行或中间恰好一个空行、词边界完整且映射唯一时
|
||||||
|
才修改。增加新词不是自动学习行为，需要先批准新的项目映射；组件算法版本不变时，运行清单仍能通过参数区分实际语义。
|
||||||
|
|
||||||
|
## 5. 两个表格组件怎样共享范围
|
||||||
|
|
||||||
|
`_html_table.py` 只识别当前转换器输出的严格子集：`<tr>` 直接位于 `<table>` 下，`<td>` / `<th>` 直接位于
|
||||||
|
`<tr>` 下，标签完整闭合，单元格中没有嵌套标签。它返回原字符串下标，不生成 DOM，也不重新渲染全文。
|
||||||
|
|
||||||
|
实体组件只处理单元格文本中的三个精确 token：
|
||||||
|
|
||||||
|
```text
|
||||||
|
&lt; → <
|
||||||
|
&gt; → >
|
||||||
|
&amp; → &
|
||||||
|
```
|
||||||
|
|
||||||
|
结果仍是合法 HTML 源码中的单层实体。标签、属性、表格外文本和其他实体不受影响。
|
||||||
|
|
||||||
|
布局组件只处理整个片段没有换行的严格表格。它原样复用 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和结束标签，
|
||||||
|
只增加外层换行与两个空格缩进。当前 9 张真实表格都有 `colspan`，所以全部保留 HTML；实现没有猜测表头，也没有转 GFM。
|
||||||
|
|
||||||
|
遇到未闭合、嵌套、额外结构标签或混合换行时，扫描失败关闭，保持原文。
|
||||||
|
|
||||||
|
## 6. 共享代码为什么仍然很小
|
||||||
|
|
||||||
|
- `_text_ranges.py` 只提供 Python 字符下标下的物理行、行尾和空行关系；
|
||||||
|
- `_html_table.py` 只提供严格 HTML 表格、行和单元格范围；
|
||||||
|
- 业务组件依赖这两个私有模块，但辅助模块不依赖组件、流水线或文件层；
|
||||||
|
- 文件实验层只接收已经组装的 `Pipeline`，不知道任何识别规则。
|
||||||
|
|
||||||
|
因此以后放宽某个业务规则通常只改一个组件及其测试；替换 HTML 识别方式不会改变 `DocumentSnapshot`、
|
||||||
|
`ProposedChange`、`Change` 或 JSON 产物；未来引入 Profile 时，也只接管当前脚本里的组件组装。
|
||||||
|
|
||||||
|
## 7. 当前验证结果和边界
|
||||||
|
|
||||||
|
2026-08-22 在 Python 3.13.11 环境完成：
|
||||||
|
|
||||||
|
- Ruff 通过；
|
||||||
|
- mypy 检查 37 个文件无问题；
|
||||||
|
- pytest 204 项通过；
|
||||||
|
- 5 份本地论文全部 `success`，合计 155 条 `Change`；
|
||||||
|
- 对 5 份成功输出再次运行，全部 `success` 且零修改；
|
||||||
|
- 输入运行前后哈希不变；
|
||||||
|
- JAMA Abstract 前内容不变，Springer 两条合法 arXiv 参考文献保留；
|
||||||
|
- 图片引用文字不变，但实验产物没有复制图片资产。
|
||||||
|
|
||||||
|
保存型实验位于本机 Git 忽略的
|
||||||
|
`artifacts/2026-08-22/runs/clindb-first-batch-v1/`。运行和复核方法见
|
||||||
|
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。
|
||||||
|
|
||||||
|
这次成功只证明批准的 8 类规则在当前 5 份输入上闭环。截断、缺表、乱码、OCR 语义错误、图片资产、修订词选择和一般
|
||||||
|
段落重排仍不在自动清洗范围内。
|
||||||
@@ -12,24 +12,34 @@
|
|||||||
- 只有 `success` 文档才产生正式的清洗后 Markdown。
|
- 只有 `success` 文档才产生正式的清洗后 Markdown。
|
||||||
|
|
||||||
已经实现的范围来自已批准的
|
已经实现的范围来自已批准的
|
||||||
[`0005-local-experiment-runner-and-artifacts.md`](../design/0005-local-experiment-runner-and-artifacts.md)。
|
[`0005-local-experiment-runner-and-artifacts.md`](../design/0005-local-experiment-runner-and-artifacts.md) 和
|
||||||
|
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。
|
||||||
精确字段、校验和函数签名以 `src/mdpolish/` 中的代码与测试为准。
|
精确字段、校验和函数签名以 `src/mdpolish/` 中的代码与测试为准。
|
||||||
|
|
||||||
## 2. 三层怎样解耦
|
## 2. 保存与评审怎样解耦
|
||||||
|
|
||||||
```text
|
```text
|
||||||
experiment.py
|
experiment.py
|
||||||
├── pipeline.py 只负责内存清洗
|
├── pipeline.py 只负责内存清洗
|
||||||
├── reporting.py 只负责 JSON、行列和 unified diff
|
├── reporting.py 只负责 JSON、行列和 unified diff
|
||||||
└── artifact_store.py 只负责日期目录、权限和原子发布
|
└── artifact_store.py 只负责日期目录、权限和原子发布
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
已发布运行目录
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
reviewer/ 只通过发布后的文件做本地只读评审
|
||||||
```
|
```
|
||||||
|
|
||||||
- `experiment.py` 严格读取调用方显式列出的 UTF-8 Markdown，逐份调用同一个 `Pipeline`；
|
- `experiment.py` 严格读取调用方显式列出的 UTF-8 Markdown，逐份调用同一个 `Pipeline`；
|
||||||
- `reporting.py` 重放并校验 `Change` 的快照链，再生成机器可读审计和人可读 diff；
|
- `reporting.py` 重放并校验 `Change` 的快照链，再生成机器可读审计和人可读 diff；
|
||||||
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
||||||
|
- 同仓库 `reviewer/server/` 只共用无文件 I/O 的 Python 快照重放模块，不导入组件或流水线；它读取 manifest、result、
|
||||||
|
成功输出和本机定位文件。
|
||||||
|
|
||||||
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
||||||
ClinDB 的 5 份论文和 arXiv 组件组合只存在于仓库内实验脚本，通用模块没有硬编码论文名。
|
ClinDB 的 5 份论文、历史 arXiv 单组件组合和当前 first-batch 组合只存在于两个仓库内实验脚本，通用模块没有硬编码
|
||||||
|
论文名或业务组件。
|
||||||
|
|
||||||
## 3. 输入怎样保持原样
|
## 3. 输入怎样保持原样
|
||||||
|
|
||||||
@@ -70,6 +80,7 @@ artifacts/
|
|||||||
└── runs/
|
└── runs/
|
||||||
└── <run_id>/
|
└── <run_id>/
|
||||||
├── manifest.json
|
├── manifest.json
|
||||||
|
├── review-locator.json
|
||||||
└── documents/
|
└── documents/
|
||||||
└── <document_id>/
|
└── <document_id>/
|
||||||
├── result.json
|
├── result.json
|
||||||
@@ -94,6 +105,9 @@ artifacts/
|
|||||||
`changes.diff` 是原始输入到最终成功输出的 unified diff，只用于人工查看。它不包含绝对路径或时间戳，
|
`changes.diff` 是原始输入到最终成功输出的 unified diff，只用于人工查看。它不包含绝对路径或时间戳，
|
||||||
也不是修改重放的权威；机器审计仍以 `result.json` 为准。
|
也不是修改重放的权威；机器审计仍以 `result.json` 为准。
|
||||||
|
|
||||||
|
`review-locator.json` 只记录本次运行目录、每份输入的绝对解析路径和输入哈希，供本地评审器重新找到完整原文。它不复制
|
||||||
|
原文，不替代 manifest 或 result，也不作为可移植运行身份。绝对路径可能泄露本机目录结构，因此该文件同样是本地敏感数据。
|
||||||
|
|
||||||
## 6. 为什么 reporter 要重放修改
|
## 6. 为什么 reporter 要重放修改
|
||||||
|
|
||||||
第二个组件看到的是第一个组件修改后的快照，因此后续 `Change.span` 不一定对应最初输入。为了生成准确行列，
|
第二个组件看到的是第一个组件修改后的快照，因此后续 `Change.span` 不一定对应最初输入。为了生成准确行列，
|
||||||
@@ -136,6 +150,9 @@ artifact store 先在同一日期的 `runs/` 下建立本次专用临时目录
|
|||||||
|
|
||||||
当前只批准对 `data/md/` 中 5 份论文副本保存产物。仓库外 GovDoc 和其他真实数据没有因此获得输出授权。
|
当前只批准对 `data/md/` 中 5 份论文副本保存产物。仓库外 GovDoc 和其他真实数据没有因此获得输出授权。
|
||||||
|
|
||||||
|
同仓库只读页面的路径验证、组件快照重放和使用边界见
|
||||||
|
[`local-markdown-reviewer.md`](local-markdown-reviewer.md)。
|
||||||
|
|
||||||
## 9. 已完成的真实验证
|
## 9. 已完成的真实验证
|
||||||
|
|
||||||
2026-08-22 使用 `paper.arxiv_submission_stamp` `1.0.0` 对 5 份本地论文副本完成一次保存型实验：
|
2026-08-22 使用 `paper.arxiv_submission_stamp` `1.0.0` 对 5 份本地论文副本完成一次保存型实验：
|
||||||
@@ -154,3 +171,25 @@ artifact store 先在同一日期的 `runs/` 下建立本次专用临时目录
|
|||||||
|
|
||||||
实际运行方法见
|
实际运行方法见
|
||||||
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
||||||
|
|
||||||
|
同日又使用 `design/0006` 的 8 组件流水线完成 ClinDB 第一批保存型实验：
|
||||||
|
|
||||||
|
| 项目 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 运行 ID | `clindb-first-batch-v1` |
|
||||||
|
| 输出位置 | `artifacts/2026-08-22/runs/clindb-first-batch-v1/` |
|
||||||
|
| 文档状态 | 5/5 `success` |
|
||||||
|
| 实际修改 | dmp 47、ejhf 9、jama 79、sim 3、springer 17，共 155 条 |
|
||||||
|
| 第二次运行 | 5/5 `success`，合计 0 条修改 |
|
||||||
|
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||||
|
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||||
|
| 内容反例 | JAMA Abstract 前内容不变；Springer 两条合法 arXiv 引用保留；图片引用文字不变 |
|
||||||
|
| 权限 | 运行目录 `0700`，产物文件 `0600` |
|
||||||
|
|
||||||
|
当前完整运行方法见
|
||||||
|
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。实验层的文件、
|
||||||
|
JSON、diff 和权限契约没有因组件增多而改变。
|
||||||
|
|
||||||
|
2026-08-23 又产生运行 `clindb-first-batch-reviewer-v1`，用于验证新定位文件和本地页面：5/5 文档为 `success`，合计
|
||||||
|
155 条修改，输入运行前后哈希不变。评审器 API 校验了 5 份原文、成功输出以及 8 个组件形成的 40 个阶段，所有文本哈希
|
||||||
|
均与审计一致。实际页面启动步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||||
|
|||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# 本地 Markdown 清洗评审器如何保持只读和可追踪
|
||||||
|
|
||||||
|
## 1. 它解决什么问题
|
||||||
|
|
||||||
|
本地清洗实验已经保存最终 Markdown、逐条审计和 unified diff，但人工评审仍需要在多个文件之间切换，也看不到某个组件
|
||||||
|
执行前后的完整文本。当前评审器把一次已发布运行变成只读页面：主视图比较原文和最终成功输出，组件时间线则比较每个
|
||||||
|
组件实际收到的快照和它产生的新快照。
|
||||||
|
|
||||||
|
实现范围来自已批准的
|
||||||
|
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。精确 API、字段和运行行为以
|
||||||
|
[`reviewer/`](../../reviewer/) 中的代码、类型和测试为准。
|
||||||
|
|
||||||
|
## 2. 同仓库怎样保持解耦
|
||||||
|
|
||||||
|
```text
|
||||||
|
mdpolish 本地实验层
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
manifest / result / cleaned / review-locator
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Python 产物适配器与共享重放 ──► 本地只读 API ──► React 页面
|
||||||
|
```
|
||||||
|
|
||||||
|
- `src/mdpolish/` 不导入 `reviewer/`；
|
||||||
|
- `reviewer/server/` 只从 `mdpolish` 导入无文件 I/O 的 `_artifact_replay.py`，不导入组件、流水线或实验入口；
|
||||||
|
- 浏览器只读取 `/api/v1/`，不解析磁盘 JSON，也不知道绝对路径；
|
||||||
|
- Python wheel 不包含前端代码或 Node.js 依赖；
|
||||||
|
- 产物 schema 以后改变时，差异集中在服务端版本适配器，不扩散到页面组件。
|
||||||
|
|
||||||
|
前端与核心位于同一 Git 仓库，方便开发和评审，但仍是独立的 Node.js package。依赖版本只在
|
||||||
|
[`reviewer/package.json`](../../reviewer/package.json) 和锁文件维护。
|
||||||
|
|
||||||
|
## 3. 运行定位文件保存什么
|
||||||
|
|
||||||
|
新实验会在运行目录根部原子保存 `review-locator.json`。它记录：
|
||||||
|
|
||||||
|
- 运行 ID 和发布时的绝对运行目录；
|
||||||
|
- `manifest.json` 的固定相对位置；
|
||||||
|
- 每份文档的 ID、实际读取的绝对源路径和输入 SHA-256。
|
||||||
|
|
||||||
|
定位文件不复制原文，也不替代 manifest 或 result。评审器由用户显式指定当前运行目录；记录的旧运行目录只用于判断目录
|
||||||
|
是否被移动。服务只根据定位文件读取对应原文，并在每次展示前重新计算哈希。源文件不存在或内容变化时，页面明确报告
|
||||||
|
不可用，不按名称搜索替代文件。
|
||||||
|
|
||||||
|
绝对路径会暴露本机目录结构，所以定位文件与其他 artifact 一样使用 `0600` 权限并按本地敏感数据处理。历史运行没有该
|
||||||
|
文件时仍可查看清单和局部审计，但不能自动展示完整原文；评审器不会回写历史目录。
|
||||||
|
|
||||||
|
## 4. 服务为什么只读取一次运行
|
||||||
|
|
||||||
|
启动时必须传入一个具体运行目录。服务不会扫描 `artifacts/`，也没有让浏览器传入任意文件路径的 API。它先校验：
|
||||||
|
|
||||||
|
1. manifest、result 和存在的 locator 都是支持的 schema、严格 UTF-8 JSON；
|
||||||
|
2. 文档、组件、状态、路径和计数彼此一致；
|
||||||
|
3. 所有 artifact 路径都留在所选运行目录；
|
||||||
|
4. 原文和成功输出的字节哈希与审计一致；
|
||||||
|
5. 成功文档确实同时具有 `cleaned.md` 和 `changes.diff`。
|
||||||
|
|
||||||
|
HTTP 只监听 `127.0.0.1` 的随机空闲端口，只接受 `GET` 和 `HEAD`。服务拒绝非本机 Host、跨域 Origin、路径穿越和
|
||||||
|
写请求，不提供删除、移动、重新清洗或 shell 执行能力。响应禁止缓存，不开放 CORS，也不向浏览器返回绝对源路径。
|
||||||
|
|
||||||
|
## 5. 组件阶段怎样准确重放
|
||||||
|
|
||||||
|
`Change.span` 使用 Python Unicode 码点位置，而 JavaScript 编辑器使用 UTF-16 code unit。共享 Python 重放模块直接按
|
||||||
|
原生码点范围逐组件处理，不让浏览器应用修改：
|
||||||
|
|
||||||
|
1. 当前完整文本哈希必须等于组件批次的 `before_sha256`；
|
||||||
|
2. 每条范围内文本必须等于 `before`；
|
||||||
|
3. 同一批次不得有冲突范围，并按位置从后向前应用；
|
||||||
|
4. 应用后完整文本哈希必须等于 `after_sha256`；
|
||||||
|
5. 全部组件结束后必须逐字等于 `cleaned.md`；
|
||||||
|
6. 服务端另外从已验证的组件前快照派生 UTF-16 `editor_range`，只供 CodeMirror 跳转。
|
||||||
|
|
||||||
|
组件没有修改时，阶段前后文本和哈希相同，但该组件仍显示在时间线中。包含中文、emoji、组合字符、BOM、CRLF 和无末尾
|
||||||
|
换行的合成测试用于保护跨语言坐标。任一重放校验失败时，页面拒绝显示组件阶段，不通过搜索或 diff 猜测位置。
|
||||||
|
|
||||||
|
中间快照只在服务内存中按需生成，不保存新的 Markdown 文件。`failed` 和 `unstable` 文档只显示错误、残留候选和已有的
|
||||||
|
局部审计，不重建一份看似正式的部分输出。
|
||||||
|
|
||||||
|
## 6. 页面当前能看什么
|
||||||
|
|
||||||
|
页面当前提供：
|
||||||
|
|
||||||
|
- 运行状态、文档状态、哈希和实际 `Change` 数量；
|
||||||
|
- 完整原文与最终成功 Markdown 的只读双栏源码比较；
|
||||||
|
- 8 个组件的实际顺序、版本和每份文档修改数量；
|
||||||
|
- 任一组件执行前后的完整文本比较；
|
||||||
|
- 修改理由、派生行列、`before` / `after` 和同候选修改关联；
|
||||||
|
- `failed`、`unstable`、路径失效、哈希变化和未知 schema 的独立错误状态。
|
||||||
|
|
||||||
|
Markdown 只作为文本交给 CodeMirror，不进入 `innerHTML`。第一版不渲染 Markdown、HTML 或图片，不加载 CDN、远程字体、
|
||||||
|
遥测和其他外部资源，也不提供编辑、审核或回写。
|
||||||
|
|
||||||
|
## 7. 当前验证结果和边界
|
||||||
|
|
||||||
|
2026-08-24 使用 Python 3.13.11 和当前用户 nvm 中的 Node.js 24.19.0 完成：
|
||||||
|
|
||||||
|
- Python Ruff、mypy 和 229 项 pytest 通过；
|
||||||
|
- reviewer ESLint、TypeScript、9 项 Vitest 和生产构建通过；
|
||||||
|
- 新运行 `clindb-first-batch-reviewer-v1` 的 5 份论文全部 `success`，共 155 条实际修改；
|
||||||
|
- 5 份原文运行前后哈希不变；
|
||||||
|
- 本地 API 成功校验 5 份文档、8 个组件和 40 个组件阶段；
|
||||||
|
- 所有原文、成功输出和阶段前后文本的 SHA-256 与运行审计一致；
|
||||||
|
- 生产页面和全部本地构建资源可以通过只读服务读取，响应没有 CORS 并包含禁止缓存和内容类型保护头。
|
||||||
|
|
||||||
|
当前环境没有可用于自动视觉检查的本地浏览器，因此布局的真实浏览器视觉效果尚未验证。当前结果证明构建、服务、数据
|
||||||
|
重放和主要 React 状态可以运行，不等于已经完成跨浏览器、极端长度、渲染预览或生产部署验证。
|
||||||
|
|
||||||
|
实际启动与评审步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
# 使用本地页面评审一次 Markdown 清洗运行
|
||||||
|
|
||||||
|
## 1. 适用范围
|
||||||
|
|
||||||
|
本指南用于打开已经发布在 `artifacts/<YYYY-MM-DD>/runs/<run_id>/` 的本地清洗运行。完整双栏比较要求运行目录包含
|
||||||
|
`review-locator.json`，并且原文仍位于运行时记录的位置且哈希未改变。
|
||||||
|
|
||||||
|
评审器只读文件，不重新运行组件、不修改原文和产物。当前不适用于 GovDoc、远程目录、多人共享或生产部署。
|
||||||
|
|
||||||
|
本指南于 2026-08-23 使用 Python 3.13.11、当前用户 nvm 中的 Node.js 24.19.0 和运行
|
||||||
|
`clindb-first-batch-reviewer-v1` 实际验证。
|
||||||
|
|
||||||
|
## 2. 准备一次可评审运行
|
||||||
|
|
||||||
|
先按 [`run-local-clindb-first-batch-experiment.md`](run-local-clindb-first-batch-experiment.md) 产生一次新的运行。成功终端摘要
|
||||||
|
会给出绝对 artifact 路径，例如：
|
||||||
|
|
||||||
|
```text
|
||||||
|
artifacts=/home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
确认该目录内存在：
|
||||||
|
|
||||||
|
```text
|
||||||
|
manifest.json
|
||||||
|
review-locator.json
|
||||||
|
documents/
|
||||||
|
```
|
||||||
|
|
||||||
|
不要编辑定位文件，也不要向旧运行目录手工补写它。历史运行缺少 locator 时，使用不同运行 ID 重新实验。
|
||||||
|
|
||||||
|
## 3. 准备评审器
|
||||||
|
|
||||||
|
评审器前端的安装、检查和构建要求 Node.js 24 LTS。当前用户 nvm 已安装与 `.nvmrc` 匹配的版本。在仓库根目录执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd reviewer
|
||||||
|
nvm use
|
||||||
|
node --version
|
||||||
|
npm --version
|
||||||
|
```
|
||||||
|
|
||||||
|
`node --version` 必须是受支持的 `v24`。本仓库不负责修改系统级 Node.js；版本不符时先在开发环境外准备正确运行时。
|
||||||
|
|
||||||
|
首次安装或锁文件变化后，仍在 `reviewer/` 目录执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm ci
|
||||||
|
```
|
||||||
|
|
||||||
|
当前有效的 Python 与 reviewer 检查命令只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后构建页面：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. 启动一次运行
|
||||||
|
|
||||||
|
回到仓库根目录，用当前 Python 虚拟环境启动只读服务并传入运行目录：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m reviewer.server \
|
||||||
|
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
启动成功时只打印运行 ID、文档数量和随机本机端口，不打印原文或绝对源路径：
|
||||||
|
|
||||||
|
```text
|
||||||
|
mdpolish 评审器已启动：http://127.0.0.1:<port>（<run_id>，<count> 份文档）
|
||||||
|
```
|
||||||
|
|
||||||
|
在本机浏览器打开该地址。评审结束后回到终端按 `Ctrl+C` 停止服务。
|
||||||
|
|
||||||
|
开发页面时开两个终端。第一个终端在仓库根目录把 Python API 固定到 Vite 代理使用的本机端口：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m reviewer.server \
|
||||||
|
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id> \
|
||||||
|
--port 4174
|
||||||
|
```
|
||||||
|
|
||||||
|
第二个终端启动只绑定 `127.0.0.1:5173` 的 Vite 页面；它只把 `/api/` 代理给上述 Python 服务，Node.js 不读取 artifact：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd reviewer
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. 页面怎么查看
|
||||||
|
|
||||||
|
1. 先确认顶部整体状态和总修改数与 `manifest.json` 一致；
|
||||||
|
2. 在左侧选择文档，主双栏默认显示清洗前和最终成功输出；
|
||||||
|
3. 在组件时间线选择一个组件，双栏切换为该组件执行前后；
|
||||||
|
4. 检查组件版本和修改数，零修改应显示 `0`，而不是从时间线消失；
|
||||||
|
5. 点击修改详情，跳到对应组件阶段的位置并核对理由、`before` 和 `after`；
|
||||||
|
6. 对 `failed` / `unstable` 只查看错误和残留候选，不寻找不存在的正式输出。
|
||||||
|
|
||||||
|
页面中的总修改数是实际 `Change` 条数，不是 diff hunk 数、字符数或问题数量。
|
||||||
|
|
||||||
|
## 6. 常见错误
|
||||||
|
|
||||||
|
### 原文路径失效或哈希改变
|
||||||
|
|
||||||
|
评审器不会搜索同名文件。确认输入没有被移动或修改；如果需要在新位置运行，使用新的运行 ID 重新执行实验。不要改 locator
|
||||||
|
绕过哈希检查。
|
||||||
|
|
||||||
|
### 历史运行没有 `review-locator.json`
|
||||||
|
|
||||||
|
历史产物仍可在页面查看清单和已有审计，也可人工查看 manifest、result 和 diff，但第一版页面不能自动找到完整原文或
|
||||||
|
组件阶段。不要回写历史目录；需要完整双栏时重新运行一次即可。
|
||||||
|
|
||||||
|
### 不支持 schema
|
||||||
|
|
||||||
|
评审器只支持当前文档列出的 schema 版本。不要删除或伪造 `schema_version`；应升级评审器适配器或使用与产物匹配的代码。
|
||||||
|
|
||||||
|
### 没有 `cleaned.md`
|
||||||
|
|
||||||
|
对应文档状态是 `failed` 或 `unstable` 时这是正常边界。页面不会从 Change 重建并冒充正式结果。
|
||||||
|
|
||||||
|
### 服务拒绝 Host、Origin 或写请求
|
||||||
|
|
||||||
|
评审器只接受本机同源的只读请求。不要通过反向代理、远程端口转发或网页跨域调用它；这些用法没有批准。
|
||||||
|
|
||||||
|
## 7. 数据边界
|
||||||
|
|
||||||
|
页面会在本机内存中读取完整原文和成功输出。不要截图、复制或通过浏览器扩展分享真实内容。运行目录继续受 Git 忽略并按
|
||||||
|
manifest 的 `retention_until` 管理；页面不会自动删除到期产物。
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
```bash
|
```bash
|
||||||
.venv/bin/python --version
|
.venv/bin/python --version
|
||||||
.venv/bin/ruff check .
|
.venv/bin/ruff check .
|
||||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
.venv/bin/pytest
|
.venv/bin/pytest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# 运行本地 ClinDB 第一批完整清洗实验
|
||||||
|
|
||||||
|
## 1. 适用范围
|
||||||
|
|
||||||
|
本指南只运行仓库内已经批准的 first-batch 实验脚本：
|
||||||
|
|
||||||
|
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||||
|
- 流水线：`design/0006` 固定的 8 个组件和顺序；
|
||||||
|
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||||
|
- 输入只读，不覆盖原文件；
|
||||||
|
- 不读取或复制图片，不处理 `/home/lihaoze/gov_test_data`。
|
||||||
|
|
||||||
|
本指南于 2026-08-23 在 Python 3.13.11 环境实际验证。
|
||||||
|
|
||||||
|
## 2. 前置检查
|
||||||
|
|
||||||
|
在仓库根目录执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python --version
|
||||||
|
.venv/bin/ruff check .
|
||||||
|
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||||
|
.venv/bin/pytest
|
||||||
|
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||||
|
```
|
||||||
|
|
||||||
|
当前检查结果只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后再确认 5 份输入存在：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||||
|
```
|
||||||
|
|
||||||
|
必须看到 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`。不要把真实论文复制进测试 fixture。
|
||||||
|
|
||||||
|
## 3. 运行实验
|
||||||
|
|
||||||
|
人工选择一个当天未使用的安全运行 ID：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/run_clindb_first_batch_experiment.py \
|
||||||
|
--run-id clindb-first-batch-review
|
||||||
|
```
|
||||||
|
|
||||||
|
成功时终端只显示运行身份和汇总，不打印原文：
|
||||||
|
|
||||||
|
```text
|
||||||
|
run_id=clindb-first-batch-review
|
||||||
|
status=success
|
||||||
|
documents=5
|
||||||
|
changes=155
|
||||||
|
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-first-batch-review
|
||||||
|
```
|
||||||
|
|
||||||
|
同一天同名目录已存在时脚本会拒绝覆盖。需要重跑时使用新 ID，不要删除旧目录来绕过检查。
|
||||||
|
|
||||||
|
## 4. 先看哪些结果
|
||||||
|
|
||||||
|
先确认运行目录根部同时存在 `manifest.json` 和 `review-locator.json`。定位文件只供本机评审器寻找原文，包含绝对路径，
|
||||||
|
不得提交或分享。然后打开 `manifest.json`，确认：
|
||||||
|
|
||||||
|
- `run.status` 是 `success`；
|
||||||
|
- `summary.document_count` 和 `summary.success_count` 都是 5；
|
||||||
|
- `summary.failed_count`、`summary.unstable_count` 都是 0；
|
||||||
|
- `summary.change_count` 是 155；
|
||||||
|
- `pipeline.components` 的顺序与 `design/0006` 一致。
|
||||||
|
|
||||||
|
然后查看每份文档目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
documents/<document_id>/
|
||||||
|
├── result.json
|
||||||
|
├── cleaned.md
|
||||||
|
└── changes.diff
|
||||||
|
```
|
||||||
|
|
||||||
|
- `changes.diff` 用于人工查看输入到最终输出的总变化；
|
||||||
|
- `result.json` 用于按组件、理由、位置和哈希追踪每条修改；
|
||||||
|
- `cleaned.md` 是成功输出全文。
|
||||||
|
|
||||||
|
当前 5 份输入的预期计数是：
|
||||||
|
|
||||||
|
| 文档 | `Change` 数 |
|
||||||
|
| --- | ---: |
|
||||||
|
| dmp | 47 |
|
||||||
|
| ejhf | 9 |
|
||||||
|
| jama | 79 |
|
||||||
|
| sim | 3 |
|
||||||
|
| springer | 17 |
|
||||||
|
| **合计** | **155** |
|
||||||
|
|
||||||
|
按组件应为：Word 批注 2、手稿行号 75、arXiv 戳 2、重复页眉 2、映射断词 6、HTML 实体 31、
|
||||||
|
HTML 表格布局 9、参考文献空行 28。
|
||||||
|
|
||||||
|
## 5. 人工复核重点
|
||||||
|
|
||||||
|
除了逐份查看 diff，至少确认：
|
||||||
|
|
||||||
|
- JAMA 的 Abstract 前作者和单位编号仍在，只删除 Abstract 后的 75 个手稿行号；
|
||||||
|
- Springer 两条 `arXiv preprint arXiv:` 合法参考文献仍在；
|
||||||
|
- Springer 正文中的编号方法列表没有被参考文献规则整理；
|
||||||
|
- dmp 的重复页眉删除后，正文句子接回，参考文献第 18、19 条之间仍有一个空行；
|
||||||
|
- 9 张表仍是 HTML，属性和单元格内容未被布局组件改写；
|
||||||
|
- 双重实体变成单层 `<`、`>` 或 `&`，没有直接生成标签边界；
|
||||||
|
- 图片引用文字保持不变。
|
||||||
|
|
||||||
|
清洗目录没有复制图片资产，所以直接打开 `cleaned.md` 时图片仍可能无法显示。这不表示图片引用被清洗组件删除；
|
||||||
|
资产打包和路径改写需要单独设计。
|
||||||
|
|
||||||
|
## 6. 验证幂等和输入不变
|
||||||
|
|
||||||
|
流水线会在每份文档结束时做最终稳定性复查。需要额外复核整个保存结果时，可以把 `cleaned.md` 作为内存输入再次运行
|
||||||
|
同一 `build_pipeline()`；5 份都应为 `success` 且合计零 `Change`。
|
||||||
|
|
||||||
|
实验层已经在发布前后复读输入并比较字节哈希。需要人工记录运行前后的摘要时，可在运行前后分别执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sha256sum data/md/*.md
|
||||||
|
```
|
||||||
|
|
||||||
|
两次输出必须逐项一致。每个 `cleaned.md` 的 SHA-256 还必须等于对应 `result.json.current_sha256`。
|
||||||
|
|
||||||
|
## 7. 常见失败
|
||||||
|
|
||||||
|
### 状态不是 `success`
|
||||||
|
|
||||||
|
查看对应 `result.json` 的 `errors` 或 `residual_proposals`。`failed` / `unstable` 文档不会有正式 `cleaned.md`，
|
||||||
|
不能把其他文档的部分成功当成整批成功。
|
||||||
|
|
||||||
|
### 修改数不是 155
|
||||||
|
|
||||||
|
先按组件和文档分组定位差异。输入变化、组件参数变化或识别边界变化都必须回到 design/reference 核对；不要放宽断言、
|
||||||
|
补跑第二轮或手工改产物。
|
||||||
|
|
||||||
|
### 图片不显示
|
||||||
|
|
||||||
|
当前运行只保存 Markdown、审计和 diff，不复制图片。不要为了显示图片而修改输入路径或把真实资产强制加入 Git。
|
||||||
|
|
||||||
|
### 私有目录无法被 Snap 工具读取
|
||||||
|
|
||||||
|
运行目录权限是 `0700`，文件是 `0600`。使用普通编辑器或当前虚拟环境中的 Python 读取，不要放宽权限。
|
||||||
|
|
||||||
|
## 8. 数据边界
|
||||||
|
|
||||||
|
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||||
|
其他仓库、云存储或外部系统。
|
||||||
|
|
||||||
|
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。当前不自动删除；到期后如需清理，必须先确认具体运行目录。
|
||||||
|
|
||||||
|
需要在只读页面中查看完整前后文和各组件阶段时，继续按
|
||||||
|
[`review-local-cleaning-run.md`](review-local-cleaning-run.md) 操作。
|
||||||
@@ -11,8 +11,8 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
||||||
是它的首批输入，未来会继续扩充同源转换产物。
|
是它的首批输入，未来会继续扩充同源转换产物。
|
||||||
|
|
||||||
本仓库（mdpolish）为该项目的数据提供清洗能力；ClinDB-ReviewBench 通过 profile
|
本仓库（mdpolish）为该项目的数据提供清洗能力。当前由仓库内 first-batch 实验脚本显式组合论文规则，
|
||||||
表达论文场景的规则组合，不把论文专属规则写进通用核心。
|
尚未建立 Profile 对象或配置格式；论文专属规则仍不写进通用核心。
|
||||||
|
|
||||||
## 2. 第一版清洗目标
|
## 2. 第一版清洗目标
|
||||||
|
|
||||||
@@ -21,14 +21,14 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
|
|
||||||
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 1 | HTML 实体双重转义（D2） | `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，还原一层；幂等 | dmp L27/L33 共 23 处、ejhf L143 共 8 处，全部在 `<table>` 行内 |
|
| 1 | HTML 实体双重转义（D2） | 只在严格 HTML 表格单元格文本中把 `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，解除源码的一层转义；幂等 | dmp 23 处、ejhf 8 处，共 31 处 |
|
||||||
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
||||||
| 3 | Word 审阅批注（B2） | 以 `Commented [xx]:` 开头的行及其批注正文整块删除 | jama L155–157 共 2 处 |
|
| 3 | Word 审阅批注（B2） | 删除严格单行 `Commented [xx]: ...` 及其后第一个空行；不猜测多行批注正文 | jama 2 处 |
|
||||||
| 4 | 手稿行号（B1） | 仅剥离**单调递增序列**的行首 `^\d{1,3} ` 与标题内 `^#{1,6} \d{1,3} `；序列中断即停止提出后续修改，防止误伤正文数字（如 "35 pediatric experts"） | jama 128 行正文 + 6 个标题（L127/149/151/165/193/195） |
|
| 4 | 手稿行号（B1） | 只处理唯一 `## Abstract` 后至少 20 个、严格递增且含至少 2 个标题证据的完整序列；剥离普通行和标题中的数字前缀 | jama 75 处，其中普通行 69、标题 6；Abstract 前 59 个作者单位编号必须保留 |
|
||||||
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
||||||
| 6 | 单行 HTML 表格展开（D1） | 无 rowspan/colspan 的表转多行 GFM；含合并属性的表保留 HTML 但按 `<tr>` 换行缩进；内容一字不改 | 全部 9 个表：dmp 7、ejhf 1、springer 1 |
|
| 6 | 单行 HTML 表格展开（D1） | 严格完整的单行表格保留 HTML、属性和单元格内容，只按 `<tr>` 换行缩进；当前不转 GFM | 全部 9 个表：dmp 7、ejhf 1、springer 1；每张都有非 `1` 的 `colspan` |
|
||||||
| 7 | 跨页断词（E2） | 行尾连字符 + 下一行首小写字母 → 合并；仅在拼出的词能通过英文词表校验时执行，否则保留原样且不提出修改 | dmp L125/127（thresh-olds）、sim L87/89（possi-bly）、L217/219（cre-ated） |
|
| 7 | 跨页断词（E2） | 只按项目批准的左右片段和结果词映射合并相邻行或只隔一个空行的片段；不使用英文词表猜测 | dmp 2 处、jama 2 处、sim 2 处，共 6 处 |
|
||||||
| 8 | 参考文献分隔统一（G3） | `^\d+\. ` 条目之间统一一个空行 | dmp refs 19–33、springer refs 9–19（连续堆叠段） |
|
| 8 | 参考文献分隔统一（G3） | 只在准确 References 章节内，对从 1 开始连续递增的编号条目统一一个空行 | dmp 13 处、springer 15 处，共 28 处 |
|
||||||
|
|
||||||
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
||||||
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
||||||
@@ -54,20 +54,24 @@ ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI
|
|||||||
|
|
||||||
## 4. 输入输出边界
|
## 4. 输入输出边界
|
||||||
|
|
||||||
- 项目输入位于 `data/<转换结果目录>/markdowns/*.md`，原文件只读；
|
- 当前本地输入是 `data/md/` 中按论文缩写命名的 5 份 Markdown 副本，原文件只读；
|
||||||
- 当前核心只接收内存 Markdown 字符串并返回内存结果；文件输出位置和保存流程尚未设计，不回写、不覆盖原文件；
|
- 核心仍只接收内存 Markdown 字符串；本地实验层按 `design/0005` 将成功输出、JSON 审计和 diff 保存到
|
||||||
- 当前批次不读取图片资产；未来 Inspector 和路径改写分别设计；
|
`artifacts/<YYYY-MM-DD>/runs/<run_id>/`，不回写、不覆盖输入；
|
||||||
- 每处实际修改必须在内存结果中记录组件、理由、原文、改后内容和批次哈希，保证可追踪。
|
- 当前批次不读取或复制图片资产；未来 Inspector、资产打包和路径改写分别设计；
|
||||||
|
- 每处实际修改记录组件、理由、原文、改后内容和批次哈希，完整落盘字段以代码和 `design/0005` 为准。
|
||||||
|
|
||||||
## 5. 验收口径（第一版）
|
## 5. 验收口径（第一版）
|
||||||
|
|
||||||
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，处理数与审计报告的实测数字一致；
|
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，共产生 155 条 `Change`；
|
||||||
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
||||||
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
||||||
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
||||||
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
||||||
等数字开头/含数字短语不受影响。
|
等数字开头/含数字短语不受影响。
|
||||||
|
|
||||||
|
2026-08-22 的保存型验收中，5 份文档全部为 `success`，各组件计数为：批注 2、行号 75、arXiv 2、页眉 2、
|
||||||
|
断词 6、HTML 实体 31、表格布局 9、参考文献空行 28。第二次运行 5 份合计 0 条修改，输入运行前后不变。
|
||||||
|
|
||||||
## 6. 与审计报告的编号对应
|
## 6. 与审计报告的编号对应
|
||||||
|
|
||||||
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
24
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Repository-local Markdown cleaning reviewer."""
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
import eslint from "@eslint/js";
|
||||||
|
import reactHooks from "eslint-plugin-react-hooks";
|
||||||
|
import reactRefresh from "eslint-plugin-react-refresh";
|
||||||
|
import globals from "globals";
|
||||||
|
import tseslint from "typescript-eslint";
|
||||||
|
|
||||||
|
export default tseslint.config(
|
||||||
|
{ ignores: ["dist", "coverage", "node_modules"] },
|
||||||
|
eslint.configs.recommended,
|
||||||
|
...tseslint.configs.strictTypeChecked,
|
||||||
|
...tseslint.configs.stylisticTypeChecked,
|
||||||
|
{
|
||||||
|
files: ["**/*.{ts,tsx}"],
|
||||||
|
languageOptions: {
|
||||||
|
globals: { ...globals.browser, ...globals.node },
|
||||||
|
parserOptions: {
|
||||||
|
projectService: true,
|
||||||
|
tsconfigRootDir: import.meta.dirname,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
plugins: {
|
||||||
|
"react-hooks": reactHooks,
|
||||||
|
"react-refresh": reactRefresh,
|
||||||
|
},
|
||||||
|
rules: {
|
||||||
|
...reactHooks.configs.flat.recommended.rules,
|
||||||
|
"react-refresh/only-export-components": ["warn", { allowConstantExport: true }],
|
||||||
|
"@typescript-eslint/consistent-type-definitions": ["error", "interface"],
|
||||||
|
"@typescript-eslint/no-confusing-void-expression": "off",
|
||||||
|
"@typescript-eslint/restrict-template-expressions": ["error", { allowNumber: true }],
|
||||||
|
"react-hooks/set-state-in-effect": "off",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
files: ["tests/**/*.{ts,tsx}"],
|
||||||
|
rules: {
|
||||||
|
"@typescript-eslint/no-non-null-assertion": "off",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<meta name="color-scheme" content="light" />
|
||||||
|
<title>mdpolish 评审器</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="root"></div>
|
||||||
|
<script type="module" src="/src/client/main.tsx"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Generated
+4599
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"name": "mdpolish-reviewer",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"private": true,
|
||||||
|
"type": "module",
|
||||||
|
"engines": {
|
||||||
|
"node": "^24.0.0"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"dev": "vite --host 127.0.0.1",
|
||||||
|
"build": "npm run typecheck && vite build",
|
||||||
|
"typecheck": "tsc --noEmit -p tsconfig.json",
|
||||||
|
"lint": "eslint src tests vite.config.ts",
|
||||||
|
"test": "vitest run",
|
||||||
|
"check": "npm run lint && npm run test && npm run build"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@codemirror/lang-markdown": "6.5.2",
|
||||||
|
"@codemirror/merge": "6.12.2",
|
||||||
|
"@codemirror/state": "6.7.1",
|
||||||
|
"@codemirror/view": "6.43.9",
|
||||||
|
"react": "19.2.8",
|
||||||
|
"react-dom": "19.2.8"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@eslint/js": "10.0.1",
|
||||||
|
"@testing-library/jest-dom": "7.0.1",
|
||||||
|
"@testing-library/react": "16.3.2",
|
||||||
|
"@types/node": "24.13.3",
|
||||||
|
"@types/react": "19.2.18",
|
||||||
|
"@types/react-dom": "19.2.4",
|
||||||
|
"@vitejs/plugin-react": "6.1.0",
|
||||||
|
"eslint": "10.9.0",
|
||||||
|
"eslint-plugin-react-hooks": "7.1.1",
|
||||||
|
"eslint-plugin-react-refresh": "0.5.4",
|
||||||
|
"globals": "17.11.0",
|
||||||
|
"jsdom": "30.0.1",
|
||||||
|
"typescript": "6.0.3",
|
||||||
|
"typescript-eslint": "8.67.0",
|
||||||
|
"vite": "8.2.2",
|
||||||
|
"vitest": "4.1.11"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""Read-only artifact adapter and local HTTP service."""
|
||||||
|
|
||||||
|
from reviewer.server.artifacts import ReviewArtifactError, ReviewArtifacts
|
||||||
|
|
||||||
|
__all__ = ["ReviewArtifactError", "ReviewArtifacts"]
|
||||||
@@ -0,0 +1,233 @@
|
|||||||
|
"""Serve one local review run through a loopback-only read-only HTTP API."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import mimetypes
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
from urllib.parse import unquote, urlsplit
|
||||||
|
|
||||||
|
from reviewer.server.artifacts import JsonObject, ReviewArtifactError, ReviewArtifacts
|
||||||
|
|
||||||
|
_SECURITY_HEADERS = {
|
||||||
|
"Cache-Control": "no-store",
|
||||||
|
"Content-Security-Policy": (
|
||||||
|
"default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'none'; "
|
||||||
|
"font-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'"
|
||||||
|
),
|
||||||
|
"Referrer-Policy": "no-referrer",
|
||||||
|
"X-Content-Type-Options": "nosniff",
|
||||||
|
"X-Frame-Options": "DENY",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ReviewerHttpServer(ThreadingHTTPServer):
|
||||||
|
"""Threaded local server whose workers never keep process shutdown alive."""
|
||||||
|
|
||||||
|
daemon_threads = True
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_local_request(handler: BaseHTTPRequestHandler) -> bool:
|
||||||
|
host = handler.headers.get("Host")
|
||||||
|
if host is None:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
parsed_host = urlsplit(f"//{host}")
|
||||||
|
if parsed_host.username is not None or parsed_host.password is not None:
|
||||||
|
return False
|
||||||
|
if parsed_host.hostname not in {"127.0.0.1", "localhost"}:
|
||||||
|
return False
|
||||||
|
if parsed_host.port is not None and not 0 < parsed_host.port < 65536:
|
||||||
|
return False
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
origin = handler.headers.get("Origin")
|
||||||
|
if origin is None:
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
parsed_origin = urlsplit(origin)
|
||||||
|
return parsed_origin.scheme == "http" and parsed_origin.netloc == host
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _api_response(repository: ReviewArtifacts, path: str) -> JsonObject:
|
||||||
|
if path == "/api/v1/run":
|
||||||
|
return repository.run_summary()
|
||||||
|
parts = [part for part in path.split("/") if part]
|
||||||
|
try:
|
||||||
|
if len(parts) == 4 and parts[:3] == ["api", "v1", "documents"]:
|
||||||
|
return repository.document_comparison(unquote(parts[3], encoding="utf-8", errors="strict"))
|
||||||
|
if len(parts) == 6 and parts[:3] == ["api", "v1", "documents"] and parts[4] == "components":
|
||||||
|
document_id = unquote(parts[3], encoding="utf-8", errors="strict")
|
||||||
|
try:
|
||||||
|
component_position = int(parts[5])
|
||||||
|
except ValueError:
|
||||||
|
raise ReviewArtifactError("unknown_component", "组件位置不存在。", 404) from None
|
||||||
|
return repository.component_stage(document_id, component_position)
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||||
|
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||||
|
|
||||||
|
|
||||||
|
def _handler_factory(repository: ReviewArtifacts, static_root: Path) -> type[BaseHTTPRequestHandler]:
|
||||||
|
class ReviewRequestHandler(BaseHTTPRequestHandler):
|
||||||
|
server_version = "mdpolish-reviewer"
|
||||||
|
sys_version = ""
|
||||||
|
|
||||||
|
def log_message(self, format_: str, *args: Any) -> None:
|
||||||
|
del format_, args
|
||||||
|
|
||||||
|
def _headers(self, status: int, content_type: str, content_length: int) -> None:
|
||||||
|
self.send_response(status)
|
||||||
|
for name, value in _SECURITY_HEADERS.items():
|
||||||
|
self.send_header(name, value)
|
||||||
|
self.send_header("Content-Type", content_type)
|
||||||
|
self.send_header("Content-Length", str(content_length))
|
||||||
|
self.end_headers()
|
||||||
|
|
||||||
|
def _json(self, status: int, payload: JsonObject, *, head_only: bool) -> None:
|
||||||
|
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||||
|
self._headers(status, "application/json; charset=utf-8", len(content))
|
||||||
|
if not head_only:
|
||||||
|
self.wfile.write(content)
|
||||||
|
|
||||||
|
def _error(self, error: Exception, *, head_only: bool) -> None:
|
||||||
|
if isinstance(error, ReviewArtifactError):
|
||||||
|
status = error.http_status
|
||||||
|
code = error.code
|
||||||
|
message = str(error)
|
||||||
|
else:
|
||||||
|
status = 500
|
||||||
|
code = "internal_error"
|
||||||
|
message = "评审器无法完成该请求。"
|
||||||
|
self._json(status, {"error": {"code": code, "message": message}}, head_only=head_only)
|
||||||
|
|
||||||
|
def _static(self, path: str, *, head_only: bool) -> None:
|
||||||
|
requested = "index.html" if path == "/" else unquote(path[1:], encoding="utf-8", errors="strict")
|
||||||
|
if "\0" in requested:
|
||||||
|
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||||
|
candidate = static_root / requested
|
||||||
|
try:
|
||||||
|
if candidate.is_symlink():
|
||||||
|
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||||
|
resolved = candidate.resolve(strict=True)
|
||||||
|
if not resolved.is_relative_to(static_root) or not resolved.is_file():
|
||||||
|
raise FileNotFoundError
|
||||||
|
except (FileNotFoundError, OSError):
|
||||||
|
if Path(requested).suffix:
|
||||||
|
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||||
|
resolved = (static_root / "index.html").resolve(strict=True)
|
||||||
|
content = resolved.read_bytes()
|
||||||
|
content_type = mimetypes.guess_type(resolved.name)[0] or "application/octet-stream"
|
||||||
|
if content_type.startswith("text/") or content_type in {"application/javascript", "application/json"}:
|
||||||
|
content_type += "; charset=utf-8"
|
||||||
|
self._headers(200, content_type, len(content))
|
||||||
|
if not head_only:
|
||||||
|
self.wfile.write(content)
|
||||||
|
|
||||||
|
def _handle(self, method: str) -> None:
|
||||||
|
head_only = method == "HEAD"
|
||||||
|
if method not in {"GET", "HEAD"}:
|
||||||
|
self.send_response(405)
|
||||||
|
for name, value in _SECURITY_HEADERS.items():
|
||||||
|
self.send_header(name, value)
|
||||||
|
self.send_header("Allow", "GET, HEAD")
|
||||||
|
payload: JsonObject = {
|
||||||
|
"error": {"code": "method_not_allowed", "message": "只允许 GET 和 HEAD。"}
|
||||||
|
}
|
||||||
|
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||||
|
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||||
|
self.send_header("Content-Length", str(len(content)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(content)
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
if not _valid_local_request(self):
|
||||||
|
raise ReviewArtifactError("invalid_origin", "只接受本机同源请求。", 403)
|
||||||
|
request_path = urlsplit(self.path).path
|
||||||
|
if request_path.startswith("/api/"):
|
||||||
|
self._json(200, _api_response(repository, request_path), head_only=head_only)
|
||||||
|
else:
|
||||||
|
self._static(request_path, head_only=head_only)
|
||||||
|
except Exception as error: # the response intentionally hides unexpected implementation details
|
||||||
|
self._error(error, head_only=head_only)
|
||||||
|
|
||||||
|
def do_GET(self) -> None:
|
||||||
|
self._handle("GET")
|
||||||
|
|
||||||
|
def do_HEAD(self) -> None:
|
||||||
|
self._handle("HEAD")
|
||||||
|
|
||||||
|
def do_POST(self) -> None:
|
||||||
|
self._handle("POST")
|
||||||
|
|
||||||
|
def do_PUT(self) -> None:
|
||||||
|
self._handle("PUT")
|
||||||
|
|
||||||
|
def do_PATCH(self) -> None:
|
||||||
|
self._handle("PATCH")
|
||||||
|
|
||||||
|
def do_DELETE(self) -> None:
|
||||||
|
self._handle("DELETE")
|
||||||
|
|
||||||
|
def do_OPTIONS(self) -> None:
|
||||||
|
self._handle("OPTIONS")
|
||||||
|
|
||||||
|
return ReviewRequestHandler
|
||||||
|
|
||||||
|
|
||||||
|
def create_server(
|
||||||
|
repository: ReviewArtifacts,
|
||||||
|
static_root: Path,
|
||||||
|
*,
|
||||||
|
port: int = 0,
|
||||||
|
) -> ReviewerHttpServer:
|
||||||
|
"""Create, but do not start, the loopback reviewer server."""
|
||||||
|
index = static_root / "index.html"
|
||||||
|
if not index.is_file():
|
||||||
|
raise ReviewArtifactError("missing_build", "未找到前端构建结果，请先运行 npm run build。", 400)
|
||||||
|
return ReviewerHttpServer(("127.0.0.1", port), _handler_factory(repository, static_root.resolve()))
|
||||||
|
|
||||||
|
|
||||||
|
def _arguments(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="只读查看一次 mdpolish 本地清洗运行。")
|
||||||
|
parser.add_argument("--run-dir", required=True, help="一次已发布运行目录的绝对或相对路径")
|
||||||
|
parser.add_argument("--port", type=int, default=0, help="本机端口；默认 0 表示自动选择")
|
||||||
|
arguments = parser.parse_args(argv)
|
||||||
|
if arguments.port < 0 or arguments.port > 65535:
|
||||||
|
parser.error("--port 必须在 0 到 65535 之间")
|
||||||
|
return arguments
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> None:
|
||||||
|
arguments = _arguments(argv)
|
||||||
|
repository = ReviewArtifacts(arguments.run_dir)
|
||||||
|
static_root = Path(__file__).resolve().parents[1] / "dist"
|
||||||
|
try:
|
||||||
|
server = create_server(repository, static_root, port=arguments.port)
|
||||||
|
except OSError as error:
|
||||||
|
raise ReviewArtifactError("server_error", "无法启动本地评审服务。", 500) from error
|
||||||
|
port = server.server_address[1]
|
||||||
|
print(
|
||||||
|
f"mdpolish 评审器已启动：http://127.0.0.1:{port}（{repository.run_id}，"
|
||||||
|
f"{len(repository.documents)} 份文档）",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
server.serve_forever()
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
server.server_close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except ReviewArtifactError as error:
|
||||||
|
raise SystemExit(f"评审器启动失败：{error}") from None
|
||||||
@@ -0,0 +1,691 @@
|
|||||||
|
"""Strict adapters from published mdpolish artifacts to the reviewer API."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from hashlib import sha256
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import NoReturn, TypeAlias, cast
|
||||||
|
|
||||||
|
from mdpolish._artifact_replay import (
|
||||||
|
LocatedChange,
|
||||||
|
ReplayChange,
|
||||||
|
ReplayComponent,
|
||||||
|
ReplayError,
|
||||||
|
ReplayResult,
|
||||||
|
replay_change_chain,
|
||||||
|
)
|
||||||
|
|
||||||
|
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||||
|
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||||
|
_STATUSES = {"success", "failed", "unstable"}
|
||||||
|
_ERROR_STAGES = {"transform", "final_review"}
|
||||||
|
|
||||||
|
|
||||||
|
class ReviewArtifactError(ValueError):
|
||||||
|
"""Published artifacts cannot be exposed as a trustworthy review response."""
|
||||||
|
|
||||||
|
def __init__(self, code: str, message: str, http_status: int = 422) -> None:
|
||||||
|
super().__init__(message)
|
||||||
|
self.code = code
|
||||||
|
self.http_status = http_status
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ComponentRecord:
|
||||||
|
component_id: str
|
||||||
|
version: str
|
||||||
|
parameters: JsonValue
|
||||||
|
applicability: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ManifestDocument:
|
||||||
|
document_id: str
|
||||||
|
source_label: str
|
||||||
|
status: str
|
||||||
|
input_sha256: str
|
||||||
|
current_sha256: str
|
||||||
|
change_count: int
|
||||||
|
result_path: str
|
||||||
|
cleaned_path: str | None
|
||||||
|
diff_path: str | None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class LocatorDocument:
|
||||||
|
document_id: str
|
||||||
|
source_path: Path
|
||||||
|
input_sha256: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ChangeRecord:
|
||||||
|
replay: ReplayChange
|
||||||
|
reason: str
|
||||||
|
recorded_line: int
|
||||||
|
recorded_column: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ResultRecord:
|
||||||
|
status: str
|
||||||
|
input_sha256: str
|
||||||
|
current_sha256: str
|
||||||
|
changes: tuple[ChangeRecord, ...]
|
||||||
|
errors: tuple[JsonObject, ...]
|
||||||
|
residual_proposals: tuple[JsonObject, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class DocumentRecord:
|
||||||
|
manifest: ManifestDocument
|
||||||
|
result: ResultRecord
|
||||||
|
locator: LocatorDocument | None
|
||||||
|
|
||||||
|
|
||||||
|
def _fail(code: str, message: str, http_status: int = 422) -> NoReturn:
|
||||||
|
raise ReviewArtifactError(code, message, http_status)
|
||||||
|
|
||||||
|
|
||||||
|
def _object(value: object, label: str) -> dict[str, object]:
|
||||||
|
if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
|
||||||
|
_fail("invalid_artifact", f"{label} 必须是 JSON 对象。")
|
||||||
|
return cast(dict[str, object], value)
|
||||||
|
|
||||||
|
|
||||||
|
def _array(value: object, label: str) -> list[object]:
|
||||||
|
if not isinstance(value, list):
|
||||||
|
_fail("invalid_artifact", f"{label} 必须是数组。")
|
||||||
|
return cast(list[object], value)
|
||||||
|
|
||||||
|
|
||||||
|
def _string(value: object, label: str, *, allow_empty: bool = False) -> str:
|
||||||
|
if not isinstance(value, str) or "\0" in value or (not allow_empty and not value):
|
||||||
|
_fail("invalid_artifact", f"{label} 必须是字符串。")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _integer(value: object, label: str, *, minimum: int = 0) -> int:
|
||||||
|
if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
|
||||||
|
_fail("invalid_artifact", f"{label} 必须是不小于 {minimum} 的整数。")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _hash(value: object, label: str) -> str:
|
||||||
|
digest = _string(value, label)
|
||||||
|
if len(digest) != 64 or any(character not in "0123456789abcdef" for character in digest):
|
||||||
|
_fail("invalid_artifact", f"{label} 必须是小写 SHA-256 摘要。")
|
||||||
|
return digest
|
||||||
|
|
||||||
|
|
||||||
|
def _status(value: object, label: str) -> str:
|
||||||
|
status = _string(value, label)
|
||||||
|
if status not in _STATUSES:
|
||||||
|
_fail("invalid_artifact", f"{label} 不是已知状态。")
|
||||||
|
return status
|
||||||
|
|
||||||
|
|
||||||
|
def _json_value(value: object, label: str) -> JsonValue:
|
||||||
|
if value is None or isinstance(value, str | int | float | bool):
|
||||||
|
return value
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [_json_value(item, label) for item in value]
|
||||||
|
if isinstance(value, dict) and all(isinstance(key, str) for key in value):
|
||||||
|
return {cast(str, key): _json_value(item, label) for key, item in value.items()}
|
||||||
|
_fail("invalid_artifact", f"{label} 包含不支持的 JSON 值。")
|
||||||
|
raise AssertionError("unreachable")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_json(path: Path, label: str) -> dict[str, object]:
|
||||||
|
try:
|
||||||
|
content = path.read_bytes()
|
||||||
|
except OSError:
|
||||||
|
_fail("missing_artifact", f"无法读取{label}。")
|
||||||
|
if content.startswith(b"\xef\xbb\xbf"):
|
||||||
|
_fail("invalid_artifact", f"{label} 不能包含 UTF-8 BOM。")
|
||||||
|
if not content.endswith(b"\n"):
|
||||||
|
_fail("invalid_artifact", f"{label} 必须以换行结尾。")
|
||||||
|
try:
|
||||||
|
text = content.decode("utf-8", errors="strict")
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
_fail("invalid_utf8", f"{label} 不是严格 UTF-8。")
|
||||||
|
try:
|
||||||
|
payload = cast(
|
||||||
|
object,
|
||||||
|
json.loads(
|
||||||
|
text,
|
||||||
|
parse_constant=lambda _value: _fail(
|
||||||
|
"invalid_artifact", f"{label} 不能包含非有限数值。"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
_fail("invalid_artifact", f"{label} 不是合法 JSON。")
|
||||||
|
return _object(payload, label)
|
||||||
|
|
||||||
|
|
||||||
|
def _read_markdown(path: Path, expected_hash: str, label: str, error_code: str) -> str:
|
||||||
|
try:
|
||||||
|
if path.is_symlink() or not path.is_file():
|
||||||
|
_fail(error_code, f"{label}不是普通文件。", 409)
|
||||||
|
content = path.read_bytes()
|
||||||
|
except OSError:
|
||||||
|
_fail(error_code, f"无法读取{label}。", 409)
|
||||||
|
if sha256(content).hexdigest() != expected_hash:
|
||||||
|
_fail("hash_mismatch", f"{label}的内容哈希已经变化。", 409)
|
||||||
|
try:
|
||||||
|
return content.decode("utf-8", errors="strict")
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
_fail("invalid_utf8", f"{label}不是严格 UTF-8。", 409)
|
||||||
|
raise AssertionError("unreachable")
|
||||||
|
|
||||||
|
|
||||||
|
def _run_path(run_directory: Path, relative_path: str, label: str) -> Path:
|
||||||
|
relative = Path(relative_path)
|
||||||
|
if relative.is_absolute() or "\0" in relative_path:
|
||||||
|
_fail("unsafe_path", f"{label}必须是运行目录内的相对路径。")
|
||||||
|
unresolved = run_directory / relative
|
||||||
|
try:
|
||||||
|
if unresolved.is_symlink() or not unresolved.is_file():
|
||||||
|
_fail("missing_artifact", f"无法读取{label}。")
|
||||||
|
resolved = unresolved.resolve(strict=True)
|
||||||
|
except OSError:
|
||||||
|
_fail("missing_artifact", f"无法读取{label}。")
|
||||||
|
if not resolved.is_relative_to(run_directory):
|
||||||
|
_fail("unsafe_path", f"{label}越过了运行目录边界。")
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
|
def _component(value: object, position: int) -> ComponentRecord:
|
||||||
|
item = _object(value, f"pipeline.components[{position}]")
|
||||||
|
return ComponentRecord(
|
||||||
|
component_id=_string(item.get("component_id"), "component_id"),
|
||||||
|
version=_string(item.get("version"), "component version"),
|
||||||
|
parameters=_json_value(item.get("parameters"), "component parameters"),
|
||||||
|
applicability=_string(item.get("applicability"), "component applicability"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _manifest_document(value: object, position: int) -> ManifestDocument:
|
||||||
|
item = _object(value, f"manifest.documents[{position}]")
|
||||||
|
document_id = _string(item.get("document_id"), "document_id")
|
||||||
|
status = _status(item.get("status"), "document status")
|
||||||
|
result_path = _string(item.get("result_path"), "result_path")
|
||||||
|
cleaned_value = item.get("cleaned_path")
|
||||||
|
diff_value = item.get("diff_path")
|
||||||
|
cleaned_path = None if cleaned_value is None else _string(cleaned_value, "cleaned_path")
|
||||||
|
diff_path = None if diff_value is None else _string(diff_value, "diff_path")
|
||||||
|
base = f"documents/{document_id}"
|
||||||
|
if (
|
||||||
|
result_path != f"{base}/result.json"
|
||||||
|
or cleaned_path != (f"{base}/cleaned.md" if status == "success" else None)
|
||||||
|
or diff_path != (f"{base}/changes.diff" if status == "success" else None)
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "manifest 文档产物路径不符合 schema 1。")
|
||||||
|
return ManifestDocument(
|
||||||
|
document_id=document_id,
|
||||||
|
source_label=_string(item.get("source_label"), "source_label"),
|
||||||
|
status=status,
|
||||||
|
input_sha256=_hash(item.get("input_sha256"), "input_sha256"),
|
||||||
|
current_sha256=_hash(item.get("current_sha256"), "current_sha256"),
|
||||||
|
change_count=_integer(item.get("change_count"), "change_count"),
|
||||||
|
result_path=result_path,
|
||||||
|
cleaned_path=cleaned_path,
|
||||||
|
diff_path=diff_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_change(value: object, position: int) -> ChangeRecord:
|
||||||
|
item = _object(value, f"changes[{position}]")
|
||||||
|
proposal = _object(item.get("proposal_ref"), "proposal_ref")
|
||||||
|
span = _object(item.get("span"), "span")
|
||||||
|
location = _object(item.get("location"), "location")
|
||||||
|
start = _integer(span.get("start"), "span.start")
|
||||||
|
end = _integer(span.get("end"), "span.end")
|
||||||
|
if end < start:
|
||||||
|
_fail("invalid_artifact", "change span 必须满足 start <= end。")
|
||||||
|
replay = ReplayChange(
|
||||||
|
component_id=_string(item.get("component_id"), "change component_id"),
|
||||||
|
component_version=_string(item.get("component_version"), "change component_version"),
|
||||||
|
component_position=_integer(item.get("component_position"), "change component_position"),
|
||||||
|
proposal_component_position=_integer(
|
||||||
|
proposal.get("component_position"), "proposal component_position"
|
||||||
|
),
|
||||||
|
proposal_snapshot_sha256=_hash(proposal.get("snapshot_sha256"), "proposal snapshot_sha256"),
|
||||||
|
proposal_index=_integer(proposal.get("proposal_index"), "proposal_index"),
|
||||||
|
edit_index=_integer(item.get("edit_index"), "edit_index"),
|
||||||
|
start=start,
|
||||||
|
end=end,
|
||||||
|
before=_string(item.get("before"), "before", allow_empty=True),
|
||||||
|
after=_string(item.get("after"), "after", allow_empty=True),
|
||||||
|
before_sha256=_hash(item.get("before_sha256"), "before_sha256"),
|
||||||
|
after_sha256=_hash(item.get("after_sha256"), "after_sha256"),
|
||||||
|
)
|
||||||
|
return ChangeRecord(
|
||||||
|
replay=replay,
|
||||||
|
reason=_string(item.get("reason"), "change reason"),
|
||||||
|
recorded_line=_integer(location.get("line"), "location.line", minimum=1),
|
||||||
|
recorded_column=_integer(location.get("column"), "location.column", minimum=1),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_error(value: object, position: int) -> JsonObject:
|
||||||
|
item = _object(value, f"errors[{position}]")
|
||||||
|
stage = _string(item.get("stage"), "error stage")
|
||||||
|
if stage not in _ERROR_STAGES:
|
||||||
|
_fail("invalid_artifact", "error stage 不是已知阶段。")
|
||||||
|
return {
|
||||||
|
"component_id": _string(item.get("component_id"), "error component_id"),
|
||||||
|
"component_version": _string(item.get("component_version"), "error component_version"),
|
||||||
|
"component_position": _integer(item.get("component_position"), "error component_position"),
|
||||||
|
"stage": stage,
|
||||||
|
"error_type": _string(item.get("error_type"), "error_type"),
|
||||||
|
"message": _string(item.get("message"), "error message"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_residual(value: object, position: int) -> JsonObject:
|
||||||
|
item = _object(value, f"residual_proposals[{position}]")
|
||||||
|
reference = _object(item.get("proposal_ref"), "residual proposal_ref")
|
||||||
|
proposal = _object(item.get("proposal"), "residual proposal")
|
||||||
|
component_position = _integer(item.get("component_position"), "residual component_position")
|
||||||
|
proposal_snapshot = _hash(proposal.get("snapshot_sha256"), "residual proposal snapshot_sha256")
|
||||||
|
if (
|
||||||
|
_integer(reference.get("component_position"), "residual reference component_position")
|
||||||
|
!= component_position
|
||||||
|
or _hash(reference.get("snapshot_sha256"), "residual reference snapshot_sha256")
|
||||||
|
!= proposal_snapshot
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "residual proposal_ref 与候选身份不一致。")
|
||||||
|
_integer(reference.get("proposal_index"), "residual proposal_index")
|
||||||
|
edits = _array(proposal.get("edits"), "residual proposal edits")
|
||||||
|
if not edits:
|
||||||
|
_fail("invalid_artifact", "residual proposal edits 不能为空。")
|
||||||
|
for edit_position, edit_value in enumerate(edits):
|
||||||
|
edit = _object(edit_value, f"residual edit[{edit_position}]")
|
||||||
|
span = _object(edit.get("span"), "residual edit span")
|
||||||
|
if _hash(edit.get("snapshot_sha256"), "residual edit snapshot_sha256") != proposal_snapshot:
|
||||||
|
_fail("invalid_artifact", "residual edit 与候选快照不一致。")
|
||||||
|
start = _integer(span.get("start"), "residual span.start")
|
||||||
|
end = _integer(span.get("end"), "residual span.end")
|
||||||
|
expected = _string(edit.get("expected_text"), "residual expected_text", allow_empty=True)
|
||||||
|
_string(edit.get("replacement"), "residual replacement", allow_empty=True)
|
||||||
|
if end < start or len(expected) != end - start:
|
||||||
|
_fail("invalid_artifact", "residual edit 范围与 expected_text 不一致。")
|
||||||
|
return {
|
||||||
|
"component_id": _string(item.get("component_id"), "residual component_id"),
|
||||||
|
"component_version": _string(item.get("component_version"), "residual component_version"),
|
||||||
|
"component_position": component_position,
|
||||||
|
"reason": _string(proposal.get("reason"), "residual reason"),
|
||||||
|
"edit_count": len(edits),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _component_json(component: ComponentRecord, position: int, change_count: int) -> JsonObject:
|
||||||
|
return {
|
||||||
|
"component_position": position,
|
||||||
|
"component_id": component.component_id,
|
||||||
|
"version": component.version,
|
||||||
|
"parameters": component.parameters,
|
||||||
|
"applicability": component.applicability,
|
||||||
|
"change_count": change_count,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ReviewArtifacts:
|
||||||
|
"""Validated, read-only view over one explicitly selected run directory."""
|
||||||
|
|
||||||
|
def __init__(self, run_directory: str | Path) -> None:
|
||||||
|
requested = Path(run_directory)
|
||||||
|
try:
|
||||||
|
if requested.is_symlink() or not requested.is_dir():
|
||||||
|
_fail("missing_run", "指定的运行目录不存在或不是普通目录。", 400)
|
||||||
|
self.run_directory = requested.resolve(strict=True)
|
||||||
|
except OSError:
|
||||||
|
_fail("missing_run", "无法读取指定的运行目录。", 400)
|
||||||
|
|
||||||
|
manifest = _read_json(_run_path(self.run_directory, "manifest.json", "manifest.json"), "manifest.json")
|
||||||
|
if manifest.get("schema_version") != 1:
|
||||||
|
_fail("unsupported_schema", "只支持 manifest.json schema 1。", 409)
|
||||||
|
run = _object(manifest.get("run"), "manifest.run")
|
||||||
|
pipeline = _object(manifest.get("pipeline"), "manifest.pipeline")
|
||||||
|
summary = _object(manifest.get("summary"), "manifest.summary")
|
||||||
|
self.run_id = _string(run.get("run_id"), "run_id")
|
||||||
|
self.run_json: JsonObject = {
|
||||||
|
"run_id": self.run_id,
|
||||||
|
"run_date": _string(run.get("run_date"), "run_date"),
|
||||||
|
"status": _status(run.get("status"), "run status"),
|
||||||
|
"started_at_utc": _string(run.get("started_at_utc"), "started_at_utc"),
|
||||||
|
"completed_at_utc": _string(run.get("completed_at_utc"), "completed_at_utc"),
|
||||||
|
"retention_until": _string(run.get("retention_until"), "retention_until"),
|
||||||
|
}
|
||||||
|
self.components = tuple(
|
||||||
|
_component(value, position)
|
||||||
|
for position, value in enumerate(_array(pipeline.get("components"), "pipeline.components"))
|
||||||
|
)
|
||||||
|
manifest_documents = tuple(
|
||||||
|
_manifest_document(value, position)
|
||||||
|
for position, value in enumerate(_array(manifest.get("documents"), "manifest.documents"))
|
||||||
|
)
|
||||||
|
if len({item.document_id for item in manifest_documents}) != len(manifest_documents):
|
||||||
|
_fail("invalid_artifact", "manifest 中的 document_id 必须唯一。")
|
||||||
|
self.summary_json: JsonObject = {
|
||||||
|
"document_count": _integer(summary.get("document_count"), "summary.document_count"),
|
||||||
|
"success_count": _integer(summary.get("success_count"), "summary.success_count"),
|
||||||
|
"failed_count": _integer(summary.get("failed_count"), "summary.failed_count"),
|
||||||
|
"unstable_count": _integer(summary.get("unstable_count"), "summary.unstable_count"),
|
||||||
|
"change_count": _integer(summary.get("change_count"), "summary.change_count"),
|
||||||
|
}
|
||||||
|
expected_summary: JsonObject = {
|
||||||
|
"document_count": len(manifest_documents),
|
||||||
|
"success_count": sum(item.status == "success" for item in manifest_documents),
|
||||||
|
"failed_count": sum(item.status == "failed" for item in manifest_documents),
|
||||||
|
"unstable_count": sum(item.status == "unstable" for item in manifest_documents),
|
||||||
|
"change_count": sum(item.change_count for item in manifest_documents),
|
||||||
|
}
|
||||||
|
if self.summary_json != expected_summary:
|
||||||
|
_fail("invalid_artifact", "manifest 汇总与文档索引不一致。")
|
||||||
|
expected_status = (
|
||||||
|
"failed"
|
||||||
|
if expected_summary["failed_count"]
|
||||||
|
else "unstable"
|
||||||
|
if expected_summary["unstable_count"]
|
||||||
|
else "success"
|
||||||
|
)
|
||||||
|
if self.run_json["status"] != expected_status:
|
||||||
|
_fail("invalid_artifact", "manifest 运行状态与文档状态不一致。")
|
||||||
|
|
||||||
|
locator_path = self.run_directory / "review-locator.json"
|
||||||
|
locators: tuple[LocatorDocument, ...] | None = None
|
||||||
|
self.original_run_location_changed = False
|
||||||
|
if locator_path.exists():
|
||||||
|
locator = _read_json(
|
||||||
|
_run_path(self.run_directory, "review-locator.json", "review-locator.json"),
|
||||||
|
"review-locator.json",
|
||||||
|
)
|
||||||
|
if locator.get("schema_version") != 1:
|
||||||
|
_fail("unsupported_schema", "只支持 review-locator.json schema 1。", 409)
|
||||||
|
locator_run = _object(locator.get("run"), "review locator run")
|
||||||
|
if (
|
||||||
|
_string(locator_run.get("run_id"), "locator run_id") != self.run_id
|
||||||
|
or locator_run.get("manifest_path") != "manifest.json"
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "review locator 与 manifest 运行身份不一致。")
|
||||||
|
recorded_directory_text = _string(locator_run.get("run_directory"), "locator run_directory")
|
||||||
|
recorded_directory = Path(recorded_directory_text)
|
||||||
|
if not recorded_directory.is_absolute() or str(recorded_directory) != str(recorded_directory.resolve()):
|
||||||
|
_fail("invalid_artifact", "locator run_directory 必须是绝对解析路径。")
|
||||||
|
self.original_run_location_changed = recorded_directory != self.run_directory
|
||||||
|
parsed_locators: list[LocatorDocument] = []
|
||||||
|
for position, value in enumerate(_array(locator.get("documents"), "locator documents")):
|
||||||
|
item = _object(value, f"locator.documents[{position}]")
|
||||||
|
source_text = _string(item.get("source_path"), "source_path")
|
||||||
|
source_path = Path(source_text)
|
||||||
|
if not source_path.is_absolute() or str(source_path) != str(source_path.resolve()):
|
||||||
|
_fail("invalid_artifact", "source_path 必须是绝对解析路径。")
|
||||||
|
parsed_locators.append(
|
||||||
|
LocatorDocument(
|
||||||
|
document_id=_string(item.get("document_id"), "locator document_id"),
|
||||||
|
source_path=source_path,
|
||||||
|
input_sha256=_hash(item.get("input_sha256"), "locator input_sha256"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
locators = tuple(parsed_locators)
|
||||||
|
if len(locators) != len(manifest_documents):
|
||||||
|
_fail("invalid_artifact", "review locator 文档数量与 manifest 不一致。")
|
||||||
|
for indexed, located in zip(manifest_documents, locators, strict=True):
|
||||||
|
if indexed.document_id != located.document_id or indexed.input_sha256 != located.input_sha256:
|
||||||
|
_fail("invalid_artifact", "review locator 文档身份与 manifest 不一致。")
|
||||||
|
|
||||||
|
records: list[DocumentRecord] = []
|
||||||
|
for position, manifest_document in enumerate(manifest_documents):
|
||||||
|
result = self._parse_result(manifest_document)
|
||||||
|
records.append(
|
||||||
|
DocumentRecord(
|
||||||
|
manifest=manifest_document,
|
||||||
|
result=result,
|
||||||
|
locator=None if locators is None else locators[position],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.documents = tuple(records)
|
||||||
|
self.documents_by_id = {item.manifest.document_id: item for item in self.documents}
|
||||||
|
|
||||||
|
def _parse_result(self, manifest: ManifestDocument) -> ResultRecord:
|
||||||
|
payload = _read_json(_run_path(self.run_directory, manifest.result_path, "result.json"), "result.json")
|
||||||
|
if payload.get("schema_version") != 1:
|
||||||
|
_fail("unsupported_schema", "只支持 result.json schema 1。", 409)
|
||||||
|
document = _object(payload.get("document"), "result.document")
|
||||||
|
if (
|
||||||
|
_string(document.get("document_id"), "result document_id") != manifest.document_id
|
||||||
|
or _string(document.get("source_label"), "result source_label") != manifest.source_label
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "result 与 manifest 文档身份不一致。")
|
||||||
|
status = _status(payload.get("status"), "result status")
|
||||||
|
input_hash = _hash(payload.get("input_sha256"), "result input_sha256")
|
||||||
|
current_hash = _hash(payload.get("current_sha256"), "result current_sha256")
|
||||||
|
changes = tuple(
|
||||||
|
_parse_change(value, position)
|
||||||
|
for position, value in enumerate(_array(payload.get("changes"), "result changes"))
|
||||||
|
)
|
||||||
|
errors = tuple(
|
||||||
|
_parse_error(value, position)
|
||||||
|
for position, value in enumerate(_array(payload.get("errors"), "result errors"))
|
||||||
|
)
|
||||||
|
residuals = tuple(
|
||||||
|
_parse_residual(value, position)
|
||||||
|
for position, value in enumerate(_array(payload.get("residual_proposals"), "result residual_proposals"))
|
||||||
|
)
|
||||||
|
output = _object(payload.get("output"), "result output")
|
||||||
|
cleaned = output.get("cleaned_path")
|
||||||
|
diff = output.get("diff_path")
|
||||||
|
expected_cleaned: object = "cleaned.md" if status == "success" else None
|
||||||
|
expected_diff: object = "changes.diff" if status == "success" else None
|
||||||
|
if cleaned != expected_cleaned or diff != expected_diff:
|
||||||
|
_fail("invalid_artifact", "result 状态与输出路径不一致。")
|
||||||
|
if (
|
||||||
|
status != manifest.status
|
||||||
|
or input_hash != manifest.input_sha256
|
||||||
|
or current_hash != manifest.current_sha256
|
||||||
|
or len(changes) != manifest.change_count
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "result 与 manifest 文档索引不一致。")
|
||||||
|
if status == "success" and (errors or residuals):
|
||||||
|
_fail("invalid_artifact", "success 文档不能包含错误或残留候选。")
|
||||||
|
if status == "failed" and (not errors or residuals):
|
||||||
|
_fail("invalid_artifact", "failed 文档的错误或残留状态不合法。")
|
||||||
|
if status == "unstable" and (errors or not residuals):
|
||||||
|
_fail("invalid_artifact", "unstable 文档的错误或残留状态不合法。")
|
||||||
|
for change in changes:
|
||||||
|
position = change.replay.component_position
|
||||||
|
if position >= len(self.components):
|
||||||
|
_fail("invalid_artifact", "change 没有对应的流水线组件。")
|
||||||
|
component = self.components[position]
|
||||||
|
if (
|
||||||
|
change.replay.component_id != component.component_id
|
||||||
|
or change.replay.component_version != component.version
|
||||||
|
):
|
||||||
|
_fail("invalid_artifact", "change 身份与流水线组件不一致。")
|
||||||
|
for evidence in (*errors, *residuals):
|
||||||
|
position_value = evidence["component_position"]
|
||||||
|
component_id = evidence["component_id"]
|
||||||
|
component_version = evidence["component_version"]
|
||||||
|
if not isinstance(position_value, int) or position_value >= len(self.components):
|
||||||
|
_fail("invalid_artifact", "审计证据没有对应的流水线组件。")
|
||||||
|
component = self.components[position_value]
|
||||||
|
if component_id != component.component_id or component_version != component.version:
|
||||||
|
_fail("invalid_artifact", "审计证据身份与流水线组件不一致。")
|
||||||
|
if manifest.diff_path is not None:
|
||||||
|
_run_path(self.run_directory, manifest.diff_path, "changes.diff")
|
||||||
|
return ResultRecord(status, input_hash, current_hash, changes, errors, residuals)
|
||||||
|
|
||||||
|
def _source(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||||
|
if record.locator is None:
|
||||||
|
return None, "这次历史运行没有 review-locator.json，无法定位完整原文。"
|
||||||
|
try:
|
||||||
|
return (
|
||||||
|
_read_markdown(
|
||||||
|
record.locator.source_path,
|
||||||
|
record.manifest.input_sha256,
|
||||||
|
"原文",
|
||||||
|
"unavailable_source",
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
except ReviewArtifactError as error:
|
||||||
|
return None, str(error)
|
||||||
|
|
||||||
|
def _cleaned(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||||
|
if record.manifest.status != "success" or record.manifest.cleaned_path is None:
|
||||||
|
return None, None
|
||||||
|
try:
|
||||||
|
path = _run_path(self.run_directory, record.manifest.cleaned_path, "cleaned.md")
|
||||||
|
return (
|
||||||
|
_read_markdown(path, record.manifest.current_sha256, "清洗结果", "missing_artifact"),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
except ReviewArtifactError as error:
|
||||||
|
return None, str(error)
|
||||||
|
|
||||||
|
def _replay(self, record: DocumentRecord, original: str, cleaned: str | None) -> ReplayResult:
|
||||||
|
try:
|
||||||
|
replayed = replay_change_chain(
|
||||||
|
input_markdown=original,
|
||||||
|
input_sha256=record.result.input_sha256,
|
||||||
|
components=tuple(ReplayComponent(item.component_id, item.version) for item in self.components),
|
||||||
|
changes=tuple(item.replay for item in record.result.changes),
|
||||||
|
current_sha256=record.result.current_sha256,
|
||||||
|
current_markdown=cleaned,
|
||||||
|
include_zero_change_stages=record.result.status == "success",
|
||||||
|
)
|
||||||
|
except ReplayError as error:
|
||||||
|
raise ReviewArtifactError("untrusted_replay", f"产物无法可信重放：{error}", 409) from error
|
||||||
|
for recorded, located in zip(record.result.changes, replayed.changes, strict=True):
|
||||||
|
if recorded.recorded_line != located.line or recorded.recorded_column != located.column:
|
||||||
|
_fail("untrusted_replay", "产物记录的位置与重放快照不一致。", 409)
|
||||||
|
return replayed
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _summary_json(
|
||||||
|
record: DocumentRecord,
|
||||||
|
original: str | None,
|
||||||
|
source_error: str | None,
|
||||||
|
cleaned: str | None,
|
||||||
|
output_error: str | None,
|
||||||
|
) -> JsonObject:
|
||||||
|
return {
|
||||||
|
"document_id": record.manifest.document_id,
|
||||||
|
"source_label": record.manifest.source_label,
|
||||||
|
"status": record.manifest.status,
|
||||||
|
"input_sha256": record.manifest.input_sha256,
|
||||||
|
"current_sha256": record.manifest.current_sha256,
|
||||||
|
"change_count": record.manifest.change_count,
|
||||||
|
"source_available": original is not None,
|
||||||
|
"output_available": cleaned is not None,
|
||||||
|
"availability_error": source_error or output_error,
|
||||||
|
}
|
||||||
|
|
||||||
|
def _summary(self, record: DocumentRecord) -> JsonObject:
|
||||||
|
original, source_error = self._source(record)
|
||||||
|
cleaned, output_error = self._cleaned(record)
|
||||||
|
return self._summary_json(record, original, source_error, cleaned, output_error)
|
||||||
|
|
||||||
|
def _component_summaries(self, changes: tuple[ChangeRecord, ...]) -> list[JsonValue]:
|
||||||
|
counts = [0] * len(self.components)
|
||||||
|
for change in changes:
|
||||||
|
if 0 <= change.replay.component_position < len(counts):
|
||||||
|
counts[change.replay.component_position] += 1
|
||||||
|
return [_component_json(item, position, counts[position]) for position, item in enumerate(self.components)]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _change_json(recorded: ChangeRecord, located: LocatedChange | None) -> JsonObject:
|
||||||
|
change = recorded.replay
|
||||||
|
return {
|
||||||
|
"component_id": change.component_id,
|
||||||
|
"component_version": change.component_version,
|
||||||
|
"component_position": change.component_position,
|
||||||
|
"proposal_ref": {
|
||||||
|
"component_position": change.proposal_component_position,
|
||||||
|
"snapshot_sha256": change.proposal_snapshot_sha256,
|
||||||
|
"proposal_index": change.proposal_index,
|
||||||
|
},
|
||||||
|
"edit_index": change.edit_index,
|
||||||
|
"reason": recorded.reason,
|
||||||
|
"location": {
|
||||||
|
"line": recorded.recorded_line if located is None else located.line,
|
||||||
|
"column": recorded.recorded_column if located is None else located.column,
|
||||||
|
},
|
||||||
|
"editor_range": (
|
||||||
|
None if located is None else {"start": located.editor_start, "end": located.editor_end}
|
||||||
|
),
|
||||||
|
"before": change.before,
|
||||||
|
"after": change.after,
|
||||||
|
}
|
||||||
|
|
||||||
|
def run_summary(self) -> JsonObject:
|
||||||
|
all_changes = tuple(change for document in self.documents for change in document.result.changes)
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": self.run_json,
|
||||||
|
"components": self._component_summaries(all_changes),
|
||||||
|
"documents": [self._summary(item) for item in self.documents],
|
||||||
|
"summary": self.summary_json,
|
||||||
|
"original_run_location_changed": self.original_run_location_changed,
|
||||||
|
}
|
||||||
|
|
||||||
|
def _document(self, document_id: str) -> DocumentRecord:
|
||||||
|
record = self.documents_by_id.get(document_id)
|
||||||
|
if record is None:
|
||||||
|
_fail("unknown_document", "文档不存在。", 404)
|
||||||
|
return record
|
||||||
|
|
||||||
|
def document_comparison(self, document_id: str) -> JsonObject:
|
||||||
|
record = self._document(document_id)
|
||||||
|
original, source_error = self._source(record)
|
||||||
|
cleaned, output_error = self._cleaned(record)
|
||||||
|
replayed = None if original is None else self._replay(record, original, cleaned)
|
||||||
|
located = () if replayed is None else replayed.changes
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"document": self._summary_json(record, original, source_error, cleaned, output_error),
|
||||||
|
"components": self._component_summaries(record.result.changes),
|
||||||
|
"original_markdown": original,
|
||||||
|
"cleaned_markdown": cleaned,
|
||||||
|
"changes": [
|
||||||
|
self._change_json(change, located[position] if position < len(located) else None)
|
||||||
|
for position, change in enumerate(record.result.changes)
|
||||||
|
],
|
||||||
|
"errors": list(record.result.errors),
|
||||||
|
"residual_proposals": list(record.result.residual_proposals),
|
||||||
|
}
|
||||||
|
|
||||||
|
def component_stage(self, document_id: str, component_position: int) -> JsonObject:
|
||||||
|
record = self._document(document_id)
|
||||||
|
if record.manifest.status != "success":
|
||||||
|
_fail("stage_unavailable", "只有 success 文档具有完整组件阶段。", 409)
|
||||||
|
if component_position < 0 or component_position >= len(self.components):
|
||||||
|
_fail("unknown_component", "组件位置不存在。", 404)
|
||||||
|
original, source_error = self._source(record)
|
||||||
|
cleaned, output_error = self._cleaned(record)
|
||||||
|
if original is None or cleaned is None:
|
||||||
|
_fail("comparison_unavailable", source_error or output_error or "组件阶段不可用。", 409)
|
||||||
|
replayed = self._replay(record, original, cleaned)
|
||||||
|
stage = replayed.stages[component_position]
|
||||||
|
component = self.components[component_position]
|
||||||
|
recorded_changes = tuple(
|
||||||
|
item for item in record.result.changes if item.replay.component_position == component_position
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"document_id": record.manifest.document_id,
|
||||||
|
"component": _component_json(component, component_position, len(stage.changes)),
|
||||||
|
"before_sha256": stage.before_sha256,
|
||||||
|
"after_sha256": stage.after_sha256,
|
||||||
|
"before_markdown": stage.before_markdown,
|
||||||
|
"after_markdown": stage.after_markdown,
|
||||||
|
"changes": [
|
||||||
|
self._change_json(change, located)
|
||||||
|
for change, located in zip(recorded_changes, stage.changes, strict=True)
|
||||||
|
],
|
||||||
|
}
|
||||||
@@ -0,0 +1,378 @@
|
|||||||
|
import { lazy, Suspense, useEffect, useMemo, useState } from "react";
|
||||||
|
|
||||||
|
import type {
|
||||||
|
ChangeDetail,
|
||||||
|
ComponentStageResponse,
|
||||||
|
DocumentComparisonResponse,
|
||||||
|
RunStatus,
|
||||||
|
RunSummaryResponse,
|
||||||
|
} from "../shared/api.js";
|
||||||
|
import { fetchComponentStage, fetchDocument, fetchRun } from "./api-client.js";
|
||||||
|
|
||||||
|
const DiffView = lazy(async () => {
|
||||||
|
const module = await import("./DiffView.js");
|
||||||
|
return { default: module.DiffView };
|
||||||
|
});
|
||||||
|
|
||||||
|
interface AsyncState<T> {
|
||||||
|
loading: boolean;
|
||||||
|
value: T | null;
|
||||||
|
error: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const emptyState = <T,>(): AsyncState<T> => ({ loading: true, value: null, error: null });
|
||||||
|
|
||||||
|
function statusLabel(status: RunStatus): string {
|
||||||
|
switch (status) {
|
||||||
|
case "success":
|
||||||
|
return "成功";
|
||||||
|
case "failed":
|
||||||
|
return "失败";
|
||||||
|
case "unstable":
|
||||||
|
return "不稳定";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function shortHash(hash: string): string {
|
||||||
|
return `${hash.slice(0, 8)}…${hash.slice(-6)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function ErrorPanel({ message }: { message: string }) {
|
||||||
|
return (
|
||||||
|
<div className="state-panel state-panel--error" role="alert">
|
||||||
|
<span className="eyebrow">无法显示</span>
|
||||||
|
<p>{message}</p>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function LoadingPanel() {
|
||||||
|
return (
|
||||||
|
<div className="state-panel" role="status">
|
||||||
|
<span className="loading-dot" />
|
||||||
|
<p>正在校验本地运行产物…</p>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ChangeList({
|
||||||
|
changes,
|
||||||
|
onSelect,
|
||||||
|
canJump,
|
||||||
|
}: {
|
||||||
|
changes: ChangeDetail[];
|
||||||
|
onSelect: (change: ChangeDetail) => void;
|
||||||
|
canJump: boolean;
|
||||||
|
}) {
|
||||||
|
if (changes.length === 0) {
|
||||||
|
return <p className="quiet-message">这个组件运行过，但没有修改当前文档。</p>;
|
||||||
|
}
|
||||||
|
return (
|
||||||
|
<ol className="change-list">
|
||||||
|
{changes.map((change) => (
|
||||||
|
<li key={`${change.proposal_ref.snapshot_sha256}-${change.proposal_ref.proposal_index}-${change.edit_index}`}>
|
||||||
|
<button type="button" onClick={() => onSelect(change)} disabled={!canJump}>
|
||||||
|
<span className="change-location">
|
||||||
|
第 {change.location.line} 行，第 {change.location.column} 列 · 候选
|
||||||
|
{change.proposal_ref.proposal_index + 1} / 编辑 {change.edit_index + 1}
|
||||||
|
</span>
|
||||||
|
<strong>{change.reason}</strong>
|
||||||
|
<span className="change-sample">
|
||||||
|
<del>{change.before || "∅"}</del>
|
||||||
|
<span aria-hidden="true">→</span>
|
||||||
|
<ins>{change.after || "∅"}</ins>
|
||||||
|
</span>
|
||||||
|
</button>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ol>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function App() {
|
||||||
|
const [runState, setRunState] = useState<AsyncState<RunSummaryResponse>>(emptyState);
|
||||||
|
const [selectedDocument, setSelectedDocument] = useState<string | null>(null);
|
||||||
|
const [documentState, setDocumentState] = useState<AsyncState<DocumentComparisonResponse>>({
|
||||||
|
loading: false,
|
||||||
|
value: null,
|
||||||
|
error: null,
|
||||||
|
});
|
||||||
|
const [selectedComponent, setSelectedComponent] = useState<number | null>(null);
|
||||||
|
const [stageState, setStageState] = useState<AsyncState<ComponentStageResponse>>({
|
||||||
|
loading: false,
|
||||||
|
value: null,
|
||||||
|
error: null,
|
||||||
|
});
|
||||||
|
const [focusRange, setFocusRange] = useState<{ start: number; end: number } | null>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
const controller = new AbortController();
|
||||||
|
fetchRun(controller.signal)
|
||||||
|
.then((run) => {
|
||||||
|
setRunState({ loading: false, value: run, error: null });
|
||||||
|
setSelectedDocument(run.documents[0]?.document_id ?? null);
|
||||||
|
})
|
||||||
|
.catch((error: unknown) => {
|
||||||
|
if (!controller.signal.aborted) {
|
||||||
|
setRunState({
|
||||||
|
loading: false,
|
||||||
|
value: null,
|
||||||
|
error: error instanceof Error ? error.message : "无法读取运行摘要。",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return () => controller.abort();
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
setSelectedComponent(null);
|
||||||
|
setFocusRange(null);
|
||||||
|
if (selectedDocument === null) {
|
||||||
|
setDocumentState({ loading: false, value: null, error: null });
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const controller = new AbortController();
|
||||||
|
setDocumentState(emptyState());
|
||||||
|
fetchDocument(selectedDocument, controller.signal)
|
||||||
|
.then((document) => setDocumentState({ loading: false, value: document, error: null }))
|
||||||
|
.catch((error: unknown) => {
|
||||||
|
if (!controller.signal.aborted) {
|
||||||
|
setDocumentState({
|
||||||
|
loading: false,
|
||||||
|
value: null,
|
||||||
|
error: error instanceof Error ? error.message : "无法读取文档。",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return () => controller.abort();
|
||||||
|
}, [selectedDocument]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (selectedDocument === null || selectedComponent === null) {
|
||||||
|
setStageState({ loading: false, value: null, error: null });
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const controller = new AbortController();
|
||||||
|
setStageState(emptyState());
|
||||||
|
fetchComponentStage(selectedDocument, selectedComponent, controller.signal)
|
||||||
|
.then((stage) => setStageState({ loading: false, value: stage, error: null }))
|
||||||
|
.catch((error: unknown) => {
|
||||||
|
if (!controller.signal.aborted) {
|
||||||
|
setStageState({
|
||||||
|
loading: false,
|
||||||
|
value: null,
|
||||||
|
error: error instanceof Error ? error.message : "无法读取组件阶段。",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return () => controller.abort();
|
||||||
|
}, [selectedDocument, selectedComponent]);
|
||||||
|
|
||||||
|
const visibleChanges = useMemo(() => {
|
||||||
|
const document = documentState.value;
|
||||||
|
if (document === null) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
if (selectedComponent === null) {
|
||||||
|
return document.changes;
|
||||||
|
}
|
||||||
|
return document.changes.filter((change) => change.component_position === selectedComponent);
|
||||||
|
}, [documentState.value, selectedComponent]);
|
||||||
|
|
||||||
|
const selectChange = (change: ChangeDetail): void => {
|
||||||
|
setSelectedComponent(change.component_position);
|
||||||
|
setFocusRange(change.editor_range);
|
||||||
|
};
|
||||||
|
|
||||||
|
if (runState.loading) {
|
||||||
|
return <LoadingPanel />;
|
||||||
|
}
|
||||||
|
if (runState.error !== null || runState.value === null) {
|
||||||
|
return <ErrorPanel message={runState.error ?? "运行摘要为空。"} />;
|
||||||
|
}
|
||||||
|
|
||||||
|
const run = runState.value;
|
||||||
|
const document = documentState.value;
|
||||||
|
const selectedSummary = run.documents.find((item) => item.document_id === selectedDocument);
|
||||||
|
const selectedStage = stageState.value;
|
||||||
|
const canCompare =
|
||||||
|
document?.document.status === "success" &&
|
||||||
|
document.document.source_available &&
|
||||||
|
document.document.output_available &&
|
||||||
|
document.original_markdown !== null &&
|
||||||
|
document.cleaned_markdown !== null;
|
||||||
|
const beforeText = selectedStage?.before_markdown ?? document?.original_markdown ?? "";
|
||||||
|
const afterText = selectedStage?.after_markdown ?? document?.cleaned_markdown ?? "";
|
||||||
|
const beforeLabel = selectedStage === null ? "清洗前" : `组件 ${selectedStage.component.component_position + 1} 执行前`;
|
||||||
|
const afterLabel = selectedStage === null ? "清洗后" : `组件 ${selectedStage.component.component_position + 1} 执行后`;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="app-shell">
|
||||||
|
<header className="topbar">
|
||||||
|
<div>
|
||||||
|
<span className="brand-mark">md</span>
|
||||||
|
<div>
|
||||||
|
<p className="eyebrow">本地清洗评审器</p>
|
||||||
|
<h1>{run.run.run_id}</h1>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="run-facts">
|
||||||
|
<span className={`status status--${run.run.status}`}>{statusLabel(run.run.status)}</span>
|
||||||
|
<span>{run.summary.document_count} 份文档</span>
|
||||||
|
<span>{run.summary.change_count} 条修改</span>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{run.original_run_location_changed ? (
|
||||||
|
<div className="notice" role="status">
|
||||||
|
这次运行目录已被移动；评审器使用你本次指定的目录读取产物。
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<div className="layout">
|
||||||
|
<aside className="sidebar" aria-label="运行导航">
|
||||||
|
<section>
|
||||||
|
<div className="section-heading">
|
||||||
|
<h2>文档</h2>
|
||||||
|
<span>{run.documents.length}</span>
|
||||||
|
</div>
|
||||||
|
<nav className="document-list" aria-label="文档列表">
|
||||||
|
{run.documents.map((item) => (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
key={item.document_id}
|
||||||
|
className={item.document_id === selectedDocument ? "is-active" : ""}
|
||||||
|
onClick={() => setSelectedDocument(item.document_id)}
|
||||||
|
aria-current={item.document_id === selectedDocument ? "page" : undefined}
|
||||||
|
>
|
||||||
|
<span className={`status-dot status-dot--${item.status}`} />
|
||||||
|
<span>
|
||||||
|
<strong>{item.source_label}</strong>
|
||||||
|
<small>{item.change_count} 条修改</small>
|
||||||
|
</span>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</nav>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section className="component-section">
|
||||||
|
<div className="section-heading">
|
||||||
|
<h2>组件时间线</h2>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="text-button"
|
||||||
|
onClick={() => {
|
||||||
|
setSelectedComponent(null);
|
||||||
|
setFocusRange(null);
|
||||||
|
}}
|
||||||
|
disabled={selectedComponent === null}
|
||||||
|
>
|
||||||
|
查看总结果
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<ol className="component-list">
|
||||||
|
{(document?.components ?? run.components).map((component) => (
|
||||||
|
<li key={component.component_id}>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className={component.component_position === selectedComponent ? "is-active" : ""}
|
||||||
|
onClick={() => {
|
||||||
|
setSelectedComponent(component.component_position);
|
||||||
|
setFocusRange(null);
|
||||||
|
}}
|
||||||
|
disabled={!canCompare}
|
||||||
|
>
|
||||||
|
<span className="component-index">{component.component_position + 1}</span>
|
||||||
|
<span>
|
||||||
|
<strong>{component.component_id}</strong>
|
||||||
|
<small>
|
||||||
|
v{component.version} · {component.change_count} 条
|
||||||
|
</small>
|
||||||
|
</span>
|
||||||
|
</button>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ol>
|
||||||
|
</section>
|
||||||
|
</aside>
|
||||||
|
|
||||||
|
<main className="workspace">
|
||||||
|
<section className="document-header">
|
||||||
|
<div>
|
||||||
|
<p className="eyebrow">当前文档</p>
|
||||||
|
<h2>{selectedSummary?.source_label ?? "未选择"}</h2>
|
||||||
|
</div>
|
||||||
|
{selectedSummary === undefined ? null : (
|
||||||
|
<div className="document-meta">
|
||||||
|
<span className={`status status--${selectedSummary.status}`}>
|
||||||
|
{statusLabel(selectedSummary.status)}
|
||||||
|
</span>
|
||||||
|
<span title={selectedSummary.input_sha256}>输入 {shortHash(selectedSummary.input_sha256)}</span>
|
||||||
|
<span title={selectedSummary.current_sha256}>当前 {shortHash(selectedSummary.current_sha256)}</span>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
|
||||||
|
{documentState.loading || stageState.loading ? <LoadingPanel /> : null}
|
||||||
|
{documentState.error !== null ? <ErrorPanel message={documentState.error} /> : null}
|
||||||
|
{stageState.error !== null ? <ErrorPanel message={stageState.error} /> : null}
|
||||||
|
|
||||||
|
{!documentState.loading && document !== null && document.document.status === "success" && !canCompare ? (
|
||||||
|
<ErrorPanel message={document.document.availability_error ?? "完整原文或清洗结果不可用。"} />
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{!documentState.loading && document !== null && document.document.status !== "success" ? (
|
||||||
|
<div className="diagnostic-panel">
|
||||||
|
<p className="eyebrow">没有正式清洗结果</p>
|
||||||
|
<h3>{statusLabel(document.document.status)}文档只展示审计证据</h3>
|
||||||
|
<p>
|
||||||
|
该状态不会生成 <code>cleaned.md</code>，因此这里不构造完整部分输出。
|
||||||
|
</p>
|
||||||
|
{document.errors.map((error) => (
|
||||||
|
<article key={`${error.component_position}-${error.stage}-${error.error_type}`}>
|
||||||
|
<strong>{error.error_type}</strong>
|
||||||
|
<span>{error.message}</span>
|
||||||
|
</article>
|
||||||
|
))}
|
||||||
|
{document.residual_proposals.map((proposal) => (
|
||||||
|
<article key={`${proposal.component_position}-${proposal.reason}`}>
|
||||||
|
<strong>最终复查仍有 {proposal.edit_count} 项候选</strong>
|
||||||
|
<span>{proposal.reason}</span>
|
||||||
|
</article>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{!documentState.loading && !stageState.loading && canCompare && stageState.error === null ? (
|
||||||
|
<Suspense fallback={<LoadingPanel />}>
|
||||||
|
<DiffView
|
||||||
|
before={beforeText}
|
||||||
|
after={afterText}
|
||||||
|
beforeLabel={beforeLabel}
|
||||||
|
afterLabel={afterLabel}
|
||||||
|
focusRange={focusRange}
|
||||||
|
/>
|
||||||
|
</Suspense>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{document !== null ? (
|
||||||
|
<section className="changes-panel" aria-label="修改详情">
|
||||||
|
<div className="changes-heading">
|
||||||
|
<div>
|
||||||
|
<p className="eyebrow">实际修改</p>
|
||||||
|
<h3>
|
||||||
|
{selectedComponent === null
|
||||||
|
? `全部组件 · ${visibleChanges.length} 条`
|
||||||
|
: `${document.components[selectedComponent]?.component_id ?? "组件"} · ${visibleChanges.length} 条`}
|
||||||
|
</h3>
|
||||||
|
</div>
|
||||||
|
{selectedStage === null ? null : <p>{selectedStage.component.applicability}</p>}
|
||||||
|
</div>
|
||||||
|
<ChangeList changes={visibleChanges} onSelect={selectChange} canJump={canCompare} />
|
||||||
|
</section>
|
||||||
|
) : null}
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
import { markdown } from "@codemirror/lang-markdown";
|
||||||
|
import { MergeView } from "@codemirror/merge";
|
||||||
|
import { EditorSelection, EditorState } from "@codemirror/state";
|
||||||
|
import { drawSelection, EditorView, lineNumbers } from "@codemirror/view";
|
||||||
|
import { useEffect, useRef } from "react";
|
||||||
|
|
||||||
|
interface DiffViewProps {
|
||||||
|
before: string;
|
||||||
|
after: string;
|
||||||
|
beforeLabel: string;
|
||||||
|
afterLabel: string;
|
||||||
|
focusRange?: { start: number; end: number } | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const editorTheme = EditorView.theme({
|
||||||
|
"&": {
|
||||||
|
height: "100%",
|
||||||
|
backgroundColor: "#fbfaf7",
|
||||||
|
color: "#262822",
|
||||||
|
fontSize: "13px",
|
||||||
|
},
|
||||||
|
".cm-scroller": {
|
||||||
|
fontFamily: '"SFMono-Regular", Consolas, "Liberation Mono", monospace',
|
||||||
|
lineHeight: "1.68",
|
||||||
|
},
|
||||||
|
".cm-gutters": {
|
||||||
|
backgroundColor: "#f2f0ea",
|
||||||
|
color: "#8a877e",
|
||||||
|
border: "none",
|
||||||
|
},
|
||||||
|
".cm-content": {
|
||||||
|
padding: "18px 0 36px",
|
||||||
|
},
|
||||||
|
".cm-line": {
|
||||||
|
padding: "0 14px",
|
||||||
|
},
|
||||||
|
"&.cm-focused": {
|
||||||
|
outline: "2px solid #a7b9ac",
|
||||||
|
outlineOffset: "-2px",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
const readOnlyExtensions = [
|
||||||
|
lineNumbers(),
|
||||||
|
markdown(),
|
||||||
|
EditorState.readOnly.of(true),
|
||||||
|
EditorView.editable.of(false),
|
||||||
|
drawSelection(),
|
||||||
|
EditorView.lineWrapping,
|
||||||
|
editorTheme,
|
||||||
|
];
|
||||||
|
|
||||||
|
export function DiffView({ before, after, beforeLabel, afterLabel, focusRange }: DiffViewProps) {
|
||||||
|
const host = useRef<HTMLDivElement>(null);
|
||||||
|
const merge = useRef<MergeView | null>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (host.current === null) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
const view = new MergeView({
|
||||||
|
parent: host.current,
|
||||||
|
a: { doc: before, extensions: readOnlyExtensions },
|
||||||
|
b: { doc: after, extensions: readOnlyExtensions },
|
||||||
|
orientation: "a-b",
|
||||||
|
gutter: true,
|
||||||
|
highlightChanges: true,
|
||||||
|
});
|
||||||
|
merge.current = view;
|
||||||
|
return () => {
|
||||||
|
view.destroy();
|
||||||
|
merge.current = null;
|
||||||
|
};
|
||||||
|
}, [before, after]);
|
||||||
|
|
||||||
|
// 切换组件时 stage 文本先到、旧视图上的提前滚动会被重建覆盖，所以聚焦
|
||||||
|
// 必须等 MergeView 换成目标文本之后再执行。
|
||||||
|
useEffect(() => {
|
||||||
|
const view = merge.current;
|
||||||
|
if (view === null || focusRange === null || focusRange === undefined) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const anchor = Math.min(Math.max(focusRange.start, 0), view.a.state.doc.length);
|
||||||
|
const head = Math.min(Math.max(focusRange.end, anchor), view.a.state.doc.length);
|
||||||
|
view.a.dispatch({
|
||||||
|
selection: EditorSelection.range(anchor, head),
|
||||||
|
effects: EditorView.scrollIntoView(anchor, { y: "center" }),
|
||||||
|
});
|
||||||
|
view.a.focus();
|
||||||
|
}, [focusRange, before, after]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section className="diff-shell" aria-label={`${beforeLabel}与${afterLabel}对比`}>
|
||||||
|
<div className="diff-labels" aria-hidden="true">
|
||||||
|
<span>{beforeLabel}</span>
|
||||||
|
<span>{afterLabel}</span>
|
||||||
|
</div>
|
||||||
|
<div className="diff-host" ref={host} />
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,288 @@
|
|||||||
|
import type {
|
||||||
|
ApiErrorResponse,
|
||||||
|
ComponentStageResponse,
|
||||||
|
DocumentComparisonResponse,
|
||||||
|
RunSummaryResponse,
|
||||||
|
} from "../shared/api.js";
|
||||||
|
|
||||||
|
export class ReviewerApiError extends Error {
|
||||||
|
readonly code: string;
|
||||||
|
|
||||||
|
constructor(code: string, message: string) {
|
||||||
|
super(message);
|
||||||
|
this.name = "ReviewerApiError";
|
||||||
|
this.code = code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type JsonRecord = Record<string, unknown>;
|
||||||
|
|
||||||
|
function invalid(label: string): never {
|
||||||
|
throw new ReviewerApiError("invalid_response", `本地服务返回的 ${label} 格式不正确。`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function record(value: unknown, label: string): JsonRecord {
|
||||||
|
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return value as JsonRecord;
|
||||||
|
}
|
||||||
|
|
||||||
|
function array(value: unknown, label: string): unknown[] {
|
||||||
|
if (!Array.isArray(value)) {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function string(value: unknown, label: string): string {
|
||||||
|
if (typeof value !== "string") {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function integer(value: unknown, label: string, minimum = 0): number {
|
||||||
|
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return value as number;
|
||||||
|
}
|
||||||
|
|
||||||
|
function boolean(value: unknown, label: string): boolean {
|
||||||
|
if (typeof value !== "boolean") {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function nullableString(value: unknown, label: string): string | null {
|
||||||
|
return value === null ? null : string(value, label);
|
||||||
|
}
|
||||||
|
|
||||||
|
function hash(value: unknown, label: string): string {
|
||||||
|
const digest = string(value, label);
|
||||||
|
if (!/^[0-9a-f]{64}$/.test(digest)) {
|
||||||
|
return invalid(label);
|
||||||
|
}
|
||||||
|
return digest;
|
||||||
|
}
|
||||||
|
|
||||||
|
function status(value: unknown): "success" | "failed" | "unstable" {
|
||||||
|
if (value !== "success" && value !== "failed" && value !== "unstable") {
|
||||||
|
return invalid("status");
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function component(value: unknown): RunSummaryResponse["components"][number] {
|
||||||
|
const item = record(value, "component");
|
||||||
|
return {
|
||||||
|
component_position: integer(item.component_position, "component_position"),
|
||||||
|
component_id: string(item.component_id, "component_id"),
|
||||||
|
version: string(item.version, "component version"),
|
||||||
|
parameters: item.parameters,
|
||||||
|
applicability: string(item.applicability, "component applicability"),
|
||||||
|
change_count: integer(item.change_count, "component change_count"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function documentSummary(value: unknown): RunSummaryResponse["documents"][number] {
|
||||||
|
const item = record(value, "document summary");
|
||||||
|
return {
|
||||||
|
document_id: string(item.document_id, "document_id"),
|
||||||
|
source_label: string(item.source_label, "source_label"),
|
||||||
|
status: status(item.status),
|
||||||
|
input_sha256: hash(item.input_sha256, "input_sha256"),
|
||||||
|
current_sha256: hash(item.current_sha256, "current_sha256"),
|
||||||
|
change_count: integer(item.change_count, "document change_count"),
|
||||||
|
source_available: boolean(item.source_available, "source_available"),
|
||||||
|
output_available: boolean(item.output_available, "output_available"),
|
||||||
|
availability_error: nullableString(item.availability_error, "availability_error"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function summary(value: unknown): RunSummaryResponse["summary"] {
|
||||||
|
const item = record(value, "summary");
|
||||||
|
return {
|
||||||
|
document_count: integer(item.document_count, "document_count"),
|
||||||
|
success_count: integer(item.success_count, "success_count"),
|
||||||
|
failed_count: integer(item.failed_count, "failed_count"),
|
||||||
|
unstable_count: integer(item.unstable_count, "unstable_count"),
|
||||||
|
change_count: integer(item.change_count, "change_count"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function change(value: unknown): DocumentComparisonResponse["changes"][number] {
|
||||||
|
const item = record(value, "change");
|
||||||
|
const proposal = record(item.proposal_ref, "proposal_ref");
|
||||||
|
const location = record(item.location, "location");
|
||||||
|
const editorValue = item.editor_range;
|
||||||
|
const editorRange =
|
||||||
|
editorValue === null
|
||||||
|
? null
|
||||||
|
: (() => {
|
||||||
|
const editor = record(editorValue, "editor_range");
|
||||||
|
const start = integer(editor.start, "editor_range.start");
|
||||||
|
const end = integer(editor.end, "editor_range.end");
|
||||||
|
if (end < start) {
|
||||||
|
return invalid("editor_range");
|
||||||
|
}
|
||||||
|
return { start, end };
|
||||||
|
})();
|
||||||
|
return {
|
||||||
|
component_id: string(item.component_id, "change component_id"),
|
||||||
|
component_version: string(item.component_version, "change component_version"),
|
||||||
|
component_position: integer(item.component_position, "change component_position"),
|
||||||
|
proposal_ref: {
|
||||||
|
component_position: integer(proposal.component_position, "proposal component_position"),
|
||||||
|
snapshot_sha256: hash(proposal.snapshot_sha256, "proposal snapshot_sha256"),
|
||||||
|
proposal_index: integer(proposal.proposal_index, "proposal_index"),
|
||||||
|
},
|
||||||
|
edit_index: integer(item.edit_index, "edit_index"),
|
||||||
|
reason: string(item.reason, "reason"),
|
||||||
|
location: {
|
||||||
|
line: integer(location.line, "location.line", 1),
|
||||||
|
column: integer(location.column, "location.column", 1),
|
||||||
|
},
|
||||||
|
editor_range: editorRange,
|
||||||
|
before: string(item.before, "before"),
|
||||||
|
after: string(item.after, "after"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function runError(value: unknown): DocumentComparisonResponse["errors"][number] {
|
||||||
|
const item = record(value, "run error");
|
||||||
|
const stage = item.stage;
|
||||||
|
if (stage !== "transform" && stage !== "final_review") {
|
||||||
|
return invalid("error stage");
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
component_id: string(item.component_id, "error component_id"),
|
||||||
|
component_version: string(item.component_version, "error component_version"),
|
||||||
|
component_position: integer(item.component_position, "error component_position"),
|
||||||
|
stage,
|
||||||
|
error_type: string(item.error_type, "error_type"),
|
||||||
|
message: string(item.message, "error message"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function residual(value: unknown): DocumentComparisonResponse["residual_proposals"][number] {
|
||||||
|
const item = record(value, "residual proposal");
|
||||||
|
return {
|
||||||
|
component_id: string(item.component_id, "residual component_id"),
|
||||||
|
component_version: string(item.component_version, "residual component_version"),
|
||||||
|
component_position: integer(item.component_position, "residual component_position"),
|
||||||
|
reason: string(item.reason, "residual reason"),
|
||||||
|
edit_count: integer(item.edit_count, "residual edit_count", 1),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseRun(value: unknown): RunSummaryResponse {
|
||||||
|
const payload = record(value, "run response");
|
||||||
|
if (payload.schema_version !== 1) {
|
||||||
|
return invalid("run schema_version");
|
||||||
|
}
|
||||||
|
const run = record(payload.run, "run");
|
||||||
|
return {
|
||||||
|
schema_version: 1,
|
||||||
|
run: {
|
||||||
|
run_id: string(run.run_id, "run_id"),
|
||||||
|
run_date: string(run.run_date, "run_date"),
|
||||||
|
status: status(run.status),
|
||||||
|
started_at_utc: string(run.started_at_utc, "started_at_utc"),
|
||||||
|
completed_at_utc: string(run.completed_at_utc, "completed_at_utc"),
|
||||||
|
retention_until: string(run.retention_until, "retention_until"),
|
||||||
|
},
|
||||||
|
components: array(payload.components, "components").map(component),
|
||||||
|
documents: array(payload.documents, "documents").map(documentSummary),
|
||||||
|
summary: summary(payload.summary),
|
||||||
|
original_run_location_changed: boolean(
|
||||||
|
payload.original_run_location_changed,
|
||||||
|
"original_run_location_changed",
|
||||||
|
),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseDocument(value: unknown): DocumentComparisonResponse {
|
||||||
|
const payload = record(value, "document response");
|
||||||
|
if (payload.schema_version !== 1) {
|
||||||
|
return invalid("document schema_version");
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
schema_version: 1,
|
||||||
|
document: documentSummary(payload.document),
|
||||||
|
components: array(payload.components, "components").map(component),
|
||||||
|
original_markdown: nullableString(payload.original_markdown, "original_markdown"),
|
||||||
|
cleaned_markdown: nullableString(payload.cleaned_markdown, "cleaned_markdown"),
|
||||||
|
changes: array(payload.changes, "changes").map(change),
|
||||||
|
errors: array(payload.errors, "errors").map(runError),
|
||||||
|
residual_proposals: array(payload.residual_proposals, "residual_proposals").map(residual),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseStage(value: unknown): ComponentStageResponse {
|
||||||
|
const payload = record(value, "component stage response");
|
||||||
|
if (payload.schema_version !== 1) {
|
||||||
|
return invalid("component stage schema_version");
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
schema_version: 1,
|
||||||
|
document_id: string(payload.document_id, "document_id"),
|
||||||
|
component: component(payload.component),
|
||||||
|
before_sha256: hash(payload.before_sha256, "before_sha256"),
|
||||||
|
after_sha256: hash(payload.after_sha256, "after_sha256"),
|
||||||
|
before_markdown: string(payload.before_markdown, "before_markdown"),
|
||||||
|
after_markdown: string(payload.after_markdown, "after_markdown"),
|
||||||
|
changes: array(payload.changes, "changes").map(change),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function getJson<T>(
|
||||||
|
pathname: string,
|
||||||
|
parse: (payload: unknown) => T,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<T> {
|
||||||
|
const response = await fetch(pathname, {
|
||||||
|
method: "GET",
|
||||||
|
cache: "no-store",
|
||||||
|
credentials: "same-origin",
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
const payload: unknown = await response.json();
|
||||||
|
if (!response.ok) {
|
||||||
|
const errorPayload = payload as Partial<ApiErrorResponse>;
|
||||||
|
throw new ReviewerApiError(
|
||||||
|
errorPayload.error?.code ?? "request_failed",
|
||||||
|
errorPayload.error?.message ?? `请求失败（HTTP ${response.status}）。`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return parse(payload);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fetchRun(signal?: AbortSignal): Promise<RunSummaryResponse> {
|
||||||
|
return getJson("/api/v1/run", parseRun, signal);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fetchDocument(
|
||||||
|
documentId: string,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<DocumentComparisonResponse> {
|
||||||
|
return getJson(
|
||||||
|
`/api/v1/documents/${encodeURIComponent(documentId)}`,
|
||||||
|
parseDocument,
|
||||||
|
signal,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fetchComponentStage(
|
||||||
|
documentId: string,
|
||||||
|
componentPosition: number,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<ComponentStageResponse> {
|
||||||
|
return getJson(
|
||||||
|
`/api/v1/documents/${encodeURIComponent(documentId)}/components/${componentPosition}`,
|
||||||
|
parseStage,
|
||||||
|
signal,
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
import { StrictMode } from "react";
|
||||||
|
import { createRoot } from "react-dom/client";
|
||||||
|
|
||||||
|
import { App } from "./App.js";
|
||||||
|
import "./styles.css";
|
||||||
|
|
||||||
|
const root = document.getElementById("root");
|
||||||
|
if (root === null) {
|
||||||
|
throw new Error("missing #root element");
|
||||||
|
}
|
||||||
|
|
||||||
|
createRoot(root).render(
|
||||||
|
<StrictMode>
|
||||||
|
<App />
|
||||||
|
</StrictMode>,
|
||||||
|
);
|
||||||
@@ -0,0 +1,558 @@
|
|||||||
|
:root {
|
||||||
|
color: #252720;
|
||||||
|
background: #ecebe5;
|
||||||
|
font-family:
|
||||||
|
Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Microsoft YaHei",
|
||||||
|
sans-serif;
|
||||||
|
font-synthesis: none;
|
||||||
|
text-rendering: optimizeLegibility;
|
||||||
|
}
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
html,
|
||||||
|
body,
|
||||||
|
#root {
|
||||||
|
min-width: 1180px;
|
||||||
|
min-height: 100%;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
button {
|
||||||
|
color: inherit;
|
||||||
|
font: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
|
button:focus-visible {
|
||||||
|
outline: 2px solid #315f4b;
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.app-shell {
|
||||||
|
min-height: 100vh;
|
||||||
|
background:
|
||||||
|
radial-gradient(circle at 12% 0%, rgb(255 255 255 / 72%), transparent 34%),
|
||||||
|
#ecebe5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar {
|
||||||
|
position: sticky;
|
||||||
|
z-index: 20;
|
||||||
|
top: 0;
|
||||||
|
display: flex;
|
||||||
|
min-height: 76px;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
padding: 12px 24px;
|
||||||
|
border-bottom: 1px solid #d7d5cd;
|
||||||
|
background: rgb(248 247 242 / 94%);
|
||||||
|
backdrop-filter: blur(18px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar > div:first-child {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.brand-mark {
|
||||||
|
display: grid;
|
||||||
|
width: 42px;
|
||||||
|
height: 42px;
|
||||||
|
place-items: center;
|
||||||
|
border-radius: 12px;
|
||||||
|
background: #284d3d;
|
||||||
|
color: #f3f4ed;
|
||||||
|
font-family: Georgia, serif;
|
||||||
|
font-size: 19px;
|
||||||
|
letter-spacing: -0.08em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.eyebrow {
|
||||||
|
margin: 0 0 3px;
|
||||||
|
color: #78796f;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.12em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1,
|
||||||
|
h2,
|
||||||
|
h3,
|
||||||
|
p {
|
||||||
|
margin-top: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar h1 {
|
||||||
|
margin: 0;
|
||||||
|
font-family: Georgia, "Songti SC", serif;
|
||||||
|
font-size: 19px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.run-facts,
|
||||||
|
.document-meta {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 10px;
|
||||||
|
color: #66685f;
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.run-facts > span:not(.status),
|
||||||
|
.document-meta > span:not(.status) {
|
||||||
|
padding-left: 10px;
|
||||||
|
border-left: 1px solid #d5d2c9;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
border: 1px solid currentColor;
|
||||||
|
border-radius: 999px;
|
||||||
|
padding: 3px 8px;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status--success {
|
||||||
|
color: #277052;
|
||||||
|
background: #edf6ef;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status--failed {
|
||||||
|
color: #a04338;
|
||||||
|
background: #fff0ed;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status--unstable {
|
||||||
|
color: #986617;
|
||||||
|
background: #fff7df;
|
||||||
|
}
|
||||||
|
|
||||||
|
.notice {
|
||||||
|
padding: 9px 24px;
|
||||||
|
border-bottom: 1px solid #e6d09c;
|
||||||
|
background: #fff7de;
|
||||||
|
color: #77561d;
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.layout {
|
||||||
|
display: grid;
|
||||||
|
min-height: calc(100vh - 76px);
|
||||||
|
grid-template-columns: 300px minmax(0, 1fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.sidebar {
|
||||||
|
position: sticky;
|
||||||
|
top: 76px;
|
||||||
|
overflow-y: auto;
|
||||||
|
height: calc(100vh - 76px);
|
||||||
|
border-right: 1px solid #d7d5cd;
|
||||||
|
background: #f7f6f1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sidebar section {
|
||||||
|
padding: 20px 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sidebar section + section {
|
||||||
|
border-top: 1px solid #dfddd5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-heading {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-heading h2 {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 12px;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.section-heading > span {
|
||||||
|
color: #888980;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list,
|
||||||
|
.component-list {
|
||||||
|
display: grid;
|
||||||
|
gap: 4px;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
list-style: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list button,
|
||||||
|
.component-list button {
|
||||||
|
display: grid;
|
||||||
|
width: 100%;
|
||||||
|
align-items: center;
|
||||||
|
border: 0;
|
||||||
|
border-radius: 9px;
|
||||||
|
background: transparent;
|
||||||
|
cursor: pointer;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list button {
|
||||||
|
grid-template-columns: 9px 1fr;
|
||||||
|
gap: 10px;
|
||||||
|
padding: 9px 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list button:hover,
|
||||||
|
.component-list button:hover:not(:disabled) {
|
||||||
|
background: #eceae2;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list button.is-active,
|
||||||
|
.component-list button.is-active {
|
||||||
|
background: #e0e8e0;
|
||||||
|
color: #234b39;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list strong,
|
||||||
|
.component-list strong {
|
||||||
|
display: block;
|
||||||
|
overflow: hidden;
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 650;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-list small,
|
||||||
|
.component-list small {
|
||||||
|
display: block;
|
||||||
|
margin-top: 3px;
|
||||||
|
color: #7d7e75;
|
||||||
|
font-size: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-dot {
|
||||||
|
width: 7px;
|
||||||
|
height: 7px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: #999;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-dot--success {
|
||||||
|
background: #348361;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-dot--failed {
|
||||||
|
background: #b64b3f;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-dot--unstable {
|
||||||
|
background: #bd831c;
|
||||||
|
}
|
||||||
|
|
||||||
|
.text-button {
|
||||||
|
border: 0;
|
||||||
|
background: transparent;
|
||||||
|
color: #315f4b;
|
||||||
|
cursor: pointer;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.text-button:disabled {
|
||||||
|
color: #aaa99f;
|
||||||
|
cursor: default;
|
||||||
|
}
|
||||||
|
|
||||||
|
.component-list {
|
||||||
|
counter-reset: components;
|
||||||
|
}
|
||||||
|
|
||||||
|
.component-list button {
|
||||||
|
grid-template-columns: 26px minmax(0, 1fr);
|
||||||
|
gap: 8px;
|
||||||
|
padding: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.component-list button:disabled {
|
||||||
|
cursor: not-allowed;
|
||||||
|
opacity: 0.55;
|
||||||
|
}
|
||||||
|
|
||||||
|
.component-index {
|
||||||
|
display: grid;
|
||||||
|
width: 24px;
|
||||||
|
height: 24px;
|
||||||
|
place-items: center;
|
||||||
|
border: 1px solid #d3d1c8;
|
||||||
|
border-radius: 50%;
|
||||||
|
color: #74766e;
|
||||||
|
font-family: Georgia, serif;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.workspace {
|
||||||
|
display: grid;
|
||||||
|
min-width: 0;
|
||||||
|
align-content: start;
|
||||||
|
gap: 14px;
|
||||||
|
padding: 18px 20px 30px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-header {
|
||||||
|
display: flex;
|
||||||
|
align-items: end;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 20px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.document-header h2 {
|
||||||
|
margin: 0;
|
||||||
|
font-family: Georgia, "Songti SC", serif;
|
||||||
|
font-size: 21px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-shell,
|
||||||
|
.changes-panel,
|
||||||
|
.diagnostic-panel,
|
||||||
|
.state-panel {
|
||||||
|
overflow: hidden;
|
||||||
|
border: 1px solid #d5d3ca;
|
||||||
|
border-radius: 13px;
|
||||||
|
background: #fbfaf7;
|
||||||
|
box-shadow: 0 12px 36px rgb(55 57 48 / 7%);
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-shell {
|
||||||
|
min-height: 510px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-labels {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
border-bottom: 1px solid #dcdbd3;
|
||||||
|
background: #f4f2ec;
|
||||||
|
color: #6f7168;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.06em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-labels span {
|
||||||
|
padding: 9px 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-labels span + span {
|
||||||
|
border-left: 1px solid #dcdbd3;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-host {
|
||||||
|
height: 510px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-host > .cm-mergeView {
|
||||||
|
height: 100%;
|
||||||
|
overflow-y: auto;
|
||||||
|
overscroll-behavior: contain;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-host .cm-mergeViewEditors {
|
||||||
|
min-height: 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diff-host .cm-editor {
|
||||||
|
min-width: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.changes-panel {
|
||||||
|
min-height: 120px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.changes-heading {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 24px;
|
||||||
|
padding: 15px 18px;
|
||||||
|
border-bottom: 1px solid #e0ded6;
|
||||||
|
}
|
||||||
|
|
||||||
|
.changes-heading h3 {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.changes-heading > p {
|
||||||
|
max-width: 58%;
|
||||||
|
margin: 0;
|
||||||
|
color: #77786f;
|
||||||
|
font-size: 11px;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list {
|
||||||
|
display: grid;
|
||||||
|
max-height: 310px;
|
||||||
|
gap: 1px;
|
||||||
|
overflow-y: auto;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
background: #e4e2da;
|
||||||
|
list-style: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list button {
|
||||||
|
display: grid;
|
||||||
|
width: 100%;
|
||||||
|
grid-template-columns: 145px minmax(240px, 1fr) minmax(260px, 0.9fr);
|
||||||
|
align-items: center;
|
||||||
|
gap: 16px;
|
||||||
|
border: 0;
|
||||||
|
padding: 11px 18px;
|
||||||
|
background: #fbfaf7;
|
||||||
|
cursor: pointer;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list button:hover {
|
||||||
|
background: #f4f5ef;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list button:disabled {
|
||||||
|
cursor: default;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-location {
|
||||||
|
color: #73756c;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list strong {
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-sample {
|
||||||
|
display: flex;
|
||||||
|
min-width: 0;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-sample del,
|
||||||
|
.change-sample ins {
|
||||||
|
overflow: hidden;
|
||||||
|
max-width: 46%;
|
||||||
|
border-radius: 4px;
|
||||||
|
padding: 2px 5px;
|
||||||
|
text-decoration: none;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-sample del {
|
||||||
|
background: #f9ded9;
|
||||||
|
color: #913e34;
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-sample ins {
|
||||||
|
background: #dcecdf;
|
||||||
|
color: #276348;
|
||||||
|
}
|
||||||
|
|
||||||
|
.quiet-message {
|
||||||
|
margin: 0;
|
||||||
|
padding: 22px 18px;
|
||||||
|
color: #77786f;
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.state-panel,
|
||||||
|
.diagnostic-panel {
|
||||||
|
padding: 28px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.state-panel {
|
||||||
|
display: grid;
|
||||||
|
min-height: 180px;
|
||||||
|
place-items: center;
|
||||||
|
align-content: center;
|
||||||
|
color: #66685f;
|
||||||
|
}
|
||||||
|
|
||||||
|
.state-panel p {
|
||||||
|
margin: 8px 0 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.state-panel--error {
|
||||||
|
border-color: #e2b6ae;
|
||||||
|
color: #8f3e34;
|
||||||
|
}
|
||||||
|
|
||||||
|
.loading-dot {
|
||||||
|
width: 11px;
|
||||||
|
height: 11px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: #3d735b;
|
||||||
|
box-shadow: 0 0 0 7px #dce9df;
|
||||||
|
animation: pulse 1.25s ease-in-out infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diagnostic-panel h3 {
|
||||||
|
margin-bottom: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diagnostic-panel > p:not(.eyebrow) {
|
||||||
|
color: #686a61;
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.diagnostic-panel article {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 220px 1fr;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 10px 0;
|
||||||
|
border-top: 1px solid #e0ded6;
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
code {
|
||||||
|
border-radius: 4px;
|
||||||
|
padding: 1px 4px;
|
||||||
|
background: #eceae3;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes pulse {
|
||||||
|
0%,
|
||||||
|
100% {
|
||||||
|
opacity: 0.45;
|
||||||
|
transform: scale(0.82);
|
||||||
|
}
|
||||||
|
|
||||||
|
50% {
|
||||||
|
opacity: 1;
|
||||||
|
transform: scale(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 1280px) {
|
||||||
|
.layout {
|
||||||
|
grid-template-columns: 270px minmax(0, 1fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.change-list button {
|
||||||
|
grid-template-columns: 125px minmax(180px, 1fr) minmax(220px, 0.8fr);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
export type RunStatus = "success" | "failed" | "unstable";
|
||||||
|
|
||||||
|
export interface ComponentSummary {
|
||||||
|
component_position: number;
|
||||||
|
component_id: string;
|
||||||
|
version: string;
|
||||||
|
parameters: unknown;
|
||||||
|
applicability: string;
|
||||||
|
change_count: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DocumentSummary {
|
||||||
|
document_id: string;
|
||||||
|
source_label: string;
|
||||||
|
status: RunStatus;
|
||||||
|
input_sha256: string;
|
||||||
|
current_sha256: string;
|
||||||
|
change_count: number;
|
||||||
|
source_available: boolean;
|
||||||
|
output_available: boolean;
|
||||||
|
availability_error: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RunSummaryResponse {
|
||||||
|
schema_version: 1;
|
||||||
|
run: {
|
||||||
|
run_id: string;
|
||||||
|
run_date: string;
|
||||||
|
status: RunStatus;
|
||||||
|
started_at_utc: string;
|
||||||
|
completed_at_utc: string;
|
||||||
|
retention_until: string;
|
||||||
|
};
|
||||||
|
components: ComponentSummary[];
|
||||||
|
documents: DocumentSummary[];
|
||||||
|
summary: {
|
||||||
|
document_count: number;
|
||||||
|
success_count: number;
|
||||||
|
failed_count: number;
|
||||||
|
unstable_count: number;
|
||||||
|
change_count: number;
|
||||||
|
};
|
||||||
|
original_run_location_changed: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ChangeDetail {
|
||||||
|
component_id: string;
|
||||||
|
component_version: string;
|
||||||
|
component_position: number;
|
||||||
|
proposal_ref: {
|
||||||
|
component_position: number;
|
||||||
|
snapshot_sha256: string;
|
||||||
|
proposal_index: number;
|
||||||
|
};
|
||||||
|
edit_index: number;
|
||||||
|
reason: string;
|
||||||
|
location: {
|
||||||
|
line: number;
|
||||||
|
column: number;
|
||||||
|
};
|
||||||
|
editor_range: {
|
||||||
|
start: number;
|
||||||
|
end: number;
|
||||||
|
} | null;
|
||||||
|
before: string;
|
||||||
|
after: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RunErrorDetail {
|
||||||
|
component_id: string;
|
||||||
|
component_version: string;
|
||||||
|
component_position: number;
|
||||||
|
stage: "transform" | "final_review";
|
||||||
|
error_type: string;
|
||||||
|
message: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ResidualProposalDetail {
|
||||||
|
component_id: string;
|
||||||
|
component_version: string;
|
||||||
|
component_position: number;
|
||||||
|
reason: string;
|
||||||
|
edit_count: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DocumentComparisonResponse {
|
||||||
|
schema_version: 1;
|
||||||
|
document: DocumentSummary;
|
||||||
|
components: ComponentSummary[];
|
||||||
|
original_markdown: string | null;
|
||||||
|
cleaned_markdown: string | null;
|
||||||
|
changes: ChangeDetail[];
|
||||||
|
errors: RunErrorDetail[];
|
||||||
|
residual_proposals: ResidualProposalDetail[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ComponentStageResponse {
|
||||||
|
schema_version: 1;
|
||||||
|
document_id: string;
|
||||||
|
component: ComponentSummary;
|
||||||
|
before_sha256: string;
|
||||||
|
after_sha256: string;
|
||||||
|
before_markdown: string;
|
||||||
|
after_markdown: string;
|
||||||
|
changes: ChangeDetail[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ApiErrorResponse {
|
||||||
|
error: {
|
||||||
|
code: string;
|
||||||
|
message: string;
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,256 @@
|
|||||||
|
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
|
||||||
|
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||||
|
|
||||||
|
import { App } from "../src/client/App.js";
|
||||||
|
import type {
|
||||||
|
ComponentStageResponse,
|
||||||
|
DocumentComparisonResponse,
|
||||||
|
RunSummaryResponse,
|
||||||
|
} from "../src/shared/api.js";
|
||||||
|
|
||||||
|
vi.mock("../src/client/DiffView.js", () => ({
|
||||||
|
DiffView: ({ beforeLabel, afterLabel }: { beforeLabel: string; afterLabel: string }) => (
|
||||||
|
<div data-testid="diff-view">
|
||||||
|
{beforeLabel} / {afterLabel}
|
||||||
|
</div>
|
||||||
|
),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const components = [
|
||||||
|
{
|
||||||
|
component_position: 0,
|
||||||
|
component_id: "paper.rule",
|
||||||
|
version: "1.0.0",
|
||||||
|
parameters: [],
|
||||||
|
applicability: "替换测试单词。",
|
||||||
|
change_count: 1,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
component_position: 1,
|
||||||
|
component_id: "paper.zero",
|
||||||
|
version: "1.0.0",
|
||||||
|
parameters: [],
|
||||||
|
applicability: "不修改当前测试文档。",
|
||||||
|
change_count: 0,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
const documentSummary = {
|
||||||
|
document_id: "paper",
|
||||||
|
source_label: "inputs/paper.md",
|
||||||
|
status: "success" as const,
|
||||||
|
input_sha256: "1".repeat(64),
|
||||||
|
current_sha256: "2".repeat(64),
|
||||||
|
change_count: 1,
|
||||||
|
source_available: true,
|
||||||
|
output_available: true,
|
||||||
|
availability_error: null,
|
||||||
|
};
|
||||||
|
|
||||||
|
const runResponse: RunSummaryResponse = {
|
||||||
|
schema_version: 1,
|
||||||
|
run: {
|
||||||
|
run_id: "review-run",
|
||||||
|
run_date: "2026-08-23",
|
||||||
|
status: "success",
|
||||||
|
started_at_utc: "2026-08-23T01:00:00Z",
|
||||||
|
completed_at_utc: "2026-08-23T01:01:00Z",
|
||||||
|
retention_until: "2026-09-22T01:01:00Z",
|
||||||
|
},
|
||||||
|
components,
|
||||||
|
documents: [documentSummary],
|
||||||
|
summary: {
|
||||||
|
document_count: 1,
|
||||||
|
success_count: 1,
|
||||||
|
failed_count: 0,
|
||||||
|
unstable_count: 0,
|
||||||
|
change_count: 1,
|
||||||
|
},
|
||||||
|
original_run_location_changed: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
const change = {
|
||||||
|
component_id: "paper.rule",
|
||||||
|
component_version: "1.0.0",
|
||||||
|
component_position: 0,
|
||||||
|
proposal_ref: {
|
||||||
|
component_position: 0,
|
||||||
|
snapshot_sha256: "1".repeat(64),
|
||||||
|
proposal_index: 0,
|
||||||
|
},
|
||||||
|
edit_index: 0,
|
||||||
|
reason: "替换测试单词",
|
||||||
|
location: { line: 1, column: 3 },
|
||||||
|
editor_range: { start: 0, end: 3 },
|
||||||
|
before: "old",
|
||||||
|
after: "new",
|
||||||
|
};
|
||||||
|
|
||||||
|
const documentResponse: DocumentComparisonResponse = {
|
||||||
|
schema_version: 1,
|
||||||
|
document: documentSummary,
|
||||||
|
components,
|
||||||
|
original_markdown: "old",
|
||||||
|
cleaned_markdown: "new",
|
||||||
|
changes: [change],
|
||||||
|
errors: [],
|
||||||
|
residual_proposals: [],
|
||||||
|
};
|
||||||
|
|
||||||
|
const stageResponse: ComponentStageResponse = {
|
||||||
|
schema_version: 1,
|
||||||
|
document_id: "paper",
|
||||||
|
component: components[0]!,
|
||||||
|
before_sha256: "1".repeat(64),
|
||||||
|
after_sha256: "2".repeat(64),
|
||||||
|
before_markdown: "old",
|
||||||
|
after_markdown: "new",
|
||||||
|
changes: [change],
|
||||||
|
};
|
||||||
|
|
||||||
|
function response(payload: unknown, status = 200): Response {
|
||||||
|
return new Response(JSON.stringify(payload), {
|
||||||
|
status,
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("App", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("shows the run, document comparison, component order and component stage", async () => {
|
||||||
|
const fetchMock = vi.fn((input: string | URL | Request) => {
|
||||||
|
const pathname =
|
||||||
|
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||||
|
if (pathname === "/api/v1/run") {
|
||||||
|
return Promise.resolve(response(runResponse));
|
||||||
|
}
|
||||||
|
if (pathname.endsWith("/components/0")) {
|
||||||
|
return Promise.resolve(response(stageResponse));
|
||||||
|
}
|
||||||
|
return Promise.resolve(response(documentResponse));
|
||||||
|
});
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
|
||||||
|
render(<App />);
|
||||||
|
|
||||||
|
expect(await screen.findByRole("heading", { name: "review-run" })).toBeInTheDocument();
|
||||||
|
expect(await screen.findByTestId("diff-view")).toHaveTextContent("清洗前 / 清洗后");
|
||||||
|
expect(screen.getByText("paper.rule")).toBeInTheDocument();
|
||||||
|
expect(screen.getByText("paper.zero")).toBeInTheDocument();
|
||||||
|
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||||
|
|
||||||
|
fireEvent.click(screen.getByRole("button", { name: /paper\.rule/ }));
|
||||||
|
await waitFor(() => {
|
||||||
|
expect(fetchMock).toHaveBeenCalledWith(expect.stringContaining("components/0"), expect.anything());
|
||||||
|
});
|
||||||
|
expect(await screen.findByTestId("diff-view")).toHaveTextContent("组件 1 执行前 / 组件 1 执行后");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("shows API failures without rendering document text", async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(() =>
|
||||||
|
Promise.resolve(
|
||||||
|
response({ error: { code: "unsupported_schema", message: "不支持这个产物版本。" } }, 409),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
render(<App />);
|
||||||
|
|
||||||
|
expect(await screen.findByRole("alert")).toHaveTextContent("不支持这个产物版本");
|
||||||
|
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("shows failed audit evidence without inventing a cleaned result", async () => {
|
||||||
|
const failedSummary = {
|
||||||
|
...documentSummary,
|
||||||
|
status: "failed" as const,
|
||||||
|
current_sha256: documentSummary.input_sha256,
|
||||||
|
change_count: 0,
|
||||||
|
output_available: false,
|
||||||
|
};
|
||||||
|
const failedRun: RunSummaryResponse = {
|
||||||
|
...runResponse,
|
||||||
|
run: { ...runResponse.run, status: "failed" },
|
||||||
|
documents: [failedSummary],
|
||||||
|
summary: {
|
||||||
|
document_count: 1,
|
||||||
|
success_count: 0,
|
||||||
|
failed_count: 1,
|
||||||
|
unstable_count: 0,
|
||||||
|
change_count: 0,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const failedDocument: DocumentComparisonResponse = {
|
||||||
|
schema_version: 1,
|
||||||
|
document: failedSummary,
|
||||||
|
components: components.map((component) => ({ ...component, change_count: 0 })),
|
||||||
|
original_markdown: "原文",
|
||||||
|
cleaned_markdown: null,
|
||||||
|
changes: [],
|
||||||
|
errors: [
|
||||||
|
{
|
||||||
|
component_id: "paper.rule",
|
||||||
|
component_version: "1.0.0",
|
||||||
|
component_position: 0,
|
||||||
|
stage: "transform",
|
||||||
|
error_type: "SyntheticError",
|
||||||
|
message: "测试组件失败。",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
residual_proposals: [],
|
||||||
|
};
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn((input: string | URL | Request) => {
|
||||||
|
const pathname =
|
||||||
|
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||||
|
return Promise.resolve(response(pathname === "/api/v1/run" ? failedRun : failedDocument));
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
render(<App />);
|
||||||
|
|
||||||
|
expect(await screen.findByRole("heading", { name: "失败文档只展示审计证据" })).toBeInTheDocument();
|
||||||
|
expect(screen.getByText("测试组件失败。")).toBeInTheDocument();
|
||||||
|
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps audit details visible when a successful run has lost its original source", async () => {
|
||||||
|
const unavailableSummary = {
|
||||||
|
...documentSummary,
|
||||||
|
source_available: false,
|
||||||
|
availability_error: "原文路径已经失效。",
|
||||||
|
};
|
||||||
|
const unavailableRun: RunSummaryResponse = {
|
||||||
|
...runResponse,
|
||||||
|
documents: [unavailableSummary],
|
||||||
|
};
|
||||||
|
const unavailableDocument: DocumentComparisonResponse = {
|
||||||
|
...documentResponse,
|
||||||
|
document: unavailableSummary,
|
||||||
|
original_markdown: null,
|
||||||
|
changes: [{ ...change, editor_range: null }],
|
||||||
|
};
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn((input: string | URL | Request) => {
|
||||||
|
const pathname =
|
||||||
|
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||||
|
return Promise.resolve(
|
||||||
|
response(pathname === "/api/v1/run" ? unavailableRun : unavailableDocument),
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
render(<App />);
|
||||||
|
|
||||||
|
expect(await screen.findByRole("alert")).toHaveTextContent("原文路径已经失效");
|
||||||
|
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||||
|
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
import { render, screen } from "@testing-library/react";
|
||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { EditorView } from "@codemirror/view";
|
||||||
|
|
||||||
|
import { DiffView } from "../src/client/DiffView.js";
|
||||||
|
|
||||||
|
function leftPaneSelection(): { from: number; to: number } | null {
|
||||||
|
const pane = document.querySelector(".diff-host .cm-editor");
|
||||||
|
if (pane === null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
const view = EditorView.findFromDOM(pane as HTMLElement);
|
||||||
|
if (view === null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
const { from, to } = view.state.selection.main;
|
||||||
|
return { from, to };
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("DiffView", () => {
|
||||||
|
it("keeps Markdown and raw HTML as inert editor text", () => {
|
||||||
|
render(
|
||||||
|
<DiffView
|
||||||
|
before={'# title\n<img src="https://example.com/private.png" onerror="alert(1)">'}
|
||||||
|
after={'# title\n<script>alert("x")</script>'}
|
||||||
|
beforeLabel="清洗前"
|
||||||
|
afterLabel="清洗后"
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(screen.getByRole("region", { name: "清洗前与清洗后对比" })).toBeInTheDocument();
|
||||||
|
expect(document.querySelector("img")).toBeNull();
|
||||||
|
expect(document.querySelector("script")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps long unchanged sections available in the full document view", () => {
|
||||||
|
const before = Array.from({ length: 30 }, (_, index) => `line ${index + 1}`);
|
||||||
|
const after = [...before];
|
||||||
|
after[14] = "changed line 15";
|
||||||
|
|
||||||
|
render(
|
||||||
|
<DiffView
|
||||||
|
before={before.join("\n")}
|
||||||
|
after={after.join("\n")}
|
||||||
|
beforeLabel="清洗前"
|
||||||
|
afterLabel="清洗后"
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(document.querySelector(".cm-collapsedLines")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("re-applies the focus selection after the compared texts change", () => {
|
||||||
|
const longText = (mark: string) =>
|
||||||
|
Array.from({ length: 30 }, (_, index) => (index === 14 ? mark : `line ${index + 1}`)).join("\n");
|
||||||
|
|
||||||
|
const { rerender } = render(
|
||||||
|
<DiffView before={longText("old")} after={longText("new")} beforeLabel="清洗前" afterLabel="清洗后" />,
|
||||||
|
);
|
||||||
|
|
||||||
|
// 模拟点击跨组件修改项：focusRange 先落在旧文本上，stage 文本随后到达。
|
||||||
|
rerender(
|
||||||
|
<DiffView
|
||||||
|
before={longText("old")}
|
||||||
|
after={longText("new")}
|
||||||
|
beforeLabel="组件 1 执行前"
|
||||||
|
afterLabel="组件 1 执行后"
|
||||||
|
focusRange={{ start: 58, end: 61 }}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
rerender(
|
||||||
|
<DiffView
|
||||||
|
before={longText("stage before")}
|
||||||
|
after={longText("stage after")}
|
||||||
|
beforeLabel="组件 2 执行前"
|
||||||
|
afterLabel="组件 2 执行后"
|
||||||
|
focusRange={{ start: 58, end: 71 }}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
|
||||||
|
// 最终视图必须是组件 2 的文本，且选区重新落在新文本的目标范围上。
|
||||||
|
expect(screen.getByRole("region", { name: "组件 2 执行前与组件 2 执行后对比" })).toBeInTheDocument();
|
||||||
|
expect(leftPaneSelection()).toEqual({ from: 58, to: 71 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("applies the focus selection when it arrives with the initial texts", () => {
|
||||||
|
render(
|
||||||
|
<DiffView
|
||||||
|
before={"a\nb"}
|
||||||
|
after={"a\nc"}
|
||||||
|
beforeLabel="清洗前"
|
||||||
|
afterLabel="清洗后"
|
||||||
|
focusRange={{ start: 0, end: 1 }}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(leftPaneSelection()).toEqual({ from: 0, to: 1 });
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
|
||||||
|
import { fetchRun } from "../src/client/api-client.js";
|
||||||
|
|
||||||
|
describe("API response validation", () => {
|
||||||
|
it("rejects a successful HTTP response with an unknown schema", async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(() =>
|
||||||
|
Promise.resolve(
|
||||||
|
new Response(JSON.stringify({ schema_version: 2 }), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(fetchRun()).rejects.toMatchObject({
|
||||||
|
code: "invalid_response",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
import "@testing-library/jest-dom/vitest";
|
||||||
|
|
||||||
|
import { afterEach } from "vitest";
|
||||||
|
import { cleanup } from "@testing-library/react";
|
||||||
|
|
||||||
|
afterEach(() => cleanup());
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2023",
|
||||||
|
"useDefineForClassFields": true,
|
||||||
|
"lib": ["ES2023", "DOM", "DOM.Iterable"],
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "Bundler",
|
||||||
|
"allowImportingTsExtensions": false,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"jsx": "react-jsx",
|
||||||
|
"strict": true,
|
||||||
|
"noUncheckedIndexedAccess": true,
|
||||||
|
"noImplicitOverride": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"types": ["node", "vite/client", "vitest/globals", "@testing-library/jest-dom/vitest"]
|
||||||
|
},
|
||||||
|
"include": ["src", "tests", "vite.config.ts"]
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import react from "@vitejs/plugin-react";
|
||||||
|
import { defineConfig } from "vitest/config";
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
plugins: [react()],
|
||||||
|
server: {
|
||||||
|
host: "127.0.0.1",
|
||||||
|
port: 5173,
|
||||||
|
strictPort: true,
|
||||||
|
proxy: {
|
||||||
|
"/api": {
|
||||||
|
target: "http://127.0.0.1:4174",
|
||||||
|
changeOrigin: true,
|
||||||
|
configure(proxy) {
|
||||||
|
proxy.on("proxyReq", (request) => request.removeHeader("origin"));
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
build: {
|
||||||
|
outDir: "dist",
|
||||||
|
emptyOutDir: true,
|
||||||
|
},
|
||||||
|
test: {
|
||||||
|
environment: "jsdom",
|
||||||
|
setupFiles: "./tests/setup.ts",
|
||||||
|
css: true,
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
"""Run the approved first-batch ClinDB pipeline over five local paper copies."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from mdpolish.components import (
|
||||||
|
ArxivSubmissionStampComponent,
|
||||||
|
HtmlTableDoubleEscapeComponent,
|
||||||
|
HtmlTableLayoutComponent,
|
||||||
|
ManuscriptLineNumberComponent,
|
||||||
|
PageBreakWordJoinComponent,
|
||||||
|
ReferenceSpacingComponent,
|
||||||
|
RepeatedRunningHeaderComponent,
|
||||||
|
WordReviewCommentComponent,
|
||||||
|
)
|
||||||
|
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||||
|
from mdpolish.models import RunStatus
|
||||||
|
from mdpolish.pipeline import Pipeline
|
||||||
|
|
||||||
|
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||||
|
CLINDB_WORD_JOIN_MAPPINGS = (
|
||||||
|
("medi-", "cal", "medical"),
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("cre-", "ated", "created"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("life-", "threatening", "life-threatening"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build_pipeline() -> Pipeline:
|
||||||
|
"""Build the frozen ClinDB first-batch component order."""
|
||||||
|
return Pipeline(
|
||||||
|
[
|
||||||
|
WordReviewCommentComponent(),
|
||||||
|
ManuscriptLineNumberComponent(),
|
||||||
|
ArxivSubmissionStampComponent(),
|
||||||
|
RepeatedRunningHeaderComponent(),
|
||||||
|
PageBreakWordJoinComponent(CLINDB_WORD_JOIN_MAPPINGS),
|
||||||
|
HtmlTableDoubleEscapeComponent(),
|
||||||
|
HtmlTableLayoutComponent(),
|
||||||
|
ReferenceSpacingComponent(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = _parse_args()
|
||||||
|
repository_root = Path(__file__).resolve().parents[1]
|
||||||
|
documents = tuple(
|
||||||
|
InputDocument(
|
||||||
|
document_id=document_id,
|
||||||
|
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||||
|
source_label=f"data/md/{document_id}.md",
|
||||||
|
)
|
||||||
|
for document_id in _DOCUMENT_IDS
|
||||||
|
)
|
||||||
|
started_at = datetime.now().astimezone()
|
||||||
|
try:
|
||||||
|
result = run_experiment(
|
||||||
|
pipeline=build_pipeline(),
|
||||||
|
documents=documents,
|
||||||
|
run_id=args.run_id,
|
||||||
|
artifacts_root=repository_root / "artifacts",
|
||||||
|
started_at=started_at,
|
||||||
|
tool=collect_tool_metadata(repository_root),
|
||||||
|
)
|
||||||
|
except Exception as error:
|
||||||
|
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print(f"run_id={args.run_id}")
|
||||||
|
print(f"status={result.status.value}")
|
||||||
|
print(f"documents={result.document_count}")
|
||||||
|
print(f"changes={result.change_count}")
|
||||||
|
print(f"artifacts={result.run_directory}")
|
||||||
|
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,217 @@
|
|||||||
|
"""Pure replay of recorded artifact changes against an exact Markdown snapshot."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
from mdpolish.models import markdown_sha256
|
||||||
|
|
||||||
|
|
||||||
|
class ReplayError(ValueError):
|
||||||
|
"""A recorded change chain cannot be replayed without losing trust."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ReplayComponent:
|
||||||
|
"""The component identity needed to validate recorded changes."""
|
||||||
|
|
||||||
|
component_id: str
|
||||||
|
version: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ReplayChange:
|
||||||
|
"""One artifact edit expressed with Python Unicode code-point indexes."""
|
||||||
|
|
||||||
|
component_id: str
|
||||||
|
component_version: str
|
||||||
|
component_position: int
|
||||||
|
proposal_component_position: int
|
||||||
|
proposal_snapshot_sha256: str
|
||||||
|
proposal_index: int
|
||||||
|
edit_index: int
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
before: str
|
||||||
|
after: str
|
||||||
|
before_sha256: str
|
||||||
|
after_sha256: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class LocatedChange:
|
||||||
|
"""A replayed edit with human and browser-editor coordinates."""
|
||||||
|
|
||||||
|
change: ReplayChange
|
||||||
|
line: int
|
||||||
|
column: int
|
||||||
|
editor_start: int
|
||||||
|
editor_end: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ReplayStage:
|
||||||
|
"""The exact snapshots immediately before and after one component."""
|
||||||
|
|
||||||
|
component_position: int
|
||||||
|
before_sha256: str
|
||||||
|
after_sha256: str
|
||||||
|
before_markdown: str
|
||||||
|
after_markdown: str
|
||||||
|
changes: tuple[LocatedChange, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ReplayResult:
|
||||||
|
"""The verified final snapshot, locations and optional component stages."""
|
||||||
|
|
||||||
|
current_markdown: str
|
||||||
|
changes: tuple[LocatedChange, ...]
|
||||||
|
stages: tuple[ReplayStage, ...]
|
||||||
|
|
||||||
|
|
||||||
|
def _line_column(markdown: str, offset: int) -> tuple[int, int]:
|
||||||
|
if offset < 0 or offset > len(markdown):
|
||||||
|
raise ReplayError("a change offset is outside its recorded snapshot")
|
||||||
|
|
||||||
|
line = 1
|
||||||
|
column = 1
|
||||||
|
position = 0
|
||||||
|
while position < offset:
|
||||||
|
character = markdown[position]
|
||||||
|
if character == "\r":
|
||||||
|
line += 1
|
||||||
|
column = 1
|
||||||
|
if position + 1 < offset and markdown[position + 1] == "\n":
|
||||||
|
position += 2
|
||||||
|
else:
|
||||||
|
position += 1
|
||||||
|
elif character == "\n":
|
||||||
|
line += 1
|
||||||
|
column = 1
|
||||||
|
position += 1
|
||||||
|
else:
|
||||||
|
column += 1
|
||||||
|
position += 1
|
||||||
|
return line, column
|
||||||
|
|
||||||
|
|
||||||
|
def _utf16_offset(markdown: str, offset: int) -> int:
|
||||||
|
return len(markdown[:offset].encode("utf-16-le")) // 2
|
||||||
|
|
||||||
|
|
||||||
|
def _changes_conflict(left: ReplayChange, right: ReplayChange) -> bool:
|
||||||
|
if left.start == left.end and right.start == right.end:
|
||||||
|
return left.start == right.start
|
||||||
|
if left.start == left.end:
|
||||||
|
return right.start <= left.start <= right.end
|
||||||
|
if right.start == right.end:
|
||||||
|
return left.start <= right.start <= left.end
|
||||||
|
return max(left.start, right.start) < min(left.end, right.end)
|
||||||
|
|
||||||
|
|
||||||
|
def replay_change_chain(
|
||||||
|
*,
|
||||||
|
input_markdown: str,
|
||||||
|
input_sha256: str,
|
||||||
|
components: tuple[ReplayComponent, ...],
|
||||||
|
changes: tuple[ReplayChange, ...],
|
||||||
|
current_sha256: str,
|
||||||
|
current_markdown: str | None,
|
||||||
|
include_zero_change_stages: bool,
|
||||||
|
) -> ReplayResult:
|
||||||
|
"""Verify and replay a complete recorded chain without file I/O."""
|
||||||
|
if markdown_sha256(input_markdown) != input_sha256:
|
||||||
|
raise ReplayError("input Markdown does not match the recorded input hash")
|
||||||
|
|
||||||
|
previous_position = -1
|
||||||
|
for change in changes:
|
||||||
|
if change.component_position < previous_position:
|
||||||
|
raise ReplayError("recorded changes are not in component order")
|
||||||
|
previous_position = change.component_position
|
||||||
|
|
||||||
|
current = input_markdown
|
||||||
|
located_changes: list[LocatedChange] = []
|
||||||
|
stages: list[ReplayStage] = []
|
||||||
|
cursor = 0
|
||||||
|
|
||||||
|
for component_position, component in enumerate(components):
|
||||||
|
before_markdown = current
|
||||||
|
batch: list[ReplayChange] = []
|
||||||
|
while cursor < len(changes) and changes[cursor].component_position == component_position:
|
||||||
|
batch.append(changes[cursor])
|
||||||
|
cursor += 1
|
||||||
|
|
||||||
|
batch_locations: list[LocatedChange] = []
|
||||||
|
if batch:
|
||||||
|
first = batch[0]
|
||||||
|
if markdown_sha256(current) != first.before_sha256:
|
||||||
|
raise ReplayError("a change batch does not follow the recorded snapshot chain")
|
||||||
|
|
||||||
|
for change in batch:
|
||||||
|
if change.component_id != component.component_id or change.component_version != component.version:
|
||||||
|
raise ReplayError("a change identity does not match the recorded component")
|
||||||
|
if change.proposal_component_position != component_position:
|
||||||
|
raise ReplayError("a change proposal reference has the wrong component position")
|
||||||
|
if change.proposal_snapshot_sha256 != change.before_sha256:
|
||||||
|
raise ReplayError("a change proposal reference targets the wrong snapshot")
|
||||||
|
if change.before_sha256 != first.before_sha256 or change.after_sha256 != first.after_sha256:
|
||||||
|
raise ReplayError("a change batch contains inconsistent snapshot hashes")
|
||||||
|
if change.start < 0 or change.end < change.start or change.end > len(current):
|
||||||
|
raise ReplayError("a change span is outside its recorded snapshot")
|
||||||
|
if len(change.before) != change.end - change.start:
|
||||||
|
raise ReplayError("a change before value does not match its span length")
|
||||||
|
if current[change.start : change.end] != change.before:
|
||||||
|
raise ReplayError("a change before value does not match its recorded snapshot")
|
||||||
|
line, column = _line_column(current, change.start)
|
||||||
|
batch_locations.append(
|
||||||
|
LocatedChange(
|
||||||
|
change=change,
|
||||||
|
line=line,
|
||||||
|
column=column,
|
||||||
|
editor_start=_utf16_offset(current, change.start),
|
||||||
|
editor_end=_utf16_offset(current, change.end),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
for left_index, left in enumerate(batch):
|
||||||
|
for right in batch[left_index + 1 :]:
|
||||||
|
if _changes_conflict(left, right):
|
||||||
|
raise ReplayError("a change batch contains conflicting edit ranges")
|
||||||
|
|
||||||
|
application_order = sorted(
|
||||||
|
batch,
|
||||||
|
key=lambda change: (
|
||||||
|
change.start,
|
||||||
|
change.end,
|
||||||
|
change.proposal_index,
|
||||||
|
change.edit_index,
|
||||||
|
),
|
||||||
|
reverse=True,
|
||||||
|
)
|
||||||
|
for change in application_order:
|
||||||
|
current = current[: change.start] + change.after + current[change.end :]
|
||||||
|
if markdown_sha256(current) != first.after_sha256:
|
||||||
|
raise ReplayError("replayed changes do not produce the recorded batch hash")
|
||||||
|
|
||||||
|
located_changes.extend(batch_locations)
|
||||||
|
if include_zero_change_stages or batch:
|
||||||
|
stages.append(
|
||||||
|
ReplayStage(
|
||||||
|
component_position=component_position,
|
||||||
|
before_sha256=markdown_sha256(before_markdown),
|
||||||
|
after_sha256=markdown_sha256(current),
|
||||||
|
before_markdown=before_markdown,
|
||||||
|
after_markdown=current,
|
||||||
|
changes=tuple(batch_locations),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if cursor != len(changes):
|
||||||
|
raise ReplayError("a change has no matching component position")
|
||||||
|
if markdown_sha256(current) != current_sha256:
|
||||||
|
raise ReplayError("replayed changes do not produce the recorded current hash")
|
||||||
|
if current_markdown is not None and current != current_markdown:
|
||||||
|
raise ReplayError("replayed changes do not produce the recorded current snapshot")
|
||||||
|
|
||||||
|
return ReplayResult(current_markdown=current, changes=tuple(located_changes), stages=tuple(stages))
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
"""Fail-closed lexical ranges for the approved raw HTML table subset."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlCell:
|
||||||
|
"""The text-content range of one strict table cell."""
|
||||||
|
|
||||||
|
content_start: int
|
||||||
|
content_end: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlRow:
|
||||||
|
"""The full source range of one strict table row."""
|
||||||
|
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
cells: tuple[HtmlCell, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class HtmlTable:
|
||||||
|
"""Exact source ranges needed by table cleaning components."""
|
||||||
|
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
opening_end: int
|
||||||
|
closing_start: int
|
||||||
|
rows: tuple[HtmlRow, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _Tag:
|
||||||
|
name: str
|
||||||
|
closing: bool
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_tag(markdown: str, start: int) -> _Tag | None:
|
||||||
|
if start >= len(markdown) or markdown[start] != "<":
|
||||||
|
return None
|
||||||
|
|
||||||
|
position = start + 1
|
||||||
|
closing = position < len(markdown) and markdown[position] == "/"
|
||||||
|
if closing:
|
||||||
|
position += 1
|
||||||
|
|
||||||
|
name_start = position
|
||||||
|
while position < len(markdown) and markdown[position].isascii() and markdown[position].isalnum():
|
||||||
|
position += 1
|
||||||
|
if position == name_start:
|
||||||
|
return None
|
||||||
|
name = markdown[name_start:position]
|
||||||
|
if position >= len(markdown) or (not markdown[position].isspace() and markdown[position] != ">"):
|
||||||
|
return None
|
||||||
|
|
||||||
|
body_start = position
|
||||||
|
quote: str | None = None
|
||||||
|
while position < len(markdown):
|
||||||
|
character = markdown[position]
|
||||||
|
if quote is not None:
|
||||||
|
if character == quote:
|
||||||
|
quote = None
|
||||||
|
elif character in {'"', "'"}:
|
||||||
|
quote = character
|
||||||
|
elif character == "<":
|
||||||
|
return None
|
||||||
|
elif character == ">":
|
||||||
|
body = markdown[body_start:position]
|
||||||
|
if closing and body.strip():
|
||||||
|
return None
|
||||||
|
if not closing and body.rstrip().endswith("/"):
|
||||||
|
return None
|
||||||
|
return _Tag(name=name, closing=closing, start=start, end=position + 1)
|
||||||
|
position += 1
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _skip_whitespace(markdown: str, position: int, limit: int) -> int:
|
||||||
|
while position < limit and markdown[position].isspace():
|
||||||
|
position += 1
|
||||||
|
return position
|
||||||
|
|
||||||
|
|
||||||
|
def _balanced_table_end(markdown: str, opening: _Tag) -> int | None:
|
||||||
|
depth = 1
|
||||||
|
position = opening.end
|
||||||
|
while position < len(markdown):
|
||||||
|
tag_start = markdown.find("<", position)
|
||||||
|
if tag_start < 0:
|
||||||
|
return None
|
||||||
|
tag = _parse_tag(markdown, tag_start)
|
||||||
|
if tag is None:
|
||||||
|
position = tag_start + 1
|
||||||
|
continue
|
||||||
|
if tag.name == "table":
|
||||||
|
depth += -1 if tag.closing else 1
|
||||||
|
if depth == 0:
|
||||||
|
return tag.end
|
||||||
|
position = tag.end
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _strict_table_at(markdown: str, start: int, balanced_end: int) -> HtmlTable | None:
|
||||||
|
opening = _parse_tag(markdown, start)
|
||||||
|
if opening is None or opening.closing or opening.name != "table":
|
||||||
|
return None
|
||||||
|
|
||||||
|
rows: list[HtmlRow] = []
|
||||||
|
position = _skip_whitespace(markdown, opening.end, balanced_end)
|
||||||
|
closing_start = -1
|
||||||
|
while position < balanced_end:
|
||||||
|
tag = _parse_tag(markdown, position)
|
||||||
|
if tag is None:
|
||||||
|
return None
|
||||||
|
if tag.closing and tag.name == "table":
|
||||||
|
closing_start = tag.start
|
||||||
|
position = tag.end
|
||||||
|
break
|
||||||
|
if tag.closing or tag.name != "tr":
|
||||||
|
return None
|
||||||
|
|
||||||
|
row_start = tag.start
|
||||||
|
position = _skip_whitespace(markdown, tag.end, balanced_end)
|
||||||
|
cells: list[HtmlCell] = []
|
||||||
|
row_end = -1
|
||||||
|
while position < balanced_end:
|
||||||
|
child = _parse_tag(markdown, position)
|
||||||
|
if child is None:
|
||||||
|
return None
|
||||||
|
if child.closing and child.name == "tr":
|
||||||
|
row_end = child.end
|
||||||
|
position = child.end
|
||||||
|
break
|
||||||
|
if child.closing or child.name not in {"td", "th"}:
|
||||||
|
return None
|
||||||
|
|
||||||
|
content_start = child.end
|
||||||
|
close_start = markdown.find("<", content_start, balanced_end)
|
||||||
|
if close_start < 0:
|
||||||
|
return None
|
||||||
|
close = _parse_tag(markdown, close_start)
|
||||||
|
if close is None or not close.closing or close.name != child.name:
|
||||||
|
return None
|
||||||
|
cells.append(HtmlCell(content_start=content_start, content_end=close_start))
|
||||||
|
position = _skip_whitespace(markdown, close.end, balanced_end)
|
||||||
|
|
||||||
|
if not cells or row_end < 0:
|
||||||
|
return None
|
||||||
|
rows.append(HtmlRow(start=row_start, end=row_end, cells=tuple(cells)))
|
||||||
|
position = _skip_whitespace(markdown, position, balanced_end)
|
||||||
|
|
||||||
|
if not rows or closing_start < 0 or position != balanced_end:
|
||||||
|
return None
|
||||||
|
return HtmlTable(
|
||||||
|
start=start,
|
||||||
|
end=balanced_end,
|
||||||
|
opening_end=opening.end,
|
||||||
|
closing_start=closing_start,
|
||||||
|
rows=tuple(rows),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def strict_html_tables(markdown: str) -> tuple[HtmlTable, ...]:
|
||||||
|
"""Return non-overlapping complete tables from the approved strict subset."""
|
||||||
|
tables: list[HtmlTable] = []
|
||||||
|
position = 0
|
||||||
|
while position < len(markdown):
|
||||||
|
start = markdown.find("<table", position)
|
||||||
|
if start < 0:
|
||||||
|
break
|
||||||
|
opening = _parse_tag(markdown, start)
|
||||||
|
if opening is None or opening.closing or opening.name != "table":
|
||||||
|
position = start + 1
|
||||||
|
continue
|
||||||
|
balanced_end = _balanced_table_end(markdown, opening)
|
||||||
|
if balanced_end is None:
|
||||||
|
position = opening.end
|
||||||
|
continue
|
||||||
|
table = _strict_table_at(markdown, start, balanced_end)
|
||||||
|
if table is not None:
|
||||||
|
tables.append(table)
|
||||||
|
position = balanced_end
|
||||||
|
return tuple(tables)
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""Exact physical-line ranges shared by text cleaning components."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PhysicalLine:
|
||||||
|
"""A content range and its optional line ending in one source string."""
|
||||||
|
|
||||||
|
content_start: int
|
||||||
|
content_end: int
|
||||||
|
full_end: int
|
||||||
|
|
||||||
|
def content(self, markdown: str) -> str:
|
||||||
|
return markdown[self.content_start : self.content_end]
|
||||||
|
|
||||||
|
def line_ending(self, markdown: str) -> str:
|
||||||
|
return markdown[self.content_end : self.full_end]
|
||||||
|
|
||||||
|
def is_blank(self, markdown: str) -> bool:
|
||||||
|
return self.content_start == self.content_end
|
||||||
|
|
||||||
|
|
||||||
|
def iter_physical_lines(markdown: str) -> Iterator[PhysicalLine]:
|
||||||
|
"""Yield exact physical lines without normalizing CR or LF endings."""
|
||||||
|
line_start = 0
|
||||||
|
position = 0
|
||||||
|
while position < len(markdown):
|
||||||
|
character = markdown[position]
|
||||||
|
if character == "\n":
|
||||||
|
yield PhysicalLine(line_start, position, position + 1)
|
||||||
|
position += 1
|
||||||
|
line_start = position
|
||||||
|
elif character == "\r":
|
||||||
|
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
|
||||||
|
yield PhysicalLine(line_start, position, line_end)
|
||||||
|
position = line_end
|
||||||
|
line_start = position
|
||||||
|
else:
|
||||||
|
position += 1
|
||||||
|
|
||||||
|
if line_start < len(markdown):
|
||||||
|
yield PhysicalLine(line_start, len(markdown), len(markdown))
|
||||||
|
|
||||||
|
|
||||||
|
def physical_lines(markdown: str) -> tuple[PhysicalLine, ...]:
|
||||||
|
"""Return all physical-line ranges as an immutable sequence."""
|
||||||
|
return tuple(iter_physical_lines(markdown))
|
||||||
|
|
||||||
|
|
||||||
|
def line_ending_styles(markdown: str) -> frozenset[str]:
|
||||||
|
"""Return the distinct non-empty physical line endings in a string."""
|
||||||
|
return frozenset(
|
||||||
|
line.line_ending(markdown)
|
||||||
|
for line in iter_physical_lines(markdown)
|
||||||
|
if line.full_end > line.content_end
|
||||||
|
)
|
||||||
@@ -131,7 +131,9 @@ def _validate_bundle(
|
|||||||
*,
|
*,
|
||||||
run_date: str,
|
run_date: str,
|
||||||
run_id: str,
|
run_id: str,
|
||||||
|
run_directory: Path,
|
||||||
manifest_json: bytes,
|
manifest_json: bytes,
|
||||||
|
review_locator_json: bytes,
|
||||||
documents: tuple[StoredDocument, ...],
|
documents: tuple[StoredDocument, ...],
|
||||||
) -> None:
|
) -> None:
|
||||||
manifest = _json_object(manifest_json, "manifest")
|
manifest = _json_object(manifest_json, "manifest")
|
||||||
@@ -216,6 +218,50 @@ def _validate_bundle(
|
|||||||
if run.get("status") != overall_status or manifest.get("summary") != expected_summary:
|
if run.get("status") != overall_status or manifest.get("summary") != expected_summary:
|
||||||
raise ArtifactStoreError("manifest status or summary does not match its documents")
|
raise ArtifactStoreError("manifest status or summary does not match its documents")
|
||||||
|
|
||||||
|
locator = _json_object(review_locator_json, "review locator")
|
||||||
|
if locator.get("schema_version") != 1:
|
||||||
|
raise ArtifactStoreError("review locator schema_version must be 1")
|
||||||
|
locator_run = _nested_object(locator, "run", "review locator")
|
||||||
|
expected_run = {
|
||||||
|
"run_id": run_id,
|
||||||
|
"run_directory": str(run_directory.resolve(strict=False)),
|
||||||
|
"manifest_path": "manifest.json",
|
||||||
|
}
|
||||||
|
if locator_run != expected_run:
|
||||||
|
raise ArtifactStoreError("review locator run identity does not match the target path")
|
||||||
|
|
||||||
|
locator_documents = locator.get("documents")
|
||||||
|
if not isinstance(locator_documents, list) or len(locator_documents) != len(documents):
|
||||||
|
raise ArtifactStoreError("review locator documents do not match the stored documents")
|
||||||
|
for position, document in enumerate(documents):
|
||||||
|
raw_locator_document = locator_documents[position]
|
||||||
|
if not isinstance(raw_locator_document, dict) or not all(
|
||||||
|
isinstance(key, str) for key in raw_locator_document
|
||||||
|
):
|
||||||
|
raise ArtifactStoreError("review locator documents must be JSON objects")
|
||||||
|
locator_document = cast(dict[str, object], raw_locator_document)
|
||||||
|
source_path_value = locator_document.get("source_path")
|
||||||
|
if not isinstance(source_path_value, str) or "\0" in source_path_value:
|
||||||
|
raise ArtifactStoreError("review locator source_path must be an absolute path")
|
||||||
|
source_path = Path(source_path_value)
|
||||||
|
if not source_path.is_absolute() or source_path.resolve(strict=False) != source_path:
|
||||||
|
raise ArtifactStoreError("review locator source_path must be an absolute resolved path")
|
||||||
|
|
||||||
|
manifest_index = cast(dict[str, object], document_indexes[position])
|
||||||
|
expected_locator_document = {
|
||||||
|
"document_id": document.document_id,
|
||||||
|
"source_path": source_path_value,
|
||||||
|
"input_sha256": manifest_index.get("input_sha256"),
|
||||||
|
}
|
||||||
|
if locator_document != expected_locator_document:
|
||||||
|
raise ArtifactStoreError("review locator document identity does not match the manifest")
|
||||||
|
try:
|
||||||
|
source_bytes = source_path.read_bytes()
|
||||||
|
except OSError as error:
|
||||||
|
raise ArtifactStoreError("review locator source_path must identify a readable regular file") from error
|
||||||
|
if not source_path.is_file() or sha256(source_bytes).hexdigest() != manifest_index.get("input_sha256"):
|
||||||
|
raise ArtifactStoreError("review locator source file does not match its input hash")
|
||||||
|
|
||||||
|
|
||||||
def _rename_no_replace(source: Path, target: Path) -> None:
|
def _rename_no_replace(source: Path, target: Path) -> None:
|
||||||
directory_fd = os.open(source.parent, os.O_RDONLY | os.O_DIRECTORY)
|
directory_fd = os.open(source.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||||
@@ -235,6 +281,7 @@ def publish_run(
|
|||||||
run_date: str,
|
run_date: str,
|
||||||
run_id: str,
|
run_id: str,
|
||||||
manifest_json: bytes,
|
manifest_json: bytes,
|
||||||
|
review_locator_json: bytes,
|
||||||
documents: tuple[StoredDocument, ...],
|
documents: tuple[StoredDocument, ...],
|
||||||
) -> Path:
|
) -> Path:
|
||||||
"""Publish one complete run directory without replacing an existing run."""
|
"""Publish one complete run directory without replacing an existing run."""
|
||||||
@@ -256,7 +303,9 @@ def publish_run(
|
|||||||
_validate_bundle(
|
_validate_bundle(
|
||||||
run_date=run_date,
|
run_date=run_date,
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
|
run_directory=artifacts_root / run_date / "runs" / run_id,
|
||||||
manifest_json=manifest_json,
|
manifest_json=manifest_json,
|
||||||
|
review_locator_json=review_locator_json,
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -279,6 +328,7 @@ def publish_run(
|
|||||||
_write_private_file(document_directory / "changes.diff", document.diff)
|
_write_private_file(document_directory / "changes.diff", document.diff)
|
||||||
|
|
||||||
_write_private_file(temporary_directory / "manifest.json", manifest_json)
|
_write_private_file(temporary_directory / "manifest.json", manifest_json)
|
||||||
|
_write_private_file(temporary_directory / "review-locator.json", review_locator_json)
|
||||||
_rename_no_replace(temporary_directory, final_directory)
|
_rename_no_replace(temporary_directory, final_directory)
|
||||||
except Exception:
|
except Exception:
|
||||||
if temporary_directory.exists():
|
if temporary_directory.exists():
|
||||||
|
|||||||
@@ -1,5 +1,22 @@
|
|||||||
"""Approved business cleaning components."""
|
"""Approved business cleaning components."""
|
||||||
|
|
||||||
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
||||||
|
from mdpolish.components.html_table_double_escape import HtmlTableDoubleEscapeComponent
|
||||||
|
from mdpolish.components.html_table_layout import HtmlTableLayoutComponent
|
||||||
|
from mdpolish.components.manuscript_line_number import ManuscriptLineNumberComponent
|
||||||
|
from mdpolish.components.page_break_word_join import PageBreakWordJoinComponent, WordJoinMapping
|
||||||
|
from mdpolish.components.reference_spacing import ReferenceSpacingComponent
|
||||||
|
from mdpolish.components.repeated_running_header import RepeatedRunningHeaderComponent
|
||||||
|
from mdpolish.components.word_review_comment import WordReviewCommentComponent
|
||||||
|
|
||||||
__all__ = ["ArxivSubmissionStampComponent"]
|
__all__ = [
|
||||||
|
"ArxivSubmissionStampComponent",
|
||||||
|
"HtmlTableDoubleEscapeComponent",
|
||||||
|
"HtmlTableLayoutComponent",
|
||||||
|
"ManuscriptLineNumberComponent",
|
||||||
|
"PageBreakWordJoinComponent",
|
||||||
|
"ReferenceSpacingComponent",
|
||||||
|
"RepeatedRunningHeaderComponent",
|
||||||
|
"WordJoinMapping",
|
||||||
|
"WordReviewCommentComponent",
|
||||||
|
]
|
||||||
|
|||||||
@@ -3,9 +3,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from collections.abc import Iterator, Mapping
|
from collections.abc import Mapping
|
||||||
from types import MappingProxyType
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import iter_physical_lines
|
||||||
from mdpolish.component import Component
|
from mdpolish.component import Component
|
||||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
@@ -18,28 +19,6 @@ _EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
|||||||
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||||
|
|
||||||
|
|
||||||
def _physical_line_ranges(markdown: str) -> Iterator[tuple[int, int, int]]:
|
|
||||||
"""Yield content start, content end, and full line end for CR/LF line endings."""
|
|
||||||
line_start = 0
|
|
||||||
position = 0
|
|
||||||
while position < len(markdown):
|
|
||||||
character = markdown[position]
|
|
||||||
if character == "\n":
|
|
||||||
yield line_start, position, position + 1
|
|
||||||
position += 1
|
|
||||||
line_start = position
|
|
||||||
elif character == "\r":
|
|
||||||
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
|
|
||||||
yield line_start, position, line_end
|
|
||||||
position = line_end
|
|
||||||
line_start = position
|
|
||||||
else:
|
|
||||||
position += 1
|
|
||||||
|
|
||||||
if line_start < len(markdown):
|
|
||||||
yield line_start, len(markdown), len(markdown)
|
|
||||||
|
|
||||||
|
|
||||||
class ArxivSubmissionStampComponent(Component):
|
class ArxivSubmissionStampComponent(Component):
|
||||||
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
||||||
|
|
||||||
@@ -61,13 +40,13 @@ class ArxivSubmissionStampComponent(Component):
|
|||||||
|
|
||||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
proposals: list[ProposedChange] = []
|
proposals: list[ProposedChange] = []
|
||||||
for line_start, content_end, line_end in _physical_line_ranges(snapshot.markdown):
|
for line in iter_physical_lines(snapshot.markdown):
|
||||||
if _STAMP_PATTERN.fullmatch(snapshot.markdown[line_start:content_end]) is None:
|
if _STAMP_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||||
continue
|
continue
|
||||||
expected_text = snapshot.markdown[line_start:line_end]
|
expected_text = snapshot.markdown[line.content_start : line.full_end]
|
||||||
edit = TextEdit(
|
edit = TextEdit(
|
||||||
snapshot_sha256=snapshot.sha256,
|
snapshot_sha256=snapshot.sha256,
|
||||||
span=TextSpan(line_start, line_end),
|
span=TextSpan(line.content_start, line.full_end),
|
||||||
expected_text=expected_text,
|
expected_text=expected_text,
|
||||||
replacement="",
|
replacement="",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
"""Remove one approved escape layer from strict HTML table cell text."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._html_table import strict_html_tables
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_ENTITY_PATTERN = re.compile(r"&(?:lt|gt|amp);")
|
||||||
|
_REPLACEMENTS = {
|
||||||
|
"&lt;": "<",
|
||||||
|
"&gt;": ">",
|
||||||
|
"&amp;": "&",
|
||||||
|
}
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "在严格 HTML 表格单元格文本中解除一层实体转义"
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlTableDoubleEscapeComponent(Component):
|
||||||
|
"""Unescape only approved double entities inside strict cell text."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "markdown.html_table_double_escape"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;，"
|
||||||
|
"排除属性、表格外文本和其他实体。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for table in strict_html_tables(snapshot.markdown):
|
||||||
|
for row in table.rows:
|
||||||
|
for cell in row.cells:
|
||||||
|
for match in _ENTITY_PATTERN.finditer(snapshot.markdown, cell.content_start, cell.content_end):
|
||||||
|
edits.append((match.start(), match.end(), _REPLACEMENTS[match.group()]))
|
||||||
|
edits.sort(key=lambda item: item[0])
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Expand strict one-line HTML tables to one row per physical line."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._html_table import strict_html_tables
|
||||||
|
from mdpolish._text_ranges import line_ending_styles
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "展开严格单行 HTML 表格的行布局"
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlTableLayoutComponent(Component):
|
||||||
|
"""Preserve strict HTML table bytes while placing each row on its own line."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "markdown.html_table_layout"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理严格完整且当前为单物理行的 HTML 表格，只规范 table/tr 外层布局；"
|
||||||
|
"排除混合行尾、复杂 HTML 和已是多行的表格。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
styles = line_ending_styles(snapshot.markdown)
|
||||||
|
if len(styles) > 1:
|
||||||
|
return ()
|
||||||
|
newline = next(iter(styles), "\n")
|
||||||
|
|
||||||
|
proposals: list[ProposedChange] = []
|
||||||
|
for table in strict_html_tables(snapshot.markdown):
|
||||||
|
expected = snapshot.markdown[table.start : table.end]
|
||||||
|
if "\r" in expected or "\n" in expected:
|
||||||
|
continue
|
||||||
|
opening = snapshot.markdown[table.start : table.opening_end]
|
||||||
|
rows = [snapshot.markdown[row.start : row.end] for row in table.rows]
|
||||||
|
closing = snapshot.markdown[table.closing_start : table.end]
|
||||||
|
replacement = opening + newline + newline.join(f" {row}" for row in rows) + newline + closing
|
||||||
|
proposals.append(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(table.start, table.end),
|
||||||
|
expected_text=expected,
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(proposals)
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Remove line-number prefixes from an evidence-backed manuscript sequence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_PLAIN_PATTERN = re.compile(r"^([1-9][0-9]{0,2}) (?=\S)")
|
||||||
|
_HEADING_PATTERN = re.compile(r"^(#{1,6}) ([1-9][0-9]{0,2}) (?=\S)")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除已确认手稿序列中的行号前缀"
|
||||||
|
_MINIMUM_CANDIDATES = 20
|
||||||
|
_MINIMUM_HEADING_CANDIDATES = 2
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate(line: PhysicalLine, markdown: str) -> tuple[int, int, int, bool] | None:
|
||||||
|
content = line.content(markdown)
|
||||||
|
heading_match = _HEADING_PATTERN.match(content)
|
||||||
|
if heading_match is not None:
|
||||||
|
number_start = line.content_start + len(heading_match.group(1)) + 1
|
||||||
|
return int(heading_match.group(2)), number_start, line.content_start + heading_match.end(), True
|
||||||
|
|
||||||
|
plain_match = _PLAIN_PATTERN.match(content)
|
||||||
|
if plain_match is None:
|
||||||
|
return None
|
||||||
|
return int(plain_match.group(1)), line.content_start, line.content_start + plain_match.end(), False
|
||||||
|
|
||||||
|
|
||||||
|
class ManuscriptLineNumberComponent(Component):
|
||||||
|
"""Strip prefixes only from one long, monotonic post-Abstract sequence."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.manuscript_line_number"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理唯一 ## Abstract 后至少 20 个严格递增且含至少 2 个标题证据的手稿行号序列，"
|
||||||
|
"排除作者单位、列表和短数字片段。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
abstract_indexes = [
|
||||||
|
index for index, line in enumerate(lines) if line.content(snapshot.markdown) == "## Abstract"
|
||||||
|
]
|
||||||
|
if len(abstract_indexes) != 1:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
candidates = [
|
||||||
|
candidate
|
||||||
|
for line in lines[abstract_indexes[0] + 1 :]
|
||||||
|
if (candidate := _candidate(line, snapshot.markdown)) is not None
|
||||||
|
]
|
||||||
|
if len(candidates) < _MINIMUM_CANDIDATES:
|
||||||
|
return ()
|
||||||
|
if sum(is_heading for _, _, _, is_heading in candidates) < _MINIMUM_HEADING_CANDIDATES:
|
||||||
|
return ()
|
||||||
|
|
||||||
|
numbers = [number for number, _, _, _ in candidates]
|
||||||
|
if any(previous >= current for previous, current in pairwise(numbers)):
|
||||||
|
return ()
|
||||||
|
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement="",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for _, start, end, _ in candidates
|
||||||
|
)
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
"""Join approved word fragments split across adjacent physical lines."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import Counter
|
||||||
|
from collections.abc import Iterable, Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import physical_lines
|
||||||
|
from mdpolish.component import Component, ComponentContractError
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
WordJoinMapping = tuple[str, str, str]
|
||||||
|
|
||||||
|
|
||||||
|
def _is_ascii_letter(character: str) -> bool:
|
||||||
|
return character in "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||||
|
|
||||||
|
|
||||||
|
def _validated_mappings(mappings: Iterable[WordJoinMapping]) -> tuple[WordJoinMapping, ...]:
|
||||||
|
normalized: list[WordJoinMapping] = []
|
||||||
|
seen_pairs: set[tuple[str, str]] = set()
|
||||||
|
for mapping in mappings:
|
||||||
|
if not isinstance(mapping, tuple) or len(mapping) != 3 or any(not isinstance(value, str) for value in mapping):
|
||||||
|
raise ComponentContractError("word join mappings must be three-string tuples")
|
||||||
|
left, right, replacement = mapping
|
||||||
|
if not left or not right or not replacement:
|
||||||
|
raise ComponentContractError("word join mapping fields must be non-empty")
|
||||||
|
if any(character.isspace() for value in mapping for character in value):
|
||||||
|
raise ComponentContractError("word join mapping fields cannot contain whitespace")
|
||||||
|
pair = (left, right)
|
||||||
|
if pair in seen_pairs:
|
||||||
|
raise ComponentContractError("word join mappings cannot repeat a fragment pair")
|
||||||
|
seen_pairs.add(pair)
|
||||||
|
normalized.append(mapping)
|
||||||
|
return tuple(sorted(normalized))
|
||||||
|
|
||||||
|
|
||||||
|
class PageBreakWordJoinComponent(Component):
|
||||||
|
"""Apply only caller-approved exact fragment-to-word mappings."""
|
||||||
|
|
||||||
|
def __init__(self, mappings: Iterable[WordJoinMapping]) -> None:
|
||||||
|
self._mappings = _validated_mappings(mappings)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.page_break_word_join"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
records = tuple(
|
||||||
|
MappingProxyType(
|
||||||
|
{
|
||||||
|
"left_fragment": left,
|
||||||
|
"right_fragment": right,
|
||||||
|
"replacement": replacement,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
for left, right, replacement in self._mappings
|
||||||
|
)
|
||||||
|
return MappingProxyType({"mappings": records})
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理显式批准的左右词片段，片段必须位于相邻物理行或只隔一个空行并满足 ASCII 词边界；"
|
||||||
|
"排除未配置词和段落级猜测。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
matches: list[tuple[int, int, str, str, str]] = []
|
||||||
|
for index, left_line in enumerate(lines[:-1]):
|
||||||
|
left_text = left_line.content(snapshot.markdown)
|
||||||
|
left_ending = left_line.line_ending(snapshot.markdown)
|
||||||
|
if not left_ending:
|
||||||
|
continue
|
||||||
|
for left, right, replacement in self._mappings:
|
||||||
|
if not left_text.endswith(left):
|
||||||
|
continue
|
||||||
|
left_start = left_line.content_end - len(left)
|
||||||
|
if left_start > left_line.content_start and _is_ascii_letter(snapshot.markdown[left_start - 1]):
|
||||||
|
continue
|
||||||
|
|
||||||
|
right_index = index + 1
|
||||||
|
if lines[right_index].is_blank(snapshot.markdown):
|
||||||
|
blank = lines[right_index]
|
||||||
|
if blank.line_ending(snapshot.markdown) != left_ending or right_index + 1 >= len(lines):
|
||||||
|
continue
|
||||||
|
right_index += 1
|
||||||
|
if lines[right_index].is_blank(snapshot.markdown):
|
||||||
|
continue
|
||||||
|
right_line = lines[right_index]
|
||||||
|
right_text = right_line.content(snapshot.markdown)
|
||||||
|
if not right_text.startswith(right):
|
||||||
|
continue
|
||||||
|
if len(right_text) > len(right) and _is_ascii_letter(right_text[len(right)]):
|
||||||
|
continue
|
||||||
|
end = right_line.content_start + len(right)
|
||||||
|
matches.append((left_start, end, replacement, left, right))
|
||||||
|
|
||||||
|
matches.sort(key=lambda item: (item[0], item[1], item[2]))
|
||||||
|
span_counts = Counter(match[:2] for match in matches)
|
||||||
|
unique_matches = [match for match in matches if span_counts[match[:2]] == 1]
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(unique_matches)):
|
||||||
|
return ()
|
||||||
|
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=f"按已批准映射修复跨页断词：{left} + {right} → {replacement}",
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement, left, right in unique_matches
|
||||||
|
)
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
"""Normalize blank lines between numbered entries in References sections."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_ATX_HEADING_PATTERN = re.compile(r"^(#{1,6}) (\S(?:.*\S)?)$")
|
||||||
|
_ENTRY_PATTERN = re.compile(r"^([1-9][0-9]*)\. \S")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "统一 References 章节中相邻编号条目之间的一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
def _heading(line: PhysicalLine, markdown: str) -> tuple[int, str] | None:
|
||||||
|
match = _ATX_HEADING_PATTERN.fullmatch(line.content(markdown))
|
||||||
|
if match is None:
|
||||||
|
return None
|
||||||
|
return len(match.group(1)), match.group(2)
|
||||||
|
|
||||||
|
|
||||||
|
class ReferenceSpacingComponent(Component):
|
||||||
|
"""Normalize only complete, consecutive numbered reference sections."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.reference_spacing"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理准确 References ATX 章节内从 1 开始逐项递增的编号条目，统一为一个空行；"
|
||||||
|
"排除章节外列表、缺号和混合行尾边界。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for heading_index, line in enumerate(lines):
|
||||||
|
heading = _heading(line, snapshot.markdown)
|
||||||
|
if heading is None or heading[1].lower() != "references":
|
||||||
|
continue
|
||||||
|
heading_level = heading[0]
|
||||||
|
section_end = len(lines)
|
||||||
|
for index in range(heading_index + 1, len(lines)):
|
||||||
|
following_heading = _heading(lines[index], snapshot.markdown)
|
||||||
|
if following_heading is not None and following_heading[0] <= heading_level:
|
||||||
|
section_end = index
|
||||||
|
break
|
||||||
|
|
||||||
|
entries: list[tuple[int, int]] = []
|
||||||
|
section_is_safe = True
|
||||||
|
for index in range(heading_index + 1, section_end):
|
||||||
|
content = lines[index].content(snapshot.markdown)
|
||||||
|
match = _ENTRY_PATTERN.match(content)
|
||||||
|
if match is not None:
|
||||||
|
entries.append((index, int(match.group(1))))
|
||||||
|
elif _heading(lines[index], snapshot.markdown) is not None:
|
||||||
|
section_is_safe = False
|
||||||
|
if not section_is_safe or len(entries) < 2:
|
||||||
|
continue
|
||||||
|
if [number for _, number in entries] != list(range(1, len(entries) + 1)):
|
||||||
|
continue
|
||||||
|
|
||||||
|
section_edits: list[tuple[int, int, str]] = []
|
||||||
|
for previous, following in pairwise(entries):
|
||||||
|
previous_index = previous[0]
|
||||||
|
next_index = following[0]
|
||||||
|
last_nonblank_index = next(
|
||||||
|
index
|
||||||
|
for index in range(next_index - 1, previous_index - 1, -1)
|
||||||
|
if not lines[index].is_blank(snapshot.markdown)
|
||||||
|
)
|
||||||
|
between = lines[last_nonblank_index + 1 : next_index]
|
||||||
|
if any(not blank.is_blank(snapshot.markdown) for blank in between):
|
||||||
|
section_is_safe = False
|
||||||
|
break
|
||||||
|
line_ending = lines[last_nonblank_index].line_ending(snapshot.markdown)
|
||||||
|
if not line_ending or any(blank.line_ending(snapshot.markdown) != line_ending for blank in between):
|
||||||
|
section_is_safe = False
|
||||||
|
break
|
||||||
|
start = lines[last_nonblank_index].content_end
|
||||||
|
end = lines[next_index].content_start
|
||||||
|
replacement = line_ending * 2
|
||||||
|
if snapshot.markdown[start:end] != replacement:
|
||||||
|
section_edits.append((start, end, replacement))
|
||||||
|
if section_is_safe:
|
||||||
|
edits.extend(section_edits)
|
||||||
|
|
||||||
|
edits.sort(key=lambda item: item[0])
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||||
|
return ()
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""Remove repeated ATX running headers with paragraph-interruption evidence."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import defaultdict
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from itertools import pairwise
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_HEADING_PATTERN = re.compile(r"^#{1,6} \S(?:.*\S)?$")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除经重复和断句证据确认的跑动页眉"
|
||||||
|
_TERMINAL_PUNCTUATION = frozenset(".?!:;。！？：；")
|
||||||
|
|
||||||
|
|
||||||
|
def _bridge_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||||
|
if index < 2 or index + 2 >= len(lines):
|
||||||
|
return None
|
||||||
|
before_blank = lines[index - 1]
|
||||||
|
after_blank = lines[index + 1]
|
||||||
|
previous = lines[index - 2]
|
||||||
|
following = lines[index + 2]
|
||||||
|
if not before_blank.is_blank(markdown) or not after_blank.is_blank(markdown):
|
||||||
|
return None
|
||||||
|
if previous.is_blank(markdown) or following.is_blank(markdown):
|
||||||
|
return None
|
||||||
|
|
||||||
|
previous_text = previous.content(markdown).rstrip()
|
||||||
|
following_text = following.content(markdown)
|
||||||
|
if not previous_text or previous_text[-1] in _TERMINAL_PUNCTUATION:
|
||||||
|
return None
|
||||||
|
if not following_text or following_text[0] not in "abcdefghijklmnopqrstuvwxyz":
|
||||||
|
return None
|
||||||
|
return previous.content_end, following.content_start
|
||||||
|
|
||||||
|
|
||||||
|
def _deletion_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||||
|
if index + 1 >= len(lines) or not lines[index + 1].is_blank(markdown):
|
||||||
|
return None
|
||||||
|
return lines[index].content_start, lines[index + 1].full_end
|
||||||
|
|
||||||
|
|
||||||
|
class RepeatedRunningHeaderComponent(Component):
|
||||||
|
"""Remove a repeated heading only when one occurrence splits a sentence."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.repeated_running_header"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return (
|
||||||
|
"处理原样重复至少两次、且至少一次由小写续句证据确认切断正文的 ATX 跑动页眉；"
|
||||||
|
"任一出现位置不安全时整组保留。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
groups: defaultdict[str, list[int]] = defaultdict(list)
|
||||||
|
for index, line in enumerate(lines):
|
||||||
|
content = line.content(snapshot.markdown)
|
||||||
|
if _HEADING_PATTERN.fullmatch(content) is not None:
|
||||||
|
groups[content].append(index)
|
||||||
|
|
||||||
|
edits: list[tuple[int, int, str]] = []
|
||||||
|
for indexes in groups.values():
|
||||||
|
if len(indexes) < 2:
|
||||||
|
continue
|
||||||
|
group_edits: list[tuple[int, int, str]] = []
|
||||||
|
bridge_found = False
|
||||||
|
for index in indexes:
|
||||||
|
bridge = _bridge_span(lines, index, snapshot.markdown)
|
||||||
|
if bridge is not None:
|
||||||
|
group_edits.append((*bridge, " "))
|
||||||
|
bridge_found = True
|
||||||
|
continue
|
||||||
|
deletion = _deletion_span(lines, index, snapshot.markdown)
|
||||||
|
if deletion is None:
|
||||||
|
group_edits = []
|
||||||
|
break
|
||||||
|
group_edits.append((*deletion, ""))
|
||||||
|
if bridge_found and group_edits:
|
||||||
|
edits.extend(group_edits)
|
||||||
|
|
||||||
|
edits.sort(key=lambda item: (item[0], item[1]))
|
||||||
|
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||||
|
return ()
|
||||||
|
return tuple(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement=replacement,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for start, end, replacement in edits
|
||||||
|
)
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""Remove strict single-line Word review comments."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
from mdpolish._text_ranges import physical_lines
|
||||||
|
from mdpolish.component import Component
|
||||||
|
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||||
|
|
||||||
|
_COMMENT_PATTERN = re.compile(r"^Commented \[[A-Za-z0-9]+\]: .*\S$")
|
||||||
|
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||||
|
_REASON = "删除严格单行 Word 审阅批注及其后一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
class WordReviewCommentComponent(Component):
|
||||||
|
"""Delete approved one-line Word comments followed by a blank line."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def component_id(self) -> str:
|
||||||
|
return "paper.word_review_comment"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return "1.0.0"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parameters(self) -> Mapping[str, object]:
|
||||||
|
return _EMPTY_PARAMETERS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def applicability(self) -> str:
|
||||||
|
return "处理独占物理行且后接空行的 Word Commented 审阅批注，排除多行正文和所有相似文本。"
|
||||||
|
|
||||||
|
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||||
|
lines = physical_lines(snapshot.markdown)
|
||||||
|
proposals: list[ProposedChange] = []
|
||||||
|
for index, line in enumerate(lines[:-1]):
|
||||||
|
following = lines[index + 1]
|
||||||
|
if _COMMENT_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||||
|
continue
|
||||||
|
if not following.is_blank(snapshot.markdown):
|
||||||
|
continue
|
||||||
|
start = line.content_start
|
||||||
|
end = following.full_end
|
||||||
|
proposals.append(
|
||||||
|
ProposedChange(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
reason=_REASON,
|
||||||
|
edits=(
|
||||||
|
TextEdit(
|
||||||
|
snapshot_sha256=snapshot.sha256,
|
||||||
|
span=TextSpan(start, end),
|
||||||
|
expected_text=snapshot.markdown[start:end],
|
||||||
|
replacement="",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(proposals)
|
||||||
@@ -276,6 +276,31 @@ def _manifest_payload(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _review_locator_payload(
|
||||||
|
*,
|
||||||
|
run_id: str,
|
||||||
|
run_directory: Path,
|
||||||
|
documents: tuple[_PreparedDocument, ...],
|
||||||
|
) -> JsonObject:
|
||||||
|
document_values: list[JsonValue] = [
|
||||||
|
{
|
||||||
|
"document_id": document.input.document_id,
|
||||||
|
"source_path": str(document.resolved_path),
|
||||||
|
"input_sha256": document.input_sha256,
|
||||||
|
}
|
||||||
|
for document in documents
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": {
|
||||||
|
"run_id": run_id,
|
||||||
|
"run_directory": str(run_directory.resolve(strict=False)),
|
||||||
|
"manifest_path": "manifest.json",
|
||||||
|
},
|
||||||
|
"documents": document_values,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def run_experiment(
|
def run_experiment(
|
||||||
*,
|
*,
|
||||||
pipeline: Pipeline,
|
pipeline: Pipeline,
|
||||||
@@ -365,6 +390,13 @@ def run_experiment(
|
|||||||
run_date=run_date,
|
run_date=run_date,
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
manifest_json=encode_json(manifest),
|
manifest_json=encode_json(manifest),
|
||||||
|
review_locator_json=encode_json(
|
||||||
|
_review_locator_payload(
|
||||||
|
run_id=run_id,
|
||||||
|
run_directory=final_directory,
|
||||||
|
documents=prepared,
|
||||||
|
)
|
||||||
|
),
|
||||||
documents=stored_documents,
|
documents=stored_documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
+35
-92
@@ -7,6 +7,12 @@ from dataclasses import dataclass
|
|||||||
from difflib import unified_diff
|
from difflib import unified_diff
|
||||||
from typing import TypeAlias
|
from typing import TypeAlias
|
||||||
|
|
||||||
|
from mdpolish._artifact_replay import (
|
||||||
|
ReplayChange,
|
||||||
|
ReplayComponent,
|
||||||
|
ReplayError,
|
||||||
|
replay_change_chain,
|
||||||
|
)
|
||||||
from mdpolish.models import (
|
from mdpolish.models import (
|
||||||
Change,
|
Change,
|
||||||
ComponentInfo,
|
ComponentInfo,
|
||||||
@@ -17,15 +23,13 @@ from mdpolish.models import (
|
|||||||
RunStatus,
|
RunStatus,
|
||||||
TextEdit,
|
TextEdit,
|
||||||
TransformResult,
|
TransformResult,
|
||||||
markdown_sha256,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||||
JsonObject: TypeAlias = dict[str, JsonValue]
|
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||||
|
|
||||||
|
|
||||||
class ReportingError(ValueError):
|
ReportingError = ReplayError
|
||||||
"""A core result cannot be represented without losing its audit contract."""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
@@ -115,99 +119,38 @@ def _run_error_json(error: RunError) -> JsonObject:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _line_column(markdown: str, offset: int) -> tuple[int, int]:
|
|
||||||
if offset < 0 or offset > len(markdown):
|
|
||||||
raise ReportingError("a change offset is outside its recorded snapshot")
|
|
||||||
|
|
||||||
line = 1
|
|
||||||
column = 1
|
|
||||||
position = 0
|
|
||||||
while position < offset:
|
|
||||||
character = markdown[position]
|
|
||||||
if character == "\r":
|
|
||||||
line += 1
|
|
||||||
column = 1
|
|
||||||
if position + 1 < offset and markdown[position + 1] == "\n":
|
|
||||||
position += 2
|
|
||||||
else:
|
|
||||||
position += 1
|
|
||||||
elif character == "\n":
|
|
||||||
line += 1
|
|
||||||
column = 1
|
|
||||||
position += 1
|
|
||||||
else:
|
|
||||||
column += 1
|
|
||||||
position += 1
|
|
||||||
return line, column
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_change_identity(change: Change, components: tuple[ComponentInfo, ...]) -> None:
|
|
||||||
if change.component_position < 0 or change.component_position >= len(components):
|
|
||||||
raise ReportingError("a change has no matching component position")
|
|
||||||
component = components[change.component_position]
|
|
||||||
if change.component_id != component.component_id or change.component_version != component.version:
|
|
||||||
raise ReportingError("a change identity does not match the recorded component")
|
|
||||||
if change.proposal_ref.component_position != change.component_position:
|
|
||||||
raise ReportingError("a change proposal reference has the wrong component position")
|
|
||||||
if change.proposal_ref.snapshot_sha256 != change.before_sha256:
|
|
||||||
raise ReportingError("a change proposal reference targets the wrong snapshot")
|
|
||||||
|
|
||||||
|
|
||||||
def _replay_changes(input_markdown: str, result: TransformResult) -> tuple[tuple[tuple[int, int], ...], str]:
|
def _replay_changes(input_markdown: str, result: TransformResult) -> tuple[tuple[tuple[int, int], ...], str]:
|
||||||
if markdown_sha256(input_markdown) != result.input_sha256:
|
|
||||||
raise ReportingError("input Markdown does not match the transform result")
|
|
||||||
|
|
||||||
current = input_markdown
|
|
||||||
locations: list[tuple[int, int]] = []
|
|
||||||
cursor = 0
|
|
||||||
while cursor < len(result.changes):
|
|
||||||
first = result.changes[cursor]
|
|
||||||
batch_key = (first.component_position, first.before_sha256, first.after_sha256)
|
|
||||||
batch: list[Change] = []
|
|
||||||
while cursor < len(result.changes):
|
|
||||||
candidate = result.changes[cursor]
|
|
||||||
candidate_key = (candidate.component_position, candidate.before_sha256, candidate.after_sha256)
|
|
||||||
if candidate_key != batch_key:
|
|
||||||
break
|
|
||||||
batch.append(candidate)
|
|
||||||
cursor += 1
|
|
||||||
|
|
||||||
if markdown_sha256(current) != first.before_sha256:
|
|
||||||
raise ReportingError("a change batch does not follow the recorded snapshot chain")
|
|
||||||
|
|
||||||
for change in batch:
|
|
||||||
_validate_change_identity(change, result.components)
|
|
||||||
if change.before_sha256 != first.before_sha256 or change.after_sha256 != first.after_sha256:
|
|
||||||
raise ReportingError("a change batch contains inconsistent snapshot hashes")
|
|
||||||
if change.span.end > len(current):
|
|
||||||
raise ReportingError("a change span is outside its recorded snapshot")
|
|
||||||
if len(change.before) != change.span.end - change.span.start:
|
|
||||||
raise ReportingError("a change before value does not match its span length")
|
|
||||||
if current[change.span.start : change.span.end] != change.before:
|
|
||||||
raise ReportingError("a change before value does not match its recorded snapshot")
|
|
||||||
locations.append(_line_column(current, change.span.start))
|
|
||||||
|
|
||||||
application_order = sorted(
|
|
||||||
batch,
|
|
||||||
key=lambda change: (
|
|
||||||
change.span.start,
|
|
||||||
change.span.end,
|
|
||||||
change.proposal_ref.proposal_index,
|
|
||||||
change.edit_index,
|
|
||||||
),
|
|
||||||
reverse=True,
|
|
||||||
)
|
|
||||||
for change in application_order:
|
|
||||||
current = current[: change.span.start] + change.after + current[change.span.end :]
|
|
||||||
if markdown_sha256(current) != first.after_sha256:
|
|
||||||
raise ReportingError("replayed changes do not produce the recorded batch hash")
|
|
||||||
|
|
||||||
current_markdown = result.output_markdown if result.status is RunStatus.SUCCESS else result.partial_markdown
|
current_markdown = result.output_markdown if result.status is RunStatus.SUCCESS else result.partial_markdown
|
||||||
if current_markdown is None:
|
if current_markdown is None:
|
||||||
raise ReportingError("a transform result does not contain its status-specific Markdown")
|
raise ReportingError("a transform result does not contain its status-specific Markdown")
|
||||||
if current != current_markdown or markdown_sha256(current) != result.current_sha256:
|
replayed = replay_change_chain(
|
||||||
raise ReportingError("replayed changes do not produce the transform result's current snapshot")
|
input_markdown=input_markdown,
|
||||||
return tuple(locations), current
|
input_sha256=result.input_sha256,
|
||||||
|
components=tuple(ReplayComponent(item.component_id, item.version) for item in result.components),
|
||||||
|
changes=tuple(
|
||||||
|
ReplayChange(
|
||||||
|
component_id=change.component_id,
|
||||||
|
component_version=change.component_version,
|
||||||
|
component_position=change.component_position,
|
||||||
|
proposal_component_position=change.proposal_ref.component_position,
|
||||||
|
proposal_snapshot_sha256=change.proposal_ref.snapshot_sha256,
|
||||||
|
proposal_index=change.proposal_ref.proposal_index,
|
||||||
|
edit_index=change.edit_index,
|
||||||
|
start=change.span.start,
|
||||||
|
end=change.span.end,
|
||||||
|
before=change.before,
|
||||||
|
after=change.after,
|
||||||
|
before_sha256=change.before_sha256,
|
||||||
|
after_sha256=change.after_sha256,
|
||||||
|
)
|
||||||
|
for change in result.changes
|
||||||
|
),
|
||||||
|
current_sha256=result.current_sha256,
|
||||||
|
current_markdown=current_markdown,
|
||||||
|
include_zero_change_stages=False,
|
||||||
|
)
|
||||||
|
locations = tuple((item.line, item.column) for item in replayed.changes)
|
||||||
|
return locations, replayed.current_markdown
|
||||||
|
|
||||||
|
|
||||||
def _change_json(change: Change, location: tuple[int, int]) -> JsonObject:
|
def _change_json(change: Change, location: tuple[int, int]) -> JsonObject:
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Test support package for repository-local integration fixtures."""
|
||||||
@@ -0,0 +1,212 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from hashlib import sha256
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, TypedDict, cast
|
||||||
|
|
||||||
|
|
||||||
|
class ReviewFixture(TypedDict):
|
||||||
|
run_directory: Path
|
||||||
|
source_path: Path
|
||||||
|
manifest_path: Path
|
||||||
|
locator_path: Path
|
||||||
|
result_path: Path
|
||||||
|
original: str
|
||||||
|
cleaned: str
|
||||||
|
|
||||||
|
|
||||||
|
def digest(text: str) -> str:
|
||||||
|
return sha256(text.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def write_json(path: Path, payload: object) -> None:
|
||||||
|
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def read_json(path: Path) -> dict[str, Any]:
|
||||||
|
return cast(dict[str, Any], json.loads(path.read_text(encoding="utf-8")))
|
||||||
|
|
||||||
|
|
||||||
|
def create_review_run(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
status: str = "success",
|
||||||
|
with_locator: bool = True,
|
||||||
|
) -> ReviewFixture:
|
||||||
|
run_directory = root / "artifacts/2026-08-23/runs/review-run"
|
||||||
|
document_directory = run_directory / "documents/paper"
|
||||||
|
document_directory.mkdir(parents=True)
|
||||||
|
source_path = root / "paper.md"
|
||||||
|
original = "\ufeff😀 old\r\nCafe\u0301\n"
|
||||||
|
cleaned = "\ufeff😀 new\r\nCafe\u0301\n"
|
||||||
|
source_path.write_text(original, encoding="utf-8")
|
||||||
|
input_hash = digest(original)
|
||||||
|
current_hash = digest(cleaned) if status == "success" else input_hash
|
||||||
|
changes: list[dict[str, object]] = []
|
||||||
|
if status == "success":
|
||||||
|
changes.append(
|
||||||
|
{
|
||||||
|
"component_id": "paper.rule",
|
||||||
|
"component_version": "1.0.0",
|
||||||
|
"component_position": 0,
|
||||||
|
"proposal_ref": {
|
||||||
|
"component_position": 0,
|
||||||
|
"snapshot_sha256": input_hash,
|
||||||
|
"proposal_index": 0,
|
||||||
|
},
|
||||||
|
"edit_index": 0,
|
||||||
|
"reason": "替换测试单词",
|
||||||
|
"span": {"start": 3, "end": 6},
|
||||||
|
"location": {"line": 1, "column": 4},
|
||||||
|
"before": "old",
|
||||||
|
"after": "new",
|
||||||
|
"before_sha256": input_hash,
|
||||||
|
"after_sha256": current_hash,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
errors: list[dict[str, object]] = []
|
||||||
|
if status == "failed":
|
||||||
|
errors.append(
|
||||||
|
{
|
||||||
|
"component_id": "paper.rule",
|
||||||
|
"component_version": "1.0.0",
|
||||||
|
"component_position": 0,
|
||||||
|
"stage": "transform",
|
||||||
|
"error_type": "SyntheticError",
|
||||||
|
"message": "测试组件失败。",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
residuals: list[dict[str, object]] = []
|
||||||
|
if status == "unstable":
|
||||||
|
residuals.append(
|
||||||
|
{
|
||||||
|
"component_id": "paper.rule",
|
||||||
|
"component_version": "1.0.0",
|
||||||
|
"component_position": 0,
|
||||||
|
"proposal_ref": {
|
||||||
|
"component_position": 0,
|
||||||
|
"snapshot_sha256": input_hash,
|
||||||
|
"proposal_index": 0,
|
||||||
|
},
|
||||||
|
"proposal": {
|
||||||
|
"snapshot_sha256": input_hash,
|
||||||
|
"reason": "仍可替换测试单词",
|
||||||
|
"edits": [
|
||||||
|
{
|
||||||
|
"snapshot_sha256": input_hash,
|
||||||
|
"span": {"start": 3, "end": 6},
|
||||||
|
"expected_text": "old",
|
||||||
|
"replacement": "new",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
result = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"document": {"document_id": "paper", "source_label": "inputs/paper.md"},
|
||||||
|
"status": status,
|
||||||
|
"input_sha256": input_hash,
|
||||||
|
"current_sha256": current_hash,
|
||||||
|
"changes": changes,
|
||||||
|
"errors": errors,
|
||||||
|
"residual_proposals": residuals,
|
||||||
|
"output": {
|
||||||
|
"cleaned_path": "cleaned.md" if status == "success" else None,
|
||||||
|
"diff_path": "changes.diff" if status == "success" else None,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result_path = document_directory / "result.json"
|
||||||
|
write_json(result_path, result)
|
||||||
|
if status == "success":
|
||||||
|
(document_directory / "cleaned.md").write_text(cleaned, encoding="utf-8")
|
||||||
|
(document_directory / "changes.diff").write_text("synthetic diff\n", encoding="utf-8")
|
||||||
|
|
||||||
|
manifest = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": {
|
||||||
|
"run_id": "review-run",
|
||||||
|
"run_date": "2026-08-23",
|
||||||
|
"utc_offset": "+08:00",
|
||||||
|
"status": status,
|
||||||
|
"started_at_utc": "2026-08-23T01:00:00Z",
|
||||||
|
"completed_at_utc": "2026-08-23T01:01:00Z",
|
||||||
|
"retention_until": "2026-09-22T01:01:00Z",
|
||||||
|
},
|
||||||
|
"tool": {
|
||||||
|
"name": "mdpolish",
|
||||||
|
"package_version": "0.1.0",
|
||||||
|
"python_version": "3.13.11",
|
||||||
|
"platform": "linux-x86_64",
|
||||||
|
"git_commit": None,
|
||||||
|
"git_dirty": None,
|
||||||
|
},
|
||||||
|
"pipeline": {
|
||||||
|
"components": [
|
||||||
|
{
|
||||||
|
"component_id": "paper.rule",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"parameters": [],
|
||||||
|
"applicability": "替换测试单词。",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"component_id": "paper.zero",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"parameters": [],
|
||||||
|
"applicability": "不修改当前测试文档。",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"documents": [
|
||||||
|
{
|
||||||
|
"document_id": "paper",
|
||||||
|
"source_label": "inputs/paper.md",
|
||||||
|
"status": status,
|
||||||
|
"input_sha256": input_hash,
|
||||||
|
"current_sha256": current_hash,
|
||||||
|
"change_count": len(changes),
|
||||||
|
"result_path": "documents/paper/result.json",
|
||||||
|
"cleaned_path": "documents/paper/cleaned.md" if status == "success" else None,
|
||||||
|
"diff_path": "documents/paper/changes.diff" if status == "success" else None,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"summary": {
|
||||||
|
"document_count": 1,
|
||||||
|
"success_count": int(status == "success"),
|
||||||
|
"failed_count": int(status == "failed"),
|
||||||
|
"unstable_count": int(status == "unstable"),
|
||||||
|
"change_count": len(changes),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
manifest_path = run_directory / "manifest.json"
|
||||||
|
write_json(manifest_path, manifest)
|
||||||
|
locator_path = run_directory / "review-locator.json"
|
||||||
|
if with_locator:
|
||||||
|
write_json(
|
||||||
|
locator_path,
|
||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": {
|
||||||
|
"run_id": "review-run",
|
||||||
|
"run_directory": str(run_directory.resolve()),
|
||||||
|
"manifest_path": "manifest.json",
|
||||||
|
},
|
||||||
|
"documents": [
|
||||||
|
{
|
||||||
|
"document_id": "paper",
|
||||||
|
"source_path": str(source_path.resolve()),
|
||||||
|
"input_sha256": input_hash,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"run_directory": run_directory,
|
||||||
|
"source_path": source_path,
|
||||||
|
"manifest_path": manifest_path,
|
||||||
|
"locator_path": locator_path,
|
||||||
|
"result_path": result_path,
|
||||||
|
"original": original,
|
||||||
|
"cleaned": cleaned,
|
||||||
|
}
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish._artifact_replay import (
|
||||||
|
ReplayChange,
|
||||||
|
ReplayComponent,
|
||||||
|
ReplayError,
|
||||||
|
replay_change_chain,
|
||||||
|
)
|
||||||
|
from mdpolish.models import markdown_sha256
|
||||||
|
|
||||||
|
|
||||||
|
def change(
|
||||||
|
*,
|
||||||
|
component_position: int,
|
||||||
|
before_text: str,
|
||||||
|
after_text: str,
|
||||||
|
start: int,
|
||||||
|
end: int,
|
||||||
|
before_sha256: str,
|
||||||
|
after_sha256: str,
|
||||||
|
proposal_index: int = 0,
|
||||||
|
edit_index: int = 0,
|
||||||
|
) -> ReplayChange:
|
||||||
|
return ReplayChange(
|
||||||
|
component_id=f"test.{component_position}",
|
||||||
|
component_version="1.0.0",
|
||||||
|
component_position=component_position,
|
||||||
|
proposal_component_position=component_position,
|
||||||
|
proposal_snapshot_sha256=before_sha256,
|
||||||
|
proposal_index=proposal_index,
|
||||||
|
edit_index=edit_index,
|
||||||
|
start=start,
|
||||||
|
end=end,
|
||||||
|
before=before_text,
|
||||||
|
after=after_text,
|
||||||
|
before_sha256=before_sha256,
|
||||||
|
after_sha256=after_sha256,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_builds_zero_change_stages_and_utf16_editor_ranges() -> None:
|
||||||
|
original = "\ufeff😀 old\r\nCafe\u0301"
|
||||||
|
final = "\ufeff😀 new\r\nCafe\u0301"
|
||||||
|
input_hash = markdown_sha256(original)
|
||||||
|
final_hash = markdown_sha256(final)
|
||||||
|
recorded = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="old",
|
||||||
|
after_text="new",
|
||||||
|
start=3,
|
||||||
|
end=6,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256=final_hash,
|
||||||
|
)
|
||||||
|
|
||||||
|
replayed = replay_change_chain(
|
||||||
|
input_markdown=original,
|
||||||
|
input_sha256=input_hash,
|
||||||
|
components=(ReplayComponent("test.0", "1.0.0"), ReplayComponent("test.1", "1.0.0")),
|
||||||
|
changes=(recorded,),
|
||||||
|
current_sha256=final_hash,
|
||||||
|
current_markdown=final,
|
||||||
|
include_zero_change_stages=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert replayed.current_markdown == final
|
||||||
|
assert len(replayed.stages) == 2
|
||||||
|
assert replayed.stages[1].before_markdown == final
|
||||||
|
assert replayed.stages[1].after_markdown == final
|
||||||
|
assert replayed.stages[1].changes == ()
|
||||||
|
assert (replayed.changes[0].line, replayed.changes[0].column) == (1, 4)
|
||||||
|
assert (replayed.changes[0].editor_start, replayed.changes[0].editor_end) == (4, 7)
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_uses_full_descending_application_key_not_record_order() -> None:
|
||||||
|
original = "abcd"
|
||||||
|
final = "aXXcYY"
|
||||||
|
input_hash = markdown_sha256(original)
|
||||||
|
final_hash = markdown_sha256(final)
|
||||||
|
right = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="d",
|
||||||
|
after_text="YY",
|
||||||
|
start=3,
|
||||||
|
end=4,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256=final_hash,
|
||||||
|
proposal_index=1,
|
||||||
|
)
|
||||||
|
left = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="b",
|
||||||
|
after_text="XX",
|
||||||
|
start=1,
|
||||||
|
end=2,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256=final_hash,
|
||||||
|
)
|
||||||
|
|
||||||
|
replayed = replay_change_chain(
|
||||||
|
input_markdown=original,
|
||||||
|
input_sha256=input_hash,
|
||||||
|
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||||
|
changes=(right, left),
|
||||||
|
current_sha256=final_hash,
|
||||||
|
current_markdown=final,
|
||||||
|
include_zero_change_stages=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert replayed.current_markdown == final
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("mutate", "message"),
|
||||||
|
[
|
||||||
|
(lambda item: replace(item, component_position=2), "component position"),
|
||||||
|
(lambda item: replace(item, proposal_component_position=1), "proposal reference"),
|
||||||
|
(lambda item: replace(item, before="bad"), "recorded snapshot"),
|
||||||
|
(lambda item: replace(item, after_sha256="0" * 64), "batch hash"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_replay_rejects_untrusted_change_chains(
|
||||||
|
mutate: Callable[[ReplayChange], ReplayChange], message: str
|
||||||
|
) -> None:
|
||||||
|
original = "old"
|
||||||
|
final = "new"
|
||||||
|
input_hash = markdown_sha256(original)
|
||||||
|
final_hash = markdown_sha256(final)
|
||||||
|
valid = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="old",
|
||||||
|
after_text="new",
|
||||||
|
start=0,
|
||||||
|
end=3,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256=final_hash,
|
||||||
|
)
|
||||||
|
tampered = mutate(valid)
|
||||||
|
|
||||||
|
with pytest.raises(ReplayError, match=message):
|
||||||
|
replay_change_chain(
|
||||||
|
input_markdown=original,
|
||||||
|
input_sha256=input_hash,
|
||||||
|
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||||
|
changes=(tampered,),
|
||||||
|
current_sha256=final_hash,
|
||||||
|
current_markdown=final,
|
||||||
|
include_zero_change_stages=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_replay_rejects_conflicting_ranges() -> None:
|
||||||
|
original = "abc"
|
||||||
|
input_hash = markdown_sha256(original)
|
||||||
|
first = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="ab",
|
||||||
|
after_text="x",
|
||||||
|
start=0,
|
||||||
|
end=2,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256="0" * 64,
|
||||||
|
)
|
||||||
|
second = change(
|
||||||
|
component_position=0,
|
||||||
|
before_text="bc",
|
||||||
|
after_text="y",
|
||||||
|
start=1,
|
||||||
|
end=3,
|
||||||
|
before_sha256=input_hash,
|
||||||
|
after_sha256="0" * 64,
|
||||||
|
proposal_index=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(ReplayError, match="conflicting"):
|
||||||
|
replay_change_chain(
|
||||||
|
input_markdown=original,
|
||||||
|
input_sha256=input_hash,
|
||||||
|
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||||
|
changes=(first, second),
|
||||||
|
current_sha256=input_hash,
|
||||||
|
current_markdown=None,
|
||||||
|
include_zero_change_stages=False,
|
||||||
|
)
|
||||||
@@ -10,6 +10,8 @@ import pytest
|
|||||||
import mdpolish.artifact_store as artifact_store
|
import mdpolish.artifact_store as artifact_store
|
||||||
from mdpolish.artifact_store import ArtifactStoreError, StoredDocument, publish_run
|
from mdpolish.artifact_store import ArtifactStoreError, StoredDocument, publish_run
|
||||||
|
|
||||||
|
_SOURCE_BYTES = b"source\n"
|
||||||
|
|
||||||
|
|
||||||
def stored_document(document_id: str = "paper", output: bytes | None = b"cleaned\n") -> StoredDocument:
|
def stored_document(document_id: str = "paper", output: bytes | None = b"cleaned\n") -> StoredDocument:
|
||||||
status = "success" if output is not None else "failed"
|
status = "success" if output is not None else "failed"
|
||||||
@@ -18,7 +20,7 @@ def stored_document(document_id: str = "paper", output: bytes | None = b"cleaned
|
|||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
"document": {"document_id": document_id, "source_label": f"inputs/{document_id}.md"},
|
"document": {"document_id": document_id, "source_label": f"inputs/{document_id}.md"},
|
||||||
"status": status,
|
"status": status,
|
||||||
"input_sha256": "1" * 64,
|
"input_sha256": sha256(_SOURCE_BYTES).hexdigest(),
|
||||||
"current_sha256": current_sha256,
|
"current_sha256": current_sha256,
|
||||||
"changes": [],
|
"changes": [],
|
||||||
"errors": [] if status == "success" else [{"error_type": "SyntheticError"}],
|
"errors": [] if status == "success" else [{"error_type": "SyntheticError"}],
|
||||||
@@ -82,15 +84,49 @@ def manifest_json(
|
|||||||
return (json.dumps(payload, indent=2) + "\n").encode()
|
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||||
|
|
||||||
|
|
||||||
|
def review_locator_json(
|
||||||
|
artifacts_root: Path,
|
||||||
|
run_date: str,
|
||||||
|
run_id: str,
|
||||||
|
documents: tuple[StoredDocument, ...],
|
||||||
|
) -> bytes:
|
||||||
|
source_root = artifacts_root.parent / f"{artifacts_root.name}-sources"
|
||||||
|
source_root.mkdir(exist_ok=True)
|
||||||
|
locator_documents: list[dict[str, object]] = []
|
||||||
|
for position, document in enumerate(documents):
|
||||||
|
source_path = source_root / f"{position}-{document.document_id}.md"
|
||||||
|
source_path.write_bytes(_SOURCE_BYTES)
|
||||||
|
report = json.loads(document.result_json)
|
||||||
|
locator_documents.append(
|
||||||
|
{
|
||||||
|
"document_id": document.document_id,
|
||||||
|
"source_path": str(source_path.resolve()),
|
||||||
|
"input_sha256": report["input_sha256"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
payload = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": {
|
||||||
|
"run_id": run_id,
|
||||||
|
"run_directory": str((artifacts_root / run_date / "runs" / run_id).resolve()),
|
||||||
|
"manifest_path": "manifest.json",
|
||||||
|
},
|
||||||
|
"documents": locator_documents,
|
||||||
|
}
|
||||||
|
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||||
|
|
||||||
|
|
||||||
def test_publish_run_creates_private_date_layout_and_status_specific_files(tmp_path: Path) -> None:
|
def test_publish_run_creates_private_date_layout_and_status_specific_files(tmp_path: Path) -> None:
|
||||||
artifacts_root = tmp_path / "artifacts"
|
artifacts_root = tmp_path / "artifacts"
|
||||||
documents = (stored_document("success"), stored_document("failed", None))
|
documents = (stored_document("success"), stored_document("failed", None))
|
||||||
|
locator_json = review_locator_json(artifacts_root, "2026-08-22", "example-run", documents)
|
||||||
|
|
||||||
run_directory = publish_run(
|
run_directory = publish_run(
|
||||||
artifacts_root=artifacts_root,
|
artifacts_root=artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="example-run",
|
run_id="example-run",
|
||||||
manifest_json=manifest_json("2026-08-22", "example-run", documents),
|
manifest_json=manifest_json("2026-08-22", "example-run", documents),
|
||||||
|
review_locator_json=locator_json,
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -100,6 +136,7 @@ def test_publish_run_creates_private_date_layout_and_status_specific_files(tmp_p
|
|||||||
assert (run_directory / "documents/success/cleaned.md").read_bytes() == b"cleaned\n"
|
assert (run_directory / "documents/success/cleaned.md").read_bytes() == b"cleaned\n"
|
||||||
assert (run_directory / "documents/success/changes.diff").read_bytes() == b""
|
assert (run_directory / "documents/success/changes.diff").read_bytes() == b""
|
||||||
assert (run_directory / "documents/failed/result.json").is_file()
|
assert (run_directory / "documents/failed/result.json").is_file()
|
||||||
|
assert (run_directory / "review-locator.json").read_bytes() == locator_json
|
||||||
assert not (run_directory / "documents/failed/cleaned.md").exists()
|
assert not (run_directory / "documents/failed/cleaned.md").exists()
|
||||||
assert not (run_directory / "documents/failed/changes.diff").exists()
|
assert not (run_directory / "documents/failed/changes.diff").exists()
|
||||||
|
|
||||||
@@ -121,11 +158,13 @@ def test_publish_run_rejects_existing_target_without_overwriting(tmp_path: Path)
|
|||||||
artifacts_root = tmp_path / "artifacts"
|
artifacts_root = tmp_path / "artifacts"
|
||||||
documents = (stored_document(),)
|
documents = (stored_document(),)
|
||||||
first_manifest = manifest_json("2026-08-22", "same-run", documents)
|
first_manifest = manifest_json("2026-08-22", "same-run", documents)
|
||||||
|
locator_json = review_locator_json(artifacts_root, "2026-08-22", "same-run", documents)
|
||||||
run_directory = publish_run(
|
run_directory = publish_run(
|
||||||
artifacts_root=artifacts_root,
|
artifacts_root=artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="same-run",
|
run_id="same-run",
|
||||||
manifest_json=first_manifest,
|
manifest_json=first_manifest,
|
||||||
|
review_locator_json=locator_json,
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -135,6 +174,7 @@ def test_publish_run_rejects_existing_target_without_overwriting(tmp_path: Path)
|
|||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="same-run",
|
run_id="same-run",
|
||||||
manifest_json=first_manifest,
|
manifest_json=first_manifest,
|
||||||
|
review_locator_json=locator_json,
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -152,13 +192,15 @@ def test_publish_run_rejects_existing_target_without_overwriting(tmp_path: Path)
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_publish_run_rejects_unsafe_date_and_run_id(tmp_path: Path, run_date: str, run_id: str) -> None:
|
def test_publish_run_rejects_unsafe_date_and_run_id(tmp_path: Path, run_date: str, run_id: str) -> None:
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
documents = (stored_document(),)
|
documents = (stored_document(),)
|
||||||
with pytest.raises(ArtifactStoreError):
|
with pytest.raises(ArtifactStoreError):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts",
|
artifacts_root=artifacts_root,
|
||||||
run_date=run_date,
|
run_date=run_date,
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
manifest_json=manifest_json(run_date, run_id, documents),
|
manifest_json=manifest_json(run_date, run_id, documents),
|
||||||
|
review_locator_json=review_locator_json(artifacts_root, run_date, run_id, documents),
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -166,22 +208,30 @@ def test_publish_run_rejects_unsafe_date_and_run_id(tmp_path: Path, run_date: st
|
|||||||
def test_publish_run_rejects_inconsistent_or_duplicate_document_artifacts(tmp_path: Path) -> None:
|
def test_publish_run_rejects_inconsistent_or_duplicate_document_artifacts(tmp_path: Path) -> None:
|
||||||
valid = stored_document()
|
valid = stored_document()
|
||||||
bad_hash = StoredDocument("paper", valid.result_json, b"output", b"", "0" * 64)
|
bad_hash = StoredDocument("paper", valid.result_json, b"output", b"", "0" * 64)
|
||||||
|
first_artifacts_root = tmp_path / "artifacts-a"
|
||||||
with pytest.raises(ArtifactStoreError, match="output hash"):
|
with pytest.raises(ArtifactStoreError, match="output hash"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts-a",
|
artifacts_root=first_artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="bad-hash",
|
run_id="bad-hash",
|
||||||
manifest_json=manifest_json("2026-08-22", "bad-hash", (bad_hash,)),
|
manifest_json=manifest_json("2026-08-22", "bad-hash", (bad_hash,)),
|
||||||
|
review_locator_json=review_locator_json(
|
||||||
|
first_artifacts_root, "2026-08-22", "bad-hash", (bad_hash,)
|
||||||
|
),
|
||||||
documents=(bad_hash,),
|
documents=(bad_hash,),
|
||||||
)
|
)
|
||||||
|
|
||||||
duplicates = (stored_document(), stored_document())
|
duplicates = (stored_document(), stored_document())
|
||||||
|
second_artifacts_root = tmp_path / "artifacts-b"
|
||||||
with pytest.raises(ArtifactStoreError, match="unique"):
|
with pytest.raises(ArtifactStoreError, match="unique"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts-b",
|
artifacts_root=second_artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="duplicate",
|
run_id="duplicate",
|
||||||
manifest_json=manifest_json("2026-08-22", "duplicate", duplicates),
|
manifest_json=manifest_json("2026-08-22", "duplicate", duplicates),
|
||||||
|
review_locator_json=review_locator_json(
|
||||||
|
second_artifacts_root, "2026-08-22", "duplicate", duplicates
|
||||||
|
),
|
||||||
documents=duplicates,
|
documents=duplicates,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -189,32 +239,81 @@ def test_publish_run_rejects_inconsistent_or_duplicate_document_artifacts(tmp_pa
|
|||||||
def test_publish_run_rejects_manifest_path_or_document_mismatch(tmp_path: Path) -> None:
|
def test_publish_run_rejects_manifest_path_or_document_mismatch(tmp_path: Path) -> None:
|
||||||
documents = (stored_document(),)
|
documents = (stored_document(),)
|
||||||
wrong_date = manifest_json("2026-08-21", "review", documents)
|
wrong_date = manifest_json("2026-08-21", "review", documents)
|
||||||
|
first_artifacts_root = tmp_path / "artifacts-a"
|
||||||
with pytest.raises(ArtifactStoreError, match="identity"):
|
with pytest.raises(ArtifactStoreError, match="identity"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts-a",
|
artifacts_root=first_artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="review",
|
run_id="review",
|
||||||
manifest_json=wrong_date,
|
manifest_json=wrong_date,
|
||||||
|
review_locator_json=review_locator_json(
|
||||||
|
first_artifacts_root, "2026-08-22", "review", documents
|
||||||
|
),
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
payload = json.loads(manifest_json("2026-08-22", "review", documents))
|
payload = json.loads(manifest_json("2026-08-22", "review", documents))
|
||||||
payload["documents"][0]["change_count"] = 99
|
payload["documents"][0]["change_count"] = 99
|
||||||
mismatched_index = (json.dumps(payload, indent=2) + "\n").encode()
|
mismatched_index = (json.dumps(payload, indent=2) + "\n").encode()
|
||||||
|
second_artifacts_root = tmp_path / "artifacts-b"
|
||||||
with pytest.raises(ArtifactStoreError, match="index"):
|
with pytest.raises(ArtifactStoreError, match="index"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts-b",
|
artifacts_root=second_artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="review",
|
run_id="review",
|
||||||
manifest_json=mismatched_index,
|
manifest_json=mismatched_index,
|
||||||
|
review_locator_json=review_locator_json(
|
||||||
|
second_artifacts_root, "2026-08-22", "review", documents
|
||||||
|
),
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_publish_run_rejects_inconsistent_review_locator(tmp_path: Path) -> None:
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
|
documents = (stored_document(),)
|
||||||
|
locator = json.loads(review_locator_json(artifacts_root, "2026-08-22", "review", documents))
|
||||||
|
locator["documents"][0]["input_sha256"] = "0" * 64
|
||||||
|
inconsistent_locator = (json.dumps(locator, indent=2) + "\n").encode()
|
||||||
|
|
||||||
|
with pytest.raises(ArtifactStoreError, match="document identity"):
|
||||||
|
publish_run(
|
||||||
|
artifacts_root=artifacts_root,
|
||||||
|
run_date="2026-08-22",
|
||||||
|
run_id="review",
|
||||||
|
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||||
|
review_locator_json=inconsistent_locator,
|
||||||
|
documents=documents,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert not artifacts_root.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_publish_run_rejects_source_changed_after_locator_creation(tmp_path: Path) -> None:
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
|
documents = (stored_document(),)
|
||||||
|
locator_json = review_locator_json(artifacts_root, "2026-08-22", "review", documents)
|
||||||
|
locator = json.loads(locator_json)
|
||||||
|
Path(locator["documents"][0]["source_path"]).write_bytes(b"changed\n")
|
||||||
|
|
||||||
|
with pytest.raises(ArtifactStoreError, match="input hash"):
|
||||||
|
publish_run(
|
||||||
|
artifacts_root=artifacts_root,
|
||||||
|
run_date="2026-08-22",
|
||||||
|
run_id="review",
|
||||||
|
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||||
|
review_locator_json=locator_json,
|
||||||
|
documents=documents,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert not artifacts_root.exists()
|
||||||
|
|
||||||
|
|
||||||
def test_publish_race_does_not_replace_a_new_target(
|
def test_publish_race_does_not_replace_a_new_target(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
documents = (stored_document(),)
|
documents = (stored_document(),)
|
||||||
original_rename = artifact_store._rename_no_replace
|
original_rename = artifact_store._rename_no_replace
|
||||||
|
|
||||||
@@ -226,10 +325,11 @@ def test_publish_race_does_not_replace_a_new_target(
|
|||||||
monkeypatch.setattr(artifact_store, "_rename_no_replace", create_competing_target)
|
monkeypatch.setattr(artifact_store, "_rename_no_replace", create_competing_target)
|
||||||
with pytest.raises(ArtifactStoreError, match="already exists"):
|
with pytest.raises(ArtifactStoreError, match="already exists"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts",
|
artifacts_root=artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="raced",
|
run_id="raced",
|
||||||
manifest_json=manifest_json("2026-08-22", "raced", documents),
|
manifest_json=manifest_json("2026-08-22", "raced", documents),
|
||||||
|
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "raced", documents),
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -253,14 +353,44 @@ def test_write_failure_cleans_temporary_directory_and_does_not_publish(
|
|||||||
original_write(path, content)
|
original_write(path, content)
|
||||||
|
|
||||||
monkeypatch.setattr(artifact_store, "_write_private_file", fail_second_write)
|
monkeypatch.setattr(artifact_store, "_write_private_file", fail_second_write)
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
documents = (stored_document(),)
|
documents = (stored_document(),)
|
||||||
|
|
||||||
with pytest.raises(OSError, match="synthetic"):
|
with pytest.raises(OSError, match="synthetic"):
|
||||||
publish_run(
|
publish_run(
|
||||||
artifacts_root=tmp_path / "artifacts",
|
artifacts_root=artifacts_root,
|
||||||
run_date="2026-08-22",
|
run_date="2026-08-22",
|
||||||
run_id="broken",
|
run_id="broken",
|
||||||
manifest_json=manifest_json("2026-08-22", "broken", documents),
|
manifest_json=manifest_json("2026-08-22", "broken", documents),
|
||||||
|
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "broken", documents),
|
||||||
|
documents=documents,
|
||||||
|
)
|
||||||
|
|
||||||
|
runs_directory = tmp_path / "artifacts/2026-08-22/runs"
|
||||||
|
assert list(runs_directory.iterdir()) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_locator_write_failure_does_not_publish(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
original_write = artifact_store._write_private_file
|
||||||
|
|
||||||
|
def fail_locator_write(path: Path, content: bytes) -> None:
|
||||||
|
if path.name == "review-locator.json":
|
||||||
|
raise OSError("synthetic locator write failure")
|
||||||
|
original_write(path, content)
|
||||||
|
|
||||||
|
monkeypatch.setattr(artifact_store, "_write_private_file", fail_locator_write)
|
||||||
|
artifacts_root = tmp_path / "artifacts"
|
||||||
|
documents = (stored_document(),)
|
||||||
|
|
||||||
|
with pytest.raises(OSError, match="locator"):
|
||||||
|
publish_run(
|
||||||
|
artifacts_root=artifacts_root,
|
||||||
|
run_date="2026-08-22",
|
||||||
|
run_id="locator-failure",
|
||||||
|
manifest_json=manifest_json("2026-08-22", "locator-failure", documents),
|
||||||
|
review_locator_json=review_locator_json(
|
||||||
|
artifacts_root, "2026-08-22", "locator-failure", documents
|
||||||
|
),
|
||||||
documents=documents,
|
documents=documents,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,162 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import (
|
||||||
|
ArxivSubmissionStampComponent,
|
||||||
|
HtmlTableDoubleEscapeComponent,
|
||||||
|
HtmlTableLayoutComponent,
|
||||||
|
ManuscriptLineNumberComponent,
|
||||||
|
PageBreakWordJoinComponent,
|
||||||
|
ReferenceSpacingComponent,
|
||||||
|
RepeatedRunningHeaderComponent,
|
||||||
|
WordReviewCommentComponent,
|
||||||
|
)
|
||||||
|
|
||||||
|
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||||
|
HEADER = "## Repeated Paper Header"
|
||||||
|
MAPPINGS = (
|
||||||
|
("medi-", "cal", "medical"),
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("cre-", "ated", "created"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("life-", "threatening", "life-threatening"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_pipeline() -> Pipeline:
|
||||||
|
return Pipeline(
|
||||||
|
[
|
||||||
|
WordReviewCommentComponent(),
|
||||||
|
ManuscriptLineNumberComponent(),
|
||||||
|
ArxivSubmissionStampComponent(),
|
||||||
|
RepeatedRunningHeaderComponent(),
|
||||||
|
PageBreakWordJoinComponent(MAPPINGS),
|
||||||
|
HtmlTableDoubleEscapeComponent(),
|
||||||
|
HtmlTableLayoutComponent(),
|
||||||
|
ReferenceSpacingComponent(),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _numbered_manuscript() -> list[str]:
|
||||||
|
return [
|
||||||
|
f"## {number} Section {number}" if number in {5, 15} else f"{number} body {number}"
|
||||||
|
for number in range(1, 21)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _combined_markdown() -> str:
|
||||||
|
return "\n".join(
|
||||||
|
(
|
||||||
|
"1 Affiliation",
|
||||||
|
"## Abstract",
|
||||||
|
*_numbered_manuscript(),
|
||||||
|
"Commented [A1]: remove this",
|
||||||
|
"",
|
||||||
|
STAMP,
|
||||||
|
"Sentence continues in",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"the next line.",
|
||||||
|
"A word is possi-",
|
||||||
|
"",
|
||||||
|
"bly split.",
|
||||||
|
"<table><tr><td>&lt;5</td></tr><tr><td>B</td></tr></table>",
|
||||||
|
"## References",
|
||||||
|
"",
|
||||||
|
"1. First",
|
||||||
|
"",
|
||||||
|
"2. Second",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"3. Third",
|
||||||
|
"4. Fourth",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_script_builds_frozen_component_order_and_parameters() -> None:
|
||||||
|
script_path = Path(__file__).parents[1] / "scripts" / "run_clindb_first_batch_experiment.py"
|
||||||
|
module = ast.parse(script_path.read_text(encoding="utf-8"))
|
||||||
|
build_function = next(
|
||||||
|
node for node in module.body if isinstance(node, ast.FunctionDef) and node.name == "build_pipeline"
|
||||||
|
)
|
||||||
|
component_names = [
|
||||||
|
call.func.id
|
||||||
|
for node in ast.walk(build_function)
|
||||||
|
if isinstance(node, ast.List)
|
||||||
|
for call in node.elts
|
||||||
|
if isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
|
||||||
|
]
|
||||||
|
mapping_assignment = next(
|
||||||
|
node
|
||||||
|
for node in module.body
|
||||||
|
if isinstance(node, ast.Assign)
|
||||||
|
and any(isinstance(target, ast.Name) and target.id == "CLINDB_WORD_JOIN_MAPPINGS" for target in node.targets)
|
||||||
|
)
|
||||||
|
assert component_names == [
|
||||||
|
"WordReviewCommentComponent",
|
||||||
|
"ManuscriptLineNumberComponent",
|
||||||
|
"ArxivSubmissionStampComponent",
|
||||||
|
"RepeatedRunningHeaderComponent",
|
||||||
|
"PageBreakWordJoinComponent",
|
||||||
|
"HtmlTableDoubleEscapeComponent",
|
||||||
|
"HtmlTableLayoutComponent",
|
||||||
|
"ReferenceSpacingComponent",
|
||||||
|
]
|
||||||
|
assert ast.literal_eval(mapping_assignment.value) == MAPPINGS
|
||||||
|
|
||||||
|
pipeline = _build_pipeline()
|
||||||
|
result = pipeline.transform("")
|
||||||
|
|
||||||
|
assert [component.component_id for component in result.components] == [
|
||||||
|
"paper.word_review_comment",
|
||||||
|
"paper.manuscript_line_number",
|
||||||
|
"paper.arxiv_submission_stamp",
|
||||||
|
"paper.repeated_running_header",
|
||||||
|
"paper.page_break_word_join",
|
||||||
|
"markdown.html_table_double_escape",
|
||||||
|
"markdown.html_table_layout",
|
||||||
|
"paper.reference_spacing",
|
||||||
|
]
|
||||||
|
assert len(MAPPINGS) == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_pipeline_is_audited_stable_and_idempotent() -> None:
|
||||||
|
pipeline = _build_pipeline()
|
||||||
|
first = pipeline.transform(_combined_markdown())
|
||||||
|
|
||||||
|
assert first.status is RunStatus.SUCCESS
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert first.residual_proposals == ()
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
for change in first.changes:
|
||||||
|
counts[change.component_id] = counts.get(change.component_id, 0) + 1
|
||||||
|
assert counts == {
|
||||||
|
"paper.word_review_comment": 1,
|
||||||
|
"paper.manuscript_line_number": 20,
|
||||||
|
"paper.arxiv_submission_stamp": 1,
|
||||||
|
"paper.repeated_running_header": 2,
|
||||||
|
"paper.page_break_word_join": 1,
|
||||||
|
"markdown.html_table_double_escape": 1,
|
||||||
|
"markdown.html_table_layout": 1,
|
||||||
|
"paper.reference_spacing": 1,
|
||||||
|
}
|
||||||
|
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.output_markdown == first.output_markdown
|
||||||
|
assert second.changes == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_business_components_are_not_exported_from_core_namespace() -> None:
|
||||||
|
import mdpolish
|
||||||
|
|
||||||
|
assert not hasattr(mdpolish, "WordReviewCommentComponent")
|
||||||
|
assert not hasattr(mdpolish, "HtmlTableLayoutComponent")
|
||||||
@@ -109,6 +109,27 @@ def test_run_experiment_publishes_manifest_reports_diff_and_preserves_inputs(tmp
|
|||||||
assert (result.run_directory / "documents/second/changes.diff").read_bytes() == b""
|
assert (result.run_directory / "documents/second/changes.diff").read_bytes() == b""
|
||||||
|
|
||||||
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
||||||
|
locator = json.loads((result.run_directory / "review-locator.json").read_bytes())
|
||||||
|
assert locator == {
|
||||||
|
"schema_version": 1,
|
||||||
|
"run": {
|
||||||
|
"run_id": "local-review",
|
||||||
|
"run_directory": str(result.run_directory.resolve()),
|
||||||
|
"manifest_path": "manifest.json",
|
||||||
|
},
|
||||||
|
"documents": [
|
||||||
|
{
|
||||||
|
"document_id": "first",
|
||||||
|
"source_path": str(first_path.resolve()),
|
||||||
|
"input_sha256": manifest["documents"][0]["input_sha256"],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"document_id": "second",
|
||||||
|
"source_path": str(second_path.resolve()),
|
||||||
|
"input_sha256": manifest["documents"][1]["input_sha256"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
assert manifest["run"]["run_date"] == "2026-08-22"
|
assert manifest["run"]["run_date"] == "2026-08-22"
|
||||||
assert manifest["run"]["utc_offset"] == "+08:00"
|
assert manifest["run"]["utc_offset"] == "+08:00"
|
||||||
assert manifest["run"]["started_at_utc"] == "2026-08-22T02:30:00Z"
|
assert manifest["run"]["started_at_utc"] == "2026-08-22T02:30:00Z"
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import HtmlTableDoubleEscapeComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([HtmlTableDoubleEscapeComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unescapes_one_layer_only_in_strict_cell_text() -> None:
|
||||||
|
markdown = (
|
||||||
|
'<table data-note="&lt;"><tr><td>&lt;5</td><td>&gt;2 &amp; x <</td></tr></table>'
|
||||||
|
"\noutside &lt;"
|
||||||
|
)
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == (
|
||||||
|
'<table data-note="&lt;"><tr><td><5</td><td>>2 & x <</td></tr></table>'
|
||||||
|
"\noutside &lt;"
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_multiple_tables_and_cells_report_source_order() -> None:
|
||||||
|
markdown = "<table><tr><td>&gt;</td></tr></table> x <table><tr><th>&lt;</th></tr></table>"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown == "<table><tr><td>></td></tr></table> x <table><tr><th><</th></tr></table>"
|
||||||
|
assert [change.span.start for change in result.changes] == sorted(change.span.start for change in result.changes)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"outside &lt;",
|
||||||
|
"<table><tr><td>&lt;</tr></table>",
|
||||||
|
"<table><tr><td><em>&lt;</em></td></tr></table>",
|
||||||
|
"<table><tr><td><table><tr><td>&lt;</td></tr></table></td></tr></table>",
|
||||||
|
"<table><tbody><tr><td>&lt;</td></tr></tbody></table>",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_non_strict_or_outside_content_is_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_fenced_table_is_not_protected() -> None:
|
||||||
|
markdown = "```html\n<table><tr><td>&lt;</td></tr></table>\n```"
|
||||||
|
assert transform(markdown).output_markdown == "```html\n<table><tr><td><</td></tr></table>\n```"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([HtmlTableDoubleEscapeComponent()])
|
||||||
|
first = pipeline.transform("<table><tr><td>&lt;</td></tr></table>")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import HtmlTableLayoutComponent
|
||||||
|
|
||||||
|
TABLE = '<table class="x"><tr><td colspan="2">A</td></tr><tr><td>B</td><td>C</td></tr></table>'
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([HtmlTableLayoutComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_expands_rows_without_changing_tags_attributes_or_cells() -> None:
|
||||||
|
result = transform(f"before\n{TABLE}\nafter")
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == (
|
||||||
|
"before\n<table class=\"x\">\n"
|
||||||
|
" <tr><td colspan=\"2\">A</td></tr>\n"
|
||||||
|
" <tr><td>B</td><td>C</td></tr>\n"
|
||||||
|
"</table>\nafter"
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_uses_the_documents_single_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = f"before{line_ending}{TABLE}{line_ending}after"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert f"<table class=\"x\">{line_ending} <tr>" in result.output_markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_table_only_document_uses_lf() -> None:
|
||||||
|
assert transform(TABLE).output_markdown == (
|
||||||
|
'<table class="x">\n'
|
||||||
|
' <tr><td colspan="2">A</td></tr>\n'
|
||||||
|
" <tr><td>B</td><td>C</td></tr>\n"
|
||||||
|
"</table>"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"before\n<table><tr><td>A</td></tr>\r\nafter",
|
||||||
|
"<table>\n <tr><td>A</td></tr>\n</table>",
|
||||||
|
"<table><tr><td>A</tr></table>",
|
||||||
|
"<table><tbody><tr><td>A</td></tr></tbody></table>",
|
||||||
|
"<table><tr><td><em>A</em></td></tr></table>",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_mixed_multiline_or_non_strict_tables_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([HtmlTableLayoutComponent()])
|
||||||
|
first = pipeline.transform(TABLE)
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import ManuscriptLineNumberComponent
|
||||||
|
|
||||||
|
|
||||||
|
def _document(
|
||||||
|
*,
|
||||||
|
count: int = 20,
|
||||||
|
heading_numbers: frozenset[int] = frozenset({5, 15}),
|
||||||
|
numbers: tuple[int, ...] | None = None,
|
||||||
|
line_ending: str = "\n",
|
||||||
|
) -> str:
|
||||||
|
values = numbers if numbers is not None else tuple(range(1, count + 1))
|
||||||
|
body = [
|
||||||
|
f"## {number} Section {number}" if number in heading_numbers else f"{number} body {number}"
|
||||||
|
for number in values
|
||||||
|
]
|
||||||
|
return line_ending.join(("1 Affiliation", "2 Institute", "## Abstract", *body))
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([ManuscriptLineNumberComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_removes_long_monotonic_sequence_but_preserves_pre_abstract_affiliations() -> None:
|
||||||
|
result = transform(_document())
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.startswith("1 Affiliation\n2 Institute\n## Abstract\nbody 1")
|
||||||
|
assert "## Section 5" in result.output_markdown
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_all_supported_line_endings(line_ending: str) -> None:
|
||||||
|
result = transform(_document(line_ending=line_ending))
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.count(line_ending) == _document(line_ending=line_ending).count(line_ending)
|
||||||
|
|
||||||
|
|
||||||
|
def test_allows_skipped_numbers_when_sequence_is_strictly_increasing() -> None:
|
||||||
|
numbers = tuple(range(10, 30))
|
||||||
|
result = transform(_document(numbers=numbers, heading_numbers=frozenset({14, 24})))
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
_document(count=19, heading_numbers=frozenset({5, 15})),
|
||||||
|
_document(heading_numbers=frozenset({5})),
|
||||||
|
_document(numbers=(*tuple(range(1, 20)), 10), heading_numbers=frozenset({5, 15})),
|
||||||
|
_document().replace("## Abstract", "## ABSTRACT"),
|
||||||
|
_document() + "\n## Abstract",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_incomplete_or_ambiguous_evidence_preserves_the_document(markdown: str) -> None:
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == markdown
|
||||||
|
assert result.changes == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_lists_years_and_numbers_inside_body_are_not_candidates() -> None:
|
||||||
|
markdown = _document() + "\n1. list\n1) list\n2024 report\nThe panel included 35 experts"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown is not None
|
||||||
|
assert result.output_markdown.endswith("1. list\n1) list\n2024 report\nThe panel included 35 experts")
|
||||||
|
assert len(result.changes) == 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([ManuscriptLineNumberComponent()])
|
||||||
|
first = pipeline.transform(_document())
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.changes == ()
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.component import ComponentContractError
|
||||||
|
from mdpolish.components import PageBreakWordJoinComponent
|
||||||
|
|
||||||
|
MAPPINGS = (
|
||||||
|
("possi-", "bly", "possibly"),
|
||||||
|
("SOFA-", "based", "SOFA-based"),
|
||||||
|
("threshold.", "olds", "thresholds"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([PageBreakWordJoinComponent(MAPPINGS)]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("markdown", "expected"),
|
||||||
|
[
|
||||||
|
("except possi-\nbly through care", "except possibly through care"),
|
||||||
|
("except possi-\n\nbly through care", "except possibly through care"),
|
||||||
|
("use SOFA-\nbased criteria", "use SOFA-based criteria"),
|
||||||
|
("at threshold.\n\nolds of eight", "at thresholds of eight"),
|
||||||
|
("except possi-\r\n\r\nbly now", "except possibly now"),
|
||||||
|
("except possi-\r\rbly now", "except possibly now"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_applies_exact_mapping_across_approved_line_shapes(markdown: str, expected: str) -> None:
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == expected
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"except unknown-\nword here",
|
||||||
|
"except possi-\n\n\nbly here",
|
||||||
|
"except possi-\r\n\nbly here",
|
||||||
|
"except POSSI-\nbly here",
|
||||||
|
"except possi-\nblymore here",
|
||||||
|
"except xSOFA-\nbased here",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_unknown_or_unsafe_boundaries_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_parameters_and_results_are_independent_of_mapping_order() -> None:
|
||||||
|
first = PageBreakWordJoinComponent(MAPPINGS)
|
||||||
|
second = PageBreakWordJoinComponent(reversed(MAPPINGS))
|
||||||
|
markdown = "possibly becomes possi-\nbly"
|
||||||
|
|
||||||
|
first_result = Pipeline([first]).transform(markdown)
|
||||||
|
second_result = Pipeline([second]).transform(markdown)
|
||||||
|
|
||||||
|
assert first_result.components == second_result.components
|
||||||
|
assert first_result.output_markdown == second_result.output_markdown
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"mappings",
|
||||||
|
[
|
||||||
|
(("", "right", "word"),),
|
||||||
|
(("left", "right", "two words"),),
|
||||||
|
(("left", "right", "word"), ("left", "right", "other")),
|
||||||
|
(("left", "right"),),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_invalid_mappings_raise_contract_error(mappings: object) -> None:
|
||||||
|
with pytest.raises(ComponentContractError):
|
||||||
|
PageBreakWordJoinComponent(mappings) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([PageBreakWordJoinComponent(MAPPINGS)])
|
||||||
|
first = pipeline.transform("except possi-\n\nbly here")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import ReferenceSpacingComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([ReferenceSpacingComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalizes_missing_and_extra_blank_lines_in_references_only() -> None:
|
||||||
|
markdown = "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n2. Second\n\n\n3. Third"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n\n2. Second\n\n3. Third"
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("heading", ["# References", "## REFERENCES", "### references", "#### ReFeReNcEs"])
|
||||||
|
def test_accepts_exact_references_heading_with_ascii_case_folding(heading: str) -> None:
|
||||||
|
markdown = f"{heading}\n\n1. First\n2. Second"
|
||||||
|
assert transform(markdown).output_markdown == f"{heading}\n\n1. First\n\n2. Second"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(("## References", "", "1. First", "2. Second"))
|
||||||
|
expected = line_ending.join(("## References", "", "1. First", "", "2. Second"))
|
||||||
|
assert transform(markdown).output_markdown == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
"## Reference\n\n1. First\n2. Second",
|
||||||
|
"## References\n\n1. First\n3. Third",
|
||||||
|
"## References\n\n2. Second\n3. Third",
|
||||||
|
"## References\n\n1. First\n### Subsection\n2. Second",
|
||||||
|
"## References\r\n\r\n1. First\r\n\n2. Second",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_ambiguous_or_mixed_sections_are_preserved(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_same_or_higher_heading_ends_section() -> None:
|
||||||
|
markdown = "## References\n\n1. First\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "## References\n\n1. First\n\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||||
|
|
||||||
|
|
||||||
|
def test_multiline_reference_uses_its_last_text_line_as_boundary() -> None:
|
||||||
|
markdown = "## References\n\n1. First line\ncontinuation\n2. Second"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "## References\n\n1. First line\ncontinuation\n\n2. Second"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([ReferenceSpacingComponent()])
|
||||||
|
first = pipeline.transform("## References\n\n1. First\n2. Second")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import RepeatedRunningHeaderComponent
|
||||||
|
|
||||||
|
HEADER = "## Repeated Paper Header"
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([RepeatedRunningHeaderComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_bridges_interrupted_sentence_and_deletes_other_occurrence(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(
|
||||||
|
(
|
||||||
|
"Sentence continues in",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"the next line.",
|
||||||
|
"",
|
||||||
|
"18. Reference",
|
||||||
|
"",
|
||||||
|
HEADER,
|
||||||
|
"",
|
||||||
|
"19. Reference",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == line_ending.join(
|
||||||
|
("Sentence continues in the next line.", "", "18. Reference", "", "19. Reference")
|
||||||
|
)
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"markdown",
|
||||||
|
[
|
||||||
|
f"before\n\n{HEADER}\n\nafter",
|
||||||
|
f"Sentence ends.\n\n{HEADER}\n\nAfter\n\nAnother sentence.\n\n{HEADER}\n\nOther",
|
||||||
|
f"Sentence continues\n\n{HEADER}\n\nAfter\n\nText ends.\n\n{HEADER}\n\nOther",
|
||||||
|
f"Sentence continues\n\n{HEADER}\n\nafter\n{HEADER}\nnot blank",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_missing_or_unsafe_group_evidence_preserves_document(markdown: str) -> None:
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_header_matching_is_exact_and_not_keyword_based() -> None:
|
||||||
|
markdown = "continues\n\n## Any Header\n\nfrom here\n\n18. Ref\n\n## Any Header\n\n19. Ref"
|
||||||
|
result = transform(markdown)
|
||||||
|
assert result.output_markdown == "continues from here\n\n18. Ref\n\n19. Ref"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([RepeatedRunningHeaderComponent()])
|
||||||
|
first = pipeline.transform(f"continues\n\n{HEADER}\n\nfrom here\n\ntext\n\n{HEADER}\n\nend")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
assert pipeline.transform(first.output_markdown).changes == ()
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, cast
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from reviewer.server import ReviewArtifactError, ReviewArtifacts
|
||||||
|
from tests.reviewer_fixture import create_review_run, read_json, write_json
|
||||||
|
|
||||||
|
|
||||||
|
def test_reviewer_returns_full_comparison_and_all_component_stages(tmp_path: Path) -> None:
|
||||||
|
fixture = create_review_run(tmp_path)
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
|
||||||
|
run = cast(dict[str, Any], repository.run_summary())
|
||||||
|
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||||
|
changed = cast(dict[str, Any], repository.component_stage("paper", 0))
|
||||||
|
unchanged = cast(dict[str, Any], repository.component_stage("paper", 1))
|
||||||
|
|
||||||
|
assert run["summary"] == {
|
||||||
|
"document_count": 1,
|
||||||
|
"success_count": 1,
|
||||||
|
"failed_count": 0,
|
||||||
|
"unstable_count": 0,
|
||||||
|
"change_count": 1,
|
||||||
|
}
|
||||||
|
assert run["documents"][0]["source_available"] is True
|
||||||
|
assert comparison["original_markdown"] == fixture["original"]
|
||||||
|
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||||
|
assert comparison["changes"][0]["editor_range"] == {"start": 4, "end": 7}
|
||||||
|
assert changed["before_markdown"] == fixture["original"]
|
||||||
|
assert changed["after_markdown"] == fixture["cleaned"]
|
||||||
|
assert unchanged["before_markdown"] == fixture["cleaned"]
|
||||||
|
assert unchanged["after_markdown"] == fixture["cleaned"]
|
||||||
|
assert unchanged["changes"] == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("status", ["failed", "unstable"])
|
||||||
|
def test_non_success_documents_expose_audit_but_no_formal_output(tmp_path: Path, status: str) -> None:
|
||||||
|
fixture = create_review_run(tmp_path, status=status)
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
|
||||||
|
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||||
|
|
||||||
|
assert comparison["original_markdown"] == fixture["original"]
|
||||||
|
assert comparison["cleaned_markdown"] is None
|
||||||
|
if status == "failed":
|
||||||
|
assert comparison["errors"]
|
||||||
|
assert comparison["residual_proposals"] == []
|
||||||
|
else:
|
||||||
|
assert comparison["errors"] == []
|
||||||
|
assert comparison["residual_proposals"]
|
||||||
|
with pytest.raises(ReviewArtifactError, match="success"):
|
||||||
|
repository.component_stage("paper", 0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_historical_run_without_locator_reports_unavailable_source(tmp_path: Path) -> None:
|
||||||
|
fixture = create_review_run(tmp_path, with_locator=False)
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
|
||||||
|
summary = cast(dict[str, Any], repository.run_summary())
|
||||||
|
|
||||||
|
assert summary["documents"][0]["source_available"] is False
|
||||||
|
assert "review-locator.json" in summary["documents"][0]["availability_error"]
|
||||||
|
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||||
|
assert comparison["original_markdown"] is None
|
||||||
|
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||||
|
assert comparison["changes"][0]["editor_range"] is None
|
||||||
|
with pytest.raises(ReviewArtifactError, match=r"review-locator\.json"):
|
||||||
|
repository.component_stage("paper", 0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_hash_change_is_not_silently_displayed(tmp_path: Path) -> None:
|
||||||
|
fixture = create_review_run(tmp_path)
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
source_path = fixture["source_path"]
|
||||||
|
assert isinstance(source_path, Path)
|
||||||
|
source_path.write_text("changed\n", encoding="utf-8")
|
||||||
|
|
||||||
|
summary = cast(dict[str, Any], repository.run_summary())
|
||||||
|
|
||||||
|
assert summary["documents"][0]["source_available"] is False
|
||||||
|
assert "哈希" in summary["documents"][0]["availability_error"]
|
||||||
|
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||||
|
assert comparison["original_markdown"] is None
|
||||||
|
assert comparison["changes"][0]["editor_range"] is None
|
||||||
|
with pytest.raises(ReviewArtifactError, match="哈希"):
|
||||||
|
repository.component_stage("paper", 0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reviewer_rejects_unknown_schema_and_artifact_path_escape(tmp_path: Path) -> None:
|
||||||
|
fixture = create_review_run(tmp_path)
|
||||||
|
manifest_path = fixture["manifest_path"]
|
||||||
|
assert isinstance(manifest_path, Path)
|
||||||
|
manifest = read_json(manifest_path)
|
||||||
|
manifest["schema_version"] = 2
|
||||||
|
write_json(manifest_path, manifest)
|
||||||
|
|
||||||
|
with pytest.raises(ReviewArtifactError) as unknown:
|
||||||
|
ReviewArtifacts(fixture["run_directory"])
|
||||||
|
assert unknown.value.code == "unsupported_schema"
|
||||||
|
|
||||||
|
second = create_review_run(tmp_path / "second")
|
||||||
|
second_manifest_path = second["manifest_path"]
|
||||||
|
assert isinstance(second_manifest_path, Path)
|
||||||
|
second_manifest = read_json(second_manifest_path)
|
||||||
|
second_manifest["documents"][0]["result_path"] = "../outside.json"
|
||||||
|
write_json(second_manifest_path, second_manifest)
|
||||||
|
with pytest.raises(ReviewArtifactError, match="路径"):
|
||||||
|
ReviewArtifacts(second["run_directory"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_reviewer_rejects_tampered_snapshot_chain_and_unknown_identity(tmp_path: Path) -> None:
|
||||||
|
fixture = create_review_run(tmp_path)
|
||||||
|
result_path = fixture["result_path"]
|
||||||
|
assert isinstance(result_path, Path)
|
||||||
|
result = read_json(result_path)
|
||||||
|
result["changes"][0]["after_sha256"] = "0" * 64
|
||||||
|
write_json(result_path, result)
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
|
||||||
|
with pytest.raises(ReviewArtifactError) as replay_error:
|
||||||
|
repository.document_comparison("paper")
|
||||||
|
assert replay_error.value.code == "untrusted_replay"
|
||||||
|
|
||||||
|
with pytest.raises(ReviewArtifactError) as document_error:
|
||||||
|
repository.document_comparison("missing")
|
||||||
|
assert document_error.value.http_status == 404
|
||||||
|
with pytest.raises(ReviewArtifactError) as component_error:
|
||||||
|
repository.component_stage("paper", 99)
|
||||||
|
assert component_error.value.http_status == 404
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import threading
|
||||||
|
from collections.abc import Generator
|
||||||
|
from http.client import HTTPConnection, HTTPResponse
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, cast
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from reviewer.server import ReviewArtifacts
|
||||||
|
from reviewer.server.__main__ import create_server
|
||||||
|
from tests.reviewer_fixture import ReviewFixture, create_review_run
|
||||||
|
|
||||||
|
|
||||||
|
def request(
|
||||||
|
port: int,
|
||||||
|
method: str,
|
||||||
|
path: str,
|
||||||
|
*,
|
||||||
|
headers: dict[str, str] | None = None,
|
||||||
|
) -> tuple[HTTPResponse, bytes]:
|
||||||
|
connection = HTTPConnection("127.0.0.1", port, timeout=3)
|
||||||
|
connection.request(method, path, headers=headers or {})
|
||||||
|
response = connection.getresponse()
|
||||||
|
content = response.read()
|
||||||
|
connection.close()
|
||||||
|
return response, content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def running_server(tmp_path: Path) -> Generator[tuple[int, ReviewFixture], None, None]:
|
||||||
|
fixture = create_review_run(tmp_path)
|
||||||
|
static_root = tmp_path / "static"
|
||||||
|
(static_root / "assets").mkdir(parents=True)
|
||||||
|
(static_root / "index.html").write_text("<main>reviewer</main>\n", encoding="utf-8")
|
||||||
|
(static_root / "assets/app.js").write_text("export {};\n", encoding="utf-8")
|
||||||
|
repository = ReviewArtifacts(fixture["run_directory"])
|
||||||
|
server = create_server(repository, static_root)
|
||||||
|
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||||
|
thread.start()
|
||||||
|
try:
|
||||||
|
yield server.server_address[1], fixture
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
server.server_close()
|
||||||
|
thread.join(timeout=3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_server_exposes_same_origin_api_and_static_build(
|
||||||
|
running_server: tuple[int, ReviewFixture],
|
||||||
|
) -> None:
|
||||||
|
port, fixture = running_server
|
||||||
|
|
||||||
|
api_response, api_content = request(port, "GET", "/api/v1/run")
|
||||||
|
page_response, page_content = request(port, "GET", "/")
|
||||||
|
asset_response, _ = request(port, "HEAD", "/assets/app.js")
|
||||||
|
|
||||||
|
payload = cast(dict[str, Any], json.loads(api_content))
|
||||||
|
assert api_response.status == 200
|
||||||
|
assert payload["run"]["run_id"] == "review-run"
|
||||||
|
assert str(fixture["source_path"]) not in api_content.decode()
|
||||||
|
assert api_response.getheader("Access-Control-Allow-Origin") is None
|
||||||
|
assert api_response.getheader("Cache-Control") == "no-store"
|
||||||
|
assert "default-src 'self'" in cast(str, api_response.getheader("Content-Security-Policy"))
|
||||||
|
assert page_response.status == 200
|
||||||
|
assert page_content == b"<main>reviewer</main>\n"
|
||||||
|
assert asset_response.status == 200
|
||||||
|
assert asset_response.getheader("Content-Type") == "text/javascript; charset=utf-8"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("method", "path", "headers", "status", "code"),
|
||||||
|
[
|
||||||
|
("POST", "/api/v1/run", None, 405, "method_not_allowed"),
|
||||||
|
("GET", "/api/v1/run", {"Host": "example.com"}, 403, "invalid_origin"),
|
||||||
|
(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/run",
|
||||||
|
{"Origin": "http://example.com"},
|
||||||
|
403,
|
||||||
|
"invalid_origin",
|
||||||
|
),
|
||||||
|
("GET", "/api/v1/documents/missing", None, 404, "unknown_document"),
|
||||||
|
("GET", "/api/v1/documents/paper/components/99", None, 404, "unknown_component"),
|
||||||
|
("GET", "/assets/missing.js", None, 404, "not_found"),
|
||||||
|
("GET", "/..%2Fsecret.txt", None, 404, "not_found"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_server_rejects_unsafe_or_unknown_requests(
|
||||||
|
running_server: tuple[int, ReviewFixture],
|
||||||
|
method: str,
|
||||||
|
path: str,
|
||||||
|
headers: dict[str, str] | None,
|
||||||
|
status: int,
|
||||||
|
code: str,
|
||||||
|
) -> None:
|
||||||
|
port, _ = running_server
|
||||||
|
|
||||||
|
response, content = request(port, method, path, headers=headers)
|
||||||
|
payload = cast(dict[str, Any], json.loads(content))
|
||||||
|
|
||||||
|
assert response.status == status
|
||||||
|
assert payload["error"]["code"] == code
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from mdpolish import Pipeline, RunStatus
|
||||||
|
from mdpolish.components import WordReviewCommentComponent
|
||||||
|
|
||||||
|
|
||||||
|
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||||
|
return Pipeline([WordReviewCommentComponent()]).transform(markdown)
|
||||||
|
|
||||||
|
|
||||||
|
def test_deletes_single_line_comment_and_one_following_blank() -> None:
|
||||||
|
markdown = "before\nCommented [AB1]: review this\n\nafter"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.status is RunStatus.SUCCESS
|
||||||
|
assert result.output_markdown == "before\nafter"
|
||||||
|
assert len(result.changes) == 1
|
||||||
|
assert result.changes[0].reason == "删除严格单行 Word 审阅批注及其后一个空行"
|
||||||
|
|
||||||
|
|
||||||
|
def test_adjacent_comment_blocks_are_deleted_without_overlap() -> None:
|
||||||
|
markdown = "before\nCommented [A1]: first\n\nCommented [B2R1]: second\n\nafter"
|
||||||
|
result = transform(markdown)
|
||||||
|
|
||||||
|
assert result.output_markdown == "before\nafter"
|
||||||
|
assert len(result.changes) == 2
|
||||||
|
assert result.changes[0].span.end == result.changes[1].span.start
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||||
|
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||||
|
markdown = line_ending.join(("before", "Commented [A1]: note", "", "after"))
|
||||||
|
assert transform(markdown).output_markdown == line_ending.join(("before", "after"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"comment",
|
||||||
|
[
|
||||||
|
"prefix Commented [A1]: note",
|
||||||
|
" Commented [A1]: note",
|
||||||
|
"Commented []: note",
|
||||||
|
"Commented [A-1]: note",
|
||||||
|
"Commented [A1]:",
|
||||||
|
"Commented [A1]: ",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_similar_lines_are_preserved(comment: str) -> None:
|
||||||
|
markdown = f"before\n{comment}\n\nafter"
|
||||||
|
assert transform(markdown).output_markdown == markdown
|
||||||
|
|
||||||
|
|
||||||
|
def test_requires_a_following_blank_line_and_does_not_delete_extra_blanks() -> None:
|
||||||
|
without_blank = "Commented [A1]: note\nafter"
|
||||||
|
with_two_blanks = "before\nCommented [A1]: note\n\n\nafter"
|
||||||
|
|
||||||
|
assert transform(without_blank).output_markdown == without_blank
|
||||||
|
assert transform(with_two_blanks).output_markdown == "before\n\nafter"
|
||||||
|
|
||||||
|
|
||||||
|
def test_successful_output_is_idempotent() -> None:
|
||||||
|
pipeline = Pipeline([WordReviewCommentComponent()])
|
||||||
|
first = pipeline.transform("Commented [A1]: note\n\nafter")
|
||||||
|
assert first.output_markdown is not None
|
||||||
|
second = pipeline.transform(first.output_markdown)
|
||||||
|
|
||||||
|
assert second.status is RunStatus.SUCCESS
|
||||||
|
assert second.changes == ()
|
||||||
Reference in New Issue
Block a user