重构为函数式通用 Markdown 修改库
This commit is contained in:
@@ -25,6 +25,9 @@ artifacts/
|
||||
outputs/
|
||||
experiments/
|
||||
research-wiki/scratch/
|
||||
!research-wiki/scratch/
|
||||
research-wiki/scratch/*
|
||||
!research-wiki/scratch/.gitkeep
|
||||
*.cleaned.md
|
||||
report.json
|
||||
|
||||
|
||||
@@ -1,203 +1,208 @@
|
||||
# mdpolish
|
||||
|
||||
实验室共用的 Markdown 清洗研究与基础工具库。项目不属于 GovDoc 专用组件，也不只服务政务文档。
|
||||
`mdpolish` 是实验室共用的、项目无关的 Python Markdown 修改库。它提供函数式 `Modifier`、精确文本编辑执行器、
|
||||
有序 `Pipeline`、正则修改器工厂，以及少量可以用合成样例完整说明的通用修改器。
|
||||
|
||||
本仓库面向实验室内不同项目复用，用于清洗 PDF、DOCX、OCR、网页等上游管线生成的 Markdown，统一解决
|
||||
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
||||
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
||||
当前版本是 `0.2.0`。库只处理内存中的 Markdown 字符串，不读取或写入文件，不提供默认流水线，也不包含任何项目的
|
||||
规则集合、数据清单、实验脚本或评审工具。
|
||||
|
||||
仓库当前已从纯文档治理进入第一版核心和 ClinDB 第一批组件实现阶段：已经提供可安装的 Python 内存处理包、
|
||||
8 个论文清洗组件、本地实验入口，以及同仓但与清洗运行解耦的只读评审前端。实验可以保存指定论文的成功输出、
|
||||
审计和 diff；评审前端可以同时查看清洗前后全文，并逐组件复放修改。仓库仍不提供面向任意数据集的完整规则集、
|
||||
公共命令行工具、通用文件适配器或生产接口，因此目前还不是拿来即可完成任意 Markdown 清洗的成品工具。
|
||||
## 当前能力
|
||||
|
||||
## 当前阶段
|
||||
| 能力 | 作用 | 明确边界 |
|
||||
| --- | --- | --- |
|
||||
| `Modifier` | 把不可变元数据与普通提议函数组合起来 | 函数只提议修改，不直接改字符串或文件 |
|
||||
| 精确编辑执行器 | 校验快照、范围、原文、重复和冲突后原子应用一个批次 | 不判断项目业务语义 |
|
||||
| `Pipeline` | 按调用方顺序运行修改器，并对最终快照做只读稳定性复查 | 不自动选规则、不重排、不循环执行 |
|
||||
| `regex_replace()` | 把非空正则匹配转换为精确编辑 | 不提供规则注册表、配置加载或默认模式 |
|
||||
| `mapped_line_join()` | 按调用方提供的映射合并跨行片段 | 库内没有默认词表，不猜测未知词 |
|
||||
| HTML 表格修改器 | 处理严格表格子集的实体和单行布局 | 不是完整 HTML parser，也不是 HTML→GFM 转换器 |
|
||||
|
||||
项目当前已经进入第一版可执行核心和真实组件验证阶段：
|
||||
一次运行会返回 `success`、`failed` 或 `unstable`：
|
||||
|
||||
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
||||
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
||||
- 已实现 ClinDB 第一批 8 个正式组件，覆盖 Word 批注、手稿行号、arXiv 戳、重复页眉、批准映射断词、
|
||||
HTML 表格实体与布局、参考文献空行；
|
||||
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计、unified diff 和评审定位文件，
|
||||
并保持输入不变；
|
||||
- 已有独立的 `reviewer/` 本地只读前端，服务只消费一次已发布的产物和定位文件，并与报告层共用纯 Python 快照重放逻辑，
|
||||
不导入组件、不调用流水线，也不重新清洗；
|
||||
- 第一批流水线已对 5 份论文 Markdown 完成保存型实验，5/5 成功，共记录 155 条修改，第二次运行零修改；
|
||||
- 当前基础检查为 Ruff、mypy、229 项 pytest 测试，以及前端 ESLint、TypeScript、9 项 Vitest 测试和生产构建，
|
||||
实际命令见本文“当前可用检查”。
|
||||
- `success`：所选修改器完成运行，且对最终结果不再提出修改；
|
||||
- `failed`：修改器、契约或编辑验证发生错误；
|
||||
- `unstable`：运行没有错误，但最终复查仍发现有效候选修改。
|
||||
|
||||
项目还没有面向任意数据集的完整清洗规则集、Markdown/HTML 通用 parser、profile 格式、通用文件输入接口、公共 CLI、
|
||||
通用批处理或生产接口。评审前端也不是公共 Web 服务：它只绑定本机回环地址，只读展示 Markdown 源文和审计，不提供
|
||||
渲染预览、在线编辑或重新清洗。当前 first-batch 实验脚本只固定运行已批准的 5 份论文和 8 个组件；它只能证明当前
|
||||
8 类确定规则已经闭环，不能据此认为论文中的缺失内容、乱码、复杂表格、图片或 GovDoc 已具备完整清洗能力。
|
||||
`success` 只代表本次选择的修改器已经稳定，不代表文档不存在其他质量问题。
|
||||
|
||||
## 服务对象与复用目标
|
||||
## 安装
|
||||
|
||||
当前已经明确的使用场景包括：
|
||||
项目仍在仓库内开发，使用项目可以从本地路径安装：
|
||||
|
||||
- **论文清洗**：`data/` 中现有内容来自师姐的项目，包含论文 PDF 的 Markdown、JSON、图片等转换产物；
|
||||
- **GovDoc**：政务、招投标、采购、合同等文档的清洗、比对和 RAG 前处理；
|
||||
- **未来实验室项目**：后续可以继续接入其他需要 Markdown 质量检查、规范化或用途派生的项目数据。
|
||||
|
||||
当前用例只用于发现真实问题和验证通用能力，不能反过来限定库的设计。核心代码不得依赖论文 DOI、
|
||||
GovDoc 目录、具体客户名称或某一转换器的固定输出路径。
|
||||
|
||||
## 测试数据
|
||||
|
||||
- **论文 Markdown**：`data/md/`，共 5 份，按 ClinDB-ReviewBench 中使用的论文缩写命名，
|
||||
供本地查看和组件只读验证；
|
||||
- **GovDoc Markdown**：`/home/lihaoze/gov_test_data/compare`，共 7 组、45 份，输入位于各组 `uploads/` 下，
|
||||
保持仓库外只读，不复制到本项目。
|
||||
|
||||
两组数据都不是可提交的自动测试 fixture。`data/` 已被 Git 忽略，清洗实验不得覆盖这些输入。
|
||||
|
||||
本地清洗实验产物位于 `artifacts/<YYYY-MM-DD>/runs/<run_id>/`。产物可能包含完整原文；其中评审定位文件还包含
|
||||
输入源的绝对路径，因此整个目录均受 Git 忽略，默认保留 30 个日历日，不得提交或复制到外部系统。当前只批准为上述
|
||||
5 份论文副本保存产物，未批准保存 GovDoc 输出。
|
||||
|
||||
## 面向复用的设计原则
|
||||
|
||||
- **通用核心**：只接收 Markdown；第一版只执行确定、可审计的精确修改，不读取 PDF、图片或转换器 JSON；
|
||||
- **输入边界**：PDF/OCR/DOCX/HTML 转换和外部材料核验由使用项目或上游流程负责，不写入共用组件契约；
|
||||
- **项目 profile**：论文、GovDoc、对比、RAG、公开脱敏等规则独立组合，不互相污染默认行为；
|
||||
- **保真优先**：不确定内容默认保留；当前核心不猜测修改，也不承担人工确认流程；
|
||||
- **可复现**：规则、配置、输入哈希、输出和每次变更都可以追踪；
|
||||
- **可扩展**：新增项目在自身边界处理上游适配，并主要组合或补充组件和 profile，而不是复制一套清洗器。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```text
|
||||
mdpolish/
|
||||
├── .gitignore
|
||||
├── AGENTS.md
|
||||
├── CLAUDE.md
|
||||
├── README.md
|
||||
├── pyproject.toml # Python 包、构建和开发检查配置
|
||||
├── src/
|
||||
│ └── mdpolish/
|
||||
│ ├── __init__.py # 第一版核心公共导出
|
||||
│ ├── _artifact_replay.py # 报告与评审器共用的纯快照重放
|
||||
│ ├── component.py # 组件基类和元数据契约
|
||||
│ ├── edits.py # 文本编辑验证与原子应用
|
||||
│ ├── experiment.py # 本地实验输入预检与批量编排
|
||||
│ ├── models.py # 不可变数据模型和运行状态
|
||||
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
||||
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
||||
│ ├── artifact_store.py # 私有产物目录和原子发布
|
||||
│ ├── _html_table.py # 严格 HTML 表格词法范围
|
||||
│ ├── _text_ranges.py # 精确物理行与换行范围
|
||||
│ ├── py.typed # 类型信息声明
|
||||
│ └── components/
|
||||
│ ├── __init__.py
|
||||
│ ├── arxiv_submission_stamp.py
|
||||
│ ├── html_table_double_escape.py
|
||||
│ ├── html_table_layout.py
|
||||
│ ├── manuscript_line_number.py
|
||||
│ ├── page_break_word_join.py
|
||||
│ ├── reference_spacing.py
|
||||
│ ├── repeated_running_header.py
|
||||
│ └── word_review_comment.py
|
||||
├── scripts/
|
||||
│ ├── run_clindb_arxiv_experiment.py # 只含 arXiv 组件的历史实验入口
|
||||
│ └── run_clindb_first_batch_experiment.py # ClinDB 第一批 8 组件实验入口
|
||||
├── reviewer/ # 与组件和流水线解耦的本地只读评审器
|
||||
│ ├── .nvmrc # 前端开发使用 Node.js 24
|
||||
│ ├── package.json # React 依赖、检查和构建命令
|
||||
│ ├── server/ # Python 产物适配、快照重放接口和本地 HTTP 服务
|
||||
│ ├── src/
|
||||
│ │ ├── client/ # React 全文对比、组件时间线和审计界面
|
||||
│ │ └── shared/ # 浏览器使用的内部 API 类型
|
||||
│ └── tests/ # 前端响应、界面和源码安全测试
|
||||
├── tests/
|
||||
│ ├── test_arxiv_submission_stamp.py
|
||||
│ ├── test_artifact_store.py
|
||||
│ ├── test_clindb_first_batch_pipeline.py
|
||||
│ ├── test_component.py
|
||||
│ ├── test_edits.py
|
||||
│ ├── test_experiment.py
|
||||
│ ├── test_html_table_double_escape.py
|
||||
│ ├── test_html_table_layout.py
|
||||
│ ├── test_manuscript_line_number.py
|
||||
│ ├── test_models.py
|
||||
│ ├── test_page_break_word_join.py
|
||||
│ ├── test_pipeline.py
|
||||
│ ├── test_reference_spacing.py
|
||||
│ ├── test_repeated_running_header.py
|
||||
│ ├── test_reporting.py
|
||||
│ ├── test_word_review_comment.py
|
||||
│ ├── test_artifact_replay.py
|
||||
│ ├── test_reviewer_artifacts.py
|
||||
│ └── test_reviewer_server.py
|
||||
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
||||
│ └── md/
|
||||
│ ├── dmp.md
|
||||
│ ├── ejhf.md
|
||||
│ ├── jama.md
|
||||
│ ├── sim.md
|
||||
│ └── springer.md
|
||||
├── artifacts/ # 本地敏感实验产物；Git 忽略
|
||||
│ └── <YYYY-MM-DD>/runs/<run_id>/
|
||||
│ ├── manifest.json
|
||||
│ ├── review-locator.json # 输入源定位和哈希；仅供本地评审
|
||||
│ └── documents/
|
||||
└── research-wiki/
|
||||
├── README.md # Wiki 分类与维护规则
|
||||
├── design/ # 批准前的选择；批准后冻结
|
||||
├── explanation/ # 当前有效机制及原因
|
||||
├── reference/ # 稳定查询事实
|
||||
├── guides/ # 已验证操作步骤
|
||||
└── scratch/ # 调研和未收敛材料
|
||||
```bash
|
||||
python -m pip install /path/to/mdpolish
|
||||
```
|
||||
|
||||
## 开始工作
|
||||
开发环境：
|
||||
|
||||
进入仓库后依次阅读：
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv/bin/python -m pip install -e '.[dev]'
|
||||
```
|
||||
|
||||
1. 本文件，确认当前阶段；
|
||||
2. `AGENTS.md` 或 `CLAUDE.md`，确认协作与安全边界；
|
||||
3. `research-wiki/README.md`，确认文档应放在哪里；
|
||||
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
||||
运行时只依赖 Python 标准库，支持 Python 3.11 及以上版本。
|
||||
|
||||
第一版核心机制见 `research-wiki/explanation/first-executable-core.md`，ClinDB 第一批组件见
|
||||
`research-wiki/explanation/clindb-first-batch-components.md`，本地实验产物机制见
|
||||
`research-wiki/explanation/local-experiment-artifacts.md`，本地评审器机制见
|
||||
`research-wiki/explanation/local-markdown-reviewer.md`。实验和评审器的实际操作分别见
|
||||
`research-wiki/guides/run-local-clindb-first-batch-experiment.md` 和 `research-wiki/guides/review-local-cleaning-run.md`。
|
||||
解析器、CLI、文件适配器、profile 格式、图片资产打包和独立检查能力仍需分别设计；当前本地实验适配器和评审器不能
|
||||
被推导成这些公共接口已经获批。
|
||||
## 组装自己的流水线
|
||||
|
||||
项目拥有规则、参数和顺序。下面的正则规则与两个通用修改器最终都是 `Modifier`：
|
||||
|
||||
```python
|
||||
from mdpolish import Pipeline, RunStatus, regex_replace
|
||||
from mdpolish.modifiers import html_table_entity_unescape, mapped_line_join
|
||||
|
||||
normalize_spaces = regex_replace(
|
||||
modifier_id="my_project.normalize_spaces",
|
||||
version="1.0.0",
|
||||
pattern=r" {2,}",
|
||||
replacement=" ",
|
||||
applicability="把正文中连续两个及以上的 ASCII 空格收敛为一个；项目需自行排除不适用区域。",
|
||||
)
|
||||
|
||||
join_fragments = mapped_line_join(
|
||||
(
|
||||
("exam-", "ple", "example"),
|
||||
("rule-", "based", "rule-based"),
|
||||
)
|
||||
)
|
||||
|
||||
pipeline = Pipeline(
|
||||
(
|
||||
normalize_spaces,
|
||||
join_fragments,
|
||||
html_table_entity_unescape(),
|
||||
)
|
||||
)
|
||||
|
||||
result = pipeline.transform("an exam-\nple text")
|
||||
if result.status is not RunStatus.SUCCESS:
|
||||
raise RuntimeError(f"cleaning did not succeed: {result.status}")
|
||||
|
||||
assert result.output_markdown == "an example text"
|
||||
```
|
||||
|
||||
库不会保存 `output_markdown`。调用项目应先检查状态，再自行决定写入位置：
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
source_path = Path("input.md")
|
||||
output_path = Path("output.md")
|
||||
|
||||
result = pipeline.transform(source_path.read_text(encoding="utf-8"))
|
||||
if result.status is RunStatus.SUCCESS and result.output_markdown is not None:
|
||||
output_path.write_text(result.output_markdown, encoding="utf-8")
|
||||
```
|
||||
|
||||
文件读取、输出命名、覆盖策略、批处理、CLI 和报告都属于调用项目，不属于 `mdpolish`。
|
||||
|
||||
## 编写项目自己的修改器
|
||||
|
||||
复杂规则使用普通函数返回精确候选修改，不需要继承库基类：
|
||||
|
||||
```python
|
||||
from mdpolish import DocumentSnapshot, Modifier, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
|
||||
def propose_marker_removal(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
marker = "[REMOVE-ME]"
|
||||
start = snapshot.markdown.find(marker)
|
||||
if start < 0:
|
||||
return ()
|
||||
end = start + len(marker)
|
||||
return (
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason="删除项目确认过的占位标记",
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=marker,
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
remove_marker = Modifier(
|
||||
modifier_id="my_project.remove_marker",
|
||||
version="1.0.0",
|
||||
parameters={"marker": "[REMOVE-ME]"},
|
||||
applicability="只删除项目声明的完整占位标记；不处理近似文本。",
|
||||
propose=propose_marker_removal,
|
||||
)
|
||||
```
|
||||
|
||||
修改器决定“建议改哪里、为什么改、改成什么”；公共执行器真正修改内存字符串。每项 `TextEdit` 都绑定当前快照哈希、
|
||||
半开字符串范围、预期原文和替换文本。任意候选无效时，该修改器当前批次不会产生部分修改。
|
||||
|
||||
项目函数应保持确定且无副作用，不读取文件、网络、环境变量、当前时间或随机数。Python 无法沙箱隔离任意函数；项目
|
||||
函数私下产生的外部副作用不属于本库的验证或审计范围。
|
||||
|
||||
## 通用修改器的严格边界
|
||||
|
||||
`mapped_line_join()` 只使用调用方显式传入的三元组：左片段、右片段和最终文本。它只处理相邻物理行或中间恰好一个
|
||||
同风格空行的情况，并检查 ASCII 词边界。
|
||||
|
||||
`html_table_entity_unescape()` 只在严格完整的 `<td>` / `<th>` 文本中处理 `&lt;`、`&gt;` 和
|
||||
`&amp;`。`html_table_layout()` 只调整严格单行表格的外层行布局，并保留标签、属性和单元格内容。
|
||||
|
||||
这两个 HTML 修改器采用保守的词法子集，不识别 Markdown 围栏。混合换行、嵌套标签、`<tbody>`、嵌套表格或损坏
|
||||
结构会保持原样。
|
||||
|
||||
## 仓库结构
|
||||
|
||||
```text
|
||||
src/mdpolish/
|
||||
├── models.py # 不可变快照、精确编辑、审计和结果模型
|
||||
├── modifier.py # 函数式 Modifier 契约
|
||||
├── edits.py # 批次验证与原子应用
|
||||
├── pipeline.py # 有序执行与最终稳定性复查
|
||||
├── regex.py # 正则修改器工厂
|
||||
└── modifiers/ # 少量项目无关的通用修改器
|
||||
tests/ # 只使用虚构文本的核心与通用修改器测试
|
||||
research-wiki/
|
||||
├── design/ # 已批准决策及被冻结的历史记录
|
||||
├── explanation/ # 当前有效机制
|
||||
├── reference/ # 代码无法完整表达的稳定查询事实
|
||||
├── guides/ # 已实际验证的操作步骤
|
||||
└── scratch/ # 不作为当前事实的本地草稿
|
||||
```
|
||||
|
||||
## 当前不提供
|
||||
|
||||
- 文件适配器、公共 CLI、配置文件、profile 或批处理协议；
|
||||
- 自动规则发现、注册表或默认流水线；
|
||||
- Markdown AST、完整 HTML parser 或新运行依赖；
|
||||
- artifact、报告、Web/桌面评审器；
|
||||
- 任何业务项目的规则、固定参数、文档 ID、数据或验收统计。
|
||||
|
||||
公共边界与原因见
|
||||
[`0008-generic-functional-library-boundary.md`](research-wiki/design/0008-generic-functional-library-boundary.md)，当前机制见
|
||||
[`functional-modifier-core.md`](research-wiki/explanation/functional-modifier-core.md)。旧 design 只保存历史决策，不代表当前
|
||||
交付能力。
|
||||
|
||||
## 当前可用检查
|
||||
|
||||
在已经安装开发依赖的仓库环境中运行：
|
||||
|
||||
```bash
|
||||
# 建立隔离环境并安装包与开发检查工具
|
||||
python -m venv .venv
|
||||
.venv/bin/python -m pip install -e '.[dev]'
|
||||
|
||||
# 第一版核心的基础验收
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/mypy src tests
|
||||
.venv/bin/pytest
|
||||
.venv/bin/python -m pip wheel . --no-deps --wheel-dir /tmp/mdpolish-wheel-check
|
||||
```
|
||||
|
||||
# 本地评审器要求 Node.js 24 LTS；安装依赖后依次执行静态检查、测试和生产构建
|
||||
cd reviewer
|
||||
nvm use
|
||||
npm ci
|
||||
npm run check
|
||||
cd ..
|
||||
检查本地变更：
|
||||
|
||||
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
||||
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||
|
||||
# 查看当前 Wiki 中实际存在的文档
|
||||
find research-wiki -maxdepth 2 -type f | sort
|
||||
|
||||
# 检查本地变更
|
||||
```bash
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
上述 Python 验收已于 2026-08-24 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 43 个源码、测试和实验脚本文件
|
||||
无问题，pytest 共 229 项测试通过。评审器验收也在当前用户 nvm 的 Node.js 24.19.0 环境实际运行：ESLint 和 TypeScript
|
||||
通过，Vitest 共 9 项测试通过，生产构建成功；并以一批真实的 5 文档、8 组件、155 条修改产物验证了接口读取和逐阶段哈希。
|
||||
当前环境没有可用的图形浏览器，因此页面视觉布局尚未进行真实浏览器人工验收。`requires-python` 仍以
|
||||
`pyproject.toml` 声明的 Python 3.11 及以上为准；本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
||||
上述检查已于 2026-08-26 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 21 个源码和测试文件无问题，
|
||||
pytest 共 104 项测试通过，`mdpolish-0.2.0-py3-none-any.whl` 构建成功。wheel 内容已单独检查，只包含通用 Python
|
||||
包、类型标记和包元数据，不包含项目规则、实验脚本、评审器或 Node.js 文件。
|
||||
|
||||
`pyproject.toml` 声明的 Python 3.11 及以上为支持范围；本次结果不表示已经在每个受支持版本上完成兼容性验证。
|
||||
|
||||
+2
-2
@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "mdpolish"
|
||||
version = "0.1.0"
|
||||
description = "Deterministic in-memory core for composing Markdown cleaning components"
|
||||
version = "0.2.0"
|
||||
description = "Deterministic functional core for composing exact Markdown modifiers"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
|
||||
@@ -0,0 +1,408 @@
|
||||
# 0008：函数式通用库边界与项目组件外置
|
||||
|
||||
## 状态
|
||||
|
||||
已于 2026-08-26 经用户明确批准。本文自批准起冻结；后续若改变这里的公共边界或修改语义，应新增 design，
|
||||
不得回写本文掩盖决策变化。
|
||||
|
||||
`supersedes: 0003`（范围有限）：本文拟替代基于 `Component` 抽象基类的扩展接口和当前公共导出形式；
|
||||
快照绑定、精确修改、整批验证、原子应用、显式顺序、单轮执行和最终稳定性复查继续保留。
|
||||
|
||||
`supersedes: 0005, 0007`（仓库职责范围）：本地实验产物和评审器不再属于通用库交付物。它们曾经完成的实验
|
||||
和验证仍是历史事实，但不继续作为 `mdpolish` 的当前能力维护。
|
||||
|
||||
`supersedes: 0006`（仓库职责范围）：ClinDB 第一批组件、固定参数、流水线和验收计数不再由通用库拥有。
|
||||
本文不否定这些规则当时在 5 份论文上的验证结果，只改变它们今后的代码归属。
|
||||
|
||||
`0001`、`0002` 中的保真优先、项目显式组装、Markdown 单一输入、核心无文件 I/O 和外部真实材料只读边界继续有效。
|
||||
历史 design 保持冻结，不回写成新的决定。
|
||||
|
||||
## 1. 问题与可观察现象
|
||||
|
||||
仓库的底层修改执行器和 `Pipeline` 不认识具体项目，但当前交付物已经与 ClinDB 深度绑定：
|
||||
|
||||
- 发布包内包含 `paper.*` 组件，其中多项识别条件来自 5 份论文的特定形态；
|
||||
- first-batch 脚本固定了 5 个文档 ID、8 个组件、6 条断词映射和执行顺序；
|
||||
- README、reference、guide 和 explanation 以 ClinDB 的 155 条修改作为主要成果；
|
||||
- artifact、review locator 和 React 评审器围绕这套本地实验流程继续扩张；
|
||||
- 外部使用者若只需要安全修改内核，仍会看到项目术语、实验入口和第二套 Node.js 工具链。
|
||||
|
||||
这与新的目标不一致。新的 `mdpolish` 应是一个真正可复用的 Python 库：它定义怎样描述、组合、校验和应用修改，
|
||||
但不拥有任何项目的规则集合、数据清单、流水线或评审流程。
|
||||
|
||||
当前 `Component` 已经被约束为确定、无副作用的对象，但外部扩展仍需要继承抽象基类、实现四个属性和一个私有方法。
|
||||
这里的继承没有提供运行时隔离，反而增加了编写简单修改器的仪式。通用库更适合把行为表达为纯函数，把身份和参数表达为
|
||||
不可变数据，再由 `Pipeline` 组合。
|
||||
|
||||
## 2. 目标与非目标
|
||||
|
||||
### 2.1 目标
|
||||
|
||||
- 把仓库收敛为可安装、项目无关的 Python 修改库；
|
||||
- 以纯函数加不可变元数据代替 `Component` 抽象基类；
|
||||
- 保留当前快照、精确编辑、冲突检查、原子应用、审计和稳定性复查不变量；
|
||||
- 提供通用的正则修改器工厂，使项目规则可以在项目仓库中声明；
|
||||
- 提供少量经合成测试验证、无需项目数据的通用修改器；
|
||||
- 让任何项目显式创建自己的修改器、参数和 `Pipeline`；
|
||||
- 从当前能力、Python 包、测试和用户文档中移除 ClinDB、论文缩写、固定映射和真实样本计数；
|
||||
- 移除通用库不再拥有的本地实验层和评审器工具链；
|
||||
- 保持运行时只依赖 Python 标准库。
|
||||
|
||||
### 2.2 非目标
|
||||
|
||||
- 不在本轮建设公共 CLI、配置文件、profile、插件发现、LSP、Web 服务或桌面应用；
|
||||
- 不建立新的 artifact schema、文件适配器或批处理协议；
|
||||
- 不把 ClinDB 组件迁移到另一个仓库；修改其他仓库需要另行授权；
|
||||
- 不修改、移动、复制或删除本地 `data/`、`artifacts/` 和仓库外真实材料；
|
||||
- 不把现有项目规则改名后冒充通用组件；
|
||||
- 不承诺任意用户函数在运行时被沙箱隔离；纯函数和无副作用是修改器契约，由内置实现和测试保证；
|
||||
- 不在本轮引入 Markdown parser、HTML parser 或新的运行依赖；
|
||||
- 不提供默认流水线。安装库或导入修改器不会自动修改任何文本。
|
||||
|
||||
## 3. 新的依赖方向
|
||||
|
||||
```text
|
||||
使用项目
|
||||
├── 项目规则函数
|
||||
├── 项目参数与顺序
|
||||
└── 项目文件、CLI、profile、报告和评审
|
||||
│
|
||||
▼
|
||||
mdpolish
|
||||
├── 不可变快照与修改模型
|
||||
├── 函数式 Modifier 协议
|
||||
├── 修改验证与原子执行器
|
||||
├── Pipeline
|
||||
├── 通用正则修改器工厂
|
||||
└── 少量通用内置修改器
|
||||
```
|
||||
|
||||
依赖只能从使用项目指向 `mdpolish`。`mdpolish` 不导入项目包，不读取项目目录，不根据文件名选择规则，
|
||||
也不保存任何项目的默认顺序或参数。
|
||||
|
||||
## 4. 函数式修改器接口
|
||||
|
||||
### 4.1 行为与元数据分开
|
||||
|
||||
修改行为使用一个普通可调用对象：
|
||||
|
||||
```python
|
||||
ModifierFunction = Callable[
|
||||
[DocumentSnapshot],
|
||||
tuple[ProposedChange, ...],
|
||||
]
|
||||
```
|
||||
|
||||
库使用不可变的 `Modifier` 保存审计所需元数据和函数：
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Modifier:
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
propose: ModifierFunction
|
||||
```
|
||||
|
||||
精确字段名可以在实现中根据类型检查做机械调整，但必须满足以下决定：
|
||||
|
||||
- 不要求外部作者继承基类；
|
||||
- 修改逻辑是接收一个快照、返回不可变候选修改的普通函数；
|
||||
- `modifier_id`、版本、参数和适用边界不得从函数名、模块路径或闭包内容自动猜测；
|
||||
- 元数据在流水线开始时冻结，运行期间变化视为契约错误；
|
||||
- 函数只能提出修改，仍不能直接应用编辑或返回一整篇新 Markdown；
|
||||
- 相同输入、身份、版本和参数必须产生相同顺序的候选修改；
|
||||
- 修改器不得读取文件、网络、环境变量、当前时间或随机数。
|
||||
|
||||
保留显式元数据是为了让外包组件仍可被审计和复现。函数式编程不等于丢弃组件身份和版本。
|
||||
|
||||
### 4.2 修改权限与数据流
|
||||
|
||||
这里把“修改权”拆成三层，避免把业务判断、字符串执行和文件写回混为一件事：
|
||||
|
||||
1. **使用项目拥有规则决策权。** 项目决定启用哪些 `Modifier`、传入什么参数、按什么顺序运行。修改器的
|
||||
`propose(snapshot)` 只判断“建议改哪里、为什么改、改成什么”，不能原地改变不可变快照，也不能用一整篇
|
||||
新 Markdown 绕过精确编辑契约。
|
||||
2. **`mdpolish` 核心拥有验证和内存执行权。** `Pipeline` 调用修改器，公共执行器检查快照、范围、原文、重复和
|
||||
冲突，然后才通过字符串切片应用编辑。一个修改器本轮的候选修改必须整批通过；任意一项无效时，该批不产生
|
||||
部分结果。
|
||||
3. **使用项目拥有持久化决定权。** `Pipeline` 只返回状态、结果文本和审计记录，不读取或写入文件。调用方检查
|
||||
`success`、`failed` 或 `unstable` 后，自行决定是否以及怎样保存；`mdpolish` 不会自动覆盖输入文件。
|
||||
|
||||
修改器至少通过 `ProposedChange` 和 `TextEdit` 向核心提交以下信息：
|
||||
|
||||
- 候选修改及每条精确编辑所绑定的 `snapshot_sha256`；
|
||||
- 人可读且非空的修改原因 `reason`；
|
||||
- 使用 Python 字符串索引表示的半开范围 `TextSpan(start, end)`；
|
||||
- 该范围当前应有的原文 `expected_text`；
|
||||
- 替换后的文本 `replacement`。
|
||||
|
||||
因此，一次调用的数据流固定为：
|
||||
|
||||
```text
|
||||
项目组装 Modifier 与顺序
|
||||
↓
|
||||
Modifier.propose(snapshot) 提出精确修改
|
||||
↓
|
||||
mdpolish 验证并在内存中原子应用
|
||||
↓
|
||||
Pipeline 返回状态、结果和审计记录
|
||||
↓
|
||||
项目决定是否写回文件
|
||||
```
|
||||
|
||||
项目规则当然可以决定自己的业务语义，也可以提出删除、替换或插入；但只有满足上述契约的候选修改才由核心执行。
|
||||
Python 无法阻止项目函数私下写文件或直接调用字符串替换，这类副作用属于调用方绕过库契约，不属于 `mdpolish`
|
||||
保证、审计或回滚的修改。
|
||||
|
||||
### 4.3 `Pipeline`
|
||||
|
||||
`Pipeline` 改为接收 `Iterable[Modifier]`，不再要求 `Component` 子类。其行为继续保持：
|
||||
|
||||
1. 预检全部修改器身份、版本、参数和重复 ID；
|
||||
2. 按调用方顺序对当前快照调用 `propose`；
|
||||
3. 由公共执行器验证和原子应用当前修改器批次；
|
||||
4. 为下一个修改器建立新快照；
|
||||
5. 全部执行一次后，对最终快照进行只读稳定性复查；
|
||||
6. 返回 `success`、`failed` 或 `unstable`，不自动开始第二轮。
|
||||
|
||||
本轮不自动重排修改器，不自动选择修改器，也不提供全局默认组合。
|
||||
|
||||
### 4.4 外部项目的两种扩展方式
|
||||
|
||||
简单规则优先使用库提供的工厂：
|
||||
|
||||
```python
|
||||
remove_stamp = regex_replace(
|
||||
modifier_id="my_project.remove_stamp",
|
||||
version="1.0.0",
|
||||
pattern=r"...",
|
||||
replacement="",
|
||||
)
|
||||
```
|
||||
|
||||
复杂规则直接提供纯函数，再显式构造 `Modifier`：
|
||||
|
||||
```python
|
||||
def propose_project_changes(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
...
|
||||
|
||||
project_modifier = Modifier(
|
||||
modifier_id="my_project.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="...",
|
||||
propose=propose_project_changes,
|
||||
)
|
||||
```
|
||||
|
||||
库不会自动注册或发现这些对象。
|
||||
|
||||
`Modifier` 是 `Pipeline` 接收的统一值类型；`regex_replace()` 是创建这种值的便捷工厂，不是与 `Modifier` 并列的
|
||||
另一套接口。它返回的也是一个完整 `Modifier`。复杂规则与正则规则进入流水线后，都遵守同一套提议、验证、执行
|
||||
和审计流程。
|
||||
|
||||
## 5. 通用正则修改器
|
||||
|
||||
第一版提供一个安全、窄边界的 `regex_replace()` 工厂：
|
||||
|
||||
- 调用方必须显式给出稳定 `modifier_id` 和版本；
|
||||
- 输入为正则 pattern、固定 replacement、flags 和适用边界说明；
|
||||
- pattern、replacement 和 flags 完整进入修改器参数，运行结果可以复核；
|
||||
- 按 Python `re.finditer()` 的确定顺序定位非重叠匹配；
|
||||
- replacement 使用 Python 正则的固定替换模板展开；
|
||||
- 每个匹配生成绑定当前快照、范围和原文的精确编辑；
|
||||
- 第一版拒绝零长度匹配，避免隐式插入和边界顺序不明确；
|
||||
- 工厂不读 YAML，不维护规则注册表，也不提供默认规则集；
|
||||
- 需要动态 replacement、跨匹配聚合或结构判断时，项目应编写自己的纯函数。
|
||||
|
||||
这个工厂提供正则能力，但不会退回“全局替换后返回整篇文本”的旧实现。所有修改仍经过公共执行器。
|
||||
|
||||
## 6. 通用内置修改器
|
||||
|
||||
第一版只保留以下与具体项目无关、能够用合成样例完整说明的能力：
|
||||
|
||||
### 6.1 映射驱动的跨行片段合并
|
||||
|
||||
- 保留当前“左片段、右片段、结果文本”的算法；
|
||||
- 修改器身份改为领域无关名称；
|
||||
- 映射必须由调用方传入并写入参数；
|
||||
- 库中不保留 ClinDB 的 6 条映射；
|
||||
- 不使用词典、模型或项目默认值猜测结果。
|
||||
|
||||
### 6.2 严格 HTML 表格实体解除一层转义
|
||||
|
||||
- 只处理严格识别的 `<td>` / `<th>` 文本；
|
||||
- 只处理明确支持的双重实体；
|
||||
- 不处理属性、表格外文本或任意 HTML；
|
||||
- 名称和说明不得宣称已经支持通用损坏 HTML 修复。
|
||||
|
||||
### 6.3 严格单行 HTML 表格布局
|
||||
|
||||
- 保留 HTML 标签、属性、行和单元格内容；
|
||||
- 只调整严格单行表格的外层行布局；
|
||||
- 混合换行和不支持的结构保持原样；
|
||||
- 它是保守的通用修改器，不是 HTML→GFM 转换器。
|
||||
|
||||
当前 `_html_table.py` 可以作为这两个修改器的私有词法辅助实现。由于它不识别 Markdown 围栏和完整方言，
|
||||
公开说明必须保留这一边界。以后扩大为通用 HTML 清洗前必须另建设计。
|
||||
|
||||
除上述三类外，当前所有正式业务组件都移出 Python 包。合成示例可以演示怎样用 `regex_replace()` 和自定义函数
|
||||
建立修改器，但不得包含论文原文、ClinDB 名称、固定文档 ID 或项目参数。
|
||||
|
||||
## 7. 从通用库移出的内容
|
||||
|
||||
设计获批后，当前工作树中以下内容不再作为 `mdpolish` 当前实现保留：
|
||||
|
||||
- `paper.word_review_comment`；
|
||||
- `paper.manuscript_line_number`；
|
||||
- `paper.arxiv_submission_stamp`；
|
||||
- `paper.repeated_running_header`；
|
||||
- `paper.reference_spacing`；
|
||||
- ClinDB 固定断词映射与 first-batch 组合；
|
||||
- `scripts/run_clindb_arxiv_experiment.py`；
|
||||
- `scripts/run_clindb_first_batch_experiment.py`；
|
||||
- `experiment.py`、`artifact_store.py`、`reporting.py` 和只服务 artifact 的重放代码；
|
||||
- `reviewer/` Python 服务、React 前端、Node.js 依赖与构建配置；
|
||||
- 对应项目测试、实验 guide、当前机制 explanation、ClinDB/GovDoc reference 和项目 scratch；
|
||||
- README 中的数据目录、实验运行、155 条修改和评审器说明。
|
||||
|
||||
`research-wiki/design/0001` 至 `0007` 继续作为冻结的历史决策保留。`0008` 负责明确它们不再描述当前交付范围，
|
||||
避免通过删除历史记录让旧方向看起来从未发生。
|
||||
|
||||
本轮只从 Git 跟踪的通用库工作树移除上述实现和当前说明。Git 历史仍可恢复这些内容。本地被忽略的 `data/`、
|
||||
`artifacts/`、虚拟环境、构建缓存和真实材料一律不删除、不移动、不读取、不复制。
|
||||
|
||||
## 8. 拟采用的源码结构
|
||||
|
||||
```text
|
||||
pyproject.toml
|
||||
src/mdpolish/
|
||||
├── __init__.py
|
||||
├── models.py
|
||||
├── modifier.py
|
||||
├── edits.py
|
||||
├── pipeline.py
|
||||
├── regex.py
|
||||
├── _text_ranges.py
|
||||
├── _html_table.py
|
||||
└── modifiers/
|
||||
├── __init__.py
|
||||
├── mapped_line_join.py
|
||||
├── html_table_entities.py
|
||||
└── html_table_layout.py
|
||||
tests/
|
||||
├── test_models.py
|
||||
├── test_modifier.py
|
||||
├── test_edits.py
|
||||
├── test_pipeline.py
|
||||
├── test_regex.py
|
||||
├── test_mapped_line_join.py
|
||||
├── test_html_table_entities.py
|
||||
└── test_html_table_layout.py
|
||||
```
|
||||
|
||||
`modifier.py` 只定义函数式修改器的身份和契约；`regex.py` 提供通用工厂；`modifiers/` 只放领域无关实现。
|
||||
业务组件不以示例为由继续进入发布包。
|
||||
|
||||
是否把 `DocumentSnapshot`、`ProposedChange`、`TextEdit` 等低层对象继续放在顶层公共命名空间，由实现时按外部作者
|
||||
能否编写自定义函数决定。本设计要求它们具有受支持的导入路径，但不要求所有对象都堆在 `mdpolish.__init__`。
|
||||
|
||||
## 9. 兼容与版本
|
||||
|
||||
这是明确的破坏性重构：
|
||||
|
||||
- 删除 `Component` 抽象基类；
|
||||
- `Pipeline` 的元素类型改变；
|
||||
- 删除项目组件和实验接口；
|
||||
- wheel 不再包含项目工作流；
|
||||
- README 的使用方式改变。
|
||||
|
||||
当前仓库仍处于 `0.x` 研究阶段，没有已批准的外部兼容承诺。实现后包版本从 `0.1.0` 更新为 `0.2.0`，不建立
|
||||
兼容旧 `Component` 子类的适配层。保留适配层会让新的公共边界继续携带旧项目架构，本轮不采用。
|
||||
|
||||
## 10. 测试与验收
|
||||
|
||||
### 10.1 核心不变量
|
||||
|
||||
现有以下测试语义必须迁移并继续通过：
|
||||
|
||||
- 空文本、中文、组合字符和不同换行；
|
||||
- 插入、删除、替换、多编辑候选和组件批次原子性；
|
||||
- 过期快照、范围越界、原文不符、重复与冲突编辑明确失败；
|
||||
- 后一个修改器读取新快照；
|
||||
- 最终复查发现连锁修改时返回 `unstable`；
|
||||
- 异常和元数据变化返回安全错误；
|
||||
- 相同输入、修改器和参数得到相同结果；
|
||||
- 成功结果再次运行零修改；
|
||||
- 输入字符串不被原地改变。
|
||||
|
||||
### 10.2 函数式 API
|
||||
|
||||
至少覆盖：
|
||||
|
||||
- 普通函数不继承任何库基类即可成为 `Modifier`；
|
||||
- 无效 ID、版本、参数、适用边界和不可调用对象明确失败；
|
||||
- 两个相同 ID 的修改器仍在预检失败；
|
||||
- 函数异常不会被吞掉，也不会把部分文本当成功输出；
|
||||
- 无效候选修改导致当前修改器整批失败，不应用其中任何一项；
|
||||
- 流水线记录实际修改器身份、版本、参数和顺序；
|
||||
- 流水线运行只产生内存结果，不读取或写回调用方文件。
|
||||
|
||||
### 10.3 正则工厂
|
||||
|
||||
至少覆盖：
|
||||
|
||||
- 普通替换、捕获组展开、多命中和零命中；
|
||||
- flags 和参数记录；
|
||||
- 零长度模式拒绝；
|
||||
- 原文、范围、理由和哈希审计正确；
|
||||
- 第二次运行稳定；
|
||||
- 没有 YAML、全局注册表或默认启用行为。
|
||||
|
||||
### 10.4 通用修改器
|
||||
|
||||
- 只使用小型虚构文本；
|
||||
- 映射合并没有内置项目词表；
|
||||
- HTML 修改器保留标签、属性、表格外文本和不支持结构；
|
||||
- 正向、反向、换行、无末尾换行、确定性和幂等性均有覆盖；
|
||||
- 不读取 `data/`、`artifacts/` 或仓库外数据。
|
||||
|
||||
### 10.5 仓库边界
|
||||
|
||||
实现完成后必须确认：
|
||||
|
||||
- wheel 只包含通用 Python 库；
|
||||
- Python 源码、测试、README、explanation 和 guide 不包含 ClinDB 文档 ID、固定映射和项目流水线；
|
||||
- 除冻结的 `research-wiki/design/0001` 至 `0007` 外，当前文档不再把项目实验描述为库能力；
|
||||
- `reviewer/`、项目脚本和 Node.js 配置不再属于当前工作树；
|
||||
- `.gitignore` 继续保护本地 `data/` 和 `artifacts/`；
|
||||
- `git status` 不出现被忽略的真实材料；
|
||||
- `AGENTS.md` 与 `CLAUDE.md` 除标题外正文一致。
|
||||
|
||||
基础检查仍使用 Ruff、mypy 和 pytest。实现后根据实际文件更新根 README 中的唯一检查命令，并实际运行后才报告通过。
|
||||
|
||||
## 11. 风险与代价
|
||||
|
||||
- **历史实验入口消失：** 当前分支不再直接运行 ClinDB；旧代码仍在 Git 历史中，未来迁移需在目标项目重新评审。
|
||||
- **评审能力不再随库提供：** 这符合最小公共库目标；多个项目产生共同需求后再设计独立扩展。
|
||||
- **函数不能被强制纯净：** Python 无法仅靠类型禁止文件和网络副作用；内置修改器通过实现与测试保证，外部修改器由调用方负责。
|
||||
- **正则工厂可能被滥用：** 它只保证修改执行安全，不保证项目正则语义正确；项目仍需为自己的模式、反例和版本负责。
|
||||
- **HTML 修改器的“通用”范围有限：** 严格失败关闭，宁可漏处理未知结构，也不扩大成未经验证的 HTML 修复器。
|
||||
- **没有文件产品入口：** 第一版调用方需要自己读取 Markdown 和处理结果；这正是新的库边界，不是遗漏。
|
||||
- **破坏现有导入：** `0.2.0` 明确表示新边界，不维护尚未承诺的旧扩展接口。
|
||||
|
||||
## 12. 批准后的实施顺序
|
||||
|
||||
用户明确批准本文后，按以下顺序实施：
|
||||
|
||||
1. 新增函数式 `Modifier`，迁移核心测试，使 `Pipeline` 不再依赖抽象基类；
|
||||
2. 实现 `regex_replace()` 和通用修改器，全部使用合成测试；
|
||||
3. 更新公共导出、包描述和版本；
|
||||
4. 移除项目组件、脚本、实验层、评审器及其测试；
|
||||
5. 清理当前 README、explanation、reference、guide 和 scratch，只保留通用库当前事实与冻结历史 design；
|
||||
6. 核对 `.gitignore`，但不触碰任何被忽略的真实输入和产物；
|
||||
7. 运行 README 中实际保留的全部检查，检查 wheel 内容、Git diff、文档镜像和工作区状态。
|
||||
|
||||
本文批准不授权提交、推送、创建 PR、发布、修改其他仓库，或删除、移动、复制本地真实材料。
|
||||
@@ -1,83 +0,0 @@
|
||||
# arXiv 提交边栏戳为什么能自动删除
|
||||
|
||||
## 1. 可观察的问题
|
||||
|
||||
部分 arXiv 论文转换为 Markdown 后，会把提交页边栏中的编号、分类和日期留下来，形成一整行独立文字。它不是
|
||||
论文正文，却会进入后续分块、检索和对比。与此同时，论文参考文献也可能包含 `arXiv:`；只要见到这个子串就删行，
|
||||
会损坏合法引用。
|
||||
|
||||
当前组件只处理前一种格式固定的独立行。它的决策来自已批准的
|
||||
[`0004-arxiv-submission-stamp-component.md`](../design/0004-arxiv-submission-stamp-component.md)，项目范围编号为 H1。
|
||||
精确模式、类名和返回对象以
|
||||
[`arxiv_submission_stamp.py`](../../src/mdpolish/components/arxiv_submission_stamp.py) 及其
|
||||
[`测试`](../../tests/test_arxiv_submission_stamp.py) 为准。
|
||||
|
||||
## 2. 当前识别边界
|
||||
|
||||
组件逐个读取物理行，只在整行同时具有以下结构时提出删除：
|
||||
|
||||
```text
|
||||
arXiv:<新版数字编号和版本> [<ASCII 分类>] <日> <英文月份缩写> <四位年份>
|
||||
```
|
||||
|
||||
首尾空格、列表或引用前缀、缺少版本、旧式编号、错误月份以及句子中的 `arXiv:` 都不会命中。组件没有参数，
|
||||
调用方不能传入更宽松的正则表达式改变同一版本的语义。
|
||||
|
||||
当前版本有意不解析 Markdown 块结构。围栏代码、HTML 注释或其他块中如果存在一行完整目标文字，同样会被删除。
|
||||
这是 `0004` 明确接受的代价，不是实现遗漏。以后出现必须保留的真实反例时，需要重新评审识别边界并更新组件版本。
|
||||
|
||||
## 3. 删除如何保持原文边界
|
||||
|
||||
每个命中行产生一个候选修改和一个删除型文本编辑。删除范围包含该行自己的 `\n`、`\r\n` 或单独 `\r`；
|
||||
没有行尾的末行只删除文字，不拿走前一行已有的行尾。
|
||||
|
||||
| 输入位置 | 当前行为 |
|
||||
| --- | --- |
|
||||
| 首行且有行尾 | 连同行尾删除，后续正文成为首行 |
|
||||
| 文档中间 | 连同目标行自己的行尾删除，前后内容保持两行 |
|
||||
| 末行且没有行尾 | 只删除目标文字，保留前一行原有行尾 |
|
||||
| 多个目标行 | 每行一个候选，按原文顺序记录，作为一个组件批次原子应用 |
|
||||
|
||||
组件不整理空行、不统一换行符，也不改变未命中的字符。候选修改绑定当前快照哈希和准确原文，仍由公共修改执行器
|
||||
验证和应用；组件本身没有文件读写或独立 `transform()`。
|
||||
|
||||
## 4. 审计与稳定性
|
||||
|
||||
组件标识为 `paper.arxiv_submission_stamp`，版本为 `1.0.0`，参数为空。每条实际删除记录固定理由，并保留删除原文、
|
||||
原始范围、组件位置以及批次修改前后的哈希。
|
||||
|
||||
删除完成后目标行已经不存在。流水线最终复查不应再得到候选修改；把成功输出再次交给同一组件，也应保持原文不变
|
||||
且产生零条实际改动。
|
||||
|
||||
## 5. 已完成验证
|
||||
|
||||
合成测试覆盖严格匹配、反向引用、首行/中间/末行、三种行尾、多个命中、围栏中仍删除、审计字段、确定性和
|
||||
第二次运行零修改。测试只使用短小的虚构字符串，不含真实论文片段。
|
||||
|
||||
2026-08-22 先对本地 5 份 ClinDB-ReviewBench Markdown 做了只读、纯内存复核：
|
||||
|
||||
| 复核项 | 结果 |
|
||||
| --- | --- |
|
||||
| 输入范围 | dmp、jama、ejhf、sim、springer 各 1 份，共 5 份 |
|
||||
| 实际删除 | sim 1 行、springer 1 行，其余 0 行，共 2 行 |
|
||||
| 合法反向样例 | springer 的 2 处 `arXiv preprint arXiv:` 修改前后均保留 |
|
||||
| 第二次运行 | 5 份合计 0 条修改 |
|
||||
| 源文件复读 | 5/5 与处理前内存内容一致，没有回写 |
|
||||
|
||||
这次早期复核没有保存清洗后 Markdown，也没有把真实原文复制进测试、日志或仓库。它是 `0004` 当时验证边界的
|
||||
历史事实。
|
||||
|
||||
`0005` 批准本地产物机制后，同日又完成一次保存型实验：5 份文档全部为 `success`，仍然只修改 sim 和 springer
|
||||
各一处；输出哈希全部匹配，输入运行前后字节不变，Springer 两处合法引用仍保留。产物位于
|
||||
`artifacts/2026-08-22/runs/clindb-arxiv-stamp-artifacts-v1/`，只在本机保留并受 Git 忽略。具体产物结构和验证结果见
|
||||
[`local-experiment-artifacts.md`](local-experiment-artifacts.md)。安装、静态检查和完整测试命令仍只在根目录
|
||||
[`README.md`](../../README.md#当前可用检查) 维护。
|
||||
|
||||
## 6. 剩余边界
|
||||
|
||||
这个组件最初证明了第一条严格删除规则能够在公共核心上闭环。此后 ClinDB 第一批另外 7 个组件已经按 `0006` 实现，
|
||||
当前完整组合见 [`clindb-first-batch-components.md`](clindb-first-batch-components.md)。这仍不表示论文内容问题全部解决；
|
||||
通用文件接口、profile、公共 CLI、图片资产和通用批处理仍不存在。
|
||||
|
||||
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
||||
提高命中数量直接修改正则表达式。
|
||||
@@ -1,116 +0,0 @@
|
||||
# ClinDB 第一批组件如何在不猜正文的前提下完成清洗
|
||||
|
||||
## 1. 它解决的实际问题
|
||||
|
||||
5 份 ClinDB 论文 Markdown 同时包含编辑痕迹、转换噪声和排版噪声。它们看起来都像“删掉几行或整理一下格式”，
|
||||
实际误删边界不同：行首数字可能是手稿行号，也可能是作者单位；`arXiv:` 可能是边栏戳，也可能是合法参考文献；
|
||||
编号列表可能属于 References，也可能是正文方法步骤。
|
||||
|
||||
当前实现没有建立一个能随意改全文的“大清洗器”，而是把第一批确定问题拆成 8 个组件。每个组件只识别一种证据，
|
||||
返回快照绑定的精确 `TextEdit`，由公共流水线统一验证、应用和记录。
|
||||
|
||||
清洗语义来自已批准的
|
||||
[`0006-clindb-first-batch-cleaning-components.md`](../design/0006-clindb-first-batch-cleaning-components.md)，
|
||||
输入范围和稳定计数见
|
||||
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。
|
||||
|
||||
## 2. 当前组件和顺序
|
||||
|
||||
```text
|
||||
Word 批注 ─┐
|
||||
手稿行号 ──┼──► arXiv 戳 ──► 重复页眉 ──► 映射断词
|
||||
│ │
|
||||
│ └────────────► 参考文献空行
|
||||
│
|
||||
└────► HTML 双重实体 ──► HTML 表格布局
|
||||
```
|
||||
|
||||
实验脚本固定按以下顺序组装：
|
||||
|
||||
| 顺序 | 组件 | 当前作用 |
|
||||
| ---: | --- | --- |
|
||||
| 1 | `paper.word_review_comment` | 删除完整单行 Word 批注及其后一个空行 |
|
||||
| 2 | `paper.manuscript_line_number` | 删除由长递增序列确认的手稿行号 |
|
||||
| 3 | `paper.arxiv_submission_stamp` | 删除严格整行提交戳 |
|
||||
| 4 | `paper.repeated_running_header` | 删除重复页眉，并接回有明确续句证据的段落 |
|
||||
| 5 | `paper.page_break_word_join` | 只应用本次运行参数中记录的词片段映射 |
|
||||
| 6 | `markdown.html_table_double_escape` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||
| 7 | `markdown.html_table_layout` | 保留 HTML 内容，把单行表格展开成一行一个 `<tr>` |
|
||||
| 8 | `paper.reference_spacing` | 只在完整连续的 References 章节中统一条目空行 |
|
||||
|
||||
顺序不是为了让结果“看起来整齐”。dmp 的一个重复页眉正好位于第 18、19 条参考文献之间；如果不先删除页眉，
|
||||
参考文献组件就不能确认这是完整连续序列。HTML 实体先修改小范围 token，表格布局再基于新快照替换整张表，
|
||||
两类修改仍能在审计中分别追踪。
|
||||
|
||||
## 3. 为什么行号不能使用全局正则
|
||||
|
||||
JAMA 文档中一共有 134 个看似 `数字 + 空格` 的行首前缀。前 59 个位于 Abstract 之前，是作者单位编号；真正的手稿
|
||||
行号只有 Abstract 之后的 75 个。
|
||||
|
||||
当前组件要求：
|
||||
|
||||
- 文档中恰好有一个严格 `## Abstract`；
|
||||
- 只看它之后的候选；
|
||||
- 候选数字全部严格递增；
|
||||
- 至少有 20 个候选，并至少有 2 个数字标题。
|
||||
|
||||
任一条件失败就整篇不改。这样会漏掉较短的带行号手稿，但不会为了提高命中率删除作者单位或零散数字段落。
|
||||
|
||||
## 4. 为什么断词使用显式映射
|
||||
|
||||
“行尾连字符加下一行小写字母”无法决定连字符应删还是保留：`possi-` + `bly` 应成为 `possibly`，而
|
||||
`SOFA-` + `based` 应保留为 `SOFA-based`。dmp 还存在 `threshold.` + `olds`，它同时包含多余句点和重复片段，
|
||||
普通词典也无法解释。
|
||||
|
||||
因此组件的实际参数记录三项：左片段、右片段和结果词。只有相邻物理行或中间恰好一个空行、词边界完整且映射唯一时
|
||||
才修改。增加新词不是自动学习行为，需要先批准新的项目映射；组件算法版本不变时，运行清单仍能通过参数区分实际语义。
|
||||
|
||||
## 5. 两个表格组件怎样共享范围
|
||||
|
||||
`_html_table.py` 只识别当前转换器输出的严格子集：`<tr>` 直接位于 `<table>` 下，`<td>` / `<th>` 直接位于
|
||||
`<tr>` 下，标签完整闭合，单元格中没有嵌套标签。它返回原字符串下标，不生成 DOM，也不重新渲染全文。
|
||||
|
||||
实体组件只处理单元格文本中的三个精确 token：
|
||||
|
||||
```text
|
||||
&lt; → <
|
||||
&gt; → >
|
||||
&amp; → &
|
||||
```
|
||||
|
||||
结果仍是合法 HTML 源码中的单层实体。标签、属性、表格外文本和其他实体不受影响。
|
||||
|
||||
布局组件只处理整个片段没有换行的严格表格。它原样复用 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和结束标签，
|
||||
只增加外层换行与两个空格缩进。当前 9 张真实表格都有 `colspan`，所以全部保留 HTML；实现没有猜测表头，也没有转 GFM。
|
||||
|
||||
遇到未闭合、嵌套、额外结构标签或混合换行时，扫描失败关闭，保持原文。
|
||||
|
||||
## 6. 共享代码为什么仍然很小
|
||||
|
||||
- `_text_ranges.py` 只提供 Python 字符下标下的物理行、行尾和空行关系；
|
||||
- `_html_table.py` 只提供严格 HTML 表格、行和单元格范围；
|
||||
- 业务组件依赖这两个私有模块，但辅助模块不依赖组件、流水线或文件层；
|
||||
- 文件实验层只接收已经组装的 `Pipeline`，不知道任何识别规则。
|
||||
|
||||
因此以后放宽某个业务规则通常只改一个组件及其测试；替换 HTML 识别方式不会改变 `DocumentSnapshot`、
|
||||
`ProposedChange`、`Change` 或 JSON 产物；未来引入 Profile 时，也只接管当前脚本里的组件组装。
|
||||
|
||||
## 7. 当前验证结果和边界
|
||||
|
||||
2026-08-22 在 Python 3.13.11 环境完成：
|
||||
|
||||
- Ruff 通过；
|
||||
- mypy 检查 37 个文件无问题；
|
||||
- pytest 204 项通过；
|
||||
- 5 份本地论文全部 `success`，合计 155 条 `Change`；
|
||||
- 对 5 份成功输出再次运行，全部 `success` 且零修改；
|
||||
- 输入运行前后哈希不变；
|
||||
- JAMA Abstract 前内容不变，Springer 两条合法 arXiv 参考文献保留；
|
||||
- 图片引用文字不变，但实验产物没有复制图片资产。
|
||||
|
||||
保存型实验位于本机 Git 忽略的
|
||||
`artifacts/2026-08-22/runs/clindb-first-batch-v1/`。运行和复核方法见
|
||||
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。
|
||||
|
||||
这次成功只证明批准的 8 类规则在当前 5 份输入上闭环。截断、缺表、乱码、OCR 语义错误、图片资产、修订词选择和一般
|
||||
段落重排仍不在自动清洗范围内。
|
||||
@@ -1,155 +0,0 @@
|
||||
# 第一版内存清洗核心如何工作
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
清洗组件如果直接返回一整篇新 Markdown，调用方只能看到修改后的结果，很难确认它实际改了哪里。组件保存的
|
||||
旧位置还可能在文本变化后误中另一段内容；同一批修改发生重叠时，按不同顺序执行也可能得到不同结果。
|
||||
|
||||
当前核心把“判断应该改什么”和“安全地执行修改”分开：组件只描述绑定当前文本的精确修改，公共执行器统一
|
||||
验证并应用。核心建立时先不引入真实清洗规则、文件读写或 Markdown parser，让组合与审计协议独立可运行；
|
||||
现在第一个真实组件已经在这套协议上完成验证，没有改变核心接口。
|
||||
|
||||
已经实现的范围来自已批准的
|
||||
[`0003-first-executable-core-architecture.md`](../design/0003-first-executable-core-architecture.md)。精确类名、字段和
|
||||
函数签名以 [`src/mdpolish/`](../../src/mdpolish/) 中的代码和测试为准，本文不维护第二份 API 清单。
|
||||
|
||||
## 2. 当前数据流
|
||||
|
||||
```text
|
||||
输入 Markdown 字符串
|
||||
│
|
||||
▼
|
||||
带内容哈希的当前快照
|
||||
│
|
||||
▼
|
||||
组件提出精确修改 ──► 整批验证 ──► 整批应用 ──► 新快照
|
||||
│ │
|
||||
└──────── 按组件顺序重复 ◄──────────────┘
|
||||
│
|
||||
▼
|
||||
最终只重新提议，不再应用
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
success unstable failed
|
||||
```
|
||||
|
||||
核心只有四层：
|
||||
|
||||
| 层次 | 当前职责 | 明确不负责 |
|
||||
| --- | --- | --- |
|
||||
| 数据模型 | 保存快照、范围、候选修改、实际改动、错误和结果 | 业务规则和文件路径 |
|
||||
| 组件基类 | 声明身份、版本、参数、适用边界并提出修改 | 应用修改和组织流水线 |
|
||||
| 修改执行器 | 统一验证并原子应用一个组件批次 | 判断 Markdown 业务语义 |
|
||||
| 流水线 | 排列组件、刷新快照、处理失败并做最终复查 | 读取文件、选择项目 profile |
|
||||
|
||||
依赖保持单向：组件基类和修改执行器只依赖数据模型，流水线可以调用前三者，底层模块不反向调用流水线。
|
||||
|
||||
## 3. 为什么修改必须绑定快照
|
||||
|
||||
每个 Markdown 快照都带有根据完整字符串计算的 SHA-256。候选修改和其中每条文本编辑都必须指向这个哈希，
|
||||
还要同时提供原文范围和该范围预期出现的文字。
|
||||
|
||||
执行时会再次检查：
|
||||
|
||||
1. 哈希仍然对应当前快照；
|
||||
2. 范围没有越过字符串边界；
|
||||
3. 当前位置的文字与组件声明的预期原文完全一致；
|
||||
4. 替换后确实会改变内容。
|
||||
|
||||
任何一项不满足，当前组件的整个批次都不会执行。这使位置只在其产生时的快照内有效，不允许把旧候选修改悄悄
|
||||
套到后来变化的 Markdown 上。
|
||||
|
||||
范围使用 Python 字符串下标，而不是 UTF-8 字节位置。核心保留输入的换行、Unicode 形式和末尾换行，不做隐式
|
||||
规范化。
|
||||
|
||||
## 4. 组件为什么只提出自动修改
|
||||
|
||||
第一版组件只返回能够立即、唯一执行的候选修改。遇到不知道正确修法的截断、损坏表格、疑似幻觉或无法读取的
|
||||
图片时，组件应忽略，不猜测修复，也不额外生成“仅检查”结果。
|
||||
|
||||
每个组件必须提供稳定标识、`MAJOR.MINOR.PATCH` 版本、可冻结的参数和非空适用边界。适用边界需要由组件作者说明
|
||||
它处理什么结构、依赖哪些严格前置条件、明确排除什么。组件还必须满足确定、无副作用和幂等约束；不能读取文件、
|
||||
网络、环境变量、当前时间或随机数。
|
||||
|
||||
最终复查能发现多个组件组合后仍会继续提出修改，但不能从有限输入证明一个组件对所有文本都幂等。因此，幂等性
|
||||
既由最终复查保护当前运行，也必须由组件自己的针对性测试证明其适用范围内的行为。
|
||||
|
||||
独立检查和人工建议目前没有实现。以后只有出现明确消费者和闭环时，才通过新 design 增加平行接口，不在当前
|
||||
组件结果中补可空字段或状态枚举。
|
||||
|
||||
## 5. 一个组件批次如何保证原子性
|
||||
|
||||
一个组件可以提出多个候选修改，每个候选修改又可以包含多个文本编辑。执行器先验证该组件本次提出的全部编辑，
|
||||
只有整批通过才从后向前应用；任意一条失败，整批保持原样。这里的原子边界是“当前组件本次执行的全部修改”，
|
||||
不是单独一条编辑。
|
||||
|
||||
当前冲突规则有意保守：
|
||||
|
||||
| 两项编辑的关系 | 结果 |
|
||||
| --- | --- |
|
||||
| 两个非空范围真正重叠 | 冲突 |
|
||||
| 两个非空范围只相邻 | 允许 |
|
||||
| 两次插入位于同一点 | 冲突 |
|
||||
| 两次插入位于不同点 | 允许 |
|
||||
| 插入点位于非空范围内部、起点或终点 | 冲突 |
|
||||
|
||||
如果业务动作需要替换一段文字并在边界追加内容，组件应把它表达成同一条替换，而不是依赖编辑执行顺序。
|
||||
|
||||
## 6. 流水线状态代表什么
|
||||
|
||||
流水线先对所有组件做元数据预检，避免运行到一半才发现重复标识或无效版本。之后每个组件只执行一次，后一个组件
|
||||
只能读取前一个组件产生的新快照。清洗阶段出现异常、契约错误或编辑验证错误时会立即停止，且不再进行最终复查。
|
||||
|
||||
所有组件完成后，流水线让它们针对最终快照重新提出一次修改，但这一阶段只验证、不应用：
|
||||
|
||||
| 状态 | 含义 | Markdown 字段 |
|
||||
| --- | --- | --- |
|
||||
| `success` | 清洗和最终复查均完成，所选组件不再提出修改 | 只提供成功输出 |
|
||||
| `unstable` | 清洗无错误，但最终复查仍有有效候选修改 | 只提供诊断用部分文本和残留候选 |
|
||||
| `failed` | 清洗或最终复查发生错误 | 只提供诊断用部分文本和错误 |
|
||||
|
||||
最终复查是只读阶段，因此某个组件失败后仍会继续复查其余组件。错误与其他组件的有效残留修改可以同时保留，最终
|
||||
状态以 `failed` 为准。流水线不会因为 `unstable` 自动开始第二轮。
|
||||
|
||||
`success` 只表示本次选中的自动清洗组件已经稳定，不表示文档没有截断、幻觉、表格损坏、图片断链或其他未实现
|
||||
规则能够发现的问题。
|
||||
|
||||
## 7. 审计记录能回答什么
|
||||
|
||||
每条实际执行的文本编辑都会生成一条改动记录，说明：
|
||||
|
||||
- 是哪个组件、哪个版本和流水线位置执行的；
|
||||
- 属于哪个候选修改，以及在该候选修改中的编辑序号；
|
||||
- 组件给出的修改理由；
|
||||
- 修改前范围、原文和替换内容；
|
||||
- 当前组件批次修改前后的快照哈希。
|
||||
|
||||
同一候选修改中的多条记录共享候选引用，同一组件批次中的所有记录共享批次前后哈希。记录只描述已经发生的修改；
|
||||
验证失败或最终复查中没有执行的候选不会冒充实际改动。
|
||||
|
||||
这些内容在核心中只存在于内存返回值中。核心外已经有一个获批的本地实验 reporter，可以校验修改链并把审计、
|
||||
成功 Markdown 和 diff 保存到私有产物目录；机制见
|
||||
[`local-experiment-artifacts.md`](local-experiment-artifacts.md)。这没有改变核心接口，也不允许把失败结果中的部分文本
|
||||
写成正式输出或写回原文件。
|
||||
|
||||
## 8. 当前验证和剩余边界
|
||||
|
||||
核心测试继续使用短小的假组件，不包含或复制真实文档。它们覆盖空文本、中文和组合 Unicode、插入/删除/替换、
|
||||
范围冲突、过期哈希、批次原子性、组件连锁影响、错误阶段、审计关联和成功结果再次运行等行为。首个真实组件另用
|
||||
合成样例测试，并在本地真实材料上只读复核；机制与结果见
|
||||
[`arxiv-submission-stamp.md`](arxiv-submission-stamp.md)。
|
||||
|
||||
实际可用的安装与验收命令、最近一次验证日期和结果只在根目录
|
||||
[`README.md`](../../README.md#当前可用检查) 维护。
|
||||
|
||||
当前仍然没有：
|
||||
|
||||
- 除严格删除 arXiv 提交边栏戳外的其他论文、GovDoc 或 HTML 表格清洗组件；
|
||||
- 独立文档检查、人工建议或审核流程；
|
||||
- Markdown parser、AST 或共享业务中间表示；
|
||||
- 通用文件输入、公共 CLI、通用批处理、项目 profile 格式和生产集成；
|
||||
- 审计结果的长期存储、自动清理或脱敏输出协议。
|
||||
|
||||
当前只有一个固定数据和组件组合的本地实验脚本，不构成上述公共能力。这些边界中的任何一项要进入实现，都需要先用
|
||||
新的 design 明确语义、代价和验收方式。
|
||||
@@ -0,0 +1,97 @@
|
||||
# 函数式修改核心如何工作
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
项目规则如果直接返回一整篇新 Markdown，库无法确认它改了哪里，也无法在文本已经变化时阻止旧位置继续执行。
|
||||
`mdpolish` 把三个职责分开：项目规则提出修改，库验证并在内存中执行，调用项目决定是否写回文件。
|
||||
|
||||
当前机制来自已批准的
|
||||
[`0008-generic-functional-library-boundary.md`](../design/0008-generic-functional-library-boundary.md)。公共类名、字段和
|
||||
函数签名以 [`src/mdpolish/`](../../src/mdpolish/) 和测试为准，本文只解释不变量与边界。
|
||||
|
||||
## 2. 当前数据流
|
||||
|
||||
```text
|
||||
项目选择 Modifier、参数和顺序
|
||||
│
|
||||
▼
|
||||
当前 Markdown 快照
|
||||
│
|
||||
▼
|
||||
Modifier 提出精确修改 ──► 整批验证 ──► 内存中整批应用 ──► 新快照
|
||||
│ │
|
||||
└────────── 按项目给定顺序重复 ◄──────────┘
|
||||
│
|
||||
▼
|
||||
最终只复查，不再应用
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
success unstable failed
|
||||
```
|
||||
|
||||
| 层次 | 负责 | 不负责 |
|
||||
| --- | --- | --- |
|
||||
| 不可变数据模型 | 保存快照、范围、候选修改、实际改动、错误和结果 | 项目业务判断 |
|
||||
| `Modifier` | 冻结身份、版本、参数、适用边界和提议函数 | 直接改变字符串或文件 |
|
||||
| 修改执行器 | 校验并原子应用一个修改器批次 | 判断规则是否符合某个项目 |
|
||||
| `Pipeline` | 按显式顺序运行修改器并做最终稳定性复查 | 自动选规则、文件读写和默认组合 |
|
||||
|
||||
`mdpolish` 不导入使用项目。项目可以使用正则工厂、通用内置修改器，也可以用普通函数建立自己的 `Modifier`。
|
||||
|
||||
## 3. 修改器拥有什么权限
|
||||
|
||||
修改函数接收不可变 `DocumentSnapshot`，返回一个 `ProposedChange` 元组。每项候选修改说明原因，并包含一条或多条
|
||||
精确 `TextEdit`。修改器拥有规则判断权，可以提议插入、删除或替换，但不能通过公共契约原地改变快照，也不能用
|
||||
整篇新文本绕过执行器。
|
||||
|
||||
`Modifier` 还保存稳定 ID、语义版本、冻结后的参数和适用边界。普通函数无需继承基类。相同输入、身份、版本和参数
|
||||
应产生相同顺序的候选修改；修改函数不得读取文件、网络、环境变量、当前时间或随机数。
|
||||
|
||||
Python 不能沙箱隔离任意调用方函数。外部函数若私下写文件，属于绕过库契约的副作用，不在 `mdpolish` 的验证、
|
||||
审计和回滚保证内。
|
||||
|
||||
## 4. 为什么修改必须绑定快照
|
||||
|
||||
每个快照都带有完整 Markdown 的 SHA-256。候选修改及其中每条编辑必须绑定这个哈希，并提供半开字符串范围、该范围
|
||||
应有的原文和替换文本。执行器再次确认：
|
||||
|
||||
1. 哈希对应当前快照；
|
||||
2. 范围没有越界；
|
||||
3. 当前位置与预期原文完全一致；
|
||||
4. 编辑不是无变化操作；
|
||||
5. 当前修改器批次没有重复或冲突范围。
|
||||
|
||||
任意检查失败，当前修改器的整批候选都不执行。范围使用 Python 字符串索引，不是 UTF-8 字节位置；核心不会隐式
|
||||
改变换行、Unicode 形式或末尾换行。
|
||||
|
||||
执行器从文本后方向前应用编辑，避免前面的修改使后面的下标失效；审计记录仍按原文位置排列。
|
||||
|
||||
## 5. `Pipeline` 的状态
|
||||
|
||||
`Pipeline` 先预检全部修改器及重复 ID，再按调用方顺序各运行一次。后一个修改器读取前一个修改器生成的新快照。
|
||||
全部执行完成后，每个修改器对最终快照再提议一次，但复查阶段只验证，不应用，也不会自动开始第二轮。
|
||||
|
||||
| 状态 | 含义 | 文本字段 |
|
||||
| --- | --- | --- |
|
||||
| `success` | 运行和复查均完成，所选修改器不再提出修改 | `output_markdown` |
|
||||
| `unstable` | 运行无错误，但最终快照仍有有效候选修改 | `partial_markdown` |
|
||||
| `failed` | 提议、契约或执行验证发生错误 | `partial_markdown` |
|
||||
|
||||
`success` 只说明调用方选择的这组修改器在这次输入上已经稳定，不说明文档不存在其他质量问题。库只返回内存结果；
|
||||
调用项目检查状态后，自己决定是否保存。
|
||||
|
||||
## 6. 当前通用能力与边界
|
||||
|
||||
除了核心，发布包只提供：
|
||||
|
||||
- `regex_replace()`：把非空正则匹配转换为精确编辑；
|
||||
- `mapped_line_join()`：按调用方映射合并跨行片段，库不附带词表；
|
||||
- `html_table_entity_unescape()`：在严格表格单元格文本中解除一层受支持的实体转义；
|
||||
- `html_table_layout()`：把严格单行 HTML 表格展开为每行一个表格行。
|
||||
|
||||
HTML 能力使用失败关闭的词法子集，不是完整 HTML parser，也不识别 Markdown 围栏。正则工厂只保证定位和执行契约，
|
||||
不保证调用方正则的业务语义正确。
|
||||
|
||||
当前没有默认流水线、文件适配器、CLI、profile、配置加载、批处理、artifact、评审器或项目规则集。安装或导入库不会
|
||||
自动修改任何文本。实际安装、示例和当前检查命令只以根目录 [`README.md`](../../README.md) 为准。
|
||||
@@ -1,195 +0,0 @@
|
||||
# 本地清洗实验如何保存 Markdown、审计和 diff
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
内存流水线可以安全地产生 `TransformResult`，但进程结束后，评审者仍需要打开清洗后的完整 Markdown、查看总 diff，
|
||||
并追溯每条修改属于哪个组件、为什么修改、修改前后是什么。
|
||||
|
||||
当前本地实验层把这些结果保存到独立目录，同时继续保持三个边界：
|
||||
|
||||
- 组件和 `Pipeline` 仍然不读写文件；
|
||||
- 输入文件永远不被覆盖；
|
||||
- 只有 `success` 文档才产生正式的清洗后 Markdown。
|
||||
|
||||
已经实现的范围来自已批准的
|
||||
[`0005-local-experiment-runner-and-artifacts.md`](../design/0005-local-experiment-runner-and-artifacts.md) 和
|
||||
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。
|
||||
精确字段、校验和函数签名以 `src/mdpolish/` 中的代码与测试为准。
|
||||
|
||||
## 2. 保存与评审怎样解耦
|
||||
|
||||
```text
|
||||
experiment.py
|
||||
├── pipeline.py 只负责内存清洗
|
||||
├── reporting.py 只负责 JSON、行列和 unified diff
|
||||
└── artifact_store.py 只负责日期目录、权限和原子发布
|
||||
│
|
||||
▼
|
||||
已发布运行目录
|
||||
│
|
||||
▼
|
||||
reviewer/ 只通过发布后的文件做本地只读评审
|
||||
```
|
||||
|
||||
- `experiment.py` 严格读取调用方显式列出的 UTF-8 Markdown，逐份调用同一个 `Pipeline`；
|
||||
- `reporting.py` 重放并校验 `Change` 的快照链，再生成机器可读审计和人可读 diff；
|
||||
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
||||
- 同仓库 `reviewer/server/` 只共用无文件 I/O 的 Python 快照重放模块，不导入组件或流水线；它读取 manifest、result、
|
||||
成功输出和本机定位文件。
|
||||
|
||||
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
||||
ClinDB 的 5 份论文、历史 arXiv 单组件组合和当前 first-batch 组合只存在于两个仓库内实验脚本，通用模块没有硬编码
|
||||
论文名或业务组件。
|
||||
|
||||
## 3. 输入怎样保持原样
|
||||
|
||||
实验层先完成整批预检：
|
||||
|
||||
1. 验证日期、运行 ID、文档 ID 和来源标签；
|
||||
2. 确认所有路径存在、是普通文件且没有重复；
|
||||
3. 以二进制读取全部输入；
|
||||
4. 使用严格 UTF-8 解码；
|
||||
5. 比较原始字节 SHA-256 与核心 Markdown SHA-256。
|
||||
|
||||
读取过程不剔除 BOM，不规范化 Unicode，不转换 `\n`、`\r\n` 或文件末尾换行。全部文档运行结束后，
|
||||
实验层再次读取每份输入并比较原始字节和哈希；任何变化都会阻止产物发布。
|
||||
|
||||
输入缺失、不是普通文件、不是有效 UTF-8 或清单冲突属于整批预检失败。这时不调用流水线，也不创建最终运行目录。
|
||||
|
||||
## 4. 状态怎样决定产物
|
||||
|
||||
每份文档独立运行，某一份发生核心错误不会阻止其他已经预检的文档继续产生结果。
|
||||
|
||||
| 文档状态 | `result.json` | `cleaned.md` | `changes.diff` |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | 有 | 有 | 有；零修改时为空文件 |
|
||||
| `failed` | 有 | 无 | 无 |
|
||||
| `unstable` | 有 | 无 | 无 |
|
||||
|
||||
`failed` 和 `unstable` 的审计仍保留已经实际发生的 `Change`、错误或残留候选，但不持久化
|
||||
`partial_markdown`，避免半成品看起来像正式结果。
|
||||
|
||||
整批状态按 `failed`、`unstable`、`success` 的优先级汇总。即使其他文档有成功产物，只要一份失败，
|
||||
`manifest.json` 就会把整批标为 `failed`。
|
||||
|
||||
## 5. 产物怎样组织
|
||||
|
||||
```text
|
||||
artifacts/
|
||||
└── <YYYY-MM-DD>/
|
||||
└── runs/
|
||||
└── <run_id>/
|
||||
├── manifest.json
|
||||
├── review-locator.json
|
||||
└── documents/
|
||||
└── <document_id>/
|
||||
├── result.json
|
||||
├── cleaned.md
|
||||
└── changes.diff
|
||||
```
|
||||
|
||||
日期取实验启动时本机时区中的日历日期。运行 ID 由调用方显式提供；同一日期下已经存在同名目录时拒绝覆盖。
|
||||
|
||||
`manifest.json` 是整次运行的索引，记录：
|
||||
|
||||
- 开始、完成和到期时间；
|
||||
- 本机日期及 UTC 偏移；
|
||||
- mdpolish、Python、平台和 Git 状态；
|
||||
- 实际组件顺序、版本和参数；
|
||||
- 每份输入的来源标签、前后哈希、状态、修改数量和产物相对路径；
|
||||
- 整批成功、失败、不稳定和修改数量。
|
||||
|
||||
每份 `result.json` 保存实际修改、错误与残留候选。每条修改包含组件、候选引用、理由、Python 字符范围、
|
||||
1-based 行列、`before`、`after` 和批次前后哈希。完整成功文本只存在于 `cleaned.md`。
|
||||
|
||||
`changes.diff` 是原始输入到最终成功输出的 unified diff，只用于人工查看。它不包含绝对路径或时间戳，
|
||||
也不是修改重放的权威；机器审计仍以 `result.json` 为准。
|
||||
|
||||
`review-locator.json` 只记录本次运行目录、每份输入的绝对解析路径和输入哈希，供本地评审器重新找到完整原文。它不复制
|
||||
原文，不替代 manifest 或 result，也不作为可移植运行身份。绝对路径可能泄露本机目录结构，因此该文件同样是本地敏感数据。
|
||||
|
||||
## 6. 为什么 reporter 要重放修改
|
||||
|
||||
第二个组件看到的是第一个组件修改后的快照，因此后续 `Change.span` 不一定对应最初输入。为了生成准确行列，
|
||||
reporter 从输入开始，按组件批次重放修改：
|
||||
|
||||
1. 当前文本哈希必须等于该批次 `before_sha256`；
|
||||
2. 每条范围内的原文必须等于 `before`；
|
||||
3. 同一批次按核心相同的从后向前顺序应用；
|
||||
4. 结果哈希必须等于 `after_sha256`；
|
||||
5. 全部批次完成后必须等于 `TransformResult.current_sha256`。
|
||||
|
||||
任何一步不一致都说明内存结果、reporter 或调用方式违反契约，整次运行不会发布最终目录。行列只是方便人查看的
|
||||
派生信息，修改权威仍是快照绑定的 Python 字符范围。
|
||||
|
||||
## 7. 文件怎样安全发布
|
||||
|
||||
artifact store 先在同一日期的 `runs/` 下建立本次专用临时目录。所有文件写入后都会重新读取校验，
|
||||
成功 Markdown 还要再次核对输出 SHA-256，manifest 的身份、状态、计数和路径也必须与各文档审计及实际文件一致。
|
||||
只有全部文件、清单和权限都通过，临时目录才会在 `runs/` 目录协作锁内重新检查目标，并原子重命名为最终运行 ID。
|
||||
|
||||
当前 Linux 本地实现使用：
|
||||
|
||||
- 目录权限 `0700`；
|
||||
- 文件权限 `0600`；
|
||||
- 已存在的目标目录拒绝覆盖；
|
||||
- 写入中途失败时不发布最终目录。
|
||||
|
||||
这保证不会发布已知不完整的结果，但不承诺跨平台断电耐久性或网络文件系统语义。
|
||||
|
||||
## 8. 隐私和保留边界
|
||||
|
||||
`cleaned.md`、diff 和 JSON 审计都可能包含真实原文，因此整个 `artifacts/` 都是本地敏感数据：
|
||||
|
||||
- 受 Git 忽略；
|
||||
- 不进入 Wiki、提交、推送或外部系统；
|
||||
- 终端只显示状态、计数和目录；
|
||||
- 默认保留 30 个日历日；
|
||||
- manifest 记录 `retention_until`；
|
||||
- 第一版不自动删除，到期后仍需用户确认具体目录再清理。
|
||||
|
||||
当前只批准对 `data/md/` 中 5 份论文副本保存产物。仓库外 GovDoc 和其他真实数据没有因此获得输出授权。
|
||||
|
||||
同仓库只读页面的路径验证、组件快照重放和使用边界见
|
||||
[`local-markdown-reviewer.md`](local-markdown-reviewer.md)。
|
||||
|
||||
## 9. 已完成的真实验证
|
||||
|
||||
2026-08-22 使用 `paper.arxiv_submission_stamp` `1.0.0` 对 5 份本地论文副本完成一次保存型实验：
|
||||
|
||||
| 项目 | 结果 |
|
||||
| --- | --- |
|
||||
| 运行 ID | `clindb-arxiv-stamp-artifacts-v1` |
|
||||
| 输出位置 | `artifacts/2026-08-22/runs/clindb-arxiv-stamp-artifacts-v1/` |
|
||||
| 文档状态 | 5/5 `success` |
|
||||
| 实际修改 | `sim` 1 条、`springer` 1 条，其余 0 条 |
|
||||
| 修改位置 | `sim` 1:1、`springer` 18:1 |
|
||||
| 合法反向样例 | Springer 两处 `arXiv preprint arXiv:` 均保留 |
|
||||
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||
| 权限 | 全部运行目录 `0700`，产物文件 `0600` |
|
||||
|
||||
实际运行方法见
|
||||
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
||||
|
||||
同日又使用 `design/0006` 的 8 组件流水线完成 ClinDB 第一批保存型实验：
|
||||
|
||||
| 项目 | 结果 |
|
||||
| --- | --- |
|
||||
| 运行 ID | `clindb-first-batch-v1` |
|
||||
| 输出位置 | `artifacts/2026-08-22/runs/clindb-first-batch-v1/` |
|
||||
| 文档状态 | 5/5 `success` |
|
||||
| 实际修改 | dmp 47、ejhf 9、jama 79、sim 3、springer 17，共 155 条 |
|
||||
| 第二次运行 | 5/5 `success`，合计 0 条修改 |
|
||||
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||
| 内容反例 | JAMA Abstract 前内容不变；Springer 两条合法 arXiv 引用保留；图片引用文字不变 |
|
||||
| 权限 | 运行目录 `0700`，产物文件 `0600` |
|
||||
|
||||
当前完整运行方法见
|
||||
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。实验层的文件、
|
||||
JSON、diff 和权限契约没有因组件增多而改变。
|
||||
|
||||
2026-08-23 又产生运行 `clindb-first-batch-reviewer-v1`，用于验证新定位文件和本地页面：5/5 文档为 `success`，合计
|
||||
155 条修改，输入运行前后哈希不变。评审器 API 校验了 5 份原文、成功输出以及 8 个组件形成的 40 个阶段，所有文本哈希
|
||||
均与审计一致。实际页面启动步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||
@@ -1,109 +0,0 @@
|
||||
# 本地 Markdown 清洗评审器如何保持只读和可追踪
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
本地清洗实验已经保存最终 Markdown、逐条审计和 unified diff，但人工评审仍需要在多个文件之间切换，也看不到某个组件
|
||||
执行前后的完整文本。当前评审器把一次已发布运行变成只读页面：主视图比较原文和最终成功输出，组件时间线则比较每个
|
||||
组件实际收到的快照和它产生的新快照。
|
||||
|
||||
实现范围来自已批准的
|
||||
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。精确 API、字段和运行行为以
|
||||
[`reviewer/`](../../reviewer/) 中的代码、类型和测试为准。
|
||||
|
||||
## 2. 同仓库怎样保持解耦
|
||||
|
||||
```text
|
||||
mdpolish 本地实验层
|
||||
│
|
||||
▼
|
||||
manifest / result / cleaned / review-locator
|
||||
│
|
||||
▼
|
||||
Python 产物适配器与共享重放 ──► 本地只读 API ──► React 页面
|
||||
```
|
||||
|
||||
- `src/mdpolish/` 不导入 `reviewer/`；
|
||||
- `reviewer/server/` 只从 `mdpolish` 导入无文件 I/O 的 `_artifact_replay.py`，不导入组件、流水线或实验入口；
|
||||
- 浏览器只读取 `/api/v1/`，不解析磁盘 JSON，也不知道绝对路径；
|
||||
- Python wheel 不包含前端代码或 Node.js 依赖；
|
||||
- 产物 schema 以后改变时，差异集中在服务端版本适配器，不扩散到页面组件。
|
||||
|
||||
前端与核心位于同一 Git 仓库，方便开发和评审，但仍是独立的 Node.js package。依赖版本只在
|
||||
[`reviewer/package.json`](../../reviewer/package.json) 和锁文件维护。
|
||||
|
||||
## 3. 运行定位文件保存什么
|
||||
|
||||
新实验会在运行目录根部原子保存 `review-locator.json`。它记录：
|
||||
|
||||
- 运行 ID 和发布时的绝对运行目录；
|
||||
- `manifest.json` 的固定相对位置；
|
||||
- 每份文档的 ID、实际读取的绝对源路径和输入 SHA-256。
|
||||
|
||||
定位文件不复制原文，也不替代 manifest 或 result。评审器由用户显式指定当前运行目录；记录的旧运行目录只用于判断目录
|
||||
是否被移动。服务只根据定位文件读取对应原文，并在每次展示前重新计算哈希。源文件不存在或内容变化时，页面明确报告
|
||||
不可用，不按名称搜索替代文件。
|
||||
|
||||
绝对路径会暴露本机目录结构，所以定位文件与其他 artifact 一样使用 `0600` 权限并按本地敏感数据处理。历史运行没有该
|
||||
文件时仍可查看清单和局部审计，但不能自动展示完整原文；评审器不会回写历史目录。
|
||||
|
||||
## 4. 服务为什么只读取一次运行
|
||||
|
||||
启动时必须传入一个具体运行目录。服务不会扫描 `artifacts/`，也没有让浏览器传入任意文件路径的 API。它先校验：
|
||||
|
||||
1. manifest、result 和存在的 locator 都是支持的 schema、严格 UTF-8 JSON；
|
||||
2. 文档、组件、状态、路径和计数彼此一致；
|
||||
3. 所有 artifact 路径都留在所选运行目录；
|
||||
4. 原文和成功输出的字节哈希与审计一致；
|
||||
5. 成功文档确实同时具有 `cleaned.md` 和 `changes.diff`。
|
||||
|
||||
HTTP 只监听 `127.0.0.1` 的随机空闲端口，只接受 `GET` 和 `HEAD`。服务拒绝非本机 Host、跨域 Origin、路径穿越和
|
||||
写请求，不提供删除、移动、重新清洗或 shell 执行能力。响应禁止缓存，不开放 CORS，也不向浏览器返回绝对源路径。
|
||||
|
||||
## 5. 组件阶段怎样准确重放
|
||||
|
||||
`Change.span` 使用 Python Unicode 码点位置，而 JavaScript 编辑器使用 UTF-16 code unit。共享 Python 重放模块直接按
|
||||
原生码点范围逐组件处理，不让浏览器应用修改：
|
||||
|
||||
1. 当前完整文本哈希必须等于组件批次的 `before_sha256`；
|
||||
2. 每条范围内文本必须等于 `before`；
|
||||
3. 同一批次不得有冲突范围，并按位置从后向前应用；
|
||||
4. 应用后完整文本哈希必须等于 `after_sha256`；
|
||||
5. 全部组件结束后必须逐字等于 `cleaned.md`；
|
||||
6. 服务端另外从已验证的组件前快照派生 UTF-16 `editor_range`，只供 CodeMirror 跳转。
|
||||
|
||||
组件没有修改时，阶段前后文本和哈希相同，但该组件仍显示在时间线中。包含中文、emoji、组合字符、BOM、CRLF 和无末尾
|
||||
换行的合成测试用于保护跨语言坐标。任一重放校验失败时，页面拒绝显示组件阶段，不通过搜索或 diff 猜测位置。
|
||||
|
||||
中间快照只在服务内存中按需生成，不保存新的 Markdown 文件。`failed` 和 `unstable` 文档只显示错误、残留候选和已有的
|
||||
局部审计，不重建一份看似正式的部分输出。
|
||||
|
||||
## 6. 页面当前能看什么
|
||||
|
||||
页面当前提供：
|
||||
|
||||
- 运行状态、文档状态、哈希和实际 `Change` 数量；
|
||||
- 完整原文与最终成功 Markdown 的只读双栏源码比较；
|
||||
- 8 个组件的实际顺序、版本和每份文档修改数量；
|
||||
- 任一组件执行前后的完整文本比较；
|
||||
- 修改理由、派生行列、`before` / `after` 和同候选修改关联；
|
||||
- `failed`、`unstable`、路径失效、哈希变化和未知 schema 的独立错误状态。
|
||||
|
||||
Markdown 只作为文本交给 CodeMirror，不进入 `innerHTML`。第一版不渲染 Markdown、HTML 或图片，不加载 CDN、远程字体、
|
||||
遥测和其他外部资源，也不提供编辑、审核或回写。
|
||||
|
||||
## 7. 当前验证结果和边界
|
||||
|
||||
2026-08-24 使用 Python 3.13.11 和当前用户 nvm 中的 Node.js 24.19.0 完成：
|
||||
|
||||
- Python Ruff、mypy 和 229 项 pytest 通过；
|
||||
- reviewer ESLint、TypeScript、9 项 Vitest 和生产构建通过；
|
||||
- 新运行 `clindb-first-batch-reviewer-v1` 的 5 份论文全部 `success`，共 155 条实际修改；
|
||||
- 5 份原文运行前后哈希不变；
|
||||
- 本地 API 成功校验 5 份文档、8 个组件和 40 个组件阶段；
|
||||
- 所有原文、成功输出和阶段前后文本的 SHA-256 与运行审计一致；
|
||||
- 生产页面和全部本地构建资源可以通过只读服务读取，响应没有 CORS 并包含禁止缓存和内容类型保护头。
|
||||
|
||||
当前环境没有可用于自动视觉检查的本地浏览器，因此布局的真实浏览器视觉效果尚未验证。当前结果证明构建、服务、数据
|
||||
重放和主要 React 状态可以运行，不等于已经完成跨浏览器、极端长度、渲染预览或生产部署验证。
|
||||
|
||||
实际启动与评审步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||
@@ -1,127 +0,0 @@
|
||||
# 使用本地页面评审一次 Markdown 清洗运行
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南用于打开已经发布在 `artifacts/<YYYY-MM-DD>/runs/<run_id>/` 的本地清洗运行。完整双栏比较要求运行目录包含
|
||||
`review-locator.json`，并且原文仍位于运行时记录的位置且哈希未改变。
|
||||
|
||||
评审器只读文件，不重新运行组件、不修改原文和产物。当前不适用于 GovDoc、远程目录、多人共享或生产部署。
|
||||
|
||||
本指南于 2026-08-23 使用 Python 3.13.11、当前用户 nvm 中的 Node.js 24.19.0 和运行
|
||||
`clindb-first-batch-reviewer-v1` 实际验证。
|
||||
|
||||
## 2. 准备一次可评审运行
|
||||
|
||||
先按 [`run-local-clindb-first-batch-experiment.md`](run-local-clindb-first-batch-experiment.md) 产生一次新的运行。成功终端摘要
|
||||
会给出绝对 artifact 路径，例如：
|
||||
|
||||
```text
|
||||
artifacts=/home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||
```
|
||||
|
||||
确认该目录内存在：
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
review-locator.json
|
||||
documents/
|
||||
```
|
||||
|
||||
不要编辑定位文件，也不要向旧运行目录手工补写它。历史运行缺少 locator 时，使用不同运行 ID 重新实验。
|
||||
|
||||
## 3. 准备评审器
|
||||
|
||||
评审器前端的安装、检查和构建要求 Node.js 24 LTS。当前用户 nvm 已安装与 `.nvmrc` 匹配的版本。在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
cd reviewer
|
||||
nvm use
|
||||
node --version
|
||||
npm --version
|
||||
```
|
||||
|
||||
`node --version` 必须是受支持的 `v24`。本仓库不负责修改系统级 Node.js；版本不符时先在开发环境外准备正确运行时。
|
||||
|
||||
首次安装或锁文件变化后，仍在 `reviewer/` 目录执行：
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
```
|
||||
|
||||
当前有效的 Python 与 reviewer 检查命令只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后构建页面：
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
## 4. 启动一次运行
|
||||
|
||||
回到仓库根目录，用当前 Python 虚拟环境启动只读服务并传入运行目录：
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m reviewer.server \
|
||||
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||
```
|
||||
|
||||
启动成功时只打印运行 ID、文档数量和随机本机端口，不打印原文或绝对源路径：
|
||||
|
||||
```text
|
||||
mdpolish 评审器已启动：http://127.0.0.1:<port>（<run_id>，<count> 份文档）
|
||||
```
|
||||
|
||||
在本机浏览器打开该地址。评审结束后回到终端按 `Ctrl+C` 停止服务。
|
||||
|
||||
开发页面时开两个终端。第一个终端在仓库根目录把 Python API 固定到 Vite 代理使用的本机端口：
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m reviewer.server \
|
||||
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id> \
|
||||
--port 4174
|
||||
```
|
||||
|
||||
第二个终端启动只绑定 `127.0.0.1:5173` 的 Vite 页面；它只把 `/api/` 代理给上述 Python 服务，Node.js 不读取 artifact：
|
||||
|
||||
```bash
|
||||
cd reviewer
|
||||
npm run dev
|
||||
```
|
||||
|
||||
## 5. 页面怎么查看
|
||||
|
||||
1. 先确认顶部整体状态和总修改数与 `manifest.json` 一致；
|
||||
2. 在左侧选择文档，主双栏默认显示清洗前和最终成功输出；
|
||||
3. 在组件时间线选择一个组件，双栏切换为该组件执行前后；
|
||||
4. 检查组件版本和修改数，零修改应显示 `0`，而不是从时间线消失；
|
||||
5. 点击修改详情，跳到对应组件阶段的位置并核对理由、`before` 和 `after`；
|
||||
6. 对 `failed` / `unstable` 只查看错误和残留候选，不寻找不存在的正式输出。
|
||||
|
||||
页面中的总修改数是实际 `Change` 条数，不是 diff hunk 数、字符数或问题数量。
|
||||
|
||||
## 6. 常见错误
|
||||
|
||||
### 原文路径失效或哈希改变
|
||||
|
||||
评审器不会搜索同名文件。确认输入没有被移动或修改；如果需要在新位置运行，使用新的运行 ID 重新执行实验。不要改 locator
|
||||
绕过哈希检查。
|
||||
|
||||
### 历史运行没有 `review-locator.json`
|
||||
|
||||
历史产物仍可在页面查看清单和已有审计，也可人工查看 manifest、result 和 diff，但第一版页面不能自动找到完整原文或
|
||||
组件阶段。不要回写历史目录；需要完整双栏时重新运行一次即可。
|
||||
|
||||
### 不支持 schema
|
||||
|
||||
评审器只支持当前文档列出的 schema 版本。不要删除或伪造 `schema_version`；应升级评审器适配器或使用与产物匹配的代码。
|
||||
|
||||
### 没有 `cleaned.md`
|
||||
|
||||
对应文档状态是 `failed` 或 `unstable` 时这是正常边界。页面不会从 Change 重建并冒充正式结果。
|
||||
|
||||
### 服务拒绝 Host、Origin 或写请求
|
||||
|
||||
评审器只接受本机同源的只读请求。不要通过反向代理、远程端口转发或网页跨域调用它；这些用法没有批准。
|
||||
|
||||
## 7. 数据边界
|
||||
|
||||
页面会在本机内存中读取完整原文和成功输出。不要截图、复制或通过浏览器扩展分享真实内容。运行目录继续受 Git 忽略并按
|
||||
manifest 的 `retention_until` 管理；页面不会自动删除到期产物。
|
||||
@@ -1,129 +0,0 @@
|
||||
# 运行本地 ClinDB arXiv 清洗实验
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南只运行仓库内已经批准的本地实验脚本：
|
||||
|
||||
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||
- 流水线：只包含 `paper.arxiv_submission_stamp` `1.0.0`；
|
||||
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||
- 输入只读，不覆盖原文件；
|
||||
- 不处理 `/home/lihaoze/gov_test_data`。
|
||||
|
||||
本指南于 2026-08-22 在 Python 3.13.11 环境实际验证。
|
||||
|
||||
## 2. 前置条件
|
||||
|
||||
在仓库根目录执行，并确认隔离环境和基础检查可用：
|
||||
|
||||
```bash
|
||||
.venv/bin/python --version
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/pytest
|
||||
```
|
||||
|
||||
确认 5 份本地输入存在：
|
||||
|
||||
```bash
|
||||
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||
```
|
||||
|
||||
预期看到：
|
||||
|
||||
```text
|
||||
dmp.md
|
||||
ejhf.md
|
||||
jama.md
|
||||
sim.md
|
||||
springer.md
|
||||
```
|
||||
|
||||
## 3. 运行实验
|
||||
|
||||
为本次实验人工选择一个小写运行 ID。同一天已经使用过的 ID 不能覆盖；需要重跑时换一个新 ID。
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/run_clindb_arxiv_experiment.py \
|
||||
--run-id clindb-arxiv-stamp-review
|
||||
```
|
||||
|
||||
成功时终端只显示运行 ID、状态、文档数、修改数和产物目录，例如：
|
||||
|
||||
```text
|
||||
run_id=clindb-arxiv-stamp-review
|
||||
status=success
|
||||
documents=5
|
||||
changes=2
|
||||
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-arxiv-stamp-review
|
||||
```
|
||||
|
||||
终端不会打印论文原文或 diff。
|
||||
|
||||
## 4. 查看结果
|
||||
|
||||
进入终端输出的运行目录。目录结构为：
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
documents/
|
||||
├── dmp/
|
||||
│ ├── result.json
|
||||
│ ├── cleaned.md
|
||||
│ └── changes.diff
|
||||
├── ejhf/
|
||||
├── jama/
|
||||
├── sim/
|
||||
└── springer/
|
||||
```
|
||||
|
||||
先看 `manifest.json` 的整批状态和汇总，再查看各文档：
|
||||
|
||||
- `cleaned.md`：成功清洗后的完整 Markdown；
|
||||
- `changes.diff`：输入到成功输出的人工对比；
|
||||
- `result.json`：组件、理由、位置、`before`、`after` 和哈希等机器审计。
|
||||
|
||||
`dmp`、`ejhf` 和 `jama` 当前应为零修改，diff 是空文件；`sim` 和 `springer` 当前各有一条删除。
|
||||
|
||||
## 5. 判断成功
|
||||
|
||||
本轮验收口径是：
|
||||
|
||||
- manifest 整体状态为 `success`；
|
||||
- 5 份文档全部为 `success`；
|
||||
- 合计 2 条修改；
|
||||
- `sim` 修改位置为 1:1；
|
||||
- `springer` 修改位置为 18:1；
|
||||
- Springer 两条合法 `arXiv preprint arXiv:` 参考文献保留；
|
||||
- 每份 `cleaned.md` 的 SHA-256 等于对应 `result.json.current_sha256`；
|
||||
- 输入文件运行前后不变。
|
||||
|
||||
只看到运行目录存在不等于成功，必须先检查 manifest 和文档状态。
|
||||
|
||||
## 6. 常见失败
|
||||
|
||||
### 运行目录已经存在
|
||||
|
||||
脚本拒绝覆盖同一日期下的同名运行目录。选择新的 `--run-id`，不要删除或覆盖旧目录来绕过检查。
|
||||
|
||||
### 输入缺失或不是 UTF-8
|
||||
|
||||
整批预检会失败，不运行任何组件，也不发布最终目录。先确认 `data/md/` 中 5 份文件存在且未被修改。
|
||||
|
||||
### 状态为 `failed` 或 `unstable`
|
||||
|
||||
对应文档只会生成 `result.json`，不会生成 `cleaned.md` 或 diff。查看错误或残留候选，不要把其他文档的部分成功
|
||||
当成整批成功。
|
||||
|
||||
### Snap 版本的 `jq` 报权限错误
|
||||
|
||||
产物目录权限是 `0700`。某些 Snap 沙箱工具不能进入私有目录，即使当前用户拥有权限。可直接用编辑器查看 JSON，
|
||||
或使用当前虚拟环境中的 Python 读取；不要为了兼容受限工具放宽产物权限。
|
||||
|
||||
## 7. 数据边界
|
||||
|
||||
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||
其他仓库、云存储或外部系统。
|
||||
|
||||
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。第一版不会自动删除；到期后如需清理，必须先确认
|
||||
具体运行目录。
|
||||
@@ -1,150 +0,0 @@
|
||||
# 运行本地 ClinDB 第一批完整清洗实验
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南只运行仓库内已经批准的 first-batch 实验脚本：
|
||||
|
||||
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||
- 流水线：`design/0006` 固定的 8 个组件和顺序；
|
||||
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||
- 输入只读，不覆盖原文件；
|
||||
- 不读取或复制图片，不处理 `/home/lihaoze/gov_test_data`。
|
||||
|
||||
本指南于 2026-08-23 在 Python 3.13.11 环境实际验证。
|
||||
|
||||
## 2. 前置检查
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
.venv/bin/python --version
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/pytest
|
||||
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||
```
|
||||
|
||||
当前检查结果只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后再确认 5 份输入存在：
|
||||
|
||||
```bash
|
||||
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||
```
|
||||
|
||||
必须看到 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`。不要把真实论文复制进测试 fixture。
|
||||
|
||||
## 3. 运行实验
|
||||
|
||||
人工选择一个当天未使用的安全运行 ID：
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/run_clindb_first_batch_experiment.py \
|
||||
--run-id clindb-first-batch-review
|
||||
```
|
||||
|
||||
成功时终端只显示运行身份和汇总，不打印原文：
|
||||
|
||||
```text
|
||||
run_id=clindb-first-batch-review
|
||||
status=success
|
||||
documents=5
|
||||
changes=155
|
||||
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-first-batch-review
|
||||
```
|
||||
|
||||
同一天同名目录已存在时脚本会拒绝覆盖。需要重跑时使用新 ID，不要删除旧目录来绕过检查。
|
||||
|
||||
## 4. 先看哪些结果
|
||||
|
||||
先确认运行目录根部同时存在 `manifest.json` 和 `review-locator.json`。定位文件只供本机评审器寻找原文，包含绝对路径，
|
||||
不得提交或分享。然后打开 `manifest.json`，确认：
|
||||
|
||||
- `run.status` 是 `success`；
|
||||
- `summary.document_count` 和 `summary.success_count` 都是 5；
|
||||
- `summary.failed_count`、`summary.unstable_count` 都是 0；
|
||||
- `summary.change_count` 是 155；
|
||||
- `pipeline.components` 的顺序与 `design/0006` 一致。
|
||||
|
||||
然后查看每份文档目录：
|
||||
|
||||
```text
|
||||
documents/<document_id>/
|
||||
├── result.json
|
||||
├── cleaned.md
|
||||
└── changes.diff
|
||||
```
|
||||
|
||||
- `changes.diff` 用于人工查看输入到最终输出的总变化；
|
||||
- `result.json` 用于按组件、理由、位置和哈希追踪每条修改；
|
||||
- `cleaned.md` 是成功输出全文。
|
||||
|
||||
当前 5 份输入的预期计数是：
|
||||
|
||||
| 文档 | `Change` 数 |
|
||||
| --- | ---: |
|
||||
| dmp | 47 |
|
||||
| ejhf | 9 |
|
||||
| jama | 79 |
|
||||
| sim | 3 |
|
||||
| springer | 17 |
|
||||
| **合计** | **155** |
|
||||
|
||||
按组件应为：Word 批注 2、手稿行号 75、arXiv 戳 2、重复页眉 2、映射断词 6、HTML 实体 31、
|
||||
HTML 表格布局 9、参考文献空行 28。
|
||||
|
||||
## 5. 人工复核重点
|
||||
|
||||
除了逐份查看 diff，至少确认：
|
||||
|
||||
- JAMA 的 Abstract 前作者和单位编号仍在，只删除 Abstract 后的 75 个手稿行号；
|
||||
- Springer 两条 `arXiv preprint arXiv:` 合法参考文献仍在；
|
||||
- Springer 正文中的编号方法列表没有被参考文献规则整理；
|
||||
- dmp 的重复页眉删除后，正文句子接回，参考文献第 18、19 条之间仍有一个空行；
|
||||
- 9 张表仍是 HTML，属性和单元格内容未被布局组件改写；
|
||||
- 双重实体变成单层 `<`、`>` 或 `&`，没有直接生成标签边界；
|
||||
- 图片引用文字保持不变。
|
||||
|
||||
清洗目录没有复制图片资产，所以直接打开 `cleaned.md` 时图片仍可能无法显示。这不表示图片引用被清洗组件删除；
|
||||
资产打包和路径改写需要单独设计。
|
||||
|
||||
## 6. 验证幂等和输入不变
|
||||
|
||||
流水线会在每份文档结束时做最终稳定性复查。需要额外复核整个保存结果时，可以把 `cleaned.md` 作为内存输入再次运行
|
||||
同一 `build_pipeline()`；5 份都应为 `success` 且合计零 `Change`。
|
||||
|
||||
实验层已经在发布前后复读输入并比较字节哈希。需要人工记录运行前后的摘要时，可在运行前后分别执行：
|
||||
|
||||
```bash
|
||||
sha256sum data/md/*.md
|
||||
```
|
||||
|
||||
两次输出必须逐项一致。每个 `cleaned.md` 的 SHA-256 还必须等于对应 `result.json.current_sha256`。
|
||||
|
||||
## 7. 常见失败
|
||||
|
||||
### 状态不是 `success`
|
||||
|
||||
查看对应 `result.json` 的 `errors` 或 `residual_proposals`。`failed` / `unstable` 文档不会有正式 `cleaned.md`，
|
||||
不能把其他文档的部分成功当成整批成功。
|
||||
|
||||
### 修改数不是 155
|
||||
|
||||
先按组件和文档分组定位差异。输入变化、组件参数变化或识别边界变化都必须回到 design/reference 核对；不要放宽断言、
|
||||
补跑第二轮或手工改产物。
|
||||
|
||||
### 图片不显示
|
||||
|
||||
当前运行只保存 Markdown、审计和 diff，不复制图片。不要为了显示图片而修改输入路径或把真实资产强制加入 Git。
|
||||
|
||||
### 私有目录无法被 Snap 工具读取
|
||||
|
||||
运行目录权限是 `0700`，文件是 `0600`。使用普通编辑器或当前虚拟环境中的 Python 读取，不要放宽权限。
|
||||
|
||||
## 8. 数据边界
|
||||
|
||||
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||
其他仓库、云存储或外部系统。
|
||||
|
||||
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。当前不自动删除；到期后如需清理，必须先确认具体运行目录。
|
||||
|
||||
需要在只读页面中查看完整前后文和各组件阶段时，继续按
|
||||
[`review-local-cleaning-run.md`](review-local-cleaning-run.md) 操作。
|
||||
@@ -1,79 +0,0 @@
|
||||
# ClinDB-ReviewBench 清洗目标（第一版）
|
||||
|
||||
> 性质：reference——本项目清洗范围的权威查询事实。
|
||||
> 权威关系：本文只定义 ClinDB-ReviewBench 第一批自动清洗"洗什么、不洗什么"；清洗语义的方案比较与批准记录
|
||||
> 属于 `research-wiki/design/`，实现后的运行方式属于 `explanation/` 与 `guides/`。未来只读检查不属于当前批次。
|
||||
> 依据：`research-wiki/scratch/data-5papers-cleaning-audit-2026-08-21.md`（问题编号 A–H 沿用该审计）。
|
||||
|
||||
## 1. 项目定位
|
||||
|
||||
ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI 命名的论文 Markdown
|
||||
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
||||
是它的首批输入，未来会继续扩充同源转换产物。
|
||||
|
||||
本仓库（mdpolish）为该项目的数据提供清洗能力。当前由仓库内 first-batch 实验脚本显式组合论文规则，
|
||||
尚未建立 Profile 对象或配置格式；论文专属规则仍不写进通用核心。
|
||||
|
||||
## 2. 第一版清洗目标
|
||||
|
||||
第一版只做"全自动、规则确定、可安全执行"的问题（审计第一档中的 8 类自动修改）。
|
||||
判定标准是三条同时满足：模式可用确定规则描述；不依赖对正文语义的理解；改错可以从 diff 直接看出。
|
||||
|
||||
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
||||
|---|---|---|---|
|
||||
| 1 | HTML 实体双重转义（D2） | 只在严格 HTML 表格单元格文本中把 `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，解除源码的一层转义；幂等 | dmp 23 处、ejhf 8 处，共 31 处 |
|
||||
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
||||
| 3 | Word 审阅批注（B2） | 删除严格单行 `Commented [xx]: ...` 及其后第一个空行；不猜测多行批注正文 | jama 2 处 |
|
||||
| 4 | 手稿行号（B1） | 只处理唯一 `## Abstract` 后至少 20 个、严格递增且含至少 2 个标题证据的完整序列；剥离普通行和标题中的数字前缀 | jama 75 处，其中普通行 69、标题 6；Abstract 前 59 个作者单位编号必须保留 |
|
||||
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
||||
| 6 | 单行 HTML 表格展开（D1） | 严格完整的单行表格保留 HTML、属性和单元格内容，只按 `<tr>` 换行缩进；当前不转 GFM | 全部 9 个表：dmp 7、ejhf 1、springer 1；每张都有非 `1` 的 `colspan` |
|
||||
| 7 | 跨页断词（E2） | 只按项目批准的左右片段和结果词映射合并相邻行或只隔一个空行的片段；不使用英文词表猜测 | dmp 2 处、jama 2 处、sim 2 处，共 6 处 |
|
||||
| 8 | 参考文献分隔统一（G3） | 只在准确 References 章节内，对从 1 开始连续递增的编号条目统一一个空行 | dmp 13 处、springer 15 处，共 28 处 |
|
||||
|
||||
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
||||
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
||||
|
||||
## 3. 明确不洗（第一版非目标）
|
||||
|
||||
以下问题已确认存在但**不在**第一版范围内，避免实施时范围蔓延：
|
||||
|
||||
- **内容级缺失（A1–A4）**：截断、表格整体缺失、图转表格、句子丢失。清洗无法恢复内容，
|
||||
处置方式（重新转换 / 标记 / 人工补录）由项目负责人决定，不由清洗管线代劳。
|
||||
- **修订语义（B3）**："defined identified by as" 等修订残留词对，哪个词是定稿词需对照定稿版判断。
|
||||
- **占位符与坏日期（B4/B5）**：等待定稿填充，清洗只可检测。
|
||||
- **乱码修复（C3/E1）**：泰文标题 `## ่วง`、`ينDS Hospital`，需要人工对照 PDF 替换。
|
||||
- **表格结构修复（D3–D5）**：空单元格、错误合并行、OCR 表头错字，需人工对照原文。
|
||||
- **空行断句合并（E3）**、**层级重建（C1）**、**标题合并（C2）**、**孤行公式编号（F1）**、
|
||||
**公式空格与纠错（F2–F4）**、**上标风格统一（G1）**、**数字格式（G2）**：属于第二档
|
||||
"自动检测+人工确认"，等第一档验证后再立项。
|
||||
- **封面页（H2）**：ejhf L1–11 的仓库封面区第一版不删——它是整块连续的正文区，删除逻辑
|
||||
与页眉类噪声不同，归入后续批次。
|
||||
- **图片断链校验（H3）**：全部 8 处图片引用是否存在属于未来只读检查；当前核心不读取图片资产、不输出断链
|
||||
报告，也不移动、复制或改写图片。需要该结果时先新增平行 Inspector 设计。
|
||||
- **一切内容改写**：原文写作瑕疵、欧式千分位、拼写（含 "Conounder"）不属于转换噪声，永不由清洗工具修改。
|
||||
|
||||
## 4. 输入输出边界
|
||||
|
||||
- 当前本地输入是 `data/md/` 中按论文缩写命名的 5 份 Markdown 副本，原文件只读；
|
||||
- 核心仍只接收内存 Markdown 字符串；本地实验层按 `design/0005` 将成功输出、JSON 审计和 diff 保存到
|
||||
`artifacts/<YYYY-MM-DD>/runs/<run_id>/`，不回写、不覆盖输入；
|
||||
- 当前批次不读取或复制图片资产；未来 Inspector、资产打包和路径改写分别设计；
|
||||
- 每处实际修改记录组件、理由、原文、改后内容和批次哈希，完整落盘字段以代码和 `design/0005` 为准。
|
||||
|
||||
## 5. 验收口径（第一版）
|
||||
|
||||
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，共产生 155 条 `Change`；
|
||||
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
||||
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
||||
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
||||
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
||||
等数字开头/含数字短语不受影响。
|
||||
|
||||
2026-08-22 的保存型验收中，5 份文档全部为 `success`，各组件计数为：批注 2、行号 75、arXiv 2、页眉 2、
|
||||
断词 6、HTML 实体 31、表格布局 9、参考文献空行 28。第二次运行 5 份合计 0 条修改，输入运行前后不变。
|
||||
|
||||
## 6. 与审计报告的编号对应
|
||||
|
||||
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
||||
D2、H1、B2、B1、C4、D1、E2、G3。H3 保留为未来只读检查候选。该审计是 scratch 材料，本文引用其编号
|
||||
仅为便于追溯，权威以本文为准。
|
||||
@@ -1,276 +0,0 @@
|
||||
# 7 组对比测试文档 Markdown 清洗审计与实施规范
|
||||
|
||||
- 审计日期：2026-08-20
|
||||
- 审计目录：`/home/lihaoze/gov_test_data/compare`
|
||||
- 清洗对象：7 个测试用例中 `uploads/` 下的 45 份 Markdown
|
||||
- 背景资料：`compare/readme.md` 与各用例 `README.md`
|
||||
- 本次操作：只读审计；没有修改任何原始测试文档
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
这批 Markdown 目前不适合直接作为“语义可靠”的清洗后基准。问题并不只是在空格、换行和标题层级，而是同时存在以下几类高风险污染：
|
||||
|
||||
1. **内容级污染**：至少 32/45 份文件命中了保守的模型幻觉特征词，合计 1,703 次；典型内容包括 `The quick brown fox...`、勾股定理说明、`The image contains...`、`The steps are:` 等与投标文件无关的文本。
|
||||
2. **图像内容丢失**：45/45 份文件都有图片引用，共 8,471 个，但目录中没有任何图片资产；其中 4,012 个引用只是字面量 `data:image/...;base64...`，并不包含可解码数据。
|
||||
3. **表格结构损坏**：41 份文件含原始 HTML 表格，共 7,550 个 `<table>`；35/41 份存在 `<table>/<tr>/<td>` 数量不平衡。另有 4 份 007 用例文档使用 GFM 表格，其中 203 个表格块里至少 60 个行列数不一致。
|
||||
4. **重复和超长噪声**：出现 261,838 字符的单行点线、数万字符的重复英语句子、LaTeX 箭头/颜色命令和重复汉字。001 两份文件的非空行重复比例分别达到 81.8% 和 84.1%。
|
||||
5. **结构扁平化**：全库 29,358 个 Markdown 标题中有 28,736 个是一级标题，占 97.9%；至少 378 个标题缺少空格、只有 `#` 或存在其他明显语法问题。
|
||||
6. **字符与格式污染**：有 10 个替换字符 `�`、20 个私用区字符、11 个 NBSP、5 个零宽字符、1,474 个多余的 `\-` 转义，以及 151,616 行尾部双空格。
|
||||
|
||||
最重要的实施建议是：**不要只产出一份“清洗 Markdown”**。应同时产出：
|
||||
|
||||
- `clean_fidelity.md`：忠实版，保留有法律/业务意义的全部内容，所有修复必须能追溯到源文件。
|
||||
- `clean_compare.md`：对比版，从忠实版派生，移除页眉页脚、失效目录页码、转换器噪声和确定的重复版式块，用于文档相似度计算。
|
||||
- `audit.json`：记录每一次删除、替换、合并、回源重提取和人工确认。
|
||||
|
||||
这样可以避免为了提升对比效果而不可逆地破坏原文，也能避免把幻觉文本、通用图片占位符和重复页眉当成“相同内容”。
|
||||
|
||||
## 2. 审计范围与统计口径
|
||||
|
||||
### 2.1 纳入和排除
|
||||
|
||||
纳入：
|
||||
|
||||
- `compare/001-2/uploads/*.md`：2 份
|
||||
- `compare/002-21/uploads/*.md`：21 份
|
||||
- `compare/003-10/uploads/*.md`：10 份
|
||||
- `compare/004-2/uploads/*.md`：2 份
|
||||
- `compare/005-3/uploads/*.md`：3 份
|
||||
- `compare/006-3/uploads/*.md`：3 份
|
||||
- `compare/007-4/uploads/*.md`：4 份
|
||||
|
||||
排除：
|
||||
|
||||
- 8 份 README：它们是测试说明，不是待清洗输入。
|
||||
- `review.json`、`blocks_*.json`、`match_index.json`、`summary.json`：它们是旧输入产生的下游结果，只用于理解测试背景，不能作为清洗真值。
|
||||
- `file_*_reviewed.docx`：本次未把它们当作 Markdown 清洗对象；后续可作为辅助核对材料，但是否能作为权威源需单独确认。
|
||||
|
||||
### 2.2 基础规模
|
||||
|
||||
45 份输入合计约 36.39 MiB、388,482 个物理行。所有文件都能按 UTF-8 读取，但“能解码”不代表内容无损，文件中仍有 `�` 和私用区字符。
|
||||
|
||||
下表中的“幻觉特征”使用一组偏保守的固定模板进行计数，包括 `The quick brown fox...`、勾股定理模板、`The image contains...`、`The concept of a concept...`、`The steps are:` 等；因此它只是明确下限，不包含全部乱码和中文重复污染。
|
||||
|
||||
| 用例 | 文档数 | 大小 MiB | 行数 | HTML 表格 | GFM 表格行 | 图片引用 | 空 Base64 占位 | 幻觉特征 | 最大单行字符数 | 总体判断 |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| 001-2 | 2 | 6.41 | 48,450 | 2,596 | 0 | 109 | 0 | 0 | 4,482 | 高风险：表格未闭合、重复率极高、图片定位不可用 |
|
||||
| 002-21 | 21 | 15.23 | 158,462 | 2,593 | 592 | 2,000 | 0 | 1,530 | 28,624 | 严重：模型幻觉和重复文本最集中 |
|
||||
| 003-10 | 10 | 9.20 | 116,358 | 1,646 | 2 | 1,793 | 0 | 137 | 261,838 | 严重：存在灾难性长行、乱码、幻觉和 LaTeX 污染 |
|
||||
| 004-2 | 2 | 0.41 | 5,726 | 109 | 0 | 36 | 0 | 7 | 8,001 | 高风险：OCR 语义错误、缺图、幻觉 |
|
||||
| 005-3 | 3 | 2.67 | 31,537 | 476 | 0 | 359 | 0 | 7 | 11,599 | 高风险：重复、缺图、表格和少量幻觉 |
|
||||
| 006-3 | 3 | 1.12 | 14,701 | 130 | 0 | 162 | 0 | 22 | 9,151 | 高风险：含 OCR 文档，仍有明显幻觉和缺图 |
|
||||
| 007-4 | 4 | 1.35 | 13,248 | 0 | 2,792 | 4,012 | 4,012 | 0 | 5,916 | 严重：图片内容全部为无效占位，表格和 Word 目录损坏 |
|
||||
|
||||
## 3. 对下游“文件对比”功能的直接影响
|
||||
|
||||
这些污染会系统性扭曲相似度，不只是影响阅读体验：
|
||||
|
||||
- `The quick brown fox...` 等同一幻觉模板出现在多个本来无关的投标文件里，会制造跨文档假阳性匹配。
|
||||
- 007 中 4,012 个完全相同的空图片占位符会被当成大量相同行。
|
||||
- 001 中同一“综合单价分析表”说明被重复约 900 次；这与 README 中 001、003 的“近似匹配占 99% 以上”和超大 `review.json` 有明显关联。这里只能判断为高度可疑的影响因素，不能在没有重新跑对比的情况下断言它是唯一原因。
|
||||
- 平铺成一级标题、把表格压成一行、把页眉页脚混入正文，会改变分块边界，并使段落/句子/近似三档匹配分布失真。
|
||||
- 如果把不同手机号、身份证号、图片都统一替换成同一个 `[PHONE]`、`[ID]`、`[IMAGE]`，又会制造新的假相同内容。因此占位符必须保留“不同原值不同 token”的性质。
|
||||
|
||||
清洗后必须重新生成 `blocks_*.json`、`match_index.json`、`review.json` 和 `summary.json`。旧匹配数量不能作为清洗后的等值验收条件，应保留为“脏输入性能基线”，另建“干净输入语义基线”。`fileIndex` 和测试用例映射必须保持不变。
|
||||
|
||||
## 4. 问题清单与清洗要求
|
||||
|
||||
### 4.1 C001：固定模板型模型幻觉
|
||||
|
||||
优先级：P0，阻断语义版交付。
|
||||
|
||||
已确认的例子包括：
|
||||
|
||||
- `The quick brown fox jumps over the lazy dog.`：全库 1,071 次。
|
||||
- `The equation $x^2 + y^2 = z^2$ represents a Pythagorean triple...`：全库 62 次。
|
||||
- `The image contains a single character...`
|
||||
- `The concept of a concept is fundamental in physics...`
|
||||
- `- The steps are:`
|
||||
- `Agree to be`
|
||||
- `汽车法规和商业期刊的出版`
|
||||
|
||||
典型证据：
|
||||
|
||||
- `002-21/file_14` 第 1,164 行：一行内重复 `quick brown fox` 约 411 次。
|
||||
- `002-21/file_3` 第 3,804 行：同类重复约 428 次。
|
||||
- `002-21/file_5` 第 2,915 行：`The image contains...` 及数字说明被重复扩展。
|
||||
- `002-21/file_8` 第 5,379 行：28,624 字符的英语概念重复行。
|
||||
- `003-10/file_2` 开头即出现无关英文、勾股定理和中文乱码。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 固定模板命中后先标记其所在段、表格单元格和推定页面，不应只删除匹配到的几个单词。
|
||||
2. 有源 PDF 时，按页或按区域重新提取；没有源文件时，将该段放入 `quarantine`，不能把删除后的残缺上下文冒充完整正文。
|
||||
3. 黑名单适合做拦截器，不适合做唯一清洗器。应再检测异常语言切换、低词汇多样性、同短语高频循环和超长单行。
|
||||
4. 清洗结果中这些已知模板必须为 0；审计文件须记录被移除的原始范围和回源依据。
|
||||
|
||||
### 4.2 C002：超长重复串和退化输出
|
||||
|
||||
优先级：P0/P1，视能否回源而定。
|
||||
|
||||
典型问题：
|
||||
|
||||
- `003-10/file_2` 第 91 行长 261,838 字符，主体是目录点线重复。
|
||||
- 同文件第 8,924 行长 19,089 字符，主体是重复的 LaTeX `\rightarrow`。
|
||||
- `002-21/file_15` 第 4,898 行长 11,267 字符，主体是重复 `\textcolor{red}{\blacksquare}`。
|
||||
- `003-10/file_7` 第 24,211 行含嵌套、重复的 `\textcolor{red}`。
|
||||
- 多处出现成千上万次的 `园`、`\cdots`、`...` 或错误短语，例如“建设工程执行”。
|
||||
|
||||
检测规则建议：
|
||||
|
||||
- 普通文本单行超过 2,000 字符告警，超过 10,000 字符阻断；结构化表格在解析后按单元格重新执行该规则。
|
||||
- 同一字符连续 20 次以上、同一 2—20 字符片段连续 10 次以上告警。
|
||||
- 单行压缩率异常高、唯一 token 比例过低、相邻重复 n-gram 比例过高时进入隔离。
|
||||
- 目录点线只保留为结构化 TOC 信息，不保留数十万字符的视觉填充。
|
||||
|
||||
不允许简单按固定长度截断，因为长 HTML 表格可能包含真实内容。必须先判断是表格、目录点线、Base64、SVG 还是普通文本。
|
||||
|
||||
### 4.3 I001：图片引用全部失效
|
||||
|
||||
优先级：P0。
|
||||
|
||||
全库有 8,471 个图片引用，本地图片资产为 0：
|
||||
|
||||
- 007：4,012 个 `` 或 PNG 变体。这里的 `...` 是文本，不是被终端隐藏的真实数据，无法解码恢复。
|
||||
- 001：109 个目标形如 `page=9,bbox=[...]`，不是标准图片路径，也没有对应裁剪图。
|
||||
- 其余：4,350 个 `images/<hash>.jpg` 等相对引用，但仓库中不存在相应文件。
|
||||
- 7,073/8,471 个图片没有 alt 文本。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 从原始 PDF/DOCX 回源导出图片，使用内容哈希命名，并生成 `assets/manifest.json`。
|
||||
2. 对印章、签名、证书、身份证件、扫描表格等“有语义图片”执行 OCR/版面识别，但仍要保留原图引用，不能只留推测文本。
|
||||
3. 001 的 `page+bbox` 必须转换为结构化来源坐标，再从对应 PDF 裁剪；不能直接当 Markdown URL。
|
||||
4. 如果确实采用纯文本模式，图片位置应写成唯一、可追踪的标记，例如 `[IMAGE_MISSING:007-4:file_1:0001]`，不能用所有文件共享的 `[IMAGE]`，否则会制造假匹配。
|
||||
5. 生产级“完整清洗”验收要求 broken reference 为 0。无法回源的图片必须明确标为未解决，不能计作完整通过。
|
||||
|
||||
### 4.4 T001：HTML 表格损坏和超长单行
|
||||
|
||||
优先级：P0/P1。
|
||||
|
||||
41 份文件中共有 7,550 个 HTML 表格。静态标签计数如下：
|
||||
|
||||
- `<table>` 7,550，`</table>` 7,155。
|
||||
- `<tr>` 101,760，`</tr>` 101,019。
|
||||
- `<td>` 657,263，`</td>` 656,575。
|
||||
- 35/41 份含 HTML 表格的文件至少有一类标签不平衡；7 份连 table 层级都不平衡。
|
||||
|
||||
001 最严重：
|
||||
|
||||
- `file_0`：`table=1315/1124`。
|
||||
- `file_1`：`table=1281/1080`。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 使用容错 HTML 解析器构建 DOM，记录解析器自动补齐了哪些标签；禁止用正则直接替换所有表格标签。
|
||||
2. 对每个表格校验 `rowspan/colspan` 展开后的网格是否矩形、行列数是否与表头一致、单元格顺序是否可追溯。
|
||||
3. 简单矩形表格可转为 GFM；包含合并单元格、斜线表头、嵌套结构的表格应保留规范化 HTML，或另存为结构化 JSON。强行转 GFM 会丢失语义。
|
||||
4. 一行一个完整 HTML 表格应格式化为多行结构，避免长行拖垮分块器和 diff，但换行必须发生在 DOM 节点之间。
|
||||
5. 类似 `004-2/file_0` 第 613 行中“国家、职位、导演、客户”等明显不符合资格审查表语境的表头，应回源确认，不能只修标签。
|
||||
|
||||
### 4.5 T002：GFM 表格行列数不一致
|
||||
|
||||
优先级：P1。
|
||||
|
||||
007 四份文件共有 203 个连续 GFM 表格块、2,792 个表格行，至少 60 个表格块出现不同行拥有不同列数：
|
||||
|
||||
- `file_0`：36 个块，21 个不一致。
|
||||
- `file_1`：55 个块，20 个不一致。
|
||||
- `file_2`：42 个块，7 个不一致。
|
||||
- `file_3`：70 个块，12 个不一致，另有 2 个块缺分隔行。
|
||||
|
||||
此外，`002-21/file_7` 有一个 584 行的大型 GFM 表格，同时出现 3 列和 5 列；`002-21/file_3` 也有不一致的孤立表格块。
|
||||
|
||||
这通常是把 Word/PDF 合并单元格硬投影到 GFM 的结果。清洗器应先恢复二维网格；无法无损表达的表格应改用规范 HTML/JSON，而不是补若干空 `|` 来“通过语法检查”。
|
||||
|
||||
### 4.6 H001：标题层级扁平、标题断裂和错误标题
|
||||
|
||||
优先级：P1。
|
||||
|
||||
问题表现：
|
||||
|
||||
- 001—006 几乎把所有章节都输出为 `#`，没有文档层级。
|
||||
- 存在 `#零星维修...`、`#(正本)` 等缺少空格的标题。
|
||||
- 存在只有 `#` 的空标题。
|
||||
- 同一个标题被版面换行切成多个连续一级标题，例如项目名称被拆成 2—3 行。
|
||||
- 普通正文、图片识别结果和幻觉句子也被错误标成标题。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 每个逻辑文档保留一个主标题 H1；主要章为 H2，节为 H3，依次递进，不跳级。
|
||||
2. 使用编号模式（“第一章”“一、”“（一）”“1.”）、目录结构、相邻上下文和重复页眉信息共同推断层级。
|
||||
3. 连续短标题行在确认属于同一版面标题后合并；不能只根据行长合并。
|
||||
4. 修复 `#标题` 为 `# 标题`，删除空标题；被判为正文的行去掉标题标记。
|
||||
5. 标题文本中的项目编号、公司名、年份不得因规范化而改变。
|
||||
|
||||
### 4.7 P001：段落、硬换行和词内断裂
|
||||
|
||||
优先级：P1。
|
||||
|
||||
151,616 行以两个空格结尾，主要来自 002—006 的转换器。这会把几乎每个物理行强制为 Markdown `<br>`，使原本同一段的文字被切碎。还存在：
|
||||
|
||||
- `物 业`、`项 目`、`服\n务项目` 等版面换行造成的词内空格或断行。
|
||||
- 页码、目录页码和正文混在同一行。
|
||||
- 句子被错误拼接成超长段，或每句话都被拆成独立段。
|
||||
- 项目编号中的连字符被转义成 `440442\-2025\-00435`，同一标识在不同文件中形式不一致。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 先识别标题、列表、表格、地址、编号、签名区，再做段落重组。
|
||||
2. 普通段落内部把版面软换行合并为空或一个空格；中文字符之间通常直接合并，中英/数字边界按规则保留空格。
|
||||
3. 列表项、诗行、地址、落款、表格单元格内的有意义换行不得统一删除。
|
||||
4. `\-` 仅在确认是转换器多余转义时还原；项目编号和负号的字符本身必须保留。
|
||||
5. Unicode 采用 NFC，不建议全局 NFKC；NFKC 可能改变罗马数字、圈号、单位和兼容字符的法律原貌。
|
||||
|
||||
### 4.8 D001：页眉、页脚、目录页码和文档边界
|
||||
|
||||
优先级：P1。
|
||||
|
||||
已检测到至少 1,166 个“独立页码样式候选”，但其中也会混入 `0/1000` 等评分或限制值，所以不能仅凭正则删除。明确例子包括 `1 / 390`、`368 / 390`、`4 / 6`。
|
||||
|
||||
清洗要求：
|
||||
|
||||
- 只有在同一文本出现在多页相近顶部/底部位置，或页码形成合理序列时，才判定为页眉页脚。
|
||||
- `clean_fidelity.md` 可通过元数据保留页码映射；`clean_compare.md` 删除纯版式页码和重复运行标题。
|
||||
- 002 中 `file_16`—`file_19` 开头分别表现为符合性审查、综合文件、商务部分、技术部分，可能是同一投标人的分卷/分册。清洗器必须保存原始 `fileIndex` 和卷册边界，不能自行拼接。
|
||||
- 目录文本可转换为结构化章节导航；目录点线和页码不应参与正文相似度。
|
||||
|
||||
### 4.9 D002：重复内容与模板内容
|
||||
|
||||
优先级：P1，但严禁盲删。
|
||||
|
||||
重复既可能是转换器错误，也可能是投标文件真实模板。典型统计：
|
||||
|
||||
- 001 两份文件非空行重复比例分别为 81.8% 和 84.1%。同一“综合单价分析表”及说明出现约 900 次。
|
||||
- `002-21/file_3` 非空行重复比例 44.8%，含“装约买箱箱装”等乱码高频重复。
|
||||
- `002-21/file_15` 中 `Agree to be` 重复 813 次。
|
||||
- `002-21/file_5` 中 `- The steps are:` 重复 818 次。
|
||||
- `002-21/file_20` 中目录式条目“施工等级 282”“工期保证措施 282”分别重复 165、164 次。
|
||||
|
||||
处理原则：
|
||||
|
||||
1. 明确的幻觉/转换器退化重复应回源替换或隔离。
|
||||
2. 重复页眉、页脚、页码在对比版删除。
|
||||
3. 合法合同条款、报价表说明和表头在忠实版保留。对比版可把同一文档内重复模板块标为同一 `template_block_id`，在相似度计算时降权，而不是直接删除。
|
||||
4. 跨文档共同出现的合法招标模板本来就是对比目标的一部分，不应因“重复”而全部抹除。项目需要明确是检测抄袭、检测共同模板，还是两者分别评分。
|
||||
|
||||
### 4.10 L001：Word/PDF 转换遗留语法
|
||||
|
||||
优先级：P1/P2。
|
||||
|
||||
包括：
|
||||
|
||||
- 007 `file_1` 有 171 个 `(#_Toc...)` Word 目录链接，但全库没有对应显式锚点。
|
||||
- 001 有 2,594 个 `<div align="center">`，用 HTML 仅表达居中版式。
|
||||
- 575 个行内数学片段中有不少重复的 `\textcolor`、`\rightarrow`、`\cdots` 和勾股定理幻觉。
|
||||
- 20 个私用区字符主要是 U+F0B7、U+F070、U+F0D8，常见于 Wingdings/项目符号映射。
|
||||
- 10 个 `�` 出现在 7 份文件中，已经无法靠 Unicode 规范化恢复原字符。
|
||||
|
||||
处理要求：
|
||||
|
||||
- Word TOC 链接要么根据清洗后标题生成真实 slug，要么移除链接只保留目录文字；不得保留悬空锚点。
|
||||
- 纯版式 `<div align>` 转成语义标题/段落；居中信息可进样式元数据。
|
||||
- 私用区字符按原字体映射或源文件回查，不能一律删除。
|
||||
- `�` 必须逐处回源，无法回源时产生显式未解决项。
|
||||
- LaTeX 只有在源文件确实包含公式/符号时保留；重复生成的视觉符号应由原图或文本含义替代。
|
||||
|
||||
@@ -1,225 +0,0 @@
|
||||
# data/ 五份论文 Markdown 审计报告（2026-08-21）
|
||||
|
||||
> 状态：scratch 草稿，供评审。清洗力度由师姐逐项决定，本报告只列问题、证据和可选处置，不做力度决策。
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
- 五份文件均为英文学术论文的 PDF→Markdown 转换产物，分属 5 个来源（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness），噪声特征差异很大，不能用一套固定规则覆盖。
|
||||
- 最严重的问题不是格式噪声，而是**内容丢失**：2 份文件中途截断，1 份（JAMA）所有表格整体缺失。这类问题清洗无法恢复，只能重新转换或人工补录。
|
||||
- JAMA 文件的原始 PDF 是**未定稿的 Word 修订稿**，行号、审阅批注、修订残留词对全部泄漏进正文，是五份中污染最重的。
|
||||
- 格式类噪声（双重转义实体 31 处、孤行公式编号 12 处、空行断句、标题层级扁平、引用上标混用等）确定可清洗，风险低。
|
||||
- 少数问题涉及**语义改变**（数学公式被 OCR 改错、乱码替换专名），自动清洗有改错正文的风险，建议人工确认。
|
||||
|
||||
## 2. 审计范围与方法
|
||||
|
||||
- 对象：`data/*/markdowns/*.md` 共 5 份，逐行人工通读；行号均指 markdowns 下的 md 文件。
|
||||
- 辅助统计（本轮实际执行）：标题层级计数、图片/表格/实体/批注/arXiv 戳的 grep 计数。
|
||||
- 判定原则：只把"转换管线引入的噪声"列为清洗对象；原文自身的写作瑕疵（如语法错误）不属于转换噪声，单独列出并标注"不建议清洗"。
|
||||
- 未验证项：未与原 PDF 逐页比对内容完整性，"缺失"结论基于文内自引用（如正文提到 Table 2 但全文无 Table 2）和文件截断位置。
|
||||
|
||||
## 3. 总体统计
|
||||
|
||||
| 文件（下文简称） | 行数 | H1 | H2 | H3+ | 图片 | HTML表格 | 双重转义 | 孤行编号 | 批注 | arXiv边栏戳 |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| dmp (Disaster Med) | 208 | 0 | 11 | 0 | 1 | 7 | 23 | 0 | 0 | 0 |
|
||||
| jama (JAMA 2024) | 349 | 2 | 10 | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
|
||||
| ejhf (EJHF 2020) | 143 | 1 | 13 | 0 | 1 | 1 | 8 | 0 | 0 | 0 |
|
||||
| sim (Stat Med/arXiv) | 247 | 2 | 11 | 0 | 2 | 0 | 0 | 12 | 0 | 1 |
|
||||
| springer (ICU LSTM) | 154 | 1 | 12 | 0 | 4 | 1 | 0 | 1 | 0 | 1 |
|
||||
|
||||
补充：jama 有 128 行以"行号+空格"开头（手稿行号泄漏）；springer 的 3 处 arXiv 字符串中 2 处是参考文献的正常引用，仅 1 处是边栏戳。
|
||||
|
||||
## 4. 问题清单
|
||||
|
||||
每项标注：【确定】= 确定是转换噪声，可安全清洗；【语义】= 涉及内容判断，建议人工确认；【丢失】= 内容缺失，清洗不可恢复。处置选项仅供师姐选择。
|
||||
|
||||
### A. 内容级问题（最严重）
|
||||
|
||||
**A1【丢失】两份文件中途截断**
|
||||
- ejhf 第 143 行在 Table 1 HTML 表格中间戛然而止（"Medical history at randomization, no. (%)" 行后为空单元格）。Table 1 后半、Tables 2–4、全部图注、参考文献整体缺失。
|
||||
- sim 第 247 行止于方法 M5 描述中间，且结尾 URL 损坏（`https://cran.r-project.org/web/p6$^{10}$`）。第 3 节（结果）、4、5 节、参考文献、附录缺失。
|
||||
- 处置选项：a) 退回重新转换；b) 接受现状并在元数据标记"截断"；c) 人工补录缺失部分。清洗管线本身无法解决。
|
||||
|
||||
**A2【丢失】jama 全部表格缺失**
|
||||
- 正文多处引用 Table、Box 1、eTables 1–3，但全文 0 个表格、0 张图片。摘要性内容（如各器官系统阈值表）完全丢失。
|
||||
- 处置选项：同 A1。
|
||||
|
||||
**A3【丢失】dmp 的 FIGURE 1 流程图被转成 HTML 表格**
|
||||
- 第 83 行：流程图（决策树）被输出为单行 HTML 表格，图形语义尽失，仅 FIGURE 2（第 111 行）保留为图片。
|
||||
- 处置选项：a) 保留现状（有总比没有好）；b) 重新转换该页；c) 人工用图片替换。
|
||||
|
||||
**A4【丢失】sim 第 93 行句子开头缺失**
|
||||
- "/unpenalized) regression models" 以斜杠开头，前文整句丢失。
|
||||
- 处置选项：标记为损坏片段，或对照原文补录。
|
||||
|
||||
### B. 草稿/修订痕迹泄漏（仅 jama，原稿是 Word 修订稿）
|
||||
|
||||
**B1【确定】手稿行号泄漏（128 行）**
|
||||
- 正文行以行号开头："25 increase in the Sequential..."（第 115 行）、"26 with suspected infection"（第 116 行）等；标题也带行号："## 116 Results/recommandations"（第 149 行）、"## 117 Criteria..."（第 151 行）、"## 143 Organ dysfunction..."（第 165 行）。
|
||||
- 处置选项：a) 剥离行首行号（注意与正常编号列表、年份区分）；b) 连同批注一起整体退回，要求提供定稿版。
|
||||
|
||||
**B2【确定】审阅批注泄漏（2 处）**
|
||||
- 第 155–157 行："Commented [LS1]: Why highlighted?"、"Commented [SW2R1]: To make sure we use capital letters..."。
|
||||
- 处置选项：整段删除。注意第 292 行还有一句删除文字与保留文字混杂的句子（"Appropriate process and balancing measures Efforts to enhance..."），需人工断句。
|
||||
|
||||
**B3【语义】Word 修订残留词对（约 7 处）**
|
||||
- "defined identified by as"（第 91 行）、"defined identified using by"（第 153 行）、"defined identified as in sepsis-septic patients"（第 116 行）、"defined indicate by as"（第 163 行）、"was were derived"（第 306 行）、"The new-Phoenix"（第 197 行）等——是"插入词+删除词"并存的痕迹。
|
||||
- 哪个词是最终保留词需要对照定稿判断，自动二选一有风险。处置选项：a) 人工逐处确认；b) 退回要定稿。
|
||||
|
||||
**B4【确定】未填占位符**
|
||||
- "XX societies"（第 101 行）、"XX-microbiological testing and YY-antibiotics"（第 302 行）、"Endorsing societies: To be populated after acceptance"（第 317 行）。
|
||||
- 处置选项：保留并标记待补，或等定稿填充后再清洗。
|
||||
|
||||
**B5【确定】日期损坏**
|
||||
- 第 85 行 "Revision date: December 3122, 2023"。
|
||||
- 处置选项：对照原文改为 2023 年内正确日期（人工）。
|
||||
|
||||
### C. 标题结构问题
|
||||
|
||||
**C1【确定】标题层级扁平**
|
||||
- dmp：0 个 H1，11 个 H2 全部同级（含 Introduction/Methods/Results 等，第 1,3,7,59,67,103,137,153 行）。
|
||||
- ejhf：主章节与子章节同为 H2（"## Methods" 第 51 行、"## Study population" 第 53 行），无层级区分。
|
||||
- 处置选项：按章节编号/语义推断层级（2.1 → H3），或统一降级保持平级。前者需人工核对推断结果。
|
||||
|
||||
**C2【确定】标题被拆分成两行**
|
||||
- jama：标题拆成两个 H1（第 1–2 行 "International Consensus Criteria..." / "The Phoenix Pediatric Sepsis Criteria"，实为正副标题）。
|
||||
- sim：主标题拆成两个 H1（第 3–4 行 "Conounder selection..." / "treatment effect estimators"，且首词 "Conounder" 是 OCR 错字，原文应为 Confounder）；2.2 节标题拆两行（第 183/185 行 "…full matching on the propensity" / "## score"）。
|
||||
- 处置选项：合并为单标题；sim 主标题的 "Conounder" 拼写需对照原文确认（【语义】）。
|
||||
|
||||
**C3【语义】乱码标题**
|
||||
- dmp 第 47 行 `## ่วง`（泰文字符）。按位置推断应为 METHODS，但不能凭推断改写正文级内容。
|
||||
- 处置选项：人工对照 PDF 改回，或删除该标题。
|
||||
|
||||
**C4【确定】跑动页眉混入正文且被标为标题**
|
||||
- dmp 第 73、191 行 "## MSOFA Score for Critical Care Triage" 出现在句中和 REFERENCES 内；第 71→75 行一句话被它从中间切断。
|
||||
- 处置选项：删除页眉行并把被切断的句子接回。
|
||||
|
||||
### D. 表格问题
|
||||
|
||||
**D1【确定】表格压缩为单行 HTML**
|
||||
- 全部 9 个表格（dmp 7、ejhf 1、springer 1）都是 `<table>...</table>` 单行输出，diff、review、编辑都极难。
|
||||
- 处置选项：转 GFM 多行表格（简单表）；对含 rowspan/colspan 的复杂表保留 HTML 但格式化缩进。
|
||||
|
||||
**D2【确定】HTML 实体双重转义（31 处）**
|
||||
- dmp 23 处（`&gt;400`、`&lt;1.2`、`MAP&lt;70`，第 27/33/39 行等）；ejhf 8 处（`A1C&lt;7`、`7≤A1C&lt;8`，第 143 行）。
|
||||
- 处置选项：还原一层转义（`&gt;` → `>`）。这是最安全的清洗之一。
|
||||
|
||||
**D3【确定】合并单元格信息丢失**
|
||||
- dmp 第 33 行 Table 2 的 Liver 行出现空 `<td></td>`，跨行合并关系丢失，数值与表头对应断裂。
|
||||
- 处置选项：对照 PDF 人工修复合并结构，或标记为低可信表格。
|
||||
|
||||
**D4【确定】表格行错误合并**
|
||||
- dmp 第 39 行 Table 3：四个器官系统 "Respiratory Coagulation Liver Cardiovascular" 被挤进一个单元格。
|
||||
- 处置选项：人工拆分修复。
|
||||
|
||||
**D5【确定】表头 OCR 乱码**
|
||||
- springer 第 111 行 Table 1 表头 "Claesther"（应为 Classifier）。
|
||||
- 处置选项：人工改正（单处，低成本）。
|
||||
|
||||
### E. 文本级损坏
|
||||
|
||||
**E1【语义】跨脚本字符替换**
|
||||
- dmp 第 43 行 "atينS Hospital"、第 55 行 "ينDS Hospital"——阿拉伯字符 ين 替换了 "LD"（LDS Hospital 是机构专名）。
|
||||
- 处置选项：人工替换回 "LDS"。自动规则可检出非拉丁字符混入，但替换动作建议人工确认。
|
||||
|
||||
**E2【确定】跨页断词**
|
||||
- dmp：第 125 行结尾 "…at the relevant thresh" + 第 127 行 "olds of 8 and 11"（单词 thresholds 被页边界切开）。
|
||||
- sim：第 87/89 行 "except possi-" / "bly via treatment"；第 217/219 行 "cre-" / "ated by permuting"。
|
||||
- 处置选项：拼接断词（去连字符合并）。需注意英语中合法的行尾连字符（如 "well-known"）不能误合并，建议只合并"行尾连字符+下一行首为小写字母且拼出的词在词表内"的情况，其余保留待审。
|
||||
|
||||
**E3【确定】句内空行断句**
|
||||
- 大量段落被空行从句子中间切开：jama 第 103→105、113→115 行；ejhf 第 47–49、57–59、67–69、87–89、113–115、131–133 行；springer 第 16→20 行（中间还被 arXiv 戳隔开，见 H1）、42→44 行。
|
||||
- 处置选项：段内合并（前段末无句号且后段首为小写/连接词时拼接）。这是对 RAG 分块影响最大的问题之一。
|
||||
|
||||
**E4【确定】段落内容错位/孤立行**
|
||||
- springer 第 63 行孤立 "1"（公式编号漂移，见 F1）；第 81–83 行 "…improve the simple Multilayer Perceptron… other deep models." 后接 "including RNNs and MLPs."，段落被错误切开。
|
||||
- 处置选项：结合上下文人工归位。
|
||||
|
||||
### F. 数学公式问题（sim 最重，springer 次之）
|
||||
|
||||
**F1【确定】公式编号漂移成孤行**
|
||||
- sim 12 处：第 63/70/81/107/117/133/145/153/161/171/179/197 行分别是 "1"、"2"、"(4)"…"(12)"；springer 第 63 行 "1"。
|
||||
- 处置选项：并入对应公式块或删除（若公式本体已带编号）。需逐处对照，不宜盲目删除。
|
||||
|
||||
**F2【语义】公式内容被 OCR 改错**
|
||||
- sim 第 139 行 `$\exp(x) = \exp(x)/\{1+\exp(x)\}$`——这是 expit 函数定义（x↦e^x/(1+e^x)），左边的 $\exp(x)$ 应为 $x$ 或 expit(x)。OCR 错误改变了数学含义，且这种错误会误导下游读者。
|
||||
- 处置选项：人工对照原文修复；自动工具只能标记"公式与上下文不符"，不宜自动改。
|
||||
- sim 第 119 行 "weights w k" 下标丢失，同类。
|
||||
|
||||
**F3【确定】LaTeX 冗余空格与风格不一**
|
||||
- sim 全文行内公式带前导空格（`$ \widehat{\psi}_{j} $`）、内部空格过多（`\widehat { \mathrm { E } } ( Y ^ { a } )`，第 150 行）。
|
||||
- 处置选项：规范化空格（不改变符号语义的前提下）。风险低但需保守，避免动 `\,` `\;` 等有意义的间距命令。
|
||||
|
||||
**F4【确定】公式 OCR 字符间距拉宽**
|
||||
- springer 第 72/78 行 cases 环境里 `\mathrm { s u r v i v o r s ~ g r o u p }` 逐字空格。
|
||||
- 处置选项：去除字母间空格（保守做法：只处理 `\mathrm{}` 内的单字母间距模式）。
|
||||
|
||||
### G. 格式不一致
|
||||
|
||||
**G1【确定】引用上标风格混用**
|
||||
- 同一文件内 LaTeX `$^{1,2}$` 与 Unicode 上标（¹²、²⁻⁴、⁴⁹ ⁵⁰,⁵¹）并存：jama（第 115–116 行 Unicode vs 第 107/187 行 `$\geq 2$`）、ejhf（第 45–49 行 LaTeX vs 第 133 行 Unicode）、springer（第 5–6 行 `$ ^{1} $` vs 第 20 行 [1,2,4,3] 方括号）。
|
||||
- 处置选项：统一为一种（选哪种由师姐定，取决于下游用途：RAG 检索倾向 Unicode 纯文本，渲染倾向 LaTeX）。
|
||||
|
||||
**G2【可能】数字格式不一致**
|
||||
- ejhf 第 55/57 行 "6068 patients" vs "4,091 patients"；dmp 第 91 行 "0.81-.85"（小数点前缺 0）。
|
||||
- 注意：springer 的 "61.532"、"46.520" 是欧式千分位，可能是原文排版而非转换噪声，不能自动统一。
|
||||
- 处置选项：统一千分位与小数风格；欧式写法是否转换需师姐定。
|
||||
|
||||
**G3【确定】参考文献列表分隔不一致**
|
||||
- dmp：refs 1–18 空行分隔，19–33 连续堆叠（第 193–208 行）；springer：refs 1–8 空行分隔，9–19 连续堆叠（第 140–154 行）。
|
||||
- 处置选项：统一为一条一空行。
|
||||
|
||||
### H. 非正文噪声与资产
|
||||
|
||||
**H1【确定】arXiv 边栏戳**
|
||||
- sim 第 1 行 "arXiv:2001.08971v3 [stat.ME] 10 Oct 2020"；springer 第 18 行同类戳插在 Introduction 段落中间，把一段话切成三截（第 16/18/20 行）。
|
||||
- 处置选项：删除戳行并接回段落。注意别误删参考文献中的合法 "arXiv preprint arXiv:…"（springer 第 143/152 行）。
|
||||
|
||||
**H2【确定】机构仓库封面页**
|
||||
- ejhf 第 1–11 行：Glasgow eprints 引用说明、版本声明、`http://eprints.gla.ac.uk/213358/`、"Deposited on: 27 April 2020"，以及封面图 ``（该图是仓库封面，不是论文插图）。
|
||||
- 处置选项：整体删除封面区；封面图一并删或移入元数据。对 RAG 是纯噪声。
|
||||
|
||||
**H3【可能】图片相对路径依赖**
|
||||
- 全部图片用 `../images/` 相对路径（dmp 第 111 行、ejhf 封面、sim 第 229/231 行、springer 第 65/115/117/119 行）。md 文件一旦脱离原目录结构，图片全部失效。
|
||||
- 处置选项：a) 保持现状（目录结构不变时无碍）；b) 清洗时把路径改写为部署目标路径；c) 校验引用的图片文件是否存在并报告断链。sim 的 Figure 1 实为左右两个面板（sub0/sub1 两张图）共享一条图注（第 233 行），springer 的 Fig. 2 为三面板（sub1–sub3）——合并还是保持多图由师姐定。
|
||||
|
||||
## 5. 不建议清洗的内容（保真边界）
|
||||
|
||||
- **原文自身的写作瑕疵**：springer 论文语言明显不通（"was went to describe"、"In the other hand"、"section 4 discuss"），这是作者问题不是转换噪声，清洗工具不得改写学术内容。
|
||||
- **欧式千分位**（springer "61.532"）：疑似原文排版，自动统一有改数风险。
|
||||
- **专名与缩写的大小写、期刊缩写风格**：不属于转换噪声。
|
||||
- **A 类内容缺失**：不要试图用生成或推测内容"补全"缺失章节——宁可留空标记。
|
||||
|
||||
## 6. 决策清单（供师姐勾选力度）
|
||||
|
||||
| 编号 | 问题 | 涉及文件 | 建议决策点 |
|
||||
|---|---|---|---|
|
||||
| A1/A2/A4 | 截断与表格缺失 | ejhf, sim, jama | 重新转换 / 接受并标记 / 人工补录 |
|
||||
| A3 | 图转表格 | dmp | 保留 / 重转 / 换图 |
|
||||
| B1 | 行号剥离 | jama | 剥离 / 退回要定稿 |
|
||||
| B2 | 批注删除 | jama | 删（基本无争议） |
|
||||
| B3 | 修订词对 | jama | 人工定稿对照（不宜自动） |
|
||||
| B4/B5 | 占位符/坏日期 | jama | 保留标记 / 人工修 |
|
||||
| C1 | 层级重建 | dmp, ejhf | 推断层级 / 保持平级 |
|
||||
| C2 | 标题合并 | jama, sim | 合并（低风险） |
|
||||
| C3/C4 | 乱码标题/页眉 | dmp | 人工改 / 删 |
|
||||
| D1 | 表格展开 | 全部 | GFM / 缩进 HTML / 不动 |
|
||||
| D2 | 实体还原 | dmp, ejhf | 还原（低风险） |
|
||||
| D3/D4/D5 | 表格结构修复 | dmp, springer | 人工修 / 标记低可信 |
|
||||
| E1 | 乱码专名 | dmp | 人工替换 |
|
||||
| E2 | 断词拼接 | dmp, sim | 保守合并+白名单 |
|
||||
| E3 | 段内合并 | jama, ejhf, springer | 启用（对 RAG 影响大） |
|
||||
| E4 | 段落归位 | springer | 人工 |
|
||||
| F1 | 编号并入 | sim, springer | 对照处理 |
|
||||
| F2 | 公式纠错 | sim | 人工（含义级） |
|
||||
| F3/F4 | 公式空格 | sim, springer | 保守规范化 |
|
||||
| G1 | 上标统一 | jama, ejhf, springer | 定一种风格 |
|
||||
| G2 | 数字格式 | ejhf, dmp | 统一 / 保留原文 |
|
||||
| G3 | 参考文献分隔 | dmp, springer | 统一空行 |
|
||||
| H1 | arXiv 戳 | sim, springer | 删+接段（勿伤引文） |
|
||||
| H2 | 封面页 | ejhf | 删 |
|
||||
| H3 | 图片路径 | 全部 | 现状 / 改写 / 断链校验 |
|
||||
|
||||
## 7. 与既有审计的关系
|
||||
|
||||
`research-wiki/reference/GOVDOC_SAAS_CLEANING_SCOPE.md`（45 份政务文档审计，原名 MARKDOWN_CLEANING_AUDIT.md）中的幻觉、重复、页眉页脚等类别在本批论文中部分复现（C4/D2/H1 对应旧审计的 D001/L001 类），但本批新增了论文特有的类别：修订稿痕迹（B 类）、公式问题（F 类）、引用上标混用（G1）、截断缺失（A 类）。若后续建立通用清洗规则库，B/F/G 类需要论文 profile，不宜进默认规则。
|
||||
@@ -1,143 +0,0 @@
|
||||
# HTML 表格清洗专题调研：业界工具在 Markdown 清洗中如何处理 HTML 表格
|
||||
|
||||
> 状态：调研记录，尚未进入任何 design。
|
||||
>
|
||||
> 调研日期：2026-08-21。
|
||||
>
|
||||
> 定位：回答一个具体问题——Markdown 清洗中遇到 HTML 表格，业界工具实际怎么做。
|
||||
> 结论用于印证或修正 [`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md`](../reference/GOVDOC_SAAS_CLEANING_SCOPE.md)
|
||||
> 中 T001/T002 的方向，不构成对任何方案的批准。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
围绕“Markdown 里的 HTML 表格怎么办”，生态里的工具分成三种流派：
|
||||
|
||||
| 流派 | 做法 | 代表 |
|
||||
|---|---|---|
|
||||
| 生成端保真 | 复杂表格直接输出 HTML，不做管道表格 | MinerU、Docling（HTML/JSON 视图） |
|
||||
| 强行归一 | 全部转成管道表格，合并单元格静默损坏或内容重复 | Turndown + gfm 插件、Docling（Markdown 视图） |
|
||||
| 保真派 | 容错解析 → 校验网格 → 简单表转 GFM、复杂表保留 HTML/JSON | 本项目审计 T001 方向、Pandoc（grid tables / AST） |
|
||||
|
||||
支撑这张表的共同事实是：**GFM 管道表格语法在原理上表达不了合并单元格**。三种流派只是对这条约束的
|
||||
不同回答——绕开它、硬转它、或者按能力分流。
|
||||
|
||||
## 2. 语法能力边界：GFM 管道表格没有合并单元格写法
|
||||
|
||||
Pandoc 手册对各家 Markdown 表格方言的原文描述（2026-08-21 从官方 MANUAL 核实）：
|
||||
|
||||
| 方言 | 合并单元格 | 单元格内块级元素 |
|
||||
|---|---|---|
|
||||
| pipe tables（≈GFM 表格） | 不支持（单元格不能跨多行） | 不能包含块级元素 |
|
||||
| multiline tables | 明确不支持跨行/跨列单元格 | 可以 |
|
||||
| grid tables | **支持**（"Cells can span multiple columns or rows"） | 可以 |
|
||||
| HTML `<table>` | 原生 `rowspan`/`colspan` | 可以 |
|
||||
|
||||
grid tables 是唯一支持合并单元格的 Markdown 表格语法，但它是 Pandoc 扩展，GitHub 不渲染，
|
||||
对“清洗后还要在 GFM 渲染器里查看”的场景不可用。所以在 GFM 方言内部，合并单元格没有任何无损写法；
|
||||
想保真只能保留 HTML，或存结构化 JSON。
|
||||
|
||||
Pandoc 手册同时警告：从表达能力更强的格式转换时"some document elements, such as complex tables,
|
||||
may not fit"，"can be expected to be lossy"。
|
||||
|
||||
来源：[Pandoc MANUAL - Tables](https://pandoc.org/MANUAL.html)
|
||||
|
||||
## 3. 生成端：PDF→Markdown 转换器为什么输出 HTML 表格
|
||||
|
||||
MD 文档里出现 HTML 表格，通常不是 bug，而是转换器面对合并单元格（`rowspan`/`colspan`）、
|
||||
多级表头等管道语法表达不了的结构时的标准回退。
|
||||
|
||||
| 工具 | 表格输出策略 | 依据 |
|
||||
|---|---|--- |
|
||||
| [MinerU](https://github.com/opendatalab/MinerU) | 所有表格一律输出 HTML 嵌在 Markdown 中，不做管道表格；支持跨页表格拼接 | 官方 README 功能列表 |
|
||||
| [Docling](https://docling.org/) | 内部 TableFormer 模型专门恢复合并单元格；同一文档可导出 HTML / Markdown / JSON 三种视图 | 官网能力页 + docling-core 2.92.0 源码 |
|
||||
|
||||
对本项目的含义：HTML 表格是合法的中间形态，不是待清除的垃圾。清洗目标不是“消灭 HTML 表格”，
|
||||
而是“识别哪些表格结构正确、哪些在转换中损坏”。
|
||||
|
||||
## 4. 反面教材一：Turndown 静默产出错位表格
|
||||
|
||||
[Turndown](https://github.com/mixmark-io/turndown) 是最流行的 HTML→Markdown 转换库之一，
|
||||
[turndown-plugin-gfm 的 tables.js](https://github.com/mixmark-io/turndown-plugin-gfm/blob/master/src/tables.js)
|
||||
（v1.0.2，2018 年发布后基本未改）只做两件事：
|
||||
|
||||
1. 首行不是全 `<th>`（无表头行）的表格：保留 HTML 不转；
|
||||
2. 其余表格：按 DOM 位置逐格输出管道符。
|
||||
|
||||
它**完全没有 `colspan`/`rowspan` 的处理代码**。合并单元格不触发上面的回退，直接按 DOM 位置压扁，
|
||||
转出列数不齐的坏表，且**不报任何错**。
|
||||
|
||||
对本项目的含义：
|
||||
|
||||
- “无表头就不转”是能力判断驱动的回退，这个思想是对的；但它的能力判断漏掉了合并单元格；
|
||||
- 连最流行的转换库在这里都会静默弄坏表格——审计要求“先建 DOM、校验网格、禁止正则替换”有真实事故支撑；
|
||||
- 选 HTML→Markdown 转换库时，“是否处理 span”必须列入验证项，不能信 README 宣称。
|
||||
|
||||
## 5. 反面教材二：Docling 的 Markdown 导出重复合并单元格内容
|
||||
|
||||
读了 docling-core 2.92.0 的源码（wheel 解包，2026-08-21）：
|
||||
|
||||
- 内部 `TableCell` 带 `row_span`/`col_span` 和起止行列偏移；`TableData.grid` 属性把同一个 cell 对象
|
||||
**铺满**它覆盖的每个 (行， 列) 位置；
|
||||
- **HTML 序列化器**（`transforms/serializer/html.py`）：遍历网格时跳过被覆盖的续位
|
||||
（`rowstart != i` 或 `colstart != j` 时 `continue`），只在起始位置输出，并正确带上
|
||||
`rowspan="N"`/`colspan="N"`——语义保真；
|
||||
- **Markdown 序列化器**（`transforms/serializer/markdown.py` 的 `MarkdownTableSerializer`）：
|
||||
直接遍历铺满后的 grid，每个位置都输出 `col.text`——一个 `row_span=3` 的单元格内容在 Markdown
|
||||
输出里**重复出现 3 次**。转义只处理换行和管道符（`\n`→空格、`|`→`|`），再用
|
||||
tabulate `tablefmt="github"` 输出管道表格。
|
||||
|
||||
即 Docling 面对“Markdown 视图必须有合并单元格”的需求，选择了**内容重复**来保住矩形形状。
|
||||
这是“强行转管道表格会丢语义”的又一个实例，和 Turndown 的压扁是同一根源的两种表现。
|
||||
|
||||
对本项目的含义：
|
||||
|
||||
- “转 GFM”不是免费的格式变换，每一家实现都发明了自己的有损映射；
|
||||
- 如果未来用 Docling 做回源提取（调研报告第 6.1 节的候选方向），它的 Markdown 导出不能直接当作
|
||||
保真输出使用，需要用它的 `DoclingDocument` JSON 或 HTML 视图；
|
||||
- 审计 T001 说“强行转 GFM 会丢失语义”，这里的机制证据是：跨行列单元格要么被压扁（Turndown）、
|
||||
要么被重复（Docling）、要么失去合并关系本身（都失去 `rowspan`/`colspan` 语义）。
|
||||
|
||||
## 6. 清洗与格式化工具：主流选择是“不动 HTML 块”
|
||||
|
||||
| 工具 | 对 Markdown 内 HTML 表格的行为 | 来源 |
|
||||
|---|---|---|
|
||||
| remark / mdformat | raw HTML 当不透明块原样传递，不重新格式化、不转换 | [mdformat](https://mdformat.readthedocs.io/) 官方文档（核心保证是格式化前后 AST 一致，HTML 块不在处理范围） |
|
||||
| rumdl MD033（no-inline-html） | 报告 `<table>`（它有 Markdown 等价物），但 `fix` 只自动转 `em/strong/code/a/img/br/hr` 等行内简单标签，**不含表格**；`allowed-inside = ["table"]` 可整块豁免 | 本地 `reference/rumdl/docs/md033.md` |
|
||||
| rumdl MD056（table-column-count） | 校验每行列数与表头一致，自动修复方式是补/删空单元格 | 本地 `reference/rumdl/docs/md056.md` |
|
||||
| rumdl MD058（blanks-around-tables） | GFM 表格前后补空行 | 本地 `reference/rumdl/docs/md058.md` |
|
||||
|
||||
两点值得注意：
|
||||
|
||||
1. **没有主流工具自动把 HTML 表格转成 GFM 表格。** 连以“消灭 HTML”为目标的 MD033 都把表格留在
|
||||
“只报告、不修复”的范围里——因为工具作者知道这个转换会弄坏表格。
|
||||
2. **MD056 的自动修复方向与本项目审计相反。** 审计 T002 反对“补空单元格凑齐列数通过语法检查”，
|
||||
因为这可能掩盖静默丢列；MD056 恰恰把补空作为修复手段。借用这类规则时必须关掉它的自动修复，
|
||||
只取检测部分。
|
||||
|
||||
## 7. 与既有材料的关系
|
||||
|
||||
- 审计 T001（`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md` 第 4.4 节）的五步法——容错解析建 DOM、
|
||||
展开 rowspan/colspan 校验二维网格、简单矩形表转 GFM / 复杂表保留 HTML 或 JSON、拆多行、
|
||||
回源确认幻觉——与本次调研的所有正面证据一致，未发现需要修正的点；
|
||||
- [`markdown-cleaning-ecosystem-research-2026-08-20.md`](markdown-cleaning-ecosystem-research-2026-08-20.md)
|
||||
第 5 节的推荐流程（html5lib 容错解析、禁正则、按合并单元格分流）同样得到印证；
|
||||
- 新增的证据是反面案例的具体机制：Turndown 的压扁路径、Docling Markdown 视图的重复路径、
|
||||
Pandoc 手册的方言能力原文、MD056 修复方向与审计相反。
|
||||
|
||||
## 8. 对本项目的待决问题（不是结论）
|
||||
|
||||
以下问题在对应 design 时需要回答，本调研只提供背景：
|
||||
|
||||
1. 简单/复杂表格的分界线，除了“有无合并单元格”，是否还要看单元格内块级元素、嵌套表格和表头层级；
|
||||
2. 复杂表保留的“规范 HTML”具体规范到什么程度（属性白名单？标签重排？缩进策略？）；
|
||||
3. JSON grid 的格式是否对齐 Docling 的 `TableCell`（row_span/col_span/offset 字段），
|
||||
还是自定义 schema——涉及与未来回源 adapter 的成本权衡；
|
||||
4. 无表头表格（Turndown 的回退条件）按哪种流派处理：补合成表头转 GFM，还是保留 HTML。
|
||||
|
||||
## 9. 验证状态
|
||||
|
||||
- Pandoc 手册、Turndown 源码、MinerU README、Docling 官网：2026-08-21 通过网络核实；
|
||||
- docling-core 2.92.0：下载 wheel 解包读源码核实，涉及
|
||||
`MarkdownTableSerializer.serialize`、`TableData.grid`、HTML 序列化器的 span 处理；
|
||||
- rumdl 三条规则：读本地 `reference/rumdl/docs/`（该目录为镜像副本，以 rumdl 上游为准）；
|
||||
- 未验证：各工具在本项目真实数据上的实际表现——需要等对应组件 design 批准后用受控样本测试。
|
||||
@@ -1,170 +0,0 @@
|
||||
# HTML 表格真实数据结构分析：45 份 GovDoc 测试 Markdown
|
||||
|
||||
> 状态：只读分析记录，尚未进入任何 design。文中"建议"部分是候选方向，不是批准的方案。
|
||||
>
|
||||
> 分析日期：2026-08-21。
|
||||
>
|
||||
> 定位：用真实数据回答"我们的 HTML 表格到底长什么样、损坏在哪"，为未来表格清洗 design 提供测量依据，
|
||||
> 并修正 [`html-table-cleaning-ecosystem-research-2026-08-21.md`](html-table-cleaning-ecosystem-research-2026-08-21.md)
|
||||
> 第 8 节中可以用数据回答的待决问题。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
按"展开 rowspan/colspan 后每行列数是否一致"给全部顶层表格分类，**剔除空 `<tr></tr>` 之后**的分布：
|
||||
|
||||
| 分类 | 定义 | 数量 | 占比 |
|
||||
|---|---|---|---|
|
||||
| A 简单矩形表 | 无合并单元格、每行列数一致、无嵌套、无游离内容 | 2474 | 53% |
|
||||
| B 合法合并表 | 有 rowspan/colspan，展开后仍是完整矩形 | 356 | 8% |
|
||||
| C 损坏表 | 参差网格、标签截断、占位冲突等 | 1879 | 40% |
|
||||
|
||||
四个改变预期的发现：
|
||||
|
||||
1. **一半的"损坏"是空 `<tr></tr>` 造成的假象**——剔除后 C 类从 54% 降到 40%；
|
||||
2. **标签不闭合会吞掉半篇文档**——最极端的一个未闭合表格吞了 2.79MB、1294 个表格；
|
||||
3. **参差网格与合并单元格强相关**——六成以上参差表带 span 属性，指向转换器丢失 colspan 标注；
|
||||
4. **无表头是常态**（A 类中 96%），**表格内图片为零**，单元格内管道符为零。
|
||||
|
||||
## 2. 数据范围与方法
|
||||
|
||||
- 输入：`/home/lihaoze/gov_test_data/compare/*/uploads/*.md`，45 份，全程只读；
|
||||
本文只含聚合数字和结构事实，不含任何原文片段；
|
||||
- 分析脚本在会话级临时目录 `/tmp/table-analysis/analyze.py`，未入库（仓库处于文档治理阶段，无源码目录）；
|
||||
本文第 2.1 节的规则描述是复现依据；
|
||||
- 解析器：lxml `HTMLParser(recover=True)`。本地未装 html5lib。
|
||||
|
||||
### 2.1 测量规则
|
||||
|
||||
1. **代码围栏遮蔽**：先标记 ```` ``` ````/`~~~` 围栏内的位置，围栏里的 `<table>` 不计入；
|
||||
2. **片段切分**：对围栏外的 `<table`/`</table>` 事件按嵌套深度配对出顶层片段；悬空的 `</table>`
|
||||
（无对应开启）单独计数并忽略；到文件尾仍未闭合的片段标记 `truncated`；
|
||||
3. **结构分析**：每个片段单独喂给 lxml 容错解析；对每个 `<table>` 展开 rowspan/colspan 建二维占位网格，
|
||||
检测占位冲突，计算每行展开后的列数，收集表头、嵌套、游离文本、块级子标签、空单元格、超长单元格、
|
||||
退化重复等特征；
|
||||
4. **分类谓词**（`truncated` 为片段级标记，其余为表格级）：
|
||||
|
||||
```text
|
||||
A_simple : 非 truncated 且 span_cells=0 且无嵌套 且每行展开列数一致
|
||||
且无占位冲突 且无游离文本 且块级子标签 ⊆ {br}
|
||||
B_span_ok: 非 truncated 且无占位冲突 且每行展开列数一致 且无游离文本
|
||||
C_broken : 其余全部
|
||||
```
|
||||
|
||||
5. **空行修复复核**：把没有任何 `td`/`th` 子元素的 `<tr>` 整行剔除后重新跑同一分类，观察迁移。
|
||||
|
||||
## 3. 总量与原始分类
|
||||
|
||||
- 45 份文档共 7550 个 `<table` 开标签、7155 个闭标签，**缺口 395 个**；
|
||||
- 深度配对得到 4648 个顶层片段；容错解析后共 **4709 个表格元素**（缺口标签导致部分片段解析出多个表格）；
|
||||
- `parse_fail=0`；lxml recover 模式记录 parser 错误的片段仅 30/4648——**recover 解析器的错误日志是弱信号，
|
||||
不能当损坏检测器用**，损坏要靠网格校验发现。
|
||||
|
||||
原始分类（4709 个表格）：
|
||||
|
||||
| 分类 | 数量 | 占比 |
|
||||
|---|---|---|
|
||||
| A_simple | 1921 | 41% |
|
||||
| B_span_ok | 262 | 5% |
|
||||
| C_broken | 2526 | 54% |
|
||||
|
||||
结构特征（占 4709 的比例）：含 span 属性 1891（40%）、参差网格 2506（53%）、有 `<th>` 259（5.5%）、
|
||||
单行表 157、截断片段 67、超长单元格（>500 字符）197、巨型片段（>10KB）70、退化重复内容 33、
|
||||
游离文本 27、占位冲突 14、嵌套表格 5、含块级标签 5、**含图片 0**。
|
||||
|
||||
## 4. 发现一：空 `<tr></tr>` 是最大的单一"假损坏"来源
|
||||
|
||||
完全没有 `td`/`th` 子元素的空行标签在参差表里出现 3000+ 行次。它们把"列数一致的好表"撑成
|
||||
"某些行 0 列"的参差表。
|
||||
|
||||
剔除空行后重新分类：
|
||||
|
||||
| 迁移路径 | 数量 |
|
||||
|---|---|
|
||||
| C_broken → A_simple | 553 |
|
||||
| C_broken → B_span_ok | 94 |
|
||||
|
||||
即分类变为 **A 2474（53%）/ B 356（8%）/ C 1879（40%）**，一条零风险修复救回 13% 的表。
|
||||
空行不含任何内容，剔除是无损的。
|
||||
|
||||
空行出现的位置：表中间 1151 处、表尾 385 处（对修复后仍为 C 的表统计），**没有出现在表头位置**——
|
||||
符合"转换器输出残留"而非"表头占位"的形态。
|
||||
|
||||
## 5. 发现二：标签不闭合会吞掉后续正文
|
||||
|
||||
最极端案例 `001-2/uploads/file_0_PDF.md`：全文件 1315 开 / 1124 闭；从第 1315 行开始的一个未闭合
|
||||
表格把后续 **2,789,092 字符、内部含 1294 个表格**的内容全部吞进一个"顶层片段"。
|
||||
|
||||
全库 395 个闭合缺口意味着：**清洗的第一步不是处理表格，而是安全切分片段**。深度配对在标签缺失时
|
||||
会把正文和后续完整表格归并进一个巨型片段，后续所有基于片段的统计和修改都会失真。
|
||||
|
||||
切分策略的可用锚点：相邻顶层表格之间，2943 对隔着真实文本、1661 对只隔空行——块边界
|
||||
（空行 + 后续正文）在实际数据中是可识别的。
|
||||
|
||||
## 6. 发现三：参差网格与 span 强相关，指向"丢 colspan 标注"
|
||||
|
||||
2506 个参差表中 **1624 个（65%）带 span 属性**——格子内容在、宽度信息没了，是转换器丢失
|
||||
colspan 标注的形态，不全是真缺内容。
|
||||
|
||||
剔除空行后仍参差的 1851 个表，参差发生位置：
|
||||
|
||||
| 位置 | 数量 | 推断成因 |
|
||||
|---|---|---|
|
||||
| 中间各行乱 | 780 | 真·结构损坏或逐行丢标注 |
|
||||
| 表头行比正文长 | 619 | 多级表头被拍平成一行、丢层级 |
|
||||
| 首行短 | 239 | 表头/首行缺格 |
|
||||
| 仅尾部截短 | 213 | 跨页截断尾巴 |
|
||||
|
||||
四类成因不同，修复策略应当不同，但**都不能靠补空单元格自动修**——那正是
|
||||
[`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md`](../reference/GOVDOC_SAAS_CLEANING_SCOPE.md) T002
|
||||
明确反对、且 rumdl MD056 的 auto-fix 方向被本仓库否决的做法。
|
||||
|
||||
## 7. 发现四：无表头是常态，转换障碍集中在 `<br>`
|
||||
|
||||
对修复后仍是 A 类的 2474 个表，检查转 GFM 管道表格的内容障碍：
|
||||
|
||||
| 障碍 | 数量 | 占 A 类 |
|
||||
|---|---|---|
|
||||
| 无 `<th>` 表头 | 2393 | 96% |
|
||||
| 单元格含 `<br>` | 501 | 20% |
|
||||
| 单行表 | 176 | 7% |
|
||||
| 超长单元格（>500 字符） | 54 | 2% |
|
||||
| 单元格文本含 `\|` | 0 | 0% |
|
||||
| 表格内 `<img>` | 0 | 0% |
|
||||
|
||||
转义压力比预期小（管道符、图片都是零），压力集中在**合成表头**和 `<br>` 处理上。
|
||||
|
||||
## 8. 建议的处理策略（候选，未批准）
|
||||
|
||||
三步走，对齐审计 T001 的分流方向，用真实数据修正边界：
|
||||
|
||||
1. **切分**：容错定位 `<table>` 片段；未闭合的在块边界截断并标记 truncated，
|
||||
绝不让深度配对吞正文；
|
||||
2. **无损修复**：只做剔除空 `<tr>` 这一级别的零风险修复（实测救回 13% 的表）；
|
||||
3. **分流**：
|
||||
- A 类（53%）→ 转 GFM：合成表头、`<br>` 转空格、管道符转义兜底；
|
||||
- B 类（8%）→ 保留 HTML，规范化输出；
|
||||
- C 类（40%）→ 不自动修，保留原样 + 按第 6 节四类成因报告分类原因，交人工确认或回源。
|
||||
|
||||
对生态调研第 8 节待决问题的数据回答：
|
||||
|
||||
- 问题 4（无表头表格怎么处理）：数据表明无表头占绝对多数（96%），"补合成表头转 GFM"
|
||||
是主流路径；空表头还是首行充当表头仍是 design 待决；
|
||||
- 问题 1（简单/复杂分界线）：分界线除了合并单元格，必须加"展开后网格是否矩形"——
|
||||
本数据中它是比 span 更强的损坏信号（53% 对 40%）；
|
||||
- 问题 2、3（规范 HTML 程度、JSON schema）本次数据没有新增证据，维持开放。
|
||||
|
||||
## 9. 局限
|
||||
|
||||
- 全部数字只来自这 45 份文档，不得外推为一般结论（CLAUDE.md 第 5 节约束）；
|
||||
- 53/8/40 依赖"剔除空 `<tr>`"这条规则被采纳；不采纳则为 41/5/54；
|
||||
- lxml recover 解析可能自行重排损坏片段，个别表格的行列统计是解析结果而非字节事实；
|
||||
- 巨型片段内部的表格统计（如 001-2/file_0 被吞的 1294 个）已计入总数，但它们在原文中的
|
||||
真实边界未经人工核对；
|
||||
- 参差位置的四分类用的是简单规则（与列数众数比较），是启发式归类，不是语义判断。
|
||||
|
||||
## 10. 验证状态
|
||||
|
||||
- 第 3 至 7 节所有数字：2026-08-21 由只读脚本在真实数据上实际运行得出，脚本未入库；
|
||||
- 分类谓词与第 2.1 节规则描述和脚本逻辑一致，可据此重建等价测量；
|
||||
- 未验证：任何修复或转换策略的实际效果——A/B/C 分流、空行剔除、块边界截断都还没有实现，
|
||||
需要等对应 design 批准后用受控样本测试。
|
||||
@@ -1,492 +0,0 @@
|
||||
# Markdown 清洗生态调研与通用架构建议
|
||||
|
||||
> 状态：调研草稿，尚未批准为项目设计。
|
||||
>
|
||||
> 调研日期：2026-08-20。
|
||||
>
|
||||
> 更新方式：候选工具、许可证、实测结果或项目范围变化时更新；形成实施决定后转写为下一编号 design。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
这个项目不应该重新实现一个“正则表达式合集”，也不应该把 Prettier、mdformat、Unstructured 或某个
|
||||
PDF→Markdown 模型直接包装成最终产品。
|
||||
|
||||
现有工具各自只解决问题的一层：
|
||||
|
||||
- Markdown parser/formatter 能统一语法，但无法知道一句话是不是模型幻觉；
|
||||
- HTML 容错解析器能补齐标签，但无法保证补出的表格在业务上正确；
|
||||
- PDF/DOCX 提取器能回源重建，但仍可能 OCR 错误或生成幻觉；
|
||||
- PII 工具能提供候选实体，但无法自动决定跨文档伪名是否应该一致；
|
||||
- 文本质量过滤器能发现重复和低熵，却常以“整篇丢弃”为目标，不适合忠实修复文档。
|
||||
|
||||
因此建议把 `govdoc-md-cleaner` 定位为：
|
||||
|
||||
> **面向多项目的、可审计的文档规范化与派生框架。Markdown 是主要输入输出格式，但核心对象是带来源、
|
||||
> 结构、置信度和问题记录的文档，而不是一串待正则替换的文本。**
|
||||
|
||||
推荐的技术组合是：
|
||||
|
||||
| 层 | 首选候选 | 在本项目中的角色 |
|
||||
|---|---|---|
|
||||
| 核心语言 | Python | 与文档解析、OCR、隐私工具及现有下游生态衔接 |
|
||||
| Markdown 解析 | `markdown-it-py` + GFM 插件 | CommonMark/GFM 结构识别、块级行号映射；不负责语义修复 |
|
||||
| Markdown 输出 | 自有受控 renderer；`mdformat` 只作可选格式化后端 | 保证 profile 输出稳定，避免 formatter 越权改原文 |
|
||||
| 损坏 HTML | `html5lib`，必要时配合 `lxml` tree builder | 按浏览器规则恢复 DOM；恢复动作必须进入审计 |
|
||||
| 回源提取 | Docling 作为默认候选 adapter | PDF/DOCX/图片转结构化文档，保留页码、bbox 和 provenance |
|
||||
| 编码异常 | `ftfy` 作为候选建议器 | 识别/建议 mojibake 修复；默认不静默应用到法律文本 |
|
||||
| 隐私 | Presidio 可选 adapter + 中文自定义 recognizer | 检测、确定性伪名和图片脱敏；不是默认核心依赖 |
|
||||
| 重复/退化 | 自有 detector，参考 DataTrove 指标 | 长行、低熵、重复字符/n-gram、语言突变和固定幻觉模板 |
|
||||
| 多格式转换 | Pandoc 可选 adapter/对照 oracle | DOCX/HTML/Markdown 转换与 AST filter；不作为忠实度权威 |
|
||||
|
||||
`remark/unified` 是 Markdown 原生变换能力最完整的候选，但它会引入 Node.js/TypeScript 运行时；本项目的
|
||||
回源、OCR、中文隐私和下游环境更偏 Python，所以建议把 remark 作为设计参照和交叉验证器，而不是第一版核心。
|
||||
|
||||
这不是最终技术选型。下一步应以本报告为输入编写 design，并用小型 spike 验证关键假设后再批准依赖。
|
||||
|
||||
## 2. 审计告诉我们的真实问题
|
||||
|
||||
本报告以 [45 份 Markdown 清洗审计](../reference/MARKDOWN_CLEANING_AUDIT.md) 为本地事实来源。
|
||||
审计发现的不是单一格式问题，而是至少五个不同层次的问题：
|
||||
|
||||
1. **字节与字符层**：替换字符、私用区字符、NBSP、零宽字符、多余转义；
|
||||
2. **Markdown/HTML 语法层**：标题扁平、悬空链接、损坏 HTML/GFM 表格、极端长行；
|
||||
3. **文档结构层**：段落断裂、阅读顺序错误、页眉页脚混入、图片与表格丢失；
|
||||
4. **内容可信度层**：模型幻觉、退化重复、OCR 语义错误、无法凭 Markdown 恢复的缺失内容；
|
||||
5. **用途与合规层**：对比、RAG、受控忠实版、公开脱敏版对内容保留规则不同。
|
||||
|
||||
其中两个结论直接改变技术路线。
|
||||
|
||||
第一，Markdown parser 解析成功不能作为质量通过条件。CommonMark 明确规定任意字符序列都是合法文档，
|
||||
因此绝大多数“脏 Markdown”仍然可以无报错解析；我们必须建立额外的结构、内容和来源验证器
|
||||
（[CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/)）。
|
||||
|
||||
第二，GFM parser 接受的表格也未必满足我们的忠实度要求。GFM 对正文行缺列会补空单元格，多出的单元格
|
||||
会被忽略；这对法律和金额表格可能造成静默丢失，所以项目必须在 parser 之上做严格矩形网格验证
|
||||
（[GFM 表格规范](https://github.github.io/gfm/#tables-extension-)）。
|
||||
|
||||
## 3. 为什么成熟 formatter 不能直接解决
|
||||
|
||||
### 3.1 Prettier、mdformat
|
||||
|
||||
[Prettier](https://prettier.io/docs/) 和 [mdformat](https://mdformat.readthedocs.io/) 都是成熟的确定性格式化器。
|
||||
它们通过“解析后重新打印”统一标题、列表、换行等书写风格。mdformat 使用 `markdown-it-py`，并提供语法扩展
|
||||
和代码围栏 formatter 插件（[mdformat 插件文档](https://mdformat.readthedocs.io/en/stable/users/plugins.html)）。
|
||||
|
||||
适合：
|
||||
|
||||
- 已经确认语义正确的 Markdown；
|
||||
- 统一输出风格；
|
||||
- 检查幂等性；
|
||||
- Wiki、README 和开发者手写文档。
|
||||
|
||||
不适合直接处理本审计数据：
|
||||
|
||||
- formatter 不知道 `The quick brown fox...` 是幻觉；
|
||||
- 重新打印会扩大 diff，使逐项审计更困难；
|
||||
- 对 raw HTML、损坏表格和未知扩展可能规范化或转义；
|
||||
- 它无法从缺失图片引用恢复资产，也无法回到 PDF bbox。
|
||||
|
||||
建议：只在结构和内容已经通过验证的节点上使用，或作为最终输出的可选 profile；永不直接覆盖原输入。
|
||||
|
||||
### 3.2 markdownlint、remark-lint
|
||||
|
||||
[remark-lint](https://github.com/remarkjs/remark-lint) 有约 70 条可组合规则，能检查标题跳级、硬换行、
|
||||
链接语法和行长等；markdownlint 也有成熟的规则集。这些工具适合开发文档质量门禁，但其规则主要面向
|
||||
作者书写风格，不认识 PDF 页、OCR 置信度、表格合并单元格或业务实体。
|
||||
|
||||
建议：用作本仓 Wiki/README 的 CI，或复用部分规则思想；不作为业务文档清洗引擎。
|
||||
|
||||
## 4. Markdown AST 候选
|
||||
|
||||
### 4.1 `remark` / `unified`
|
||||
|
||||
[remark](https://github.com/remarkjs/remark) 提供 Markdown→mdast→Markdown 的完整插件流水线；
|
||||
[mdast](https://github.com/syntax-tree/mdast) 对 CommonMark、GFM 表格、图片、raw HTML 等节点有稳定模型，
|
||||
unist 生态还提供位置、source extraction、遍历、lint 和 vfile 消息。它是本次调研中最完整的
|
||||
Markdown-native 变换生态。
|
||||
|
||||
优点：
|
||||
|
||||
- parser、AST、visitor、transformer、lint、stringifier 是同一生态；
|
||||
- 节点通常带行、列、offset，适合生成诊断；
|
||||
- GFM、frontmatter、数学、directives 等扩展成熟；
|
||||
- TypeScript 类型和插件边界清晰。
|
||||
|
||||
代价：
|
||||
|
||||
- 核心运行时是 Node.js/ESM；
|
||||
- PDF/DOCX/OCR、中文文本处理和当前下游大多仍在 Python；
|
||||
- 双运行时会增加部署、版本锁定和跨语言 IR 的维护成本。
|
||||
|
||||
判断：如果项目只清洗开发者 Markdown，remark 是首选；对当前“文档回源 + 多项目 profile”目标，第一版
|
||||
不建议为它引入第二套运行时。可把它用于 conformance 对照或以后提供 TypeScript 前端。
|
||||
|
||||
### 4.2 `markdown-it-py` + `mdformat`
|
||||
|
||||
[markdown-it-py](https://markdown-it-py.readthedocs.io/en/latest/) 遵循 CommonMark，支持插件、自定义规则和
|
||||
GFM 相关扩展；Token 的 `map` 字段提供块级起止行号
|
||||
（[Token 文档](https://markdown-it-py.readthedocs.io/en/v4.2.0/_modules/markdown_it/token.html)）。它活跃、
|
||||
MIT、Python 原生，适合本项目第一版。
|
||||
|
||||
局限也需要明确：
|
||||
|
||||
- 它主要是 parser/HTML renderer，不是完整的 Markdown transformation framework；
|
||||
- 行号映射主要在块级，细粒度字符 offset 和跨回源 bbox 仍需我们维护；
|
||||
- raw HTML 会成为特殊 token，表格恢复仍要交给 HTML parser；
|
||||
- CommonMark 合法不等于文档内容可信。
|
||||
|
||||
建议：把它用于“识别现有 Markdown 的结构和边界”，再投影到项目自己的 Document IR；不要直接在 token
|
||||
列表上堆满业务规则。`mdformat` 可为确认安全的 AST 提供稳定输出，但 renderer 行为必须通过回归样本冻结。
|
||||
|
||||
### 4.3 Pandoc
|
||||
|
||||
[Pandoc](https://pandoc.org/MANUAL.html) 使用 reader→AST→writer 架构，Lua/JSON filter 可以按顺序变换 AST
|
||||
（[Pandoc filter 文档](https://pandoc.org/filters.html)）。它的多格式覆盖和长期稳定性很强。
|
||||
|
||||
适合：
|
||||
|
||||
- DOCX、HTML、Markdown 等格式导入导出；
|
||||
- 做第二实现的转换对照；
|
||||
- 用户明确接受 Pandoc 方言规范化的 profile。
|
||||
|
||||
不适合担任忠实版核心：
|
||||
|
||||
- reader/writer round-trip 会改变原始 Markdown 表达；
|
||||
- Pandoc AST 不是为逐字符审计和 PDF bbox 设计的；
|
||||
- 外部二进制与 [GPL-2.0 许可证](https://github.com/jgm/pandoc/blob/main/COPYING.md)需要独立部署评估；
|
||||
- 不能修复不存在于输入中的图片和内容。
|
||||
|
||||
判断：可选 adapter，不作为唯一内部表示。
|
||||
|
||||
### 4.4 Marko、Mistune 等 Python parser
|
||||
|
||||
[Marko](https://marko-py.readthedocs.io/en/latest/) 提供纯 Python CommonMark AST 和扩展机制，Mistune 偏向
|
||||
高速渲染。它们都能用于特定场景，但相较 `markdown-it-py`，当前项目更看重现成插件、维护活跃度、
|
||||
生态采用以及块级 source map。第一轮 spike 不必同时维护三个 Python parser。
|
||||
|
||||
## 5. 损坏 HTML 与表格恢复
|
||||
|
||||
[html5lib](https://html5lib.readthedocs.io/en/stable/) 按 WHATWG 浏览器解析算法处理可能损坏的 HTML，
|
||||
可以输出 ElementTree 或使用 lxml tree builder；[lxml 的 HTML5 接口](https://lxml.de/4.5/apidoc/lxml.html.html5parser.html)
|
||||
也支持 fragment 解析。
|
||||
|
||||
推荐流程：
|
||||
|
||||
1. 从 Markdown AST 中只取 raw HTML fragment，不把整篇 Markdown 当 HTML；
|
||||
2. 保存原 fragment、source span 和哈希；
|
||||
3. 使用 html5lib 容错解析并收集 parser errors；
|
||||
4. 构建显式二维 table grid，展开 `rowspan`/`colspan`；
|
||||
5. 校验每个输出 cell 都能映射到原节点或 source 区域；
|
||||
6. 仅无合并单元格的简单矩形表格输出 GFM；
|
||||
7. 复杂表格输出规范 HTML，并并行保留 JSON grid；
|
||||
8. parser 自动补齐的标签只说明“语法可恢复”，不能自动标为“语义已验证”。
|
||||
|
||||
不能采用旧实现那样用正则匹配 `<table>...</table>`：审计已经证明大量闭合标签缺失，正则既无法正确嵌套，
|
||||
也会把后续正文吞入表格。
|
||||
|
||||
## 6. PDF、DOCX 和图片回源候选
|
||||
|
||||
### 6.1 Docling：默认候选 adapter
|
||||
|
||||
[Docling](https://docling.org/) 支持 PDF、Office、HTML、Markdown、图片等格式，能输出 Markdown 和结构化
|
||||
`DoclingDocument`；后者包含表格、层级、bbox 和 provenance
|
||||
（[DoclingDocument 说明](https://github.com/docling-project/docling/blob/main/docs/concepts/docling_document.md)）。
|
||||
项目是 Python/MIT，OCR 后端可插拔。
|
||||
|
||||
它与审计需求最匹配的不是“Markdown 看起来更漂亮”，而是能先保存结构化、带位置的中间结果，再由我们
|
||||
生成 fidelity/profile 输出。因此建议把 Docling 作为第一批回源 adapter 的基准候选。
|
||||
|
||||
但它仍不能成为无条件真值：OCR、阅读顺序和表格模型都会出错，VLM 路径也可能生成内容。必须在 001、003
|
||||
有原始 PDF 的受控样本上验证字符、数字、表格和图片，不以官方 demo 或总准确率代替本项目测试。
|
||||
|
||||
### 6.2 Unstructured
|
||||
|
||||
[Unstructured partition](https://docs.unstructured.io/open-source/core-functionality/partitioning) 能把多种格式切成
|
||||
`Title`、`NarrativeText`、`ListItem`、`Table` 等元素，一些格式保留页码、坐标和 table HTML，适合作为
|
||||
另一种 source adapter 或元素分类对照。
|
||||
|
||||
其 `cleaners` 不能整体照搬。例如官方实现中的 `clean_dashes` 会替换连字符，`clean_bullets` 会删除项目符号，
|
||||
`clean_non_ascii_chars` 会丢弃非 ASCII 字符；这对中文法律文本、项目编号和列表结构明显过于激进
|
||||
（[cleaners 源码](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py)）。
|
||||
|
||||
判断：可评估 partition/metadata；不采用通用 `clean(...)` 作为默认清洗策略。
|
||||
|
||||
### 6.3 MinerU、Marker、MarkItDown
|
||||
|
||||
- [MinerU](https://github.com/opendatalab/MinerU) 支持 PDF/Office/图片到 Markdown、JSON 和图片资产，能力覆盖广，
|
||||
但本地审计已经展示某些现有解析产物中的幻觉与退化；此外它当前是 Apache-2.0 加附加商业与署名条款，
|
||||
不是无条件的标准 Apache-2.0（[MinerU 许可证](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md)）。
|
||||
- [Marker](https://github.com/datalab-to/marker) 能输出 Markdown、JSON、HTML 和 chunks，也暴露页/块结构；代码为
|
||||
Apache-2.0，但模型权重采用带商业门槛和用途限制的修改版 OpenRAIL-M，必须把代码与模型许可分开审查
|
||||
（[Marker 模型许可证](https://github.com/datalab-to/marker/blob/master/MODEL_LICENSE)）。
|
||||
- [Microsoft MarkItDown](https://github.com/microsoft/markitdown) 是轻量多格式→Markdown 工具，适合低成本文本提取，
|
||||
但它的目标不是页级 provenance、复杂表格忠实恢复或审计账本。
|
||||
|
||||
判断：三者都可成为 benchmark adapter，不应把任何一个输出直接标为 fidelity 真值。第一轮优先比较
|
||||
Docling、MinerU 和 Marker 的结构化 JSON，而不是只比较最终 Markdown 的视觉效果。
|
||||
|
||||
## 7. 编码、内容退化和隐私工具
|
||||
|
||||
### 7.1 `ftfy`
|
||||
|
||||
[ftfy](https://ftfy.readthedocs.io/en/latest/) 用保守启发式修复 Unicode mojibake，目标之一是避免把正常文本
|
||||
误改。它适合发现和建议典型 UTF-8/Windows-1252 误解码。
|
||||
|
||||
边界：
|
||||
|
||||
- 已经变成 `�` 的原字符信息不在字符串中，ftfy 无法凭空恢复；
|
||||
- 私用区字符需要字体或源文件映射；
|
||||
- 中文旧编码误解码和法律文本中的兼容字符仍需专门验证；
|
||||
- 即使候选看起来合理，也要保留 before/after、置信度和规则版本。
|
||||
|
||||
建议：作为 detector/candidate fixer；默认 profile 只自动应用有严格前置条件、通过实体保护检查的修复。
|
||||
|
||||
### 7.2 重复、低熵和幻觉模板
|
||||
|
||||
[DataTrove](https://github.com/huggingface/datatrove) 是大规模文本过滤/去重框架，已有行重复率、长行比例、
|
||||
标点比例、语言分数和 contamination 等统计。它的默认任务是筛掉低质量训练语料，而本项目需要定位并修复
|
||||
文档中的局部区域。
|
||||
|
||||
建议借鉴指标，不把 DataTrove 作为核心依赖：
|
||||
|
||||
- 最大行长与结构白名单；
|
||||
- 字符/短片段 run-length；
|
||||
- 唯一字符、token 和 n-gram 比例；
|
||||
- 压缩率与局部信息熵；
|
||||
- 相邻和非相邻重复块；
|
||||
- 文档主要语言与局部语言突变；
|
||||
- 已知转换器/模型幻觉签名。
|
||||
|
||||
detector 只产生 issue 和范围。没有可信来源时，默认隔离或人工确认，不自动编写替代内容。
|
||||
|
||||
### 7.3 Presidio
|
||||
|
||||
[Presidio](https://microsoft.github.io/presidio/) 支持文本、图片和结构化数据中的 PII 检测与匿名化，并允许使用
|
||||
正则、校验和、上下文、NER 和自定义 recognizer。官方也明确说明自动检测不能保证找到全部敏感信息。
|
||||
|
||||
适合：
|
||||
|
||||
- 作为可选 privacy adapter；
|
||||
- 为中国身份证、统一社会信用代码、手机号、银行账号等实现校验和与上下文 recognizer；
|
||||
- 用 custom operator 实现稳定、按实体区分的伪名；
|
||||
- 把文本、表格单元格和图片脱敏放进同一 profile。
|
||||
|
||||
不适合：
|
||||
|
||||
- 默认把所有候选直接覆盖；
|
||||
- 把不同值统一变成同一 `[PHONE]`/`[ID]`；
|
||||
- 认为通用 NER 已覆盖中文政务/合同实体；
|
||||
- 把 privacy profile 与 fidelity 修复写死在一起。
|
||||
|
||||
## 8. 推荐的领域无关架构
|
||||
|
||||
下面是候选架构，不是已批准契约：
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Markdown / HTML / PDF / DOCX / Image] --> B[Immutable source artifact]
|
||||
B --> C[Preflight detectors]
|
||||
B --> D[Parser / source adapters]
|
||||
C --> E[Issue ledger]
|
||||
D --> F[Document IR]
|
||||
F --> G[Validation and routing]
|
||||
E --> G
|
||||
G --> H[Safe deterministic repair]
|
||||
G --> I[Source-verified repair]
|
||||
G --> J[Quarantine / human review]
|
||||
H --> K[Canonical fidelity document]
|
||||
I --> K
|
||||
J --> K
|
||||
K --> L[Fidelity profile]
|
||||
K --> M[Retrieval profile]
|
||||
K --> N[Compare profile]
|
||||
K --> O[Public/privacy profile]
|
||||
L --> P[Markdown + assets + audit]
|
||||
M --> P
|
||||
N --> P
|
||||
O --> P
|
||||
```
|
||||
|
||||
### 8.1 Immutable source artifact
|
||||
|
||||
任何输入先冻结：原始 bytes、SHA-256、媒体类型、来源 ID 和接收时间。后续全部是新产物，不原地覆盖。
|
||||
只有路径而没有内容哈希不足以复现。
|
||||
|
||||
### 8.2 Document IR
|
||||
|
||||
IR 至少需要表达：
|
||||
|
||||
- block/node 类型、层级和子节点；
|
||||
- 原始 byte/line/column span；
|
||||
- 有源文档时的 page、bbox、source artifact hash；
|
||||
- 文本、表格网格、图片 asset ref 和文档边界；
|
||||
- parser/extractor、版本和置信度；
|
||||
- issue、annotation、repair 和 unresolved 关联。
|
||||
|
||||
不要让 Markdown AST 直接承担全部职责：mdast 或 markdown-it token 不包含完整的 PDF provenance、表格网格、
|
||||
修复证据和多 profile 状态。也不要直接把 DoclingDocument 定为公共契约，否则核心会被某个提取器绑定。
|
||||
|
||||
### 8.3 Detector 与 transformer 分离
|
||||
|
||||
每条规则先检测，再决定是否变换。建议规则声明：
|
||||
|
||||
- `rule_id` 与版本；
|
||||
- 支持的 node/input 类型；
|
||||
- source span 与证据；
|
||||
- safety level；
|
||||
- 是否确定性、幂等、可逆；
|
||||
- 影响文本、数字、实体、结构、资产或下游权重；
|
||||
- 验收 predicate。
|
||||
|
||||
安全级别建议：
|
||||
|
||||
| 级别 | 含义 | 默认行为 |
|
||||
|---|---|---|
|
||||
| `detect_only` | 只能确认异常，不能确认正确内容 | 记录 issue，不修改 |
|
||||
| `deterministic` | 不改变语义、前置条件严格 | 可自动执行并记录 |
|
||||
| `source_verified` | 新内容可回链到可信源区域 | 自动或抽检执行 |
|
||||
| `heuristic` | 有合理推断但可能误伤 | profile 显式开启或人工确认 |
|
||||
| `forbidden_without_source` | 金额、编号、缺失正文等无法猜测 | 隔离/未解决 |
|
||||
|
||||
### 8.4 Canonical fidelity 与 profiles
|
||||
|
||||
核心不应硬编码只有 `clean_fidelity.md` 和 `clean_compare.md`。更通用的方式是先生成 canonical fidelity
|
||||
document，再由 profile 派生：
|
||||
|
||||
| Profile | 目标 | 典型变化 |
|
||||
|---|---|---|
|
||||
| `fidelity` | 法律/业务忠实与可回源 | 只含确定性和源验证修复 |
|
||||
| `retrieval` | 搜索、RAG 和可读分块 | 规范段落、结构化 chunk、保留来源 |
|
||||
| `compare` | 相似度与模板分析 | 页眉页脚降噪、稳定图片 token、模板标注/降权 |
|
||||
| `public` | 可共享样例或外部处理 | 确定性伪名、图片/二维码脱敏、严格日志脱敏 |
|
||||
|
||||
未来项目可以增加自己的 profile；GovDoc 的章节模式、投标模板、compare 权重和中国政务字段放在插件/配置包，
|
||||
不能污染领域无关 core。
|
||||
|
||||
## 9. 建议的第一版产品边界
|
||||
|
||||
第一版不要一开始就做“全自动修复所有 Markdown”。建议逐层交付。
|
||||
|
||||
### M0：只读 audit
|
||||
|
||||
- 接受 Markdown；
|
||||
- 冻结哈希并解析 CommonMark/GFM/raw HTML 边界；
|
||||
- 输出 issue、source span、统计和阻断等级；
|
||||
- 覆盖编码、超长行、重复退化、链接/图片、标题、HTML/GFM 表格和 PII 候选；
|
||||
- 不改输入，不需要 PDF 模型。
|
||||
|
||||
这一阶段可以最早验证规则召回、误报、性能和审计 schema，不把修复风险混进来。
|
||||
|
||||
### M1：安全规范化
|
||||
|
||||
- 只执行严格确定性的 LF、NFC、尾空白、空标题等修复；
|
||||
- 每项变更有 source span 和 before/after hash；
|
||||
- 输出 fidelity Markdown、audit 和 unresolved；
|
||||
- 强制幂等、确定性和原输入不覆盖。
|
||||
|
||||
### M2：结构与回源
|
||||
|
||||
- raw HTML fragment 恢复和 table grid；
|
||||
- Docling source adapter；
|
||||
- 图片 asset store 与 manifest；
|
||||
- 页/区域级 re-extract；
|
||||
- 标题、列表、段落只在来源或高置信结构证据下恢复。
|
||||
|
||||
### M3：多用途 profile
|
||||
|
||||
- `retrieval`、`compare`、`public`；
|
||||
- privacy adapter；
|
||||
- 模板标注/权重和稳定实体/图片 token；
|
||||
- 各 profile 的差异可回链到 fidelity。
|
||||
|
||||
### M4：插件与规模化
|
||||
|
||||
- 稳定 rule/adapter/profile API；
|
||||
- 项目专属配置包；
|
||||
- 并行批处理、缓存、可恢复任务和机器可读报告；
|
||||
- 再评估 CLI、Python SDK、服务接口和跨语言消费。
|
||||
|
||||
## 10. 选型 spike 与验收建议
|
||||
|
||||
进入实现前建议建立下一份 design，并批准两个小型 spike。
|
||||
|
||||
### Spike A：Markdown/HTML 核心
|
||||
|
||||
使用脱敏合成 fixture 和审计列出的结构模式，比较：
|
||||
|
||||
- `markdown-it-py` + 自有 IR/renderer；
|
||||
- remark/mdast 作为对照；
|
||||
- html5lib 与 lxml recover 对损坏 table fragment 的差异。
|
||||
|
||||
至少覆盖：未闭合 table、GFM 多/少列、raw HTML 与 Markdown 交错、代码围栏内伪标签、超长行、中文硬换行、
|
||||
图片 URL、Word `_Toc`、标题断裂和多余转义。
|
||||
|
||||
验收关注：source span 完整率、round-trip 语义一致、没有静默 cell 丢失、幂等性、峰值内存和每 MiB 耗时。
|
||||
|
||||
### Spike B：回源提取
|
||||
|
||||
只在受控环境抽取 001、003 的风险分层页面，对 Docling、MinerU、Marker 做 A/B：
|
||||
|
||||
- 原文字符和关键实体准确率；
|
||||
- 表格网格、合并单元格和阅读顺序；
|
||||
- 图片数量、bbox 和 asset 引用；
|
||||
- 已知幻觉与重复退化命中；
|
||||
- CPU/GPU、耗时、峰值内存、模型版本和许可证约束。
|
||||
|
||||
不能只比较“生成的 Markdown 肉眼是否整齐”，也不能把某个引擎自己的置信度当作金标。
|
||||
|
||||
### 回归体系
|
||||
|
||||
- 公开/合成 fixture 进入 Git，复现结构问题但不包含客户原文；
|
||||
- 真实样本只在外部受控目录运行，以 case/file/page ID 和聚合指标报告；
|
||||
- P0 页面 100% 人工核对，其他页面风险分层抽样；
|
||||
- 金额、日期、项目编号、公司名、身份证候选做前后对账；
|
||||
- 每个 transformer 测幂等、确定性、边界和反例；
|
||||
- fidelity、retrieval、compare、public 分别验收，不能用单一“清洗率”。
|
||||
|
||||
## 11. 明确不建议的路线
|
||||
|
||||
- 不恢复旧版逐行正则清洗器作为默认基线；
|
||||
- 不对原文件直接运行 Prettier/mdformat 并覆盖；
|
||||
- 不用正则解析或补齐 HTML 表格；
|
||||
- 不把 parser 无报错当成 Markdown 正确；
|
||||
- 不对全文执行 `clean_extra_whitespace`、`clean_dashes`、全局 NFKC 或非 ASCII 删除；
|
||||
- 不因重复就删除合同条款，不因语言突变就自动删除段落；
|
||||
- 不用生成模型补写缺失文字、表格单元格或图片说明；
|
||||
- 不让不同实体、图片和缺失区域坍缩成同一个通用 token；
|
||||
- 不把某个 PDF 提取器的 Markdown 直接当权威真值；
|
||||
- 不把 GovDoc 的投标/采购规则写进通用 core。
|
||||
|
||||
## 12. 下一份 design 需要决定的事项
|
||||
|
||||
1. 是否批准“Python core + adapter/profile”方向；
|
||||
2. 第一阶段是否只做 Markdown audit，暂不引入 PDF/OCR 重依赖；
|
||||
3. 内部 IR 的最小字段和版本策略；
|
||||
4. audit/unresolved 的事件粒度与敏感信息保存边界；
|
||||
5. `fidelity`、`retrieval`、`compare`、`public` 哪些进入第一版；
|
||||
6. Markdown dialect 是 CommonMark + GFM，还是还要支持 frontmatter、math、directives；
|
||||
7. Docling/MinerU/Marker 的 benchmark 范围和许可证审查责任；
|
||||
8. 真实数据输出目录、保留周期、人工审核和脱敏规则；
|
||||
9. Python SDK、CLI、配置文件和插件 API 哪些属于首个可交付范围。
|
||||
|
||||
在这些事项获得批准前，本报告只代表调研判断，不代表依赖、schema 或产品行为已经确定。
|
||||
|
||||
## 13. 主要资料来源
|
||||
|
||||
本次优先使用官方文档、规范和上游仓库，GitHub 活跃度与许可证检查日期为 2026-08-20：
|
||||
|
||||
- [CommonMark 规范](https://spec.commonmark.org/0.31.2/)
|
||||
- [GitHub Flavored Markdown 规范](https://github.github.io/gfm/)
|
||||
- [markdown-it-py 文档](https://markdown-it-py.readthedocs.io/en/latest/)
|
||||
- [mdformat 文档](https://mdformat.readthedocs.io/en/stable/)
|
||||
- [remark](https://github.com/remarkjs/remark) 与 [mdast](https://github.com/syntax-tree/mdast)
|
||||
- [Pandoc filters](https://pandoc.org/filters.html)
|
||||
- [Docling](https://docling.org/) 与 [DoclingDocument](https://github.com/docling-project/docling/blob/main/docs/concepts/docling_document.md)
|
||||
- [Unstructured partition/cleaning](https://docs.unstructured.io/open-source/core-functionality/partitioning)
|
||||
- [html5lib](https://html5lib.readthedocs.io/en/stable/)
|
||||
- [ftfy](https://ftfy.readthedocs.io/en/latest/)
|
||||
- [Presidio](https://microsoft.github.io/presidio/)
|
||||
- [DataTrove](https://github.com/huggingface/datatrove)
|
||||
- [MinerU](https://github.com/opendatalab/MinerU)
|
||||
- [Marker](https://github.com/datalab-to/marker)
|
||||
- [MarkItDown](https://github.com/microsoft/markitdown)
|
||||
@@ -1 +0,0 @@
|
||||
"""Repository-local Markdown cleaning reviewer."""
|
||||
@@ -1,40 +0,0 @@
|
||||
import eslint from "@eslint/js";
|
||||
import reactHooks from "eslint-plugin-react-hooks";
|
||||
import reactRefresh from "eslint-plugin-react-refresh";
|
||||
import globals from "globals";
|
||||
import tseslint from "typescript-eslint";
|
||||
|
||||
export default tseslint.config(
|
||||
{ ignores: ["dist", "coverage", "node_modules"] },
|
||||
eslint.configs.recommended,
|
||||
...tseslint.configs.strictTypeChecked,
|
||||
...tseslint.configs.stylisticTypeChecked,
|
||||
{
|
||||
files: ["**/*.{ts,tsx}"],
|
||||
languageOptions: {
|
||||
globals: { ...globals.browser, ...globals.node },
|
||||
parserOptions: {
|
||||
projectService: true,
|
||||
tsconfigRootDir: import.meta.dirname,
|
||||
},
|
||||
},
|
||||
plugins: {
|
||||
"react-hooks": reactHooks,
|
||||
"react-refresh": reactRefresh,
|
||||
},
|
||||
rules: {
|
||||
...reactHooks.configs.flat.recommended.rules,
|
||||
"react-refresh/only-export-components": ["warn", { allowConstantExport: true }],
|
||||
"@typescript-eslint/consistent-type-definitions": ["error", "interface"],
|
||||
"@typescript-eslint/no-confusing-void-expression": "off",
|
||||
"@typescript-eslint/restrict-template-expressions": ["error", { allowNumber: true }],
|
||||
"react-hooks/set-state-in-effect": "off",
|
||||
},
|
||||
},
|
||||
{
|
||||
files: ["tests/**/*.{ts,tsx}"],
|
||||
rules: {
|
||||
"@typescript-eslint/no-non-null-assertion": "off",
|
||||
},
|
||||
},
|
||||
);
|
||||
@@ -1,13 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="color-scheme" content="light" />
|
||||
<title>mdpolish 评审器</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/client/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
-4599
File diff suppressed because it is too large
Load Diff
@@ -1,43 +0,0 @@
|
||||
{
|
||||
"name": "mdpolish-reviewer",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": "^24.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "vite --host 127.0.0.1",
|
||||
"build": "npm run typecheck && vite build",
|
||||
"typecheck": "tsc --noEmit -p tsconfig.json",
|
||||
"lint": "eslint src tests vite.config.ts",
|
||||
"test": "vitest run",
|
||||
"check": "npm run lint && npm run test && npm run build"
|
||||
},
|
||||
"dependencies": {
|
||||
"@codemirror/lang-markdown": "6.5.2",
|
||||
"@codemirror/merge": "6.12.2",
|
||||
"@codemirror/state": "6.7.1",
|
||||
"@codemirror/view": "6.43.9",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "10.0.1",
|
||||
"@testing-library/jest-dom": "7.0.1",
|
||||
"@testing-library/react": "16.3.2",
|
||||
"@types/node": "24.13.3",
|
||||
"@types/react": "19.2.18",
|
||||
"@types/react-dom": "19.2.4",
|
||||
"@vitejs/plugin-react": "6.1.0",
|
||||
"eslint": "10.9.0",
|
||||
"eslint-plugin-react-hooks": "7.1.1",
|
||||
"eslint-plugin-react-refresh": "0.5.4",
|
||||
"globals": "17.11.0",
|
||||
"jsdom": "30.0.1",
|
||||
"typescript": "6.0.3",
|
||||
"typescript-eslint": "8.67.0",
|
||||
"vite": "8.2.2",
|
||||
"vitest": "4.1.11"
|
||||
}
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Read-only artifact adapter and local HTTP service."""
|
||||
|
||||
from reviewer.server.artifacts import ReviewArtifactError, ReviewArtifacts
|
||||
|
||||
__all__ = ["ReviewArtifactError", "ReviewArtifacts"]
|
||||
@@ -1,233 +0,0 @@
|
||||
"""Serve one local review run through a loopback-only read-only HTTP API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import mimetypes
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urlsplit
|
||||
|
||||
from reviewer.server.artifacts import JsonObject, ReviewArtifactError, ReviewArtifacts
|
||||
|
||||
_SECURITY_HEADERS = {
|
||||
"Cache-Control": "no-store",
|
||||
"Content-Security-Policy": (
|
||||
"default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'none'; "
|
||||
"font-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'"
|
||||
),
|
||||
"Referrer-Policy": "no-referrer",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"X-Frame-Options": "DENY",
|
||||
}
|
||||
|
||||
|
||||
class ReviewerHttpServer(ThreadingHTTPServer):
|
||||
"""Threaded local server whose workers never keep process shutdown alive."""
|
||||
|
||||
daemon_threads = True
|
||||
|
||||
|
||||
def _valid_local_request(handler: BaseHTTPRequestHandler) -> bool:
|
||||
host = handler.headers.get("Host")
|
||||
if host is None:
|
||||
return False
|
||||
try:
|
||||
parsed_host = urlsplit(f"//{host}")
|
||||
if parsed_host.username is not None or parsed_host.password is not None:
|
||||
return False
|
||||
if parsed_host.hostname not in {"127.0.0.1", "localhost"}:
|
||||
return False
|
||||
if parsed_host.port is not None and not 0 < parsed_host.port < 65536:
|
||||
return False
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
origin = handler.headers.get("Origin")
|
||||
if origin is None:
|
||||
return True
|
||||
try:
|
||||
parsed_origin = urlsplit(origin)
|
||||
return parsed_origin.scheme == "http" and parsed_origin.netloc == host
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _api_response(repository: ReviewArtifacts, path: str) -> JsonObject:
|
||||
if path == "/api/v1/run":
|
||||
return repository.run_summary()
|
||||
parts = [part for part in path.split("/") if part]
|
||||
try:
|
||||
if len(parts) == 4 and parts[:3] == ["api", "v1", "documents"]:
|
||||
return repository.document_comparison(unquote(parts[3], encoding="utf-8", errors="strict"))
|
||||
if len(parts) == 6 and parts[:3] == ["api", "v1", "documents"] and parts[4] == "components":
|
||||
document_id = unquote(parts[3], encoding="utf-8", errors="strict")
|
||||
try:
|
||||
component_position = int(parts[5])
|
||||
except ValueError:
|
||||
raise ReviewArtifactError("unknown_component", "组件位置不存在。", 404) from None
|
||||
return repository.component_stage(document_id, component_position)
|
||||
except UnicodeDecodeError:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
|
||||
|
||||
def _handler_factory(repository: ReviewArtifacts, static_root: Path) -> type[BaseHTTPRequestHandler]:
|
||||
class ReviewRequestHandler(BaseHTTPRequestHandler):
|
||||
server_version = "mdpolish-reviewer"
|
||||
sys_version = ""
|
||||
|
||||
def log_message(self, format_: str, *args: Any) -> None:
|
||||
del format_, args
|
||||
|
||||
def _headers(self, status: int, content_type: str, content_length: int) -> None:
|
||||
self.send_response(status)
|
||||
for name, value in _SECURITY_HEADERS.items():
|
||||
self.send_header(name, value)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(content_length))
|
||||
self.end_headers()
|
||||
|
||||
def _json(self, status: int, payload: JsonObject, *, head_only: bool) -> None:
|
||||
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||
self._headers(status, "application/json; charset=utf-8", len(content))
|
||||
if not head_only:
|
||||
self.wfile.write(content)
|
||||
|
||||
def _error(self, error: Exception, *, head_only: bool) -> None:
|
||||
if isinstance(error, ReviewArtifactError):
|
||||
status = error.http_status
|
||||
code = error.code
|
||||
message = str(error)
|
||||
else:
|
||||
status = 500
|
||||
code = "internal_error"
|
||||
message = "评审器无法完成该请求。"
|
||||
self._json(status, {"error": {"code": code, "message": message}}, head_only=head_only)
|
||||
|
||||
def _static(self, path: str, *, head_only: bool) -> None:
|
||||
requested = "index.html" if path == "/" else unquote(path[1:], encoding="utf-8", errors="strict")
|
||||
if "\0" in requested:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
candidate = static_root / requested
|
||||
try:
|
||||
if candidate.is_symlink():
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
resolved = candidate.resolve(strict=True)
|
||||
if not resolved.is_relative_to(static_root) or not resolved.is_file():
|
||||
raise FileNotFoundError
|
||||
except (FileNotFoundError, OSError):
|
||||
if Path(requested).suffix:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||
resolved = (static_root / "index.html").resolve(strict=True)
|
||||
content = resolved.read_bytes()
|
||||
content_type = mimetypes.guess_type(resolved.name)[0] or "application/octet-stream"
|
||||
if content_type.startswith("text/") or content_type in {"application/javascript", "application/json"}:
|
||||
content_type += "; charset=utf-8"
|
||||
self._headers(200, content_type, len(content))
|
||||
if not head_only:
|
||||
self.wfile.write(content)
|
||||
|
||||
def _handle(self, method: str) -> None:
|
||||
head_only = method == "HEAD"
|
||||
if method not in {"GET", "HEAD"}:
|
||||
self.send_response(405)
|
||||
for name, value in _SECURITY_HEADERS.items():
|
||||
self.send_header(name, value)
|
||||
self.send_header("Allow", "GET, HEAD")
|
||||
payload: JsonObject = {
|
||||
"error": {"code": "method_not_allowed", "message": "只允许 GET 和 HEAD。"}
|
||||
}
|
||||
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(content)))
|
||||
self.end_headers()
|
||||
self.wfile.write(content)
|
||||
return
|
||||
try:
|
||||
if not _valid_local_request(self):
|
||||
raise ReviewArtifactError("invalid_origin", "只接受本机同源请求。", 403)
|
||||
request_path = urlsplit(self.path).path
|
||||
if request_path.startswith("/api/"):
|
||||
self._json(200, _api_response(repository, request_path), head_only=head_only)
|
||||
else:
|
||||
self._static(request_path, head_only=head_only)
|
||||
except Exception as error: # the response intentionally hides unexpected implementation details
|
||||
self._error(error, head_only=head_only)
|
||||
|
||||
def do_GET(self) -> None:
|
||||
self._handle("GET")
|
||||
|
||||
def do_HEAD(self) -> None:
|
||||
self._handle("HEAD")
|
||||
|
||||
def do_POST(self) -> None:
|
||||
self._handle("POST")
|
||||
|
||||
def do_PUT(self) -> None:
|
||||
self._handle("PUT")
|
||||
|
||||
def do_PATCH(self) -> None:
|
||||
self._handle("PATCH")
|
||||
|
||||
def do_DELETE(self) -> None:
|
||||
self._handle("DELETE")
|
||||
|
||||
def do_OPTIONS(self) -> None:
|
||||
self._handle("OPTIONS")
|
||||
|
||||
return ReviewRequestHandler
|
||||
|
||||
|
||||
def create_server(
|
||||
repository: ReviewArtifacts,
|
||||
static_root: Path,
|
||||
*,
|
||||
port: int = 0,
|
||||
) -> ReviewerHttpServer:
|
||||
"""Create, but do not start, the loopback reviewer server."""
|
||||
index = static_root / "index.html"
|
||||
if not index.is_file():
|
||||
raise ReviewArtifactError("missing_build", "未找到前端构建结果，请先运行 npm run build。", 400)
|
||||
return ReviewerHttpServer(("127.0.0.1", port), _handler_factory(repository, static_root.resolve()))
|
||||
|
||||
|
||||
def _arguments(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="只读查看一次 mdpolish 本地清洗运行。")
|
||||
parser.add_argument("--run-dir", required=True, help="一次已发布运行目录的绝对或相对路径")
|
||||
parser.add_argument("--port", type=int, default=0, help="本机端口；默认 0 表示自动选择")
|
||||
arguments = parser.parse_args(argv)
|
||||
if arguments.port < 0 or arguments.port > 65535:
|
||||
parser.error("--port 必须在 0 到 65535 之间")
|
||||
return arguments
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> None:
|
||||
arguments = _arguments(argv)
|
||||
repository = ReviewArtifacts(arguments.run_dir)
|
||||
static_root = Path(__file__).resolve().parents[1] / "dist"
|
||||
try:
|
||||
server = create_server(repository, static_root, port=arguments.port)
|
||||
except OSError as error:
|
||||
raise ReviewArtifactError("server_error", "无法启动本地评审服务。", 500) from error
|
||||
port = server.server_address[1]
|
||||
print(
|
||||
f"mdpolish 评审器已启动：http://127.0.0.1:{port}（{repository.run_id}，"
|
||||
f"{len(repository.documents)} 份文档）",
|
||||
flush=True,
|
||||
)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except ReviewArtifactError as error:
|
||||
raise SystemExit(f"评审器启动失败：{error}") from None
|
||||
@@ -1,691 +0,0 @@
|
||||
"""Strict adapters from published mdpolish artifacts to the reviewer API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import NoReturn, TypeAlias, cast
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
LocatedChange,
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
ReplayResult,
|
||||
replay_change_chain,
|
||||
)
|
||||
|
||||
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||
_STATUSES = {"success", "failed", "unstable"}
|
||||
_ERROR_STAGES = {"transform", "final_review"}
|
||||
|
||||
|
||||
class ReviewArtifactError(ValueError):
|
||||
"""Published artifacts cannot be exposed as a trustworthy review response."""
|
||||
|
||||
def __init__(self, code: str, message: str, http_status: int = 422) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.http_status = http_status
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ComponentRecord:
|
||||
component_id: str
|
||||
version: str
|
||||
parameters: JsonValue
|
||||
applicability: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ManifestDocument:
|
||||
document_id: str
|
||||
source_label: str
|
||||
status: str
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
change_count: int
|
||||
result_path: str
|
||||
cleaned_path: str | None
|
||||
diff_path: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LocatorDocument:
|
||||
document_id: str
|
||||
source_path: Path
|
||||
input_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ChangeRecord:
|
||||
replay: ReplayChange
|
||||
reason: str
|
||||
recorded_line: int
|
||||
recorded_column: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ResultRecord:
|
||||
status: str
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
changes: tuple[ChangeRecord, ...]
|
||||
errors: tuple[JsonObject, ...]
|
||||
residual_proposals: tuple[JsonObject, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DocumentRecord:
|
||||
manifest: ManifestDocument
|
||||
result: ResultRecord
|
||||
locator: LocatorDocument | None
|
||||
|
||||
|
||||
def _fail(code: str, message: str, http_status: int = 422) -> NoReturn:
|
||||
raise ReviewArtifactError(code, message, http_status)
|
||||
|
||||
|
||||
def _object(value: object, label: str) -> dict[str, object]:
|
||||
if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
|
||||
_fail("invalid_artifact", f"{label} 必须是 JSON 对象。")
|
||||
return cast(dict[str, object], value)
|
||||
|
||||
|
||||
def _array(value: object, label: str) -> list[object]:
|
||||
if not isinstance(value, list):
|
||||
_fail("invalid_artifact", f"{label} 必须是数组。")
|
||||
return cast(list[object], value)
|
||||
|
||||
|
||||
def _string(value: object, label: str, *, allow_empty: bool = False) -> str:
|
||||
if not isinstance(value, str) or "\0" in value or (not allow_empty and not value):
|
||||
_fail("invalid_artifact", f"{label} 必须是字符串。")
|
||||
return value
|
||||
|
||||
|
||||
def _integer(value: object, label: str, *, minimum: int = 0) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
|
||||
_fail("invalid_artifact", f"{label} 必须是不小于 {minimum} 的整数。")
|
||||
return value
|
||||
|
||||
|
||||
def _hash(value: object, label: str) -> str:
|
||||
digest = _string(value, label)
|
||||
if len(digest) != 64 or any(character not in "0123456789abcdef" for character in digest):
|
||||
_fail("invalid_artifact", f"{label} 必须是小写 SHA-256 摘要。")
|
||||
return digest
|
||||
|
||||
|
||||
def _status(value: object, label: str) -> str:
|
||||
status = _string(value, label)
|
||||
if status not in _STATUSES:
|
||||
_fail("invalid_artifact", f"{label} 不是已知状态。")
|
||||
return status
|
||||
|
||||
|
||||
def _json_value(value: object, label: str) -> JsonValue:
|
||||
if value is None or isinstance(value, str | int | float | bool):
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return [_json_value(item, label) for item in value]
|
||||
if isinstance(value, dict) and all(isinstance(key, str) for key in value):
|
||||
return {cast(str, key): _json_value(item, label) for key, item in value.items()}
|
||||
_fail("invalid_artifact", f"{label} 包含不支持的 JSON 值。")
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def _read_json(path: Path, label: str) -> dict[str, object]:
|
||||
try:
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
if content.startswith(b"\xef\xbb\xbf"):
|
||||
_fail("invalid_artifact", f"{label} 不能包含 UTF-8 BOM。")
|
||||
if not content.endswith(b"\n"):
|
||||
_fail("invalid_artifact", f"{label} 必须以换行结尾。")
|
||||
try:
|
||||
text = content.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError:
|
||||
_fail("invalid_utf8", f"{label} 不是严格 UTF-8。")
|
||||
try:
|
||||
payload = cast(
|
||||
object,
|
||||
json.loads(
|
||||
text,
|
||||
parse_constant=lambda _value: _fail(
|
||||
"invalid_artifact", f"{label} 不能包含非有限数值。"
|
||||
),
|
||||
),
|
||||
)
|
||||
except json.JSONDecodeError:
|
||||
_fail("invalid_artifact", f"{label} 不是合法 JSON。")
|
||||
return _object(payload, label)
|
||||
|
||||
|
||||
def _read_markdown(path: Path, expected_hash: str, label: str, error_code: str) -> str:
|
||||
try:
|
||||
if path.is_symlink() or not path.is_file():
|
||||
_fail(error_code, f"{label}不是普通文件。", 409)
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
_fail(error_code, f"无法读取{label}。", 409)
|
||||
if sha256(content).hexdigest() != expected_hash:
|
||||
_fail("hash_mismatch", f"{label}的内容哈希已经变化。", 409)
|
||||
try:
|
||||
return content.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError:
|
||||
_fail("invalid_utf8", f"{label}不是严格 UTF-8。", 409)
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def _run_path(run_directory: Path, relative_path: str, label: str) -> Path:
|
||||
relative = Path(relative_path)
|
||||
if relative.is_absolute() or "\0" in relative_path:
|
||||
_fail("unsafe_path", f"{label}必须是运行目录内的相对路径。")
|
||||
unresolved = run_directory / relative
|
||||
try:
|
||||
if unresolved.is_symlink() or not unresolved.is_file():
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
resolved = unresolved.resolve(strict=True)
|
||||
except OSError:
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
if not resolved.is_relative_to(run_directory):
|
||||
_fail("unsafe_path", f"{label}越过了运行目录边界。")
|
||||
return resolved
|
||||
|
||||
|
||||
def _component(value: object, position: int) -> ComponentRecord:
|
||||
item = _object(value, f"pipeline.components[{position}]")
|
||||
return ComponentRecord(
|
||||
component_id=_string(item.get("component_id"), "component_id"),
|
||||
version=_string(item.get("version"), "component version"),
|
||||
parameters=_json_value(item.get("parameters"), "component parameters"),
|
||||
applicability=_string(item.get("applicability"), "component applicability"),
|
||||
)
|
||||
|
||||
|
||||
def _manifest_document(value: object, position: int) -> ManifestDocument:
|
||||
item = _object(value, f"manifest.documents[{position}]")
|
||||
document_id = _string(item.get("document_id"), "document_id")
|
||||
status = _status(item.get("status"), "document status")
|
||||
result_path = _string(item.get("result_path"), "result_path")
|
||||
cleaned_value = item.get("cleaned_path")
|
||||
diff_value = item.get("diff_path")
|
||||
cleaned_path = None if cleaned_value is None else _string(cleaned_value, "cleaned_path")
|
||||
diff_path = None if diff_value is None else _string(diff_value, "diff_path")
|
||||
base = f"documents/{document_id}"
|
||||
if (
|
||||
result_path != f"{base}/result.json"
|
||||
or cleaned_path != (f"{base}/cleaned.md" if status == "success" else None)
|
||||
or diff_path != (f"{base}/changes.diff" if status == "success" else None)
|
||||
):
|
||||
_fail("invalid_artifact", "manifest 文档产物路径不符合 schema 1。")
|
||||
return ManifestDocument(
|
||||
document_id=document_id,
|
||||
source_label=_string(item.get("source_label"), "source_label"),
|
||||
status=status,
|
||||
input_sha256=_hash(item.get("input_sha256"), "input_sha256"),
|
||||
current_sha256=_hash(item.get("current_sha256"), "current_sha256"),
|
||||
change_count=_integer(item.get("change_count"), "change_count"),
|
||||
result_path=result_path,
|
||||
cleaned_path=cleaned_path,
|
||||
diff_path=diff_path,
|
||||
)
|
||||
|
||||
|
||||
def _parse_change(value: object, position: int) -> ChangeRecord:
|
||||
item = _object(value, f"changes[{position}]")
|
||||
proposal = _object(item.get("proposal_ref"), "proposal_ref")
|
||||
span = _object(item.get("span"), "span")
|
||||
location = _object(item.get("location"), "location")
|
||||
start = _integer(span.get("start"), "span.start")
|
||||
end = _integer(span.get("end"), "span.end")
|
||||
if end < start:
|
||||
_fail("invalid_artifact", "change span 必须满足 start <= end。")
|
||||
replay = ReplayChange(
|
||||
component_id=_string(item.get("component_id"), "change component_id"),
|
||||
component_version=_string(item.get("component_version"), "change component_version"),
|
||||
component_position=_integer(item.get("component_position"), "change component_position"),
|
||||
proposal_component_position=_integer(
|
||||
proposal.get("component_position"), "proposal component_position"
|
||||
),
|
||||
proposal_snapshot_sha256=_hash(proposal.get("snapshot_sha256"), "proposal snapshot_sha256"),
|
||||
proposal_index=_integer(proposal.get("proposal_index"), "proposal_index"),
|
||||
edit_index=_integer(item.get("edit_index"), "edit_index"),
|
||||
start=start,
|
||||
end=end,
|
||||
before=_string(item.get("before"), "before", allow_empty=True),
|
||||
after=_string(item.get("after"), "after", allow_empty=True),
|
||||
before_sha256=_hash(item.get("before_sha256"), "before_sha256"),
|
||||
after_sha256=_hash(item.get("after_sha256"), "after_sha256"),
|
||||
)
|
||||
return ChangeRecord(
|
||||
replay=replay,
|
||||
reason=_string(item.get("reason"), "change reason"),
|
||||
recorded_line=_integer(location.get("line"), "location.line", minimum=1),
|
||||
recorded_column=_integer(location.get("column"), "location.column", minimum=1),
|
||||
)
|
||||
|
||||
|
||||
def _parse_error(value: object, position: int) -> JsonObject:
|
||||
item = _object(value, f"errors[{position}]")
|
||||
stage = _string(item.get("stage"), "error stage")
|
||||
if stage not in _ERROR_STAGES:
|
||||
_fail("invalid_artifact", "error stage 不是已知阶段。")
|
||||
return {
|
||||
"component_id": _string(item.get("component_id"), "error component_id"),
|
||||
"component_version": _string(item.get("component_version"), "error component_version"),
|
||||
"component_position": _integer(item.get("component_position"), "error component_position"),
|
||||
"stage": stage,
|
||||
"error_type": _string(item.get("error_type"), "error_type"),
|
||||
"message": _string(item.get("message"), "error message"),
|
||||
}
|
||||
|
||||
|
||||
def _parse_residual(value: object, position: int) -> JsonObject:
|
||||
item = _object(value, f"residual_proposals[{position}]")
|
||||
reference = _object(item.get("proposal_ref"), "residual proposal_ref")
|
||||
proposal = _object(item.get("proposal"), "residual proposal")
|
||||
component_position = _integer(item.get("component_position"), "residual component_position")
|
||||
proposal_snapshot = _hash(proposal.get("snapshot_sha256"), "residual proposal snapshot_sha256")
|
||||
if (
|
||||
_integer(reference.get("component_position"), "residual reference component_position")
|
||||
!= component_position
|
||||
or _hash(reference.get("snapshot_sha256"), "residual reference snapshot_sha256")
|
||||
!= proposal_snapshot
|
||||
):
|
||||
_fail("invalid_artifact", "residual proposal_ref 与候选身份不一致。")
|
||||
_integer(reference.get("proposal_index"), "residual proposal_index")
|
||||
edits = _array(proposal.get("edits"), "residual proposal edits")
|
||||
if not edits:
|
||||
_fail("invalid_artifact", "residual proposal edits 不能为空。")
|
||||
for edit_position, edit_value in enumerate(edits):
|
||||
edit = _object(edit_value, f"residual edit[{edit_position}]")
|
||||
span = _object(edit.get("span"), "residual edit span")
|
||||
if _hash(edit.get("snapshot_sha256"), "residual edit snapshot_sha256") != proposal_snapshot:
|
||||
_fail("invalid_artifact", "residual edit 与候选快照不一致。")
|
||||
start = _integer(span.get("start"), "residual span.start")
|
||||
end = _integer(span.get("end"), "residual span.end")
|
||||
expected = _string(edit.get("expected_text"), "residual expected_text", allow_empty=True)
|
||||
_string(edit.get("replacement"), "residual replacement", allow_empty=True)
|
||||
if end < start or len(expected) != end - start:
|
||||
_fail("invalid_artifact", "residual edit 范围与 expected_text 不一致。")
|
||||
return {
|
||||
"component_id": _string(item.get("component_id"), "residual component_id"),
|
||||
"component_version": _string(item.get("component_version"), "residual component_version"),
|
||||
"component_position": component_position,
|
||||
"reason": _string(proposal.get("reason"), "residual reason"),
|
||||
"edit_count": len(edits),
|
||||
}
|
||||
|
||||
|
||||
def _component_json(component: ComponentRecord, position: int, change_count: int) -> JsonObject:
|
||||
return {
|
||||
"component_position": position,
|
||||
"component_id": component.component_id,
|
||||
"version": component.version,
|
||||
"parameters": component.parameters,
|
||||
"applicability": component.applicability,
|
||||
"change_count": change_count,
|
||||
}
|
||||
|
||||
|
||||
class ReviewArtifacts:
|
||||
"""Validated, read-only view over one explicitly selected run directory."""
|
||||
|
||||
def __init__(self, run_directory: str | Path) -> None:
|
||||
requested = Path(run_directory)
|
||||
try:
|
||||
if requested.is_symlink() or not requested.is_dir():
|
||||
_fail("missing_run", "指定的运行目录不存在或不是普通目录。", 400)
|
||||
self.run_directory = requested.resolve(strict=True)
|
||||
except OSError:
|
||||
_fail("missing_run", "无法读取指定的运行目录。", 400)
|
||||
|
||||
manifest = _read_json(_run_path(self.run_directory, "manifest.json", "manifest.json"), "manifest.json")
|
||||
if manifest.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 manifest.json schema 1。", 409)
|
||||
run = _object(manifest.get("run"), "manifest.run")
|
||||
pipeline = _object(manifest.get("pipeline"), "manifest.pipeline")
|
||||
summary = _object(manifest.get("summary"), "manifest.summary")
|
||||
self.run_id = _string(run.get("run_id"), "run_id")
|
||||
self.run_json: JsonObject = {
|
||||
"run_id": self.run_id,
|
||||
"run_date": _string(run.get("run_date"), "run_date"),
|
||||
"status": _status(run.get("status"), "run status"),
|
||||
"started_at_utc": _string(run.get("started_at_utc"), "started_at_utc"),
|
||||
"completed_at_utc": _string(run.get("completed_at_utc"), "completed_at_utc"),
|
||||
"retention_until": _string(run.get("retention_until"), "retention_until"),
|
||||
}
|
||||
self.components = tuple(
|
||||
_component(value, position)
|
||||
for position, value in enumerate(_array(pipeline.get("components"), "pipeline.components"))
|
||||
)
|
||||
manifest_documents = tuple(
|
||||
_manifest_document(value, position)
|
||||
for position, value in enumerate(_array(manifest.get("documents"), "manifest.documents"))
|
||||
)
|
||||
if len({item.document_id for item in manifest_documents}) != len(manifest_documents):
|
||||
_fail("invalid_artifact", "manifest 中的 document_id 必须唯一。")
|
||||
self.summary_json: JsonObject = {
|
||||
"document_count": _integer(summary.get("document_count"), "summary.document_count"),
|
||||
"success_count": _integer(summary.get("success_count"), "summary.success_count"),
|
||||
"failed_count": _integer(summary.get("failed_count"), "summary.failed_count"),
|
||||
"unstable_count": _integer(summary.get("unstable_count"), "summary.unstable_count"),
|
||||
"change_count": _integer(summary.get("change_count"), "summary.change_count"),
|
||||
}
|
||||
expected_summary: JsonObject = {
|
||||
"document_count": len(manifest_documents),
|
||||
"success_count": sum(item.status == "success" for item in manifest_documents),
|
||||
"failed_count": sum(item.status == "failed" for item in manifest_documents),
|
||||
"unstable_count": sum(item.status == "unstable" for item in manifest_documents),
|
||||
"change_count": sum(item.change_count for item in manifest_documents),
|
||||
}
|
||||
if self.summary_json != expected_summary:
|
||||
_fail("invalid_artifact", "manifest 汇总与文档索引不一致。")
|
||||
expected_status = (
|
||||
"failed"
|
||||
if expected_summary["failed_count"]
|
||||
else "unstable"
|
||||
if expected_summary["unstable_count"]
|
||||
else "success"
|
||||
)
|
||||
if self.run_json["status"] != expected_status:
|
||||
_fail("invalid_artifact", "manifest 运行状态与文档状态不一致。")
|
||||
|
||||
locator_path = self.run_directory / "review-locator.json"
|
||||
locators: tuple[LocatorDocument, ...] | None = None
|
||||
self.original_run_location_changed = False
|
||||
if locator_path.exists():
|
||||
locator = _read_json(
|
||||
_run_path(self.run_directory, "review-locator.json", "review-locator.json"),
|
||||
"review-locator.json",
|
||||
)
|
||||
if locator.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 review-locator.json schema 1。", 409)
|
||||
locator_run = _object(locator.get("run"), "review locator run")
|
||||
if (
|
||||
_string(locator_run.get("run_id"), "locator run_id") != self.run_id
|
||||
or locator_run.get("manifest_path") != "manifest.json"
|
||||
):
|
||||
_fail("invalid_artifact", "review locator 与 manifest 运行身份不一致。")
|
||||
recorded_directory_text = _string(locator_run.get("run_directory"), "locator run_directory")
|
||||
recorded_directory = Path(recorded_directory_text)
|
||||
if not recorded_directory.is_absolute() or str(recorded_directory) != str(recorded_directory.resolve()):
|
||||
_fail("invalid_artifact", "locator run_directory 必须是绝对解析路径。")
|
||||
self.original_run_location_changed = recorded_directory != self.run_directory
|
||||
parsed_locators: list[LocatorDocument] = []
|
||||
for position, value in enumerate(_array(locator.get("documents"), "locator documents")):
|
||||
item = _object(value, f"locator.documents[{position}]")
|
||||
source_text = _string(item.get("source_path"), "source_path")
|
||||
source_path = Path(source_text)
|
||||
if not source_path.is_absolute() or str(source_path) != str(source_path.resolve()):
|
||||
_fail("invalid_artifact", "source_path 必须是绝对解析路径。")
|
||||
parsed_locators.append(
|
||||
LocatorDocument(
|
||||
document_id=_string(item.get("document_id"), "locator document_id"),
|
||||
source_path=source_path,
|
||||
input_sha256=_hash(item.get("input_sha256"), "locator input_sha256"),
|
||||
)
|
||||
)
|
||||
locators = tuple(parsed_locators)
|
||||
if len(locators) != len(manifest_documents):
|
||||
_fail("invalid_artifact", "review locator 文档数量与 manifest 不一致。")
|
||||
for indexed, located in zip(manifest_documents, locators, strict=True):
|
||||
if indexed.document_id != located.document_id or indexed.input_sha256 != located.input_sha256:
|
||||
_fail("invalid_artifact", "review locator 文档身份与 manifest 不一致。")
|
||||
|
||||
records: list[DocumentRecord] = []
|
||||
for position, manifest_document in enumerate(manifest_documents):
|
||||
result = self._parse_result(manifest_document)
|
||||
records.append(
|
||||
DocumentRecord(
|
||||
manifest=manifest_document,
|
||||
result=result,
|
||||
locator=None if locators is None else locators[position],
|
||||
)
|
||||
)
|
||||
self.documents = tuple(records)
|
||||
self.documents_by_id = {item.manifest.document_id: item for item in self.documents}
|
||||
|
||||
def _parse_result(self, manifest: ManifestDocument) -> ResultRecord:
|
||||
payload = _read_json(_run_path(self.run_directory, manifest.result_path, "result.json"), "result.json")
|
||||
if payload.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 result.json schema 1。", 409)
|
||||
document = _object(payload.get("document"), "result.document")
|
||||
if (
|
||||
_string(document.get("document_id"), "result document_id") != manifest.document_id
|
||||
or _string(document.get("source_label"), "result source_label") != manifest.source_label
|
||||
):
|
||||
_fail("invalid_artifact", "result 与 manifest 文档身份不一致。")
|
||||
status = _status(payload.get("status"), "result status")
|
||||
input_hash = _hash(payload.get("input_sha256"), "result input_sha256")
|
||||
current_hash = _hash(payload.get("current_sha256"), "result current_sha256")
|
||||
changes = tuple(
|
||||
_parse_change(value, position)
|
||||
for position, value in enumerate(_array(payload.get("changes"), "result changes"))
|
||||
)
|
||||
errors = tuple(
|
||||
_parse_error(value, position)
|
||||
for position, value in enumerate(_array(payload.get("errors"), "result errors"))
|
||||
)
|
||||
residuals = tuple(
|
||||
_parse_residual(value, position)
|
||||
for position, value in enumerate(_array(payload.get("residual_proposals"), "result residual_proposals"))
|
||||
)
|
||||
output = _object(payload.get("output"), "result output")
|
||||
cleaned = output.get("cleaned_path")
|
||||
diff = output.get("diff_path")
|
||||
expected_cleaned: object = "cleaned.md" if status == "success" else None
|
||||
expected_diff: object = "changes.diff" if status == "success" else None
|
||||
if cleaned != expected_cleaned or diff != expected_diff:
|
||||
_fail("invalid_artifact", "result 状态与输出路径不一致。")
|
||||
if (
|
||||
status != manifest.status
|
||||
or input_hash != manifest.input_sha256
|
||||
or current_hash != manifest.current_sha256
|
||||
or len(changes) != manifest.change_count
|
||||
):
|
||||
_fail("invalid_artifact", "result 与 manifest 文档索引不一致。")
|
||||
if status == "success" and (errors or residuals):
|
||||
_fail("invalid_artifact", "success 文档不能包含错误或残留候选。")
|
||||
if status == "failed" and (not errors or residuals):
|
||||
_fail("invalid_artifact", "failed 文档的错误或残留状态不合法。")
|
||||
if status == "unstable" and (errors or not residuals):
|
||||
_fail("invalid_artifact", "unstable 文档的错误或残留状态不合法。")
|
||||
for change in changes:
|
||||
position = change.replay.component_position
|
||||
if position >= len(self.components):
|
||||
_fail("invalid_artifact", "change 没有对应的流水线组件。")
|
||||
component = self.components[position]
|
||||
if (
|
||||
change.replay.component_id != component.component_id
|
||||
or change.replay.component_version != component.version
|
||||
):
|
||||
_fail("invalid_artifact", "change 身份与流水线组件不一致。")
|
||||
for evidence in (*errors, *residuals):
|
||||
position_value = evidence["component_position"]
|
||||
component_id = evidence["component_id"]
|
||||
component_version = evidence["component_version"]
|
||||
if not isinstance(position_value, int) or position_value >= len(self.components):
|
||||
_fail("invalid_artifact", "审计证据没有对应的流水线组件。")
|
||||
component = self.components[position_value]
|
||||
if component_id != component.component_id or component_version != component.version:
|
||||
_fail("invalid_artifact", "审计证据身份与流水线组件不一致。")
|
||||
if manifest.diff_path is not None:
|
||||
_run_path(self.run_directory, manifest.diff_path, "changes.diff")
|
||||
return ResultRecord(status, input_hash, current_hash, changes, errors, residuals)
|
||||
|
||||
def _source(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||
if record.locator is None:
|
||||
return None, "这次历史运行没有 review-locator.json，无法定位完整原文。"
|
||||
try:
|
||||
return (
|
||||
_read_markdown(
|
||||
record.locator.source_path,
|
||||
record.manifest.input_sha256,
|
||||
"原文",
|
||||
"unavailable_source",
|
||||
),
|
||||
None,
|
||||
)
|
||||
except ReviewArtifactError as error:
|
||||
return None, str(error)
|
||||
|
||||
def _cleaned(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||
if record.manifest.status != "success" or record.manifest.cleaned_path is None:
|
||||
return None, None
|
||||
try:
|
||||
path = _run_path(self.run_directory, record.manifest.cleaned_path, "cleaned.md")
|
||||
return (
|
||||
_read_markdown(path, record.manifest.current_sha256, "清洗结果", "missing_artifact"),
|
||||
None,
|
||||
)
|
||||
except ReviewArtifactError as error:
|
||||
return None, str(error)
|
||||
|
||||
def _replay(self, record: DocumentRecord, original: str, cleaned: str | None) -> ReplayResult:
|
||||
try:
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=record.result.input_sha256,
|
||||
components=tuple(ReplayComponent(item.component_id, item.version) for item in self.components),
|
||||
changes=tuple(item.replay for item in record.result.changes),
|
||||
current_sha256=record.result.current_sha256,
|
||||
current_markdown=cleaned,
|
||||
include_zero_change_stages=record.result.status == "success",
|
||||
)
|
||||
except ReplayError as error:
|
||||
raise ReviewArtifactError("untrusted_replay", f"产物无法可信重放：{error}", 409) from error
|
||||
for recorded, located in zip(record.result.changes, replayed.changes, strict=True):
|
||||
if recorded.recorded_line != located.line or recorded.recorded_column != located.column:
|
||||
_fail("untrusted_replay", "产物记录的位置与重放快照不一致。", 409)
|
||||
return replayed
|
||||
|
||||
@staticmethod
|
||||
def _summary_json(
|
||||
record: DocumentRecord,
|
||||
original: str | None,
|
||||
source_error: str | None,
|
||||
cleaned: str | None,
|
||||
output_error: str | None,
|
||||
) -> JsonObject:
|
||||
return {
|
||||
"document_id": record.manifest.document_id,
|
||||
"source_label": record.manifest.source_label,
|
||||
"status": record.manifest.status,
|
||||
"input_sha256": record.manifest.input_sha256,
|
||||
"current_sha256": record.manifest.current_sha256,
|
||||
"change_count": record.manifest.change_count,
|
||||
"source_available": original is not None,
|
||||
"output_available": cleaned is not None,
|
||||
"availability_error": source_error or output_error,
|
||||
}
|
||||
|
||||
def _summary(self, record: DocumentRecord) -> JsonObject:
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
return self._summary_json(record, original, source_error, cleaned, output_error)
|
||||
|
||||
def _component_summaries(self, changes: tuple[ChangeRecord, ...]) -> list[JsonValue]:
|
||||
counts = [0] * len(self.components)
|
||||
for change in changes:
|
||||
if 0 <= change.replay.component_position < len(counts):
|
||||
counts[change.replay.component_position] += 1
|
||||
return [_component_json(item, position, counts[position]) for position, item in enumerate(self.components)]
|
||||
|
||||
@staticmethod
|
||||
def _change_json(recorded: ChangeRecord, located: LocatedChange | None) -> JsonObject:
|
||||
change = recorded.replay
|
||||
return {
|
||||
"component_id": change.component_id,
|
||||
"component_version": change.component_version,
|
||||
"component_position": change.component_position,
|
||||
"proposal_ref": {
|
||||
"component_position": change.proposal_component_position,
|
||||
"snapshot_sha256": change.proposal_snapshot_sha256,
|
||||
"proposal_index": change.proposal_index,
|
||||
},
|
||||
"edit_index": change.edit_index,
|
||||
"reason": recorded.reason,
|
||||
"location": {
|
||||
"line": recorded.recorded_line if located is None else located.line,
|
||||
"column": recorded.recorded_column if located is None else located.column,
|
||||
},
|
||||
"editor_range": (
|
||||
None if located is None else {"start": located.editor_start, "end": located.editor_end}
|
||||
),
|
||||
"before": change.before,
|
||||
"after": change.after,
|
||||
}
|
||||
|
||||
def run_summary(self) -> JsonObject:
|
||||
all_changes = tuple(change for document in self.documents for change in document.result.changes)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": self.run_json,
|
||||
"components": self._component_summaries(all_changes),
|
||||
"documents": [self._summary(item) for item in self.documents],
|
||||
"summary": self.summary_json,
|
||||
"original_run_location_changed": self.original_run_location_changed,
|
||||
}
|
||||
|
||||
def _document(self, document_id: str) -> DocumentRecord:
|
||||
record = self.documents_by_id.get(document_id)
|
||||
if record is None:
|
||||
_fail("unknown_document", "文档不存在。", 404)
|
||||
return record
|
||||
|
||||
def document_comparison(self, document_id: str) -> JsonObject:
|
||||
record = self._document(document_id)
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
replayed = None if original is None else self._replay(record, original, cleaned)
|
||||
located = () if replayed is None else replayed.changes
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"document": self._summary_json(record, original, source_error, cleaned, output_error),
|
||||
"components": self._component_summaries(record.result.changes),
|
||||
"original_markdown": original,
|
||||
"cleaned_markdown": cleaned,
|
||||
"changes": [
|
||||
self._change_json(change, located[position] if position < len(located) else None)
|
||||
for position, change in enumerate(record.result.changes)
|
||||
],
|
||||
"errors": list(record.result.errors),
|
||||
"residual_proposals": list(record.result.residual_proposals),
|
||||
}
|
||||
|
||||
def component_stage(self, document_id: str, component_position: int) -> JsonObject:
|
||||
record = self._document(document_id)
|
||||
if record.manifest.status != "success":
|
||||
_fail("stage_unavailable", "只有 success 文档具有完整组件阶段。", 409)
|
||||
if component_position < 0 or component_position >= len(self.components):
|
||||
_fail("unknown_component", "组件位置不存在。", 404)
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
if original is None or cleaned is None:
|
||||
_fail("comparison_unavailable", source_error or output_error or "组件阶段不可用。", 409)
|
||||
replayed = self._replay(record, original, cleaned)
|
||||
stage = replayed.stages[component_position]
|
||||
component = self.components[component_position]
|
||||
recorded_changes = tuple(
|
||||
item for item in record.result.changes if item.replay.component_position == component_position
|
||||
)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"document_id": record.manifest.document_id,
|
||||
"component": _component_json(component, component_position, len(stage.changes)),
|
||||
"before_sha256": stage.before_sha256,
|
||||
"after_sha256": stage.after_sha256,
|
||||
"before_markdown": stage.before_markdown,
|
||||
"after_markdown": stage.after_markdown,
|
||||
"changes": [
|
||||
self._change_json(change, located)
|
||||
for change, located in zip(recorded_changes, stage.changes, strict=True)
|
||||
],
|
||||
}
|
||||
@@ -1,378 +0,0 @@
|
||||
import { lazy, Suspense, useEffect, useMemo, useState } from "react";
|
||||
|
||||
import type {
|
||||
ChangeDetail,
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunStatus,
|
||||
RunSummaryResponse,
|
||||
} from "../shared/api.js";
|
||||
import { fetchComponentStage, fetchDocument, fetchRun } from "./api-client.js";
|
||||
|
||||
const DiffView = lazy(async () => {
|
||||
const module = await import("./DiffView.js");
|
||||
return { default: module.DiffView };
|
||||
});
|
||||
|
||||
interface AsyncState<T> {
|
||||
loading: boolean;
|
||||
value: T | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
const emptyState = <T,>(): AsyncState<T> => ({ loading: true, value: null, error: null });
|
||||
|
||||
function statusLabel(status: RunStatus): string {
|
||||
switch (status) {
|
||||
case "success":
|
||||
return "成功";
|
||||
case "failed":
|
||||
return "失败";
|
||||
case "unstable":
|
||||
return "不稳定";
|
||||
}
|
||||
}
|
||||
|
||||
function shortHash(hash: string): string {
|
||||
return `${hash.slice(0, 8)}…${hash.slice(-6)}`;
|
||||
}
|
||||
|
||||
function ErrorPanel({ message }: { message: string }) {
|
||||
return (
|
||||
<div className="state-panel state-panel--error" role="alert">
|
||||
<span className="eyebrow">无法显示</span>
|
||||
<p>{message}</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function LoadingPanel() {
|
||||
return (
|
||||
<div className="state-panel" role="status">
|
||||
<span className="loading-dot" />
|
||||
<p>正在校验本地运行产物…</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ChangeList({
|
||||
changes,
|
||||
onSelect,
|
||||
canJump,
|
||||
}: {
|
||||
changes: ChangeDetail[];
|
||||
onSelect: (change: ChangeDetail) => void;
|
||||
canJump: boolean;
|
||||
}) {
|
||||
if (changes.length === 0) {
|
||||
return <p className="quiet-message">这个组件运行过，但没有修改当前文档。</p>;
|
||||
}
|
||||
return (
|
||||
<ol className="change-list">
|
||||
{changes.map((change) => (
|
||||
<li key={`${change.proposal_ref.snapshot_sha256}-${change.proposal_ref.proposal_index}-${change.edit_index}`}>
|
||||
<button type="button" onClick={() => onSelect(change)} disabled={!canJump}>
|
||||
<span className="change-location">
|
||||
第 {change.location.line} 行，第 {change.location.column} 列 · 候选
|
||||
{change.proposal_ref.proposal_index + 1} / 编辑 {change.edit_index + 1}
|
||||
</span>
|
||||
<strong>{change.reason}</strong>
|
||||
<span className="change-sample">
|
||||
<del>{change.before || "∅"}</del>
|
||||
<span aria-hidden="true">→</span>
|
||||
<ins>{change.after || "∅"}</ins>
|
||||
</span>
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
);
|
||||
}
|
||||
|
||||
export function App() {
|
||||
const [runState, setRunState] = useState<AsyncState<RunSummaryResponse>>(emptyState);
|
||||
const [selectedDocument, setSelectedDocument] = useState<string | null>(null);
|
||||
const [documentState, setDocumentState] = useState<AsyncState<DocumentComparisonResponse>>({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: null,
|
||||
});
|
||||
const [selectedComponent, setSelectedComponent] = useState<number | null>(null);
|
||||
const [stageState, setStageState] = useState<AsyncState<ComponentStageResponse>>({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: null,
|
||||
});
|
||||
const [focusRange, setFocusRange] = useState<{ start: number; end: number } | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
fetchRun(controller.signal)
|
||||
.then((run) => {
|
||||
setRunState({ loading: false, value: run, error: null });
|
||||
setSelectedDocument(run.documents[0]?.document_id ?? null);
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setRunState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取运行摘要。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
setSelectedComponent(null);
|
||||
setFocusRange(null);
|
||||
if (selectedDocument === null) {
|
||||
setDocumentState({ loading: false, value: null, error: null });
|
||||
return undefined;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
setDocumentState(emptyState());
|
||||
fetchDocument(selectedDocument, controller.signal)
|
||||
.then((document) => setDocumentState({ loading: false, value: document, error: null }))
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setDocumentState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取文档。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [selectedDocument]);
|
||||
|
||||
useEffect(() => {
|
||||
if (selectedDocument === null || selectedComponent === null) {
|
||||
setStageState({ loading: false, value: null, error: null });
|
||||
return undefined;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
setStageState(emptyState());
|
||||
fetchComponentStage(selectedDocument, selectedComponent, controller.signal)
|
||||
.then((stage) => setStageState({ loading: false, value: stage, error: null }))
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setStageState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取组件阶段。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [selectedDocument, selectedComponent]);
|
||||
|
||||
const visibleChanges = useMemo(() => {
|
||||
const document = documentState.value;
|
||||
if (document === null) {
|
||||
return [];
|
||||
}
|
||||
if (selectedComponent === null) {
|
||||
return document.changes;
|
||||
}
|
||||
return document.changes.filter((change) => change.component_position === selectedComponent);
|
||||
}, [documentState.value, selectedComponent]);
|
||||
|
||||
const selectChange = (change: ChangeDetail): void => {
|
||||
setSelectedComponent(change.component_position);
|
||||
setFocusRange(change.editor_range);
|
||||
};
|
||||
|
||||
if (runState.loading) {
|
||||
return <LoadingPanel />;
|
||||
}
|
||||
if (runState.error !== null || runState.value === null) {
|
||||
return <ErrorPanel message={runState.error ?? "运行摘要为空。"} />;
|
||||
}
|
||||
|
||||
const run = runState.value;
|
||||
const document = documentState.value;
|
||||
const selectedSummary = run.documents.find((item) => item.document_id === selectedDocument);
|
||||
const selectedStage = stageState.value;
|
||||
const canCompare =
|
||||
document?.document.status === "success" &&
|
||||
document.document.source_available &&
|
||||
document.document.output_available &&
|
||||
document.original_markdown !== null &&
|
||||
document.cleaned_markdown !== null;
|
||||
const beforeText = selectedStage?.before_markdown ?? document?.original_markdown ?? "";
|
||||
const afterText = selectedStage?.after_markdown ?? document?.cleaned_markdown ?? "";
|
||||
const beforeLabel = selectedStage === null ? "清洗前" : `组件 ${selectedStage.component.component_position + 1} 执行前`;
|
||||
const afterLabel = selectedStage === null ? "清洗后" : `组件 ${selectedStage.component.component_position + 1} 执行后`;
|
||||
|
||||
return (
|
||||
<div className="app-shell">
|
||||
<header className="topbar">
|
||||
<div>
|
||||
<span className="brand-mark">md</span>
|
||||
<div>
|
||||
<p className="eyebrow">本地清洗评审器</p>
|
||||
<h1>{run.run.run_id}</h1>
|
||||
</div>
|
||||
</div>
|
||||
<div className="run-facts">
|
||||
<span className={`status status--${run.run.status}`}>{statusLabel(run.run.status)}</span>
|
||||
<span>{run.summary.document_count} 份文档</span>
|
||||
<span>{run.summary.change_count} 条修改</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
{run.original_run_location_changed ? (
|
||||
<div className="notice" role="status">
|
||||
这次运行目录已被移动；评审器使用你本次指定的目录读取产物。
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="layout">
|
||||
<aside className="sidebar" aria-label="运行导航">
|
||||
<section>
|
||||
<div className="section-heading">
|
||||
<h2>文档</h2>
|
||||
<span>{run.documents.length}</span>
|
||||
</div>
|
||||
<nav className="document-list" aria-label="文档列表">
|
||||
{run.documents.map((item) => (
|
||||
<button
|
||||
type="button"
|
||||
key={item.document_id}
|
||||
className={item.document_id === selectedDocument ? "is-active" : ""}
|
||||
onClick={() => setSelectedDocument(item.document_id)}
|
||||
aria-current={item.document_id === selectedDocument ? "page" : undefined}
|
||||
>
|
||||
<span className={`status-dot status-dot--${item.status}`} />
|
||||
<span>
|
||||
<strong>{item.source_label}</strong>
|
||||
<small>{item.change_count} 条修改</small>
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</nav>
|
||||
</section>
|
||||
|
||||
<section className="component-section">
|
||||
<div className="section-heading">
|
||||
<h2>组件时间线</h2>
|
||||
<button
|
||||
type="button"
|
||||
className="text-button"
|
||||
onClick={() => {
|
||||
setSelectedComponent(null);
|
||||
setFocusRange(null);
|
||||
}}
|
||||
disabled={selectedComponent === null}
|
||||
>
|
||||
查看总结果
|
||||
</button>
|
||||
</div>
|
||||
<ol className="component-list">
|
||||
{(document?.components ?? run.components).map((component) => (
|
||||
<li key={component.component_id}>
|
||||
<button
|
||||
type="button"
|
||||
className={component.component_position === selectedComponent ? "is-active" : ""}
|
||||
onClick={() => {
|
||||
setSelectedComponent(component.component_position);
|
||||
setFocusRange(null);
|
||||
}}
|
||||
disabled={!canCompare}
|
||||
>
|
||||
<span className="component-index">{component.component_position + 1}</span>
|
||||
<span>
|
||||
<strong>{component.component_id}</strong>
|
||||
<small>
|
||||
v{component.version} · {component.change_count} 条
|
||||
</small>
|
||||
</span>
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
</section>
|
||||
</aside>
|
||||
|
||||
<main className="workspace">
|
||||
<section className="document-header">
|
||||
<div>
|
||||
<p className="eyebrow">当前文档</p>
|
||||
<h2>{selectedSummary?.source_label ?? "未选择"}</h2>
|
||||
</div>
|
||||
{selectedSummary === undefined ? null : (
|
||||
<div className="document-meta">
|
||||
<span className={`status status--${selectedSummary.status}`}>
|
||||
{statusLabel(selectedSummary.status)}
|
||||
</span>
|
||||
<span title={selectedSummary.input_sha256}>输入 {shortHash(selectedSummary.input_sha256)}</span>
|
||||
<span title={selectedSummary.current_sha256}>当前 {shortHash(selectedSummary.current_sha256)}</span>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
{documentState.loading || stageState.loading ? <LoadingPanel /> : null}
|
||||
{documentState.error !== null ? <ErrorPanel message={documentState.error} /> : null}
|
||||
{stageState.error !== null ? <ErrorPanel message={stageState.error} /> : null}
|
||||
|
||||
{!documentState.loading && document !== null && document.document.status === "success" && !canCompare ? (
|
||||
<ErrorPanel message={document.document.availability_error ?? "完整原文或清洗结果不可用。"} />
|
||||
) : null}
|
||||
|
||||
{!documentState.loading && document !== null && document.document.status !== "success" ? (
|
||||
<div className="diagnostic-panel">
|
||||
<p className="eyebrow">没有正式清洗结果</p>
|
||||
<h3>{statusLabel(document.document.status)}文档只展示审计证据</h3>
|
||||
<p>
|
||||
该状态不会生成 <code>cleaned.md</code>，因此这里不构造完整部分输出。
|
||||
</p>
|
||||
{document.errors.map((error) => (
|
||||
<article key={`${error.component_position}-${error.stage}-${error.error_type}`}>
|
||||
<strong>{error.error_type}</strong>
|
||||
<span>{error.message}</span>
|
||||
</article>
|
||||
))}
|
||||
{document.residual_proposals.map((proposal) => (
|
||||
<article key={`${proposal.component_position}-${proposal.reason}`}>
|
||||
<strong>最终复查仍有 {proposal.edit_count} 项候选</strong>
|
||||
<span>{proposal.reason}</span>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{!documentState.loading && !stageState.loading && canCompare && stageState.error === null ? (
|
||||
<Suspense fallback={<LoadingPanel />}>
|
||||
<DiffView
|
||||
before={beforeText}
|
||||
after={afterText}
|
||||
beforeLabel={beforeLabel}
|
||||
afterLabel={afterLabel}
|
||||
focusRange={focusRange}
|
||||
/>
|
||||
</Suspense>
|
||||
) : null}
|
||||
|
||||
{document !== null ? (
|
||||
<section className="changes-panel" aria-label="修改详情">
|
||||
<div className="changes-heading">
|
||||
<div>
|
||||
<p className="eyebrow">实际修改</p>
|
||||
<h3>
|
||||
{selectedComponent === null
|
||||
? `全部组件 · ${visibleChanges.length} 条`
|
||||
: `${document.components[selectedComponent]?.component_id ?? "组件"} · ${visibleChanges.length} 条`}
|
||||
</h3>
|
||||
</div>
|
||||
{selectedStage === null ? null : <p>{selectedStage.component.applicability}</p>}
|
||||
</div>
|
||||
<ChangeList changes={visibleChanges} onSelect={selectChange} canJump={canCompare} />
|
||||
</section>
|
||||
) : null}
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
import { markdown } from "@codemirror/lang-markdown";
|
||||
import { MergeView } from "@codemirror/merge";
|
||||
import { EditorSelection, EditorState } from "@codemirror/state";
|
||||
import { drawSelection, EditorView, lineNumbers } from "@codemirror/view";
|
||||
import { useEffect, useRef } from "react";
|
||||
|
||||
interface DiffViewProps {
|
||||
before: string;
|
||||
after: string;
|
||||
beforeLabel: string;
|
||||
afterLabel: string;
|
||||
focusRange?: { start: number; end: number } | null;
|
||||
}
|
||||
|
||||
const editorTheme = EditorView.theme({
|
||||
"&": {
|
||||
height: "100%",
|
||||
backgroundColor: "#fbfaf7",
|
||||
color: "#262822",
|
||||
fontSize: "13px",
|
||||
},
|
||||
".cm-scroller": {
|
||||
fontFamily: '"SFMono-Regular", Consolas, "Liberation Mono", monospace',
|
||||
lineHeight: "1.68",
|
||||
},
|
||||
".cm-gutters": {
|
||||
backgroundColor: "#f2f0ea",
|
||||
color: "#8a877e",
|
||||
border: "none",
|
||||
},
|
||||
".cm-content": {
|
||||
padding: "18px 0 36px",
|
||||
},
|
||||
".cm-line": {
|
||||
padding: "0 14px",
|
||||
},
|
||||
"&.cm-focused": {
|
||||
outline: "2px solid #a7b9ac",
|
||||
outlineOffset: "-2px",
|
||||
},
|
||||
});
|
||||
|
||||
const readOnlyExtensions = [
|
||||
lineNumbers(),
|
||||
markdown(),
|
||||
EditorState.readOnly.of(true),
|
||||
EditorView.editable.of(false),
|
||||
drawSelection(),
|
||||
EditorView.lineWrapping,
|
||||
editorTheme,
|
||||
];
|
||||
|
||||
export function DiffView({ before, after, beforeLabel, afterLabel, focusRange }: DiffViewProps) {
|
||||
const host = useRef<HTMLDivElement>(null);
|
||||
const merge = useRef<MergeView | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (host.current === null) {
|
||||
return undefined;
|
||||
}
|
||||
const view = new MergeView({
|
||||
parent: host.current,
|
||||
a: { doc: before, extensions: readOnlyExtensions },
|
||||
b: { doc: after, extensions: readOnlyExtensions },
|
||||
orientation: "a-b",
|
||||
gutter: true,
|
||||
highlightChanges: true,
|
||||
});
|
||||
merge.current = view;
|
||||
return () => {
|
||||
view.destroy();
|
||||
merge.current = null;
|
||||
};
|
||||
}, [before, after]);
|
||||
|
||||
// 切换组件时 stage 文本先到、旧视图上的提前滚动会被重建覆盖，所以聚焦
|
||||
// 必须等 MergeView 换成目标文本之后再执行。
|
||||
useEffect(() => {
|
||||
const view = merge.current;
|
||||
if (view === null || focusRange === null || focusRange === undefined) {
|
||||
return;
|
||||
}
|
||||
const anchor = Math.min(Math.max(focusRange.start, 0), view.a.state.doc.length);
|
||||
const head = Math.min(Math.max(focusRange.end, anchor), view.a.state.doc.length);
|
||||
view.a.dispatch({
|
||||
selection: EditorSelection.range(anchor, head),
|
||||
effects: EditorView.scrollIntoView(anchor, { y: "center" }),
|
||||
});
|
||||
view.a.focus();
|
||||
}, [focusRange, before, after]);
|
||||
|
||||
return (
|
||||
<section className="diff-shell" aria-label={`${beforeLabel}与${afterLabel}对比`}>
|
||||
<div className="diff-labels" aria-hidden="true">
|
||||
<span>{beforeLabel}</span>
|
||||
<span>{afterLabel}</span>
|
||||
</div>
|
||||
<div className="diff-host" ref={host} />
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -1,288 +0,0 @@
|
||||
import type {
|
||||
ApiErrorResponse,
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunSummaryResponse,
|
||||
} from "../shared/api.js";
|
||||
|
||||
export class ReviewerApiError extends Error {
|
||||
readonly code: string;
|
||||
|
||||
constructor(code: string, message: string) {
|
||||
super(message);
|
||||
this.name = "ReviewerApiError";
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
type JsonRecord = Record<string, unknown>;
|
||||
|
||||
function invalid(label: string): never {
|
||||
throw new ReviewerApiError("invalid_response", `本地服务返回的 ${label} 格式不正确。`);
|
||||
}
|
||||
|
||||
function record(value: unknown, label: string): JsonRecord {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value as JsonRecord;
|
||||
}
|
||||
|
||||
function array(value: unknown, label: string): unknown[] {
|
||||
if (!Array.isArray(value)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function string(value: unknown, label: string): string {
|
||||
if (typeof value !== "string") {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function integer(value: unknown, label: string, minimum = 0): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
function boolean(value: unknown, label: string): boolean {
|
||||
if (typeof value !== "boolean") {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function nullableString(value: unknown, label: string): string | null {
|
||||
return value === null ? null : string(value, label);
|
||||
}
|
||||
|
||||
function hash(value: unknown, label: string): string {
|
||||
const digest = string(value, label);
|
||||
if (!/^[0-9a-f]{64}$/.test(digest)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return digest;
|
||||
}
|
||||
|
||||
function status(value: unknown): "success" | "failed" | "unstable" {
|
||||
if (value !== "success" && value !== "failed" && value !== "unstable") {
|
||||
return invalid("status");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function component(value: unknown): RunSummaryResponse["components"][number] {
|
||||
const item = record(value, "component");
|
||||
return {
|
||||
component_position: integer(item.component_position, "component_position"),
|
||||
component_id: string(item.component_id, "component_id"),
|
||||
version: string(item.version, "component version"),
|
||||
parameters: item.parameters,
|
||||
applicability: string(item.applicability, "component applicability"),
|
||||
change_count: integer(item.change_count, "component change_count"),
|
||||
};
|
||||
}
|
||||
|
||||
function documentSummary(value: unknown): RunSummaryResponse["documents"][number] {
|
||||
const item = record(value, "document summary");
|
||||
return {
|
||||
document_id: string(item.document_id, "document_id"),
|
||||
source_label: string(item.source_label, "source_label"),
|
||||
status: status(item.status),
|
||||
input_sha256: hash(item.input_sha256, "input_sha256"),
|
||||
current_sha256: hash(item.current_sha256, "current_sha256"),
|
||||
change_count: integer(item.change_count, "document change_count"),
|
||||
source_available: boolean(item.source_available, "source_available"),
|
||||
output_available: boolean(item.output_available, "output_available"),
|
||||
availability_error: nullableString(item.availability_error, "availability_error"),
|
||||
};
|
||||
}
|
||||
|
||||
function summary(value: unknown): RunSummaryResponse["summary"] {
|
||||
const item = record(value, "summary");
|
||||
return {
|
||||
document_count: integer(item.document_count, "document_count"),
|
||||
success_count: integer(item.success_count, "success_count"),
|
||||
failed_count: integer(item.failed_count, "failed_count"),
|
||||
unstable_count: integer(item.unstable_count, "unstable_count"),
|
||||
change_count: integer(item.change_count, "change_count"),
|
||||
};
|
||||
}
|
||||
|
||||
function change(value: unknown): DocumentComparisonResponse["changes"][number] {
|
||||
const item = record(value, "change");
|
||||
const proposal = record(item.proposal_ref, "proposal_ref");
|
||||
const location = record(item.location, "location");
|
||||
const editorValue = item.editor_range;
|
||||
const editorRange =
|
||||
editorValue === null
|
||||
? null
|
||||
: (() => {
|
||||
const editor = record(editorValue, "editor_range");
|
||||
const start = integer(editor.start, "editor_range.start");
|
||||
const end = integer(editor.end, "editor_range.end");
|
||||
if (end < start) {
|
||||
return invalid("editor_range");
|
||||
}
|
||||
return { start, end };
|
||||
})();
|
||||
return {
|
||||
component_id: string(item.component_id, "change component_id"),
|
||||
component_version: string(item.component_version, "change component_version"),
|
||||
component_position: integer(item.component_position, "change component_position"),
|
||||
proposal_ref: {
|
||||
component_position: integer(proposal.component_position, "proposal component_position"),
|
||||
snapshot_sha256: hash(proposal.snapshot_sha256, "proposal snapshot_sha256"),
|
||||
proposal_index: integer(proposal.proposal_index, "proposal_index"),
|
||||
},
|
||||
edit_index: integer(item.edit_index, "edit_index"),
|
||||
reason: string(item.reason, "reason"),
|
||||
location: {
|
||||
line: integer(location.line, "location.line", 1),
|
||||
column: integer(location.column, "location.column", 1),
|
||||
},
|
||||
editor_range: editorRange,
|
||||
before: string(item.before, "before"),
|
||||
after: string(item.after, "after"),
|
||||
};
|
||||
}
|
||||
|
||||
function runError(value: unknown): DocumentComparisonResponse["errors"][number] {
|
||||
const item = record(value, "run error");
|
||||
const stage = item.stage;
|
||||
if (stage !== "transform" && stage !== "final_review") {
|
||||
return invalid("error stage");
|
||||
}
|
||||
return {
|
||||
component_id: string(item.component_id, "error component_id"),
|
||||
component_version: string(item.component_version, "error component_version"),
|
||||
component_position: integer(item.component_position, "error component_position"),
|
||||
stage,
|
||||
error_type: string(item.error_type, "error_type"),
|
||||
message: string(item.message, "error message"),
|
||||
};
|
||||
}
|
||||
|
||||
function residual(value: unknown): DocumentComparisonResponse["residual_proposals"][number] {
|
||||
const item = record(value, "residual proposal");
|
||||
return {
|
||||
component_id: string(item.component_id, "residual component_id"),
|
||||
component_version: string(item.component_version, "residual component_version"),
|
||||
component_position: integer(item.component_position, "residual component_position"),
|
||||
reason: string(item.reason, "residual reason"),
|
||||
edit_count: integer(item.edit_count, "residual edit_count", 1),
|
||||
};
|
||||
}
|
||||
|
||||
function parseRun(value: unknown): RunSummaryResponse {
|
||||
const payload = record(value, "run response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("run schema_version");
|
||||
}
|
||||
const run = record(payload.run, "run");
|
||||
return {
|
||||
schema_version: 1,
|
||||
run: {
|
||||
run_id: string(run.run_id, "run_id"),
|
||||
run_date: string(run.run_date, "run_date"),
|
||||
status: status(run.status),
|
||||
started_at_utc: string(run.started_at_utc, "started_at_utc"),
|
||||
completed_at_utc: string(run.completed_at_utc, "completed_at_utc"),
|
||||
retention_until: string(run.retention_until, "retention_until"),
|
||||
},
|
||||
components: array(payload.components, "components").map(component),
|
||||
documents: array(payload.documents, "documents").map(documentSummary),
|
||||
summary: summary(payload.summary),
|
||||
original_run_location_changed: boolean(
|
||||
payload.original_run_location_changed,
|
||||
"original_run_location_changed",
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
function parseDocument(value: unknown): DocumentComparisonResponse {
|
||||
const payload = record(value, "document response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("document schema_version");
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
document: documentSummary(payload.document),
|
||||
components: array(payload.components, "components").map(component),
|
||||
original_markdown: nullableString(payload.original_markdown, "original_markdown"),
|
||||
cleaned_markdown: nullableString(payload.cleaned_markdown, "cleaned_markdown"),
|
||||
changes: array(payload.changes, "changes").map(change),
|
||||
errors: array(payload.errors, "errors").map(runError),
|
||||
residual_proposals: array(payload.residual_proposals, "residual_proposals").map(residual),
|
||||
};
|
||||
}
|
||||
|
||||
function parseStage(value: unknown): ComponentStageResponse {
|
||||
const payload = record(value, "component stage response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("component stage schema_version");
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
document_id: string(payload.document_id, "document_id"),
|
||||
component: component(payload.component),
|
||||
before_sha256: hash(payload.before_sha256, "before_sha256"),
|
||||
after_sha256: hash(payload.after_sha256, "after_sha256"),
|
||||
before_markdown: string(payload.before_markdown, "before_markdown"),
|
||||
after_markdown: string(payload.after_markdown, "after_markdown"),
|
||||
changes: array(payload.changes, "changes").map(change),
|
||||
};
|
||||
}
|
||||
|
||||
async function getJson<T>(
|
||||
pathname: string,
|
||||
parse: (payload: unknown) => T,
|
||||
signal?: AbortSignal,
|
||||
): Promise<T> {
|
||||
const response = await fetch(pathname, {
|
||||
method: "GET",
|
||||
cache: "no-store",
|
||||
credentials: "same-origin",
|
||||
signal,
|
||||
});
|
||||
const payload: unknown = await response.json();
|
||||
if (!response.ok) {
|
||||
const errorPayload = payload as Partial<ApiErrorResponse>;
|
||||
throw new ReviewerApiError(
|
||||
errorPayload.error?.code ?? "request_failed",
|
||||
errorPayload.error?.message ?? `请求失败（HTTP ${response.status}）。`,
|
||||
);
|
||||
}
|
||||
return parse(payload);
|
||||
}
|
||||
|
||||
export function fetchRun(signal?: AbortSignal): Promise<RunSummaryResponse> {
|
||||
return getJson("/api/v1/run", parseRun, signal);
|
||||
}
|
||||
|
||||
export function fetchDocument(
|
||||
documentId: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<DocumentComparisonResponse> {
|
||||
return getJson(
|
||||
`/api/v1/documents/${encodeURIComponent(documentId)}`,
|
||||
parseDocument,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
|
||||
export function fetchComponentStage(
|
||||
documentId: string,
|
||||
componentPosition: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<ComponentStageResponse> {
|
||||
return getJson(
|
||||
`/api/v1/documents/${encodeURIComponent(documentId)}/components/${componentPosition}`,
|
||||
parseStage,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
import { StrictMode } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
|
||||
import { App } from "./App.js";
|
||||
import "./styles.css";
|
||||
|
||||
const root = document.getElementById("root");
|
||||
if (root === null) {
|
||||
throw new Error("missing #root element");
|
||||
}
|
||||
|
||||
createRoot(root).render(
|
||||
<StrictMode>
|
||||
<App />
|
||||
</StrictMode>,
|
||||
);
|
||||
@@ -1,558 +0,0 @@
|
||||
:root {
|
||||
color: #252720;
|
||||
background: #ecebe5;
|
||||
font-family:
|
||||
Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Microsoft YaHei",
|
||||
sans-serif;
|
||||
font-synthesis: none;
|
||||
text-rendering: optimizeLegibility;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body,
|
||||
#root {
|
||||
min-width: 1180px;
|
||||
min-height: 100%;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
button {
|
||||
color: inherit;
|
||||
font: inherit;
|
||||
}
|
||||
|
||||
button:focus-visible {
|
||||
outline: 2px solid #315f4b;
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
.app-shell {
|
||||
min-height: 100vh;
|
||||
background:
|
||||
radial-gradient(circle at 12% 0%, rgb(255 255 255 / 72%), transparent 34%),
|
||||
#ecebe5;
|
||||
}
|
||||
|
||||
.topbar {
|
||||
position: sticky;
|
||||
z-index: 20;
|
||||
top: 0;
|
||||
display: flex;
|
||||
min-height: 76px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px 24px;
|
||||
border-bottom: 1px solid #d7d5cd;
|
||||
background: rgb(248 247 242 / 94%);
|
||||
backdrop-filter: blur(18px);
|
||||
}
|
||||
|
||||
.topbar > div:first-child {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.brand-mark {
|
||||
display: grid;
|
||||
width: 42px;
|
||||
height: 42px;
|
||||
place-items: center;
|
||||
border-radius: 12px;
|
||||
background: #284d3d;
|
||||
color: #f3f4ed;
|
||||
font-family: Georgia, serif;
|
||||
font-size: 19px;
|
||||
letter-spacing: -0.08em;
|
||||
}
|
||||
|
||||
.eyebrow {
|
||||
margin: 0 0 3px;
|
||||
color: #78796f;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.12em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
h1,
|
||||
h2,
|
||||
h3,
|
||||
p {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
.topbar h1 {
|
||||
margin: 0;
|
||||
font-family: Georgia, "Songti SC", serif;
|
||||
font-size: 19px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.run-facts,
|
||||
.document-meta {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
color: #66685f;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.run-facts > span:not(.status),
|
||||
.document-meta > span:not(.status) {
|
||||
padding-left: 10px;
|
||||
border-left: 1px solid #d5d2c9;
|
||||
}
|
||||
|
||||
.status {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
border: 1px solid currentColor;
|
||||
border-radius: 999px;
|
||||
padding: 3px 8px;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.status--success {
|
||||
color: #277052;
|
||||
background: #edf6ef;
|
||||
}
|
||||
|
||||
.status--failed {
|
||||
color: #a04338;
|
||||
background: #fff0ed;
|
||||
}
|
||||
|
||||
.status--unstable {
|
||||
color: #986617;
|
||||
background: #fff7df;
|
||||
}
|
||||
|
||||
.notice {
|
||||
padding: 9px 24px;
|
||||
border-bottom: 1px solid #e6d09c;
|
||||
background: #fff7de;
|
||||
color: #77561d;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.layout {
|
||||
display: grid;
|
||||
min-height: calc(100vh - 76px);
|
||||
grid-template-columns: 300px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
position: sticky;
|
||||
top: 76px;
|
||||
overflow-y: auto;
|
||||
height: calc(100vh - 76px);
|
||||
border-right: 1px solid #d7d5cd;
|
||||
background: #f7f6f1;
|
||||
}
|
||||
|
||||
.sidebar section {
|
||||
padding: 20px 16px;
|
||||
}
|
||||
|
||||
.sidebar section + section {
|
||||
border-top: 1px solid #dfddd5;
|
||||
}
|
||||
|
||||
.section-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.section-heading h2 {
|
||||
margin: 0;
|
||||
font-size: 12px;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.section-heading > span {
|
||||
color: #888980;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.document-list,
|
||||
.component-list {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.document-list button,
|
||||
.component-list button {
|
||||
display: grid;
|
||||
width: 100%;
|
||||
align-items: center;
|
||||
border: 0;
|
||||
border-radius: 9px;
|
||||
background: transparent;
|
||||
cursor: pointer;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.document-list button {
|
||||
grid-template-columns: 9px 1fr;
|
||||
gap: 10px;
|
||||
padding: 9px 10px;
|
||||
}
|
||||
|
||||
.document-list button:hover,
|
||||
.component-list button:hover:not(:disabled) {
|
||||
background: #eceae2;
|
||||
}
|
||||
|
||||
.document-list button.is-active,
|
||||
.component-list button.is-active {
|
||||
background: #e0e8e0;
|
||||
color: #234b39;
|
||||
}
|
||||
|
||||
.document-list strong,
|
||||
.component-list strong {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
font-size: 12px;
|
||||
font-weight: 650;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.document-list small,
|
||||
.component-list small {
|
||||
display: block;
|
||||
margin-top: 3px;
|
||||
color: #7d7e75;
|
||||
font-size: 10px;
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 7px;
|
||||
height: 7px;
|
||||
border-radius: 50%;
|
||||
background: #999;
|
||||
}
|
||||
|
||||
.status-dot--success {
|
||||
background: #348361;
|
||||
}
|
||||
|
||||
.status-dot--failed {
|
||||
background: #b64b3f;
|
||||
}
|
||||
|
||||
.status-dot--unstable {
|
||||
background: #bd831c;
|
||||
}
|
||||
|
||||
.text-button {
|
||||
border: 0;
|
||||
background: transparent;
|
||||
color: #315f4b;
|
||||
cursor: pointer;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.text-button:disabled {
|
||||
color: #aaa99f;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.component-list {
|
||||
counter-reset: components;
|
||||
}
|
||||
|
||||
.component-list button {
|
||||
grid-template-columns: 26px minmax(0, 1fr);
|
||||
gap: 8px;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
.component-list button:disabled {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.55;
|
||||
}
|
||||
|
||||
.component-index {
|
||||
display: grid;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
place-items: center;
|
||||
border: 1px solid #d3d1c8;
|
||||
border-radius: 50%;
|
||||
color: #74766e;
|
||||
font-family: Georgia, serif;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.workspace {
|
||||
display: grid;
|
||||
min-width: 0;
|
||||
align-content: start;
|
||||
gap: 14px;
|
||||
padding: 18px 20px 30px;
|
||||
}
|
||||
|
||||
.document-header {
|
||||
display: flex;
|
||||
align-items: end;
|
||||
justify-content: space-between;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.document-header h2 {
|
||||
margin: 0;
|
||||
font-family: Georgia, "Songti SC", serif;
|
||||
font-size: 21px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.diff-shell,
|
||||
.changes-panel,
|
||||
.diagnostic-panel,
|
||||
.state-panel {
|
||||
overflow: hidden;
|
||||
border: 1px solid #d5d3ca;
|
||||
border-radius: 13px;
|
||||
background: #fbfaf7;
|
||||
box-shadow: 0 12px 36px rgb(55 57 48 / 7%);
|
||||
}
|
||||
|
||||
.diff-shell {
|
||||
min-height: 510px;
|
||||
}
|
||||
|
||||
.diff-labels {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
border-bottom: 1px solid #dcdbd3;
|
||||
background: #f4f2ec;
|
||||
color: #6f7168;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.diff-labels span {
|
||||
padding: 9px 14px;
|
||||
}
|
||||
|
||||
.diff-labels span + span {
|
||||
border-left: 1px solid #dcdbd3;
|
||||
}
|
||||
|
||||
.diff-host {
|
||||
height: 510px;
|
||||
}
|
||||
|
||||
.diff-host > .cm-mergeView {
|
||||
height: 100%;
|
||||
overflow-y: auto;
|
||||
overscroll-behavior: contain;
|
||||
}
|
||||
|
||||
.diff-host .cm-mergeViewEditors {
|
||||
min-height: 100%;
|
||||
}
|
||||
|
||||
.diff-host .cm-editor {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.changes-panel {
|
||||
min-height: 120px;
|
||||
}
|
||||
|
||||
.changes-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 24px;
|
||||
padding: 15px 18px;
|
||||
border-bottom: 1px solid #e0ded6;
|
||||
}
|
||||
|
||||
.changes-heading h3 {
|
||||
margin: 0;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.changes-heading > p {
|
||||
max-width: 58%;
|
||||
margin: 0;
|
||||
color: #77786f;
|
||||
font-size: 11px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.change-list {
|
||||
display: grid;
|
||||
max-height: 310px;
|
||||
gap: 1px;
|
||||
overflow-y: auto;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background: #e4e2da;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.change-list button {
|
||||
display: grid;
|
||||
width: 100%;
|
||||
grid-template-columns: 145px minmax(240px, 1fr) minmax(260px, 0.9fr);
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
border: 0;
|
||||
padding: 11px 18px;
|
||||
background: #fbfaf7;
|
||||
cursor: pointer;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.change-list button:hover {
|
||||
background: #f4f5ef;
|
||||
}
|
||||
|
||||
.change-list button:disabled {
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.change-location {
|
||||
color: #73756c;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.change-list strong {
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.change-sample {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.change-sample del,
|
||||
.change-sample ins {
|
||||
overflow: hidden;
|
||||
max-width: 46%;
|
||||
border-radius: 4px;
|
||||
padding: 2px 5px;
|
||||
text-decoration: none;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.change-sample del {
|
||||
background: #f9ded9;
|
||||
color: #913e34;
|
||||
}
|
||||
|
||||
.change-sample ins {
|
||||
background: #dcecdf;
|
||||
color: #276348;
|
||||
}
|
||||
|
||||
.quiet-message {
|
||||
margin: 0;
|
||||
padding: 22px 18px;
|
||||
color: #77786f;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.state-panel,
|
||||
.diagnostic-panel {
|
||||
padding: 28px;
|
||||
}
|
||||
|
||||
.state-panel {
|
||||
display: grid;
|
||||
min-height: 180px;
|
||||
place-items: center;
|
||||
align-content: center;
|
||||
color: #66685f;
|
||||
}
|
||||
|
||||
.state-panel p {
|
||||
margin: 8px 0 0;
|
||||
}
|
||||
|
||||
.state-panel--error {
|
||||
border-color: #e2b6ae;
|
||||
color: #8f3e34;
|
||||
}
|
||||
|
||||
.loading-dot {
|
||||
width: 11px;
|
||||
height: 11px;
|
||||
border-radius: 50%;
|
||||
background: #3d735b;
|
||||
box-shadow: 0 0 0 7px #dce9df;
|
||||
animation: pulse 1.25s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.diagnostic-panel h3 {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.diagnostic-panel > p:not(.eyebrow) {
|
||||
color: #686a61;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.diagnostic-panel article {
|
||||
display: grid;
|
||||
grid-template-columns: 220px 1fr;
|
||||
gap: 12px;
|
||||
padding: 10px 0;
|
||||
border-top: 1px solid #e0ded6;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
code {
|
||||
border-radius: 4px;
|
||||
padding: 1px 4px;
|
||||
background: #eceae3;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 0.45;
|
||||
transform: scale(0.82);
|
||||
}
|
||||
|
||||
50% {
|
||||
opacity: 1;
|
||||
transform: scale(1);
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 1280px) {
|
||||
.layout {
|
||||
grid-template-columns: 270px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.change-list button {
|
||||
grid-template-columns: 125px minmax(180px, 1fr) minmax(220px, 0.8fr);
|
||||
}
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
export type RunStatus = "success" | "failed" | "unstable";
|
||||
|
||||
export interface ComponentSummary {
|
||||
component_position: number;
|
||||
component_id: string;
|
||||
version: string;
|
||||
parameters: unknown;
|
||||
applicability: string;
|
||||
change_count: number;
|
||||
}
|
||||
|
||||
export interface DocumentSummary {
|
||||
document_id: string;
|
||||
source_label: string;
|
||||
status: RunStatus;
|
||||
input_sha256: string;
|
||||
current_sha256: string;
|
||||
change_count: number;
|
||||
source_available: boolean;
|
||||
output_available: boolean;
|
||||
availability_error: string | null;
|
||||
}
|
||||
|
||||
export interface RunSummaryResponse {
|
||||
schema_version: 1;
|
||||
run: {
|
||||
run_id: string;
|
||||
run_date: string;
|
||||
status: RunStatus;
|
||||
started_at_utc: string;
|
||||
completed_at_utc: string;
|
||||
retention_until: string;
|
||||
};
|
||||
components: ComponentSummary[];
|
||||
documents: DocumentSummary[];
|
||||
summary: {
|
||||
document_count: number;
|
||||
success_count: number;
|
||||
failed_count: number;
|
||||
unstable_count: number;
|
||||
change_count: number;
|
||||
};
|
||||
original_run_location_changed: boolean;
|
||||
}
|
||||
|
||||
export interface ChangeDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
proposal_ref: {
|
||||
component_position: number;
|
||||
snapshot_sha256: string;
|
||||
proposal_index: number;
|
||||
};
|
||||
edit_index: number;
|
||||
reason: string;
|
||||
location: {
|
||||
line: number;
|
||||
column: number;
|
||||
};
|
||||
editor_range: {
|
||||
start: number;
|
||||
end: number;
|
||||
} | null;
|
||||
before: string;
|
||||
after: string;
|
||||
}
|
||||
|
||||
export interface RunErrorDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
stage: "transform" | "final_review";
|
||||
error_type: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export interface ResidualProposalDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
reason: string;
|
||||
edit_count: number;
|
||||
}
|
||||
|
||||
export interface DocumentComparisonResponse {
|
||||
schema_version: 1;
|
||||
document: DocumentSummary;
|
||||
components: ComponentSummary[];
|
||||
original_markdown: string | null;
|
||||
cleaned_markdown: string | null;
|
||||
changes: ChangeDetail[];
|
||||
errors: RunErrorDetail[];
|
||||
residual_proposals: ResidualProposalDetail[];
|
||||
}
|
||||
|
||||
export interface ComponentStageResponse {
|
||||
schema_version: 1;
|
||||
document_id: string;
|
||||
component: ComponentSummary;
|
||||
before_sha256: string;
|
||||
after_sha256: string;
|
||||
before_markdown: string;
|
||||
after_markdown: string;
|
||||
changes: ChangeDetail[];
|
||||
}
|
||||
|
||||
export interface ApiErrorResponse {
|
||||
error: {
|
||||
code: string;
|
||||
message: string;
|
||||
};
|
||||
}
|
||||
@@ -1,256 +0,0 @@
|
||||
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { App } from "../src/client/App.js";
|
||||
import type {
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunSummaryResponse,
|
||||
} from "../src/shared/api.js";
|
||||
|
||||
vi.mock("../src/client/DiffView.js", () => ({
|
||||
DiffView: ({ beforeLabel, afterLabel }: { beforeLabel: string; afterLabel: string }) => (
|
||||
<div data-testid="diff-view">
|
||||
{beforeLabel} / {afterLabel}
|
||||
</div>
|
||||
),
|
||||
}));
|
||||
|
||||
const components = [
|
||||
{
|
||||
component_position: 0,
|
||||
component_id: "paper.rule",
|
||||
version: "1.0.0",
|
||||
parameters: [],
|
||||
applicability: "替换测试单词。",
|
||||
change_count: 1,
|
||||
},
|
||||
{
|
||||
component_position: 1,
|
||||
component_id: "paper.zero",
|
||||
version: "1.0.0",
|
||||
parameters: [],
|
||||
applicability: "不修改当前测试文档。",
|
||||
change_count: 0,
|
||||
},
|
||||
];
|
||||
|
||||
const documentSummary = {
|
||||
document_id: "paper",
|
||||
source_label: "inputs/paper.md",
|
||||
status: "success" as const,
|
||||
input_sha256: "1".repeat(64),
|
||||
current_sha256: "2".repeat(64),
|
||||
change_count: 1,
|
||||
source_available: true,
|
||||
output_available: true,
|
||||
availability_error: null,
|
||||
};
|
||||
|
||||
const runResponse: RunSummaryResponse = {
|
||||
schema_version: 1,
|
||||
run: {
|
||||
run_id: "review-run",
|
||||
run_date: "2026-08-23",
|
||||
status: "success",
|
||||
started_at_utc: "2026-08-23T01:00:00Z",
|
||||
completed_at_utc: "2026-08-23T01:01:00Z",
|
||||
retention_until: "2026-09-22T01:01:00Z",
|
||||
},
|
||||
components,
|
||||
documents: [documentSummary],
|
||||
summary: {
|
||||
document_count: 1,
|
||||
success_count: 1,
|
||||
failed_count: 0,
|
||||
unstable_count: 0,
|
||||
change_count: 1,
|
||||
},
|
||||
original_run_location_changed: false,
|
||||
};
|
||||
|
||||
const change = {
|
||||
component_id: "paper.rule",
|
||||
component_version: "1.0.0",
|
||||
component_position: 0,
|
||||
proposal_ref: {
|
||||
component_position: 0,
|
||||
snapshot_sha256: "1".repeat(64),
|
||||
proposal_index: 0,
|
||||
},
|
||||
edit_index: 0,
|
||||
reason: "替换测试单词",
|
||||
location: { line: 1, column: 3 },
|
||||
editor_range: { start: 0, end: 3 },
|
||||
before: "old",
|
||||
after: "new",
|
||||
};
|
||||
|
||||
const documentResponse: DocumentComparisonResponse = {
|
||||
schema_version: 1,
|
||||
document: documentSummary,
|
||||
components,
|
||||
original_markdown: "old",
|
||||
cleaned_markdown: "new",
|
||||
changes: [change],
|
||||
errors: [],
|
||||
residual_proposals: [],
|
||||
};
|
||||
|
||||
const stageResponse: ComponentStageResponse = {
|
||||
schema_version: 1,
|
||||
document_id: "paper",
|
||||
component: components[0]!,
|
||||
before_sha256: "1".repeat(64),
|
||||
after_sha256: "2".repeat(64),
|
||||
before_markdown: "old",
|
||||
after_markdown: "new",
|
||||
changes: [change],
|
||||
};
|
||||
|
||||
function response(payload: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(payload), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
describe("App", () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("shows the run, document comparison, component order and component stage", async () => {
|
||||
const fetchMock = vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
if (pathname === "/api/v1/run") {
|
||||
return Promise.resolve(response(runResponse));
|
||||
}
|
||||
if (pathname.endsWith("/components/0")) {
|
||||
return Promise.resolve(response(stageResponse));
|
||||
}
|
||||
return Promise.resolve(response(documentResponse));
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("heading", { name: "review-run" })).toBeInTheDocument();
|
||||
expect(await screen.findByTestId("diff-view")).toHaveTextContent("清洗前 / 清洗后");
|
||||
expect(screen.getByText("paper.rule")).toBeInTheDocument();
|
||||
expect(screen.getByText("paper.zero")).toBeInTheDocument();
|
||||
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: /paper\.rule/ }));
|
||||
await waitFor(() => {
|
||||
expect(fetchMock).toHaveBeenCalledWith(expect.stringContaining("components/0"), expect.anything());
|
||||
});
|
||||
expect(await screen.findByTestId("diff-view")).toHaveTextContent("组件 1 执行前 / 组件 1 执行后");
|
||||
});
|
||||
|
||||
it("shows API failures without rendering document text", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
response({ error: { code: "unsupported_schema", message: "不支持这个产物版本。" } }, 409),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("alert")).toHaveTextContent("不支持这个产物版本");
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("shows failed audit evidence without inventing a cleaned result", async () => {
|
||||
const failedSummary = {
|
||||
...documentSummary,
|
||||
status: "failed" as const,
|
||||
current_sha256: documentSummary.input_sha256,
|
||||
change_count: 0,
|
||||
output_available: false,
|
||||
};
|
||||
const failedRun: RunSummaryResponse = {
|
||||
...runResponse,
|
||||
run: { ...runResponse.run, status: "failed" },
|
||||
documents: [failedSummary],
|
||||
summary: {
|
||||
document_count: 1,
|
||||
success_count: 0,
|
||||
failed_count: 1,
|
||||
unstable_count: 0,
|
||||
change_count: 0,
|
||||
},
|
||||
};
|
||||
const failedDocument: DocumentComparisonResponse = {
|
||||
schema_version: 1,
|
||||
document: failedSummary,
|
||||
components: components.map((component) => ({ ...component, change_count: 0 })),
|
||||
original_markdown: "原文",
|
||||
cleaned_markdown: null,
|
||||
changes: [],
|
||||
errors: [
|
||||
{
|
||||
component_id: "paper.rule",
|
||||
component_version: "1.0.0",
|
||||
component_position: 0,
|
||||
stage: "transform",
|
||||
error_type: "SyntheticError",
|
||||
message: "测试组件失败。",
|
||||
},
|
||||
],
|
||||
residual_proposals: [],
|
||||
};
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
return Promise.resolve(response(pathname === "/api/v1/run" ? failedRun : failedDocument));
|
||||
}),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("heading", { name: "失败文档只展示审计证据" })).toBeInTheDocument();
|
||||
expect(screen.getByText("测试组件失败。")).toBeInTheDocument();
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("keeps audit details visible when a successful run has lost its original source", async () => {
|
||||
const unavailableSummary = {
|
||||
...documentSummary,
|
||||
source_available: false,
|
||||
availability_error: "原文路径已经失效。",
|
||||
};
|
||||
const unavailableRun: RunSummaryResponse = {
|
||||
...runResponse,
|
||||
documents: [unavailableSummary],
|
||||
};
|
||||
const unavailableDocument: DocumentComparisonResponse = {
|
||||
...documentResponse,
|
||||
document: unavailableSummary,
|
||||
original_markdown: null,
|
||||
changes: [{ ...change, editor_range: null }],
|
||||
};
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
return Promise.resolve(
|
||||
response(pathname === "/api/v1/run" ? unavailableRun : unavailableDocument),
|
||||
);
|
||||
}),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("alert")).toHaveTextContent("原文路径已经失效");
|
||||
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -1,99 +0,0 @@
|
||||
import { render, screen } from "@testing-library/react";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { EditorView } from "@codemirror/view";
|
||||
|
||||
import { DiffView } from "../src/client/DiffView.js";
|
||||
|
||||
function leftPaneSelection(): { from: number; to: number } | null {
|
||||
const pane = document.querySelector(".diff-host .cm-editor");
|
||||
if (pane === null) {
|
||||
return null;
|
||||
}
|
||||
const view = EditorView.findFromDOM(pane as HTMLElement);
|
||||
if (view === null) {
|
||||
return null;
|
||||
}
|
||||
const { from, to } = view.state.selection.main;
|
||||
return { from, to };
|
||||
}
|
||||
|
||||
describe("DiffView", () => {
|
||||
it("keeps Markdown and raw HTML as inert editor text", () => {
|
||||
render(
|
||||
<DiffView
|
||||
before={'# title\n<img src="https://example.com/private.png" onerror="alert(1)">'}
|
||||
after={'# title\n<script>alert("x")</script>'}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(screen.getByRole("region", { name: "清洗前与清洗后对比" })).toBeInTheDocument();
|
||||
expect(document.querySelector("img")).toBeNull();
|
||||
expect(document.querySelector("script")).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps long unchanged sections available in the full document view", () => {
|
||||
const before = Array.from({ length: 30 }, (_, index) => `line ${index + 1}`);
|
||||
const after = [...before];
|
||||
after[14] = "changed line 15";
|
||||
|
||||
render(
|
||||
<DiffView
|
||||
before={before.join("\n")}
|
||||
after={after.join("\n")}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(document.querySelector(".cm-collapsedLines")).toBeNull();
|
||||
});
|
||||
|
||||
it("re-applies the focus selection after the compared texts change", () => {
|
||||
const longText = (mark: string) =>
|
||||
Array.from({ length: 30 }, (_, index) => (index === 14 ? mark : `line ${index + 1}`)).join("\n");
|
||||
|
||||
const { rerender } = render(
|
||||
<DiffView before={longText("old")} after={longText("new")} beforeLabel="清洗前" afterLabel="清洗后" />,
|
||||
);
|
||||
|
||||
// 模拟点击跨组件修改项：focusRange 先落在旧文本上，stage 文本随后到达。
|
||||
rerender(
|
||||
<DiffView
|
||||
before={longText("old")}
|
||||
after={longText("new")}
|
||||
beforeLabel="组件 1 执行前"
|
||||
afterLabel="组件 1 执行后"
|
||||
focusRange={{ start: 58, end: 61 }}
|
||||
/>,
|
||||
);
|
||||
rerender(
|
||||
<DiffView
|
||||
before={longText("stage before")}
|
||||
after={longText("stage after")}
|
||||
beforeLabel="组件 2 执行前"
|
||||
afterLabel="组件 2 执行后"
|
||||
focusRange={{ start: 58, end: 71 }}
|
||||
/>,
|
||||
);
|
||||
|
||||
// 最终视图必须是组件 2 的文本，且选区重新落在新文本的目标范围上。
|
||||
expect(screen.getByRole("region", { name: "组件 2 执行前与组件 2 执行后对比" })).toBeInTheDocument();
|
||||
expect(leftPaneSelection()).toEqual({ from: 58, to: 71 });
|
||||
});
|
||||
|
||||
it("applies the focus selection when it arrives with the initial texts", () => {
|
||||
render(
|
||||
<DiffView
|
||||
before={"a\nb"}
|
||||
after={"a\nc"}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
focusRange={{ start: 0, end: 1 }}
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(leftPaneSelection()).toEqual({ from: 0, to: 1 });
|
||||
});
|
||||
});
|
||||
@@ -1,23 +0,0 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { fetchRun } from "../src/client/api-client.js";
|
||||
|
||||
describe("API response validation", () => {
|
||||
it("rejects a successful HTTP response with an unknown schema", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
new Response(JSON.stringify({ schema_version: 2 }), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
}),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
await expect(fetchRun()).rejects.toMatchObject({
|
||||
code: "invalid_response",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,6 +0,0 @@
|
||||
import "@testing-library/jest-dom/vitest";
|
||||
|
||||
import { afterEach } from "vitest";
|
||||
import { cleanup } from "@testing-library/react";
|
||||
|
||||
afterEach(() => cleanup());
|
||||
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2023",
|
||||
"useDefineForClassFields": true,
|
||||
"lib": ["ES2023", "DOM", "DOM.Iterable"],
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"allowImportingTsExtensions": false,
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"esModuleInterop": true,
|
||||
"jsx": "react-jsx",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"noImplicitOverride": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"types": ["node", "vite/client", "vitest/globals", "@testing-library/jest-dom/vitest"]
|
||||
},
|
||||
"include": ["src", "tests", "vite.config.ts"]
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
import react from "@vitejs/plugin-react";
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
host: "127.0.0.1",
|
||||
port: 5173,
|
||||
strictPort: true,
|
||||
proxy: {
|
||||
"/api": {
|
||||
target: "http://127.0.0.1:4174",
|
||||
changeOrigin: true,
|
||||
configure(proxy) {
|
||||
proxy.on("proxyReq", (request) => request.removeHeader("origin"));
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
build: {
|
||||
outDir: "dist",
|
||||
emptyOutDir: true,
|
||||
},
|
||||
test: {
|
||||
environment: "jsdom",
|
||||
setupFiles: "./tests/setup.ts",
|
||||
css: true,
|
||||
},
|
||||
});
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Run the approved arXiv component over the five local ClinDB paper copies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.components import ArxivSubmissionStampComponent
|
||||
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||
from mdpolish.models import RunStatus
|
||||
from mdpolish.pipeline import Pipeline
|
||||
|
||||
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||
|
||||
|
||||
def _parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _parse_args()
|
||||
repository_root = Path(__file__).resolve().parents[1]
|
||||
documents = tuple(
|
||||
InputDocument(
|
||||
document_id=document_id,
|
||||
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||
source_label=f"data/md/{document_id}.md",
|
||||
)
|
||||
for document_id in _DOCUMENT_IDS
|
||||
)
|
||||
started_at = datetime.now().astimezone()
|
||||
try:
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ArxivSubmissionStampComponent()]),
|
||||
documents=documents,
|
||||
run_id=args.run_id,
|
||||
artifacts_root=repository_root / "artifacts",
|
||||
started_at=started_at,
|
||||
tool=collect_tool_metadata(repository_root),
|
||||
)
|
||||
except Exception as error:
|
||||
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"run_id={args.run_id}")
|
||||
print(f"status={result.status.value}")
|
||||
print(f"documents={result.document_count}")
|
||||
print(f"changes={result.change_count}")
|
||||
print(f"artifacts={result.run_directory}")
|
||||
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,91 +0,0 @@
|
||||
"""Run the approved first-batch ClinDB pipeline over five local paper copies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.components import (
|
||||
ArxivSubmissionStampComponent,
|
||||
HtmlTableDoubleEscapeComponent,
|
||||
HtmlTableLayoutComponent,
|
||||
ManuscriptLineNumberComponent,
|
||||
PageBreakWordJoinComponent,
|
||||
ReferenceSpacingComponent,
|
||||
RepeatedRunningHeaderComponent,
|
||||
WordReviewCommentComponent,
|
||||
)
|
||||
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||
from mdpolish.models import RunStatus
|
||||
from mdpolish.pipeline import Pipeline
|
||||
|
||||
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||
CLINDB_WORD_JOIN_MAPPINGS = (
|
||||
("medi-", "cal", "medical"),
|
||||
("possi-", "bly", "possibly"),
|
||||
("cre-", "ated", "created"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("life-", "threatening", "life-threatening"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def build_pipeline() -> Pipeline:
|
||||
"""Build the frozen ClinDB first-batch component order."""
|
||||
return Pipeline(
|
||||
[
|
||||
WordReviewCommentComponent(),
|
||||
ManuscriptLineNumberComponent(),
|
||||
ArxivSubmissionStampComponent(),
|
||||
RepeatedRunningHeaderComponent(),
|
||||
PageBreakWordJoinComponent(CLINDB_WORD_JOIN_MAPPINGS),
|
||||
HtmlTableDoubleEscapeComponent(),
|
||||
HtmlTableLayoutComponent(),
|
||||
ReferenceSpacingComponent(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _parse_args()
|
||||
repository_root = Path(__file__).resolve().parents[1]
|
||||
documents = tuple(
|
||||
InputDocument(
|
||||
document_id=document_id,
|
||||
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||
source_label=f"data/md/{document_id}.md",
|
||||
)
|
||||
for document_id in _DOCUMENT_IDS
|
||||
)
|
||||
started_at = datetime.now().astimezone()
|
||||
try:
|
||||
result = run_experiment(
|
||||
pipeline=build_pipeline(),
|
||||
documents=documents,
|
||||
run_id=args.run_id,
|
||||
artifacts_root=repository_root / "artifacts",
|
||||
started_at=started_at,
|
||||
tool=collect_tool_metadata(repository_root),
|
||||
)
|
||||
except Exception as error:
|
||||
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"run_id={args.run_id}")
|
||||
print(f"status={result.status.value}")
|
||||
print(f"documents={result.document_count}")
|
||||
print(f"changes={result.change_count}")
|
||||
print(f"artifacts={result.run_directory}")
|
||||
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,12 +1,11 @@
|
||||
"""Approved public API for the first mdpolish in-memory core."""
|
||||
"""Public API for the project-independent mdpolish in-memory core."""
|
||||
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.edits import AppliedBatch, EditValidationError, apply_component_batch, validate_component_batch
|
||||
from mdpolish.edits import AppliedBatch, EditValidationError, apply_modifier_batch, validate_modifier_batch
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
@@ -17,17 +16,20 @@ from mdpolish.models import (
|
||||
TransformResult,
|
||||
markdown_sha256,
|
||||
)
|
||||
from mdpolish.modifier import Modifier, ModifierContractError, ModifierFunction
|
||||
from mdpolish.pipeline import Pipeline, PipelineContractError
|
||||
from mdpolish.regex import regex_replace
|
||||
|
||||
__all__ = [
|
||||
"AppliedBatch",
|
||||
"Change",
|
||||
"Component",
|
||||
"ComponentContractError",
|
||||
"ComponentInfo",
|
||||
"DocumentSnapshot",
|
||||
"EditValidationError",
|
||||
"ErrorStage",
|
||||
"Modifier",
|
||||
"ModifierContractError",
|
||||
"ModifierFunction",
|
||||
"ModifierInfo",
|
||||
"Pipeline",
|
||||
"PipelineContractError",
|
||||
"ProposalReference",
|
||||
@@ -38,7 +40,8 @@ __all__ = [
|
||||
"TextEdit",
|
||||
"TextSpan",
|
||||
"TransformResult",
|
||||
"apply_component_batch",
|
||||
"apply_modifier_batch",
|
||||
"markdown_sha256",
|
||||
"validate_component_batch",
|
||||
"regex_replace",
|
||||
"validate_modifier_batch",
|
||||
]
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
"""Pure replay of recorded artifact changes against an exact Markdown snapshot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from mdpolish.models import markdown_sha256
|
||||
|
||||
|
||||
class ReplayError(ValueError):
|
||||
"""A recorded change chain cannot be replayed without losing trust."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayComponent:
|
||||
"""The component identity needed to validate recorded changes."""
|
||||
|
||||
component_id: str
|
||||
version: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayChange:
|
||||
"""One artifact edit expressed with Python Unicode code-point indexes."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
proposal_component_position: int
|
||||
proposal_snapshot_sha256: str
|
||||
proposal_index: int
|
||||
edit_index: int
|
||||
start: int
|
||||
end: int
|
||||
before: str
|
||||
after: str
|
||||
before_sha256: str
|
||||
after_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LocatedChange:
|
||||
"""A replayed edit with human and browser-editor coordinates."""
|
||||
|
||||
change: ReplayChange
|
||||
line: int
|
||||
column: int
|
||||
editor_start: int
|
||||
editor_end: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayStage:
|
||||
"""The exact snapshots immediately before and after one component."""
|
||||
|
||||
component_position: int
|
||||
before_sha256: str
|
||||
after_sha256: str
|
||||
before_markdown: str
|
||||
after_markdown: str
|
||||
changes: tuple[LocatedChange, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayResult:
|
||||
"""The verified final snapshot, locations and optional component stages."""
|
||||
|
||||
current_markdown: str
|
||||
changes: tuple[LocatedChange, ...]
|
||||
stages: tuple[ReplayStage, ...]
|
||||
|
||||
|
||||
def _line_column(markdown: str, offset: int) -> tuple[int, int]:
|
||||
if offset < 0 or offset > len(markdown):
|
||||
raise ReplayError("a change offset is outside its recorded snapshot")
|
||||
|
||||
line = 1
|
||||
column = 1
|
||||
position = 0
|
||||
while position < offset:
|
||||
character = markdown[position]
|
||||
if character == "\r":
|
||||
line += 1
|
||||
column = 1
|
||||
if position + 1 < offset and markdown[position + 1] == "\n":
|
||||
position += 2
|
||||
else:
|
||||
position += 1
|
||||
elif character == "\n":
|
||||
line += 1
|
||||
column = 1
|
||||
position += 1
|
||||
else:
|
||||
column += 1
|
||||
position += 1
|
||||
return line, column
|
||||
|
||||
|
||||
def _utf16_offset(markdown: str, offset: int) -> int:
|
||||
return len(markdown[:offset].encode("utf-16-le")) // 2
|
||||
|
||||
|
||||
def _changes_conflict(left: ReplayChange, right: ReplayChange) -> bool:
|
||||
if left.start == left.end and right.start == right.end:
|
||||
return left.start == right.start
|
||||
if left.start == left.end:
|
||||
return right.start <= left.start <= right.end
|
||||
if right.start == right.end:
|
||||
return left.start <= right.start <= left.end
|
||||
return max(left.start, right.start) < min(left.end, right.end)
|
||||
|
||||
|
||||
def replay_change_chain(
|
||||
*,
|
||||
input_markdown: str,
|
||||
input_sha256: str,
|
||||
components: tuple[ReplayComponent, ...],
|
||||
changes: tuple[ReplayChange, ...],
|
||||
current_sha256: str,
|
||||
current_markdown: str | None,
|
||||
include_zero_change_stages: bool,
|
||||
) -> ReplayResult:
|
||||
"""Verify and replay a complete recorded chain without file I/O."""
|
||||
if markdown_sha256(input_markdown) != input_sha256:
|
||||
raise ReplayError("input Markdown does not match the recorded input hash")
|
||||
|
||||
previous_position = -1
|
||||
for change in changes:
|
||||
if change.component_position < previous_position:
|
||||
raise ReplayError("recorded changes are not in component order")
|
||||
previous_position = change.component_position
|
||||
|
||||
current = input_markdown
|
||||
located_changes: list[LocatedChange] = []
|
||||
stages: list[ReplayStage] = []
|
||||
cursor = 0
|
||||
|
||||
for component_position, component in enumerate(components):
|
||||
before_markdown = current
|
||||
batch: list[ReplayChange] = []
|
||||
while cursor < len(changes) and changes[cursor].component_position == component_position:
|
||||
batch.append(changes[cursor])
|
||||
cursor += 1
|
||||
|
||||
batch_locations: list[LocatedChange] = []
|
||||
if batch:
|
||||
first = batch[0]
|
||||
if markdown_sha256(current) != first.before_sha256:
|
||||
raise ReplayError("a change batch does not follow the recorded snapshot chain")
|
||||
|
||||
for change in batch:
|
||||
if change.component_id != component.component_id or change.component_version != component.version:
|
||||
raise ReplayError("a change identity does not match the recorded component")
|
||||
if change.proposal_component_position != component_position:
|
||||
raise ReplayError("a change proposal reference has the wrong component position")
|
||||
if change.proposal_snapshot_sha256 != change.before_sha256:
|
||||
raise ReplayError("a change proposal reference targets the wrong snapshot")
|
||||
if change.before_sha256 != first.before_sha256 or change.after_sha256 != first.after_sha256:
|
||||
raise ReplayError("a change batch contains inconsistent snapshot hashes")
|
||||
if change.start < 0 or change.end < change.start or change.end > len(current):
|
||||
raise ReplayError("a change span is outside its recorded snapshot")
|
||||
if len(change.before) != change.end - change.start:
|
||||
raise ReplayError("a change before value does not match its span length")
|
||||
if current[change.start : change.end] != change.before:
|
||||
raise ReplayError("a change before value does not match its recorded snapshot")
|
||||
line, column = _line_column(current, change.start)
|
||||
batch_locations.append(
|
||||
LocatedChange(
|
||||
change=change,
|
||||
line=line,
|
||||
column=column,
|
||||
editor_start=_utf16_offset(current, change.start),
|
||||
editor_end=_utf16_offset(current, change.end),
|
||||
)
|
||||
)
|
||||
|
||||
for left_index, left in enumerate(batch):
|
||||
for right in batch[left_index + 1 :]:
|
||||
if _changes_conflict(left, right):
|
||||
raise ReplayError("a change batch contains conflicting edit ranges")
|
||||
|
||||
application_order = sorted(
|
||||
batch,
|
||||
key=lambda change: (
|
||||
change.start,
|
||||
change.end,
|
||||
change.proposal_index,
|
||||
change.edit_index,
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
for change in application_order:
|
||||
current = current[: change.start] + change.after + current[change.end :]
|
||||
if markdown_sha256(current) != first.after_sha256:
|
||||
raise ReplayError("replayed changes do not produce the recorded batch hash")
|
||||
|
||||
located_changes.extend(batch_locations)
|
||||
if include_zero_change_stages or batch:
|
||||
stages.append(
|
||||
ReplayStage(
|
||||
component_position=component_position,
|
||||
before_sha256=markdown_sha256(before_markdown),
|
||||
after_sha256=markdown_sha256(current),
|
||||
before_markdown=before_markdown,
|
||||
after_markdown=current,
|
||||
changes=tuple(batch_locations),
|
||||
)
|
||||
)
|
||||
|
||||
if cursor != len(changes):
|
||||
raise ReplayError("a change has no matching component position")
|
||||
if markdown_sha256(current) != current_sha256:
|
||||
raise ReplayError("replayed changes do not produce the recorded current hash")
|
||||
if current_markdown is not None and current != current_markdown:
|
||||
raise ReplayError("replayed changes do not produce the recorded current snapshot")
|
||||
|
||||
return ReplayResult(current_markdown=current, changes=tuple(located_changes), stages=tuple(stages))
|
||||
@@ -24,7 +24,7 @@ class HtmlRow:
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HtmlTable:
|
||||
"""Exact source ranges needed by table cleaning components."""
|
||||
"""Exact source ranges needed by table cleaning modifiers."""
|
||||
|
||||
start: int
|
||||
end: int
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Exact physical-line ranges shared by text cleaning components."""
|
||||
"""Exact physical-line ranges shared by text cleaning modifiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,340 +0,0 @@
|
||||
"""Private local storage for complete experiment artifact directories."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
_SAFE_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
|
||||
_DIRECTORY_MODE = 0o700
|
||||
_FILE_MODE = 0o600
|
||||
|
||||
|
||||
class ArtifactStoreError(RuntimeError):
|
||||
"""A local artifact directory cannot be published safely."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StoredDocument:
|
||||
"""Complete encoded artifacts for one document."""
|
||||
|
||||
document_id: str
|
||||
result_json: bytes
|
||||
cleaned_markdown: bytes | None
|
||||
diff: bytes | None
|
||||
output_sha256: str | None
|
||||
|
||||
|
||||
def _validate_identifier(value: str, field_name: str) -> None:
|
||||
if not isinstance(value, str) or _SAFE_ID_PATTERN.fullmatch(value) is None or ".." in value:
|
||||
raise ArtifactStoreError(f"{field_name} must be a safe lowercase identifier")
|
||||
|
||||
|
||||
def _validate_run_date(run_date: str) -> None:
|
||||
if not isinstance(run_date, str):
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD")
|
||||
try:
|
||||
parsed = date.fromisoformat(run_date)
|
||||
except ValueError as error:
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD") from error
|
||||
if parsed.isoformat() != run_date:
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD")
|
||||
|
||||
|
||||
def _require_private_directory(path: Path) -> None:
|
||||
if path.is_symlink():
|
||||
raise ArtifactStoreError(f"artifact directory cannot be a symlink: {path}")
|
||||
if not path.exists():
|
||||
path.mkdir(mode=_DIRECTORY_MODE, parents=False)
|
||||
path.chmod(_DIRECTORY_MODE)
|
||||
if not path.is_dir():
|
||||
raise ArtifactStoreError(f"artifact path is not a directory: {path}")
|
||||
mode = stat.S_IMODE(path.stat().st_mode)
|
||||
if mode != _DIRECTORY_MODE:
|
||||
raise ArtifactStoreError(f"artifact directory must have mode 0700: {path}")
|
||||
|
||||
|
||||
def _create_private_directory(path: Path) -> None:
|
||||
path.mkdir(mode=_DIRECTORY_MODE)
|
||||
path.chmod(_DIRECTORY_MODE)
|
||||
_require_private_directory(path)
|
||||
|
||||
|
||||
def _prepare_runs_directory(artifacts_root: Path, run_date: str) -> Path:
|
||||
if not artifacts_root.exists():
|
||||
artifacts_root.mkdir(mode=_DIRECTORY_MODE, parents=True)
|
||||
artifacts_root.chmod(_DIRECTORY_MODE)
|
||||
_require_private_directory(artifacts_root)
|
||||
|
||||
date_directory = artifacts_root / run_date
|
||||
_require_private_directory(date_directory)
|
||||
runs_directory = date_directory / "runs"
|
||||
_require_private_directory(runs_directory)
|
||||
return runs_directory
|
||||
|
||||
|
||||
def _write_private_file(path: Path, content: bytes) -> None:
|
||||
if not isinstance(content, bytes):
|
||||
raise ArtifactStoreError("artifact content must be bytes")
|
||||
with path.open("xb") as artifact_file:
|
||||
artifact_file.write(content)
|
||||
artifact_file.flush()
|
||||
os.fsync(artifact_file.fileno())
|
||||
path.chmod(_FILE_MODE)
|
||||
if stat.S_IMODE(path.stat().st_mode) != _FILE_MODE:
|
||||
raise ArtifactStoreError(f"artifact file must have mode 0600: {path}")
|
||||
if path.read_bytes() != content:
|
||||
raise ArtifactStoreError(f"artifact readback verification failed: {path}")
|
||||
|
||||
|
||||
def _validate_document(document: StoredDocument) -> None:
|
||||
_validate_identifier(document.document_id, "document_id")
|
||||
has_output = document.cleaned_markdown is not None
|
||||
if has_output != (document.diff is not None) or has_output != (document.output_sha256 is not None):
|
||||
raise ArtifactStoreError("cleaned Markdown, diff, and output hash must be present together")
|
||||
if document.cleaned_markdown is not None:
|
||||
digest = sha256(document.cleaned_markdown).hexdigest()
|
||||
if digest != document.output_sha256:
|
||||
raise ArtifactStoreError("cleaned Markdown does not match its output hash")
|
||||
|
||||
|
||||
def _json_object(content: bytes, label: str) -> dict[str, object]:
|
||||
if not isinstance(content, bytes) or not content.endswith(b"\n") or content.startswith(b"\xef\xbb\xbf"):
|
||||
raise ArtifactStoreError(f"{label} must be BOM-free UTF-8 JSON ending in a newline")
|
||||
try:
|
||||
decoded = json.loads(content)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as error:
|
||||
raise ArtifactStoreError(f"{label} must contain valid UTF-8 JSON") from error
|
||||
if not isinstance(decoded, dict) or not all(isinstance(key, str) for key in decoded):
|
||||
raise ArtifactStoreError(f"{label} must contain a JSON object")
|
||||
return cast(dict[str, object], decoded)
|
||||
|
||||
|
||||
def _nested_object(payload: dict[str, object], field_name: str, label: str) -> dict[str, object]:
|
||||
value = payload.get(field_name)
|
||||
if not isinstance(value, dict) or not all(isinstance(key, str) for key in value):
|
||||
raise ArtifactStoreError(f"{label}.{field_name} must be a JSON object")
|
||||
return cast(dict[str, object], value)
|
||||
|
||||
|
||||
def _validate_bundle(
|
||||
*,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
run_directory: Path,
|
||||
manifest_json: bytes,
|
||||
review_locator_json: bytes,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> None:
|
||||
manifest = _json_object(manifest_json, "manifest")
|
||||
if manifest.get("schema_version") != 1:
|
||||
raise ArtifactStoreError("manifest schema_version must be 1")
|
||||
run = _nested_object(manifest, "run", "manifest")
|
||||
if run.get("run_date") != run_date or run.get("run_id") != run_id:
|
||||
raise ArtifactStoreError("manifest run identity does not match the target path")
|
||||
|
||||
document_indexes = manifest.get("documents")
|
||||
if not isinstance(document_indexes, list) or len(document_indexes) != len(documents):
|
||||
raise ArtifactStoreError("manifest documents do not match the stored documents")
|
||||
|
||||
success_count = 0
|
||||
failed_count = 0
|
||||
unstable_count = 0
|
||||
change_count = 0
|
||||
for position, document in enumerate(documents):
|
||||
raw_index = document_indexes[position]
|
||||
if not isinstance(raw_index, dict) or not all(isinstance(key, str) for key in raw_index):
|
||||
raise ArtifactStoreError("manifest document indexes must be JSON objects")
|
||||
index = cast(dict[str, object], raw_index)
|
||||
report = _json_object(document.result_json, f"result for {document.document_id}")
|
||||
report_document = _nested_object(report, "document", f"result for {document.document_id}")
|
||||
report_output = _nested_object(report, "output", f"result for {document.document_id}")
|
||||
status = report.get("status")
|
||||
changes = report.get("changes")
|
||||
if (
|
||||
report.get("schema_version") != 1
|
||||
or report_document.get("document_id") != document.document_id
|
||||
or status not in {"success", "failed", "unstable"}
|
||||
or not isinstance(changes, list)
|
||||
):
|
||||
raise ArtifactStoreError("a document result does not match its stored identity")
|
||||
|
||||
has_output = document.cleaned_markdown is not None
|
||||
expected_cleaned_name = "cleaned.md" if has_output else None
|
||||
expected_diff_name = "changes.diff" if has_output else None
|
||||
base = f"documents/{document.document_id}"
|
||||
expected_cleaned_path = f"{base}/cleaned.md" if has_output else None
|
||||
expected_diff_path = f"{base}/changes.diff" if has_output else None
|
||||
if has_output != (status == "success"):
|
||||
raise ArtifactStoreError("document output presence does not match its result status")
|
||||
if report_output != {
|
||||
"cleaned_path": expected_cleaned_name,
|
||||
"diff_path": expected_diff_name,
|
||||
}:
|
||||
raise ArtifactStoreError("document result paths do not match its stored files")
|
||||
if document.output_sha256 != (report.get("current_sha256") if has_output else None):
|
||||
raise ArtifactStoreError("document output hash does not match its result")
|
||||
|
||||
expected_index = {
|
||||
"document_id": document.document_id,
|
||||
"source_label": report_document.get("source_label"),
|
||||
"status": status,
|
||||
"input_sha256": report.get("input_sha256"),
|
||||
"current_sha256": report.get("current_sha256"),
|
||||
"change_count": len(changes),
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": expected_cleaned_path,
|
||||
"diff_path": expected_diff_path,
|
||||
}
|
||||
if index != expected_index:
|
||||
raise ArtifactStoreError("a manifest document index does not match its result and files")
|
||||
|
||||
change_count += len(changes)
|
||||
if status == "success":
|
||||
success_count += 1
|
||||
elif status == "failed":
|
||||
failed_count += 1
|
||||
else:
|
||||
unstable_count += 1
|
||||
|
||||
overall_status = "failed" if failed_count else "unstable" if unstable_count else "success"
|
||||
expected_summary = {
|
||||
"document_count": len(documents),
|
||||
"success_count": success_count,
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
}
|
||||
if run.get("status") != overall_status or manifest.get("summary") != expected_summary:
|
||||
raise ArtifactStoreError("manifest status or summary does not match its documents")
|
||||
|
||||
locator = _json_object(review_locator_json, "review locator")
|
||||
if locator.get("schema_version") != 1:
|
||||
raise ArtifactStoreError("review locator schema_version must be 1")
|
||||
locator_run = _nested_object(locator, "run", "review locator")
|
||||
expected_run = {
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_directory.resolve(strict=False)),
|
||||
"manifest_path": "manifest.json",
|
||||
}
|
||||
if locator_run != expected_run:
|
||||
raise ArtifactStoreError("review locator run identity does not match the target path")
|
||||
|
||||
locator_documents = locator.get("documents")
|
||||
if not isinstance(locator_documents, list) or len(locator_documents) != len(documents):
|
||||
raise ArtifactStoreError("review locator documents do not match the stored documents")
|
||||
for position, document in enumerate(documents):
|
||||
raw_locator_document = locator_documents[position]
|
||||
if not isinstance(raw_locator_document, dict) or not all(
|
||||
isinstance(key, str) for key in raw_locator_document
|
||||
):
|
||||
raise ArtifactStoreError("review locator documents must be JSON objects")
|
||||
locator_document = cast(dict[str, object], raw_locator_document)
|
||||
source_path_value = locator_document.get("source_path")
|
||||
if not isinstance(source_path_value, str) or "\0" in source_path_value:
|
||||
raise ArtifactStoreError("review locator source_path must be an absolute path")
|
||||
source_path = Path(source_path_value)
|
||||
if not source_path.is_absolute() or source_path.resolve(strict=False) != source_path:
|
||||
raise ArtifactStoreError("review locator source_path must be an absolute resolved path")
|
||||
|
||||
manifest_index = cast(dict[str, object], document_indexes[position])
|
||||
expected_locator_document = {
|
||||
"document_id": document.document_id,
|
||||
"source_path": source_path_value,
|
||||
"input_sha256": manifest_index.get("input_sha256"),
|
||||
}
|
||||
if locator_document != expected_locator_document:
|
||||
raise ArtifactStoreError("review locator document identity does not match the manifest")
|
||||
try:
|
||||
source_bytes = source_path.read_bytes()
|
||||
except OSError as error:
|
||||
raise ArtifactStoreError("review locator source_path must identify a readable regular file") from error
|
||||
if not source_path.is_file() or sha256(source_bytes).hexdigest() != manifest_index.get("input_sha256"):
|
||||
raise ArtifactStoreError("review locator source file does not match its input hash")
|
||||
|
||||
|
||||
def _rename_no_replace(source: Path, target: Path) -> None:
|
||||
directory_fd = os.open(source.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
fcntl.flock(directory_fd, fcntl.LOCK_EX)
|
||||
if target.exists() or target.is_symlink():
|
||||
raise ArtifactStoreError("the target run directory already exists")
|
||||
os.rename(source, target)
|
||||
finally:
|
||||
fcntl.flock(directory_fd, fcntl.LOCK_UN)
|
||||
os.close(directory_fd)
|
||||
|
||||
|
||||
def publish_run(
|
||||
*,
|
||||
artifacts_root: Path,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
manifest_json: bytes,
|
||||
review_locator_json: bytes,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> Path:
|
||||
"""Publish one complete run directory without replacing an existing run."""
|
||||
if not isinstance(artifacts_root, Path):
|
||||
raise TypeError("artifacts_root must be a Path")
|
||||
_validate_run_date(run_date)
|
||||
_validate_identifier(run_id, "run_id")
|
||||
if not documents:
|
||||
raise ArtifactStoreError("an artifact run requires at least one document")
|
||||
|
||||
seen_ids: set[str] = set()
|
||||
for document in documents:
|
||||
if not isinstance(document, StoredDocument):
|
||||
raise ArtifactStoreError("documents must contain only StoredDocument values")
|
||||
_validate_document(document)
|
||||
if document.document_id in seen_ids:
|
||||
raise ArtifactStoreError("document_id values must be unique")
|
||||
seen_ids.add(document.document_id)
|
||||
_validate_bundle(
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
run_directory=artifacts_root / run_date / "runs" / run_id,
|
||||
manifest_json=manifest_json,
|
||||
review_locator_json=review_locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = _prepare_runs_directory(artifacts_root, run_date)
|
||||
final_directory = runs_directory / run_id
|
||||
if final_directory.exists() or final_directory.is_symlink():
|
||||
raise ArtifactStoreError("the target run directory already exists")
|
||||
|
||||
temporary_directory = Path(tempfile.mkdtemp(prefix=f".{run_id}.", dir=runs_directory))
|
||||
temporary_directory.chmod(_DIRECTORY_MODE)
|
||||
try:
|
||||
documents_directory = temporary_directory / "documents"
|
||||
_create_private_directory(documents_directory)
|
||||
for document in documents:
|
||||
document_directory = documents_directory / document.document_id
|
||||
_create_private_directory(document_directory)
|
||||
_write_private_file(document_directory / "result.json", document.result_json)
|
||||
if document.cleaned_markdown is not None and document.diff is not None:
|
||||
_write_private_file(document_directory / "cleaned.md", document.cleaned_markdown)
|
||||
_write_private_file(document_directory / "changes.diff", document.diff)
|
||||
|
||||
_write_private_file(temporary_directory / "manifest.json", manifest_json)
|
||||
_write_private_file(temporary_directory / "review-locator.json", review_locator_json)
|
||||
_rename_no_replace(temporary_directory, final_directory)
|
||||
except Exception:
|
||||
if temporary_directory.exists():
|
||||
shutil.rmtree(temporary_directory)
|
||||
raise
|
||||
|
||||
if stat.S_IMODE(final_directory.stat().st_mode) != _DIRECTORY_MODE:
|
||||
raise ArtifactStoreError("published run directory does not have mode 0700")
|
||||
return final_directory
|
||||
@@ -1,105 +0,0 @@
|
||||
"""Component extension contract for the mdpolish core."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Mapping
|
||||
from typing import cast, final
|
||||
|
||||
from mdpolish.models import ComponentInfo, DocumentSnapshot, Parameters, ParameterValue, ProposedChange
|
||||
|
||||
_COMPONENT_ID_PATTERN = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
|
||||
_SEMVER_PATTERN = re.compile(r"^(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)$")
|
||||
|
||||
|
||||
class ComponentContractError(ValueError):
|
||||
"""A component does not satisfy the approved extension contract."""
|
||||
|
||||
|
||||
def _freeze_parameter(value: object) -> ParameterValue:
|
||||
if value is None or isinstance(value, (str, bool)) or type(value) is int:
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ComponentContractError("component parameters cannot contain non-finite floats")
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
pairs: list[tuple[str, ParameterValue]] = []
|
||||
for key, nested_value in value.items():
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ComponentContractError("component parameter mapping keys must be non-empty strings")
|
||||
pairs.append((key, _freeze_parameter(nested_value)))
|
||||
return tuple(sorted(pairs, key=lambda pair: pair[0]))
|
||||
if isinstance(value, (list, tuple)):
|
||||
return tuple(_freeze_parameter(item) for item in value)
|
||||
raise ComponentContractError("component parameters must contain only deterministic basic data")
|
||||
|
||||
|
||||
def _freeze_parameters(parameters: object) -> Parameters:
|
||||
if not isinstance(parameters, Mapping):
|
||||
raise ComponentContractError("component parameters must be a mapping")
|
||||
frozen = _freeze_parameter(parameters)
|
||||
if not isinstance(frozen, tuple):
|
||||
raise AssertionError("a mapping must normalize to a tuple")
|
||||
return cast(Parameters, frozen)
|
||||
|
||||
|
||||
class Component(ABC):
|
||||
"""Base class for deterministic, side-effect-free cleaning components."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def component_id(self) -> str:
|
||||
"""Return the stable lowercase identity of this component."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def version(self) -> str:
|
||||
"""Return this component's MAJOR.MINOR.PATCH version."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
"""Return the current deterministic parameter values."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def applicability(self) -> str:
|
||||
"""Describe handled structures, strict preconditions, and exclusions."""
|
||||
|
||||
@abstractmethod
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
"""Return exact, automatically applicable changes for this snapshot."""
|
||||
|
||||
@final
|
||||
def _component_info(self) -> ComponentInfo:
|
||||
component_id = self.component_id
|
||||
version = self.version
|
||||
applicability = self.applicability
|
||||
|
||||
if not isinstance(component_id, str) or _COMPONENT_ID_PATTERN.fullmatch(component_id) is None:
|
||||
raise ComponentContractError("component_id must be a stable lowercase identifier")
|
||||
if not isinstance(version, str) or _SEMVER_PATTERN.fullmatch(version) is None:
|
||||
raise ComponentContractError("component version must use MAJOR.MINOR.PATCH")
|
||||
if not isinstance(applicability, str) or not applicability.strip():
|
||||
raise ComponentContractError("component applicability must be a non-empty string")
|
||||
|
||||
return ComponentInfo(
|
||||
component_id=component_id,
|
||||
version=version,
|
||||
parameters=_freeze_parameters(self.parameters),
|
||||
applicability=applicability,
|
||||
)
|
||||
|
||||
@final
|
||||
def _collect_proposals(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise ComponentContractError("components require a DocumentSnapshot")
|
||||
proposals = self._propose_changes(snapshot)
|
||||
if not isinstance(proposals, tuple):
|
||||
raise ComponentContractError("_propose_changes must return a tuple")
|
||||
if any(not isinstance(proposal, ProposedChange) for proposal in proposals):
|
||||
raise ComponentContractError("_propose_changes must return only ProposedChange values")
|
||||
return proposals
|
||||
@@ -1,22 +0,0 @@
|
||||
"""Approved business cleaning components."""
|
||||
|
||||
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
||||
from mdpolish.components.html_table_double_escape import HtmlTableDoubleEscapeComponent
|
||||
from mdpolish.components.html_table_layout import HtmlTableLayoutComponent
|
||||
from mdpolish.components.manuscript_line_number import ManuscriptLineNumberComponent
|
||||
from mdpolish.components.page_break_word_join import PageBreakWordJoinComponent, WordJoinMapping
|
||||
from mdpolish.components.reference_spacing import ReferenceSpacingComponent
|
||||
from mdpolish.components.repeated_running_header import RepeatedRunningHeaderComponent
|
||||
from mdpolish.components.word_review_comment import WordReviewCommentComponent
|
||||
|
||||
__all__ = [
|
||||
"ArxivSubmissionStampComponent",
|
||||
"HtmlTableDoubleEscapeComponent",
|
||||
"HtmlTableLayoutComponent",
|
||||
"ManuscriptLineNumberComponent",
|
||||
"PageBreakWordJoinComponent",
|
||||
"ReferenceSpacingComponent",
|
||||
"RepeatedRunningHeaderComponent",
|
||||
"WordJoinMapping",
|
||||
"WordReviewCommentComponent",
|
||||
]
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Remove exact arXiv submission stamp lines from converted papers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import iter_physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_STAMP_PATTERN = re.compile(
|
||||
r"^arXiv:[0-9]{4}\.[0-9]+v[0-9]+ \[[A-Za-z0-9_.-]+\] "
|
||||
r"(?:[1-9]|[12][0-9]|3[01]) "
|
||||
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{4}$"
|
||||
)
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||
|
||||
|
||||
class ArxivSubmissionStampComponent(Component):
|
||||
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.arxiv_submission_stamp"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理 PDF/arXiv 论文转换产生的独立提交戳行，要求严格整行匹配，排除所有相似文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
proposals: list[ProposedChange] = []
|
||||
for line in iter_physical_lines(snapshot.markdown):
|
||||
if _STAMP_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||
continue
|
||||
expected_text = snapshot.markdown[line.content_start : line.full_end]
|
||||
edit = TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(line.content_start, line.full_end),
|
||||
expected_text=expected_text,
|
||||
replacement="",
|
||||
)
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(edit,),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
@@ -1,93 +0,0 @@
|
||||
"""Remove line-number prefixes from an evidence-backed manuscript sequence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_PLAIN_PATTERN = re.compile(r"^([1-9][0-9]{0,2}) (?=\S)")
|
||||
_HEADING_PATTERN = re.compile(r"^(#{1,6}) ([1-9][0-9]{0,2}) (?=\S)")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除已确认手稿序列中的行号前缀"
|
||||
_MINIMUM_CANDIDATES = 20
|
||||
_MINIMUM_HEADING_CANDIDATES = 2
|
||||
|
||||
|
||||
def _candidate(line: PhysicalLine, markdown: str) -> tuple[int, int, int, bool] | None:
|
||||
content = line.content(markdown)
|
||||
heading_match = _HEADING_PATTERN.match(content)
|
||||
if heading_match is not None:
|
||||
number_start = line.content_start + len(heading_match.group(1)) + 1
|
||||
return int(heading_match.group(2)), number_start, line.content_start + heading_match.end(), True
|
||||
|
||||
plain_match = _PLAIN_PATTERN.match(content)
|
||||
if plain_match is None:
|
||||
return None
|
||||
return int(plain_match.group(1)), line.content_start, line.content_start + plain_match.end(), False
|
||||
|
||||
|
||||
class ManuscriptLineNumberComponent(Component):
|
||||
"""Strip prefixes only from one long, monotonic post-Abstract sequence."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.manuscript_line_number"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理唯一 ## Abstract 后至少 20 个严格递增且含至少 2 个标题证据的手稿行号序列，"
|
||||
"排除作者单位、列表和短数字片段。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
abstract_indexes = [
|
||||
index for index, line in enumerate(lines) if line.content(snapshot.markdown) == "## Abstract"
|
||||
]
|
||||
if len(abstract_indexes) != 1:
|
||||
return ()
|
||||
|
||||
candidates = [
|
||||
candidate
|
||||
for line in lines[abstract_indexes[0] + 1 :]
|
||||
if (candidate := _candidate(line, snapshot.markdown)) is not None
|
||||
]
|
||||
if len(candidates) < _MINIMUM_CANDIDATES:
|
||||
return ()
|
||||
if sum(is_heading for _, _, _, is_heading in candidates) < _MINIMUM_HEADING_CANDIDATES:
|
||||
return ()
|
||||
|
||||
numbers = [number for number, _, _, _ in candidates]
|
||||
if any(previous >= current for previous, current in pairwise(numbers)):
|
||||
return ()
|
||||
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
)
|
||||
for _, start, end, _ in candidates
|
||||
)
|
||||
@@ -1,120 +0,0 @@
|
||||
"""Normalize blank lines between numbered entries in References sections."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_ATX_HEADING_PATTERN = re.compile(r"^(#{1,6}) (\S(?:.*\S)?)$")
|
||||
_ENTRY_PATTERN = re.compile(r"^([1-9][0-9]*)\. \S")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "统一 References 章节中相邻编号条目之间的一个空行"
|
||||
|
||||
|
||||
def _heading(line: PhysicalLine, markdown: str) -> tuple[int, str] | None:
|
||||
match = _ATX_HEADING_PATTERN.fullmatch(line.content(markdown))
|
||||
if match is None:
|
||||
return None
|
||||
return len(match.group(1)), match.group(2)
|
||||
|
||||
|
||||
class ReferenceSpacingComponent(Component):
|
||||
"""Normalize only complete, consecutive numbered reference sections."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.reference_spacing"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理准确 References ATX 章节内从 1 开始逐项递增的编号条目，统一为一个空行；"
|
||||
"排除章节外列表、缺号和混合行尾边界。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for heading_index, line in enumerate(lines):
|
||||
heading = _heading(line, snapshot.markdown)
|
||||
if heading is None or heading[1].lower() != "references":
|
||||
continue
|
||||
heading_level = heading[0]
|
||||
section_end = len(lines)
|
||||
for index in range(heading_index + 1, len(lines)):
|
||||
following_heading = _heading(lines[index], snapshot.markdown)
|
||||
if following_heading is not None and following_heading[0] <= heading_level:
|
||||
section_end = index
|
||||
break
|
||||
|
||||
entries: list[tuple[int, int]] = []
|
||||
section_is_safe = True
|
||||
for index in range(heading_index + 1, section_end):
|
||||
content = lines[index].content(snapshot.markdown)
|
||||
match = _ENTRY_PATTERN.match(content)
|
||||
if match is not None:
|
||||
entries.append((index, int(match.group(1))))
|
||||
elif _heading(lines[index], snapshot.markdown) is not None:
|
||||
section_is_safe = False
|
||||
if not section_is_safe or len(entries) < 2:
|
||||
continue
|
||||
if [number for _, number in entries] != list(range(1, len(entries) + 1)):
|
||||
continue
|
||||
|
||||
section_edits: list[tuple[int, int, str]] = []
|
||||
for previous, following in pairwise(entries):
|
||||
previous_index = previous[0]
|
||||
next_index = following[0]
|
||||
last_nonblank_index = next(
|
||||
index
|
||||
for index in range(next_index - 1, previous_index - 1, -1)
|
||||
if not lines[index].is_blank(snapshot.markdown)
|
||||
)
|
||||
between = lines[last_nonblank_index + 1 : next_index]
|
||||
if any(not blank.is_blank(snapshot.markdown) for blank in between):
|
||||
section_is_safe = False
|
||||
break
|
||||
line_ending = lines[last_nonblank_index].line_ending(snapshot.markdown)
|
||||
if not line_ending or any(blank.line_ending(snapshot.markdown) != line_ending for blank in between):
|
||||
section_is_safe = False
|
||||
break
|
||||
start = lines[last_nonblank_index].content_end
|
||||
end = lines[next_index].content_start
|
||||
replacement = line_ending * 2
|
||||
if snapshot.markdown[start:end] != replacement:
|
||||
section_edits.append((start, end, replacement))
|
||||
if section_is_safe:
|
||||
edits.extend(section_edits)
|
||||
|
||||
edits.sort(key=lambda item: item[0])
|
||||
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||
return ()
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement=replacement,
|
||||
),
|
||||
),
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
@@ -1,115 +0,0 @@
|
||||
"""Remove repeated ATX running headers with paragraph-interruption evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_HEADING_PATTERN = re.compile(r"^#{1,6} \S(?:.*\S)?$")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除经重复和断句证据确认的跑动页眉"
|
||||
_TERMINAL_PUNCTUATION = frozenset(".?!:;。！？：；")
|
||||
|
||||
|
||||
def _bridge_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||
if index < 2 or index + 2 >= len(lines):
|
||||
return None
|
||||
before_blank = lines[index - 1]
|
||||
after_blank = lines[index + 1]
|
||||
previous = lines[index - 2]
|
||||
following = lines[index + 2]
|
||||
if not before_blank.is_blank(markdown) or not after_blank.is_blank(markdown):
|
||||
return None
|
||||
if previous.is_blank(markdown) or following.is_blank(markdown):
|
||||
return None
|
||||
|
||||
previous_text = previous.content(markdown).rstrip()
|
||||
following_text = following.content(markdown)
|
||||
if not previous_text or previous_text[-1] in _TERMINAL_PUNCTUATION:
|
||||
return None
|
||||
if not following_text or following_text[0] not in "abcdefghijklmnopqrstuvwxyz":
|
||||
return None
|
||||
return previous.content_end, following.content_start
|
||||
|
||||
|
||||
def _deletion_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||
if index + 1 >= len(lines) or not lines[index + 1].is_blank(markdown):
|
||||
return None
|
||||
return lines[index].content_start, lines[index + 1].full_end
|
||||
|
||||
|
||||
class RepeatedRunningHeaderComponent(Component):
|
||||
"""Remove a repeated heading only when one occurrence splits a sentence."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.repeated_running_header"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理原样重复至少两次、且至少一次由小写续句证据确认切断正文的 ATX 跑动页眉；"
|
||||
"任一出现位置不安全时整组保留。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
groups: defaultdict[str, list[int]] = defaultdict(list)
|
||||
for index, line in enumerate(lines):
|
||||
content = line.content(snapshot.markdown)
|
||||
if _HEADING_PATTERN.fullmatch(content) is not None:
|
||||
groups[content].append(index)
|
||||
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for indexes in groups.values():
|
||||
if len(indexes) < 2:
|
||||
continue
|
||||
group_edits: list[tuple[int, int, str]] = []
|
||||
bridge_found = False
|
||||
for index in indexes:
|
||||
bridge = _bridge_span(lines, index, snapshot.markdown)
|
||||
if bridge is not None:
|
||||
group_edits.append((*bridge, " "))
|
||||
bridge_found = True
|
||||
continue
|
||||
deletion = _deletion_span(lines, index, snapshot.markdown)
|
||||
if deletion is None:
|
||||
group_edits = []
|
||||
break
|
||||
group_edits.append((*deletion, ""))
|
||||
if bridge_found and group_edits:
|
||||
edits.extend(group_edits)
|
||||
|
||||
edits.sort(key=lambda item: (item[0], item[1]))
|
||||
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||
return ()
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement=replacement,
|
||||
),
|
||||
),
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
@@ -1,62 +0,0 @@
|
||||
"""Remove strict single-line Word review comments."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_COMMENT_PATTERN = re.compile(r"^Commented \[[A-Za-z0-9]+\]: .*\S$")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除严格单行 Word 审阅批注及其后一个空行"
|
||||
|
||||
|
||||
class WordReviewCommentComponent(Component):
|
||||
"""Delete approved one-line Word comments followed by a blank line."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.word_review_comment"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理独占物理行且后接空行的 Word Commented 审阅批注，排除多行正文和所有相似文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
proposals: list[ProposedChange] = []
|
||||
for index, line in enumerate(lines[:-1]):
|
||||
following = lines[index + 1]
|
||||
if _COMMENT_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||
continue
|
||||
if not following.is_blank(snapshot.markdown):
|
||||
continue
|
||||
start = line.content_start
|
||||
end = following.full_end
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
+18
-18
@@ -6,8 +6,8 @@ from dataclasses import dataclass
|
||||
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
TextEdit,
|
||||
@@ -15,12 +15,12 @@ from mdpolish.models import (
|
||||
|
||||
|
||||
class EditValidationError(ValueError):
|
||||
"""A component edit batch cannot be applied safely."""
|
||||
"""A modifier edit batch cannot be applied safely."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AppliedBatch:
|
||||
"""The new snapshot and audit entries from one atomic component batch."""
|
||||
"""The new snapshot and audit entries from one atomic modifier batch."""
|
||||
|
||||
snapshot: DocumentSnapshot
|
||||
changes: tuple[Change, ...]
|
||||
@@ -47,21 +47,21 @@ def _edits_conflict(left: TextEdit, right: TextEdit) -> bool:
|
||||
return max(left_span.start, right_span.start) < min(left_span.end, right_span.end)
|
||||
|
||||
|
||||
def validate_component_batch(
|
||||
def validate_modifier_batch(
|
||||
snapshot: DocumentSnapshot,
|
||||
proposals: tuple[ProposedChange, ...],
|
||||
) -> tuple[_IndexedEdit, ...]:
|
||||
"""Validate a complete component batch without changing the snapshot."""
|
||||
"""Validate a complete modifier batch without changing the snapshot."""
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise TypeError("snapshot must be a DocumentSnapshot")
|
||||
if not isinstance(proposals, tuple):
|
||||
raise EditValidationError("component proposals must be a tuple")
|
||||
raise EditValidationError("modifier proposals must be a tuple")
|
||||
|
||||
indexed_edits: list[_IndexedEdit] = []
|
||||
seen_edits: set[TextEdit] = set()
|
||||
for proposal_index, proposal in enumerate(proposals):
|
||||
if not isinstance(proposal, ProposedChange):
|
||||
raise EditValidationError("a component batch must contain only ProposedChange values")
|
||||
raise EditValidationError("a modifier batch must contain only ProposedChange values")
|
||||
if proposal.snapshot_sha256 != snapshot.sha256:
|
||||
raise EditValidationError("a proposal targets a stale document snapshot")
|
||||
for edit_index, edit in enumerate(proposal.edits):
|
||||
@@ -72,7 +72,7 @@ def validate_component_batch(
|
||||
if snapshot.markdown[edit.span.start : edit.span.end] != edit.expected_text:
|
||||
raise EditValidationError("an edit's expected_text does not match the document snapshot")
|
||||
if edit in seen_edits:
|
||||
raise EditValidationError("a component batch contains a duplicate edit")
|
||||
raise EditValidationError("a modifier batch contains a duplicate edit")
|
||||
seen_edits.add(edit)
|
||||
indexed_edits.append(
|
||||
_IndexedEdit(
|
||||
@@ -86,19 +86,19 @@ def validate_component_batch(
|
||||
for left_index, left in enumerate(indexed_edits):
|
||||
for right in indexed_edits[left_index + 1 :]:
|
||||
if _edits_conflict(left.edit, right.edit):
|
||||
raise EditValidationError("a component batch contains conflicting edit ranges")
|
||||
raise EditValidationError("a modifier batch contains conflicting edit ranges")
|
||||
|
||||
return tuple(indexed_edits)
|
||||
|
||||
|
||||
def apply_component_batch(
|
||||
def apply_modifier_batch(
|
||||
snapshot: DocumentSnapshot,
|
||||
proposals: tuple[ProposedChange, ...],
|
||||
component: ComponentInfo,
|
||||
component_position: int,
|
||||
modifier: ModifierInfo,
|
||||
modifier_position: int,
|
||||
) -> AppliedBatch:
|
||||
"""Atomically apply one fully validated component batch."""
|
||||
indexed_edits = validate_component_batch(snapshot, proposals)
|
||||
"""Atomically apply one fully validated modifier batch."""
|
||||
indexed_edits = validate_modifier_batch(snapshot, proposals)
|
||||
if not indexed_edits:
|
||||
return AppliedBatch(snapshot=snapshot, changes=())
|
||||
|
||||
@@ -129,11 +129,11 @@ def apply_component_batch(
|
||||
)
|
||||
changes = tuple(
|
||||
Change(
|
||||
component_id=component.component_id,
|
||||
component_version=component.version,
|
||||
component_position=component_position,
|
||||
modifier_id=modifier.modifier_id,
|
||||
modifier_version=modifier.version,
|
||||
modifier_position=modifier_position,
|
||||
proposal_ref=ProposalReference(
|
||||
component_position=component_position,
|
||||
modifier_position=modifier_position,
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
proposal_index=item.proposal_index,
|
||||
),
|
||||
|
||||
@@ -1,412 +0,0 @@
|
||||
"""Explicit local file experiments built around the in-memory pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import platform as platform_module
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from hashlib import sha256
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.artifact_store import StoredDocument, publish_run
|
||||
from mdpolish.models import ComponentInfo, RunStatus, markdown_sha256
|
||||
from mdpolish.pipeline import Pipeline
|
||||
from mdpolish.reporting import (
|
||||
DocumentReport,
|
||||
JsonObject,
|
||||
JsonValue,
|
||||
build_document_report,
|
||||
component_info_json,
|
||||
encode_json,
|
||||
)
|
||||
|
||||
_SAFE_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
|
||||
_UTC_OFFSET_PATTERN = re.compile(r"^[+-][0-9]{2}:[0-9]{2}$")
|
||||
|
||||
|
||||
class ExperimentError(RuntimeError):
|
||||
"""A local experiment cannot be completed without violating its contract."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class InputDocument:
|
||||
"""One explicitly selected local Markdown input."""
|
||||
|
||||
document_id: str
|
||||
source_path: Path
|
||||
source_label: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ToolMetadata:
|
||||
"""Source-safe tool and environment facts for a run manifest."""
|
||||
|
||||
name: str
|
||||
package_version: str
|
||||
python_version: str
|
||||
platform: str
|
||||
git_commit: str | None
|
||||
git_dirty: bool | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ExperimentResult:
|
||||
"""Non-sensitive terminal summary of one published local run."""
|
||||
|
||||
run_directory: Path
|
||||
status: RunStatus
|
||||
document_count: int
|
||||
success_count: int
|
||||
failed_count: int
|
||||
unstable_count: int
|
||||
change_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _PreparedDocument:
|
||||
input: InputDocument
|
||||
resolved_path: Path
|
||||
source_bytes: bytes
|
||||
markdown: str
|
||||
input_sha256: str
|
||||
|
||||
|
||||
def _safe_identifier(value: str, field_name: str) -> None:
|
||||
if not isinstance(value, str) or _SAFE_ID_PATTERN.fullmatch(value) is None or ".." in value:
|
||||
raise ExperimentError(f"{field_name} must be a safe lowercase identifier")
|
||||
|
||||
|
||||
def _run_git(repository_root: Path, *arguments: str) -> str | None:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
("git", *arguments),
|
||||
cwd=repository_root,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
return completed.stdout.strip()
|
||||
|
||||
|
||||
def collect_tool_metadata(repository_root: Path) -> ToolMetadata:
|
||||
"""Collect only the reproducibility facts approved for the manifest."""
|
||||
if not isinstance(repository_root, Path):
|
||||
raise TypeError("repository_root must be a Path")
|
||||
try:
|
||||
package_version = version("mdpolish")
|
||||
except PackageNotFoundError:
|
||||
package_version = "uninstalled"
|
||||
|
||||
git_commit = _run_git(repository_root, "rev-parse", "HEAD")
|
||||
git_status = _run_git(repository_root, "status", "--porcelain")
|
||||
return ToolMetadata(
|
||||
name="mdpolish",
|
||||
package_version=package_version,
|
||||
python_version=platform_module.python_version(),
|
||||
platform=f"{sys.platform}-{platform_module.machine()}",
|
||||
git_commit=git_commit,
|
||||
git_dirty=None if git_status is None else bool(git_status),
|
||||
)
|
||||
|
||||
|
||||
def _utc_text(value: datetime) -> str:
|
||||
if value.tzinfo is None or value.utcoffset() is None:
|
||||
raise ExperimentError("experiment timestamps must be timezone-aware")
|
||||
utc_value = value.astimezone(UTC)
|
||||
return utc_value.isoformat(timespec="seconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def _utc_offset_text(value: datetime) -> str:
|
||||
offset = value.utcoffset()
|
||||
if offset is None:
|
||||
raise ExperimentError("experiment timestamps must be timezone-aware")
|
||||
total_minutes = int(offset.total_seconds() // 60)
|
||||
sign = "+" if total_minutes >= 0 else "-"
|
||||
absolute_minutes = abs(total_minutes)
|
||||
hours, minutes = divmod(absolute_minutes, 60)
|
||||
text = f"{sign}{hours:02d}:{minutes:02d}"
|
||||
if _UTC_OFFSET_PATTERN.fullmatch(text) is None or hours > 23 or minutes > 59:
|
||||
raise ExperimentError("the experiment UTC offset cannot be represented safely")
|
||||
return text
|
||||
|
||||
|
||||
def _prepare_documents(
|
||||
documents: tuple[InputDocument, ...],
|
||||
*,
|
||||
final_directory: Path,
|
||||
) -> tuple[_PreparedDocument, ...]:
|
||||
if not isinstance(documents, tuple) or not documents:
|
||||
raise ExperimentError("documents must be a non-empty tuple")
|
||||
|
||||
seen_ids: set[str] = set()
|
||||
seen_paths: set[Path] = set()
|
||||
prepared: list[_PreparedDocument] = []
|
||||
final_resolved = final_directory.resolve(strict=False)
|
||||
for document in documents:
|
||||
if not isinstance(document, InputDocument):
|
||||
raise ExperimentError("documents must contain only InputDocument values")
|
||||
_safe_identifier(document.document_id, "document_id")
|
||||
if document.document_id in seen_ids:
|
||||
raise ExperimentError("document_id values must be unique")
|
||||
seen_ids.add(document.document_id)
|
||||
if not isinstance(document.source_path, Path):
|
||||
raise ExperimentError("source_path must be a Path")
|
||||
if (
|
||||
not isinstance(document.source_label, str)
|
||||
or not document.source_label.strip()
|
||||
or "\0" in document.source_label
|
||||
):
|
||||
raise ExperimentError("source_label must be a non-empty display string")
|
||||
|
||||
try:
|
||||
resolved_path = document.source_path.resolve(strict=True)
|
||||
except OSError as error:
|
||||
raise ExperimentError(f"input file cannot be resolved: {document.source_label}") from error
|
||||
if not resolved_path.is_file():
|
||||
raise ExperimentError(f"input path is not a regular file: {document.source_label}")
|
||||
if resolved_path in seen_paths:
|
||||
raise ExperimentError("source_path values must be unique")
|
||||
seen_paths.add(resolved_path)
|
||||
if (
|
||||
resolved_path == final_resolved
|
||||
or final_resolved in resolved_path.parents
|
||||
or resolved_path in final_resolved.parents
|
||||
):
|
||||
raise ExperimentError("an input file and target run directory cannot overlap")
|
||||
|
||||
try:
|
||||
source_bytes = resolved_path.read_bytes()
|
||||
markdown = source_bytes.decode("utf-8", errors="strict")
|
||||
except (OSError, UnicodeDecodeError) as error:
|
||||
raise ExperimentError(f"input file must be readable strict UTF-8: {document.source_label}") from error
|
||||
input_sha256 = sha256(source_bytes).hexdigest()
|
||||
if markdown_sha256(markdown) != input_sha256:
|
||||
raise ExperimentError("decoded Markdown does not reproduce the input byte hash")
|
||||
prepared.append(
|
||||
_PreparedDocument(
|
||||
input=document,
|
||||
resolved_path=resolved_path,
|
||||
source_bytes=source_bytes,
|
||||
markdown=markdown,
|
||||
input_sha256=input_sha256,
|
||||
)
|
||||
)
|
||||
return tuple(prepared)
|
||||
|
||||
|
||||
def _overall_status(reports: tuple[DocumentReport, ...]) -> RunStatus:
|
||||
if any(report.status is RunStatus.FAILED for report in reports):
|
||||
return RunStatus.FAILED
|
||||
if any(report.status is RunStatus.UNSTABLE for report in reports):
|
||||
return RunStatus.UNSTABLE
|
||||
return RunStatus.SUCCESS
|
||||
|
||||
|
||||
def _document_index(report: DocumentReport) -> JsonObject:
|
||||
base = f"documents/{report.document_id}"
|
||||
return {
|
||||
"document_id": report.document_id,
|
||||
"source_label": report.source_label,
|
||||
"status": report.status.value,
|
||||
"input_sha256": report.input_sha256,
|
||||
"current_sha256": report.current_sha256,
|
||||
"change_count": report.change_count,
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": f"{base}/cleaned.md" if report.status is RunStatus.SUCCESS else None,
|
||||
"diff_path": f"{base}/changes.diff" if report.status is RunStatus.SUCCESS else None,
|
||||
}
|
||||
|
||||
|
||||
def _manifest_payload(
|
||||
*,
|
||||
run_id: str,
|
||||
run_date: str,
|
||||
utc_offset: str,
|
||||
started_at: datetime,
|
||||
completed_at: datetime,
|
||||
tool: ToolMetadata,
|
||||
components: tuple[ComponentInfo, ...],
|
||||
reports: tuple[DocumentReport, ...],
|
||||
) -> JsonObject:
|
||||
status = _overall_status(reports)
|
||||
component_values: list[JsonValue] = [component_info_json(component) for component in components]
|
||||
document_values: list[JsonValue] = [_document_index(report) for report in reports]
|
||||
success_count = sum(report.status is RunStatus.SUCCESS for report in reports)
|
||||
failed_count = sum(report.status is RunStatus.FAILED for report in reports)
|
||||
unstable_count = sum(report.status is RunStatus.UNSTABLE for report in reports)
|
||||
change_count = sum(report.change_count for report in reports)
|
||||
retention_until = completed_at.astimezone(UTC) + timedelta(days=30)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_date": run_date,
|
||||
"utc_offset": utc_offset,
|
||||
"status": status.value,
|
||||
"started_at_utc": _utc_text(started_at),
|
||||
"completed_at_utc": _utc_text(completed_at),
|
||||
"retention_until": _utc_text(retention_until),
|
||||
},
|
||||
"tool": {
|
||||
"name": tool.name,
|
||||
"package_version": tool.package_version,
|
||||
"python_version": tool.python_version,
|
||||
"platform": tool.platform,
|
||||
"git_commit": tool.git_commit,
|
||||
"git_dirty": tool.git_dirty,
|
||||
},
|
||||
"pipeline": {"components": component_values},
|
||||
"documents": document_values,
|
||||
"summary": {
|
||||
"document_count": len(reports),
|
||||
"success_count": success_count,
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _review_locator_payload(
|
||||
*,
|
||||
run_id: str,
|
||||
run_directory: Path,
|
||||
documents: tuple[_PreparedDocument, ...],
|
||||
) -> JsonObject:
|
||||
document_values: list[JsonValue] = [
|
||||
{
|
||||
"document_id": document.input.document_id,
|
||||
"source_path": str(document.resolved_path),
|
||||
"input_sha256": document.input_sha256,
|
||||
}
|
||||
for document in documents
|
||||
]
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_directory.resolve(strict=False)),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": document_values,
|
||||
}
|
||||
|
||||
|
||||
def run_experiment(
|
||||
*,
|
||||
pipeline: Pipeline,
|
||||
documents: tuple[InputDocument, ...],
|
||||
run_id: str,
|
||||
artifacts_root: Path,
|
||||
started_at: datetime,
|
||||
tool: ToolMetadata,
|
||||
completed_at: datetime | None = None,
|
||||
) -> ExperimentResult:
|
||||
"""Run explicit inputs and publish one complete private artifact directory."""
|
||||
if not isinstance(pipeline, Pipeline):
|
||||
raise TypeError("pipeline must be a Pipeline")
|
||||
_safe_identifier(run_id, "run_id")
|
||||
if not isinstance(artifacts_root, Path):
|
||||
raise TypeError("artifacts_root must be a Path")
|
||||
if not isinstance(tool, ToolMetadata):
|
||||
raise TypeError("tool must be ToolMetadata")
|
||||
if started_at.tzinfo is None or started_at.utcoffset() is None:
|
||||
raise ExperimentError("started_at must be timezone-aware")
|
||||
|
||||
run_date = started_at.date().isoformat()
|
||||
utc_offset = _utc_offset_text(started_at)
|
||||
final_directory = artifacts_root / run_date / "runs" / run_id
|
||||
if final_directory.exists() or final_directory.is_symlink():
|
||||
raise ExperimentError("the target run directory already exists")
|
||||
prepared = _prepare_documents(documents, final_directory=final_directory)
|
||||
|
||||
reports: list[DocumentReport] = []
|
||||
expected_components: tuple[ComponentInfo, ...] | None = None
|
||||
for document in prepared:
|
||||
result = pipeline.transform(document.markdown)
|
||||
if expected_components is None:
|
||||
expected_components = result.components
|
||||
elif result.components != expected_components:
|
||||
raise ExperimentError("pipeline component metadata changed between documents")
|
||||
reports.append(
|
||||
build_document_report(
|
||||
document_id=document.input.document_id,
|
||||
source_label=document.input.source_label,
|
||||
input_markdown=document.markdown,
|
||||
result=result,
|
||||
)
|
||||
)
|
||||
|
||||
for document in prepared:
|
||||
try:
|
||||
current_bytes = document.resolved_path.read_bytes()
|
||||
except OSError as error:
|
||||
raise ExperimentError(
|
||||
f"input file cannot be verified after the run: {document.input.source_label}"
|
||||
) from error
|
||||
if current_bytes != document.source_bytes or sha256(current_bytes).hexdigest() != document.input_sha256:
|
||||
raise ExperimentError(f"input file changed during the run: {document.input.source_label}")
|
||||
|
||||
report_tuple = tuple(reports)
|
||||
if expected_components is None:
|
||||
raise ExperimentError("a non-empty experiment produced no component metadata")
|
||||
completion = datetime.now(UTC) if completed_at is None else completed_at
|
||||
if completion.tzinfo is None or completion.utcoffset() is None:
|
||||
raise ExperimentError("completed_at must be timezone-aware")
|
||||
if completion.astimezone(UTC) < started_at.astimezone(UTC):
|
||||
raise ExperimentError("completed_at cannot be earlier than started_at")
|
||||
|
||||
manifest = _manifest_payload(
|
||||
run_id=run_id,
|
||||
run_date=run_date,
|
||||
utc_offset=utc_offset,
|
||||
started_at=started_at,
|
||||
completed_at=completion,
|
||||
tool=tool,
|
||||
components=expected_components,
|
||||
reports=report_tuple,
|
||||
)
|
||||
stored_documents = tuple(
|
||||
StoredDocument(
|
||||
document_id=report.document_id,
|
||||
result_json=report.result_json,
|
||||
cleaned_markdown=report.cleaned_markdown,
|
||||
diff=report.diff,
|
||||
output_sha256=report.current_sha256 if report.status is RunStatus.SUCCESS else None,
|
||||
)
|
||||
for report in report_tuple
|
||||
)
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
manifest_json=encode_json(manifest),
|
||||
review_locator_json=encode_json(
|
||||
_review_locator_payload(
|
||||
run_id=run_id,
|
||||
run_directory=final_directory,
|
||||
documents=prepared,
|
||||
)
|
||||
),
|
||||
documents=stored_documents,
|
||||
)
|
||||
|
||||
status = _overall_status(report_tuple)
|
||||
return ExperimentResult(
|
||||
run_directory=run_directory,
|
||||
status=status,
|
||||
document_count=len(report_tuple),
|
||||
success_count=sum(report.status is RunStatus.SUCCESS for report in report_tuple),
|
||||
failed_count=sum(report.status is RunStatus.FAILED for report in report_tuple),
|
||||
unstable_count=sum(report.status is RunStatus.UNSTABLE for report in report_tuple),
|
||||
change_count=sum(report.change_count for report in report_tuple),
|
||||
)
|
||||
+19
-19
@@ -94,7 +94,7 @@ class TextEdit:
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ProposedChange:
|
||||
"""One atomic, automatically applicable action proposed by a component."""
|
||||
"""One atomic, automatically applicable action proposed by a modifier."""
|
||||
|
||||
snapshot_sha256: str
|
||||
reason: str
|
||||
@@ -113,10 +113,10 @@ class ProposedChange:
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ComponentInfo:
|
||||
"""Validated, immutable component metadata recorded in a run."""
|
||||
class ModifierInfo:
|
||||
"""Validated, immutable modifier metadata recorded in a run."""
|
||||
|
||||
component_id: str
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
@@ -126,13 +126,13 @@ class ComponentInfo:
|
||||
class ProposalReference:
|
||||
"""A deterministic reference scoped to one transform result."""
|
||||
|
||||
component_position: int
|
||||
modifier_position: int
|
||||
snapshot_sha256: str
|
||||
proposal_index: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if type(self.component_position) is not int or self.component_position < 0:
|
||||
raise ValueError("component_position must be a non-negative integer")
|
||||
if type(self.modifier_position) is not int or self.modifier_position < 0:
|
||||
raise ValueError("modifier_position must be a non-negative integer")
|
||||
if type(self.proposal_index) is not int or self.proposal_index < 0:
|
||||
raise ValueError("proposal_index must be a non-negative integer")
|
||||
_require_sha256(self.snapshot_sha256, "snapshot_sha256")
|
||||
@@ -142,9 +142,9 @@ class ProposalReference:
|
||||
class Change:
|
||||
"""One text edit that was actually applied."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
proposal_ref: ProposalReference
|
||||
edit_index: int
|
||||
reason: str
|
||||
@@ -159,9 +159,9 @@ class Change:
|
||||
class ResidualProposal:
|
||||
"""A valid proposal found by the final read-only stability review."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
proposal_ref: ProposalReference
|
||||
proposal: ProposedChange
|
||||
|
||||
@@ -183,11 +183,11 @@ class ErrorStage(StrEnum):
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RunError:
|
||||
"""A source-safe component, contract, or edit error."""
|
||||
"""A source-safe modifier, contract, or edit error."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
stage: ErrorStage
|
||||
error_type: str
|
||||
message: str
|
||||
@@ -200,7 +200,7 @@ class TransformResult:
|
||||
status: RunStatus
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
components: tuple[ComponentInfo, ...] = ()
|
||||
modifiers: tuple[ModifierInfo, ...] = ()
|
||||
changes: tuple[Change, ...] = ()
|
||||
errors: tuple[RunError, ...] = ()
|
||||
residual_proposals: tuple[ResidualProposal, ...] = ()
|
||||
@@ -212,7 +212,7 @@ class TransformResult:
|
||||
_require_sha256(self.current_sha256, "current_sha256")
|
||||
if not isinstance(self.status, RunStatus):
|
||||
raise TypeError("status must be a RunStatus")
|
||||
for field_name in ("components", "changes", "errors", "residual_proposals"):
|
||||
for field_name in ("modifiers", "changes", "errors", "residual_proposals"):
|
||||
if not isinstance(getattr(self, field_name), tuple):
|
||||
raise TypeError(f"{field_name} must be a tuple")
|
||||
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Functional modifier contract for the mdpolish core."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import cast
|
||||
|
||||
from mdpolish.models import DocumentSnapshot, ModifierInfo, Parameters, ParameterValue, ProposedChange
|
||||
|
||||
ModifierFunction = Callable[[DocumentSnapshot], tuple[ProposedChange, ...]]
|
||||
|
||||
_MODIFIER_ID_PATTERN = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
|
||||
_SEMVER_PATTERN = re.compile(r"^(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)$")
|
||||
|
||||
|
||||
class ModifierContractError(ValueError):
|
||||
"""A modifier does not satisfy the approved extension contract."""
|
||||
|
||||
|
||||
def _freeze_parameter(value: object) -> ParameterValue:
|
||||
if value is None or isinstance(value, (str, bool)) or type(value) is int:
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ModifierContractError("modifier parameters cannot contain non-finite floats")
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
pairs: list[tuple[str, ParameterValue]] = []
|
||||
for key, nested_value in value.items():
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ModifierContractError("modifier parameter mapping keys must be non-empty strings")
|
||||
pairs.append((key, _freeze_parameter(nested_value)))
|
||||
return tuple(sorted(pairs, key=lambda pair: pair[0]))
|
||||
if isinstance(value, (list, tuple)):
|
||||
return tuple(_freeze_parameter(item) for item in value)
|
||||
raise ModifierContractError("modifier parameters must contain only deterministic basic data")
|
||||
|
||||
|
||||
def _freeze_parameters(parameters: Mapping[str, object] | Parameters) -> Parameters:
|
||||
if isinstance(parameters, Mapping):
|
||||
frozen = _freeze_parameter(parameters)
|
||||
if not isinstance(frozen, tuple):
|
||||
raise AssertionError("a mapping must normalize to a tuple")
|
||||
return cast(Parameters, frozen)
|
||||
if not isinstance(parameters, tuple):
|
||||
raise ModifierContractError("modifier parameters must be a mapping or normalized parameter tuple")
|
||||
|
||||
normalized: dict[str, ParameterValue] = {}
|
||||
for pair in parameters:
|
||||
if not isinstance(pair, tuple) or len(pair) != 2:
|
||||
raise ModifierContractError("normalized modifier parameters must contain key-value tuples")
|
||||
key, value = pair
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ModifierContractError("modifier parameter keys must be non-empty strings")
|
||||
if key in normalized:
|
||||
raise ModifierContractError("modifier parameter keys must be unique")
|
||||
normalized[key] = _freeze_parameter(value)
|
||||
return tuple(sorted(normalized.items(), key=lambda pair: pair[0]))
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True, init=False)
|
||||
class Modifier:
|
||||
"""Immutable metadata paired with a pure proposal function."""
|
||||
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
propose: ModifierFunction
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
modifier_id: str,
|
||||
version: str,
|
||||
parameters: Mapping[str, object] | Parameters,
|
||||
applicability: str,
|
||||
propose: ModifierFunction,
|
||||
) -> None:
|
||||
if not isinstance(modifier_id, str) or _MODIFIER_ID_PATTERN.fullmatch(modifier_id) is None:
|
||||
raise ModifierContractError("modifier_id must be a stable lowercase identifier")
|
||||
if not isinstance(version, str) or _SEMVER_PATTERN.fullmatch(version) is None:
|
||||
raise ModifierContractError("modifier version must use MAJOR.MINOR.PATCH")
|
||||
if not isinstance(applicability, str) or not applicability.strip():
|
||||
raise ModifierContractError("modifier applicability must be a non-empty string")
|
||||
if not callable(propose):
|
||||
raise ModifierContractError("modifier propose must be callable")
|
||||
|
||||
object.__setattr__(self, "modifier_id", modifier_id)
|
||||
object.__setattr__(self, "version", version)
|
||||
object.__setattr__(self, "parameters", _freeze_parameters(parameters))
|
||||
object.__setattr__(self, "applicability", applicability)
|
||||
object.__setattr__(self, "propose", propose)
|
||||
|
||||
def _info(self) -> ModifierInfo:
|
||||
return ModifierInfo(
|
||||
modifier_id=self.modifier_id,
|
||||
version=self.version,
|
||||
parameters=self.parameters,
|
||||
applicability=self.applicability,
|
||||
)
|
||||
|
||||
|
||||
def collect_proposals(modifier: Modifier, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
"""Call one modifier and validate the proposal container contract."""
|
||||
if not isinstance(modifier, Modifier):
|
||||
raise ModifierContractError("pipeline entries must be Modifier instances")
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise ModifierContractError("modifiers require a DocumentSnapshot")
|
||||
proposals = modifier.propose(snapshot)
|
||||
if not isinstance(proposals, tuple):
|
||||
raise ModifierContractError("modifier propose must return a tuple")
|
||||
if any(not isinstance(proposal, ProposedChange) for proposal in proposals):
|
||||
raise ModifierContractError("modifier propose must return only ProposedChange values")
|
||||
return proposals
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Small project-independent modifiers shipped as examples and reusable tools."""
|
||||
|
||||
from mdpolish.modifiers.html_table_entities import html_table_entity_unescape
|
||||
from mdpolish.modifiers.html_table_layout import html_table_layout
|
||||
from mdpolish.modifiers.mapped_line_join import LineJoinMapping, mapped_line_join
|
||||
|
||||
__all__ = [
|
||||
"LineJoinMapping",
|
||||
"html_table_entity_unescape",
|
||||
"html_table_layout",
|
||||
"mapped_line_join",
|
||||
]
|
||||
+18
-28
@@ -1,14 +1,12 @@
|
||||
"""Remove one approved escape layer from strict HTML table cell text."""
|
||||
"""Unescape one approved entity layer inside strict HTML table cell text."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._html_table import strict_html_tables
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier
|
||||
|
||||
_ENTITY_PATTERN = re.compile(r"&(?:lt|gt|amp);")
|
||||
_REPLACEMENTS = {
|
||||
@@ -16,33 +14,17 @@ _REPLACEMENTS = {
|
||||
"&gt;": ">",
|
||||
"&amp;": "&",
|
||||
}
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "在严格 HTML 表格单元格文本中解除一层实体转义"
|
||||
|
||||
|
||||
class HtmlTableDoubleEscapeComponent(Component):
|
||||
"""Unescape only approved double entities inside strict cell text."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "markdown.html_table_double_escape"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;，"
|
||||
"排除属性、表格外文本和其他实体。"
|
||||
_APPLICABILITY = (
|
||||
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;；"
|
||||
"不处理属性、表格外文本、其他实体或任意损坏 HTML。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
|
||||
def html_table_entity_unescape() -> Modifier:
|
||||
"""Create the strict HTML table entity modifier."""
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for table in strict_html_tables(snapshot.markdown):
|
||||
for row in table.rows:
|
||||
@@ -65,3 +47,11 @@ class HtmlTableDoubleEscapeComponent(Component):
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="markdown.html_table_entity_unescape",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=_APPLICABILITY,
|
||||
propose=propose,
|
||||
)
|
||||
+16
-27
@@ -2,41 +2,22 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._html_table import strict_html_tables
|
||||
from mdpolish._text_ranges import line_ending_styles
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier
|
||||
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "展开严格单行 HTML 表格的行布局"
|
||||
|
||||
|
||||
class HtmlTableLayoutComponent(Component):
|
||||
"""Preserve strict HTML table bytes while placing each row on its own line."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "markdown.html_table_layout"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
_APPLICABILITY = (
|
||||
"处理严格完整且当前为单物理行的 HTML 表格，只规范 table/tr 外层布局；"
|
||||
"排除混合行尾、复杂 HTML 和已是多行的表格。"
|
||||
"不处理混合行尾、复杂 HTML 或已经换行的表格。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
|
||||
def html_table_layout() -> Modifier:
|
||||
"""Create the strict one-line HTML table layout modifier."""
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
styles = line_ending_styles(snapshot.markdown)
|
||||
if len(styles) > 1:
|
||||
return ()
|
||||
@@ -66,3 +47,11 @@ class HtmlTableLayoutComponent(Component):
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="markdown.html_table_layout",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=_APPLICABILITY,
|
||||
propose=propose,
|
||||
)
|
||||
+42
-49
@@ -1,78 +1,55 @@
|
||||
"""Join approved word fragments split across adjacent physical lines."""
|
||||
"""Join exact caller-mapped fragments across nearby physical lines."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from collections.abc import Iterable, Mapping
|
||||
from collections.abc import Iterable
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import physical_lines
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier, ModifierContractError
|
||||
|
||||
WordJoinMapping = tuple[str, str, str]
|
||||
LineJoinMapping = tuple[str, str, str]
|
||||
|
||||
_APPLICABILITY = (
|
||||
"处理调用方显式提供的左右片段映射；片段必须位于相邻物理行或只隔一个空行，并满足 ASCII 词边界；"
|
||||
"不猜测未配置词语或段落结构。"
|
||||
)
|
||||
|
||||
|
||||
def _is_ascii_letter(character: str) -> bool:
|
||||
return character in "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||
|
||||
|
||||
def _validated_mappings(mappings: Iterable[WordJoinMapping]) -> tuple[WordJoinMapping, ...]:
|
||||
normalized: list[WordJoinMapping] = []
|
||||
def _validated_mappings(mappings: Iterable[LineJoinMapping]) -> tuple[LineJoinMapping, ...]:
|
||||
normalized: list[LineJoinMapping] = []
|
||||
seen_pairs: set[tuple[str, str]] = set()
|
||||
for mapping in mappings:
|
||||
try:
|
||||
iterator = iter(mappings)
|
||||
except TypeError as error:
|
||||
raise ModifierContractError("line join mappings must be iterable") from error
|
||||
for mapping in iterator:
|
||||
if not isinstance(mapping, tuple) or len(mapping) != 3 or any(not isinstance(value, str) for value in mapping):
|
||||
raise ComponentContractError("word join mappings must be three-string tuples")
|
||||
raise ModifierContractError("line join mappings must be three-string tuples")
|
||||
left, right, replacement = mapping
|
||||
if not left or not right or not replacement:
|
||||
raise ComponentContractError("word join mapping fields must be non-empty")
|
||||
raise ModifierContractError("line join mapping fields must be non-empty")
|
||||
if any(character.isspace() for value in mapping for character in value):
|
||||
raise ComponentContractError("word join mapping fields cannot contain whitespace")
|
||||
raise ModifierContractError("line join mapping fields cannot contain whitespace")
|
||||
pair = (left, right)
|
||||
if pair in seen_pairs:
|
||||
raise ComponentContractError("word join mappings cannot repeat a fragment pair")
|
||||
raise ModifierContractError("line join mappings cannot repeat a fragment pair")
|
||||
seen_pairs.add(pair)
|
||||
normalized.append(mapping)
|
||||
return tuple(sorted(normalized))
|
||||
|
||||
|
||||
class PageBreakWordJoinComponent(Component):
|
||||
"""Apply only caller-approved exact fragment-to-word mappings."""
|
||||
def mapped_line_join(mappings: Iterable[LineJoinMapping]) -> Modifier:
|
||||
"""Create a modifier from exact left-fragment, right-fragment, replacement mappings."""
|
||||
normalized_mappings = _validated_mappings(mappings)
|
||||
|
||||
def __init__(self, mappings: Iterable[WordJoinMapping]) -> None:
|
||||
self._mappings = _validated_mappings(mappings)
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.page_break_word_join"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
records = tuple(
|
||||
MappingProxyType(
|
||||
{
|
||||
"left_fragment": left,
|
||||
"right_fragment": right,
|
||||
"replacement": replacement,
|
||||
}
|
||||
)
|
||||
for left, right, replacement in self._mappings
|
||||
)
|
||||
return MappingProxyType({"mappings": records})
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理显式批准的左右词片段，片段必须位于相邻物理行或只隔一个空行并满足 ASCII 词边界；"
|
||||
"排除未配置词和段落级猜测。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
matches: list[tuple[int, int, str, str, str]] = []
|
||||
for index, left_line in enumerate(lines[:-1]):
|
||||
@@ -80,7 +57,7 @@ class PageBreakWordJoinComponent(Component):
|
||||
left_ending = left_line.line_ending(snapshot.markdown)
|
||||
if not left_ending:
|
||||
continue
|
||||
for left, right, replacement in self._mappings:
|
||||
for left, right, replacement in normalized_mappings:
|
||||
if not left_text.endswith(left):
|
||||
continue
|
||||
left_start = left_line.content_end - len(left)
|
||||
@@ -113,7 +90,7 @@ class PageBreakWordJoinComponent(Component):
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=f"按已批准映射修复跨页断词：{left} + {right} → {replacement}",
|
||||
reason=f"按显式映射合并跨行片段：{left} + {right} → {replacement}",
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
@@ -125,3 +102,19 @@ class PageBreakWordJoinComponent(Component):
|
||||
)
|
||||
for start, end, replacement, left, right in unique_matches
|
||||
)
|
||||
|
||||
records = tuple(
|
||||
{
|
||||
"left_fragment": left,
|
||||
"replacement": replacement,
|
||||
"right_fragment": right,
|
||||
}
|
||||
for left, right, replacement in normalized_mappings
|
||||
)
|
||||
return Modifier(
|
||||
modifier_id="markdown.mapped_line_join",
|
||||
version="1.0.0",
|
||||
parameters={"mappings": records},
|
||||
applicability=_APPLICABILITY,
|
||||
propose=propose,
|
||||
)
|
||||
+103
-78
@@ -1,16 +1,15 @@
|
||||
"""Sequential orchestration and final stability review."""
|
||||
"""Sequential modifier orchestration and final stability review."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.edits import apply_component_batch, validate_component_batch
|
||||
from mdpolish.edits import apply_modifier_batch, validate_modifier_batch
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
@@ -18,6 +17,7 @@ from mdpolish.models import (
|
||||
RunStatus,
|
||||
TransformResult,
|
||||
)
|
||||
from mdpolish.modifier import Modifier, ModifierContractError, collect_proposals
|
||||
|
||||
|
||||
class PipelineContractError(ValueError):
|
||||
@@ -25,32 +25,32 @@ class PipelineContractError(ValueError):
|
||||
|
||||
|
||||
class Pipeline:
|
||||
"""Run selected components once, then review the final snapshot for stability."""
|
||||
"""Run selected modifiers once, then review the final snapshot for stability."""
|
||||
|
||||
def __init__(self, components: Iterable[Component]) -> None:
|
||||
self._components = tuple(components)
|
||||
def __init__(self, modifiers: Iterable[Modifier]) -> None:
|
||||
self._modifiers = tuple(modifiers)
|
||||
|
||||
@property
|
||||
def components(self) -> tuple[Component, ...]:
|
||||
return self._components
|
||||
def modifiers(self) -> tuple[Modifier, ...]:
|
||||
return self._modifiers
|
||||
|
||||
def transform(self, markdown: str) -> TransformResult:
|
||||
input_snapshot = DocumentSnapshot(markdown)
|
||||
component_infos, preflight_error = self._preflight_components()
|
||||
modifier_infos, preflight_error = self._preflight_modifiers()
|
||||
if preflight_error is not None:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=input_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=(),
|
||||
errors=(preflight_error,),
|
||||
)
|
||||
|
||||
current_snapshot = input_snapshot
|
||||
changes: list[Change] = []
|
||||
for position, (component, expected_info) in enumerate(zip(self._components, component_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_component_info(
|
||||
component=component,
|
||||
for position, (modifier, expected_info) in enumerate(zip(self._modifiers, modifier_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
@@ -59,14 +59,14 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(metadata_error,),
|
||||
)
|
||||
|
||||
proposals, proposal_error = self._proposals(
|
||||
component=component,
|
||||
component_info=current_info,
|
||||
modifier=modifier,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
snapshot=current_snapshot,
|
||||
@@ -75,31 +75,46 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(proposal_error,),
|
||||
)
|
||||
|
||||
_, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
)
|
||||
if metadata_error is not None:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(metadata_error,),
|
||||
)
|
||||
|
||||
try:
|
||||
applied_batch = apply_component_batch(
|
||||
applied_batch = apply_modifier_batch(
|
||||
snapshot=current_snapshot,
|
||||
proposals=proposals,
|
||||
component=current_info,
|
||||
component_position=position,
|
||||
modifier=current_info,
|
||||
modifier_position=position,
|
||||
)
|
||||
except Exception as error:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(
|
||||
self._run_error(
|
||||
component_info=current_info,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="component edit batch does not satisfy the edit contract",
|
||||
unexpected_message="modifier edit batch does not satisfy the edit contract",
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -109,9 +124,9 @@ class Pipeline:
|
||||
|
||||
review_errors: list[RunError] = []
|
||||
residual_proposals: list[ResidualProposal] = []
|
||||
for position, (component, expected_info) in enumerate(zip(self._components, component_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_component_info(
|
||||
component=component,
|
||||
for position, (modifier, expected_info) in enumerate(zip(self._modifiers, modifier_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
@@ -121,8 +136,8 @@ class Pipeline:
|
||||
continue
|
||||
|
||||
proposals, proposal_error = self._proposals(
|
||||
component=component,
|
||||
component_info=current_info,
|
||||
modifier=modifier,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
snapshot=current_snapshot,
|
||||
@@ -131,27 +146,37 @@ class Pipeline:
|
||||
review_errors.append(proposal_error)
|
||||
continue
|
||||
|
||||
_, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
)
|
||||
if metadata_error is not None:
|
||||
review_errors.append(metadata_error)
|
||||
continue
|
||||
|
||||
try:
|
||||
validate_component_batch(current_snapshot, proposals)
|
||||
validate_modifier_batch(current_snapshot, proposals)
|
||||
except Exception as error:
|
||||
review_errors.append(
|
||||
self._run_error(
|
||||
component_info=current_info,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
error=error,
|
||||
unexpected_message="component edit batch does not satisfy the edit contract",
|
||||
unexpected_message="modifier edit batch does not satisfy the edit contract",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
residual_proposals.extend(
|
||||
ResidualProposal(
|
||||
component_id=current_info.component_id,
|
||||
component_version=current_info.version,
|
||||
component_position=position,
|
||||
modifier_id=current_info.modifier_id,
|
||||
modifier_version=current_info.version,
|
||||
modifier_position=position,
|
||||
proposal_ref=ProposalReference(
|
||||
component_position=position,
|
||||
modifier_position=position,
|
||||
snapshot_sha256=current_snapshot.sha256,
|
||||
proposal_index=proposal_index,
|
||||
),
|
||||
@@ -164,7 +189,7 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=tuple(review_errors),
|
||||
residual_proposals=tuple(residual_proposals),
|
||||
@@ -174,7 +199,7 @@ class Pipeline:
|
||||
status=RunStatus.UNSTABLE,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
residual_proposals=tuple(residual_proposals),
|
||||
partial_markdown=current_snapshot.markdown,
|
||||
@@ -183,100 +208,100 @@ class Pipeline:
|
||||
status=RunStatus.SUCCESS,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
output_markdown=current_snapshot.markdown,
|
||||
)
|
||||
|
||||
def _preflight_components(self) -> tuple[tuple[ComponentInfo, ...], RunError | None]:
|
||||
component_infos: list[ComponentInfo] = []
|
||||
def _preflight_modifiers(self) -> tuple[tuple[ModifierInfo, ...], RunError | None]:
|
||||
modifier_infos: list[ModifierInfo] = []
|
||||
seen_ids: set[str] = set()
|
||||
for position, component in enumerate(self._components):
|
||||
if not isinstance(component, Component):
|
||||
error = ComponentContractError("pipeline entries must be Component instances")
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=None,
|
||||
for position, modifier in enumerate(self._modifiers):
|
||||
if not isinstance(modifier, Modifier):
|
||||
error = ModifierContractError("pipeline entries must be Modifier instances")
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=None,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="a pipeline entry is not a Component instance",
|
||||
unexpected_message="a pipeline entry is not a Modifier instance",
|
||||
)
|
||||
try:
|
||||
component_info = component._component_info()
|
||||
modifier_info = modifier._info()
|
||||
except Exception as error:
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=None,
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=None,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="component metadata does not satisfy the component contract",
|
||||
unexpected_message="modifier metadata does not satisfy the modifier contract",
|
||||
)
|
||||
component_infos.append(component_info)
|
||||
if component_info.component_id in seen_ids:
|
||||
duplicate_error = PipelineContractError("pipeline component_id values must be unique")
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=component_info,
|
||||
modifier_infos.append(modifier_info)
|
||||
if modifier_info.modifier_id in seen_ids:
|
||||
duplicate_error = PipelineContractError("pipeline modifier_id values must be unique")
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=modifier_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=duplicate_error,
|
||||
unexpected_message="pipeline component_id values must be unique",
|
||||
unexpected_message="pipeline modifier_id values must be unique",
|
||||
)
|
||||
seen_ids.add(component_info.component_id)
|
||||
return tuple(component_infos), None
|
||||
seen_ids.add(modifier_info.modifier_id)
|
||||
return tuple(modifier_infos), None
|
||||
|
||||
def _current_component_info(
|
||||
def _current_modifier_info(
|
||||
self,
|
||||
component: Component,
|
||||
expected_info: ComponentInfo,
|
||||
modifier: Modifier,
|
||||
expected_info: ModifierInfo,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
) -> tuple[ComponentInfo, RunError | None]:
|
||||
) -> tuple[ModifierInfo, RunError | None]:
|
||||
try:
|
||||
current_info = component._component_info()
|
||||
current_info = modifier._info()
|
||||
if current_info != expected_info:
|
||||
raise ComponentContractError("component metadata changed during pipeline execution")
|
||||
raise ModifierContractError("modifier metadata changed during pipeline execution")
|
||||
except Exception as error:
|
||||
return expected_info, self._run_error(
|
||||
component_info=expected_info,
|
||||
modifier_info=expected_info,
|
||||
position=position,
|
||||
stage=stage,
|
||||
error=error,
|
||||
unexpected_message="component metadata does not satisfy the component contract",
|
||||
unexpected_message="modifier metadata does not satisfy the modifier contract",
|
||||
)
|
||||
return current_info, None
|
||||
|
||||
def _proposals(
|
||||
self,
|
||||
component: Component,
|
||||
component_info: ComponentInfo,
|
||||
modifier: Modifier,
|
||||
modifier_info: ModifierInfo,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
snapshot: DocumentSnapshot,
|
||||
) -> tuple[tuple[ProposedChange, ...], RunError | None]:
|
||||
try:
|
||||
proposals = component._collect_proposals(snapshot)
|
||||
proposals = collect_proposals(modifier, snapshot)
|
||||
except Exception as error:
|
||||
return (), self._run_error(
|
||||
component_info=component_info,
|
||||
modifier_info=modifier_info,
|
||||
position=position,
|
||||
stage=stage,
|
||||
error=error,
|
||||
unexpected_message="component could not produce contract-valid proposals",
|
||||
unexpected_message="modifier could not produce contract-valid proposals",
|
||||
)
|
||||
return proposals, None
|
||||
|
||||
@staticmethod
|
||||
def _run_error(
|
||||
component_info: ComponentInfo | None,
|
||||
modifier_info: ModifierInfo | None,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
error: Exception,
|
||||
unexpected_message: str,
|
||||
) -> RunError:
|
||||
return RunError(
|
||||
component_id=component_info.component_id if component_info is not None else "<invalid>",
|
||||
component_version=component_info.version if component_info is not None else "<invalid>",
|
||||
component_position=position,
|
||||
modifier_id=modifier_info.modifier_id if modifier_info is not None else "<invalid>",
|
||||
modifier_version=modifier_info.version if modifier_info is not None else "<invalid>",
|
||||
modifier_position=position,
|
||||
stage=stage,
|
||||
error_type=type(error).__name__,
|
||||
message=unexpected_message,
|
||||
@@ -286,7 +311,7 @@ class Pipeline:
|
||||
def _failed_result(
|
||||
input_snapshot: DocumentSnapshot,
|
||||
current_snapshot: DocumentSnapshot,
|
||||
component_infos: tuple[ComponentInfo, ...],
|
||||
modifier_infos: tuple[ModifierInfo, ...],
|
||||
changes: tuple[Change, ...],
|
||||
errors: tuple[RunError, ...],
|
||||
residual_proposals: tuple[ResidualProposal, ...] = (),
|
||||
@@ -295,7 +320,7 @@ class Pipeline:
|
||||
status=RunStatus.FAILED,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=changes,
|
||||
errors=errors,
|
||||
residual_proposals=residual_proposals,
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Safe factory for exact regular-expression replacements."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier, ModifierContractError
|
||||
|
||||
_DEFAULT_APPLICABILITY = "处理调用方正则表达式明确匹配的文本；不推断文档结构，也不自动启用。"
|
||||
_REASON = "应用调用方声明的正则表达式替换"
|
||||
|
||||
|
||||
def regex_replace(
|
||||
*,
|
||||
modifier_id: str,
|
||||
version: str,
|
||||
pattern: str,
|
||||
replacement: str,
|
||||
flags: int | re.RegexFlag = 0,
|
||||
applicability: str = _DEFAULT_APPLICABILITY,
|
||||
) -> Modifier:
|
||||
"""Create a modifier that turns non-empty regex matches into exact edits."""
|
||||
if not isinstance(pattern, str):
|
||||
raise ModifierContractError("regex pattern must be a string")
|
||||
if not isinstance(replacement, str):
|
||||
raise ModifierContractError("regex replacement must be a string")
|
||||
if type(flags) is not int and not isinstance(flags, re.RegexFlag):
|
||||
raise ModifierContractError("regex flags must be an integer or RegexFlag")
|
||||
try:
|
||||
compiled = re.compile(pattern, flags)
|
||||
except (re.error, ValueError) as error:
|
||||
raise ModifierContractError("regex pattern and flags must compile successfully") from error
|
||||
|
||||
empty_match = compiled.search("")
|
||||
if empty_match is not None and empty_match.start() == empty_match.end():
|
||||
raise ModifierContractError("regex patterns that produce zero-length matches are not supported")
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
proposals: list[ProposedChange] = []
|
||||
for match in compiled.finditer(snapshot.markdown):
|
||||
if match.start() == match.end():
|
||||
raise ModifierContractError("regex patterns that produce zero-length matches are not supported")
|
||||
expanded = match.expand(replacement)
|
||||
expected = match.group()
|
||||
if expected == expanded:
|
||||
continue
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(match.start(), match.end()),
|
||||
expected_text=expected,
|
||||
replacement=expanded,
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version=version,
|
||||
parameters={"flags": int(flags), "pattern": pattern, "replacement": replacement},
|
||||
applicability=applicability,
|
||||
propose=propose,
|
||||
)
|
||||
@@ -1,258 +0,0 @@
|
||||
"""Pure JSON and diff representations for local experiment results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from difflib import unified_diff
|
||||
from typing import TypeAlias
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
replay_change_chain,
|
||||
)
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
ParameterValue,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
RunError,
|
||||
RunStatus,
|
||||
TextEdit,
|
||||
TransformResult,
|
||||
)
|
||||
|
||||
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||
|
||||
|
||||
ReportingError = ReplayError
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DocumentReport:
|
||||
"""Rendered artifacts and index facts for one document result."""
|
||||
|
||||
document_id: str
|
||||
source_label: str
|
||||
status: RunStatus
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
change_count: int
|
||||
result_json: bytes
|
||||
cleaned_markdown: bytes | None
|
||||
diff: bytes | None
|
||||
|
||||
|
||||
def _parameter_value_json(value: ParameterValue) -> JsonValue:
|
||||
if isinstance(value, tuple):
|
||||
return [_parameter_value_json(item) for item in value]
|
||||
return value
|
||||
|
||||
|
||||
def component_info_json(component: ComponentInfo) -> JsonObject:
|
||||
"""Represent validated component metadata without changing tuple order."""
|
||||
parameters: list[JsonValue] = []
|
||||
for name, value in component.parameters:
|
||||
parameters.append([name, _parameter_value_json(value)])
|
||||
return {
|
||||
"component_id": component.component_id,
|
||||
"version": component.version,
|
||||
"parameters": parameters,
|
||||
"applicability": component.applicability,
|
||||
}
|
||||
|
||||
|
||||
def _proposal_reference_json(change: Change) -> JsonObject:
|
||||
reference = change.proposal_ref
|
||||
return {
|
||||
"component_position": reference.component_position,
|
||||
"snapshot_sha256": reference.snapshot_sha256,
|
||||
"proposal_index": reference.proposal_index,
|
||||
}
|
||||
|
||||
|
||||
def _text_edit_json(edit: TextEdit) -> JsonObject:
|
||||
return {
|
||||
"snapshot_sha256": edit.snapshot_sha256,
|
||||
"span": {"start": edit.span.start, "end": edit.span.end},
|
||||
"expected_text": edit.expected_text,
|
||||
"replacement": edit.replacement,
|
||||
}
|
||||
|
||||
|
||||
def _proposed_change_json(proposal: ProposedChange) -> JsonObject:
|
||||
edits: list[JsonValue] = [_text_edit_json(edit) for edit in proposal.edits]
|
||||
return {
|
||||
"snapshot_sha256": proposal.snapshot_sha256,
|
||||
"reason": proposal.reason,
|
||||
"edits": edits,
|
||||
}
|
||||
|
||||
|
||||
def _residual_proposal_json(residual: ResidualProposal) -> JsonObject:
|
||||
reference = residual.proposal_ref
|
||||
return {
|
||||
"component_id": residual.component_id,
|
||||
"component_version": residual.component_version,
|
||||
"component_position": residual.component_position,
|
||||
"proposal_ref": {
|
||||
"component_position": reference.component_position,
|
||||
"snapshot_sha256": reference.snapshot_sha256,
|
||||
"proposal_index": reference.proposal_index,
|
||||
},
|
||||
"proposal": _proposed_change_json(residual.proposal),
|
||||
}
|
||||
|
||||
|
||||
def _run_error_json(error: RunError) -> JsonObject:
|
||||
return {
|
||||
"component_id": error.component_id,
|
||||
"component_version": error.component_version,
|
||||
"component_position": error.component_position,
|
||||
"stage": error.stage.value,
|
||||
"error_type": error.error_type,
|
||||
"message": error.message,
|
||||
}
|
||||
|
||||
|
||||
def _replay_changes(input_markdown: str, result: TransformResult) -> tuple[tuple[tuple[int, int], ...], str]:
|
||||
current_markdown = result.output_markdown if result.status is RunStatus.SUCCESS else result.partial_markdown
|
||||
if current_markdown is None:
|
||||
raise ReportingError("a transform result does not contain its status-specific Markdown")
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=input_markdown,
|
||||
input_sha256=result.input_sha256,
|
||||
components=tuple(ReplayComponent(item.component_id, item.version) for item in result.components),
|
||||
changes=tuple(
|
||||
ReplayChange(
|
||||
component_id=change.component_id,
|
||||
component_version=change.component_version,
|
||||
component_position=change.component_position,
|
||||
proposal_component_position=change.proposal_ref.component_position,
|
||||
proposal_snapshot_sha256=change.proposal_ref.snapshot_sha256,
|
||||
proposal_index=change.proposal_ref.proposal_index,
|
||||
edit_index=change.edit_index,
|
||||
start=change.span.start,
|
||||
end=change.span.end,
|
||||
before=change.before,
|
||||
after=change.after,
|
||||
before_sha256=change.before_sha256,
|
||||
after_sha256=change.after_sha256,
|
||||
)
|
||||
for change in result.changes
|
||||
),
|
||||
current_sha256=result.current_sha256,
|
||||
current_markdown=current_markdown,
|
||||
include_zero_change_stages=False,
|
||||
)
|
||||
locations = tuple((item.line, item.column) for item in replayed.changes)
|
||||
return locations, replayed.current_markdown
|
||||
|
||||
|
||||
def _change_json(change: Change, location: tuple[int, int]) -> JsonObject:
|
||||
line, column = location
|
||||
return {
|
||||
"component_id": change.component_id,
|
||||
"component_version": change.component_version,
|
||||
"component_position": change.component_position,
|
||||
"proposal_ref": _proposal_reference_json(change),
|
||||
"edit_index": change.edit_index,
|
||||
"reason": change.reason,
|
||||
"span": {"start": change.span.start, "end": change.span.end},
|
||||
"location": {"line": line, "column": column},
|
||||
"before": change.before,
|
||||
"after": change.after,
|
||||
"before_sha256": change.before_sha256,
|
||||
"after_sha256": change.after_sha256,
|
||||
}
|
||||
|
||||
|
||||
def encode_json(payload: JsonObject) -> bytes:
|
||||
"""Encode a schema object using the approved stable local representation."""
|
||||
return (json.dumps(payload, ensure_ascii=False, indent=2) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def build_unified_diff(document_id: str, before: str, after: str) -> bytes:
|
||||
"""Build the human review view using logical document names only."""
|
||||
if before == after:
|
||||
return b""
|
||||
lines = unified_diff(
|
||||
before.splitlines(keepends=True),
|
||||
after.splitlines(keepends=True),
|
||||
fromfile=f"a/{document_id}.md",
|
||||
tofile=f"b/{document_id}.md",
|
||||
n=3,
|
||||
lineterm="\n",
|
||||
)
|
||||
normalized: list[str] = []
|
||||
line_endings = ("\r\n", "\n", "\r", "\v", "\f", "\x1c", "\x1d", "\x1e", "\x85", "\u2028", "\u2029")
|
||||
for line in lines:
|
||||
for ending in line_endings:
|
||||
if line.endswith(ending):
|
||||
normalized.append(line[: -len(ending)] + "\n")
|
||||
break
|
||||
else:
|
||||
normalized.append(line + "\n\\ No newline at end of file\n")
|
||||
return "".join(normalized).encode("utf-8")
|
||||
|
||||
|
||||
def build_document_report(
|
||||
*,
|
||||
document_id: str,
|
||||
source_label: str,
|
||||
input_markdown: str,
|
||||
result: TransformResult,
|
||||
) -> DocumentReport:
|
||||
"""Render one core result after replaying and verifying every applied batch."""
|
||||
locations, current_markdown = _replay_changes(input_markdown, result)
|
||||
changes: list[JsonValue] = [
|
||||
_change_json(change, location) for change, location in zip(result.changes, locations, strict=True)
|
||||
]
|
||||
errors: list[JsonValue] = [_run_error_json(error) for error in result.errors]
|
||||
residual_proposals: list[JsonValue] = [
|
||||
_residual_proposal_json(residual) for residual in result.residual_proposals
|
||||
]
|
||||
|
||||
if result.status is RunStatus.SUCCESS:
|
||||
cleaned_path: JsonValue = "cleaned.md"
|
||||
diff_path: JsonValue = "changes.diff"
|
||||
cleaned_markdown = current_markdown.encode("utf-8")
|
||||
diff = build_unified_diff(document_id, input_markdown, current_markdown)
|
||||
else:
|
||||
cleaned_path = None
|
||||
diff_path = None
|
||||
cleaned_markdown = None
|
||||
diff = None
|
||||
|
||||
payload: JsonObject = {
|
||||
"schema_version": 1,
|
||||
"document": {
|
||||
"document_id": document_id,
|
||||
"source_label": source_label,
|
||||
},
|
||||
"status": result.status.value,
|
||||
"input_sha256": result.input_sha256,
|
||||
"current_sha256": result.current_sha256,
|
||||
"changes": changes,
|
||||
"errors": errors,
|
||||
"residual_proposals": residual_proposals,
|
||||
"output": {
|
||||
"cleaned_path": cleaned_path,
|
||||
"diff_path": diff_path,
|
||||
},
|
||||
}
|
||||
return DocumentReport(
|
||||
document_id=document_id,
|
||||
source_label=source_label,
|
||||
status=result.status,
|
||||
input_sha256=result.input_sha256,
|
||||
current_sha256=result.current_sha256,
|
||||
change_count=len(result.changes),
|
||||
result_json=encode_json(payload),
|
||||
cleaned_markdown=cleaned_markdown,
|
||||
diff=diff,
|
||||
)
|
||||
@@ -1,212 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import Any, TypedDict, cast
|
||||
|
||||
|
||||
class ReviewFixture(TypedDict):
|
||||
run_directory: Path
|
||||
source_path: Path
|
||||
manifest_path: Path
|
||||
locator_path: Path
|
||||
result_path: Path
|
||||
original: str
|
||||
cleaned: str
|
||||
|
||||
|
||||
def digest(text: str) -> str:
|
||||
return sha256(text.encode()).hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, payload: object) -> None:
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
return cast(dict[str, Any], json.loads(path.read_text(encoding="utf-8")))
|
||||
|
||||
|
||||
def create_review_run(
|
||||
root: Path,
|
||||
*,
|
||||
status: str = "success",
|
||||
with_locator: bool = True,
|
||||
) -> ReviewFixture:
|
||||
run_directory = root / "artifacts/2026-08-23/runs/review-run"
|
||||
document_directory = run_directory / "documents/paper"
|
||||
document_directory.mkdir(parents=True)
|
||||
source_path = root / "paper.md"
|
||||
original = "\ufeff😀 old\r\nCafe\u0301\n"
|
||||
cleaned = "\ufeff😀 new\r\nCafe\u0301\n"
|
||||
source_path.write_text(original, encoding="utf-8")
|
||||
input_hash = digest(original)
|
||||
current_hash = digest(cleaned) if status == "success" else input_hash
|
||||
changes: list[dict[str, object]] = []
|
||||
if status == "success":
|
||||
changes.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"proposal_ref": {
|
||||
"component_position": 0,
|
||||
"snapshot_sha256": input_hash,
|
||||
"proposal_index": 0,
|
||||
},
|
||||
"edit_index": 0,
|
||||
"reason": "替换测试单词",
|
||||
"span": {"start": 3, "end": 6},
|
||||
"location": {"line": 1, "column": 4},
|
||||
"before": "old",
|
||||
"after": "new",
|
||||
"before_sha256": input_hash,
|
||||
"after_sha256": current_hash,
|
||||
}
|
||||
)
|
||||
errors: list[dict[str, object]] = []
|
||||
if status == "failed":
|
||||
errors.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"stage": "transform",
|
||||
"error_type": "SyntheticError",
|
||||
"message": "测试组件失败。",
|
||||
}
|
||||
)
|
||||
residuals: list[dict[str, object]] = []
|
||||
if status == "unstable":
|
||||
residuals.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"proposal_ref": {
|
||||
"component_position": 0,
|
||||
"snapshot_sha256": input_hash,
|
||||
"proposal_index": 0,
|
||||
},
|
||||
"proposal": {
|
||||
"snapshot_sha256": input_hash,
|
||||
"reason": "仍可替换测试单词",
|
||||
"edits": [
|
||||
{
|
||||
"snapshot_sha256": input_hash,
|
||||
"span": {"start": 3, "end": 6},
|
||||
"expected_text": "old",
|
||||
"replacement": "new",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"document": {"document_id": "paper", "source_label": "inputs/paper.md"},
|
||||
"status": status,
|
||||
"input_sha256": input_hash,
|
||||
"current_sha256": current_hash,
|
||||
"changes": changes,
|
||||
"errors": errors,
|
||||
"residual_proposals": residuals,
|
||||
"output": {
|
||||
"cleaned_path": "cleaned.md" if status == "success" else None,
|
||||
"diff_path": "changes.diff" if status == "success" else None,
|
||||
},
|
||||
}
|
||||
result_path = document_directory / "result.json"
|
||||
write_json(result_path, result)
|
||||
if status == "success":
|
||||
(document_directory / "cleaned.md").write_text(cleaned, encoding="utf-8")
|
||||
(document_directory / "changes.diff").write_text("synthetic diff\n", encoding="utf-8")
|
||||
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "review-run",
|
||||
"run_date": "2026-08-23",
|
||||
"utc_offset": "+08:00",
|
||||
"status": status,
|
||||
"started_at_utc": "2026-08-23T01:00:00Z",
|
||||
"completed_at_utc": "2026-08-23T01:01:00Z",
|
||||
"retention_until": "2026-09-22T01:01:00Z",
|
||||
},
|
||||
"tool": {
|
||||
"name": "mdpolish",
|
||||
"package_version": "0.1.0",
|
||||
"python_version": "3.13.11",
|
||||
"platform": "linux-x86_64",
|
||||
"git_commit": None,
|
||||
"git_dirty": None,
|
||||
},
|
||||
"pipeline": {
|
||||
"components": [
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"version": "1.0.0",
|
||||
"parameters": [],
|
||||
"applicability": "替换测试单词。",
|
||||
},
|
||||
{
|
||||
"component_id": "paper.zero",
|
||||
"version": "1.0.0",
|
||||
"parameters": [],
|
||||
"applicability": "不修改当前测试文档。",
|
||||
},
|
||||
]
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "paper",
|
||||
"source_label": "inputs/paper.md",
|
||||
"status": status,
|
||||
"input_sha256": input_hash,
|
||||
"current_sha256": current_hash,
|
||||
"change_count": len(changes),
|
||||
"result_path": "documents/paper/result.json",
|
||||
"cleaned_path": "documents/paper/cleaned.md" if status == "success" else None,
|
||||
"diff_path": "documents/paper/changes.diff" if status == "success" else None,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"document_count": 1,
|
||||
"success_count": int(status == "success"),
|
||||
"failed_count": int(status == "failed"),
|
||||
"unstable_count": int(status == "unstable"),
|
||||
"change_count": len(changes),
|
||||
},
|
||||
}
|
||||
manifest_path = run_directory / "manifest.json"
|
||||
write_json(manifest_path, manifest)
|
||||
locator_path = run_directory / "review-locator.json"
|
||||
if with_locator:
|
||||
write_json(
|
||||
locator_path,
|
||||
{
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "review-run",
|
||||
"run_directory": str(run_directory.resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "paper",
|
||||
"source_path": str(source_path.resolve()),
|
||||
"input_sha256": input_hash,
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
return {
|
||||
"run_directory": run_directory,
|
||||
"source_path": source_path,
|
||||
"manifest_path": manifest_path,
|
||||
"locator_path": locator_path,
|
||||
"result_path": result_path,
|
||||
"original": original,
|
||||
"cleaned": cleaned,
|
||||
}
|
||||
@@ -1,189 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
replay_change_chain,
|
||||
)
|
||||
from mdpolish.models import markdown_sha256
|
||||
|
||||
|
||||
def change(
|
||||
*,
|
||||
component_position: int,
|
||||
before_text: str,
|
||||
after_text: str,
|
||||
start: int,
|
||||
end: int,
|
||||
before_sha256: str,
|
||||
after_sha256: str,
|
||||
proposal_index: int = 0,
|
||||
edit_index: int = 0,
|
||||
) -> ReplayChange:
|
||||
return ReplayChange(
|
||||
component_id=f"test.{component_position}",
|
||||
component_version="1.0.0",
|
||||
component_position=component_position,
|
||||
proposal_component_position=component_position,
|
||||
proposal_snapshot_sha256=before_sha256,
|
||||
proposal_index=proposal_index,
|
||||
edit_index=edit_index,
|
||||
start=start,
|
||||
end=end,
|
||||
before=before_text,
|
||||
after=after_text,
|
||||
before_sha256=before_sha256,
|
||||
after_sha256=after_sha256,
|
||||
)
|
||||
|
||||
|
||||
def test_replay_builds_zero_change_stages_and_utf16_editor_ranges() -> None:
|
||||
original = "\ufeff😀 old\r\nCafe\u0301"
|
||||
final = "\ufeff😀 new\r\nCafe\u0301"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
recorded = change(
|
||||
component_position=0,
|
||||
before_text="old",
|
||||
after_text="new",
|
||||
start=3,
|
||||
end=6,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"), ReplayComponent("test.1", "1.0.0")),
|
||||
changes=(recorded,),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
assert replayed.current_markdown == final
|
||||
assert len(replayed.stages) == 2
|
||||
assert replayed.stages[1].before_markdown == final
|
||||
assert replayed.stages[1].after_markdown == final
|
||||
assert replayed.stages[1].changes == ()
|
||||
assert (replayed.changes[0].line, replayed.changes[0].column) == (1, 4)
|
||||
assert (replayed.changes[0].editor_start, replayed.changes[0].editor_end) == (4, 7)
|
||||
|
||||
|
||||
def test_replay_uses_full_descending_application_key_not_record_order() -> None:
|
||||
original = "abcd"
|
||||
final = "aXXcYY"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
right = change(
|
||||
component_position=0,
|
||||
before_text="d",
|
||||
after_text="YY",
|
||||
start=3,
|
||||
end=4,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
proposal_index=1,
|
||||
)
|
||||
left = change(
|
||||
component_position=0,
|
||||
before_text="b",
|
||||
after_text="XX",
|
||||
start=1,
|
||||
end=2,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(right, left),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
assert replayed.current_markdown == final
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mutate", "message"),
|
||||
[
|
||||
(lambda item: replace(item, component_position=2), "component position"),
|
||||
(lambda item: replace(item, proposal_component_position=1), "proposal reference"),
|
||||
(lambda item: replace(item, before="bad"), "recorded snapshot"),
|
||||
(lambda item: replace(item, after_sha256="0" * 64), "batch hash"),
|
||||
],
|
||||
)
|
||||
def test_replay_rejects_untrusted_change_chains(
|
||||
mutate: Callable[[ReplayChange], ReplayChange], message: str
|
||||
) -> None:
|
||||
original = "old"
|
||||
final = "new"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
valid = change(
|
||||
component_position=0,
|
||||
before_text="old",
|
||||
after_text="new",
|
||||
start=0,
|
||||
end=3,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
tampered = mutate(valid)
|
||||
|
||||
with pytest.raises(ReplayError, match=message):
|
||||
replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(tampered,),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
|
||||
def test_replay_rejects_conflicting_ranges() -> None:
|
||||
original = "abc"
|
||||
input_hash = markdown_sha256(original)
|
||||
first = change(
|
||||
component_position=0,
|
||||
before_text="ab",
|
||||
after_text="x",
|
||||
start=0,
|
||||
end=2,
|
||||
before_sha256=input_hash,
|
||||
after_sha256="0" * 64,
|
||||
)
|
||||
second = change(
|
||||
component_position=0,
|
||||
before_text="bc",
|
||||
after_text="y",
|
||||
start=1,
|
||||
end=3,
|
||||
before_sha256=input_hash,
|
||||
after_sha256="0" * 64,
|
||||
proposal_index=1,
|
||||
)
|
||||
|
||||
with pytest.raises(ReplayError, match="conflicting"):
|
||||
replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(first, second),
|
||||
current_sha256=input_hash,
|
||||
current_markdown=None,
|
||||
include_zero_change_stages=False,
|
||||
)
|
||||
@@ -1,398 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import stat
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
import mdpolish.artifact_store as artifact_store
|
||||
from mdpolish.artifact_store import ArtifactStoreError, StoredDocument, publish_run
|
||||
|
||||
_SOURCE_BYTES = b"source\n"
|
||||
|
||||
|
||||
def stored_document(document_id: str = "paper", output: bytes | None = b"cleaned\n") -> StoredDocument:
|
||||
status = "success" if output is not None else "failed"
|
||||
current_sha256 = sha256(output).hexdigest() if output is not None else "2" * 64
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"document": {"document_id": document_id, "source_label": f"inputs/{document_id}.md"},
|
||||
"status": status,
|
||||
"input_sha256": sha256(_SOURCE_BYTES).hexdigest(),
|
||||
"current_sha256": current_sha256,
|
||||
"changes": [],
|
||||
"errors": [] if status == "success" else [{"error_type": "SyntheticError"}],
|
||||
"residual_proposals": [],
|
||||
"output": {
|
||||
"cleaned_path": "cleaned.md" if output is not None else None,
|
||||
"diff_path": "changes.diff" if output is not None else None,
|
||||
},
|
||||
}
|
||||
return StoredDocument(
|
||||
document_id=document_id,
|
||||
result_json=(json.dumps(payload, indent=2) + "\n").encode(),
|
||||
cleaned_markdown=output,
|
||||
diff=b"" if output is not None else None,
|
||||
output_sha256=sha256(output).hexdigest() if output is not None else None,
|
||||
)
|
||||
|
||||
|
||||
def manifest_json(
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> bytes:
|
||||
indexes: list[dict[str, object]] = []
|
||||
statuses: list[str] = []
|
||||
change_count = 0
|
||||
for document in documents:
|
||||
report = json.loads(document.result_json)
|
||||
status = report["status"]
|
||||
statuses.append(status)
|
||||
change_count += len(report["changes"])
|
||||
base = f"documents/{document.document_id}"
|
||||
indexes.append(
|
||||
{
|
||||
"document_id": document.document_id,
|
||||
"source_label": report["document"]["source_label"],
|
||||
"status": status,
|
||||
"input_sha256": report["input_sha256"],
|
||||
"current_sha256": report["current_sha256"],
|
||||
"change_count": len(report["changes"]),
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": f"{base}/cleaned.md" if status == "success" else None,
|
||||
"diff_path": f"{base}/changes.diff" if status == "success" else None,
|
||||
}
|
||||
)
|
||||
failed_count = statuses.count("failed")
|
||||
unstable_count = statuses.count("unstable")
|
||||
overall_status = "failed" if failed_count else "unstable" if unstable_count else "success"
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"run": {"run_id": run_id, "run_date": run_date, "status": overall_status},
|
||||
"documents": indexes,
|
||||
"summary": {
|
||||
"document_count": len(documents),
|
||||
"success_count": statuses.count("success"),
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
},
|
||||
}
|
||||
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||
|
||||
|
||||
def review_locator_json(
|
||||
artifacts_root: Path,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> bytes:
|
||||
source_root = artifacts_root.parent / f"{artifacts_root.name}-sources"
|
||||
source_root.mkdir(exist_ok=True)
|
||||
locator_documents: list[dict[str, object]] = []
|
||||
for position, document in enumerate(documents):
|
||||
source_path = source_root / f"{position}-{document.document_id}.md"
|
||||
source_path.write_bytes(_SOURCE_BYTES)
|
||||
report = json.loads(document.result_json)
|
||||
locator_documents.append(
|
||||
{
|
||||
"document_id": document.document_id,
|
||||
"source_path": str(source_path.resolve()),
|
||||
"input_sha256": report["input_sha256"],
|
||||
}
|
||||
)
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_directory": str((artifacts_root / run_date / "runs" / run_id).resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": locator_documents,
|
||||
}
|
||||
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||
|
||||
|
||||
def test_publish_run_creates_private_date_layout_and_status_specific_files(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document("success"), stored_document("failed", None))
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "example-run", documents)
|
||||
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="example-run",
|
||||
manifest_json=manifest_json("2026-08-22", "example-run", documents),
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert run_directory == artifacts_root / "2026-08-22" / "runs" / "example-run"
|
||||
assert json.loads((run_directory / "manifest.json").read_bytes())["run"]["status"] == "failed"
|
||||
assert (run_directory / "documents/success/result.json").is_file()
|
||||
assert (run_directory / "documents/success/cleaned.md").read_bytes() == b"cleaned\n"
|
||||
assert (run_directory / "documents/success/changes.diff").read_bytes() == b""
|
||||
assert (run_directory / "documents/failed/result.json").is_file()
|
||||
assert (run_directory / "review-locator.json").read_bytes() == locator_json
|
||||
assert not (run_directory / "documents/failed/cleaned.md").exists()
|
||||
assert not (run_directory / "documents/failed/changes.diff").exists()
|
||||
|
||||
for directory in (
|
||||
artifacts_root,
|
||||
artifacts_root / "2026-08-22",
|
||||
artifacts_root / "2026-08-22/runs",
|
||||
run_directory,
|
||||
run_directory / "documents",
|
||||
run_directory / "documents/success",
|
||||
):
|
||||
assert stat.S_IMODE(directory.stat().st_mode) == 0o700
|
||||
for artifact_file in run_directory.rglob("*"):
|
||||
if artifact_file.is_file():
|
||||
assert stat.S_IMODE(artifact_file.stat().st_mode) == 0o600
|
||||
|
||||
|
||||
def test_publish_run_rejects_existing_target_without_overwriting(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
first_manifest = manifest_json("2026-08-22", "same-run", documents)
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "same-run", documents)
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="same-run",
|
||||
manifest_json=first_manifest,
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="already exists"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="same-run",
|
||||
manifest_json=first_manifest,
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert (run_directory / "manifest.json").read_bytes() == first_manifest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("run_date", "run_id"),
|
||||
[
|
||||
("2026-8-22", "valid"),
|
||||
("2026-02-30", "valid"),
|
||||
("2026-08-22", "Uppercase"),
|
||||
("2026-08-22", "../escape"),
|
||||
("2026-08-22", "two..dots"),
|
||||
],
|
||||
)
|
||||
def test_publish_run_rejects_unsafe_date_and_run_id(tmp_path: Path, run_date: str, run_id: str) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
with pytest.raises(ArtifactStoreError):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
manifest_json=manifest_json(run_date, run_id, documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, run_date, run_id, documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_inconsistent_or_duplicate_document_artifacts(tmp_path: Path) -> None:
|
||||
valid = stored_document()
|
||||
bad_hash = StoredDocument("paper", valid.result_json, b"output", b"", "0" * 64)
|
||||
first_artifacts_root = tmp_path / "artifacts-a"
|
||||
with pytest.raises(ArtifactStoreError, match="output hash"):
|
||||
publish_run(
|
||||
artifacts_root=first_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="bad-hash",
|
||||
manifest_json=manifest_json("2026-08-22", "bad-hash", (bad_hash,)),
|
||||
review_locator_json=review_locator_json(
|
||||
first_artifacts_root, "2026-08-22", "bad-hash", (bad_hash,)
|
||||
),
|
||||
documents=(bad_hash,),
|
||||
)
|
||||
|
||||
duplicates = (stored_document(), stored_document())
|
||||
second_artifacts_root = tmp_path / "artifacts-b"
|
||||
with pytest.raises(ArtifactStoreError, match="unique"):
|
||||
publish_run(
|
||||
artifacts_root=second_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="duplicate",
|
||||
manifest_json=manifest_json("2026-08-22", "duplicate", duplicates),
|
||||
review_locator_json=review_locator_json(
|
||||
second_artifacts_root, "2026-08-22", "duplicate", duplicates
|
||||
),
|
||||
documents=duplicates,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_manifest_path_or_document_mismatch(tmp_path: Path) -> None:
|
||||
documents = (stored_document(),)
|
||||
wrong_date = manifest_json("2026-08-21", "review", documents)
|
||||
first_artifacts_root = tmp_path / "artifacts-a"
|
||||
with pytest.raises(ArtifactStoreError, match="identity"):
|
||||
publish_run(
|
||||
artifacts_root=first_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=wrong_date,
|
||||
review_locator_json=review_locator_json(
|
||||
first_artifacts_root, "2026-08-22", "review", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
payload = json.loads(manifest_json("2026-08-22", "review", documents))
|
||||
payload["documents"][0]["change_count"] = 99
|
||||
mismatched_index = (json.dumps(payload, indent=2) + "\n").encode()
|
||||
second_artifacts_root = tmp_path / "artifacts-b"
|
||||
with pytest.raises(ArtifactStoreError, match="index"):
|
||||
publish_run(
|
||||
artifacts_root=second_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=mismatched_index,
|
||||
review_locator_json=review_locator_json(
|
||||
second_artifacts_root, "2026-08-22", "review", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_inconsistent_review_locator(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
locator = json.loads(review_locator_json(artifacts_root, "2026-08-22", "review", documents))
|
||||
locator["documents"][0]["input_sha256"] = "0" * 64
|
||||
inconsistent_locator = (json.dumps(locator, indent=2) + "\n").encode()
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="document identity"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||
review_locator_json=inconsistent_locator,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_publish_run_rejects_source_changed_after_locator_creation(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "review", documents)
|
||||
locator = json.loads(locator_json)
|
||||
Path(locator["documents"][0]["source_path"]).write_bytes(b"changed\n")
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="input hash"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_publish_race_does_not_replace_a_new_target(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
original_rename = artifact_store._rename_no_replace
|
||||
|
||||
def create_competing_target(source: Path, target: Path) -> None:
|
||||
target.mkdir(mode=0o700)
|
||||
(target / "keep").write_bytes(b"existing")
|
||||
original_rename(source, target)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_rename_no_replace", create_competing_target)
|
||||
with pytest.raises(ArtifactStoreError, match="already exists"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="raced",
|
||||
manifest_json=manifest_json("2026-08-22", "raced", documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "raced", documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
target = tmp_path / "artifacts/2026-08-22/runs/raced"
|
||||
assert (target / "keep").read_bytes() == b"existing"
|
||||
assert not any(path.name.startswith(".raced.") for path in target.parent.iterdir())
|
||||
|
||||
|
||||
def test_write_failure_cleans_temporary_directory_and_does_not_publish(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
original_write = artifact_store._write_private_file
|
||||
call_count = 0
|
||||
|
||||
def fail_second_write(path: Path, content: bytes) -> None:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 2:
|
||||
raise OSError("synthetic write failure")
|
||||
original_write(path, content)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_write_private_file", fail_second_write)
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
|
||||
with pytest.raises(OSError, match="synthetic"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="broken",
|
||||
manifest_json=manifest_json("2026-08-22", "broken", documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "broken", documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = tmp_path / "artifacts/2026-08-22/runs"
|
||||
assert list(runs_directory.iterdir()) == []
|
||||
|
||||
|
||||
def test_locator_write_failure_does_not_publish(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
original_write = artifact_store._write_private_file
|
||||
|
||||
def fail_locator_write(path: Path, content: bytes) -> None:
|
||||
if path.name == "review-locator.json":
|
||||
raise OSError("synthetic locator write failure")
|
||||
original_write(path, content)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_write_private_file", fail_locator_write)
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
|
||||
with pytest.raises(OSError, match="locator"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="locator-failure",
|
||||
manifest_json=manifest_json("2026-08-22", "locator-failure", documents),
|
||||
review_locator_json=review_locator_json(
|
||||
artifacts_root, "2026-08-22", "locator-failure", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = tmp_path / "artifacts/2026-08-22/runs"
|
||||
assert list(runs_directory.iterdir()) == []
|
||||
@@ -1,155 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import mdpolish
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ArxivSubmissionStampComponent
|
||||
|
||||
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||
OTHER_STAMP = "arXiv:2301.7v1 [cs.AI] 1 Jan 2023"
|
||||
REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||
|
||||
|
||||
def transform(markdown: str) -> mdpolish.TransformResult:
|
||||
return Pipeline([ArxivSubmissionStampComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_component_metadata_and_package_export() -> None:
|
||||
result = transform("")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.components[0].component_id == "paper.arxiv_submission_stamp"
|
||||
assert result.components[0].version == "1.0.0"
|
||||
assert result.components[0].parameters == ()
|
||||
assert result.components[0].applicability
|
||||
assert not hasattr(mdpolish, "ArxivSubmissionStampComponent")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("markdown", ["", "普通正文", "中文\nCafe\u0301\n🙂\n"])
|
||||
def test_no_target_returns_unchanged_success(markdown: str) -> None:
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
(f"{STAMP}\n正文", "正文"),
|
||||
(f"正文\n{STAMP}\n后文", "正文\n后文"),
|
||||
(f"正文\n{STAMP}", "正文\n"),
|
||||
(STAMP, ""),
|
||||
(f"正文\r\n{STAMP}\r\n后文", "正文\r\n后文"),
|
||||
(f"正文\r{STAMP}\r后文", "正文\r后文"),
|
||||
],
|
||||
)
|
||||
def test_deletes_target_at_approved_line_boundaries(markdown: str, expected: str) -> None:
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == expected
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_multiple_targets_are_reported_in_source_order_with_one_atomic_batch() -> None:
|
||||
markdown = f"{STAMP}\n保留\n{OTHER_STAMP}"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "保留\n"
|
||||
assert [change.before for change in result.changes] == [f"{STAMP}\n", OTHER_STAMP]
|
||||
assert [change.proposal_ref.proposal_index for change in result.changes] == [0, 1]
|
||||
assert [change.span.start for change in result.changes] == sorted(change.span.start for change in result.changes)
|
||||
assert {change.before_sha256 for change in result.changes} == {result.input_sha256}
|
||||
assert {change.after_sha256 for change in result.changes} == {result.current_sha256}
|
||||
|
||||
|
||||
def test_adjacent_targets_use_non_overlapping_delete_ranges() -> None:
|
||||
result = transform(f"{STAMP}\n{OTHER_STAMP}")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == ""
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].span.end == result.changes[1].span.start
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"line",
|
||||
[
|
||||
f" {STAMP}",
|
||||
f"{STAMP} ",
|
||||
f"- {STAMP}",
|
||||
f"> {STAMP}",
|
||||
"1. Example. arXiv preprint arXiv:2104.12345v2 [stat.ME], 2021.",
|
||||
"See arXiv:2104.12345v2 for details.",
|
||||
"arXiv:2104.12345 [stat.ME] 31 Dec 2021",
|
||||
"arXiv:hep-ph/9901001 [hep-ph] 31 Dec 1999",
|
||||
"arXiv:2104.12345v2 31 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 0 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 32 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 31 December 2021",
|
||||
"arXiv:2104.12345v2 [统计] 31 Dec 2021",
|
||||
],
|
||||
)
|
||||
def test_similar_arxiv_text_is_preserved(line: str) -> None:
|
||||
markdown = f"前文\n{line}\n后文"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
def test_matching_line_inside_fenced_code_is_not_protected() -> None:
|
||||
markdown = f"```text\n{STAMP}\n```\n"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "```text\n```\n"
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_change_audit_records_identity_reason_source_and_batch_hashes() -> None:
|
||||
result = transform(f"{STAMP}\n正文")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
change = result.changes[0]
|
||||
assert change.component_id == "paper.arxiv_submission_stamp"
|
||||
assert change.component_version == "1.0.0"
|
||||
assert change.component_position == 0
|
||||
assert change.proposal_ref.component_position == 0
|
||||
assert change.proposal_ref.proposal_index == 0
|
||||
assert change.edit_index == 0
|
||||
assert change.reason == REASON
|
||||
assert change.before == f"{STAMP}\n"
|
||||
assert change.after == ""
|
||||
assert change.before_sha256 == result.input_sha256
|
||||
assert change.after_sha256 == result.current_sha256
|
||||
|
||||
|
||||
def test_successful_output_is_stable_and_second_run_has_no_changes() -> None:
|
||||
pipeline = Pipeline([ArxivSubmissionStampComponent()])
|
||||
|
||||
first = pipeline.transform(f"{STAMP}\n正文")
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
assert first.output_markdown == "正文"
|
||||
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.output_markdown == "正文"
|
||||
assert second.changes == ()
|
||||
assert second.residual_proposals == ()
|
||||
|
||||
|
||||
def test_same_input_produces_same_ordered_result() -> None:
|
||||
pipeline = Pipeline([ArxivSubmissionStampComponent()])
|
||||
markdown = f"{STAMP}\n正文\n{OTHER_STAMP}\n"
|
||||
|
||||
assert pipeline.transform(markdown) == pipeline.transform(markdown)
|
||||
@@ -1,162 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import (
|
||||
ArxivSubmissionStampComponent,
|
||||
HtmlTableDoubleEscapeComponent,
|
||||
HtmlTableLayoutComponent,
|
||||
ManuscriptLineNumberComponent,
|
||||
PageBreakWordJoinComponent,
|
||||
ReferenceSpacingComponent,
|
||||
RepeatedRunningHeaderComponent,
|
||||
WordReviewCommentComponent,
|
||||
)
|
||||
|
||||
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||
HEADER = "## Repeated Paper Header"
|
||||
MAPPINGS = (
|
||||
("medi-", "cal", "medical"),
|
||||
("possi-", "bly", "possibly"),
|
||||
("cre-", "ated", "created"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("life-", "threatening", "life-threatening"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def _build_pipeline() -> Pipeline:
|
||||
return Pipeline(
|
||||
[
|
||||
WordReviewCommentComponent(),
|
||||
ManuscriptLineNumberComponent(),
|
||||
ArxivSubmissionStampComponent(),
|
||||
RepeatedRunningHeaderComponent(),
|
||||
PageBreakWordJoinComponent(MAPPINGS),
|
||||
HtmlTableDoubleEscapeComponent(),
|
||||
HtmlTableLayoutComponent(),
|
||||
ReferenceSpacingComponent(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _numbered_manuscript() -> list[str]:
|
||||
return [
|
||||
f"## {number} Section {number}" if number in {5, 15} else f"{number} body {number}"
|
||||
for number in range(1, 21)
|
||||
]
|
||||
|
||||
|
||||
def _combined_markdown() -> str:
|
||||
return "\n".join(
|
||||
(
|
||||
"1 Affiliation",
|
||||
"## Abstract",
|
||||
*_numbered_manuscript(),
|
||||
"Commented [A1]: remove this",
|
||||
"",
|
||||
STAMP,
|
||||
"Sentence continues in",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"the next line.",
|
||||
"A word is possi-",
|
||||
"",
|
||||
"bly split.",
|
||||
"<table><tr><td>&lt;5</td></tr><tr><td>B</td></tr></table>",
|
||||
"## References",
|
||||
"",
|
||||
"1. First",
|
||||
"",
|
||||
"2. Second",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"3. Third",
|
||||
"4. Fourth",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_script_builds_frozen_component_order_and_parameters() -> None:
|
||||
script_path = Path(__file__).parents[1] / "scripts" / "run_clindb_first_batch_experiment.py"
|
||||
module = ast.parse(script_path.read_text(encoding="utf-8"))
|
||||
build_function = next(
|
||||
node for node in module.body if isinstance(node, ast.FunctionDef) and node.name == "build_pipeline"
|
||||
)
|
||||
component_names = [
|
||||
call.func.id
|
||||
for node in ast.walk(build_function)
|
||||
if isinstance(node, ast.List)
|
||||
for call in node.elts
|
||||
if isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
|
||||
]
|
||||
mapping_assignment = next(
|
||||
node
|
||||
for node in module.body
|
||||
if isinstance(node, ast.Assign)
|
||||
and any(isinstance(target, ast.Name) and target.id == "CLINDB_WORD_JOIN_MAPPINGS" for target in node.targets)
|
||||
)
|
||||
assert component_names == [
|
||||
"WordReviewCommentComponent",
|
||||
"ManuscriptLineNumberComponent",
|
||||
"ArxivSubmissionStampComponent",
|
||||
"RepeatedRunningHeaderComponent",
|
||||
"PageBreakWordJoinComponent",
|
||||
"HtmlTableDoubleEscapeComponent",
|
||||
"HtmlTableLayoutComponent",
|
||||
"ReferenceSpacingComponent",
|
||||
]
|
||||
assert ast.literal_eval(mapping_assignment.value) == MAPPINGS
|
||||
|
||||
pipeline = _build_pipeline()
|
||||
result = pipeline.transform("")
|
||||
|
||||
assert [component.component_id for component in result.components] == [
|
||||
"paper.word_review_comment",
|
||||
"paper.manuscript_line_number",
|
||||
"paper.arxiv_submission_stamp",
|
||||
"paper.repeated_running_header",
|
||||
"paper.page_break_word_join",
|
||||
"markdown.html_table_double_escape",
|
||||
"markdown.html_table_layout",
|
||||
"paper.reference_spacing",
|
||||
]
|
||||
assert len(MAPPINGS) == 6
|
||||
|
||||
|
||||
def test_full_pipeline_is_audited_stable_and_idempotent() -> None:
|
||||
pipeline = _build_pipeline()
|
||||
first = pipeline.transform(_combined_markdown())
|
||||
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
assert first.output_markdown is not None
|
||||
assert first.residual_proposals == ()
|
||||
counts: dict[str, int] = {}
|
||||
for change in first.changes:
|
||||
counts[change.component_id] = counts.get(change.component_id, 0) + 1
|
||||
assert counts == {
|
||||
"paper.word_review_comment": 1,
|
||||
"paper.manuscript_line_number": 20,
|
||||
"paper.arxiv_submission_stamp": 1,
|
||||
"paper.repeated_running_header": 2,
|
||||
"paper.page_break_word_join": 1,
|
||||
"markdown.html_table_double_escape": 1,
|
||||
"markdown.html_table_layout": 1,
|
||||
"paper.reference_spacing": 1,
|
||||
}
|
||||
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.output_markdown == first.output_markdown
|
||||
assert second.changes == ()
|
||||
|
||||
|
||||
def test_business_components_are_not_exported_from_core_namespace() -> None:
|
||||
import mdpolish
|
||||
|
||||
assert not hasattr(mdpolish, "WordReviewCommentComponent")
|
||||
assert not hasattr(mdpolish, "HtmlTableLayoutComponent")
|
||||
@@ -1,121 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, ComponentContractError, DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
|
||||
class ExampleComponent(Component):
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
component_id: object = "test.example",
|
||||
version: object = "1.2.3",
|
||||
parameters: object = None,
|
||||
applicability: object = "处理测试标记，要求精确匹配，排除所有其他内容。",
|
||||
) -> None:
|
||||
self._component_id = component_id
|
||||
self._version = version
|
||||
self._parameters = {} if parameters is None else parameters
|
||||
self._applicability = applicability
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return cast(str, self._component_id)
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return cast(str, self._version)
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return cast(Mapping[str, object], self._parameters)
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return cast(str, self._applicability)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if not snapshot.markdown:
|
||||
return ()
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), snapshot.markdown[0], "X")
|
||||
return (ProposedChange(snapshot.sha256, "replace first character", (edit,)),)
|
||||
|
||||
|
||||
class ListReturningComponent(ExampleComponent):
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
|
||||
|
||||
class WrongValueComponent(ExampleComponent):
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], ("wrong",))
|
||||
|
||||
|
||||
def test_component_metadata_is_validated_and_parameters_are_frozen_deterministically() -> None:
|
||||
component = ExampleComponent(
|
||||
parameters={
|
||||
"z": [1, {"b": False, "a": None}],
|
||||
"a": "value",
|
||||
}
|
||||
)
|
||||
|
||||
info = component._component_info()
|
||||
|
||||
assert info.component_id == "test.example"
|
||||
assert info.version == "1.2.3"
|
||||
assert info.parameters == (
|
||||
("a", "value"),
|
||||
("z", (1, (("a", None), ("b", False)))),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("component_id", ["", "Uppercase", "has space", "two..dots", "_leading"])
|
||||
def test_invalid_component_id_is_a_contract_error(component_id: str) -> None:
|
||||
with pytest.raises(ComponentContractError, match="component_id"):
|
||||
ExampleComponent(component_id=component_id)._component_info()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", ["1", "1.2", "v1.2.3", "01.2.3", "1.2.3-alpha"])
|
||||
def test_invalid_version_is_a_contract_error(version: str) -> None:
|
||||
with pytest.raises(ComponentContractError, match=r"MAJOR.MINOR.PATCH"):
|
||||
ExampleComponent(version=version)._component_info()
|
||||
|
||||
|
||||
def test_empty_applicability_is_a_contract_error() -> None:
|
||||
with pytest.raises(ComponentContractError, match="applicability"):
|
||||
ExampleComponent(applicability=" \n")._component_info()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"parameters",
|
||||
[
|
||||
{"bad": {1, 2}},
|
||||
{"bad": float("inf")},
|
||||
{"bad": float("nan")},
|
||||
{1: "non-string key"},
|
||||
["not", "a", "mapping"],
|
||||
],
|
||||
)
|
||||
def test_unrepresentable_parameters_are_contract_errors(parameters: object) -> None:
|
||||
with pytest.raises(ComponentContractError, match="parameter"):
|
||||
ExampleComponent(parameters=parameters)._component_info()
|
||||
|
||||
|
||||
def test_collect_proposals_requires_a_tuple_of_proposed_changes() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
|
||||
with pytest.raises(ComponentContractError, match="return a tuple"):
|
||||
ListReturningComponent()._collect_proposals(snapshot)
|
||||
with pytest.raises(ComponentContractError, match="only ProposedChange"):
|
||||
WrongValueComponent()._collect_proposals(snapshot)
|
||||
|
||||
|
||||
def test_component_exposes_no_public_check_or_transform_shortcut() -> None:
|
||||
component = ExampleComponent()
|
||||
|
||||
assert not hasattr(component, "check")
|
||||
assert not hasattr(component, "transform")
|
||||
+25
-25
@@ -3,17 +3,17 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import (
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
EditValidationError,
|
||||
ModifierInfo,
|
||||
ProposedChange,
|
||||
TextEdit,
|
||||
TextSpan,
|
||||
apply_component_batch,
|
||||
apply_modifier_batch,
|
||||
)
|
||||
|
||||
COMPONENT = ComponentInfo(
|
||||
component_id="test.component",
|
||||
MODIFIER = ModifierInfo(
|
||||
modifier_id="test.modifier",
|
||||
version="1.2.3",
|
||||
parameters=(),
|
||||
applicability="测试精确文本编辑，只处理测试字符串，排除其他输入。",
|
||||
@@ -38,22 +38,22 @@ def test_applies_insert_delete_and_replace() -> None:
|
||||
delete_snapshot = DocumentSnapshot("abc")
|
||||
replace_snapshot = DocumentSnapshot("abc")
|
||||
|
||||
inserted = apply_component_batch(
|
||||
inserted = apply_modifier_batch(
|
||||
insert_snapshot,
|
||||
(make_proposal(insert_snapshot, make_edit(insert_snapshot, 1, 1, "X")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
deleted = apply_component_batch(
|
||||
deleted = apply_modifier_batch(
|
||||
delete_snapshot,
|
||||
(make_proposal(delete_snapshot, make_edit(delete_snapshot, 1, 2, "")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
replaced = apply_component_batch(
|
||||
replaced = apply_modifier_batch(
|
||||
replace_snapshot,
|
||||
(make_proposal(replace_snapshot, make_edit(replace_snapshot, 1, 2, "X")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
|
||||
@@ -71,7 +71,7 @@ def test_multiple_edits_apply_backwards_but_report_in_source_order() -> None:
|
||||
reason="normalize two locations",
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 3)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 3)
|
||||
|
||||
assert applied.snapshot.markdown == "AbcdF"
|
||||
assert [change.span.start for change in applied.changes] == [0, 4]
|
||||
@@ -82,7 +82,7 @@ def test_multiple_edits_apply_backwards_but_report_in_source_order() -> None:
|
||||
}
|
||||
assert {change.reason for change in applied.changes} == {"normalize two locations"}
|
||||
assert [change.edit_index for change in applied.changes] == [1, 0]
|
||||
assert all(change.component_position == 3 for change in applied.changes)
|
||||
assert all(change.modifier_position == 3 for change in applied.changes)
|
||||
|
||||
|
||||
def test_adjacent_nonempty_ranges_are_allowed() -> None:
|
||||
@@ -93,7 +93,7 @@ def test_adjacent_nonempty_ranges_are_allowed() -> None:
|
||||
make_edit(snapshot, 2, 4, "D"),
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot.markdown == "AD"
|
||||
|
||||
@@ -107,7 +107,7 @@ def test_overlapping_ranges_fail_without_changing_snapshot() -> None:
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert snapshot.markdown == "abcdef"
|
||||
assert snapshot.sha256 == DocumentSnapshot("abcdef").sha256
|
||||
@@ -119,7 +119,7 @@ def test_duplicate_edits_fail_explicitly() -> None:
|
||||
proposal = make_proposal(snapshot, edit, edit)
|
||||
|
||||
with pytest.raises(EditValidationError, match="duplicate"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_distinct_insert_points_are_allowed() -> None:
|
||||
@@ -130,7 +130,7 @@ def test_distinct_insert_points_are_allowed() -> None:
|
||||
make_edit(snapshot, 3, 3, "Y"),
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot.markdown == "aXbcYd"
|
||||
|
||||
@@ -144,7 +144,7 @@ def test_same_insert_point_conflicts() -> None:
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("insert_position", [1, 2, 3])
|
||||
@@ -157,7 +157,7 @@ def test_insert_at_start_inside_or_end_of_nonempty_range_conflicts(insert_positi
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_stale_hash_out_of_range_and_expected_text_mismatch_fail() -> None:
|
||||
@@ -166,34 +166,34 @@ def test_stale_hash_out_of_range_and_expected_text_mismatch_fail() -> None:
|
||||
stale = make_proposal(original, make_edit(original, 0, 1, "A"))
|
||||
|
||||
with pytest.raises(EditValidationError, match="stale"):
|
||||
apply_component_batch(current, (stale,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (stale,), MODIFIER, 0)
|
||||
|
||||
out_of_range_edit = TextEdit(current.sha256, TextSpan(2, 5), "dxx", "D")
|
||||
out_of_range = make_proposal(current, out_of_range_edit)
|
||||
with pytest.raises(EditValidationError, match="outside"):
|
||||
apply_component_batch(current, (out_of_range,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (out_of_range,), MODIFIER, 0)
|
||||
|
||||
mismatch_edit = TextEdit(current.sha256, TextSpan(0, 1), "z", "A")
|
||||
mismatch = make_proposal(current, mismatch_edit)
|
||||
with pytest.raises(EditValidationError, match="expected_text"):
|
||||
apply_component_batch(current, (mismatch,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (mismatch,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_conflict_across_proposals_rejects_whole_component_batch() -> None:
|
||||
def test_conflict_across_proposals_rejects_whole_modifier_batch() -> None:
|
||||
snapshot = DocumentSnapshot("abcdef")
|
||||
first = make_proposal(snapshot, make_edit(snapshot, 0, 3, "X"), reason="first")
|
||||
second = make_proposal(snapshot, make_edit(snapshot, 2, 4, "Y"), reason="second")
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (first, second), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (first, second), MODIFIER, 0)
|
||||
|
||||
assert snapshot.markdown == "abcdef"
|
||||
|
||||
|
||||
def test_empty_component_batch_keeps_same_snapshot_and_records_nothing() -> None:
|
||||
def test_empty_modifier_batch_keeps_same_snapshot_and_records_nothing() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
|
||||
applied = apply_component_batch(snapshot, (), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot is snapshot
|
||||
assert applied.changes == ()
|
||||
|
||||
@@ -1,356 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from datetime import UTC, datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, DocumentSnapshot, Pipeline, ProposedChange, RunStatus, TextEdit, TextSpan
|
||||
from mdpolish.experiment import ExperimentError, InputDocument, ToolMetadata, run_experiment
|
||||
|
||||
|
||||
class ConditionalComponent(Component):
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "test.conditional"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {"needle": "old", "replacement": "new"}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "替换测试标记 old，遇到 boom 时模拟组件失败，排除其他内容。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if snapshot.markdown == "boom":
|
||||
raise RuntimeError("private source")
|
||||
position = snapshot.markdown.find("old")
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(position, position + 3), "old", "new")
|
||||
return (ProposedChange(snapshot.sha256, "replace old test marker", (edit,)),)
|
||||
|
||||
|
||||
class NonIdempotentComponent(Component):
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "test.non-idempotent"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "在测试文本末尾反复插入标记，只用于验证 unstable 产物边界。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = len(snapshot.markdown)
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(position, position), "", "!")
|
||||
return (ProposedChange(snapshot.sha256, "append synthetic marker", (edit,)),)
|
||||
|
||||
|
||||
def tool_metadata() -> ToolMetadata:
|
||||
return ToolMetadata(
|
||||
name="mdpolish",
|
||||
package_version="0.1.0",
|
||||
python_version="3.13.11",
|
||||
platform="linux-x86_64",
|
||||
git_commit="a" * 40,
|
||||
git_dirty=False,
|
||||
)
|
||||
|
||||
|
||||
def input_document(path: Path, document_id: str) -> InputDocument:
|
||||
return InputDocument(document_id, path, f"inputs/{path.name}")
|
||||
|
||||
|
||||
def test_run_experiment_publishes_manifest_reports_diff_and_preserves_inputs(tmp_path: Path) -> None:
|
||||
first_path = tmp_path / "first.md"
|
||||
second_path = tmp_path / "second.md"
|
||||
first_path.write_bytes(b"before old\r\nafter\r\n")
|
||||
second_path.write_text("unchanged\n", encoding="utf-8")
|
||||
original_first = first_path.read_bytes()
|
||||
original_second = second_path.read_bytes()
|
||||
local_timezone = timezone(timedelta(hours=8))
|
||||
started_at = datetime(2026, 8, 22, 10, 30, tzinfo=local_timezone)
|
||||
completed_at = datetime(2026, 8, 22, 2, 31, tzinfo=UTC)
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(first_path, "first"), input_document(second_path, "second")),
|
||||
run_id="local-review",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.run_directory == tmp_path / "artifacts/2026-08-22/runs/local-review"
|
||||
assert result.document_count == 2
|
||||
assert result.success_count == 2
|
||||
assert result.change_count == 1
|
||||
assert first_path.read_bytes() == original_first
|
||||
assert second_path.read_bytes() == original_second
|
||||
assert (result.run_directory / "documents/first/cleaned.md").read_bytes() == b"before new\r\nafter\r\n"
|
||||
assert (result.run_directory / "documents/second/cleaned.md").read_bytes() == original_second
|
||||
assert (result.run_directory / "documents/second/changes.diff").read_bytes() == b""
|
||||
|
||||
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
||||
locator = json.loads((result.run_directory / "review-locator.json").read_bytes())
|
||||
assert locator == {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "local-review",
|
||||
"run_directory": str(result.run_directory.resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "first",
|
||||
"source_path": str(first_path.resolve()),
|
||||
"input_sha256": manifest["documents"][0]["input_sha256"],
|
||||
},
|
||||
{
|
||||
"document_id": "second",
|
||||
"source_path": str(second_path.resolve()),
|
||||
"input_sha256": manifest["documents"][1]["input_sha256"],
|
||||
},
|
||||
],
|
||||
}
|
||||
assert manifest["run"]["run_date"] == "2026-08-22"
|
||||
assert manifest["run"]["utc_offset"] == "+08:00"
|
||||
assert manifest["run"]["started_at_utc"] == "2026-08-22T02:30:00Z"
|
||||
assert manifest["run"]["completed_at_utc"] == "2026-08-22T02:31:00Z"
|
||||
assert manifest["run"]["retention_until"] == "2026-09-21T02:31:00Z"
|
||||
assert manifest["pipeline"]["components"][0]["component_id"] == "test.conditional"
|
||||
assert manifest["pipeline"]["components"][0]["parameters"] == [
|
||||
["needle", "old"],
|
||||
["replacement", "new"],
|
||||
]
|
||||
assert manifest["summary"] == {
|
||||
"document_count": 2,
|
||||
"success_count": 2,
|
||||
"failed_count": 0,
|
||||
"unstable_count": 0,
|
||||
"change_count": 1,
|
||||
}
|
||||
first_report = json.loads((result.run_directory / "documents/first/result.json").read_bytes())
|
||||
assert first_report["changes"][0]["location"] == {"line": 1, "column": 8}
|
||||
assert first_report["changes"][0]["before"] == "old"
|
||||
assert b"--- a/first.md\n+++ b/first.md\n" in (
|
||||
result.run_directory / "documents/first/changes.diff"
|
||||
).read_bytes()
|
||||
|
||||
|
||||
def test_document_failure_is_isolated_and_never_writes_partial_markdown(tmp_path: Path) -> None:
|
||||
good_path = tmp_path / "good.md"
|
||||
bad_path = tmp_path / "bad.md"
|
||||
good_path.write_text("old", encoding="utf-8")
|
||||
bad_path.write_text("boom", encoding="utf-8")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(good_path, "good"), input_document(bad_path, "bad")),
|
||||
run_id="mixed",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, 10, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 10, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.success_count == 1
|
||||
assert result.failed_count == 1
|
||||
assert (result.run_directory / "documents/good/cleaned.md").read_text() == "new"
|
||||
assert (result.run_directory / "documents/bad/result.json").is_file()
|
||||
assert not (result.run_directory / "documents/bad/cleaned.md").exists()
|
||||
assert not (result.run_directory / "documents/bad/changes.diff").exists()
|
||||
bad_report = json.loads((result.run_directory / "documents/bad/result.json").read_bytes())
|
||||
assert bad_report["status"] == "failed"
|
||||
assert "partial_markdown" not in bad_report
|
||||
|
||||
|
||||
def test_preflight_rejects_invalid_utf8_without_running_or_creating_artifacts(tmp_path: Path) -> None:
|
||||
invalid_path = tmp_path / "invalid.md"
|
||||
invalid_path.write_bytes(b"\xff")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
|
||||
with pytest.raises(ExperimentError, match="strict UTF-8"):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(invalid_path, "invalid"),),
|
||||
run_id="invalid-input",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_preflight_rejects_invalid_document_manifests_before_creating_artifacts(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
directory_path = tmp_path / "directory"
|
||||
directory_path.mkdir()
|
||||
missing_path = tmp_path / "missing.md"
|
||||
cases = (
|
||||
(input_document(source_path, "duplicate"), input_document(source_path, "duplicate")),
|
||||
(input_document(source_path, "first"), input_document(source_path, "second")),
|
||||
(input_document(missing_path, "missing"),),
|
||||
(input_document(directory_path, "directory"),),
|
||||
(input_document(source_path, "two..dots"),),
|
||||
)
|
||||
|
||||
for index, documents in enumerate(cases):
|
||||
artifacts_root = tmp_path / f"artifacts-{index}"
|
||||
with pytest.raises(ExperimentError):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=documents,
|
||||
run_id="preflight",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_preflight_rejects_an_output_path_nested_under_an_input_file(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ExperimentError, match="cannot overlap"):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="overlap",
|
||||
artifacts_root=source_path,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert source_path.read_text(encoding="utf-8") == "content"
|
||||
|
||||
|
||||
def test_utf8_bom_crlf_and_missing_final_newline_are_preserved_exactly(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "bom.md"
|
||||
empty_path = tmp_path / "empty.md"
|
||||
source_bytes = b"\xef\xbb\xbfhead\r\nlast"
|
||||
source_path.write_bytes(source_bytes)
|
||||
empty_path.write_bytes(b"")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "bom"), input_document(empty_path, "empty")),
|
||||
run_id="byte-preservation",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert source_path.read_bytes() == source_bytes
|
||||
assert (result.run_directory / "documents/bom/cleaned.md").read_bytes() == source_bytes
|
||||
assert (result.run_directory / "documents/bom/changes.diff").read_bytes() == b""
|
||||
assert empty_path.read_bytes() == b""
|
||||
assert (result.run_directory / "documents/empty/cleaned.md").read_bytes() == b""
|
||||
assert (result.run_directory / "documents/empty/changes.diff").read_bytes() == b""
|
||||
|
||||
|
||||
def test_unstable_document_only_publishes_a_result_json(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([NonIdempotentComponent()]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="unstable",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.UNSTABLE
|
||||
assert result.unstable_count == 1
|
||||
assert (result.run_directory / "documents/paper/result.json").is_file()
|
||||
assert not (result.run_directory / "documents/paper/cleaned.md").exists()
|
||||
assert not (result.run_directory / "documents/paper/changes.diff").exists()
|
||||
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
||||
assert manifest["run"]["status"] == "unstable"
|
||||
|
||||
|
||||
def test_existing_same_date_run_is_rejected_without_overwriting(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("old", encoding="utf-8")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
pipeline = Pipeline([ConditionalComponent()])
|
||||
documents = (input_document(source_path, "paper"),)
|
||||
started_at = datetime(2026, 8, 22, tzinfo=UTC)
|
||||
completed_at = datetime(2026, 8, 22, 0, 1, tzinfo=UTC)
|
||||
tool = tool_metadata()
|
||||
first = run_experiment(
|
||||
pipeline=pipeline,
|
||||
documents=documents,
|
||||
run_id="same",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool,
|
||||
)
|
||||
original_manifest = (first.run_directory / "manifest.json").read_bytes()
|
||||
|
||||
with pytest.raises(ExperimentError, match="already exists"):
|
||||
run_experiment(
|
||||
pipeline=pipeline,
|
||||
documents=documents,
|
||||
run_id="same",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool,
|
||||
)
|
||||
|
||||
assert (first.run_directory / "manifest.json").read_bytes() == original_manifest
|
||||
|
||||
|
||||
def test_same_run_id_on_another_local_date_uses_a_separate_directory(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("unchanged", encoding="utf-8")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
|
||||
first = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="daily",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, 23, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 23, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
second = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="daily",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 23, 0, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 23, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert first.run_directory.parent.parent.name == "2026-08-22"
|
||||
assert second.run_directory.parent.parent.name == "2026-08-23"
|
||||
@@ -3,11 +3,11 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import HtmlTableDoubleEscapeComponent
|
||||
from mdpolish.modifiers import html_table_entity_unescape
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([HtmlTableDoubleEscapeComponent()]).transform(markdown)
|
||||
return Pipeline([html_table_entity_unescape()]).transform(markdown)
|
||||
|
||||
|
||||
def test_unescapes_one_layer_only_in_strict_cell_text() -> None:
|
||||
@@ -47,13 +47,13 @@ def test_non_strict_or_outside_content_is_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_fenced_table_is_not_protected() -> None:
|
||||
def test_fenced_table_is_not_protected_by_the_lexical_subset() -> None:
|
||||
markdown = "```html\n<table><tr><td>&lt;</td></tr></table>\n```"
|
||||
assert transform(markdown).output_markdown == "```html\n<table><tr><td><</td></tr></table>\n```"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([HtmlTableDoubleEscapeComponent()])
|
||||
pipeline = Pipeline([html_table_entity_unescape()])
|
||||
first = pipeline.transform("<table><tr><td>&lt;</td></tr></table>")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -3,13 +3,13 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import HtmlTableLayoutComponent
|
||||
from mdpolish.modifiers import html_table_layout
|
||||
|
||||
TABLE = '<table class="x"><tr><td colspan="2">A</td></tr><tr><td>B</td><td>C</td></tr></table>'
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([HtmlTableLayoutComponent()]).transform(markdown)
|
||||
return Pipeline([html_table_layout()]).transform(markdown)
|
||||
|
||||
|
||||
def test_expands_rows_without_changing_tags_attributes_or_cells() -> None:
|
||||
@@ -57,7 +57,7 @@ def test_mixed_multiline_or_non_strict_tables_are_preserved(markdown: str) -> No
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([HtmlTableLayoutComponent()])
|
||||
pipeline = Pipeline([html_table_layout()])
|
||||
first = pipeline.transform(TABLE)
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ManuscriptLineNumberComponent
|
||||
|
||||
|
||||
def _document(
|
||||
*,
|
||||
count: int = 20,
|
||||
heading_numbers: frozenset[int] = frozenset({5, 15}),
|
||||
numbers: tuple[int, ...] | None = None,
|
||||
line_ending: str = "\n",
|
||||
) -> str:
|
||||
values = numbers if numbers is not None else tuple(range(1, count + 1))
|
||||
body = [
|
||||
f"## {number} Section {number}" if number in heading_numbers else f"{number} body {number}"
|
||||
for number in values
|
||||
]
|
||||
return line_ending.join(("1 Affiliation", "2 Institute", "## Abstract", *body))
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([ManuscriptLineNumberComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_removes_long_monotonic_sequence_but_preserves_pre_abstract_affiliations() -> None:
|
||||
result = transform(_document())
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.startswith("1 Affiliation\n2 Institute\n## Abstract\nbody 1")
|
||||
assert "## Section 5" in result.output_markdown
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_all_supported_line_endings(line_ending: str) -> None:
|
||||
result = transform(_document(line_ending=line_ending))
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.count(line_ending) == _document(line_ending=line_ending).count(line_ending)
|
||||
|
||||
|
||||
def test_allows_skipped_numbers_when_sequence_is_strictly_increasing() -> None:
|
||||
numbers = tuple(range(10, 30))
|
||||
result = transform(_document(numbers=numbers, heading_numbers=frozenset({14, 24})))
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
_document(count=19, heading_numbers=frozenset({5, 15})),
|
||||
_document(heading_numbers=frozenset({5})),
|
||||
_document(numbers=(*tuple(range(1, 20)), 10), heading_numbers=frozenset({5, 15})),
|
||||
_document().replace("## Abstract", "## ABSTRACT"),
|
||||
_document() + "\n## Abstract",
|
||||
],
|
||||
)
|
||||
def test_incomplete_or_ambiguous_evidence_preserves_the_document(markdown: str) -> None:
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
def test_lists_years_and_numbers_inside_body_are_not_candidates() -> None:
|
||||
markdown = _document() + "\n1. list\n1) list\n2024 report\nThe panel included 35 experts"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.endswith("1. list\n1) list\n2024 report\nThe panel included 35 experts")
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([ManuscriptLineNumberComponent()])
|
||||
first = pipeline.transform(_document())
|
||||
assert first.output_markdown is not None
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.changes == ()
|
||||
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import ModifierContractError, Pipeline, RunStatus
|
||||
from mdpolish.modifiers import mapped_line_join
|
||||
|
||||
MAPPINGS = (
|
||||
("exam-", "ple", "example"),
|
||||
("rule-", "based", "rule-based"),
|
||||
("value.", "ues", "values"),
|
||||
)
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([mapped_line_join(MAPPINGS)]).transform(markdown)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
("an exam-\nple here", "an example here"),
|
||||
("an exam-\n\nple here", "an example here"),
|
||||
("a rule-\nbased method", "a rule-based method"),
|
||||
("the value.\n\nues differ", "the values differ"),
|
||||
("an exam-\r\n\r\nple here", "an example here"),
|
||||
("an exam-\r\rple here", "an example here"),
|
||||
],
|
||||
)
|
||||
def test_applies_exact_mapping_across_supported_line_shapes(markdown: str, expected: str) -> None:
|
||||
result = transform(markdown)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == expected
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"an unknown-\nword here",
|
||||
"an exam-\n\n\nple here",
|
||||
"an exam-\r\n\nple here",
|
||||
"an EXAM-\nple here",
|
||||
"an exam-\nplemore here",
|
||||
"an xrule-\nbased method",
|
||||
],
|
||||
)
|
||||
def test_unknown_or_unsafe_boundaries_are_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_parameters_and_results_are_independent_of_mapping_order() -> None:
|
||||
first = mapped_line_join(MAPPINGS)
|
||||
second = mapped_line_join(reversed(MAPPINGS))
|
||||
markdown = "example becomes exam-\nple"
|
||||
|
||||
first_result = Pipeline([first]).transform(markdown)
|
||||
second_result = Pipeline([second]).transform(markdown)
|
||||
|
||||
assert first_result.modifiers == second_result.modifiers
|
||||
assert first_result.output_markdown == second_result.output_markdown
|
||||
|
||||
|
||||
def test_library_contains_no_default_mapping() -> None:
|
||||
modifier = mapped_line_join(())
|
||||
result = Pipeline([modifier]).transform("an exam-\nple here")
|
||||
|
||||
assert modifier.parameters == (("mappings", ()),)
|
||||
assert result.output_markdown == "an exam-\nple here"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mappings",
|
||||
[
|
||||
(("", "right", "word"),),
|
||||
(("left", "right", "two words"),),
|
||||
(("left", "right", "word"), ("left", "right", "other")),
|
||||
(("left", "right"),),
|
||||
],
|
||||
)
|
||||
def test_invalid_mappings_raise_contract_error(mappings: object) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
mapped_line_join(mappings) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([mapped_line_join(MAPPINGS)])
|
||||
first = pipeline.transform("an exam-\n\nple here")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -95,13 +95,13 @@ def test_proposal_requires_reason_edits_and_one_matching_digest() -> None:
|
||||
|
||||
def test_transform_result_enforces_status_specific_output_fields() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
error = RunError("component", "1.0.0", 0, ErrorStage.TRANSFORM, "ExampleError", "safe")
|
||||
error = RunError("modifier", "1.0.0", 0, ErrorStage.TRANSFORM, "ExampleError", "safe")
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), "a", "A")
|
||||
proposal = ProposedChange(snapshot.sha256, "reason", (edit,))
|
||||
residual = ResidualProposal(
|
||||
component_id="component",
|
||||
component_version="1.0.0",
|
||||
component_position=0,
|
||||
modifier_id="modifier",
|
||||
modifier_version="1.0.0",
|
||||
modifier_position=0,
|
||||
proposal_ref=ProposalReference(0, snapshot.sha256, 0),
|
||||
proposal=proposal,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import DocumentSnapshot, Modifier, ModifierContractError, ProposedChange
|
||||
|
||||
|
||||
def no_changes(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return ()
|
||||
|
||||
|
||||
def test_plain_function_and_mapping_parameters_create_immutable_modifier() -> None:
|
||||
source_parameters: dict[str, object] = {
|
||||
"enabled": True,
|
||||
"nested": {"count": 2},
|
||||
"values": ["a", "b"],
|
||||
}
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.2.3",
|
||||
parameters=source_parameters,
|
||||
applicability="处理虚构测试输入；排除其他文本。",
|
||||
propose=no_changes,
|
||||
)
|
||||
source_parameters["enabled"] = False
|
||||
|
||||
assert modifier.propose is no_changes
|
||||
assert modifier.parameters == (
|
||||
("enabled", True),
|
||||
("nested", (("count", 2),)),
|
||||
("values", ("a", "b")),
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
modifier.version = "2.0.0" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_normalized_parameters_are_sorted_and_validated() -> None:
|
||||
modifier = Modifier(
|
||||
modifier_id="example.normalized",
|
||||
version="1.0.0",
|
||||
parameters=(("z", 1), ("a", ("x", 2))),
|
||||
applicability="处理虚构测试输入。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
assert modifier.parameters == (("a", ("x", 2)), ("z", 1))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("modifier_id", ["", "Upper", "has space", ".leading", "trailing."])
|
||||
def test_invalid_modifier_id_is_rejected(modifier_id: str) -> None:
|
||||
with pytest.raises(ModifierContractError, match="modifier_id"):
|
||||
Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", ["", "1", "1.0", "v1.0.0", "01.0.0", "1.0.0-beta"])
|
||||
def test_invalid_version_is_rejected(version: str) -> None:
|
||||
with pytest.raises(ModifierContractError, match="version"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version=version,
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"parameters",
|
||||
[
|
||||
{"bad": {1}},
|
||||
{"bad": float("inf")},
|
||||
(("duplicate", 1), ("duplicate", 2)),
|
||||
(("bad", object()),),
|
||||
("not-a-pair",),
|
||||
],
|
||||
)
|
||||
def test_invalid_parameters_are_rejected(parameters: object) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=parameters, # type: ignore[arg-type]
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
def test_empty_applicability_and_non_callable_proposal_are_rejected() -> None:
|
||||
with pytest.raises(ModifierContractError, match="applicability"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=" ",
|
||||
propose=no_changes,
|
||||
)
|
||||
with pytest.raises(ModifierContractError, match="callable"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=cast(object, None), # type: ignore[arg-type]
|
||||
)
|
||||
@@ -1,83 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.component import ComponentContractError
|
||||
from mdpolish.components import PageBreakWordJoinComponent
|
||||
|
||||
MAPPINGS = (
|
||||
("possi-", "bly", "possibly"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([PageBreakWordJoinComponent(MAPPINGS)]).transform(markdown)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
("except possi-\nbly through care", "except possibly through care"),
|
||||
("except possi-\n\nbly through care", "except possibly through care"),
|
||||
("use SOFA-\nbased criteria", "use SOFA-based criteria"),
|
||||
("at threshold.\n\nolds of eight", "at thresholds of eight"),
|
||||
("except possi-\r\n\r\nbly now", "except possibly now"),
|
||||
("except possi-\r\rbly now", "except possibly now"),
|
||||
],
|
||||
)
|
||||
def test_applies_exact_mapping_across_approved_line_shapes(markdown: str, expected: str) -> None:
|
||||
result = transform(markdown)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == expected
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"except unknown-\nword here",
|
||||
"except possi-\n\n\nbly here",
|
||||
"except possi-\r\n\nbly here",
|
||||
"except POSSI-\nbly here",
|
||||
"except possi-\nblymore here",
|
||||
"except xSOFA-\nbased here",
|
||||
],
|
||||
)
|
||||
def test_unknown_or_unsafe_boundaries_are_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_parameters_and_results_are_independent_of_mapping_order() -> None:
|
||||
first = PageBreakWordJoinComponent(MAPPINGS)
|
||||
second = PageBreakWordJoinComponent(reversed(MAPPINGS))
|
||||
markdown = "possibly becomes possi-\nbly"
|
||||
|
||||
first_result = Pipeline([first]).transform(markdown)
|
||||
second_result = Pipeline([second]).transform(markdown)
|
||||
|
||||
assert first_result.components == second_result.components
|
||||
assert first_result.output_markdown == second_result.output_markdown
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mappings",
|
||||
[
|
||||
(("", "right", "word"),),
|
||||
(("left", "right", "two words"),),
|
||||
(("left", "right", "word"), ("left", "right", "other")),
|
||||
(("left", "right"),),
|
||||
],
|
||||
)
|
||||
def test_invalid_mappings_raise_contract_error(mappings: object) -> None:
|
||||
with pytest.raises(ComponentContractError):
|
||||
PageBreakWordJoinComponent(mappings) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([PageBreakWordJoinComponent(MAPPINGS)])
|
||||
first = pipeline.transform("except possi-\n\nbly here")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
+116
-113
@@ -1,14 +1,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import (
|
||||
Component,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
Modifier,
|
||||
Pipeline,
|
||||
ProposedChange,
|
||||
RunStatus,
|
||||
@@ -17,97 +16,84 @@ from mdpolish import (
|
||||
)
|
||||
|
||||
|
||||
class ReplaceComponent(Component):
|
||||
def __init__(
|
||||
self,
|
||||
def replace_modifier(
|
||||
needle: str,
|
||||
replacement: str,
|
||||
*,
|
||||
component_id: str,
|
||||
modifier_id: str,
|
||||
version: str = "1.0.0",
|
||||
parameters: object = None,
|
||||
applicability: str = "处理精确测试字符串，要求完整匹配，排除其他内容。",
|
||||
) -> None:
|
||||
self.needle = needle
|
||||
self.replacement = replacement
|
||||
self._component_id = component_id
|
||||
self._version = version
|
||||
self._parameters = {"needle": needle, "replacement": replacement} if parameters is None else parameters
|
||||
self._applicability = applicability
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return self._component_id
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return self._version
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return cast(Mapping[str, object], self._parameters)
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return self._applicability
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(self.needle)
|
||||
) -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(needle)
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(position, position + len(self.needle)),
|
||||
expected_text=self.needle,
|
||||
replacement=self.replacement,
|
||||
span=TextSpan(position, position + len(needle)),
|
||||
expected_text=needle,
|
||||
replacement=replacement,
|
||||
)
|
||||
return (
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=f"replace test token for {self.component_id}",
|
||||
reason=f"replace test token for {modifier_id}",
|
||||
edits=(edit,),
|
||||
),
|
||||
)
|
||||
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version=version,
|
||||
parameters={"needle": needle, "replacement": replacement},
|
||||
applicability="处理精确测试字符串；排除其他内容。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
class ExplodingComponent(ReplaceComponent):
|
||||
def __init__(self, *, trigger: str | None = None, component_id: str = "test.exploding") -> None:
|
||||
super().__init__("unused", "unused-replacement", component_id=component_id)
|
||||
self.trigger = trigger
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if self.trigger is None or snapshot.markdown == self.trigger:
|
||||
def exploding_modifier(*, trigger: str | None = None, modifier_id: str = "test.exploding") -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if trigger is None or snapshot.markdown == trigger:
|
||||
raise RuntimeError(f"SECRET source: {snapshot.markdown}")
|
||||
return ()
|
||||
|
||||
|
||||
class InvalidReturnComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.invalid-return")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version="1.0.0",
|
||||
parameters={"trigger": trigger},
|
||||
applicability="仅用于测试异常路径。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
|
||||
class StaleProposalComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.stale")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def stale_proposal_modifier() -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
stale_snapshot = DocumentSnapshot(snapshot.markdown + "!")
|
||||
edit = TextEdit(stale_snapshot.sha256, TextSpan(0, 1), stale_snapshot.markdown[0], "X")
|
||||
return (ProposedChange(stale_snapshot.sha256, "stale test proposal", (edit,)),)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="test.stale",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试过期快照。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
class OverlapComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.overlap")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def overlap_modifier() -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
first = TextEdit(snapshot.sha256, TextSpan(0, 3), snapshot.markdown[0:3], "X")
|
||||
second = TextEdit(snapshot.sha256, TextSpan(2, 4), snapshot.markdown[2:4], "Y")
|
||||
return (ProposedChange(snapshot.sha256, "overlapping test proposal", (first, second)),)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="test.overlap",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试冲突范围。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
|
||||
def test_empty_pipeline_returns_unchanged_success_for_empty_unicode_text() -> None:
|
||||
for markdown in ("", "中文\nCafe\u0301\n🙂"):
|
||||
@@ -117,14 +103,14 @@ def test_empty_pipeline_returns_unchanged_success_for_empty_unicode_text() -> No
|
||||
assert result.output_markdown == markdown
|
||||
assert result.partial_markdown is None
|
||||
assert result.changes == ()
|
||||
assert result.components == ()
|
||||
assert result.modifiers == ()
|
||||
|
||||
|
||||
def test_later_component_reads_snapshot_produced_by_earlier_component() -> None:
|
||||
def test_later_modifier_reads_snapshot_produced_by_earlier_modifier() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("初", "中", component_id="test.first"),
|
||||
ReplaceComponent("中", "终", component_id="test.second"),
|
||||
replace_modifier("初", "中", modifier_id="test.first"),
|
||||
replace_modifier("中", "终", modifier_id="test.second"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -132,15 +118,15 @@ def test_later_component_reads_snapshot_produced_by_earlier_component() -> None:
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "终"
|
||||
assert [change.component_id for change in result.changes] == ["test.first", "test.second"]
|
||||
assert [change.modifier_id for change in result.changes] == ["test.first", "test.second"]
|
||||
assert result.changes[1].before_sha256 == result.changes[0].after_sha256
|
||||
|
||||
|
||||
def test_same_input_components_and_parameters_produce_same_ordered_result() -> None:
|
||||
def test_same_input_modifiers_and_parameters_produce_same_ordered_result() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.first"),
|
||||
ReplaceComponent("b", "c", component_id="test.second"),
|
||||
replace_modifier("a", "b", modifier_id="test.first"),
|
||||
replace_modifier("b", "c", modifier_id="test.second"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -148,7 +134,7 @@ def test_same_input_components_and_parameters_produce_same_ordered_result() -> N
|
||||
|
||||
|
||||
def test_successful_pipeline_is_idempotent_on_its_output() -> None:
|
||||
pipeline = Pipeline([ReplaceComponent("old", "new", component_id="test.replace")])
|
||||
pipeline = Pipeline([replace_modifier("old", "new", modifier_id="test.replace")])
|
||||
|
||||
first = pipeline.transform("old value")
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
@@ -161,22 +147,12 @@ def test_successful_pipeline_is_idempotent_on_its_output() -> None:
|
||||
assert second.changes == ()
|
||||
|
||||
|
||||
def test_single_component_runs_through_pipeline_without_shortcut() -> None:
|
||||
component = ReplaceComponent("a", "A", component_id="test.single")
|
||||
|
||||
result = Pipeline([component]).transform("a")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "A"
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_transform_error_stops_later_components_and_keeps_only_partial_text() -> None:
|
||||
def test_transform_error_stops_later_modifiers_and_keeps_only_partial_text() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.first"),
|
||||
ExplodingComponent(),
|
||||
ReplaceComponent("b", "c", component_id="test.never-runs"),
|
||||
replace_modifier("a", "b", modifier_id="test.first"),
|
||||
exploding_modifier(),
|
||||
replace_modifier("b", "c", modifier_id="test.never-runs"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -185,14 +161,14 @@ def test_transform_error_stops_later_components_and_keeps_only_partial_text() ->
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.output_markdown is None
|
||||
assert result.partial_markdown == "b"
|
||||
assert [change.component_id for change in result.changes] == ["test.first"]
|
||||
assert [change.modifier_id for change in result.changes] == ["test.first"]
|
||||
assert len(result.errors) == 1
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
assert result.residual_proposals == ()
|
||||
|
||||
|
||||
def test_unexpected_component_error_does_not_leak_source_or_exception_message() -> None:
|
||||
result = Pipeline([ExplodingComponent()]).transform("private markdown")
|
||||
def test_unexpected_modifier_error_does_not_leak_source_or_exception_message() -> None:
|
||||
result = Pipeline([exploding_modifier()]).transform("private markdown")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.errors[0].error_type == "RuntimeError"
|
||||
@@ -201,17 +177,27 @@ def test_unexpected_component_error_does_not_leak_source_or_exception_message()
|
||||
|
||||
|
||||
def test_invalid_proposal_return_is_a_transform_contract_failure() -> None:
|
||||
result = Pipeline([InvalidReturnComponent()]).transform("abc")
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="test.invalid-return",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试非法返回值。",
|
||||
propose=propose,
|
||||
)
|
||||
result = Pipeline([modifier]).transform("abc")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "abc"
|
||||
assert result.errors[0].error_type == "ComponentContractError"
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
|
||||
|
||||
@pytest.mark.parametrize("component", [StaleProposalComponent(), OverlapComponent()])
|
||||
def test_invalid_edit_batch_fails_atomically(component: Component) -> None:
|
||||
result = Pipeline([component]).transform("abcd")
|
||||
@pytest.mark.parametrize("modifier", [stale_proposal_modifier(), overlap_modifier()])
|
||||
def test_invalid_edit_batch_fails_atomically(modifier: Modifier) -> None:
|
||||
result = Pipeline([modifier]).transform("abcd")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "abcd"
|
||||
@@ -219,11 +205,11 @@ def test_invalid_edit_batch_fails_atomically(component: Component) -> None:
|
||||
assert result.errors[0].error_type == "EditValidationError"
|
||||
|
||||
|
||||
def test_duplicate_component_ids_fail_during_preflight_before_modification() -> None:
|
||||
def test_duplicate_modifier_ids_fail_during_preflight_before_modification() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.duplicate"),
|
||||
ReplaceComponent("b", "c", component_id="test.duplicate"),
|
||||
replace_modifier("a", "b", modifier_id="test.duplicate"),
|
||||
replace_modifier("b", "c", modifier_id="test.duplicate"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -235,28 +221,45 @@ def test_duplicate_component_ids_fail_during_preflight_before_modification() ->
|
||||
assert result.errors[0].error_type == "PipelineContractError"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"component",
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.bad-version", version="1.0"),
|
||||
ReplaceComponent("a", "b", component_id="test.bad-parameters", parameters={"bad": {1}}),
|
||||
ReplaceComponent("a", "b", component_id="test.bad-applicability", applicability=""),
|
||||
],
|
||||
def test_non_modifier_entry_fails_during_preflight() -> None:
|
||||
pipeline = Pipeline(cast(list[Modifier], [object()]))
|
||||
|
||||
result = pipeline.transform("a")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
|
||||
|
||||
def test_metadata_changed_by_proposal_function_fails_before_application() -> None:
|
||||
modifier: Modifier
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
object.__setattr__(modifier, "version", "2.0.0")
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), "a", "A")
|
||||
return (ProposedChange(snapshot.sha256, "mutating metadata", (edit,)),)
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="test.mutating-metadata",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试运行期元数据变化。",
|
||||
propose=propose,
|
||||
)
|
||||
def test_invalid_component_metadata_fails_before_modification(component: Component) -> None:
|
||||
result = Pipeline([component]).transform("a")
|
||||
|
||||
result = Pipeline([modifier]).transform("a")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.changes == ()
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
|
||||
|
||||
def test_cross_component_chain_is_reported_unstable_without_a_second_round() -> None:
|
||||
def test_cross_modifier_chain_is_reported_unstable_without_a_second_round() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("bad", "good", component_id="test.to-good"),
|
||||
ReplaceComponent("good", "bad", component_id="test.to-bad"),
|
||||
replace_modifier("bad", "good", modifier_id="test.to-good"),
|
||||
replace_modifier("good", "bad", modifier_id="test.to-bad"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -267,16 +270,16 @@ def test_cross_component_chain_is_reported_unstable_without_a_second_round() ->
|
||||
assert result.partial_markdown == "bad"
|
||||
assert len(result.changes) == 2
|
||||
assert len(result.residual_proposals) == 1
|
||||
assert result.residual_proposals[0].component_id == "test.to-good"
|
||||
assert result.residual_proposals[0].modifier_id == "test.to-good"
|
||||
assert result.errors == ()
|
||||
|
||||
|
||||
def test_final_review_continues_after_error_and_keeps_valid_residual_proposal() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ExplodingComponent(trigger="done", component_id="test.review-error"),
|
||||
ReplaceComponent("done", "clean", component_id="test.residual"),
|
||||
ReplaceComponent("start", "done", component_id="test.producer"),
|
||||
exploding_modifier(trigger="done", modifier_id="test.review-error"),
|
||||
replace_modifier("done", "clean", modifier_id="test.residual"),
|
||||
replace_modifier("start", "done", modifier_id="test.producer"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -287,7 +290,7 @@ def test_final_review_continues_after_error_and_keeps_valid_residual_proposal()
|
||||
assert result.partial_markdown == "done"
|
||||
assert len(result.errors) == 1
|
||||
assert result.errors[0].stage is ErrorStage.FINAL_REVIEW
|
||||
assert result.errors[0].component_id == "test.review-error"
|
||||
assert result.errors[0].modifier_id == "test.review-error"
|
||||
assert len(result.residual_proposals) == 1
|
||||
assert result.residual_proposals[0].component_id == "test.residual"
|
||||
assert result.residual_proposals[0].modifier_id == "test.residual"
|
||||
assert result.residual_proposals[0].proposal_ref.snapshot_sha256 == result.current_sha256
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ReferenceSpacingComponent
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([ReferenceSpacingComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_normalizes_missing_and_extra_blank_lines_in_references_only() -> None:
|
||||
markdown = "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n2. Second\n\n\n3. Third"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n\n2. Second\n\n3. Third"
|
||||
assert len(result.changes) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("heading", ["# References", "## REFERENCES", "### references", "#### ReFeReNcEs"])
|
||||
def test_accepts_exact_references_heading_with_ascii_case_folding(heading: str) -> None:
|
||||
markdown = f"{heading}\n\n1. First\n2. Second"
|
||||
assert transform(markdown).output_markdown == f"{heading}\n\n1. First\n\n2. Second"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||
markdown = line_ending.join(("## References", "", "1. First", "2. Second"))
|
||||
expected = line_ending.join(("## References", "", "1. First", "", "2. Second"))
|
||||
assert transform(markdown).output_markdown == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"## Reference\n\n1. First\n2. Second",
|
||||
"## References\n\n1. First\n3. Third",
|
||||
"## References\n\n2. Second\n3. Third",
|
||||
"## References\n\n1. First\n### Subsection\n2. Second",
|
||||
"## References\r\n\r\n1. First\r\n\n2. Second",
|
||||
],
|
||||
)
|
||||
def test_ambiguous_or_mixed_sections_are_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_same_or_higher_heading_ends_section() -> None:
|
||||
markdown = "## References\n\n1. First\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "## References\n\n1. First\n\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||
|
||||
|
||||
def test_multiline_reference_uses_its_last_text_line_as_boundary() -> None:
|
||||
markdown = "## References\n\n1. First line\ncontinuation\n2. Second"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "## References\n\n1. First line\ncontinuation\n\n2. Second"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([ReferenceSpacingComponent()])
|
||||
first = pipeline.transform("## References\n\n1. First\n2. Second")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -0,0 +1,108 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Modifier, ModifierContractError, Pipeline, RunStatus, regex_replace
|
||||
|
||||
|
||||
def test_factory_returns_modifier_and_expands_capture_groups() -> None:
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.swap-date",
|
||||
version="1.0.0",
|
||||
pattern=r"(\d{4})-(\d{2})-(\d{2})",
|
||||
replacement=r"\3/\2/\1",
|
||||
applicability="处理 ISO 形状的虚构日期；不验证真实日历日期。",
|
||||
)
|
||||
|
||||
result = Pipeline([modifier]).transform("A 2026-08-26, B 2027-01-02")
|
||||
|
||||
assert isinstance(modifier, Modifier)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "A 26/08/2026, B 02/01/2027"
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].before == "2026-08-26"
|
||||
assert result.changes[0].after == "26/08/2026"
|
||||
|
||||
|
||||
def test_flags_and_factory_parameters_are_recorded() -> None:
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.case",
|
||||
version="2.0.0",
|
||||
pattern="token",
|
||||
replacement="value",
|
||||
flags=re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
|
||||
result = Pipeline([modifier]).transform("TOKEN")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.modifiers[0].modifier_id == modifier.modifier_id
|
||||
assert result.modifiers[0].parameters == modifier.parameters
|
||||
assert dict(modifier.parameters) == {
|
||||
"flags": int(re.IGNORECASE | re.MULTILINE),
|
||||
"pattern": "token",
|
||||
"replacement": "value",
|
||||
}
|
||||
|
||||
|
||||
def test_zero_match_and_no_op_replacement_are_stable() -> None:
|
||||
missing = Pipeline(
|
||||
[regex_replace(modifier_id="example.missing", version="1.0.0", pattern="missing", replacement="new")]
|
||||
).transform("text")
|
||||
no_op = Pipeline(
|
||||
[regex_replace(modifier_id="example.no-op", version="1.0.0", pattern="text", replacement="text")]
|
||||
).transform("text")
|
||||
|
||||
assert missing.output_markdown == "text"
|
||||
assert missing.changes == ()
|
||||
assert no_op.status is RunStatus.SUCCESS
|
||||
assert no_op.changes == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pattern", [r"", r"^", r"a*", r"(?=a)"])
|
||||
def test_zero_length_matches_are_rejected(pattern: str) -> None:
|
||||
if pattern == r"(?=a)":
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.zero",
|
||||
version="1.0.0",
|
||||
pattern=pattern,
|
||||
replacement="x",
|
||||
)
|
||||
result = Pipeline([modifier]).transform("a")
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.changes == ()
|
||||
return
|
||||
|
||||
with pytest.raises(ModifierContractError, match="zero-length"):
|
||||
regex_replace(
|
||||
modifier_id="example.zero",
|
||||
version="1.0.0",
|
||||
pattern=pattern,
|
||||
replacement="x",
|
||||
)
|
||||
|
||||
|
||||
def test_invalid_pattern_flags_and_input_types_are_rejected() -> None:
|
||||
with pytest.raises(ModifierContractError, match="compile"):
|
||||
regex_replace(modifier_id="example.invalid", version="1.0.0", pattern="(", replacement="x")
|
||||
with pytest.raises(ModifierContractError, match="flags"):
|
||||
regex_replace(
|
||||
modifier_id="example.flags",
|
||||
version="1.0.0",
|
||||
pattern="x",
|
||||
replacement="y",
|
||||
flags=True,
|
||||
)
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline(
|
||||
[regex_replace(modifier_id="example.space", version="1.0.0", pattern=r" {2,}", replacement=" ")]
|
||||
)
|
||||
first = pipeline.transform("a b")
|
||||
assert first.output_markdown == "a b"
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -1,64 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import RepeatedRunningHeaderComponent
|
||||
|
||||
HEADER = "## Repeated Paper Header"
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([RepeatedRunningHeaderComponent()]).transform(markdown)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_bridges_interrupted_sentence_and_deletes_other_occurrence(line_ending: str) -> None:
|
||||
markdown = line_ending.join(
|
||||
(
|
||||
"Sentence continues in",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"the next line.",
|
||||
"",
|
||||
"18. Reference",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"19. Reference",
|
||||
)
|
||||
)
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == line_ending.join(
|
||||
("Sentence continues in the next line.", "", "18. Reference", "", "19. Reference")
|
||||
)
|
||||
assert len(result.changes) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
f"before\n\n{HEADER}\n\nafter",
|
||||
f"Sentence ends.\n\n{HEADER}\n\nAfter\n\nAnother sentence.\n\n{HEADER}\n\nOther",
|
||||
f"Sentence continues\n\n{HEADER}\n\nAfter\n\nText ends.\n\n{HEADER}\n\nOther",
|
||||
f"Sentence continues\n\n{HEADER}\n\nafter\n{HEADER}\nnot blank",
|
||||
],
|
||||
)
|
||||
def test_missing_or_unsafe_group_evidence_preserves_document(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_header_matching_is_exact_and_not_keyword_based() -> None:
|
||||
markdown = "continues\n\n## Any Header\n\nfrom here\n\n18. Ref\n\n## Any Header\n\n19. Ref"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "continues from here\n\n18. Ref\n\n19. Ref"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([RepeatedRunningHeaderComponent()])
|
||||
first = pipeline.transform(f"continues\n\n{HEADER}\n\nfrom here\n\ntext\n\n{HEADER}\n\nend")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -1,177 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, DocumentSnapshot, Pipeline, ProposedChange, RunStatus, TextEdit, TextSpan
|
||||
from mdpolish.reporting import ReportingError, build_document_report, build_unified_diff
|
||||
|
||||
|
||||
class ReplaceComponent(Component):
|
||||
def __init__(self, needle: str, replacement: str, component_id: str) -> None:
|
||||
self.needle = needle
|
||||
self.replacement = replacement
|
||||
self._component_id = component_id
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return self._component_id
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {"needle": self.needle, "replacement": self.replacement}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理精确测试标记，要求完整匹配，排除其他文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(self.needle)
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(
|
||||
snapshot.sha256,
|
||||
TextSpan(position, position + len(self.needle)),
|
||||
self.needle,
|
||||
self.replacement,
|
||||
)
|
||||
return (ProposedChange(snapshot.sha256, f"replace {self.needle}", (edit,)),)
|
||||
|
||||
|
||||
class ExplodingComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("unused", "unused-replacement", "test.exploding")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
raise RuntimeError(f"private: {snapshot.markdown}")
|
||||
|
||||
|
||||
def test_success_report_replays_multiple_component_snapshots_and_derives_locations() -> None:
|
||||
markdown = "x\r\ntarget\n"
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("x", "XX", "test.expand"),
|
||||
ReplaceComponent("target", "done", "test.target"),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform(markdown)
|
||||
|
||||
report = build_document_report(
|
||||
document_id="paper",
|
||||
source_label="data/md/paper.md",
|
||||
input_markdown=markdown,
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert report.cleaned_markdown == b"XX\r\ndone\n"
|
||||
assert payload["changes"][0]["location"] == {"line": 1, "column": 1}
|
||||
assert payload["changes"][1]["location"] == {"line": 2, "column": 1}
|
||||
assert payload["changes"][1]["before"] == "target"
|
||||
assert payload["changes"][1]["after"] == "done"
|
||||
assert payload["changes"][1]["before_sha256"] == result.changes[1].before_sha256
|
||||
assert payload["changes"][1]["after_sha256"] == result.changes[1].after_sha256
|
||||
assert report.diff is not None
|
||||
assert b"--- a/paper.md\n+++ b/paper.md\n" in report.diff
|
||||
assert b"data/md/paper.md" not in report.diff
|
||||
|
||||
|
||||
def test_zero_change_success_still_has_cleaned_markdown_and_empty_diff() -> None:
|
||||
markdown = "中文\nCafe\u0301\n"
|
||||
result = Pipeline([]).transform(markdown)
|
||||
|
||||
report = build_document_report(
|
||||
document_id="unchanged",
|
||||
source_label="论文.md",
|
||||
input_markdown=markdown,
|
||||
result=result,
|
||||
)
|
||||
|
||||
assert report.cleaned_markdown == markdown.encode()
|
||||
assert report.diff == b""
|
||||
assert report.change_count == 0
|
||||
assert report.result_json.endswith(b"\n")
|
||||
assert "论文".encode() in report.result_json
|
||||
assert b"\\u4e2d" not in report.result_json
|
||||
|
||||
|
||||
def test_diff_preserves_final_newline_changes_and_uses_report_newlines() -> None:
|
||||
diff = build_unified_diff("paper", "same\r\n", "same")
|
||||
|
||||
assert b"--- a/paper.md\n+++ b/paper.md\n" in diff
|
||||
assert b"-same\n+same\n\\ No newline at end of file\n" in diff
|
||||
assert b"\r" not in diff
|
||||
|
||||
|
||||
def test_failed_report_keeps_change_audit_but_does_not_render_partial_markdown() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", "test.first"),
|
||||
ExplodingComponent(),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform("a")
|
||||
|
||||
report = build_document_report(
|
||||
document_id="failed",
|
||||
source_label="failed.md",
|
||||
input_markdown="a",
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert report.cleaned_markdown is None
|
||||
assert report.diff is None
|
||||
assert payload["output"] == {"cleaned_path": None, "diff_path": None}
|
||||
assert len(payload["changes"]) == 1
|
||||
assert payload["errors"][0]["error_type"] == "RuntimeError"
|
||||
assert "private" not in payload["errors"][0]["message"]
|
||||
assert "partial_markdown" not in payload
|
||||
|
||||
|
||||
def test_unstable_report_serializes_residual_proposals_without_partial_markdown() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("bad", "good", "test.to-good"),
|
||||
ReplaceComponent("good", "bad", "test.to-bad"),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform("bad")
|
||||
|
||||
report = build_document_report(
|
||||
document_id="unstable",
|
||||
source_label="unstable.md",
|
||||
input_markdown="bad",
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.UNSTABLE
|
||||
assert report.cleaned_markdown is None
|
||||
assert report.diff is None
|
||||
assert payload["residual_proposals"][0]["proposal"]["reason"] == "replace bad"
|
||||
assert payload["residual_proposals"][0]["proposal"]["edits"][0]["expected_text"] == "bad"
|
||||
assert "partial_markdown" not in payload
|
||||
|
||||
|
||||
def test_report_rejects_a_tampered_change_hash_chain() -> None:
|
||||
result = Pipeline([ReplaceComponent("a", "b", "test.replace")]).transform("a")
|
||||
tampered_change = replace(result.changes[0], after_sha256="0" * 64)
|
||||
tampered_result = replace(result, changes=(tampered_change,))
|
||||
|
||||
with pytest.raises(ReportingError, match="batch hash"):
|
||||
build_document_report(
|
||||
document_id="tampered",
|
||||
source_label="tampered.md",
|
||||
input_markdown="a",
|
||||
result=tampered_result,
|
||||
)
|
||||
@@ -1,132 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from reviewer.server import ReviewArtifactError, ReviewArtifacts
|
||||
from tests.reviewer_fixture import create_review_run, read_json, write_json
|
||||
|
||||
|
||||
def test_reviewer_returns_full_comparison_and_all_component_stages(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
run = cast(dict[str, Any], repository.run_summary())
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
changed = cast(dict[str, Any], repository.component_stage("paper", 0))
|
||||
unchanged = cast(dict[str, Any], repository.component_stage("paper", 1))
|
||||
|
||||
assert run["summary"] == {
|
||||
"document_count": 1,
|
||||
"success_count": 1,
|
||||
"failed_count": 0,
|
||||
"unstable_count": 0,
|
||||
"change_count": 1,
|
||||
}
|
||||
assert run["documents"][0]["source_available"] is True
|
||||
assert comparison["original_markdown"] == fixture["original"]
|
||||
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||
assert comparison["changes"][0]["editor_range"] == {"start": 4, "end": 7}
|
||||
assert changed["before_markdown"] == fixture["original"]
|
||||
assert changed["after_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["before_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["after_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["changes"] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", ["failed", "unstable"])
|
||||
def test_non_success_documents_expose_audit_but_no_formal_output(tmp_path: Path, status: str) -> None:
|
||||
fixture = create_review_run(tmp_path, status=status)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
|
||||
assert comparison["original_markdown"] == fixture["original"]
|
||||
assert comparison["cleaned_markdown"] is None
|
||||
if status == "failed":
|
||||
assert comparison["errors"]
|
||||
assert comparison["residual_proposals"] == []
|
||||
else:
|
||||
assert comparison["errors"] == []
|
||||
assert comparison["residual_proposals"]
|
||||
with pytest.raises(ReviewArtifactError, match="success"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_historical_run_without_locator_reports_unavailable_source(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path, with_locator=False)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
summary = cast(dict[str, Any], repository.run_summary())
|
||||
|
||||
assert summary["documents"][0]["source_available"] is False
|
||||
assert "review-locator.json" in summary["documents"][0]["availability_error"]
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
assert comparison["original_markdown"] is None
|
||||
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||
assert comparison["changes"][0]["editor_range"] is None
|
||||
with pytest.raises(ReviewArtifactError, match=r"review-locator\.json"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_source_hash_change_is_not_silently_displayed(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
source_path = fixture["source_path"]
|
||||
assert isinstance(source_path, Path)
|
||||
source_path.write_text("changed\n", encoding="utf-8")
|
||||
|
||||
summary = cast(dict[str, Any], repository.run_summary())
|
||||
|
||||
assert summary["documents"][0]["source_available"] is False
|
||||
assert "哈希" in summary["documents"][0]["availability_error"]
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
assert comparison["original_markdown"] is None
|
||||
assert comparison["changes"][0]["editor_range"] is None
|
||||
with pytest.raises(ReviewArtifactError, match="哈希"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_reviewer_rejects_unknown_schema_and_artifact_path_escape(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
manifest_path = fixture["manifest_path"]
|
||||
assert isinstance(manifest_path, Path)
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["schema_version"] = 2
|
||||
write_json(manifest_path, manifest)
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as unknown:
|
||||
ReviewArtifacts(fixture["run_directory"])
|
||||
assert unknown.value.code == "unsupported_schema"
|
||||
|
||||
second = create_review_run(tmp_path / "second")
|
||||
second_manifest_path = second["manifest_path"]
|
||||
assert isinstance(second_manifest_path, Path)
|
||||
second_manifest = read_json(second_manifest_path)
|
||||
second_manifest["documents"][0]["result_path"] = "../outside.json"
|
||||
write_json(second_manifest_path, second_manifest)
|
||||
with pytest.raises(ReviewArtifactError, match="路径"):
|
||||
ReviewArtifacts(second["run_directory"])
|
||||
|
||||
|
||||
def test_reviewer_rejects_tampered_snapshot_chain_and_unknown_identity(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
result_path = fixture["result_path"]
|
||||
assert isinstance(result_path, Path)
|
||||
result = read_json(result_path)
|
||||
result["changes"][0]["after_sha256"] = "0" * 64
|
||||
write_json(result_path, result)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as replay_error:
|
||||
repository.document_comparison("paper")
|
||||
assert replay_error.value.code == "untrusted_replay"
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as document_error:
|
||||
repository.document_comparison("missing")
|
||||
assert document_error.value.http_status == 404
|
||||
with pytest.raises(ReviewArtifactError) as component_error:
|
||||
repository.component_stage("paper", 99)
|
||||
assert component_error.value.http_status == 404
|
||||
@@ -1,105 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from collections.abc import Generator
|
||||
from http.client import HTTPConnection, HTTPResponse
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from reviewer.server import ReviewArtifacts
|
||||
from reviewer.server.__main__ import create_server
|
||||
from tests.reviewer_fixture import ReviewFixture, create_review_run
|
||||
|
||||
|
||||
def request(
|
||||
port: int,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
headers: dict[str, str] | None = None,
|
||||
) -> tuple[HTTPResponse, bytes]:
|
||||
connection = HTTPConnection("127.0.0.1", port, timeout=3)
|
||||
connection.request(method, path, headers=headers or {})
|
||||
response = connection.getresponse()
|
||||
content = response.read()
|
||||
connection.close()
|
||||
return response, content
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def running_server(tmp_path: Path) -> Generator[tuple[int, ReviewFixture], None, None]:
|
||||
fixture = create_review_run(tmp_path)
|
||||
static_root = tmp_path / "static"
|
||||
(static_root / "assets").mkdir(parents=True)
|
||||
(static_root / "index.html").write_text("<main>reviewer</main>\n", encoding="utf-8")
|
||||
(static_root / "assets/app.js").write_text("export {};\n", encoding="utf-8")
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
server = create_server(repository, static_root)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
yield server.server_address[1], fixture
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(timeout=3)
|
||||
|
||||
|
||||
def test_server_exposes_same_origin_api_and_static_build(
|
||||
running_server: tuple[int, ReviewFixture],
|
||||
) -> None:
|
||||
port, fixture = running_server
|
||||
|
||||
api_response, api_content = request(port, "GET", "/api/v1/run")
|
||||
page_response, page_content = request(port, "GET", "/")
|
||||
asset_response, _ = request(port, "HEAD", "/assets/app.js")
|
||||
|
||||
payload = cast(dict[str, Any], json.loads(api_content))
|
||||
assert api_response.status == 200
|
||||
assert payload["run"]["run_id"] == "review-run"
|
||||
assert str(fixture["source_path"]) not in api_content.decode()
|
||||
assert api_response.getheader("Access-Control-Allow-Origin") is None
|
||||
assert api_response.getheader("Cache-Control") == "no-store"
|
||||
assert "default-src 'self'" in cast(str, api_response.getheader("Content-Security-Policy"))
|
||||
assert page_response.status == 200
|
||||
assert page_content == b"<main>reviewer</main>\n"
|
||||
assert asset_response.status == 200
|
||||
assert asset_response.getheader("Content-Type") == "text/javascript; charset=utf-8"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "headers", "status", "code"),
|
||||
[
|
||||
("POST", "/api/v1/run", None, 405, "method_not_allowed"),
|
||||
("GET", "/api/v1/run", {"Host": "example.com"}, 403, "invalid_origin"),
|
||||
(
|
||||
"GET",
|
||||
"/api/v1/run",
|
||||
{"Origin": "http://example.com"},
|
||||
403,
|
||||
"invalid_origin",
|
||||
),
|
||||
("GET", "/api/v1/documents/missing", None, 404, "unknown_document"),
|
||||
("GET", "/api/v1/documents/paper/components/99", None, 404, "unknown_component"),
|
||||
("GET", "/assets/missing.js", None, 404, "not_found"),
|
||||
("GET", "/..%2Fsecret.txt", None, 404, "not_found"),
|
||||
],
|
||||
)
|
||||
def test_server_rejects_unsafe_or_unknown_requests(
|
||||
running_server: tuple[int, ReviewFixture],
|
||||
method: str,
|
||||
path: str,
|
||||
headers: dict[str, str] | None,
|
||||
status: int,
|
||||
code: str,
|
||||
) -> None:
|
||||
port, _ = running_server
|
||||
|
||||
response, content = request(port, method, path, headers=headers)
|
||||
payload = cast(dict[str, Any], json.loads(content))
|
||||
|
||||
assert response.status == status
|
||||
assert payload["error"]["code"] == code
|
||||
@@ -1,69 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import WordReviewCommentComponent
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([WordReviewCommentComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_deletes_single_line_comment_and_one_following_blank() -> None:
|
||||
markdown = "before\nCommented [AB1]: review this\n\nafter"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "before\nafter"
|
||||
assert len(result.changes) == 1
|
||||
assert result.changes[0].reason == "删除严格单行 Word 审阅批注及其后一个空行"
|
||||
|
||||
|
||||
def test_adjacent_comment_blocks_are_deleted_without_overlap() -> None:
|
||||
markdown = "before\nCommented [A1]: first\n\nCommented [B2R1]: second\n\nafter"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.output_markdown == "before\nafter"
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].span.end == result.changes[1].span.start
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||
markdown = line_ending.join(("before", "Commented [A1]: note", "", "after"))
|
||||
assert transform(markdown).output_markdown == line_ending.join(("before", "after"))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"comment",
|
||||
[
|
||||
"prefix Commented [A1]: note",
|
||||
" Commented [A1]: note",
|
||||
"Commented []: note",
|
||||
"Commented [A-1]: note",
|
||||
"Commented [A1]:",
|
||||
"Commented [A1]: ",
|
||||
],
|
||||
)
|
||||
def test_similar_lines_are_preserved(comment: str) -> None:
|
||||
markdown = f"before\n{comment}\n\nafter"
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_requires_a_following_blank_line_and_does_not_delete_extra_blanks() -> None:
|
||||
without_blank = "Commented [A1]: note\nafter"
|
||||
with_two_blanks = "before\nCommented [A1]: note\n\n\nafter"
|
||||
|
||||
assert transform(without_blank).output_markdown == without_blank
|
||||
assert transform(with_two_blanks).output_markdown == "before\n\nafter"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([WordReviewCommentComponent()])
|
||||
first = pipeline.transform("Commented [A1]: note\n\nafter")
|
||||
assert first.output_markdown is not None
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.changes == ()
|
||||
Reference in New Issue
Block a user