Merge pull request #4 from Bepr4/refactor/generic-functional-core
Refactor/generic functional core
This commit is contained in:
Bepr4
@@ -25,6 +25,9 @@ artifacts/
|
||||
outputs/
|
||||
experiments/
|
||||
research-wiki/scratch/
|
||||
!research-wiki/scratch/
|
||||
research-wiki/scratch/*
|
||||
!research-wiki/scratch/.gitkeep
|
||||
*.cleaned.md
|
||||
report.json
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
>
|
||||
> 1. 本仓库是实验室共用的 Markdown 清洗研究与基础工具库；已有实现不等于完整清洗工具或生产能力。
|
||||
> 2. 真实文档和外部数据默认只读，不修改、不复制、不提交。
|
||||
> 3. 面向用户的说明使用简体中文；代码、命令、路径和标识符使用英文。
|
||||
> 3. 面向用户的说明使用简体中文，代码注释使用中文；代码、命令、路径和标识符使用英文。
|
||||
> 4. `AGENTS.md` 与 `CLAUDE.md` 是同步镜像，除第一行标题外正文必须一致。
|
||||
|
||||
## 0. 事实权威
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
>
|
||||
> 1. 本仓库是实验室共用的 Markdown 清洗研究与基础工具库；已有实现不等于完整清洗工具或生产能力。
|
||||
> 2. 真实文档和外部数据默认只读，不修改、不复制、不提交。
|
||||
> 3. 面向用户的说明使用简体中文；代码、命令、路径和标识符使用英文。
|
||||
> 3. 面向用户的说明使用简体中文，代码注释使用中文；代码、命令、路径和标识符使用英文。
|
||||
> 4. `AGENTS.md` 与 `CLAUDE.md` 是同步镜像，除第一行标题外正文必须一致。
|
||||
|
||||
## 0. 事实权威
|
||||
|
||||
@@ -1,203 +1,287 @@
|
||||
# mdpolish
|
||||
|
||||
实验室共用的 Markdown 清洗研究与基础工具库。项目不属于 GovDoc 专用组件，也不只服务政务文档。
|
||||
`mdpolish` 是实验室共用的、项目无关的 Python Markdown 修改库。它提供函数式 `Modifier`、精确文本编辑执行器、
|
||||
有序 `Pipeline`、正则修改器工厂，以及少量可以用合成样例完整说明的通用修改器。
|
||||
|
||||
本仓库面向实验室内不同项目复用，用于清洗 PDF、DOCX、OCR、网页等上游管线生成的 Markdown，统一解决
|
||||
格式噪声、结构损坏、内容异常、修改追踪和多用途派生问题。各项目共享通用清洗能力，再通过独立配置或
|
||||
profile 表达论文、政务文档、RAG、文档对比等不同需求。
|
||||
当前源码包版本是 `0.3.0`。库只处理内存中的 Markdown 字符串，不读取或写入文件，不提供默认流水线，也不包含任何
|
||||
项目的规则集合、数据清单、实验脚本或评审工具。`v0.3.0` tag 和 GitHub Release 尚未创建，因此当前源码是交付候选，
|
||||
不是已经发布的版本。
|
||||
|
||||
仓库当前已从纯文档治理进入第一版核心和 ClinDB 第一批组件实现阶段：已经提供可安装的 Python 内存处理包、
|
||||
8 个论文清洗组件、本地实验入口，以及同仓但与清洗运行解耦的只读评审前端。实验可以保存指定论文的成功输出、
|
||||
审计和 diff；评审前端可以同时查看清洗前后全文，并逐组件复放修改。仓库仍不提供面向任意数据集的完整规则集、
|
||||
公共命令行工具、通用文件适配器或生产接口，因此目前还不是拿来即可完成任意 Markdown 清洗的成品工具。
|
||||
## 当前能力
|
||||
|
||||
## 当前阶段
|
||||
| 能力 | 作用 | 明确边界 |
|
||||
| --- | --- | --- |
|
||||
| `Modifier` | 把不可变元数据与普通提议函数组合起来 | 函数只提议修改，不直接改字符串或文件 |
|
||||
| 精确编辑执行器 | 校验快照、范围、原文、重复和冲突后原子应用一个批次 | 不判断项目业务语义 |
|
||||
| `Pipeline` | 按调用方顺序运行修改器，并对最终快照做只读稳定性复查 | 不自动选规则、不重排、不循环执行 |
|
||||
| `regex_replace()` | 把非空正则匹配转换为精确编辑 | 不提供规则注册表、配置加载或默认模式 |
|
||||
| `mapped_line_join()` | 用精确、正则或可选本地词典规则合并跨行片段 | 无默认规则；代码、表格、未知结构和歧义失败关闭 |
|
||||
| HTML 表格修改器 | 处理严格表格子集的实体和单行布局 | 不是完整 HTML parser，也不是 HTML→GFM 转换器 |
|
||||
|
||||
项目当前已经进入第一版可执行核心和真实组件验证阶段：
|
||||
一次运行会返回 `success`、`failed` 或 `unstable`：
|
||||
|
||||
- 提供可安装的 Python 3.11+ 内存处理包，运行时只依赖标准库；
|
||||
- 已实现不可变数据契约、组件基类、原子修改执行器、顺序流水线、审计记录和最终稳定性复查；
|
||||
- 已实现 ClinDB 第一批 8 个正式组件，覆盖 Word 批注、手稿行号、arXiv 戳、重复页眉、批准映射断词、
|
||||
HTML 表格实体与布局、参考文献空行；
|
||||
- 已有仓库内实验运行层，能严格读取显式清单、保存成功 Markdown、JSON 审计、unified diff 和评审定位文件，
|
||||
并保持输入不变；
|
||||
- 已有独立的 `reviewer/` 本地只读前端，服务只消费一次已发布的产物和定位文件，并与报告层共用纯 Python 快照重放逻辑，
|
||||
不导入组件、不调用流水线，也不重新清洗；
|
||||
- 第一批流水线已对 5 份论文 Markdown 完成保存型实验，5/5 成功，共记录 155 条修改，第二次运行零修改；
|
||||
- 当前基础检查为 Ruff、mypy、229 项 pytest 测试，以及前端 ESLint、TypeScript、9 项 Vitest 测试和生产构建，
|
||||
实际命令见本文“当前可用检查”。
|
||||
- `success`：所选修改器完成运行，且对最终结果不再提出修改；
|
||||
- `failed`：修改器、契约或编辑验证发生错误；
|
||||
- `unstable`：运行没有错误，但最终复查仍发现有效候选修改。
|
||||
|
||||
项目还没有面向任意数据集的完整清洗规则集、Markdown/HTML 通用 parser、profile 格式、通用文件输入接口、公共 CLI、
|
||||
通用批处理或生产接口。评审前端也不是公共 Web 服务：它只绑定本机回环地址，只读展示 Markdown 源文和审计，不提供
|
||||
渲染预览、在线编辑或重新清洗。当前 first-batch 实验脚本只固定运行已批准的 5 份论文和 8 个组件；它只能证明当前
|
||||
8 类确定规则已经闭环，不能据此认为论文中的缺失内容、乱码、复杂表格、图片或 GovDoc 已具备完整清洗能力。
|
||||
`success` 只代表本次选择的修改器已经稳定，不代表文档不存在其他质量问题。
|
||||
|
||||
## 服务对象与复用目标
|
||||
## 安装
|
||||
|
||||
当前已经明确的使用场景包括：
|
||||
正式版本只计划通过 GitHub tag 和 GitHub Release 交付，不发布到 PyPI 或其他 Python 包索引。`v0.3.0` 发布后，
|
||||
使用项目可以固定不可移动的 tag：
|
||||
|
||||
- **论文清洗**：`data/` 中现有内容来自师姐的项目，包含论文 PDF 的 Markdown、JSON、图片等转换产物；
|
||||
- **GovDoc**：政务、招投标、采购、合同等文档的清洗、比对和 RAG 前处理；
|
||||
- **未来实验室项目**：后续可以继续接入其他需要 Markdown 质量检查、规范化或用途派生的项目数据。
|
||||
|
||||
当前用例只用于发现真实问题和验证通用能力，不能反过来限定库的设计。核心代码不得依赖论文 DOI、
|
||||
GovDoc 目录、具体客户名称或某一转换器的固定输出路径。
|
||||
|
||||
## 测试数据
|
||||
|
||||
- **论文 Markdown**：`data/md/`，共 5 份，按 ClinDB-ReviewBench 中使用的论文缩写命名，
|
||||
供本地查看和组件只读验证；
|
||||
- **GovDoc Markdown**：`/home/lihaoze/gov_test_data/compare`，共 7 组、45 份，输入位于各组 `uploads/` 下，
|
||||
保持仓库外只读，不复制到本项目。
|
||||
|
||||
两组数据都不是可提交的自动测试 fixture。`data/` 已被 Git 忽略，清洗实验不得覆盖这些输入。
|
||||
|
||||
本地清洗实验产物位于 `artifacts/<YYYY-MM-DD>/runs/<run_id>/`。产物可能包含完整原文；其中评审定位文件还包含
|
||||
输入源的绝对路径，因此整个目录均受 Git 忽略，默认保留 30 个日历日，不得提交或复制到外部系统。当前只批准为上述
|
||||
5 份论文副本保存产物，未批准保存 GovDoc 输出。
|
||||
|
||||
## 面向复用的设计原则
|
||||
|
||||
- **通用核心**：只接收 Markdown；第一版只执行确定、可审计的精确修改，不读取 PDF、图片或转换器 JSON；
|
||||
- **输入边界**：PDF/OCR/DOCX/HTML 转换和外部材料核验由使用项目或上游流程负责，不写入共用组件契约；
|
||||
- **项目 profile**：论文、GovDoc、对比、RAG、公开脱敏等规则独立组合，不互相污染默认行为；
|
||||
- **保真优先**：不确定内容默认保留；当前核心不猜测修改，也不承担人工确认流程；
|
||||
- **可复现**：规则、配置、输入哈希、输出和每次变更都可以追踪；
|
||||
- **可扩展**：新增项目在自身边界处理上游适配，并主要组合或补充组件和 profile，而不是复制一套清洗器。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```text
|
||||
mdpolish/
|
||||
├── .gitignore
|
||||
├── AGENTS.md
|
||||
├── CLAUDE.md
|
||||
├── README.md
|
||||
├── pyproject.toml # Python 包、构建和开发检查配置
|
||||
├── src/
|
||||
│ └── mdpolish/
|
||||
│ ├── __init__.py # 第一版核心公共导出
|
||||
│ ├── _artifact_replay.py # 报告与评审器共用的纯快照重放
|
||||
│ ├── component.py # 组件基类和元数据契约
|
||||
│ ├── edits.py # 文本编辑验证与原子应用
|
||||
│ ├── experiment.py # 本地实验输入预检与批量编排
|
||||
│ ├── models.py # 不可变数据模型和运行状态
|
||||
│ ├── pipeline.py # 顺序执行和最终稳定性复查
|
||||
│ ├── reporting.py # JSON 审计、行列位置和 unified diff
|
||||
│ ├── artifact_store.py # 私有产物目录和原子发布
|
||||
│ ├── _html_table.py # 严格 HTML 表格词法范围
|
||||
│ ├── _text_ranges.py # 精确物理行与换行范围
|
||||
│ ├── py.typed # 类型信息声明
|
||||
│ └── components/
|
||||
│ ├── __init__.py
|
||||
│ ├── arxiv_submission_stamp.py
|
||||
│ ├── html_table_double_escape.py
|
||||
│ ├── html_table_layout.py
|
||||
│ ├── manuscript_line_number.py
|
||||
│ ├── page_break_word_join.py
|
||||
│ ├── reference_spacing.py
|
||||
│ ├── repeated_running_header.py
|
||||
│ └── word_review_comment.py
|
||||
├── scripts/
|
||||
│ ├── run_clindb_arxiv_experiment.py # 只含 arXiv 组件的历史实验入口
|
||||
│ └── run_clindb_first_batch_experiment.py # ClinDB 第一批 8 组件实验入口
|
||||
├── reviewer/ # 与组件和流水线解耦的本地只读评审器
|
||||
│ ├── .nvmrc # 前端开发使用 Node.js 24
|
||||
│ ├── package.json # React 依赖、检查和构建命令
|
||||
│ ├── server/ # Python 产物适配、快照重放接口和本地 HTTP 服务
|
||||
│ ├── src/
|
||||
│ │ ├── client/ # React 全文对比、组件时间线和审计界面
|
||||
│ │ └── shared/ # 浏览器使用的内部 API 类型
|
||||
│ └── tests/ # 前端响应、界面和源码安全测试
|
||||
├── tests/
|
||||
│ ├── test_arxiv_submission_stamp.py
|
||||
│ ├── test_artifact_store.py
|
||||
│ ├── test_clindb_first_batch_pipeline.py
|
||||
│ ├── test_component.py
|
||||
│ ├── test_edits.py
|
||||
│ ├── test_experiment.py
|
||||
│ ├── test_html_table_double_escape.py
|
||||
│ ├── test_html_table_layout.py
|
||||
│ ├── test_manuscript_line_number.py
|
||||
│ ├── test_models.py
|
||||
│ ├── test_page_break_word_join.py
|
||||
│ ├── test_pipeline.py
|
||||
│ ├── test_reference_spacing.py
|
||||
│ ├── test_repeated_running_header.py
|
||||
│ ├── test_reporting.py
|
||||
│ ├── test_word_review_comment.py
|
||||
│ ├── test_artifact_replay.py
|
||||
│ ├── test_reviewer_artifacts.py
|
||||
│ └── test_reviewer_server.py
|
||||
├── data/ # 本地测试数据；Git 忽略；此处只展开常用入口
|
||||
│ └── md/
|
||||
│ ├── dmp.md
|
||||
│ ├── ejhf.md
|
||||
│ ├── jama.md
|
||||
│ ├── sim.md
|
||||
│ └── springer.md
|
||||
├── artifacts/ # 本地敏感实验产物；Git 忽略
|
||||
│ └── <YYYY-MM-DD>/runs/<run_id>/
|
||||
│ ├── manifest.json
|
||||
│ ├── review-locator.json # 输入源定位和哈希；仅供本地评审
|
||||
│ └── documents/
|
||||
└── research-wiki/
|
||||
├── README.md # Wiki 分类与维护规则
|
||||
├── design/ # 批准前的选择；批准后冻结
|
||||
├── explanation/ # 当前有效机制及原因
|
||||
├── reference/ # 稳定查询事实
|
||||
├── guides/ # 已验证操作步骤
|
||||
└── scratch/ # 调研和未收敛材料
|
||||
```bash
|
||||
python -m pip install 'mdpolish @ git+https://github.com/Bepr4/mdpolish.git@v0.3.0'
|
||||
python -m pip install 'mdpolish[lexical] @ git+https://github.com/Bepr4/mdpolish.git@v0.3.0'
|
||||
python -m pip install 'mdpolish[frequency] @ git+https://github.com/Bepr4/mdpolish.git@v0.3.0'
|
||||
```
|
||||
|
||||
## 开始工作
|
||||
也可以安装同一 GitHub Release 附带的 wheel；下面的 URL 只有在 Release 实际创建后才存在：
|
||||
|
||||
进入仓库后依次阅读：
|
||||
```bash
|
||||
python -m pip install 'mdpolish[lexical] @ https://github.com/Bepr4/mdpolish/releases/download/v0.3.0/mdpolish-0.3.0-py3-none-any.whl'
|
||||
```
|
||||
|
||||
1. 本文件，确认当前阶段；
|
||||
2. `AGENTS.md` 或 `CLAUDE.md`，确认协作与安全边界；
|
||||
3. `research-wiki/README.md`，确认文档应放在哪里；
|
||||
4. 与任务直接相关的 `research-wiki/design/` 记录。
|
||||
Release 页面同时提供 wheel 的 SHA-256 校验值。仓库或 Release 如果是私有的，调用方需要自行配置 GitHub 访问权限；
|
||||
库不会保存凭据。开发环境仍从本地工作树安装：
|
||||
|
||||
第一版核心机制见 `research-wiki/explanation/first-executable-core.md`，ClinDB 第一批组件见
|
||||
`research-wiki/explanation/clindb-first-batch-components.md`，本地实验产物机制见
|
||||
`research-wiki/explanation/local-experiment-artifacts.md`，本地评审器机制见
|
||||
`research-wiki/explanation/local-markdown-reviewer.md`。实验和评审器的实际操作分别见
|
||||
`research-wiki/guides/run-local-clindb-first-batch-experiment.md` 和 `research-wiki/guides/review-local-cleaning-run.md`。
|
||||
解析器、CLI、文件适配器、profile 格式、图片资产打包和独立检查能力仍需分别设计；当前本地实验适配器和评审器不能
|
||||
被推导成这些公共接口已经获批。
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv/bin/python -m pip install -e '.[dev]'
|
||||
```
|
||||
|
||||
核心运行时只依赖 Python 标准库，支持 Python 3.11 及以上版本。自动词典规则需要调用方明确安装并选择对应 extra：
|
||||
|
||||
```bash
|
||||
python -m pip install '/path/to/mdpolish[lexical]' # pyspellchecker + Pyphen
|
||||
python -m pip install '/path/to/mdpolish[frequency]' # wordfreq，体积和传递依赖更大
|
||||
```
|
||||
|
||||
安装 extra 不会自动启用规则，也不会触发在线下载或改变精确/正则规则行为。
|
||||
|
||||
## 组装自己的流水线
|
||||
|
||||
项目拥有规则、参数和顺序。下面的正则规则与两个通用修改器最终都是 `Modifier`：
|
||||
|
||||
```python
|
||||
from mdpolish import Pipeline, RunStatus, regex_replace
|
||||
from mdpolish.modifiers import html_table_entity_unescape, mapped_line_join
|
||||
|
||||
normalize_spaces = regex_replace(
|
||||
modifier_id="my_project.normalize_spaces",
|
||||
version="1.0.0",
|
||||
pattern=r" {2,}",
|
||||
replacement=" ",
|
||||
applicability="把正文中连续两个及以上的 ASCII 空格收敛为一个；项目需自行排除不适用区域。",
|
||||
)
|
||||
|
||||
join_fragments = mapped_line_join(
|
||||
(
|
||||
("exam-", "ple", "example"),
|
||||
("rule-", "based", "rule-based"),
|
||||
)
|
||||
)
|
||||
|
||||
pipeline = Pipeline(
|
||||
(
|
||||
normalize_spaces,
|
||||
join_fragments,
|
||||
html_table_entity_unescape(),
|
||||
)
|
||||
)
|
||||
|
||||
result = pipeline.transform("an exam-\nple text")
|
||||
if result.status is not RunStatus.SUCCESS:
|
||||
raise RuntimeError(f"cleaning did not succeed: {result.status}")
|
||||
|
||||
assert result.output_markdown == "an example text"
|
||||
```
|
||||
|
||||
## 自动英文断词
|
||||
|
||||
普通英文断词不需要逐词维护映射。调用方安装 `lexical` extra 后，可以明确选择 `pyspellchecker` 的本地词典，
|
||||
并用 Pyphen 排除不合法的断词位置：
|
||||
|
||||
```python
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.modifiers import mapped_line_join
|
||||
from mdpolish.modifiers.mapped_line_join import (
|
||||
LexicalCandidateForm,
|
||||
LexicalLineJoinRule,
|
||||
LexiconBackend,
|
||||
)
|
||||
|
||||
join_english_wraps = mapped_line_join(
|
||||
(
|
||||
LexicalLineJoinRule(
|
||||
rule_id="english.dehyphenate",
|
||||
left_pattern=r"[A-Za-z]+",
|
||||
right_pattern=r"[a-z]+",
|
||||
separator="-",
|
||||
backend=LexiconBackend.SPELLCHECKER,
|
||||
language="en",
|
||||
candidate_forms=(
|
||||
LexicalCandidateForm.JOINED,
|
||||
LexicalCandidateForm.HYPHENATED,
|
||||
),
|
||||
minimum_score=1.0,
|
||||
minimum_score_margin=0.5,
|
||||
hyphenation_language="en_US",
|
||||
case_sensitive=False,
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
result = Pipeline((join_english_wraps,)).transform("an exam-\nple")
|
||||
if result.status is not RunStatus.SUCCESS:
|
||||
raise RuntimeError(f"cleaning did not succeed: {result.status}")
|
||||
|
||||
assert result.output_markdown == "an example"
|
||||
```
|
||||
|
||||
`pyspellchecker` 的内置语言词典不支持大小写敏感查询，因此这个 backend 要求调用方明确写出
|
||||
`case_sensitive=False`；库不会替调用方改变该开关或切换 backend。上面的分数和 margin 只用于演示完整配置，
|
||||
不是适合所有项目的生产阈值。`JOINED` 与源 `HYPHENATED` 会共同参与候选选择；词典不能确认唯一结果时保持原文。
|
||||
|
||||
库不会保存 `output_markdown`。调用项目应先检查状态，再自行决定写入位置：
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
source_path = Path("input.md")
|
||||
output_path = Path("output.md")
|
||||
|
||||
result = pipeline.transform(source_path.read_text(encoding="utf-8"))
|
||||
if result.status is RunStatus.SUCCESS and result.output_markdown is not None:
|
||||
output_path.write_text(result.output_markdown, encoding="utf-8")
|
||||
```
|
||||
|
||||
文件读取、输出命名、覆盖策略、批处理、CLI 和报告都属于调用项目，不属于 `mdpolish`。
|
||||
|
||||
## 编写项目自己的修改器
|
||||
|
||||
复杂规则使用普通函数返回精确候选修改，不需要继承库基类：
|
||||
|
||||
```python
|
||||
from mdpolish import DocumentSnapshot, Modifier, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
|
||||
def propose_marker_removal(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
marker = "[REMOVE-ME]"
|
||||
start = snapshot.markdown.find(marker)
|
||||
if start < 0:
|
||||
return ()
|
||||
end = start + len(marker)
|
||||
return (
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason="删除项目确认过的占位标记",
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=marker,
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
remove_marker = Modifier(
|
||||
modifier_id="my_project.remove_marker",
|
||||
version="1.0.0",
|
||||
parameters={"marker": "[REMOVE-ME]"},
|
||||
applicability="只删除项目声明的完整占位标记；不处理近似文本。",
|
||||
propose=propose_marker_removal,
|
||||
)
|
||||
```
|
||||
|
||||
修改器决定“建议改哪里、为什么改、改成什么”；公共执行器真正修改内存字符串。每项 `TextEdit` 都绑定当前快照哈希、
|
||||
半开字符串范围、预期原文和替换文本。任意候选无效时，该修改器当前批次不会产生部分修改。
|
||||
|
||||
项目函数应保持确定且无副作用，不读取文件、网络、环境变量、当前时间或随机数。Python 无法沙箱隔离任意函数；项目
|
||||
函数私下产生的外部副作用不属于本库的验证或审计范围。
|
||||
|
||||
## 通用修改器的严格边界
|
||||
|
||||
`mapped_line_join()` 保留 `(left, right, replacement)` 三元组，也接受模块
|
||||
`mdpolish.modifiers.mapped_line_join` 中的不可变规则值。规则可以使用精确片段、两侧命名正则、只约束右侧的行尾正则，
|
||||
或从行尾与行首自动生成 `JOINED` / `HYPHENATED` / `SPACED` 候选并查询显式选择的本地词典。自动规则不要求逐词维护
|
||||
映射；精确规则和 `KeepLineJoinRule` 用于项目词、例外与否决。
|
||||
|
||||
调用方还要显式决定块范围、换行处理、冲突策略、大小写和 Unicode 规范化。默认区分大小写且不规范化；支持相邻行或
|
||||
中间最多一个同风格空行，并可在一次提议内完成多行链式合并。保守词法扫描只正向识别段落、ATX 标题 continuation、
|
||||
列表 continuation 和同深度引用；代码块、GFM pipe table、raw HTML table、混合候选行尾及无法确认的容器保持原文。
|
||||
完整公共模型、选择流程和限制见
|
||||
[`0009-generalized-mapped-line-join.md`](research-wiki/design/0009-generalized-mapped-line-join.md)，真实 backend 验收、
|
||||
版本身份和 GitHub 发布边界见
|
||||
[`0010-first-cross-project-library-delivery.md`](research-wiki/design/0010-first-cross-project-library-delivery.md)。
|
||||
|
||||
`html_table_entity_unescape()` 只在严格完整的 `<td>` / `<th>` 文本中处理 `&lt;`、`&gt;` 和
|
||||
`&amp;`。`html_table_layout()` 只调整严格单行表格的外层行布局，并保留标签、属性和单元格内容。
|
||||
|
||||
这两个 HTML 修改器采用保守的词法子集，不识别 Markdown 围栏。混合换行、嵌套标签、`<tbody>`、嵌套表格或损坏
|
||||
结构会保持原样。
|
||||
|
||||
## 仓库结构
|
||||
|
||||
```text
|
||||
src/mdpolish/
|
||||
├── models.py # 不可变快照、精确编辑、审计和结果模型
|
||||
├── modifier.py # 函数式 Modifier 契约
|
||||
├── edits.py # 批次验证与原子应用
|
||||
├── pipeline.py # 有序执行与最终稳定性复查
|
||||
├── regex.py # 正则修改器工厂
|
||||
└── modifiers/ # 少量项目无关的通用修改器
|
||||
tests/ # 只使用虚构文本的核心与通用修改器测试
|
||||
research-wiki/
|
||||
├── design/ # 已批准决策及被冻结的历史记录
|
||||
├── explanation/ # 当前有效机制
|
||||
├── reference/ # 代码无法完整表达的稳定查询事实
|
||||
├── guides/ # 已实际验证的操作步骤
|
||||
└── scratch/ # 不作为当前事实的本地草稿
|
||||
```
|
||||
|
||||
## 当前不提供
|
||||
|
||||
- 文件适配器、公共 CLI、配置文件、profile 或批处理协议；
|
||||
- 自动规则发现、注册表或默认流水线；
|
||||
- Markdown AST、完整 HTML parser 或必装的第三方运行依赖；
|
||||
- artifact、报告、Web/桌面评审器；
|
||||
- 任何业务项目的规则、固定参数、文档 ID、数据或验收统计。
|
||||
|
||||
公共边界与原因见
|
||||
[`0008-generic-functional-library-boundary.md`](research-wiki/design/0008-generic-functional-library-boundary.md)，当前机制见
|
||||
[`functional-modifier-core.md`](research-wiki/explanation/functional-modifier-core.md)。旧 design 只保存历史决策，不代表当前
|
||||
交付能力。
|
||||
|
||||
## 当前可用检查
|
||||
|
||||
在已经安装开发依赖的仓库环境中运行：
|
||||
|
||||
```bash
|
||||
# 建立隔离环境并安装包与开发检查工具
|
||||
python -m venv .venv
|
||||
.venv/bin/python -m pip install -e '.[dev]'
|
||||
|
||||
# 第一版核心的基础验收
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/mypy src tests
|
||||
.venv/bin/pytest
|
||||
.venv/bin/python -m pip wheel . --no-deps --wheel-dir /tmp/mdpolish-wheel-check
|
||||
```
|
||||
|
||||
# 本地评审器要求 Node.js 24 LTS；安装依赖后依次执行静态检查、测试和生产构建
|
||||
cd reviewer
|
||||
nvm use
|
||||
npm ci
|
||||
npm run check
|
||||
cd ..
|
||||
检查本地变更：
|
||||
|
||||
# 两份 Agent 入口除标题外必须一致；无输出且退出码为 0 表示通过
|
||||
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||
|
||||
# 查看当前 Wiki 中实际存在的文档
|
||||
find research-wiki -maxdepth 2 -type f | sort
|
||||
|
||||
# 检查本地变更
|
||||
```bash
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
上述 Python 验收已于 2026-08-24 在 Python 3.13.11 环境实际运行：Ruff 通过，mypy 检查 43 个源码、测试和实验脚本文件
|
||||
无问题，pytest 共 229 项测试通过。评审器验收也在当前用户 nvm 的 Node.js 24.19.0 环境实际运行：ESLint 和 TypeScript
|
||||
通过，Vitest 共 9 项测试通过，生产构建成功；并以一批真实的 5 文档、8 组件、155 条修改产物验证了接口读取和逐阶段哈希。
|
||||
当前环境没有可用的图形浏览器，因此页面视觉布局尚未进行真实浏览器人工验收。`requires-python` 仍以
|
||||
`pyproject.toml` 声明的 Python 3.11 及以上为准；本次结果不等于已经在每个受支持版本上完成兼容性验证。
|
||||
上述检查已于 2026-08-27 实际运行。Python 3.13.11 核心开发环境中 Ruff 和 mypy 通过，pytest 为
|
||||
`170 passed, 3 skipped`；三个 skip 是该环境没有安装的真实 optional backend 路径，不计入发布验收。
|
||||
|
||||
从 `mdpolish-0.3.0-py3-none-any.whl` 安装全部 extras 后，Python 3.11.16 和 Python 3.13.11 环境分别得到
|
||||
`173 passed`，没有 skip。仓库外消费者 smoke test 已覆盖核心精确/正则规则、`pyspellchecker + Pyphen` 和
|
||||
`wordfreq` 的最终输出与版本审计。wheel 共 16 个文件，只包含通用 Python 包、类型标记和包元数据；不包含 tests、
|
||||
Wiki、真实数据或项目规则。当前候选 wheel 的 SHA-256 是
|
||||
`a510ae1755c281f7b40262e242441882f3ccbc3fbc2174c6f5db3e8a8ae85060`。
|
||||
|
||||
上述结果证明当前候选可安装并按合成契约运行，不代表任意词典阈值已经在真实业务语料上达到生产准确率。
|
||||
|
||||
+9
-2
@@ -4,12 +4,19 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "mdpolish"
|
||||
version = "0.1.0"
|
||||
description = "Deterministic in-memory core for composing Markdown cleaning components"
|
||||
version = "0.3.0"
|
||||
description = "Deterministic functional core for composing exact Markdown modifiers"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.optional-dependencies]
|
||||
frequency = [
|
||||
"wordfreq>=3.1.1,<4",
|
||||
]
|
||||
lexical = [
|
||||
"pyphen>=0.18.1,<1",
|
||||
"pyspellchecker>=0.9,<1",
|
||||
]
|
||||
dev = [
|
||||
"mypy>=1.15,<2",
|
||||
"pytest>=8.3,<10",
|
||||
|
||||
@@ -0,0 +1,408 @@
|
||||
# 0008：函数式通用库边界与项目组件外置
|
||||
|
||||
## 状态
|
||||
|
||||
已于 2026-08-26 经用户明确批准。本文自批准起冻结；后续若改变这里的公共边界或修改语义，应新增 design，
|
||||
不得回写本文掩盖决策变化。
|
||||
|
||||
`supersedes: 0003`（范围有限）：本文拟替代基于 `Component` 抽象基类的扩展接口和当前公共导出形式；
|
||||
快照绑定、精确修改、整批验证、原子应用、显式顺序、单轮执行和最终稳定性复查继续保留。
|
||||
|
||||
`supersedes: 0005, 0007`（仓库职责范围）：本地实验产物和评审器不再属于通用库交付物。它们曾经完成的实验
|
||||
和验证仍是历史事实，但不继续作为 `mdpolish` 的当前能力维护。
|
||||
|
||||
`supersedes: 0006`（仓库职责范围）：ClinDB 第一批组件、固定参数、流水线和验收计数不再由通用库拥有。
|
||||
本文不否定这些规则当时在 5 份论文上的验证结果，只改变它们今后的代码归属。
|
||||
|
||||
`0001`、`0002` 中的保真优先、项目显式组装、Markdown 单一输入、核心无文件 I/O 和外部真实材料只读边界继续有效。
|
||||
历史 design 保持冻结，不回写成新的决定。
|
||||
|
||||
## 1. 问题与可观察现象
|
||||
|
||||
仓库的底层修改执行器和 `Pipeline` 不认识具体项目，但当前交付物已经与 ClinDB 深度绑定：
|
||||
|
||||
- 发布包内包含 `paper.*` 组件，其中多项识别条件来自 5 份论文的特定形态；
|
||||
- first-batch 脚本固定了 5 个文档 ID、8 个组件、6 条断词映射和执行顺序；
|
||||
- README、reference、guide 和 explanation 以 ClinDB 的 155 条修改作为主要成果；
|
||||
- artifact、review locator 和 React 评审器围绕这套本地实验流程继续扩张；
|
||||
- 外部使用者若只需要安全修改内核，仍会看到项目术语、实验入口和第二套 Node.js 工具链。
|
||||
|
||||
这与新的目标不一致。新的 `mdpolish` 应是一个真正可复用的 Python 库：它定义怎样描述、组合、校验和应用修改，
|
||||
但不拥有任何项目的规则集合、数据清单、流水线或评审流程。
|
||||
|
||||
当前 `Component` 已经被约束为确定、无副作用的对象，但外部扩展仍需要继承抽象基类、实现四个属性和一个私有方法。
|
||||
这里的继承没有提供运行时隔离，反而增加了编写简单修改器的仪式。通用库更适合把行为表达为纯函数，把身份和参数表达为
|
||||
不可变数据，再由 `Pipeline` 组合。
|
||||
|
||||
## 2. 目标与非目标
|
||||
|
||||
### 2.1 目标
|
||||
|
||||
- 把仓库收敛为可安装、项目无关的 Python 修改库；
|
||||
- 以纯函数加不可变元数据代替 `Component` 抽象基类；
|
||||
- 保留当前快照、精确编辑、冲突检查、原子应用、审计和稳定性复查不变量；
|
||||
- 提供通用的正则修改器工厂，使项目规则可以在项目仓库中声明；
|
||||
- 提供少量经合成测试验证、无需项目数据的通用修改器；
|
||||
- 让任何项目显式创建自己的修改器、参数和 `Pipeline`；
|
||||
- 从当前能力、Python 包、测试和用户文档中移除 ClinDB、论文缩写、固定映射和真实样本计数；
|
||||
- 移除通用库不再拥有的本地实验层和评审器工具链；
|
||||
- 保持运行时只依赖 Python 标准库。
|
||||
|
||||
### 2.2 非目标
|
||||
|
||||
- 不在本轮建设公共 CLI、配置文件、profile、插件发现、LSP、Web 服务或桌面应用；
|
||||
- 不建立新的 artifact schema、文件适配器或批处理协议；
|
||||
- 不把 ClinDB 组件迁移到另一个仓库；修改其他仓库需要另行授权；
|
||||
- 不修改、移动、复制或删除本地 `data/`、`artifacts/` 和仓库外真实材料；
|
||||
- 不把现有项目规则改名后冒充通用组件；
|
||||
- 不承诺任意用户函数在运行时被沙箱隔离；纯函数和无副作用是修改器契约，由内置实现和测试保证；
|
||||
- 不在本轮引入 Markdown parser、HTML parser 或新的运行依赖；
|
||||
- 不提供默认流水线。安装库或导入修改器不会自动修改任何文本。
|
||||
|
||||
## 3. 新的依赖方向
|
||||
|
||||
```text
|
||||
使用项目
|
||||
├── 项目规则函数
|
||||
├── 项目参数与顺序
|
||||
└── 项目文件、CLI、profile、报告和评审
|
||||
│
|
||||
▼
|
||||
mdpolish
|
||||
├── 不可变快照与修改模型
|
||||
├── 函数式 Modifier 协议
|
||||
├── 修改验证与原子执行器
|
||||
├── Pipeline
|
||||
├── 通用正则修改器工厂
|
||||
└── 少量通用内置修改器
|
||||
```
|
||||
|
||||
依赖只能从使用项目指向 `mdpolish`。`mdpolish` 不导入项目包，不读取项目目录，不根据文件名选择规则，
|
||||
也不保存任何项目的默认顺序或参数。
|
||||
|
||||
## 4. 函数式修改器接口
|
||||
|
||||
### 4.1 行为与元数据分开
|
||||
|
||||
修改行为使用一个普通可调用对象：
|
||||
|
||||
```python
|
||||
ModifierFunction = Callable[
|
||||
[DocumentSnapshot],
|
||||
tuple[ProposedChange, ...],
|
||||
]
|
||||
```
|
||||
|
||||
库使用不可变的 `Modifier` 保存审计所需元数据和函数：
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Modifier:
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
propose: ModifierFunction
|
||||
```
|
||||
|
||||
精确字段名可以在实现中根据类型检查做机械调整，但必须满足以下决定：
|
||||
|
||||
- 不要求外部作者继承基类；
|
||||
- 修改逻辑是接收一个快照、返回不可变候选修改的普通函数；
|
||||
- `modifier_id`、版本、参数和适用边界不得从函数名、模块路径或闭包内容自动猜测；
|
||||
- 元数据在流水线开始时冻结，运行期间变化视为契约错误；
|
||||
- 函数只能提出修改，仍不能直接应用编辑或返回一整篇新 Markdown；
|
||||
- 相同输入、身份、版本和参数必须产生相同顺序的候选修改；
|
||||
- 修改器不得读取文件、网络、环境变量、当前时间或随机数。
|
||||
|
||||
保留显式元数据是为了让外包组件仍可被审计和复现。函数式编程不等于丢弃组件身份和版本。
|
||||
|
||||
### 4.2 修改权限与数据流
|
||||
|
||||
这里把“修改权”拆成三层，避免把业务判断、字符串执行和文件写回混为一件事：
|
||||
|
||||
1. **使用项目拥有规则决策权。** 项目决定启用哪些 `Modifier`、传入什么参数、按什么顺序运行。修改器的
|
||||
`propose(snapshot)` 只判断“建议改哪里、为什么改、改成什么”，不能原地改变不可变快照，也不能用一整篇
|
||||
新 Markdown 绕过精确编辑契约。
|
||||
2. **`mdpolish` 核心拥有验证和内存执行权。** `Pipeline` 调用修改器，公共执行器检查快照、范围、原文、重复和
|
||||
冲突，然后才通过字符串切片应用编辑。一个修改器本轮的候选修改必须整批通过；任意一项无效时，该批不产生
|
||||
部分结果。
|
||||
3. **使用项目拥有持久化决定权。** `Pipeline` 只返回状态、结果文本和审计记录，不读取或写入文件。调用方检查
|
||||
`success`、`failed` 或 `unstable` 后，自行决定是否以及怎样保存；`mdpolish` 不会自动覆盖输入文件。
|
||||
|
||||
修改器至少通过 `ProposedChange` 和 `TextEdit` 向核心提交以下信息：
|
||||
|
||||
- 候选修改及每条精确编辑所绑定的 `snapshot_sha256`；
|
||||
- 人可读且非空的修改原因 `reason`；
|
||||
- 使用 Python 字符串索引表示的半开范围 `TextSpan(start, end)`；
|
||||
- 该范围当前应有的原文 `expected_text`；
|
||||
- 替换后的文本 `replacement`。
|
||||
|
||||
因此，一次调用的数据流固定为：
|
||||
|
||||
```text
|
||||
项目组装 Modifier 与顺序
|
||||
↓
|
||||
Modifier.propose(snapshot) 提出精确修改
|
||||
↓
|
||||
mdpolish 验证并在内存中原子应用
|
||||
↓
|
||||
Pipeline 返回状态、结果和审计记录
|
||||
↓
|
||||
项目决定是否写回文件
|
||||
```
|
||||
|
||||
项目规则当然可以决定自己的业务语义，也可以提出删除、替换或插入；但只有满足上述契约的候选修改才由核心执行。
|
||||
Python 无法阻止项目函数私下写文件或直接调用字符串替换，这类副作用属于调用方绕过库契约，不属于 `mdpolish`
|
||||
保证、审计或回滚的修改。
|
||||
|
||||
### 4.3 `Pipeline`
|
||||
|
||||
`Pipeline` 改为接收 `Iterable[Modifier]`，不再要求 `Component` 子类。其行为继续保持：
|
||||
|
||||
1. 预检全部修改器身份、版本、参数和重复 ID；
|
||||
2. 按调用方顺序对当前快照调用 `propose`；
|
||||
3. 由公共执行器验证和原子应用当前修改器批次；
|
||||
4. 为下一个修改器建立新快照；
|
||||
5. 全部执行一次后，对最终快照进行只读稳定性复查；
|
||||
6. 返回 `success`、`failed` 或 `unstable`，不自动开始第二轮。
|
||||
|
||||
本轮不自动重排修改器，不自动选择修改器，也不提供全局默认组合。
|
||||
|
||||
### 4.4 外部项目的两种扩展方式
|
||||
|
||||
简单规则优先使用库提供的工厂：
|
||||
|
||||
```python
|
||||
remove_stamp = regex_replace(
|
||||
modifier_id="my_project.remove_stamp",
|
||||
version="1.0.0",
|
||||
pattern=r"...",
|
||||
replacement="",
|
||||
)
|
||||
```
|
||||
|
||||
复杂规则直接提供纯函数，再显式构造 `Modifier`：
|
||||
|
||||
```python
|
||||
def propose_project_changes(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
...
|
||||
|
||||
project_modifier = Modifier(
|
||||
modifier_id="my_project.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="...",
|
||||
propose=propose_project_changes,
|
||||
)
|
||||
```
|
||||
|
||||
库不会自动注册或发现这些对象。
|
||||
|
||||
`Modifier` 是 `Pipeline` 接收的统一值类型；`regex_replace()` 是创建这种值的便捷工厂，不是与 `Modifier` 并列的
|
||||
另一套接口。它返回的也是一个完整 `Modifier`。复杂规则与正则规则进入流水线后，都遵守同一套提议、验证、执行
|
||||
和审计流程。
|
||||
|
||||
## 5. 通用正则修改器
|
||||
|
||||
第一版提供一个安全、窄边界的 `regex_replace()` 工厂：
|
||||
|
||||
- 调用方必须显式给出稳定 `modifier_id` 和版本；
|
||||
- 输入为正则 pattern、固定 replacement、flags 和适用边界说明；
|
||||
- pattern、replacement 和 flags 完整进入修改器参数，运行结果可以复核；
|
||||
- 按 Python `re.finditer()` 的确定顺序定位非重叠匹配；
|
||||
- replacement 使用 Python 正则的固定替换模板展开；
|
||||
- 每个匹配生成绑定当前快照、范围和原文的精确编辑；
|
||||
- 第一版拒绝零长度匹配，避免隐式插入和边界顺序不明确；
|
||||
- 工厂不读 YAML，不维护规则注册表，也不提供默认规则集；
|
||||
- 需要动态 replacement、跨匹配聚合或结构判断时，项目应编写自己的纯函数。
|
||||
|
||||
这个工厂提供正则能力，但不会退回“全局替换后返回整篇文本”的旧实现。所有修改仍经过公共执行器。
|
||||
|
||||
## 6. 通用内置修改器
|
||||
|
||||
第一版只保留以下与具体项目无关、能够用合成样例完整说明的能力：
|
||||
|
||||
### 6.1 映射驱动的跨行片段合并
|
||||
|
||||
- 保留当前“左片段、右片段、结果文本”的算法；
|
||||
- 修改器身份改为领域无关名称；
|
||||
- 映射必须由调用方传入并写入参数；
|
||||
- 库中不保留 ClinDB 的 6 条映射；
|
||||
- 不使用词典、模型或项目默认值猜测结果。
|
||||
|
||||
### 6.2 严格 HTML 表格实体解除一层转义
|
||||
|
||||
- 只处理严格识别的 `<td>` / `<th>` 文本；
|
||||
- 只处理明确支持的双重实体；
|
||||
- 不处理属性、表格外文本或任意 HTML；
|
||||
- 名称和说明不得宣称已经支持通用损坏 HTML 修复。
|
||||
|
||||
### 6.3 严格单行 HTML 表格布局
|
||||
|
||||
- 保留 HTML 标签、属性、行和单元格内容；
|
||||
- 只调整严格单行表格的外层行布局；
|
||||
- 混合换行和不支持的结构保持原样；
|
||||
- 它是保守的通用修改器，不是 HTML→GFM 转换器。
|
||||
|
||||
当前 `_html_table.py` 可以作为这两个修改器的私有词法辅助实现。由于它不识别 Markdown 围栏和完整方言，
|
||||
公开说明必须保留这一边界。以后扩大为通用 HTML 清洗前必须另建设计。
|
||||
|
||||
除上述三类外，当前所有正式业务组件都移出 Python 包。合成示例可以演示怎样用 `regex_replace()` 和自定义函数
|
||||
建立修改器，但不得包含论文原文、ClinDB 名称、固定文档 ID 或项目参数。
|
||||
|
||||
## 7. 从通用库移出的内容
|
||||
|
||||
设计获批后，当前工作树中以下内容不再作为 `mdpolish` 当前实现保留：
|
||||
|
||||
- `paper.word_review_comment`；
|
||||
- `paper.manuscript_line_number`；
|
||||
- `paper.arxiv_submission_stamp`；
|
||||
- `paper.repeated_running_header`；
|
||||
- `paper.reference_spacing`；
|
||||
- ClinDB 固定断词映射与 first-batch 组合；
|
||||
- `scripts/run_clindb_arxiv_experiment.py`；
|
||||
- `scripts/run_clindb_first_batch_experiment.py`；
|
||||
- `experiment.py`、`artifact_store.py`、`reporting.py` 和只服务 artifact 的重放代码；
|
||||
- `reviewer/` Python 服务、React 前端、Node.js 依赖与构建配置；
|
||||
- 对应项目测试、实验 guide、当前机制 explanation、ClinDB/GovDoc reference 和项目 scratch；
|
||||
- README 中的数据目录、实验运行、155 条修改和评审器说明。
|
||||
|
||||
`research-wiki/design/0001` 至 `0007` 继续作为冻结的历史决策保留。`0008` 负责明确它们不再描述当前交付范围，
|
||||
避免通过删除历史记录让旧方向看起来从未发生。
|
||||
|
||||
本轮只从 Git 跟踪的通用库工作树移除上述实现和当前说明。Git 历史仍可恢复这些内容。本地被忽略的 `data/`、
|
||||
`artifacts/`、虚拟环境、构建缓存和真实材料一律不删除、不移动、不读取、不复制。
|
||||
|
||||
## 8. 拟采用的源码结构
|
||||
|
||||
```text
|
||||
pyproject.toml
|
||||
src/mdpolish/
|
||||
├── __init__.py
|
||||
├── models.py
|
||||
├── modifier.py
|
||||
├── edits.py
|
||||
├── pipeline.py
|
||||
├── regex.py
|
||||
├── _text_ranges.py
|
||||
├── _html_table.py
|
||||
└── modifiers/
|
||||
├── __init__.py
|
||||
├── mapped_line_join.py
|
||||
├── html_table_entities.py
|
||||
└── html_table_layout.py
|
||||
tests/
|
||||
├── test_models.py
|
||||
├── test_modifier.py
|
||||
├── test_edits.py
|
||||
├── test_pipeline.py
|
||||
├── test_regex.py
|
||||
├── test_mapped_line_join.py
|
||||
├── test_html_table_entities.py
|
||||
└── test_html_table_layout.py
|
||||
```
|
||||
|
||||
`modifier.py` 只定义函数式修改器的身份和契约；`regex.py` 提供通用工厂；`modifiers/` 只放领域无关实现。
|
||||
业务组件不以示例为由继续进入发布包。
|
||||
|
||||
是否把 `DocumentSnapshot`、`ProposedChange`、`TextEdit` 等低层对象继续放在顶层公共命名空间，由实现时按外部作者
|
||||
能否编写自定义函数决定。本设计要求它们具有受支持的导入路径，但不要求所有对象都堆在 `mdpolish.__init__`。
|
||||
|
||||
## 9. 兼容与版本
|
||||
|
||||
这是明确的破坏性重构：
|
||||
|
||||
- 删除 `Component` 抽象基类；
|
||||
- `Pipeline` 的元素类型改变；
|
||||
- 删除项目组件和实验接口；
|
||||
- wheel 不再包含项目工作流；
|
||||
- README 的使用方式改变。
|
||||
|
||||
当前仓库仍处于 `0.x` 研究阶段，没有已批准的外部兼容承诺。实现后包版本从 `0.1.0` 更新为 `0.2.0`，不建立
|
||||
兼容旧 `Component` 子类的适配层。保留适配层会让新的公共边界继续携带旧项目架构，本轮不采用。
|
||||
|
||||
## 10. 测试与验收
|
||||
|
||||
### 10.1 核心不变量
|
||||
|
||||
现有以下测试语义必须迁移并继续通过：
|
||||
|
||||
- 空文本、中文、组合字符和不同换行；
|
||||
- 插入、删除、替换、多编辑候选和组件批次原子性；
|
||||
- 过期快照、范围越界、原文不符、重复与冲突编辑明确失败；
|
||||
- 后一个修改器读取新快照；
|
||||
- 最终复查发现连锁修改时返回 `unstable`；
|
||||
- 异常和元数据变化返回安全错误；
|
||||
- 相同输入、修改器和参数得到相同结果；
|
||||
- 成功结果再次运行零修改；
|
||||
- 输入字符串不被原地改变。
|
||||
|
||||
### 10.2 函数式 API
|
||||
|
||||
至少覆盖：
|
||||
|
||||
- 普通函数不继承任何库基类即可成为 `Modifier`；
|
||||
- 无效 ID、版本、参数、适用边界和不可调用对象明确失败；
|
||||
- 两个相同 ID 的修改器仍在预检失败；
|
||||
- 函数异常不会被吞掉，也不会把部分文本当成功输出；
|
||||
- 无效候选修改导致当前修改器整批失败，不应用其中任何一项；
|
||||
- 流水线记录实际修改器身份、版本、参数和顺序；
|
||||
- 流水线运行只产生内存结果，不读取或写回调用方文件。
|
||||
|
||||
### 10.3 正则工厂
|
||||
|
||||
至少覆盖：
|
||||
|
||||
- 普通替换、捕获组展开、多命中和零命中；
|
||||
- flags 和参数记录；
|
||||
- 零长度模式拒绝；
|
||||
- 原文、范围、理由和哈希审计正确；
|
||||
- 第二次运行稳定；
|
||||
- 没有 YAML、全局注册表或默认启用行为。
|
||||
|
||||
### 10.4 通用修改器
|
||||
|
||||
- 只使用小型虚构文本；
|
||||
- 映射合并没有内置项目词表；
|
||||
- HTML 修改器保留标签、属性、表格外文本和不支持结构；
|
||||
- 正向、反向、换行、无末尾换行、确定性和幂等性均有覆盖；
|
||||
- 不读取 `data/`、`artifacts/` 或仓库外数据。
|
||||
|
||||
### 10.5 仓库边界
|
||||
|
||||
实现完成后必须确认：
|
||||
|
||||
- wheel 只包含通用 Python 库；
|
||||
- Python 源码、测试、README、explanation 和 guide 不包含 ClinDB 文档 ID、固定映射和项目流水线；
|
||||
- 除冻结的 `research-wiki/design/0001` 至 `0007` 外，当前文档不再把项目实验描述为库能力；
|
||||
- `reviewer/`、项目脚本和 Node.js 配置不再属于当前工作树；
|
||||
- `.gitignore` 继续保护本地 `data/` 和 `artifacts/`；
|
||||
- `git status` 不出现被忽略的真实材料；
|
||||
- `AGENTS.md` 与 `CLAUDE.md` 除标题外正文一致。
|
||||
|
||||
基础检查仍使用 Ruff、mypy 和 pytest。实现后根据实际文件更新根 README 中的唯一检查命令，并实际运行后才报告通过。
|
||||
|
||||
## 11. 风险与代价
|
||||
|
||||
- **历史实验入口消失：** 当前分支不再直接运行 ClinDB；旧代码仍在 Git 历史中，未来迁移需在目标项目重新评审。
|
||||
- **评审能力不再随库提供：** 这符合最小公共库目标；多个项目产生共同需求后再设计独立扩展。
|
||||
- **函数不能被强制纯净：** Python 无法仅靠类型禁止文件和网络副作用；内置修改器通过实现与测试保证，外部修改器由调用方负责。
|
||||
- **正则工厂可能被滥用：** 它只保证修改执行安全，不保证项目正则语义正确；项目仍需为自己的模式、反例和版本负责。
|
||||
- **HTML 修改器的“通用”范围有限：** 严格失败关闭，宁可漏处理未知结构，也不扩大成未经验证的 HTML 修复器。
|
||||
- **没有文件产品入口：** 第一版调用方需要自己读取 Markdown 和处理结果；这正是新的库边界，不是遗漏。
|
||||
- **破坏现有导入：** `0.2.0` 明确表示新边界，不维护尚未承诺的旧扩展接口。
|
||||
|
||||
## 12. 批准后的实施顺序
|
||||
|
||||
用户明确批准本文后，按以下顺序实施：
|
||||
|
||||
1. 新增函数式 `Modifier`，迁移核心测试，使 `Pipeline` 不再依赖抽象基类；
|
||||
2. 实现 `regex_replace()` 和通用修改器，全部使用合成测试；
|
||||
3. 更新公共导出、包描述和版本；
|
||||
4. 移除项目组件、脚本、实验层、评审器及其测试；
|
||||
5. 清理当前 README、explanation、reference、guide 和 scratch，只保留通用库当前事实与冻结历史 design；
|
||||
6. 核对 `.gitignore`，但不触碰任何被忽略的真实输入和产物；
|
||||
7. 运行 README 中实际保留的全部检查，检查 wheel 内容、Git diff、文档镜像和工作区状态。
|
||||
|
||||
本文批准不授权提交、推送、创建 PR、发布、修改其他仓库，或删除、移动、复制本地真实材料。
|
||||
@@ -0,0 +1,762 @@
|
||||
# 0009：通用映射驱动的跨行片段合并
|
||||
|
||||
## 状态
|
||||
|
||||
已批准（2026-08-26）。
|
||||
|
||||
`supersedes: 0008`（范围有限）：本文替代 `0008` 第 6.1 节中“保留当前左片段、右片段、结果文本算法”
|
||||
的窄接口，并为调用方显式启用的词典规则放宽“运行时只依赖标准库”这一点。核心安装仍然零依赖；精确和正则规则
|
||||
仍然只处理内存字符串。词典规则只允许在构造修改器时读取已安装 optional package 自带的版本化词典资源，
|
||||
不读取调用方文档、任意路径或网络。`propose()` 仍然是只读快照的纯函数。
|
||||
|
||||
`0008` 确定的函数式 `Modifier`、无默认启用行为、无项目固定词表、无网络和无模型推断等边界继续有效。
|
||||
|
||||
本文批准后才能实施。批准本文不等于批准提交、推送、创建 PR、发布、修改其他仓库或处理真实材料。
|
||||
|
||||
## 1. 问题与可观察现象
|
||||
|
||||
当前 `mapped_line_join()` 只接受 `(left, right, replacement)` 三元组。它能处理调用方已经确认的少量精确断词，
|
||||
但遇到下列情况时，调用方只能复制整个修改器或在库外另写一套边界扫描逻辑：
|
||||
|
||||
- 同一类断词需要用命名正则捕获不同词干；
|
||||
- 只需要匹配下一行开头，左侧只要求位于当前行尾；
|
||||
- OCR 把一个词连续拆到三行以上，需要在一次提议中完成链式合并；
|
||||
- 同一位置有多条规则，需要显式选择第一条、最高优先级或报错；
|
||||
- 规则只允许用于段落、标题、列表项或引用，不能进入表格和代码块；
|
||||
- 合并后需要保留换行、删除换行、换成空格或形成段落分隔；
|
||||
- 文本和规则可能使用不同 Unicode 规范形式，或调用方明确需要忽略大小写。
|
||||
|
||||
更根本的问题是：如果每个 `example`、`international`、`database` 都要先手写一条映射，调用方实际上要自己维护
|
||||
一份英语词典。成熟的通用方案应能从行尾和行首自动生成候选词，查询明确配置的词典或词频资源；精确映射只负责
|
||||
项目术语、误判否决和其他例外，不应成为处理普通英文断词的唯一入口。
|
||||
|
||||
当前实现还会在整个物理行内容上匹配。它不知道 Markdown 围栏、缩进代码、pipe table、列表和引用前缀，
|
||||
因此无法可靠表达“只在某类块中生效”。
|
||||
|
||||
这些都属于清洗语义、规则格式和冲突策略变化，不能作为 `0008` 已批准实现的机械扩展。
|
||||
|
||||
## 2. 调研结论及其边界
|
||||
|
||||
本次调研只用于确定库边界，不把外部工具行为直接变成 `mdpolish` 的默认规则。
|
||||
|
||||
### 2.1 Pandoc：软换行语义与源码重排是两件事
|
||||
|
||||
Pandoc 默认把段落内普通换行当作空格。`hard_line_breaks` 会把段落内每个换行解释为硬换行，
|
||||
`ignore_line_breaks` 和 `east_asian_line_breaks` 则提供另外两种读取语义。输出侧的 `--wrap=auto|none|preserve`
|
||||
决定生成源码怎样折行，不负责判断 OCR 断词。
|
||||
|
||||
当前 Pandoc 手册没有 `reflowed_text` 扩展。与“reflowed text”最接近的当前公共能力是输出侧 `--wrap`，
|
||||
不能把这个非现行名称设计成库兼容目标。
|
||||
|
||||
来源：
|
||||
|
||||
- <https://pandoc.org/MANUAL.html#paragraphs>
|
||||
- <https://pandoc.org/MANUAL.html#extension-hard_line_breaks>
|
||||
- <https://pandoc.org/MANUAL.html#option--wrap>
|
||||
|
||||
因此，本修改器不能把所有物理换行统一解释成一种语义。每条规则必须明确怎样处理命中的边界，代码块和表格等结构
|
||||
必须先排除。
|
||||
|
||||
### 2.2 OCR 去连字符：成熟方案仍然需要证据来源和取舍
|
||||
|
||||
常见 OCR 后处理会组合以下信号：
|
||||
|
||||
| 信号 | 能解决的问题 | 不能单独保证的事情 |
|
||||
| --- | --- | --- |
|
||||
| 词典查表 | 判断拼接词是否为已知词 | 专名、新词、领域词和真正带连字符的词 |
|
||||
| 词缀与形态分析 | 识别屈折、派生和复合词 | 依赖语言及词典质量 |
|
||||
| 词频或统计语言模型 | 在“连写、保留连字符、加空格”之间排序 | 结果依赖训练语料和领域分布 |
|
||||
| OCR 置信度与版面元数据 | 利用识别器对字符、词或断词的显式证据 | 普通 Markdown 通常已经丢失这些信息 |
|
||||
|
||||
ALTO 可以用 `SUBS_TYPE=HypPart1/HypPart2` 和 `SUBS_CONTENT` 表达断词及完整词，并提供词置信度；hOCR 定义了
|
||||
`x_wconf`。这说明如果上游仍持有结构化 OCR 证据，优先在上游使用它比从 Markdown 猜测更可靠。
|
||||
|
||||
来源：
|
||||
|
||||
- <https://www.loc.gov/standards/alto/>
|
||||
- <https://kba.github.io/hocr-spec/1.2/>
|
||||
- <https://aclanthology.org/L18-1113/>
|
||||
- <https://arxiv.org/abs/1204.0188>
|
||||
|
||||
本仓库当前只有 Markdown 字符串，没有 OCR 置信度、坐标或上游候选。因此本轮只实现调用方显式配置的本地词典判断
|
||||
和统计打分，不实现依赖版面证据的判断或模型推断。词典结果是候选证据，不绕过歧义策略和精确覆盖规则。
|
||||
|
||||
### 2.3 Python 词典与断词库可以提供证据，但必须显式选择
|
||||
|
||||
| 工具 | 主要能力 | 本轮不直接集成的原因 |
|
||||
| --- | --- | --- |
|
||||
| `wordninja` | 按 unigram 概率拆分粘连英文词 | 目标是拆词，不是恢复 Markdown 跨行结构；默认模型有语言和语料偏置 |
|
||||
| `pyspellchecker` | 基于词频和编辑距离给出拼写候选 | 候选不是唯一正确修改，默认大小写和词典行为也需项目决定 |
|
||||
| `PyEnchant` | 通过 Enchant provider 检查和建议拼写 | 依赖系统 provider 与外部词典，安装结果不完全由 Python 包锁定 |
|
||||
| `wordfreq` | 查询多语言词频 | 能排序候选，但不能确定是否应保留自然连字符 |
|
||||
| `PyHyphen` / `Pyphen` | 使用 TeX/Hunspell 断词模式找合法断点 | 正向排版断词不等于逆向 OCR 去断词 |
|
||||
| Hunspell | 拼写检查、词缀、复合词和形态分析 | 需要语言词典；不同词典的形态字段和能力不同 |
|
||||
|
||||
来源：
|
||||
|
||||
- <https://github.com/keredson/wordninja>
|
||||
- <https://pyspellchecker.readthedocs.io/>
|
||||
- <https://pyenchant.github.io/pyenchant/>
|
||||
- <https://rspeer.github.io/wordfreq/>
|
||||
- <https://pyhyphen.readthedocs.io/>
|
||||
- <https://pyphen.org/>
|
||||
- <https://github.com/hunspell/hunspell>
|
||||
|
||||
调研后的采用边界如下：
|
||||
|
||||
- `pyspellchecker` 具有随包分发的多语言词频词典，能够同时做精确 `known()` 查询和频率比较，适合作为第一种较轻的
|
||||
自动词典 backend；不使用它的编辑距离纠错，因为本任务只在几个明确候选之间选择，不改写 OCR 字符；
|
||||
- `wordfreq` 提供可比较的 Zipf 频率和更广的多语言数据，适合作为可选的频率 backend；它的数据主要截至 2021 年，
|
||||
且官方明确说明多 token 查询会高估罕见组合，因此不能把任意短语分数当作可靠词典命中；
|
||||
- `Pyphen` 能列出一个词的合法断词位置，适合作为“这个行尾位置是否可能是排版断词”的附加门槛，但它不能单独证明
|
||||
拼接词真实存在；
|
||||
- `PyEnchant` / Hunspell 能处理词缀、复合词和自然连字符，但依赖系统 provider 和外部词典，环境可复现性较差；
|
||||
本轮记录边界但不实现该 backend；
|
||||
- `wordninja` 的目标是把粘连字符串拆成多个词，方向与本任务相反，本轮不集成。
|
||||
|
||||
这些 backend 必须由 `LexicalLineJoinRule` 显式选择。缺少请求的 extra 时立即报错，不退回另一个词典，也不改成只靠
|
||||
正则猜测。
|
||||
|
||||
### 2.4 Markdown lint 工具负责语法和风格，不负责 OCR 语义修复
|
||||
|
||||
`markdownlint` 的规则集中有行长、尾随空格、空行、标题和列表等结构/风格检查；`remark-lint` 的规则检查 mdast，
|
||||
并允许项目自行编写插件。两者都没有“判断行尾连字符是不是排版断词并自动拼回”的内置规则。
|
||||
|
||||
这不是遗漏：lint 工具可以判断 Markdown 是否符合某种书写约定，却没有词典、原始页面或 OCR 置信度来唯一决定正文
|
||||
应该怎样改。OCR 语义修复应由上游转换器、专用后处理器或项目明确规则负责。
|
||||
|
||||
来源：
|
||||
|
||||
- <https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md>
|
||||
- <https://github.com/remarkjs/remark-lint>
|
||||
|
||||
## 3. 目标与非目标
|
||||
|
||||
### 3.1 目标
|
||||
|
||||
- 保留旧三元组精确映射的调用方式；
|
||||
- 增加不可变、带类型的精确规则和正则规则；
|
||||
- 增加自动词典规则：从行尾、separator 和行首生成候选，查询显式配置的本地词典或词频资源；
|
||||
- 让一条语言级规则可以复用于任意文档，不要求调用方逐词枚举普通英文；
|
||||
- 所有规则仍由调用方显式传入，不内置词表或模式；
|
||||
- 在一次 `propose()` 中处理独立命中和跨多行链式命中；
|
||||
- 显式配置块范围、边界替换、冲突策略、大小写和 Unicode 规范化；
|
||||
- 对代码块和表格失败关闭；
|
||||
- 最终只产生绑定当前快照的非重叠精确 `TextEdit`；
|
||||
- 规则、选项和顺序完整记录在 `Modifier.parameters` 中。
|
||||
|
||||
### 3.2 非目标
|
||||
|
||||
- 不实现自动语言识别、在线词典发现或模型调用；语言、backend、候选形式和阈值必须显式配置；
|
||||
- 不读取 ALTO、hOCR、PDF、DOCX、图片或外部文件；
|
||||
- 核心安装不引入 Markdown parser 或 OCR 库；词典能力只通过明确安装的 optional extra 启用；
|
||||
- 不声称这个词法块分类器覆盖全部 CommonMark、GFM、Pandoc Markdown 或任意嵌套 HTML；
|
||||
- 不自动选择规则、重排规则或循环运行整个 `Pipeline`；
|
||||
- 不修改 `_text_ranges.py`；
|
||||
- 不提供配置文件、CLI、默认映射或默认流水线。
|
||||
|
||||
## 4. 公共规则模型
|
||||
|
||||
新值类型均放在 `mapped_line_join.py`，使用 `@dataclass(frozen=True, slots=True)`。为避免扩大原始任务的源码范围，
|
||||
它们先通过 `mdpolish.modifiers.mapped_line_join` 模块路径提供；本轮不要求修改 `modifiers/__init__.py`。
|
||||
|
||||
### 4.1 枚举
|
||||
|
||||
```python
|
||||
class LineJoinBlock(StrEnum):
|
||||
PARAGRAPH = "paragraph"
|
||||
HEADING = "heading"
|
||||
LIST_ITEM = "list_item"
|
||||
BLOCK_QUOTE = "block_quote"
|
||||
|
||||
|
||||
class LineBreakPolicy(StrEnum):
|
||||
PRESERVE = "preserve"
|
||||
DELETE = "delete"
|
||||
SPACE = "space"
|
||||
PARAGRAPH = "paragraph"
|
||||
|
||||
|
||||
class LineJoinConflictPolicy(StrEnum):
|
||||
PRIORITY = "priority"
|
||||
FIRST = "first"
|
||||
ERROR = "error"
|
||||
|
||||
|
||||
class UnicodeNormalization(StrEnum):
|
||||
NFC = "NFC"
|
||||
NFD = "NFD"
|
||||
NFKC = "NFKC"
|
||||
NFKD = "NFKD"
|
||||
|
||||
|
||||
class LexiconBackend(StrEnum):
|
||||
SPELLCHECKER = "pyspellchecker"
|
||||
WORDFREQ = "wordfreq"
|
||||
|
||||
|
||||
class LexicalCandidateForm(StrEnum):
|
||||
JOINED = "joined"
|
||||
HYPHENATED = "hyphenated"
|
||||
SPACED = "spaced"
|
||||
|
||||
|
||||
class LexicalAmbiguityPolicy(StrEnum):
|
||||
KEEP = "keep"
|
||||
ERROR = "error"
|
||||
```
|
||||
|
||||
默认可匹配块为四种 `LineJoinBlock` 的不可变集合。代码块和表格不是可选块类型，因为它们始终排除，调用方不能通过
|
||||
普通规则意外打开。
|
||||
|
||||
### 4.2 精确规则
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ExactLineJoinRule:
|
||||
rule_id: str
|
||||
left: str
|
||||
right: str
|
||||
replacement: str
|
||||
separator: str = ""
|
||||
blocks: frozenset[LineJoinBlock] = ALL_LINE_JOIN_BLOCKS
|
||||
line_break: LineBreakPolicy = LineBreakPolicy.DELETE
|
||||
priority: int = 0
|
||||
case_sensitive: bool = True
|
||||
normalization: UnicodeNormalization | None = None
|
||||
require_word_boundaries: bool = True
|
||||
```
|
||||
|
||||
`separator` 是紧跟在 `left` 后、位于物理换行前的显式文本。它可以是 `"-"`，也可以是空字符串：
|
||||
|
||||
```python
|
||||
ExactLineJoinRule(
|
||||
rule_id="example.dehyphenate",
|
||||
left="exam",
|
||||
separator="-",
|
||||
right="ple",
|
||||
replacement="example",
|
||||
)
|
||||
|
||||
ExactLineJoinRule(
|
||||
rule_id="example.join_split_word",
|
||||
left="exam",
|
||||
separator="",
|
||||
right="ple",
|
||||
replacement="example",
|
||||
)
|
||||
```
|
||||
|
||||
这样不需要靠“`left` 是否碰巧以连字符结尾”推断规则类型。`replacement` 是完整的替换片段；库不会自动拼接
|
||||
`left`、`separator` 或 `right`。
|
||||
|
||||
自动词典不可避免会遇到项目专名和自然连字符例外。为让映射表能明确否决而不是只能增加修改，再提供同样不可变的
|
||||
精确保护规则：
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class KeepLineJoinRule:
|
||||
rule_id: str
|
||||
left: str
|
||||
right: str
|
||||
separator: str = ""
|
||||
blocks: frozenset[LineJoinBlock] = ALL_LINE_JOIN_BLOCKS
|
||||
priority: int = 0
|
||||
case_sensitive: bool = True
|
||||
normalization: UnicodeNormalization | None = None
|
||||
require_word_boundaries: bool = True
|
||||
```
|
||||
|
||||
它命中并赢得冲突选择后保留原文，不产生无效 `TextEdit`，同时阻止低优先级词典规则处理该边界。保护规则会进入
|
||||
`Modifier.parameters`；由于没有实际修改，不伪造 `Change`。
|
||||
|
||||
### 4.3 左右正则规则
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RegexLineJoinRule:
|
||||
rule_id: str
|
||||
left_pattern: str
|
||||
right_pattern: str
|
||||
replacement: str
|
||||
separator: str = ""
|
||||
blocks: frozenset[LineJoinBlock] = ALL_LINE_JOIN_BLOCKS
|
||||
line_break: LineBreakPolicy = LineBreakPolicy.DELETE
|
||||
priority: int = 0
|
||||
case_sensitive: bool = True
|
||||
normalization: UnicodeNormalization | None = None
|
||||
```
|
||||
|
||||
- `left_pattern` 必须匹配左侧逻辑内容的非空后缀；
|
||||
- `right_pattern` 必须匹配右侧逻辑内容的非空前缀；
|
||||
- 捕获组必须命名，左右两侧的组名不能重复；
|
||||
- `replacement` 只支持 `\g<name>` 命名反向引用，不支持数字组和动态回调；
|
||||
- 非捕获组、lookaround 和普通正则语法可以使用，但最终左右匹配本身必须消耗字符；
|
||||
- 正则按 Python `re` 语义编译，不启用 `MULTILINE` 或 `DOTALL`，因为物理行边界由修改器负责。
|
||||
|
||||
示例：
|
||||
|
||||
```python
|
||||
RegexLineJoinRule(
|
||||
rule_id="example.named_regex",
|
||||
left_pattern=r"(?P<stem>[A-Za-z]+)",
|
||||
separator="-",
|
||||
right_pattern=r"(?P<suffix>[a-z]+)",
|
||||
replacement=r"\g<stem>\g<suffix>",
|
||||
)
|
||||
```
|
||||
|
||||
### 4.4 只指定右侧的行尾正则规则
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LineEndRegexRule:
|
||||
rule_id: str
|
||||
right_pattern: str
|
||||
separator: str = ""
|
||||
blocks: frozenset[LineJoinBlock] = ALL_LINE_JOIN_BLOCKS
|
||||
line_break: LineBreakPolicy = LineBreakPolicy.DELETE
|
||||
priority: int = 0
|
||||
case_sensitive: bool = True
|
||||
normalization: UnicodeNormalization | None = None
|
||||
```
|
||||
|
||||
这类规则把左侧定义为逻辑行尾的零宽锚点。`right_pattern` 必须匹配右侧逻辑内容的非空前缀，但只作为边界条件；
|
||||
右侧匹配文本不进入编辑范围，也不会被重新输出。实际编辑只覆盖可选 `separator`、物理行尾、允许的空行和右侧重复
|
||||
块前缀：
|
||||
|
||||
```python
|
||||
LineEndRegexRule(
|
||||
rule_id="example.line_end_and_right",
|
||||
right_pattern=r"(?P<right>[a-z])",
|
||||
line_break=LineBreakPolicy.SPACE,
|
||||
)
|
||||
```
|
||||
|
||||
上例把 `foo\nbar` 变成 `foo bar`，而不是把空格放到 `b` 之后。规则不会猜测左侧单词，只明确表示
|
||||
“当前逻辑行尾 + 下一逻辑行首命中该正则时怎样改”。它仍然是映射表规则，不是对所有小写行首自动生效的隐藏默认值。
|
||||
|
||||
`LineEndRegexRule` 不允许 `PRESERVE`，因为它不会改变 separator、边界或右侧文本，只会产生无效编辑。
|
||||
|
||||
### 4.5 自动词典规则
|
||||
|
||||
普通英文断词不要求逐词映射。调用方配置一条语言级规则，修改器从每个合格边界提取左右单词片段并自动查询本地词典：
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LexicalLineJoinRule:
|
||||
rule_id: str
|
||||
left_pattern: str
|
||||
right_pattern: str
|
||||
separator: str
|
||||
backend: LexiconBackend
|
||||
language: str
|
||||
candidate_forms: tuple[LexicalCandidateForm, ...]
|
||||
minimum_score: float
|
||||
minimum_score_margin: float = 0.0
|
||||
hyphenation_language: str | None = None
|
||||
ambiguity: LexicalAmbiguityPolicy = LexicalAmbiguityPolicy.KEEP
|
||||
blocks: frozenset[LineJoinBlock] = ALL_LINE_JOIN_BLOCKS
|
||||
priority: int = 0
|
||||
case_sensitive: bool = True
|
||||
normalization: UnicodeNormalization | None = None
|
||||
```
|
||||
|
||||
示例配置一次后可以复用于任意英文文档：
|
||||
|
||||
```python
|
||||
LexicalLineJoinRule(
|
||||
rule_id="english.dictionary_dehyphenation",
|
||||
left_pattern=r"[A-Za-z]{2,}",
|
||||
right_pattern=r"[a-z]{2,}",
|
||||
separator="-",
|
||||
backend=LexiconBackend.SPELLCHECKER,
|
||||
language="en",
|
||||
candidate_forms=(
|
||||
LexicalCandidateForm.JOINED,
|
||||
LexicalCandidateForm.HYPHENATED,
|
||||
),
|
||||
minimum_score=1.0,
|
||||
minimum_score_margin=0.5,
|
||||
hyphenation_language="en_US",
|
||||
)
|
||||
```
|
||||
|
||||
这条规则会自动处理 `exam-\nple`、`inter-\nnational` 等候选，不需要为 `example`、`international` 分别写映射。
|
||||
规则表在这里表达的是“什么文本形态、用哪个词典、比较哪些候选、阈值多保守”，不是穷举英语单词。
|
||||
|
||||
左右 pattern 分别锚定逻辑行尾和行首，整个非空匹配就是候选片段。`separator` 本轮只允许 `"-"` 或 `""`。
|
||||
修改器生成调用方列出的候选：
|
||||
|
||||
| 候选 | 输出 |
|
||||
| --- | --- |
|
||||
| `JOINED` | `left + right` |
|
||||
| `HYPHENATED` | `left + "-" + right` |
|
||||
| `SPACED` | `left + " " + right` |
|
||||
|
||||
规则必须包含 `JOINED`，并包含与源边界对应的竞争形式：`separator="-"` 时包含 `HYPHENATED`，空 separator 时包含
|
||||
`SPACED`。这样“拼接词在词典中”不会自动压过同样合理的自然连字符或两个独立单词。
|
||||
|
||||
backend 行为：
|
||||
|
||||
- `SPELLCHECKER` 用 `SpellChecker.known()` 判断单 token 是否存在，并把 `word_usage_frequency()` 换算到每十亿词的
|
||||
对数尺度。`SPACED` 或需要拆成多个单 token 验证的 `HYPHENATED` 只在各组成 token 分别已知时成立，准入分数取
|
||||
各 token 的最低值；这个值不是短语频率，也不能与单 token 分数按 `minimum_score_margin` 排序；
|
||||
- `WORDFREQ` 使用 `zipf_frequency(candidate, language)`；它可以评价多 token 字符串，但调用方必须考虑官方说明的
|
||||
罕见组合高估问题，并通过 `minimum_score_margin`、精确否决规则和合成反例收紧；
|
||||
- `hyphenation_language` 非空时，再用 Pyphen 检查 `len(left)` 是否在 `positions(left + right)` 中。这个门槛只作用于
|
||||
`JOINED` 候选，不替代词典存在性和候选比较。
|
||||
|
||||
`minimum_score` 和 `minimum_score_margin` 是 backend 内部的阈值，不是跨 backend 的统一物理量。对 `WORDFREQ`，二者
|
||||
都是 Zipf 尺度；对 `SPELLCHECKER`，`minimum_score` 可以过滤单 token 或各组成 token 中明显低频的候选，但
|
||||
`minimum_score_margin` 只适用于两个都有可比单 token 分数的候选。调用方不能把同一个数值配置在两个 backend 间
|
||||
直接搬用并期待相同含义。
|
||||
|
||||
选择流程固定为：
|
||||
|
||||
1. 为每个显式候选查询 backend，得到“是否存在”、准入分数和可选的可比分数；
|
||||
2. 丢弃不存在、准入分数低于 `minimum_score` 或未通过可选 Pyphen 门槛的候选；
|
||||
3. 没有候选时保留原文；
|
||||
4. 只有一个候选时选择它；
|
||||
5. 多个候选且所有候选都有同一尺度的可比分数时，只有存在唯一最高分，且它比第二名至少高
|
||||
`minimum_score_margin` 才选择；
|
||||
6. 存在不可排序候选、仍然并列或差距不足时，按 `ambiguity` 处理：`KEEP` 保留原文，`ERROR` 抛出包含
|
||||
`rule_id` 和位置但不包含正文的错误；
|
||||
7. 选中候选后，把左右片段和物理边界合成一个精确替换；若选中的是自然连字符或空格形式，也会删除物理换行，
|
||||
但保留 `-` 或一个空格。
|
||||
|
||||
因此，`SPELLCHECKER + SPACED` 的两个单 token 最低频率只用于准入，不冒充短语频率。它与 `JOINED` 同时存活时属于
|
||||
不可排序歧义，不能靠 margin 自动裁决。`SPELLCHECKER + HYPHENATED` 在 backend 只能按组成 token 验证时遵循同一规则。
|
||||
|
||||
因此，“拼起来查词典，命中就合并”是自动规则的基础，但不是唯一判断。至少还要让源形式参与竞争，并对多候选歧义
|
||||
失败关闭。项目专名、缩写和已知例外仍可用更高优先级的精确/正则规则覆盖。
|
||||
|
||||
### 4.6 旧三元组兼容
|
||||
|
||||
以下调用继续有效：
|
||||
|
||||
```python
|
||||
mapped_line_join((("exam-", "ple", "example"),))
|
||||
```
|
||||
|
||||
旧三元组等价于一条精确规则：`left` 保持原值、`separator=""`、删除换行、区分大小写、不做 Unicode 规范化、
|
||||
启用词边界并允许四种文本块。它不会被改写成词典或正则规则。
|
||||
|
||||
旧三元组没有 `rule_id`。工厂按声明位置为审计和冲突消息生成 `legacy.0000`、`legacy.0001` 等内部 ID；
|
||||
因此旧映射的声明顺序也会进入参数记录。这里保留的是调用语法和精确映射能力，不承诺继续在代码块或表格中修改，
|
||||
因为本文明确把这两类区域改为始终排除。
|
||||
|
||||
旧类型别名保留：
|
||||
|
||||
```python
|
||||
LineJoinMapping = tuple[str, str, str]
|
||||
LineJoinRule = ExactLineJoinRule | KeepLineJoinRule | RegexLineJoinRule | LineEndRegexRule | LexicalLineJoinRule
|
||||
```
|
||||
|
||||
工厂签名为：
|
||||
|
||||
```python
|
||||
def mapped_line_join(
|
||||
mappings: Iterable[LineJoinMapping | LineJoinRule],
|
||||
*,
|
||||
conflict_policy: LineJoinConflictPolicy = LineJoinConflictPolicy.PRIORITY,
|
||||
max_intervening_blank_lines: int = 1,
|
||||
) -> Modifier:
|
||||
...
|
||||
```
|
||||
|
||||
保留 `max_intervening_blank_lines=1` 是为了兼容当前“可以隔一个同风格空行”的已实现行为。调用方可以设为 `0`，
|
||||
禁止跨空行。这里只接受类型严格为 `int` 的 `0` 或 `1`，`bool`、负数和 `2` 以上都拒绝。两个及以上连续空行更像
|
||||
结构边界；在没有 OCR provenance、版面置信度或真实样本证据时，本轮不允许用一个任意整数把搜索扩大到更远段落。
|
||||
如果后续证据表明某个上游稳定地产生多空行误断，应以有上限的新设计明确扩大范围，而不是静默放宽本参数。
|
||||
|
||||
## 5. 匹配与替换语义
|
||||
|
||||
### 5.1 逻辑内容和锚点
|
||||
|
||||
修改器仍使用 `_text_ranges.py` 的 `PhysicalLine` 和 `physical_lines()` 获取准确范围，不改变底层工具。
|
||||
|
||||
目标文件内新增保守的块扫描，把每条物理行分成：
|
||||
|
||||
- Markdown 容器/块前缀，例如 `> `、列表 marker 后的缩进或 `# `；
|
||||
- 可匹配逻辑内容；
|
||||
- 精确物理行尾。
|
||||
|
||||
左规则只在逻辑内容末尾匹配，右规则只在逻辑内容开头匹配。删除或替换物理边界时，右行为了表达同一块而重复的
|
||||
引用前缀或 continuation 缩进一并按第 7 节处理，不会残留在拼接词中。
|
||||
|
||||
### 5.2 精确词边界
|
||||
|
||||
`require_word_boundaries=True` 时：
|
||||
|
||||
- 左匹配前一个字符不能是 Unicode 字母、数字或下划线；
|
||||
- 右匹配后一个字符不能是 Unicode 字母、数字或下划线。
|
||||
|
||||
这替代当前只检查 ASCII 字母的窄规则。调用方确实要匹配词内片段时必须显式设为 `False`。正则规则不另加自动词边界，
|
||||
调用方用正则本身表达边界。
|
||||
|
||||
### 5.3 空规则集和无命中
|
||||
|
||||
空映射仍创建合法的 no-op `Modifier`。没有规则命中时返回空提议，不修改换行、Unicode 或任何其他内容。
|
||||
|
||||
## 6. Markdown 块范围
|
||||
|
||||
本轮不引入 parser。目标文件内使用失败关闭的词法扫描，只为跨行候选提供最小结构边界。
|
||||
|
||||
扫描器内部必须有独立的 `UNKNOWN` 分类；它不是公共 `LineJoinBlock`，任何规则都不能选择它。`PARAGRAPH` 必须由
|
||||
“普通文本行、容器前缀一致且两侧都没有未识别块 marker”正向识别，不能作为“不像其他块”的兜底。`EXCLUDED`
|
||||
用于已确认的代码和表格范围，`UNKNOWN` 用于 marker 损坏、容器深度不一致、缩进关系不足或方言结构无法确认的范围；
|
||||
两者都不产生候选。
|
||||
|
||||
### 6.1 始终排除
|
||||
|
||||
- backtick 或 tilde fenced code block，包括 fence 行；
|
||||
- 保守识别的四空格或 tab 缩进代码；
|
||||
- 有 delimiter row 的 GFM pipe table 整个连续区域；
|
||||
- 从 `<table` 开始到配对 `</table>` 的 raw HTML table 区域；不完整 opening 从该处排除到文档末尾。
|
||||
|
||||
围栏识别支持至多三个前导空格、相同 fence 字符和不短于 opening 的 closing。它不解释 fence 内语言。
|
||||
|
||||
### 6.2 可选块
|
||||
|
||||
| `LineJoinBlock` | 词法范围 |
|
||||
| --- | --- |
|
||||
| `PARAGRAPH` | 不属于其他支持块的连续普通文本行 |
|
||||
| `HEADING` | ATX heading 的内容及其紧邻、没有新块 marker 的 OCR continuation；Setext underline 本身不参与匹配 |
|
||||
| `LIST_ITEM` | list marker 行与达到该 item 内容列的 continuation；遇到下一个同级 marker 或块边界结束 |
|
||||
| `BLOCK_QUOTE` | 具有相同 quote depth 的普通引用内容；引用内的 heading/list 使用更具体的块类型 |
|
||||
|
||||
块类型取最内层可识别类型。例如引用中的列表项是 `LIST_ITEM`，引用中的普通段落是 `BLOCK_QUOTE`。
|
||||
|
||||
Markdown 允许 lazy continuation，标题后也允许紧接普通段落；仅靠文本无法总是区分这些情况。因此：
|
||||
|
||||
- 规则映射仍是决定修改正确性的主要证据；
|
||||
- 块分类只缩小规则适用范围，不证明词语一定应该合并；
|
||||
- 无法确认同一块、容器深度不一致或前缀损坏时归入 `UNKNOWN`，不回退成 `PARAGRAPH`，也不提出修改；
|
||||
- 合法且前缀对称的多层引用或嵌套列表可以正向识别；深度、item content column 或 continuation 缩进不一致时
|
||||
失败关闭。`> > >` 和多级有序列表本身不自动等于损坏语法，测试必须同时固定合法嵌套和损坏变体。
|
||||
|
||||
## 7. 换行策略
|
||||
|
||||
一个候选的“原始边界”包括左物理行尾、允许的空行，以及右侧重复的块前缀。精确和左右正则规则先把左右匹配片段
|
||||
变成 `replacement`，再把所选边界输出放在 replacement 之后、右侧未匹配内容之前。行尾正则规则不消费右侧匹配，
|
||||
只按下表替换原始边界：
|
||||
|
||||
| 策略 | 输出 |
|
||||
| --- | --- |
|
||||
| `PRESERVE` | 保留原始物理行尾、空行和右侧块前缀 |
|
||||
| `DELETE` | 删除整个边界和右侧重复前缀 |
|
||||
| `SPACE` | 用一个 ASCII 空格替换整个边界和右侧重复前缀 |
|
||||
| `PARAGRAPH` | 用左物理行尾样式生成恰好一个空行，并恢复右侧所需块前缀 |
|
||||
|
||||
`PARAGRAPH` 在普通段落中分别产生 `EOL + EOL`；在 block quote 中产生合法的空 quote 行和下一行 quote 前缀；
|
||||
在 list item 中保留 continuation 所需缩进。具体字节由合成测试固定，不根据操作系统默认换行。
|
||||
|
||||
候选内部出现混合 CR、LF、CRLF 时失败关闭，不匹配该候选。文档其他位置可以使用不同换行风格，不做全文规范化。
|
||||
|
||||
## 8. 冲突与链式合并
|
||||
|
||||
### 8.1 同一边界多规则冲突
|
||||
|
||||
每个候选边界先按调用方给出的规则顺序收集全部命中，再执行：
|
||||
|
||||
| 策略 | 行为 |
|
||||
| --- | --- |
|
||||
| `FIRST` | 选择声明顺序最早的规则 |
|
||||
| `PRIORITY` | 选择最高 `priority`；最高值相同时选择声明顺序最早的规则 |
|
||||
| `ERROR` | 两条及以上规则命中即抛出 `ModifierContractError`，消息只含位置和 `rule_id` |
|
||||
|
||||
自动词典规则只有在结构 pattern 命中且第 4.5 节选出了一个候选形式后才进入冲突集合；无词典候选或 `ambiguity=KEEP`
|
||||
不算命中。`ambiguity=ERROR` 的词典歧义直接报错，不先交给规则冲突策略掩盖。
|
||||
|
||||
规则顺序因此是参数和行为的一部分，不再为了得到顺序无关结果而排序。`PRIORITY` 的同优先级 tie-break 明确采用
|
||||
声明顺序，不把集合迭代顺序或 replacement 字典序当作隐藏规则。
|
||||
|
||||
`KeepLineJoinRule` 与其他规则参加同一次冲突选择；它只有在赢得选择后才阻止修改。这样精确项目例外可以用更高
|
||||
优先级覆盖语言级词典规则，而不会让一条低优先级保护规则意外屏蔽更明确的替换。
|
||||
|
||||
### 8.2 跨多行链式合并
|
||||
|
||||
修改器从前到后处理物理边界，并维护只存在于当前 `propose()` 调用内的不可变中间片段。选中一条规则后：
|
||||
|
||||
1. 组合该边界的虚拟输出；
|
||||
2. 标记该原始物理边界已经消费；
|
||||
3. 如果输出与下一原始物理行形成新的候选，继续匹配下一边界；
|
||||
4. 每个原始物理边界最多消费一次，因此最多执行 `physical_line_count - 1` 次，不存在无界循环；
|
||||
5. 相连边界最终合成一个覆盖原始连续范围的 `TextEdit`。
|
||||
|
||||
例如两条显式规则可以把：
|
||||
|
||||
```text
|
||||
exam-
|
||||
ple-
|
||||
based
|
||||
```
|
||||
|
||||
在一次提议中变成 `example-based`。修改器不会依赖 `Pipeline` 自动运行第二轮，也不会先提交两个重叠编辑再让执行器猜测。
|
||||
|
||||
不相连的链生成不同且不重叠的 `TextEdit`。最终按原文起点排序。若内部规划仍产生重叠、无实际变化或无法映射回原始
|
||||
范围，修改器抛出契约错误，不静默丢弃整批正确候选。
|
||||
|
||||
## 9. 大小写和 Unicode 规范化
|
||||
|
||||
- `case_sensitive=True` 是默认值；
|
||||
- `normalization=None` 是默认值；
|
||||
- 规范化只建立匹配视图，不规范化整篇文档；
|
||||
- 固定 replacement 按调用方原样输出；
|
||||
- 正则反向引用使用映射回原文的捕获文本，不借匹配视图静默改变大小写或规范形式；
|
||||
- 如果规范化后的匹配边界不能唯一映射回 Python 原字符串索引，该候选失败关闭；
|
||||
- 兼容等价匹配、NFKC/NFKD 兼容匹配和忽略大小写都必须由规则显式开启。
|
||||
|
||||
实现可以为一个逻辑内容建立规范化视图及源索引边界表，但不能把规范化后的全文作为 `expected_text`，也不能绕过
|
||||
`TextSpan` 的原始 Python 字符索引契约。
|
||||
|
||||
## 10. 验证、错误与审计
|
||||
|
||||
工厂构造时验证：
|
||||
|
||||
- 输入可迭代且每项是合法三元组或规则 dataclass；
|
||||
- `rule_id` 非空、稳定、唯一；
|
||||
- 枚举、布尔值、整数和块集合类型准确，不接受用真值冒充布尔值；
|
||||
- 左右精确片段、separator 和 replacement 不包含 CR/LF；
|
||||
- 精确规则的 `left`、`right` 和 replacement 非空；separator 可以为空；
|
||||
- 正则可以编译，左右正则规则的两侧及行尾正则规则的右侧不会只产生零长度匹配；
|
||||
- 捕获组均有名称、左右名称不重复、replacement 引用存在；
|
||||
- 行尾正则规则不接受 `PRESERVE`；
|
||||
- 自动词典规则的 separator 只能是 `"-"` 或 `""`，pattern 必须消费非空片段；
|
||||
- 自动词典规则必须包含 `JOINED` 和对应源形式，候选形式不能重复；
|
||||
- backend、非空语言代码、有限且非负的 `minimum_score` / `minimum_score_margin` 和可选 Pyphen 语言合法；
|
||||
- 请求的 backend 或 Pyphen 不可用时明确报错，不创建行为不完整的修改器；
|
||||
- `max_intervening_blank_lines` 的类型严格为 `int` 且只能是 `0` 或 `1`；
|
||||
- 旧三元组仍拒绝空字段、换行和重复的 `(left, right)`。
|
||||
|
||||
运行时冲突、无法映射的内部范围和不变量破坏抛出明确异常。普通无匹配、结构排除和不能唯一映射的 Unicode 候选是
|
||||
失败关闭条件，返回无提议，不是异常。
|
||||
|
||||
每个实际链生成一个 `ProposedChange` 和一个 `TextEdit`。`reason` 包含所用 `rule_id` 序列，不记录未命中规则，
|
||||
也不输出候选周围正文。`Modifier.parameters` 记录：
|
||||
|
||||
- 调用顺序中的完整规则；
|
||||
- 规则类型、pattern/literal、replacement、separator、块范围、换行策略、优先级、大小写和规范化选项；
|
||||
- 冲突策略和允许的空行数。
|
||||
|
||||
修改器版本从 `1.0.0` 更新为 `2.0.0`，表示规则语义和参数记录发生变化。
|
||||
|
||||
## 11. 依赖和函数式边界
|
||||
|
||||
- 核心安装仍只依赖 Python 标准库；
|
||||
- `pyproject.toml` 新增 `lexical` optional extra，包含受限版本的 `pyspellchecker` 和 `pyphen`；
|
||||
- `wordfreq` 体积和传递依赖更大，放入单独的 `frequency` optional extra，不随 `lexical` 或核心安装；
|
||||
- optional import 使用模块级 `try/except ImportError` 保存“不可用”状态；只有规则明确请求对应 backend 时才报
|
||||
`ModifierContractError`，精确和正则规则不受未安装 extra 影响；
|
||||
- backend 和 Pyphen 实例在 `mapped_line_join()` 构造修改器时建立一次，不在每个文档或每个候选上重复加载词典；
|
||||
- 参数记录 backend、第三方包版本、语言、阈值、候选形式和 Pyphen 语言；
|
||||
- 不在 import 时读取词典、环境变量、当前时间或网络；
|
||||
- 规则 dataclass、编译后的内部规则和扫描结果不可变；
|
||||
- 闭包只捕获构造时验证和冻结的数据；
|
||||
- `propose()` 只读取 `DocumentSnapshot` 并返回 tuple，不应用编辑或写文件。
|
||||
|
||||
`pyspellchecker` 和 `wordfreq` 自带的数据版本随包版本冻结；本轮不下载更新。Pyphen 使用其已安装包内词典。
|
||||
调用方仅安装 extra 不会自动启用任何规则，因而不会出现“环境里碰巧多了一个包，清洗结果就改变”的隐式行为。
|
||||
|
||||
PyEnchant/Hunspell backend 若以后进入运行时，需要新的 design 解决系统 provider、词典版本和可复现记录，不能在本轮
|
||||
用宽泛 `try/except` 悄悄替代已选择的 backend。
|
||||
|
||||
## 12. 文档与兼容范围
|
||||
|
||||
实现后的模块顶层 docstring 记录第 2 节的简短结论、支持模式和限制，不新增同目录 `NOTES.md`，避免形成第二份接口事实。
|
||||
具体签名和运行行为以代码与测试为准。
|
||||
|
||||
根 `README.md` 当前明确写着 `mapped_line_join()` 只使用三元组并只处理当前窄边界。实现本文后该说明会过期。
|
||||
按照 README 的事实权威要求，实施范围必须允许同步更新 README 的当前能力和示例。若仍要求 Git diff 只能包含目标源码
|
||||
和 tests，则本文不能实施。
|
||||
|
||||
本轮不承诺新 dataclass 从 `mdpolish.modifiers` 聚合模块导出。受支持导入路径是：
|
||||
|
||||
```python
|
||||
from mdpolish.modifiers import mapped_line_join
|
||||
from mdpolish.modifiers.mapped_line_join import (
|
||||
ExactLineJoinRule,
|
||||
KeepLineJoinRule,
|
||||
LexicalLineJoinRule,
|
||||
LineEndRegexRule,
|
||||
RegexLineJoinRule,
|
||||
)
|
||||
```
|
||||
|
||||
旧 `mapped_line_join()` 和 `LineJoinMapping` 聚合导入保持可用。
|
||||
|
||||
## 13. 测试与验收
|
||||
|
||||
测试只使用虚构 Markdown，不读取真实或外部数据。至少覆盖：
|
||||
|
||||
- 旧三元组精确匹配和新 `ExactLineJoinRule`；
|
||||
- `separator="-"` 与 `separator=""`；
|
||||
- 左右命名正则和跨两侧 backreference；
|
||||
- `LineEndRegexRule` 的行尾锚点 + 行首正则，并确认右侧条件文本不被消费；
|
||||
- 小型合成词典 backend 自动把行尾和行首组成 `JOINED`、`HYPHENATED`、`SPACED` 候选；
|
||||
- 拼接词唯一命中自动合并，不需要逐词精确映射；
|
||||
- 拼接词与源形式都有可比分数时按分差选择，分差不足按 `KEEP`/`ERROR` 处理；
|
||||
- Pyphen 合法断点门槛允许和拒绝 `JOINED` 候选；
|
||||
- 高优先级 `KeepLineJoinRule` 能否决自动词典规则；
|
||||
- 请求未安装 backend 明确失败，绝不退回另一 backend 或正则猜测；
|
||||
- 已安装 optional extra 时分别做最小真实 adapter 测试；未安装时明确 skip 并报告，不把 skip 写成通过；
|
||||
- 段落、ATX 标题 continuation、列表 continuation 和同深度引用的正向范围；
|
||||
- 合法的三层引用和正确缩进的嵌套有序列表可以合并；
|
||||
- 引用深度不一致、损坏 quote 前缀、相邻同级列表项、缩进不足的嵌套列表 continuation，以及未识别的 marker-like
|
||||
结构归入内部 `UNKNOWN`，不合并且绝不回退成段落；
|
||||
- heading/list/quote 前缀在四种换行策略下的准确输出；
|
||||
- fenced code、缩进代码、GFM pipe table 和 raw HTML table 不合并；
|
||||
- 空文档、空规则、文末无换行；
|
||||
- LF、CR、CRLF 以及候选内混合行尾失败关闭；
|
||||
- 隔零个或一个空行的配置，以及 `-1`、`True`、`2` 被构造期拒绝；
|
||||
- 三行及以上链式合并、同一合并后逻辑行继续匹配；
|
||||
- 一篇文档中的多个独立链按原文顺序报告；
|
||||
- `FIRST`、`PRIORITY`、同优先级 tie-break 和 `ERROR`；
|
||||
- 默认区分大小写和显式忽略大小写；
|
||||
- 默认不规范化、NFC/NFD 匹配、不能唯一回映时失败关闭；
|
||||
- Unicode 词边界和显式关闭词边界；
|
||||
- 无效 dataclass 字段、正则、捕获组、replacement 引用和重复 `rule_id` 明确失败；
|
||||
- 无效 backend、语言、候选组合、score 阈值和 margin 明确失败；
|
||||
- `WORDFREQ` 候选按 Zipf margin 比较；`SPELLCHECKER` 单 token 与 `SPACED`/拆分验证的 `HYPHENATED`
|
||||
同时存活时按不可排序歧义处理，不用 margin 强行选择；
|
||||
- 参数记录完整且确定；
|
||||
- 所有正向输出再次运行零修改；
|
||||
- 精确/正则规则不读取文件、网络、词典或环境；词典规则只读取已安装 extra 自带资源，且不访问网络或任意路径。
|
||||
|
||||
实现后运行根 README 届时列出的全部基础检查：
|
||||
|
||||
```bash
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests
|
||||
.venv/bin/pytest
|
||||
.venv/bin/python -m pip wheel . --no-deps --wheel-dir /tmp/mdpolish-wheel-check
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
最终报告只贴本轮真实输出。失败必须修复或明确报告，不能放宽断言掩盖。
|
||||
|
||||
## 14. 风险与代价
|
||||
|
||||
- **词法块扫描不是完整 parser：** 它能保守排除常见代码和表格，并处理约定的四类块；复杂嵌套或方言结构可能漏匹配。
|
||||
- **显式正则仍可能写错：** 库保证锚点、冲突和精确编辑安全，不保证调用方规则符合具体业务语义。
|
||||
- **heading continuation 有歧义：** 紧随标题的普通行可能本来就是段落；只有映射本身已获项目确认时才应允许 heading 规则。
|
||||
- **兼容规范化可能多对一：** 无法唯一回映时选择漏处理，不猜测源范围。
|
||||
- **链式规划增加实现复杂度：** 用每个物理边界最多消费一次和单个连续编辑限制状态空间，换取一次提议内稳定完成。
|
||||
- **规则顺序成为契约：** 这是支持 `FIRST` 以及优先级 tie-break 的必要代价，调用方必须把顺序纳入版本和评审。
|
||||
- **词典仍会漏掉专名和新词：** 自动规则对无命中和歧义失败关闭；项目可用精确规则补充，不把低覆盖伪装成全文正确。
|
||||
|
||||
## 15. 批准后的实施边界
|
||||
|
||||
批准本文将授权：
|
||||
|
||||
1. 重构 `src/mdpolish/modifiers/mapped_line_join.py`；
|
||||
2. 新增或扩展 `tests/test_mapped_line_join.py`，必要时只在 `tests/` 增加同主题测试文件；
|
||||
3. 更新 `pyproject.toml`，只增加第 11 节批准的 optional extras；
|
||||
4. 更新根 `README.md` 中已经过期的当前能力边界、安装方式和调用示例；
|
||||
5. 在不改变正文的前提下检查 `AGENTS.md` 与 `CLAUDE.md` 镜像；
|
||||
6. 运行第 13 节检查并查看 Git diff、wheel 内容和工作区状态。
|
||||
|
||||
批准本文不授权修改 `_text_ranges.py`、其他 modifier、其他 Wiki 文档、真实数据、外部系统，
|
||||
也不授权提交、推送、创建 PR 或发布。
|
||||
@@ -0,0 +1,359 @@
|
||||
# 0010：第一版跨项目库交付
|
||||
|
||||
## 状态
|
||||
|
||||
已于 2026-08-27 经用户明确批准。本文自批准起冻结；后续若改变这里的 backend 契约、版本身份或发布渠道，
|
||||
应新增 design，不得回写本文。
|
||||
|
||||
`supersedes: 0009`（范围有限）：本文只替代 `0009` 中没有被真实 optional dependency 验证覆盖的
|
||||
`SPELLCHECKER` 大小写适配、发布前验收和版本身份部分。`0009` 已批准的规则模型、候选比较、失败关闭、
|
||||
Markdown 块边界、无默认规则、无网络和函数式 `Modifier` 契约继续有效。
|
||||
|
||||
本文批准后只授权实现和验证，不自动授权提交、创建 tag、push、创建 GitHub Release、上传产物、修改其他仓库
|
||||
或读取真实材料。
|
||||
|
||||
## 1. 问题与可观察现象
|
||||
|
||||
提交 `11349e0` 已经实现通用 `mapped_line_join()`，仓库基础检查在 Python 3.13.11 下得到：
|
||||
|
||||
```text
|
||||
168 passed, 2 skipped
|
||||
```
|
||||
|
||||
两个 skip 分别对应没有安装的 `pyspellchecker` 和 `wordfreq`。这意味着核心契约和合成 backend 通过了测试，
|
||||
但最关键的真实词典适配器没有进入同一次验收。
|
||||
|
||||
2026-08-27 在临时干净 venv 中从 wheel 安装全部 extras 后，实际版本为：
|
||||
|
||||
| 包 | 实际版本 |
|
||||
| --- | --- |
|
||||
| `mdpolish` | `0.2.0` |
|
||||
| `pyspellchecker` | `0.9.0` |
|
||||
| `pyphen` | `0.18.1` |
|
||||
| `wordfreq` | `3.1.1` |
|
||||
|
||||
`wordfreq`、`Pyphen` 和显式 `case_sensitive=False` 的 `pyspellchecker` 都能把合成输入
|
||||
`an exam-\nple` 稳定处理成 `an example`。但是，当前 `LexicalLineJoinRule.case_sensitive` 默认是 `True`，
|
||||
实现会把它直接传给：
|
||||
|
||||
```python
|
||||
SpellChecker(language="en", case_sensitive=True)
|
||||
```
|
||||
|
||||
真实上游立即拒绝：
|
||||
|
||||
```text
|
||||
ValueError: case_sensitive can only be True when not using a language dictionary.
|
||||
```
|
||||
|
||||
`pyspellchecker` 的源码和官方 quickstart 都说明，大小写敏感模式只适用于不加载内置 language dictionary 的实例：
|
||||
|
||||
- <https://github.com/barrust/pyspellchecker/blob/master/spellchecker/spellchecker.py>
|
||||
- <https://github.com/barrust/pyspellchecker/blob/master/docs/source/quickstart.rst>
|
||||
|
||||
这不是词典无命中，而是构造阶段的能力冲突。当前实现把它包装成泛化的“无法加载语言”错误，调用方无法知道
|
||||
应该怎样改配置。
|
||||
|
||||
另一个交付问题是包版本仍为 `0.2.0`。这个版本已经用于 `0008` 后的函数式核心；如果新旧内容继续生成同名 wheel，
|
||||
其他项目、pip 缓存和问题报告都无法可靠区分实际安装的是哪一份代码。
|
||||
|
||||
因此，当前提交适合受控试接，不适合直接作为一个带稳定版本身份的跨项目交付。
|
||||
|
||||
## 2. 目标与非目标
|
||||
|
||||
### 2.1 目标
|
||||
|
||||
- 对 `SPELLCHECKER` 不支持的大小写组合给出构造期、可操作、不会泄露正文的错误；
|
||||
- 保留 `case_sensitive=True` 的公共默认值，不静默开启忽略大小写；
|
||||
- 用真实安装的 `pyspellchecker`、Pyphen 和 `wordfreq` 验证 wheel，而不是把 skip 当成通过；
|
||||
- 为这次新增公共能力分配唯一包版本 `0.3.0`；
|
||||
- 将修复后的 `mapped_line_join` 修改器版本更新为 `2.0.1`，使审计记录可以区分修复前后；
|
||||
- 在 README 提供其他项目可以直接复制的安装、导入和自动英文断词示例；
|
||||
- 将 GitHub tag 和 GitHub Release 确定为计划内的正式发布渠道，明确不使用 PyPI；
|
||||
- 明确第一版固定版本方式和消费者责任；
|
||||
- 保持核心安装零第三方运行依赖，extras 仍然只由调用方显式安装和启用。
|
||||
|
||||
### 2.2 非目标
|
||||
|
||||
- 不为 `pyspellchecker` 的内置语言词典自行实现大小写敏感查询；
|
||||
- 不读取、复制或改写 `pyspellchecker` 的内部压缩词典资源；
|
||||
- 不把 `case_sensitive` 默认值改成 `False`；
|
||||
- 不引入 Hunspell、Enchant、wordninja、在线词典、模型或 OCR 版面接口；
|
||||
- 不新增默认规则、默认流水线、项目 profile、配置文件或 CLI；
|
||||
- 不声称词典规则已经在真实业务语料上达到生产准确率；
|
||||
- 不在本轮定义标注指标、接受阈值或真实材料实验；
|
||||
- 不发布到 PyPI 或其他 Python 包索引，不提交 wheel 到 Git；
|
||||
- 不因批准本文就自动创建 GitHub Release 或上传产物；
|
||||
- 不修改其他 modifier、`_text_ranges.py`、真实数据或其他仓库。
|
||||
|
||||
## 3. `SPELLCHECKER` 大小写适配
|
||||
|
||||
### 3.1 候选方案
|
||||
|
||||
| 方案 | 优点 | 代价与问题 | 选择 |
|
||||
| --- | --- | --- | --- |
|
||||
| 把 `LexicalLineJoinRule.case_sensitive` 默认改成 `False` | 默认示例可以直接运行 | 违反“不默认忽略大小写”，并静默改变公共语义 | 否决 |
|
||||
| 忽略调用方的 `True`，内部总用大小写不敏感词典 | 代码最少 | 参数记录与实际行为不一致，属于静默降级 | 否决 |
|
||||
| 读取上游包内 JSON 频率表，自己实现大小写敏感词典 | 理论上可以支持 `True` | 绑定上游内部资源路径和格式，扩大维护面；第一版没有证据需要它 | 本轮否决 |
|
||||
| `SPELLCHECKER` 明确要求 `case_sensitive=False` | 行为诚实、改动小、调用方必须主动选择 | 该 backend 暂不提供大小写敏感语言词典 | 采用 |
|
||||
|
||||
### 3.2 决定
|
||||
|
||||
`case_sensitive` 继续是所有规则共有的显式开关，默认仍为 `True`。新增 backend 组合验证：
|
||||
|
||||
```text
|
||||
backend == SPELLCHECKER and case_sensitive is True
|
||||
```
|
||||
|
||||
在 `mapped_line_join()` 构造阶段立即抛出 `ModifierContractError`。错误消息必须说明：
|
||||
|
||||
- `pyspellchecker` 的内置 language dictionary 不支持大小写敏感模式；
|
||||
- 调用方若接受大小写不敏感匹配，需要显式设置 `case_sensitive=False`；
|
||||
- 不建议切换 backend，也不执行自动回退。
|
||||
|
||||
错误只包含 backend、`rule_id` 和配置字段，不包含输入正文。因为错误发生在构造修改器时，也不应等到
|
||||
`Pipeline.transform()` 才暴露。
|
||||
|
||||
`WORDFREQ` 保留现有 `case_sensitive=True/False` 行为。不能因为一个 backend 的限制，把共同字段的默认值改成
|
||||
另一个含义。
|
||||
|
||||
### 3.3 为什么第一版不复制词典
|
||||
|
||||
`pyspellchecker` 官方允许用 `language=None, case_sensitive=True` 加载调用方自己的词典，但当前规则只接受语言代码，
|
||||
设计也禁止读取任意路径。为了绕过限制而解析依赖包内部的 `resources/<language>.json.gz`，会新增一套未获保证的
|
||||
资源格式契约。
|
||||
|
||||
第一版更重要的是不误导调用方。明确拒绝不支持的组合，比复制上游实现、静默忽略参数或假装拥有大小写敏感语言词典
|
||||
更符合失败关闭原则。如果未来项目确实需要该能力，应以真实样本和新 design 决定是增加版本化自带词典、扩展 backend，
|
||||
还是接受自定义词典输入；不能回写本文。
|
||||
|
||||
## 4. 版本身份
|
||||
|
||||
### 4.1 包版本
|
||||
|
||||
实现本文时把 `pyproject.toml` 和 README 中的包版本从 `0.2.0` 更新为 `0.3.0`。
|
||||
|
||||
选择 `0.3.0` 而不是 `1.0.0`，原因是：
|
||||
|
||||
- 新能力是向现有函数式核心增加公共规则和 optional backend，属于 `0.x` 阶段的次版本变化；
|
||||
- 公共规则还没有经过多个调用项目验证；
|
||||
- Markdown 块扫描仍是保守词法子集；
|
||||
- 没有真实语料准确率、默认 profile、CLI 或生产写入协议。
|
||||
|
||||
`0.3.0` 表示“可以被其他项目固定版本试用”，不表示“自动词典规则已经适合任意文档生产启用”。
|
||||
|
||||
### 4.2 修改器版本
|
||||
|
||||
`mapped_line_join()` 生成的 `Modifier.version` 从 `2.0.0` 更新为 `2.0.1`。
|
||||
|
||||
规则 dataclass 和序列化参数格式不改变，因此不升到 `3.0.0`。补丁版本用于表示：
|
||||
|
||||
- 真实 `SPELLCHECKER` backend 的能力验证变得准确；
|
||||
- 不支持的组合从上游泛化异常变成稳定的构造期契约错误；
|
||||
- 已支持组合的候选选择语义不改变。
|
||||
|
||||
### 4.3 版本记录
|
||||
|
||||
词典规则继续在 `Modifier.parameters` 中记录 backend 包版本、语言、候选形式和阈值。调用项目还必须固定
|
||||
`mdpolish` 包版本；只记录 `Modifier.version` 不能替代安装依赖锁定。
|
||||
|
||||
## 5. 第一版交付渠道
|
||||
|
||||
### 5.1 采用范围
|
||||
|
||||
项目计划通过 GitHub 发布版本，不发布到 PyPI 或其他 Python 包索引。不可移动的 Git tag 是源码身份，GitHub Release
|
||||
是面向使用方的正式发布记录。第一版候选 tag 和 Release 名称为：
|
||||
|
||||
```text
|
||||
v0.3.0
|
||||
```
|
||||
|
||||
tag 必须指向通过本文全部验收的唯一提交，创建后不得移动；GitHub Release 必须绑定这个 tag，不能指向分支头。
|
||||
其他项目可以通过 GitHub Git 地址和 tag 固定依赖，也可以安装该 Release 附带的 wheel。Python Packaging 规范允许
|
||||
集成方使用 direct reference，但它不是包索引发布物：
|
||||
|
||||
- <https://packaging.python.org/en/latest/specifications/version-specifiers/#direct-references>
|
||||
- <https://packaging.python.org/en/latest/specifications/dependency-specifiers/>
|
||||
|
||||
README 应给出 GitHub tag direct reference 和 Release wheel 两种安装形式，但不把仓库凭据或某个调用项目配置写入库代码。
|
||||
仓库及 Release 是公开还是私有，由 GitHub 仓库权限决定；本文不授权改变仓库可见性。
|
||||
|
||||
### 5.2 wheel 的角色
|
||||
|
||||
wheel 在发布前首先是验证产物，不提交到 Git，也不在仓库内建立 artifact 目录。验收必须从最终源码构建 wheel，并在
|
||||
干净环境从这个 wheel 安装，而不是依赖当前仓库的 editable install。
|
||||
|
||||
实际创建 GitHub Release 时，只能上传通过第 7 节验收的同一个 wheel，并同时提供 SHA-256 校验值。若 tag 后重新构建，
|
||||
必须重新执行 wheel 元数据和消费者 smoke test，不能把不同构建物当作已经验收的产物。GitHub 自动生成的源码归档与
|
||||
Release wheel 共同保存在 GitHub；本仓库不另存一份二进制副本。
|
||||
|
||||
PyPI、GitHub Packages、内部 Python 包索引和其他 artifact 仓库均不在计划内。将来若要增加其他发布渠道，必须用新的
|
||||
design 改变本文，而不能只改发布脚本或 README。
|
||||
|
||||
### 5.3 tag 和 push 的确认门
|
||||
|
||||
本文批准后可以完成版本修改、测试、构建和提交前候选检查，但不能自动提交、创建或推送 `v0.3.0`，也不能创建
|
||||
GitHub Release 或上传 wheel。
|
||||
|
||||
只有在用户看到最终 diff、真实测试输出和 wheel 元数据后，才能明确授权：
|
||||
|
||||
1. 提交交付改动；
|
||||
|
||||
提交完成并向用户报告提交哈希和 tag 的准确目标后，才能再明确授权：
|
||||
|
||||
1. 创建 `v0.3.0` tag；
|
||||
2. push 提交和 tag；
|
||||
3. 创建绑定 `v0.3.0` 的 GitHub Release；
|
||||
4. 上传已验收的 wheel 和 SHA-256 校验值。
|
||||
|
||||
这些动作不因“设计已批准”而自动获得授权。
|
||||
|
||||
## 6. 公共调用契约与 README
|
||||
|
||||
### 6.1 支持的导入路径
|
||||
|
||||
保持 `0009` 的决定，不扩大聚合导出：
|
||||
|
||||
```python
|
||||
from mdpolish.modifiers import mapped_line_join
|
||||
from mdpolish.modifiers.mapped_line_join import (
|
||||
LexicalCandidateForm,
|
||||
LexicalLineJoinRule,
|
||||
LexiconBackend,
|
||||
)
|
||||
```
|
||||
|
||||
本轮不把全部枚举和 dataclass 重新导出到 `mdpolish.modifiers` 或包根。减少顶层公共表面积比缩短一行导入更重要。
|
||||
|
||||
### 6.2 README 示例
|
||||
|
||||
README 保留旧三元组示例，并新增一个最小自动英文断词示例。示例必须明确写出：
|
||||
|
||||
- 通过 GitHub tag direct reference 或 GitHub Release wheel 安装 `lexical` extra；
|
||||
- `backend=LexiconBackend.SPELLCHECKER`；
|
||||
- `case_sensitive=False`；
|
||||
- `JOINED` 与源 `HYPHENATED` 共同参与候选；
|
||||
- 分数阈值只是示例配置，不是库推荐的通用生产阈值；
|
||||
- 检查 `RunStatus` 后才能消费输出。
|
||||
|
||||
README 还要明确区分：
|
||||
|
||||
| 说法 | 当前是否成立 |
|
||||
| --- | --- |
|
||||
| wheel 可以安装，公共 API 可以运行 | 是，验收通过后成立 |
|
||||
| 自动词典不需要逐词维护映射 | 是 |
|
||||
| 库自带默认英文清洗规则 | 否 |
|
||||
| 某个阈值适合所有项目 | 否 |
|
||||
| 已在真实业务文档证明生产准确率 | 否 |
|
||||
|
||||
### 6.3 调用项目责任
|
||||
|
||||
调用项目必须：
|
||||
|
||||
- 固定 `mdpolish` 版本或不可移动 tag；
|
||||
- 把规则、顺序、backend、语言、阈值和例外作为项目配置评审；
|
||||
- 检查 `success`、`failed`、`unstable`，不能只读取可能为空的输出；
|
||||
- 自己负责文件读写、覆盖策略、批处理、日志和回滚；
|
||||
- 在自己的语料上验证误合并，不能把库的合成测试当作领域准确率。
|
||||
|
||||
## 7. 验证矩阵
|
||||
|
||||
### 7.1 核心环境
|
||||
|
||||
不安装 `lexical` 或 `frequency` extras，运行根 README 的全部基础检查，确认：
|
||||
|
||||
- 精确、正则、链式和 Markdown 失败关闭测试通过；
|
||||
- 导入模块不会加载词典；
|
||||
- 请求缺失 backend 时给出稳定错误；
|
||||
- optional backend 测试可以明确 skip，但 skip 不能计入真实 backend 验收。
|
||||
|
||||
### 7.2 extras 环境
|
||||
|
||||
另建干净环境，安装最终 wheel 的 `lexical` 和 `frequency` extras，并另行安装仓库测试工具，运行同一测试集。
|
||||
该环境的发布门要求：
|
||||
|
||||
- `pyspellchecker`、Pyphen 和 `wordfreq` 真实 adapter 测试全部执行，不得 skip；
|
||||
- `SPELLCHECKER + case_sensitive=True` 在构造期得到预期契约错误；
|
||||
- `SPELLCHECKER + case_sensitive=False` 的唯一拼接候选可以合并；
|
||||
- Pyphen 合法和非法断点分别影响 `JOINED`；
|
||||
- `WORDFREQ` 唯一胜者、margin 不足和歧义路径都符合 `0009`；
|
||||
- 包版本和 backend 版本进入审计参数；
|
||||
- 全套测试没有因安装 extras 而改变精确/正则规则结果。
|
||||
|
||||
测试不能只断言“构造成功”。至少一个真实 backend 用例必须通过 `Pipeline.transform()` 检查最终状态和完整输出。
|
||||
|
||||
### 7.3 wheel 消费者 smoke test
|
||||
|
||||
从最终提交构建 wheel 后，在不位于仓库源码目录的临时环境验证：
|
||||
|
||||
1. 只安装核心 wheel，运行精确和正则示例；
|
||||
2. 安装 `lexical` extra，运行 `SPELLCHECKER + Pyphen` 示例；
|
||||
3. 安装 `frequency` extra，运行 `WORDFREQ` 示例；
|
||||
4. 检查 `mdpolish`、修改器和 backend 版本记录；
|
||||
5. 检查 wheel 不包含 tests、Wiki、真实数据、项目规则或临时产物；
|
||||
6. 生成并记录待上传 wheel 的 SHA-256 校验值；
|
||||
7. 确认执行期间不访问网络、调用模型或读取任意文档路径。
|
||||
|
||||
依赖安装本身可以访问配置的包索引；“运行期间无网络”指安装完成后的库行为，不把安装包与执行清洗混为一谈。
|
||||
|
||||
### 7.4 Python 支持范围
|
||||
|
||||
`pyproject.toml` 当前声明 Python 3.11 及以上。第一版交付前至少验证：
|
||||
|
||||
- 最低支持版本 Python 3.11；
|
||||
- 当前开发版本 Python 3.13。
|
||||
|
||||
如果本地缺少其中一个解释器，不能把单版本结果描述成完整支持矩阵；应在可复现 CI 或受控环境补齐后再创建 tag。
|
||||
本文不新增 tox、nox、CI provider 或容器配置。若现有环境无法完成双版本验证，报告阻塞而不是降低声明。
|
||||
|
||||
## 8. 消费项目试用边界
|
||||
|
||||
通过第 7 节只证明“库可以被安装并按契约运行”，不证明“某组词典参数适合目标项目”。
|
||||
|
||||
第一个调用项目应先做只读或影子试用：保存提议和审计信息，由人复核后再决定是否应用。至少观察：
|
||||
|
||||
- 正确合并与错误合并；
|
||||
- 本应合并但被保留的候选；
|
||||
- `JOINED`、`HYPHENATED`、`SPACED` 的选择分布；
|
||||
- 按段落、标题、列表和引用拆分的行为；
|
||||
- 词典歧义、结构 `UNKNOWN`、代码和表格排除；
|
||||
- 项目专名和自然连字符是否需要 `KeepLineJoinRule`。
|
||||
|
||||
具体样本范围、标注方法、指标、接受阈值、输出目录和真实数据权限必须在调用项目或新的实验 design 中确认。
|
||||
本文不授权读取 `/home/lihaoze/gov_test_data`，也不授权修改任何调用项目。
|
||||
|
||||
## 9. 风险与代价
|
||||
|
||||
- **`SPELLCHECKER` 能力不对称：** 使用内置语言词典时必须显式忽略大小写；需要精确大小写的项目应使用其他 backend
|
||||
或等待新的词典设计。
|
||||
- **真实 extras 增加验收成本：** `wordfreq` wheel 和传递依赖较大，但不能为了节省安装时间继续跳过发布关键路径。
|
||||
- **GitHub 可用性与权限：** direct reference 需要 Git 和相应仓库权限；Release wheel 也受仓库可见性和 GitHub
|
||||
可用性约束。
|
||||
- **Release 产物一致性：** tag、Release 和 wheel 来自不同操作步骤，必须用提交哈希、版本元数据和 SHA-256 防止
|
||||
上传错误构建物。
|
||||
- **`0.3.0` 仍是预览契约：** 其他项目必须固定版本，不能跟随分支头自动升级。
|
||||
- **双 Python 版本可能受环境限制：** 缺少最低版本验证时，tag 会被阻塞。
|
||||
- **合成测试不代表领域准确率：** 即使全部 backend 测试通过，词典仍会漏掉专名、新词并误判自然连字符。
|
||||
- **保守失败关闭降低覆盖率：** 这是第一版为了正文保真接受的代价，不用放宽块扫描来追求漂亮数字。
|
||||
|
||||
## 10. 实施与验收范围
|
||||
|
||||
本文获批后授权：
|
||||
|
||||
1. 修改 `src/mdpolish/modifiers/mapped_line_join.py`，增加 `SPELLCHECKER` 组合验证并把修改器版本更新到 `2.0.1`；
|
||||
2. 扩展 `tests/test_mapped_line_join.py`，覆盖真实 backend、明确错误和版本记录；
|
||||
3. 把 `pyproject.toml` 包版本更新到 `0.3.0`，不改变已批准 extras 的依赖集合；
|
||||
4. 更新根 README 的当前版本、自动词典示例、交付边界和实际验证结果；
|
||||
5. 在临时目录构建和安装 wheel，运行第 7 节验证；
|
||||
6. 只读检查 `AGENTS.md` 与 `CLAUDE.md` 镜像、Git diff、工作区状态和最终提交候选范围。
|
||||
|
||||
本文获批后仍不授权：
|
||||
|
||||
- 修改已冻结的 `0009`；
|
||||
- 修改 `_text_ranges.py`、其他 modifier、其他 Wiki 文档或调用项目；
|
||||
- 读取或复制真实材料；
|
||||
- 新增默认规则、CLI、profile、artifact 目录、CI 配置或第三方 backend；
|
||||
- 提交、创建 `v0.3.0` tag、push、创建 GitHub Release、上传 wheel 或创建 PR。
|
||||
|
||||
实施完成的最终报告必须分别给出核心环境、extras 环境、Python 版本和 wheel smoke test 的真实输出。任何必需环境未验证时，
|
||||
明确写“未验证”或报告阻塞，不能用合成 adapter 测试替代真实 backend 结果。
|
||||
@@ -1,83 +0,0 @@
|
||||
# arXiv 提交边栏戳为什么能自动删除
|
||||
|
||||
## 1. 可观察的问题
|
||||
|
||||
部分 arXiv 论文转换为 Markdown 后，会把提交页边栏中的编号、分类和日期留下来，形成一整行独立文字。它不是
|
||||
论文正文，却会进入后续分块、检索和对比。与此同时，论文参考文献也可能包含 `arXiv:`；只要见到这个子串就删行，
|
||||
会损坏合法引用。
|
||||
|
||||
当前组件只处理前一种格式固定的独立行。它的决策来自已批准的
|
||||
[`0004-arxiv-submission-stamp-component.md`](../design/0004-arxiv-submission-stamp-component.md)，项目范围编号为 H1。
|
||||
精确模式、类名和返回对象以
|
||||
[`arxiv_submission_stamp.py`](../../src/mdpolish/components/arxiv_submission_stamp.py) 及其
|
||||
[`测试`](../../tests/test_arxiv_submission_stamp.py) 为准。
|
||||
|
||||
## 2. 当前识别边界
|
||||
|
||||
组件逐个读取物理行，只在整行同时具有以下结构时提出删除：
|
||||
|
||||
```text
|
||||
arXiv:<新版数字编号和版本> [<ASCII 分类>] <日> <英文月份缩写> <四位年份>
|
||||
```
|
||||
|
||||
首尾空格、列表或引用前缀、缺少版本、旧式编号、错误月份以及句子中的 `arXiv:` 都不会命中。组件没有参数，
|
||||
调用方不能传入更宽松的正则表达式改变同一版本的语义。
|
||||
|
||||
当前版本有意不解析 Markdown 块结构。围栏代码、HTML 注释或其他块中如果存在一行完整目标文字，同样会被删除。
|
||||
这是 `0004` 明确接受的代价，不是实现遗漏。以后出现必须保留的真实反例时，需要重新评审识别边界并更新组件版本。
|
||||
|
||||
## 3. 删除如何保持原文边界
|
||||
|
||||
每个命中行产生一个候选修改和一个删除型文本编辑。删除范围包含该行自己的 `\n`、`\r\n` 或单独 `\r`；
|
||||
没有行尾的末行只删除文字，不拿走前一行已有的行尾。
|
||||
|
||||
| 输入位置 | 当前行为 |
|
||||
| --- | --- |
|
||||
| 首行且有行尾 | 连同行尾删除，后续正文成为首行 |
|
||||
| 文档中间 | 连同目标行自己的行尾删除，前后内容保持两行 |
|
||||
| 末行且没有行尾 | 只删除目标文字，保留前一行原有行尾 |
|
||||
| 多个目标行 | 每行一个候选，按原文顺序记录，作为一个组件批次原子应用 |
|
||||
|
||||
组件不整理空行、不统一换行符，也不改变未命中的字符。候选修改绑定当前快照哈希和准确原文，仍由公共修改执行器
|
||||
验证和应用；组件本身没有文件读写或独立 `transform()`。
|
||||
|
||||
## 4. 审计与稳定性
|
||||
|
||||
组件标识为 `paper.arxiv_submission_stamp`，版本为 `1.0.0`，参数为空。每条实际删除记录固定理由，并保留删除原文、
|
||||
原始范围、组件位置以及批次修改前后的哈希。
|
||||
|
||||
删除完成后目标行已经不存在。流水线最终复查不应再得到候选修改；把成功输出再次交给同一组件，也应保持原文不变
|
||||
且产生零条实际改动。
|
||||
|
||||
## 5. 已完成验证
|
||||
|
||||
合成测试覆盖严格匹配、反向引用、首行/中间/末行、三种行尾、多个命中、围栏中仍删除、审计字段、确定性和
|
||||
第二次运行零修改。测试只使用短小的虚构字符串，不含真实论文片段。
|
||||
|
||||
2026-08-22 先对本地 5 份 ClinDB-ReviewBench Markdown 做了只读、纯内存复核：
|
||||
|
||||
| 复核项 | 结果 |
|
||||
| --- | --- |
|
||||
| 输入范围 | dmp、jama、ejhf、sim、springer 各 1 份，共 5 份 |
|
||||
| 实际删除 | sim 1 行、springer 1 行，其余 0 行，共 2 行 |
|
||||
| 合法反向样例 | springer 的 2 处 `arXiv preprint arXiv:` 修改前后均保留 |
|
||||
| 第二次运行 | 5 份合计 0 条修改 |
|
||||
| 源文件复读 | 5/5 与处理前内存内容一致，没有回写 |
|
||||
|
||||
这次早期复核没有保存清洗后 Markdown，也没有把真实原文复制进测试、日志或仓库。它是 `0004` 当时验证边界的
|
||||
历史事实。
|
||||
|
||||
`0005` 批准本地产物机制后，同日又完成一次保存型实验：5 份文档全部为 `success`，仍然只修改 sim 和 springer
|
||||
各一处；输出哈希全部匹配，输入运行前后字节不变，Springer 两处合法引用仍保留。产物位于
|
||||
`artifacts/2026-08-22/runs/clindb-arxiv-stamp-artifacts-v1/`，只在本机保留并受 Git 忽略。具体产物结构和验证结果见
|
||||
[`local-experiment-artifacts.md`](local-experiment-artifacts.md)。安装、静态检查和完整测试命令仍只在根目录
|
||||
[`README.md`](../../README.md#当前可用检查) 维护。
|
||||
|
||||
## 6. 剩余边界
|
||||
|
||||
这个组件最初证明了第一条严格删除规则能够在公共核心上闭环。此后 ClinDB 第一批另外 7 个组件已经按 `0006` 实现，
|
||||
当前完整组合见 [`clindb-first-batch-components.md`](clindb-first-batch-components.md)。这仍不表示论文内容问题全部解决；
|
||||
通用文件接口、profile、公共 CLI、图片资产和通用批处理仍不存在。
|
||||
|
||||
如果出现新的提交戳格式，默认行为是保留。必须先补充真实证据、反向样例和 design，再决定是否放宽模式，不能为了
|
||||
提高命中数量直接修改正则表达式。
|
||||
@@ -1,116 +0,0 @@
|
||||
# ClinDB 第一批组件如何在不猜正文的前提下完成清洗
|
||||
|
||||
## 1. 它解决的实际问题
|
||||
|
||||
5 份 ClinDB 论文 Markdown 同时包含编辑痕迹、转换噪声和排版噪声。它们看起来都像“删掉几行或整理一下格式”，
|
||||
实际误删边界不同：行首数字可能是手稿行号，也可能是作者单位；`arXiv:` 可能是边栏戳，也可能是合法参考文献；
|
||||
编号列表可能属于 References，也可能是正文方法步骤。
|
||||
|
||||
当前实现没有建立一个能随意改全文的“大清洗器”，而是把第一批确定问题拆成 8 个组件。每个组件只识别一种证据，
|
||||
返回快照绑定的精确 `TextEdit`，由公共流水线统一验证、应用和记录。
|
||||
|
||||
清洗语义来自已批准的
|
||||
[`0006-clindb-first-batch-cleaning-components.md`](../design/0006-clindb-first-batch-cleaning-components.md)，
|
||||
输入范围和稳定计数见
|
||||
[`CLINDB_REVIEWBENCH_CLEANING_SCOPE.md`](../reference/CLINDB_REVIEWBENCH_CLEANING_SCOPE.md)。
|
||||
|
||||
## 2. 当前组件和顺序
|
||||
|
||||
```text
|
||||
Word 批注 ─┐
|
||||
手稿行号 ──┼──► arXiv 戳 ──► 重复页眉 ──► 映射断词
|
||||
│ │
|
||||
│ └────────────► 参考文献空行
|
||||
│
|
||||
└────► HTML 双重实体 ──► HTML 表格布局
|
||||
```
|
||||
|
||||
实验脚本固定按以下顺序组装：
|
||||
|
||||
| 顺序 | 组件 | 当前作用 |
|
||||
| ---: | --- | --- |
|
||||
| 1 | `paper.word_review_comment` | 删除完整单行 Word 批注及其后一个空行 |
|
||||
| 2 | `paper.manuscript_line_number` | 删除由长递增序列确认的手稿行号 |
|
||||
| 3 | `paper.arxiv_submission_stamp` | 删除严格整行提交戳 |
|
||||
| 4 | `paper.repeated_running_header` | 删除重复页眉，并接回有明确续句证据的段落 |
|
||||
| 5 | `paper.page_break_word_join` | 只应用本次运行参数中记录的词片段映射 |
|
||||
| 6 | `markdown.html_table_double_escape` | 只在严格表格单元格文本中解除一层实体转义 |
|
||||
| 7 | `markdown.html_table_layout` | 保留 HTML 内容，把单行表格展开成一行一个 `<tr>` |
|
||||
| 8 | `paper.reference_spacing` | 只在完整连续的 References 章节中统一条目空行 |
|
||||
|
||||
顺序不是为了让结果“看起来整齐”。dmp 的一个重复页眉正好位于第 18、19 条参考文献之间；如果不先删除页眉，
|
||||
参考文献组件就不能确认这是完整连续序列。HTML 实体先修改小范围 token，表格布局再基于新快照替换整张表，
|
||||
两类修改仍能在审计中分别追踪。
|
||||
|
||||
## 3. 为什么行号不能使用全局正则
|
||||
|
||||
JAMA 文档中一共有 134 个看似 `数字 + 空格` 的行首前缀。前 59 个位于 Abstract 之前，是作者单位编号；真正的手稿
|
||||
行号只有 Abstract 之后的 75 个。
|
||||
|
||||
当前组件要求：
|
||||
|
||||
- 文档中恰好有一个严格 `## Abstract`；
|
||||
- 只看它之后的候选；
|
||||
- 候选数字全部严格递增；
|
||||
- 至少有 20 个候选，并至少有 2 个数字标题。
|
||||
|
||||
任一条件失败就整篇不改。这样会漏掉较短的带行号手稿，但不会为了提高命中率删除作者单位或零散数字段落。
|
||||
|
||||
## 4. 为什么断词使用显式映射
|
||||
|
||||
“行尾连字符加下一行小写字母”无法决定连字符应删还是保留：`possi-` + `bly` 应成为 `possibly`，而
|
||||
`SOFA-` + `based` 应保留为 `SOFA-based`。dmp 还存在 `threshold.` + `olds`，它同时包含多余句点和重复片段，
|
||||
普通词典也无法解释。
|
||||
|
||||
因此组件的实际参数记录三项：左片段、右片段和结果词。只有相邻物理行或中间恰好一个空行、词边界完整且映射唯一时
|
||||
才修改。增加新词不是自动学习行为，需要先批准新的项目映射；组件算法版本不变时，运行清单仍能通过参数区分实际语义。
|
||||
|
||||
## 5. 两个表格组件怎样共享范围
|
||||
|
||||
`_html_table.py` 只识别当前转换器输出的严格子集：`<tr>` 直接位于 `<table>` 下，`<td>` / `<th>` 直接位于
|
||||
`<tr>` 下，标签完整闭合，单元格中没有嵌套标签。它返回原字符串下标，不生成 DOM，也不重新渲染全文。
|
||||
|
||||
实体组件只处理单元格文本中的三个精确 token：
|
||||
|
||||
```text
|
||||
&lt; → <
|
||||
&gt; → >
|
||||
&amp; → &
|
||||
```
|
||||
|
||||
结果仍是合法 HTML 源码中的单层实体。标签、属性、表格外文本和其他实体不受影响。
|
||||
|
||||
布局组件只处理整个片段没有换行的严格表格。它原样复用 `<table>` 起始标签、每个完整 `<tr>...</tr>` 和结束标签，
|
||||
只增加外层换行与两个空格缩进。当前 9 张真实表格都有 `colspan`，所以全部保留 HTML；实现没有猜测表头，也没有转 GFM。
|
||||
|
||||
遇到未闭合、嵌套、额外结构标签或混合换行时，扫描失败关闭，保持原文。
|
||||
|
||||
## 6. 共享代码为什么仍然很小
|
||||
|
||||
- `_text_ranges.py` 只提供 Python 字符下标下的物理行、行尾和空行关系；
|
||||
- `_html_table.py` 只提供严格 HTML 表格、行和单元格范围；
|
||||
- 业务组件依赖这两个私有模块，但辅助模块不依赖组件、流水线或文件层；
|
||||
- 文件实验层只接收已经组装的 `Pipeline`，不知道任何识别规则。
|
||||
|
||||
因此以后放宽某个业务规则通常只改一个组件及其测试；替换 HTML 识别方式不会改变 `DocumentSnapshot`、
|
||||
`ProposedChange`、`Change` 或 JSON 产物；未来引入 Profile 时，也只接管当前脚本里的组件组装。
|
||||
|
||||
## 7. 当前验证结果和边界
|
||||
|
||||
2026-08-22 在 Python 3.13.11 环境完成：
|
||||
|
||||
- Ruff 通过；
|
||||
- mypy 检查 37 个文件无问题；
|
||||
- pytest 204 项通过；
|
||||
- 5 份本地论文全部 `success`，合计 155 条 `Change`；
|
||||
- 对 5 份成功输出再次运行，全部 `success` 且零修改；
|
||||
- 输入运行前后哈希不变；
|
||||
- JAMA Abstract 前内容不变，Springer 两条合法 arXiv 参考文献保留；
|
||||
- 图片引用文字不变，但实验产物没有复制图片资产。
|
||||
|
||||
保存型实验位于本机 Git 忽略的
|
||||
`artifacts/2026-08-22/runs/clindb-first-batch-v1/`。运行和复核方法见
|
||||
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。
|
||||
|
||||
这次成功只证明批准的 8 类规则在当前 5 份输入上闭环。截断、缺表、乱码、OCR 语义错误、图片资产、修订词选择和一般
|
||||
段落重排仍不在自动清洗范围内。
|
||||
@@ -1,155 +0,0 @@
|
||||
# 第一版内存清洗核心如何工作
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
清洗组件如果直接返回一整篇新 Markdown，调用方只能看到修改后的结果，很难确认它实际改了哪里。组件保存的
|
||||
旧位置还可能在文本变化后误中另一段内容；同一批修改发生重叠时，按不同顺序执行也可能得到不同结果。
|
||||
|
||||
当前核心把“判断应该改什么”和“安全地执行修改”分开：组件只描述绑定当前文本的精确修改，公共执行器统一
|
||||
验证并应用。核心建立时先不引入真实清洗规则、文件读写或 Markdown parser，让组合与审计协议独立可运行；
|
||||
现在第一个真实组件已经在这套协议上完成验证，没有改变核心接口。
|
||||
|
||||
已经实现的范围来自已批准的
|
||||
[`0003-first-executable-core-architecture.md`](../design/0003-first-executable-core-architecture.md)。精确类名、字段和
|
||||
函数签名以 [`src/mdpolish/`](../../src/mdpolish/) 中的代码和测试为准，本文不维护第二份 API 清单。
|
||||
|
||||
## 2. 当前数据流
|
||||
|
||||
```text
|
||||
输入 Markdown 字符串
|
||||
│
|
||||
▼
|
||||
带内容哈希的当前快照
|
||||
│
|
||||
▼
|
||||
组件提出精确修改 ──► 整批验证 ──► 整批应用 ──► 新快照
|
||||
│ │
|
||||
└──────── 按组件顺序重复 ◄──────────────┘
|
||||
│
|
||||
▼
|
||||
最终只重新提议，不再应用
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
success unstable failed
|
||||
```
|
||||
|
||||
核心只有四层：
|
||||
|
||||
| 层次 | 当前职责 | 明确不负责 |
|
||||
| --- | --- | --- |
|
||||
| 数据模型 | 保存快照、范围、候选修改、实际改动、错误和结果 | 业务规则和文件路径 |
|
||||
| 组件基类 | 声明身份、版本、参数、适用边界并提出修改 | 应用修改和组织流水线 |
|
||||
| 修改执行器 | 统一验证并原子应用一个组件批次 | 判断 Markdown 业务语义 |
|
||||
| 流水线 | 排列组件、刷新快照、处理失败并做最终复查 | 读取文件、选择项目 profile |
|
||||
|
||||
依赖保持单向：组件基类和修改执行器只依赖数据模型，流水线可以调用前三者，底层模块不反向调用流水线。
|
||||
|
||||
## 3. 为什么修改必须绑定快照
|
||||
|
||||
每个 Markdown 快照都带有根据完整字符串计算的 SHA-256。候选修改和其中每条文本编辑都必须指向这个哈希，
|
||||
还要同时提供原文范围和该范围预期出现的文字。
|
||||
|
||||
执行时会再次检查：
|
||||
|
||||
1. 哈希仍然对应当前快照；
|
||||
2. 范围没有越过字符串边界；
|
||||
3. 当前位置的文字与组件声明的预期原文完全一致；
|
||||
4. 替换后确实会改变内容。
|
||||
|
||||
任何一项不满足，当前组件的整个批次都不会执行。这使位置只在其产生时的快照内有效，不允许把旧候选修改悄悄
|
||||
套到后来变化的 Markdown 上。
|
||||
|
||||
范围使用 Python 字符串下标，而不是 UTF-8 字节位置。核心保留输入的换行、Unicode 形式和末尾换行，不做隐式
|
||||
规范化。
|
||||
|
||||
## 4. 组件为什么只提出自动修改
|
||||
|
||||
第一版组件只返回能够立即、唯一执行的候选修改。遇到不知道正确修法的截断、损坏表格、疑似幻觉或无法读取的
|
||||
图片时，组件应忽略，不猜测修复，也不额外生成“仅检查”结果。
|
||||
|
||||
每个组件必须提供稳定标识、`MAJOR.MINOR.PATCH` 版本、可冻结的参数和非空适用边界。适用边界需要由组件作者说明
|
||||
它处理什么结构、依赖哪些严格前置条件、明确排除什么。组件还必须满足确定、无副作用和幂等约束；不能读取文件、
|
||||
网络、环境变量、当前时间或随机数。
|
||||
|
||||
最终复查能发现多个组件组合后仍会继续提出修改，但不能从有限输入证明一个组件对所有文本都幂等。因此，幂等性
|
||||
既由最终复查保护当前运行，也必须由组件自己的针对性测试证明其适用范围内的行为。
|
||||
|
||||
独立检查和人工建议目前没有实现。以后只有出现明确消费者和闭环时，才通过新 design 增加平行接口，不在当前
|
||||
组件结果中补可空字段或状态枚举。
|
||||
|
||||
## 5. 一个组件批次如何保证原子性
|
||||
|
||||
一个组件可以提出多个候选修改，每个候选修改又可以包含多个文本编辑。执行器先验证该组件本次提出的全部编辑，
|
||||
只有整批通过才从后向前应用；任意一条失败，整批保持原样。这里的原子边界是“当前组件本次执行的全部修改”，
|
||||
不是单独一条编辑。
|
||||
|
||||
当前冲突规则有意保守：
|
||||
|
||||
| 两项编辑的关系 | 结果 |
|
||||
| --- | --- |
|
||||
| 两个非空范围真正重叠 | 冲突 |
|
||||
| 两个非空范围只相邻 | 允许 |
|
||||
| 两次插入位于同一点 | 冲突 |
|
||||
| 两次插入位于不同点 | 允许 |
|
||||
| 插入点位于非空范围内部、起点或终点 | 冲突 |
|
||||
|
||||
如果业务动作需要替换一段文字并在边界追加内容，组件应把它表达成同一条替换，而不是依赖编辑执行顺序。
|
||||
|
||||
## 6. 流水线状态代表什么
|
||||
|
||||
流水线先对所有组件做元数据预检，避免运行到一半才发现重复标识或无效版本。之后每个组件只执行一次，后一个组件
|
||||
只能读取前一个组件产生的新快照。清洗阶段出现异常、契约错误或编辑验证错误时会立即停止，且不再进行最终复查。
|
||||
|
||||
所有组件完成后，流水线让它们针对最终快照重新提出一次修改，但这一阶段只验证、不应用：
|
||||
|
||||
| 状态 | 含义 | Markdown 字段 |
|
||||
| --- | --- | --- |
|
||||
| `success` | 清洗和最终复查均完成，所选组件不再提出修改 | 只提供成功输出 |
|
||||
| `unstable` | 清洗无错误，但最终复查仍有有效候选修改 | 只提供诊断用部分文本和残留候选 |
|
||||
| `failed` | 清洗或最终复查发生错误 | 只提供诊断用部分文本和错误 |
|
||||
|
||||
最终复查是只读阶段，因此某个组件失败后仍会继续复查其余组件。错误与其他组件的有效残留修改可以同时保留，最终
|
||||
状态以 `failed` 为准。流水线不会因为 `unstable` 自动开始第二轮。
|
||||
|
||||
`success` 只表示本次选中的自动清洗组件已经稳定，不表示文档没有截断、幻觉、表格损坏、图片断链或其他未实现
|
||||
规则能够发现的问题。
|
||||
|
||||
## 7. 审计记录能回答什么
|
||||
|
||||
每条实际执行的文本编辑都会生成一条改动记录，说明：
|
||||
|
||||
- 是哪个组件、哪个版本和流水线位置执行的；
|
||||
- 属于哪个候选修改，以及在该候选修改中的编辑序号；
|
||||
- 组件给出的修改理由；
|
||||
- 修改前范围、原文和替换内容；
|
||||
- 当前组件批次修改前后的快照哈希。
|
||||
|
||||
同一候选修改中的多条记录共享候选引用，同一组件批次中的所有记录共享批次前后哈希。记录只描述已经发生的修改；
|
||||
验证失败或最终复查中没有执行的候选不会冒充实际改动。
|
||||
|
||||
这些内容在核心中只存在于内存返回值中。核心外已经有一个获批的本地实验 reporter，可以校验修改链并把审计、
|
||||
成功 Markdown 和 diff 保存到私有产物目录；机制见
|
||||
[`local-experiment-artifacts.md`](local-experiment-artifacts.md)。这没有改变核心接口，也不允许把失败结果中的部分文本
|
||||
写成正式输出或写回原文件。
|
||||
|
||||
## 8. 当前验证和剩余边界
|
||||
|
||||
核心测试继续使用短小的假组件，不包含或复制真实文档。它们覆盖空文本、中文和组合 Unicode、插入/删除/替换、
|
||||
范围冲突、过期哈希、批次原子性、组件连锁影响、错误阶段、审计关联和成功结果再次运行等行为。首个真实组件另用
|
||||
合成样例测试，并在本地真实材料上只读复核；机制与结果见
|
||||
[`arxiv-submission-stamp.md`](arxiv-submission-stamp.md)。
|
||||
|
||||
实际可用的安装与验收命令、最近一次验证日期和结果只在根目录
|
||||
[`README.md`](../../README.md#当前可用检查) 维护。
|
||||
|
||||
当前仍然没有：
|
||||
|
||||
- 除严格删除 arXiv 提交边栏戳外的其他论文、GovDoc 或 HTML 表格清洗组件；
|
||||
- 独立文档检查、人工建议或审核流程；
|
||||
- Markdown parser、AST 或共享业务中间表示；
|
||||
- 通用文件输入、公共 CLI、通用批处理、项目 profile 格式和生产集成；
|
||||
- 审计结果的长期存储、自动清理或脱敏输出协议。
|
||||
|
||||
当前只有一个固定数据和组件组合的本地实验脚本，不构成上述公共能力。这些边界中的任何一项要进入实现，都需要先用
|
||||
新的 design 明确语义、代价和验收方式。
|
||||
@@ -0,0 +1,97 @@
|
||||
# 函数式修改核心如何工作
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
项目规则如果直接返回一整篇新 Markdown，库无法确认它改了哪里，也无法在文本已经变化时阻止旧位置继续执行。
|
||||
`mdpolish` 把三个职责分开：项目规则提出修改，库验证并在内存中执行，调用项目决定是否写回文件。
|
||||
|
||||
当前机制来自已批准的
|
||||
[`0008-generic-functional-library-boundary.md`](../design/0008-generic-functional-library-boundary.md)。公共类名、字段和
|
||||
函数签名以 [`src/mdpolish/`](../../src/mdpolish/) 和测试为准，本文只解释不变量与边界。
|
||||
|
||||
## 2. 当前数据流
|
||||
|
||||
```text
|
||||
项目选择 Modifier、参数和顺序
|
||||
│
|
||||
▼
|
||||
当前 Markdown 快照
|
||||
│
|
||||
▼
|
||||
Modifier 提出精确修改 ──► 整批验证 ──► 内存中整批应用 ──► 新快照
|
||||
│ │
|
||||
└────────── 按项目给定顺序重复 ◄──────────┘
|
||||
│
|
||||
▼
|
||||
最终只复查，不再应用
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
success unstable failed
|
||||
```
|
||||
|
||||
| 层次 | 负责 | 不负责 |
|
||||
| --- | --- | --- |
|
||||
| 不可变数据模型 | 保存快照、范围、候选修改、实际改动、错误和结果 | 项目业务判断 |
|
||||
| `Modifier` | 冻结身份、版本、参数、适用边界和提议函数 | 直接改变字符串或文件 |
|
||||
| 修改执行器 | 校验并原子应用一个修改器批次 | 判断规则是否符合某个项目 |
|
||||
| `Pipeline` | 按显式顺序运行修改器并做最终稳定性复查 | 自动选规则、文件读写和默认组合 |
|
||||
|
||||
`mdpolish` 不导入使用项目。项目可以使用正则工厂、通用内置修改器，也可以用普通函数建立自己的 `Modifier`。
|
||||
|
||||
## 3. 修改器拥有什么权限
|
||||
|
||||
修改函数接收不可变 `DocumentSnapshot`，返回一个 `ProposedChange` 元组。每项候选修改说明原因，并包含一条或多条
|
||||
精确 `TextEdit`。修改器拥有规则判断权，可以提议插入、删除或替换，但不能通过公共契约原地改变快照，也不能用
|
||||
整篇新文本绕过执行器。
|
||||
|
||||
`Modifier` 还保存稳定 ID、语义版本、冻结后的参数和适用边界。普通函数无需继承基类。相同输入、身份、版本和参数
|
||||
应产生相同顺序的候选修改；修改函数不得读取文件、网络、环境变量、当前时间或随机数。
|
||||
|
||||
Python 不能沙箱隔离任意调用方函数。外部函数若私下写文件，属于绕过库契约的副作用，不在 `mdpolish` 的验证、
|
||||
审计和回滚保证内。
|
||||
|
||||
## 4. 为什么修改必须绑定快照
|
||||
|
||||
每个快照都带有完整 Markdown 的 SHA-256。候选修改及其中每条编辑必须绑定这个哈希，并提供半开字符串范围、该范围
|
||||
应有的原文和替换文本。执行器再次确认：
|
||||
|
||||
1. 哈希对应当前快照；
|
||||
2. 范围没有越界；
|
||||
3. 当前位置与预期原文完全一致；
|
||||
4. 编辑不是无变化操作；
|
||||
5. 当前修改器批次没有重复或冲突范围。
|
||||
|
||||
任意检查失败，当前修改器的整批候选都不执行。范围使用 Python 字符串索引，不是 UTF-8 字节位置；核心不会隐式
|
||||
改变换行、Unicode 形式或末尾换行。
|
||||
|
||||
执行器从文本后方向前应用编辑，避免前面的修改使后面的下标失效；审计记录仍按原文位置排列。
|
||||
|
||||
## 5. `Pipeline` 的状态
|
||||
|
||||
`Pipeline` 先预检全部修改器及重复 ID，再按调用方顺序各运行一次。后一个修改器读取前一个修改器生成的新快照。
|
||||
全部执行完成后，每个修改器对最终快照再提议一次，但复查阶段只验证，不应用，也不会自动开始第二轮。
|
||||
|
||||
| 状态 | 含义 | 文本字段 |
|
||||
| --- | --- | --- |
|
||||
| `success` | 运行和复查均完成，所选修改器不再提出修改 | `output_markdown` |
|
||||
| `unstable` | 运行无错误，但最终快照仍有有效候选修改 | `partial_markdown` |
|
||||
| `failed` | 提议、契约或执行验证发生错误 | `partial_markdown` |
|
||||
|
||||
`success` 只说明调用方选择的这组修改器在这次输入上已经稳定，不说明文档不存在其他质量问题。库只返回内存结果；
|
||||
调用项目检查状态后，自己决定是否保存。
|
||||
|
||||
## 6. 当前通用能力与边界
|
||||
|
||||
除了核心，发布包只提供：
|
||||
|
||||
- `regex_replace()`：把非空正则匹配转换为精确编辑；
|
||||
- `mapped_line_join()`：按调用方映射合并跨行片段，库不附带词表；
|
||||
- `html_table_entity_unescape()`：在严格表格单元格文本中解除一层受支持的实体转义；
|
||||
- `html_table_layout()`：把严格单行 HTML 表格展开为每行一个表格行。
|
||||
|
||||
HTML 能力使用失败关闭的词法子集，不是完整 HTML parser，也不识别 Markdown 围栏。正则工厂只保证定位和执行契约，
|
||||
不保证调用方正则的业务语义正确。
|
||||
|
||||
当前没有默认流水线、文件适配器、CLI、profile、配置加载、批处理、artifact、评审器或项目规则集。安装或导入库不会
|
||||
自动修改任何文本。实际安装、示例和当前检查命令只以根目录 [`README.md`](../../README.md) 为准。
|
||||
@@ -1,195 +0,0 @@
|
||||
# 本地清洗实验如何保存 Markdown、审计和 diff
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
内存流水线可以安全地产生 `TransformResult`，但进程结束后，评审者仍需要打开清洗后的完整 Markdown、查看总 diff，
|
||||
并追溯每条修改属于哪个组件、为什么修改、修改前后是什么。
|
||||
|
||||
当前本地实验层把这些结果保存到独立目录，同时继续保持三个边界：
|
||||
|
||||
- 组件和 `Pipeline` 仍然不读写文件；
|
||||
- 输入文件永远不被覆盖；
|
||||
- 只有 `success` 文档才产生正式的清洗后 Markdown。
|
||||
|
||||
已经实现的范围来自已批准的
|
||||
[`0005-local-experiment-runner-and-artifacts.md`](../design/0005-local-experiment-runner-and-artifacts.md) 和
|
||||
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。
|
||||
精确字段、校验和函数签名以 `src/mdpolish/` 中的代码与测试为准。
|
||||
|
||||
## 2. 保存与评审怎样解耦
|
||||
|
||||
```text
|
||||
experiment.py
|
||||
├── pipeline.py 只负责内存清洗
|
||||
├── reporting.py 只负责 JSON、行列和 unified diff
|
||||
└── artifact_store.py 只负责日期目录、权限和原子发布
|
||||
│
|
||||
▼
|
||||
已发布运行目录
|
||||
│
|
||||
▼
|
||||
reviewer/ 只通过发布后的文件做本地只读评审
|
||||
```
|
||||
|
||||
- `experiment.py` 严格读取调用方显式列出的 UTF-8 Markdown，逐份调用同一个 `Pipeline`；
|
||||
- `reporting.py` 重放并校验 `Change` 的快照链，再生成机器可读审计和人可读 diff；
|
||||
- `artifact_store.py` 不理解清洗规则，只把已经生成的字节写入私有临时目录，校验后一次性发布。
|
||||
- 同仓库 `reviewer/server/` 只共用无文件 I/O 的 Python 快照重放模块，不导入组件或流水线；它读取 manifest、result、
|
||||
成功输出和本机定位文件。
|
||||
|
||||
因此，新增组件不会改变文件层；调整目录布局不会影响清洗和报告；修改 JSON 或 diff 时也不需要碰流水线。
|
||||
ClinDB 的 5 份论文、历史 arXiv 单组件组合和当前 first-batch 组合只存在于两个仓库内实验脚本，通用模块没有硬编码
|
||||
论文名或业务组件。
|
||||
|
||||
## 3. 输入怎样保持原样
|
||||
|
||||
实验层先完成整批预检：
|
||||
|
||||
1. 验证日期、运行 ID、文档 ID 和来源标签；
|
||||
2. 确认所有路径存在、是普通文件且没有重复；
|
||||
3. 以二进制读取全部输入；
|
||||
4. 使用严格 UTF-8 解码；
|
||||
5. 比较原始字节 SHA-256 与核心 Markdown SHA-256。
|
||||
|
||||
读取过程不剔除 BOM，不规范化 Unicode，不转换 `\n`、`\r\n` 或文件末尾换行。全部文档运行结束后，
|
||||
实验层再次读取每份输入并比较原始字节和哈希；任何变化都会阻止产物发布。
|
||||
|
||||
输入缺失、不是普通文件、不是有效 UTF-8 或清单冲突属于整批预检失败。这时不调用流水线，也不创建最终运行目录。
|
||||
|
||||
## 4. 状态怎样决定产物
|
||||
|
||||
每份文档独立运行，某一份发生核心错误不会阻止其他已经预检的文档继续产生结果。
|
||||
|
||||
| 文档状态 | `result.json` | `cleaned.md` | `changes.diff` |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | 有 | 有 | 有；零修改时为空文件 |
|
||||
| `failed` | 有 | 无 | 无 |
|
||||
| `unstable` | 有 | 无 | 无 |
|
||||
|
||||
`failed` 和 `unstable` 的审计仍保留已经实际发生的 `Change`、错误或残留候选，但不持久化
|
||||
`partial_markdown`，避免半成品看起来像正式结果。
|
||||
|
||||
整批状态按 `failed`、`unstable`、`success` 的优先级汇总。即使其他文档有成功产物，只要一份失败，
|
||||
`manifest.json` 就会把整批标为 `failed`。
|
||||
|
||||
## 5. 产物怎样组织
|
||||
|
||||
```text
|
||||
artifacts/
|
||||
└── <YYYY-MM-DD>/
|
||||
└── runs/
|
||||
└── <run_id>/
|
||||
├── manifest.json
|
||||
├── review-locator.json
|
||||
└── documents/
|
||||
└── <document_id>/
|
||||
├── result.json
|
||||
├── cleaned.md
|
||||
└── changes.diff
|
||||
```
|
||||
|
||||
日期取实验启动时本机时区中的日历日期。运行 ID 由调用方显式提供；同一日期下已经存在同名目录时拒绝覆盖。
|
||||
|
||||
`manifest.json` 是整次运行的索引，记录：
|
||||
|
||||
- 开始、完成和到期时间；
|
||||
- 本机日期及 UTC 偏移；
|
||||
- mdpolish、Python、平台和 Git 状态；
|
||||
- 实际组件顺序、版本和参数；
|
||||
- 每份输入的来源标签、前后哈希、状态、修改数量和产物相对路径；
|
||||
- 整批成功、失败、不稳定和修改数量。
|
||||
|
||||
每份 `result.json` 保存实际修改、错误与残留候选。每条修改包含组件、候选引用、理由、Python 字符范围、
|
||||
1-based 行列、`before`、`after` 和批次前后哈希。完整成功文本只存在于 `cleaned.md`。
|
||||
|
||||
`changes.diff` 是原始输入到最终成功输出的 unified diff，只用于人工查看。它不包含绝对路径或时间戳，
|
||||
也不是修改重放的权威；机器审计仍以 `result.json` 为准。
|
||||
|
||||
`review-locator.json` 只记录本次运行目录、每份输入的绝对解析路径和输入哈希，供本地评审器重新找到完整原文。它不复制
|
||||
原文，不替代 manifest 或 result，也不作为可移植运行身份。绝对路径可能泄露本机目录结构，因此该文件同样是本地敏感数据。
|
||||
|
||||
## 6. 为什么 reporter 要重放修改
|
||||
|
||||
第二个组件看到的是第一个组件修改后的快照，因此后续 `Change.span` 不一定对应最初输入。为了生成准确行列，
|
||||
reporter 从输入开始，按组件批次重放修改：
|
||||
|
||||
1. 当前文本哈希必须等于该批次 `before_sha256`；
|
||||
2. 每条范围内的原文必须等于 `before`；
|
||||
3. 同一批次按核心相同的从后向前顺序应用；
|
||||
4. 结果哈希必须等于 `after_sha256`；
|
||||
5. 全部批次完成后必须等于 `TransformResult.current_sha256`。
|
||||
|
||||
任何一步不一致都说明内存结果、reporter 或调用方式违反契约，整次运行不会发布最终目录。行列只是方便人查看的
|
||||
派生信息，修改权威仍是快照绑定的 Python 字符范围。
|
||||
|
||||
## 7. 文件怎样安全发布
|
||||
|
||||
artifact store 先在同一日期的 `runs/` 下建立本次专用临时目录。所有文件写入后都会重新读取校验，
|
||||
成功 Markdown 还要再次核对输出 SHA-256，manifest 的身份、状态、计数和路径也必须与各文档审计及实际文件一致。
|
||||
只有全部文件、清单和权限都通过，临时目录才会在 `runs/` 目录协作锁内重新检查目标，并原子重命名为最终运行 ID。
|
||||
|
||||
当前 Linux 本地实现使用：
|
||||
|
||||
- 目录权限 `0700`；
|
||||
- 文件权限 `0600`；
|
||||
- 已存在的目标目录拒绝覆盖；
|
||||
- 写入中途失败时不发布最终目录。
|
||||
|
||||
这保证不会发布已知不完整的结果，但不承诺跨平台断电耐久性或网络文件系统语义。
|
||||
|
||||
## 8. 隐私和保留边界
|
||||
|
||||
`cleaned.md`、diff 和 JSON 审计都可能包含真实原文，因此整个 `artifacts/` 都是本地敏感数据：
|
||||
|
||||
- 受 Git 忽略；
|
||||
- 不进入 Wiki、提交、推送或外部系统；
|
||||
- 终端只显示状态、计数和目录；
|
||||
- 默认保留 30 个日历日；
|
||||
- manifest 记录 `retention_until`；
|
||||
- 第一版不自动删除，到期后仍需用户确认具体目录再清理。
|
||||
|
||||
当前只批准对 `data/md/` 中 5 份论文副本保存产物。仓库外 GovDoc 和其他真实数据没有因此获得输出授权。
|
||||
|
||||
同仓库只读页面的路径验证、组件快照重放和使用边界见
|
||||
[`local-markdown-reviewer.md`](local-markdown-reviewer.md)。
|
||||
|
||||
## 9. 已完成的真实验证
|
||||
|
||||
2026-08-22 使用 `paper.arxiv_submission_stamp` `1.0.0` 对 5 份本地论文副本完成一次保存型实验：
|
||||
|
||||
| 项目 | 结果 |
|
||||
| --- | --- |
|
||||
| 运行 ID | `clindb-arxiv-stamp-artifacts-v1` |
|
||||
| 输出位置 | `artifacts/2026-08-22/runs/clindb-arxiv-stamp-artifacts-v1/` |
|
||||
| 文档状态 | 5/5 `success` |
|
||||
| 实际修改 | `sim` 1 条、`springer` 1 条，其余 0 条 |
|
||||
| 修改位置 | `sim` 1:1、`springer` 18:1 |
|
||||
| 合法反向样例 | Springer 两处 `arXiv preprint arXiv:` 均保留 |
|
||||
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||
| 权限 | 全部运行目录 `0700`，产物文件 `0600` |
|
||||
|
||||
实际运行方法见
|
||||
[`run-local-clindb-arxiv-experiment.md`](../guides/run-local-clindb-arxiv-experiment.md)。
|
||||
|
||||
同日又使用 `design/0006` 的 8 组件流水线完成 ClinDB 第一批保存型实验：
|
||||
|
||||
| 项目 | 结果 |
|
||||
| --- | --- |
|
||||
| 运行 ID | `clindb-first-batch-v1` |
|
||||
| 输出位置 | `artifacts/2026-08-22/runs/clindb-first-batch-v1/` |
|
||||
| 文档状态 | 5/5 `success` |
|
||||
| 实际修改 | dmp 47、ejhf 9、jama 79、sim 3、springer 17，共 155 条 |
|
||||
| 第二次运行 | 5/5 `success`，合计 0 条修改 |
|
||||
| 输出校验 | 5/5 `cleaned.md` 与 `current_sha256` 一致 |
|
||||
| 输入只读 | 5/5 运行前后字节和哈希不变 |
|
||||
| 内容反例 | JAMA Abstract 前内容不变；Springer 两条合法 arXiv 引用保留；图片引用文字不变 |
|
||||
| 权限 | 运行目录 `0700`，产物文件 `0600` |
|
||||
|
||||
当前完整运行方法见
|
||||
[`run-local-clindb-first-batch-experiment.md`](../guides/run-local-clindb-first-batch-experiment.md)。实验层的文件、
|
||||
JSON、diff 和权限契约没有因组件增多而改变。
|
||||
|
||||
2026-08-23 又产生运行 `clindb-first-batch-reviewer-v1`，用于验证新定位文件和本地页面：5/5 文档为 `success`，合计
|
||||
155 条修改，输入运行前后哈希不变。评审器 API 校验了 5 份原文、成功输出以及 8 个组件形成的 40 个阶段，所有文本哈希
|
||||
均与审计一致。实际页面启动步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||
@@ -1,109 +0,0 @@
|
||||
# 本地 Markdown 清洗评审器如何保持只读和可追踪
|
||||
|
||||
## 1. 它解决什么问题
|
||||
|
||||
本地清洗实验已经保存最终 Markdown、逐条审计和 unified diff，但人工评审仍需要在多个文件之间切换，也看不到某个组件
|
||||
执行前后的完整文本。当前评审器把一次已发布运行变成只读页面：主视图比较原文和最终成功输出，组件时间线则比较每个
|
||||
组件实际收到的快照和它产生的新快照。
|
||||
|
||||
实现范围来自已批准的
|
||||
[`0007-local-markdown-reviewer.md`](../design/0007-local-markdown-reviewer.md)。精确 API、字段和运行行为以
|
||||
[`reviewer/`](../../reviewer/) 中的代码、类型和测试为准。
|
||||
|
||||
## 2. 同仓库怎样保持解耦
|
||||
|
||||
```text
|
||||
mdpolish 本地实验层
|
||||
│
|
||||
▼
|
||||
manifest / result / cleaned / review-locator
|
||||
│
|
||||
▼
|
||||
Python 产物适配器与共享重放 ──► 本地只读 API ──► React 页面
|
||||
```
|
||||
|
||||
- `src/mdpolish/` 不导入 `reviewer/`；
|
||||
- `reviewer/server/` 只从 `mdpolish` 导入无文件 I/O 的 `_artifact_replay.py`，不导入组件、流水线或实验入口；
|
||||
- 浏览器只读取 `/api/v1/`，不解析磁盘 JSON，也不知道绝对路径；
|
||||
- Python wheel 不包含前端代码或 Node.js 依赖；
|
||||
- 产物 schema 以后改变时，差异集中在服务端版本适配器，不扩散到页面组件。
|
||||
|
||||
前端与核心位于同一 Git 仓库，方便开发和评审，但仍是独立的 Node.js package。依赖版本只在
|
||||
[`reviewer/package.json`](../../reviewer/package.json) 和锁文件维护。
|
||||
|
||||
## 3. 运行定位文件保存什么
|
||||
|
||||
新实验会在运行目录根部原子保存 `review-locator.json`。它记录：
|
||||
|
||||
- 运行 ID 和发布时的绝对运行目录；
|
||||
- `manifest.json` 的固定相对位置；
|
||||
- 每份文档的 ID、实际读取的绝对源路径和输入 SHA-256。
|
||||
|
||||
定位文件不复制原文，也不替代 manifest 或 result。评审器由用户显式指定当前运行目录；记录的旧运行目录只用于判断目录
|
||||
是否被移动。服务只根据定位文件读取对应原文，并在每次展示前重新计算哈希。源文件不存在或内容变化时，页面明确报告
|
||||
不可用，不按名称搜索替代文件。
|
||||
|
||||
绝对路径会暴露本机目录结构，所以定位文件与其他 artifact 一样使用 `0600` 权限并按本地敏感数据处理。历史运行没有该
|
||||
文件时仍可查看清单和局部审计，但不能自动展示完整原文；评审器不会回写历史目录。
|
||||
|
||||
## 4. 服务为什么只读取一次运行
|
||||
|
||||
启动时必须传入一个具体运行目录。服务不会扫描 `artifacts/`，也没有让浏览器传入任意文件路径的 API。它先校验：
|
||||
|
||||
1. manifest、result 和存在的 locator 都是支持的 schema、严格 UTF-8 JSON；
|
||||
2. 文档、组件、状态、路径和计数彼此一致；
|
||||
3. 所有 artifact 路径都留在所选运行目录；
|
||||
4. 原文和成功输出的字节哈希与审计一致；
|
||||
5. 成功文档确实同时具有 `cleaned.md` 和 `changes.diff`。
|
||||
|
||||
HTTP 只监听 `127.0.0.1` 的随机空闲端口，只接受 `GET` 和 `HEAD`。服务拒绝非本机 Host、跨域 Origin、路径穿越和
|
||||
写请求，不提供删除、移动、重新清洗或 shell 执行能力。响应禁止缓存，不开放 CORS，也不向浏览器返回绝对源路径。
|
||||
|
||||
## 5. 组件阶段怎样准确重放
|
||||
|
||||
`Change.span` 使用 Python Unicode 码点位置，而 JavaScript 编辑器使用 UTF-16 code unit。共享 Python 重放模块直接按
|
||||
原生码点范围逐组件处理，不让浏览器应用修改：
|
||||
|
||||
1. 当前完整文本哈希必须等于组件批次的 `before_sha256`；
|
||||
2. 每条范围内文本必须等于 `before`；
|
||||
3. 同一批次不得有冲突范围，并按位置从后向前应用；
|
||||
4. 应用后完整文本哈希必须等于 `after_sha256`；
|
||||
5. 全部组件结束后必须逐字等于 `cleaned.md`；
|
||||
6. 服务端另外从已验证的组件前快照派生 UTF-16 `editor_range`，只供 CodeMirror 跳转。
|
||||
|
||||
组件没有修改时，阶段前后文本和哈希相同，但该组件仍显示在时间线中。包含中文、emoji、组合字符、BOM、CRLF 和无末尾
|
||||
换行的合成测试用于保护跨语言坐标。任一重放校验失败时，页面拒绝显示组件阶段，不通过搜索或 diff 猜测位置。
|
||||
|
||||
中间快照只在服务内存中按需生成，不保存新的 Markdown 文件。`failed` 和 `unstable` 文档只显示错误、残留候选和已有的
|
||||
局部审计，不重建一份看似正式的部分输出。
|
||||
|
||||
## 6. 页面当前能看什么
|
||||
|
||||
页面当前提供：
|
||||
|
||||
- 运行状态、文档状态、哈希和实际 `Change` 数量；
|
||||
- 完整原文与最终成功 Markdown 的只读双栏源码比较；
|
||||
- 8 个组件的实际顺序、版本和每份文档修改数量；
|
||||
- 任一组件执行前后的完整文本比较；
|
||||
- 修改理由、派生行列、`before` / `after` 和同候选修改关联；
|
||||
- `failed`、`unstable`、路径失效、哈希变化和未知 schema 的独立错误状态。
|
||||
|
||||
Markdown 只作为文本交给 CodeMirror，不进入 `innerHTML`。第一版不渲染 Markdown、HTML 或图片，不加载 CDN、远程字体、
|
||||
遥测和其他外部资源，也不提供编辑、审核或回写。
|
||||
|
||||
## 7. 当前验证结果和边界
|
||||
|
||||
2026-08-24 使用 Python 3.13.11 和当前用户 nvm 中的 Node.js 24.19.0 完成：
|
||||
|
||||
- Python Ruff、mypy 和 229 项 pytest 通过；
|
||||
- reviewer ESLint、TypeScript、9 项 Vitest 和生产构建通过；
|
||||
- 新运行 `clindb-first-batch-reviewer-v1` 的 5 份论文全部 `success`，共 155 条实际修改；
|
||||
- 5 份原文运行前后哈希不变；
|
||||
- 本地 API 成功校验 5 份文档、8 个组件和 40 个组件阶段；
|
||||
- 所有原文、成功输出和阶段前后文本的 SHA-256 与运行审计一致；
|
||||
- 生产页面和全部本地构建资源可以通过只读服务读取，响应没有 CORS 并包含禁止缓存和内容类型保护头。
|
||||
|
||||
当前环境没有可用于自动视觉检查的本地浏览器，因此布局的真实浏览器视觉效果尚未验证。当前结果证明构建、服务、数据
|
||||
重放和主要 React 状态可以运行，不等于已经完成跨浏览器、极端长度、渲染预览或生产部署验证。
|
||||
|
||||
实际启动与评审步骤见 [`review-local-cleaning-run.md`](../guides/review-local-cleaning-run.md)。
|
||||
@@ -1,127 +0,0 @@
|
||||
# 使用本地页面评审一次 Markdown 清洗运行
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南用于打开已经发布在 `artifacts/<YYYY-MM-DD>/runs/<run_id>/` 的本地清洗运行。完整双栏比较要求运行目录包含
|
||||
`review-locator.json`，并且原文仍位于运行时记录的位置且哈希未改变。
|
||||
|
||||
评审器只读文件，不重新运行组件、不修改原文和产物。当前不适用于 GovDoc、远程目录、多人共享或生产部署。
|
||||
|
||||
本指南于 2026-08-23 使用 Python 3.13.11、当前用户 nvm 中的 Node.js 24.19.0 和运行
|
||||
`clindb-first-batch-reviewer-v1` 实际验证。
|
||||
|
||||
## 2. 准备一次可评审运行
|
||||
|
||||
先按 [`run-local-clindb-first-batch-experiment.md`](run-local-clindb-first-batch-experiment.md) 产生一次新的运行。成功终端摘要
|
||||
会给出绝对 artifact 路径，例如：
|
||||
|
||||
```text
|
||||
artifacts=/home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||
```
|
||||
|
||||
确认该目录内存在：
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
review-locator.json
|
||||
documents/
|
||||
```
|
||||
|
||||
不要编辑定位文件，也不要向旧运行目录手工补写它。历史运行缺少 locator 时，使用不同运行 ID 重新实验。
|
||||
|
||||
## 3. 准备评审器
|
||||
|
||||
评审器前端的安装、检查和构建要求 Node.js 24 LTS。当前用户 nvm 已安装与 `.nvmrc` 匹配的版本。在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
cd reviewer
|
||||
nvm use
|
||||
node --version
|
||||
npm --version
|
||||
```
|
||||
|
||||
`node --version` 必须是受支持的 `v24`。本仓库不负责修改系统级 Node.js；版本不符时先在开发环境外准备正确运行时。
|
||||
|
||||
首次安装或锁文件变化后，仍在 `reviewer/` 目录执行：
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
```
|
||||
|
||||
当前有效的 Python 与 reviewer 检查命令只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后构建页面：
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
## 4. 启动一次运行
|
||||
|
||||
回到仓库根目录，用当前 Python 虚拟环境启动只读服务并传入运行目录：
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m reviewer.server \
|
||||
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id>
|
||||
```
|
||||
|
||||
启动成功时只打印运行 ID、文档数量和随机本机端口，不打印原文或绝对源路径：
|
||||
|
||||
```text
|
||||
mdpolish 评审器已启动：http://127.0.0.1:<port>（<run_id>，<count> 份文档）
|
||||
```
|
||||
|
||||
在本机浏览器打开该地址。评审结束后回到终端按 `Ctrl+C` 停止服务。
|
||||
|
||||
开发页面时开两个终端。第一个终端在仓库根目录把 Python API 固定到 Vite 代理使用的本机端口：
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m reviewer.server \
|
||||
--run-dir /home/lihaoze/work/mdpolish/artifacts/<YYYY-MM-DD>/runs/<run_id> \
|
||||
--port 4174
|
||||
```
|
||||
|
||||
第二个终端启动只绑定 `127.0.0.1:5173` 的 Vite 页面；它只把 `/api/` 代理给上述 Python 服务，Node.js 不读取 artifact：
|
||||
|
||||
```bash
|
||||
cd reviewer
|
||||
npm run dev
|
||||
```
|
||||
|
||||
## 5. 页面怎么查看
|
||||
|
||||
1. 先确认顶部整体状态和总修改数与 `manifest.json` 一致；
|
||||
2. 在左侧选择文档，主双栏默认显示清洗前和最终成功输出；
|
||||
3. 在组件时间线选择一个组件，双栏切换为该组件执行前后；
|
||||
4. 检查组件版本和修改数，零修改应显示 `0`，而不是从时间线消失；
|
||||
5. 点击修改详情，跳到对应组件阶段的位置并核对理由、`before` 和 `after`；
|
||||
6. 对 `failed` / `unstable` 只查看错误和残留候选，不寻找不存在的正式输出。
|
||||
|
||||
页面中的总修改数是实际 `Change` 条数，不是 diff hunk 数、字符数或问题数量。
|
||||
|
||||
## 6. 常见错误
|
||||
|
||||
### 原文路径失效或哈希改变
|
||||
|
||||
评审器不会搜索同名文件。确认输入没有被移动或修改；如果需要在新位置运行，使用新的运行 ID 重新执行实验。不要改 locator
|
||||
绕过哈希检查。
|
||||
|
||||
### 历史运行没有 `review-locator.json`
|
||||
|
||||
历史产物仍可在页面查看清单和已有审计，也可人工查看 manifest、result 和 diff，但第一版页面不能自动找到完整原文或
|
||||
组件阶段。不要回写历史目录；需要完整双栏时重新运行一次即可。
|
||||
|
||||
### 不支持 schema
|
||||
|
||||
评审器只支持当前文档列出的 schema 版本。不要删除或伪造 `schema_version`；应升级评审器适配器或使用与产物匹配的代码。
|
||||
|
||||
### 没有 `cleaned.md`
|
||||
|
||||
对应文档状态是 `failed` 或 `unstable` 时这是正常边界。页面不会从 Change 重建并冒充正式结果。
|
||||
|
||||
### 服务拒绝 Host、Origin 或写请求
|
||||
|
||||
评审器只接受本机同源的只读请求。不要通过反向代理、远程端口转发或网页跨域调用它；这些用法没有批准。
|
||||
|
||||
## 7. 数据边界
|
||||
|
||||
页面会在本机内存中读取完整原文和成功输出。不要截图、复制或通过浏览器扩展分享真实内容。运行目录继续受 Git 忽略并按
|
||||
manifest 的 `retention_until` 管理；页面不会自动删除到期产物。
|
||||
@@ -1,129 +0,0 @@
|
||||
# 运行本地 ClinDB arXiv 清洗实验
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南只运行仓库内已经批准的本地实验脚本：
|
||||
|
||||
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||
- 流水线：只包含 `paper.arxiv_submission_stamp` `1.0.0`；
|
||||
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||
- 输入只读，不覆盖原文件；
|
||||
- 不处理 `/home/lihaoze/gov_test_data`。
|
||||
|
||||
本指南于 2026-08-22 在 Python 3.13.11 环境实际验证。
|
||||
|
||||
## 2. 前置条件
|
||||
|
||||
在仓库根目录执行，并确认隔离环境和基础检查可用：
|
||||
|
||||
```bash
|
||||
.venv/bin/python --version
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/pytest
|
||||
```
|
||||
|
||||
确认 5 份本地输入存在：
|
||||
|
||||
```bash
|
||||
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||
```
|
||||
|
||||
预期看到：
|
||||
|
||||
```text
|
||||
dmp.md
|
||||
ejhf.md
|
||||
jama.md
|
||||
sim.md
|
||||
springer.md
|
||||
```
|
||||
|
||||
## 3. 运行实验
|
||||
|
||||
为本次实验人工选择一个小写运行 ID。同一天已经使用过的 ID 不能覆盖；需要重跑时换一个新 ID。
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/run_clindb_arxiv_experiment.py \
|
||||
--run-id clindb-arxiv-stamp-review
|
||||
```
|
||||
|
||||
成功时终端只显示运行 ID、状态、文档数、修改数和产物目录，例如：
|
||||
|
||||
```text
|
||||
run_id=clindb-arxiv-stamp-review
|
||||
status=success
|
||||
documents=5
|
||||
changes=2
|
||||
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-arxiv-stamp-review
|
||||
```
|
||||
|
||||
终端不会打印论文原文或 diff。
|
||||
|
||||
## 4. 查看结果
|
||||
|
||||
进入终端输出的运行目录。目录结构为：
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
documents/
|
||||
├── dmp/
|
||||
│ ├── result.json
|
||||
│ ├── cleaned.md
|
||||
│ └── changes.diff
|
||||
├── ejhf/
|
||||
├── jama/
|
||||
├── sim/
|
||||
└── springer/
|
||||
```
|
||||
|
||||
先看 `manifest.json` 的整批状态和汇总，再查看各文档：
|
||||
|
||||
- `cleaned.md`：成功清洗后的完整 Markdown；
|
||||
- `changes.diff`：输入到成功输出的人工对比；
|
||||
- `result.json`：组件、理由、位置、`before`、`after` 和哈希等机器审计。
|
||||
|
||||
`dmp`、`ejhf` 和 `jama` 当前应为零修改，diff 是空文件；`sim` 和 `springer` 当前各有一条删除。
|
||||
|
||||
## 5. 判断成功
|
||||
|
||||
本轮验收口径是：
|
||||
|
||||
- manifest 整体状态为 `success`；
|
||||
- 5 份文档全部为 `success`；
|
||||
- 合计 2 条修改；
|
||||
- `sim` 修改位置为 1:1；
|
||||
- `springer` 修改位置为 18:1；
|
||||
- Springer 两条合法 `arXiv preprint arXiv:` 参考文献保留；
|
||||
- 每份 `cleaned.md` 的 SHA-256 等于对应 `result.json.current_sha256`；
|
||||
- 输入文件运行前后不变。
|
||||
|
||||
只看到运行目录存在不等于成功，必须先检查 manifest 和文档状态。
|
||||
|
||||
## 6. 常见失败
|
||||
|
||||
### 运行目录已经存在
|
||||
|
||||
脚本拒绝覆盖同一日期下的同名运行目录。选择新的 `--run-id`，不要删除或覆盖旧目录来绕过检查。
|
||||
|
||||
### 输入缺失或不是 UTF-8
|
||||
|
||||
整批预检会失败，不运行任何组件，也不发布最终目录。先确认 `data/md/` 中 5 份文件存在且未被修改。
|
||||
|
||||
### 状态为 `failed` 或 `unstable`
|
||||
|
||||
对应文档只会生成 `result.json`，不会生成 `cleaned.md` 或 diff。查看错误或残留候选，不要把其他文档的部分成功
|
||||
当成整批成功。
|
||||
|
||||
### Snap 版本的 `jq` 报权限错误
|
||||
|
||||
产物目录权限是 `0700`。某些 Snap 沙箱工具不能进入私有目录，即使当前用户拥有权限。可直接用编辑器查看 JSON，
|
||||
或使用当前虚拟环境中的 Python 读取；不要为了兼容受限工具放宽产物权限。
|
||||
|
||||
## 7. 数据边界
|
||||
|
||||
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||
其他仓库、云存储或外部系统。
|
||||
|
||||
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。第一版不会自动删除；到期后如需清理，必须先确认
|
||||
具体运行目录。
|
||||
@@ -1,150 +0,0 @@
|
||||
# 运行本地 ClinDB 第一批完整清洗实验
|
||||
|
||||
## 1. 适用范围
|
||||
|
||||
本指南只运行仓库内已经批准的 first-batch 实验脚本：
|
||||
|
||||
- 输入：`data/md/` 中的 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`；
|
||||
- 流水线：`design/0006` 固定的 8 个组件和顺序；
|
||||
- 输出：`artifacts/<YYYY-MM-DD>/runs/<run_id>/`；
|
||||
- 输入只读，不覆盖原文件；
|
||||
- 不读取或复制图片，不处理 `/home/lihaoze/gov_test_data`。
|
||||
|
||||
本指南于 2026-08-23 在 Python 3.13.11 环境实际验证。
|
||||
|
||||
## 2. 前置检查
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
.venv/bin/python --version
|
||||
.venv/bin/ruff check .
|
||||
.venv/bin/mypy src tests scripts/run_clindb_arxiv_experiment.py scripts/run_clindb_first_batch_experiment.py
|
||||
.venv/bin/pytest
|
||||
diff -u <(tail -n +2 AGENTS.md) <(tail -n +2 CLAUDE.md)
|
||||
```
|
||||
|
||||
当前检查结果只以根目录 [`README.md`](../../README.md#当前可用检查) 为准。检查通过后再确认 5 份输入存在：
|
||||
|
||||
```bash
|
||||
find data/md -maxdepth 1 -type f -name '*.md' -printf '%f\n' | sort
|
||||
```
|
||||
|
||||
必须看到 `dmp.md`、`ejhf.md`、`jama.md`、`sim.md` 和 `springer.md`。不要把真实论文复制进测试 fixture。
|
||||
|
||||
## 3. 运行实验
|
||||
|
||||
人工选择一个当天未使用的安全运行 ID：
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/run_clindb_first_batch_experiment.py \
|
||||
--run-id clindb-first-batch-review
|
||||
```
|
||||
|
||||
成功时终端只显示运行身份和汇总，不打印原文：
|
||||
|
||||
```text
|
||||
run_id=clindb-first-batch-review
|
||||
status=success
|
||||
documents=5
|
||||
changes=155
|
||||
artifacts=/.../mdpolish/artifacts/<YYYY-MM-DD>/runs/clindb-first-batch-review
|
||||
```
|
||||
|
||||
同一天同名目录已存在时脚本会拒绝覆盖。需要重跑时使用新 ID，不要删除旧目录来绕过检查。
|
||||
|
||||
## 4. 先看哪些结果
|
||||
|
||||
先确认运行目录根部同时存在 `manifest.json` 和 `review-locator.json`。定位文件只供本机评审器寻找原文，包含绝对路径，
|
||||
不得提交或分享。然后打开 `manifest.json`，确认：
|
||||
|
||||
- `run.status` 是 `success`；
|
||||
- `summary.document_count` 和 `summary.success_count` 都是 5；
|
||||
- `summary.failed_count`、`summary.unstable_count` 都是 0；
|
||||
- `summary.change_count` 是 155；
|
||||
- `pipeline.components` 的顺序与 `design/0006` 一致。
|
||||
|
||||
然后查看每份文档目录：
|
||||
|
||||
```text
|
||||
documents/<document_id>/
|
||||
├── result.json
|
||||
├── cleaned.md
|
||||
└── changes.diff
|
||||
```
|
||||
|
||||
- `changes.diff` 用于人工查看输入到最终输出的总变化；
|
||||
- `result.json` 用于按组件、理由、位置和哈希追踪每条修改；
|
||||
- `cleaned.md` 是成功输出全文。
|
||||
|
||||
当前 5 份输入的预期计数是：
|
||||
|
||||
| 文档 | `Change` 数 |
|
||||
| --- | ---: |
|
||||
| dmp | 47 |
|
||||
| ejhf | 9 |
|
||||
| jama | 79 |
|
||||
| sim | 3 |
|
||||
| springer | 17 |
|
||||
| **合计** | **155** |
|
||||
|
||||
按组件应为：Word 批注 2、手稿行号 75、arXiv 戳 2、重复页眉 2、映射断词 6、HTML 实体 31、
|
||||
HTML 表格布局 9、参考文献空行 28。
|
||||
|
||||
## 5. 人工复核重点
|
||||
|
||||
除了逐份查看 diff，至少确认：
|
||||
|
||||
- JAMA 的 Abstract 前作者和单位编号仍在，只删除 Abstract 后的 75 个手稿行号；
|
||||
- Springer 两条 `arXiv preprint arXiv:` 合法参考文献仍在；
|
||||
- Springer 正文中的编号方法列表没有被参考文献规则整理；
|
||||
- dmp 的重复页眉删除后，正文句子接回，参考文献第 18、19 条之间仍有一个空行；
|
||||
- 9 张表仍是 HTML，属性和单元格内容未被布局组件改写；
|
||||
- 双重实体变成单层 `<`、`>` 或 `&`，没有直接生成标签边界；
|
||||
- 图片引用文字保持不变。
|
||||
|
||||
清洗目录没有复制图片资产，所以直接打开 `cleaned.md` 时图片仍可能无法显示。这不表示图片引用被清洗组件删除；
|
||||
资产打包和路径改写需要单独设计。
|
||||
|
||||
## 6. 验证幂等和输入不变
|
||||
|
||||
流水线会在每份文档结束时做最终稳定性复查。需要额外复核整个保存结果时，可以把 `cleaned.md` 作为内存输入再次运行
|
||||
同一 `build_pipeline()`；5 份都应为 `success` 且合计零 `Change`。
|
||||
|
||||
实验层已经在发布前后复读输入并比较字节哈希。需要人工记录运行前后的摘要时，可在运行前后分别执行：
|
||||
|
||||
```bash
|
||||
sha256sum data/md/*.md
|
||||
```
|
||||
|
||||
两次输出必须逐项一致。每个 `cleaned.md` 的 SHA-256 还必须等于对应 `result.json.current_sha256`。
|
||||
|
||||
## 7. 常见失败
|
||||
|
||||
### 状态不是 `success`
|
||||
|
||||
查看对应 `result.json` 的 `errors` 或 `residual_proposals`。`failed` / `unstable` 文档不会有正式 `cleaned.md`，
|
||||
不能把其他文档的部分成功当成整批成功。
|
||||
|
||||
### 修改数不是 155
|
||||
|
||||
先按组件和文档分组定位差异。输入变化、组件参数变化或识别边界变化都必须回到 design/reference 核对；不要放宽断言、
|
||||
补跑第二轮或手工改产物。
|
||||
|
||||
### 图片不显示
|
||||
|
||||
当前运行只保存 Markdown、审计和 diff，不复制图片。不要为了显示图片而修改输入路径或把真实资产强制加入 Git。
|
||||
|
||||
### 私有目录无法被 Snap 工具读取
|
||||
|
||||
运行目录权限是 `0700`，文件是 `0600`。使用普通编辑器或当前虚拟环境中的 Python 读取，不要放宽权限。
|
||||
|
||||
## 8. 数据边界
|
||||
|
||||
产物包含完整论文和原文片段，只能保存在本机 Git 忽略的 `artifacts/`。不得执行 `git add -f`，不得复制到 Wiki、
|
||||
其他仓库、云存储或外部系统。
|
||||
|
||||
`manifest.json` 中的 `retention_until` 是默认 30 天到期时间。当前不自动删除；到期后如需清理，必须先确认具体运行目录。
|
||||
|
||||
需要在只读页面中查看完整前后文和各组件阶段时，继续按
|
||||
[`review-local-cleaning-run.md`](review-local-cleaning-run.md) 操作。
|
||||
@@ -1,79 +0,0 @@
|
||||
# ClinDB-ReviewBench 清洗目标（第一版）
|
||||
|
||||
> 性质：reference——本项目清洗范围的权威查询事实。
|
||||
> 权威关系：本文只定义 ClinDB-ReviewBench 第一批自动清洗"洗什么、不洗什么"；清洗语义的方案比较与批准记录
|
||||
> 属于 `research-wiki/design/`，实现后的运行方式属于 `explanation/` 与 `guides/`。未来只读检查不属于当前批次。
|
||||
> 依据：`research-wiki/scratch/data-5papers-cleaning-audit-2026-08-21.md`（问题编号 A–H 沿用该审计）。
|
||||
|
||||
## 1. 项目定位
|
||||
|
||||
ClinDB-ReviewBench 是师姐的论文清洗项目。`data/` 下当前 5 份 DOI 命名的论文 Markdown
|
||||
（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness）
|
||||
是它的首批输入，未来会继续扩充同源转换产物。
|
||||
|
||||
本仓库（mdpolish）为该项目的数据提供清洗能力。当前由仓库内 first-batch 实验脚本显式组合论文规则，
|
||||
尚未建立 Profile 对象或配置格式；论文专属规则仍不写进通用核心。
|
||||
|
||||
## 2. 第一版清洗目标
|
||||
|
||||
第一版只做"全自动、规则确定、可安全执行"的问题（审计第一档中的 8 类自动修改）。
|
||||
判定标准是三条同时满足：模式可用确定规则描述；不依赖对正文语义的理解；改错可以从 diff 直接看出。
|
||||
|
||||
| # | 问题（审计编号） | 规则要点 | 触发范围（本轮实测） |
|
||||
|---|---|---|---|
|
||||
| 1 | HTML 实体双重转义（D2） | 只在严格 HTML 表格单元格文本中把 `&gt;`→`>`、`&lt;`→`<`、`&amp;`→`&`，解除源码的一层转义；幂等 | dmp 23 处、ejhf 8 处，共 31 处 |
|
||||
| 2 | arXiv 边栏戳（H1） | 只有整行满足 `design/0004-arxiv-submission-stamp-component.md` 第 4 节的严格格式才删除；编号条目内的 "arXiv preprint arXiv:…" 不受影响 | sim L1、springer L18；springer L143/L152 是合法参考文献，必须不误删 |
|
||||
| 3 | Word 审阅批注（B2） | 删除严格单行 `Commented [xx]: ...` 及其后第一个空行；不猜测多行批注正文 | jama 2 处 |
|
||||
| 4 | 手稿行号（B1） | 只处理唯一 `## Abstract` 后至少 20 个、严格递增且含至少 2 个标题证据的完整序列；剥离普通行和标题中的数字前缀 | jama 75 处，其中普通行 69、标题 6；Abstract 前 59 个作者单位编号必须保留 |
|
||||
| 5 | 跑动页眉（C4） | 同一文本行原样重复 ≥2 次（且非正文引用对象）判为页眉，删除并把被切断的上下文段落接回 | dmp L73/L191（"MSOFA Score for Critical Care Triage"），L71→L75 句子被切断 |
|
||||
| 6 | 单行 HTML 表格展开（D1） | 严格完整的单行表格保留 HTML、属性和单元格内容，只按 `<tr>` 换行缩进；当前不转 GFM | 全部 9 个表：dmp 7、ejhf 1、springer 1；每张都有非 `1` 的 `colspan` |
|
||||
| 7 | 跨页断词（E2） | 只按项目批准的左右片段和结果词映射合并相邻行或只隔一个空行的片段；不使用英文词表猜测 | dmp 2 处、jama 2 处、sim 2 处，共 6 处 |
|
||||
| 8 | 参考文献分隔统一（G3） | 只在准确 References 章节内，对从 1 开始连续递增的编号条目统一一个空行 | dmp 13 处、springer 15 处，共 28 处 |
|
||||
|
||||
本表只包含当前自动清洗核心能够承载的修改。原审计中的图片断链校验修复不了 Markdown，已移到第 3 节等待
|
||||
未来独立 Inspector 设计，不计入这 8 类自动清洗目标。
|
||||
|
||||
## 3. 明确不洗（第一版非目标）
|
||||
|
||||
以下问题已确认存在但**不在**第一版范围内，避免实施时范围蔓延：
|
||||
|
||||
- **内容级缺失（A1–A4）**：截断、表格整体缺失、图转表格、句子丢失。清洗无法恢复内容，
|
||||
处置方式（重新转换 / 标记 / 人工补录）由项目负责人决定，不由清洗管线代劳。
|
||||
- **修订语义（B3）**："defined identified by as" 等修订残留词对，哪个词是定稿词需对照定稿版判断。
|
||||
- **占位符与坏日期（B4/B5）**：等待定稿填充，清洗只可检测。
|
||||
- **乱码修复（C3/E1）**：泰文标题 `## ่วง`、`ينDS Hospital`，需要人工对照 PDF 替换。
|
||||
- **表格结构修复（D3–D5）**：空单元格、错误合并行、OCR 表头错字，需人工对照原文。
|
||||
- **空行断句合并（E3）**、**层级重建（C1）**、**标题合并（C2）**、**孤行公式编号（F1）**、
|
||||
**公式空格与纠错（F2–F4）**、**上标风格统一（G1）**、**数字格式（G2）**：属于第二档
|
||||
"自动检测+人工确认"，等第一档验证后再立项。
|
||||
- **封面页（H2）**：ejhf L1–11 的仓库封面区第一版不删——它是整块连续的正文区，删除逻辑
|
||||
与页眉类噪声不同，归入后续批次。
|
||||
- **图片断链校验（H3）**：全部 8 处图片引用是否存在属于未来只读检查；当前核心不读取图片资产、不输出断链
|
||||
报告，也不移动、复制或改写图片。需要该结果时先新增平行 Inspector 设计。
|
||||
- **一切内容改写**：原文写作瑕疵、欧式千分位、拼写（含 "Conounder"）不属于转换噪声，永不由清洗工具修改。
|
||||
|
||||
## 4. 输入输出边界
|
||||
|
||||
- 当前本地输入是 `data/md/` 中按论文缩写命名的 5 份 Markdown 副本，原文件只读；
|
||||
- 核心仍只接收内存 Markdown 字符串；本地实验层按 `design/0005` 将成功输出、JSON 审计和 diff 保存到
|
||||
`artifacts/<YYYY-MM-DD>/runs/<run_id>/`，不回写、不覆盖输入；
|
||||
- 当前批次不读取或复制图片资产；未来 Inspector、资产打包和路径改写分别设计；
|
||||
- 每处实际修改记录组件、理由、原文、改后内容和批次哈希，完整落盘字段以代码和 `design/0005` 为准。
|
||||
|
||||
## 5. 验收口径（第一版）
|
||||
|
||||
- 上述 8 类问题在本轮 5 份文件上的触发处全部按规则处理，共产生 155 条 `Change`；
|
||||
- 5 份文件中未被任何规则命中的正文零变更——除表中列出的触发处外不得有任何其他 diff；
|
||||
- 幂等性：同一输入清洗两次，第二次产出与第一次完全一致；
|
||||
- 规则 2（arXiv 戳）在 springer 上的验收必须包含反向用例：L143/L152 参考文献原文保留；
|
||||
- 规则 4（行号）验收必须包含反向用例：正文中的 "35 pediatric experts"、"10 sites"、"4 continents"
|
||||
等数字开头/含数字短语不受影响。
|
||||
|
||||
2026-08-22 的保存型验收中，5 份文档全部为 `success`，各组件计数为：批注 2、行号 75、arXiv 2、页眉 2、
|
||||
断词 6、HTML 实体 31、表格布局 9、参考文献空行 28。第二次运行 5 份合计 0 条修改，输入运行前后不变。
|
||||
|
||||
## 6. 与审计报告的编号对应
|
||||
|
||||
本文的 8 类自动清洗规则对应 `scratch/data-5papers-cleaning-audit-2026-08-21.md` 的决策清单行：
|
||||
D2、H1、B2、B1、C4、D1、E2、G3。H3 保留为未来只读检查候选。该审计是 scratch 材料，本文引用其编号
|
||||
仅为便于追溯，权威以本文为准。
|
||||
@@ -1,276 +0,0 @@
|
||||
# 7 组对比测试文档 Markdown 清洗审计与实施规范
|
||||
|
||||
- 审计日期：2026-08-20
|
||||
- 审计目录：`/home/lihaoze/gov_test_data/compare`
|
||||
- 清洗对象：7 个测试用例中 `uploads/` 下的 45 份 Markdown
|
||||
- 背景资料：`compare/readme.md` 与各用例 `README.md`
|
||||
- 本次操作：只读审计；没有修改任何原始测试文档
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
这批 Markdown 目前不适合直接作为“语义可靠”的清洗后基准。问题并不只是在空格、换行和标题层级，而是同时存在以下几类高风险污染：
|
||||
|
||||
1. **内容级污染**：至少 32/45 份文件命中了保守的模型幻觉特征词，合计 1,703 次；典型内容包括 `The quick brown fox...`、勾股定理说明、`The image contains...`、`The steps are:` 等与投标文件无关的文本。
|
||||
2. **图像内容丢失**：45/45 份文件都有图片引用，共 8,471 个，但目录中没有任何图片资产；其中 4,012 个引用只是字面量 `data:image/...;base64...`，并不包含可解码数据。
|
||||
3. **表格结构损坏**：41 份文件含原始 HTML 表格，共 7,550 个 `<table>`；35/41 份存在 `<table>/<tr>/<td>` 数量不平衡。另有 4 份 007 用例文档使用 GFM 表格，其中 203 个表格块里至少 60 个行列数不一致。
|
||||
4. **重复和超长噪声**：出现 261,838 字符的单行点线、数万字符的重复英语句子、LaTeX 箭头/颜色命令和重复汉字。001 两份文件的非空行重复比例分别达到 81.8% 和 84.1%。
|
||||
5. **结构扁平化**：全库 29,358 个 Markdown 标题中有 28,736 个是一级标题，占 97.9%；至少 378 个标题缺少空格、只有 `#` 或存在其他明显语法问题。
|
||||
6. **字符与格式污染**：有 10 个替换字符 `�`、20 个私用区字符、11 个 NBSP、5 个零宽字符、1,474 个多余的 `\-` 转义，以及 151,616 行尾部双空格。
|
||||
|
||||
最重要的实施建议是：**不要只产出一份“清洗 Markdown”**。应同时产出：
|
||||
|
||||
- `clean_fidelity.md`：忠实版，保留有法律/业务意义的全部内容，所有修复必须能追溯到源文件。
|
||||
- `clean_compare.md`：对比版，从忠实版派生，移除页眉页脚、失效目录页码、转换器噪声和确定的重复版式块，用于文档相似度计算。
|
||||
- `audit.json`：记录每一次删除、替换、合并、回源重提取和人工确认。
|
||||
|
||||
这样可以避免为了提升对比效果而不可逆地破坏原文，也能避免把幻觉文本、通用图片占位符和重复页眉当成“相同内容”。
|
||||
|
||||
## 2. 审计范围与统计口径
|
||||
|
||||
### 2.1 纳入和排除
|
||||
|
||||
纳入：
|
||||
|
||||
- `compare/001-2/uploads/*.md`：2 份
|
||||
- `compare/002-21/uploads/*.md`：21 份
|
||||
- `compare/003-10/uploads/*.md`：10 份
|
||||
- `compare/004-2/uploads/*.md`：2 份
|
||||
- `compare/005-3/uploads/*.md`：3 份
|
||||
- `compare/006-3/uploads/*.md`：3 份
|
||||
- `compare/007-4/uploads/*.md`：4 份
|
||||
|
||||
排除：
|
||||
|
||||
- 8 份 README：它们是测试说明，不是待清洗输入。
|
||||
- `review.json`、`blocks_*.json`、`match_index.json`、`summary.json`：它们是旧输入产生的下游结果，只用于理解测试背景，不能作为清洗真值。
|
||||
- `file_*_reviewed.docx`：本次未把它们当作 Markdown 清洗对象；后续可作为辅助核对材料，但是否能作为权威源需单独确认。
|
||||
|
||||
### 2.2 基础规模
|
||||
|
||||
45 份输入合计约 36.39 MiB、388,482 个物理行。所有文件都能按 UTF-8 读取，但“能解码”不代表内容无损，文件中仍有 `�` 和私用区字符。
|
||||
|
||||
下表中的“幻觉特征”使用一组偏保守的固定模板进行计数，包括 `The quick brown fox...`、勾股定理模板、`The image contains...`、`The concept of a concept...`、`The steps are:` 等；因此它只是明确下限，不包含全部乱码和中文重复污染。
|
||||
|
||||
| 用例 | 文档数 | 大小 MiB | 行数 | HTML 表格 | GFM 表格行 | 图片引用 | 空 Base64 占位 | 幻觉特征 | 最大单行字符数 | 总体判断 |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| 001-2 | 2 | 6.41 | 48,450 | 2,596 | 0 | 109 | 0 | 0 | 4,482 | 高风险：表格未闭合、重复率极高、图片定位不可用 |
|
||||
| 002-21 | 21 | 15.23 | 158,462 | 2,593 | 592 | 2,000 | 0 | 1,530 | 28,624 | 严重：模型幻觉和重复文本最集中 |
|
||||
| 003-10 | 10 | 9.20 | 116,358 | 1,646 | 2 | 1,793 | 0 | 137 | 261,838 | 严重：存在灾难性长行、乱码、幻觉和 LaTeX 污染 |
|
||||
| 004-2 | 2 | 0.41 | 5,726 | 109 | 0 | 36 | 0 | 7 | 8,001 | 高风险：OCR 语义错误、缺图、幻觉 |
|
||||
| 005-3 | 3 | 2.67 | 31,537 | 476 | 0 | 359 | 0 | 7 | 11,599 | 高风险：重复、缺图、表格和少量幻觉 |
|
||||
| 006-3 | 3 | 1.12 | 14,701 | 130 | 0 | 162 | 0 | 22 | 9,151 | 高风险：含 OCR 文档，仍有明显幻觉和缺图 |
|
||||
| 007-4 | 4 | 1.35 | 13,248 | 0 | 2,792 | 4,012 | 4,012 | 0 | 5,916 | 严重：图片内容全部为无效占位，表格和 Word 目录损坏 |
|
||||
|
||||
## 3. 对下游“文件对比”功能的直接影响
|
||||
|
||||
这些污染会系统性扭曲相似度，不只是影响阅读体验：
|
||||
|
||||
- `The quick brown fox...` 等同一幻觉模板出现在多个本来无关的投标文件里，会制造跨文档假阳性匹配。
|
||||
- 007 中 4,012 个完全相同的空图片占位符会被当成大量相同行。
|
||||
- 001 中同一“综合单价分析表”说明被重复约 900 次；这与 README 中 001、003 的“近似匹配占 99% 以上”和超大 `review.json` 有明显关联。这里只能判断为高度可疑的影响因素，不能在没有重新跑对比的情况下断言它是唯一原因。
|
||||
- 平铺成一级标题、把表格压成一行、把页眉页脚混入正文，会改变分块边界，并使段落/句子/近似三档匹配分布失真。
|
||||
- 如果把不同手机号、身份证号、图片都统一替换成同一个 `[PHONE]`、`[ID]`、`[IMAGE]`，又会制造新的假相同内容。因此占位符必须保留“不同原值不同 token”的性质。
|
||||
|
||||
清洗后必须重新生成 `blocks_*.json`、`match_index.json`、`review.json` 和 `summary.json`。旧匹配数量不能作为清洗后的等值验收条件，应保留为“脏输入性能基线”，另建“干净输入语义基线”。`fileIndex` 和测试用例映射必须保持不变。
|
||||
|
||||
## 4. 问题清单与清洗要求
|
||||
|
||||
### 4.1 C001：固定模板型模型幻觉
|
||||
|
||||
优先级：P0，阻断语义版交付。
|
||||
|
||||
已确认的例子包括：
|
||||
|
||||
- `The quick brown fox jumps over the lazy dog.`：全库 1,071 次。
|
||||
- `The equation $x^2 + y^2 = z^2$ represents a Pythagorean triple...`：全库 62 次。
|
||||
- `The image contains a single character...`
|
||||
- `The concept of a concept is fundamental in physics...`
|
||||
- `- The steps are:`
|
||||
- `Agree to be`
|
||||
- `汽车法规和商业期刊的出版`
|
||||
|
||||
典型证据：
|
||||
|
||||
- `002-21/file_14` 第 1,164 行：一行内重复 `quick brown fox` 约 411 次。
|
||||
- `002-21/file_3` 第 3,804 行：同类重复约 428 次。
|
||||
- `002-21/file_5` 第 2,915 行：`The image contains...` 及数字说明被重复扩展。
|
||||
- `002-21/file_8` 第 5,379 行：28,624 字符的英语概念重复行。
|
||||
- `003-10/file_2` 开头即出现无关英文、勾股定理和中文乱码。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 固定模板命中后先标记其所在段、表格单元格和推定页面，不应只删除匹配到的几个单词。
|
||||
2. 有源 PDF 时，按页或按区域重新提取；没有源文件时，将该段放入 `quarantine`，不能把删除后的残缺上下文冒充完整正文。
|
||||
3. 黑名单适合做拦截器，不适合做唯一清洗器。应再检测异常语言切换、低词汇多样性、同短语高频循环和超长单行。
|
||||
4. 清洗结果中这些已知模板必须为 0；审计文件须记录被移除的原始范围和回源依据。
|
||||
|
||||
### 4.2 C002：超长重复串和退化输出
|
||||
|
||||
优先级：P0/P1，视能否回源而定。
|
||||
|
||||
典型问题：
|
||||
|
||||
- `003-10/file_2` 第 91 行长 261,838 字符，主体是目录点线重复。
|
||||
- 同文件第 8,924 行长 19,089 字符，主体是重复的 LaTeX `\rightarrow`。
|
||||
- `002-21/file_15` 第 4,898 行长 11,267 字符，主体是重复 `\textcolor{red}{\blacksquare}`。
|
||||
- `003-10/file_7` 第 24,211 行含嵌套、重复的 `\textcolor{red}`。
|
||||
- 多处出现成千上万次的 `园`、`\cdots`、`...` 或错误短语，例如“建设工程执行”。
|
||||
|
||||
检测规则建议：
|
||||
|
||||
- 普通文本单行超过 2,000 字符告警，超过 10,000 字符阻断；结构化表格在解析后按单元格重新执行该规则。
|
||||
- 同一字符连续 20 次以上、同一 2—20 字符片段连续 10 次以上告警。
|
||||
- 单行压缩率异常高、唯一 token 比例过低、相邻重复 n-gram 比例过高时进入隔离。
|
||||
- 目录点线只保留为结构化 TOC 信息，不保留数十万字符的视觉填充。
|
||||
|
||||
不允许简单按固定长度截断，因为长 HTML 表格可能包含真实内容。必须先判断是表格、目录点线、Base64、SVG 还是普通文本。
|
||||
|
||||
### 4.3 I001：图片引用全部失效
|
||||
|
||||
优先级：P0。
|
||||
|
||||
全库有 8,471 个图片引用，本地图片资产为 0：
|
||||
|
||||
- 007：4,012 个 `` 或 PNG 变体。这里的 `...` 是文本，不是被终端隐藏的真实数据，无法解码恢复。
|
||||
- 001：109 个目标形如 `page=9,bbox=[...]`，不是标准图片路径，也没有对应裁剪图。
|
||||
- 其余：4,350 个 `images/<hash>.jpg` 等相对引用，但仓库中不存在相应文件。
|
||||
- 7,073/8,471 个图片没有 alt 文本。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 从原始 PDF/DOCX 回源导出图片，使用内容哈希命名，并生成 `assets/manifest.json`。
|
||||
2. 对印章、签名、证书、身份证件、扫描表格等“有语义图片”执行 OCR/版面识别，但仍要保留原图引用，不能只留推测文本。
|
||||
3. 001 的 `page+bbox` 必须转换为结构化来源坐标，再从对应 PDF 裁剪；不能直接当 Markdown URL。
|
||||
4. 如果确实采用纯文本模式，图片位置应写成唯一、可追踪的标记，例如 `[IMAGE_MISSING:007-4:file_1:0001]`，不能用所有文件共享的 `[IMAGE]`，否则会制造假匹配。
|
||||
5. 生产级“完整清洗”验收要求 broken reference 为 0。无法回源的图片必须明确标为未解决，不能计作完整通过。
|
||||
|
||||
### 4.4 T001：HTML 表格损坏和超长单行
|
||||
|
||||
优先级：P0/P1。
|
||||
|
||||
41 份文件中共有 7,550 个 HTML 表格。静态标签计数如下：
|
||||
|
||||
- `<table>` 7,550，`</table>` 7,155。
|
||||
- `<tr>` 101,760，`</tr>` 101,019。
|
||||
- `<td>` 657,263，`</td>` 656,575。
|
||||
- 35/41 份含 HTML 表格的文件至少有一类标签不平衡；7 份连 table 层级都不平衡。
|
||||
|
||||
001 最严重：
|
||||
|
||||
- `file_0`：`table=1315/1124`。
|
||||
- `file_1`：`table=1281/1080`。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 使用容错 HTML 解析器构建 DOM，记录解析器自动补齐了哪些标签；禁止用正则直接替换所有表格标签。
|
||||
2. 对每个表格校验 `rowspan/colspan` 展开后的网格是否矩形、行列数是否与表头一致、单元格顺序是否可追溯。
|
||||
3. 简单矩形表格可转为 GFM；包含合并单元格、斜线表头、嵌套结构的表格应保留规范化 HTML，或另存为结构化 JSON。强行转 GFM 会丢失语义。
|
||||
4. 一行一个完整 HTML 表格应格式化为多行结构，避免长行拖垮分块器和 diff，但换行必须发生在 DOM 节点之间。
|
||||
5. 类似 `004-2/file_0` 第 613 行中“国家、职位、导演、客户”等明显不符合资格审查表语境的表头，应回源确认，不能只修标签。
|
||||
|
||||
### 4.5 T002：GFM 表格行列数不一致
|
||||
|
||||
优先级：P1。
|
||||
|
||||
007 四份文件共有 203 个连续 GFM 表格块、2,792 个表格行，至少 60 个表格块出现不同行拥有不同列数：
|
||||
|
||||
- `file_0`：36 个块，21 个不一致。
|
||||
- `file_1`：55 个块，20 个不一致。
|
||||
- `file_2`：42 个块，7 个不一致。
|
||||
- `file_3`：70 个块，12 个不一致，另有 2 个块缺分隔行。
|
||||
|
||||
此外，`002-21/file_7` 有一个 584 行的大型 GFM 表格，同时出现 3 列和 5 列；`002-21/file_3` 也有不一致的孤立表格块。
|
||||
|
||||
这通常是把 Word/PDF 合并单元格硬投影到 GFM 的结果。清洗器应先恢复二维网格；无法无损表达的表格应改用规范 HTML/JSON，而不是补若干空 `|` 来“通过语法检查”。
|
||||
|
||||
### 4.6 H001：标题层级扁平、标题断裂和错误标题
|
||||
|
||||
优先级：P1。
|
||||
|
||||
问题表现：
|
||||
|
||||
- 001—006 几乎把所有章节都输出为 `#`，没有文档层级。
|
||||
- 存在 `#零星维修...`、`#(正本)` 等缺少空格的标题。
|
||||
- 存在只有 `#` 的空标题。
|
||||
- 同一个标题被版面换行切成多个连续一级标题，例如项目名称被拆成 2—3 行。
|
||||
- 普通正文、图片识别结果和幻觉句子也被错误标成标题。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 每个逻辑文档保留一个主标题 H1；主要章为 H2，节为 H3，依次递进，不跳级。
|
||||
2. 使用编号模式（“第一章”“一、”“（一）”“1.”）、目录结构、相邻上下文和重复页眉信息共同推断层级。
|
||||
3. 连续短标题行在确认属于同一版面标题后合并；不能只根据行长合并。
|
||||
4. 修复 `#标题` 为 `# 标题`，删除空标题；被判为正文的行去掉标题标记。
|
||||
5. 标题文本中的项目编号、公司名、年份不得因规范化而改变。
|
||||
|
||||
### 4.7 P001：段落、硬换行和词内断裂
|
||||
|
||||
优先级：P1。
|
||||
|
||||
151,616 行以两个空格结尾，主要来自 002—006 的转换器。这会把几乎每个物理行强制为 Markdown `<br>`，使原本同一段的文字被切碎。还存在：
|
||||
|
||||
- `物 业`、`项 目`、`服\n务项目` 等版面换行造成的词内空格或断行。
|
||||
- 页码、目录页码和正文混在同一行。
|
||||
- 句子被错误拼接成超长段，或每句话都被拆成独立段。
|
||||
- 项目编号中的连字符被转义成 `440442\-2025\-00435`，同一标识在不同文件中形式不一致。
|
||||
|
||||
清洗要求：
|
||||
|
||||
1. 先识别标题、列表、表格、地址、编号、签名区，再做段落重组。
|
||||
2. 普通段落内部把版面软换行合并为空或一个空格；中文字符之间通常直接合并，中英/数字边界按规则保留空格。
|
||||
3. 列表项、诗行、地址、落款、表格单元格内的有意义换行不得统一删除。
|
||||
4. `\-` 仅在确认是转换器多余转义时还原；项目编号和负号的字符本身必须保留。
|
||||
5. Unicode 采用 NFC，不建议全局 NFKC；NFKC 可能改变罗马数字、圈号、单位和兼容字符的法律原貌。
|
||||
|
||||
### 4.8 D001：页眉、页脚、目录页码和文档边界
|
||||
|
||||
优先级：P1。
|
||||
|
||||
已检测到至少 1,166 个“独立页码样式候选”，但其中也会混入 `0/1000` 等评分或限制值，所以不能仅凭正则删除。明确例子包括 `1 / 390`、`368 / 390`、`4 / 6`。
|
||||
|
||||
清洗要求：
|
||||
|
||||
- 只有在同一文本出现在多页相近顶部/底部位置，或页码形成合理序列时，才判定为页眉页脚。
|
||||
- `clean_fidelity.md` 可通过元数据保留页码映射；`clean_compare.md` 删除纯版式页码和重复运行标题。
|
||||
- 002 中 `file_16`—`file_19` 开头分别表现为符合性审查、综合文件、商务部分、技术部分，可能是同一投标人的分卷/分册。清洗器必须保存原始 `fileIndex` 和卷册边界，不能自行拼接。
|
||||
- 目录文本可转换为结构化章节导航；目录点线和页码不应参与正文相似度。
|
||||
|
||||
### 4.9 D002：重复内容与模板内容
|
||||
|
||||
优先级：P1，但严禁盲删。
|
||||
|
||||
重复既可能是转换器错误，也可能是投标文件真实模板。典型统计：
|
||||
|
||||
- 001 两份文件非空行重复比例分别为 81.8% 和 84.1%。同一“综合单价分析表”及说明出现约 900 次。
|
||||
- `002-21/file_3` 非空行重复比例 44.8%，含“装约买箱箱装”等乱码高频重复。
|
||||
- `002-21/file_15` 中 `Agree to be` 重复 813 次。
|
||||
- `002-21/file_5` 中 `- The steps are:` 重复 818 次。
|
||||
- `002-21/file_20` 中目录式条目“施工等级 282”“工期保证措施 282”分别重复 165、164 次。
|
||||
|
||||
处理原则：
|
||||
|
||||
1. 明确的幻觉/转换器退化重复应回源替换或隔离。
|
||||
2. 重复页眉、页脚、页码在对比版删除。
|
||||
3. 合法合同条款、报价表说明和表头在忠实版保留。对比版可把同一文档内重复模板块标为同一 `template_block_id`，在相似度计算时降权，而不是直接删除。
|
||||
4. 跨文档共同出现的合法招标模板本来就是对比目标的一部分，不应因“重复”而全部抹除。项目需要明确是检测抄袭、检测共同模板，还是两者分别评分。
|
||||
|
||||
### 4.10 L001：Word/PDF 转换遗留语法
|
||||
|
||||
优先级：P1/P2。
|
||||
|
||||
包括：
|
||||
|
||||
- 007 `file_1` 有 171 个 `(#_Toc...)` Word 目录链接，但全库没有对应显式锚点。
|
||||
- 001 有 2,594 个 `<div align="center">`，用 HTML 仅表达居中版式。
|
||||
- 575 个行内数学片段中有不少重复的 `\textcolor`、`\rightarrow`、`\cdots` 和勾股定理幻觉。
|
||||
- 20 个私用区字符主要是 U+F0B7、U+F070、U+F0D8，常见于 Wingdings/项目符号映射。
|
||||
- 10 个 `�` 出现在 7 份文件中，已经无法靠 Unicode 规范化恢复原字符。
|
||||
|
||||
处理要求：
|
||||
|
||||
- Word TOC 链接要么根据清洗后标题生成真实 slug，要么移除链接只保留目录文字；不得保留悬空锚点。
|
||||
- 纯版式 `<div align>` 转成语义标题/段落；居中信息可进样式元数据。
|
||||
- 私用区字符按原字体映射或源文件回查，不能一律删除。
|
||||
- `�` 必须逐处回源，无法回源时产生显式未解决项。
|
||||
- LaTeX 只有在源文件确实包含公式/符号时保留；重复生成的视觉符号应由原图或文本含义替代。
|
||||
|
||||
@@ -1,225 +0,0 @@
|
||||
# data/ 五份论文 Markdown 审计报告（2026-08-21）
|
||||
|
||||
> 状态：scratch 草稿，供评审。清洗力度由师姐逐项决定，本报告只列问题、证据和可选处置，不做力度决策。
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
- 五份文件均为英文学术论文的 PDF→Markdown 转换产物，分属 5 个来源（JAMA、EJHF、Statistics in Medicine/arXiv、Springer/arXiv、Disaster Med Public Health Preparedness），噪声特征差异很大，不能用一套固定规则覆盖。
|
||||
- 最严重的问题不是格式噪声，而是**内容丢失**：2 份文件中途截断，1 份（JAMA）所有表格整体缺失。这类问题清洗无法恢复，只能重新转换或人工补录。
|
||||
- JAMA 文件的原始 PDF 是**未定稿的 Word 修订稿**，行号、审阅批注、修订残留词对全部泄漏进正文，是五份中污染最重的。
|
||||
- 格式类噪声（双重转义实体 31 处、孤行公式编号 12 处、空行断句、标题层级扁平、引用上标混用等）确定可清洗，风险低。
|
||||
- 少数问题涉及**语义改变**（数学公式被 OCR 改错、乱码替换专名），自动清洗有改错正文的风险，建议人工确认。
|
||||
|
||||
## 2. 审计范围与方法
|
||||
|
||||
- 对象：`data/*/markdowns/*.md` 共 5 份，逐行人工通读；行号均指 markdowns 下的 md 文件。
|
||||
- 辅助统计（本轮实际执行）：标题层级计数、图片/表格/实体/批注/arXiv 戳的 grep 计数。
|
||||
- 判定原则：只把"转换管线引入的噪声"列为清洗对象；原文自身的写作瑕疵（如语法错误）不属于转换噪声，单独列出并标注"不建议清洗"。
|
||||
- 未验证项：未与原 PDF 逐页比对内容完整性，"缺失"结论基于文内自引用（如正文提到 Table 2 但全文无 Table 2）和文件截断位置。
|
||||
|
||||
## 3. 总体统计
|
||||
|
||||
| 文件（下文简称） | 行数 | H1 | H2 | H3+ | 图片 | HTML表格 | 双重转义 | 孤行编号 | 批注 | arXiv边栏戳 |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| dmp (Disaster Med) | 208 | 0 | 11 | 0 | 1 | 7 | 23 | 0 | 0 | 0 |
|
||||
| jama (JAMA 2024) | 349 | 2 | 10 | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
|
||||
| ejhf (EJHF 2020) | 143 | 1 | 13 | 0 | 1 | 1 | 8 | 0 | 0 | 0 |
|
||||
| sim (Stat Med/arXiv) | 247 | 2 | 11 | 0 | 2 | 0 | 0 | 12 | 0 | 1 |
|
||||
| springer (ICU LSTM) | 154 | 1 | 12 | 0 | 4 | 1 | 0 | 1 | 0 | 1 |
|
||||
|
||||
补充：jama 有 128 行以"行号+空格"开头（手稿行号泄漏）；springer 的 3 处 arXiv 字符串中 2 处是参考文献的正常引用，仅 1 处是边栏戳。
|
||||
|
||||
## 4. 问题清单
|
||||
|
||||
每项标注：【确定】= 确定是转换噪声，可安全清洗；【语义】= 涉及内容判断，建议人工确认；【丢失】= 内容缺失，清洗不可恢复。处置选项仅供师姐选择。
|
||||
|
||||
### A. 内容级问题（最严重）
|
||||
|
||||
**A1【丢失】两份文件中途截断**
|
||||
- ejhf 第 143 行在 Table 1 HTML 表格中间戛然而止（"Medical history at randomization, no. (%)" 行后为空单元格）。Table 1 后半、Tables 2–4、全部图注、参考文献整体缺失。
|
||||
- sim 第 247 行止于方法 M5 描述中间，且结尾 URL 损坏（`https://cran.r-project.org/web/p6$^{10}$`）。第 3 节（结果）、4、5 节、参考文献、附录缺失。
|
||||
- 处置选项：a) 退回重新转换；b) 接受现状并在元数据标记"截断"；c) 人工补录缺失部分。清洗管线本身无法解决。
|
||||
|
||||
**A2【丢失】jama 全部表格缺失**
|
||||
- 正文多处引用 Table、Box 1、eTables 1–3，但全文 0 个表格、0 张图片。摘要性内容（如各器官系统阈值表）完全丢失。
|
||||
- 处置选项：同 A1。
|
||||
|
||||
**A3【丢失】dmp 的 FIGURE 1 流程图被转成 HTML 表格**
|
||||
- 第 83 行：流程图（决策树）被输出为单行 HTML 表格，图形语义尽失，仅 FIGURE 2（第 111 行）保留为图片。
|
||||
- 处置选项：a) 保留现状（有总比没有好）；b) 重新转换该页；c) 人工用图片替换。
|
||||
|
||||
**A4【丢失】sim 第 93 行句子开头缺失**
|
||||
- "/unpenalized) regression models" 以斜杠开头，前文整句丢失。
|
||||
- 处置选项：标记为损坏片段，或对照原文补录。
|
||||
|
||||
### B. 草稿/修订痕迹泄漏（仅 jama，原稿是 Word 修订稿）
|
||||
|
||||
**B1【确定】手稿行号泄漏（128 行）**
|
||||
- 正文行以行号开头："25 increase in the Sequential..."（第 115 行）、"26 with suspected infection"（第 116 行）等；标题也带行号："## 116 Results/recommandations"（第 149 行）、"## 117 Criteria..."（第 151 行）、"## 143 Organ dysfunction..."（第 165 行）。
|
||||
- 处置选项：a) 剥离行首行号（注意与正常编号列表、年份区分）；b) 连同批注一起整体退回，要求提供定稿版。
|
||||
|
||||
**B2【确定】审阅批注泄漏（2 处）**
|
||||
- 第 155–157 行："Commented [LS1]: Why highlighted?"、"Commented [SW2R1]: To make sure we use capital letters..."。
|
||||
- 处置选项：整段删除。注意第 292 行还有一句删除文字与保留文字混杂的句子（"Appropriate process and balancing measures Efforts to enhance..."），需人工断句。
|
||||
|
||||
**B3【语义】Word 修订残留词对（约 7 处）**
|
||||
- "defined identified by as"（第 91 行）、"defined identified using by"（第 153 行）、"defined identified as in sepsis-septic patients"（第 116 行）、"defined indicate by as"（第 163 行）、"was were derived"（第 306 行）、"The new-Phoenix"（第 197 行）等——是"插入词+删除词"并存的痕迹。
|
||||
- 哪个词是最终保留词需要对照定稿判断，自动二选一有风险。处置选项：a) 人工逐处确认；b) 退回要定稿。
|
||||
|
||||
**B4【确定】未填占位符**
|
||||
- "XX societies"（第 101 行）、"XX-microbiological testing and YY-antibiotics"（第 302 行）、"Endorsing societies: To be populated after acceptance"（第 317 行）。
|
||||
- 处置选项：保留并标记待补，或等定稿填充后再清洗。
|
||||
|
||||
**B5【确定】日期损坏**
|
||||
- 第 85 行 "Revision date: December 3122, 2023"。
|
||||
- 处置选项：对照原文改为 2023 年内正确日期（人工）。
|
||||
|
||||
### C. 标题结构问题
|
||||
|
||||
**C1【确定】标题层级扁平**
|
||||
- dmp：0 个 H1，11 个 H2 全部同级（含 Introduction/Methods/Results 等，第 1,3,7,59,67,103,137,153 行）。
|
||||
- ejhf：主章节与子章节同为 H2（"## Methods" 第 51 行、"## Study population" 第 53 行），无层级区分。
|
||||
- 处置选项：按章节编号/语义推断层级（2.1 → H3），或统一降级保持平级。前者需人工核对推断结果。
|
||||
|
||||
**C2【确定】标题被拆分成两行**
|
||||
- jama：标题拆成两个 H1（第 1–2 行 "International Consensus Criteria..." / "The Phoenix Pediatric Sepsis Criteria"，实为正副标题）。
|
||||
- sim：主标题拆成两个 H1（第 3–4 行 "Conounder selection..." / "treatment effect estimators"，且首词 "Conounder" 是 OCR 错字，原文应为 Confounder）；2.2 节标题拆两行（第 183/185 行 "…full matching on the propensity" / "## score"）。
|
||||
- 处置选项：合并为单标题；sim 主标题的 "Conounder" 拼写需对照原文确认（【语义】）。
|
||||
|
||||
**C3【语义】乱码标题**
|
||||
- dmp 第 47 行 `## ่วง`（泰文字符）。按位置推断应为 METHODS，但不能凭推断改写正文级内容。
|
||||
- 处置选项：人工对照 PDF 改回，或删除该标题。
|
||||
|
||||
**C4【确定】跑动页眉混入正文且被标为标题**
|
||||
- dmp 第 73、191 行 "## MSOFA Score for Critical Care Triage" 出现在句中和 REFERENCES 内；第 71→75 行一句话被它从中间切断。
|
||||
- 处置选项：删除页眉行并把被切断的句子接回。
|
||||
|
||||
### D. 表格问题
|
||||
|
||||
**D1【确定】表格压缩为单行 HTML**
|
||||
- 全部 9 个表格（dmp 7、ejhf 1、springer 1）都是 `<table>...</table>` 单行输出，diff、review、编辑都极难。
|
||||
- 处置选项：转 GFM 多行表格（简单表）；对含 rowspan/colspan 的复杂表保留 HTML 但格式化缩进。
|
||||
|
||||
**D2【确定】HTML 实体双重转义（31 处）**
|
||||
- dmp 23 处（`&gt;400`、`&lt;1.2`、`MAP&lt;70`，第 27/33/39 行等）；ejhf 8 处（`A1C&lt;7`、`7≤A1C&lt;8`，第 143 行）。
|
||||
- 处置选项：还原一层转义（`&gt;` → `>`）。这是最安全的清洗之一。
|
||||
|
||||
**D3【确定】合并单元格信息丢失**
|
||||
- dmp 第 33 行 Table 2 的 Liver 行出现空 `<td></td>`，跨行合并关系丢失，数值与表头对应断裂。
|
||||
- 处置选项：对照 PDF 人工修复合并结构，或标记为低可信表格。
|
||||
|
||||
**D4【确定】表格行错误合并**
|
||||
- dmp 第 39 行 Table 3：四个器官系统 "Respiratory Coagulation Liver Cardiovascular" 被挤进一个单元格。
|
||||
- 处置选项：人工拆分修复。
|
||||
|
||||
**D5【确定】表头 OCR 乱码**
|
||||
- springer 第 111 行 Table 1 表头 "Claesther"（应为 Classifier）。
|
||||
- 处置选项：人工改正（单处，低成本）。
|
||||
|
||||
### E. 文本级损坏
|
||||
|
||||
**E1【语义】跨脚本字符替换**
|
||||
- dmp 第 43 行 "atينS Hospital"、第 55 行 "ينDS Hospital"——阿拉伯字符 ين 替换了 "LD"（LDS Hospital 是机构专名）。
|
||||
- 处置选项：人工替换回 "LDS"。自动规则可检出非拉丁字符混入，但替换动作建议人工确认。
|
||||
|
||||
**E2【确定】跨页断词**
|
||||
- dmp：第 125 行结尾 "…at the relevant thresh" + 第 127 行 "olds of 8 and 11"（单词 thresholds 被页边界切开）。
|
||||
- sim：第 87/89 行 "except possi-" / "bly via treatment"；第 217/219 行 "cre-" / "ated by permuting"。
|
||||
- 处置选项：拼接断词（去连字符合并）。需注意英语中合法的行尾连字符（如 "well-known"）不能误合并，建议只合并"行尾连字符+下一行首为小写字母且拼出的词在词表内"的情况，其余保留待审。
|
||||
|
||||
**E3【确定】句内空行断句**
|
||||
- 大量段落被空行从句子中间切开：jama 第 103→105、113→115 行；ejhf 第 47–49、57–59、67–69、87–89、113–115、131–133 行；springer 第 16→20 行（中间还被 arXiv 戳隔开，见 H1）、42→44 行。
|
||||
- 处置选项：段内合并（前段末无句号且后段首为小写/连接词时拼接）。这是对 RAG 分块影响最大的问题之一。
|
||||
|
||||
**E4【确定】段落内容错位/孤立行**
|
||||
- springer 第 63 行孤立 "1"（公式编号漂移，见 F1）；第 81–83 行 "…improve the simple Multilayer Perceptron… other deep models." 后接 "including RNNs and MLPs."，段落被错误切开。
|
||||
- 处置选项：结合上下文人工归位。
|
||||
|
||||
### F. 数学公式问题（sim 最重，springer 次之）
|
||||
|
||||
**F1【确定】公式编号漂移成孤行**
|
||||
- sim 12 处：第 63/70/81/107/117/133/145/153/161/171/179/197 行分别是 "1"、"2"、"(4)"…"(12)"；springer 第 63 行 "1"。
|
||||
- 处置选项：并入对应公式块或删除（若公式本体已带编号）。需逐处对照，不宜盲目删除。
|
||||
|
||||
**F2【语义】公式内容被 OCR 改错**
|
||||
- sim 第 139 行 `$\exp(x) = \exp(x)/\{1+\exp(x)\}$`——这是 expit 函数定义（x↦e^x/(1+e^x)），左边的 $\exp(x)$ 应为 $x$ 或 expit(x)。OCR 错误改变了数学含义，且这种错误会误导下游读者。
|
||||
- 处置选项：人工对照原文修复；自动工具只能标记"公式与上下文不符"，不宜自动改。
|
||||
- sim 第 119 行 "weights w k" 下标丢失，同类。
|
||||
|
||||
**F3【确定】LaTeX 冗余空格与风格不一**
|
||||
- sim 全文行内公式带前导空格（`$ \widehat{\psi}_{j} $`）、内部空格过多（`\widehat { \mathrm { E } } ( Y ^ { a } )`，第 150 行）。
|
||||
- 处置选项：规范化空格（不改变符号语义的前提下）。风险低但需保守，避免动 `\,` `\;` 等有意义的间距命令。
|
||||
|
||||
**F4【确定】公式 OCR 字符间距拉宽**
|
||||
- springer 第 72/78 行 cases 环境里 `\mathrm { s u r v i v o r s ~ g r o u p }` 逐字空格。
|
||||
- 处置选项：去除字母间空格（保守做法：只处理 `\mathrm{}` 内的单字母间距模式）。
|
||||
|
||||
### G. 格式不一致
|
||||
|
||||
**G1【确定】引用上标风格混用**
|
||||
- 同一文件内 LaTeX `$^{1,2}$` 与 Unicode 上标（¹²、²⁻⁴、⁴⁹ ⁵⁰,⁵¹）并存：jama（第 115–116 行 Unicode vs 第 107/187 行 `$\geq 2$`）、ejhf（第 45–49 行 LaTeX vs 第 133 行 Unicode）、springer（第 5–6 行 `$ ^{1} $` vs 第 20 行 [1,2,4,3] 方括号）。
|
||||
- 处置选项：统一为一种（选哪种由师姐定，取决于下游用途：RAG 检索倾向 Unicode 纯文本，渲染倾向 LaTeX）。
|
||||
|
||||
**G2【可能】数字格式不一致**
|
||||
- ejhf 第 55/57 行 "6068 patients" vs "4,091 patients"；dmp 第 91 行 "0.81-.85"（小数点前缺 0）。
|
||||
- 注意：springer 的 "61.532"、"46.520" 是欧式千分位，可能是原文排版而非转换噪声，不能自动统一。
|
||||
- 处置选项：统一千分位与小数风格；欧式写法是否转换需师姐定。
|
||||
|
||||
**G3【确定】参考文献列表分隔不一致**
|
||||
- dmp：refs 1–18 空行分隔，19–33 连续堆叠（第 193–208 行）；springer：refs 1–8 空行分隔，9–19 连续堆叠（第 140–154 行）。
|
||||
- 处置选项：统一为一条一空行。
|
||||
|
||||
### H. 非正文噪声与资产
|
||||
|
||||
**H1【确定】arXiv 边栏戳**
|
||||
- sim 第 1 行 "arXiv:2001.08971v3 [stat.ME] 10 Oct 2020"；springer 第 18 行同类戳插在 Introduction 段落中间，把一段话切成三截（第 16/18/20 行）。
|
||||
- 处置选项：删除戳行并接回段落。注意别误删参考文献中的合法 "arXiv preprint arXiv:…"（springer 第 143/152 行）。
|
||||
|
||||
**H2【确定】机构仓库封面页**
|
||||
- ejhf 第 1–11 行：Glasgow eprints 引用说明、版本声明、`http://eprints.gla.ac.uk/213358/`、"Deposited on: 27 April 2020"，以及封面图 ``（该图是仓库封面，不是论文插图）。
|
||||
- 处置选项：整体删除封面区；封面图一并删或移入元数据。对 RAG 是纯噪声。
|
||||
|
||||
**H3【可能】图片相对路径依赖**
|
||||
- 全部图片用 `../images/` 相对路径（dmp 第 111 行、ejhf 封面、sim 第 229/231 行、springer 第 65/115/117/119 行）。md 文件一旦脱离原目录结构，图片全部失效。
|
||||
- 处置选项：a) 保持现状（目录结构不变时无碍）；b) 清洗时把路径改写为部署目标路径；c) 校验引用的图片文件是否存在并报告断链。sim 的 Figure 1 实为左右两个面板（sub0/sub1 两张图）共享一条图注（第 233 行），springer 的 Fig. 2 为三面板（sub1–sub3）——合并还是保持多图由师姐定。
|
||||
|
||||
## 5. 不建议清洗的内容（保真边界）
|
||||
|
||||
- **原文自身的写作瑕疵**：springer 论文语言明显不通（"was went to describe"、"In the other hand"、"section 4 discuss"），这是作者问题不是转换噪声，清洗工具不得改写学术内容。
|
||||
- **欧式千分位**（springer "61.532"）：疑似原文排版，自动统一有改数风险。
|
||||
- **专名与缩写的大小写、期刊缩写风格**：不属于转换噪声。
|
||||
- **A 类内容缺失**：不要试图用生成或推测内容"补全"缺失章节——宁可留空标记。
|
||||
|
||||
## 6. 决策清单（供师姐勾选力度）
|
||||
|
||||
| 编号 | 问题 | 涉及文件 | 建议决策点 |
|
||||
|---|---|---|---|
|
||||
| A1/A2/A4 | 截断与表格缺失 | ejhf, sim, jama | 重新转换 / 接受并标记 / 人工补录 |
|
||||
| A3 | 图转表格 | dmp | 保留 / 重转 / 换图 |
|
||||
| B1 | 行号剥离 | jama | 剥离 / 退回要定稿 |
|
||||
| B2 | 批注删除 | jama | 删（基本无争议） |
|
||||
| B3 | 修订词对 | jama | 人工定稿对照（不宜自动） |
|
||||
| B4/B5 | 占位符/坏日期 | jama | 保留标记 / 人工修 |
|
||||
| C1 | 层级重建 | dmp, ejhf | 推断层级 / 保持平级 |
|
||||
| C2 | 标题合并 | jama, sim | 合并（低风险） |
|
||||
| C3/C4 | 乱码标题/页眉 | dmp | 人工改 / 删 |
|
||||
| D1 | 表格展开 | 全部 | GFM / 缩进 HTML / 不动 |
|
||||
| D2 | 实体还原 | dmp, ejhf | 还原（低风险） |
|
||||
| D3/D4/D5 | 表格结构修复 | dmp, springer | 人工修 / 标记低可信 |
|
||||
| E1 | 乱码专名 | dmp | 人工替换 |
|
||||
| E2 | 断词拼接 | dmp, sim | 保守合并+白名单 |
|
||||
| E3 | 段内合并 | jama, ejhf, springer | 启用（对 RAG 影响大） |
|
||||
| E4 | 段落归位 | springer | 人工 |
|
||||
| F1 | 编号并入 | sim, springer | 对照处理 |
|
||||
| F2 | 公式纠错 | sim | 人工（含义级） |
|
||||
| F3/F4 | 公式空格 | sim, springer | 保守规范化 |
|
||||
| G1 | 上标统一 | jama, ejhf, springer | 定一种风格 |
|
||||
| G2 | 数字格式 | ejhf, dmp | 统一 / 保留原文 |
|
||||
| G3 | 参考文献分隔 | dmp, springer | 统一空行 |
|
||||
| H1 | arXiv 戳 | sim, springer | 删+接段（勿伤引文） |
|
||||
| H2 | 封面页 | ejhf | 删 |
|
||||
| H3 | 图片路径 | 全部 | 现状 / 改写 / 断链校验 |
|
||||
|
||||
## 7. 与既有审计的关系
|
||||
|
||||
`research-wiki/reference/GOVDOC_SAAS_CLEANING_SCOPE.md`（45 份政务文档审计，原名 MARKDOWN_CLEANING_AUDIT.md）中的幻觉、重复、页眉页脚等类别在本批论文中部分复现（C4/D2/H1 对应旧审计的 D001/L001 类），但本批新增了论文特有的类别：修订稿痕迹（B 类）、公式问题（F 类）、引用上标混用（G1）、截断缺失（A 类）。若后续建立通用清洗规则库，B/F/G 类需要论文 profile，不宜进默认规则。
|
||||
@@ -1,143 +0,0 @@
|
||||
# HTML 表格清洗专题调研：业界工具在 Markdown 清洗中如何处理 HTML 表格
|
||||
|
||||
> 状态：调研记录，尚未进入任何 design。
|
||||
>
|
||||
> 调研日期：2026-08-21。
|
||||
>
|
||||
> 定位：回答一个具体问题——Markdown 清洗中遇到 HTML 表格，业界工具实际怎么做。
|
||||
> 结论用于印证或修正 [`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md`](../reference/GOVDOC_SAAS_CLEANING_SCOPE.md)
|
||||
> 中 T001/T002 的方向，不构成对任何方案的批准。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
围绕“Markdown 里的 HTML 表格怎么办”，生态里的工具分成三种流派：
|
||||
|
||||
| 流派 | 做法 | 代表 |
|
||||
|---|---|---|
|
||||
| 生成端保真 | 复杂表格直接输出 HTML，不做管道表格 | MinerU、Docling（HTML/JSON 视图） |
|
||||
| 强行归一 | 全部转成管道表格，合并单元格静默损坏或内容重复 | Turndown + gfm 插件、Docling（Markdown 视图） |
|
||||
| 保真派 | 容错解析 → 校验网格 → 简单表转 GFM、复杂表保留 HTML/JSON | 本项目审计 T001 方向、Pandoc（grid tables / AST） |
|
||||
|
||||
支撑这张表的共同事实是：**GFM 管道表格语法在原理上表达不了合并单元格**。三种流派只是对这条约束的
|
||||
不同回答——绕开它、硬转它、或者按能力分流。
|
||||
|
||||
## 2. 语法能力边界：GFM 管道表格没有合并单元格写法
|
||||
|
||||
Pandoc 手册对各家 Markdown 表格方言的原文描述（2026-08-21 从官方 MANUAL 核实）：
|
||||
|
||||
| 方言 | 合并单元格 | 单元格内块级元素 |
|
||||
|---|---|---|
|
||||
| pipe tables（≈GFM 表格） | 不支持（单元格不能跨多行） | 不能包含块级元素 |
|
||||
| multiline tables | 明确不支持跨行/跨列单元格 | 可以 |
|
||||
| grid tables | **支持**（"Cells can span multiple columns or rows"） | 可以 |
|
||||
| HTML `<table>` | 原生 `rowspan`/`colspan` | 可以 |
|
||||
|
||||
grid tables 是唯一支持合并单元格的 Markdown 表格语法，但它是 Pandoc 扩展，GitHub 不渲染，
|
||||
对“清洗后还要在 GFM 渲染器里查看”的场景不可用。所以在 GFM 方言内部，合并单元格没有任何无损写法；
|
||||
想保真只能保留 HTML，或存结构化 JSON。
|
||||
|
||||
Pandoc 手册同时警告：从表达能力更强的格式转换时"some document elements, such as complex tables,
|
||||
may not fit"，"can be expected to be lossy"。
|
||||
|
||||
来源：[Pandoc MANUAL - Tables](https://pandoc.org/MANUAL.html)
|
||||
|
||||
## 3. 生成端：PDF→Markdown 转换器为什么输出 HTML 表格
|
||||
|
||||
MD 文档里出现 HTML 表格，通常不是 bug，而是转换器面对合并单元格（`rowspan`/`colspan`）、
|
||||
多级表头等管道语法表达不了的结构时的标准回退。
|
||||
|
||||
| 工具 | 表格输出策略 | 依据 |
|
||||
|---|---|--- |
|
||||
| [MinerU](https://github.com/opendatalab/MinerU) | 所有表格一律输出 HTML 嵌在 Markdown 中，不做管道表格；支持跨页表格拼接 | 官方 README 功能列表 |
|
||||
| [Docling](https://docling.org/) | 内部 TableFormer 模型专门恢复合并单元格；同一文档可导出 HTML / Markdown / JSON 三种视图 | 官网能力页 + docling-core 2.92.0 源码 |
|
||||
|
||||
对本项目的含义：HTML 表格是合法的中间形态，不是待清除的垃圾。清洗目标不是“消灭 HTML 表格”，
|
||||
而是“识别哪些表格结构正确、哪些在转换中损坏”。
|
||||
|
||||
## 4. 反面教材一：Turndown 静默产出错位表格
|
||||
|
||||
[Turndown](https://github.com/mixmark-io/turndown) 是最流行的 HTML→Markdown 转换库之一，
|
||||
[turndown-plugin-gfm 的 tables.js](https://github.com/mixmark-io/turndown-plugin-gfm/blob/master/src/tables.js)
|
||||
（v1.0.2，2018 年发布后基本未改）只做两件事：
|
||||
|
||||
1. 首行不是全 `<th>`（无表头行）的表格：保留 HTML 不转；
|
||||
2. 其余表格：按 DOM 位置逐格输出管道符。
|
||||
|
||||
它**完全没有 `colspan`/`rowspan` 的处理代码**。合并单元格不触发上面的回退，直接按 DOM 位置压扁，
|
||||
转出列数不齐的坏表，且**不报任何错**。
|
||||
|
||||
对本项目的含义：
|
||||
|
||||
- “无表头就不转”是能力判断驱动的回退，这个思想是对的；但它的能力判断漏掉了合并单元格；
|
||||
- 连最流行的转换库在这里都会静默弄坏表格——审计要求“先建 DOM、校验网格、禁止正则替换”有真实事故支撑；
|
||||
- 选 HTML→Markdown 转换库时，“是否处理 span”必须列入验证项，不能信 README 宣称。
|
||||
|
||||
## 5. 反面教材二：Docling 的 Markdown 导出重复合并单元格内容
|
||||
|
||||
读了 docling-core 2.92.0 的源码（wheel 解包，2026-08-21）：
|
||||
|
||||
- 内部 `TableCell` 带 `row_span`/`col_span` 和起止行列偏移；`TableData.grid` 属性把同一个 cell 对象
|
||||
**铺满**它覆盖的每个 (行， 列) 位置；
|
||||
- **HTML 序列化器**（`transforms/serializer/html.py`）：遍历网格时跳过被覆盖的续位
|
||||
（`rowstart != i` 或 `colstart != j` 时 `continue`），只在起始位置输出，并正确带上
|
||||
`rowspan="N"`/`colspan="N"`——语义保真；
|
||||
- **Markdown 序列化器**（`transforms/serializer/markdown.py` 的 `MarkdownTableSerializer`）：
|
||||
直接遍历铺满后的 grid，每个位置都输出 `col.text`——一个 `row_span=3` 的单元格内容在 Markdown
|
||||
输出里**重复出现 3 次**。转义只处理换行和管道符（`\n`→空格、`|`→`|`），再用
|
||||
tabulate `tablefmt="github"` 输出管道表格。
|
||||
|
||||
即 Docling 面对“Markdown 视图必须有合并单元格”的需求，选择了**内容重复**来保住矩形形状。
|
||||
这是“强行转管道表格会丢语义”的又一个实例，和 Turndown 的压扁是同一根源的两种表现。
|
||||
|
||||
对本项目的含义：
|
||||
|
||||
- “转 GFM”不是免费的格式变换，每一家实现都发明了自己的有损映射；
|
||||
- 如果未来用 Docling 做回源提取（调研报告第 6.1 节的候选方向），它的 Markdown 导出不能直接当作
|
||||
保真输出使用，需要用它的 `DoclingDocument` JSON 或 HTML 视图；
|
||||
- 审计 T001 说“强行转 GFM 会丢失语义”，这里的机制证据是：跨行列单元格要么被压扁（Turndown）、
|
||||
要么被重复（Docling）、要么失去合并关系本身（都失去 `rowspan`/`colspan` 语义）。
|
||||
|
||||
## 6. 清洗与格式化工具：主流选择是“不动 HTML 块”
|
||||
|
||||
| 工具 | 对 Markdown 内 HTML 表格的行为 | 来源 |
|
||||
|---|---|---|
|
||||
| remark / mdformat | raw HTML 当不透明块原样传递，不重新格式化、不转换 | [mdformat](https://mdformat.readthedocs.io/) 官方文档（核心保证是格式化前后 AST 一致，HTML 块不在处理范围） |
|
||||
| rumdl MD033（no-inline-html） | 报告 `<table>`（它有 Markdown 等价物），但 `fix` 只自动转 `em/strong/code/a/img/br/hr` 等行内简单标签，**不含表格**；`allowed-inside = ["table"]` 可整块豁免 | 本地 `reference/rumdl/docs/md033.md` |
|
||||
| rumdl MD056（table-column-count） | 校验每行列数与表头一致，自动修复方式是补/删空单元格 | 本地 `reference/rumdl/docs/md056.md` |
|
||||
| rumdl MD058（blanks-around-tables） | GFM 表格前后补空行 | 本地 `reference/rumdl/docs/md058.md` |
|
||||
|
||||
两点值得注意：
|
||||
|
||||
1. **没有主流工具自动把 HTML 表格转成 GFM 表格。** 连以“消灭 HTML”为目标的 MD033 都把表格留在
|
||||
“只报告、不修复”的范围里——因为工具作者知道这个转换会弄坏表格。
|
||||
2. **MD056 的自动修复方向与本项目审计相反。** 审计 T002 反对“补空单元格凑齐列数通过语法检查”，
|
||||
因为这可能掩盖静默丢列；MD056 恰恰把补空作为修复手段。借用这类规则时必须关掉它的自动修复，
|
||||
只取检测部分。
|
||||
|
||||
## 7. 与既有材料的关系
|
||||
|
||||
- 审计 T001（`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md` 第 4.4 节）的五步法——容错解析建 DOM、
|
||||
展开 rowspan/colspan 校验二维网格、简单矩形表转 GFM / 复杂表保留 HTML 或 JSON、拆多行、
|
||||
回源确认幻觉——与本次调研的所有正面证据一致，未发现需要修正的点；
|
||||
- [`markdown-cleaning-ecosystem-research-2026-08-20.md`](markdown-cleaning-ecosystem-research-2026-08-20.md)
|
||||
第 5 节的推荐流程（html5lib 容错解析、禁正则、按合并单元格分流）同样得到印证；
|
||||
- 新增的证据是反面案例的具体机制：Turndown 的压扁路径、Docling Markdown 视图的重复路径、
|
||||
Pandoc 手册的方言能力原文、MD056 修复方向与审计相反。
|
||||
|
||||
## 8. 对本项目的待决问题（不是结论）
|
||||
|
||||
以下问题在对应 design 时需要回答，本调研只提供背景：
|
||||
|
||||
1. 简单/复杂表格的分界线，除了“有无合并单元格”，是否还要看单元格内块级元素、嵌套表格和表头层级；
|
||||
2. 复杂表保留的“规范 HTML”具体规范到什么程度（属性白名单？标签重排？缩进策略？）；
|
||||
3. JSON grid 的格式是否对齐 Docling 的 `TableCell`（row_span/col_span/offset 字段），
|
||||
还是自定义 schema——涉及与未来回源 adapter 的成本权衡；
|
||||
4. 无表头表格（Turndown 的回退条件）按哪种流派处理：补合成表头转 GFM，还是保留 HTML。
|
||||
|
||||
## 9. 验证状态
|
||||
|
||||
- Pandoc 手册、Turndown 源码、MinerU README、Docling 官网：2026-08-21 通过网络核实；
|
||||
- docling-core 2.92.0：下载 wheel 解包读源码核实，涉及
|
||||
`MarkdownTableSerializer.serialize`、`TableData.grid`、HTML 序列化器的 span 处理；
|
||||
- rumdl 三条规则：读本地 `reference/rumdl/docs/`（该目录为镜像副本，以 rumdl 上游为准）；
|
||||
- 未验证：各工具在本项目真实数据上的实际表现——需要等对应组件 design 批准后用受控样本测试。
|
||||
@@ -1,170 +0,0 @@
|
||||
# HTML 表格真实数据结构分析：45 份 GovDoc 测试 Markdown
|
||||
|
||||
> 状态：只读分析记录，尚未进入任何 design。文中"建议"部分是候选方向，不是批准的方案。
|
||||
>
|
||||
> 分析日期：2026-08-21。
|
||||
>
|
||||
> 定位：用真实数据回答"我们的 HTML 表格到底长什么样、损坏在哪"，为未来表格清洗 design 提供测量依据，
|
||||
> 并修正 [`html-table-cleaning-ecosystem-research-2026-08-21.md`](html-table-cleaning-ecosystem-research-2026-08-21.md)
|
||||
> 第 8 节中可以用数据回答的待决问题。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
按"展开 rowspan/colspan 后每行列数是否一致"给全部顶层表格分类，**剔除空 `<tr></tr>` 之后**的分布：
|
||||
|
||||
| 分类 | 定义 | 数量 | 占比 |
|
||||
|---|---|---|---|
|
||||
| A 简单矩形表 | 无合并单元格、每行列数一致、无嵌套、无游离内容 | 2474 | 53% |
|
||||
| B 合法合并表 | 有 rowspan/colspan，展开后仍是完整矩形 | 356 | 8% |
|
||||
| C 损坏表 | 参差网格、标签截断、占位冲突等 | 1879 | 40% |
|
||||
|
||||
四个改变预期的发现：
|
||||
|
||||
1. **一半的"损坏"是空 `<tr></tr>` 造成的假象**——剔除后 C 类从 54% 降到 40%；
|
||||
2. **标签不闭合会吞掉半篇文档**——最极端的一个未闭合表格吞了 2.79MB、1294 个表格；
|
||||
3. **参差网格与合并单元格强相关**——六成以上参差表带 span 属性，指向转换器丢失 colspan 标注；
|
||||
4. **无表头是常态**（A 类中 96%），**表格内图片为零**，单元格内管道符为零。
|
||||
|
||||
## 2. 数据范围与方法
|
||||
|
||||
- 输入：`/home/lihaoze/gov_test_data/compare/*/uploads/*.md`，45 份，全程只读；
|
||||
本文只含聚合数字和结构事实，不含任何原文片段；
|
||||
- 分析脚本在会话级临时目录 `/tmp/table-analysis/analyze.py`，未入库（仓库处于文档治理阶段，无源码目录）；
|
||||
本文第 2.1 节的规则描述是复现依据；
|
||||
- 解析器：lxml `HTMLParser(recover=True)`。本地未装 html5lib。
|
||||
|
||||
### 2.1 测量规则
|
||||
|
||||
1. **代码围栏遮蔽**：先标记 ```` ``` ````/`~~~` 围栏内的位置，围栏里的 `<table>` 不计入；
|
||||
2. **片段切分**：对围栏外的 `<table`/`</table>` 事件按嵌套深度配对出顶层片段；悬空的 `</table>`
|
||||
（无对应开启）单独计数并忽略；到文件尾仍未闭合的片段标记 `truncated`；
|
||||
3. **结构分析**：每个片段单独喂给 lxml 容错解析；对每个 `<table>` 展开 rowspan/colspan 建二维占位网格，
|
||||
检测占位冲突，计算每行展开后的列数，收集表头、嵌套、游离文本、块级子标签、空单元格、超长单元格、
|
||||
退化重复等特征；
|
||||
4. **分类谓词**（`truncated` 为片段级标记，其余为表格级）：
|
||||
|
||||
```text
|
||||
A_simple : 非 truncated 且 span_cells=0 且无嵌套 且每行展开列数一致
|
||||
且无占位冲突 且无游离文本 且块级子标签 ⊆ {br}
|
||||
B_span_ok: 非 truncated 且无占位冲突 且每行展开列数一致 且无游离文本
|
||||
C_broken : 其余全部
|
||||
```
|
||||
|
||||
5. **空行修复复核**：把没有任何 `td`/`th` 子元素的 `<tr>` 整行剔除后重新跑同一分类，观察迁移。
|
||||
|
||||
## 3. 总量与原始分类
|
||||
|
||||
- 45 份文档共 7550 个 `<table` 开标签、7155 个闭标签，**缺口 395 个**；
|
||||
- 深度配对得到 4648 个顶层片段；容错解析后共 **4709 个表格元素**（缺口标签导致部分片段解析出多个表格）；
|
||||
- `parse_fail=0`；lxml recover 模式记录 parser 错误的片段仅 30/4648——**recover 解析器的错误日志是弱信号，
|
||||
不能当损坏检测器用**，损坏要靠网格校验发现。
|
||||
|
||||
原始分类（4709 个表格）：
|
||||
|
||||
| 分类 | 数量 | 占比 |
|
||||
|---|---|---|
|
||||
| A_simple | 1921 | 41% |
|
||||
| B_span_ok | 262 | 5% |
|
||||
| C_broken | 2526 | 54% |
|
||||
|
||||
结构特征（占 4709 的比例）：含 span 属性 1891（40%）、参差网格 2506（53%）、有 `<th>` 259（5.5%）、
|
||||
单行表 157、截断片段 67、超长单元格（>500 字符）197、巨型片段（>10KB）70、退化重复内容 33、
|
||||
游离文本 27、占位冲突 14、嵌套表格 5、含块级标签 5、**含图片 0**。
|
||||
|
||||
## 4. 发现一：空 `<tr></tr>` 是最大的单一"假损坏"来源
|
||||
|
||||
完全没有 `td`/`th` 子元素的空行标签在参差表里出现 3000+ 行次。它们把"列数一致的好表"撑成
|
||||
"某些行 0 列"的参差表。
|
||||
|
||||
剔除空行后重新分类：
|
||||
|
||||
| 迁移路径 | 数量 |
|
||||
|---|---|
|
||||
| C_broken → A_simple | 553 |
|
||||
| C_broken → B_span_ok | 94 |
|
||||
|
||||
即分类变为 **A 2474（53%）/ B 356（8%）/ C 1879（40%）**，一条零风险修复救回 13% 的表。
|
||||
空行不含任何内容，剔除是无损的。
|
||||
|
||||
空行出现的位置：表中间 1151 处、表尾 385 处（对修复后仍为 C 的表统计），**没有出现在表头位置**——
|
||||
符合"转换器输出残留"而非"表头占位"的形态。
|
||||
|
||||
## 5. 发现二：标签不闭合会吞掉后续正文
|
||||
|
||||
最极端案例 `001-2/uploads/file_0_PDF.md`：全文件 1315 开 / 1124 闭；从第 1315 行开始的一个未闭合
|
||||
表格把后续 **2,789,092 字符、内部含 1294 个表格**的内容全部吞进一个"顶层片段"。
|
||||
|
||||
全库 395 个闭合缺口意味着：**清洗的第一步不是处理表格，而是安全切分片段**。深度配对在标签缺失时
|
||||
会把正文和后续完整表格归并进一个巨型片段，后续所有基于片段的统计和修改都会失真。
|
||||
|
||||
切分策略的可用锚点：相邻顶层表格之间，2943 对隔着真实文本、1661 对只隔空行——块边界
|
||||
（空行 + 后续正文）在实际数据中是可识别的。
|
||||
|
||||
## 6. 发现三：参差网格与 span 强相关，指向"丢 colspan 标注"
|
||||
|
||||
2506 个参差表中 **1624 个（65%）带 span 属性**——格子内容在、宽度信息没了，是转换器丢失
|
||||
colspan 标注的形态，不全是真缺内容。
|
||||
|
||||
剔除空行后仍参差的 1851 个表，参差发生位置：
|
||||
|
||||
| 位置 | 数量 | 推断成因 |
|
||||
|---|---|---|
|
||||
| 中间各行乱 | 780 | 真·结构损坏或逐行丢标注 |
|
||||
| 表头行比正文长 | 619 | 多级表头被拍平成一行、丢层级 |
|
||||
| 首行短 | 239 | 表头/首行缺格 |
|
||||
| 仅尾部截短 | 213 | 跨页截断尾巴 |
|
||||
|
||||
四类成因不同，修复策略应当不同，但**都不能靠补空单元格自动修**——那正是
|
||||
[`../reference/GOVDOC_SAAS_CLEANING_SCOPE.md`](../reference/GOVDOC_SAAS_CLEANING_SCOPE.md) T002
|
||||
明确反对、且 rumdl MD056 的 auto-fix 方向被本仓库否决的做法。
|
||||
|
||||
## 7. 发现四：无表头是常态，转换障碍集中在 `<br>`
|
||||
|
||||
对修复后仍是 A 类的 2474 个表，检查转 GFM 管道表格的内容障碍：
|
||||
|
||||
| 障碍 | 数量 | 占 A 类 |
|
||||
|---|---|---|
|
||||
| 无 `<th>` 表头 | 2393 | 96% |
|
||||
| 单元格含 `<br>` | 501 | 20% |
|
||||
| 单行表 | 176 | 7% |
|
||||
| 超长单元格（>500 字符） | 54 | 2% |
|
||||
| 单元格文本含 `\|` | 0 | 0% |
|
||||
| 表格内 `<img>` | 0 | 0% |
|
||||
|
||||
转义压力比预期小（管道符、图片都是零），压力集中在**合成表头**和 `<br>` 处理上。
|
||||
|
||||
## 8. 建议的处理策略（候选，未批准）
|
||||
|
||||
三步走，对齐审计 T001 的分流方向，用真实数据修正边界：
|
||||
|
||||
1. **切分**：容错定位 `<table>` 片段；未闭合的在块边界截断并标记 truncated，
|
||||
绝不让深度配对吞正文；
|
||||
2. **无损修复**：只做剔除空 `<tr>` 这一级别的零风险修复（实测救回 13% 的表）；
|
||||
3. **分流**：
|
||||
- A 类（53%）→ 转 GFM：合成表头、`<br>` 转空格、管道符转义兜底；
|
||||
- B 类（8%）→ 保留 HTML，规范化输出；
|
||||
- C 类（40%）→ 不自动修，保留原样 + 按第 6 节四类成因报告分类原因，交人工确认或回源。
|
||||
|
||||
对生态调研第 8 节待决问题的数据回答：
|
||||
|
||||
- 问题 4（无表头表格怎么处理）：数据表明无表头占绝对多数（96%），"补合成表头转 GFM"
|
||||
是主流路径；空表头还是首行充当表头仍是 design 待决；
|
||||
- 问题 1（简单/复杂分界线）：分界线除了合并单元格，必须加"展开后网格是否矩形"——
|
||||
本数据中它是比 span 更强的损坏信号（53% 对 40%）；
|
||||
- 问题 2、3（规范 HTML 程度、JSON schema）本次数据没有新增证据，维持开放。
|
||||
|
||||
## 9. 局限
|
||||
|
||||
- 全部数字只来自这 45 份文档，不得外推为一般结论（CLAUDE.md 第 5 节约束）；
|
||||
- 53/8/40 依赖"剔除空 `<tr>`"这条规则被采纳；不采纳则为 41/5/54；
|
||||
- lxml recover 解析可能自行重排损坏片段，个别表格的行列统计是解析结果而非字节事实；
|
||||
- 巨型片段内部的表格统计（如 001-2/file_0 被吞的 1294 个）已计入总数，但它们在原文中的
|
||||
真实边界未经人工核对；
|
||||
- 参差位置的四分类用的是简单规则（与列数众数比较），是启发式归类，不是语义判断。
|
||||
|
||||
## 10. 验证状态
|
||||
|
||||
- 第 3 至 7 节所有数字：2026-08-21 由只读脚本在真实数据上实际运行得出，脚本未入库；
|
||||
- 分类谓词与第 2.1 节规则描述和脚本逻辑一致，可据此重建等价测量；
|
||||
- 未验证：任何修复或转换策略的实际效果——A/B/C 分流、空行剔除、块边界截断都还没有实现，
|
||||
需要等对应 design 批准后用受控样本测试。
|
||||
@@ -1,492 +0,0 @@
|
||||
# Markdown 清洗生态调研与通用架构建议
|
||||
|
||||
> 状态：调研草稿，尚未批准为项目设计。
|
||||
>
|
||||
> 调研日期：2026-08-20。
|
||||
>
|
||||
> 更新方式：候选工具、许可证、实测结果或项目范围变化时更新；形成实施决定后转写为下一编号 design。
|
||||
|
||||
## 1. 结论先行
|
||||
|
||||
这个项目不应该重新实现一个“正则表达式合集”，也不应该把 Prettier、mdformat、Unstructured 或某个
|
||||
PDF→Markdown 模型直接包装成最终产品。
|
||||
|
||||
现有工具各自只解决问题的一层：
|
||||
|
||||
- Markdown parser/formatter 能统一语法，但无法知道一句话是不是模型幻觉；
|
||||
- HTML 容错解析器能补齐标签，但无法保证补出的表格在业务上正确；
|
||||
- PDF/DOCX 提取器能回源重建，但仍可能 OCR 错误或生成幻觉；
|
||||
- PII 工具能提供候选实体，但无法自动决定跨文档伪名是否应该一致；
|
||||
- 文本质量过滤器能发现重复和低熵，却常以“整篇丢弃”为目标，不适合忠实修复文档。
|
||||
|
||||
因此建议把 `govdoc-md-cleaner` 定位为：
|
||||
|
||||
> **面向多项目的、可审计的文档规范化与派生框架。Markdown 是主要输入输出格式，但核心对象是带来源、
|
||||
> 结构、置信度和问题记录的文档，而不是一串待正则替换的文本。**
|
||||
|
||||
推荐的技术组合是：
|
||||
|
||||
| 层 | 首选候选 | 在本项目中的角色 |
|
||||
|---|---|---|
|
||||
| 核心语言 | Python | 与文档解析、OCR、隐私工具及现有下游生态衔接 |
|
||||
| Markdown 解析 | `markdown-it-py` + GFM 插件 | CommonMark/GFM 结构识别、块级行号映射；不负责语义修复 |
|
||||
| Markdown 输出 | 自有受控 renderer；`mdformat` 只作可选格式化后端 | 保证 profile 输出稳定，避免 formatter 越权改原文 |
|
||||
| 损坏 HTML | `html5lib`，必要时配合 `lxml` tree builder | 按浏览器规则恢复 DOM；恢复动作必须进入审计 |
|
||||
| 回源提取 | Docling 作为默认候选 adapter | PDF/DOCX/图片转结构化文档，保留页码、bbox 和 provenance |
|
||||
| 编码异常 | `ftfy` 作为候选建议器 | 识别/建议 mojibake 修复；默认不静默应用到法律文本 |
|
||||
| 隐私 | Presidio 可选 adapter + 中文自定义 recognizer | 检测、确定性伪名和图片脱敏；不是默认核心依赖 |
|
||||
| 重复/退化 | 自有 detector，参考 DataTrove 指标 | 长行、低熵、重复字符/n-gram、语言突变和固定幻觉模板 |
|
||||
| 多格式转换 | Pandoc 可选 adapter/对照 oracle | DOCX/HTML/Markdown 转换与 AST filter；不作为忠实度权威 |
|
||||
|
||||
`remark/unified` 是 Markdown 原生变换能力最完整的候选，但它会引入 Node.js/TypeScript 运行时；本项目的
|
||||
回源、OCR、中文隐私和下游环境更偏 Python，所以建议把 remark 作为设计参照和交叉验证器，而不是第一版核心。
|
||||
|
||||
这不是最终技术选型。下一步应以本报告为输入编写 design，并用小型 spike 验证关键假设后再批准依赖。
|
||||
|
||||
## 2. 审计告诉我们的真实问题
|
||||
|
||||
本报告以 [45 份 Markdown 清洗审计](../reference/MARKDOWN_CLEANING_AUDIT.md) 为本地事实来源。
|
||||
审计发现的不是单一格式问题，而是至少五个不同层次的问题：
|
||||
|
||||
1. **字节与字符层**：替换字符、私用区字符、NBSP、零宽字符、多余转义；
|
||||
2. **Markdown/HTML 语法层**：标题扁平、悬空链接、损坏 HTML/GFM 表格、极端长行；
|
||||
3. **文档结构层**：段落断裂、阅读顺序错误、页眉页脚混入、图片与表格丢失；
|
||||
4. **内容可信度层**：模型幻觉、退化重复、OCR 语义错误、无法凭 Markdown 恢复的缺失内容；
|
||||
5. **用途与合规层**：对比、RAG、受控忠实版、公开脱敏版对内容保留规则不同。
|
||||
|
||||
其中两个结论直接改变技术路线。
|
||||
|
||||
第一，Markdown parser 解析成功不能作为质量通过条件。CommonMark 明确规定任意字符序列都是合法文档，
|
||||
因此绝大多数“脏 Markdown”仍然可以无报错解析；我们必须建立额外的结构、内容和来源验证器
|
||||
（[CommonMark 0.31.2](https://spec.commonmark.org/0.31.2/)）。
|
||||
|
||||
第二，GFM parser 接受的表格也未必满足我们的忠实度要求。GFM 对正文行缺列会补空单元格，多出的单元格
|
||||
会被忽略；这对法律和金额表格可能造成静默丢失，所以项目必须在 parser 之上做严格矩形网格验证
|
||||
（[GFM 表格规范](https://github.github.io/gfm/#tables-extension-)）。
|
||||
|
||||
## 3. 为什么成熟 formatter 不能直接解决
|
||||
|
||||
### 3.1 Prettier、mdformat
|
||||
|
||||
[Prettier](https://prettier.io/docs/) 和 [mdformat](https://mdformat.readthedocs.io/) 都是成熟的确定性格式化器。
|
||||
它们通过“解析后重新打印”统一标题、列表、换行等书写风格。mdformat 使用 `markdown-it-py`，并提供语法扩展
|
||||
和代码围栏 formatter 插件（[mdformat 插件文档](https://mdformat.readthedocs.io/en/stable/users/plugins.html)）。
|
||||
|
||||
适合：
|
||||
|
||||
- 已经确认语义正确的 Markdown；
|
||||
- 统一输出风格；
|
||||
- 检查幂等性；
|
||||
- Wiki、README 和开发者手写文档。
|
||||
|
||||
不适合直接处理本审计数据：
|
||||
|
||||
- formatter 不知道 `The quick brown fox...` 是幻觉；
|
||||
- 重新打印会扩大 diff，使逐项审计更困难；
|
||||
- 对 raw HTML、损坏表格和未知扩展可能规范化或转义；
|
||||
- 它无法从缺失图片引用恢复资产，也无法回到 PDF bbox。
|
||||
|
||||
建议：只在结构和内容已经通过验证的节点上使用，或作为最终输出的可选 profile；永不直接覆盖原输入。
|
||||
|
||||
### 3.2 markdownlint、remark-lint
|
||||
|
||||
[remark-lint](https://github.com/remarkjs/remark-lint) 有约 70 条可组合规则，能检查标题跳级、硬换行、
|
||||
链接语法和行长等；markdownlint 也有成熟的规则集。这些工具适合开发文档质量门禁，但其规则主要面向
|
||||
作者书写风格，不认识 PDF 页、OCR 置信度、表格合并单元格或业务实体。
|
||||
|
||||
建议：用作本仓 Wiki/README 的 CI，或复用部分规则思想；不作为业务文档清洗引擎。
|
||||
|
||||
## 4. Markdown AST 候选
|
||||
|
||||
### 4.1 `remark` / `unified`
|
||||
|
||||
[remark](https://github.com/remarkjs/remark) 提供 Markdown→mdast→Markdown 的完整插件流水线；
|
||||
[mdast](https://github.com/syntax-tree/mdast) 对 CommonMark、GFM 表格、图片、raw HTML 等节点有稳定模型，
|
||||
unist 生态还提供位置、source extraction、遍历、lint 和 vfile 消息。它是本次调研中最完整的
|
||||
Markdown-native 变换生态。
|
||||
|
||||
优点：
|
||||
|
||||
- parser、AST、visitor、transformer、lint、stringifier 是同一生态；
|
||||
- 节点通常带行、列、offset，适合生成诊断；
|
||||
- GFM、frontmatter、数学、directives 等扩展成熟；
|
||||
- TypeScript 类型和插件边界清晰。
|
||||
|
||||
代价：
|
||||
|
||||
- 核心运行时是 Node.js/ESM；
|
||||
- PDF/DOCX/OCR、中文文本处理和当前下游大多仍在 Python；
|
||||
- 双运行时会增加部署、版本锁定和跨语言 IR 的维护成本。
|
||||
|
||||
判断：如果项目只清洗开发者 Markdown，remark 是首选；对当前“文档回源 + 多项目 profile”目标，第一版
|
||||
不建议为它引入第二套运行时。可把它用于 conformance 对照或以后提供 TypeScript 前端。
|
||||
|
||||
### 4.2 `markdown-it-py` + `mdformat`
|
||||
|
||||
[markdown-it-py](https://markdown-it-py.readthedocs.io/en/latest/) 遵循 CommonMark，支持插件、自定义规则和
|
||||
GFM 相关扩展；Token 的 `map` 字段提供块级起止行号
|
||||
（[Token 文档](https://markdown-it-py.readthedocs.io/en/v4.2.0/_modules/markdown_it/token.html)）。它活跃、
|
||||
MIT、Python 原生，适合本项目第一版。
|
||||
|
||||
局限也需要明确：
|
||||
|
||||
- 它主要是 parser/HTML renderer，不是完整的 Markdown transformation framework；
|
||||
- 行号映射主要在块级，细粒度字符 offset 和跨回源 bbox 仍需我们维护；
|
||||
- raw HTML 会成为特殊 token，表格恢复仍要交给 HTML parser；
|
||||
- CommonMark 合法不等于文档内容可信。
|
||||
|
||||
建议：把它用于“识别现有 Markdown 的结构和边界”，再投影到项目自己的 Document IR；不要直接在 token
|
||||
列表上堆满业务规则。`mdformat` 可为确认安全的 AST 提供稳定输出，但 renderer 行为必须通过回归样本冻结。
|
||||
|
||||
### 4.3 Pandoc
|
||||
|
||||
[Pandoc](https://pandoc.org/MANUAL.html) 使用 reader→AST→writer 架构，Lua/JSON filter 可以按顺序变换 AST
|
||||
（[Pandoc filter 文档](https://pandoc.org/filters.html)）。它的多格式覆盖和长期稳定性很强。
|
||||
|
||||
适合：
|
||||
|
||||
- DOCX、HTML、Markdown 等格式导入导出；
|
||||
- 做第二实现的转换对照；
|
||||
- 用户明确接受 Pandoc 方言规范化的 profile。
|
||||
|
||||
不适合担任忠实版核心：
|
||||
|
||||
- reader/writer round-trip 会改变原始 Markdown 表达；
|
||||
- Pandoc AST 不是为逐字符审计和 PDF bbox 设计的；
|
||||
- 外部二进制与 [GPL-2.0 许可证](https://github.com/jgm/pandoc/blob/main/COPYING.md)需要独立部署评估；
|
||||
- 不能修复不存在于输入中的图片和内容。
|
||||
|
||||
判断：可选 adapter，不作为唯一内部表示。
|
||||
|
||||
### 4.4 Marko、Mistune 等 Python parser
|
||||
|
||||
[Marko](https://marko-py.readthedocs.io/en/latest/) 提供纯 Python CommonMark AST 和扩展机制，Mistune 偏向
|
||||
高速渲染。它们都能用于特定场景，但相较 `markdown-it-py`，当前项目更看重现成插件、维护活跃度、
|
||||
生态采用以及块级 source map。第一轮 spike 不必同时维护三个 Python parser。
|
||||
|
||||
## 5. 损坏 HTML 与表格恢复
|
||||
|
||||
[html5lib](https://html5lib.readthedocs.io/en/stable/) 按 WHATWG 浏览器解析算法处理可能损坏的 HTML，
|
||||
可以输出 ElementTree 或使用 lxml tree builder；[lxml 的 HTML5 接口](https://lxml.de/4.5/apidoc/lxml.html.html5parser.html)
|
||||
也支持 fragment 解析。
|
||||
|
||||
推荐流程：
|
||||
|
||||
1. 从 Markdown AST 中只取 raw HTML fragment，不把整篇 Markdown 当 HTML；
|
||||
2. 保存原 fragment、source span 和哈希；
|
||||
3. 使用 html5lib 容错解析并收集 parser errors；
|
||||
4. 构建显式二维 table grid，展开 `rowspan`/`colspan`；
|
||||
5. 校验每个输出 cell 都能映射到原节点或 source 区域；
|
||||
6. 仅无合并单元格的简单矩形表格输出 GFM；
|
||||
7. 复杂表格输出规范 HTML，并并行保留 JSON grid；
|
||||
8. parser 自动补齐的标签只说明“语法可恢复”，不能自动标为“语义已验证”。
|
||||
|
||||
不能采用旧实现那样用正则匹配 `<table>...</table>`：审计已经证明大量闭合标签缺失，正则既无法正确嵌套，
|
||||
也会把后续正文吞入表格。
|
||||
|
||||
## 6. PDF、DOCX 和图片回源候选
|
||||
|
||||
### 6.1 Docling：默认候选 adapter
|
||||
|
||||
[Docling](https://docling.org/) 支持 PDF、Office、HTML、Markdown、图片等格式，能输出 Markdown 和结构化
|
||||
`DoclingDocument`；后者包含表格、层级、bbox 和 provenance
|
||||
（[DoclingDocument 说明](https://github.com/docling-project/docling/blob/main/docs/concepts/docling_document.md)）。
|
||||
项目是 Python/MIT，OCR 后端可插拔。
|
||||
|
||||
它与审计需求最匹配的不是“Markdown 看起来更漂亮”，而是能先保存结构化、带位置的中间结果，再由我们
|
||||
生成 fidelity/profile 输出。因此建议把 Docling 作为第一批回源 adapter 的基准候选。
|
||||
|
||||
但它仍不能成为无条件真值：OCR、阅读顺序和表格模型都会出错，VLM 路径也可能生成内容。必须在 001、003
|
||||
有原始 PDF 的受控样本上验证字符、数字、表格和图片，不以官方 demo 或总准确率代替本项目测试。
|
||||
|
||||
### 6.2 Unstructured
|
||||
|
||||
[Unstructured partition](https://docs.unstructured.io/open-source/core-functionality/partitioning) 能把多种格式切成
|
||||
`Title`、`NarrativeText`、`ListItem`、`Table` 等元素，一些格式保留页码、坐标和 table HTML，适合作为
|
||||
另一种 source adapter 或元素分类对照。
|
||||
|
||||
其 `cleaners` 不能整体照搬。例如官方实现中的 `clean_dashes` 会替换连字符，`clean_bullets` 会删除项目符号，
|
||||
`clean_non_ascii_chars` 会丢弃非 ASCII 字符；这对中文法律文本、项目编号和列表结构明显过于激进
|
||||
（[cleaners 源码](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py)）。
|
||||
|
||||
判断：可评估 partition/metadata；不采用通用 `clean(...)` 作为默认清洗策略。
|
||||
|
||||
### 6.3 MinerU、Marker、MarkItDown
|
||||
|
||||
- [MinerU](https://github.com/opendatalab/MinerU) 支持 PDF/Office/图片到 Markdown、JSON 和图片资产，能力覆盖广，
|
||||
但本地审计已经展示某些现有解析产物中的幻觉与退化；此外它当前是 Apache-2.0 加附加商业与署名条款，
|
||||
不是无条件的标准 Apache-2.0（[MinerU 许可证](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md)）。
|
||||
- [Marker](https://github.com/datalab-to/marker) 能输出 Markdown、JSON、HTML 和 chunks，也暴露页/块结构；代码为
|
||||
Apache-2.0，但模型权重采用带商业门槛和用途限制的修改版 OpenRAIL-M，必须把代码与模型许可分开审查
|
||||
（[Marker 模型许可证](https://github.com/datalab-to/marker/blob/master/MODEL_LICENSE)）。
|
||||
- [Microsoft MarkItDown](https://github.com/microsoft/markitdown) 是轻量多格式→Markdown 工具，适合低成本文本提取，
|
||||
但它的目标不是页级 provenance、复杂表格忠实恢复或审计账本。
|
||||
|
||||
判断：三者都可成为 benchmark adapter，不应把任何一个输出直接标为 fidelity 真值。第一轮优先比较
|
||||
Docling、MinerU 和 Marker 的结构化 JSON，而不是只比较最终 Markdown 的视觉效果。
|
||||
|
||||
## 7. 编码、内容退化和隐私工具
|
||||
|
||||
### 7.1 `ftfy`
|
||||
|
||||
[ftfy](https://ftfy.readthedocs.io/en/latest/) 用保守启发式修复 Unicode mojibake，目标之一是避免把正常文本
|
||||
误改。它适合发现和建议典型 UTF-8/Windows-1252 误解码。
|
||||
|
||||
边界：
|
||||
|
||||
- 已经变成 `�` 的原字符信息不在字符串中，ftfy 无法凭空恢复；
|
||||
- 私用区字符需要字体或源文件映射；
|
||||
- 中文旧编码误解码和法律文本中的兼容字符仍需专门验证；
|
||||
- 即使候选看起来合理，也要保留 before/after、置信度和规则版本。
|
||||
|
||||
建议：作为 detector/candidate fixer；默认 profile 只自动应用有严格前置条件、通过实体保护检查的修复。
|
||||
|
||||
### 7.2 重复、低熵和幻觉模板
|
||||
|
||||
[DataTrove](https://github.com/huggingface/datatrove) 是大规模文本过滤/去重框架，已有行重复率、长行比例、
|
||||
标点比例、语言分数和 contamination 等统计。它的默认任务是筛掉低质量训练语料，而本项目需要定位并修复
|
||||
文档中的局部区域。
|
||||
|
||||
建议借鉴指标，不把 DataTrove 作为核心依赖：
|
||||
|
||||
- 最大行长与结构白名单；
|
||||
- 字符/短片段 run-length；
|
||||
- 唯一字符、token 和 n-gram 比例；
|
||||
- 压缩率与局部信息熵；
|
||||
- 相邻和非相邻重复块；
|
||||
- 文档主要语言与局部语言突变；
|
||||
- 已知转换器/模型幻觉签名。
|
||||
|
||||
detector 只产生 issue 和范围。没有可信来源时，默认隔离或人工确认，不自动编写替代内容。
|
||||
|
||||
### 7.3 Presidio
|
||||
|
||||
[Presidio](https://microsoft.github.io/presidio/) 支持文本、图片和结构化数据中的 PII 检测与匿名化，并允许使用
|
||||
正则、校验和、上下文、NER 和自定义 recognizer。官方也明确说明自动检测不能保证找到全部敏感信息。
|
||||
|
||||
适合：
|
||||
|
||||
- 作为可选 privacy adapter；
|
||||
- 为中国身份证、统一社会信用代码、手机号、银行账号等实现校验和与上下文 recognizer；
|
||||
- 用 custom operator 实现稳定、按实体区分的伪名；
|
||||
- 把文本、表格单元格和图片脱敏放进同一 profile。
|
||||
|
||||
不适合：
|
||||
|
||||
- 默认把所有候选直接覆盖；
|
||||
- 把不同值统一变成同一 `[PHONE]`/`[ID]`；
|
||||
- 认为通用 NER 已覆盖中文政务/合同实体；
|
||||
- 把 privacy profile 与 fidelity 修复写死在一起。
|
||||
|
||||
## 8. 推荐的领域无关架构
|
||||
|
||||
下面是候选架构，不是已批准契约：
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Markdown / HTML / PDF / DOCX / Image] --> B[Immutable source artifact]
|
||||
B --> C[Preflight detectors]
|
||||
B --> D[Parser / source adapters]
|
||||
C --> E[Issue ledger]
|
||||
D --> F[Document IR]
|
||||
F --> G[Validation and routing]
|
||||
E --> G
|
||||
G --> H[Safe deterministic repair]
|
||||
G --> I[Source-verified repair]
|
||||
G --> J[Quarantine / human review]
|
||||
H --> K[Canonical fidelity document]
|
||||
I --> K
|
||||
J --> K
|
||||
K --> L[Fidelity profile]
|
||||
K --> M[Retrieval profile]
|
||||
K --> N[Compare profile]
|
||||
K --> O[Public/privacy profile]
|
||||
L --> P[Markdown + assets + audit]
|
||||
M --> P
|
||||
N --> P
|
||||
O --> P
|
||||
```
|
||||
|
||||
### 8.1 Immutable source artifact
|
||||
|
||||
任何输入先冻结：原始 bytes、SHA-256、媒体类型、来源 ID 和接收时间。后续全部是新产物，不原地覆盖。
|
||||
只有路径而没有内容哈希不足以复现。
|
||||
|
||||
### 8.2 Document IR
|
||||
|
||||
IR 至少需要表达：
|
||||
|
||||
- block/node 类型、层级和子节点；
|
||||
- 原始 byte/line/column span；
|
||||
- 有源文档时的 page、bbox、source artifact hash；
|
||||
- 文本、表格网格、图片 asset ref 和文档边界；
|
||||
- parser/extractor、版本和置信度；
|
||||
- issue、annotation、repair 和 unresolved 关联。
|
||||
|
||||
不要让 Markdown AST 直接承担全部职责：mdast 或 markdown-it token 不包含完整的 PDF provenance、表格网格、
|
||||
修复证据和多 profile 状态。也不要直接把 DoclingDocument 定为公共契约，否则核心会被某个提取器绑定。
|
||||
|
||||
### 8.3 Detector 与 transformer 分离
|
||||
|
||||
每条规则先检测，再决定是否变换。建议规则声明：
|
||||
|
||||
- `rule_id` 与版本；
|
||||
- 支持的 node/input 类型；
|
||||
- source span 与证据；
|
||||
- safety level；
|
||||
- 是否确定性、幂等、可逆；
|
||||
- 影响文本、数字、实体、结构、资产或下游权重；
|
||||
- 验收 predicate。
|
||||
|
||||
安全级别建议：
|
||||
|
||||
| 级别 | 含义 | 默认行为 |
|
||||
|---|---|---|
|
||||
| `detect_only` | 只能确认异常，不能确认正确内容 | 记录 issue，不修改 |
|
||||
| `deterministic` | 不改变语义、前置条件严格 | 可自动执行并记录 |
|
||||
| `source_verified` | 新内容可回链到可信源区域 | 自动或抽检执行 |
|
||||
| `heuristic` | 有合理推断但可能误伤 | profile 显式开启或人工确认 |
|
||||
| `forbidden_without_source` | 金额、编号、缺失正文等无法猜测 | 隔离/未解决 |
|
||||
|
||||
### 8.4 Canonical fidelity 与 profiles
|
||||
|
||||
核心不应硬编码只有 `clean_fidelity.md` 和 `clean_compare.md`。更通用的方式是先生成 canonical fidelity
|
||||
document，再由 profile 派生：
|
||||
|
||||
| Profile | 目标 | 典型变化 |
|
||||
|---|---|---|
|
||||
| `fidelity` | 法律/业务忠实与可回源 | 只含确定性和源验证修复 |
|
||||
| `retrieval` | 搜索、RAG 和可读分块 | 规范段落、结构化 chunk、保留来源 |
|
||||
| `compare` | 相似度与模板分析 | 页眉页脚降噪、稳定图片 token、模板标注/降权 |
|
||||
| `public` | 可共享样例或外部处理 | 确定性伪名、图片/二维码脱敏、严格日志脱敏 |
|
||||
|
||||
未来项目可以增加自己的 profile；GovDoc 的章节模式、投标模板、compare 权重和中国政务字段放在插件/配置包，
|
||||
不能污染领域无关 core。
|
||||
|
||||
## 9. 建议的第一版产品边界
|
||||
|
||||
第一版不要一开始就做“全自动修复所有 Markdown”。建议逐层交付。
|
||||
|
||||
### M0：只读 audit
|
||||
|
||||
- 接受 Markdown；
|
||||
- 冻结哈希并解析 CommonMark/GFM/raw HTML 边界；
|
||||
- 输出 issue、source span、统计和阻断等级；
|
||||
- 覆盖编码、超长行、重复退化、链接/图片、标题、HTML/GFM 表格和 PII 候选；
|
||||
- 不改输入，不需要 PDF 模型。
|
||||
|
||||
这一阶段可以最早验证规则召回、误报、性能和审计 schema，不把修复风险混进来。
|
||||
|
||||
### M1：安全规范化
|
||||
|
||||
- 只执行严格确定性的 LF、NFC、尾空白、空标题等修复；
|
||||
- 每项变更有 source span 和 before/after hash；
|
||||
- 输出 fidelity Markdown、audit 和 unresolved；
|
||||
- 强制幂等、确定性和原输入不覆盖。
|
||||
|
||||
### M2：结构与回源
|
||||
|
||||
- raw HTML fragment 恢复和 table grid；
|
||||
- Docling source adapter；
|
||||
- 图片 asset store 与 manifest；
|
||||
- 页/区域级 re-extract；
|
||||
- 标题、列表、段落只在来源或高置信结构证据下恢复。
|
||||
|
||||
### M3：多用途 profile
|
||||
|
||||
- `retrieval`、`compare`、`public`；
|
||||
- privacy adapter；
|
||||
- 模板标注/权重和稳定实体/图片 token；
|
||||
- 各 profile 的差异可回链到 fidelity。
|
||||
|
||||
### M4：插件与规模化
|
||||
|
||||
- 稳定 rule/adapter/profile API；
|
||||
- 项目专属配置包；
|
||||
- 并行批处理、缓存、可恢复任务和机器可读报告；
|
||||
- 再评估 CLI、Python SDK、服务接口和跨语言消费。
|
||||
|
||||
## 10. 选型 spike 与验收建议
|
||||
|
||||
进入实现前建议建立下一份 design，并批准两个小型 spike。
|
||||
|
||||
### Spike A：Markdown/HTML 核心
|
||||
|
||||
使用脱敏合成 fixture 和审计列出的结构模式，比较：
|
||||
|
||||
- `markdown-it-py` + 自有 IR/renderer；
|
||||
- remark/mdast 作为对照；
|
||||
- html5lib 与 lxml recover 对损坏 table fragment 的差异。
|
||||
|
||||
至少覆盖：未闭合 table、GFM 多/少列、raw HTML 与 Markdown 交错、代码围栏内伪标签、超长行、中文硬换行、
|
||||
图片 URL、Word `_Toc`、标题断裂和多余转义。
|
||||
|
||||
验收关注：source span 完整率、round-trip 语义一致、没有静默 cell 丢失、幂等性、峰值内存和每 MiB 耗时。
|
||||
|
||||
### Spike B：回源提取
|
||||
|
||||
只在受控环境抽取 001、003 的风险分层页面，对 Docling、MinerU、Marker 做 A/B：
|
||||
|
||||
- 原文字符和关键实体准确率；
|
||||
- 表格网格、合并单元格和阅读顺序；
|
||||
- 图片数量、bbox 和 asset 引用；
|
||||
- 已知幻觉与重复退化命中；
|
||||
- CPU/GPU、耗时、峰值内存、模型版本和许可证约束。
|
||||
|
||||
不能只比较“生成的 Markdown 肉眼是否整齐”，也不能把某个引擎自己的置信度当作金标。
|
||||
|
||||
### 回归体系
|
||||
|
||||
- 公开/合成 fixture 进入 Git，复现结构问题但不包含客户原文；
|
||||
- 真实样本只在外部受控目录运行，以 case/file/page ID 和聚合指标报告；
|
||||
- P0 页面 100% 人工核对，其他页面风险分层抽样；
|
||||
- 金额、日期、项目编号、公司名、身份证候选做前后对账；
|
||||
- 每个 transformer 测幂等、确定性、边界和反例；
|
||||
- fidelity、retrieval、compare、public 分别验收，不能用单一“清洗率”。
|
||||
|
||||
## 11. 明确不建议的路线
|
||||
|
||||
- 不恢复旧版逐行正则清洗器作为默认基线；
|
||||
- 不对原文件直接运行 Prettier/mdformat 并覆盖；
|
||||
- 不用正则解析或补齐 HTML 表格；
|
||||
- 不把 parser 无报错当成 Markdown 正确；
|
||||
- 不对全文执行 `clean_extra_whitespace`、`clean_dashes`、全局 NFKC 或非 ASCII 删除；
|
||||
- 不因重复就删除合同条款，不因语言突变就自动删除段落；
|
||||
- 不用生成模型补写缺失文字、表格单元格或图片说明；
|
||||
- 不让不同实体、图片和缺失区域坍缩成同一个通用 token；
|
||||
- 不把某个 PDF 提取器的 Markdown 直接当权威真值；
|
||||
- 不把 GovDoc 的投标/采购规则写进通用 core。
|
||||
|
||||
## 12. 下一份 design 需要决定的事项
|
||||
|
||||
1. 是否批准“Python core + adapter/profile”方向；
|
||||
2. 第一阶段是否只做 Markdown audit，暂不引入 PDF/OCR 重依赖；
|
||||
3. 内部 IR 的最小字段和版本策略；
|
||||
4. audit/unresolved 的事件粒度与敏感信息保存边界；
|
||||
5. `fidelity`、`retrieval`、`compare`、`public` 哪些进入第一版；
|
||||
6. Markdown dialect 是 CommonMark + GFM，还是还要支持 frontmatter、math、directives；
|
||||
7. Docling/MinerU/Marker 的 benchmark 范围和许可证审查责任；
|
||||
8. 真实数据输出目录、保留周期、人工审核和脱敏规则；
|
||||
9. Python SDK、CLI、配置文件和插件 API 哪些属于首个可交付范围。
|
||||
|
||||
在这些事项获得批准前，本报告只代表调研判断，不代表依赖、schema 或产品行为已经确定。
|
||||
|
||||
## 13. 主要资料来源
|
||||
|
||||
本次优先使用官方文档、规范和上游仓库，GitHub 活跃度与许可证检查日期为 2026-08-20：
|
||||
|
||||
- [CommonMark 规范](https://spec.commonmark.org/0.31.2/)
|
||||
- [GitHub Flavored Markdown 规范](https://github.github.io/gfm/)
|
||||
- [markdown-it-py 文档](https://markdown-it-py.readthedocs.io/en/latest/)
|
||||
- [mdformat 文档](https://mdformat.readthedocs.io/en/stable/)
|
||||
- [remark](https://github.com/remarkjs/remark) 与 [mdast](https://github.com/syntax-tree/mdast)
|
||||
- [Pandoc filters](https://pandoc.org/filters.html)
|
||||
- [Docling](https://docling.org/) 与 [DoclingDocument](https://github.com/docling-project/docling/blob/main/docs/concepts/docling_document.md)
|
||||
- [Unstructured partition/cleaning](https://docs.unstructured.io/open-source/core-functionality/partitioning)
|
||||
- [html5lib](https://html5lib.readthedocs.io/en/stable/)
|
||||
- [ftfy](https://ftfy.readthedocs.io/en/latest/)
|
||||
- [Presidio](https://microsoft.github.io/presidio/)
|
||||
- [DataTrove](https://github.com/huggingface/datatrove)
|
||||
- [MinerU](https://github.com/opendatalab/MinerU)
|
||||
- [Marker](https://github.com/datalab-to/marker)
|
||||
- [MarkItDown](https://github.com/microsoft/markitdown)
|
||||
@@ -1 +0,0 @@
|
||||
"""Repository-local Markdown cleaning reviewer."""
|
||||
@@ -1,40 +0,0 @@
|
||||
import eslint from "@eslint/js";
|
||||
import reactHooks from "eslint-plugin-react-hooks";
|
||||
import reactRefresh from "eslint-plugin-react-refresh";
|
||||
import globals from "globals";
|
||||
import tseslint from "typescript-eslint";
|
||||
|
||||
export default tseslint.config(
|
||||
{ ignores: ["dist", "coverage", "node_modules"] },
|
||||
eslint.configs.recommended,
|
||||
...tseslint.configs.strictTypeChecked,
|
||||
...tseslint.configs.stylisticTypeChecked,
|
||||
{
|
||||
files: ["**/*.{ts,tsx}"],
|
||||
languageOptions: {
|
||||
globals: { ...globals.browser, ...globals.node },
|
||||
parserOptions: {
|
||||
projectService: true,
|
||||
tsconfigRootDir: import.meta.dirname,
|
||||
},
|
||||
},
|
||||
plugins: {
|
||||
"react-hooks": reactHooks,
|
||||
"react-refresh": reactRefresh,
|
||||
},
|
||||
rules: {
|
||||
...reactHooks.configs.flat.recommended.rules,
|
||||
"react-refresh/only-export-components": ["warn", { allowConstantExport: true }],
|
||||
"@typescript-eslint/consistent-type-definitions": ["error", "interface"],
|
||||
"@typescript-eslint/no-confusing-void-expression": "off",
|
||||
"@typescript-eslint/restrict-template-expressions": ["error", { allowNumber: true }],
|
||||
"react-hooks/set-state-in-effect": "off",
|
||||
},
|
||||
},
|
||||
{
|
||||
files: ["tests/**/*.{ts,tsx}"],
|
||||
rules: {
|
||||
"@typescript-eslint/no-non-null-assertion": "off",
|
||||
},
|
||||
},
|
||||
);
|
||||
@@ -1,13 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="color-scheme" content="light" />
|
||||
<title>mdpolish 评审器</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/client/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
-4599
File diff suppressed because it is too large
Load Diff
@@ -1,43 +0,0 @@
|
||||
{
|
||||
"name": "mdpolish-reviewer",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": "^24.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "vite --host 127.0.0.1",
|
||||
"build": "npm run typecheck && vite build",
|
||||
"typecheck": "tsc --noEmit -p tsconfig.json",
|
||||
"lint": "eslint src tests vite.config.ts",
|
||||
"test": "vitest run",
|
||||
"check": "npm run lint && npm run test && npm run build"
|
||||
},
|
||||
"dependencies": {
|
||||
"@codemirror/lang-markdown": "6.5.2",
|
||||
"@codemirror/merge": "6.12.2",
|
||||
"@codemirror/state": "6.7.1",
|
||||
"@codemirror/view": "6.43.9",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "10.0.1",
|
||||
"@testing-library/jest-dom": "7.0.1",
|
||||
"@testing-library/react": "16.3.2",
|
||||
"@types/node": "24.13.3",
|
||||
"@types/react": "19.2.18",
|
||||
"@types/react-dom": "19.2.4",
|
||||
"@vitejs/plugin-react": "6.1.0",
|
||||
"eslint": "10.9.0",
|
||||
"eslint-plugin-react-hooks": "7.1.1",
|
||||
"eslint-plugin-react-refresh": "0.5.4",
|
||||
"globals": "17.11.0",
|
||||
"jsdom": "30.0.1",
|
||||
"typescript": "6.0.3",
|
||||
"typescript-eslint": "8.67.0",
|
||||
"vite": "8.2.2",
|
||||
"vitest": "4.1.11"
|
||||
}
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Read-only artifact adapter and local HTTP service."""
|
||||
|
||||
from reviewer.server.artifacts import ReviewArtifactError, ReviewArtifacts
|
||||
|
||||
__all__ = ["ReviewArtifactError", "ReviewArtifacts"]
|
||||
@@ -1,233 +0,0 @@
|
||||
"""Serve one local review run through a loopback-only read-only HTTP API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import mimetypes
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urlsplit
|
||||
|
||||
from reviewer.server.artifacts import JsonObject, ReviewArtifactError, ReviewArtifacts
|
||||
|
||||
_SECURITY_HEADERS = {
|
||||
"Cache-Control": "no-store",
|
||||
"Content-Security-Policy": (
|
||||
"default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'none'; "
|
||||
"font-src 'self'; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'"
|
||||
),
|
||||
"Referrer-Policy": "no-referrer",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"X-Frame-Options": "DENY",
|
||||
}
|
||||
|
||||
|
||||
class ReviewerHttpServer(ThreadingHTTPServer):
|
||||
"""Threaded local server whose workers never keep process shutdown alive."""
|
||||
|
||||
daemon_threads = True
|
||||
|
||||
|
||||
def _valid_local_request(handler: BaseHTTPRequestHandler) -> bool:
|
||||
host = handler.headers.get("Host")
|
||||
if host is None:
|
||||
return False
|
||||
try:
|
||||
parsed_host = urlsplit(f"//{host}")
|
||||
if parsed_host.username is not None or parsed_host.password is not None:
|
||||
return False
|
||||
if parsed_host.hostname not in {"127.0.0.1", "localhost"}:
|
||||
return False
|
||||
if parsed_host.port is not None and not 0 < parsed_host.port < 65536:
|
||||
return False
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
origin = handler.headers.get("Origin")
|
||||
if origin is None:
|
||||
return True
|
||||
try:
|
||||
parsed_origin = urlsplit(origin)
|
||||
return parsed_origin.scheme == "http" and parsed_origin.netloc == host
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _api_response(repository: ReviewArtifacts, path: str) -> JsonObject:
|
||||
if path == "/api/v1/run":
|
||||
return repository.run_summary()
|
||||
parts = [part for part in path.split("/") if part]
|
||||
try:
|
||||
if len(parts) == 4 and parts[:3] == ["api", "v1", "documents"]:
|
||||
return repository.document_comparison(unquote(parts[3], encoding="utf-8", errors="strict"))
|
||||
if len(parts) == 6 and parts[:3] == ["api", "v1", "documents"] and parts[4] == "components":
|
||||
document_id = unquote(parts[3], encoding="utf-8", errors="strict")
|
||||
try:
|
||||
component_position = int(parts[5])
|
||||
except ValueError:
|
||||
raise ReviewArtifactError("unknown_component", "组件位置不存在。", 404) from None
|
||||
return repository.component_stage(document_id, component_position)
|
||||
except UnicodeDecodeError:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
|
||||
|
||||
def _handler_factory(repository: ReviewArtifacts, static_root: Path) -> type[BaseHTTPRequestHandler]:
|
||||
class ReviewRequestHandler(BaseHTTPRequestHandler):
|
||||
server_version = "mdpolish-reviewer"
|
||||
sys_version = ""
|
||||
|
||||
def log_message(self, format_: str, *args: Any) -> None:
|
||||
del format_, args
|
||||
|
||||
def _headers(self, status: int, content_type: str, content_length: int) -> None:
|
||||
self.send_response(status)
|
||||
for name, value in _SECURITY_HEADERS.items():
|
||||
self.send_header(name, value)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(content_length))
|
||||
self.end_headers()
|
||||
|
||||
def _json(self, status: int, payload: JsonObject, *, head_only: bool) -> None:
|
||||
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||
self._headers(status, "application/json; charset=utf-8", len(content))
|
||||
if not head_only:
|
||||
self.wfile.write(content)
|
||||
|
||||
def _error(self, error: Exception, *, head_only: bool) -> None:
|
||||
if isinstance(error, ReviewArtifactError):
|
||||
status = error.http_status
|
||||
code = error.code
|
||||
message = str(error)
|
||||
else:
|
||||
status = 500
|
||||
code = "internal_error"
|
||||
message = "评审器无法完成该请求。"
|
||||
self._json(status, {"error": {"code": code, "message": message}}, head_only=head_only)
|
||||
|
||||
def _static(self, path: str, *, head_only: bool) -> None:
|
||||
requested = "index.html" if path == "/" else unquote(path[1:], encoding="utf-8", errors="strict")
|
||||
if "\0" in requested:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
candidate = static_root / requested
|
||||
try:
|
||||
if candidate.is_symlink():
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404)
|
||||
resolved = candidate.resolve(strict=True)
|
||||
if not resolved.is_relative_to(static_root) or not resolved.is_file():
|
||||
raise FileNotFoundError
|
||||
except (FileNotFoundError, OSError):
|
||||
if Path(requested).suffix:
|
||||
raise ReviewArtifactError("not_found", "请求的资源不存在。", 404) from None
|
||||
resolved = (static_root / "index.html").resolve(strict=True)
|
||||
content = resolved.read_bytes()
|
||||
content_type = mimetypes.guess_type(resolved.name)[0] or "application/octet-stream"
|
||||
if content_type.startswith("text/") or content_type in {"application/javascript", "application/json"}:
|
||||
content_type += "; charset=utf-8"
|
||||
self._headers(200, content_type, len(content))
|
||||
if not head_only:
|
||||
self.wfile.write(content)
|
||||
|
||||
def _handle(self, method: str) -> None:
|
||||
head_only = method == "HEAD"
|
||||
if method not in {"GET", "HEAD"}:
|
||||
self.send_response(405)
|
||||
for name, value in _SECURITY_HEADERS.items():
|
||||
self.send_header(name, value)
|
||||
self.send_header("Allow", "GET, HEAD")
|
||||
payload: JsonObject = {
|
||||
"error": {"code": "method_not_allowed", "message": "只允许 GET 和 HEAD。"}
|
||||
}
|
||||
content = (json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n").encode()
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(content)))
|
||||
self.end_headers()
|
||||
self.wfile.write(content)
|
||||
return
|
||||
try:
|
||||
if not _valid_local_request(self):
|
||||
raise ReviewArtifactError("invalid_origin", "只接受本机同源请求。", 403)
|
||||
request_path = urlsplit(self.path).path
|
||||
if request_path.startswith("/api/"):
|
||||
self._json(200, _api_response(repository, request_path), head_only=head_only)
|
||||
else:
|
||||
self._static(request_path, head_only=head_only)
|
||||
except Exception as error: # the response intentionally hides unexpected implementation details
|
||||
self._error(error, head_only=head_only)
|
||||
|
||||
def do_GET(self) -> None:
|
||||
self._handle("GET")
|
||||
|
||||
def do_HEAD(self) -> None:
|
||||
self._handle("HEAD")
|
||||
|
||||
def do_POST(self) -> None:
|
||||
self._handle("POST")
|
||||
|
||||
def do_PUT(self) -> None:
|
||||
self._handle("PUT")
|
||||
|
||||
def do_PATCH(self) -> None:
|
||||
self._handle("PATCH")
|
||||
|
||||
def do_DELETE(self) -> None:
|
||||
self._handle("DELETE")
|
||||
|
||||
def do_OPTIONS(self) -> None:
|
||||
self._handle("OPTIONS")
|
||||
|
||||
return ReviewRequestHandler
|
||||
|
||||
|
||||
def create_server(
|
||||
repository: ReviewArtifacts,
|
||||
static_root: Path,
|
||||
*,
|
||||
port: int = 0,
|
||||
) -> ReviewerHttpServer:
|
||||
"""Create, but do not start, the loopback reviewer server."""
|
||||
index = static_root / "index.html"
|
||||
if not index.is_file():
|
||||
raise ReviewArtifactError("missing_build", "未找到前端构建结果，请先运行 npm run build。", 400)
|
||||
return ReviewerHttpServer(("127.0.0.1", port), _handler_factory(repository, static_root.resolve()))
|
||||
|
||||
|
||||
def _arguments(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="只读查看一次 mdpolish 本地清洗运行。")
|
||||
parser.add_argument("--run-dir", required=True, help="一次已发布运行目录的绝对或相对路径")
|
||||
parser.add_argument("--port", type=int, default=0, help="本机端口；默认 0 表示自动选择")
|
||||
arguments = parser.parse_args(argv)
|
||||
if arguments.port < 0 or arguments.port > 65535:
|
||||
parser.error("--port 必须在 0 到 65535 之间")
|
||||
return arguments
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> None:
|
||||
arguments = _arguments(argv)
|
||||
repository = ReviewArtifacts(arguments.run_dir)
|
||||
static_root = Path(__file__).resolve().parents[1] / "dist"
|
||||
try:
|
||||
server = create_server(repository, static_root, port=arguments.port)
|
||||
except OSError as error:
|
||||
raise ReviewArtifactError("server_error", "无法启动本地评审服务。", 500) from error
|
||||
port = server.server_address[1]
|
||||
print(
|
||||
f"mdpolish 评审器已启动：http://127.0.0.1:{port}（{repository.run_id}，"
|
||||
f"{len(repository.documents)} 份文档）",
|
||||
flush=True,
|
||||
)
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except ReviewArtifactError as error:
|
||||
raise SystemExit(f"评审器启动失败：{error}") from None
|
||||
@@ -1,691 +0,0 @@
|
||||
"""Strict adapters from published mdpolish artifacts to the reviewer API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import NoReturn, TypeAlias, cast
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
LocatedChange,
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
ReplayResult,
|
||||
replay_change_chain,
|
||||
)
|
||||
|
||||
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||
_STATUSES = {"success", "failed", "unstable"}
|
||||
_ERROR_STAGES = {"transform", "final_review"}
|
||||
|
||||
|
||||
class ReviewArtifactError(ValueError):
|
||||
"""Published artifacts cannot be exposed as a trustworthy review response."""
|
||||
|
||||
def __init__(self, code: str, message: str, http_status: int = 422) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.http_status = http_status
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ComponentRecord:
|
||||
component_id: str
|
||||
version: str
|
||||
parameters: JsonValue
|
||||
applicability: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ManifestDocument:
|
||||
document_id: str
|
||||
source_label: str
|
||||
status: str
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
change_count: int
|
||||
result_path: str
|
||||
cleaned_path: str | None
|
||||
diff_path: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LocatorDocument:
|
||||
document_id: str
|
||||
source_path: Path
|
||||
input_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ChangeRecord:
|
||||
replay: ReplayChange
|
||||
reason: str
|
||||
recorded_line: int
|
||||
recorded_column: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ResultRecord:
|
||||
status: str
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
changes: tuple[ChangeRecord, ...]
|
||||
errors: tuple[JsonObject, ...]
|
||||
residual_proposals: tuple[JsonObject, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DocumentRecord:
|
||||
manifest: ManifestDocument
|
||||
result: ResultRecord
|
||||
locator: LocatorDocument | None
|
||||
|
||||
|
||||
def _fail(code: str, message: str, http_status: int = 422) -> NoReturn:
|
||||
raise ReviewArtifactError(code, message, http_status)
|
||||
|
||||
|
||||
def _object(value: object, label: str) -> dict[str, object]:
|
||||
if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
|
||||
_fail("invalid_artifact", f"{label} 必须是 JSON 对象。")
|
||||
return cast(dict[str, object], value)
|
||||
|
||||
|
||||
def _array(value: object, label: str) -> list[object]:
|
||||
if not isinstance(value, list):
|
||||
_fail("invalid_artifact", f"{label} 必须是数组。")
|
||||
return cast(list[object], value)
|
||||
|
||||
|
||||
def _string(value: object, label: str, *, allow_empty: bool = False) -> str:
|
||||
if not isinstance(value, str) or "\0" in value or (not allow_empty and not value):
|
||||
_fail("invalid_artifact", f"{label} 必须是字符串。")
|
||||
return value
|
||||
|
||||
|
||||
def _integer(value: object, label: str, *, minimum: int = 0) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < minimum:
|
||||
_fail("invalid_artifact", f"{label} 必须是不小于 {minimum} 的整数。")
|
||||
return value
|
||||
|
||||
|
||||
def _hash(value: object, label: str) -> str:
|
||||
digest = _string(value, label)
|
||||
if len(digest) != 64 or any(character not in "0123456789abcdef" for character in digest):
|
||||
_fail("invalid_artifact", f"{label} 必须是小写 SHA-256 摘要。")
|
||||
return digest
|
||||
|
||||
|
||||
def _status(value: object, label: str) -> str:
|
||||
status = _string(value, label)
|
||||
if status not in _STATUSES:
|
||||
_fail("invalid_artifact", f"{label} 不是已知状态。")
|
||||
return status
|
||||
|
||||
|
||||
def _json_value(value: object, label: str) -> JsonValue:
|
||||
if value is None or isinstance(value, str | int | float | bool):
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return [_json_value(item, label) for item in value]
|
||||
if isinstance(value, dict) and all(isinstance(key, str) for key in value):
|
||||
return {cast(str, key): _json_value(item, label) for key, item in value.items()}
|
||||
_fail("invalid_artifact", f"{label} 包含不支持的 JSON 值。")
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def _read_json(path: Path, label: str) -> dict[str, object]:
|
||||
try:
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
if content.startswith(b"\xef\xbb\xbf"):
|
||||
_fail("invalid_artifact", f"{label} 不能包含 UTF-8 BOM。")
|
||||
if not content.endswith(b"\n"):
|
||||
_fail("invalid_artifact", f"{label} 必须以换行结尾。")
|
||||
try:
|
||||
text = content.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError:
|
||||
_fail("invalid_utf8", f"{label} 不是严格 UTF-8。")
|
||||
try:
|
||||
payload = cast(
|
||||
object,
|
||||
json.loads(
|
||||
text,
|
||||
parse_constant=lambda _value: _fail(
|
||||
"invalid_artifact", f"{label} 不能包含非有限数值。"
|
||||
),
|
||||
),
|
||||
)
|
||||
except json.JSONDecodeError:
|
||||
_fail("invalid_artifact", f"{label} 不是合法 JSON。")
|
||||
return _object(payload, label)
|
||||
|
||||
|
||||
def _read_markdown(path: Path, expected_hash: str, label: str, error_code: str) -> str:
|
||||
try:
|
||||
if path.is_symlink() or not path.is_file():
|
||||
_fail(error_code, f"{label}不是普通文件。", 409)
|
||||
content = path.read_bytes()
|
||||
except OSError:
|
||||
_fail(error_code, f"无法读取{label}。", 409)
|
||||
if sha256(content).hexdigest() != expected_hash:
|
||||
_fail("hash_mismatch", f"{label}的内容哈希已经变化。", 409)
|
||||
try:
|
||||
return content.decode("utf-8", errors="strict")
|
||||
except UnicodeDecodeError:
|
||||
_fail("invalid_utf8", f"{label}不是严格 UTF-8。", 409)
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def _run_path(run_directory: Path, relative_path: str, label: str) -> Path:
|
||||
relative = Path(relative_path)
|
||||
if relative.is_absolute() or "\0" in relative_path:
|
||||
_fail("unsafe_path", f"{label}必须是运行目录内的相对路径。")
|
||||
unresolved = run_directory / relative
|
||||
try:
|
||||
if unresolved.is_symlink() or not unresolved.is_file():
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
resolved = unresolved.resolve(strict=True)
|
||||
except OSError:
|
||||
_fail("missing_artifact", f"无法读取{label}。")
|
||||
if not resolved.is_relative_to(run_directory):
|
||||
_fail("unsafe_path", f"{label}越过了运行目录边界。")
|
||||
return resolved
|
||||
|
||||
|
||||
def _component(value: object, position: int) -> ComponentRecord:
|
||||
item = _object(value, f"pipeline.components[{position}]")
|
||||
return ComponentRecord(
|
||||
component_id=_string(item.get("component_id"), "component_id"),
|
||||
version=_string(item.get("version"), "component version"),
|
||||
parameters=_json_value(item.get("parameters"), "component parameters"),
|
||||
applicability=_string(item.get("applicability"), "component applicability"),
|
||||
)
|
||||
|
||||
|
||||
def _manifest_document(value: object, position: int) -> ManifestDocument:
|
||||
item = _object(value, f"manifest.documents[{position}]")
|
||||
document_id = _string(item.get("document_id"), "document_id")
|
||||
status = _status(item.get("status"), "document status")
|
||||
result_path = _string(item.get("result_path"), "result_path")
|
||||
cleaned_value = item.get("cleaned_path")
|
||||
diff_value = item.get("diff_path")
|
||||
cleaned_path = None if cleaned_value is None else _string(cleaned_value, "cleaned_path")
|
||||
diff_path = None if diff_value is None else _string(diff_value, "diff_path")
|
||||
base = f"documents/{document_id}"
|
||||
if (
|
||||
result_path != f"{base}/result.json"
|
||||
or cleaned_path != (f"{base}/cleaned.md" if status == "success" else None)
|
||||
or diff_path != (f"{base}/changes.diff" if status == "success" else None)
|
||||
):
|
||||
_fail("invalid_artifact", "manifest 文档产物路径不符合 schema 1。")
|
||||
return ManifestDocument(
|
||||
document_id=document_id,
|
||||
source_label=_string(item.get("source_label"), "source_label"),
|
||||
status=status,
|
||||
input_sha256=_hash(item.get("input_sha256"), "input_sha256"),
|
||||
current_sha256=_hash(item.get("current_sha256"), "current_sha256"),
|
||||
change_count=_integer(item.get("change_count"), "change_count"),
|
||||
result_path=result_path,
|
||||
cleaned_path=cleaned_path,
|
||||
diff_path=diff_path,
|
||||
)
|
||||
|
||||
|
||||
def _parse_change(value: object, position: int) -> ChangeRecord:
|
||||
item = _object(value, f"changes[{position}]")
|
||||
proposal = _object(item.get("proposal_ref"), "proposal_ref")
|
||||
span = _object(item.get("span"), "span")
|
||||
location = _object(item.get("location"), "location")
|
||||
start = _integer(span.get("start"), "span.start")
|
||||
end = _integer(span.get("end"), "span.end")
|
||||
if end < start:
|
||||
_fail("invalid_artifact", "change span 必须满足 start <= end。")
|
||||
replay = ReplayChange(
|
||||
component_id=_string(item.get("component_id"), "change component_id"),
|
||||
component_version=_string(item.get("component_version"), "change component_version"),
|
||||
component_position=_integer(item.get("component_position"), "change component_position"),
|
||||
proposal_component_position=_integer(
|
||||
proposal.get("component_position"), "proposal component_position"
|
||||
),
|
||||
proposal_snapshot_sha256=_hash(proposal.get("snapshot_sha256"), "proposal snapshot_sha256"),
|
||||
proposal_index=_integer(proposal.get("proposal_index"), "proposal_index"),
|
||||
edit_index=_integer(item.get("edit_index"), "edit_index"),
|
||||
start=start,
|
||||
end=end,
|
||||
before=_string(item.get("before"), "before", allow_empty=True),
|
||||
after=_string(item.get("after"), "after", allow_empty=True),
|
||||
before_sha256=_hash(item.get("before_sha256"), "before_sha256"),
|
||||
after_sha256=_hash(item.get("after_sha256"), "after_sha256"),
|
||||
)
|
||||
return ChangeRecord(
|
||||
replay=replay,
|
||||
reason=_string(item.get("reason"), "change reason"),
|
||||
recorded_line=_integer(location.get("line"), "location.line", minimum=1),
|
||||
recorded_column=_integer(location.get("column"), "location.column", minimum=1),
|
||||
)
|
||||
|
||||
|
||||
def _parse_error(value: object, position: int) -> JsonObject:
|
||||
item = _object(value, f"errors[{position}]")
|
||||
stage = _string(item.get("stage"), "error stage")
|
||||
if stage not in _ERROR_STAGES:
|
||||
_fail("invalid_artifact", "error stage 不是已知阶段。")
|
||||
return {
|
||||
"component_id": _string(item.get("component_id"), "error component_id"),
|
||||
"component_version": _string(item.get("component_version"), "error component_version"),
|
||||
"component_position": _integer(item.get("component_position"), "error component_position"),
|
||||
"stage": stage,
|
||||
"error_type": _string(item.get("error_type"), "error_type"),
|
||||
"message": _string(item.get("message"), "error message"),
|
||||
}
|
||||
|
||||
|
||||
def _parse_residual(value: object, position: int) -> JsonObject:
|
||||
item = _object(value, f"residual_proposals[{position}]")
|
||||
reference = _object(item.get("proposal_ref"), "residual proposal_ref")
|
||||
proposal = _object(item.get("proposal"), "residual proposal")
|
||||
component_position = _integer(item.get("component_position"), "residual component_position")
|
||||
proposal_snapshot = _hash(proposal.get("snapshot_sha256"), "residual proposal snapshot_sha256")
|
||||
if (
|
||||
_integer(reference.get("component_position"), "residual reference component_position")
|
||||
!= component_position
|
||||
or _hash(reference.get("snapshot_sha256"), "residual reference snapshot_sha256")
|
||||
!= proposal_snapshot
|
||||
):
|
||||
_fail("invalid_artifact", "residual proposal_ref 与候选身份不一致。")
|
||||
_integer(reference.get("proposal_index"), "residual proposal_index")
|
||||
edits = _array(proposal.get("edits"), "residual proposal edits")
|
||||
if not edits:
|
||||
_fail("invalid_artifact", "residual proposal edits 不能为空。")
|
||||
for edit_position, edit_value in enumerate(edits):
|
||||
edit = _object(edit_value, f"residual edit[{edit_position}]")
|
||||
span = _object(edit.get("span"), "residual edit span")
|
||||
if _hash(edit.get("snapshot_sha256"), "residual edit snapshot_sha256") != proposal_snapshot:
|
||||
_fail("invalid_artifact", "residual edit 与候选快照不一致。")
|
||||
start = _integer(span.get("start"), "residual span.start")
|
||||
end = _integer(span.get("end"), "residual span.end")
|
||||
expected = _string(edit.get("expected_text"), "residual expected_text", allow_empty=True)
|
||||
_string(edit.get("replacement"), "residual replacement", allow_empty=True)
|
||||
if end < start or len(expected) != end - start:
|
||||
_fail("invalid_artifact", "residual edit 范围与 expected_text 不一致。")
|
||||
return {
|
||||
"component_id": _string(item.get("component_id"), "residual component_id"),
|
||||
"component_version": _string(item.get("component_version"), "residual component_version"),
|
||||
"component_position": component_position,
|
||||
"reason": _string(proposal.get("reason"), "residual reason"),
|
||||
"edit_count": len(edits),
|
||||
}
|
||||
|
||||
|
||||
def _component_json(component: ComponentRecord, position: int, change_count: int) -> JsonObject:
|
||||
return {
|
||||
"component_position": position,
|
||||
"component_id": component.component_id,
|
||||
"version": component.version,
|
||||
"parameters": component.parameters,
|
||||
"applicability": component.applicability,
|
||||
"change_count": change_count,
|
||||
}
|
||||
|
||||
|
||||
class ReviewArtifacts:
|
||||
"""Validated, read-only view over one explicitly selected run directory."""
|
||||
|
||||
def __init__(self, run_directory: str | Path) -> None:
|
||||
requested = Path(run_directory)
|
||||
try:
|
||||
if requested.is_symlink() or not requested.is_dir():
|
||||
_fail("missing_run", "指定的运行目录不存在或不是普通目录。", 400)
|
||||
self.run_directory = requested.resolve(strict=True)
|
||||
except OSError:
|
||||
_fail("missing_run", "无法读取指定的运行目录。", 400)
|
||||
|
||||
manifest = _read_json(_run_path(self.run_directory, "manifest.json", "manifest.json"), "manifest.json")
|
||||
if manifest.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 manifest.json schema 1。", 409)
|
||||
run = _object(manifest.get("run"), "manifest.run")
|
||||
pipeline = _object(manifest.get("pipeline"), "manifest.pipeline")
|
||||
summary = _object(manifest.get("summary"), "manifest.summary")
|
||||
self.run_id = _string(run.get("run_id"), "run_id")
|
||||
self.run_json: JsonObject = {
|
||||
"run_id": self.run_id,
|
||||
"run_date": _string(run.get("run_date"), "run_date"),
|
||||
"status": _status(run.get("status"), "run status"),
|
||||
"started_at_utc": _string(run.get("started_at_utc"), "started_at_utc"),
|
||||
"completed_at_utc": _string(run.get("completed_at_utc"), "completed_at_utc"),
|
||||
"retention_until": _string(run.get("retention_until"), "retention_until"),
|
||||
}
|
||||
self.components = tuple(
|
||||
_component(value, position)
|
||||
for position, value in enumerate(_array(pipeline.get("components"), "pipeline.components"))
|
||||
)
|
||||
manifest_documents = tuple(
|
||||
_manifest_document(value, position)
|
||||
for position, value in enumerate(_array(manifest.get("documents"), "manifest.documents"))
|
||||
)
|
||||
if len({item.document_id for item in manifest_documents}) != len(manifest_documents):
|
||||
_fail("invalid_artifact", "manifest 中的 document_id 必须唯一。")
|
||||
self.summary_json: JsonObject = {
|
||||
"document_count": _integer(summary.get("document_count"), "summary.document_count"),
|
||||
"success_count": _integer(summary.get("success_count"), "summary.success_count"),
|
||||
"failed_count": _integer(summary.get("failed_count"), "summary.failed_count"),
|
||||
"unstable_count": _integer(summary.get("unstable_count"), "summary.unstable_count"),
|
||||
"change_count": _integer(summary.get("change_count"), "summary.change_count"),
|
||||
}
|
||||
expected_summary: JsonObject = {
|
||||
"document_count": len(manifest_documents),
|
||||
"success_count": sum(item.status == "success" for item in manifest_documents),
|
||||
"failed_count": sum(item.status == "failed" for item in manifest_documents),
|
||||
"unstable_count": sum(item.status == "unstable" for item in manifest_documents),
|
||||
"change_count": sum(item.change_count for item in manifest_documents),
|
||||
}
|
||||
if self.summary_json != expected_summary:
|
||||
_fail("invalid_artifact", "manifest 汇总与文档索引不一致。")
|
||||
expected_status = (
|
||||
"failed"
|
||||
if expected_summary["failed_count"]
|
||||
else "unstable"
|
||||
if expected_summary["unstable_count"]
|
||||
else "success"
|
||||
)
|
||||
if self.run_json["status"] != expected_status:
|
||||
_fail("invalid_artifact", "manifest 运行状态与文档状态不一致。")
|
||||
|
||||
locator_path = self.run_directory / "review-locator.json"
|
||||
locators: tuple[LocatorDocument, ...] | None = None
|
||||
self.original_run_location_changed = False
|
||||
if locator_path.exists():
|
||||
locator = _read_json(
|
||||
_run_path(self.run_directory, "review-locator.json", "review-locator.json"),
|
||||
"review-locator.json",
|
||||
)
|
||||
if locator.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 review-locator.json schema 1。", 409)
|
||||
locator_run = _object(locator.get("run"), "review locator run")
|
||||
if (
|
||||
_string(locator_run.get("run_id"), "locator run_id") != self.run_id
|
||||
or locator_run.get("manifest_path") != "manifest.json"
|
||||
):
|
||||
_fail("invalid_artifact", "review locator 与 manifest 运行身份不一致。")
|
||||
recorded_directory_text = _string(locator_run.get("run_directory"), "locator run_directory")
|
||||
recorded_directory = Path(recorded_directory_text)
|
||||
if not recorded_directory.is_absolute() or str(recorded_directory) != str(recorded_directory.resolve()):
|
||||
_fail("invalid_artifact", "locator run_directory 必须是绝对解析路径。")
|
||||
self.original_run_location_changed = recorded_directory != self.run_directory
|
||||
parsed_locators: list[LocatorDocument] = []
|
||||
for position, value in enumerate(_array(locator.get("documents"), "locator documents")):
|
||||
item = _object(value, f"locator.documents[{position}]")
|
||||
source_text = _string(item.get("source_path"), "source_path")
|
||||
source_path = Path(source_text)
|
||||
if not source_path.is_absolute() or str(source_path) != str(source_path.resolve()):
|
||||
_fail("invalid_artifact", "source_path 必须是绝对解析路径。")
|
||||
parsed_locators.append(
|
||||
LocatorDocument(
|
||||
document_id=_string(item.get("document_id"), "locator document_id"),
|
||||
source_path=source_path,
|
||||
input_sha256=_hash(item.get("input_sha256"), "locator input_sha256"),
|
||||
)
|
||||
)
|
||||
locators = tuple(parsed_locators)
|
||||
if len(locators) != len(manifest_documents):
|
||||
_fail("invalid_artifact", "review locator 文档数量与 manifest 不一致。")
|
||||
for indexed, located in zip(manifest_documents, locators, strict=True):
|
||||
if indexed.document_id != located.document_id or indexed.input_sha256 != located.input_sha256:
|
||||
_fail("invalid_artifact", "review locator 文档身份与 manifest 不一致。")
|
||||
|
||||
records: list[DocumentRecord] = []
|
||||
for position, manifest_document in enumerate(manifest_documents):
|
||||
result = self._parse_result(manifest_document)
|
||||
records.append(
|
||||
DocumentRecord(
|
||||
manifest=manifest_document,
|
||||
result=result,
|
||||
locator=None if locators is None else locators[position],
|
||||
)
|
||||
)
|
||||
self.documents = tuple(records)
|
||||
self.documents_by_id = {item.manifest.document_id: item for item in self.documents}
|
||||
|
||||
def _parse_result(self, manifest: ManifestDocument) -> ResultRecord:
|
||||
payload = _read_json(_run_path(self.run_directory, manifest.result_path, "result.json"), "result.json")
|
||||
if payload.get("schema_version") != 1:
|
||||
_fail("unsupported_schema", "只支持 result.json schema 1。", 409)
|
||||
document = _object(payload.get("document"), "result.document")
|
||||
if (
|
||||
_string(document.get("document_id"), "result document_id") != manifest.document_id
|
||||
or _string(document.get("source_label"), "result source_label") != manifest.source_label
|
||||
):
|
||||
_fail("invalid_artifact", "result 与 manifest 文档身份不一致。")
|
||||
status = _status(payload.get("status"), "result status")
|
||||
input_hash = _hash(payload.get("input_sha256"), "result input_sha256")
|
||||
current_hash = _hash(payload.get("current_sha256"), "result current_sha256")
|
||||
changes = tuple(
|
||||
_parse_change(value, position)
|
||||
for position, value in enumerate(_array(payload.get("changes"), "result changes"))
|
||||
)
|
||||
errors = tuple(
|
||||
_parse_error(value, position)
|
||||
for position, value in enumerate(_array(payload.get("errors"), "result errors"))
|
||||
)
|
||||
residuals = tuple(
|
||||
_parse_residual(value, position)
|
||||
for position, value in enumerate(_array(payload.get("residual_proposals"), "result residual_proposals"))
|
||||
)
|
||||
output = _object(payload.get("output"), "result output")
|
||||
cleaned = output.get("cleaned_path")
|
||||
diff = output.get("diff_path")
|
||||
expected_cleaned: object = "cleaned.md" if status == "success" else None
|
||||
expected_diff: object = "changes.diff" if status == "success" else None
|
||||
if cleaned != expected_cleaned or diff != expected_diff:
|
||||
_fail("invalid_artifact", "result 状态与输出路径不一致。")
|
||||
if (
|
||||
status != manifest.status
|
||||
or input_hash != manifest.input_sha256
|
||||
or current_hash != manifest.current_sha256
|
||||
or len(changes) != manifest.change_count
|
||||
):
|
||||
_fail("invalid_artifact", "result 与 manifest 文档索引不一致。")
|
||||
if status == "success" and (errors or residuals):
|
||||
_fail("invalid_artifact", "success 文档不能包含错误或残留候选。")
|
||||
if status == "failed" and (not errors or residuals):
|
||||
_fail("invalid_artifact", "failed 文档的错误或残留状态不合法。")
|
||||
if status == "unstable" and (errors or not residuals):
|
||||
_fail("invalid_artifact", "unstable 文档的错误或残留状态不合法。")
|
||||
for change in changes:
|
||||
position = change.replay.component_position
|
||||
if position >= len(self.components):
|
||||
_fail("invalid_artifact", "change 没有对应的流水线组件。")
|
||||
component = self.components[position]
|
||||
if (
|
||||
change.replay.component_id != component.component_id
|
||||
or change.replay.component_version != component.version
|
||||
):
|
||||
_fail("invalid_artifact", "change 身份与流水线组件不一致。")
|
||||
for evidence in (*errors, *residuals):
|
||||
position_value = evidence["component_position"]
|
||||
component_id = evidence["component_id"]
|
||||
component_version = evidence["component_version"]
|
||||
if not isinstance(position_value, int) or position_value >= len(self.components):
|
||||
_fail("invalid_artifact", "审计证据没有对应的流水线组件。")
|
||||
component = self.components[position_value]
|
||||
if component_id != component.component_id or component_version != component.version:
|
||||
_fail("invalid_artifact", "审计证据身份与流水线组件不一致。")
|
||||
if manifest.diff_path is not None:
|
||||
_run_path(self.run_directory, manifest.diff_path, "changes.diff")
|
||||
return ResultRecord(status, input_hash, current_hash, changes, errors, residuals)
|
||||
|
||||
def _source(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||
if record.locator is None:
|
||||
return None, "这次历史运行没有 review-locator.json，无法定位完整原文。"
|
||||
try:
|
||||
return (
|
||||
_read_markdown(
|
||||
record.locator.source_path,
|
||||
record.manifest.input_sha256,
|
||||
"原文",
|
||||
"unavailable_source",
|
||||
),
|
||||
None,
|
||||
)
|
||||
except ReviewArtifactError as error:
|
||||
return None, str(error)
|
||||
|
||||
def _cleaned(self, record: DocumentRecord) -> tuple[str | None, str | None]:
|
||||
if record.manifest.status != "success" or record.manifest.cleaned_path is None:
|
||||
return None, None
|
||||
try:
|
||||
path = _run_path(self.run_directory, record.manifest.cleaned_path, "cleaned.md")
|
||||
return (
|
||||
_read_markdown(path, record.manifest.current_sha256, "清洗结果", "missing_artifact"),
|
||||
None,
|
||||
)
|
||||
except ReviewArtifactError as error:
|
||||
return None, str(error)
|
||||
|
||||
def _replay(self, record: DocumentRecord, original: str, cleaned: str | None) -> ReplayResult:
|
||||
try:
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=record.result.input_sha256,
|
||||
components=tuple(ReplayComponent(item.component_id, item.version) for item in self.components),
|
||||
changes=tuple(item.replay for item in record.result.changes),
|
||||
current_sha256=record.result.current_sha256,
|
||||
current_markdown=cleaned,
|
||||
include_zero_change_stages=record.result.status == "success",
|
||||
)
|
||||
except ReplayError as error:
|
||||
raise ReviewArtifactError("untrusted_replay", f"产物无法可信重放：{error}", 409) from error
|
||||
for recorded, located in zip(record.result.changes, replayed.changes, strict=True):
|
||||
if recorded.recorded_line != located.line or recorded.recorded_column != located.column:
|
||||
_fail("untrusted_replay", "产物记录的位置与重放快照不一致。", 409)
|
||||
return replayed
|
||||
|
||||
@staticmethod
|
||||
def _summary_json(
|
||||
record: DocumentRecord,
|
||||
original: str | None,
|
||||
source_error: str | None,
|
||||
cleaned: str | None,
|
||||
output_error: str | None,
|
||||
) -> JsonObject:
|
||||
return {
|
||||
"document_id": record.manifest.document_id,
|
||||
"source_label": record.manifest.source_label,
|
||||
"status": record.manifest.status,
|
||||
"input_sha256": record.manifest.input_sha256,
|
||||
"current_sha256": record.manifest.current_sha256,
|
||||
"change_count": record.manifest.change_count,
|
||||
"source_available": original is not None,
|
||||
"output_available": cleaned is not None,
|
||||
"availability_error": source_error or output_error,
|
||||
}
|
||||
|
||||
def _summary(self, record: DocumentRecord) -> JsonObject:
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
return self._summary_json(record, original, source_error, cleaned, output_error)
|
||||
|
||||
def _component_summaries(self, changes: tuple[ChangeRecord, ...]) -> list[JsonValue]:
|
||||
counts = [0] * len(self.components)
|
||||
for change in changes:
|
||||
if 0 <= change.replay.component_position < len(counts):
|
||||
counts[change.replay.component_position] += 1
|
||||
return [_component_json(item, position, counts[position]) for position, item in enumerate(self.components)]
|
||||
|
||||
@staticmethod
|
||||
def _change_json(recorded: ChangeRecord, located: LocatedChange | None) -> JsonObject:
|
||||
change = recorded.replay
|
||||
return {
|
||||
"component_id": change.component_id,
|
||||
"component_version": change.component_version,
|
||||
"component_position": change.component_position,
|
||||
"proposal_ref": {
|
||||
"component_position": change.proposal_component_position,
|
||||
"snapshot_sha256": change.proposal_snapshot_sha256,
|
||||
"proposal_index": change.proposal_index,
|
||||
},
|
||||
"edit_index": change.edit_index,
|
||||
"reason": recorded.reason,
|
||||
"location": {
|
||||
"line": recorded.recorded_line if located is None else located.line,
|
||||
"column": recorded.recorded_column if located is None else located.column,
|
||||
},
|
||||
"editor_range": (
|
||||
None if located is None else {"start": located.editor_start, "end": located.editor_end}
|
||||
),
|
||||
"before": change.before,
|
||||
"after": change.after,
|
||||
}
|
||||
|
||||
def run_summary(self) -> JsonObject:
|
||||
all_changes = tuple(change for document in self.documents for change in document.result.changes)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": self.run_json,
|
||||
"components": self._component_summaries(all_changes),
|
||||
"documents": [self._summary(item) for item in self.documents],
|
||||
"summary": self.summary_json,
|
||||
"original_run_location_changed": self.original_run_location_changed,
|
||||
}
|
||||
|
||||
def _document(self, document_id: str) -> DocumentRecord:
|
||||
record = self.documents_by_id.get(document_id)
|
||||
if record is None:
|
||||
_fail("unknown_document", "文档不存在。", 404)
|
||||
return record
|
||||
|
||||
def document_comparison(self, document_id: str) -> JsonObject:
|
||||
record = self._document(document_id)
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
replayed = None if original is None else self._replay(record, original, cleaned)
|
||||
located = () if replayed is None else replayed.changes
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"document": self._summary_json(record, original, source_error, cleaned, output_error),
|
||||
"components": self._component_summaries(record.result.changes),
|
||||
"original_markdown": original,
|
||||
"cleaned_markdown": cleaned,
|
||||
"changes": [
|
||||
self._change_json(change, located[position] if position < len(located) else None)
|
||||
for position, change in enumerate(record.result.changes)
|
||||
],
|
||||
"errors": list(record.result.errors),
|
||||
"residual_proposals": list(record.result.residual_proposals),
|
||||
}
|
||||
|
||||
def component_stage(self, document_id: str, component_position: int) -> JsonObject:
|
||||
record = self._document(document_id)
|
||||
if record.manifest.status != "success":
|
||||
_fail("stage_unavailable", "只有 success 文档具有完整组件阶段。", 409)
|
||||
if component_position < 0 or component_position >= len(self.components):
|
||||
_fail("unknown_component", "组件位置不存在。", 404)
|
||||
original, source_error = self._source(record)
|
||||
cleaned, output_error = self._cleaned(record)
|
||||
if original is None or cleaned is None:
|
||||
_fail("comparison_unavailable", source_error or output_error or "组件阶段不可用。", 409)
|
||||
replayed = self._replay(record, original, cleaned)
|
||||
stage = replayed.stages[component_position]
|
||||
component = self.components[component_position]
|
||||
recorded_changes = tuple(
|
||||
item for item in record.result.changes if item.replay.component_position == component_position
|
||||
)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"document_id": record.manifest.document_id,
|
||||
"component": _component_json(component, component_position, len(stage.changes)),
|
||||
"before_sha256": stage.before_sha256,
|
||||
"after_sha256": stage.after_sha256,
|
||||
"before_markdown": stage.before_markdown,
|
||||
"after_markdown": stage.after_markdown,
|
||||
"changes": [
|
||||
self._change_json(change, located)
|
||||
for change, located in zip(recorded_changes, stage.changes, strict=True)
|
||||
],
|
||||
}
|
||||
@@ -1,378 +0,0 @@
|
||||
import { lazy, Suspense, useEffect, useMemo, useState } from "react";
|
||||
|
||||
import type {
|
||||
ChangeDetail,
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunStatus,
|
||||
RunSummaryResponse,
|
||||
} from "../shared/api.js";
|
||||
import { fetchComponentStage, fetchDocument, fetchRun } from "./api-client.js";
|
||||
|
||||
const DiffView = lazy(async () => {
|
||||
const module = await import("./DiffView.js");
|
||||
return { default: module.DiffView };
|
||||
});
|
||||
|
||||
interface AsyncState<T> {
|
||||
loading: boolean;
|
||||
value: T | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
const emptyState = <T,>(): AsyncState<T> => ({ loading: true, value: null, error: null });
|
||||
|
||||
function statusLabel(status: RunStatus): string {
|
||||
switch (status) {
|
||||
case "success":
|
||||
return "成功";
|
||||
case "failed":
|
||||
return "失败";
|
||||
case "unstable":
|
||||
return "不稳定";
|
||||
}
|
||||
}
|
||||
|
||||
function shortHash(hash: string): string {
|
||||
return `${hash.slice(0, 8)}…${hash.slice(-6)}`;
|
||||
}
|
||||
|
||||
function ErrorPanel({ message }: { message: string }) {
|
||||
return (
|
||||
<div className="state-panel state-panel--error" role="alert">
|
||||
<span className="eyebrow">无法显示</span>
|
||||
<p>{message}</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function LoadingPanel() {
|
||||
return (
|
||||
<div className="state-panel" role="status">
|
||||
<span className="loading-dot" />
|
||||
<p>正在校验本地运行产物…</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ChangeList({
|
||||
changes,
|
||||
onSelect,
|
||||
canJump,
|
||||
}: {
|
||||
changes: ChangeDetail[];
|
||||
onSelect: (change: ChangeDetail) => void;
|
||||
canJump: boolean;
|
||||
}) {
|
||||
if (changes.length === 0) {
|
||||
return <p className="quiet-message">这个组件运行过，但没有修改当前文档。</p>;
|
||||
}
|
||||
return (
|
||||
<ol className="change-list">
|
||||
{changes.map((change) => (
|
||||
<li key={`${change.proposal_ref.snapshot_sha256}-${change.proposal_ref.proposal_index}-${change.edit_index}`}>
|
||||
<button type="button" onClick={() => onSelect(change)} disabled={!canJump}>
|
||||
<span className="change-location">
|
||||
第 {change.location.line} 行，第 {change.location.column} 列 · 候选
|
||||
{change.proposal_ref.proposal_index + 1} / 编辑 {change.edit_index + 1}
|
||||
</span>
|
||||
<strong>{change.reason}</strong>
|
||||
<span className="change-sample">
|
||||
<del>{change.before || "∅"}</del>
|
||||
<span aria-hidden="true">→</span>
|
||||
<ins>{change.after || "∅"}</ins>
|
||||
</span>
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
);
|
||||
}
|
||||
|
||||
export function App() {
|
||||
const [runState, setRunState] = useState<AsyncState<RunSummaryResponse>>(emptyState);
|
||||
const [selectedDocument, setSelectedDocument] = useState<string | null>(null);
|
||||
const [documentState, setDocumentState] = useState<AsyncState<DocumentComparisonResponse>>({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: null,
|
||||
});
|
||||
const [selectedComponent, setSelectedComponent] = useState<number | null>(null);
|
||||
const [stageState, setStageState] = useState<AsyncState<ComponentStageResponse>>({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: null,
|
||||
});
|
||||
const [focusRange, setFocusRange] = useState<{ start: number; end: number } | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
fetchRun(controller.signal)
|
||||
.then((run) => {
|
||||
setRunState({ loading: false, value: run, error: null });
|
||||
setSelectedDocument(run.documents[0]?.document_id ?? null);
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setRunState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取运行摘要。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
setSelectedComponent(null);
|
||||
setFocusRange(null);
|
||||
if (selectedDocument === null) {
|
||||
setDocumentState({ loading: false, value: null, error: null });
|
||||
return undefined;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
setDocumentState(emptyState());
|
||||
fetchDocument(selectedDocument, controller.signal)
|
||||
.then((document) => setDocumentState({ loading: false, value: document, error: null }))
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setDocumentState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取文档。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [selectedDocument]);
|
||||
|
||||
useEffect(() => {
|
||||
if (selectedDocument === null || selectedComponent === null) {
|
||||
setStageState({ loading: false, value: null, error: null });
|
||||
return undefined;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
setStageState(emptyState());
|
||||
fetchComponentStage(selectedDocument, selectedComponent, controller.signal)
|
||||
.then((stage) => setStageState({ loading: false, value: stage, error: null }))
|
||||
.catch((error: unknown) => {
|
||||
if (!controller.signal.aborted) {
|
||||
setStageState({
|
||||
loading: false,
|
||||
value: null,
|
||||
error: error instanceof Error ? error.message : "无法读取组件阶段。",
|
||||
});
|
||||
}
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [selectedDocument, selectedComponent]);
|
||||
|
||||
const visibleChanges = useMemo(() => {
|
||||
const document = documentState.value;
|
||||
if (document === null) {
|
||||
return [];
|
||||
}
|
||||
if (selectedComponent === null) {
|
||||
return document.changes;
|
||||
}
|
||||
return document.changes.filter((change) => change.component_position === selectedComponent);
|
||||
}, [documentState.value, selectedComponent]);
|
||||
|
||||
const selectChange = (change: ChangeDetail): void => {
|
||||
setSelectedComponent(change.component_position);
|
||||
setFocusRange(change.editor_range);
|
||||
};
|
||||
|
||||
if (runState.loading) {
|
||||
return <LoadingPanel />;
|
||||
}
|
||||
if (runState.error !== null || runState.value === null) {
|
||||
return <ErrorPanel message={runState.error ?? "运行摘要为空。"} />;
|
||||
}
|
||||
|
||||
const run = runState.value;
|
||||
const document = documentState.value;
|
||||
const selectedSummary = run.documents.find((item) => item.document_id === selectedDocument);
|
||||
const selectedStage = stageState.value;
|
||||
const canCompare =
|
||||
document?.document.status === "success" &&
|
||||
document.document.source_available &&
|
||||
document.document.output_available &&
|
||||
document.original_markdown !== null &&
|
||||
document.cleaned_markdown !== null;
|
||||
const beforeText = selectedStage?.before_markdown ?? document?.original_markdown ?? "";
|
||||
const afterText = selectedStage?.after_markdown ?? document?.cleaned_markdown ?? "";
|
||||
const beforeLabel = selectedStage === null ? "清洗前" : `组件 ${selectedStage.component.component_position + 1} 执行前`;
|
||||
const afterLabel = selectedStage === null ? "清洗后" : `组件 ${selectedStage.component.component_position + 1} 执行后`;
|
||||
|
||||
return (
|
||||
<div className="app-shell">
|
||||
<header className="topbar">
|
||||
<div>
|
||||
<span className="brand-mark">md</span>
|
||||
<div>
|
||||
<p className="eyebrow">本地清洗评审器</p>
|
||||
<h1>{run.run.run_id}</h1>
|
||||
</div>
|
||||
</div>
|
||||
<div className="run-facts">
|
||||
<span className={`status status--${run.run.status}`}>{statusLabel(run.run.status)}</span>
|
||||
<span>{run.summary.document_count} 份文档</span>
|
||||
<span>{run.summary.change_count} 条修改</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
{run.original_run_location_changed ? (
|
||||
<div className="notice" role="status">
|
||||
这次运行目录已被移动；评审器使用你本次指定的目录读取产物。
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="layout">
|
||||
<aside className="sidebar" aria-label="运行导航">
|
||||
<section>
|
||||
<div className="section-heading">
|
||||
<h2>文档</h2>
|
||||
<span>{run.documents.length}</span>
|
||||
</div>
|
||||
<nav className="document-list" aria-label="文档列表">
|
||||
{run.documents.map((item) => (
|
||||
<button
|
||||
type="button"
|
||||
key={item.document_id}
|
||||
className={item.document_id === selectedDocument ? "is-active" : ""}
|
||||
onClick={() => setSelectedDocument(item.document_id)}
|
||||
aria-current={item.document_id === selectedDocument ? "page" : undefined}
|
||||
>
|
||||
<span className={`status-dot status-dot--${item.status}`} />
|
||||
<span>
|
||||
<strong>{item.source_label}</strong>
|
||||
<small>{item.change_count} 条修改</small>
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</nav>
|
||||
</section>
|
||||
|
||||
<section className="component-section">
|
||||
<div className="section-heading">
|
||||
<h2>组件时间线</h2>
|
||||
<button
|
||||
type="button"
|
||||
className="text-button"
|
||||
onClick={() => {
|
||||
setSelectedComponent(null);
|
||||
setFocusRange(null);
|
||||
}}
|
||||
disabled={selectedComponent === null}
|
||||
>
|
||||
查看总结果
|
||||
</button>
|
||||
</div>
|
||||
<ol className="component-list">
|
||||
{(document?.components ?? run.components).map((component) => (
|
||||
<li key={component.component_id}>
|
||||
<button
|
||||
type="button"
|
||||
className={component.component_position === selectedComponent ? "is-active" : ""}
|
||||
onClick={() => {
|
||||
setSelectedComponent(component.component_position);
|
||||
setFocusRange(null);
|
||||
}}
|
||||
disabled={!canCompare}
|
||||
>
|
||||
<span className="component-index">{component.component_position + 1}</span>
|
||||
<span>
|
||||
<strong>{component.component_id}</strong>
|
||||
<small>
|
||||
v{component.version} · {component.change_count} 条
|
||||
</small>
|
||||
</span>
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
</section>
|
||||
</aside>
|
||||
|
||||
<main className="workspace">
|
||||
<section className="document-header">
|
||||
<div>
|
||||
<p className="eyebrow">当前文档</p>
|
||||
<h2>{selectedSummary?.source_label ?? "未选择"}</h2>
|
||||
</div>
|
||||
{selectedSummary === undefined ? null : (
|
||||
<div className="document-meta">
|
||||
<span className={`status status--${selectedSummary.status}`}>
|
||||
{statusLabel(selectedSummary.status)}
|
||||
</span>
|
||||
<span title={selectedSummary.input_sha256}>输入 {shortHash(selectedSummary.input_sha256)}</span>
|
||||
<span title={selectedSummary.current_sha256}>当前 {shortHash(selectedSummary.current_sha256)}</span>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
{documentState.loading || stageState.loading ? <LoadingPanel /> : null}
|
||||
{documentState.error !== null ? <ErrorPanel message={documentState.error} /> : null}
|
||||
{stageState.error !== null ? <ErrorPanel message={stageState.error} /> : null}
|
||||
|
||||
{!documentState.loading && document !== null && document.document.status === "success" && !canCompare ? (
|
||||
<ErrorPanel message={document.document.availability_error ?? "完整原文或清洗结果不可用。"} />
|
||||
) : null}
|
||||
|
||||
{!documentState.loading && document !== null && document.document.status !== "success" ? (
|
||||
<div className="diagnostic-panel">
|
||||
<p className="eyebrow">没有正式清洗结果</p>
|
||||
<h3>{statusLabel(document.document.status)}文档只展示审计证据</h3>
|
||||
<p>
|
||||
该状态不会生成 <code>cleaned.md</code>，因此这里不构造完整部分输出。
|
||||
</p>
|
||||
{document.errors.map((error) => (
|
||||
<article key={`${error.component_position}-${error.stage}-${error.error_type}`}>
|
||||
<strong>{error.error_type}</strong>
|
||||
<span>{error.message}</span>
|
||||
</article>
|
||||
))}
|
||||
{document.residual_proposals.map((proposal) => (
|
||||
<article key={`${proposal.component_position}-${proposal.reason}`}>
|
||||
<strong>最终复查仍有 {proposal.edit_count} 项候选</strong>
|
||||
<span>{proposal.reason}</span>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{!documentState.loading && !stageState.loading && canCompare && stageState.error === null ? (
|
||||
<Suspense fallback={<LoadingPanel />}>
|
||||
<DiffView
|
||||
before={beforeText}
|
||||
after={afterText}
|
||||
beforeLabel={beforeLabel}
|
||||
afterLabel={afterLabel}
|
||||
focusRange={focusRange}
|
||||
/>
|
||||
</Suspense>
|
||||
) : null}
|
||||
|
||||
{document !== null ? (
|
||||
<section className="changes-panel" aria-label="修改详情">
|
||||
<div className="changes-heading">
|
||||
<div>
|
||||
<p className="eyebrow">实际修改</p>
|
||||
<h3>
|
||||
{selectedComponent === null
|
||||
? `全部组件 · ${visibleChanges.length} 条`
|
||||
: `${document.components[selectedComponent]?.component_id ?? "组件"} · ${visibleChanges.length} 条`}
|
||||
</h3>
|
||||
</div>
|
||||
{selectedStage === null ? null : <p>{selectedStage.component.applicability}</p>}
|
||||
</div>
|
||||
<ChangeList changes={visibleChanges} onSelect={selectChange} canJump={canCompare} />
|
||||
</section>
|
||||
) : null}
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
import { markdown } from "@codemirror/lang-markdown";
|
||||
import { MergeView } from "@codemirror/merge";
|
||||
import { EditorSelection, EditorState } from "@codemirror/state";
|
||||
import { drawSelection, EditorView, lineNumbers } from "@codemirror/view";
|
||||
import { useEffect, useRef } from "react";
|
||||
|
||||
interface DiffViewProps {
|
||||
before: string;
|
||||
after: string;
|
||||
beforeLabel: string;
|
||||
afterLabel: string;
|
||||
focusRange?: { start: number; end: number } | null;
|
||||
}
|
||||
|
||||
const editorTheme = EditorView.theme({
|
||||
"&": {
|
||||
height: "100%",
|
||||
backgroundColor: "#fbfaf7",
|
||||
color: "#262822",
|
||||
fontSize: "13px",
|
||||
},
|
||||
".cm-scroller": {
|
||||
fontFamily: '"SFMono-Regular", Consolas, "Liberation Mono", monospace',
|
||||
lineHeight: "1.68",
|
||||
},
|
||||
".cm-gutters": {
|
||||
backgroundColor: "#f2f0ea",
|
||||
color: "#8a877e",
|
||||
border: "none",
|
||||
},
|
||||
".cm-content": {
|
||||
padding: "18px 0 36px",
|
||||
},
|
||||
".cm-line": {
|
||||
padding: "0 14px",
|
||||
},
|
||||
"&.cm-focused": {
|
||||
outline: "2px solid #a7b9ac",
|
||||
outlineOffset: "-2px",
|
||||
},
|
||||
});
|
||||
|
||||
const readOnlyExtensions = [
|
||||
lineNumbers(),
|
||||
markdown(),
|
||||
EditorState.readOnly.of(true),
|
||||
EditorView.editable.of(false),
|
||||
drawSelection(),
|
||||
EditorView.lineWrapping,
|
||||
editorTheme,
|
||||
];
|
||||
|
||||
export function DiffView({ before, after, beforeLabel, afterLabel, focusRange }: DiffViewProps) {
|
||||
const host = useRef<HTMLDivElement>(null);
|
||||
const merge = useRef<MergeView | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
if (host.current === null) {
|
||||
return undefined;
|
||||
}
|
||||
const view = new MergeView({
|
||||
parent: host.current,
|
||||
a: { doc: before, extensions: readOnlyExtensions },
|
||||
b: { doc: after, extensions: readOnlyExtensions },
|
||||
orientation: "a-b",
|
||||
gutter: true,
|
||||
highlightChanges: true,
|
||||
});
|
||||
merge.current = view;
|
||||
return () => {
|
||||
view.destroy();
|
||||
merge.current = null;
|
||||
};
|
||||
}, [before, after]);
|
||||
|
||||
// 切换组件时 stage 文本先到、旧视图上的提前滚动会被重建覆盖，所以聚焦
|
||||
// 必须等 MergeView 换成目标文本之后再执行。
|
||||
useEffect(() => {
|
||||
const view = merge.current;
|
||||
if (view === null || focusRange === null || focusRange === undefined) {
|
||||
return;
|
||||
}
|
||||
const anchor = Math.min(Math.max(focusRange.start, 0), view.a.state.doc.length);
|
||||
const head = Math.min(Math.max(focusRange.end, anchor), view.a.state.doc.length);
|
||||
view.a.dispatch({
|
||||
selection: EditorSelection.range(anchor, head),
|
||||
effects: EditorView.scrollIntoView(anchor, { y: "center" }),
|
||||
});
|
||||
view.a.focus();
|
||||
}, [focusRange, before, after]);
|
||||
|
||||
return (
|
||||
<section className="diff-shell" aria-label={`${beforeLabel}与${afterLabel}对比`}>
|
||||
<div className="diff-labels" aria-hidden="true">
|
||||
<span>{beforeLabel}</span>
|
||||
<span>{afterLabel}</span>
|
||||
</div>
|
||||
<div className="diff-host" ref={host} />
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -1,288 +0,0 @@
|
||||
import type {
|
||||
ApiErrorResponse,
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunSummaryResponse,
|
||||
} from "../shared/api.js";
|
||||
|
||||
export class ReviewerApiError extends Error {
|
||||
readonly code: string;
|
||||
|
||||
constructor(code: string, message: string) {
|
||||
super(message);
|
||||
this.name = "ReviewerApiError";
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
type JsonRecord = Record<string, unknown>;
|
||||
|
||||
function invalid(label: string): never {
|
||||
throw new ReviewerApiError("invalid_response", `本地服务返回的 ${label} 格式不正确。`);
|
||||
}
|
||||
|
||||
function record(value: unknown, label: string): JsonRecord {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value as JsonRecord;
|
||||
}
|
||||
|
||||
function array(value: unknown, label: string): unknown[] {
|
||||
if (!Array.isArray(value)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function string(value: unknown, label: string): string {
|
||||
if (typeof value !== "string") {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function integer(value: unknown, label: string, minimum = 0): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||
return invalid(label);
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
function boolean(value: unknown, label: string): boolean {
|
||||
if (typeof value !== "boolean") {
|
||||
return invalid(label);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function nullableString(value: unknown, label: string): string | null {
|
||||
return value === null ? null : string(value, label);
|
||||
}
|
||||
|
||||
function hash(value: unknown, label: string): string {
|
||||
const digest = string(value, label);
|
||||
if (!/^[0-9a-f]{64}$/.test(digest)) {
|
||||
return invalid(label);
|
||||
}
|
||||
return digest;
|
||||
}
|
||||
|
||||
function status(value: unknown): "success" | "failed" | "unstable" {
|
||||
if (value !== "success" && value !== "failed" && value !== "unstable") {
|
||||
return invalid("status");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function component(value: unknown): RunSummaryResponse["components"][number] {
|
||||
const item = record(value, "component");
|
||||
return {
|
||||
component_position: integer(item.component_position, "component_position"),
|
||||
component_id: string(item.component_id, "component_id"),
|
||||
version: string(item.version, "component version"),
|
||||
parameters: item.parameters,
|
||||
applicability: string(item.applicability, "component applicability"),
|
||||
change_count: integer(item.change_count, "component change_count"),
|
||||
};
|
||||
}
|
||||
|
||||
function documentSummary(value: unknown): RunSummaryResponse["documents"][number] {
|
||||
const item = record(value, "document summary");
|
||||
return {
|
||||
document_id: string(item.document_id, "document_id"),
|
||||
source_label: string(item.source_label, "source_label"),
|
||||
status: status(item.status),
|
||||
input_sha256: hash(item.input_sha256, "input_sha256"),
|
||||
current_sha256: hash(item.current_sha256, "current_sha256"),
|
||||
change_count: integer(item.change_count, "document change_count"),
|
||||
source_available: boolean(item.source_available, "source_available"),
|
||||
output_available: boolean(item.output_available, "output_available"),
|
||||
availability_error: nullableString(item.availability_error, "availability_error"),
|
||||
};
|
||||
}
|
||||
|
||||
function summary(value: unknown): RunSummaryResponse["summary"] {
|
||||
const item = record(value, "summary");
|
||||
return {
|
||||
document_count: integer(item.document_count, "document_count"),
|
||||
success_count: integer(item.success_count, "success_count"),
|
||||
failed_count: integer(item.failed_count, "failed_count"),
|
||||
unstable_count: integer(item.unstable_count, "unstable_count"),
|
||||
change_count: integer(item.change_count, "change_count"),
|
||||
};
|
||||
}
|
||||
|
||||
function change(value: unknown): DocumentComparisonResponse["changes"][number] {
|
||||
const item = record(value, "change");
|
||||
const proposal = record(item.proposal_ref, "proposal_ref");
|
||||
const location = record(item.location, "location");
|
||||
const editorValue = item.editor_range;
|
||||
const editorRange =
|
||||
editorValue === null
|
||||
? null
|
||||
: (() => {
|
||||
const editor = record(editorValue, "editor_range");
|
||||
const start = integer(editor.start, "editor_range.start");
|
||||
const end = integer(editor.end, "editor_range.end");
|
||||
if (end < start) {
|
||||
return invalid("editor_range");
|
||||
}
|
||||
return { start, end };
|
||||
})();
|
||||
return {
|
||||
component_id: string(item.component_id, "change component_id"),
|
||||
component_version: string(item.component_version, "change component_version"),
|
||||
component_position: integer(item.component_position, "change component_position"),
|
||||
proposal_ref: {
|
||||
component_position: integer(proposal.component_position, "proposal component_position"),
|
||||
snapshot_sha256: hash(proposal.snapshot_sha256, "proposal snapshot_sha256"),
|
||||
proposal_index: integer(proposal.proposal_index, "proposal_index"),
|
||||
},
|
||||
edit_index: integer(item.edit_index, "edit_index"),
|
||||
reason: string(item.reason, "reason"),
|
||||
location: {
|
||||
line: integer(location.line, "location.line", 1),
|
||||
column: integer(location.column, "location.column", 1),
|
||||
},
|
||||
editor_range: editorRange,
|
||||
before: string(item.before, "before"),
|
||||
after: string(item.after, "after"),
|
||||
};
|
||||
}
|
||||
|
||||
function runError(value: unknown): DocumentComparisonResponse["errors"][number] {
|
||||
const item = record(value, "run error");
|
||||
const stage = item.stage;
|
||||
if (stage !== "transform" && stage !== "final_review") {
|
||||
return invalid("error stage");
|
||||
}
|
||||
return {
|
||||
component_id: string(item.component_id, "error component_id"),
|
||||
component_version: string(item.component_version, "error component_version"),
|
||||
component_position: integer(item.component_position, "error component_position"),
|
||||
stage,
|
||||
error_type: string(item.error_type, "error_type"),
|
||||
message: string(item.message, "error message"),
|
||||
};
|
||||
}
|
||||
|
||||
function residual(value: unknown): DocumentComparisonResponse["residual_proposals"][number] {
|
||||
const item = record(value, "residual proposal");
|
||||
return {
|
||||
component_id: string(item.component_id, "residual component_id"),
|
||||
component_version: string(item.component_version, "residual component_version"),
|
||||
component_position: integer(item.component_position, "residual component_position"),
|
||||
reason: string(item.reason, "residual reason"),
|
||||
edit_count: integer(item.edit_count, "residual edit_count", 1),
|
||||
};
|
||||
}
|
||||
|
||||
function parseRun(value: unknown): RunSummaryResponse {
|
||||
const payload = record(value, "run response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("run schema_version");
|
||||
}
|
||||
const run = record(payload.run, "run");
|
||||
return {
|
||||
schema_version: 1,
|
||||
run: {
|
||||
run_id: string(run.run_id, "run_id"),
|
||||
run_date: string(run.run_date, "run_date"),
|
||||
status: status(run.status),
|
||||
started_at_utc: string(run.started_at_utc, "started_at_utc"),
|
||||
completed_at_utc: string(run.completed_at_utc, "completed_at_utc"),
|
||||
retention_until: string(run.retention_until, "retention_until"),
|
||||
},
|
||||
components: array(payload.components, "components").map(component),
|
||||
documents: array(payload.documents, "documents").map(documentSummary),
|
||||
summary: summary(payload.summary),
|
||||
original_run_location_changed: boolean(
|
||||
payload.original_run_location_changed,
|
||||
"original_run_location_changed",
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
function parseDocument(value: unknown): DocumentComparisonResponse {
|
||||
const payload = record(value, "document response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("document schema_version");
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
document: documentSummary(payload.document),
|
||||
components: array(payload.components, "components").map(component),
|
||||
original_markdown: nullableString(payload.original_markdown, "original_markdown"),
|
||||
cleaned_markdown: nullableString(payload.cleaned_markdown, "cleaned_markdown"),
|
||||
changes: array(payload.changes, "changes").map(change),
|
||||
errors: array(payload.errors, "errors").map(runError),
|
||||
residual_proposals: array(payload.residual_proposals, "residual_proposals").map(residual),
|
||||
};
|
||||
}
|
||||
|
||||
function parseStage(value: unknown): ComponentStageResponse {
|
||||
const payload = record(value, "component stage response");
|
||||
if (payload.schema_version !== 1) {
|
||||
return invalid("component stage schema_version");
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
document_id: string(payload.document_id, "document_id"),
|
||||
component: component(payload.component),
|
||||
before_sha256: hash(payload.before_sha256, "before_sha256"),
|
||||
after_sha256: hash(payload.after_sha256, "after_sha256"),
|
||||
before_markdown: string(payload.before_markdown, "before_markdown"),
|
||||
after_markdown: string(payload.after_markdown, "after_markdown"),
|
||||
changes: array(payload.changes, "changes").map(change),
|
||||
};
|
||||
}
|
||||
|
||||
async function getJson<T>(
|
||||
pathname: string,
|
||||
parse: (payload: unknown) => T,
|
||||
signal?: AbortSignal,
|
||||
): Promise<T> {
|
||||
const response = await fetch(pathname, {
|
||||
method: "GET",
|
||||
cache: "no-store",
|
||||
credentials: "same-origin",
|
||||
signal,
|
||||
});
|
||||
const payload: unknown = await response.json();
|
||||
if (!response.ok) {
|
||||
const errorPayload = payload as Partial<ApiErrorResponse>;
|
||||
throw new ReviewerApiError(
|
||||
errorPayload.error?.code ?? "request_failed",
|
||||
errorPayload.error?.message ?? `请求失败（HTTP ${response.status}）。`,
|
||||
);
|
||||
}
|
||||
return parse(payload);
|
||||
}
|
||||
|
||||
export function fetchRun(signal?: AbortSignal): Promise<RunSummaryResponse> {
|
||||
return getJson("/api/v1/run", parseRun, signal);
|
||||
}
|
||||
|
||||
export function fetchDocument(
|
||||
documentId: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<DocumentComparisonResponse> {
|
||||
return getJson(
|
||||
`/api/v1/documents/${encodeURIComponent(documentId)}`,
|
||||
parseDocument,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
|
||||
export function fetchComponentStage(
|
||||
documentId: string,
|
||||
componentPosition: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<ComponentStageResponse> {
|
||||
return getJson(
|
||||
`/api/v1/documents/${encodeURIComponent(documentId)}/components/${componentPosition}`,
|
||||
parseStage,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
import { StrictMode } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
|
||||
import { App } from "./App.js";
|
||||
import "./styles.css";
|
||||
|
||||
const root = document.getElementById("root");
|
||||
if (root === null) {
|
||||
throw new Error("missing #root element");
|
||||
}
|
||||
|
||||
createRoot(root).render(
|
||||
<StrictMode>
|
||||
<App />
|
||||
</StrictMode>,
|
||||
);
|
||||
@@ -1,558 +0,0 @@
|
||||
:root {
|
||||
color: #252720;
|
||||
background: #ecebe5;
|
||||
font-family:
|
||||
Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC", "Microsoft YaHei",
|
||||
sans-serif;
|
||||
font-synthesis: none;
|
||||
text-rendering: optimizeLegibility;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body,
|
||||
#root {
|
||||
min-width: 1180px;
|
||||
min-height: 100%;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
button {
|
||||
color: inherit;
|
||||
font: inherit;
|
||||
}
|
||||
|
||||
button:focus-visible {
|
||||
outline: 2px solid #315f4b;
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
.app-shell {
|
||||
min-height: 100vh;
|
||||
background:
|
||||
radial-gradient(circle at 12% 0%, rgb(255 255 255 / 72%), transparent 34%),
|
||||
#ecebe5;
|
||||
}
|
||||
|
||||
.topbar {
|
||||
position: sticky;
|
||||
z-index: 20;
|
||||
top: 0;
|
||||
display: flex;
|
||||
min-height: 76px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px 24px;
|
||||
border-bottom: 1px solid #d7d5cd;
|
||||
background: rgb(248 247 242 / 94%);
|
||||
backdrop-filter: blur(18px);
|
||||
}
|
||||
|
||||
.topbar > div:first-child {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.brand-mark {
|
||||
display: grid;
|
||||
width: 42px;
|
||||
height: 42px;
|
||||
place-items: center;
|
||||
border-radius: 12px;
|
||||
background: #284d3d;
|
||||
color: #f3f4ed;
|
||||
font-family: Georgia, serif;
|
||||
font-size: 19px;
|
||||
letter-spacing: -0.08em;
|
||||
}
|
||||
|
||||
.eyebrow {
|
||||
margin: 0 0 3px;
|
||||
color: #78796f;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.12em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
h1,
|
||||
h2,
|
||||
h3,
|
||||
p {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
.topbar h1 {
|
||||
margin: 0;
|
||||
font-family: Georgia, "Songti SC", serif;
|
||||
font-size: 19px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.run-facts,
|
||||
.document-meta {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
color: #66685f;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.run-facts > span:not(.status),
|
||||
.document-meta > span:not(.status) {
|
||||
padding-left: 10px;
|
||||
border-left: 1px solid #d5d2c9;
|
||||
}
|
||||
|
||||
.status {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
border: 1px solid currentColor;
|
||||
border-radius: 999px;
|
||||
padding: 3px 8px;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.status--success {
|
||||
color: #277052;
|
||||
background: #edf6ef;
|
||||
}
|
||||
|
||||
.status--failed {
|
||||
color: #a04338;
|
||||
background: #fff0ed;
|
||||
}
|
||||
|
||||
.status--unstable {
|
||||
color: #986617;
|
||||
background: #fff7df;
|
||||
}
|
||||
|
||||
.notice {
|
||||
padding: 9px 24px;
|
||||
border-bottom: 1px solid #e6d09c;
|
||||
background: #fff7de;
|
||||
color: #77561d;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.layout {
|
||||
display: grid;
|
||||
min-height: calc(100vh - 76px);
|
||||
grid-template-columns: 300px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
position: sticky;
|
||||
top: 76px;
|
||||
overflow-y: auto;
|
||||
height: calc(100vh - 76px);
|
||||
border-right: 1px solid #d7d5cd;
|
||||
background: #f7f6f1;
|
||||
}
|
||||
|
||||
.sidebar section {
|
||||
padding: 20px 16px;
|
||||
}
|
||||
|
||||
.sidebar section + section {
|
||||
border-top: 1px solid #dfddd5;
|
||||
}
|
||||
|
||||
.section-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.section-heading h2 {
|
||||
margin: 0;
|
||||
font-size: 12px;
|
||||
letter-spacing: 0.08em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.section-heading > span {
|
||||
color: #888980;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.document-list,
|
||||
.component-list {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.document-list button,
|
||||
.component-list button {
|
||||
display: grid;
|
||||
width: 100%;
|
||||
align-items: center;
|
||||
border: 0;
|
||||
border-radius: 9px;
|
||||
background: transparent;
|
||||
cursor: pointer;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.document-list button {
|
||||
grid-template-columns: 9px 1fr;
|
||||
gap: 10px;
|
||||
padding: 9px 10px;
|
||||
}
|
||||
|
||||
.document-list button:hover,
|
||||
.component-list button:hover:not(:disabled) {
|
||||
background: #eceae2;
|
||||
}
|
||||
|
||||
.document-list button.is-active,
|
||||
.component-list button.is-active {
|
||||
background: #e0e8e0;
|
||||
color: #234b39;
|
||||
}
|
||||
|
||||
.document-list strong,
|
||||
.component-list strong {
|
||||
display: block;
|
||||
overflow: hidden;
|
||||
font-size: 12px;
|
||||
font-weight: 650;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.document-list small,
|
||||
.component-list small {
|
||||
display: block;
|
||||
margin-top: 3px;
|
||||
color: #7d7e75;
|
||||
font-size: 10px;
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 7px;
|
||||
height: 7px;
|
||||
border-radius: 50%;
|
||||
background: #999;
|
||||
}
|
||||
|
||||
.status-dot--success {
|
||||
background: #348361;
|
||||
}
|
||||
|
||||
.status-dot--failed {
|
||||
background: #b64b3f;
|
||||
}
|
||||
|
||||
.status-dot--unstable {
|
||||
background: #bd831c;
|
||||
}
|
||||
|
||||
.text-button {
|
||||
border: 0;
|
||||
background: transparent;
|
||||
color: #315f4b;
|
||||
cursor: pointer;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.text-button:disabled {
|
||||
color: #aaa99f;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.component-list {
|
||||
counter-reset: components;
|
||||
}
|
||||
|
||||
.component-list button {
|
||||
grid-template-columns: 26px minmax(0, 1fr);
|
||||
gap: 8px;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
.component-list button:disabled {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.55;
|
||||
}
|
||||
|
||||
.component-index {
|
||||
display: grid;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
place-items: center;
|
||||
border: 1px solid #d3d1c8;
|
||||
border-radius: 50%;
|
||||
color: #74766e;
|
||||
font-family: Georgia, serif;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.workspace {
|
||||
display: grid;
|
||||
min-width: 0;
|
||||
align-content: start;
|
||||
gap: 14px;
|
||||
padding: 18px 20px 30px;
|
||||
}
|
||||
|
||||
.document-header {
|
||||
display: flex;
|
||||
align-items: end;
|
||||
justify-content: space-between;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.document-header h2 {
|
||||
margin: 0;
|
||||
font-family: Georgia, "Songti SC", serif;
|
||||
font-size: 21px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.diff-shell,
|
||||
.changes-panel,
|
||||
.diagnostic-panel,
|
||||
.state-panel {
|
||||
overflow: hidden;
|
||||
border: 1px solid #d5d3ca;
|
||||
border-radius: 13px;
|
||||
background: #fbfaf7;
|
||||
box-shadow: 0 12px 36px rgb(55 57 48 / 7%);
|
||||
}
|
||||
|
||||
.diff-shell {
|
||||
min-height: 510px;
|
||||
}
|
||||
|
||||
.diff-labels {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
border-bottom: 1px solid #dcdbd3;
|
||||
background: #f4f2ec;
|
||||
color: #6f7168;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.diff-labels span {
|
||||
padding: 9px 14px;
|
||||
}
|
||||
|
||||
.diff-labels span + span {
|
||||
border-left: 1px solid #dcdbd3;
|
||||
}
|
||||
|
||||
.diff-host {
|
||||
height: 510px;
|
||||
}
|
||||
|
||||
.diff-host > .cm-mergeView {
|
||||
height: 100%;
|
||||
overflow-y: auto;
|
||||
overscroll-behavior: contain;
|
||||
}
|
||||
|
||||
.diff-host .cm-mergeViewEditors {
|
||||
min-height: 100%;
|
||||
}
|
||||
|
||||
.diff-host .cm-editor {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.changes-panel {
|
||||
min-height: 120px;
|
||||
}
|
||||
|
||||
.changes-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 24px;
|
||||
padding: 15px 18px;
|
||||
border-bottom: 1px solid #e0ded6;
|
||||
}
|
||||
|
||||
.changes-heading h3 {
|
||||
margin: 0;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.changes-heading > p {
|
||||
max-width: 58%;
|
||||
margin: 0;
|
||||
color: #77786f;
|
||||
font-size: 11px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.change-list {
|
||||
display: grid;
|
||||
max-height: 310px;
|
||||
gap: 1px;
|
||||
overflow-y: auto;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background: #e4e2da;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.change-list button {
|
||||
display: grid;
|
||||
width: 100%;
|
||||
grid-template-columns: 145px minmax(240px, 1fr) minmax(260px, 0.9fr);
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
border: 0;
|
||||
padding: 11px 18px;
|
||||
background: #fbfaf7;
|
||||
cursor: pointer;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.change-list button:hover {
|
||||
background: #f4f5ef;
|
||||
}
|
||||
|
||||
.change-list button:disabled {
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.change-location {
|
||||
color: #73756c;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.change-list strong {
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.change-sample {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.change-sample del,
|
||||
.change-sample ins {
|
||||
overflow: hidden;
|
||||
max-width: 46%;
|
||||
border-radius: 4px;
|
||||
padding: 2px 5px;
|
||||
text-decoration: none;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.change-sample del {
|
||||
background: #f9ded9;
|
||||
color: #913e34;
|
||||
}
|
||||
|
||||
.change-sample ins {
|
||||
background: #dcecdf;
|
||||
color: #276348;
|
||||
}
|
||||
|
||||
.quiet-message {
|
||||
margin: 0;
|
||||
padding: 22px 18px;
|
||||
color: #77786f;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.state-panel,
|
||||
.diagnostic-panel {
|
||||
padding: 28px;
|
||||
}
|
||||
|
||||
.state-panel {
|
||||
display: grid;
|
||||
min-height: 180px;
|
||||
place-items: center;
|
||||
align-content: center;
|
||||
color: #66685f;
|
||||
}
|
||||
|
||||
.state-panel p {
|
||||
margin: 8px 0 0;
|
||||
}
|
||||
|
||||
.state-panel--error {
|
||||
border-color: #e2b6ae;
|
||||
color: #8f3e34;
|
||||
}
|
||||
|
||||
.loading-dot {
|
||||
width: 11px;
|
||||
height: 11px;
|
||||
border-radius: 50%;
|
||||
background: #3d735b;
|
||||
box-shadow: 0 0 0 7px #dce9df;
|
||||
animation: pulse 1.25s ease-in-out infinite;
|
||||
}
|
||||
|
||||
.diagnostic-panel h3 {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.diagnostic-panel > p:not(.eyebrow) {
|
||||
color: #686a61;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.diagnostic-panel article {
|
||||
display: grid;
|
||||
grid-template-columns: 220px 1fr;
|
||||
gap: 12px;
|
||||
padding: 10px 0;
|
||||
border-top: 1px solid #e0ded6;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
code {
|
||||
border-radius: 4px;
|
||||
padding: 1px 4px;
|
||||
background: #eceae3;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 0.45;
|
||||
transform: scale(0.82);
|
||||
}
|
||||
|
||||
50% {
|
||||
opacity: 1;
|
||||
transform: scale(1);
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 1280px) {
|
||||
.layout {
|
||||
grid-template-columns: 270px minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.change-list button {
|
||||
grid-template-columns: 125px minmax(180px, 1fr) minmax(220px, 0.8fr);
|
||||
}
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
export type RunStatus = "success" | "failed" | "unstable";
|
||||
|
||||
export interface ComponentSummary {
|
||||
component_position: number;
|
||||
component_id: string;
|
||||
version: string;
|
||||
parameters: unknown;
|
||||
applicability: string;
|
||||
change_count: number;
|
||||
}
|
||||
|
||||
export interface DocumentSummary {
|
||||
document_id: string;
|
||||
source_label: string;
|
||||
status: RunStatus;
|
||||
input_sha256: string;
|
||||
current_sha256: string;
|
||||
change_count: number;
|
||||
source_available: boolean;
|
||||
output_available: boolean;
|
||||
availability_error: string | null;
|
||||
}
|
||||
|
||||
export interface RunSummaryResponse {
|
||||
schema_version: 1;
|
||||
run: {
|
||||
run_id: string;
|
||||
run_date: string;
|
||||
status: RunStatus;
|
||||
started_at_utc: string;
|
||||
completed_at_utc: string;
|
||||
retention_until: string;
|
||||
};
|
||||
components: ComponentSummary[];
|
||||
documents: DocumentSummary[];
|
||||
summary: {
|
||||
document_count: number;
|
||||
success_count: number;
|
||||
failed_count: number;
|
||||
unstable_count: number;
|
||||
change_count: number;
|
||||
};
|
||||
original_run_location_changed: boolean;
|
||||
}
|
||||
|
||||
export interface ChangeDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
proposal_ref: {
|
||||
component_position: number;
|
||||
snapshot_sha256: string;
|
||||
proposal_index: number;
|
||||
};
|
||||
edit_index: number;
|
||||
reason: string;
|
||||
location: {
|
||||
line: number;
|
||||
column: number;
|
||||
};
|
||||
editor_range: {
|
||||
start: number;
|
||||
end: number;
|
||||
} | null;
|
||||
before: string;
|
||||
after: string;
|
||||
}
|
||||
|
||||
export interface RunErrorDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
stage: "transform" | "final_review";
|
||||
error_type: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export interface ResidualProposalDetail {
|
||||
component_id: string;
|
||||
component_version: string;
|
||||
component_position: number;
|
||||
reason: string;
|
||||
edit_count: number;
|
||||
}
|
||||
|
||||
export interface DocumentComparisonResponse {
|
||||
schema_version: 1;
|
||||
document: DocumentSummary;
|
||||
components: ComponentSummary[];
|
||||
original_markdown: string | null;
|
||||
cleaned_markdown: string | null;
|
||||
changes: ChangeDetail[];
|
||||
errors: RunErrorDetail[];
|
||||
residual_proposals: ResidualProposalDetail[];
|
||||
}
|
||||
|
||||
export interface ComponentStageResponse {
|
||||
schema_version: 1;
|
||||
document_id: string;
|
||||
component: ComponentSummary;
|
||||
before_sha256: string;
|
||||
after_sha256: string;
|
||||
before_markdown: string;
|
||||
after_markdown: string;
|
||||
changes: ChangeDetail[];
|
||||
}
|
||||
|
||||
export interface ApiErrorResponse {
|
||||
error: {
|
||||
code: string;
|
||||
message: string;
|
||||
};
|
||||
}
|
||||
@@ -1,256 +0,0 @@
|
||||
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { App } from "../src/client/App.js";
|
||||
import type {
|
||||
ComponentStageResponse,
|
||||
DocumentComparisonResponse,
|
||||
RunSummaryResponse,
|
||||
} from "../src/shared/api.js";
|
||||
|
||||
vi.mock("../src/client/DiffView.js", () => ({
|
||||
DiffView: ({ beforeLabel, afterLabel }: { beforeLabel: string; afterLabel: string }) => (
|
||||
<div data-testid="diff-view">
|
||||
{beforeLabel} / {afterLabel}
|
||||
</div>
|
||||
),
|
||||
}));
|
||||
|
||||
const components = [
|
||||
{
|
||||
component_position: 0,
|
||||
component_id: "paper.rule",
|
||||
version: "1.0.0",
|
||||
parameters: [],
|
||||
applicability: "替换测试单词。",
|
||||
change_count: 1,
|
||||
},
|
||||
{
|
||||
component_position: 1,
|
||||
component_id: "paper.zero",
|
||||
version: "1.0.0",
|
||||
parameters: [],
|
||||
applicability: "不修改当前测试文档。",
|
||||
change_count: 0,
|
||||
},
|
||||
];
|
||||
|
||||
const documentSummary = {
|
||||
document_id: "paper",
|
||||
source_label: "inputs/paper.md",
|
||||
status: "success" as const,
|
||||
input_sha256: "1".repeat(64),
|
||||
current_sha256: "2".repeat(64),
|
||||
change_count: 1,
|
||||
source_available: true,
|
||||
output_available: true,
|
||||
availability_error: null,
|
||||
};
|
||||
|
||||
const runResponse: RunSummaryResponse = {
|
||||
schema_version: 1,
|
||||
run: {
|
||||
run_id: "review-run",
|
||||
run_date: "2026-08-23",
|
||||
status: "success",
|
||||
started_at_utc: "2026-08-23T01:00:00Z",
|
||||
completed_at_utc: "2026-08-23T01:01:00Z",
|
||||
retention_until: "2026-09-22T01:01:00Z",
|
||||
},
|
||||
components,
|
||||
documents: [documentSummary],
|
||||
summary: {
|
||||
document_count: 1,
|
||||
success_count: 1,
|
||||
failed_count: 0,
|
||||
unstable_count: 0,
|
||||
change_count: 1,
|
||||
},
|
||||
original_run_location_changed: false,
|
||||
};
|
||||
|
||||
const change = {
|
||||
component_id: "paper.rule",
|
||||
component_version: "1.0.0",
|
||||
component_position: 0,
|
||||
proposal_ref: {
|
||||
component_position: 0,
|
||||
snapshot_sha256: "1".repeat(64),
|
||||
proposal_index: 0,
|
||||
},
|
||||
edit_index: 0,
|
||||
reason: "替换测试单词",
|
||||
location: { line: 1, column: 3 },
|
||||
editor_range: { start: 0, end: 3 },
|
||||
before: "old",
|
||||
after: "new",
|
||||
};
|
||||
|
||||
const documentResponse: DocumentComparisonResponse = {
|
||||
schema_version: 1,
|
||||
document: documentSummary,
|
||||
components,
|
||||
original_markdown: "old",
|
||||
cleaned_markdown: "new",
|
||||
changes: [change],
|
||||
errors: [],
|
||||
residual_proposals: [],
|
||||
};
|
||||
|
||||
const stageResponse: ComponentStageResponse = {
|
||||
schema_version: 1,
|
||||
document_id: "paper",
|
||||
component: components[0]!,
|
||||
before_sha256: "1".repeat(64),
|
||||
after_sha256: "2".repeat(64),
|
||||
before_markdown: "old",
|
||||
after_markdown: "new",
|
||||
changes: [change],
|
||||
};
|
||||
|
||||
function response(payload: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(payload), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
describe("App", () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("shows the run, document comparison, component order and component stage", async () => {
|
||||
const fetchMock = vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
if (pathname === "/api/v1/run") {
|
||||
return Promise.resolve(response(runResponse));
|
||||
}
|
||||
if (pathname.endsWith("/components/0")) {
|
||||
return Promise.resolve(response(stageResponse));
|
||||
}
|
||||
return Promise.resolve(response(documentResponse));
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("heading", { name: "review-run" })).toBeInTheDocument();
|
||||
expect(await screen.findByTestId("diff-view")).toHaveTextContent("清洗前 / 清洗后");
|
||||
expect(screen.getByText("paper.rule")).toBeInTheDocument();
|
||||
expect(screen.getByText("paper.zero")).toBeInTheDocument();
|
||||
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: /paper\.rule/ }));
|
||||
await waitFor(() => {
|
||||
expect(fetchMock).toHaveBeenCalledWith(expect.stringContaining("components/0"), expect.anything());
|
||||
});
|
||||
expect(await screen.findByTestId("diff-view")).toHaveTextContent("组件 1 执行前 / 组件 1 执行后");
|
||||
});
|
||||
|
||||
it("shows API failures without rendering document text", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
response({ error: { code: "unsupported_schema", message: "不支持这个产物版本。" } }, 409),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("alert")).toHaveTextContent("不支持这个产物版本");
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("shows failed audit evidence without inventing a cleaned result", async () => {
|
||||
const failedSummary = {
|
||||
...documentSummary,
|
||||
status: "failed" as const,
|
||||
current_sha256: documentSummary.input_sha256,
|
||||
change_count: 0,
|
||||
output_available: false,
|
||||
};
|
||||
const failedRun: RunSummaryResponse = {
|
||||
...runResponse,
|
||||
run: { ...runResponse.run, status: "failed" },
|
||||
documents: [failedSummary],
|
||||
summary: {
|
||||
document_count: 1,
|
||||
success_count: 0,
|
||||
failed_count: 1,
|
||||
unstable_count: 0,
|
||||
change_count: 0,
|
||||
},
|
||||
};
|
||||
const failedDocument: DocumentComparisonResponse = {
|
||||
schema_version: 1,
|
||||
document: failedSummary,
|
||||
components: components.map((component) => ({ ...component, change_count: 0 })),
|
||||
original_markdown: "原文",
|
||||
cleaned_markdown: null,
|
||||
changes: [],
|
||||
errors: [
|
||||
{
|
||||
component_id: "paper.rule",
|
||||
component_version: "1.0.0",
|
||||
component_position: 0,
|
||||
stage: "transform",
|
||||
error_type: "SyntheticError",
|
||||
message: "测试组件失败。",
|
||||
},
|
||||
],
|
||||
residual_proposals: [],
|
||||
};
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
return Promise.resolve(response(pathname === "/api/v1/run" ? failedRun : failedDocument));
|
||||
}),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("heading", { name: "失败文档只展示审计证据" })).toBeInTheDocument();
|
||||
expect(screen.getByText("测试组件失败。")).toBeInTheDocument();
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("keeps audit details visible when a successful run has lost its original source", async () => {
|
||||
const unavailableSummary = {
|
||||
...documentSummary,
|
||||
source_available: false,
|
||||
availability_error: "原文路径已经失效。",
|
||||
};
|
||||
const unavailableRun: RunSummaryResponse = {
|
||||
...runResponse,
|
||||
documents: [unavailableSummary],
|
||||
};
|
||||
const unavailableDocument: DocumentComparisonResponse = {
|
||||
...documentResponse,
|
||||
document: unavailableSummary,
|
||||
original_markdown: null,
|
||||
changes: [{ ...change, editor_range: null }],
|
||||
};
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn((input: string | URL | Request) => {
|
||||
const pathname =
|
||||
typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
|
||||
return Promise.resolve(
|
||||
response(pathname === "/api/v1/run" ? unavailableRun : unavailableDocument),
|
||||
);
|
||||
}),
|
||||
);
|
||||
|
||||
render(<App />);
|
||||
|
||||
expect(await screen.findByRole("alert")).toHaveTextContent("原文路径已经失效");
|
||||
expect(screen.getByText("替换测试单词")).toBeInTheDocument();
|
||||
expect(screen.queryByTestId("diff-view")).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -1,99 +0,0 @@
|
||||
import { render, screen } from "@testing-library/react";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { EditorView } from "@codemirror/view";
|
||||
|
||||
import { DiffView } from "../src/client/DiffView.js";
|
||||
|
||||
function leftPaneSelection(): { from: number; to: number } | null {
|
||||
const pane = document.querySelector(".diff-host .cm-editor");
|
||||
if (pane === null) {
|
||||
return null;
|
||||
}
|
||||
const view = EditorView.findFromDOM(pane as HTMLElement);
|
||||
if (view === null) {
|
||||
return null;
|
||||
}
|
||||
const { from, to } = view.state.selection.main;
|
||||
return { from, to };
|
||||
}
|
||||
|
||||
describe("DiffView", () => {
|
||||
it("keeps Markdown and raw HTML as inert editor text", () => {
|
||||
render(
|
||||
<DiffView
|
||||
before={'# title\n<img src="https://example.com/private.png" onerror="alert(1)">'}
|
||||
after={'# title\n<script>alert("x")</script>'}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(screen.getByRole("region", { name: "清洗前与清洗后对比" })).toBeInTheDocument();
|
||||
expect(document.querySelector("img")).toBeNull();
|
||||
expect(document.querySelector("script")).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps long unchanged sections available in the full document view", () => {
|
||||
const before = Array.from({ length: 30 }, (_, index) => `line ${index + 1}`);
|
||||
const after = [...before];
|
||||
after[14] = "changed line 15";
|
||||
|
||||
render(
|
||||
<DiffView
|
||||
before={before.join("\n")}
|
||||
after={after.join("\n")}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(document.querySelector(".cm-collapsedLines")).toBeNull();
|
||||
});
|
||||
|
||||
it("re-applies the focus selection after the compared texts change", () => {
|
||||
const longText = (mark: string) =>
|
||||
Array.from({ length: 30 }, (_, index) => (index === 14 ? mark : `line ${index + 1}`)).join("\n");
|
||||
|
||||
const { rerender } = render(
|
||||
<DiffView before={longText("old")} after={longText("new")} beforeLabel="清洗前" afterLabel="清洗后" />,
|
||||
);
|
||||
|
||||
// 模拟点击跨组件修改项：focusRange 先落在旧文本上，stage 文本随后到达。
|
||||
rerender(
|
||||
<DiffView
|
||||
before={longText("old")}
|
||||
after={longText("new")}
|
||||
beforeLabel="组件 1 执行前"
|
||||
afterLabel="组件 1 执行后"
|
||||
focusRange={{ start: 58, end: 61 }}
|
||||
/>,
|
||||
);
|
||||
rerender(
|
||||
<DiffView
|
||||
before={longText("stage before")}
|
||||
after={longText("stage after")}
|
||||
beforeLabel="组件 2 执行前"
|
||||
afterLabel="组件 2 执行后"
|
||||
focusRange={{ start: 58, end: 71 }}
|
||||
/>,
|
||||
);
|
||||
|
||||
// 最终视图必须是组件 2 的文本，且选区重新落在新文本的目标范围上。
|
||||
expect(screen.getByRole("region", { name: "组件 2 执行前与组件 2 执行后对比" })).toBeInTheDocument();
|
||||
expect(leftPaneSelection()).toEqual({ from: 58, to: 71 });
|
||||
});
|
||||
|
||||
it("applies the focus selection when it arrives with the initial texts", () => {
|
||||
render(
|
||||
<DiffView
|
||||
before={"a\nb"}
|
||||
after={"a\nc"}
|
||||
beforeLabel="清洗前"
|
||||
afterLabel="清洗后"
|
||||
focusRange={{ start: 0, end: 1 }}
|
||||
/>,
|
||||
);
|
||||
|
||||
expect(leftPaneSelection()).toEqual({ from: 0, to: 1 });
|
||||
});
|
||||
});
|
||||
@@ -1,23 +0,0 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { fetchRun } from "../src/client/api-client.js";
|
||||
|
||||
describe("API response validation", () => {
|
||||
it("rejects a successful HTTP response with an unknown schema", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
new Response(JSON.stringify({ schema_version: 2 }), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
}),
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
await expect(fetchRun()).rejects.toMatchObject({
|
||||
code: "invalid_response",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,6 +0,0 @@
|
||||
import "@testing-library/jest-dom/vitest";
|
||||
|
||||
import { afterEach } from "vitest";
|
||||
import { cleanup } from "@testing-library/react";
|
||||
|
||||
afterEach(() => cleanup());
|
||||
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2023",
|
||||
"useDefineForClassFields": true,
|
||||
"lib": ["ES2023", "DOM", "DOM.Iterable"],
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"allowImportingTsExtensions": false,
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"esModuleInterop": true,
|
||||
"jsx": "react-jsx",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"noImplicitOverride": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"types": ["node", "vite/client", "vitest/globals", "@testing-library/jest-dom/vitest"]
|
||||
},
|
||||
"include": ["src", "tests", "vite.config.ts"]
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
import react from "@vitejs/plugin-react";
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
host: "127.0.0.1",
|
||||
port: 5173,
|
||||
strictPort: true,
|
||||
proxy: {
|
||||
"/api": {
|
||||
target: "http://127.0.0.1:4174",
|
||||
changeOrigin: true,
|
||||
configure(proxy) {
|
||||
proxy.on("proxyReq", (request) => request.removeHeader("origin"));
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
build: {
|
||||
outDir: "dist",
|
||||
emptyOutDir: true,
|
||||
},
|
||||
test: {
|
||||
environment: "jsdom",
|
||||
setupFiles: "./tests/setup.ts",
|
||||
css: true,
|
||||
},
|
||||
});
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Run the approved arXiv component over the five local ClinDB paper copies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.components import ArxivSubmissionStampComponent
|
||||
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||
from mdpolish.models import RunStatus
|
||||
from mdpolish.pipeline import Pipeline
|
||||
|
||||
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||
|
||||
|
||||
def _parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _parse_args()
|
||||
repository_root = Path(__file__).resolve().parents[1]
|
||||
documents = tuple(
|
||||
InputDocument(
|
||||
document_id=document_id,
|
||||
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||
source_label=f"data/md/{document_id}.md",
|
||||
)
|
||||
for document_id in _DOCUMENT_IDS
|
||||
)
|
||||
started_at = datetime.now().astimezone()
|
||||
try:
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ArxivSubmissionStampComponent()]),
|
||||
documents=documents,
|
||||
run_id=args.run_id,
|
||||
artifacts_root=repository_root / "artifacts",
|
||||
started_at=started_at,
|
||||
tool=collect_tool_metadata(repository_root),
|
||||
)
|
||||
except Exception as error:
|
||||
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"run_id={args.run_id}")
|
||||
print(f"status={result.status.value}")
|
||||
print(f"documents={result.document_count}")
|
||||
print(f"changes={result.change_count}")
|
||||
print(f"artifacts={result.run_directory}")
|
||||
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,91 +0,0 @@
|
||||
"""Run the approved first-batch ClinDB pipeline over five local paper copies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.components import (
|
||||
ArxivSubmissionStampComponent,
|
||||
HtmlTableDoubleEscapeComponent,
|
||||
HtmlTableLayoutComponent,
|
||||
ManuscriptLineNumberComponent,
|
||||
PageBreakWordJoinComponent,
|
||||
ReferenceSpacingComponent,
|
||||
RepeatedRunningHeaderComponent,
|
||||
WordReviewCommentComponent,
|
||||
)
|
||||
from mdpolish.experiment import InputDocument, collect_tool_metadata, run_experiment
|
||||
from mdpolish.models import RunStatus
|
||||
from mdpolish.pipeline import Pipeline
|
||||
|
||||
_DOCUMENT_IDS = ("dmp", "ejhf", "jama", "sim", "springer")
|
||||
CLINDB_WORD_JOIN_MAPPINGS = (
|
||||
("medi-", "cal", "medical"),
|
||||
("possi-", "bly", "possibly"),
|
||||
("cre-", "ated", "created"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("life-", "threatening", "life-threatening"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def build_pipeline() -> Pipeline:
|
||||
"""Build the frozen ClinDB first-batch component order."""
|
||||
return Pipeline(
|
||||
[
|
||||
WordReviewCommentComponent(),
|
||||
ManuscriptLineNumberComponent(),
|
||||
ArxivSubmissionStampComponent(),
|
||||
RepeatedRunningHeaderComponent(),
|
||||
PageBreakWordJoinComponent(CLINDB_WORD_JOIN_MAPPINGS),
|
||||
HtmlTableDoubleEscapeComponent(),
|
||||
HtmlTableLayoutComponent(),
|
||||
ReferenceSpacingComponent(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run-id", required=True, help="safe identifier within today's artifact directory")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _parse_args()
|
||||
repository_root = Path(__file__).resolve().parents[1]
|
||||
documents = tuple(
|
||||
InputDocument(
|
||||
document_id=document_id,
|
||||
source_path=repository_root / "data" / "md" / f"{document_id}.md",
|
||||
source_label=f"data/md/{document_id}.md",
|
||||
)
|
||||
for document_id in _DOCUMENT_IDS
|
||||
)
|
||||
started_at = datetime.now().astimezone()
|
||||
try:
|
||||
result = run_experiment(
|
||||
pipeline=build_pipeline(),
|
||||
documents=documents,
|
||||
run_id=args.run_id,
|
||||
artifacts_root=repository_root / "artifacts",
|
||||
started_at=started_at,
|
||||
tool=collect_tool_metadata(repository_root),
|
||||
)
|
||||
except Exception as error:
|
||||
print(f"experiment failed: {type(error).__name__}: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"run_id={args.run_id}")
|
||||
print(f"status={result.status.value}")
|
||||
print(f"documents={result.document_count}")
|
||||
print(f"changes={result.change_count}")
|
||||
print(f"artifacts={result.run_directory}")
|
||||
return 0 if result.status is RunStatus.SUCCESS else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,12 +1,11 @@
|
||||
"""Approved public API for the first mdpolish in-memory core."""
|
||||
"""Public API for the project-independent mdpolish in-memory core."""
|
||||
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.edits import AppliedBatch, EditValidationError, apply_component_batch, validate_component_batch
|
||||
from mdpolish.edits import AppliedBatch, EditValidationError, apply_modifier_batch, validate_modifier_batch
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
@@ -17,17 +16,20 @@ from mdpolish.models import (
|
||||
TransformResult,
|
||||
markdown_sha256,
|
||||
)
|
||||
from mdpolish.modifier import Modifier, ModifierContractError, ModifierFunction
|
||||
from mdpolish.pipeline import Pipeline, PipelineContractError
|
||||
from mdpolish.regex import regex_replace
|
||||
|
||||
__all__ = [
|
||||
"AppliedBatch",
|
||||
"Change",
|
||||
"Component",
|
||||
"ComponentContractError",
|
||||
"ComponentInfo",
|
||||
"DocumentSnapshot",
|
||||
"EditValidationError",
|
||||
"ErrorStage",
|
||||
"Modifier",
|
||||
"ModifierContractError",
|
||||
"ModifierFunction",
|
||||
"ModifierInfo",
|
||||
"Pipeline",
|
||||
"PipelineContractError",
|
||||
"ProposalReference",
|
||||
@@ -38,7 +40,8 @@ __all__ = [
|
||||
"TextEdit",
|
||||
"TextSpan",
|
||||
"TransformResult",
|
||||
"apply_component_batch",
|
||||
"apply_modifier_batch",
|
||||
"markdown_sha256",
|
||||
"validate_component_batch",
|
||||
"regex_replace",
|
||||
"validate_modifier_batch",
|
||||
]
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
"""Pure replay of recorded artifact changes against an exact Markdown snapshot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from mdpolish.models import markdown_sha256
|
||||
|
||||
|
||||
class ReplayError(ValueError):
|
||||
"""A recorded change chain cannot be replayed without losing trust."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayComponent:
|
||||
"""The component identity needed to validate recorded changes."""
|
||||
|
||||
component_id: str
|
||||
version: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayChange:
|
||||
"""One artifact edit expressed with Python Unicode code-point indexes."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
proposal_component_position: int
|
||||
proposal_snapshot_sha256: str
|
||||
proposal_index: int
|
||||
edit_index: int
|
||||
start: int
|
||||
end: int
|
||||
before: str
|
||||
after: str
|
||||
before_sha256: str
|
||||
after_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LocatedChange:
|
||||
"""A replayed edit with human and browser-editor coordinates."""
|
||||
|
||||
change: ReplayChange
|
||||
line: int
|
||||
column: int
|
||||
editor_start: int
|
||||
editor_end: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayStage:
|
||||
"""The exact snapshots immediately before and after one component."""
|
||||
|
||||
component_position: int
|
||||
before_sha256: str
|
||||
after_sha256: str
|
||||
before_markdown: str
|
||||
after_markdown: str
|
||||
changes: tuple[LocatedChange, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayResult:
|
||||
"""The verified final snapshot, locations and optional component stages."""
|
||||
|
||||
current_markdown: str
|
||||
changes: tuple[LocatedChange, ...]
|
||||
stages: tuple[ReplayStage, ...]
|
||||
|
||||
|
||||
def _line_column(markdown: str, offset: int) -> tuple[int, int]:
|
||||
if offset < 0 or offset > len(markdown):
|
||||
raise ReplayError("a change offset is outside its recorded snapshot")
|
||||
|
||||
line = 1
|
||||
column = 1
|
||||
position = 0
|
||||
while position < offset:
|
||||
character = markdown[position]
|
||||
if character == "\r":
|
||||
line += 1
|
||||
column = 1
|
||||
if position + 1 < offset and markdown[position + 1] == "\n":
|
||||
position += 2
|
||||
else:
|
||||
position += 1
|
||||
elif character == "\n":
|
||||
line += 1
|
||||
column = 1
|
||||
position += 1
|
||||
else:
|
||||
column += 1
|
||||
position += 1
|
||||
return line, column
|
||||
|
||||
|
||||
def _utf16_offset(markdown: str, offset: int) -> int:
|
||||
return len(markdown[:offset].encode("utf-16-le")) // 2
|
||||
|
||||
|
||||
def _changes_conflict(left: ReplayChange, right: ReplayChange) -> bool:
|
||||
if left.start == left.end and right.start == right.end:
|
||||
return left.start == right.start
|
||||
if left.start == left.end:
|
||||
return right.start <= left.start <= right.end
|
||||
if right.start == right.end:
|
||||
return left.start <= right.start <= left.end
|
||||
return max(left.start, right.start) < min(left.end, right.end)
|
||||
|
||||
|
||||
def replay_change_chain(
|
||||
*,
|
||||
input_markdown: str,
|
||||
input_sha256: str,
|
||||
components: tuple[ReplayComponent, ...],
|
||||
changes: tuple[ReplayChange, ...],
|
||||
current_sha256: str,
|
||||
current_markdown: str | None,
|
||||
include_zero_change_stages: bool,
|
||||
) -> ReplayResult:
|
||||
"""Verify and replay a complete recorded chain without file I/O."""
|
||||
if markdown_sha256(input_markdown) != input_sha256:
|
||||
raise ReplayError("input Markdown does not match the recorded input hash")
|
||||
|
||||
previous_position = -1
|
||||
for change in changes:
|
||||
if change.component_position < previous_position:
|
||||
raise ReplayError("recorded changes are not in component order")
|
||||
previous_position = change.component_position
|
||||
|
||||
current = input_markdown
|
||||
located_changes: list[LocatedChange] = []
|
||||
stages: list[ReplayStage] = []
|
||||
cursor = 0
|
||||
|
||||
for component_position, component in enumerate(components):
|
||||
before_markdown = current
|
||||
batch: list[ReplayChange] = []
|
||||
while cursor < len(changes) and changes[cursor].component_position == component_position:
|
||||
batch.append(changes[cursor])
|
||||
cursor += 1
|
||||
|
||||
batch_locations: list[LocatedChange] = []
|
||||
if batch:
|
||||
first = batch[0]
|
||||
if markdown_sha256(current) != first.before_sha256:
|
||||
raise ReplayError("a change batch does not follow the recorded snapshot chain")
|
||||
|
||||
for change in batch:
|
||||
if change.component_id != component.component_id or change.component_version != component.version:
|
||||
raise ReplayError("a change identity does not match the recorded component")
|
||||
if change.proposal_component_position != component_position:
|
||||
raise ReplayError("a change proposal reference has the wrong component position")
|
||||
if change.proposal_snapshot_sha256 != change.before_sha256:
|
||||
raise ReplayError("a change proposal reference targets the wrong snapshot")
|
||||
if change.before_sha256 != first.before_sha256 or change.after_sha256 != first.after_sha256:
|
||||
raise ReplayError("a change batch contains inconsistent snapshot hashes")
|
||||
if change.start < 0 or change.end < change.start or change.end > len(current):
|
||||
raise ReplayError("a change span is outside its recorded snapshot")
|
||||
if len(change.before) != change.end - change.start:
|
||||
raise ReplayError("a change before value does not match its span length")
|
||||
if current[change.start : change.end] != change.before:
|
||||
raise ReplayError("a change before value does not match its recorded snapshot")
|
||||
line, column = _line_column(current, change.start)
|
||||
batch_locations.append(
|
||||
LocatedChange(
|
||||
change=change,
|
||||
line=line,
|
||||
column=column,
|
||||
editor_start=_utf16_offset(current, change.start),
|
||||
editor_end=_utf16_offset(current, change.end),
|
||||
)
|
||||
)
|
||||
|
||||
for left_index, left in enumerate(batch):
|
||||
for right in batch[left_index + 1 :]:
|
||||
if _changes_conflict(left, right):
|
||||
raise ReplayError("a change batch contains conflicting edit ranges")
|
||||
|
||||
application_order = sorted(
|
||||
batch,
|
||||
key=lambda change: (
|
||||
change.start,
|
||||
change.end,
|
||||
change.proposal_index,
|
||||
change.edit_index,
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
for change in application_order:
|
||||
current = current[: change.start] + change.after + current[change.end :]
|
||||
if markdown_sha256(current) != first.after_sha256:
|
||||
raise ReplayError("replayed changes do not produce the recorded batch hash")
|
||||
|
||||
located_changes.extend(batch_locations)
|
||||
if include_zero_change_stages or batch:
|
||||
stages.append(
|
||||
ReplayStage(
|
||||
component_position=component_position,
|
||||
before_sha256=markdown_sha256(before_markdown),
|
||||
after_sha256=markdown_sha256(current),
|
||||
before_markdown=before_markdown,
|
||||
after_markdown=current,
|
||||
changes=tuple(batch_locations),
|
||||
)
|
||||
)
|
||||
|
||||
if cursor != len(changes):
|
||||
raise ReplayError("a change has no matching component position")
|
||||
if markdown_sha256(current) != current_sha256:
|
||||
raise ReplayError("replayed changes do not produce the recorded current hash")
|
||||
if current_markdown is not None and current != current_markdown:
|
||||
raise ReplayError("replayed changes do not produce the recorded current snapshot")
|
||||
|
||||
return ReplayResult(current_markdown=current, changes=tuple(located_changes), stages=tuple(stages))
|
||||
@@ -24,7 +24,7 @@ class HtmlRow:
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class HtmlTable:
|
||||
"""Exact source ranges needed by table cleaning components."""
|
||||
"""Exact source ranges needed by table cleaning modifiers."""
|
||||
|
||||
start: int
|
||||
end: int
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Exact physical-line ranges shared by text cleaning components."""
|
||||
"""Exact physical-line ranges shared by text cleaning modifiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,340 +0,0 @@
|
||||
"""Private local storage for complete experiment artifact directories."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
_SAFE_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
|
||||
_DIRECTORY_MODE = 0o700
|
||||
_FILE_MODE = 0o600
|
||||
|
||||
|
||||
class ArtifactStoreError(RuntimeError):
|
||||
"""A local artifact directory cannot be published safely."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StoredDocument:
|
||||
"""Complete encoded artifacts for one document."""
|
||||
|
||||
document_id: str
|
||||
result_json: bytes
|
||||
cleaned_markdown: bytes | None
|
||||
diff: bytes | None
|
||||
output_sha256: str | None
|
||||
|
||||
|
||||
def _validate_identifier(value: str, field_name: str) -> None:
|
||||
if not isinstance(value, str) or _SAFE_ID_PATTERN.fullmatch(value) is None or ".." in value:
|
||||
raise ArtifactStoreError(f"{field_name} must be a safe lowercase identifier")
|
||||
|
||||
|
||||
def _validate_run_date(run_date: str) -> None:
|
||||
if not isinstance(run_date, str):
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD")
|
||||
try:
|
||||
parsed = date.fromisoformat(run_date)
|
||||
except ValueError as error:
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD") from error
|
||||
if parsed.isoformat() != run_date:
|
||||
raise ArtifactStoreError("run_date must use YYYY-MM-DD")
|
||||
|
||||
|
||||
def _require_private_directory(path: Path) -> None:
|
||||
if path.is_symlink():
|
||||
raise ArtifactStoreError(f"artifact directory cannot be a symlink: {path}")
|
||||
if not path.exists():
|
||||
path.mkdir(mode=_DIRECTORY_MODE, parents=False)
|
||||
path.chmod(_DIRECTORY_MODE)
|
||||
if not path.is_dir():
|
||||
raise ArtifactStoreError(f"artifact path is not a directory: {path}")
|
||||
mode = stat.S_IMODE(path.stat().st_mode)
|
||||
if mode != _DIRECTORY_MODE:
|
||||
raise ArtifactStoreError(f"artifact directory must have mode 0700: {path}")
|
||||
|
||||
|
||||
def _create_private_directory(path: Path) -> None:
|
||||
path.mkdir(mode=_DIRECTORY_MODE)
|
||||
path.chmod(_DIRECTORY_MODE)
|
||||
_require_private_directory(path)
|
||||
|
||||
|
||||
def _prepare_runs_directory(artifacts_root: Path, run_date: str) -> Path:
|
||||
if not artifacts_root.exists():
|
||||
artifacts_root.mkdir(mode=_DIRECTORY_MODE, parents=True)
|
||||
artifacts_root.chmod(_DIRECTORY_MODE)
|
||||
_require_private_directory(artifacts_root)
|
||||
|
||||
date_directory = artifacts_root / run_date
|
||||
_require_private_directory(date_directory)
|
||||
runs_directory = date_directory / "runs"
|
||||
_require_private_directory(runs_directory)
|
||||
return runs_directory
|
||||
|
||||
|
||||
def _write_private_file(path: Path, content: bytes) -> None:
|
||||
if not isinstance(content, bytes):
|
||||
raise ArtifactStoreError("artifact content must be bytes")
|
||||
with path.open("xb") as artifact_file:
|
||||
artifact_file.write(content)
|
||||
artifact_file.flush()
|
||||
os.fsync(artifact_file.fileno())
|
||||
path.chmod(_FILE_MODE)
|
||||
if stat.S_IMODE(path.stat().st_mode) != _FILE_MODE:
|
||||
raise ArtifactStoreError(f"artifact file must have mode 0600: {path}")
|
||||
if path.read_bytes() != content:
|
||||
raise ArtifactStoreError(f"artifact readback verification failed: {path}")
|
||||
|
||||
|
||||
def _validate_document(document: StoredDocument) -> None:
|
||||
_validate_identifier(document.document_id, "document_id")
|
||||
has_output = document.cleaned_markdown is not None
|
||||
if has_output != (document.diff is not None) or has_output != (document.output_sha256 is not None):
|
||||
raise ArtifactStoreError("cleaned Markdown, diff, and output hash must be present together")
|
||||
if document.cleaned_markdown is not None:
|
||||
digest = sha256(document.cleaned_markdown).hexdigest()
|
||||
if digest != document.output_sha256:
|
||||
raise ArtifactStoreError("cleaned Markdown does not match its output hash")
|
||||
|
||||
|
||||
def _json_object(content: bytes, label: str) -> dict[str, object]:
|
||||
if not isinstance(content, bytes) or not content.endswith(b"\n") or content.startswith(b"\xef\xbb\xbf"):
|
||||
raise ArtifactStoreError(f"{label} must be BOM-free UTF-8 JSON ending in a newline")
|
||||
try:
|
||||
decoded = json.loads(content)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as error:
|
||||
raise ArtifactStoreError(f"{label} must contain valid UTF-8 JSON") from error
|
||||
if not isinstance(decoded, dict) or not all(isinstance(key, str) for key in decoded):
|
||||
raise ArtifactStoreError(f"{label} must contain a JSON object")
|
||||
return cast(dict[str, object], decoded)
|
||||
|
||||
|
||||
def _nested_object(payload: dict[str, object], field_name: str, label: str) -> dict[str, object]:
|
||||
value = payload.get(field_name)
|
||||
if not isinstance(value, dict) or not all(isinstance(key, str) for key in value):
|
||||
raise ArtifactStoreError(f"{label}.{field_name} must be a JSON object")
|
||||
return cast(dict[str, object], value)
|
||||
|
||||
|
||||
def _validate_bundle(
|
||||
*,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
run_directory: Path,
|
||||
manifest_json: bytes,
|
||||
review_locator_json: bytes,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> None:
|
||||
manifest = _json_object(manifest_json, "manifest")
|
||||
if manifest.get("schema_version") != 1:
|
||||
raise ArtifactStoreError("manifest schema_version must be 1")
|
||||
run = _nested_object(manifest, "run", "manifest")
|
||||
if run.get("run_date") != run_date or run.get("run_id") != run_id:
|
||||
raise ArtifactStoreError("manifest run identity does not match the target path")
|
||||
|
||||
document_indexes = manifest.get("documents")
|
||||
if not isinstance(document_indexes, list) or len(document_indexes) != len(documents):
|
||||
raise ArtifactStoreError("manifest documents do not match the stored documents")
|
||||
|
||||
success_count = 0
|
||||
failed_count = 0
|
||||
unstable_count = 0
|
||||
change_count = 0
|
||||
for position, document in enumerate(documents):
|
||||
raw_index = document_indexes[position]
|
||||
if not isinstance(raw_index, dict) or not all(isinstance(key, str) for key in raw_index):
|
||||
raise ArtifactStoreError("manifest document indexes must be JSON objects")
|
||||
index = cast(dict[str, object], raw_index)
|
||||
report = _json_object(document.result_json, f"result for {document.document_id}")
|
||||
report_document = _nested_object(report, "document", f"result for {document.document_id}")
|
||||
report_output = _nested_object(report, "output", f"result for {document.document_id}")
|
||||
status = report.get("status")
|
||||
changes = report.get("changes")
|
||||
if (
|
||||
report.get("schema_version") != 1
|
||||
or report_document.get("document_id") != document.document_id
|
||||
or status not in {"success", "failed", "unstable"}
|
||||
or not isinstance(changes, list)
|
||||
):
|
||||
raise ArtifactStoreError("a document result does not match its stored identity")
|
||||
|
||||
has_output = document.cleaned_markdown is not None
|
||||
expected_cleaned_name = "cleaned.md" if has_output else None
|
||||
expected_diff_name = "changes.diff" if has_output else None
|
||||
base = f"documents/{document.document_id}"
|
||||
expected_cleaned_path = f"{base}/cleaned.md" if has_output else None
|
||||
expected_diff_path = f"{base}/changes.diff" if has_output else None
|
||||
if has_output != (status == "success"):
|
||||
raise ArtifactStoreError("document output presence does not match its result status")
|
||||
if report_output != {
|
||||
"cleaned_path": expected_cleaned_name,
|
||||
"diff_path": expected_diff_name,
|
||||
}:
|
||||
raise ArtifactStoreError("document result paths do not match its stored files")
|
||||
if document.output_sha256 != (report.get("current_sha256") if has_output else None):
|
||||
raise ArtifactStoreError("document output hash does not match its result")
|
||||
|
||||
expected_index = {
|
||||
"document_id": document.document_id,
|
||||
"source_label": report_document.get("source_label"),
|
||||
"status": status,
|
||||
"input_sha256": report.get("input_sha256"),
|
||||
"current_sha256": report.get("current_sha256"),
|
||||
"change_count": len(changes),
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": expected_cleaned_path,
|
||||
"diff_path": expected_diff_path,
|
||||
}
|
||||
if index != expected_index:
|
||||
raise ArtifactStoreError("a manifest document index does not match its result and files")
|
||||
|
||||
change_count += len(changes)
|
||||
if status == "success":
|
||||
success_count += 1
|
||||
elif status == "failed":
|
||||
failed_count += 1
|
||||
else:
|
||||
unstable_count += 1
|
||||
|
||||
overall_status = "failed" if failed_count else "unstable" if unstable_count else "success"
|
||||
expected_summary = {
|
||||
"document_count": len(documents),
|
||||
"success_count": success_count,
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
}
|
||||
if run.get("status") != overall_status or manifest.get("summary") != expected_summary:
|
||||
raise ArtifactStoreError("manifest status or summary does not match its documents")
|
||||
|
||||
locator = _json_object(review_locator_json, "review locator")
|
||||
if locator.get("schema_version") != 1:
|
||||
raise ArtifactStoreError("review locator schema_version must be 1")
|
||||
locator_run = _nested_object(locator, "run", "review locator")
|
||||
expected_run = {
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_directory.resolve(strict=False)),
|
||||
"manifest_path": "manifest.json",
|
||||
}
|
||||
if locator_run != expected_run:
|
||||
raise ArtifactStoreError("review locator run identity does not match the target path")
|
||||
|
||||
locator_documents = locator.get("documents")
|
||||
if not isinstance(locator_documents, list) or len(locator_documents) != len(documents):
|
||||
raise ArtifactStoreError("review locator documents do not match the stored documents")
|
||||
for position, document in enumerate(documents):
|
||||
raw_locator_document = locator_documents[position]
|
||||
if not isinstance(raw_locator_document, dict) or not all(
|
||||
isinstance(key, str) for key in raw_locator_document
|
||||
):
|
||||
raise ArtifactStoreError("review locator documents must be JSON objects")
|
||||
locator_document = cast(dict[str, object], raw_locator_document)
|
||||
source_path_value = locator_document.get("source_path")
|
||||
if not isinstance(source_path_value, str) or "\0" in source_path_value:
|
||||
raise ArtifactStoreError("review locator source_path must be an absolute path")
|
||||
source_path = Path(source_path_value)
|
||||
if not source_path.is_absolute() or source_path.resolve(strict=False) != source_path:
|
||||
raise ArtifactStoreError("review locator source_path must be an absolute resolved path")
|
||||
|
||||
manifest_index = cast(dict[str, object], document_indexes[position])
|
||||
expected_locator_document = {
|
||||
"document_id": document.document_id,
|
||||
"source_path": source_path_value,
|
||||
"input_sha256": manifest_index.get("input_sha256"),
|
||||
}
|
||||
if locator_document != expected_locator_document:
|
||||
raise ArtifactStoreError("review locator document identity does not match the manifest")
|
||||
try:
|
||||
source_bytes = source_path.read_bytes()
|
||||
except OSError as error:
|
||||
raise ArtifactStoreError("review locator source_path must identify a readable regular file") from error
|
||||
if not source_path.is_file() or sha256(source_bytes).hexdigest() != manifest_index.get("input_sha256"):
|
||||
raise ArtifactStoreError("review locator source file does not match its input hash")
|
||||
|
||||
|
||||
def _rename_no_replace(source: Path, target: Path) -> None:
|
||||
directory_fd = os.open(source.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
fcntl.flock(directory_fd, fcntl.LOCK_EX)
|
||||
if target.exists() or target.is_symlink():
|
||||
raise ArtifactStoreError("the target run directory already exists")
|
||||
os.rename(source, target)
|
||||
finally:
|
||||
fcntl.flock(directory_fd, fcntl.LOCK_UN)
|
||||
os.close(directory_fd)
|
||||
|
||||
|
||||
def publish_run(
|
||||
*,
|
||||
artifacts_root: Path,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
manifest_json: bytes,
|
||||
review_locator_json: bytes,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> Path:
|
||||
"""Publish one complete run directory without replacing an existing run."""
|
||||
if not isinstance(artifacts_root, Path):
|
||||
raise TypeError("artifacts_root must be a Path")
|
||||
_validate_run_date(run_date)
|
||||
_validate_identifier(run_id, "run_id")
|
||||
if not documents:
|
||||
raise ArtifactStoreError("an artifact run requires at least one document")
|
||||
|
||||
seen_ids: set[str] = set()
|
||||
for document in documents:
|
||||
if not isinstance(document, StoredDocument):
|
||||
raise ArtifactStoreError("documents must contain only StoredDocument values")
|
||||
_validate_document(document)
|
||||
if document.document_id in seen_ids:
|
||||
raise ArtifactStoreError("document_id values must be unique")
|
||||
seen_ids.add(document.document_id)
|
||||
_validate_bundle(
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
run_directory=artifacts_root / run_date / "runs" / run_id,
|
||||
manifest_json=manifest_json,
|
||||
review_locator_json=review_locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = _prepare_runs_directory(artifacts_root, run_date)
|
||||
final_directory = runs_directory / run_id
|
||||
if final_directory.exists() or final_directory.is_symlink():
|
||||
raise ArtifactStoreError("the target run directory already exists")
|
||||
|
||||
temporary_directory = Path(tempfile.mkdtemp(prefix=f".{run_id}.", dir=runs_directory))
|
||||
temporary_directory.chmod(_DIRECTORY_MODE)
|
||||
try:
|
||||
documents_directory = temporary_directory / "documents"
|
||||
_create_private_directory(documents_directory)
|
||||
for document in documents:
|
||||
document_directory = documents_directory / document.document_id
|
||||
_create_private_directory(document_directory)
|
||||
_write_private_file(document_directory / "result.json", document.result_json)
|
||||
if document.cleaned_markdown is not None and document.diff is not None:
|
||||
_write_private_file(document_directory / "cleaned.md", document.cleaned_markdown)
|
||||
_write_private_file(document_directory / "changes.diff", document.diff)
|
||||
|
||||
_write_private_file(temporary_directory / "manifest.json", manifest_json)
|
||||
_write_private_file(temporary_directory / "review-locator.json", review_locator_json)
|
||||
_rename_no_replace(temporary_directory, final_directory)
|
||||
except Exception:
|
||||
if temporary_directory.exists():
|
||||
shutil.rmtree(temporary_directory)
|
||||
raise
|
||||
|
||||
if stat.S_IMODE(final_directory.stat().st_mode) != _DIRECTORY_MODE:
|
||||
raise ArtifactStoreError("published run directory does not have mode 0700")
|
||||
return final_directory
|
||||
@@ -1,105 +0,0 @@
|
||||
"""Component extension contract for the mdpolish core."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Mapping
|
||||
from typing import cast, final
|
||||
|
||||
from mdpolish.models import ComponentInfo, DocumentSnapshot, Parameters, ParameterValue, ProposedChange
|
||||
|
||||
_COMPONENT_ID_PATTERN = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
|
||||
_SEMVER_PATTERN = re.compile(r"^(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)$")
|
||||
|
||||
|
||||
class ComponentContractError(ValueError):
|
||||
"""A component does not satisfy the approved extension contract."""
|
||||
|
||||
|
||||
def _freeze_parameter(value: object) -> ParameterValue:
|
||||
if value is None or isinstance(value, (str, bool)) or type(value) is int:
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ComponentContractError("component parameters cannot contain non-finite floats")
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
pairs: list[tuple[str, ParameterValue]] = []
|
||||
for key, nested_value in value.items():
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ComponentContractError("component parameter mapping keys must be non-empty strings")
|
||||
pairs.append((key, _freeze_parameter(nested_value)))
|
||||
return tuple(sorted(pairs, key=lambda pair: pair[0]))
|
||||
if isinstance(value, (list, tuple)):
|
||||
return tuple(_freeze_parameter(item) for item in value)
|
||||
raise ComponentContractError("component parameters must contain only deterministic basic data")
|
||||
|
||||
|
||||
def _freeze_parameters(parameters: object) -> Parameters:
|
||||
if not isinstance(parameters, Mapping):
|
||||
raise ComponentContractError("component parameters must be a mapping")
|
||||
frozen = _freeze_parameter(parameters)
|
||||
if not isinstance(frozen, tuple):
|
||||
raise AssertionError("a mapping must normalize to a tuple")
|
||||
return cast(Parameters, frozen)
|
||||
|
||||
|
||||
class Component(ABC):
|
||||
"""Base class for deterministic, side-effect-free cleaning components."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def component_id(self) -> str:
|
||||
"""Return the stable lowercase identity of this component."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def version(self) -> str:
|
||||
"""Return this component's MAJOR.MINOR.PATCH version."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
"""Return the current deterministic parameter values."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def applicability(self) -> str:
|
||||
"""Describe handled structures, strict preconditions, and exclusions."""
|
||||
|
||||
@abstractmethod
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
"""Return exact, automatically applicable changes for this snapshot."""
|
||||
|
||||
@final
|
||||
def _component_info(self) -> ComponentInfo:
|
||||
component_id = self.component_id
|
||||
version = self.version
|
||||
applicability = self.applicability
|
||||
|
||||
if not isinstance(component_id, str) or _COMPONENT_ID_PATTERN.fullmatch(component_id) is None:
|
||||
raise ComponentContractError("component_id must be a stable lowercase identifier")
|
||||
if not isinstance(version, str) or _SEMVER_PATTERN.fullmatch(version) is None:
|
||||
raise ComponentContractError("component version must use MAJOR.MINOR.PATCH")
|
||||
if not isinstance(applicability, str) or not applicability.strip():
|
||||
raise ComponentContractError("component applicability must be a non-empty string")
|
||||
|
||||
return ComponentInfo(
|
||||
component_id=component_id,
|
||||
version=version,
|
||||
parameters=_freeze_parameters(self.parameters),
|
||||
applicability=applicability,
|
||||
)
|
||||
|
||||
@final
|
||||
def _collect_proposals(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise ComponentContractError("components require a DocumentSnapshot")
|
||||
proposals = self._propose_changes(snapshot)
|
||||
if not isinstance(proposals, tuple):
|
||||
raise ComponentContractError("_propose_changes must return a tuple")
|
||||
if any(not isinstance(proposal, ProposedChange) for proposal in proposals):
|
||||
raise ComponentContractError("_propose_changes must return only ProposedChange values")
|
||||
return proposals
|
||||
@@ -1,22 +0,0 @@
|
||||
"""Approved business cleaning components."""
|
||||
|
||||
from mdpolish.components.arxiv_submission_stamp import ArxivSubmissionStampComponent
|
||||
from mdpolish.components.html_table_double_escape import HtmlTableDoubleEscapeComponent
|
||||
from mdpolish.components.html_table_layout import HtmlTableLayoutComponent
|
||||
from mdpolish.components.manuscript_line_number import ManuscriptLineNumberComponent
|
||||
from mdpolish.components.page_break_word_join import PageBreakWordJoinComponent, WordJoinMapping
|
||||
from mdpolish.components.reference_spacing import ReferenceSpacingComponent
|
||||
from mdpolish.components.repeated_running_header import RepeatedRunningHeaderComponent
|
||||
from mdpolish.components.word_review_comment import WordReviewCommentComponent
|
||||
|
||||
__all__ = [
|
||||
"ArxivSubmissionStampComponent",
|
||||
"HtmlTableDoubleEscapeComponent",
|
||||
"HtmlTableLayoutComponent",
|
||||
"ManuscriptLineNumberComponent",
|
||||
"PageBreakWordJoinComponent",
|
||||
"ReferenceSpacingComponent",
|
||||
"RepeatedRunningHeaderComponent",
|
||||
"WordJoinMapping",
|
||||
"WordReviewCommentComponent",
|
||||
]
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Remove exact arXiv submission stamp lines from converted papers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import iter_physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_STAMP_PATTERN = re.compile(
|
||||
r"^arXiv:[0-9]{4}\.[0-9]+v[0-9]+ \[[A-Za-z0-9_.-]+\] "
|
||||
r"(?:[1-9]|[12][0-9]|3[01]) "
|
||||
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{4}$"
|
||||
)
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||
|
||||
|
||||
class ArxivSubmissionStampComponent(Component):
|
||||
"""Delete physical lines that exactly match the approved arXiv stamp format."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.arxiv_submission_stamp"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理 PDF/arXiv 论文转换产生的独立提交戳行，要求严格整行匹配，排除所有相似文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
proposals: list[ProposedChange] = []
|
||||
for line in iter_physical_lines(snapshot.markdown):
|
||||
if _STAMP_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||
continue
|
||||
expected_text = snapshot.markdown[line.content_start : line.full_end]
|
||||
edit = TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(line.content_start, line.full_end),
|
||||
expected_text=expected_text,
|
||||
replacement="",
|
||||
)
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(edit,),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
@@ -1,93 +0,0 @@
|
||||
"""Remove line-number prefixes from an evidence-backed manuscript sequence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_PLAIN_PATTERN = re.compile(r"^([1-9][0-9]{0,2}) (?=\S)")
|
||||
_HEADING_PATTERN = re.compile(r"^(#{1,6}) ([1-9][0-9]{0,2}) (?=\S)")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除已确认手稿序列中的行号前缀"
|
||||
_MINIMUM_CANDIDATES = 20
|
||||
_MINIMUM_HEADING_CANDIDATES = 2
|
||||
|
||||
|
||||
def _candidate(line: PhysicalLine, markdown: str) -> tuple[int, int, int, bool] | None:
|
||||
content = line.content(markdown)
|
||||
heading_match = _HEADING_PATTERN.match(content)
|
||||
if heading_match is not None:
|
||||
number_start = line.content_start + len(heading_match.group(1)) + 1
|
||||
return int(heading_match.group(2)), number_start, line.content_start + heading_match.end(), True
|
||||
|
||||
plain_match = _PLAIN_PATTERN.match(content)
|
||||
if plain_match is None:
|
||||
return None
|
||||
return int(plain_match.group(1)), line.content_start, line.content_start + plain_match.end(), False
|
||||
|
||||
|
||||
class ManuscriptLineNumberComponent(Component):
|
||||
"""Strip prefixes only from one long, monotonic post-Abstract sequence."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.manuscript_line_number"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理唯一 ## Abstract 后至少 20 个严格递增且含至少 2 个标题证据的手稿行号序列，"
|
||||
"排除作者单位、列表和短数字片段。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
abstract_indexes = [
|
||||
index for index, line in enumerate(lines) if line.content(snapshot.markdown) == "## Abstract"
|
||||
]
|
||||
if len(abstract_indexes) != 1:
|
||||
return ()
|
||||
|
||||
candidates = [
|
||||
candidate
|
||||
for line in lines[abstract_indexes[0] + 1 :]
|
||||
if (candidate := _candidate(line, snapshot.markdown)) is not None
|
||||
]
|
||||
if len(candidates) < _MINIMUM_CANDIDATES:
|
||||
return ()
|
||||
if sum(is_heading for _, _, _, is_heading in candidates) < _MINIMUM_HEADING_CANDIDATES:
|
||||
return ()
|
||||
|
||||
numbers = [number for number, _, _, _ in candidates]
|
||||
if any(previous >= current for previous, current in pairwise(numbers)):
|
||||
return ()
|
||||
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
)
|
||||
for _, start, end, _ in candidates
|
||||
)
|
||||
@@ -1,127 +0,0 @@
|
||||
"""Join approved word fragments split across adjacent physical lines."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from collections.abc import Iterable, Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import physical_lines
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
WordJoinMapping = tuple[str, str, str]
|
||||
|
||||
|
||||
def _is_ascii_letter(character: str) -> bool:
|
||||
return character in "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
|
||||
|
||||
|
||||
def _validated_mappings(mappings: Iterable[WordJoinMapping]) -> tuple[WordJoinMapping, ...]:
|
||||
normalized: list[WordJoinMapping] = []
|
||||
seen_pairs: set[tuple[str, str]] = set()
|
||||
for mapping in mappings:
|
||||
if not isinstance(mapping, tuple) or len(mapping) != 3 or any(not isinstance(value, str) for value in mapping):
|
||||
raise ComponentContractError("word join mappings must be three-string tuples")
|
||||
left, right, replacement = mapping
|
||||
if not left or not right or not replacement:
|
||||
raise ComponentContractError("word join mapping fields must be non-empty")
|
||||
if any(character.isspace() for value in mapping for character in value):
|
||||
raise ComponentContractError("word join mapping fields cannot contain whitespace")
|
||||
pair = (left, right)
|
||||
if pair in seen_pairs:
|
||||
raise ComponentContractError("word join mappings cannot repeat a fragment pair")
|
||||
seen_pairs.add(pair)
|
||||
normalized.append(mapping)
|
||||
return tuple(sorted(normalized))
|
||||
|
||||
|
||||
class PageBreakWordJoinComponent(Component):
|
||||
"""Apply only caller-approved exact fragment-to-word mappings."""
|
||||
|
||||
def __init__(self, mappings: Iterable[WordJoinMapping]) -> None:
|
||||
self._mappings = _validated_mappings(mappings)
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.page_break_word_join"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
records = tuple(
|
||||
MappingProxyType(
|
||||
{
|
||||
"left_fragment": left,
|
||||
"right_fragment": right,
|
||||
"replacement": replacement,
|
||||
}
|
||||
)
|
||||
for left, right, replacement in self._mappings
|
||||
)
|
||||
return MappingProxyType({"mappings": records})
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理显式批准的左右词片段，片段必须位于相邻物理行或只隔一个空行并满足 ASCII 词边界；"
|
||||
"排除未配置词和段落级猜测。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
matches: list[tuple[int, int, str, str, str]] = []
|
||||
for index, left_line in enumerate(lines[:-1]):
|
||||
left_text = left_line.content(snapshot.markdown)
|
||||
left_ending = left_line.line_ending(snapshot.markdown)
|
||||
if not left_ending:
|
||||
continue
|
||||
for left, right, replacement in self._mappings:
|
||||
if not left_text.endswith(left):
|
||||
continue
|
||||
left_start = left_line.content_end - len(left)
|
||||
if left_start > left_line.content_start and _is_ascii_letter(snapshot.markdown[left_start - 1]):
|
||||
continue
|
||||
|
||||
right_index = index + 1
|
||||
if lines[right_index].is_blank(snapshot.markdown):
|
||||
blank = lines[right_index]
|
||||
if blank.line_ending(snapshot.markdown) != left_ending or right_index + 1 >= len(lines):
|
||||
continue
|
||||
right_index += 1
|
||||
if lines[right_index].is_blank(snapshot.markdown):
|
||||
continue
|
||||
right_line = lines[right_index]
|
||||
right_text = right_line.content(snapshot.markdown)
|
||||
if not right_text.startswith(right):
|
||||
continue
|
||||
if len(right_text) > len(right) and _is_ascii_letter(right_text[len(right)]):
|
||||
continue
|
||||
end = right_line.content_start + len(right)
|
||||
matches.append((left_start, end, replacement, left, right))
|
||||
|
||||
matches.sort(key=lambda item: (item[0], item[1], item[2]))
|
||||
span_counts = Counter(match[:2] for match in matches)
|
||||
unique_matches = [match for match in matches if span_counts[match[:2]] == 1]
|
||||
if any(previous[1] > current[0] for previous, current in pairwise(unique_matches)):
|
||||
return ()
|
||||
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=f"按已批准映射修复跨页断词：{left} + {right} → {replacement}",
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement=replacement,
|
||||
),
|
||||
),
|
||||
)
|
||||
for start, end, replacement, left, right in unique_matches
|
||||
)
|
||||
@@ -1,120 +0,0 @@
|
||||
"""Normalize blank lines between numbered entries in References sections."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_ATX_HEADING_PATTERN = re.compile(r"^(#{1,6}) (\S(?:.*\S)?)$")
|
||||
_ENTRY_PATTERN = re.compile(r"^([1-9][0-9]*)\. \S")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "统一 References 章节中相邻编号条目之间的一个空行"
|
||||
|
||||
|
||||
def _heading(line: PhysicalLine, markdown: str) -> tuple[int, str] | None:
|
||||
match = _ATX_HEADING_PATTERN.fullmatch(line.content(markdown))
|
||||
if match is None:
|
||||
return None
|
||||
return len(match.group(1)), match.group(2)
|
||||
|
||||
|
||||
class ReferenceSpacingComponent(Component):
|
||||
"""Normalize only complete, consecutive numbered reference sections."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.reference_spacing"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理准确 References ATX 章节内从 1 开始逐项递增的编号条目，统一为一个空行；"
|
||||
"排除章节外列表、缺号和混合行尾边界。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for heading_index, line in enumerate(lines):
|
||||
heading = _heading(line, snapshot.markdown)
|
||||
if heading is None or heading[1].lower() != "references":
|
||||
continue
|
||||
heading_level = heading[0]
|
||||
section_end = len(lines)
|
||||
for index in range(heading_index + 1, len(lines)):
|
||||
following_heading = _heading(lines[index], snapshot.markdown)
|
||||
if following_heading is not None and following_heading[0] <= heading_level:
|
||||
section_end = index
|
||||
break
|
||||
|
||||
entries: list[tuple[int, int]] = []
|
||||
section_is_safe = True
|
||||
for index in range(heading_index + 1, section_end):
|
||||
content = lines[index].content(snapshot.markdown)
|
||||
match = _ENTRY_PATTERN.match(content)
|
||||
if match is not None:
|
||||
entries.append((index, int(match.group(1))))
|
||||
elif _heading(lines[index], snapshot.markdown) is not None:
|
||||
section_is_safe = False
|
||||
if not section_is_safe or len(entries) < 2:
|
||||
continue
|
||||
if [number for _, number in entries] != list(range(1, len(entries) + 1)):
|
||||
continue
|
||||
|
||||
section_edits: list[tuple[int, int, str]] = []
|
||||
for previous, following in pairwise(entries):
|
||||
previous_index = previous[0]
|
||||
next_index = following[0]
|
||||
last_nonblank_index = next(
|
||||
index
|
||||
for index in range(next_index - 1, previous_index - 1, -1)
|
||||
if not lines[index].is_blank(snapshot.markdown)
|
||||
)
|
||||
between = lines[last_nonblank_index + 1 : next_index]
|
||||
if any(not blank.is_blank(snapshot.markdown) for blank in between):
|
||||
section_is_safe = False
|
||||
break
|
||||
line_ending = lines[last_nonblank_index].line_ending(snapshot.markdown)
|
||||
if not line_ending or any(blank.line_ending(snapshot.markdown) != line_ending for blank in between):
|
||||
section_is_safe = False
|
||||
break
|
||||
start = lines[last_nonblank_index].content_end
|
||||
end = lines[next_index].content_start
|
||||
replacement = line_ending * 2
|
||||
if snapshot.markdown[start:end] != replacement:
|
||||
section_edits.append((start, end, replacement))
|
||||
if section_is_safe:
|
||||
edits.extend(section_edits)
|
||||
|
||||
edits.sort(key=lambda item: item[0])
|
||||
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||
return ()
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement=replacement,
|
||||
),
|
||||
),
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
@@ -1,115 +0,0 @@
|
||||
"""Remove repeated ATX running headers with paragraph-interruption evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from collections.abc import Mapping
|
||||
from itertools import pairwise
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_HEADING_PATTERN = re.compile(r"^#{1,6} \S(?:.*\S)?$")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除经重复和断句证据确认的跑动页眉"
|
||||
_TERMINAL_PUNCTUATION = frozenset(".?!:;。！？：；")
|
||||
|
||||
|
||||
def _bridge_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||
if index < 2 or index + 2 >= len(lines):
|
||||
return None
|
||||
before_blank = lines[index - 1]
|
||||
after_blank = lines[index + 1]
|
||||
previous = lines[index - 2]
|
||||
following = lines[index + 2]
|
||||
if not before_blank.is_blank(markdown) or not after_blank.is_blank(markdown):
|
||||
return None
|
||||
if previous.is_blank(markdown) or following.is_blank(markdown):
|
||||
return None
|
||||
|
||||
previous_text = previous.content(markdown).rstrip()
|
||||
following_text = following.content(markdown)
|
||||
if not previous_text or previous_text[-1] in _TERMINAL_PUNCTUATION:
|
||||
return None
|
||||
if not following_text or following_text[0] not in "abcdefghijklmnopqrstuvwxyz":
|
||||
return None
|
||||
return previous.content_end, following.content_start
|
||||
|
||||
|
||||
def _deletion_span(lines: tuple[PhysicalLine, ...], index: int, markdown: str) -> tuple[int, int] | None:
|
||||
if index + 1 >= len(lines) or not lines[index + 1].is_blank(markdown):
|
||||
return None
|
||||
return lines[index].content_start, lines[index + 1].full_end
|
||||
|
||||
|
||||
class RepeatedRunningHeaderComponent(Component):
|
||||
"""Remove a repeated heading only when one occurrence splits a sentence."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.repeated_running_header"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理原样重复至少两次、且至少一次由小写续句证据确认切断正文的 ATX 跑动页眉；"
|
||||
"任一出现位置不安全时整组保留。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
groups: defaultdict[str, list[int]] = defaultdict(list)
|
||||
for index, line in enumerate(lines):
|
||||
content = line.content(snapshot.markdown)
|
||||
if _HEADING_PATTERN.fullmatch(content) is not None:
|
||||
groups[content].append(index)
|
||||
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for indexes in groups.values():
|
||||
if len(indexes) < 2:
|
||||
continue
|
||||
group_edits: list[tuple[int, int, str]] = []
|
||||
bridge_found = False
|
||||
for index in indexes:
|
||||
bridge = _bridge_span(lines, index, snapshot.markdown)
|
||||
if bridge is not None:
|
||||
group_edits.append((*bridge, " "))
|
||||
bridge_found = True
|
||||
continue
|
||||
deletion = _deletion_span(lines, index, snapshot.markdown)
|
||||
if deletion is None:
|
||||
group_edits = []
|
||||
break
|
||||
group_edits.append((*deletion, ""))
|
||||
if bridge_found and group_edits:
|
||||
edits.extend(group_edits)
|
||||
|
||||
edits.sort(key=lambda item: (item[0], item[1]))
|
||||
if any(previous[1] > current[0] for previous, current in pairwise(edits)):
|
||||
return ()
|
||||
return tuple(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement=replacement,
|
||||
),
|
||||
),
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
@@ -1,62 +0,0 @@
|
||||
"""Remove strict single-line Word review comments."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._text_ranges import physical_lines
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
_COMMENT_PATTERN = re.compile(r"^Commented \[[A-Za-z0-9]+\]: .*\S$")
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "删除严格单行 Word 审阅批注及其后一个空行"
|
||||
|
||||
|
||||
class WordReviewCommentComponent(Component):
|
||||
"""Delete approved one-line Word comments followed by a blank line."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "paper.word_review_comment"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理独占物理行且后接空行的 Word Commented 审阅批注，排除多行正文和所有相似文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
proposals: list[ProposedChange] = []
|
||||
for index, line in enumerate(lines[:-1]):
|
||||
following = lines[index + 1]
|
||||
if _COMMENT_PATTERN.fullmatch(line.content(snapshot.markdown)) is None:
|
||||
continue
|
||||
if not following.is_blank(snapshot.markdown):
|
||||
continue
|
||||
start = line.content_start
|
||||
end = following.full_end
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(start, end),
|
||||
expected_text=snapshot.markdown[start:end],
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
+18
-18
@@ -6,8 +6,8 @@ from dataclasses import dataclass
|
||||
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
TextEdit,
|
||||
@@ -15,12 +15,12 @@ from mdpolish.models import (
|
||||
|
||||
|
||||
class EditValidationError(ValueError):
|
||||
"""A component edit batch cannot be applied safely."""
|
||||
"""A modifier edit batch cannot be applied safely."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AppliedBatch:
|
||||
"""The new snapshot and audit entries from one atomic component batch."""
|
||||
"""The new snapshot and audit entries from one atomic modifier batch."""
|
||||
|
||||
snapshot: DocumentSnapshot
|
||||
changes: tuple[Change, ...]
|
||||
@@ -47,21 +47,21 @@ def _edits_conflict(left: TextEdit, right: TextEdit) -> bool:
|
||||
return max(left_span.start, right_span.start) < min(left_span.end, right_span.end)
|
||||
|
||||
|
||||
def validate_component_batch(
|
||||
def validate_modifier_batch(
|
||||
snapshot: DocumentSnapshot,
|
||||
proposals: tuple[ProposedChange, ...],
|
||||
) -> tuple[_IndexedEdit, ...]:
|
||||
"""Validate a complete component batch without changing the snapshot."""
|
||||
"""Validate a complete modifier batch without changing the snapshot."""
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise TypeError("snapshot must be a DocumentSnapshot")
|
||||
if not isinstance(proposals, tuple):
|
||||
raise EditValidationError("component proposals must be a tuple")
|
||||
raise EditValidationError("modifier proposals must be a tuple")
|
||||
|
||||
indexed_edits: list[_IndexedEdit] = []
|
||||
seen_edits: set[TextEdit] = set()
|
||||
for proposal_index, proposal in enumerate(proposals):
|
||||
if not isinstance(proposal, ProposedChange):
|
||||
raise EditValidationError("a component batch must contain only ProposedChange values")
|
||||
raise EditValidationError("a modifier batch must contain only ProposedChange values")
|
||||
if proposal.snapshot_sha256 != snapshot.sha256:
|
||||
raise EditValidationError("a proposal targets a stale document snapshot")
|
||||
for edit_index, edit in enumerate(proposal.edits):
|
||||
@@ -72,7 +72,7 @@ def validate_component_batch(
|
||||
if snapshot.markdown[edit.span.start : edit.span.end] != edit.expected_text:
|
||||
raise EditValidationError("an edit's expected_text does not match the document snapshot")
|
||||
if edit in seen_edits:
|
||||
raise EditValidationError("a component batch contains a duplicate edit")
|
||||
raise EditValidationError("a modifier batch contains a duplicate edit")
|
||||
seen_edits.add(edit)
|
||||
indexed_edits.append(
|
||||
_IndexedEdit(
|
||||
@@ -86,19 +86,19 @@ def validate_component_batch(
|
||||
for left_index, left in enumerate(indexed_edits):
|
||||
for right in indexed_edits[left_index + 1 :]:
|
||||
if _edits_conflict(left.edit, right.edit):
|
||||
raise EditValidationError("a component batch contains conflicting edit ranges")
|
||||
raise EditValidationError("a modifier batch contains conflicting edit ranges")
|
||||
|
||||
return tuple(indexed_edits)
|
||||
|
||||
|
||||
def apply_component_batch(
|
||||
def apply_modifier_batch(
|
||||
snapshot: DocumentSnapshot,
|
||||
proposals: tuple[ProposedChange, ...],
|
||||
component: ComponentInfo,
|
||||
component_position: int,
|
||||
modifier: ModifierInfo,
|
||||
modifier_position: int,
|
||||
) -> AppliedBatch:
|
||||
"""Atomically apply one fully validated component batch."""
|
||||
indexed_edits = validate_component_batch(snapshot, proposals)
|
||||
"""Atomically apply one fully validated modifier batch."""
|
||||
indexed_edits = validate_modifier_batch(snapshot, proposals)
|
||||
if not indexed_edits:
|
||||
return AppliedBatch(snapshot=snapshot, changes=())
|
||||
|
||||
@@ -129,11 +129,11 @@ def apply_component_batch(
|
||||
)
|
||||
changes = tuple(
|
||||
Change(
|
||||
component_id=component.component_id,
|
||||
component_version=component.version,
|
||||
component_position=component_position,
|
||||
modifier_id=modifier.modifier_id,
|
||||
modifier_version=modifier.version,
|
||||
modifier_position=modifier_position,
|
||||
proposal_ref=ProposalReference(
|
||||
component_position=component_position,
|
||||
modifier_position=modifier_position,
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
proposal_index=item.proposal_index,
|
||||
),
|
||||
|
||||
@@ -1,412 +0,0 @@
|
||||
"""Explicit local file experiments built around the in-memory pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import platform as platform_module
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from hashlib import sha256
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish.artifact_store import StoredDocument, publish_run
|
||||
from mdpolish.models import ComponentInfo, RunStatus, markdown_sha256
|
||||
from mdpolish.pipeline import Pipeline
|
||||
from mdpolish.reporting import (
|
||||
DocumentReport,
|
||||
JsonObject,
|
||||
JsonValue,
|
||||
build_document_report,
|
||||
component_info_json,
|
||||
encode_json,
|
||||
)
|
||||
|
||||
_SAFE_ID_PATTERN = re.compile(r"^[a-z0-9][a-z0-9._-]*$")
|
||||
_UTC_OFFSET_PATTERN = re.compile(r"^[+-][0-9]{2}:[0-9]{2}$")
|
||||
|
||||
|
||||
class ExperimentError(RuntimeError):
|
||||
"""A local experiment cannot be completed without violating its contract."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class InputDocument:
|
||||
"""One explicitly selected local Markdown input."""
|
||||
|
||||
document_id: str
|
||||
source_path: Path
|
||||
source_label: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ToolMetadata:
|
||||
"""Source-safe tool and environment facts for a run manifest."""
|
||||
|
||||
name: str
|
||||
package_version: str
|
||||
python_version: str
|
||||
platform: str
|
||||
git_commit: str | None
|
||||
git_dirty: bool | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ExperimentResult:
|
||||
"""Non-sensitive terminal summary of one published local run."""
|
||||
|
||||
run_directory: Path
|
||||
status: RunStatus
|
||||
document_count: int
|
||||
success_count: int
|
||||
failed_count: int
|
||||
unstable_count: int
|
||||
change_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _PreparedDocument:
|
||||
input: InputDocument
|
||||
resolved_path: Path
|
||||
source_bytes: bytes
|
||||
markdown: str
|
||||
input_sha256: str
|
||||
|
||||
|
||||
def _safe_identifier(value: str, field_name: str) -> None:
|
||||
if not isinstance(value, str) or _SAFE_ID_PATTERN.fullmatch(value) is None or ".." in value:
|
||||
raise ExperimentError(f"{field_name} must be a safe lowercase identifier")
|
||||
|
||||
|
||||
def _run_git(repository_root: Path, *arguments: str) -> str | None:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
("git", *arguments),
|
||||
cwd=repository_root,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
return completed.stdout.strip()
|
||||
|
||||
|
||||
def collect_tool_metadata(repository_root: Path) -> ToolMetadata:
|
||||
"""Collect only the reproducibility facts approved for the manifest."""
|
||||
if not isinstance(repository_root, Path):
|
||||
raise TypeError("repository_root must be a Path")
|
||||
try:
|
||||
package_version = version("mdpolish")
|
||||
except PackageNotFoundError:
|
||||
package_version = "uninstalled"
|
||||
|
||||
git_commit = _run_git(repository_root, "rev-parse", "HEAD")
|
||||
git_status = _run_git(repository_root, "status", "--porcelain")
|
||||
return ToolMetadata(
|
||||
name="mdpolish",
|
||||
package_version=package_version,
|
||||
python_version=platform_module.python_version(),
|
||||
platform=f"{sys.platform}-{platform_module.machine()}",
|
||||
git_commit=git_commit,
|
||||
git_dirty=None if git_status is None else bool(git_status),
|
||||
)
|
||||
|
||||
|
||||
def _utc_text(value: datetime) -> str:
|
||||
if value.tzinfo is None or value.utcoffset() is None:
|
||||
raise ExperimentError("experiment timestamps must be timezone-aware")
|
||||
utc_value = value.astimezone(UTC)
|
||||
return utc_value.isoformat(timespec="seconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def _utc_offset_text(value: datetime) -> str:
|
||||
offset = value.utcoffset()
|
||||
if offset is None:
|
||||
raise ExperimentError("experiment timestamps must be timezone-aware")
|
||||
total_minutes = int(offset.total_seconds() // 60)
|
||||
sign = "+" if total_minutes >= 0 else "-"
|
||||
absolute_minutes = abs(total_minutes)
|
||||
hours, minutes = divmod(absolute_minutes, 60)
|
||||
text = f"{sign}{hours:02d}:{minutes:02d}"
|
||||
if _UTC_OFFSET_PATTERN.fullmatch(text) is None or hours > 23 or minutes > 59:
|
||||
raise ExperimentError("the experiment UTC offset cannot be represented safely")
|
||||
return text
|
||||
|
||||
|
||||
def _prepare_documents(
|
||||
documents: tuple[InputDocument, ...],
|
||||
*,
|
||||
final_directory: Path,
|
||||
) -> tuple[_PreparedDocument, ...]:
|
||||
if not isinstance(documents, tuple) or not documents:
|
||||
raise ExperimentError("documents must be a non-empty tuple")
|
||||
|
||||
seen_ids: set[str] = set()
|
||||
seen_paths: set[Path] = set()
|
||||
prepared: list[_PreparedDocument] = []
|
||||
final_resolved = final_directory.resolve(strict=False)
|
||||
for document in documents:
|
||||
if not isinstance(document, InputDocument):
|
||||
raise ExperimentError("documents must contain only InputDocument values")
|
||||
_safe_identifier(document.document_id, "document_id")
|
||||
if document.document_id in seen_ids:
|
||||
raise ExperimentError("document_id values must be unique")
|
||||
seen_ids.add(document.document_id)
|
||||
if not isinstance(document.source_path, Path):
|
||||
raise ExperimentError("source_path must be a Path")
|
||||
if (
|
||||
not isinstance(document.source_label, str)
|
||||
or not document.source_label.strip()
|
||||
or "\0" in document.source_label
|
||||
):
|
||||
raise ExperimentError("source_label must be a non-empty display string")
|
||||
|
||||
try:
|
||||
resolved_path = document.source_path.resolve(strict=True)
|
||||
except OSError as error:
|
||||
raise ExperimentError(f"input file cannot be resolved: {document.source_label}") from error
|
||||
if not resolved_path.is_file():
|
||||
raise ExperimentError(f"input path is not a regular file: {document.source_label}")
|
||||
if resolved_path in seen_paths:
|
||||
raise ExperimentError("source_path values must be unique")
|
||||
seen_paths.add(resolved_path)
|
||||
if (
|
||||
resolved_path == final_resolved
|
||||
or final_resolved in resolved_path.parents
|
||||
or resolved_path in final_resolved.parents
|
||||
):
|
||||
raise ExperimentError("an input file and target run directory cannot overlap")
|
||||
|
||||
try:
|
||||
source_bytes = resolved_path.read_bytes()
|
||||
markdown = source_bytes.decode("utf-8", errors="strict")
|
||||
except (OSError, UnicodeDecodeError) as error:
|
||||
raise ExperimentError(f"input file must be readable strict UTF-8: {document.source_label}") from error
|
||||
input_sha256 = sha256(source_bytes).hexdigest()
|
||||
if markdown_sha256(markdown) != input_sha256:
|
||||
raise ExperimentError("decoded Markdown does not reproduce the input byte hash")
|
||||
prepared.append(
|
||||
_PreparedDocument(
|
||||
input=document,
|
||||
resolved_path=resolved_path,
|
||||
source_bytes=source_bytes,
|
||||
markdown=markdown,
|
||||
input_sha256=input_sha256,
|
||||
)
|
||||
)
|
||||
return tuple(prepared)
|
||||
|
||||
|
||||
def _overall_status(reports: tuple[DocumentReport, ...]) -> RunStatus:
|
||||
if any(report.status is RunStatus.FAILED for report in reports):
|
||||
return RunStatus.FAILED
|
||||
if any(report.status is RunStatus.UNSTABLE for report in reports):
|
||||
return RunStatus.UNSTABLE
|
||||
return RunStatus.SUCCESS
|
||||
|
||||
|
||||
def _document_index(report: DocumentReport) -> JsonObject:
|
||||
base = f"documents/{report.document_id}"
|
||||
return {
|
||||
"document_id": report.document_id,
|
||||
"source_label": report.source_label,
|
||||
"status": report.status.value,
|
||||
"input_sha256": report.input_sha256,
|
||||
"current_sha256": report.current_sha256,
|
||||
"change_count": report.change_count,
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": f"{base}/cleaned.md" if report.status is RunStatus.SUCCESS else None,
|
||||
"diff_path": f"{base}/changes.diff" if report.status is RunStatus.SUCCESS else None,
|
||||
}
|
||||
|
||||
|
||||
def _manifest_payload(
|
||||
*,
|
||||
run_id: str,
|
||||
run_date: str,
|
||||
utc_offset: str,
|
||||
started_at: datetime,
|
||||
completed_at: datetime,
|
||||
tool: ToolMetadata,
|
||||
components: tuple[ComponentInfo, ...],
|
||||
reports: tuple[DocumentReport, ...],
|
||||
) -> JsonObject:
|
||||
status = _overall_status(reports)
|
||||
component_values: list[JsonValue] = [component_info_json(component) for component in components]
|
||||
document_values: list[JsonValue] = [_document_index(report) for report in reports]
|
||||
success_count = sum(report.status is RunStatus.SUCCESS for report in reports)
|
||||
failed_count = sum(report.status is RunStatus.FAILED for report in reports)
|
||||
unstable_count = sum(report.status is RunStatus.UNSTABLE for report in reports)
|
||||
change_count = sum(report.change_count for report in reports)
|
||||
retention_until = completed_at.astimezone(UTC) + timedelta(days=30)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_date": run_date,
|
||||
"utc_offset": utc_offset,
|
||||
"status": status.value,
|
||||
"started_at_utc": _utc_text(started_at),
|
||||
"completed_at_utc": _utc_text(completed_at),
|
||||
"retention_until": _utc_text(retention_until),
|
||||
},
|
||||
"tool": {
|
||||
"name": tool.name,
|
||||
"package_version": tool.package_version,
|
||||
"python_version": tool.python_version,
|
||||
"platform": tool.platform,
|
||||
"git_commit": tool.git_commit,
|
||||
"git_dirty": tool.git_dirty,
|
||||
},
|
||||
"pipeline": {"components": component_values},
|
||||
"documents": document_values,
|
||||
"summary": {
|
||||
"document_count": len(reports),
|
||||
"success_count": success_count,
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _review_locator_payload(
|
||||
*,
|
||||
run_id: str,
|
||||
run_directory: Path,
|
||||
documents: tuple[_PreparedDocument, ...],
|
||||
) -> JsonObject:
|
||||
document_values: list[JsonValue] = [
|
||||
{
|
||||
"document_id": document.input.document_id,
|
||||
"source_path": str(document.resolved_path),
|
||||
"input_sha256": document.input_sha256,
|
||||
}
|
||||
for document in documents
|
||||
]
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_directory.resolve(strict=False)),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": document_values,
|
||||
}
|
||||
|
||||
|
||||
def run_experiment(
|
||||
*,
|
||||
pipeline: Pipeline,
|
||||
documents: tuple[InputDocument, ...],
|
||||
run_id: str,
|
||||
artifacts_root: Path,
|
||||
started_at: datetime,
|
||||
tool: ToolMetadata,
|
||||
completed_at: datetime | None = None,
|
||||
) -> ExperimentResult:
|
||||
"""Run explicit inputs and publish one complete private artifact directory."""
|
||||
if not isinstance(pipeline, Pipeline):
|
||||
raise TypeError("pipeline must be a Pipeline")
|
||||
_safe_identifier(run_id, "run_id")
|
||||
if not isinstance(artifacts_root, Path):
|
||||
raise TypeError("artifacts_root must be a Path")
|
||||
if not isinstance(tool, ToolMetadata):
|
||||
raise TypeError("tool must be ToolMetadata")
|
||||
if started_at.tzinfo is None or started_at.utcoffset() is None:
|
||||
raise ExperimentError("started_at must be timezone-aware")
|
||||
|
||||
run_date = started_at.date().isoformat()
|
||||
utc_offset = _utc_offset_text(started_at)
|
||||
final_directory = artifacts_root / run_date / "runs" / run_id
|
||||
if final_directory.exists() or final_directory.is_symlink():
|
||||
raise ExperimentError("the target run directory already exists")
|
||||
prepared = _prepare_documents(documents, final_directory=final_directory)
|
||||
|
||||
reports: list[DocumentReport] = []
|
||||
expected_components: tuple[ComponentInfo, ...] | None = None
|
||||
for document in prepared:
|
||||
result = pipeline.transform(document.markdown)
|
||||
if expected_components is None:
|
||||
expected_components = result.components
|
||||
elif result.components != expected_components:
|
||||
raise ExperimentError("pipeline component metadata changed between documents")
|
||||
reports.append(
|
||||
build_document_report(
|
||||
document_id=document.input.document_id,
|
||||
source_label=document.input.source_label,
|
||||
input_markdown=document.markdown,
|
||||
result=result,
|
||||
)
|
||||
)
|
||||
|
||||
for document in prepared:
|
||||
try:
|
||||
current_bytes = document.resolved_path.read_bytes()
|
||||
except OSError as error:
|
||||
raise ExperimentError(
|
||||
f"input file cannot be verified after the run: {document.input.source_label}"
|
||||
) from error
|
||||
if current_bytes != document.source_bytes or sha256(current_bytes).hexdigest() != document.input_sha256:
|
||||
raise ExperimentError(f"input file changed during the run: {document.input.source_label}")
|
||||
|
||||
report_tuple = tuple(reports)
|
||||
if expected_components is None:
|
||||
raise ExperimentError("a non-empty experiment produced no component metadata")
|
||||
completion = datetime.now(UTC) if completed_at is None else completed_at
|
||||
if completion.tzinfo is None or completion.utcoffset() is None:
|
||||
raise ExperimentError("completed_at must be timezone-aware")
|
||||
if completion.astimezone(UTC) < started_at.astimezone(UTC):
|
||||
raise ExperimentError("completed_at cannot be earlier than started_at")
|
||||
|
||||
manifest = _manifest_payload(
|
||||
run_id=run_id,
|
||||
run_date=run_date,
|
||||
utc_offset=utc_offset,
|
||||
started_at=started_at,
|
||||
completed_at=completion,
|
||||
tool=tool,
|
||||
components=expected_components,
|
||||
reports=report_tuple,
|
||||
)
|
||||
stored_documents = tuple(
|
||||
StoredDocument(
|
||||
document_id=report.document_id,
|
||||
result_json=report.result_json,
|
||||
cleaned_markdown=report.cleaned_markdown,
|
||||
diff=report.diff,
|
||||
output_sha256=report.current_sha256 if report.status is RunStatus.SUCCESS else None,
|
||||
)
|
||||
for report in report_tuple
|
||||
)
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
manifest_json=encode_json(manifest),
|
||||
review_locator_json=encode_json(
|
||||
_review_locator_payload(
|
||||
run_id=run_id,
|
||||
run_directory=final_directory,
|
||||
documents=prepared,
|
||||
)
|
||||
),
|
||||
documents=stored_documents,
|
||||
)
|
||||
|
||||
status = _overall_status(report_tuple)
|
||||
return ExperimentResult(
|
||||
run_directory=run_directory,
|
||||
status=status,
|
||||
document_count=len(report_tuple),
|
||||
success_count=sum(report.status is RunStatus.SUCCESS for report in report_tuple),
|
||||
failed_count=sum(report.status is RunStatus.FAILED for report in report_tuple),
|
||||
unstable_count=sum(report.status is RunStatus.UNSTABLE for report in report_tuple),
|
||||
change_count=sum(report.change_count for report in report_tuple),
|
||||
)
|
||||
+19
-19
@@ -94,7 +94,7 @@ class TextEdit:
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ProposedChange:
|
||||
"""One atomic, automatically applicable action proposed by a component."""
|
||||
"""One atomic, automatically applicable action proposed by a modifier."""
|
||||
|
||||
snapshot_sha256: str
|
||||
reason: str
|
||||
@@ -113,10 +113,10 @@ class ProposedChange:
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ComponentInfo:
|
||||
"""Validated, immutable component metadata recorded in a run."""
|
||||
class ModifierInfo:
|
||||
"""Validated, immutable modifier metadata recorded in a run."""
|
||||
|
||||
component_id: str
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
@@ -126,13 +126,13 @@ class ComponentInfo:
|
||||
class ProposalReference:
|
||||
"""A deterministic reference scoped to one transform result."""
|
||||
|
||||
component_position: int
|
||||
modifier_position: int
|
||||
snapshot_sha256: str
|
||||
proposal_index: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if type(self.component_position) is not int or self.component_position < 0:
|
||||
raise ValueError("component_position must be a non-negative integer")
|
||||
if type(self.modifier_position) is not int or self.modifier_position < 0:
|
||||
raise ValueError("modifier_position must be a non-negative integer")
|
||||
if type(self.proposal_index) is not int or self.proposal_index < 0:
|
||||
raise ValueError("proposal_index must be a non-negative integer")
|
||||
_require_sha256(self.snapshot_sha256, "snapshot_sha256")
|
||||
@@ -142,9 +142,9 @@ class ProposalReference:
|
||||
class Change:
|
||||
"""One text edit that was actually applied."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
proposal_ref: ProposalReference
|
||||
edit_index: int
|
||||
reason: str
|
||||
@@ -159,9 +159,9 @@ class Change:
|
||||
class ResidualProposal:
|
||||
"""A valid proposal found by the final read-only stability review."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
proposal_ref: ProposalReference
|
||||
proposal: ProposedChange
|
||||
|
||||
@@ -183,11 +183,11 @@ class ErrorStage(StrEnum):
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RunError:
|
||||
"""A source-safe component, contract, or edit error."""
|
||||
"""A source-safe modifier, contract, or edit error."""
|
||||
|
||||
component_id: str
|
||||
component_version: str
|
||||
component_position: int
|
||||
modifier_id: str
|
||||
modifier_version: str
|
||||
modifier_position: int
|
||||
stage: ErrorStage
|
||||
error_type: str
|
||||
message: str
|
||||
@@ -200,7 +200,7 @@ class TransformResult:
|
||||
status: RunStatus
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
components: tuple[ComponentInfo, ...] = ()
|
||||
modifiers: tuple[ModifierInfo, ...] = ()
|
||||
changes: tuple[Change, ...] = ()
|
||||
errors: tuple[RunError, ...] = ()
|
||||
residual_proposals: tuple[ResidualProposal, ...] = ()
|
||||
@@ -212,7 +212,7 @@ class TransformResult:
|
||||
_require_sha256(self.current_sha256, "current_sha256")
|
||||
if not isinstance(self.status, RunStatus):
|
||||
raise TypeError("status must be a RunStatus")
|
||||
for field_name in ("components", "changes", "errors", "residual_proposals"):
|
||||
for field_name in ("modifiers", "changes", "errors", "residual_proposals"):
|
||||
if not isinstance(getattr(self, field_name), tuple):
|
||||
raise TypeError(f"{field_name} must be a tuple")
|
||||
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Functional modifier contract for the mdpolish core."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import cast
|
||||
|
||||
from mdpolish.models import DocumentSnapshot, ModifierInfo, Parameters, ParameterValue, ProposedChange
|
||||
|
||||
ModifierFunction = Callable[[DocumentSnapshot], tuple[ProposedChange, ...]]
|
||||
|
||||
_MODIFIER_ID_PATTERN = re.compile(r"^[a-z][a-z0-9]*(?:[._-][a-z0-9]+)*$")
|
||||
_SEMVER_PATTERN = re.compile(r"^(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)$")
|
||||
|
||||
|
||||
class ModifierContractError(ValueError):
|
||||
"""A modifier does not satisfy the approved extension contract."""
|
||||
|
||||
|
||||
def _freeze_parameter(value: object) -> ParameterValue:
|
||||
if value is None or isinstance(value, (str, bool)) or type(value) is int:
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ModifierContractError("modifier parameters cannot contain non-finite floats")
|
||||
return value
|
||||
if isinstance(value, Mapping):
|
||||
pairs: list[tuple[str, ParameterValue]] = []
|
||||
for key, nested_value in value.items():
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ModifierContractError("modifier parameter mapping keys must be non-empty strings")
|
||||
pairs.append((key, _freeze_parameter(nested_value)))
|
||||
return tuple(sorted(pairs, key=lambda pair: pair[0]))
|
||||
if isinstance(value, (list, tuple)):
|
||||
return tuple(_freeze_parameter(item) for item in value)
|
||||
raise ModifierContractError("modifier parameters must contain only deterministic basic data")
|
||||
|
||||
|
||||
def _freeze_parameters(parameters: Mapping[str, object] | Parameters) -> Parameters:
|
||||
if isinstance(parameters, Mapping):
|
||||
frozen = _freeze_parameter(parameters)
|
||||
if not isinstance(frozen, tuple):
|
||||
raise AssertionError("a mapping must normalize to a tuple")
|
||||
return cast(Parameters, frozen)
|
||||
if not isinstance(parameters, tuple):
|
||||
raise ModifierContractError("modifier parameters must be a mapping or normalized parameter tuple")
|
||||
|
||||
normalized: dict[str, ParameterValue] = {}
|
||||
for pair in parameters:
|
||||
if not isinstance(pair, tuple) or len(pair) != 2:
|
||||
raise ModifierContractError("normalized modifier parameters must contain key-value tuples")
|
||||
key, value = pair
|
||||
if not isinstance(key, str) or not key:
|
||||
raise ModifierContractError("modifier parameter keys must be non-empty strings")
|
||||
if key in normalized:
|
||||
raise ModifierContractError("modifier parameter keys must be unique")
|
||||
normalized[key] = _freeze_parameter(value)
|
||||
return tuple(sorted(normalized.items(), key=lambda pair: pair[0]))
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True, init=False)
|
||||
class Modifier:
|
||||
"""Immutable metadata paired with a pure proposal function."""
|
||||
|
||||
modifier_id: str
|
||||
version: str
|
||||
parameters: Parameters
|
||||
applicability: str
|
||||
propose: ModifierFunction
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
modifier_id: str,
|
||||
version: str,
|
||||
parameters: Mapping[str, object] | Parameters,
|
||||
applicability: str,
|
||||
propose: ModifierFunction,
|
||||
) -> None:
|
||||
if not isinstance(modifier_id, str) or _MODIFIER_ID_PATTERN.fullmatch(modifier_id) is None:
|
||||
raise ModifierContractError("modifier_id must be a stable lowercase identifier")
|
||||
if not isinstance(version, str) or _SEMVER_PATTERN.fullmatch(version) is None:
|
||||
raise ModifierContractError("modifier version must use MAJOR.MINOR.PATCH")
|
||||
if not isinstance(applicability, str) or not applicability.strip():
|
||||
raise ModifierContractError("modifier applicability must be a non-empty string")
|
||||
if not callable(propose):
|
||||
raise ModifierContractError("modifier propose must be callable")
|
||||
|
||||
object.__setattr__(self, "modifier_id", modifier_id)
|
||||
object.__setattr__(self, "version", version)
|
||||
object.__setattr__(self, "parameters", _freeze_parameters(parameters))
|
||||
object.__setattr__(self, "applicability", applicability)
|
||||
object.__setattr__(self, "propose", propose)
|
||||
|
||||
def _info(self) -> ModifierInfo:
|
||||
return ModifierInfo(
|
||||
modifier_id=self.modifier_id,
|
||||
version=self.version,
|
||||
parameters=self.parameters,
|
||||
applicability=self.applicability,
|
||||
)
|
||||
|
||||
|
||||
def collect_proposals(modifier: Modifier, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
"""Call one modifier and validate the proposal container contract."""
|
||||
if not isinstance(modifier, Modifier):
|
||||
raise ModifierContractError("pipeline entries must be Modifier instances")
|
||||
if not isinstance(snapshot, DocumentSnapshot):
|
||||
raise ModifierContractError("modifiers require a DocumentSnapshot")
|
||||
proposals = modifier.propose(snapshot)
|
||||
if not isinstance(proposals, tuple):
|
||||
raise ModifierContractError("modifier propose must return a tuple")
|
||||
if any(not isinstance(proposal, ProposedChange) for proposal in proposals):
|
||||
raise ModifierContractError("modifier propose must return only ProposedChange values")
|
||||
return proposals
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Small project-independent modifiers shipped as examples and reusable tools."""
|
||||
|
||||
from mdpolish.modifiers.html_table_entities import html_table_entity_unescape
|
||||
from mdpolish.modifiers.html_table_layout import html_table_layout
|
||||
from mdpolish.modifiers.mapped_line_join import LineJoinMapping, mapped_line_join
|
||||
|
||||
__all__ = [
|
||||
"LineJoinMapping",
|
||||
"html_table_entity_unescape",
|
||||
"html_table_layout",
|
||||
"mapped_line_join",
|
||||
]
|
||||
+17
-27
@@ -1,14 +1,12 @@
|
||||
"""Remove one approved escape layer from strict HTML table cell text."""
|
||||
"""Unescape one approved entity layer inside strict HTML table cell text."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._html_table import strict_html_tables
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier
|
||||
|
||||
_ENTITY_PATTERN = re.compile(r"&(?:lt|gt|amp);")
|
||||
_REPLACEMENTS = {
|
||||
@@ -16,33 +14,17 @@ _REPLACEMENTS = {
|
||||
"&gt;": ">",
|
||||
"&amp;": "&",
|
||||
}
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "在严格 HTML 表格单元格文本中解除一层实体转义"
|
||||
_APPLICABILITY = (
|
||||
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;；"
|
||||
"不处理属性、表格外文本、其他实体或任意损坏 HTML。"
|
||||
)
|
||||
|
||||
|
||||
class HtmlTableDoubleEscapeComponent(Component):
|
||||
"""Unescape only approved double entities inside strict cell text."""
|
||||
def html_table_entity_unescape() -> Modifier:
|
||||
"""Create the strict HTML table entity modifier."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "markdown.html_table_double_escape"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
"处理严格完整 HTML 表格 td/th 文本中的 &lt;、&gt; 和 &amp;，"
|
||||
"排除属性、表格外文本和其他实体。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
edits: list[tuple[int, int, str]] = []
|
||||
for table in strict_html_tables(snapshot.markdown):
|
||||
for row in table.rows:
|
||||
@@ -65,3 +47,11 @@ class HtmlTableDoubleEscapeComponent(Component):
|
||||
)
|
||||
for start, end, replacement in edits
|
||||
)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="markdown.html_table_entity_unescape",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=_APPLICABILITY,
|
||||
propose=propose,
|
||||
)
|
||||
+17
-28
@@ -2,41 +2,22 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from types import MappingProxyType
|
||||
|
||||
from mdpolish._html_table import strict_html_tables
|
||||
from mdpolish._text_ranges import line_ending_styles
|
||||
from mdpolish.component import Component
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier
|
||||
|
||||
_EMPTY_PARAMETERS: Mapping[str, object] = MappingProxyType({})
|
||||
_REASON = "展开严格单行 HTML 表格的行布局"
|
||||
|
||||
|
||||
class HtmlTableLayoutComponent(Component):
|
||||
"""Preserve strict HTML table bytes while placing each row on its own line."""
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "markdown.html_table_layout"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return _EMPTY_PARAMETERS
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return (
|
||||
_APPLICABILITY = (
|
||||
"处理严格完整且当前为单物理行的 HTML 表格，只规范 table/tr 外层布局；"
|
||||
"排除混合行尾、复杂 HTML 和已是多行的表格。"
|
||||
)
|
||||
"不处理混合行尾、复杂 HTML 或已经换行的表格。"
|
||||
)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
|
||||
def html_table_layout() -> Modifier:
|
||||
"""Create the strict one-line HTML table layout modifier."""
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
styles = line_ending_styles(snapshot.markdown)
|
||||
if len(styles) > 1:
|
||||
return ()
|
||||
@@ -66,3 +47,11 @@ class HtmlTableLayoutComponent(Component):
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="markdown.html_table_layout",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=_APPLICABILITY,
|
||||
propose=propose,
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
+103
-78
@@ -1,16 +1,15 @@
|
||||
"""Sequential orchestration and final stability review."""
|
||||
"""Sequential modifier orchestration and final stability review."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
from mdpolish.component import Component, ComponentContractError
|
||||
from mdpolish.edits import apply_component_batch, validate_component_batch
|
||||
from mdpolish.edits import apply_modifier_batch, validate_modifier_batch
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
ModifierInfo,
|
||||
ProposalReference,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
@@ -18,6 +17,7 @@ from mdpolish.models import (
|
||||
RunStatus,
|
||||
TransformResult,
|
||||
)
|
||||
from mdpolish.modifier import Modifier, ModifierContractError, collect_proposals
|
||||
|
||||
|
||||
class PipelineContractError(ValueError):
|
||||
@@ -25,32 +25,32 @@ class PipelineContractError(ValueError):
|
||||
|
||||
|
||||
class Pipeline:
|
||||
"""Run selected components once, then review the final snapshot for stability."""
|
||||
"""Run selected modifiers once, then review the final snapshot for stability."""
|
||||
|
||||
def __init__(self, components: Iterable[Component]) -> None:
|
||||
self._components = tuple(components)
|
||||
def __init__(self, modifiers: Iterable[Modifier]) -> None:
|
||||
self._modifiers = tuple(modifiers)
|
||||
|
||||
@property
|
||||
def components(self) -> tuple[Component, ...]:
|
||||
return self._components
|
||||
def modifiers(self) -> tuple[Modifier, ...]:
|
||||
return self._modifiers
|
||||
|
||||
def transform(self, markdown: str) -> TransformResult:
|
||||
input_snapshot = DocumentSnapshot(markdown)
|
||||
component_infos, preflight_error = self._preflight_components()
|
||||
modifier_infos, preflight_error = self._preflight_modifiers()
|
||||
if preflight_error is not None:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=input_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=(),
|
||||
errors=(preflight_error,),
|
||||
)
|
||||
|
||||
current_snapshot = input_snapshot
|
||||
changes: list[Change] = []
|
||||
for position, (component, expected_info) in enumerate(zip(self._components, component_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_component_info(
|
||||
component=component,
|
||||
for position, (modifier, expected_info) in enumerate(zip(self._modifiers, modifier_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
@@ -59,14 +59,14 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(metadata_error,),
|
||||
)
|
||||
|
||||
proposals, proposal_error = self._proposals(
|
||||
component=component,
|
||||
component_info=current_info,
|
||||
modifier=modifier,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
snapshot=current_snapshot,
|
||||
@@ -75,31 +75,46 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(proposal_error,),
|
||||
)
|
||||
|
||||
_, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
)
|
||||
if metadata_error is not None:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(metadata_error,),
|
||||
)
|
||||
|
||||
try:
|
||||
applied_batch = apply_component_batch(
|
||||
applied_batch = apply_modifier_batch(
|
||||
snapshot=current_snapshot,
|
||||
proposals=proposals,
|
||||
component=current_info,
|
||||
component_position=position,
|
||||
modifier=current_info,
|
||||
modifier_position=position,
|
||||
)
|
||||
except Exception as error:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=(
|
||||
self._run_error(
|
||||
component_info=current_info,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="component edit batch does not satisfy the edit contract",
|
||||
unexpected_message="modifier edit batch does not satisfy the edit contract",
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -109,9 +124,9 @@ class Pipeline:
|
||||
|
||||
review_errors: list[RunError] = []
|
||||
residual_proposals: list[ResidualProposal] = []
|
||||
for position, (component, expected_info) in enumerate(zip(self._components, component_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_component_info(
|
||||
component=component,
|
||||
for position, (modifier, expected_info) in enumerate(zip(self._modifiers, modifier_infos, strict=True)):
|
||||
current_info, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
@@ -121,8 +136,8 @@ class Pipeline:
|
||||
continue
|
||||
|
||||
proposals, proposal_error = self._proposals(
|
||||
component=component,
|
||||
component_info=current_info,
|
||||
modifier=modifier,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
snapshot=current_snapshot,
|
||||
@@ -131,27 +146,37 @@ class Pipeline:
|
||||
review_errors.append(proposal_error)
|
||||
continue
|
||||
|
||||
_, metadata_error = self._current_modifier_info(
|
||||
modifier=modifier,
|
||||
expected_info=expected_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
)
|
||||
if metadata_error is not None:
|
||||
review_errors.append(metadata_error)
|
||||
continue
|
||||
|
||||
try:
|
||||
validate_component_batch(current_snapshot, proposals)
|
||||
validate_modifier_batch(current_snapshot, proposals)
|
||||
except Exception as error:
|
||||
review_errors.append(
|
||||
self._run_error(
|
||||
component_info=current_info,
|
||||
modifier_info=current_info,
|
||||
position=position,
|
||||
stage=ErrorStage.FINAL_REVIEW,
|
||||
error=error,
|
||||
unexpected_message="component edit batch does not satisfy the edit contract",
|
||||
unexpected_message="modifier edit batch does not satisfy the edit contract",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
residual_proposals.extend(
|
||||
ResidualProposal(
|
||||
component_id=current_info.component_id,
|
||||
component_version=current_info.version,
|
||||
component_position=position,
|
||||
modifier_id=current_info.modifier_id,
|
||||
modifier_version=current_info.version,
|
||||
modifier_position=position,
|
||||
proposal_ref=ProposalReference(
|
||||
component_position=position,
|
||||
modifier_position=position,
|
||||
snapshot_sha256=current_snapshot.sha256,
|
||||
proposal_index=proposal_index,
|
||||
),
|
||||
@@ -164,7 +189,7 @@ class Pipeline:
|
||||
return self._failed_result(
|
||||
input_snapshot=input_snapshot,
|
||||
current_snapshot=current_snapshot,
|
||||
component_infos=component_infos,
|
||||
modifier_infos=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
errors=tuple(review_errors),
|
||||
residual_proposals=tuple(residual_proposals),
|
||||
@@ -174,7 +199,7 @@ class Pipeline:
|
||||
status=RunStatus.UNSTABLE,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
residual_proposals=tuple(residual_proposals),
|
||||
partial_markdown=current_snapshot.markdown,
|
||||
@@ -183,100 +208,100 @@ class Pipeline:
|
||||
status=RunStatus.SUCCESS,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=tuple(changes),
|
||||
output_markdown=current_snapshot.markdown,
|
||||
)
|
||||
|
||||
def _preflight_components(self) -> tuple[tuple[ComponentInfo, ...], RunError | None]:
|
||||
component_infos: list[ComponentInfo] = []
|
||||
def _preflight_modifiers(self) -> tuple[tuple[ModifierInfo, ...], RunError | None]:
|
||||
modifier_infos: list[ModifierInfo] = []
|
||||
seen_ids: set[str] = set()
|
||||
for position, component in enumerate(self._components):
|
||||
if not isinstance(component, Component):
|
||||
error = ComponentContractError("pipeline entries must be Component instances")
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=None,
|
||||
for position, modifier in enumerate(self._modifiers):
|
||||
if not isinstance(modifier, Modifier):
|
||||
error = ModifierContractError("pipeline entries must be Modifier instances")
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=None,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="a pipeline entry is not a Component instance",
|
||||
unexpected_message="a pipeline entry is not a Modifier instance",
|
||||
)
|
||||
try:
|
||||
component_info = component._component_info()
|
||||
modifier_info = modifier._info()
|
||||
except Exception as error:
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=None,
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=None,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=error,
|
||||
unexpected_message="component metadata does not satisfy the component contract",
|
||||
unexpected_message="modifier metadata does not satisfy the modifier contract",
|
||||
)
|
||||
component_infos.append(component_info)
|
||||
if component_info.component_id in seen_ids:
|
||||
duplicate_error = PipelineContractError("pipeline component_id values must be unique")
|
||||
return tuple(component_infos), self._run_error(
|
||||
component_info=component_info,
|
||||
modifier_infos.append(modifier_info)
|
||||
if modifier_info.modifier_id in seen_ids:
|
||||
duplicate_error = PipelineContractError("pipeline modifier_id values must be unique")
|
||||
return tuple(modifier_infos), self._run_error(
|
||||
modifier_info=modifier_info,
|
||||
position=position,
|
||||
stage=ErrorStage.TRANSFORM,
|
||||
error=duplicate_error,
|
||||
unexpected_message="pipeline component_id values must be unique",
|
||||
unexpected_message="pipeline modifier_id values must be unique",
|
||||
)
|
||||
seen_ids.add(component_info.component_id)
|
||||
return tuple(component_infos), None
|
||||
seen_ids.add(modifier_info.modifier_id)
|
||||
return tuple(modifier_infos), None
|
||||
|
||||
def _current_component_info(
|
||||
def _current_modifier_info(
|
||||
self,
|
||||
component: Component,
|
||||
expected_info: ComponentInfo,
|
||||
modifier: Modifier,
|
||||
expected_info: ModifierInfo,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
) -> tuple[ComponentInfo, RunError | None]:
|
||||
) -> tuple[ModifierInfo, RunError | None]:
|
||||
try:
|
||||
current_info = component._component_info()
|
||||
current_info = modifier._info()
|
||||
if current_info != expected_info:
|
||||
raise ComponentContractError("component metadata changed during pipeline execution")
|
||||
raise ModifierContractError("modifier metadata changed during pipeline execution")
|
||||
except Exception as error:
|
||||
return expected_info, self._run_error(
|
||||
component_info=expected_info,
|
||||
modifier_info=expected_info,
|
||||
position=position,
|
||||
stage=stage,
|
||||
error=error,
|
||||
unexpected_message="component metadata does not satisfy the component contract",
|
||||
unexpected_message="modifier metadata does not satisfy the modifier contract",
|
||||
)
|
||||
return current_info, None
|
||||
|
||||
def _proposals(
|
||||
self,
|
||||
component: Component,
|
||||
component_info: ComponentInfo,
|
||||
modifier: Modifier,
|
||||
modifier_info: ModifierInfo,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
snapshot: DocumentSnapshot,
|
||||
) -> tuple[tuple[ProposedChange, ...], RunError | None]:
|
||||
try:
|
||||
proposals = component._collect_proposals(snapshot)
|
||||
proposals = collect_proposals(modifier, snapshot)
|
||||
except Exception as error:
|
||||
return (), self._run_error(
|
||||
component_info=component_info,
|
||||
modifier_info=modifier_info,
|
||||
position=position,
|
||||
stage=stage,
|
||||
error=error,
|
||||
unexpected_message="component could not produce contract-valid proposals",
|
||||
unexpected_message="modifier could not produce contract-valid proposals",
|
||||
)
|
||||
return proposals, None
|
||||
|
||||
@staticmethod
|
||||
def _run_error(
|
||||
component_info: ComponentInfo | None,
|
||||
modifier_info: ModifierInfo | None,
|
||||
position: int,
|
||||
stage: ErrorStage,
|
||||
error: Exception,
|
||||
unexpected_message: str,
|
||||
) -> RunError:
|
||||
return RunError(
|
||||
component_id=component_info.component_id if component_info is not None else "<invalid>",
|
||||
component_version=component_info.version if component_info is not None else "<invalid>",
|
||||
component_position=position,
|
||||
modifier_id=modifier_info.modifier_id if modifier_info is not None else "<invalid>",
|
||||
modifier_version=modifier_info.version if modifier_info is not None else "<invalid>",
|
||||
modifier_position=position,
|
||||
stage=stage,
|
||||
error_type=type(error).__name__,
|
||||
message=unexpected_message,
|
||||
@@ -286,7 +311,7 @@ class Pipeline:
|
||||
def _failed_result(
|
||||
input_snapshot: DocumentSnapshot,
|
||||
current_snapshot: DocumentSnapshot,
|
||||
component_infos: tuple[ComponentInfo, ...],
|
||||
modifier_infos: tuple[ModifierInfo, ...],
|
||||
changes: tuple[Change, ...],
|
||||
errors: tuple[RunError, ...],
|
||||
residual_proposals: tuple[ResidualProposal, ...] = (),
|
||||
@@ -295,7 +320,7 @@ class Pipeline:
|
||||
status=RunStatus.FAILED,
|
||||
input_sha256=input_snapshot.sha256,
|
||||
current_sha256=current_snapshot.sha256,
|
||||
components=component_infos,
|
||||
modifiers=modifier_infos,
|
||||
changes=changes,
|
||||
errors=errors,
|
||||
residual_proposals=residual_proposals,
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Safe factory for exact regular-expression replacements."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier, ModifierContractError
|
||||
|
||||
_DEFAULT_APPLICABILITY = "处理调用方正则表达式明确匹配的文本；不推断文档结构，也不自动启用。"
|
||||
_REASON = "应用调用方声明的正则表达式替换"
|
||||
|
||||
|
||||
def regex_replace(
|
||||
*,
|
||||
modifier_id: str,
|
||||
version: str,
|
||||
pattern: str,
|
||||
replacement: str,
|
||||
flags: int | re.RegexFlag = 0,
|
||||
applicability: str = _DEFAULT_APPLICABILITY,
|
||||
) -> Modifier:
|
||||
"""Create a modifier that turns non-empty regex matches into exact edits."""
|
||||
if not isinstance(pattern, str):
|
||||
raise ModifierContractError("regex pattern must be a string")
|
||||
if not isinstance(replacement, str):
|
||||
raise ModifierContractError("regex replacement must be a string")
|
||||
if type(flags) is not int and not isinstance(flags, re.RegexFlag):
|
||||
raise ModifierContractError("regex flags must be an integer or RegexFlag")
|
||||
try:
|
||||
compiled = re.compile(pattern, flags)
|
||||
except (re.error, ValueError) as error:
|
||||
raise ModifierContractError("regex pattern and flags must compile successfully") from error
|
||||
|
||||
empty_match = compiled.search("")
|
||||
if empty_match is not None and empty_match.start() == empty_match.end():
|
||||
raise ModifierContractError("regex patterns that produce zero-length matches are not supported")
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
proposals: list[ProposedChange] = []
|
||||
for match in compiled.finditer(snapshot.markdown):
|
||||
if match.start() == match.end():
|
||||
raise ModifierContractError("regex patterns that produce zero-length matches are not supported")
|
||||
expanded = match.expand(replacement)
|
||||
expected = match.group()
|
||||
if expected == expanded:
|
||||
continue
|
||||
proposals.append(
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=_REASON,
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(match.start(), match.end()),
|
||||
expected_text=expected,
|
||||
replacement=expanded,
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(proposals)
|
||||
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version=version,
|
||||
parameters={"flags": int(flags), "pattern": pattern, "replacement": replacement},
|
||||
applicability=applicability,
|
||||
propose=propose,
|
||||
)
|
||||
@@ -1,258 +0,0 @@
|
||||
"""Pure JSON and diff representations for local experiment results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from difflib import unified_diff
|
||||
from typing import TypeAlias
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
replay_change_chain,
|
||||
)
|
||||
from mdpolish.models import (
|
||||
Change,
|
||||
ComponentInfo,
|
||||
ParameterValue,
|
||||
ProposedChange,
|
||||
ResidualProposal,
|
||||
RunError,
|
||||
RunStatus,
|
||||
TextEdit,
|
||||
TransformResult,
|
||||
)
|
||||
|
||||
JsonValue: TypeAlias = bool | int | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
|
||||
JsonObject: TypeAlias = dict[str, JsonValue]
|
||||
|
||||
|
||||
ReportingError = ReplayError
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DocumentReport:
|
||||
"""Rendered artifacts and index facts for one document result."""
|
||||
|
||||
document_id: str
|
||||
source_label: str
|
||||
status: RunStatus
|
||||
input_sha256: str
|
||||
current_sha256: str
|
||||
change_count: int
|
||||
result_json: bytes
|
||||
cleaned_markdown: bytes | None
|
||||
diff: bytes | None
|
||||
|
||||
|
||||
def _parameter_value_json(value: ParameterValue) -> JsonValue:
|
||||
if isinstance(value, tuple):
|
||||
return [_parameter_value_json(item) for item in value]
|
||||
return value
|
||||
|
||||
|
||||
def component_info_json(component: ComponentInfo) -> JsonObject:
|
||||
"""Represent validated component metadata without changing tuple order."""
|
||||
parameters: list[JsonValue] = []
|
||||
for name, value in component.parameters:
|
||||
parameters.append([name, _parameter_value_json(value)])
|
||||
return {
|
||||
"component_id": component.component_id,
|
||||
"version": component.version,
|
||||
"parameters": parameters,
|
||||
"applicability": component.applicability,
|
||||
}
|
||||
|
||||
|
||||
def _proposal_reference_json(change: Change) -> JsonObject:
|
||||
reference = change.proposal_ref
|
||||
return {
|
||||
"component_position": reference.component_position,
|
||||
"snapshot_sha256": reference.snapshot_sha256,
|
||||
"proposal_index": reference.proposal_index,
|
||||
}
|
||||
|
||||
|
||||
def _text_edit_json(edit: TextEdit) -> JsonObject:
|
||||
return {
|
||||
"snapshot_sha256": edit.snapshot_sha256,
|
||||
"span": {"start": edit.span.start, "end": edit.span.end},
|
||||
"expected_text": edit.expected_text,
|
||||
"replacement": edit.replacement,
|
||||
}
|
||||
|
||||
|
||||
def _proposed_change_json(proposal: ProposedChange) -> JsonObject:
|
||||
edits: list[JsonValue] = [_text_edit_json(edit) for edit in proposal.edits]
|
||||
return {
|
||||
"snapshot_sha256": proposal.snapshot_sha256,
|
||||
"reason": proposal.reason,
|
||||
"edits": edits,
|
||||
}
|
||||
|
||||
|
||||
def _residual_proposal_json(residual: ResidualProposal) -> JsonObject:
|
||||
reference = residual.proposal_ref
|
||||
return {
|
||||
"component_id": residual.component_id,
|
||||
"component_version": residual.component_version,
|
||||
"component_position": residual.component_position,
|
||||
"proposal_ref": {
|
||||
"component_position": reference.component_position,
|
||||
"snapshot_sha256": reference.snapshot_sha256,
|
||||
"proposal_index": reference.proposal_index,
|
||||
},
|
||||
"proposal": _proposed_change_json(residual.proposal),
|
||||
}
|
||||
|
||||
|
||||
def _run_error_json(error: RunError) -> JsonObject:
|
||||
return {
|
||||
"component_id": error.component_id,
|
||||
"component_version": error.component_version,
|
||||
"component_position": error.component_position,
|
||||
"stage": error.stage.value,
|
||||
"error_type": error.error_type,
|
||||
"message": error.message,
|
||||
}
|
||||
|
||||
|
||||
def _replay_changes(input_markdown: str, result: TransformResult) -> tuple[tuple[tuple[int, int], ...], str]:
|
||||
current_markdown = result.output_markdown if result.status is RunStatus.SUCCESS else result.partial_markdown
|
||||
if current_markdown is None:
|
||||
raise ReportingError("a transform result does not contain its status-specific Markdown")
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=input_markdown,
|
||||
input_sha256=result.input_sha256,
|
||||
components=tuple(ReplayComponent(item.component_id, item.version) for item in result.components),
|
||||
changes=tuple(
|
||||
ReplayChange(
|
||||
component_id=change.component_id,
|
||||
component_version=change.component_version,
|
||||
component_position=change.component_position,
|
||||
proposal_component_position=change.proposal_ref.component_position,
|
||||
proposal_snapshot_sha256=change.proposal_ref.snapshot_sha256,
|
||||
proposal_index=change.proposal_ref.proposal_index,
|
||||
edit_index=change.edit_index,
|
||||
start=change.span.start,
|
||||
end=change.span.end,
|
||||
before=change.before,
|
||||
after=change.after,
|
||||
before_sha256=change.before_sha256,
|
||||
after_sha256=change.after_sha256,
|
||||
)
|
||||
for change in result.changes
|
||||
),
|
||||
current_sha256=result.current_sha256,
|
||||
current_markdown=current_markdown,
|
||||
include_zero_change_stages=False,
|
||||
)
|
||||
locations = tuple((item.line, item.column) for item in replayed.changes)
|
||||
return locations, replayed.current_markdown
|
||||
|
||||
|
||||
def _change_json(change: Change, location: tuple[int, int]) -> JsonObject:
|
||||
line, column = location
|
||||
return {
|
||||
"component_id": change.component_id,
|
||||
"component_version": change.component_version,
|
||||
"component_position": change.component_position,
|
||||
"proposal_ref": _proposal_reference_json(change),
|
||||
"edit_index": change.edit_index,
|
||||
"reason": change.reason,
|
||||
"span": {"start": change.span.start, "end": change.span.end},
|
||||
"location": {"line": line, "column": column},
|
||||
"before": change.before,
|
||||
"after": change.after,
|
||||
"before_sha256": change.before_sha256,
|
||||
"after_sha256": change.after_sha256,
|
||||
}
|
||||
|
||||
|
||||
def encode_json(payload: JsonObject) -> bytes:
|
||||
"""Encode a schema object using the approved stable local representation."""
|
||||
return (json.dumps(payload, ensure_ascii=False, indent=2) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def build_unified_diff(document_id: str, before: str, after: str) -> bytes:
|
||||
"""Build the human review view using logical document names only."""
|
||||
if before == after:
|
||||
return b""
|
||||
lines = unified_diff(
|
||||
before.splitlines(keepends=True),
|
||||
after.splitlines(keepends=True),
|
||||
fromfile=f"a/{document_id}.md",
|
||||
tofile=f"b/{document_id}.md",
|
||||
n=3,
|
||||
lineterm="\n",
|
||||
)
|
||||
normalized: list[str] = []
|
||||
line_endings = ("\r\n", "\n", "\r", "\v", "\f", "\x1c", "\x1d", "\x1e", "\x85", "\u2028", "\u2029")
|
||||
for line in lines:
|
||||
for ending in line_endings:
|
||||
if line.endswith(ending):
|
||||
normalized.append(line[: -len(ending)] + "\n")
|
||||
break
|
||||
else:
|
||||
normalized.append(line + "\n\\ No newline at end of file\n")
|
||||
return "".join(normalized).encode("utf-8")
|
||||
|
||||
|
||||
def build_document_report(
|
||||
*,
|
||||
document_id: str,
|
||||
source_label: str,
|
||||
input_markdown: str,
|
||||
result: TransformResult,
|
||||
) -> DocumentReport:
|
||||
"""Render one core result after replaying and verifying every applied batch."""
|
||||
locations, current_markdown = _replay_changes(input_markdown, result)
|
||||
changes: list[JsonValue] = [
|
||||
_change_json(change, location) for change, location in zip(result.changes, locations, strict=True)
|
||||
]
|
||||
errors: list[JsonValue] = [_run_error_json(error) for error in result.errors]
|
||||
residual_proposals: list[JsonValue] = [
|
||||
_residual_proposal_json(residual) for residual in result.residual_proposals
|
||||
]
|
||||
|
||||
if result.status is RunStatus.SUCCESS:
|
||||
cleaned_path: JsonValue = "cleaned.md"
|
||||
diff_path: JsonValue = "changes.diff"
|
||||
cleaned_markdown = current_markdown.encode("utf-8")
|
||||
diff = build_unified_diff(document_id, input_markdown, current_markdown)
|
||||
else:
|
||||
cleaned_path = None
|
||||
diff_path = None
|
||||
cleaned_markdown = None
|
||||
diff = None
|
||||
|
||||
payload: JsonObject = {
|
||||
"schema_version": 1,
|
||||
"document": {
|
||||
"document_id": document_id,
|
||||
"source_label": source_label,
|
||||
},
|
||||
"status": result.status.value,
|
||||
"input_sha256": result.input_sha256,
|
||||
"current_sha256": result.current_sha256,
|
||||
"changes": changes,
|
||||
"errors": errors,
|
||||
"residual_proposals": residual_proposals,
|
||||
"output": {
|
||||
"cleaned_path": cleaned_path,
|
||||
"diff_path": diff_path,
|
||||
},
|
||||
}
|
||||
return DocumentReport(
|
||||
document_id=document_id,
|
||||
source_label=source_label,
|
||||
status=result.status,
|
||||
input_sha256=result.input_sha256,
|
||||
current_sha256=result.current_sha256,
|
||||
change_count=len(result.changes),
|
||||
result_json=encode_json(payload),
|
||||
cleaned_markdown=cleaned_markdown,
|
||||
diff=diff,
|
||||
)
|
||||
@@ -1,212 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
from typing import Any, TypedDict, cast
|
||||
|
||||
|
||||
class ReviewFixture(TypedDict):
|
||||
run_directory: Path
|
||||
source_path: Path
|
||||
manifest_path: Path
|
||||
locator_path: Path
|
||||
result_path: Path
|
||||
original: str
|
||||
cleaned: str
|
||||
|
||||
|
||||
def digest(text: str) -> str:
|
||||
return sha256(text.encode()).hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, payload: object) -> None:
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
return cast(dict[str, Any], json.loads(path.read_text(encoding="utf-8")))
|
||||
|
||||
|
||||
def create_review_run(
|
||||
root: Path,
|
||||
*,
|
||||
status: str = "success",
|
||||
with_locator: bool = True,
|
||||
) -> ReviewFixture:
|
||||
run_directory = root / "artifacts/2026-08-23/runs/review-run"
|
||||
document_directory = run_directory / "documents/paper"
|
||||
document_directory.mkdir(parents=True)
|
||||
source_path = root / "paper.md"
|
||||
original = "\ufeff😀 old\r\nCafe\u0301\n"
|
||||
cleaned = "\ufeff😀 new\r\nCafe\u0301\n"
|
||||
source_path.write_text(original, encoding="utf-8")
|
||||
input_hash = digest(original)
|
||||
current_hash = digest(cleaned) if status == "success" else input_hash
|
||||
changes: list[dict[str, object]] = []
|
||||
if status == "success":
|
||||
changes.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"proposal_ref": {
|
||||
"component_position": 0,
|
||||
"snapshot_sha256": input_hash,
|
||||
"proposal_index": 0,
|
||||
},
|
||||
"edit_index": 0,
|
||||
"reason": "替换测试单词",
|
||||
"span": {"start": 3, "end": 6},
|
||||
"location": {"line": 1, "column": 4},
|
||||
"before": "old",
|
||||
"after": "new",
|
||||
"before_sha256": input_hash,
|
||||
"after_sha256": current_hash,
|
||||
}
|
||||
)
|
||||
errors: list[dict[str, object]] = []
|
||||
if status == "failed":
|
||||
errors.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"stage": "transform",
|
||||
"error_type": "SyntheticError",
|
||||
"message": "测试组件失败。",
|
||||
}
|
||||
)
|
||||
residuals: list[dict[str, object]] = []
|
||||
if status == "unstable":
|
||||
residuals.append(
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"component_version": "1.0.0",
|
||||
"component_position": 0,
|
||||
"proposal_ref": {
|
||||
"component_position": 0,
|
||||
"snapshot_sha256": input_hash,
|
||||
"proposal_index": 0,
|
||||
},
|
||||
"proposal": {
|
||||
"snapshot_sha256": input_hash,
|
||||
"reason": "仍可替换测试单词",
|
||||
"edits": [
|
||||
{
|
||||
"snapshot_sha256": input_hash,
|
||||
"span": {"start": 3, "end": 6},
|
||||
"expected_text": "old",
|
||||
"replacement": "new",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"document": {"document_id": "paper", "source_label": "inputs/paper.md"},
|
||||
"status": status,
|
||||
"input_sha256": input_hash,
|
||||
"current_sha256": current_hash,
|
||||
"changes": changes,
|
||||
"errors": errors,
|
||||
"residual_proposals": residuals,
|
||||
"output": {
|
||||
"cleaned_path": "cleaned.md" if status == "success" else None,
|
||||
"diff_path": "changes.diff" if status == "success" else None,
|
||||
},
|
||||
}
|
||||
result_path = document_directory / "result.json"
|
||||
write_json(result_path, result)
|
||||
if status == "success":
|
||||
(document_directory / "cleaned.md").write_text(cleaned, encoding="utf-8")
|
||||
(document_directory / "changes.diff").write_text("synthetic diff\n", encoding="utf-8")
|
||||
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "review-run",
|
||||
"run_date": "2026-08-23",
|
||||
"utc_offset": "+08:00",
|
||||
"status": status,
|
||||
"started_at_utc": "2026-08-23T01:00:00Z",
|
||||
"completed_at_utc": "2026-08-23T01:01:00Z",
|
||||
"retention_until": "2026-09-22T01:01:00Z",
|
||||
},
|
||||
"tool": {
|
||||
"name": "mdpolish",
|
||||
"package_version": "0.1.0",
|
||||
"python_version": "3.13.11",
|
||||
"platform": "linux-x86_64",
|
||||
"git_commit": None,
|
||||
"git_dirty": None,
|
||||
},
|
||||
"pipeline": {
|
||||
"components": [
|
||||
{
|
||||
"component_id": "paper.rule",
|
||||
"version": "1.0.0",
|
||||
"parameters": [],
|
||||
"applicability": "替换测试单词。",
|
||||
},
|
||||
{
|
||||
"component_id": "paper.zero",
|
||||
"version": "1.0.0",
|
||||
"parameters": [],
|
||||
"applicability": "不修改当前测试文档。",
|
||||
},
|
||||
]
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "paper",
|
||||
"source_label": "inputs/paper.md",
|
||||
"status": status,
|
||||
"input_sha256": input_hash,
|
||||
"current_sha256": current_hash,
|
||||
"change_count": len(changes),
|
||||
"result_path": "documents/paper/result.json",
|
||||
"cleaned_path": "documents/paper/cleaned.md" if status == "success" else None,
|
||||
"diff_path": "documents/paper/changes.diff" if status == "success" else None,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"document_count": 1,
|
||||
"success_count": int(status == "success"),
|
||||
"failed_count": int(status == "failed"),
|
||||
"unstable_count": int(status == "unstable"),
|
||||
"change_count": len(changes),
|
||||
},
|
||||
}
|
||||
manifest_path = run_directory / "manifest.json"
|
||||
write_json(manifest_path, manifest)
|
||||
locator_path = run_directory / "review-locator.json"
|
||||
if with_locator:
|
||||
write_json(
|
||||
locator_path,
|
||||
{
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "review-run",
|
||||
"run_directory": str(run_directory.resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "paper",
|
||||
"source_path": str(source_path.resolve()),
|
||||
"input_sha256": input_hash,
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
return {
|
||||
"run_directory": run_directory,
|
||||
"source_path": source_path,
|
||||
"manifest_path": manifest_path,
|
||||
"locator_path": locator_path,
|
||||
"result_path": result_path,
|
||||
"original": original,
|
||||
"cleaned": cleaned,
|
||||
}
|
||||
@@ -1,189 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish._artifact_replay import (
|
||||
ReplayChange,
|
||||
ReplayComponent,
|
||||
ReplayError,
|
||||
replay_change_chain,
|
||||
)
|
||||
from mdpolish.models import markdown_sha256
|
||||
|
||||
|
||||
def change(
|
||||
*,
|
||||
component_position: int,
|
||||
before_text: str,
|
||||
after_text: str,
|
||||
start: int,
|
||||
end: int,
|
||||
before_sha256: str,
|
||||
after_sha256: str,
|
||||
proposal_index: int = 0,
|
||||
edit_index: int = 0,
|
||||
) -> ReplayChange:
|
||||
return ReplayChange(
|
||||
component_id=f"test.{component_position}",
|
||||
component_version="1.0.0",
|
||||
component_position=component_position,
|
||||
proposal_component_position=component_position,
|
||||
proposal_snapshot_sha256=before_sha256,
|
||||
proposal_index=proposal_index,
|
||||
edit_index=edit_index,
|
||||
start=start,
|
||||
end=end,
|
||||
before=before_text,
|
||||
after=after_text,
|
||||
before_sha256=before_sha256,
|
||||
after_sha256=after_sha256,
|
||||
)
|
||||
|
||||
|
||||
def test_replay_builds_zero_change_stages_and_utf16_editor_ranges() -> None:
|
||||
original = "\ufeff😀 old\r\nCafe\u0301"
|
||||
final = "\ufeff😀 new\r\nCafe\u0301"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
recorded = change(
|
||||
component_position=0,
|
||||
before_text="old",
|
||||
after_text="new",
|
||||
start=3,
|
||||
end=6,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"), ReplayComponent("test.1", "1.0.0")),
|
||||
changes=(recorded,),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
assert replayed.current_markdown == final
|
||||
assert len(replayed.stages) == 2
|
||||
assert replayed.stages[1].before_markdown == final
|
||||
assert replayed.stages[1].after_markdown == final
|
||||
assert replayed.stages[1].changes == ()
|
||||
assert (replayed.changes[0].line, replayed.changes[0].column) == (1, 4)
|
||||
assert (replayed.changes[0].editor_start, replayed.changes[0].editor_end) == (4, 7)
|
||||
|
||||
|
||||
def test_replay_uses_full_descending_application_key_not_record_order() -> None:
|
||||
original = "abcd"
|
||||
final = "aXXcYY"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
right = change(
|
||||
component_position=0,
|
||||
before_text="d",
|
||||
after_text="YY",
|
||||
start=3,
|
||||
end=4,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
proposal_index=1,
|
||||
)
|
||||
left = change(
|
||||
component_position=0,
|
||||
before_text="b",
|
||||
after_text="XX",
|
||||
start=1,
|
||||
end=2,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
|
||||
replayed = replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(right, left),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
assert replayed.current_markdown == final
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mutate", "message"),
|
||||
[
|
||||
(lambda item: replace(item, component_position=2), "component position"),
|
||||
(lambda item: replace(item, proposal_component_position=1), "proposal reference"),
|
||||
(lambda item: replace(item, before="bad"), "recorded snapshot"),
|
||||
(lambda item: replace(item, after_sha256="0" * 64), "batch hash"),
|
||||
],
|
||||
)
|
||||
def test_replay_rejects_untrusted_change_chains(
|
||||
mutate: Callable[[ReplayChange], ReplayChange], message: str
|
||||
) -> None:
|
||||
original = "old"
|
||||
final = "new"
|
||||
input_hash = markdown_sha256(original)
|
||||
final_hash = markdown_sha256(final)
|
||||
valid = change(
|
||||
component_position=0,
|
||||
before_text="old",
|
||||
after_text="new",
|
||||
start=0,
|
||||
end=3,
|
||||
before_sha256=input_hash,
|
||||
after_sha256=final_hash,
|
||||
)
|
||||
tampered = mutate(valid)
|
||||
|
||||
with pytest.raises(ReplayError, match=message):
|
||||
replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(tampered,),
|
||||
current_sha256=final_hash,
|
||||
current_markdown=final,
|
||||
include_zero_change_stages=True,
|
||||
)
|
||||
|
||||
|
||||
def test_replay_rejects_conflicting_ranges() -> None:
|
||||
original = "abc"
|
||||
input_hash = markdown_sha256(original)
|
||||
first = change(
|
||||
component_position=0,
|
||||
before_text="ab",
|
||||
after_text="x",
|
||||
start=0,
|
||||
end=2,
|
||||
before_sha256=input_hash,
|
||||
after_sha256="0" * 64,
|
||||
)
|
||||
second = change(
|
||||
component_position=0,
|
||||
before_text="bc",
|
||||
after_text="y",
|
||||
start=1,
|
||||
end=3,
|
||||
before_sha256=input_hash,
|
||||
after_sha256="0" * 64,
|
||||
proposal_index=1,
|
||||
)
|
||||
|
||||
with pytest.raises(ReplayError, match="conflicting"):
|
||||
replay_change_chain(
|
||||
input_markdown=original,
|
||||
input_sha256=input_hash,
|
||||
components=(ReplayComponent("test.0", "1.0.0"),),
|
||||
changes=(first, second),
|
||||
current_sha256=input_hash,
|
||||
current_markdown=None,
|
||||
include_zero_change_stages=False,
|
||||
)
|
||||
@@ -1,398 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import stat
|
||||
from hashlib import sha256
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
import mdpolish.artifact_store as artifact_store
|
||||
from mdpolish.artifact_store import ArtifactStoreError, StoredDocument, publish_run
|
||||
|
||||
_SOURCE_BYTES = b"source\n"
|
||||
|
||||
|
||||
def stored_document(document_id: str = "paper", output: bytes | None = b"cleaned\n") -> StoredDocument:
|
||||
status = "success" if output is not None else "failed"
|
||||
current_sha256 = sha256(output).hexdigest() if output is not None else "2" * 64
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"document": {"document_id": document_id, "source_label": f"inputs/{document_id}.md"},
|
||||
"status": status,
|
||||
"input_sha256": sha256(_SOURCE_BYTES).hexdigest(),
|
||||
"current_sha256": current_sha256,
|
||||
"changes": [],
|
||||
"errors": [] if status == "success" else [{"error_type": "SyntheticError"}],
|
||||
"residual_proposals": [],
|
||||
"output": {
|
||||
"cleaned_path": "cleaned.md" if output is not None else None,
|
||||
"diff_path": "changes.diff" if output is not None else None,
|
||||
},
|
||||
}
|
||||
return StoredDocument(
|
||||
document_id=document_id,
|
||||
result_json=(json.dumps(payload, indent=2) + "\n").encode(),
|
||||
cleaned_markdown=output,
|
||||
diff=b"" if output is not None else None,
|
||||
output_sha256=sha256(output).hexdigest() if output is not None else None,
|
||||
)
|
||||
|
||||
|
||||
def manifest_json(
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> bytes:
|
||||
indexes: list[dict[str, object]] = []
|
||||
statuses: list[str] = []
|
||||
change_count = 0
|
||||
for document in documents:
|
||||
report = json.loads(document.result_json)
|
||||
status = report["status"]
|
||||
statuses.append(status)
|
||||
change_count += len(report["changes"])
|
||||
base = f"documents/{document.document_id}"
|
||||
indexes.append(
|
||||
{
|
||||
"document_id": document.document_id,
|
||||
"source_label": report["document"]["source_label"],
|
||||
"status": status,
|
||||
"input_sha256": report["input_sha256"],
|
||||
"current_sha256": report["current_sha256"],
|
||||
"change_count": len(report["changes"]),
|
||||
"result_path": f"{base}/result.json",
|
||||
"cleaned_path": f"{base}/cleaned.md" if status == "success" else None,
|
||||
"diff_path": f"{base}/changes.diff" if status == "success" else None,
|
||||
}
|
||||
)
|
||||
failed_count = statuses.count("failed")
|
||||
unstable_count = statuses.count("unstable")
|
||||
overall_status = "failed" if failed_count else "unstable" if unstable_count else "success"
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"run": {"run_id": run_id, "run_date": run_date, "status": overall_status},
|
||||
"documents": indexes,
|
||||
"summary": {
|
||||
"document_count": len(documents),
|
||||
"success_count": statuses.count("success"),
|
||||
"failed_count": failed_count,
|
||||
"unstable_count": unstable_count,
|
||||
"change_count": change_count,
|
||||
},
|
||||
}
|
||||
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||
|
||||
|
||||
def review_locator_json(
|
||||
artifacts_root: Path,
|
||||
run_date: str,
|
||||
run_id: str,
|
||||
documents: tuple[StoredDocument, ...],
|
||||
) -> bytes:
|
||||
source_root = artifacts_root.parent / f"{artifacts_root.name}-sources"
|
||||
source_root.mkdir(exist_ok=True)
|
||||
locator_documents: list[dict[str, object]] = []
|
||||
for position, document in enumerate(documents):
|
||||
source_path = source_root / f"{position}-{document.document_id}.md"
|
||||
source_path.write_bytes(_SOURCE_BYTES)
|
||||
report = json.loads(document.result_json)
|
||||
locator_documents.append(
|
||||
{
|
||||
"document_id": document.document_id,
|
||||
"source_path": str(source_path.resolve()),
|
||||
"input_sha256": report["input_sha256"],
|
||||
}
|
||||
)
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": run_id,
|
||||
"run_directory": str((artifacts_root / run_date / "runs" / run_id).resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": locator_documents,
|
||||
}
|
||||
return (json.dumps(payload, indent=2) + "\n").encode()
|
||||
|
||||
|
||||
def test_publish_run_creates_private_date_layout_and_status_specific_files(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document("success"), stored_document("failed", None))
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "example-run", documents)
|
||||
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="example-run",
|
||||
manifest_json=manifest_json("2026-08-22", "example-run", documents),
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert run_directory == artifacts_root / "2026-08-22" / "runs" / "example-run"
|
||||
assert json.loads((run_directory / "manifest.json").read_bytes())["run"]["status"] == "failed"
|
||||
assert (run_directory / "documents/success/result.json").is_file()
|
||||
assert (run_directory / "documents/success/cleaned.md").read_bytes() == b"cleaned\n"
|
||||
assert (run_directory / "documents/success/changes.diff").read_bytes() == b""
|
||||
assert (run_directory / "documents/failed/result.json").is_file()
|
||||
assert (run_directory / "review-locator.json").read_bytes() == locator_json
|
||||
assert not (run_directory / "documents/failed/cleaned.md").exists()
|
||||
assert not (run_directory / "documents/failed/changes.diff").exists()
|
||||
|
||||
for directory in (
|
||||
artifacts_root,
|
||||
artifacts_root / "2026-08-22",
|
||||
artifacts_root / "2026-08-22/runs",
|
||||
run_directory,
|
||||
run_directory / "documents",
|
||||
run_directory / "documents/success",
|
||||
):
|
||||
assert stat.S_IMODE(directory.stat().st_mode) == 0o700
|
||||
for artifact_file in run_directory.rglob("*"):
|
||||
if artifact_file.is_file():
|
||||
assert stat.S_IMODE(artifact_file.stat().st_mode) == 0o600
|
||||
|
||||
|
||||
def test_publish_run_rejects_existing_target_without_overwriting(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
first_manifest = manifest_json("2026-08-22", "same-run", documents)
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "same-run", documents)
|
||||
run_directory = publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="same-run",
|
||||
manifest_json=first_manifest,
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="already exists"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="same-run",
|
||||
manifest_json=first_manifest,
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert (run_directory / "manifest.json").read_bytes() == first_manifest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("run_date", "run_id"),
|
||||
[
|
||||
("2026-8-22", "valid"),
|
||||
("2026-02-30", "valid"),
|
||||
("2026-08-22", "Uppercase"),
|
||||
("2026-08-22", "../escape"),
|
||||
("2026-08-22", "two..dots"),
|
||||
],
|
||||
)
|
||||
def test_publish_run_rejects_unsafe_date_and_run_id(tmp_path: Path, run_date: str, run_id: str) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
with pytest.raises(ArtifactStoreError):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date=run_date,
|
||||
run_id=run_id,
|
||||
manifest_json=manifest_json(run_date, run_id, documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, run_date, run_id, documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_inconsistent_or_duplicate_document_artifacts(tmp_path: Path) -> None:
|
||||
valid = stored_document()
|
||||
bad_hash = StoredDocument("paper", valid.result_json, b"output", b"", "0" * 64)
|
||||
first_artifacts_root = tmp_path / "artifacts-a"
|
||||
with pytest.raises(ArtifactStoreError, match="output hash"):
|
||||
publish_run(
|
||||
artifacts_root=first_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="bad-hash",
|
||||
manifest_json=manifest_json("2026-08-22", "bad-hash", (bad_hash,)),
|
||||
review_locator_json=review_locator_json(
|
||||
first_artifacts_root, "2026-08-22", "bad-hash", (bad_hash,)
|
||||
),
|
||||
documents=(bad_hash,),
|
||||
)
|
||||
|
||||
duplicates = (stored_document(), stored_document())
|
||||
second_artifacts_root = tmp_path / "artifacts-b"
|
||||
with pytest.raises(ArtifactStoreError, match="unique"):
|
||||
publish_run(
|
||||
artifacts_root=second_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="duplicate",
|
||||
manifest_json=manifest_json("2026-08-22", "duplicate", duplicates),
|
||||
review_locator_json=review_locator_json(
|
||||
second_artifacts_root, "2026-08-22", "duplicate", duplicates
|
||||
),
|
||||
documents=duplicates,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_manifest_path_or_document_mismatch(tmp_path: Path) -> None:
|
||||
documents = (stored_document(),)
|
||||
wrong_date = manifest_json("2026-08-21", "review", documents)
|
||||
first_artifacts_root = tmp_path / "artifacts-a"
|
||||
with pytest.raises(ArtifactStoreError, match="identity"):
|
||||
publish_run(
|
||||
artifacts_root=first_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=wrong_date,
|
||||
review_locator_json=review_locator_json(
|
||||
first_artifacts_root, "2026-08-22", "review", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
payload = json.loads(manifest_json("2026-08-22", "review", documents))
|
||||
payload["documents"][0]["change_count"] = 99
|
||||
mismatched_index = (json.dumps(payload, indent=2) + "\n").encode()
|
||||
second_artifacts_root = tmp_path / "artifacts-b"
|
||||
with pytest.raises(ArtifactStoreError, match="index"):
|
||||
publish_run(
|
||||
artifacts_root=second_artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=mismatched_index,
|
||||
review_locator_json=review_locator_json(
|
||||
second_artifacts_root, "2026-08-22", "review", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
|
||||
def test_publish_run_rejects_inconsistent_review_locator(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
locator = json.loads(review_locator_json(artifacts_root, "2026-08-22", "review", documents))
|
||||
locator["documents"][0]["input_sha256"] = "0" * 64
|
||||
inconsistent_locator = (json.dumps(locator, indent=2) + "\n").encode()
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="document identity"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||
review_locator_json=inconsistent_locator,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_publish_run_rejects_source_changed_after_locator_creation(tmp_path: Path) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
locator_json = review_locator_json(artifacts_root, "2026-08-22", "review", documents)
|
||||
locator = json.loads(locator_json)
|
||||
Path(locator["documents"][0]["source_path"]).write_bytes(b"changed\n")
|
||||
|
||||
with pytest.raises(ArtifactStoreError, match="input hash"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="review",
|
||||
manifest_json=manifest_json("2026-08-22", "review", documents),
|
||||
review_locator_json=locator_json,
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_publish_race_does_not_replace_a_new_target(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
original_rename = artifact_store._rename_no_replace
|
||||
|
||||
def create_competing_target(source: Path, target: Path) -> None:
|
||||
target.mkdir(mode=0o700)
|
||||
(target / "keep").write_bytes(b"existing")
|
||||
original_rename(source, target)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_rename_no_replace", create_competing_target)
|
||||
with pytest.raises(ArtifactStoreError, match="already exists"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="raced",
|
||||
manifest_json=manifest_json("2026-08-22", "raced", documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "raced", documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
target = tmp_path / "artifacts/2026-08-22/runs/raced"
|
||||
assert (target / "keep").read_bytes() == b"existing"
|
||||
assert not any(path.name.startswith(".raced.") for path in target.parent.iterdir())
|
||||
|
||||
|
||||
def test_write_failure_cleans_temporary_directory_and_does_not_publish(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
original_write = artifact_store._write_private_file
|
||||
call_count = 0
|
||||
|
||||
def fail_second_write(path: Path, content: bytes) -> None:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count == 2:
|
||||
raise OSError("synthetic write failure")
|
||||
original_write(path, content)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_write_private_file", fail_second_write)
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
|
||||
with pytest.raises(OSError, match="synthetic"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="broken",
|
||||
manifest_json=manifest_json("2026-08-22", "broken", documents),
|
||||
review_locator_json=review_locator_json(artifacts_root, "2026-08-22", "broken", documents),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = tmp_path / "artifacts/2026-08-22/runs"
|
||||
assert list(runs_directory.iterdir()) == []
|
||||
|
||||
|
||||
def test_locator_write_failure_does_not_publish(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
original_write = artifact_store._write_private_file
|
||||
|
||||
def fail_locator_write(path: Path, content: bytes) -> None:
|
||||
if path.name == "review-locator.json":
|
||||
raise OSError("synthetic locator write failure")
|
||||
original_write(path, content)
|
||||
|
||||
monkeypatch.setattr(artifact_store, "_write_private_file", fail_locator_write)
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
documents = (stored_document(),)
|
||||
|
||||
with pytest.raises(OSError, match="locator"):
|
||||
publish_run(
|
||||
artifacts_root=artifacts_root,
|
||||
run_date="2026-08-22",
|
||||
run_id="locator-failure",
|
||||
manifest_json=manifest_json("2026-08-22", "locator-failure", documents),
|
||||
review_locator_json=review_locator_json(
|
||||
artifacts_root, "2026-08-22", "locator-failure", documents
|
||||
),
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
runs_directory = tmp_path / "artifacts/2026-08-22/runs"
|
||||
assert list(runs_directory.iterdir()) == []
|
||||
@@ -1,155 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
import mdpolish
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ArxivSubmissionStampComponent
|
||||
|
||||
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||
OTHER_STAMP = "arXiv:2301.7v1 [cs.AI] 1 Jan 2023"
|
||||
REASON = "删除完整匹配的 arXiv 提交边栏戳"
|
||||
|
||||
|
||||
def transform(markdown: str) -> mdpolish.TransformResult:
|
||||
return Pipeline([ArxivSubmissionStampComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_component_metadata_and_package_export() -> None:
|
||||
result = transform("")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.components[0].component_id == "paper.arxiv_submission_stamp"
|
||||
assert result.components[0].version == "1.0.0"
|
||||
assert result.components[0].parameters == ()
|
||||
assert result.components[0].applicability
|
||||
assert not hasattr(mdpolish, "ArxivSubmissionStampComponent")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("markdown", ["", "普通正文", "中文\nCafe\u0301\n🙂\n"])
|
||||
def test_no_target_returns_unchanged_success(markdown: str) -> None:
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
(f"{STAMP}\n正文", "正文"),
|
||||
(f"正文\n{STAMP}\n后文", "正文\n后文"),
|
||||
(f"正文\n{STAMP}", "正文\n"),
|
||||
(STAMP, ""),
|
||||
(f"正文\r\n{STAMP}\r\n后文", "正文\r\n后文"),
|
||||
(f"正文\r{STAMP}\r后文", "正文\r后文"),
|
||||
],
|
||||
)
|
||||
def test_deletes_target_at_approved_line_boundaries(markdown: str, expected: str) -> None:
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == expected
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_multiple_targets_are_reported_in_source_order_with_one_atomic_batch() -> None:
|
||||
markdown = f"{STAMP}\n保留\n{OTHER_STAMP}"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "保留\n"
|
||||
assert [change.before for change in result.changes] == [f"{STAMP}\n", OTHER_STAMP]
|
||||
assert [change.proposal_ref.proposal_index for change in result.changes] == [0, 1]
|
||||
assert [change.span.start for change in result.changes] == sorted(change.span.start for change in result.changes)
|
||||
assert {change.before_sha256 for change in result.changes} == {result.input_sha256}
|
||||
assert {change.after_sha256 for change in result.changes} == {result.current_sha256}
|
||||
|
||||
|
||||
def test_adjacent_targets_use_non_overlapping_delete_ranges() -> None:
|
||||
result = transform(f"{STAMP}\n{OTHER_STAMP}")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == ""
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].span.end == result.changes[1].span.start
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"line",
|
||||
[
|
||||
f" {STAMP}",
|
||||
f"{STAMP} ",
|
||||
f"- {STAMP}",
|
||||
f"> {STAMP}",
|
||||
"1. Example. arXiv preprint arXiv:2104.12345v2 [stat.ME], 2021.",
|
||||
"See arXiv:2104.12345v2 for details.",
|
||||
"arXiv:2104.12345 [stat.ME] 31 Dec 2021",
|
||||
"arXiv:hep-ph/9901001 [hep-ph] 31 Dec 1999",
|
||||
"arXiv:2104.12345v2 31 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 0 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 32 Dec 2021",
|
||||
"arXiv:2104.12345v2 [stat.ME] 31 December 2021",
|
||||
"arXiv:2104.12345v2 [统计] 31 Dec 2021",
|
||||
],
|
||||
)
|
||||
def test_similar_arxiv_text_is_preserved(line: str) -> None:
|
||||
markdown = f"前文\n{line}\n后文"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
def test_matching_line_inside_fenced_code_is_not_protected() -> None:
|
||||
markdown = f"```text\n{STAMP}\n```\n"
|
||||
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "```text\n```\n"
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_change_audit_records_identity_reason_source_and_batch_hashes() -> None:
|
||||
result = transform(f"{STAMP}\n正文")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
change = result.changes[0]
|
||||
assert change.component_id == "paper.arxiv_submission_stamp"
|
||||
assert change.component_version == "1.0.0"
|
||||
assert change.component_position == 0
|
||||
assert change.proposal_ref.component_position == 0
|
||||
assert change.proposal_ref.proposal_index == 0
|
||||
assert change.edit_index == 0
|
||||
assert change.reason == REASON
|
||||
assert change.before == f"{STAMP}\n"
|
||||
assert change.after == ""
|
||||
assert change.before_sha256 == result.input_sha256
|
||||
assert change.after_sha256 == result.current_sha256
|
||||
|
||||
|
||||
def test_successful_output_is_stable_and_second_run_has_no_changes() -> None:
|
||||
pipeline = Pipeline([ArxivSubmissionStampComponent()])
|
||||
|
||||
first = pipeline.transform(f"{STAMP}\n正文")
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
assert first.output_markdown == "正文"
|
||||
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.output_markdown == "正文"
|
||||
assert second.changes == ()
|
||||
assert second.residual_proposals == ()
|
||||
|
||||
|
||||
def test_same_input_produces_same_ordered_result() -> None:
|
||||
pipeline = Pipeline([ArxivSubmissionStampComponent()])
|
||||
markdown = f"{STAMP}\n正文\n{OTHER_STAMP}\n"
|
||||
|
||||
assert pipeline.transform(markdown) == pipeline.transform(markdown)
|
||||
@@ -1,162 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import (
|
||||
ArxivSubmissionStampComponent,
|
||||
HtmlTableDoubleEscapeComponent,
|
||||
HtmlTableLayoutComponent,
|
||||
ManuscriptLineNumberComponent,
|
||||
PageBreakWordJoinComponent,
|
||||
ReferenceSpacingComponent,
|
||||
RepeatedRunningHeaderComponent,
|
||||
WordReviewCommentComponent,
|
||||
)
|
||||
|
||||
STAMP = "arXiv:2104.12345v2 [stat.ME] 31 Dec 2021"
|
||||
HEADER = "## Repeated Paper Header"
|
||||
MAPPINGS = (
|
||||
("medi-", "cal", "medical"),
|
||||
("possi-", "bly", "possibly"),
|
||||
("cre-", "ated", "created"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("life-", "threatening", "life-threatening"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def _build_pipeline() -> Pipeline:
|
||||
return Pipeline(
|
||||
[
|
||||
WordReviewCommentComponent(),
|
||||
ManuscriptLineNumberComponent(),
|
||||
ArxivSubmissionStampComponent(),
|
||||
RepeatedRunningHeaderComponent(),
|
||||
PageBreakWordJoinComponent(MAPPINGS),
|
||||
HtmlTableDoubleEscapeComponent(),
|
||||
HtmlTableLayoutComponent(),
|
||||
ReferenceSpacingComponent(),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _numbered_manuscript() -> list[str]:
|
||||
return [
|
||||
f"## {number} Section {number}" if number in {5, 15} else f"{number} body {number}"
|
||||
for number in range(1, 21)
|
||||
]
|
||||
|
||||
|
||||
def _combined_markdown() -> str:
|
||||
return "\n".join(
|
||||
(
|
||||
"1 Affiliation",
|
||||
"## Abstract",
|
||||
*_numbered_manuscript(),
|
||||
"Commented [A1]: remove this",
|
||||
"",
|
||||
STAMP,
|
||||
"Sentence continues in",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"the next line.",
|
||||
"A word is possi-",
|
||||
"",
|
||||
"bly split.",
|
||||
"<table><tr><td>&lt;5</td></tr><tr><td>B</td></tr></table>",
|
||||
"## References",
|
||||
"",
|
||||
"1. First",
|
||||
"",
|
||||
"2. Second",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"3. Third",
|
||||
"4. Fourth",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_script_builds_frozen_component_order_and_parameters() -> None:
|
||||
script_path = Path(__file__).parents[1] / "scripts" / "run_clindb_first_batch_experiment.py"
|
||||
module = ast.parse(script_path.read_text(encoding="utf-8"))
|
||||
build_function = next(
|
||||
node for node in module.body if isinstance(node, ast.FunctionDef) and node.name == "build_pipeline"
|
||||
)
|
||||
component_names = [
|
||||
call.func.id
|
||||
for node in ast.walk(build_function)
|
||||
if isinstance(node, ast.List)
|
||||
for call in node.elts
|
||||
if isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
|
||||
]
|
||||
mapping_assignment = next(
|
||||
node
|
||||
for node in module.body
|
||||
if isinstance(node, ast.Assign)
|
||||
and any(isinstance(target, ast.Name) and target.id == "CLINDB_WORD_JOIN_MAPPINGS" for target in node.targets)
|
||||
)
|
||||
assert component_names == [
|
||||
"WordReviewCommentComponent",
|
||||
"ManuscriptLineNumberComponent",
|
||||
"ArxivSubmissionStampComponent",
|
||||
"RepeatedRunningHeaderComponent",
|
||||
"PageBreakWordJoinComponent",
|
||||
"HtmlTableDoubleEscapeComponent",
|
||||
"HtmlTableLayoutComponent",
|
||||
"ReferenceSpacingComponent",
|
||||
]
|
||||
assert ast.literal_eval(mapping_assignment.value) == MAPPINGS
|
||||
|
||||
pipeline = _build_pipeline()
|
||||
result = pipeline.transform("")
|
||||
|
||||
assert [component.component_id for component in result.components] == [
|
||||
"paper.word_review_comment",
|
||||
"paper.manuscript_line_number",
|
||||
"paper.arxiv_submission_stamp",
|
||||
"paper.repeated_running_header",
|
||||
"paper.page_break_word_join",
|
||||
"markdown.html_table_double_escape",
|
||||
"markdown.html_table_layout",
|
||||
"paper.reference_spacing",
|
||||
]
|
||||
assert len(MAPPINGS) == 6
|
||||
|
||||
|
||||
def test_full_pipeline_is_audited_stable_and_idempotent() -> None:
|
||||
pipeline = _build_pipeline()
|
||||
first = pipeline.transform(_combined_markdown())
|
||||
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
assert first.output_markdown is not None
|
||||
assert first.residual_proposals == ()
|
||||
counts: dict[str, int] = {}
|
||||
for change in first.changes:
|
||||
counts[change.component_id] = counts.get(change.component_id, 0) + 1
|
||||
assert counts == {
|
||||
"paper.word_review_comment": 1,
|
||||
"paper.manuscript_line_number": 20,
|
||||
"paper.arxiv_submission_stamp": 1,
|
||||
"paper.repeated_running_header": 2,
|
||||
"paper.page_break_word_join": 1,
|
||||
"markdown.html_table_double_escape": 1,
|
||||
"markdown.html_table_layout": 1,
|
||||
"paper.reference_spacing": 1,
|
||||
}
|
||||
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.output_markdown == first.output_markdown
|
||||
assert second.changes == ()
|
||||
|
||||
|
||||
def test_business_components_are_not_exported_from_core_namespace() -> None:
|
||||
import mdpolish
|
||||
|
||||
assert not hasattr(mdpolish, "WordReviewCommentComponent")
|
||||
assert not hasattr(mdpolish, "HtmlTableLayoutComponent")
|
||||
@@ -1,121 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, ComponentContractError, DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
|
||||
|
||||
class ExampleComponent(Component):
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
component_id: object = "test.example",
|
||||
version: object = "1.2.3",
|
||||
parameters: object = None,
|
||||
applicability: object = "处理测试标记，要求精确匹配，排除所有其他内容。",
|
||||
) -> None:
|
||||
self._component_id = component_id
|
||||
self._version = version
|
||||
self._parameters = {} if parameters is None else parameters
|
||||
self._applicability = applicability
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return cast(str, self._component_id)
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return cast(str, self._version)
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return cast(Mapping[str, object], self._parameters)
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return cast(str, self._applicability)
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if not snapshot.markdown:
|
||||
return ()
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), snapshot.markdown[0], "X")
|
||||
return (ProposedChange(snapshot.sha256, "replace first character", (edit,)),)
|
||||
|
||||
|
||||
class ListReturningComponent(ExampleComponent):
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
|
||||
|
||||
class WrongValueComponent(ExampleComponent):
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], ("wrong",))
|
||||
|
||||
|
||||
def test_component_metadata_is_validated_and_parameters_are_frozen_deterministically() -> None:
|
||||
component = ExampleComponent(
|
||||
parameters={
|
||||
"z": [1, {"b": False, "a": None}],
|
||||
"a": "value",
|
||||
}
|
||||
)
|
||||
|
||||
info = component._component_info()
|
||||
|
||||
assert info.component_id == "test.example"
|
||||
assert info.version == "1.2.3"
|
||||
assert info.parameters == (
|
||||
("a", "value"),
|
||||
("z", (1, (("a", None), ("b", False)))),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("component_id", ["", "Uppercase", "has space", "two..dots", "_leading"])
|
||||
def test_invalid_component_id_is_a_contract_error(component_id: str) -> None:
|
||||
with pytest.raises(ComponentContractError, match="component_id"):
|
||||
ExampleComponent(component_id=component_id)._component_info()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", ["1", "1.2", "v1.2.3", "01.2.3", "1.2.3-alpha"])
|
||||
def test_invalid_version_is_a_contract_error(version: str) -> None:
|
||||
with pytest.raises(ComponentContractError, match=r"MAJOR.MINOR.PATCH"):
|
||||
ExampleComponent(version=version)._component_info()
|
||||
|
||||
|
||||
def test_empty_applicability_is_a_contract_error() -> None:
|
||||
with pytest.raises(ComponentContractError, match="applicability"):
|
||||
ExampleComponent(applicability=" \n")._component_info()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"parameters",
|
||||
[
|
||||
{"bad": {1, 2}},
|
||||
{"bad": float("inf")},
|
||||
{"bad": float("nan")},
|
||||
{1: "non-string key"},
|
||||
["not", "a", "mapping"],
|
||||
],
|
||||
)
|
||||
def test_unrepresentable_parameters_are_contract_errors(parameters: object) -> None:
|
||||
with pytest.raises(ComponentContractError, match="parameter"):
|
||||
ExampleComponent(parameters=parameters)._component_info()
|
||||
|
||||
|
||||
def test_collect_proposals_requires_a_tuple_of_proposed_changes() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
|
||||
with pytest.raises(ComponentContractError, match="return a tuple"):
|
||||
ListReturningComponent()._collect_proposals(snapshot)
|
||||
with pytest.raises(ComponentContractError, match="only ProposedChange"):
|
||||
WrongValueComponent()._collect_proposals(snapshot)
|
||||
|
||||
|
||||
def test_component_exposes_no_public_check_or_transform_shortcut() -> None:
|
||||
component = ExampleComponent()
|
||||
|
||||
assert not hasattr(component, "check")
|
||||
assert not hasattr(component, "transform")
|
||||
+25
-25
@@ -3,17 +3,17 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import (
|
||||
ComponentInfo,
|
||||
DocumentSnapshot,
|
||||
EditValidationError,
|
||||
ModifierInfo,
|
||||
ProposedChange,
|
||||
TextEdit,
|
||||
TextSpan,
|
||||
apply_component_batch,
|
||||
apply_modifier_batch,
|
||||
)
|
||||
|
||||
COMPONENT = ComponentInfo(
|
||||
component_id="test.component",
|
||||
MODIFIER = ModifierInfo(
|
||||
modifier_id="test.modifier",
|
||||
version="1.2.3",
|
||||
parameters=(),
|
||||
applicability="测试精确文本编辑，只处理测试字符串，排除其他输入。",
|
||||
@@ -38,22 +38,22 @@ def test_applies_insert_delete_and_replace() -> None:
|
||||
delete_snapshot = DocumentSnapshot("abc")
|
||||
replace_snapshot = DocumentSnapshot("abc")
|
||||
|
||||
inserted = apply_component_batch(
|
||||
inserted = apply_modifier_batch(
|
||||
insert_snapshot,
|
||||
(make_proposal(insert_snapshot, make_edit(insert_snapshot, 1, 1, "X")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
deleted = apply_component_batch(
|
||||
deleted = apply_modifier_batch(
|
||||
delete_snapshot,
|
||||
(make_proposal(delete_snapshot, make_edit(delete_snapshot, 1, 2, "")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
replaced = apply_component_batch(
|
||||
replaced = apply_modifier_batch(
|
||||
replace_snapshot,
|
||||
(make_proposal(replace_snapshot, make_edit(replace_snapshot, 1, 2, "X")),),
|
||||
COMPONENT,
|
||||
MODIFIER,
|
||||
0,
|
||||
)
|
||||
|
||||
@@ -71,7 +71,7 @@ def test_multiple_edits_apply_backwards_but_report_in_source_order() -> None:
|
||||
reason="normalize two locations",
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 3)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 3)
|
||||
|
||||
assert applied.snapshot.markdown == "AbcdF"
|
||||
assert [change.span.start for change in applied.changes] == [0, 4]
|
||||
@@ -82,7 +82,7 @@ def test_multiple_edits_apply_backwards_but_report_in_source_order() -> None:
|
||||
}
|
||||
assert {change.reason for change in applied.changes} == {"normalize two locations"}
|
||||
assert [change.edit_index for change in applied.changes] == [1, 0]
|
||||
assert all(change.component_position == 3 for change in applied.changes)
|
||||
assert all(change.modifier_position == 3 for change in applied.changes)
|
||||
|
||||
|
||||
def test_adjacent_nonempty_ranges_are_allowed() -> None:
|
||||
@@ -93,7 +93,7 @@ def test_adjacent_nonempty_ranges_are_allowed() -> None:
|
||||
make_edit(snapshot, 2, 4, "D"),
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot.markdown == "AD"
|
||||
|
||||
@@ -107,7 +107,7 @@ def test_overlapping_ranges_fail_without_changing_snapshot() -> None:
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert snapshot.markdown == "abcdef"
|
||||
assert snapshot.sha256 == DocumentSnapshot("abcdef").sha256
|
||||
@@ -119,7 +119,7 @@ def test_duplicate_edits_fail_explicitly() -> None:
|
||||
proposal = make_proposal(snapshot, edit, edit)
|
||||
|
||||
with pytest.raises(EditValidationError, match="duplicate"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_distinct_insert_points_are_allowed() -> None:
|
||||
@@ -130,7 +130,7 @@ def test_distinct_insert_points_are_allowed() -> None:
|
||||
make_edit(snapshot, 3, 3, "Y"),
|
||||
)
|
||||
|
||||
applied = apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot.markdown == "aXbcYd"
|
||||
|
||||
@@ -144,7 +144,7 @@ def test_same_insert_point_conflicts() -> None:
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("insert_position", [1, 2, 3])
|
||||
@@ -157,7 +157,7 @@ def test_insert_at_start_inside_or_end_of_nonempty_range_conflicts(insert_positi
|
||||
)
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (proposal,), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (proposal,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_stale_hash_out_of_range_and_expected_text_mismatch_fail() -> None:
|
||||
@@ -166,34 +166,34 @@ def test_stale_hash_out_of_range_and_expected_text_mismatch_fail() -> None:
|
||||
stale = make_proposal(original, make_edit(original, 0, 1, "A"))
|
||||
|
||||
with pytest.raises(EditValidationError, match="stale"):
|
||||
apply_component_batch(current, (stale,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (stale,), MODIFIER, 0)
|
||||
|
||||
out_of_range_edit = TextEdit(current.sha256, TextSpan(2, 5), "dxx", "D")
|
||||
out_of_range = make_proposal(current, out_of_range_edit)
|
||||
with pytest.raises(EditValidationError, match="outside"):
|
||||
apply_component_batch(current, (out_of_range,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (out_of_range,), MODIFIER, 0)
|
||||
|
||||
mismatch_edit = TextEdit(current.sha256, TextSpan(0, 1), "z", "A")
|
||||
mismatch = make_proposal(current, mismatch_edit)
|
||||
with pytest.raises(EditValidationError, match="expected_text"):
|
||||
apply_component_batch(current, (mismatch,), COMPONENT, 0)
|
||||
apply_modifier_batch(current, (mismatch,), MODIFIER, 0)
|
||||
|
||||
|
||||
def test_conflict_across_proposals_rejects_whole_component_batch() -> None:
|
||||
def test_conflict_across_proposals_rejects_whole_modifier_batch() -> None:
|
||||
snapshot = DocumentSnapshot("abcdef")
|
||||
first = make_proposal(snapshot, make_edit(snapshot, 0, 3, "X"), reason="first")
|
||||
second = make_proposal(snapshot, make_edit(snapshot, 2, 4, "Y"), reason="second")
|
||||
|
||||
with pytest.raises(EditValidationError, match="conflicting"):
|
||||
apply_component_batch(snapshot, (first, second), COMPONENT, 0)
|
||||
apply_modifier_batch(snapshot, (first, second), MODIFIER, 0)
|
||||
|
||||
assert snapshot.markdown == "abcdef"
|
||||
|
||||
|
||||
def test_empty_component_batch_keeps_same_snapshot_and_records_nothing() -> None:
|
||||
def test_empty_modifier_batch_keeps_same_snapshot_and_records_nothing() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
|
||||
applied = apply_component_batch(snapshot, (), COMPONENT, 0)
|
||||
applied = apply_modifier_batch(snapshot, (), MODIFIER, 0)
|
||||
|
||||
assert applied.snapshot is snapshot
|
||||
assert applied.changes == ()
|
||||
|
||||
@@ -1,356 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from datetime import UTC, datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, DocumentSnapshot, Pipeline, ProposedChange, RunStatus, TextEdit, TextSpan
|
||||
from mdpolish.experiment import ExperimentError, InputDocument, ToolMetadata, run_experiment
|
||||
|
||||
|
||||
class ConditionalComponent(Component):
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "test.conditional"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {"needle": "old", "replacement": "new"}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "替换测试标记 old，遇到 boom 时模拟组件失败，排除其他内容。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if snapshot.markdown == "boom":
|
||||
raise RuntimeError("private source")
|
||||
position = snapshot.markdown.find("old")
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(position, position + 3), "old", "new")
|
||||
return (ProposedChange(snapshot.sha256, "replace old test marker", (edit,)),)
|
||||
|
||||
|
||||
class NonIdempotentComponent(Component):
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return "test.non-idempotent"
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "在测试文本末尾反复插入标记，只用于验证 unstable 产物边界。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = len(snapshot.markdown)
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(position, position), "", "!")
|
||||
return (ProposedChange(snapshot.sha256, "append synthetic marker", (edit,)),)
|
||||
|
||||
|
||||
def tool_metadata() -> ToolMetadata:
|
||||
return ToolMetadata(
|
||||
name="mdpolish",
|
||||
package_version="0.1.0",
|
||||
python_version="3.13.11",
|
||||
platform="linux-x86_64",
|
||||
git_commit="a" * 40,
|
||||
git_dirty=False,
|
||||
)
|
||||
|
||||
|
||||
def input_document(path: Path, document_id: str) -> InputDocument:
|
||||
return InputDocument(document_id, path, f"inputs/{path.name}")
|
||||
|
||||
|
||||
def test_run_experiment_publishes_manifest_reports_diff_and_preserves_inputs(tmp_path: Path) -> None:
|
||||
first_path = tmp_path / "first.md"
|
||||
second_path = tmp_path / "second.md"
|
||||
first_path.write_bytes(b"before old\r\nafter\r\n")
|
||||
second_path.write_text("unchanged\n", encoding="utf-8")
|
||||
original_first = first_path.read_bytes()
|
||||
original_second = second_path.read_bytes()
|
||||
local_timezone = timezone(timedelta(hours=8))
|
||||
started_at = datetime(2026, 8, 22, 10, 30, tzinfo=local_timezone)
|
||||
completed_at = datetime(2026, 8, 22, 2, 31, tzinfo=UTC)
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(first_path, "first"), input_document(second_path, "second")),
|
||||
run_id="local-review",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.run_directory == tmp_path / "artifacts/2026-08-22/runs/local-review"
|
||||
assert result.document_count == 2
|
||||
assert result.success_count == 2
|
||||
assert result.change_count == 1
|
||||
assert first_path.read_bytes() == original_first
|
||||
assert second_path.read_bytes() == original_second
|
||||
assert (result.run_directory / "documents/first/cleaned.md").read_bytes() == b"before new\r\nafter\r\n"
|
||||
assert (result.run_directory / "documents/second/cleaned.md").read_bytes() == original_second
|
||||
assert (result.run_directory / "documents/second/changes.diff").read_bytes() == b""
|
||||
|
||||
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
||||
locator = json.loads((result.run_directory / "review-locator.json").read_bytes())
|
||||
assert locator == {
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "local-review",
|
||||
"run_directory": str(result.run_directory.resolve()),
|
||||
"manifest_path": "manifest.json",
|
||||
},
|
||||
"documents": [
|
||||
{
|
||||
"document_id": "first",
|
||||
"source_path": str(first_path.resolve()),
|
||||
"input_sha256": manifest["documents"][0]["input_sha256"],
|
||||
},
|
||||
{
|
||||
"document_id": "second",
|
||||
"source_path": str(second_path.resolve()),
|
||||
"input_sha256": manifest["documents"][1]["input_sha256"],
|
||||
},
|
||||
],
|
||||
}
|
||||
assert manifest["run"]["run_date"] == "2026-08-22"
|
||||
assert manifest["run"]["utc_offset"] == "+08:00"
|
||||
assert manifest["run"]["started_at_utc"] == "2026-08-22T02:30:00Z"
|
||||
assert manifest["run"]["completed_at_utc"] == "2026-08-22T02:31:00Z"
|
||||
assert manifest["run"]["retention_until"] == "2026-09-21T02:31:00Z"
|
||||
assert manifest["pipeline"]["components"][0]["component_id"] == "test.conditional"
|
||||
assert manifest["pipeline"]["components"][0]["parameters"] == [
|
||||
["needle", "old"],
|
||||
["replacement", "new"],
|
||||
]
|
||||
assert manifest["summary"] == {
|
||||
"document_count": 2,
|
||||
"success_count": 2,
|
||||
"failed_count": 0,
|
||||
"unstable_count": 0,
|
||||
"change_count": 1,
|
||||
}
|
||||
first_report = json.loads((result.run_directory / "documents/first/result.json").read_bytes())
|
||||
assert first_report["changes"][0]["location"] == {"line": 1, "column": 8}
|
||||
assert first_report["changes"][0]["before"] == "old"
|
||||
assert b"--- a/first.md\n+++ b/first.md\n" in (
|
||||
result.run_directory / "documents/first/changes.diff"
|
||||
).read_bytes()
|
||||
|
||||
|
||||
def test_document_failure_is_isolated_and_never_writes_partial_markdown(tmp_path: Path) -> None:
|
||||
good_path = tmp_path / "good.md"
|
||||
bad_path = tmp_path / "bad.md"
|
||||
good_path.write_text("old", encoding="utf-8")
|
||||
bad_path.write_text("boom", encoding="utf-8")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(good_path, "good"), input_document(bad_path, "bad")),
|
||||
run_id="mixed",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, 10, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 10, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.success_count == 1
|
||||
assert result.failed_count == 1
|
||||
assert (result.run_directory / "documents/good/cleaned.md").read_text() == "new"
|
||||
assert (result.run_directory / "documents/bad/result.json").is_file()
|
||||
assert not (result.run_directory / "documents/bad/cleaned.md").exists()
|
||||
assert not (result.run_directory / "documents/bad/changes.diff").exists()
|
||||
bad_report = json.loads((result.run_directory / "documents/bad/result.json").read_bytes())
|
||||
assert bad_report["status"] == "failed"
|
||||
assert "partial_markdown" not in bad_report
|
||||
|
||||
|
||||
def test_preflight_rejects_invalid_utf8_without_running_or_creating_artifacts(tmp_path: Path) -> None:
|
||||
invalid_path = tmp_path / "invalid.md"
|
||||
invalid_path.write_bytes(b"\xff")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
|
||||
with pytest.raises(ExperimentError, match="strict UTF-8"):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([ConditionalComponent()]),
|
||||
documents=(input_document(invalid_path, "invalid"),),
|
||||
run_id="invalid-input",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_preflight_rejects_invalid_document_manifests_before_creating_artifacts(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
directory_path = tmp_path / "directory"
|
||||
directory_path.mkdir()
|
||||
missing_path = tmp_path / "missing.md"
|
||||
cases = (
|
||||
(input_document(source_path, "duplicate"), input_document(source_path, "duplicate")),
|
||||
(input_document(source_path, "first"), input_document(source_path, "second")),
|
||||
(input_document(missing_path, "missing"),),
|
||||
(input_document(directory_path, "directory"),),
|
||||
(input_document(source_path, "two..dots"),),
|
||||
)
|
||||
|
||||
for index, documents in enumerate(cases):
|
||||
artifacts_root = tmp_path / f"artifacts-{index}"
|
||||
with pytest.raises(ExperimentError):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=documents,
|
||||
run_id="preflight",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
assert not artifacts_root.exists()
|
||||
|
||||
|
||||
def test_preflight_rejects_an_output_path_nested_under_an_input_file(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ExperimentError, match="cannot overlap"):
|
||||
run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="overlap",
|
||||
artifacts_root=source_path,
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert source_path.read_text(encoding="utf-8") == "content"
|
||||
|
||||
|
||||
def test_utf8_bom_crlf_and_missing_final_newline_are_preserved_exactly(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "bom.md"
|
||||
empty_path = tmp_path / "empty.md"
|
||||
source_bytes = b"\xef\xbb\xbfhead\r\nlast"
|
||||
source_path.write_bytes(source_bytes)
|
||||
empty_path.write_bytes(b"")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "bom"), input_document(empty_path, "empty")),
|
||||
run_id="byte-preservation",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert source_path.read_bytes() == source_bytes
|
||||
assert (result.run_directory / "documents/bom/cleaned.md").read_bytes() == source_bytes
|
||||
assert (result.run_directory / "documents/bom/changes.diff").read_bytes() == b""
|
||||
assert empty_path.read_bytes() == b""
|
||||
assert (result.run_directory / "documents/empty/cleaned.md").read_bytes() == b""
|
||||
assert (result.run_directory / "documents/empty/changes.diff").read_bytes() == b""
|
||||
|
||||
|
||||
def test_unstable_document_only_publishes_a_result_json(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("content", encoding="utf-8")
|
||||
|
||||
result = run_experiment(
|
||||
pipeline=Pipeline([NonIdempotentComponent()]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="unstable",
|
||||
artifacts_root=tmp_path / "artifacts",
|
||||
started_at=datetime(2026, 8, 22, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert result.status is RunStatus.UNSTABLE
|
||||
assert result.unstable_count == 1
|
||||
assert (result.run_directory / "documents/paper/result.json").is_file()
|
||||
assert not (result.run_directory / "documents/paper/cleaned.md").exists()
|
||||
assert not (result.run_directory / "documents/paper/changes.diff").exists()
|
||||
manifest = json.loads((result.run_directory / "manifest.json").read_bytes())
|
||||
assert manifest["run"]["status"] == "unstable"
|
||||
|
||||
|
||||
def test_existing_same_date_run_is_rejected_without_overwriting(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("old", encoding="utf-8")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
pipeline = Pipeline([ConditionalComponent()])
|
||||
documents = (input_document(source_path, "paper"),)
|
||||
started_at = datetime(2026, 8, 22, tzinfo=UTC)
|
||||
completed_at = datetime(2026, 8, 22, 0, 1, tzinfo=UTC)
|
||||
tool = tool_metadata()
|
||||
first = run_experiment(
|
||||
pipeline=pipeline,
|
||||
documents=documents,
|
||||
run_id="same",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool,
|
||||
)
|
||||
original_manifest = (first.run_directory / "manifest.json").read_bytes()
|
||||
|
||||
with pytest.raises(ExperimentError, match="already exists"):
|
||||
run_experiment(
|
||||
pipeline=pipeline,
|
||||
documents=documents,
|
||||
run_id="same",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=started_at,
|
||||
completed_at=completed_at,
|
||||
tool=tool,
|
||||
)
|
||||
|
||||
assert (first.run_directory / "manifest.json").read_bytes() == original_manifest
|
||||
|
||||
|
||||
def test_same_run_id_on_another_local_date_uses_a_separate_directory(tmp_path: Path) -> None:
|
||||
source_path = tmp_path / "paper.md"
|
||||
source_path.write_text("unchanged", encoding="utf-8")
|
||||
artifacts_root = tmp_path / "artifacts"
|
||||
|
||||
first = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="daily",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 22, 23, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 22, 23, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
second = run_experiment(
|
||||
pipeline=Pipeline([]),
|
||||
documents=(input_document(source_path, "paper"),),
|
||||
run_id="daily",
|
||||
artifacts_root=artifacts_root,
|
||||
started_at=datetime(2026, 8, 23, 0, tzinfo=UTC),
|
||||
completed_at=datetime(2026, 8, 23, 0, 1, tzinfo=UTC),
|
||||
tool=tool_metadata(),
|
||||
)
|
||||
|
||||
assert first.run_directory.parent.parent.name == "2026-08-22"
|
||||
assert second.run_directory.parent.parent.name == "2026-08-23"
|
||||
@@ -3,11 +3,11 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import HtmlTableDoubleEscapeComponent
|
||||
from mdpolish.modifiers import html_table_entity_unescape
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([HtmlTableDoubleEscapeComponent()]).transform(markdown)
|
||||
return Pipeline([html_table_entity_unescape()]).transform(markdown)
|
||||
|
||||
|
||||
def test_unescapes_one_layer_only_in_strict_cell_text() -> None:
|
||||
@@ -47,13 +47,13 @@ def test_non_strict_or_outside_content_is_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_fenced_table_is_not_protected() -> None:
|
||||
def test_fenced_table_is_not_protected_by_the_lexical_subset() -> None:
|
||||
markdown = "```html\n<table><tr><td>&lt;</td></tr></table>\n```"
|
||||
assert transform(markdown).output_markdown == "```html\n<table><tr><td><</td></tr></table>\n```"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([HtmlTableDoubleEscapeComponent()])
|
||||
pipeline = Pipeline([html_table_entity_unescape()])
|
||||
first = pipeline.transform("<table><tr><td>&lt;</td></tr></table>")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -3,13 +3,13 @@ from __future__ import annotations
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import HtmlTableLayoutComponent
|
||||
from mdpolish.modifiers import html_table_layout
|
||||
|
||||
TABLE = '<table class="x"><tr><td colspan="2">A</td></tr><tr><td>B</td><td>C</td></tr></table>'
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([HtmlTableLayoutComponent()]).transform(markdown)
|
||||
return Pipeline([html_table_layout()]).transform(markdown)
|
||||
|
||||
|
||||
def test_expands_rows_without_changing_tags_attributes_or_cells() -> None:
|
||||
@@ -57,7 +57,7 @@ def test_mixed_multiline_or_non_strict_tables_are_preserved(markdown: str) -> No
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([HtmlTableLayoutComponent()])
|
||||
pipeline = Pipeline([html_table_layout()])
|
||||
first = pipeline.transform(TABLE)
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ManuscriptLineNumberComponent
|
||||
|
||||
|
||||
def _document(
|
||||
*,
|
||||
count: int = 20,
|
||||
heading_numbers: frozenset[int] = frozenset({5, 15}),
|
||||
numbers: tuple[int, ...] | None = None,
|
||||
line_ending: str = "\n",
|
||||
) -> str:
|
||||
values = numbers if numbers is not None else tuple(range(1, count + 1))
|
||||
body = [
|
||||
f"## {number} Section {number}" if number in heading_numbers else f"{number} body {number}"
|
||||
for number in values
|
||||
]
|
||||
return line_ending.join(("1 Affiliation", "2 Institute", "## Abstract", *body))
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([ManuscriptLineNumberComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_removes_long_monotonic_sequence_but_preserves_pre_abstract_affiliations() -> None:
|
||||
result = transform(_document())
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.startswith("1 Affiliation\n2 Institute\n## Abstract\nbody 1")
|
||||
assert "## Section 5" in result.output_markdown
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_all_supported_line_endings(line_ending: str) -> None:
|
||||
result = transform(_document(line_ending=line_ending))
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.count(line_ending) == _document(line_ending=line_ending).count(line_ending)
|
||||
|
||||
|
||||
def test_allows_skipped_numbers_when_sequence_is_strictly_increasing() -> None:
|
||||
numbers = tuple(range(10, 30))
|
||||
result = transform(_document(numbers=numbers, heading_numbers=frozenset({14, 24})))
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
_document(count=19, heading_numbers=frozenset({5, 15})),
|
||||
_document(heading_numbers=frozenset({5})),
|
||||
_document(numbers=(*tuple(range(1, 20)), 10), heading_numbers=frozenset({5, 15})),
|
||||
_document().replace("## Abstract", "## ABSTRACT"),
|
||||
_document() + "\n## Abstract",
|
||||
],
|
||||
)
|
||||
def test_incomplete_or_ambiguous_evidence_preserves_the_document(markdown: str) -> None:
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == markdown
|
||||
assert result.changes == ()
|
||||
|
||||
|
||||
def test_lists_years_and_numbers_inside_body_are_not_candidates() -> None:
|
||||
markdown = _document() + "\n1. list\n1) list\n2024 report\nThe panel included 35 experts"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.output_markdown is not None
|
||||
assert result.output_markdown.endswith("1. list\n1) list\n2024 report\nThe panel included 35 experts")
|
||||
assert len(result.changes) == 20
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([ManuscriptLineNumberComponent()])
|
||||
first = pipeline.transform(_document())
|
||||
assert first.output_markdown is not None
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.changes == ()
|
||||
@@ -0,0 +1,606 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from importlib import import_module
|
||||
from importlib.metadata import version as distribution_version
|
||||
from typing import TYPE_CHECKING, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import DocumentSnapshot, ModifierContractError, Pipeline, RunStatus
|
||||
from mdpolish.modifiers import mapped_line_join
|
||||
from mdpolish.modifiers.mapped_line_join import (
|
||||
ExactLineJoinRule,
|
||||
KeepLineJoinRule,
|
||||
LexicalAmbiguityPolicy,
|
||||
LexicalCandidateForm,
|
||||
LexicalLineJoinRule,
|
||||
LexiconBackend,
|
||||
LineBreakPolicy,
|
||||
LineEndRegexRule,
|
||||
LineJoinBlock,
|
||||
LineJoinConflictPolicy,
|
||||
LineJoinRule,
|
||||
RegexLineJoinRule,
|
||||
UnicodeNormalization,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from _pytest.monkeypatch import MonkeyPatch
|
||||
|
||||
mapped_module = import_module("mdpolish.modifiers.mapped_line_join")
|
||||
|
||||
LEGACY_MAPPINGS = (
|
||||
("exam-", "ple", "example"),
|
||||
("rule-", "based", "rule-based"),
|
||||
("value.", "ues", "values"),
|
||||
)
|
||||
|
||||
|
||||
def transform(
|
||||
markdown: str,
|
||||
rules: tuple[tuple[str, str, str] | LineJoinRule, ...] = LEGACY_MAPPINGS,
|
||||
**options: object,
|
||||
) -> str:
|
||||
modifier = mapped_line_join(rules, **options) # type: ignore[arg-type]
|
||||
result = Pipeline((modifier,)).transform(markdown)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown is not None
|
||||
return result.output_markdown
|
||||
|
||||
|
||||
def exact_rule(**overrides: object) -> ExactLineJoinRule:
|
||||
fields: dict[str, object] = {
|
||||
"rule_id": "example.join",
|
||||
"left": "exam",
|
||||
"right": "ple",
|
||||
"replacement": "example",
|
||||
"separator": "-",
|
||||
}
|
||||
fields.update(overrides)
|
||||
return ExactLineJoinRule(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
("an exam-\nple here", "an example here"),
|
||||
("an exam-\n\nple here", "an example here"),
|
||||
("an exam-\r\n\r\nple here", "an example here"),
|
||||
("an exam-\r\rple here", "an example here"),
|
||||
("a rule-\nbased method", "a rule-based method"),
|
||||
("the value.\nues differ", "the values differ"),
|
||||
],
|
||||
)
|
||||
def test_legacy_exact_mapping_supports_lf_crlf_cr_and_one_blank_line(markdown: str, expected: str) -> None:
|
||||
assert transform(markdown) == expected
|
||||
|
||||
|
||||
def test_exact_rules_distinguish_explicit_and_empty_separator() -> None:
|
||||
hyphenated = exact_rule()
|
||||
unseparated = exact_rule(rule_id="example.unseparated", separator="")
|
||||
|
||||
assert transform("exam-\nple", (hyphenated,)) == "example"
|
||||
assert transform("exam\nple", (unseparated,)) == "example"
|
||||
assert transform("exam\nple", (hyphenated,)) == "exam\nple"
|
||||
|
||||
|
||||
def test_named_regex_combines_backreferences_from_both_sides() -> None:
|
||||
rule = RegexLineJoinRule(
|
||||
rule_id="regex.join",
|
||||
left_pattern=r"(?P<stem>[A-Za-z]+)",
|
||||
right_pattern=r"(?P<suffix>[a-z]+)",
|
||||
replacement=r"\g<stem>_\g<suffix>",
|
||||
separator="-",
|
||||
)
|
||||
|
||||
assert transform("prefix exam-\nple suffix", (rule,)) == "prefix exam_ple suffix"
|
||||
|
||||
|
||||
def test_regex_keeps_its_inline_flags() -> None:
|
||||
rule = RegexLineJoinRule(
|
||||
rule_id="regex.flags",
|
||||
left_pattern=r"(?i:(?P<stem>exam))",
|
||||
right_pattern=r"(?i:(?P<suffix>ple))",
|
||||
replacement=r"\g<stem>\g<suffix>",
|
||||
separator="-",
|
||||
)
|
||||
|
||||
assert transform("EXAM-\nPLE", (rule,)) == "EXAMPLE"
|
||||
|
||||
|
||||
def test_regex_case_insensitive_option_applies_to_the_pattern() -> None:
|
||||
rule = RegexLineJoinRule(
|
||||
rule_id="regex.ignore-case",
|
||||
left_pattern=r"(?P<stem>EXAM)",
|
||||
right_pattern=r"(?P<suffix>PLE)",
|
||||
replacement=r"\g<stem>\g<suffix>",
|
||||
separator="-",
|
||||
case_sensitive=False,
|
||||
)
|
||||
|
||||
assert transform("exam-\nple", (rule,)) == "example"
|
||||
|
||||
|
||||
def test_regex_suffix_matching_considers_overlapping_starts() -> None:
|
||||
rule = RegexLineJoinRule(
|
||||
rule_id="regex.overlap",
|
||||
left_pattern=r"(?P<stem>aba)",
|
||||
right_pattern=r"(?P<suffix>x)",
|
||||
replacement=r"\g<stem>\g<suffix>",
|
||||
separator="-",
|
||||
)
|
||||
|
||||
assert transform("ababa-\nx", (rule,)) == "ababax"
|
||||
|
||||
|
||||
def test_line_end_regex_uses_right_match_only_as_a_condition() -> None:
|
||||
rule = LineEndRegexRule(
|
||||
rule_id="line-end.space",
|
||||
right_pattern=r"(?P<initial>[a-z])",
|
||||
line_break=LineBreakPolicy.SPACE,
|
||||
)
|
||||
|
||||
assert transform("foo\nbar", (rule,)) == "foo bar"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("policy", "expected"),
|
||||
[
|
||||
(LineBreakPolicy.PRESERVE, "example\n"),
|
||||
(LineBreakPolicy.DELETE, "example"),
|
||||
(LineBreakPolicy.SPACE, "example "),
|
||||
(LineBreakPolicy.PARAGRAPH, "example\n\n"),
|
||||
],
|
||||
)
|
||||
def test_all_line_break_policies_are_exact(policy: LineBreakPolicy, expected: str) -> None:
|
||||
assert transform("exam-\nple", (exact_rule(line_break=policy),)) == expected
|
||||
|
||||
|
||||
def test_paragraph_policy_restores_quote_and_list_prefixes() -> None:
|
||||
rule = exact_rule(line_break=LineBreakPolicy.PARAGRAPH)
|
||||
|
||||
assert transform("> exam-\n> ple", (rule,)) == "> example\n>\n> "
|
||||
assert transform("- exam-\n ple", (rule,)) == "- example\n\n "
|
||||
assert transform("> - exam-\n> ple", (rule,)) == "> - example\n>\n> "
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
("plain exam-\nple", "plain example"),
|
||||
("# exam-\nple", "# example"),
|
||||
("- exam-\n ple", "- example"),
|
||||
("> exam-\n> ple", "> example"),
|
||||
("> > > exam-\n> > > ple", "> > > example"),
|
||||
("1. outer\n 2. exam-\n ple", "1. outer\n 2. example"),
|
||||
],
|
||||
)
|
||||
def test_supported_markdown_blocks_and_valid_nested_containers(markdown: str, expected: str) -> None:
|
||||
assert transform(markdown, (exact_rule(),)) == expected
|
||||
|
||||
|
||||
def test_block_scope_is_explicit() -> None:
|
||||
heading_only = exact_rule(blocks=frozenset({LineJoinBlock.HEADING}))
|
||||
|
||||
assert transform("# exam-\nple", (heading_only,)) == "# example"
|
||||
assert transform("exam-\nple", (heading_only,)) == "exam-\nple"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"> > > exam-\n> > ple",
|
||||
"> #broken exam-\n> ple",
|
||||
"1. exam-\n2. ple",
|
||||
"1. outer\n 2. exam-\n ple",
|
||||
"- exam-\n ple",
|
||||
"#broken exam-\nple",
|
||||
],
|
||||
)
|
||||
def test_unknown_or_mismatched_containers_fail_closed_instead_of_becoming_paragraphs(markdown: str) -> None:
|
||||
assert transform(markdown, (exact_rule(),)) == markdown
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"```text\nexam-\nple\n```",
|
||||
" exam-\n ple",
|
||||
"> exam-\n> ple",
|
||||
"- ```\n exam-\n ple\n ```",
|
||||
"| word |\n| --- |\n| exam- |\n| ple |",
|
||||
"<table>\nexam-\nple\n</table>",
|
||||
"<table>\nexam-\nple",
|
||||
],
|
||||
)
|
||||
def test_code_and_table_regions_are_always_excluded(markdown: str) -> None:
|
||||
assert transform(markdown, (exact_rule(),)) == markdown
|
||||
|
||||
|
||||
def test_three_line_chain_is_one_proposal_and_uses_virtual_output() -> None:
|
||||
rules = (
|
||||
exact_rule(rule_id="chain.first"),
|
||||
ExactLineJoinRule(
|
||||
rule_id="chain.second",
|
||||
left="example",
|
||||
right="based",
|
||||
replacement="example-based",
|
||||
separator="-",
|
||||
),
|
||||
)
|
||||
modifier = mapped_line_join(rules)
|
||||
proposals = modifier.propose(DocumentSnapshot("exam-\nple-\nbased"))
|
||||
|
||||
assert len(proposals) == 1
|
||||
assert len(proposals[0].edits) == 1
|
||||
assert "chain.first, chain.second" in proposals[0].reason
|
||||
assert transform("exam-\nple-\nbased", rules) == "example-based"
|
||||
|
||||
|
||||
def test_mixed_line_endings_fail_the_entire_connected_chain_closed() -> None:
|
||||
rules = (
|
||||
exact_rule(rule_id="chain.first"),
|
||||
ExactLineJoinRule("chain.second", "example", "based", "example-based", separator="-"),
|
||||
)
|
||||
markdown = "exam-\r\nple-\nbased"
|
||||
|
||||
assert transform(markdown, rules) == markdown
|
||||
|
||||
|
||||
def test_multiple_independent_chains_are_reported_in_source_order() -> None:
|
||||
modifier = mapped_line_join((exact_rule(),))
|
||||
snapshot = DocumentSnapshot("exam-\nple and\n\nexam-\nple")
|
||||
proposals = modifier.propose(snapshot)
|
||||
|
||||
assert len(proposals) == 2
|
||||
assert proposals[0].edits[0].span.start < proposals[1].edits[0].span.start
|
||||
|
||||
|
||||
def test_keep_rule_can_veto_a_lower_priority_replacement() -> None:
|
||||
keep = KeepLineJoinRule("keep.example", "exam", "ple", separator="-", priority=10)
|
||||
replace = exact_rule(priority=0)
|
||||
|
||||
assert transform("exam-\nple", (replace, keep)) == "exam-\nple"
|
||||
|
||||
|
||||
def test_conflict_first_and_priority_are_deterministic() -> None:
|
||||
first = exact_rule(rule_id="first", replacement="first", priority=0)
|
||||
second = exact_rule(rule_id="second", replacement="second", priority=10)
|
||||
|
||||
assert transform(
|
||||
"exam-\nple",
|
||||
(first, second),
|
||||
conflict_policy=LineJoinConflictPolicy.FIRST,
|
||||
) == "first"
|
||||
assert transform(
|
||||
"exam-\nple",
|
||||
(first, second),
|
||||
conflict_policy=LineJoinConflictPolicy.PRIORITY,
|
||||
) == "second"
|
||||
|
||||
|
||||
def test_priority_tie_uses_declaration_order() -> None:
|
||||
first = exact_rule(rule_id="first", replacement="first")
|
||||
second = exact_rule(rule_id="second", replacement="second")
|
||||
|
||||
assert transform("exam-\nple", (second, first)) == "second"
|
||||
|
||||
|
||||
def test_conflict_error_names_rules_but_not_surrounding_text() -> None:
|
||||
modifier = mapped_line_join(
|
||||
(exact_rule(rule_id="first"), exact_rule(rule_id="second")),
|
||||
conflict_policy=LineJoinConflictPolicy.ERROR,
|
||||
)
|
||||
|
||||
with pytest.raises(ModifierContractError, match=r"first, second") as raised:
|
||||
modifier.propose(DocumentSnapshot("secret exam-\nple material"))
|
||||
assert "secret" not in str(raised.value)
|
||||
|
||||
|
||||
def test_case_insensitive_matching_is_opt_in() -> None:
|
||||
assert transform("EXAM-\nPLE", (exact_rule(),)) == "EXAM-\nPLE"
|
||||
assert transform("EXAM-\nPLE", (exact_rule(case_sensitive=False),)) == "example"
|
||||
|
||||
|
||||
def test_unicode_normalization_is_opt_in_and_preserves_source_index_mapping() -> None:
|
||||
decomposed_left = "cafe\N{COMBINING ACUTE ACCENT}"
|
||||
markdown = "CAFÉ-\nteria"
|
||||
default = ExactLineJoinRule(
|
||||
"unicode.default",
|
||||
decomposed_left,
|
||||
"teria",
|
||||
"cafeteria",
|
||||
separator="-",
|
||||
case_sensitive=False,
|
||||
)
|
||||
normalized = ExactLineJoinRule(
|
||||
"unicode.nfc",
|
||||
decomposed_left,
|
||||
"teria",
|
||||
"cafeteria",
|
||||
separator="-",
|
||||
case_sensitive=False,
|
||||
normalization=UnicodeNormalization.NFC,
|
||||
)
|
||||
|
||||
assert transform(markdown, (default,)) == markdown
|
||||
assert transform(markdown, (normalized,)) == "cafeteria"
|
||||
|
||||
|
||||
def test_unicode_word_boundaries_and_explicit_override() -> None:
|
||||
strict = exact_rule()
|
||||
permissive = exact_rule(rule_id="permissive", require_word_boundaries=False)
|
||||
markdown = "éexam-\nplemore"
|
||||
|
||||
assert transform(markdown, (strict,)) == markdown
|
||||
assert transform(markdown, (permissive,)) == "éexamplemore"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("max_blank_lines", [0, 1])
|
||||
def test_blank_line_limit_is_configurable(max_blank_lines: int) -> None:
|
||||
expected = "exam-\n\nple" if max_blank_lines == 0 else "example"
|
||||
assert transform(
|
||||
"exam-\n\nple",
|
||||
(exact_rule(),),
|
||||
max_intervening_blank_lines=max_blank_lines,
|
||||
) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("invalid", [-1, True, 2])
|
||||
def test_invalid_blank_line_limits_are_rejected(invalid: object) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
mapped_line_join((), max_intervening_blank_lines=invalid) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_empty_document_and_empty_rule_set_are_no_ops() -> None:
|
||||
modifier = mapped_line_join(())
|
||||
|
||||
assert transform("", ()) == ""
|
||||
assert transform("exam-\nple", ()) == "exam-\nple"
|
||||
assert modifier.version == "2.0.1"
|
||||
assert dict(modifier.parameters)["rules"] == ()
|
||||
|
||||
|
||||
def test_installed_package_version_matches_delivery_candidate() -> None:
|
||||
assert distribution_version("mdpolish") == "0.3.0"
|
||||
|
||||
|
||||
def test_parameters_preserve_rule_order_and_record_all_options() -> None:
|
||||
first = mapped_line_join((exact_rule(rule_id="first"), exact_rule(rule_id="second")))
|
||||
second = mapped_line_join((exact_rule(rule_id="second"), exact_rule(rule_id="first")))
|
||||
|
||||
assert first.parameters != second.parameters
|
||||
records = dict(first.parameters)["rules"]
|
||||
assert isinstance(records, tuple)
|
||||
first_record = cast(tuple[tuple[str, object], ...], records[0])
|
||||
assert ("rule_id", "first") in first_record
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"rules",
|
||||
[
|
||||
(("", "right", "word"),),
|
||||
(("left", "right", "two words"),),
|
||||
(("left", "right", "word"), ("left", "right", "other")),
|
||||
(("left", "right"),),
|
||||
(exact_rule(rule_id="same"), exact_rule(rule_id="same")),
|
||||
(exact_rule(case_sensitive=1),),
|
||||
(exact_rule(blocks={LineJoinBlock.PARAGRAPH}),),
|
||||
(RegexLineJoinRule("regex", "", r"(?P<right>x)", "x"),),
|
||||
(RegexLineJoinRule("regex", r"(x)", r"(?P<right>x)", "x"),),
|
||||
(RegexLineJoinRule("regex", r"(?P<x>x)", r"(?P<x>x)", r"\g<x>"),),
|
||||
(RegexLineJoinRule("regex", r"(?P<x>x)", r"(?P<y>x)", r"\g<missing>"),),
|
||||
(LineEndRegexRule("line-end", r"(?P<x>x)", line_break=LineBreakPolicy.PRESERVE),),
|
||||
],
|
||||
)
|
||||
def test_invalid_rules_raise_contract_error(rules: object) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
mapped_line_join(rules) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class FakeLexicon:
|
||||
package_version = "test-lexicon"
|
||||
|
||||
def __init__(self, scores: dict[str, tuple[float, float | None]]) -> None:
|
||||
self._scores = scores
|
||||
|
||||
def evidence(
|
||||
self,
|
||||
candidate: str,
|
||||
form: LexicalCandidateForm,
|
||||
left: str,
|
||||
right: str,
|
||||
) -> object:
|
||||
del form, left, right
|
||||
score = self._scores.get(candidate)
|
||||
return None if score is None else mapped_module._LexicalEvidence(*score)
|
||||
|
||||
|
||||
def lexical_rule(**overrides: object) -> LexicalLineJoinRule:
|
||||
fields: dict[str, object] = {
|
||||
"rule_id": "english.lexical",
|
||||
"left_pattern": r"[A-Za-z]+",
|
||||
"right_pattern": r"[a-z]+",
|
||||
"separator": "-",
|
||||
"backend": LexiconBackend.WORDFREQ,
|
||||
"language": "en",
|
||||
"candidate_forms": (LexicalCandidateForm.JOINED, LexicalCandidateForm.HYPHENATED),
|
||||
"minimum_score": 1.0,
|
||||
"minimum_score_margin": 0.5,
|
||||
}
|
||||
fields.update(overrides)
|
||||
return LexicalLineJoinRule(**fields) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def install_fake_lexicon(monkeypatch: MonkeyPatch, scores: dict[str, tuple[float, float | None]]) -> None:
|
||||
def build(rule: LexicalLineJoinRule) -> FakeLexicon:
|
||||
del rule
|
||||
return FakeLexicon(scores)
|
||||
|
||||
monkeypatch.setattr(mapped_module, "_build_lexicon", build)
|
||||
|
||||
|
||||
def test_lexical_rule_joins_a_unique_dictionary_candidate_without_per_word_mapping(
|
||||
monkeypatch: MonkeyPatch,
|
||||
) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"example": (5.0, 5.0)})
|
||||
|
||||
assert transform("exam-\nple", (lexical_rule(),)) == "example"
|
||||
|
||||
|
||||
def test_lexical_rule_can_select_the_natural_hyphenated_form_by_score(monkeypatch: MonkeyPatch) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"rulebased": (3.0, 3.0), "rule-based": (6.0, 6.0)})
|
||||
|
||||
assert transform("rule-\nbased", (lexical_rule(),)) == "rule-based"
|
||||
|
||||
|
||||
def test_lexical_margin_and_keep_policy_preserve_an_ambiguous_boundary(monkeypatch: MonkeyPatch) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"rulebased": (5.0, 5.0), "rule-based": (4.8, 4.8)})
|
||||
|
||||
assert transform("rule-\nbased", (lexical_rule(minimum_score_margin=0.5),)) == "rule-\nbased"
|
||||
|
||||
|
||||
def test_spellchecker_style_spaced_proxy_is_not_ranked_against_single_token(monkeypatch: MonkeyPatch) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"inside": (6.0, 6.0), "in side": (5.0, None)})
|
||||
rule = lexical_rule(
|
||||
separator="",
|
||||
candidate_forms=(LexicalCandidateForm.JOINED, LexicalCandidateForm.SPACED),
|
||||
)
|
||||
|
||||
assert transform("in\nside", (rule,)) == "in\nside"
|
||||
|
||||
|
||||
def test_lexical_ambiguity_error_is_source_safe(monkeypatch: MonkeyPatch) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"inside": (6.0, 6.0), "in side": (5.0, None)})
|
||||
rule = lexical_rule(
|
||||
separator="",
|
||||
candidate_forms=(LexicalCandidateForm.JOINED, LexicalCandidateForm.SPACED),
|
||||
ambiguity=LexicalAmbiguityPolicy.ERROR,
|
||||
)
|
||||
modifier = mapped_line_join((rule,))
|
||||
|
||||
with pytest.raises(ModifierContractError, match=r"english\.lexical") as raised:
|
||||
modifier.propose(DocumentSnapshot("secret in\nside"))
|
||||
assert "secret" not in str(raised.value)
|
||||
|
||||
|
||||
class FakeHyphenator:
|
||||
def __init__(self, positions: tuple[int, ...]) -> None:
|
||||
self._positions = positions
|
||||
|
||||
def positions(self, word: str) -> tuple[int, ...]:
|
||||
del word
|
||||
return self._positions
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("positions", "expected"), [((4,), "example"), ((), "exam-ple")])
|
||||
def test_pyphen_gate_only_controls_the_joined_candidate(
|
||||
monkeypatch: MonkeyPatch,
|
||||
positions: tuple[int, ...],
|
||||
expected: str,
|
||||
) -> None:
|
||||
install_fake_lexicon(monkeypatch, {"example": (6.0, 6.0), "exam-ple": (4.0, 4.0)})
|
||||
monkeypatch.setattr(
|
||||
mapped_module,
|
||||
"_build_hyphenator",
|
||||
lambda language: (FakeHyphenator(positions), f"test-{language}"),
|
||||
)
|
||||
rule = lexical_rule(hyphenation_language="en_US", minimum_score_margin=0.5)
|
||||
|
||||
assert transform("exam-\nple", (rule,)) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"overrides",
|
||||
[
|
||||
{"separator": "/"},
|
||||
{"candidate_forms": (LexicalCandidateForm.JOINED,)},
|
||||
{"candidate_forms": (LexicalCandidateForm.JOINED, LexicalCandidateForm.JOINED)},
|
||||
{"minimum_score": -1.0},
|
||||
{"minimum_score": float("nan")},
|
||||
{"minimum_score_margin": True},
|
||||
{"language": ""},
|
||||
{"hyphenation_language": ""},
|
||||
],
|
||||
)
|
||||
def test_invalid_lexical_rules_fail_before_loading_a_backend(overrides: dict[str, object]) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
mapped_line_join((lexical_rule(**overrides),))
|
||||
|
||||
|
||||
def test_missing_requested_optional_backend_raises_without_fallback(monkeypatch: MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(mapped_module, "_zipf_frequency", None)
|
||||
|
||||
with pytest.raises(ModifierContractError, match="frequency"):
|
||||
mapped_line_join((lexical_rule(),))
|
||||
|
||||
|
||||
def test_spellchecker_case_sensitive_dictionary_is_rejected_before_backend_loading(
|
||||
monkeypatch: MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(mapped_module, "_SpellChecker", None)
|
||||
rule = lexical_rule(backend=LexiconBackend.SPELLCHECKER)
|
||||
|
||||
with pytest.raises(ModifierContractError, match=r"english\.lexical") as raised:
|
||||
mapped_line_join((rule,))
|
||||
|
||||
message = str(raised.value)
|
||||
assert "pyspellchecker" in message
|
||||
assert "case_sensitive=False" in message
|
||||
assert "lexical' extra" not in message
|
||||
|
||||
|
||||
def test_real_spellchecker_and_pyphen_transform_and_record_versions() -> None:
|
||||
if mapped_module._SpellChecker is None or mapped_module._Pyphen is None:
|
||||
pytest.skip("optional lexical backend is not installed")
|
||||
rule = lexical_rule(
|
||||
backend=LexiconBackend.SPELLCHECKER,
|
||||
case_sensitive=False,
|
||||
hyphenation_language="en_US",
|
||||
)
|
||||
modifier = mapped_line_join((rule,))
|
||||
result = Pipeline((modifier,)).transform("an exam-\nple")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "an example"
|
||||
records = cast(tuple[tuple[tuple[str, object], ...], ...], dict(modifier.parameters)["rules"])
|
||||
package_versions = dict(cast(tuple[tuple[str, str], ...], dict(records[0])["package_versions"]))
|
||||
assert package_versions == {
|
||||
"pyphen": distribution_version("pyphen"),
|
||||
"pyspellchecker": distribution_version("pyspellchecker"),
|
||||
}
|
||||
|
||||
|
||||
def test_real_pyphen_invalid_breakpoint_keeps_joined_candidate_out() -> None:
|
||||
if mapped_module._SpellChecker is None or mapped_module._Pyphen is None:
|
||||
pytest.skip("optional lexical backend is not installed")
|
||||
rule = lexical_rule(
|
||||
backend=LexiconBackend.SPELLCHECKER,
|
||||
case_sensitive=False,
|
||||
hyphenation_language="en_US",
|
||||
)
|
||||
|
||||
assert transform("exa-\nmple", (rule,)) == "exa-\nmple"
|
||||
|
||||
|
||||
def test_real_wordfreq_transforms_and_records_version() -> None:
|
||||
if mapped_module._zipf_frequency is None:
|
||||
pytest.skip("optional frequency backend is not installed")
|
||||
modifier = mapped_line_join((lexical_rule(),))
|
||||
result = Pipeline((modifier,)).transform("an exam-\nple")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "an example"
|
||||
records = cast(tuple[tuple[tuple[str, object], ...], ...], dict(modifier.parameters)["rules"])
|
||||
package_versions = dict(cast(tuple[tuple[str, str], ...], dict(records[0])["package_versions"]))
|
||||
assert package_versions == {"wordfreq": distribution_version("wordfreq")}
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
rules = (exact_rule(),)
|
||||
pipeline = Pipeline((mapped_line_join(rules),))
|
||||
first = pipeline.transform("an exam-\n\nple here")
|
||||
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -95,13 +95,13 @@ def test_proposal_requires_reason_edits_and_one_matching_digest() -> None:
|
||||
|
||||
def test_transform_result_enforces_status_specific_output_fields() -> None:
|
||||
snapshot = DocumentSnapshot("abc")
|
||||
error = RunError("component", "1.0.0", 0, ErrorStage.TRANSFORM, "ExampleError", "safe")
|
||||
error = RunError("modifier", "1.0.0", 0, ErrorStage.TRANSFORM, "ExampleError", "safe")
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), "a", "A")
|
||||
proposal = ProposedChange(snapshot.sha256, "reason", (edit,))
|
||||
residual = ResidualProposal(
|
||||
component_id="component",
|
||||
component_version="1.0.0",
|
||||
component_position=0,
|
||||
modifier_id="modifier",
|
||||
modifier_version="1.0.0",
|
||||
modifier_position=0,
|
||||
proposal_ref=ProposalReference(0, snapshot.sha256, 0),
|
||||
proposal=proposal,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import DocumentSnapshot, Modifier, ModifierContractError, ProposedChange
|
||||
|
||||
|
||||
def no_changes(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return ()
|
||||
|
||||
|
||||
def test_plain_function_and_mapping_parameters_create_immutable_modifier() -> None:
|
||||
source_parameters: dict[str, object] = {
|
||||
"enabled": True,
|
||||
"nested": {"count": 2},
|
||||
"values": ["a", "b"],
|
||||
}
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.2.3",
|
||||
parameters=source_parameters,
|
||||
applicability="处理虚构测试输入；排除其他文本。",
|
||||
propose=no_changes,
|
||||
)
|
||||
source_parameters["enabled"] = False
|
||||
|
||||
assert modifier.propose is no_changes
|
||||
assert modifier.parameters == (
|
||||
("enabled", True),
|
||||
("nested", (("count", 2),)),
|
||||
("values", ("a", "b")),
|
||||
)
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
modifier.version = "2.0.0" # type: ignore[misc]
|
||||
|
||||
|
||||
def test_normalized_parameters_are_sorted_and_validated() -> None:
|
||||
modifier = Modifier(
|
||||
modifier_id="example.normalized",
|
||||
version="1.0.0",
|
||||
parameters=(("z", 1), ("a", ("x", 2))),
|
||||
applicability="处理虚构测试输入。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
assert modifier.parameters == (("a", ("x", 2)), ("z", 1))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("modifier_id", ["", "Upper", "has space", ".leading", "trailing."])
|
||||
def test_invalid_modifier_id_is_rejected(modifier_id: str) -> None:
|
||||
with pytest.raises(ModifierContractError, match="modifier_id"):
|
||||
Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", ["", "1", "1.0", "v1.0.0", "01.0.0", "1.0.0-beta"])
|
||||
def test_invalid_version_is_rejected(version: str) -> None:
|
||||
with pytest.raises(ModifierContractError, match="version"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version=version,
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"parameters",
|
||||
[
|
||||
{"bad": {1}},
|
||||
{"bad": float("inf")},
|
||||
(("duplicate", 1), ("duplicate", 2)),
|
||||
(("bad", object()),),
|
||||
("not-a-pair",),
|
||||
],
|
||||
)
|
||||
def test_invalid_parameters_are_rejected(parameters: object) -> None:
|
||||
with pytest.raises(ModifierContractError):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=parameters, # type: ignore[arg-type]
|
||||
applicability="测试。",
|
||||
propose=no_changes,
|
||||
)
|
||||
|
||||
|
||||
def test_empty_applicability_and_non_callable_proposal_are_rejected() -> None:
|
||||
with pytest.raises(ModifierContractError, match="applicability"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability=" ",
|
||||
propose=no_changes,
|
||||
)
|
||||
with pytest.raises(ModifierContractError, match="callable"):
|
||||
Modifier(
|
||||
modifier_id="example.rule",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="测试。",
|
||||
propose=cast(object, None), # type: ignore[arg-type]
|
||||
)
|
||||
@@ -1,83 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.component import ComponentContractError
|
||||
from mdpolish.components import PageBreakWordJoinComponent
|
||||
|
||||
MAPPINGS = (
|
||||
("possi-", "bly", "possibly"),
|
||||
("SOFA-", "based", "SOFA-based"),
|
||||
("threshold.", "olds", "thresholds"),
|
||||
)
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([PageBreakWordJoinComponent(MAPPINGS)]).transform(markdown)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("markdown", "expected"),
|
||||
[
|
||||
("except possi-\nbly through care", "except possibly through care"),
|
||||
("except possi-\n\nbly through care", "except possibly through care"),
|
||||
("use SOFA-\nbased criteria", "use SOFA-based criteria"),
|
||||
("at threshold.\n\nolds of eight", "at thresholds of eight"),
|
||||
("except possi-\r\n\r\nbly now", "except possibly now"),
|
||||
("except possi-\r\rbly now", "except possibly now"),
|
||||
],
|
||||
)
|
||||
def test_applies_exact_mapping_across_approved_line_shapes(markdown: str, expected: str) -> None:
|
||||
result = transform(markdown)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == expected
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"except unknown-\nword here",
|
||||
"except possi-\n\n\nbly here",
|
||||
"except possi-\r\n\nbly here",
|
||||
"except POSSI-\nbly here",
|
||||
"except possi-\nblymore here",
|
||||
"except xSOFA-\nbased here",
|
||||
],
|
||||
)
|
||||
def test_unknown_or_unsafe_boundaries_are_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_parameters_and_results_are_independent_of_mapping_order() -> None:
|
||||
first = PageBreakWordJoinComponent(MAPPINGS)
|
||||
second = PageBreakWordJoinComponent(reversed(MAPPINGS))
|
||||
markdown = "possibly becomes possi-\nbly"
|
||||
|
||||
first_result = Pipeline([first]).transform(markdown)
|
||||
second_result = Pipeline([second]).transform(markdown)
|
||||
|
||||
assert first_result.components == second_result.components
|
||||
assert first_result.output_markdown == second_result.output_markdown
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mappings",
|
||||
[
|
||||
(("", "right", "word"),),
|
||||
(("left", "right", "two words"),),
|
||||
(("left", "right", "word"), ("left", "right", "other")),
|
||||
(("left", "right"),),
|
||||
],
|
||||
)
|
||||
def test_invalid_mappings_raise_contract_error(mappings: object) -> None:
|
||||
with pytest.raises(ComponentContractError):
|
||||
PageBreakWordJoinComponent(mappings) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([PageBreakWordJoinComponent(MAPPINGS)])
|
||||
first = pipeline.transform("except possi-\n\nbly here")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
+117
-114
@@ -1,14 +1,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import (
|
||||
Component,
|
||||
DocumentSnapshot,
|
||||
ErrorStage,
|
||||
Modifier,
|
||||
Pipeline,
|
||||
ProposedChange,
|
||||
RunStatus,
|
||||
@@ -17,97 +16,84 @@ from mdpolish import (
|
||||
)
|
||||
|
||||
|
||||
class ReplaceComponent(Component):
|
||||
def __init__(
|
||||
self,
|
||||
def replace_modifier(
|
||||
needle: str,
|
||||
replacement: str,
|
||||
*,
|
||||
component_id: str,
|
||||
modifier_id: str,
|
||||
version: str = "1.0.0",
|
||||
parameters: object = None,
|
||||
applicability: str = "处理精确测试字符串，要求完整匹配，排除其他内容。",
|
||||
) -> None:
|
||||
self.needle = needle
|
||||
self.replacement = replacement
|
||||
self._component_id = component_id
|
||||
self._version = version
|
||||
self._parameters = {"needle": needle, "replacement": replacement} if parameters is None else parameters
|
||||
self._applicability = applicability
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return self._component_id
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return self._version
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return cast(Mapping[str, object], self._parameters)
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return self._applicability
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(self.needle)
|
||||
) -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(needle)
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(position, position + len(self.needle)),
|
||||
expected_text=self.needle,
|
||||
replacement=self.replacement,
|
||||
span=TextSpan(position, position + len(needle)),
|
||||
expected_text=needle,
|
||||
replacement=replacement,
|
||||
)
|
||||
return (
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason=f"replace test token for {self.component_id}",
|
||||
reason=f"replace test token for {modifier_id}",
|
||||
edits=(edit,),
|
||||
),
|
||||
)
|
||||
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version=version,
|
||||
parameters={"needle": needle, "replacement": replacement},
|
||||
applicability="处理精确测试字符串；排除其他内容。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
class ExplodingComponent(ReplaceComponent):
|
||||
def __init__(self, *, trigger: str | None = None, component_id: str = "test.exploding") -> None:
|
||||
super().__init__("unused", "unused-replacement", component_id=component_id)
|
||||
self.trigger = trigger
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if self.trigger is None or snapshot.markdown == self.trigger:
|
||||
def exploding_modifier(*, trigger: str | None = None, modifier_id: str = "test.exploding") -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
if trigger is None or snapshot.markdown == trigger:
|
||||
raise RuntimeError(f"SECRET source: {snapshot.markdown}")
|
||||
return ()
|
||||
|
||||
|
||||
class InvalidReturnComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.invalid-return")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
return Modifier(
|
||||
modifier_id=modifier_id,
|
||||
version="1.0.0",
|
||||
parameters={"trigger": trigger},
|
||||
applicability="仅用于测试异常路径。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
|
||||
class StaleProposalComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.stale")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def stale_proposal_modifier() -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
stale_snapshot = DocumentSnapshot(snapshot.markdown + "!")
|
||||
edit = TextEdit(stale_snapshot.sha256, TextSpan(0, 1), stale_snapshot.markdown[0], "X")
|
||||
return (ProposedChange(stale_snapshot.sha256, "stale test proposal", (edit,)),)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="test.stale",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试过期快照。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
class OverlapComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("a", "A", component_id="test.overlap")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
def overlap_modifier() -> Modifier:
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
first = TextEdit(snapshot.sha256, TextSpan(0, 3), snapshot.markdown[0:3], "X")
|
||||
second = TextEdit(snapshot.sha256, TextSpan(2, 4), snapshot.markdown[2:4], "Y")
|
||||
return (ProposedChange(snapshot.sha256, "overlapping test proposal", (first, second)),)
|
||||
|
||||
return Modifier(
|
||||
modifier_id="test.overlap",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试冲突范围。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
|
||||
def test_empty_pipeline_returns_unchanged_success_for_empty_unicode_text() -> None:
|
||||
for markdown in ("", "中文\nCafe\u0301\n🙂"):
|
||||
@@ -117,14 +103,14 @@ def test_empty_pipeline_returns_unchanged_success_for_empty_unicode_text() -> No
|
||||
assert result.output_markdown == markdown
|
||||
assert result.partial_markdown is None
|
||||
assert result.changes == ()
|
||||
assert result.components == ()
|
||||
assert result.modifiers == ()
|
||||
|
||||
|
||||
def test_later_component_reads_snapshot_produced_by_earlier_component() -> None:
|
||||
def test_later_modifier_reads_snapshot_produced_by_earlier_modifier() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("初", "中", component_id="test.first"),
|
||||
ReplaceComponent("中", "终", component_id="test.second"),
|
||||
replace_modifier("初", "中", modifier_id="test.first"),
|
||||
replace_modifier("中", "终", modifier_id="test.second"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -132,15 +118,15 @@ def test_later_component_reads_snapshot_produced_by_earlier_component() -> None:
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "终"
|
||||
assert [change.component_id for change in result.changes] == ["test.first", "test.second"]
|
||||
assert [change.modifier_id for change in result.changes] == ["test.first", "test.second"]
|
||||
assert result.changes[1].before_sha256 == result.changes[0].after_sha256
|
||||
|
||||
|
||||
def test_same_input_components_and_parameters_produce_same_ordered_result() -> None:
|
||||
def test_same_input_modifiers_and_parameters_produce_same_ordered_result() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.first"),
|
||||
ReplaceComponent("b", "c", component_id="test.second"),
|
||||
replace_modifier("a", "b", modifier_id="test.first"),
|
||||
replace_modifier("b", "c", modifier_id="test.second"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -148,7 +134,7 @@ def test_same_input_components_and_parameters_produce_same_ordered_result() -> N
|
||||
|
||||
|
||||
def test_successful_pipeline_is_idempotent_on_its_output() -> None:
|
||||
pipeline = Pipeline([ReplaceComponent("old", "new", component_id="test.replace")])
|
||||
pipeline = Pipeline([replace_modifier("old", "new", modifier_id="test.replace")])
|
||||
|
||||
first = pipeline.transform("old value")
|
||||
assert first.status is RunStatus.SUCCESS
|
||||
@@ -161,22 +147,12 @@ def test_successful_pipeline_is_idempotent_on_its_output() -> None:
|
||||
assert second.changes == ()
|
||||
|
||||
|
||||
def test_single_component_runs_through_pipeline_without_shortcut() -> None:
|
||||
component = ReplaceComponent("a", "A", component_id="test.single")
|
||||
|
||||
result = Pipeline([component]).transform("a")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "A"
|
||||
assert len(result.changes) == 1
|
||||
|
||||
|
||||
def test_transform_error_stops_later_components_and_keeps_only_partial_text() -> None:
|
||||
def test_transform_error_stops_later_modifiers_and_keeps_only_partial_text() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.first"),
|
||||
ExplodingComponent(),
|
||||
ReplaceComponent("b", "c", component_id="test.never-runs"),
|
||||
replace_modifier("a", "b", modifier_id="test.first"),
|
||||
exploding_modifier(),
|
||||
replace_modifier("b", "c", modifier_id="test.never-runs"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -185,14 +161,14 @@ def test_transform_error_stops_later_components_and_keeps_only_partial_text() ->
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.output_markdown is None
|
||||
assert result.partial_markdown == "b"
|
||||
assert [change.component_id for change in result.changes] == ["test.first"]
|
||||
assert [change.modifier_id for change in result.changes] == ["test.first"]
|
||||
assert len(result.errors) == 1
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
assert result.residual_proposals == ()
|
||||
|
||||
|
||||
def test_unexpected_component_error_does_not_leak_source_or_exception_message() -> None:
|
||||
result = Pipeline([ExplodingComponent()]).transform("private markdown")
|
||||
def test_unexpected_modifier_error_does_not_leak_source_or_exception_message() -> None:
|
||||
result = Pipeline([exploding_modifier()]).transform("private markdown")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.errors[0].error_type == "RuntimeError"
|
||||
@@ -201,17 +177,27 @@ def test_unexpected_component_error_does_not_leak_source_or_exception_message()
|
||||
|
||||
|
||||
def test_invalid_proposal_return_is_a_transform_contract_failure() -> None:
|
||||
result = Pipeline([InvalidReturnComponent()]).transform("abc")
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
return cast(tuple[ProposedChange, ...], [])
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="test.invalid-return",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试非法返回值。",
|
||||
propose=propose,
|
||||
)
|
||||
result = Pipeline([modifier]).transform("abc")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "abc"
|
||||
assert result.errors[0].error_type == "ComponentContractError"
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
|
||||
|
||||
@pytest.mark.parametrize("component", [StaleProposalComponent(), OverlapComponent()])
|
||||
def test_invalid_edit_batch_fails_atomically(component: Component) -> None:
|
||||
result = Pipeline([component]).transform("abcd")
|
||||
@pytest.mark.parametrize("modifier", [stale_proposal_modifier(), overlap_modifier()])
|
||||
def test_invalid_edit_batch_fails_atomically(modifier: Modifier) -> None:
|
||||
result = Pipeline([modifier]).transform("abcd")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "abcd"
|
||||
@@ -219,11 +205,11 @@ def test_invalid_edit_batch_fails_atomically(component: Component) -> None:
|
||||
assert result.errors[0].error_type == "EditValidationError"
|
||||
|
||||
|
||||
def test_duplicate_component_ids_fail_during_preflight_before_modification() -> None:
|
||||
def test_duplicate_modifier_ids_fail_during_preflight_before_modification() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.duplicate"),
|
||||
ReplaceComponent("b", "c", component_id="test.duplicate"),
|
||||
replace_modifier("a", "b", modifier_id="test.duplicate"),
|
||||
replace_modifier("b", "c", modifier_id="test.duplicate"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -235,28 +221,45 @@ def test_duplicate_component_ids_fail_during_preflight_before_modification() ->
|
||||
assert result.errors[0].error_type == "PipelineContractError"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"component",
|
||||
[
|
||||
ReplaceComponent("a", "b", component_id="test.bad-version", version="1.0"),
|
||||
ReplaceComponent("a", "b", component_id="test.bad-parameters", parameters={"bad": {1}}),
|
||||
ReplaceComponent("a", "b", component_id="test.bad-applicability", applicability=""),
|
||||
],
|
||||
)
|
||||
def test_invalid_component_metadata_fails_before_modification(component: Component) -> None:
|
||||
result = Pipeline([component]).transform("a")
|
||||
def test_non_modifier_entry_fails_during_preflight() -> None:
|
||||
pipeline = Pipeline(cast(list[Modifier], [object()]))
|
||||
|
||||
result = pipeline.transform("a")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
|
||||
|
||||
def test_metadata_changed_by_proposal_function_fails_before_application() -> None:
|
||||
modifier: Modifier
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
object.__setattr__(modifier, "version", "2.0.0")
|
||||
edit = TextEdit(snapshot.sha256, TextSpan(0, 1), "a", "A")
|
||||
return (ProposedChange(snapshot.sha256, "mutating metadata", (edit,)),)
|
||||
|
||||
modifier = Modifier(
|
||||
modifier_id="test.mutating-metadata",
|
||||
version="1.0.0",
|
||||
parameters=(),
|
||||
applicability="仅用于测试运行期元数据变化。",
|
||||
propose=propose,
|
||||
)
|
||||
|
||||
result = Pipeline([modifier]).transform("a")
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.changes == ()
|
||||
assert result.errors[0].stage is ErrorStage.TRANSFORM
|
||||
assert result.errors[0].error_type == "ModifierContractError"
|
||||
|
||||
|
||||
def test_cross_component_chain_is_reported_unstable_without_a_second_round() -> None:
|
||||
def test_cross_modifier_chain_is_reported_unstable_without_a_second_round() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("bad", "good", component_id="test.to-good"),
|
||||
ReplaceComponent("good", "bad", component_id="test.to-bad"),
|
||||
replace_modifier("bad", "good", modifier_id="test.to-good"),
|
||||
replace_modifier("good", "bad", modifier_id="test.to-bad"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -267,16 +270,16 @@ def test_cross_component_chain_is_reported_unstable_without_a_second_round() ->
|
||||
assert result.partial_markdown == "bad"
|
||||
assert len(result.changes) == 2
|
||||
assert len(result.residual_proposals) == 1
|
||||
assert result.residual_proposals[0].component_id == "test.to-good"
|
||||
assert result.residual_proposals[0].modifier_id == "test.to-good"
|
||||
assert result.errors == ()
|
||||
|
||||
|
||||
def test_final_review_continues_after_error_and_keeps_valid_residual_proposal() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ExplodingComponent(trigger="done", component_id="test.review-error"),
|
||||
ReplaceComponent("done", "clean", component_id="test.residual"),
|
||||
ReplaceComponent("start", "done", component_id="test.producer"),
|
||||
exploding_modifier(trigger="done", modifier_id="test.review-error"),
|
||||
replace_modifier("done", "clean", modifier_id="test.residual"),
|
||||
replace_modifier("start", "done", modifier_id="test.producer"),
|
||||
]
|
||||
)
|
||||
|
||||
@@ -287,7 +290,7 @@ def test_final_review_continues_after_error_and_keeps_valid_residual_proposal()
|
||||
assert result.partial_markdown == "done"
|
||||
assert len(result.errors) == 1
|
||||
assert result.errors[0].stage is ErrorStage.FINAL_REVIEW
|
||||
assert result.errors[0].component_id == "test.review-error"
|
||||
assert result.errors[0].modifier_id == "test.review-error"
|
||||
assert len(result.residual_proposals) == 1
|
||||
assert result.residual_proposals[0].component_id == "test.residual"
|
||||
assert result.residual_proposals[0].modifier_id == "test.residual"
|
||||
assert result.residual_proposals[0].proposal_ref.snapshot_sha256 == result.current_sha256
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import ReferenceSpacingComponent
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([ReferenceSpacingComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_normalizes_missing_and_extra_blank_lines_in_references_only() -> None:
|
||||
markdown = "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n2. Second\n\n\n3. Third"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "1. Method\n2. Method\n\n## REFERENCES\n\n1. First\n\n2. Second\n\n3. Third"
|
||||
assert len(result.changes) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("heading", ["# References", "## REFERENCES", "### references", "#### ReFeReNcEs"])
|
||||
def test_accepts_exact_references_heading_with_ascii_case_folding(heading: str) -> None:
|
||||
markdown = f"{heading}\n\n1. First\n2. Second"
|
||||
assert transform(markdown).output_markdown == f"{heading}\n\n1. First\n\n2. Second"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||
markdown = line_ending.join(("## References", "", "1. First", "2. Second"))
|
||||
expected = line_ending.join(("## References", "", "1. First", "", "2. Second"))
|
||||
assert transform(markdown).output_markdown == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
"## Reference\n\n1. First\n2. Second",
|
||||
"## References\n\n1. First\n3. Third",
|
||||
"## References\n\n2. Second\n3. Third",
|
||||
"## References\n\n1. First\n### Subsection\n2. Second",
|
||||
"## References\r\n\r\n1. First\r\n\n2. Second",
|
||||
],
|
||||
)
|
||||
def test_ambiguous_or_mixed_sections_are_preserved(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_same_or_higher_heading_ends_section() -> None:
|
||||
markdown = "## References\n\n1. First\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "## References\n\n1. First\n\n2. Second\n\n## Appendix\n\n1. Keep\n2. Keep"
|
||||
|
||||
|
||||
def test_multiline_reference_uses_its_last_text_line_as_boundary() -> None:
|
||||
markdown = "## References\n\n1. First line\ncontinuation\n2. Second"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "## References\n\n1. First line\ncontinuation\n\n2. Second"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([ReferenceSpacingComponent()])
|
||||
first = pipeline.transform("## References\n\n1. First\n2. Second")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -0,0 +1,108 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Modifier, ModifierContractError, Pipeline, RunStatus, regex_replace
|
||||
|
||||
|
||||
def test_factory_returns_modifier_and_expands_capture_groups() -> None:
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.swap-date",
|
||||
version="1.0.0",
|
||||
pattern=r"(\d{4})-(\d{2})-(\d{2})",
|
||||
replacement=r"\3/\2/\1",
|
||||
applicability="处理 ISO 形状的虚构日期；不验证真实日历日期。",
|
||||
)
|
||||
|
||||
result = Pipeline([modifier]).transform("A 2026-08-26, B 2027-01-02")
|
||||
|
||||
assert isinstance(modifier, Modifier)
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "A 26/08/2026, B 02/01/2027"
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].before == "2026-08-26"
|
||||
assert result.changes[0].after == "26/08/2026"
|
||||
|
||||
|
||||
def test_flags_and_factory_parameters_are_recorded() -> None:
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.case",
|
||||
version="2.0.0",
|
||||
pattern="token",
|
||||
replacement="value",
|
||||
flags=re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
|
||||
result = Pipeline([modifier]).transform("TOKEN")
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.modifiers[0].modifier_id == modifier.modifier_id
|
||||
assert result.modifiers[0].parameters == modifier.parameters
|
||||
assert dict(modifier.parameters) == {
|
||||
"flags": int(re.IGNORECASE | re.MULTILINE),
|
||||
"pattern": "token",
|
||||
"replacement": "value",
|
||||
}
|
||||
|
||||
|
||||
def test_zero_match_and_no_op_replacement_are_stable() -> None:
|
||||
missing = Pipeline(
|
||||
[regex_replace(modifier_id="example.missing", version="1.0.0", pattern="missing", replacement="new")]
|
||||
).transform("text")
|
||||
no_op = Pipeline(
|
||||
[regex_replace(modifier_id="example.no-op", version="1.0.0", pattern="text", replacement="text")]
|
||||
).transform("text")
|
||||
|
||||
assert missing.output_markdown == "text"
|
||||
assert missing.changes == ()
|
||||
assert no_op.status is RunStatus.SUCCESS
|
||||
assert no_op.changes == ()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("pattern", [r"", r"^", r"a*", r"(?=a)"])
|
||||
def test_zero_length_matches_are_rejected(pattern: str) -> None:
|
||||
if pattern == r"(?=a)":
|
||||
modifier = regex_replace(
|
||||
modifier_id="example.zero",
|
||||
version="1.0.0",
|
||||
pattern=pattern,
|
||||
replacement="x",
|
||||
)
|
||||
result = Pipeline([modifier]).transform("a")
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert result.partial_markdown == "a"
|
||||
assert result.changes == ()
|
||||
return
|
||||
|
||||
with pytest.raises(ModifierContractError, match="zero-length"):
|
||||
regex_replace(
|
||||
modifier_id="example.zero",
|
||||
version="1.0.0",
|
||||
pattern=pattern,
|
||||
replacement="x",
|
||||
)
|
||||
|
||||
|
||||
def test_invalid_pattern_flags_and_input_types_are_rejected() -> None:
|
||||
with pytest.raises(ModifierContractError, match="compile"):
|
||||
regex_replace(modifier_id="example.invalid", version="1.0.0", pattern="(", replacement="x")
|
||||
with pytest.raises(ModifierContractError, match="flags"):
|
||||
regex_replace(
|
||||
modifier_id="example.flags",
|
||||
version="1.0.0",
|
||||
pattern="x",
|
||||
replacement="y",
|
||||
flags=True,
|
||||
)
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline(
|
||||
[regex_replace(modifier_id="example.space", version="1.0.0", pattern=r" {2,}", replacement=" ")]
|
||||
)
|
||||
first = pipeline.transform("a b")
|
||||
assert first.output_markdown == "a b"
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -1,64 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import RepeatedRunningHeaderComponent
|
||||
|
||||
HEADER = "## Repeated Paper Header"
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([RepeatedRunningHeaderComponent()]).transform(markdown)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_bridges_interrupted_sentence_and_deletes_other_occurrence(line_ending: str) -> None:
|
||||
markdown = line_ending.join(
|
||||
(
|
||||
"Sentence continues in",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"the next line.",
|
||||
"",
|
||||
"18. Reference",
|
||||
"",
|
||||
HEADER,
|
||||
"",
|
||||
"19. Reference",
|
||||
)
|
||||
)
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == line_ending.join(
|
||||
("Sentence continues in the next line.", "", "18. Reference", "", "19. Reference")
|
||||
)
|
||||
assert len(result.changes) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"markdown",
|
||||
[
|
||||
f"before\n\n{HEADER}\n\nafter",
|
||||
f"Sentence ends.\n\n{HEADER}\n\nAfter\n\nAnother sentence.\n\n{HEADER}\n\nOther",
|
||||
f"Sentence continues\n\n{HEADER}\n\nAfter\n\nText ends.\n\n{HEADER}\n\nOther",
|
||||
f"Sentence continues\n\n{HEADER}\n\nafter\n{HEADER}\nnot blank",
|
||||
],
|
||||
)
|
||||
def test_missing_or_unsafe_group_evidence_preserves_document(markdown: str) -> None:
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_header_matching_is_exact_and_not_keyword_based() -> None:
|
||||
markdown = "continues\n\n## Any Header\n\nfrom here\n\n18. Ref\n\n## Any Header\n\n19. Ref"
|
||||
result = transform(markdown)
|
||||
assert result.output_markdown == "continues from here\n\n18. Ref\n\n19. Ref"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([RepeatedRunningHeaderComponent()])
|
||||
first = pipeline.transform(f"continues\n\n{HEADER}\n\nfrom here\n\ntext\n\n{HEADER}\n\nend")
|
||||
assert first.output_markdown is not None
|
||||
assert pipeline.transform(first.output_markdown).changes == ()
|
||||
@@ -1,177 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Component, DocumentSnapshot, Pipeline, ProposedChange, RunStatus, TextEdit, TextSpan
|
||||
from mdpolish.reporting import ReportingError, build_document_report, build_unified_diff
|
||||
|
||||
|
||||
class ReplaceComponent(Component):
|
||||
def __init__(self, needle: str, replacement: str, component_id: str) -> None:
|
||||
self.needle = needle
|
||||
self.replacement = replacement
|
||||
self._component_id = component_id
|
||||
|
||||
@property
|
||||
def component_id(self) -> str:
|
||||
return self._component_id
|
||||
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
|
||||
@property
|
||||
def parameters(self) -> Mapping[str, object]:
|
||||
return {"needle": self.needle, "replacement": self.replacement}
|
||||
|
||||
@property
|
||||
def applicability(self) -> str:
|
||||
return "处理精确测试标记，要求完整匹配，排除其他文本。"
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
position = snapshot.markdown.find(self.needle)
|
||||
if position < 0:
|
||||
return ()
|
||||
edit = TextEdit(
|
||||
snapshot.sha256,
|
||||
TextSpan(position, position + len(self.needle)),
|
||||
self.needle,
|
||||
self.replacement,
|
||||
)
|
||||
return (ProposedChange(snapshot.sha256, f"replace {self.needle}", (edit,)),)
|
||||
|
||||
|
||||
class ExplodingComponent(ReplaceComponent):
|
||||
def __init__(self) -> None:
|
||||
super().__init__("unused", "unused-replacement", "test.exploding")
|
||||
|
||||
def _propose_changes(self, snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
raise RuntimeError(f"private: {snapshot.markdown}")
|
||||
|
||||
|
||||
def test_success_report_replays_multiple_component_snapshots_and_derives_locations() -> None:
|
||||
markdown = "x\r\ntarget\n"
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("x", "XX", "test.expand"),
|
||||
ReplaceComponent("target", "done", "test.target"),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform(markdown)
|
||||
|
||||
report = build_document_report(
|
||||
document_id="paper",
|
||||
source_label="data/md/paper.md",
|
||||
input_markdown=markdown,
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert report.cleaned_markdown == b"XX\r\ndone\n"
|
||||
assert payload["changes"][0]["location"] == {"line": 1, "column": 1}
|
||||
assert payload["changes"][1]["location"] == {"line": 2, "column": 1}
|
||||
assert payload["changes"][1]["before"] == "target"
|
||||
assert payload["changes"][1]["after"] == "done"
|
||||
assert payload["changes"][1]["before_sha256"] == result.changes[1].before_sha256
|
||||
assert payload["changes"][1]["after_sha256"] == result.changes[1].after_sha256
|
||||
assert report.diff is not None
|
||||
assert b"--- a/paper.md\n+++ b/paper.md\n" in report.diff
|
||||
assert b"data/md/paper.md" not in report.diff
|
||||
|
||||
|
||||
def test_zero_change_success_still_has_cleaned_markdown_and_empty_diff() -> None:
|
||||
markdown = "中文\nCafe\u0301\n"
|
||||
result = Pipeline([]).transform(markdown)
|
||||
|
||||
report = build_document_report(
|
||||
document_id="unchanged",
|
||||
source_label="论文.md",
|
||||
input_markdown=markdown,
|
||||
result=result,
|
||||
)
|
||||
|
||||
assert report.cleaned_markdown == markdown.encode()
|
||||
assert report.diff == b""
|
||||
assert report.change_count == 0
|
||||
assert report.result_json.endswith(b"\n")
|
||||
assert "论文".encode() in report.result_json
|
||||
assert b"\\u4e2d" not in report.result_json
|
||||
|
||||
|
||||
def test_diff_preserves_final_newline_changes_and_uses_report_newlines() -> None:
|
||||
diff = build_unified_diff("paper", "same\r\n", "same")
|
||||
|
||||
assert b"--- a/paper.md\n+++ b/paper.md\n" in diff
|
||||
assert b"-same\n+same\n\\ No newline at end of file\n" in diff
|
||||
assert b"\r" not in diff
|
||||
|
||||
|
||||
def test_failed_report_keeps_change_audit_but_does_not_render_partial_markdown() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("a", "b", "test.first"),
|
||||
ExplodingComponent(),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform("a")
|
||||
|
||||
report = build_document_report(
|
||||
document_id="failed",
|
||||
source_label="failed.md",
|
||||
input_markdown="a",
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.FAILED
|
||||
assert report.cleaned_markdown is None
|
||||
assert report.diff is None
|
||||
assert payload["output"] == {"cleaned_path": None, "diff_path": None}
|
||||
assert len(payload["changes"]) == 1
|
||||
assert payload["errors"][0]["error_type"] == "RuntimeError"
|
||||
assert "private" not in payload["errors"][0]["message"]
|
||||
assert "partial_markdown" not in payload
|
||||
|
||||
|
||||
def test_unstable_report_serializes_residual_proposals_without_partial_markdown() -> None:
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
ReplaceComponent("bad", "good", "test.to-good"),
|
||||
ReplaceComponent("good", "bad", "test.to-bad"),
|
||||
]
|
||||
)
|
||||
result = pipeline.transform("bad")
|
||||
|
||||
report = build_document_report(
|
||||
document_id="unstable",
|
||||
source_label="unstable.md",
|
||||
input_markdown="bad",
|
||||
result=result,
|
||||
)
|
||||
payload = json.loads(report.result_json)
|
||||
|
||||
assert result.status is RunStatus.UNSTABLE
|
||||
assert report.cleaned_markdown is None
|
||||
assert report.diff is None
|
||||
assert payload["residual_proposals"][0]["proposal"]["reason"] == "replace bad"
|
||||
assert payload["residual_proposals"][0]["proposal"]["edits"][0]["expected_text"] == "bad"
|
||||
assert "partial_markdown" not in payload
|
||||
|
||||
|
||||
def test_report_rejects_a_tampered_change_hash_chain() -> None:
|
||||
result = Pipeline([ReplaceComponent("a", "b", "test.replace")]).transform("a")
|
||||
tampered_change = replace(result.changes[0], after_sha256="0" * 64)
|
||||
tampered_result = replace(result, changes=(tampered_change,))
|
||||
|
||||
with pytest.raises(ReportingError, match="batch hash"):
|
||||
build_document_report(
|
||||
document_id="tampered",
|
||||
source_label="tampered.md",
|
||||
input_markdown="a",
|
||||
result=tampered_result,
|
||||
)
|
||||
@@ -1,132 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from reviewer.server import ReviewArtifactError, ReviewArtifacts
|
||||
from tests.reviewer_fixture import create_review_run, read_json, write_json
|
||||
|
||||
|
||||
def test_reviewer_returns_full_comparison_and_all_component_stages(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
run = cast(dict[str, Any], repository.run_summary())
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
changed = cast(dict[str, Any], repository.component_stage("paper", 0))
|
||||
unchanged = cast(dict[str, Any], repository.component_stage("paper", 1))
|
||||
|
||||
assert run["summary"] == {
|
||||
"document_count": 1,
|
||||
"success_count": 1,
|
||||
"failed_count": 0,
|
||||
"unstable_count": 0,
|
||||
"change_count": 1,
|
||||
}
|
||||
assert run["documents"][0]["source_available"] is True
|
||||
assert comparison["original_markdown"] == fixture["original"]
|
||||
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||
assert comparison["changes"][0]["editor_range"] == {"start": 4, "end": 7}
|
||||
assert changed["before_markdown"] == fixture["original"]
|
||||
assert changed["after_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["before_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["after_markdown"] == fixture["cleaned"]
|
||||
assert unchanged["changes"] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", ["failed", "unstable"])
|
||||
def test_non_success_documents_expose_audit_but_no_formal_output(tmp_path: Path, status: str) -> None:
|
||||
fixture = create_review_run(tmp_path, status=status)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
|
||||
assert comparison["original_markdown"] == fixture["original"]
|
||||
assert comparison["cleaned_markdown"] is None
|
||||
if status == "failed":
|
||||
assert comparison["errors"]
|
||||
assert comparison["residual_proposals"] == []
|
||||
else:
|
||||
assert comparison["errors"] == []
|
||||
assert comparison["residual_proposals"]
|
||||
with pytest.raises(ReviewArtifactError, match="success"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_historical_run_without_locator_reports_unavailable_source(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path, with_locator=False)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
summary = cast(dict[str, Any], repository.run_summary())
|
||||
|
||||
assert summary["documents"][0]["source_available"] is False
|
||||
assert "review-locator.json" in summary["documents"][0]["availability_error"]
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
assert comparison["original_markdown"] is None
|
||||
assert comparison["cleaned_markdown"] == fixture["cleaned"]
|
||||
assert comparison["changes"][0]["editor_range"] is None
|
||||
with pytest.raises(ReviewArtifactError, match=r"review-locator\.json"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_source_hash_change_is_not_silently_displayed(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
source_path = fixture["source_path"]
|
||||
assert isinstance(source_path, Path)
|
||||
source_path.write_text("changed\n", encoding="utf-8")
|
||||
|
||||
summary = cast(dict[str, Any], repository.run_summary())
|
||||
|
||||
assert summary["documents"][0]["source_available"] is False
|
||||
assert "哈希" in summary["documents"][0]["availability_error"]
|
||||
comparison = cast(dict[str, Any], repository.document_comparison("paper"))
|
||||
assert comparison["original_markdown"] is None
|
||||
assert comparison["changes"][0]["editor_range"] is None
|
||||
with pytest.raises(ReviewArtifactError, match="哈希"):
|
||||
repository.component_stage("paper", 0)
|
||||
|
||||
|
||||
def test_reviewer_rejects_unknown_schema_and_artifact_path_escape(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
manifest_path = fixture["manifest_path"]
|
||||
assert isinstance(manifest_path, Path)
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["schema_version"] = 2
|
||||
write_json(manifest_path, manifest)
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as unknown:
|
||||
ReviewArtifacts(fixture["run_directory"])
|
||||
assert unknown.value.code == "unsupported_schema"
|
||||
|
||||
second = create_review_run(tmp_path / "second")
|
||||
second_manifest_path = second["manifest_path"]
|
||||
assert isinstance(second_manifest_path, Path)
|
||||
second_manifest = read_json(second_manifest_path)
|
||||
second_manifest["documents"][0]["result_path"] = "../outside.json"
|
||||
write_json(second_manifest_path, second_manifest)
|
||||
with pytest.raises(ReviewArtifactError, match="路径"):
|
||||
ReviewArtifacts(second["run_directory"])
|
||||
|
||||
|
||||
def test_reviewer_rejects_tampered_snapshot_chain_and_unknown_identity(tmp_path: Path) -> None:
|
||||
fixture = create_review_run(tmp_path)
|
||||
result_path = fixture["result_path"]
|
||||
assert isinstance(result_path, Path)
|
||||
result = read_json(result_path)
|
||||
result["changes"][0]["after_sha256"] = "0" * 64
|
||||
write_json(result_path, result)
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as replay_error:
|
||||
repository.document_comparison("paper")
|
||||
assert replay_error.value.code == "untrusted_replay"
|
||||
|
||||
with pytest.raises(ReviewArtifactError) as document_error:
|
||||
repository.document_comparison("missing")
|
||||
assert document_error.value.http_status == 404
|
||||
with pytest.raises(ReviewArtifactError) as component_error:
|
||||
repository.component_stage("paper", 99)
|
||||
assert component_error.value.http_status == 404
|
||||
@@ -1,105 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
from collections.abc import Generator
|
||||
from http.client import HTTPConnection, HTTPResponse
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from reviewer.server import ReviewArtifacts
|
||||
from reviewer.server.__main__ import create_server
|
||||
from tests.reviewer_fixture import ReviewFixture, create_review_run
|
||||
|
||||
|
||||
def request(
|
||||
port: int,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
headers: dict[str, str] | None = None,
|
||||
) -> tuple[HTTPResponse, bytes]:
|
||||
connection = HTTPConnection("127.0.0.1", port, timeout=3)
|
||||
connection.request(method, path, headers=headers or {})
|
||||
response = connection.getresponse()
|
||||
content = response.read()
|
||||
connection.close()
|
||||
return response, content
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def running_server(tmp_path: Path) -> Generator[tuple[int, ReviewFixture], None, None]:
|
||||
fixture = create_review_run(tmp_path)
|
||||
static_root = tmp_path / "static"
|
||||
(static_root / "assets").mkdir(parents=True)
|
||||
(static_root / "index.html").write_text("<main>reviewer</main>\n", encoding="utf-8")
|
||||
(static_root / "assets/app.js").write_text("export {};\n", encoding="utf-8")
|
||||
repository = ReviewArtifacts(fixture["run_directory"])
|
||||
server = create_server(repository, static_root)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
yield server.server_address[1], fixture
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(timeout=3)
|
||||
|
||||
|
||||
def test_server_exposes_same_origin_api_and_static_build(
|
||||
running_server: tuple[int, ReviewFixture],
|
||||
) -> None:
|
||||
port, fixture = running_server
|
||||
|
||||
api_response, api_content = request(port, "GET", "/api/v1/run")
|
||||
page_response, page_content = request(port, "GET", "/")
|
||||
asset_response, _ = request(port, "HEAD", "/assets/app.js")
|
||||
|
||||
payload = cast(dict[str, Any], json.loads(api_content))
|
||||
assert api_response.status == 200
|
||||
assert payload["run"]["run_id"] == "review-run"
|
||||
assert str(fixture["source_path"]) not in api_content.decode()
|
||||
assert api_response.getheader("Access-Control-Allow-Origin") is None
|
||||
assert api_response.getheader("Cache-Control") == "no-store"
|
||||
assert "default-src 'self'" in cast(str, api_response.getheader("Content-Security-Policy"))
|
||||
assert page_response.status == 200
|
||||
assert page_content == b"<main>reviewer</main>\n"
|
||||
assert asset_response.status == 200
|
||||
assert asset_response.getheader("Content-Type") == "text/javascript; charset=utf-8"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "headers", "status", "code"),
|
||||
[
|
||||
("POST", "/api/v1/run", None, 405, "method_not_allowed"),
|
||||
("GET", "/api/v1/run", {"Host": "example.com"}, 403, "invalid_origin"),
|
||||
(
|
||||
"GET",
|
||||
"/api/v1/run",
|
||||
{"Origin": "http://example.com"},
|
||||
403,
|
||||
"invalid_origin",
|
||||
),
|
||||
("GET", "/api/v1/documents/missing", None, 404, "unknown_document"),
|
||||
("GET", "/api/v1/documents/paper/components/99", None, 404, "unknown_component"),
|
||||
("GET", "/assets/missing.js", None, 404, "not_found"),
|
||||
("GET", "/..%2Fsecret.txt", None, 404, "not_found"),
|
||||
],
|
||||
)
|
||||
def test_server_rejects_unsafe_or_unknown_requests(
|
||||
running_server: tuple[int, ReviewFixture],
|
||||
method: str,
|
||||
path: str,
|
||||
headers: dict[str, str] | None,
|
||||
status: int,
|
||||
code: str,
|
||||
) -> None:
|
||||
port, _ = running_server
|
||||
|
||||
response, content = request(port, method, path, headers=headers)
|
||||
payload = cast(dict[str, Any], json.loads(content))
|
||||
|
||||
assert response.status == status
|
||||
assert payload["error"]["code"] == code
|
||||
@@ -1,69 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish import Pipeline, RunStatus
|
||||
from mdpolish.components import WordReviewCommentComponent
|
||||
|
||||
|
||||
def transform(markdown: str): # type: ignore[no-untyped-def]
|
||||
return Pipeline([WordReviewCommentComponent()]).transform(markdown)
|
||||
|
||||
|
||||
def test_deletes_single_line_comment_and_one_following_blank() -> None:
|
||||
markdown = "before\nCommented [AB1]: review this\n\nafter"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.status is RunStatus.SUCCESS
|
||||
assert result.output_markdown == "before\nafter"
|
||||
assert len(result.changes) == 1
|
||||
assert result.changes[0].reason == "删除严格单行 Word 审阅批注及其后一个空行"
|
||||
|
||||
|
||||
def test_adjacent_comment_blocks_are_deleted_without_overlap() -> None:
|
||||
markdown = "before\nCommented [A1]: first\n\nCommented [B2R1]: second\n\nafter"
|
||||
result = transform(markdown)
|
||||
|
||||
assert result.output_markdown == "before\nafter"
|
||||
assert len(result.changes) == 2
|
||||
assert result.changes[0].span.end == result.changes[1].span.start
|
||||
|
||||
|
||||
@pytest.mark.parametrize("line_ending", ["\n", "\r\n", "\r"])
|
||||
def test_preserves_line_ending_style(line_ending: str) -> None:
|
||||
markdown = line_ending.join(("before", "Commented [A1]: note", "", "after"))
|
||||
assert transform(markdown).output_markdown == line_ending.join(("before", "after"))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"comment",
|
||||
[
|
||||
"prefix Commented [A1]: note",
|
||||
" Commented [A1]: note",
|
||||
"Commented []: note",
|
||||
"Commented [A-1]: note",
|
||||
"Commented [A1]:",
|
||||
"Commented [A1]: ",
|
||||
],
|
||||
)
|
||||
def test_similar_lines_are_preserved(comment: str) -> None:
|
||||
markdown = f"before\n{comment}\n\nafter"
|
||||
assert transform(markdown).output_markdown == markdown
|
||||
|
||||
|
||||
def test_requires_a_following_blank_line_and_does_not_delete_extra_blanks() -> None:
|
||||
without_blank = "Commented [A1]: note\nafter"
|
||||
with_two_blanks = "before\nCommented [A1]: note\n\n\nafter"
|
||||
|
||||
assert transform(without_blank).output_markdown == without_blank
|
||||
assert transform(with_two_blanks).output_markdown == "before\n\nafter"
|
||||
|
||||
|
||||
def test_successful_output_is_idempotent() -> None:
|
||||
pipeline = Pipeline([WordReviewCommentComponent()])
|
||||
first = pipeline.transform("Commented [A1]: note\n\nafter")
|
||||
assert first.output_markdown is not None
|
||||
second = pipeline.transform(first.output_markdown)
|
||||
|
||||
assert second.status is RunStatus.SUCCESS
|
||||
assert second.changes == ()
|
||||
Reference in New Issue
Block a user