feat: 公开精确物理行范围接口

This commit is contained in:
2026-08-28 15:00:56 +08:00
parent 8b689eebe1
commit 61b7ea7534
9 changed files with 466 additions and 82 deletions
+38 -16
View File
@@ -3,8 +3,8 @@
`mdpolish` 是实验室共用的、项目无关的 Python Markdown 修改库。它提供函数式 `Modifier`、精确文本编辑执行器、 `mdpolish` 是实验室共用的、项目无关的 Python Markdown 修改库。它提供函数式 `Modifier`、精确文本编辑执行器、
有序 `Pipeline`、正则修改器工厂,以及少量可以用合成样例完整说明的通用修改器。 有序 `Pipeline`、正则修改器工厂,以及少量可以用合成样例完整说明的通用修改器。
当前发布版本是 [`v0.4.0`](https://github.com/Bepr4/mdpolish/releases/tag/v0.4.0),当前源码树的下一候选版本是 当前发布版本是 [`v0.6.0`](https://github.com/Bepr4/mdpolish/releases/tag/v0.6.0)。库只处理内存中的 Markdown 字符串,
`0.5.0`。库只处理内存中的 Markdown 字符串,不读取或写入文件,不提供默认流水线,也不包含任何项目的规则集合、 不读取或写入文件,不提供默认流水线,也不包含任何项目的规则集合、
数据清单、实验脚本或评审界面。 数据清单、实验脚本或评审界面。
## 当前能力 ## 当前能力
@@ -18,6 +18,7 @@
| `render_markdown_report()` | 把评审视图编排成完整 Markdown 源码报告字符串 | 只返回内存字符串,不创建文件或业务页面 | | `render_markdown_report()` | 把评审视图编排成完整 Markdown 源码报告字符串 | 只返回内存字符串,不创建文件或业务页面 |
| `review_document_to_dict()` | 按 schema `1.0` 把评审视图投影成普通 JSON 基本值 | 单向投影,不反序列化或重新应用修改 | | `review_document_to_dict()` | 按 schema `1.0` 把评审视图投影成普通 JSON 基本值 | 单向投影,不反序列化或重新应用修改 |
| `render_json_report()` | 复用正式 dict 投影生成确定的内存 JSON 字符串 | 不创建文件;默认摘要不等于公开安全日志 | | `render_json_report()` | 复用正式 dict 投影生成确定的内存 JSON 字符串 | 不创建文件;默认摘要不等于公开安全日志 |
| `mdpolish.text_ranges` | 返回 CR/LF/CRLF 物理行的精确不可变原文范围 | 不解析 Markdown 块,不自动执行或修改文本 |
| `regex_replace()` | 把非空正则匹配转换为精确编辑 | 不提供规则注册表、配置加载或默认模式 | | `regex_replace()` | 把非空正则匹配转换为精确编辑 | 不提供规则注册表、配置加载或默认模式 |
| `mapped_line_join()` | 用精确、正则或可选本地词典规则合并跨行片段 | 无默认规则;代码、表格、未知结构和歧义失败关闭 | | `mapped_line_join()` | 用精确、正则或可选本地词典规则合并跨行片段 | 无默认规则;代码、表格、未知结构和歧义失败关闭 |
| HTML 表格修改器 | 处理严格表格子集的实体和单行布局 | 不是完整 HTML parser,也不是 HTML→GFM 转换器 | | HTML 表格修改器 | 处理严格表格子集的实体和单行布局 | 不是完整 HTML parser,也不是 HTML→GFM 转换器 |
@@ -36,20 +37,19 @@
不可移动的 tag 不可移动的 tag
```bash ```bash
python -m pip install 'mdpolish @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0' python -m pip install 'mdpolish @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
python -m pip install 'mdpolish[lexical] @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0' python -m pip install 'mdpolish[lexical] @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
python -m pip install 'mdpolish[frequency] @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0' python -m pip install 'mdpolish[frequency] @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
``` ```
也可以安装同一 GitHub Release 附带的 wheel 也可以安装同一 GitHub Release 附带的 wheel
```bash ```bash
python -m pip install 'mdpolish[lexical] @ https://github.com/Bepr4/mdpolish/releases/download/v0.4.0/mdpolish-0.4.0-py3-none-any.whl' python -m pip install 'mdpolish[lexical] @ https://github.com/Bepr4/mdpolish/releases/download/v0.6.0/mdpolish-0.6.0-py3-none-any.whl'
``` ```
Release 页面同时提供 wheel 的 SHA-256 校验值。仓库或 Release 如果是私有的,调用方需要自行配置 GitHub 访问权限; Release 页面同时提供 wheel 的 SHA-256 校验值。仓库或 Release 如果是私有的,调用方需要自行配置 GitHub 访问权限;
库不会保存凭据。以上命令当前安装的是已发布的 `v0.4.0`,不包含本源码树尚未发布的 `0.5.0` 机器投影。开发环境仍从 库不会保存凭据。开发环境仍从本地工作树安装:
本地工作树安装:
```bash ```bash
python -m venv .venv python -m venv .venv
@@ -212,6 +212,26 @@ Markdown reporter 会包含完整输入、当前全文、统一 diff 以及实
dict 和 JSON 都是单向派生视图,不用于恢复 `ReviewDocument` 或重新应用修改。完整 schema、坐标、哈希和兼容口径见 dict 和 JSON 都是单向派生视图,不用于恢复 `ReviewDocument` 或重新应用修改。完整 schema、坐标、哈希和兼容口径见
[`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。 [`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。
## 扫描精确物理行
项目 Modifier 如果需要识别独占行、检查相邻行或连同行尾删除一行,可以按原文 code point 范围扫描:
```python
from mdpolish.text_ranges import physical_lines
source = "first\r\n\r\nlast"
lines = physical_lines(source)
assert lines[0].content(source) == "first"
assert lines[0].line_ending(source) == "\r\n"
assert lines[1].is_empty
assert source[lines[2].content_start : lines[2].full_end] == "last"
```
扫描只识别 LF、CR 和 CRLF,不规范化原文,也不判断段落、标题、列表、引用、代码或表格。它不会成为 Pipeline 的全局预处理;
只有显式调用它的 Modifier 才会扫描当前阶段快照。精确空文档、尾换行、offset 和失败关闭口径见
[`physical-line-ranges.md`](research-wiki/reference/physical-line-ranges.md)。
## 编写项目自己的修改器 ## 编写项目自己的修改器
复杂规则使用普通函数返回精确候选修改,不需要继承库基类: 复杂规则使用普通函数返回精确候选修改,不需要继承库基类:
@@ -288,6 +308,7 @@ src/mdpolish/
├── pipeline.py # 有序执行与最终稳定性复查 ├── pipeline.py # 有序执行与最终稳定性复查
├── review.py # 可信评审视图、机器投影及内存 JSON/Markdown reporter ├── review.py # 可信评审视图、机器投影及内存 JSON/Markdown reporter
├── regex.py # 正则修改器工厂 ├── regex.py # 正则修改器工厂
├── text_ranges.py # 公共精确物理行范围
└── modifiers/ # 少量项目无关的通用修改器 └── modifiers/ # 少量项目无关的通用修改器
tests/ # 只使用虚构文本的核心与通用修改器测试 tests/ # 只使用虚构文本的核心与通用修改器测试
research-wiki/ research-wiki/
@@ -312,8 +333,9 @@ research-wiki/
[`0011-generic-review-projection-and-reporting.md`](research-wiki/design/0011-generic-review-projection-and-reporting.md),当前机制见 [`0011-generic-review-projection-and-reporting.md`](research-wiki/design/0011-generic-review-projection-and-reporting.md),当前机制见
[`review-projection.md`](research-wiki/explanation/review-projection.md)。正式机器投影的批准边界见 [`review-projection.md`](research-wiki/explanation/review-projection.md)。正式机器投影的批准边界见
[`0012-review-document-machine-projection.md`](research-wiki/design/0012-review-document-machine-projection.md)schema `1.0` 的稳定 [`0012-review-document-machine-projection.md`](research-wiki/design/0012-review-document-machine-projection.md)schema `1.0` 的稳定
查询口径见 [`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。旧 design 只保存历史 查询口径见 [`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。精确物理行公共接口的边界见
决策,不代表当前交付能力。 [`0013-public-physical-line-ranges.md`](research-wiki/design/0013-public-physical-line-ranges.md),稳定查询口径见
[`physical-line-ranges.md`](research-wiki/reference/physical-line-ranges.md)。旧 design 只保存历史决策,不代表当前交付能力。
## 当前可用检查 ## 当前可用检查
@@ -344,10 +366,10 @@ Python 标准库。Release wheel 的 SHA-256 是
上述结果证明当前版本可安装并按合成契约运行,不代表任意词典阈值已经在真实业务语料上达到生产准确率。 上述结果证明当前版本可安装并按合成契约运行,不代表任意词典阈值已经在真实业务语料上达到生产准确率。
`0.5.0` 候选于 2026-08-28 在 Python 3.13.11 开发环境中实际得到:mypy 通过,pytest 为 `v0.6.0` 于 2026-08-28 在 Python 3.13.11 开发环境中实际得到:Ruff 和 mypy 通过,pytest 为
`228 passed, 3 skipped`;三个 skip 仍是没有安装的 optional backend。除工作区已有的 `src/mdpolish/regex.py` 中文注释 `290 passed, 3 skipped`;三个 skip 仍是没有安装的 optional backend。
改动外,Ruff 全部通过;未排除该文件的全仓 Ruff 因其中 30 个 `RUF002` / `RUF003` 失败,本轮没有擅自修改该用户改动。
候选 `mdpolish-0.5.0-py3-none-any.whl` 构建成功,共 17 个文件,包含更新后的 `review.py``py.typed`,不包含 tests、Wiki、 Release wheel `mdpolish-0.6.0-py3-none-any.whl` 共 17 个文件,包含 `text_ranges.py``review.py``py.typed`,不包含
报告或真实数据;在仓库外全新虚拟环境中无依赖安装后,dict 投影与 JSON reporter smoke test 通过。该临时 wheel 不是 Release `_text_ranges.py`、tests、Wiki、报告或真实数据;在仓库外全新虚拟环境中无依赖安装后,版本、公共导入、精确混合行尾范围、
资产,其哈希不构成发布身份;完成全仓 Ruff、提交、合并、tag 和 Release 仍需要分别确认。 行尾集合、空行判断和 wheel 清单 smoke test 通过。Release wheel 的 SHA-256 是
`695502b1a443d4e98dbf63e8bdcee59452baea2185cb7e1e13160127f70c920f`
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project] [project]
name = "mdpolish" name = "mdpolish"
version = "0.5.0" version = "0.6.0"
description = "Deterministic functional core for composing exact Markdown modifiers" description = "Deterministic functional core for composing exact Markdown modifiers"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [] dependencies = []
@@ -0,0 +1,137 @@
# 精确物理行范围
本文记录 `mdpolish.text_ranges` 当前稳定的公共查询口径。实现和运行校验以
`src/mdpolish/text_ranges.py` 与测试为准;设计理由、方案取舍和施工边界见
[`0013-public-physical-line-ranges.md`](../design/0013-public-physical-line-ranges.md)。
## 1. 公共入口
```python
from mdpolish.text_ranges import (
PhysicalLine,
iter_physical_lines,
line_ending_styles,
physical_lines,
)
```
这些名称只从 `mdpolish.text_ranges` 导出,不在包根重复导出。`mdpolish._text_ranges` 不是兼容路径。
模块只接收内存中的 `str`。它不读取文件、不改写文本、不建立 Markdown AST,也不会在 `Pipeline` 运行前自动执行。
## 2. 什么是物理行
物理行由以下任一边界结束:
| 原文边界 | 行尾字符串 | code point 长度 |
| --- | --- | ---: |
| LF | `"\n"` | 1 |
| CR | `"\r"` | 1 |
| CRLF | `"\r\n"` | 2 |
| 文档末尾 | `""` | 0 |
CRLF 是一个物理行尾。U+2028、U+2029、vertical tab、form feed、NUL 以及其他字符都属于行内容,不会被当成物理行尾。
扫描不做 Unicode 或换行规范化。
空文档产生零条物理行。原文以行尾结束时,不会在末尾虚构额外空行;连续行尾之间确实存在的零长度行仍会返回。
## 3. `PhysicalLine`
`PhysicalLine``dataclass(frozen=True, slots=True)`
```python
PhysicalLine(
content_start=0,
content_end=4,
full_end=6,
)
```
三个 offset 都是相对于同一个 Python `str` 的 0-based Unicode code point index
- 内容范围:`[content_start, content_end)`
- 行尾范围:`[content_end, full_end)`
- 完整物理行范围:`[content_start, full_end)`
它们不是 UTF-8 byte、UTF-16 code unit、终端显示列或人类使用的 1-based 行列。
公共成员:
| 成员 | 结果 |
| --- | --- |
| `content(source)` | 返回内容范围的精确切片 |
| `line_ending(source)` | 返回 `""`、LF、CR 或 CRLF |
| `is_empty` | 内容范围长度是否为零 |
`is_empty` 不等于 Markdown blank line。`" \n"``"\t\n"` 的内容不是零长度,因此均为 `False`。项目若要把 space、Tab
或其他 Unicode whitespace 当作空白,必须在自己的 Modifier 中显式定义。
手工构造时,offset 必须是非负的严格 `int`,不接受 `bool`,并满足:
```text
content_start <= content_end <= full_end
full_end - content_end <= 2
```
访问器还会拒绝非字符串来源、超出来源长度的范围,以及不是 `""`、LF、CR、CRLF 的行尾切片。错误信息不包含原文。
## 4. 扫描函数
`iter_physical_lines(source)` 返回惰性迭代器,但会在函数调用当下检查 `source` 类型。它单次从左到右扫描,不预先复制原文。
`physical_lines(source)` 返回同一扫描结果的不可变 tuple,适合需要查看前后行或重复遍历的 Modifier。
`line_ending_styles(source)` 返回原文实际出现过的非空行尾集合,只可能包含 `"\n"``"\r"``"\r\n"`。空文档或
没有换行的文档返回空 `frozenset`
## 5. 精确示例
下表的结果写作 `(content_start, content_end, full_end)`
| source | 结果 |
| --- | --- |
| `""` | `()` |
| `"a"` | `((0, 1, 1),)` |
| `"a\n"` | `((0, 1, 2),)` |
| `"\n"` | `((0, 0, 1),)` |
| `"\r\n"` | `((0, 0, 2),)` |
| `"a\n\n"` | `((0, 1, 2), (2, 2, 3))` |
| `"\ntext"` | `((0, 0, 1), (1, 5, 5))` |
| `"a\r\nb\rc\n"` | `((0, 1, 3), (3, 4, 5), (5, 6, 7))` |
| `"a\u2028b"` | `((0, 3, 3),)` |
按顺序拼接所有 `source[line.content_start:line.full_end]` 必须逐 code point 还原原文。
## 6. 在项目 Modifier 中使用
物理行范围只是定位工具,不会自动成为修改:
```python
from mdpolish import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
from mdpolish.text_ranges import physical_lines
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
lines = physical_lines(snapshot.markdown)
if not lines or lines[0].content(snapshot.markdown) != "[REMOVE-ME]":
return ()
line = lines[0]
expected = snapshot.markdown[line.content_start : line.full_end]
return (
ProposedChange(
snapshot_sha256=snapshot.sha256,
reason="删除项目确认的独占行标记",
edits=(
TextEdit(
snapshot_sha256=snapshot.sha256,
span=TextSpan(line.content_start, line.full_end),
expected_text=expected,
replacement="",
),
),
),
)
```
项目仍负责判断标题、列表、引用、代码、表格和业务语义。公共编辑执行器仍负责验证快照哈希、范围、预期原文与冲突。
-60
View File
@@ -1,60 +0,0 @@
"""Exact physical-line ranges shared by text cleaning modifiers."""
from __future__ import annotations
from collections.abc import Iterator
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class PhysicalLine:
"""A content range and its optional line ending in one source string."""
content_start: int
content_end: int
full_end: int
def content(self, markdown: str) -> str:
return markdown[self.content_start : self.content_end]
def line_ending(self, markdown: str) -> str:
return markdown[self.content_end : self.full_end]
def is_blank(self, markdown: str) -> bool:
return self.content_start == self.content_end
def iter_physical_lines(markdown: str) -> Iterator[PhysicalLine]:
"""Yield exact physical lines without normalizing CR or LF endings."""
line_start = 0
position = 0
while position < len(markdown):
character = markdown[position]
if character == "\n":
yield PhysicalLine(line_start, position, position + 1)
position += 1
line_start = position
elif character == "\r":
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
yield PhysicalLine(line_start, position, line_end)
position = line_end
line_start = position
else:
position += 1
if line_start < len(markdown):
yield PhysicalLine(line_start, len(markdown), len(markdown))
def physical_lines(markdown: str) -> tuple[PhysicalLine, ...]:
"""Return all physical-line ranges as an immutable sequence."""
return tuple(iter_physical_lines(markdown))
def line_ending_styles(markdown: str) -> frozenset[str]:
"""Return the distinct non-empty physical line endings in a string."""
return frozenset(
line.line_ending(markdown)
for line in iter_physical_lines(markdown)
if line.full_end > line.content_end
)
+1 -1
View File
@@ -3,9 +3,9 @@
from __future__ import annotations from __future__ import annotations
from mdpolish._html_table import strict_html_tables from mdpolish._html_table import strict_html_tables
from mdpolish._text_ranges import line_ending_styles
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
from mdpolish.modifier import Modifier from mdpolish.modifier import Modifier
from mdpolish.text_ranges import line_ending_styles
_REASON = "展开严格单行 HTML 表格的行布局" _REASON = "展开严格单行 HTML 表格的行布局"
_APPLICABILITY = ( _APPLICABILITY = (
+3 -3
View File
@@ -48,9 +48,9 @@ from importlib.metadata import version as package_version
from itertools import pairwise from itertools import pairwise
from typing import Protocol, TypeAlias from typing import Protocol, TypeAlias
from mdpolish._text_ranges import PhysicalLine, physical_lines
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
from mdpolish.modifier import Modifier, ModifierContractError from mdpolish.modifier import Modifier, ModifierContractError
from mdpolish.text_ranges import PhysicalLine, physical_lines
try: try:
from spellchecker import SpellChecker as _SpellChecker # type: ignore[import-not-found] from spellchecker import SpellChecker as _SpellChecker # type: ignore[import-not-found]
@@ -543,7 +543,7 @@ def _scan_lines(markdown: str) -> tuple[_ScannedLine, ...]:
blank_run = 0 blank_run = 0
for index, line in enumerate(lines): for index, line in enumerate(lines):
raw = line.content(markdown) raw = line.content(markdown)
if line.is_blank(markdown): if line.is_empty:
blank_run += 1 blank_run += 1
active_heading = None active_heading = None
if blank_run > 1: if blank_run > 1:
@@ -1143,7 +1143,7 @@ def _boundary_between(
if right.physical_index <= left.physical_index: if right.physical_index <= left.physical_index:
raise ModifierContractError("line join scanner produced non-monotonic physical lines") raise ModifierContractError("line join scanner produced non-monotonic physical lines")
intervening = lines[left.physical_index + 1 : right.physical_index] intervening = lines[left.physical_index + 1 : right.physical_index]
if len(intervening) > max_blank_lines or any(not line.is_blank(markdown) for line in intervening): if len(intervening) > max_blank_lines or any(not line.is_empty for line in intervening):
return None return None
ending = left.physical.line_ending(markdown) ending = left.physical.line_ending(markdown)
if not ending or any(line.line_ending(markdown) != ending for line in intervening): if not ending or any(line.line_ending(markdown) != ending for line in intervening):
+111
View File
@@ -0,0 +1,111 @@
"""Exact physical-line ranges for project and library modifiers.
This module scans only CR, LF, and CRLF boundaries. It preserves the source
string and reports Python Unicode code point offsets; it does not parse
Markdown blocks, normalize line endings, read files, or propose edits.
"""
from __future__ import annotations
from collections.abc import Iterator
from dataclasses import dataclass
__all__ = [
"PhysicalLine",
"iter_physical_lines",
"line_ending_styles",
"physical_lines",
]
_LINE_ENDINGS = frozenset(("", "\n", "\r", "\r\n"))
def _require_source(source: object) -> str:
if not isinstance(source, str):
raise TypeError("source must be a string")
return source
@dataclass(frozen=True, slots=True)
class PhysicalLine:
"""A source string's exact content range and optional line-ending range."""
content_start: int
content_end: int
full_end: int
def __post_init__(self) -> None:
for field_name in ("content_start", "content_end", "full_end"):
if type(getattr(self, field_name)) is not int:
raise TypeError(f"{field_name} must be an integer")
if self.content_start < 0 or self.content_end < 0 or self.full_end < 0:
raise ValueError("physical line offsets must be non-negative")
if not self.content_start <= self.content_end <= self.full_end:
raise ValueError("physical line offsets must be ordered")
if self.full_end - self.content_end > 2:
raise ValueError("physical line ending range must contain at most two code points")
def _validated_source(self, source: object) -> str:
validated = _require_source(source)
if self.full_end > len(validated):
raise ValueError("physical line range exceeds source length")
return validated
def content(self, source: str) -> str:
"""Return this line's content without its physical line ending."""
validated = self._validated_source(source)
return validated[self.content_start : self.content_end]
def line_ending(self, source: str) -> str:
"""Return this line's exact CR, LF, CRLF, or absent line ending."""
validated = self._validated_source(source)
ending = validated[self.content_end : self.full_end]
if ending not in _LINE_ENDINGS:
raise ValueError("physical line ending must be empty, LF, CR, or CRLF")
return ending
@property
def is_empty(self) -> bool:
"""Return whether the content range has zero length."""
return self.content_start == self.content_end
def _iter_physical_lines(source: str) -> Iterator[PhysicalLine]:
line_start = 0
position = 0
while position < len(source):
character = source[position]
if character == "\n":
yield PhysicalLine(line_start, position, position + 1)
position += 1
line_start = position
elif character == "\r":
line_end = position + 2 if position + 1 < len(source) and source[position + 1] == "\n" else position + 1
yield PhysicalLine(line_start, position, line_end)
position = line_end
line_start = position
else:
position += 1
if line_start < len(source):
yield PhysicalLine(line_start, len(source), len(source))
def iter_physical_lines(source: str) -> Iterator[PhysicalLine]:
"""Return a lazy iterator over exact physical lines in ``source``."""
return _iter_physical_lines(_require_source(source))
def physical_lines(source: str) -> tuple[PhysicalLine, ...]:
"""Return all exact physical-line ranges as an immutable sequence."""
return tuple(iter_physical_lines(source))
def line_ending_styles(source: str) -> frozenset[str]:
"""Return the distinct non-empty physical line endings in ``source``."""
validated = _require_source(source)
return frozenset(
line.line_ending(validated)
for line in iter_physical_lines(validated)
if line.full_end > line.content_end
)
+1 -1
View File
@@ -362,7 +362,7 @@ def test_empty_document_and_empty_rule_set_are_no_ops() -> None:
def test_installed_package_version_matches_delivery_candidate() -> None: def test_installed_package_version_matches_delivery_candidate() -> None:
assert distribution_version("mdpolish") == "0.4.0" assert distribution_version("mdpolish") == "0.6.0"
def test_parameters_preserve_rule_order_and_record_all_options() -> None: def test_parameters_preserve_rule_order_and_record_all_options() -> None:
+174
View File
@@ -0,0 +1,174 @@
from __future__ import annotations
from collections.abc import Callable
from dataclasses import FrozenInstanceError
from pathlib import Path
from typing import Any, cast
import pytest
from mdpolish.text_ranges import PhysicalLine, iter_physical_lines, line_ending_styles, physical_lines
def _offsets(lines: tuple[PhysicalLine, ...]) -> tuple[tuple[int, int, int], ...]:
return tuple((line.content_start, line.content_end, line.full_end) for line in lines)
@pytest.mark.parametrize(
("source", "expected"),
[
("", ()),
("a", ((0, 1, 1),)),
("a\n", ((0, 1, 2),)),
("\n", ((0, 0, 1),)),
("\r\n", ((0, 0, 2),)),
("a\n\n", ((0, 1, 2), (2, 2, 3))),
("\ntext", ((0, 0, 1), (1, 5, 5))),
("a\r\nb\rc\n", ((0, 1, 3), (3, 4, 5), (5, 6, 7))),
("a\u2028b", ((0, 3, 3),)),
],
)
def test_scans_exact_physical_line_offsets(
source: str,
expected: tuple[tuple[int, int, int], ...],
) -> None:
assert _offsets(physical_lines(source)) == expected
@pytest.mark.parametrize(
"source",
[
"",
"one line",
"\nleading empty",
"a\n\n",
"a\r\nb\rc\n",
"\ufeff中文😀e\u0301\u2028\x00\t\f",
],
)
def test_ranges_reconstruct_source_without_normalization(source: str) -> None:
lines = physical_lines(source)
assert "".join(source[line.content_start : line.full_end] for line in lines) == source
assert lines == tuple(iter_physical_lines(source))
assert lines == physical_lines(source)
for line in lines:
assert line.content(source) == source[line.content_start : line.content_end]
assert line.line_ending(source) == source[line.content_end : line.full_end]
def test_unicode_separators_and_control_characters_remain_content() -> None:
source = "\ufeff中文😀e\u0301\u2028\u2029\x00\t\f"
line = physical_lines(source)[0]
assert (line.content_start, line.content_end, line.full_end) == (0, len(source), len(source))
assert line.content(source) == source
assert line.line_ending(source) == ""
@pytest.mark.parametrize(
("source", "expected"),
[
("", frozenset()),
("no ending", frozenset()),
("a\nb\n", frozenset(("\n",))),
("a\r\nb\r\n", frozenset(("\r\n",))),
("a\rb\r", frozenset(("\r",))),
("a\nb\r\nc\r", frozenset(("\n", "\r\n", "\r"))),
],
)
def test_reports_only_line_endings_present_in_source(source: str, expected: frozenset[str]) -> None:
assert line_ending_styles(source) == expected
def test_empty_means_zero_length_content_not_whitespace() -> None:
empty, space, tab = physical_lines("\n \n\t\n")
assert empty.is_empty
assert not space.is_empty
assert not tab.is_empty
def test_physical_line_is_frozen_and_slotted() -> None:
line = PhysicalLine(0, 1, 1)
mutable_view = cast(Any, line)
with pytest.raises(FrozenInstanceError):
mutable_view.content_start = 1
assert not hasattr(line, "__dict__")
@pytest.mark.parametrize("field_index", [0, 1, 2])
@pytest.mark.parametrize("invalid", [True, 1.0, "1", None])
def test_rejects_non_integer_offsets(field_index: int, invalid: object) -> None:
values = [0, 0, 0]
values[field_index] = cast(int, invalid)
with pytest.raises(TypeError):
PhysicalLine(*values)
@pytest.mark.parametrize(
"offsets",
[
(-1, 0, 0),
(0, -1, 0),
(0, 0, -1),
(1, 0, 1),
(0, 2, 1),
(0, 0, 3),
],
)
def test_rejects_invalid_offset_ranges(offsets: tuple[int, int, int]) -> None:
with pytest.raises(ValueError):
PhysicalLine(*offsets)
@pytest.mark.parametrize(
"scanner",
[iter_physical_lines, physical_lines, line_ending_styles],
)
@pytest.mark.parametrize("invalid", [b"text", Path("text.md"), None, 1])
def test_scanners_reject_non_string_sources_immediately(
scanner: Callable[[str], object],
invalid: object,
) -> None:
with pytest.raises(TypeError, match="source must be a string"):
scanner(cast(str, invalid))
def test_iter_scanner_validates_before_iteration_begins() -> None:
with pytest.raises(TypeError, match="source must be a string"):
iter_physical_lines(cast(str, b"deferred validation would be wrong"))
def test_accessors_reject_non_string_or_short_sources() -> None:
line = PhysicalLine(0, 1, 2)
with pytest.raises(TypeError, match="source must be a string"):
line.content(cast(str, b"a\n"))
with pytest.raises(TypeError, match="source must be a string"):
line.line_ending(cast(str, None))
with pytest.raises(ValueError, match="exceeds source length"):
line.content("a")
with pytest.raises(ValueError, match="exceeds source length"):
line.line_ending("a")
@pytest.mark.parametrize(
("line", "source"),
[
(PhysicalLine(0, 1, 2), "ab"),
(PhysicalLine(0, 0, 2), "\n\r"),
],
)
def test_line_ending_rejects_non_physical_slices(line: PhysicalLine, source: str) -> None:
with pytest.raises(ValueError, match="empty, LF, CR, or CRLF"):
line.line_ending(source)
@pytest.mark.parametrize("source", ["a", "a\n", "a\r", "a\r\n"])
def test_line_ending_accepts_every_supported_slice(source: str) -> None:
line = physical_lines(source)[0]
assert line.line_ending(source) in {"", "\n", "\r", "\r\n"}