feat: 公开精确物理行范围接口
This commit is contained in:
@@ -3,8 +3,8 @@
|
||||
`mdpolish` 是实验室共用的、项目无关的 Python Markdown 修改库。它提供函数式 `Modifier`、精确文本编辑执行器、
|
||||
有序 `Pipeline`、正则修改器工厂，以及少量可以用合成样例完整说明的通用修改器。
|
||||
|
||||
当前发布版本是 [`v0.4.0`](https://github.com/Bepr4/mdpolish/releases/tag/v0.4.0)，当前源码树的下一候选版本是
|
||||
`0.5.0`。库只处理内存中的 Markdown 字符串，不读取或写入文件，不提供默认流水线，也不包含任何项目的规则集合、
|
||||
当前发布版本是 [`v0.6.0`](https://github.com/Bepr4/mdpolish/releases/tag/v0.6.0)。库只处理内存中的 Markdown 字符串，
|
||||
不读取或写入文件，不提供默认流水线，也不包含任何项目的规则集合、
|
||||
数据清单、实验脚本或评审界面。
|
||||
|
||||
## 当前能力
|
||||
@@ -18,6 +18,7 @@
|
||||
| `render_markdown_report()` | 把评审视图编排成完整 Markdown 源码报告字符串 | 只返回内存字符串，不创建文件或业务页面 |
|
||||
| `review_document_to_dict()` | 按 schema `1.0` 把评审视图投影成普通 JSON 基本值 | 单向投影，不反序列化或重新应用修改 |
|
||||
| `render_json_report()` | 复用正式 dict 投影生成确定的内存 JSON 字符串 | 不创建文件；默认摘要不等于公开安全日志 |
|
||||
| `mdpolish.text_ranges` | 返回 CR/LF/CRLF 物理行的精确不可变原文范围 | 不解析 Markdown 块，不自动执行或修改文本 |
|
||||
| `regex_replace()` | 把非空正则匹配转换为精确编辑 | 不提供规则注册表、配置加载或默认模式 |
|
||||
| `mapped_line_join()` | 用精确、正则或可选本地词典规则合并跨行片段 | 无默认规则；代码、表格、未知结构和歧义失败关闭 |
|
||||
| HTML 表格修改器 | 处理严格表格子集的实体和单行布局 | 不是完整 HTML parser，也不是 HTML→GFM 转换器 |
|
||||
@@ -36,20 +37,19 @@
|
||||
不可移动的 tag：
|
||||
|
||||
```bash
|
||||
python -m pip install 'mdpolish @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0'
|
||||
python -m pip install 'mdpolish[lexical] @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0'
|
||||
python -m pip install 'mdpolish[frequency] @ git+https://github.com/Bepr4/mdpolish.git@v0.4.0'
|
||||
python -m pip install 'mdpolish @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
|
||||
python -m pip install 'mdpolish[lexical] @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
|
||||
python -m pip install 'mdpolish[frequency] @ git+https://github.com/Bepr4/mdpolish.git@v0.6.0'
|
||||
```
|
||||
|
||||
也可以安装同一 GitHub Release 附带的 wheel：
|
||||
|
||||
```bash
|
||||
python -m pip install 'mdpolish[lexical] @ https://github.com/Bepr4/mdpolish/releases/download/v0.4.0/mdpolish-0.4.0-py3-none-any.whl'
|
||||
python -m pip install 'mdpolish[lexical] @ https://github.com/Bepr4/mdpolish/releases/download/v0.6.0/mdpolish-0.6.0-py3-none-any.whl'
|
||||
```
|
||||
|
||||
Release 页面同时提供 wheel 的 SHA-256 校验值。仓库或 Release 如果是私有的，调用方需要自行配置 GitHub 访问权限；
|
||||
库不会保存凭据。以上命令当前安装的是已发布的 `v0.4.0`，不包含本源码树尚未发布的 `0.5.0` 机器投影。开发环境仍从
|
||||
本地工作树安装：
|
||||
库不会保存凭据。开发环境仍从本地工作树安装：
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
@@ -212,6 +212,26 @@ Markdown reporter 会包含完整输入、当前全文、统一 diff 以及实
|
||||
dict 和 JSON 都是单向派生视图，不用于恢复 `ReviewDocument` 或重新应用修改。完整 schema、坐标、哈希和兼容口径见
|
||||
[`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。
|
||||
|
||||
## 扫描精确物理行
|
||||
|
||||
项目 Modifier 如果需要识别独占行、检查相邻行或连同行尾删除一行，可以按原文 code point 范围扫描：
|
||||
|
||||
```python
|
||||
from mdpolish.text_ranges import physical_lines
|
||||
|
||||
source = "first\r\n\r\nlast"
|
||||
lines = physical_lines(source)
|
||||
|
||||
assert lines[0].content(source) == "first"
|
||||
assert lines[0].line_ending(source) == "\r\n"
|
||||
assert lines[1].is_empty
|
||||
assert source[lines[2].content_start : lines[2].full_end] == "last"
|
||||
```
|
||||
|
||||
扫描只识别 LF、CR 和 CRLF，不规范化原文，也不判断段落、标题、列表、引用、代码或表格。它不会成为 Pipeline 的全局预处理；
|
||||
只有显式调用它的 Modifier 才会扫描当前阶段快照。精确空文档、尾换行、offset 和失败关闭口径见
|
||||
[`physical-line-ranges.md`](research-wiki/reference/physical-line-ranges.md)。
|
||||
|
||||
## 编写项目自己的修改器
|
||||
|
||||
复杂规则使用普通函数返回精确候选修改，不需要继承库基类：
|
||||
@@ -288,6 +308,7 @@ src/mdpolish/
|
||||
├── pipeline.py # 有序执行与最终稳定性复查
|
||||
├── review.py # 可信评审视图、机器投影及内存 JSON/Markdown reporter
|
||||
├── regex.py # 正则修改器工厂
|
||||
├── text_ranges.py # 公共精确物理行范围
|
||||
└── modifiers/ # 少量项目无关的通用修改器
|
||||
tests/ # 只使用虚构文本的核心与通用修改器测试
|
||||
research-wiki/
|
||||
@@ -312,8 +333,9 @@ research-wiki/
|
||||
[`0011-generic-review-projection-and-reporting.md`](research-wiki/design/0011-generic-review-projection-and-reporting.md)，当前机制见
|
||||
[`review-projection.md`](research-wiki/explanation/review-projection.md)。正式机器投影的批准边界见
|
||||
[`0012-review-document-machine-projection.md`](research-wiki/design/0012-review-document-machine-projection.md)，schema `1.0` 的稳定
|
||||
查询口径见 [`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。旧 design 只保存历史
|
||||
决策，不代表当前交付能力。
|
||||
查询口径见 [`review-projection-schema-v1.md`](research-wiki/reference/review-projection-schema-v1.md)。精确物理行公共接口的边界见
|
||||
[`0013-public-physical-line-ranges.md`](research-wiki/design/0013-public-physical-line-ranges.md)，稳定查询口径见
|
||||
[`physical-line-ranges.md`](research-wiki/reference/physical-line-ranges.md)。旧 design 只保存历史决策，不代表当前交付能力。
|
||||
|
||||
## 当前可用检查
|
||||
|
||||
@@ -344,10 +366,10 @@ Python 标准库。Release wheel 的 SHA-256 是
|
||||
|
||||
上述结果证明当前版本可安装并按合成契约运行，不代表任意词典阈值已经在真实业务语料上达到生产准确率。
|
||||
|
||||
`0.5.0` 候选于 2026-08-28 在 Python 3.13.11 开发环境中实际得到：mypy 通过，pytest 为
|
||||
`228 passed, 3 skipped`；三个 skip 仍是没有安装的 optional backend。除工作区已有的 `src/mdpolish/regex.py` 中文注释
|
||||
改动外，Ruff 全部通过；未排除该文件的全仓 Ruff 因其中 30 个 `RUF002` / `RUF003` 失败，本轮没有擅自修改该用户改动。
|
||||
`v0.6.0` 于 2026-08-28 在 Python 3.13.11 开发环境中实际得到：Ruff 和 mypy 通过，pytest 为
|
||||
`290 passed, 3 skipped`；三个 skip 仍是没有安装的 optional backend。
|
||||
|
||||
候选 `mdpolish-0.5.0-py3-none-any.whl` 构建成功，共 17 个文件，包含更新后的 `review.py` 和 `py.typed`，不包含 tests、Wiki、
|
||||
报告或真实数据；在仓库外全新虚拟环境中无依赖安装后，dict 投影与 JSON reporter smoke test 通过。该临时 wheel 不是 Release
|
||||
资产，其哈希不构成发布身份；完成全仓 Ruff、提交、合并、tag 和 Release 仍需要分别确认。
|
||||
Release wheel `mdpolish-0.6.0-py3-none-any.whl` 共 17 个文件，包含 `text_ranges.py`、`review.py` 和 `py.typed`，不包含
|
||||
`_text_ranges.py`、tests、Wiki、报告或真实数据；在仓库外全新虚拟环境中无依赖安装后，版本、公共导入、精确混合行尾范围、
|
||||
行尾集合、空行判断和 wheel 清单 smoke test 通过。Release wheel 的 SHA-256 是
|
||||
`695502b1a443d4e98dbf63e8bdcee59452baea2185cb7e1e13160127f70c920f`。
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "mdpolish"
|
||||
version = "0.5.0"
|
||||
version = "0.6.0"
|
||||
description = "Deterministic functional core for composing exact Markdown modifiers"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
# 精确物理行范围
|
||||
|
||||
本文记录 `mdpolish.text_ranges` 当前稳定的公共查询口径。实现和运行校验以
|
||||
`src/mdpolish/text_ranges.py` 与测试为准；设计理由、方案取舍和施工边界见
|
||||
[`0013-public-physical-line-ranges.md`](../design/0013-public-physical-line-ranges.md)。
|
||||
|
||||
## 1. 公共入口
|
||||
|
||||
```python
|
||||
from mdpolish.text_ranges import (
|
||||
PhysicalLine,
|
||||
iter_physical_lines,
|
||||
line_ending_styles,
|
||||
physical_lines,
|
||||
)
|
||||
```
|
||||
|
||||
这些名称只从 `mdpolish.text_ranges` 导出，不在包根重复导出。`mdpolish._text_ranges` 不是兼容路径。
|
||||
|
||||
模块只接收内存中的 `str`。它不读取文件、不改写文本、不建立 Markdown AST，也不会在 `Pipeline` 运行前自动执行。
|
||||
|
||||
## 2. 什么是物理行
|
||||
|
||||
物理行由以下任一边界结束：
|
||||
|
||||
| 原文边界 | 行尾字符串 | code point 长度 |
|
||||
| --- | --- | ---: |
|
||||
| LF | `"\n"` | 1 |
|
||||
| CR | `"\r"` | 1 |
|
||||
| CRLF | `"\r\n"` | 2 |
|
||||
| 文档末尾 | `""` | 0 |
|
||||
|
||||
CRLF 是一个物理行尾。U+2028、U+2029、vertical tab、form feed、NUL 以及其他字符都属于行内容，不会被当成物理行尾。
|
||||
扫描不做 Unicode 或换行规范化。
|
||||
|
||||
空文档产生零条物理行。原文以行尾结束时，不会在末尾虚构额外空行；连续行尾之间确实存在的零长度行仍会返回。
|
||||
|
||||
## 3. `PhysicalLine`
|
||||
|
||||
`PhysicalLine` 是 `dataclass(frozen=True, slots=True)`：
|
||||
|
||||
```python
|
||||
PhysicalLine(
|
||||
content_start=0,
|
||||
content_end=4,
|
||||
full_end=6,
|
||||
)
|
||||
```
|
||||
|
||||
三个 offset 都是相对于同一个 Python `str` 的 0-based Unicode code point index：
|
||||
|
||||
- 内容范围：`[content_start, content_end)`；
|
||||
- 行尾范围：`[content_end, full_end)`；
|
||||
- 完整物理行范围：`[content_start, full_end)`。
|
||||
|
||||
它们不是 UTF-8 byte、UTF-16 code unit、终端显示列或人类使用的 1-based 行列。
|
||||
|
||||
公共成员：
|
||||
|
||||
| 成员 | 结果 |
|
||||
| --- | --- |
|
||||
| `content(source)` | 返回内容范围的精确切片 |
|
||||
| `line_ending(source)` | 返回 `""`、LF、CR 或 CRLF |
|
||||
| `is_empty` | 内容范围长度是否为零 |
|
||||
|
||||
`is_empty` 不等于 Markdown blank line。`" \n"` 和 `"\t\n"` 的内容不是零长度，因此均为 `False`。项目若要把 space、Tab
|
||||
或其他 Unicode whitespace 当作空白，必须在自己的 Modifier 中显式定义。
|
||||
|
||||
手工构造时，offset 必须是非负的严格 `int`，不接受 `bool`，并满足：
|
||||
|
||||
```text
|
||||
content_start <= content_end <= full_end
|
||||
full_end - content_end <= 2
|
||||
```
|
||||
|
||||
访问器还会拒绝非字符串来源、超出来源长度的范围，以及不是 `""`、LF、CR、CRLF 的行尾切片。错误信息不包含原文。
|
||||
|
||||
## 4. 扫描函数
|
||||
|
||||
`iter_physical_lines(source)` 返回惰性迭代器，但会在函数调用当下检查 `source` 类型。它单次从左到右扫描，不预先复制原文。
|
||||
|
||||
`physical_lines(source)` 返回同一扫描结果的不可变 tuple，适合需要查看前后行或重复遍历的 Modifier。
|
||||
|
||||
`line_ending_styles(source)` 返回原文实际出现过的非空行尾集合，只可能包含 `"\n"`、`"\r"`、`"\r\n"`。空文档或
|
||||
没有换行的文档返回空 `frozenset`。
|
||||
|
||||
## 5. 精确示例
|
||||
|
||||
下表的结果写作 `(content_start, content_end, full_end)`：
|
||||
|
||||
| source | 结果 |
|
||||
| --- | --- |
|
||||
| `""` | `()` |
|
||||
| `"a"` | `((0, 1, 1),)` |
|
||||
| `"a\n"` | `((0, 1, 2),)` |
|
||||
| `"\n"` | `((0, 0, 1),)` |
|
||||
| `"\r\n"` | `((0, 0, 2),)` |
|
||||
| `"a\n\n"` | `((0, 1, 2), (2, 2, 3))` |
|
||||
| `"\ntext"` | `((0, 0, 1), (1, 5, 5))` |
|
||||
| `"a\r\nb\rc\n"` | `((0, 1, 3), (3, 4, 5), (5, 6, 7))` |
|
||||
| `"a\u2028b"` | `((0, 3, 3),)` |
|
||||
|
||||
按顺序拼接所有 `source[line.content_start:line.full_end]` 必须逐 code point 还原原文。
|
||||
|
||||
## 6. 在项目 Modifier 中使用
|
||||
|
||||
物理行范围只是定位工具，不会自动成为修改：
|
||||
|
||||
```python
|
||||
from mdpolish import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.text_ranges import physical_lines
|
||||
|
||||
|
||||
def propose(snapshot: DocumentSnapshot) -> tuple[ProposedChange, ...]:
|
||||
lines = physical_lines(snapshot.markdown)
|
||||
if not lines or lines[0].content(snapshot.markdown) != "[REMOVE-ME]":
|
||||
return ()
|
||||
|
||||
line = lines[0]
|
||||
expected = snapshot.markdown[line.content_start : line.full_end]
|
||||
return (
|
||||
ProposedChange(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
reason="删除项目确认的独占行标记",
|
||||
edits=(
|
||||
TextEdit(
|
||||
snapshot_sha256=snapshot.sha256,
|
||||
span=TextSpan(line.content_start, line.full_end),
|
||||
expected_text=expected,
|
||||
replacement="",
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
项目仍负责判断标题、列表、引用、代码、表格和业务语义。公共编辑执行器仍负责验证快照哈希、范围、预期原文与冲突。
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Exact physical-line ranges shared by text cleaning modifiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PhysicalLine:
|
||||
"""A content range and its optional line ending in one source string."""
|
||||
|
||||
content_start: int
|
||||
content_end: int
|
||||
full_end: int
|
||||
|
||||
def content(self, markdown: str) -> str:
|
||||
return markdown[self.content_start : self.content_end]
|
||||
|
||||
def line_ending(self, markdown: str) -> str:
|
||||
return markdown[self.content_end : self.full_end]
|
||||
|
||||
def is_blank(self, markdown: str) -> bool:
|
||||
return self.content_start == self.content_end
|
||||
|
||||
|
||||
def iter_physical_lines(markdown: str) -> Iterator[PhysicalLine]:
|
||||
"""Yield exact physical lines without normalizing CR or LF endings."""
|
||||
line_start = 0
|
||||
position = 0
|
||||
while position < len(markdown):
|
||||
character = markdown[position]
|
||||
if character == "\n":
|
||||
yield PhysicalLine(line_start, position, position + 1)
|
||||
position += 1
|
||||
line_start = position
|
||||
elif character == "\r":
|
||||
line_end = position + 2 if position + 1 < len(markdown) and markdown[position + 1] == "\n" else position + 1
|
||||
yield PhysicalLine(line_start, position, line_end)
|
||||
position = line_end
|
||||
line_start = position
|
||||
else:
|
||||
position += 1
|
||||
|
||||
if line_start < len(markdown):
|
||||
yield PhysicalLine(line_start, len(markdown), len(markdown))
|
||||
|
||||
|
||||
def physical_lines(markdown: str) -> tuple[PhysicalLine, ...]:
|
||||
"""Return all physical-line ranges as an immutable sequence."""
|
||||
return tuple(iter_physical_lines(markdown))
|
||||
|
||||
|
||||
def line_ending_styles(markdown: str) -> frozenset[str]:
|
||||
"""Return the distinct non-empty physical line endings in a string."""
|
||||
return frozenset(
|
||||
line.line_ending(markdown)
|
||||
for line in iter_physical_lines(markdown)
|
||||
if line.full_end > line.content_end
|
||||
)
|
||||
@@ -3,9 +3,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from mdpolish._html_table import strict_html_tables
|
||||
from mdpolish._text_ranges import line_ending_styles
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier
|
||||
from mdpolish.text_ranges import line_ending_styles
|
||||
|
||||
_REASON = "展开严格单行 HTML 表格的行布局"
|
||||
_APPLICABILITY = (
|
||||
|
||||
@@ -48,9 +48,9 @@ from importlib.metadata import version as package_version
|
||||
from itertools import pairwise
|
||||
from typing import Protocol, TypeAlias
|
||||
|
||||
from mdpolish._text_ranges import PhysicalLine, physical_lines
|
||||
from mdpolish.models import DocumentSnapshot, ProposedChange, TextEdit, TextSpan
|
||||
from mdpolish.modifier import Modifier, ModifierContractError
|
||||
from mdpolish.text_ranges import PhysicalLine, physical_lines
|
||||
|
||||
try:
|
||||
from spellchecker import SpellChecker as _SpellChecker # type: ignore[import-not-found]
|
||||
@@ -543,7 +543,7 @@ def _scan_lines(markdown: str) -> tuple[_ScannedLine, ...]:
|
||||
blank_run = 0
|
||||
for index, line in enumerate(lines):
|
||||
raw = line.content(markdown)
|
||||
if line.is_blank(markdown):
|
||||
if line.is_empty:
|
||||
blank_run += 1
|
||||
active_heading = None
|
||||
if blank_run > 1:
|
||||
@@ -1143,7 +1143,7 @@ def _boundary_between(
|
||||
if right.physical_index <= left.physical_index:
|
||||
raise ModifierContractError("line join scanner produced non-monotonic physical lines")
|
||||
intervening = lines[left.physical_index + 1 : right.physical_index]
|
||||
if len(intervening) > max_blank_lines or any(not line.is_blank(markdown) for line in intervening):
|
||||
if len(intervening) > max_blank_lines or any(not line.is_empty for line in intervening):
|
||||
return None
|
||||
ending = left.physical.line_ending(markdown)
|
||||
if not ending or any(line.line_ending(markdown) != ending for line in intervening):
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Exact physical-line ranges for project and library modifiers.
|
||||
|
||||
This module scans only CR, LF, and CRLF boundaries. It preserves the source
|
||||
string and reports Python Unicode code point offsets; it does not parse
|
||||
Markdown blocks, normalize line endings, read files, or propose edits.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
__all__ = [
|
||||
"PhysicalLine",
|
||||
"iter_physical_lines",
|
||||
"line_ending_styles",
|
||||
"physical_lines",
|
||||
]
|
||||
|
||||
_LINE_ENDINGS = frozenset(("", "\n", "\r", "\r\n"))
|
||||
|
||||
|
||||
def _require_source(source: object) -> str:
|
||||
if not isinstance(source, str):
|
||||
raise TypeError("source must be a string")
|
||||
return source
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PhysicalLine:
|
||||
"""A source string's exact content range and optional line-ending range."""
|
||||
|
||||
content_start: int
|
||||
content_end: int
|
||||
full_end: int
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for field_name in ("content_start", "content_end", "full_end"):
|
||||
if type(getattr(self, field_name)) is not int:
|
||||
raise TypeError(f"{field_name} must be an integer")
|
||||
if self.content_start < 0 or self.content_end < 0 or self.full_end < 0:
|
||||
raise ValueError("physical line offsets must be non-negative")
|
||||
if not self.content_start <= self.content_end <= self.full_end:
|
||||
raise ValueError("physical line offsets must be ordered")
|
||||
if self.full_end - self.content_end > 2:
|
||||
raise ValueError("physical line ending range must contain at most two code points")
|
||||
|
||||
def _validated_source(self, source: object) -> str:
|
||||
validated = _require_source(source)
|
||||
if self.full_end > len(validated):
|
||||
raise ValueError("physical line range exceeds source length")
|
||||
return validated
|
||||
|
||||
def content(self, source: str) -> str:
|
||||
"""Return this line's content without its physical line ending."""
|
||||
validated = self._validated_source(source)
|
||||
return validated[self.content_start : self.content_end]
|
||||
|
||||
def line_ending(self, source: str) -> str:
|
||||
"""Return this line's exact CR, LF, CRLF, or absent line ending."""
|
||||
validated = self._validated_source(source)
|
||||
ending = validated[self.content_end : self.full_end]
|
||||
if ending not in _LINE_ENDINGS:
|
||||
raise ValueError("physical line ending must be empty, LF, CR, or CRLF")
|
||||
return ending
|
||||
|
||||
@property
|
||||
def is_empty(self) -> bool:
|
||||
"""Return whether the content range has zero length."""
|
||||
return self.content_start == self.content_end
|
||||
|
||||
|
||||
def _iter_physical_lines(source: str) -> Iterator[PhysicalLine]:
|
||||
line_start = 0
|
||||
position = 0
|
||||
while position < len(source):
|
||||
character = source[position]
|
||||
if character == "\n":
|
||||
yield PhysicalLine(line_start, position, position + 1)
|
||||
position += 1
|
||||
line_start = position
|
||||
elif character == "\r":
|
||||
line_end = position + 2 if position + 1 < len(source) and source[position + 1] == "\n" else position + 1
|
||||
yield PhysicalLine(line_start, position, line_end)
|
||||
position = line_end
|
||||
line_start = position
|
||||
else:
|
||||
position += 1
|
||||
|
||||
if line_start < len(source):
|
||||
yield PhysicalLine(line_start, len(source), len(source))
|
||||
|
||||
|
||||
def iter_physical_lines(source: str) -> Iterator[PhysicalLine]:
|
||||
"""Return a lazy iterator over exact physical lines in ``source``."""
|
||||
return _iter_physical_lines(_require_source(source))
|
||||
|
||||
|
||||
def physical_lines(source: str) -> tuple[PhysicalLine, ...]:
|
||||
"""Return all exact physical-line ranges as an immutable sequence."""
|
||||
return tuple(iter_physical_lines(source))
|
||||
|
||||
|
||||
def line_ending_styles(source: str) -> frozenset[str]:
|
||||
"""Return the distinct non-empty physical line endings in ``source``."""
|
||||
validated = _require_source(source)
|
||||
return frozenset(
|
||||
line.line_ending(validated)
|
||||
for line in iter_physical_lines(validated)
|
||||
if line.full_end > line.content_end
|
||||
)
|
||||
@@ -362,7 +362,7 @@ def test_empty_document_and_empty_rule_set_are_no_ops() -> None:
|
||||
|
||||
|
||||
def test_installed_package_version_matches_delivery_candidate() -> None:
|
||||
assert distribution_version("mdpolish") == "0.4.0"
|
||||
assert distribution_version("mdpolish") == "0.6.0"
|
||||
|
||||
|
||||
def test_parameters_preserve_rule_order_and_record_all_options() -> None:
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import FrozenInstanceError
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
from mdpolish.text_ranges import PhysicalLine, iter_physical_lines, line_ending_styles, physical_lines
|
||||
|
||||
|
||||
def _offsets(lines: tuple[PhysicalLine, ...]) -> tuple[tuple[int, int, int], ...]:
|
||||
return tuple((line.content_start, line.content_end, line.full_end) for line in lines)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("source", "expected"),
|
||||
[
|
||||
("", ()),
|
||||
("a", ((0, 1, 1),)),
|
||||
("a\n", ((0, 1, 2),)),
|
||||
("\n", ((0, 0, 1),)),
|
||||
("\r\n", ((0, 0, 2),)),
|
||||
("a\n\n", ((0, 1, 2), (2, 2, 3))),
|
||||
("\ntext", ((0, 0, 1), (1, 5, 5))),
|
||||
("a\r\nb\rc\n", ((0, 1, 3), (3, 4, 5), (5, 6, 7))),
|
||||
("a\u2028b", ((0, 3, 3),)),
|
||||
],
|
||||
)
|
||||
def test_scans_exact_physical_line_offsets(
|
||||
source: str,
|
||||
expected: tuple[tuple[int, int, int], ...],
|
||||
) -> None:
|
||||
assert _offsets(physical_lines(source)) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"source",
|
||||
[
|
||||
"",
|
||||
"one line",
|
||||
"\nleading empty",
|
||||
"a\n\n",
|
||||
"a\r\nb\rc\n",
|
||||
"\ufeff中文😀e\u0301\u2028\x00\t\f",
|
||||
],
|
||||
)
|
||||
def test_ranges_reconstruct_source_without_normalization(source: str) -> None:
|
||||
lines = physical_lines(source)
|
||||
|
||||
assert "".join(source[line.content_start : line.full_end] for line in lines) == source
|
||||
assert lines == tuple(iter_physical_lines(source))
|
||||
assert lines == physical_lines(source)
|
||||
for line in lines:
|
||||
assert line.content(source) == source[line.content_start : line.content_end]
|
||||
assert line.line_ending(source) == source[line.content_end : line.full_end]
|
||||
|
||||
|
||||
def test_unicode_separators_and_control_characters_remain_content() -> None:
|
||||
source = "\ufeff中文😀e\u0301\u2028\u2029\x00\t\f"
|
||||
line = physical_lines(source)[0]
|
||||
|
||||
assert (line.content_start, line.content_end, line.full_end) == (0, len(source), len(source))
|
||||
assert line.content(source) == source
|
||||
assert line.line_ending(source) == ""
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("source", "expected"),
|
||||
[
|
||||
("", frozenset()),
|
||||
("no ending", frozenset()),
|
||||
("a\nb\n", frozenset(("\n",))),
|
||||
("a\r\nb\r\n", frozenset(("\r\n",))),
|
||||
("a\rb\r", frozenset(("\r",))),
|
||||
("a\nb\r\nc\r", frozenset(("\n", "\r\n", "\r"))),
|
||||
],
|
||||
)
|
||||
def test_reports_only_line_endings_present_in_source(source: str, expected: frozenset[str]) -> None:
|
||||
assert line_ending_styles(source) == expected
|
||||
|
||||
|
||||
def test_empty_means_zero_length_content_not_whitespace() -> None:
|
||||
empty, space, tab = physical_lines("\n \n\t\n")
|
||||
|
||||
assert empty.is_empty
|
||||
assert not space.is_empty
|
||||
assert not tab.is_empty
|
||||
|
||||
|
||||
def test_physical_line_is_frozen_and_slotted() -> None:
|
||||
line = PhysicalLine(0, 1, 1)
|
||||
mutable_view = cast(Any, line)
|
||||
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
mutable_view.content_start = 1
|
||||
assert not hasattr(line, "__dict__")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field_index", [0, 1, 2])
|
||||
@pytest.mark.parametrize("invalid", [True, 1.0, "1", None])
|
||||
def test_rejects_non_integer_offsets(field_index: int, invalid: object) -> None:
|
||||
values = [0, 0, 0]
|
||||
values[field_index] = cast(int, invalid)
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
PhysicalLine(*values)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"offsets",
|
||||
[
|
||||
(-1, 0, 0),
|
||||
(0, -1, 0),
|
||||
(0, 0, -1),
|
||||
(1, 0, 1),
|
||||
(0, 2, 1),
|
||||
(0, 0, 3),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_offset_ranges(offsets: tuple[int, int, int]) -> None:
|
||||
with pytest.raises(ValueError):
|
||||
PhysicalLine(*offsets)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"scanner",
|
||||
[iter_physical_lines, physical_lines, line_ending_styles],
|
||||
)
|
||||
@pytest.mark.parametrize("invalid", [b"text", Path("text.md"), None, 1])
|
||||
def test_scanners_reject_non_string_sources_immediately(
|
||||
scanner: Callable[[str], object],
|
||||
invalid: object,
|
||||
) -> None:
|
||||
with pytest.raises(TypeError, match="source must be a string"):
|
||||
scanner(cast(str, invalid))
|
||||
|
||||
|
||||
def test_iter_scanner_validates_before_iteration_begins() -> None:
|
||||
with pytest.raises(TypeError, match="source must be a string"):
|
||||
iter_physical_lines(cast(str, b"deferred validation would be wrong"))
|
||||
|
||||
|
||||
def test_accessors_reject_non_string_or_short_sources() -> None:
|
||||
line = PhysicalLine(0, 1, 2)
|
||||
|
||||
with pytest.raises(TypeError, match="source must be a string"):
|
||||
line.content(cast(str, b"a\n"))
|
||||
with pytest.raises(TypeError, match="source must be a string"):
|
||||
line.line_ending(cast(str, None))
|
||||
with pytest.raises(ValueError, match="exceeds source length"):
|
||||
line.content("a")
|
||||
with pytest.raises(ValueError, match="exceeds source length"):
|
||||
line.line_ending("a")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("line", "source"),
|
||||
[
|
||||
(PhysicalLine(0, 1, 2), "ab"),
|
||||
(PhysicalLine(0, 0, 2), "\n\r"),
|
||||
],
|
||||
)
|
||||
def test_line_ending_rejects_non_physical_slices(line: PhysicalLine, source: str) -> None:
|
||||
with pytest.raises(ValueError, match="empty, LF, CR, or CRLF"):
|
||||
line.line_ending(source)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("source", ["a", "a\n", "a\r", "a\r\n"])
|
||||
def test_line_ending_accepts_every_supported_slice(source: str) -> None:
|
||||
line = physical_lines(source)[0]
|
||||
|
||||
assert line.line_ending(source) in {"", "\n", "\r", "\r\n"}
|
||||
Reference in New Issue
Block a user