test: keep gateway-dependent e2e out of the commit gate

The pre-commit hook runs the whole suite, and tests/e2e/ talks to a real
LLM gateway, so whether a commit is allowed depended on how fast that
gateway happened to be. During the issue 14 work it blocked two commits
on two different cases; both passed when rerun alone, and the suite went
from 165s to 336s that hour.

The wasted minutes are not the real cost. Retrying on red teaches you to
read "test failed" as "gateway was slow", and a genuinely flaky bug then
gets retried away too. An alarm that cries wolf stops being an alarm.

test_thinking_live.py already carried the slow marker; the other three
files now match it, and the release checklist gains an explicit
`pytest -m slow` step so they still run where a human is watching --
without that step this change would just delete the coverage.

Also raises test_flat_legacy_keys_assemble's LLM_TIMEOUT from 120 to
300, matching .env. At 120 the case allowed half of what production
allows, on a gateway that needs the full 300 -- it measured 116s in a
solo run. The assertion is that the flat key name parses into
SourceConfig.timeout_s; the value itself was never under test.
This commit is contained in:
2026-08-20 05:49:28 -04:00
parent f31f7caf99
commit 620b426ede
4 changed files with 34 additions and 11 deletions
+2 -1
View File
@@ -91,7 +91,7 @@ make ci # 只读验证(check + test)
| 1 | **更新 README** | 打包会把当时的 README 固化进 sdist,**发布后再改就来不及了**(包里那份永远是旧的)。逐项核对: 安装命令的版本约束(`==1.1.*` 这类**极易漏改**,漏了下游就被锁在旧版)、能力表是否覆盖新行为、数字型断言是否仍成立(如遥测字段数,须用 `inspect.signature` 实测而非凭记忆) | | 1 | **更新 README** | 打包会把当时的 README 固化进 sdist,**发布后再改就来不及了**(包里那份永远是旧的)。逐项核对: 安装命令的版本约束(`==1.1.*` 这类**极易漏改**,漏了下游就被锁在旧版)、能力表是否覆盖新行为、数字型断言是否仍成立(如遥测字段数,须用 `inspect.signature` 实测而非凭记忆) |
| 2 | CHANGELOG 定版 | "未发布" → `## X.Y.Z(日期)` | | 2 | CHANGELOG 定版 | "未发布" → `## X.Y.Z(日期)` |
| 3 | 版本号 | `pyproject.toml` + `src/polygateway/__init__.py` 两处必须一致 | | 3 | 版本号 | `pyproject.toml` + `src/polygateway/__init__.py` 两处必须一致 |
| 4 | 合并 main + push | `--no-ff`;合并后在 main 上重跑 `make lint` 与全套件 | | 4 | 合并 main + push | `--no-ff`;合并后在 main 上重跑 `make lint` 与全套件,**外加 `pytest -m slow`** ——真实网关 e2e 与 Redis 时间语义变体被 `addopts = "-m 'not slow'"` 默认排除,**不显式跑就等于没跑**(约 20-40 分钟,取决于网关快慢)。它们不进日常提交是有意的: pre-commit 关卡跑全套件,网关一抖就挡住与之无关的提交,久了会把"测试红了先怀疑网关"变成惯性,真 bug 也会被当成抖动重试掉;代价是这道门必须由本清单兜住 |
| 5 | **打 tag 并 push** | `git tag -a vX.Y.Z -m "..."` + `git push origin vX.Y.Z`。历史上多个版本漏打 | | 5 | **打 tag 并 push** | `git tag -a vX.Y.Z -m "..."` + `git push origin vX.Y.Z`。历史上多个版本漏打 |
| 6 | 构建 | `rm -rf dist && python -m build && python -m twine check dist/*` | | 6 | 构建 | `rm -rf dist && python -m build && python -m twine check dist/*` |
| 7 | **上传 registry** | 凭据在 `~/.config/tea/config.yml`(tea CLI 的 Gitea token,**不在** `~/.pypirc`);token 走 `TWINE_PASSWORD` 环境变量,不进命令行<br>`TWINE_USERNAME=iomgaa TWINE_PASSWORD=$TOKEN python -m twine upload --repository-url https://gitea.iomgaa.online/api/packages/iomgaa/pypi dist/*` | | 7 | **上传 registry** | 凭据在 `~/.config/tea/config.yml`(tea CLI 的 Gitea token,**不在** `~/.pypirc`);token 走 `TWINE_PASSWORD` 环境变量,不进命令行<br>`TWINE_USERNAME=iomgaa TWINE_PASSWORD=$TOKEN python -m twine upload --repository-url https://gitea.iomgaa.online/api/packages/iomgaa/pypi dist/*` |
@@ -113,6 +113,7 @@ Gitea 包 registry 是 **owner 级**(`/iomgaa/-/packages/`)不是仓库级;PyPI
- 覆盖率目标 80%;并发/韧性行为是一等测试对象: 重试穿透取消、熔断开路半开、限流结算退款、Redis 掉线降级方向、缓存 key 隔离。 - 覆盖率目标 80%;并发/韧性行为是一等测试对象: 重试穿透取消、熔断开路半开、限流结算退款、Redis 掉线降级方向、缓存 key 隔离。
- Redis 相关测试用真实 Redis(integration),不 mock Lua 行为;限流契约测试随实现一起交付(参考 CHSAnalyzer `tests/contracts_limiter.py`)。 - Redis 相关测试用真实 Redis(integration),不 mock Lua 行为;限流契约测试随实现一起交付(参考 CHSAnalyzer `tests/contracts_limiter.py`)。
- 涉及真实 LLM 的测试输出结构化 Markdown 至 `tests/outputs/<module>/<test>_<ts>.md` - 涉及真实 LLM 的测试输出结构化 Markdown 至 `tests/outputs/<module>/<test>_<ts>.md`
- **成败取决于外部服务当下状态的测试一律标 `slow`**(`tests/e2e/` 四个文件与 Redis 时间语义变体):它们默认不进日常套件,由发布清单第 4 步统一跑。判据是"重跑一次可能就绿了"——这种测试留在提交关卡里会污染信号。同理,给它们的超时不得紧于 `.env` 的生产配置,否则是设计上就会间歇红。
## 5. 项目结构 ## 5. 项目结构
+13 -3
View File
@@ -18,9 +18,16 @@ _REPO = Path(__file__).resolve().parents[2]
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None} _ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV) _HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
pytestmark = pytest.mark.skipif( # 真实网关调用: 与 test_thinking_live.py 同待遇标 slow(pytest addopts 默认排除,
# 显式 `pytest -m slow` 运行)。理由是这些用例的成败取决于网关此刻快不快,而
# pre-commit 关卡跑全套件——网关一抖就挡住与之无关的提交,久了会把"测试红了
# 先怀疑网关"变成惯性,真 bug 也会被当成抖动重试掉。发版清单负责让它们真跑。
pytestmark = [
pytest.mark.slow,
pytest.mark.skipif(
not _HAS_SOURCE, reason="需真实网关凭据: 在 .env 配置 LLM__{PROVIDER}__1__*" not _HAS_SOURCE, reason="需真实网关凭据: 在 .env 配置 LLM__{PROVIDER}__1__*"
) ),
]
@pytest.fixture @pytest.fixture
@@ -69,7 +76,10 @@ class TestVideoTreeOnboarding:
source_keys = {k: v for k, v in _ENV.items() if k.split("__")[0] == "LLM" and "__" in k} source_keys = {k: v for k, v in _ENV.items() if k.split("__")[0] == "LLM" and "__" in k}
flat_env = { flat_env = {
**source_keys, **source_keys,
"LLM_TIMEOUT": "120", # 与 .env 的 LLM__MINIMAX__1__TIMEOUT_S 同值。取 120(VT 旧值)会让本用例的
# 超时比生产配置还紧一半,在慢网关上必然间歇红——而本用例断言的是平铺
# 键名能否解析成 SourceConfig.timeout_s,超时取值本身不是被测对象
"LLM_TIMEOUT": "300",
"LLM_MAX_RETRIES": "3", "LLM_MAX_RETRIES": "3",
"LLM_RETRY_BASE_DELAY": "2.0", "LLM_RETRY_BASE_DELAY": "2.0",
"LLM_RETRY_MAX_DELAY": "30.0", "LLM_RETRY_MAX_DELAY": "30.0",
+9 -3
View File
@@ -21,9 +21,15 @@ from polygateway.types import SourceConfig
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None} _ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
pytestmark = pytest.mark.skipif( # 真实网关调用: 与 test_thinking_live.py 同待遇标 slow(pytest addopts 默认排除,
"LLM__MINIMAX__1__BASE_URL" not in _ENV, reason="缺真实网关配置(.env)" # 显式 `pytest -m slow` 运行)。理由见 test_compat_projects.py 同处注释。
) pytestmark = [
pytest.mark.slow,
pytest.mark.skipif(
"LLM__MINIMAX__1__BASE_URL" not in _ENV,
reason="缺真实网关配置(.env)",
),
]
_OUT = Path("tests/outputs/embedding") _OUT = Path("tests/outputs/embedding")
+9 -3
View File
@@ -19,9 +19,15 @@ from polygateway import GatewayClient
_ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None} _ENV = {k: v for k, v in {**dotenv_values(".env"), **os.environ}.items() if v is not None}
_HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV) _HAS_SOURCE = any(k.split("__")[0] == "LLM" and k.endswith("__API_KEY") for k in _ENV)
pytestmark = pytest.mark.skipif( # 真实网关调用: 与 test_thinking_live.py 同待遇标 slow(pytest addopts 默认排除,
not _HAS_SOURCE, reason="需真实网关凭据: 在 .env 配置 LLM__{PROVIDER}__1__*(M1 验收前必须真跑)" # 显式 `pytest -m slow` 运行)。理由见 test_compat_projects.py 同处注释。
) pytestmark = [
pytest.mark.slow,
pytest.mark.skipif(
not _HAS_SOURCE,
reason="需真实网关凭据: 在 .env 配置 LLM__{PROVIDER}__1__*(M1 验收前必须真跑)",
),
]
_OUT_DIR = Path("tests/outputs/e2e") _OUT_DIR = Path("tests/outputs/e2e")