Commit a1cf2cd
fix(deps): declare packaging, numpy, pygments and xxhash as runtime dependencies (#2268)
## The bug
`python/xorq/ibis_yaml/packager.py:40` imports `packaging` at module
level, but `packaging` was never declared in `[project].dependencies`.
It reaches every development environment transitively — `pytest`,
`black`, `matplotlib` and `snowflake-connector-python` all depend on it
— so nothing local ever noticed.
Installing only the declared dependencies fails:
```console
$ uv tool run --isolated --python 3.12 --with xorq==0.4.0 xorq run builds/f68b0e436e96
File ".../xorq/ibis_yaml/packager.py", line 40, in <module>
from packaging.specifiers import SpecifierSet
ModuleNotFoundError: No module named 'packaging'
```
`xorq build` survives, because it doesn't reach `packager` on that path.
`xorq run`, `xorq catalog` and the TUI (`catalog/tui.py:82`) are dead on
a clean install, as is `backends/pandas/executor.py`.
**This affects 0.4.0 as published, so it needs a patch release, not just
a fix on main.**
## The fix
Declared `packaging`. Auditing the rest of the always-loaded code
surfaced three more module-level imports of undeclared distributions,
arriving via `pandas`, `rich`/`textual` and `xorq-dasher`:
| import | reached today via | declared range of the provider |
| --- | --- | --- |
| `numpy` | `pandas` | `pandas>=2.2.3,<3` |
| `pygments` | `rich`, `textual` | `rich>=13.9.4` |
| `xxhash` | `xorq-dasher` | `xorq-dasher>=0.1.1` |
Every one of those is an open-ended range in someone else's metadata.
Declared all three rather than depend on them.
### Correction: declaring is not free under `--resolution lowest-direct`
My first pass claimed this "adds no install surface". That was wrong,
and `ci-test-lowest-direct (3.13)` caught it. Floor-lowering applies to
**direct** dependencies, so promoting `numpy` and `xxhash` from
transitive to declared newly subjected them to the floor lock, where a
bare `>=` selects a release predating the interpreter:
```
numpy==1.26.0 # no cp313 wheel -> sdist -> meson: "No BLAS library detected!"
xxhash==3.0.0 # no cp313 wheel either; queued to fail right after numpy
```
Previously `pandas`' own per-interpreter numpy floors happened to keep
this installable. Floors are now the first release shipping wheels for
each supported interpreter, following the existing precedent for
`matplotlib` and `scikit-learn` in the `examples` extra:
| | 3.10 | 3.11 | 3.12 | 3.13 |
| --- | --- | --- | --- | --- |
| `numpy` | 1.22.4 | 1.23.2 | 1.26.0 | 2.1.0 |
| `xxhash` | 3.0 | 3.2 | 3.4 | 3.5 |
Gated rather than raised outright, so numpy 1.x stays usable on older
interpreters — xorq only touches `numpy.ndarray` and `numpy.dtype`, so
forcing 2.1 everywhere would constrain downstreams for nothing.
`pygments` is pure python and needs no gate.
Verified with `uv pip compile --resolution lowest-direct` for each of
3.10–3.13, cross-checking every resolved version against the PyPI file
list — all twelve now ship a usable wheel.
Everything else that turned up in the audit is either extras-gated
(`pyiceberg`, `adbc_*`, `snowflake`, `databricks`, `sklearn`, `gcsfs`)
or `try`/`except ImportError`-guarded (`regex` in
`backends/pandas/kernels.py:10`, `importlib_metadata` in
`__init__.py:1`).
## Tests
### `python/xorq/tests/test_declared_dependencies.py` — static, `core`
marker
AST-scans **all of `python/xorq`**, minus twelve extras-gated modules
named by path, for third-party imports at module scope and asserts each
is declared. Scoping by exclusion rather than inclusion means a new
module anywhere in xorq is guarded by default;
`test_extras_gated_modules_are_all_still_needed` stops the exclusion
list becoming a place to park problems. Walking `tree.body` rather than
`ast.walk` is what makes it precise: imports nested in `try`/`except
ImportError`, `if` blocks or function bodies are guarded or deferred on
purpose, so their absence is already handled. That classification is
itself parametrized and tested.
**There is deliberately no allowlist** for "it arrives via some other
dependency":
- An allowlist encodes the *same* inference that shipped this bug.
`packaging` was already an unwritten allowlist entry — *"pytest pulls
it, we're fine."* True in every dev env, false in every user env.
Writing it down makes it auditable, not true.
- It asserts a fact that isn't ours to hold. "pygments comes via rich"
is a claim about rich's metadata under an open range; rich can drop it
and nothing in this repo changes.
- A guard test over `uv.lock` doesn't fix that. The lock is one dev
resolution at one moment, while users resolve fresh from PyPI. And
*reachable ≠ present*: `pandas` lists `numpy` twice under split
`python_version` markers, so a naive graph walk reports "covered" for an
edge whose marker is false on the user's interpreter.
- The cost is upside down — lock parsing plus marker evaluation,
maintained forever, to avoid writing three lines.
Since there's no exception mechanism, there's no configuration surface
to argue about: an undeclared module-level import is a hard failure with
a one-line fix.
Runs in the `core` job and in `ci-test-lowest-direct`, which also
exercises the new floors under `--resolution lowest-direct`.
### `python/xorq/tests/test_bare_install.py` — end-to-end,
`bare_install` marker
Builds the wheel and drives the CLI through `uv tool run --isolated`,
which installs `[project].dependencies` and nothing else. This covers
what the static scan structurally cannot: most packager imports in
`cli.py` are lazy, inside function bodies (`# noqa: PLC0415`).
It asserts a **build → run round trip**, not just `--help`, because that
distinction is the whole bug.
## Verified both tests fail before the fix
Deleting `"packaging>=22"` from `pyproject.toml` and re-running:
```
drop packaging -> 3 failed, 19 passed # static scan
-> 2 failed, 5 passed # bare install
drop numpy -> 2 failed, 20 passed
drop xxhash -> 2 failed, 20 passed
drop pygments -> 2 failed, 20 passed
```
with the original traceback reproduced through `cli.py:348 ->
packager.py:40`. Note what still passes on a `packaging` drop: **every
`--help` test stayed green** — which is why the round trip, not
`--help`, is the thing worth asserting.
### What each layer actually guards
Only `packaging` is caught by *both* layers. Removing `numpy`,
`pygments` or `xxhash` leaves all bare-install tests green, because each
still arrives transitively via `pandas`, `rich`/`textual` and
`xorq-dasher`. The bare-install layer exercises those import paths;
**the static scan is what guards the declarations.** An earlier revision
of this PR overstated that, and the docstring now says it plainly.
Restored, all 29 pass.
## CI
`ci-test-library` and `ci-test-install` already installed the wheel into
a bare environment — but only smoke-imported `xorq` itself, which
succeeds even when broken. Worse, `ci-test-library`'s pytest step runs
`--with pytest --with pytest-cov`, and **pytest depends on
`packaging`**, so that environment is contaminated against exactly this
class of bug.
Both workflows now import the modules on the build, run and catalog
paths in the bare environment, with no pytest present, on wheel and
sdist. `ci-test-install` covers that across 3.10–3.13 and three
operating systems.
The module list lives in one place —
`python/xorq/tests/bare_install_modules.txt`, driven by
`check_bare_imports.py` — read by the test and by all four workflow
steps, rather than being repeated as five hand-maintained `python -c
"import …"` one-liners.
The `bare_install` tests also needed a step of their own: `ci-test.yml`
filters on the backend matrix name, `ci-test-library` on `"library or
xorq"`, and every other workflow names explicit paths — so nothing
selected that marker and the round trip ran only locally. Now wired into
the `core` matrix entry, which already has uv and a synced project; the
test builds its own wheel and spawns its own isolated environment, so
the surrounding dev/test groups don't contaminate what's under test.
## Known limitations
Stated rather than left to be discovered:
- **`vendor/` is outside the static scan.** Its module-level imports are
upstream ibis's and re-vendoring would churn any exclusion list kept
here. Not a hole: `xorq.api` imports `vendor/ibis`, so the bare-install
layer catches an undeclared import there that is genuinely absent from a
bare install.
- **The exclusion-minimality guard only catches import roots that match
their distribution name.** An aliased one (`sklearn` → `scikit-learn`)
would not be flagged as stale, because pre-registering that alias
collides with `test_import_root_mapping_has_no_stale_entries`.
- **The twelve exclusions are verified non-stale, not verified
extras-gated.** Several of those distributions aren't declared in any
extra — pre-existing, and out of scope here.
- **The `>= '3.13'` floors are open upward**, safe only while
`requires-python` caps at `<3.14`. Noted in `pyproject.toml` where the
next bump will see it.
## Follow-ups
- **0.4.0 on PyPI carries this bug.** Merging fixes `main`; it does not
help anyone already installed. Needs a patch release.
- **#2269** — unrelated, found while diagnosing CI here: Flight replaces
exception messages over ~1–2 KB with an opaque gRPC metadata-size error,
so real failures with chained tracebacks surface as transport errors.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent f7948a8 commit a1cf2cd
9 files changed
Lines changed: 443 additions & 0 deletions
File tree
- .github/workflows
- python/xorq/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
| 114 | + | |
114 | 115 | | |
115 | 116 | | |
116 | 117 | | |
| |||
125 | 126 | | |
126 | 127 | | |
127 | 128 | | |
| 129 | + | |
128 | 130 | | |
129 | 131 | | |
130 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
56 | 58 | | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
60 | 62 | | |
| 63 | + | |
| 64 | + | |
61 | 65 | | |
62 | 66 | | |
63 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
160 | 173 | | |
161 | 174 | | |
162 | 175 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
36 | 52 | | |
37 | 53 | | |
38 | 54 | | |
| |||
324 | 340 | | |
325 | 341 | | |
326 | 342 | | |
| 343 | + | |
327 | 344 | | |
328 | 345 | | |
329 | 346 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
0 commit comments