Skip to content

Commit 63349f5

Browse files
Claptarclaude
andcommitted
Release 0.6.0
Adds `adata convert`, and fixes four quadratic paths that made `concat` and `split` appear to hang on real data. Three of the four were found by the new complexity guards rather than by a user; the first was reported from a pipeline that killed twelve tasks after 98 minutes. A minor rather than a patch: `convert` is a new command, and the benchmark and guards are new machinery. Nothing in the existing surface changed behaviour, except that `--merge drop` is now accepted where it was previously rejected despite being the documented default. Also folds the two `### Added` sections that had accumulated under Unreleased into one, and moves the two fixes that had been filed under Added into Fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 7b58432 commit 63349f5

3 files changed

Lines changed: 36 additions & 26 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,12 @@
33
Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
44
`v` prefix.
55

6-
## Unreleased
6+
## 0.6.0
7+
8+
Adds `adata convert`, and fixes four quadratic paths that made `concat` and
9+
`split` appear to hang on real data. Three of the four were found by a new
10+
suite of complexity guards, which now run on every pull request; the first
11+
was reported from a pipeline that had to kill twelve tasks after 98 minutes.
712

813
### Added
914

@@ -37,6 +42,30 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
3742
matrix encoding. The check itself is not new, but nothing tested it and it
3843
could not suggest a fix, because there was none.
3944

45+
- **Complexity guards in the test suite** (`tests/test_performance.py`).
46+
Cost regressions now fail at merge time. They count operations rather than
47+
seconds -- h5py and zarr reads, Zarr store traffic, Python allocation and
48+
executed lines -- and assert that successive increments grow no faster than
49+
linearly, so nothing here can fail because a CI runner was busy. See
50+
[docs/TESTING.md](docs/TESTING.md#performance). Every subcommand is covered:
51+
`ls`, `view`, `create`, all five `export` and all five `import` variants,
52+
`split` on both axes, and the `concat` options nothing else reached.
53+
Two claims are now enforced rather than described -- `view` and `ls` read
54+
**zero** data elements at any store size, and streaming stays far below the
55+
input curve at a fixed `--chunk`.
56+
57+
- **A comparative benchmark** (`benchmarks/`), run on every tag against
58+
anndata and against scanpy where scanpy has a real equivalent. Reports peak
59+
RSS, wall time and output size; publishes to
60+
[docs/BENCHMARKS.md](docs/BENCHMARKS.md) and the release notes. Report-only
61+
-- it never fails a build. Fifteen cases, covering every command with a real
62+
baseline, including `h5ls -r` for `ls` and the rows where adata-cli is the
63+
slower of the two.
64+
65+
- **`--merge drop` and `--uns-merge drop` are accepted.** `drop` was already
66+
the documented default behaviour but was rejected as a value, so a config
67+
could not state it explicitly.
68+
4069
### Fixed
4170

4271
- **`concat --merge` never finished on a real store.** Aligning a var column
@@ -47,34 +76,24 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
4776
of 13 pipeline tasks had to be killed after 98 minutes. The column is now
4877
read once per input, and `--merge first` / `--merge only`, which decide on
4978
presence alone, read no column values at all.
79+
5080
- **`concat` was quadratic in the number of categories** in an obs column.
5181
Category merging probed a list rather than a dict: 2,096,128 string
5282
comparisons to union 1,024 categories, and around 5e9 for a 100k-category
5383
column. Found by the new guards.
84+
5485
- **`split --by` was quadratic**, O(n_rows x n_groups). `group_indices` grouped
5586
rows with `np.nonzero(values == label)` inside a loop over distinct labels,
5687
rescanning each chunk once per label: at 4,096 rows, 16,384 elements scanned
5788
for 4 groups and 1,048,576 for 256. A million cells split by a thousand
5889
samples is ~10^9 comparisons. One `np.unique` pass per chunk makes it flat
5990
in the group count. Found by the new guards; order of first appearance,
6091
which names the output files, is unchanged.
92+
6193
- **`concat` built a Python object per row** for nullable and string obs
6294
columns, then walked the list twice more. Filling a typed buffer by slice
6395
removes three full passes over every such column.
6496

65-
### Added
66-
67-
- **Complexity guards in the test suite** (`tests/test_performance.py`).
68-
Cost regressions now fail at merge time. They count operations rather than
69-
seconds -- h5py and zarr reads, Zarr store traffic, Python allocation and
70-
executed lines -- and assert that successive increments grow no faster than
71-
linearly, so nothing here can fail because a CI runner was busy. See
72-
[docs/TESTING.md](docs/TESTING.md#performance). Every subcommand is covered:
73-
`ls`, `view`, `create`, all five `export` and all five `import` variants,
74-
`split` on both axes, and the `concat` options nothing else reached.
75-
Two claims are now enforced rather than described -- `view` and `ls` read
76-
**zero** data elements at any store size, and streaming stays far below the
77-
input curve at a fixed `--chunk`.
7897
- **Copying variable-length strings ignored its own read budget.** The width
7998
of a vlen element was assumed to be 64 bytes, because h5py reports the
8099
itemsize of a pointer, so the step was the same 524,288 elements whatever
@@ -84,6 +103,7 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
84103
sampled from the first 256 elements. Reported by an automated review on
85104
PR #14 and confirmed by measurement; `uns` can hold arbitrary text, so this
86105
was not a width the layer could assume.
106+
87107
- **Peak RSS in the benchmark was floored by the runner's own memory on Linux.**
88108
A forked child inherits its parent's resident pages and `execve` folds that
89109
into the `maxrss` the kernel reports, so every contender would have measured
@@ -93,16 +113,6 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
93113
child goes from 326 MB to 8 MB. Caught by `test_benchmark_harness.py`, which
94114
exists for exactly this. The published figures were measured on macOS, which
95115
resets the high-water mark at exec, and are unchanged.
96-
- **A comparative benchmark** (`benchmarks/`), run on every tag against
97-
anndata and against scanpy where scanpy has a real equivalent. Reports peak
98-
RSS, wall time and output size; publishes to
99-
[docs/BENCHMARKS.md](docs/BENCHMARKS.md) and the release notes. Report-only
100-
-- it never fails a build. Fifteen cases, covering every command with a real
101-
baseline, including `h5ls -r` for `ls` and the rows where adata-cli is the
102-
slower of the two.
103-
- **`--merge drop` and `--uns-merge drop` are accepted.** `drop` was already
104-
the documented default behaviour but was rejected as a value, so a config
105-
could not state it explicitly.
106116

107117
## 0.5.1
108118

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# so the distribution is published as `pyadata-cli`. The import package
44
# and the command are both still `adata`.
55
name = "pyadata-cli"
6-
version = "0.5.1"
6+
version = "0.6.0"
77
description = "Streaming CLI for exploring and editing large AnnData .h5ad and .zarr stores"
88
readme = "README.md"
99
requires-python = ">=3.12"

‎uv.lock‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)