33Notable changes to ` adata-cli ` . Versions are ` MAJOR.MINOR.PATCH ` ; tags carry no
44` v ` prefix.
55
6- ## Unreleased
6+ ## 0.6.0
7+
8+ Adds ` adata convert ` , and fixes four quadratic paths that made ` concat ` and
9+ ` split ` appear to hang on real data. Three of the four were found by a new
10+ suite of complexity guards, which now run on every pull request; the first
11+ was reported from a pipeline that had to kill twelve tasks after 98 minutes.
712
813### Added
914
@@ -37,6 +42,30 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
3742 matrix encoding. The check itself is not new, but nothing tested it and it
3843 could not suggest a fix, because there was none.
3944
45+ - ** Complexity guards in the test suite** (` tests/test_performance.py ` ).
46+ Cost regressions now fail at merge time. They count operations rather than
47+ seconds -- h5py and zarr reads, Zarr store traffic, Python allocation and
48+ executed lines -- and assert that successive increments grow no faster than
49+ linearly, so nothing here can fail because a CI runner was busy. See
50+ [ docs/TESTING.md] ( docs/TESTING.md#performance ) . Every subcommand is covered:
51+ ` ls ` , ` view ` , ` create ` , all five ` export ` and all five ` import ` variants,
52+ ` split ` on both axes, and the ` concat ` options nothing else reached.
53+ Two claims are now enforced rather than described -- ` view ` and ` ls ` read
54+ ** zero** data elements at any store size, and streaming stays far below the
55+ input curve at a fixed ` --chunk ` .
56+
57+ - ** A comparative benchmark** (` benchmarks/ ` ), run on every tag against
58+ anndata and against scanpy where scanpy has a real equivalent. Reports peak
59+ RSS, wall time and output size; publishes to
60+ [ docs/BENCHMARKS.md] ( docs/BENCHMARKS.md ) and the release notes. Report-only
61+ -- it never fails a build. Fifteen cases, covering every command with a real
62+ baseline, including ` h5ls -r ` for ` ls ` and the rows where adata-cli is the
63+ slower of the two.
64+
65+ - ** ` --merge drop ` and ` --uns-merge drop ` are accepted.** ` drop ` was already
66+ the documented default behaviour but was rejected as a value, so a config
67+ could not state it explicitly.
68+
4069### Fixed
4170
4271- ** ` concat --merge ` never finished on a real store.** Aligning a var column
@@ -47,34 +76,24 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
4776 of 13 pipeline tasks had to be killed after 98 minutes. The column is now
4877 read once per input, and ` --merge first ` / ` --merge only ` , which decide on
4978 presence alone, read no column values at all.
79+
5080- ** ` concat ` was quadratic in the number of categories** in an obs column.
5181 Category merging probed a list rather than a dict: 2,096,128 string
5282 comparisons to union 1,024 categories, and around 5e9 for a 100k-category
5383 column. Found by the new guards.
84+
5485- ** ` split --by ` was quadratic** , O(n_rows x n_groups). ` group_indices ` grouped
5586 rows with ` np.nonzero(values == label) ` inside a loop over distinct labels,
5687 rescanning each chunk once per label: at 4,096 rows, 16,384 elements scanned
5788 for 4 groups and 1,048,576 for 256. A million cells split by a thousand
5889 samples is ~ 10^9 comparisons. One ` np.unique ` pass per chunk makes it flat
5990 in the group count. Found by the new guards; order of first appearance,
6091 which names the output files, is unchanged.
92+
6193- ** ` concat ` built a Python object per row** for nullable and string obs
6294 columns, then walked the list twice more. Filling a typed buffer by slice
6395 removes three full passes over every such column.
6496
65- ### Added
66-
67- - ** Complexity guards in the test suite** (` tests/test_performance.py ` ).
68- Cost regressions now fail at merge time. They count operations rather than
69- seconds -- h5py and zarr reads, Zarr store traffic, Python allocation and
70- executed lines -- and assert that successive increments grow no faster than
71- linearly, so nothing here can fail because a CI runner was busy. See
72- [ docs/TESTING.md] ( docs/TESTING.md#performance ) . Every subcommand is covered:
73- ` ls ` , ` view ` , ` create ` , all five ` export ` and all five ` import ` variants,
74- ` split ` on both axes, and the ` concat ` options nothing else reached.
75- Two claims are now enforced rather than described -- ` view ` and ` ls ` read
76- ** zero** data elements at any store size, and streaming stays far below the
77- input curve at a fixed ` --chunk ` .
7897- ** Copying variable-length strings ignored its own read budget.** The width
7998 of a vlen element was assumed to be 64 bytes, because h5py reports the
8099 itemsize of a pointer, so the step was the same 524,288 elements whatever
@@ -84,6 +103,7 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
84103 sampled from the first 256 elements. Reported by an automated review on
85104 PR #14 and confirmed by measurement; ` uns ` can hold arbitrary text, so this
86105 was not a width the layer could assume.
106+
87107- ** Peak RSS in the benchmark was floored by the runner's own memory on Linux.**
88108 A forked child inherits its parent's resident pages and ` execve ` folds that
89109 into the ` maxrss ` the kernel reports, so every contender would have measured
@@ -93,16 +113,6 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
93113 child goes from 326 MB to 8 MB. Caught by ` test_benchmark_harness.py ` , which
94114 exists for exactly this. The published figures were measured on macOS, which
95115 resets the high-water mark at exec, and are unchanged.
96- - ** A comparative benchmark** (` benchmarks/ ` ), run on every tag against
97- anndata and against scanpy where scanpy has a real equivalent. Reports peak
98- RSS, wall time and output size; publishes to
99- [ docs/BENCHMARKS.md] ( docs/BENCHMARKS.md ) and the release notes. Report-only
100- -- it never fails a build. Fifteen cases, covering every command with a real
101- baseline, including ` h5ls -r ` for ` ls ` and the rows where adata-cli is the
102- slower of the two.
103- - ** ` --merge drop ` and ` --uns-merge drop ` are accepted.** ` drop ` was already
104- the documented default behaviour but was rejected as a value, so a config
105- could not state it explicitly.
106116
107117## 0.5.1
108118
0 commit comments