Skip to content

Commit 8a69561

Browse files
github-actions[bot]github-actions[bot]
committed
Benchmark results for 0.7.0 [skip ci]
1 parent 60b2c4b commit 8a69561

2 files changed

Lines changed: 877 additions & 51 deletions

File tree

‎docs/BENCHMARKS.md‎

Lines changed: 53 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,10 @@ for time, and a table that hid the cost would not be worth publishing.
1212

1313
## What is measured
1414

15-
Fifteen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
16-
subset by query, h5ad→zarr, `view`, `ls`, `create`, `export` dataframe / array /
17-
sparse, `import dataframe`, `split`, and a peak-RSS-against-input-size sweep. Inputs
15+
Seventeen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
16+
subset by query, h5ad→zarr (`convert`), `view`, `ls`, `create`, `export` dataframe /
17+
array / sparse, `import dataframe`, `split`, `recast` dtype and layout, and a
18+
peak-RSS-against-input-size sweep. Inputs
1819
are 50,000 × 20,000 at 5% density, plus the 2,000 × 36,601 shape that hung in 0.5.1.
1920

2021
Every run also reports a startup floor (`--version` for each contender), because the
@@ -25,7 +26,7 @@ the entire measurement.
2526
library offers an equivalent, so the rows would only ever read `n/a`. They are covered
2627
by the complexity guards instead.
2728

28-
## Results — `0.6.1`
29+
## Results — `0.7.0`
2930

3031
| | |
3132
|---|---|
@@ -35,28 +36,28 @@ by the complexity guards instead.
3536
| Address-space ceiling | 12.9 GB per process |
3637
| Repeats | 3 (fastest shown) |
3738
| Platform | Linux x86_64, Python 3.12.3 |
38-
| Versions | adata-cli 0.6.1, anndata 0.13.4, scanpy 1.12.4 |
39-
| Generated | 2026-09-24T15:41:03Z |
39+
| Versions | adata-cli 0.7.0, anndata 0.13.4, scanpy 1.12.4 |
40+
| Generated | 2026-10-15T12:48:08Z |
4041

4142
### `startup`
4243

4344
What does each contender cost before it does any work?
4445

4546
| Contender | Wall time | Peak RSS | Output | vs previous |
4647
|---|---|---|---|---|
47-
| adata-cli | 0.56 s | 69.4 MB | - | no change |
48-
| anndata | 1.52 s | 132.8 MB | - | no change |
49-
| scanpy | 3.30 s | 265.2 MB | - | no change |
48+
| adata-cli | 0.63 s | 67.9 MB | - | time +13% |
49+
| anndata | 1.65 s | 133.5 MB | - | no change |
50+
| scanpy | 3.57 s | 260.3 MB | - | no change |
5051

5152
### `concat-inner`
5253

5354
Concatenate two stores on the obs axis, inner join.
5455

5556
| Contender | Wall time | Peak RSS | Output | vs previous |
5657
|---|---|---|---|---|
57-
| adata-cli | 12.26 s | 177.2 MB | 583.3 MB | time +108% |
58-
| anndata (concat_on_disk) | 6.18 s | 451.0 MB | 831.9 MB | no change |
59-
| anndata (in memory) | 12.26 s | 1.8 GB | 580.3 MB | no change |
58+
| adata-cli | 9.29 s | 170.3 MB | 583.3 MB | time -24% |
59+
| anndata (concat_on_disk) | 5.94 s | 447.3 MB | 831.9 MB | no change |
60+
| anndata (in memory) | 9.09 s | 1.8 GB | 580.3 MB | time -26% |
6061
| scanpy | n/a | n/a | n/a | - |
6162
| | *scanpy has no concat of its own; it re-exports anndata's* | | | |
6263

@@ -66,29 +67,29 @@ Concatenate carrying var columns forward -- the REQ-71798 case.
6667

6768
| Contender | Wall time | Peak RSS | Output | vs previous |
6869
|---|---|---|---|---|
69-
| adata-cli | 1.74 s | 195.6 MB | 45.7 MB | time +40% |
70-
| anndata (concat_on_disk) | 2.26 s | 247.6 MB | 63.2 MB | no change |
71-
| anndata (in memory) | 2.67 s | 288.0 MB | 44.6 MB | no change |
70+
| adata-cli | 1.50 s | 193.1 MB | 45.7 MB | time -14% |
71+
| anndata (concat_on_disk) | 2.24 s | 246.3 MB | 63.2 MB | no change |
72+
| anndata (in memory) | 2.42 s | 288.9 MB | 44.6 MB | no change |
7273

7374
### `subset-query`
7475

7576
Keep the obs rows matching a predicate on an existing column.
7677

7778
| Contender | Wall time | Peak RSS | Output | vs previous |
7879
|---|---|---|---|---|
79-
| adata-cli | 4.10 s | 212.0 MB | 147.1 MB | time +26%, RSS -39% |
80-
| anndata (backed) | 4.94 s | 626.5 MB | 146.6 MB | no change |
81-
| anndata (in memory) | 5.21 s | 984.4 MB | 146.6 MB | no change |
82-
| scanpy (filter_cells) | 9.02 s | 1.6 GB | 291.2 MB | no change |
80+
| adata-cli | 3.35 s | 209.5 MB | 147.1 MB | time -18% |
81+
| anndata (backed) | 4.26 s | 683.4 MB | 146.6 MB | time -14% |
82+
| anndata (in memory) | 4.49 s | 985.2 MB | 146.6 MB | time -14% |
83+
| scanpy (filter_cells) | 7.69 s | 1.6 GB | 291.2 MB | time -15% |
8384

8485
### `h5ad-to-zarr`
8586

8687
Convert a store to Zarr.
8788

8889
| Contender | Wall time | Peak RSS | Output | vs previous |
8990
|---|---|---|---|---|
90-
| adata-cli | 7.81 s | 232.9 MB | 151.6 MB in 4717 files | time +29% |
91-
| anndata (in memory) | 6.15 s | 814.9 MB | 136.8 MB in 55 files | no change |
91+
| adata-cli | 6.14 s | 154.2 MB | 151.6 MB in 4717 files | time -21%, RSS -34% |
92+
| anndata (in memory) | 5.70 s | 778.4 MB | 136.8 MB in 55 files | no change |
9293
| scanpy | n/a | n/a | n/a | - |
9394
| | *no streaming converter; scanpy defers to anndata* | | | |
9495

@@ -98,37 +99,37 @@ Report what is in the store, without reading the matrix.
9899

99100
| Contender | Wall time | Peak RSS | Output | vs previous |
100101
|---|---|---|---|---|
101-
| adata-cli | 0.58 s | 71.4 MB | - | no change |
102-
| anndata (read_elem on obs) | 1.76 s | 159.9 MB | - | no change |
103-
| anndata (full load) | 3.20 s | 583.7 MB | - | no change |
102+
| adata-cli | 0.61 s | 69.8 MB | - | no change |
103+
| anndata (read_elem on obs) | 1.76 s | 160.3 MB | - | no change |
104+
| anndata (full load) | 3.19 s | 584.3 MB | - | no change |
104105

105106
### `export-obs`
106107

107108
Write the obs table out as CSV.
108109

109110
| Contender | Wall time | Peak RSS | Output | vs previous |
110111
|---|---|---|---|---|
111-
| adata-cli | 0.74 s | 85.1 MB | 1.8 MB | no change |
112-
| anndata (read_elem on obs) | 1.77 s | 160.5 MB | 1.8 MB | no change |
113-
| anndata (full load) | 3.28 s | 583.9 MB | 1.8 MB | no change |
112+
| adata-cli | 0.77 s | 83.3 MB | 1.8 MB | no change |
113+
| anndata (read_elem on obs) | 1.78 s | 161.1 MB | 1.8 MB | no change |
114+
| anndata (full load) | 3.22 s | 584.4 MB | 1.8 MB | no change |
114115

115116
### `split-by-sample`
116117

117118
Write one store per distinct value of an obs column.
118119

119120
| Contender | Wall time | Peak RSS | Output | vs previous |
120121
|---|---|---|---|---|
121-
| adata-cli | 6.65 s | 198.4 MB | 302.6 MB in 9 files | time -88%, RSS -74% |
122-
| anndata (hand-written loop) | 7.81 s | 684.9 MB | 301.7 MB in 8 files | no change |
122+
| adata-cli | 4.99 s | 196.5 MB | 302.6 MB in 9 files | time -25% |
123+
| anndata (hand-written loop) | 6.34 s | 684.4 MB | 301.7 MB in 8 files | time -19% |
123124

124125
### `ls`
125126

126127
List everything in the store.
127128

128129
| Contender | Wall time | Peak RSS | Output | vs previous |
129130
|---|---|---|---|---|
130-
| adata-cli | 0.59 s | 70.5 MB | - | no change |
131-
| h5py (visit) | 1.66 s | 136.3 MB | - | no change |
131+
| adata-cli | 0.63 s | 68.9 MB | - | no change |
132+
| h5py (visit) | 1.67 s | 136.4 MB | - | no change |
132133
| h5ls -r | n/a | n/a | n/a | - |
133134
| | *h5ls is not installed* | | | |
134135

@@ -138,56 +139,56 @@ Write an empty store with a given obs/var shape.
138139

139140
| Contender | Wall time | Peak RSS | Output | vs previous |
140141
|---|---|---|---|---|
141-
| adata-cli | 0.64 s | 84.5 MB | 3.4 MB | no change |
142-
| anndata | 8.73 s | 228.1 MB | 49.2 MB | no change |
142+
| adata-cli | 0.66 s | 82.5 MB | 3.4 MB | no change |
143+
| anndata | 9.00 s | 229.1 MB | 49.2 MB | no change |
143144

144145
### `export-array`
145146

146147
Write a dense element out as .npy.
147148

148149
| Contender | Wall time | Peak RSS | Output | vs previous |
149150
|---|---|---|---|---|
150-
| adata-cli | 0.60 s | 82.6 MB | 10.0 MB | no change |
151-
| anndata (read_elem) | 1.67 s | 154.3 MB | 10.0 MB | no change |
152-
| anndata (full load) | 3.27 s | 583.8 MB | 10.0 MB | no change |
151+
| adata-cli | 0.66 s | 81.3 MB | 10.0 MB | time +10% |
152+
| anndata (read_elem) | 1.68 s | 154.6 MB | 10.0 MB | no change |
153+
| anndata (full load) | 3.22 s | 584.3 MB | 10.0 MB | no change |
153154

154155
### `export-sparse`
155156

156157
Write X out as Matrix Market text.
157158

158159
| Contender | Wall time | Peak RSS | Output | vs previous |
159160
|---|---|---|---|---|
160-
| adata-cli | 25.52 s | 76.3 MB | 761.3 MB | no change |
161-
| anndata (sparse_dataset) | 7.12 s | 968.7 MB | 661.6 MB | no change |
162-
| anndata (full load) | 7.23 s | 775.1 MB | 661.6 MB | no change |
161+
| adata-cli | 21.22 s | 74.7 MB | 761.3 MB | time -17% |
162+
| anndata (sparse_dataset) | 6.53 s | 968.5 MB | 661.6 MB | no change |
163+
| anndata (full load) | 6.72 s | 774.4 MB | 661.6 MB | no change |
163164

164165
### `import-dataframe`
165166

166167
Replace obs from a CSV.
167168

168169
| Contender | Wall time | Peak RSS | Output | vs previous |
169170
|---|---|---|---|---|
170-
| adata-cli | 1.21 s | 114.2 MB | 296.0 MB | no change |
171-
| anndata | 6.88 s | 596.5 MB | 291.2 MB | no change |
171+
| adata-cli | 1.96 s | 112.4 MB | 296.0 MB | time +63% |
172+
| anndata | 5.36 s | 600.5 MB | 291.2 MB | time -22% |
172173

173174
### `concat-outer`
174175

175176
Concatenate two stores keeping the union of variables.
176177

177178
| Contender | Wall time | Peak RSS | Output | vs previous |
178179
|---|---|---|---|---|
179-
| adata-cli | 12.20 s | 172.0 MB | 583.3 MB | time +106% |
180-
| anndata (concat_on_disk) | 6.26 s | 450.5 MB | 831.9 MB | no change |
181-
| anndata (in memory) | 12.18 s | 1.8 GB | 580.3 MB | no change |
180+
| adata-cli | 9.15 s | 175.6 MB | 583.3 MB | time -25% |
181+
| anndata (concat_on_disk) | 5.93 s | 449.6 MB | 831.9 MB | no change |
182+
| anndata (in memory) | 9.07 s | 1.8 GB | 580.3 MB | time -26% |
182183

183184
### `convert-dtype`
184185

185186
Rewrite X as float32.
186187

187188
| Contender | Wall time | Peak RSS | Output | vs previous |
188189
|---|---|---|---|---|
189-
| adata-cli | 5.57 s | 90.2 MB | 291.1 MB | no change |
190-
| anndata (in memory) | 6.91 s | 983.8 MB | 291.2 MB | no change |
190+
| adata-cli | 4.02 s | 88.2 MB | 291.1 MB | time -28% |
191+
| anndata (in memory) | 5.35 s | 984.0 MB | 291.2 MB | time -23% |
191192
| scanpy | n/a | n/a | n/a | - |
192193
| | *no dtype rewrite; scanpy defers to anndata* | | | |
193194

@@ -197,18 +198,18 @@ Transpose X from CSR to CSC.
197198

198199
| Contender | Wall time | Peak RSS | Output | vs previous |
199200
|---|---|---|---|---|
200-
| adata-cli (streaming) | 13.62 s | 369.8 MB | 1.2 GB | no change |
201-
| adata-cli (--in-memory) | 9.52 s | 2.3 GB | 297.0 MB | no change |
202-
| anndata (in memory) | 7.22 s | 984.2 MB | 297.0 MB | no change |
201+
| adata-cli (streaming) | 18.18 s | 362.2 MB | 297.0 MB | time +33% |
202+
| adata-cli (--in-memory) | 8.85 s | 2.3 GB | 297.0 MB | no change |
203+
| anndata (in memory) | 6.04 s | 984.7 MB | 297.0 MB | time -16% |
203204

204205
### `rss-vs-size`
205206

206207
Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.
207208

208209
| Contender | Wall time | Peak RSS | Output | vs previous |
209210
|---|---|---|---|---|
210-
| adata-cli | 12.13 s | 173.8 MB | 583.3 MB | time +105% |
211-
| anndata (in memory) | 12.18 s | 1.8 GB | 580.3 MB | no change |
211+
| adata-cli | 9.19 s | 180.6 MB | 583.3 MB | time -24% |
212+
| anndata (in memory) | 8.99 s | 1.8 GB | 580.3 MB | time -26% |
212213

213214
---
214215

@@ -222,6 +223,7 @@ Rows marked `n/a` are operations the baseline does not offer; that is a result,
222223

223224
| Run | Ref | Tier | Wall time | Peak RSS | Raw |
224225
|---|---|---|---|---|---|
226+
| 2026-10-15T12:48:08Z | `0.7.0` | ci | 9.29 s | 170.3 MB | [json](benchmarks/0.7.0.json) |
225227
| 2026-09-24T15:41:03Z | `0.6.1` | ci | 12.26 s | 177.2 MB | [json](benchmarks/0.6.1.json) |
226228
| 2026-09-24T10:35:13Z | `0.6.0` | ci | 5.90 s | 176.2 MB | [json](benchmarks/0.6.0.json) |
227229

0 commit comments

Comments
 (0)