Merge pull request #33 from cellgeni/feat/recast-and-convert

Rename convert to recast; add format conversion as convert (h5ad ↔ zarr)

Author: Claptar

CHANGELOG.md

@@ -5,6 +5,42 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
 
 ## Unreleased
 
+### Added
+
+- **`adata convert` changes a store's format**
+  ([#20](https://github.com/cellgeni/adata-cli/issues/20),
+  [#22](https://github.com/cellgeni/adata-cli/issues/22)). This release
+  converts between h5ad and Zarr v2/v3 in any direction; the output's
+  extension picks the format. Every element is copied as it is, streamed,
+  keeping chunking and, where the target can express it, compression;
+  `--compression` chooses instead. A Zarr output keeps the source's version
+  unless `--zarr-format` says otherwise.
+
+  Everything is checked before anything is written, and the output is
+  written beside its destination and moved into place only once complete,
+  so a failed conversion leaves nothing behind. An existing output is
+  refused unless `--overwrite` is given. A flag that does not apply to the
+  conversion asked for is refused rather than ignored.
+
+  Formats are registered in `adata.conversion`, each with the converters
+  that read and write it; 10x HDF5, MTX folders, loom and others follow as
+  separate issues under #20.
+
+### Changed
+
+- **The matrix command `convert` is now `recast`**
+  ([#21](https://github.com/cellgeni/adata-cli/issues/21)). It changes a
+  matrix's dtype, layout or density exactly as `convert` did in 0.6; the
+  name `convert` now means format conversion. The hint `concat` prints for
+  mismatched encodings names `recast`.
+
+### Deprecated
+
+- **`convert` with `recast`'s flags.** Given any of `--dtype`,
+  `--indices-dtype`, `--layout`, `--all` or `--inplace`, `convert` still
+  runs `recast` as before, with a warning. **This is removed in 1.0.0**,
+  with the `h5ad` command name and the `info` alias.
+
 ### Fixed
 
 - **Writing an h5ad with a `None` in it to Zarr failed.** anndata stores

README.md

@@ -8,7 +8,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
 - Auto-detects `.h5ad` files vs `.zarr` directories
 - Chunked processing for dense and sparse matrices (CSR/CSC)
 - Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
-- Converts between HDF5 and Zarr (v2 and v3) in either direction
+- Converts between HDF5 and Zarr (v2 and v3) in either direction with `adata convert`
 - Rich terminal output with progress indicators, kept on stderr so results pipe cleanly
 
 **Documentation: [cellgeni.github.io/adata-cli](https://cellgeni.github.io/adata-cli/)**
@@ -56,7 +56,8 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
 - `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
 - `split` – write one store per distinct value of an annotation column, with a CSV manifest.
 - `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
-- `convert` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced.
+- `convert` – change a store's format: h5ad ↔ Zarr v2/v3, streaming, with the output written beside its destination and moved into place only once complete.
+- `recast` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced. Called `convert` until 0.7.0.
 - `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
 - `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
 
@@ -82,16 +83,23 @@ adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 ### Shrinking a store, and making encodings agree
 
 ```bash
-adata convert data.h5ad X --inplace --dtype float32
-adata convert data.h5ad --all -o small.h5ad --dtype float32 --indices-dtype int32
-adata convert data.h5ad X -o csc.h5ad --layout csc
+adata recast data.h5ad X --inplace --dtype float32
+adata recast data.h5ad --all -o small.h5ad --dtype float32 --indices-dtype int32
+adata recast data.h5ad X -o csc.h5ad --layout csc
 ```
 
-Counts written as float64 halve with no loss — and `convert` proves that
+Counts written as float64 halve with no loss — and `recast` proves that
 before it writes, by casting every value and casting it back. `concat`
 refuses inputs whose matrices disagree about CSR versus CSC; `--layout` is
 how you make them agree.
 
+### Changing format
+
+```bash
+adata convert data.h5ad -o data.zarr
+adata convert data.zarr -o data.h5ad --compression gzip
+```
+
 ## Documentation
 
 - [Get started](docs/GET_STARTED.md) — a short tutorial

benchmarks/cases.py

@@ -232,10 +232,7 @@ def _py(body: str) -> str:
         contenders=[
             Contender(
                 "adata-cli",
-                argv=[
-                    "adata", "subset", "{input0}", "-o", "{output}",
-                    "-q", "quality >= 0",
-                ],
+                argv=["adata", "convert", "{input0}", "-o", "{output}"],
             ),
             Contender(
                 "anndata (in memory)",
@@ -485,15 +482,15 @@ def _py(body: str) -> str:
             ),
         ],
     ),
-    # -- convert ------------------------------------------------------------
+    # -- recast (named convert until 0.7.0; the case names keep the history) --
     Case(
         name="convert-dtype",
         question="Rewrite X as float32.",
         contenders=[
             Contender(
                 "adata-cli",
                 argv=[
-                    "adata", "convert", "{input0}", "X", "-o", "{output}",
+                    "adata", "recast", "{input0}", "X", "-o", "{output}",
                     "--dtype", "float32", "--force",
                 ],
             ),
@@ -518,14 +515,14 @@ def _py(body: str) -> str:
             Contender(
                 "adata-cli (streaming)",
                 argv=[
-                    "adata", "convert", "{input0}", "X", "-o", "{output}",
+                    "adata", "recast", "{input0}", "X", "-o", "{output}",
                     "--layout", "csc",
                 ],
             ),
             Contender(
                 "adata-cli (--in-memory)",
                 argv=[
-                    "adata", "convert", "{input0}", "X", "-o", "{output}",
+                    "adata", "recast", "{input0}", "X", "-o", "{output}",
                     "--layout", "csc", "--in-memory",
                 ],
             ),

benchmarks/page_template.md

@@ -12,9 +12,10 @@ for time, and a table that hid the cost would not be worth publishing.
 
 ## What is measured
 
-Fifteen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
-subset by query, h5ad→zarr, `view`, `ls`, `create`, `export` dataframe / array /
-sparse, `import dataframe`, `split`, and a peak-RSS-against-input-size sweep. Inputs
+Seventeen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
+subset by query, h5ad→zarr (`convert`), `view`, `ls`, `create`, `export` dataframe /
+array / sparse, `import dataframe`, `split`, `recast` dtype and layout, and a
+peak-RSS-against-input-size sweep. Inputs
 are 50,000 × 20,000 at 5% density, plus the 2,000 × 36,601 shape that hung in 0.5.1.
 
 Every run also reports a startup floor (`--version` for each contender), because the

docs/COMMANDS.md

@@ -19,6 +19,7 @@ Run `adata <command> --help` for the authoritative flag list.
 - [split](#split)
 - [concat](#concat)
 - [convert](#convert)
+- [recast](#recast)
 - [create](#create)
 - [export](#export)
 - [import](#import)
@@ -84,27 +85,72 @@ holds more genes than the main object.
 
 ## `convert`
 
+Change a store's format. The output's extension picks the format; `--to`
+names it when the extension does not say, and `--from` names the input's
+when it cannot be detected from its content.
+
+```bash
+adata convert data.h5ad -o data.zarr
+adata convert data.zarr -o data.h5ad --compression gzip
+adata convert old.zarr -o new.zarr --zarr-format 3
+adata convert data.h5ad -o data.zarr --overwrite
+```
+
+| Flag | Meaning |
+|---|---|
+| `--output`, `-o` | Output path (required) |
+| `--from` | Input format, if it cannot be detected |
+| `--to` | Output format, if the extension does not say |
+| `--zarr-format` | Zarr version to write; defaults to the source's, else 3 |
+| `--compression` | `none`, `gzip` or `lzf` for h5ad; `none`, `gzip`, `zstd` or `blosc` for Zarr |
+| `--overwrite` | Replace an existing output |
+| `--force` | Convert despite a lossy or inflating result |
+
+| Format | Extension | Read | Write |
+|---|---|---|---|
+| `h5ad` | `.h5ad` | yes | yes |
+| `zarr` | `.zarr` (v2 or v3) | yes | yes |
+
+Between h5ad and Zarr every element is copied as it is, streamed, so the
+store's size is not a limit. Chunking carries over, and so does compression
+where the target can express it: Zarr codecs travel between stores of the same
+Zarr version, a compressed Zarr array becomes gzip in an h5ad, and an h5ad
+written to Zarr gets zarr-python's default codec. `--compression` replaces all
+of that with one choice. Elements are not re-encoded, so a store written by
+anndata 0.7 keeps its legacy layout.
+
+Every check runs before anything is written. The output is written beside
+its destination (`out.adata-tmp.zarr`) and moved into place only once
+complete, so a failed conversion leaves nothing behind. An existing output is
+refused unless `--overwrite` is given, and is replaced only once its
+successor is finished.
+
+To change a matrix's dtype, layout or density instead, use
+[`recast`](#recast).
+
+## `recast`
+
 Change a matrix's dtype, layout or density. Counts held as float64 cost twice
 the disk and twice the read for no information; a tool that wants CSC cannot
 use a CSR store; and `concat` refuses inputs whose encodings disagree.
 
 ```bash
-adata convert data.h5ad X -o out.h5ad --dtype float32
-adata convert data.h5ad X -o out.h5ad --layout csc
-adata convert data.h5ad X --inplace --dtype float32 --indices-dtype int32
-adata convert data.h5ad --all -o out.h5ad --dtype float32
-adata convert data.h5ad layers/counts -o out.h5ad --layout dense --force
+adata recast data.h5ad X -o out.h5ad --dtype float32
+adata recast data.h5ad X -o out.h5ad --layout csc
+adata recast data.h5ad X --inplace --dtype float32 --indices-dtype int32
+adata recast data.h5ad --all -o out.h5ad --dtype float32
+adata recast data.h5ad layers/counts -o out.h5ad --layout dense --force
 ```
 
 | Flag | Meaning |
 |---|---|
 | `--output`, `-o` | Output path. Required unless `--inplace` |
 | `--inplace` | Replace the source (written to a temporary path first) |
-| `--all` | Convert `X`, every layer and `raw/X` |
+| `--all` | Recast `X`, every layer and `raw/X` |
 | `--dtype` | New dtype for the values, e.g. `float32` |
 | `--indices-dtype` | New dtype for sparse indices: `int32` or `int64` |
 | `--layout` | `csr`, `csc`, `dense` or `sparse` |
-| `--force` | Convert despite a lossy cast or a large size increase |
+| `--force` | Recast despite a lossy cast or a large size increase |
 | `--in-memory` | Transpose in memory rather than streaming |
 | `--chunk`, `-C` | Row chunk size for dense matrices |
 | `--zarr-format` | Zarr version to write; defaults to the source's |
@@ -128,6 +174,10 @@ Transposing streams by default and works on matrices too large to load, at
 the cost of two extra passes over the nonzeros. `--in-memory` is faster when
 the matrix fits.
 
+Until 0.7.0 this command was called `convert`. Until 1.0.0, `convert` given
+any of `--dtype`, `--indices-dtype`, `--layout`, `--all` or `--inplace` still
+runs `recast`, with a warning.
+
 ## `split`
 
 One store per distinct value of a column.
@@ -277,13 +327,14 @@ adata subset data.h5ad -o cortex.h5ad --obs barcodes.txt
 ## Zarr format selection
 
 New Zarr stores follow the source store's version, so a v2 input is not
-silently upgraded. `--zarr-format 2|3` on `create`, `subset`, `split` and
-`concat` overrides that. Writing a Zarr store from an `.h5ad` source defaults
-to v3.
+silently upgraded. `--zarr-format 2|3` on `create`, `subset`, `split`,
+`concat`, `convert` and `recast` overrides that. Writing a Zarr store from
+an `.h5ad` source defaults to v3.
 
 ```bash
 adata create out.zarr --n-obs 100 --n-var 50 --zarr-format 2
 adata subset data.zarr -o out.zarr --obs keep.txt --zarr-format 3
 adata split data.zarr --by sample -o parts/ --suffix .zarr --zarr-format 3
 adata concat a.zarr b.zarr -o merged.zarr --zarr-format 3
+adata convert data.zarr -o v3.zarr --zarr-format 3
 ```

docs/TESTING.md

@@ -23,6 +23,8 @@ and the compatibility suite as a separate job.
 | `test_anndata_roundtrip.py` | anndata writes the fixtures, reads back our output |
 | `test_anndata_versions.py` | Compatibility with six real anndata releases (see below) |
 | `test_commands_phase2.py`, `test_commands_coverage.py`, `test_cli.py` | Command surfaces and error paths |
+| `test_convert.py` | `convert` between every backend pair, read back by anndata; its refusals, staging, and the forwarding of recast's old flags |
+| `test_recast.py` | `recast` against scipy: casts, transposes, density changes and every refusal |
 | `test_performance.py` | Complexity guards -- what an operation costs, not how long it takes (see below) |
 | `test_benchmark_harness.py` | That the benchmark's measurement is trustworthy |
 | `test_subset.py`, `test_export.py`, `test_import.py`, `test_info_read.py`, `test_zarr.py`, `test_query.py` | Per-feature unit tests |
@@ -232,7 +234,8 @@ should scale with, scale only that, and hold everything else fixed.
 | `export dataframe` | n_obs, n_obs_columns | linear |
 | `export array`, `sparse` (both paths), `dict`, `image` | elements, nnz, keys, pixels | linear |
 | `import dataframe`, `array`, `sparse`, `dict`, `image` | rows, elements, nnz, keys, pixels | linear in writes |
-| h5ad to zarr | n_obs | linear |
+| h5ad to zarr (`subset`), `convert` both directions | n_obs | linear |
+| `recast --dtype`, `--layout csc`, `--layout dense` | nnz, n_obs | linear; the streaming transpose's **peak memory** is bounded |
 
 The `slow` marker is on the three guards that build a 65,536-row store; they still gate
 merges, and `-m "not slow"` skips them locally. `perf` is on the tracing guards.

docs/index.md

@@ -51,7 +51,8 @@ adata subset data.h5ad -o cortex.h5ad --obs-query "cluster == Cortex_2"
 adata split  data.h5ad --by sample -o per_sample/
 adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 
-adata convert data.h5ad X -o small.h5ad --dtype float32
+adata convert data.h5ad -o data.zarr
+adata recast  data.h5ad X -o small.h5ad --dtype float32
 
 adata create new.h5ad --n-obs 5000 --n-var 2000
 adata import sparse new.h5ad X counts.mtx --inplace
@@ -62,9 +63,9 @@ adata import sparse new.h5ad X counts.mtx --inplace
 Every AnnData on-disk layout is readable, from anndata 0.7.x through the
 current spec; everything written is in the current spec. This is verified in
 CI against stores written by anndata 0.8, 0.9, 0.10, 0.11, 0.12 and 0.13, in
-both formats. Both backends are
-supported in both directions, so `adata subset in.h5ad -o out.zarr` converts as
-it filters.
+both formats. Both backends are supported in both directions: `adata convert
+in.h5ad -o out.zarr` converts a whole store, and `subset`, `split` and
+`concat` convert as they write when the output's backend differs.
 
 | Element | Read | Write |
 |---|---|---|