Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,12 @@ Each layer calls only the layers below it:
- add a scanpy comparison in `tests/test_external_readers.py`.

Content sniffing runs before extensions, because extensions lie (`.h5`).
A format with no extension, such as an MTX folder (`10x-mtx`), is chosen as
an output only with `--to`.
- **`formats/`**: the per-file-type halves of `export` and `import`: npy, mtx,
csv, json and images.
csv, json and images. `mtx.py` is a streaming Matrix Market codec (two
passes in, block writes out) shared by `import`/`export sparse` and the
`10x-mtx` converter; `textio.py` holds its gzip-aware block I/O.
- **`util/`**: `staging.py` and `path.py` (below), plus `names.py`.

## Rules the code depends on
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
- Auto-detects `.h5ad` files vs `.zarr` directories
- Chunked processing for dense and sparse matrices (CSR/CSC)
- Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
- Converts between HDF5 and Zarr (v2 and v3) in either direction with `adata convert`, and to and from 10x Genomics HDF5 matrices
- Converts between HDF5 and Zarr (v2 and v3) in either direction with `adata convert`, and to and from 10x Genomics HDF5 matrices and 10x / STARsolo MTX folders
- Rich terminal output with progress indicators, kept on stderr so results pipe cleanly

**Documentation: [cellgeni.github.io/adata-cli](https://cellgeni.github.io/adata-cli/)**
Expand Down Expand Up @@ -56,7 +56,7 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
- `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
- `split` – write one store per distinct value of an annotation column, with a CSV manifest.
- `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
- `convert` – change a store's format: h5ad ↔ Zarr v2/v3, and 10x Genomics HDF5 (Cell Ranger's `*_feature_bc_matrix.h5`) in and out, streaming, with the output written beside its destination and moved into place only once complete.
- `convert` – change a store's format: h5ad ↔ Zarr v2/v3, 10x Genomics HDF5 (Cell Ranger's `*_feature_bc_matrix.h5`) and 10x / STARsolo MTX folders in and out, streaming, with the output written beside its destination and moved into place only once complete.
- `recast` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced. Called `convert` until 0.7.0.
- `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
- `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
Expand Down Expand Up @@ -100,6 +100,7 @@ adata convert data.h5ad -o data.zarr
adata convert data.zarr -o data.h5ad --compression gzip
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad --feature-type "Gene Expression"
adata convert counts.h5ad -o for_cellbender.h5 --layer counts
adata convert Solo.out/Velocyto/raw -o velocyto.h5ad
```

## Documentation
Expand Down
36 changes: 33 additions & 3 deletions benchmarks/cases.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,10 @@ class Case:
#: Report the output's size against the input's, as before -> after. For
#: cases whose point is what they do to the file's size.
compare_size: bool = False
#: An auxiliary input the case needs, e.g. the CSV an import reads.
#: Built once from the first input and offered to CLI contenders as
#: `{sidecar_csv}` and to scripts as $BENCH_SIDECAR_CSV.
#: An auxiliary input the case needs: "csv", the CSV an import reads, or
#: "10x-mtx", an MTX folder of the first input's counts. Built once from
#: the first input and offered to CLI contenders as `{sidecar}` and to
#: scripts as $BENCH_SIDECAR (`{sidecar_csv}` and $BENCH_SIDECAR_CSV too).
sidecar: Optional[str] = None
tags: List[str] = field(default_factory=list)

Expand Down Expand Up @@ -458,6 +459,35 @@ def _py(body: str) -> str:
),
],
),
Case(
name="import-10x-mtx",
question=(
"Read a gzipped Cell Ranger 3 MTX folder of the same counts into "
"an h5ad. adata-cli streams it in two passes and keeps integer "
"counts; scanpy loads it and converts them to float32."
),
sidecar="10x-mtx",
contenders=[
Contender(
"adata-cli",
argv=[
"adata", "convert", "{sidecar}", "-o", "{output}",
"--compression", COMPRESSION,
],
),
Contender(
"scanpy",
env="scanpy",
script=_py(
"import os, scanpy as sc\n"
"obj = sc.read_10x_mtx(os.environ['BENCH_SIDECAR'], "
"gex_only=False, make_unique=False)\n"
"obj.write_h5ad(OUT, compression=COMPRESSION)\n"
),
),
],
tags=["headline"],
),
# -- concat, outer ------------------------------------------------------
Case(
name="concat-outer",
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/page_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ for time, and a table that hid the cost would not be worth publishing.

## What is measured

Eighteen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
Nineteen cases on a GitHub-hosted runner: concat (inner, outer, `--merge same`),
subset by query, h5ad→zarr (`convert`), `view`, `ls`, `create`, `export` dataframe /
array / sparse, `import dataframe`, `split`, `recast` narrowing and layout, and a
array / sparse, `import dataframe`, MTX folder import (`convert`), `split`, `recast` narrowing and layout, and a
peak-RSS-against-input-size sweep. Inputs
are 50,000 × 20,000 at 5% density, plus the 2,000 × 36,601 shape that hung in 0.5.1.
`recast` narrowing starts from a copy of the main input stored as float64 values
Expand Down
35 changes: 34 additions & 1 deletion benchmarks/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,13 +145,15 @@ def _run_contender(
substitutions = {
"output": str(output),
"outdir": str(outdir),
"sidecar": str(sidecar) if sidecar else "",
"sidecar_csv": str(sidecar) if sidecar else "",
"n_obs": str(shape[0]),
"n_var": str(shape[1]),
**{f"input{i}": str(p) for i, p in enumerate(inputs)},
}
env = {"BENCH_N_OBS": str(shape[0]), "BENCH_N_VAR": str(shape[1])}
if sidecar:
env["BENCH_SIDECAR"] = str(sidecar)
env["BENCH_SIDECAR_CSV"] = str(sidecar)

if contender.argv is not None:
Expand Down Expand Up @@ -181,9 +183,13 @@ def _build_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Optional[Pa
Derived from the real store rather than invented, so an import writes
back something the file could plausibly have held.
"""
if case.sidecar == "10x-mtx":
return _build_mtx_sidecar(case, inputs, workdir)
if case.sidecar != "csv":
return None
path = workdir / f"{case.name}-sidecar.csv"
# Named for its input too: a work directory reused across tiers must not
# hand one tier's sidecar to another.
path = workdir / f"{case.name}-{inputs[0].stem}-sidecar.csv"
if path.exists():
return path
import h5py
Expand All @@ -194,6 +200,33 @@ def _build_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Optional[Pa
return path


def _build_mtx_sidecar(case: Case, inputs: List[Path], workdir: Path) -> Path:
"""A Cell Ranger 3 MTX folder holding the first input's counts.

Written by adata-cli itself, which is the one tool here that can: integer
counts first, as Cell Ranger writes them, then the gzipped folder. The
fixture's var names are ids and its symbols are a column, so the symbols
become the names, and placeholders the ids.
"""
path = workdir / f"{case.name}-{inputs[0].stem}-sidecar"
if path.exists():
return path
counts = workdir / f"{case.name}-{inputs[0].stem}-counts.h5ad"
if counts.exists():
counts.unlink()
for command in (
["adata", "recast", str(inputs[0]), "X", "-o", str(counts), "--dtype", "int32"],
["adata", "convert", str(counts), "-o", str(path), "--to", "10x-mtx",
"--gene-names-column", "gene_symbol", "--feature-types-column",
"feature_type", "--force"],
):
done = subprocess.run(command, capture_output=True, text=True)
if done.returncode != 0:
raise RuntimeError(f"could not build the MTX sidecar:\n{done.stderr or done.stdout}")
counts.unlink()
return path


def run(
tier_name: str,
*,
Expand Down
54 changes: 53 additions & 1 deletion docs/COMMANDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ adata convert old.zarr -o new.zarr --zarr-format 3
adata convert data.h5ad -o data.zarr --overwrite
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
adata convert counts.h5ad -o for_cellbender.h5 --layer counts
adata convert Solo.out/Gene/filtered -o counts.h5ad
adata convert counts.h5ad -o counts_mtx --to 10x-mtx
```

| Flag | Meaning |
Expand All @@ -111,7 +113,7 @@ adata convert counts.h5ad -o for_cellbender.h5 --layer counts
| `--from` | Input format, if it cannot be detected |
| `--to` | Output format, if the extension does not say |
| `--zarr-format` | Zarr version to write; defaults to the source's, else 3 |
| `--compression` | `none`, `gzip` or `lzf` for h5ad; `none`, `gzip`, `zstd` or `blosc` for Zarr |
| `--compression` | `none`, `gzip` or `lzf` for h5ad; `none`, `gzip`, `zstd` or `blosc` for Zarr; `gzip` (default) or `none` for an MTX folder |
| `--overwrite` | Replace an existing output |
| `--force` | Convert despite a lossy, inflating or placeholder result |

Expand All @@ -120,6 +122,7 @@ adata convert counts.h5ad -o for_cellbender.h5 --layer counts
| `h5ad` | `.h5ad` | yes | yes |
| `zarr` | `.zarr` (v2 or v3) | yes | yes |
| `10x-h5` | `.h5` | Cell Ranger 2 and 3+, Space Ranger, Xenium | Cell Ranger 3+ layout |
| `10x-mtx` | a folder (`--to 10x-mtx`) | Cell Ranger 2 and 3+, STARsolo (velocyto too) | Cell Ranger 3+ layout, gzipped |

Formats other than h5ad and Zarr convert to and from AnnData only: to turn a
10x file into another format, convert it to `.h5ad` first.
Expand Down Expand Up @@ -193,6 +196,55 @@ is refused, and `--force` writes the names as ids and `Gene Expression` as
the type. A matrix of floats is written as it is, with a warning. A file
imported by `convert` exports with its run metadata and tag fields intact.

### 10x Genomics and STARsolo MTX folders

A folder holding `matrix.mtx`, `barcodes.tsv` and `features.tsv` (or
Cell Ranger 2's `genes.tsv`), each plain or gzipped, is recognised as one.
Every shape Cell Ranger and STARsolo write is read:

* Cell Ranger 3+ (`filtered_feature_bc_matrix/`), with id, name and feature
type per feature;
* Cell Ranger 2 (`filtered_gene_bc_matrices/`), one subfolder per genome:
give the parent and `--genome` picks one, or the only one is taken;
* STARsolo `Solo.out/Gene/filtered/`, `GeneFull/` and so on, uncompressed,
with two or three feature columns;
* STARsolo `Solo.out/Velocyto/raw/`: `spliced.mtx`, `unspliced.mtx` and
`ambiguous.mtx` become layers of those names, and X is a copy of
`spliced`, the layout scvelo expects;
* any of these with every file under one prefix, as GEO serves them
(`GSM123_matrix.mtx.gz`, ...).

```bash
adata convert filtered_feature_bc_matrix -o counts.h5ad
adata convert Solo.out/Gene/filtered -o counts.h5ad
adata convert Solo.out/Velocyto/raw -o velocyto.h5ad
adata convert filtered_gene_bc_matrices -o mouse.h5ad --genome mm10
adata convert GSE000_RAW -o sample.h5ad --split-feature-types
adata convert counts.h5ad -o counts_mtx --to 10x-mtx --layer counts
```

The import flags are those of the HDF5 format above (`--genome`,
`--var-names`, `--make-unique`, `--feature-type`, `--split-feature-types`),
plus `--compression` for the output store, which is gzip on h5ad unless
given. Reading gives what `scanpy.read_10x_mtx(gex_only=False)` gives,
column for column, except that integer counts stay integers (int32, int64
when a value needs it) where scanpy converts them to float32; a `real` matrix
is read as float64. Duplicate entries are summed, as scipy does.

The matrix is read in two streaming passes, so its size is not a limit: the
first checks the whole folder -- barcode and feature counts against the
matrix, every entry against its size -- before any output exists, and the
second writes X. Files in barcode order, as Cell Ranger and STARsolo write
them, are written straight through; others are sorted through scratch space
beside the output.

Exporting needs `--to 10x-mtx`, since a folder has no extension to say so,
and writes a Cell Ranger 3 folder: gzipped (`--compression none` for plain
files), three feature columns, entries in barcode order, from X or the
matrix `--layer` names. Feature ids and types come from var as for the HDF5
export, with the same `--force` placeholders. A barcode or feature holding a
tab or a line break is refused, since it would break the TSV file.

## `recast`

Change a matrix's dtype, layout or density. Counts held as float64 cost twice
Expand Down
8 changes: 7 additions & 1 deletion docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ separate jobs.
| `test_convert.py` | `convert` between every backend pair, read back by anndata; its refusals, staging, and the forwarding of recast's old flags |
| `test_recast.py` | `recast` against scipy: casts, transposes, density changes and every refusal |
| `test_tenx.py` | 10x HDF5 import and export against Cell Ranger-shaped fixtures (v2, v3, multimodal, probe-based): mapping, filters, round trips, refusals |
| `test_external_readers.py` | 10x conversions checked by scanpy and PyTables in their own uv environments (`integration`) |
| `test_mtx.py` | 10x / STARsolo MTX folders against hand-written fixtures (Cell Ranger 2 and 3, STARsolo, velocyto, GEO prefixes): mapping, entry orders, refusals that leave nothing, round trips |
| `test_mtx_codec.py`, `test_csr_build.py` | The streaming Matrix Market reader and writer against scipy, and the bucket sort they and `recast` share |
| `test_external_readers.py` | 10x conversions, HDF5 and MTX, checked by scanpy and PyTables in their own uv environments (`integration`) |
| `test_performance.py` | Complexity guards -- what an operation costs, not how long it takes (see below) |
| `test_benchmark_harness.py` | That the benchmark's measurement is trustworthy |
| `test_subset.py`, `test_export.py`, `test_import.py`, `test_info_read.py`, `test_zarr.py`, `test_query.py` | Per-feature unit tests |
Expand Down Expand Up @@ -236,9 +238,13 @@ should scale with, scale only that, and hold everything else fixed.
| `split --by`, `--axis var` | n_groups | linear reads, **flat** scan work |
| `export dataframe` | n_obs, n_obs_columns | linear |
| `export array`, `sparse` (both paths), `dict`, `image` | elements, nnz, keys, pixels | linear |
| `export sparse` | n_obs | **flat** read calls: one per block, not per row |
| `import dataframe`, `array`, `sparse`, `dict`, `image` | rows, elements, nnz, keys, pixels | linear in writes |
| `import sparse` | n_obs | **flat** write calls |
| h5ad to zarr (`subset`), `convert` both directions | n_obs | linear |
| `convert` 10x import (whole, `--split-feature-types`), export | n_barcodes | linear |
| `convert` 10x MTX import (in order, shuffled, `--split-feature-types`) | n_barcodes | linear; **flat** write calls |
| `convert` to 10x MTX | n_obs | **flat** read calls |
| `recast --dtype`, `--layout csc`, `--layout dense` | nnz, n_obs | linear; the streaming transpose's **peak memory** is bounded |

The `slow` marker is on the three guards that build a 65,536-row store; they still gate
Expand Down
4 changes: 3 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ both formats. Both backends are supported in both directions: `adata convert
in.h5ad -o out.zarr` converts a whole store, and `subset`, `split` and
`concat` convert as they write when the output's backend differs.
`convert` also reads and writes 10x Genomics HDF5 matrices (Cell Ranger's
`*_feature_bc_matrix.h5`), matching what `scanpy.read_10x_h5` reads.
`*_feature_bc_matrix.h5`) and 10x / STARsolo Matrix Market folders,
velocyto's included, matching what scanpy's `read_10x_h5` and `read_10x_mtx`
read.

| Element | Read | Write |
|---|---|---|
Expand Down
25 changes: 16 additions & 9 deletions src/adata/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -553,7 +553,7 @@ def convert(
ctx: typer.Context,
file: Path = typer.Argument(
...,
help="Input file or store",
help="Input file, store, or 10x/STARsolo matrix folder",
exists=True,
readable=True,
dir_okay=True,
Expand All @@ -563,7 +563,8 @@ def convert(
None,
"--output",
"-o",
help="Output path; its extension picks the format (.h5ad, .zarr, .h5)",
help="Output path; its extension picks the format (.h5ad, .zarr, .h5), "
"or a folder with --to 10x-mtx",
dir_okay=True,
file_okay=True,
),
Expand All @@ -586,7 +587,7 @@ def convert(
None,
"--compression",
help="none, gzip or lzf for h5ad; none, gzip, zstd or blosc for zarr "
"(defaults to the source's)",
"(defaults to the source's); gzip or none for a 10x-mtx folder",
),
force: bool = typer.Option(
False,
Expand All @@ -601,7 +602,8 @@ def convert(
genome: Optional[str] = typer.Option(
None,
"--genome",
help="10x input: the genome to keep (picks the group in a Cell Ranger 2 file)",
help="10x input: the genome to keep (picks the group or subfolder of a "
"Cell Ranger 2 matrix)",
),
var_names: str = typer.Option(
"gene_symbols",
Expand Down Expand Up @@ -657,11 +659,13 @@ def convert(
Convert a store to another format.

The output's extension picks the format: .h5ad, .zarr, or .h5 for a 10x
Genomics matrix (Cell Ranger's *_feature_bc_matrix.h5). Between h5ad and
zarr every element is copied, streamed, keeping the source's chunking and
compression where the target can express them. A 10x matrix is read as
scanpy.read_10x_h5(gex_only=False) reads it, keeping integer counts, and
written as Cell Ranger writes it.
Genomics matrix (Cell Ranger's *_feature_bc_matrix.h5); --to 10x-mtx
writes a Matrix Market folder. Between h5ad and zarr every element is
copied, streamed, keeping the source's chunking and compression where the
target can express them. A 10x matrix is read as scanpy's read_10x_h5 or
read_10x_mtx (gex_only=False) reads it, keeping integer counts, and
written as Cell Ranger writes it. MTX folders from Cell Ranger and
STARsolo, velocyto's included, are read in two streaming passes.

Nothing is written until every check has passed, and the output is moved
into place only once complete: a failed conversion leaves nothing
Expand All @@ -677,6 +681,9 @@ def convert(
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
adata convert raw_feature_bc_matrix.h5 -o rna.h5ad --feature-type "Gene Expression"
adata convert counts.h5ad -o for_cellbender.h5 --layer counts
adata convert Solo.out/Gene/filtered -o counts.h5ad
adata convert Solo.out/Velocyto/raw -o velocyto.h5ad
adata convert counts.h5ad -o counts_mtx --to 10x-mtx --layer counts
"""
entries = list(ctx.args)
recast_flags = _given(ctx, _RECAST_FLAGS)
Expand Down
Loading
Loading