Merge pull request #34 from cellgeni/feat/convert-10x-h5

convert: read and write 10x Genomics HDF5 matrices

Author: Claptar

.github/workflows/tests.yml

@@ -127,3 +127,44 @@ jobs:
           if ran < 150:
               sys.exit(f"expected at least 150 cases to run, got {ran}")
           EOF
+
+  # scanpy and PyTables read the 10x files `convert` writes, and scanpy is
+  # the reference for what it reads. Each runs in its own uv environment.
+  external-readers:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    name: 10x h5 against scanpy and PyTables
+
+    permissions:
+      contents: read
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      - name: Install dependencies (frozen)
+        run: uv sync --extra dev --frozen
+
+      - name: Run external reader tests
+        env:
+          # An environment that will not build must fail the job, not skip.
+          ADATA_REQUIRE_VERSION_FIXTURES: "1"
+        run: |
+          uv run pytest -v -W default tests/test_external_readers.py \
+            -m integration --junitxml=external-results.xml
+
+      - name: Confirm the cases actually ran
+        if: always()
+        run: |
+          python - <<'PY'
+          import sys, xml.etree.ElementTree as ET
+          root = ET.parse("external-results.xml").getroot()
+          suite = root.find("testsuite") if root.tag == "testsuites" else root
+          ran = int(suite.get("tests", 0)) - int(suite.get("skipped", 0))
+          print(f"{ran} external reader cases ran")
+          if ran < 8:
+              sys.exit(f"expected at least 8 cases to run, got {ran}")
+          PY

CHANGELOG.md

@@ -23,8 +23,25 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
   conversion asked for is refused rather than ignored.
 
   Formats are registered in `adata.conversion`, each with the converters
-  that read and write it; 10x HDF5, MTX folders, loom and others follow as
-  separate issues under #20.
+  that read and write it; MTX folders, loom and others follow as separate
+  issues under #20.
+
+- **`convert` reads and writes 10x Genomics HDF5 matrices**
+  ([#23](https://github.com/cellgeni/adata-cli/issues/23)): Cell Ranger's
+  `*_feature_bc_matrix.h5`, v2 and v3+, and the same layout from Space
+  Ranger, Xenium and CellBender. Reading gives what
+  `scanpy.read_10x_h5(gex_only=False)` gives, column for column -- checked
+  against scanpy itself in CI -- except that `X` keeps the integer counts
+  and the run's metadata is kept in `uns["cellranger"]`. `--feature-type`
+  keeps one type, `--split-feature-types` writes one output per type from a
+  single pass over the matrix, `--genome` picks a genome, and `--var-names`
+  and `--make-unique` do what scanpy's options of the same name do.
+
+  Writing produces the Cell Ranger 3+ layout from X, `raw/X` or any layer
+  (`--layer`), for CellBender, SoupX, DropletUtils and Seurat: gzip and
+  fixed-width strings, which PyTables -- and so CellBender -- needs. A file
+  imported by `convert` exports with its metadata intact, and the round
+  trip reproduces Cell Ranger's file dataset for dataset.
 
 ### Changed
 
@@ -43,6 +60,13 @@ Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
 
 ### Fixed
 
+- **`recast --layout` left its h5ad output up to three times too large.**
+  The streaming transpose kept its intermediate buckets in the output and
+  deleted them when done, but HDF5 never returns a deleted dataset's space
+  to the file. On a 20,000 x 2,000 matrix at 5% the output was 55 MB where
+  the in-memory transpose wrote 17 MB. The buckets now go in a separate
+  file beside the output, removed afterwards; Zarr outputs were unaffected.
+
 - **Writing an h5ad with a `None` in it to Zarr failed.** anndata stores
   `None` as a null, which on HDF5 is an empty dataspace with no value to
   read, and copying one raised `TypeError: float() argument must be a string

README.md

@@ -8,7 +8,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
 - Auto-detects `.h5ad` files vs `.zarr` directories
 - Chunked processing for dense and sparse matrices (CSR/CSC)
 - Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
-- Converts between HDF5 and Zarr (v2 and v3) in either direction with `adata convert`
+- Converts between HDF5 and Zarr (v2 and v3) in either direction with `adata convert`, and to and from 10x Genomics HDF5 matrices
 - Rich terminal output with progress indicators, kept on stderr so results pipe cleanly
 
 **Documentation: [cellgeni.github.io/adata-cli](https://cellgeni.github.io/adata-cli/)**
@@ -56,7 +56,7 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
 - `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
 - `split` – write one store per distinct value of an annotation column, with a CSV manifest.
 - `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
-- `convert` – change a store's format: h5ad ↔ Zarr v2/v3, streaming, with the output written beside its destination and moved into place only once complete.
+- `convert` – change a store's format: h5ad ↔ Zarr v2/v3, and 10x Genomics HDF5 (Cell Ranger's `*_feature_bc_matrix.h5`) in and out, streaming, with the output written beside its destination and moved into place only once complete.
 - `recast` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced. Called `convert` until 0.7.0.
 - `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
 - `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
@@ -98,6 +98,8 @@ how you make them agree.
 ```bash
 adata convert data.h5ad -o data.zarr
 adata convert data.zarr -o data.h5ad --compression gzip
+adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad --feature-type "Gene Expression"
+adata convert counts.h5ad -o for_cellbender.h5 --layer counts
 ```
 
 ## Documentation

docs/COMMANDS.md

@@ -94,6 +94,8 @@ adata convert data.h5ad -o data.zarr
 adata convert data.zarr -o data.h5ad --compression gzip
 adata convert old.zarr -o new.zarr --zarr-format 3
 adata convert data.h5ad -o data.zarr --overwrite
+adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
+adata convert counts.h5ad -o for_cellbender.h5 --layer counts
 ```
 
 | Flag | Meaning |
@@ -104,12 +106,16 @@ adata convert data.h5ad -o data.zarr --overwrite
 | `--zarr-format` | Zarr version to write; defaults to the source's, else 3 |
 | `--compression` | `none`, `gzip` or `lzf` for h5ad; `none`, `gzip`, `zstd` or `blosc` for Zarr |
 | `--overwrite` | Replace an existing output |
-| `--force` | Convert despite a lossy or inflating result |
+| `--force` | Convert despite a lossy, inflating or placeholder result |
 
 | Format | Extension | Read | Write |
 |---|---|---|---|
 | `h5ad` | `.h5ad` | yes | yes |
 | `zarr` | `.zarr` (v2 or v3) | yes | yes |
+| `10x-h5` | `.h5` | Cell Ranger 2 and 3+, Space Ranger, Xenium | Cell Ranger 3+ layout |
+
+Formats other than h5ad and Zarr convert to and from AnnData only: to turn a
+10x file into another format, convert it to `.h5ad` first.
 
 Between h5ad and Zarr every element is copied as it is, streamed, so the
 store's size is not a limit. Chunking carries over, and so does compression
@@ -128,6 +134,58 @@ successor is finished.
 To change a matrix's dtype, layout or density instead, use
 [`recast`](#recast).
 
+### 10x Genomics HDF5
+
+Cell Ranger's `filtered_feature_bc_matrix.h5` and `raw_feature_bc_matrix.h5`,
+and the same layout from Space Ranger, Xenium and CellBender. The file is
+recognised by its content, whatever its name.
+
+```bash
+adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
+adata convert raw_feature_bc_matrix.h5 -o rna.h5ad --feature-type "Gene Expression"
+adata convert multiome.h5 -o sample.h5ad --split-feature-types
+adata convert cellranger2.h5 -o mouse.h5ad --genome mm10
+adata convert counts.h5ad -o out.h5 --layer counts
+```
+
+| Flag | Meaning |
+|---|---|
+| `--genome` | Keep this genome's features; picks the group in a Cell Ranger 2 file holding several |
+| `--var-names` | `gene_symbols` (default) or `gene_ids` |
+| `--make-unique` | Suffix repeated var names as `var_names_make_unique` does |
+| `--feature-type` | Keep one feature type, e.g. `"Gene Expression"` |
+| `--split-feature-types` | Write one output per type, `OUT.<type>.h5ad` |
+| `--layer` | Export: the matrix to write, `X` (default), `raw/X` or a layer |
+| `--gene-ids-column` | Export: var column with gene ids (default `gene_ids`) |
+| `--gene-names-column` | Export: var column with gene names (default the var names) |
+| `--feature-types-column` | Export: var column with feature types (default `feature_types`) |
+
+Reading gives what `scanpy.read_10x_h5(gex_only=False)` gives, column for
+column: `gene_ids`, `feature_types`, every other per-feature field Cell Ranger
+recorded (`genome`, and `pattern`, `read` and so on for antibody and CRISPR
+features), `probe_ids` for a probe-based (Flex) matrix, and
+`filtered_barcodes` in obs where present. Unlike scanpy, every feature type
+is kept unless `--feature-type` says otherwise, and `X` keeps Cell Ranger's
+integer counts, where scanpy converts them to float32; run `adata recast
+out.h5ad X --inplace --dtype float32` if a tool needs floats. Repeated var
+names are kept and reported, as scanpy does. The run's metadata, which scanpy
+drops, goes in `uns["cellranger"]`.
+
+The matrix is stored genes x cells in CSC, which is byte for byte cells x
+genes in CSR, so `data`, `indices` and `indptr` stream straight across;
+`--split-feature-types` reads them once for every output.
+
+Writing produces the Cell Ranger 3+ layout for CellBender, SoupX,
+DropletUtils, Seurat and anything else that reads Cell Ranger output: gzip
+compression and fixed-width strings, as Cell Ranger writes them, since
+PyTables (which CellBender reads with) cannot read the variable-length kind.
+A CSC or dense matrix is transposed or sparsified on the way, streaming.
+Each feature needs an id and a type: if var has no `gene_ids` and no
+`feature_types` column (or the columns named by the flags above), the export
+is refused, and `--force` writes the names as ids and `Gene Expression` as
+the type. A matrix of floats is written as it is, with a warning. A file
+imported by `convert` exports with its run metadata and tag fields intact.
+
 ## `recast`
 
 Change a matrix's dtype, layout or density. Counts held as float64 cost twice

docs/TESTING.md

@@ -5,11 +5,12 @@ uv sync --extra dev
 uv run pytest                      # everything
 uv run pytest -m "not integration" # fast: no environment building, ~40s
 uv run pytest -m perf              # just the tracing complexity guards
-uv run pytest -m integration       # compatibility across anndata releases
+uv run pytest -m integration       # anndata releases, scanpy and PyTables
 ```
 
 CI runs the unit suite on Python 3.12 and 3.13 with a **90% coverage floor**,
-and the compatibility suite as a separate job.
+and the anndata compatibility suite and the 10x external-reader suite as
+separate jobs.
 
 ## How the suite is organised
 
@@ -25,6 +26,8 @@ and the compatibility suite as a separate job.
 | `test_commands_phase2.py`, `test_commands_coverage.py`, `test_cli.py` | Command surfaces and error paths |
 | `test_convert.py` | `convert` between every backend pair, read back by anndata; its refusals, staging, and the forwarding of recast's old flags |
 | `test_recast.py` | `recast` against scipy: casts, transposes, density changes and every refusal |
+| `test_tenx.py` | 10x HDF5 import and export against Cell Ranger-shaped fixtures (v2, v3, multimodal, probe-based): mapping, filters, round trips, refusals |
+| `test_external_readers.py` | 10x conversions checked by scanpy and PyTables in their own uv environments (`integration`) |
 | `test_performance.py` | Complexity guards -- what an operation costs, not how long it takes (see below) |
 | `test_benchmark_harness.py` | That the benchmark's measurement is trustworthy |
 | `test_subset.py`, `test_export.py`, `test_import.py`, `test_info_read.py`, `test_zarr.py`, `test_query.py` | Per-feature unit tests |
@@ -235,6 +238,7 @@ should scale with, scale only that, and hold everything else fixed.
 | `export array`, `sparse` (both paths), `dict`, `image` | elements, nnz, keys, pixels | linear |
 | `import dataframe`, `array`, `sparse`, `dict`, `image` | rows, elements, nnz, keys, pixels | linear in writes |
 | h5ad to zarr (`subset`), `convert` both directions | n_obs | linear |
+| `convert` 10x import (whole, `--split-feature-types`), export | n_barcodes | linear |
 | `recast --dtype`, `--layout csc`, `--layout dense` | nnz, n_obs | linear; the streaming transpose's **peak memory** is bounded |
 
 The `slow` marker is on the three guards that build a 65,536-row store; they still gate

docs/index.md

@@ -52,6 +52,7 @@ adata split  data.h5ad --by sample -o per_sample/
 adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 
 adata convert data.h5ad -o data.zarr
+adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
 adata recast  data.h5ad X -o small.h5ad --dtype float32
 
 adata create new.h5ad --n-obs 5000 --n-var 2000
@@ -66,6 +67,8 @@ CI against stores written by anndata 0.8, 0.9, 0.10, 0.11, 0.12 and 0.13, in
 both formats. Both backends are supported in both directions: `adata convert
 in.h5ad -o out.zarr` converts a whole store, and `subset`, `split` and
 `concat` convert as they write when the output's backend differs.
+`convert` also reads and writes 10x Genomics HDF5 matrices (Cell Ranger's
+`*_feature_bc_matrix.h5`), matching what `scanpy.read_10x_h5` reads.
 
 | Element | Read | Write |
 |---|---|---|

src/adata/cli.py

@@ -503,6 +503,27 @@ def concat(
     "compression": "--compression",
     "force": "--force",
     "overwrite": "--overwrite",
+    "genome": "--genome",
+    "var_names": "--var-names",
+    "make_unique": "--make-unique",
+    "feature_type": "--feature-type",
+    "split_feature_types": "--split-feature-types",
+    "layer": "--layer",
+    "gene_ids_column": "--gene-ids-column",
+    "gene_names_column": "--gene-names-column",
+    "feature_types_column": "--feature-types-column",
+}
+
+#: Flags that only make sense for format conversion; with recast's flags,
+#: they mean two commands' worth of request at once.
+_FORMAT_ONLY_FLAGS = {
+    "from_format": "--from",
+    "to_format": "--to",
+    **{
+        name: flag
+        for name, flag in _CONVERT_FLAGS.items()
+        if flag not in ("--force", "--zarr-format")
+    },
 }
 
 
@@ -532,7 +553,7 @@ def convert(
         None,
         "--output",
         "-o",
-        help="Output path; its extension picks the format (.h5ad, .zarr)",
+        help="Output path; its extension picks the format (.h5ad, .zarr, .h5)",
         dir_okay=True,
         file_okay=True,
     ),
@@ -567,6 +588,51 @@ def convert(
         "--overwrite",
         help="Replace an existing output",
     ),
+    genome: Optional[str] = typer.Option(
+        None,
+        "--genome",
+        help="10x input: the genome to keep (picks the group in a Cell Ranger 2 file)",
+    ),
+    var_names: str = typer.Option(
+        "gene_symbols",
+        "--var-names",
+        help="10x input: name features by gene_symbols or gene_ids",
+    ),
+    make_unique: bool = typer.Option(
+        False,
+        "--make-unique",
+        help="10x input: suffix repeated var names, as var_names_make_unique does",
+    ),
+    feature_type: Optional[str] = typer.Option(
+        None,
+        "--feature-type",
+        help="10x input: keep only this feature type, e.g. 'Gene Expression'",
+    ),
+    split_feature_types: bool = typer.Option(
+        False,
+        "--split-feature-types",
+        help="10x input: write one output per feature type, OUT.<type>.h5ad",
+    ),
+    layer: Optional[str] = typer.Option(
+        None,
+        "--layer",
+        help="10x output: the matrix to write: X (default), raw/X, or a layer",
+    ),
+    gene_ids_column: Optional[str] = typer.Option(
+        None,
+        "--gene-ids-column",
+        help="10x output: var column holding gene ids (default gene_ids)",
+    ),
+    gene_names_column: Optional[str] = typer.Option(
+        None,
+        "--gene-names-column",
+        help="10x output: var column holding gene names (default: var_names)",
+    ),
+    feature_types_column: Optional[str] = typer.Option(
+        None,
+        "--feature-types-column",
+        help="10x output: var column holding feature types (default feature_types)",
+    ),
     # Deprecated: recast's flags, so `convert` keeps working as it did until
     # 1.0.0. Hidden from --help; the old matrix paths arrive in ctx.args.
     inplace: bool = typer.Option(False, "--inplace", hidden=True),
@@ -580,9 +646,12 @@ def convert(
     """
     Convert a store to another format.
 
-    The output's extension picks the format: .h5ad or .zarr. Converting
-    between them copies every element, streamed, keeping the source's
-    chunking and compression where the target can express them.
+    The output's extension picks the format: .h5ad, .zarr, or .h5 for a 10x
+    Genomics matrix (Cell Ranger's *_feature_bc_matrix.h5). Between h5ad and
+    zarr every element is copied, streamed, keeping the source's chunking and
+    compression where the target can express them. A 10x matrix is read as
+    scanpy.read_10x_h5(gex_only=False) reads it, keeping integer counts, and
+    written as Cell Ranger writes it.
 
     Nothing is written until every check has passed, and the output is moved
     into place only once complete: a failed conversion leaves nothing
@@ -595,21 +664,16 @@ def convert(
         adata convert data.h5ad -o data.zarr
         adata convert data.zarr -o data.h5ad --compression gzip
         adata convert data.zarr -o v3.zarr --zarr-format 3
+        adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
+        adata convert raw_feature_bc_matrix.h5 -o rna.h5ad --feature-type "Gene Expression"
+        adata convert counts.h5ad -o for_cellbender.h5 --layer counts
     """
     entries = list(ctx.args)
     recast_flags = _given(ctx, _RECAST_FLAGS)
 
     try:
         if recast_flags:
-            mixed = _given(
-                ctx,
-                {
-                    "from_format": "--from",
-                    "to_format": "--to",
-                    "compression": "--compression",
-                    "overwrite": "--overwrite",
-                },
-            )
+            mixed = _given(ctx, _FORMAT_ONLY_FLAGS)
             if mixed:
                 raise ValueError(
                     f"{', '.join(mixed)} cannot be combined with "
@@ -677,6 +741,15 @@ def convert(
                 compression=compression,
                 force=force,
                 overwrite=overwrite,
+                genome=genome,
+                var_names=var_names,
+                make_unique=make_unique,
+                feature_type=feature_type,
+                split_feature_types=split_feature_types,
+                layer=layer,
+                gene_ids_column=gene_ids_column,
+                gene_names_column=gene_names_column,
+                feature_types_column=feature_types_column,
             ),
         )
     except Exception as e: