Skip to content

convert: 10x and STARsolo Matrix Market folders, in and out - #49

Merged
Claptar merged 4 commits into
mainfrom
feat/convert-10x-mtx
Oct 15, 2026
Merged

Claptar merged 4 commits into
mainfrom
feat/convert-10x-mtx

Conversation

@Claptar

@Claptar Claptar commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Fixes #24. Part 3 of 3, stacked on #48 (and through it on #47).

adata convert FOLDER -o out.h5ad reads every folder shape Cell Ranger and STARsolo write:

  • Cell Ranger 3 (features.tsv);
  • Cell Ranger 2 (genes.tsv, with a subfolder per genome);
  • STARsolo Gene and GeneFull;
  • STARsolo Velocyto, where spliced, unspliced and ambiguous become layers and X is a copy of spliced.

Each file may be plain or gzipped, under one GEO-style prefix or none. --to 10x-mtx writes a gzipped Cell Ranger 3 folder back.

adata convert Solo.out/Gene/filtered -o counts.h5ad
adata convert Solo.out/Velocyto/raw -o velocyto.h5ad
adata convert filtered_gene_bc_matrices -o mouse.h5ad --genome mm10
adata convert counts.h5ad -o counts_mtx --to 10x-mtx --layer counts
  • Two streaming passes:
    • the first checks the folder (barcode and feature counts, and every entry) before any output exists;
    • the second writes X, straight through when the entries are in barcode order, as Cell Ranger and STARsolo write them.
  • scanpy parity: the result matches scanpy.read_10x_mtx(gex_only=False) column for column, except that integer counts stay integers.
  • Flags: the 10x HDF5 flags apply (--genome, --var-names, --make-unique, --feature-type, --split-feature-types), plus --compression.
  • Refusals leave nothing behind, not even a temporary copy or a scratch file.

Tests

  • tests/test_mtx.py uses hand-written fixtures for every layout. It covers entry order, duplicates, real values, prefixes, genomes, velocyto, run metadata in both directions, refusals, the export round trip (byte-identical after decompression), CSR/CSC/dense export, and overwriting.

  • tests/test_external_readers.py (integration) checks against scanpy for:

    • Cell Ranger 3 with a prefix, with and without --make-unique;
    • Cell Ranger 2, STARsolo and velocyto;
    • scanpy reading back our export.

    All of these pass locally.

  • Perf guards: linear work for sorted, shuffled and split input, flat write calls on import, and flat read calls on export.

  • Benchmark: new case import-10x-mtx, against scanpy's read_10x_mtx followed by a write. At the ci size (50M entries, warm cache, run locally):

Wall time Peak RSS
adata-cli 10.5 s 572 MB
scanpy 10.6 s 3.1 GB

Python's peak was 222 MB at both 5M and 50M entries.

Not yet done: a run on a real STARsolo folder from a cellgeni ticket.

🤖 Generated with Claude Code

Claptar and others added 2 commits October 15, 2026 16:28
Fixes #24. `adata convert FOLDER -o out.h5ad` reads the folder shapes
Cell Ranger and STARsolo write -- Cell Ranger 3 (features.tsv), Cell
Ranger 2 (genes.tsv, a subfolder per genome), STARsolo Gene/GeneFull,
and STARsolo Velocyto (spliced/unspliced/ambiguous -> layers, X a copy
of spliced) -- each file plain or gzipped, under one GEO-style prefix
or none. `--to 10x-mtx` writes a gzipped Cell Ranger 3 folder back.

Import streams through formats/mtx in two passes: the first checks the
folder (barcode and feature counts, every entry) before any output
exists, the second writes X, straight through when entries arrive in
barcode order and through on-disk buckets when not. The result matches
scanpy's read_10x_mtx(gex_only=False) column for column, checked in
test_external_readers, except that integer counts stay integers. The
10x HDF5 flags apply (--genome, --var-names, --make-unique,
--feature-type, --split-feature-types), plus --compression.

Also a benchmark case against scanpy, perf guards (linear work, flat
write calls on import, flat read calls on export), and docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A work directory reused across tiers handed one tier's sidecar to the
next: a ci run measured the smoke tier's MTX folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 21:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T22:00:15.827401Z 660cf1e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (py3.13)

1 406 tests  +62   1 406 ✅ +62   3m 12s ⏱️ +2s
    1 suites ± 0       0 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit f9e0839. ± Comparison against base commit 27bf873.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (py3.12)

1 406 tests  +62   1 406 ✅ +62   3m 50s ⏱️ + 1m 47s
    1 suites ± 0       0 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit f9e0839. ± Comparison against base commit 27bf873.

♻️ This comment has been updated with latest results.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 660cf1ef78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/adata/conversion/mtx.py
Comment thread src/adata/conversion/mtx.py
@Claptar
Claptar changed the base branch from feat/mtx-codec to main October 15, 2026 22:24
@Claptar
Claptar merged commit 6e42649 into main Oct 15, 2026
6 checks passed
@Claptar
Claptar deleted the feat/convert-10x-mtx branch October 15, 2026 22:26
@Claptar Claptar mentioned this pull request Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Convert: MTX and 10x MTX folders, import and export (streaming)

2 participants