Repository navigation
Stream Matrix Market files: a codec behind import/export sparse - #48
Conversation
`import sparse` built a Python tuple per entry and `export sparse` read indices and data once per row. Both now go through formats/mtx, which reads and writes a block of about 16 MB at a time: - import makes two passes: the first counts entries per row and checks the whole file before the store is touched, the second appends in CSR order -- directly when rows arrive in order, through on-disk buckets (core/csr_build) when they do not. Gzipped files are read as such. - export reads indices and data once per block, and gzips when the output ends in .gz. User-visible changes: - integer values stay integers (int32, int64 when needed) and real ones are float64; all were float32, which loses counts above 2^24; - symmetric, skew-symmetric, hermitian, complex and dense files are refused; a symmetric file was silently read as half its matrix; - duplicate entries are summed, as scipy does; indptr widens to int64 past 2^31 entries instead of overflowing; - HDF5 data is gzipped, and a parent group made on the way is a dict; - exported floats read back exactly, and whole numbers print as ints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91b4c7ee05
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Duplicate sums: the data dtype is chosen in pass 1, before the store is touched, but duplicates are summed only in pass 2. Two int32 values whose sum is not int32 therefore failed after `import sparse --inplace` had removed the matrix it was replacing. Pass 1 now also bounds each row's total magnitude, which no summed entry can exceed, and a file that may hold duplicates is typed by that bound. - The int64 limits: fromstring saturates instead of failing, so a value equal to INT64_MAX or INT64_MIN was refused as overflow. Such a block is now checked token by token, and only a real overflow is refused. - The formatter: magnitudes are computed in uint64, so INT64_MIN (whose int64 abs is itself, and hung the digit loop) and uint64 values above INT64_MAX (which wrapped negative) are written exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part 2 of 3 for #24. Stacked on #47.
import sparsebuilt a Python tuple per entry, andexport sparsereadindicesanddataonce per row. Both now go through a new streaming codec,formats/mtx.py, which reads and writes about 16 MB of text at a time. Its gzip-aware block I/O and vectorised formatting are informats/textio.py.Import makes two passes:
core/csr_build) when they don't.Export reads
indicesanddataonce per block, and gzips when the output name ends in.gz.User-visible changes (for the 0.8.0 CHANGELOG)
symmetric,skew-symmetric,hermitian,complexand densearrayfiles are refused. A symmetric file used to be read silently as half its matrix.indptrwidens to int64 past 2^31 entries, instead of overflowing.dict.Tests
tests/test_mtx_codec.pychecks against scipy'smmreadas the oracle. It covers every field, both gzip and plain files, sorted and shuffled input, duplicates, every refusal, line endings, and that the file is read exactly twice.test_formats.pynow uses scipy as the oracle instead of the removed private reader.export sparseread calls stay flat asn_obsgrows. This fails on the old code: 64× more calls over a 64× span.import sparsewrite calls stay flat asn_obsgrows.🤖 Generated with Claude Code