Motivation. The engine allocates full (N, K, K) normal-equation caches (JTJ_cache, JTr_cache) and (N, M, K) Jacobians. For millions of spectra this can exhaust RAM, and a mid-run cancel/crash loses all progress.
Approach. Add an optional chunk size so VBFthread/VBFengine fit the map in memory-bounded batches of rows, streaming results back into SpectraStore per chunk. Optionally persist a checkpoint (completed row range + params) so a cancelled/interrupted large fit can resume.
Scope.
- Chunked driver loop (reuse the existing single-batch path per chunk; results identical since spectra are independent).
- Progress already per-spectrum via progress_callback — extend to aggregate across chunks (vbf_thread.py already tracks processed_spectra).
- Optional checkpoint file (params + success mask + last completed index).
- Tests: chunked vs single-shot fit give identical results; resume-from-checkpoint reproduces a full run.
Risks. Must keep per-chunk results bit-identical to the all-at-once fit (they are, since the batch is only a vectorization, not a coupling). Checkpoint I/O should be optional and off by default.
Motivation. The engine allocates full (N, K, K) normal-equation caches (JTJ_cache, JTr_cache) and (N, M, K) Jacobians. For millions of spectra this can exhaust RAM, and a mid-run cancel/crash loses all progress.
Approach. Add an optional chunk size so VBFthread/VBFengine fit the map in memory-bounded batches of rows, streaming results back into SpectraStore per chunk. Optionally persist a checkpoint (completed row range + params) so a cancelled/interrupted large fit can resume.
Scope.
Risks. Must keep per-chunk results bit-identical to the all-at-once fit (they are, since the batch is only a vectorization, not a coupling). Checkpoint I/O should be optional and off by default.