v4.9.0 - torch compile speedup, MegaCache import/export, NextLat, Explorative Modeling, block swap optimisations

Breaking Change

• Diffusers' group offload is removed in PR 3150
• Torch inductor backend switched from python to c++ wrapper
• MiniMax Music 3 training objectives changed; don't upgrade mid-run

Features

• Uses torch inductor's C++ wrapper by default now for a large performance improvement for LTX 2.5 torch compiled training

• Load and export torch Mega-Cache from/to huggingface hub or local disk via new --dynamo.. options

• Built-in RVC trainer for music model dataset expansion, configurable inside dataloader

• NextLatent (NextLat: https://arxiv.org/abs/2511.05963) trains a model to "form compact world models with coherent belief states", regularising the hidden states so the transition through the model is more smooth

• Explorative Modeling (XM: https://explorative-modeling.github.io) improves output fidelity by training on only the "best" out of several possible loss candidates on each step. Trades more training-time compute for better results during inference.

• Better MiniMax Music 3 training objectives

  • continuation mode trains the model to continue a generation from a given full starting prefix, up to a certain max length
  • random crop mode without full prefixing/continuation now anchors positions better

• RamTorch now shares quantised weights across multiple GPU processes, reducing system VRAM consumption

Bugfixes

• MiniMax H3 single file checkpoint loader / dataset (torchcodec compatibility) fixes
• Generated checkpoint README/model card refined and includes MiniMax LM training mode info now
• Musubi block swap is optimised a little by allowing it to completely replace streamed weights instead of moving them back to GPU
• Musubi block swap and gradient checkpointing now work together correctly, not increasing VRAM more than required
• torch compile now enables correctly (eg. regional or full dynamo compile modes)
• SDNQ with torch compile now enabled and working fully for all configurations
• torch compile reduce-overhead mode now works with activation checkpointing
• LTX 2.5 graph breaks for torch compile are removed, incl when using Flash Attention 2

What's Changed

• add NextLat and Explorative Modeling by @bghira in #3133
• update lyrics scraper to remove on-page junk text by @bghira in #3136
• torchcodec + H3 single file fixes by @bghira in #3135
• minimax music3: cache rvq codes as if its a VAE by @bghira in #3137
• Fix audio RVQ caching for LM training by @bghira in #3138
• Fix H3 audio-only XM and NextLat runtime paths by @bghira in #3139
• add XM and NextLat info to model card, plus audio-specific fixes by @bghira in #3140
• Add causal continuation windows for MiniMax Music 3 LM training by @bghira in #3141
• Fix MiniMax Music LM crop boundaries and metadata by @bghira in #3142
• Fix Musubi block residency across checkpointed backward by @bghira in #3148
• Share quantized RamTorch storage across local ranks by @bghira in #3149
• Remove Diffusers group offload integration by @bghira in #3150
• Stream checkpointed Musubi blocks through a packed H2D ring by @bghira in #3151
• Fix regional compilation not executing under mixed precision by @hjinnkim in #3147
• Fix SDNQ compilation across FakeTensor and checkpoint contexts by @bghira in #3154
• Document LTX regional compilation performance caveat by @bghira in #3152
• Overlap large prefetched batch transfers with CUDA compute by @bghira in #3153
• Handle reduce-overhead with activation checkpointing by @bghira in #3155
• Hoist LTX attention mask preparation, removing FA2 graph break by @bghira in #3156
• Make LTX regional compilation fully capturable by @bghira in #3157
• Persist and reuse Torch compiler Mega-Caches by @bghira in #3158
• data_transforms dataloader block for automatically RVCing inputs by @bghira in #3143
• minimax music continuation objective fixes by @bghira in #3161
• merge by @bghira in #3162

Full Changelog: v4.8.0...v4.9.0