Merge branch 'main' into pr4_omni3_packseq_sequence_utils

Author: hbhflw2000

docker/Dockerfile.ci

@@ -91,10 +91,12 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     # the uv sync below. Point the compilers at the CCCL/libcu++ headers (under cccl/ in this base
     # image, not the default CUDA include path) and scope the build to the target archs.
     export FLASH_MLA_DISABLE_SM90=1 NVCC_THREADS=16 \
-        CFLAGS="-I/usr/local/cuda/include/cccl${CFLAGS:+ $CFLAGS}" \
-        CXXFLAGS="-I/usr/local/cuda/include/cccl${CXXFLAGS:+ $CXXFLAGS}" && \
+        CFLAGS="-I/usr/local/cuda/include/cccl${CFLAGS:+ $CFLAGS} -DNDEBUG" \
+        CXXFLAGS="-I/usr/local/cuda/include/cccl${CXXFLAGS:+ $CXXFLAGS} -DNDEBUG" && \
     # Reinstall nvidia-cutlass-dsl in system site packages
     # uv install can run into non-deterministic install issues
+    # Keep this pin compatible with cuDNN Frontend; mismatched versions can break
+    # its CUTLASS-backed kernels.
     pip uninstall -y nvidia-cutlass-dsl nvidia-cutlass-dsl-libs-base nvidia-cutlass-dsl-libs-cu13 && \
     pip install nvidia-cutlass-dsl[cu13]==4.5.0 && \
     if [ "$MCORE_TRIGGERED_TESTING" = "true" ]; then \
@@ -110,4 +112,4 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     ln -sf "$(ldconfig -p | awk '/libcudart\.so\.[0-9]+ /{print $NF; exit}')" /opt/venv/lib/python3.12/site-packages/tilelang/lib/libcudart_stub.so && \
     uv cache prune ${UV_CACHE_PRUNE_ARGS}
 
-COPY --chmod=644 . /opt/Megatron-Bridge
+COPY --chmod=644 . /opt/Megatron-Bridge

docker/common/fw_pyproject.toml

@@ -103,7 +103,7 @@ override-dependencies = [
     "nvidia-cusparse-cu12; sys_platform == 'never'",
     "nvidia-cusparselt-cu12; sys_platform == 'never'",
     "nvidia-nccl-cu12; sys_platform == 'never'",
-    "nvidia-cudnn-frontend==1.24.0",
+    "nvidia-cudnn-frontend==1.26.0",
     "nvidia-cutlass-dsl[cu13]; sys_platform == 'never'", # Rely on system site packages install
     "tensorrt-llm; sys_platform == 'never'",
     "tensorrt; sys_platform == 'never'",

docker/common/import_check_skip.txt

@@ -23,6 +23,8 @@ deep_ep_cpp
 hybrid_ep_cpp
 torch_tensorrt
 quack  # imports nvidia-cutlass-dsl MLIR libs, which dlopen libcuda at import
+peft # transitively imports optional CUDA-backed modules that dlopen libcuda
+transformer_engine # dlopens libcuda at import
 
 # Requires the libfuse system library (multi-storage-client FUSE backend; optional):
 mfusepy

docs/fern/versions/_nav_order.yml

@@ -17,6 +17,7 @@
 "./versions/nightly/pages/models/llama/llama2.mdx": 1
 "./versions/nightly/pages/models/mimo/mimo.mdx": 1
 "./versions/nightly/pages/models/minimax/minimax-m2.mdx": 1
+"./versions/nightly/pages/models/minimax/minimax-m3.mdx": 2
 "./versions/nightly/pages/models/mistral/mistral.mdx": 1
 "./versions/nightly/pages/models/moonlight/moonlight.mdx": 1
 "./versions/nightly/pages/models/nemotron/llama-nemotron.mdx": 1

docs/fern/versions/nightly.yml

@@ -90,6 +90,8 @@ navigation:
   - contents:
     - page: MiniMax-M2
       path: ./nightly/pages/models/minimax/minimax-m2.mdx
+    - page: MiniMax-M3
+      path: ./nightly/pages/models/minimax/minimax-m3.mdx
     section: MiniMax
   - contents:
     - page: Mistral

docs/fern/versions/nightly/pages/models/minimax/index.mdx

@@ -5,3 +5,4 @@ MiniMax model documentation is organized by model variant.
 | Variant | Guide |
 |---------|-------|
 | MiniMax-M2 / M2.5 / M2.7 | [minimax-m2.md](minimax-m2.md) |
+| MiniMax-M3 | [minimax-m3.md](minimax-m3.md) |

docs/fern/versions/nightly/pages/models/minimax/minimax-m3.mdx

@@ -0,0 +1,54 @@
+# MiniMax-M3
+
+[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) is a natively multimodal sparse MoE model from MiniMaxAI (428B total, ~23B active parameters). Megatron Bridge supports the M3 *language model* through the `MiniMaxM3Bridge`: the text backbone of the `MiniMaxM3SparseForConditionalGeneration` checkpoint is converted to a Megatron-Core `GPTModel`.
+
+## Supported Variants
+
+| Variant | Hugging Face ID | Notes |
+|---------|-----------------|-------|
+| MiniMax-M3 | `MiniMaxAI/MiniMax-M3` | Language model only (bf16 weights) |
+
+## Architecture Notes
+
+- Mixed dense/MoE decoder: 60 layers, the first 3 dense, the rest with 128 routed experts (top-4) plus one shared expert.
+- Sigmoid router with expert-bias correction and `routed_scaling_factor` applied to the normalized top-k weights (DeepSeek-V3-style routing); the checkpoint's FP32 router weights remain FP32 during import.
+- SwiGLU-OAI activation in every MLP and expert: clamped gate/up projections with a `+1` linear offset, mapped to `activation_func_clamp_value` / `glu_linear_offset` (same mechanism as GPT-OSS).
+- Gemma-style RMSNorm (`x * (1 + w)`) on every norm, mapped to `layernorm_zero_centered_gamma`.
+- GQA attention (64 query heads, 4 KV heads) with per-head QK RMSNorm and partial RoPE (64 of 128 head channels rotated, theta 5e6).
+
+## Known Limitations
+
+- **Language model only.** The CLIP-style vision tower, multimodal projector, and patch-merge MLP are not mapped.
+- **Full attention only.** The lightning-indexer block-sparse attention branch (`self_attn.index_*` weights) is not mapped; the Megatron model runs full causal attention on every layer. Block selection keeps `index_topk_blocks * index_block_size` (2048) key tokens per query, so full attention is mathematically identical up to that sequence length and an approximation beyond it.
+- **MTP modules are not mapped.** The released checkpoint advertises `num_nextn_predict_layers` in its config but ships no `mtp.*` weights.
+- **Auxiliary-loss scoring differs for training.** The recipes use MCore's token-global load-balancing loss over normalized sigmoid scores. Hugging Face leaves its optional router loss disabled by default and uses softmax scores when enabled.
+- The MXFP8 variant (`MiniMaxAI/MiniMax-M3-MXFP8`) is not supported; use the bf16 checkpoint.
+
+## Conversion
+
+```python
+from megatron.bridge import AutoBridge
+
+bridge = AutoBridge.from_hf_pretrained("MiniMaxAI/MiniMax-M3", trust_remote_code=True)
+provider = bridge.to_megatron_provider()
+```
+
+The bridge imports the language backbone from the multimodal checkpoint.
+Standalone Hugging Face checkpoint export is not supported because the vision,
+projector, lightning-indexer, and MTP weights are intentionally not mapped;
+native Megatron checkpoints and in-memory weight round-trip verification are
+supported.
+
+## Examples
+
+For real-checkpoint Slurm conversion, inference, hardware requirements, and
+validated parallelism settings, see the [MiniMax-M3 examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/minimax/minimax_m3/README.md).
+
+## Recipes
+
+Pretraining and packed-sequence (THD) SFT recipes are available under [`src/megatron/bridge/recipes/minimax`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/recipes/minimax) (`minimax_m3_pretrain_256gpu_h100_bf16_config`, `minimax_m3_sft_128gpu_h100_bf16_config`), using a TP=2 / PP=4 / EP=32 baseline layout.
+
+## Related Implementation
+
+- Bridge implementation: [`src/megatron/bridge/models/minimax_m3`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/minimax_m3)
+- Examples: [`examples/models/minimax/minimax_m3`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/minimax/minimax_m3)

docs/fern/versions/nightly/pages/models/qwen/qwen2.5-vl.mdx

@@ -61,29 +61,25 @@ Before training, ensure the following environment variables are set.
 
 ### Full Finetuning
 
-Example usage for full parameter finetuning:
+Example usage for 3B full-parameter finetuning on one H100:
 
 ```bash
-uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/qwen/qwen_vl/finetune_qwen25_vl.py \
---pretrained-checkpoint $MEGATRON_MODEL_PATH \
---recipe qwen25_vl_3b_finetune_config \
---dataset-type hf \
-dataset.source.dataset_name=cord_v2 \
-train.global_batch_size=<batch size> \
-train.train_iters=<number of iterations> \
-logger.wandb_project=<optional wandb project name> \
-logger.wandb_save_dir=$SAVE_DIR \
-checkpoint.save=$SAVE_DIR/<experiment name>
+uv run python -m torch.distributed.run --standalone --nproc_per_node=1 \
+  scripts/training/run_recipe.py \
+  --recipe qwen25_vl_3b_sft_config \
+  --dataset vlm-hf \
+  --step_func vlm_step \
+  checkpoint.pretrained_checkpoint=$MEGATRON_MODEL_PATH \
+  checkpoint.save=$SAVE_DIR/qwen25-vl-3b-sft \
+  dataset.source.dataset_name=cord_v2 \
+  train.global_batch_size=2 \
+  train.train_iters=NUM_ITERATIONS
 ```
 
 Note:
-- The `--recipe` parameter selects the model size configuration. Available options:
-  - `qwen25_vl_3b_finetune_config` - for 3B model
-  - `qwen25_vl_7b_finetune_config` - for 7B model
-  - `qwen25_vl_32b_finetune_config` - for 32B model
-  - `qwen25_vl_72b_finetune_config` - for 72B model
+- Full-SFT recipes are `qwen25_vl_{3b,7b,32b,72b}_sft_config`; their defaults require 1, 2, 16, and 32 GPUs respectively.
 - The config file `examples/models/qwen/qwen_vl/conf/qwen25_vl_pretrain_override_example.yaml` contains a list of arguments
-  that can be overridden in the command. For example, you can set `train.global_batch_size=&lt;batch size&gt;` in the command.
+  that can be overridden in the command. For example, you can set `train.global_batch_size=BATCH_SIZE` in the command.
 - The dataset format should be JSONL with conversation format (see dataset section below).
 - After training, you can run inference with `hf_to_megatron_generate_vlm.py` by supplying the trained megatron checkpoint.
   You can also export the trained checkpoint to Hugging Face format.
@@ -92,14 +88,15 @@ Note:
 Parameter-efficient finetuning (PEFT) using LoRA or DoRA is supported. You can use the `--peft_scheme` argument to enable PEFT training:
 
 ```bash
-uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/qwen/qwen_vl/finetune_qwen25_vl.py \
---pretrained-checkpoint $MEGATRON_MODEL_PATH \
---recipe qwen25_vl_3b_finetune_config \
---peft_scheme lora \
---dataset-type hf \
-dataset.source.dataset_name=cord_v2 \
-train.global_batch_size=<batch size> \
-checkpoint.save=$SAVE_DIR/<experiment name>
+uv run python -m torch.distributed.run --standalone --nproc_per_node=1 \
+  scripts/training/run_recipe.py \
+  --recipe qwen25_vl_3b_peft_config \
+  --dataset vlm-hf \
+  --step_func vlm_step \
+  --peft_scheme lora \
+  checkpoint.pretrained_checkpoint=$MEGATRON_MODEL_PATH \
+  checkpoint.save=$SAVE_DIR/qwen25-vl-3b-lora \
+  dataset.source.dataset_name=cord_v2
 ```
 
 PEFT options:
@@ -112,16 +109,21 @@ You can also combine PEFT with freeze options to control which components are tr
 
 Example with LoRA and freeze options:
 ```bash
-uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/qwen/qwen_vl/finetune_qwen25_vl.py \
---pretrained-checkpoint $MEGATRON_MODEL_PATH \
---recipe qwen25_vl_3b_finetune_config \
---peft_scheme lora \
-model.freeze_language_model=True \
-model.freeze_vision_model=False \
-model.freeze_vision_projection=False \
-checkpoint.save=$SAVE_DIR/<experiment name>
+uv run python -m torch.distributed.run --standalone --nproc_per_node=1 \
+  scripts/training/run_recipe.py \
+  --recipe qwen25_vl_3b_peft_config \
+  --dataset vlm-hf \
+  --step_func vlm_step \
+  --peft_scheme lora \
+  checkpoint.pretrained_checkpoint=$MEGATRON_MODEL_PATH \
+  model.freeze_language_model=True \
+  model.freeze_vision_model=False \
+  model.freeze_vision_projection=False \
+  checkpoint.save=$SAVE_DIR/qwen25-vl-3b-lora
 ```
 
+For local processor-native image JSONL and sharded Energon preparation patterns, see the [multimodal data tutorials](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/tutorials/data/README.md).
+
 ## Example Datasets
 
 Megatron Bridge supports various vision-language dataset examples which can be used to finetune Qwen 2.5 VL:

docs/fern/versions/nightly/pages/recipe-usage.mdx

@@ -23,7 +23,7 @@ Training mode follows the recipe and dataset type:
 | LLM pretraining or continued pretraining | `GPTDatasetConfig` | `pretrain()` | No checkpoint for from-scratch runs; use `checkpoint.load` for full resume or `checkpoint.pretrained_checkpoint` for model-weight initialization |
 | Full SFT | `GPTSFTDatasetConfig` for local JSONL or Hugging Face source data | `finetune()` | Use `checkpoint.pretrained_checkpoint` for the base model, or `checkpoint.load` for a full native Megatron resume |
 | PEFT / LoRA / DoRA | Same as SFT, plus `cfg.peft` | `finetune()` | `checkpoint.pretrained_checkpoint` is required for the frozen base model; `checkpoint.load` resumes adapter training |
-| VLM SFT or PEFT | `DirectHFSFTDatasetConfig` + builder, Energon, or a specialized/preloaded provider | `finetune()` with a VLM step function | Use the model-specific checkpoint guidance in the recipe or model docs |
+| VLM SFT or PEFT | `DirectHFSFTDatasetConfig` with a Hugging Face source, including its JSON loader, or Energon | `finetune()` with a VLM step function | Use the model-specific checkpoint guidance in the recipe or model docs |
 
 For dataset fields, prefer `seq_length` in Bridge examples. LLM pretraining uses `GPTDatasetConfig` with `data_path`, `blend`, or `blend_per_split`; SFT and PEFT use `dataset_root` for local JSONL data. Do not use `data_path` for SFT/PEFT JSONL roots.