Merge main into the Nemotron 3 Ultra 384-GPU SFT recipe branch

Resolve the conflict in recipes/nemotronh/h100/__init__.py: keep main's
removal of the retired Nemotron Nano v2 and NemotronH entries (#6370) and
this branch's rename of the Ultra SFT recipe to
nemotron_3_ultra_sft_384gpu_h100_bf16_openmathinstruct2_packed_config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>

Author: kamran-nvidia

.github/ISSUE_TEMPLATE/model_support_request.yml

@@ -27,9 +27,9 @@ body:
       description: Pick the closest fit for the requested model.
       options:
         - Dense decoder-only (e.g. Llama, Qwen, Gemma, Mistral, Nemotron)
-        - MoE decoder-only (e.g. DeepSeek V2/V3, OLMoE, Qwen3-MoE, MiniMax-M2)
+        - MoE decoder-only (e.g. DeepSeek V3, OLMoE, Qwen3-MoE, MiniMax-M2)
         - VLM / multimodal (e.g. Qwen-VL, Gemma-VL, Nemotron-VL)
-        - SSM / hybrid (e.g. Mamba, Nemotron-H)
+        - SSM / hybrid (e.g. Mamba, Nemotron 3)
         - Encoder-decoder (e.g. T5)
         - Other / unsure
     validations:

.github/actions/test-template/action.yml

@@ -177,6 +177,9 @@ runs:
           set -e
           export GH_TOKEN=${{ inputs.github-token }}
 
+          # Keep the FlashAttention generation policy scoped to CI tests.
+          export NVTE_FLASH_ATTN_V2=1 NVTE_FLASH_ATTN_V3=0 NVTE_FLASH_ATTN_V4=0
+
           if [[ "${{ contains(inputs.runner, 'gcp') }}" == "true" ]]; then
             export NCCL_ENV_PLUGIN=none
             export NCCL_NET_PLUGIN=none

.main.commit

@@ -1 +1 @@
-8949def81eb744b9219e2d794c80265928c40a5b
+e0c5e18673bf4d4fd89cf87c4f1afbe521fe297f

3rdparty/Megatron-LM

@@ -221,34 +221,28 @@ For a deeper dive into conversion design and advanced usage, see the [models REA
 
 Megatron Bridge provides out-of-the-box bridges and training recipes for a wide range of models, built on top of base model architectures from [Megatron Core](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core). Refer to the [models directory](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models) for the full list of model bridges.
 
-> **Deprecation notice:** Gemma 1 (2B/7B), Gemma 2 (2B/9B/27B), Llama 2,
-> Mistral 7B, Mistral Small 3 24B, Nemotron H v1 (4B/8B/47B/56B), and the
-> legacy `NemotronForCausalLM` bridge previously documented for Nemotron-4 340B
-> are no longer actively maintained or tested against current upstream
-> checkpoints. They will be removed in Megatron Bridge 0.7.0.
-
 | Family | Supported variants |
 |----------------|--------------------|
 | [**BAGEL**](docs/models/bagel/bagel.md) | BAGEL-7B-MoT (requires the tested MCore `dev` revision and `bagel` extra) |
 | [**Bailing**](docs/models/bailing/index.md) | Ling 2.0 / Ling MoE V2 (Bailing) |
 | [**BERT**](docs/models/bert/index.md) | Megatron-Style BERT (`MegatronBertForMaskedLM`) |
-| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V2 / V2 Lite (deprecated), DeepSeek V3, DeepSeek V4 / V4 Flash |
+| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V3, DeepSeek V4 / V4 Flash |
 | [**Diffusion**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/diffusion/models) | FLUX, LLaDA 1.5, Nemotron-Labs Diffusion, WAN |
 | **Ernie** | [Ernie 4.5 MoE](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/ernie), [Ernie 4.5 VL MoE](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/ernie_vl) |
 | [**EXAONE**](docs/models/exaone/exaone.md) | EXAONE 4.0, EXAONE 4.5 VL, K-EXAONE MoE |
 | [**Falcon**](docs/models/falcon/index.md) | Falcon H1 |
-| [**Gemma**](docs/models/gemma/index.md) | Gemma / Gemma 2 (deprecated), Gemma 3, Gemma 3-VL, Gemma 4 (26B-A4B MoE / 31B dense), Gemma 4-VL (26B-A4B MoE) |
+| [**Gemma**](docs/models/gemma/index.md) | Gemma 3, Gemma 3-VL, Gemma 4 (26B-A4B MoE / 31B dense), Gemma 4-VL (26B-A4B MoE) |
 | [**GLM**](docs/models/glm/index.md) | GLM-4.5 / GLM-4.7 / GLM-4.7-Flash, GLM-4.5V, GLM-5 / GLM-5.1 / GLM-5.2 / [GLM-5.3](docs/models/glm/glm5-2.md) |
 | [**GPT-OSS**](docs/models/gpt_oss/index.md) | GPT-oss |
 | [**HY V3**](https://huggingface.co/tencent/Hy3-preview-Base) | Hy3 preview-Base (HF → Megatron checkpoint conversion) |
 | [**Kimi**](docs/models/kimi/index.md) | Kimi K2, Kimi-K2.5-VL, Kimi K3 |
-| [**Llama**](docs/models/llama/index.md) | Llama 2 (deprecated), Llama 3 / 3.1 / 3.2 / 3.3 |
+| [**Llama**](docs/models/llama/index.md) | Llama 3 / 3.1 / 3.2 / 3.3 |
 | [**MiniMax**](docs/models/minimax/index.md) | MiniMax-M2 / M2.5 / M2.7, MiniMax-M3 |
-| [**Mistral**](docs/models/mistral/index.md) | Mistral 7B / Small 3 24B (deprecated), Ministral 3 (3B/8B/14B) |
+| [**Mistral**](docs/models/mistral/index.md) | Ministral 3 (3B/8B/14B) |
 | [**Xiaomi-MiMo**](docs/models/mimo/index.md) | Xiaomi-MiMo, MiMo-V2-Flash |
 | [**Moonlight**](docs/models/moonlight/index.md) | Moonlight |
 | [**Muse Glimmer**](docs/models/muse_glimmer/muse-glimmer.md) | Muse Glimmer 30B |
-| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron H v1 (deprecated), Nemotron Nano v2 (deprecated), Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Llama Nemotron (deprecated), Nemotron Nano v2 VL (deprecated), Nemotron-3 Nano Omni |
+| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Nemotron-3 Nano Omni |
 | [**OLMoE**](docs/models/olmoe/index.md) | OLMoE |
 | [**Qwen**](docs/models/qwen/index.md) | Qwen2 / Qwen2.5, Qwen3, Qwen3-MoE, Qwen3 Next, Qwen3.5 (dense/MoE), Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, Qwen3.6-VL, Qwen2 Audio, Qwen2.5-Omni, Qwen3-Omni, Qwen3-ASR |
 | [**Sarvam**](docs/models/sarvam/index.md) | Sarvam |

README.md

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:26.08-py3
+ARG BASE_IMAGE=nvcr.io/nvidia/pytorch:26.09-py3
 FROM ${BASE_IMAGE} AS base
 
 ENV PATH="/opt/venv/bin:/usr/local/bin:$PATH" \

docker/Dockerfile.ci

@@ -61,7 +61,9 @@ override-dependencies = [
     "pyzmq==25.1.2",                            # @URL: https://github.com/NVIDIA/NeMo/issues/13625
     "tiktoken>=0.9.0",                          # let megatron-hub win over nemo-toolkit
     "datasets==4.8.5",
-    "pyarrow>=21.0.0",                          # datasets 4.8.5 needs >=21; override export-deploy's <21 cap (not exercised by training)
+    # datasets 4.8.5 needs >=21, overriding export-deploy's <21 cap.
+    # PyArrow 26 requires NumPy 2; preserve MBridge's NumPy 1 runtime.
+    "pyarrow>=21.0.0,<26",
     "fsspec[http]>=2023.1.0,<=2024.9.0",
     "transformers; sys_platform == 'never'",    # Defer to transformers in MBridge install
     "packaging>=25.0",

docker/common/fw_pyproject.toml

@@ -1347,35 +1347,35 @@
           "workflow": "pretrain"
         },
         {
-          "bridge_commit": "330f87c0a391eb2770cbf2306352cc807211d7ba",
+          "bridge_commit": "b2ebece6a78fbee6a734655cc6993c13a08b60a4",
           "command_topologies": [
             {
-              "gpus_per_node": "1",
+              "gpus_per_node": "8",
               "nodes": "1",
               "recipe": "gemma3_vl_4b_sft_config",
               "sequence_length": "4096",
-              "total_gpus": 1
+              "total_gpus": 8
             }
           ],
           "commands": [
-            "./scripts/training/train.sh --nodes 1 --gpus-per-node 1 --recipe gemma3_vl_4b_sft_config --mode sft --dataset cord-v2 --pretrained_checkpoint work/model-verification/gemma-3-4b-it/imported-megatron/iter_0000000 --max_steps 10 --seq_length 4096 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gemma-3-4b-it/sft-checkpoints --save_interval 10 logger.log_interval=1 logger.log_throughput=true"
+            "./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe gemma3_vl_4b_sft_config --mode sft --dataset cord-v2 --pretrained_checkpoint work/model-verification/gemma-3-4b-it/imported-megatron/iter_0000000 --max_steps 10 --seq_length 4096 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gemma-3-4b-it/sft-checkpoints --save_interval 10 logger.log_interval=1 logger.log_throughput=true"
           ],
           "dimensions": {},
           "enabled_features": {},
           "entry_id": "gemma-3-4b-it-sft-h100",
-          "expected_result": "Pending exact-revision CORD-v2 full-SFT verification with the public Gemma 3 VL 4B H100 recipe. The run must complete at least ten optimizer steps with finite loss, no skipped or NaN iterations, all five metrics, and a reloadable final full-model checkpoint.\n",
+          "expected_result": "On 8 H100s, the TP1/PP1 recipe completes exactly ten exact-revision CORD-v2 full-SFT steps at global batch size 32 and sequence length 4096. Loss is finite from 1.163850 to 0.143927 with no skipped or NaN iterations. The ten-step metric window includes the first-step cold-start cost by policy, and the final iteration-10 checkpoint saves successfully.\n",
           "hardware": "H100",
-          "last_verified": null,
+          "last_verified": "2026-10-09",
           "metrics": {
-            "final_loss": null,
-            "initial_loss": null,
-            "last_10_steps_model_tflops_per_gpu_avg": null,
-            "last_10_steps_step_time_ms_avg": null,
-            "last_10_steps_tokens_per_second_per_gpu_avg": null
+            "final_loss": 0.143927,
+            "initial_loss": 1.16385,
+            "last_10_steps_model_tflops_per_gpu_avg": 22.45,
+            "last_10_steps_step_time_ms_avg": 5461.01,
+            "last_10_steps_tokens_per_second_per_gpu_avg": 3000.178
           },
           "precision": "bf16",
           "source_pointer": "items.sft.H100",
-          "status": "unverified",
+          "status": "verified",
           "variant": null,
           "workflow": "sft"
         },
@@ -1481,7 +1481,7 @@
       "min_transformers_version": "5.8.0",
       "slug": "gemma-3-4b-it",
       "source_card": "examples/model_verification_cards/gemma-3-4b-it/card.yaml",
-      "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Gemma 3 4B Instruct exact-checkpoint CPU and TP2/PP2 GPU import, CPU and TP2/PP2 GPU export, manual multimodal forward parity, and deterministic TP2/PP2 inference are verified. SFT, post-SFT export/inference, and PEFT remain pending. Pretraining and checkpoint resume are unsupported because there is no public exact-model pretraining recipe. Long-context SFT is unsupported because the required dense local/global attention biases currently exclude context parallelism and in-batch THD packing. Existing toy-model tests are not used as exact-model evidence.\n",
+      "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Gemma 3 4B Instruct exact-checkpoint CPU and TP2/PP2 GPU import, CPU and TP2/PP2 GPU export, manual multimodal forward parity, deterministic TP2/PP2 inference, and 8-GPU H100 full SFT are verified. Post-SFT export/inference and PEFT remain pending. Pretraining and checkpoint resume are unsupported because there is no public exact-model pretraining recipe. Long-context SFT is unsupported because the required dense local/global attention biases currently exclude context parallelism and in-batch THD packing. Existing toy-model tests are not used as exact-model evidence.\n",
       "title": "gemma_3_4b_it"
     },
     {
@@ -4818,7 +4818,7 @@
           "workflow": "sft_export_inference"
         },
         {
-          "bridge_commit": "44f3031a2e77ff455eb128efbaa19322ad6397a6",
+          "bridge_commit": "ffeb5a7ae817a5a6ccfa3850d7dee937faad7f0b",
           "command_topologies": [
             {
               "gpus_per_node": "8",
@@ -4837,15 +4837,15 @@
             "sequence_packing": "offline"
           },
           "entry_id": "nemotron-3-super-120b-a12b-sft-long-context-h100",
-          "expected_result": "On 16 H100s, the dedicated TP1/PP8/CP2/EP2 recipe completes exactly ten packed 32K SFT steps with pad-4 alignment, full recompute, and full optimizer CPU offload. Loss is finite from 1.168852 to 1.008992 with no skipped or NaN iterations. The ten-step metric window includes the first-step cold-start cost by policy, the resolved configuration is persisted, and the final model-only checkpoint saves successfully.\n",
+          "expected_result": "On 16 H100s, the dedicated TP1/PP8/CP2/EP2 recipe completes exactly ten packed 32K SFT steps with pad-4 alignment, full recompute, and full optimizer CPU offload. Loss is finite from 1.168852 to 1.011637 with no skipped or NaN iterations. The ten-step metric window includes the first-step cold-start cost by policy, the resolved configuration is persisted, and the final model-only checkpoint saves successfully.\n",
           "hardware": "H100",
-          "last_verified": "2026-07-24",
+          "last_verified": "2026-10-08",
           "metrics": {
-            "final_loss": 1.008992,
+            "final_loss": 1.011637,
             "initial_loss": 1.168852,
-            "last_10_steps_model_tflops_per_gpu_avg": 5.58,
-            "last_10_steps_step_time_ms_avg": 132859.58,
-            "last_10_steps_tokens_per_second_per_gpu_avg": 30.83
+            "last_10_steps_model_tflops_per_gpu_avg": 6.89,
+            "last_10_steps_step_time_ms_avg": 123114.4,
+            "last_10_steps_tokens_per_second_per_gpu_avg": 33.27
           },
           "precision": "bf16",
           "source_pointer": "items.sft_long_context.H100",

docs/_static/model-verification/catalog-v1.json

@@ -63,7 +63,7 @@ from megatron.bridge import AutoBridge
 
 # Load with specific settings
 bridge = AutoBridge.from_hf_pretrained(
-    "meta-llama/Llama-2-7b-hf",
+    "meta-llama/Meta-Llama-3-8B",
     torch_dtype=torch.float16,
     device_map="auto",
     trust_remote_code=True,
@@ -346,7 +346,7 @@ AutoBridge.to_megatron_model(load_weights: bool = True, hf_path: str | Path | No
 AutoBridge.load_hf_weights(model: list[MegatronModule], hf_path: str | Path | None = None) -> None
 
 # Megatron → HF conversion
-AutoBridge.export_hf_weights(model: list[MegatronModule], cpu: bool = False, show_progress: bool = True, conversion_tasks: Optional[list[WeightConversionTask]] = None) -> Iterable[HFWeightTuple]
+AutoBridge.export_hf_weights(model: list[MegatronModule], cpu: bool = False, show_progress: bool = True, conversion_tasks: Optional[list[WeightConversionTask]] = None, current_pp_stage_only: bool = False) -> Iterable[HFWeightTuple]
 AutoBridge.save_hf_pretrained(model: list[MegatronModule], path: str | Path, show_progress: bool = True) -> None
 AutoBridge.save_hf_weights(model: list[MegatronModule], path: str | Path, show_progress: bool = True) -> None
 

docs/bridge-guide.md

@@ -352,7 +352,28 @@ for name, weight in bridge.export_hf_weights(megatron_model, cpu=True, show_prog
 
 Point your inference engine (e.g., vLLM) to `"/path/to/hf_export"`.
 
-### B) Zero-copy streaming via ZMQ (fast refit, colocated)
+### B) Pipeline-stage-local streaming for RL weight synchronization
+
+When inference workers collectively receive shards from every training pipeline stage, avoid replicating the full
+HF stream across PP ranks:
+
+```python
+for name, weight in bridge.export_hf_weights(
+    megatron_model,
+    show_progress=False,
+    current_pp_stage_only=True,
+):
+    send_to_inference_workers(name, weight)
+```
+
+- `current_pp_stage_only` changes ownership only across the PP axis.
+- TP/EP ranks in the owning stage still gather shards and perform normal HF layout conversion.
+- Combining one representative stream from every PP stage reconstructs the normal complete export.
+- Source-only HF passthrough tensors are assigned to PP stage 0; duplicate Megatron parameters use the lowest PP owner.
+
+This mode is opt-in. Omitting the flag preserves the complete stream on every rank.
+
+### C) Zero-copy streaming via ZMQ (fast refit, colocated)
 
 Stream tensors from the training side to your inference runtime without writing to disk. The transport is ZMQ peer-to-peer with async send/recv and ping‑pong buffers for overlap; Ray is used only for lightweight coordination. This replaces the earlier ad‑hoc per‑tensor IPC handle passing and aligns with the refactor in [NVIDIA-NeMo/RL#1267](https://github.com/NVIDIA-NeMo/RL/pull/1267).