diff --git a/README.md b/README.md index a03980cd15..2602e15dd0 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ ## πŸ“£ News - [08/19/2026] **Megatron Bridge 0.6.0 released!** Highlights include GLM-5.2 with cuDNN fused DSA for 128K THD-packed context-parallel training, new LLM/VLM/diffusion models, more reliable VLM data and sequence packing, improved checkpoint and LoRA workflows, unified training and conversion launchers, and refreshed performance recipes. Huge thanks to our community contributors! See the [full release notes](https://github.com/NVIDIA-NeMo/Megatron-Bridge/releases/tag/v0.6.0). -- [08/11/2026] **NVIDIA Nemotron 3.5 Lightning is released!** Read the technical blog [here](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/). See the [Nemotron 3.5 Lightning README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3/lightning/README.md) for details. +- [08/11/2026] **NVIDIA Nemotron 3.5 Lightning is released!** Read the technical blog [here](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/). See the [Nemotron 3.5 Lightning README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_5_lightning/README.md) for details. - [07/31/2026] **[K-EXAONE-2](https://huggingface.co/LGAI-EXAONE/K-EXAONE-2.0-750B-A37B) is now supported**! Day-0 support for LG AI Research's K-EXAONE-2 (750B-A37B MoE) is now available on the [k-exaone-2 branch](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/k-exaone-2). Users can convert checkpoints, run inference, fine-tune with SFT or PEFT (LoRA), and use the NVFP4 inference-optimized checkpoint released alongside the model. See the branch documentation for setup instructions and examples. @@ -31,13 +31,13 @@ - [05/20/2026] [**DeepSeek V4**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/deepseek_v4) is now merged on **main**! See the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/deepseek_v4/README.md) for conversion and inference details. -- [05/20/2026] [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) day-0 branch support is now merged on **main**! The 30B-A3B MoE multimodal model supports image, video, audio, and text workflows with checkpoint conversion, inference, SFT, and PEFT (LoRA) examples. Read the [NVIDIA Blog](https://blogs.nvidia.com/blog/nemotron-3-nano-omni-multimodal-ai-agents/) and see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) for the full walkthrough. +- [05/20/2026] [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) day-0 branch support is now merged on **main**! The 30B-A3B MoE multimodal model supports image, video, audio, and text workflows with checkpoint conversion, inference, SFT, and PEFT (LoRA) examples. Read the [NVIDIA Blog](https://blogs.nvidia.com/blog/nemotron-3-nano-omni-multimodal-ai-agents/) and see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) for the full walkthrough. - [05/19/2026] [**Nemotron-Labs Diffusion**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron_labs_diffusion) is now supported on **main** with autoregressive-to-diffusion conversion, continuous pretraining, checkpoint conversion, and inference workflows. Read the [NVIDIA Research blog](https://research.nvidia.com/publication/2026-05_nemotron-labs-diffusion-tri-mode-language-model-unifying-autoregressive) for the tri-mode language model overview. - [05/06/2026] [**Gemma 4 VL 26B-A4B**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/gemma/gemma4_vl) is now supported! Checkpoint conversion, SFT, and PEFT (LoRA) recipes for Google's MoE vision-language model (26B total / 4B active params, 128 experts top-k=8, dual sliding/global attention with K=V tying on full-attention layers) are available on **main**. See the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/gemma/gemma4_vl/README.md) for the full walkthrough. -- [04/28/2026] Day 0 support for [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16), a 30B-A3B MoE multimodal model that jointly processes image, video, audio, and text. Checkpoint conversion, SFT, and LoRA recipes are available on **main** β€” see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) for the full walkthrough. +- [04/28/2026] Day 0 support for [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16), a 30B-A3B MoE multimodal model that jointly processes image, video, audio, and text. Checkpoint conversion, SFT, and LoRA recipes are available on **main** β€” see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) for the full walkthrough. - [04/19/2026] [**Qwen3.6-35B-A3B**](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is now supported! Qwen3.6 uses the same architecture as Qwen3.5 VL MoE (`Qwen3_5MoeForConditionalGeneration`) and works with the existing [Qwen3.5-VL bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/qwen_vl) out of the box β€” no code changes needed. HFβ†’Megatron conversion and inference verified. @@ -55,13 +55,13 @@ - [03/31/2026] **Agent Skills for Megatron Bridge!** We've added a [`skills/`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/skills) directory with structured guides that AI coding agents (Cursor, Claude Code, Codex, etc.) can use to help you add model support, set up dev environments, tune performance, and more. Try them out, and PRs to improve or add new skills are very welcome! -- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). +- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). - [03/12/2026] **Deprecating Python 3.10 support:** We're officially dropping Python 3.10 support with the upcoming 0.4.0 release. Downstream applications must raise their lower boundary to 3.12 to stay compatible with Megatron-Bridge. - [12/16/2025] [Mind Lab](https://alpha.macaron.im/mindlab) successfully used Megatron-bridge and [VeRL](https://github.com/volcengine/verl) to trained GRPO Lora for Trillion-parameter model on 64 H800 - See their [techblog](https://alpha.macaron.im/mindlab/research/building-trillion-parameter-reasoning-rl-with-10-gpus). -- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3/nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) +- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3_nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) ## Overview diff --git a/docs/_static/model-verification/catalog-v1.json b/docs/_static/model-verification/catalog-v1.json index e99a3a2ccf..bbbbda2bc5 100644 --- a/docs/_static/model-verification/catalog-v1.json +++ b/docs/_static/model-verification/catalog-v1.json @@ -3430,59 +3430,49 @@ "title": "moonlight_16b_a3b" }, { - "architecture": "NemotronH_Nano_Omni_Reasoning_V3", - "base_container": "nvcr.io/nvidia/nemo:26.06", - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "architecture": "NemotronHForCausalLM", + "base_container": "nvcr.io/nvidia/nemo:26.08", + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", "entries": [ { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", - "command_topologies": [ - { - "expert_parallel": "1", - "expert_tensor_parallel": "1", - "nodes": "1", - "pipeline_parallel": "1", - "tensor_parallel": "1" - } - ], - "commands": [ - "./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-megatron-clean --torch-dtype bfloat16 --tp 1 --pp 1 --ep 1 --etp 1 --trust-remote-code" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-hf-to-megatron-cpu", - "expected_result": "The offline, immutable-revision CPU import exits successfully after mapping 7,333 model parameters and creates a reloadable iter_0000000 torch_dist checkpoint with 33,015,546,816 parameters on the single model parallel rank.\n", + "entry_id": "nemotron-3-nano-30b-a3b-hf-to-megatron-cpu", + "expected_result": "This workflow remains unverified and requires a public run against the pinned Hugging Face revision with all applicable verification gates.\n", "hardware": null, - "last_verified": "2026-07-30", + "last_verified": null, "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_cpu", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "hf_to_megatron_cpu" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", "command_topologies": [ { - "expert_parallel": "4", + "expert_parallel": "8", "expert_tensor_parallel": "1", "gpus_per_node": "8", "nodes": "1", "pipeline_parallel": "1", - "tensor_parallel": "2", + "tensor_parallel": "1", "total_gpus": 8 } ], "commands": [ - "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 4 --etp 1 --trust-remote-code --low-memory-save" + "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-timeout-minutes 120 --trust-remote-code --low-memory-save" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-hf-to-megatron-gpu", - "expected_result": "The command exits successfully at TP2/PP1/EP4/ETP1 and creates a reloadable iter_0000000 checkpoint. The paired strict GPU export contains the source checkpoint's exact 7,349-key set, shapes, dtypes, and values.\n", + "entry_id": "nemotron-3-nano-30b-a3b-hf-to-megatron-gpu", + "expected_result": "The pinned one-node 8-H100 import exits successfully at TP1/PP1/EP8/ETP1, persists iter_0000000, and reloads it for inference. A separate public GPU round-trip at the same topology exhaustively compares all 6,243 exported BF16 weights against the immutable source: 6,243 of 6,243 match with no skipped FP8 tensors or mismatches.\n", "hardware": null, - "last_verified": "2026-07-30", + "last_verified": "2026-08-25", "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_gpu", @@ -3491,19 +3481,19 @@ "workflow": "hf_to_megatron_gpu" }, { - "bridge_commit": "7783fc356b7d4768525eabf9825f9108c6544f4c", + "bridge_commit": "60442bb9adb5435b47db22c6c20aacdf772fbddc", "command_topologies": [ { "nodes": "1" } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-megatron-clean/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-hf-export-clean --torch-dtype bfloat16 --trust-remote-code" + "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-30b-a3b/cpu-hf-export --torch-dtype bfloat16 --trust-remote-code" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-megatron-to-hf-cpu", - "expected_result": "CPU export completes in 14 indexed shards totaling 66,031,270,520 bytes. Its exact comparison contains all 7,349 source tensors and 33,015,632,238 values, with identical keys, shapes, dtypes, and values and no dtype conversions. Transformers reloads the local custom-code export natively with the source-equivalent warning set: one generated vision summary buffer is missing and four source-only audio/vision buffers are unexpected.\n", + "entry_id": "nemotron-3-nano-30b-a3b-megatron-to-hf-cpu", + "expected_result": "The synchronous CPU export writes a complete indexed checkpoint. All 6,243 source keys and shapes match across 31,577,940,288 values. The 46 source FP32 tensors cast to BF16 are the 23 per-layer mixer A_log tensors and 23 per-layer mixer D tensors; each equals the export exactly after that declared cast, and every remaining tensor matches exactly at zero tolerance. Native loading succeeds as NemotronHForCausalLM with no missing, unexpected, mismatched, or errored weights, and the tokenizer reloads from the export.\n", "hardware": null, "last_verified": "2026-08-25", "metrics": {}, @@ -3514,25 +3504,25 @@ "workflow": "megatron_to_hf_cpu" }, { - "bridge_commit": "7783fc356b7d4768525eabf9825f9108c6544f4c", + "bridge_commit": "60442bb9adb5435b47db22c6c20aacdf772fbddc", "command_topologies": [ { - "expert_parallel": "4", + "expert_parallel": "8", "expert_tensor_parallel": "1", "gpus_per_node": "8", "nodes": "1", "pipeline_parallel": "1", - "tensor_parallel": "2", + "tensor_parallel": "1", "total_gpus": 8 } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-hf-export-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 4 --etp 1 --trust-remote-code --distributed-save --save-every-n-ranks 1" + "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-30b-a3b/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-save --save-every-n-ranks 1 --trust-remote-code" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-megatron-to-hf-gpu", - "expected_result": "Distributed export completes in 17 indexed shards totaling 66,031,270,520 bytes. All 7,349 tensors and 33,015,632,238 values match the immutable HF source exactly in keys, shapes, dtypes, and values with no dtype conversions. Transformers reloads the local custom-code export natively with the same one missing generated buffer and four unexpected source-only buffers reported when reloading the pinned source itself.\n", + "entry_id": "nemotron-3-nano-30b-a3b-megatron-to-hf-gpu", + "expected_result": "The synchronous TP1/PP1/EP8/ETP1 export writes a complete indexed checkpoint. All 6,243 source keys and shapes match across 31,577,940,288 values. The 69 source FP32 tensors cast to BF16 are the same 46 mixer A_log and D tensors plus 23 per-layer mixer gate e_score_correction_bias tensors. The legacy distributed GPU export model wrapper downcast those gate biases to BF16, while the CPU export preserved them in FP32; all 69 equal the export exactly after this declared cast, and every remaining tensor matches exactly at zero tolerance. Native loading succeeds as NemotronHForCausalLM with no missing, unexpected, mismatched, or errored weights, and the tokenizer reloads from the export.\n", "hardware": null, "last_verified": "2026-08-25", "metrics": {}, @@ -3543,22 +3533,13 @@ "workflow": "megatron_to_hf_gpu" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", - "command_topologies": [ - { - "expert_parallel": "2", - "expert_tensor_parallel": "1", - "pipeline_parallel": "4", - "tensor_parallel": "1" - } - ], - "commands": [ - "uv run python -m torch.distributed.run --standalone --nproc_per_node=8 examples/conversion/compare_hf_and_megatron/compare.py --hf_model_path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron_model_path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --tp 1 --pp 4 --ep 2 --etp 1 --prompt \"The capital of France is \" --trust-remote-code" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-manual-forward-pass", - "expected_result": "The pinned one-step run exits successfully and both implementations predict token ID 6993 (\" Paris\"), but cosine similarity is 0.969760 and therefore does not satisfy the required 0.99 verification gate. The maximum and mean absolute logit differences are 3.527344 and 0.527820, respectively.\n", + "entry_id": "nemotron-3-nano-30b-a3b-manual-forward-pass", + "expected_result": "This workflow remains unverified and requires a public run against the pinned Hugging Face revision with all applicable verification gates.\n", "hardware": null, "last_verified": null, "metrics": {}, @@ -3569,24 +3550,27 @@ "workflow": "manual_forward_pass" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", "command_topologies": [ { - "expert_parallel": "2", + "expert_parallel": "8", "expert_tensor_parallel": "1", - "pipeline_parallel": "4", - "tensor_parallel": "1" + "gpus_per_node": "8", + "nodes": "1", + "pipeline_parallel": "1", + "tensor_parallel": "1", + "total_gpus": 8 } ], "commands": [ - "uv run python -m torch.distributed.run --standalone --nproc_per_node=8 examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py --hf_model_path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron_model_path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --prompt \"The capital of France is\" --max_new_tokens 4 --tp 1 --pp 4 --ep 2 --etp 1" + "./scripts/inference/infer.sh --nodes 1 --gpus-per-node 8 --task legacy-full-prefix-generation --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron_model_path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --tp 1 --pp 1 --ep 8 --etp 1 --prompt \"Explain why the sky appears blue in one sentence.\" --max_new_tokens 32 --apply-chat-template --thinking-mode disabled --trust-remote-code --legacy-full-prefix" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-inference", - "expected_result": "Deterministic greedy decoding returns the exact 4-token result and\nterminates with the model end token. Literal completion: \"\nParis.\"", + "entry_id": "nemotron-3-nano-30b-a3b-inference", + "expected_result": "The public launcher reloaded the pinned BF16 checkpoint and produced exactly 32 generated tokens across generation steps 0 through 31, using the 32-token maximum without an earlier EOS. The literal completion is 'User asks: \"Explain why the sky appears blue in one sentence.\" So we need to give a single sentence explanation. Provide concise answer. Probably: \"'.\n", "hardware": null, - "last_verified": "2026-07-30", + "last_verified": "2026-08-25", "metrics": {}, "precision": "bf16", "source_pointer": "items.inference", @@ -3595,14 +3579,14 @@ "workflow": "inference" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", "command_topologies": [], "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-pretrain-all", - "expected_result": "Megatron Bridge does not publish a pretraining recipe for this multimodal conditional-generation model; the supported package provides supervised CORD v2 SFT and LoRA PEFT workflows.\n", - "hardware": "all", + "entry_id": "nemotron-3-nano-30b-a3b-pretrain-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "hardware": "H100", "last_verified": null, "metrics": { "final_loss": null, @@ -3611,68 +3595,43 @@ "last_10_steps_step_time_ms_avg": null, "last_10_steps_tokens_per_second_per_gpu_avg": null }, - "precision": null, - "source_pointer": "items.pretrain.all", - "status": "unsupported", + "precision": "bf16", + "source_pointer": "items.pretrain.H100", + "status": "unverified", "variant": null, "workflow": "pretrain" }, { - "bridge_commit": "9b69b47a321de14b0453c230dd130e18ae0f00f7", - "command_topologies": [ - { - "gpus_per_node": "8", - "nodes": "1", - "recipe": "nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config", - "total_gpus": 8 - } - ], - "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config --mode sft --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-4k-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, - "enabled_features": { - "moe_dispatcher": "hybridep" - }, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-h100", - "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 full-SFT optimizer steps on eight H100 GPUs at TP2/PP1/CP1/EP8/ETP1 and GBS/MBS 64/4 with natural-routing HybridEP. Execution uses selective MoE/layernorm recompute, fused attention, TE cross entropy, fused weighted squared-ReLU, and router fusion without changing the image-text objective or routing policy. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.281308 to 0.03330838, the final ten steps average 3,603.67 ms and 197.10 TFLOP/s/GPU, sampled peak memory is 76,349 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 at 197.07 final-ten TFLOP/s/GPU with finite loss and zero skipped or NaN iterations.\n", + "enabled_features": {}, + "entry_id": "nemotron-3-nano-30b-a3b-sft-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", "hardware": "H100", - "last_verified": "2026-08-12", + "last_verified": null, "metrics": { - "final_loss": 0.03330838, - "initial_loss": 1.281308, - "last_10_steps_model_tflops_per_gpu_avg": 197.1, - "last_10_steps_step_time_ms_avg": 3603.67, - "last_10_steps_tokens_per_second_per_gpu_avg": 9092.952 + "final_loss": null, + "initial_loss": null, + "last_10_steps_model_tflops_per_gpu_avg": null, + "last_10_steps_step_time_ms_avg": null, + "last_10_steps_tokens_per_second_per_gpu_avg": null }, "precision": "bf16", "source_pointer": "items.sft.H100", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "sft" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", - "command_topologies": [ - { - "expert_parallel": "8", - "expert_tensor_parallel": "1", - "gpus_per_node": "8", - "nodes": "1", - "pipeline_parallel": "1", - "tensor_parallel": "2", - "total_gpus": 8 - }, - {} - ], - "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-4k-checkpoints/iter_0000100 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-hf-export-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 8 --etp 1 --trust-remote-code --distributed-save --not-strict", - "uv run python skills/create-model-verification-card/scripts/verify_hf_inference.py --hf-model work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-hf-export-clean --prompt \"The capital of France is\" --max-new-tokens 32 --chat-template --disable-thinking --trust-remote-code --device cuda --dtype bfloat16" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-export-inference-h100", - "expected_result": "Non-strict export succeeds and writes 6,637 indexed tensors in 17 shards. The CORD v2 recipe intentionally disables sound, so 712 source audio tensors are absent. Native Transformers generation remains unverified because local dynamic-code loading omits configuration_nemotron_h before model weights can reload.\n", + "entry_id": "nemotron-3-nano-30b-a3b-sft-export-inference-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", "hardware": "H100", "last_verified": null, "metrics": {}, @@ -3683,83 +3642,60 @@ "workflow": "sft_export_inference" }, { - "bridge_commit": "eb7cf72392ac07e7a095b87d9b1d6bf2b38c5b6e", - "command_topologies": [ - { - "gpus_per_node": "8", - "nodes": "1", - "recipe": "nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config", - "total_gpus": 8 - } - ], - "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config --mode sft --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-8k-packed-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, - "enabled_features": { - "context_parallel_size": 2, - "sequence_packing": "in_batch" - }, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-long-context-h100", - "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 full-SFT optimizer steps on eight H100 GPUs at TP4/PP1/CP2/EP8/ETP1 and GBS/MBS 64/2 with 8K in-batch packing and the standard all-to-all dispatcher. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.251364 to 0.01020405, the final ten steps average 15,577.9 ms and 93.73 TFLOP/s/GPU, sampled peak memory is 75,197 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 with finite loss and zero skipped or NaN iterations.\n", + "enabled_features": {}, + "entry_id": "nemotron-3-nano-30b-a3b-sft-long-context-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", "hardware": "H100", - "last_verified": "2026-08-11", + "last_verified": null, "metrics": { - "final_loss": 0.01020405, - "initial_loss": 1.251364, - "last_10_steps_model_tflops_per_gpu_avg": 93.73, - "last_10_steps_step_time_ms_avg": 15577.9, - "last_10_steps_tokens_per_second_per_gpu_avg": 4206.986 + "final_loss": null, + "initial_loss": null, + "last_10_steps_model_tflops_per_gpu_avg": null, + "last_10_steps_step_time_ms_avg": null, + "last_10_steps_tokens_per_second_per_gpu_avg": null }, "precision": "bf16", "source_pointer": "items.sft_long_context.H100", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "sft_long_context" }, { - "bridge_commit": "eb7cf72392ac07e7a095b87d9b1d6bf2b38c5b6e", - "command_topologies": [ - { - "gpus_per_node": "8", - "nodes": "1", - "recipe": "nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config", - "total_gpus": 8 - } - ], - "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config --mode lora --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/peft-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" - ], + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "command_topologies": [], + "commands": [], "dimensions": {}, - "enabled_features": { - "moe_dispatcher": "hybridep" - }, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-peft-h100", - "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 LoRA optimizer steps on eight H100 GPUs at TP2/PP1/CP1/EP8/ETP1 and GBS/MBS 64/1 with natural-routing HybridEP. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.282941 to 0.02634745, the final ten steps average 11,621.42 ms and 61.25 TFLOP/s/GPU, sampled peak memory is 44,873 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 with finite loss and zero skipped or NaN iterations.\n", + "enabled_features": {}, + "entry_id": "nemotron-3-nano-30b-a3b-peft-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", "hardware": "H100", - "last_verified": "2026-08-11", + "last_verified": null, "metrics": { - "final_loss": 0.02634745, - "initial_loss": 1.282941, - "last_10_steps_model_tflops_per_gpu_avg": 61.25, - "last_10_steps_step_time_ms_avg": 11621.42, - "last_10_steps_tokens_per_second_per_gpu_avg": 2819.621 + "final_loss": null, + "initial_loss": null, + "last_10_steps_model_tflops_per_gpu_avg": null, + "last_10_steps_step_time_ms_avg": null, + "last_10_steps_tokens_per_second_per_gpu_avg": null }, "precision": "bf16", "source_pointer": "items.peft.H100", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "peft" }, { - "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", "command_topologies": [], "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-checkpoint-resume-all", - "expected_result": "There is no supported pretraining reference workflow for this model, so no model-wide optimizer and RNG checkpoint-resume contract is available for sentinel comparison.\n", - "hardware": "all", + "entry_id": "nemotron-3-nano-30b-a3b-checkpoint-resume-h100", + "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "hardware": "H100", "last_verified": null, "metrics": { "final_loss": null, @@ -3768,65 +3704,101 @@ "last_10_steps_step_time_ms_avg": null, "last_10_steps_tokens_per_second_per_gpu_avg": null }, - "precision": null, - "source_pointer": "items.checkpoint_resume.all", - "status": "unsupported", + "precision": "bf16", + "source_pointer": "items.checkpoint_resume.H100", + "status": "unverified", "variant": null, "workflow": "checkpoint_resume" + }, + { + "bridge_commit": "0480586879f2513958fd8634fc529693ae13e536", + "command_topologies": [ + { + "gpus_per_node": "4", + "nodes": "2", + "recipe": "nemotron_3_nano_pretrain_8gpu_gb300_fp8mx_config", + "sequence_length": "8192", + "total_gpus": 8 + } + ], + "commands": [ + "./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 4 --recipe nemotron_3_nano_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 8192 logger.save_config_filepath=work/model-verification/nemotron-3-nano-30b-a3b/gb300-performance/ConfigContainer.yaml" + ], + "dimensions": {}, + "enabled_features": {}, + "entry_id": "nemotron-3-nano-30b-a3b-pretrain-performance-gb300", + "expected_result": "On exactly 8 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP8/ETP1, GBS/MBS 512/4, and sequence length 8192. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.18025 to 0.004964447; the final ten steps average 12455.140 ms, 937.770 TFLOP/s/GPU, and 42094.107 tokens/s/GPU. The resolved configuration persists.\n", + "hardware": "GB300", + "last_verified": "2026-08-18", + "metrics": { + "final_loss": 0.004964447, + "initial_loss": 12.18025, + "last_10_steps_model_tflops_per_gpu_avg": 937.77, + "last_10_steps_step_time_ms_avg": 12455.14, + "last_10_steps_tokens_per_second_per_gpu_avg": 42094.107 + }, + "precision": "fp8_mx", + "source_pointer": "items.pretrain_performance.GB300", + "status": "verified", + "variant": null, + "workflow": "pretrain_performance" } ], - "hf_id": "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16", - "hf_revision": "24e67ea000b7c2837fc8f9488aa2008524fac8ba", - "min_transformers_version": "5.8.0", - "slug": "nemotron-3-nano-omni-30b-a3b-reasoning", - "source_card": "examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml", - "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Verification uses the immutable public model and CORD v2 revisions. CPU and distributed GPU import and export, deterministic Megatron inference, 100-step one-node H100 full-model SFT, in-batch-packed 8K SFT, and LoRA PEFT runs completed. The H100 fine-tuning variants preserve real image-text samples and natural routing while using lower-precision optimizer state; 4K SFT and PEFT use HybridEP, while packed CP2 uses the standard all-to-all dispatcher. Strict CPU and GPU round trips preserve all 7,349 tensors bitwise and reload with the same native-loader notices as the pinned Hugging Face source. The one-step HF/Megatron comparison predicts the same token but remains below the 0.99 cosine gate. Unsupported and incomplete workflows remain explicitly identified rather than inferred from focused unit coverage.\n", - "title": "nemotron_3_nano_omni_30b_a3b_reasoning" + "hf_id": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", + "hf_revision": "2d59de1cbd51c0adf384eb906b766d1aee0e0517", + "min_transformers_version": "5.8.1", + "slug": "nemotron-3-nano-30b-a3b", + "source_card": "examples/model_verification_cards/nemotron-3-nano-30b-a3b/card.yaml", + "summary": "The pinned checkpoint has verified one-node GPU import, independent CPU and distributed GPU BF16 export, and checkpoint-backed deterministic inference. Both exports strictly reload with Transformers and exhaustively match the source after the explicitly requested BF16 casts. Performance scope: pretrain_performance.GB300 uses the tuned canonical 8-GPU MXFP8 recipe. CPU import, manual forward, and functional training workflows remain unverified in this card.\n", + "title": "nemotron_3_nano_30b_a3b" }, { "architecture": "NemotronHForCausalLM", - "base_container": "nvcr.io/nvidia/nemo:26.08", - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "base_container": "nvcr.io/nvidia/pytorch:26.04-py3", + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", "entries": [ { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "nodes": "1" + } + ], + "commands": [ + "./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --torch-dtype bfloat16" + ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-hf-to-megatron-cpu", - "expected_result": "This workflow remains unverified and requires a public run against the pinned Hugging Face revision with all applicable verification gates.\n", + "entry_id": "nemotron-3-nano-4b-hf-to-megatron-cpu", + "expected_result": "The command exits successfully and creates iter_0000000. After the paired CPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value, with maximum difference zero.\n", "hardware": null, - "last_verified": null, + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_cpu", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "hf_to_megatron_cpu" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", "command_topologies": [ { - "expert_parallel": "8", - "expert_tensor_parallel": "1", - "gpus_per_node": "8", + "gpus_per_node": "1", "nodes": "1", - "pipeline_parallel": "1", "tensor_parallel": "1", - "total_gpus": 8 + "total_gpus": 1 } ], "commands": [ - "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-timeout-minutes 120 --trust-remote-code --low-memory-save" + "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --torch-dtype bfloat16" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-hf-to-megatron-gpu", - "expected_result": "The pinned one-node 8-H100 import exits successfully at TP1/PP1/EP8/ETP1, persists iter_0000000, and reloads it for inference. A separate public GPU round-trip at the same topology exhaustively compares all 6,243 exported BF16 weights against the immutable source: 6,243 of 6,243 match with no skipped FP8 tensors or mismatches.\n", + "entry_id": "nemotron-3-nano-4b-hf-to-megatron-gpu", + "expected_result": "The command exits successfully and creates a reloadable iter_0000000. After the paired GPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value.\n", "hardware": null, - "last_verified": "2026-08-25", + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_gpu", @@ -3835,21 +3807,21 @@ "workflow": "hf_to_megatron_gpu" }, { - "bridge_commit": "60442bb9adb5435b47db22c6c20aacdf772fbddc", + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", "command_topologies": [ { "nodes": "1" } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-30b-a3b/cpu-hf-export --torch-dtype bfloat16 --trust-remote-code" + "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --hf-path work/model-verification/nemotron-3-nano-4b/cpu-hf-export --torch-dtype bfloat16" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-megatron-to-hf-cpu", - "expected_result": "The synchronous CPU export writes a complete indexed checkpoint. All 6,243 source keys and shapes match across 31,577,940,288 values. The 46 source FP32 tensors cast to BF16 are the 23 per-layer mixer A_log tensors and 23 per-layer mixer D tensors; each equals the export exactly after that declared cast, and every remaining tensor matches exactly at zero tolerance. Native loading succeeds as NemotronHForCausalLM with no missing, unexpected, mismatched, or errored weights, and the tokenizer reloads from the export.\n", + "entry_id": "nemotron-3-nano-4b-megatron-to-hf-cpu", + "expected_result": "Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the 171-byte generation configuration is preserved byte-for-byte, and Transformers reloads the output natively as NemotronHForCausalLM.\n", "hardware": null, - "last_verified": "2026-08-25", + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.megatron_to_hf_cpu", @@ -3858,27 +3830,24 @@ "workflow": "megatron_to_hf_cpu" }, { - "bridge_commit": "60442bb9adb5435b47db22c6c20aacdf772fbddc", + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", "command_topologies": [ { - "expert_parallel": "8", - "expert_tensor_parallel": "1", - "gpus_per_node": "8", + "gpus_per_node": "1", "nodes": "1", - "pipeline_parallel": "1", "tensor_parallel": "1", - "total_gpus": 8 + "total_gpus": 1 } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron-path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-30b-a3b/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1 --distributed-save --save-every-n-ranks 1 --trust-remote-code" + "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --hf-path work/model-verification/nemotron-3-nano-4b/hf-export --torch-dtype bfloat16" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-megatron-to-hf-gpu", - "expected_result": "The synchronous TP1/PP1/EP8/ETP1 export writes a complete indexed checkpoint. All 6,243 source keys and shapes match across 31,577,940,288 values. The 69 source FP32 tensors cast to BF16 are the same 46 mixer A_log and D tensors plus 23 per-layer mixer gate e_score_correction_bias tensors. The legacy distributed GPU export model wrapper downcast those gate biases to BF16, while the CPU export preserved them in FP32; all 69 equal the export exactly after this declared cast, and every remaining tensor matches exactly at zero tolerance. Native loading succeeds as NemotronHForCausalLM with no missing, unexpected, mismatched, or errored weights, and the tokenizer reloads from the export.\n", + "entry_id": "nemotron-3-nano-4b-megatron-to-hf-gpu", + "expected_result": "Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the generation configuration is exact, and Transformers reloads the output natively as NemotronHForCausalLM. A_log, D, dt_bias, and out_proj are exact for all 21 Mamba layers.\n", "hardware": null, - "last_verified": "2026-08-25", + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.megatron_to_hf_gpu", @@ -3887,44 +3856,44 @@ "workflow": "megatron_to_hf_gpu" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "tensor_parallel": "1" + } + ], + "commands": [ + "uv run python -m torch.distributed.run --standalone --nproc_per_node=1 examples/conversion/compare_hf_and_megatron/compare.py --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron_model_path work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --tp 1 --prompt \"The capital of France is the city of\"" + ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-manual-forward-pass", - "expected_result": "This workflow remains unverified and requires a public run against the pinned Hugging Face revision with all applicable verification gates.\n", + "entry_id": "nemotron-3-nano-4b-manual-forward-pass", + "expected_result": "The eight-token one-step comparison exits successfully using native Transformers Nemotron-H and a real Megatron inference context. Both paths have next-token predictions that match at token ID 6993 (\" Paris\"); cosine similarity is 0.999982 and the maximum and mean absolute logit differences are 0.093750 and 0.018522.\n", "hardware": null, - "last_verified": null, + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.manual_forward_pass", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "manual_forward_pass" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", + "bridge_commit": "78ecfad7644ee67efa4e8af4b0e7cf09f6bc9ecc", "command_topologies": [ { - "expert_parallel": "8", - "expert_tensor_parallel": "1", - "gpus_per_node": "8", - "nodes": "1", - "pipeline_parallel": "1", - "tensor_parallel": "1", - "total_gpus": 8 + "tensor_parallel": "1" } ], "commands": [ - "./scripts/inference/infer.sh --nodes 1 --gpus-per-node 8 --task legacy-full-prefix-generation --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --hf-revision 2d59de1cbd51c0adf384eb906b766d1aee0e0517 --megatron_model_path work/model-verification/nemotron-3-nano-30b-a3b/import-gpu/iter_0000000 --tp 1 --pp 1 --ep 8 --etp 1 --prompt \"Explain why the sky appears blue in one sentence.\" --max_new_tokens 32 --apply-chat-template --thinking-mode disabled --trust-remote-code --legacy-full-prefix" + "uv run python -m torch.distributed.run --standalone --nproc_per_node=1 examples/conversion/hf_to_megatron_generate_text.py --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron_model_path work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --tp 1 --prompt \"The capital of France is\" --max_new_tokens 2" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-inference", - "expected_result": "The public launcher reloaded the pinned BF16 checkpoint and produced exactly 32 generated tokens across generation steps 0 through 31, using the 32-token maximum without an earlier EOS. The literal completion is 'User asks: \"Explain why the sky appears blue in one sentence.\" So we need to give a single sentence explanation. Provide concise answer. Probably: \"'.\n", + "entry_id": "nemotron-3-nano-4b-inference", + "expected_result": "Two independent runs use greedy decoding, produce byte-identical token IDs, and return exactly 2 generated tokens with this literal completion, including its leading space: ' Paris.\"'.\n", "hardware": null, - "last_verified": "2026-08-25", + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.inference", @@ -3933,200 +3902,241 @@ "workflow": "inference" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "gpus_per_node": "8", + "nodes": "1", + "recipe": "nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config", + "sequence_length": "4096", + "total_gpus": 8 + } + ], + "commands": [ + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[[\"work/data/rp2/head_01\"],null]' dataset.path_to_cache=work/cache/nemotron-3-nano-4b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null" + ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-pretrain-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "entry_id": "nemotron-3-nano-4b-pretrain-h100", + "expected_result": "The uninterrupted bounded random-initialization RP2 reference finishes all 100 steps at recipe-owned GBS/MBS 1024/1 with the five recorded metrics, finite loss, and no skipped or NaN iterations. It reaches peak learning rate 3e-4 at step 40 and completes cosine decay to 3e-5 at step 100. Both iter_0000050 and iter_0000100 contain all eight distributed shards plus metadata, optimizer/RNG train state, run config, and tokenizer; iter_0000100 reloads at step 100 without an extra optimizer step.\n", "hardware": "H100", - "last_verified": null, + "last_verified": "2026-07-20", "metrics": { - "final_loss": null, - "initial_loss": null, - "last_10_steps_model_tflops_per_gpu_avg": null, - "last_10_steps_step_time_ms_avg": null, - "last_10_steps_tokens_per_second_per_gpu_avg": null + "final_loss": 5.511951, + "initial_loss": 12.4325, + "last_10_steps_model_tflops_per_gpu_avg": 402.51, + "last_10_steps_step_time_ms_avg": 26775.11, + "last_10_steps_tokens_per_second_per_gpu_avg": 19581.171 }, "precision": "bf16", "source_pointer": "items.pretrain.H100", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "pretrain" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "gpus_per_node": "8", + "nodes": "1", + "recipe": "nemotron_3_nano_4b_sft_8gpu_h100_bf16_config", + "sequence_length": "2048", + "total_gpus": 8 + } + ], + "commands": [ + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-sft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:1}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + ], "dimensions": {}, - "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-sft-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "enabled_features": { + "sequence_packing": "offline" + }, + "entry_id": "nemotron-3-nano-4b-sft-h100", + "expected_result": "Pad-1 offline packing is 99.27% efficient. The TP1/DP8 run uses four microbatches per optimizer step and completes all 100 steps with the four recorded metrics, finite loss, and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,187,630 supervised tokens. The complete eight-shard iter_0000100 full-model checkpoint is saved and reloads for export.\n", "hardware": "H100", - "last_verified": null, + "last_verified": "2026-07-20", "metrics": { - "final_loss": null, - "initial_loss": null, - "last_10_steps_model_tflops_per_gpu_avg": null, - "last_10_steps_step_time_ms_avg": null, - "last_10_steps_tokens_per_second_per_gpu_avg": null + "final_loss": 1.234906, + "initial_loss": 1.839282, + "last_10_steps_model_tflops_per_gpu_avg": 262.89, + "last_10_steps_step_time_ms_avg": 627.0, + "last_10_steps_tokens_per_second_per_gpu_avg": 13065.391 }, "precision": "bf16", "source_pointer": "items.sft.H100", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "sft" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "gpus_per_node": "1", + "nodes": "1", + "tensor_parallel": "1", + "total_gpus": 1 + }, + {} + ], + "commands": [ + "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/sft-checkpoints/iter_0000100 --hf-path work/model-verification/nemotron-3-nano-4b/sft-hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16", + "uv run python skills/create-model-verification-card/scripts/verify_hf_inference.py --hf-model work/model-verification/nemotron-3-nano-4b/sft-hf-export --prompt \"Name the capital of France and explain its role in one sentence.\" --max-new-tokens 45 --chat-template --disable-thinking" + ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-sft-export-inference-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "entry_id": "nemotron-3-nano-4b-sft-export-inference-h100", + "expected_result": "The final full-SFT checkpoint exports as one BF16 safetensors file with 263 weights and reloads natively as NemotronHForCausalLM. The helper runs greedy generation twice, obtains identical token IDs and exactly 45 new tokens, and produces this literal completion with twelve trailing newline characters: \"The capital of France is Paris, which serves as the country's political, cultural, and economic center.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\".\n", "hardware": "H100", - "last_verified": null, + "last_verified": "2026-07-20", "metrics": {}, "precision": "bf16", "source_pointer": "items.sft_export_inference.H100", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "sft_export_inference" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "context_parallel": "2", + "gpus_per_node": "8", + "nodes": "1", + "recipe": "nemotron_3_nano_4b_sft_8gpu_h100_bf16_32k_config", + "sequence_length": "32768", + "tensor_parallel": "2", + "total_gpus": 8 + } + ], + "commands": [ + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 32768 --tensor_model_parallel_size 2 --context_parallel_size 2 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-long-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.sequence_parallel=true model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.calculate_per_token_loss=true ddp.average_in_collective=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/long-sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + ], "dimensions": {}, - "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-sft-long-context-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "enabled_features": { + "context_parallel_size": 2, + "sequence_packing": "offline" + }, + "entry_id": "nemotron-3-nano-4b-sft-long-context-h100", + "expected_result": "Pad-4 offline packing is 99.35% efficient. The TP2/CP2/SP-on/DP2 run uses four microbatches per optimizer step and completes all 100 32K steps with the five recorded metrics, finite loss, and no skipped or NaN iterations. Across 26,214,400 token slots, assistant-only loss masks contain 16,734,347 supervised tokens. The complete eight-shard iter_0000100 checkpoint is saved and reloads at the same topology.\n", "hardware": "H100", - "last_verified": null, + "last_verified": "2026-07-20", "metrics": { - "final_loss": null, - "initial_loss": null, - "last_10_steps_model_tflops_per_gpu_avg": null, - "last_10_steps_step_time_ms_avg": null, - "last_10_steps_tokens_per_second_per_gpu_avg": null + "final_loss": 1.100197, + "initial_loss": 1.65866, + "last_10_steps_model_tflops_per_gpu_avg": 142.06, + "last_10_steps_step_time_ms_avg": 5575.59, + "last_10_steps_tokens_per_second_per_gpu_avg": 5877.046 }, "precision": "bf16", "source_pointer": "items.sft_long_context.H100", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "sft_long_context" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "command_topologies": [ + { + "gpus_per_node": "8", + "nodes": "1", + "recipe": "nemotron_3_nano_4b_peft_8gpu_h100_bf16_config", + "sequence_length": "2048", + "total_gpus": 8 + } + ], + "commands": [ + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_peft_8gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-peft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:4}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + ], "dimensions": {}, - "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-peft-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", + "enabled_features": { + "sequence_packing": "offline" + }, + "entry_id": "nemotron-3-nano-4b-peft-h100", + "expected_result": "Pad-4 offline packing is 99.36% efficient. The frozen-base LoRA run uses rank 8, alpha 16, zero dropout, and linear_qkv/linear_proj targets. Its TP1/DP8 layout uses four microbatches per optimizer step and completes all 100 steps with finite loss and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,178,503 supervised tokens. The complete eight-shard iter_0000100 adapter checkpoint reloads over the pinned base model at step 100.\n", "hardware": "H100", - "last_verified": null, + "last_verified": "2026-07-20", "metrics": { - "final_loss": null, - "initial_loss": null, - "last_10_steps_model_tflops_per_gpu_avg": null, - "last_10_steps_step_time_ms_avg": null, - "last_10_steps_tokens_per_second_per_gpu_avg": null + "final_loss": 1.256916, + "initial_loss": 1.769892, + "last_10_steps_model_tflops_per_gpu_avg": 431.9, + "last_10_steps_step_time_ms_avg": 381.69, + "last_10_steps_tokens_per_second_per_gpu_avg": 21462.443 }, "precision": "bf16", "source_pointer": "items.peft.H100", - "status": "unverified", + "status": "verified", "variant": null, "workflow": "peft" }, { - "bridge_commit": "d352aceda8ed4136f1db787bcf449c1b210a2438", - "command_topologies": [], - "commands": [], - "dimensions": {}, - "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-checkpoint-resume-h100", - "expected_result": "This workflow remains unverified and requires a bounded public run with the model's pinned revision and all applicable verification gates.\n", - "hardware": "H100", - "last_verified": null, - "metrics": { - "final_loss": null, - "initial_loss": null, - "last_10_steps_model_tflops_per_gpu_avg": null, - "last_10_steps_step_time_ms_avg": null, - "last_10_steps_tokens_per_second_per_gpu_avg": null - }, - "precision": "bf16", - "source_pointer": "items.checkpoint_resume.H100", - "status": "unverified", - "variant": null, - "workflow": "checkpoint_resume" - }, - { - "bridge_commit": "0480586879f2513958fd8634fc529693ae13e536", + "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", "command_topologies": [ { - "gpus_per_node": "4", - "nodes": "2", - "recipe": "nemotron_3_nano_pretrain_8gpu_gb300_fp8mx_config", - "sequence_length": "8192", + "gpus_per_node": "8", + "nodes": "1", + "recipe": "nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config", + "sequence_length": "4096", "total_gpus": 8 } ], "commands": [ - "./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 4 --recipe nemotron_3_nano_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 8192 logger.save_config_filepath=work/model-verification/nemotron-3-nano-30b-a3b/gb300-performance/ConfigContainer.yaml" + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[[\"work/data/rp2/head_01\"],null]' dataset.path_to_cache=work/cache/nemotron-3-nano-4b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true --load_dir work/model-verification/nemotron-3-nano-4b/pretrain-reference-checkpoints checkpoint.ckpt_step=50 --save_dir work/model-verification/nemotron-3-nano-4b/pretrain-resumed-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-30b-a3b-pretrain-performance-gb300", - "expected_result": "On exactly 8 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP8/ETP1, GBS/MBS 512/4, and sequence length 8192. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.18025 to 0.004964447; the final ten steps average 12455.140 ms, 937.770 TFLOP/s/GPU, and 42094.107 tokens/s/GPU. The resolved configuration persists.\n", - "hardware": "GB300", - "last_verified": "2026-08-18", + "entry_id": "nemotron-3-nano-4b-checkpoint-resume-h100", + "expected_result": "The continuation restores optimizer, scheduler, data-order, and RNG state directly from iter_0000050 and executes exactly steps 51 through 100 into a distinct output root with the five recorded metrics, finite loss, and no skipped or NaN iterations. Step 51 matches the uninterrupted reference exactly at 6.173212. At step 100, resumed loss 5.512038 differs from reference loss 5.511951 by 0.000087 (0.00158%), inside the declared 1% tolerance. Its complete eight-shard iter_0000100 checkpoint contains metadata, optimizer/RNG train state, run config, and tokenizer, and reloads at step 100 without an extra optimizer step.\n", + "hardware": "H100", + "last_verified": "2026-07-20", "metrics": { - "final_loss": 0.004964447, - "initial_loss": 12.18025, - "last_10_steps_model_tflops_per_gpu_avg": 937.77, - "last_10_steps_step_time_ms_avg": 12455.14, - "last_10_steps_tokens_per_second_per_gpu_avg": 42094.107 + "final_loss": 5.512038, + "initial_loss": 6.173212, + "last_10_steps_model_tflops_per_gpu_avg": 400.11, + "last_10_steps_step_time_ms_avg": 26943.07, + "last_10_steps_tokens_per_second_per_gpu_avg": 19459.104 }, - "precision": "fp8_mx", - "source_pointer": "items.pretrain_performance.GB300", + "precision": "bf16", + "source_pointer": "items.checkpoint_resume.H100", "status": "verified", "variant": null, - "workflow": "pretrain_performance" + "workflow": "checkpoint_resume" } ], - "hf_id": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", - "hf_revision": "2d59de1cbd51c0adf384eb906b766d1aee0e0517", + "hf_id": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16", + "hf_revision": "dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f", "min_transformers_version": "5.8.1", - "slug": "nemotron-3-nano-30b-a3b", - "source_card": "examples/model_verification_cards/nemotron-3-nano-30b-a3b/card.yaml", - "summary": "The pinned checkpoint has verified one-node GPU import, independent CPU and distributed GPU BF16 export, and checkpoint-backed deterministic inference. Both exports strictly reload with Transformers and exhaustively match the source after the explicitly requested BF16 casts. Performance scope: pretrain_performance.GB300 uses the tuned canonical 8-GPU MXFP8 recipe. CPU import, manual forward, and functional training workflows remain unverified in this card.\n", - "title": "nemotron_3_nano_30b_a3b" + "slug": "nemotron-3-nano-4b", + "source_card": "examples/model_verification_cards/nemotron-3-nano-4b/card.yaml", + "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Nemotron 3 Nano 4B support verification covers conversion, inference, and training.\n", + "title": "nemotron_3_nano_4b" }, { - "architecture": "NemotronHForCausalLM", - "base_container": "nvcr.io/nvidia/pytorch:26.04-py3", - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "architecture": "NemotronH_Nano_Omni_Reasoning_V3", + "base_container": "nvcr.io/nvidia/nemo:26.06", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "entries": [ { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "command_topologies": [ { - "nodes": "1" + "expert_parallel": "1", + "expert_tensor_parallel": "1", + "nodes": "1", + "pipeline_parallel": "1", + "tensor_parallel": "1" } ], "commands": [ - "./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --torch-dtype bfloat16" + "./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-megatron-clean --torch-dtype bfloat16 --tp 1 --pp 1 --ep 1 --etp 1 --trust-remote-code" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-hf-to-megatron-cpu", - "expected_result": "The command exits successfully and creates iter_0000000. After the paired CPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value, with maximum difference zero.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-hf-to-megatron-cpu", + "expected_result": "The offline, immutable-revision CPU import exits successfully after mapping 7,333 model parameters and creates a reloadable iter_0000000 torch_dist checkpoint with 33,015,546,816 parameters on the single model parallel rank.\n", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": "2026-07-30", "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_cpu", @@ -4135,24 +4145,27 @@ "workflow": "hf_to_megatron_cpu" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "command_topologies": [ { - "gpus_per_node": "1", + "expert_parallel": "4", + "expert_tensor_parallel": "1", + "gpus_per_node": "8", "nodes": "1", - "tensor_parallel": "1", - "total_gpus": 1 + "pipeline_parallel": "1", + "tensor_parallel": "2", + "total_gpus": 8 } ], "commands": [ - "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --torch-dtype bfloat16" + "./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 4 --etp 1 --trust-remote-code --low-memory-save" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-hf-to-megatron-gpu", - "expected_result": "The command exits successfully and creates a reloadable iter_0000000. After the paired GPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-hf-to-megatron-gpu", + "expected_result": "The command exits successfully at TP2/PP1/EP4/ETP1 and creates a reloadable iter_0000000 checkpoint. The paired strict GPU export contains the source checkpoint's exact 7,349-key set, shapes, dtypes, and values.\n", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": "2026-07-30", "metrics": {}, "precision": "bf16", "source_pointer": "items.hf_to_megatron_gpu", @@ -4161,21 +4174,21 @@ "workflow": "hf_to_megatron_gpu" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "7783fc356b7d4768525eabf9825f9108c6544f4c", "command_topologies": [ { "nodes": "1" } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --hf-path work/model-verification/nemotron-3-nano-4b/cpu-hf-export --torch-dtype bfloat16" + "./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-megatron-clean/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/cpu-hf-export-clean --torch-dtype bfloat16 --trust-remote-code" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-megatron-to-hf-cpu", - "expected_result": "Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the 171-byte generation configuration is preserved byte-for-byte, and Transformers reloads the output natively as NemotronHForCausalLM.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-megatron-to-hf-cpu", + "expected_result": "CPU export completes in 14 indexed shards totaling 66,031,270,520 bytes. Its exact comparison contains all 7,349 source tensors and 33,015,632,238 values, with identical keys, shapes, dtypes, and values and no dtype conversions. Transformers reloads the local custom-code export natively with the source-equivalent warning set: one generated vision summary buffer is missing and four source-only audio/vision buffers are unexpected.\n", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": "2026-08-25", "metrics": {}, "precision": "bf16", "source_pointer": "items.megatron_to_hf_cpu", @@ -4184,24 +4197,27 @@ "workflow": "megatron_to_hf_cpu" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "7783fc356b7d4768525eabf9825f9108c6544f4c", "command_topologies": [ { - "gpus_per_node": "1", + "expert_parallel": "4", + "expert_tensor_parallel": "1", + "gpus_per_node": "8", "nodes": "1", - "tensor_parallel": "1", - "total_gpus": 1 + "pipeline_parallel": "1", + "tensor_parallel": "2", + "total_gpus": 8 } ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --hf-path work/model-verification/nemotron-3-nano-4b/hf-export --torch-dtype bfloat16" + "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-hf-export-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 4 --etp 1 --trust-remote-code --distributed-save --save-every-n-ranks 1" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-megatron-to-hf-gpu", - "expected_result": "Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the generation configuration is exact, and Transformers reloads the output natively as NemotronHForCausalLM. A_log, D, dt_bias, and out_proj are exact for all 21 Mamba layers.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-megatron-to-hf-gpu", + "expected_result": "Distributed export completes in 17 indexed shards totaling 66,031,270,520 bytes. All 7,349 tensors and 33,015,632,238 values match the immutable HF source exactly in keys, shapes, dtypes, and values with no dtype conversions. Transformers reloads the local custom-code export natively with the same one missing generated buffer and four unexpected source-only buffers reported when reloading the pinned source itself.\n", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": "2026-08-25", "metrics": {}, "precision": "bf16", "source_pointer": "items.megatron_to_hf_gpu", @@ -4210,44 +4226,50 @@ "workflow": "megatron_to_hf_gpu" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "command_topologies": [ { + "expert_parallel": "2", + "expert_tensor_parallel": "1", + "pipeline_parallel": "4", "tensor_parallel": "1" } ], "commands": [ - "uv run python -m torch.distributed.run --standalone --nproc_per_node=1 examples/conversion/compare_hf_and_megatron/compare.py --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron_model_path work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --tp 1 --prompt \"The capital of France is the city of\"" + "uv run python -m torch.distributed.run --standalone --nproc_per_node=8 examples/conversion/compare_hf_and_megatron/compare.py --hf_model_path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron_model_path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --tp 1 --pp 4 --ep 2 --etp 1 --prompt \"The capital of France is \" --trust-remote-code" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-manual-forward-pass", - "expected_result": "The eight-token one-step comparison exits successfully using native Transformers Nemotron-H and a real Megatron inference context. Both paths have next-token predictions that match at token ID 6993 (\" Paris\"); cosine similarity is 0.999982 and the maximum and mean absolute logit differences are 0.093750 and 0.018522.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-manual-forward-pass", + "expected_result": "The pinned one-step run exits successfully and both implementations predict token ID 6993 (\" Paris\"), but cosine similarity is 0.969760 and therefore does not satisfy the required 0.99 verification gate. The maximum and mean absolute logit differences are 3.527344 and 0.527820, respectively.\n", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": null, "metrics": {}, "precision": "bf16", "source_pointer": "items.manual_forward_pass", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "manual_forward_pass" }, { - "bridge_commit": "78ecfad7644ee67efa4e8af4b0e7cf09f6bc9ecc", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "command_topologies": [ { + "expert_parallel": "2", + "expert_tensor_parallel": "1", + "pipeline_parallel": "4", "tensor_parallel": "1" } ], "commands": [ - "uv run python -m torch.distributed.run --standalone --nproc_per_node=1 examples/conversion/hf_to_megatron_generate_text.py --hf_model_path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron_model_path work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --tp 1 --prompt \"The capital of France is\" --max_new_tokens 2" + "uv run python -m torch.distributed.run --standalone --nproc_per_node=8 examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py --hf_model_path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron_model_path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --prompt \"The capital of France is\" --max_new_tokens 4 --tp 1 --pp 4 --ep 2 --etp 1" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-inference", - "expected_result": "Two independent runs use greedy decoding, produce byte-identical token IDs, and return exactly 2 generated tokens with this literal completion, including its leading space: ' Paris.\"'.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-inference", + "expected_result": "Deterministic greedy decoding returns the exact 4-token result and\nterminates with the model end token. Literal completion: \"\nParis.\"", "hardware": null, - "last_verified": "2026-07-20", + "last_verified": "2026-07-30", "metrics": {}, "precision": "bf16", "source_pointer": "items.inference", @@ -4256,66 +4278,55 @@ "workflow": "inference" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", - "command_topologies": [ - { - "gpus_per_node": "8", - "nodes": "1", - "recipe": "nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config", - "sequence_length": "4096", - "total_gpus": 8 - } - ], - "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[[\"work/data/rp2/head_01\"],null]' dataset.path_to_cache=work/cache/nemotron-3-nano-4b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null" - ], + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "command_topologies": [], + "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-pretrain-h100", - "expected_result": "The uninterrupted bounded random-initialization RP2 reference finishes all 100 steps at recipe-owned GBS/MBS 1024/1 with the five recorded metrics, finite loss, and no skipped or NaN iterations. It reaches peak learning rate 3e-4 at step 40 and completes cosine decay to 3e-5 at step 100. Both iter_0000050 and iter_0000100 contain all eight distributed shards plus metadata, optimizer/RNG train state, run config, and tokenizer; iter_0000100 reloads at step 100 without an extra optimizer step.\n", - "hardware": "H100", - "last_verified": "2026-07-20", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-pretrain-all", + "expected_result": "Megatron Bridge does not publish a pretraining recipe for this multimodal conditional-generation model; the supported package provides supervised CORD v2 SFT and LoRA PEFT workflows.\n", + "hardware": "all", + "last_verified": null, "metrics": { - "final_loss": 5.511951, - "initial_loss": 12.4325, - "last_10_steps_model_tflops_per_gpu_avg": 402.51, - "last_10_steps_step_time_ms_avg": 26775.11, - "last_10_steps_tokens_per_second_per_gpu_avg": 19581.171 + "final_loss": null, + "initial_loss": null, + "last_10_steps_model_tflops_per_gpu_avg": null, + "last_10_steps_step_time_ms_avg": null, + "last_10_steps_tokens_per_second_per_gpu_avg": null }, - "precision": "bf16", - "source_pointer": "items.pretrain.H100", - "status": "verified", + "precision": null, + "source_pointer": "items.pretrain.all", + "status": "unsupported", "variant": null, "workflow": "pretrain" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "9b69b47a321de14b0453c230dd130e18ae0f00f7", "command_topologies": [ { "gpus_per_node": "8", "nodes": "1", - "recipe": "nemotron_3_nano_4b_sft_8gpu_h100_bf16_config", - "sequence_length": "2048", + "recipe": "nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config", "total_gpus": 8 } ], "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-sft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:1}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config --mode sft --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-4k-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" ], "dimensions": {}, "enabled_features": { - "sequence_packing": "offline" + "moe_dispatcher": "hybridep" }, - "entry_id": "nemotron-3-nano-4b-sft-h100", - "expected_result": "Pad-1 offline packing is 99.27% efficient. The TP1/DP8 run uses four microbatches per optimizer step and completes all 100 steps with the four recorded metrics, finite loss, and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,187,630 supervised tokens. The complete eight-shard iter_0000100 full-model checkpoint is saved and reloads for export.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-h100", + "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 full-SFT optimizer steps on eight H100 GPUs at TP2/PP1/CP1/EP8/ETP1 and GBS/MBS 64/4 with natural-routing HybridEP. Execution uses selective MoE/layernorm recompute, fused attention, TE cross entropy, fused weighted squared-ReLU, and router fusion without changing the image-text objective or routing policy. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.281308 to 0.03330838, the final ten steps average 3,603.67 ms and 197.10 TFLOP/s/GPU, sampled peak memory is 76,349 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 at 197.07 final-ten TFLOP/s/GPU with finite loss and zero skipped or NaN iterations.\n", "hardware": "H100", - "last_verified": "2026-07-20", + "last_verified": "2026-08-12", "metrics": { - "final_loss": 1.234906, - "initial_loss": 1.839282, - "last_10_steps_model_tflops_per_gpu_avg": 262.89, - "last_10_steps_step_time_ms_avg": 627.0, - "last_10_steps_tokens_per_second_per_gpu_avg": 13065.391 + "final_loss": 0.03330838, + "initial_loss": 1.281308, + "last_10_steps_model_tflops_per_gpu_avg": 197.1, + "last_10_steps_step_time_ms_avg": 3603.67, + "last_10_steps_tokens_per_second_per_gpu_avg": 9092.952 }, "precision": "bf16", "source_pointer": "items.sft.H100", @@ -4324,64 +4335,64 @@ "workflow": "sft" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", "command_topologies": [ { - "gpus_per_node": "1", + "expert_parallel": "8", + "expert_tensor_parallel": "1", + "gpus_per_node": "8", "nodes": "1", - "tensor_parallel": "1", - "total_gpus": 1 + "pipeline_parallel": "1", + "tensor_parallel": "2", + "total_gpus": 8 }, {} ], "commands": [ - "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/sft-checkpoints/iter_0000100 --hf-path work/model-verification/nemotron-3-nano-4b/sft-hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16", - "uv run python skills/create-model-verification-card/scripts/verify_hf_inference.py --hf-model work/model-verification/nemotron-3-nano-4b/sft-hf-export --prompt \"Name the capital of France and explain its role in one sentence.\" --max-new-tokens 45 --chat-template --disable-thinking" + "./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-4k-checkpoints/iter_0000100 --hf-path work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-hf-export-clean --torch-dtype bfloat16 --tp 2 --pp 1 --ep 8 --etp 1 --trust-remote-code --distributed-save --not-strict", + "uv run python skills/create-model-verification-card/scripts/verify_hf_inference.py --hf-model work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-hf-export-clean --prompt \"The capital of France is\" --max-new-tokens 32 --chat-template --disable-thinking --trust-remote-code --device cuda --dtype bfloat16" ], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-sft-export-inference-h100", - "expected_result": "The final full-SFT checkpoint exports as one BF16 safetensors file with 263 weights and reloads natively as NemotronHForCausalLM. The helper runs greedy generation twice, obtains identical token IDs and exactly 45 new tokens, and produces this literal completion with twelve trailing newline characters: \"The capital of France is Paris, which serves as the country's political, cultural, and economic center.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\".\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-export-inference-h100", + "expected_result": "Non-strict export succeeds and writes 6,637 indexed tensors in 17 shards. The CORD v2 recipe intentionally disables sound, so 712 source audio tensors are absent. Native Transformers generation remains unverified because local dynamic-code loading omits configuration_nemotron_h before model weights can reload.\n", "hardware": "H100", - "last_verified": "2026-07-20", + "last_verified": null, "metrics": {}, "precision": "bf16", "source_pointer": "items.sft_export_inference.H100", - "status": "verified", + "status": "unverified", "variant": null, "workflow": "sft_export_inference" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "eb7cf72392ac07e7a095b87d9b1d6bf2b38c5b6e", "command_topologies": [ { - "context_parallel": "2", "gpus_per_node": "8", "nodes": "1", - "recipe": "nemotron_3_nano_4b_sft_8gpu_h100_bf16_32k_config", - "sequence_length": "32768", - "tensor_parallel": "2", + "recipe": "nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config", "total_gpus": 8 } ], "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 32768 --tensor_model_parallel_size 2 --context_parallel_size 2 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-long-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.sequence_parallel=true model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.calculate_per_token_loss=true ddp.average_in_collective=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/long-sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config --mode sft --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/sft-8k-packed-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" ], "dimensions": {}, "enabled_features": { "context_parallel_size": 2, - "sequence_packing": "offline" + "sequence_packing": "in_batch" }, - "entry_id": "nemotron-3-nano-4b-sft-long-context-h100", - "expected_result": "Pad-4 offline packing is 99.35% efficient. The TP2/CP2/SP-on/DP2 run uses four microbatches per optimizer step and completes all 100 32K steps with the five recorded metrics, finite loss, and no skipped or NaN iterations. Across 26,214,400 token slots, assistant-only loss masks contain 16,734,347 supervised tokens. The complete eight-shard iter_0000100 checkpoint is saved and reloads at the same topology.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-sft-long-context-h100", + "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 full-SFT optimizer steps on eight H100 GPUs at TP4/PP1/CP2/EP8/ETP1 and GBS/MBS 64/2 with 8K in-batch packing and the standard all-to-all dispatcher. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.251364 to 0.01020405, the final ten steps average 15,577.9 ms and 93.73 TFLOP/s/GPU, sampled peak memory is 75,197 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 with finite loss and zero skipped or NaN iterations.\n", "hardware": "H100", - "last_verified": "2026-07-20", + "last_verified": "2026-08-11", "metrics": { - "final_loss": 1.100197, - "initial_loss": 1.65866, - "last_10_steps_model_tflops_per_gpu_avg": 142.06, - "last_10_steps_step_time_ms_avg": 5575.59, - "last_10_steps_tokens_per_second_per_gpu_avg": 5877.046 + "final_loss": 0.01020405, + "initial_loss": 1.251364, + "last_10_steps_model_tflops_per_gpu_avg": 93.73, + "last_10_steps_step_time_ms_avg": 15577.9, + "last_10_steps_tokens_per_second_per_gpu_avg": 4206.986 }, "precision": "bf16", "source_pointer": "items.sft_long_context.H100", @@ -4390,33 +4401,32 @@ "workflow": "sft_long_context" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", + "bridge_commit": "eb7cf72392ac07e7a095b87d9b1d6bf2b38c5b6e", "command_topologies": [ { "gpus_per_node": "8", "nodes": "1", - "recipe": "nemotron_3_nano_4b_peft_8gpu_h100_bf16_config", - "sequence_length": "2048", + "recipe": "nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config", "total_gpus": 8 } ], "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_peft_8gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split=\"train[:10000]\"' 'dataset.hf_dataset.load_kwargs={revision:\"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be\"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-peft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:4}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true" + "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config --mode lora --step_func nemotron_omni_step --pretrained_checkpoint work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/gpu-megatron-clean/iter_0000000 --max_steps 100 --save_dir work/model-verification/nemotron-3-nano-omni-30b-a3b-reasoning/peft-checkpoints --save_interval 50 'dataset.source.load_kwargs={revision:\"7f0115a4b758a71d6473b8d085751692da2fef98\"}' dataset.do_validation=false dataset.do_test=false validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null logger.log_interval=1 logger.log_throughput=true rng.seed=5678" ], "dimensions": {}, "enabled_features": { - "sequence_packing": "offline" + "moe_dispatcher": "hybridep" }, - "entry_id": "nemotron-3-nano-4b-peft-h100", - "expected_result": "Pad-4 offline packing is 99.36% efficient. The frozen-base LoRA run uses rank 8, alpha 16, zero dropout, and linear_qkv/linear_proj targets. Its TP1/DP8 layout uses four microbatches per optimizer step and completes all 100 steps with finite loss and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,178,503 supervised tokens. The complete eight-shard iter_0000100 adapter checkpoint reloads over the pinned base model at step 100.\n", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-peft-h100", + "expected_result": "The immutable-revision, real image-text CORD v2 run completes exactly 100 LoRA optimizer steps on eight H100 GPUs at TP2/PP1/CP1/EP8/ETP1 and GBS/MBS 64/1 with natural-routing HybridEP. Model parameters, main gradients, and Adam moments use BF16; optimizer main parameters use scaled FP16 without parameter remainders. LM loss is finite from 1.282941 to 0.02634745, the final ten steps average 11,621.42 ms and 61.25 TFLOP/s/GPU, sampled peak memory is 44,873 MiB, and no iteration is skipped or NaN. Complete optimizer and RNG checkpoints are saved at steps 50 and 100. Direct step-50 resume loads the complete state and reaches step 100 with finite loss and zero skipped or NaN iterations.\n", "hardware": "H100", - "last_verified": "2026-07-20", + "last_verified": "2026-08-11", "metrics": { - "final_loss": 1.256916, - "initial_loss": 1.769892, - "last_10_steps_model_tflops_per_gpu_avg": 431.9, - "last_10_steps_step_time_ms_avg": 381.69, - "last_10_steps_tokens_per_second_per_gpu_avg": 21462.443 + "final_loss": 0.02634745, + "initial_loss": 1.282941, + "last_10_steps_model_tflops_per_gpu_avg": 61.25, + "last_10_steps_step_time_ms_avg": 11621.42, + "last_10_steps_tokens_per_second_per_gpu_avg": 2819.621 }, "precision": "bf16", "source_pointer": "items.peft.H100", @@ -4425,46 +4435,36 @@ "workflow": "peft" }, { - "bridge_commit": "0e6c1837edeb442b07c9175b855b3d288c5cad33", - "command_topologies": [ - { - "gpus_per_node": "8", - "nodes": "1", - "recipe": "nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config", - "sequence_length": "4096", - "total_gpus": 8 - } - ], - "commands": [ - "./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[[\"work/data/rp2/head_01\"],null]' dataset.path_to_cache=work/cache/nemotron-3-nano-4b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true --load_dir work/model-verification/nemotron-3-nano-4b/pretrain-reference-checkpoints checkpoint.ckpt_step=50 --save_dir work/model-verification/nemotron-3-nano-4b/pretrain-resumed-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null" - ], + "bridge_commit": "fbbafc7ddfa818ef91d5eb64c8a00dcaa7bee78d", + "command_topologies": [], + "commands": [], "dimensions": {}, "enabled_features": {}, - "entry_id": "nemotron-3-nano-4b-checkpoint-resume-h100", - "expected_result": "The continuation restores optimizer, scheduler, data-order, and RNG state directly from iter_0000050 and executes exactly steps 51 through 100 into a distinct output root with the five recorded metrics, finite loss, and no skipped or NaN iterations. Step 51 matches the uninterrupted reference exactly at 6.173212. At step 100, resumed loss 5.512038 differs from reference loss 5.511951 by 0.000087 (0.00158%), inside the declared 1% tolerance. Its complete eight-shard iter_0000100 checkpoint contains metadata, optimizer/RNG train state, run config, and tokenizer, and reloads at step 100 without an extra optimizer step.\n", - "hardware": "H100", - "last_verified": "2026-07-20", + "entry_id": "nemotron-3-nano-omni-30b-a3b-reasoning-checkpoint-resume-all", + "expected_result": "There is no supported pretraining reference workflow for this model, so no model-wide optimizer and RNG checkpoint-resume contract is available for sentinel comparison.\n", + "hardware": "all", + "last_verified": null, "metrics": { - "final_loss": 5.512038, - "initial_loss": 6.173212, - "last_10_steps_model_tflops_per_gpu_avg": 400.11, - "last_10_steps_step_time_ms_avg": 26943.07, - "last_10_steps_tokens_per_second_per_gpu_avg": 19459.104 + "final_loss": null, + "initial_loss": null, + "last_10_steps_model_tflops_per_gpu_avg": null, + "last_10_steps_step_time_ms_avg": null, + "last_10_steps_tokens_per_second_per_gpu_avg": null }, - "precision": "bf16", - "source_pointer": "items.checkpoint_resume.H100", - "status": "verified", + "precision": null, + "source_pointer": "items.checkpoint_resume.all", + "status": "unsupported", "variant": null, "workflow": "checkpoint_resume" } ], - "hf_id": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16", - "hf_revision": "dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f", - "min_transformers_version": "5.8.1", - "slug": "nemotron-3-nano-4b", - "source_card": "examples/model_verification_cards/nemotron-3-nano-4b/card.yaml", - "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Nemotron 3 Nano 4B support verification covers conversion, inference, and training.\n", - "title": "nemotron_3_nano_4b" + "hf_id": "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16", + "hf_revision": "24e67ea000b7c2837fc8f9488aa2008524fac8ba", + "min_transformers_version": "5.8.0", + "slug": "nemotron-3-nano-omni-30b-a3b-reasoning", + "source_card": "examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml", + "summary": "Performance disclaimer: this model has not been performance-tuned; reported timing and throughput metrics are sanity checks, not optimized performance results. Verification uses the immutable public model and CORD v2 revisions. CPU and distributed GPU import and export, deterministic Megatron inference, 100-step one-node H100 full-model SFT, in-batch-packed 8K SFT, and LoRA PEFT runs completed. The H100 fine-tuning variants preserve real image-text samples and natural routing while using lower-precision optimizer state; 4K SFT and PEFT use HybridEP, while packed CP2 uses the standard all-to-all dispatcher. Strict CPU and GPU round trips preserve all 7,349 tensors bitwise and reload with the same native-loader notices as the pinned Hugging Face source. The one-step HF/Megatron comparison predicts the same token but remains below the 0.99 cosine gate. Unsupported and incomplete workflows remain explicitly identified rather than inferred from focused unit coverage.\n", + "title": "nemotron_3_nano_omni_30b_a3b_reasoning" }, { "architecture": "NemotronHForCausalLM", diff --git a/docs/fern/versions/0.4.2/pages/index.mdx b/docs/fern/versions/0.4.2/pages/index.mdx index fd84f1a195..ffa1b15f2c 100644 --- a/docs/fern/versions/0.4.2/pages/index.mdx +++ b/docs/fern/versions/0.4.2/pages/index.mdx @@ -20,13 +20,13 @@ - [03/31/2026] **Agent Skills for Megatron Bridge!** We've added a [`skills/`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/skills) directory with structured guides that AI coding agents (Cursor, Claude Code, Codex, etc.) can use to help you add model support, set up dev environments, tune performance, and more. Try them out, and PRs to improve or add new skills are very welcome! -- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). +- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). - [03/12/2026] **Deprecating Python 3.10 support:** We're officially dropping Python 3.10 support with the upcoming 0.4.0 release. Downstream applications must raise their lower boundary to 3.12 to stay compatible with Megatron-Bridge. - [12/16/2025] [Mind Lab](https://macaron.im/mindlab) successfully used Megatron-bridge and [VeRL](https://github.com/volcengine/verl) to trained GRPO Lora for Trillion-parameter model on 64 H800 - See their [techblog](https://macaron.im/mindlab/research/building-trillion-parameter-reasoning-rl-with-10-gpus). -- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3/nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) +- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3_nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) ## Overview diff --git a/docs/fern/versions/0.4.2/pages/models/llm/nemotron3-super.mdx b/docs/fern/versions/0.4.2/pages/models/llm/nemotron3-super.mdx index 61da73db8a..88c57937c5 100644 --- a/docs/fern/versions/0.4.2/pages/models/llm/nemotron3-super.mdx +++ b/docs/fern/versions/0.4.2/pages/models/llm/nemotron3-super.mdx @@ -123,7 +123,7 @@ torchrun --nproc-per-node=8 examples/conversion/compare_hf_and_megatron/compare. BLEND_PATH=/path/to/dataset/blend.json CHECKPOINT_DIR=/path/to/checkpoints -torchrun --nproc-per-node=8 examples/models/nemotron_3/pretrain_nemotron_3_super.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/super/pretrain_nemotron_3_super.py \ --per-split-data-args-path=${BLEND_PATH} \ logger.wandb_project=your_project \ logger.wandb_entity=nvidia \ @@ -156,7 +156,7 @@ For quick testing without a dataset: ```bash CHECKPOINT_DIR=/path/to/checkpoints -torchrun --nproc-per-node=8 examples/models/nemotron_3/pretrain_nemotron_3_super.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/super/pretrain_nemotron_3_super.py \ logger.wandb_project=your_project \ logger.wandb_entity=nvidia \ checkpoint.load=${CHECKPOINT_DIR} \ @@ -181,7 +181,7 @@ Notes: MEGATRON_PATH=/path/to/pretrained/megatron/ckpt CHECKPOINT_DIR=/path/to/finetuned/checkpoints -torchrun --nproc-per-node=8 examples/models/nemotron_3/finetune_nemotron_3_super.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/super/finetune_nemotron_3_super.py \ logger.wandb_project=your_project \ logger.wandb_entity=nvidia \ logger.log_interval=5 \ @@ -209,7 +209,7 @@ To enable LoRA fine-tuning, pass `--peft lora` to the script: MEGATRON_PATH=/path/to/pretrained/megatron/ckpt CHECKPOINT_DIR=/path/to/lora/checkpoints -torchrun --nproc-per-node=8 examples/models/nemotron_3/finetune_nemotron_3_super.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/super/finetune_nemotron_3_super.py \ --peft lora \ logger.wandb_project=your_project \ logger.wandb_entity=nvidia \ @@ -304,7 +304,7 @@ After quantization, further improve model quality with QAT by continuing trainin MEGATRON_PATH=/path/to/quantized/megatron/ckpt CHECKPOINT_DIR=/path/to/qat/checkpoints -torchrun --nproc-per-node=8 examples/models/nemotron_3/qat_nemotron_3_super.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/super/qat_nemotron_3_super.py \ --megatron-load-path=${MEGATRON_PATH} \ --seq-length=8192 \ --packed-sequence \ diff --git a/docs/fern/versions/0.4.2/pages/models/llm/nemotron3.mdx b/docs/fern/versions/0.4.2/pages/models/llm/nemotron3.mdx index b97ba2455a..025381ddf2 100644 --- a/docs/fern/versions/0.4.2/pages/models/llm/nemotron3.mdx +++ b/docs/fern/versions/0.4.2/pages/models/llm/nemotron3.mdx @@ -41,7 +41,7 @@ python examples/conversion/convert_checkpoints.py export \ BLEND_PATH=/path/to/dataset/blend TOKENIZER_MODEL=/path/to/tiktok/tokenizer/model -torchrun --nproc-per-node=8 examples/models/nemotron_3/pretrain_nemotron_3_nano.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/nano/pretrain_nemotron_3_nano.py \ --per-split-data-args-path=${BLEND_PATH} \ --tokenizer-model=${TOKENIZER_MODEL} \ train.global_batch_size=3072 \ @@ -58,7 +58,7 @@ Notes: ### Full Parameter Fine-Tuning ```bash -torchrun --nproc-per-node=8 examples/models/nemotron_3/finetune_nemotron_3_nano.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/nano/finetune_nemotron_3_nano.py \ train.global_batch_size=128 \ train.train_iters=100 \ scheduler.lr_warmup_iters=10 \ @@ -73,7 +73,7 @@ Notes: ### LoRA Fine-Tuning To enable LoRA fine-tuning, pass `--peft lora` to script ```bash -torchrun --nproc-per-node=8 examples/models/nemotron_3/finetune_nemotron_3_nano.py \ +torchrun --nproc-per-node=8 examples/models/nemotron_3/nano/finetune_nemotron_3_nano.py \ --peft lora \ train.global_batch_size=128 \ train.train_iters=100 \ diff --git a/docs/fern/versions/_nav_order.yml b/docs/fern/versions/_nav_order.yml index 7e45e569f2..a001d1c2e1 100644 --- a/docs/fern/versions/_nav_order.yml +++ b/docs/fern/versions/_nav_order.yml @@ -67,7 +67,7 @@ "./versions/nightly/pages/models/gemma/gemma3-vl.mdx": 4 "./versions/nightly/pages/models/glm/glm5.mdx": 4 "./versions/nightly/pages/models/glm/glm5-2.mdx": 5 -"./versions/nightly/pages/models/nemotron/nemotron3-super.mdx": 5 +"./versions/nightly/pages/models/nemotron/nemotron3-super.mdx": 6 "./versions/nightly/pages/models/qwen/qwen2.5-vl.mdx": 6 "./versions/nightly/pages/nemo2-migration-guide.mdx": 4 "./versions/nightly/pages/releases/changelog.mdx": 4 @@ -75,13 +75,13 @@ "./versions/nightly/pages/megatron-lm-to-megatron-bridge.mdx": 5 "./versions/nightly/pages/models/gemma/gemma4-vl.mdx": 5 "./versions/nightly/pages/models/gemma/index.mdx": 5 -"./versions/nightly/pages/models/nemotron/nemotron3.5-lightning.mdx": 6 -"./versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx": 7 +"./versions/nightly/pages/models/nemotron/nemotron3.5-lightning.mdx": 8 +"./versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx": 11 "./versions/nightly/pages/models/qwen/qwen3-vl.mdx": 7 "./versions/nightly/pages/releases/known-issues.mdx": 5 "./versions/nightly/pages/training/optimizer-scheduler.mdx": 5 "./versions/nightly/pages/models/glm/index.mdx": 6 -"./versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx": 8 +"./versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx": 5 "./versions/nightly/pages/models/qwen/qwen3.6-35b-a3b.mdx": 8 "./versions/nightly/pages/models/qwen/qwen3.8-27b.mdx": 9 "./versions/nightly/pages/training/logging.mdx": 6 @@ -118,3 +118,6 @@ "./versions/nightly/pages/training/packed-sequences.mdx": 21 "./versions/nightly/pages/training/multi-token-prediction.mdx": 22 "./versions/nightly/pages/training/callbacks.mdx": 23 +"./versions/nightly/pages/models/nemotron/nemotron3-ultra.mdx": 7 +"./versions/nightly/pages/models/nemotron/nemotron3.5-super-vl.mdx": 9 +"./versions/nightly/pages/models/nemotron/nemotron3.5-super-vl-text-only.mdx": 10 diff --git a/docs/fern/versions/nightly.yml b/docs/fern/versions/nightly.yml index 8440d055ad..77fb26f210 100644 --- a/docs/fern/versions/nightly.yml +++ b/docs/fern/versions/nightly.yml @@ -130,17 +130,17 @@ navigation: path: ./nightly/pages/models/nemotron/nemotron3-nano.mdx - page: Nemotron 3 Nano 4B path: ./nightly/pages/models/nemotron/nemotron3-nano-4b.mdx + - page: Nemotron-3 Nano Omni + path: ./nightly/pages/models/nemotron/nemotron-3-omni.mdx - page: Nemotron 3 Super path: ./nightly/pages/models/nemotron/nemotron3-super.mdx + - page: Nemotron 3 Ultra + path: ./nightly/pages/models/nemotron/nemotron3-ultra.mdx - page: Nemotron 3.5 Lightning path: ./nightly/pages/models/nemotron/nemotron3.5-lightning.mdx # Super VL guides remain in GitHub, but are excluded from publication. - - page: Nemotron 3 Ultra - path: ./nightly/pages/models/nemotron/nemotron3-ultra.mdx - page: Nemotron Nano V2 VL path: ./nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx - - page: Nemotron-3 Nano Omni - path: ./nightly/pages/models/nemotron/nemotron-3-omni.mdx section: Nemotron - contents: - page: OLMoE diff --git a/docs/fern/versions/nightly/pages/index.mdx b/docs/fern/versions/nightly/pages/index.mdx index 450eec4263..766fc0d0f5 100644 --- a/docs/fern/versions/nightly/pages/index.mdx +++ b/docs/fern/versions/nightly/pages/index.mdx @@ -21,13 +21,13 @@ - [05/20/2026] [**DeepSeek V4**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/deepseek_v4) is now merged on **main**! See the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/deepseek_v4/README.md) for conversion and inference details. -- [05/20/2026] [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) day-0 branch support is now merged on **main**! The 30B-A3B MoE multimodal model supports image, video, audio, and text workflows with checkpoint conversion, inference, SFT, and PEFT (LoRA) examples. Read the [NVIDIA Blog](https://blogs.nvidia.com/blog/nemotron-3-nano-omni-multimodal-ai-agents/) and see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) for the full walkthrough. +- [05/20/2026] [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) day-0 branch support is now merged on **main**! The 30B-A3B MoE multimodal model supports image, video, audio, and text workflows with checkpoint conversion, inference, SFT, and PEFT (LoRA) examples. Read the [NVIDIA Blog](https://blogs.nvidia.com/blog/nemotron-3-nano-omni-multimodal-ai-agents/) and see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) for the full walkthrough. - [05/19/2026] [**Nemotron-Labs Diffusion**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron_labs_diffusion) is now supported on **main** with autoregressive-to-diffusion conversion, continuous pretraining, checkpoint conversion, and inference workflows. Read the [NVIDIA Research blog](https://research.nvidia.com/publication/2026-05_nemotron-labs-diffusion-tri-mode-language-model-unifying-autoregressive) for the tri-mode language model overview. - [05/06/2026] [**Gemma 4 VL 26B-A4B**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/gemma/gemma4_vl) is now supported! Checkpoint conversion, SFT, and PEFT (LoRA) recipes for Google's MoE vision-language model (26B total / 4B active params, 128 experts top-k=8, dual sliding/global attention with K=V tying on full-attention layers) are available on **main**. See the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/gemma/gemma4_vl/README.md) for the full walkthrough. -- [04/28/2026] Day 0 support for [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16), a 30B-A3B MoE multimodal model that jointly processes image, video, audio, and text. Checkpoint conversion, SFT, and LoRA recipes are available on **main** β€” see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) for the full walkthrough. +- [04/28/2026] Day 0 support for [**Nemotron-3 Nano Omni**](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16), a 30B-A3B MoE multimodal model that jointly processes image, video, audio, and text. Checkpoint conversion, SFT, and LoRA recipes are available on **main** β€” see the [examples README](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) for the full walkthrough. - [04/19/2026] [**Qwen3.6-35B-A3B**](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is now supported! Qwen3.6 uses the same architecture as Qwen3.5 VL MoE (`Qwen3_5MoeForConditionalGeneration`) and works with the existing [Qwen3.5-VL bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/qwen_vl) out of the box β€” no code changes needed. HFβ†’Megatron conversion and inference verified. @@ -45,13 +45,13 @@ - [03/31/2026] **Agent Skills for Megatron Bridge!** We've added a [`skills/`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/skills) directory with structured guides that AI coding agents (Cursor, Claude Code, Codex, etc.) can use to help you add model support, set up dev environments, tune performance, and more. Try them out, and PRs to improve or add new skills are very welcome! -- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). +- [03/26/2026] [**Nemotron 3 Super**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron) is now on **main**! Checkpoint conversion and SFT/LoRA recipes (120B-A12B) are available in the main branch. Read the [blog post](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/). - [03/12/2026] **Deprecating Python 3.10 support:** We're officially dropping Python 3.10 support with the upcoming 0.4.0 release. Downstream applications must raise their lower boundary to 3.12 to stay compatible with Megatron-Bridge. - [12/16/2025] [Mind Lab](https://macaron.im/mindlab) successfully used Megatron-bridge and [VeRL](https://github.com/volcengine/verl) to trained GRPO Lora for Trillion-parameter model on 64 H800 - See their [techblog](https://macaron.im/mindlab/research/building-trillion-parameter-reasoning-rl-with-10-gpus). -- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3/nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) +- [12/15/2025] Day 0 support for [NVIDIA-NeMotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)! [Reproducible code](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/nemotron/nemotron_3_nano) and custom NGC container: [nvcr.io/nvidia/nemo:25.11.nemotron_3_nano](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=25.11.nemotron_3_nano) ## Overview diff --git a/docs/fern/versions/nightly/pages/models/README.mdx b/docs/fern/versions/nightly/pages/models/README.mdx index 0897365610..9af151354d 100644 --- a/docs/fern/versions/nightly/pages/models/README.mdx +++ b/docs/fern/versions/nightly/pages/models/README.mdx @@ -115,10 +115,6 @@ Choose a model to inspect its recorded import/export, training, and precision co 6 models
- - Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 - ◐ Partial - NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 ◐ Partial @@ -127,6 +123,10 @@ Choose a model to inspect its recorded import/export, training, and precision co NVIDIA-Nemotron-3-Nano-4B-BF16 βœ“ Verified + + Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 + ◐ Partial + NVIDIA-Nemotron-3-Super-120B-A12B-BF16 {/* pragma: allowlist secret */ } diff --git a/docs/fern/versions/nightly/pages/models/nemotron/index.mdx b/docs/fern/versions/nightly/pages/models/nemotron/index.mdx index afcb193a9e..d7458b721f 100644 --- a/docs/fern/versions/nightly/pages/models/nemotron/index.mdx +++ b/docs/fern/versions/nightly/pages/models/nemotron/index.mdx @@ -8,13 +8,13 @@ Nemotron model documentation is organized by model variant. | Nemotron H and Nemotron Nano v2 | [nemotronh.md](nemotronh.md) | | Nemotron-3 Nano 30B-A3B | [nemotron3-nano.md](nemotron3-nano.md) | | Nemotron-3 Nano 4B | [nemotron3-nano-4b.md](nemotron3-nano-4b.md) | +| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) | | Nemotron-3 Super | [nemotron3-super.md](nemotron3-super.md) | -| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) | | Nemotron-3 Ultra | [nemotron3-ultra.md](nemotron3-ultra.md) | +| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) | | Nemotron Nano V2 VL | [nemotron-nano-v2-vl.md](nemotron-nano-v2-vl.md) | -| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) | ## Examples -- Nemotron-3 examples: [`examples/models/nemotron/nemotron_3/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3/README.md) -- Nemotron-3 Omni examples: [`examples/models/nemotron/nemotron_3_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) +- Nemotron-3 examples: [`examples/models/nemotron/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/README.md) +- Nemotron-3 Omni examples: [`examples/models/nemotron/nemotron_3_nano_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) diff --git a/docs/fern/versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx b/docs/fern/versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx index bf4287309f..5d54a4c9b1 100644 --- a/docs/fern/versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx +++ b/docs/fern/versions/nightly/pages/models/nemotron/nemotron-3-omni.mdx @@ -23,7 +23,7 @@ deployment. For the full setup, conversion, inference, training, evaluation, and LoRA merge / adapter export workflows, see -[`examples/models/nemotron/nemotron_3_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md). +[`examples/models/nemotron/nemotron_3_nano_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md). {/* BEGIN GENERATED VERIFIED CONFIGURATIONS */} diff --git a/docs/fern/versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx b/docs/fern/versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx index 4537d74691..7ddce24ca6 100644 --- a/docs/fern/versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx +++ b/docs/fern/versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx @@ -78,122 +78,18 @@ Note: e.g. `--prompt="Describe what you see."`. ## Finetuning Recipes -Before training, ensure the following environment variables are set. -1. `SAVE_DIR`: to specify a checkpoint and log saving directory, used in the commands below. -2. `HF_TOKEN`: to download models from HF Hub. -3. `HF_HOME`: (optional) to avoid re-downloading models and datasets every time. -4. `WANDB_API_KEY`: (optional) to enable WandB logging. -### Full Finetuning -Example usage for full parameter finetuning using the -[Raven dataset](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron/viewer/raven): +The deprecated Nano v2 VL example script and its YAML override files have +been removed from `main`. The public recipe entry points are unchanged: -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ -``` - -Note: -- The config file `examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_override_example.yaml` contains a list of arguments - that can be overridden in the command. For example, you can set `train.global_batch_size=<batch size>` in the command. -- To change to another built-in dataset, set its `dataset.source.dataset_name` preset. To replace the recipe preset with a custom native source from the CLI, set `dataset.source.dataset_name=null` plus `dataset.source.path_or_dataset=`; custom schemas also require an explicit adapter. See the dataset section below. -- After training, you can run inference with `hf_to_megatron_generate_vlm.py` by supplying the trained megatron checkpoint. - You can also export the trained checkpoint to Hugging Face format. -- This full finetuning recipe requires at least 4xH100 (80G) GPUs. - -### Parameter-Efficient Finetuning (PEFT) -Parameter-efficient finetuning (PEFT) using LoRA is supported. -LoRA can be independently applied to the vision model, vision projection, and language model. We support two commonly used -settings out of the box in the example script: -1. Apply LoRA to the language model, and fully finetune the vision model and projection (used when the visual - distribution is substantially different from pretrained.) - -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---lora-on-language-model \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -model.freeze_language_model=True \ -model.freeze_vision_model=False \ -model.freeze_vision_projection=False -``` - -2. Apply LoRA to all linear layers in attention and MLP modules of the vision model, vision projection, and the language model. - -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---lora-on-language-model \ ---lora-on-vision-model \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -model.freeze_language_model=True \ -model.freeze_vision_model=True \ -model.freeze_vision_projection=True +```python +from megatron.bridge.recipes.nemotron_vl import ( + nemotron_nano_v2_vl_12b_sft_config, + nemotron_nano_v2_vl_12b_peft_config, +) ``` -These LoRA finetuning recipe requires at least 2xH100 (80G) GPUs. - -A LoRA checkpoint only contains the learnable adapter weights. In order to convert the LoRA checkpoint to Hugging Face -format for downstream evaluation, it is necessary to merge the LoRA adapters back to the base model. - -```bash -uv run python examples/peft/merge_lora.py \ ---hf-model-path $HF_MODEL_PATH \ ---lora-checkpoint /iter_N \ ---output -``` -The output is a merged Hugging Face checkpoint for downstream evaluation. - -## Example Datasets - -Megatron Bridge supports various vision-language dataset examples which can be used to finetune Nemotron Nano V2 VL: -| Dataset | Dataset preset | Description | -|---------|------------|-------------| -| [cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) | `cord_v2` | OCR receipts: Single-image-text dataset for receipt understanding, outputs xml-like annotated text. | -| [MedPix-VQA](https://huggingface.co/datasets/mmoukouba/MedPix-VQA) | `medpix` | Medical VQA: Single-image question-answer dataset covering clinical medical images and free-form answers. | -| [The Cauldron (Raven subset)](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) | `raven` | Visual reasoning: Multi-image, vision reasoning dataset for analogical reasoning in different visual layouts. | -| [LLaVA-Video-178K (0_30_s_nextqa subset)](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K) | `llava_video_178k` | Video understanding: video question-answer dataset covering every-day scenarios. | - -`cord-v2` is a tiny dataset and is included for demonstration only. This dataset is not recommended for PEFT tuning for this model since the XML output format interacts with the special tokens, leading to unexpected results. - -Note on video training example: -- We provide a video config yaml file instead of the default config yaml file that overwrites a few commands. Please - pass in `--config-file "examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml"`. -- The LLaVA video dataset requires manual download beforehand. Please place the downloaded and extracted video files - in a folder `VIDEO_ROOT` and pass it to the adapter with `dataset.source.adapter_kwargs={"video_root_path":$VIDEO_ROOT}`. - In the nextqa subset example, `VIDEO_ROOT` should look like - ``` - $VIDEO_ROOT/ - β”œβ”€β”€ NextQA/ - β”‚ └── NExTVideo/ - β”‚ └── 0000/ - β”‚ └── 2440175990.mp4 - β”‚ └── 0001/ - β”‚ └── ... - └── ... - ``` - -Full video training example command: -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---config-file "examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml" \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -dataset.source.adapter_kwargs={"video_root_path":$VIDEO_ROOT} -``` +For the historical full-finetuning, component-specific LoRA, and video +examples, use the [Bridge 0.6.0 guide](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/v0.6.0/docs/models/nemotron/nemotron-nano-v2-vl.md#finetuning-recipes) +with a `v0.6.0` checkout. Those commands and YAML files belong to that release +and are no longer available on `main`. diff --git a/docs/fern/versions/nightly/pages/models/nemotron/nemotron3-ultra.mdx b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3-ultra.mdx index 91ce1fb4a9..ca05305440 100644 --- a/docs/fern/versions/nightly/pages/models/nemotron/nemotron3-ultra.mdx +++ b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3-ultra.mdx @@ -8,7 +8,7 @@ Hugging Face to Megatron conversion, inference, DCLM pretraining, packed OpenMathInstruct-2 full SFT, and packed OpenMathInstruct-2 LoRA PEFT. Use the main example README for setup and scripts: -[`examples/models/nemotron/nemotron_3/ultra/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3/ultra/README.md). +[`examples/models/nemotron/nemotron_3_ultra/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_ultra/README.md). {/* BEGIN GENERATED VERIFIED CONFIGURATIONS */} diff --git a/docs/models/README.md b/docs/models/README.md index fe6b8fcb1a..89c8b9c0ee 100644 --- a/docs/models/README.md +++ b/docs/models/README.md @@ -115,10 +115,6 @@ Choose a model to inspect its recorded import/export, training, and precision co 6 models
- - Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 - ◐ Partial - NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 ◐ Partial @@ -127,6 +123,10 @@ Choose a model to inspect its recorded import/export, training, and precision co NVIDIA-Nemotron-3-Nano-4B-BF16 βœ“ Verified + + Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 + ◐ Partial + NVIDIA-Nemotron-3-Super-120B-A12B-BF16 ◐ Partial diff --git a/docs/models/nemotron/index.md b/docs/models/nemotron/index.md index 959f5692a6..075bd1c255 100644 --- a/docs/models/nemotron/index.md +++ b/docs/models/nemotron/index.md @@ -9,11 +9,11 @@ llama-nemotron.md nemotronh.md nemotron3-nano.md nemotron3-nano-4b.md +nemotron-3-omni.md nemotron3-super.md -nemotron3.5-lightning.md nemotron3-ultra.md +nemotron3.5-lightning.md nemotron-nano-v2-vl.md -nemotron-3-omni.md ``` | Variant | Guide | @@ -22,13 +22,13 @@ nemotron-3-omni.md | Nemotron H and Nemotron Nano v2 (deprecated) | [nemotronh.md](nemotronh.md) | | Nemotron-3 Nano 30B-A3B | [nemotron3-nano.md](nemotron3-nano.md) | | Nemotron-3 Nano 4B | [nemotron3-nano-4b.md](nemotron3-nano-4b.md) | +| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) | | Nemotron-3 Super | [nemotron3-super.md](nemotron3-super.md) | -| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) | | Nemotron-3 Ultra | [nemotron3-ultra.md](nemotron3-ultra.md) | +| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) | | Nemotron Nano V2 VL (deprecated) | [nemotron-nano-v2-vl.md](nemotron-nano-v2-vl.md) | -| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) | ## Examples -- Nemotron-3 examples: [`examples/models/nemotron/nemotron_3/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3/README.md) -- Nemotron-3 Omni examples: [`examples/models/nemotron/nemotron_3_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md) +- Nemotron-3 examples: [`examples/models/nemotron/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/README.md) +- Nemotron-3 Omni examples: [`examples/models/nemotron/nemotron_3_nano_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md) diff --git a/docs/models/nemotron/nemotron-3-omni.md b/docs/models/nemotron/nemotron-3-omni.md index d5e0ab0583..f0e31a5063 100644 --- a/docs/models/nemotron/nemotron-3-omni.md +++ b/docs/models/nemotron/nemotron-3-omni.md @@ -23,7 +23,7 @@ deployment. For the full setup, conversion, inference, training, evaluation, and LoRA merge / adapter export workflows, see -[`examples/models/nemotron/nemotron_3_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_omni/README.md). +[`examples/models/nemotron/nemotron_3_nano_omni/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_nano_omni/README.md). diff --git a/docs/models/nemotron/nemotron-nano-v2-vl.md b/docs/models/nemotron/nemotron-nano-v2-vl.md index 090cb7b235..957892f097 100644 --- a/docs/models/nemotron/nemotron-nano-v2-vl.md +++ b/docs/models/nemotron/nemotron-nano-v2-vl.md @@ -78,123 +78,18 @@ Note: ## Finetuning Recipes -Before training, ensure the following environment variables are set. -1. `SAVE_DIR`: to specify a checkpoint and log saving directory, used in the commands below. -2. `HF_TOKEN`: to download models from HF Hub. -3. `HF_HOME`: (optional) to avoid re-downloading models and datasets every time. -4. `WANDB_API_KEY`: (optional) to enable WandB logging. -### Full Finetuning -Example usage for full parameter finetuning using the -[Raven dataset](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron/viewer/raven): +The deprecated Nano v2 VL example script and its YAML override files have +been removed from `main`. The public recipe entry points are unchanged: -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ -``` - -Note: -- The config file `examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_override_example.yaml` contains a list of arguments - that can be overridden in the command. For example, you can set `train.global_batch_size=` in the command. -- To change to another built-in dataset, set its `dataset.source.dataset_name` preset. To replace the recipe preset with a custom native source from the CLI, set `dataset.source.dataset_name=null` plus `dataset.source.path_or_dataset=`; custom schemas also require an explicit adapter. See the dataset section below. -- After training, you can run inference with `hf_to_megatron_generate_vlm.py` by supplying the trained megatron checkpoint. - You can also export the trained checkpoint to Hugging Face format. -- This full finetuning recipe requires at least 4xH100 (80G) GPUs. - -### Parameter-Efficient Finetuning (PEFT) -Parameter-efficient finetuning (PEFT) using LoRA is supported. -LoRA can be independently applied to the vision model, vision projection, and language model. We support two commonly used -settings out of the box in the example script: -1. Apply LoRA to the language model, and fully finetune the vision model and projection (used when the visual - distribution is substantially different from pretrained.) - -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---lora-on-language-model \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -model.freeze_language_model=True \ -model.freeze_vision_model=False \ -model.freeze_vision_projection=False -``` - -2. Apply LoRA to all linear layers in attention and MLP modules of the vision model, vision projection, and the language model. - -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---lora-on-language-model \ ---lora-on-vision-model \ -dataset.source.dataset_name=raven \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -model.freeze_language_model=True \ -model.freeze_vision_model=True \ -model.freeze_vision_projection=True +```python +from megatron.bridge.recipes.nemotron_vl import ( + nemotron_nano_v2_vl_12b_sft_config, + nemotron_nano_v2_vl_12b_peft_config, +) ``` -These LoRA finetuning recipe requires at least 2xH100 (80G) GPUs. - -A LoRA checkpoint only contains the learnable adapter weights. In order to convert the LoRA checkpoint to Hugging Face -format for downstream evaluation, it is necessary to merge the LoRA adapters back to the base model. - -```bash -uv run python examples/peft/merge_lora.py \ ---hf-model-path $HF_MODEL_PATH \ ---lora-checkpoint /iter_N \ ---output -``` -The output is a merged Hugging Face checkpoint for downstream evaluation. - - -## Example Datasets - -Megatron Bridge supports various vision-language dataset examples which can be used to finetune Nemotron Nano V2 VL: -| Dataset | Dataset preset | Description | -|---------|------------|-------------| -| [cord-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) | `cord_v2` | OCR receipts: Single-image-text dataset for receipt understanding, outputs xml-like annotated text. | -| [MedPix-VQA](https://huggingface.co/datasets/mmoukouba/MedPix-VQA) | `medpix` | Medical VQA: Single-image question-answer dataset covering clinical medical images and free-form answers. | -| [The Cauldron (Raven subset)](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) | `raven` | Visual reasoning: Multi-image, vision reasoning dataset for analogical reasoning in different visual layouts. | -| [LLaVA-Video-178K (0_30_s_nextqa subset)](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K) | `llava_video_178k` | Video understanding: video question-answer dataset covering every-day scenarios. | - -`cord-v2` is a tiny dataset and is included for demonstration only. This dataset is not recommended for PEFT tuning for this model since the XML output format interacts with the special tokens, leading to unexpected results. - -Note on video training example: -- We provide a video config yaml file instead of the default config yaml file that overwrites a few commands. Please - pass in `--config-file "examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml"`. -- The LLaVA video dataset requires manual download beforehand. Please place the downloaded and extracted video files - in a folder `VIDEO_ROOT` and pass it to the adapter with `dataset.source.adapter_kwargs={"video_root_path":$VIDEO_ROOT}`. - In the nextqa subset example, `VIDEO_ROOT` should look like - ``` - $VIDEO_ROOT/ - β”œβ”€β”€ NextQA/ - β”‚ └── NExTVideo/ - β”‚ └── 0000/ - β”‚ └── 2440175990.mp4 - β”‚ └── 0001/ - β”‚ └── ... - └── ... - ``` - -Full video training example command: -```bash -uv run python -m torch.distributed.run --nproc-per-node=8 examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py \ ---hf-model-path $HF_MODEL_PATH \ ---pretrained-checkpoint $MEGATRON_MODEL_PATH \ ---config-file "examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml" \ -logger.wandb_project= \ -logger.wandb_save_dir=$SAVE_DIR \ -checkpoint.save=$SAVE_DIR/ \ -dataset.source.adapter_kwargs={"video_root_path":$VIDEO_ROOT} -``` +For the historical full-finetuning, component-specific LoRA, and video +examples, use the [Bridge 0.6.0 guide](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/v0.6.0/docs/models/nemotron/nemotron-nano-v2-vl.md#finetuning-recipes) +with a `v0.6.0` checkout. Those commands and YAML files belong to that release +and are no longer available on `main`. diff --git a/docs/models/nemotron/nemotron3-ultra.md b/docs/models/nemotron/nemotron3-ultra.md index ca59ad9dee..e04d66cdf6 100644 --- a/docs/models/nemotron/nemotron3-ultra.md +++ b/docs/models/nemotron/nemotron3-ultra.md @@ -8,7 +8,7 @@ Hugging Face to Megatron conversion, inference, DCLM pretraining, packed OpenMathInstruct-2 full SFT, and packed OpenMathInstruct-2 LoRA PEFT. Use the main example README for setup and scripts: -[`examples/models/nemotron/nemotron_3/ultra/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3/ultra/README.md). +[`examples/models/nemotron/nemotron_3_ultra/README.md`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/models/nemotron/nemotron_3_ultra/README.md). diff --git a/examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml b/examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml index dd58724ff7..6c71f59866 100644 --- a/examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml +++ b/examples/model_verification_cards/nemotron-3-nano-omni-30b-a3b-reasoning/card.yaml @@ -149,7 +149,7 @@ items: precision: bf16 command: > uv run python -m torch.distributed.run --standalone --nproc_per_node=8 - examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py + examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py --hf_model_path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --hf-revision 24e67ea000b7c2837fc8f9488aa2008524fac8ba --megatron_model_path diff --git a/examples/models/nemotron/README.md b/examples/models/nemotron/README.md new file mode 100644 index 0000000000..17cc21e4f3 --- /dev/null +++ b/examples/models/nemotron/README.md @@ -0,0 +1,87 @@ +# Nemotron Examples + +This directory contains model-specific examples for the Nemotron family: + +| Model | Parameters | Active Parameters | Subdirectory | +|-------|-----------|-------------------|--------------| +| Nemotron 3 Nano | 30B | A3B | [nemotron_3_nano/](nemotron_3_nano/) | +| Nemotron 3 Nano Omni | 30B | A3B | [nemotron_3_nano_omni/](nemotron_3_nano_omni/) | +| Nemotron 3 Super | 120B | A12B | [nemotron_3_super/](nemotron_3_super/) | +| Nemotron 3 Ultra | 550B | A55B | [nemotron_3_ultra/](nemotron_3_ultra/) | +| Nemotron 3.5 Lightning | 30B | A3B | [nemotron_3_5_lightning/](nemotron_3_5_lightning/) | +| Nemotron 3.5 Super VL | 120B language model | A12B | [nemotron_3_5_super_vl/](nemotron_3_5_super_vl/) (draft) | + +## Path migration + +Model examples now live directly under this directory. Update saved commands +and example imports to use the model-specific paths above: the former +`nemotron_3/` hierarchy is flattened, Lightning and Super VL use +`nemotron_3_5_*`, and Omni uses `nemotron_3_nano_omni`. Old example paths +are intentionally removed. The deprecated Nano v2 VL example script and +YAML overrides have also been removed. Public `megatron.bridge` recipe names and model +conversion entry points are unchanged. Historical release branches retain +their own layouts; use the README from the branch you check out. + +## Workspace Configuration + +All scripts use a `WORKSPACE` environment variable to define the base directory for checkpoints and results. By default, this is set to `/workspace`. You can override it: + +```bash +export WORKSPACE=/your/custom/path +``` + +Directory structure: +- `${WORKSPACE}/models/` - Converted checkpoints +- `${WORKSPACE}/results/` - Training outputs and experiment results + +## Checkpoint Conversion + +Nano and Super have conversion scripts: [nemotron_3_nano/conversion.sh](nemotron_3_nano/conversion.sh), [nemotron_3_super/conversion.sh](nemotron_3_super/conversion.sh). Ultra has Slurm examples for multi-node conversion, inference, and OpenMath training; see [nemotron_3_ultra/](nemotron_3_ultra/) and [Ultra documentation](../../../docs/models/nemotron/nemotron3-ultra.md). + +## Training Recipes + +Available recipes: + +**Nano** ([source](../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_nano.py)): +- `nemotron_3_nano_pretrain_config`: Pretraining +- `nemotron_3_nano_sft_config`: Supervised fine-tuning +- `nemotron_3_nano_peft_config`: PEFT with LoRA support + +**Super** ([source](../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_super.py)): +- `nemotron_3_super_pretrain_config`: Pretraining +- `nemotron_3_super_sft_config`: Supervised fine-tuning +- `nemotron_3_super_peft_config`: PEFT with LoRA support + +**Ultra** ([source](../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_ultra.py)): +- `nemotron_3_ultra_pretrain_config`: Pretraining +- `nemotron_3_ultra_sft_openmathinstruct2_packed_config`: Packed OpenMathInstruct-2 SFT +- `nemotron_3_ultra_peft_openmathinstruct2_packed_config`: Packed OpenMathInstruct-2 PEFT + +Before training, ensure the following are configured: +1. **Container Image**: Set `CONTAINER_IMAGE` in the SLURM scripts to your container path +2. **Container Mounts**: (optional) Set `CONTAINER_MOUNTS` for data and workspace directories +3. **Environment Variables**: + - `HF_TOKEN`: to download models from HF Hub (if required) + - `HF_HOME`: (optional) to avoid re-downloading models and datasets + - `WANDB_API_KEY`: (optional) to enable WandB logging + +All training scripts use SLURM for containerized multi-node training. + +### Nano + +See the SLURM scripts in [nemotron_3_nano/](nemotron_3_nano/): [slurm_pretrain.sh](nemotron_3_nano/slurm_pretrain.sh), [slurm_pretrain_fsdp.sh](nemotron_3_nano/slurm_pretrain_fsdp.sh) - fsdp_dtensor is the ckpt format supported as of now. To convert to torch_dcp format follow this [guide](https://github.com/NVIDIA/Megatron-LM/tree/main/examples/megatron_fsdp#sbatch_checkpoint_convertsh) , [slurm_sft.sh](nemotron_3_nano/slurm_sft.sh), [slurm_peft.sh](nemotron_3_nano/slurm_peft.sh). + +### Super + +See the SLURM scripts in [nemotron_3_super/](nemotron_3_super/): [slurm_pretrain.sh](nemotron_3_super/slurm_pretrain.sh), [slurm_sft.sh](nemotron_3_super/slurm_sft.sh), [slurm_peft.sh](nemotron_3_super/slurm_peft.sh). + +### Ultra + +See [nemotron_3_ultra/slurm_inference.sh](nemotron_3_ultra/slurm_inference.sh) for the 4-node inference pattern. +For OpenMath training, use [nemotron_3_ultra/slurm_sft.sh](nemotron_3_ultra/slurm_sft.sh) and +[nemotron_3_ultra/slurm_peft.sh](nemotron_3_ultra/slurm_peft.sh), which default to the current +OpenMath tuning starting points. + +## Evaluation + +Coming soon. diff --git a/examples/models/nemotron/nemotron_3/README.md b/examples/models/nemotron/nemotron_3/README.md deleted file mode 100644 index 65c432fcec..0000000000 --- a/examples/models/nemotron/nemotron_3/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Nemotron 3 Examples - -This directory contains example scripts for Nemotron 3 language models: - -| Model | Parameters | Active Parameters | Subdirectory | -|-------|-----------|-------------------|--------------| -| Nemotron 3 Nano | 30B | A3B | [nano/](nano/) | -| Nemotron 3.5 Lightning | 30B | A3B | [lightning/](lightning/) | -| Nemotron 3 Super | 120B | A12B | [super/](super/) | -| Nemotron 3 Ultra | 550B | A55B | [ultra/](ultra/) | - -## Workspace Configuration - -All scripts use a `WORKSPACE` environment variable to define the base directory for checkpoints and results. By default, this is set to `/workspace`. You can override it: - -```bash -export WORKSPACE=/your/custom/path -``` - -Directory structure: -- `${WORKSPACE}/models/` - Converted checkpoints -- `${WORKSPACE}/results/` - Training outputs and experiment results - -## Checkpoint Conversion - -Nano and Super have conversion scripts: [nano/conversion.sh](nano/conversion.sh), [super/conversion.sh](super/conversion.sh). Ultra has Slurm examples for multi-node conversion, inference, and OpenMath training; see [ultra/](ultra/) and [Ultra documentation](../../../../docs/models/nemotron/nemotron3-ultra.md). - -## Training Recipes - -Available recipes: - -**Nano** ([source](../../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_nano.py)): -- `nemotron_3_nano_pretrain_config`: Pretraining -- `nemotron_3_nano_sft_config`: Supervised fine-tuning -- `nemotron_3_nano_peft_config`: PEFT with LoRA support - -**Super** ([source](../../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_super.py)): -- `nemotron_3_super_pretrain_config`: Pretraining -- `nemotron_3_super_sft_config`: Supervised fine-tuning -- `nemotron_3_super_peft_config`: PEFT with LoRA support - -**Ultra** ([source](../../../../src/megatron/bridge/recipes/nemotronh/nemotron_3_ultra.py)): -- `nemotron_3_ultra_pretrain_config`: Pretraining -- `nemotron_3_ultra_sft_openmathinstruct2_packed_config`: Packed OpenMathInstruct-2 SFT -- `nemotron_3_ultra_peft_openmathinstruct2_packed_config`: Packed OpenMathInstruct-2 PEFT - -Before training, ensure the following are configured: -1. **Container Image**: Set `CONTAINER_IMAGE` in the SLURM scripts to your container path -2. **Container Mounts**: (optional) Set `CONTAINER_MOUNTS` for data and workspace directories -3. **Environment Variables**: - - `HF_TOKEN`: to download models from HF Hub (if required) - - `HF_HOME`: (optional) to avoid re-downloading models and datasets - - `WANDB_API_KEY`: (optional) to enable WandB logging - -All training scripts use SLURM for containerized multi-node training. - -### Nano - -See the SLURM scripts in [nano/](nano/): [slurm_pretrain.sh](nano/slurm_pretrain.sh), [slurm_pretrain_fsdp.sh](nano/slurm_pretrain_fsdp.sh) - fsdp_dtensor is the ckpt format supported as of now. To convert to torch_dcp format follow this [guide](https://github.com/NVIDIA/Megatron-LM/tree/main/examples/megatron_fsdp#sbatch_checkpoint_convertsh) , [slurm_sft.sh](nano/slurm_sft.sh), [slurm_peft.sh](nano/slurm_peft.sh). - -### Super - -See the SLURM scripts in [super/](super/): [slurm_pretrain.sh](super/slurm_pretrain.sh), [slurm_sft.sh](super/slurm_sft.sh), [slurm_peft.sh](super/slurm_peft.sh). - -### Ultra - -See [ultra/slurm_inference.sh](ultra/slurm_inference.sh) for the 4-node inference pattern. -For OpenMath training, use [ultra/slurm_sft.sh](ultra/slurm_sft.sh) and -[ultra/slurm_peft.sh](ultra/slurm_peft.sh), which default to the current -OpenMath tuning starting points. - -## Evaluation - -Coming soon. diff --git a/examples/models/nemotron/nemotron_3/lightning/README.md b/examples/models/nemotron/nemotron_3_5_lightning/README.md similarity index 100% rename from examples/models/nemotron/nemotron_3/lightning/README.md rename to examples/models/nemotron/nemotron_3_5_lightning/README.md diff --git a/examples/models/nemotron/nemotron_3/super_vl/README.md b/examples/models/nemotron/nemotron_3_5_super_vl/README.md similarity index 96% rename from examples/models/nemotron/nemotron_3/super_vl/README.md rename to examples/models/nemotron/nemotron_3_5_super_vl/README.md index 734d168929..36509cc5a1 100644 --- a/examples/models/nemotron/nemotron_3/super_vl/README.md +++ b/examples/models/nemotron/nemotron_3_5_super_vl/README.md @@ -13,7 +13,7 @@ For language-only conversion and reuse of the existing Nemotron 3 Super recipes, see [the text-only bridge guide](text-only.md). The functional recipes are defined under -[`src/megatron/bridge/recipes/nemotronh_multimodal/`](../../../../../src/megatron/bridge/recipes/nemotronh_multimodal/): +[`src/megatron/bridge/recipes/nemotronh_multimodal/`](../../../../src/megatron/bridge/recipes/nemotronh_multimodal/): - `nemotron_35_super_vl_pretrain_config` - `nemotron_35_super_vl_sft_config` @@ -92,4 +92,4 @@ must preserve that qualification if language-only adapters are intended. - Add deterministic image and video inference examples. - Add SFT and PEFT dataset requirements and launch examples. - Document dynamic-resolution image and temporal-video preprocessing. -- Link the completed page from the Nemotron 3 example index. +- Replace the draft label in the Nemotron example index when the page is complete. diff --git a/examples/models/nemotron/nemotron_3/super_vl/text-only.md b/examples/models/nemotron/nemotron_3_5_super_vl/text-only.md similarity index 100% rename from examples/models/nemotron/nemotron_3/super_vl/text-only.md rename to examples/models/nemotron/nemotron_3_5_super_vl/text-only.md diff --git a/examples/models/nemotron/nemotron_3/nano/conversion.sh b/examples/models/nemotron/nemotron_3_nano/conversion.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/conversion.sh rename to examples/models/nemotron/nemotron_3_nano/conversion.sh diff --git a/examples/models/nemotron/nemotron_3/nano/finetune_nemotron_3_nano.py b/examples/models/nemotron/nemotron_3_nano/finetune_nemotron_3_nano.py similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/finetune_nemotron_3_nano.py rename to examples/models/nemotron/nemotron_3_nano/finetune_nemotron_3_nano.py diff --git a/examples/models/nemotron/nemotron_3/nano/pretrain_nemotron_3_nano.py b/examples/models/nemotron/nemotron_3_nano/pretrain_nemotron_3_nano.py similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/pretrain_nemotron_3_nano.py rename to examples/models/nemotron/nemotron_3_nano/pretrain_nemotron_3_nano.py diff --git a/examples/models/nemotron/nemotron_3/nano/slurm_peft.sh b/examples/models/nemotron/nemotron_3_nano/slurm_peft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/slurm_peft.sh rename to examples/models/nemotron/nemotron_3_nano/slurm_peft.sh diff --git a/examples/models/nemotron/nemotron_3/nano/slurm_pretrain.sh b/examples/models/nemotron/nemotron_3_nano/slurm_pretrain.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/slurm_pretrain.sh rename to examples/models/nemotron/nemotron_3_nano/slurm_pretrain.sh diff --git a/examples/models/nemotron/nemotron_3/nano/slurm_pretrain_fsdp.sh b/examples/models/nemotron/nemotron_3_nano/slurm_pretrain_fsdp.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/slurm_pretrain_fsdp.sh rename to examples/models/nemotron/nemotron_3_nano/slurm_pretrain_fsdp.sh diff --git a/examples/models/nemotron/nemotron_3/nano/slurm_sft.sh b/examples/models/nemotron/nemotron_3_nano/slurm_sft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/nano/slurm_sft.sh rename to examples/models/nemotron/nemotron_3_nano/slurm_sft.sh diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_nano_omni/README.md similarity index 92% rename from examples/models/nemotron/nemotron_3_omni/README.md rename to examples/models/nemotron/nemotron_3_nano_omni/README.md index 0ce859daf6..95f184711e 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_nano_omni/README.md @@ -43,11 +43,14 @@ The scripts expect the following layout: - `${WORKSPACE}/assets/` β€” image / video / audio files used for `inference.sh` (auto-downloaded from the public HF model card on first run) - `${WORKSPACE}/results/` β€” training outputs (checkpoints, tensorboard logs) -## Day-0 Code +## Code checkout -Use the NeMo 26.04 container as the base image: `nvcr.io/nvidia/nemo:26.04`. +Use a NeMo container compatible with your checkout; see the +[development environment instructions](../../../../CONTRIBUTING.md). -The Day-0 code lives on the following public branches: +The original Day-0 code is retained on the following public branches. Use +`main` for the flat example paths documented below; use the README on the +Day-0 branch when working with that historical checkout: | Repo | Branch | Remote | |---|---|---| @@ -56,16 +59,15 @@ The Day-0 code lives on the following public branches: ```bash cd $WORKSPACE -git clone -b nemotron_3_omni https://github.com/NVIDIA-NeMo/Megatron-Bridge.git +git clone -b main https://github.com/NVIDIA-NeMo/Megatron-Bridge.git cd Megatron-Bridge git submodule update --init --recursive --depth 1 uv lock uv sync ``` -The `.gitmodules` already points the `3rdparty/Megatron-LM` submodule at -`https://github.com/NVIDIA/Megatron-LM.git`, and the recorded gitlink is -the tip of its `nemotron_3_omni` branch β€” `git submodule update --init +The `.gitmodules` points the `3rdparty/Megatron-LM` submodule at +`https://github.com/NVIDIA/Megatron-LM.git`. Use the recorded submodule commit β€” `git submodule update --init --recursive --depth 1` checks it out as a shallow clone automatically; no extra remote/fetch step needed. The `--depth 1` flag dramatically reduces clone time and disk usage (avoids fetching Megatron-LM's full history). @@ -73,14 +75,8 @@ clone time and disk usage (avoids fetching Megatron-LM's full history). `3rdparty/Megatron-LM/` submodule rather than any pre-installed copy from the container. `uv sync` then materializes the resulting environment. -> **`uv lock && uv sync` are mandatory before running any script in this -> repo.** The `uv.lock` in this repo pins `flashinfer-python==0.6.8.post1` -> to match the `flashinfer-cubin` pre-installed in the NeMo container. -> Skipping `uv sync` will leave a stale version installed, producing: -> ``` -> RuntimeError: flashinfer-cubin version (0.6.8.post1) does not match flashinfer version (X). -> ``` -> Re-run `uv sync` from `$WORKSPACE/Megatron-Bridge` to resolve it. +Run `uv lock` and `uv sync` before the examples so the environment uses +the dependencies required by this checkout. Verify that `megatron.core` and `megatron.bridge` resolve to the cloned checkout (and not a pre-installed copy from the container): @@ -138,13 +134,13 @@ export, and a multi-GPU HF↔Megatron round-trip verification. Run from `$WORKSPACE/Megatron-Bridge`: ```bash -bash examples/models/nemotron/nemotron_3_omni/conversion.sh +bash examples/models/nemotron/nemotron_3_nano_omni/conversion.sh ``` ## Inference [inference.sh](inference.sh) drives -`examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py` over the four +`examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py` over the four modality combinations exercised by the model: | # | Modality | GPUs | Parallelism | @@ -194,7 +190,7 @@ checkpoint. Run from `$WORKSPACE/Megatron-Bridge`: ```bash -bash examples/models/nemotron/nemotron_3_omni/inference.sh +bash examples/models/nemotron/nemotron_3_nano_omni/inference.sh ``` ## Training @@ -325,8 +321,8 @@ selective recompute. LoRA targets `linear_qkv`, `linear_proj`, `in_proj`, projections frozen. ```bash -sbatch examples/models/nemotron/nemotron_3_omni/slurm_sft_cord_v2.sh -sbatch examples/models/nemotron/nemotron_3_omni/slurm_peft_cord_v2.sh +sbatch examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_cord_v2.sh +sbatch examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_cord_v2.sh ``` ### Audio-Video-Text β€” VALOR32K-AVQA @@ -369,8 +365,8 @@ LR=1e-4 (adapters target the language model only; vision encoder, vision projection, sound encoder, and sound projection are frozen). ```bash -sbatch examples/models/nemotron/nemotron_3_omni/slurm_sft_valor32k_avqa.sh -sbatch examples/models/nemotron/nemotron_3_omni/slurm_peft_valor32k_avqa.sh +sbatch examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_valor32k_avqa.sh +sbatch examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_valor32k_avqa.sh ``` ### Expected Training Dynamics @@ -395,7 +391,7 @@ so substitute the matching `RUN_TAG` from your training job into ```bash uv run torchrun --nproc-per-node=8 \ - examples/models/nemotron/nemotron_3_omni/cord_v2_inference.py \ + examples/models/nemotron/nemotron_3_nano_omni/cord_v2_inference.py \ --hf_model_path "$HF_MODEL_ID" \ --megatron_model_path ${WORKSPACE}/results/nemotron_omni_cord_v2_sft_config_sft_/checkpoints \ --tp 4 --ep 2 \ @@ -403,7 +399,7 @@ uv run torchrun --nproc-per-node=8 \ --output ${WORKSPACE}/results/cord_v2_eval.json uv run torchrun --nproc-per-node=8 \ - examples/models/nemotron/nemotron_3_omni/valor32k_avqa_inference.py \ + examples/models/nemotron/nemotron_3_nano_omni/valor32k_avqa_inference.py \ --hf_model_path "$HF_MODEL_ID" \ --megatron_model_path ${WORKSPACE}/results/nemotron_omni_valor32k_sft_config_sft_/checkpoints \ --data_root ${WORKSPACE}/datasets/valor32k_avqa \ diff --git a/examples/models/nemotron/nemotron_3_omni/conversion.sh b/examples/models/nemotron/nemotron_3_nano_omni/conversion.sh similarity index 100% rename from examples/models/nemotron/nemotron_3_omni/conversion.sh rename to examples/models/nemotron/nemotron_3_nano_omni/conversion.sh diff --git a/examples/models/nemotron/nemotron_3_omni/cord_v2_inference.py b/examples/models/nemotron/nemotron_3_nano_omni/cord_v2_inference.py similarity index 99% rename from examples/models/nemotron/nemotron_3_omni/cord_v2_inference.py rename to examples/models/nemotron/nemotron_3_nano_omni/cord_v2_inference.py index 681337ea56..9171dd2cb9 100644 --- a/examples/models/nemotron/nemotron_3_omni/cord_v2_inference.py +++ b/examples/models/nemotron/nemotron_3_nano_omni/cord_v2_inference.py @@ -18,7 +18,7 @@ (matches `nemotron_omni_cord_v2_sft_config` with the updated dynamic-resolution training). Usage: - uv run torchrun --nproc-per-node=8 examples/models/nemotron/nemotron_3_omni/cord_v2_inference.py \ + uv run torchrun --nproc-per-node=8 examples/models/nemotron/nemotron_3_nano_omni/cord_v2_inference.py \ --hf_model_path /chcui/pretrained_models/Nemotron-3-Nano-Omni-30B-A3B-Reasoning \ --megatron_model_path /path/to/cord_v2/checkpoints \ --tp 4 --ep 2 \ diff --git a/examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py b/examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py similarity index 99% rename from examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py rename to examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py index 7077f34a13..b177167986 100644 --- a/examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py +++ b/examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py @@ -40,28 +40,28 @@ Examples: # Single image: - uv run python examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py \ + uv run python examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py \ --hf_model_path="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" \ --image_path="https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/resolve/main/images/table.png" \ --prompt="Describe this image." \ --max_new_tokens 300 # Multiple images: - uv run python examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py \ + uv run python examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py \ --hf_model_path="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" \ --image_path="https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/resolve/main/images/example1a.jpeg,https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/resolve/main/images/example1b.jpeg" \ --prompt="Describe the two images in detail." \ --max_new_tokens 300 # Video description: - uv run python examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py \ + uv run python examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py \ --hf_model_path="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" \ --video_path="https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/resolve/main/images/demo.mp4" \ --prompt="Describe what you see." \ --max_new_tokens 300 # Audio transcription: - uv run python examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py \ + uv run python examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py \ --hf_model_path="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" \ --audio_path="/path/to/audio.wav" \ --prompt="Transcribe the audio." \ diff --git a/examples/models/nemotron/nemotron_3_omni/inference.sh b/examples/models/nemotron/nemotron_3_nano_omni/inference.sh similarity index 97% rename from examples/models/nemotron/nemotron_3_omni/inference.sh rename to examples/models/nemotron/nemotron_3_nano_omni/inference.sh index cf998fac02..ebf537a627 100755 --- a/examples/models/nemotron/nemotron_3_omni/inference.sh +++ b/examples/models/nemotron/nemotron_3_nano_omni/inference.sh @@ -45,7 +45,7 @@ mkdir -p "$(dirname "$IMAGE_PATH")" "$(dirname "$VIDEO_PATH")" "$(dirname "$AUDI [[ -f "$VIDEO_PATH" ]] || curl -fL "${_HF_ASSETS}/demo.mp4" -o "$VIDEO_PATH" [[ -f "$AUDIO_PATH" ]] || ffmpeg -y -i "$VIDEO_PATH" -vn -acodec pcm_s16le -ar 16000 -ac 1 "$AUDIO_PATH" -GEN_SCRIPT=examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py +GEN_SCRIPT=examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py # 1) Image + Text (1 GPU) uv run python -m torch.distributed.run --nproc_per_node=1 ${GEN_SCRIPT} \ diff --git a/examples/models/nemotron/nemotron_3_omni/slurm_peft_cord_v2.sh b/examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_cord_v2.sh similarity index 100% rename from examples/models/nemotron/nemotron_3_omni/slurm_peft_cord_v2.sh rename to examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_cord_v2.sh diff --git a/examples/models/nemotron/nemotron_3_omni/slurm_peft_valor32k_avqa.sh b/examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_valor32k_avqa.sh similarity index 100% rename from examples/models/nemotron/nemotron_3_omni/slurm_peft_valor32k_avqa.sh rename to examples/models/nemotron/nemotron_3_nano_omni/slurm_peft_valor32k_avqa.sh diff --git a/examples/models/nemotron/nemotron_3_omni/slurm_sft_cord_v2.sh b/examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_cord_v2.sh similarity index 100% rename from examples/models/nemotron/nemotron_3_omni/slurm_sft_cord_v2.sh rename to examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_cord_v2.sh diff --git a/examples/models/nemotron/nemotron_3_omni/slurm_sft_valor32k_avqa.sh b/examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_valor32k_avqa.sh similarity index 100% rename from examples/models/nemotron/nemotron_3_omni/slurm_sft_valor32k_avqa.sh rename to examples/models/nemotron/nemotron_3_nano_omni/slurm_sft_valor32k_avqa.sh diff --git a/examples/models/nemotron/nemotron_3_omni/valor32k_avqa_inference.py b/examples/models/nemotron/nemotron_3_nano_omni/valor32k_avqa_inference.py similarity index 99% rename from examples/models/nemotron/nemotron_3_omni/valor32k_avqa_inference.py rename to examples/models/nemotron/nemotron_3_nano_omni/valor32k_avqa_inference.py index 00ef12ce10..830dfd0145 100644 --- a/examples/models/nemotron/nemotron_3_omni/valor32k_avqa_inference.py +++ b/examples/models/nemotron/nemotron_3_nano_omni/valor32k_avqa_inference.py @@ -20,7 +20,7 @@ exercises the trained `video_embedder`. Usage: - torchrun --nproc-per-node=8 examples/models/nemotron/nemotron_3_omni/valor32k_avqa_inference.py \ + torchrun --nproc-per-node=8 examples/models/nemotron/nemotron_3_nano_omni/valor32k_avqa_inference.py \ --hf_model_path /path/to/nemotron-3-nano-omni-ea1_v1.0 \ --megatron_model_path /path/to/checkpoint \ --data_root /path/to/valor32k_avqa \ @@ -93,7 +93,7 @@ def _build_vision_packed_seq_params(imgs_sizes: Optional[torch.Tensor]) -> Optio # --------------------------------------------------------------------------- -# Forward step (same as examples/models/nemotron/nemotron_3_omni/hf_to_megatron_generate_nemotron_omni.py) +# Forward step (same as examples/models/nemotron/nemotron_3_nano_omni/hf_to_megatron_generate_nemotron_omni.py) # --------------------------------------------------------------------------- diff --git a/examples/models/nemotron/nemotron_3/super/conversion.sh b/examples/models/nemotron/nemotron_3_super/conversion.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/super/conversion.sh rename to examples/models/nemotron/nemotron_3_super/conversion.sh diff --git a/examples/models/nemotron/nemotron_3/super/finetune_nemotron_3_super.py b/examples/models/nemotron/nemotron_3_super/finetune_nemotron_3_super.py similarity index 100% rename from examples/models/nemotron/nemotron_3/super/finetune_nemotron_3_super.py rename to examples/models/nemotron/nemotron_3_super/finetune_nemotron_3_super.py diff --git a/examples/models/nemotron/nemotron_3/super/pretrain_nemotron_3_super.py b/examples/models/nemotron/nemotron_3_super/pretrain_nemotron_3_super.py similarity index 100% rename from examples/models/nemotron/nemotron_3/super/pretrain_nemotron_3_super.py rename to examples/models/nemotron/nemotron_3_super/pretrain_nemotron_3_super.py diff --git a/examples/models/nemotron/nemotron_3/super/qat_nemotron_3_super.py b/examples/models/nemotron/nemotron_3_super/qat_nemotron_3_super.py similarity index 100% rename from examples/models/nemotron/nemotron_3/super/qat_nemotron_3_super.py rename to examples/models/nemotron/nemotron_3_super/qat_nemotron_3_super.py diff --git a/examples/models/nemotron/nemotron_3/super/slurm_peft.sh b/examples/models/nemotron/nemotron_3_super/slurm_peft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/super/slurm_peft.sh rename to examples/models/nemotron/nemotron_3_super/slurm_peft.sh diff --git a/examples/models/nemotron/nemotron_3/super/slurm_pretrain.sh b/examples/models/nemotron/nemotron_3_super/slurm_pretrain.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/super/slurm_pretrain.sh rename to examples/models/nemotron/nemotron_3_super/slurm_pretrain.sh diff --git a/examples/models/nemotron/nemotron_3/super/slurm_sft.sh b/examples/models/nemotron/nemotron_3_super/slurm_sft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/super/slurm_sft.sh rename to examples/models/nemotron/nemotron_3_super/slurm_sft.sh diff --git a/examples/models/nemotron/nemotron_3/ultra/README.md b/examples/models/nemotron/nemotron_3_ultra/README.md similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/README.md rename to examples/models/nemotron/nemotron_3_ultra/README.md diff --git a/examples/models/nemotron/nemotron_3/ultra/conversion.sh b/examples/models/nemotron/nemotron_3_ultra/conversion.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/conversion.sh rename to examples/models/nemotron/nemotron_3_ultra/conversion.sh diff --git a/examples/models/nemotron/nemotron_3/ultra/pack_data_job.sh b/examples/models/nemotron/nemotron_3_ultra/pack_data_job.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/pack_data_job.sh rename to examples/models/nemotron/nemotron_3_ultra/pack_data_job.sh diff --git a/examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh b/examples/models/nemotron/nemotron_3_ultra/slurm_inference.sh similarity index 98% rename from examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh rename to examples/models/nemotron/nemotron_3_ultra/slurm_inference.sh index be5ff72756..b07997a0f6 100755 --- a/examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh +++ b/examples/models/nemotron/nemotron_3_ultra/slurm_inference.sh @@ -19,7 +19,7 @@ # Usage: # 1. Set CONTAINER_IMAGE, CONTAINER_MOUNTS, and cache/token environment variables. # 2. Optionally set HF_MODEL_PATH to a local Hugging Face snapshot. -# 3. Submit with: sbatch examples/models/nemotron/nemotron_3/ultra/slurm_inference.sh +# 3. Submit with: sbatch examples/models/nemotron/nemotron_3_ultra/slurm_inference.sh # ============================================================================== #SBATCH --job-name=nemotron-ultra-inference diff --git a/examples/models/nemotron/nemotron_3/ultra/slurm_peft.sh b/examples/models/nemotron/nemotron_3_ultra/slurm_peft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/slurm_peft.sh rename to examples/models/nemotron/nemotron_3_ultra/slurm_peft.sh diff --git a/examples/models/nemotron/nemotron_3/ultra/slurm_pretrain.sh b/examples/models/nemotron/nemotron_3_ultra/slurm_pretrain.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/slurm_pretrain.sh rename to examples/models/nemotron/nemotron_3_ultra/slurm_pretrain.sh diff --git a/examples/models/nemotron/nemotron_3/ultra/slurm_sft.sh b/examples/models/nemotron/nemotron_3_ultra/slurm_sft.sh similarity index 100% rename from examples/models/nemotron/nemotron_3/ultra/slurm_sft.sh rename to examples/models/nemotron/nemotron_3_ultra/slurm_sft.sh diff --git a/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_override_example.yaml b/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_override_example.yaml deleted file mode 100644 index c500dcc09d..0000000000 --- a/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_override_example.yaml +++ /dev/null @@ -1,57 +0,0 @@ -# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Example override file for Nemotron Nano V2 VL - -model: - seq_length: 4096 - freeze_language_model: False - freeze_vision_model: False - freeze_vision_projection: False - -train: - train_iters: 2000 - global_batch_size: 32 - micro_batch_size: 1 - eval_iters: 0 - -optimizer: - lr: 2e-5 - min_lr: 2e-6 - -scheduler: - lr_warmup_iters: 5 - -checkpoint: - # Directory to save to. If null, no checkpoint will be saved. - save: ??? - save_interval: 200 - -dist: - use_megatron_fsdp: false - use_torch_fsdp2: false - -logger: - log_interval: 1 - -dataset: - seq_length: 4096 - source: - dataset_name: raven - -rng: - seed: 42 - -ddp: - grad_reduce_in_fp32: true diff --git a/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml b/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml deleted file mode 100644 index 15eb0f6c3c..0000000000 --- a/examples/models/nemotron/nemotron_vl/conf/nemotron_nano_v2_vl_video.yaml +++ /dev/null @@ -1,58 +0,0 @@ -# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Example override file for Nemotron Nano V2 VL - -model: - seq_length: 8192 - freeze_language_model: False - freeze_vision_model: False - freeze_vision_projection: False - tensor_model_parallel_size: 8 - -train: - train_iters: 300 - global_batch_size: 16 - micro_batch_size: 1 - eval_iters: 0 - -optimizer: - lr: 2e-6 - min_lr: 2e-7 - -scheduler: - lr_warmup_iters: 20 - -checkpoint: - save_interval: 50 - -dist: - use_megatron_fsdp: false - use_torch_fsdp2: false - -logger: - log_interval: 1 - -dataset: - seq_length: 8192 - source: - dataset_name: llava_video_178k - adapter_kwargs: - video_root_path: ??? - -rng: - seed: 42 - -ddp: - grad_reduce_in_fp32: true diff --git a/examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py b/examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py deleted file mode 100644 index cf07d100aa..0000000000 --- a/examples/models/nemotron/nemotron_vl/finetune_nemotron_nano_v2_vl.py +++ /dev/null @@ -1,203 +0,0 @@ -#!/usr/bin/env python3 -# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -""" -Finetune Nemotron Nano V2 VL with YAML and CLI Configuration Overrides. -""" - -from __future__ import annotations - -import argparse -import logging -import os -import sys -from pathlib import Path -from typing import Tuple - -import torch -from omegaconf import OmegaConf - -from megatron.bridge.recipes.nemotron_vl.nemotron_nano_v2_vl import ( - nemotron_nano_v2_vl_12b_peft_config, - nemotron_nano_v2_vl_12b_sft_config, -) -from megatron.bridge.training.config import ConfigContainer -from megatron.bridge.training.finetune import finetune -from megatron.bridge.training.llava_step import forward_step -from megatron.bridge.training.utils.omegaconf_utils import ( - apply_overrides, - create_omegaconf_dict_config, - parse_hydra_overrides, -) -from megatron.bridge.utils.common_utils import get_rank_safe - - -logger: logging.Logger = logging.getLogger(__name__) - -SCRIPT_DIR: Path = Path(__file__).parent.resolve() -DEFAULT_CONFIG_FILENAME: str = "nemotron_nano_v2_vl_override_example.yaml" -DEFAULT_CONFIG_FILE_PATH: Path = SCRIPT_DIR / "conf" / DEFAULT_CONFIG_FILENAME - -_ALL_COMPONENT_LORA_TARGET_MODULES = ["linear_qkv", "linear_proj", "linear_fc1", "linear_fc2"] -_LANGUAGE_LORA_TARGET_MODULES = [ - "*language_model*.linear_qkv", - "*language_model*.linear_proj", - "*language_model*.linear_fc1", - "*language_model*.linear_fc2", -] -_VISION_LORA_TARGET_MODULES = [ - "*vision_model*.linear_qkv", - "*vision_model*.linear_proj", - "*vision_model*.linear_fc1", - "*vision_model*.linear_fc2", - "*vision_projection*.linear_fc1", - "*vision_projection*.linear_fc2", -] - - -def parse_cli_args() -> Tuple[argparse.Namespace, list[str]]: - """Parse command-line flags and return `(argparse.Namespace, overrides)`.""" - - parser = argparse.ArgumentParser( - description="Finetune Nemotron Nano V2 VL with YAML and CLI overrides", - formatter_class=argparse.RawTextHelpFormatter, - ) - - # Same flags as pre-training script - parser.add_argument( - "--config-file", - type=str, - default=str(DEFAULT_CONFIG_FILE_PATH), - help="Path to the YAML OmegaConf override file. Default: conf/nemotron_nano_v2_vl_override_example.yaml", - ) - parser.add_argument( - "--hf-model-path", - type=str, - default="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16", - help="Path to the HuggingFace model to load weights from. Default: nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16", - ) - # Finetune-specific flags - parser.add_argument( - "--pretrained-checkpoint", - type=str, - required=True, - help="Path to a Megatron-Bridge checkpoint directory or HuggingFace model name to load weights from.", - ) - parser.add_argument( - "--lora-on-language-model", - action="store_true", - help="Use PEFT for finetuning on the language model.", - ) - parser.add_argument( - "--lora-on-vision-model", - action="store_true", - help="Use PEFT for finetuning on the vision model.", - ) - - parser.add_argument("--debug", action="store_true", help="Enable debug logging") - - args, cli_dotlist_overrides = parser.parse_known_args() - return args, cli_dotlist_overrides - - -def _apply_lora_component_selection( - cfg: ConfigContainer, - *, - lora_on_language_model: bool, - lora_on_vision_model: bool, -) -> None: - """Apply the example script's component-level LoRA target selection.""" - if not hasattr(cfg.peft, "target_modules"): - raise ValueError("The selected PEFT config does not expose target_modules.") - - if lora_on_language_model and lora_on_vision_model: - cfg.peft.target_modules = _ALL_COMPONENT_LORA_TARGET_MODULES.copy() - elif lora_on_language_model: - cfg.peft.target_modules = _LANGUAGE_LORA_TARGET_MODULES.copy() - if hasattr(cfg.peft, "freeze_vision_model"): - cfg.peft.freeze_vision_model = False - if hasattr(cfg.peft, "freeze_vision_projection"): - cfg.peft.freeze_vision_projection = False - elif lora_on_vision_model: - cfg.peft.target_modules = _VISION_LORA_TARGET_MODULES.copy() - if hasattr(cfg.peft, "freeze_language_model"): - cfg.peft.freeze_language_model = False - else: - raise ValueError("At least one of --lora-on-language-model or --lora-on-vision-model must be set.") - - -def main() -> None: - """Main function to finetune Nemotron Nano V2 VL.""" - args, cli_overrides = parse_cli_args() - - logger.info("Megatron-Bridge Nemotron Nano V2 VL Finetuning Script with YAML & CLI Overrides") - logger.info("--------------------------------------------------------------------------") - - logging.basicConfig( - level=logging.DEBUG if args.debug else logging.INFO, - format="%(asctime)s [%(levelname)s] %(message)s", - datefmt="%Y-%m-%d %H:%M:%S", - ) - - if args.lora_on_language_model or args.lora_on_vision_model: - cfg: ConfigContainer = nemotron_nano_v2_vl_12b_peft_config() - _apply_lora_component_selection( - cfg, - lora_on_language_model=args.lora_on_language_model, - lora_on_vision_model=args.lora_on_vision_model, - ) - logger.info("Loaded base configuration for PEFT") - else: - cfg = nemotron_nano_v2_vl_12b_sft_config() - logger.info("Loaded base configuration for SFT") - - cfg.dataset.hf_processor_path = args.hf_model_path - cfg.checkpoint.pretrained_checkpoint = args.pretrained_checkpoint - - if get_rank_safe() == 0: - cfg.print_yaml() - - # Convert to OmegaConf so we can merge overrides - merged_omega_conf, excluded_fields = create_omegaconf_dict_config(cfg) - - if args.config_file: - logger.debug(f"Loading YAML overrides from: {args.config_file}") - if not os.path.exists(args.config_file): - logger.error(f"Override YAML file not found: {args.config_file}") - sys.exit(1) - yaml_overrides_omega = OmegaConf.load(args.config_file) - merged_omega_conf = OmegaConf.merge(merged_omega_conf, yaml_overrides_omega) - - if cli_overrides: - logger.debug(f"Applying Hydra-style command-line overrides: {cli_overrides}") - merged_omega_conf = parse_hydra_overrides(merged_omega_conf, cli_overrides) - - final_overrides_as_dict = OmegaConf.to_container(merged_omega_conf, resolve=True) - apply_overrides(cfg, final_overrides_as_dict, excluded_fields) - - if get_rank_safe() == 0: - logger.info("--- Final Merged Configuration (Finetune) ---") - cfg.print_yaml() - logger.info("--------------------------------------------") - - # Nemotron Nano V2 VL uses the LLaVaModel class for the model, so we use the llava_step.forward_step function. - finetune(config=cfg, forward_step_func=forward_step) - - if torch.distributed.is_initialized(): - torch.distributed.destroy_process_group() - - -if __name__ == "__main__": - main() diff --git a/scripts/docs/generate_model_verification_catalog.py b/scripts/docs/generate_model_verification_catalog.py index 240873aaaf..b507d4391c 100644 --- a/scripts/docs/generate_model_verification_catalog.py +++ b/scripts/docs/generate_model_verification_catalog.py @@ -450,7 +450,14 @@ def build_catalog(repo_root: Path) -> dict[str, object]: if not cards: raise CatalogError(f"no cards matched {CARD_GLOB}") models = [normalize_card(card, repo_root) for card in cards] - models.sort(key=lambda model: (str(model["hf_id"]).casefold(), str(model["slug"]))) + # Ignore the repeated vendor prefix so Nano precedes Nano Omni, followed + # by Super, Ultra, Lightning, and Super VL in every generated model list. + models.sort( + key=lambda model: ( + str(model["hf_id"]).casefold().replace("nvidia/nvidia-nemotron-", "nvidia/nemotron-"), + str(model["slug"]), + ) + ) return { "schema_version": CATALOG_SCHEMA_VERSION, "source": CARD_GLOB, diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md index ee4dfcd337..d4e4c7cc23 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md @@ -9,28 +9,18 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nemo-mbridge-perf-vision-dp-over-cp` -- Evaluation date: 2026-08-27 -- Evaluator version: `1.3.2` +- Evaluation date: 2026-10-09 +- Evaluator version: `1.5.6` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) - Tasks: 3 evaluation tasks (2 positive, 1 negative) - Dataset digest: `sha256:1f686f2a6deb8b2b442751f0639aafb30c6e89513befad8532f4ef5754f59593` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 - Environment: `k8s-sandbox` +- Tier 2 evidence: required for publication - Tier 3 evidence: required for publication Each task attempt ran in its own isolated sandbox pod. -## Execution and Provenance - -- Validation status: `passed` -- Report generation: `complete` -- Evaluator version: `1.3.2` -- Git commit: `eda36f54e38357fe24c321a2ffae85589de9bd48` -- Content type: requested `auto`, detected `skill` -- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-eda36f54e38357fe24c321a2ffae85589de9bd48` -- Container image digest: `not recorded` -- Tier 3: requested `true`, executed `true`, status `succeeded` - ## What This Report Answers The three-tier evaluation checks whether the skill: @@ -45,23 +35,43 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline β†’ Skill Uplift) | Codex (Baseline β†’ Skill Uplift) | |---|---:|---:| -| Overall | 67% β†’ 100% (+32 points) | 59% β†’ 95% (+36 points) | -| Security | 100% β†’ 100% (Β±0 points) | 67% β†’ 100% (+33 points) | -| Correctness | 47% β†’ 100% (+53 points) | 60% β†’ 93% (+33 points) | -| Discoverability | 83% β†’ 100% (+17 points) | 58% β†’ 96% (+38 points) | -| Effectiveness | 34% β†’ 99% (+65 points) | 50% β†’ 88% (+38 points) | -| Efficiency | 73% β†’ 100% (+27 points) | 62% β†’ 100% (+38 points) | +| Overall | 98.3% β€” baseline ran, but no comparable score was available; uplift unavailable | 93.2% β€” baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% β†’ 100.0% (Β±0.0 points) | 100.0% β†’ 100.0% (Β±0.0 points) | +| Correctness | 33.3% β†’ 100.0% (+66.7 points) | 93.3% β†’ 93.3% (Β±0.0 points) | +| Discoverability | 100.0% β€” baseline ran, but no comparable score was available; uplift unavailable | 95.0% β€” baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 35.0% β†’ 93.1% (+58.1 points) | 70.8% β†’ 80.8% (+10.0 points) | +| Efficiency | 98.5% β€” baseline ran, but no comparable score was available; uplift unavailable | 97.0% β€” baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% β†’ 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +## Token Usage + +Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. -**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. +| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | +|---|---|---:|---:|---:|---:|---| +| claude-code | All cases | 162,821 | 1,009,164 | -846,343 | -83.87% | skill 3/3; base 3/3 | +| claude-code | vision-dp-over-cp-negative-hierarchical-cp | 29,190 | 156,718 | -127,528 | -81.37% | skill 1/1; base 1/1 | +| claude-code | vision-dp-over-cp-positive-enablement | 67,014 | 505,211 | -438,197 | -86.74% | skill 1/1; base 1/1 | +| claude-code | vision-dp-over-cp-positive-load-imbalance | 66,617 | 347,235 | -280,618 | -80.82% | skill 1/1; base 1/1 | +| codex | All cases | 159,842 | 169,449 | -9,607 | -5.67% | skill 3/3; base 3/3 | +| codex | vision-dp-over-cp-negative-hierarchical-cp | 99,310 | 74,593 | +24,717 | +33.14% | skill 1/1; base 1/1 | +| codex | vision-dp-over-cp-positive-enablement | 30,650 | 40,224 | -9,574 | -23.80% | skill 1/1; base 1/1 | +| codex | vision-dp-over-cp-positive-load-imbalance | 29,882 | 54,632 | -24,750 | -45.30% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 322,663 | 1,178,613 | -855,950 | -72.62% | skill 6/6; base 6/6 | -Example: `47% β†’ 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated. ## Tier Status | Tier | Purpose | Status | Evidence | |---|---|---|---| -| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) | -| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 15 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | | Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) | ## Findings and Observations @@ -69,10 +79,12 @@ Example: `47% β†’ 92% (+45 points)` means the skill-assisted run scored 92%, 45
Show detailed findings and successful checks +- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) - **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) - **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) - **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) -- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml`) +- 10 additional finding(s) are available in the full evaluation artifacts.
@@ -87,23 +99,24 @@ Example: `47% β†’ 92% (+45 points)` means the skill-assisted run scored 92%, 45 | Correctness | Is the answer correct? | `accuracy` (100%) | | Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | | Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | -| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | +| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) | - Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. - Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. - Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. - The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. - Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). -- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. +- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial. Signals present in this run: - `security` (Security): unsafe operations, secret leakage, and unauthorized access. -- `skill_execution` (Skill Execution): whether the expected skill was found and executed. -- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. +- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed. +- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability). - `accuracy` (Accuracy): final-answer correctness against the reference answer. - `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. - `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. +- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency). diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml index 242cd5b5c2..66ba88c827 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml @@ -83,7 +83,7 @@ evidence: - src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py - src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py - tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py - - examples/models/nemotron/nemotron_3_omni/README.md + - examples/models/nemotron/nemotron_3_nano_omni/README.md - 3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py - 3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py follow_up_validation: diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md index 848868ce1a..a2a8603d54 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md @@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers optimizing VLM training performance who need to reduce peak memory and improve throughput by sharding the vision encoder's image inputs across context-parallel ranks in Megatron-Bridge.
+Developers and engineers training vision-language models with Megatron-Bridge use this skill to enable, verify, and troubleshoot `vision_dp_over_cp`, which shards images or temporal tubelets across the language model's context-parallel ranks to reduce vision-encoder memory and step time when CP>1.
### Deployment Geography for Use:
Global
@@ -25,14 +25,16 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
+- [card.yaml (validation status and feature meanings)](card.yaml)
+- [nemo-mbridge-perf-moe-vlm-training skill](../nemo-mbridge-perf-moe-vlm-training/SKILL.md)
+- [nemo-mbridge-perf-memory-tuning skill](../nemo-mbridge-perf-memory-tuning/SKILL.md)
+- [nemo-mbridge-perf-hierarchical-context-parallel skill](../nemo-mbridge-perf-hierarchical-context-parallel/SKILL.md)
- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
-- [SKILL.md](SKILL.md)
-- [card.yaml](card.yaml)
## Skill Output:
-**Output Type(s):** [Configuration instructions, Analysis]
-**Output Format:** [Markdown with inline Python code blocks]
+**Output Type(s):** [Analysis, Configuration instructions, Shell commands]
+**Output Format:** [Markdown with inline Python and bash code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
@@ -43,43 +45,39 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-Evaluated against 3 evaluation tasks (2 positive, 1 negative) in isolated k8s-sandbox pods with 1 attempt per task.
+3 evaluation tasks (2 positive, 1 negative), 1 attempt per task, each run in an isolated sandbox pod and compared against a no-skill baseline.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access).
-- Correctness: Whether the skill produces correct answers against the reference.
-- Discoverability: Whether the right skill is loaded and activated when needed.
-- Effectiveness: Whether the skill helps complete the user's goal and expected workflow (goal completion + behavior adherence).
-- Efficiency: Whether the skill avoids wasted tool or skill usage (routing quality and productive tool use).
+- Security: Is it safe to use? Scored from the `security` signal.
+- Correctness: Is the answer correct? Scored from the `accuracy` signal.
+- Discoverability: Was the right skill loaded when needed? Scored from the `skill_execution` signal.
+- Effectiveness: Did the skill help complete the task? Equal-weight mean of `goal_accuracy` and `behavior_check`.
+- Efficiency: Did it avoid wasted tool calls and token usage? 50% `skill_efficiency` plus 50% `token_efficiency`.
Underlying evaluation signals used in this run:
- `security`: Unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Whether the expected skill was found and executed.
-- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
+- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency`: Tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy`: Final-answer correctness against the reference answer.
- `goal_accuracy`: Whether the user's goal was achieved.
- `behavior_check`: Whether the expected workflow behavior was followed.
+- `token_efficiency`: Actual uncached prompt plus completion usage (50% of Efficiency).
## Evaluation Results:
| Measure | Claude Code (Baseline β†’ Skill Uplift) | Codex (Baseline β†’ Skill Uplift) | |---|---:|---:| -| Overall | 67% β†’ 100% (+32 points) | 59% β†’ 95% (+36 points) | -| Security | 100% β†’ 100% (Β±0 points) | 67% β†’ 100% (+33 points) | -| Correctness | 47% β†’ 100% (+53 points) | 60% β†’ 93% (+33 points) | -| Discoverability | 83% β†’ 100% (+17 points) | 58% β†’ 96% (+38 points) | -| Effectiveness | 34% β†’ 99% (+65 points) | 50% β†’ 88% (+38 points) | -| Efficiency | 73% β†’ 100% (+27 points) | 62% β†’ 100% (+38 points) | - -## Testing Completed:
-**[x] Agent Red-Teaming**
-**[ ] Network Security**
-**[ ] Product Security**
+| Overall | 98.3% β€” baseline ran, but no comparable score was available; uplift unavailable | 93.2% β€” baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% β†’ 100.0% (Β±0.0 points) | 100.0% β†’ 100.0% (Β±0.0 points) | +| Correctness | 33.3% β†’ 100.0% (+66.7 points) | 93.3% β†’ 93.3% (Β±0.0 points) | +| Discoverability | 100.0% β€” baseline ran, but no comparable score was available; uplift unavailable | 95.0% β€” baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 35.0% β†’ 93.1% (+58.1 points) | 70.8% β†’ 80.8% (+10.0 points) | +| Efficiency | 98.5% β€” baseline ran, but no comparable score was available; uplift unavailable | 97.0% β€” baseline ran, but no comparable score was available; uplift unavailable | ## Skill Version(s):
-1.0.0+b7643bd (source: pyproject.toml)
+1.0.0+9edee0c (source: pyproject.toml)
## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig index e1b5805ea0..b576d64b24 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWRwLW92ZXItY3AiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiMTU2ZmQwMjI0MzgwNDU1MmI2NTAxMDU2MDU4Zjk1YmY2NDFlOWM2YzUzNDY1NDdhYmU2NDZhNjc2ZTdjYjg2ZSIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzA2NTg3ZmIxNzM3ZWU2ODA5NDJlNWQ1M2QwMDk0MzI4NzJlZTQ1ZmNkZDcwZDk3NjU2YTU3MmIzMmQ1M2QxZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJjNGFjZTEwMWVkZmM2OGJhNzc0Y2I3MWRlNDBiZTdlOTk2NDZhMjFiNzA4MDEwYjFhNjAyYjI4NmRjMTRmMzg2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3ZTIxMDQyZGRiNjM3ZmNmN2JiNjEwOTcxYWY2ZjEyZTI5OWRjMmU1YmNlY2M1ZjM2OTM5MzRiMGVjZDUwNTkyIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiOWEyNWJlYTM4M2Q4NjU0ZjMyY2E3Y2EzZjI1ODZmOWYyZTJlODdiNjQ4N2U1NzJmNDcyOTFlMzk3ZTMzNjkzYiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImM4OGNiZjEzYmJiYmVmNzE1MDU0OTExNmU2MmY1NjU5ZTU3OTY1ZjNlNTNmNDgwMTg1MWI3YjU5NWJmZjNhYzgiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDwd/MAPl2kWNx2d2eJ3XaNNCQl8x401GN0EFcYDECzBr5KQj3668JZgX5KHvtyCsECMCeibyjnUd+tRvezzdXiUZJEeuE07vbfe4PdF6RygUCNd4MBop2pR/s7Ny5fznkiKA==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWRwLW92ZXItY3AiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiZWM3Nzg4YmYwZmYzZTQ2MGI2MzY2ZGIwYzIwMTUxYTI2MTI0ODEyNmE5MzNmYzFiOGFkMmRhMDNlODA2MzIyNCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjIwYTQzZWJhOGI3NTcyYTgwZDY4ZDJjYzhjNzBlYTgzMDFmMTdhNDZjYWMyMTdjNmYxNTkzY2I4NTk0NTY0OTUiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImM0YWNlMTAxZWRmYzY4YmE3NzRjYjcxZGU0MGJlN2U5OTY0NmEyMWI3MDgwMTBiMWE2MDJiMjg2ZGMxNGYzODYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzJiYmMyZTI2NmU2N2U4ODY5YTUxMTJmMTE4MDdkY2RkNjdhZDI0MTZlNjJmYzQyOGNhNDVmODA3NWM1NzU3NCIsCiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiOWEyNWJlYTM4M2Q4NjU0ZjMyY2E3Y2EzZjI1ODZmOWYyZTJlODdiNjQ4N2U1NzJmNDcyOTFlMzk3ZTMzNjkzYiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImE5YWNjMjA3NDlkMTQzMjNmYjgxZWMyMDk2YWU0NjEwYzI5ZjRjYmYxZmI5MjA2MDc5NDAyMWU3MTM3YTRhMmMiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQC97XEEVHp6umiN8vp4FEzka9qaM+R1PsqxFanIdPoNSOV4hHhyEf5vrzRGSHVagr4CMQCaOMAq4Jzrvc+UTMqMqGE0qoEwjJpAXRvUlbdJo9iLi5XtbPyczK1OZLPqpCglI5Q=","keyid":""}]}} \ No newline at end of file diff --git a/tests/unit_tests/doc_consistency/test_nemotron_layout.py b/tests/unit_tests/doc_consistency/test_nemotron_layout.py new file mode 100644 index 0000000000..4f5a806976 --- /dev/null +++ b/tests/unit_tests/doc_consistency/test_nemotron_layout.py @@ -0,0 +1,87 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Check example discovery and links without importing GPU dependencies.""" + +import importlib +import re +import subprocess +from pathlib import Path +from urllib.parse import unquote, urlsplit + +import pytest + + +pytestmark = pytest.mark.unit +REPO_ROOT = Path(__file__).resolve().parents[3] +EXAMPLE_ROOT = REPO_ROOT / "examples/models/nemotron" +MODEL_DIRECTORIES = ( + "nemotron_3_nano", + "nemotron_3_nano_omni", + "nemotron_3_super", + "nemotron_3_ultra", + "nemotron_3_5_lightning", + "nemotron_3_5_super_vl", +) + + +def test_model_directories_are_flat() -> None: + directories = {path.name for path in EXAMPLE_ROOT.iterdir() if path.is_dir() and path.name != "__pycache__"} + assert directories == set(MODEL_DIRECTORIES) + + +@pytest.mark.parametrize("name", MODEL_DIRECTORIES) +def test_model_example_namespace_is_importable(name: str, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.syspath_prepend(str(REPO_ROOT)) + package = importlib.import_module(f"examples.models.nemotron.{name}") + assert (EXAMPLE_ROOT / name).resolve() in {Path(path).resolve() for path in package.__path__} + + +@pytest.mark.parametrize("readme", sorted(EXAMPLE_ROOT.rglob("*.md"))) +def test_example_relative_links(readme: Path) -> None: + for target in re.findall(r"\]\(([^)]+)\)", readme.read_text()): + url = urlsplit(target) + if url.scheme or not url.path: + continue + destination = (readme.parent / unquote(url.path)).resolve() + assert destination.exists(), f"{readme.relative_to(REPO_ROOT)}: {target}" + if url.fragment and destination.suffix == ".md": + headings = re.findall(r"^#+\s+(.+)$", destination.read_text(), re.MULTILINE) + anchors = {re.sub(r"[^\w\- ]", "", heading.lower()).replace(" ", "-") for heading in headings} + assert url.fragment in anchors, f"{destination}: #{url.fragment}" + + +def test_current_nemotron_example_references_resolve() -> None: + tracked = subprocess.check_output(["git", "ls-files", "-z"], cwd=REPO_ROOT).decode().split("\0") + references = 0 + for name in tracked: + path = REPO_ROOT / name + # Versioned commands describe their release checkout. Links to main + # within those pages must still resolve against the current tree. + if not path.is_file(): + continue + content = path.read_bytes() + if b"\0" in content: + continue + text = content.decode("utf-8", errors="replace") + if name.startswith("docs/fern/versions/") and not name.startswith("docs/fern/versions/nightly/"): + text = "\n".join( + re.findall(r"https://github.com/NVIDIA-NeMo/Megatron-Bridge/(?:blob|tree)/main/[^\s)]+", text) + ) + text = re.sub(r"https://github.com/NVIDIA-NeMo/Megatron-Bridge/(?:blob|tree)/(?!main/)[^\s)]+", "", text) + for target in re.findall(r"examples/models/nemotron/[\w./-]+", text): + target = target.rstrip(".") + references += 1 + assert (REPO_ROOT / target).exists(), f"{name}: {target}" + assert references > 20 diff --git a/tests/unit_tests/doc_consistency/test_readme_consistency.py b/tests/unit_tests/doc_consistency/test_readme_consistency.py index 2ce1b41995..2e9450bfca 100644 --- a/tests/unit_tests/doc_consistency/test_readme_consistency.py +++ b/tests/unit_tests/doc_consistency/test_readme_consistency.py @@ -427,11 +427,11 @@ def test_model_examples_use_current_run_recipe_arguments(): def test_nemotron_and_qwen2_audio_finetune_launchers_use_exported_recipes(): """SFT and PEFT example recipes must match their launcher-visible aliases.""" launcher_expectations = { - MODEL_EXAMPLES / "nemotron" / "nemotron_3" / "nano" / "slurm_sft.sh": ( + MODEL_EXAMPLES / "nemotron" / "nemotron_3_nano" / "slurm_sft.sh": ( "${MODEL_NAME}_sft_config", "nemotron_3_nano_finetune_config", ), - MODEL_EXAMPLES / "nemotron" / "nemotron_3" / "nano" / "slurm_peft.sh": ( + MODEL_EXAMPLES / "nemotron" / "nemotron_3_nano" / "slurm_peft.sh": ( "${MODEL_NAME}_peft_config", "nemotron_3_nano_finetune_config", ), diff --git a/tests/unit_tests/examples/test_nemotron_3_ultra_pack_data.py b/tests/unit_tests/examples/test_nemotron_3_ultra_pack_data.py index 2a79fc1910..7347fd5ac1 100644 --- a/tests/unit_tests/examples/test_nemotron_3_ultra_pack_data.py +++ b/tests/unit_tests/examples/test_nemotron_3_ultra_pack_data.py @@ -22,7 +22,7 @@ pytestmark = pytest.mark.unit REPO_ROOT = Path(__file__).resolve().parents[3] -PACK_DATA_SCRIPT = REPO_ROOT / "examples/models/nemotron/nemotron_3/ultra/pack_data_job.sh" +PACK_DATA_SCRIPT = REPO_ROOT / "examples/models/nemotron/nemotron_3_ultra/pack_data_job.sh" def test_nemotron_3_ultra_pack_data_uses_recipe_defaults(tmp_path: Path) -> None: diff --git a/tests/unit_tests/examples/test_nemotron_omni_inference.py b/tests/unit_tests/examples/test_nemotron_omni_inference.py index d4d735763c..1639a25179 100644 --- a/tests/unit_tests/examples/test_nemotron_omni_inference.py +++ b/tests/unit_tests/examples/test_nemotron_omni_inference.py @@ -22,7 +22,7 @@ import megatron.bridge.models.nemotron_vl.nemotron_vl_utils as vl_utils -_EXAMPLE_ROOT = Path(__file__).parents[3] / "examples" / "models" / "nemotron" / "nemotron_3_omni" +_EXAMPLE_ROOT = Path(__file__).parents[3] / "examples" / "models" / "nemotron" / "nemotron_3_nano_omni" class _VideoImageProcessor: