Skip to content

Commit 4fdfa75

Browse files
tomlifutomlifu
authored
Merge branch 'main' into lifuz/qwen35-35b-8gpu-gb300-mxfp8-recipe
2 parents d9d98b9 + 7feb75e commit 4fdfa75

166 files changed

Lines changed: 672 additions & 11589 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/model_support_request.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ body:
2727
description: Pick the closest fit for the requested model.
2828
options:
2929
- Dense decoder-only (e.g. Llama, Qwen, Gemma, Mistral, Nemotron)
30-
- MoE decoder-only (e.g. DeepSeek V2/V3, OLMoE, Qwen3-MoE, MiniMax-M2)
30+
- MoE decoder-only (e.g. DeepSeek V3, OLMoE, Qwen3-MoE, MiniMax-M2)
3131
- VLM / multimodal (e.g. Qwen-VL, Gemma-VL, Nemotron-VL)
32-
- SSM / hybrid (e.g. Mamba, Nemotron-H)
32+
- SSM / hybrid (e.g. Mamba, Nemotron 3)
3333
- Encoder-decoder (e.g. T5)
3434
- Other / unsure
3535
validations:

‎.main.commit‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
fde43b6d617752433310b4668c6b5c55e503521a
1+
e0c5e18673bf4d4fd89cf87c4f1afbe521fe297f

‎3rdparty/Megatron-LM‎

Submodule Megatron-LM updated 206 files

‎README.md‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -221,34 +221,28 @@ For a deeper dive into conversion design and advanced usage, see the [models REA
221221

222222
Megatron Bridge provides out-of-the-box bridges and training recipes for a wide range of models, built on top of base model architectures from [Megatron Core](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core). Refer to the [models directory](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models) for the full list of model bridges.
223223

224-
> **Deprecation notice:** Gemma 1 (2B/7B), Gemma 2 (2B/9B/27B), Llama 2,
225-
> Mistral 7B, Mistral Small 3 24B, Nemotron H v1 (4B/8B/47B/56B), and the
226-
> legacy `NemotronForCausalLM` bridge previously documented for Nemotron-4 340B
227-
> are no longer actively maintained or tested against current upstream
228-
> checkpoints. They will be removed in Megatron Bridge 0.7.0.
229-
230224
| Family | Supported variants |
231225
|----------------|--------------------|
232226
| [**BAGEL**](docs/models/bagel/bagel.md) | BAGEL-7B-MoT (requires the tested MCore `dev` revision and `bagel` extra) |
233227
| [**Bailing**](docs/models/bailing/index.md) | Ling 2.0 / Ling MoE V2 (Bailing) |
234228
| [**BERT**](docs/models/bert/index.md) | Megatron-Style BERT (`MegatronBertForMaskedLM`) |
235-
| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V2 / V2 Lite (deprecated), DeepSeek V3, DeepSeek V4 / V4 Flash |
229+
| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V3, DeepSeek V4 / V4 Flash |
236230
| [**Diffusion**](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/diffusion/models) | FLUX, LLaDA 1.5, Nemotron-Labs Diffusion, WAN |
237231
| **Ernie** | [Ernie 4.5 MoE](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/ernie), [Ernie 4.5 VL MoE](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/ernie_vl) |
238232
| [**EXAONE**](docs/models/exaone/exaone.md) | EXAONE 4.0, EXAONE 4.5 VL, K-EXAONE MoE |
239233
| [**Falcon**](docs/models/falcon/index.md) | Falcon H1 |
240-
| [**Gemma**](docs/models/gemma/index.md) | Gemma / Gemma 2 (deprecated), Gemma 3, Gemma 3-VL, Gemma 4 (26B-A4B MoE / 31B dense), Gemma 4-VL (26B-A4B MoE) |
234+
| [**Gemma**](docs/models/gemma/index.md) | Gemma 3, Gemma 3-VL, Gemma 4 (26B-A4B MoE / 31B dense), Gemma 4-VL (26B-A4B MoE) |
241235
| [**GLM**](docs/models/glm/index.md) | GLM-4.5 / GLM-4.7 / GLM-4.7-Flash, GLM-4.5V, GLM-5 / GLM-5.1 / GLM-5.2 / [GLM-5.3](docs/models/glm/glm5-2.md) |
242236
| [**GPT-OSS**](docs/models/gpt_oss/index.md) | GPT-oss |
243237
| [**HY V3**](https://huggingface.co/tencent/Hy3-preview-Base) | Hy3 preview-Base (HF → Megatron checkpoint conversion) |
244238
| [**Kimi**](docs/models/kimi/index.md) | Kimi K2, Kimi-K2.5-VL, Kimi K3 |
245-
| [**Llama**](docs/models/llama/index.md) | Llama 2 (deprecated), Llama 3 / 3.1 / 3.2 / 3.3 |
239+
| [**Llama**](docs/models/llama/index.md) | Llama 3 / 3.1 / 3.2 / 3.3 |
246240
| [**MiniMax**](docs/models/minimax/index.md) | MiniMax-M2 / M2.5 / M2.7, MiniMax-M3 |
247-
| [**Mistral**](docs/models/mistral/index.md) | Mistral 7B / Small 3 24B (deprecated), Ministral 3 (3B/8B/14B) |
241+
| [**Mistral**](docs/models/mistral/index.md) | Ministral 3 (3B/8B/14B) |
248242
| [**Xiaomi-MiMo**](docs/models/mimo/index.md) | Xiaomi-MiMo, MiMo-V2-Flash |
249243
| [**Moonlight**](docs/models/moonlight/index.md) | Moonlight |
250244
| [**Muse Glimmer**](docs/models/muse_glimmer/muse-glimmer.md) | Muse Glimmer 30B |
251-
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron H v1 (deprecated), Nemotron Nano v2 (deprecated), Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Llama Nemotron (deprecated), Nemotron Nano v2 VL (deprecated), Nemotron-3 Nano Omni |
245+
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Nemotron-3 Nano Omni |
252246
| [**OLMoE**](docs/models/olmoe/index.md) | OLMoE |
253247
| [**Qwen**](docs/models/qwen/index.md) | Qwen2 / Qwen2.5, Qwen3, Qwen3-MoE, Qwen3 Next, Qwen3.5 (dense/MoE), Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, Qwen3.6-VL, Qwen2 Audio, Qwen2.5-Omni, Qwen3-Omni, Qwen3-ASR |
254248
| [**Sarvam**](docs/models/sarvam/index.md) | Sarvam |

‎docs/_static/model-verification/catalog-v1.json‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -691,14 +691,47 @@
691691
"status": "verified",
692692
"variant": null,
693693
"workflow": "pretrain_performance"
694+
},
695+
{
696+
"bridge_commit": "797a3141d3ccc4ec20a45cc653a4b69f9af0cdbf",
697+
"command_topologies": [
698+
{
699+
"gpus_per_node": "4",
700+
"nodes": "64",
701+
"recipe": "deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config",
702+
"sequence_length": "4096",
703+
"total_gpus": 256
704+
}
705+
],
706+
"commands": [
707+
"./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yaml"
708+
],
709+
"dimensions": {},
710+
"enabled_features": {},
711+
"entry_id": "deepseek-v3-pretrain-performance-hsdp-gb300",
712+
"expected_result": "On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1. Dense parameters use ZeRO-2 optim_grads sharding while expert parameters use optim_grads_params across four distributed optimizer instances, with full-iteration CUDA graphs and MoE paged stashing enabled. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and 1,332.420 model TFLOP/s/GPU. Do not compare this item against pretrain_performance.GB300: that recipe runs a different convergence contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU throughput are not commensurable. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.\n",
713+
"hardware": "GB300",
714+
"last_verified": "2026-10-09",
715+
"metrics": {
716+
"final_loss": 5.146079,
717+
"initial_loss": 11.89824,
718+
"last_10_steps_model_tflops_per_gpu_avg": 1332.42,
719+
"last_10_steps_step_time_ms_avg": 3197.04,
720+
"last_10_steps_tokens_per_second_per_gpu_avg": 5124.74
721+
},
722+
"precision": "fp8_mx",
723+
"source_pointer": "items.pretrain_performance_hsdp.GB300",
724+
"status": "verified",
725+
"variant": null,
726+
"workflow": "pretrain_performance_hsdp"
694727
}
695728
],
696729
"hf_id": "deepseek-ai/DeepSeek-V3",
697730
"hf_revision": "e815299b0bcbac849fa540c768ef21845365c9eb",
698731
"min_transformers_version": "5.8.0",
699732
"slug": "deepseek-v3",
700733
"source_card": "examples/model_verification_cards/deepseek-v3/card.yaml",
701-
"summary": "Distributed H100 GPU import, BF16 export, and deterministic inference are verified against the pinned blockwise-FP8 checkpoint. The canonical pretrain_performance.H100 1024-GPU BF16, pretrain_performance.GB200 256-GPU MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified. A distributed CPU export of the same pinned checkpoint is also verified. Timing and throughput metrics from future functional training items remain sanity checks rather than optimized performance results. CPU import, forward correlation, functional training, fine-tuning, and checkpoint-resume verification remain deferred.\n",
734+
"summary": "Distributed H100 GPU import, BF16 export, and deterministic inference are verified against the pinned blockwise-FP8 checkpoint. The canonical pretrain_performance.H100 1024-GPU BF16, pretrain_performance.GB200 256-GPU MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified, as is the pretrain_performance_hsdp.GB300 256-GPU MXFP8 Hybrid FSDP recipe. A distributed CPU export of the same pinned checkpoint is also verified. Timing and throughput metrics from future functional training items remain sanity checks rather than optimized performance results. CPU import, forward correlation, functional training, fine-tuning, and checkpoint-resume verification remain deferred.\n",
702735
"title": "deepseek_v3"
703736
},
704737
{

‎docs/bridge-guide.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ from megatron.bridge import AutoBridge
6363

6464
# Load with specific settings
6565
bridge = AutoBridge.from_hf_pretrained(
66-
"meta-llama/Llama-2-7b-hf",
66+
"meta-llama/Meta-Llama-3-8B",
6767
torch_dtype=torch.float16,
6868
device_map="auto",
6969
trust_remote_code=True,

‎docs/fern/versions/_nav_order.yml‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,20 +8,15 @@
88
"./versions/nightly/pages/modelopt/quantization.mdx": 1
99
"./versions/nightly/pages/models/README.mdx": 1
1010
"./versions/nightly/pages/models/bailing/ling-2.mdx": 1
11-
"./versions/nightly/pages/models/deepseek/deepseek-v2.mdx": 1
1211
"./versions/nightly/pages/models/falcon/falcon-h1.mdx": 1
13-
"./versions/nightly/pages/models/gemma/gemma.mdx": 1
1412
"./versions/nightly/pages/models/glm/glm45.mdx": 1
1513
"./versions/nightly/pages/models/gpt_oss/gpt-oss-20b.mdx": 1
1614
"./versions/nightly/pages/models/gpt_oss/gpt-oss-120b.mdx": 2
1715
"./versions/nightly/pages/models/kimi/kimi-k2.mdx": 1
18-
"./versions/nightly/pages/models/llama/llama2.mdx": 1
1916
"./versions/nightly/pages/models/mimo/mimo.mdx": 1
2017
"./versions/nightly/pages/models/minimax/minimax-m2.mdx": 1
2118
"./versions/nightly/pages/models/minimax/minimax-m3.mdx": 2
22-
"./versions/nightly/pages/models/mistral/mistral.mdx": 1
2319
"./versions/nightly/pages/models/moonlight/moonlight.mdx": 1
24-
"./versions/nightly/pages/models/nemotron/llama-nemotron.mdx": 1
2520
"./versions/nightly/pages/models/olmoe/olmoe.mdx": 1
2621
"./versions/nightly/pages/models/qwen/qwen3-235b-a22b.mdx": 1
2722
"./versions/nightly/pages/models/qwen/qwen3-30b-a3b.mdx": 2
@@ -37,12 +32,10 @@
3732
"./versions/nightly/pages/modelopt/distillation.mdx": 2
3833
"./versions/nightly/pages/models/bailing/index.mdx": 2
3934
"./versions/nightly/pages/models/deepseek/deepseek-v3.mdx": 2
40-
"./versions/nightly/pages/models/gemma/gemma2.mdx": 2
4135
"./versions/nightly/pages/models/glm/glm-45v.mdx": 2
4236
"./versions/nightly/pages/models/kimi/kimi-k25-vl.mdx": 2
4337
"./versions/nightly/pages/models/llama/llama3.mdx": 2
4438
"./versions/nightly/pages/models/mistral/ministral3.mdx": 2
45-
"./versions/nightly/pages/models/nemotron/nemotronh.mdx": 2
4639
"./versions/nightly/pages/models/qwen/qwen3-moe.mdx": 4
4740
"./versions/nightly/pages/performance-guide.mdx": 2
4841
"./versions/nightly/pages/performance-summary-archive.mdx": 2
@@ -75,8 +68,7 @@
7568
"./versions/nightly/pages/megatron-lm-to-megatron-bridge.mdx": 5
7669
"./versions/nightly/pages/models/gemma/gemma4-vl.mdx": 5
7770
"./versions/nightly/pages/models/gemma/index.mdx": 5
78-
"./versions/nightly/pages/models/nemotron/nemotron3.5-lightning.mdx": 8
79-
"./versions/nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx": 11
71+
"./versions/nightly/pages/models/nemotron/nemotron3.5-lightning.mdx": 6
8072
"./versions/nightly/pages/models/qwen/qwen3-vl.mdx": 7
8173
"./versions/nightly/pages/releases/known-issues.mdx": 5
8274
"./versions/nightly/pages/training/optimizer-scheduler.mdx": 5

‎docs/fern/versions/nightly.yml‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,6 @@ navigation:
4646
path: ./nightly/pages/models/bailing/ling-2.mdx
4747
section: Bailing
4848
- contents:
49-
- page: DeepSeek V2
50-
path: ./nightly/pages/models/deepseek/deepseek-v2.mdx
5149
- page: DeepSeek V3
5250
path: ./nightly/pages/models/deepseek/deepseek-v3.mdx
5351
- page: DeepSeek V4
@@ -58,10 +56,6 @@ navigation:
5856
path: ./nightly/pages/models/falcon/falcon-h1.mdx
5957
section: Falcon
6058
- contents:
61-
- page: Gemma
62-
path: ./nightly/pages/models/gemma/gemma.mdx
63-
- page: Gemma 2
64-
path: ./nightly/pages/models/gemma/gemma2.mdx
6559
- page: Gemma 3
6660
path: ./nightly/pages/models/gemma/gemma3.mdx
6761
- page: Gemma 3 VL (Vision-Language)
@@ -96,8 +90,6 @@ navigation:
9690
path: ./nightly/pages/models/kimi/kimi-k3.mdx
9791
section: Kimi
9892
- contents:
99-
- page: Llama 2
100-
path: ./nightly/pages/models/llama/llama2.mdx
10193
- page: Llama 3
10294
path: ./nightly/pages/models/llama/llama3.mdx
10395
section: Llama
@@ -108,8 +100,6 @@ navigation:
108100
path: ./nightly/pages/models/minimax/minimax-m3.mdx
109101
section: MiniMax
110102
- contents:
111-
- page: Mistral
112-
path: ./nightly/pages/models/mistral/mistral.mdx
113103
- page: Ministral 3
114104
path: ./nightly/pages/models/mistral/ministral3.mdx
115105
section: Mistral
@@ -122,10 +112,6 @@ navigation:
122112
path: ./nightly/pages/models/moonlight/moonlight.mdx
123113
section: Moonlight
124114
- contents:
125-
- page: Llama Nemotron
126-
path: ./nightly/pages/models/nemotron/llama-nemotron.mdx
127-
- page: Nemotron H and Nemotron Nano v2
128-
path: ./nightly/pages/models/nemotron/nemotronh.mdx
129115
- page: Nemotron 3 Nano 30B-A3B
130116
path: ./nightly/pages/models/nemotron/nemotron3-nano.mdx
131117
- page: Nemotron 3 Nano 4B
@@ -139,8 +125,6 @@ navigation:
139125
- page: Nemotron 3.5 Lightning
140126
path: ./nightly/pages/models/nemotron/nemotron3.5-lightning.mdx
141127
# Super VL guides remain in GitHub, but are excluded from publication.
142-
- page: Nemotron Nano V2 VL
143-
path: ./nightly/pages/models/nemotron/nemotron-nano-v2-vl.mdx
144128
section: Nemotron
145129
- contents:
146130
- page: OLMoE

‎docs/fern/versions/nightly/pages/bridge-guide.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ from megatron.bridge import AutoBridge
6363

6464
# Load with specific settings
6565
bridge = AutoBridge.from_hf_pretrained(
66-
"meta-llama/Llama-2-7b-hf",
66+
"meta-llama/Meta-Llama-3-8B",
6767
torch_dtype=torch.float16,
6868
device_map="auto",
6969
trust_remote_code=True,

‎docs/fern/versions/nightly/pages/index.mdx‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -206,18 +206,18 @@ Megatron Bridge provides out-of-the-box bridges and training recipes for a wide
206206
| Family | Supported variants |
207207
|----------------|--------------------|
208208
| [**Bailing**](docs/models/bailing/index.md) | Ling 2.0 (Bailing) |
209-
| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V2 / V2 Lite, DeepSeek V3, DeepSeek V4 |
209+
| [**DeepSeek**](docs/models/deepseek/index.md) | DeepSeek V3, DeepSeek V4 |
210210
| [**Falcon**](docs/models/falcon/index.md) | Falcon H1 |
211-
| [**Gemma**](docs/models/gemma/index.md) | Gemma / Gemma 2, Gemma 3, Gemma 3-VL, Gemma 4-VL (26B-A4B MoE) |
211+
| [**Gemma**](docs/models/gemma/index.md) | Gemma 3, Gemma 3-VL, Gemma 4-VL (26B-A4B MoE) |
212212
| [**GLM**](docs/models/glm/index.md) | GLM-4.5 / 4.7 / 4.7-Flash, GLM-4.5V, GLM-5 / 5.1 |
213213
| [**GPT-OSS**](docs/models/gpt_oss/index.md) | GPT-oss |
214214
| [**Kimi**](docs/models/kimi/index.md) | Kimi K2, Kimi-K2.5-VL |
215-
| [**Llama**](docs/models/llama/index.md) | Llama 2, Llama 3 / 3.1 / 3.2 / 3.3 |
215+
| [**Llama**](docs/models/llama/index.md) | Llama 3 / 3.1 / 3.2 / 3.3 |
216216
| [**MiniMax**](docs/models/minimax/index.md) | MiniMax-M2 / M2.5 / M2.7 |
217-
| [**Mistral**](docs/models/mistral/index.md) | Mistral, Ministral 3 (3B/8B/14B) |
217+
| [**Mistral**](docs/models/mistral/index.md) | Ministral 3 (3B/8B/14B) |
218218
| [**Xiaomi-MiMo**](docs/models/mimo/index.md) | Xiaomi-MiMo |
219219
| [**Moonlight**](docs/models/moonlight/index.md) | Moonlight |
220-
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron H, Nemotron Nano v2, Nemotron-3 Nano, Nemotron-3 Super, Llama Nemotron, Nemotron Nano v2 VL, Nemotron-3 Nano Omni |
220+
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Nano Omni |
221221
| [**OLMoE**](docs/models/olmoe/index.md) | OLMoE |
222222
| [**Qwen**](docs/models/qwen/index.md) | Qwen2 / Qwen2.5, Qwen3, Qwen3-MoE, Qwen3 Next, Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, Qwen3.6-VL, Qwen2 Audio, Qwen2.5-Omni, Qwen3-Omni, Qwen3-ASR |
223223
| [**Sarvam**](docs/models/sarvam/index.md) | Sarvam |

0 commit comments

Comments
 (0)