Skip to content

Commit 7feb75e

Browse files
gautham-kollugautham-kollu
authored
Non-uniform HSDP Update (#6382)
Signed-off-by: Gautham Kollu <gkollu@nvidia.com>
1 parent 618c6c8 commit 7feb75e

15 files changed

Lines changed: 293 additions & 21 deletions

File tree

‎docs/_static/model-verification/catalog-v1.json‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -691,14 +691,47 @@
691691
"status": "verified",
692692
"variant": null,
693693
"workflow": "pretrain_performance"
694+
},
695+
{
696+
"bridge_commit": "797a3141d3ccc4ec20a45cc653a4b69f9af0cdbf",
697+
"command_topologies": [
698+
{
699+
"gpus_per_node": "4",
700+
"nodes": "64",
701+
"recipe": "deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config",
702+
"sequence_length": "4096",
703+
"total_gpus": 256
704+
}
705+
],
706+
"commands": [
707+
"./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yaml"
708+
],
709+
"dimensions": {},
710+
"enabled_features": {},
711+
"entry_id": "deepseek-v3-pretrain-performance-hsdp-gb300",
712+
"expected_result": "On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1. Dense parameters use ZeRO-2 optim_grads sharding while expert parameters use optim_grads_params across four distributed optimizer instances, with full-iteration CUDA graphs and MoE paged stashing enabled. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and 1,332.420 model TFLOP/s/GPU. Do not compare this item against pretrain_performance.GB300: that recipe runs a different convergence contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU throughput are not commensurable. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.\n",
713+
"hardware": "GB300",
714+
"last_verified": "2026-10-09",
715+
"metrics": {
716+
"final_loss": 5.146079,
717+
"initial_loss": 11.89824,
718+
"last_10_steps_model_tflops_per_gpu_avg": 1332.42,
719+
"last_10_steps_step_time_ms_avg": 3197.04,
720+
"last_10_steps_tokens_per_second_per_gpu_avg": 5124.74
721+
},
722+
"precision": "fp8_mx",
723+
"source_pointer": "items.pretrain_performance_hsdp.GB300",
724+
"status": "verified",
725+
"variant": null,
726+
"workflow": "pretrain_performance_hsdp"
694727
}
695728
],
696729
"hf_id": "deepseek-ai/DeepSeek-V3",
697730
"hf_revision": "e815299b0bcbac849fa540c768ef21845365c9eb",
698731
"min_transformers_version": "5.8.0",
699732
"slug": "deepseek-v3",
700733
"source_card": "examples/model_verification_cards/deepseek-v3/card.yaml",
701-
"summary": "Distributed H100 GPU import, BF16 export, and deterministic inference are verified against the pinned blockwise-FP8 checkpoint. The canonical pretrain_performance.H100 1024-GPU BF16, pretrain_performance.GB200 256-GPU MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified. A distributed CPU export of the same pinned checkpoint is also verified. Timing and throughput metrics from future functional training items remain sanity checks rather than optimized performance results. CPU import, forward correlation, functional training, fine-tuning, and checkpoint-resume verification remain deferred.\n",
734+
"summary": "Distributed H100 GPU import, BF16 export, and deterministic inference are verified against the pinned blockwise-FP8 checkpoint. The canonical pretrain_performance.H100 1024-GPU BF16, pretrain_performance.GB200 256-GPU MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified, as is the pretrain_performance_hsdp.GB300 256-GPU MXFP8 Hybrid FSDP recipe. A distributed CPU export of the same pinned checkpoint is also verified. Timing and throughput metrics from future functional training items remain sanity checks rather than optimized performance results. CPU import, forward correlation, functional training, fine-tuning, and checkpoint-resume verification remain deferred.\n",
702735
"title": "deepseek_v3"
703736
},
704737
{

‎docs/fern/versions/nightly/pages/models/README.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
Choose a model to open its interactive import/export and training configurations. Every option comes directly from an authoritative YAML verification card; independent fields are never combined into synthetic commands.
66

7-
Current inventory: **26 model cards** and **364 concrete configurations**.
7+
Current inventory: **26 model cards** and **365 concrete configurations**.
88

99
Each model page lets you select import/export, pretraining, benchmarking, SFT, LoRA, or long-context SFT and immediately see the exact command and expected result for the recorded precision and GPU.
1010

‎docs/fern/versions/nightly/pages/models/deepseek/deepseek-v3.mdx‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,13 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
121121
</span>
122122
<span class="verification-combination-meta">FP8 MX</span>
123123
</button>
124+
<button type="button" class="verification-combination" data-capability="benchmark" data-precision="fp8_mx" data-hardware="GB300" data-status="verified" data-entry="deepseek-v3-pretrain-performance-hsdp-gb300" aria-controls="deepseek-v3-pretrain-performance-hsdp-gb300" aria-pressed="false">
125+
<span class="verification-combination-heading">
126+
<strong>Benchmark · HSDP · GB300</strong>
127+
<span class="verification-status verification-status--verified" title="Verified">✓ Verified</span>
128+
</span>
129+
<span class="verification-combination-meta">FP8 MX</span>
130+
</button>
124131
</div>
125132
<div class="verification-model-details">
126133
<article id="deepseek-v3-hf-to-megatron-cpu" class="verification-model-detail" data-entry-detail="deepseek-v3-hf-to-megatron-cpu" tabindex="-1">
@@ -543,6 +550,56 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
543550
<p>On exactly 256 GB300 GPUs, the canonical 50-step mock-data MXFP8 recipe completed at TP1/PP2/VP8/CP1/EP32/ETP1 and GBS/MBS 4096/1. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations, and the resolved configuration was saved. Steps 41-50 averaged 10,667.300 ms and 1,597.310 model TFLOP/s/GPU, passing gates of at most 11,000 ms and at least 1,550 TFLOP/s/GPU. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.</p>
544551
</section>
545552
</article>
553+
<article id="deepseek-v3-pretrain-performance-hsdp-gb300" class="verification-model-detail" data-entry-detail="deepseek-v3-pretrain-performance-hsdp-gb300" tabindex="-1">
554+
<header class="verification-model-detail-heading">
555+
<h4>Benchmark · HSDP · GB300</h4>
556+
<span class="verification-status verification-status--verified" title="Verified">✓ Verified</span>
557+
</header>
558+
<dl class="verification-model-detail-meta">
559+
<div><dt>Hardware</dt><dd>GB300</dd></div>
560+
<div><dt>Precision</dt><dd>FP8 MX</dd></div>
561+
<div><dt>Last verified</dt><dd>2026-10-09</dd></div>
562+
</dl>
563+
<section class="verification-recorded-metrics">
564+
<h5>Recorded metrics</h5>
565+
<dl class="verification-metric-list">
566+
<div>
567+
<dt>Initial loss</dt>
568+
<dd>11.89824</dd>
569+
</div>
570+
<div>
571+
<dt>Final loss</dt>
572+
<dd>5.146079</dd>
573+
</div>
574+
<div>
575+
<dt>Step time · last 10 avg</dt>
576+
<dd>3,197.040 ms</dd>
577+
</div>
578+
<div>
579+
<dt>Model throughput · last 10 avg</dt>
580+
<dd>1,332.420 TFLOP/s/GPU</dd>
581+
</div>
582+
<div>
583+
<dt>Token throughput · last 10 avg</dt>
584+
<dd>5,124.740 tokens/s/GPU</dd>
585+
</div>
586+
</dl>
587+
</section>
588+
<section class="verification-command-section">
589+
<h5>Exact command</h5>
590+
<div class="verification-command">
591+
<div class="verification-command-heading">
592+
<span>Command</span>
593+
<button type="button" class="verification-copy-command">Copy</button>
594+
</div>
595+
<pre><code class="language-bash">./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yaml</code></pre>
596+
</div>
597+
</section>
598+
<section class="verification-expected-result">
599+
<h5>Expected result</h5>
600+
<p>On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1. Dense parameters use ZeRO-2 optim_grads sharding while expert parameters use optim_grads_params across four distributed optimizer instances, with full-iteration CUDA graphs and MoE paged stashing enabled. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and 1,332.420 model TFLOP/s/GPU. Do not compare this item against pretrain_performance.GB300: that recipe runs a different convergence contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU throughput are not commensurable. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.</p>
601+
</section>
602+
</article>
546603
</div>
547604
</div>
548605

‎docs/models/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
Choose a model to open its interactive import/export and training configurations. Every option comes directly from an authoritative YAML verification card; independent fields are never combined into synthetic commands.
66

7-
Current inventory: **26 model cards** and **364 concrete configurations**.
7+
Current inventory: **26 model cards** and **365 concrete configurations**.
88

99
Each model page lets you select import/export, pretraining, benchmarking, SFT, LoRA, or long-context SFT and immediately see the exact command and expected result for the recorded precision and GPU.
1010

‎docs/models/deepseek/deepseek-v3.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,13 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
121121
</span>
122122
<span class="verification-combination-meta">FP8 MX</span>
123123
</button>
124+
<button type="button" class="verification-combination" data-capability="benchmark" data-precision="fp8_mx" data-hardware="GB300" data-status="verified" data-entry="deepseek-v3-pretrain-performance-hsdp-gb300" aria-controls="deepseek-v3-pretrain-performance-hsdp-gb300" aria-pressed="false">
125+
<span class="verification-combination-heading">
126+
<strong>Benchmark · HSDP · GB300</strong>
127+
<span class="verification-status verification-status--verified" title="Verified">✓ Verified</span>
128+
</span>
129+
<span class="verification-combination-meta">FP8 MX</span>
130+
</button>
124131
</div>
125132
<div class="verification-model-details">
126133
<article id="deepseek-v3-hf-to-megatron-cpu" class="verification-model-detail" data-entry-detail="deepseek-v3-hf-to-megatron-cpu" tabindex="-1">
@@ -551,6 +558,57 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
551558
<section class="verification-expected-result">
552559
<h5>Expected result</h5>
553560
<p>On exactly 256 GB300 GPUs, the canonical 50-step mock-data MXFP8 recipe completed at TP1/PP2/VP8/CP1/EP32/ETP1 and GBS/MBS 4096/1. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations, and the resolved configuration was saved. Steps 41-50 averaged 10,667.300 ms and 1,597.310 model TFLOP/s/GPU, passing gates of at most 11,000 ms and at least 1,550 TFLOP/s/GPU. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.
561+
</p>
562+
</section>
563+
</article>
564+
<article id="deepseek-v3-pretrain-performance-hsdp-gb300" class="verification-model-detail" data-entry-detail="deepseek-v3-pretrain-performance-hsdp-gb300" tabindex="-1">
565+
<header class="verification-model-detail-heading">
566+
<h4>Benchmark · HSDP · GB300</h4>
567+
<span class="verification-status verification-status--verified" title="Verified">✓ Verified</span>
568+
</header>
569+
<dl class="verification-model-detail-meta">
570+
<div><dt>Hardware</dt><dd>GB300</dd></div>
571+
<div><dt>Precision</dt><dd>FP8 MX</dd></div>
572+
<div><dt>Last verified</dt><dd>2026-10-09</dd></div>
573+
</dl>
574+
<section class="verification-recorded-metrics">
575+
<h5>Recorded metrics</h5>
576+
<dl class="verification-metric-list">
577+
<div>
578+
<dt>Initial loss</dt>
579+
<dd>11.89824</dd>
580+
</div>
581+
<div>
582+
<dt>Final loss</dt>
583+
<dd>5.146079</dd>
584+
</div>
585+
<div>
586+
<dt>Step time · last 10 avg</dt>
587+
<dd>3,197.040 ms</dd>
588+
</div>
589+
<div>
590+
<dt>Model throughput · last 10 avg</dt>
591+
<dd>1,332.420 TFLOP/s/GPU</dd>
592+
</div>
593+
<div>
594+
<dt>Token throughput · last 10 avg</dt>
595+
<dd>5,124.740 tokens/s/GPU</dd>
596+
</div>
597+
</dl>
598+
</section>
599+
<section class="verification-command-section">
600+
<h5>Exact command</h5>
601+
<div class="verification-command">
602+
<div class="verification-command-heading">
603+
<span>Command</span>
604+
<button type="button" class="verification-copy-command">Copy</button>
605+
</div>
606+
<pre><code class="language-bash">./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yaml</code></pre>
607+
</div>
608+
</section>
609+
<section class="verification-expected-result">
610+
<h5>Expected result</h5>
611+
<p>On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1. Dense parameters use ZeRO-2 optim_grads sharding while expert parameters use optim_grads_params across four distributed optimizer instances, with full-iteration CUDA graphs and MoE paged stashing enabled. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and 1,332.420 model TFLOP/s/GPU. Do not compare this item against pretrain_performance.GB300: that recipe runs a different convergence contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU throughput are not commensurable. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.
554612
</p>
555613
</section>
556614
</article>

‎examples/model_verification_cards/deepseek-v3/card.yaml‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ summary: >
66
Distributed H100 GPU import, BF16 export, and deterministic inference are
77
verified against the pinned blockwise-FP8 checkpoint. The canonical
88
pretrain_performance.H100 1024-GPU BF16, pretrain_performance.GB200 256-GPU
9-
MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified.
9+
MXFP8, and pretrain_performance.GB300 256-GPU MXFP8 recipes are also verified,
10+
as is the pretrain_performance_hsdp.GB300 256-GPU MXFP8 Hybrid FSDP recipe.
1011
A distributed CPU export of the same pinned checkpoint is also verified.
1112
Timing and throughput metrics from future functional training items remain
1213
sanity checks rather than optimized performance results. CPU import, forward
@@ -29,6 +30,8 @@ verification_index:
2930
H100: verified
3031
GB200: verified
3132
GB300: verified
33+
performance_hsdp:
34+
GB300: verified
3235
model:
3336
hf_id: deepseek-ai/DeepSeek-V3
3437
hf_revision: e815299b0bcbac849fa540c768ef21845365c9eb # pragma: allowlist secret
@@ -339,3 +342,35 @@ items:
339342
11,000 ms and at least 1,550 TFLOP/s/GPU. Mock data, forced expert
340343
balancing, and reduced-precision optimizer moments make this
341344
benchmark-only evidence, not convergence evidence.
345+
346+
pretrain_performance_hsdp:
347+
GB300:
348+
status: verified
349+
precision: fp8_mx
350+
bridge_commit: 797a3141d3ccc4ec20a45cc653a4b69f9af0cdbf # pragma: allowlist secret
351+
command: >
352+
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4
353+
--recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config
354+
--mode pretrain --max_steps 50 --seq_length 4096
355+
logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yaml
356+
last_verified: 2026-10-09
357+
metrics:
358+
initial_loss: 11.898240
359+
final_loss: 5.146079
360+
last_10_steps_step_time_ms_avg: 3197.040
361+
last_10_steps_model_tflops_per_gpu_avg: 1332.420
362+
last_10_steps_tokens_per_second_per_gpu_avg: 5124.740
363+
expected_result: >
364+
On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid
365+
FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1.
366+
Dense parameters use ZeRO-2 optim_grads sharding while expert
367+
parameters use optim_grads_params across four distributed optimizer
368+
instances, with full-iteration CUDA graphs and MoE paged stashing
369+
enabled. All 50 keyed rows had finite loss and throughput with zero
370+
skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and
371+
1,332.420 model TFLOP/s/GPU. Do not compare this item against
372+
pretrain_performance.GB300: that recipe runs a different convergence
373+
contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU
374+
throughput are not commensurable. Mock data, forced expert balancing,
375+
and reduced-precision optimizer moments make this benchmark-only
376+
evidence, not convergence evidence.

‎scripts/docs/generate_model_verification_catalog.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@
5959
"Pretrain",
6060
("pretrain", "pretrain_fsdp", "pretrain_weak_scaling"),
6161
),
62-
("benchmark", "Benchmark", ("pretrain_performance",)),
62+
("benchmark", "Benchmark", ("pretrain_performance", "pretrain_performance_hsdp")),
6363
("sft", "SFT", ("sft",)),
6464
("lora", "LoRA", ("peft",)),
6565
("long-context", "Long Context", ("sft_long_context",)),
@@ -602,6 +602,7 @@ def _entry_option_label(entry: dict[str, object]) -> str:
602602
"pretrain": "Pretrain",
603603
"pretrain_fsdp": "Pretrain · FSDP",
604604
"pretrain_performance": "Benchmark",
605+
"pretrain_performance_hsdp": "Benchmark · HSDP",
605606
"pretrain_weak_scaling": "Pretrain · Weak scaling",
606607
"sft": "SFT",
607608
"peft": "LoRA",

‎scripts/performance/dump_perf_configs.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,7 @@
167167
("deepseek", "deepseek_v3", "pretrain", 256, "gb300", "nvfp4"),
168168
("deepseek", "deepseek_v3", "pretrain", 64, "gb300", "bf16", "fsdp"),
169169
("deepseek", "deepseek_v3", "pretrain", 64, "gb300", "fp8_mx", "fsdp"),
170+
("deepseek", "deepseek_v3", "pretrain", 256, "gb300", "fp8_mx", "hsdp"),
170171
("deepseek", "deepseek_v3", "pretrain", 256, "gb200", "bf16"),
171172
("deepseek", "deepseek_v3", "pretrain", 256, "gb200", "fp8_cs"),
172173
("deepseek", "deepseek_v3", "pretrain", 256, "gb200", "fp8_mx"),

‎skills/create-model-verification-card/SKILL.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,16 @@ item; agents can derive valid comparisons from the standalone commands,
8585
resolved recipes, and raw metrics. The FSDP item does not claim checkpoint
8686
save/load unless a separate functional item records that evidence.
8787

88+
Add `pretrain_performance_hsdp` as an optional hardware-scoped item when a
89+
tuned Hybrid FSDP (HSDP) performance recipe exists for the exact model
90+
variant. Use it instead of `pretrain_fsdp` when the recipe shards dense and
91+
expert parameters under different strategies, which the single
92+
`enabled_features.megatron_fsdp` scalar cannot express. Record the result
93+
under `pretrain_performance_hsdp.<hardware>` and mirror it in
94+
`verification_index.performance_hsdp`. Keep it separate from
95+
`pretrain_performance`: an HSDP recipe usually runs a different convergence
96+
contract, so the two are not commensurable.
97+
8898
When the same hardware has multiple first-class FSDP runs with different
8999
precisions or convergence contracts, use one aggregate hardware container and
90100
key its standalone leaves by precision:

0 commit comments

Comments
 (0)