You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 358b5b7
Browse filesBrowse the repository at this point in the historyBrowse files
fix(recipe)!: run Nemotron 3 Ultra H100 SFT on the documented 48-node layout
Replace nemotron_3_ultra_sft_192gpu_h100_bf16_openmathinstruct2_packed_config
with nemotron_3_ultra_sft_384gpu_h100_bf16_openmathinstruct2_packed_config. The
new recipe uses the 48-node layout that the Ultra README and slurm_sft.sh
already document as the H100 starting point for packed OpenMath full SFT:
TP2/PP12/EP16, a 90-minute distributed timeout, and
CUDA_DEVICE_MAX_CONNECTIONS=1.
The previous 24-node TP2/PP6/EP32 default was never verified, and the example
scripts launched the recipe with 48-node overrides instead. At expert-DP 1
each GPU keeps unsharded expert parameters, fp32 gradients, and Adam state;
runs of that layout ran out of memory in the first training step, and the
first step's serialized startup work can also outlast the default 10-minute
NCCL timeout.
The recipe splits the 108 decoder layers unevenly across the 12 pipeline
stages: 9 on each of stages 0-7, 10 on stages 8 and 9, 11 on stage 10, and 5
on the last stage, which also holds the output layer and the two-depth MTP
block. With the even split of 9 layers per stage, the last stage ran out of
memory in the second training step.
Recompute stays selective, on moe, core_attn, and moe_act, which fits the
48-node layout. The pre-MLP norms are no longer recomputed: on Transformer
Engine 2.18 and later, a recomputed TE norm that is a pipeline stage's first
backward operation fails with "invalid device context" on GPU 0, and most
stages of this layout end with an MoE layer. The recipe also sets
NVTE_NORM_FWD_USE_CUDNN=1 and NVTE_NORM_BWD_USE_CUDNN=1 so normalization runs
on cuDNN kernels instead of TE's NVRTC path, covering any other norm that
becomes a stage's first backward operation. Full-layer recompute, which the
README previously suggested, hits the same failure, so slurm_sft.sh no longer
overrides the recipe's recompute settings by default.
The generic nemotron_3_ultra_sft_openmathinstruct2_packed_config alias now
points to the 384-GPU recipe. Optimizer, schedule, data, and batch settings
are unchanged. Update the verification card command to 48 nodes, the README
starting points, and the generated model guides.
Addresses NVBug 6674155.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
"expected_result": "The public 192-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.\n",
5377
+
"expected_result": "The public 384-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.\n",
<p>The public 192-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.</p>
416
+
<p>The public 384-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.</p>
<p>The public 192-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.
422
+
<p>The public 384-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.
0 commit comments