You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 02873e5
Browse filesBrowse the repository at this point in the historyBrowse files
"expected_result": "The uninterrupted 352-H100 TP1/PP11/CP1/EP32/ETP1 run completed 100 real-data steps at GBS/MBS 1024/1 with cuDNN DSA, sparse indexer loss coefficient 0.001, MTP1, and flex/DeepEP dispatch. Loss remained finite from 13.15907 to 6.385243, final grad norm was 0.796, and every step recorded zero skipped and NaN iterations. The final 10 steps averaged 31,204.920 ms and 54.595 TFLOPS/GPU; peak observed GPU memory was 80,999 MiB. The matching step-50 and step-100 checkpoints contain model, optimizer, scheduler, data-order, and RNG state; the step-50 checkpoint reloads successfully.\n",
11326
+
"expected_result": "The 352-H100 TP1/PP11/CP1/EP32/ETP1 run warm-starts from a Bridge GPU import of the GLM-5.2 weights that includes the MTP layer and completes exactly 20 bounded steps at GBS/MBS 1024/1 and sequence length 2048 with cuDNN DSA, sparse indexer loss coefficient 0.001, MTP1, HybridEP flex expert dispatch, and natural routing, without deterministic mode. Data is the WikiText-103 raw train split grouped into 29,444 articles and tokenized with the pinned GLM-5.2 tokenizer (120,372,951 tokens). The learning rate warms up to 3e-6 at step 10 and decays to 3e-7 at step 20. Loss decreases from 1.078868 to 0.9303461 with a maximum grad norm of 4.478, a final grad norm of 1.725, and zero skipped or NaN iterations. HybridEP JIT-compiles its kernels at each pipeline stage's first dispatch, one stage after another, so step 1 takes 1,558.3 s and the distributed timeout is raised from 10 to 60 minutes. Steps 11-20 average 16,505.840 ms, 103.280 model TFLOP/s/GPU, and 360.952 token slots/s/GPU. The step-10 and step-20 checkpoints each contain model, optimizer, scheduler, data-order, and RNG state in 359 files (about 10.56 TB).\n",
"expected_result": "The run reloads the corrected reference's step-50 model, optimizer, scheduler, data-order, and RNG state and completes steps 51-100 with zero skipped or NaN iterations. Step 51 matches exactly at 7.219822. Step 100 is 6.372624 versus the uninterrupted reference's 6.385243, an absolute difference of 0.012619 within the allowed 0.063853. Final grad norm is 2.169; the last 10 steps average 32,002.290 ms and 53.235 TFLOPS/GPU. Peak observed allocator memory is 65.230 GiB allocated and 69.684 GiB reserved.\n",
11658
+
"expected_result": "The run reloads the warm-start reference's step-10 model, optimizer, scheduler, data-order, and RNG state and completes steps 11-20 into a new root with zero skipped or NaN iterations. Step 11 is 0.9305546, identical to the uninterrupted reference, and step 20 is 0.9313419 versus 0.9303461, a relative difference of 0.107%. Every resumed LM and MTP loss stays within the declared bound, with a largest LM relative difference of 0.113% at step 19. The resolved reference and resume configs differ only in the checkpoint source, step, pretrained-checkpoint fallback, and output paths. The resume requires PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True in the launch environment; the reference ran without it. The recipe's own allocator setting is applied after CUDA initializes and does not take effect, and without expandable segments the first resumed step exhausts device memory when NCCL allocates buffers for the data-parallel gradient reduce-scatter on the 8-layer pipeline stages. The final-10-step window is the whole resumed segment, so its averages include HybridEP JIT compilation at step 11 (1,835.1 s); steps 12-20 average 17,201.3 ms.\n",
"summary": "Performance disclaimer: this card does not record a canonical pretrain performance result; reported timing and throughput metrics are functional verification observations, not standalone optimized performance results. GLM-5.2 H100 support verification currently covers exact GPU checkpoint round-trip on CPU and GPU (the CPU path excludes the disabled MTP layer), HF-to-Megatron forward equivalence, bounded full SFT with export/reload inference, 200K-context CP32 SFT, bounded PEFT, and bounded pretraining with a reloadable middle checkpoint and verified continuation. GB200 support verification currently covers bounded warm-start pretraining, bounded full SFT, 128K packed long-context SFT, bounded PEFT, bounded checkpoint resume, and deterministic legacy Megatron inference; post-SFT export/reload inference remains pending.\n",
11713
+
"summary": "Performance disclaimer: this card does not record a canonical pretrain performance result; reported timing and throughput metrics are functional verification observations, not standalone optimized performance results. GLM-5.2 H100 support verification currently covers exact GPU checkpoint round-trip on CPU and GPU (the CPU path excludes the disabled MTP layer), HF-to-Megatron forward equivalence, bounded full SFT with export/reload inference, 200K-context CP32 SFT, bounded PEFT, bounded warm-start pretraining, and bounded checkpoint resume from its step-10 checkpoint. GB200 support verification currently covers bounded warm-start pretraining, bounded full SFT, 128K packed long-context SFT, bounded PEFT, bounded checkpoint resume, and deterministic legacy Megatron inference; post-SFT export/reload inference remains pending.\n",
<p>The uninterrupted 352-H100 TP1/PP11/CP1/EP32/ETP1 run completed 100 real-data steps at GBS/MBS 1024/1 with cuDNN DSA, sparse indexer loss coefficient 0.001, MTP1, and flex/DeepEP dispatch. Loss remained finite from 13.15907 to 6.385243, final grad norm was 0.796, and every step recorded zero skipped and NaN iterations. The final 10 steps averaged 31,204.920 ms and 54.595 TFLOPS/GPU; peak observed GPU memory was 80,999 MiB. The matching step-50 and step-100 checkpoints contain model, optimizer, scheduler, data-order, and RNG state; the step-50 checkpoint reloads successfully.</p>
308
+
<p>The 352-H100 TP1/PP11/CP1/EP32/ETP1 run warm-starts from a Bridge GPU import of the GLM-5.2 weights that includes the MTP layer and completes exactly 20 bounded steps at GBS/MBS 1024/1 and sequence length 2048 with cuDNN DSA, sparse indexer loss coefficient 0.001, MTP1, HybridEP flex expert dispatch, and natural routing, without deterministic mode. Data is the WikiText-103 raw train split grouped into 29,444 articles and tokenized with the pinned GLM-5.2 tokenizer (120,372,951 tokens). The learning rate warms up to 3e-6 at step 10 and decays to 3e-7 at step 20. Loss decreases from 1.078868 to 0.9303461 with a maximum grad norm of 4.478, a final grad norm of 1.725, and zero skipped or NaN iterations. HybridEP JIT-compiles its kernels at each pipeline stage's first dispatch, one stage after another, so step 1 takes 1,558.3 s and the distributed timeout is raised from 10 to 60 minutes. Steps 11-20 average 16,505.840 ms, 103.280 model TFLOP/s/GPU, and 360.952 token slots/s/GPU. The step-10 and step-20 checkpoints each contain model, optimizer, scheduler, data-order, and RNG state in 359 files (about 10.56 TB).</p>
0 commit comments