Skip to content

Commit a124b60

Browse files
committed
pretrain perf verified
Signed-off-by: Maanu Grover <maanug@nvidia.com>
1 parent 97deb4a commit a124b60

1 file changed

Lines changed: 22 additions & 19 deletions

File tree

  • examples/model_verification_cards/gpt-oss-120b

‎examples/model_verification_cards/gpt-oss-120b/card.yaml‎

Lines changed: 22 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ summary: >
1010
manual HF/Megatron forward-pass correlation, deterministic Megatron
1111
inference, bounded H100 pretraining, H100 SFT, H100 PEFT, and H100
1212
checkpoint resume, H100 SFT export/inference, and H100 long-context SFT are
13-
verified. CPU conversion and performance verification remain planned but not
14-
yet verified.
13+
verified. GB200 pretraining performance is verified. CPU conversion remains
14+
planned but not yet verified.
1515
verification_index:
1616
model_level:
1717
verified:
@@ -26,7 +26,7 @@ verification_index:
2626
H100:
2727
verified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
2828
performance:
29-
GB200: unverified
29+
GB200: verified
3030
model:
3131
hf_id: openai/gpt-oss-120b
3232
hf_revision: b5c939de8f754692c1647ca79fbf85e8c1e70f8a # pragma: allowlist secret
@@ -414,26 +414,29 @@ items:
414414
415415
pretrain_performance:
416416
GB200:
417-
status: unverified
417+
status: verified
418418
precision: bf16
419419
command: >
420420
./scripts/training/train.sh --nodes 16 --gpus-per-node 4
421421
--recipe gpt_oss_120b_pretrain_64gpu_gb200_bf16_config
422-
--mode pretrain --max_steps 50
423-
last_verified: null
422+
--mode pretrain --max_steps 50 logger.log_throughput=true
423+
last_verified: 2026-08-17
424424
metrics:
425-
initial_loss: null
426-
final_loss: null
427-
last_10_steps_step_time_ms_avg: null
428-
last_10_steps_model_tflops_per_gpu_avg: null
429-
last_10_steps_tokens_per_second_per_gpu_avg: null
425+
initial_loss: 12.79104
426+
final_loss: 0.03246120
427+
last_10_steps_step_time_ms_avg: 4995.630
428+
last_10_steps_model_tflops_per_gpu_avg: 535.230
429+
last_10_steps_tokens_per_second_per_gpu_avg: 16398.332
430430
expected_result: >
431-
Future verification must run the exact canonical 64-GB200 BF16
432-
mock-data performance recipe for 50 optimizer steps, record finite
433-
losses with zero skipped or NaN iterations and all required metrics,
434-
and define measured step-time and model-TFLOP/s/GPU thresholds before
435-
this item is marked verified. The recipe uses TP1/PP1/EP64,
436-
GBS1280/MBS4, forced router balancing, selective layernorm and MoE
437-
activation recompute, and expert-parallel communication overlap; this
438-
benchmark batch contract makes it performance evidence, not
431+
On exactly 64 GB200s, the canonical BF16 mock-data performance recipe
432+
completes 50 optimizer steps at TP1/PP1/CP1/EP64/ETP1, GBS/MBS
433+
1280/4, forced router balancing, selective layernorm and MoE
434+
activation recompute, and expert-parallel communication overlap. All
435+
50 optimizer-step rows are present exactly once with finite loss, zero
436+
skipped iterations, and zero NaN iterations. Loss decreases from
437+
12.79104 to 0.03246120. The final ten logged step times average
438+
4995.630 ms, measured model throughput averages 535.230 TFLOP/s/GPU,
439+
and derived token-slot throughput averages 16398.332 tokens/s/GPU over
440+
the same final-ten window. The resolved configuration persists. Mock
441+
data and forced routing make the loss a finiteness check rather than
439442
convergence evidence.

0 commit comments

Comments
 (0)