Skip to content

Commit db4ea1f

Browse files
committed
long context sft verified
Signed-off-by: Maanu Grover <maanug@nvidia.com>
1 parent f8d318a commit db4ea1f

1 file changed

Lines changed: 42 additions & 19 deletions

File tree

  • examples/model_verification_cards/gpt-oss-120b

‎examples/model_verification_cards/gpt-oss-120b/card.yaml‎

Lines changed: 42 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@ summary: >
99
GPT-OSS 120B H100 GPU HF-to-Megatron import, GPU Megatron-to-HF export,
1010
manual HF/Megatron forward-pass correlation, deterministic Megatron
1111
inference, bounded H100 pretraining, H100 SFT, H100 PEFT, and H100
12-
checkpoint resume, and H100 SFT export/inference are verified. CPU
13-
conversion, long-context SFT, and performance verification remain planned
14-
but not yet verified.
12+
checkpoint resume, H100 SFT export/inference, and H100 long-context SFT are
13+
verified. CPU conversion and performance verification remain planned but not
14+
yet verified.
1515
verification_index:
1616
model_level:
1717
verified:
@@ -24,8 +24,7 @@ verification_index:
2424
- megatron_to_hf_cpu
2525
training:
2626
H100:
27-
verified: [pretrain, sft, sft_export_inference, peft, checkpoint_resume]
28-
unverified: [sft_long_context]
27+
verified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
2928
performance:
3029
GB200: unverified
3130
model:
@@ -276,23 +275,47 @@ items:
276275
277276
sft_long_context:
278277
H100:
279-
status: unverified
278+
status: verified
280279
precision: bf16
281-
enabled_features: {}
282-
command: null
283-
last_verified: null
280+
enabled_features:
281+
sequence_packing: offline
282+
context_parallel_size: 4
283+
command: >
284+
./scripts/training/train.sh --nodes 6 --gpus-per-node 8
285+
--recipe gpt_oss_120b_sft_48gpu_h100_bf16_32k_config --mode sft
286+
--dataset tulu3
287+
--pretrained_checkpoint work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000
288+
--max_steps 20 --seq_length 32768
289+
--lr 1e-6 --min_lr 0 --warmup_iters 2
290+
'dataset.hf_dataset.split="train[:10000]"'
291+
'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}'
292+
'++tokenizer.hf_tokenizer_kwargs.revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a"'
293+
dataset.hf_output_root=work/data/tulu3/gpt-oss-120b-sft-long-context-32k-cp4-pp6-nofuse-gbs4-b14afda60f1b
294+
dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678
295+
dataset.do_validation=false dataset.hf_validation_proportion=null
296+
dataset.enable_offline_packing=true
297+
+dataset.offline_packing_specs.pad_seq_to_mult=8
298+
scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0
299+
checkpoint.load=null checkpoint.save=null
300+
logger.log_interval=1 logger.log_throughput=true
301+
logger.tensorboard_dir=null
302+
last_verified: 2026-08-13
284303
metrics:
285-
initial_loss: null
286-
final_loss: null
287-
last_10_steps_step_time_ms_avg: null
288-
last_10_steps_model_tflops_per_gpu_avg: null
304+
initial_loss: 6.666666
305+
final_loss: 1.238030
306+
last_10_steps_step_time_ms_avg: 24388.780
307+
last_10_steps_model_tflops_per_gpu_avg: 5.070
289308
expected_result: >
290-
A model-appropriate long-context topology and memory plan must be
291-
resolved before a public command is recorded. Future verification must
292-
exercise sequence packing with context parallelism greater than one,
293-
use the topology-derived packing alignment, complete at least 10 steps
294-
with finite losses and zero skipped or NaN iterations, and record all
295-
required metrics.
309+
Completed 20 long-context full-SFT optimizer steps from the imported
310+
checkpoint on 48x H100 with 32K sequence length, immutable Tulu3
311+
selection, assistant-token masking, offline packing, TP2/SP, CP4 with
312+
a2a context-parallel communication, PP6, EP4, disabled cross-entropy
313+
and bias-activation fusions, and selective MoE-activation recompute.
314+
All 20 optimizer-step rows are present exactly once with finite losses,
315+
zero skipped iterations, and zero NaN iterations. Loss decreases from
316+
6.666666 to 1.238030, all four metrics are recorded, and no checkpoint
317+
is saved because this bounded long-context verification run disables
318+
checkpoint output.
296319
297320
peft:
298321
H100:

0 commit comments

Comments
 (0)