@@ -8,9 +8,10 @@ summary: >
88 items remain sanity checks rather than optimized performance results.
99 GPT-OSS 120B H100 GPU HF-to-Megatron import, GPU Megatron-to-HF export,
1010 manual HF/Megatron forward-pass correlation, deterministic Megatron
11- inference, bounded H100 pretraining, H100 PEFT, and H100 checkpoint resume
12- are verified. CPU conversion, SFT, and performance verification remain
13- planned but not yet verified.
11+ inference, bounded H100 pretraining, H100 SFT, H100 PEFT, and H100
12+ checkpoint resume, and H100 SFT export/inference are verified. CPU
13+ conversion, long-context SFT, and performance verification remain planned
14+ but not yet verified.
1415verification_index :
1516 model_level :
1617 verified :
@@ -23,8 +24,8 @@ verification_index:
2324 - megatron_to_hf_cpu
2425 training :
2526 H100 :
26- verified : [pretrain, peft, checkpoint_resume]
27- unverified : [sft, sft_export_inference, sft_long_context]
27+ verified : [pretrain, sft, sft_export_inference, peft, checkpoint_resume]
28+ unverified : [sft_long_context]
2829 performance :
2930 GB200 : unverified
3031model :
@@ -195,7 +196,7 @@ items:
195196
196197 sft :
197198 H100 :
198- status : unverified
199+ status : verified
199200 precision : bf16
200201 enabled_features :
201202 sequence_packing : offline
@@ -214,28 +215,34 @@ items:
214215 dataset.do_validation=false dataset.hf_validation_proportion=null
215216 dataset.enable_offline_packing=true
216217 +dataset.offline_packing_specs.pad_seq_to_mult=1
218+ model.tensor_model_parallel_size=2 model.sequence_parallel=true
219+ model.recompute_granularity=selective
220+ 'model.recompute_modules=[moe_act]'
217221 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0
218222 checkpoint.load=null
219223 --save_dir work/model-verification/gpt-oss-120b/sft-2k-checkpoints
220224 --save_interval 100 logger.log_interval=1 logger.log_throughput=true
221225 logger.tensorboard_dir=null
222- last_verified : null
226+ last_verified : 2026-08-12
223227 metrics :
224- initial_loss : null
225- final_loss : null
226- last_10_steps_step_time_ms_avg : null
227- last_10_steps_model_tflops_per_gpu_avg : null
228+ initial_loss : 6.751403
229+ final_loss : 0.7468084
230+ last_10_steps_step_time_ms_avg : 8063.150
231+ last_10_steps_model_tflops_per_gpu_avg : 33.800
228232 expected_result : >
229- Future support verification must complete 100 full-SFT steps from the
230- imported checkpoint with immutable Tulu3 selection, assistant-token
231- masking, offline packing, finite losses, zero skipped or NaN iterations,
232- all required metrics, a persisted post-setup ConfigContainer, and a
233- complete reloadable final full-model checkpoint. The recipe-owned batch
234- contract is not a qwen3_30b_a3b_convergence_v2 claim.
233+ Completed 100 full-SFT steps from the imported checkpoint on 32x H100
234+ with immutable Tulu3 selection, assistant-token masking, offline
235+ packing, TP2/SP, and selective MoE-activation recompute. All 100
236+ optimizer-step rows are present with finite losses, zero skipped
237+ iterations, and zero NaN iterations; loss decreases from 6.751403 to
238+ 0.7468084. All four metrics are recorded, the checkpoint-local
239+ post-setup run_config.yaml persists, and the final full-model
240+ checkpoint at iter_0000100 is complete with 32 DistCP shards, metadata,
241+ train state, tokenizer artifacts, and latest-checkpoint marker 100.
235242
236243 sft_export_inference :
237244 H100 :
238- status : unverified
245+ status : verified
239246 precision : bf16
240247 depends_on : sft
241248 commands :
@@ -246,21 +253,26 @@ items:
246253 --megatron-path work/model-verification/gpt-oss-120b/sft-2k-checkpoints/iter_0000100
247254 --hf-path work/model-verification/gpt-oss-120b/sft-hf-export
248255 --torch-dtype bfloat16 --export-weight-dtype bfloat16
249- --distributed-save --tp 1 --pp 4 --ep 8 --etp 1
256+ --distributed-save --tp 2 --pp 4 --ep 8 --etp 1
250257 --trust-remote-code
251258 - >
252259 ./scripts/inference/infer.sh --task hf-inference
253260 --nodes 1 --gpus-per-node 1
254261 --hf-model work/model-verification/gpt-oss-120b/sft-hf-export
255262 --prompt "Name the capital of France and explain its role in one sentence."
256- --max-new-tokens 45 --chat-template
257- last_verified : null
258- expected_result : >
259- After SFT is verified, future verification must export its final
260- checkpoint to a strict-reloadable BF16 Hugging Face checkpoint with
261- attention_bias=true, then run deterministic greedy HF inference and
262- record the actual generated-token count and literal completion within
263- the 45-token bound.
263+ --max-new-tokens 45 --chat-template --trust-remote-code --device cpu
264+ last_verified : 2026-08-13
265+ expected_result : |
266+ The verified step-100 SFT checkpoint exports to a strict-reloadable BF16
267+ Hugging Face checkpoint with architecture GptOssForCausalLM and
268+ attention_bias=true. The exported checkpoint has 73 safetensors shards,
269+ 615 tensors, model.safetensors.index.json total_size 233,658,313,344,
270+ tokenizer artifacts, generation_config.json, and chat_template.jinja.
271+ The deterministic HF verifier reloads the export on CPU through
272+ Transformers with no missing, unexpected, mismatched, or error loading
273+ keys and generates exactly 45 new tokens under the 45-token bound. The
274+ literal completion is:
275+ "finalThe capital of France is Paris, and it serves as the political, economic, and cultural center of the country, housing important institutions such as the French government, major corporations, and world-renowned museums and landmarks."
264276
265277 sft_long_context :
266278 H100 :
0 commit comments