@@ -11,11 +11,12 @@ summary: >
1111 forward comparison remains unverified because the recorded run compared a
1212 tensor-parallel padding position rather than the prompt's next token.
1313 Decoder-focused SFT is verified on one 8-GPU H100 node with TP4/PP1/EP8/ETP1,
14- a frozen vision encoder, and selective core-attention and MoE-activation
15- recompute. The real CORD-v2 run completed 10 optimizer steps with stable
16- memory and saved iter_0000010. LoRA PEFT is also verified on four H100 GPUs
17- with TP2/PP1/EP4/ETP1 and a complete adapter checkpoint. Long-context SFT is
18- pending. The published Bridge recipes do not currently provide VLM pretraining.
14+ a frozen vision encoder, and full-layer uniform activation recompute with one
15+ layer per recompute unit. The real CORD-v2 run completed 10 optimizer steps
16+ with stable memory and saved iter_0000010. LoRA PEFT is also verified on four
17+ H100 GPUs with TP2/PP1/EP4/ETP1 and a complete adapter checkpoint.
18+ Long-context SFT is pending. The published Bridge recipes do not currently
19+ provide VLM pretraining.
1920verification_index :
2021 model_level :
2122 verified :
@@ -181,39 +182,43 @@ items:
181182 status : verified
182183 precision : bf16
183184 enabled_features : {}
184- bridge_commit : 81e559ca4d579cf2d9ed5b8f3f48bef82e06451f # pragma: allowlist secret
185+ bridge_commit : 07d182e7e8df4421e445906b29e6b5f4a2f4a6f1 # pragma: allowlist secret
185186 command : >
186187 ./scripts/training/train.sh --nodes 1 --gpus-per-node 8
187188 --recipe gemma4_vl_26b_sft_config --mode sft --dataset cord-v2
188189 --pretrained_checkpoint work/model-verification/gemma-4-26b-a4b-it/imported-megatron/iter_0000000
189190 --max_steps 10 --seq_length 4096
190191 'dataset.source.load_kwargs={revision:"7f0115a4b758a71d6473b8d085751692da2fef98"}'
191192 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null
193+ model.recompute_granularity=full model.recompute_modules=null
194+ model.recompute_method=uniform model.recompute_num_layers=1
195+ model.cuda_graph_impl=none
192196 --save_dir work/model-verification/gemma-4-26b-a4b-it/sft-checkpoints
193197 --save_interval 10 logger.log_interval=1 logger.log_throughput=true
194198 logger.save_config_filepath=work/model-verification/gemma-4-26b-a4b-it/sft-post-setup.yaml
195- last_verified : 2026-08-04
199+ last_verified : 2026-08-11
196200 metrics :
197- initial_loss : 1.268018
198- final_loss : 0.06543536
199- last_10_steps_step_time_ms_avg : 18699.99
200- last_10_steps_model_tflops_per_gpu_avg : 26.47
201- peak_allocated_memory_gib : 70.624
202- peak_reserved_memory_gib : 71.021
201+ initial_loss : 1.282612
202+ final_loss : 0.07524271
203+ last_10_steps_step_time_ms_avg : 15600.68
204+ last_10_steps_model_tflops_per_gpu_avg : 32.95
205+ peak_allocated_memory_gib : 65.223
206+ peak_reserved_memory_gib : 72.356
203207 expected_result : >
204- On 2026-08-04 the single-node TP4/PP1/EP8/ETP1 run loaded the immutable
208+ On 2026-08-11 the single-node TP4/PP1/EP8/ETP1 run loaded the immutable
205209 BF16 checkpoint and real CORD-v2 revision, then completed exactly 10
206210 optimizer steps at MBS1 and GBS32. The persisted post-setup config
207211 confirmed a frozen vision encoder, trainable projection and language
208- model, expandable CUDA allocator segments, and selective core_attn plus
209- moe_act recompute with CUDA graphs and CPU offload disabled. Loss moved
210- from 1.268018 to 0.06543536, final grad norm was 1.561, and every step
211- reported zero skipped and zero NaN iterations. Across all eight ranks,
212- allocated memory returned to 56.493 GiB after each step, while
213- cumulative peak allocated and reserved memory remained at or below
214- 70.624 and 71.021 GiB. The run saved a complete iter_0000010 torch_dist
215- checkpoint with eight rank shards, finalized metadata, training state,
216- and a latest-checkpoint tracker containing iteration 10.
212+ model, expandable CUDA allocator segments, and full-layer uniform
213+ recompute with one layer per recompute unit while CUDA graphs and CPU
214+ offload remained disabled. Loss moved from 1.282612 to 0.07524271,
215+ final grad norm was 2.094, and every step reported zero skipped and zero
216+ NaN iterations. After iteration 2, logger-reported ranks returned to
217+ 62.134-62.145 GiB allocated, while cumulative peak allocated and
218+ reserved memory remained at or below 65.223 and 72.356 GiB. The run
219+ saved a complete iter_0000010 torch_dist checkpoint with eight rank
220+ shards, finalized metadata, run config, training state, and a
221+ latest-checkpoint tracker containing iteration 10.
217222
218223 sft_export_inference :
219224 H100 :
0 commit comments