Google's Gemma 4 26B-A4B is a Mixture-of-Experts vision-language model (26B total, 4B active parameters). It pairs a 128-expert top-k=8 MoE language backbone with a SigLIP vision tower, dual sliding/global attention, and K=V tying on the full-attention layers.
NeMo Megatron Bridge supports HF↔Megatron conversion, full SFT, and LoRA PEFT on image-text datasets. The finetuned model can be re-exported to 🤗 Hugging Face format for downstream evaluation or deployment.
For the full setup, conversion, inference, training, and LoRA merge /
adapter export workflows, see
examples/models/gemma/gemma4_vl/README.md.
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
- Hardware
- not specified
- Precision
- BF16
- Last verified
- 2026-07-23
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model google/gemma-4-26B-A4B-it --hf-revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 --megatron-path work/model-verification/gemma-4-26b-a4b-it/cpu-megatron --torch-dtype bfloat16The command completed all 8,548 conversion mappings and saved iter_0000000. All 383 tensor entries in the distributed-checkpoint metadata were torch.bfloat16. The run configuration recorded bf16=true, fp16=false, and BF16 parameter and autocast dtypes. The artifact reloaded successfully for CPU export.
- Hardware
- not specified
- Precision
- BF16
- Last verified
- 2026-07-23
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model google/gemma-4-26B-A4B-it --hf-revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 --megatron-path work/model-verification/gemma-4-26b-a4b-it/imported-megatron --torch-dtype bfloat16 --tp 4 --pp 2 --ep 1The command completed successfully at TP4/PP2/EP1. It loaded 8,548 conversion mappings from the immutable Hugging Face revision and saved the full multimodal checkpoint as iter_0000000. All 383 tensor entries in the distributed-checkpoint metadata were torch.bfloat16. The artifact then reloaded successfully for distributed export and direct-model inference.
- Hardware
- not specified
- Precision
- BF16
- Last verified
- 2026-07-23
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model google/gemma-4-26B-A4B-it --hf-revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 --megatron-path work/model-verification/gemma-4-26b-a4b-it/cpu-megatron/iter_0000000 --hf-path work/model-verification/gemma-4-26b-a4b-it/cpu-hf-export --torch-dtype bfloat16The command completed all 8,548 conversion mappings and wrote 11 BF16 shards. All 1,013 exported tensor entries matched the immutable source exactly in key, shape, dtype, and value. Native Gemma4ForConditionalGeneration reload reported model.dtype=torch.bfloat16 with no missing, unexpected, mismatched, or error keys.
- Hardware
- not specified
- Precision
- BF16
- Last verified
- 2026-07-23
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model google/gemma-4-26B-A4B-it --hf-revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 --megatron-path work/model-verification/gemma-4-26b-a4b-it/imported-megatron/iter_0000000 --hf-path work/model-verification/gemma-4-26b-a4b-it/hf-export-verified --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --tp 2 --pp 1 --ep 4 --etp 1Distributed TP2/PP1/EP4 export completed all 2,788 conversion mappings. The exported config preserved sliding-attention rope_theta 10000 and full-attention rope_theta 1000000. All 1,013 exported tensor entries were BF16 and matched the immutable source exactly in key, shape, dtype, and value. Native Gemma4ForConditionalGeneration reload reported model.dtype=torch.bfloat16 with no missing, unexpected, mismatched, or error keys.
- Hardware
- all
- Precision
- NOT SPECIFIED
- Last verified
- —
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
No runnable command is recorded for this status.
Megatron Bridge does not publish a pretraining recipe for this Gemma 4 conditional-generation variant; the current model package provides SFT and PEFT recipes only.
- Hardware
- H100
- Precision
- BF16
- Last verified
- 2026-08-11
- Initial loss
- 1.282612
- Final loss
- 0.09127883
- Step time · last 10 avg
- 15,936.250 ms
- Model throughput · last 10 avg
- 32.200 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,028.096 tokens/s/GPU
- Peak allocated memory
- 65.223 GiB
- Peak reserved memory
- 72.356 GiB
./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe gemma4_vl_26b_sft_config --mode sft --dataset cord-v2 --pretrained_checkpoint work/model-verification/gemma-4-26b-a4b-it/imported-megatron/iter_0000000 --max_steps 10 --seq_length 4096 'dataset.source.load_kwargs={revision:"7f0115a4b758a71d6473b8d085751692da2fef98"}' validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gemma-4-26b-a4b-it/sft-checkpoints --save_interval 10 logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/gemma-4-26b-a4b-it/sft-post-setup.yamlOn 2026-08-11 the single-node TP4/PP1/EP8/ETP1 run loaded the immutable BF16 checkpoint and real CORD-v2 revision, then completed exactly 10 optimizer steps at MBS1 and GBS32. The persisted post-setup config confirmed a frozen vision encoder, trainable projection and language model, expandable CUDA allocator segments, and full-layer uniform recompute with one layer per recompute unit directly from the recipe defaults, while CUDA graphs and CPU offload remained disabled. Loss moved from 1.282612 to 0.09127883, final grad norm was 3.090, and all 10 steps reported zero skipped and zero NaN iterations. After iteration 2, logger-reported ranks returned to 62.134-62.145 GiB allocated, while cumulative peak allocated and reserved memory remained at or below 65.223 and 72.356 GiB. The run saved a complete iter_0000010 torch_dist checkpoint with eight rank shards, finalized metadata, run config, training state, and a latest-checkpoint tracker containing iteration 10.
- Hardware
- H100
- Precision
- BF16
- Last verified
- —
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
No runnable command is recorded for this status.
No Gemma 4 VL long-context recipe or run was verified. A future workload must establish sequence-packing and context-parallel settings, complete at least 10 optimizer steps with finite loss and no skipped or NaN iterations, report all five required metrics, and save a reloadable final checkpoint.
- Hardware
- H100
- Precision
- BF16
- Last verified
- 2026-08-04
- Initial loss
- 1.288328
- Final loss
- 0.1072377
- Step time · last 10 avg
- 14,163.250 ms
- Model throughput · last 10 avg
- 65.010 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,313.593 tokens/s/GPU
- Peak allocated memory
- 44.252 GiB
- Peak reserved memory
- 45.637 GiB
./scripts/training/train.sh --nodes 1 --gpus-per-node 4 --recipe gemma4_vl_26b_peft_config --mode lora --dataset cord-v2 --pretrained_checkpoint work/model-verification/gemma-4-26b-a4b-it/imported-megatron/iter_0000000 --max_steps 10 --seq_length 4096 'dataset.source.load_kwargs={revision:"7f0115a4b758a71d6473b8d085751692da2fef98"}' validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gemma-4-26b-a4b-it/peft-checkpoints --save_interval 10 logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/gemma-4-26b-a4b-it/peft-post-setup.yamlOn 2026-08-04 the 4-GPU TP2/PP1/EP4/ETP1 LoRA run loaded the immutable BF16 checkpoint and real CORD-v2 revision, then completed exactly 10 optimizer steps at MBS1 and GBS32. The persisted post-setup config recorded rank-32 adapters with alpha 32 and zero dropout on linear_qkv, linear_proj, linear_fc1, and linear_fc2. Runtime transformation froze the base model and reported 22,272,000 trainable adapter parameters, 0.30% of the local model shard. Loss moved from 1.288328 to 0.1072377, final grad norm was 22.994, and every step reported zero skipped and zero NaN iterations. Across all four ranks, allocated memory returned to 14.582 GiB after each step, while cumulative peak allocated and reserved memory remained at or below 44.252 and 45.637 GiB. The run saved a complete iter_0000010 adapter checkpoint with four rank shards, finalized metadata, training state, and a latest-checkpoint tracker containing iteration 10.