glm5.md

GLM-5

GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3 use the shared GLM5Bridge for their MoE, Multi-Latent Attention, and Dynamic Sparse Attention architecture. GLM-5.3 shares GLM-5.2's architecture and import mappings; full-model GLM-5.3 verification remains pending.

GLM-5.2 / GLM-5.3 compatibility

The pinned publisher configs for GLM-5.2 and GLM-5.3 have identical architecture fields: 78 decoder layers, 256 routed experts with top-8 routing, MLA, and the same DSA IndexShare pattern. All 59,585 base checkpoint tensor names and shapes also match. AutoBridge resolves both to GlmMoeDsaForCausalLM and GLM5Bridge; no separate provider is needed. GLM-5.3-Flash is a different architecture and is outside this support statement.

The checkpoints have important differences:

• Precision: GLM-5.2 stores BF16 weights (plus FP32 router biases). GLM-5.3 stores 59,044 weights as E4M3 FP8 with FP32 scales per 128×128 block. The existing GLM bridge dequantizes these weights to BF16 on import. This is distinct from enabling FP8 training.
• Config metadata: GLM-5.3 adds quantization_config and records Transformers 5.15.0 instead of 5.12.0. Use the repository's supported dependency versions.
• Tokenizer and generation defaults: tokenizer.json, tokenizer_config.json, and generation_config.json are identical at those revisions, including token IDs and sampling defaults.
• Chat formatting: GLM-5.3's template adds low reasoning effort, retains historical reasoning by default, always opens a thinking response (it does not honor enable_thinking=False), and changes structured tool-result handling and ordering. Load the template from the GLM-5.3 checkpoint; do not substitute GLM-5.2's template.

For config/provider loading without weights:

from megatron.bridge import AutoBridge

bridge = AutoBridge.from_hf_pretrained(
    "zai-org/GLM-5.3", revision="aca966e4e02791568aa6a4ced368624b3d897f42"  # pragma: allowlist secret (public HF revision)
)
provider = bridge.to_megatron_provider(load_weights=False)

For BF16 export using the FP8 source as the reference, explicitly select --export-weight-dtype bfloat16 with the GPU or distributed-CPU conversion backend (API: weight_dtype=torch.bfloat16). This removes FP8 scale tensors from strict source-key validation and writes a BF16 artifact rather than claiming a bitwise FP8 round trip. The serial-CPU backend does not support this option. MTP is disabled by default; the appended MTP layer is outside that default inference graph.

Verification limits

The recorded commands and results below belong to the checkpoint named in each verification card. GLM-5.2 results do not establish GLM-5.3 full-checkpoint conversion/export, forward parity, generation, SFT/PEFT, resume, long-context behavior, or performance. Those GLM-5.3 checks remain unverified. The glm52_* recipes are pinned to GLM-5.2 and retain that name and attribution; changing only their display name would not select GLM-5.3 weights or its chat template.

Production qualification tracker #5476 remains open. It includes repeated-MTP training semantics, precision controls, dynamic context parallelism, production recipes, checkpoint continuity, convergence, and hardware-specific performance qualification. See also the open export context-length fix #5996, router-bias fix #6036, and recipe-selection fix #5608.

HybridModel migration

The shared GLM-5/5.1/5.2/5.3 bridge constructs HybridModel through GLM5ModelProvider. The outer language model is Core's native HybridModel; GLM-specific runtime adaptations are supplied through the stack specification. Each HF block becomes a D- (dense) or DE (MoE) pair; 78 HF blocks therefore become 156 physical Hybrid layers. A thin GLMDSAttention subclass reads the per-forward index-sharing state from a context variable owned by the GLM stage; the stack translates the HF index-sharing cadence into physical layer coordinates on private per-layer config copies. HF export retains the original layer count, and MTP uses a separate /DE pattern when enabled.

Import the pinned HF checkpoint into a new Hybrid checkpoint directory. The generic Core GPT-to-Hybrid converter does not support this DSA architecture; old GPT optimizer/checkpoint resume is outside this migration. get_model_config() is not supported for this family yet; use to_megatron_provider().

The initial recipes use BF16 and non-VPP pipeline segments. Pattern boundaries must begin on DSA compute layers. Full recomputation replays an entire pipeline stage to preserve forward-local top-k sharing, so memory and throughput must be remeasured. Selective core_attn recomputation is rejected because its backward replay does not preserve the forward-local DSA sharing state. Verified Hybrid conversion, forward correlation, inference, and GB200 training/resume results, including MTP and packed CP, are recorded below. Historical GPT results do not verify Hybrid.

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import & Export Pretrain Benchmark SFT LoRA Long Context

Precision All BF16 FP8 MX NVFP4

GPU All GB200

Import · CPU ✓ Verified BF16 Import · GPU ✓ Verified BF16 Export · CPU ✓ Verified BF16 Export · GPU ✓ Verified BF16 Pretrain · GB200 ✓ Verified BF16 SFT · GB200 ✓ Verified BF16 Long Context · GB200 ✓ Verified BF16 LoRA · GB200 ✓ Verified BF16

Import · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-22

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 4 --cpu-processes-per-node 8 --cpus-per-task 16 --mem 0 --exclusive --hf-model zai-org/GLM-5 --hf-revision 4e6698ba8e85059d749020e3c4d2123719f23926 --megatron-path work/model-verification/glm5/hybrid/cpu-megatron --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 2 --distributed-timeout-minutes 240

Expected result

Distributed CPU HF-to-Hybrid import with 32 Gloo processes on the host memory of 4 GB200 nodes maps all 6201 pinned source tensors through GLM5Bridge and saves a 1.4 TB torch_dist checkpoint (iter_0000000 with run_config.yaml) in about 13 minutes. The checkpoint is strictly reloadable, as exercised by a distributed CPU export of it whose 59079 tensors match the pinned source keys, shapes, dtypes and values exactly (the 791 tensors of the appended MTP layer, model.layers.78.*, are excluded because MTP is disabled by default). Weights stay on CPU, but Megatron-Core's CUDA RNG tracker still requires a visible CUDA device on each node during model-parallel initialization, so the run cannot execute on GPU-less nodes. Attention, dense and shared-expert weights are replicated across the 16 expert ranks of each pipeline stage, so peak host memory reached about 930 GB of the 952 GB per node (up to 177 GB for a single process); a single-process CPU import would need more than 1.5 TB on one host.

Import · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model zai-org/GLM-5 --hf-revision 4e6698ba8e85059d749020e3c4d2123719f23926 --megatron-path work/model-verification/glm5/hybrid/gpu-megatron --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 1 --distributed-timeout-minutes 60 --low-memory-save

Expected result

Distributed HF-to-Hybrid import on 16 GB200 GPUs maps all 6201 pinned source tensors through GLM5Bridge and saves a 1.4 TB distributed checkpoint (iter_0000000 with run_config.yaml) in about 10 minutes. The checkpoint is strictly reloadable, as exercised by the verified GPU export. The appended MTP layer stays disabled by default.

Export · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 4 --cpu-processes-per-node 8 --cpus-per-task 16 --mem 0 --exclusive --hf-model zai-org/GLM-5 --hf-revision 4e6698ba8e85059d749020e3c4d2123719f23926 --megatron-path work/model-verification/glm5/hybrid/gpu-megatron/iter_0000000 --hf-path work/model-verification/glm5/hybrid/cpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 2 --distributed-timeout-minutes 240 --distributed-save --save-every-n-ranks 1 --no-progress

Expected result

Distributed CPU Hybrid-to-HF export with 32 Gloo processes on the host memory of 4 GB200 nodes strictly reloads the imported checkpoint and writes 280 safetensors shards plus model.safetensors.index.json in about 28 minutes. All 59079 exported tensors match the pinned source keys, shapes, dtypes and values exactly; the 791 tensors of the appended MTP layer (model.layers.78.*) are excluded because MTP is disabled by default. Weights stay on CPU, but Megatron-Core's CUDA RNG tracker still requires a visible CUDA device on each node during model-parallel initialization, so the run cannot execute on GPU-less nodes. generation_config.json is preserved from the source because Transformers strict validation rejects re-saving it.

Export · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model zai-org/GLM-5 --hf-revision 4e6698ba8e85059d749020e3c4d2123719f23926 --megatron-path work/model-verification/glm5/hybrid/gpu-megatron/iter_0000000 --hf-path work/model-verification/glm5/hybrid/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 1 --distributed-timeout-minutes 60 --distributed-save --save-every-n-ranks 1

Expected result

Distributed Hybrid-to-HF export on 16 GB200 GPUs strictly reloads the imported checkpoint and writes 280 safetensors shards plus model.safetensors.index.json in about 10 minutes. All 59079 exported tensors match the pinned source keys, shapes, dtypes and values exactly; the 791 tensors of the appended MTP layer (model.layers.78.*) are excluded because MTP is disabled by default. generation_config.json is preserved from the source because Transformers strict validation rejects re-saving it.

Pretrain · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

0.989893

Final loss

0.7442691

Step time · last 10 avg

33,684.140 ms

Model throughput · last 10 avg

204.370 TFLOP/s/GPU

Token throughput · last 10 avg

648.535 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm5_pretrain_192gpu_gb200_bf16_config --mode pretrain --max_steps 100 --save_dir work/model-verification/glm5/hybrid/pretrain-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 50 checkpoint.load=null checkpoint.save_optim=true checkpoint.save_rng=true logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5/hybrid/pretrain-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5/hybrid/hf-4e6698ba8e85059d749020e3c4d2123719f23926 --dataset megatron-indexed --seq_length 4096 --lr 3e-6 --min_lr 3e-7 --warmup_iters 40 'dataset.blend=[["work/data/wikitext103-glm5-glm-remaining-20261007-175000/wikitext103_glm5_text_document"],null]' dataset.path_to_cache=work/cache/glm5/wikitext-glm-remaining-20261007-175000 dataset.random_seed=1234 dataset.num_workers=8 tokenizer.use_tokenizer_vocab_size=false rng.seed=1234 scheduler.lr_decay_iters=100 model.moe_router_force_load_balancing=false

Expected result

Completed 100 BF16 Hybrid warm-start steps on 192 GB200 GPUs with WikiText-103 raw train tokenized by the pinned GLM-5 tokenizer, 4K sequences, GBS/MBS 1024/1, MTP1 and natural routing. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved complete step-50 and step-100 checkpoints with optimizer, scheduler, RNG state and post-setup configuration.

SFT · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

1.456939

Final loss

0.3063886

Step time · last 10 avg

6,468.400 ms

Model throughput · last 10 avg

14.180 TFLOP/s/GPU

Token throughput · last 10 avg

52.769 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm5_sft_192gpu_gb200_bf16_config --mode sft --max_steps 100 --save_dir work/model-verification/glm5/hybrid/sft-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 100 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5/hybrid/sft-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5/hybrid/hf-4e6698ba8e85059d749020e3c4d2123719f23926 dataset.hf_output_root=work/data/glm5/sft-glm-remaining-20261007-175000

Expected result

Completed 100 BF16 Hybrid full-SFT steps with pinned Tulu 3 train[:10000], 8K packing, GBS/MBS 8/1 and CP4 on 192 GB200 GPUs, with MTP1, HybridEP and natural routing. The exact runtime passed full-shape causal top-k reference checks with explicit cuDNN query offsets before training. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved the complete final model checkpoint and post-setup configuration. Strictly reloaded the step-100 checkpoint with raise_all and completed a further training step. Metrics cover the original 100 steps only.

Long Context · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

3.555943

Final loss

3.37788

Step time · last 10 avg

149,688.470 ms

Model throughput · last 10 avg

96.150 TFLOP/s/GPU

Token throughput · last 10 avg

255.393 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm5_sft_192gpu_gb200_bf16_128k_config --mode sft --max_steps 20 --save_dir work/model-verification/glm5/hybrid/long-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 20 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5/hybrid/long-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5/hybrid/hf-4e6698ba8e85059d749020e3c4d2123719f23926 dataset.dataset_root=work/data/glm5/long-glm-remaining-20261007-175000

Expected result

Completed 20 BF16 Hybrid full-SFT steps with 128K synthetic packed sequences, GBS/MBS 56/1 and CP32 on 192 GB200 GPUs, with MTP1, HybridEP and natural routing. The exact runtime passed full-shape causal top-k reference checks with explicit cuDNN query offsets before training. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved the complete final model checkpoint and post-setup configuration.

LoRA · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

1.544981

Final loss

0.941183

Step time · last 10 avg

7,883.890 ms

Model throughput · last 10 avg

11.710 TFLOP/s/GPU

Token throughput · last 10 avg

43.295 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm5_peft_192gpu_gb200_bf16_config --mode lora --max_steps 100 --save_dir work/model-verification/glm5/hybrid/peft-fixedwidth-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 100 checkpoint.load=null checkpoint.save_optim=true checkpoint.save_rng=true logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5/hybrid/peft-fixedwidth-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5/hybrid/hf-4e6698ba8e85059d749020e3c4d2123719f23926 dataset.hf_output_root=work/data/glm5/peft-fixedwidth-glm-remaining-20261007-175000

Expected result

Completed 100 BF16 Hybrid LoRA steps on 192 GB200 GPUs with pinned Tulu 3 train[:10000], 2K offline packing, GBS/MBS 32/1, MTP1 and natural routing. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved adapter, optimizer, scheduler and RNG state, strictly reloaded the step-100 adapters with raise_all, and completed step 101 at zero LR. All 790 saved adapter tensors matched the reloaded checkpoint exactly; only the five configured MLA projection targets were adapted. Metrics cover training steps 1-100 only.