glm5-2.md

GLM-5.2

GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3 use the shared GLM5Bridge for their MoE, Multi-Latent Attention, and Dynamic Sparse Attention architecture. GLM-5.3 shares GLM-5.2's architecture and import mappings; full-model GLM-5.3 verification remains pending.

GLM-5.2 / GLM-5.3 compatibility

The pinned publisher configs for GLM-5.2 and GLM-5.3 have identical architecture fields: 78 decoder layers, 256 routed experts with top-8 routing, MLA, and the same DSA IndexShare pattern. All 59,585 base checkpoint tensor names and shapes also match. AutoBridge resolves both to GlmMoeDsaForCausalLM and GLM5Bridge; no separate provider is needed. GLM-5.3-Flash is a different architecture and is outside this support statement.

The checkpoints have important differences:

• Precision: GLM-5.2 stores BF16 weights (plus FP32 router biases). GLM-5.3 stores 59,044 weights as E4M3 FP8 with FP32 scales per 128×128 block. The existing GLM bridge dequantizes these weights to BF16 on import. This is distinct from enabling FP8 training.
• Config metadata: GLM-5.3 adds quantization_config and records Transformers 5.15.0 instead of 5.12.0. Use the repository's supported dependency versions.
• Tokenizer and generation defaults: tokenizer.json, tokenizer_config.json, and generation_config.json are identical at those revisions, including token IDs and sampling defaults.
• Chat formatting: GLM-5.3's template adds low reasoning effort, retains historical reasoning by default, always opens a thinking response (it does not honor enable_thinking=False), and changes structured tool-result handling and ordering. Load the template from the GLM-5.3 checkpoint; do not substitute GLM-5.2's template.

For config/provider loading without weights:

from megatron.bridge import AutoBridge

bridge = AutoBridge.from_hf_pretrained(
    "zai-org/GLM-5.3", revision="aca966e4e02791568aa6a4ced368624b3d897f42"  # pragma: allowlist secret (public HF revision)
)
provider = bridge.to_megatron_provider(load_weights=False)

For BF16 export using the FP8 source as the reference, explicitly select --export-weight-dtype bfloat16 with the GPU or distributed-CPU conversion backend (API: weight_dtype=torch.bfloat16). This removes FP8 scale tensors from strict source-key validation and writes a BF16 artifact rather than claiming a bitwise FP8 round trip. The serial-CPU backend does not support this option. MTP is disabled by default; the appended MTP layer is outside that default inference graph.

Verification limits

The recorded commands and results below belong to the checkpoint named in each verification card. GLM-5.2 results do not establish GLM-5.3 full-checkpoint conversion/export, forward parity, generation, SFT/PEFT, resume, long-context behavior, or performance. Those GLM-5.3 checks remain unverified. The glm52_* recipes are pinned to GLM-5.2 and retain that name and attribution; changing only their display name would not select GLM-5.3 weights or its chat template.

Production qualification tracker #5476 remains open. It includes repeated-MTP training semantics, precision controls, dynamic context parallelism, production recipes, checkpoint continuity, convergence, and hardware-specific performance qualification. See also the open export context-length fix #5996, router-bias fix #6036, and recipe-selection fix #5608.

HybridModel migration

The shared GLM-5/5.1/5.2/5.3 bridge constructs HybridModel through GLM5ModelProvider. The outer language model is Core's native HybridModel; GLM-specific runtime adaptations are supplied through the stack specification. Each HF block becomes a D- (dense) or DE (MoE) pair; 78 HF blocks therefore become 156 physical Hybrid layers. A thin GLMDSAttention subclass reads the per-forward index-sharing state from a context variable owned by the GLM stage; the stack translates the HF index-sharing cadence into physical layer coordinates on private per-layer config copies. HF export retains the original layer count, and MTP uses a separate /DE pattern when enabled.

Import the pinned HF checkpoint into a new Hybrid checkpoint directory. The generic Core GPT-to-Hybrid converter does not support this DSA architecture; old GPT optimizer/checkpoint resume is outside this migration. get_model_config() is not supported for this family yet; use to_megatron_provider().

The initial recipes use BF16 and non-VPP pipeline segments. Pattern boundaries must begin on DSA compute layers. Full recomputation replays an entire pipeline stage to preserve forward-local top-k sharing, so memory and throughput must be remeasured. Selective core_attn recomputation is rejected because its backward replay does not preserve the forward-local DSA sharing state. Verified Hybrid conversion, forward correlation, inference, and GB200 training/resume results, including MTP and packed CP, are recorded below. Historical GPT results do not verify Hybrid.

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import & Export Pretrain Benchmark SFT LoRA Long Context

Precision All BF16 FP8 MX NVFP4

GPU All GB200

Import · CPU ✓ Verified BF16 Import · GPU ✓ Verified BF16 Export · CPU ✓ Verified BF16 Export · GPU ✓ Verified BF16 Pretrain · GB200 ✓ Verified BF16 SFT · GB200 ✓ Verified BF16 Long Context · GB200 ✓ Verified BF16 LoRA · GB200 ✓ Verified BF16

Import · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-22

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 4 --cpu-processes-per-node 8 --cpus-per-task 16 --mem 0 --exclusive --hf-model zai-org/GLM-5.2 --hf-revision 4d67f66cc64d3219133b767c253b2ad1425c6c88 --megatron-path work/model-verification/glm5-2/hybrid/cpu-megatron --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 2 --distributed-timeout-minutes 240

Expected result

Distributed CPU HF-to-Hybrid import with 32 Gloo processes on the host memory of 4 GB200 nodes maps all 5916 pinned source tensors through GLM5Bridge and saves a 1.4 TB torch_dist checkpoint (iter_0000000 with run_config.yaml) in about 14 minutes. The checkpoint is strictly reloadable, as exercised by a distributed CPU export of it whose 58794 tensors match the pinned source keys, shapes, dtypes and values exactly (the 791 tensors of the appended MTP layer, model.layers.78.*, are excluded because MTP is disabled by default). Weights stay on CPU, but Megatron-Core's CUDA RNG tracker still requires a visible CUDA device on each node during model-parallel initialization, so the run cannot execute on GPU-less nodes. Attention, dense and shared-expert weights are replicated across the 16 expert ranks of each pipeline stage, so peak host memory reached about 930 GB of the 952 GB per node (up to 177 GB for a single process); a single-process CPU import would need more than 1.5 TB on one host.

Import · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model zai-org/GLM-5.2 --hf-revision 4d67f66cc64d3219133b767c253b2ad1425c6c88 --megatron-path work/model-verification/glm5-2/hybrid/gpu-megatron --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 1 --distributed-timeout-minutes 60 --low-memory-save

Expected result

Distributed HF-to-Hybrid import on 16 GB200 GPUs maps all 5916 pinned source tensors through GLM5Bridge (shared-indexer layers carry no indexer tensors) and saves a 1.4 TB distributed checkpoint (iter_0000000 with run_config.yaml) in about 12 minutes. The checkpoint is strictly reloadable, as exercised by the GPU export. The appended MTP layer stays disabled by default.

Export · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 4 --cpu-processes-per-node 8 --cpus-per-task 16 --mem 0 --exclusive --hf-model zai-org/GLM-5.2 --hf-revision 4d67f66cc64d3219133b767c253b2ad1425c6c88 --megatron-path work/model-verification/glm5-2/hybrid/gpu-megatron/iter_0000000 --hf-path work/model-verification/glm5-2/hybrid/cpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 2 --distributed-timeout-minutes 240 --distributed-save --save-every-n-ranks 1 --no-progress

Expected result

Distributed CPU Hybrid-to-HF export with 32 Gloo processes on the host memory of 4 GB200 nodes strictly reloads the imported checkpoint and writes 279 safetensors shards plus model.safetensors.index.json in about 30 minutes. All 58794 exported tensors match the pinned source keys, shapes, dtypes and values exactly; the 791 tensors of the appended MTP layer (model.layers.78.*) are excluded because MTP is disabled by default, and shared-indexer layers carry no indexer tensors on either side. Weights stay on CPU, but Megatron-Core's CUDA RNG tracker still requires a visible CUDA device on each node during model-parallel initialization, so the run cannot execute on GPU-less nodes. generation_config.json is preserved from the source because Transformers strict validation rejects re-saving it.

Export · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-09-19

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model zai-org/GLM-5.2 --hf-revision 4d67f66cc64d3219133b767c253b2ad1425c6c88 --megatron-path work/model-verification/glm5-2/hybrid/gpu-megatron/iter_0000000 --hf-path work/model-verification/glm5-2/hybrid/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 2 --ep 8 --etp 1 --distributed-timeout-minutes 60 --distributed-save --save-every-n-ranks 1

Expected result

Distributed Hybrid-to-HF export on 16 GB200 GPUs strictly reloads the imported checkpoint and writes 279 safetensors shards plus model.safetensors.index.json in about 13 minutes. All 58794 exported tensors match the pinned source keys, shapes, dtypes and values exactly; the 791 tensors of the appended MTP layer (model.layers.78.*) are excluded because MTP is disabled by default, and shared-indexer layers carry no indexer tensors on either side. generation_config.json is preserved from the source because Transformers strict validation rejects re-saving it.

Pretrain · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

1.039136

Final loss

0.8746876

Step time · last 10 avg

32,331.530 ms

Model throughput · last 10 avg

219.080 TFLOP/s/GPU

Token throughput · last 10 avg

675.667 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm52_pretrain_192gpu_gb200_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 20 --lr 3e-6 --min_lr 3e-7 --warmup_iters 10 --pretrained_checkpoint work/model-verification/glm5-2/hybrid/gpu-megatron-mtp-20261007-135936/iter_0000000 'dataset.blend=[["work/data/wikitext103/wikitext103_glm52_text_document"],null]' dataset.path_to_cache=work/cache/glm5-2/wikitext-glm52-gb200 dataset.random_seed=1234 dataset.num_workers=8 tokenizer.use_tokenizer_vocab_size=false rng.seed=1234 scheduler.lr_decay_iters=20 model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.save_optim=true checkpoint.save_rng=true checkpoint.load_optim=true checkpoint.load_rng=true checkpoint.finetune=false --save_dir work/model-verification/glm5-2/hybrid/pretrain-wikitext-gb200-20261007-135936/checkpoints --save_interval 10 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5-2/hybrid/pretrain-wikitext-gb200-20261007-135936/resolved-config.yaml dist.distributed_timeout_minutes=60

Expected result

Completed 20 BF16 Hybrid warm-start steps on 192 GB200 GPUs with WikiText-103 raw train (29444 articles, 120372951 tokens), tokenized with the pinned GLM-5.2 tokenizer. Used 4K sequences, GBS/MBS 1024/1, MTP1, all-to-all expert dispatch and natural routing. LR warmed up to 3e-6 at step 10 and decayed to 3e-7 at step 20. Saved full step-10 and step-20 checkpoints with optimizer, scheduler and RNG state, plus the post-setup config. All losses and gradients were finite with zero skipped or NaN iterations; the step-10 checkpoint passed the separate resume comparison.

SFT · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

1.913098

Final loss

0.7940679

Step time · last 10 avg

6,738.850 ms

Model throughput · last 10 avg

17.140 TFLOP/s/GPU

Token throughput · last 10 avg

50.652 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm52_gb200_sft_config --mode sft --max_steps 100 --save_dir work/model-verification/glm5-2/hybrid/sft-causal-fixed-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 100 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5-2/hybrid/sft-causal-fixed-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5-2/hybrid/hf-4d67f66cc64d3219133b767c253b2ad1425c6c88 dataset.hf_output_root=work/data/glm5-2/tulu3-sft-20261006-142455

Expected result

Completed 100 BF16 Hybrid full-SFT steps with pinned Tulu 3 train[:10000], 8K packing, GBS/MBS 8/1 and CP4 on 192 GB200 GPUs, with MTP1, HybridEP and natural routing. The exact runtime passed full-shape causal top-k reference checks with explicit cuDNN query offsets before training. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved the complete final model checkpoint and post-setup configuration. Strictly reloaded the step-100 checkpoint with raise_all and completed a further training step. Metrics cover the original 100 steps only.

Long Context · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-08

Recorded metrics

Initial loss

3.501637

Final loss

3.345199

Step time · last 10 avg

115,202.340 ms

Model throughput · last 10 avg

110.510 TFLOP/s/GPU

Token throughput · last 10 avg

331.845 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm52_sft_192gpu_gb200_bf16_128k_config --mode sft --max_steps 20 --save_dir work/model-verification/glm5-2/hybrid/long-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 20 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5-2/hybrid/long-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5-2/hybrid/hf-4d67f66cc64d3219133b767c253b2ad1425c6c88 dataset.dataset_root=work/data/glm5-2/long-glm-remaining-20261007-175000

Expected result

Completed 20 BF16 Hybrid full-SFT steps with 128K synthetic packed sequences, GBS/MBS 56/1 and CP32 on 192 GB200 GPUs, with MTP1, HybridEP and natural routing. The exact runtime passed full-shape causal top-k reference checks with explicit cuDNN query offsets before training. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved the complete final model checkpoint and post-setup configuration.

LoRA · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-07

Recorded metrics

Initial loss

1.903576

Final loss

0.8791012

Step time · last 10 avg

4,322.970 ms

Model throughput · last 10 avg

23.030 TFLOP/s/GPU

Token throughput · last 10 avg

78.958 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 48 --gpus-per-node 4 --recipe glm52_peft_192gpu_gb200_bf16_config --mode lora --max_steps 100 --save_dir work/model-verification/glm5-2/hybrid/peft-gb200-glm-remaining-20261007-175000/checkpoints --save_interval 100 checkpoint.load=null checkpoint.save_optim=true checkpoint.save_rng=true logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/glm5-2/hybrid/peft-gb200-glm-remaining-20261007-175000/launch-config.yaml dist.distributed_timeout_minutes=60 --pretrained_checkpoint work/model-verification/glm5-2/hybrid/hf-4d67f66cc64d3219133b767c253b2ad1425c6c88 dataset.hf_output_root=work/data/glm5-2/peft-glm-remaining-20261007-175000

Expected result

Completed 100 BF16 Hybrid LoRA steps on 192 GB200 GPUs with pinned Tulu 3 train[:10000], 2K offline packing, GBS/MBS 32/1, MTP1 and natural routing. All LM/MTP losses and gradients were finite with zero skipped or NaN iterations. Saved adapter, optimizer, scheduler and RNG state, strictly reloaded the step-100 adapters with raise_all, and completed step 101 at zero LR. All 790 saved adapter tensors matched the reloaded checkpoint exactly; only the five configured MLA projection targets were adapted. Metrics cover training steps 1-100 only.