|
2 | 2 |
|
3 | 3 | Megatron Bridge is Python-first: configure models, data, and training via typed Python APIs. All configuration lives in a structured `ConfigContainer` (see [Configuration overview](training/config-container-overview.md)). Any field can be overridden from the command line using Hydra/OmegaConf syntax in the example training scripts. |
4 | 4 |
|
| 5 | +## Automated config translation script |
| 6 | + |
| 7 | +`scripts/translate_mlm_to_bridge.py` translates bidirectionally between Megatron-LM `pretrain_gpt.py` CLI arguments and Megatron Bridge `run_recipe.py` Hydra overrides. It is useful for running loss-correlation experiments between the two frameworks and for migrating existing MLM configs. |
| 8 | + |
| 9 | +### MLM → Bridge (default direction) |
| 10 | + |
| 11 | +```bash |
| 12 | +# From a YAML config file (MODEL_ARGS section) |
| 13 | +python scripts/translate_mlm_to_bridge.py --yaml model_configs/DeepSeek-V3.yaml |
| 14 | + |
| 15 | +# From inline CLI args |
| 16 | +python scripts/translate_mlm_to_bridge.py \ |
| 17 | + --args "--num-layers 32 --hidden-size 4096 --num-attention-heads 32 --bf16 --swiglu" |
| 18 | + |
| 19 | +# Emit a standalone Bridge recipe Python file (output goes to stdout; use -o to write to a file) |
| 20 | +python scripts/translate_mlm_to_bridge.py \ |
| 21 | + --yaml DeepSeek-V3.yaml --emit recipe --recipe-name deepseek_v3 |
| 22 | + |
| 23 | +# Write output to a file instead of stdout |
| 24 | +python scripts/translate_mlm_to_bridge.py \ |
| 25 | + --yaml DeepSeek-V3.yaml -o bridge_overrides.txt |
| 26 | +``` |
| 27 | + |
| 28 | +### Bridge → MLM (reverse direction) |
| 29 | + |
| 30 | +```bash |
| 31 | +# From a Bridge recipe name (defaults exported as MLM args) |
| 32 | +python scripts/translate_mlm_to_bridge.py --reverse \ |
| 33 | + --recipe llama32_1b_pretrain_config |
| 34 | + |
| 35 | +# From a recipe plus inline overrides |
| 36 | +python scripts/translate_mlm_to_bridge.py --reverse \ |
| 37 | + --recipe llama32_1b_pretrain_config \ |
| 38 | + --args "train.train_iters=1000 model.tensor_model_parallel_size=2" |
| 39 | + |
| 40 | +# From Bridge overrides only (no recipe) |
| 41 | +python scripts/translate_mlm_to_bridge.py --reverse \ |
| 42 | + --args "model.num_layers=32 model.activation_func=silu model.gated_linear_unit=true" |
| 43 | + |
| 44 | +# From a Bridge YAML/OmegaConf config file (e.g. exported ConfigContainer) |
| 45 | +python scripts/translate_mlm_to_bridge.py --reverse \ |
| 46 | + --yaml bridge_config.yaml |
| 47 | +``` |
| 48 | + |
| 49 | +### Key mappings |
| 50 | + |
| 51 | +| MLM flag | Bridge override | Notes | |
| 52 | +|---|---|---| |
| 53 | +| `--num-layers N` | `model.num_layers=N` | | |
| 54 | +| `--hidden-size N` | `model.hidden_size=N` | | |
| 55 | +| `--ffn-hidden-size N` | `model.ffn_hidden_size=N` | | |
| 56 | +| `--num-attention-heads N` | `model.num_attention_heads=N` | | |
| 57 | +| `--num-query-groups N` | `model.num_query_groups=N` | | |
| 58 | +| `--seq-length N` | `model.seq_length=N dataset.sequence_length=N` | Dual mapping | |
| 59 | +| `--swiglu` | `model.gated_linear_unit=true model.activation_func=silu` | Expanded to two keys | |
| 60 | +| `--squared-relu` | `model.activation_func=squared_relu` | | |
| 61 | +| `--data-path PATH [W PATH...]` | `dataset.data_path=PATH` | Space-separated paths (and optional weights) | |
| 62 | +| `--bf16` | `mixed_precision=bf16_mixed` | | |
| 63 | +| `--fp16` | `mixed_precision=16-mixed` | | |
| 64 | +| `--disable-bias-linear` | `model.add_bias_linear=false` | Inverted flag | |
| 65 | +| `--untie-embeddings-and-output-weights` | `model.share_embeddings_and_output_weights=false` | Inverted flag | |
| 66 | +| `--sequence-parallel` | `model.sequence_parallel=true` | | |
| 67 | +| `--tensor-model-parallel-size N` | `model.tensor_model_parallel_size=N` | | |
| 68 | +| `--pipeline-model-parallel-size N` | `model.pipeline_model_parallel_size=N` | | |
| 69 | +| `--context-parallel-size N` | `model.context_parallel_size=N` | | |
| 70 | +| `--micro-batch-size N` | `train.micro_batch_size=N` | | |
| 71 | +| `--global-batch-size N` | `train.global_batch_size=N` | | |
| 72 | +| `--train-iters N` | `train.train_iters=N` | | |
| 73 | +| `--lr LR` | `optimizer.lr=LR` | | |
| 74 | +| `--min-lr LR` | `optimizer.min_lr=LR` | | |
| 75 | +| `--weight-decay WD` | `optimizer.weight_decay=WD` | | |
| 76 | +| `--clip-grad CG` | `optimizer.clip_grad=CG` | | |
| 77 | +| `--lr-decay-style S` | `scheduler.lr_decay_style=S` | | |
| 78 | +| `--lr-warmup-iters N` | `scheduler.lr_warmup_iters=N` | | |
| 79 | +| `--seed S` | `rng.seed=S` | | |
| 80 | +| `--save PATH` | `checkpoint.save=PATH` | | |
| 81 | +| `--load PATH` | `checkpoint.load=PATH` | | |
| 82 | +| `--save-interval N` | `checkpoint.save_interval=N` | | |
| 83 | +| `--tokenizer-type T` | `tokenizer.tokenizer_type=T` | | |
| 84 | +| `--tokenizer-model M` | `tokenizer.tokenizer_model=M` | | |
| 85 | +| `--normalization N` | `model.normalization=N` | | |
| 86 | +| `--position-embedding-type T` | `model.position_embedding_type=T` | | |
| 87 | +| `--rotary-base N` | `model.rotary_base=N` | | |
| 88 | +| `--num-experts N` | `model.num_moe_experts=N` | | |
| 89 | +| `--moe-router-topk K` | `model.moe_router_topk=K` | | |
| 90 | +| `--mock-data` | `dataset.mock=true` | Use synthetic data (no files needed) | |
| 91 | + |
| 92 | +Flags not present in Bridge (e.g., `--use-mcore-models`, `--use-flash-attn`) are silently skipped with a comment. `--mock-data` translates to `dataset.mock=true`. Unknown flags are listed in a separate section so you can handle them manually. |
| 93 | + |
| 94 | +> **Activation function CLI overrides**: `model.activation_func` can now be set via Hydra CLI string override (e.g. `model.activation_func=silu`, `model.activation_func=gelu`). The string is resolved to the callable in `TransformerConfig.finalize()`. This makes `--swiglu` → `model.gated_linear_unit=true model.activation_func=silu` round-trippable from the CLI. |
| 95 | +
|
5 | 96 | ## Quick start |
6 | 97 |
|
7 | 98 | Run your example training entrypoint and override config keys directly: |
|
0 commit comments