|
1 | | -# Nemotron H |
| 1 | +# Nemotron H v1 support removed |
2 | 2 |
|
3 | | -> **Deprecation notice:** Nemotron H v1 support (4B, 8B, 47B, and 56B) is no longer actively maintained or tested against current upstream checkpoints and will be removed in Megatron Bridge 0.7.0. Nemotron 3 is not deprecated. |
| 3 | +{/* Keep this notice for previews that use navigation from main. */} |
4 | 4 |
|
5 | | -[Nemotron H](https://huggingface.co/collections/nvidia/nemotron-h) is a family of hybrid SSM-Attention models combining Mamba and attention layers. |
| 5 | +Support for Nemotron H v1 (4B, 8B, 47B, and 56B) has been removed from Megatron Bridge. The former conversion and training instructions no longer apply. |
6 | 6 |
|
7 | | -The Nemotron H family includes models from 4B to 56B parameters with 8K context length. |
8 | | - |
9 | | -## Model Families |
10 | | - |
11 | | -### Nemotron H |
12 | | -- **4B**: 52 layers, 3072 hidden size, 8K context |
13 | | -- **8B**: 52 layers, 4096 hidden size, 8K context |
14 | | -- **47B**: 98 layers, 8192 hidden size, 8K context |
15 | | -- **56B**: 118 layers, 8192 hidden size, 8K context |
16 | | - |
17 | | -## Model Architecture |
18 | | - |
19 | | -### Common Features Across All Models |
20 | | -- **Architecture**: Hybrid SSM-Attention (Mamba + Multi-Query Attention) |
21 | | -- **SSM**: Mamba-2 selective state space layers |
22 | | -- **Attention**: Multi-query attention with QK LayerNorm and RoPE |
23 | | -- **Activation**: Squared ReLU (SwiGLU in FFN) |
24 | | -- **Normalization**: RMSNorm |
25 | | -- **Position Embedding**: RoPE (Rotary Position Embeddings) |
26 | | -- **Hybrid Pattern**: Configurable layer-wise mixing of Mamba ("M") and Attention ("*") layers |
27 | | - |
28 | | -### Nemotron H 4B Specifications |
29 | | -- **Parameters**: 4B |
30 | | -- **Layers**: 52 (Hybrid pattern: `M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-`) |
31 | | -- **Hidden size**: 3072 |
32 | | -- **FFN hidden size**: 12288 |
33 | | -- **Attention heads**: 32 query heads, 8 key-value groups |
34 | | -- **KV channels**: 128 |
35 | | -- **Mamba heads**: 112 |
36 | | -- **Mamba head dim**: 64 |
37 | | -- **Mamba state dim**: 128 |
38 | | -- **Context Length**: 8K tokens |
39 | | - |
40 | | -### Nemotron H 8B Specifications |
41 | | -- **Parameters**: 8B |
42 | | -- **Layers**: 52 (Hybrid pattern: `M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-`) |
43 | | -- **Hidden size**: 4096 |
44 | | -- **FFN hidden size**: 21504 |
45 | | -- **Attention heads**: 32 query heads, 8 key-value groups |
46 | | -- **KV channels**: 128 |
47 | | -- **Mamba heads**: 128 |
48 | | -- **Mamba head dim**: 64 |
49 | | -- **Mamba state dim**: 128 |
50 | | -- **Context Length**: 8K tokens |
51 | | - |
52 | | -### Nemotron H 47B Specifications |
53 | | -- **Parameters**: 47B |
54 | | -- **Layers**: 98 |
55 | | -- **Hidden size**: 8192 |
56 | | -- **FFN hidden size**: 30720 |
57 | | -- **Attention heads**: 64 query heads, 8 key-value groups |
58 | | -- **KV channels**: 128 |
59 | | -- **Mamba heads**: 256 |
60 | | -- **Mamba head dim**: 64 |
61 | | -- **Mamba state dim**: 256 |
62 | | -- **Context Length**: 8K tokens |
63 | | - |
64 | | -### Nemotron H 56B Specifications |
65 | | -- **Parameters**: 56B |
66 | | -- **Layers**: 118 |
67 | | -- **Hidden size**: 8192 |
68 | | -- **FFN hidden size**: 32768 |
69 | | -- **Attention heads**: 64 query heads, 8 key-value groups |
70 | | -- **KV channels**: 128 |
71 | | -- **Mamba heads**: 256 |
72 | | -- **Mamba head dim**: 64 |
73 | | -- **Mamba state dim**: 256 |
74 | | -- **Context Length**: 8K tokens |
75 | | - |
76 | | -## Key Features |
77 | | - |
78 | | -### Hybrid SSM-Attention Architecture |
79 | | -- **Mamba Layers (M)**: State space model layers for efficient long-range modeling |
80 | | -- **Attention Layers (*)**: Standard multi-query attention for complex reasoning |
81 | | -- **Configurable Pattern**: Each model has a predefined hybrid pattern balancing efficiency and performance |
82 | | - |
83 | | -### Advanced Optimizations |
84 | | -- **Squared ReLU Activation**: Enhanced non-linearity for better gradient flow |
85 | | -- **QK LayerNorm**: Applies LayerNorm to query and key projections for training stability |
86 | | -- **RoPE**: Rotary Position Embeddings with base 10000 |
87 | | -- **Multi-Query Attention**: Efficient attention with shared key-value heads |
88 | | -- **Selective State Space**: Mamba-2 architecture with selective gating |
89 | | - |
90 | | -## Conversion with 🤗 Hugging Face |
91 | | - |
92 | | -### Load HF → Megatron |
93 | | - |
94 | | -#### Nemotron H Models |
95 | | -```python |
96 | | -from megatron.bridge import AutoBridge |
97 | | - |
98 | | -# Example: Nemotron H 8B |
99 | | -bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-8B-Base-8K", trust_remote_code=True) |
100 | | -provider = bridge.to_megatron_provider() |
101 | | - |
102 | | -# Configure parallelism before instantiating the model |
103 | | -provider.tensor_model_parallel_size = 2 |
104 | | -provider.pipeline_model_parallel_size = 1 |
105 | | -provider.context_parallel_size = 1 |
106 | | -provider.sequence_parallel = True |
107 | | - |
108 | | -provider.finalize() |
109 | | -model = provider.provide_distributed_model(wrap_with_ddp=False) |
110 | | - |
111 | | -# Other models: |
112 | | -# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-4B-Base-8K", trust_remote_code=True) |
113 | | -# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-47B-Base-8K", trust_remote_code=True) |
114 | | -# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-56B-Base-8K", trust_remote_code=True) |
115 | | -``` |
116 | | - |
117 | | -### Export Megatron → HF |
118 | | -```python |
119 | | -# Convert from a Megatron checkpoint directory to HF format |
120 | | -bridge.export_ckpt( |
121 | | - megatron_path="/results/nemotronh_8b/checkpoints/iter_0500000", |
122 | | - hf_path="./nemotronh-8b-hf-export", |
123 | | -) |
124 | | -``` |
125 | | - |
126 | | -## Examples |
127 | | - |
128 | | -- Checkpoint conversion: [scripts/conversion/convert.sh](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/scripts/conversion/convert.sh) |
129 | | -- Training scripts: [examples/models/train_any_basic.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/tutorials/recipes/llama/00_quickstart_pretrain.py) |
130 | | - |
131 | | -## Finetuning Recipes |
132 | | - |
133 | | -### Nemotron H 4B Finetuning |
134 | | - |
135 | | -#### LoRA Finetuning |
136 | | -```python |
137 | | -from megatron.bridge.recipes.nemotronh import nemotronh_4b_peft_config |
138 | | - |
139 | | -cfg = nemotronh_4b_peft_config( |
140 | | - tokenizer_path="nvidia/Nemotron-H-4B-Base-8K", |
141 | | - name="nemotronh_4b_lora", |
142 | | - pretrained_checkpoint="path/to/nemotronh/4b/checkpoint", |
143 | | - peft_scheme="lora", # or "dora" for DoRA |
144 | | - train_iters=1000, |
145 | | - global_batch_size=128, |
146 | | - finetune_lr=1e-4, |
147 | | -) |
148 | | -``` |
149 | | - |
150 | | -#### Full Supervised Finetuning (SFT) |
151 | | -```python |
152 | | -from megatron.bridge.recipes.nemotronh import nemotronh_4b_sft_config |
153 | | - |
154 | | -cfg = nemotronh_4b_sft_config( |
155 | | - tokenizer_path="nvidia/Nemotron-H-4B-Base-8K", |
156 | | - name="nemotronh_4b_sft", |
157 | | - pretrained_checkpoint="path/to/nemotronh/4b/checkpoint", |
158 | | - train_iters=1000, |
159 | | - global_batch_size=128, |
160 | | - finetune_lr=5e-6, # Lower LR for full SFT |
161 | | -) |
162 | | -``` |
163 | | - |
164 | | -### Nemotron H 8B Finetuning |
165 | | - |
166 | | -```python |
167 | | -from megatron.bridge.recipes.nemotronh import nemotronh_8b_peft_config |
168 | | - |
169 | | -# LoRA finetuning |
170 | | -cfg = nemotronh_8b_peft_config( |
171 | | - tokenizer_path="nvidia/Nemotron-H-8B-Base-8K", |
172 | | - name="nemotronh_8b_lora", |
173 | | - pretrained_checkpoint="path/to/nemotronh/8b/checkpoint", |
174 | | - peft_scheme="lora", |
175 | | - train_iters=1000, |
176 | | - global_batch_size=128, |
177 | | - finetune_lr=1e-4, |
178 | | -) |
179 | | -``` |
180 | | - |
181 | | -### Nemotron H 47B Finetuning |
182 | | - |
183 | | -```python |
184 | | -from megatron.bridge.recipes.nemotronh import nemotronh_47b_peft_config |
185 | | - |
186 | | -# LoRA finetuning (recommended for 47B) |
187 | | -cfg = nemotronh_47b_peft_config( |
188 | | - tokenizer_path="nvidia/Nemotron-H-47B-Base-8K", |
189 | | - name="nemotronh_47b_lora", |
190 | | - pretrained_checkpoint="path/to/nemotronh/47b/checkpoint", |
191 | | - peft_scheme="lora", |
192 | | - train_iters=1000, |
193 | | - global_batch_size=128, |
194 | | - finetune_lr=1e-4, |
195 | | -) |
196 | | -``` |
197 | | - |
198 | | -### Nemotron H 56B Finetuning |
199 | | - |
200 | | -```python |
201 | | -from megatron.bridge.recipes.nemotronh import nemotronh_56b_peft_config |
202 | | - |
203 | | -# LoRA finetuning (recommended for 56B) |
204 | | -cfg = nemotronh_56b_peft_config( |
205 | | - tokenizer_path="nvidia/Nemotron-H-56B-Base-8K", |
206 | | - name="nemotronh_56b_lora", |
207 | | - pretrained_checkpoint="path/to/nemotronh/56b/checkpoint", |
208 | | - peft_scheme="lora", |
209 | | - train_iters=1000, |
210 | | - global_batch_size=128, |
211 | | - finetune_lr=1e-4, |
212 | | -) |
213 | | -``` |
214 | | - |
215 | | -## Default Configurations |
216 | | - |
217 | | -### Nemotron H Models |
218 | | - |
219 | | -#### 4B - LoRA (1 node, 8 GPUs) |
220 | | -- TP=1, PP=1, CP=1, LR=1e-4 |
221 | | -- Sequence Parallel: False |
222 | | -- Precision: BF16 mixed |
223 | | -- Optimized for single-GPU finetuning |
224 | | - |
225 | | -#### 4B - Full SFT (1 node, 8 GPUs) |
226 | | -- TP=1, PP=1, CP=1, LR=5e-6 |
227 | | -- Sequence Parallel: False |
228 | | -- Precision: BF16 mixed |
229 | | - |
230 | | -#### 8B - LoRA (1 node, 8 GPUs) |
231 | | -- TP=1, PP=1, CP=1, LR=1e-4 |
232 | | -- Sequence Parallel: False |
233 | | -- Precision: BF16 mixed |
234 | | - |
235 | | -#### 8B - Full SFT (1 node, 8 GPUs) |
236 | | -- TP=2, PP=1, CP=1, LR=5e-6 |
237 | | -- Sequence Parallel: True |
238 | | -- Precision: BF16 mixed |
239 | | - |
240 | | -#### 47B - LoRA (2+ nodes) |
241 | | -- TP=4, PP=1, CP=1, LR=1e-4 |
242 | | -- Sequence Parallel: False |
243 | | -- Precision: FP8 hybrid (recommended) |
244 | | - |
245 | | -#### 47B - Full SFT (4+ nodes) |
246 | | -- TP=8, PP=1, CP=1, LR=5e-6 |
247 | | -- Sequence Parallel: True |
248 | | -- Precision: FP8 hybrid |
249 | | - |
250 | | -#### 56B - LoRA (2+ nodes) |
251 | | -- TP=4, PP=1, CP=1, LR=1e-4 |
252 | | -- Sequence Parallel: False |
253 | | -- Precision: FP8 hybrid (recommended) |
254 | | - |
255 | | -#### 56B - Full SFT (4+ nodes) |
256 | | -- TP=8, PP=1, CP=1, LR=5e-6 |
257 | | -- Sequence Parallel: True |
258 | | -- Precision: FP8 hybrid |
259 | | - |
260 | | -## API Reference |
261 | | - |
262 | | -### Nemotron H |
263 | | -- Nemotron H recipes: [bridge.recipes.nemotronh](../../apidocs/bridge/bridge.recipes.nemotronh.md) |
264 | | - |
265 | | -## Performance Optimizations |
266 | | - |
267 | | -### Memory Efficiency |
268 | | -- **Selective Recomputation**: Reduces activation memory for larger models |
269 | | -- **Sequence Parallelism**: Distributes sequence dimension across GPUs (enabled for 8B+) |
270 | | -- **Manual GC**: Aggressive garbage collection for stable memory usage |
271 | | -- **Precision-aware optimizer**: BF16/FP8 gradients with FP32 master weights |
272 | | - |
273 | | -### Compute Efficiency |
274 | | -- **Mamba-2 Optimizations**: Efficient selective state space computations |
275 | | -- **Hybrid Architecture**: Balanced mix of Mamba and Attention layers |
276 | | -- **Squared ReLU**: Efficient activation function with good gradient properties |
277 | | -- **RoPE Fusion**: Optional optimization for position embeddings |
278 | | -- **Multi-Query Attention**: Reduced KV cache memory and compute |
279 | | - |
280 | | -### Hybrid Pattern Optimization |
281 | | -The hybrid override pattern determines which layers use Mamba (M) vs Attention (*): |
282 | | -- **Mamba layers**: Fast, memory-efficient, good for long-range dependencies |
283 | | -- **Attention layers**: Better for complex reasoning and multi-token relationships |
284 | | -- **Optimal patterns**: Pre-configured per model size based on extensive experimentation |
285 | | - |
286 | | -## Pipeline Parallelism Layouts |
287 | | - |
288 | | -Nemotron H models support several PP configurations with pre-defined layouts: |
289 | | -- **PP=1**: No pipelining (default for most configurations) |
290 | | -- **PP=2**: Supported with symmetric layer splits |
291 | | -- **PP=4**: Supported for larger models (47B, 56B) |
292 | | -- **VP (Virtual Pipeline)**: Supported for reducing pipeline bubbles |
293 | | - |
294 | | -## Hugging Face Model Cards |
295 | | - |
296 | | -### Nemotron H Models |
297 | | -- **4B Base**: [nvidia/Nemotron-H-4B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-4B-Base-8K) |
298 | | -- **8B Base**: [nvidia/Nemotron-H-8B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K) |
299 | | -- **47B Base**: [nvidia/Nemotron-H-47B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-47B-Base-8K) |
300 | | -- **56B Base**: [nvidia/Nemotron-H-56B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K) |
301 | | - |
302 | | -## Technical Resources |
303 | | - |
304 | | -### Research Papers |
305 | | -- **Nemotron Technical Report**: [arXiv:2508.14444](https://arxiv.org/abs/2508.14444) |
306 | | -- **Mamba-2**: [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060) |
307 | | - |
308 | | -## Related Documentation |
309 | | - |
310 | | -- Recipe usage and customization: [Recipe usage](../../recipe-usage.md) |
311 | | -- Training configuration: [Configuration overview](../../training/config-container-overview.md) |
312 | | -- Training entry points: [Entry points](../../training/entry-points.md) |
313 | | -- PEFT methods (LoRA, DoRA): [PEFT Guide](../../training/peft.md) |
| 7 | +See the [family overview](index.mdx) for supported Nemotron 3 and 3.5 models. |
0 commit comments