Skip to content

Commit 130e14d

Browse files
committed
refactor(model)!: remove Nemotron H v1 and deprecation checks
Signed-off-by: Chen Cui <chcui@nvidia.com>
1 parent 0e39526 commit 130e14d

44 files changed

Lines changed: 193 additions & 3193 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/model_support_request.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ body:
2929
- Dense decoder-only (e.g. Llama, Qwen, Gemma, Mistral, Nemotron)
3030
- MoE decoder-only (e.g. DeepSeek V3, OLMoE, Qwen3-MoE, MiniMax-M2)
3131
- VLM / multimodal (e.g. Qwen-VL, Gemma-VL, Nemotron-VL)
32-
- SSM / hybrid (e.g. Mamba, Nemotron-H)
32+
- SSM / hybrid (e.g. Mamba, Nemotron 3)
3333
- Encoder-decoder (e.g. T5)
3434
- Other / unsure
3535
validations:

‎README.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -221,8 +221,6 @@ For a deeper dive into conversion design and advanced usage, see the [models REA
221221

222222
Megatron Bridge provides out-of-the-box bridges and training recipes for a wide range of models, built on top of base model architectures from [Megatron Core](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core). Refer to the [models directory](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models) for the full list of model bridges.
223223

224-
> **Deprecation notice:** Nemotron H v1 (4B/8B/47B/56B) is no longer actively maintained and will be removed in Megatron Bridge 0.7.0.
225-
226224
| Family | Supported variants |
227225
|----------------|--------------------|
228226
| [**BAGEL**](docs/models/bagel/bagel.md) | BAGEL-7B-MoT (requires the tested MCore `dev` revision and `bagel` extra) |
@@ -244,7 +242,7 @@ Megatron Bridge provides out-of-the-box bridges and training recipes for a wide
244242
| [**Xiaomi-MiMo**](docs/models/mimo/index.md) | Xiaomi-MiMo, MiMo-V2-Flash |
245243
| [**Moonlight**](docs/models/moonlight/index.md) | Moonlight |
246244
| [**Muse Glimmer**](docs/models/muse_glimmer/muse-glimmer.md) | Muse Glimmer 30B |
247-
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron H v1 (deprecated), Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Nemotron-3 Nano Omni |
245+
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Ultra, Nemotron-3 Nano Omni |
248246
| [**OLMoE**](docs/models/olmoe/index.md) | OLMoE |
249247
| [**Qwen**](docs/models/qwen/index.md) | Qwen2 / Qwen2.5, Qwen3, Qwen3-MoE, Qwen3 Next, Qwen3.5 (dense/MoE), Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, Qwen3.6-VL, Qwen2 Audio, Qwen2.5-Omni, Qwen3-Omni, Qwen3-ASR |
250248
| [**Sarvam**](docs/models/sarvam/index.md) | Sarvam |

‎docs/fern/versions/_nav_order.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@
3636
"./versions/nightly/pages/models/kimi/kimi-k25-vl.mdx": 2
3737
"./versions/nightly/pages/models/llama/llama3.mdx": 2
3838
"./versions/nightly/pages/models/mistral/ministral3.mdx": 2
39-
"./versions/nightly/pages/models/nemotron/nemotronh.mdx": 2
4039
"./versions/nightly/pages/models/qwen/qwen3-moe.mdx": 4
4140
"./versions/nightly/pages/performance-guide.mdx": 2
4241
"./versions/nightly/pages/performance-summary-archive.mdx": 2

‎docs/fern/versions/nightly.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,6 @@ navigation:
112112
path: ./nightly/pages/models/moonlight/moonlight.mdx
113113
section: Moonlight
114114
- contents:
115-
- page: Nemotron H
116-
path: ./nightly/pages/models/nemotron/nemotronh.mdx
117115
- page: Nemotron 3 Nano 30B-A3B
118116
path: ./nightly/pages/models/nemotron/nemotron3-nano.mdx
119117
- page: Nemotron 3 Nano 4B

‎docs/fern/versions/nightly/pages/index.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,7 @@ Megatron Bridge provides out-of-the-box bridges and training recipes for a wide
217217
| [**Mistral**](docs/models/mistral/index.md) | Ministral 3 (3B/8B/14B) |
218218
| [**Xiaomi-MiMo**](docs/models/mimo/index.md) | Xiaomi-MiMo |
219219
| [**Moonlight**](docs/models/moonlight/index.md) | Moonlight |
220-
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron H, Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Nano Omni |
220+
| [**Nemotron**](docs/models/nemotron/index.md) | Nemotron-3 Nano, Nemotron-3 Super, Nemotron-3 Nano Omni |
221221
| [**OLMoE**](docs/models/olmoe/index.md) | OLMoE |
222222
| [**Qwen**](docs/models/qwen/index.md) | Qwen2 / Qwen2.5, Qwen3, Qwen3-MoE, Qwen3 Next, Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, Qwen3.6-VL, Qwen2 Audio, Qwen2.5-Omni, Qwen3-Omni, Qwen3-ASR |
223223
| [**Sarvam**](docs/models/sarvam/index.md) | Sarvam |

‎docs/fern/versions/nightly/pages/modelopt/quantization.mdx‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,6 @@ and ModelOpt scale suffixes, so a pattern can target the logical parameter name
188188
| Llama-3.2-1B | ✅ | ✅ |
189189
| Qwen3-8B | ✅ | ✅ |
190190
| Qwen3-30B-A3B | ✅ | ✅ |
191-
| Nemotron-H-8B-Base-8K | ✅ | ✅ |
192191
| Qwen3-VL-8B-Instruct | ✅ | ✅ |
193192
| Qwen3-VL-30B-A3B-Instruct | ✅ | ✅ |
194193

‎docs/fern/versions/nightly/pages/models/nemotron/index.mdx‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ Nemotron model documentation is organized by model variant.
44

55
| Variant | Guide |
66
|---------|-------|
7-
| Nemotron H | [nemotronh.md](nemotronh.md) |
87
| Nemotron-3 Nano 30B-A3B | [nemotron3-nano.md](nemotron3-nano.md) |
98
| Nemotron-3 Nano 4B | [nemotron3-nano-4b.md](nemotron3-nano-4b.md) |
109
| Nemotron-3 Super | [nemotron3-super.md](nemotron3-super.md) |
Lines changed: 4 additions & 310 deletions
Original file line numberDiff line numberDiff line change
@@ -1,313 +1,7 @@
1-
# Nemotron H
1+
# Nemotron H v1 support removed
22

3-
> **Deprecation notice:** Nemotron H v1 support (4B, 8B, 47B, and 56B) is no longer actively maintained or tested against current upstream checkpoints and will be removed in Megatron Bridge 0.7.0. Nemotron 3 is not deprecated.
3+
{/* Keep this notice for previews that use navigation from main. */}
44

5-
[Nemotron H](https://huggingface.co/collections/nvidia/nemotron-h) is a family of hybrid SSM-Attention models combining Mamba and attention layers.
5+
Support for Nemotron H v1 (4B, 8B, 47B, and 56B) has been removed from Megatron Bridge. The former conversion and training instructions no longer apply.
66

7-
The Nemotron H family includes models from 4B to 56B parameters with 8K context length.
8-
9-
## Model Families
10-
11-
### Nemotron H
12-
- **4B**: 52 layers, 3072 hidden size, 8K context
13-
- **8B**: 52 layers, 4096 hidden size, 8K context
14-
- **47B**: 98 layers, 8192 hidden size, 8K context
15-
- **56B**: 118 layers, 8192 hidden size, 8K context
16-
17-
## Model Architecture
18-
19-
### Common Features Across All Models
20-
- **Architecture**: Hybrid SSM-Attention (Mamba + Multi-Query Attention)
21-
- **SSM**: Mamba-2 selective state space layers
22-
- **Attention**: Multi-query attention with QK LayerNorm and RoPE
23-
- **Activation**: Squared ReLU (SwiGLU in FFN)
24-
- **Normalization**: RMSNorm
25-
- **Position Embedding**: RoPE (Rotary Position Embeddings)
26-
- **Hybrid Pattern**: Configurable layer-wise mixing of Mamba ("M") and Attention ("*") layers
27-
28-
### Nemotron H 4B Specifications
29-
- **Parameters**: 4B
30-
- **Layers**: 52 (Hybrid pattern: `M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-`)
31-
- **Hidden size**: 3072
32-
- **FFN hidden size**: 12288
33-
- **Attention heads**: 32 query heads, 8 key-value groups
34-
- **KV channels**: 128
35-
- **Mamba heads**: 112
36-
- **Mamba head dim**: 64
37-
- **Mamba state dim**: 128
38-
- **Context Length**: 8K tokens
39-
40-
### Nemotron H 8B Specifications
41-
- **Parameters**: 8B
42-
- **Layers**: 52 (Hybrid pattern: `M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-`)
43-
- **Hidden size**: 4096
44-
- **FFN hidden size**: 21504
45-
- **Attention heads**: 32 query heads, 8 key-value groups
46-
- **KV channels**: 128
47-
- **Mamba heads**: 128
48-
- **Mamba head dim**: 64
49-
- **Mamba state dim**: 128
50-
- **Context Length**: 8K tokens
51-
52-
### Nemotron H 47B Specifications
53-
- **Parameters**: 47B
54-
- **Layers**: 98
55-
- **Hidden size**: 8192
56-
- **FFN hidden size**: 30720
57-
- **Attention heads**: 64 query heads, 8 key-value groups
58-
- **KV channels**: 128
59-
- **Mamba heads**: 256
60-
- **Mamba head dim**: 64
61-
- **Mamba state dim**: 256
62-
- **Context Length**: 8K tokens
63-
64-
### Nemotron H 56B Specifications
65-
- **Parameters**: 56B
66-
- **Layers**: 118
67-
- **Hidden size**: 8192
68-
- **FFN hidden size**: 32768
69-
- **Attention heads**: 64 query heads, 8 key-value groups
70-
- **KV channels**: 128
71-
- **Mamba heads**: 256
72-
- **Mamba head dim**: 64
73-
- **Mamba state dim**: 256
74-
- **Context Length**: 8K tokens
75-
76-
## Key Features
77-
78-
### Hybrid SSM-Attention Architecture
79-
- **Mamba Layers (M)**: State space model layers for efficient long-range modeling
80-
- **Attention Layers (*)**: Standard multi-query attention for complex reasoning
81-
- **Configurable Pattern**: Each model has a predefined hybrid pattern balancing efficiency and performance
82-
83-
### Advanced Optimizations
84-
- **Squared ReLU Activation**: Enhanced non-linearity for better gradient flow
85-
- **QK LayerNorm**: Applies LayerNorm to query and key projections for training stability
86-
- **RoPE**: Rotary Position Embeddings with base 10000
87-
- **Multi-Query Attention**: Efficient attention with shared key-value heads
88-
- **Selective State Space**: Mamba-2 architecture with selective gating
89-
90-
## Conversion with 🤗 Hugging Face
91-
92-
### Load HF → Megatron
93-
94-
#### Nemotron H Models
95-
```python
96-
from megatron.bridge import AutoBridge
97-
98-
# Example: Nemotron H 8B
99-
bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-8B-Base-8K", trust_remote_code=True)
100-
provider = bridge.to_megatron_provider()
101-
102-
# Configure parallelism before instantiating the model
103-
provider.tensor_model_parallel_size = 2
104-
provider.pipeline_model_parallel_size = 1
105-
provider.context_parallel_size = 1
106-
provider.sequence_parallel = True
107-
108-
provider.finalize()
109-
model = provider.provide_distributed_model(wrap_with_ddp=False)
110-
111-
# Other models:
112-
# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-4B-Base-8K", trust_remote_code=True)
113-
# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-47B-Base-8K", trust_remote_code=True)
114-
# bridge = AutoBridge.from_hf_pretrained("nvidia/Nemotron-H-56B-Base-8K", trust_remote_code=True)
115-
```
116-
117-
### Export Megatron → HF
118-
```python
119-
# Convert from a Megatron checkpoint directory to HF format
120-
bridge.export_ckpt(
121-
megatron_path="/results/nemotronh_8b/checkpoints/iter_0500000",
122-
hf_path="./nemotronh-8b-hf-export",
123-
)
124-
```
125-
126-
## Examples
127-
128-
- Checkpoint conversion: [scripts/conversion/convert.sh](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/scripts/conversion/convert.sh)
129-
- Training scripts: [examples/models/train_any_basic.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/tutorials/recipes/llama/00_quickstart_pretrain.py)
130-
131-
## Finetuning Recipes
132-
133-
### Nemotron H 4B Finetuning
134-
135-
#### LoRA Finetuning
136-
```python
137-
from megatron.bridge.recipes.nemotronh import nemotronh_4b_peft_config
138-
139-
cfg = nemotronh_4b_peft_config(
140-
tokenizer_path="nvidia/Nemotron-H-4B-Base-8K",
141-
name="nemotronh_4b_lora",
142-
pretrained_checkpoint="path/to/nemotronh/4b/checkpoint",
143-
peft_scheme="lora", # or "dora" for DoRA
144-
train_iters=1000,
145-
global_batch_size=128,
146-
finetune_lr=1e-4,
147-
)
148-
```
149-
150-
#### Full Supervised Finetuning (SFT)
151-
```python
152-
from megatron.bridge.recipes.nemotronh import nemotronh_4b_sft_config
153-
154-
cfg = nemotronh_4b_sft_config(
155-
tokenizer_path="nvidia/Nemotron-H-4B-Base-8K",
156-
name="nemotronh_4b_sft",
157-
pretrained_checkpoint="path/to/nemotronh/4b/checkpoint",
158-
train_iters=1000,
159-
global_batch_size=128,
160-
finetune_lr=5e-6, # Lower LR for full SFT
161-
)
162-
```
163-
164-
### Nemotron H 8B Finetuning
165-
166-
```python
167-
from megatron.bridge.recipes.nemotronh import nemotronh_8b_peft_config
168-
169-
# LoRA finetuning
170-
cfg = nemotronh_8b_peft_config(
171-
tokenizer_path="nvidia/Nemotron-H-8B-Base-8K",
172-
name="nemotronh_8b_lora",
173-
pretrained_checkpoint="path/to/nemotronh/8b/checkpoint",
174-
peft_scheme="lora",
175-
train_iters=1000,
176-
global_batch_size=128,
177-
finetune_lr=1e-4,
178-
)
179-
```
180-
181-
### Nemotron H 47B Finetuning
182-
183-
```python
184-
from megatron.bridge.recipes.nemotronh import nemotronh_47b_peft_config
185-
186-
# LoRA finetuning (recommended for 47B)
187-
cfg = nemotronh_47b_peft_config(
188-
tokenizer_path="nvidia/Nemotron-H-47B-Base-8K",
189-
name="nemotronh_47b_lora",
190-
pretrained_checkpoint="path/to/nemotronh/47b/checkpoint",
191-
peft_scheme="lora",
192-
train_iters=1000,
193-
global_batch_size=128,
194-
finetune_lr=1e-4,
195-
)
196-
```
197-
198-
### Nemotron H 56B Finetuning
199-
200-
```python
201-
from megatron.bridge.recipes.nemotronh import nemotronh_56b_peft_config
202-
203-
# LoRA finetuning (recommended for 56B)
204-
cfg = nemotronh_56b_peft_config(
205-
tokenizer_path="nvidia/Nemotron-H-56B-Base-8K",
206-
name="nemotronh_56b_lora",
207-
pretrained_checkpoint="path/to/nemotronh/56b/checkpoint",
208-
peft_scheme="lora",
209-
train_iters=1000,
210-
global_batch_size=128,
211-
finetune_lr=1e-4,
212-
)
213-
```
214-
215-
## Default Configurations
216-
217-
### Nemotron H Models
218-
219-
#### 4B - LoRA (1 node, 8 GPUs)
220-
- TP=1, PP=1, CP=1, LR=1e-4
221-
- Sequence Parallel: False
222-
- Precision: BF16 mixed
223-
- Optimized for single-GPU finetuning
224-
225-
#### 4B - Full SFT (1 node, 8 GPUs)
226-
- TP=1, PP=1, CP=1, LR=5e-6
227-
- Sequence Parallel: False
228-
- Precision: BF16 mixed
229-
230-
#### 8B - LoRA (1 node, 8 GPUs)
231-
- TP=1, PP=1, CP=1, LR=1e-4
232-
- Sequence Parallel: False
233-
- Precision: BF16 mixed
234-
235-
#### 8B - Full SFT (1 node, 8 GPUs)
236-
- TP=2, PP=1, CP=1, LR=5e-6
237-
- Sequence Parallel: True
238-
- Precision: BF16 mixed
239-
240-
#### 47B - LoRA (2+ nodes)
241-
- TP=4, PP=1, CP=1, LR=1e-4
242-
- Sequence Parallel: False
243-
- Precision: FP8 hybrid (recommended)
244-
245-
#### 47B - Full SFT (4+ nodes)
246-
- TP=8, PP=1, CP=1, LR=5e-6
247-
- Sequence Parallel: True
248-
- Precision: FP8 hybrid
249-
250-
#### 56B - LoRA (2+ nodes)
251-
- TP=4, PP=1, CP=1, LR=1e-4
252-
- Sequence Parallel: False
253-
- Precision: FP8 hybrid (recommended)
254-
255-
#### 56B - Full SFT (4+ nodes)
256-
- TP=8, PP=1, CP=1, LR=5e-6
257-
- Sequence Parallel: True
258-
- Precision: FP8 hybrid
259-
260-
## API Reference
261-
262-
### Nemotron H
263-
- Nemotron H recipes: [bridge.recipes.nemotronh](../../apidocs/bridge/bridge.recipes.nemotronh.md)
264-
265-
## Performance Optimizations
266-
267-
### Memory Efficiency
268-
- **Selective Recomputation**: Reduces activation memory for larger models
269-
- **Sequence Parallelism**: Distributes sequence dimension across GPUs (enabled for 8B+)
270-
- **Manual GC**: Aggressive garbage collection for stable memory usage
271-
- **Precision-aware optimizer**: BF16/FP8 gradients with FP32 master weights
272-
273-
### Compute Efficiency
274-
- **Mamba-2 Optimizations**: Efficient selective state space computations
275-
- **Hybrid Architecture**: Balanced mix of Mamba and Attention layers
276-
- **Squared ReLU**: Efficient activation function with good gradient properties
277-
- **RoPE Fusion**: Optional optimization for position embeddings
278-
- **Multi-Query Attention**: Reduced KV cache memory and compute
279-
280-
### Hybrid Pattern Optimization
281-
The hybrid override pattern determines which layers use Mamba (M) vs Attention (*):
282-
- **Mamba layers**: Fast, memory-efficient, good for long-range dependencies
283-
- **Attention layers**: Better for complex reasoning and multi-token relationships
284-
- **Optimal patterns**: Pre-configured per model size based on extensive experimentation
285-
286-
## Pipeline Parallelism Layouts
287-
288-
Nemotron H models support several PP configurations with pre-defined layouts:
289-
- **PP=1**: No pipelining (default for most configurations)
290-
- **PP=2**: Supported with symmetric layer splits
291-
- **PP=4**: Supported for larger models (47B, 56B)
292-
- **VP (Virtual Pipeline)**: Supported for reducing pipeline bubbles
293-
294-
## Hugging Face Model Cards
295-
296-
### Nemotron H Models
297-
- **4B Base**: [nvidia/Nemotron-H-4B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-4B-Base-8K)
298-
- **8B Base**: [nvidia/Nemotron-H-8B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K)
299-
- **47B Base**: [nvidia/Nemotron-H-47B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-47B-Base-8K)
300-
- **56B Base**: [nvidia/Nemotron-H-56B-Base-8K](https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K)
301-
302-
## Technical Resources
303-
304-
### Research Papers
305-
- **Nemotron Technical Report**: [arXiv:2508.14444](https://arxiv.org/abs/2508.14444)
306-
- **Mamba-2**: [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060)
307-
308-
## Related Documentation
309-
310-
- Recipe usage and customization: [Recipe usage](../../recipe-usage.md)
311-
- Training configuration: [Configuration overview](../../training/config-container-overview.md)
312-
- Training entry points: [Entry points](../../training/entry-points.md)
313-
- PEFT methods (LoRA, DoRA): [PEFT Guide](../../training/peft.md)
7+
See the [family overview](index.mdx) for supported Nemotron 3 and 3.5 models.

‎docs/fern/versions/nightly/pages/nemo2-migration-guide.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Megatron Bridge supports the following model families through bridge mappings an
2929
- **Llama**: Llama3, Llama3.1, Llama3.2, CodeLlama, Llama4
3030
- **Qwen**: Qwen2, Qwen2.5, Qwen3, Qwen3MoE, Qwen2.5VL
3131
- **DeepSeek**: DeepSeek V3, DeepSeek V4
32-
- **Nemotron**: Nemotron 3, Nemotron 3.5, Nemotron H (deprecated)
32+
- **Nemotron**: Nemotron 3, Nemotron 3.5
3333
- **NVIDIA Mamba**: Mamba variants and hybrid models
3434

3535
For a complete list of bridge mappings, providers, and their parameters, see `bridge.models`.

0 commit comments

Comments
 (0)