|
1 | | -# DeepSeek V2 |
| 1 | +# DeepSeek V2 support removed |
2 | 2 |
|
3 | | -> **Deprecation notice:** DeepSeek V2 and DeepSeek V2 Lite support is no |
4 | | -> longer actively maintained or tested against current upstream checkpoints and |
5 | | -> will be removed in Megatron Bridge 0.7.0. |
| 3 | +{/* Keep this notice for previews that use navigation from main. */} |
6 | 4 |
|
7 | | -[DeepSeek-V2](https://github.com/deepseek-ai/DeepSeek-V2) is a Mixture-of-Experts (MoE) language model that uses innovative Multi-head Latent Attention (MLA) for efficient inference and DeepSeekMoE architecture for economical training and inference. The model achieves performance comparable to GPT-4 while using significantly fewer activated parameters. More information is available in the companion paper ["DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model"](https://arxiv.org/abs/2405.04434). |
| 5 | +Support for DeepSeek V2 and V2 Lite has been removed from Megatron Bridge. The former conversion and training instructions no longer apply. |
8 | 6 |
|
9 | | -DeepSeek V2 models are supported via the Bridge system with auto-detected configuration and weight mapping. |
10 | | - |
11 | | -## Available Models |
12 | | - |
13 | | -Megatron Bridge supports the following DeepSeek V2 model variants: |
14 | | - |
15 | | -- **DeepSeek-V2**: 236B parameters (21B activated per token) |
16 | | -- **DeepSeek-V2-Lite**: 16B parameters (2.4B activated per token) |
17 | | - |
18 | | -Both models support pretraining with expert parallelism for efficient MoE training. |
19 | | - |
20 | | -## Model Architecture Features |
21 | | - |
22 | | -- **Multi-head Latent Attention (MLA)**: Novel attention mechanism that reduces KV cache requirements |
23 | | -- **DeepSeekMoE**: Efficient MoE architecture with routed and shared experts |
24 | | -- **Expert Parallelism**: Distributes experts across GPUs for scalable training |
25 | | -- **RoPE Embeddings**: Rotary position embeddings for position encoding |
26 | | -- **128K Context Length**: Native support for long sequences (DeepSeek-V2) |
27 | | -- **Pre-normalization**: RMSNorm before each transformer sub-layer |
28 | | - |
29 | | -## Conversion with 🤗 Hugging Face |
30 | | - |
31 | | -### Load HF → Megatron |
32 | | - |
33 | | -```python |
34 | | -from megatron.bridge import AutoBridge |
35 | | - |
36 | | -# Example: DeepSeek-V2-Lite |
37 | | -bridge = AutoBridge.from_hf_pretrained("deepseek-ai/DeepSeek-V2-Lite", trust_remote_code=True) |
38 | | -provider = bridge.to_megatron_provider() |
39 | | - |
40 | | -# Optionally configure parallelism before instantiating the model |
41 | | -provider.tensor_model_parallel_size = 1 |
42 | | -provider.pipeline_model_parallel_size = 1 |
43 | | -provider.expert_model_parallel_size = 8 |
44 | | - |
45 | | -model = provider.provide_distributed_model(wrap_with_ddp=False) |
46 | | -``` |
47 | | - |
48 | | -### Import Checkpoint from HF |
49 | | - |
50 | | -```bash |
51 | | -./scripts/conversion/convert.sh import \ |
52 | | - --hf-model deepseek-ai/DeepSeek-V2-Lite \ |
53 | | - --megatron-path /checkpoints/deepseek_v2_lite_megatron \ |
54 | | - --trust-remote-code |
55 | | -``` |
56 | | - |
57 | | -### Export Megatron → HF |
58 | | - |
59 | | -```python |
60 | | -from megatron.bridge import AutoBridge |
61 | | - |
62 | | -# Load the bridge from HF model ID |
63 | | -bridge = AutoBridge.from_hf_pretrained("deepseek-ai/DeepSeek-V2-Lite", trust_remote_code=True) |
64 | | - |
65 | | -# Export a trained Megatron checkpoint to HF format |
66 | | -bridge.export_ckpt( |
67 | | - megatron_path="/results/deepseek_v2_lite/checkpoints/iter_0000500", |
68 | | - hf_path="/exports/deepseek_v2_lite_hf", |
69 | | -) |
70 | | -``` |
71 | | - |
72 | | -### Run Inference on Converted Checkpoint |
73 | | - |
74 | | -```bash |
75 | | -uv run python examples/conversion/hf_to_megatron_generate_text.py \ |
76 | | - --hf_model_path deepseek-ai/DeepSeek-V2-Lite \ |
77 | | - --megatron_model_path /checkpoints/deepseek_v2_lite_megatron \ |
78 | | - --prompt "What is artificial intelligence?" \ |
79 | | - --max_new_tokens 100 \ |
80 | | - --ep 8 \ |
81 | | - --trust-remote-code |
82 | | -``` |
83 | | - |
84 | | -For more details, see [examples/conversion/hf_to_megatron_generate_text.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/examples/conversion/hf_to_megatron_generate_text.py) |
85 | | - |
86 | | -## Recipes |
87 | | - |
88 | | -See: [bridge.recipes.deepseek.deepseek_v2](../../apidocs/bridge/bridge.recipes.deepseek.deepseek_v2.md) |
89 | | - |
90 | | -### Available Recipes |
91 | | - |
92 | | -- **Pretrain recipes**: |
93 | | - - `deepseek_v2_lite_pretrain_config`: Pre-training for DeepSeek-V2-Lite (16B parameters, 2.4B activated per token) |
94 | | - - `deepseek_v2_pretrain_config`: Pre-training for DeepSeek-V2 (236B parameters, 21B activated per token) |
95 | | - |
96 | | -### Parallelism Configurations |
97 | | - |
98 | | -| Model | TP | PP | EP | Total GPUs | Use Case | |
99 | | -|-------|----|----|----|-----------:|----------| |
100 | | -| **DeepSeek-V2-Lite** | 1 | 1 | 8 | 8 | Pre-training (single node) | |
101 | | -| **DeepSeek-V2** | 1 | 4 | 32 | 128 | Pre-training (16 nodes) | |
102 | | - |
103 | | -**Key Features**: |
104 | | -- **Expert Parallelism**: EP=8 (V2-Lite) or EP=32 (V2) for efficient MoE training |
105 | | -- **Selective Recomputation**: Enabled by default for memory optimization |
106 | | -- **Sequence Length**: Default 4096, V2 supports up to 128K tokens |
107 | | - |
108 | | -### Pre-training Examples |
109 | | - |
110 | | -#### DeepSeek-V2-Lite (16B) |
111 | | - |
112 | | -```python |
113 | | -from megatron.bridge.recipes.deepseek import deepseek_v2_lite_pretrain_config |
114 | | - |
115 | | -config = deepseek_v2_lite_pretrain_config() |
116 | | -config.dataset.blend = [["/path/to/dataset_text_document"], None] |
117 | | -config.checkpoint.save = "/results/deepseek_v2_lite/checkpoints" |
118 | | -config.checkpoint.load = "/results/deepseek_v2_lite/checkpoints" |
119 | | -config.logger.tensorboard_dir = "/results/deepseek_v2_lite/tb_logs" |
120 | | -config.train.train_iters = 500_000 |
121 | | -config.train.global_batch_size = 512 |
122 | | -config.model.seq_length = 4096 |
123 | | -config.dataset.seq_length = 4096 |
124 | | -# Uses TP=1, PP=1, EP=8 (8 GPUs) automatically |
125 | | -``` |
126 | | - |
127 | | -#### DeepSeek-V2 (236B) |
128 | | - |
129 | | -```python |
130 | | -from megatron.bridge.recipes.deepseek import deepseek_v2_pretrain_config |
131 | | - |
132 | | -config = deepseek_v2_pretrain_config() |
133 | | -config.dataset.blend = [["/path/to/dataset_text_document"], None] |
134 | | -config.checkpoint.save = "/results/deepseek_v2/checkpoints" |
135 | | -config.checkpoint.load = "/results/deepseek_v2/checkpoints" |
136 | | -config.logger.tensorboard_dir = "/results/deepseek_v2/tb_logs" |
137 | | -config.train.train_iters = 500_000 |
138 | | -config.train.global_batch_size = 512 |
139 | | -config.model.seq_length = 4096 |
140 | | -config.dataset.seq_length = 4096 |
141 | | -# Uses TP=1, PP=4, EP=32 (128 GPUs) automatically |
142 | | -``` |
143 | | - |
144 | | -### Finetuning Recipes |
145 | | - |
146 | | -Finetuning recipes for DeepSeek V2 models are not currently available. |
147 | | - |
148 | | -## Hugging Face Model Cards & References |
149 | | - |
150 | | -### Hugging Face Model Cards |
151 | | -- DeepSeek-V2: https://huggingface.co/deepseek-ai/DeepSeek-V2 |
152 | | -- DeepSeek-V2-Lite: https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite |
153 | | -- DeepSeek-V2-Chat: https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat |
154 | | -- DeepSeek-V2-Lite-Chat: https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat |
155 | | - |
156 | | -### Technical Papers |
157 | | -- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model: [arXiv:2405.04434](https://arxiv.org/abs/2405.04434) |
158 | | - |
159 | | -### Additional Resources |
160 | | -- GitHub Repository: https://github.com/deepseek-ai/DeepSeek-V2 |
161 | | - |
162 | | -## Related Docs |
163 | | -- Recipe usage: [Recipe usage](../../recipe-usage.md) |
164 | | -- Customizing the training recipe configuration: [Configuration overview](../../training/config-container-overview.md) |
165 | | -- Training entry points: [Entry points](../../training/entry-points.md) |
| 7 | +See the [family overview](index.mdx) for supported models. |
0 commit comments