Skip to content

Commit d5edbb4

Browse files
chenopischenopis
committed
docs(fern): fix MDX syntax errors blocking the build
fern docs dev failed to parse 3 files (defects fern check missed): - nemo2-to-megatron-bridge.mdx + training/performance-tuning.mdx: un-converted HTML comments (<!-- -->) -> MDX comment syntax ({/* */}). - training/hierarchical-context-parallel.mdx: raw <= read as a JSX tag -> escaped (\<=). Verified: all 124 MDX compile clean via @mdx-js/mdx batch compile. Signed-off-by: Andrew Chen <andrewch@nvidia.com>
1 parent 9c030ac commit d5edbb4

3 files changed

Lines changed: 24 additions & 24 deletions

File tree

‎docs/get-started/migration/nemo2-to-megatron-bridge.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Megatron Bridge supports the following model families through bridge mappings an
4141

4242
For a complete list of bridge mappings, providers, and their parameters, see `bridge.models`.
4343

44-
<!-- TODO: Create a dedicated model support table with tested HF checkpoint mappings -->
44+
{/* TODO: Create a dedicated model support table with tested HF checkpoint mappings */}
4545

4646
---
4747

‎docs/training/hierarchical-context-parallel.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ several CP communication backends:
2222
|---|---|---|---|
2323
| `"p2p"` | Ring-exchange of KV chunks | Yes | None |
2424
| `"all_gather"` | All-gather full KV before attention | No | None |
25-
| `"a2a"` | All-to-all: scatter heads, gather full sequence (Ulysses-style) | N/A | **CP <= num_kv_heads** |
25+
| `"a2a"` | All-to-all: scatter heads, gather full sequence (Ulysses-style) | N/A | **CP \<= num_kv_heads** |
2626
| `"a2a+p2p"` | Hierarchical: a2a within inner group, p2p across outer group | Partial (p2p part) | Requires `hierarchical_context_parallel_sizes` |
2727

2828
**HCP (`a2a+p2p`)** exists to scale CP beyond the KV head count by combining
@@ -70,7 +70,7 @@ Model: 8 KV heads. Cluster: 4 nodes x 8 GPUs. Goal: train 128K sequences.
7070
| Option | TP | CP | `cp_comm_type` | Notes |
7171
|---|---|---|---|---|
7272
| A | 1 | 16 | `a2a+p2p` with `[8,2]` | a2a intra-node (8 GPUs), p2p across 2 node-groups |
73-
| B | 2 | 4 | `a2a` | CP=4 <= 8 KV heads. Simpler. Often same throughput. |
73+
| B | 2 | 4 | `a2a` | CP=4 \<= 8 KV heads. Simpler. Often same throughput. |
7474
| C | 1 | 16 | `p2p` | Works but no a2a bandwidth benefit intra-node |
7575

7676
In practice, **option B is usually preferred** -- benchmarks showed identical

‎docs/training/performance-tuning.mdx‎

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -110,23 +110,23 @@ Additionally, because CP shards activations, it also partitions optimizer states
110110
> > 2. In FSDP training, activation storage remains as the main memory bottleneck because FSDP only shards model state memory, and a large per-GPU activation is needed to hide the costly FSDP communication. On GB200 GPUs, Megatron-Bridge offers an option to offload activations to the host memory via a high-speed chip-to-chip interconnect.
111111
> > 3. Baseline training is host performance-bound, but FSDP allows for larger per-GPU tensor sizes by eliminating TP or enabling a larger micro-batch size.
112112
113-
<!-- TODO: support megatron custom fsdp -->
114-
<!-- > 1. Megatron-Bridge supports two Fully Sharded Data Parallelism (FSDP) implementations: PyTorch-native FSDP and a custom Megatron FSDP built within Megatron Core. While both follow the same sharding principles, the custom implementation is further optimized for performance. The performance gain of the custom FSDP comes primarily from minimizing the data movement to the communication tensors and reusing communication buffers. Both FSDP methods can be used in combination with per-tensor sharding methods. -->
115-
<!-- > -->
116-
<!-- > > 1. To use PyTorch FSDP2: -->
117-
<!-- > > -->
118-
<!-- > > > 1. `DistributedInitConfig.use_torch_fsdp2=True` -->
119-
<!-- > > -->
120-
<!-- > > 2. To use Custom Megatron FSDP: -->
121-
<!-- > > -->
122-
<!-- > > > 1. `recipe.trainer.strategy.fsdp="megatron"` -->
123-
<!-- > > > 2. `recipe.trainer.strategy.ddp.data_parallel_sharding_strategy="optim_grads_params"` -->
124-
<!-- > -->
125-
<!-- > 2. FSDP can be preferred over TP+PP+DP mappings in the following scenarios: -->
126-
<!-- > -->
127-
<!-- > > 1. Small models with a large sequence, thus the parameter AllGather and gradient ReduceScatter can effectively be hidden under computation and the short communication overlap causes minor interference to the computation under overlap. -->
128-
<!-- > > 2. In FSDP training, activation storage remains as the main memory bottleneck because FSDP only shards model state memory, and a large per-GPU activation is needed to hide the costly FSDP communication. On GB200 GPUs, Megatron-Bridge offers an option to offload activations to the host memory via a high-speed chip-to-chip interconnect. -->
129-
<!-- > > 3. Baseline training is host performance-bound, but FSDP allows for larger per-GPU tensor sizes by eliminating TP or enabling a larger micro-batch size. -->
113+
{/* TODO: support megatron custom fsdp */}
114+
{/* > 1. Megatron-Bridge supports two Fully Sharded Data Parallelism (FSDP) implementations: PyTorch-native FSDP and a custom Megatron FSDP built within Megatron Core. While both follow the same sharding principles, the custom implementation is further optimized for performance. The performance gain of the custom FSDP comes primarily from minimizing the data movement to the communication tensors and reusing communication buffers. Both FSDP methods can be used in combination with per-tensor sharding methods. */}
115+
{/* > */}
116+
{/* > > 1. To use PyTorch FSDP2: */}
117+
{/* > > */}
118+
{/* > > > 1. `DistributedInitConfig.use_torch_fsdp2=True` */}
119+
{/* > > */}
120+
{/* > > 2. To use Custom Megatron FSDP: */}
121+
{/* > > */}
122+
{/* > > > 1. `recipe.trainer.strategy.fsdp="megatron"` */}
123+
{/* > > > 2. `recipe.trainer.strategy.ddp.data_parallel_sharding_strategy="optim_grads_params"` */}
124+
{/* > */}
125+
{/* > 2. FSDP can be preferred over TP+PP+DP mappings in the following scenarios: */}
126+
{/* > */}
127+
{/* > > 1. Small models with a large sequence, thus the parameter AllGather and gradient ReduceScatter can effectively be hidden under computation and the short communication overlap causes minor interference to the computation under overlap. */}
128+
{/* > > 2. In FSDP training, activation storage remains as the main memory bottleneck because FSDP only shards model state memory, and a large per-GPU activation is needed to hide the costly FSDP communication. On GB200 GPUs, Megatron-Bridge offers an option to offload activations to the host memory via a high-speed chip-to-chip interconnect. */}
129+
{/* > > 3. Baseline training is host performance-bound, but FSDP allows for larger per-GPU tensor sizes by eliminating TP or enabling a larger micro-batch size. */}
130130

131131
4. Heterogeneous Encoder Parallelism
132132

@@ -188,9 +188,9 @@ Additionally, because CP shards activations, it also partitions optimizer states
188188
> > > 1. `NVTE_FWD_LAYERNORM_SM_MARGIN=<#SM for DP collectives = 16>`
189189
> > > 2. `NVTE_BWD_LAYERNORM_SM_MARGIN=<#SM for DP collectives = 16>`
190190
191-
<!-- 2. Custom Megatron FSDP -->
191+
{/* 2. Custom Megatron FSDP */}
192192

193-
<!-- > 1. Unless you specify the communication bucket size, MCORE FSDP uses fixed communication overlap that overlaps the parameter AllGather and gradient ReduceScatter of each Transformer layer with its associated forward and backward computations. -->
193+
{/* > 1. Unless you specify the communication bucket size, MCORE FSDP uses fixed communication overlap that overlaps the parameter AllGather and gradient ReduceScatter of each Transformer layer with its associated forward and backward computations. */}
194194

195195
3. Tensor-parallel (TP) communication (with sequence parallelism)
196196

@@ -793,5 +793,5 @@ uv run python -u /home/dpsk_a2a/deepep/tests/test_internode.py
793793
- `TPOverlapCfg.fp8_buf`
794794
- `TPOverlapCfg.num_sm`
795795
- `TPOverlapCfg.num_split`
796-
<!-- - `garbageCollectionCallback.gc_interval_val` -->
797-
<!-- - `NsysPlugin` -->
796+
{/* - `garbageCollectionCallback.gc_interval_val` */}
797+
{/* - `NsysPlugin` */}

0 commit comments

Comments
 (0)