Skip to content

Commit 36c128d

Browse files
committed
[docs] rename hierarchical context parallel docs
Signed-off-by: Chen Cui <chcui@nvidia.com>
1 parent 31c49b5 commit 36c128d

9 files changed

Lines changed: 19 additions & 26 deletions

File tree

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ training/megatron-fsdp.md
5252
training/resiliency.md
5353
training/mixed-precision.md
5454
training/cuda-graphs.md
55-
training/hybrid-context-parallel.md
55+
training/hierarchical-context-parallel.md
5656
training/communication-overlap.md
5757
training/attention-optimizations.md
5858
training/activation-recomputation.md

‎docs/training/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ This directory contains comprehensive documentation for training and customizing
4141
| **[Optimizer & Scheduler](optimizer-scheduler.md)** | Optimizer and learning rate scheduler configuration | Setting up optimization |
4242
| **[Mixed Precision](mixed-precision.md)** | Mixed precision training for memory efficiency | Reducing memory usage |
4343
| **[Communication Overlap](communication-overlap.md)** | Overlapping communication with computation | Optimizing distributed training |
44-
| **[Hybrid Context Parallel](hybrid-context-parallel.md)** | Hierarchical `a2a+p2p` context parallel guidance | Advanced long-sequence scaling |
44+
| **[Hierarchical Context Parallel](hierarchical-context-parallel.md)** | Hierarchical `a2a+p2p` context parallel guidance | Advanced long-sequence scaling |
4545
| **[Attention Optimizations](attention-optimizations.md)** | Optimizing attention mechanisms | Improving training speed |
4646
| **[Activation Recomputation](activation-recomputation.md)** | Gradient checkpointing strategies | Reducing memory footprint |
4747
| **[CPU Offloading](cpu-offloading.md)** | Offloading to CPU for memory management | Working with limited GPU memory |

‎docs/training/communication-overlap.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ correct question is always "which communication path is exposed in this run?"
157157

158158
- [docs/performance-guide.md](../performance-guide.md)
159159
- [docs/training/cuda-graphs.md](cuda-graphs.md)
160-
- [docs/training/hybrid-context-parallel.md](hybrid-context-parallel.md)
160+
- [docs/training/hierarchical-context-parallel.md](hierarchical-context-parallel.md)
161161
- [skills/perf-tp-dp-comm-overlap/SKILL.md](../skills/perf-tp-dp-comm-overlap/SKILL.md)
162162
- [skills/perf-expert-parallel-overlap/SKILL.md](../skills/perf-expert-parallel-overlap/SKILL.md)
163163
- [skills/perf-moe-comm-overlap/SKILL.md](../skills/perf-moe-comm-overlap/SKILL.md)

docs/training/hybrid-context-parallel.md renamed to docs/training/hierarchical-context-parallel.md

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
1-
# Hybrid / Hierarchical Context Parallel
1+
# Hierarchical Context Parallel
22

3-
This page covers the stable Bridge-facing meaning of hierarchical context
4-
parallelism, especially the `a2a+p2p` transport path and
3+
This page covers hierarchical context parallelism in Megatron Bridge,
4+
especially the `a2a+p2p` transport path and
55
`hierarchical_context_parallel_sizes`.
66

77
For operational setup, code anchors, and verification commands, see
8-
[skills/perf-hybrid-context-parallel/SKILL.md](../skills/perf-hybrid-context-parallel/SKILL.md).
8+
[the hierarchical CP skill](../skills/perf-hybrid-context-parallel/SKILL.md).
99

1010
## What It Is
1111

@@ -24,11 +24,6 @@ several CP communication backends:
2424
a2a (fast, head-parallel) on intra-node links with p2p (async,
2525
sequence-parallel) on inter-node links.
2626

27-
It is important to separate this from the upstream boolean
28-
`hybrid_context_parallel`, which is a different feature for balancing packed or
29-
variable-length workloads. The two concepts should not be treated as
30-
interchangeable.
31-
3227
### Why a2a is limited by KV heads
3328

3429
a2a transposes the parallelism dimension: each rank trades its sequence chunk
@@ -109,5 +104,5 @@ all Bridge initialization modes.
109104

110105
- [docs/performance-guide.md](../performance-guide.md)
111106
- [docs/training/communication-overlap.md](communication-overlap.md)
112-
- [skills/perf-hybrid-context-parallel/SKILL.md](../skills/perf-hybrid-context-parallel/SKILL.md)
113-
- [skills/perf-hybrid-context-parallel/card.yaml](../skills/perf-hybrid-context-parallel/card.yaml)
107+
- [Hierarchical CP skill](../skills/perf-hybrid-context-parallel/SKILL.md)
108+
- [Hierarchical CP skill card](../skills/perf-hybrid-context-parallel/card.yaml)

‎docs/training/packed-sequences.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ problems:
7373
For long-sequence training guidance, see:
7474

7575
- `docs/performance-guide.md`
76-
- `docs/training/hybrid-context-parallel.md`
76+
- `docs/training/hierarchical-context-parallel.md`
7777

7878
## Practical Caveats
7979

@@ -89,6 +89,6 @@ The most stable caveats to remember are:
8989

9090
- [docs/training/multi-token-prediction.md](multi-token-prediction.md)
9191
- [docs/performance-guide.md](../performance-guide.md)
92-
- [docs/training/hybrid-context-parallel.md](hybrid-context-parallel.md)
92+
- [docs/training/hierarchical-context-parallel.md](hierarchical-context-parallel.md)
9393
- [skills/perf-sequence-packing/SKILL.md](../skills/perf-sequence-packing/SKILL.md)
9494
- [skills/perf-sequence-packing/card.yaml](../skills/perf-sequence-packing/card.yaml)

‎skills/perf-hybrid-context-parallel/SKILL.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,12 @@ when_to_use: Scaling context parallelism beyond KV heads, or investigating a com
88

99
This skill covers hierarchical context parallelism: nested context-parallel process
1010
groups used by `cp_comm_type="a2a+p2p"` and configured with
11-
`hierarchical_context_parallel_sizes`. Some file paths still contain `hybrid`
12-
as legacy naming; the feature described here is hierarchical CP.
11+
`hierarchical_context_parallel_sizes`.
1312

1413
For what hierarchical CP is, when to use it, and the decision tree
1514
(`a2a+p2p` vs pure `a2a` vs `p2p`), see:
1615

17-
- @docs/training/hybrid-context-parallel.md
16+
- @docs/training/hierarchical-context-parallel.md
1817
- @skills/perf-hybrid-context-parallel/card.yaml
1918

2019
## Enablement

‎skills/perf-hybrid-context-parallel/card.yaml‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
title: hybrid_context_parallel
1+
title: hierarchical_context_parallel
22
validated_on: "2026-03-14"
33
summary: >
44
Megatron-Bridge currently supports hierarchical context parallelism
@@ -27,9 +27,9 @@ feature_meaning:
2727
hierarchical_context_parallel_sizes: >
2828
Per-level subgroup sizes for hierarchical context parallelism. The product
2929
must equal context_parallel_size.
30-
hybrid_context_parallel_flag: >
31-
Separate upstream feature for balancing packed or variable-length workloads.
32-
It is not the same feature as a2a+p2p.
30+
inner_outer_cp_groups: >
31+
Hierarchical CP creates inner and outer context-parallel groups. With
32+
a2a+p2p, the inner group uses a2a and the outer group uses p2p.
3333
recommended_path:
3434
model.context_parallel_size: 4
3535
model.cp_comm_type: a2a+p2p
@@ -47,7 +47,6 @@ known_limitations:
4747
- The decentralized-PG path initializes flat CP groups and leaves HCP unset.
4848
- No checked-in Bridge recipe sets cp_comm_type=a2a+p2p.
4949
- No checked-in Bridge functional test runs an end-to-end HCP training step.
50-
- Bridge docs do not currently call out the decentralized-PG limitation.
5150
evidence:
5251
- src/megatron/bridge/training/initialize.py
5352
- src/megatron/bridge/training/config.py

‎skills/perf-parallelism-strategies/card.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ evidence:
6262
- docs/parallelisms.md
6363
- docs/performance-guide.md
6464
- docs/training/communication-overlap.md
65-
- docs/training/hybrid-context-parallel.md
65+
- docs/training/hierarchical-context-parallel.md
6666
- src/megatron/bridge/training/initialize.py
6767
- src/megatron/bridge/training/config.py
6868
- src/megatron/bridge/models/common/unimodal.py

‎skills/perf-sequence-packing/card.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ evidence:
6767
- docs/performance-guide.md
6868
- docs/training/multi-token-prediction.md
6969
- docs/models/llm/llama3.md
70-
- docs/training/hybrid-context-parallel.md
70+
- docs/training/hierarchical-context-parallel.md
7171
- src/megatron/bridge/data/datasets/packed_sequence.py
7272
- src/megatron/bridge/data/datasets/sft.py
7373
- src/megatron/bridge/training/utils/packed_seq_utils.py

0 commit comments

Comments
 (0)