You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f4b705a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: skills/perf-hierarchical-context-parallel/SKILL.md
+24-17Lines changed: 24 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,15 +1,20 @@
1
1
---
2
-
name: perf-hybrid-context-parallel
2
+
name: perf-hierarchical-context-parallel
3
3
description: Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.
4
-
when_to_use: Scaling context parallelism beyond KV heads, or investigating a commit that changed CP config and caused OOM or a regression; 'hierarchical_context_parallel_sizes', 'a2a+p2p', 'hybrid context parallel', 'CP beyond KV heads', 'multi-level CP'.
4
+
when_to_use: Scaling context parallelism beyond KV heads, or investigating a commit that changed CP config and caused OOM or a regression; 'hierarchical_context_parallel_sizes', 'a2a+p2p', 'hierarchical CP', 'CP beyond KV heads', 'multi-level CP'.
5
5
---
6
6
7
-
# Hybrid / Hierarchical Context Parallel Skill
7
+
# Hierarchical Context Parallel Skill
8
8
9
-
For what HCP is, when to use it, and the decision tree (a2a+p2p vs pure a2a vs p2p), see:
9
+
This skill covers hierarchical context parallelism: nested context-parallel process
10
+
groups used by `cp_comm_type="a2a+p2p"` and configured with
11
+
`hierarchical_context_parallel_sizes`.
10
12
11
-
-@docs/training/hybrid-context-parallel.md
12
-
-@skills/perf-hybrid-context-parallel/card.yaml
13
+
For what hierarchical CP is, when to use it, and the decision tree
when HCP sizes are provided via `create_hierarchical_groups`. Bridge currently
118
+
gets those groups through the MPU-backed `ProcessGroupCollection`.
112
119
113
120
### TE integration
114
121
115
-
`TEDotProductAttention` passes the hierarchical groups to Transformer Engine when `a2a+p2p` is used. Requires **Transformer Engine >= 1.12.0**.
122
+
`TEDotProductAttention` passes the hierarchical groups to Transformer Engine
123
+
when `a2a+p2p` is used. Requires **Transformer Engine >= 1.12.0**.
116
124
117
125
## Pitfalls
118
126
119
-
1.**Different features**: `a2a+p2p` and upstream `hybrid_context_parallel=True` are different features. The latter is for balancing packed/variable-length workloads.
120
-
2.**Bridge HCP is MPU-only today**: If `use_decentralized_pg=True`, Bridge initializes flat CP groups and leaves HCP unset.
121
-
3.**No checked-in Bridge recipe** currently exercises HCP directly.
5.**Silent broken training**: If you use `a2a+p2p` without setting `hierarchical_context_parallel_sizes`, MCore now asserts. Older versions would silently disable CP communication — each rank attended only to its local chunk, producing artificially high throughput but completely broken gradients.
124
-
6.**Product must match**: `prod(hierarchical_context_parallel_sizes)` must exactly equal `context_parallel_size`. A mismatch triggers an assertion.
125
-
7.**Verify in logs**: Look for the process group initialization output. You should see `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created. If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active.
127
+
1.**Bridge HCP is MPU-only today**: If `use_decentralized_pg=True`, Bridge initializes flat CP groups and leaves HCP unset.
128
+
2.**No checked-in Bridge recipe** currently exercises HCP directly.
4.**Silent broken training on old stacks**: If you use `a2a+p2p` without setting `hierarchical_context_parallel_sizes`, MCore now asserts. Older versions would silently disable CP communication, so each rank attended only to its local chunk and produced artificially high throughput with broken gradients.
131
+
5.**Product must match**: `prod(hierarchical_context_parallel_sizes)` must exactly equal `context_parallel_size`. A mismatch triggers an assertion.
132
+
6.**Verify in logs**: Look for the process group initialization output. You should see `HIERARCHICAL_CONTEXT_PARALLEL_GROUPS` being created. If you only see `CONTEXT_PARALLEL_GROUP`, HCP is not active.
126
133
127
134
## Verification
128
135
129
-
No dedicated Bridge end-to-end test exists yet for HCP (see @skills/perf-hybrid-context-parallel/card.yaml
136
+
No dedicated Bridge end-to-end test exists yet for HCP (see @skills/perf-hierarchical-context-parallel/card.yaml
130
137
`follow_up_validation`). Use the existing unit tests and log inspection instead.
131
138
132
139
Run the decentralized-PG unit test to confirm the flat-CP behavior is preserved:
0 commit comments