You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4a92bd0
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: skills/nemo-mbridge-perf-activation-recompute/SKILL.md
+11-3Lines changed: 11 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ Activation recompute (activation checkpointing) trades additional forward work d
26
26
-**Grouped MoE:** start with `moe_act` when the expert intermediate activation dominates; add `layernorm` when norm outputs are material. Use whole `moe` recompute only after accounting for the extra expert compute and communication it replays.
27
27
-**Dense FFN:**`mlp` can save the whole dense-MLP activation region, but it usually costs more compute than a narrow output-discard boundary.
28
28
4. Change one label at a time. Record per-rank allocated/reserved peaks plus steady-state step time or throughput; do not infer a global module ranking from one recipe.
29
-
5. Full-layer recompute is allowed temporarily for diagnostic bring-up. Retain it in a final recipe only when target-scale measurements show that no recompute and supported selective boundaries still fail to fit; document the memory/OOM evidence, boundaries tested, and steady-state performance cost. Full recompute can severely reduce throughput and must not become the default from a bring-up success alone.
29
+
5. Full-layer recompute is allowed temporarily for diagnostic bring-up. Retain it in a final recipe only when target-scale measurements show that no recompute and supported selective boundaries still fail to fit, or matched runs show better end-to-end throughput with full recompute (for example, by enabling larger MBS or lower TP). Document the memory/OOM evidence, boundaries and layouts tested, and steady-state performance of the alternatives. Full recompute can severely reduce throughput and must not become the default from a bring-up success alone.
30
30
6. Treat CUDA graphs, FP8, context-parallel communication, and overlap features as compatibility constraints, not afterthoughts.
31
31
32
32
Megatron Core's `cpu_offloading=True` is an alternative when PCIe/NVLink transfer overhead is preferable to replayed compute. It cannot be combined with activation recompute and is not compatible with pipeline parallelism greater than one.
@@ -108,7 +108,14 @@ For every candidate, capture:
108
108
- steady-state step time or throughput after warmup;
109
109
- a short convergence or numerical-sanity check appropriate to the task.
110
110
111
-
Use a matched no-recompute control and change one recompute choice at a time. Peak memory from different jobs, backends, or parallel layouts is not a module-ranking benchmark.
111
+
Use a matched no-recompute control when it fits and change one recompute choice
112
+
at a time to isolate its effect. For a throughput-driven full-recompute choice,
113
+
also compare viable no-recompute/selective layouts with the full-recompute
114
+
layout, including any larger MBS or lower TP it enables. Keep hardware,
115
+
workload, precision, GBS, and convergence settings fixed; record all execution
116
+
changes and check loss. Compare useful tokens/s/GPU or complete step time,
117
+
not replayed FLOPs. Such a layout comparison supports the final recipe choice,
118
+
not a ranking of individual modules or checkpoint boundaries.
112
119
113
120
Do not call a candidate successful merely because it advances farther than the
114
121
control. Run through optimizer-state initialization and multiple steady-state
@@ -186,7 +193,8 @@ control, and cheap selective `moe_act` offered additional headroom at modest
Copy file name to clipboardExpand all lines: skills/nemo-mbridge-recipe-recommender/evals/evals.json
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -20,12 +20,12 @@
20
20
"question": "Use the nemo-mbridge-recipe-recommender skill. Give me a rule-of-thumb tuning loop for a Qwen MoE recipe on 8 versus 32 GPUs. Cover TP, EP, selective versus full recompute including Qwen3.5 GDN, HybridEP versus alltoall on NVL8 BF16, and what to do after the first config fits.",
"ground_truth": "The answer should use the lowest legal TP that still fits dense state and prefer recompute over raising TP solely for activation memory. It should use the largest legal EP supported by expert divisibility and both parallel meshes, with EP8 on one 8-GPU node and EP32 on 32 GPUs described as starting candidates rather than universal defaults. It must state that EP shards expert weights only, so EP8 can still require full recompute. It should prefer architecture-specific selective recompute; standard-attention Qwen can start with core_attn, while Qwen3.5 GDN needs the exact revision's gdn_norm_out boundary to be tested and can fall back to full recompute when the complete optimizer step still does not fit. It should keep alltoall as the correctness baseline and treat HybridEP as a matched candidate because NVL8 BF16 results can be close. After the run fits, it should inspect W&B memory and confirm every rank's peak allocated and reserved memory in runtime logs because rank 0 can miss the hot EP or PP rank, then A/B one lower-TP step or higher MBS while keeping GBS and optimizer boundaries fixed. Acceptance requires improved end-to-end tokens/s/GPU or step time, healthy loss, no skipped or NaN iterations, and memory margin for transients.",
23
+
"ground_truth": "The answer should use the lowest legal TP that still fits dense state and prefer recompute over raising TP solely for activation memory. It should use the largest legal EP supported by expert divisibility and both parallel meshes, with EP8 on one 8-GPU node and EP32 on 32 GPUs described as starting candidates rather than universal defaults. It must state that EP shards expert weights only and that EP8 alone does not justify full recompute. No recompute and cheap supported selective modules are equally valid starting choices; standard-attention Qwen can test core_attn, while Qwen3.5 GDN needs the exact revision's gdn_norm_out boundary tested. Full recompute may be temporary diagnostic bring-up, or retained with target-scale evidence that lighter choices fail to fit or matched viable layouts show better end-to-end throughput, including gains from larger MBS or lower TP. Record memory and steady-state performance with workload, hardware, precision, GBS, and convergence settings fixed. It should keep alltoall as the correctness baseline and treat HybridEP as a matched candidate because NVL8 BF16 results can be close. After the run fits, it should inspect W&B memory and confirm every rank's peak allocated and reserved memory in runtime logs because rank 0 can miss the hot EP or PP rank, then A/B one lower-TP step or higher MBS while keeping GBS and optimizer boundaries fixed. Acceptance requires improved end-to-end tokens/s/GPU or step time, healthy loss, no skipped or NaN iterations, and memory margin for transients.",
24
24
"expected_behavior": [
25
25
"Read the nemo-mbridge-recipe-recommender skill before answering.",
26
26
"Keep dense TP as low as capacity allows and make EP as large as the legal expert mesh permits.",
27
27
"Scope EP8 and EP32 as starting candidates and explain that EP does not shard dense or activation state.",
28
-
"Prefer selective recompute, distinguish standard Qwen attention from Qwen3.5 GDN, and retain full recompute as a measured fallback.",
28
+
"Treat no recompute and cheap selective modules as equally valid, distinguish standard Qwen attention from Qwen3.5 GDN, and require measured fit or matched throughput evidence to retain full recompute.",
29
29
"Use a matched alltoall control when testing HybridEP on NVL8 BF16.",
30
30
"After fit, use per-rank memory evidence to A/B lower TP or higher MBS with fixed GBS and explicit safety headroom."
0 commit comments