Skip to content

Commit 4a92bd0

Browse files
committed
docs(recipe): scope pretraining readiness and address review feedback
Signed-off-by: Chen Cui <chcui@nvidia.com>
1 parent 6dd1bac commit 4a92bd0

6 files changed

Lines changed: 77 additions & 41 deletions

File tree

‎skills/create-model-verification-card/SKILL.md‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -156,8 +156,10 @@ recipe exists for that hardware, including one explicitly marked **early,
156156
unoptimized**. Its item status states whether the card's benchmark run has been
157157
verified; an `unverified` leaf still records the existence of the recipe.
158158
Existence or functional verification alone does not justify a tuned/optimized
159-
claim: require measured performance meeting the readiness guidance below and
160-
preserve any early, unoptimized label in the summary.
159+
claim: require workload-appropriate measured performance and preserve any
160+
early, unoptimized label in both the summary and the affected item's
161+
`expected_result` so generated pages display it. The library-pretraining
162+
readiness targets below are not benchmark-recipe acceptance thresholds.
161163
If the card has no concrete performance leaf,
162164
start `summary` with this exact disclaimer before the functional-support
163165
summary:
@@ -435,9 +437,12 @@ on cheap supported modules are both appropriate starting choices. Choose by
435437
measured memory headroom and steady-state performance; selective recompute
436438
does not require a no-recompute OOM first. Full recompute is acceptable
437439
temporarily for diagnostic bring-up, but can severely reduce throughput.
438-
Retain it only when target-scale measurements show no recompute and supported
439-
selective boundaries still fail to fit. Record memory/OOM
440-
evidence, boundaries tested, and steady-state performance cost in the recipe's
440+
Retain it only when target-scale measurements show lighter choices still fail
441+
to fit, or matched runs show better end-to-end throughput with full recompute,
442+
for example through larger MBS or lower TP. Keep the workload, GBS, and
443+
convergence settings fixed; validate recipe-owned MBS/layout changes in the
444+
recipe rather than overriding batch size in the card. Record memory evidence,
445+
boundaries/layouts tested, and steady-state performance in the recipe's
441446
supporting record; keep public card metrics standalone as required below.
442447

443448
Treat micro batch size and gradient-accumulation layout as execution
@@ -825,12 +830,16 @@ Optionally record `peak_allocated_memory_gib` and
825830
`peak_reserved_memory_gib`. They are required on verified `pretrain_fsdp`
826831
leaves, including every verified precision variant.
827832

828-
For any new recipe used by a card, apply the
829-
[performance-readiness guidance](../nemo-mbridge-recipe-recommender/SKILL.md#performance-readiness-for-new-recipes):
830-
aim for roughly 200 model TFLOP/s/GPU on H100 or 400 on GB200. Use the measured
831-
steady-state training metrics for the intended workload. If performance is
832-
severely below the target, withhold the recipe or mark it **early, unoptimized**
833-
in the recipe, card summary, and PR, with its measured performance and limits.
833+
For a new **pretraining library recipe** used by a card's `pretrain` item,
834+
apply the [performance-readiness guidance](../nemo-mbridge-recipe-recommender/SKILL.md#performance-readiness-for-new-recipes):
835+
for BF16, aim for roughly 200 model TFLOP/s/GPU on H100 or 400 on GB200 using
836+
steady-state metrics from the intended library workload. SFT, PEFT, and
837+
long-context SFT are excluded. A `pretrain_performance` benchmark result does
838+
not satisfy this library-recipe requirement. For other hardware/precisions,
839+
use a workload-matched measured target as described in that guidance.
840+
If performance is severely below the target, withhold the library recipe or
841+
mark it **early, unoptimized** in the recipe, card summary, affected `pretrain`
842+
item's `expected_result`, and PR, with its measured performance and limits.
834843
Functional verification alone does not establish that a recipe is optimized.
835844

836845
Record raw metrics for each run only. Do not store comparison payloads,

‎skills/nemo-mbridge-perf-activation-recompute/SKILL.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Activation recompute (activation checkpointing) trades additional forward work d
2626
- **Grouped MoE:** start with `moe_act` when the expert intermediate activation dominates; add `layernorm` when norm outputs are material. Use whole `moe` recompute only after accounting for the extra expert compute and communication it replays.
2727
- **Dense FFN:** `mlp` can save the whole dense-MLP activation region, but it usually costs more compute than a narrow output-discard boundary.
2828
4. Change one label at a time. Record per-rank allocated/reserved peaks plus steady-state step time or throughput; do not infer a global module ranking from one recipe.
29-
5. Full-layer recompute is allowed temporarily for diagnostic bring-up. Retain it in a final recipe only when target-scale measurements show that no recompute and supported selective boundaries still fail to fit; document the memory/OOM evidence, boundaries tested, and steady-state performance cost. Full recompute can severely reduce throughput and must not become the default from a bring-up success alone.
29+
5. Full-layer recompute is allowed temporarily for diagnostic bring-up. Retain it in a final recipe only when target-scale measurements show that no recompute and supported selective boundaries still fail to fit, or matched runs show better end-to-end throughput with full recompute (for example, by enabling larger MBS or lower TP). Document the memory/OOM evidence, boundaries and layouts tested, and steady-state performance of the alternatives. Full recompute can severely reduce throughput and must not become the default from a bring-up success alone.
3030
6. Treat CUDA graphs, FP8, context-parallel communication, and overlap features as compatibility constraints, not afterthoughts.
3131

3232
Megatron Core's `cpu_offloading=True` is an alternative when PCIe/NVLink transfer overhead is preferable to replayed compute. It cannot be combined with activation recompute and is not compatible with pipeline parallelism greater than one.
@@ -108,7 +108,14 @@ For every candidate, capture:
108108
- steady-state step time or throughput after warmup;
109109
- a short convergence or numerical-sanity check appropriate to the task.
110110

111-
Use a matched no-recompute control and change one recompute choice at a time. Peak memory from different jobs, backends, or parallel layouts is not a module-ranking benchmark.
111+
Use a matched no-recompute control when it fits and change one recompute choice
112+
at a time to isolate its effect. For a throughput-driven full-recompute choice,
113+
also compare viable no-recompute/selective layouts with the full-recompute
114+
layout, including any larger MBS or lower TP it enables. Keep hardware,
115+
workload, precision, GBS, and convergence settings fixed; record all execution
116+
changes and check loss. Compare useful tokens/s/GPU or complete step time,
117+
not replayed FLOPs. Such a layout comparison supports the final recipe choice,
118+
not a ranking of individual modules or checkpoint boundaries.
112119

113120
Do not call a candidate successful merely because it advances farther than the
114121
control. Run through optimizer-state initialization and multiple steady-state
@@ -186,7 +193,8 @@ control, and cheap selective `moe_act` offered additional headroom at modest
186193
cost. Nemotron required broad whole-`moe` recompute; historical dense Llama
187194
evidence found whole-`mlp` replay costly and lacked an empty control. Start with
188195
no recompute or cheap selective boundaries, choose by measured memory and
189-
performance, and broaden only as needed to pass the complete step.
196+
performance, and broaden only when measured capacity or end-to-end throughput
197+
benefits justify the replay cost.
190198

191199
## Compatibility and Validation
192200

‎skills/nemo-mbridge-perf-moe-vlm-training/SKILL.md‎

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -78,21 +78,26 @@ choices**. Choose using measured memory headroom and steady-state performance
7878
at target scale; selective recompute does not require a no-recompute OOM first.
7979
Full recompute can severely reduce throughput. It is acceptable temporarily
8080
during memory-constrained diagnostic bring-up, but a successful bring-up run
81-
does not justify publishing it as the default. Retain full recompute only when target-scale measurements show that
82-
no recompute and supported selective boundaries still fail to fit. Document
83-
the per-rank memory peaks/OOM evidence, boundaries tested, and steady-state
84-
performance cost.
81+
does not justify publishing it as the default. Retain full recompute only when
82+
target-scale measurements show that no recompute and supported selective
83+
boundaries still fail to fit, or matched runs show better end-to-end throughput
84+
with full recompute, for example through larger MBS or lower TP. Keep the
85+
workload, GBS, and convergence settings fixed. Document layout changes,
86+
per-rank memory peaks/OOM evidence, boundaries tested, and steady-state
87+
performance for the alternatives tested.
8588

8689
### Check performance before publishing
8790

88-
Every new recipe needs measured steady-state performance on its intended
89-
workload and allocation: aim for roughly **200 model TFLOP/s/GPU on H100** or
90-
**400 on GB200**. For VLMs, use realistic multimodal input. If performance is
91-
severely below these approximate targets, withhold the recipe or clearly mark
92-
it **early, unoptimized** in the recipe and its public documentation/card and
93-
PR. Follow the [recipe publication guidance](../nemo-mbridge-recipe-recommender/SKILL.md#performance-readiness-for-new-recipes)
94-
for the measurement details; a successful full-recompute bring-up does not
95-
waive this requirement.
91+
Every new **pretraining library recipe** needs measured steady-state
92+
performance on its intended workload and allocation. For BF16, aim for roughly
93+
**200 model TFLOP/s/GPU on H100** or **400 on GB200**. For VLMs, use realistic
94+
multimodal input. These targets assess the base library recipe, not a
95+
`perf_recipes` benchmark; SFT/PEFT and long-context SFT are excluded. If
96+
performance is severely below the target, withhold the recipe or clearly mark
97+
it **early, unoptimized** in the recipe, public documentation/card, and PR.
98+
Follow the [recipe publication guidance](../nemo-mbridge-recipe-recommender/SKILL.md#performance-readiness-for-new-recipes)
99+
for other hardware/precisions and visible card labels. A successful
100+
full-recompute bring-up does not waive this requirement.
96101

97102
## Key Tuning Knobs
98103

@@ -175,7 +180,7 @@ fits is the baseline; it is not automatically the performance recipe.
175180
TP=1 CP=1 PP=1
176181
EP sized to the expert topology, often large
177182
Dispatcher: HybridEP on GB200-class systems
178-
Recompute: none or cheap supported selective modules; measure memory and throughput
183+
Recompute: none or cheap selective by default; full only with measured fit/throughput justification
179184
```
180185

181186
### 3D-parallel GB200 path

‎skills/nemo-mbridge-perf-moe-vlm-training/card.yaml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,11 @@ validated_on: "2026-08-25"
33
summary: >
44
Training strategies for MoE vision-language models (VLMs), comparing FSDP
55
vs 3D Parallel approaches. Backed by empirical data from Qwen3-VL 235B
6-
and Qwen3.5-35B multimodal experiments on GB200 and B200.
6+
and Qwen3.5-35B multimodal experiments on GB200 and B200. The best_configs
7+
below are historical measured configurations, not default recommendations.
8+
Retain full recompute only with target-scale fit evidence or a matched
9+
throughput advantage over viable no-recompute/selective layouts; the
10+
historical rows alone do not establish that comparison.
711
812
validation_status:
913
qwen3_vl_fsdp_gb200:
@@ -26,8 +30,9 @@ training_dimensions:
2630
effect: "FSDP shards more aggressively; 3D Parallel often needs recompute"
2731
confidence: medium
2832
rationale: >
29-
3D Parallel with real vision data often needs full recompute for initial
30-
fit, then should be retested with supported selective modules.
33+
Full recompute can help diagnostic bring-up. For final recipes, measure
34+
no recompute or cheap selective modules and retain full recompute only
35+
for demonstrated capacity need or a matched end-to-end throughput gain.
3136
scale:
3237
effect: "Both approaches tested at 64–128 GPUs"
3338
confidence: medium

‎skills/nemo-mbridge-recipe-recommender/SKILL.md‎

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,24 @@ index details:
4444

4545
## Performance Readiness for New Recipes
4646

47-
Before publishing any new library, performance, or verification-card training
48-
recipe, measure steady-state model TFLOP/s/GPU on its intended workload and
49-
GPU allocation. Aim for roughly **200 TFLOP/s/GPU on H100** and **400
50-
TFLOP/s/GPU on GB200**. These are approximate readiness targets, not exact
51-
pass/fail cutoffs; completing a run alone does not establish decent performance.
47+
Before publishing a new **pretraining library recipe** under
48+
`src/megatron/bridge/recipes/`, measure steady-state model TFLOP/s/GPU on its
49+
intended workload and GPU allocation. For BF16, aim for roughly **200
50+
TFLOP/s/GPU on H100** and **400 TFLOP/s/GPU on GB200**. These are approximate
51+
readiness targets, not exact pass/fail cutoffs. This requirement excludes SFT,
52+
PEFT, and long-context SFT; it assesses the base pretraining library recipe,
53+
not a `perf_recipes` benchmark or its result. For other precisions or hardware
54+
such as B200/GB300, establish a workload-matched measured target rather than
55+
silently reusing the BF16 H100/GB200 numbers.
5256

5357
If measured performance is severely below the target, either withhold the
5458
recipe until it is improved or clearly label it **early, unoptimized** in the
5559
recipe docstring, accompanying public documentation/card, and PR description.
5660
Record the measured throughput, hardware, precision, workload, and GPU layout
57-
alongside the limitation. Do not present it as a tuned recommendation or change
58-
the intended training workload or convergence settings merely to meet a target.
61+
alongside the limitation. In a card, include the label in both the summary and
62+
the affected training item's `expected_result` so generated pages display it.
63+
Do not present it as a tuned recommendation or change the intended training
64+
workload or convergence settings merely to meet a target.
5965

6066
## Configuration Layers and Change Control
6167

@@ -282,8 +288,11 @@ For an MoE recipe on a fixed allocation, use this practical order:
282288
performance; selective recompute does not require a no-recompute OOM first.
283289
Full recompute is acceptable temporarily for diagnostic bring-up,
284290
but can severely reduce throughput. Retain it only when target-scale
285-
measurements show no recompute and supported selective boundaries still fail
286-
to fit; document memory/OOM evidence and steady-state performance cost. For
291+
measurements show that lighter choices still fail to fit, or matched runs
292+
show better end-to-end throughput with full recompute (for example, by
293+
enabling larger MBS or lower TP). Keep the workload, GBS, and convergence
294+
settings fixed; document layout changes, memory evidence, and steady-state
295+
performance for the alternatives tested. For
287296
Qwen3.5 GDN layers, verify `gdn_norm_out` on the exact Megatron Core revision
288297
instead of assuming `core_attn` covers the GDN peak.
289298
4. After a stable baseline completes optimizer initialization and several

‎skills/nemo-mbridge-recipe-recommender/evals/evals.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,12 @@
2020
"question": "Use the nemo-mbridge-recipe-recommender skill. Give me a rule-of-thumb tuning loop for a Qwen MoE recipe on 8 versus 32 GPUs. Cover TP, EP, selective versus full recompute including Qwen3.5 GDN, HybridEP versus alltoall on NVL8 BF16, and what to do after the first config fits.",
2121
"expected_skill": "nemo-mbridge-recipe-recommender",
2222
"expected_script": null,
23-
"ground_truth": "The answer should use the lowest legal TP that still fits dense state and prefer recompute over raising TP solely for activation memory. It should use the largest legal EP supported by expert divisibility and both parallel meshes, with EP8 on one 8-GPU node and EP32 on 32 GPUs described as starting candidates rather than universal defaults. It must state that EP shards expert weights only, so EP8 can still require full recompute. It should prefer architecture-specific selective recompute; standard-attention Qwen can start with core_attn, while Qwen3.5 GDN needs the exact revision's gdn_norm_out boundary to be tested and can fall back to full recompute when the complete optimizer step still does not fit. It should keep alltoall as the correctness baseline and treat HybridEP as a matched candidate because NVL8 BF16 results can be close. After the run fits, it should inspect W&B memory and confirm every rank's peak allocated and reserved memory in runtime logs because rank 0 can miss the hot EP or PP rank, then A/B one lower-TP step or higher MBS while keeping GBS and optimizer boundaries fixed. Acceptance requires improved end-to-end tokens/s/GPU or step time, healthy loss, no skipped or NaN iterations, and memory margin for transients.",
23+
"ground_truth": "The answer should use the lowest legal TP that still fits dense state and prefer recompute over raising TP solely for activation memory. It should use the largest legal EP supported by expert divisibility and both parallel meshes, with EP8 on one 8-GPU node and EP32 on 32 GPUs described as starting candidates rather than universal defaults. It must state that EP shards expert weights only and that EP8 alone does not justify full recompute. No recompute and cheap supported selective modules are equally valid starting choices; standard-attention Qwen can test core_attn, while Qwen3.5 GDN needs the exact revision's gdn_norm_out boundary tested. Full recompute may be temporary diagnostic bring-up, or retained with target-scale evidence that lighter choices fail to fit or matched viable layouts show better end-to-end throughput, including gains from larger MBS or lower TP. Record memory and steady-state performance with workload, hardware, precision, GBS, and convergence settings fixed. It should keep alltoall as the correctness baseline and treat HybridEP as a matched candidate because NVL8 BF16 results can be close. After the run fits, it should inspect W&B memory and confirm every rank's peak allocated and reserved memory in runtime logs because rank 0 can miss the hot EP or PP rank, then A/B one lower-TP step or higher MBS while keeping GBS and optimizer boundaries fixed. Acceptance requires improved end-to-end tokens/s/GPU or step time, healthy loss, no skipped or NaN iterations, and memory margin for transients.",
2424
"expected_behavior": [
2525
"Read the nemo-mbridge-recipe-recommender skill before answering.",
2626
"Keep dense TP as low as capacity allows and make EP as large as the legal expert mesh permits.",
2727
"Scope EP8 and EP32 as starting candidates and explain that EP does not shard dense or activation state.",
28-
"Prefer selective recompute, distinguish standard Qwen attention from Qwen3.5 GDN, and retain full recompute as a measured fallback.",
28+
"Treat no recompute and cheap selective modules as equally valid, distinguish standard Qwen attention from Qwen3.5 GDN, and require measured fit or matched throughput evidence to retain full recompute.",
2929
"Use a matched alltoall control when testing HybridEP on NVL8 BF16.",
3030
"After fit, use per-rank memory evidence to A/B lower TP or higher MBS with fixed GBS and explicit safety headroom."
3131
]

0 commit comments

Comments
 (0)