diff --git a/docs/skills-index.md b/docs/skills-index.md index 0b191667d3..49b5461b13 100644 --- a/docs/skills-index.md +++ b/docs/skills-index.md @@ -50,6 +50,7 @@ skills/nemo-mbridge-perf-tp-dp-comm-overlap/SKILL skills/nemo-mbridge-perf-megatron-fsdp/SKILL skills/nemo-mbridge-perf-sequence-packing/SKILL skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL +skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL skills/nemo-mbridge-perf-expert-parallel-overlap/SKILL skills/nemo-mbridge-perf-moe-comm-overlap/SKILL skills/nemo-mbridge-perf-activation-recompute/SKILL diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index a66d6b9e6e..eb5e3ae43d 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -257,12 +257,41 @@ Set the four flags below as a matched column — a mismatched set produces incor | Field | Dynamic resolution (images, variable H×W) | Temporal video (videos, fused pairs, 512²) | |----------------------------------------------------|-------------------------------------------|--------------------------------------------| | `dataset.task_encoder.use_temporal_video_embedder` | `False` | `True` | -| `model.temporal_patchrecipe_dim` | `1` | `2` | +| `model.temporal_patch_dim` | `1` | `2` | | `model.separate_video_embedder` | `False` | `True` | | `model.temporal_ckpt_compat` | `False` | `True` | Note:`dataset.task_encoder.use_temporal_video_embedder` only applies to the Energon data path. +### Vision DP Over CP + +`model.vision_dp_over_cp` shards the microbatch's images (or +temporal tubelets) across the language model's context-parallel group, so each +CP rank encodes `1/CP` of them instead of all of them. This is data parallelism +over images, not ring attention — RADIO already attends within each image via +`cu_seqlens`, so nothing is split along the sequence dimension and no +cross-rank attention communication is introduced. The vision projector runs on +the rank-local shard, and a single gather afterwards restores the full feature +set that every rank needs to merge media into the packed text sequence. + +The flag defaults to `True` and is a no-op unless +`model.context_parallel_size > 1`. Both recipes below ship with CP=1, so it only +takes effect once you raise CP. Set `model.vision_dp_over_cp=False` to +encode every image redundantly on every CP rank. + +Measured on 2 nodes / 16 GPUs (PP=1, DP=2, EP=8, ETP=1, MBS=2, GBS=64, GA=16), +flipping only this flag: + +| Dataset | TP | CP | Step time | Peak memory | +|---------------|----|----|----------------|---------------| +| InfoVQA | 4 | 2 | 13.2 → 12.5 s | 81 → 73 GB | +| Mantis | 2 | 4 | 12.2 → 11.6 s | 81 → 77 GB | +| llava_video | 4 | 2 | 13.6 → 13.0 s | 84 → 75 GB | + +Images are split by count, with the remainder assigned to the last rank. +Multi-image data whose per-sample image count varies (Mantis at CP=4) therefore +balances less evenly than uniform data, which is why its memory saving is the +smallest of the three. ### Image-Text — CORD-V2 diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md new file mode 100644 index 0000000000..ee4dfcd337 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md @@ -0,0 +1,112 @@ +# Skill Benchmark: nemo-mbridge-perf-vision-dp-over-cp + +> ✅ **Overall verdict: PASS — Recommended for publication** + +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +## Evaluation Metadata + +- Skill: `nemo-mbridge-perf-vision-dp-over-cp` +- Evaluation date: 2026-08-27 +- Evaluator version: `1.3.2` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Tasks: 3 evaluation tasks (2 positive, 1 negative) +- Dataset digest: `sha256:1f686f2a6deb8b2b442751f0639aafb30c6e89513befad8532f4ef5754f59593` (skill-evaluator-dataset-snapshot/1) +- Attempts per task: 1 +- Environment: `k8s-sandbox` +- Tier 3 evidence: required for publication + +Each task attempt ran in its own isolated sandbox pod. + +## Execution and Provenance + +- Validation status: `passed` +- Report generation: `complete` +- Evaluator version: `1.3.2` +- Git commit: `eda36f54e38357fe24c321a2ffae85589de9bd48` +- Content type: requested `auto`, detected `skill` +- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-eda36f54e38357fe24c321a2ffae85589de9bd48` +- Container image digest: `not recorded` +- Tier 3: requested `true`, executed `true`, status `succeeded` + +## What This Report Answers + +The three-tier evaluation checks whether the skill: + +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. + +## Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 67% → 100% (+32 points) | 59% → 95% (+36 points) | +| Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | +| Correctness | 47% → 100% (+53 points) | 60% → 93% (+33 points) | +| Discoverability | 83% → 100% (+17 points) | 58% → 96% (+38 points) | +| Effectiveness | 34% → 99% (+65 points) | 50% → 88% (+38 points) | +| Efficiency | 73% → 100% (+27 points) | 62% → 100% (+38 points) | + +**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. + +Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. + +## Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) | +| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) | + +## Findings and Observations + +
+Show detailed findings and successful checks + +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml`) + +
+ +## Scoring Methodology + +
+Show dimension definitions, source signals, and thresholds + +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | + +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. + +Signals present in this run: + +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was found and executed. +- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. + +
+ +## Freshness + +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md new file mode 100644 index 0000000000..89d8900fa2 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md @@ -0,0 +1,217 @@ +--- +name: nemo-mbridge-perf-vision-dp-over-cp +description: Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact. +license: Apache-2.0 +when_to_use: The vision tower replicates work and activations on every CP rank while the language model is already sharded, or a VLM run at CP>1 OOMs in the encoder; 'vision_dp_over_cp', 'vision CP', 'vision DP over CP', 'image sharding across CP ranks', 'encoder replicated across CP'. +--- + +# Vision DP Over CP Skill + +This skill covers `vision_dp_over_cp`: reusing the language model's +context-parallel process group to shard the **images** (or temporal tubelets) of +a microbatch across CP ranks, so each rank encodes `1/CP` of them instead of all +of them. + +It is **not** ring attention and shares nothing with +`nemo-mbridge-perf-hierarchical-context-parallel` beyond the process group. +Vision encoders such as RADIO already attend *within* each image via +`cu_seqlens`, so there is no long sequence to split and no cross-rank attention +communication. The correct mental model is **data parallelism over images, +borrowing the CP group** — a rank encodes whole images, never partial ones. + +Shipped for Nemotron Omni. The MCore split/gather helpers are model-agnostic, so +the pattern generalizes to any Bridge VLM with a dynamic-resolution encoder. + +## Do Not Confuse With `dist_train.vision_context_parallel_size` + +Bridge already has a similarly named and completely different knob: +`vision_context_parallel_size` on the dist-train path +(`src/megatron/bridge/models/qwen_vl/qwen3_vl_provider.py`, consumed in +`src/megatron/bridge/training/initialize.py`). + +| | `model.vision_dp_over_cp` (bool) | `dist_train.vision_context_parallel_size` (int) | +|---|---|---| +| Ranks | Reuses the language model's CP ranks | Gives the vision tower its **own** ranks | +| Splits | Whole images across those ranks | The vision encoder's own sequence dimension | +| Encoder `context_parallel_size` | Stays `1` | Set to this value | +| Models | Nemotron Omni | Qwen3-VL dist-train | + +They are not alternatives and do not compose in any tested configuration. + +## Enablement + +```python +cfg.model.context_parallel_size = 2 # the flag is a no-op at CP=1 +cfg.model.vision_dp_over_cp = False # opt out; on by default +``` + +Defaults to `True`. It self-disables when `context_parallel_size == 1`, so a +recipe that does not raise CP behaves exactly as it did before the flag existed +and reports no error. Opting out is only useful for A/B measurement or to +isolate a suspected split/gather bug. + +## Code Anchors + +Provider field, `src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py`: + +```python +vision_dp_over_cp: bool = True +``` + +Gating, in `NemotronOmniModel.__init__` +(`src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py`): + +```python +self.context_parallel_lm = language_transformer_config.context_parallel_size +self.vision_dp_over_cp = vision_dp_over_cp and self.context_parallel_lm > 1 +``` + +The split itself wraps an upstream MCore helper: + +```python +return split_to_context_parallel_ranks_dynamic_res( + images, + imgs_sizes, + vision_packed_seq_params, + patch_dim=self.patch_dim, + fp8_enabled=False, + fp8_recipe=getattr(self.config, "fp8_recipe", None), + num_frames=num_frames, + temporal_patch_size=temporal_patch_size, +) +``` + +Project-then-gather, at the end of `_encode_images`: + +```python +if shard_vision: + # Project before the gather so the projector stays sharded, then + # restore the global feature set every CP rank's media merge needs. + projected = _project_multimodal_embeddings(self.vision_projection, encoded) + gathered = gather_from_context_parallel_ranks_dynamic_res(projected, num_padded_ranks) + return gathered.contiguous() +``` + +Upstream helpers (do not modify — changes go through the MCore repo): + +- `3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py` + — `split_to_context_parallel_ranks_dynamic_res`, + `gather_from_context_parallel_ranks_dynamic_res` +- `3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py` + — performs the identical split/encode/gather on its temporal path + +## Implementation Map + +1. **Split.** The MCore helper partitions the patched image tensor on image (or + tubelet) boundaries and rewrites `imgs_sizes` and the vision + `PackedSeqParams` (`cu_seqlens`, `max_seqlen`) to describe only the rank-local + share. Sharding is by **image count**, not patch count; the remainder goes to + the **last** rank. +2. **Placeholder padding.** When a microbatch has fewer images than CP ranks, + the helper injects `num_padded_imgs = max(0, cp_size - num_imgs)` dummy `1x1` + images so no rank runs an empty encoder, and returns `num_padded_ranks` so the + gather can drop them again. +3. **Encode.** Each rank runs the full vision tower on its own images only. + Activations, not just compute, are what drop by `1/CP`. The vision encoder's + own `context_parallel_size` stays hardcoded to `1` in + `_build_vision_config` — that is correct and unchanged by this feature, since + the sharding happens *outside* the encoder, over its inputs. +4. **Even-grid padding.** `_pad_patch_grid_to_even` zero-pads an odd patch grid + before pixel shuffle. Only the injected `1x1` placeholders hit this path; + real frames are already even in both extents. +5. **Project, then gather.** The vision projector runs on the rank-local shard, + so it stays sharded too, and only the smaller post-projector features cross + the wire. The gather is the **only** collective this feature adds. +6. **Merge.** After the gather, every rank holds the full global feature set, + which is exactly what the unchanged media-merge path expects. + +## Measured Impact + +Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 +GBS=64 GA=16, with only `vision_dp_over_cp` flipped: + +| Dataset | TP | CP | Step time | Peak memory | +|-------------|----|----|---------------|-------------| +| InfoVQA | 4 | 2 | 13.2 -> 12.5 s | 81 -> 73 GB | +| Mantis | 2 | 4 | 12.2 -> 11.6 s | 81 -> 77 GB | +| llava_video | 4 | 2 | 13.6 -> 13.0 s | 84 -> 75 GB | + +It is both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step +time). Do not describe it as memory-only; a single early video A/B suggested step +time was flat, and the three-dataset sweep contradicted that. + +The Mantis row is the weakest memory result, and that is the load-balance +signature described under Pitfalls, not noise. + +## Pitfalls + +1. **Silent no-op at CP=1.** The flag ANDs with `context_parallel_lm > 1`. Both + checked-in Nemotron Omni recipes ship CP=1, so the on-by-default value changes + nothing there. Always confirm CP>1 before attributing a result to this flag, + and note that raising CP now silently enables image sharding too. +2. **Count-based split leaves the tail rank hot.** Images are divided by count + with the remainder on the last rank. Multi-image data with a wide per-sample + image-count spread (e.g. Mantis, 1-5 images/sample at CP=4) balances poorly: + peak memory is set by the busiest rank, so the win shrinks. A + patch-count-balanced greedy split on image boundaries is the fix, and it + belongs **upstream in MCore**, not reimplemented in Bridge. Since the flag is + on by default, this imbalance is what a multi-image CP>1 run gets unless the + user opts out. +3. **Process-group divergence is checked, not assumed.** The MCore splitter + resolves the CP group from global parallel state, so a `ProcessGroupCollection` + whose `cp` size disagrees with `context_parallel_size` would shard against the + wrong ranks. Bridge raises instead: + `"Nemotron Omni vision_dp_over_cp does not match its process group"`. +4. **Odd patch grids break pixel shuffle.** Omni's factor-2 pixel shuffle rejects + an odd grid, and MCore's `1x1` placeholder images are odd by construction. + Without `_pad_patch_grid_to_even`, the CP>images case fails only for the + specific microbatches that trigger placeholder injection — an intermittent + failure that looks data-dependent. +5. **Do not gather before projecting.** Gathering encoder output first would + replicate the projector on every rank and move a larger tensor. The ordering + is deliberate. +6. **The placeholder path is unvalidated end-to-end.** A two-rank unit test + proves the sharded encoder matches the unsharded one when a microbatch holds + fewer images than ranks, but no multi-node training run has exercised it, and + nothing covers CP=8. + +## Verification + +Unit tests. The second file self-launches a two-rank `torch.distributed` job and +asserts the sharded tower matches the unsharded one for fewer images than ranks, +an exact split, and a remainder; it skips when fewer than two GPUs are visible: + +```bash +uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py -q +uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_vision_dp_over_cp_distributed.py -q +``` + +Multi-GPU A/B. Run the same recipe twice at CP>1, flipping only the flag, and +compare reported step time and peak allocated memory: + +```bash +uv run python -m torch.distributed.run --nproc_per_node=8 \ + scripts/training/run_recipe.py \ + --recipe nemotron_omni_cord_v2_sft_config \ + model.context_parallel_size=2 \ + model.vision_dp_over_cp=false \ + train.train_iters=20 +``` + +Success criteria: + +- Peak memory drops relative to the `false` arm; step time does not regress. +- Loss curves of the two arms track each other. The gather makes this + mathematically equivalent, so a divergence means the split or gather is wrong, + not that the feature is lossy. +- If nothing changes at all, check that `context_parallel_size > 1` — the flag + silently self-disables at CP=1. + +## Related + +- @skills/nemo-mbridge-perf-moe-vlm-training/SKILL.md — VLM training strategy, + FSDP vs 3D parallel, and why mock image data invalidates VLM benchmarks +- @skills/nemo-mbridge-perf-memory-tuning/SKILL.md +- @skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL.md — the + *sequence*-splitting use of the same process group; unrelated mechanism +- @skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml new file mode 100644 index 0000000000..242cd5b5c2 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml @@ -0,0 +1,93 @@ +title: vision_dp_over_cp +validated_on: "2026-08-18" +summary: > + Megatron-Bridge can shard a VLM's images or temporal tubelets across the + language model's context-parallel group so each CP rank encodes 1/CP of the + microbatch instead of all of it. This is data parallelism over images reusing + the CP group, not ring attention: the encoder attends within each image via + cu_seqlens, so no sequence is split and the only added collective is the + post-projector gather. Controlled by model.vision_dp_over_cp, default + True, and a silent no-op at context_parallel_size=1. +validation_status: + upstream_mcore_split_gather_helpers: + - code_verified + bridge_nemotron_omni_passthrough: + - code_verified + bridge_even_grid_padding: + - unit_tested + bridge_process_group_mismatch_guard: + - code_verified + bridge_multi_gpu_ab_cp2_cp4: + - benchmark_verified + bridge_cp8_placeholder_padding_end_to_end: + - unclear + bridge_vision_cp_recipes: + - unclear + bridge_vision_cp_docs: + - doc_only +feature_meaning: + vision_dp_over_cp: > + Bridge model flag that shards the microbatch's images or tubelets across the + language model's CP ranks. Whole images only; no image is split. + split_to_context_parallel_ranks_dynamic_res: > + Upstream Megatron-Core helper that partitions the patched image tensor on + image boundaries and rewrites imgs_sizes and the vision PackedSeqParams to + describe the rank-local share. + gather_from_context_parallel_ranks_dynamic_res: > + Upstream Megatron-Core helper that restores the global feature set after the + vision projector and drops the injected placeholder images. + placeholder_images: > + When a microbatch holds fewer images than CP ranks, the splitter injects + max(0, cp_size - num_imgs) dummy 1x1 images so no rank runs an empty + encoder. They are removed during the gather. +recommended_path: + model.context_parallel_size: 2 + model.vision_dp_over_cp: true +measured_impact: + setup: > + Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 + GBS=64 GA=16, only vision_dp_over_cp flipped. + results: + - dataset: InfoVQA + tp: 4 + cp: 2 + step_time_s: "13.2 -> 12.5" + peak_memory_gb: "81 -> 73" + - dataset: Mantis + tp: 2 + cp: 4 + step_time_s: "12.2 -> 11.6" + peak_memory_gb: "81 -> 77" + - dataset: llava_video + tp: 4 + cp: 2 + step_time_s: "13.6 -> 13.0" + peak_memory_gb: "84 -> 75" + interpretation: > + Both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step + time). Mantis shows the smallest memory saving because its per-sample image + count varies from 1 to 5, which the count-based split balances poorly at + CP=4. +known_constraints: + - Requires context_parallel_size > 1; the flag self-disables at CP=1 without error. + - Requires an MCore pin that exposes megatron.core.models.multimodal.context_parallel. + - The CP process group size must equal context_parallel_size, otherwise Bridge raises. + - Applies to the dynamic-resolution vision path; the encoder must attend per image via cu_seqlens. + - Pixel shuffle requires an even patch grid, so injected placeholder images are zero-padded first. +known_limitations: + - Images are split by count with the remainder on the last rank, so multi-image data with a wide image-count spread balances unevenly. + - The CP=8 case where a microbatch holds fewer images than ranks has unit tests but no end-to-end run. + - No checked-in Bridge recipe enables vision_dp_over_cp; both Nemotron Omni recipes ship CP=1. + - Only wired into Nemotron Omni today, although the MCore helpers are model-agnostic. +evidence: + - src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py + - src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py + - tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py + - examples/models/nemotron/nemotron_3_omni/README.md + - 3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py + - 3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py +follow_up_validation: + - Run an end-to-end job where the microbatch holds fewer images than CP ranks to exercise placeholder padding. + - Replace the count-based split with a patch-count-balanced greedy split on image boundaries, upstream in Megatron-Core. + - Add a Bridge functional test that completes a training step with vision_dp_over_cp=true at CP=2. + - Extend the flag to other Bridge VLM families using the same MCore helpers. diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json b/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json new file mode 100644 index 0000000000..0b18b0ab2a --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json @@ -0,0 +1,43 @@ +[ + { + "id": "vision-dp-over-cp-positive-enablement", + "question": "Use the nemo-mbridge-perf-vision-dp-over-cp skill. Our Nemotron Omni VLM run at CP=2 shows the vision encoder doing identical work on every context-parallel rank. Give the exact Bridge config values to shard it, explain what collective this adds, and state how to prove the flag actually took effect.", + "expected_skill": "nemo-mbridge-perf-vision-dp-over-cp", + "expected_script": null, + "ground_truth": "The answer should use the vision DP over CP skill. It should note that cfg.model.vision_dp_over_cp already defaults to True, so the only required change is cfg.model.context_parallel_size>1, and that the flag self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-dp-over-cp skill before answering.", + "State that model.vision_dp_over_cp is already True by default and require context_parallel_size>1.", + "State that the flag is a silent no-op at CP=1.", + "Explain the split is over whole images, not over the sequence dimension.", + "Identify the post-projector gather as the only added collective, and that projection happens before the gather.", + "Propose an A/B verification comparing peak memory, step time, and loss parity." + ] + }, + { + "id": "vision-dp-over-cp-positive-load-imbalance", + "question": "Use the nemo-mbridge-perf-vision-dp-over-cp skill. We enabled vision DP over CP at CP=4 on a multi-image dataset where samples carry between 1 and 5 images, and the memory saving is much smaller than on our single-image datasets. Explain why and say where the fix belongs.", + "expected_skill": "nemo-mbridge-perf-vision-dp-over-cp", + "expected_script": null, + "ground_truth": "The answer should attribute the weak saving to the split being by image count rather than patch count, with the remainder assigned to the last rank, so a wide per-sample image-count spread leaves the tail rank overloaded and peak memory is set by the busiest rank. It should cite the measured Mantis result (CP=4, peak memory 81 -> 77 GB) as smaller than InfoVQA or llava_video (roughly -10%). It should say the fix is a patch-count-balanced greedy split on image boundaries and that it belongs upstream in Megatron-Core, not reimplemented in Bridge, since 3rdparty/Megatron-LM must not be modified in this repo.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-dp-over-cp skill before answering.", + "Identify the count-based split with remainder-on-last-rank as the cause.", + "Explain that peak memory is set by the busiest rank.", + "Reference the measured Mantis result as the load-imbalance signature.", + "Locate the fix upstream in Megatron-Core rather than in Megatron-Bridge." + ] + }, + { + "id": "vision-dp-over-cp-negative-hierarchical-cp", + "question": "Our text-only MoE model has 8 KV heads and we need CP=16 for 128K sequences. Which cp_comm_type should we use and what else must be set?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "This is a hierarchical context parallelism question about splitting a long text sequence across ranks, not about sharding vision-encoder images. The vision DP over CP skill should not be loaded. A correct answer discusses cp_comm_type=a2a+p2p with hierarchical_context_parallel_sizes whose product equals context_parallel_size, and the KV-head limit on pure a2a.", + "expected_behavior": [ + "Do not load the nemo-mbridge-perf-vision-dp-over-cp skill.", + "Recognize the question concerns sequence-dimension context parallelism for a text-only model.", + "Route to hierarchical context parallelism guidance instead." + ] + } +] diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md new file mode 100644 index 0000000000..848868ce1a --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md @@ -0,0 +1,88 @@ +## Description:
+Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and engineers optimizing VLM training performance who need to reduce peak memory and improve throughput by sharding the vision encoder's image inputs across context-parallel ranks in Megatron-Bridge.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
+- [SKILL.md](SKILL.md)
+- [card.yaml](card.yaml)
+ + +## Skill Output:
+**Output Type(s):** [Configuration instructions, Analysis]
+**Output Format:** [Markdown with inline Python code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Evaluation Agents Used:
+- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
+- Codex (`openai/openai/gpt-5.5`)
+ + + +## Evaluation Tasks:
+Evaluated against 3 evaluation tasks (2 positive, 1 negative) in isolated k8s-sandbox pods with 1 attempt per task.
+ +## Evaluation Metrics Used:
+Reported benchmark dimensions:
+- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access).
+- Correctness: Whether the skill produces correct answers against the reference.
+- Discoverability: Whether the right skill is loaded and activated when needed.
+- Effectiveness: Whether the skill helps complete the user's goal and expected workflow (goal completion + behavior adherence).
+- Efficiency: Whether the skill avoids wasted tool or skill usage (routing quality and productive tool use).
+ +Underlying evaluation signals used in this run:
+- `security`: Unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Whether the expected skill was found and executed.
+- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy`: Final-answer correctness against the reference answer.
+- `goal_accuracy`: Whether the user's goal was achieved.
+- `behavior_check`: Whether the expected workflow behavior was followed.
+ + + +## Evaluation Results:
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 67% → 100% (+32 points) | 59% → 95% (+36 points) | +| Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | +| Correctness | 47% → 100% (+53 points) | 60% → 93% (+33 points) | +| Discoverability | 83% → 100% (+17 points) | 58% → 96% (+38 points) | +| Effectiveness | 34% → 99% (+65 points) | 50% → 88% (+38 points) | +| Efficiency | 73% → 100% (+27 points) | 62% → 100% (+38 points) | + +## Testing Completed:
+**[x] Agent Red-Teaming**
+**[ ] Network Security**
+**[ ] Product Security**
+ +## Skill Version(s):
+1.0.0+b7643bd (source: pyproject.toml)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig new file mode 100644 index 0000000000..e1b5805ea0 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWRwLW92ZXItY3AiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiMTU2ZmQwMjI0MzgwNDU1MmI2NTAxMDU2MDU4Zjk1YmY2NDFlOWM2YzUzNDY1NDdhYmU2NDZhNjc2ZTdjYjg2ZSIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzA2NTg3ZmIxNzM3ZWU2ODA5NDJlNWQ1M2QwMDk0MzI4NzJlZTQ1ZmNkZDcwZDk3NjU2YTU3MmIzMmQ1M2QxZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJjNGFjZTEwMWVkZmM2OGJhNzc0Y2I3MWRlNDBiZTdlOTk2NDZhMjFiNzA4MDEwYjFhNjAyYjI4NmRjMTRmMzg2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3ZTIxMDQyZGRiNjM3ZmNmN2JiNjEwOTcxYWY2ZjEyZTI5OWRjMmU1YmNlY2M1ZjM2OTM5MzRiMGVjZDUwNTkyIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiOWEyNWJlYTM4M2Q4NjU0ZjMyY2E3Y2EzZjI1ODZmOWYyZTJlODdiNjQ4N2U1NzJmNDcyOTFlMzk3ZTMzNjkzYiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImM4OGNiZjEzYmJiYmVmNzE1MDU0OTExNmU2MmY1NjU5ZTU3OTY1ZjNlNTNmNDgwMTg1MWI3YjU5NWJmZjNhYzgiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDwd/MAPl2kWNx2d2eJ3XaNNCQl8x401GN0EFcYDECzBr5KQj3668JZgX5KHvtyCsECMCeibyjnUd+tRvezzdXiUZJEeuE07vbfe4PdF6RygUCNd4MBop2pR/s7Ny5fznkiKA==","keyid":""}]}} \ No newline at end of file