From c70996c9bc06cbf184ee7a0196f780a40c7d9162 Mon Sep 17 00:00:00 2001 From: Huy Vu Date: Tue, 18 Aug 2026 08:15:27 -0700 Subject: [PATCH 1/8] docs(model): document Nemotron Omni vision context parallelism Add a Vision Context Parallelism section to the Nemotron-3 Omni example README covering the model.vision_context_parallel flag, its CP>1 gating, and the measured step-time/memory A/B across three datasets. Also fixes a typo in the vision input mode table (temporal_patchrecipe_dim). Signed-off-by: Huy Vu --- .../models/nemotron/nemotron_3_omni/README.md | 30 ++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index a66d6b9e6e..52050b752a 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -257,12 +257,40 @@ Set the four flags below as a matched column — a mismatched set produces incor | Field | Dynamic resolution (images, variable H×W) | Temporal video (videos, fused pairs, 512²) | |----------------------------------------------------|-------------------------------------------|--------------------------------------------| | `dataset.task_encoder.use_temporal_video_embedder` | `False` | `True` | -| `model.temporal_patchrecipe_dim` | `1` | `2` | +| `model.temporal_patch_dim` | `1` | `2` | | `model.separate_video_embedder` | `False` | `True` | | `model.temporal_ckpt_compat` | `False` | `True` | Note:`dataset.task_encoder.use_temporal_video_embedder` only applies to the Energon data path. +### Vision Context Parallelism + +`model.vision_context_parallel=True` shards the microbatch's images (or +temporal tubelets) across the language model's context-parallel group, so each +CP rank encodes `1/CP` of them instead of all of them. This is data parallelism +over images, not ring attention — RADIO already attends within each image via +`cu_seqlens`, so nothing is split along the sequence dimension and no +cross-rank attention communication is introduced. The vision projector runs on +the rank-local shard, and a single gather afterwards restores the full feature +set that every rank needs to merge media into the packed text sequence. + +The flag defaults to `False` and is a no-op unless +`model.context_parallel_size > 1`. Both recipes below ship with CP=1, so enable +it only when you raise CP. + +Measured on 2 nodes / 16 GPUs (PP=1, DP=2, EP=8, ETP=1, MBS=2, GBS=64, GA=16), +flipping only this flag: + +| Dataset | TP | CP | Step time | Peak memory | +|---------------|----|----|----------------|---------------| +| InfoVQA | 4 | 2 | 13.2 → 12.5 s | 81 → 73 GB | +| Mantis | 2 | 4 | 12.2 → 11.6 s | 81 → 77 GB | +| llava_video | 4 | 2 | 13.6 → 13.0 s | 84 → 75 GB | + +Images are split by count, with the remainder assigned to the last rank. +Multi-image data whose per-sample image count varies (Mantis at CP=4) therefore +balances less evenly than uniform data, which is why its memory saving is the +smallest of the three. ### Image-Text — CORD-V2 From dc0ccfff5b9419e4671c46959976318503b34f70 Mon Sep 17 00:00:00 2001 From: Huy Vu Date: Tue, 18 Aug 2026 08:28:44 -0700 Subject: [PATCH 2/8] docs(model): add vision context parallel perf skill Capture the vision_context_parallel findings as a proper perf skill: SKILL.md with enablement, code anchors, implementation map, the measured 3-dataset A/B, and the load-balance pitfalls; card.yaml with validation status, constraints, and follow-up validation; an evals dataset with two positive tasks and one negative routing task against hierarchical CP, which shares the "context parallel" vocabulary but not the mechanism. BENCHMARK.md and skill-card.md are committed with empty result tables: those numbers come from an NVSkills-Eval run that has not happened yet and must not be hand-authored. skill.oms.sig is likewise omitted since it is a Sigstore bundle from the NVIDIA signing pipeline. Signed-off-by: Huy Vu --- docs/skills-index.md | 1 + .../models/nemotron/nemotron_3_omni/README.md | 3 + .../BENCHMARK.md | 85 ++++++++ .../SKILL.md | 190 ++++++++++++++++++ .../card.yaml | 93 +++++++++ .../evals/evals.json | 43 ++++ .../skill-card.md | 87 ++++++++ 7 files changed, 502 insertions(+) create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/card.yaml create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md diff --git a/docs/skills-index.md b/docs/skills-index.md index 0b191667d3..25c32ff3a9 100644 --- a/docs/skills-index.md +++ b/docs/skills-index.md @@ -50,6 +50,7 @@ skills/nemo-mbridge-perf-tp-dp-comm-overlap/SKILL skills/nemo-mbridge-perf-megatron-fsdp/SKILL skills/nemo-mbridge-perf-sequence-packing/SKILL skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL +skills/nemo-mbridge-perf-vision-context-parallel/SKILL skills/nemo-mbridge-perf-expert-parallel-overlap/SKILL skills/nemo-mbridge-perf-moe-comm-overlap/SKILL skills/nemo-mbridge-perf-activation-recompute/SKILL diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index 52050b752a..558df26cb0 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -292,6 +292,9 @@ Multi-image data whose per-sample image count varies (Mantis at CP=4) therefore balances less evenly than uniform data, which is why its memory saving is the smallest of the three. +For code anchors, pitfalls, and the A/B verification procedure, see +[the vision context parallel skill](../../../../skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md). + ### Image-Text — CORD-V2 [CORD-V2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) is a diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md new file mode 100644 index 0000000000..dfad07e2f0 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md @@ -0,0 +1,85 @@ +# Evaluation Report + +Evaluation of the `nemo-mbridge-perf-vision-context-parallel` skill before publication through NVSkills-Eval. + +This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use. + +## Evaluation Summary + +- Skill: `nemo-mbridge-perf-vision-context-parallel` +- Evaluation date: not yet run +- NVSkills-Eval profile: `external` +- Environment: `local` +- Dataset: 3 evaluation tasks +- Attempts per task: 2 +- Pass threshold: 50% +- Overall verdict: **PENDING — NVSkills-Eval has not been run against this skill** + +> This file is a placeholder committed alongside the skill. The result tables +> below are intentionally empty. Populate them by running NVSkills-Eval with the +> `external` profile against `evals/evals.json`, then replace this notice. +> Do not hand-author scores. + +## Agents Used + +- `claude-code` +- `codex` + +## Metrics Used + +Reported benchmark dimensions: + +- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. +- Correctness: checks whether the agent follows the expected workflow and produces the correct final output. +- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant. +- Effectiveness: checks whether the agent performs measurably better with the skill than without it. +- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work. + +Underlying evaluation signals used in this run: + +- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow. +- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage. +- `accuracy` (Accuracy): grades final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully. +- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations. +- `token_efficiency` (Token Efficiency): compares token usage with and without the skill. + +## Test Tasks + +The benchmark dataset contains 3 evaluation tasks: + +- Positive tasks: 2 tasks where the skill was expected to activate — enablement and the load-imbalance diagnosis. +- Negative tasks: 1 task where no skill was expected — a hierarchical context parallelism question for a text-only model, included because "context parallel" is a shared keyword and the two skills describe unrelated mechanisms. +- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred. + +Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases. + +## Results + +| Dimension | Num | `claude-code` | `codex` | +|---|---:|---:|---:| +| Security | — | — | — | +| Correctness | — | — | — | +| Discoverability | — | — | — | +| Effectiveness | — | — | — | +| Efficiency | — | — | — | + +Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. + +## Tier 1: Static Validation Summary + +Not yet run. + +## Tier 2: Deduplication Summary + +Not yet run. Deduplication is the tier to watch for this skill: it shares the +"context parallel" vocabulary with `nemo-mbridge-perf-hierarchical-context-parallel` +while describing an unrelated mechanism, which is why the dataset includes an +explicit negative routing task. + +## Publication Recommendation + +Withhold from NVSkills-Eval publication until the evaluation above has been run +and this file updated with real results. The skill content itself is usable in +the repository in the meantime. diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md new file mode 100644 index 0000000000..817c1e91c3 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md @@ -0,0 +1,190 @@ +--- +name: nemo-mbridge-perf-vision-context-parallel +description: Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact. +license: Apache-2.0 +when_to_use: The vision tower replicates work and activations on every CP rank while the language model is already sharded, or a VLM run at CP>1 OOMs in the encoder; 'vision_context_parallel', 'vision CP', 'vision DP over CP', 'image sharding across CP ranks', 'encoder replicated across CP'. +--- + +# Vision Context Parallel Skill + +This skill covers `vision_context_parallel`: reusing the language model's +context-parallel process group to shard the **images** (or temporal tubelets) of +a microbatch across CP ranks, so each rank encodes `1/CP` of them instead of all +of them. + +It is **not** ring attention and shares nothing with +`nemo-mbridge-perf-hierarchical-context-parallel` beyond the process group. +Vision encoders such as RADIO already attend *within* each image via +`cu_seqlens`, so there is no long sequence to split and no cross-rank attention +communication. The correct mental model is **data parallelism over images, +borrowing the CP group** — a rank encodes whole images, never partial ones. + +Shipped for Nemotron Omni. The MCore split/gather helpers are model-agnostic, so +the pattern generalizes to any Bridge VLM with a dynamic-resolution encoder. + +## Enablement + +```python +cfg.model.context_parallel_size = 2 # the flag is a no-op at CP=1 +cfg.model.vision_context_parallel = True +``` + +Defaults to `False`. It self-disables when `context_parallel_size == 1`, so +turning it on in a recipe that does not raise CP changes nothing and reports no +error. + +## Code Anchors + +Provider field, `src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py`: + +```python +vision_context_parallel: bool = False +``` + +Gating, in `NemotronOmniModel.__init__` +(`src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py`): + +```python +self.context_parallel_lm = language_transformer_config.context_parallel_size +self.vision_context_parallel = vision_context_parallel and self.context_parallel_lm > 1 +``` + +The split itself wraps an upstream MCore helper: + +```python +return split_to_context_parallel_ranks_dynamic_res( + images, + imgs_sizes, + vision_packed_seq_params, + patch_dim=self.patch_dim, + fp8_enabled=False, + fp8_recipe=getattr(self.config, "fp8_recipe", None), + num_frames=num_frames, + temporal_patch_size=temporal_patch_size, +) +``` + +Project-then-gather, at the end of `_encode_images`: + +```python +if shard_vision: + # Project before the gather so the projector stays sharded, then + # restore the global feature set every CP rank's media merge needs. + projected = self.vision_projection(encoded.unsqueeze(1)).squeeze(1) + gathered = gather_from_context_parallel_ranks_dynamic_res(projected, num_padded_ranks) + return gathered.contiguous() +``` + +Upstream helpers (do not modify — changes go through the MCore repo): + +- `3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py` + — `split_to_context_parallel_ranks_dynamic_res`, + `gather_from_context_parallel_ranks_dynamic_res` +- `3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py` + — performs the identical split/encode/gather on its temporal path + +## Implementation Map + +1. **Split.** The MCore helper partitions the patched image tensor on image (or + tubelet) boundaries and rewrites `imgs_sizes` and the vision + `PackedSeqParams` (`cu_seqlens`, `max_seqlen`) to describe only the rank-local + share. Sharding is by **image count**, not patch count; the remainder goes to + the **last** rank. +2. **Placeholder padding.** When a microbatch has fewer images than CP ranks, + the helper injects `num_padded_imgs = max(0, cp_size - num_imgs)` dummy `1x1` + images so no rank runs an empty encoder, and returns `num_padded_ranks` so the + gather can drop them again. +3. **Encode.** Each rank runs the full vision tower on its own images only. + Activations, not just compute, are what drop by `1/CP`. +4. **Even-grid padding.** `_pad_patch_grid_to_even` zero-pads an odd patch grid + before pixel shuffle. Only the injected `1x1` placeholders hit this path; + real frames are already even in both extents. +5. **Project, then gather.** The vision projector runs on the rank-local shard, + so it stays sharded too, and only the smaller post-projector features cross + the wire. The gather is the **only** collective this feature adds. +6. **Merge.** After the gather, every rank holds the full global feature set, + which is exactly what the unchanged media-merge path expects. + +## Measured Impact + +Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 +GBS=64 GA=16, with only `vision_context_parallel` flipped: + +| Dataset | TP | CP | Step time | Peak memory | +|-------------|----|----|---------------|-------------| +| InfoVQA | 4 | 2 | 13.2 -> 12.5 s | 81 -> 73 GB | +| Mantis | 2 | 4 | 12.2 -> 11.6 s | 81 -> 77 GB | +| llava_video | 4 | 2 | 13.6 -> 13.0 s | 84 -> 75 GB | + +It is both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step +time). Do not describe it as memory-only; a single early video A/B suggested step +time was flat, and the three-dataset sweep contradicted that. + +The Mantis row is the weakest memory result, and that is the load-balance +signature described under Pitfalls, not noise. + +## Pitfalls + +1. **Silent no-op at CP=1.** The flag ANDs with `context_parallel_lm > 1`. Both + checked-in Nemotron Omni recipes ship CP=1, so enabling it there changes + nothing. Always confirm CP>1 before attributing a result to this flag. +2. **Count-based split leaves the tail rank hot.** Images are divided by count + with the remainder on the last rank. Multi-image data with a wide per-sample + image-count spread (e.g. Mantis, 1-5 images/sample at CP=4) balances poorly: + peak memory is set by the busiest rank, so the win shrinks. A + patch-count-balanced greedy split on image boundaries is the fix, and it + belongs **upstream in MCore**, not reimplemented in Bridge. +3. **Process-group divergence is checked, not assumed.** The MCore splitter + resolves the CP group from global parallel state, so a `ProcessGroupCollection` + whose `cp` size disagrees with `context_parallel_size` would shard against the + wrong ranks. Bridge raises instead: + `"Nemotron Omni vision context parallelism does not match its process group"`. +4. **Odd patch grids break pixel shuffle.** Omni's factor-2 pixel shuffle rejects + an odd grid, and MCore's `1x1` placeholder images are odd by construction. + Without `_pad_patch_grid_to_even`, the CP>images case fails only for the + specific microbatches that trigger placeholder injection — an intermittent + failure that looks data-dependent. +5. **Do not gather before projecting.** Gathering encoder output first would + replicate the projector on every rank and move a larger tensor. The ordering + is deliberate. +6. **CP=8 placeholder path is unvalidated end-to-end.** Unit tests cover the + even-grid padding, but no multi-node run has exercised a microbatch with fewer + images than ranks. + +## Verification + +Unit tests for the padding helper: + +```bash +uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py -q +``` + +Multi-GPU A/B. Run the same recipe twice at CP>1, flipping only the flag, and +compare reported step time and peak allocated memory: + +```bash +uv run python -m torch.distributed.run --nproc_per_node=8 \ + scripts/training/run_recipe.py \ + --recipe nemotron_omni_cord_v2_sft_config \ + model.context_parallel_size=2 \ + model.vision_context_parallel=true \ + train.train_iters=20 +``` + +Success criteria: + +- Peak memory drops relative to the `false` arm; step time does not regress. +- Loss curves of the two arms track each other. The gather makes this + mathematically equivalent, so a divergence means the split or gather is wrong, + not that the feature is lossy. +- If nothing changes at all, check that `context_parallel_size > 1` — the flag + silently self-disables at CP=1. + +## Related + +- @skills/nemo-mbridge-perf-moe-vlm-training/SKILL.md — VLM training strategy, + FSDP vs 3D parallel, and why mock image data invalidates VLM benchmarks +- @skills/nemo-mbridge-perf-memory-tuning/SKILL.md +- @skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL.md — the + *sequence*-splitting use of the same process group; unrelated mechanism +- @skills/nemo-mbridge-perf-vision-context-parallel/card.yaml diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml b/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml new file mode 100644 index 0000000000..441f15bf8f --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml @@ -0,0 +1,93 @@ +title: vision_context_parallel +validated_on: "2026-08-18" +summary: > + Megatron-Bridge can shard a VLM's images or temporal tubelets across the + language model's context-parallel group so each CP rank encodes 1/CP of the + microbatch instead of all of it. This is data parallelism over images reusing + the CP group, not ring attention: the encoder attends within each image via + cu_seqlens, so no sequence is split and the only added collective is the + post-projector gather. Opt-in through model.vision_context_parallel, default + False, and a silent no-op at context_parallel_size=1. +validation_status: + upstream_mcore_split_gather_helpers: + - code_verified + bridge_nemotron_omni_passthrough: + - code_verified + bridge_even_grid_padding: + - unit_tested + bridge_process_group_mismatch_guard: + - code_verified + bridge_multi_gpu_ab_cp2_cp4: + - benchmark_verified + bridge_cp8_placeholder_padding_end_to_end: + - unclear + bridge_vision_cp_recipes: + - unclear + bridge_vision_cp_docs: + - doc_only +feature_meaning: + vision_context_parallel: > + Bridge model flag that shards the microbatch's images or tubelets across the + language model's CP ranks. Whole images only; no image is split. + split_to_context_parallel_ranks_dynamic_res: > + Upstream Megatron-Core helper that partitions the patched image tensor on + image boundaries and rewrites imgs_sizes and the vision PackedSeqParams to + describe the rank-local share. + gather_from_context_parallel_ranks_dynamic_res: > + Upstream Megatron-Core helper that restores the global feature set after the + vision projector and drops the injected placeholder images. + placeholder_images: > + When a microbatch holds fewer images than CP ranks, the splitter injects + max(0, cp_size - num_imgs) dummy 1x1 images so no rank runs an empty + encoder. They are removed during the gather. +recommended_path: + model.context_parallel_size: 2 + model.vision_context_parallel: true +measured_impact: + setup: > + Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 + GBS=64 GA=16, only vision_context_parallel flipped. + results: + - dataset: InfoVQA + tp: 4 + cp: 2 + step_time_s: "13.2 -> 12.5" + peak_memory_gb: "81 -> 73" + - dataset: Mantis + tp: 2 + cp: 4 + step_time_s: "12.2 -> 11.6" + peak_memory_gb: "81 -> 77" + - dataset: llava_video + tp: 4 + cp: 2 + step_time_s: "13.6 -> 13.0" + peak_memory_gb: "84 -> 75" + interpretation: > + Both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step + time). Mantis shows the smallest memory saving because its per-sample image + count varies from 1 to 5, which the count-based split balances poorly at + CP=4. +known_constraints: + - Requires context_parallel_size > 1; the flag self-disables at CP=1 without error. + - Requires an MCore pin that exposes megatron.core.models.multimodal.context_parallel. + - The CP process group size must equal context_parallel_size, otherwise Bridge raises. + - Applies to the dynamic-resolution vision path; the encoder must attend per image via cu_seqlens. + - Pixel shuffle requires an even patch grid, so injected placeholder images are zero-padded first. +known_limitations: + - Images are split by count with the remainder on the last rank, so multi-image data with a wide image-count spread balances unevenly. + - The CP=8 case where a microbatch holds fewer images than ranks has unit tests but no end-to-end run. + - No checked-in Bridge recipe enables vision_context_parallel; both Nemotron Omni recipes ship CP=1. + - Only wired into Nemotron Omni today, although the MCore helpers are model-agnostic. +evidence: + - src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py + - src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py + - tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py + - examples/models/nemotron/nemotron_3_omni/README.md + - 3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py + - 3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py +follow_up_validation: + - Run an end-to-end job where the microbatch holds fewer images than CP ranks to exercise placeholder padding. + - Replace the count-based split with a patch-count-balanced greedy split on image boundaries, upstream in Megatron-Core. + - Add a Bridge functional test that completes a training step with vision_context_parallel=true at CP=2. + - Extend the flag to other Bridge VLM families using the same MCore helpers. diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json b/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json new file mode 100644 index 0000000000..6d44880494 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json @@ -0,0 +1,43 @@ +[ + { + "id": "vision-context-parallel-positive-enablement", + "question": "Use the nemo-mbridge-perf-vision-context-parallel skill. Our Nemotron Omni VLM run at CP=2 shows the vision encoder doing identical work on every context-parallel rank. Give the exact Bridge config values to shard it, explain what collective this adds, and state how to prove the flag actually took effect.", + "expected_skill": "nemo-mbridge-perf-vision-context-parallel", + "expected_script": null, + "ground_truth": "The answer should use the vision context parallel skill. It should set cfg.model.vision_context_parallel=True with cfg.model.context_parallel_size>1, and note the flag defaults to False and self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-context-parallel skill before answering.", + "Set model.vision_context_parallel=True and require context_parallel_size>1.", + "State that the flag is a silent no-op at CP=1.", + "Explain the split is over whole images, not over the sequence dimension.", + "Identify the post-projector gather as the only added collective, and that projection happens before the gather.", + "Propose an A/B verification comparing peak memory, step time, and loss parity." + ] + }, + { + "id": "vision-context-parallel-positive-load-imbalance", + "question": "Use the nemo-mbridge-perf-vision-context-parallel skill. We enabled vision context parallelism at CP=4 on a multi-image dataset where samples carry between 1 and 5 images, and the memory saving is much smaller than on our single-image datasets. Explain why and say where the fix belongs.", + "expected_skill": "nemo-mbridge-perf-vision-context-parallel", + "expected_script": null, + "ground_truth": "The answer should attribute the weak saving to the split being by image count rather than patch count, with the remainder assigned to the last rank, so a wide per-sample image-count spread leaves the tail rank overloaded and peak memory is set by the busiest rank. It should cite the measured Mantis result (CP=4, peak memory 81 -> 77 GB) as smaller than InfoVQA or llava_video (roughly -10%). It should say the fix is a patch-count-balanced greedy split on image boundaries and that it belongs upstream in Megatron-Core, not reimplemented in Bridge, since 3rdparty/Megatron-LM must not be modified in this repo.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-context-parallel skill before answering.", + "Identify the count-based split with remainder-on-last-rank as the cause.", + "Explain that peak memory is set by the busiest rank.", + "Reference the measured Mantis result as the load-imbalance signature.", + "Locate the fix upstream in Megatron-Core rather than in Megatron-Bridge." + ] + }, + { + "id": "vision-context-parallel-negative-hierarchical-cp", + "question": "Our text-only MoE model has 8 KV heads and we need CP=16 for 128K sequences. Which cp_comm_type should we use and what else must be set?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "This is a hierarchical context parallelism question about splitting a long text sequence across ranks, not about sharding vision-encoder images. The vision context parallel skill should not be loaded. A correct answer discusses cp_comm_type=a2a+p2p with hierarchical_context_parallel_sizes whose product equals context_parallel_size, and the KV-head limit on pure a2a.", + "expected_behavior": [ + "Do not load the nemo-mbridge-perf-vision-context-parallel skill.", + "Recognize the question concerns sequence-dimension context parallelism for a text-only model.", + "Route to hierarchical context parallelism guidance instead." + ] + } +] diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md new file mode 100644 index 0000000000..1eb615af78 --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md @@ -0,0 +1,87 @@ +## Description:
+Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache 2.0
+## Use Case:
+Developers and engineers training vision-language models in Megatron-Bridge whose vision tower replicates work and activations on every context-parallel rank while the language model is already sharded, or who need to diagnose why enabling that sharding produced a smaller-than-expected memory saving.
+ +### Deployment Geography for Use:
+Global
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +Risk: The flag silently self-disables at `context_parallel_size=1`, so guidance could be applied to a configuration where it has no effect.
+Mitigation: The skill states the CP>1 requirement in the enablement snippet, the pitfalls list, and the verification success criteria.
+ +## Reference(s):
+- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
+- [Nemotron-3 Omni example README](../../examples/models/nemotron/nemotron_3_omni/README.md)
+- [card.yaml](card.yaml)
+ + +## Skill Output:
+**Output Type(s):** [Configuration instructions, Shell commands, Analysis]
+**Output Format:** [Markdown with inline bash and python code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Evaluation Agents Used:
+- Claude Code (`claude-code`)
+- Codex (`codex`)
+ + + +## Evaluation Tasks:
+Dataset prepared with 3 evaluation tasks (2 positive skill-activation cases, 1 negative routing case) for the NVSkills-Eval external profile. The evaluation has not yet been run.
+ +## Evaluation Metrics Used:
+Reported benchmark dimensions:
+- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
+- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output.
+- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
+- Effectiveness: Checks whether the agent performs measurably better with the skill than without it.
+- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work.
+ +Underlying evaluation signals used in this run:
+- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Verifies that the agent loaded the expected skill and workflow.
+- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage.
+- `accuracy`: Grades final-answer correctness against the reference answer.
+- `goal_accuracy`: Checks whether the overall user task completed successfully.
+- `behavior_check`: Verifies expected behavior steps, including safety expectations.
+- `token_efficiency`: Compares token usage with and without the skill.
+ + + +## Evaluation Results:
+Pending — NVSkills-Eval has not been run against this skill. See [BENCHMARK.md](BENCHMARK.md). Do not hand-author scores into this table.
+ +| Dimension | Num | `claude-code` | `codex` | +|---|---:|---:|---:| +| Security | — | — | — | +| Correctness | — | — | — | +| Discoverability | — | — | — | +| Effectiveness | — | — | — | +| Efficiency | — | — | — | + +## Testing Completed:
+**[ ] Agent Red-Teaming**
+**[ ] Network Security**
+**[ ] Product Security**
+ +## Skill Version(s):
+Unreleased — pending first NVSkills-Eval run and signing.
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
From 88e05adf47550d04b04d234d2654863728241500 Mon Sep 17 00:00:00 2001 From: nvskills-svc-account Date: Tue, 18 Aug 2026 15:48:28 +0000 Subject: [PATCH 3/8] Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account --- .../BENCHMARK.md | 132 ++++++++++-------- .../skill-card.md | 67 ++++----- .../skill.oms.sig | 1 + 3 files changed, 109 insertions(+), 91 deletions(-) create mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md index dfad07e2f0..11c2d945a3 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md @@ -1,85 +1,101 @@ -# Evaluation Report +# Skill Benchmark: nemo-mbridge-perf-vision-context-parallel -Evaluation of the `nemo-mbridge-perf-vision-context-parallel` skill before publication through NVSkills-Eval. +> ✅ **Overall verdict: PASS — Recommended for publication** -This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use. +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. -## Evaluation Summary +## Evaluation Metadata - Skill: `nemo-mbridge-perf-vision-context-parallel` -- Evaluation date: not yet run -- NVSkills-Eval profile: `external` +- Evaluation date: 2026-08-18 +- Evaluator version: `1.2.7` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Tasks: 3 evaluation tasks (2 positive, 1 negative) +- Dataset digest: `sha256:d50a225a6e6651369d09c6c55904396dc1b180aae3e274bd683ea74ed58066f0` (skill-evaluator-dataset-snapshot/1) +- Attempts per task: 1 - Environment: `local` -- Dataset: 3 evaluation tasks -- Attempts per task: 2 -- Pass threshold: 50% -- Overall verdict: **PENDING — NVSkills-Eval has not been run against this skill** +- Tier 3 evidence: required for publication -> This file is a placeholder committed alongside the skill. The result tables -> below are intentionally empty. Populate them by running NVSkills-Eval with the -> `external` profile against `evals/evals.json`, then replace this notice. -> Do not hand-author scores. +Tasks ran on the trusted local host; local mode is not sandboxed. -## Agents Used +## What This Report Answers -- `claude-code` -- `codex` +The three-tier evaluation checks whether the skill: -## Metrics Used +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. -Reported benchmark dimensions: +## Results at a Glance -- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. -- Correctness: checks whether the agent follows the expected workflow and produces the correct final output. -- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant. -- Effectiveness: checks whether the agent performs measurably better with the skill than without it. -- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work. +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 70% → 98% (+28 points) | 79% → 95% (+17 points) | +| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Correctness | 33% → 100% (+67 points) | 87% → 93% (+7 points) | +| Discoverability | 100% → 100% (±0 points) | 73% → 96% (+23 points) | +| Effectiveness | 18% → 88% (+70 points) | 53% → 87% (+34 points) | +| Efficiency | 97% → 100% (+3 points) | 80% → 100% (+20 points) | -Underlying evaluation signals used in this run: +**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. -- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access. -- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow. -- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage. -- `accuracy` (Accuracy): grades final-answer correctness against the reference answer. -- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully. -- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations. -- `token_efficiency` (Token Efficiency): compares token usage with and without the skill. +Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. -## Test Tasks +## Tier Status -The benchmark dataset contains 3 evaluation tasks: +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) | +| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) | -- Positive tasks: 2 tasks where the skill was expected to activate — enablement and the load-imbalance diagnosis. -- Negative tasks: 1 task where no skill was expected — a hierarchical context parallelism question for a text-only model, included because "context parallel" is a shared keyword and the two skills describe unrelated mechanisms. -- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred. +## Findings and Observations -Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases. +
+Show detailed findings and successful checks -## Results +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) +- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) +- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-context-parallel/card.yaml`) -| Dimension | Num | `claude-code` | `codex` | -|---|---:|---:|---:| -| Security | — | — | — | -| Correctness | — | — | — | -| Discoverability | — | — | — | -| Effectiveness | — | — | — | -| Efficiency | — | — | — | +
-Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. +## Scoring Methodology -## Tier 1: Static Validation Summary +
+Show dimension definitions, source signals, and thresholds -Not yet run. +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | -## Tier 2: Deduplication Summary +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. -Not yet run. Deduplication is the tier to watch for this skill: it shares the -"context parallel" vocabulary with `nemo-mbridge-perf-hierarchical-context-parallel` -while describing an unrelated mechanism, which is why the dataset includes an -explicit negative routing task. +Signals present in this run: -## Publication Recommendation +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was found and executed. +- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. + +
+ +## Freshness -Withhold from NVSkills-Eval publication until the evaluation above has been run -and this file updated with real results. The skill content itself is usable in -the repository in the meantime. +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md index 1eb615af78..e626542285 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md @@ -9,76 +9,77 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers training vision-language models in Megatron-Bridge whose vision tower replicates work and activations on every context-parallel rank while the language model is already sharded, or who need to diagnose why enabling that sharding produced a smaller-than-expected memory saving.
+Developers and engineers optimizing Vision-Language Model training throughput and memory usage by sharding the vision encoder across context-parallel ranks in Megatron-Bridge.
### Deployment Geography for Use:
Global
+## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Not Specified]
+**Credential Type(s):** [None identified]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ ## Known Risks and Mitigations:
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
Mitigation: Review and scan skill before deployment.
-Risk: The flag silently self-disables at `context_parallel_size=1`, so guidance could be applied to a configuration where it has no effect.
-Mitigation: The skill states the CP>1 requirement in the enablement snippet, the pitfalls list, and the verification success criteria.
- ## Reference(s):
-- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
-- [Nemotron-3 Omni example README](../../examples/models/nemotron/nemotron_3_omni/README.md)
-- [card.yaml](card.yaml)
+- [SKILL.md](skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md)
+- [Megatron-Bridge Performance Tuning Guide](docs/performance-guide.md)
+- [Megatron-Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
## Skill Output:
**Output Type(s):** [Configuration instructions, Shell commands, Analysis]
-**Output Format:** [Markdown with inline bash and python code blocks]
+**Output Format:** [Markdown with inline bash and Python code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
## Evaluation Agents Used:
-- Claude Code (`claude-code`)
-- Codex (`codex`)
+- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
+- Codex (`openai/openai/gpt-5.5`)
## Evaluation Tasks:
-Dataset prepared with 3 evaluation tasks (2 positive skill-activation cases, 1 negative routing case) for the NVSkills-Eval external profile. The evaluation has not yet been run.
+Evaluated against 3 evaluation tasks (2 positive, 1 negative) using evaluator version 1.2.7.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
-- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output.
-- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
-- Effectiveness: Checks whether the agent performs measurably better with the skill than without it.
-- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work.
+- Security: Whether the skill is safe to use, checking for unsafe operations, secret leakage, and unauthorized access.
+- Correctness: Whether the final answer is correct against the reference answer.
+- Discoverability: Whether the right skill was found and activated when needed.
+- Effectiveness: Whether the skill helped complete the user's goal and followed the expected workflow.
+- Efficiency: Whether the skill avoided wasted tool or skill usage.
Underlying evaluation signals used in this run:
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Verifies that the agent loaded the expected skill and workflow.
-- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage.
-- `accuracy`: Grades final-answer correctness against the reference answer.
-- `goal_accuracy`: Checks whether the overall user task completed successfully.
-- `behavior_check`: Verifies expected behavior steps, including safety expectations.
-- `token_efficiency`: Compares token usage with and without the skill.
+- `skill_execution`: Whether the expected skill was found and executed.
+- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy`: Final-answer correctness against the reference answer.
+- `goal_accuracy`: Whether the user's goal was achieved.
+- `behavior_check`: Whether the expected workflow behavior was followed.
## Evaluation Results:
-Pending — NVSkills-Eval has not been run against this skill. See [BENCHMARK.md](BENCHMARK.md). Do not hand-author scores into this table.
- -| Dimension | Num | `claude-code` | `codex` | -|---|---:|---:|---:| -| Security | — | — | — | -| Correctness | — | — | — | -| Discoverability | — | — | — | -| Effectiveness | — | — | — | -| Efficiency | — | — | — | +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 70% → 98% (+28 points) | 79% → 95% (+17 points) | +| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | +| Correctness | 33% → 100% (+67 points) | 87% → 93% (+7 points) | +| Discoverability | 100% → 100% (±0 points) | 73% → 96% (+23 points) | +| Effectiveness | 18% → 88% (+70 points) | 53% → 87% (+34 points) | +| Efficiency | 97% → 100% (+3 points) | 80% → 100% (+20 points) | ## Testing Completed:
-**[ ] Agent Red-Teaming**
+**[x] Agent Red-Teaming**
**[ ] Network Security**
**[ ] Product Security**
## Skill Version(s):
-Unreleased — pending first NVSkills-Eval run and signing.
+1.0.0+b7643bd (source: pyproject.toml)
## Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig b/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig new file mode 100644 index 0000000000..383280b4af --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWNvbnRleHQtcGFyYWxsZWwiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiZWUyMDZiYWVkNDJiZDNhZjYzMTkzN2Q3NjU3MGQyMjUxMDViNDg0MzgxZDVmZDhjMjMxN2UyMjg4OGYyY2Y2NCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIgogICAgICBdLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAibWV0aG9kIjogImZpbGVzIgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImI4OGRkM2Q1YzY2YzdmYWE1ODNlM2RhNzM5MDY4MGEwZDViYWQwNjEwOWUyODEyOTk4ZjI5ZjNiMDRhYzdlNTgiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMjc3OWM5MTEyZjE4ODE5ODY2YTBhMTUyOTJjOTg4ZDY1OTgzNTk2NjNjMDQ0YzM1NDU2ZTc3ODNmYTZmODMwNyIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImNhcmQueWFtbCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMDgzNWNmNDUyNzg5MWQwNmZkNGNjNmQ0YTIzMGRjZTE4MWZiNjUzZDNlNTY4ZjI3NmU4YWNlMWFkZmEyZmM5MCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjg3Y2RiNTYzMDdlYzI5ZGIxZjZkNjk0ZGE5MmU0NmE1YmIwZTBiNDE1YzJjNTdhOTkwNmFjYmNhZjAyMmQzNGQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI4N2M0MDUwYTMzZjhlZDAzOTk3YWIyMGY1ODYwYzM2ZmI2ZWI5OWM3ODlhNDBjOGMxOGIxNWM2M2JhZTBhY2JkIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQCoWR72SuDRXWU7mUoEKH7f27OYvQcTO4EJYzPuxWUgRhhTC+mM+6Bclkg9cB8JUfMCMCU47W+hGnEJl/V1Au1SWSfb+EDD4U7TrVecjH/s9/fbDC7zmz3Owi/R99OHdKzJWg==","keyid":""}]}} \ No newline at end of file From 31d48a79b3d92bb6b2a44787facf472ce357f5a1 Mon Sep 17 00:00:00 2001 From: Huy Vu Date: Tue, 18 Aug 2026 08:51:18 -0700 Subject: [PATCH 4/8] docs(model): disambiguate vision_context_parallel from dist-train knob Bridge already exposes dist_train.vision_context_parallel_size, an unrelated Qwen3-VL knob that gives the vision tower its own ranks and sets the encoder's own context_parallel_size. Add a comparison table so the two are not mistaken for alternatives, and record that the encoder config intentionally keeps context_parallel_size=1 under this feature because the sharding happens outside the encoder, over its inputs. Drops the README link into the skill tree per review feedback. Signed-off-by: Huy Vu --- .../models/nemotron/nemotron_3_omni/README.md | 3 --- .../SKILL.md | 21 ++++++++++++++++++- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index 558df26cb0..52050b752a 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -292,9 +292,6 @@ Multi-image data whose per-sample image count varies (Mantis at CP=4) therefore balances less evenly than uniform data, which is why its memory saving is the smallest of the three. -For code anchors, pitfalls, and the A/B verification procedure, see -[the vision context parallel skill](../../../../skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md). - ### Image-Text — CORD-V2 [CORD-V2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) is a diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md index 817c1e91c3..76d54fbb4a 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md @@ -22,6 +22,22 @@ borrowing the CP group** — a rank encodes whole images, never partial ones. Shipped for Nemotron Omni. The MCore split/gather helpers are model-agnostic, so the pattern generalizes to any Bridge VLM with a dynamic-resolution encoder. +## Do Not Confuse With `dist_train.vision_context_parallel_size` + +Bridge already has a similarly named and completely different knob: +`vision_context_parallel_size` on the dist-train path +(`src/megatron/bridge/models/qwen_vl/qwen3_vl_provider.py`, consumed in +`src/megatron/bridge/training/initialize.py`). + +| | `model.vision_context_parallel` (bool) | `dist_train.vision_context_parallel_size` (int) | +|---|---|---| +| Ranks | Reuses the language model's CP ranks | Gives the vision tower its **own** ranks | +| Splits | Whole images across those ranks | The vision encoder's own sequence dimension | +| Encoder `context_parallel_size` | Stays `1` | Set to this value | +| Models | Nemotron Omni | Qwen3-VL dist-train | + +They are not alternatives and do not compose in any tested configuration. + ## Enablement ```python @@ -95,7 +111,10 @@ Upstream helpers (do not modify — changes go through the MCore repo): images so no rank runs an empty encoder, and returns `num_padded_ranks` so the gather can drop them again. 3. **Encode.** Each rank runs the full vision tower on its own images only. - Activations, not just compute, are what drop by `1/CP`. + Activations, not just compute, are what drop by `1/CP`. The vision encoder's + own `context_parallel_size` stays hardcoded to `1` in + `_build_vision_config` — that is correct and unchanged by this feature, since + the sharding happens *outside* the encoder, over its inputs. 4. **Even-grid padding.** `_pad_patch_grid_to_even` zero-pads an odd patch grid before pixel shuffle. Only the injected `1x1` placeholders hit this path; real frames are already even in both extents. From a24bdc68d881f77fb727e9c6dd8571634d93366a Mon Sep 17 00:00:00 2001 From: Huy Vu Date: Tue, 25 Aug 2026 08:40:10 -0700 Subject: [PATCH 5/8] docs(model): track the vision_context_parallel default flip to True PR #5645 turned image sharding on by default and moved the sharded projection onto the FP8-alignment helper, which invalidated the enablement guidance, the code anchors, and the A/B baseline arm documented here. Also records that a two-rank unit test now covers the placeholder path CP=8 still lacks. Signed-off-by: Huy Vu --- .../models/nemotron/nemotron_3_omni/README.md | 9 ++--- .../SKILL.md | 36 +++++++++++-------- .../card.yaml | 4 +-- .../evals/evals.json | 4 +-- 4 files changed, 31 insertions(+), 22 deletions(-) diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index 52050b752a..d71c56841d 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -265,7 +265,7 @@ Note:`dataset.task_encoder.use_temporal_video_embedder` only applies to the Ener ### Vision Context Parallelism -`model.vision_context_parallel=True` shards the microbatch's images (or +`model.vision_context_parallel` shards the microbatch's images (or temporal tubelets) across the language model's context-parallel group, so each CP rank encodes `1/CP` of them instead of all of them. This is data parallelism over images, not ring attention — RADIO already attends within each image via @@ -274,9 +274,10 @@ cross-rank attention communication is introduced. The vision projector runs on the rank-local shard, and a single gather afterwards restores the full feature set that every rank needs to merge media into the packed text sequence. -The flag defaults to `False` and is a no-op unless -`model.context_parallel_size > 1`. Both recipes below ship with CP=1, so enable -it only when you raise CP. +The flag defaults to `True` and is a no-op unless +`model.context_parallel_size > 1`. Both recipes below ship with CP=1, so it only +takes effect once you raise CP. Set `model.vision_context_parallel=False` to +encode every image redundantly on every CP rank. Measured on 2 nodes / 16 GPUs (PP=1, DP=2, EP=8, ETP=1, MBS=2, GBS=64, GA=16), flipping only this flag: diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md index 76d54fbb4a..4b61a38b9b 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md @@ -42,19 +42,20 @@ They are not alternatives and do not compose in any tested configuration. ```python cfg.model.context_parallel_size = 2 # the flag is a no-op at CP=1 -cfg.model.vision_context_parallel = True +cfg.model.vision_context_parallel = False # opt out; on by default ``` -Defaults to `False`. It self-disables when `context_parallel_size == 1`, so -turning it on in a recipe that does not raise CP changes nothing and reports no -error. +Defaults to `True`. It self-disables when `context_parallel_size == 1`, so a +recipe that does not raise CP behaves exactly as it did before the flag existed +and reports no error. Opting out is only useful for A/B measurement or to +isolate a suspected split/gather bug. ## Code Anchors Provider field, `src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py`: ```python -vision_context_parallel: bool = False +vision_context_parallel: bool = True ``` Gating, in `NemotronOmniModel.__init__` @@ -86,7 +87,7 @@ Project-then-gather, at the end of `_encode_images`: if shard_vision: # Project before the gather so the projector stays sharded, then # restore the global feature set every CP rank's media merge needs. - projected = self.vision_projection(encoded.unsqueeze(1)).squeeze(1) + projected = _project_multimodal_embeddings(self.vision_projection, encoded) gathered = gather_from_context_parallel_ranks_dynamic_res(projected, num_padded_ranks) return gathered.contiguous() ``` @@ -145,14 +146,17 @@ signature described under Pitfalls, not noise. ## Pitfalls 1. **Silent no-op at CP=1.** The flag ANDs with `context_parallel_lm > 1`. Both - checked-in Nemotron Omni recipes ship CP=1, so enabling it there changes - nothing. Always confirm CP>1 before attributing a result to this flag. + checked-in Nemotron Omni recipes ship CP=1, so the on-by-default value changes + nothing there. Always confirm CP>1 before attributing a result to this flag, + and note that raising CP now silently enables image sharding too. 2. **Count-based split leaves the tail rank hot.** Images are divided by count with the remainder on the last rank. Multi-image data with a wide per-sample image-count spread (e.g. Mantis, 1-5 images/sample at CP=4) balances poorly: peak memory is set by the busiest rank, so the win shrinks. A patch-count-balanced greedy split on image boundaries is the fix, and it - belongs **upstream in MCore**, not reimplemented in Bridge. + belongs **upstream in MCore**, not reimplemented in Bridge. Since the flag is + on by default, this imbalance is what a multi-image CP>1 run gets unless the + user opts out. 3. **Process-group divergence is checked, not assumed.** The MCore splitter resolves the CP group from global parallel state, so a `ProcessGroupCollection` whose `cp` size disagrees with `context_parallel_size` would shard against the @@ -166,16 +170,20 @@ signature described under Pitfalls, not noise. 5. **Do not gather before projecting.** Gathering encoder output first would replicate the projector on every rank and move a larger tensor. The ordering is deliberate. -6. **CP=8 placeholder path is unvalidated end-to-end.** Unit tests cover the - even-grid padding, but no multi-node run has exercised a microbatch with fewer - images than ranks. +6. **The placeholder path is unvalidated end-to-end.** A two-rank unit test + proves the sharded encoder matches the unsharded one when a microbatch holds + fewer images than ranks, but no multi-node training run has exercised it, and + nothing covers CP=8. ## Verification -Unit tests for the padding helper: +Unit tests. The second file self-launches a two-rank `torch.distributed` job and +asserts the sharded tower matches the unsharded one for fewer images than ranks, +an exact split, and a remainder; it skips when fewer than two GPUs are visible: ```bash uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py -q +uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_vision_context_parallel_distributed.py -q ``` Multi-GPU A/B. Run the same recipe twice at CP>1, flipping only the flag, and @@ -186,7 +194,7 @@ uv run python -m torch.distributed.run --nproc_per_node=8 \ scripts/training/run_recipe.py \ --recipe nemotron_omni_cord_v2_sft_config \ model.context_parallel_size=2 \ - model.vision_context_parallel=true \ + model.vision_context_parallel=false \ train.train_iters=20 ``` diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml b/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml index 441f15bf8f..2dcdd94d59 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml +++ b/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml @@ -6,8 +6,8 @@ summary: > microbatch instead of all of it. This is data parallelism over images reusing the CP group, not ring attention: the encoder attends within each image via cu_seqlens, so no sequence is split and the only added collective is the - post-projector gather. Opt-in through model.vision_context_parallel, default - False, and a silent no-op at context_parallel_size=1. + post-projector gather. Controlled by model.vision_context_parallel, default + True, and a silent no-op at context_parallel_size=1. validation_status: upstream_mcore_split_gather_helpers: - code_verified diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json b/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json index 6d44880494..21ad4c68eb 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json +++ b/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json @@ -4,10 +4,10 @@ "question": "Use the nemo-mbridge-perf-vision-context-parallel skill. Our Nemotron Omni VLM run at CP=2 shows the vision encoder doing identical work on every context-parallel rank. Give the exact Bridge config values to shard it, explain what collective this adds, and state how to prove the flag actually took effect.", "expected_skill": "nemo-mbridge-perf-vision-context-parallel", "expected_script": null, - "ground_truth": "The answer should use the vision context parallel skill. It should set cfg.model.vision_context_parallel=True with cfg.model.context_parallel_size>1, and note the flag defaults to False and self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", + "ground_truth": "The answer should use the vision context parallel skill. It should note that cfg.model.vision_context_parallel already defaults to True, so the only required change is cfg.model.context_parallel_size>1, and that the flag self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", "expected_behavior": [ "Read the nemo-mbridge-perf-vision-context-parallel skill before answering.", - "Set model.vision_context_parallel=True and require context_parallel_size>1.", + "State that model.vision_context_parallel is already True by default and require context_parallel_size>1.", "State that the flag is a silent no-op at CP=1.", "Explain the split is over whole images, not over the sequence dimension.", "Identify the post-projector gather as the only added collective, and that projection happens before the gather.", From 319ba51a221ea89d1a410848ba3b423f1b3bfe5a Mon Sep 17 00:00:00 2001 From: nvskills-svc-account Date: Tue, 25 Aug 2026 16:05:01 +0000 Subject: [PATCH 6/8] Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account --- .../BENCHMARK.md | 29 ++++++++++----- .../skill-card.md | 35 ++++++++++--------- .../skill.oms.sig | 2 +- 3 files changed, 39 insertions(+), 27 deletions(-) diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md index 11c2d945a3..fdd9d1c9d3 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md @@ -9,17 +9,28 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nemo-mbridge-perf-vision-context-parallel` -- Evaluation date: 2026-08-18 -- Evaluator version: `1.2.7` +- Evaluation date: 2026-08-25 +- Evaluator version: `1.3.2` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) - Tasks: 3 evaluation tasks (2 positive, 1 negative) -- Dataset digest: `sha256:d50a225a6e6651369d09c6c55904396dc1b180aae3e274bd683ea74ed58066f0` (skill-evaluator-dataset-snapshot/1) +- Dataset digest: `sha256:2040e9e066e5932a1594234dfa6b37b8c793fdfc0c05a80edaa212b85b18e921` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 - Environment: `local` - Tier 3 evidence: required for publication Tasks ran on the trusted local host; local mode is not sandboxed. +## Execution and Provenance + +- Validation status: `passed` +- Report generation: `complete` +- Evaluator version: `1.3.2` +- Git commit: `0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Content type: requested `auto`, detected `skill` +- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Container image digest: `not recorded` +- Tier 3: requested `true`, executed `true`, status `succeeded` + ## What This Report Answers The three-tier evaluation checks whether the skill: @@ -34,12 +45,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 70% → 98% (+28 points) | 79% → 95% (+17 points) | -| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 33% → 100% (+67 points) | 87% → 93% (+7 points) | -| Discoverability | 100% → 100% (±0 points) | 73% → 96% (+23 points) | -| Effectiveness | 18% → 88% (+70 points) | 53% → 87% (+34 points) | -| Efficiency | 97% → 100% (+3 points) | 80% → 100% (+20 points) | +| Overall | 71% → 98% (+27 points) | 56% → 94% (+39 points) | +| Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | +| Correctness | 27% → 93% (+67 points) | 60% → 93% (+33 points) | +| Discoverability | 100% → 100% (±0 points) | 58% → 96% (+38 points) | +| Effectiveness | 30% → 98% (+68 points) | 40% → 81% (+41 points) | +| Efficiency | 98% → 100% (+2 points) | 53% → 100% (+47 points) | **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md index e626542285..e7bc769b86 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md +++ b/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md @@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers optimizing Vision-Language Model training throughput and memory usage by sharding the vision encoder across context-parallel ranks in Megatron-Bridge.
+Developers and engineers optimizing Vision-Language Model (VLM) training throughput and memory usage when running with context parallelism greater than 1 in Megatron-Bridge.
### Deployment Geography for Use:
Global
@@ -25,14 +25,15 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [SKILL.md](skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md)
- [Megatron-Bridge Performance Tuning Guide](docs/performance-guide.md)
+- [Nemotron Omni Modeling Implementation](src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py)
+- [MCore Context Parallel Helpers](3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py)
- [Megatron-Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
## Skill Output:
-**Output Type(s):** [Configuration instructions, Shell commands, Analysis]
-**Output Format:** [Markdown with inline bash and Python code blocks]
+**Output Type(s):** [Configuration instructions, Analysis, Shell commands]
+**Output Format:** [Markdown with inline code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
@@ -43,35 +44,35 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-Evaluated against 3 evaluation tasks (2 positive, 1 negative) using evaluator version 1.2.7.
+Evaluated against 3 evaluation tasks (2 positive, 1 negative) from a curated skill-evaluator dataset.
## Evaluation Metrics Used:
Reported benchmark dimensions:
- Security: Whether the skill is safe to use, checking for unsafe operations, secret leakage, and unauthorized access.
-- Correctness: Whether the final answer is correct against the reference answer.
-- Discoverability: Whether the right skill was found and activated when needed.
-- Effectiveness: Whether the skill helped complete the user's goal and followed the expected workflow.
-- Efficiency: Whether the skill avoided wasted tool or skill usage.
+- Correctness: Whether the skill produces correct answers against the reference answer.
+- Discoverability: Whether the right skill is loaded and activated when needed.
+- Effectiveness: Whether the skill helps the agent complete the user's goal and expected workflow (equal-weight mean of goal completion and behavior adherence).
+- Efficiency: Whether the skill avoids wasted tool or skill usage, measuring routing quality and productive tool use.
Underlying evaluation signals used in this run:
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Whether the expected skill was found and executed.
-- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
- `accuracy`: Final-answer correctness against the reference answer.
+- `skill_execution`: Whether the expected skill was found and executed.
- `goal_accuracy`: Whether the user's goal was achieved.
- `behavior_check`: Whether the expected workflow behavior was followed.
+- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
## Evaluation Results:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 70% → 98% (+28 points) | 79% → 95% (+17 points) | -| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | -| Correctness | 33% → 100% (+67 points) | 87% → 93% (+7 points) | -| Discoverability | 100% → 100% (±0 points) | 73% → 96% (+23 points) | -| Effectiveness | 18% → 88% (+70 points) | 53% → 87% (+34 points) | -| Efficiency | 97% → 100% (+3 points) | 80% → 100% (+20 points) | +| Overall | 71% → 98% (+27 points) | 56% → 94% (+39 points) | +| Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | +| Correctness | 27% → 93% (+67 points) | 60% → 93% (+33 points) | +| Discoverability | 100% → 100% (±0 points) | 58% → 96% (+38 points) | +| Effectiveness | 30% → 98% (+68 points) | 40% → 81% (+41 points) | +| Efficiency | 98% → 100% (+2 points) | 53% → 100% (+47 points) | ## Testing Completed:
**[x] Agent Red-Teaming**
diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig b/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig index 383280b4af..358787c9f0 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig +++ b/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWNvbnRleHQtcGFyYWxsZWwiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiZWUyMDZiYWVkNDJiZDNhZjYzMTkzN2Q3NjU3MGQyMjUxMDViNDg0MzgxZDVmZDhjMjMxN2UyMjg4OGYyY2Y2NCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIgogICAgICBdLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAibWV0aG9kIjogImZpbGVzIgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImI4OGRkM2Q1YzY2YzdmYWE1ODNlM2RhNzM5MDY4MGEwZDViYWQwNjEwOWUyODEyOTk4ZjI5ZjNiMDRhYzdlNTgiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMjc3OWM5MTEyZjE4ODE5ODY2YTBhMTUyOTJjOTg4ZDY1OTgzNTk2NjNjMDQ0YzM1NDU2ZTc3ODNmYTZmODMwNyIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImNhcmQueWFtbCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMDgzNWNmNDUyNzg5MWQwNmZkNGNjNmQ0YTIzMGRjZTE4MWZiNjUzZDNlNTY4ZjI3NmU4YWNlMWFkZmEyZmM5MCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjg3Y2RiNTYzMDdlYzI5ZGIxZjZkNjk0ZGE5MmU0NmE1YmIwZTBiNDE1YzJjNTdhOTkwNmFjYmNhZjAyMmQzNGQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI4N2M0MDUwYTMzZjhlZDAzOTk3YWIyMGY1ODYwYzM2ZmI2ZWI5OWM3ODlhNDBjOGMxOGIxNWM2M2JhZTBhY2JkIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQCoWR72SuDRXWU7mUoEKH7f27OYvQcTO4EJYzPuxWUgRhhTC+mM+6Bclkg9cB8JUfMCMCU47W+hGnEJl/V1Au1SWSfb+EDD4U7TrVecjH/s9/fbDC7zmz3Owi/R99OHdKzJWg==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWNvbnRleHQtcGFyYWxsZWwiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiODU2MmU5OThjZDVhNzgyMWRhN2MxNzFjNTdmYjQwZTdhZDRkODZhZDY0NWUyNzBhMzk3ZTdkZDJhZWFmY2QyNCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdGh1YiIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiLAogICAgICAgICJkaWdlc3QiOiAiMzNmOWJjYTJmOTU1YmFjYWFiZGNjNzk3OGMyYTRmZmFlNGYyMzI0ODg1ZDY3ZTBlZjQxYTUxZWNjYzU2MjJlMCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIwNzg0MzJjMmFkYTllZmM2NGFhOTUxMWM2NDA1YTYyNzRlNjM5OWIzNjNhNTExN2IwYjFlYjFlY2MwMTkwNjE1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImNhcmQueWFtbCIsCiAgICAgICAgImRpZ2VzdCI6ICJmYjM0YmViZmMyMjg5MDBmNTAxMmZkMDQ3M2RmZmQzY2NlZmUzYTkzYTc3NWVmOTliZDNiMGI2YjNiODM3MDA1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJkaWdlc3QiOiAiNjgzMWNlZjlmYTE4Y2FlYTQ3MmVkYjVhNzc2MTcyYWMyYzY1MmUzNjY2MzUwZjhkN2ViNGVhOGViMDNmNGZjZiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjM1ZjMzOGE5NzEwZWVlYThlOThkMjgzNDM0OWMxZTUyYWI0OTEwYjg4MDY3ZmM2NTNmYjg3YmVlNDVkZDlhODUiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDoHtASj9vKQuomMMPDzUv9rNjNUb910wPq75IUQFrLojPw3+YYxSfbUpOuVEV/NR4CMCe4ys0s1z1yb3Jz8/hChZ0o/HEIpfaIgobphMleU9DdssdOPplC9eB30tniNrwXCw==","keyid":""}]}} \ No newline at end of file From f47757652d2944eb9c5ebde3e01c000878807b4c Mon Sep 17 00:00:00 2001 From: Huy Vu Date: Thu, 27 Aug 2026 07:14:32 -0700 Subject: [PATCH 7/8] docs(model): rename vision_context_parallel to vision_dp_over_cp Follows the flag rename in the implementation PR. The skill directory becomes nemo-mbridge-perf-vision-dp-over-cp; skill.oms.sig is now stale (subject name and file digests both changed) and needs an NVSkills pipeline re-run. Signed-off-by: Huy Vu --- docs/skills-index.md | 2 +- .../models/nemotron/nemotron_3_omni/README.md | 6 +-- .../evals/evals.json | 43 ------------------- .../BENCHMARK.md | 12 +++--- .../SKILL.md | 26 +++++------ .../card.yaml | 14 +++--- .../evals/evals.json | 43 +++++++++++++++++++ .../skill-card.md | 0 .../skill.oms.sig | 0 9 files changed, 73 insertions(+), 73 deletions(-) delete mode 100644 skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json rename skills/{nemo-mbridge-perf-vision-context-parallel => nemo-mbridge-perf-vision-dp-over-cp}/BENCHMARK.md (92%) rename skills/{nemo-mbridge-perf-vision-context-parallel => nemo-mbridge-perf-vision-dp-over-cp}/SKILL.md (91%) rename skills/{nemo-mbridge-perf-vision-context-parallel => nemo-mbridge-perf-vision-dp-over-cp}/card.yaml (91%) create mode 100644 skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json rename skills/{nemo-mbridge-perf-vision-context-parallel => nemo-mbridge-perf-vision-dp-over-cp}/skill-card.md (100%) rename skills/{nemo-mbridge-perf-vision-context-parallel => nemo-mbridge-perf-vision-dp-over-cp}/skill.oms.sig (100%) diff --git a/docs/skills-index.md b/docs/skills-index.md index 25c32ff3a9..49b5461b13 100644 --- a/docs/skills-index.md +++ b/docs/skills-index.md @@ -50,7 +50,7 @@ skills/nemo-mbridge-perf-tp-dp-comm-overlap/SKILL skills/nemo-mbridge-perf-megatron-fsdp/SKILL skills/nemo-mbridge-perf-sequence-packing/SKILL skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL -skills/nemo-mbridge-perf-vision-context-parallel/SKILL +skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL skills/nemo-mbridge-perf-expert-parallel-overlap/SKILL skills/nemo-mbridge-perf-moe-comm-overlap/SKILL skills/nemo-mbridge-perf-activation-recompute/SKILL diff --git a/examples/models/nemotron/nemotron_3_omni/README.md b/examples/models/nemotron/nemotron_3_omni/README.md index d71c56841d..eb5e3ae43d 100644 --- a/examples/models/nemotron/nemotron_3_omni/README.md +++ b/examples/models/nemotron/nemotron_3_omni/README.md @@ -263,9 +263,9 @@ Set the four flags below as a matched column — a mismatched set produces incor Note:`dataset.task_encoder.use_temporal_video_embedder` only applies to the Energon data path. -### Vision Context Parallelism +### Vision DP Over CP -`model.vision_context_parallel` shards the microbatch's images (or +`model.vision_dp_over_cp` shards the microbatch's images (or temporal tubelets) across the language model's context-parallel group, so each CP rank encodes `1/CP` of them instead of all of them. This is data parallelism over images, not ring attention — RADIO already attends within each image via @@ -276,7 +276,7 @@ set that every rank needs to merge media into the packed text sequence. The flag defaults to `True` and is a no-op unless `model.context_parallel_size > 1`. Both recipes below ship with CP=1, so it only -takes effect once you raise CP. Set `model.vision_context_parallel=False` to +takes effect once you raise CP. Set `model.vision_dp_over_cp=False` to encode every image redundantly on every CP rank. Measured on 2 nodes / 16 GPUs (PP=1, DP=2, EP=8, ETP=1, MBS=2, GBS=64, GA=16), diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json b/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json deleted file mode 100644 index 21ad4c68eb..0000000000 --- a/skills/nemo-mbridge-perf-vision-context-parallel/evals/evals.json +++ /dev/null @@ -1,43 +0,0 @@ -[ - { - "id": "vision-context-parallel-positive-enablement", - "question": "Use the nemo-mbridge-perf-vision-context-parallel skill. Our Nemotron Omni VLM run at CP=2 shows the vision encoder doing identical work on every context-parallel rank. Give the exact Bridge config values to shard it, explain what collective this adds, and state how to prove the flag actually took effect.", - "expected_skill": "nemo-mbridge-perf-vision-context-parallel", - "expected_script": null, - "ground_truth": "The answer should use the vision context parallel skill. It should note that cfg.model.vision_context_parallel already defaults to True, so the only required change is cfg.model.context_parallel_size>1, and that the flag self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", - "expected_behavior": [ - "Read the nemo-mbridge-perf-vision-context-parallel skill before answering.", - "State that model.vision_context_parallel is already True by default and require context_parallel_size>1.", - "State that the flag is a silent no-op at CP=1.", - "Explain the split is over whole images, not over the sequence dimension.", - "Identify the post-projector gather as the only added collective, and that projection happens before the gather.", - "Propose an A/B verification comparing peak memory, step time, and loss parity." - ] - }, - { - "id": "vision-context-parallel-positive-load-imbalance", - "question": "Use the nemo-mbridge-perf-vision-context-parallel skill. We enabled vision context parallelism at CP=4 on a multi-image dataset where samples carry between 1 and 5 images, and the memory saving is much smaller than on our single-image datasets. Explain why and say where the fix belongs.", - "expected_skill": "nemo-mbridge-perf-vision-context-parallel", - "expected_script": null, - "ground_truth": "The answer should attribute the weak saving to the split being by image count rather than patch count, with the remainder assigned to the last rank, so a wide per-sample image-count spread leaves the tail rank overloaded and peak memory is set by the busiest rank. It should cite the measured Mantis result (CP=4, peak memory 81 -> 77 GB) as smaller than InfoVQA or llava_video (roughly -10%). It should say the fix is a patch-count-balanced greedy split on image boundaries and that it belongs upstream in Megatron-Core, not reimplemented in Bridge, since 3rdparty/Megatron-LM must not be modified in this repo.", - "expected_behavior": [ - "Read the nemo-mbridge-perf-vision-context-parallel skill before answering.", - "Identify the count-based split with remainder-on-last-rank as the cause.", - "Explain that peak memory is set by the busiest rank.", - "Reference the measured Mantis result as the load-imbalance signature.", - "Locate the fix upstream in Megatron-Core rather than in Megatron-Bridge." - ] - }, - { - "id": "vision-context-parallel-negative-hierarchical-cp", - "question": "Our text-only MoE model has 8 KV heads and we need CP=16 for 128K sequences. Which cp_comm_type should we use and what else must be set?", - "expected_skill": null, - "expected_script": null, - "ground_truth": "This is a hierarchical context parallelism question about splitting a long text sequence across ranks, not about sharding vision-encoder images. The vision context parallel skill should not be loaded. A correct answer discusses cp_comm_type=a2a+p2p with hierarchical_context_parallel_sizes whose product equals context_parallel_size, and the KV-head limit on pure a2a.", - "expected_behavior": [ - "Do not load the nemo-mbridge-perf-vision-context-parallel skill.", - "Recognize the question concerns sequence-dimension context parallelism for a text-only model.", - "Route to hierarchical context parallelism guidance instead." - ] - } -] diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md similarity index 92% rename from skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md rename to skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md index fdd9d1c9d3..d3976b9899 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md @@ -1,4 +1,4 @@ -# Skill Benchmark: nemo-mbridge-perf-vision-context-parallel +# Skill Benchmark: nemo-mbridge-perf-vision-dp-over-cp > ✅ **Overall verdict: PASS — Recommended for publication** @@ -8,7 +8,7 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata -- Skill: `nemo-mbridge-perf-vision-context-parallel` +- Skill: `nemo-mbridge-perf-vision-dp-over-cp` - Evaluation date: 2026-08-25 - Evaluator version: `1.3.2` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) @@ -69,10 +69,10 @@ Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45
Show detailed findings and successful checks -- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) -- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) -- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md`) -- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-context-parallel/card.yaml`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`) +- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml`)
diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md similarity index 91% rename from skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md rename to skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md index 4b61a38b9b..89d8900fa2 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/SKILL.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md @@ -1,13 +1,13 @@ --- -name: nemo-mbridge-perf-vision-context-parallel +name: nemo-mbridge-perf-vision-dp-over-cp description: Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact. license: Apache-2.0 -when_to_use: The vision tower replicates work and activations on every CP rank while the language model is already sharded, or a VLM run at CP>1 OOMs in the encoder; 'vision_context_parallel', 'vision CP', 'vision DP over CP', 'image sharding across CP ranks', 'encoder replicated across CP'. +when_to_use: The vision tower replicates work and activations on every CP rank while the language model is already sharded, or a VLM run at CP>1 OOMs in the encoder; 'vision_dp_over_cp', 'vision CP', 'vision DP over CP', 'image sharding across CP ranks', 'encoder replicated across CP'. --- -# Vision Context Parallel Skill +# Vision DP Over CP Skill -This skill covers `vision_context_parallel`: reusing the language model's +This skill covers `vision_dp_over_cp`: reusing the language model's context-parallel process group to shard the **images** (or temporal tubelets) of a microbatch across CP ranks, so each rank encodes `1/CP` of them instead of all of them. @@ -29,7 +29,7 @@ Bridge already has a similarly named and completely different knob: (`src/megatron/bridge/models/qwen_vl/qwen3_vl_provider.py`, consumed in `src/megatron/bridge/training/initialize.py`). -| | `model.vision_context_parallel` (bool) | `dist_train.vision_context_parallel_size` (int) | +| | `model.vision_dp_over_cp` (bool) | `dist_train.vision_context_parallel_size` (int) | |---|---|---| | Ranks | Reuses the language model's CP ranks | Gives the vision tower its **own** ranks | | Splits | Whole images across those ranks | The vision encoder's own sequence dimension | @@ -42,7 +42,7 @@ They are not alternatives and do not compose in any tested configuration. ```python cfg.model.context_parallel_size = 2 # the flag is a no-op at CP=1 -cfg.model.vision_context_parallel = False # opt out; on by default +cfg.model.vision_dp_over_cp = False # opt out; on by default ``` Defaults to `True`. It self-disables when `context_parallel_size == 1`, so a @@ -55,7 +55,7 @@ isolate a suspected split/gather bug. Provider field, `src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py`: ```python -vision_context_parallel: bool = True +vision_dp_over_cp: bool = True ``` Gating, in `NemotronOmniModel.__init__` @@ -63,7 +63,7 @@ Gating, in `NemotronOmniModel.__init__` ```python self.context_parallel_lm = language_transformer_config.context_parallel_size -self.vision_context_parallel = vision_context_parallel and self.context_parallel_lm > 1 +self.vision_dp_over_cp = vision_dp_over_cp and self.context_parallel_lm > 1 ``` The split itself wraps an upstream MCore helper: @@ -128,7 +128,7 @@ Upstream helpers (do not modify — changes go through the MCore repo): ## Measured Impact Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 -GBS=64 GA=16, with only `vision_context_parallel` flipped: +GBS=64 GA=16, with only `vision_dp_over_cp` flipped: | Dataset | TP | CP | Step time | Peak memory | |-------------|----|----|---------------|-------------| @@ -161,7 +161,7 @@ signature described under Pitfalls, not noise. resolves the CP group from global parallel state, so a `ProcessGroupCollection` whose `cp` size disagrees with `context_parallel_size` would shard against the wrong ranks. Bridge raises instead: - `"Nemotron Omni vision context parallelism does not match its process group"`. + `"Nemotron Omni vision_dp_over_cp does not match its process group"`. 4. **Odd patch grids break pixel shuffle.** Omni's factor-2 pixel shuffle rejects an odd grid, and MCore's `1x1` placeholder images are odd by construction. Without `_pad_patch_grid_to_even`, the CP>images case fails only for the @@ -183,7 +183,7 @@ an exact split, and a remainder; it skips when fewer than two GPUs are visible: ```bash uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py -q -uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_vision_context_parallel_distributed.py -q +uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_vision_dp_over_cp_distributed.py -q ``` Multi-GPU A/B. Run the same recipe twice at CP>1, flipping only the flag, and @@ -194,7 +194,7 @@ uv run python -m torch.distributed.run --nproc_per_node=8 \ scripts/training/run_recipe.py \ --recipe nemotron_omni_cord_v2_sft_config \ model.context_parallel_size=2 \ - model.vision_context_parallel=false \ + model.vision_dp_over_cp=false \ train.train_iters=20 ``` @@ -214,4 +214,4 @@ Success criteria: - @skills/nemo-mbridge-perf-memory-tuning/SKILL.md - @skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL.md — the *sequence*-splitting use of the same process group; unrelated mechanism -- @skills/nemo-mbridge-perf-vision-context-parallel/card.yaml +- @skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml similarity index 91% rename from skills/nemo-mbridge-perf-vision-context-parallel/card.yaml rename to skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml index 2dcdd94d59..242cd5b5c2 100644 --- a/skills/nemo-mbridge-perf-vision-context-parallel/card.yaml +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml @@ -1,4 +1,4 @@ -title: vision_context_parallel +title: vision_dp_over_cp validated_on: "2026-08-18" summary: > Megatron-Bridge can shard a VLM's images or temporal tubelets across the @@ -6,7 +6,7 @@ summary: > microbatch instead of all of it. This is data parallelism over images reusing the CP group, not ring attention: the encoder attends within each image via cu_seqlens, so no sequence is split and the only added collective is the - post-projector gather. Controlled by model.vision_context_parallel, default + post-projector gather. Controlled by model.vision_dp_over_cp, default True, and a silent no-op at context_parallel_size=1. validation_status: upstream_mcore_split_gather_helpers: @@ -26,7 +26,7 @@ validation_status: bridge_vision_cp_docs: - doc_only feature_meaning: - vision_context_parallel: > + vision_dp_over_cp: > Bridge model flag that shards the microbatch's images or tubelets across the language model's CP ranks. Whole images only; no image is split. split_to_context_parallel_ranks_dynamic_res: > @@ -42,11 +42,11 @@ feature_meaning: encoder. They are removed during the gather. recommended_path: model.context_parallel_size: 2 - model.vision_context_parallel: true + model.vision_dp_over_cp: true measured_impact: setup: > Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2 - GBS=64 GA=16, only vision_context_parallel flipped. + GBS=64 GA=16, only vision_dp_over_cp flipped. results: - dataset: InfoVQA tp: 4 @@ -77,7 +77,7 @@ known_constraints: known_limitations: - Images are split by count with the remainder on the last rank, so multi-image data with a wide image-count spread balances unevenly. - The CP=8 case where a microbatch holds fewer images than ranks has unit tests but no end-to-end run. - - No checked-in Bridge recipe enables vision_context_parallel; both Nemotron Omni recipes ship CP=1. + - No checked-in Bridge recipe enables vision_dp_over_cp; both Nemotron Omni recipes ship CP=1. - Only wired into Nemotron Omni today, although the MCore helpers are model-agnostic. evidence: - src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py @@ -89,5 +89,5 @@ evidence: follow_up_validation: - Run an end-to-end job where the microbatch holds fewer images than CP ranks to exercise placeholder padding. - Replace the count-based split with a patch-count-balanced greedy split on image boundaries, upstream in Megatron-Core. - - Add a Bridge functional test that completes a training step with vision_context_parallel=true at CP=2. + - Add a Bridge functional test that completes a training step with vision_dp_over_cp=true at CP=2. - Extend the flag to other Bridge VLM families using the same MCore helpers. diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json b/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json new file mode 100644 index 0000000000..0b18b0ab2a --- /dev/null +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/evals/evals.json @@ -0,0 +1,43 @@ +[ + { + "id": "vision-dp-over-cp-positive-enablement", + "question": "Use the nemo-mbridge-perf-vision-dp-over-cp skill. Our Nemotron Omni VLM run at CP=2 shows the vision encoder doing identical work on every context-parallel rank. Give the exact Bridge config values to shard it, explain what collective this adds, and state how to prove the flag actually took effect.", + "expected_skill": "nemo-mbridge-perf-vision-dp-over-cp", + "expected_script": null, + "ground_truth": "The answer should use the vision DP over CP skill. It should note that cfg.model.vision_dp_over_cp already defaults to True, so the only required change is cfg.model.context_parallel_size>1, and that the flag self-disables silently at CP=1. It should explain that images or tubelets are split whole across the CP ranks (data parallelism over images, not ring attention, because the encoder attends within each image via cu_seqlens), that the vision projector runs on the rank-local shard, and that the only added collective is the post-projector gather. Verification should be an A/B at CP>1 flipping only the flag, expecting lower peak memory with no step-time regression and matching loss curves, plus a reminder to confirm context_parallel_size>1 if nothing changes.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-dp-over-cp skill before answering.", + "State that model.vision_dp_over_cp is already True by default and require context_parallel_size>1.", + "State that the flag is a silent no-op at CP=1.", + "Explain the split is over whole images, not over the sequence dimension.", + "Identify the post-projector gather as the only added collective, and that projection happens before the gather.", + "Propose an A/B verification comparing peak memory, step time, and loss parity." + ] + }, + { + "id": "vision-dp-over-cp-positive-load-imbalance", + "question": "Use the nemo-mbridge-perf-vision-dp-over-cp skill. We enabled vision DP over CP at CP=4 on a multi-image dataset where samples carry between 1 and 5 images, and the memory saving is much smaller than on our single-image datasets. Explain why and say where the fix belongs.", + "expected_skill": "nemo-mbridge-perf-vision-dp-over-cp", + "expected_script": null, + "ground_truth": "The answer should attribute the weak saving to the split being by image count rather than patch count, with the remainder assigned to the last rank, so a wide per-sample image-count spread leaves the tail rank overloaded and peak memory is set by the busiest rank. It should cite the measured Mantis result (CP=4, peak memory 81 -> 77 GB) as smaller than InfoVQA or llava_video (roughly -10%). It should say the fix is a patch-count-balanced greedy split on image boundaries and that it belongs upstream in Megatron-Core, not reimplemented in Bridge, since 3rdparty/Megatron-LM must not be modified in this repo.", + "expected_behavior": [ + "Read the nemo-mbridge-perf-vision-dp-over-cp skill before answering.", + "Identify the count-based split with remainder-on-last-rank as the cause.", + "Explain that peak memory is set by the busiest rank.", + "Reference the measured Mantis result as the load-imbalance signature.", + "Locate the fix upstream in Megatron-Core rather than in Megatron-Bridge." + ] + }, + { + "id": "vision-dp-over-cp-negative-hierarchical-cp", + "question": "Our text-only MoE model has 8 KV heads and we need CP=16 for 128K sequences. Which cp_comm_type should we use and what else must be set?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "This is a hierarchical context parallelism question about splitting a long text sequence across ranks, not about sharding vision-encoder images. The vision DP over CP skill should not be loaded. A correct answer discusses cp_comm_type=a2a+p2p with hierarchical_context_parallel_sizes whose product equals context_parallel_size, and the KV-head limit on pure a2a.", + "expected_behavior": [ + "Do not load the nemo-mbridge-perf-vision-dp-over-cp skill.", + "Recognize the question concerns sequence-dimension context parallelism for a text-only model.", + "Route to hierarchical context parallelism guidance instead." + ] + } +] diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md similarity index 100% rename from skills/nemo-mbridge-perf-vision-context-parallel/skill-card.md rename to skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md diff --git a/skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig similarity index 100% rename from skills/nemo-mbridge-perf-vision-context-parallel/skill.oms.sig rename to skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig From 155154e5d4cdc5fe7af4873dd9206f227f825a8d Mon Sep 17 00:00:00 2001 From: nvskills-svc-account Date: Thu, 27 Aug 2026 14:31:25 +0000 Subject: [PATCH 8/8] Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account --- .../BENCHMARK.md | 22 +++++------ .../skill-card.md | 39 +++++++++---------- .../skill.oms.sig | 2 +- 3 files changed, 31 insertions(+), 32 deletions(-) diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md index d3976b9899..ee4dfcd337 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md @@ -9,25 +9,25 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nemo-mbridge-perf-vision-dp-over-cp` -- Evaluation date: 2026-08-25 +- Evaluation date: 2026-08-27 - Evaluator version: `1.3.2` - Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) - Tasks: 3 evaluation tasks (2 positive, 1 negative) -- Dataset digest: `sha256:2040e9e066e5932a1594234dfa6b37b8c793fdfc0c05a80edaa212b85b18e921` (skill-evaluator-dataset-snapshot/1) +- Dataset digest: `sha256:1f686f2a6deb8b2b442751f0639aafb30c6e89513befad8532f4ef5754f59593` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 -- Environment: `local` +- Environment: `k8s-sandbox` - Tier 3 evidence: required for publication -Tasks ran on the trusted local host; local mode is not sandboxed. +Each task attempt ran in its own isolated sandbox pod. ## Execution and Provenance - Validation status: `passed` - Report generation: `complete` - Evaluator version: `1.3.2` -- Git commit: `0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Git commit: `eda36f54e38357fe24c321a2ffae85589de9bd48` - Content type: requested `auto`, detected `skill` -- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-0117bc2e3e54da4244a656466526c5b1b5a559ea` +- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-eda36f54e38357fe24c321a2ffae85589de9bd48` - Container image digest: `not recorded` - Tier 3: requested `true`, executed `true`, status `succeeded` @@ -45,12 +45,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 71% → 98% (+27 points) | 56% → 94% (+39 points) | +| Overall | 67% → 100% (+32 points) | 59% → 95% (+36 points) | | Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | -| Correctness | 27% → 93% (+67 points) | 60% → 93% (+33 points) | -| Discoverability | 100% → 100% (±0 points) | 58% → 96% (+38 points) | -| Effectiveness | 30% → 98% (+68 points) | 40% → 81% (+41 points) | -| Efficiency | 98% → 100% (+2 points) | 53% → 100% (+47 points) | +| Correctness | 47% → 100% (+53 points) | 60% → 93% (+33 points) | +| Discoverability | 83% → 100% (+17 points) | 58% → 96% (+38 points) | +| Effectiveness | 34% → 99% (+65 points) | 50% → 88% (+38 points) | +| Efficiency | 73% → 100% (+27 points) | 62% → 100% (+38 points) | **How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md index e7bc769b86..848868ce1a 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md @@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and engineers optimizing Vision-Language Model (VLM) training throughput and memory usage when running with context parallelism greater than 1 in Megatron-Bridge.
+Developers and engineers optimizing VLM training performance who need to reduce peak memory and improve throughput by sharding the vision encoder's image inputs across context-parallel ranks in Megatron-Bridge.
### Deployment Geography for Use:
Global
@@ -25,15 +25,14 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [Megatron-Bridge Performance Tuning Guide](docs/performance-guide.md)
-- [Nemotron Omni Modeling Implementation](src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py)
-- [MCore Context Parallel Helpers](3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py)
-- [Megatron-Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
+- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/)
+- [SKILL.md](SKILL.md)
+- [card.yaml](card.yaml)
## Skill Output:
-**Output Type(s):** [Configuration instructions, Analysis, Shell commands]
-**Output Format:** [Markdown with inline code blocks]
+**Output Type(s):** [Configuration instructions, Analysis]
+**Output Format:** [Markdown with inline Python code blocks]
**Output Parameters:** [1D]
**Other Properties Related to Output:** [None]
@@ -44,35 +43,35 @@ Mitigation: Review and scan skill before deployment.
## Evaluation Tasks:
-Evaluated against 3 evaluation tasks (2 positive, 1 negative) from a curated skill-evaluator dataset.
+Evaluated against 3 evaluation tasks (2 positive, 1 negative) in isolated k8s-sandbox pods with 1 attempt per task.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Whether the skill is safe to use, checking for unsafe operations, secret leakage, and unauthorized access.
-- Correctness: Whether the skill produces correct answers against the reference answer.
+- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access).
+- Correctness: Whether the skill produces correct answers against the reference.
- Discoverability: Whether the right skill is loaded and activated when needed.
-- Effectiveness: Whether the skill helps the agent complete the user's goal and expected workflow (equal-weight mean of goal completion and behavior adherence).
-- Efficiency: Whether the skill avoids wasted tool or skill usage, measuring routing quality and productive tool use.
+- Effectiveness: Whether the skill helps complete the user's goal and expected workflow (goal completion + behavior adherence).
+- Efficiency: Whether the skill avoids wasted tool or skill usage (routing quality and productive tool use).
Underlying evaluation signals used in this run:
-- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
-- `accuracy`: Final-answer correctness against the reference answer.
+- `security`: Unsafe operations, secret leakage, and unauthorized access.
- `skill_execution`: Whether the expected skill was found and executed.
+- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy`: Final-answer correctness against the reference answer.
- `goal_accuracy`: Whether the user's goal was achieved.
- `behavior_check`: Whether the expected workflow behavior was followed.
-- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
## Evaluation Results:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 71% → 98% (+27 points) | 56% → 94% (+39 points) | +| Overall | 67% → 100% (+32 points) | 59% → 95% (+36 points) | | Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) | -| Correctness | 27% → 93% (+67 points) | 60% → 93% (+33 points) | -| Discoverability | 100% → 100% (±0 points) | 58% → 96% (+38 points) | -| Effectiveness | 30% → 98% (+68 points) | 40% → 81% (+41 points) | -| Efficiency | 98% → 100% (+2 points) | 53% → 100% (+47 points) | +| Correctness | 47% → 100% (+53 points) | 60% → 93% (+33 points) | +| Discoverability | 83% → 100% (+17 points) | 58% → 96% (+38 points) | +| Effectiveness | 34% → 99% (+65 points) | 50% → 88% (+38 points) | +| Efficiency | 73% → 100% (+27 points) | 62% → 100% (+38 points) | ## Testing Completed:
**[x] Agent Red-Teaming**
diff --git a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig index 358787c9f0..e1b5805ea0 100644 --- a/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig +++ b/skills/nemo-mbridge-perf-vision-dp-over-cp/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWNvbnRleHQtcGFyYWxsZWwiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiODU2MmU5OThjZDVhNzgyMWRhN2MxNzFjNTdmYjQwZTdhZDRkODZhZDY0NWUyNzBhMzk3ZTdkZDJhZWFmY2QyNCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdGh1YiIKICAgICAgXSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiLAogICAgICAgICJkaWdlc3QiOiAiMzNmOWJjYTJmOTU1YmFjYWFiZGNjNzk3OGMyYTRmZmFlNGYyMzI0ODg1ZDY3ZTBlZjQxYTUxZWNjYzU2MjJlMCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIwNzg0MzJjMmFkYTllZmM2NGFhOTUxMWM2NDA1YTYyNzRlNjM5OWIzNjNhNTExN2IwYjFlYjFlY2MwMTkwNjE1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImNhcmQueWFtbCIsCiAgICAgICAgImRpZ2VzdCI6ICJmYjM0YmViZmMyMjg5MDBmNTAxMmZkMDQ3M2RmZmQzY2NlZmUzYTkzYTc3NWVmOTliZDNiMGI2YjNiODM3MDA1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJkaWdlc3QiOiAiNjgzMWNlZjlmYTE4Y2FlYTQ3MmVkYjVhNzc2MTcyYWMyYzY1MmUzNjY2MzUwZjhkN2ViNGVhOGViMDNmNGZjZiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjM1ZjMzOGE5NzEwZWVlYThlOThkMjgzNDM0OWMxZTUyYWI0OTEwYjg4MDY3ZmM2NTNmYjg3YmVlNDVkZDlhODUiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDoHtASj9vKQuomMMPDzUv9rNjNUb910wPq75IUQFrLojPw3+YYxSfbUpOuVEV/NR4CMCe4ys0s1z1yb3Jz8/hChZ0o/HEIpfaIgobphMleU9DdssdOPplC9eB30tniNrwXCw==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLXBlcmYtdmlzaW9uLWRwLW92ZXItY3AiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiMTU2ZmQwMjI0MzgwNDU1MmI2NTAxMDU2MDU4Zjk1YmY2NDFlOWM2YzUzNDY1NDdhYmU2NDZhNjc2ZTdjYjg2ZSIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzA2NTg3ZmIxNzM3ZWU2ODA5NDJlNWQ1M2QwMDk0MzI4NzJlZTQ1ZmNkZDcwZDk3NjU2YTU3MmIzMmQ1M2QxZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJjNGFjZTEwMWVkZmM2OGJhNzc0Y2I3MWRlNDBiZTdlOTk2NDZhMjFiNzA4MDEwYjFhNjAyYjI4NmRjMTRmMzg2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3ZTIxMDQyZGRiNjM3ZmNmN2JiNjEwOTcxYWY2ZjEyZTI5OWRjMmU1YmNlY2M1ZjM2OTM5MzRiMGVjZDUwNTkyIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiOWEyNWJlYTM4M2Q4NjU0ZjMyY2E3Y2EzZjI1ODZmOWYyZTJlODdiNjQ4N2U1NzJmNDcyOTFlMzk3ZTMzNjkzYiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImM4OGNiZjEzYmJiYmVmNzE1MDU0OTExNmU2MmY1NjU5ZTU3OTY1ZjNlNTNmNDgwMTg1MWI3YjU5NWJmZjNhYzgiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDwd/MAPl2kWNx2d2eJ3XaNNCQl8x401GN0EFcYDECzBr5KQj3668JZgX5KHvtyCsECMCeibyjnUd+tRvezzdXiUZJEeuE07vbfe4PdF6RygUCNd4MBop2pR/s7Ny5fznkiKA==","keyid":""}]}} \ No newline at end of file