docs(model): document Nemotron Omni vision_dp_over_cp

docs/skills-index.md

@@ -50,6 +50,7 @@ skills/nemo-mbridge-perf-tp-dp-comm-overlap/SKILL
 skills/nemo-mbridge-perf-megatron-fsdp/SKILL
 skills/nemo-mbridge-perf-sequence-packing/SKILL
 skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL
+skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL
 skills/nemo-mbridge-perf-expert-parallel-overlap/SKILL
 skills/nemo-mbridge-perf-moe-comm-overlap/SKILL
 skills/nemo-mbridge-perf-activation-recompute/SKILL

examples/models/nemotron/nemotron_3_omni/README.md

@@ -257,12 +257,41 @@ Set the four flags below as a matched column — a mismatched set produces incor
 | Field                                              | Dynamic resolution (images, variable H×W) | Temporal video (videos, fused pairs, 512²) |
 |----------------------------------------------------|-------------------------------------------|--------------------------------------------|
 | `dataset.task_encoder.use_temporal_video_embedder` | `False`                                   | `True`                                     |
-| `model.temporal_patchrecipe_dim`                         | `1`                                       | `2`                                        |
+| `model.temporal_patch_dim`                         | `1`                                       | `2`                                        |
 | `model.separate_video_embedder`                    | `False`                                   | `True`                                     |
 | `model.temporal_ckpt_compat`                       | `False`                                   | `True`                                     |
 
 Note:`dataset.task_encoder.use_temporal_video_embedder` only applies to the Energon data path.
 
+### Vision DP Over CP
+
+`model.vision_dp_over_cp` shards the microbatch's images (or
+temporal tubelets) across the language model's context-parallel group, so each
+CP rank encodes `1/CP` of them instead of all of them. This is data parallelism
+over images, not ring attention — RADIO already attends within each image via
+`cu_seqlens`, so nothing is split along the sequence dimension and no
+cross-rank attention communication is introduced. The vision projector runs on
+the rank-local shard, and a single gather afterwards restores the full feature
+set that every rank needs to merge media into the packed text sequence.
+
+The flag defaults to `True` and is a no-op unless
+`model.context_parallel_size > 1`. Both recipes below ship with CP=1, so it only
+takes effect once you raise CP. Set `model.vision_dp_over_cp=False` to
+encode every image redundantly on every CP rank.
+
+Measured on 2 nodes / 16 GPUs (PP=1, DP=2, EP=8, ETP=1, MBS=2, GBS=64, GA=16),
+flipping only this flag:
+
+| Dataset       | TP | CP | Step time      | Peak memory   |
+|---------------|----|----|----------------|---------------|
+| InfoVQA       | 4  | 2  | 13.2 → 12.5 s  | 81 → 73 GB    |
+| Mantis        | 2  | 4  | 12.2 → 11.6 s  | 81 → 77 GB    |
+| llava_video   | 4  | 2  | 13.6 → 13.0 s  | 84 → 75 GB    |
+
+Images are split by count, with the remainder assigned to the last rank.
+Multi-image data whose per-sample image count varies (Mantis at CP=4) therefore
+balances less evenly than uniform data, which is why its memory saving is the
+smallest of the three.
 
 ### Image-Text — CORD-V2
 

skills/nemo-mbridge-perf-vision-dp-over-cp/BENCHMARK.md

@@ -0,0 +1,112 @@
+# Skill Benchmark: nemo-mbridge-perf-vision-dp-over-cp
+
+> ✅ **Overall verdict: PASS — Recommended for publication**
+
+## Publication Recommendation
+
+Recommended for publication based on the completed evaluation evidence in this report.
+
+## Evaluation Metadata
+
+- Skill: `nemo-mbridge-perf-vision-dp-over-cp`
+- Evaluation date: 2026-08-27
+- Evaluator version: `1.3.2`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
+- Tasks: 3 evaluation tasks (2 positive, 1 negative)
+- Dataset digest: `sha256:1f686f2a6deb8b2b442751f0639aafb30c6e89513befad8532f4ef5754f59593` (skill-evaluator-dataset-snapshot/1)
+- Attempts per task: 1
+- Environment: `k8s-sandbox`
+- Tier 3 evidence: required for publication
+
+Each task attempt ran in its own isolated sandbox pod.
+
+## Execution and Provenance
+
+- Validation status: `passed`
+- Report generation: `complete`
+- Evaluator version: `1.3.2`
+- Git commit: `eda36f54e38357fe24c321a2ffae85589de9bd48`
+- Content type: requested `auto`, detected `skill`
+- Container image: `gitlab-master.nvidia.com:5005/nvcarps/ci-group/nvcarps-ci/skillevaluator-ci:sha-eda36f54e38357fe24c321a2ffae85589de9bd48`
+- Container image digest: `not recorded`
+- Tier 3: requested `true`, executed `true`, status `succeeded`
+
+## What This Report Answers
+
+The three-tier evaluation checks whether the skill:
+
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
+
+## Results at a Glance
+
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | 67% → 100% (+32 points) | 59% → 95% (+36 points) |
+| Security | 100% → 100% (±0 points) | 67% → 100% (+33 points) |
+| Correctness | 47% → 100% (+53 points) | 60% → 93% (+33 points) |
+| Discoverability | 83% → 100% (+17 points) | 58% → 96% (+38 points) |
+| Effectiveness | 34% → 99% (+65 points) | 50% → 88% (+38 points) |
+| Efficiency | 73% → 100% (+27 points) | 62% → 100% (+38 points) |
+
+**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
+
+Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
+
+## Tier Status
+
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) |
+| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |
+| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) |
+
+## Findings and Observations
+
+<details>
+<summary>Show detailed findings and successful checks</summary>
+
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`)
+- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`)
+- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml`)
+
+</details>
+
+## Scoring Methodology
+
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
+
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
+
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
+
+Signals present in this run:
+
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was found and executed.
+- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
+
+</details>
+
+## Freshness
+
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md

@@ -0,0 +1,217 @@
+---
+name: nemo-mbridge-perf-vision-dp-over-cp
+description: Operational guide for sharding a VLM vision encoder across the language model's context-parallel ranks in Megatron-Bridge, including config knobs, code anchors, load-balance pitfalls, and measured impact.
+license: Apache-2.0
+when_to_use: The vision tower replicates work and activations on every CP rank while the language model is already sharded, or a VLM run at CP>1 OOMs in the encoder; 'vision_dp_over_cp', 'vision CP', 'vision DP over CP', 'image sharding across CP ranks', 'encoder replicated across CP'.
+---
+
+# Vision DP Over CP Skill
+
+This skill covers `vision_dp_over_cp`: reusing the language model's
+context-parallel process group to shard the **images** (or temporal tubelets) of
+a microbatch across CP ranks, so each rank encodes `1/CP` of them instead of all
+of them.
+
+It is **not** ring attention and shares nothing with
+`nemo-mbridge-perf-hierarchical-context-parallel` beyond the process group.
+Vision encoders such as RADIO already attend *within* each image via
+`cu_seqlens`, so there is no long sequence to split and no cross-rank attention
+communication. The correct mental model is **data parallelism over images,
+borrowing the CP group** — a rank encodes whole images, never partial ones.
+
+Shipped for Nemotron Omni. The MCore split/gather helpers are model-agnostic, so
+the pattern generalizes to any Bridge VLM with a dynamic-resolution encoder.
+
+## Do Not Confuse With `dist_train.vision_context_parallel_size`
+
+Bridge already has a similarly named and completely different knob:
+`vision_context_parallel_size` on the dist-train path
+(`src/megatron/bridge/models/qwen_vl/qwen3_vl_provider.py`, consumed in
+`src/megatron/bridge/training/initialize.py`).
+
+| | `model.vision_dp_over_cp` (bool) | `dist_train.vision_context_parallel_size` (int) |
+|---|---|---|
+| Ranks | Reuses the language model's CP ranks | Gives the vision tower its **own** ranks |
+| Splits | Whole images across those ranks | The vision encoder's own sequence dimension |
+| Encoder `context_parallel_size` | Stays `1` | Set to this value |
+| Models | Nemotron Omni | Qwen3-VL dist-train |
+
+They are not alternatives and do not compose in any tested configuration.
+
+## Enablement
+
+```python
+cfg.model.context_parallel_size = 2      # the flag is a no-op at CP=1
+cfg.model.vision_dp_over_cp = False      # opt out; on by default
+```
+
+Defaults to `True`. It self-disables when `context_parallel_size == 1`, so a
+recipe that does not raise CP behaves exactly as it did before the flag existed
+and reports no error. Opting out is only useful for A/B measurement or to
+isolate a suspected split/gather bug.
+
+## Code Anchors
+
+Provider field, `src/megatron/bridge/models/nemotron_omni/nemotron_omni_provider.py`:
+
+```python
+vision_dp_over_cp: bool = True
+```
+
+Gating, in `NemotronOmniModel.__init__`
+(`src/megatron/bridge/models/nemotron_omni/modeling_nemotron_omni.py`):
+
+```python
+self.context_parallel_lm = language_transformer_config.context_parallel_size
+self.vision_dp_over_cp = vision_dp_over_cp and self.context_parallel_lm > 1
+```
+
+The split itself wraps an upstream MCore helper:
+
+```python
+return split_to_context_parallel_ranks_dynamic_res(
+    images,
+    imgs_sizes,
+    vision_packed_seq_params,
+    patch_dim=self.patch_dim,
+    fp8_enabled=False,
+    fp8_recipe=getattr(self.config, "fp8_recipe", None),
+    num_frames=num_frames,
+    temporal_patch_size=temporal_patch_size,
+)
+```
+
+Project-then-gather, at the end of `_encode_images`:
+
+```python
+if shard_vision:
+    # Project before the gather so the projector stays sharded, then
+    # restore the global feature set every CP rank's media merge needs.
+    projected = _project_multimodal_embeddings(self.vision_projection, encoded)
+    gathered = gather_from_context_parallel_ranks_dynamic_res(projected, num_padded_ranks)
+    return gathered.contiguous()
+```
+
+Upstream helpers (do not modify — changes go through the MCore repo):
+
+- `3rdparty/Megatron-LM/megatron/core/models/multimodal/context_parallel.py`
+  — `split_to_context_parallel_ranks_dynamic_res`,
+  `gather_from_context_parallel_ranks_dynamic_res`
+- `3rdparty/Megatron-LM/megatron/core/models/multimodal/llava_model.py`
+  — performs the identical split/encode/gather on its temporal path
+
+## Implementation Map
+
+1. **Split.** The MCore helper partitions the patched image tensor on image (or
+   tubelet) boundaries and rewrites `imgs_sizes` and the vision
+   `PackedSeqParams` (`cu_seqlens`, `max_seqlen`) to describe only the rank-local
+   share. Sharding is by **image count**, not patch count; the remainder goes to
+   the **last** rank.
+2. **Placeholder padding.** When a microbatch has fewer images than CP ranks,
+   the helper injects `num_padded_imgs = max(0, cp_size - num_imgs)` dummy `1x1`
+   images so no rank runs an empty encoder, and returns `num_padded_ranks` so the
+   gather can drop them again.
+3. **Encode.** Each rank runs the full vision tower on its own images only.
+   Activations, not just compute, are what drop by `1/CP`. The vision encoder's
+   own `context_parallel_size` stays hardcoded to `1` in
+   `_build_vision_config` — that is correct and unchanged by this feature, since
+   the sharding happens *outside* the encoder, over its inputs.
+4. **Even-grid padding.** `_pad_patch_grid_to_even` zero-pads an odd patch grid
+   before pixel shuffle. Only the injected `1x1` placeholders hit this path;
+   real frames are already even in both extents.
+5. **Project, then gather.** The vision projector runs on the rank-local shard,
+   so it stays sharded too, and only the smaller post-projector features cross
+   the wire. The gather is the **only** collective this feature adds.
+6. **Merge.** After the gather, every rank holds the full global feature set,
+   which is exactly what the unchanged media-merge path expects.
+
+## Measured Impact
+
+Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2
+GBS=64 GA=16, with only `vision_dp_over_cp` flipped:
+
+| Dataset     | TP | CP | Step time     | Peak memory |
+|-------------|----|----|---------------|-------------|
+| InfoVQA     | 4  | 2  | 13.2 -> 12.5 s | 81 -> 73 GB |
+| Mantis      | 2  | 4  | 12.2 -> 11.6 s | 81 -> 77 GB |
+| llava_video | 4  | 2  | 13.6 -> 13.0 s | 84 -> 75 GB |
+
+It is both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step
+time). Do not describe it as memory-only; a single early video A/B suggested step
+time was flat, and the three-dataset sweep contradicted that.
+
+The Mantis row is the weakest memory result, and that is the load-balance
+signature described under Pitfalls, not noise.
+
+## Pitfalls
+
+1. **Silent no-op at CP=1.** The flag ANDs with `context_parallel_lm > 1`. Both
+   checked-in Nemotron Omni recipes ship CP=1, so the on-by-default value changes
+   nothing there. Always confirm CP>1 before attributing a result to this flag,
+   and note that raising CP now silently enables image sharding too.
+2. **Count-based split leaves the tail rank hot.** Images are divided by count
+   with the remainder on the last rank. Multi-image data with a wide per-sample
+   image-count spread (e.g. Mantis, 1-5 images/sample at CP=4) balances poorly:
+   peak memory is set by the busiest rank, so the win shrinks. A
+   patch-count-balanced greedy split on image boundaries is the fix, and it
+   belongs **upstream in MCore**, not reimplemented in Bridge. Since the flag is
+   on by default, this imbalance is what a multi-image CP>1 run gets unless the
+   user opts out.
+3. **Process-group divergence is checked, not assumed.** The MCore splitter
+   resolves the CP group from global parallel state, so a `ProcessGroupCollection`
+   whose `cp` size disagrees with `context_parallel_size` would shard against the
+   wrong ranks. Bridge raises instead:
+   `"Nemotron Omni vision_dp_over_cp does not match its process group"`.
+4. **Odd patch grids break pixel shuffle.** Omni's factor-2 pixel shuffle rejects
+   an odd grid, and MCore's `1x1` placeholder images are odd by construction.
+   Without `_pad_patch_grid_to_even`, the CP>images case fails only for the
+   specific microbatches that trigger placeholder injection — an intermittent
+   failure that looks data-dependent.
+5. **Do not gather before projecting.** Gathering encoder output first would
+   replicate the projector on every rank and move a larger tensor. The ordering
+   is deliberate.
+6. **The placeholder path is unvalidated end-to-end.** A two-rank unit test
+   proves the sharded encoder matches the unsharded one when a microbatch holds
+   fewer images than ranks, but no multi-node training run has exercised it, and
+   nothing covers CP=8.
+
+## Verification
+
+Unit tests. The second file self-launches a two-rank `torch.distributed` job and
+asserts the sharded tower matches the unsharded one for fewer images than ranks,
+an exact split, and a remainder; it skips when fewer than two GPUs are visible:
+
+```bash
+uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_nemotron_omni_model.py -q
+uv run python -m pytest tests/unit_tests/models/nemotron_omni/test_vision_dp_over_cp_distributed.py -q
+```
+
+Multi-GPU A/B. Run the same recipe twice at CP>1, flipping only the flag, and
+compare reported step time and peak allocated memory:
+
+```bash
+uv run python -m torch.distributed.run --nproc_per_node=8 \
+  scripts/training/run_recipe.py \
+  --recipe nemotron_omni_cord_v2_sft_config \
+  model.context_parallel_size=2 \
+  model.vision_dp_over_cp=false \
+  train.train_iters=20
+```
+
+Success criteria:
+
+- Peak memory drops relative to the `false` arm; step time does not regress.
+- Loss curves of the two arms track each other. The gather makes this
+  mathematically equivalent, so a divergence means the split or gather is wrong,
+  not that the feature is lossy.
+- If nothing changes at all, check that `context_parallel_size > 1` — the flag
+  silently self-disables at CP=1.
+
+## Related
+
+- @skills/nemo-mbridge-perf-moe-vlm-training/SKILL.md — VLM training strategy,
+  FSDP vs 3D parallel, and why mock image data invalidates VLM benchmarks
+- @skills/nemo-mbridge-perf-memory-tuning/SKILL.md
+- @skills/nemo-mbridge-perf-hierarchical-context-parallel/SKILL.md — the
+  *sequence*-splitting use of the same process group; unrelated mechanism
+- @skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml