Repository navigation
docs(model): document Nemotron Omni vision_dp_over_cp - #5646
Conversation
Add a Vision Context Parallelism section to the Nemotron-3 Omni example README covering the model.vision_context_parallel flag, its CP>1 gating, and the measured step-time/memory A/B across three datasets. Also fixes a typo in the vision input mode table (temporal_patchrecipe_dim). Signed-off-by: Huy Vu <huvu@nvidia.com>
huvunvidia
added
docs
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| ### Vision Context Parallelism | ||
|
|
||
| `model.vision_context_parallel=True` shards the microbatch's images (or |
There was a problem hiding this comment.
This documents model.vision_context_parallel=True, but that field does not exist on the Nemotron Omni provider in this tree. The provider hardcodes vision_cfg.context_parallel_size = 1 (nemotron_omni_provider.py:105) and defines no vision_context_parallel attribute, and the modeling code contains no image-count sharding + gather (the only CP logic in modeling_nemotron_omni.py shards the text sequence, not images).
Setting an undefined key on a config dataclass silently creates a phantom field, so users following this section would get a no-op with no error. Please either land the implementing change first, or gate this doc behind it and confirm the exact flag name.
=> Huy: False alarm, because this PR will go in after #5645 |
Capture the vision_context_parallel findings as a proper perf skill: SKILL.md with enablement, code anchors, implementation map, the measured 3-dataset A/B, and the load-balance pitfalls; card.yaml with validation status, constraints, and follow-up validation; an evals dataset with two positive tasks and one negative routing task against hierarchical CP, which shares the "context parallel" vocabulary but not the mechanism. BENCHMARK.md and skill-card.md are committed with empty result tables: those numbers come from an NVSkills-Eval run that has not happened yet and must not be hand-authored. skill.oms.sig is likewise omitted since it is a Sigstore bundle from the NVIDIA signing pipeline. Signed-off-by: Huy Vu <huvu@nvidia.com>
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Bridge already exposes dist_train.vision_context_parallel_size, an unrelated Qwen3-VL knob that gives the vision tower its own ranks and sets the encoder's own context_parallel_size. Add a comparison table so the two are not mistaken for alternatives, and record that the encoder config intentionally keeps context_parallel_size=1 under this feature because the sharding happens outside the encoder, over its inputs. Drops the README link into the skill tree per review feedback. Signed-off-by: Huy Vu <huvu@nvidia.com>
|
/nvskills-ci |
|
/ok to test 31d48a7 |
|
/nvskills-ci |
PR #5645 turned image sharding on by default and moved the sharded projection onto the FP8-alignment helper, which invalidated the enablement guidance, the code anchors, and the A/B baseline arm documented here. Also records that a two-rank unit test now covers the placeholder path CP=8 still lacks. Signed-off-by: Huy Vu <huvu@nvidia.com>
…m/NVIDIA-NeMo/Megatron-Bridge into huvu/vision_module_DP_docs_skills
|
/nvskills-ci |
|
/ok to test 67e0579 |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Follows the flag rename in the implementation PR. The skill directory becomes nemo-mbridge-perf-vision-dp-over-cp; skill.oms.sig is now stale (subject name and file digests both changed) and needs an NVSkills pipeline re-run. Signed-off-by: Huy Vu <huvu@nvidia.com>
|
/nvskills-ci |
|
/ok to test cf63462 |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
huvunvidia
changed the title
|
/ok to test 155154e |
cuichenx
added
ready-to-merge
4 participants
Summary
Documents the
model.vision_dp_over_cpflag added by #5645, in two places.1. Example README (
examples/models/nemotron/nemotron_3_omni/README.md)## Training: what the flag does(images/tubelets sharded over the LM's CP group — data parallelism over
images, not ring attention), the fact that it defaults to
Trueand is asilent no-op at CP=1, and the measured A/B results.
the last rank, so multi-image data with varying image counts balances unevenly.
model.temporal_patchrecipe_dim->model.temporal_patch_dim.2. New perf skill (
skills/nemo-mbridge-perf-vision-dp-over-cp/)SKILL.md— enablement, a disambiguation table against the pre-existing andunrelated
dist_train.vision_context_parallel_size, code anchors,implementation map (split, placeholder padding, encode, even-grid padding,
project-then-gather, merge), measured impact, six pitfalls, and an A/B
verification procedure.
card.yaml— validation status, feature meaning, constraints, knownlimitations, evidence files, follow-up validation.
evals/evals.json— two positive activation tasks plus one negative routingtask, because this skill shares the "context parallel" vocabulary with
nemo-mbridge-perf-hierarchical-context-parallelwhile describing an unrelatedmechanism.
BENCHMARK.md,skill-card.md,skill.oms.sig— generated and signed by theNVSkills pipeline. Tier 3 verdict: PASS, overall 70% -> 98% for Claude Code
and 79% -> 95% for Codex; the negative routing task was answered correctly.
docs/skills-index.mdunder Performance Techniques.Rename
The flag was called
vision_context_parallelin earlier revisions. It wasrenamed to
vision_dp_over_cpat review request, because the old name read assequence-dimension context parallelism (which this is not) and collided visually
with the unrelated
dist_train.vision_context_parallel_size. This PR followsthat rename: the skill directory moved from
skills/nemo-mbridge-perf-vision-context-parallel/toskills/nemo-mbridge-perf-vision-dp-over-cp/, and the flag name was updated inthe README,
SKILL.md,card.yaml,evals/evals.json,BENCHMARK.md, anddocs/skills-index.md.The disambiguation section in
SKILL.mddeliberately still namesdist_train.vision_context_parallel_size— that knob is unchanged, and tellingthe two apart is the reason the section exists.
skill.oms.signeeds re-signing before mergeThe rename invalidated the attached signature on both counts: the signed subject
name is still
nemo-mbridge-perf-vision-context-parallel, and the pinned SHA-256digests for
SKILL.md,card.yaml,evals/evals.json, andBENCHMARK.mdnolonger match their contents. The file was moved with the rest of the directory
but deliberately not hand-edited. The NVSkills pipeline needs to re-run and
re-sign before this merges.
Merge order
Please merge #5645 first — it introduces the flag these docs describe.
Measurements quoted in the docs
Nemotron-3 Nano Omni 30B-A3B, 2 nodes / 16 GPUs, PP=1 DP=2 EP=8 ETP=1 MBS=2
GBS=64 GA=16, flag flipped only:
Both a memory win (-5% to -11% peak) and a throughput win (-4% to -5% step time).
Test plan
evals.jsonparses as JSON;card.yamlparses as YAML.skills-index.mdtoctree entry.skill.oms.sigafter the rename (see above).