Skip to content

feat(model): add Nemotron 3.5 Super VL support - #6100

Merged
cuichenx merged 65 commits into
mainfrom
chcui/sage/nmvl
Sep 21, 2026
Merged

cuichenx merged 65 commits into
mainfrom
chcui/sage/nmvl

Conversation

@cuichenx

@cuichenx cuichenx commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds Megatron Bridge support for Nemotron 3.5 Super VL, combining the Nemotron 3 Super language/MTP stack with the shared Nemotron Omni image and temporal-video stack, without an audio encoder.

The Bridge translates one serialized HF MTP block into two repeated Megatron prediction depths and preserves the HF block count on export. This PR is ready for code review; current-head model verification is still being refreshed and is not a merge-readiness claim.

Changelog

  • Register NemotronH_Omni_Reasoning_V3 and normalize its hybrid/MTP configuration through the shared Nemotron-H bridge.
  • Reuse Omni's RADIO tower, dynamic-resolution media support, and separate temporal video embedder without audio or EVS token pruning.
  • Support nested MTP checkpoint weights, the vision projector final norm, and the deterministic RADIO summary-index buffer, including strict re-export of an exported HF checkpoint.
  • Add H100/GB200 pretrain, SFT, PEFT, and GB200 long-context SFT recipes under recipes/, with independent language, vision, and projector freeze controls.
  • Add focused model, conversion, recipe, and inference tests and a model verification card with generated model documentation.
  • Fix shared visual inference helpers to retain post-resize image/video metadata and use the same native processor inputs for HF comparisons. These helpers serve both Omni and Super VL.
  • Extend the public conversion, inference, and training launchers with bounded Slurm polling, shared allocation-level options, and an opt-in direct-Enroot runtime. Pyxis remains the default. These reusable changes make the documented verification workflows usable without private launcher forks and have focused backward-compatibility tests.
  • Merge current main, including its pinned-HF-revision loader fix, and regenerate the model indexes to retain both branches' entries.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) — No new required dependencies.
    • Reviewer: Does the PR have correct import guards for all optional libraries? — Existing optional boundaries retained.

Additional Information

  • The final merged tree passed 348 focused launcher, visual-input, inference-helper, comparison, and generated-document tests; the model-card validator, generated-catalog consistency check, and all-files pre-commit also passed.
  • Fresh independent review covered model/conversion/training, recipes/documentation, and shared launchers/helpers. Both findings were fixed and independently reviewed: strict re-export now preserves the RADIO summary buffer, and direct-Enroot cache defaults now honor forwarded HF_HOME and explicit cache overrides. Four export regression cases passed an isolated CPU check using the real strict safetensors writer; full conversion-test collection still requires the model runtime dependencies.
  • A separate recipe-registry test fails identically on main and this branch: test_known_cross_package_collisions_have_explicit_benchmark_precedence reports the existing nemotron_3_super_pretrain_64gpu_gb200_bf16_config name collision. This unrelated base failure is not patched in the model-addition PR.
  • The verification card deliberately retains historical evidence at its recorded revisions. Current-head numerical parity, inference, and affected training/export/resume verification remain pending; no old result is relabeled as a result of this merge. Full GPU model tests were not rerun by the local CPU checks above.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Use the HF/vLLM source-timeline prompt unconditionally and remove the legacy prompt-mode switch. Add the copyright header required by CI for the new video utility test.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Use the processor-owned video grid and actual per-tubelet vision token counts in the shipped inference examples. Reuse the training prompt labels so inference preserves source timestamps without adding a legacy video prefix.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx cuichenx removed needs-more-tests Requires additional L0 and L1 test coverage before merge full-test-suite labels Sep 18, 2026
@cuichenx cuichenx changed the title feat(model): add Nemotron 3.5 Super VL support Draft: feat(model): add Nemotron 3.5 Super VL support Sep 18, 2026
@cuichenx cuichenx changed the title Draft: feat(model): add Nemotron 3.5 Super VL support [draft] feat(model): add Nemotron 3.5 Super VL support Sep 18, 2026
@cuichenx
cuichenx marked this pull request as draft September 18, 2026 19:46
cuichenx and others added 9 commits September 18, 2026 13:38
…allbacks"

This reverts commit 5fb6151. Use the shared runtime-config callback binding from #6067 instead.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Apply the callback fix and regression tests from #6067 (commits e3e2f13, 65bece8, and 33c47e4).

Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Retain both Bagel and Nemotron 3.5 Super VL forward-step registrations.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Add an opt-in shared backend for training, conversion and inference when site-managed Pyxis runtime placement cannot meet storage requirements. Preserve Pyxis defaults and worker semantics, validate Lustre storage and effective Enroot settings, and cover generated wrappers with focused tests.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Merge upstream/main, regenerate both model indexes, preserve the derived RADIO summary buffer on strict re-export, and honor forwarded HF cache roots in direct Enroot.

Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx cuichenx changed the title [draft] feat(model): add Nemotron 3.5 Super VL support feat(model): add Nemotron 3.5 Super VL support Sep 21, 2026
@cuichenx
cuichenx marked this pull request as ready for review September 21, 2026 17:56
@cuichenx

Copy link
Copy Markdown
Contributor Author

/ok to test b049c95

Signed-off-by: Chen Cui <chcui@nvidia.com>
@claude

claude Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Light review — LGTM

Reviewed the diff against the mandatory workflow (read diff, loaded adding-model-support + review-pr skills, then reviewed). Scope: critical bugs/logic, typos, test coverage, doc accuracy.

No blocking issues found. Notes on the one item I chased down:

  • Nemotron35SuperVLBridge.megatron_to_hf_config replaces hf_config[llm_config] wholesale with {num_nextn_predict_layers: 1}. This is safe: export runs the dict through conform_config_to_reference, which recurses into the nested llm_config, keeps the exported num_nextn_predict_layers, and backfills every other llm_config key from the reference HF config. Covered by test_nemotron_35_super_vl_export_preserves_shared_mtp_serialization.
  • MTP shared-block invariants (one serialized HF block maps to two repeated Megatron depths, pattern star-E), RADIO summary-buffer shard write (atomic tmp+replace, index/total_size update, idempotent re-export), list-based layers_block_type schema normalization, and vision recompute validation are all guarded with explicit errors and have direct unit coverage.
  • No typos, doc drift, or missing-coverage gaps spotted in the changed code. is_nemotron_omni correctly matches NemotronH_Omni_Reasoning_V3.

Suggested test cases

No perf tests impacted (no scripts/performance/configs/ targets touched by this PR).

Unit cases exercised/derived from the diff (all present in the PR, listed for the record):

  • test_nemotron_35_super_vl_architecture_is_registered
  • test_nemotron_35_super_vl_provider_reuses_omni_with_list_based_mtp
  • test_nemotron_35_super_vl_rejects_unexpected_serialized_mtp_depth
  • test_nemotron_35_super_vl_export_preserves_shared_mtp_serialization
  • test_nemotron_35_super_vl_export_closes_summary_idxs_buffer
  • test_nemotron_35_super_vl_export_requires_weight_index
  • test_nemotron_35_super_vl_strict_reexport_preserves_summary_buffer
  • test_nemotron_35_super_vl_export_requires_direct_modeling_entrypoint
  • test_nemotron_35_super_vl_mapping_uses_nested_mtp_and_vision_final_norm
  • test_list_based_transformers_layer_schema_is_normalized
  • test_list_based_transformers_layer_schema_rejects_unknown_types
  • test_canonical_model_exposes_nested_language_decoder_for_cuda_graph_helper
  • test_canonical_model_rejects_variable_sequences_with_cuda_graphs
  • test_vision_recompute_requires_effective_granularity
  • test_full_vision_recompute_requires_complete_policy
  • test_full_vision_recompute_rejects_invalid_layer_count
  • test_vision_recompute_rejects_forced_radio_eval_mode
  • test_super_vl_rectangular_image_and_video_use_exact_visual_token_counts (GPU)

@yaoyu-33 yaoyu-33 added full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge labels Sep 21, 2026
@cuichenx cuichenx added ready-to-merge PR is approved, current, and only waiting for CI to pass before merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Sep 21, 2026
@cuichenx
cuichenx merged commit 9ac0caa into main Sep 21, 2026
85 of 88 checks passed
@cuichenx
cuichenx deleted the chcui/sage/nmvl branch September 21, 2026 22:49

This branch was successfully deployed

1 active deployment
test — 46269e7b Deployed Sep 21, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22064
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants