feat(nemotron): native HF 5.19 Super VL conversion support

[cuichenx]

What does this PR do ?

Draft native Transformers 5.19 support for Nemotron 3.5 Super VL conversion, preserving legacy remote-code loading.

Work in progress—not ready to merge. Conversion passes exact weight checks, but the exported model is not yet a fully functional standalone HF checkpoint with trust_remote_code=False.

Changelog

• Read native text_config and image-token fields while retaining the legacy llm_config path.
• Preserve shared MTP serialization in either HF config schema.
• Do not require remote modeling files for native-schema exports.
• Add a model-local, HF-5.19.0-guarded processor adapter: ignore the legacy derived video field in memory and omit the unused audio extractor for audio-free Super VL.
• Preserve the source checkpoint and installed Transformers code; do not apply global monkey patches.
• Add focused tests and document compatibility boundaries.
• Expand the supported Transformers range to >=5.15,<5.20 and lock Transformers 5.19.0.
• Keep the full MiniMax M3 VL rotary correction only on 5.17.0. Use unpatched upstream geometry from HF #49164 on 5.18/5.19, accepting the native BF16/FP16 buffer-rounding precision difference. Document that policy and test native arithmetic without weakening the geometry checks.

GitHub Actions CI

Local verification used nvcr.io/nvidia/nemo:26.08.01 with an isolated Transformers 5.19.0 environment:

• 38 focused tests passed, including native/remote loader selection, native export reload with either trust flag, older-HF remote loading, and processor/conversion behavior.
• General-upgrade matrix before the final MiniMax policy adjustment: 2,868 focused tests passed per version on 5.17.0, 5.18.0, and 5.19.0 (three documented skips each); 52 MiniMax/CLIP checks passed each on 5.15.1 and 5.16.1; 158 PEFT 0.20 compatibility checks passed on 5.19.0. That matrix used the earlier extended precision workaround; focused MiniMax reruns validate the final unpatched-5.18/5.19 policy separately. These are focused tests, not all-model real-checkpoint parity or full functional CI.
• Native and remote-code HF → Megatron → HF roundtrips: 43,078/43,078 source weights exactly matched, atol=rtol=0, TP1/PP2/EP2/ETP1 on four H100s. Independently saved HF exports have identical tensor keys/shapes/dtypes/bytes. A subsequent persisted Megatron readback comparison passed: 1,104 logical tensors byte-identical, with no metadata or model-config differences.
• Processor artifact save/reload through Bridge preserved tensors and token IDs exactly for three image sizes and a synthetic rectangular two-frame video; chat template and source metadata hashes were preserved.
• A fresh standalone HF-only process, without the Bridge adapter and with trust_remote_code=False, failed end-to-end validation:
  • AutoProcessor still requires an absent audio feature extractor.
  • AutoModelForImageTextToText loaded, but reported four unexpected vision normalization/buffer keys.
  • Text forward with installed Mamba kernels produced non-finite logits. A controlled follow-up disabled only mamba-ssm within the HF interpreter; the PyTorch fallback produced finite logits. With that explicit runtime condition, source/export logits were bitwise identical on three prompts (cosine 1, max difference 0), and all 16-token greedy continuations matched. This is not an automatic Bridge production workaround or a shared-package uninstall.
  • Image inference was blocked by processor loading.
• With visual inputs prepared by the Bridge processor adapter, native HF source image forward/generation completed for square and rectangular images. Native video forward failed on the original source checkpoint: torch.cat receives a tensor-valued video pooler_output, rather than a sequence. Exported visual parity remains unverified. This visual test is not a standalone HF processor workflow.
• Standalone text-only checkpoint import/save/reload/export/inference verification is still running. No existing verification-card claims are changed.
• Final MiniMax policy rerun: 51 MiniMax tests passed on each of 5.15.1, 5.16.1, 5.17.0, 5.18.0, and 5.19.0 (255 total, no skips). Only 5.17.0 is patched; 5.18/5.19 retain native geometry and native rounded-buffer arithmetic.

Before your PR is "Ready for review"

Pre checks:

• Read and followed contributor guidelines.
• Added focused tests.
• Updated documentation.
• Version-gated native imports and retained older-HF remote-code behavior.
• Resolve standalone processor loading without an audio extractor.
• Isolate non-finite text logits to the installed Mamba path; verify source/export text inference with the explicit fallback.
• Resolve native vision-final LayerNorm naming compatibility for non-identity fine-tuned weights; Base identity affine values do not establish that support.
• Fix native hf_pretrained.model accessor selection using AutoModelForImageTextToText; retain CausalLM for text and actual remote registrations, not merely the trust permission.
• Complete native HF inference/parity and CI validation before making ready.

Additional Information

Scope is Nemotron 3.5 Super VL only, using nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 for verification. This is not a Nemotron Omni native-HF migration.

Independent reviews covered the complete diff and follow-up loader changes. Native HF-saved checkpoint key layouts and full standalone visual inference are not claimed supported by these original-checkpoint conversion tests. Remaining upstream/runtime limitations are explicit above.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[cuichenx]

/review

[github-actions]

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-6471-c98f2f57394f.docs.buildwithfern.com/nemo/megatron-bridge

[cuichenx]

/review

[svcnvidia-nemo-ci]

One blocking correctness finding: config-only HF exports bypass the new processor adapter and can silently omit processor artifacts.

Reviewed all 15 changed files and directly affected loading/export paths. This was a static review; tests were not executed. The documented standalone HF inference limitations are not treated as verified conversion support.

No perf tests impacted.

Suggested test cases

• test_super_vl_config_only_export_preserves_native_processor — proposed regression covering both artifact-source options described inline.
• test_native_processor_load_save_reload[False] — retain exact image/video tensor and token-ID equality.
• test_native_processor_load_save_reload[True] — retain native reload when remote-code execution is permitted but not registered.
• test_super_vl_native_export_does_not_require_remote_modeling_file.
• test_super_vl_native_config_preserves_language_and_image_settings.
• test_vision_rope_correction_is_version_gated_and_idempotent.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

[svcnvidia-nemo-ci]

[P2] Apply the native wrapper selection to config-only artifact export

Failure: This resolver is used by from_hf_pretrained, but the config-only branch of save_hf_pretrained still constructs PreTrainedCausalLM directly. Consequently, AutoBridge.from_auto_config(..., native_reference, trust_remote_code=False).save_hf_pretrained(...) does not use Nemotron35VLProcessor. For an audio-free 5.19.0 source requiring this adapter, ordinary AutoProcessor fails on the missing audio extractor, and the base wrapper silently returns None. Artifact saving therefore omits processor_config.json and processor/video settings instead of preserving the reloadable Bridge processor. Saving an image processor separately does not preserve those artifacts. The new roundtrip test exercises the specialized wrapper directly, so it misses this public export entrypoint.

Correction: In the non-text-only config-only artifact branch, use this resolver for causal-family wrappers. Preserve the existing artifact saver for non-causal heads rather than changing their artifact contracts. Suggested replacement for the direct construction at line 1266 (not executed; verify before adopting):

artifact_cls = _resolve_pretrained_wrapper_cls(self.hf_pretrained)
if not issubclass(artifact_cls, PreTrainedCausalLM):
    artifact_cls = PreTrainedCausalLM
artifact_source = artifact_cls.from_pretrained(
    artifact_source_path, **artifact_kwargs
)

The invariant is that config-only native Super-VL exports use the same processor compatibility path as wrapper-backed exports, without materializing model weights.

Preservation and acceptance: Add proposed test_super_vl_config_only_export_preserves_native_processor, reusing the local processor_source fixture plus a native-schema config. Stub weight writing/postprocessing, export through a config-only bridge using both hf_model_id and source_path, and assert that processor_config.json exists, video_temporal_patch_dim remains 2, and Bridge reload reproduces the source image/video tensors and token IDs exactly. Retain the existing config-only reference-artifact test assertions that processor artifacts survive and reference weights are never loaded. Legacy Super-VL with a trusted remote registration must still delegate to the base processor loader and preserve remote modeling files; ordinary causal checkpoints must remain unchanged. Leave the separate text-only branch media-free, and retain the existing base artifact saver for masked/token-classification configurations.