diff --git a/docs/fern/versions/nightly/pages/models/nemotron/index.mdx b/docs/fern/versions/nightly/pages/models/nemotron/index.mdx
index afcb193a9e..9d2924111d 100644
--- a/docs/fern/versions/nightly/pages/models/nemotron/index.mdx
+++ b/docs/fern/versions/nightly/pages/models/nemotron/index.mdx
@@ -10,6 +10,8 @@ Nemotron model documentation is organized by model variant.
| Nemotron-3 Nano 4B | [nemotron3-nano-4b.md](nemotron3-nano-4b.md) |
| Nemotron-3 Super | [nemotron3-super.md](nemotron3-super.md) |
| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) |
+| Nemotron-3.5 Super VL | [nemotron3.5-super-vl.md](nemotron3.5-super-vl.md) |
+| Nemotron-3.5 Super Text-only | [nemotron3.5-super-vl-text-only.md](nemotron3.5-super-vl-text-only.md) |
| Nemotron-3 Ultra | [nemotron3-ultra.md](nemotron3-ultra.md) |
| Nemotron Nano V2 VL | [nemotron-nano-v2-vl.md](nemotron-nano-v2-vl.md) |
| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) |
diff --git a/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl-text-only.mdx b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl-text-only.mdx
new file mode 100644
index 0000000000..8bfd546455
--- /dev/null
+++ b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl-text-only.mdx
@@ -0,0 +1,613 @@
+# Nemotron-3.5 Super Text-only
+
+The text-only bridge imports language and shared Multi-Token Prediction (MTP) weights from the Nemotron 3.5 Super VL checkpoint while excluding the vision encoder, projector, and temporal video embedder. Export produces a standalone `NemotronHForCausalLM` checkpoint that uses the native Nemotron-H text bridge.
+
+This workflow reuses [Nemotron 3 Super](nemotron3-super.md) text recipes with the language-only checkpoint as their initialization source. The records below describe the exact verified configurations, not every recipe or parallelism combination. See the [full-VL guide](nemotron3.5-super-vl.md) for image/video workflows.
+
+The public checkpoint revision remains unbound. CPU conversion, long-context training, and text-only pretrain/resume verification are deferred; their unverified status does not imply lack of implementation support.
+
+
+
+## Verified configurations
+
+Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
+
+
+### Run a configuration
+
+Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Precision
+
+
+
+
+
+
+ GPU
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Import · CPU
+ ○ Unverified
+
+
+
Hardware
not specified
+
Precision
BF16
+
Last verified
—
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
CPU conversion is deferred; no CPU-only verification is claimed.
+
Distributed GPU import and paired export passed on H100 and GB200. All 42,683 language tensors, including 1,040 MTP tensors, matched in keys, shapes, dtypes and values across the saved round trip. Compared 247,222,108,160 source tensor bytes; no media tensors were exported. This establishes weight conversion, not forward or training parity.
+
+
+
+
+
+
Export · CPU
+ ○ Unverified
+
+
+
Hardware
not specified
+
Precision
BF16
+
Last verified
—
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
CPU conversion is deferred; no CPU-only verification is claimed.
+
Exported a standalone native NemotronHForCausalLM checkpoint with unprefixed language and MTP weights and zero media tensors. All 42,683 tensors matched the source exactly, including 1,040 MTP tensors. The tokenizer was preserved, native HF config loading and normal text bridge dispatch passed, and the tensor-name/shape structure matched Nemotron 3 Super. This does not claim identical weights to that model.
+
+
+
+
+
+
Pretrain · H100
+ ○ Unverified
+
+
+
Hardware
H100
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+
+
+
+
+
+
Pretrain · GB200
+ ○ Unverified
+
+
+
Hardware
GB200
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+
Completed a fresh-root continuation from the original complete step-250 checkpoint through step 500 after the reference run failed at final save. All 1250 loss comparisons across steps 251-500 passed the unchanged 1% plus 1e-6 bound. Metrics below describe only that continuation, not an uninterrupted 500-step execution. Uses the native Nemotron 3 Super SFT recipe on 64 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super SFT recipe on 64 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
+
+
+
+
+
Long Context · H100
+ ○ Unverified
+
+
+
Hardware
H100
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Long-context text-only training is deferred; no long-context verification is claimed.
+
+
+
+
+
+
Long Context · GB200
+ ○ Unverified
+
+
+
Hardware
GB200
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Long-context text-only training is deferred; no long-context verification is claimed.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super PEFT recipe on 16 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super PEFT recipe on 16 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
+
+
+
+
+
+
diff --git a/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl.mdx b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl.mdx
index a1bec248d3..cb06b1cad7 100644
--- a/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl.mdx
+++ b/docs/fern/versions/nightly/pages/models/nemotron/nemotron3.5-super-vl.mdx
@@ -6,7 +6,9 @@ Megatron Bridge provides checkpoint conversion and pretraining, SFT, and PEFT re
The `freeze_vision_model`, `freeze_vision_projection`, and `freeze_language_model` provider options independently control which model components are trained. The Super-VL SFT recipes freeze the vision encoder while keeping the projection and language model trainable.
-> Verification scope: pretrain, SFT, PEFT, checkpoint resume, and dependent post-SFT export/inference remain unverified after the shared-provider callback fix. The commands, metrics, and artifact observations below are historical only: the copied provider configuration could omit training finalization callbacks, so finite losses and successful reloads do not establish synchronized, token-normalized updates. These runs also predate the current two-depth shared-MTP objective, FP32 optimizer-state and weight-decay defaults, and language-only PEFT targets. Pure conversion verification is unaffected. Historical throughput is not a corrected-training or optimized-performance baseline.
+The records below contain refreshed conversion, inference, and training verification with the corrected training callbacks and shared-MTP objective. Verification is scoped to each recorded workflow and hardware configuration; the GB200 pretrain/resume comparison and stronger visual diagnostics are not all passing. The public checkpoint revision remains unbound, and functional throughput measurements are not optimized-performance claims.
+
+For language-only conversion and training without media components, see [the text-only companion guide](nemotron3.5-super-vl-text-only.md).
@@ -75,10 +77,10 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
BF16
-
-
Expected result
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 H100s completed all 120 optimizer steps from the learned vision-pretrained checkpoint on DataComp image-caption samples at TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. Natural dropless routing and full/uniform one-layer language recompute, selective vision recompute, and 25% optimizer-state CPU offload plus the recipe's BF16 precision-aware optimizer produced finite loss, zero skipped or NaN iterations, and the resolved ConfigContainer. Full optimizer/RNG checkpoints at steps 60 and 120 each contain all 64 distributed model shards and 32 Energon data-parallel states.
+
Completed 200 finite optimizer steps on 64 H100s with zero skipped or NaN iterations, full checkpoints at steps 100 and 200, and a fresh final-checkpoint native inference reload. DataComp image-caption training warm-started from converted pretrained weights at sequence length 4096, GBS/MBS 1280/1, TP2/PP2/EP32/ETP1, natural HybridEP routing and FP32 optimizer state. Shared MTP uses two prediction depths and loss scale 0.3. Native finalization and update preflights passed for this workload. Rates are padded token slots per GPU, not actual supervised-token counts; production supervised totals were not recorded. This is bounded support verification, not full convergence.
@@ -313,52 +315,46 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. On exactly 64 GB200s within one NVL72 domain, complete all 120 optimizer steps from the learned vision-pretrained checkpoint on DataComp image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS 512/1. Natural dropless routing, finite loss, zero skipped or NaN iterations, no active language or vision recompute, scoped Transformer Engine CUDA-graph capture, and the resolved ConfigContainer are required. Full optimizer/RNG checkpoints at steps 60 and 120 each contain all 64 distributed model shards and 32 Energon data-parallel states.
+
Refreshed DataComp pretraining used 64 GB200 GPUs, pretrained weights, natural HybridEP routing, FP32 optimizer state, TP2/PP1/EP64/ETP1, sequence length 8192, GBS/MBS 512/1 and selective moe_act recompute. The original 500-step schedule reached step 475 before its four-hour limit. A fresh-root continuation from step 250 reached step 500 and saved a complete final checkpoint, whose native model reload passed. This is not an uninterrupted 500-step reference. One of 1125 matched loss observations exceeded the unchanged resume tolerance; independent replay remains pending, so no verified pretrain/resume pair is claimed.
SFT · H100
- ○ Unverified
+ ✓ Verified
Hardware
H100
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-23
Recorded metrics
Initial loss
-
None
+
3.375419
Final loss
-
None
+
1.958105
Step time · last 10 avg
-
None ms
+
42,907.800 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
159.840 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,909.210 tokens/s/GPU
@@ -369,47 +365,47 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 H100s completed 100 BF16 SFT optimizer steps from the immutable source checkpoint on 128,000 DataComp image-caption samples at TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. The shifted production loss masks contain 2,900,779 supervised tokens. Natural routing, HybridEP flex dispatch, fixed expert capacity, selective language recompute with vision recompute disabled, and the precision-aware optimizer produced finite loss with zero skipped or NaN iterations. The model-only step-100 checkpoint contains all 64 distributed shards and 32 Energon states and reloaded successfully. This dataset verifies image-caption batching; it does not contain video samples.
+
Completed 200 finite optimizer steps on 64 H100 GPUs with zero skipped or NaN iterations and a complete final checkpoint. DataComp image-caption training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP routing, FP32 optimizer state, and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Native finalization/update preflights passed. Final export and native HF reload are checked separately. Token rates are padded slots, not measured supervised-token totals. This verifies image-caption training, not video training or full convergence.
SFT · GB200
- ○ Unverified
+ ✓ Verified
Hardware
GB200
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-23
Recorded metrics
Initial loss
-
None
+
3.428863
Final loss
-
None
+
1.934663
Step time · last 10 avg
-
None ms
+
55,224.620 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
124.210 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,483.396 tokens/s/GPU
@@ -420,12 +416,12 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 GB200s within one NVL72 domain completed 100 BF16 SFT optimizer steps from the immutable source checkpoint on 128,000 DataComp image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS 1280/1. The shifted production loss masks contain 2,900,779 supervised tokens. Natural routing, eager all-to-all dispatch, fixed expert capacity, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The model-only step-100 checkpoint contains all 64 distributed shards and 32 Energon states and reloaded successfully. This dataset verifies image-caption batching; it does not contain video samples.
+
Completed 200 finite optimizer steps on 64 GB200 GPUs with zero skipped or NaN iterations and a complete final checkpoint. DataComp image-caption training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP routing, FP32 optimizer state, and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Native finalization/update preflights passed. Final export and native HF reload are checked separately. Token rates are padded slots, not measured supervised-token totals. This verifies image-caption training, not video training or full convergence.
@@ -522,35 +518,35 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
LoRA · H100
- ○ Unverified
+ ✓ Verified
Hardware
H100
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-24
Recorded metrics
Initial loss
-
None
+
2.483866
Final loss
-
None
+
2.596869
Step time · last 10 avg
-
None ms
+
5,634.590 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
60.860 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
726.938 tokens/s/GPU
@@ -561,47 +557,47 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 16 H100s completed the rank-32, alpha-32, zero-dropout LoRA objective for 100 optimizer steps from the immutable source checkpoint on 1,600 DataComp image-caption samples at TP4/PP2/CP1/EP8/ETP1 and GBS/MBS 16/1. The shifted production loss masks contain 35,540 supervised tokens. Frozen base weights, natural routing, dropless all-to-all dispatch, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The step-100 adapter contains all 16 distributed shards and two Energon states and reloaded successfully. This dataset does not contain video samples. This historical recipe used unqualified adapter targets that also matched vision and projector modules; it does not verify the current language-only adapter placement.
+
Completed 500 finite LoRA optimizer steps on 16 H100 GPUs with zero skipped or NaN iterations and a complete final native adapter checkpoint. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets. Native save/reload and merge preserve updated router expert-bias buffers as well as adapter tensors. Full merged HF reload and bounded image generation passed; stronger post-merge numerical diagnostics are not all passing. Standalone HF adapter export does not preserve these trained router buffers; use native reload and full-model merge/export. Token rates are padded slots, not measured supervised-token totals.
LoRA · GB200
- ○ Unverified
+ ✓ Verified
Hardware
GB200
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-24
Recorded metrics
Initial loss
-
None
+
3.45039
Final loss
-
None
+
2.059837
Step time · last 10 avg
-
None ms
+
3,640.690 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
94.230 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,125.061 tokens/s/GPU
@@ -612,12 +608,12 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 16 GB200s completed the rank-32, alpha-32, zero-dropout LoRA objective for 100 optimizer steps from the immutable source checkpoint on 1,600 DataComp image-caption samples at TP2/PP1/CP1/EP16/ETP1 and GBS/MBS 16/1. The shifted production loss masks contain 34,537 supervised tokens. Frozen base weights, natural routing, dropless all-to-all dispatch, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The step-100 adapter contains all 16 distributed shards and eight Energon states and reloaded successfully. This dataset does not contain video samples. This historical recipe used unqualified adapter targets that also matched vision and projector modules; it does not verify the current language-only adapter placement.
+
Completed 500 finite LoRA optimizer steps on 16 GB200 GPUs with zero skipped or NaN iterations and a complete final native adapter checkpoint. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets. Native save/reload and merge preserve updated router expert-bias buffers as well as adapter tensors. Full merged HF reload and bounded image generation passed; stronger post-merge numerical diagnostics are not all passing. Standalone HF adapter export does not preserve these trained router buffers; use native reload and full-model merge/export. Token rates are padded slots, not measured supervised-token totals.
diff --git a/docs/models/README.md b/docs/models/README.md
index 8980d68e87..b16a2599cb 100644
--- a/docs/models/README.md
+++ b/docs/models/README.md
@@ -4,7 +4,7 @@
Choose a model to open its interactive import/export and training configurations. Every option comes directly from an authoritative YAML verification card; independent fields are never combined into synthetic commands.
-Current inventory: **27 model cards** and **387 concrete configurations**.
+Current inventory: **28 model cards** and **405 concrete configurations**.
Each model page lets you select import/export, pretraining, benchmarking, SFT, LoRA, or long-context SFT and immediately see the exact command and expected result for the recorded precision and GPU.
@@ -112,7 +112,7 @@ Choose a model to inspect its recorded import/export, training, and precision co
NVIDIA
- 7 models
+ 8 models
diff --git a/docs/models/nemotron/index.md b/docs/models/nemotron/index.md
index 247861d86f..dc11355e4f 100644
--- a/docs/models/nemotron/index.md
+++ b/docs/models/nemotron/index.md
@@ -12,6 +12,7 @@ nemotron3-nano-4b.md
nemotron3-super.md
nemotron3.5-lightning.md
nemotron3.5-super-vl.md
+nemotron3.5-super-vl-text-only.md
nemotron3-ultra.md
nemotron-nano-v2-vl.md
nemotron-3-omni.md
@@ -26,6 +27,7 @@ nemotron-3-omni.md
| Nemotron-3 Super | [nemotron3-super.md](nemotron3-super.md) |
| Nemotron-3.5 Lightning | [nemotron3.5-lightning.md](nemotron3.5-lightning.md) |
| Nemotron-3.5 Super VL | [nemotron3.5-super-vl.md](nemotron3.5-super-vl.md) |
+| Nemotron-3.5 Super Text-only | [nemotron3.5-super-vl-text-only.md](nemotron3.5-super-vl-text-only.md) |
| Nemotron-3 Ultra | [nemotron3-ultra.md](nemotron3-ultra.md) |
| Nemotron Nano V2 VL (deprecated) | [nemotron-nano-v2-vl.md](nemotron-nano-v2-vl.md) |
| Nemotron-3 Nano Omni | [nemotron-3-omni.md](nemotron-3-omni.md) |
diff --git a/docs/models/nemotron/nemotron3.5-super-vl-text-only.md b/docs/models/nemotron/nemotron3.5-super-vl-text-only.md
new file mode 100644
index 0000000000..8bfd546455
--- /dev/null
+++ b/docs/models/nemotron/nemotron3.5-super-vl-text-only.md
@@ -0,0 +1,613 @@
+# Nemotron-3.5 Super Text-only
+
+The text-only bridge imports language and shared Multi-Token Prediction (MTP) weights from the Nemotron 3.5 Super VL checkpoint while excluding the vision encoder, projector, and temporal video embedder. Export produces a standalone `NemotronHForCausalLM` checkpoint that uses the native Nemotron-H text bridge.
+
+This workflow reuses [Nemotron 3 Super](nemotron3-super.md) text recipes with the language-only checkpoint as their initialization source. The records below describe the exact verified configurations, not every recipe or parallelism combination. See the [full-VL guide](nemotron3.5-super-vl.md) for image/video workflows.
+
+The public checkpoint revision remains unbound. CPU conversion, long-context training, and text-only pretrain/resume verification are deferred; their unverified status does not imply lack of implementation support.
+
+
+
+## Verified configurations
+
+Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
+
+
+### Run a configuration
+
+Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Precision
+
+
+
+
+
+
+ GPU
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Import · CPU
+ ○ Unverified
+
+
+
Hardware
not specified
+
Precision
BF16
+
Last verified
—
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
CPU conversion is deferred; no CPU-only verification is claimed.
+
Distributed GPU import and paired export passed on H100 and GB200. All 42,683 language tensors, including 1,040 MTP tensors, matched in keys, shapes, dtypes and values across the saved round trip. Compared 247,222,108,160 source tensor bytes; no media tensors were exported. This establishes weight conversion, not forward or training parity.
+
+
+
+
+
+
Export · CPU
+ ○ Unverified
+
+
+
Hardware
not specified
+
Precision
BF16
+
Last verified
—
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
CPU conversion is deferred; no CPU-only verification is claimed.
+
Exported a standalone native NemotronHForCausalLM checkpoint with unprefixed language and MTP weights and zero media tensors. All 42,683 tensors matched the source exactly, including 1,040 MTP tensors. The tokenizer was preserved, native HF config loading and normal text bridge dispatch passed, and the tensor-name/shape structure matched Nemotron 3 Super. This does not claim identical weights to that model.
+
+
+
+
+
+
Pretrain · H100
+ ○ Unverified
+
+
+
Hardware
H100
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+
+
+
+
+
+
Pretrain · GB200
+ ○ Unverified
+
+
+
Hardware
GB200
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+
Completed a fresh-root continuation from the original complete step-250 checkpoint through step 500 after the reference run failed at final save. All 1250 loss comparisons across steps 251-500 passed the unchanged 1% plus 1e-6 bound. Metrics below describe only that continuation, not an uninterrupted 500-step execution. Uses the native Nemotron 3 Super SFT recipe on 64 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super SFT recipe on 64 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
+
+
+
+
+
Long Context · H100
+ ○ Unverified
+
+
+
Hardware
H100
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Long-context text-only training is deferred; no long-context verification is claimed.
+
+
+
+
+
+
Long Context · GB200
+ ○ Unverified
+
+
+
Hardware
GB200
+
Precision
BF16
+
Last verified
—
+
+
+
Recorded metrics
+
+
+
Initial loss
+
None
+
+
+
Final loss
+
None
+
+
+
Step time · last 10 avg
+
None ms
+
+
+
Model throughput · last 10 avg
+
None TFLOP/s/GPU
+
+
+
Token throughput · last 10 avg
+
None tokens/s/GPU
+
+
+
+
+
Exact command
+
No runnable command is recorded for this status.
+
+
+
Expected result
+
Long-context text-only training is deferred; no long-context verification is claimed.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super PEFT recipe on 16 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint. Uses the native Nemotron 3 Super PEFT recipe on 16 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1. Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3. LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers. Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+
+
+
+
+
+
+
diff --git a/docs/models/nemotron/nemotron3.5-super-vl.md b/docs/models/nemotron/nemotron3.5-super-vl.md
index a1bec248d3..cb06b1cad7 100644
--- a/docs/models/nemotron/nemotron3.5-super-vl.md
+++ b/docs/models/nemotron/nemotron3.5-super-vl.md
@@ -6,7 +6,9 @@ Megatron Bridge provides checkpoint conversion and pretraining, SFT, and PEFT re
The `freeze_vision_model`, `freeze_vision_projection`, and `freeze_language_model` provider options independently control which model components are trained. The Super-VL SFT recipes freeze the vision encoder while keeping the projection and language model trainable.
-> Verification scope: pretrain, SFT, PEFT, checkpoint resume, and dependent post-SFT export/inference remain unverified after the shared-provider callback fix. The commands, metrics, and artifact observations below are historical only: the copied provider configuration could omit training finalization callbacks, so finite losses and successful reloads do not establish synchronized, token-normalized updates. These runs also predate the current two-depth shared-MTP objective, FP32 optimizer-state and weight-decay defaults, and language-only PEFT targets. Pure conversion verification is unaffected. Historical throughput is not a corrected-training or optimized-performance baseline.
+The records below contain refreshed conversion, inference, and training verification with the corrected training callbacks and shared-MTP objective. Verification is scoped to each recorded workflow and hardware configuration; the GB200 pretrain/resume comparison and stronger visual diagnostics are not all passing. The public checkpoint revision remains unbound, and functional throughput measurements are not optimized-performance claims.
+
+For language-only conversion and training without media components, see [the text-only companion guide](nemotron3.5-super-vl-text-only.md).
@@ -75,10 +77,10 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
BF16
-
-
Expected result
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 H100s completed all 120 optimizer steps from the learned vision-pretrained checkpoint on DataComp image-caption samples at TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. Natural dropless routing and full/uniform one-layer language recompute, selective vision recompute, and 25% optimizer-state CPU offload plus the recipe's BF16 precision-aware optimizer produced finite loss, zero skipped or NaN iterations, and the resolved ConfigContainer. Full optimizer/RNG checkpoints at steps 60 and 120 each contain all 64 distributed model shards and 32 Energon data-parallel states.
+
Completed 200 finite optimizer steps on 64 H100s with zero skipped or NaN iterations, full checkpoints at steps 100 and 200, and a fresh final-checkpoint native inference reload. DataComp image-caption training warm-started from converted pretrained weights at sequence length 4096, GBS/MBS 1280/1, TP2/PP2/EP32/ETP1, natural HybridEP routing and FP32 optimizer state. Shared MTP uses two prediction depths and loss scale 0.3. Native finalization and update preflights passed for this workload. Rates are padded token slots per GPU, not actual supervised-token counts; production supervised totals were not recorded. This is bounded support verification, not full convergence.
@@ -313,52 +315,46 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. On exactly 64 GB200s within one NVL72 domain, complete all 120 optimizer steps from the learned vision-pretrained checkpoint on DataComp image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS 512/1. Natural dropless routing, finite loss, zero skipped or NaN iterations, no active language or vision recompute, scoped Transformer Engine CUDA-graph capture, and the resolved ConfigContainer are required. Full optimizer/RNG checkpoints at steps 60 and 120 each contain all 64 distributed model shards and 32 Energon data-parallel states.
+
Refreshed DataComp pretraining used 64 GB200 GPUs, pretrained weights, natural HybridEP routing, FP32 optimizer state, TP2/PP1/EP64/ETP1, sequence length 8192, GBS/MBS 512/1 and selective moe_act recompute. The original 500-step schedule reached step 475 before its four-hour limit. A fresh-root continuation from step 250 reached step 500 and saved a complete final checkpoint, whose native model reload passed. This is not an uninterrupted 500-step reference. One of 1125 matched loss observations exceeded the unchanged resume tolerance; independent replay remains pending, so no verified pretrain/resume pair is claimed.
SFT · H100
- ○ Unverified
+ ✓ Verified
Hardware
H100
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-23
Recorded metrics
Initial loss
-
None
+
3.375419
Final loss
-
None
+
1.958105
Step time · last 10 avg
-
None ms
+
42,907.800 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
159.840 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,909.210 tokens/s/GPU
@@ -369,47 +365,47 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 H100s completed 100 BF16 SFT optimizer steps from the immutable source checkpoint on 128,000 DataComp image-caption samples at TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. The shifted production loss masks contain 2,900,779 supervised tokens. Natural routing, HybridEP flex dispatch, fixed expert capacity, selective language recompute with vision recompute disabled, and the precision-aware optimizer produced finite loss with zero skipped or NaN iterations. The model-only step-100 checkpoint contains all 64 distributed shards and 32 Energon states and reloaded successfully. This dataset verifies image-caption batching; it does not contain video samples.
+
Completed 200 finite optimizer steps on 64 H100 GPUs with zero skipped or NaN iterations and a complete final checkpoint. DataComp image-caption training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP routing, FP32 optimizer state, and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Native finalization/update preflights passed. Final export and native HF reload are checked separately. Token rates are padded slots, not measured supervised-token totals. This verifies image-caption training, not video training or full convergence.
SFT · GB200
- ○ Unverified
+ ✓ Verified
Hardware
GB200
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-23
Recorded metrics
Initial loss
-
None
+
3.428863
Final loss
-
None
+
1.934663
Step time · last 10 avg
-
None ms
+
55,224.620 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
124.210 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,483.396 tokens/s/GPU
@@ -420,12 +416,12 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 64 GB200s within one NVL72 domain completed 100 BF16 SFT optimizer steps from the immutable source checkpoint on 128,000 DataComp image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS 1280/1. The shifted production loss masks contain 2,900,779 supervised tokens. Natural routing, eager all-to-all dispatch, fixed expert capacity, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The model-only step-100 checkpoint contains all 64 distributed shards and 32 Energon states and reloaded successfully. This dataset verifies image-caption batching; it does not contain video samples.
+
Completed 200 finite optimizer steps on 64 GB200 GPUs with zero skipped or NaN iterations and a complete final checkpoint. DataComp image-caption training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP routing, FP32 optimizer state, and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Native finalization/update preflights passed. Final export and native HF reload are checked separately. Token rates are padded slots, not measured supervised-token totals. This verifies image-caption training, not video training or full convergence.
@@ -522,35 +518,35 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
LoRA · H100
- ○ Unverified
+ ✓ Verified
Hardware
H100
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-24
Recorded metrics
Initial loss
-
None
+
2.483866
Final loss
-
None
+
2.596869
Step time · last 10 avg
-
None ms
+
5,634.590 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
60.860 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
726.938 tokens/s/GPU
@@ -561,47 +557,47 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-29. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 16 H100s completed the rank-32, alpha-32, zero-dropout LoRA objective for 100 optimizer steps from the immutable source checkpoint on 1,600 DataComp image-caption samples at TP4/PP2/CP1/EP8/ETP1 and GBS/MBS 16/1. The shifted production loss masks contain 35,540 supervised tokens. Frozen base weights, natural routing, dropless all-to-all dispatch, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The step-100 adapter contains all 16 distributed shards and two Energon states and reloaded successfully. This dataset does not contain video samples. This historical recipe used unqualified adapter targets that also matched vision and projector modules; it does not verify the current language-only adapter placement.
+
Completed 500 finite LoRA optimizer steps on 16 H100 GPUs with zero skipped or NaN iterations and a complete final native adapter checkpoint. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets. Native save/reload and merge preserve updated router expert-bias buffers as well as adapter tensors. Full merged HF reload and bounded image generation passed; stronger post-merge numerical diagnostics are not all passing. Standalone HF adapter export does not preserve these trained router buffers; use native reload and full-model merge/export. Token rates are padded slots, not measured supervised-token totals.
LoRA · GB200
- ○ Unverified
+ ✓ Verified
Hardware
GB200
Precision
BF16
-
Last verified
—
+
Last verified
2026-09-24
Recorded metrics
Initial loss
-
None
+
3.45039
Final loss
-
None
+
2.059837
Step time · last 10 avg
-
None ms
+
3,640.690 ms
Model throughput · last 10 avg
-
None TFLOP/s/GPU
+
94.230 TFLOP/s/GPU
Token throughput · last 10 avg
-
None tokens/s/GPU
+
1,125.061 tokens/s/GPU
@@ -612,12 +608,12 @@ Choose a workflow, precision, and exact recorded combination. The command and ex
Command
-
Historical-only observations recorded on 2026-08-28. The command, metrics and artifact observations do not establish synchronized, token-normalized training updates. Recipe-specific reverification after the shared-provider callback fix is pending. Exactly 16 GB200s completed the rank-32, alpha-32, zero-dropout LoRA objective for 100 optimizer steps from the immutable source checkpoint on 1,600 DataComp image-caption samples at TP2/PP1/CP1/EP16/ETP1 and GBS/MBS 16/1. The shifted production loss masks contain 34,537 supervised tokens. Frozen base weights, natural routing, dropless all-to-all dispatch, and no active language or vision recompute produced finite loss with zero skipped or NaN iterations. The step-100 adapter contains all 16 distributed shards and eight Energon states and reloaded successfully. This dataset does not contain video samples. This historical recipe used unqualified adapter targets that also matched vision and projector modules; it does not verify the current language-only adapter placement.
+
Completed 500 finite LoRA optimizer steps on 16 GB200 GPUs with zero skipped or NaN iterations and a complete final native adapter checkpoint. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets. Native save/reload and merge preserve updated router expert-bias buffers as well as adapter tensors. Full merged HF reload and bounded image generation passed; stronger post-merge numerical diagnostics are not all passing. Standalone HF adapter export does not preserve these trained router buffers; use native reload and full-model merge/export. Token rates are padded slots, not measured supervised-token totals.
diff --git a/docs/training/peft.md b/docs/training/peft.md
index 35f1bbceb7..5fc33d0a50 100644
--- a/docs/training/peft.md
+++ b/docs/training/peft.md
@@ -42,6 +42,25 @@ config = ConfigContainer(
When resuming adapter training, keep `checkpoint.pretrained_checkpoint` pointed at the same base model and set `checkpoint.load` to the PEFT adapter checkpoint directory.
```
+### MoE router state and checkpoint compatibility
+
+When expert-bias routing is enabled, gradient finalization updates the router's
+`expert_bias` buffer even when the router weights are frozen. PEFT checkpoints
+therefore retain these buffers alongside adapter parameters. Adapter reload and
+merge into a full model preserve this state.
+
+Older adapter checkpoints may omit these buffers. The adapter inference/merge
+loader rejects such checkpoints for a bias-enabled model, including when its
+`strict=False` option is used: that option does not permit missing distributed
+checkpoint tensors. The separate training-resume loader follows
+`checkpoint.dist_ckpt_strictness`; a permissive policy may retain the base-model
+bias, which is not exact recovery of the trained router state. Missing trained
+biases cannot be reconstructed from adapter weights alone.
+
+Standalone Hugging Face adapter export does not serialize router-bias buffers.
+For models whose biases changed during PEFT, merge the native adapter checkpoint
+into the base model and export the resulting full Hugging Face model instead.
+
## Supported PEFT Methods
### [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
@@ -402,7 +421,7 @@ The PEFT framework introduces a modular design for integrating adapters into lar
1. **Base PEFT Class**: All PEFT methods inherit from the abstract {py:class}`bridge.peft.base.PEFT` base class, which defines the core interface for module transformation.
2. **Module Transformation**: PEFT traverses the model structure to identify and transform target modules individually.
3. **Adapter Integration**: Adapters are injected into selected modules using a pre-wrap hook during model initialization.
-4. **Checkpoint Integration**: Only adapter parameters are saved and loaded during checkpointing; base model weights remain frozen and unchanged.
+4. **Checkpoint Integration**: Adapter parameters and updated MoE router expert-bias buffers are saved and loaded during checkpointing; base model weights remain frozen.
### PEFT Workflow in Training
@@ -410,8 +429,8 @@ The training workflow for PEFT follows a structured sequence that ensures effici
1. **Model Loading**: The base model is initialized from a specified pretrained checkpoint.
2. **PEFT Application**: Adapter transformations are applied after Megatron Core model initialization, but before distributed wrapping.
3. **Parameter Freezing**: Base model parameters are frozen to reduce training complexity; only adapter parameters are updated.
-4. **Adapter Weight Loading**: When resuming training, adapter weights are restored from the checkpoint.
-5. **Checkpoint Saving**: Only adapter states are saved, resulting in significantly smaller checkpoint files.
+4. **Adapter Weight Loading**: When resuming training, adapter weights and saved router expert-bias buffers are restored from the checkpoint.
+5. **Checkpoint Saving**: Adapter states and router expert-bias buffers are saved, resulting in significantly smaller checkpoint files than full-model checkpoints.
### Key Benefits
diff --git a/examples/conversion/compare_hf_and_megatron/compare.py b/examples/conversion/compare_hf_and_megatron/compare.py
index f5e11fb0b1..be1e53d4eb 100644
--- a/examples/conversion/compare_hf_and_megatron/compare.py
+++ b/examples/conversion/compare_hf_and_megatron/compare.py
@@ -782,6 +782,18 @@ def _load_megatron_model(args):
"pipeline_model_parallel_size": pp,
"expert_model_parallel_size": ep,
"expert_tensor_parallel_size": etp,
+ # Preserve the saved dispatcher/backend, but never drop tokens or
+ # use synthetic routing when comparing against HF inference.
+ "moe_expert_capacity_factor": None,
+ "moe_expert_rank_capacity_factor": None,
+ "moe_paged_stash": False,
+ # Zero-copy NCCL-EP requires the fixed capacity disabled above.
+ "moe_ncclep_zero_copy": False,
+ "moe_pad_expert_input_to_capacity": False,
+ "moe_router_force_load_balancing": False,
+ "moe_router_force_biased": None,
+ # Eager prompts can have unequal or unaligned token counts.
+ "moe_hybridep_pad_uneven_dispatch_inputs": True,
},
wrap_with_ddp=False,
)
diff --git a/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b-text-only/card.yaml b/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b-text-only/card.yaml
new file mode 100644
index 0000000000..bde81e7282
--- /dev/null
+++ b/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b-text-only/card.yaml
@@ -0,0 +1,486 @@
+# Agent-readable companion card for the language-only workflow.
+# Results are populated independently of the full-VL card.
+title: nemotron_3_5_super_vl_120b_a12b_text_only
+summary: >
+ Performance disclaimer: this card does not record a canonical pretrain
+ performance result; reported timing and throughput metrics are functional
+ verification observations, not standalone optimized performance results.
+ This companion card covers importing only language and MTP weights from
+ the VL checkpoint, reusing Nemotron 3 Super text recipes, and exporting a
+ standalone NemotronHForCausalLM checkpoint. It excludes all media encoders,
+ projectors, and video embedding weights. The full-VL card is separate.
+ SFT and PEFT use the prepared full Tulu 3 training split: 892,375 examples
+ after the native 95/5 split, data and packing seed 5678, model seed 1234,
+ 2048-token offline packs aligned to eight tokens, and assistant-only loss.
+ H100 and GB200 use byte-identical packed training and validation files;
+ their training metrics count padded token slots, not supervised tokens.
+ The public revision is intentionally blank; results must not be interpreted
+ as bound to an immutable public release revision. Refreshed SFT and PEFT,
+ dependent export, merge, and reload checks are recorded for H100 and GB200.
+verification_index:
+ model_level:
+ verified: [hf_to_megatron_gpu, megatron_to_hf_gpu, manual_forward_pass, inference]
+ unverified: [hf_to_megatron_cpu, megatron_to_hf_cpu]
+ training:
+ H100:
+ verified: [sft, sft_export_inference, peft]
+ unverified: [pretrain, sft_long_context, checkpoint_resume]
+ GB200:
+ verified: [sft, sft_export_inference, peft]
+ unverified: [pretrain, sft_long_context, checkpoint_resume]
+model:
+ hf_id: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ hf_revision:
+ architecture: NemotronHForCausalLM
+ min_transformers_version: "5.12.1"
+verification_environment:
+ base_container: nvcr.io/nvidia/nemo:26.08.01
+ bridge_commit: 5e63b9d9ebb0230199aaf6177efa86e06bbce5d3 # pragma: allowlist secret
+items:
+ hf_to_megatron_cpu:
+ status: unverified
+ precision: bf16
+ command: null
+ last_verified: null
+ expected_result: >
+ CPU conversion is deferred; no CPU-only verification is claimed.
+ hf_to_megatron_gpu:
+ status: verified
+ precision: bf16
+ bridge_commit: 3a5b4f610a5b3608b1bded146418459672492cd9 # pragma: allowlist secret
+ command: >
+ ./scripts/conversion/convert.sh import --executor slurm --device gpu
+ --nodes 1 --gpus-per-node 4
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --text-only --trust-remote-code --torch-dtype bfloat16
+ --tp 1 --pp 2 --ep 2 --etp 1
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-text-only/megatron
+ last_verified: 2026-09-23
+ expected_result: >
+ Distributed GPU import and paired export passed on H100 and GB200.
+ All 42,683 language tensors, including 1,040 MTP tensors, matched in
+ keys, shapes, dtypes and values across the saved round trip. Compared
+ 247,222,108,160 source tensor bytes; no media tensors were exported.
+ This establishes weight conversion, not forward or training parity.
+ megatron_to_hf_cpu:
+ status: unverified
+ precision: bf16
+ command: null
+ last_verified: null
+ expected_result: >
+ CPU conversion is deferred; no CPU-only verification is claimed.
+ megatron_to_hf_gpu:
+ status: verified
+ precision: bf16
+ bridge_commit: 3a5b4f610a5b3608b1bded146418459672492cd9 # pragma: allowlist secret
+ command: >
+ ./scripts/conversion/convert.sh export --executor slurm --device gpu
+ --nodes 1 --gpus-per-node 4
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --text-only --trust-remote-code --torch-dtype bfloat16
+ --tp 1 --pp 2 --ep 2 --etp 1 --distributed-save
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-text-only/megatron
+ --hf-path work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ last_verified: 2026-09-23
+ expected_result: >
+ Exported a standalone native NemotronHForCausalLM checkpoint with
+ unprefixed language and MTP weights and zero media tensors. All 42,683
+ tensors matched the source exactly, including 1,040 MTP tensors.
+ The tokenizer was preserved, native HF config loading and normal text
+ bridge dispatch passed, and the tensor-name/shape structure matched
+ Nemotron 3 Super. This does not claim identical weights to that model.
+ manual_forward_pass:
+ status: verified
+ precision: bf16
+ command: >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --task model-comparison
+ --hf_model_path work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ --megatron_model_path work/model-verification/nemotron-3.5-super-vl-text-only/megatron/iter_0000000
+ --hf-logits-path work/model-verification/nemotron-3.5-super-vl-text-only/hf-reference.pt
+ --prompt $'<|im_start|>system\n<|im_end|>\n<|im_start|>user\nWhat is two plus two?<|im_end|>\n<|im_start|>assistant\n'
+ --tp 1 --pp 1 --ep 4 --etp 1 --trust-remote-code
+ last_verified: 2026-09-27
+ expected_result: >
+ The maintained public comparison passed on four GB200 GPUs with the
+ native 22-token chat prompt and a separately saved native-VL HF
+ no-media reference for the identical source weights and input. This
+ compares that reference with the saved text-only Megatron checkpoint;
+ standalone text HF reload is verified separately below.
+ The next token matches: 16489 (Two).
+ Cosine similarity is 0.999996; maximum and mean absolute logit
+ differences are 0.500000 and 0.063002. Bash ANSI-C quoting preserves
+ the native prompt's newline characters.
+ This is single-prompt forward correlation, not strict probability
+ equality, generation, or MTP forward verification. Regenerate the HF
+ reference if the checkpoint changes. The public revision remains unbound.
+ inference:
+ status: verified
+ precision: bf16
+ command: >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --task text-generation
+ --hf_model_path work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ --megatron_model_path work/model-verification/nemotron-3.5-super-vl-text-only/megatron/iter_0000000
+ --prompt $'<|im_start|>system\n<|im_end|>\n<|im_start|>user\nWhat is two plus two?<|im_end|>\n<|im_start|>assistant\n'
+ --tp 1 --pp 1 --ep 4 --etp 1 --max_new_tokens 32 --max_seq_length 256
+ --temperature 0 --top_k 1 --use-coordinator --coordinator-host 127.0.0.1
+ --return-log-probs --skip-prompt-log-probs --trust-remote-code
+ last_verified: 2026-09-27
+ expected_result: >
+ The maintained cached text-generation path completed on four GB200
+ GPUs, using the same native 22-token chat prompt as the comparison.
+ The literal completion was "Two plus two equals four." Generated 7
+ tokens including EOS under the 32-token maximum, confirmed by the
+ returned per-token log probabilities. Greedy sampling used temperature
+ zero and top-k one. The loopback coordinator is for this single-node
+ execution only. This is one bounded generation sanity check, not
+ cross-process repeatability or broad HF generation equality.
+ pretrain:
+ H100:
+ status: unverified
+ precision: bf16
+ enabled_features: {}
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ expected_result: >
+ Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+ GB200:
+ status: unverified
+ precision: bf16
+ enabled_features: {}
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ expected_result: >
+ Additional text-only pretrain and resume verification is deferred. Recipe construction alone does not establish training or resume correctness.
+ sft:
+ H100:
+ status: verified
+ precision: bf16
+ enabled_features:
+ sequence_packing: offline
+ moe_dispatcher: hybridep
+ command: >
+ ./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8
+ --recipe nemotron_3_super_sft_16gpu_h100_bf16_config --mode sft --dataset tulu3
+ --seq_length 2048 --max_steps 500 --save_interval 250
+ --load_dir work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft checkpoint.ckpt_step=250 checkpoint.finetune=false
+ --save_dir work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-recovery
+ model.hf_model_id=work/model-verification/nemotron-3.5-super-vl-text-only/hf tokenizer.tokenizer_model=work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ ++tokenizer.hf_tokenizer_kwargs.trust_remote_code=false
+ model.expert_model_parallel_size=64 model.moe_token_dispatcher_type=flex
+ model.moe_flex_dispatcher_backend=hybridep model.moe_shared_expert_overlap=false
+ model.moe_router_force_load_balancing=false
+ optimizer.use_precision_aware_optimizer=false optimizer.main_grads_dtype=float32
+ optimizer.main_params_dtype=float32 optimizer.exp_avg_dtype=float32 optimizer.exp_avg_sq_dtype=float32
+ mixed_precision.grad_reduce_in_fp32=true ddp.grad_reduce_in_fp32=true
+ dataset.hf_dataset=null dataset.hf_validation_proportion=null
+ dataset.dataset_root=work/data/tulu3/jsonl dataset.do_validation=false dataset.do_test=false
+ dataset.enable_offline_packing=true ++dataset.dataset_kwargs.pad_to_max_length=true
+ ++dataset.dataset_kwargs.index_mapping_dir=work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-recovery-indices
+ 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:8,packed_train_data_path:work/data/tulu3/packed/training.idx.parquet,packed_val_data_path:work/data/tulu3/packed/validation.idx.parquet,packed_metadata_path:work/data/tulu3/packed/metadata.jsonl}'
+ scheduler.lr_decay_iters=500 validation.eval_iters=0 validation.eval_interval=0
+ checkpoint.load_optim=true checkpoint.load_rng=true
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-recovery-config/ConfigContainer.yaml
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-27
+ metrics:
+ initial_loss: 0.6722764
+ final_loss: 0.5672588
+ last_10_steps_step_time_ms_avg: 3792.22
+ last_10_steps_model_tflops_per_gpu_avg: 23.57
+ last_10_steps_tokens_per_second_per_gpu_avg: 270.02652799679345
+ expected_result: >
+ Completed a fresh-root continuation from the original complete step-250 checkpoint through step 500 after the reference run failed at final save. All 1250 loss comparisons across steps 251-500 passed the unchanged 1% plus 1e-6 bound. Metrics below describe only that continuation, not an uninterrupted 500-step execution.
+ Uses the native Nemotron 3 Super SFT recipe on 64 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1.
+ Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3.
+ Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality.
+ Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+ GB200:
+ status: verified
+ precision: bf16
+ enabled_features:
+ sequence_packing: offline
+ moe_dispatcher: hybridep
+ command: >
+ ./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4
+ --recipe nemotron_3_super_sft_16gpu_h100_bf16_config --mode sft --dataset tulu3
+ --seq_length 2048 --max_steps 500 --save_interval 250
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-text-only/megatron/iter_0000000
+ --save_dir work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft
+ model.hf_model_id=work/model-verification/nemotron-3.5-super-vl-text-only/hf tokenizer.tokenizer_model=work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ ++tokenizer.hf_tokenizer_kwargs.trust_remote_code=false
+ model.expert_model_parallel_size=64 model.moe_token_dispatcher_type=flex
+ model.moe_flex_dispatcher_backend=hybridep model.moe_shared_expert_overlap=false
+ model.moe_router_force_load_balancing=false
+ optimizer.use_precision_aware_optimizer=false optimizer.main_grads_dtype=float32
+ optimizer.main_params_dtype=float32 optimizer.exp_avg_dtype=float32 optimizer.exp_avg_sq_dtype=float32
+ mixed_precision.grad_reduce_in_fp32=true ddp.grad_reduce_in_fp32=true
+ dataset.hf_dataset=null dataset.hf_validation_proportion=null
+ dataset.dataset_root=work/data/tulu3/jsonl dataset.do_validation=false dataset.do_test=false
+ dataset.enable_offline_packing=true ++dataset.dataset_kwargs.pad_to_max_length=true
+ ++dataset.dataset_kwargs.index_mapping_dir=work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft-indices
+ 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:8,packed_train_data_path:work/data/tulu3/packed/training.idx.parquet,packed_val_data_path:work/data/tulu3/packed/validation.idx.parquet,packed_metadata_path:work/data/tulu3/packed/metadata.jsonl}'
+ scheduler.lr_decay_iters=500 validation.eval_iters=0 validation.eval_interval=0
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft-config/ConfigContainer.yaml
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-27
+ metrics:
+ initial_loss: 0.5975891
+ final_loss: 0.5676602
+ last_10_steps_step_time_ms_avg: 6013.67
+ last_10_steps_model_tflops_per_gpu_avg: 14.42
+ last_10_steps_tokens_per_second_per_gpu_avg: 170.2787149943379
+ expected_result: >
+ Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint.
+ Uses the native Nemotron 3 Super SFT recipe on 64 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP64/ETP1, sequence length 2048 and recipe GBS/MBS 32/1.
+ Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3.
+ Final export, native HF reload and one matched numerical fixture passed; those do not establish general probability equality.
+ Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+ sft_export_inference:
+ H100:
+ status: verified
+ precision: bf16
+ depends_on: sft
+ commands:
+ - >
+ ./scripts/conversion/convert.sh export --executor slurm --device gpu
+ --nodes 1 --gpus-per-node 4
+ --hf-model work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-recovery/iter_0000500
+ --hf-path work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-hf
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --distributed-timeout-minutes 60 --distributed-save
+ - >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --tasks-per-node 1 --task hf-inference
+ --hf-model work/model-verification/nemotron-3.5-super-vl-text-only/h100-sft-hf
+ --prompt "What is two plus two?" --max-new-tokens 32
+ --chat-template --disable-thinking --device cuda --device-map balanced
+ --dtype bfloat16 --require-gpu-only
+ last_verified: 2026-09-27
+ expected_result: >
+ Exported the final step-500 language checkpoint through its standalone
+ native text source, retaining 42,683 language tensors including 1,040
+ MTP tensors and zero media tensors. The only config differences from
+ the pretrained text source were initializer_range=0.014 and
+ max_position_embeddings=2048, both checked against the saved training
+ config. Strict native Transformers reload covered 862 modules and
+ generated exactly 7 tokens, stopping at EOS before the 32-token maximum. The literal
+ completion was "Two plus two equals four.". This validates full-model
+ export and one greedy completion, not native HF MTP execution or broad
+ numerical equivalence. The separate matched six-position forward
+ comparison also passed without changing numerical tolerances.
+ GB200:
+ status: verified
+ precision: bf16
+ depends_on: sft
+ commands:
+ - >
+ ./scripts/conversion/convert.sh export --executor slurm --device gpu
+ --nodes 1 --gpus-per-node 4
+ --hf-model work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft/iter_0000500
+ --hf-path work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft-hf
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --distributed-timeout-minutes 60 --distributed-save
+ - >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --tasks-per-node 1 --task hf-inference
+ --hf-model work/model-verification/nemotron-3.5-super-vl-text-only/gb200-sft-hf
+ --prompt "What is two plus two?" --max-new-tokens 32
+ --chat-template --disable-thinking --device cuda --device-map balanced
+ --dtype bfloat16 --require-gpu-only
+ last_verified: 2026-09-27
+ expected_result: >
+ Exported the final step-500 language checkpoint through its standalone
+ native text source, retaining 42,683 language tensors including 1,040
+ MTP tensors and zero media tensors. The only config differences from
+ the pretrained text source were initializer_range=0.014 and
+ max_position_embeddings=2048, both checked against the saved training
+ config. Strict native Transformers reload covered 862 modules and
+ generated exactly 7 tokens, stopping at EOS before the 32-token maximum. The literal
+ completion was "Two plus two equals four.". This validates full-model
+ export and one greedy completion, not native HF MTP execution or broad
+ numerical equivalence. The separate matched six-position forward
+ comparison also passed without changing numerical tolerances.
+ sft_long_context:
+ H100:
+ status: unverified
+ precision: bf16
+ enabled_features: {}
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ expected_result: >
+ Long-context text-only training is deferred; no long-context verification is claimed.
+ GB200:
+ status: unverified
+ precision: bf16
+ enabled_features: {}
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ expected_result: >
+ Long-context text-only training is deferred; no long-context verification is claimed.
+ peft:
+ H100:
+ status: verified
+ precision: bf16
+ enabled_features:
+ sequence_packing: offline
+ moe_dispatcher: hybridep
+ command: >
+ ./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8
+ --recipe nemotron_3_super_peft_16gpu_h100_bf16_config --mode lora --dataset tulu3
+ --seq_length 2048 --max_steps 500 --save_interval 250
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-text-only/megatron/iter_0000000
+ --save_dir work/model-verification/nemotron-3.5-super-vl-text-only/h100-peft
+ model.hf_model_id=work/model-verification/nemotron-3.5-super-vl-text-only/hf tokenizer.tokenizer_model=work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ ++tokenizer.hf_tokenizer_kwargs.trust_remote_code=false
+ model.expert_model_parallel_size=16 model.moe_token_dispatcher_type=flex
+ model.moe_flex_dispatcher_backend=hybridep model.moe_shared_expert_overlap=false
+ model.moe_router_force_load_balancing=false
+ optimizer.use_precision_aware_optimizer=false optimizer.main_grads_dtype=float32
+ optimizer.main_params_dtype=float32 optimizer.exp_avg_dtype=float32 optimizer.exp_avg_sq_dtype=float32
+ ddp.grad_reduce_in_fp32=true
+ dataset.hf_dataset=null dataset.hf_validation_proportion=null
+ dataset.dataset_root=work/data/tulu3/jsonl dataset.do_validation=false dataset.do_test=false
+ dataset.enable_offline_packing=true ++dataset.dataset_kwargs.pad_to_max_length=true
+ ++dataset.dataset_kwargs.index_mapping_dir=work/model-verification/nemotron-3.5-super-vl-text-only/h100-peft-indices
+ 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:8,packed_train_data_path:work/data/tulu3/packed/training.idx.parquet,packed_val_data_path:work/data/tulu3/packed/validation.idx.parquet,packed_metadata_path:work/data/tulu3/packed/metadata.jsonl}'
+ scheduler.lr_decay_iters=500 validation.eval_iters=0 validation.eval_interval=0
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-text-only/h100-peft-config/ConfigContainer.yaml
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-27
+ metrics:
+ initial_loss: 0.6401206
+ final_loss: 0.6625465
+ last_10_steps_step_time_ms_avg: 8122.16
+ last_10_steps_model_tflops_per_gpu_avg: 21.22
+ last_10_steps_tokens_per_second_per_gpu_avg: 252.14967447083043
+ expected_result: >
+ Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint.
+ Uses the native Nemotron 3 Super PEFT recipe on 16 H100 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1.
+ Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3.
+ LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers.
+ Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+ GB200:
+ status: verified
+ precision: bf16
+ enabled_features:
+ sequence_packing: offline
+ moe_dispatcher: hybridep
+ command: >
+ ./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4
+ --recipe nemotron_3_super_peft_16gpu_h100_bf16_config --mode lora --dataset tulu3
+ --seq_length 2048 --max_steps 500 --save_interval 250
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-text-only/megatron/iter_0000000
+ --save_dir work/model-verification/nemotron-3.5-super-vl-text-only/gb200-peft
+ model.hf_model_id=work/model-verification/nemotron-3.5-super-vl-text-only/hf tokenizer.tokenizer_model=work/model-verification/nemotron-3.5-super-vl-text-only/hf
+ ++tokenizer.hf_tokenizer_kwargs.trust_remote_code=false
+ model.expert_model_parallel_size=16 model.moe_token_dispatcher_type=flex
+ model.moe_flex_dispatcher_backend=hybridep model.moe_shared_expert_overlap=false
+ model.moe_router_force_load_balancing=false
+ optimizer.use_precision_aware_optimizer=false optimizer.main_grads_dtype=float32
+ optimizer.main_params_dtype=float32 optimizer.exp_avg_dtype=float32 optimizer.exp_avg_sq_dtype=float32
+ ddp.grad_reduce_in_fp32=true
+ dataset.hf_dataset=null dataset.hf_validation_proportion=null
+ dataset.dataset_root=work/data/tulu3/jsonl dataset.do_validation=false dataset.do_test=false
+ dataset.enable_offline_packing=true ++dataset.dataset_kwargs.pad_to_max_length=true
+ ++dataset.dataset_kwargs.index_mapping_dir=work/model-verification/nemotron-3.5-super-vl-text-only/gb200-peft-indices
+ 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:8,packed_train_data_path:work/data/tulu3/packed/training.idx.parquet,packed_val_data_path:work/data/tulu3/packed/validation.idx.parquet,packed_metadata_path:work/data/tulu3/packed/metadata.jsonl}'
+ scheduler.lr_decay_iters=500 validation.eval_iters=0 validation.eval_interval=0
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-text-only/gb200-peft-config/ConfigContainer.yaml
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-27
+ metrics:
+ initial_loss: 0.6392546
+ final_loss: 0.6631947
+ last_10_steps_step_time_ms_avg: 13720.5
+ last_10_steps_model_tflops_per_gpu_avg: 12.72
+ last_10_steps_tokens_per_second_per_gpu_avg: 149.26569731423783
+ expected_result: >
+ Completed 500 finite optimizer steps with zero skipped or NaN iterations and a complete final checkpoint.
+ Uses the native Nemotron 3 Super PEFT recipe on 16 GB200 GPUs with this model's converted language weights and tokenizer, no media modules, TP8/PP1/EP16/ETP1, sequence length 2048 and recipe GBS/MBS 16/1.
+ Tulu 3 offline-packed data uses assistant-only main and MTP loss masking, natural HybridEP routing and FP32 optimizer state. Both shared MTP prediction depths use loss scale 0.3.
+ LoRA rank/alpha 32/32 targets native language projections with zero dropout. Native adapter and trained router-bias buffers reloaded exactly; full merge, strict HF reload and one matched numerical fixture passed. Standalone HF adapter export does not preserve trained router-bias buffers.
+ Token rates are padded slots, not measured supervised-token counts. This is bounded support verification, not full convergence or a cross-model convergence comparison.
+ checkpoint_resume:
+ H100:
+ status: unverified
+ precision: bf16
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ depends_on: pretrain
+ resume_comparison:
+ reference_item: pretrain
+ sentinel_steps: null
+ loss_relative_tolerance: 0.01
+ loss_absolute_tolerance: 0.000001
+ sentinels_match: null
+ expected_result: >
+ Additional text-only pretrain and resume verification is deferred; no cross-hardware or VL checkpoint-resume result is substituted.
+ GB200:
+ status: unverified
+ precision: bf16
+ command: null
+ last_verified: null
+ metrics:
+ initial_loss: null
+ final_loss: null
+ last_10_steps_step_time_ms_avg: null
+ last_10_steps_model_tflops_per_gpu_avg: null
+ last_10_steps_tokens_per_second_per_gpu_avg: null
+ depends_on: pretrain
+ resume_comparison:
+ reference_item: pretrain
+ sentinel_steps: null
+ loss_relative_tolerance: 0.01
+ loss_absolute_tolerance: 0.000001
+ sentinels_match: null
+ expected_result: >
+ Additional text-only pretrain and resume verification is deferred; no cross-hardware or VL checkpoint-resume result is substituted.
diff --git a/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b/card.yaml b/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b/card.yaml
index 353facedf3..e2b1c5b052 100644
--- a/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b/card.yaml
+++ b/examples/model_verification_cards/nemotron-3.5-super-vl-120b-a12b/card.yaml
@@ -6,52 +6,41 @@ summary: >
Performance disclaimer: this card does not record a canonical pretrain
performance result; reported timing and throughput metrics are functional
verification observations, not standalone optimized performance results.
- The model ID is the release-facing name. Recorded revisions and results
- refer to historical development checkpoints, not verification of the release
- repository; the displayed commands require an available matching checkpoint.
- Historical training commands, metrics, checkpoint and reload observations
- are retained below, but pretrain, SFT, PEFT and checkpoint-resume correctness
- are unverified on both H100 and GB200. The historical shared-provider config
- copy could omit training finalization callbacks, so finite losses and
- matching resume losses do not establish synchronized, token-normalized
- updates. Recipe-specific reverification after the callback fix is pending.
- Those runs also used one MTP prediction depth and predate the current
- two-depth shared-block objective, FP32 optimizer-state and weight-decay
- defaults, and language-only PEFT adapter targets. Their metrics are not
- corrected-training convergence or throughput baselines. The H100 and GB200
- workloads also used different sequence and global batch sizes.
- The pinned eight-GPU BF16 import and distributed export remain verified:
- all 43,078 tensors matched the immutable HF source bitwise, and the exported
- package reloaded natively through Transformers on eight H100s. This
- conversion evidence is independent of training finalization. Model-level
- deterministic generation remains pending. Historical post-SFT export and
- dynamic-image inference demonstrate artifact execution only; their
- dependent verification awaits a corrected SFT checkpoint. The recipes
- reuse the Nemotron Omni image/video stack without audio and retain the
- checkpoint's separate temporal embedder.
+ Refreshed verification uses the release-facing model name and the official
+ container below. The public revision is intentionally blank; these results
+ do not establish correspondence to an immutable public release revision.
+ The model reuses the Omni image/video stack without audio and retains a
+ separate temporal embedder. Training uses fixed-resolution images;
+ inference exercises dynamic-resolution processing. This is bounded support
+ verification, not a cross-model convergence comparison. Stronger visual
+ parity diagnostics and the GB200 resume comparison are not all passing.
+ CPU conversion and long-context training are deferred. The companion card
+ separately covers text-only workflows.
verification_index:
model_level:
verified:
- hf_to_megatron_gpu
- megatron_to_hf_gpu
+ - manual_forward_pass
+ - inference
unverified:
- hf_to_megatron_cpu
- megatron_to_hf_cpu
- - manual_forward_pass
- - inference
training:
H100:
- unverified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
+ verified: [pretrain, sft, sft_export_inference, peft, checkpoint_resume]
+ unverified: [sft_long_context]
GB200:
- unverified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
+ verified: [sft, sft_export_inference, peft]
+ unverified: [pretrain, sft_long_context, checkpoint_resume]
model:
- hf_id: nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
- hf_revision: e86197a3bad449de618a5835f26835ce770c6f10 # pragma: allowlist secret
+ hf_id: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ hf_revision:
architecture: NemotronH_Omni_Reasoning_V3
min_transformers_version: "5.12.1"
verification_environment:
- base_container: nvcr.io/nvidia/nemo:26.08
- bridge_commit: f375ef553a06b782f5129c1bc9a060a55259da4f # pragma: allowlist secret
+ base_container: nvcr.io/nvidia/nemo:26.08.01
+ bridge_commit: 2449c2fcd49741cfe5c112542b901d0c46a90d34 # pragma: allowlist secret
items:
hf_to_megatron_cpu:
@@ -66,21 +55,21 @@ items:
hf_to_megatron_gpu:
status: verified
precision: bf16
+ bridge_commit: 3a5b4f610a5b3608b1bded146418459672492cd9 # pragma: allowlist secret
command: >
./scripts/conversion/convert.sh import --executor slurm --device gpu
- --nodes 1 --gpus-per-node 8
- --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
- --hf-revision e86197a3bad449de618a5835f26835ce770c6f10
+ --nodes 1 --gpus-per-node 4
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
--megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron
- --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1
- --trust-remote-code --low-memory-save
- last_verified: 2026-08-25
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --trust-remote-code
+ last_verified: 2026-09-23
expected_result: >
- The pinned eight-GPU BF16 import exits successfully after mapping 6,281
- parameters per rank and creates a TP1/PP1/EP8/ETP1 iter_0000000
- checkpoint containing the complete language, MTP, image, projector, and
- temporal-video weights. The paired export reloads this saved checkpoint
- successfully.
+ Distributed BF16 GPU import and paired export passed with all 43,079
+ source tensors exact in keys, shapes, dtypes and values. Compared
+ 248,906,833,960 source bytes, including language, MTP, image encoder,
+ projector, temporal embedder and the vision summary-index buffer.
+ This establishes conversion, not training or numerical forward parity.
megatron_to_hf_cpu:
status: unverified
@@ -94,321 +83,288 @@ items:
megatron_to_hf_gpu:
status: verified
precision: bf16
+ bridge_commit: 3a5b4f610a5b3608b1bded146418459672492cd9 # pragma: allowlist secret
command: >
./scripts/conversion/convert.sh export --executor slurm --device gpu
- --nodes 1 --gpus-per-node 8
- --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
- --hf-revision e86197a3bad449de618a5835f26835ce770c6f10
- --megatron-path
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --nodes 1 --gpus-per-node 4
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron
--hf-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-hf-export
- --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1
- --trust-remote-code --distributed-save --save-every-n-ranks 1
- last_verified: 2026-08-25
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --trust-remote-code --distributed-save
+ last_verified: 2026-09-23
expected_result: >
- Strict distributed export exits successfully in 63 indexed shards. All
- 43,078 source tensors match in keys, shapes, dtypes, and values, with
- maximum difference zero. AutoConfig, AutoProcessor, and
- AutoModelForImageTextToText reload the local-only export natively as
- NemotronH_Omni_Reasoning_V3 across all eight visible H100s.
+ Saved round-trip comparison passed for all 43,079 tensors with no
+ missing or extra keys and zero value differences. The exported package
+ reloaded through native Transformers and performed bounded image
+ generation. Exact conversion is separate from forward-logit correlation.
manual_forward_pass:
- status: unverified
+ status: verified
precision: bf16
- command: null
- last_verified: null
+ command: >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --task model-comparison
+ --hf_model_path nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --megatron_model_path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --hf-logits-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/hf-reference.pt
+ --image_path docs/images/mtp_loss.png --prompt "Describe this image."
+ --tp 1 --pp 1 --ep 4 --etp 1 --trust-remote-code
+ last_verified: 2026-09-23
expected_result: >
- Verification requires the public model-comparison launcher to preserve
- Nemotron Omni processor metadata for one rectangular image. Hugging Face
- and Megatron must predict the same next token with cosine similarity of
- at least 0.99, with maximum and mean absolute logit differences reported
- as diagnostics.
+ The maintained comparison passed for this rectangular-image prompt on
+ four GB200 GPUs using a separately saved native HF reference from the
+ same checkpoint and inputs. The next token matches: 1784 (The).
+ Cosine similarity is 0.999980. Maximum and mean absolute logit
+ differences are 1.250000 and 0.216821. The input had 314 text tokens
+ and 1160 vision patches. This establishes single-prompt functional
+ correlation, not strict probability equality or comprehensive visual
+ parity. Stronger image/video diagnostics are not all passing. The
+ reference must be regenerated for any changed checkpoint; the blank
+ public revision does not bind these results to a public release commit.
inference:
- status: unverified
+ status: verified
precision: bf16
- command: null
- last_verified: null
+ command: >
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --task vlm-generation
+ --hf_model_path nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --megatron_model_path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --image_path docs/images/mtp_loss.png --prompt "Describe this image."
+ --tp 1 --pp 1 --ep 4 --etp 1 --max_new_tokens 32
+ --deterministic --trust_remote_code
+ last_verified: 2026-09-23
expected_result: >
- Verification requires synchronous deterministic greedy image and video
- generation through the public VLM launcher, including exact post-resize
- grids and per-tubelet visual-token counts. Record the literal completion
- and actual generated-token count under an explicit maximum bound.
+ Native dynamic-resolution image generation completed on four GB200
+ GPUs with deterministic execution requested. The literal completion was
+ "The user wants a description of the provided image.\n\n1. **Identify the overall structure:** The image contains two side-by-side line charts. They look".
+ Exactly 32 tokens were generated under the 32-token maximum, so the
+ response is truncated, not an EOS-completed answer. This is one image
+ generation sanity check, not HF generation equality, cross-process
+ repeatability, exhaustive video verification, or MTP forward validation.
pretrain:
H100:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 1c459be13a9a229de15a90de8bcad0bc21ad767a on 2026-08-29. # pragma: allowlist secret
enabled_features:
moe_dispatcher: hybridep
command: >
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8
- --recipe nemotron_35_super_vl_pretrain_64gpu_h100_bf16_config
- --mode pretrain --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/vision-pretrained/iter_0000000
- --max_steps 120
- dataset.path=work/data/datacomp-525k-energon
- checkpoint.load=null
+ --recipe nemotron_35_super_vl_pretrain_64gpu_h100_bf16_config --mode pretrain
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --max_steps 200 --save_interval 100
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-pretrain
- --save_interval 60 logger.log_interval=1 logger.log_throughput=true
- optimizer.optimizer_cpu_offload=true
- optimizer.optimizer_offload_fraction=0.25
- optimizer.overlap_cpu_optimizer_d2h_h2d=false
- model.recompute_granularity=full model.recompute_method=uniform
- model.recompute_num_layers=1 model.recompute_modules=null
- logger.log_device_memory_used=true
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ model.tensor_model_parallel_size=2 model.sequence_parallel=true
+ model.cross_entropy_fusion_impl=native
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-pretrain-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-25
metrics:
- # Historical-only initial_loss: 2.999004
- initial_loss: null
- # Historical-only final_loss: 2.104368
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 49743.2
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 130.34
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 1646.858
- last_10_steps_tokens_per_second_per_gpu_avg: null
+ initial_loss: 3.368271
+ final_loss: 1.918807
+ last_10_steps_step_time_ms_avg: 60028.32
+ last_10_steps_model_tflops_per_gpu_avg: 114.33
+ last_10_steps_tokens_per_second_per_gpu_avg: 1364.6892000309188
expected_result: >
- Historical-only observations recorded on 2026-08-29. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- Exactly 64 H100s completed all 120 optimizer steps from the learned
- vision-pretrained checkpoint on DataComp image-caption samples at
- TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. Natural dropless routing and
- full/uniform one-layer language recompute, selective vision recompute,
- and 25% optimizer-state CPU offload plus the recipe's BF16
- precision-aware optimizer produced finite loss, zero skipped or NaN
- iterations, and the resolved ConfigContainer. Full optimizer/RNG
- checkpoints at steps 60 and 120 each contain all 64 distributed model
- shards and 32 Energon data-parallel states.
+ Completed 200 finite optimizer steps on 64 H100s with zero skipped or
+ NaN iterations, full checkpoints at steps 100 and 200, and a fresh
+ final-checkpoint native inference reload. DataComp image-caption
+ training warm-started from converted pretrained weights at sequence
+ length 4096, GBS/MBS 1280/1, TP2/PP2/EP32/ETP1, natural HybridEP
+ routing and FP32 optimizer state. Shared MTP uses two prediction
+ depths and loss scale 0.3. Native finalization and update preflights
+ passed for this workload. Rates are padded token slots per GPU, not
+ actual supervised-token counts; production supervised totals were
+ not recorded. This is bounded support verification, not full convergence.
GB200:
status: unverified
precision: bf16
- # Historical execution: Bridge a3fc1e2e6b970b17cfea247ad1f7b840be67e522 on 2026-08-28. # pragma: allowlist secret
enabled_features:
moe_dispatcher: hybridep
- cuda_graph:
- implementation: transformer_engine
- scopes: [attn, mamba, moe_router, moe_preprocess]
- command: >
- ./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4
- --recipe nemotron_35_super_vl_pretrain_64gpu_gb200_bf16_config
- --mode pretrain --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/vision-pretrained/iter_0000000
- --max_steps 120
- dataset.path=work/data/datacomp-525k-energon
- checkpoint.load=null
- --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-pretrain
- --save_interval 60
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true
- logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-pretrain-config/ConfigContainer.yaml
+ command: null
last_verified: null
metrics:
- # Historical-only initial_loss: 3.062157
initial_loss: null
- # Historical-only final_loss: 2.038754
final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 20385.85
last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 257.33
last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 3214.779
last_10_steps_tokens_per_second_per_gpu_avg: null
expected_result: >
- Historical-only observations recorded on 2026-08-28. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- On exactly 64 GB200s within one NVL72 domain, complete all 120 optimizer
- steps from the learned vision-pretrained checkpoint on DataComp
- image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS 512/1.
- Natural dropless routing, finite loss, zero skipped or NaN iterations,
- no active language or vision recompute, scoped Transformer Engine
- CUDA-graph capture, and the resolved ConfigContainer are required.
- Full optimizer/RNG checkpoints at steps 60 and 120 each contain all 64
- distributed model shards and 32 Energon data-parallel states.
+ Refreshed DataComp pretraining used 64 GB200 GPUs, pretrained weights,
+ natural HybridEP routing, FP32 optimizer state, TP2/PP1/EP64/ETP1,
+ sequence length 8192, GBS/MBS 512/1 and selective moe_act recompute.
+ The original 500-step schedule reached step 475 before its four-hour
+ limit. A fresh-root continuation from step 250 reached step 500 and
+ saved a complete final checkpoint, whose native model reload passed.
+ This is not an uninterrupted 500-step reference. One of 1125 matched
+ loss observations exceeded the unchanged resume tolerance; independent
+ replay remains pending, so no verified pretrain/resume pair is claimed.
sft:
H100:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 2f1c677ee0f948b3842a16ce052190557ed71981 on 2026-08-28. # pragma: allowlist secret
enabled_features:
moe_dispatcher: hybridep
command: >
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8
- --recipe nemotron_35_super_vl_sft_64gpu_h100_bf16_config
- --mode sft
- --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
- --max_steps 100
- dataset.path=work/data/datacomp-525k-energon
- dataset.do_validation=false dataset.do_test=false
- validation.eval_iters=0 validation.eval_interval=0
- scheduler.lr_decay_iters=100
- checkpoint.load=null
+ --recipe nemotron_35_super_vl_sft_64gpu_h100_bf16_config --mode sft
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --max_steps 200 --save_interval 100
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft
- --save_interval 100 checkpoint.save_optim=false checkpoint.save_rng=false
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true logger.tensorboard_dir=null
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ model.tensor_model_parallel_size=2 model.sequence_parallel=true
+ model.recompute_granularity=full model.recompute_method=uniform
+ model.recompute_num_layers=1 model.recompute_modules=[]
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-23
metrics:
- # Historical-only initial_loss: 5.412096
- initial_loss: null
- # Historical-only final_loss: 2.293077
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 30878.25
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 209.98
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 2653.0
- last_10_steps_tokens_per_second_per_gpu_avg: null
+ initial_loss: 3.375419
+ final_loss: 1.958105
+ last_10_steps_step_time_ms_avg: 42907.8
+ last_10_steps_model_tflops_per_gpu_avg: 159.84
+ last_10_steps_tokens_per_second_per_gpu_avg: 1909.2099804697514
expected_result: >
- Historical-only observations recorded on 2026-08-28. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- Exactly 64 H100s completed 100 BF16 SFT optimizer steps from the
- immutable source checkpoint on 128,000 DataComp image-caption samples
- at TP1/PP2/CP1/EP32/ETP1 and GBS/MBS 1280/1. The shifted production
- loss masks contain 2,900,779 supervised tokens. Natural routing,
- HybridEP flex dispatch, fixed expert capacity, selective language
- recompute with vision recompute disabled, and the precision-aware
- optimizer produced finite loss with zero skipped or NaN iterations.
- The model-only step-100 checkpoint contains all 64 distributed shards
- and 32 Energon states and reloaded successfully. This dataset verifies
- image-caption batching; it does not contain video samples.
+ Completed 200 finite optimizer steps on 64 H100 GPUs with zero skipped
+ or NaN iterations and a complete final checkpoint. DataComp image-caption
+ training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP
+ routing, FP32 optimizer state, and two shared MTP prediction depths at
+ loss scale 0.3. The vision encoder is frozen; the language model and
+ projection are trainable. Native finalization/update preflights passed.
+ Final export and native HF reload are checked separately. Token rates
+ are padded slots, not measured supervised-token totals. This verifies
+ image-caption training, not video training or full convergence.
GB200:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 2f1c677ee0f948b3842a16ce052190557ed71981 on 2026-08-29. # pragma: allowlist secret
- enabled_features: {}
+ enabled_features:
+ moe_dispatcher: hybridep
command: >
./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4
- --recipe nemotron_35_super_vl_sft_64gpu_gb200_bf16_config
- --mode sft
- --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
- --max_steps 100
- dataset.path=work/data/datacomp-525k-energon
- dataset.do_validation=false dataset.do_test=false
- validation.eval_iters=0 validation.eval_interval=0
- scheduler.lr_decay_iters=100
- checkpoint.load=null
+ --recipe nemotron_35_super_vl_sft_64gpu_gb200_bf16_config --mode sft
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --max_steps 200 --save_interval 100
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft
- --save_interval 100 checkpoint.save_optim=false checkpoint.save_rng=false
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true logger.tensorboard_dir=null
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-23
metrics:
- # Historical-only initial_loss: 5.393576
- initial_loss: null
- # Historical-only final_loss: 2.303741
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 47924.55
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 135.3
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 1709.354
- last_10_steps_tokens_per_second_per_gpu_avg: null
+ initial_loss: 3.428863
+ final_loss: 1.934663
+ last_10_steps_step_time_ms_avg: 55224.62
+ last_10_steps_model_tflops_per_gpu_avg: 124.21
+ last_10_steps_tokens_per_second_per_gpu_avg: 1483.39635474178
expected_result: >
- Historical-only observations recorded on 2026-08-29. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- Exactly 64 GB200s within one NVL72 domain completed 100 BF16 SFT
- optimizer steps from the immutable source checkpoint on 128,000
- DataComp image-caption samples at TP2/PP1/CP1/EP64/ETP1 and GBS/MBS
- 1280/1. The shifted production loss masks contain 2,900,779 supervised
- tokens. Natural routing, eager all-to-all dispatch, fixed expert
- capacity, and no active language or vision recompute produced finite
- loss with zero skipped or NaN iterations. The model-only step-100
- checkpoint contains all 64 distributed shards and 32 Energon states
- and reloaded successfully. This dataset verifies image-caption batching;
- it does not contain video samples.
+ Completed 200 finite optimizer steps on 64 GB200 GPUs with zero skipped
+ or NaN iterations and a complete final checkpoint. DataComp image-caption
+ training uses sequence length 4096 and GBS/MBS 1280/1, natural HybridEP
+ routing, FP32 optimizer state, and two shared MTP prediction depths at
+ loss scale 0.3. The vision encoder is frozen; the language model and
+ projection are trainable. Native finalization/update preflights passed.
+ Final export and native HF reload are checked separately. Token rates
+ are padded slots, not measured supervised-token totals. This verifies
+ image-caption training, not video training or full convergence.
sft_export_inference:
H100:
- status: unverified
+ status: verified
precision: bf16
+ bridge_commit: fb082ba947e6edfe0318587bdf2fd8679608aeac # pragma: allowlist secret
depends_on: sft
commands:
- >
./scripts/conversion/convert.sh export --executor slurm --device gpu
- --nodes 1 --gpus-per-node 8
- --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
- --hf-revision e86197a3bad449de618a5835f26835ce770c6f10
- --megatron-path
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft/iter_0000100
+ --nodes 1 --gpus-per-node 4
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft/iter_0000200
--hf-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft-hf-export
- --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1
- --trust-remote-code --distributed-save --save-every-n-ranks 1
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --distributed-timeout-minutes 60 --trust-remote-code --distributed-save
- >
- ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 8
- --task hf-inference
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --tasks-per-node 1 --task hf-inference
--hf-model work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft-hf-export
- --prompt "Describe the image." --image docs/images/tp1.png
- --max-new-tokens 32 --chat-template --trust-remote-code
- --device cuda --dtype bfloat16
- last_verified: null
+ --prompt "Describe this image." --image docs/images/mtp_loss.png
+ --max-new-tokens 32 --chat-template --disable-thinking
+ --separate-image-processing --trust-remote-code
+ --device cuda --device-map balanced --dtype bfloat16 --require-gpu-only
+ last_verified: 2026-09-23
expected_result: >
- Strict export must reproduce a natively reloadable Transformers
- checkpoint. Deterministic greedy image generation must then complete
- successfully, with the literal completion and actual generated-token
- count recorded under the 32-token maximum.
+ Exported the final step-200 full-SFT checkpoint, retaining the native
+ model package and reasoning-parser file. Strict Transformers reload
+ checked missing, unexpected and mismatched keys, covered 1453 modules,
+ and passed GPU-only placement on four H100 GPUs. Greedy generation
+ stopped at EOS after exactly 7 tokens under the 32-token maximum.
+ The literal completion was "Losses of the two models". Export used
+ the leaf Bridge commit; subsequent HF inference used
+ e2cf6cbb30f3c93d99e727f8a12c143324308012.
+ This verifies export and one native HF image completion, not
+ HF/Bridge numerical equality. Separate stronger post-SFT visual
+ forward diagnostics have failures and are not reclassified as passing.
GB200:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 215e20e596cb9fedb60fa3f8a7c4e8f618905aab on 2026-08-30. # pragma: allowlist secret
+ bridge_commit: fb082ba947e6edfe0318587bdf2fd8679608aeac # pragma: allowlist secret
depends_on: sft
commands:
- >
./scripts/conversion/convert.sh export --executor slurm --device gpu
--nodes 1 --gpus-per-node 4
- --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
- --hf-revision e86197a3bad449de618a5835f26835ce770c6f10
- --megatron-path
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft/iter_0000100
- --hf-path
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-hf-export
- --torch-dtype bfloat16 --export-weight-dtype bfloat16
- --tp 1 --pp 1 --ep 4 --etp 1 --trust-remote-code
- --distributed-save --save-every-n-ranks 1
+ --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
+ --megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft/iter_0000200
+ --hf-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-hf-export
+ --torch-dtype bfloat16 --tp 1 --pp 2 --ep 2 --etp 1
+ --distributed-timeout-minutes 60 --trust-remote-code --distributed-save
- >
- uv run python
- skills/create-model-verification-card/scripts/verify_hf_inference.py
- --hf-model
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-hf-export
- --prompt "Describe the image." --image docs/images/tp1.png
- --max-new-tokens 32 --chat-template --separate-image-processing
- --trust-remote-code --device cuda --device-map balanced_low_0
- --dtype bfloat16
- last_verified: null
+ ./scripts/inference/infer.sh --nodes 1 --gpus-per-node 4
+ --tasks-per-node 1 --task hf-inference
+ --hf-model work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-hf-export
+ --prompt "Describe this image." --image docs/images/mtp_loss.png
+ --max-new-tokens 32 --chat-template --disable-thinking
+ --separate-image-processing --trust-remote-code
+ --device cuda --device-map balanced --dtype bfloat16 --require-gpu-only
+ last_verified: 2026-09-23
expected_result: >
- Historical-only artifact observations recorded on 2026-08-30. The
- commands and export/reload/generation results do not establish correct
- synchronized, token-normalized SFT updates. This dependent item awaits
- export and inference from a newly validated SFT checkpoint.
- The step-100 full-SFT checkpoint exports as a native BF16 Transformers
- package with 43,079 indexed tensors. The post-weight-derived
- vision_model.summary_idxs tensor is stored as int64 [0, 1] in its own
- deterministic shard. The HF export reloads natively through strict
- Transformers loading with no missing or unexpected keys, the
- dynamic-resolution image processor succeeds, and greedy generation
- emits exactly 32 new tokens under a 32-token maximum.
- The literal completion is "The user wants a description of the provided
- image.\n\n1. **Identify the main subject:** The image is a diagram
- illustrating a \"Pre-Layern".
+ Exported the final step-200 full-SFT checkpoint, retaining the native
+ model package and reasoning-parser file. Strict Transformers reload
+ checked missing, unexpected and mismatched keys, covered 1453 modules,
+ and passed GPU-only placement on four GB200 GPUs. Greedy generation
+ stopped at EOS after exactly 7 tokens under the 32-token maximum.
+ The literal completion was "Losses of the two models". Export used
+ the leaf Bridge commit; subsequent HF inference used
+ e2cf6cbb30f3c93d99e727f8a12c143324308012.
+ This verifies export and one native HF image completion, not
+ HF/Bridge numerical equality. Separate stronger post-SFT visual
+ forward diagnostics have failures and are not reclassified as passing.
sft_long_context:
H100:
@@ -443,208 +399,159 @@ items:
peft:
H100:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge b1d2932f32839aa5244edbb342fefc4a88714068 on 2026-08-29. # pragma: allowlist secret
+ bridge_commit: fb082ba947e6edfe0318587bdf2fd8679608aeac # pragma: allowlist secret
enabled_features: {}
command: >
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8
- --recipe nemotron_35_super_vl_peft_16gpu_h100_bf16_config
- --mode lora
- --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
- --max_steps 100
- dataset.path=work/data/datacomp-525k-energon
- dataset.do_validation=false dataset.do_test=false
- validation.eval_iters=0 validation.eval_interval=0
- checkpoint.load=null
+ --recipe nemotron_35_super_vl_peft_16gpu_h100_bf16_config --mode lora
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --max_steps 500 --save_interval 250
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-peft
- --save_interval 100 checkpoint.save_optim=false checkpoint.save_rng=false
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true logger.tensorboard_dir=null
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=500
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-peft-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-24
metrics:
- # Historical-only initial_loss: 3.857522
- initial_loss: null
- # Historical-only final_loss: 2.157877
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 5427.27
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 59.74
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 754.707
- last_10_steps_tokens_per_second_per_gpu_avg: null
+ initial_loss: 2.483866
+ final_loss: 2.596869
+ last_10_steps_step_time_ms_avg: 5634.59
+ last_10_steps_model_tflops_per_gpu_avg: 60.86
+ last_10_steps_tokens_per_second_per_gpu_avg: 726.9384285280739
expected_result: >
- Historical-only observations recorded on 2026-08-29. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- Exactly 16 H100s completed the rank-32, alpha-32, zero-dropout LoRA
- objective for 100 optimizer steps from the immutable source checkpoint
- on 1,600 DataComp image-caption samples at TP4/PP2/CP1/EP8/ETP1 and
- GBS/MBS 16/1. The shifted production loss masks contain 35,540
- supervised tokens. Frozen base weights, natural routing, dropless
- all-to-all dispatch, and no active language or vision recompute
- produced finite loss with zero skipped or NaN iterations. The
- step-100 adapter contains all 16 distributed shards and two Energon
- states and reloaded successfully. This dataset does not contain video
- samples. This historical recipe used unqualified adapter targets that
- also matched vision and projector modules; it does not verify the
- current language-only adapter placement.
+ Completed 500 finite LoRA optimizer steps on 16 H100 GPUs with zero
+ skipped or NaN iterations and a complete final native adapter checkpoint.
+ DataComp image-caption training uses sequence length 4096, GBS/MBS
+ 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets.
+ Native save/reload and merge preserve updated router expert-bias buffers
+ as well as adapter tensors. Full merged HF reload and bounded image
+ generation passed; stronger post-merge numerical diagnostics are not
+ all passing. Standalone HF adapter export does not preserve these
+ trained router buffers; use native reload and full-model merge/export.
+ Token rates are padded slots, not measured supervised-token totals.
GB200:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 2f1c677ee0f948b3842a16ce052190557ed71981 on 2026-08-28. # pragma: allowlist secret
+ bridge_commit: fb082ba947e6edfe0318587bdf2fd8679608aeac # pragma: allowlist secret
enabled_features: {}
command: >
./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4
- --recipe nemotron_35_super_vl_peft_16gpu_gb200_bf16_config
- --mode lora
- --pretrained_checkpoint
- work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
- --max_steps 100
- dataset.path=work/data/datacomp-525k-energon
- dataset.do_validation=false dataset.do_test=false
- validation.eval_iters=0 validation.eval_interval=0
- checkpoint.load=null
+ --recipe nemotron_35_super_vl_peft_16gpu_gb200_bf16_config --mode lora
+ --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000
+ --max_steps 500 --save_interval 250
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-peft
- --save_interval 100 checkpoint.save_optim=false checkpoint.save_rng=false
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true logger.tensorboard_dir=null
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=500
+ checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-peft-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-24
metrics:
- # Historical-only initial_loss: 5.485836
- initial_loss: null
- # Historical-only final_loss: 2.260179
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 3472.77
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 93.55
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 1179.462
- last_10_steps_tokens_per_second_per_gpu_avg: null
+ initial_loss: 3.45039
+ final_loss: 2.059837
+ last_10_steps_step_time_ms_avg: 3640.69
+ last_10_steps_model_tflops_per_gpu_avg: 94.23
+ last_10_steps_tokens_per_second_per_gpu_avg: 1125.0614581301895
expected_result: >
- Historical-only observations recorded on 2026-08-28. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- Exactly 16 GB200s completed the rank-32, alpha-32, zero-dropout LoRA
- objective for 100 optimizer steps from the immutable source checkpoint
- on 1,600 DataComp image-caption samples at TP2/PP1/CP1/EP16/ETP1 and
- GBS/MBS 16/1. The shifted production loss masks contain 34,537
- supervised tokens. Frozen base weights, natural routing, dropless
- all-to-all dispatch, and no active language or vision recompute
- produced finite loss with zero skipped or NaN iterations. The
- step-100 adapter contains all 16 distributed shards and eight Energon
- states and reloaded successfully. This dataset does not contain video
- samples. This historical recipe used unqualified adapter targets that
- also matched vision and projector modules; it does not verify the
- current language-only adapter placement.
+ Completed 500 finite LoRA optimizer steps on 16 GB200 GPUs with zero
+ skipped or NaN iterations and a complete final native adapter checkpoint.
+ DataComp image-caption training uses sequence length 4096, GBS/MBS
+ 16/1, rank 32, alpha 32, zero adapter dropout and language-only targets.
+ Native save/reload and merge preserve updated router expert-bias buffers
+ as well as adapter tensors. Full merged HF reload and bounded image
+ generation passed; stronger post-merge numerical diagnostics are not
+ all passing. Standalone HF adapter export does not preserve these
+ trained router buffers; use native reload and full-model merge/export.
+ Token rates are padded slots, not measured supervised-token totals.
checkpoint_resume:
H100:
- status: unverified
+ status: verified
precision: bf16
- # Historical execution: Bridge 1c459be13a9a229de15a90de8bcad0bc21ad767a on 2026-08-29. # pragma: allowlist secret
depends_on: pretrain
command: >
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8
- --recipe nemotron_35_super_vl_pretrain_64gpu_h100_bf16_config
- --mode pretrain --max_steps 120
- dataset.path=work/data/datacomp-525k-energon
+ --recipe nemotron_35_super_vl_pretrain_64gpu_h100_bf16_config --mode pretrain
--load_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-pretrain
- checkpoint.ckpt_step=60
+ --max_steps 200 --save_interval 100
--save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-resume
- --save_interval 120
- optimizer.optimizer_cpu_offload=true
- optimizer.optimizer_offload_fraction=0.25
- optimizer.overlap_cpu_optimizer_d2h_h2d=false
- model.recompute_granularity=full model.recompute_method=uniform
- model.recompute_num_layers=1 model.recompute_modules=null
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true
+ dataset.path=work/data/datacomp-525k-energon
+ dataset.do_validation=false dataset.num_workers=2
+ model.tensor_model_parallel_size=2 model.sequence_parallel=true
+ model.cross_entropy_fusion_impl=native
+ validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200
+ checkpoint.ckpt_step=100
+ checkpoint.finetune=false checkpoint.load_optim=true checkpoint.load_rng=true
+ checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true
+ logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true
+ logger.tensorboard_dir=null
logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-resume-config/ConfigContainer.yaml
- last_verified: null
+ ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true
+ rerun_state_machine.check_for_nan_in_loss=true
+ last_verified: 2026-09-25
metrics:
- # Historical-only initial_loss: 2.064014
- initial_loss: null
- # Historical-only final_loss: 2.117677
- final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 50238.95
- last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 129.05
- last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 1630.607
- last_10_steps_tokens_per_second_per_gpu_avg: null
- # Historical comparison only; not proof of correct training updates.
+ initial_loss: 1.951292
+ final_loss: 1.919275
+ last_10_steps_step_time_ms_avg: 51787.35
+ last_10_steps_model_tflops_per_gpu_avg: 132.45
+ last_10_steps_tokens_per_second_per_gpu_avg: 1581.8534835244516
resume_comparison:
reference_item: pretrain
- sentinel_steps: [61, 120]
+ sentinel_steps: [101, 200]
loss_relative_tolerance: 1.0e-2
loss_absolute_tolerance: 1.0e-6
sentinels_match: true
expected_result: >
- Historical-only observations recorded on 2026-08-29. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- The distinct resume root loaded pretrain.H100 step 60 directly,
- restored optimizer, RNG, and Energon state, and completed exactly steps
- 61--120 on the same 64-H100 topology and execution fingerprint. Every
- resumed LM and MTP loss matched the uninterrupted reference within 1%
- relative plus 1e-6 absolute; the maximum absolute deltas across all 60
- resumed steps were 0.013309 and 0.017865. LM deltas at sentinels 61 and
- 120 were 0 and 0.013309. The final full-state checkpoint contains 64
- model shards and 32 Energon states.
+ Loaded pretrain.H100 step 100 directly into a fresh output directory,
+ restored optimizer, RNG and Energon state, and completed steps 101-200.
+ All 500 logged loss comparisons across the 100 resumed steps passed
+ the unchanged tolerance. The final checkpoint contains 64 distributed
+ shards and 16 Energon states; a fresh native model reload and one-token
+ image-conditioned forward passed separately. Model, data, schedule and
+ parallel topology match the reference. Compilation was serialized for
+ execution stability without changing training arithmetic. Rates are
+ padded token slots; actual supervised-token totals were not recorded.
GB200:
status: unverified
precision: bf16
- # Historical execution: Bridge a3fc1e2e6b970b17cfea247ad1f7b840be67e522 on 2026-08-28. # pragma: allowlist secret
depends_on: pretrain
- command: >
- ./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4
- --recipe nemotron_35_super_vl_pretrain_64gpu_gb200_bf16_config
- --mode pretrain --max_steps 120
- dataset.path=work/data/datacomp-525k-energon
- --load_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-pretrain
- checkpoint.ckpt_step=60
- --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-resume
- --save_interval 120
- logger.log_interval=1 logger.log_throughput=true
- logger.log_device_memory_used=true
- logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-resume-config/ConfigContainer.yaml
+ command: null
last_verified: null
metrics:
- # Historical-only initial_loss: 2.204572
initial_loss: null
- # Historical-only final_loss: 2.040755
final_loss: null
- # Historical-only last_10_steps_step_time_ms_avg: 19945.03
last_10_steps_step_time_ms_avg: null
- # Historical-only last_10_steps_model_tflops_per_gpu_avg: 263.04
last_10_steps_model_tflops_per_gpu_avg: null
- # Historical-only last_10_steps_tokens_per_second_per_gpu_avg: 3285.831
last_10_steps_tokens_per_second_per_gpu_avg: null
- # Historical comparison only; not proof of correct training updates.
resume_comparison:
reference_item: pretrain
- sentinel_steps: [61, 120]
+ sentinel_steps: [251, 430, 475]
loss_relative_tolerance: 1.0e-2
loss_absolute_tolerance: 1.0e-6
- sentinels_match: true
+ sentinels_match: false
expected_result: >
- Historical-only observations recorded on 2026-08-28. The command,
- metrics and artifact observations do not establish synchronized,
- token-normalized training updates. Recipe-specific reverification
- after the shared-provider callback fix is pending.
- The distinct resume root loaded pretrain.GB200 step 60 directly,
- restored optimizer, RNG, and Energon state, and completed exactly steps
- 61--120 on the same 64-GB200 one-domain topology. Every resumed LM and
- MTP loss matched the uninterrupted reference within 1% relative plus
- 1e-6 absolute; LM deltas at sentinels 61 and 120 were 0.000077 and
- 0.002001. The final full-state checkpoint contains 64 model shards and
- 32 Energon states.
+ The fresh-root step-250 continuation restored optimizer, RNG and
+ Energon state and completed steps 251-500, retaining the original
+ 500-step schedule. The final checkpoint contains 64 distributed model
+ shards and 32 Energon states and passed native model-weight reload.
+ Comparison against the available original steps 251-475 passed
+ 1124 of 1125 logged-loss checks, but step-430 LM loss differed by
+ 2.55%, exceeding the fixed 1% plus 1e-6 tolerance. The original run
+ has no reference rows for steps 476-500. The comparison remains
+ failed, with independent diagnostic replay pending; successful
+ checkpoint loading does not establish numerical resume equivalence.
diff --git a/examples/models/nemotron/nemotron_3/super_vl/text-only.md b/examples/models/nemotron/nemotron_3/super_vl/text-only.md
index bb8db998de..903284ad1e 100644
--- a/examples/models/nemotron/nemotron_3/super_vl/text-only.md
+++ b/examples/models/nemotron/nemotron_3/super_vl/text-only.md
@@ -1,4 +1,4 @@
-# Nemotron 3.5 Super VL as a text-only Nemotron-H checkpoint
+# Nemotron-3.5 Super Text-only
Use `text_only=True` to import the language model and MTP weights directly from
the VL checkpoint. This selects the existing Nemotron-H bridge and plain hybrid
@@ -15,7 +15,7 @@ performs conversion; there is no separate text-only model or pretrained subclass
from megatron.bridge import AutoBridge
bridge = AutoBridge.from_hf_pretrained(
- "nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B",
+ "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
text_only=True,
trust_remote_code=True,
revision="",
@@ -36,7 +36,7 @@ The shared CLI accepts `--text-only` for import and export. For example, in a
GPU-enabled environment with sufficient aggregate memory:
```bash
-HF_MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B
+HF_MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
HF_REVISION=''
bash scripts/conversion/convert.sh import \
@@ -60,11 +60,15 @@ are retained; keeping media tokens in the vocabulary does not retain the media
encoder. The lazy source wrapper itself is for conversion, not HF generation;
use the Megatron model or the standalone HF export for inference.
-Super VL stores one shared attention+MoE MTP block. The text config expresses
-two runtime prediction depths, matching the existing Super training convention,
-with `mtp_use_repeated_layer=True`. This does not duplicate the serialized
-weights. Native Transformers inference may ignore the MTP weights; a successful
-HF generation test alone therefore does not validate MTP training.
+Super VL stores one shared attention+MoE MTP block. Text-only import and HF
+export preserve `num_nextn_predict_layers=1`. The Super training recipes
+explicitly set `mtp_num_layers=2` and `mtp_use_repeated_layer=True`, applying
+that same block at two prediction depths without duplicating its weights.
+When building a training provider directly through `AutoBridge` instead of a
+Super recipe, set these two training options explicitly. A bare HF import
+does not infer the repetition count from the block's attention and MoE layers.
+Native Transformers inference may ignore the MTP weights; a successful HF
+generation test alone therefore does not validate MTP training.
## Reuse the Super recipes
@@ -75,7 +79,7 @@ H100/GB200 variants and legacy aliases) accept the same source options:
from megatron.bridge.recipes.nemotronh import nemotron_3_super_sft_config
cfg = nemotron_3_super_sft_config(
- hf_path="nvidia/NVIDIA-Nemotron-3.5-Super-120B-A12B",
+ hf_path="nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
text_only=True,
revision="",
trust_remote_code=True,
diff --git a/scripts/docs/generate_model_verification_catalog.py b/scripts/docs/generate_model_verification_catalog.py
index 7a1db2bfcd..258a18946c 100644
--- a/scripts/docs/generate_model_verification_catalog.py
+++ b/scripts/docs/generate_model_verification_catalog.py
@@ -108,6 +108,7 @@
"nemotron-3-ultra-550b-a55b": "models/nemotron/nemotron3-ultra.md",
"nemotron-3.5-lightning": "models/nemotron/nemotron3.5-lightning.md",
"nemotron-3.5-super-vl-120b-a12b": "models/nemotron/nemotron3.5-super-vl.md",
+ "nemotron-3.5-super-vl-120b-a12b-text-only": "models/nemotron/nemotron3.5-super-vl-text-only.md",
"qwen3-235b-a22b": "models/qwen/qwen3-235b-a22b.md",
"qwen3-30b-a3b": "models/qwen/qwen3-30b-a3b.md",
"qwen3-8b": "models/qwen/qwen3-8b.md",
@@ -122,6 +123,7 @@
"gpt-oss-20b": "GPT OSS 20B",
"nemotron-3-nano-4b": "Nemotron 3 Nano 4B",
"nemotron-3.5-lightning": "Nemotron 3.5 Lightning",
+ "nemotron-3.5-super-vl-120b-a12b-text-only": "Nemotron-3.5 Super Text-only",
"qwen3-235b-a22b": "Qwen3-235B-A22B",
"qwen3-30b-a3b": "Qwen3-30B-A3B",
"qwen3-8b": "Qwen3-8B",
diff --git a/scripts/inference/infer.sh b/scripts/inference/infer.sh
index 3e6dccdeec..876becafe9 100755
--- a/scripts/inference/infer.sh
+++ b/scripts/inference/infer.sh
@@ -21,6 +21,6 @@ cd "${REPO_ROOT}"
UV_ARGS=(--no-project --with nemo-run==0.10.0)
if [[ -n "${VIRTUAL_ENV:-}" ]]; then
- UV_ARGS=(--active --no-sync --with nemo-run==0.10.0)
+ UV_ARGS=(--active --no-sync)
fi
exec uv run "${UV_ARGS[@]}" python "${SCRIPT_DIR}/setup_inference.py" "$@"
diff --git a/scripts/inference/vlm_generation.py b/scripts/inference/vlm_generation.py
index 23b45c4c96..cd6cd3d467 100644
--- a/scripts/inference/vlm_generation.py
+++ b/scripts/inference/vlm_generation.py
@@ -293,6 +293,15 @@ def main(args) -> None:
"expert_tensor_parallel_size": etp,
"pipeline_dtype": torch.bfloat16,
"deterministic_mode": args.deterministic,
+ # Preserve the saved dispatcher, but never drop tokens or force routing in inference.
+ "moe_expert_capacity_factor": None,
+ "moe_expert_rank_capacity_factor": None,
+ "moe_paged_stash": False,
+ "moe_ncclep_zero_copy": False,
+ "moe_pad_expert_input_to_capacity": False,
+ "moe_router_force_load_balancing": False,
+ "moe_router_force_biased": None,
+ "moe_hybridep_pad_uneven_dispatch_inputs": True,
}
if args.pp_layout:
mp_overrides["pipeline_model_parallel_layout"] = args.pp_layout
diff --git a/skills/create-model-verification-card/SKILL.md b/skills/create-model-verification-card/SKILL.md
index 0d182e8b04..813cc257e8 100644
--- a/skills/create-model-verification-card/SKILL.md
+++ b/skills/create-model-verification-card/SKILL.md
@@ -367,6 +367,13 @@ Never record or reproduce:
- tokens, token-loading commands, private URLs, or private registry references;
- environment-specific launcher overlays.
+Literal chat delimiters (`<|im_start|>`, `<|im_end|>`, ``, and
+``) are prompt or output data, not placeholders or storage paths.
+Loopback IP literals (IPv4 `127.0.0.0/8` and IPv6 `::1`) may identify a
+single-node inference coordinator; they do not identify private infrastructure.
+These narrow exceptions do not exempt text enclosed by chat delimiters, other
+IP addresses, URLs, runtime orchestration flags, or caller-supplied deny terms.
+
Keep private run notes outside the tracked repository and public PR text. If a
private codename cannot be recognized generically, pass it to the validator via
`--deny-term` or an untracked file through `--denylist "$PRIVATE_DENYLIST"`.
diff --git a/skills/create-model-verification-card/scripts/validate_card.py b/skills/create-model-verification-card/scripts/validate_card.py
index 18d32b07bb..3611bff120 100644
--- a/skills/create-model-verification-card/scripts/validate_card.py
+++ b/skills/create-model-verification-card/scripts/validate_card.py
@@ -181,6 +181,7 @@
)
PLACEHOLDER_RE = re.compile(r"\b(?:TODO|TBD|PLACEHOLDER)\b|<[^>]+>", re.IGNORECASE)
+CHAT_TOKEN_RE = re.compile(r"<\|im_start\|>|<\|im_end\|>||")
HF_ID_RE = re.compile(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+")
REVISION_RE = re.compile(r"[0-9a-f]{40}")
VERSION_RE = re.compile(r"\d+\.\d+\.\d+")
@@ -585,13 +586,30 @@ def _is_finite_number(value: object) -> TypeGuard[int | float]:
return isinstance(value, int | float) and not isinstance(value, bool) and math.isfinite(float(value))
-def _contains_ipv6(value: str) -> bool:
+def _mask_chat_tokens(value: str) -> str:
+ # Mask only exact delimiters, not their contents, and preserve diagnostic offsets.
+ return CHAT_TOKEN_RE.sub(lambda match: " " * len(match.group(0)), value)
+
+
+def _contains_placeholder(value: str) -> bool:
+ return PLACEHOLDER_RE.search(_mask_chat_tokens(value)) is not None
+
+
+def _contains_non_loopback_ip(value: str) -> bool:
+ for match in IPV4_RE.finditer(value):
+ try:
+ address = ipaddress.ip_address(match.group(0))
+ except ValueError:
+ # Preserve the existing conservative rejection of IPv4-like text.
+ return True
+ if not address.is_loopback:
+ return True
for match in IPV6_CANDIDATE_RE.finditer(value):
try:
address = ipaddress.ip_address(match.group(0))
except ValueError:
continue
- if address.version == 6:
+ if address.version == 6 and address != ipaddress.IPv6Address("::1"):
return True
return False
@@ -1584,11 +1602,11 @@ def _validate_item(
if len(complete_commands) != expected_count:
errors.append(f"{_pointer(*path, command_field)}: verified item requires {expected_count} command(s)")
for entry_path, entry in command_entries:
- if PLACEHOLDER_RE.search(entry):
+ if _contains_placeholder(entry):
errors.append(f"{_pointer(*entry_path)}: verified command contains a placeholder")
if not isinstance(expected, str) or not expected.strip():
errors.append(f"{_pointer(*path, 'expected_result')}: verified items require a concrete result")
- elif PLACEHOLDER_RE.search(expected):
+ elif _contains_placeholder(expected):
errors.append(f"{_pointer(*path, 'expected_result')}: verified result contains a placeholder")
if not _is_iso_date(item.get("last_verified")):
errors.append(f"{_pointer(*path, 'last_verified')}: verified items require an ISO date")
@@ -1597,7 +1615,7 @@ def _validate_item(
errors.append(f"{_pointer(*path, command_field)}: must be null for status {status}")
if item.get("last_verified") is not None:
errors.append(f"{_pointer(*path, 'last_verified')}: must be null for status {status}")
- if not isinstance(expected, str) or not expected.strip() or PLACEHOLDER_RE.search(expected):
+ if not isinstance(expected, str) or not expected.strip() or _contains_placeholder(expected):
errors.append(f"{_pointer(*path, 'expected_result')}: explain the public limitation")
if status == "verified" and item_name in CONVERSION_ITEMS:
@@ -1763,7 +1781,7 @@ def _validate_weak_scaling_group(
expected_result = group.get("expected_result")
if not isinstance(expected_result, str) or not expected_result.strip():
errors.append(f"{_pointer(*path, 'expected_result')}: verified items require a concrete result")
- elif PLACEHOLDER_RE.search(expected_result):
+ elif _contains_placeholder(expected_result):
errors.append(f"{_pointer(*path, 'expected_result')}: verified result contains a placeholder")
points = group.get("points")
@@ -1806,7 +1824,7 @@ def _validate_weak_scaling_group(
if not isinstance(command, str) or not command.strip():
errors.append(f"{_pointer(*point_path, 'command')}: expected a non-empty command string")
command = None
- elif PLACEHOLDER_RE.search(command):
+ elif _contains_placeholder(command):
errors.append(f"{_pointer(*point_path, 'command')}: verified command contains a placeholder")
sequence_length: int | None = None
@@ -1933,7 +1951,7 @@ def _validate_privacy(raw: str, card: Mapping[str, Any], deny_terms: tuple[str,
errors.append("/: execution-environment names do not belong in the card")
if URL_RE.search(privacy_raw):
errors.append("/: URLs are forbidden; use a public model name or repository-relative path")
- if EMAIL_RE.search(privacy_raw) or IPV4_RE.search(privacy_raw) or _contains_ipv6(privacy_raw):
+ if EMAIL_RE.search(privacy_raw) or _contains_non_loopback_ip(privacy_raw):
errors.append("/: email or IP address detected")
if REMOTE_COMMAND_RE.search(privacy_raw) or REMOTE_COPY_RE.search(privacy_raw):
errors.append("/: remote host commands are forbidden")
@@ -1972,7 +1990,7 @@ def _validate_privacy(raw: str, card: Mapping[str, Any], deny_terms: tuple[str,
if "../" in privacy_raw:
errors.append("/: parent-directory traversal is forbidden in cards")
- raw_without_urls = URL_RE.sub("", privacy_raw)
+ raw_without_urls = _mask_chat_tokens(URL_RE.sub("", privacy_raw))
for match in ABSOLUTE_PATH_RE.finditer(raw_without_urls):
errors.append(f"/: absolute path detected at character {match.start()}")
diff --git a/src/megatron/bridge/inference/text_generation.py b/src/megatron/bridge/inference/text_generation.py
index e4e20f6956..44ceb66113 100644
--- a/src/megatron/bridge/inference/text_generation.py
+++ b/src/megatron/bridge/inference/text_generation.py
@@ -146,6 +146,16 @@ def _apply_provider_parallelism(
setattr(provider, "pipeline_dtype", dtype)
setattr(provider, "bf16", dtype == torch.bfloat16)
setattr(provider, "fp16", dtype == torch.float16)
+ # Training capacity and forced routing must not change inference predictions.
+ setattr(provider, "moe_expert_capacity_factor", None)
+ setattr(provider, "moe_expert_rank_capacity_factor", None)
+ # Backward-only paged stashing also requires a fixed rank capacity.
+ setattr(provider, "moe_paged_stash", False)
+ # NCCL-EP zero-copy requires fixed capacity and cannot serve dropless eager inference.
+ setattr(provider, "moe_ncclep_zero_copy", False)
+ setattr(provider, "moe_pad_expert_input_to_capacity", False)
+ setattr(provider, "moe_router_force_load_balancing", False)
+ setattr(provider, "moe_router_force_biased", None)
if attention_backend is not None:
setattr(provider, "attention_backend", AttnBackend[attention_backend])
is_mla_model = bool(getattr(provider, "multi_latent_attention", False))
@@ -185,6 +195,13 @@ def _megatron_checkpoint_overrides(
"pipeline_dtype": dtype,
"bf16": dtype == torch.bfloat16,
"fp16": dtype == torch.float16,
+ "moe_expert_capacity_factor": None,
+ "moe_expert_rank_capacity_factor": None,
+ "moe_paged_stash": False,
+ "moe_ncclep_zero_copy": False,
+ "moe_pad_expert_input_to_capacity": False,
+ "moe_router_force_load_balancing": False,
+ "moe_router_force_biased": None,
}
if attention_backend is not None:
overrides["attention_backend"] = AttnBackend[attention_backend]
diff --git a/src/megatron/bridge/models/model_provider.py b/src/megatron/bridge/models/model_provider.py
index 0c3172631d..fddd27c2e1 100644
--- a/src/megatron/bridge/models/model_provider.py
+++ b/src/megatron/bridge/models/model_provider.py
@@ -548,7 +548,7 @@ class GetModelKwargs(TypedDict, total=False):
class ModelParallelKwargs(TypedDict, total=False):
- """Model-parallel override kwargs.
+ """Model-parallel and checkpoint inference override kwargs.
Attributes map to `TransformerConfig`/provider fields that control parallelism.
Only provided values are applied as overrides.
@@ -566,6 +566,14 @@ class ModelParallelKwargs(TypedDict, total=False):
hierarchical_context_parallel_sizes: list[int] | None
pipeline_model_parallel_layout: list[list[str]] | None
pipeline_dtype: torch.dtype
+ moe_expert_capacity_factor: float | None
+ moe_expert_rank_capacity_factor: float | None
+ moe_paged_stash: bool
+ moe_ncclep_zero_copy: bool
+ moe_pad_expert_input_to_capacity: bool
+ moe_router_force_load_balancing: bool
+ moe_router_force_biased: float | None
+ moe_hybridep_pad_uneven_dispatch_inputs: bool
def get_model(
diff --git a/src/megatron/bridge/models/nemotron_omni/data/collate_fn.py b/src/megatron/bridge/models/nemotron_omni/data/collate_fn.py
index 5382e0253c..fe032f84cf 100644
--- a/src/megatron/bridge/models/nemotron_omni/data/collate_fn.py
+++ b/src/megatron/bridge/models/nemotron_omni/data/collate_fn.py
@@ -245,6 +245,18 @@ def _pixel_shuffled_token_count(*, height: int, width: int, patch_dim: int) -> i
return (patch_rows // PIXEL_SHUFFLE_FACTOR) * (patch_cols // PIXEL_SHUFFLE_FACTOR)
+def _join_content_parts(parts: Sequence[str]) -> str:
+ """Separate structured parts without duplicating source-owned newlines."""
+ rendered: list[str] = []
+ for index, part in enumerate(parts):
+ # Energon preserves the newline after a literal /