docs(model): re-verify DeepSeek V4 Flash GB200 performance at the current recipe

[pruprakash]

What does this PR do ?

Re-verifies the DeepSeek V4 Flash pretrain_performance.GB200 card leaf at the current perf recipe and records the measured layout and metrics. Follows @cuichenx's direction on #6463 (closed): perf recipes stay as tuned; the verification card is updated instead.

Summary

The GB200 leaf recorded EP64 / GBS/MBS 2048/1 from its 2026-08-06 run at 3ecb452af. #5845 later retuned deepseek_v4_flash_pretrain_128gpu_gb200_fp8mx_config to EP32 with 32 HybridEP ranks, so the card no longer described what its published command builds. The published command was re-run unchanged at b53e66c47 and the leaf now records that run.

Changelog

• examples/model_verification_cards/deepseek-v4-flash/card.yaml (pretrain_performance.GB200): bridge_commit b53e66c, last_verified 2026-10-14, the five metrics from the run, expected_result layout PP1/EP32/CP1/TP1/DP4, GBS/MBS 2048/1.
• docs/_static/model-verification/catalog-v1.json, docs/models/deepseek/deepseek-v4.md, docs/fern/.../deepseek-v4.mdx: regenerated.
• tests/unit_tests/recipes/test_verification_card_performance_layouts.py: checks that the EP and GBS/MBS a verified performance leaf records match what its published recipe builds.

Verification

• Published card command, unchanged, on 32 x 4 GB200, nvcr.io/nvidia/nemo:26.08 with Bridge b53e66c47 (MCore 1e3c0c62): 50/50 steps, 0 skipped, 0 NaN. Resolved config: TP1/PP1/CP1/EP32/ETP1, GBS/MBS 2048/1, seq 4096, MXFP8, HybridEP (32 ranks per NVLink domain), full-iteration CUDA graph, forced routing, expert-rank capacity 1.5, paged stash. Loss 12.59972 -> 4.25929; last 10 steps 8146.440 ms, 731.830 TFLOP/s/GPU, 8061.248 tokens/s/GPU.
• New test, run in the same container: main's card fails assert (64, 2048, 1) == (32, 2048, 1); this branch passes (2 passed).
• validate_card.py valid; catalog --check clean; pre-commit passed.

Additional Information

• pretrain_performance.GB300 (also drifted: card EP64/MBS1, recipe EP32/MBS2) and Nemotron 3 Super pretrain_performance.GB300 (card EP64, recipe EP16 with the NCCL EP dispatcher) are not re-run yet; they keep their recorded evidence at their pinned commits and will follow in this PR or a follow-up. The new test covers only the re-verified GB200 leaf.

• Reviewer: @cuichenx

• NVBug: 6875251.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pruprakash]

/review

[pruprakash]

/ok to test 2f92295

[github-actions]

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-6467-1877c24ee602.docs.buildwithfern.com/nemo/megatron-bridge

[svcnvidia-nemo-ci]

Completed the light review of all five changed files and their directly affected recipe, test, and documentation paths. Two blocking correctness issues remain: the token-throughput metric contradicts its defined calculation, and the new test's offline setup depends on recipe import order.

Static source review only; no tests or training workloads were executed.

Suggested test cases

• test_recorded_layout_parses_the_card_wording — retain the (32, 2048, 1) assertion in the new unit tests.
• test_verified_performance_layout_matches_the_published_recipe[deepseek-v4-flash-GB200-megatron.bridge.perf_recipes.deepseek] — run independently in a fresh process; expect offline construction and a passing layout comparison.
• Proposed test_gb200_token_throughput_matches_recorded_step_time — expect approximately 8044.741 tokens/s/GPU from the recorded workload and mean step time.
• test_generated_outputs_and_navigation_are_current — retain the clean generation-check assertion in the catalog consistency tests.
• No perf tests impacted.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

[svcnvidia-nemo-ci]

[P2] Import recipe modules before installing the offline patches

Failure: In a fresh targeted pytest process, this file has not imported its recipe module when this fixture calls the helper. The helper only patches AutoBridge on recipe modules already present in sys.modules; its subsequent Flux import loads library recipes after that scan. Consequently, the GB200 factory's AutoBridge.from_hf_pretrained call can retain the real bridge. It receives the helper's mocked AutoConfig result, which lacks supported architectures, and configuration validation rejects it before the layout comparison. Full-suite collection can hide this because existing recipe tests import factories before fixture setup.

Correction: In this fixture, preload the selected modules before invoking the existing helper:

    for _, _, module in PERFORMANCE_LEAVES:
        importlib.import_module(module)
    patch_recipe_construction_dependencies(monkeypatch)

This ensures the factory and its inherited library recipes exist when their dependencies are patched. It is a suggested, unexecuted replacement for the fixture body; verify it before adopting. No shared-helper change is needed.

Preservation: Keep actual recipe construction and the card-derived recipe name, rather than mocking the layout tuple. Preserve the parser's (32, 2048, 1) case, detection of stale EP64 evidence, and compatibility with collection alongside the existing recipe tests.

Acceptance: Run test_verification_card_performance_layouts.py alone in a fresh process without an HF model cache: both tests should pass without network access or architecture-validation errors. Run it alongside test_deepseek_v4_flash_perf_recipes.py: both sets should still pass. As a proposed negative regression check, temporarily supply card wording with EP64 while retaining the current factory; the comparison must fail with (64, 2048, 1) != (32, 2048, 1), proving the offline fix has not bypassed the intended guard.

[pruprakash]

/review

[pruprakash]

/ok to test 1877c24

[svcnvidia-nemo-ci]

Completed the light follow-up against the previously reviewed head. The GB200 throughput correction now satisfies 4096 * 2048 / 8.14644 / 128 ≈ 8044.741, and the catalog and both generated pages agree. That prior finding is resolved.

The prior offline-fixture finding remains unresolved: selecting the layout test alone in a fresh process still leaves its inherited recipe's AutoBridge unpatched. One blocking finding is retained below; no new eligible regressions were identified in the follow-up delta.

Static source review only; no tests or training workloads were executed.

Suggested test cases

• test_verified_performance_layout_matches_the_published_recipe[deepseek-v4-flash-GB200-megatron.bridge.perf_recipes.deepseek] — select this case alone in a fresh process; expect offline construction and (32, 2048, 1) in the layout test.
• test_recorded_layout_parses_the_card_wording — retain the (32, 2048, 1) assertion in the parser test.
• test_deepseek_v4_flash_128gpu_gb200_fp8mx_config — run alongside the new tests; retain EP32, GBS2048, MBS1 and the existing benchmark-setting assertions in the recipe regression test.
• Proposed test_gb200_token_throughput_matches_recorded_step_time — load the GB200 leaf and assert TPS equals 4096 * 2048 / (step_time_ms / 1000) / 128 with absolute tolerance 0.0005; expect 8044.741.
• test_generated_outputs_and_navigation_are_current — retain assert generator.generate(REPO_ROOT, check=True) in the catalog consistency test.
• No perf tests impacted.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

[svcnvidia-nemo-ci]

[P2] Import recipe modules before installing the offline patches

Failure: The prior finding remains unresolved; this fixture is unchanged from the reviewed version. The smallest trigger is selecting only test_verified_performance_layout_matches_the_published_recipe in a fresh pytest process, without collecting another recipe test. The helper scans currently imported recipe modules for AutoBridge, then imports Flux/DeepSeek library recipes after that scan. The inherited GB200 factory therefore retains the real bridge. Its config loader receives the helper's namespace without supported architectures, and validation raises ValueError before comparing layouts. Expected behavior is independent offline construction and a passing comparison. Running the whole file can mask this: the parser test's earlier autouse-fixture invocation has already imported the library modules before the layout test's fixture scans them; full-suite collection can likewise mask it.

Correction: Replace this fixture's body with:

    for _, _, module in PERFORMANCE_LEAVES:
        importlib.import_module(module)
    patch_recipe_construction_dependencies(monkeypatch)

The selected performance package imports its inherited library owners before the helper scans them. The invariant is that every recipe-owned AutoBridge reached during construction is patched regardless of prior test collection or execution. No shared-helper or production-recipe change is needed. This is a source-traced, unexecuted suggestion; verify it before adopting.

Preservation: Keep actual recipe construction and the card-derived --recipe selection, not a mocked layout tuple. Retain the valid EP32/GBS2048/MBS1 comparison and parser assertion, rejection of stale EP64 wording, and compatibility with already-imported recipes. The proposed change only moves imports before patch installation; it does not change the recipe's layout or benchmark settings.

Acceptance: Run uv run python -m pytest tests/unit_tests/recipes/test_verification_card_performance_layouts.py::test_verified_performance_layout_matches_the_published_recipe alone in a fresh process without an HF model cache: expect one passing case, no network access, and no architecture-validation error. Also run the whole file and then alongside the existing DeepSeek V4 performance-recipe tests; retain the parser's (32, 2048, 1) assertion and test_deepseek_v4_flash_128gpu_gb200_fp8mx_config assertions for EP32/GBS2048/MBS1. Proposed negative regression test_stale_ep64_layout_is_rejected: use wording PP1/EP64/CP1/TP1/DP2 with GBS/MBS 2048/1 with the unchanged factory; expect the comparison to fail as (64, 2048, 1) != (32, 2048, 1).

[qiaochuz-nv]

Retrospective QA-fix tracking issue: #6519