Skip to content

docs(misc): point GB200 fine-tunes at the card's own import - #6366

Merged
cuichenx merged 1 commit into
mainfrom
pruprakash/fix_misc_gb200_finetune_checkpoint_paths
Oct 9, 2026
Merged

cuichenx merged 1 commit into
mainfrom
pruprakash/fix_misc_gb200_finetune_checkpoint_paths

Conversation

@pruprakash

@pruprakash pruprakash commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Fixes five verified GB200 fine-tune commands on the Qwen3.6-35B-A3B and DeepSeek-V4-Flash cards that load a --pretrained_checkpoint no item on their card writes, by pointing them at the card's own GPU import.

Changelog

  • examples/model_verification_cards/qwen3.6-35b-a3b/card.yaml: GB200 sft/peft read imported-megatron/iter_0000000 instead of gb200-imported/iter_0000000.
  • examples/model_verification_cards/deepseek-v4-flash/card.yaml: GB200 sft/sft_long_context/peft read dsv4-flash/import-gpu/iter_0000000 instead of work/models/deepseek-v4-flash-megatron.
  • docs/_static/model-verification/catalog-v1.json, docs/models/..., docs/fern/.../*.mdx: regenerated (--check passes).

Additional Information

  • Root cause: the card's import writes one directory and the fine-tunes read another. The H100 siblings (Qwen) and pretrain.GB200 (DeepSeek) already read the import path. Same class as docs: correct Nemotron verification artifact paths #6286.
  • Unchanged: statuses, dates, metrics and hardware claims. Only the paths changed. No GB200 rerun was done.
  • Regression? No. The paths have been wrong since the items were added (feat(qwen-vl): add Qwen3.6 verification card #5055, docs: add DeepSeek V4 Flash performance verification #5724).
  • Verification: each new path equals the --megatron-path written by the card's own hf_to_megatron_gpu item. validate_card.py passes for both cards, generate_model_verification_catalog.py --check is clean, and the card/catalog/doc-consistency unit tests pass (70). QA's card audit (nmfw_tests) flagged exactly these 5 leaves on main.
  • Left for maintainers: GLM-5.2 GB200 sft/sft_long_context read a Hugging Face directory (glm5-2/hf-<rev>) that no item stages. They are not changed here, because repointing would change the checkpoint format.
  • NVBug: 6875021.

@pruprakash pruprakash self-assigned this Oct 8, 2026
@pruprakash pruprakash added docs Documentation-only updates or documentation debt area:misc Cross-cutting utilities, logging, helpers, and other changes needs-review PR is ready for code review and waiting on a reviewer qa_rcca_done labels Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pruprakash

Copy link
Copy Markdown
Contributor Author

/review

@pruprakash
pruprakash requested a review from cuichenx October 8, 2026 22:37

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all eight changed files. The five checkpoint corrections match the cards’ published GPU import destinations, and the generated catalog, Sphinx pages, and Fern mirrors consistently contain the corrected paths. Two blocking coverage gaps remain in the new consistency guard, detailed inline.

Static review only: no tests or GPU workloads were executed.

Suggested test cases

Review generated by codex

Comment thread tests/unit_tests/doc_consistency/test_verification_card_published_commands.py Outdated
Comment thread tests/unit_tests/doc_consistency/test_verification_card_published_commands.py Outdated
The Qwen3.6-35B-A3B GB200 SFT/PEFT items and the DeepSeek-V4-Flash GB200
SFT, long-context SFT and PEFT items read --pretrained_checkpoint
directories that no item on their card writes. Point them at the path the
card's GPU import writes, as the sibling items already do, and regenerate
the catalog and model pages.

Detected by: megatron-bridge QA (NVBug 6875021)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
@pruprakash
pruprakash force-pushed the pruprakash/fix_misc_gb200_finetune_checkpoint_paths branch from 3a2884c to 761a0fe Compare October 8, 2026 23:06
@pruprakash

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review complete; no eligible findings remain.

Both prior findings no longer apply: the checkpoint guard and its regression test were removed, and the published-command test module matches base. The five corrected readers match their cards’ published GPU-import destinations; the inspected catalog, Sphinx pages, and Fern mirrors contain the same corrected paths.

The previously reviewed head tree was unavailable, so a complete inter-review comparison was not possible. This review verified the prior findings and claimed removal, plus the current path corrections and generated copies. Static review only; no tests or GPU workloads were executed.

Suggested test cases

Review generated by codex

@pruprakash

This comment was marked as outdated.

@copy-pr-bot

This comment was marked as off-topic.

@pruprakash

Copy link
Copy Markdown
Contributor Author

/ok to test 761a0fe

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@cuichenx
cuichenx enabled auto-merge (squash) October 9, 2026 00:05
@cuichenx
cuichenx merged commit 40546e9 into main Oct 9, 2026
87 checks passed
@cuichenx
cuichenx deleted the pruprakash/fix_misc_gb200_finetune_checkpoint_paths branch October 9, 2026 02:08
@qiaochuz-nv

Copy link
Copy Markdown

Retrospective QA-fix tracking issue: #6385. It records the fix authored by @pruprakash and delivered by this merged PR.

This branch was successfully deployed

1 active deployment
test — 761a0feb Deployed Oct 8, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22639
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:misc Cross-cutting utilities, logging, helpers, and other changes docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer qa_rcca_done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants