feat(conversion): add a failure exit mode to HF/Megatron comparison

[yuchenwang3]

What does this PR do?

Add --fail-on-mismatch to the HF/Megatron forward comparison. Today a different next token or cosine similarity below 0.99 is printed, but the command still exits successfully. The new flag makes those cases fail the command, so a validation job can use its exit status.

Changelog

• Keep the existing diagnostic behavior when the flag is omitted.
• Require matching next tokens within the HF vocabulary and cosine similarity >= 0.99. Absolute logit differences remain diagnostic metrics.
• Reduce the verdict across all ranks before raising, including non-last pipeline stages, and destroy the process group on failure.
• Document the flag and cover mismatched tokens, low cosine similarity, NaN logits, padded vocabulary entries, and failure propagation from either rank.

Validation

• 38 targeted CPU tests passed on macOS, Python 3.12.13 and PyTorch 2.14.1. Model loads/forwards are mocked in the CPU tests; verdict propagation uses a real two-process Gloo group.
• Unpatched main fails the three regression cases that expect an error for token mismatch, low correlation, or NaN logits.
• All repository pre-commit hooks passed.
• Strict mypy reports the same 39 existing diagnostics on main and this patch, with no new diagnostic.
• CUDA/NCCL and full-model inference were not run for this patch.

Local test command (isolates the CPU test file from GPU-dependent conftest imports):

uv run --no-project --python 3.12 --with torch --with pytest --with rich --with pre-commit --with mypy \
  python -m pytest --noconftest tests/unit_tests/test_compare_mask_handling.py -q

GitHub Actions CI

External contributor; NVIDIA runner approval is needed for CI.

Before your PR is ready for review

• Read the contributor guidelines.
• Added necessary tests.
• Updated the example documentation.
• Changes optional components (not applicable).

Additional Information

Implementation and tests were written with OpenAI Codex.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yuchenwang3]

@cuichenx Could you review the opt-in --fail-on-mismatch mode and authorize CI for 2153756f2? All 38 CPU tests pass, including two-rank Gloo failure propagation. CUDA/NCCL still needs CI.

[cuichenx]

/ok to test 2153756

[github-actions]

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

[cuichenx]

/ok to test bb4937a