[jupiterepoch]

• I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Support use_gigatoken=True while maintaining HF parity (with fallback on a fixed set of unicode characters). The default path remains use_gigatoken=False, this change requires user to explicitly opt in via use_gigatoken=Trueor the--use-gigatoken` flag. The tokenization encoding time is 10x - 40x faster depending on input. The functional support is achieved through:

1. Support right-aligned numeric grouping Gigatoken PR. The current Gigatoken supports NT3, where a number string "12345" is broken into "1", "2", "3", "4", "5", whereas NT4 would break it into "12", "345".
2. Wire gigatoken into Megatron-LM
3. Fall-back to HF when Gigatoken process unicode characters differently. Gigatoken's ICU 2.2 Unicode 17 character classes differ from HF tokenizers 0.22's Unicode 16 regex. A compiled guard covers the 28 affected ranges and routes the entire input or batch through HF to preserve token boundaries. ASCII skips the regex; compatible Unicode uses Gigatoken. The guard also checks characters introduced by rendered chat templates. Direct native encoding and bulk encode_files bypass this guard.

Validation

Validated with end2end run, with transformers 5.15.1 and tokenizers 0.22.2:

• All 38 new tokenizer tests and 283 existing serving tests passed. Tests are added to the existing tokenizer test file.
• Local parity checks against a complete HF tokenizer configuration covered all 1,112,064 valid Unicode scalars in contextual inputs, in both normal and ordinary-text encoding modes: zero ID or round-trip differences.
• Public HTTP handlers with the real tokenizer configuration and a canned inference client passed prompt-ID and response comparisons, native/fallback routing, template-introduced Unicode, tokenizer copying and 16 concurrent requests.

Setup for right-aligned tokenizer configurations

Install the required fork into the same Python environment used by the serving/RL processes, including the container used on worker nodes. After the normal Megatron environment and Gigatoken dependencies are installed, replace only Gigatoken with the tested revision:

python -m pip install --force-reinstall --no-deps \
  "gigatoken @ git+https://github.com/jupiterepoch/gigatoken.git@522b4fadabd2a6d475fc355b0b4b54ee691c24f2"

This source build requires Git and the Rust nightly toolchain specified by the fork's rust-toolchain.toml. Verify the loaded module and recorded source revision:

python - <<'PYTHON'
import gigatoken as gt
from importlib.metadata import distribution

print(gt.__file__)
print(distribution("gigatoken").read_text("direct_url.json"))
PYTHON

The recorded repository should be https://github.com/jupiterepoch/gigatoken.git and vcs_info.commit_id should be 522b4fadabd2a6d475fc355b0b4b54ee691c24f2.

Contribution process

Pre-checks

• I have added relevant unit tests
• I have added relevant functional tests
• If this PR adds or changes a GPU kernel (Triton, jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it in tests/unit_tests/determinism/kernels/manifest.py (guide)
• I have added proper typing to my code Typing guidelines
• [ x I have added relevant documentation
• I have run the autoformatter.sh on my PR

[github-actions]

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

1. Add the oncall reviewer (optional reviewer)
2. Add required review teams based on your changes

See the contribution guide for more details.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.