Skip to content

build: support Transformers 5.14.1 - #5286

Closed
bvolpato wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
bvolpato:bvolpato/transformers-5.14.1
Closed

bvolpato wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
bvolpato:bvolpato/transformers-5.14.1

Conversation

@bvolpato

@bvolpato bvolpato commented Aug 3, 2026 •

Copy link
Copy Markdown

What does this PR do?

Extends Megatron Bridge support from Transformers 5.12.1 through 5.14.1.

Changelog

  • Raise Transformers upper bound from <=5.12.1 to <=5.14.1.
  • Regenerate uv.lock at Transformers 5.14.1.
  • Update GLM-5 config coverage for Transformers 5.14.
  • Keep Qwen3-Omni audio forward compatible with the Transformers 5.14 API change by using Bridge's existing audio-length helper.

Why

NovaSky-AI/SkyRL#1974 adds GLM-5.2 config support but limits Transformers to 5.11 because of the Megatron Bridge compatibility range. SkyRL's GLM-5.2 stack works with Transformers 5.14.1, so raising Bridge's upper bound removes that constraint.

Transformers 5.14 also removed Qwen3OmniMoeAudioEncoder._get_feat_extract_output_lengths. Bridge still called that method, which broke Qwen3-Omni audio forward and its training smoke test. Bridge already has the matching length helper for Qwen3-Omni RoPE, so this PR uses it in the audio path too.

GitHub Actions CI

Labels needs-more-tests and full-test-suite request H100 L0, L1, and L2 coverage against the locked maximum endpoint.

Local validation used the NVIDIA PyTorch 26.06 container on an RTX 5070 Ti with CUDA 13.3 and Transformers 5.14.1:

  • uv lock --check
  • 166 unit tests passed across Qwen3-Omni, Qwen2.5-Omni, Qwen3-ASR, Qwen2 Audio, Ministral3, and GLM-5
  • Both Qwen3-Omni tests that failed in the first CI run passed on CUDA
  • uv run pre-commit run --all-files

Full GLM-5.2 training and GB200 hardware were not tested locally. GB200 jobs are skipped for external-contributor PR runs, so this PR does not claim GB200 validation.

Before your PR is "Ready for review"

Pre checks:

  • Read and followed contributor guidelines.
  • Existing tests cover the Transformers 5.14 regressions found by CI.
  • No documentation changes needed. Supported range is declared in package metadata.
  • No optional components added.

Additional Information

  • Follows #4928, which expanded support through Transformers 5.12.1.
  • Qwen3-ASR native registration protection from #4876 is already on main.
  • TorchAudio remains a downstream binary-pairing concern and is not added as a Bridge dependency.
  • Minimum supported Transformers version remains 5.8.

@bvolpato
bvolpato requested a review from a team as a code owner August 3, 2026 23:56
@copy-pr-bot

copy-pr-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@bvolpato

bvolpato commented Aug 3, 2026

Copy link
Copy Markdown
Author

/ok to test 1613992

@bvolpato

bvolpato commented Aug 3, 2026

Copy link
Copy Markdown
Author

CI request after contributor vetting: please add needs-more-tests and full-test-suite. This dependency ceiling update needs L0/L1/L2 coverage, especially Qwen2 Audio, Qwen3-ASR, Ministral3, and model conversion jobs.

@yaoyu-33 yaoyu-33 added area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Aug 4, 2026
@bvolpato

Copy link
Copy Markdown
Author

Any changes to get this in? It would simplify things a lot on our side!

@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test afbe93a

@bvolpato
bvolpato force-pushed the bvolpato/transformers-5.14.1 branch from afbe93a to c35abb2 Compare August 13, 2026 01:53
@bvolpato bvolpato changed the title build(deps): upgrade Transformers to 5.14.1 build: support Transformers 5.14.1 Aug 13, 2026
@bvolpato

Copy link
Copy Markdown
Author

/ok to test c35abb2

@bvolpato

Copy link
Copy Markdown
Author

Rebased onto current main and fixed the two Qwen3-Omni failures from the first CI run. Local CUDA validation on Transformers 5.14.1 now passes 166 targeted tests across Qwen3-Omni, Qwen3-ASR, Qwen2 Audio, Ministral3, and GLM-5. Ready for another /ok to test c35abb2e2d619e326415416b31edc3972221120a when convenient.

bvolpato and others added 3 commits August 14, 2026 00:34
Signed-off-by: bvolpato <brunocvcunha@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: bvolpato <brunocvcunha@gmail.com>
@bvolpato
bvolpato force-pushed the bvolpato/transformers-5.14.1 branch from c35abb2 to f3fd952 Compare August 14, 2026 04:38
@bvolpato

Copy link
Copy Markdown
Author

/ok to test f3fd952

Rebased onto current main and regenerated uv.lock at Transformers 5.14.1 while preserving latest MCore and TransformerEngine pins.

Validation:

  • uv lock --check
  • 35 targeted CUDA unit tests passed, including both prior Qwen3-Omni failures
  • uv run pre-commit run --all-files

Ready for full L0/L1/L2 CI.

@bvolpato

Copy link
Copy Markdown
Author

@cuichenx, would you mind taking a look at this Transformers 5.14.1 compatibility update? You handled the previous Transformers bump and have recently worked in both the Qwen3-Omni and GLM paths, so your context would be helpful. If someone else owns this path now, happy to redirect.

@bvolpato

bvolpato commented Sep 2, 2026

Copy link
Copy Markdown
Author

Closing as superseded by #5479, which moved the project to Transformers 5.15.0 and included the corresponding Qwen3-Omni and GLM-5 compatibility updates.

@bvolpato bvolpato closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work community-request full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants