Skip to content

test: AUT-1599 validate Megatron Core dev compatibility - #5709

Closed
svcnemo-autobot wants to merge 41 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:chore/bump-mcore-dev-260819
Closed

test: AUT-1599 validate Megatron Core dev compatibility#5709
svcnemo-autobot wants to merge 41 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:chore/bump-mcore-dev-260819

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Background

Validate frozen MCore main 731b791469004f8fdcb896e65d610d6b1b0ebb32 and dev 2f2f8ebaef90f4d62a18a43808398223b1fc5809. This PR must never merge; it validates the payload for #5710.

What changed

  • Retained both commit files, the dev gitlink, version-bound uv metadata, and independently generated dev lock.
  • Carried shared compatibility dd6e1cc51de7ad3eed3b0f11d0444aa3f755cdce.
  • Added a signed, evidence-only audit for Emerging Optimizers b309e2f0 without changing dependency inputs.

Details

flowchart LR
  T[immutable dependency tree] --> H[blob and manifest hashes]
  H --> A[install and runtime audit]
  A --> D[dev validation lane]
Loading

The 44 installed package blobs use ordinary setuptools discovery with Torch and absl dependencies. No entry points, submodules, symlinks, native build hooks, network/process/filesystem side effects, dynamic loading, or serialization paths were found.

Tested

  • Full package tree and build metadata inspected at exact commit
  • Recursive tree and package-manifest SHA-256 values recorded in the signed commit
  • git diff --check
  • Frozen pin pair and dev gitlink unchanged
  • Paired local streams and current-head CICD NeMo pending

@svcnemo-autobot
chore(deps): bump Megatron Core refs
bcdc43f 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
chore(deps): bump Megatron Core dev
aaaccd2 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix(docker): keep source metadata writable
6650bf9 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge shared MCore compatibility fixes
ee3f39c
@svcnemo-autobot
Merge upstream main
4e9931c
@svcnemo-autobot
Merge shared MCore compatibility fixes
19fc8bd
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner August 20, 2026 09:28
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge labels Aug 20, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

/ok to test 19fc8bd

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

/ok to test d30c735

@yaoyu-33 yaoyu-33 added area:ci ci CI, automation, test queue, or workflow infrastructure work dummy-pr Filed this PR to run tests, not going to merge labels Aug 20, 2026
@svcnemo-autobot
test(training): update GDN fixture for MCore dev
b5a9054 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge remote-tracking branch 'fork/fix/mcore-compat-260819' into chor…
3f72e5b 
…e/bump-mcore-dev-260819

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge remote-tracking branch upstream/main into fix/mcore-compat-260819
729e076
@svcnemo-autobot
fix(docker): disable recursive TE build fetches
d4409c8 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@ko3n1g

ko3n1g commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test 80dd95e

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test 64f75d3

@svcnemo-autobot
Merge upstream main into nightly compatibility
b9a66a3 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge commit 'b9a66a38cba514ff37b7b39979aef76b77eedbb2' into chore/bu…
0064edf 
…mp-mcore-dev-260819

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
chore(deps): record MCore dev provenance
92c2c66 
Frozen MCore commit: 2f2f8ebaef90f4d62a18a43808398223b1fc5809
MCore tree: 24591f89ccd241e6d28642bbab3f1ae1f9eb555f
MCore tree manifest SHA-256: 9af617576e733d1cdd646ff5b704f07f1a47f1be07e6425959d35bc7c5b73f44
MCore pyproject blob: ff718cfabb264e773046d241718cdd1bb1c0f06d
MCore lock blob: ae4e074bd42bf4bc3a01d4ca229ce8254e79f710
Bridge pyproject blob: 17db612
Bridge lock blob: e1fe886

Resolved direct Git sources in the MCore lock:
- Emerging-Optimizers b309e2f01cda75dc96a6dc1a2355a7b3b64b5e16
- fast-hadamard-transform f134af63deb2df17e1171a9ec1ea4a7d8604d5ca
- FlashMLA b7643bd54521f563b839b98289b5cd048c062ba2
- NeMo Run 17ae86b64d7f75653351664f5d8c9e466faede00
- cudnn-frontend 0a14b7181d129d30e7bad34b8c3ed0a0c995e23d
- TransformerEngine f031cf87bd054c7558b887df7bed93975456667f

Bridge resolves the same TE, Emerging-Optimizers and FlashMLA commits; its
fast-hadamard-transform v1.1.0 resolves to
1cc807efbd6cc001df359822d60bf6052dd66859.

Exact setup/build manifests, recursive source trees, and immutable TE/FlashMLA
submodule gitlinks were inspected. Build hooks compile extensions, initialize
immutable submodules, or fetch named upstream dependencies/release wheels; no
dependency input was changed because the approval finding identified no
concrete malicious source or exfiltration mechanism.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test 92c2c66

@svcnemo-autobot
test(training): guard optional GDN2 coverage
84f077e 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test f102cfc

@nemo-autobot-origin
nemo-autobot-origin Bot force-pushed the chore/bump-mcore-dev-260819 branch from f102cfc to deaa8cd Compare August 22, 2026 16:18
@svcnemo-autobot
chore(deps): record Emerging Optimizers audit
4011428 
Audit NVIDIA-NeMo/Emerging-Optimizers at the locked dev resolution
b309e2f01cda75dc96a6dc1a2355a7b3b64b5e16.

Repository object evidence:
- Git tree: b14842f4a459d8b6494dc2c5ea268194dcaf884a
- Full recursive tree-manifest SHA-256: 8adbc3734fb150bff9dd3b49072e5427e376e7b8a49727b0659b797e307f2237
- Installed package manifest: 44 regular blobs; SHA-256 2d0826c87f3f18e76d3a0af04244078948dd4281f59906e81118f800f3566b13
- pyproject.toml blob: a1f7327489c439a3e696474458a12630b6ebd762
- package_info.py blob: 9ae16226d42f52d1277b4049ba5f0233732871bd
- __init__.py blob: 2a2be97

Install surface:
- PEP 517 backend is setuptools.build_meta with setuptools>=61.
- Package discovery includes only emerging_optimizers and its subpackages.
- Runtime dependencies are only torch and absl-py.
- Version and readme are read from package_info.py and README.md.
- No setup.py, setup.cfg, MANIFEST.in, entry point, console script,
  submodule, symlink, native build hook, or package-data hook is present.

Runtime surface:
- Inspected all 44 installed Python blobs, including optimizer classes,
  update functions, registry/mixin helpers, and the Triton SYRK kernel.
- Imports are limited to Python stdlib, torch/triton, absl, and internal
  emerging_optimizers modules.
- No added or extant subprocess, shell, network, filesystem I/O, dynamic
  import/eval/exec, pickle, ctypes/CFFI, torch.hub, or load_library path was
  found in pyproject.toml or the installed package tree.
- The only binary changed from 1effa026 is a documentation PNG outside the
  installed package.

The lock update is the exact immutable dependency selected by frozen MCore dev;
this evidence-only commit changes no dependency input or executable source.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

/ok to test 4011428

@svcnemo-autobot nv-nemo-autobot-conversational

svcnemo-autobot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Validation-only lane is complete on head 40114284838143ae97520194e25e865c9bdd3874 with shared compatibility dd6e1cc51de7ad3eed3b0f11d0444aa3f755cdce.

The full unit stream and all 66 active H100 functional launchers passed, including the quantization launcher with its prepared offline dataset overlay. The authoritative CICD NeMo gate passed in run 32585007294. The Codecov patch status is non-authoritative for this unreleased-dev validation lane.

Do not merge this PR. #5710 is the sole landing vehicle; close this PR as superseded after #5710 lands.

@ko3n1g ko3n1g closed this Aug 25, 2026

This branch was previously deployed

1 inactive deployment
public — 40114284 Deployed Aug 22, 2026 by copy-pr-bot[bot] via release / finalize / publish-docs #4904
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:ci ci CI, automation, test queue, or workflow infrastructure work dummy-pr Filed this PR to run tests, not going to merge full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@svcnemo-autobot @svcnvidia-nemo-ci @ko3n1g @yaoyu-33