Skip to content

[otel-4/7] Own startup, linked training traces, and exit cleanup in telemetry - #8277

Draft
rhewett-nv wants to merge 4 commits into
NVIDIA:mainfrom
rhewett-nv:rhewett/otel/public-main-04-trainer-lifecycle
Draft

rhewett-nv wants to merge 4 commits into
NVIDIA:mainfrom
rhewett-nv:rhewett/otel/public-main-04-trainer-lifecycle

Conversation

@rhewett-nv

@rhewett-nv rhewett-nv commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Completes the trainer telemetry lifecycle from Python startup through linked training traces and ordered exit cleanup.

This is PR 4 of 7 and is intended to be reviewed relative to PR 3.

Summary

  • Captures startup timing for the GPT and MIMO Python entrypoints and records startup work beneath the trainer lifecycle.
  • Links training-loop traces and final checkpoint work without changing application control flow.
  • Routes normal and signal-driven cleanup through one lifecycle owner so open spans finish before telemetry providers shut down.

Issue tracking

Linked issue: N/A — follow-up integration work.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • This PR does not add or change a GPU kernel, so no determinism manifest update is required.
  • I have added proper typing to my code (typing guidelines)
  • I have added relevant documentation
  • I have run the autoformatter checks on my PR

Validation

  • Focused telemetry unit suite at this boundary: 201 passed.
  • Scoped Ruff, Black, adjacent-diff, DCO, and signature checks passed.
  • No local GPU or distributed run was performed.

@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rhewett-nv rhewett-nv changed the title Own startup, linked training traces, and exit cleanup in telemetry [otel-4/7] Own startup, linked training traces, and exit cleanup in telemetry Oct 15, 2026
@rhewett-nv
rhewett-nv force-pushed the rhewett/otel/public-main-04-trainer-lifecycle branch 2 times, most recently from 49efebc to c05e8b0 Compare October 16, 2026 23:29
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
@rhewett-nv
rhewett-nv force-pushed the rhewett/otel/public-main-04-trainer-lifecycle branch from c05e8b0 to 106f1a0 Compare October 16, 2026 23:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant