Expose enable_continuous_profiling CLI flag in MaxText RL - #5061
Open
youchunni wants to merge 4 commits into
Open
Expose enable_continuous_profiling CLI flag in MaxText RL#5061youchunni wants to merge 4 commits into
youchunni wants to merge 4 commits into
Conversation
youchunni
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
richjames0,
shralex,
shuningjin,
vipannalla and
xibinliu
as code owners
August 29, 2026 00:35
There was a problem hiding this comment.
Code Review
This pull request introduces a new configuration option, enable_continuous_profiling, to support saving traces larger than 2GB in the RL/Tunix profiler workflow. The reviewer noted a typo in the JAX version specified in the configuration comment (suggesting 0.4.11 instead of 0.11.1) and recommended clarifying that this option is currently only supported in the RL/Tunix profiler workflow.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Expose the
enable_continuous_profilingCLI configuration flag in MaxText RL workflow to support saving trace profiles larger than 2GB as.xplane.riegeli.Context & Solution:
When profiling large model workloads or multi-step training runs, monolithic
.xplane.pbtrace files can exceed Protocol Buffer's 2GB wire format limit, leading to failed file upload.Setting
enable_continuous_profiling=Trueconfigures the profiler stream smaller trace chunks into Riegeli record containers (.xplane.riegeli), bypassing the 2GB single-protobuf size limitation.Note: Direct streaming of
.xplane.riegelifiles to GCS paths (gs://) is currently not supported. Profiles should be saved locally on the host path and uploaded to GCS.Specific Changes:
Added
enable_continuous_profiling(boolean, defaultFalse) tosrc/maxtext/configs/base.ymlandsrc/maxtext/configs/types.py.Updated
src/maxtext/trainers/post_train/rl/train_rl.pyto forwardenable_continuous_profilingto the profiler configuration. Related Tunix PR: Supportenable_continuous_profilingoption in the Tunix JAX profiler wrapper google/tunix#2036Added documentation notes in
base.ymlclarifying that continuous profiling requires JAX >= 0.11.1.Tests
enable_continuous_profiling=Trueenables continuous trace collection and produces.xplane.riegelicontainer files on local path when the profile is >2GB. See screenshot:enable_continuous_profiling=Falseretains existing JAX profiling behavior.Checklist
gemini-reviewlabel.