diff --git a/llm_post_training/README.md b/llm_post_training/README.md index 8036c3256..d4c60e3ea 100644 --- a/llm_post_training/README.md +++ b/llm_post_training/README.md @@ -33,18 +33,19 @@ through Ray: in per-task Apptainer/Singularity containers. The benchmark implementation is provided by -[NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL/tree/mlperf-training-qwen35-main). +[NVIDIA NeMo-RL](https://github.com/NVIDIA-NeMo/RL/tree/mlperf-training-qwen35-next-v2). This repository checks out NeMo-RL under the local submodule path -`llm_moe_grpo/RL`, pinned to commit -`3fca04c9b313d313302923a5bb6b0c8dc0340ed6`. The source commit, recursive +`llm_post_training/RL`, pinned to commit +`f987c0596af2bd6cfb9958e768deb8df951d7504`. The source commit, recursive submodule revisions, Python dependency lock, and downstream build-time patches are recorded in the benchmark container. -The principal entrypoints are: +The principal entrypoints and reference sections are: -| Purpose | Path | +| Purpose | Path or section | |---|---| -| Dataset download and preprocessing | `download_dataset.sh` | +| Qualified dataset download | [Training and validation JSONL](#training-and-validation-jsonl) | +| Dataset production from source | `download_dataset.sh` | | Dataset identity verification | `verify_dataset.sh` | | Pretrained-checkpoint download | `download_checkpoint.sh` | | Pretrained-checkpoint conversion | `RL/docker/mlperf/data_scripts/convert_ckpt.sh` | @@ -83,11 +84,11 @@ scheduler nodes. From the root of this repository: ```bash -git submodule update --init --recursive llm_moe_grpo/RL -cd llm_moe_grpo/RL +git submodule update --init --recursive llm_post_training/RL +cd llm_post_training/RL test "$(git rev-parse HEAD)" = \ - "3fca04c9b313d313302923a5bb6b0c8dc0340ed6" + "f987c0596af2bd6cfb9958e768deb8df951d7504" git submodule update --init --recursive ``` @@ -103,7 +104,7 @@ submodules, installs the locked dependencies, applies the benchmark's guarded patches, builds the Arm HybridEP dependency, installs Apptainer, and prefetches the Ray and NeMo Gym virtual environments. -From the `llm_moe_grpo/RL` directory: +From the `llm_post_training/RL` directory: ```bash source_commit="$(git rev-parse HEAD)" @@ -152,12 +153,38 @@ The run requires five external artifacts: |---|---|---| | Qwen3.5 policy checkpoint | `HF_CKPT_PATH` | `Qwen/Qwen3.5-397B-A17B` snapshot `8472618112abcbd45acbcdc58436aff4233c23f7` | | Megatron checkpoint cache | `NRL_MEGATRON_CHECKPOINT_DIR` | Writable directory; an empty directory is converted from the HF checkpoint on first use | -| Curriculum training JSONL | `QWEN35_CURRICULUM_DATA_PATH` | Exact 685-row curriculum-v2 artifact described below | -| Validation JSONL | `NEMO_GYM_SWE_VALIDATION_DATA_PATH` | 256 held-out R2E-Gym tasks | -| Per-task SIF images | `NEMO_GYM_SWE_SIF_DIR` | All 941 environments referenced by the qualified train/validation lists; the checked-in helper builds a compatible 977-image superset | +| Curriculum training JSONL | `QWEN35_CURRICULUM_DATA_PATH` | Exact 700-row artifact described below | +| Validation JSONL | `NEMO_GYM_SWE_VALIDATION_DATA_PATH` | 251 held-out R2E-Gym tasks | +| Per-task SIF images | `NEMO_GYM_SWE_SIF_DIR` | All 951 environments referenced by the qualified train/validation files | ### Policy checkpoint +Download the qualified Hugging Face checkpoint from +[MLCommons storage](https://training.mlcommons-storage.org/index.html#qwen3-5-397b-benchmark): + +```bash +cd llm_moe_grpo + +checkpoint_dir="/qwen35-397b-a17b" + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${checkpoint_dir}" \ + https://training.mlcommons-storage.org/metadata/qwen35-397b-a17b.uri + +export HF_CKPT_PATH="${checkpoint_dir}" +mkdir -p "/qwen35-mcore-cache" +export NRL_MEGATRON_CHECKPOINT_DIR="/qwen35-mcore-cache" +``` + +The downloader verifies the package's MD5 manifest. `HF_CKPT_PATH` must name +the directory containing `config.json` and the model weight shards. The +MLCommons package mirrors Hugging Face revision +`98d1a504ba52e88924b3a3a008447cf2fdbd518c`; its model, configuration, and +tokenizer files match revision `8472618112abcbd45acbcdc58436aff4233c23f7` +pinned by the helper below. Those revisions differ only in `README.md`. + +#### Optionally, download the checkpoint from Hugging Face + The checked-in download helper pins the exact Hugging Face model revision and uses the standard Hugging Face cache layout: @@ -190,22 +217,16 @@ incompatible NeMo-RL/Megatron Bridge/Megatron Core stack is not supported. The benchmark training input is: ```text -benchmark_r2e_gym_easy_train.filtered.curriculum-v2-classic-cycles2-seed20260710.jsonl +benchmark_r2e_gym_easy_train.jsonl ``` Its known identity is: | Property | Value | |---|---:| -| Rows | 685 | -| Size | 444,421,669 bytes | -| SHA-256 | `c07bcd64ed1c558e28d091239104e38295a5e696c1d21bb0b61f0346c7eaa0f7` | - -The checked-in converter and instance-ID lists reproduce this filtered, -curriculum-ordered input. Commit -`9e64cc37197c9d00954ee8285c623fd7e53595e2`, which is included in the pinned -NeMo-RL revision, removes 36 unusable environments from the former 721-row -training list and orders the remaining 685 rows deterministically. +| Rows | 700 | +| Size | 446,364,412 bytes | +| SHA-256 | `e772af09599270ab16a04ccbbcef395bcb2929607f050478033a84b087a606fc` | The held-out validation input is: @@ -217,17 +238,38 @@ Its reference identity is: | Property | Value | |---|---:| -| Rows | 256 | -| Size | 173,801,096 bytes | -| SHA-256 | `452d0e6b3c1973669334062dc24931355de51749df1ab51fc9bb71a129f7bb5c` | +| Rows | 251 | +| Size | 172,078,629 bytes | +| SHA-256 | `2d5bddf717afb4389510a7d197f26ce5eb7f610aab4b20d6af6e0150de6f2c79` | -The package pins source dataset revision -`e8b9fcbce43eaca0dc2c0d4798ee6f3e965f590a` of -[`R2E-Gym/R2E-Gym-Subset`](https://huggingface.co/datasets/R2E-Gym/R2E-Gym-Subset). -The download script retrieves that revision, runs the pinned converter, gives -the curriculum output its canonical name, and invokes the independent -verification script. The preprocessing host requires `uv`/`uvx` and Git -network access for repository metadata not already present in `cache_dir`. +Download both qualified files from the +[Qwen3.5 397B benchmark page](https://training.mlcommons-storage.org/index.html#qwen3-5-397b-benchmark) +with the MLCommons R2 Downloader: + +```bash +cd llm_moe_grpo + +dataset_dir="/benchmark-r2e-gym-easy" + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${dataset_dir}" \ + https://training.mlcommons-storage.org/metadata/benchmark-r2e-gym-easy.uri + +export QWEN35_CURRICULUM_DATA_PATH="${dataset_dir}/benchmark_r2e_gym_easy_train.jsonl" +export NEMO_GYM_SWE_VALIDATION_DATA_PATH="${dataset_dir}/benchmark_r2e_gym_easy_val.jsonl" +``` + +The downloader verifies the package's MD5 manifest. Verify the SHA-256 values +above before running because `data.shuffle: false` makes training row order +part of the benchmark input. + +#### Optionally, produce the dataset from source + +The checked-in converter and instance-ID lists reproduce the qualified, +curriculum-ordered input from the pinned +[`R2E-Gym/R2E-Gym-Subset`](https://huggingface.co/datasets/R2E-Gym/R2E-Gym-Subset) +source revision. The preprocessing host requires `uv`/`uvx` and Git network +access for repository metadata not already present in `cache_dir`. ```bash cd llm_moe_grpo @@ -246,33 +288,43 @@ cache_dir="/r2e-repository-cache" `download_dataset.sh` uses `RL/tools/create_r2e_gym_easy_subset_jsonl.py`, -`RL/tools/train-instance-ids.txt`, and `RL/tools/val-instance-ids.txt`. -The similarly named converter under `RL/docker/mlperf/data_scripts/` retains -the earlier 721-row, non-curriculum training split and is not the qualified -data generator. - -The curriculum algorithm is fixed in the script: seed `20260710`, 16 examples -per curriculum batch, and two easy-to-hard cycles. It ranks difficulty using -75% inverse classic pass rate and 25% changed-line count capped at 20, divides -the training set into 25% easy, 50% medium, and 25% hard buckets, and changes -the per-batch mixture through warmup, core, and hardening phases. The converter -also validates row structure, verifies that all requested IDs were found, and -rejects overlapping train and validation ID sets. The hash checks above are -the final identity gate because `data.shuffle: false` makes row order part of -the benchmark input. +`RL/tools/train-instance-ids.txt`, and `RL/tools/val-instance-ids.txt`. The +converter validates row structure, verifies that all requested IDs were +found, and rejects overlapping train and validation ID sets. The verification +script checks the resulting files against the row counts, sizes, and SHA-256 +identities above. ### R2E-Gym task containers -The qualified lists under `RL/tools` contain 685 training IDs and 256 -validation IDs, for 941 task environments used by the benchmark. The SIF -builder's committed manifest is a 977-environment superset: it also contains -the 36 training environments filtered out by the qualified training list. -Those extra images are never selected by the generated JSONL. The helper below -builds all 977 Arm Docker images, pushes them to a user-supplied registry, and -converts them to SIF: +The qualified dataset contains 700 training tasks and 251 validation tasks, +for 951 task environments used by the benchmark. The SIF builder's committed +manifest contains those 951 environments. + +Download the SIF bundle for the target host architecture from +[MLCommons storage](https://training.mlcommons-storage.org/index.html#qwen3-5-397b-benchmark). +The qualified Arm reference uses the `aarch64` bundle: ```bash -cd llm_moe_grpo/RL +sif_package_dir="/r2egym-sifs-aarch64" + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${sif_package_dir}" \ + https://training.mlcommons-storage.org/metadata/r2egym-sifs-aarch64.uri + +export NEMO_GYM_SWE_SIF_DIR="${sif_package_dir}/images" +``` + +For an x86_64 host, use +`https://training.mlcommons-storage.org/metadata/r2egym-sifs-x86_64.uri` +instead. Both packages store the SIF files under `images/`. + +#### Optionally, build the Arm SIF files from source + +The helper below builds all 951 Arm Docker images, pushes them to a +user-supplied registry, and converts them to SIF: + +```bash +cd llm_post_training/RL export DOCKER_REGISTRY="" export DOCKER_USER="" @@ -326,7 +378,7 @@ installation requires them. ## Run and time the benchmark -Run the submission wrapper from the `llm_moe_grpo/RL` directory. Supply the +Run the submission wrapper from the `llm_post_training/RL` directory. Supply the container, external data configuration, shared result root, and Slurm routing: ```bash @@ -436,11 +488,11 @@ The task data originates from [`R2E-Gym/R2E-Gym-Subset`](https://huggingface.co/datasets/R2E-Gym/R2E-Gym-Subset). The fixed easy split retains tasks whose environments can be built for the target architecture and whose source changes satisfy the easy-subset -selection. The qualified instance-ID files under `RL/tools` define disjoint -685-task training and 256-task validation sets. +selection. The qualified MLCommons package defines disjoint 700-task training +and 251-task validation sets. -The benchmark trains on the fixed 685-row curriculum-v2 artifact identified in -Section 2. It validates on all 256 held-out tasks. Runtime data shuffling is +The benchmark trains on the fixed 700-row artifact identified in Section 2. It +validates on all 251 held-out tasks. Runtime data shuffling is disabled, so training row order is preserved. ## NeMo Gym and OpenHands configuration @@ -487,8 +539,10 @@ Relevant environment settings are: | Validation concurrency | 256 | | Training agent timeout | 1,800 seconds | | Validation agent timeout | 1,800 seconds | -| Training test timeout | 300 seconds | -| Validation test timeout | 180 seconds | +| Training test timeout | 60 seconds | +| Validation test timeout | 60 seconds | +| Training command timeout | 60 seconds | +| Validation command timeout | 60 seconds | | Task runtime | Apptainer/Singularity SIF | | Model reasoning parser | `qwen3` | | Tool parser | `qwen3_xml` | @@ -566,7 +620,7 @@ The pinned runtime identities are: | Component | Version | Revision or source | |---|---|---| -| NeMo-RL | source reports `0.6.0` | `3fca04c9b313d313302923a5bb6b0c8dc0340ed6` | +| NeMo-RL | source reports `0.6.0` | `f987c0596af2bd6cfb9958e768deb8df951d7504` | | NeMo Gym | `0.4.0rc0` | `610a08ab5fe9f8f5fb5fff36b170429ea67f0f92` | | Megatron Bridge | `0.6.0` | `554c7b9324225aa863eee52e8b8fdde7abced2b1` | | Megatron Core | `0.19.0` | `002255075c3728fded9a2e435677840b08560d55` | @@ -662,7 +716,7 @@ measured run. ## Quality metric -The quality metric is observed grouped `pass@4` on the 256-task validation +The quality metric is observed grouped `pass@4` on the 251-task validation set. Each task is evaluated with four independently sampled trajectories. A task passes when at least one of its four rewards is positive: @@ -670,7 +724,7 @@ task passes when at least one of its four rewards is positive: pass@4 = mean_over_tasks(any(reward[task, generation] > 0)) ``` -One complete validation therefore executes 1,024 agent trajectories. NeMo-RL +One complete validation therefore executes 1,004 agent trajectories. NeMo-RL also reports all-four-pass `pass^4` and average pass@1 across the four trajectories as diagnostics, but the MLPerf `eval_accuracy` and convergence decision use `pass@4`. @@ -683,8 +737,8 @@ The quality target is: pass@4 >= 0.69 ``` -On a 256-task validation set, the first representable score meeting this -threshold is `177 / 256 = 0.69140625`: at least 177 held-out tasks must be +On a 251-task validation set, the first representable score meeting this +threshold is `174 / 251 = 0.6932270916`: at least 174 held-out tasks must be solved by one or more of their four attempts. The checked-in RCP logs provide six independent seeds at each qualified GBS: @@ -695,10 +749,9 @@ The checked-in RCP logs provide six independent seeds at each qualified GBS: | 512 | `rcp_logs/512/seed_*.out` | 6 / 6 | Step 10 | 0.6953125 - 0.73046875 | | 1024 | `rcp_logs/1024/seed_*.out` | 6 / 6 | Step 7 | 0.70703125 - 0.75000000 | -All 18 runs therefore reach the target at the first scheduled validation. -This is stable observed convergence at the configured evaluation schedule; it -does not imply zero latent crossing-step variance before the first -observation. +All 18 runs reach the target at the first scheduled validation. This is stable +observed convergence at the configured evaluation schedule; it does not imply +zero latent crossing-step variance before the first observation. When an evaluation reaches the target, the MLPerf logger emits a successful `run_stop` and terminates training. The submission wrapper must receive @@ -713,16 +766,35 @@ qualified time-to-target run. | 512 | No | 10 | Every step | No | 20 | | 1024 | No | 7 | Every step | No | 10 | -The validation dataloader uses the entire 256-task validation JSONL at every +The validation dataloader uses the entire 251-task validation JSONL at every evaluation. The four trajectories for each task use validation temperature `0.1` and top-p `0.95`. -Each validation executes 1,024 complete agent trajectories, so evaluation is +Each validation executes 1,004 complete agent trajectories, so evaluation is substantially more expensive than a conventional forward-only validation pass. The schedule delays the first evaluation to the GBS-specific RCP convergence window, then evaluates every step so a later crossing is detected within one additional training step. +## Offline evaluation + +The benchmark rules require offline evaluation for this benchmark: +`qwen35_397b_grpo` does not validate inline. Training must save a checkpoint +after every step from the table's first-evaluation step until training stops, +each checkpoint must record the timestamp of its latest weight update, and the +saved checkpoints are evaluated after the run until one reaches the target +accuracy. The run's `run_stop` event is emitted with the passing checkpoint's +weight-update timestamp, so checkpoint writing and evaluation time are +excluded from the measured time. + +With `DEFERRED_OFFLINE_EVAL=1` (see `RL/docker/mlperf/README.md`), the +reference implements this directly: training runs without inline validation +and saves a weights-only checkpoint after every step from the table's +first-evaluation step through the final step, recording each checkpoint's +weight-update end time in its metadata. After training stops, the launcher +evaluates the saved checkpoints in the same allocation, in step order, +stopping at the first checkpoint that reaches the target. + # 6. Approximate runtime The reference results were produced on NVIDIA GB300 NVL72 racks, using 64 diff --git a/llm_post_training/RL b/llm_post_training/RL index 3fca04c9b..f987c0596 160000 --- a/llm_post_training/RL +++ b/llm_post_training/RL @@ -1 +1 @@ -Subproject commit 3fca04c9b313d313302923a5bb6b0c8dc0340ed6 +Subproject commit f987c0596af2bd6cfb9958e768deb8df951d7504 diff --git a/llm_post_training/qwen35_grpo_rcp_launch_instructions.md b/llm_post_training/qwen35_grpo_rcp_launch_instructions.md index 9a0624692..007be95fa 100644 --- a/llm_post_training/qwen35_grpo_rcp_launch_instructions.md +++ b/llm_post_training/qwen35_grpo_rcp_launch_instructions.md @@ -31,15 +31,37 @@ export NRL_SOURCE_OVERLAY=0 ## Data configuration +Download the checkpoint, Arm SIF bundle, 700-row training file, and 251-row +validation file from the +[Qwen3.5 397B benchmark page](https://training.mlcommons-storage.org/index.html#qwen3-5-397b-benchmark): + +```bash +checkpoint_dir="/qwen35-397b-a17b" +sif_package_dir="/r2egym-sifs-aarch64" +dataset_dir="/benchmark-r2e-gym-easy" + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${checkpoint_dir}" \ + https://training.mlcommons-storage.org/metadata/qwen35-397b-a17b.uri + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${sif_package_dir}" \ + https://training.mlcommons-storage.org/metadata/r2egym-sifs-aarch64.uri + +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \ + -d "${dataset_dir}" \ + https://training.mlcommons-storage.org/metadata/benchmark-r2e-gym-easy.uri +``` + Create a shell file outside the repository containing the paths and MLPerf metadata for the target installation: ```bash -export HF_CKPT_PATH="" +export HF_CKPT_PATH="" export NRL_MEGATRON_CHECKPOINT_DIR="" -export NEMO_GYM_SWE_SIF_DIR="" -export NEMO_GYM_SWE_VALIDATION_DATA_PATH="" -export QWEN35_CURRICULUM_DATA_PATH="" +export NEMO_GYM_SWE_SIF_DIR="/images" +export NEMO_GYM_SWE_VALIDATION_DATA_PATH="/benchmark_r2e_gym_easy_val.jsonl" +export QWEN35_CURRICULUM_DATA_PATH="/benchmark_r2e_gym_easy_train.jsonl" export MLPERF_SUBMITTER="" export MLPERF_STATUS="" @@ -271,6 +293,8 @@ generation: speculative decoding: disabled and not permitted train and validation concurrency: 256 train and validation agent timeout: 1800 seconds + train and validation test timeout: 60 seconds + train and validation command timeout: 60 seconds prefix caching: enabled chunked prefill: enabled max_num_batched_tokens: 16384 diff --git a/llm_post_training/verify_dataset.sh b/llm_post_training/verify_dataset.sh index e60b98e8c..d0c81252e 100755 --- a/llm_post_training/verify_dataset.sh +++ b/llm_post_training/verify_dataset.sh @@ -16,15 +16,15 @@ set -euo pipefail -readonly TRAIN_FILENAME="benchmark_r2e_gym_easy_train.filtered.curriculum-v2-classic-cycles2-seed20260710.jsonl" -readonly TRAIN_SHA256="c07bcd64ed1c558e28d091239104e38295a5e696c1d21bb0b61f0346c7eaa0f7" -readonly TRAIN_ROWS="685" -readonly TRAIN_BYTES="444421669" +readonly TRAIN_FILENAME="benchmark_r2e_gym_easy_train.jsonl" +readonly TRAIN_SHA256="e772af09599270ab16a04ccbbcef395bcb2929607f050478033a84b087a606fc" +readonly TRAIN_ROWS="700" +readonly TRAIN_BYTES="446364412" readonly VAL_FILENAME="benchmark_r2e_gym_easy_val.jsonl" -readonly VAL_SHA256="452d0e6b3c1973669334062dc24931355de51749df1ab51fc9bb71a129f7bb5c" -readonly VAL_ROWS="256" -readonly VAL_BYTES="173801096" +readonly VAL_SHA256="2d5bddf717afb4389510a7d197f26ce5eb7f610aab4b20d6af6e0150de6f2c79" +readonly VAL_ROWS="251" +readonly VAL_BYTES="172078629" usage() { cat <<'EOF'