Skip to content

Match Tunix peft_trainer_v2 performance in MaxTextTrainingEngine #6330

Match Tunix peft_trainer_v2 performance in MaxTextTrainingEngine

Match Tunix peft_trainer_v2 performance in MaxTextTrainingEngine #6330

Triggered via pull request September 1, 2026 03:52
Status Failure
Total duration 28m 59s
Artifacts 18

ci_pipeline.yml

on: pull_request
Analyze Code Changes for Test Orchestration
21s
Analyze Code Changes for Test Orchestration
Code Quality Check  /  Pre-commit Linters
48s
Code Quality Check / Pre-commit Linters
Documentation Build Check  /  Build Sphinx Docs
1m 18s
Documentation Build Check / Build Sphinx Docs
Build MaxText Package  /  Build Wheel
31s
Build MaxText Package / Build Wheel
Gate and Formalize Parameters
3s
Gate and Formalize Parameters
Jupyter Notebook Tests  /  Discover Notebooks to Run
Jupyter Notebook Tests / Discover Notebooks to Run
Matrix: CPU Posttrain Tests
Matrix: CPU Pretrain Tests
Matrix: GPU Tests
Matrix: TPU Posttrain Tests
Matrix: TPU Pretrain Tests
Matrix: TPU Pathways Unit Tests
TPU Pathways Integration Tests  /  run
TPU Pathways Integration Tests / run
Matrix: TPU7X Tests
Waiting for pending jobs
Matrix: Jupyter Notebook Tests / run
Waiting for pending jobs
Track Test Performance  /  Track Test Duration
11s
Track Test Performance / Track Test Duration
All Required Tests Passed
2s
All Required Tests Passed
Notify failed build
2s
Notify failed build
All Notebooks Passed
3s
All Notebooks Passed
Investigate failed build  /  investigate
Investigate failed build / investigate
Fit to window
Zoom out
Zoom in

Annotations

4 errors and 25 warnings
TPU Posttrain Tests (tpu-post-training-integration) / Execute Tests (1) / tpu-post-training-integration
Executing the custom container implementation failed. Please contact your self hosted runner administrator.
TPU Posttrain Tests (tpu-post-training-integration) / Execute Tests (1) / tpu-post-training-integration
Error: ScriptExecutorError when trying to execute: "Error: Job failed with exit code 1."
All Required Tests Passed
Process completed with exit code 1.
Analyze Code Changes for Test Orchestration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@11d5960a326750d5838078e36cf38b85af677262. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Code Quality Check / Pre-commit Linters
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Build MaxText Package / Build Wheel
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Pretrain Tests (cpu-unit) / Execute Tests (3) / cpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Pretrain Tests (cpu-unit) / Execute Tests (2) / cpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Posttrain Tests (tpu-post-training-integration) / Execute Tests (1) / tpu-post-training-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Posttrain Tests (tpu-post-training-integration) / Execute Tests (1) / tpu-post-training-integration
Retrying execution for ECONNREFUSED or UNAVAILABLE "Error: Error execing #!/bin/sh -l\n\ncd /__w/maxtext/maxtext && exec env \"INPUT_REF=6904b7ecc0689f91fb3161925bec9fbdb7f06791\" \"INPUT_FETCH-DEPTH=0\" \"INPUT_PERSIST-CREDENTIALS=false\" \"INPUT_REPOSITORY=AI-Hypercomputer/maxtext\" \"INPUT_TOKEN=***\" \"INPUT_SSH-KEY=\" \"INPUT_SSH-KNOWN-HOSTS=\" \"INPUT_SSH-STRICT=true\" \"INPUT_SSH-USER=git\" \"INPUT_PATH=\" \"INPUT_CLEAN=true\" \"INPUT_FILTER=\" \"INPUT_SPARSE-CHECKOUT=\" \"INPUT_SPARSE-CHECKOUT-CONE-MODE=true\" \"INPUT_FETCH-TAGS=false\" \"INPUT_SHOW-PROGRESS=true\" \"INPUT_LFS=false\" \"INPUT_SUBMODULES=false\" \"INPUT_SET-SAFE-DIRECTORY=true\" \"INPUT_GITHUB-SERVER-URL=\" \"INPUT_ALLOW-UNSAFE-PR-CHECKOUT=false\" \"GITHUB_JOB=run\" \"GITHUB_REF=refs/pull/5060/merge\" \"GITHUB_SHA=6904b7ecc0689f91fb3161925bec9fbdb7f06791\" \"GITHUB_REPOSITORY=AI-Hypercomputer/maxtext\" \"GITHUB_REPOSITORY_OWNER=AI-Hypercomputer\" \"GITHUB_REPOSITORY_OWNER_ID=181000646\" \"GITHUB_RUN_ID=33467756955\" \"GITHUB_RUN_NUMBER=6330\" \"GITHUB_RETENTION_DAYS=90\" \"GITHUB_RUN_ATTEMPT=1\" \"GITHUB_ACTOR_ID=215185133\" \"GITHUB_ACTOR=NuojCheng\" \"GITHUB_WORKFLOW=MaxText Package Tests\" \"GITHUB_HEAD_REF=chengnuojin-trainer-fix\" \"GITHUB_BASE_REF=main\" \"GITHUB_EVENT_NAME=pull_request\" \"GITHUB_SERVER_URL=https://github.com\" \"GITHUB_API_URL=https://api.github.com\" \"GITHUB_GRAPHQL_URL=https://api.github.com/graphql\" \"GITHUB_REF_NAME=5060/merge\" \"GITHUB_REF_PROTECTED=false\" \"GITHUB_REF_TYPE=branch\" \"GITHUB_WORKFLOW_REF=AI-Hypercomputer/maxtext/.github/workflows/ci_pipeline.yml@refs/pull/5060/merge\" \"GITHUB_WORKFLOW_SHA=6904b7ecc0689f91fb3161925bec9fbdb7f06791\" \"GITHUB_REPOSITORY_ID=607845880\" \"GITHUB_TRIGGERING_ACTOR=NuojCheng\" \"GITHUB_WORKSPACE=/__w/maxtext/maxtext\" \"GITHUB_ACTION=__actions_checkout\" \"GITHUB_EVENT_PATH=/github/workflow/event.json\" \"GITHUB_ACTION_REPOSITORY=actions/checkout\" \"GITHUB_ACTION_REF=fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09\" \"GITHUB_PATH=/__w/_temp/_runner_file_commands/add_path_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_ENV=/__w/_temp/_runner_file_commands/set_env_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_STEP_SUMMARY=/__w/_temp/_runner_file_commands/step_summary_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_STATE=/__w/_temp/_runner_file_commands/save_state_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_OUTPUT=/__w/_temp/_runner_file_commands/set_output_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_ARTIFACTS=/__w/_temp/_runner_file_commands/artifacts_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"GITHUB_ARTIFACTS_LIST=/__w/_temp/_runner_file_commands/artifacts_list_02e732ad-3fcd-4ca3-beac-14bd04e7c5a4\" \"RUNNER_OS=Linux\" \"RUNNER_ARCH=X64\" \"RUNNER_NAME=linux-x86-ct6e-180-4tpu-ss64f-runner-zdlc4\" \"RUNNER_ENVIRONMENT=self-hosted\" \"RUNNER_TOOL_CACHE=/__w/_tool\" \"RUNNER_TEMP=/__w/_temp\" \"RUNNER_WORKSPACE=/__w/maxtext\" \"ACTIONS_RUNTIME_URL=https://pipelinesghubeus13.actions.githubusercontent.com/ifAZyBvORBJf4huTr2IHYuErk1LmwopbBEkmaEiGPaUn249lFR/\" \"ACTIONS_RUNTIME_TOKEN=***" \"ACTIONS_CACHE_URL=https://acghubeus2.actions.githubusercontent.com/ifAZyBvORBJf4huTr2IHYuErk1LmwopbBEkmaEiGPaUn249lFR/\" \"ACTIONS_RESULTS_URL=https://results-receiver.actions.githubusercontent.com/\" \"ACTIONS_CACHE_SERVICE_V2=True\" \"ACTIONS_ORCHESTRATION_ID=d9f41777-1f94-41c3-a588-1da33bb17128.tpu-posttrain-tests.tpu-post-training-integra.execute-test-package._1.run.__default\" /__e/node24/bin/node /__w/_actions/actions/checkout/fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09/dist/index.js\n: Error: 14 UNAVAILABLE: No connection established. Last error: Error: connect ECONNREFUSED 34.118.239.84:50051".
TPU Posttrain Tests (tpu-post-training-unit) / Execute Tests (1) / tpu-post-training-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests (2) / cpu-post-training-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Posttrain Tests (cpu-post-training-integration) / Execute Tests (1) / cpu-post-training-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Pretrain Tests (cpu-unit) / Execute Tests (1) / cpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests (4) / cpu-post-training-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests (1) / cpu-post-training-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
GPU Tests (gpu-unit) / Execute Tests (1) / gpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Posttrain Tests (cpu-post-training-unit) / Execute Tests (3) / cpu-post-training-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pathways Unit Tests (1) / tpu-pathways-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Pretrain Tests (cpu-integration) / Execute Tests (1) / cpu-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pathways Integration Tests / tpu-pathways-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
GPU Tests (gpu-integration) / Execute Tests (1) / gpu-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pathways Unit Tests (2) / tpu-pathways-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pretrain Tests (tpu-unit) / Execute Tests (1) / tpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pretrain Tests (tpu-integration) / Execute Tests (1) / tpu-integration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
CPU Pretrain Tests (cpu-unit) / Execute Tests (4) / cpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
TPU Pretrain Tests (tpu-unit) / Execute Tests (2) / tpu-unit
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093, actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea, actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Track Test Performance / Track Test Duration
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0, benchmark-action/github-action-benchmark@3d3bca03e83647895ef4f911fa57de3c7a391aaf. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Artifacts

Produced during runtime
Name Size Digest
maxtext-wheel
12.6 MB
sha256:5d3a41e221d1c41fbe06e934d92a72c40545023197cbac1ba07152ff921882e8
test-results-cpu-integration-1-33467756955
1.72 KB
sha256:ccc6abe4b495eed132216b99e0302ee1874b9407cc1b4a1710544027cacf6529
test-results-cpu-post-training-integration-1-33467756955
572 Bytes
sha256:a5c9a753ca75fe863d73c35ccb4457b5d2d5e25a82a6fa01ac5f774943d50713
test-results-cpu-post-training-unit-1-33467756955
2.65 KB
sha256:7f7e435831553645c4ed6c473bb7f444b675b4a88a951074eebaff6ea120a783
test-results-cpu-post-training-unit-2-33467756955
2.61 KB
sha256:a5a3cf4a11ac3877702448ca004936e3afce6db1524caa99e199938ad1eec3c7
test-results-cpu-post-training-unit-3-33467756955
2.63 KB
sha256:a8fc439429833f1ab23e3b7d8de0726979eb1205e1c4d52f7875e879bd0fab56
test-results-cpu-post-training-unit-4-33467756955
2.88 KB
sha256:dda1884f9eaf1598be3dbf4b691c354a81bf8ead5e565e6e20402b2b077818e8
test-results-cpu-unit-1-33467756955
17.2 KB
sha256:78ae120694ed97033c5e599563d9c71cf509eaad5f8db2e2eb51927f507ed1a9
test-results-cpu-unit-2-33467756955
12.6 KB
sha256:8b39c893454d1d8b6c7de75cb67328c8d6140861f602a44959ab1f34cd266afa
test-results-cpu-unit-3-33467756955
4.38 KB
sha256:b1b0339cecd49d375b01ecac50c31929f762c33a91a8e9956a926180d13073e9
test-results-cpu-unit-4-33467756955
12.4 KB
sha256:1528176c2cb64f652102b8b88b86f85fe1a3a558bee48212e06e73e0363dbc59
test-results-gpu-integration-1-33467756955
1.42 KB
sha256:95e23a3ff3b431a42999103e971b5bfd14b2b6e6b5ebb9d8b923f0e07edd03c3
test-results-gpu-unit-1-33467756955
957 Bytes
sha256:16253e951415cb45a6ec9bcd16101dd775b4c83dd40dc5480c5817f586cae08a
test-results-tpu-integration-1-33467756955
2.33 KB
sha256:73b50ef5a8aa41f98bec7e50ede16ae394701fe7dbaea6099dde0e3b4a243710
test-results-tpu-post-training-integration-1-33467756955
1.68 KB
sha256:a99bc203a877186af72f37a0d8cd26ed93c3ee723b38fed38e079caa66e39ddd
test-results-tpu-post-training-unit-1-33467756955
1.96 KB
sha256:378cbec723017ad1165364d17235a4e55c791dafea2c8124161e93d6a5b0f0f9
test-results-tpu-unit-1-33467756955
2.32 KB
sha256:6095781a68acf8e7a0bdb6ada42e62425b35f2c69ed6645bae7085411178e437
test-results-tpu-unit-2-33467756955
2.56 KB
sha256:7753605bef317054fd1901330721580cba04226ca3934392efd8442634d422af