Skip to content

fix(recipe): partition GLM-4.5 pipeline layers #20354

fix(recipe): partition GLM-4.5 pipeline layers

fix(recipe): partition GLM-4.5 pipeline layers #20354

Triggered via push August 13, 2026 08:26
Status Success
Total duration 1h 32m 18s
Artifacts 41

cicd-main.yml

on: push
pre-flight  /  pre-flight
15s
pre-flight / pre-flight
configure
4s
configure
Lint check
25s
Lint check
cicd-wait-in-queue
2s
cicd-wait-in-queue
cicd-compute-build-matrix
2s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
13m 12s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 9s
Launch_Unit_Tests_Diffusion
generate-test-matrix
6s
generate-test-matrix
generate-gb200-test-matrix
7s
generate-gb200-test-matrix
Launch_Import_Check
1m 21s
Launch_Import_Check
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
13s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Aug 13, 2026
test

Automatically approved by queue manager

Annotations

82 warnings and 39 notices
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Coverage (unit-test)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Coverage (e2e)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_training_callbacks — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L0_Launch_training_decentralized_pg — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L0_Launch_training_distributed_save — PASSED
Result
✅ L0_Launch_training_local_checkpointing — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_training_finetune — PASSED
Result
✅ L0_Launch_training_tensor_inspect — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_resiliency — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L0_Launch_training_pretrain — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-31682088009-07788926-1802-48b3-97e7-52b3057f87f3
366 KB
sha256:a87fd9d60ab87fe9b207df17b94afc0ef979f738283b2f2b6757a4ac4d2eafd2
coverage-e2e-31682088009-0d79be26-fa6d-425b-aa23-00ebcc003d0a
418 KB
sha256:4b4661c533ee9b5e69853c5ad2618640c80bf0f28a7b031ccb0948b281f2f2ea
coverage-e2e-31682088009-0db47846-4483-4e7b-ab9f-515dfb70a301
842 KB
sha256:7ea1f29db0bcae93be2e1d50caf27f46e04bbc07409abb428aea2785daf84595
coverage-e2e-31682088009-19ee3d56-17f6-48bf-884b-db3fe40a1ded
366 KB
sha256:99718c87390e0913e1980c4ea8b0104963e504b4822ef928e16c88443d382508
coverage-e2e-31682088009-465a6dce-8746-45cf-b197-072acafeb1ad
317 KB
sha256:8816f8283f70c3e0b240fc45c18af191a44aedee35158950e5e496c97590fb89
coverage-e2e-31682088009-4965ae30-b3bb-46a6-a370-118f0b43fde1
315 KB
sha256:eb510937f2bb151637724fd7fc38fb8e75a9b5e591d51f13adc648ccb8eac0b9
coverage-e2e-31682088009-49d3532f-6f53-49dd-a21f-8980ec928da1
405 KB
sha256:c5b66cf46dd5590a6f00efc61b6f7ef4e3c3210d799a352cf895a9a1b6a1a0b6
coverage-e2e-31682088009-4a958a62-ddea-4a6c-a3d5-49824929ec1d
844 KB
sha256:920859faf0d129aeea108c12c6116b162c0e6128dd8862d9c0a24f86714ea212
coverage-e2e-31682088009-4abb98c8-2e7e-476a-b2d9-7c95db1bc365
851 KB
sha256:9b808492713a8f6367725c4ab23d7023f3c4974b04675a2c96996644401f5e3b
coverage-e2e-31682088009-5ad6c679-7c7a-46c5-8ec5-b66bb23340e3
488 KB
sha256:b67d367ddb8b0d7d18820534c062474e0c398de5d7b9e37f1f7f92efb96b2247
coverage-e2e-31682088009-5e7fad81-c1e6-438a-b0cd-ed2d2e7a5694
346 KB
sha256:194afc18fd54e72d6b51814c64a828721ef9d7b1663804fe3f53e01bfa1e2991
coverage-e2e-31682088009-5e950740-d993-4263-b730-ad647f477822
788 KB
sha256:b85963ba73352581533f26e80f3ccaf2299887ade754ced08a0742bda3dbb572
coverage-e2e-31682088009-61295f83-c182-405c-9d78-a9813703423b
376 KB
sha256:98da744b4f2fa5abf2e46ef7d72b113a1de7262eda9035cb39e2125ba7372414
coverage-e2e-31682088009-61296fbd-d88f-49cf-8ae2-5d29bc5ea6ac
315 KB
sha256:9e57d6d51b4643e81c0be3fb543ee044bbf98947d0f36ef3ae97e43e1d137272
coverage-e2e-31682088009-6bb06a8f-2dcc-4122-b5b0-cacbf9f72c33
373 KB
sha256:a98c951123f1d6a3457e9e8dbeb75b41df69ebbceac3b7ba26ff6f5dbbd50f7a
coverage-e2e-31682088009-7090ba6a-7159-4e27-81db-a109fc865fcb
275 KB
sha256:1798a6bf3ffb8d1d974d29814cfa56bdb5f28ab51187439424781187e1acb845
coverage-e2e-31682088009-81597872-5f72-49a5-b141-58adddd67b12
386 KB
sha256:8cb6aae29c9e9c4ec5b5ae835897762c5cbb82f172f92eec5eb3a9873a8043ed
coverage-e2e-31682088009-8557b6cd-7073-4a4d-81d4-509723ada2a3
418 KB
sha256:745fce68d078eb7ad1d0f6ac5a088ba0146332403de44a2b71fbd0531e565096
coverage-e2e-31682088009-888d5039-ce07-4a92-b557-2d25b2064673
844 KB
sha256:3a74473125f2129ea7da6e7199cafae481741c88be096180fc8de9442657bcbf
coverage-e2e-31682088009-99bb72dc-9072-4215-b0b6-d568ee68bc0a
790 KB
sha256:3443320f59e8326dfd6081741ae5c241e6480ac4a35932b6b73475a1c46900a3
coverage-e2e-31682088009-9b527097-a7dd-4734-93e1-4126179f6659
346 KB
sha256:ed4daf90c4055180601883691e0fcae92b3b19c5cbf5d6e4affe29d0bd5ca426
coverage-e2e-31682088009-a27ae7db-419f-467b-bfe7-8ca1241ee31a
770 KB
sha256:b545da5d0ecaea14ece0ad624ebe7613702b3bf98205dd9367c1b20f8286d795
coverage-e2e-31682088009-a57462f9-6d8d-4ab0-83d3-9d15ce806806
770 KB
sha256:1a16c9e7e2f5e5164f62eef23070a1d48de4b7d5d5d7d6f8bb78325361e8bd00
coverage-e2e-31682088009-a602a6f4-b541-426c-8100-160c0172f307
309 KB
sha256:0047c7eed69f6d682c7fb508140732ec5dd1eed0064d5046ddef45f1184dcc49
coverage-e2e-31682088009-a6c3a7a5-9dae-4c1f-bc44-15ed0193c2b0
376 KB
sha256:005e4847e1bc3b7318b2e70fc7f7e55d079209e2d269171b3096226cd5bcc563
coverage-e2e-31682088009-ab92ff9e-4ecb-4bf8-ad1d-d196bcbc461c
386 KB
sha256:2fd89c3e6d6ffb9ea81fbae3f9896749eb102e2243c552f4f8728f05eaf56592
coverage-e2e-31682088009-b011a5e0-6e37-4953-a165-27d6f4ee09a7
373 KB
sha256:b2ac45afcb3deec735ef08921334bb05a387fefbd0d58954d9527dd5f95903e1
coverage-e2e-31682088009-b746fce6-0fdd-4f47-9d8f-ba5ffae5d101
547 KB
sha256:d75670a31a935457779f288ba82d1c5acbe333ce5abdff19c9a96a38e4c80195
coverage-e2e-31682088009-b7f2b68d-fc88-4b06-a54c-a334a12d69bb
769 KB
sha256:8366782738f820a9cf130217db0158735646b049a028ae0be7e0767af13d09c7
coverage-e2e-31682088009-b8334a8a-a550-4714-bde5-76b413857765
403 KB
sha256:5a5427c0eba9562bff97fff143aefc36cedc8394b80bdee2c156be0573f195f6
coverage-e2e-31682088009-b8354be8-f5f1-44c0-a52a-98ec6bb04434
275 KB
sha256:1682a2c4afdab41e14c430d1d869fb82101cff41136dbdf37961751577dc228d
coverage-e2e-31682088009-b984d994-6c1f-4b7a-8f0c-49173c02f199
770 KB
sha256:c90c7c20a16cfaa7f6d6bc461c583e04c13d484ae742e567f91574657ad4d06a
coverage-e2e-31682088009-bc818001-79d4-4a2c-a022-9bb56b025d87
772 KB
sha256:aba20ffa11e78135a5439c7a84cb97384e119dc7daa5b21c6683d0fcf471b2c6
coverage-e2e-31682088009-bdc38b04-77fc-40bf-be4a-8cd38db22196
344 KB
sha256:e571a50121fa9f98b7e358058bb675e51bfbc983f618c749bd2bc90577cd7707
coverage-e2e-31682088009-e7259e4d-b5db-4b10-b2b7-5dfc17566051
1.29 KB
sha256:8a1025bb43ef5111bb361ee93252a905f7d777a2cd08e7c5d571f01ba55981dd
coverage-e2e-31682088009-ebb91a2f-4535-446b-a88c-caedcc1bb298
275 KB
sha256:9577b46c3144740870ee148834c74f4fb85605a06939e473f77eac4d4dceafa7
coverage-e2e-31682088009-ed398ab9-c29f-4d86-b85b-3ca2c03d4107
790 KB
sha256:76b767e9d9cddf30b4b4a1fc00e5ce6881d23ba6cd6272e41b7405d495a24724
coverage-e2e-aggregated
154 KB
sha256:84fcc271962b662575073ab9323ca9999943c4d5cee0ffb1a22e0a176429db9d
coverage-unit-test-31682088009-7532952a-350d-49d5-92fc-6ab950aafd09
912 KB
sha256:818e56e0d0778ab0d0143e313c20eb92aaa13f5d6ea507f69ba77dd4629a043e
coverage-unit-test-31682088009-a0575c54-648b-42fd-aa0a-5216ff923aac
372 KB
sha256:8788a418038f4f5a172864a22f74176a5d654ebee64ed09602c28133860bae8a
coverage-unit-test-aggregated
158 KB
sha256:34b3248b6fcb97d5cf16052c81065fb0a4a2f04d7504115e74fd0e9f039f34fb