Repository navigation
[training] fix: respect module-local MIMO loss normalization #20374
copy-pr-bot[bot]cicd-main.yml
on: push
pre-flight
/
pre-flight
19s
Matrix: cicd-container-build
Launch_Unit_Tests_Core
13m 5s
Launch_Unit_Tests_Diffusion
3m 8s
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Coverage_Fake
Matrix: Coverage
Deployment protection rules
Reviewers, timers, and other rules protecting deployments in this run
| Event | Environments |
|---|---|
svcnvidia-nemo-ci
approved
|
test |
Annotations
82 warnings and 39 notices
|
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
|
|
Coverage (unit-test)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Coverage (e2e)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
|
|
Result
✅ Launch_Unit_Tests_Core — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L0_Launch_converter_fsdp — PASSED
|
|
Result
✅ L0_Launch_models_deepseek — PASSED
|
|
Result
✅ L0_Launch_data — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L0_Launch_training_callbacks — PASSED
|
|
Result
✅ L0_Launch_megatron_fsdp — PASSED
|
|
Result
✅ L0_Launch_recipes_llama_1b — PASSED
|
|
Result
✅ L0_Launch_recipes_deepseek — PASSED
|
|
Result
✅ L0_Launch_models_nemotron_omni — PASSED
|
|
Result
✅ L0_Launch_recipes_perf_config — PASSED
|
|
Result
✅ L0_Launch_models_flux — PASSED
|
|
Result
✅ L0_Launch_models_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_training_decentralized_pg — PASSED
|
|
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L0_Launch_training_distributed_save — PASSED
|
|
Result
✅ L0_Launch_converter_generate — PASSED
|
|
Result
✅ L0_Launch_utils — PASSED
|
|
Result
✅ L0_Launch_training_local_checkpointing — PASSED
|
|
Result
✅ L0_Launch_training_megatron_mimo — PASSED
|
|
Result
✅ L0_Launch_models_qwen3 — PASSED
|
|
Result
✅ L0_Launch_training_resiliency — PASSED
|
|
Result
✅ L0_Launch_training_tensor_inspect — PASSED
|
|
Result
✅ L0_Launch_training_pretrain — PASSED
|
|
Result
✅ L0_Launch_training_finetune — PASSED
|
|
Result
✅ L0_Launch_converter — PASSED
|
|
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
coverage-e2e-31692154668-08ffbdca-750c-4318-b982-0caf52f7509e
|
790 KB |
sha256:7a80a495f6250cb37d377655099bf50956504a50f8429151f490e1c03c79c847
|
|
|
coverage-e2e-31692154668-1de47ae2-9e5e-475c-8aed-ef8dad982f47
|
547 KB |
sha256:668d2d9e245c44ce7008928cba6debdd14bf34a4a44e90622727fb657103b5e2
|
|
|
coverage-e2e-31692154668-1f20e681-3d5a-4fb9-b825-9d31a94a41af
|
844 KB |
sha256:19039eee89c90bc6644e73f24e9921ff81e1b0e186cbfa75e9678e1236531cb4
|
|
|
coverage-e2e-31692154668-26e56b78-d389-4dc4-a9de-20bf0591bcf2
|
405 KB |
sha256:4921f02ea2615d09ef6ee66f85dfc98a91879d3c6c61ca6e68a21017fec8261a
|
|
|
coverage-e2e-31692154668-2dedc3de-730c-4024-bcaa-c08a3fb8eb23
|
373 KB |
sha256:06d0324db151e2a1a829b1b7bdbf2a19f87b8ddb2150b84005db3aed8f8ea439
|
|
|
coverage-e2e-31692154668-2e271cf9-ff6e-479f-aa77-99691ff1089a
|
275 KB |
sha256:fd2d987ce436cbca19a99fc064a6520c979b39ca6bff5ee28df8008579b3b159
|
|
|
coverage-e2e-31692154668-3820e19b-5aa7-48cf-9515-661d60b868bc
|
366 KB |
sha256:e82dcc0344ddfa538b0435c3ab3ee97d50399f58dbdec0c2e8e8498af35a9664
|
|
|
coverage-e2e-31692154668-3940f3d2-d739-430c-9748-8a24c07afaca
|
851 KB |
sha256:9fd696487ec2ec989d6fa7636d7a3a322482b36169637115ce37d962a1512c2d
|
|
|
coverage-e2e-31692154668-3bbf18ee-eb30-47bb-a4d9-09e74f001cd6
|
366 KB |
sha256:22e085684e4ebea9bf0e07ba75dfea7e4a15f28ec2611c932d4db214a5a40d9c
|
|
|
coverage-e2e-31692154668-432476d7-50c6-42af-b7c1-78f556072c0f
|
344 KB |
sha256:3cedcaca6fd8a1daf630be559cf281633888f276db0e86b116a2e446a72d581c
|
|
|
coverage-e2e-31692154668-4344368a-695b-47a5-9368-eabed523a988
|
788 KB |
sha256:115e6941e6776d45dc84d5da4c30fbdc4c3940e4be054bd0395cdd6188f16d54
|
|
|
coverage-e2e-31692154668-54117403-ad79-4e52-8f24-dfcb10f3dbb7
|
376 KB |
sha256:6b889428448dad3ddd25fe71776c90c3dedc8ab53d154e824402a9768acb7e92
|
|
|
coverage-e2e-31692154668-57d60630-5a84-4eda-ad69-ff03c9df961a
|
488 KB |
sha256:0145505ae0938101590b6c4d1e4c1824f7bc8ceb9dc9e707fc32da7312ba2a13
|
|
|
coverage-e2e-31692154668-5b0ab384-346a-4123-96fc-101edc1466f7
|
346 KB |
sha256:b9b4c1e08b1ad8d9ca8a3b6e74a1ef6cc7021e0492d0a2bd4c175b680eca9505
|
|
|
coverage-e2e-31692154668-6a9fa1c3-aff7-4434-85e1-1df80dd52cc9
|
418 KB |
sha256:d1809ff67b06f3cf648251ad6a06db4121fb66ef3eadccb4d9361cfb1435fe99
|
|
|
coverage-e2e-31692154668-6c58fcc5-18a8-4d70-87c6-7c03a07ae97a
|
376 KB |
sha256:678a7d51c0d40925a1e9086f4a6573a9407017e896fa83d7eed27fe4f5db6e84
|
|
|
coverage-e2e-31692154668-7102a132-a0ff-4668-8494-65ca435f29db
|
418 KB |
sha256:c846c96c4eb91be0414744ab0b9879f3e701e5c07d49ad30bccda4be78aab05f
|
|
|
coverage-e2e-31692154668-71581c76-e8e7-4c75-a1d3-cb77dfd1b88d
|
315 KB |
sha256:a7710ea3088ac0448951b57b985ee470cb6399e40dabfb6796be5a5c216e45b4
|
|
|
coverage-e2e-31692154668-783ecdcd-4295-4d6a-9cda-3fea33aa6a15
|
373 KB |
sha256:e722b4319fb0b8b61cdaf79c8c775d2f273080f13daa6bb036b69224cfabf37a
|
|
|
coverage-e2e-31692154668-8816321d-e82b-45d2-879c-457acd4dd248
|
315 KB |
sha256:784e030ce6ac231f4c6167e71342c88dec943df0119ff19081a8440df9106d50
|
|
|
coverage-e2e-31692154668-8ab02f73-1372-4b25-bea9-25bcaf216574
|
770 KB |
sha256:e2901ecb48525a80f9e4cd85775b1c2f92ab8bb72ef91645b0c17c8b5e6b404b
|
|
|
coverage-e2e-31692154668-8b8f579f-0f95-401a-880c-72c9b20e1aa2
|
309 KB |
sha256:5970978929d6ed7163782b99ef951ef483c184ab4e708cfbb9fc26dd188d1fac
|
|
|
coverage-e2e-31692154668-8cc1282d-0758-4c85-a303-10e70da4d468
|
770 KB |
sha256:a2d11fa28484c8f01bc97d19550c9ce29fbac6b82b5f745550ffaf45508c6128
|
|
|
coverage-e2e-31692154668-8e0c37d1-18e9-41ad-9d30-77c7325c9622
|
770 KB |
sha256:04b0d73d0049a7dfaaaf18c8cad27e73362d5cc791bd33120d49f26f4b16d220
|
|
|
coverage-e2e-31692154668-968c0dce-da20-48e0-bc20-6cde1f56be30
|
769 KB |
sha256:4bd510f61acc7c12cefccdaac66302399794251347f2b6b43cfc4f8fe82666dc
|
|
|
coverage-e2e-31692154668-995ffa9d-eb9c-4216-8ea6-c73a6f1638b2
|
346 KB |
sha256:7ce399346bfb73ae61498c3f6271d92d8dd06a163f9188dd3937c33a91ca35ff
|
|
|
coverage-e2e-31692154668-b1586b24-e6e1-49bb-a646-faed1cff8d60
|
317 KB |
sha256:183c386d878b6c16301cd2bef7890b33f881613dbc2975198e5b839129e8fdea
|
|
|
coverage-e2e-31692154668-c7783105-78aa-4850-9af9-493cdbcc6a9e
|
1.29 KB |
sha256:c9e71cd1159cb9f1ab8f3ba7049bbad1f08c3e1f044d09f0d5c1b041bafe3a48
|
|
|
coverage-e2e-31692154668-d0377834-7947-4fa6-9db7-63f7674eefdb
|
275 KB |
sha256:d78b41110712ea8c4a2f553b6a719a35e9c6b34d6e2a649b14342392c4698c07
|
|
|
coverage-e2e-31692154668-d09f124c-f91f-4182-9ac2-5f87282893ed
|
790 KB |
sha256:0a843121e4c682c4d7a4790b6769479104d905b6754d3b5158d396287657185f
|
|
|
coverage-e2e-31692154668-d49c36d1-6a59-486e-a4dd-5dbca2eedd28
|
386 KB |
sha256:7ecc56cb95d6811b7518d260c1acfa419e529e13a31810fb1fab53cfa865789e
|
|
|
coverage-e2e-31692154668-d51e8036-91d5-4425-b920-392f6b76d10f
|
386 KB |
sha256:af25c580d262c12a879bfa838ad7c9043e6990150dce16021548b7c8bef9ce61
|
|
|
coverage-e2e-31692154668-d767c4c6-6701-49d0-a1a1-23bcb65851b5
|
403 KB |
sha256:0a74e235a15b401b102fde6caedd8b3ed68a69d7d1701d126833ab516f89f40b
|
|
|
coverage-e2e-31692154668-ebfd173b-f61f-4d33-8af6-0bdeac63fb21
|
275 KB |
sha256:e3eb8ebf6379213c8dd89447a52fe32933b0aba40f198981bb3c7e7b7a50d42b
|
|
|
coverage-e2e-31692154668-fd60dc51-1796-4e94-9901-493b5296e396
|
772 KB |
sha256:055cdebbec45d0b792dace3221f398b089af746a17f236236a7779e508c859a3
|
|
|
coverage-e2e-31692154668-fe2f5103-31e5-45c9-b972-a31e2a947056
|
844 KB |
sha256:bd21fd45d41fecca86202f20dec448b984594cac9dcc546a7a22bbdd8ec1c07e
|
|
|
coverage-e2e-31692154668-feec9804-ca83-4a9e-9354-34b32af3be72
|
842 KB |
sha256:15b136e651bd9b9e3ce6d369a1bbd295b442c605493b113748cf0983085f236e
|
|
|
coverage-e2e-aggregated
|
154 KB |
sha256:56c70f387ecd5551099b8afe476183ae33540aac96899e16e0cc82edb1eafaa3
|
|
|
coverage-unit-test-31692154668-048c14dd-b830-4cdb-a475-245a9565cdf6
|
912 KB |
sha256:1b1202ec388207867e61b02b2669ce5b876780d2ddba4396dcbe25abe20dc03d
|
|
|
coverage-unit-test-31692154668-f4378d2c-e642-410a-9ce8-9845973f4e5e
|
372 KB |
sha256:1aaffc56e347f4f670e0802b02d8be04344bb9e9545af9c288149803a2ae9058
|
|
|
coverage-unit-test-aggregated
|
158 KB |
sha256:d4c0df6c293fd93f32c8f28c6e192fc6dfc3d80c46aa945a04d93cf1e6e33998
|
|