Skip to content

CICD NeMo

CICD NeMo #20334

Manually triggered August 13, 2026 01:26
Status Failure
Total duration 2h 14m 53s
Artifacts 81

cicd-main.yml

on: workflow_dispatch
pre-flight  /  pre-flight
12s
pre-flight / pre-flight
configure
4s
configure
Lint check
30s
Lint check
cicd-wait-in-queue
3s
cicd-wait-in-queue
cicd-compute-build-matrix
2s
cicd-compute-build-matrix
Matrix: cicd-container-build
Launch_Unit_Tests_Core
12m 12s
Launch_Unit_Tests_Core
Launch_Unit_Tests_Diffusion
3m 30s
Launch_Unit_Tests_Diffusion
generate-test-matrix
8s
generate-test-matrix
generate-gb200-test-matrix
8s
generate-gb200-test-matrix
Launch_Import_Check
27s
Launch_Import_Check
Matrix: cicd-functional-tests-l0
Matrix: cicd-functional-tests-l1
Matrix: cicd-functional-tests-l2
Matrix: cicd-functional-tests-flaky
Matrix: cicd-functional-tests-gb200-l0
Matrix: cicd-functional-tests-gb200-l1
Matrix: cicd-functional-tests-gb200-l2
Matrix: cicd-functional-tests-gb200-flaky
Nemo_CICD_Test
16s
Nemo_CICD_Test
Coverage_Fake
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
svcnvidia-nemo-ci
approved Aug 13, 2026
test

Automatically approved by queue manager

Annotations

4 errors, 166 warnings, and 81 notices
L0_Launch_models_qwen3
Process completed with exit code 1.
Result
❌ L0_Launch_models_qwen3 — FAILED (exit 1)
L0_Launch_models_qwen3
Process completed with exit code 1.
Nemo_CICD_Test
Process completed with exit code 1.
cicd-container-build (gcp, us-east4-docker.pkg.dev/nv-projdgxchipp-20260113193621/megatron-bridge...
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
cicd-container-build (aws, 766267172432.dkr.ecr.us-east-1.amazonaws.com, nemo-ci-aws-gpu-x2)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25, docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Diffusion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Launch_Unit_Tests_Core
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen35_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen35_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen3_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen3_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_llama_3b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_llama_3b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_stepfun
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_stepfun
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_gemma3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_gemma3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_glm
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_glm
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_kimi_k25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_kimi_k25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_recipes_deepseek_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen3_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen3_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_models_qwen2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_qwen
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_qwen
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen3_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen3_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_vlm_inference
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_vlm_inference
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen3_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen3_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_megatron_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_gemma
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_glm
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_glm
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_llada15
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_llada15
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_kimi_k25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_kimi_k25_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_callbacks
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_generate
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen35_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen35_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter_fsdp
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_perf_config
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_qwen_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
gb200_L1_Launch_recipes_qwen_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_glm_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_gemma3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_gemma3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_decentralized_pg
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_converter
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_llama_1b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_deepseek
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_qwen2
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_recipes_gemma4_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_gpt_oss
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_stepfun
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_models_stepfun
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_flux
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_data
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_ministral3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_models_qwen3
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_llama_3b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_llama_3b
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_distributed_save
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_nemotron_omni
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_qwen
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_qwen
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_vlm_inference
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_vlm_inference
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_local_checkpointing
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_megatron_mimo
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_nemotronh
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_nemotronh
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_pretrain
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_resiliency
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_finetune
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_utils
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_training_tensor_inspect
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L0_Launch_wan_ckpt_conversion
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_qwen_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
L1_Launch_recipes_qwen_vl
Input 'app-id' has been deprecated with message: Use 'client-id' instead.
Result
✅ Launch_Unit_Tests_Diffusion — PASSED
Result
✅ Launch_Unit_Tests_Core — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L1_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L1_Launch_models_qwen35_vl — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L1_Launch_models_qwen25_vl — PASSED
Result
✅ L0_Launch_recipes_gemma — PASSED
Result
✅ L1_Launch_models_qwen3_omni — PASSED
Result
✅ L1_Launch_recipes_llama_3b — PASSED
Result
✅ L1_Launch_models_ministral3 — PASSED
Result
✅ L1_Launch_models_stepfun — PASSED
Result
✅ L1_Launch_models_glm_vl — PASSED
Result
✅ L1_Launch_models_gemma3 — PASSED
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
Result
✅ L1_Launch_models_glm — PASSED
Result
✅ L1_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_recipes_deepseek_fsdp — PASSED
Result
✅ L1_Launch_models_qwen3_vl — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L1_Launch_recipes_ministral3 — PASSED
Result
✅ L1_Launch_models_gpt_oss — PASSED
Result
✅ L1_Launch_models_qwen2 — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L1_Launch_recipes_qwen — PASSED
Result
✅ L1_Launch_recipes_nemotron_omni — PASSED
Result
✅ L1_Launch_models_qwen3_omni — PASSED
Result
✅ L1_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_models_qwen3 — PASSED
Result
✅ L1_Launch_models_gpt_oss — PASSED
Result
✅ L1_Launch_recipes_glm_vl — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L1_Launch_models_qwen3_vl — PASSED
Result
✅ L0_Launch_megatron_fsdp — PASSED
Result
✅ L1_Launch_recipes_gemma — PASSED
Result
✅ L1_Launch_models_glm — PASSED
Result
✅ L1_Launch_models_qwen25_vl — PASSED
Result
✅ L0_Launch_models_nemotron_omni — PASSED
Result
✅ L1_Launch_models_llada15 — PASSED
Result
✅ L1_Launch_models_kimi_k25_vl — PASSED
Result
✅ L0_Launch_training_callbacks — PASSED
Result
✅ L0_Launch_converter_generate — PASSED
Result
✅ L1_Launch_models_qwen35_vl — PASSED
Result
✅ L0_Launch_converter_fsdp — PASSED
Result
✅ L0_Launch_recipes_perf_config — PASSED
Result
✅ L0_Launch_models_deepseek — PASSED
Result
✅ L1_Launch_recipes_qwen_vl — PASSED
Result
✅ L1_Launch_models_glm_vl — PASSED
Result
✅ L1_Launch_models_gemma3 — PASSED
Result
✅ L0_Launch_training_decentralized_pg — PASSED
Result
✅ L0_Launch_converter — PASSED
Result
✅ L0_Launch_recipes_llama_1b — PASSED
Result
✅ L0_Launch_recipes_deepseek — PASSED
Result
✅ L1_Launch_models_ministral3 — PASSED
Result
✅ L1_Launch_models_qwen2 — PASSED
Result
✅ L0_Launch_models_gemma4_vl — PASSED
Result
✅ L0_Launch_recipes_gemma4_vl — PASSED
Result
✅ L1_Launch_recipes_gpt_oss — PASSED
Result
✅ L1_Launch_models_stepfun — PASSED
Result
✅ L0_Launch_models_flux — PASSED
Result
✅ L0_Launch_data — PASSED
Result
✅ L1_Launch_recipes_ministral3 — PASSED
Result
✅ L1_Launch_recipes_llama_3b — PASSED
Result
✅ L0_Launch_training_distributed_save — PASSED
Result
✅ L1_Launch_recipes_nemotron_omni — PASSED
Result
✅ L1_Launch_recipes_qwen — PASSED
Result
✅ L1_Launch_vlm_inference — PASSED
Result
✅ L0_Launch_training_local_checkpointing — PASSED
Result
✅ L0_Launch_training_megatron_mimo — PASSED
Result
✅ L1_Launch_recipes_nemotronh — PASSED
Result
✅ L0_Launch_training_pretrain — PASSED
Result
✅ L0_Launch_training_resiliency — PASSED
Result
✅ L0_Launch_training_finetune — PASSED
Result
✅ L0_Launch_utils — PASSED
Result
✅ L0_Launch_training_tensor_inspect — PASSED
Result
✅ L0_Launch_wan_ckpt_conversion — PASSED
Result
✅ L1_Launch_recipes_qwen_vl — PASSED

Artifacts

Produced during runtime
Name Size Digest
coverage-e2e-31657694462-0bbe2040-fc4c-4455-bbfa-68f646d87e48
772 KB
sha256:ebb0045d396b642da365b794341c392b9a6dcc2c9ab80ab78cce372761d1940e
coverage-e2e-31657694462-1e872980-e686-4266-87f9-41a05548b569
275 KB
sha256:8a8d889ff92e21a11b796ac4993f0b28383bbfd5efe34f78d690bfbd65e2365b
coverage-e2e-31657694462-259c237e-112c-4d88-a874-1e41d606c9c3
344 KB
sha256:27c80ea8ceca43e5835db575fbbe52454ba7e2a3a09f93ee75efdb2f6371bd75
coverage-e2e-31657694462-260803a3-fce8-47c3-827e-1cfab2b9750d
274 KB
sha256:1950371755060b702559f75f2203e2b42eb3dc409fb5f02c908d405d72b7aacc
coverage-e2e-31657694462-2a0ff478-1c9a-45ae-83cd-2e1265982db2
844 KB
sha256:8ede3444fb275108ffdb1407bdf701cfb52b532f6cf40b3e4911a105dac9e2f4
coverage-e2e-31657694462-2fa02a6d-54ea-4bdb-ba70-2c1346356fc9
324 KB
sha256:7c32dd11171d446d8d2c98adf41e1e35c95f8e928f321867293b3a9880c6a13b
coverage-e2e-31657694462-31d97dfc-a93b-42aa-83fd-60d88bcf28d0
769 KB
sha256:3711dc85c6e46d45072bacadc9fcd637fa01469325c18ccab61f2b91aa456f68
coverage-e2e-31657694462-3bc7f22a-604c-48d3-9b71-e84c44d73038
843 KB
sha256:fbd5157e2c9a61aa4901afdc1ff30df421186155d81c7b5f319cc8441ea4923b
coverage-e2e-31657694462-3d9eeff5-6dab-411c-99b1-bd79b1d42af6
375 KB
sha256:ffad479069b8b4a81dcf0b566daaacb54c931abd46320023bd0292fbc0f12c0d
coverage-e2e-31657694462-3df6fa65-b098-4e13-9a6d-dff71cdbbacf
844 KB
sha256:d0491e083aa99211c0db0eedb0bc3d75179242460bd7bf902991f0abe1612a58
coverage-e2e-31657694462-43395ebe-2f7b-446b-b20b-6499b1480a1d
769 KB
sha256:3f824ae2c5f2355237a6f117e6e13266e5f97d8bfe322728a8cb0e8891e091cf
coverage-e2e-31657694462-44d4a1ac-2e42-4127-ae84-b239af0ebdf5
274 KB
sha256:51e7756aabcf07874b232019b37dfc93447c75148ed9475cf79f34fbcd5c2b04
coverage-e2e-31657694462-46049426-8bfc-46ab-aa1d-51f4fc4fb271
376 KB
sha256:6a93ed787b0671699e6d14e06eb11e32f2c3337fb8869aba67a3c2aa99baa2b7
coverage-e2e-31657694462-484219de-591b-49c8-abb3-240a6848aeee
422 KB
sha256:8c2641c1f312b9d4f829d0d9f0afebe34bbc7a528ab6f196873f621c9e8cd874
coverage-e2e-31657694462-4d23ddcd-c03b-466c-9bcc-f824e5fd242e
274 KB
sha256:aabd7faf3e2a2680273c46ada233a92c2c17d839c13b2254fbf2ce1cc88244ce
coverage-e2e-31657694462-5054f8ca-f721-4e7e-891b-ac446c2573e8
277 KB
sha256:fe5e78e4784a04703447d4aa8b74eb3e6ea3a2f3d09920f201409110ac1c0e3f
coverage-e2e-31657694462-50f32127-94d9-43a0-86be-994fa527f329
319 KB
sha256:d4c54218167cb560ff6ef08264d5c4406d6611ac88440cdd28b8a1216525e390
coverage-e2e-31657694462-52335deb-39a0-4d5a-b9c0-75f94246b290
275 KB
sha256:a5f47c68f1c4748d48a682f011775a2266a67876bae2ebf109a3963c900c58b0
coverage-e2e-31657694462-575bf12a-7b29-4b1a-9fea-fe0b917e58c5
309 KB
sha256:fad6333a200d6497b6414d4426f4233259cfc6675dae5659fa2640b6e062b5cb
coverage-e2e-31657694462-5a73b6ba-8e4a-4b35-9f1e-51781ad9b02a
405 KB
sha256:0d70554b0a14d7c68461bf85be124b5c9f589dd5ee23e0628c79666696107e36
coverage-e2e-31657694462-5e7686c0-f251-406c-a8dc-dc1584ea79aa
844 KB
sha256:d59656128224f99a5071f5b3e327b7c9a3b0d19d990460f376d40ecac659f2b0
coverage-e2e-31657694462-614fbb2f-5879-4f61-b78c-ac6bc40293b2
849 KB
sha256:01c75bc2377782a3678ecc952369d419223ba7e7d75493cc71ebaaaff007df2a
coverage-e2e-31657694462-63acb4d1-bf20-4e87-a661-c53050e2d51e
327 KB
sha256:0f5dd7b22e2ecfd6b87463b2cd54cda8ac90bafebb97009af0b31fbf168b19ff
coverage-e2e-31657694462-642147da-17de-45f7-8d04-e5ca8b05aff1
772 KB
sha256:cca37d1a82a331a762c204d891c9c2c5e8f29e3493057494f7614b4b36e9d414
coverage-e2e-31657694462-6490e382-413a-44a0-9bb2-7288b803357d
275 KB
sha256:bc0e16b93e1a3f6712cd2499f6fc64d9e5721d276ea01f190f09a55cd2160984
coverage-e2e-31657694462-676ace21-1015-454c-9005-eb9f40486233
315 KB
sha256:c40509ab38287d9a4cb728348089a2b58249e7319452518149dcec1b4628664c
coverage-e2e-31657694462-68a7d4f5-700b-4d0e-9a8c-ca6120b6d874
1.29 KB
sha256:97c3d48b88d32d81e050c8e4ab26677a0b5630ba7bfa1a30ac873c11258db58a
coverage-e2e-31657694462-68f685d9-500d-4b53-9835-f988029c0d69
379 KB
sha256:a0cd10b92a9b944fbb8d9efe3b3d1801eaf71b8159cebd8a67d5675699fb7fec
coverage-e2e-31657694462-69c467fa-77ee-43d9-99bd-038f5f8702f3
851 KB
sha256:e1c549066020e54ffb325b5a7c6a4c649fe35026d0bf083b4274e848d64e2025
coverage-e2e-31657694462-6af81a57-cc6b-42d6-a9f6-2da7d09dbcec
772 KB
sha256:c5d2d66d71b2939732245d6821263523fd4167fda20c5f385cb2ef81dc2ffa25
coverage-e2e-31657694462-7827b224-22f6-4ac8-b884-f6575356aa03
420 KB
sha256:7f54ddf57e973267f97f39dbfa88b14624fd9328cbf2e87eb9accfb1a025a255
coverage-e2e-31657694462-7d428b24-245b-4a59-b365-33c8d93e431b
278 KB
sha256:dc9763e05eba52bac767a2128b75647b2b44f521904afa9964733b2993bddf11
coverage-e2e-31657694462-7edfcb9d-63e6-4256-9d4d-ac231a39b268
842 KB
sha256:8621cd64d96a24cfb263d605998c883ab69f13e0f8042467c4abe793dccac092
coverage-e2e-31657694462-7f429ff1-50e3-48e7-9d64-63f1a622bfb8
275 KB
sha256:659bf259c25d619f8ca23c06f39c5f5375e530afbae6a28d549bf63c9a4cacae
coverage-e2e-31657694462-8604b744-9c01-4a5d-86ab-f8b3ea1a2e4a
389 KB
sha256:a5dbbae6b0a19c74b8fd5e0aba1d4970e96085522017f848ff248a19837cfb8b
coverage-e2e-31657694462-879629fa-a27b-4ecc-a83c-49d0a5da4ed5
274 KB
sha256:32ebd567bb8dae0c1ac3c3a761258892735954afaf6ddeb61f26ed42956fb699
coverage-e2e-31657694462-87be1f03-1ba1-4cec-bfea-5ab1fe082331
302 KB
sha256:c46fd16e841e0d5c3372f2f1cf1599eb722eeff9afaae91d8a7d73255d26ab66
coverage-e2e-31657694462-8e7c344f-dc5c-44a4-8a97-beb0b1f9540a
772 KB
sha256:757caecacf4c5bdd92b8d52c46452a9b8f1ebde1f957373e73bf2fdd49d664b6
coverage-e2e-31657694462-9083c1b7-540d-4f23-bed5-04ba9256b025
406 KB
sha256:d31900d9f755da624401e500db08bf7c7d1e2ade0306a0db71bae7a8b02effa3
coverage-e2e-31657694462-911a2a34-89f9-4cb0-bbfe-276cce7e58d8
774 KB
sha256:f4c47e187baa8ada9ab7e0e4db5a1f92508874d4fc78b000dc197c5b6a3111f5
coverage-e2e-31657694462-9254d608-f3a3-427f-987a-0c716be9c92b
302 KB
sha256:a990df773288f38406826bf291cf9ed0a035406c135bf664297a1b51bc6225a3
coverage-e2e-31657694462-94d88386-2592-4b5e-90d9-3310fbb077cd
844 KB
sha256:15b0e9958ae4c547704a7b8e04aed9395c703400194a381450181ba6b7c97c0b
coverage-e2e-31657694462-974e706a-d377-48f6-8ea0-2371022c49c6
839 KB
sha256:39295e7d7568e843dfb71243f04c59b5b8a77c325981f98b648c6a500f5badbb
coverage-e2e-31657694462-9783f21d-c18e-46d2-b185-cdbeb9dc894f
550 KB
sha256:4f955f8d06d37183f14fc4e261fec9a6b789c736762e0dddb32142d6e6f69361
coverage-e2e-31657694462-99200aae-d830-4b2b-932a-590f1ee1174f
373 KB
sha256:badcb5ee2efb1e71bc8bebaf50f31637f472c9db6a497db5b4e99130fd3649c7
coverage-e2e-31657694462-9d895a02-40bb-438b-93fa-d2bfd248f038
793 KB
sha256:ef0380e9bdf3de4a8e9ff2d31aa0d1f3957e5105a699d893850c12dd22a92e05
coverage-e2e-31657694462-9e68c6d2-56f3-4597-9e8f-100b8047d30e
411 KB
sha256:53e4e62c98290184e0a1e30545da846469db4e70b0ded29e18f1375c1e272a93
coverage-e2e-31657694462-a18cf14c-9dfe-49e8-97be-77480a42ec10
274 KB
sha256:7ae40cb2c5c4d5b7f3299e3166085ac32a3b38107ef59b167015c51e32fdea29
coverage-e2e-31657694462-a3959896-1389-408e-bf9e-54768eb7ec93
773 KB
sha256:87f4368f78ae53a4acb79f2b5240fb3fb9752bce54dd9880c10a038ace01e1ae
coverage-e2e-31657694462-a5529888-5743-490c-aedd-9deec117d9c1
492 KB
sha256:948940c706eb018b26c15c4a431bea5374e27136d0ec77214844d62e4c5351b5
coverage-e2e-31657694462-a5f0233c-312d-472c-a13a-97f7466b7ae4
420 KB
sha256:101f4468fb5731ac12cd6fe1aa0b4e1b6414f1d10705ec44c23fbf731bcd6436
coverage-e2e-31657694462-a826a8b8-ec97-4d8b-bffa-2a693632bf37
844 KB
sha256:7773ab31cd3f8b9e14b82c0cee48a2a6726b8b0e09bb21b6c8f674c4f44960ca
coverage-e2e-31657694462-a9e2535e-23d2-4c22-9823-b969cb286482
346 KB
sha256:c14b72c0c21e2696dbc6e98a36bf3f8c64e17512275a44fba3fb984b8a0cd6a0
coverage-e2e-31657694462-ac88d4fd-2eca-48b2-a1cf-ed1aff145574
794 KB
sha256:10f9504927258bbbce61c39c98a6456118c42a1798ac5260ee5f1a52962dcc3f
coverage-e2e-31657694462-adecd799-0f89-49f6-a122-4d8351b3cdd4
346 KB
sha256:ad825ffb504e6d7dc045ace2212074b4ab842609b14575a4c8c4f14f21c4a7c3
coverage-e2e-31657694462-b21c87fb-4373-4aa6-89db-ed7c294782a3
366 KB
sha256:e11e6d7cfa0ca29f3828185c6e743f2d90ee0815a1fd51d0de520b231dd296e2
coverage-e2e-31657694462-b32812aa-aa3d-44ee-ba8c-dc4557cf5ccf
375 KB
sha256:743b75bcfe8d6cc03d7e76fe053de38abedc099aa49aa68b9da5dba96036c379
coverage-e2e-31657694462-bbcf28c9-424e-4a6d-b61c-41f13add786c
274 KB
sha256:beaaa32e4a5c7fb6b7fcd36aa965c02989fe91cd484b0a75307b0a8ec22d8e21
coverage-e2e-31657694462-bdbd7276-66eb-4e37-8698-2fed2e00721d
771 KB
sha256:77299e4ef4dc021fe12054aa0676fb74a75ceb3f5b110c9ecf0afeb92ce83796
coverage-e2e-31657694462-c765b906-e92e-47d5-b65d-226f349e02de
842 KB
sha256:65d0b3ad7cc94276dc3341d0fbe61d0f297fd78b6bf12fc9eedcbe464356e959
coverage-e2e-31657694462-c8582026-1a5c-4a5e-9450-5113723bcd6e
274 KB
sha256:dada259f6e4a2a6cf239225a719b5ef9f3e6d216872961106e3039547bb23b36
coverage-e2e-31657694462-c963f0b6-52b7-452b-ada4-bb6a2aedbb1c
423 KB
sha256:acb115dc3b37e69bd03a2b25fc100eb28baaf73ce5bc542ee5ebb02a3efe972d
coverage-e2e-31657694462-cc20350b-bd5f-4a7e-8486-641162489b47
315 KB
sha256:dde5725c65a47dadd2620822209af24693d8aebb5eb47417047438ee5367461a
coverage-e2e-31657694462-cd5ad195-8e0a-4a94-bf08-f8c59fdff8b7
769 KB
sha256:27604a07cb714a180d91bdd0091431446a62ab1b1f4d5b4e1a75c77920421f19
coverage-e2e-31657694462-cf6b804b-64df-4e3e-bc75-fbad8635d693
271 KB
sha256:744f4773e820e23032cfd81927b146a74c657c487808e681b4a5e1e56d43a540
coverage-e2e-31657694462-d4066de0-a41c-46f6-8f32-5c8a154680bd
366 KB
sha256:1ec52032e75de1e64679b397da11eea4df97210a8e033a43aff8a4a549ee8a73
coverage-e2e-31657694462-da08e420-e885-4b64-a469-9879300e8303
315 KB
sha256:a5ee2241322418237926b5f1d5bd52f135d70ebe4272173eeddbd00e09d48f97
coverage-e2e-31657694462-dd4b419c-923d-4a58-af08-fd8b53de84e0
386 KB
sha256:a1f40fa29160c7ea3ee07d49d59876e9d8568cc0fa05c868d8803b8e5c4cf9d3
coverage-e2e-31657694462-dde7ed67-da80-4077-8929-8547ba6b01bf
788 KB
sha256:32cf39697705102fc3003d08a3883cd7d5e0892fe0b98e624fe4ca5f0ec363e0
coverage-e2e-31657694462-e076cc4f-9512-45c4-96b3-be586d740291
376 KB
sha256:dc2b7be37362a0f797134367a7d5b574a670dc7dba63e5cfa9cd942eb26336d3
coverage-e2e-31657694462-ed96f4c4-79f5-4784-aeb0-77b113a89e1e
413 KB
sha256:229919586d9c57bcaf7b83380f5cf2e631fb85a99584183db8b0c085224af8d2
coverage-e2e-31657694462-efd54a3a-fc15-42fd-a27f-9592a130fcde
276 KB
sha256:f02db9889143974eb9b22bfbca840c4967f0881e0fe03c356a766e114cd53c74
coverage-e2e-31657694462-f42db282-aaea-4c8b-a5f9-04d370947121
769 KB
sha256:5b8fc6b75a2a088711f7eae44be3fb4b7435c6141aa2f393f35bccf584839dcd
coverage-e2e-31657694462-f4805952-d53e-4ba0-8ed6-f0c9f77c13dd
315 KB
sha256:9f10265b90ceeaf238d73774fe4e279df98efa240804424e402d06a71ee8589c
coverage-e2e-31657694462-f85be950-13da-44cc-af30-d8ec48853f2c
771 KB
sha256:9218a7a146c4f99824a807785bb7a3923519dced1bfcc0e168d1f2fe7612003c
coverage-e2e-31657694462-f96348c2-fce1-4941-a158-0097d7d46a0a
274 KB
sha256:c2d5ef0abcf289f93c2efeff4681e73b17c5daa30a7fce4993059f0233a1002b
coverage-e2e-31657694462-fb3db854-a781-4a95-8c02-f2c40d34385f
420 KB
sha256:c5b33fc27acaef2a2717e96739fdc9560521ffe52f5f002b72f7f9e8123eee5c
coverage-e2e-31657694462-fcd57e78-faf6-4481-aa7c-370cd033e71b
418 KB
sha256:fc1302cd30e999dcfeb2db0f722a0c0fbbe61421937e5715d0fcea8035d3dfa4
coverage-e2e-31657694462-fe54bb67-f54a-4bd1-b6fc-9100ba7988dc
847 KB
sha256:125cbb6e36c49f6f3270f8bd68ab49a1ee4a4fb8a9770fa1031837261382a9a0
coverage-unit-test-31657694462-a4027262-5ad4-40bb-aebb-35cfbd21e395
375 KB
sha256:2d02f4e91b14c5ffc0ac2b359fbbc9a0cc137b83b3f074f6942d994bdbe2dee4
coverage-unit-test-31657694462-b6897911-b951-43fa-bf16-e5403feb64ed
912 KB
sha256:df2476564faa9593529883c5b335c09e3c04e87bd6d81485c2a9e75cb13250a9