ci: let a flaky download fail its own job, not the whole cluster

A transient PyPI/uv fetch failure was recorded as a cluster-wide outage, which then skipped every other job on that cluster for 20 minutes. On this branch's CI that turned one bad download into red AMD lanes whose tests had already reported 0 failed.

The breaker was justified by 17 Frontier jobs spending ~33 minutes each rediscovering an outage -- but the dependency install happens before any compute is committed, on the login node for Frontier, so a failed fetch costs no allocation and there is nothing to protect the matrix from. uv already retries internally.

Removes classify-build-failure.sh entirely: this was its only rule. Net -128 lines. The outage breaker itself stays for cases that genuinely are cluster-wide; nothing marks one for a download any more.

Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Author: sbryngelson

.github/scripts/classify-build-failure.sh

@@ -75,11 +75,6 @@ run_build_step() {
     local rc=${PIPESTATUS[0]}
     set -e
     if [ "$rc" -ne 0 ]; then
-        local cls=0
-        bash .github/scripts/classify-build-failure.sh "$log" "$job_cluster" || cls=$?
-        if [ "$cls" -ne 0 ]; then
-            exit "$cls"
-        fi
         exit "$rc"
     fi
 }

.github/workflows/common/build.sh

@@ -24,20 +24,16 @@ clean_build
 
 source .github/scripts/retry-build.sh
 
-# This login-node step is where Frontier's dependency install actually happens,
-# and so where a PyPI outage actually lands -- 17 jobs spent ~33 minutes each
-# rediscovering one on 2026-08-28. Tee the output and classify a failure so the
-# first job to hit it records it and the rest of the matrix can skip.
+# Frontier's dependency install happens here, on the login node -- so a failed
+# download costs no allocation and there is nothing to protect the matrix from.
+# A flaky PyPI fetch used to be recorded as a cluster-wide outage, which then
+# skipped every other job on that cluster: one bad download turned into a red
+# matrix, including jobs whose tests had already passed. uv already retries.
 # No set -e in this script, so capture the status rather than toggling it.
 deps_log="deps-${cluster_name}-${job_device}-${job_interface}.log"
 retry_build ./mfc.sh build --deps-only -j 8 $build_opts 2>&1 | tee "$deps_log"
 deps_rc=${PIPESTATUS[0]}
 
 if [ "$deps_rc" -ne 0 ]; then
-    cls=0
-    bash .github/scripts/classify-build-failure.sh "$deps_log" "$cluster_name" || cls=$?
-    if [ "$cls" -ne 0 ]; then
-        exit "$cls"
-    fi
     exit 1
 fi

.github/workflows/frontier/build.sh

@@ -122,17 +122,22 @@ def test_a_failing_probe_stops_before_the_solver_build(workspace):
     assert len(trace.read_text().splitlines()) == 1, "solver build must not be attempted"
 
 
-def test_a_pypi_failure_records_a_cluster_outage(workspace):
+def test_a_pypi_failure_is_an_ordinary_build_failure(workspace):
+    """A flaky download must not red out the rest of the cluster.
+
+    The dependency install happens before any compute is committed -- on the
+    login node for Frontier -- so a failed fetch costs no allocation and there
+    is nothing to protect the matrix from. Recording it as a cluster-wide outage
+    skipped every other job on that cluster, including ones whose tests had
+    already passed, and uv already retries internally.
+    """
     tmp_path, install_mfc, run, _ = workspace
     install_mfc(full_build_stdout=PYPI_FAILURE, full_build_rc=1)
-    run()
-    assert outage_recorded(tmp_path)
 
+    result = run()
 
-def test_a_pypi_failure_reports_the_outage_exit_code(workspace):
-    tmp_path, install_mfc, run, _ = workspace
-    install_mfc(full_build_stdout=PYPI_FAILURE, full_build_rc=1)
-    assert run().returncode == 78
+    assert result.returncode == 1, "a failed download must fail only its own job"
+    assert not outage_recorded(tmp_path), "a failed download must not trip the cluster breaker"
 
 
 def test_an_ordinary_compile_error_is_not_treated_as_an_outage(workspace):
@@ -152,23 +157,6 @@ def test_the_build_output_is_still_shown_when_it_fails(workspace):
     assert "ftn-2116" in result.stdout + result.stderr
 
 
-def test_a_pypi_failure_during_the_probe_build_records_an_outage(workspace):
-    # The probe build is now the first mfc.sh call in the job, so it is what
-    # bootstraps build/venv from PyPI -- and on Phoenix clean_build has just
-    # deleted that venv, so it is rebuilt every time. Classifying only the solver
-    # build leaves the breaker blind to the outage it exists for.
-    tmp_path, install_mfc, run, _ = workspace
-    install_mfc(probe_build_stdout=PYPI_FAILURE, probe_build_rc=1)
-    run()
-    assert outage_recorded(tmp_path)
-
-
-def test_a_pypi_failure_during_the_probe_build_reports_the_outage_exit_code(workspace):
-    tmp_path, install_mfc, run, _ = workspace
-    install_mfc(probe_build_stdout=PYPI_FAILURE, probe_build_rc=1)
-    assert run().returncode == 78
-
-
 def test_an_ordinary_probe_build_failure_is_not_an_outage(workspace):
     tmp_path, install_mfc, run, _ = workspace
     install_mfc(probe_build_stdout="NVFORTRAN-S-0034-Syntax error\n", probe_build_rc=1)

toolchain/mfc/test_build_preflight.py

@@ -74,17 +74,22 @@ def test_a_successful_dependency_fetch_records_nothing(workspace):
     assert not outage_recorded(tmp_path)
 
 
-def test_a_pypi_outage_on_the_login_node_is_recorded(workspace):
+def test_a_pypi_failure_is_an_ordinary_build_failure(workspace):
+    """A flaky download must not red out the rest of the cluster.
+
+    The dependency install happens before any compute is committed -- on the
+    login node for Frontier -- so a failed fetch costs no allocation and there
+    is nothing to protect the matrix from. Recording it as a cluster-wide outage
+    skipped every other job on that cluster, including ones whose tests had
+    already passed, and uv already retries internally.
+    """
     tmp_path, install_mfc, run = workspace
     install_mfc(stdout=PYPI_FAILURE, rc=1)
-    run()
-    assert outage_recorded(tmp_path)
 
+    result = run()
 
-def test_a_pypi_outage_on_the_login_node_reports_the_outage_exit_code(workspace):
-    tmp_path, install_mfc, run = workspace
-    install_mfc(stdout=PYPI_FAILURE, rc=1)
-    assert run().returncode == 78
+    assert result.returncode == 1, "a failed download must fail only its own job"
+    assert not outage_recorded(tmp_path), "a failed download must not trip the cluster breaker"
 
 
 def test_an_ordinary_dependency_failure_is_not_an_outage(workspace):