[Bug]: TPU nightlies latch red permanently after a single DWS timeout

[LukeAVanDrie]

Contact Details

lukevandrie@google.com

What happened?

All four GKE TPU nightly lanes are red because the Request TPUs (DWS) step is not idempotent. One night of genuine TPU scarcity makes the lane fail forever after.

reusable-ci-nightly-benchmark.yaml#L538-L608 applies a tpu-request-job, waits for its pods, then deletes it to release the chips. The delete (L607) only runs if the wait (L600) succeeds. When the wait times out, the Job is orphaned. The namespace is not cleaned up either, because the teardown step exits early with llmdbenchmark was not even installed, will not attempt to teardown.

Every subsequent run then finds the Job already present. kubectl apply reports unchanged, creates no pods, and kubectl wait immediately returns no matching resources found.

Expected: a transient DWS capacity shortage fails one run, and the next run retries from a clean state.

Actual: the first capacity failure is permanent. The lane never recovers without manual intervention, and every later failure misreports as a TPU shortage when it is really a leftover Job.

Affected lanes in llm-d, pass rate over the last 7 scheduled runs:

Lane	Rate	Note
pd-disaggregation-gke-acc-tpu-vllm-x	5/7	latched Jul 26
precise-prefix-cache-gke-acc-tpu-vllm-x	1/7	6 consecutive
tiered-prefix-cache-gke-cpu-tpu-vllm-native	1/7	6 consecutive
optimized-baseline-gke-acc-tpu-vllm-x	2/7	cleared on its own Jul 27

Suggested fix

• Delete the orphaned tpu-request-job in each stuck llm-d-nightly-*-gke-tpu namespace to unstick the current runs.
• Add kubectl delete job tpu-request-job -n "$LLMDBENCH_CICD_NS" --ignore-not-found before the heredoc at L574, or switch the Job to generateName.
• Move the release-the-chips delete at L607 into its own if: always() step so a wait timeout cannot orphan the Job.

Separately, no matching resources found is a confusing failure message here; it reads as scarcity when it means stale state.

Version

main

Steps to Reproduce

1. Run any GKE TPU nightly lane at a time when TPU v6e capacity is unavailable in us-east5, so that "Request TPUs (DWS)" times out.
2. Observe the step fail with "timed out waiting for the condition on pods/tpu-request-job-*", leaving the Job and namespace in place.
3. Re-run the same lane at any later time, including when capacity is free.
4. Observe it fail in under a second with "job.batch/tpu-request-job unchanged" followed by "error: no matching resources found".

Environment

- Kubernetes: GKE, cluster `llm-d-e2e-us-east5`, zone `us-east5`
- Cloud provider: GCP, project `llm-d-scale`
- Accelerator: TPU v6e (`tpu-v6e-slice`), topology 2x4, 16 chips, 2 VMs
- Provisioning: DWS flex-start (`cloud.google.com/gke-flex-start: "true"`)
- Workflow: llm-d-infra `reusable-ci-nightly-benchmark.yaml@main`

Relevant log output

# Jul 26 — the original, genuine capacity failure
# https://github.com/llm-d/llm-d/actions/runs/30187492637
2026-07-26T04:16:51.3668729Z job.batch/tpu-request-job created
2026-07-26T05:16:51.6383026Z timed out waiting for the condition on pods/tpu-request-job-74cl8
2026-07-26T05:16:51.6384387Z timed out waiting for the condition on pods/tpu-request-job-t24zx
# Waited the full 60 minutes. The "kubectl delete job" on L607 never ran.

# Jul 27 — same lane, now latched
# https://github.com/llm-d/llm-d/actions/runs/30236796772
2026-07-27T04:21:16.1871986Z Dynamic TPU request parameters calculated:
2026-07-27T04:21:16.6485014Z job.batch/tpu-request-job unchanged
2026-07-27T04:21:16.8880766Z error: no matching resources found
# 240 ms from apply to error. Not a capacity timeout.

# Teardown declines to clean up, so the Job survives to the next run
### llmdbenchmark was not even installed, will not attempt to teardown

# For contrast — optimized-baseline-gke-acc-tpu on a clean namespace, same day
# https://github.com/llm-d/llm-d/actions/runs/30260698411
2026-07-27T11:08:08.9705074Z job.batch/tpu-request-job created
2026-07-27T11:18:56.6184960Z pod/tpu-request-job-c8qdl condition met
2026-07-27T11:18:59.4675501Z pod/tpu-request-job-kjf54 condition met
2026-07-27T11:18:59.7697184Z job.batch "tpu-request-job" deleted from llm-d-nightly-optimized-baseline-gke-tpu namespace
# Chips acquired in ~10 min. Same cluster, same step, no leftover Job.

[LukeAVanDrie]

cc @rlakhtakia and @nicolexin

[nicolexin]

/assign @nicolexin