smurf-tf-workflow: apply credentials expire during the approval wait, leaving orphaned resources and a stuck lock #442

Summary

In .github/workflows/smurf-tf-workflow.yml, the apply job mints credentials before the manual approval gate and gives them 900 seconds:

role-duration-seconds: 900     (:209, :359)
🟦 Configure AWS               (:350)   before
⏳ Approval                    (:443)   after

So a single 15-minute budget has to cover init, however long a human takes to approve, and the apply itself.

Why this is worse than a timeout

When the budget runs out mid-apply, the failure is not clean. Terraform can create resources and then be unable to write state:

Error: failed to upload state: operation error S3: PutObject ... api error ExpiredToken
Error: Failed to persist state to backend

Observed consequences on a root with a slow resource (a CloudFront VPC origin, measured at 9m15s to create):

• four resources created in the cloud and recorded in no state file
• the state lock left held by a runner that no longer existed, blocking every subsequent apply on that root
• the orphans only discoverable by reading the apply log, then deleting them one at a time by hand

A plain timeout would have been recoverable. Losing the state write after creating resources is not.

Not fixable by the caller

timeout is a workflow input, role-duration-seconds is not. Raising the first alone only changes which of the two limits kills the run.

Suggested fix

Two changes, both small:

1. Move Configure AWS after the approval gate, or re-authenticate between the gate and the apply. A human reading a plan should not consume the apply's credential lifetime.
2. Raise role-duration-seconds to something an apply can finish inside, or expose it as an input so callers with slow roots can.

Re-authenticating after the gate is the higher-value half: it makes the wait unbounded without affecting the apply, and the saved plan is unchanged by a new session.

Workaround in the meantime

Callers with slow roots can plan and apply in a single job of their own, re-authenticating after the gate and applying a saved plan file. That also sidesteps the separate problem that a re-plan at apply time is not necessarily the plan a human approved.