Handle backoff with new error

[wild-endeavor]

Summary

Handle RESOURCE_EXHAUSTED in the remote controller instead of the generic RPC retry interceptor.

• Introduce ResourceExhaustedError as a specialized SlowDownError.
• Retry with capped exponential backoff while the controller is running.
• Track resource-exhaustion attempts separately without incrementing action.retries.
• Log a warning for each retry.

Why

Resource exhaustion is not a terminal action error and may persist for an arbitrary amount of time. Consuming the normal bounded retry budget would eventually fail runs that should continue waiting.

Testing

• Added coverage confirming RESOURCE_EXHAUSTED bypasses the generic retry interceptor.
• Added coverage confirming repeated resource exhaustion does not consume the action retry budget.
• Tested in staging with:
  • run_concurrency: 1
  • action_concurrency: 2
  • depth: 5
• Observed retries backing off from 0.5s to 1s, 2s, 4s, and 8s.

[pingsutw]

Should we add a max_resource_exhausted_retries?

[pingsutw]

or do we want to retry forever?

[wild-endeavor]

my vote is retry forever. upcoming observability should expose long running tasks to the user to manually abort.

[pingsutw]

we don't want to use RetryUnaryInterceptor? why?

[wild-endeavor]

maybe we do... my reasoning is that unavailable is a network level thing. resource_exhausted we think is not a thing that's sent by nginx/envoy/nlbs, so basically the network request successfully went through - but the server said hold off.

honestly i don't know why we retry on code.internal either - we should not, that would just add to the storm... but i suppose that does give us time to live patch something. we should probably add a sentry on this to see how often we hit this/recover.