Skip to content

Back off job polling from 250ms up to the polling interval - #3890

Open
sethboyles wants to merge 1 commit into
cloudfoundry:mainfrom
sethboyles:feature/cli-job-poll-backoff
Open

sethboyles wants to merge 1 commit into
cloudfoundry:mainfrom
sethboyles:feature/cli-job-poll-backoff

Conversation

@sethboyles

@sethboyles sethboyles commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Description of the Change

ccv3 job polling (PollJob, PollJobForState, PollJobToEventStream in api/cloudcontroller/ccv3/job.go) checks the job once immediately and then sleeps a fixed polling interval (3s, util/configv3/defaults.go) between checks. Short Cloud Controller jobs, such as app and route deletes, are rarely complete at that first check. So every command that waits on one takes at least 3s, even when the job finishes in well under a second.

This PR replaces the fixed sleep with a short ramp of early checks that then joins the existing polling grid:

  • Schedule: with the default 3s interval, checks happen at 0, 0.25, 0.75 and 1.75s, then at 3, 6, 9s… (today: 0, 3, 6, 9s…). Every check today's CLI makes still happens, so a job is never detected later than before. Check times are measured from the start of the poll call, so time spent in a GET doesn't push later checks off the grid. Ramp points (250ms, 750ms, 1.75s, …) are only used while they fall below the polling interval.
  • Streaming: PollJobToEventStream now closes the stream right after the terminal event, instead of sleeping one more interval before closing.
  • Unchanged: the job polling timeout and where it is checked, terminal-state handling, and event contents and order. Each poll call starts its own schedule.
  • Testability: the ccv3 Clock interface gains Sleep(time.Duration) (real implementation time.Sleep; fake regenerated), so the schedule is tested deterministically instead of against wall-clock bounds.

Only ccv3 job polling changes. Build/staging, package, app-start and task polling still use the existing fixed interval.

Request-count impact

  • Jobs done within ~250ms: 2 GETs, the same as today.
  • Jobs done between ~250ms and 3s: 3–5 GETs instead of 2.
  • Longer jobs: 3 extra GETs in total (at 0.25, 0.75 and 1.75s), then the same 3s cadence as today.

GET /v3/jobs/:guid is a small read: it looks up the job, its warnings and the linked resource. Each poll GET keeps the existing retry behavior for 5xx responses (api/cloudcontroller/wrapper/retry_request.go). If Cloud Controller rate limiting is enabled, these GETs count against the per-user limit as they do today. Operators can watch Cloud Controller request rates and 429 responses after rolling out a CLI version that includes this.

Why Is This PR Valuable?

Every async CLI command that waits on a job returns sooner when its job is short, including cf delete, cf delete -r, route deletes and service operations. For example, cf delete -r polls the app-delete job and then the route-delete jobs, so it currently pays at least two 3s waits.

Measured against a small Cloud Foundry foundation, building main and this branch and alternating them for 10 runs each. Each run deleted a freshly created app; for -r the app had one mapped route:

Command main mean / median This PR mean / median
cf delete -f 7.02s / 6.41s 6.42s / 6.36s
cf delete -f -r 15.68s / 15.68s 8.40s / 8.28s

On that foundation, app-delete jobs usually took a few seconds to be picked up by a worker. So cf delete -f mostly finished at the 6s grid point either way, but it no longer slipped to the 9s point. cf delete -f -r runs a second round of job polling for the route deletes, which are short, so it roughly halves. In an earlier acceptance-test run, cf delete -f -r averaged 9.1s over 165 calls. CI pipelines and test suites that run many such commands benefit the most.

Applicable Issues

None filed.

How Urgent Is The Change?

Not urgent.

Other Relevant Parties

Anyone running many async CLI commands, such as CI pipelines and acceptance-test suites. I will open a separate PR for v8 once this is accepted.

🤖 Generated with Claude Code

Job polling checked the job once immediately and then slept a fixed
polling interval (3s) between checks. Short asynchronous jobs, such as
app and route deletes, are rarely complete at the first check, so every
such command took at least 3s even when the job finished in well under
a second.

Poll at 250ms, 750ms and 1.75s after the start, then on the existing
polling-interval grid (3s, 6s, ...), so every check today's CLI makes
still happens and no job is detected later than before. The event stream
now closes right after the terminal event instead of sleeping one more
interval first. Timeout checks are unchanged.

Add Sleep to the ccv3 Clock so the delay sequence can be tested
deterministically.

ai-assisted=yes
AI-Tool-Used: Claude, Codex
AI-Tool-Use-Level: High
AI-Code-Category: Production

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sethboyles
sethboyles force-pushed the feature/cli-job-poll-backoff branch from 6c90d43 to 14e0808 Compare November 2, 2026 22:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant