Skip to content

Tolerate missing valkey socket in RedisStore - #5663

Open
kathap wants to merge 2 commits into
mainfrom
valkey-drain-resilience
Open

kathap wants to merge 2 commits into
mainfrom
valkey-drain-resilience

Conversation

@kathap

@kathap kathap commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

StatsdUpdater's RedisStore reaches Valkey over a unix socket on every request to maintain the outstanding-requests gauge. If a Redis call failed, for example Valkey being briefly unavailable, the error propagated out of the request-metrics middleware and the request returned a 500.

This makes the metrics path fail open: a metrics write must never fail a user request. Redis errors are caught, logged, and the operation returns nil.

CC has three code paths that talk to Valkey, and they handled Redis errors three different ways. This aligns StatsdUpdater with the existing, already-merged handling in the other two so all three behave the same:

File Rescues Log level On error
middleware/base_rate_limiter.rb Redis::BaseError error fail open
middleware/concurrent_request_counter.rb Redis::BaseError error fail open
lib/cloud_controller/metrics/statsd_updater.rb (this change) Redis::BaseError error fail open
  • A short explanation of the proposed change:
    Make StatsdUpdater's RedisStore fail open on Valkey errors. The metrics store is reached over a unix socket on every request to update the outstanding-requests gauge; any Redis error used to propagate out of the request-metrics middleware and 500 the request. Now Redis errors are caught, logged, and the operation returns nil. This aligns StatsdUpdater's error handling with the existing Valkey callers base_rate_limiter.rb and concurrent_request_counter.rb: all three rescue Redis::BaseError and log at error with the shared "Redis error: - " wording. Each still fails open in the way that fits its own call site (StatsdUpdater returns nil; the rate limiter returns a non-limiting count; the concurrent counter rescues a StoreError one level up).
  • An explanation of the use cases your change solves
    CC updates the StatsD/Valkey metrics gauge on every request. When Valkey is momentarily unavailable (e.g. briefly down or restarting), the Redis call raised and the request returned a 500 to the caller instead of completing, a metrics write should never fail a user request. Failing open keeps requests succeeding while Valkey is unreachable, and rescuing/logging the same way as the other two Valkey callers makes the error handling consistent across CC.

This change was developed with AI assistance; all code was reviewed and tested by me.

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

When valkey is stopped ahead of cloud_controller_ng draining during an
api-group rolling update, its unix socket is removed while CC-NG still
serves traffic. Every request then hit RedisStore on the dead socket and
raised Redis::CannotConnectError, which surfaced as a 500 with a leaked
backtrace.

Wrap the RedisStore metrics writes so connection errors are logged and
swallowed — a metrics write must never fail a user request. Genuine
errors (e.g. Redis::CommandError) still propagate.

@philippthun philippthun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should come up with a single pattern for all places where we interact with Valkey: base_rate_limiter.rb, concurrent_request_counter.rb and statsd_updater.rb. All three should rescue the same exceptions, write at the same log level and use similar wording (fail open vs. tolerate).

Both the commit message and the PR description mention a draining behavior that is wrong (i.e. caused by another bug and fixed with cloudfoundry/capi-release#698) and thus should be updated.

StatsdUpdater's RedisStore updates the outstanding-requests gauge in
Valkey on every request. Any Redis error propagated out of the
request-metrics middleware and failed the request with a 500.

Wrap the RedisStore operations in a fail_open helper that logs and
swallows Redis errors, matching the existing handling in
base_rate_limiter.rb and concurrent_request_counter.rb: rescue
Redis::BaseError, log at error with "Redis error: <class> - <message>".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants