Repository navigation
Conversation
When valkey is stopped ahead of cloud_controller_ng draining during an api-group rolling update, its unix socket is removed while CC-NG still serves traffic. Every request then hit RedisStore on the dead socket and raised Redis::CannotConnectError, which surfaced as a 500 with a leaked backtrace. Wrap the RedisStore metrics writes so connection errors are logged and swallowed — a metrics write must never fail a user request. Genuine errors (e.g. Redis::CommandError) still propagate.
kathap
force-pushed
the
valkey-drain-resilience
branch
from
November 12, 2026 12:14
cee89e1 to
79c5b80
Compare
2 of 3 tasks
philippthun
reviewed
Nov 13, 2026
philippthun
left a comment
Member
There was a problem hiding this comment.
We should come up with a single pattern for all places where we interact with Valkey: base_rate_limiter.rb, concurrent_request_counter.rb and statsd_updater.rb. All three should rescue the same exceptions, write at the same log level and use similar wording (fail open vs. tolerate).
Both the commit message and the PR description mention a draining behavior that is wrong (i.e. caused by another bug and fixed with cloudfoundry/capi-release#698) and thus should be updated.
StatsdUpdater's RedisStore updates the outstanding-requests gauge in Valkey on every request. Any Redis error propagated out of the request-metrics middleware and failed the request with a 500. Wrap the RedisStore operations in a fail_open helper that logs and swallows Redis errors, matching the existing handling in base_rate_limiter.rb and concurrent_request_counter.rb: rescue Redis::BaseError, log at error with "Redis error: <class> - <message>".
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StatsdUpdater's RedisStore reaches Valkey over a unix socket on every request to maintain the outstanding-requests gauge. If a Redis call failed, for example Valkey being briefly unavailable, the error propagated out of the request-metrics middleware and the request returned a 500.
This makes the metrics path fail open: a metrics write must never fail a user request. Redis errors are caught, logged, and the operation returns nil.
CC has three code paths that talk to Valkey, and they handled Redis errors three different ways. This aligns StatsdUpdater with the existing, already-merged handling in the other two so all three behave the same:
Make StatsdUpdater's RedisStore fail open on Valkey errors. The metrics store is reached over a unix socket on every request to update the outstanding-requests gauge; any Redis error used to propagate out of the request-metrics middleware and 500 the request. Now Redis errors are caught, logged, and the operation returns nil. This aligns StatsdUpdater's error handling with the existing Valkey callers base_rate_limiter.rb and concurrent_request_counter.rb: all three rescue Redis::BaseError and log at error with the shared "Redis error: - " wording. Each still fails open in the way that fits its own call site (StatsdUpdater returns nil; the rate limiter returns a non-limiting count; the concurrent counter rescues a StoreError one level up).
CC updates the StatsD/Valkey metrics gauge on every request. When Valkey is momentarily unavailable (e.g. briefly down or restarting), the Redis call raised and the request returned a 500 to the caller instead of completing, a metrics write should never fail a user request. Failing open keeps requests succeeding while Valkey is unreachable, and rescuing/logging the same way as the other two Valkey callers makes the error handling consistent across CC.
This change was developed with AI assistance; all code was reviewed and tested by me.
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests