feat: add dependency resilience policy, billing replay fallback, and hardening - #3
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR closes a bunch of edge-case and failure-mode gaps, plus it adds one system-level improvement that felt worth doing while we were in there. No breaking changes: everything new is opt-in or defaults to the old behaviour.
What changed
We now have a small resilience helper that wraps calls to backing services. When Redis is unavailable, we used to either swallow the error silently or let it blow up in an ugly way. Now you can tell the package what to do per dependency: fail open and return a fallback, or fail closed with a clean exception that includes a Retry-After header. It logs the problem, marks the OpenTelemetry span, and emits a single event that ops can alert on, so any part of the system that relies on Redis will trigger the same signal. We applied this to quotas, rate limiting, cache, and metrics, so they degrade sensibly instead of breaking unpredictably.
Billing replay runs were fragile because Stripe drops events after about 30 days. If a tenant's replay job ran late, it would just fail to retrieve the event. We now keep a minimal, PII-free copy of each event locally, just enough to rebuild what the dispatcher needs, and fall back to that when Stripe can't give us the original. This makes late replays work again. While there, we also fixed some wrong JSDoc that claimed a UUID idempotency key when it was actually a deterministic minute bucket.
On the hardening side, the circuit breaker now reloads its state from Redis on startup, so if a tenant database was marked as down before a deploy, we won't waste time probing it again right away. The schema driver no longer silently swallows connection release errors. We dropped a dead cache key from the feature flag service and made a logger lazy-loaded so unit tests don't need a real one.
All of this is covered by new unit and integration specs, and we turned on coverage checks to keep it that way. We also added proper docs pages for configuration, exceptions, troubleshooting, and an API reference, because reading the source code to figure out how things work wasn't great.