Repository navigation
fix(engine): don't log routine concurrency index builds as errors - #5220
Closed
igor-kupczynski wants to merge 1 commit into
Closed
igor-kupczynski wants to merge 1 commit into
igor-kupczynski wants to merge 1 commit into
Conversation
Every scheduler restart or tenant rebalance makes the new owner wait out the previous holder's 30s outbox lease before the in-memory index builds. Run reported each of those ticks as an error, flooding alerts on rollouts. Builds under 60s now return ErrIndexBuildInProgress, logged at debug under a separate message; longer builds still log as errors.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Author
Closing for now in favour of #5214. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Every scheduler restart, scale event or tenant rebalance makes the new owner wait out the previous holder's 30s outbox lease before the in-memory concurrency index builds.
Runlogged each tick of that wait aserror running concurrency strategy, so every rollout floods the error alerts.Builds under 60s now return
ErrIndexBuildInProgress, logged at debug under a separate message. Builds past 60s (actually stuck) still log at error. Observed build waits top out around 35s.Type of change
🤖 AI Disclosure
I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
Details: Claude Code investigated the alert, wrote the change and test, and ran a separate review pass.