Ignore errors when attempting to restore messages if the message was acked - #2552
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2552 +/- ##
=======================================
Coverage 82.75% 82.75%
=======================================
Files 79 79
Lines 10261 10262 +1
Branches 1174 1175 +1
=======================================
+ Hits 8491 8492 +1
Misses 1569 1569
Partials 201 201 ☔ View full report in Codecov by Harness. |
8db85ac to
66232cc
Compare
There was a problem hiding this comment.
Pull request overview
This PR targets a shutdown/concurrency race in kombu.transport.virtual.base.QoS.restore_unacked() where a message can be acked by another thread after _flush() but before/during a restore attempt (notably surfacing as noisy SQS errors). The change aims to suppress restore errors when the message is already known to have been acked.
Changes:
- Track the delivery-tag key when popping from
_deliveredso restore failures can be correlated to_dirty(acked/rejected) tags. - Suppress restore exceptions for messages whose delivery tag is now present in
_dirty. - Add a unit test asserting errors are ignored when restore raises and the message becomes acked/dirty.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
kombu/transport/virtual/base.py |
Ignores restore exceptions when the delivery tag is marked dirty (acked) during restore_unacked processing. |
t/unit/transport/virtual/test_base.py |
Adds a regression test covering “restore raises but message is acked” behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
is there any open or relevant issue this PR is handling? |
I can't see any associated issues in Github for this - but it is a real issue I am experiencing in my production environment |
I am seeing a lot of errors from SQS in my logs coming from here (through the
restore_unacked_oncepath) with the message:My assumption is that while we are closing the channel, we call
restore_unacked_once, but some task invocations are still completing while we are restoring unacked messages. This results in a race condition where messages may be acked after the '_flush' call earlier in this function, and the loop iteration where we attempt to restore that particular message.If we encounter an exception when we attempt to restore a message but then find that the message has already been acked by another thread - then we should ignore the error.