Skip to content

Ignore errors when attempting to restore messages if the message was acked - #2552

Merged
auvipy merged 4 commits into
celery:mainfrom
soceanainn:ignore-restore-errors-for-acked-messages
Jun 10, 2026
Merged

Ignore errors when attempting to restore messages if the message was acked#2552
auvipy merged 4 commits into
celery:mainfrom
soceanainn:ignore-restore-errors-for-acked-messages

Conversation

@soceanainn

Copy link
Copy Markdown
Contributor

I am seeing a lot of errors from SQS in my logs coming from here (through the restore_unacked_once path) with the message:

Message does not exist or is not available for visibility timeout change.

My assumption is that while we are closing the channel, we call restore_unacked_once, but some task invocations are still completing while we are restoring unacked messages. This results in a race condition where messages may be acked after the '_flush' call earlier in this function, and the loop iteration where we attempt to restore that particular message.

If we encounter an exception when we attempt to restore a message but then find that the message has already been acked by another thread - then we should ignore the error.

@codecov

codecov Bot commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.75%. Comparing base (0b293c7) to head (6fd1cc6).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2552   +/-   ##
=======================================
  Coverage   82.75%   82.75%           
=======================================
  Files          79       79           
  Lines       10261    10262    +1     
  Branches     1174     1175    +1     
=======================================
+ Hits         8491     8492    +1     
  Misses       1569     1569           
  Partials      201      201           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@soceanainn
soceanainn force-pushed the ignore-restore-errors-for-acked-messages branch from 8db85ac to 66232cc Compare June 4, 2026 15:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets a shutdown/concurrency race in kombu.transport.virtual.base.QoS.restore_unacked() where a message can be acked by another thread after _flush() but before/during a restore attempt (notably surfacing as noisy SQS errors). The change aims to suppress restore errors when the message is already known to have been acked.

Changes:

  • Track the delivery-tag key when popping from _delivered so restore failures can be correlated to _dirty (acked/rejected) tags.
  • Suppress restore exceptions for messages whose delivery tag is now present in _dirty.
  • Add a unit test asserting errors are ignored when restore raises and the message becomes acked/dirty.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
kombu/transport/virtual/base.py Ignores restore exceptions when the delivery tag is marked dirty (acked) during restore_unacked processing.
t/unit/transport/virtual/test_base.py Adds a regression test covering “restore raises but message is acked” behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread kombu/transport/virtual/base.py
Comment thread t/unit/transport/virtual/test_base.py
@auvipy auvipy added this to the 5.7.0 milestone Jun 9, 2026
@auvipy

auvipy commented Jun 10, 2026

Copy link
Copy Markdown
Member

is there any open or relevant issue this PR is handling?

@soceanainn

Copy link
Copy Markdown
Contributor Author

is there any open or relevant issue this PR is handling?

I can't see any associated issues in Github for this - but it is a real issue I am experiencing in my production environment

@auvipy
auvipy merged commit 3b7bc66 into celery:main Jun 10, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants