Redis: propagate polling_interval from transport options to _brpop_start timeout by ghirailghiro · Pull Request #2346 · celery/kombu

ghirailghiro · 2025-08-08T23:40:58Z

This PR fixes an issue in the Redis transport where polling_interval provided in transport_options were not being propagated to the Channel’s _brpop_start timeout.

Changes included:

Implemented logic to propagate polling_interval from the connection’s transport to Channel._brpop_start timeout.
Updated the documentation in Redis transport section.
Added a dedicated unit test to ensure correct propagation behavior.

Why:

Without this fix, custom polling_interval or brpop_timeout values passed via transport_options are ignored by _brpop_start

Related issue:

celery/celery#6625

Example:

Here a quick example using celery:

from celery import Celery
app = Celery("demo", broker="redis://redis:6379/0")
app.conf.broker_transport_options = {'polling_interval': 10}

@app.task
def ping():
    return "pong"

Before changing on kombu:

After changing on kombu:

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Ensure that when either `polling_interval` or `brpop_timeout` is set in `transport_options`, the value is propagated from the connection’s transport to the `Channel` (`self.brpop_timeout`). Includes: - Implementation of the propagation logic - Updated documentation for `polling_interval` / `brpop_timeout` - Unit tests covering the behavior Related to celery/celery#6625.

auvipy · 2025-08-09T06:41:12Z

it seems we need to fix or remove the lib rabbitmq tests in a separate pr

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ghirailghiro · 2025-08-09T07:26:46Z

Ok can i help with this?

auvipy · 2025-08-09T07:59:16Z

sure, please!

Update automatic test to be aligned on the logic modified during review process

ghirailghiro · 2025-08-09T12:04:01Z

I update the test that I added so it follow that the polling_interval is now not always none, so that the test will not fail

ghirailghiro · 2025-08-09T13:07:01Z

Hi @auvipy I have fixed the failing test on this pr #2347 , let me now if that's ok

auvipy · 2025-08-10T03:11:25Z

thank for fixing the failing one in another pr. i have updated the branch, hope ci will be fine now

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull Request Overview

This PR fixes an issue where the Redis transport was not properly using custom polling_interval values from transport_options for the BRPOP timeout. The fix propagates the polling interval to the channel's _brpop_start timeout parameter, allowing users to configure how long the Redis BRPOP command waits for messages.

Propagates polling_interval from transport options to brpop_timeout in the Transport class
Modifies Channel to use the transport's brpop_timeout value instead of a hardcoded default
Updates _brpop_start method to accept optional timeout parameter with fallback to instance attribute

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
kombu/transport/redis.py	Implements brpop_timeout propagation from transport to channel and modifies _brpop_start method
t/unit/transport/test_redis.py	Adds unit test to verify polling_interval is properly propagated to brpop_timeout
docs/reference/kombu.transport.redis.rst	Documents the new polling_interval support for Redis transport

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

auvipy · 2025-08-10T16:48:07Z

thanks a lot!

Bumps [kombu](https://github.com/celery/kombu) from 5.5.4 to 5.6.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/celery/kombu/releases">kombu's releases</a>.</em></p> <blockquote> <h2>v5.6.1</h2> <h2>What's Changed</h2> <ul> <li>fix: ensure hub close does also remove global event loop reference by <a href="https://github.com/oliverhaas"><code>@oliverhaas</code></a> in <a href="https://redirect.github.com/celery/kombu/pull/2404">celery/kombu#2404</a></li> <li>fix: default value for SQS's receive message by <a href="https://github.com/cuducos"><code>@cuducos</code></a> in <a href="https://redirect.github.com/celery/kombu/pull/2405">celery/kombu#2405</a></li> <li>Feat: add support for credential_provider to redis broker by <a href="https://github.com/alaminopu"><code>@alaminopu</code></a> in <a href="https://redirect.github.com/celery/kombu/pull/2408">celery/kombu#2408</a></li> <li>Prepare for release: v5.6.1 by <a href="https://github.com/Nusnus"><code>@Nusnus</code></a> in <a href="https://redirect.github.com/celery/kombu/pull/2416">celery/kombu#2416</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/oliverhaas"><code>@oliverhaas</code></a> made their first contribution in <a href="https://redirect.github.com/celery/kombu/pull/2404">celery/kombu#2404</a></li> <li><a href="https://github.com/cuducos"><code>@cuducos</code></a> made their first contribution in <a href="https://redirect.github.com/celery/kombu/pull/2405">celery/kombu#2405</a></li> <li><a href="https://github.com/alaminopu"><code>@alaminopu</code></a> made their first contribution in <a href="https://redirect.github.com/celery/kombu/pull/2408">celery/kombu#2408</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/celery/kombu/compare/v5.6.0...v5.6.1">https://github.com/celery/kombu/compare/v5.6.0...v5.6.1</a></p> <h2>v5.6.0</h2> <h2>Key Highlights</h2> <h3>QoS Max Prefetch Limit <a href="https://redirect.github.com/celery/kombu/pull/2348">celery/kombu#2348</a></h3> <p>Prevent Out Of Memory crashes when queues flood with ETA/countdown tasks. The new optional <code>max_prefetch</code> parameter caps how many messages workers hold in memory. Defaults to unlimited (<code>None</code>) to preserve existing behavior.</p> <pre lang="python"><code>from kombu.common import QoS <h1>Limit prefetch to maximum 100 messages</h1> <p>qos = QoS(callback=consumer.qos, initial_value=10, max_prefetch=100) </code></pre></p> <h3>Redis Polling Interval Support <a href="https://redirect.github.com/celery/kombu/pull/2346">celery/kombu#2346</a></h3> <p>Fix Redis transport to properly propagate <code>polling_interval</code> and <code>brpop_timeout</code> from <code>transport_options</code> to the Channel's <code>_brpop_start</code> timeout.</p> <pre lang="python"><code>app.conf.broker_transport_options = {"polling_interval": 10} </code></pre> <p>Leave it unset to keep the familiar 1-second default, or raise it to slow down idle polling.</p> <h3>Pidbox RabbitMQ 4.x Compatibility <a href="https://redirect.github.com/celery/kombu/pull/2338">celery/kombu#2338</a></h3> <p>Let pidbox queues work on RabbitMQ 4.x brokers that reject transient, non-exclusive queues.</p> <h3>MongoDB Transport Improvements <a href="https://redirect.github.com/celery/kombu/pull/2347">celery/kombu#2347</a></h3> <p>URI options now come through lowercase and flattened again, so settings like <code>replicaSet=test_rs</code> show up as <code>options['replicaset']</code>.</p> <h3>Resource Pool Gevent Compatibility <a href="https://redirect.github.com/celery/kombu/pull/2314">celery/kombu#2314</a></h3> <p>Restore compatibility with recent gevent releases that monkey-patch the standard library queue.</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/celery/kombu/blob/main/Changelog.rst">kombu's changelog</a>.</em></p> <blockquote> <h1>5.6.1</h1> <p>:release-date: 25 November, 2025 :release-by: Tomer Nosrati</p> <p>What's Changed</p> <pre><code> - fix: ensure hub close does also remove global event loop reference ([#2404](celery/kombu#2404)) - fix: default value for SQS's receive message ([#2405](celery/kombu#2405)) - Feat: add support for credential_provider to redis broker ([#2408](celery/kombu#2408)) - Prepare for release: v5.6.1 ([#2416](celery/kombu#2416)) <p>.. _version-5.6.0:</p> <h1>5.6.0</h1> <p>:release-date: 1 November, 2025 :release-by: Tomer Nosrati</p> <p>Key Highlights </code></pre></p> <h2>QoS Max Prefetch Limit</h2> <p><code>PR [#2348](celery/kombu#2348) <https://github.com/celery/kombu/pull/2348></code>_</p> <p>Prevent Out Of Memory crashes when queues flood with ETA/countdown tasks. The new optional <code>max_prefetch</code> parameter caps how many messages workers hold in memory. Defaults to unlimited (<code>None</code>) to preserve existing behavior.</p> <p>.. code-block:: python</p> <pre><code>from kombu.common import QoS <h1>Limit prefetch to maximum 100 messages</h1> <p>qos = QoS(callback=consumer.qos, initial_value=10, max_prefetch=100) </code></pre></p> <h2>Redis Polling Interval Support</h2> <p><code>PR [#2346](celery/kombu#2346) <https://github.com/celery/kombu/pull/2346></code>_</p> <p>Fix Redis transport to properly propagate <code>polling_interval</code> and <code>brpop_timeout</code> from <code>transport_options</code> to the Channel's <code>_brpop_start</code> timeout.</p> <p>.. code-block:: python</p> <pre><code>app.conf.broker_transport_options = {"polling_interval": 10} </code></pre> <p>Leave it unset to keep the familiar 1-second default, or raise it to slow down idle polling.</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/celery/kombu/commit/5208431c95bda47c7f422638dd273e086ab34be9"><code>5208431</code></a> Prepare for release: v5.6.1 (<a href="https://redirect.github.com/celery/kombu/issues/2416">#2416</a>)</li> <li><a href="https://github.com/celery/kombu/commit/24ade8f73eb43af96ad81ef953493a58a54d28b0"><code>24ade8f</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/celery/kombu/issues/2415">#2415</a>)</li> <li><a href="https://github.com/celery/kombu/commit/88225114201a6a0e13f9fb67f49131a250cc7bf7"><code>8822511</code></a> Bump actions/checkout from 5 to 6 (<a href="https://redirect.github.com/celery/kombu/issues/2412">#2412</a>)</li> <li><a href="https://github.com/celery/kombu/commit/4a57bfe1fec85cd5b8f52493e1c11c9b7afcf9c8"><code>4a57bfe</code></a> Feat: add support for credential_provider to redis broker (<a href="https://redirect.github.com/celery/kombu/issues/2408">#2408</a>)</li> <li><a href="https://github.com/celery/kombu/commit/b67cbe631b24bdfa2ce944b99663070317e53268"><code>b67cbe6</code></a> Revert "Bump protobuf from 6.32.1 to 6.33.1 (<a href="https://redirect.github.com/celery/kombu/issues/2409">#2409</a>)"</li> <li><a href="https://github.com/celery/kombu/commit/3454c8879643a2deb7d4778779690f43e4185db3"><code>3454c88</code></a> Bump protobuf from 6.32.1 to 6.33.1 (<a href="https://redirect.github.com/celery/kombu/issues/2409">#2409</a>)</li> <li><a href="https://github.com/celery/kombu/commit/baa625da51ec01b3c64920f7519d0732d02229b3"><code>baa625d</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/celery/kombu/issues/2407">#2407</a>)</li> <li><a href="https://github.com/celery/kombu/commit/c89be8c9ebd8bd65bc38da1bf589a9e487335a08"><code>c89be8c</code></a> fix: default value for SQS's receive message (<a href="https://redirect.github.com/celery/kombu/issues/2405">#2405</a>)</li> <li><a href="https://github.com/celery/kombu/commit/ecbb733080e09cfe8f60eb8b2fc1db2f5f2af502"><code>ecbb733</code></a> fix: ensure hub close does also remove global event loop reference (<a href="https://redirect.github.com/celery/kombu/issues/2404">#2404</a>)</li> <li><a href="https://github.com/celery/kombu/commit/4e74395e44d6cf5a06b6cd1a40e3b3d3fac2c476"><code>4e74395</code></a> Prepare for release: v5.6.0 (<a href="https://redirect.github.com/celery/kombu/issues/2402">#2402</a>)</li> <li>Additional commits viewable in <a href="https://github.com/celery/kombu/compare/v5.5.4...v5.6.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=kombu&package-manager=pip&previous-version=5.5.4&new-version=5.6.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…art timeout (celery#2346) * Pidbox: add queue_exclusive / queue_durable flags * Update kombu/pidbox.py * update pidbox documentation * update documentation * redis: propagate brpop_timeout from transport options Ensure that when either `polling_interval` or `brpop_timeout` is set in `transport_options`, the value is propagated from the connection’s transport to the `Channel` (`self.brpop_timeout`). Includes: - Implementation of the propagation logic - Updated documentation for `polling_interval` / `brpop_timeout` - Unit tests covering the behavior Related to celery/celery#6625. * Update kombu/transport/redis.py * Update test_redis.py Update automatic test to be aligned on the logic modified during review process * Update kombu/transport/redis.py * Update kombu/transport/redis.py * Update docs/reference/kombu.transport.redis.rst * Update kombu/transport/redis.py --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <auvipy@gmail.com>

ghirailghiro and others added 7 commits July 20, 2025 22:55

Pidbox: add queue_exclusive / queue_durable flags

e3b1c09

Update kombu/pidbox.py

0bb2fec

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

update pidbox documentation

4381712

Merge remote-tracking branch 'origin/main'

0dc9824

update documentation

f977bb6

Merge branch 'celery:main' into main

d782ed9

auvipy requested review from auvipy and Copilot and removed request for Copilot August 9, 2025 06:30

This comment was marked as outdated.

Sign in to view

auvipy added the Component: Redis Broker label Aug 9, 2025

auvipy added this to the 5.6.0 milestone Aug 9, 2025

Update kombu/transport/redis.py

596f4f7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update test_redis.py

6dddcb2

Update automatic test to be aligned on the logic modified during review process

Merge branch 'main' into main

a1b6386

auvipy requested a review from Copilot August 10, 2025 03:29

This comment was marked as outdated.

Sign in to view

Update kombu/transport/redis.py

a79d267

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

auvipy reviewed Aug 10, 2025

View reviewed changes

Comment thread docs/reference/kombu.transport.redis.rst Outdated

auvipy requested a review from Copilot August 10, 2025 15:20

Copilot AI reviewed Aug 10, 2025

View reviewed changes

Comment thread kombu/transport/redis.py Outdated

Comment thread kombu/transport/redis.py

auvipy requested changes Aug 10, 2025

View reviewed changes

Comment thread kombu/transport/redis.py Outdated

Update kombu/transport/redis.py

a90297f

auvipy reviewed Aug 10, 2025

View reviewed changes

Comment thread docs/reference/kombu.transport.redis.rst Outdated

auvipy and others added 2 commits August 10, 2025 21:34

Update docs/reference/kombu.transport.redis.rst

474fabe

Update kombu/transport/redis.py

2e14c3c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

auvipy approved these changes Aug 10, 2025

View reviewed changes

auvipy added the PR Type: Feature label Aug 10, 2025

auvipy merged commit 883086d into celery:main Aug 10, 2025
40 checks passed

Uh oh!

Uh oh!

Conversation

ghirailghiro commented Aug 8, 2025

Changes included:

Why:

Related issue:

Example:

Uh oh!

This comment was marked as outdated.

Uh oh!

auvipy commented Aug 9, 2025

Uh oh!

ghirailghiro commented Aug 9, 2025

Uh oh!

auvipy commented Aug 9, 2025

Uh oh!

ghirailghiro commented Aug 9, 2025

Uh oh!

ghirailghiro commented Aug 9, 2025

Uh oh!

auvipy commented Aug 10, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

auvipy commented Aug 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants