Feature: urllib3 http client v2

[spawn-guy]

as it was reported urllib3 might be slow(er) than pycurl as an http client. (or even "not working" in 1 unverified report)

this PR brings the curl back. and uses pycurl when available, reverting to urllib3 when not

previous PR super-seeds #2269 and updates urllib3_client implementation with multi-threading (similar to CurlMulti) that brings speeds to 98% of pycurl version

this PR super-seeds #2312 and depends on #2586 which was extracted from my work

as the pycurl dependency was removed from being required by sqs extra module
to use pycurl - users need to explicitly add and install pycurl library on their own.

the last required version in pip/requirements.txt format was

pycurl>=7.43.0.5; sys_platform != 'win32' and platform_python_implementation=="CPython"

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.33%. Comparing base (74f2d25) to head (695f55a).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2587      +/-   ##
==========================================
+ Coverage   83.15%   83.33%   +0.17%     
==========================================
  Files          79       80       +1     
  Lines       10626    10728     +102     
  Branches     1241     1256      +15     
==========================================
+ Hits         8836     8940     +104     
+ Misses       1584     1583       -1     
+ Partials      206      205       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[auvipy]

we should keep pycurl as default, and urllib3 as fallback

[spawn-guy]

@auvipy as described - if pycurl is available it will be used. otherwise it will fall back to, already required, urllib3(boto3).
one dependency fewer.
i am also having problems installing pycurl on aws linux (it is possible, but not out-of-the-box) - so "fresh" celery/kombu usage and deployment experience will be easier

[spawn-guy]

rebased on recent changes

[Copilot]

Pull request overview

This PR reintroduces an optional pycurl-backed async HTTP client for Kombu’s asynchronous AWS/SQS stack, falling back to a new urllib3-based client when pycurl isn’t available.

Changes:

• Add a new Urllib3Client implementation using a thread pool.
• Update kombu.asynchronous.http.Client() to prefer CurlClient when pycurl is present, otherwise use Urllib3Client.
• Add unit tests and reference docs for the new urllib3 client and selection logic.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
t/unit/asynchronous/http/test_urllib3.py	Adds unit tests for the new Urllib3Client.
t/unit/asynchronous/http/test_http.py	Adds tests for curl-vs-urllib3 client selection and get_client caching.
kombu/asynchronous/http/urllib3_client.py	Introduces a new thread-pool-based urllib3 HTTP client implementation.
kombu/asynchronous/http/base.py	Exposes BaseClient in __all__.
kombu/asynchronous/http/init.py	Implements runtime selection between curl and urllib3 clients and exports get_client.
docs/reference/kombu.asynchronous.http.urllib3_client.rst	Adds API reference stub for urllib3_client.
docs/reference/index.rst	Adds urllib3_client to the reference index.
AUTHORS	Adds a contributor entry.

Suppressed comments (3)

t/unit/asynchronous/http/test_urllib3.py:130

• Patching urllib3.util.make_headers won't affect this client because make_headers is imported into kombu.asynchronous.http.urllib3_client at import time. Patch the symbol used by the code under test so the call_args_list assertions are meaningful.

                with patch('urllib3.util.make_headers') as mock_make_headers:

kombu/asynchronous/http/urllib3_client.py:171

• The comment says redirects are handled manually to match pycurl, but no manual redirect handling exists here. Also, passing retries=False while setting redirect=request.follow_redirects is inconsistent and may prevent redirects from being followed depending on urllib3 behavior. Please align the implementation so follow_redirects is reliably honored (either via an explicit Retry configuration or manual redirect loop).

                preload_content=True,  # We want to preload content for compatibility
                redirect=request.follow_redirects,
                retries=False,  # Handle redirects manually to match pycurl behavior
            )

kombu/asynchronous/http/urllib3_client.py:156

• CurlClient always sends an explicit empty body for POST/PUT when request.body is falsy. Here body stays None, which can change request semantics (e.g., missing Content-Length) compared to the curl implementation.

        body = None
        if request.method in ('POST', 'PUT') and request.body:
            body = request.body if isinstance(request.body, bytes) else request.body.encode('utf-8')

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[spawn-guy]

@auvipy fixed code and tests. also enabled unit-tests even when pycurl is not installed