You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6d9d72b
Browse filesBrowse the repository at this point in the historyBrowse files
docs: Document the transport-hook contract and the built-in HTTPX client (#1013)
Rewrites the HTTP clients concepts page around the three-layer hierarchy
and the transport hooks, documents the built-in HTTPX client from #1004
(installation, sync/async examples, timeout semantics), and replaces the
old HTTPX guide with "Build a custom HTTP client", which implements full
AIOHTTP and requests clients including response adapters for the
`HttpResponse` protocol.
Stack: #1022 (pipeline) → #1004 (HTTPX client) → **#1013 (this)**. Split
out of #1006.
*✍️ Drafted by Claude Code*
Copy file name to clipboardExpand all lines: README.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -135,7 +135,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
135
135
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
136
136
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
137
137
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
138
-
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([Custom HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
138
+
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
139
139
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).
140
140
- **Debug logging** — opt-in structured logging on the `apify_client` logger captures request URLs, status codes, retry attempts, and more ([Logging](https://docs.apify.com/api/client/python/docs/concepts/logging)).
141
141
@@ -202,8 +202,8 @@ The full documentation lives at **[docs.apify.com/api/client/python](https://doc
202
202
| --- | --- |
203
203
| [Introduction](https://docs.apify.com/api/client/python/docs) | Overview, prerequisites, and a tour of the client. |
204
204
| [Quick start](https://docs.apify.com/api/client/python/docs/quick-start) | Authenticate, run an Actor, and fetch its results step by step. |
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), use HTTPX as the HTTP client. |
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), build a custom HTTP client. |
207
207
| [Upgrading](https://docs.apify.com/api/client/python/docs/upgrading/upgrading-to-v3) | Migrating between major versions. |
208
208
| [API reference](https://docs.apify.com/api/client/python/reference) | Generated reference for every class, method, and model. |
209
209
| [Changelog](https://docs.apify.com/api/client/python/docs/changelog) | Release history and breaking changes. |
Copy file name to clipboardExpand all lines: docs/01_introduction/index.mdx
+17Lines changed: 17 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -63,6 +63,23 @@ For better request-body compression, opt in to `brotli`, which compresses better
63
63
64
64
For details, see [HTTP compression](../02_concepts/13_http_compression.mdx).
65
65
66
+
The client uses [Impit](https://github.com/apify/impit) as its default HTTP transport. To use the built-in [HTTPX](https://github.com/pydantic/httpx2) transport, install the optional `httpx` extra. The extra installs `httpx2`, Pydantic's maintained continuation of HTTPX:
The Apify API client uses a pluggable HTTP client architecture. By default, it ships with an [Impit](https://github.com/apify/impit)-based HTTP client that handles retries, timeouts, passing headers, and more. You can replace it with your own implementation for use cases like custom logging, proxying, request modification, or integrating with a different HTTP library.
22
+
The Apify API client uses a pluggable HTTP layer. It ships with an [Impit](https://github.com/apify/impit)-based default, offers [HTTPX](https://github.com/pydantic/httpx2) as an optional built-in alternative, and accepts custom synchronous or asynchronous implementations.
21
23
22
24
## Default HTTP client
23
25
24
26
When you create an <ApiLinkto="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLinkto="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> instance, it automatically uses the built-in <ApiLinkto="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink> (or <ApiLinkto="class/ImpitHttpClientAsync">`ImpitHttpClientAsync`</ApiLink>). This default client provides:
25
27
26
28
- Automatic retries with exponential backoff for network errors, HTTP 429, and HTTP 5xx responses.
27
29
- Configurable timeouts.
28
-
-Preparing request data and headers according to the API requirements, including authentication.
29
-
-Collecting requests statistics for monitoring and debugging.
30
+
-Request compression and preparation of API-compatible data, query parameters, and headers, including authentication.
31
+
-API error handling, structured logging, and request statistics.
30
32
31
33
You can configure the default client through the <ApiLinkto="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLinkto="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor:
32
34
@@ -43,35 +45,65 @@ You can configure the default client through the <ApiLink to="class/ApifyClient"
43
45
</TabItem>
44
46
</Tabs>
45
47
48
+
## Built-in HTTPX client
49
+
50
+
The package also provides <ApiLinkto="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink> and <ApiLinkto="class/HttpxHttpClientAsync">`HttpxHttpClientAsync`</ApiLink>. They use the same request preparation, compression, retry policy, timeout tiers and growth, error handling, logging, and statistics as the default Impit clients, with [HTTPX](https://github.com/pydantic/httpx2) as the transport.
51
+
52
+
HTTPX is an optional dependency. Install `apify-client[httpx]`, then pass the appropriate client to <ApiLinkto="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink>. The extra installs `httpx2`, Pydantic's maintained continuation of HTTPX. Impit remains the default even when the HTTPX extra is installed.
Configure retries, timeout tiers, default headers, and compression on the HTTPX client instance. The token passed to `with_custom_http_client` is applied automatically unless the HTTP client already has an `Authorization` header. The examples use the clients as context managers so their connection pools are closed deterministically. If a context manager doesn't fit your application's lifecycle, call `close()` on `HttpxHttpClient` or `await aclose()` on `HttpxHttpClientAsync` during shutdown.
74
+
75
+
Timeout values are passed to the selected transport. Impit enforces them as a deadline for the whole request, body included. HTTPX applies them to each socket operation instead, so a response whose body arrives slowly keeps resetting the timeout and can outlast both the requested timeout and `timeout_max`. The `no_timeout` option disables HTTPX's timeouts.
76
+
46
77
## Architecture
47
78
48
-
The HTTP client system is built on two key abstractions:
79
+
Internally, the HTTP client hierarchy has three layers:
80
+
81
+
- A common internal base contains configuration and utilities shared by synchronous and asynchronous clients, including headers, request-body preparation, parameters, compression, and timeout tiers. It isn't a public extension point.
82
+
- <ApiLinkto="class/HttpClient">`HttpClient`</ApiLink> and <ApiLinkto="class/HttpClientAsync">`HttpClientAsync`</ApiLink> add the synchronous or asynchronous request pipeline, retry loop, transport hooks, and lifecycle interface.
83
+
- The built-in Impit and HTTPX classes inherit directly from the corresponding sync or async class and adapt the underlying transport.
49
84
50
-
- <ApiLinkto="class/HttpClient">`HttpClient`</ApiLink> / <ApiLinkto="class/HttpClientAsync">`HttpClientAsync`</ApiLink> - Abstract base classes that define the interface. Extend one of these to create a custom HTTP client by implementing the `call` method.
51
-
- <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> - A [runtime-checkable protocol](https://docs.python.org/3/library/typing.html#typing.runtime_checkable) that defines the expected response shape. Any object with the required attributes and methods satisfies the protocol — no inheritance needed.
85
+
`HttpClient.is_timeout_error(exc)` and `HttpClientAsync.is_timeout_error(exc)` are the public, transport-neutral way to tell whether an exception is a timeout, so code built on the client, such as streamed logs, doesn't need to know which transport raised it.
52
86
53
-
To plug in your custom implementation, use the <ApiLinkto="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink> class method.
87
+
Responses have their own abstraction. <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> is a [runtime-checkable protocol](https://docs.python.org/3/library/typing.html#typing.runtime_checkable) that defines the expected response shape. Any object with the required attributes and methods satisfies the protocol, so no inheritance is needed.
54
88
55
-
All of these are available as top-level imports from the `apify_client` package:
89
+
Custom transport adapters implement the request, error-classification, and lifecycle hooks. They inherit request preparation, retries, timeout growth, API error conversion, logging, and statistics from the base.
90
+
91
+
The base classes and the response protocol are available from the `apify_client.http_clients` module:
56
92
57
93
<CodeBlockclassName="language-python">
58
94
{ArchitectureImportsExample}
59
95
</CodeBlock>
60
96
61
-
### The call method
97
+
### The transport contract
62
98
63
-
The `call` method receives all the information needed to make an HTTP request:
99
+
The public `call` method provides the shared request pipeline. A concrete transport implements these hooks:
-`params` - Query parameters to append to the URL.
69
-
-`data` - Raw request body (mutually exclusive with `json`).
70
-
-`json` - JSON-serializable request body (mutually exclusive with `data`).
71
-
-`stream` - Whether to stream the response body.
72
-
-`timeout` - Timeout for the request as a `timedelta`.
101
+
-`send_request(...)` sends one prepared request and returns an `HttpResponse`, error statuses included. It receives the URL with the query parameters already encoded into it, the headers with the client's default headers already merged in, the body already serialized and compressed, and the timeout for this attempt in seconds. The inherited `call` needs it, so every transport adapter has to implement it. Let the HTTP library's exceptions propagate unwrapped, and leave status handling and <ApiLinkto="class/ApifyApiError">`ApifyApiError`</ApiLink> to `call`.
102
+
-`is_retryable_transport_error(exc)` classifies transport failures for the shared retry loop. The default classifies nothing as retryable, so a transport that doesn't override it gives up on the first connection failure.
103
+
-`is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The default recognizes Python's `TimeoutError`. Timeout classification is independent of retryability, so a timeout the retry loop should retry has to be listed in `is_retryable_transport_error` too.
104
+
-`close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a transport that owns no pool or session.
73
105
74
-
It must return an object satisfying the <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> protocol.
106
+
Decorate your implementations with `@override`, as the built-in Impit and HTTPX adapters do, so a type checker catches a misspelled or incompatible override.
75
107
76
108
### The HTTP response protocol
77
109
@@ -92,7 +124,9 @@ It must return an object satisfying the <ApiLink to="class/HttpResponse">`HttpRe
92
124
|`aiter_bytes() -> AsyncIterator[bytes]`| Iterate body in chunks (async) |
93
125
94
126
:::note
95
-
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX](https://www.python-httpx.org/) already satisfy this protocol out of the box.
127
+
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX](https://github.com/pydantic/httpx2) already satisfy this protocol out of the box.
128
+
129
+
For a streamed response, consume the body inside the streaming context manager with `iter_bytes()` / `aiter_bytes()`, or call `read()` / `aread()` before accessing `content`. Some transports, including HTTPX, intentionally reject `content` on an unread streamed response.
96
130
:::
97
131
98
132
### Plugging it in
@@ -115,18 +149,18 @@ Use the <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.wit
115
149
After that, all API calls made through the client will go through your custom HTTP client.
116
150
117
151
:::warning
118
-
When using a custom HTTP client, you are responsible for constructing the request, handling retries, timeouts, and errors yourself. The default retry logic is not applied.
152
+
If you override `call` itself, your implementation becomes responsible for request preparation, retries, timeouts, API error conversion, logging, and statistics. Implementing the transport hooks and inheriting `call` keeps the shared behavior.
119
153
:::
120
154
121
155
## Use cases
122
156
123
-
Custom HTTP clients might be useful when you need to:
157
+
Custom HTTP clients might be useful when the built-in Impit and HTTPX clients don't cover your requirements, for example when you need to:
124
158
125
-
-**Use a different HTTP library** - Swap Impit for [httpx](https://www.python-httpx.org/), [requests](https://requests.readthedocs.io/), or [aiohttp](https://docs.aiohttp.org/).
159
+
-**Use a different HTTP library** - Integrate [requests](https://requests.readthedocs.io/), [aiohttp](https://docs.aiohttp.org/), or another transport.
126
160
-**Route through a proxy** - Add proxy support or request routing.
127
161
-**Implement custom retry logic** - Use different backoff strategies or retry conditions.
128
162
-**Log requests and responses** - Track API calls for debugging or auditing.
129
163
-**Modify requests** - Add custom fields, modify the body, or change headers.
130
164
-**Collect custom metrics** - Measure request latency, track error rates, or count API calls.
131
165
132
-
For a step-by-step walkthrough of building a custom HTTP client, see the [Using HTTPX as the HTTP client](/api/client/python/docs/guides/custom-http-client-httpx) guide.
166
+
For complete synchronous and asynchronous implementations over a transport with a different response API, see [Build a custom HTTP client](../03_guides/05_custom_http_client.mdx). The <ApiLinkto="class/HttpClient">`HttpClient`</ApiLink> and <ApiLinkto="class/HttpClientAsync">`HttpClientAsync`</ApiLink> API references document the full contract.
0 commit comments