You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 424c3f0
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -124,7 +124,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
124
124
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
125
125
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
126
126
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
127
-
- **Pluggable HTTP layer** — swap the default [Impit](https://github.com/apify/impit)-based HTTP client for`httpx`, `requests`, `aiohttp`, or any custom implementation ([Custom HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
127
+
- **Pluggable HTTP layer** — swap the default [Impit](https://github.com/apify/impit)-based HTTP client for`httpx`, `requests`, `aiohttp`, or any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
128
128
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).
129
129
- **Debug logging** — opt-in structured logging on the `apify_client` logger captures request URLs, status codes, retry attempts, and more ([Logging](https://docs.apify.com/api/client/python/docs/concepts/logging)).
130
130
@@ -192,7 +192,7 @@ The full documentation lives at **[docs.apify.com/api/client/python](https://doc
192
192
| [Introduction](https://docs.apify.com/api/client/python/docs) | Overview, prerequisites, and a tour of the client. |
193
193
| [Quick start](https://docs.apify.com/api/client/python/docs/quick-start) | Authenticate, run an Actor, and fetch its results step by step. |
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), use HTTPX as the HTTP client. |
195
+
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), build a custom HTTP client. |
196
196
| [Upgrading](https://docs.apify.com/api/client/python/docs/upgrading/upgrading-to-v3) | Migrating between major versions. |
197
197
| [API reference](https://docs.apify.com/api/client/python/reference) | Generated reference for every class, method, and model. |
198
198
| [Changelog](https://docs.apify.com/api/client/python/docs/changelog) | Release history and breaking changes. |
The Apify API client uses a pluggable HTTP client architecture. By default, it ships with an [Impit](https://github.com/apify/impit)-based HTTP client that handles retries, timeouts, passing headers, and more. You can replace it with your own implementation for use cases like custom logging, proxying, request modification, or integrating with a different HTTP library.
20
+
The Apify API client uses a pluggable HTTP layer. It ships with an [Impit](https://github.com/apify/impit)-based default
21
+
and accepts fully custom synchronous or asynchronous implementations.
21
22
22
23
## Default HTTP client
23
24
24
25
When you create an <ApiLinkto="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLinkto="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> instance, it automatically uses the built-in <ApiLinkto="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink> (or <ApiLinkto="class/ImpitHttpClientAsync">`ImpitHttpClientAsync`</ApiLink>). This default client provides:
25
26
26
27
- Automatic retries with exponential backoff for network errors, HTTP 429, and HTTP 5xx responses.
27
28
- Configurable timeouts.
28
-
-Preparing request data and headers according to the API requirements, including authentication.
29
-
-Collecting requests statistics for monitoring and debugging.
29
+
-Request compression and preparation of API-compatible data, query parameters, and headers, including authentication.
30
+
-API error handling, structured logging, and request statistics.
30
31
31
32
You can configure the default client through the <ApiLinkto="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLinkto="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor:
32
33
@@ -45,33 +46,53 @@ You can configure the default client through the <ApiLink to="class/ApifyClient"
45
46
46
47
## Architecture
47
48
48
-
The HTTP client system is built on two key abstractions:
49
+
Internally, the HTTP client hierarchy has three layers:
50
+
51
+
- A common internal base contains configuration and utilities shared by synchronous and asynchronous clients, including
52
+
headers, request-body preparation, parameters, compression, and timeout tiers. It is not a public extension point.
53
+
- <ApiLinkto="class/HttpClient">`HttpClient`</ApiLink> and <ApiLinkto="class/HttpClientAsync">`HttpClientAsync`</ApiLink>
54
+
add the synchronous or asynchronous request pipeline, retry loop, transport hooks, and lifecycle interface.
55
+
- The built-in Impit classes inherit directly from the corresponding sync or async class and adapt the underlying
56
+
transport.
57
+
58
+
`HttpClient.is_timeout_error(exc)` and `HttpClientAsync.is_timeout_error(exc)` provide the public, transport-neutral way
59
+
to determine whether an exception is a timeout. Their shared implementation recognizes Python's `TimeoutError`;
60
+
transport adapters override it when their HTTP library defines additional timeout exception types. This lets
61
+
higher-level features such as streamed logs classify timeouts without depending on Impit or private implementation
62
+
details.
63
+
64
+
Responses use one separate abstraction:
49
65
50
-
- <ApiLinkto="class/HttpClient">`HttpClient`</ApiLink> / <ApiLinkto="class/HttpClientAsync">`HttpClientAsync`</ApiLink> - Abstract base classes that define the interface. Extend one of these to create a custom HTTP client by implementing the `call` method.
51
66
- <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> - A [runtime-checkable protocol](https://docs.python.org/3/library/typing.html#typing.runtime_checkable) that defines the expected response shape. Any object with the required attributes and methods satisfies the protocol — no inheritance needed.
52
67
53
68
To plug in your custom implementation, use the <ApiLinkto="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink> class method.
54
69
55
-
All of these are available as top-level imports from the `apify_client` package:
70
+
The built-in Impit classes are thin transport adapters over the request implementation in `HttpClient` and
71
+
`HttpClientAsync`. Custom transport adapters implement the request, error-classification, and lifecycle hooks. They
72
+
inherit request construction, retries, timeout growth, API error conversion, logging, and statistics from the base.
73
+
74
+
All of these are available from the `apify_client.http_clients` module:
56
75
57
76
<CodeBlockclassName="language-python">
58
77
{ArchitectureImportsExample}
59
78
</CodeBlock>
60
79
61
-
### The call method
80
+
### The transport contract
62
81
63
-
The `call` method receives all the information needed to make an HTTP request:
82
+
The public `call` method provides the shared request pipeline. A concrete transport implements these hooks:
-`params` - Query parameters to append to the URL.
69
-
-`data` - Raw request body (mutually exclusive with `json`).
70
-
-`json` - JSON-serializable request body (mutually exclusive with `data`).
71
-
-`stream` - Whether to stream the response body.
72
-
-`timeout` - Timeout for the request as a `timedelta`.
84
+
-`send_request(...)` sends one prepared request and returns an `HttpResponse`. The inherited `call` needs it, so
85
+
every transport adapter has to implement it.
86
+
-`is_retryable_transport_error(exc)` classifies transport failures for the shared retry loop. The default classifies
87
+
nothing as retryable, so a transport that skips it gives up on the first connection failure.
88
+
-`is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The
89
+
default recognizes Python's `TimeoutError`. Timeout classification is independent of retryability, so a timeout
90
+
the retry loop should retry has to be listed in `is_retryable_transport_error` too.
91
+
-`close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a
92
+
transport that owns no pool or session.
73
93
74
-
It must return an object satisfying the <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> protocol.
94
+
The `@override` decorators in the built-in Impit adapters make these implementations explicit and allow type checkers
95
+
to catch misspelled or incompatible overrides.
75
96
76
97
### The HTTP response protocol
77
98
@@ -93,6 +114,10 @@ It must return an object satisfying the <ApiLink to="class/HttpResponse">`HttpRe
93
114
94
115
:::note
95
116
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX](https://www.python-httpx.org/) already satisfy this protocol out of the box.
117
+
118
+
For a streamed response, consume the body inside its context manager with `iter_bytes()` / `aiter_bytes()`, or call
119
+
`read()` / `aread()` before accessing `content`. Some transports, including HTTPX, intentionally reject `content` on
120
+
an unread streamed response.
96
121
:::
97
122
98
123
### Plugging it in
@@ -115,18 +140,23 @@ Use the <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.wit
115
140
After that, all API calls made through the client will go through your custom HTTP client.
116
141
117
142
:::warning
118
-
When using a custom HTTP client, you are responsible for constructing the request, handling retries, timeouts, and errors yourself. The default retry logic is not applied.
143
+
If you override `call` itself, your implementation becomes responsible for request preparation, retries, timeouts, API
144
+
error conversion, logging, and statistics. Implementing the transport hooks and inheriting `call` keeps the shared
145
+
behavior.
119
146
:::
120
147
121
148
## Use cases
122
149
123
-
Custom HTTP clients might be useful when you need to:
150
+
Custom HTTP clients might be useful when the built-in Impit clients don't cover your requirements, for example when
151
+
you need to:
124
152
125
-
-**Use a different HTTP library** - Swap Impit for [httpx](https://www.python-httpx.org/), [requests](https://requests.readthedocs.io/), or [aiohttp](https://docs.aiohttp.org/).
153
+
-**Use a different HTTP library** - Integrate [httpx](https://www.python-httpx.org/), [requests](https://requests.readthedocs.io/), [aiohttp](https://docs.aiohttp.org/), or another transport.
126
154
-**Route through a proxy** - Add proxy support or request routing.
127
155
-**Implement custom retry logic** - Use different backoff strategies or retry conditions.
128
156
-**Log requests and responses** - Track API calls for debugging or auditing.
129
157
-**Modify requests** - Add custom fields, modify the body, or change headers.
130
158
-**Collect custom metrics** - Measure request latency, track error rates, or count API calls.
131
159
132
-
For a step-by-step walkthrough of building a custom HTTP client, see the [Using HTTPX as the HTTP client](/api/client/python/docs/guides/custom-http-client-httpx) guide.
160
+
For complete synchronous and asynchronous implementations over a transport with a different response API, see
161
+
[Build a custom HTTP client](../03_guides/05_custom_http_client.mdx). You can also refer to the
162
+
<ApiLinkto="class/HttpClient">`HttpClient` API reference</ApiLink> for the synchronous contract.
0 commit comments