You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9be215b
Browse filesBrowse the repository at this point in the historyBrowse files
feat: Stream request bodies from files, iterables, and responses (#1060)
## Summary
`set_record`, the `run_input` of `start` / `call` / `metamorph`, and
`HttpClient.call(data=...)` accept an `io.IOBase` stream such as an open
file, an iterable of `bytes` or `str` chunks, or a streamed
`HttpResponse`. The client uploads it in chunks without holding it in
memory, so one Actor's `OUTPUT` record can become another Actor's input.
The async client also accepts async iterables and `aiofiles`-style
readers.
Streaming is experimental. The docs say so, and the first streamed body
in a process emits a `UserWarning`.
## Behavior
- An `io.IOBase` source is read in 64 KiB chunks. Any other object with
a `read` method is read whole and sent as one chunk.
- Only a seekable `io.IOBase` source is retried, rewound before each
attempt. Other sources get a single attempt.
- Streamed bodies are never compressed. A file-like `set_record` value
used to be compressed, so uploading from a file handle now sends more
bytes; `file.read()` keeps the old behavior.
- If the API answers 408 because the body didn't arrive within about 5
minutes, the client logs a warning suggesting a temporary file.
- `send_request` of a custom transport receives `bytes | Iterator[bytes]
| None` (async: `AsyncIterator[bytes]`). It only sees an iterator when a
caller passes a streamable value, so existing transports keep working.
## Docs and tests
A "Streaming uploads" section on the streaming concept page, plus
updates to the compression and HTTP client pages, the custom transport
examples, and the README. Unit tests cover `StreamedRequestBody`, the
retry pipeline, and `set_record` over Impit and HTTPX2; two integration
tests cover a 3 MiB upload and a record piped between stores.
## Issues
- Closes#972
*✍️ Drafted by Claude Code*
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -133,7 +133,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
133
133
- **Fully typed responses** — every method returns a [Pydantic](https://docs.pydantic.dev/) model generated from the Apify OpenAPI spec, with IDE autocomplete and runtime validation ([Typed models](https://docs.apify.com/api/client/python/docs/concepts/typed-models)).
134
134
- **Automatic retries** — exponential backoff for network errors, HTTP 429, and 5xx responses, configurable per client ([Retries](https://docs.apify.com/api/client/python/docs/concepts/retries)).
135
135
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
136
-
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
136
+
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering, and upload large records straight from a file or another stream ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
137
137
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
138
138
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX2](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
139
139
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).
Certain resources, such as dataset items, key-value store records, and logs, support streaming directly from the Apify API. This allows you to process large resources incrementally without downloading them entirely into memory, making it ideal for handling large or continuously updated data.
20
+
Large resources don't have to pass through memory whole. The client streams downloads of dataset items, key-value store records, and logs as they arrive, and it streams uploads of key-value store records and Actor inputs from a file or another stream as they're read. Both directions work with the synchronous and the asynchronous client. A download can also feed an upload directly, so one Actor's output becomes another Actor's input without passing through memory.
17
21
18
-
Supported streaming methods:
22
+
## Streaming downloads
23
+
24
+
Dataset items, key-value store records, and logs can be streamed directly from the Apify API, so you process them incrementally instead of downloading them whole. The supported methods are:
19
25
20
26
- <ApiLinkto="class/DatasetClient#stream_items">`DatasetClient.stream_items`</ApiLink> - Stream dataset items incrementally. Yields a raw streaming <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink>.
21
27
- <ApiLinkto="class/KeyValueStoreClient#stream_record">`KeyValueStoreClient.stream_record`</ApiLink> - Stream a key-value store record as raw data. Yields a `dict` with the `key`, `value`, and `content_type` fields, where `value` holds the raw streaming response, or `None` when the record doesn't exist.
22
28
- <ApiLinkto="class/LogClient#stream">`LogClient.stream`</ApiLink> - Stream logs in real time. Yields a raw streaming <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink>, or `None` when the log doesn't exist.
23
29
24
-
All three methods are context managers. Consume the streamed data within a `with` block to ensure that the connection is closed automatically, preventing memory leaks or unclosed connections.
30
+
All three methods are context managers. Consume the streamed data within a `with` block, so the connection is closed automatically.
25
31
26
32
The following example shows how to stream the logs of an Actor run incrementally:
27
33
@@ -38,4 +44,79 @@ The following example shows how to stream the logs of an Actor run incrementally
38
44
</TabItem>
39
45
</Tabs>
40
46
41
-
Streaming is ideal for processing large logs, datasets, or files incrementally without downloading them entirely into memory.
47
+
## Streaming uploads
48
+
49
+
:::warning Experimental feature
50
+
Streaming uploads are experimental. Their behavior and interface may change in future releases.
51
+
:::
52
+
53
+
<ApiLinkto="class/KeyValueStoreClient#set_record">`KeyValueStoreClient.set_record`</ApiLink> and the `run_input` of <ApiLinkto="class/ActorClient#start">`ActorClient.start`</ApiLink>, <ApiLinkto="class/ActorClient#call">`ActorClient.call`</ApiLink>, and <ApiLinkto="class/RunClient#metamorph">`RunClient.metamorph`</ApiLink> accept a value the client streams to the API in chunks:
54
+
55
+
- An `io.IOBase` stream: an open file, an `io.BytesIO`, a member of a ZIP archive, or a pipe. The client reads it in chunks of 64 KiB, or 64 Ki characters for a text-mode file, from its current position and doesn't close it.
56
+
- Any other object with a `read` method. The client can't tell how that `read` behaves, so it calls `read()` once and sends the result as a single chunk, which holds the whole source in memory.
57
+
- An iterable of `bytes` or `str` chunks: an iterator such as a generator, or an object that only implements `__iter__`, for example a class that yields chunks from a generator method. The iterable decides the chunk sizes. An iterator is consumed by the upload, so passing the same one to a second upload sends an empty body. A generator is closed once the upload ends, even when the upload fails before reaching its end. A `str`, `bytes`, `list`, `tuple`, `dict`, or other container, and a pydantic model, are never streamed even though they can be iterated - they are the values the client uploads whole or serializes as JSON.
58
+
- A streaming <ApiLinkto="class/HttpResponse">`HttpResponse`</ApiLink> from one of the download methods, whose body is forwarded chunk by chunk.
59
+
60
+
The asynchronous client also accepts an async iterable - an async generator or an object that only implements `__aiter__` - and an object with an `async def read`, such as a file opened with `aiofiles`, which is read whole like any other non-`io.IOBase` object. The synchronous client rejects those with a `TypeError`. A synchronous file or iterator works with both clients. The asynchronous client reads it in a worker thread, so the event loop stays free.
61
+
62
+
Apart from a non-`io.IOBase` object with a `read` method, only the chunk being sent is held in memory, so the process uploading a file doesn't need memory for the whole file. What the API accepts doesn't change: an Actor input is still capped at 9 MB, and a streamed body over that size is rejected with a `413` during the upload.
63
+
64
+
The following example uploads a file from disk. Both variants pass a plain file, which the asynchronous client reads in a worker thread.
A file the standard library opened in text mode is uploaded as `text/plain; charset=utf-8`, encoded to UTF-8 chunk by chunk. Any other streamed value is uploaded as `application/octet-stream` unless you pass `content_type`, an `aiofiles` text file included - the client can't tell it from a binary one. An iterator or a response carries no type of its own, so set `content_type` explicitly, as the examples do.
97
+
98
+
### Compression
99
+
100
+
A streamed body is never compressed, because the client sees each chunk only as it's sent. To upload compressed data, compress it yourself and declare the encoding with `content_encoding`. The client then forwards the bytes and the header as they are. For details, see [Pre-compressed bodies](./13_http_compression.mdx#pre-compressed-bodies).
101
+
102
+
### Retries
103
+
104
+
The client retries a failed request only when it can send the body again:
105
+
106
+
- A seekable `io.IOBase` stream, such as an open file or an `io.BytesIO`, is sought back to where it started before each retry, so the upload is retried like any other request.
107
+
- An iterator, a streaming response, a pipe, an object with a `read` method outside `io.IOBase`, or any asynchronous source is consumed by the attempt that sends it. The request gets a single attempt, and a failure raises immediately.
108
+
- A failure inside the source itself, such as a disk error while reading the file, is raised as that error and isn't retried, whatever the source.
109
+
110
+
If an upload from a source that can't be rewound needs retries, download the data to a temporary file first and upload the file. The object `tempfile.NamedTemporaryFile()` returns, which is also what `tempfile.TemporaryFile()` returns on Windows, isn't an `io.IOBase` stream, so pass its `file` attribute to have it read in chunks and retried. In the asynchronous client, prefer a file opened with `open()` over one opened with `aiofiles`: the plain file is read in chunks in a worker thread and can be sought back for a retry, while the `aiofiles` file is read whole in one call and gets a single attempt.
111
+
112
+
### Time limits
113
+
114
+
The API ends a request whose body doesn't arrive in full within about 5 minutes with a `408 Request Timeout`. The client doesn't retry the error and logs a warning that names the limit.
115
+
116
+
The client's own timeout applies to an upload as well. With the default Impit HTTP client, the timeout covers the whole request, so `set_record` gives up on an upload after its default of 360 seconds, and `ActorClient.start` after 30 seconds. For a slow upload that still fits within the API's limit, pass a longer `timeout`.
117
+
118
+
To upload data that a source produces slowly, such as the results of a long-running query, write it to a temporary file first and upload the file. The upload then runs at network speed, and a failed attempt can be retried. For details, see [Retries](#retries).
119
+
120
+
### Custom HTTP clients
121
+
122
+
A custom transport receives a streamed body as an iterator of `bytes` chunks, or as an async iterator in the asynchronous client, so any HTTP library that sends iterable bodies can send it. The <ApiLinkto="class/StreamedRequestBody">`StreamedRequestBody`</ApiLink> class documents the rest of the contract. For details, see [The transport contract](./10_custom_http_clients.mdx#the-transport-contract).
Copy file name to clipboardExpand all lines: docs/02_concepts/10_custom_http_clients.mdx
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,7 +98,7 @@ The base classes and the response protocol are available from the `apify_client.
98
98
99
99
The public `call` method provides the shared request pipeline. A concrete transport implements these hooks:
100
100
101
-
-`send_request(...)` sends one prepared request and returns an `HttpResponse`, error statuses included. It receives the URL with the query parameters already encoded into it, the headers with the client's default headers already merged in, the bodyalready serialized and compressed, and the timeout for this attempt in seconds. The inherited `call` needs it, so every transport adapter has to implement it. Let the HTTP library's exceptions propagate unwrapped, and leave status handling and <ApiLinkto="class/ApifyApiError">`ApifyApiError`</ApiLink> to `call`.
101
+
-`send_request(...)` sends one prepared request and returns an `HttpResponse`, error statuses included. It receives the URL with the query parameters already encoded into it, the headers with the client's default headers already merged in, the body, and the timeout for this attempt in seconds. The body is either `bytes`, already serialized and compressed, or an iterator of `bytes` chunks for a streamed body, which is an async iterator in the asynchronous client. The inherited `call` needs it, so every transport adapter has to implement it. Let the HTTP library's exceptions propagate unwrapped, and leave status handling and <ApiLinkto="class/ApifyApiError">`ApifyApiError`</ApiLink> to `call`.
102
102
-`is_retryable_transport_error(exc)` classifies transport failures for the shared retry loop. The default classifies nothing as retryable, so a transport that doesn't override it gives up on the first connection failure.
103
103
-`is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The default recognizes Python's `TimeoutError`. Timeout classification is independent of retryability, so a timeout the retry loop should retry has to be listed in `is_retryable_transport_error` too.
104
104
-`close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a transport that owns no pool or session.
@@ -152,6 +152,8 @@ After that, all API calls made through the client will go through your custom HT
152
152
If you override `call` itself, your implementation becomes responsible for request preparation, retries, timeouts, API error conversion, logging, and statistics. Implementing the transport hooks and inheriting `call` keeps the shared behavior.
153
153
:::
154
154
155
+
If you override `call` and prepare requests with the inherited helpers, a streamed body arrives as a <ApiLinkto="class/StreamedRequestBody">`StreamedRequestBody`</ApiLink>. Send its `iter_bytes()` or `aiter_bytes()`, and send the body a second time only when `rewindable` is true, after calling `rewind()`. For what the client streams and how, see [Streaming uploads](./09_streaming.mdx#streaming-uploads).
156
+
155
157
## Use cases
156
158
157
159
Custom HTTP clients might be useful when the built-in Impit and HTTPX2 clients don't cover your requirements, for example when you need to:
Copy file name to clipboardExpand all lines: docs/02_concepts/13_http_compression.mdx
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ The Apify client compresses request bodies before sending them to the API. It re
17
17
18
18
## How it works
19
19
20
-
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, and the request carries no `Content-Encoding` of its own. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), and [Pre-compressed bodies](#pre-compressed-bodies).
20
+
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, the request carries no `Content-Encoding` of its own, and the body isn't streamed from a file or an iterator. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), [Pre-compressed bodies](#pre-compressed-bodies), and [Streaming uploads](./09_streaming.mdx#streaming-uploads).
21
21
22
22
## Minimum body size
23
23
@@ -47,7 +47,7 @@ Two kinds of media type are compressed anyway: raw formats such as `image/bmp`,
47
47
</TabItem>
48
48
</Tabs>
49
49
50
-
Without an explicit content type, a `bytes` value is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. File-like values are read into memory before they're sent, so they follow the same rules as any other body.
50
+
Without an explicit content type, a `bytes` value is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. A streamed value, such as an open file or an iterator of chunks, is never compressed. For details, see [Streaming uploads](./09_streaming.mdx#streaming-uploads).
51
51
52
52
## Pre-compressed bodies
53
53
@@ -66,7 +66,7 @@ A payload can reach the client already encoded, for example a gzipped file read
66
66
</TabItem>
67
67
</Tabs>
68
68
69
-
The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
69
+
The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. It's forwarded for a streamed value too, so an open gzipped file can be passed in place of its bytes. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
70
70
71
71
A value that can't be compressed at all - a string, an object serialized to JSON, or a file-like value opened in text mode - is rejected with a `TypeError` when `content_encoding` names a compression. Beyond that the client can't verify that the bytes match the header, so set `Content-Encoding` only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail.
0 commit comments