Skip to content

Commit 9be215b

Browse files
authored
feat: Stream request bodies from files, iterables, and responses (#1060)
## Summary `set_record`, the `run_input` of `start` / `call` / `metamorph`, and `HttpClient.call(data=...)` accept an `io.IOBase` stream such as an open file, an iterable of `bytes` or `str` chunks, or a streamed `HttpResponse`. The client uploads it in chunks without holding it in memory, so one Actor's `OUTPUT` record can become another Actor's input. The async client also accepts async iterables and `aiofiles`-style readers. Streaming is experimental. The docs say so, and the first streamed body in a process emits a `UserWarning`. ## Behavior - An `io.IOBase` source is read in 64 KiB chunks. Any other object with a `read` method is read whole and sent as one chunk. - Only a seekable `io.IOBase` source is retried, rewound before each attempt. Other sources get a single attempt. - Streamed bodies are never compressed. A file-like `set_record` value used to be compressed, so uploading from a file handle now sends more bytes; `file.read()` keeps the old behavior. - If the API answers 408 because the body didn't arrive within about 5 minutes, the client logs a warning suggesting a temporary file. - `send_request` of a custom transport receives `bytes | Iterator[bytes] | None` (async: `AsyncIterator[bytes]`). It only sees an iterator when a caller passes a streamable value, so existing transports keep working. ## Docs and tests A "Streaming uploads" section on the streaming concept page, plus updates to the compression and HTTP client pages, the custom transport examples, and the README. Unit tests cover `StreamedRequestBody`, the retry pipeline, and `set_record` over Impit and HTTPX2; two integration tests cover a 3 MiB upload and a record piped between stores. ## Issues - Closes #972 *✍️ Drafted by Claude Code*
1 parent 4dcc54c commit 9be215b

33 files changed

Lines changed: 2488 additions & 213 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
133133
- **Fully typed responses** — every method returns a [Pydantic](https://docs.pydantic.dev/) model generated from the Apify OpenAPI spec, with IDE autocomplete and runtime validation ([Typed models](https://docs.apify.com/api/client/python/docs/concepts/typed-models)).
134134
- **Automatic retries** — exponential backoff for network errors, HTTP 429, and 5xx responses, configurable per client ([Retries](https://docs.apify.com/api/client/python/docs/concepts/retries)).
135135
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
136-
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
136+
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering, and upload large records straight from a file or another stream ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
137137
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
138138
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX2](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
139139
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).

‎docs/02_concepts/09_streaming.mdx‎

Lines changed: 86 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
id: streaming-resources
33
title: Streaming resources
4-
description: Stream large datasets, key-value store records, and logs without loading them into memory.
4+
description: Stream large datasets, key-value store records, and logs in both directions without loading them into memory.
55
---
66

77
import Tabs from '@theme/Tabs';
@@ -12,16 +12,22 @@ import ApiLink from '@theme/ApiLink';
1212

1313
import StreamingAsyncExample from '!!raw-loader!./code/09_streaming_async.py';
1414
import StreamingSyncExample from '!!raw-loader!./code/09_streaming_sync.py';
15+
import UploadFileAsyncExample from '!!raw-loader!./code/09_upload_file_async.py';
16+
import UploadFileSyncExample from '!!raw-loader!./code/09_upload_file_sync.py';
17+
import UploadGeneratorAsyncExample from '!!raw-loader!./code/09_upload_generator_async.py';
18+
import UploadGeneratorSyncExample from '!!raw-loader!./code/09_upload_generator_sync.py';
1519

16-
Certain resources, such as dataset items, key-value store records, and logs, support streaming directly from the Apify API. This allows you to process large resources incrementally without downloading them entirely into memory, making it ideal for handling large or continuously updated data.
20+
Large resources don't have to pass through memory whole. The client streams downloads of dataset items, key-value store records, and logs as they arrive, and it streams uploads of key-value store records and Actor inputs from a file or another stream as they're read. Both directions work with the synchronous and the asynchronous client. A download can also feed an upload directly, so one Actor's output becomes another Actor's input without passing through memory.
1721

18-
Supported streaming methods:
22+
## Streaming downloads
23+
24+
Dataset items, key-value store records, and logs can be streamed directly from the Apify API, so you process them incrementally instead of downloading them whole. The supported methods are:
1925

2026
- <ApiLink to="class/DatasetClient#stream_items">`DatasetClient.stream_items`</ApiLink> - Stream dataset items incrementally. Yields a raw streaming <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink>.
2127
- <ApiLink to="class/KeyValueStoreClient#stream_record">`KeyValueStoreClient.stream_record`</ApiLink> - Stream a key-value store record as raw data. Yields a `dict` with the `key`, `value`, and `content_type` fields, where `value` holds the raw streaming response, or `None` when the record doesn't exist.
2228
- <ApiLink to="class/LogClient#stream">`LogClient.stream`</ApiLink> - Stream logs in real time. Yields a raw streaming <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink>, or `None` when the log doesn't exist.
2329

24-
All three methods are context managers. Consume the streamed data within a `with` block to ensure that the connection is closed automatically, preventing memory leaks or unclosed connections.
30+
All three methods are context managers. Consume the streamed data within a `with` block, so the connection is closed automatically.
2531

2632
The following example shows how to stream the logs of an Actor run incrementally:
2733

@@ -38,4 +44,79 @@ The following example shows how to stream the logs of an Actor run incrementally
3844
</TabItem>
3945
</Tabs>
4046

41-
Streaming is ideal for processing large logs, datasets, or files incrementally without downloading them entirely into memory.
47+
## Streaming uploads
48+
49+
:::warning Experimental feature
50+
Streaming uploads are experimental. Their behavior and interface may change in future releases.
51+
:::
52+
53+
<ApiLink to="class/KeyValueStoreClient#set_record">`KeyValueStoreClient.set_record`</ApiLink> and the `run_input` of <ApiLink to="class/ActorClient#start">`ActorClient.start`</ApiLink>, <ApiLink to="class/ActorClient#call">`ActorClient.call`</ApiLink>, and <ApiLink to="class/RunClient#metamorph">`RunClient.metamorph`</ApiLink> accept a value the client streams to the API in chunks:
54+
55+
- An `io.IOBase` stream: an open file, an `io.BytesIO`, a member of a ZIP archive, or a pipe. The client reads it in chunks of 64 KiB, or 64 Ki characters for a text-mode file, from its current position and doesn't close it.
56+
- Any other object with a `read` method. The client can't tell how that `read` behaves, so it calls `read()` once and sends the result as a single chunk, which holds the whole source in memory.
57+
- An iterable of `bytes` or `str` chunks: an iterator such as a generator, or an object that only implements `__iter__`, for example a class that yields chunks from a generator method. The iterable decides the chunk sizes. An iterator is consumed by the upload, so passing the same one to a second upload sends an empty body. A generator is closed once the upload ends, even when the upload fails before reaching its end. A `str`, `bytes`, `list`, `tuple`, `dict`, or other container, and a pydantic model, are never streamed even though they can be iterated - they are the values the client uploads whole or serializes as JSON.
58+
- A streaming <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> from one of the download methods, whose body is forwarded chunk by chunk.
59+
60+
The asynchronous client also accepts an async iterable - an async generator or an object that only implements `__aiter__` - and an object with an `async def read`, such as a file opened with `aiofiles`, which is read whole like any other non-`io.IOBase` object. The synchronous client rejects those with a `TypeError`. A synchronous file or iterator works with both clients. The asynchronous client reads it in a worker thread, so the event loop stays free.
61+
62+
Apart from a non-`io.IOBase` object with a `read` method, only the chunk being sent is held in memory, so the process uploading a file doesn't need memory for the whole file. What the API accepts doesn't change: an Actor input is still capped at 9 MB, and a streamed body over that size is rejected with a `413` during the upload.
63+
64+
The following example uploads a file from disk. Both variants pass a plain file, which the asynchronous client reads in a worker thread.
65+
66+
<Tabs>
67+
<TabItem value="UploadFileAsyncExample" label="Async client" default>
68+
<CodeBlock className="language-python">
69+
{UploadFileAsyncExample}
70+
</CodeBlock>
71+
</TabItem>
72+
<TabItem value="UploadFileSyncExample" label="Sync client">
73+
<CodeBlock className="language-python">
74+
{UploadFileSyncExample}
75+
</CodeBlock>
76+
</TabItem>
77+
</Tabs>
78+
79+
With a generator, you produce the data while it uploads, for example from pages of a database query:
80+
81+
<Tabs>
82+
<TabItem value="UploadGeneratorAsyncExample" label="Async client" default>
83+
<CodeBlock className="language-python">
84+
{UploadGeneratorAsyncExample}
85+
</CodeBlock>
86+
</TabItem>
87+
<TabItem value="UploadGeneratorSyncExample" label="Sync client">
88+
<CodeBlock className="language-python">
89+
{UploadGeneratorSyncExample}
90+
</CodeBlock>
91+
</TabItem>
92+
</Tabs>
93+
94+
### Content type
95+
96+
A file the standard library opened in text mode is uploaded as `text/plain; charset=utf-8`, encoded to UTF-8 chunk by chunk. Any other streamed value is uploaded as `application/octet-stream` unless you pass `content_type`, an `aiofiles` text file included - the client can't tell it from a binary one. An iterator or a response carries no type of its own, so set `content_type` explicitly, as the examples do.
97+
98+
### Compression
99+
100+
A streamed body is never compressed, because the client sees each chunk only as it's sent. To upload compressed data, compress it yourself and declare the encoding with `content_encoding`. The client then forwards the bytes and the header as they are. For details, see [Pre-compressed bodies](./13_http_compression.mdx#pre-compressed-bodies).
101+
102+
### Retries
103+
104+
The client retries a failed request only when it can send the body again:
105+
106+
- A seekable `io.IOBase` stream, such as an open file or an `io.BytesIO`, is sought back to where it started before each retry, so the upload is retried like any other request.
107+
- An iterator, a streaming response, a pipe, an object with a `read` method outside `io.IOBase`, or any asynchronous source is consumed by the attempt that sends it. The request gets a single attempt, and a failure raises immediately.
108+
- A failure inside the source itself, such as a disk error while reading the file, is raised as that error and isn't retried, whatever the source.
109+
110+
If an upload from a source that can't be rewound needs retries, download the data to a temporary file first and upload the file. The object `tempfile.NamedTemporaryFile()` returns, which is also what `tempfile.TemporaryFile()` returns on Windows, isn't an `io.IOBase` stream, so pass its `file` attribute to have it read in chunks and retried. In the asynchronous client, prefer a file opened with `open()` over one opened with `aiofiles`: the plain file is read in chunks in a worker thread and can be sought back for a retry, while the `aiofiles` file is read whole in one call and gets a single attempt.
111+
112+
### Time limits
113+
114+
The API ends a request whose body doesn't arrive in full within about 5 minutes with a `408 Request Timeout`. The client doesn't retry the error and logs a warning that names the limit.
115+
116+
The client's own timeout applies to an upload as well. With the default Impit HTTP client, the timeout covers the whole request, so `set_record` gives up on an upload after its default of 360 seconds, and `ActorClient.start` after 30 seconds. For a slow upload that still fits within the API's limit, pass a longer `timeout`.
117+
118+
To upload data that a source produces slowly, such as the results of a long-running query, write it to a temporary file first and upload the file. The upload then runs at network speed, and a failed attempt can be retried. For details, see [Retries](#retries).
119+
120+
### Custom HTTP clients
121+
122+
A custom transport receives a streamed body as an iterator of `bytes` chunks, or as an async iterator in the asynchronous client, so any HTTP library that sends iterable bodies can send it. The <ApiLink to="class/StreamedRequestBody">`StreamedRequestBody`</ApiLink> class documents the rest of the contract. For details, see [The transport contract](./10_custom_http_clients.mdx#the-transport-contract).

‎docs/02_concepts/10_custom_http_clients.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ The base classes and the response protocol are available from the `apify_client.
9898

9999
The public `call` method provides the shared request pipeline. A concrete transport implements these hooks:
100100

101-
- `send_request(...)` sends one prepared request and returns an `HttpResponse`, error statuses included. It receives the URL with the query parameters already encoded into it, the headers with the client's default headers already merged in, the body already serialized and compressed, and the timeout for this attempt in seconds. The inherited `call` needs it, so every transport adapter has to implement it. Let the HTTP library's exceptions propagate unwrapped, and leave status handling and <ApiLink to="class/ApifyApiError">`ApifyApiError`</ApiLink> to `call`.
101+
- `send_request(...)` sends one prepared request and returns an `HttpResponse`, error statuses included. It receives the URL with the query parameters already encoded into it, the headers with the client's default headers already merged in, the body, and the timeout for this attempt in seconds. The body is either `bytes`, already serialized and compressed, or an iterator of `bytes` chunks for a streamed body, which is an async iterator in the asynchronous client. The inherited `call` needs it, so every transport adapter has to implement it. Let the HTTP library's exceptions propagate unwrapped, and leave status handling and <ApiLink to="class/ApifyApiError">`ApifyApiError`</ApiLink> to `call`.
102102
- `is_retryable_transport_error(exc)` classifies transport failures for the shared retry loop. The default classifies nothing as retryable, so a transport that doesn't override it gives up on the first connection failure.
103103
- `is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The default recognizes Python's `TimeoutError`. Timeout classification is independent of retryability, so a timeout the retry loop should retry has to be listed in `is_retryable_transport_error` too.
104104
- `close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a transport that owns no pool or session.
@@ -152,6 +152,8 @@ After that, all API calls made through the client will go through your custom HT
152152
If you override `call` itself, your implementation becomes responsible for request preparation, retries, timeouts, API error conversion, logging, and statistics. Implementing the transport hooks and inheriting `call` keeps the shared behavior.
153153
:::
154154

155+
If you override `call` and prepare requests with the inherited helpers, a streamed body arrives as a <ApiLink to="class/StreamedRequestBody">`StreamedRequestBody`</ApiLink>. Send its `iter_bytes()` or `aiter_bytes()`, and send the body a second time only when `rewindable` is true, after calling `rewind()`. For what the client streams and how, see [Streaming uploads](./09_streaming.mdx#streaming-uploads).
156+
155157
## Use cases
156158

157159
Custom HTTP clients might be useful when the built-in Impit and HTTPX2 clients don't cover your requirements, for example when you need to:

‎docs/02_concepts/13_http_compression.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ The Apify client compresses request bodies before sending them to the API. It re
1717

1818
## How it works
1919

20-
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, and the request carries no `Content-Encoding` of its own. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), and [Pre-compressed bodies](#pre-compressed-bodies).
20+
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, the request carries no `Content-Encoding` of its own, and the body isn't streamed from a file or an iterator. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), [Pre-compressed bodies](#pre-compressed-bodies), and [Streaming uploads](./09_streaming.mdx#streaming-uploads).
2121

2222
## Minimum body size
2323

@@ -47,7 +47,7 @@ Two kinds of media type are compressed anyway: raw formats such as `image/bmp`,
4747
</TabItem>
4848
</Tabs>
4949

50-
Without an explicit content type, a `bytes` value is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. File-like values are read into memory before they're sent, so they follow the same rules as any other body.
50+
Without an explicit content type, a `bytes` value is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. A streamed value, such as an open file or an iterator of chunks, is never compressed. For details, see [Streaming uploads](./09_streaming.mdx#streaming-uploads).
5151

5252
## Pre-compressed bodies
5353

@@ -66,7 +66,7 @@ A payload can reach the client already encoded, for example a gzipped file read
6666
</TabItem>
6767
</Tabs>
6868

69-
The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
69+
The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. It's forwarded for a streamed value too, so an open gzipped file can be passed in place of its bytes. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
7070

7171
A value that can't be compressed at all - a string, an object serialized to JSON, or a file-like value opened in text mode - is rejected with a `TypeError` when `content_encoding` names a compression. Beyond that the client can't verify that the bytes match the header, so set `Content-Encoding` only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail.
7272

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
import asyncio
2+
from pathlib import Path
3+
4+
from apify_client import ApifyClientAsync
5+
6+
TOKEN = 'MY-APIFY-TOKEN'
7+
8+
9+
async def main() -> None:
10+
apify_client = ApifyClientAsync(TOKEN)
11+
kvs_client = apify_client.key_value_store('MY-KVS-ID')
12+
13+
# The file is read in chunks in a worker thread as it uploads, so its size
14+
# doesn't matter.
15+
backup = await asyncio.to_thread(Path('backup.tar.gz').open, 'rb')
16+
with backup:
17+
await kvs_client.set_record(
18+
'backup.tar.gz', backup, content_type='application/gzip'
19+
)
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
from pathlib import Path
2+
3+
from apify_client import ApifyClient
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
def main() -> None:
9+
apify_client = ApifyClient(TOKEN)
10+
kvs_client = apify_client.key_value_store('MY-KVS-ID')
11+
12+
# The file is read in chunks as it uploads, so its size doesn't matter.
13+
with Path('backup.tar.gz').open('rb') as backup:
14+
kvs_client.set_record('backup.tar.gz', backup, content_type='application/gzip')

0 commit comments

Comments
 (0)