Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@
uv add "apify-client[brotli]"
```

To compress with `zstd`, pass `compression='zstd'`. Python 3.14 and newer have it in the standard library. On older
versions, install the optional `zstd` extra: `pip install "apify-client[zstd]"`.

[Impit](https://github.com/apify/impit) is the default HTTP client and is installed automatically. To use the
built-in [HTTPX2](https://github.com/pydantic/httpx2) client instead, install the optional `httpx2` extra, which
provides Pydantic's maintained continuation of HTTPX, and pass `http_client=Httpx2HttpClient()` to
Expand Down
2 changes: 2 additions & 0 deletions docs/01_introduction/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ For better request-body compression, opt in to `brotli`, which compresses better
</TabItem>
</Tabs>

To compress with `zstd` instead, pass `compression='zstd'`. On Python 3.14 and newer it uses the standard library. On older versions, install the optional `zstd` extra with `pip install "apify-client[zstd]"`.

For details, see [HTTP compression](../02_concepts/13_http_compression.mdx).

The client uses [Impit](https://github.com/apify/impit) as its default HTTP transport. To use the built-in [HTTPX2](https://github.com/pydantic/httpx2) transport, install the optional `httpx2` extra, which provides Pydantic's maintained continuation of HTTPX:
Expand Down
52 changes: 37 additions & 15 deletions docs/02_concepts/13_http_compression.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: http-compression
title: HTTP compression
description: The client compresses request bodies automatically using gzip by default, with optional brotli via an explicit opt-in.
description: The client compresses request bodies automatically using gzip by default, with optional brotli or zstd via an explicit opt-in.
---

import Tabs from '@theme/Tabs';
Expand All @@ -17,7 +17,7 @@ The Apify client compresses request bodies before sending them to the API. It re

## How it works

The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, the request carries no `Content-Encoding` of its own, and the body isn't streamed from a file or an iterator. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), [Pre-compressed bodies](#pre-compressed-bodies), and [Streaming uploads](./09_streaming.mdx#streaming-uploads).
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports gzip, brotli, and zstd and decompresses the request body transparently. A body is compressed only when it's large enough to benefit, its content type isn't already compressed, the request carries no `Content-Encoding` of its own, and the body isn't streamed from a file or an iterator. For details, see [Minimum body size](#minimum-body-size), [Already-compressed payloads](#already-compressed-payloads), [Pre-compressed bodies](#pre-compressed-bodies), and [Streaming uploads](./09_streaming.mdx#streaming-uploads).

## Minimum body size

Expand Down Expand Up @@ -66,7 +66,7 @@ A payload can reach the client already encoded, for example a gzipped file read
</TabItem>
</Tabs>

The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. It's forwarded for a streamed value too, so an open gzipped file can be passed in place of its bytes. The API accepts `gzip`, `br`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.
The header is forwarded verbatim, so it also covers encodings the client ships no compressor for, such as `deflate`. It's forwarded for a streamed value too, so an open gzipped file can be passed in place of its bytes. The API accepts `gzip`, `br`, `zstd`, `deflate`, and `identity`. Passing `identity` turns compression off for a single request without changing how the client is configured.

A value that can't be compressed at all - a string, an object serialized to JSON, or a file-like value opened in text mode - is rejected with a `TypeError` when `content_encoding` names a compression. Beyond that the client can't verify that the bytes match the header, so set `Content-Encoding` only when the payload really is encoded that way. Key-value store records are stored exactly as you upload them, which makes the header part of the stored record rather than a transport detail.

Expand All @@ -82,6 +82,9 @@ client = ApifyClient(token='MY-APIFY-TOKEN')

# Opt in to brotli (requires apify-client[brotli])
client = ApifyClient(token='MY-APIFY-TOKEN', compression='brotli')

# Opt in to zstd (requires apify-client[zstd] before Python 3.14)
client = ApifyClient(token='MY-APIFY-TOKEN', compression='zstd')
```

## Enabling brotli
Expand All @@ -96,19 +99,38 @@ uv add "apify-client[brotli]"

Then pass `compression='brotli'` to the client constructor. If you request brotli without installing the extra, the client raises a clear `ImportError`. There is no silent fallback.

## Enabling zstd

Python 3.14 and newer ship zstd in the standard library as `compression.zstd`, so `compression='zstd'` works there without an extra. On older Python versions, install the optional extra, which brings in the `backports.zstd` package:

```bash
pip install "apify-client[zstd]"
# or
uv add "apify-client[zstd]"
```

As with brotli, requesting zstd without the extra on Python older than 3.14, or on a Python 3.14 build without zstd support, raises a clear `ImportError`.

## Custom quality and advanced control

For fine-grained control over compression quality, inject an `HttpCompressor` instance directly instead of a string literal:

```python
from apify_client import ApifyClient
from apify_client.http_compressors import BrotliHttpCompressor, GzipHttpCompressor
from apify_client.http_compressors import (
BrotliHttpCompressor,
GzipHttpCompressor,
ZstdHttpCompressor,
)

# Brotli at maximum quality
client = ApifyClient(token='MY-APIFY-TOKEN', compression=BrotliHttpCompressor(quality=11))

# Gzip at maximum quality
client = ApifyClient(token='MY-APIFY-TOKEN', compression=GzipHttpCompressor(quality=9))

# Zstd at the highest accepted level
client = ApifyClient(token='MY-APIFY-TOKEN', compression=ZstdHttpCompressor(quality=19))
```

You can also implement a fully custom compressor by subclassing `HttpCompressor`. The client calls it only for bodies that reach the [minimum body size](#minimum-body-size) and aren't [already compressed](#already-compressed-payloads) or [pre-compressed by the caller](#pre-compressed-bodies):
Expand Down Expand Up @@ -139,17 +161,17 @@ client = ApifyClient(token='MY-APIFY-TOKEN', compression=IdentityCompressor())

## Comparison

| | Brotli | Gzip |
|-------------------------------|-----------------------------------------|---------------------------------|
| **Compression ratio** | Typically better than gzip | Good |
| **CPU cost** | Moderate, depends on quality | Low |
| **Availability** | Requires the `brotli` extra | Built-in, no extra needed |
| **`Content-Encoding` header** | `br` | `gzip` |
| **Quality range** | `0–11` | `1–9` |
| **Default quality** | `6` | `6` |
| **Enable via config** | `compression='brotli'` | `compression='gzip'` (default) |
| **Best for** | Large payloads where bandwidth matters | Minimal-dependency environments |
| | Brotli | Gzip | Zstd |
|-------------------------------|-----------------------------------------|---------------------------------|-------------------------------------------------------|
| **Compression ratio** | Typically better than gzip | Good | Close to brotli at high levels |
| **CPU cost** | Moderate, depends on quality | Low | Low at the default level, grows with it |
| **Availability** | Requires the `brotli` extra | Built-in, no extra needed | Built-in from Python 3.14, else the `zstd` extra |
| **`Content-Encoding` header** | `br` | `gzip` | `zstd` |
| **Quality range** | `0–11` | `1–9` | `1–19` |
| **Default quality** | `6` | `6` | `3` |
| **Enable via config** | `compression='brotli'` | `compression='gzip'` (default) | `compression='zstd'` |
| **Best for** | Large payloads where bandwidth matters | Minimal-dependency environments | Fast compression at a good ratio |

:::tip
For most workloads, the bandwidth savings from brotli outweigh the CPU costs. Install the `brotli` extra and pass `compression='brotli'` unless you can't install additional packages in your environment.
For most workloads, the bandwidth savings from brotli outweigh the CPU costs. Install the `brotli` extra and pass `compression='brotli'` unless you can't install additional packages in your environment. On Python 3.14 and newer, `compression='zstd'` is an alternative that needs no extra package.
:::
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ dependencies = [
[project.optional-dependencies]
brotli = ["brotli>=1.0.9"]
httpx2 = ["httpx2>=2.0.0"]
# zstd is in the standard library (`compression.zstd`) from Python 3.14; older versions use the `backports.zstd` package.
zstd = ["backports.zstd>=1.0.0; python_version < '3.14'"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uv.lock isn't updated for the new zstd extra, so uv lock --check fails. CI passes because it doesn't sync with --locked, but every uv sync rewrites the lock and leaves the tree dirty. Please run uv lock and commit the result.

✍️ Drafted by Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 19d5957: uv.lock now includes backports-zstd for the zstd extra, and uv lock --check passes.


[project.urls]
"Apify Homepage" = "https://apify.com"
Expand Down Expand Up @@ -205,6 +207,10 @@ exclude = ["website/versioned_docs"]
[tool.ty.rules]
unused-ignore-comment = "error"

[tool.ty.analysis]
# Installed only below Python 3.14 (the `zstd` extra), so it cannot resolve when type checking under 3.14 and newer.
allowed-unresolved-imports = ["backports.zstd"]

[[tool.ty.overrides]]
include = ["docs/**/*.py", "website/**/*.py"]
[tool.ty.overrides.rules]
Expand Down
14 changes: 10 additions & 4 deletions src/apify_client/_apify_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,14 @@ def __init__(
compression: Compression algorithm for request bodies. Defaults to `'gzip'`. Pass a string literal to select
an algorithm, or an `HttpCompressor` instance for finer-grained control. `'brotli'` needs the `brotli`
extra (`pip install "apify-client[brotli]"`). It is not the default because the `brotli` package is a C
extension, while gzip support is part of the Python standard library.
extension, while gzip support is part of the Python standard library. `'zstd'` uses the standard
library on Python 3.14 and newer when the Python build includes zstd, and needs the `zstd` extra
(`pip install "apify-client[zstd]"`) before that.

Raises:
ValueError: If `max_retries` is negative.
ImportError: If `compression` is `'brotli'` and the `brotli` extra is not installed.
ImportError: If `compression` is `'brotli'` or `'zstd'` and the extra it needs is not installed, or, for
`'zstd'` on Python 3.14 and newer, if the Python build lacks zstd support.
"""
if max_retries < 0:
raise ValueError(f'max_retries must be a non-negative integer, got {max_retries}')
Expand Down Expand Up @@ -540,11 +543,14 @@ def __init__(
compression: Compression algorithm for request bodies. Defaults to `'gzip'`. Pass a string literal to select
an algorithm, or an `HttpCompressor` instance for finer-grained control. `'brotli'` needs the `brotli`
extra (`pip install "apify-client[brotli]"`). It is not the default because the `brotli` package is a C
extension, while gzip support is part of the Python standard library.
extension, while gzip support is part of the Python standard library. `'zstd'` uses the standard
library on Python 3.14 and newer when the Python build includes zstd, and needs the `zstd` extra
(`pip install "apify-client[zstd]"`) before that.

Raises:
ValueError: If `max_retries` is negative.
ImportError: If `compression` is `'brotli'` and the `brotli` extra is not installed.
ImportError: If `compression` is `'brotli'` or `'zstd'` and the extra it needs is not installed, or, for
`'zstd'` on Python 3.14 and newer, if the Python build lacks zstd support.
"""
if max_retries < 0:
raise ValueError(f'max_retries must be a non-negative integer, got {max_retries}')
Expand Down
6 changes: 4 additions & 2 deletions src/apify_client/_resource_clients/actor_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,8 @@ def request(
Raises:
ApifyClientError: If the Actor is not a Server Actor.
NotFoundError: If the Actor does not exist.
ImportError: If `compression` needs an optional extra that is not installed.
ImportError: If `compression` needs an optional extra that is not installed, or, for `'zstd'` on Python 3.14
and newer, if the Python build lacks zstd support.
ValueError: If both `data` and `json` are passed, or `compression` is not a supported algorithm.
"""
server_url = self.get_url()
Expand Down Expand Up @@ -399,7 +400,8 @@ async def request(
Raises:
ApifyClientError: If the Actor is not a Server Actor.
NotFoundError: If the Actor does not exist.
ImportError: If `compression` needs an optional extra that is not installed.
ImportError: If `compression` needs an optional extra that is not installed, or, for `'zstd'` on Python 3.14
and newer, if the Python build lacks zstd support.
ValueError: If both `data` and `json` are passed, or `compression` is not a supported algorithm.
"""
server_url = await self.get_url()
Expand Down
12 changes: 6 additions & 6 deletions src/apify_client/_resource_clients/key_value_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -384,9 +384,9 @@ def set_record(
content_type: The content type of the saved value.
content_encoding: The encoding already applied to `value`, sent as the `Content-Encoding` header. Pass it
to upload a pre-compressed value - the client then forwards the bytes as they are instead of
compressing them itself. The API accepts `gzip`, `br`, `deflate`, and `identity`, and stores the
record exactly as uploaded, so this also becomes the encoding the record is served with. A `str`, a
JSON-serializable object, or a text-mode file cannot be carrying a compression and raises
compressing them itself. The API accepts `gzip`, `br`, `zstd`, `deflate`, and `identity`, and stores
the record exactly as uploaded, so this also becomes the encoding the record is served with. A `str`,
a JSON-serializable object, or a text-mode file cannot be carrying a compression and raises
`TypeError`; a streamed value is taken at its word, since its bytes are only seen as they are sent.
timeout: Timeout for the API HTTP request.
"""
Expand Down Expand Up @@ -828,9 +828,9 @@ async def set_record(
content_type: The content type of the saved value.
content_encoding: The encoding already applied to `value`, sent as the `Content-Encoding` header. Pass it
to upload a pre-compressed value - the client then forwards the bytes as they are instead of
compressing them itself. The API accepts `gzip`, `br`, `deflate`, and `identity`, and stores the
record exactly as uploaded, so this also becomes the encoding the record is served with. A `str`, a
JSON-serializable object, or a text-mode file cannot be carrying a compression and raises
compressing them itself. The API accepts `gzip`, `br`, `zstd`, `deflate`, and `identity`, and stores
the record exactly as uploaded, so this also becomes the encoding the record is served with. A `str`,
a JSON-serializable object, or a text-mode file cannot be carrying a compression and raises
`TypeError`; a streamed value is taken at its word, since its bytes are only seen as they are sent.
timeout: Timeout for the API HTTP request.
"""
Expand Down
4 changes: 2 additions & 2 deletions src/apify_client/_utils/http.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,8 @@ def to_path_segment(value: str) -> str:
def is_compressible_content_type(content_type: str | None) -> bool:
"""Decide whether a request body with the given content type is worth compressing.

Images, audio, video and archives already carry their own compression. Running them through gzip or brotli
burns CPU, holds a second full copy of the body in memory, and usually produces output slightly larger than
Images, audio, video and archives already carry their own compression. Running them through gzip, brotli, or
zstd burns CPU, holds a second full copy of the body in memory, and usually produces output slightly larger than
the input. Formats that are raw despite such a media type, for example `image/bmp` or `audio/wav`, are still
compressed. A body with no content type is assumed to be compressible.

Expand Down
20 changes: 12 additions & 8 deletions src/apify_client/_utils/try_import.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,28 @@ class ImportState:


@contextmanager
def try_import(module_name: str, *symbol_names: str, dependency_name: str, extra_name: str) -> Generator[ImportState]:
def try_import(
module_name: str, *symbol_names: str, dependency_name: str, extra_name: str | None
) -> Generator[ImportState]:
"""Context manager to attempt importing symbols into a module.

If the named optional dependency is missing, the symbols are replaced with `FailedImport` objects whose message
names the extra that installs it. Import errors caused by the importing module itself or by another dependency
are propagated instead of being masked.
names the extra that installs it, or is the import error's own message when `extra_name` is `None` because no extra
can install the dependency. A dotted `dependency_name` also counts as missing when its parent package is missing.
Import errors caused by the importing module itself or by another dependency are propagated instead of being masked.
"""
state = ImportState()
try:
yield state
except ModuleNotFoundError as exc:
if exc.name != dependency_name:
if exc.name is None or not (dependency_name == exc.name or dependency_name.startswith(f'{exc.name}.')):
raise
state.available = False
message = (
f"{exc.args[0]}. Install the optional '{extra_name}' extra to use it: "
f"pip install 'apify-client[{extra_name}]'"
)
message = exc.args[0]
if extra_name is not None:
message += (
f". Install the optional '{extra_name}' extra to use it: pip install 'apify-client[{extra_name}]'"
)
for symbol_name in symbol_names:
setattr(sys.modules[module_name], symbol_name, FailedImport(message))

Expand Down
Loading
Loading