Skip to content

Commit 808bdde

Browse files
authored
perf: Skip request-body compression for already-compressed content types (#987)
### What `_prepare_request_call` now checks the request's `Content-Type` before compressing the body and skips compression for media types that already carry their own compression: - any `image/*`, `audio/*`, or `video/*` type, - archives (`application/zip`, `application/gzip`, `application/x-7z-compressed`, rar, bzip2, xz, zstd, jar), - web fonts (`font/woff`, `font/woff2`). Types with a `+json`/`+xml` structured suffix stay compressible, so `image/svg+xml` is not caught by the `image/` prefix. The media type is normalized before matching (parameters stripped, lowercased, trimmed). When compression is skipped, any caller-supplied `Content-Encoding` header is dropped as well — the body goes out verbatim, so the header would otherwise misdescribe it. ### Why The client compressed every request body unconditionally. For already-compressed payloads that burns CPU and holds a second full copy of the body in memory, while typically producing output slightly *larger* than the input. Key-value store records — screenshots, video, archives — are exactly this case. End-to-end `set_record` with a 200 MB incompressible payload: | Content type | Peak RSS | Wall / CPU | |---|---|---| | `application/octet-stream` (unchanged) | 604 MB - 3.02x | 7.8 s / 6.9 s user | | `image/png` (new path) | 403 MB - 2.02x | 1.05 s / 0.40 s user | `application/octet-stream` is deliberately not on the list: it is the catch-all for unknown binary, which may well be uncompressed data. It is also the encoder's default when no `content_type` is passed, so the optimization only applies when the caller sets an accurate content type — documented in the compression concept page. The remaining 2x comes from handing a `bytes` body to impit's `content=`, tracked separately in #972. ### Notes This overlaps with #934 (skip compression for small payloads) — both add a condition to the same block in `_prepare_request_call`. Whichever lands second needs the two conditions combined. *✍️ Drafted by Claude Code*
1 parent 7a5ed88 commit 808bdde

9 files changed

Lines changed: 303 additions & 15 deletions

File tree

‎docs/02_concepts/13_http_compression.mdx‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,48 @@
11
---
22
id: http-compression
33
title: HTTP compression
4-
description: The client compresses every request body automatically using gzip by default, with optional brotli via an explicit opt-in.
4+
description: The client compresses request bodies automatically using gzip by default, with optional brotli via an explicit opt-in.
55
---
66

7-
The Apify client compresses every request body before sending it to the API. It reduces the amount of data transferred over the network, resulting in faster requests and lower bandwidth usage, especially for large payloads such as Actor inputs, dataset uploads, or key-value store records.
7+
import Tabs from '@theme/Tabs';
8+
import TabItem from '@theme/TabItem';
9+
import CodeBlock from '@theme/CodeBlock';
10+
11+
import SkipCompressionAsyncExample from '!!raw-loader!./code/13_skip_compression_async.py';
12+
import SkipCompressionSyncExample from '!!raw-loader!./code/13_skip_compression_sync.py';
13+
14+
The Apify client compresses request bodies before sending them to the API. It reduces the amount of data transferred over the network, resulting in faster requests and lower bandwidth usage, especially for large payloads such as Actor inputs, dataset uploads, or key-value store records.
815

916
## How it works
1017

1118
The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently.
1219

20+
## Already-compressed payloads
21+
22+
Some payloads carry their own compression, so compressing them again costs CPU and memory while making the request slightly larger. The client skips compression when the request's `Content-Type` is one of these:
23+
24+
- any `image/*`, `audio/*`, or `video/*` type
25+
- archives such as `application/zip`, `application/gzip`, or `application/x-7z-compressed`
26+
- office documents and packages built on ZIP, such as `.docx`, `.xlsx`, `.epub`, or `.apk`
27+
- web fonts (`font/woff`, `font/woff2`)
28+
29+
Two kinds of media type are compressed anyway: raw formats such as `image/bmp`, `image/tiff`, and `audio/wav`, and subtypes with a structured syntax suffix such as `image/svg+xml`. Set an accurate `content_type` when uploading media to a key-value store:
30+
31+
<Tabs>
32+
<TabItem value="AsyncExample" label="Async client" default>
33+
<CodeBlock className="language-python">
34+
{SkipCompressionAsyncExample}
35+
</CodeBlock>
36+
</TabItem>
37+
<TabItem value="SyncExample" label="Sync client">
38+
<CodeBlock className="language-python">
39+
{SkipCompressionSyncExample}
40+
</CodeBlock>
41+
</TabItem>
42+
</Tabs>
43+
44+
Without an explicit content type, a `bytes` value is sent as `application/octet-stream`, which the client can't tell apart from uncompressed binary data and therefore still compresses. File-like values are streamed rather than buffered, so they're never compressed regardless of their content type.
45+
1346
## Configuration
1447

1548
To choose the compression algorithm, pass `compression` to the client constructor:
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import asyncio
2+
from pathlib import Path
3+
4+
from apify_client import ApifyClientAsync
5+
6+
TOKEN = 'MY-APIFY-TOKEN'
7+
8+
9+
async def main() -> None:
10+
apify_client = ApifyClientAsync(TOKEN)
11+
kvs_client = apify_client.key_value_store('MY-KVS-ID')
12+
13+
screenshot = await asyncio.to_thread(Path('screenshot.png').read_bytes)
14+
15+
# The explicit content type lets the client skip compressing the PNG.
16+
await kvs_client.set_record('screenshot', screenshot, content_type='image/png')
17+
18+
19+
if __name__ == '__main__':
20+
asyncio.run(main())
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
from pathlib import Path
2+
3+
from apify_client import ApifyClient
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
def main() -> None:
9+
apify_client = ApifyClient(TOKEN)
10+
kvs_client = apify_client.key_value_store('MY-KVS-ID')
11+
12+
screenshot = Path('screenshot.png').read_bytes()
13+
14+
# The explicit content type lets the client skip compressing the PNG.
15+
kvs_client.set_record('screenshot', screenshot, content_type='image/png')

‎src/apify_client/_consts.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,56 @@
3434

3535
OVERRIDABLE_DEFAULT_HEADERS = {'Accept', 'Authorization', 'Accept-Encoding', 'User-Agent'}
3636
"""Headers that can be overridden by users, but will trigger a warning if they do so, as it may lead to API errors."""
37+
38+
ALREADY_COMPRESSED_MEDIA_TYPE_PREFIXES = ('audio/', 'image/', 'video/')
39+
"""Media type prefixes whose payloads carry their own compression, so compressing the request body is wasted work."""
40+
41+
ALREADY_COMPRESSED_MEDIA_TYPES = frozenset(
42+
{
43+
'application/epub+zip',
44+
'application/gzip',
45+
'application/java-archive',
46+
'application/vnd.android.package-archive',
47+
'application/vnd.openxmlformats-officedocument.presentationml.presentation',
48+
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
49+
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
50+
'application/vnd.rar',
51+
'application/x-7z-compressed',
52+
'application/x-bzip',
53+
'application/x-bzip2',
54+
'application/x-gzip',
55+
'application/x-rar-compressed',
56+
'application/x-xz',
57+
'application/x-zip-compressed',
58+
'application/zip',
59+
'application/zstd',
60+
'font/woff',
61+
'font/woff2',
62+
}
63+
)
64+
"""Exact media types whose payloads carry their own compression."""
65+
66+
COMPRESSIBLE_MEDIA_TYPES = frozenset(
67+
{
68+
'audio/aiff',
69+
'audio/basic',
70+
'audio/l16',
71+
'audio/l24',
72+
'audio/midi',
73+
'audio/vnd.wave',
74+
'audio/wav',
75+
'audio/wave',
76+
'audio/x-aiff',
77+
'audio/x-wav',
78+
'image/bmp',
79+
'image/tiff',
80+
'image/vnd.adobe.photoshop',
81+
'image/vnd.microsoft.icon',
82+
'image/x-icon',
83+
'image/x-ms-bmp',
84+
}
85+
)
86+
"""Uncompressed media types that sit under an already-compressed prefix, so compressing them still pays off."""
87+
88+
COMPRESSIBLE_MEDIA_TYPE_SUFFIXES = ('+json', '+xml')
89+
"""Structured syntax suffixes marking a media type as text even under an already-compressed prefix (`image/svg+xml`)."""

‎src/apify_client/_utils/http.py‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,13 @@
33
import warnings
44
from typing import TYPE_CHECKING
55

6-
from apify_client._consts import OVERRIDABLE_DEFAULT_HEADERS
6+
from apify_client._consts import (
7+
ALREADY_COMPRESSED_MEDIA_TYPE_PREFIXES,
8+
ALREADY_COMPRESSED_MEDIA_TYPES,
9+
COMPRESSIBLE_MEDIA_TYPE_SUFFIXES,
10+
COMPRESSIBLE_MEDIA_TYPES,
11+
OVERRIDABLE_DEFAULT_HEADERS,
12+
)
713

814
if TYPE_CHECKING:
915
from apify_client.http_clients import HttpResponse
@@ -21,6 +27,34 @@ def to_safe_id(id: str) -> str:
2127
return id.replace('/', '~')
2228

2329

30+
def is_compressible_content_type(content_type: str | None) -> bool:
31+
"""Decide whether a request body with the given content type is worth compressing.
32+
33+
Images, audio, video and archives already carry their own compression. Running them through gzip or brotli
34+
burns CPU, holds a second full copy of the body in memory, and usually produces output slightly larger than
35+
the input. Formats that are raw despite such a media type, for example `image/bmp` or `audio/wav`, are still
36+
compressed. A body with no content type is assumed to be compressible.
37+
38+
Args:
39+
content_type: The value of the `Content-Type` header, if any.
40+
41+
Returns:
42+
`True` if the body should be compressed before it is sent.
43+
"""
44+
if not content_type:
45+
return True
46+
47+
# `Content-Type` is case-insensitive and may carry parameters, for example `text/plain; charset=utf-8`.
48+
media_type = content_type.split(';', 1)[0].strip().lower()
49+
50+
if media_type in COMPRESSIBLE_MEDIA_TYPES or media_type.endswith(COMPRESSIBLE_MEDIA_TYPE_SUFFIXES):
51+
return True
52+
53+
return not (
54+
media_type in ALREADY_COMPRESSED_MEDIA_TYPES or media_type.startswith(ALREADY_COMPRESSED_MEDIA_TYPE_PREFIXES)
55+
)
56+
57+
2458
def response_to_dict(response: HttpResponse) -> dict:
2559
"""Parse the API response as a dictionary and validate its type.
2660

‎src/apify_client/http_clients/_base.py‎

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
)
2020
from apify_client._docs import docs_group
2121
from apify_client._statistics import ClientStatistics
22+
from apify_client._utils.http import is_compressible_content_type
2223
from apify_client._utils.time import to_seconds
2324
from apify_client.http_compressors._gzip import GzipHttpCompressor
2425

@@ -232,29 +233,41 @@ def _prepare_request_call(
232233
) -> tuple[dict[str, str], dict[str, Any] | None, bytes | None]:
233234
"""Prepare headers, params, and body for an HTTP request.
234235
235-
Merges the client's default headers (including authorization) with per-request headers,
236-
serializes JSON and compresses the body. Header names are treated case-insensitively and
237-
per-request values win over the client defaults. For JSON bodies, a `Content-Type` header
238-
is set unless the caller supplied one.
236+
Merges the client's default headers (including authorization) with per-request headers, serializes JSON
237+
and compresses the body unless its content type says the payload is already compressed. Header names are
238+
treated case-insensitively and per-request values win over the client defaults. For JSON bodies, a
239+
`Content-Type` header is set unless the caller supplied one. `Content-Encoding` always describes what was
240+
actually applied to the body, so a caller-supplied value is dropped whenever nothing was compressed.
239241
"""
240242
if json is not None and data is not None:
241243
raise ValueError('Cannot pass both "json" and "data" parameters at the same time!')
242244

243245
headers = self._merge_headers(self._headers, headers)
244246

245-
# Dump JSON data to string so it can be compressed.
247+
# Dump JSON data to a string so it can be sent as a request body.
246248
if json is not None:
247249
data = jsonlib.dumps(json, ensure_ascii=False, allow_nan=False, default=str).encode('utf-8')
248250
if not any(key.lower() == 'content-type' for key in headers):
249251
headers['Content-Type'] = 'application/json'
250252

253+
compressed = False
254+
251255
if isinstance(data, (str, bytes, bytearray)):
252256
if isinstance(data, str):
253257
data = data.encode('utf-8')
254258
elif isinstance(data, bytearray):
255259
data = bytes(data)
256-
data = self._http_compressor.compress(data)
257-
headers = self._merge_headers(headers, {'Content-Encoding': self._http_compressor.content_encoding})
260+
261+
content_type = next((value for key, value in headers.items() if key.lower() == 'content-type'), None)
262+
if is_compressible_content_type(content_type):
263+
data = self._http_compressor.compress(data)
264+
headers = self._merge_headers(headers, {'Content-Encoding': self._http_compressor.content_encoding})
265+
compressed = True
266+
267+
# Anything left uncompressed goes out as-is - a file-like body included - so a caller-supplied encoding
268+
# would misdescribe it.
269+
if data is not None and not compressed:
270+
headers = {key: value for key, value in headers.items() if key.lower() != 'content-encoding'}
258271

259272
return (headers, self._parse_params(params), data)
260273

‎src/apify_client/http_clients/_impit.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -395,8 +395,8 @@ async def call(
395395
self._statistics.calls += 1
396396

397397
# Serializing and compressing a request body is CPU-bound and would block the event loop, so
398-
# offload request preparation to a worker thread whenever there is a body to compress. Bodyless
399-
# requests skip the thread hop, as they have no expensive work to move off the loop.
398+
# offload request preparation to a worker thread whenever there is a body. Bodyless requests
399+
# skip the thread hop, as they have no expensive work to move off the loop.
400400
if json is not None or data is not None:
401401
prepared_headers, prepared_params, content = await asyncio.to_thread(
402402
self._prepare_request_call,

‎tests/unit/test_http_clients.py‎

Lines changed: 80 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@
22

33
import asyncio
44
import gzip
5+
import io
56
import threading
67
import time
78
from datetime import UTC, datetime, timedelta
8-
from typing import TYPE_CHECKING
9+
from typing import TYPE_CHECKING, cast
910
from unittest.mock import AsyncMock, Mock
1011

1112
import brotli
@@ -461,6 +462,83 @@ def test_prepare_request_call_with_bytearray_data(compressor_case: tuple) -> Non
461462
assert decompress(data) == b'test bytearray'
462463

463464

465+
@pytest.mark.parametrize(
466+
'content_type',
467+
[
468+
pytest.param('image/png', id='image'),
469+
pytest.param('video/mp4', id='video'),
470+
pytest.param('application/zip', id='archive'),
471+
],
472+
)
473+
def test_prepare_request_call_skips_compression_for_already_compressed_content(content_type: str) -> None:
474+
"""An already-compressed body is sent verbatim, carries no `Content-Encoding`, and keeps every other header."""
475+
client = _ConcreteHttpClient(token='test_token', http_compressor=GzipHttpCompressor())
476+
477+
headers, _params, data = client._prepare_request_call(
478+
headers={'content-type': content_type},
479+
data=b'\x89PNG binary',
480+
)
481+
482+
assert data == b'\x89PNG binary'
483+
assert not any(key.lower() == 'content-encoding' for key in headers)
484+
assert headers['Authorization'] == 'Bearer test_token'
485+
assert headers['content-type'] == content_type
486+
assert headers['User-Agent'] == client._headers['User-Agent']
487+
488+
489+
def test_prepare_request_call_drops_caller_content_encoding_when_compression_is_skipped() -> None:
490+
"""Skipping compression also strips a caller-supplied `Content-Encoding`, which would misdescribe the body."""
491+
client = _ConcreteHttpClient(http_compressor=GzipHttpCompressor())
492+
493+
headers, _params, data = client._prepare_request_call(
494+
headers={'content-type': 'image/jpeg', 'content-encoding': 'br'},
495+
data=b'jpeg binary',
496+
)
497+
498+
assert data == b'jpeg binary'
499+
assert not any(key.lower() == 'content-encoding' for key in headers)
500+
501+
502+
def test_prepare_request_call_drops_caller_content_encoding_for_a_streamed_body() -> None:
503+
"""A body that is streamed rather than compressed, such as a file-like object, also loses `Content-Encoding`."""
504+
client = _ConcreteHttpClient(http_compressor=GzipHttpCompressor())
505+
stream = io.BytesIO(b'raw payload')
506+
507+
headers, _params, data = client._prepare_request_call(
508+
headers={'content-encoding': 'gzip'},
509+
data=cast('bytes', stream),
510+
)
511+
512+
assert data is stream
513+
assert not any(key.lower() == 'content-encoding' for key in headers)
514+
515+
516+
@pytest.mark.parametrize(
517+
'content_type',
518+
[
519+
pytest.param('image/svg+xml', id='structured xml suffix'),
520+
pytest.param('image/bmp', id='raw bitmap'),
521+
pytest.param('audio/wav', id='raw audio'),
522+
],
523+
)
524+
def test_prepare_request_call_compresses_exceptions_to_compressed_prefixes(
525+
content_type: str,
526+
compressor_case: tuple,
527+
) -> None:
528+
"""Types that are text or raw are compressed even when they sit under an already-compressed prefix."""
529+
compressor, content_encoding, decompress = compressor_case
530+
client = _ConcreteHttpClient(http_compressor=compressor)
531+
532+
headers, _params, data = client._prepare_request_call(
533+
headers={'content-type': content_type},
534+
data=b'raw payload',
535+
)
536+
537+
assert headers['Content-Encoding'] == content_encoding
538+
assert isinstance(data, bytes)
539+
assert decompress(data) == b'raw payload'
540+
541+
464542
def test_prepare_request_call_json_and_data_error() -> None:
465543
"""Test _prepare_request_call raises error when both json and data are provided."""
466544
client = _ConcreteHttpClient()
@@ -520,7 +598,7 @@ def test_prepare_request_call_json_keeps_caller_content_type() -> None:
520598

521599

522600
def test_prepare_request_call_replaces_caller_content_encoding() -> None:
523-
"""The Content-Encoding header always reflects the compressor actually applied, replacing any caller value."""
601+
"""A compressed body reports the compressor actually applied, replacing any caller-supplied Content-Encoding."""
524602
client = _ConcreteHttpClient(http_compressor=GzipHttpCompressor())
525603

526604
headers, _params, _data = client._prepare_request_call(headers={'content-encoding': 'br'}, data='payload')

0 commit comments

Comments
 (0)