Skip to content

Commit 4281530

Browse files
authored
fix: Warn when a requested timeout is capped at timeout_max (#962)
An explicit per-call `timeout` timedelta larger than `timeout_max` (default 360 s), or a tier configured above it, is clamped to `timeout_max` on every attempt. For example, `dataset.get_items_as_bytes(timeout=timedelta(minutes=30))` runs every attempt with a 360 s timeout and surfaces as repeated `impit.TimeoutException`, with no hint that the requested 30 minutes never took effect. The behavior stays unchanged: `timeout_max` remains a hard ceiling for any single request attempt. Only visibility changes. `_compute_timeout` now logs a warning when the resolved base timeout exceeds `timeout_max`, naming both values and pointing at `timeout_max` as the knob to raise. The warning goes through `LoggerOnce`, a small dedup helper ported from Crawlee for Python, so a recurring cut-off is reported once per timeout kind rather than on every attempt of every call. Docs and docstrings now state that the cap applies to tier and per-call timeouts alike. *✍️ Drafted by Claude Code*
1 parent e69e837 commit 4281530

7 files changed

Lines changed: 181 additions & 16 deletions

File tree

‎docs/02_concepts/11_timeouts.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Every client method has a pre-assigned tier that matches the expected duration o
2525

2626
## Configuring default timeouts
2727

28-
You can override the default values for each tier in the <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor. The `timeout_max` parameter sets an upper cap on the timeout for any individual API request, limiting exponential growth during retries.
28+
You can override the default values for each tier in the <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor. The `timeout_max` parameter sets an upper cap on the timeout for any individual API request, limiting exponential growth during retries. It caps tier and per-call timeouts as well, so raise it whenever you need a timeout longer than the default 360 seconds. A tier configured above the cap is capped too, and the client logs a warning to make the cut-off visible.
2929

3030
<Tabs>
3131
<TabItem value="AsyncExample" label="Async client" default>
@@ -57,6 +57,8 @@ client.dataset('id').list_items(timeout='long')
5757
client.dataset('id').list_items(timeout='no_timeout')
5858
```
5959

60+
A `timedelta` longer than `timeout_max` is capped at `timeout_max`, and the client logs a warning. To let such a call use its full timeout, raise `timeout_max` in the client constructor.
61+
6062
## Interaction with retries
6163

6264
Timeouts work together with the [retry system](/api/client/python/docs/concepts/retries). When a request times out, it counts as a failed attempt and triggers a retry (up to `max_retries`). The timeout applies to each individual attempt, not the total time across all retries.

‎src/apify_client/_apify_client.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ def __init__(
146146
timeout_short: Default timeout for short-duration API operations (simple CRUD operations, ...).
147147
timeout_medium: Default timeout for medium-duration API operations (batch operations, listing, ...).
148148
timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...).
149-
timeout_max: Maximum timeout cap for exponential timeout growth across retries.
149+
timeout_max: Maximum timeout cap for any single request attempt, including tier and per-call timeouts.
150150
headers: Additional HTTP headers to include in all API requests.
151151
compression: Compression algorithm for request bodies. Pass a string literal to select an algorithm,
152152
or an `HttpCompressor` instance for finer-grained control.
@@ -508,7 +508,7 @@ def __init__(
508508
timeout_short: Default timeout for short-duration API operations (simple CRUD operations, ...).
509509
timeout_medium: Default timeout for medium-duration API operations (batch operations, listing, ...).
510510
timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...).
511-
timeout_max: Maximum timeout cap for exponential timeout growth across retries.
511+
timeout_max: Maximum timeout cap for any single request attempt, including tier and per-call timeouts.
512512
headers: Additional HTTP headers to include in all API requests.
513513
compression: Compression algorithm for request bodies. Pass a string literal to select an algorithm,
514514
or an `HttpCompressor` instance for finer-grained control.

‎src/apify_client/_logging.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import functools
44
import inspect
55
import logging
6+
import threading
67
from contextvars import ContextVar
78
from typing import TYPE_CHECKING, Any, NamedTuple
89

@@ -42,6 +43,38 @@ class LogContext(NamedTuple):
4243
)
4344

4445

46+
class LoggerOnce:
47+
"""Emits each log message at most once, keyed by an explicit string.
48+
49+
Useful for diagnostic warnings that would otherwise spam the log when the same condition recurs (per-request
50+
misconfiguration warnings, repeated fallback paths, etc.). Deduplication scope follows the lifetime of the
51+
instance - a module-level instance gives process-wide dedup; an attribute on a class gives per-instance dedup.
52+
53+
Safe to call from multiple threads, since the sync client can be used from several threads at once.
54+
"""
55+
56+
def __init__(self, logger: logging.Logger) -> None:
57+
self._logger = logger
58+
self._seen: set[str] = set()
59+
self._lock = threading.Lock()
60+
61+
def log(self, message: str, *, key: str, level: int = logging.INFO) -> None:
62+
"""Log `message` at `level` the first time `key` is seen on this instance; later calls are no-ops.
63+
64+
Args:
65+
message: The message to log.
66+
key: Deduplication key. Two calls with the same key emit at most once.
67+
level: Standard `logging` level (e.g. `logging.WARNING`). Defaults to `logging.INFO`.
68+
"""
69+
# The check and the insert have to be atomic, otherwise two threads racing on the same key both emit.
70+
with self._lock:
71+
if key in self._seen:
72+
return
73+
self._seen.add(key)
74+
75+
self._logger.log(level, message)
76+
77+
4578
class WithLogDetailsClient(type):
4679
"""Metaclass that wraps public methods to inject client details into log context."""
4780

‎src/apify_client/http_clients/_base.py‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from __future__ import annotations
22

33
import json as jsonlib
4+
import logging
45
import os
56
import sys
67
from abc import ABC, abstractmethod
@@ -19,6 +20,7 @@
1920
MIN_COMPRESSION_SIZE,
2021
)
2122
from apify_client._docs import docs_group
23+
from apify_client._logging import LoggerOnce, logger_name
2224
from apify_client._statistics import ClientStatistics
2325
from apify_client._utils.http import is_compressible_content_type
2426
from apify_client._utils.time import to_seconds
@@ -30,6 +32,9 @@
3032
from apify_client.http_compressors._base import HttpCompressor
3133
from apify_client.types import JsonSerializable, Timeout
3234

35+
logger = logging.getLogger(logger_name)
36+
logger_once = LoggerOnce(logger)
37+
3338

3439
@docs_group('HTTP clients')
3540
@runtime_checkable
@@ -112,7 +117,7 @@ def __init__(
112117
timeout_short: Default timeout for short-duration API operations (simple CRUD operations, ...).
113118
timeout_medium: Default timeout for medium-duration API operations (batch operations, listing, ...).
114119
timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...).
115-
timeout_max: Maximum timeout cap for exponential timeout growth across retries.
120+
timeout_max: Maximum timeout cap for any single request attempt, including tier and per-call timeouts.
116121
max_retries: Maximum number of retries for failed requests.
117122
min_delay_between_retries: Minimum delay between retries.
118123
statistics: Statistics tracker for API calls. Created automatically if not provided.
@@ -205,7 +210,8 @@ def _compute_timeout(self, timeout: Timeout, *, attempt: int) -> int | float | N
205210
"""Resolve a timeout tier and compute the timeout for a request attempt with exponential increase.
206211
207212
For `no_timeout`, returns `None` to indicate no timeout. For tier literals and explicit `timedelta` values,
208-
doubles the timeout with each attempt but caps at `timeout_max`.
213+
doubles the timeout with each attempt but caps at `timeout_max`. A base timeout above `timeout_max` is
214+
capped too, which warns once per timeout kind since the requested value does not take effect in full.
209215
210216
Args:
211217
timeout: The timeout specification to resolve (tier literal or explicit `timedelta`).
@@ -226,6 +232,16 @@ def _compute_timeout(self, timeout: Timeout, *, attempt: int) -> int | float | N
226232
else:
227233
resolved = timeout
228234

235+
if resolved > self._timeout_max:
236+
# Keyed per timeout kind, so retries and repeated calls do not spam the log with the same warning.
237+
logger_once.log(
238+
f'The requested timeout of {to_seconds(resolved)}s exceeds `timeout_max` '
239+
f'({to_seconds(self._timeout_max)}s) and is capped at it. Raise `timeout_max` on the client '
240+
'to allow longer request timeouts.',
241+
key=f'timeout-capped-{timeout if isinstance(timeout, str) else "explicit"}',
242+
level=logging.WARNING,
243+
)
244+
229245
new_timeout = min(resolved * (2 ** (attempt - 1)), self._timeout_max)
230246
return to_seconds(new_timeout)
231247

@@ -350,8 +366,8 @@ def call(
350366
json: JSON-serializable data for the request body. Cannot be used together with data.
351367
stream: Whether to stream the response body.
352368
timeout: Timeout for the API HTTP request. Use `short`, `medium`, or `long` tier literals for
353-
preconfigured timeouts. A `timedelta` overrides it for this call, and `no_timeout` disables
354-
the timeout entirely.
369+
preconfigured timeouts. A `timedelta` overrides it for this call (capped at `timeout_max`), and
370+
`no_timeout` disables the timeout entirely.
355371
356372
Returns:
357373
The HTTP response object.
@@ -394,8 +410,8 @@ async def call(
394410
json: JSON-serializable data for the request body. Cannot be used together with data.
395411
stream: Whether to stream the response body.
396412
timeout: Timeout for the API HTTP request. Use `short`, `medium`, or `long` tier literals for
397-
preconfigured timeouts. A `timedelta` overrides it for this call, and `no_timeout` disables
398-
the timeout entirely.
413+
preconfigured timeouts. A `timedelta` overrides it for this call (capped at `timeout_max`), and
414+
`no_timeout` disables the timeout entirely.
399415
400416
Returns:
401417
The HTTP response object.

‎src/apify_client/http_clients/_impit.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ def __init__(
8383
timeout_short: Default timeout for short-duration API operations (simple CRUD operations, ...).
8484
timeout_medium: Default timeout for medium-duration API operations (batch operations, listing, ...).
8585
timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...).
86-
timeout_max: Maximum timeout cap for exponential timeout growth across retries.
86+
timeout_max: Maximum timeout cap for any single request attempt, including tier and per-call timeouts.
8787
max_retries: Maximum number of retry attempts for failed requests.
8888
min_delay_between_retries: Minimum delay between retries (increases exponentially with each attempt).
8989
statistics: Statistics tracker for API calls. Created automatically if not provided.
@@ -130,8 +130,8 @@ def call(
130130
json: JSON-serializable data for the request body. Cannot be used together with data.
131131
stream: Whether to stream the response body.
132132
timeout: Timeout for the API HTTP request. Use `short`, `medium`, or `long` tier literals for
133-
preconfigured timeouts. A `timedelta` overrides it for this call, and `no_timeout` disables
134-
the timeout entirely.
133+
preconfigured timeouts. A `timedelta` overrides it for this call (capped at `timeout_max`), and
134+
`no_timeout` disables the timeout entirely.
135135
136136
Returns:
137137
The HTTP response object.
@@ -332,7 +332,7 @@ def __init__(
332332
timeout_short: Default timeout for short-duration API operations (simple CRUD operations, ...).
333333
timeout_medium: Default timeout for medium-duration API operations (batch operations, listing, ...).
334334
timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...).
335-
timeout_max: Maximum timeout cap for exponential timeout growth across retries.
335+
timeout_max: Maximum timeout cap for any single request attempt, including tier and per-call timeouts.
336336
max_retries: Maximum number of retry attempts for failed requests.
337337
min_delay_between_retries: Minimum delay between retries (increases exponentially with each attempt).
338338
statistics: Statistics tracker for API calls. Created automatically if not provided.
@@ -379,8 +379,8 @@ async def call(
379379
json: JSON-serializable data for the request body. Cannot be used together with data.
380380
stream: Whether to stream the response body.
381381
timeout: Timeout for the API HTTP request. Use `short`, `medium`, or `long` tier literals for
382-
preconfigured timeouts. A `timedelta` overrides it for this call, and `no_timeout` disables
383-
the timeout entirely.
382+
preconfigured timeouts. A `timedelta` overrides it for this call (capped at `timeout_max`), and
383+
`no_timeout` disables the timeout entirely.
384384
385385
Returns:
386386
The HTTP response object.

‎tests/unit/test_client_timeouts.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,22 @@
11
from __future__ import annotations
22

3+
import logging
34
from datetime import timedelta
45
from typing import TYPE_CHECKING, Any
56
from unittest.mock import Mock
67

78
import pytest
89
from impit import HTTPError, Response, TimeoutException
910

11+
from apify_client._logging import LoggerOnce, logger_name
1012
from apify_client.http_clients import ImpitHttpClient, ImpitHttpClientAsync
13+
from apify_client.http_clients import _base as http_client_base
1114

1215
if TYPE_CHECKING:
1316
from collections.abc import Iterator
1417

18+
from _pytest.logging import LogCaptureFixture
19+
1520

1621
class EndOfTestError(Exception):
1722
"""Custom exception that is raised after the relevant part of the code is executed to stop the test."""
@@ -34,6 +39,12 @@ async def mock_request_async(*args: Any, **kwargs: Any) -> None:
3439
monkeypatch.undo()
3540

3641

42+
@pytest.fixture
43+
def fresh_logger_once(monkeypatch: pytest.MonkeyPatch) -> None:
44+
"""Replace the module-level `LoggerOnce`, whose dedup state would otherwise leak between tests."""
45+
monkeypatch.setattr(http_client_base, 'logger_once', LoggerOnce(http_client_base.logger))
46+
47+
3748
def test_no_timeout_passes_large_value_to_impit_sync(patch_request: list) -> None:
3849
"""Test that `no_timeout` passes a large timeout to impit to effectively disable the timeout."""
3950
client = ImpitHttpClient(timeout_short=timedelta(seconds=10))
@@ -149,6 +160,47 @@ def test_compute_timeout_no_timeout_returns_none() -> None:
149160
assert client._compute_timeout('no_timeout', attempt=1) is None
150161

151162

163+
@pytest.mark.usefixtures('fresh_logger_once')
164+
def test_compute_timeout_explicit_timedelta_above_max_warns(caplog: LogCaptureFixture) -> None:
165+
"""Test an explicit timedelta larger than timeout_max is capped, and the cut-off is logged once."""
166+
client = ImpitHttpClient(timeout_max=timedelta(seconds=360))
167+
168+
with caplog.at_level(logging.WARNING, logger=logger_name):
169+
assert client._compute_timeout(timedelta(minutes=30), attempt=1) == 360.0
170+
# Retries and later calls recompute the timeout, which must not repeat the warning.
171+
assert client._compute_timeout(timedelta(minutes=30), attempt=2) == 360.0
172+
assert client._compute_timeout(timedelta(minutes=45), attempt=1) == 360.0
173+
174+
assert len(caplog.records) == 1
175+
assert '1800.0s exceeds `timeout_max` (360.0s)' in caplog.records[0].message
176+
177+
178+
@pytest.mark.usefixtures('fresh_logger_once')
179+
def test_compute_timeout_tier_above_max_warns(caplog: LogCaptureFixture) -> None:
180+
"""Test a tier configured larger than timeout_max is capped, and the cut-off is logged too."""
181+
client = ImpitHttpClient(timeout_long=timedelta(seconds=600), timeout_max=timedelta(seconds=360))
182+
183+
with caplog.at_level(logging.WARNING, logger=logger_name):
184+
assert client._compute_timeout('long', attempt=1) == 360.0
185+
186+
assert len(caplog.records) == 1
187+
assert '600.0s exceeds `timeout_max` (360.0s)' in caplog.records[0].message
188+
189+
190+
@pytest.mark.usefixtures('fresh_logger_once')
191+
def test_compute_timeout_within_max_does_not_warn(caplog: LogCaptureFixture) -> None:
192+
"""Test a base timeout within timeout_max is used as-is, without a warning."""
193+
client = ImpitHttpClient(timeout_long=timedelta(seconds=300), timeout_max=timedelta(seconds=360))
194+
195+
with caplog.at_level(logging.WARNING, logger=logger_name):
196+
assert client._compute_timeout(timedelta(seconds=120), attempt=1) == 120.0
197+
assert client._compute_timeout('long', attempt=1) == 300.0
198+
# Growth capped at `timeout_max` on a retry is expected, so it is not warned about.
199+
assert client._compute_timeout('long', attempt=2) == 360.0
200+
201+
assert caplog.records == []
202+
203+
152204
async def test_dynamic_timeout_async_client(monkeypatch: pytest.MonkeyPatch) -> None:
153205
"""Tests timeout values for request with retriable errors.
154206

‎tests/unit/test_logging.py‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
from werkzeug import Request, Response
1414

1515
from apify_client import ApifyClient, ApifyClientAsync
16-
from apify_client._logging import RedirectLogFormatter
16+
from apify_client._logging import LoggerOnce, RedirectLogFormatter
1717
from apify_client._status_message_watcher import StatusMessageWatcherBase
1818
from apify_client._streamed_log import StreamedLog, StreamedLogAsync, StreamedLogBase
1919

@@ -1004,3 +1004,65 @@ def generate_logs() -> Iterator[bytes]:
10041004
assert not error_records, f'async task logged an error on stream timeout: {[r.message for r in error_records]}'
10051005
# The line received before the timeout must still have been redirected.
10061006
assert any('ACTOR: still running' in record.message for record in caplog.records)
1007+
1008+
1009+
def test_logger_once_logs_the_first_call(caplog: LogCaptureFixture) -> None:
1010+
"""Test the first call with a given key is logged."""
1011+
logger = logging.getLogger('apify_client.tests.log_once_first')
1012+
logger_once = LoggerOnce(logger)
1013+
1014+
with caplog.at_level(logging.INFO, logger=logger.name):
1015+
logger_once.log('first', key='k1')
1016+
1017+
assert [record.getMessage() for record in caplog.records] == ['first']
1018+
1019+
1020+
def test_logger_once_suppresses_a_repeated_key(caplog: LogCaptureFixture) -> None:
1021+
"""Test a second call with an already seen key is a no-op."""
1022+
logger = logging.getLogger('apify_client.tests.log_once_repeated')
1023+
logger_once = LoggerOnce(logger)
1024+
1025+
with caplog.at_level(logging.INFO, logger=logger.name):
1026+
logger_once.log('first', key='k1')
1027+
logger_once.log('second', key='k1')
1028+
1029+
assert [record.getMessage() for record in caplog.records] == ['first']
1030+
1031+
1032+
def test_logger_once_logs_distinct_keys(caplog: LogCaptureFixture) -> None:
1033+
"""Test each distinct key is logged once on its own."""
1034+
logger = logging.getLogger('apify_client.tests.log_once_distinct')
1035+
logger_once = LoggerOnce(logger)
1036+
1037+
with caplog.at_level(logging.INFO, logger=logger.name):
1038+
logger_once.log('msg-a', key='k1')
1039+
logger_once.log('msg-b', key='k2')
1040+
1041+
assert [record.getMessage() for record in caplog.records] == ['msg-a', 'msg-b']
1042+
1043+
1044+
def test_logger_once_instances_keep_independent_state(caplog: LogCaptureFixture) -> None:
1045+
"""Test dedup state is per instance, so two instances sharing a logger both emit the same key."""
1046+
logger = logging.getLogger('apify_client.tests.log_once_independent')
1047+
logger_once_a = LoggerOnce(logger)
1048+
logger_once_b = LoggerOnce(logger)
1049+
1050+
with caplog.at_level(logging.INFO, logger=logger.name):
1051+
logger_once_a.log('from-a', key='k1')
1052+
logger_once_b.log('from-b', key='k1')
1053+
1054+
assert [record.getMessage() for record in caplog.records] == ['from-a', 'from-b']
1055+
1056+
1057+
def test_logger_once_logs_at_the_requested_level(caplog: LogCaptureFixture) -> None:
1058+
"""Test the level defaults to info and is honored when passed explicitly."""
1059+
logger = logging.getLogger('apify_client.tests.log_once_levels')
1060+
logger_once = LoggerOnce(logger)
1061+
1062+
with caplog.at_level(logging.DEBUG, logger=logger.name):
1063+
logger_once.log('default-msg', key='k_default')
1064+
logger_once.log('warn-msg', key='k_warn', level=logging.WARNING)
1065+
1066+
levels = {record.getMessage(): record.levelno for record in caplog.records}
1067+
assert levels['default-msg'] == logging.INFO
1068+
assert levels['warn-msg'] == logging.WARNING

0 commit comments

Comments
 (0)