Skip to content

Commit 508f544

Browse files
authored
test: deflake the persistent 429 stall test with a stopped throttler clock (#2331)
`test_persistent_429_raises_after_stall[retry_on_blocked]` failed on ubuntu / Python 3.10 ([run](https://github.com/apify/crawlee-python/actions/runs/38694178175/job/116286777865)) with `DID NOT RAISE PersistentRateLimitError`. The other 17 legs of the same run passed. The test put the domain into a 50 to 100 ms backoff and let real time pass through a 1.2 s `max_domain_stall`. The crawler defers a 429 only if `is_throttled()` still holds when it handles the error, and on a loaded runner the backoff ran out first. With `max_request_retries=0` the request then took the ordinary-error path, got marked as handled, and the run ended without an error. A 120 ms delay injected into that check reproduces the failure on every run. The throttler now runs on a stopped `Mock` clock with a 1-minute backoff, as in #2275. The clock moves one backoff forward each time the idle crawler asks for a stall reason, and `max_domain_stall` sits between one and two backoffs, so the domain stalls on the check after its second 429. The assertions are unchanged, and the test still fails when `_is_held_back` is forced to `False`. It also finishes in well under a second. The stall tests in the basic crawler suite keep the 50 ms backoff. They run with the default three retries, so a lost race there only spends one of the retries. *✍️ Drafted by Claude Code*
1 parent d5e8f71 commit 508f544

1 file changed

Lines changed: 20 additions & 5 deletions

File tree

‎tests/unit/crawlers/_http/test_http_crawler.py‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import json
44
import logging
5-
from datetime import timedelta
5+
from datetime import datetime, timedelta, timezone
66
from typing import TYPE_CHECKING, Any
77
from unittest.mock import AsyncMock, Mock
88
from urllib.parse import parse_qs, urlencode
@@ -13,7 +13,7 @@
1313
from crawlee.crawlers import HttpCrawler
1414
from crawlee.errors import PersistentRateLimitError
1515
from crawlee.http_clients import HttpClient, HttpCrawlingResult
16-
from crawlee.request_loaders import ThrottlingRequestManager
16+
from crawlee.request_loaders import ThrottlingRequestManager, _throttling_request_manager
1717
from crawlee.sessions import SessionPool
1818
from crawlee.statistics import Statistics
1919
from crawlee.storages import RequestQueue
@@ -750,18 +750,33 @@ async def handler(context: HttpCrawlingContext) -> None:
750750
async def test_persistent_429_raises_after_stall(
751751
server_url: URL,
752752
caplog: pytest.LogCaptureFixture,
753+
monkeypatch: pytest.MonkeyPatch,
753754
*,
754755
retry_on_blocked: bool,
755756
) -> None:
756757
"""A domain that rate-limits every request for longer than `max_domain_stall` ends the run with an error."""
758+
# The throttler runs on a stopped clock that moves one backoff forward on each stall check. The crawler checks for
759+
# a stall only while idle, so a backoff can't run out before the crawler decides to defer a 429. The stall window
760+
# sits between one and two backoffs, so the domain counts as stalled on the check after its second 429.
761+
backoff = timedelta(minutes=1)
762+
clock = Mock()
763+
clock.now.return_value = datetime.now(timezone.utc)
764+
monkeypatch.setattr(_throttling_request_manager, 'datetime', clock)
757765
throttler = ThrottlingRequestManager(
758766
await RequestQueue.open(alias='throttle-429-inner'),
759767
domains=[server_url.host or ''],
760768
request_manager_opener=RequestQueue.open,
761-
base_delay=timedelta(milliseconds=50),
762-
max_delay=timedelta(milliseconds=100),
763-
max_domain_stall=timedelta(milliseconds=1200),
769+
base_delay=backoff,
770+
max_delay=backoff,
771+
max_domain_stall=backoff * 1.5,
764772
)
773+
get_stall_reason = throttler.get_stall_reason
774+
775+
async def get_stall_reason_after_backoff() -> str | None:
776+
clock.now.return_value += backoff
777+
return await get_stall_reason()
778+
779+
monkeypatch.setattr(throttler, 'get_stall_reason', get_stall_reason_after_backoff)
765780
failed_request_handler = AsyncMock()
766781
crawler = HttpCrawler(
767782
request_handler=AsyncMock(),

0 commit comments

Comments
 (0)