Repository navigation
Commit 508f544
authored
test: deflake the persistent 429 stall test with a stopped throttler clock (#2331)
`test_persistent_429_raises_after_stall[retry_on_blocked]` failed on
ubuntu / Python 3.10
([run](https://github.com/apify/crawlee-python/actions/runs/38694178175/job/116286777865))
with `DID NOT RAISE PersistentRateLimitError`. The other 17 legs of the
same run passed.
The test put the domain into a 50 to 100 ms backoff and let real time
pass through a 1.2 s `max_domain_stall`. The crawler defers a 429 only
if `is_throttled()` still holds when it handles the error, and on a
loaded runner the backoff ran out first. With `max_request_retries=0`
the request then took the ordinary-error path, got marked as handled,
and the run ended without an error. A 120 ms delay injected into that
check reproduces the failure on every run.
The throttler now runs on a stopped `Mock` clock with a 1-minute
backoff, as in #2275. The clock moves one backoff forward each time the
idle crawler asks for a stall reason, and `max_domain_stall` sits
between one and two backoffs, so the domain stalls on the check after
its second 429. The assertions are unchanged, and the test still fails
when `_is_held_back` is forced to `False`. It also finishes in well
under a second.
The stall tests in the basic crawler suite keep the 50 ms backoff. They
run with the default three retries, so a lost race there only spends one
of the retries.
*✍️ Drafted by Claude Code*1 parent d5e8f71 commit 508f544
1 file changed
Lines changed: 20 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
| 16 | + | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| |||
750 | 750 | | |
751 | 751 | | |
752 | 752 | | |
| 753 | + | |
753 | 754 | | |
754 | 755 | | |
755 | 756 | | |
756 | 757 | | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
757 | 765 | | |
758 | 766 | | |
759 | 767 | | |
760 | 768 | | |
761 | | - | |
762 | | - | |
763 | | - | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
764 | 772 | | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
765 | 780 | | |
766 | 781 | | |
767 | 782 | | |
| |||
0 commit comments