Add spider retry middleware and politeness delays per domain

[kamsirichard]

Context

The spider occasionally fails on network timeouts or rate-limit responses (429) from source sites. These failures are silent — the spider just moves on, and some opportunities are missed. Scrapy has built-in retry and autothrottle middleware but it's not configured.

Task

Enable and tune Scrapy's built-in middleware in scoutbot/settings.py:

1. Retry middleware

RETRY_ENABLED = True
RETRY_TIMES   = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

2. AutoThrottle (politeness)

AUTOTHROTTLE_ENABLED     = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY   = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

3. Respect robots.txt

ROBOTSTXT_OBEY = True

(Check if any current sources would be blocked before enabling)

Files to touch

• scoutbot/settings.py

Notes

• Test by running scrapy crawl opportunities locally and checking for 429 or timeout errors
• AutoThrottle automatically slows down when servers are responding slowly

[BMmubeez]

To systematically handle persistent 429 status codes and timeout challenges during scaling, relying purely on basic retry loops or standard AutoThrottle settings can sometimes fall short if the target domain tracks IP reputations.

Here are a few architectural steps you can implement alongside the middleware upgrades:

1. Integrate Residential Proxy Pools: Standard datacenter IPs are easily flagged in batches. Utilizing an enterprise residential proxy network—like Magnetic Proxy—allows your spider traffic to route through authentic consumer ISP nodes, dynamically rotating exit IPs to bypass severe rate limits.
2. Configure Custom Retry Middleware: Ensure your middleware captures 429 and 503 status codes explicitly, triggering an exponential backoff strategy with randomized jitter to prevent hammering the destination server in uniform intervals.
3. Synchronize Headers and Fingerprints: Pair your request pacing with dynamic User-Agent rotation that matches valid TLS/JA4 browser fingerprints, avoiding sudden blocks from enterprise security layers.