Skip to content

Apply crawl_delay per request instead of per page - #212

Open
thatbudakguy wants to merge 1 commit into
207-each-documentfrom
209-per-request-crawl-delay
Open

thatbudakguy wants to merge 1 commit into
207-each-documentfrom
209-per-request-crawl-delay

Conversation

@thatbudakguy

@thatbudakguy thatbudakguy commented Sep 17, 2026

Copy link
Copy Markdown
Member

Closes #209

Moves the requests a harvest makes into a GeoBlacklightHarvester::HttpClient that waits out the crawl delay before each request. Previously the delay was applied once per page of search results, so a harvest could issue 100 document requests back to back before it took effect. This resulted in ~20% of requests failing outright against our local firewall at Stanford, for example.

Each request continues to get its own connection rather than reusing one, which is an architectural difference from something like GitHub downloading, where we instead want a persistent connection pool (using Faraday). Firewalls are somewhat less likely to reject a request if it's a new request every time instead of a persistent connection making big bursts of requests.

Fractional delays work too, so you can tune things to get an acceptable timing for large harvests.

Harvest timing changes

Pacing per request is the point of #209, but it makes a harvest considerably slower than it was: a 1 second delay against a GeoBlacklight with 10,000 records is roughly three hours of waiting, where before it was ~100 page delays.

GeoCombine is much more useful when doing incremental harvesting instead of full dumps anyway, so the assumption is that longer delays are acceptable when you're pulling directly from a GeoBlacklight (especially if it's your own). The "happy path" is getting it from the OGM GitHub anyway, which is much faster.

@thatbudakguy
thatbudakguy added this pull request to stack #218 September 17, 2026 23:43
@thatbudakguy
thatbudakguy force-pushed the 209-per-request-crawl-delay branch from 24755ed to b9ac8e3 Compare September 17, 2026 23:43
@thatbudakguy
thatbudakguy marked this pull request as draft September 17, 2026 23:46
@thatbudakguy
thatbudakguy removed this pull request from stack #218 September 18, 2026 20:18
Moves the requests a harvest makes into an HttpClient that waits out the
crawl delay before each one. Previously the delay was applied once per
page of search results, so a Blacklight 7 harvest could issue 100
document requests back to back before it took effect.

Each request continues to get its own connection rather than reusing one,
which is likewise more likely to be accepted by a WAF; the client is now
the single place that behavior is decided. Fractional delays work too.

A response class builds a client for itself when it isn't given one, and
passes along the logger it was given so request logging doesn't quietly
go somewhere else.

Harvester request specs now stub HTTP with webmock rather than mocking
Net::HTTP.get, since the request path is no longer a single class method.

Closes #209

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thatbudakguy
thatbudakguy force-pushed the 209-per-request-crawl-delay branch from b9ac8e3 to fefb98e Compare September 18, 2026 20:19
@thatbudakguy
thatbudakguy marked this pull request as ready for review September 18, 2026 20:31
@thatbudakguy
thatbudakguy added this pull request to stack #219 September 18, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant