Skip to content

Retry failed harvest requests, and fail loudly when they don't recover - #214

Draft
thatbudakguy wants to merge 1 commit into
210-configured-headersfrom
208-fail-loudly
Draft

thatbudakguy wants to merge 1 commit into
210-configured-headersfrom
208-fail-loudly

Conversation

@thatbudakguy

@thatbudakguy thatbudakguy commented Sep 17, 2026

Copy link
Copy Markdown
Member

Closes #208

Both response classes rescued every error while paging, nil'd the response and left their loop, so #index returned normally after a single transient failure — the 6-of-15-documents-and-exit-0 case in the issue. Individual document failures were dropped the same way, by compacting them out of the page.

Requests now go through retries with an exponential backoff for failures that tend to be transient — connection resets, broken pipes, timeouts, TLS errors, 5xx, rate limiting — and a request that still can't be completed raises GeoCombine::Exceptions::HarvestError. A 404 for a single document is logged and skipped instead, since a record can be indexed but unreadable.

Responses are checked rather than trusted, since a status code alone can't tell a page of records from a rejection:

  • a 200 carrying HTML, or any body that isn't a JSON object, is treated as a failure worth retrying rather than parsed as a document
  • every page is validated as a page of search results, not just the first one through the response factory. A 200 carrying parseable JSON that isn't one (a WAF or API gateway rejection, say) used to read as "no more results" and end the harvest quietly; so did a body of the literal null, via JSON.parse returning nil
  • a result with no link to itself, and a page whose every document was skipped, are logged rather than quietly dropped

Decisions to check

  • LegacyBlacklightResponse is fixed too, though the issue title names only ModernBlacklightResponse — it had the identical swallow, and the central fix covers both. Leaving one behind would have meant Legacy still truncating harvests silently.
  • 429 is retried, and retries are configurable (max_retries, retry_delay, globally or per site). Both go slightly beyond the issue's literal list; rate limiting is the most likely response to an aggressive harvester and the thing Change crawl_delay to apply per-request #209's pacing exists to avoid, so it seemed in the spirit of "failures are actually relatively common in production". Easy to drop if you'd rather keep to the list.
  • A wholly-skipped page logs at error but does not raise. The issue explicitly carves 404s out as non-fatal, so raising would contradict it — but a /raw route that moves shouldn't turn into a zero-document success that looks like an empty site either. Error-level logging is the compromise; happy to make it fatal instead.
  • Redirects are not followed (they never were). A 3xx now raises with the status rather than failing to parse an empty body.

Note that Net::HTTP already retries once, without delay, for most of the error classes the issue lists; these retries stack on top of that.

@thatbudakguy
thatbudakguy added this pull request to stack #218 September 17, 2026 23:43
Both response classes rescued every error while paging, nil'd the
response, and left their loop, so #index returned normally after a single
transient failure: a harvest that indexed 6 of 15 documents exited 0 and
looked like a success. Individual document failures were dropped the same
way, by compacting them out of the page.

Requests now go through retries with an exponential backoff for failures
that tend to be transient -- connection resets, broken pipes, timeouts,
TLS errors, 5xx, rate limiting -- and a request that still can't be
completed raises GeoCombine::Exceptions::HarvestError. A 404 for a single
document is logged and skipped instead, since a record can be indexed but
unreadable.

Responses are checked rather than trusted, since a status code alone
can't tell a page of records from a rejection:

  - a 200 carrying HTML, or any body that isn't a JSON object, is treated
    as a failure worth retrying rather than parsed as a document
  - every page is validated as a page of search results, not just the
    first one through the response factory; a 200 carrying JSON that
    isn't one used to read as "no more results" and end the harvest
  - a result with no link to itself, and a page whose every document was
    skipped, are logged rather than quietly dropped

Closes #208

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thatbudakguy
thatbudakguy marked this pull request as draft September 17, 2026 23:47
@thatbudakguy
thatbudakguy removed this pull request from stack #218 September 18, 2026 20:18
@thatbudakguy
thatbudakguy added this pull request to stack #219 September 18, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant