Narrow litfetch to the fetch surface: source ladder, resolvers, HTTP session - #1
Merged
Merged
Conversation
…session litfetch resolves a scholarly-article identifier bundle (ArticleIds: pmid / pmcid / doi) to its retrievable files and fetches their bytes. It owns identity, the file-set model, and the act of fetching; rendering and storage stay the consumer's (see CONTEXT.md). Surface: - fetch_body walks a Fetcher ladder for the article body (JATS / publisher XML): PMC OA (S3), Europe PMC, Elsevier, Springer OpenAccess. bioRxiv/medRxiv is opt-in (litfetch[biorxiv]; browser-fingerprint fetch, off the default ladder). - list_files / fetch_file enumerate and materialise the file-set across FileSources: PMC OA renditions + supplementary material, plus PDF renditions from Unpaywall, Semantic Scholar, Crossref TDM, and Springer (Meta API). A discovered PDF is a BODY rendition, never a fetch_body result. - Identifier resolvers (Europe PMC, NCBI ID Converter, Semantic Scholar) enrich the bundle; chain() composes them. - resolve_access reports licence / OA terms (from Unpaywall, or read from the artifact bytes); related_ids links preprint <-> published versions. HTTP session (ADR 0001): Session is the object callers hold -- it owns the pooled client (injectable client_factory), per-host polite pacing, and retry with exponential backoff / 429 handling; the operations are methods on it. session.scope() adds a short-lived response cache so a duplicate upstream call within one unit of work is served once. The HTTP primitives (the Http protocol, Rate, RetryPolicy) live in _http, so sources depend on the narrow Http protocol, not the concrete Session. File downloads follow redirects; DOIs are validated and percent-encoded before use. Entitlement: a source marks a subscription File with credential_key=INSTITUTIONAL, which the consumer routes through an EZproxy-style client_factory (documented pattern; no proxy code in litfetch). Docs: README (usage), CONTEXT.md (domain model), docs/api.md (reference), docs/adr/0001 (the Session decision), docs/source-expansion-plan.md, and source references under docs/sources/. Type-checked (ships py.typed); 127 tests; the static-check gate runs via pre-commit.
folded
commented
Jul 3, 2026
folded
left a comment
folded
left a comment
Collaborator
Author
There was a problem hiding this comment.
Docs-vs-code consistency pass. One finding is a runnable bug (the README resolver signature); the rest are readable-consistency polish, two with apply-able suggestions. Note: the original audit's #2 (requires unset) and #3 (chain stopping condition) were checked against the code and are not issues — requires is set on every fetcher and chain._run does gate on all three ids — so they're dropped.
- README: my_resolver now takes the (ArticleIds, Http) the Resolver protocol requires (was ArticleIds -> ArticleIds, which would TypeError when the ladder passes the session); prose + import updated to match. - api.md: NcbiIdConverterResolver noted as always-keyless (no keyed variant); S2 Rate rows given req/s intervals like the NCBI rows; note that PmcOaFetcher implements both Fetcher and FileSource (no PmcOaFileSource); serde functions noted as litfetch.serde (not top-level re-exported); Contact-defaults names the two email= sites.
Collaborator
Author
|
Addressed all six in fbc2aae — README resolver signature now takes |
folded
commented
Jul 3, 2026
The maintainer's email is no longer baked into the source. Session gains a contact param (default None): it appends (mailto:...) to the User-Agent only when set, and flows to the polite-pool params via a new Http.contact attribute. Sources read http.contact for their email/mailto -- Unpaywall (required email; skipped when absent), Crossref mailto (omitted when absent), NCBI email (omitted). NcbiIdConverterResolver drops its email arg; resolve_access / UnpaywallFileSource keep an email override defaulting to the session contact. A scope inherits the parent's contact. Tests supply an explicit test contact where a request is expected, plus new tests: contact->User-Agent, scope inheritance, and Unpaywall declining without a contact. Docs (README, api.md) updated; the api.md 'Contact' section rewritten.
- artifacts: INSTITUTIONAL -> 'litfetch:institutional' so the entitlement sentinel can't collide with a user credentials key; annotate the public string constants Final. (Media types stay str -- an open domain, arbitrary content-types -- so no enum; the closed sets are already enums.) - Log a warning on every silent 'except ValueError' guarding resp.json() (crossref, unpaywall, semantic_scholar, resolvers._get_json, relations, fetchers x2) -- a malformed 200 no longer returns None without a trace, per the never-swallow-silently rule. Left the deliberate broad excepts (defusedxml parse in source_metadata, curl_cffi in _fetch_impersonated): both already log and their error surfaces are unspecified/open.
…lations - RetryPolicy rejects max_attempts < 1 (0 fell through to the unreachable guard). - Session.scope() entered before its parent now raises a named RuntimeError instead of the generic client-property error. - related_ids dedupes on a case-folded DOI key (DOIs are case-insensitive), so bioRxiv and Crossref naming one DOI in different case yield one entry. - _extract_jats_article anchors on <article followed by whitespace or >, so a <article-set>/<article-meta> wrapper cannot be mistaken for the root. - Free resolve_access passes email to Unpaywall only, not the session contact. - Tests for each of the above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
litfetch resolves a scholarly-article identifier bundle (ArticleIds: pmid /
pmcid / doi) to its retrievable files and fetches their bytes. It owns identity,
the file-set model, and the act of fetching; rendering and storage stay the
consumer's (see CONTEXT.md).
Surface:
PMC OA (S3), Europe PMC, Elsevier, Springer OpenAccess. bioRxiv/medRxiv is
opt-in (litfetch[biorxiv]; browser-fingerprint fetch, off the default ladder).
FileSources: PMC OA renditions + supplementary material, plus PDF renditions
from Unpaywall, Semantic Scholar, Crossref TDM, and Springer (Meta API). A
discovered PDF is a BODY rendition, never a fetch_body result.
the bundle; chain() composes them.
artifact bytes); related_ids links preprint <-> published versions.
HTTP session (ADR 0001): Session is the object callers hold -- it owns the
pooled client (injectable client_factory), per-host polite pacing, and retry
with exponential backoff / 429 handling; the operations are methods on it.
session.scope() adds a short-lived response cache so a duplicate upstream call
within one unit of work is served once. The HTTP primitives (the Http protocol,
Rate, RetryPolicy) live in _http, so sources depend on the narrow Http protocol,
not the concrete Session. File downloads follow redirects; DOIs are validated
and percent-encoded before use.
Entitlement: a source marks a subscription File with
credential_key=INSTITUTIONAL, which the consumer routes through an EZproxy-style
client_factory (documented pattern; no proxy code in litfetch).
Docs: README (usage), CONTEXT.md (domain model), docs/api.md (reference),
docs/adr/0001 (the Session decision), docs/source-expansion-plan.md, and source
references under docs/sources/. Type-checked (ships py.typed); 127 tests; the
static-check gate runs via pre-commit.