Add a Europe PMC Bookshelf rung for NCBI book parts - #4
Merged
Merged
Conversation
NCBI Bookshelf book parts -- GeneReviews chapters and some 40k other
parts -- have no PMCID, so nothing on the ladder could reach them.
Europe PMC serves their full text under the sibling of the article
endpoint, `/{bookid}/bookXML`, as BITS (the JATS Book Interchange Tag
Set, root `<book-part-wrapper>`). The accession is a Bookshelf id
(`NBK` + digits) that PubMed's own record carries.
ArticleIds gains `bookid`, carried through `merge`. `ids.RESOLVABLE`
names the three identifiers a resolver can supply; `fetch_body`'s
demand-driven resolution, `chain`'s stop condition and `chain_batch`'s
`required` (its default and its validation) all read it, so a rung that
needs only a `bookid` never costs a resolver call and `bookid` stays
caller-supplied.
EuropePmcBookshelfFetcher (`requires {'bookid'}`) heads the default
ladder: a present bookid is decisive (a book part has no PMCID) and
costs one GET, so a `pmid+bookid` bundle is served before any PMC rung
or resolver call is spent chasing a PMCID that does not exist. It
shares EuropePmcFetcher's GET-and-decline handling, which now also
recognises how the book endpoint reports a miss: HTTP 200 carrying a
`<fullTextXMLBean>` message envelope (the article endpoint 404s). The
root tag is sniffed from the first start event, not a full parse. The
bookid is gated to `NBK<digits>` (ASCII letters and digits,
case-insensitive, upper-cased) before it is interpolated; any other
shape raises ValueError, matching how `_doi` treats a malformed DOI,
rather than silently declining as if the source had no copy.
Source metadata: `_from_jats` walked every `<license>` in document
order, which for a book part means the book's `<book-meta>` terms
always won because they precede the part. A `<book-part-wrapper>` is
now read from `book-part/book-part-meta/permissions` first, then
`book-meta/permissions`, and nowhere else (a figure's permissions are
not the part's); any other root keeps the whole-document walk. The
NBK1247 payload states its terms only in `<book-meta>` and extracts as
before.
CONTEXT.md settles the vocabulary: *Article* covers a Bookshelf book
part, BITS's term for a chapter or section.
Version 0.4.0 (pyproject and uv.lock's own entry); `__version__` and
the HTTP User-Agent read the installed distribution's metadata rather
than hand-kept strings.
lgruen-cpg
marked this pull request as ready for review
September 2, 2026 05:51
folded
reviewed
Sep 2, 2026
folded
reviewed
Sep 2, 2026
folded
previously approved these changes
Sep 2, 2026
`defusedxml.ElementTree` re-exports the stdlib `ParseError`, so the root sniff in `fetchers.py` needs no import of `xml.etree.ElementTree` of its own.
Contributor
Author
The rule as enforced is ruff's S314: parsing goes through defusedxml only, and it does — |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a fetcher for NCBI Bookshelf book parts — GeneReviews chapters and the rest of the Bookshelf corpus — and bumps the version to
0.4.0.A book part has no PMCID, the identifier the PMC rungs key on. Its accession is a Bookshelf id (
NBK+ digits), which PubMed's own record carries, and Europe PMC serves the full text as BITS XML (JATS's book tag set, root<book-part-wrapper>) under/{bookid}/bookXML, the sibling of the article endpoint.ArticleIdsgainsbookid; the consumer supplies it from the PubMed record, since no resolver can.The rung heads the default ladder. A present bookid is decisive — the part has no PMCID for the PMC rungs to find — and costs one GET, so a
pmid+bookidbundle is served before any PMC rung or resolver call is spent chasing a PMCID that does not exist.ids.RESOLVABLEnames the three identifiers a resolver can supply (pmid,pmcid,doi);fetch_body's demand-driven resolution,chain's stop condition andchain_batch'srequiredall read it, so a rung whose unmet requirement isbookidnever triggers the resolver.A
bookXMLmiss is HTTP 200. The book endpoint answers an unknown id with a<fullTextXMLBean>message envelope rather than the 404 the article endpoint returns. Both Europe PMC fetchers share one GET-and-decline helper; it sniffs the root tag from the first start event, not a full parse, and declines on that envelope.BITS licence precedence. A book part can carry two
<permissions>blocks: the book's in<book-meta>and the part's in<book-part-meta>, with the book's first in document order._from_jatsreads a<book-part-wrapper>frombook-part/book-part-meta/permissionsfirst, thenbook-meta/permissions, and nowhere else — a figure's permissions are not the part's. Any other root keeps the whole-document walk. The real GeneReviews payload (NBK1247) states its licence only at book level, as anopen-access-typed link to its usage terms, and extracts under this rule.__version__and the HTTPUser-Agentare read from the installed distribution's metadata, so the version is stated once, inpyproject.toml.Where to look.
litfetch/fetchers.py(the rung, the shared GET-and-decline helper, the accession gate),litfetch/sessions.py(theRESOLVABLEgate on resolution),litfetch/source_metadata.py(the BITS licence scopes), and the tests alongside each.Release. After merge, publish GitHub Release
v0.4.0(matchingpyproject.toml); the release workflow publishes it to PyPI.