Skip to content

Add a Europe PMC Bookshelf rung for NCBI book parts - #4

Merged
lgruen-cpg merged 2 commits into
mainfrom
feat/bookshelf-fetcher
Sep 2, 2026
Merged

lgruen-cpg merged 2 commits into
mainfrom
feat/bookshelf-fetcher

Conversation

@lgruen-cpg

Copy link
Copy Markdown
Contributor

Adds a fetcher for NCBI Bookshelf book parts — GeneReviews chapters and the rest of the Bookshelf corpus — and bumps the version to 0.4.0.

A book part has no PMCID, the identifier the PMC rungs key on. Its accession is a Bookshelf id (NBK + digits), which PubMed's own record carries, and Europe PMC serves the full text as BITS XML (JATS's book tag set, root <book-part-wrapper>) under /{bookid}/bookXML, the sibling of the article endpoint. ArticleIds gains bookid; the consumer supplies it from the PubMed record, since no resolver can.

The rung heads the default ladder. A present bookid is decisive — the part has no PMCID for the PMC rungs to find — and costs one GET, so a pmid+bookid bundle is served before any PMC rung or resolver call is spent chasing a PMCID that does not exist. ids.RESOLVABLE names the three identifiers a resolver can supply (pmid, pmcid, doi); fetch_body's demand-driven resolution, chain's stop condition and chain_batch's required all read it, so a rung whose unmet requirement is bookid never triggers the resolver.

A bookXML miss is HTTP 200. The book endpoint answers an unknown id with a <fullTextXMLBean> message envelope rather than the 404 the article endpoint returns. Both Europe PMC fetchers share one GET-and-decline helper; it sniffs the root tag from the first start event, not a full parse, and declines on that envelope.

BITS licence precedence. A book part can carry two <permissions> blocks: the book's in <book-meta> and the part's in <book-part-meta>, with the book's first in document order. _from_jats reads a <book-part-wrapper> from book-part/book-part-meta/permissions first, then book-meta/permissions, and nowhere else — a figure's permissions are not the part's. Any other root keeps the whole-document walk. The real GeneReviews payload (NBK1247) states its licence only at book level, as an open-access-typed link to its usage terms, and extracts under this rule.

__version__ and the HTTP User-Agent are read from the installed distribution's metadata, so the version is stated once, in pyproject.toml.

Where to look. litfetch/fetchers.py (the rung, the shared GET-and-decline helper, the accession gate), litfetch/sessions.py (the RESOLVABLE gate on resolution), litfetch/source_metadata.py (the BITS licence scopes), and the tests alongside each.

Release. After merge, publish GitHub Release v0.4.0 (matching pyproject.toml); the release workflow publishes it to PyPI.

NCBI Bookshelf book parts -- GeneReviews chapters and some 40k other
parts -- have no PMCID, so nothing on the ladder could reach them.
Europe PMC serves their full text under the sibling of the article
endpoint, `/{bookid}/bookXML`, as BITS (the JATS Book Interchange Tag
Set, root `<book-part-wrapper>`). The accession is a Bookshelf id
(`NBK` + digits) that PubMed's own record carries.

ArticleIds gains `bookid`, carried through `merge`. `ids.RESOLVABLE`
names the three identifiers a resolver can supply; `fetch_body`'s
demand-driven resolution, `chain`'s stop condition and `chain_batch`'s
`required` (its default and its validation) all read it, so a rung that
needs only a `bookid` never costs a resolver call and `bookid` stays
caller-supplied.

EuropePmcBookshelfFetcher (`requires {'bookid'}`) heads the default
ladder: a present bookid is decisive (a book part has no PMCID) and
costs one GET, so a `pmid+bookid` bundle is served before any PMC rung
or resolver call is spent chasing a PMCID that does not exist. It
shares EuropePmcFetcher's GET-and-decline handling, which now also
recognises how the book endpoint reports a miss: HTTP 200 carrying a
`<fullTextXMLBean>` message envelope (the article endpoint 404s). The
root tag is sniffed from the first start event, not a full parse. The
bookid is gated to `NBK<digits>` (ASCII letters and digits,
case-insensitive, upper-cased) before it is interpolated; any other
shape raises ValueError, matching how `_doi` treats a malformed DOI,
rather than silently declining as if the source had no copy.

Source metadata: `_from_jats` walked every `<license>` in document
order, which for a book part means the book's `<book-meta>` terms
always won because they precede the part. A `<book-part-wrapper>` is
now read from `book-part/book-part-meta/permissions` first, then
`book-meta/permissions`, and nowhere else (a figure's permissions are
not the part's); any other root keeps the whole-document walk. The
NBK1247 payload states its terms only in `<book-meta>` and extracts as
before.

CONTEXT.md settles the vocabulary: *Article* covers a Bookshelf book
part, BITS's term for a chapter or section.

Version 0.4.0 (pyproject and uv.lock's own entry); `__version__` and
the HTTP User-Agent read the installed distribution's metadata rather
than hand-kept strings.
@lgruen-cpg
lgruen-cpg marked this pull request as ready for review September 2, 2026 05:51
@lgruen-cpg
lgruen-cpg requested a review from folded as a code owner September 2, 2026 05:51
Comment thread litfetch/ids.py
Comment thread litfetch/source_metadata.py
folded
folded previously approved these changes Sep 2, 2026
`defusedxml.ElementTree` re-exports the stdlib `ParseError`, so the root
sniff in `fetchers.py` needs no import of `xml.etree.ElementTree` of its
own.
@lgruen-cpg

Copy link
Copy Markdown
Contributor Author

import xml.etree.ElementTree as ET — what's the rule about using defusedxml?

The rule as enforced is ruff's S314: parsing goes through defusedxml only, and it does — fromstring in both files, iterparse for the root sniff. The stdlib import in source_metadata.py is for the Element type in annotations, which defusedxml doesn't export (it returns stdlib Elements). The one in fetchers.py was only there for ParseError, which defusedxml re-exports; dropped in 654da15.

@lgruen-cpg
lgruen-cpg merged commit c73999f into main Sep 2, 2026
5 of 6 checks passed
@lgruen-cpg
lgruen-cpg deleted the feat/bookshelf-fetcher branch September 2, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants