Render every block child of a list item, cell, footnote, box and quote

[lgruen-cpg]

What

render_list built each <list-item> from item.findall('p') alone and joined the paragraphs into one line. A <list> or <def-list> nested as a sibling of the paragraph, and the item's own <label>, were dropped without a trace — every sub-bullet in a Bookshelf chapter's management and diagnosis sections, for instance. The same collect-a-fixed-subset shape recurred in a dozen other containers in both dialects (definition lists, boxed text, quotes, statements, glossaries, structured abstracts, the body itself, table cells, <table-wrap-foot> — whose <fn-group> footnotes were dropped — and the Elsevier list, def-list, display, quote, enunciation, table entry and footnote renderers).

How

One mechanism rather than a patch per site: a document-order content walker (jats._render_content, elsevier._Renderer._content) renders each child through its block renderer and each run of text and inline children between blocks as a paragraph, returning typed (tag, markdown) fragments. A tag without a block renderer is inline content, so nothing falls through silently; the only exclusions are named metadata children (sec-meta, permissions, object-id) and the children a caller renders itself (a label, a title). JATS block dispatch is one table, _BLOCK_RENDERERS; a caller that needs a tag rendered its own way (a structured abstract's <sec> as a bold title, a section's <ref-list> headed against the section title) passes an override, so document order is kept.

Three shared layout helpers in common turn the fragments into markdown: list_item stacks an item's blocks under its marker, indenting by the marker's width (CommonMark's content offset) so a nested list nests and a later paragraph continues the item; definition_item does the same for a - **term** — definition entry (the definition joins the term line only when it opens with a paragraph); md_cell joins a cell's blocks with <br>. A <label> becomes the marker only when CommonMark would parse it as one (common.list_marker: an ordinal such as 3. or 10) verbatim, a bare number with its period, a bullet glyph as -); any other label ((i), Step 1:, **(a)**, a math bullet) is the item's enumerator and opens the content behind a bullet, whatever the list type, so the line is a list item and its sub-list nests; a lead that would open a heading, quote or fence is escaped. A nested list is separated from the paragraph above by a blank line unless its first line can interrupt a paragraph (a bullet or 1.), since - Steps:\n 3. Third. would otherwise be a lazy continuation. Verified with pandoc. A flat list of single-paragraph items renders byte-identically to before.

Where to look

• litdown/jats.py: _render_content, _BLOCK_RENDERERS (end of module), render_list, render_def_list, render_table / _render_table_foot.
• litdown/common.py: list_marker, list_item, definition_item, md_cell, text_carrier.
• litdown/elsevier.py: _Renderer._content, _para_typed, _list.
• tests/test_block_content_unit.py: one hand-written document per container shape; every (multi-character) text node of the input must appear in the output, plus the exact nesting layout.

Also rendered where previously dropped

A section's <label> (joins the heading), a list's <title>, <term-head>/<def-head>, a term's inline markup (findtext read only the text before the first child), a labelled and captioned <media>/<graphic> in a paragraph, an <inline-graphic> (as an image), bare <mml:math>/<tex-math> in running text (as inline LaTeX instead of their source; a bare <alternatives> yields one alternative), a titled <app-group>, <bio>, and an Elsevier quote's <source>. Cells iterate <th>/<td> in document order rather than all <td> first. A table foot is italicised as a whole only when no asterisk inside it (an italic span, a literal significance marker) could pair with the wrapper — so a foot mentioning an italic gene name is now plain; dropping the wrapper for every foot would be the consistent alternative, at the cost of churning every table foot. Reference entries are blank-line separated with none trailing, so a mid-section <ref-list> leaves no triple newline (a document ending in references loses its trailing newline). Inline runs trim ASCII whitespace only (common.flat, as table cells and labels already did; headings and attribs still use str.strip()), so em-space indented pseudocode keeps its indentation; a run that is only non-ASCII whitespace still renders nothing.

Measurement

Over a corpus of 1,013 Bookshelf chapters (BITS XML; 961 render, 52 are <book-app>/<toc> units the wrapper refuses by design): 960 of 961 renderings change; content words (tokens with a letter or digit, excluding list markers and <br>) go from 8,096,392 to 8,324,705 (+228,313, +2.8%); no chapter loses words. Body/back text nodes absent from the output drop from 18,568 in 891 chapters to 6,713 in 77 — what remains is a <back><index> in one chapter (not rendered, before or after) and reference-field text (<supplement>, <comment>, <elocation-id>) the citation renderer does not emit, both outside this change.

Of the 39 PMC article fixtures, all 39 renderings change (every article ends in a reference list, which loses its trailing newline); 16 change beyond whitespace: list-bearing cells gain <br>-separated items, multi-paragraph footnotes keep their paragraphs, fn-group table footnotes and eLife's competing-interest/author-contribution groups appear, inline graphics render as images, a glossary's heading gains a blank line, foots with emphasis inside lose the italic wrapper; body text nodes missing from the output go from 1,111 to 1,091. Of the 11 Elsevier fixtures, 5 change (bullet-glyph and (1) labels now list items, a bulleted cell, a quote's <source>, a table placed at its anchor inside a list item instead of the end sweep); no body text node is missing before or after.

Release

Version bumped to 0.7.0 (rendered output changes for most documents; consumers caching renderings re-render). Publishing follows docs/releasing.md: after merge, publish a GitHub Release tagged v0.7.0; the release workflow checks the tag against pyproject.toml, builds, and publishes to PyPI via Trusted Publishing.

[folded]

I just realised that now scr needs to be able to post issues for me. :)