Render every block child of a list item, cell, footnote, box and quote (#11)

* Render every block child of a list item, cell, footnote, box and quote

render_list built each <list-item> from item.findall('p') alone and
joined the paragraphs into one line, so a <list> or <def-list> nested
as a sibling of the paragraph, and the item's own <label>, were dropped
without a trace. Across a corpus of 1,013 Bookshelf chapters (BITS XML)
list items hold 2,951 nested lists — about 196,000 words, every
chapter's sub-bullets in its management and diagnosis sections.

The same shape — collect a fixed subset of children, drop the rest —
recurred in render_def_list, render_boxed_text, render_disp_quote,
render_statement, render_glossary, render_abstract's structured
sections, render_body, render_floats_group, table cells and
<table-wrap-foot> (whose <fn-group> footnotes were dropped), and in the
Elsevier _list, _render_deflist, _render_display, _render_quote,
_render_enunciation, table entries and footnotes. So the fix is one
mechanism rather than a patch per site: a document-order content walker
(jats._render_content, elsevier._Renderer._content) renders each child
through its block renderer — a <p> as its own fragments, a run of text
and inline children between blocks as a paragraph — and returns typed
(tag, markdown) fragments. A tag with no block renderer is inline
content, so nothing falls through; the only exclusions are named
metadata children (sec-meta, permissions, object-id) and the children a
caller renders itself (a label, a title). JATS block dispatch is one
table, _BLOCK_RENDERERS, keyed by tag.

The fragments are laid out by three shared helpers in common: list_item
stacks an item's blocks under its marker, indenting by the marker's
width (CommonMark's content offset) so a nested list nests and a later
paragraph continues the item; definition_item does the same for a
`- **term** — definition` entry; md_cell joins a cell's blocks with
<br>, a GFM row being single-line. The item marker is the <label> when
present (verbatim, as the Elsevier renderer already did), else the
ordinal or a bullet. A flat list of single-paragraph items renders
byte-identically to before.

Also rendered where previously dropped: a section's <label> (joins the
heading), a list's <title>, <term-head>/<def-head>, a term's inline
markup (findtext read only the text before the first child), a
labelled and captioned <media> or <graphic> in a paragraph, an
<inline-graphic> (as an image), bare <mml:math>/<tex-math> in running
text (as inline LaTeX instead of their source), a titled <app-group>,
<bio>, and an Elsevier quote's <source>. Cells iterate <th>/<td> in
document order rather than all <td> first. Inline runs trim ASCII
whitespace only (common.flat), as every other edge does, so an em-space
indented pseudocode line keeps its indentation.

Tests: one hand-written document per container shape, checked both for
the invariant that every text node of the input appears in the output
and for the exact markdown layout of the nesting.

Bump version to 0.7.0: rendered output changes for most documents with
nested lists, list-bearing cells, numbered sections or multi-paragraph
footnotes, so consumers that cache renderings need to re-render.

* Emit only CommonMark list markers; keep heads, titles and floats in place

Review fixes for the block-content walker.

A verbatim <label> is not a CommonMark list marker: "• Outer\n  ◦ Inner"
parses as one paragraph, and a label three or more characters wide
("(i)", "Step 1:") indents its continuation paragraphs into an indented
code block. common.list_marker now resolves a label: an ordinal
("3.", "10)") is the marker verbatim, a bare number gets its period, a
bullet glyph becomes "-", and anything else is not a marker at all — the
list's default marker is emitted and the label opens the item's content
("- (i) First."). The indent follows the marker actually emitted. The
Elsevier renderer applies the same rule, keeping the inline rendering of
a non-marker label ("- **(a)** …"). Verified with pandoc: every shape
parses as a (nested) list.

A definition entry, a <statement> and an Elsevier <enunciation> join
their head onto the first block only when that block is a paragraph; a
fence, table or formula now starts beneath the head line.

A run whose ASCII-trimmed text is only non-ASCII whitespace
(<p>&#160;</p>) renders nothing, as before; the trim stays ASCII-only,
which table cells and labels already used (headings, attribs and
statement heads trim with str.strip()).

An Elsevier <textbox> without <textbox-body> no longer repeats its
label and head from the body walk. A paragraph nested in another run —
a list item's, a footnote's — restores the run's queued floats instead
of resetting them, so a float anchored there is placed at the
paragraph, not in the end sweep.

A nested, titled <def-list> is preceded by a blank line so its title
does not lazily continue the entry above. A table foot part that is one
italic span is unwrapped before the foot is italicised, so an
<attrib><italic> no longer renders bold. A bare <alternatives> in
running text yields one alternative in _formula_body's preference
order instead of every alternative in turn. A structured abstract's
<sec>s, and a section's <ref-list>, are dispatched inside the walk (an
overrides table per caller), keeping document order for a paragraph
that follows them.

Tests: the text-node invariant now uses multi-character tokens
throughout (a one-character node matches almost any output) and asserts
that; layout-only shapes — a bullet-glyph label, an <alternatives> —
are checked for their markdown alone. New cases: bare-number, ordinal,
glyph, roman and word labels, a definition opening with code, a
statement opening with a table, a titled nested def-list, an italic
attrib foot, whitespace-only paragraphs, a bold Elsevier label and a
body-less textbox.

* Separate a nested list that cannot interrupt a paragraph; one italic foot

Second-pass review fixes for the block-content walker.

Only a bullet or a "1." item can interrupt a paragraph in CommonMark;
"- Steps:\n  3. Third." lazily continues the paragraph, and an empty
first nested item ("-" alone) underlines it as a setext heading.
common.list_item now puts a blank line before a nested list whose first
line is not "[-*+] x" or "1[.)] x" — the same path lays out definition
entries and Elsevier items.

A label that is not a marker is the item's enumerator, so it opens the
content behind a bullet whatever the list type ("- (a) Alpha.", not
"1. (a) Alpha."); the marker is decided on the stripped label (so a
glyph followed by a no-break space counts), the glyph set gains the
dashes and the remaining common bullet shapes, and a lead that would
open a heading, quote or fence is backslash-escaped. An Elsevier label
with no text but an inline rendering (a <math> bullet) is kept as the
lead rather than dropped. list_marker's docstring says that a verbatim
ordinal keeps the source numbering in the markdown while a renderer
renumbers from the first item.

A table foot is italicised as a whole only when no asterisk inside it
— an italic span or a literal significance marker — could pair with the
wrapper, so a partly italic <attrib> no longer renders
"**Adapted* from Doe.*" (unwrapping the inner emphasis was tried and
rejected: it also ate literal asterisks, "DQB1*0302"). A bare
<alternatives> prefers its <tex-math>,
then its <math> whether namespaced or not, before a graphic, so an
un-namespaced <math> beside a <graphic> renders as math. Reference
entries are blank-line separated with none trailing, so a <ref-list>
inside a section no longer leaves a triple newline before the paragraph
that follows it (a document ending in references loses its trailing
newline).

Tests: the italic-foot assertion is anchored on the preceding blank
line and asserts no bold; new shapes cover a nested list opening with
"3.", "10)", a bare "3", "0" and an empty item, an order-typed list with
"(a)" labels, "#"/">" labels, an <alternatives> with un-namespaced math,
a glyph-with-NBSP label, an Elsevier math label and a mid-section
<ref-list>. Verified with pandoc that each shape parses as intended.

* Render a table foot as plain text

A table foot was set in italics as a whole. The wrapper collided with
emphasis inside the foot — an italic gene name at its edge doubled up
to bold — and with the literal asterisks footnotes use as significance
markers; making the wrapper conditional on the foot's content left a
foot's rendering depending on whether it mentions an italic gene. So
neither dialect wraps the foot: it is plain text below the table, its
footnote markers and its own markup as they are.

---------

Co-authored-by: Leonhard Gruenschloss <l.gruenschloss@vcgs.org.au>

Author: lgruen-cpg

README.md

@@ -96,7 +96,7 @@ the regression suite re-runs the converter over them on every test run.
 pytest                                # full suite
 ```
 
-Six test files:
+Seven test files:
 
 - `tests/test_mml_unit.py` — exhaustive per-element MathML cases.
 - `tests/test_jats_articles.py` — structural assertions over real PMC
@@ -119,6 +119,10 @@ Six test files:
 - `tests/test_whitespace_unit.py` — single-line Markdown constructs (headings,
   table rows, list items) survive pretty-printed source, one hand-written
   document per construct.
+- `tests/test_block_content_unit.py` — block content inside a container (a
+  nested list, a second paragraph or a definition list under a list item;
+  lists in table cells, footnotes, boxed text and quotes) survives in both
+  dialects and nests as Markdown; one hand-written document per shape.
 
 ### Fetching test fixtures
 

litdown/common.py

@@ -4,8 +4,9 @@
 bits both :mod:`litdown.jats` and :mod:`litdown.elsevier` would otherwise
 duplicate verbatim: namespace-stripping tag helpers, the xlink href
 accessor, source-whitespace normalization, table-cell escaping, the inline
-typographic leaf formatters, and the markdown-table grid builder
-(colspan/rowspan expansion + multi-row header collapse).
+typographic leaf formatters, the markdown-table grid builder
+(colspan/rowspan expansion + multi-row header collapse), and the layout of
+block fragments into a list item, a definition entry or a table cell.
 
 The inline *dispatchers* are deliberately NOT shared — JATS and Elsevier
 diverge on cross-ref/link attribute handling enough that one config-driven
@@ -16,7 +17,9 @@
 from __future__ import annotations
 
 import re
+import textwrap
 import xml.etree.ElementTree as ET
+from collections.abc import Iterable, Sequence
 
 XLINK_NS = 'http://www.w3.org/1999/xlink'
 MML_NS = 'http://www.w3.org/1998/Math/MathML'
@@ -73,6 +76,125 @@ def md_escape_cell(text: str) -> str:
     return flat(text).replace('|', '\\|')
 
 
+def text_carrier(text: str | None) -> ET.Element:
+    """An element whose only content is ``text``.
+
+    Lets a bare string — an element's ``.text``, or the ``.tail`` of a child
+    that is rendered separately — take its place in a list of children that
+    is rendered as one inline run. Both dialects' inline dispatchers render
+    an unknown tag as its inner text, which is all a carrier has.
+    """
+    carrier = ET.Element('#text')
+    carrier.text = text
+    return carrier
+
+
+# ---------------------------------------------------------------------------
+# Block-fragment layout
+# ---------------------------------------------------------------------------
+
+# A block fragment is ``(tag, markdown)``: the source tag it was rendered from
+# (``p``/``para`` for a run of inline content), and its markdown. The tag
+# decides how the fragment joins its neighbours inside a list item.
+BlockFragment = tuple[str, str]
+
+_PARAGRAPH_TAGS = frozenset({'p', 'para'})
+_NESTED_LIST_TAGS = frozenset({'list', 'def-list'})
+
+# The list markers CommonMark recognises: an ordinal of up to nine digits
+# closed by ``.`` or ``)``, or a bullet. Bullet glyphs publishers put in a
+# <label> stand for the bullet marker.
+_ORDINAL_MARKER_RE = re.compile(r'\d{1,9}[.)]')
+_BARE_ORDINAL_RE = re.compile(r'\d{1,9}')
+_BULLET_GLYPHS = frozenset({'•', '◦', '▪', '●', '○', '·', '‣', '⁃', '■', '◆', '▸', '►', '-', '–', '—', '*', '+'})
+# Only a bullet or a ``1.``/``1)`` item can interrupt a paragraph; any other
+# list line after a paragraph is a lazy continuation of it.
+_INTERRUPTING_LIST_RE = re.compile(r'(?:[-*+]|1[.)]) \S')
+# At the start of an item's content these would open a heading, a quote or a fence.
+_BLOCK_OPENERS = ('#', '>', '`', '~')
+
+
+def list_marker(label: str, default: str) -> tuple[str, str]:
+    """Resolve a list item's label to ``(marker, lead)``: a CommonMark marker, and content that opens the item.
+
+    An ordinal label (``3.``, ``10)``) is the marker verbatim — the markdown
+    keeps the source numbering, though a renderer renumbers from the first
+    item — and a bare number gets its period; a bullet glyph is the bullet
+    marker. Anything else (``(i)``, ``a.``, ``Step 1:``) is not a marker
+    CommonMark would parse — the line would be a paragraph and nothing could
+    nest under it — so the label opens the item's content behind a bullet
+    (``- (i) First.``): the label is the enumerator, so the list type's
+    ordinal is not emitted beside it. An empty label yields ``default``.
+    """
+    label = label.strip()
+    if not label:
+        return default, ''
+    if _ORDINAL_MARKER_RE.fullmatch(label):
+        return label, ''
+    if _BARE_ORDINAL_RE.fullmatch(label):
+        return f'{label}.', ''
+    if label in _BULLET_GLYPHS:
+        return '-', ''
+    return '-', label
+
+
+def list_item(marker: str, blocks: Sequence[BlockFragment], lead: str = '') -> str:
+    """Lay out one markdown list item from its block fragments.
+
+    The first fragment follows ``marker``; every later line is indented by
+    the marker's width — CommonMark's content offset — so it stays inside
+    the item. A nested list (a fragment tagged ``list`` or ``def-list``)
+    follows the line above directly; any other block is a paragraph of its
+    own, separated by a blank line — as is a nested list whose first line
+    could not interrupt a paragraph (an ordinal other than ``1``, an empty
+    item). ``lead`` (a label that is not a marker, see :func:`list_marker`)
+    opens the first paragraph, or is a paragraph of its own when the item
+    opens with another block; escaped when it would otherwise open a block
+    of its own. An item without content is the bare marker.
+    """
+    if lead:
+        if lead.startswith(_BLOCK_OPENERS):
+            lead = f'\\{lead}'
+        if blocks and blocks[0][0] in _PARAGRAPH_TAGS:
+            blocks = [(blocks[0][0], f'{lead} {blocks[0][1]}'), *blocks[1:]]
+        else:
+            blocks = [('p', lead), *blocks]
+    indent = ' ' * (len(marker) + 1)
+    lines: list[str] = []
+    for i, (tag, md) in enumerate(blocks):
+        if i and (tag not in _NESTED_LIST_TAGS or not _INTERRUPTING_LIST_RE.match(md)):
+            lines.append('')
+        lines.append(textwrap.indent(md, indent))
+    body = '\n'.join(lines)
+    return f'{marker} {body[len(indent) :]}' if body else marker
+
+
+def definition_item(term: str, blocks: Sequence[BlockFragment]) -> str:
+    """One definition-list entry as a bullet: ``- **term** — definition``.
+
+    The definition's first block joins the term line when it is a paragraph;
+    the rest stack under it as in :func:`list_item`. A definition that opens
+    with any other block — a nested list, a code fence, a table — keeps the
+    term line to itself, the blocks beneath it.
+    """
+    head = f'**{term}**' if term else ''
+    if blocks and blocks[0][0] in _PARAGRAPH_TAGS:
+        first = blocks[0][1]
+        head = f'{head} — {first}' if head else first
+        blocks = blocks[1:]
+    return list_item('-', [('p', head), *blocks] if head else list(blocks))
+
+
+def md_cell(blocks: Iterable[str]) -> str:
+    """One GFM cell from block fragments.
+
+    A row cannot span lines, so ``<br>`` separates the fragments and the
+    lines within each; the result is escaped and flattened by
+    :func:`md_escape_cell`.
+    """
+    return md_escape_cell('<br>'.join('<br>'.join(md.splitlines()) for md in blocks))
+
+
 # ---------------------------------------------------------------------------
 # Inline leaf formatters
 # ---------------------------------------------------------------------------