Optional source ranges and provenance in the Pandoc AST

[dereuromark]

Pandoc's AST is excellent for semantic conversion, but tools built around it cannot generally relate a node back to its original source range. This affects diagnostics, language servers and source-to-source tools.

For example, a filter can report that an image is missing alt text, but after reading Markdown through Pandoc it cannot reliably point to the original line and columns.

A related case, from the bridge side: our own AST does retain deliberate source spellings - which bullet character was typed, whether a dash came from --, whether a character was escaped, the order the parts of an attribute were written in. What we cannot do is carry any of it through Pandoc, because the nodes that hold those spellings (BulletList, Str, HorizontalRule, Para) are exactly the ones with no Attr.

Would there be interest in an optional, standardized provenance mechanism carrying at least:

source file
start line/column
end line/column

I would expect it to be opt-in and safe for readers, filters and writers to ignore. It should not require every filter to maintain exact positions after editing; stale or absent provenance is preferable to pretending it is always valid.

Possible designs include node attributes where available, a document-level side table with stable node IDs, or an annotated AST/API outside the normal JSON representation.

We have since implemented the first of those privately, and the experience is worth offering: local attribute wrappers travel with their node when a filter reorders content, which a positional side table does not. We started out preferring the side table on the grounds that it leaves the tree untouched, and changed our minds once filters were in the picture. The remaining weakness is that a wrapper only works where a node can hold an Attr, which is the gap described above.

I would still like to understand what would fit Pandoc without burdening ordinary conversion.

[jgm]

See https://pandoc.org/MANUAL.html#extension-sourcepos
Note that this also works for djot (need to document that):

% pandoc -f djot+sourcepos -t native
- hi
[ Div
    ( ""
    , []
    , [ ( "wrapper" , "1" ) , ( "data-pos" , "1:1-2:0" ) ]
    )
    [ BulletList
        [ [ Div
              ( ""
              , []
              , [ ( "wrapper" , "1" ) , ( "data-pos" , "1:3-2:0" ) ]
              )
              [ Para
                  [ Span
                      ( ""
                      , []
                      , [ ( "wrapper" , "1" )
                        , ( "data-pos" , "1:3-1:4" )
                        ]
                      )
                      [ Str "hi" ]
                  ]
              ]
          ]
        ]
    ]
]

It may be possible to support it in some other formats.

[jgm]

For commonmark, this will also give source files, e.g.

data-pos="MANUAL.txt@8148:32-8148:33

but for djot this doesn't work, because the djoths parser just takes a string (the concatenation of input sources).

[dereuromark]

That covers the position half, I think. Three things I measured on 3.10.2 while trying it:

Inline ranges in djot are per text run, not per node, so they cannot locate a spelling (as of now anyway).
a -- b \*x\* c gives every inline the same range:

Str "a"        1:1-1:14
Space          1:1-1:14
Str "\8211"    1:1-1:14
Str "*x*"      1:1-1:14
Str "c"        1:1-1:14

Even a b puts both words and the space at 1:1-1:3. Commonmark splits per node
(1:1-1:6, 1:6-1:7, 1:12-1:17), so slicing the source to see whether the en dash was
typed --, or whether those asterisks were escaped, works there and cannot work in djot.

On the concatenated-string point: for djot it is not only the missing filename. Reading
f1.djot f2.djot, the first line of the second file reports as 3:1-4:0, a position in
a document that does not exist on disk, with nothing marking the switch. Commonmark gives
f2.md@1:1-3:1. So a consumer that trusts djot positions points at the wrong line of the
wrong file rather than failing.

And the wrappers are not free for writers, which matters if provenance is meant to ride
along an ordinary conversion rather than a dedicated inspection pass. On - hi:

$ pandoc -f djot -t html
<ul>
<li>hi</li>
</ul>

$ pandoc -f djot+sourcepos -t html
<ul data-pos="1:1-2:0">
<li><p data-pos="1:3-2:0"><span data-wrapper="1" data-pos="1:3-1:4">hi</span></p></li>
</ul>

-t djot writes the positions back out as {data-pos="1\:1-2\:0"} attribute lines, and
JSON for a two-item list goes from 169 to 514 bytes.

On what shape avoids the first point, since it is the one that blocks the spelling use
case: I ended up not needing a range for most of it in Carve, because the reader keeps the
distinction as a node type. a -- b \*x\* c parses to text, smart_punctuation,
text, escaped_text, text, escaped_text, text, so "was that en dash typed --"
and "were those asterisks escaped" are answered from the tree rather than from the
source. Where a range is still needed it is one per node, as UTF-8 byte offsets into the
source rather than line and column:

[0,2)   "a "
[2,4)   "--"
[4,7)   " b "
[7,9)   "\*"
[9,10)  "x"
[10,12) "\*"
[12,14) " c"

Bytes sidestep the unit question entirely - I had columns counted in UTF-16 in one
engine and codepoints in another before pinning it - and the sidecar carries the source
string it indexes, so there is no filename to lose when inputs are concatenated.

[jgm]

Inline ranges in djot are per text run, not per node

Yes, because djoths doesn't have separate Str and Space elements; it just parses the whole run.
We have to split this up to fit the pandoc types.

On the concatenated-string point: for djot it is not only the missing filename. Reading
f1.djot f2.djot, the first line of the second file reports as 3:1-4:0, a position in
a document that does not exist on disk

Yes, as I said, these are just positions in the total string passed to djoths, which is concatenated from the input files.

[massifrg]

I faced a similar problem: from a point (a text line) in a PDF, (inside an editor) get back to the position in the source file that ends up to that point of the PDF.
You are likely facing a different problem, so I'm a bit off topic, but what follows is still under the "how to make positional information pass through (mostly) isomorphic transformations of a document?" question.

My source files are Pandoc JSON documents, while PDF is generated by ConTeXt.
Two problems here:

1. how to get from the click point on the PDF page to the information about the corresponding source file and line
2. what's a line in a Pandoc JSON document?

The answers I found:

1. ConTeXt supports SyncTeX, a tool to match rectangular areas of a PDF page with a source file name and a position in the source, usually plain text (LaTeX/ConTeXt .tex files, but I know ConTeXt is used also with the "SyncTeX-XML sources" combination); once you click in a PDF viewer that supports SyncTeX and is properly configured, your favorite editor pops up with the right source file and the cursor at the line matching the point you clicked
2. since a JSON file is a text file, you first think of the source line in the JSON file; but it may be a one-liner, and even in a prettified form, the source line is not that significant, to retrieve the point to move the cursor to in your editor; the solution was to answer the question: "What's a line in a Pandoc AST?" (so not just in a JSON, but in the AST serialized with that JSON)

Getting from the paragraph in the PDF to the paragraph in the source file was good enough, so the first answer was: "a line in a Pandoc AST is a paragraph"; then I extended this concept to all the block-like containers of Inlines:

• Para (most common)
• Header
• Plain
• terms in DefinitionList
• lines in a LineBlock
• short Caption

How to get to the nth "line" (paragraph-like item) in the editor? Just count those elements in a "topdown filter" way and stop at the nth one (I'm omitting how to inject that number into the PDF+SyncTeX in the first place).

For better granularity, that concept could be extended to the nth character-like item inside the "line".