Skip to content

Commit cdc637f

Browse files
cdervcderv
authored
Add Pandoc update tooling: --skip-archive and smoke-test override (#15291)
Regenerating the Pandoc reference templates and format-extension.ts only ran as part of the S3 archival step, so anyone without S3 credentials could not prepare a Pandoc bump. CI could also smoke-test a new Pandoc only after archival. Both steps can now run before a release is archived. - Add `quarto-bld update-pandoc <version> --skip-archive`. It regenerates the templates and format-extension.ts from the binary QUARTO_PANDOC points at and leaves configuration untouched. It fails when QUARTO_PANDOC is unset or reports another version, since the tool lookup would otherwise fall back to the bundled Pandoc silently. - Add a `pandoc-override-version` input to test-smokes.yml. A composite action installs that release and sets QUARTO_PANDOC for the run only, on Linux, Windows and macOS. - When QUARTO_TEST_EXPECTED_PANDOC is set, the version-check test requires the reported Pandoc version to match it exactly, so a run with the wrong binary still fails.
1 parent 089ba53 commit cdc637f

14 files changed

Lines changed: 404 additions & 91 deletions

File tree

‎.claude/rules/dev-tools/dependency-updates.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
description: "Bundled dependency version bumps (Deno, Pandoc, Dart Sass, Typst, esbuild, veraPDF)"
3+
paths:
4+
- "configuration"
5+
- "src/command/check/check.ts"
6+
---
7+
8+
# Bundled Dependency Updates
9+
10+
When updating a bundled binary, follow `dev-docs/upgrade-dependencies.md`. For Pandoc, also follow `dev-docs/update-pandoc-checklist.md`.

‎.claude/rules/formats/pandoc-templates.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,19 @@
11
---
22
paths:
33
- "src/resources/formats/pdf/pandoc/**"
4+
- "src/resources/formats/beamer/pandoc/**"
45
- "src/resources/formats/typst/pandoc/**"
6+
- "src/resources/formats/html/pandoc/**"
7+
- "src/resources/formats/revealjs/pandoc/**"
8+
- "src/resources/formats/asciidoc/pandoc/**"
59
- "package/src/common/update-pandoc.ts"
10+
- "dev-docs/update-pandoc-checklist.md"
611
---
712

8-
# Pandoc Templates (LaTeX & Typst)
13+
# Pandoc Templates (LaTeX, Typst, HTML, Reveal.js, AsciiDoc)
914

1015
For how Pandoc's LaTeX templates are copied into Quarto and restructured into a modular form (`latex.template`, `latex.common`, …), see `llm-docs/pandoc-quarto-latex-templates.md`.
1116

1217
For the equivalent Typst template integration (`typst.template`, `template.typst`), see `llm-docs/pandoc-quarto-typst-templates.md`.
18+
19+
When updating Pandoc templates, read `dev-docs/update-pandoc-checklist.md` for the per-format file list, reference-copy maintenance, and filename checks.

‎.github/actions/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Composite Actions
2+
3+
Reusable steps shared across workflows in this repo. See also `.github/workflows/actions/` for another set of composite actions (historical split, no functional difference).
4+
5+
| Action | Purpose |
6+
|--------|---------|
7+
| [`build-dist-tarball`](build-dist-tarball/action.yml) | Builds and uploads a versioned Quarto distribution tarball from the current checkout. |
8+
| [`cache-typst`](cache-typst/action.yml) | Configures caching for Typst packages. |
9+
| [`docker`](docker/action.yml) | Builds and pushes a Quarto Docker container to the GitHub Container Registry. |
10+
| [`merge-extension-tests`](merge-extension-tests/action.yml) | Copies test files from extension subtrees into the main test directories. |

‎.github/workflows/actions/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Composite Actions
2+
3+
Reusable steps shared across workflows in this repo. See also `.github/actions/` for another set of composite actions (historical split, no functional difference).
4+
5+
| Action | Purpose |
6+
|--------|---------|
7+
| [`amplitude`](amplitude/action.yml) | Sends an event to Amplitude. |
8+
| [`archive`](archive/action.yml) | Archives a dependency binary to S3. |
9+
| [`keychain`](keychain/action.yml) | Configures a P12 certificate into the keychain for signing. |
10+
| [`pandoc-override`](pandoc-override/action.yml) | Installs a Pandoc release for smoke tests before S3 archival and selects it through `QUARTO_PANDOC`. |
11+
| [`prevent-rerun`](prevent-rerun/action.yml) | Fails if the workflow is re-run (`github.run_attempt > 1`). |
12+
| [`quarto-dev`](quarto-dev/action.yml) | Configures the image for Quarto development (runs `configure.sh`/`.cmd`). |
13+
| [`sign-files`](sign-files/action.yml) | Installs and configures the environment to sign files. |

‎.github/workflows/actions/pandoc-override/action.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: "Pandoc Override"
2+
description: "Installs a Pandoc release for smoke tests before S3 archival and selects it through QUARTO_PANDOC. If pandoc-version is empty, Quarto uses the configured binary."
3+
4+
inputs:
5+
pandoc-version:
6+
description: "Pandoc release tag to install and use instead of the archived/configured version (e.g. 3.10.1). Leave empty to skip."
7+
required: false
8+
default: ""
9+
10+
runs:
11+
using: "composite"
12+
steps:
13+
- name: Install Pandoc override
14+
if: inputs.pandoc-version != ''
15+
uses: r-lib/actions/setup-pandoc@v2
16+
with:
17+
pandoc-version: ${{ inputs.pandoc-version }}
18+
19+
- name: Point QUARTO_PANDOC at override (Linux, macOS)
20+
if: runner.os != 'Windows' && inputs.pandoc-version != ''
21+
shell: bash
22+
run: echo "QUARTO_PANDOC=$(command -v pandoc)" >> "$GITHUB_ENV"
23+
24+
- name: Point QUARTO_PANDOC at override (Windows)
25+
if: runner.os == 'Windows' && inputs.pandoc-version != ''
26+
shell: pwsh
27+
run: |
28+
"QUARTO_PANDOC=$((Get-Command pandoc).Source)" >> $env:GITHUB_ENV
29+
30+
# Lets check-versions-no-mismatch.test.ts tell an intentional override
31+
# apart from a real bundled-version mismatch. Passed via env so the
32+
# dispatch input is never interpolated into the shell script.
33+
- name: Record expected Pandoc version for the version-mismatch test
34+
if: inputs.pandoc-version != ''
35+
shell: bash
36+
env:
37+
PANDOC_VERSION: ${{ inputs.pandoc-version }}
38+
run: echo "QUARTO_TEST_EXPECTED_PANDOC=$PANDOC_VERSION" >> "$GITHUB_ENV"
39+
40+
- name: Confirm Pandoc override is live (Linux, macOS)
41+
if: runner.os != 'Windows' && inputs.pandoc-version != ''
42+
shell: bash
43+
run: quarto pandoc --version
44+
45+
- name: Confirm Pandoc override is live (Windows)
46+
if: runner.os == 'Windows' && inputs.pandoc-version != ''
47+
shell: pwsh
48+
run: quarto pandoc --version

‎.github/workflows/test-smokes.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ on:
7171
required: false
7272
type: string
7373
default: ""
74+
pandoc-override-version:
75+
description: "Pandoc release tag to download from GitHub for smoke tests before S3 archival (e.g. 3.10.1). Quarto selects it through QUARTO_PANDOC. Leave empty to use the configured binary."
76+
required: false
77+
type: string
78+
default: ""
7479

7580
# this is required to cleanup julia cache
7681
permissions:
@@ -504,6 +509,12 @@ jobs:
504509
echo "QUARTO_TEST_EXPECTED_VERSION=$QUARTO_VERSION_INPUT" >> "$GITHUB_ENV"
505510
fi
506511
512+
# Guarded at the step so runs on older refs, which lack this action, never load it.
513+
- if: inputs.pandoc-override-version != ''
514+
uses: ./.github/workflows/actions/pandoc-override
515+
with:
516+
pandoc-version: ${{ inputs.pandoc-override-version }}
517+
507518
- name: Install Tinytex
508519
env:
509520
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎dev-docs/update-pandoc-checklist.md‎

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,39 @@
11
## Updating the bundled version of Pandoc
22

3-
Carlos needs to run this:
3+
Updating Pandoc happens in two phases. Phase 1 (prep) needs no S3 access and can be done by anyone; Phase 2 (archival) is done by whoever holds S3 write credentials, right before merge.
44

5-
- [ ] Ensure archives are upgraded
6-
- [ ] Run `AWS_PROFILE=... ./package/src/quarto-bld update-pandoc PANDOC_VERSION`
7-
- [ ] look at `git diff`, specifically for changes in Pandoc templates, and adjust as needed.
5+
### Phase 1 - prep (no S3, anyone)
86

9-
As a reminder, our templates are kept in the same directories as Pandoc's templates, but with different names. `git diff` will show the diff in Pandoc's template; we have to manually patch
10-
ours. (We can't just use `patch` because the templates have diverged too much)
7+
Regenerate the reference copies and `src/core/pandoc/format-extension.ts`, port relevant changes to Quarto's templates, and verify rendering. Keep `configuration`'s `export PANDOC=` line at the archived version so `./configure.sh`/`.cmd` continues to work.
8+
9+
Run the template/variant regeneration with `--skip-archive`, pointing `QUARTO_PANDOC` at an executable of the target version:
10+
11+
```bash
12+
QUARTO_PANDOC=/path/to/pandoc-3.10.1/bin/pandoc \
13+
./package/src/quarto-bld update-pandoc 3.10.1 --skip-archive
14+
```
15+
16+
`--skip-archive` skips S3 archival, the `configureDependency` download, and the `configuration` rewrite. It still runs `writePandocTemplates` to regenerate the mapped template files and `writeVariants` to regenerate `format-extension.ts`.
17+
18+
`QUARTO_PANDOC` is required: `writeVariants` uses that binary to enumerate formats and extensions. The command checks its version before regenerating files. Review and commit the resulting working-tree diff.
19+
20+
`writePandocTemplates` overwrites the reference copies and mapped Pandoc partials such as `fonts.latex` and `hypersetup.latex`. Port relevant changes to Quarto's templates using the [file list below](#pandoc-templates). Review generated changes to `format-extension.ts` as well. The same review applies to the full Phase 2 run.
21+
22+
To smoke-test rendering before archival, dispatch `.github/workflows/test-smokes.yml` with `pandoc-override-version` set to the new release tag (e.g. `3.10.1`). The workflow downloads Pandoc from GitHub and uses it via `QUARTO_PANDOC` for that run, without S3 access. Limit `buckets` to the relevant smoke-all directories (e.g. `typst`, `latex`, `table`) for a faster run.
23+
24+
For a full-suite run, the action sets `QUARTO_TEST_EXPECTED_PANDOC`. The version check accepts either OK or a strict-mismatch note for the requested version, normalized as in `check.ts`.
25+
26+
### Phase 2 - archival (S3 credential holder, before merge)
27+
28+
Whoever holds S3 write credentials does this last, right before merging:
29+
30+
- [ ] Bump `configuration`'s `export PANDOC=` line and the `Pandoc` entry in `src/command/check/check.ts`'s `versionConstraints` array to the new version (this constraint tracks the bundled version exactly).
31+
- [ ] Ensure archives are upgraded.
32+
- [ ] Run `AWS_PROFILE=... ./package/src/quarto-bld update-pandoc PANDOC_VERSION` (without `--skip-archive`) to archive the binary to S3, configure it locally, and regenerate templates/variants.
33+
- [ ] Review `git diff` for regenerated reference copies, mapped Pandoc partials, and `format-extension.ts`. Port any remaining upstream changes to Quarto's templates and reapply any Quarto changes overwritten in mapped partials.
34+
- [ ] Run the `create-release.yml` dry-run signing/notarization gate (see `dev-docs/upgrade-dependencies.md`), then merge.
35+
36+
Quarto's templates share directories with the Pandoc reference copies, but use different names. Review the reference-copy diffs and port relevant changes manually; the templates have diverged too much to use `patch`.
1137

1238
### Pandoc templates
1339

@@ -25,6 +51,36 @@ Partials:
2551
- Ours:
2652
- src/resources/formats/beamer/pandoc/common.latex
2753

54+
#### pdf / latex
55+
56+
- Pandoc's: src/resources/formats/pdf/pandoc/latex.template
57+
- Ours: src/resources/formats/pdf/pandoc/template.tex (see `llm-docs/pandoc-quarto-latex-templates.md` for the full partial breakdown, including `latex.common`/`common.latex`)
58+
59+
#### html
60+
61+
- Pandoc's: src/resources/formats/html/pandoc/html.template, src/resources/formats/html/pandoc/html.styles
62+
- Ours: src/resources/formats/html/pandoc/template.html, src/resources/formats/html/pandoc/styles.html
63+
64+
#### revealjs
65+
66+
- Pandoc's: src/resources/formats/revealjs/pandoc/revealjs.template
67+
- Ours: src/resources/formats/revealjs/pandoc/template.html
68+
69+
#### asciidoc
70+
71+
- Pandoc's: src/resources/formats/asciidoc/pandoc/asciidoc.template
72+
- Ours: src/resources/formats/asciidoc/pandoc/template.asciidoc
73+
74+
#### typst
75+
76+
Typst keeps Pandoc's `template.typst` partial under its original name alongside the renamed `typst.template`. See `llm-docs/pandoc-quarto-typst-templates.md` for Quarto's partials.
77+
78+
The files labelled "Pandoc's" are reference copies used to review upstream changes. Rendering uses the corresponding "Ours" files and their partials, such as `tables.tex` and `toc.tex`.
79+
80+
- Regenerate reference copies from upstream with `writePandocTemplates`; apply rendering changes to Quarto's templates and partials.
81+
- Keep reference copies identical to upstream, including any upstream typos. Manual edits are overwritten on regeneration. Fix upstream bugs in Quarto's templates.
82+
- List the directory without filtering by extension when checking the files: `*.latex` excludes `latex.common`, which ends in `.common`.
83+
2884
## Manual steps
2985

3086
- [ ] Update schemas by inspecting [their changelog](https://github.com/jgm/pandoc/blob/main/changelog.md) for new commands, deprecation removals, etc

‎dev-docs/upgrade-dependencies.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
Change version numbers in `./configuration` to correspond to new versions.
22

3-
Update hardcoded version strings in `src/command/check/check.ts` (the `versionConstraints` array in `checkVersions`) so that they match the new versions in `configuration`. The `configuration` file warns about this in a comment, and `tests/smoke/check/check-versions-no-mismatch.test.ts` fails until they match.
3+
Update hardcoded version strings in `src/command/check/check.ts` (the `versionConstraints` array in `checkVersions`) so that they match the new versions in `configuration`. The `configuration` file warns about this in a comment, and `tests/smoke/check/check-versions-no-mismatch.test.ts` fails until they match. The constraint tracks the bundled version exactly, not a minimum floor.
4+
5+
For Pandoc, follow [update-pandoc-checklist.md](update-pandoc-checklist.md) for template updates and filename conventions, schema and Lua type checks, and CI smoke tests using `QUARTO_PANDOC` before S3 archival.
46

57
## Verify installer signing & notarization before merging (bundled binaries)
68

‎llm-docs/localization-architecture.md‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,13 @@ key_files:
1616
- src/resources/filters/modules/authors.lua
1717
- src/resources/filters/modules/callouts.lua
1818
- src/resources/filters/layout/meta.lua
19-
- src/resources/formats/html/pandoc/html.template
19+
- src/resources/formats/html/pandoc/template.html
20+
- src/resources/formats/html/pandoc/toc.html
2021
- src/resources/formats/html/pandoc/title-block.html
2122
- src/resources/formats/html/templates/title-metadata.html
22-
- src/resources/formats/pdf/pandoc/latex.template
2323
- src/resources/formats/pdf/pandoc/babel-lang.tex
2424
- src/resources/formats/pdf/pandoc/toc.tex
25-
- src/resources/formats/beamer/pandoc/beamer.template
26-
- src/resources/formats/typst/pandoc/typst.template
25+
- src/resources/formats/beamer/pandoc/toc.tex
2726
- src/resources/formats/typst/pandoc/quarto/typst-template.typ
2827
- src/resources/formats/typst/pandoc/quarto/typst-show.typ
2928
- src/resources/extension-subtrees/orange-book/_extensions/orange-book/typst-show.typ
@@ -151,12 +150,14 @@ These bypass both Pandoc metadata and Lua filters. The string lands directly in
151150

152151
`quarto.*` is an internal namespace. The user-facing override path for localized strings is the top-level `language:` YAML key (resolved by `formatLanguage`, already merged into `options.format.language` before the builder runs). The schema does not advertise `variables.quarto.*` as a user option, but the merge in `generateDefaults` honors a user-set value on collision: it deep-merges via `mergeConfigs` (`src/core/config.ts`) so user-supplied keys win at any nesting depth while all other localized values remain available. For example, a user setting `variables.quarto.language.crossref-ch-prefix: Bouquin` overrides only that one leaf — `$quarto.language.toc-title-document$` still resolves to the localized fallback. (Same helper is used in `src/core/language.ts:formatLanguage` to merge user-supplied `language:` onto `_language.yml` defaults.) A non-object `variables.quarto` (string, number, array) is ignored defensively. New contributors to this file should design around the `language:` path, not around the escape hatch.
153152

154-
From any Pandoc template — built-in (`html.template`, `latex.template`, `typst-template.typ`), extension partial, or custom user template — every localized key is then accessible as:
153+
From any Pandoc template — built-in (`template.html`, `template.tex`, `typst-template.typ`), extension partial, or custom user template — every localized key is then accessible as:
155154

156155
```pandoc
157156
$quarto.language.<key>$
158157
```
159158

159+
> **Naming note**: Citations below refer to Quarto's rendering templates: `template.html` (HTML), `template.tex` (LaTeX/PDF and Beamer), and `quarto/typst-template.typ` (Typst). See `dev-docs/update-pandoc-checklist.md` for the corresponding Pandoc reference copies.
160+
160161
For example, `$quarto.language.crossref-ch-prefix$` resolves to `Chapitre` when `lang: fr` is set. Hyphens within the leaf segment are valid Pandoc template-variable syntax (manual: *"Variable names can include letters, numbers, underscores, hyphens, and periods"*).
161162

162163
This is the channel of choice for new template consumers because:
@@ -173,9 +174,9 @@ It does **not** replace 2a — Lua filters still need `param("key")` because Pan
173174

174175
Localization paths used:
175176

176-
- **Document `lang` attribute**: `_language.yml` lookup not involved. `lang:` flows as Pandoc-native metadata, template renders `<html lang="$lang$" xml:lang="$lang$">` (`html.template:2`).
177-
- **TOC title**: `$toc-title$` resolved via channel 2b. Used in `html.template:60`, `toc.html:3`.
178-
- **Abstract title**: `$abstract-title$` resolved via 2b. Used in `html.template:51`, `title-block.html:15`.
177+
- **Document `lang` attribute**: `_language.yml` lookup not involved. `lang:` flows as Pandoc-native metadata, template renders `<html$if(lang)$ lang="$lang$" xml:lang="$lang$"$endif$>` (`template.html:2`).
178+
- **TOC title**: `$toc-title$` resolved via channel 2b. Used in `toc.html:3`.
179+
- **Abstract title**: `$abstract-title$` resolved via 2b. Used in `title-block.html:15`.
179180
- **Title block labels** (`Authors`, `Affiliations`, `Published`, `Modified`, `Doi`, `Abstract`, `Keywords`): `$labels.*$` written into meta by `modules/authors.lua:854-913` (`computeLabels`). Templates: `title-metadata.html:3,4,32,43,52,61,74,83`, `manuscript/title-metadata.html:6,7,37,48,57,66,83,92`.
180181
- **Crossref text** (`Figure 1.1`, `Table 2.1`): assembled in Lua by `crossref/format.lua` using `title()` / `refPrefix()` which call `param("crossref-<type>-title"/"-prefix")`. Written as inline text directly into the AST. By the time Pandoc renders HTML, the localized prefix is already document content.
181182
- **Callout titles** (`Tip`, `Note`, etc.): `modules/callouts.lua:15,185` reads `param("callout-<type>-title", default)`, writes into the callout node.
@@ -191,7 +192,7 @@ Three localization paths layered:
191192

192193
Other Lua filters use `metaInjectLatex` for non-language LaTeX customization (`crossref/custom.lua:78`, `layout/meta.lua`, `quarto-post/landscape.lua`, etc. — they inject packages or styling, not localized strings).
193194

194-
3. **Pandoc template `$var$`** — `pdf/pandoc/toc.tex:3` and `pdf/pandoc/latex.template:93-94` use `$toc-title$` to set `\contentsname`. Beamer templates (`beamer/pandoc/beamer.template:146-151`, `beamer/pandoc/toc.tex:3`) likewise.
195+
3. **Pandoc template `$var$`** — `pdf/pandoc/toc.tex:2-3` uses `$toc-title$` to set `\contentsname`. Beamer's `beamer/pandoc/toc.tex:2-3` does the same.
195196

196197
PDF-side TS code does not read `format.language` directly except `src/format/pdf/format-pdf.ts:242` which registers `"babel-lang"` as a Quarto partial.
197198

0 commit comments

Comments
 (0)