|
| 1 | +# Scanning a site with `quarto call axe` |
| 2 | + |
| 3 | +`quarto call axe` scans a rendered Quarto site for accessibility |
| 4 | +violations with axe-core. It drives headless Chrome over every page, at |
| 5 | +desktop and mobile widths, in each colour mode the page ships. It groups |
| 6 | +violations by root cause, compares them against a committed baseline, and |
| 7 | +writes a report you can read, commit review comments from, or gate CI on. |
| 8 | + |
| 9 | +**Status: experimental.** The command is hidden (it does not show in |
| 10 | +`quarto call` help) and makes no stability promise — flags, artifact shapes, |
| 11 | +and semantics can change between prereleases. This page is contributor-facing |
| 12 | +documentation; it graduates to quarto.org when the command is unhidden (the |
| 13 | +same path the `axe:` render option's docs took). |
| 14 | + |
| 15 | +Prerequisite: a Chromium the scanner can find. `quarto install |
| 16 | +chrome-headless-shell` is the reliable route; an installed system Chrome or |
| 17 | +Edge also works. |
| 18 | + |
| 19 | +## When to reach for it |
| 20 | + |
| 21 | +The render-time `axe:` option checks the page you are looking at, in your |
| 22 | +browser, while you author. This command checks the *site*: every page, both |
| 23 | +viewports, light and dark, at audit time or in CI. Reach for it when you |
| 24 | +want a site-wide inventory, a regression gate, or output an agent can work |
| 25 | +through. |
| 26 | + |
| 27 | +## First scan |
| 28 | + |
| 29 | +```sh |
| 30 | +quarto render |
| 31 | +quarto call axe _site |
| 32 | +``` |
| 33 | + |
| 34 | +The scan prints its matrix up front (`43 pages (40 light+dark, 3 default) × |
| 35 | +2 viewports — 166 cells`), then one line per cell, then a findings summary. |
| 36 | +Artifacts land at the project root (the nearest `_quarto.yml` at or above |
| 37 | +the site dir): |
| 38 | + |
| 39 | +- `_axe-checks/findings.json` — every finding, machine-readable. |
| 40 | +- `_axe-checks/report.md` — the human summary, GitHub-flavored markdown. |
| 41 | +- `_axe-checks/README.md` — generated docs for the artifacts, including the |
| 42 | + baseline how-to. Read this after a scan; it is written for whoever (or |
| 43 | + whatever) has only the artifact directory in front of them. |
| 44 | +- `_axe-checks/cells/` — raw axe output per page × viewport × mode cell. |
| 45 | + |
| 46 | +`_axe-checks/` ignores itself (`.gitignore` with `*`): it is a disposable |
| 47 | +snapshot, not a thing to commit. |
| 48 | + |
| 49 | +### Reading report.md |
| 50 | + |
| 51 | +The report is GitHub-flavored markdown, made to be read where markdown |
| 52 | +already renders: your editor's preview, GitHub, or the sticky PR comment |
| 53 | +from the CI recipe below. Rendering it is optional. For a standalone HTML |
| 54 | +view: |
| 55 | + |
| 56 | +```sh |
| 57 | +quarto render _axe-checks/report.md |
| 58 | +``` |
| 59 | + |
| 60 | +then open `_axe-checks/report.html` in your browser. The output lands |
| 61 | +beside the report, inside the self-ignoring artifact directory. |
| 62 | + |
| 63 | +`quarto preview _axe-checks/report.md` does **not** work from inside a |
| 64 | +project: `_`-prefixed directories are not project inputs, so preview stops |
| 65 | +with `No output created by quarto render report.md`. Use render-then-open, |
| 66 | +or pass `--report` a path inside your site source (see the flags table) to |
| 67 | +render and preview the report as part of the site. |
| 68 | + |
| 69 | +A finding is one *root cause*, not one element: an alt-less image in a |
| 70 | +shared include shows up as one finding with an instance count, not once per |
| 71 | +page. Grouping keys on axe's element selector (normalized), so this holds |
| 72 | +when the pages describe the element the same way — reliably true for |
| 73 | +Quarto's own chrome and for repeated template output, and occasionally |
| 74 | +wrong for an anonymous element whose surrounding DOM differs page to page |
| 75 | +(axe then picks different selectors, and one cause splits into two |
| 76 | +findings). Findings on many pages usually come from a shared source — a |
| 77 | +template, the theme, Quarto's own chrome — and one fix clears them all. |
| 78 | + |
| 79 | +## Flags |
| 80 | + |
| 81 | +| flag | default | | |
| 82 | +|---|---|---| |
| 83 | +| `--pages <globs>` | all `*.html` | comma-separated site-relative globs | |
| 84 | +| `--exclude <globs>` | — | skip globs, applied after `--pages` | |
| 85 | +| `--max-pages <n>` | ∞ | deterministic cap (sorted, first n scannable pages — redirect stubs don't use up the cap) | |
| 86 | +| `--viewports <WxH,...>` | `1440x900,320x568` | | |
| 87 | +| `--themes <light,dark>` | `light,dark` | filters two-mode pages; one-mode pages always scan once | |
| 88 | +| `--timeout <ms>` | `30000` | per-cell budget | |
| 89 | +| `--settle <ms>` | `50` | extra delay after the page reports ready | |
| 90 | +| `--fail-on <impact>` | off | exit 1 on new findings at/above `minor`/`moderate`/`serious`/`critical` | |
| 91 | +| `--report <path>` | `_axe-checks/report.md` | put the report elsewhere, e.g. inside your site source — can't name the baseline, `findings.json`, `README.md`, `.gitignore`, or anything inside `cells/` | |
| 92 | + |
| 93 | +The narrow default viewport is 320 CSS px — the width WCAG's reflow |
| 94 | +criterion (SC 1.4.10) names, equivalent to 400% zoom on a 1280 px window — |
| 95 | +so every rule runs against the reflowed mobile layout. |
| 96 | + |
| 97 | +A subset scan (`--pages`, `--exclude`, `--max-pages`) says so loudly in |
| 98 | +every artifact: its counts describe the subset, not the site. |
| 99 | + |
| 100 | +## Running it as a post-render step |
| 101 | + |
| 102 | +The command works as a project `post-render` script, so a full render scans |
| 103 | +itself: |
| 104 | + |
| 105 | +```yaml |
| 106 | +project: |
| 107 | + type: website |
| 108 | + post-render: |
| 109 | + - quarto call axe _site/ |
| 110 | +``` |
| 111 | +
|
| 112 | +The script runs from the project directory after the outputs are written, |
| 113 | +so the relative site dir and the artifact anchor resolve exactly as they do |
| 114 | +on the command line. Exit codes propagate: without `--fail-on`, findings |
| 115 | +never fail the render; with `--fail-on`, a new finding at the threshold — |
| 116 | +or an incomplete scan — fails `quarto render` itself, with the scan's error |
| 117 | +line in the render output. This adds the scan's runtime to each full render. |
| 118 | + |
| 119 | +Only *full* renders scan. Quarto runs post-render scripts on incremental |
| 120 | +renders and on preview reloads too, and the scan skips both with a note and |
| 121 | +exit 0 — a partly-rebuilt site would mix this render's output with the last |
| 122 | +one's. Run `quarto render` with no file argument to scan. |
| 123 | + |
| 124 | +## The baseline workflow |
| 125 | + |
| 126 | +The first scan of a real site reports findings you will not fix today: |
| 127 | +upstream defects, deferred best-practice items, known false positives. The |
| 128 | +baseline is the committed ledger of those decisions — `_axe-baseline.json` |
| 129 | +at the project root, beside (not inside) `_axe-checks/`. |
| 130 | + |
| 131 | +1. Scan, then fix what you can from `report.md`. |
| 132 | +2. For each finding you are accepting instead: copy it out of |
| 133 | + `findings.json` into the baseline's `findings` array, trim it, and write |
| 134 | + a `note` saying why. The generated `_axe-checks/README.md` documents the |
| 135 | + entry shape and the scoping rules (`pages: []` accepts site-wide; |
| 136 | + a listed `pages` re-alerts anywhere else). |
| 137 | +3. Commit `_axe-baseline.json`. |
| 138 | + |
| 139 | +From then on, reports separate **new** findings from **baselined** ones, |
| 140 | +and only new findings can fail CI. A baselined finding that escalates in |
| 141 | +impact, or shows up on a page outside its scope, re-alerts as new. Entries |
| 142 | +a scan no longer sees are reported as stale — prune them by hand, but only |
| 143 | +after a scan of the full matrix (every page, viewport and mode) in which |
| 144 | +every cell completed. Narrowing any axis with `--pages`, `--viewports` or |
| 145 | +`--themes` means the scan can't tell "fixed" from "not scanned", and neither |
| 146 | +can a full scan that lost a cell; the report says which applies to it. |
| 147 | + |
| 148 | +There is deliberately no `--update-baseline`: every entry exists because |
| 149 | +someone wrote it and said why. |
| 150 | + |
| 151 | +## Exit codes and the CI recipe |
| 152 | + |
| 153 | +| exit | meaning | |
| 154 | +|---|---| |
| 155 | +| `0` | scan complete; no new findings at/above `--fail-on` (when given) | |
| 156 | +| `1` | complete scan, new findings at/above the `--fail-on` threshold | |
| 157 | +| `2` | scan incomplete — a cell timed out or errored, no browser, nothing to scan. Takes precedence over 1: an incomplete scan never reads as a pass | |
| 158 | +| `3` | usage error — a flag value the command can't accept (`--fail-on serius`), or a filter that matches nothing (`--themes dark` on a site with no dark mode). A typo is not a result, so it gets its own code | |
| 159 | + |
| 160 | +A minimal GitHub Actions gate: |
| 161 | + |
| 162 | +```yaml |
| 163 | +- uses: quarto-dev/quarto-actions/setup@v2 |
| 164 | +- run: quarto install chrome-headless-shell --no-prompt |
| 165 | +- run: quarto render |
| 166 | +- run: quarto call axe _site --fail-on serious |
| 167 | +``` |
| 168 | + |
| 169 | +`report.md` is GitHub-flavored markdown, so a workflow can post it straight |
| 170 | +into the PR — no Quarto-side tooling needed. Use a sticky comment, which |
| 171 | +updates in place on each push rather than adding one comment per run: |
| 172 | + |
| 173 | +```yaml |
| 174 | +- if: always() |
| 175 | + uses: marocchino/sticky-pull-request-comment@v2 |
| 176 | + with: |
| 177 | + header: axe |
| 178 | + path: _axe-checks/report.md |
| 179 | +``` |
| 180 | + |
| 181 | +(`if: always()` keeps the comment current when `--fail-on` fails the job; |
| 182 | +`header` keys the comment so other sticky comments are untouched. GitHub |
| 183 | +caps comment bodies at 65,536 characters; a whole-site report on a large |
| 184 | +site can exceed it — trim or attach as an artifact instead.) |
| 185 | + |
| 186 | +## Reading the output as an agent |
| 187 | + |
| 188 | +Point the agent at `_axe-checks/` and let it read the generated `README.md` |
| 189 | +first. Finding ids are stable across runs (`image-alt-6e3b76`), so "fix |
| 190 | +`image-alt-6e3b76`" is a well-defined instruction, and each finding's |
| 191 | +`occurrences[]` carries real selectors and HTML excerpts. Fixes belong in |
| 192 | +Quarto *source* (`.qmd`, `_quarto.yml`, `_brand.yml`, theme `.scss`) — never |
| 193 | +in the rendered site directory. |
| 194 | +
|
| 195 | +## Where the pieces are documented |
| 196 | +
|
| 197 | +- How it works and why: `llm-docs/axe-scan-architecture.md`. |
| 198 | +- What the artifacts mean, baseline entry shape: the generated |
| 199 | + `_axe-checks/README.md` (regenerated every scan, always matches the build |
| 200 | + that wrote it). |
| 201 | +- Render-time single-page checking: the `axe:` HTML format option. |
0 commit comments