Skip to content

Copy local files that Markdown links point at into the output (bd-twfep08m) - #967

Merged
cscheid merged 1 commit into
mainfrom
feature/bd-twfep08m-linked-resource-discovery
Oct 29, 2026
Merged

cscheid merged 1 commit into
mainfrom
feature/bd-twfep08m-linked-resource-discovery

Conversation

@cscheid

@cscheid cscheid commented Oct 29, 2026

Copy link
Copy Markdown
Member

Problem

A Markdown link to a local file that isn't a document, such as [Data](data.csv) or [Dashboard](dashboard.json){download="my-dashboard.json"}, rendered with its href unchanged, but q2 never copied the file into the output directory. The link was broken on the published site and the render printed nothing. Quarto 1 copies these files, so sites ported from it (for example the Posit Connect docs) silently lost their downloads.

What changed

  • Discovery. A new LinkResourceDiscoveryTransform runs in the Finalization phase, immediately before link-rewrite, so it sees links produced by shortcodes and pre filters while targets are still source-relative. For each Link target it drops any ?query/#fragment, percent-decodes the path, and resolves it against the document's directory (a leading / resolves against the project root). It keeps the target if it is an existing file inside the project that is none of the following:

    • a .qmd file or any other project input;
    • a hidden path or a path under _freeze/;
    • a file under the output directory;
    • a likely render output of a project document, such as report.pdf next to report.qmd, which would otherwise overwrite the fresh output with a stale in-place copy.

    Each kept file is appended to meta.resources.

  • Attribution and Lua control. ResourceReportStage publishes the discovered entries that are still in meta.resources after the post filters, under a new origin, ResourceOrigin::AutoDiscovery { via: link }. That origin is what appears in .quarto/render-manifest.json. A post filter can therefore remove discovered entries. Entries the author declared in YAML still can't be removed, as before.

  • Opt-out. resource-discovery: false is ordinary merged metadata. It works in front matter, _metadata.yml, _quarto.yml, a format: block, -M, or from a pre filter.

  • New error Q-5-52. Any value other than a boolean is an error that fails the document, so a typo like flase can't publish files the author meant to keep out. To make the error point at the right place, AstTransformsStage now passes a transform's QuartoError::Parse through as PipelineError::Structured. The span then resolves into _quarto.yml and _metadata.yml as well as front matter. No transform returned that variant before, so nothing else changes.

  • Images still go through ResourceCollectorTransform, which Typst needs: images have to be in the output directory before typst compile runs.

  • Docs.

    • paths.qmd: a new section, "Linked files are copied".
    • lua-filters.qmd: a new section on meta.resources, covering pruning from a post filter and the pre-filter alternative.
    • A new Q-5-52 error page.

Plan, with the measured before/after repros and the design decisions: claude-notes/plans/2026-10-29-linked-resource-discovery.md. Braid: bd-twfep08m.

Things worth a look in review

  • Pruning in docx/pptx/typst. For these formats, post filters run inside pandoc, after ResourceReportStage, so editing meta.resources there can't remove anything. The Lua docs tell those users to use resource-discovery: false instead.
  • Pipeline classification. The new transform is classified B3 in BUCKETS (a shared post-core service) and runs for every format, because the resources channel itself is format-agnostic. It is also registered in the q2-preview profile, like resource-collector. In preview's VFS-root mode it does nothing.
  • Render-output check. Deciding that a file is a render output of a project document is a static rule. A file is skipped when its extension is one of html htm pdf docx pptx odt epub typ tex and its path without the extension matches a project input.
  • Out of scope, with follow-ups:
    • navbar, sidebar and footer href: values pointing at files are still not copied (Quarto 1 copies them);
    • an optional lint for local link targets that don't exist;
    • a separate bug, bd-mejgdrlh: a body link that climbs above the project root is silently retargeted inside the site.

Testing

  • Integration tests. crates/quarto-core/tests/integration/linked_resources.rs has 17 tests. They cover:
    • the issue's repro, plus query, fragment and %20 targets;
    • root-relative links, and links in footnotes, tables and callouts;
    • links produced by a shortcode or a pre filter;
    • every case that must not be copied;
    • the stale-output check and an unrendered notebook;
    • the opt-out at each metadata layer (front matter, _metadata.yml, project level, format level), including a document overriding the project;
    • Q-5-52, post-filter pruning combined with the defense of author entries, the pre-filter opt-out, and the manifest origin;
    • a default project with a separate output-dir.
  • Unit tests for link-target parsing, hidden-path detection, appending to meta.resources, and the report stage's classification.
  • CLI check by hand. I checked the Q-5-52 output for a typo in front matter, in _quarto.yml and via -M (correct spans, exit code 1). I also checked that -M resource-discovery:false disables discovery.
  • Verify. cargo xtask verify passes, including the hub-client build and tests.
  • Windows. quarto-core can't be type-checked for Windows from macOS, because the C build scripts of wasmtime and tree-sitter fail. The change adds no platform-specific APIs. Paths go through Path::join, pathdiff and to_forward_slashes.
  • No changelog entry. The only changelog, hub-client/changelog.md, covers hub-client changes, and discovery does nothing in hub-client.

🤖 Generated with Claude Code

…ep08m)

A link such as [Data](data.csv) or [Dashboard](dashboard.json){download=…}
rendered its href unchanged but never copied the file, so websites ported
from Quarto 1 silently lost their downloads. Quarto 1 finds these with an
HTML postprocessor; q2 now finds them on the AST.

LinkResourceDiscoveryTransform (Finalization, just before link-rewrite)
walks Link targets, keeps existing in-project files that are not .qmd,
project inputs, hidden/_freeze paths, files under the output dir, or a
likely render output of a project document (report.pdf beside
report.qmd), and appends them to meta.resources. ResourceReportStage
publishes the survivors of the post filters as the new
ResourceOrigin::AutoDiscovery { via: link }, so a post filter can prune
discovered entries (author-declared ones stay defended, as before).
Images stay on ResourceCollectorTransform.

`resource-discovery: false` (ordinary merged metadata: front matter,
_metadata.yml, _quarto.yml, format blocks, -M, or a pre filter) turns it
off. Any non-boolean value is the new Q-5-52 error and fails the
document, so a typo cannot publish files the author meant to keep out.
AstTransformsStage now passes a transform's QuartoError::Parse through
as PipelineError::Structured so that span resolves into config files.

Plan: claude-notes/plans/2026-10-29-linked-resource-discovery.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@posit-snyk-bot

posit-snyk-bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
cscheid merged commit 63112fe into main Oct 29, 2026
17 checks passed
@cscheid
cscheid deleted the feature/bd-twfep08m-linked-resource-discovery branch October 29, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants