Skip to content

Use up-tree lib_dir (rmarkdown 2.32) instead of rewriting lib paths after render #88

Description

@nuest

Summary

rmarkdown 2.32 (merged, not yet on CRAN) contains two upstream fixes that let us delete a substantial amount of workaround code in the rendering pipeline:

  • rstudio/rmarkdown#2632html_document_base() no longer errors when lib_dir points outside the output directory. An up-tree lib_dir: "../../libs" now works and dependencies are referenced with up-tree relative hrefs instead of failing with "The path <file> does not appear to be a descendant of <dir>". See the maintainer's comment on #1859.
  • rstudio/rmarkdown#1632 (also in the 2.32 NEWS) — render() in a fork cluster no longer deletes sibling renders' rmarkdown-str*.html temp files from the shared tempdir().

The first is the reason the "descendant" error forced our render-locally-then-rewrite approach. The second is the cause of the flaky parallel renders we currently sweep up after.

What we do today

Every page renders its HTML dependencies into a page-local libs/, and we then rewrite the hrefs and delete that directory:

  • R/utils_render_register_htmls.R:23generate_html_document_yml() hardcodes lib_dir: libs (with self_contained: false)
  • R/utils_render_register_htmls.R:288edit_html_lib_paths() computes the depth to docs/ and regex-rewrites ="(../)*libs/ in the rendered HTML
  • R/utils_render_register_htmls.R:276-281 and R/utils_render_cert_htmls.R:321-324 — call it, then unlink() the libs/ directory that was just written

On top of that, R/utils_render_cert_htmls.R:232-256 sweeps up after failed parallel renders: stray libs/ directories and stray index_header.html / index_prefix.html / index_postfix.html / html_document.yml / temp.md files. The inline comment names the cause precisely — "forked processes share /tmp, causing occasional pandoc temp file conflicts" — which is rmarkdown#1632.

register/Makefile:49 (make clean) additionally has to find docs/certs -type d -name "libs" -exec rm -rf {} +.

Proposed approach

  1. Compute the relative depth from the page's output directory to docs/libs in generate_html_document_yml() and emit lib_dir: <../ × depth>libs instead of the fixed lib_dir: libs. The depth logic already exists inside edit_html_lib_paths() and can move there.
  2. Delete edit_html_lib_paths() and its two call sites, together with the two unlink(file.path(output_dir, "libs")) calls.
  3. Remove the parallel-render cleanup block in R/utils_render_cert_htmls.R:232-256 once a full parallel render is confirmed clean under 2.32.
  4. Drop the libs removal from make clean in the register repository.

Path resolution is unambiguous for us: temp.md is written into output_dir before rendering, so the input directory and the output directory are the same and a relative lib_dir resolves identically either way.

Open question: concurrent writes into the shared docs/libs

Today each parallel worker owns a private libs/ directory. Afterwards they all copy into the same one, and htmltools' dependency copy is not locked, so two workers can write the same file at the same time. Mitigation options, cheapest first:

  • render one certificate serially to populate docs/libs, then fan out
  • pre-seed docs/libs before the parallel section, reusing the machinery in R/utils_external_libs.R (external_library_specs(), libs_are_current())

This needs to be settled before the cleanup block in step 3 is removed, otherwise we trade one race for another.

Dependency and Dockerfile

rmarkdown 2.32 is not on CRAN yet (CRAN is at 2.31). Until it is released:

  • DESCRIPTION needs rmarkdown (>= 2.32) in Imports once we rely on the new behaviour, plus a Remotes: rstudio/rmarkdown entry while it is unreleased
  • the register's Dockerfile installs the package with remotes::install_github("codecheckers/codecheck") and ENV R_REMOTES_UPGRADE="never"; it must additionally install the development rmarkdown, e.g. an explicit remotes::install_github("rstudio/rmarkdown") step before the codecheck install, so the image does not silently fall back to the CRAN 2.31 and fail every render with the descendant error

Both the Remotes: entry and the extra Dockerfile step should be removed again once 2.32 reaches CRAN.

Suggested order of work

Prototype against the GitHub build of rmarkdown and time a full parallel render before committing to it; merge when 2.32 ships.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions