Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/testing/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Run configuration script to set up:
```

Managed via:
- **R**: renv (`renv.lock`)
- **R**: rig project (`rproj.toml`, `rproj.lock`); `run-tests` sets `QUARTO_R` to `tests/.rvenv/bin`
- **Python**: uv (`pyproject.toml`, `uv.lock`)
- **Julia**: Pkg.jl (`Project.toml`, `Manifest.toml`)

Expand Down
18 changes: 0 additions & 18 deletions .claude/rules/testing/unit-test-with-r.md

This file was deleted.

3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,6 @@ tools/codemod/** text eol=lf
*.otf binary


# rig proj sync rewrites this file with LF line endings; a CRLF checkout
# would then show it as modified after every sync.
tests/.Renviron text eol=lf
40 changes: 40 additions & 0 deletions .github/actions/setup-rig/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: "Set up rig"
description: "Installs a pinned rig release and adds it to PATH. rig proj is experimental, so callers get one pinned version rather than latest."
inputs:
version:
description: "rig version to install, without the leading v"
required: false
default: "0.11.1"

runs:
using: "composite"
steps:
- name: Install rig (Windows)
if: runner.os == 'Windows'
shell: pwsh
env:
RIG_VERSION: ${{ inputs.version }}
run: |
$zip = Join-Path $env:RUNNER_TEMP "rig.zip"
$dest = Join-Path $env:RUNNER_TEMP "rig"
Invoke-WebRequest -Uri "https://github.com/r-lib/rig/releases/download/v$($env:RIG_VERSION)/rig-windows-x86_64-$($env:RIG_VERSION).zip" -OutFile $zip
Expand-Archive -Path $zip -DestinationPath $dest -Force
Join-Path $dest "bin" | Out-File -FilePath $env:GITHUB_PATH -Append -Encoding utf8
& (Join-Path $dest "bin/rig.exe") --version

- name: Install rig
if: runner.os != 'Windows'
shell: bash
env:
RIG_VERSION: ${{ inputs.version }}
run: |
case "$RUNNER_OS-$RUNNER_ARCH" in
Linux-X64) asset="rig-linux-x86_64-${RIG_VERSION}.tar.gz" ;;
macOS-ARM64) asset="rig-macos-arm64-${RIG_VERSION}.tar.gz" ;;
*) echo "::error::No rig build configured for $RUNNER_OS-$RUNNER_ARCH"; exit 1 ;;
esac
dest="$RUNNER_TEMP/rig"
mkdir -p "$dest"
curl -fsSL "https://github.com/r-lib/rig/releases/download/v${RIG_VERSION}/${asset}" | tar -xz -C "$dest"
echo "$dest/bin" >> "$GITHUB_PATH"
"$dest/bin/rig" --version
4 changes: 2 additions & 2 deletions .github/workflows/test-ff-matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
workflow_call:
inputs:
extra-r-packages:
description: "Comma-separated R packages to pass to renv::install"
description: "Comma-separated R packages to pass to rig pkg install (see test-smokes.yml)"
required: false
type: string
default: ""
Expand Down Expand Up @@ -46,7 +46,7 @@ on:
workflow_dispatch:
inputs:
extra-r-packages:
description: "extra R package to install for the runs (like a dev version of one of the deps) - comma separated, passed to renv::install"
description: "extra R packages installed on top of tests/rproj.lock (like a dev version of one of the deps) - comma separated, passed to rig pkg install: pkg, owner/repo[@ref], gitlab::group/project, url::https://...; pkg@1.2.3 means exactly 1.2.3"
required: false
type: string
default: ""
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-smokes-parallel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
default: 10
type: number
extra-r-packages:
description: "extra R package to install for the tests runs (like a dev version of one of the deps) - comma separated, passed to renv::install"
description: "extra R packages installed on top of tests/rproj.lock (like a dev version of one of the deps) - comma separated, passed to rig pkg install: pkg, owner/repo[@ref], gitlab::group/project, url::https://...; pkg@1.2.3 means exactly 1.2.3"
required: false
type: string
default: ""
Expand Down
162 changes: 77 additions & 85 deletions .github/workflows/test-smokes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ on:
type: boolean
default: false
extra-r-packages:
description: "extra R package to install for the runs (like a dev version of one of the deps) - comma separated, passed to renv::install"
description: "extra R packages installed on top of tests/rproj.lock (like a dev version of one of the deps) - comma separated, passed to rig pkg install: pkg, owner/repo[@ref], gitlab::group/project, url::https://...; pkg@1.2.3 means exactly 1.2.3"
required: false
type: string
default: ""
Expand Down Expand Up @@ -67,7 +67,7 @@ on:
type: boolean
default: false
extra-r-packages:
description: "extra R package to install for the runs (like a dev version of one of the deps) - comma separated, passed to renv::install"
description: "extra R packages installed on top of tests/rproj.lock (like a dev version of one of the deps) - comma separated, passed to rig pkg install: pkg, owner/repo[@ref], gitlab::group/project, url::https://...; pkg@1.2.3 means exactly 1.2.3"
required: false
type: string
default: ""
Expand Down Expand Up @@ -110,6 +110,16 @@ jobs:
# Empty keeps the standard checkout behavior.
ref: ${{ inputs.ref }}

- name: Check the ref supports the rig test environment
shell: bash
run: |
for f in tests/rproj.lock .github/actions/setup-rig/action.yml; do
if [ ! -f "$f" ]; then
echo "::error::This checkout has no $f. The R test environment needs a ref that uses the rig project (releases before it was introduced are not supported)."
exit 1
fi
done

# macOS runners boot with an unresolvable `.local` hostname, so any process
# resolving its own hostname stalls ~35s (actions/runner-images#14568).
- name: Set a resolvable hostname (macOS)
Expand All @@ -128,7 +138,7 @@ jobs:
shell: bash

# Drops a slow azure.archive.ubuntu.com from the apt mirror list so the job uses
# archive.ubuntu.com (actions/runner-images#14823): sampled here before setup-r,
# archive.ubuntu.com (actions/runner-images#14823): sampled here before rig installs R,
# and after a failed attempt in later apt steps run via $APT_WITH_FALLBACK.
# Inline, not a local action, because release mode checks out older tags.
- name: Configure apt mirror fallback
Expand Down Expand Up @@ -191,13 +201,8 @@ jobs:
fi
exit 0

- name: Set up R
uses: r-lib/actions/setup-r@v2
with:
r-version: "4.6.1"
use-public-rspm: true
# required to avoid rtools bin in path
windows-path-include-rtools: false
- name: Set up rig
uses: ./.github/actions/setup-rig

- name: Install node (for Playwright, MECA)
uses: actions/setup-node@v6
Expand Down Expand Up @@ -292,29 +297,6 @@ jobs:
if: ${{ runner.os != 'Windows' }}
run: npm install -g meca --loglevel=error --no-audit

- name: Set RENV_PATHS_ROOT
shell: bash
run: |
echo "RENV_PATHS_ROOT=${{ runner.temp }}/renv" >> $GITHUB_ENV

- name: Get R and OS version
id: get-version
run: |
cat("os-version=", sessionInfo()$running, "\n", file = Sys.getenv("GITHUB_OUTPUT"), sep = "", append = TRUE)
cat("r-version=", R.Version()$version.string, "\n", file = Sys.getenv("GITHUB_OUTPUT"), sep = "", append = TRUE)
shell: Rscript {0}

- name: Restore Renv package cache
id: cache-renv-packages-restore
uses: actions/cache/restore@v5
with:
path: |
${{ env.RENV_PATHS_ROOT }}
renv/library
key: ${{ steps.get-version.outputs.os-version }}-${{ steps.get-version.outputs.r-version }}-renv-2-${{ hashFiles('tests/renv.lock') }}
restore-keys: |
${{ steps.get-version.outputs.os-version }}-${{ steps.get-version.outputs.r-version }}-renv-2-

- name: Install missing system deps
if: runner.os == 'Linux'
run: |
Expand All @@ -340,56 +322,77 @@ jobs:
brew install --cask xquartz
brew install poppler librsvg

- name: Restore R packages
id: restore-r-packages
- name: Install R from rproj.lock
working-directory: tests
shell: bash
run: |
cat("::group::Installing renv if needed\n")
if (!requireNamespace('renv', quietly = TRUE)) install.packages('renv')
cat("::endgroup::\n")
cat("::group::Restoring R packages from renv.lock\n")
renv::restore()
cat("::endgroup::\n")
cat("::group::Verifying R packages are synchronized after restore\n")
status <- renv::status()
if (!isTRUE(status$synchronized)) {
cat("::error::R packages are not synchronized after restore - some packages may have failed to install\n")
quit(status = 1)
}
cat("::endgroup::\n")
cat("::group::Installing dev versions of knitr and rmarkdown\n")
# Install dev versions for our testing
# Use r-universe to avoid github api calls
try(install.packages('rmarkdown', repos = c('https://rstudio.r-universe.dev', getOption('repos'))))
try(install.packages('knitr', repos = c('https://yihui.r-universe.dev', getOption('repos'))))
cat("::endgroup::\n")
# A failed dev install keeps the renv-restored version; flag it instead of failing setup
for (pkg in c('rmarkdown', 'knitr')) {
# A separate step because on Linux rig proj sync re-runs itself under sudo
# to install R, which leaves the project library owned by root.
r_version=$(sed -n 's/^r_version = "\(.*\)"/\1/p' rproj.lock | head -n 1 | tr -d '\r')
rig add "$r_version"

- name: Sync R packages from rproj.lock
working-directory: tests
shell: bash
run: |
# Refs that predate the rig project are rejected by the preflight step after checkout.
echo "::group::rig proj sync"
rig proj sync --locked --no-install-r
echo "::endgroup::"

- name: Install dev and extra R packages
working-directory: tests
shell: bash
env:
EXTRA_R_PACKAGES: ${{ inputs.extra-r-packages }}
run: |
# Installs on top of the lock must target the project library and the
# exact R the sync used; rig's default is the default R's user library.
r_version=$(sed -n 's/^r-version = //p' .rvenv/rvenv.cfg | tr -d '\r')
r_lib="$(cd .rvenv/lib && { pwd -W 2>/dev/null || pwd; })"
overlay() { rig pkg install --r-version "$r_version" --library "$r_lib" "$@"; }

echo "::group::Installing dev versions of knitr and rmarkdown"
# Use r-universe to avoid GitHub API calls. A failed dev install keeps
# the locked version; the check below flags it instead of failing setup.
overlay --with-repos https://rstudio.r-universe.dev rmarkdown || true
overlay --with-repos https://yihui.r-universe.dev knitr || true
echo "::endgroup::"

if [ -n "$EXTRA_R_PACKAGES" ]; then
echo "::notice::Running with the following extra R packages: $EXTRA_R_PACKAGES"
echo "::group::Installing extra R packages"
specs=()
IFS=',' read -ra items <<< "$EXTRA_R_PACKAGES"
for item in "${items[@]}"; do
item="${item#"${item%%[![:space:]]*}"}"
item="${item%"${item##*[![:space:]]}"}"
# renv's pkg@1.2.3 is an exact version; rig reads it as a minimum.
if [[ "$item" =~ ^[A-Za-z][A-Za-z0-9.]*@[0-9] ]]; then item="${item/@/@=}"; fi
specs+=("$item")
done
overlay "${specs[@]}"
echo "::endgroup::"
fi

echo "::group::Installing sessioninfo"
overlay sessioninfo
echo "::endgroup::"

# Run through the wrapper so the check sees the library the tests load.
cat > .rvenv/ci-r-check.R <<'EOF'
for (pkg in c("rmarkdown", "knitr")) {
repo <- packageDescription(pkg)$Repository
if (is.null(repo) || !grepl('r-universe.dev', repo, fixed = TRUE)) {
if (is.null(repo) || !grepl("r-universe.dev", repo, fixed = TRUE)) {
cat(sprintf("::notice::Dev version of %s not installed; tests use %s from %s\n",
pkg, packageVersion(pkg), if (is.null(repo)) "unknown source" else repo))
pkg, format(packageVersion(pkg)), if (is.null(repo)) "unknown source" else repo))
}
}
if ('${{ inputs.extra-r-packages }}' != '') {
cat(sprintf("::notice::Running with the following extra R packages for renv: %s\n", "${{ inputs.extra-r-packages }}"))
cat("::group::Installing extra R packages\n")
renv::install(strsplit("${{ inputs.extra-r-packages }}", split = ",")[[1]])
cat("::endgroup::\n")
}
shell: Rscript {0}
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

- name: Show R session information
working-directory: tests
run: |
# Session info
cat("::group::Session info\n")
renv::install("sessioninfo")
sessioninfo::session_info(pkgs = "installed", include_base = TRUE)
cat("::endgroup::\n")
shell: Rscript {0}
EOF
.rvenv/bin/Rscript .rvenv/ci-r-check.R

- name: Install uv for Python
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
Expand Down Expand Up @@ -719,17 +722,6 @@ jobs:
key: ${{ steps.cache-typst.outputs.cache-primary-key }}
path: ${{ steps.cache-typst.outputs.cache-path }}

- name: Save Renv package cache
# don't save cache if we have extra R packages, or if the cache restore or
# renv::restore did not complete (partial library, see Typst cache above)
if: ${{ !cancelled() && steps.restore-r-packages.outcome == 'success' && steps.cache-renv-packages-restore.outputs.cache-hit != 'true' && inputs.extra-r-packages == '' }}
uses: actions/cache/save@v5
with:
path: |
${{ env.RENV_PATHS_ROOT }}
renv/library
key: ${{ steps.cache-renv-packages-restore.outputs.cache-primary-key }}

- uses: actions/upload-artifact@v7
# Include the OS because sibling jobs share the workflow run.
if: ${{ !cancelled() && runner.os != 'Windows' && hashFiles('tests/integration/playwright/playwright-report/**/*') != '' }}
Expand Down
2 changes: 1 addition & 1 deletion .luarc.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,5 @@
"Lua.runtime.version": "Lua 5.3",
"Lua.workspace.checkThirdParty": false,
"Lua.workspace.library": ["src/resources/lua-types"],
"Lua.workspace.ignoreDir": ["tests/renv", ".vscode", "**/node_modules/*"]
"Lua.workspace.ignoreDir": ["tests/.rvenv", ".vscode", "**/node_modules/*"]
}
1 change: 1 addition & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
"QUARTO_ROOT": "${workspaceFolder}",
"QUARTO_BIN_PATH": "${workspaceFolder}/package/dist/bin",
"QUARTO_SHARE_PATH": "${workspaceFolder}/src/resources",
"QUARTO_R": "${workspaceFolder}/tests/.rvenv/bin",
"QUARTO_DEBUG": "true"
},
"attachSimplePort": 9229,
Expand Down
7 changes: 2 additions & 5 deletions dev-docs/configuring-test-deps.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,7 @@

## R

- Add the dependencies in `DESCRIPTION` file
- Run `renv::install(<package>)` to install the dependencies
- Run `renv::snapshot()` to lock the versions in `renv.lock`
- Run `rig proj add <package>` from `tests/`: it adds the dependency to `rproj.toml`, locks it in `rproj.lock` and installs it

## Julia

Expand All @@ -82,6 +80,5 @@

# Updating test dependencies

- When `main` updates the test environment (`tests/renv.lock`, `tests/uv.lock`, `tests/Manifest.toml`, or the R / Python / Julia versions in `.github/workflows/test-smokes.yml`), backport the update to the current stable branch (e.g. `v1.10`) while it still receives patch releases, as its own PR (see `checklist-backport-a-pr.md`).
- When `main` updates the test environment (`tests/rproj.lock`, `tests/uv.lock`, `tests/Manifest.toml`, or the R / Python / Julia versions in `.github/workflows/test-smokes.yml`), backport the update to the current stable branch (e.g. `v1.10`) while it still receives patch releases, as its own PR (see `checklist-backport-a-pr.md`).
- Patch releases are then tested against the same current R, Python, Julia and package versions users install.
- CI caches are shared: the R cache key is built from the OS, the R version and the `tests/renv.lock` hash, and any branch can read caches saved on `main`. A stable branch with identical lock files gets cache hits; a diverged one does a slow cold restore on each PR.
3 changes: 2 additions & 1 deletion llm-docs/built-version-testing-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ In practice:
- For signed Windows artifacts from a branch, dispatch `create-release` with `publish-release=false` and `smoke-artifacts-only=true`. The resulting workflow run starts the available test legs automatically (D7).
- Dispatch `nightly` with a `run-id` to retest an earlier `create-release` run.
- Dispatch `release` for post-publish verification, such as the optional prerelease checklist step.
It works only for releases cut after the harness support merged (D10).
It works only for releases cut after the harness support and the rig test environment merged (D10).

## Built-mode test legs (scheduler layout)

Expand Down Expand Up @@ -298,6 +298,7 @@ All built legs currently run after every completed `create-release` run, unless
### D10. Release mode only works for post-harness tags

Release mode checks out the tag, so the harness at that tag must already contain `tests/quarto-cmd.ts` — a preflight fails clearly for older releases.
The smoke legs also need the rig test environment (`tests/rproj.lock`): on a tag from before it, each leg fails at its "Check the ref supports the rig test environment" step.
Testing an older binary with the current harness would require decoupling the harness from the target ref.

Nightly and release legs use the harness from the target ref. Refs that contain `tests/quarto-cmd.ts` but predate the `quartoSpawnEnvOptions()` fix in `playwright-tests.test.ts` (#14706) run Playwright with dev-tree resources.
Expand Down
Loading
Loading