Centralize the complete benchmark harness in v3

.github/workflows/ci.yml

@@ -13,10 +13,20 @@ jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           persist-credentials: false
+      - uses: julia-actions/setup-julia@fa02766e078afaaf09b14210362cee14137e6a32 # v3
+        with:
+          version: '1.12.6'
       - name: Check shell syntax
-        run: bash -n run-benchmark.sh test/run-benchmark.sh test/fixture-workspace/head/benchmark/precompile/run.sh
-      - name: Test benchmark invocation
-        run: test/run-benchmark.sh
+        run: bash -n build-release.sh run.sh test/integration.sh
+      - name: Parse Julia tools and fixture
+        run: |
+          julia --startup-file=no --history-file=no -e '
+              for path in ARGS
+                  Meta.parseall(read(path, String); filename = path)
+              end
+          ' scenario-driver.jl summarize.jl test/fixture/src/PrecompileBenchmarkFixture.jl test/fixture/lib/FixtureSupport/src/FixtureSupport.jl test/fixture/benchmark/precompile/scenarios.jl
+      - name: Run the fixture comparison and release checks
+        run: test/integration.sh

README.md

@@ -1,12 +1,10 @@
 # Julia precompile benchmark
 
-A composite GitHub Action that compares Julia package-cache build, import, and
-first-use latency between the exact base and head of a pull request.
-
-The package owns the benchmark scenarios and measurement logic in
-`benchmark/precompile/run.sh`. This action provides the shared CI orchestration:
-it checks out both revisions, installs Julia, runs the head revision's harness,
-writes its Markdown comparison to the job summary, and uploads all raw results.
+This composite GitHub Action compares Julia package-cache build, import, and
+first-use latency between the exact base and head revisions of a pull request.
+It owns the isolated consumer environment, measurement schedule, scenario
+driver, result validation, summaries, and artifact upload. A package supplies
+only `benchmark/precompile/scenarios.jl` with its package-specific workloads.
 
 ## Usage
 
@@ -35,36 +33,75 @@ jobs:
   compare:
     name: Compare base and head
     runs-on: ubuntu-latest
-    timeout-minutes: 120
+    timeout-minutes: 180
     steps:
-      - uses: QuantumSavory/julia-precompile-benchmark@v2
+      - uses: QuantumSavory/julia-precompile-benchmark@v3
 ```
 
-The path filter belongs to the caller because GitHub Actions cannot define the
-events that invoke a consumer workflow. The example runs for every change under
-`ext`, `src`, or `benchmark/precompile`; dependency and workflow changes also
-run it.
+The caller owns the trigger because an action cannot define workflow events.
+The example therefore runs for every change under `ext`, `src`, or
+`benchmark/precompile`, as well as dependency and workflow changes.
+
+The action uses Julia 1.12.6. Its optional `builds` and `samples` inputs default
+to two independent package-cache builds and five fresh-process measurements per
+scenario and build.
 
-## Inputs
+## Scenario registry
 
-| Input | Required | Default | Description |
-| --- | --- | --- | --- |
-| `julia-version` | no | `1.12.6` | Julia version used for the comparison |
-| `builds` | no | `2` | Independent package-cache builds |
-| `samples` | no | `5` | Fresh-process samples per scenario and build |
+The driver imports the package named by the checkout's root `Project.toml`, then
+includes `benchmark/precompile/scenarios.jl`. That file defines workload
+functions and an ordered, nonempty `NamedTuple` named
+`PRECOMPILE_BENCHMARKS`:
 
-The action invokes the head checkout as follows:
+```julia
+exercise_api() = check(MyPackage.answer() == 42, "unexpected answer")
+
+const PRECOMPILE_BENCHMARKS = (
+    exercise_api = exercise_api,
+)
+```
 
-```text
-benchmark/precompile/run.sh RESULTS BASE=CHECKOUT HEAD=CHECKOUT
+Every registered workload runs in its own fresh Julia process. The central
+driver measures package import, first use, compilation, total latency, and a
+second warm call. Relative path dependencies declared in the package's root
+`[sources]` table are discovered and moved between base and head together with
+the package.
+
+## Local comparisons
+
+Download the versioned tool bundle so `run.sh`, `scenario-driver.jl`, and
+`summarize.jl` stay matched:
+
+```sh
+gh release download v3.0.0 \
+  --repo QuantumSavory/julia-precompile-benchmark \
+  --pattern julia-precompile-benchmark-v3.0.0.tar.gz
+tar -xzf julia-precompile-benchmark-v3.0.0.tar.gz
+julia-precompile-benchmark-v3.0.0/run.sh \
+  benchmark/precompile/scenarios.jl \
+  /tmp/precompile-results \
+  base=/path/to/base \
+  head=/path/to/head
 ```
 
-The harness must be executable. It must write `summary.md` and any raw data
-under the supplied results directory. The action passes repetition controls as
-`PRECOMPILE_BENCHMARK_BUILDS` and `PRECOMPILE_BENCHMARK_SAMPLES`. The package
-harness owns its ordered `PRECOMPILE_BENCHMARKS` registry and runs every
-registered benchmark; callers do not select scenarios through the action.
+The runner requires GNU/Linux, clean committed checkouts, and Julia 1.12.6.
+Its main environment controls are:
+
+- `PRECOMPILE_BENCHMARK_BUILDS` and `PRECOMPILE_BENCHMARK_SAMPLES`
+- `PRECOMPILE_BENCHMARK_BASELINES`, a comma-separated `CANDIDATE=BASELINE` map
+- `PRECOMPILE_BENCHMARK_CONSUMER_PROJECT` and
+  `PRECOMPILE_BENCHMARK_CONSUMER_MANIFEST`, set together to reuse a normalized
+  environment from an earlier run
+- `PRECOMPILE_BENCHMARK_KEEP_TMP=1` to retain the temporary environment
+
+`PRECOMPILE_BENCHMARK_ALLOW_DIRTY=1` and
+`PRECOMPILE_BENCHMARK_ALLOW_JULIA_MISMATCH=1` are available for non-reportable
+smoke runs.
+
+Each successful comparison writes `raw.tsv`, `summary.tsv`,
+`build-summary.tsv`, `summary.md`, `metadata.txt`, `consumer-Project.toml`, and
+`consumer-Manifest.toml`.
 
-Use this action only from `pull_request` workflows. The action executes the
-head revision's benchmark script, so callers should grant only read access to
-the GitHub token.
+Use this action only from `pull_request` workflows. It executes code from the
+pull request head, so callers should keep the GitHub token read-only and must
+not expose secrets.

VERSION

@@ -0,0 +1 @@
+3.0.0

action.yml

@@ -7,10 +7,6 @@ branding:
   color: blue
 
 inputs:
-  julia-version:
-    description: Julia version used by the benchmark harness
-    required: false
-    default: '1.12.6'
   builds:
     description: Number of independent package-cache builds
     required: false
@@ -23,33 +19,50 @@ inputs:
 runs:
   using: composite
   steps:
+    - name: Validate the event
+      shell: bash
+      env:
+        EVENT_NAME: ${{ github.event_name }}
+      run: |
+        [[ $EVENT_NAME == pull_request ]] || {
+          echo "julia-precompile-benchmark supports only pull_request workflows" >&2
+          exit 2
+        }
+
     - name: Check out the pull request base
-      uses: actions/checkout@v7
+      uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       with:
         repository: ${{ github.event.pull_request.base.repo.full_name }}
         ref: ${{ github.event.pull_request.base.sha }}
         path: base
         persist-credentials: false
 
     - name: Check out the pull request head
-      uses: actions/checkout@v7
+      uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       with:
         repository: ${{ github.event.pull_request.head.repo.full_name }}
         ref: ${{ github.event.pull_request.head.sha }}
         path: head
         persist-credentials: false
 
     - name: Set up Julia
-      uses: julia-actions/setup-julia@v3
+      uses: julia-actions/setup-julia@fa02766e078afaaf09b14210362cee14137e6a32 # v3
       with:
-        version: ${{ inputs.julia-version }}
+        version: '1.12.6'
 
     - name: Measure cold-start latency
       shell: bash
       env:
-        INPUT_BUILDS: ${{ inputs.builds }}
-        INPUT_SAMPLES: ${{ inputs.samples }}
-      run: "$GITHUB_ACTION_PATH/run-benchmark.sh"
+        PRECOMPILE_BENCHMARK_BUILDS: ${{ inputs.builds }}
+        PRECOMPILE_BENCHMARK_SAMPLES: ${{ inputs.samples }}
+        PRECOMPILE_BENCHMARK_HARNESS_REPOSITORY: ${{ github.action_repository }}
+        PRECOMPILE_BENCHMARK_HARNESS_REF: ${{ github.action_ref }}
+      run: |
+        "$GITHUB_ACTION_PATH/run.sh" \
+          "$GITHUB_WORKSPACE/head/benchmark/precompile/scenarios.jl" \
+          "$GITHUB_WORKSPACE/cold-start-results" \
+          "base=$GITHUB_WORKSPACE/base" \
+          "head=$GITHUB_WORKSPACE/head"
 
     - name: Add comparison to the job summary
       if: always() && hashFiles('cold-start-results/summary.md') != ''
@@ -58,7 +71,7 @@ runs:
 
     - name: Upload raw results
       if: always() && hashFiles('cold-start-results/**') != ''
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
       with:
         name: cold-start-results
         path: cold-start-results

build-release.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+usage() {
+    echo "usage: $0 OUTPUT_DIR" >&2
+    exit 2
+}
+
+[[ $# == 1 ]] || usage
+
+repository_root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)
+version=$(<"$repository_root/VERSION")
+[[ $version =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
+    echo "invalid VERSION: $version" >&2
+    exit 2
+}
+
+output_dir=$1
+mkdir -p -- "$output_dir"
+output_dir=$(cd -- "$output_dir" && pwd -P)
+package_name="julia-precompile-benchmark-v$version"
+archive_path="$output_dir/$package_name.tar.gz"
+checksum_path="$output_dir/SHA256SUMS"
+for output_path in "$archive_path" "$checksum_path"; do
+    [[ ! -e $output_path && ! -L $output_path ]] || {
+        echo "refusing to overwrite release output: $output_path" >&2
+        exit 2
+    }
+done
+
+temporary_root=$(mktemp -d)
+trap 'rm -rf -- "$temporary_root"' EXIT
+package_root="$temporary_root/$package_name"
+install -d -m 0755 -- "$package_root"
+
+regular_files=(LICENSE README.md VERSION scenario-driver.jl summarize.jl)
+executable_files=(run.sh)
+for relative_path in "${regular_files[@]}" "${executable_files[@]}"; do
+    [[ -f "$repository_root/$relative_path" && -r "$repository_root/$relative_path" ]] || {
+        echo "release input is missing or unreadable: $relative_path" >&2
+        exit 2
+    }
+done
+for relative_path in "${regular_files[@]}"; do
+    install -m 0644 -- "$repository_root/$relative_path" "$package_root/$relative_path"
+done
+for relative_path in "${executable_files[@]}"; do
+    install -m 0755 -- "$repository_root/$relative_path" "$package_root/$relative_path"
+done
+
+tar \
+    --sort=name \
+    --mtime='UTC 1970-01-01' \
+    --owner=0 \
+    --group=0 \
+    --numeric-owner \
+    -C "$temporary_root" \
+    -cf - \
+    "$package_name" | gzip -n > "$archive_path"
+
+(
+    cd -- "$output_dir"
+    sha256sum "$package_name.tar.gz"
+) > "$checksum_path"