Run SDK integration against the matching native client

Author: rustyconover

.github/workflows/integration.yml

@@ -1,13 +1,12 @@
 # Runs the canonical Query-farm/vgi integration sqllogictest suite against the
-# Python example worker on every push / PR. Rather than building the C++ extension
-# from source, it drives a prebuilt standalone `haybarn-unittest` and installs the
-# SIGNED vgi extension from the Haybarn community channel + its deps from core.
-# The .test files come from a pinned Query-farm/vgi checkout; ci/run-integration.sh
-# + ci/preprocess-require.awk adapt them for the standalone runner. See ci/README.md.
+# Python example worker on every push / PR. The native runner comes from the
+# Query-farm/vgi Actions build of the same commit as the test suite and statically
+# links the matching extension. Public package releases remain separate.
+# ci/run-integration.sh also supports the published standalone runner locally.
 #
 # (The separate ci.yml covers lint / type-check / unit + conformance tests / docs.
 # Those drive the pure-Python in-process Client; this workflow is the only place
-# the worker is exercised through the real published DuckDB extension over the wire.)
+# the worker is exercised through the real DuckDB extension over the wire.)
 name: Integration suite
 
 on:
@@ -19,20 +18,16 @@ on:
 
 permissions:
   contents: read
+  actions: read
 
 concurrency:
   group: integration-${{ github.ref }}
   cancel-in-progress: true
 
 env:
-  # The Query-farm/vgi ref whose test/sql/integration suite we run. For now this
-  # tracks the latest `main` (resolved at checkout time) so the Python port is
-  # always validated against the newest C++ test suite. Pin to a specific commit
-  # SHA here if/when reproducible, deliberate bumps are wanted (see ci/README.md).
-  VGI_REF: main
-  # The Haybarn release providing the prebuilt haybarn-unittest binary. Must be
-  # ABI-compatible with the community-published vgi extension.
-  HAYBARN_RELEASE: haybarn-v1.5.5-rc1
+  # The test suite and native Actions artifact must come from this exact ref.
+  # The downloader verifies its provenance against the checked-out commit.
+  VGI_REF: 3a2249f815ff6e2a8b5289a3e6284326084e3630
   # CI is not a user: its ATTACHes should not land in Query.Farm's adoption data.
   # The vgi extension fires a telemetry event per successful ATTACH, and this
   # suite performs several hundred of them per lane, so opting out keeps the
@@ -81,18 +76,13 @@ jobs:
       - name: Install dependencies (vgi + fixtures + all extras)
         run: uv sync --all-extras --python 3.14
 
-      - name: Download haybarn-unittest
+      - name: Download the matching native integration runner
         run: |
-          gh release download "$HAYBARN_RELEASE" \
-            --repo Query-farm-haybarn/haybarn \
-            --pattern 'haybarn_unittest-linux-amd64.zip' \
-            --output /tmp/haybarn-unittest.zip --clobber
-          unzip -o -q /tmp/haybarn-unittest.zip -d /tmp/haybarn-unittest
-          UNITTEST=$(find /tmp/haybarn-unittest -name 'haybarn-unittest' -type f | head -1)
-          chmod +x "$UNITTEST"
-          echo "HAYBARN_UNITTEST=$UNITTEST" >> "$GITHUB_ENV"
+          python3 "$VGI_SRC/scripts/ci_fetch_native_runner.py" \
+            --source "$VGI_SRC" --output "$RUNNER_TEMP/vgi-native"
         env:
           GH_TOKEN: ${{ github.token }}
+          VGI_SRC: ${{ github.workspace }}/vgi-upstream
 
       - name: Run integration suite (${{ matrix.lane }})
         run: ci/run-integration.sh

ci/README.md

@@ -4,19 +4,18 @@
 runs the canonical [Query-farm/vgi](https://github.com/Query-farm/vgi)
 integration sqllogictest suite against this repo's Python example worker on
 every push / PR. The same `.test` files run against the Python, Java, and Go
-ports, so a green run here is real wire-compatibility evidence — it exercises
-the worker through the *published* DuckDB extension, not a mock, and not the
-pure-Python in-process `Client` that the conformance tests in `ci.yml` use.
+ports, so a green run here verifies the worker through the native DuckDB
+extension over the wire. The native runner and test suite come from the same
+VGI commit. Public package releases remain separate from this development check.
 
 (The separate [`ci.yml`](../.github/workflows/ci.yml) covers lint / mypy / unit
 + conformance tests / docs / LocalStack S3 offload.)
 
 ## How it works (no C++ build)
 
-Rather than building the vgi DuckDB extension from source (which needs the
-Haybarn vcpkg pipeline), CI drives a **prebuilt** standalone `haybarn-unittest`
-(the DuckDB/Haybarn sqllogictest runner, published in Haybarn's releases) and
-installs the **signed** vgi extension from the Haybarn community channel:
+CI downloads the native integration runner built by Query-farm/vgi Actions.
+It links vgi, httpfs, json and parquet, so the worker's wire contract is checked
+against the matching client without waiting for a community extension release.
 
 1. **Install the workers** — `uv sync --all-extras` installs the `vgi-fixtures`
    workspace member, which puts the fixture console scripts
@@ -27,17 +26,27 @@ installs the **signed** vgi extension from the Haybarn community channel:
    `--http`, and the launcher's `--unix`, so one install covers every lane.
 2. **Checkout the test suite** — `Query-farm/vgi` at a pinned commit; its
    `test/sql/integration/*.test` files are the suite.
-3. **Download the runner** — `haybarn_unittest-linux-amd64.zip` from the pinned
-   Haybarn release.
-4. **Preprocess** — the standalone runner links none of the extensions the tests
-   gate on, so [`preprocess-require.awk`](preprocess-require.awk) rewrites each
-   `require <ext>` into an explicit signed `INSTALL <ext> FROM {community,core};
-   LOAD <ext>;`. `require-env` and everything else pass through.
-5. **Run** — [`run-integration.sh`](run-integration.sh) stages the preprocessed
-   tree, wires the `VGI_*_WORKER` env vars at the `.venv/bin` scripts for the
-   selected transport, `FORCE INSTALL`s the vgi extension (so the run uses what
-   users can install today), then runs the suite in a single `haybarn-unittest`
-   invocation. Any failed assertion exits non-zero and fails the job.
+3. **Download the runner** — the pinned checkout's `ci_fetch_native_runner.py`
+   retrieves its Actions artifact and checks its provenance against that exact
+   source commit. It exports `HAYBARN_UNITTEST`, `HAYBARN_CLI` and
+   `RUNNER_LINKS_EXTENSIONS=1` for the next step.
+4. **Stage** — [`run-integration.sh`](run-integration.sh) copies the selected
+   tests unchanged when extensions are linked into the runner. Their normal
+   `require` gates load linked extensions; unavailable optional extensions keep
+   their normal skip behavior.
+5. **Run** — the harness wires the `VGI_*_WORKER` environment variables to the
+   `.venv/bin` scripts for the selected transport and executes the suite, with
+   `simple_writable` in its own process. Native mode passes `--test-dir` so the
+   runner discovers the staged tree instead of its compiled source directory.
+   Failed assertions, empty runs, fatal signals and dead shared HTTP workers
+   fail the job. The no-error-skip configuration keeps HTTP errors from silently
+   skipping tests.
+
+For local checks against the published extension, leave
+`RUNNER_LINKS_EXTENSIONS` unset or set it to `0`. This preserves the standalone
+runner path: [`preprocess-require.awk`](preprocess-require.awk) converts
+`require` gates to signed `INSTALL` and `LOAD` statements, and the harness
+warms the community/core extension cache before running.
 
 The env wiring mirrors the vgi repo's Makefile `test_subprocess` / `test_shm` /
 `test_launcher` / `test_http` targets — the canonical local way to run this
@@ -77,9 +86,9 @@ matrix; `stdio` remains available for local compatibility testing:
   `test_launcher`), so those tests skip here; run the local stdio lane to cover
   them. One deliberate divergence from `test_launcher`: see
   `filter_echo_partitioned.test` below.
-- **`http`** — the whole suite over the stateless HTTP transport. Staging
-  injects `LOAD httpfs` before each worker ATTACH (the prebuilt binary doesn't
-  statically link httpfs).
+- **`http`** — the whole suite over the stateless HTTP transport. The native
+  runner links httpfs; standalone mode injects `LOAD httpfs` before worker
+  attachments.
 
 The `simple_writable/*.test` write-path tests (INSERT/UPDATE/DELETE/RETURNING)
 run on **every** lane against `VGI_SIMPLE_WRITABLE_WORKER`
@@ -147,36 +156,32 @@ even though the fixture (`vgi-fixture-bad-enum-worker`) is installed.
 ```bash
 uv sync --all-extras
 VGI_SRC=~/Development/vgi \
-HAYBARN_UNITTEST=/path/to/haybarn-unittest \
+HAYBARN_UNITTEST=~/Development/vgi/build/release/test/unittest \
+RUNNER_LINKS_EXTENSIONS=1 \
 TRANSPORT=stdio \
   ci/run-integration.sh
 ```
 
-Download `haybarn-unittest` for your platform from the pinned Haybarn release:
+To check the published extension instead, download `haybarn-unittest` for your
+platform from a compatible Haybarn release and leave `RUNNER_LINKS_EXTENSIONS=0`:
 
 ```bash
 gh release download "$HAYBARN_RELEASE" --repo Query-farm-haybarn/haybarn \
   --pattern 'haybarn_unittest-*.zip'
 ```
 
-For day-to-day local work against a hand-built C++ extension, prefer the vgi
-repo's own targets (`make test_subprocess`, `make test_http`, …) /
-`scripts/run_all_tests.sh` as described in `CLAUDE.md`; this harness exists for
-CI, where building the extension from source is impractical.
+The vgi repo's own targets (`make test_subprocess`, `make test_http`, …) and
+`scripts/run_all_tests.sh` remain available as described in `CLAUDE.md`.
 
 ## Version pins (and their coupling)
 
-Two pins live in the workflow's `env:` block:
+The workflow's `VGI_REF` pins the test suite and the native Actions artifact:
 
 | Pin | What | Why |
 |-----|------|-----|
-| `VGI_REF` | the `Query-farm/vgi` ref supplying the `.test` files | **for now tracks `main`** (latest at checkout time) so the port is always validated against the newest suite; switch to a commit SHA for reproducible, deliberate bumps |
-| `HAYBARN_RELEASE` | the Haybarn release supplying `haybarn-unittest` | must be ABI-compatible with the community vgi extension |
-
-**The coupling to know about:** the vgi extension is pulled live from the
-community channel (`FORCE INSTALL vgi FROM community`), which always serves the
-*currently published* build — it is not version-pinned here. So CI verifies the
-worker against what users can install today. If `VGI_REF` points at a commit
-whose tests exercise a protocol feature the published extension doesn't yet ship
-(or vice-versa), that test can fail or skip — bump `VGI_REF` deliberately and
-re-validate against the current community extension.
+| `VGI_REF` | the `Query-farm/vgi` commit supplying the tests and native runner | the downloader requires an artifact built from that exact commit |
+
+The optional standalone mode still installs the currently published community
+extension. Use a compatible test suite and worker when checking that release;
+an unreleased minor wire revision requires the matching native development
+runner. Changing the CI pin does not publish an extension or an SDK package.

ci/run-integration.sh

@@ -2,8 +2,8 @@
 # Copyright 2025, 2026 Query Farm LLC - https://query.farm
 #
 # Run the canonical Query-farm/vgi integration sqllogictest suite against the
-# Python example worker, using a prebuilt standalone `haybarn-unittest` and the
-# signed community vgi extension — no C++ build from source. See ci/README.md.
+# Python example worker, using either a native runner with linked extensions or
+# standalone `haybarn-unittest` and the signed community extension. See ci/README.md.
 #
 # Ported from vgi-go's harness. The Python differences:
 #   * The "worker binaries" are console scripts installed into the project venv
@@ -29,6 +29,8 @@
 # Optional:
 #   BIN_DIR           dir holding the fixture console scripts (default: $REPO/.venv/bin)
 #   TRANSPORT         stdio | shm | launch | http   (default: launch)
+#   RUNNER_LINKS_EXTENSIONS  1 for a native runner linking vgi/httpfs/json/parquet;
+#                     0 for the published standalone runner (default)
 #   VGI_RPC_SHM_SIZE_BYTES  shm side-channel segment size (the shm lane)
 #   STAGE             scratch dir for the preprocessed test tree (default: mktemp)
 set -euo pipefail
@@ -41,6 +43,7 @@ REPO="$(cd "$HERE/.." && pwd)"
 BIN_DIR="${BIN_DIR:-$REPO/.venv/bin}"
 STAGE="${STAGE:-$(mktemp -d)}"
 TRANSPORT="${TRANSPORT:-launch}"
+RUNNER_LINKS_EXTENSIONS="${RUNNER_LINKS_EXTENSIONS:-0}"
 INTEGRATION="$VGI_SRC/test/sql/integration"
 [ -d "$INTEGRATION" ] || { echo "::error::no test/sql/integration under VGI_SRC=$VGI_SRC"; exit 1; }
 
@@ -118,7 +121,11 @@ mkdir -p "$STAGE/test/sql/integration"
        -not -name 'nested_type_combinations.test' \
        "${EXTRA_SKIP[@]}" | while read -r f; do
     mkdir -p "$STAGE/test/sql/integration/$(dirname "$f")"
-    awk -v http="$AWK_HTTP" -f "$HERE/preprocess-require.awk" "$f" > "$STAGE/test/sql/integration/$f"
+    if [ "$RUNNER_LINKS_EXTENSIONS" = "1" ]; then
+      cp "$f" "$STAGE/test/sql/integration/$f"
+    else
+      awk -v http="$AWK_HTTP" -f "$HERE/preprocess-require.awk" "$f" > "$STAGE/test/sql/integration/$f"
+    fi
   done )
 
 # Some integration cases package an executable from the upstream support tree.
@@ -310,6 +317,9 @@ esac
 
 cd "$STAGE"
 
+if [ "$RUNNER_LINKS_EXTENSIONS" = "1" ]; then
+  echo "Using extensions linked into the matching native runner."
+else
 echo "Warming the extension cache (vgi from community, deps from core) ..."
 mkdir -p "$STAGE/test"
 # FORCE INSTALL vgi re-downloads the currently-published community build,
@@ -335,6 +345,7 @@ INSTALL spatial FROM core;
 EOF
 "$HAYBARN_UNITTEST" "test/_warm.test" >/dev/null 2>&1 || echo "::warning::extension warm step did not fully succeed"
 rm -f "$STAGE/test/_warm.test"
+fi
 
 # DuckDB's sqllogictest runner, given no --test-config, turns any error whose
 # text contains "HTTP" or "Unable to connect" into a SKIP and exits 0. Over the
@@ -343,6 +354,14 @@ rm -f "$STAGE/test/_warm.test"
 TEST_CONFIG="$VGI_SRC/test/configs/no_error_skip.json"
 [ -f "$TEST_CONFIG" ] || { echo "::error::missing $TEST_CONFIG (VGI_REF predates it?)"; exit 1; }
 
+# Native unittest defaults to its compiled source directory even after `cd`.
+# Select the staged tree explicitly so discovery respects the exclusions above
+# and spawned workers resolve relative fixture paths from the same directory.
+RUNNER_ARGS=()
+if [ "$RUNNER_LINKS_EXTENSIONS" = "1" ]; then
+  RUNNER_ARGS=(--test-dir "$STAGE")
+fi
+
 # Run the lane, streaming the native sqllogictest report (a progress line per
 # file + the final "All tests passed (.. N assertions ..)" summary). Out-of-scope
 # tests were dropped at staging; any failed assertion exits non-zero.
@@ -373,7 +392,7 @@ run_unittest() {
   # `set +e` rather than `|| true`: the latter runs before PIPESTATUS is read and
   # overwrites it with true's 0, silently swallowing every real test failure.
   set +e
-  "$HAYBARN_UNITTEST" --test-config "$TEST_CONFIG" "$@" 2>&1 | tee "$log"
+  "$HAYBARN_UNITTEST" "${RUNNER_ARGS[@]}" --test-config "$TEST_CONFIG" "$@" 2>&1 | tee "$log"
   rc="${PIPESTATUS[0]}"
   set -e
   if grep -q 'due to a fatal error condition' "$log"; then
@@ -382,6 +401,10 @@ run_unittest() {
          "otherwise have passed. This is invisible to the exit code by construction."
     rc=1
   fi
+  if grep -Eq 'No tests ran|No test cases matched' "$log"; then
+    echo "::error::the runner did not execute the selected tests"
+    rc=1
+  fi
   rm -f "$log"
   return "$rc"
 }