Skip to content

Commit f17acaa

Browse files
rustyconoverclaude
rustyconover
and
claude
committed
ci: run the vgi integration sqllogictest suite against the Python worker
Adds a dedicated integration workflow mirroring vgi-go / vgi-java: it drives a prebuilt standalone haybarn-unittest and the signed community vgi extension (no C++ build from source) to run the canonical Query-farm/vgi test/sql/integration suite against the Python example worker over a matrix of transport lanes (stdio, shm, launch, http). This is the only place the worker is exercised through the real published DuckDB extension over the wire; ci.yml's conformance tests drive the pure-Python in-process Client. VGI_REF tracks Query-farm/vgi main for now (latest at checkout time); HAYBARN_RELEASE pins haybarn-v1.5.4-rc1. - ci/run-integration.sh stages + preprocesses the suite, wires the .venv/bin/vgi-fixture-* workers per lane - ci/preprocess-require.awk rewrites `require <ext>` -> signed INSTALL/LOAD - ci/README.md design, lanes, exclusions, version-pin coupling - .github/workflows/integration.yml the 4-lane matrix workflow Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 456eda4 commit f17acaa

4 files changed

Lines changed: 522 additions & 0 deletions

File tree

‎.github/workflows/integration.yml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Runs the canonical Query-farm/vgi integration sqllogictest suite against the
2+
# Python example worker on every push / PR. Rather than building the C++ extension
3+
# from source, it drives a prebuilt standalone `haybarn-unittest` and installs the
4+
# SIGNED vgi extension from the Haybarn community channel + its deps from core.
5+
# The .test files come from a pinned Query-farm/vgi checkout; ci/run-integration.sh
6+
# + ci/preprocess-require.awk adapt them for the standalone runner. See ci/README.md.
7+
#
8+
# (The separate ci.yml covers lint / type-check / unit + conformance tests / docs.
9+
# Those drive the pure-Python in-process Client; this workflow is the only place
10+
# the worker is exercised through the real published DuckDB extension over the wire.)
11+
name: Integration suite
12+
13+
on:
14+
push:
15+
branches: [main]
16+
pull_request:
17+
branches: [main]
18+
workflow_dispatch:
19+
20+
permissions:
21+
contents: read
22+
23+
concurrency:
24+
group: integration-${{ github.ref }}
25+
cancel-in-progress: true
26+
27+
env:
28+
# The Query-farm/vgi ref whose test/sql/integration suite we run. For now this
29+
# tracks the latest `main` (resolved at checkout time) so the Python port is
30+
# always validated against the newest C++ test suite. Pin to a specific commit
31+
# SHA here if/when reproducible, deliberate bumps are wanted (see ci/README.md).
32+
VGI_REF: main
33+
# The Haybarn release providing the prebuilt haybarn-unittest binary. Must be
34+
# ABI-compatible with the community-published vgi extension.
35+
HAYBARN_RELEASE: haybarn-v1.5.4-rc1
36+
37+
jobs:
38+
integration:
39+
runs-on: ubuntu-latest
40+
timeout-minutes: 45
41+
strategy:
42+
fail-fast: false
43+
matrix:
44+
include:
45+
# stdio: the subprocess transport (whole suite — the primary lane).
46+
# shm: stdio + the POSIX shared-memory side channel.
47+
# launch: the AF_UNIX launcher-only tests.
48+
# http: the whole suite over the stateless HTTP transport.
49+
- { lane: stdio }
50+
- { lane: shm, shm_bytes: '67108864' }
51+
- { lane: launch }
52+
- { lane: http }
53+
name: integration (${{ matrix.lane }})
54+
steps:
55+
- name: Checkout vgi-python
56+
uses: actions/checkout@v5
57+
58+
- name: Checkout pinned vgi test suite
59+
uses: actions/checkout@v5
60+
with:
61+
repository: Query-farm/vgi
62+
ref: ${{ env.VGI_REF }}
63+
path: vgi-upstream
64+
token: ${{ secrets.GITHUB_TOKEN }}
65+
66+
- name: Install uv
67+
uses: astral-sh/setup-uv@v8.2.0
68+
69+
- name: Set up Python 3.13
70+
run: uv python install 3.13
71+
72+
- name: Install dependencies (vgi + fixtures + all extras)
73+
run: uv sync --all-extras --python 3.13
74+
75+
- name: Download haybarn-unittest
76+
run: |
77+
gh release download "$HAYBARN_RELEASE" \
78+
--repo Query-farm-haybarn/haybarn \
79+
--pattern 'haybarn_unittest-linux-amd64.zip' \
80+
--output /tmp/haybarn-unittest.zip --clobber
81+
unzip -o -q /tmp/haybarn-unittest.zip -d /tmp/haybarn-unittest
82+
UNITTEST=$(find /tmp/haybarn-unittest -name 'haybarn-unittest' -type f | head -1)
83+
chmod +x "$UNITTEST"
84+
echo "HAYBARN_UNITTEST=$UNITTEST" >> "$GITHUB_ENV"
85+
env:
86+
GH_TOKEN: ${{ github.token }}
87+
88+
- name: Run integration suite (${{ matrix.lane }})
89+
run: ci/run-integration.sh
90+
env:
91+
VGI_SRC: ${{ github.workspace }}/vgi-upstream
92+
TRANSPORT: ${{ matrix.lane }}
93+
# Empty on every lane except shm; run-integration.sh drops an empty value.
94+
VGI_RPC_SHM_SIZE_BYTES: ${{ matrix.shm_bytes || '' }}

‎ci/README.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# CI: the vgi integration suite
2+
3+
[`.github/workflows/integration.yml`](../.github/workflows/integration.yml)
4+
runs the canonical [Query-farm/vgi](https://github.com/Query-farm/vgi)
5+
integration sqllogictest suite against this repo's Python example worker on
6+
every push / PR. The same `.test` files run against the Python, Java, and Go
7+
ports, so a green run here is real wire-compatibility evidence — it exercises
8+
the worker through the *published* DuckDB extension, not a mock, and not the
9+
pure-Python in-process `Client` that the conformance tests in `ci.yml` use.
10+
11+
(The separate [`ci.yml`](../.github/workflows/ci.yml) covers lint / mypy / unit
12+
+ conformance tests / docs / LocalStack S3 offload.)
13+
14+
## How it works (no C++ build)
15+
16+
Rather than building the vgi DuckDB extension from source (which needs the
17+
Haybarn vcpkg pipeline), CI drives a **prebuilt** standalone `haybarn-unittest`
18+
(the DuckDB/Haybarn sqllogictest runner, published in Haybarn's releases) and
19+
installs the **signed** vgi extension from the Haybarn community channel:
20+
21+
1. **Install the workers** — `uv sync --all-extras` installs the `vgi-fixtures`
22+
workspace member, which puts the fixture console scripts
23+
(`vgi-fixture-worker`, `-versioned-worker`, `-versioned-tables-worker`,
24+
`-attach-options-worker`, `-bad-protocol-worker`, `-simple-writable-worker`)
25+
into `.venv/bin`. Every per-catalog worker is the same Python program with a
26+
different entrypoint; the base `WorkerClass.main()` understands stdin/stdout,
27+
`--http`, and the launcher's `--unix`, so one install covers every lane.
28+
2. **Checkout the test suite** — `Query-farm/vgi` at a pinned commit; its
29+
`test/sql/integration/*.test` files are the suite.
30+
3. **Download the runner** — `haybarn_unittest-linux-amd64.zip` from the pinned
31+
Haybarn release.
32+
4. **Preprocess** — the standalone runner links none of the extensions the tests
33+
gate on, so [`preprocess-require.awk`](preprocess-require.awk) rewrites each
34+
`require <ext>` into an explicit signed `INSTALL <ext> FROM {community,core};
35+
LOAD <ext>;`. `require-env` and everything else pass through.
36+
5. **Run** — [`run-integration.sh`](run-integration.sh) stages the preprocessed
37+
tree, wires the `VGI_*_WORKER` env vars at the `.venv/bin` scripts for the
38+
selected transport, `FORCE INSTALL`s the vgi extension (so the run uses what
39+
users can install today), then runs the suite in a single `haybarn-unittest`
40+
invocation. Any failed assertion exits non-zero and fails the job.
41+
42+
The env wiring mirrors the vgi repo's Makefile `test_subprocess` / `test_shm` /
43+
`test_launcher` / `test_http` targets — the canonical local way to run this
44+
suite against the Python worker (see the project `CLAUDE.md`).
45+
46+
## Transport lanes
47+
48+
`run-integration.sh` honours `TRANSPORT=stdio|shm|launch|http` (the workflow
49+
runs them as a matrix):
50+
51+
- **`stdio`** — the subprocess transport (the primary lane); the whole suite.
52+
Also boots the versioned and versioned-tables workers as background HTTP
53+
servers (`VGI_VERSIONED_HTTP_WORKER` / `VGI_VERSIONED_TABLES_HTTP_WORKER`) so
54+
the `attach/versioned_tables_*_http` and `versioning_http` tests run.
55+
- **`shm`** — `stdio` plus the POSIX shared-memory side channel
56+
(`VGI_RPC_SHM_SIZE_BYTES`); the whole suite. vgi-rpc-python attaches the
57+
segment transparently, so the env var alone flips the transport.
58+
- **`launch`** — the AF_UNIX launcher transport; the launcher-only tests
59+
(`launcher/*`), which the other lanes skip via `require-env`.
60+
- **`http`** — the whole suite over the stateless HTTP transport. Staging
61+
injects `LOAD httpfs` before each worker ATTACH (the prebuilt binary doesn't
62+
statically link httpfs).
63+
64+
The `simple_writable/*.test` write-path tests (INSERT/UPDATE/DELETE/RETURNING)
65+
run on the subprocess lanes against `VGI_SIMPLE_WRITABLE_WORKER`
66+
(`vgi-fixture-simple-writable-worker`), in their own `haybarn-unittest`
67+
invocation so their warm pooled connections don't perturb the immediately
68+
following crash-recovery test. They self-skip on the http lane
69+
(skip-on-error `'HTTP'`).
70+
71+
### Excluded tests
72+
73+
Out of scope on **every** lane:
74+
75+
- `writable/` — the opt-in *generic* writable catalog
76+
(`VGI_WORKER_ENABLE_WRITABLE`), not modelled by a cross-language fixture (the
77+
vgi Makefile excludes it too). `simple_writable/` is *not* excluded.
78+
- `nested_type_combinations.test` — segfaults the prebuilt standalone runner (a
79+
property of that C++ build, not the worker, which passes it against a
80+
locally-built unittest).
81+
- `bool_in_union.test` — a pre-existing, arch-dependent union-bool bug whose
82+
pinned expected output matches arm64 but not amd64 (CI is amd64).
83+
84+
Dropped on the **http** lane only:
85+
86+
- `projection_pushdown_repro.test` — one POST per two rows; transport-agnostic,
87+
fully covered by stdio.
88+
- `dynamic_filter.test` — Top-N + dynamic-filter continuation terminates early
89+
over http in the prebuilt binary (a property of that C++ build).
90+
- `partitioned_sequence.test` and `table_in_out/buffer_input/sizes.test` —
91+
partition-local / input-buffering state that the stateless HTTP transport does
92+
not preserve across exchanges (the known Python HTTP limitations documented in
93+
`CLAUDE.md`). `buffer_input/scale.test_slow` is a `.test_slow` file, which the
94+
harness never stages (it only finds `*.test`).
95+
96+
The HTTP-attach / bearer-auth / dynamic-code / schema-reconcile-only tests skip
97+
via their `require-env` gates when we don't set the corresponding worker.
98+
99+
## Run it locally
100+
101+
```bash
102+
uv sync --all-extras
103+
VGI_SRC=~/Development/vgi \
104+
HAYBARN_UNITTEST=/path/to/haybarn-unittest \
105+
TRANSPORT=stdio \
106+
ci/run-integration.sh
107+
```
108+
109+
Download `haybarn-unittest` for your platform from the pinned Haybarn release:
110+
111+
```bash
112+
gh release download "$HAYBARN_RELEASE" --repo Query-farm-haybarn/haybarn \
113+
--pattern 'haybarn_unittest-*.zip'
114+
```
115+
116+
For day-to-day local work against a hand-built C++ extension, prefer the vgi
117+
repo's own targets (`make test_subprocess`, `make test_http`, …) /
118+
`scripts/run_all_tests.sh` as described in `CLAUDE.md`; this harness exists for
119+
CI, where building the extension from source is impractical.
120+
121+
## Version pins (and their coupling)
122+
123+
Two pins live in the workflow's `env:` block:
124+
125+
| Pin | What | Why |
126+
|-----|------|-----|
127+
| `VGI_REF` | the `Query-farm/vgi` ref supplying the `.test` files | **for now tracks `main`** (latest at checkout time) so the port is always validated against the newest suite; switch to a commit SHA for reproducible, deliberate bumps |
128+
| `HAYBARN_RELEASE` | the Haybarn release supplying `haybarn-unittest` | must be ABI-compatible with the community vgi extension |
129+
130+
**The coupling to know about:** the vgi extension is pulled live from the
131+
community channel (`FORCE INSTALL vgi FROM community`), which always serves the
132+
*currently published* build — it is not version-pinned here. So CI verifies the
133+
worker against what users can install today. If `VGI_REF` points at a commit
134+
whose tests exercise a protocol feature the published extension doesn't yet ship
135+
(or vice-versa), that test can fail or skip — bump `VGI_REF` deliberately and
136+
re-validate against the current community extension.

‎ci/preprocess-require.awk‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
2+
#
3+
# Rewrite each `require <ext>` gate in an upstream vgi sqllogictest into an
4+
# explicit signed INSTALL+LOAD, so the prebuilt standalone `haybarn-unittest`
5+
# (which links none of these extensions) can run the suite. The vgi extension
6+
# comes from the signed community channel; httpfs/json/parquet/spatial from the
7+
# signed core channel. `require-env` and every other directive pass through
8+
# untouched. See ci/README.md.
9+
#
10+
# With `-v http=1`, also inject a signed `INSTALL httpfs FROM core; LOAD httpfs;`
11+
# before the first worker ATTACH (keyed off `require vgi` or `require-env
12+
# VGI_TEST_WORKER`). The prebuilt `haybarn-unittest` does not statically link
13+
# httpfs, so `ATTACH ... (TYPE vgi, LOCATION 'http://...')` fails with a binder
14+
# error unless httpfs is loaded into the connection first. Upstream's own
15+
# `unittest` build links httpfs, so its `make test_http` needs no such step.
16+
BEGIN { injected = 0 }
17+
function inject_httpfs() {
18+
if (http != 1 || injected) return
19+
print "";
20+
print "statement ok"; print "INSTALL httpfs FROM core;"; print "";
21+
print "statement ok"; print "LOAD httpfs;";
22+
injected = 1
23+
}
24+
/^require[ \t]+vgi[ \t]*$/ {
25+
print "statement ok"; print "INSTALL vgi FROM community;"; print "";
26+
print "statement ok"; print "LOAD vgi;";
27+
inject_httpfs();
28+
next
29+
}
30+
/^require[ \t]+(httpfs|json|parquet|spatial)[ \t]*$/ {
31+
ext = $2
32+
print "statement ok"; print "INSTALL " ext " FROM core;"; print "";
33+
print "statement ok"; print "LOAD " ext ";"; next
34+
}
35+
/^require-env[ \t]+VGI_TEST_WORKER[ \t]*$/ {
36+
print
37+
inject_httpfs();
38+
next
39+
}
40+
{ print }

0 commit comments

Comments
 (0)