You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a4d5b61
Browse filesBrowse the repository at this point in the historyBrowse files
build(daemon): link ducklake into the daemon from DuckDB's pinned config
daemon-compile could not pass on a DuckDB 2.0 development build: the
default local-ducklake mode INSTALLs ducklake at startup, no ducklake is
published for a dev build, and the 2.0 alpha's binary null-derefs on load
against the branch tip. No version string fixes that, because the problem
is the coupling itself -- the daemon only worked when a published extension
build happened to match its ABI.
So the daemon stops downloading the extension its default mode needs:
- DUCKDB_OTLP_BUILT_IN_EXTENSIONS (CMake list) links extensions into
DuckDB. extension_config.cmake takes each from DuckDB's own pinned config
(duckdb/.github/config/extensions/<name>.cmake: the commit and patches
DuckDB builds against this source), so the ABI matches by construction.
- `make server-release` -- the path Docker and both daemon CI jobs take --
sets it to ducklake and first builds CRoaring, ducklake's one dependency,
into build/deps, since those builds run without vcpkg. v4.7.2 rather
than vcpkg's 4.4.2, whose package config CMake 4 rejects. The extension
distribution builds are unchanged.
- Core(name) reports BUILT_IN when LinkedExtensionRegistry says the binary
links it, so no INSTALL/LOAD is emitted and the banner says "built in".
Asking the binary rather than a build flag means the two cannot drift.
- The image stops priming ducklake.
quack stays a runtime install: on 2.0 it draws its tokens from the
writable crypto module only httpfs (OpenSSL) provides, so building it in
means building httpfs in, against the image's no-libssl design.
test_init_sql_adds_a_parquet_destination_next_to_the_ducklake_catalog read
the daemon's catalog back with the host's duckdb==1.5.6, which cannot open
a 2.0-format database (storage version 69). It now reads it back through
the binary under test (`duckdb-otlp query`), which is still a real DuckLake
read and no longer depends on the host's DuckDB version.
Verified: local-ducklake daemon with an empty extension cache ingests,
seals and reads back with zero downloads; pytest test/server 247/247 both
on macOS and against the Docker-built linux/arm64 daemon-export binary;
make test 23/23, seal harness 204, arrow harness 6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: AGENTS.md
+13-2Lines changed: 13 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,6 +22,17 @@ This file provides guidance to coding agents when working with code in this repo
22
22
`Copy Function "parquet" is not in the catalog`. Separately, loading an extension
23
23
config no longer links it into the shell/`unittest`: `extension_config.cmake` asks for that with
24
24
`duckdb_extension_statically_link(otlp)`.
25
+
-**Built-in runtime extensions (`DUCKDB_OTLP_BUILT_IN_EXTENSIONS`).** A CMake list of extensions
26
+
linked into DuckDB instead of `INSTALL`ed by the daemon at startup. `extension_config.cmake`
27
+
takes each from DuckDB's own pinned config (`duckdb/.github/config/extensions/<name>.cmake`,
28
+
the commit and patches DuckDB builds against this source), so its ABI matches by construction.
29
+
`make server-release` (Docker, both daemon CI jobs) sets `DAEMON_BUILT_IN_EXTENSIONS ?= ducklake`
30
+
and first builds CRoaring, ducklake's one dependency, into `build/deps/croaring`, because the
31
+
daemon builds run without vcpkg (v4.7.2, not vcpkg's 4.4.2, whose package config CMake 4
32
+
rejects). The extension distribution builds leave the list empty. `quack` is deliberately not
33
+
in it: on DuckDB 2.0 it draws tokens from the writable crypto module only httpfs (OpenSSL)
34
+
provides, so building it in means building httpfs in, against the runtime image's no-libssl
35
+
design. Linked extensions load in every command, not only `serve`.
25
36
-**Windows (`windows_amd64`, MSVC) builds the extension only.**`CMakeLists.txt` gates the
26
37
native daemon (`duckdb_otlp_server`) and the C++ harnesses (`otlp_seal_harness`,
27
38
`otlp_arrow_harness`) on `NOT WIN32`, because `make release` builds the default ALL target
@@ -89,7 +100,7 @@ CI in `.github/workflows/MainDistributionPipeline.yml`: `daemon-compile` is a ch
89
100
90
101
On a `v*` tag, `daemon-release` attaches those four binaries to the GitHub release as `duckdb-otlp-<tag>-<platform>.tar.gz` plus `SHA256SUMS`. It recompiles nothing (it downloads the artifacts the build jobs uploaded) and gates on `docker-smoke`, so a binary that failed the e2e never reaches a release. The Darwin artifacts are deliberately named `duckdb-otlp-darwin-*` rather than `duckdb-otlp-server-*`: the latter is the glob `docker-image` downloads, which must only match the Linux binaries. `daemon-release` creates the release when the tag was pushed without one, which in turn triggers `pages.yml` to attach the extension repository.
91
102
92
-
Extension offline-cache coupling: the `deps` stage pre-`INSTALL`s ducklake/iceberg/httpfs/aws/postgres/quack and community gcs with a throwaway DuckDB CLI into `HOME=/duckdb-home` (copied into the final image, owned by the nonroot user; the CLI itself is not copied). The daemon reuses that cache only because it runs with the same `HOME` and is pinned to the same `DUCKDB_VERSION`. If those diverge, the daemon's startup `INSTALL`/`LOAD` re-downloads (or fails when offline). The `otlp` extension is statically embedded in the daemon and is intentionally not installed. Bind-mounted `/data` must be writable by uid 65532; named/anonymous volumes inherit the image's nonroot ownership automatically.
103
+
Extension offline-cache coupling: the `deps` stage pre-`INSTALL`s iceberg/httpfs/aws/postgres/quack and community gcs with a throwaway DuckDB CLI into `HOME=/duckdb-home` (copied into the final image, owned by the nonroot user; the CLI itself is not copied). The daemon reuses that cache only because it runs with the same `HOME` and is pinned to the same `DUCKDB_VERSION`. If those diverge, the daemon's startup `INSTALL`/`LOAD` re-downloads (or fails when offline). The `otlp` extension is statically embedded in the daemon and is intentionally not installed, and so is `ducklake` (see *Built-in runtime extensions* below), which is what lets the default `local-ducklake` mode run on a DuckDB build that has no published extensions — a 2.0 development build, for one. Bind-mounted `/data` must be writable by uid 65532; named/anonymous volumes inherit the image's nonroot ownership automatically.
93
104
94
105
### Building for WebAssembly
95
106
```bash
@@ -290,7 +301,7 @@ Prefer one canonical page per topic and link to it instead of duplicating exampl
290
301
-**Standard OTLP env vars**: `OTEL_EXPORTER_OTLP_PROTOCOL` (only when no port is explicit), and `OTEL_EXPORTER_OTLP_HEADERS` (`Authorization=Bearer`) are honored. `OTEL_EXPORTER_OTLP_ENDPOINT` is deliberately **not read** (it usually points at the user's real collector, so reading it as a bind address is a footgun); the per-signal endpoint and TLS variables are **rejected at startup** rather than ignored, because silently dropping them would misrepresent the deployment.
291
302
-**AWS credentials use a multi-source chain**: `BuildCredentialChainSecret` emits `CHAIN 'env;config;instance'` when no `AWS_PROFILE` is set (and `CHAIN config` + `PROFILE` when one is). A single-source chain is a trap in both directions — `CHAIN env` cannot see an EC2/ECS instance role, `CHAIN instance` cannot see the standard AWS variables — and this mode set has been bitten by each: `aws-ducklake` hardcoded `CHAIN instance`, and consolidating it onto the shared builder briefly swapped that for `CHAIN env`, taking instance-role credentials away from exactly the deployments that have nothing else. `AwsCredentialsSource` must keep naming the sources the chain actually contains; it previously claimed "environment, then instance role" while the SQL said `env`.
292
303
-**`parquet` locality is `IsRemotePath`, not `IsS3Path`**: `IsS3Path` answers "does this need the S3 credential set", which is a different question from "is this a local directory". Using it as the locality test sent a `gcss://` export path down the local branch — no filesystem extension loaded, empty setup SQL, `validate` reporting success — and the first seal was the first sign. A remote non-S3 path is now recorded as unusable at configuration time. The R2 modes' default DuckLake `DATA_PATH` keeps its trailing slash (`ObjectStorePath` + `"/"`), because DuckLake treats it as a directory prefix and dropping it would repoint an existing deployment at a different prefix than the one in its catalog.
293
-
- **A mode declares its extensions once**: `ServerConfig::mode_extensions` is a list of `{name, ExtensionSource}` (`BUILT_IN` / `CORE` / `COMMUNITY`), and both the `INSTALL`/`LOAD` prelude (`ExtensionSetupSql`, spliced ahead of each mode's SQL) and the startup banner are derived from it. Each mode used to state the same list twice — once as strings for the banner, once as hand-typed SQL — so they could disagree with no symptom beyond a misleading banner. Provenance is the only thing that varies and it decides everything: `BUILT_IN` (otlp, statically embedded) emits nothing, so an `INSTALL otlp` that would fetch a *different* published build is unrepresentable; `COMMUNITY` emits `FROM community`, without which the install resolves nowhere. `test_generated_sql_installs_exactly_what_the_banner_reports` pins the two renderings together per mode. The image's offline extension cache (`docker/duckdb-otlp-server/Dockerfile`) is a third copy that cannot share the declaration — different language, and `deps` must not depend on `builder` — so `test_the_image_primes_every_extension_some_mode_needs` asserts it covers the union instead; without it, a mode gaining an extension silently sends the container to the network at startup, or fails outright when offline. Quack is declared the same way but kept out of `mode_extensions`, because it is loaded by the serve path only and `query`/`export` must not pull a server extension in: `QUACK_EXTENSION` is declared once and consumed by `StartQuackSql` and by `AllExtensions()` (what the banner prints). It is `CORE` — `INSTALL quack FROM community` 404s while a bare `INSTALL quack` resolves, which is also how the image primes it. Its `LOAD` previously carried no `INSTALL`, so `--quack` failed with `Extension "quack" not found. Install it first` on every host except the container, whose image primes the cache.
304
+
- **A mode declares its extensions once**: `ServerConfig::mode_extensions` is a list of `{name, ExtensionSource}` (`BUILT_IN` / `CORE` / `COMMUNITY`), and both the `INSTALL`/`LOAD` prelude (`ExtensionSetupSql`, spliced ahead of each mode's SQL) and the startup banner are derived from it. Each mode used to state the same list twice — once as strings for the banner, once as hand-typed SQL — so they could disagree with no symptom beyond a misleading banner. Provenance is the only thing that varies and it decides everything: `BUILT_IN` (otlp, plus any extension the binary links — `Core(name)` asks `LinkedExtensionRegistry` at runtime rather than a build flag, so the answer cannot drift from the link line) emits nothing, so an `INSTALL otlp` that would fetch a *different* published build is unrepresentable; `COMMUNITY` emits `FROM community`, without which the install resolves nowhere. `test_generated_sql_installs_exactly_what_the_banner_reports` pins the two renderings together per mode. The image's offline extension cache (`docker/duckdb-otlp-server/Dockerfile`) is a third copy that cannot share the declaration — different language, and `deps` must not depend on `builder` — so `test_the_image_primes_every_extension_some_mode_needs` asserts it covers the union instead; without it, a mode gaining an extension silently sends the container to the network at startup, or fails outright when offline. Quack is declared the same way but kept out of `mode_extensions`, because it is loaded by the serve path only and `query`/`export` must not pull a server extension in: `QuackExtension()` is declared once and consumed by `StartQuackSql` and by `AllExtensions()` (what the banner prints); it is a function rather than a constant because the linked-extension registry it consults is filled during static initialization. It is `CORE` — `INSTALL quack FROM community` 404s while a bare `INSTALL quack` resolves, which is also how the image primes it. Its `LOAD` previously carried no `INSTALL`, so `--quack` failed with `Extension "quack" not found. Install it first` on every host except the container, whose image primes the cache.
294
305
- **One name per setting, one error per run**: storage configuration used to spell the same concept differently per mode (five names for an R2 access key, `NEON_PG*` vs `PG*` for the same Postgres catalog, six for a bucket) — about 70 variable names for ~25 concepts. Each mode now reads one canonical name: the cloud's own convention where one exists (`AWS_*`, `PG*` libpq names), `R2_*` for R2, `S3_*` for S3, `DUCKDB_OTLP_*` only for this daemon's own settings. `ConfigProblems` (`server_config.cpp`) accumulates every problem with a mode's configuration — absent values via `Add`, set-but-unusable ones via `Invalid` — and `ConfigureMode` throws once naming all of them, deduplicated by variable. Malformed values are recorded rather than thrown so they do not short-circuit the rest of the pass, which is what made `DUCKLAKE_DATA_PATH=/local/path` report the path on one run and `AWS_REGION` only on the next — configuring `r2-data-catalog` took six runs to satisfy when each error named a single variable. `MODES` is one table of `{name, configure}` so the supported-mode list in the error cannot drift from the set actually handled.
295
306
- **`parquet` mode reads back through views, defined in one place**: the mode has no catalog, so a signal is readable only through a view over `<root>/<table>/**/*.parquet`. `SealParquet` already creates that view after each successful export (it can only be created once files exist, which is why mode setup cannot do it), so serving and reading on one host was already covered. What was broken is narrower than it looks: `ExistingSignalTables` asked `duckdb_tables()` only, so `export` found **zero** signal tables and refused to export data that was sitting there readable — it now unions `duckdb_views()`. `RegisterParquetExportViews` (`commands.cpp`, from `OpenConfiguredDatabase`) covers the remaining gap: a dataset written by another process or host, whose control database has never been sealed into. Both sites build the SELECT from `ParquetDatasetSelect` (`otlp_sql_util.hpp`) so the glob depth and the two load-bearing `read_parquet` options cannot drift — `hive_partitioning=false` because the COPY sets `WRITE_PARTITION_COLUMNS false` (year/month/day live only in the path, and inferring them appends three columns no catalog mode has), `union_by_name=true` so a schema change still reads. Registration is scoped to the signals the caller could read — the resolved `--signal` list for `export`, the signal names appearing in the SQL for `query` — because each probe is a directory listing, and a remote LIST apiece against an `s3://` root. A read-only database cannot hold a view, so that path falls back to session-scoped `TEMP VIEW`s.
296
307
-**`--mode none`**: attaches nothing, installs nothing, sets no catalog — the destination is entirely `--init-sql` plus `--catalog`. Every mode contributes only setup SQL and a catalog name (see `BootSql`), both already expressible that way, but without an opt-out an operator's script was layered on top of an unwanted `local-ducklake` ATTACH. The eight presets are a hand-picked subset of (3 catalog kinds × 2 catalog stores × 4 storage backends); `none` is how an uncovered combination (S3 data files with a Postgres catalog, say) is reached without adding a ninth.
0 commit comments