Skip to content

Commit a4d5b61

Browse files
smithclayclaude
andcommitted
build(daemon): link ducklake into the daemon from DuckDB's pinned config
daemon-compile could not pass on a DuckDB 2.0 development build: the default local-ducklake mode INSTALLs ducklake at startup, no ducklake is published for a dev build, and the 2.0 alpha's binary null-derefs on load against the branch tip. No version string fixes that, because the problem is the coupling itself -- the daemon only worked when a published extension build happened to match its ABI. So the daemon stops downloading the extension its default mode needs: - DUCKDB_OTLP_BUILT_IN_EXTENSIONS (CMake list) links extensions into DuckDB. extension_config.cmake takes each from DuckDB's own pinned config (duckdb/.github/config/extensions/<name>.cmake: the commit and patches DuckDB builds against this source), so the ABI matches by construction. - `make server-release` -- the path Docker and both daemon CI jobs take -- sets it to ducklake and first builds CRoaring, ducklake's one dependency, into build/deps, since those builds run without vcpkg. v4.7.2 rather than vcpkg's 4.4.2, whose package config CMake 4 rejects. The extension distribution builds are unchanged. - Core(name) reports BUILT_IN when LinkedExtensionRegistry says the binary links it, so no INSTALL/LOAD is emitted and the banner says "built in". Asking the binary rather than a build flag means the two cannot drift. - The image stops priming ducklake. quack stays a runtime install: on 2.0 it draws its tokens from the writable crypto module only httpfs (OpenSSL) provides, so building it in means building httpfs in, against the image's no-libssl design. test_init_sql_adds_a_parquet_destination_next_to_the_ducklake_catalog read the daemon's catalog back with the host's duckdb==1.5.6, which cannot open a 2.0-format database (storage version 69). It now reads it back through the binary under test (`duckdb-otlp query`), which is still a real DuckLake read and no longer depends on the host's DuckDB version. Verified: local-ducklake daemon with an empty extension cache ingests, seals and reads back with zero downloads; pytest test/server 247/247 both on macOS and against the Docker-built linux/arm64 daemon-export binary; make test 23/23, seal harness 204, arrow harness 6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 2e98434 commit a4d5b61

6 files changed

Lines changed: 101 additions & 21 deletions

File tree

‎AGENTS.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,17 @@ This file provides guidance to coding agents when working with code in this repo
2222
`Copy Function "parquet" is not in the catalog`. Separately, loading an extension
2323
config no longer links it into the shell/`unittest`: `extension_config.cmake` asks for that with
2424
`duckdb_extension_statically_link(otlp)`.
25+
- **Built-in runtime extensions (`DUCKDB_OTLP_BUILT_IN_EXTENSIONS`).** A CMake list of extensions
26+
linked into DuckDB instead of `INSTALL`ed by the daemon at startup. `extension_config.cmake`
27+
takes each from DuckDB's own pinned config (`duckdb/.github/config/extensions/<name>.cmake`,
28+
the commit and patches DuckDB builds against this source), so its ABI matches by construction.
29+
`make server-release` (Docker, both daemon CI jobs) sets `DAEMON_BUILT_IN_EXTENSIONS ?= ducklake`
30+
and first builds CRoaring, ducklake's one dependency, into `build/deps/croaring`, because the
31+
daemon builds run without vcpkg (v4.7.2, not vcpkg's 4.4.2, whose package config CMake 4
32+
rejects). The extension distribution builds leave the list empty. `quack` is deliberately not
33+
in it: on DuckDB 2.0 it draws tokens from the writable crypto module only httpfs (OpenSSL)
34+
provides, so building it in means building httpfs in, against the runtime image's no-libssl
35+
design. Linked extensions load in every command, not only `serve`.
2536
- **Windows (`windows_amd64`, MSVC) builds the extension only.** `CMakeLists.txt` gates the
2637
native daemon (`duckdb_otlp_server`) and the C++ harnesses (`otlp_seal_harness`,
2738
`otlp_arrow_harness`) on `NOT WIN32`, because `make release` builds the default ALL target
@@ -89,7 +100,7 @@ CI in `.github/workflows/MainDistributionPipeline.yml`: `daemon-compile` is a ch
89100

90101
On a `v*` tag, `daemon-release` attaches those four binaries to the GitHub release as `duckdb-otlp-<tag>-<platform>.tar.gz` plus `SHA256SUMS`. It recompiles nothing (it downloads the artifacts the build jobs uploaded) and gates on `docker-smoke`, so a binary that failed the e2e never reaches a release. The Darwin artifacts are deliberately named `duckdb-otlp-darwin-*` rather than `duckdb-otlp-server-*`: the latter is the glob `docker-image` downloads, which must only match the Linux binaries. `daemon-release` creates the release when the tag was pushed without one, which in turn triggers `pages.yml` to attach the extension repository.
91102

92-
Extension offline-cache coupling: the `deps` stage pre-`INSTALL`s ducklake/iceberg/httpfs/aws/postgres/quack and community gcs with a throwaway DuckDB CLI into `HOME=/duckdb-home` (copied into the final image, owned by the nonroot user; the CLI itself is not copied). The daemon reuses that cache only because it runs with the same `HOME` and is pinned to the same `DUCKDB_VERSION`. If those diverge, the daemon's startup `INSTALL`/`LOAD` re-downloads (or fails when offline). The `otlp` extension is statically embedded in the daemon and is intentionally not installed. Bind-mounted `/data` must be writable by uid 65532; named/anonymous volumes inherit the image's nonroot ownership automatically.
103+
Extension offline-cache coupling: the `deps` stage pre-`INSTALL`s iceberg/httpfs/aws/postgres/quack and community gcs with a throwaway DuckDB CLI into `HOME=/duckdb-home` (copied into the final image, owned by the nonroot user; the CLI itself is not copied). The daemon reuses that cache only because it runs with the same `HOME` and is pinned to the same `DUCKDB_VERSION`. If those diverge, the daemon's startup `INSTALL`/`LOAD` re-downloads (or fails when offline). The `otlp` extension is statically embedded in the daemon and is intentionally not installed, and so is `ducklake` (see *Built-in runtime extensions* below), which is what lets the default `local-ducklake` mode run on a DuckDB build that has no published extensions — a 2.0 development build, for one. Bind-mounted `/data` must be writable by uid 65532; named/anonymous volumes inherit the image's nonroot ownership automatically.
93104

94105
### Building for WebAssembly
95106
```bash
@@ -290,7 +301,7 @@ Prefer one canonical page per topic and link to it instead of duplicating exampl
290301
- **Standard OTLP env vars**: `OTEL_EXPORTER_OTLP_PROTOCOL` (only when no port is explicit), and `OTEL_EXPORTER_OTLP_HEADERS` (`Authorization=Bearer`) are honored. `OTEL_EXPORTER_OTLP_ENDPOINT` is deliberately **not read** (it usually points at the user's real collector, so reading it as a bind address is a footgun); the per-signal endpoint and TLS variables are **rejected at startup** rather than ignored, because silently dropping them would misrepresent the deployment.
291302
- **AWS credentials use a multi-source chain**: `BuildCredentialChainSecret` emits `CHAIN 'env;config;instance'` when no `AWS_PROFILE` is set (and `CHAIN config` + `PROFILE` when one is). A single-source chain is a trap in both directions — `CHAIN env` cannot see an EC2/ECS instance role, `CHAIN instance` cannot see the standard AWS variables — and this mode set has been bitten by each: `aws-ducklake` hardcoded `CHAIN instance`, and consolidating it onto the shared builder briefly swapped that for `CHAIN env`, taking instance-role credentials away from exactly the deployments that have nothing else. `AwsCredentialsSource` must keep naming the sources the chain actually contains; it previously claimed "environment, then instance role" while the SQL said `env`.
292303
- **`parquet` locality is `IsRemotePath`, not `IsS3Path`**: `IsS3Path` answers "does this need the S3 credential set", which is a different question from "is this a local directory". Using it as the locality test sent a `gcss://` export path down the local branch — no filesystem extension loaded, empty setup SQL, `validate` reporting success — and the first seal was the first sign. A remote non-S3 path is now recorded as unusable at configuration time. The R2 modes' default DuckLake `DATA_PATH` keeps its trailing slash (`ObjectStorePath` + `"/"`), because DuckLake treats it as a directory prefix and dropping it would repoint an existing deployment at a different prefix than the one in its catalog.
293-
- **A mode declares its extensions once**: `ServerConfig::mode_extensions` is a list of `{name, ExtensionSource}` (`BUILT_IN` / `CORE` / `COMMUNITY`), and both the `INSTALL`/`LOAD` prelude (`ExtensionSetupSql`, spliced ahead of each mode's SQL) and the startup banner are derived from it. Each mode used to state the same list twice — once as strings for the banner, once as hand-typed SQL — so they could disagree with no symptom beyond a misleading banner. Provenance is the only thing that varies and it decides everything: `BUILT_IN` (otlp, statically embedded) emits nothing, so an `INSTALL otlp` that would fetch a *different* published build is unrepresentable; `COMMUNITY` emits `FROM community`, without which the install resolves nowhere. `test_generated_sql_installs_exactly_what_the_banner_reports` pins the two renderings together per mode. The image's offline extension cache (`docker/duckdb-otlp-server/Dockerfile`) is a third copy that cannot share the declaration — different language, and `deps` must not depend on `builder` — so `test_the_image_primes_every_extension_some_mode_needs` asserts it covers the union instead; without it, a mode gaining an extension silently sends the container to the network at startup, or fails outright when offline. Quack is declared the same way but kept out of `mode_extensions`, because it is loaded by the serve path only and `query`/`export` must not pull a server extension in: `QUACK_EXTENSION` is declared once and consumed by `StartQuackSql` and by `AllExtensions()` (what the banner prints). It is `CORE` — `INSTALL quack FROM community` 404s while a bare `INSTALL quack` resolves, which is also how the image primes it. Its `LOAD` previously carried no `INSTALL`, so `--quack` failed with `Extension "quack" not found. Install it first` on every host except the container, whose image primes the cache.
304+
- **A mode declares its extensions once**: `ServerConfig::mode_extensions` is a list of `{name, ExtensionSource}` (`BUILT_IN` / `CORE` / `COMMUNITY`), and both the `INSTALL`/`LOAD` prelude (`ExtensionSetupSql`, spliced ahead of each mode's SQL) and the startup banner are derived from it. Each mode used to state the same list twice — once as strings for the banner, once as hand-typed SQL — so they could disagree with no symptom beyond a misleading banner. Provenance is the only thing that varies and it decides everything: `BUILT_IN` (otlp, plus any extension the binary links — `Core(name)` asks `LinkedExtensionRegistry` at runtime rather than a build flag, so the answer cannot drift from the link line) emits nothing, so an `INSTALL otlp` that would fetch a *different* published build is unrepresentable; `COMMUNITY` emits `FROM community`, without which the install resolves nowhere. `test_generated_sql_installs_exactly_what_the_banner_reports` pins the two renderings together per mode. The image's offline extension cache (`docker/duckdb-otlp-server/Dockerfile`) is a third copy that cannot share the declaration — different language, and `deps` must not depend on `builder` — so `test_the_image_primes_every_extension_some_mode_needs` asserts it covers the union instead; without it, a mode gaining an extension silently sends the container to the network at startup, or fails outright when offline. Quack is declared the same way but kept out of `mode_extensions`, because it is loaded by the serve path only and `query`/`export` must not pull a server extension in: `QuackExtension()` is declared once and consumed by `StartQuackSql` and by `AllExtensions()` (what the banner prints); it is a function rather than a constant because the linked-extension registry it consults is filled during static initialization. It is `CORE` — `INSTALL quack FROM community` 404s while a bare `INSTALL quack` resolves, which is also how the image primes it. Its `LOAD` previously carried no `INSTALL`, so `--quack` failed with `Extension "quack" not found. Install it first` on every host except the container, whose image primes the cache.
294305
- **One name per setting, one error per run**: storage configuration used to spell the same concept differently per mode (five names for an R2 access key, `NEON_PG*` vs `PG*` for the same Postgres catalog, six for a bucket) — about 70 variable names for ~25 concepts. Each mode now reads one canonical name: the cloud's own convention where one exists (`AWS_*`, `PG*` libpq names), `R2_*` for R2, `S3_*` for S3, `DUCKDB_OTLP_*` only for this daemon's own settings. `ConfigProblems` (`server_config.cpp`) accumulates every problem with a mode's configuration — absent values via `Add`, set-but-unusable ones via `Invalid` — and `ConfigureMode` throws once naming all of them, deduplicated by variable. Malformed values are recorded rather than thrown so they do not short-circuit the rest of the pass, which is what made `DUCKLAKE_DATA_PATH=/local/path` report the path on one run and `AWS_REGION` only on the next — configuring `r2-data-catalog` took six runs to satisfy when each error named a single variable. `MODES` is one table of `{name, configure}` so the supported-mode list in the error cannot drift from the set actually handled.
295306
- **`parquet` mode reads back through views, defined in one place**: the mode has no catalog, so a signal is readable only through a view over `<root>/<table>/**/*.parquet`. `SealParquet` already creates that view after each successful export (it can only be created once files exist, which is why mode setup cannot do it), so serving and reading on one host was already covered. What was broken is narrower than it looks: `ExistingSignalTables` asked `duckdb_tables()` only, so `export` found **zero** signal tables and refused to export data that was sitting there readable — it now unions `duckdb_views()`. `RegisterParquetExportViews` (`commands.cpp`, from `OpenConfiguredDatabase`) covers the remaining gap: a dataset written by another process or host, whose control database has never been sealed into. Both sites build the SELECT from `ParquetDatasetSelect` (`otlp_sql_util.hpp`) so the glob depth and the two load-bearing `read_parquet` options cannot drift — `hive_partitioning=false` because the COPY sets `WRITE_PARTITION_COLUMNS false` (year/month/day live only in the path, and inferring them appends three columns no catalog mode has), `union_by_name=true` so a schema change still reads. Registration is scoped to the signals the caller could read — the resolved `--signal` list for `export`, the signal names appearing in the SQL for `query` — because each probe is a directory listing, and a remote LIST apiece against an `s3://` root. A read-only database cannot hold a view, so that path falls back to session-scoped `TEMP VIEW`s.
296307
- **`--mode none`**: attaches nothing, installs nothing, sets no catalog — the destination is entirely `--init-sql` plus `--catalog`. Every mode contributes only setup SQL and a catalog name (see `BootSql`), both already expressible that way, but without an opt-out an operator's script was layered on top of an unwanted `local-ducklake` ATTACH. The eight presets are a hand-picked subset of (3 catalog kinds × 2 catalog stores × 4 storage backends); `none` is how an uncovered combination (S3 data files with a Postgres catalog, say) is reached without adding a ninth.

‎Makefile‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,34 @@ wasm_relink: wasm_link
8686

8787
.PHONY: server-release docker-image docker-image-local docker-image-multiarch
8888

89-
server-release: ${EXTENSION_CONFIG_STEP}
89+
# Runtime extensions the daemon links instead of INSTALLing at startup (see
90+
# extension_config.cmake). ducklake is the default mode's catalog. quack is not
91+
# here: on DuckDB 2.0 it draws its tokens from the writable crypto module httpfs
92+
# (OpenSSL) provides, so building it in means building httpfs in too.
93+
DAEMON_BUILT_IN_EXTENSIONS ?= ducklake
94+
95+
# ducklake needs CRoaring. The extension builds take it from vcpkg, but the daemon
96+
# builds (Docker, macOS CI) run without vcpkg, so build it into a private prefix,
97+
# for the same architecture as the daemon. v4.7.2 rather than vcpkg's 4.4.2: the
98+
# 4.4.2 package config declares cmake_minimum_required(2.8), which CMake 4 rejects.
99+
CROARING_VERSION := v4.7.2
100+
CROARING_PREFIX := $(PROJ_DIR)build/deps/croaring
101+
CROARING_ARCH_FLAG := $(if $(OSX_BUILD_ARCH),-DCMAKE_OSX_ARCHITECTURES=$(OSX_BUILD_ARCH),)
102+
103+
$(CROARING_PREFIX)/.built-$(CROARING_VERSION):
104+
rm -rf build/deps/croaring-src $(CROARING_PREFIX)
105+
git clone --quiet --depth 1 --branch $(CROARING_VERSION) https://github.com/RoaringBitmap/CRoaring build/deps/croaring-src
106+
cmake $(GENERATOR) -S build/deps/croaring-src -B build/deps/croaring-build -DCMAKE_BUILD_TYPE=Release \
107+
-DCMAKE_INSTALL_PREFIX=$(CROARING_PREFIX) -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=OFF \
108+
-DENABLE_ROARING_TESTS=OFF -DROARING_USE_CPM=OFF $(CROARING_ARCH_FLAG)
109+
cmake --build build/deps/croaring-build --target install
110+
touch $@
111+
112+
server-release: ${EXTENSION_CONFIG_STEP} $(CROARING_PREFIX)/.built-$(CROARING_VERSION)
90113
mkdir -p build/release
91-
cmake $(GENERATOR) $(BUILD_FLAGS) $(EXT_RELEASE_FLAGS) $(VCPKG_MANIFEST_FLAGS) -DCMAKE_BUILD_TYPE=Release -S $(DUCKDB_SRCDIR) -B build/release
114+
cmake $(GENERATOR) $(BUILD_FLAGS) $(EXT_RELEASE_FLAGS) $(VCPKG_MANIFEST_FLAGS) -DCMAKE_BUILD_TYPE=Release \
115+
-DDUCKDB_OTLP_BUILT_IN_EXTENSIONS="$(DAEMON_BUILT_IN_EXTENSIONS)" -DCMAKE_PREFIX_PATH=$(CROARING_PREFIX) \
116+
-S $(DUCKDB_SRCDIR) -B build/release
92117
cmake --build build/release --config Release --target duckdb_otlp_server
93118

94119
docker-image: docker-image-local

‎docker/duckdb-otlp-server/Dockerfile‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,12 @@ RUN set -eux; \
7070
rm -f /tmp/duckdb.zip
7171

7272
# Prime every mode's extensions into HOME=/duckdb-home so the daemon's startup INSTALL/LOAD
73-
# is offline. The daemon (otlp) is statically embedded and is intentionally not installed.
73+
# is offline. Extensions statically linked into the daemon are intentionally not installed:
74+
# otlp, plus ducklake (`make server-release` builds it from DuckDB's pinned config, so the
75+
# default local-ducklake mode downloads nothing and does not depend on a published build
76+
# matching the daemon's DuckDB ABI).
7477
RUN set -eux; mkdir -p "${HOME}"; \
75-
for ext in ducklake iceberg httpfs aws postgres quack; do \
78+
for ext in iceberg httpfs aws postgres quack; do \
7679
/tmp/duckdb-cli/duckdb -unsigned -c "INSTALL ${ext};"; \
7780
done; \
7881
/tmp/duckdb-cli/duckdb -c "INSTALL gcs FROM community; LOAD gcs;"; \

‎extension_config.cmake‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,21 @@ endif()
1616
# when a config asks for it; loading it no longer implies linking it.
1717
duckdb_extension_statically_link(otlp)
1818

19+
# Runtime extensions compiled into the binary instead of INSTALLed when it
20+
# starts, e.g. "ducklake;quack" (set by `make server-release`). Each one is
21+
# taken from DuckDB's own pinned config -- the commit and patches DuckDB builds
22+
# and tests against this source -- so its ABI matches the daemon by
23+
# construction, which no published binary can promise for a development build.
24+
# The daemon notices at runtime that an extension is linked and stops emitting
25+
# INSTALL/LOAD for it.
26+
set(DUCKDB_OTLP_BUILT_IN_EXTENSIONS
27+
""
28+
CACHE STRING "Extensions to link into DuckDB from DuckDB's pinned configs")
29+
foreach(built_in IN LISTS DUCKDB_OTLP_BUILT_IN_EXTENSIONS)
30+
include(
31+
"${CMAKE_CURRENT_LIST_DIR}/duckdb/.github/config/extensions/${built_in}.cmake"
32+
)
33+
duckdb_extension_statically_link(${built_in})
34+
endforeach()
35+
1936
# Any extra extensions that should be built e.g.: duckdb_extension_load(json)

‎src/server/server_config.cpp‎

Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#include "duckdb/common/error_data.hpp"
44
#include "duckdb/common/exception.hpp"
55
#include "duckdb/common/string_util.hpp"
6+
#include "duckdb/main/extension/linked_extension_registry.hpp"
67
#include "env_source.hpp"
78
#include "otlp_sql_util.hpp"
89
#include "otlp_uri.hpp"
@@ -479,9 +480,24 @@ CREATE OR REPLACE SECRET %s (
479480
return secret_sql;
480481
}
481482

482-
//! Shorthands for declaring a mode's extension list.
483+
//! Whether an extension is compiled into this binary (`DUCKDB_OTLP_BUILT_IN_EXTENSIONS` at build
484+
//! time). Asked of the binary itself rather than of a build flag, so the answer cannot drift
485+
//! from what was actually linked.
486+
bool IsLinkedIntoBinary(const string &name) {
487+
for (auto &linked : duckdb::LinkedExtensionRegistry::Get()) {
488+
if (StringUtil::CIEquals(linked.name, name)) {
489+
return true;
490+
}
491+
}
492+
return false;
493+
}
494+
495+
//! Shorthands for declaring a mode's extension list. A core extension linked into the binary
496+
//! is built in: DuckDB loads every linked extension when the database opens, and INSTALLing it
497+
//! would fetch a published build that is not the one running -- for a DuckDB development build,
498+
//! one with a different ABI, if it exists at all.
483499
ModeExtension Core(const char *name) {
484-
return {name, ExtensionSource::CORE};
500+
return {name, IsLinkedIntoBinary(name) ? ExtensionSource::BUILT_IN : ExtensionSource::CORE};
485501
}
486502

487503
ModeExtension Community(const char *name) {
@@ -493,8 +509,11 @@ ModeExtension Community(const char *name) {
493509
//! LOAD, which is why the banner used to omit it even when Quack was running.
494510
//!
495511
//! CORE, not COMMUNITY: `INSTALL quack FROM community` 404s, while a bare `INSTALL quack`
496-
//! resolves from the core repository (which is also how the image primes it).
497-
const ModeExtension QUACK_EXTENSION = {"quack", ExtensionSource::CORE};
512+
//! resolves from the core repository. A function, not a constant: whether it is built in is read
513+
//! from the linked-extension registry, which is filled during static initialization.
514+
ModeExtension QuackExtension() {
515+
return Core("quack");
516+
}
498517

499518
//! The otlp extension, statically embedded in the daemon. Declared so the banner reports it,
500519
//! never emitted as SQL -- INSTALLing it would reach for a published build that is not the one
@@ -1331,7 +1350,7 @@ FROM %s(
13311350
std::vector<ModeExtension> ServerConfig::AllExtensions() const {
13321351
auto extensions = mode_extensions;
13331352
if (quack_enabled) {
1334-
extensions.push_back(QUACK_EXTENSION);
1353+
extensions.push_back(QuackExtension());
13351354
}
13361355
return extensions;
13371356
}
@@ -1340,7 +1359,7 @@ string ServerConfig::StartQuackSql() const {
13401359
if (!quack_enabled) {
13411360
return "";
13421361
}
1343-
return ExtensionSetupSql({QUACK_EXTENSION}) + StringUtil::Format(R"SQL(
1362+
return ExtensionSetupSql({QuackExtension()}) + StringUtil::Format(R"SQL(
13441363
SELECT listen_uri
13451364
FROM quack_serve(
13461365
%s,

0 commit comments

Comments
 (0)