Skip to content

Commit 992d9c0

Browse files
smithclaysmithclay
authored
docs(demo): add a recorded demo reel, and make --format box render a box (#80)
* fix(cli): render --format box as an actual box, on stdout `PrintBox` called `MaterializedQueryResult::ToString()`, which is DuckDB's debug rendering, not a box: it prints a column-types row and a `[ Rows: N]` line ahead of the data and draws no border. So `duckdb-otlp query` -- whose default on a terminal is `box` -- looked nothing like the `duckdb` CLI it mirrors, and an explicit `--format box` did not produce one. Go through `duckdb::BoxRenderer`, the renderer the duckdb CLI itself uses, via `ToString` rather than `Print`: `BoxRenderer::Print` writes through `duckdb::Printer` to stderr, and a result set is the command's data, which this CLI puts on stdout. Piped output is unaffected -- it still defaults to CSV. * docs(demo): add a recorded demo reel and embed it in the README Three beats, deliberately: receive -> ingest -> query. `duckdb-otlp serve` with no flags, a synthetic checkout flow sent over OTLP/HTTP, and a p95-by-service query run with plain `duckdb` straight over the Parquet the server has already written -- while it is still receiving. Anything outside those three beats is cut; the reel is a hook that autoplays muted, not a tutorial, so it stays at ~36s and 432K (a README GIF that is slow to load is not shown at all). The reel is a VHS tape rather than a screen recording, so it lives in the diff and re-records reproducibly. `demo/record.sh` holds everything a viewer should not watch -- the throwaway workspace, PATH, the prompt -- so `demo.tape` contains only what ends up on camera. Beat 3 uses `duckdb` rather than `duckdb-otlp query` because a running server holds the control database's write lock, so the CLI cannot read its own catalog until it stops (`--readonly` does not help). Reading the Parquet directly is the only live option, and it makes the better point anyway. `demo/seed.py` generates the telemetry locally -- stdlib only, no recorded traffic -- with `payment-service` deliberately the slow and failing one, so the query on screen has something real to find. record.sh refuses to run under VHS 0.12.0, which records the tape, prints "Creating <file>...", exits 0 and writes nothing (charmbracelet/vhs#787). * docs(demo): hand-wave the data source, and size the GIF for the README Two things the first cut got wrong. The ingest caption named the scenario ("Here, a checkout flow"), which is detail nobody needs in beat 2. It now says what a viewer would actually do -- point their own apps or agents at :4318 -- and gestures at the OpenTelemetry Demo as something to send. Deliberately as a suggestion, not as a claim about what is on screen: the traffic is generated by demo/seed.py, and a reel that implied it came from the upstream demo would be asserting a compatibility result it has not shown. seed.py's progress lines drop "checkout" for the same reason -- they repeated the scenario eight times on camera. And the reel was recorded at 1200px, which GitHub scales into its ~900px README column, softening every glyph. It is now recorded at the width it is displayed at, so the terminal text is pixel-crisp where it is actually read (and 320K rather than 432K). The height is load-bearing, not cosmetic. At 520px the terminal held 25 rows, two short of the startup banner plus the lines typed above it, so the banner scrolled past `Wait /Send OTLP/` before it could match. The failure mode is nasty: the recording dies a minute later, at a beat that has nothing to do with the cause, with the server up and listening the whole time. record.sh gained the guard that would have caught this run's other stall: the cleanup pkill pattern was built from $BIN's absolute path, but the tape starts the server off PATH as `duckdb-otlp serve`, so it matched nothing and left a server holding 4318 for the next run. It now matches the real command line, and a port check up front turns "recording failed" into a sentence that says which port and what is on it. * docs(readme): lead with what it does, not what it is The old line -- "DuckDB extension for querying and storing OpenTelemetry traces, logs, and metrics with SQL" -- opened on a category rather than a benefit, and "storing" is doing a lot of quiet work: the thing that is actually novel here is that it *receives* live OTLP and OTAP over HTTP and gRPC and lands it in a lakehouse you own. A reader deciding in fifteen seconds could not tell there was a server in the box at all. So: a promise first ("Your telemetry, in your own storage, answerable in SQL"), then the specifics that back it -- the protocols in, the destinations out, and the three things you do not have to run to get them. The H1 moves to `duckdb-otlp` for the same reason "Extension" led the old one: the binary is half the project. The "As of v0.6" paragraph went with it. It said what the new opening says, with the same links, so it had become an echo. In its place, the one thing the tagline cannot carry: that this ships both as a DuckDB extension and as a standalone binary, and what each gives you. Every destination link is preserved, and "DuckDB extension" and "OpenTelemetry" both still appear above the fold, so nothing that people search for has been dropped. The single-node and unproven-query-volume caveats further down are untouched, and keep "no cluster to size" honest. Also drops the caption under the demo GIF.
1 parent fdcaad6 commit 992d9c0

9 files changed

Lines changed: 587 additions & 7 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,17 @@
1-
# DuckDB OpenTelemetry Extension
1+
# duckdb-otlp
22

33
[![Build](https://github.com/smithclay/duckdb-otlp/actions/workflows/MainDistributionPipeline.yml/badge.svg)](https://github.com/smithclay/duckdb-otlp/actions/workflows/MainDistributionPipeline.yml)
44
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
55
[![DuckDB](https://img.shields.io/badge/DuckDB-%E2%89%A5%201.5.5-FFF000?logo=duckdb&logoColor=black)](https://duckdb.org)
66
[![Docs](https://img.shields.io/badge/docs-online-4c9aff)](https://smithclay.github.io/duckdb-otlp/)
77

8-
DuckDB extension for querying and storing OpenTelemetry traces, logs, and metrics with SQL.
8+
**Your telemetry, in your own storage, answerable in SQL.**
99

10-
As of v0.6, the extension has an embedded HTTP/gRPC server that lets you stream live telemetry using the OpenTelemetry Protocol (including the [new OTel-Arrow protocol](https://github.com/open-telemetry/otel-arrow#what-is-otap)) into [parquet files](https://smithclay.github.io/duckdb-otlp/guides/stream-to-parquet/), [DuckLake](https://smithclay.github.io/duckdb-otlp/guides/stream-to-ducklake/), or Iceberg catalogs like [Amazon S3 Tables](https://smithclay.github.io/duckdb-otlp/guides/stream-to-s3-tables/) and [Cloudflare R2 Data Catalog](https://smithclay.github.io/duckdb-otlp/guides/stream-to-r2-data-catalog/).
10+
Receives OpenTelemetry over OTLP and OTAP and streams it into [Parquet files](https://smithclay.github.io/duckdb-otlp/guides/stream-to-parquet/), [DuckLake](https://smithclay.github.io/duckdb-otlp/guides/stream-to-ducklake/), or Iceberg catalogs like [Amazon S3 Tables](https://smithclay.github.io/duckdb-otlp/guides/stream-to-s3-tables/) and [Cloudflare R2 Data Catalog](https://smithclay.github.io/duckdb-otlp/guides/stream-to-r2-data-catalog/) — no collector to run, no cluster to size, no vendor to pay. Query it with DuckDB, or anything else that reads Parquet.
11+
12+
![duckdb-otlp: receive OpenTelemetry over OTLP, query it with SQL](demo/duckdb-otlp.gif)
13+
14+
It ships two ways. As a **DuckDB extension**, it adds `read_otlp_*` / `read_otap_*` table functions and an embedded HTTP/gRPC ingest server to any DuckDB session. As a **single binary** (`duckdb-otlp`), it runs that same server standalone or in Docker, and doubles as a CLI for one-off conversion, export, and queries. Both speak OTLP and the [OpenTelemetry Arrow protocol](https://github.com/open-telemetry/otel-arrow#what-is-otap) (OTAP).
1115

1216
## Quickstart: Read OpenTelemetry data
1317

‎demo/README.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Demo reel
2+
3+
The GIF at the top of the repository README, and the MP4 beside it.
4+
5+
```sh
6+
./demo/record.sh
7+
```
8+
9+
That writes `demo/duckdb-otlp.gif` (for the README) and `demo/duckdb-otlp.mp4`
10+
(for social posts, where a video beats a GIF on both quality and size).
11+
12+
## What it shows
13+
14+
Three beats, deliberately: **receive → ingest → query**.
15+
16+
1. `duckdb-otlp serve` with no flags at all — OTLP/HTTP on 4318, OTLP/gRPC on
17+
4317, a local DuckLake, no configuration file.
18+
2. `demo/seed.py` stands in for your own apps or agents, sending synthetic
19+
traffic across five services over OTLP/HTTP. Point the real OpenTelemetry
20+
Demo, an SDK, or a collector at the same port and nothing else changes.
21+
3. Plain `duckdb` runs a p95-by-service query straight over the Parquet the
22+
server has already written — *while the server keeps receiving*.
23+
24+
Anything outside those three beats was cut. The reel is a hook that autoplays
25+
muted on a timeline, not a tutorial, so it stays under a minute.
26+
27+
The queries in beat 3 use `duckdb` rather than `duckdb-otlp query` for a
28+
concrete reason: in a catalog mode the running server holds the control
29+
database's write lock, so `duckdb-otlp query` cannot read it until the server
30+
stops (even with `--readonly`). Reading the Parquet directly is both the only
31+
live option and the better story — the data is open the moment it lands.
32+
33+
## The data is synthetic
34+
35+
`demo/seed.py` generates every span, log, and metric locally; nothing on screen
36+
is recorded traffic, and the reel never claims otherwise -- the caption points
37+
viewers at the OpenTelemetry Demo as something they could send, not as the
38+
source of what is being shown. `payment-service` is deliberately the slow, failing one, so
39+
the p95 query has something to find. Pass `--seed N` to fix the RNG.
40+
41+
## Requirements
42+
43+
`vhs`, `ffmpeg`, `duckdb`, `python3`, and a built `duckdb-otlp`
44+
(`GEN=ninja make`, or point `DUCKDB_OTLP_BIN` at one).
45+
46+
**VHS 0.12.0 cannot record.** It runs the tape, prints `Creating <file>...`,
47+
exits 0 and writes nothing — [charmbracelet/vhs#787][vhs787]. It captures the
48+
frames and then never encodes them, so the failure looks exactly like success.
49+
`record.sh` detects that version and stops. Use 0.11.0:
50+
51+
```sh
52+
curl -fsSL -o /tmp/vhs.tar.gz \
53+
https://github.com/charmbracelet/vhs/releases/download/v0.11.0/vhs_0.11.0_Darwin_arm64.tar.gz
54+
tar -xzf /tmp/vhs.tar.gz -C /tmp
55+
VHS_BIN=/tmp/vhs_0.11.0_Darwin_arm64/vhs ./demo/record.sh
56+
```
57+
58+
[vhs787]: https://github.com/charmbracelet/vhs/issues/787
59+
60+
## Editing the reel
61+
62+
`demo/demo.tape` is the script — VHS records exactly what it says, so the reel
63+
is reproducible and reviewable in a diff. Setup that a viewer should not watch
64+
(the workspace, `PATH`, the prompt) lives in `record.sh` and the tape's `Hide`
65+
block, never on camera.

‎demo/demo.tape‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# ===========================================================================
2+
# duckdb-otlp -- demo reel
3+
#
4+
# Record with ./demo/record.sh, which builds the off-camera workspace this
5+
# tape expects ($DEMO_WORKDIR, duckdb-otlp on PATH, DUCKDB_OTLP_DATA_DIR).
6+
# Running `vhs demo/demo.tape` directly will not work.
7+
#
8+
# Three beats, and nothing else: RECEIVE -> INGEST -> QUERY. Anything that is
9+
# not one of those three is cut, however good it looks -- this is a hook that
10+
# autoplays muted on a timeline, not a tutorial. The one addition is a closing
11+
# install line, because a reel with no next action wastes the attention it won.
12+
#
13+
# The typed `#` lines are the narration; they are real shell comments, so
14+
# nothing on screen is staged. Every number shown is produced by the run.
15+
# Pacing: a caption gets ~900ms to land before its command runs; a result
16+
# gets ~4s. SQL is typed faster than prose (Type@18ms) -- it is read as a
17+
# shape, not word by word, and at prose speed it dominates the runtime.
18+
# ===========================================================================
19+
20+
Output demo/duckdb-otlp.gif
21+
Output demo/duckdb-otlp.mp4
22+
23+
Require duckdb
24+
Require python3
25+
26+
# 900x600 is not arbitrary: GitHub renders README images into a ~900px column, so
27+
# anything wider is scaled down and every glyph softens. Recording at the display
28+
# width keeps the terminal text pixel-crisp where it is actually read. The height is
29+
# set by the tallest beat: the startup banner runs ~24 lines, and a terminal too short
30+
# to hold it plus the two typed lines above scrolls it away before `Wait /Send OTLP/`
31+
# can match -- the recording then fails a minute later with the server up and happily
32+
# listening, which is a confusing way to discover the window is two rows short.
33+
Set Shell bash
34+
Set FontFamily "JetBrains Mono"
35+
Set FontSize 14
36+
Set Width 900
37+
Set Height 600
38+
Set Padding 20
39+
Set Theme "Catppuccin Mocha"
40+
Set TypingSpeed 28ms
41+
Set Framerate 10
42+
Set WaitTimeout 90s
43+
44+
# --- off camera: clean workspace, plain prompt -----------------------------
45+
# PS1 is "$ " rather than a fancy glyph: a prompt character the recording font
46+
# lacks renders as a stray ")" in the GIF.
47+
Hide
48+
Type `export PS1="$ "` Enter
49+
Type "cd $DEMO_WORKDIR" Enter
50+
Type "clear" Enter
51+
Show
52+
53+
# --- 1. RECEIVE: one binary, zero configuration ----------------------------
54+
Type "# OpenTelemetry, queryable with SQL -- no collector, no cluster, no vendor."
55+
Sleep 900ms
56+
Enter
57+
Type "duckdb-otlp serve &"
58+
Sleep 500ms
59+
Enter
60+
Wait+Screen@60s /Send OTLP/
61+
Sleep 2600ms
62+
63+
Hide
64+
Type "clear" Enter
65+
Show
66+
67+
# --- 2. INGEST: real OTLP over the wire ------------------------------------
68+
Type "# Configure your apps or agents to send OTLP to :4318 (try the OTel demo)."
69+
Sleep 900ms
70+
Enter
71+
Type "python3 seed.py --traces 2000"
72+
Sleep 400ms
73+
Enter
74+
Wait+Screen@60s /buffered/
75+
Sleep 2400ms
76+
77+
Hide
78+
Type "clear" Enter
79+
Show
80+
81+
# --- 3. QUERY: SQL over telemetry that is already open Parquet -------------
82+
Type "# It lands as Parquet. Query it with plain DuckDB -- while ingest runs."
83+
Sleep 1000ms
84+
Enter
85+
Type "duckdb -dark-mode"
86+
Sleep 300ms
87+
Enter
88+
Sleep 1500ms
89+
90+
Type@18ms "SELECT service_name, count(*) AS spans,"
91+
Enter
92+
Type@18ms " round(quantile_cont(duration_time_unix_nano, .95)/1e6, 1) AS p95_ms,"
93+
Enter
94+
Type@18ms " sum(status_code = 2) AS errors"
95+
Enter
96+
Type@18ms "FROM 'otel/ducklake/storage/main/otlp_traces/*.parquet'"
97+
Enter
98+
Type@18ms "GROUP BY 1 ORDER BY p95_ms DESC;"
99+
Sleep 500ms
100+
Enter
101+
Sleep 4200ms
102+
103+
# --- close: the one next action -------------------------------------------
104+
Type ".quit"
105+
Enter
106+
Sleep 700ms
107+
Type "# curl -fsSL https://smithclay.github.io/duckdb-otlp/install.sh | sh"
108+
Sleep 1000ms
109+
Enter
110+
Sleep 3200ms

‎demo/duckdb-otlp.gif‎

298 KB
Loading

‎demo/duckdb-otlp.mp4‎

285 KB
Binary file not shown.

‎demo/record.sh‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bash
2+
# Record the demo reel. Everything the tape needs is set up here, off camera, so
3+
# demo.tape holds only what a viewer should see.
4+
#
5+
# ./demo/record.sh # record into demo/duckdb-otlp.{gif,mp4}
6+
#
7+
# Requires: vhs, ffmpeg, duckdb, python3, and a built duckdb-otlp binary.
8+
set -euo pipefail
9+
10+
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
11+
cd "$REPO_ROOT"
12+
13+
BIN="${DUCKDB_OTLP_BIN:-$REPO_ROOT/build/release/extension/otlp/duckdb-otlp}"
14+
if [[ ! -x "$BIN" ]]; then
15+
echo "record.sh: no duckdb-otlp binary at $BIN" >&2
16+
echo "record.sh: build one with \`GEN=ninja make\` or set DUCKDB_OTLP_BIN." >&2
17+
exit 1
18+
fi
19+
20+
for tool in duckdb python3 ffmpeg; do
21+
command -v "$tool" >/dev/null || { echo "record.sh: $tool is not installed." >&2; exit 1; }
22+
done
23+
24+
# VHS 0.12.0 runs the tape, prints "Creating <file>...", exits 0 and writes nothing --
25+
# charmbracelet/vhs#787. It captures frames fine and then never encodes them, so the
26+
# failure is silent and looks like success. Pin to 0.11.0 until that is fixed upstream.
27+
VHS="${VHS_BIN:-vhs}"
28+
command -v "$VHS" >/dev/null || { echo "record.sh: vhs is not installed." >&2; exit 1; }
29+
VHS_VERSION="$("$VHS" --version 2>/dev/null | grep -oE 'v?[0-9]+\.[0-9]+\.[0-9]+' | head -1)"
30+
if [[ "$VHS_VERSION" == "v0.12.0" || "$VHS_VERSION" == "0.12.0" ]]; then
31+
cat >&2 <<'EOF'
32+
record.sh: vhs 0.12.0 cannot record -- it exits 0 and writes no file
33+
(https://github.com/charmbracelet/vhs/issues/787).
34+
35+
Install 0.11.0 and point VHS_BIN at it:
36+
37+
curl -fsSL -o /tmp/vhs.tar.gz \
38+
https://github.com/charmbracelet/vhs/releases/download/v0.11.0/vhs_0.11.0_Darwin_arm64.tar.gz
39+
tar -xzf /tmp/vhs.tar.gz -C /tmp
40+
VHS_BIN=/tmp/vhs_0.11.0_Darwin_arm64/vhs ./demo/record.sh
41+
EOF
42+
exit 1
43+
fi
44+
45+
# A fresh workspace every take, so the reel never shows a previous run's data and
46+
# the row counts on screen are always the ones this run produced.
47+
# The reel binds the default ports. If something is already on them the server in
48+
# the tape never comes up, `Wait /Send OTLP/` sits there until it times out, and
49+
# the only symptom is "recording failed" a minute later -- so check up front and
50+
# say so. Checking first also makes the cleanup below safe to be broad: nothing
51+
# was listening when we started, so any server running at exit is one we started.
52+
for port in 4318 4317; do
53+
if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
54+
echo "record.sh: port $port is already in use; the demo server cannot bind it." >&2
55+
echo "record.sh: stop whatever is listening (lsof -nP -iTCP:$port -sTCP:LISTEN) and retry." >&2
56+
exit 1
57+
fi
58+
done
59+
60+
WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/duckdb-otlp-demo.XXXXXX")"
61+
cleanup() {
62+
# Match how the process actually appears in `ps`: the tape runs `duckdb-otlp serve`
63+
# off PATH, so argv[0] is the bare name -- a pattern built from $BIN's absolute
64+
# path matches nothing, and the server survives to hold the port for the next run.
65+
pkill -f 'duckdb-otlp serve' 2>/dev/null || true
66+
rm -rf "$WORKDIR"
67+
}
68+
trap cleanup EXIT
69+
70+
cp "$REPO_ROOT/demo/seed.py" "$WORKDIR/seed.py"
71+
72+
# The tape reads these; exporting them keeps absolute paths out of the tape itself.
73+
export DEMO_WORKDIR="$WORKDIR"
74+
export PATH="$(dirname "$BIN"):$PATH"
75+
# Keeps the banner's data path short and the take reproducible. Set here rather
76+
# than as a `serve` flag so the reel can show `duckdb-otlp serve` with no flags.
77+
export DUCKDB_OTLP_DATA_DIR="./otel"
78+
79+
echo "record.sh: workspace $WORKDIR"
80+
"$VHS" "$REPO_ROOT/demo/demo.tape"
81+
82+
echo
83+
echo "record.sh: wrote"
84+
for f in demo/duckdb-otlp.gif demo/duckdb-otlp.mp4; do
85+
[[ -f "$f" ]] && printf ' %-28s %s\n' "$f" "$(du -h "$f" | cut -f1)"
86+
done

0 commit comments

Comments
 (0)