Skip to content

Commit ad5fc88

Browse files
authored
feat(demos): Demos (#24)
Signed-off-by: Julio Jimenez <julio@julioj.com>
1 parent 1547473 commit ad5fc88

39 files changed

Lines changed: 1435 additions & 40 deletions

‎.github/workflows/build-static.yml‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ name: build-static
99
on:
1010
workflow_call: {}
1111

12+
permissions:
13+
contents: read
14+
1215
jobs:
1316
build:
1417
name: static ${{ matrix.arch }}
@@ -22,25 +25,29 @@ jobs:
2225
arch: aarch64
2326
runs-on: ${{ matrix.runner }}
2427
steps:
25-
- uses: actions/checkout@v7
28+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
2629

2730
# ~200MB static ECL toolchain, rebuilt only when the version or the
2831
# build script changes. runner.arch is in the key because runner.os
2932
# is "Linux" on both architectures.
3033
- name: Cache static ECL toolchain
31-
uses: actions/cache@v6
34+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
3235
with:
3336
path: .cache/ecl-static
3437
key: ecl-static-26.5.5-${{ runner.arch }}-${{ hashFiles('scripts/build-static.sh') }}
3538

3639
- name: Build static binary in Alpine
3740
run: |
3841
mkdir -p .cache/ecl-static
42+
# alpine:3.22 pinned to its manifest-list digest (covers amd64+arm64);
43+
# dependabot cannot see inside run: blocks, so bump it by hand with
44+
# `docker manifest inspect alpine:3.22` (or the registry API) on upgrades.
3945
docker run --rm \
4046
-v "$GITHUB_WORKSPACE:/src" \
4147
-v "$GITHUB_WORKSPACE/.cache/ecl-static:/cache" \
4248
-w /src \
43-
alpine:3.22 sh scripts/build-static.sh
49+
alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce \
50+
sh scripts/build-static.sh
4451
# the container runs as root; later steps and the cache post-step
4552
# run as the unprivileged runner user
4653
sudo chown -R "$(id -u):$(id -g)" dist bin .cache/ecl-static
@@ -53,7 +60,7 @@ jobs:
5360
"./dist/clicklisp-linux-${{ matrix.arch }}" version
5461
echo '(select (1))' | "./dist/clicklisp-linux-${{ matrix.arch }}" compile
5562
56-
- uses: actions/upload-artifact@v7
63+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
5764
with:
5865
name: clicklisp-linux-${{ matrix.arch }}
5966
path: dist/clicklisp-linux-${{ matrix.arch }}

‎.github/workflows/ci.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ on:
55
branches: [main]
66
pull_request:
77

8+
permissions:
9+
contents: read
10+
811
jobs:
912
# Unit tests + a full binary build with the distro ECL, on both
1013
# architectures.
@@ -16,7 +19,7 @@ jobs:
1619
runner: [ubuntu-24.04, ubuntu-24.04-arm]
1720
runs-on: ${{ matrix.runner }}
1821
steps:
19-
- uses: actions/checkout@v7
22+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
2023
- name: Install ECL
2124
run: |
2225
sudo apt-get update
@@ -40,6 +43,7 @@ jobs:
4043
runs-on: ${{ matrix.runner }}
4144
env:
4245
ECL_VERSION: "26.5.5"
46+
ECL_SHA256: "a01a5bcda8c5b73e59dda3494fd13e5fec5db6aa1dad782c3cc3bb57f1633435"
4347
# object-size, shift-base and pointer-overflow fire by design in
4448
# ECL's core (tagged fixnums, stack-allocated frames) -- ~120 known
4549
# reports. Everything else in UBSan plus all of ASan stays on.
@@ -54,14 +58,14 @@ jobs:
5458
ASAN_OPTIONS: "detect_leaks=0:detect_stack_use_after_return=0:allow_user_segv_handler=1"
5559
UBSAN_OPTIONS: "print_stacktrace=1"
5660
steps:
57-
- uses: actions/checkout@v7
61+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
5862
- name: Install dependencies
5963
run: |
6064
sudo apt-get update
6165
sudo apt-get install -y libgmp-dev
6266
- name: Cache sanitized ECL
6367
id: cache-ecl
64-
uses: actions/cache@v6
68+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
6569
with:
6670
path: ~/ecl-asan
6771
# bump the -vN suffix when SAN_CFLAGS or configure flags change
@@ -70,6 +74,7 @@ jobs:
7074
if: steps.cache-ecl.outputs.cache-hit != 'true'
7175
run: |
7276
curl -fsSL "https://ecl.common-lisp.dev/static/files/release/ecl-${ECL_VERSION}.tgz" -o ecl.tgz
77+
echo "${ECL_SHA256} ecl.tgz" | sha256sum -c -
7378
mkdir ecl-src
7479
tar xzf ecl.tgz -C ecl-src --strip-components=1
7580
cd ecl-src

‎.github/workflows/pages.yml‎

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,9 @@ name: pages
33
on:
44
push:
55
branches: [main]
6-
paths: ["site/**", "logo.png", ".github/workflows/pages.yml"]
6+
paths: ["site/**", "logo.png", "examples/**", "src/**", "clicklisp.asd", "Makefile", "build/build.lisp", "version.sexp", "scripts/gen-site-data.sh", ".github/workflows/pages.yml"]
77
workflow_dispatch:
88

9-
permissions:
10-
contents: read
11-
pages: write
12-
id-token: write
13-
149
# One deploy at a time; let an in-flight production deploy finish.
1510
concurrency:
1611
group: pages
@@ -19,12 +14,19 @@ concurrency:
1914
jobs:
2015
build:
2116
runs-on: ubuntu-24.04
17+
permissions:
18+
contents: read
19+
pages: write # configure-pages (enablement: true) reads/creates the Pages site via the API
2220
steps:
23-
- uses: actions/checkout@v7
24-
- uses: actions/configure-pages@v6
21+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
22+
- uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
2523
with:
2624
enablement: true
27-
- uses: actions/setup-node@v7
25+
- name: Install ECL
26+
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ecl
27+
- name: Build clicklisp and generate demo data
28+
run: make all site-data
29+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
2830
with:
2931
node-version: 22
3032
cache: npm
@@ -35,16 +37,19 @@ jobs:
3537
cp ../logo.png public/logo.png # keep the site logo in sync with the repo's
3638
npm ci
3739
npm run build
38-
- uses: actions/upload-pages-artifact@v5
40+
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5
3941
with:
4042
path: site/dist
4143

4244
deploy:
4345
needs: build
4446
runs-on: ubuntu-24.04
47+
permissions:
48+
pages: write
49+
id-token: write
4550
environment:
4651
name: github-pages
4752
url: ${{ steps.deployment.outputs.page_url }}
4853
steps:
4954
- id: deployment
50-
uses: actions/deploy-pages@v5
55+
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5

‎.github/workflows/release.yml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,13 @@ on:
44
push:
55
tags: ["v*"]
66

7-
permissions:
8-
contents: write
9-
107
jobs:
118
check-version:
129
runs-on: ubuntu-24.04
10+
permissions:
11+
contents: read
1312
steps:
14-
- uses: actions/checkout@v7
13+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
1514
- name: Tag must match version.sexp
1615
run: |
1716
tag="${GITHUB_REF_NAME#v}"
@@ -28,9 +27,11 @@ jobs:
2827
release:
2928
needs: build
3029
runs-on: ubuntu-24.04
30+
permissions:
31+
contents: write
3132
steps:
32-
- uses: actions/checkout@v7
33-
- uses: actions/download-artifact@v8
33+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
34+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
3435
with:
3536
path: dist
3637
merge-multiple: true

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,5 @@ dist/
1313
.claude/CLAUDE.md
1414
site/node_modules/
1515
site/dist/
16+
site/src/generated/
1617
*.tsbuildinfo

‎Makefile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ ECL ?= ecl
22
BIN := bin/clicklisp
33
SOURCES := clicklisp.asd version.sexp $(wildcard src/*.lisp)
44

5-
.PHONY: all test smoke clean
5+
.PHONY: all test smoke site-data clean
66

77
all: $(BIN)
88

@@ -23,7 +23,13 @@ smoke: $(BIN)
2323
$(BIN) rules sql --all --load examples/analytics/uk-price-paid.lisp
2424
$(BIN) rules sql --all --load examples/analytics/repo-health.lisp
2525
$(BIN) rules sql --all --load examples/analytics/hackernews.lisp
26+
$(BIN) rules sql --all --load examples/analytics/github-events.lisp
27+
$(BIN) rules sql --all --load examples/analytics/repo-health.lisp --load examples/analytics/repo-health-playground.lisp
28+
$(BIN) rules json --all --load examples/rules.lisp | python3 -m json.tool > /dev/null
2629
@echo smoke OK
2730

31+
site-data: $(BIN)
32+
scripts/gen-site-data.sh
33+
2834
clean:
2935
rm -rf bin

‎README.md‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,12 @@ parameters; `CLICKLISP_FORMAT` and `CLICKLISP_PLAY` override the output
8484
format and the endpoint. One statement per invocation (the ClickHouse
8585
HTTP interface is single-statement).
8686

87+
The same queries run as [live demos](https://cl.clickhouse.com/#demos)
88+
in the browser: the site's deploy workflow builds the binary, compiles
89+
the example libraries with `rules json`, and each demo page sends the
90+
compiled SQL to the playground with typed `param_` bindings — no server
91+
of ours in the path.
92+
8793
## The query language
8894

8995
Operators and clause names are matched by symbol *name*, so forms can be
@@ -168,8 +174,15 @@ Rules are validated (compiled) at load time and rendered on demand:
168174
clicklisp rules --load rules.lisp # list
169175
clicklisp rules sql --load rules.lisp --all # emit SQL for all
170176
clicklisp rules sql --load rules.lisp ssh-bruteforce
177+
clicklisp rules json --load rules.lisp --all # machine-readable JSON
171178
```
172179

180+
`rules json` dumps each rule as JSON — `name`, `description`, `severity`,
181+
`tags`, the extracted `{name:Type}` `params`, compact `sql`, multiline
182+
`sql_pretty`, and the reconstructed `form` — for anything downstream that
183+
wants rules as data; it is what feeds the
184+
[live demos](https://cl.clickhouse.com/#demos) at build time.
185+
173186
See [examples/rules.lisp](examples/rules.lisp).
174187

175188
Exit codes: `0` success, `1` bad input (a query that does not compile),
@@ -179,7 +192,7 @@ Exit codes: `0` success, `1` bad input (a query that does not compile),
179192

180193
`defquery` is `defrule` minus the detection metadata: named queries for
181194
any domain, served by the same `rules` commands.
182-
[examples/analytics/](examples/analytics/) has three libraries built on
195+
[examples/analytics/](examples/analytics/) has four libraries built on
183196
the playground's example datasets:
184197

185198
- [uk-price-paid.lisp](examples/analytics/uk-price-paid.lisp) — UK
@@ -189,11 +202,21 @@ the playground's example datasets:
189202
- [repo-health.lisp](examples/analytics/repo-health.lisp) — engineering
190203
analytics (code churn, bus factor, comment-to-code ratio) over
191204
`clickhouse git-import` tables; the whole library is one macro
192-
parameterized by table names, so it runs against *your* repo locally
193-
or the playground's pre-imported ClickHouse and Grafana mirrors.
205+
parameterized by table names, so it runs against *your* repo locally,
206+
and [repo-health-playground.lisp](examples/analytics/repo-health-playground.lisp)
207+
re-invokes it against the playground's pre-imported ClickHouse mirror
208+
(load both files; the second registration wins).
194209
- [hackernews.lisp](examples/analytics/hackernews.lisp) — text search,
195210
array lambdas, CTE composition, and a query that calls back into a
196211
clicklisp executable UDF.
212+
- [github-events.lisp](examples/analytics/github-events.lisp) — a
213+
`defrule` threat pack over 11 billion rows of public GitHub events
214+
(`github.events`): branch-reset storms, mass ref deletion, tag
215+
retargeting, star storms, bulk collaborator adds, commit-hour
216+
anomalies, each tagged with MITRE ATT&CK technique IDs. Two rules
217+
default to historical windows because the dataset's loader stopped
218+
populating `push_size` and tag `CreateEvent`s around October 2025 —
219+
see the file header.
197220

198221
Rule files are full Common Lisp, and queries are data — so a macro can
199222
generate a family of queries with no copy-paste and no templating
@@ -365,6 +388,15 @@ compiling Lisp at `CREATE FUNCTION` time.
365388
- **release** — pushing a tag `vX.Y.Z` (which must match `version.sexp`)
366389
builds both static binaries and publishes them with SHA256SUMS to a
367390
GitHub release.
391+
- **pages** — builds the binary with the distro ECL, generates the demo
392+
data (`make site-data`), builds the site, and deploys it to
393+
[cl.clickhouse.com](https://cl.clickhouse.com) — so the demos are
394+
compiled by the same commit they document.
395+
396+
Supply-chain posture: every workflow action is pinned to a commit SHA
397+
(dependabot keeps the pins fresh), workflows run with least-privilege
398+
`permissions`, the ECL source tarball is SHA256-verified before it is
399+
built, and the Alpine build image is pinned by digest.
368400

369401
## Layout
370402

@@ -377,6 +409,8 @@ src/main.lisp CLI entry point
377409
build/build.lisp ECL AOT build (c:build-program; works on static ECL)
378410
scripts/build-static.sh static musl build, run inside alpine:3.22
379411
scripts/play.sh pipe SQL from stdin to the public ClickHouse playground
412+
scripts/gen-site-data.sh compile the example libraries to JSON for the site demos
380413
examples/ detection rules, analytics libraries, UDFs, server config
414+
site/ cl.clickhouse.com — docs plus live demos of the libraries
381415
tests/ zero-dependency harness + suite
382416
```

‎clicklisp.asd‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
:serial t
1818
:components ((:file "tests/harness")
1919
(:file "tests/compiler-tests")
20+
(:file "tests/rules-json-tests")
2021
(:file "tests/udf-tests"))
2122
:perform (test-op (o c)
2223
(unless (symbol-call '#:clicklisp/test '#:run-tests)

0 commit comments

Comments
 (0)