Skip to content

Commit 982eab1

Browse files
committed
feat(clickhouse): migration 0029 drops the ten hourly rollup tables
Nothing has read them since 0.7.0 moved every chart to fifteen-minute rollups, and with the fill checking against events_raw nothing needs them. 0029 drops the eight view targets plus session_rollups_1h and revenue_1h; the daily tables stay. Site deletion stops naming them in the same release (ClickHouse 44 -> 34 targets), their grants go, and a deletion started before the drop settles a dropped table as empty instead of retrying it forever. CHANGELOG 0.8.0 and SELF-HOSTING cover the upgrade, the 0.6 -> 0.8 skip, and why going back is a restore.
1 parent b93dba5 commit 982eab1

35 files changed

Lines changed: 669 additions & 307 deletions

‎CHANGELOG.md‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,43 @@ taking.
88
Releases before 0.6.0 have their notes on the
99
[GitHub releases page](https://github.com/OpenLabs-so/openanalytics/releases).
1010

11+
## [0.8.0] - 2026-09-19
12+
13+
**Upgrade notes: nothing by hand — and going back is a restore.**
14+
`./upgrade.sh` as always. ClickHouse migration 0029 drops the ten hourly
15+
rollup tables 0.7.0 stopped writing, so **returning to 0.7.0 is not a matter
16+
of changing the image tag**: the older images would find those tables gone.
17+
Go back with `./rollback.sh` to the snapshot the upgrade took. Skipping 0.7.0
18+
(0.6.0 → 0.8.0) is supported: migrations 0025–0029 run in one pass and the
19+
migrate container fills the fifteen-minute rollups from your raw events
20+
afterwards, exactly as in 0.7.0. Details in `SELF-HOSTING.md`, "Upgrades and
21+
going back".
22+
23+
### Fixed
24+
25+
- **No more false "history fill failed".** The fill that builds the
26+
fifteen-minute rollups from history now checks its result against the raw
27+
events instead of the frozen hourly tables. In 0.7.0, an event the worker
28+
delivered from its queue after the switch — with a timestamp inside the
29+
filled history — reached the new tables and the raw events but not the
30+
hourly ones, and the first start logged `history fill backfill-15m failed`
31+
with `additive_mismatch` over data that was correct. The check now fails
32+
only when the filled history disagrees with the raw events, and still
33+
allows the one quarter hour per site that the views had already started
34+
writing when the fill ran. The session rollups are checked against the
35+
session facts in the same way.
36+
- **A site deletion started before an upgrade finishes after it.** A deletion
37+
records its list of tables when it starts; a table dropped by a later
38+
migration is now settled as empty instead of being retried forever.
39+
40+
### Removed
41+
42+
- **The hourly rollup tables.** `metrics_1h`, `pages_1h`, `sources_1h`,
43+
`geography_1h`, `devices_1h`, `custom_events_1h`, `performance_1h`,
44+
`custom_event_samples_1h`, `session_rollups_1h` and `revenue_1h`
45+
(ClickHouse migration 0029), with their ClickHouse grants and their entries
46+
in site deletion. Nothing has read them since 0.7.0; the daily tables stay.
47+
1148
## [0.7.0] - 2026-09-18
1249

1350
**Upgrade notes: nothing by hand, but the first start takes longer.**
@@ -22,7 +59,8 @@ views are created counts only the events that arrived after that moment, and
2259
the first start can log one `history fill backfill-15m failed` with
2360
`additive_mismatch` (late events reach the new tables but not the frozen
2461
hourly ones the fill checks against; the history is complete and the line
25-
does not repeat). The new rollups take up to four rows for every hourly one.
62+
does not repeat; fixed in 0.8.0). The new rollups take up to four rows for
63+
every hourly one.
2664
The new ClickHouse grants arrive with the new image. Details in `SELF-HOSTING.md`, "Upgrades and
2765
going back".
2866

‎SELF-HOSTING.md‎

Lines changed: 31 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ part doing the work: a candidate publishes images under its own tag and sorts
7777
_above_ the release it is a candidate for, so `git tag --sort=-v:refname` lists
7878
`v0.1.0-rc.1` before `v0.1.0` and `git describe` would hand you the candidate.
7979
Dropping every tag with a `-` in it leaves only releases. To take a specific
80-
one, name it instead: `git checkout v0.7.0`.
80+
one, name it instead: `git checkout v0.8.0`.
8181

8282
Add `--with-geoip` to that last command to download the country and city
8383
database in the same pass — see [GeoIP](#geoip). It is the one thing in the
@@ -116,7 +116,7 @@ because you checked out the tag before running it:
116116
> your version over the baked one, which still wins.
117117
118118
```sh
119-
grep OA_IMAGE .env # ghcr.io/openlabs-so/openanalytics, v0.7.0
119+
grep OA_IMAGE .env # ghcr.io/openlabs-so/openanalytics, v0.8.0
120120
docker compose pull
121121
docker compose up -d
122122
docker compose logs -f migrate # schemas, both stores, from empty
@@ -643,7 +643,7 @@ Two things worth knowing before you rely on any of it:
643643

644644
```sh
645645
git fetch --tags
646-
git checkout v0.7.0 # the release you are moving to
646+
git checkout v0.8.0 # the release you are moving to
647647
cd infra/selfhost
648648
./upgrade.sh # tells you what it costs, then does it
649649
```
@@ -664,6 +664,31 @@ standing on, takes a snapshot, points `.env` at the new images, pulls them and
664664
brings everything up. On an architecture with no published images,
665665
`./upgrade.sh --from-source` builds instead.
666666

667+
**0.7.0 → 0.8.0 asks nothing else of you either**, and it is short:
668+
669+
- **The hourly rollup tables are dropped.** ClickHouse migration 0029 removes
670+
the ten `*_1h` tables 0.7.0 stopped writing; nothing had read them since
671+
0.7.0 moved every chart to the fifteen-minute rollups. Site deletion stops
672+
naming them in the same release. The `*_1d` tables stay.
673+
- **The history fill checks itself against your raw events.** In 0.7.0 it
674+
compared its sums with the frozen hourly tables, so an upgrade with the
675+
worker still running could log one `history fill backfill-15m failed` with
676+
`additive_mismatch` over history that was complete. It now compares with
677+
`events_raw`, which every rollup is built from, so that line no longer
678+
appears for that reason — when it does appear, the filled history really
679+
disagrees with the raw events, and the report above it says where.
680+
- **Going back from 0.8.0 is a restore, not an image tag.** 0029 deletes the
681+
hourly tables, so the 0.7.0 images would find tables missing. Use
682+
`./rollback.sh` to the snapshot `./upgrade.sh` took, as below.
683+
684+
**Skipping 0.7.0 (0.6.0 → 0.8.0) is supported and asks nothing extra.** The
685+
migrate container applies ClickHouse migrations 0025 through 0029 in one run
686+
and fills the fifteen-minute rollups from your raw events afterwards, so
687+
everything 0.7.0 describes below happens too — only the hourly tables are gone
688+
by the time the fill runs. That is safe because the fill reads `events_raw`
689+
and checks itself against `events_raw`; no step of it needs an hourly table.
690+
Read the 0.7.0 notes below for what the fill costs on a large install.
691+
667692
**0.6.0 → 0.7.0 asks nothing else of you either, but it does more while it
668693
runs**, so it is worth knowing what you are watching:
669694

@@ -686,12 +711,9 @@ runs**, so it is worth knowing what you are watching:
686711
- **If the worker keeps running through the upgrade** — a platform redeploy, or
687712
`docker compose pull && docker compose up -d` without `./upgrade.sh` — one
688713
quarter hour can come out short: the one in which the new views were created
689-
counts only the events that arrived after that moment. The first start can
690-
also log one `history fill backfill-15m failed` with `additive_mismatch`:
691-
the fill checks its sums against the hourly tables, and late events that
692-
arrive after the switch reach the new tables but not the frozen hourly ones.
693-
The history itself is complete, and the next start does not repeat the line.
694-
`./upgrade.sh` stops everything first, so it has neither.
714+
counts only the events that arrived after that moment. `./upgrade.sh`
715+
stops everything first, so it does not have that. (0.7.0 could also log one
716+
false `additive_mismatch` here; 0.8.0 no longer does.)
695717
- **ClickHouse gets new grants**, for the two rollups the worker writes itself.
696718
They are rendered when its container is created, and the new image tag is
697719
what recreates it, so `./upgrade.sh` and a platform redeploy deliver them

‎apps/collector/src/events.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -453,7 +453,7 @@ function buildPersistedEvent(input: BuildPersistedInput): PersistedEvent {
453453
const page = event.page ? sanitizeUrl(event.page.url, { redactQueryKeys }) : null
454454
// The site's own hosts are not an acquisition source, and one source has one
455455
// spelling (ADR-0028). Both rules are applied here, on the way in, because
456-
// `referrer_domain` is a rollup grouping key: `sources_1h` has already grouped
456+
// `referrer_domain` is a rollup grouping key: `sources_15m` has already grouped
457457
// by it before any reader exists to repair it.
458458
const referrer = resolveReferrer(event.referrer, {
459459
siteDomains: config.allowedDomains,
@@ -466,7 +466,7 @@ function buildPersistedEvent(input: BuildPersistedInput): PersistedEvent {
466466
// of the channel is the click id the platform put in the landing URL. It is
467467
// read here, on the way in, for the reason ADR-0028 gives for resolving the
468468
// referrer here: `referrer_domain` is the rollup grouping key and
469-
// `sources_1h` has already grouped by it before any reader exists to repair
469+
// `sources_15m` has already grouped by it before any reader exists to repair
470470
// it. Repairing it at read time would have to be repeated in every future
471471
// reader — the sources report, the session fact's entry attribution, revenue
472472
// first/last touch — instead of once.

‎apps/worker/src/jobs/site-deletion.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import {
2+
DELETION_CLICKHOUSE_TARGETS,
23
DELETION_REDIS_TARGETS,
34
SITE_DELETION_PHASES,
45
isSiteDeletionPhase,
@@ -708,6 +709,8 @@ async function objectPurge(context: PhaseContext): Promise<PhaseOutcome> {
708709
* since ClickHouse queues mutations per table regardless of when they arrive.
709710
* The job runner's own `limit: 1` is what keeps *deletions* from overlapping.
710711
*/
712+
const RETAINED_CLICKHOUSE_TARGETS: ReadonlySet<string> = new Set(DELETION_CLICKHOUSE_TARGETS)
713+
711714
async function clickhousePurge(context: PhaseContext): Promise<PhaseOutcome> {
712715
const maintenance = context.job.resources?.clickhouse
713716
if (!maintenance) {
@@ -735,6 +738,28 @@ async function clickhousePurge(context: PhaseContext): Promise<PhaseOutcome> {
735738
for (const target of targets) {
736739
if (target.phase === 'completed') continue
737740

741+
// A target the vocabulary no longer names is a table a later migration
742+
// dropped — the ten hour tables, ClickHouse 0029 (v0.8.0) — while this
743+
// deletion's snapshot, taken at its start, still lists it. The table holds
744+
// no row of any site, so there is nothing to purge and nothing a mutation
745+
// could be submitted against: ClickHouse would refuse it on every attempt
746+
// and the deletion would never finish. Settled as verified-empty, with the
747+
// reason, and never submitted.
748+
if (!RETAINED_CLICKHOUSE_TARGETS.has(target.target)) {
749+
await completeTarget(context, target, {
750+
count: 0,
751+
mutation_id: null,
752+
retired: 'target left the deletion vocabulary: its table was dropped by a migration',
753+
verified_at: new Date().toISOString(),
754+
})
755+
context.job.logger.info('site_deletion_target_retired', {
756+
job_id: context.job.job.id,
757+
site_id: context.siteId,
758+
table: target.target,
759+
})
760+
continue
761+
}
762+
738763
let mutationId = target.mutationId
739764
// Whether ClickHouse has *reported* this mutation finished. It is not the
740765
// same question as "are the rows gone", and the difference is what the

‎apps/worker/src/revenue/rollup.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ const MS_PER_DAY = 86_400_000
8484

8585
// Two grains since ADR-0079 step 4: nothing has read `revenue_1h` since step 3
8686
// moved every composition to the quarter, so the hour swap was a third of this
87-
// job's writes for no reader. The table is frozen, not dropped.
87+
// job's writes for no reader. Migration 0029 dropped the table (v0.8.0).
8888
const UNITS: readonly RevenueRollupUnit[] = ['15m', '1d']
8989

9090
export interface RevenueRollupDeps {

‎apps/worker/src/sessions/finalizer.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -192,8 +192,8 @@ export async function finalizeSite(
192192
let rollupSwaps = 0
193193
// Two grains since ADR-0079 step 4 (migration 0027): the hour rollup was
194194
// read by nothing after step 3 moved every composition to the quarter, so
195-
// writing it was a third of this loop's work for nobody. `session_rollups_1h`
196-
// still exists and still holds what it held -- it is simply frozen.
195+
// writing it was a third of this loop's work for nobody. Migration 0029
196+
// (v0.8.0) then dropped `session_rollups_1h` itself.
197197
const unitBuckets: Record<SessionRollupUnit, readonly number[]> = {
198198
'15m': plan.affectedQuarterBucketsMs,
199199
'1d': plan.affectedDayBucketsMs,

‎infra/selfhost/clickhouse/oa-entrypoint.sh‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,11 +84,9 @@ cat >"$TARGET" <<XML
8484
<grants>
8585
<query>GRANT INSERT ON analytics.events_raw</query>
8686
<query>GRANT INSERT ON analytics.session_facts_versions</query>
87-
<query>GRANT INSERT ON analytics.session_rollups_1h</query>
8887
<query>GRANT INSERT ON analytics.session_rollups_1d</query>
8988
<query>GRANT SELECT ON analytics.events_raw</query>
9089
<query>GRANT SELECT ON analytics.session_facts_versions</query>
91-
<query>GRANT SELECT ON analytics.session_rollups_1h</query>
9290
<query>GRANT SELECT ON analytics.session_rollups_1d</query>
9391
9492
<!-- Import staging targets the prepare job writes. -->
@@ -111,8 +109,6 @@ cat >"$TARGET" <<XML
111109
<query>GRANT SELECT ON analytics.revenue_events</query>
112110
<query>GRANT INSERT ON analytics.revenue_attributions</query>
113111
<query>GRANT SELECT ON analytics.revenue_attributions</query>
114-
<query>GRANT INSERT ON analytics.revenue_1h</query>
115-
<query>GRANT SELECT ON analytics.revenue_1h</query>
116112
<query>GRANT INSERT ON analytics.revenue_1d</query>
117113
<query>GRANT SELECT ON analytics.revenue_1d</query>
118114
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
-- ADR-0079 step 4, completed (v0.8.0): the hour rollup TABLES go.
2+
--
3+
-- Migration 0027 dropped the eight hour views and kept their tables, for two
4+
-- reasons it named. Both are gone now:
5+
--
6+
-- * the 15m backfill's equality gate compared each family against its hour
7+
-- twin. It now compares against `events_raw` itself -- the table every view
8+
-- reads, which is never behind. (The hour twin WAS behind: 0027 froze it,
9+
-- so any event delivered after 0027 -- a queue backlog drained during the
10+
-- upgrade -- sat in the 15m rows and not in the reference, and the gate
11+
-- reported a failed fill over data that was right.)
12+
-- * the deletion workflow named these tables as targets. It stops naming
13+
-- them in the same release (`packages/domain/src/deletion.ts`, 44 -> 34
14+
-- ClickHouse targets), which is the only order that works: a target naming
15+
-- a table that no longer exists fails every deletion, and the rows a target
16+
-- would purge leave with the table.
17+
--
18+
-- Ten tables: the eight view targets of 0006-0012 and 0021, plus the two swap
19+
-- targets of 0014 and 0018 that the worker stopped writing in the same step
20+
-- (`'1h'` left both unit unions). Nothing reads any of them: step 3 moved every
21+
-- read to the fifteen-minute family, and the compiler has refused a write since
22+
-- step 4.
23+
--
24+
-- The `_1d` family stays. It is read directly for every UTC day and week, and
25+
-- its views keep firing.
26+
--
27+
-- `IF EXISTS` so an install that never had one of them (none should, but a
28+
-- hand-dropped table is not a reason to stop the stack) passes. An install
29+
-- skipping 0.7.0 applies 0025-0029 in one run: the 15m family is created by
30+
-- 0025/0026 and filled from `events_raw` by the migrate container after the
31+
-- migrations, and nothing in that fill looks at an hour table any more, so
32+
-- dropping them first is safe.
33+
--
34+
-- Reversal, if the hour grain is ever wanted back: this migration deletes
35+
-- data, so it is not reversed by running an older image. Re-create the tables
36+
-- and views from 0006-0012, 0014, 0018 and 0021 as a new migration and replay
37+
-- from `events_raw` (views) and the session facts / revenue facts (swaps), all
38+
-- of which are kept. A self-hosted install restores the pre-upgrade backup
39+
-- instead (`./rollback.sh`).
40+
41+
DROP TABLE IF EXISTS metrics_1h;
42+
43+
DROP TABLE IF EXISTS pages_1h;
44+
45+
DROP TABLE IF EXISTS sources_1h;
46+
47+
DROP TABLE IF EXISTS geography_1h;
48+
49+
DROP TABLE IF EXISTS devices_1h;
50+
51+
DROP TABLE IF EXISTS custom_events_1h;
52+
53+
DROP TABLE IF EXISTS performance_1h;
54+
55+
DROP TABLE IF EXISTS custom_event_samples_1h;
56+
57+
DROP TABLE IF EXISTS session_rollups_1h;
58+
59+
DROP TABLE IF EXISTS revenue_1h;

0 commit comments

Comments
 (0)