docs(cutover): frame cross-replica ON CLUSTER skew for multi-replica production + maintenance note

The prior comment-precision pass described the cross-host ON CLUSTER case as "a non-issue in
the default single-node deployment", which downplays it: production is multi-replica. Correct
the framing in the stage-B/C rollback SQL, finalize.sh and the runbook, and add the operator
maintenance note.

- Cross-replica: ON CLUSTER runs synchronously (the client blocks until every reachable replica
  applies it, or throws naming a laggard that then converges via the DDL queue), so there is no
  durable mixed topology — only a sub-second cross-replica skew as it propagates, the same
  accepted skew as the wrap. Not globally atomic (no ClickHouse primitive for that); nil on a
  single replica.
- Rollback stages B/C rename the LIVE `traces`, so reads hit the skew: rollback.sh now prints a
  maintenance NOTE and the runbook says to run the rollback in a maintenance moment / with reads
  quiesced on a multi-replica cluster.
- finalize is EXEMPT: its TRUNCATE+RENAME touch only the parked backup / disposable shadow, never
  the live `traces`, so there is no live-read skew and no maintenance window is needed — stated
  explicitly in finalize.sh and the runbook.

Comment/doc-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: andrescrz

apps/opik-backend/data-migrations/traces-local-v2-cutover/README.md

@@ -481,6 +481,14 @@ Pick the stage by how far the cutover got (`cutover_start` is the value `exchang
   `RENAME` promotes the original (`traces_pre_cutover_backup`) back to `traces` and parks the successor as
   `traces_post_rollback_backup`, then the reverse replay. (Guarded: aborts unless `traces` is `Distributed`.)
 
+> **Multi-replica note (production is multi-replica).** Stages B and C promote via a single `ON CLUSTER` RENAME of the
+> **live** `traces`. It runs synchronously across the shard's replicas — the client blocks until each applies it, or fails
+> loudly naming a laggard, which then converges via the DDL queue — so there is no durable mixed topology, only a brief
+> sub-second cross-replica skew as it propagates, during which a read on a not-yet-renamed replica sees the pre-rollback
+> `traces`. This is the same accepted `ON CLUSTER` skew as the wrap; on a multi-replica cluster run the rollback in a
+> maintenance moment / with reads quiesced. `finalize.sh` is **exempt** — it renames only the parked backup / disposable
+> shadow, never the live `traces`, so it has no live-read skew and needs no maintenance window.
+
 **Recovering from an interrupted rollback.** Each promote stage runs its table-swap and then the reverse-replay as two
 statements, so a failure *between* them needs a restart path:
 

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/db-app-analytics/000004_rollback_stage_b_exchange_back.sql

@@ -8,10 +8,12 @@
 -- restoring the canonical state: traces = original live, traces_post_rollback_backup = successor parked. Gapless and with
 -- no orphan risk: the multi-target RENAME is one statement, atomic PER HOST (all clauses apply or none), so on any node
 -- there is no window where a partial failure strands the successor under a wrong name — the flaw of a separate
--- EXCHANGE + RENAME, which has that window even on a single host. ACROSS hosts, ON CLUSTER is serialized through the DDL
--- queue and eventually-consistent, NOT globally atomic: a briefly lagging host converges on its own (or the ON CLUSTER
--- call fails loudly naming it) — the same accepted skew as any ON CLUSTER DDL, and a non-issue in the default single-node
--- deployment (one host). Non-destructive.
+-- EXCHANGE + RENAME, which has that window even on a single host. ACROSS the shard's replicas (production is multi-replica)
+-- ON CLUSTER runs synchronously — the client blocks until every reachable replica applies it, or throws naming a laggard
+-- that then converges via the DDL queue — so there is no durable mixed topology, only a sub-second cross-replica skew as
+-- it propagates, during which a read on a not-yet-renamed replica sees the old `traces`. That is the same accepted
+-- ON CLUSTER skew as the wrap (nil on a single replica); run this in the rollback maintenance moment (see the runbook).
+-- Non-destructive.
 -- rollback.sh runs the reverse-replay (000004_rollback_reverse_replay.sql) right after this so deletes since
 -- cutover_start do not resurrect. rollback.sh asserts the post-EXCHANGE, pre-wrap topology (traces = successor schema,
 -- not Distributed) before running it.

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/db-app-analytics/000004_rollback_stage_c_promote_original.sql

@@ -8,9 +8,11 @@
 -- `traces` (the name freed by the first clause), and the successor shard (`traces_local`) parks as
 -- `traces_post_rollback_backup` (a retained backup, dropped only by finalize.sh — NOT the disposable `traces_local_v2`
 -- shadow) — ending in the canonical state (traces = original live, traces_post_rollback_backup = successor parked). So
--- `traces` is never absent on a node. ACROSS hosts, as for any ON CLUSTER DDL, the rename is eventually-consistent (a
--- lagging host converges on its own, or the call fails loudly naming it), NOT globally atomic — a non-issue in the
--- default single-node deployment (one host).
+-- `traces` is never absent on a node. ACROSS the shard's replicas (production is multi-replica) ON CLUSTER runs
+-- synchronously — the client blocks until every reachable replica applies it, or throws naming a laggard that then
+-- converges via the DDL queue — NOT globally atomic, so the only exposure is a sub-second cross-replica skew as it
+-- propagates, during which a read on a not-yet-renamed replica sees the pre-rollback `traces` (the same accepted
+-- ON CLUSTER skew as the wrap; nil on a single replica). Run this in the rollback maintenance moment (see the runbook).
 --
 -- Then drop the ex-wrapper. It is dropped under `traces_dist_old` — a fresh name that ONLY the data-less wrapper ever
 -- occupied — so the DROP cannot hit the original data regardless of per-replica DDL timing (the concern with dropping a

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/finalize.sh

@@ -89,10 +89,11 @@ if [[ "$BACKUP" == "traces_post_rollback_backup" ]]; then
     # is fixed at CREATE and unchanged by the rename) — back into an empty `traces_local_v2`. ClickHouse has no single
     # truncate-and-rename, so this is two statements, each atomic PER HOST and ON CLUSTER; ordered TRUNCATE-then-RENAME so
     # the only state a crash between them can leave is an empty `traces_post_rollback_backup`, which re-running finalize
-    # recovers (RENAME-first could strand a populated `traces_local_v2` that a retry backfill would mis-skip). ACROSS hosts,
-    # ON CLUSTER is eventually-consistent, NOT globally atomic — the same accepted skew as any ON CLUSTER DDL and a
-    # non-issue in the default single-node deployment (one host): a lagging host converges on its own via the DDL queue,
-    # or the ON CLUSTER call fails loudly naming it.
+    # recovers (RENAME-first could strand a populated `traces_local_v2` that a retry backfill would mis-skip). ACROSS the
+    # shard's replicas ON CLUSTER runs synchronously (the client blocks until every reachable replica applies it, or throws
+    # naming a laggard that then converges via the DDL queue), NOT globally atomic. Both statements touch only the parked
+    # backup / disposable shadow — never the live `traces` — so unlike the rollback promote and the wrap (which rename live
+    # `traces`) the brief cross-replica skew is invisible to readers, and finalize needs no maintenance window.
     if [[ "$CONFIRM" != "1" ]]; then
         echo "DRY RUN: would recycle $DATABASE.$BACKUP into an empty $DATABASE.traces_local_v2 (TRUNCATE + RENAME)."
         echo "         Re-run with --confirm."

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/rollback.sh

@@ -203,6 +203,9 @@ assert_topology
 if [[ "$STAGE" == "B" || "$STAGE" == "C" ]]; then
     echo "NOTE: promoting the frozen backup now. Traces the successor accepted after cutover_start ($CUTOVER_START) will" >&2
     echo "      stop being live; recover them from the parked traces_post_rollback_backup (kept until finalize.sh) if needed." >&2
+    echo "NOTE: the promote is a single ON CLUSTER RENAME of the live 'traces' — synchronous across the shard's replicas, but" >&2
+    echo "      with a brief sub-second cross-replica read skew as it propagates (a read on a lagging replica sees the" >&2
+    echo "      pre-rollback 'traces'). On a multi-replica cluster run this in a maintenance moment / with reads quiesced, as for the wrap." >&2
 fi
 
 case "$STAGE" in