Skip to content

Commit 2ba13ed

Browse files
andrescrzclaude
andcommitted
fix(cutover): make finalize.sh empty-row guard cluster-aware
Follow-up to the cluster-aware detection: the LIVE_ROWS/BACKUP_ROWS emptiness guard still counted on the connected replica only, so the "refuse if live is empty but the backup holds data" check reflected one node, not the cluster — inconsistent with the clusterAllReplicas detection in the same script (Baz review). Add a max_rows() helper that takes the max row count across replicas (clusterAllReplicas grouped by host; fail-loud on a down replica, like classify) and use it for both counts. Behavior is unchanged on a single replica (max over one host = that host's count); on multi-replica it now reflects the most-caught-up replica, so a lagging/empty connected node can no longer skew the guard. The post-cutover DROP path's Distributed `traces` already aggregates the cluster and the max-over-hosts still yields its true total. Guard still fails safe either way. Validated live: max_rows returns the correct counts against the single-node estate; bash -n clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent df6ee20 commit 2ba13ed

1 file changed

Lines changed: 14 additions & 5 deletions

File tree

  • apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts

apps/opik-backend/data-migrations/traces-local-v2-cutover/scripts/finalize.sh

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,9 @@
1818
# Detection is CLUSTER-WIDE (via clusterAllReplicas, like exchange_and_wrap.sh's settle gate): because finalize is the one
1919
# irreversible step and production is multi-replica, a name present on only SOME replicas means an ON CLUSTER DDL has not
2020
# finished propagating, so acting on the connected node's partial view could recycle/drop mid-transition — it refuses
21-
# loudly instead. It also refuses if the live `traces` is empty while the backup is not (the live table may be unhealthy
22-
# and the "backup" the only copy), if BOTH parked names exist (an ambiguous state a human must resolve), and — before a
21+
# loudly instead. It also refuses if the live `traces` is empty ACROSS THE CLUSTER (max rows over replicas) while the
22+
# backup is not (the live table may be unhealthy and the "backup" the only copy), if BOTH parked names exist (an
23+
# ambiguous state a human must resolve), and — before a
2324
# recycle — if `traces_local_v2` already exists (recycle renames the backup INTO that name; a stray shadow means a retry
2425
# cutover started before the rollback was finalized).
2526
#
@@ -76,6 +77,14 @@ classify() {
7677
fi
7778
}
7879

80+
# Row counts as the MAX across replicas (per-host via clusterAllReplicas), so the emptiness guard below reflects the whole
81+
# cluster, not just the connected node — consistent with classify. A Replicated table returns a full copy per replica, so
82+
# group by host and take the most-caught-up one; the post-cutover DROP path's Distributed `traces` already aggregates the
83+
# cluster and this still yields its true total. Fail-loud on a down replica, like classify.
84+
max_rows() {
85+
ch "SELECT max(c) FROM (SELECT count() AS c FROM clusterAllReplicas('$CLUSTER', $DATABASE.$1) GROUP BY hostName())"
86+
}
87+
7988
classify traces
8089
[[ "$CLUSTER_HAS" == "1" ]] || { echo "ERROR: live 'traces' table not found on all replicas in '$DATABASE'." >&2; exit 1; }
8190

@@ -97,10 +106,10 @@ else
97106
exit 0
98107
fi
99108

100-
LIVE_ROWS="$(ch "SELECT count() FROM traces")"
101-
BACKUP_ROWS="$(ch "SELECT count() FROM $BACKUP")"
109+
LIVE_ROWS="$(max_rows traces)"
110+
BACKUP_ROWS="$(max_rows "$BACKUP")"
102111

103-
# Refuse the dangerous case: a live table that looks empty while the backup holds data.
112+
# Refuse the dangerous case: a live table that looks empty across the cluster while the backup holds data.
104113
if [[ "$LIVE_ROWS" == "0" && "$BACKUP_ROWS" != "0" ]]; then
105114
echo "ERROR: live 'traces' is empty but '$BACKUP' has $BACKUP_ROWS rows. Refusing to drop the backup —" >&2
106115
echo " verify the live table is the healthy one before finalizing." >&2

0 commit comments

Comments
 (0)